== sanity test 123ad: Verify batching statahead works correctly ========================================================== 10:55:37 (1786805737) batch: statahead_max=32 statahead_batch_max=32 mdc.lustre-MDT0000-mdc-ffff9f64d0b0c800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9f64d0b0c800.batch_stats=0 llite.lustre-ffff9f64d0b0c800.statahead_max=32 llite.lustre-ffff9f64d0b0c800.statahead_batch_max=32 seq.cli-lustre-OST0000-super.width=0x1ffffff seq.cli-lustre-OST0001-super.width=0x1ffffff kvm mdc.lustre-MDT0000-mdc-ffff9f64d0b0c800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9f64d0b0c800.batch_stats=0 striped dir -i1 -c2 -H crush /mnt/lustre/d123ad.sanity total: 100 open/close in 2.30 seconds: 43.46 ops/second llite.lustre-ffff9f64d0b0c800.statahead_max=0 101 real 0m2.758s user 0m0.002s sys 0m0.351s 'ls -l' 100 files without statahead: 3 sec llite.lustre-ffff9f64d0b0c800.statahead_max=32 llite.lustre-ffff9f64d0b0c800.statahead_max=32 101 real 0m1.004s user 0m0.003s sys 0m0.189s 'ls -l' 100 files with statahead: 1 sec statahead total: 105 statahead wrong: 1 agl total: 98 list_total: 31 fname_total: 72 hit_total: 3583 miss_total: 256 swrong=1 ewrong=1 - open/close 890 (time 1786805764.92 total 10.00 last 88.98) total: 900 open/close in 10.06 seconds: 89.42 ops/second llite.lustre-ffff9f64d0b0c800.statahead_max=0 1001 real 0m17.110s user 0m0.032s sys 0m3.099s 'ls -l' 1000 files without statahead: 17 sec llite.lustre-ffff9f64d0b0c800.statahead_max=32 llite.lustre-ffff9f64d0b0c800.statahead_max=32 1001 real 0m5.903s user 0m0.024s sys 0m1.571s 'ls -l' 1000 files with statahead: 6 sec statahead total: 106 statahead wrong: 1 agl total: 99 list_total: 32 fname_total: 72 hit_total: 4581 miss_total: 258 swrong=1 ewrong=1 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 18 seconds llite.lustre-ffff9f64d0b0c800.statahead_stats= statahead total: 106 statahead wrong: 1 agl total: 99 list_total: 32 fname_total: 72 hit_total: 4581 miss_total: 258 mdc.lustre-MDT0000-mdc-ffff9f64d0b0c800.batch_stats= snapshot_time: 1786805815.031737033 secs.nsecs start_time: 1786805742.490842543 secs.nsecs elapsed_time: 72.540894490 secs.nsecs subreqs per batch batches %% cum %% 1: 1 2 2 2: 2 5 8 4: 2 5 13 8: 2 5 18 16: 27 72 91 32: 3 8 100 mdc.lustre-MDT0001-mdc-ffff9f64d0b0c800.batch_stats= snapshot_time: 1786805815.031809173 secs.nsecs start_time: 1786805742.490937441 secs.nsecs elapsed_time: 72.540871732 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 2 5 5 4: 1 2 7 8: 4 10 17 16: 8 20 37 32: 25 62 100 - open/close 577 (time 1786805828.30 total 10.02 last 57.59) - open/close 1334 (time 1786805838.30 total 20.02 last 75.68) - open/close 2130 (time 1786805848.30 total 30.02 last 79.58) - open/close 3070 (time 1786805858.31 total 40.03 last 93.97) - open/close 3973 (time 1786805868.31 total 50.03 last 90.25) - open/close 4888 (time 1786805878.32 total 60.04 last 91.46) - open/close 5967 (time 1786805888.32 total 70.04 last 107.85) - open/close 6974 (time 1786805898.35 total 80.07 last 100.45) - open/close 7998 (time 1786805908.35 total 90.07 last 102.40) - open/close 9037 (time 1786805918.35 total 100.07 last 103.87) total: 10000 open/close in 109.36 seconds: 91.44 ops/second llite.lustre-ffff9f64d0b0c800.statahead_batch_max=0 llite.lustre-ffff9f64d0b0c800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9f64d0b0c800.stats=clear mdc.lustre-MDT0001-mdc-ffff9f64d0b0c800.stats=clear ls -l files in d123ad.sanity: 10001 real 1m11.931s user 0m0.304s sys 0m16.459s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff9f64d0b0c800.statahead_batch_max=32 llite.lustre-ffff9f64d0b0c800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9f64d0b0c800.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff9f64d0b0c800.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff9f64d0b0c800.stats=clear mdc.lustre-MDT0001-mdc-ffff9f64d0b0c800.stats=clear ls -l files in d123ad.sanity: 10001 real 1m40.591s user 0m0.306s sys 0m15.923s batched statahead RPC= hit count=9979 mdc.lustre-MDT0000-mdc-ffff9f64d0b0c800.batch_stats= snapshot_time: 1786806117.355261732 secs.nsecs start_time: 1786806006.618525634 secs.nsecs elapsed_time: 110.736736098 secs.nsecs subreqs per batch batches %% cum %% 1: 1 0 0 2: 0 0 0 4: 0 0 0 8: 1 0 0 16: 1 0 0 32: 313 99 100 mdc.lustre-MDT0001-mdc-ffff9f64d0b0c800.batch_stats= snapshot_time: 1786806117.355321396 secs.nsecs start_time: 1786806006.618762951 secs.nsecs elapsed_time: 110.736558445 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 batch: statahead_max=2048 statahead_batch_max=256 mdc.lustre-MDT0000-mdc-ffff9f64d0b0c800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9f64d0b0c800.batch_stats=0 llite.lustre-ffff9f64d0b0c800.statahead_max=2048 llite.lustre-ffff9f64d0b0c800.statahead_batch_max=256 kvm mdc.lustre-MDT0000-mdc-ffff9f64d0b0c800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9f64d0b0c800.batch_stats=0 striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d123ad.sanity total: 100 open/close in 2.21 seconds: 45.18 ops/second llite.lustre-ffff9f64d0b0c800.statahead_max=0 101 real 0m2.049s user 0m0.003s sys 0m0.340s 'ls -l' 100 files without statahead: 2 sec llite.lustre-ffff9f64d0b0c800.statahead_max=2048 llite.lustre-ffff9f64d0b0c800.statahead_max=2048 101 real 0m0.875s user 0m0.007s sys 0m0.188s 'ls -l' 100 files with statahead: 1 sec statahead total: 2 statahead wrong: 0 agl total: 2 list_total: 2 fname_total: 0 hit_total: 10078 miss_total: 22 swrong=0 ewrong=0 total: 900 open/close in 9.96 seconds: 90.32 ops/second llite.lustre-ffff9f64d0b0c800.statahead_max=0 1001 real 0m23.220s user 0m0.036s sys 0m3.154s 'ls -l' 1000 files without statahead: 23 sec llite.lustre-ffff9f64d0b0c800.statahead_max=2048 llite.lustre-ffff9f64d0b0c800.statahead_max=2048 1001 real 0m6.804s user 0m0.025s sys 0m1.684s 'ls -l' 1000 files with statahead: 7 sec statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11077 miss_total: 23 swrong=0 ewrong=0 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 17 seconds llite.lustre-ffff9f64d0b0c800.statahead_stats= statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11077 miss_total: 23 mdc.lustre-MDT0000-mdc-ffff9f64d0b0c800.batch_stats= snapshot_time: 1786806447.469286555 secs.nsecs start_time: 1786806119.805071427 secs.nsecs elapsed_time: 327.664215128 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 1 11 11 4: 0 0 11 8: 3 33 44 16: 2 22 66 32: 1 11 77 64: 0 0 77 128: 0 0 77 256: 2 22 100 mdc.lustre-MDT0001-mdc-ffff9f64d0b0c800.batch_stats= snapshot_time: 1786806447.469355307 secs.nsecs start_time: 1786806119.805273407 secs.nsecs elapsed_time: 327.664081900 secs.nsecs subreqs per batch batches %% cum %% 1: 1 11 11 2: 0 0 11 4: 2 22 33 8: 3 33 66 16: 0 0 66 32: 0 0 66 64: 1 11 77 128: 0 0 77 256: 2 22 100 - open/close 886 (time 1786806459.54 total 10.00 last 88.58) - open/close 1876 (time 1786806469.55 total 20.01 last 98.92) - open/close 2925 (time 1786806479.55 total 30.02 last 104.85) - open/close 3981 (time 1786806489.55 total 40.02 last 105.57) - open/close 5086 (time 1786806499.56 total 50.03 last 110.43) - open/close 6028 (time 1786806509.57 total 60.03 last 94.14) - open/close 7001 (time 1786806519.57 total 70.03 last 97.28) - open/close 7831 (time 1786806529.58 total 80.04 last 82.94) - open/close 8876 (time 1786806539.58 total 90.05 last 104.45) - open/close 9932 (time 1786806549.59 total 100.05 last 105.55) total: 10000 open/close in 100.70 seconds: 99.31 ops/second llite.lustre-ffff9f64d0b0c800.statahead_batch_max=0 llite.lustre-ffff9f64d0b0c800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9f64d0b0c800.stats=clear mdc.lustre-MDT0001-mdc-ffff9f64d0b0c800.stats=clear ls -l files in d123ad.sanity: 10001 real 0m49.142s user 0m0.289s sys 0m19.137s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff9f64d0b0c800.statahead_batch_max=256 llite.lustre-ffff9f64d0b0c800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9f64d0b0c800.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff9f64d0b0c800.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff9f64d0b0c800.stats=clear mdc.lustre-MDT0001-mdc-ffff9f64d0b0c800.stats=clear ls -l files in d123ad.sanity: 10001 real 0m54.053s user 0m0.289s sys 0m16.976s batched statahead RPC= hit count=9999 mdc.lustre-MDT0000-mdc-ffff9f64d0b0c800.batch_stats= snapshot_time: 1786806674.188065173 secs.nsecs start_time: 1786806607.357831836 secs.nsecs elapsed_time: 66.830233337 secs.nsecs subreqs per batch batches %% cum %% 1: 140 48 48 2: 39 13 62 4: 26 9 71 8: 16 5 77 16: 13 4 81 32: 9 3 84 64: 7 2 87 128: 2 0 88 256: 34 11 100 mdc.lustre-MDT0001-mdc-ffff9f64d0b0c800.batch_stats= snapshot_time: 1786806674.188158933 secs.nsecs start_time: 1786806607.357998249 secs.nsecs elapsed_time: 66.830160684 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second seq.cli-lustre-OST0000-super.width=65536 seq.cli-lustre-OST0001-super.width=65536 llite.lustre-ffff9f64d0b0c800.statahead_batch_max=64 llite.lustre-ffff9f64d0b0c800.statahead_max=128