== sanity test 123ad: Verify batching statahead works correctly ========================================================== 19:15:24 (1785885324) batch: statahead_max=32 statahead_batch_max=32 mdc.lustre-MDT0000-mdc-ffff9e803df57000.batch_stats=0 llite.lustre-ffff9e803df57000.statahead_max=32 llite.lustre-ffff9e803df57000.statahead_batch_max=32 seq.cli-lustre-OST0000-super.width=0x1ffffff seq.cli-lustre-OST0001-super.width=0x1ffffff kvm mdc.lustre-MDT0000-mdc-ffff9e803df57000.batch_stats=0 total: 100 open/close in 4.58 seconds: 21.82 ops/second llite.lustre-ffff9e803df57000.statahead_max=0 101 real 0m5.154s user 0m0.005s sys 0m0.350s 'ls -l' 100 files without statahead: 5 sec llite.lustre-ffff9e803df57000.statahead_max=32 llite.lustre-ffff9e803df57000.statahead_max=32 101 real 0m2.971s user 0m0.005s sys 0m0.200s 'ls -l' 100 files with statahead: 3 sec statahead total: 164 statahead wrong: 2 agl total: 157 list_total: 30 fname_total: 134 hit_total: 3609 miss_total: 1668 swrong=2 ewrong=2 - open/close 404 (time 1785885368.38 total 10.02 last 40.31) total: 900 open/close in 19.83 seconds: 45.38 ops/second llite.lustre-ffff9e803df57000.statahead_max=0 1001 real 0m31.291s user 0m0.035s sys 0m3.183s 'ls -l' 1000 files without statahead: 31 sec llite.lustre-ffff9e803df57000.statahead_max=32 llite.lustre-ffff9e803df57000.statahead_max=32 1001 real 0m18.683s user 0m0.025s sys 0m1.583s 'ls -l' 1000 files with statahead: 19 sec statahead total: 165 statahead wrong: 2 agl total: 158 list_total: 31 fname_total: 134 hit_total: 4602 miss_total: 1675 swrong=2 ewrong=2 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 29 seconds llite.lustre-ffff9e803df57000.statahead_stats= statahead total: 165 statahead wrong: 2 agl total: 158 list_total: 31 fname_total: 134 hit_total: 4602 miss_total: 1675 mdc.lustre-MDT0000-mdc-ffff9e803df57000.batch_stats= snapshot_time: 1785885481.561668828 secs.nsecs start_time: 1785885333.017522502 secs.nsecs elapsed_time: 148.544146326 secs.nsecs subreqs per batch batches %% cum %% 1: 1 2 2 2: 0 0 2 4: 0 0 2 8: 2 5 7 16: 3 7 15 32: 34 85 100 - open/close 340 (time 1785885495.59 total 10.03 last 33.88) - open/close 732 (time 1785885505.59 total 20.04 last 39.19) - open/close 1192 (time 1785885515.60 total 30.05 last 45.96) - open/close 1716 (time 1785885525.61 total 40.05 last 52.37) - open/close 2265 (time 1785885535.61 total 50.05 last 54.89) - open/close 2824 (time 1785885545.62 total 60.06 last 55.84) - open/close 3335 (time 1785885555.63 total 70.07 last 51.05) - open/close 3848 (time 1785885565.63 total 80.08 last 51.27) - open/close 4267 (time 1785885575.63 total 90.08 last 41.89) - open/close 4752 (time 1785885585.66 total 100.11 last 48.37) - open/close 5178 (time 1785885595.67 total 110.12 last 42.56) - open/close 5735 (time 1785885605.68 total 120.13 last 55.63) - open/close 6112 (time 1785885615.70 total 130.15 last 37.62) - open/close 6667 (time 1785885625.71 total 140.16 last 55.47) - open/close 7532 (time 1785885635.72 total 150.17 last 86.42) - open/close 8213 (time 1785885645.73 total 160.17 last 68.06) - open/close 8848 (time 1785885655.74 total 170.19 last 63.41) - open/close 9380 (time 1785885665.76 total 180.21 last 53.09) - open/close 9861 (time 1785885675.77 total 190.22 last 48.05) total: 10000 open/close in 192.82 seconds: 51.86 ops/second llite.lustre-ffff9e803df57000.statahead_batch_max=0 llite.lustre-ffff9e803df57000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9e803df57000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m25.197s user 0m0.279s sys 0m14.583s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff9e803df57000.statahead_batch_max=32 llite.lustre-ffff9e803df57000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9e803df57000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff9e803df57000.stats=clear ls -l files in d123ad.sanity: 10001 real 2m52.700s user 0m0.290s sys 0m15.166s batched statahead RPC= hit count=9973 mdc.lustre-MDT0000-mdc-ffff9e803df57000.batch_stats= snapshot_time: 1785885968.264387744 secs.nsecs start_time: 1785885777.892321527 secs.nsecs elapsed_time: 190.372066217 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 0 0 0 8: 2 0 0 16: 1 0 0 32: 313 99 100 batch: statahead_max=2048 statahead_batch_max=256 mdc.lustre-MDT0000-mdc-ffff9e803df57000.batch_stats=0 llite.lustre-ffff9e803df57000.statahead_max=2048 llite.lustre-ffff9e803df57000.statahead_batch_max=256 kvm mdc.lustre-MDT0000-mdc-ffff9e803df57000.batch_stats=0 total: 100 open/close in 3.16 seconds: 31.68 ops/second llite.lustre-ffff9e803df57000.statahead_max=0 101 real 0m2.468s user 0m0.006s sys 0m0.307s 'ls -l' 100 files without statahead: 3 sec llite.lustre-ffff9e803df57000.statahead_max=2048 llite.lustre-ffff9e803df57000.statahead_max=2048 101 real 0m1.063s user 0m0.003s sys 0m0.153s 'ls -l' 100 files with statahead: 1 sec statahead total: 2 statahead wrong: 0 agl total: 2 list_total: 2 fname_total: 0 hit_total: 10072 miss_total: 28 swrong=0 ewrong=0 - open/close 364 (time 1785886403.52 total 10.01 last 36.36) - open/close 828 (time 1785886413.52 total 20.01 last 46.40) total: 900 open/close in 21.09 seconds: 42.67 ops/second llite.lustre-ffff9e803df57000.statahead_max=0 1001 real 0m28.567s user 0m0.036s sys 0m3.089s 'ls -l' 1000 files without statahead: 29 sec llite.lustre-ffff9e803df57000.statahead_max=2048 llite.lustre-ffff9e803df57000.statahead_max=2048 1001 real 0m8.969s user 0m0.021s sys 0m1.410s 'ls -l' 1000 files with statahead: 9 sec statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11071 miss_total: 29 swrong=0 ewrong=0 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 37 seconds llite.lustre-ffff9e803df57000.statahead_stats= statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11071 miss_total: 29 mdc.lustre-MDT0000-mdc-ffff9e803df57000.batch_stats= snapshot_time: 1785886508.424467912 secs.nsecs start_time: 1785885975.061125998 secs.nsecs elapsed_time: 533.363341914 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 0 0 0 8: 2 18 18 16: 2 18 36 32: 2 18 54 64: 0 0 54 128: 1 9 63 256: 4 36 100 - open/close 402 (time 1785886522.09 total 10.06 last 39.97) - open/close 941 (time 1785886532.10 total 20.08 last 53.80) - open/close 1559 (time 1785886542.11 total 30.08 last 61.79) - open/close 2190 (time 1785886552.11 total 40.08 last 63.10) - open/close 2737 (time 1785886562.11 total 50.08 last 54.69) - open/close 3214 (time 1785886572.13 total 60.10 last 47.60) - open/close 3881 (time 1785886582.14 total 70.11 last 66.64) - open/close 4465 (time 1785886592.14 total 80.11 last 58.39) - open/close 4965 (time 1785886602.15 total 90.13 last 49.92) - open/close 5427 (time 1785886612.17 total 100.14 last 46.12) - open/close 5884 (time 1785886622.17 total 110.15 last 45.69) - open/close 6453 (time 1785886632.17 total 120.15 last 56.90) - open/close 7003 (time 1785886642.20 total 130.17 last 54.87) - open/close 7606 (time 1785886652.20 total 140.17 last 60.27) - open/close 8089 (time 1785886662.22 total 150.19 last 48.22) - open/close 8460 (time 1785886672.24 total 160.21 last 37.02) - open/close 8932 (time 1785886682.25 total 170.22 last 47.17) - open/close 9509 (time 1785886692.26 total 180.23 last 57.65) total: 10000 open/close in 187.62 seconds: 53.30 ops/second llite.lustre-ffff9e803df57000.statahead_batch_max=0 llite.lustre-ffff9e803df57000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9e803df57000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m22.284s user 0m0.242s sys 0m14.413s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff9e803df57000.statahead_batch_max=256 llite.lustre-ffff9e803df57000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9e803df57000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff9e803df57000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m29.416s user 0m0.259s sys 0m14.893s batched statahead RPC= hit count=9999 mdc.lustre-MDT0000-mdc-ffff9e803df57000.batch_stats= snapshot_time: 1785886900.514171057 secs.nsecs start_time: 1785886794.221732505 secs.nsecs elapsed_time: 106.292438552 secs.nsecs subreqs per batch batches %% cum %% 1: 118 45 45 2: 36 13 59 4: 23 8 68 8: 16 6 74 16: 15 5 80 32: 12 4 85 64: 1 0 85 128: 1 0 86 256: 36 13 100 unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second seq.cli-lustre-OST0000-super.width=65536 seq.cli-lustre-OST0001-super.width=65536 llite.lustre-ffff9e803df57000.statahead_batch_max=64 llite.lustre-ffff9e803df57000.statahead_max=128