== sanity test 123ad: Verify batching statahead works correctly ========================================================== 22:23:53 (1787797433) batch: statahead_max=32 statahead_batch_max=32 mdc.lustre-MDT0000-mdc-ffff99caa7bf0800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff99caa7bf0800.batch_stats=0 llite.lustre-ffff99caa7bf0800.statahead_max=32 llite.lustre-ffff99caa7bf0800.statahead_batch_max=32 seq.cli-lustre-OST0000-super.width=0x1ffffff seq.cli-lustre-OST0001-super.width=0x1ffffff kvm mdc.lustre-MDT0000-mdc-ffff99caa7bf0800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff99caa7bf0800.batch_stats=0 striped dir -i1 -c2 -H all_char /mnt/lustre/d123ad.sanity total: 100 open/close in 3.04 seconds: 32.93 ops/second llite.lustre-ffff99caa7bf0800.statahead_max=0 101 real 0m2.310s user 0m0.006s sys 0m0.316s 'ls -l' 100 files without statahead: 2 sec llite.lustre-ffff99caa7bf0800.statahead_max=32 llite.lustre-ffff99caa7bf0800.statahead_max=32 101 real 0m1.299s user 0m0.003s sys 0m0.164s 'ls -l' 100 files with statahead: 1 sec statahead total: 134 statahead wrong: 2 agl total: 129 list_total: 32 fname_total: 95 hit_total: 4361 miss_total: 1279 swrong=2 ewrong=2 - open/close 818 (time 1787797462.03 total 10.01 last 81.75) total: 900 open/close in 11.12 seconds: 80.92 ops/second llite.lustre-ffff99caa7bf0800.statahead_max=0 1001 real 0m22.554s user 0m0.038s sys 0m2.941s 'ls -l' 1000 files without statahead: 22 sec llite.lustre-ffff99caa7bf0800.statahead_max=32 llite.lustre-ffff99caa7bf0800.statahead_max=32 1001 real 0m9.814s user 0m0.026s sys 0m1.441s 'ls -l' 1000 files with statahead: 10 sec statahead total: 135 statahead wrong: 2 agl total: 130 list_total: 33 fname_total: 95 hit_total: 5356 miss_total: 1284 swrong=2 ewrong=2 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 22 seconds llite.lustre-ffff99caa7bf0800.statahead_stats= statahead total: 135 statahead wrong: 2 agl total: 130 list_total: 33 fname_total: 95 hit_total: 5356 miss_total: 1284 mdc.lustre-MDT0000-mdc-ffff99caa7bf0800.batch_stats= snapshot_time: 1787797532.382879293 secs.nsecs start_time: 1787797438.814390968 secs.nsecs elapsed_time: 93.568488325 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 1 2 2 4: 2 5 7 8: 3 7 15 16: 21 53 69 32: 12 30 100 mdc.lustre-MDT0001-mdc-ffff99caa7bf0800.batch_stats= snapshot_time: 1787797532.382945078 secs.nsecs start_time: 1787797438.817010791 secs.nsecs elapsed_time: 93.565934287 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 1 2 2 4: 2 5 7 8: 3 7 15 16: 17 43 58 32: 16 41 100 - open/close 582 (time 1787797545.49 total 10.02 last 58.10) - open/close 1238 (time 1787797555.50 total 20.03 last 65.54) - open/close 2066 (time 1787797565.50 total 30.03 last 82.79) - open/close 2870 (time 1787797575.50 total 40.03 last 80.37) - open/close 3702 (time 1787797585.50 total 50.03 last 83.20) - open/close 4530 (time 1787797595.50 total 60.03 last 82.79) - open/close 5365 (time 1787797605.50 total 70.03 last 83.49) - open/close 6175 (time 1787797615.51 total 80.04 last 80.97) - open/close 7001 (time 1787797625.51 total 90.04 last 82.56) - open/close 7751 (time 1787797635.55 total 100.08 last 74.69) - open/close 8507 (time 1787797645.56 total 110.09 last 75.54) - open/close 9328 (time 1787797655.57 total 120.09 last 82.07) total: 10000 open/close in 128.44 seconds: 77.86 ops/second llite.lustre-ffff99caa7bf0800.statahead_batch_max=0 llite.lustre-ffff99caa7bf0800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff99caa7bf0800.stats=clear mdc.lustre-MDT0001-mdc-ffff99caa7bf0800.stats=clear ls -l files in d123ad.sanity: 10001 real 1m0.515s user 0m0.316s sys 0m14.821s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff99caa7bf0800.statahead_batch_max=32 llite.lustre-ffff99caa7bf0800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff99caa7bf0800.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff99caa7bf0800.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff99caa7bf0800.stats=clear mdc.lustre-MDT0001-mdc-ffff99caa7bf0800.stats=clear ls -l files in d123ad.sanity: 10001 real 1m36.655s user 0m0.312s sys 0m14.369s batched statahead RPC= hit count=9979 mdc.lustre-MDT0000-mdc-ffff99caa7bf0800.batch_stats= snapshot_time: 1787797841.405707969 secs.nsecs start_time: 1787797733.896259263 secs.nsecs elapsed_time: 107.509448706 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 0 0 0 8: 1 0 0 16: 1 0 0 32: 313 99 100 mdc.lustre-MDT0001-mdc-ffff99caa7bf0800.batch_stats= snapshot_time: 1787797841.405786632 secs.nsecs start_time: 1787797733.896518858 secs.nsecs elapsed_time: 107.509267774 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 batch: statahead_max=2048 statahead_batch_max=256 mdc.lustre-MDT0000-mdc-ffff99caa7bf0800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff99caa7bf0800.batch_stats=0 llite.lustre-ffff99caa7bf0800.statahead_max=2048 llite.lustre-ffff99caa7bf0800.statahead_batch_max=256 kvm mdc.lustre-MDT0000-mdc-ffff99caa7bf0800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff99caa7bf0800.batch_stats=0 striped dir -i1 -c2 -H crush2 /mnt/lustre/d123ad.sanity total: 100 open/close in 2.72 seconds: 36.75 ops/second llite.lustre-ffff99caa7bf0800.statahead_max=0 101 real 0m1.934s user 0m0.006s sys 0m0.318s 'ls -l' 100 files without statahead: 2 sec llite.lustre-ffff99caa7bf0800.statahead_max=2048 llite.lustre-ffff99caa7bf0800.statahead_max=2048 101 real 0m0.809s user 0m0.006s sys 0m0.190s 'ls -l' 100 files with statahead: 1 sec statahead total: 2 statahead wrong: 0 agl total: 2 list_total: 2 fname_total: 0 hit_total: 10078 miss_total: 22 swrong=0 ewrong=0 - open/close 524 (time 1787798136.21 total 10.00 last 52.40) total: 900 open/close in 14.58 seconds: 61.74 ops/second llite.lustre-ffff99caa7bf0800.statahead_max=0 1001 real 0m22.025s user 0m0.036s sys 0m3.003s 'ls -l' 1000 files without statahead: 22 sec llite.lustre-ffff99caa7bf0800.statahead_max=2048 llite.lustre-ffff99caa7bf0800.statahead_max=2048 1001 real 0m6.529s user 0m0.024s sys 0m1.512s 'ls -l' 1000 files with statahead: 7 sec statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11077 miss_total: 23 swrong=0 ewrong=0 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 19 seconds llite.lustre-ffff99caa7bf0800.statahead_stats= statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11077 miss_total: 23 mdc.lustre-MDT0000-mdc-ffff99caa7bf0800.batch_stats= snapshot_time: 1787798199.631518248 secs.nsecs start_time: 1787797844.064161567 secs.nsecs elapsed_time: 355.567356681 secs.nsecs subreqs per batch batches %% cum %% 1: 2 25 25 2: 0 0 25 4: 3 37 62 8: 1 12 75 16: 0 0 75 32: 0 0 75 64: 0 0 75 128: 0 0 75 256: 2 25 100 mdc.lustre-MDT0001-mdc-ffff99caa7bf0800.batch_stats= snapshot_time: 1787798199.631599895 secs.nsecs start_time: 1787797844.067518173 secs.nsecs elapsed_time: 355.564081722 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 0 0 0 8: 3 30 30 16: 1 10 40 32: 4 40 80 64: 0 0 80 128: 0 0 80 256: 2 20 100 - open/close 866 (time 1787798212.15 total 10.01 last 86.48) - open/close 1634 (time 1787798222.16 total 20.02 last 76.77) - open/close 2513 (time 1787798232.16 total 30.02 last 87.85) - open/close 3402 (time 1787798242.17 total 40.03 last 88.86) - open/close 4228 (time 1787798252.17 total 50.03 last 82.59) - open/close 5124 (time 1787798262.18 total 60.04 last 89.47) - open/close 5954 (time 1787798272.19 total 70.05 last 82.95) - open/close 6807 (time 1787798282.20 total 80.06 last 85.23) - open/close 7624 (time 1787798292.21 total 90.07 last 81.61) - open/close 8414 (time 1787798302.22 total 100.08 last 78.95) - open/close 9202 (time 1787798312.24 total 110.11 last 78.57) total: 10000 open/close in 119.63 seconds: 83.59 ops/second llite.lustre-ffff99caa7bf0800.statahead_batch_max=0 llite.lustre-ffff99caa7bf0800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff99caa7bf0800.stats=clear mdc.lustre-MDT0001-mdc-ffff99caa7bf0800.stats=clear ls -l files in d123ad.sanity: 10001 real 0m57.859s user 0m0.284s sys 0m15.085s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff99caa7bf0800.statahead_batch_max=256 llite.lustre-ffff99caa7bf0800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff99caa7bf0800.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff99caa7bf0800.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff99caa7bf0800.stats=clear mdc.lustre-MDT0001-mdc-ffff99caa7bf0800.stats=clear ls -l files in d123ad.sanity: 10001 real 0m48.864s user 0m0.279s sys 0m15.723s batched statahead RPC= hit count=9999 mdc.lustre-MDT0000-mdc-ffff99caa7bf0800.batch_stats= snapshot_time: 1787798446.622302601 secs.nsecs start_time: 1787798387.456916199 secs.nsecs elapsed_time: 59.165386402 secs.nsecs subreqs per batch batches %% cum %% 1: 131 50 50 2: 37 14 65 4: 17 6 71 8: 9 3 75 16: 7 2 77 32: 12 4 82 64: 9 3 86 128: 2 0 86 256: 34 13 100 mdc.lustre-MDT0001-mdc-ffff99caa7bf0800.batch_stats= snapshot_time: 1787798446.622374002 secs.nsecs start_time: 1787798387.457037729 secs.nsecs elapsed_time: 59.165336273 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second seq.cli-lustre-OST0000-super.width=65536 seq.cli-lustre-OST0001-super.width=65536 llite.lustre-ffff99caa7bf0800.statahead_batch_max=64 llite.lustre-ffff99caa7bf0800.statahead_max=128