== sanity test 123ad: Verify batching statahead works correctly ========================================================== 20:53:52 (1786841632) batch: statahead_max=32 statahead_batch_max=32 mdc.lustre-MDT0000-mdc-ffff9ba3885bf800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9ba3885bf800.batch_stats=0 llite.lustre-ffff9ba3885bf800.statahead_max=32 llite.lustre-ffff9ba3885bf800.statahead_batch_max=32 seq.cli-lustre-OST0000-super.width=0x1ffffff seq.cli-lustre-OST0001-super.width=0x1ffffff kvm mdc.lustre-MDT0000-mdc-ffff9ba3885bf800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9ba3885bf800.batch_stats=0 striped dir -i1 -c2 -H crush /mnt/lustre/d123ad.sanity total: 100 open/close in 0.68 seconds: 147.76 ops/second llite.lustre-ffff9ba3885bf800.statahead_max=0 101 real 0m0.577s user 0m0.003s sys 0m0.203s 'ls -l' 100 files without statahead: 1 sec llite.lustre-ffff9ba3885bf800.statahead_max=32 llite.lustre-ffff9ba3885bf800.statahead_max=32 101 real 0m0.401s user 0m0.004s sys 0m0.213s 'ls -l' 100 files with statahead: 0 sec statahead total: 54 statahead wrong: 1 agl total: 50 list_total: 25 fname_total: 29 hit_total: 13608 miss_total: 198 swrong=1 ewrong=1 total: 900 open/close in 1.92 seconds: 468.28 ops/second llite.lustre-ffff9ba3885bf800.statahead_max=0 1001 real 0m5.563s user 0m0.014s sys 0m1.931s 'ls -l' 1000 files without statahead: 5 sec llite.lustre-ffff9ba3885bf800.statahead_max=32 llite.lustre-ffff9ba3885bf800.statahead_max=32 1001 real 0m4.267s user 0m0.013s sys 0m1.902s 'ls -l' 1000 files with statahead: 4 sec statahead total: 55 statahead wrong: 1 agl total: 51 list_total: 26 fname_total: 29 hit_total: 14607 miss_total: 199 swrong=1 ewrong=1 - open/close 4745 (time 1786841662.49 total 10.00 last 474.46) total: 9000 open/close in 19.31 seconds: 466.13 ops/second llite.lustre-ffff9ba3885bf800.statahead_max=0 10001 real 1m0.097s user 0m0.154s sys 0m21.537s 'ls -l' 10000 files without statahead: 61 sec llite.lustre-ffff9ba3885bf800.statahead_max=32 llite.lustre-ffff9ba3885bf800.statahead_max=32 10001 real 0m41.382s user 0m0.159s sys 0m19.395s 'ls -l' 10000 files with statahead: 41 sec statahead total: 56 statahead wrong: 1 agl total: 52 list_total: 27 fname_total: 29 hit_total: 24606 miss_total: 200 swrong=1 ewrong=1 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 78 seconds llite.lustre-ffff9ba3885bf800.statahead_stats= statahead total: 56 statahead wrong: 1 agl total: 52 list_total: 27 fname_total: 29 hit_total: 24606 miss_total: 200 mdc.lustre-MDT0000-mdc-ffff9ba3885bf800.batch_stats= snapshot_time: 1786841864.124189154 secs.nsecs start_time: 1786841634.162116712 secs.nsecs elapsed_time: 229.962072442 secs.nsecs subreqs per batch batches %% cum %% 1: 3 0 0 2: 0 0 0 4: 5 1 2 8: 6 1 3 16: 235 66 70 32: 103 29 100 mdc.lustre-MDT0001-mdc-ffff9ba3885bf800.batch_stats= snapshot_time: 1786841864.124296711 secs.nsecs start_time: 1786841634.162167095 secs.nsecs elapsed_time: 229.962129616 secs.nsecs subreqs per batch batches %% cum %% 1: 1 0 0 2: 1 0 0 4: 1 0 0 8: 4 1 1 16: 165 46 48 32: 182 51 100 - open/close 2385 (time 1786841875.39 total 10.00 last 238.50) - open/close 4715 (time 1786841885.40 total 20.00 last 232.91) - open/close 7300 (time 1786841895.40 total 30.01 last 258.44) total: 10000 open/close in 39.42 seconds: 253.67 ops/second llite.lustre-ffff9ba3885bf800.statahead_batch_max=0 llite.lustre-ffff9ba3885bf800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9ba3885bf800.stats=clear mdc.lustre-MDT0001-mdc-ffff9ba3885bf800.stats=clear ls -l files in d123ad.sanity: 10001 real 0m52.298s user 0m0.161s sys 0m48.833s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff9ba3885bf800.statahead_batch_max=32 llite.lustre-ffff9ba3885bf800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9ba3885bf800.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff9ba3885bf800.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff9ba3885bf800.stats=clear mdc.lustre-MDT0001-mdc-ffff9ba3885bf800.stats=clear ls -l files in d123ad.sanity: 10001 real 0m56.104s user 0m0.155s sys 0m32.922s batched statahead RPC= hit count=9999 mdc.lustre-MDT0000-mdc-ffff9ba3885bf800.batch_stats= snapshot_time: 1786842023.730010031 secs.nsecs start_time: 1786841960.473236582 secs.nsecs elapsed_time: 63.256773449 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 0 0 0 8: 1 0 0 16: 2 0 0 32: 312 99 100 mdc.lustre-MDT0001-mdc-ffff9ba3885bf800.batch_stats= snapshot_time: 1786842023.730057296 secs.nsecs start_time: 1786841960.473330524 secs.nsecs elapsed_time: 63.256726772 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 batch: statahead_max=2048 statahead_batch_max=256 mdc.lustre-MDT0000-mdc-ffff9ba3885bf800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9ba3885bf800.batch_stats=0 llite.lustre-ffff9ba3885bf800.statahead_max=2048 llite.lustre-ffff9ba3885bf800.statahead_batch_max=256 kvm mdc.lustre-MDT0000-mdc-ffff9ba3885bf800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9ba3885bf800.batch_stats=0 striped dir -i1 -c2 -H crush2 /mnt/lustre/d123ad.sanity total: 100 open/close in 1.08 seconds: 92.34 ops/second llite.lustre-ffff9ba3885bf800.statahead_max=0 101 real 0m1.020s user 0m0.003s sys 0m0.382s 'ls -l' 100 files without statahead: 1 sec llite.lustre-ffff9ba3885bf800.statahead_max=2048 llite.lustre-ffff9ba3885bf800.statahead_max=2048 101 real 0m0.472s user 0m0.003s sys 0m0.292s 'ls -l' 100 files with statahead: 1 sec statahead total: 2 statahead wrong: 0 agl total: 2 list_total: 2 fname_total: 0 hit_total: 10098 miss_total: 2 swrong=0 ewrong=0 total: 900 open/close in 3.84 seconds: 234.37 ops/second llite.lustre-ffff9ba3885bf800.statahead_max=0 1001 real 0m7.438s user 0m0.020s sys 0m2.628s 'ls -l' 1000 files without statahead: 8 sec llite.lustre-ffff9ba3885bf800.statahead_max=2048 llite.lustre-ffff9ba3885bf800.statahead_max=2048 1001 real 0m3.325s user 0m0.009s sys 0m1.728s 'ls -l' 1000 files with statahead: 3 sec statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11097 miss_total: 3 swrong=0 ewrong=0 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 7 seconds llite.lustre-ffff9ba3885bf800.statahead_stats= statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11097 miss_total: 3 mdc.lustre-MDT0000-mdc-ffff9ba3885bf800.batch_stats= snapshot_time: 1786842174.553653634 secs.nsecs start_time: 1786842024.815085844 secs.nsecs elapsed_time: 149.738567790 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 2 40 40 8: 1 20 60 16: 0 0 60 32: 0 0 60 64: 0 0 60 128: 0 0 60 256: 2 40 100 mdc.lustre-MDT0001-mdc-ffff9ba3885bf800.batch_stats= snapshot_time: 1786842174.553731819 secs.nsecs start_time: 1786842024.815242052 secs.nsecs elapsed_time: 149.738489767 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 0 0 0 8: 3 37 37 16: 1 12 50 32: 0 0 50 64: 1 12 62 128: 1 12 75 256: 2 25 100 - open/close 3403 (time 1786842185.38 total 10.00 last 340.15) - open/close 5648 (time 1786842195.38 total 20.01 last 224.48) - open/close 8197 (time 1786842205.38 total 30.01 last 254.89) total: 10000 open/close in 35.03 seconds: 285.44 ops/second llite.lustre-ffff9ba3885bf800.statahead_batch_max=0 llite.lustre-ffff9ba3885bf800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9ba3885bf800.stats=clear mdc.lustre-MDT0001-mdc-ffff9ba3885bf800.stats=clear ls -l files in d123ad.sanity: 10001 real 0m41.540s user 0m0.147s sys 0m35.946s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff9ba3885bf800.statahead_batch_max=256 llite.lustre-ffff9ba3885bf800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9ba3885bf800.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff9ba3885bf800.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff9ba3885bf800.stats=clear mdc.lustre-MDT0001-mdc-ffff9ba3885bf800.stats=clear ls -l files in d123ad.sanity: 10001 real 0m35.861s user 0m0.138s sys 0m24.779s batched statahead RPC= hit count=9999 mdc.lustre-MDT0000-mdc-ffff9ba3885bf800.batch_stats= snapshot_time: 1786842297.532100760 secs.nsecs start_time: 1786842254.921336241 secs.nsecs elapsed_time: 42.610764519 secs.nsecs subreqs per batch batches %% cum %% 1: 511 71 71 2: 107 14 86 4: 33 4 90 8: 4 0 91 16: 2 0 91 32: 23 3 94 64: 3 0 95 128: 0 0 95 256: 33 4 100 mdc.lustre-MDT0001-mdc-ffff9ba3885bf800.batch_stats= snapshot_time: 1786842297.532150224 secs.nsecs start_time: 1786842254.921441554 secs.nsecs elapsed_time: 42.610708670 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second seq.cli-lustre-OST0000-super.width=65536 seq.cli-lustre-OST0001-super.width=65536 llite.lustre-ffff9ba3885bf800.statahead_batch_max=64 llite.lustre-ffff9ba3885bf800.statahead_max=128