== sanity test 123ad: Verify batching statahead works correctly ========================================================== 12:34:37 (1785083677) batch: statahead_max=32 statahead_batch_max=32 mdc.lustre-MDT0000-mdc-ffff996e44515800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff996e44515800.batch_stats=0 llite.lustre-ffff996e44515800.statahead_max=32 llite.lustre-ffff996e44515800.statahead_batch_max=32 seq.cli-lustre-OST0000-super.width=0x1ffffff seq.cli-lustre-OST0001-super.width=0x1ffffff kvm mdc.lustre-MDT0000-mdc-ffff996e44515800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff996e44515800.batch_stats=0 striped dir -i1 -c2 -H crush2 /mnt/lustre/d123ad.sanity total: 100 open/close in 2.77 seconds: 36.09 ops/second llite.lustre-ffff996e44515800.statahead_max=0 101 real 0m3.193s user 0m0.011s sys 0m0.349s 'ls -l' 100 files without statahead: 3 sec llite.lustre-ffff996e44515800.statahead_max=32 llite.lustre-ffff996e44515800.statahead_max=32 101 real 0m0.805s user 0m0.003s sys 0m0.172s 'ls -l' 100 files with statahead: 1 sec statahead total: 125 statahead wrong: 2 agl total: 117 list_total: 31 fname_total: 94 hit_total: 4002 miss_total: 1329 swrong=2 ewrong=2 - open/close 824 (time 1785083706.10 total 10.00 last 82.38) total: 900 open/close in 11.15 seconds: 80.72 ops/second llite.lustre-ffff996e44515800.statahead_max=0 1001 real 0m24.319s user 0m0.031s sys 0m3.230s 'ls -l' 1000 files without statahead: 24 sec llite.lustre-ffff996e44515800.statahead_max=32 llite.lustre-ffff996e44515800.statahead_max=32 1001 real 0m9.444s user 0m0.035s sys 0m1.499s 'ls -l' 1000 files with statahead: 10 sec statahead total: 126 statahead wrong: 2 agl total: 118 list_total: 32 fname_total: 94 hit_total: 5000 miss_total: 1331 swrong=2 ewrong=2 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 17 seconds llite.lustre-ffff996e44515800.statahead_stats= statahead total: 126 statahead wrong: 2 agl total: 118 list_total: 32 fname_total: 94 hit_total: 5000 miss_total: 1331 mdc.lustre-MDT0000-mdc-ffff996e44515800.batch_stats= snapshot_time: 1785083772.037475214 secs.nsecs start_time: 1785083682.238165281 secs.nsecs elapsed_time: 89.799309933 secs.nsecs subreqs per batch batches %% cum %% 1: 2 5 5 2: 2 5 10 4: 2 5 16 8: 0 0 16 16: 27 72 89 32: 4 10 100 mdc.lustre-MDT0001-mdc-ffff996e44515800.batch_stats= snapshot_time: 1785083772.037543075 secs.nsecs start_time: 1785083682.238260776 secs.nsecs elapsed_time: 89.799282299 secs.nsecs subreqs per batch batches %% cum %% 1: 1 2 2 2: 0 0 2 4: 1 2 5 8: 4 10 15 16: 11 27 42 32: 23 57 100 - open/close 822 (time 1785083784.20 total 10.01 last 82.13) - open/close 1644 (time 1785083794.21 total 20.02 last 82.14) - open/close 2549 (time 1785083804.21 total 30.02 last 90.45) - open/close 3558 (time 1785083814.22 total 40.03 last 100.85) - open/close 4461 (time 1785083824.22 total 50.03 last 90.23) - open/close 5379 (time 1785083834.23 total 60.04 last 91.76) - open/close 6358 (time 1785083844.23 total 70.05 last 97.84) - open/close 7411 (time 1785083854.23 total 80.05 last 105.29) - open/close 8261 (time 1785083864.26 total 90.07 last 84.78) - open/close 9135 (time 1785083874.28 total 100.09 last 87.20) - open/close 9873 (time 1785083884.30 total 110.11 last 73.67) total: 10000 open/close in 112.01 seconds: 89.28 ops/second llite.lustre-ffff996e44515800.statahead_batch_max=0 llite.lustre-ffff996e44515800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff996e44515800.stats=clear mdc.lustre-MDT0001-mdc-ffff996e44515800.stats=clear ls -l files in d123ad.sanity: 10001 real 1m11.320s user 0m0.307s sys 0m16.219s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff996e44515800.statahead_batch_max=32 llite.lustre-ffff996e44515800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff996e44515800.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff996e44515800.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff996e44515800.stats=clear mdc.lustre-MDT0001-mdc-ffff996e44515800.stats=clear ls -l files in d123ad.sanity: 10001 real 1m42.476s user 0m0.312s sys 0m15.303s batched statahead RPC= hit count=9984 mdc.lustre-MDT0000-mdc-ffff996e44515800.batch_stats= snapshot_time: 1785084089.036889035 secs.nsecs start_time: 1785083969.644556661 secs.nsecs elapsed_time: 119.392332374 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 0 0 0 8: 1 0 0 16: 0 0 0 32: 313 99 100 mdc.lustre-MDT0001-mdc-ffff996e44515800.batch_stats= snapshot_time: 1785084089.037061603 secs.nsecs start_time: 1785083969.646158986 secs.nsecs elapsed_time: 119.390902617 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 batch: statahead_max=2048 statahead_batch_max=256 mdc.lustre-MDT0000-mdc-ffff996e44515800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff996e44515800.batch_stats=0 llite.lustre-ffff996e44515800.statahead_max=2048 llite.lustre-ffff996e44515800.statahead_batch_max=256 kvm mdc.lustre-MDT0000-mdc-ffff996e44515800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff996e44515800.batch_stats=0 striped dir -i1 -c2 -H crush2 /mnt/lustre/d123ad.sanity total: 100 open/close in 1.90 seconds: 52.69 ops/second llite.lustre-ffff996e44515800.statahead_max=0 101 real 0m2.193s user 0m0.009s sys 0m0.327s 'ls -l' 100 files without statahead: 2 sec llite.lustre-ffff996e44515800.statahead_max=2048 llite.lustre-ffff996e44515800.statahead_max=2048 101 real 0m1.056s user 0m0.007s sys 0m0.209s 'ls -l' 100 files with statahead: 1 sec statahead total: 2 statahead wrong: 0 agl total: 2 list_total: 2 fname_total: 0 hit_total: 10082 miss_total: 18 swrong=0 ewrong=0 total: 900 open/close in 9.91 seconds: 90.80 ops/second llite.lustre-ffff996e44515800.statahead_max=0 1001 real 0m23.754s user 0m0.029s sys 0m3.237s 'ls -l' 1000 files without statahead: 24 sec llite.lustre-ffff996e44515800.statahead_max=2048 llite.lustre-ffff996e44515800.statahead_max=2048 1001 real 0m7.917s user 0m0.029s sys 0m1.703s 'ls -l' 1000 files with statahead: 7 sec statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11081 miss_total: 19 swrong=0 ewrong=0 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 21 seconds llite.lustre-ffff996e44515800.statahead_stats= statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11081 miss_total: 19 mdc.lustre-MDT0000-mdc-ffff996e44515800.batch_stats= snapshot_time: 1785084447.333053809 secs.nsecs start_time: 1785084091.646711170 secs.nsecs elapsed_time: 355.686342639 secs.nsecs subreqs per batch batches %% cum %% 1: 1 16 16 2: 0 0 16 4: 3 50 66 8: 0 0 66 16: 0 0 66 32: 0 0 66 64: 0 0 66 128: 0 0 66 256: 2 33 100 mdc.lustre-MDT0001-mdc-ffff996e44515800.batch_stats= snapshot_time: 1785084447.335769173 secs.nsecs start_time: 1785084091.647754965 secs.nsecs elapsed_time: 355.688014208 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 0 0 0 8: 3 33 33 16: 1 11 44 32: 1 11 55 64: 2 22 77 128: 0 0 77 256: 2 22 100 - open/close 982 (time 1785084459.80 total 10.00 last 98.18) - open/close 1657 (time 1785084469.80 total 20.01 last 67.47) - open/close 2669 (time 1785084479.81 total 30.01 last 101.16) - open/close 3600 (time 1785084489.81 total 40.02 last 93.03) - open/close 4346 (time 1785084499.82 total 50.02 last 74.57) - open/close 5285 (time 1785084509.83 total 60.03 last 93.80) - open/close 6196 (time 1785084519.83 total 70.03 last 91.08) - open/close 7006 (time 1785084529.83 total 80.04 last 80.98) - open/close 7795 (time 1785084539.84 total 90.04 last 78.87) - open/close 8511 (time 1785084549.84 total 100.05 last 71.56) - open/close 9392 (time 1785084559.86 total 110.06 last 87.98) total: 10000 open/close in 117.65 seconds: 85.00 ops/second llite.lustre-ffff996e44515800.statahead_batch_max=0 llite.lustre-ffff996e44515800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff996e44515800.stats=clear mdc.lustre-MDT0001-mdc-ffff996e44515800.stats=clear ls -l files in d123ad.sanity: 10001 real 1m12.456s user 0m0.321s sys 0m16.485s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff996e44515800.statahead_batch_max=256 llite.lustre-ffff996e44515800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff996e44515800.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff996e44515800.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff996e44515800.stats=clear mdc.lustre-MDT0001-mdc-ffff996e44515800.stats=clear ls -l files in d123ad.sanity: 10001 real 1m5.445s user 0m0.309s sys 0m15.770s batched statahead RPC= hit count=9999 mdc.lustre-MDT0000-mdc-ffff996e44515800.batch_stats= snapshot_time: 1785084735.880756308 secs.nsecs start_time: 1785084652.391062576 secs.nsecs elapsed_time: 83.489693732 secs.nsecs subreqs per batch batches %% cum %% 1: 249 52 52 2: 78 16 69 4: 51 10 80 8: 23 4 84 16: 18 3 88 32: 12 2 91 64: 8 1 93 128: 0 0 93 256: 33 6 100 mdc.lustre-MDT0001-mdc-ffff996e44515800.batch_stats= snapshot_time: 1785084735.880823405 secs.nsecs start_time: 1785084652.391525883 secs.nsecs elapsed_time: 83.489297522 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second seq.cli-lustre-OST0000-super.width=65536 seq.cli-lustre-OST0001-super.width=65536 llite.lustre-ffff996e44515800.statahead_batch_max=64 llite.lustre-ffff996e44515800.statahead_max=128