== sanity test 123ad: Verify batching statahead works correctly ========================================================== 10:09:34 (1789308574) batch: statahead_max=32 statahead_batch_max=32 mdc.lustre-MDT0000-mdc-ffff953520c88800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff953520c88800.batch_stats=0 llite.lustre-ffff953520c88800.statahead_max=32 llite.lustre-ffff953520c88800.statahead_batch_max=32 seq.cli-lustre-OST0000-super.width=0x1ffffff seq.cli-lustre-OST0001-super.width=0x1ffffff kvm mdc.lustre-MDT0000-mdc-ffff953520c88800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff953520c88800.batch_stats=0 striped dir -i1 -c2 -H crush /mnt/lustre/d123ad.sanity total: 100 open/close in 1.00 seconds: 100.32 ops/second llite.lustre-ffff953520c88800.statahead_max=0 101 real 0m0.867s user 0m0.003s sys 0m0.269s 'ls -l' 100 files without statahead: 1 sec llite.lustre-ffff953520c88800.statahead_max=32 llite.lustre-ffff953520c88800.statahead_max=32 101 real 0m0.414s user 0m0.003s sys 0m0.226s 'ls -l' 100 files with statahead: 1 sec statahead total: 125 statahead wrong: 2 agl total: 120 list_total: 31 fname_total: 78 hit_total: 3867 miss_total: 1298 swrong=2 ewrong=2 total: 900 open/close in 3.40 seconds: 264.44 ops/second llite.lustre-ffff953520c88800.statahead_max=0 1001 real 0m8.614s user 0m0.034s sys 0m2.547s 'ls -l' 1000 files without statahead: 9 sec llite.lustre-ffff953520c88800.statahead_max=32 llite.lustre-ffff953520c88800.statahead_max=32 1001 real 0m3.711s user 0m0.019s sys 0m1.522s 'ls -l' 1000 files with statahead: 4 sec statahead total: 126 statahead wrong: 2 agl total: 121 list_total: 32 fname_total: 78 hit_total: 4865 miss_total: 1300 swrong=2 ewrong=2 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 9 seconds llite.lustre-ffff953520c88800.statahead_stats= statahead total: 126 statahead wrong: 2 agl total: 121 list_total: 32 fname_total: 78 hit_total: 4865 miss_total: 1300 mdc.lustre-MDT0000-mdc-ffff953520c88800.batch_stats= snapshot_time: 1789308608.935684426 secs.nsecs start_time: 1789308576.058503665 secs.nsecs elapsed_time: 32.877180761 secs.nsecs subreqs per batch batches %% cum %% 1: 2 5 5 2: 0 0 5 4: 2 5 10 8: 4 10 21 16: 25 67 89 32: 4 10 100 mdc.lustre-MDT0001-mdc-ffff953520c88800.batch_stats= snapshot_time: 1789308608.935735611 secs.nsecs start_time: 1789308576.058605843 secs.nsecs elapsed_time: 32.877129768 secs.nsecs subreqs per batch batches %% cum %% 1: 1 2 2 2: 1 2 5 4: 2 5 10 8: 1 2 12 16: 10 25 37 32: 25 62 100 - open/close 2228 (time 1789308620.25 total 10.00 last 222.78) - open/close 4741 (time 1789308630.26 total 20.00 last 251.21) - open/close 7384 (time 1789308640.26 total 30.01 last 264.26) - open/close 9664 (time 1789308650.26 total 40.01 last 227.95) total: 10000 open/close in 40.94 seconds: 244.26 ops/second llite.lustre-ffff953520c88800.statahead_batch_max=0 llite.lustre-ffff953520c88800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff953520c88800.stats=clear mdc.lustre-MDT0001-mdc-ffff953520c88800.stats=clear ls -l files in d123ad.sanity: 10001 real 0m30.189s user 0m0.237s sys 0m27.795s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff953520c88800.statahead_batch_max=32 llite.lustre-ffff953520c88800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff953520c88800.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff953520c88800.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff953520c88800.stats=clear mdc.lustre-MDT0001-mdc-ffff953520c88800.stats=clear ls -l files in d123ad.sanity: 10001 real 0m37.792s user 0m0.211s sys 0m18.602s batched statahead RPC= hit count=9997 mdc.lustre-MDT0000-mdc-ffff953520c88800.batch_stats= snapshot_time: 1789308726.829800594 secs.nsecs start_time: 1789308684.314345672 secs.nsecs elapsed_time: 42.515454922 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 0 0 0 8: 1 0 0 16: 2 0 0 32: 312 99 100 mdc.lustre-MDT0001-mdc-ffff953520c88800.batch_stats= snapshot_time: 1789308726.829850212 secs.nsecs start_time: 1789308684.314412925 secs.nsecs elapsed_time: 42.515437287 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 batch: statahead_max=2048 statahead_batch_max=256 mdc.lustre-MDT0000-mdc-ffff953520c88800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff953520c88800.batch_stats=0 llite.lustre-ffff953520c88800.statahead_max=2048 llite.lustre-ffff953520c88800.statahead_batch_max=256 kvm mdc.lustre-MDT0000-mdc-ffff953520c88800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff953520c88800.batch_stats=0 striped dir -i1 -c2 -H crush2 /mnt/lustre/d123ad.sanity total: 100 open/close in 1.02 seconds: 98.50 ops/second llite.lustre-ffff953520c88800.statahead_max=0 101 real 0m0.747s user 0m0.004s sys 0m0.255s 'ls -l' 100 files without statahead: 0 sec llite.lustre-ffff953520c88800.statahead_max=2048 llite.lustre-ffff953520c88800.statahead_max=2048 101 real 0m0.334s user 0m0.004s sys 0m0.238s 'ls -l' 100 files with statahead: 0 sec statahead total: 2 statahead wrong: 0 agl total: 2 list_total: 2 fname_total: 0 hit_total: 10096 miss_total: 4 swrong=0 ewrong=0 total: 900 open/close in 2.83 seconds: 318.53 ops/second llite.lustre-ffff953520c88800.statahead_max=0 1001 real 0m8.351s user 0m0.033s sys 0m2.615s 'ls -l' 1000 files without statahead: 9 sec llite.lustre-ffff953520c88800.statahead_max=2048 llite.lustre-ffff953520c88800.statahead_max=2048 1001 real 0m2.914s user 0m0.019s sys 0m1.533s 'ls -l' 1000 files with statahead: 3 sec statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11095 miss_total: 5 swrong=0 ewrong=0 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 7 seconds llite.lustre-ffff953520c88800.statahead_stats= statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11095 miss_total: 5 mdc.lustre-MDT0000-mdc-ffff953520c88800.batch_stats= snapshot_time: 1789308866.001005297 secs.nsecs start_time: 1789308727.873596528 secs.nsecs elapsed_time: 138.127408769 secs.nsecs subreqs per batch batches %% cum %% 1: 1 16 16 2: 0 0 16 4: 1 16 33 8: 2 33 66 16: 0 0 66 32: 0 0 66 64: 0 0 66 128: 0 0 66 256: 2 33 100 mdc.lustre-MDT0001-mdc-ffff953520c88800.batch_stats= snapshot_time: 1789308866.001073245 secs.nsecs start_time: 1789308727.873738168 secs.nsecs elapsed_time: 138.127335077 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 1 11 11 4: 1 11 22 8: 1 11 33 16: 1 11 44 32: 1 11 55 64: 2 22 77 128: 0 0 77 256: 2 22 100 - open/close 2575 (time 1789308877.02 total 10.00 last 257.45) - open/close 4967 (time 1789308887.02 total 20.00 last 239.19) - open/close 7358 (time 1789308897.02 total 30.00 last 239.10) total: 10000 open/close in 37.29 seconds: 268.20 ops/second llite.lustre-ffff953520c88800.statahead_batch_max=0 llite.lustre-ffff953520c88800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff953520c88800.stats=clear mdc.lustre-MDT0001-mdc-ffff953520c88800.stats=clear ls -l files in d123ad.sanity: 10001 real 0m33.265s user 0m0.236s sys 0m27.737s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff953520c88800.statahead_batch_max=256 llite.lustre-ffff953520c88800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff953520c88800.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff953520c88800.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff953520c88800.stats=clear mdc.lustre-MDT0001-mdc-ffff953520c88800.stats=clear ls -l files in d123ad.sanity: 10001 real 0m24.404s user 0m0.170s sys 0m18.180s batched statahead RPC= hit count=9999 mdc.lustre-MDT0000-mdc-ffff953520c88800.batch_stats= snapshot_time: 1789308969.590533576 secs.nsecs start_time: 1789308940.505011886 secs.nsecs elapsed_time: 29.085521690 secs.nsecs subreqs per batch batches %% cum %% 1: 151 55 55 2: 35 12 68 4: 17 6 74 8: 6 2 76 16: 2 0 77 32: 27 9 87 64: 0 0 87 128: 0 0 87 256: 35 12 100 mdc.lustre-MDT0001-mdc-ffff953520c88800.batch_stats= snapshot_time: 1789308969.590591154 secs.nsecs start_time: 1789308940.505080898 secs.nsecs elapsed_time: 29.085510256 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second seq.cli-lustre-OST0000-super.width=65536 seq.cli-lustre-OST0001-super.width=65536 llite.lustre-ffff953520c88800.statahead_batch_max=64 llite.lustre-ffff953520c88800.statahead_max=128