== sanity test 123ad: Verify batching statahead works correctly ========================================================== 10:31:42 (1789482702) batch: statahead_max=32 statahead_batch_max=32 mdc.lustre-MDT0000-mdc-ffff89c0916f2000.batch_stats=0 llite.lustre-ffff89c0916f2000.statahead_max=32 llite.lustre-ffff89c0916f2000.statahead_batch_max=32 seq.cli-lustre-OST0000-super.width=0x1ffffff seq.cli-lustre-OST0001-super.width=0x1ffffff kvm mdc.lustre-MDT0000-mdc-ffff89c0916f2000.batch_stats=0 total: 100 open/close in 4.02 seconds: 24.86 ops/second llite.lustre-ffff89c0916f2000.statahead_max=0 101 real 0m3.095s user 0m0.006s sys 0m0.334s 'ls -l' 100 files without statahead: 3 sec llite.lustre-ffff89c0916f2000.statahead_max=32 llite.lustre-ffff89c0916f2000.statahead_max=32 101 real 0m1.339s user 0m0.002s sys 0m0.163s 'ls -l' 100 files with statahead: 2 sec statahead total: 148 statahead wrong: 2 agl total: 142 list_total: 30 fname_total: 118 hit_total: 3549 miss_total: 1366 swrong=2 ewrong=2 - open/close 629 (time 1789482733.24 total 10.01 last 62.84) total: 900 open/close in 14.47 seconds: 62.20 ops/second llite.lustre-ffff89c0916f2000.statahead_max=0 1001 real 0m26.029s user 0m0.034s sys 0m2.902s 'ls -l' 1000 files without statahead: 26 sec llite.lustre-ffff89c0916f2000.statahead_max=32 llite.lustre-ffff89c0916f2000.statahead_max=32 1001 real 0m10.895s user 0m0.031s sys 0m1.370s 'ls -l' 1000 files with statahead: 11 sec statahead total: 149 statahead wrong: 2 agl total: 143 list_total: 31 fname_total: 118 hit_total: 4545 miss_total: 1370 swrong=2 ewrong=2 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 21 seconds llite.lustre-ffff89c0916f2000.statahead_stats= statahead total: 149 statahead wrong: 2 agl total: 143 list_total: 31 fname_total: 118 hit_total: 4545 miss_total: 1370 mdc.lustre-MDT0000-mdc-ffff89c0916f2000.batch_stats= snapshot_time: 1789482810.102207374 secs.nsecs start_time: 1789482707.593897743 secs.nsecs elapsed_time: 102.508309631 secs.nsecs subreqs per batch batches %% cum %% 1: 1 2 2 2: 0 0 2 4: 0 0 2 8: 3 7 10 16: 2 5 15 32: 34 85 100 - open/close 588 (time 1789482822.40 total 10.00 last 58.79) - open/close 1169 (time 1789482832.40 total 20.00 last 58.10) - open/close 1724 (time 1789482842.41 total 30.02 last 55.41) - open/close 2281 (time 1789482852.41 total 40.02 last 55.69) - open/close 2863 (time 1789482862.45 total 50.05 last 57.99) - open/close 3325 (time 1789482872.46 total 60.06 last 46.16) - open/close 3857 (time 1789482882.48 total 70.08 last 53.10) - open/close 4366 (time 1789482892.48 total 80.09 last 50.88) - open/close 4741 (time 1789482902.51 total 90.12 last 37.38) - open/close 5081 (time 1789482912.52 total 100.12 last 33.99) - open/close 5483 (time 1789482922.55 total 110.16 last 40.06) - open/close 5996 (time 1789482932.56 total 120.17 last 51.24) - open/close 6546 (time 1789482942.57 total 130.18 last 54.95) - open/close 7125 (time 1789482952.58 total 140.18 last 57.87) - open/close 7753 (time 1789482962.60 total 150.21 last 62.64) - open/close 8319 (time 1789482972.61 total 160.22 last 56.55) - open/close 8949 (time 1789482982.62 total 170.22 last 62.95) - open/close 9638 (time 1789482992.64 total 180.25 last 68.74) total: 10000 open/close in 185.64 seconds: 53.87 ops/second llite.lustre-ffff89c0916f2000.statahead_batch_max=0 llite.lustre-ffff89c0916f2000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff89c0916f2000.stats=clear ls -l files in d123ad.sanity: 10001 real 0m57.543s user 0m0.291s sys 0m14.696s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff89c0916f2000.statahead_batch_max=32 llite.lustre-ffff89c0916f2000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff89c0916f2000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff89c0916f2000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m33.936s user 0m0.268s sys 0m12.989s batched statahead RPC= hit count=9975 mdc.lustre-MDT0000-mdc-ffff89c0916f2000.batch_stats= snapshot_time: 1789483167.601583032 secs.nsecs start_time: 1789483063.221731589 secs.nsecs elapsed_time: 104.379851443 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 0 0 0 8: 2 0 0 16: 1 0 0 32: 313 99 100 batch: statahead_max=2048 statahead_batch_max=256 mdc.lustre-MDT0000-mdc-ffff89c0916f2000.batch_stats=0 llite.lustre-ffff89c0916f2000.statahead_max=2048 llite.lustre-ffff89c0916f2000.statahead_batch_max=256 kvm mdc.lustre-MDT0000-mdc-ffff89c0916f2000.batch_stats=0 total: 100 open/close in 2.50 seconds: 40.00 ops/second llite.lustre-ffff89c0916f2000.statahead_max=0 101 real 0m1.982s user 0m0.004s sys 0m0.297s 'ls -l' 100 files without statahead: 2 sec llite.lustre-ffff89c0916f2000.statahead_max=2048 llite.lustre-ffff89c0916f2000.statahead_max=2048 101 real 0m0.668s user 0m0.002s sys 0m0.143s 'ls -l' 100 files with statahead: 1 sec statahead total: 2 statahead wrong: 0 agl total: 2 list_total: 2 fname_total: 0 hit_total: 10074 miss_total: 26 swrong=0 ewrong=0 - open/close 697 (time 1789483501.07 total 10.02 last 69.58) total: 900 open/close in 13.16 seconds: 68.37 ops/second llite.lustre-ffff89c0916f2000.statahead_max=0 1001 real 0m22.688s user 0m0.030s sys 0m2.774s 'ls -l' 1000 files without statahead: 23 sec llite.lustre-ffff89c0916f2000.statahead_max=2048 llite.lustre-ffff89c0916f2000.statahead_max=2048 1001 real 0m6.332s user 0m0.022s sys 0m1.389s 'ls -l' 1000 files with statahead: 6 sec statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11073 miss_total: 27 swrong=0 ewrong=0 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 20 seconds llite.lustre-ffff89c0916f2000.statahead_stats= statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11073 miss_total: 27 mdc.lustre-MDT0000-mdc-ffff89c0916f2000.batch_stats= snapshot_time: 1789483564.628223844 secs.nsecs start_time: 1789483170.358068036 secs.nsecs elapsed_time: 394.270155808 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 0 0 0 8: 2 20 20 16: 2 20 40 32: 1 10 50 64: 0 0 50 128: 1 10 60 256: 4 40 100 - open/close 739 (time 1789483577.31 total 10.00 last 73.90) - open/close 1434 (time 1789483587.31 total 20.00 last 69.50) - open/close 2110 (time 1789483597.32 total 30.01 last 67.57) - open/close 2826 (time 1789483607.33 total 40.02 last 71.47) - open/close 3573 (time 1789483617.34 total 50.03 last 74.64) - open/close 4235 (time 1789483627.34 total 60.03 last 66.19) - open/close 4870 (time 1789483637.35 total 70.04 last 63.44) - open/close 5543 (time 1789483647.36 total 80.05 last 67.25) - open/close 6115 (time 1789483657.37 total 90.05 last 57.18) - open/close 6844 (time 1789483667.37 total 100.06 last 72.85) - open/close 7614 (time 1789483677.37 total 110.06 last 77.00) - open/close 8432 (time 1789483687.37 total 120.06 last 81.80) - open/close 9224 (time 1789483697.39 total 130.08 last 79.06) - open/close 9977 (time 1789483707.39 total 140.08 last 75.29) total: 10000 open/close in 140.34 seconds: 71.25 ops/second llite.lustre-ffff89c0916f2000.statahead_batch_max=0 llite.lustre-ffff89c0916f2000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff89c0916f2000.stats=clear ls -l files in d123ad.sanity: 10001 real 0m54.540s user 0m0.284s sys 0m14.421s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff89c0916f2000.statahead_batch_max=256 llite.lustre-ffff89c0916f2000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff89c0916f2000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff89c0916f2000.stats=clear ls -l files in d123ad.sanity: 10001 real 0m46.209s user 0m0.296s sys 0m14.679s batched statahead RPC= hit count=9999 mdc.lustre-MDT0000-mdc-ffff89c0916f2000.batch_stats= snapshot_time: 1789483826.195287654 secs.nsecs start_time: 1789483770.693250960 secs.nsecs elapsed_time: 55.502036694 secs.nsecs subreqs per batch batches %% cum %% 1: 158 50 50 2: 39 12 62 4: 26 8 70 8: 23 7 78 16: 12 3 81 32: 14 4 86 64: 9 2 89 128: 0 0 89 256: 34 10 100 unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second seq.cli-lustre-OST0000-super.width=65536 seq.cli-lustre-OST0001-super.width=65536 llite.lustre-ffff89c0916f2000.statahead_batch_max=64 llite.lustre-ffff89c0916f2000.statahead_max=128