== sanity test 123ad: Verify batching statahead works correctly ========================================================== 05:51:27 (1788256287) batch: statahead_max=32 statahead_batch_max=32 mdc.lustre-MDT0000-mdc-ffff9eeaefac3000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9eeaefac3000.batch_stats=0 llite.lustre-ffff9eeaefac3000.statahead_max=32 llite.lustre-ffff9eeaefac3000.statahead_batch_max=32 seq.cli-lustre-OST0000-super.width=0x1ffffff seq.cli-lustre-OST0001-super.width=0x1ffffff kvm mdc.lustre-MDT0000-mdc-ffff9eeaefac3000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9eeaefac3000.batch_stats=0 striped dir -i1 -c2 -H crush2 /mnt/lustre/d123ad.sanity total: 100 open/close in 2.31 seconds: 43.29 ops/second llite.lustre-ffff9eeaefac3000.statahead_max=0 101 real 0m2.500s user 0m0.003s sys 0m0.306s 'ls -l' 100 files without statahead: 2 sec llite.lustre-ffff9eeaefac3000.statahead_max=32 llite.lustre-ffff9eeaefac3000.statahead_max=32 101 real 0m1.047s user 0m0.004s sys 0m0.141s 'ls -l' 100 files with statahead: 1 sec statahead total: 123 statahead wrong: 2 agl total: 118 list_total: 31 fname_total: 77 hit_total: 3591 miss_total: 1487 swrong=2 ewrong=2 - open/close 792 (time 1788256314.70 total 10.01 last 79.15) total: 900 open/close in 11.58 seconds: 77.75 ops/second llite.lustre-ffff9eeaefac3000.statahead_max=0 1001 real 0m20.532s user 0m0.028s sys 0m2.865s 'ls -l' 1000 files without statahead: 20 sec llite.lustre-ffff9eeaefac3000.statahead_max=32 llite.lustre-ffff9eeaefac3000.statahead_max=32 1001 real 0m8.437s user 0m0.033s sys 0m1.504s 'ls -l' 1000 files with statahead: 8 sec statahead total: 124 statahead wrong: 2 agl total: 119 list_total: 32 fname_total: 77 hit_total: 4590 miss_total: 1488 swrong=2 ewrong=2 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 20 seconds llite.lustre-ffff9eeaefac3000.statahead_stats= statahead total: 124 statahead wrong: 2 agl total: 119 list_total: 32 fname_total: 77 hit_total: 4590 miss_total: 1488 mdc.lustre-MDT0000-mdc-ffff9eeaefac3000.batch_stats= snapshot_time: 1788256376.149188775 secs.nsecs start_time: 1788256291.149611232 secs.nsecs elapsed_time: 84.999577543 secs.nsecs subreqs per batch batches %% cum %% 1: 3 7 7 2: 2 5 13 4: 2 5 18 8: 2 5 23 16: 22 57 81 32: 7 18 100 mdc.lustre-MDT0001-mdc-ffff9eeaefac3000.batch_stats= snapshot_time: 1788256376.149294033 secs.nsecs start_time: 1788256291.149688812 secs.nsecs elapsed_time: 84.999605221 secs.nsecs subreqs per batch batches %% cum %% 1: 1 2 2 2: 1 2 5 4: 0 0 5 8: 4 10 15 16: 12 30 45 32: 22 55 100 - open/close 726 (time 1788256388.53 total 10.01 last 72.55) - open/close 1565 (time 1788256398.53 total 20.01 last 83.89) - open/close 2339 (time 1788256408.53 total 30.01 last 77.36) - open/close 3159 (time 1788256418.53 total 40.01 last 82.00) - open/close 3954 (time 1788256428.53 total 50.02 last 79.48) - open/close 4651 (time 1788256438.54 total 60.02 last 69.64) - open/close 5286 (time 1788256448.54 total 70.02 last 63.50) - open/close 5972 (time 1788256458.55 total 80.03 last 68.55) - open/close 6772 (time 1788256468.56 total 90.04 last 79.96) - open/close 7671 (time 1788256478.56 total 100.04 last 89.87) - open/close 8597 (time 1788256488.57 total 110.05 last 92.49) - open/close 9519 (time 1788256498.57 total 120.06 last 92.17) total: 10000 open/close in 125.13 seconds: 79.92 ops/second llite.lustre-ffff9eeaefac3000.statahead_batch_max=0 llite.lustre-ffff9eeaefac3000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9eeaefac3000.stats=clear mdc.lustre-MDT0001-mdc-ffff9eeaefac3000.stats=clear ls -l files in d123ad.sanity: 10001 real 0m55.072s user 0m0.307s sys 0m15.882s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff9eeaefac3000.statahead_batch_max=32 llite.lustre-ffff9eeaefac3000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9eeaefac3000.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff9eeaefac3000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff9eeaefac3000.stats=clear mdc.lustre-MDT0001-mdc-ffff9eeaefac3000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m29.608s user 0m0.274s sys 0m13.801s batched statahead RPC= hit count=9981 mdc.lustre-MDT0000-mdc-ffff9eeaefac3000.batch_stats= snapshot_time: 1788256666.519339086 secs.nsecs start_time: 1788256566.999480830 secs.nsecs elapsed_time: 99.519858256 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 0 0 0 8: 1 0 0 16: 1 0 0 32: 313 99 100 mdc.lustre-MDT0001-mdc-ffff9eeaefac3000.batch_stats= snapshot_time: 1788256666.519403258 secs.nsecs start_time: 1788256566.999559334 secs.nsecs elapsed_time: 99.519843924 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 batch: statahead_max=2048 statahead_batch_max=256 mdc.lustre-MDT0000-mdc-ffff9eeaefac3000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9eeaefac3000.batch_stats=0 llite.lustre-ffff9eeaefac3000.statahead_max=2048 llite.lustre-ffff9eeaefac3000.statahead_batch_max=256 kvm mdc.lustre-MDT0000-mdc-ffff9eeaefac3000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9eeaefac3000.batch_stats=0 striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d123ad.sanity total: 100 open/close in 2.22 seconds: 45.11 ops/second llite.lustre-ffff9eeaefac3000.statahead_max=0 101 real 0m2.018s user 0m0.002s sys 0m0.308s 'ls -l' 100 files without statahead: 2 sec llite.lustre-ffff9eeaefac3000.statahead_max=2048 llite.lustre-ffff9eeaefac3000.statahead_max=2048 101 real 0m0.885s user 0m0.005s sys 0m0.172s 'ls -l' 100 files with statahead: 1 sec statahead total: 2 statahead wrong: 0 agl total: 2 list_total: 2 fname_total: 0 hit_total: 10080 miss_total: 20 swrong=0 ewrong=0 - open/close 850 (time 1788256943.66 total 10.00 last 84.98) total: 900 open/close in 10.57 seconds: 85.18 ops/second llite.lustre-ffff9eeaefac3000.statahead_max=0 1001 real 0m18.419s user 0m0.025s sys 0m2.858s 'ls -l' 1000 files without statahead: 18 sec llite.lustre-ffff9eeaefac3000.statahead_max=2048 llite.lustre-ffff9eeaefac3000.statahead_max=2048 1001 real 0m5.411s user 0m0.018s sys 0m1.535s 'ls -l' 1000 files with statahead: 6 sec statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11079 miss_total: 21 swrong=0 ewrong=0 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 16 seconds llite.lustre-ffff9eeaefac3000.statahead_stats= statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11079 miss_total: 21 mdc.lustre-MDT0000-mdc-ffff9eeaefac3000.batch_stats= snapshot_time: 1788256993.933257271 secs.nsecs start_time: 1788256668.490030681 secs.nsecs elapsed_time: 325.443226590 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 1 11 11 4: 1 11 22 8: 3 33 55 16: 1 11 66 32: 1 11 77 64: 0 0 77 128: 0 0 77 256: 2 22 100 mdc.lustre-MDT0001-mdc-ffff9eeaefac3000.batch_stats= snapshot_time: 1788256993.933325861 secs.nsecs start_time: 1788256668.490173931 secs.nsecs elapsed_time: 325.443151930 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 2 22 22 8: 3 33 55 16: 1 11 66 32: 0 0 66 64: 1 11 77 128: 0 0 77 256: 2 22 100 - open/close 878 (time 1788257006.17 total 10.00 last 87.76) - open/close 1774 (time 1788257016.17 total 20.01 last 89.57) - open/close 2703 (time 1788257026.17 total 30.01 last 92.87) - open/close 3617 (time 1788257036.18 total 40.01 last 91.37) - open/close 4567 (time 1788257046.20 total 50.04 last 94.74) - open/close 5491 (time 1788257056.21 total 60.05 last 92.35) - open/close 6440 (time 1788257066.22 total 70.05 last 94.84) - open/close 7415 (time 1788257076.22 total 80.06 last 97.48) - open/close 8344 (time 1788257086.22 total 90.06 last 92.85) - open/close 9257 (time 1788257096.23 total 100.07 last 91.22) total: 10000 open/close in 108.39 seconds: 92.26 ops/second llite.lustre-ffff9eeaefac3000.statahead_batch_max=0 llite.lustre-ffff9eeaefac3000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9eeaefac3000.stats=clear mdc.lustre-MDT0001-mdc-ffff9eeaefac3000.stats=clear ls -l files in d123ad.sanity: 10001 real 0m58.768s user 0m0.293s sys 0m15.321s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff9eeaefac3000.statahead_batch_max=256 llite.lustre-ffff9eeaefac3000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9eeaefac3000.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff9eeaefac3000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff9eeaefac3000.stats=clear mdc.lustre-MDT0001-mdc-ffff9eeaefac3000.stats=clear ls -l files in d123ad.sanity: 10001 real 0m43.285s user 0m0.288s sys 0m15.840s batched statahead RPC= hit count=9999 mdc.lustre-MDT0000-mdc-ffff9eeaefac3000.batch_stats= snapshot_time: 1788257223.370144084 secs.nsecs start_time: 1788257172.354367534 secs.nsecs elapsed_time: 51.015776550 secs.nsecs subreqs per batch batches %% cum %% 1: 155 50 50 2: 38 12 63 4: 31 10 73 8: 17 5 78 16: 9 2 81 32: 12 3 85 64: 9 2 88 128: 1 0 88 256: 34 11 100 mdc.lustre-MDT0001-mdc-ffff9eeaefac3000.batch_stats= snapshot_time: 1788257223.370203656 secs.nsecs start_time: 1788257172.354440659 secs.nsecs elapsed_time: 51.015762997 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second seq.cli-lustre-OST0000-super.width=65536 seq.cli-lustre-OST0001-super.width=65536 llite.lustre-ffff9eeaefac3000.statahead_batch_max=64 llite.lustre-ffff9eeaefac3000.statahead_max=128