== sanity test 123ad: Verify batching statahead works correctly ========================================================== 14:45:28 (1783968328) batch: statahead_max=32 statahead_batch_max=32 mdc.lustre-MDT0000-mdc-ffff9404c8ca3800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9404c8ca3800.batch_stats=0 llite.lustre-ffff9404c8ca3800.statahead_max=32 llite.lustre-ffff9404c8ca3800.statahead_batch_max=32 seq.cli-lustre-OST0000-super.width=0x1ffffff seq.cli-lustre-OST0001-super.width=0x1ffffff kvm mdc.lustre-MDT0000-mdc-ffff9404c8ca3800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9404c8ca3800.batch_stats=0 striped dir -i1 -c2 -H crush /mnt/lustre/d123ad.sanity total: 100 open/close in 1.89 seconds: 52.89 ops/second llite.lustre-ffff9404c8ca3800.statahead_max=0 101 real 0m3.154s user 0m0.005s sys 0m0.371s 'ls -l' 100 files without statahead: 3 sec llite.lustre-ffff9404c8ca3800.statahead_max=32 llite.lustre-ffff9404c8ca3800.statahead_max=32 101 real 0m1.599s user 0m0.001s sys 0m0.206s 'ls -l' 100 files with statahead: 1 sec statahead total: 127 statahead wrong: 2 agl total: 121 list_total: 32 fname_total: 81 hit_total: 3642 miss_total: 1337 swrong=2 ewrong=2 - open/close 638 (time 1783968359.96 total 10.01 last 63.74) total: 900 open/close in 13.97 seconds: 64.40 ops/second llite.lustre-ffff9404c8ca3800.statahead_max=0 1001 real 0m28.103s user 0m0.040s sys 0m3.259s 'ls -l' 1000 files without statahead: 28 sec llite.lustre-ffff9404c8ca3800.statahead_max=32 llite.lustre-ffff9404c8ca3800.statahead_max=32 1001 real 0m8.726s user 0m0.029s sys 0m1.456s 'ls -l' 1000 files with statahead: 9 sec statahead total: 128 statahead wrong: 2 agl total: 122 list_total: 33 fname_total: 81 hit_total: 4638 miss_total: 1341 swrong=2 ewrong=2 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 17 seconds llite.lustre-ffff9404c8ca3800.statahead_stats= statahead total: 128 statahead wrong: 2 agl total: 122 list_total: 33 fname_total: 81 hit_total: 4638 miss_total: 1341 mdc.lustre-MDT0000-mdc-ffff9404c8ca3800.batch_stats= snapshot_time: 1783968430.093028898 secs.nsecs start_time: 1783968334.680872290 secs.nsecs elapsed_time: 95.412156608 secs.nsecs subreqs per batch batches %% cum %% 1: 4 10 10 2: 2 5 15 4: 0 0 15 8: 1 2 18 16: 24 63 81 32: 7 18 100 mdc.lustre-MDT0001-mdc-ffff9404c8ca3800.batch_stats= snapshot_time: 1783968430.093095149 secs.nsecs start_time: 1783968334.680966653 secs.nsecs elapsed_time: 95.412128496 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 1 2 2 8: 5 12 15 16: 10 25 41 32: 23 58 100 - open/close 985 (time 1783968442.12 total 10.00 last 98.47) - open/close 1889 (time 1783968452.12 total 20.00 last 90.39) - open/close 2814 (time 1783968462.13 total 30.02 last 92.33) - open/close 3678 (time 1783968472.14 total 40.03 last 86.33) - open/close 4520 (time 1783968482.15 total 50.03 last 84.16) - open/close 5438 (time 1783968492.15 total 60.04 last 91.74) - open/close 6451 (time 1783968502.16 total 70.05 last 101.24) - open/close 7368 (time 1783968512.16 total 80.05 last 91.70) - open/close 8209 (time 1783968522.18 total 90.07 last 83.96) - open/close 9228 (time 1783968532.18 total 100.07 last 101.86) total: 10000 open/close in 107.18 seconds: 93.30 ops/second llite.lustre-ffff9404c8ca3800.statahead_batch_max=0 llite.lustre-ffff9404c8ca3800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9404c8ca3800.stats=clear mdc.lustre-MDT0001-mdc-ffff9404c8ca3800.stats=clear ls -l files in d123ad.sanity: 10001 real 1m4.072s user 0m0.306s sys 0m15.513s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff9404c8ca3800.statahead_batch_max=32 llite.lustre-ffff9404c8ca3800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9404c8ca3800.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff9404c8ca3800.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff9404c8ca3800.stats=clear mdc.lustre-MDT0001-mdc-ffff9404c8ca3800.stats=clear ls -l files in d123ad.sanity: 10001 real 1m48.703s user 0m0.320s sys 0m15.267s batched statahead RPC= hit count=9982 mdc.lustre-MDT0000-mdc-ffff9404c8ca3800.batch_stats= snapshot_time: 1783968738.980524437 secs.nsecs start_time: 1783968614.666336160 secs.nsecs elapsed_time: 124.314188277 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 0 0 0 8: 1 0 0 16: 1 0 0 32: 313 99 100 mdc.lustre-MDT0001-mdc-ffff9404c8ca3800.batch_stats= snapshot_time: 1783968738.980588877 secs.nsecs start_time: 1783968614.672060901 secs.nsecs elapsed_time: 124.308527976 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 batch: statahead_max=2048 statahead_batch_max=256 mdc.lustre-MDT0000-mdc-ffff9404c8ca3800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9404c8ca3800.batch_stats=0 llite.lustre-ffff9404c8ca3800.statahead_max=2048 llite.lustre-ffff9404c8ca3800.statahead_batch_max=256 kvm mdc.lustre-MDT0000-mdc-ffff9404c8ca3800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9404c8ca3800.batch_stats=0 striped dir -i1 -c2 -H crush2 /mnt/lustre/d123ad.sanity total: 100 open/close in 2.05 seconds: 48.69 ops/second llite.lustre-ffff9404c8ca3800.statahead_max=0 101 real 0m2.293s user 0m0.004s sys 0m0.322s 'ls -l' 100 files without statahead: 3 sec llite.lustre-ffff9404c8ca3800.statahead_max=2048 llite.lustre-ffff9404c8ca3800.statahead_max=2048 101 real 0m0.838s user 0m0.002s sys 0m0.192s 'ls -l' 100 files with statahead: 1 sec statahead total: 2 statahead wrong: 0 agl total: 2 list_total: 2 fname_total: 0 hit_total: 10081 miss_total: 19 swrong=0 ewrong=0 total: 900 open/close in 9.35 seconds: 96.21 ops/second llite.lustre-ffff9404c8ca3800.statahead_max=0 1001 real 0m26.994s user 0m0.035s sys 0m3.524s 'ls -l' 1000 files without statahead: 27 sec llite.lustre-ffff9404c8ca3800.statahead_max=2048 llite.lustre-ffff9404c8ca3800.statahead_max=2048 1001 real 0m7.352s user 0m0.029s sys 0m1.460s 'ls -l' 1000 files with statahead: 7 sec statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11080 miss_total: 20 swrong=0 ewrong=0 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 21 seconds llite.lustre-ffff9404c8ca3800.statahead_stats= statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11080 miss_total: 20 mdc.lustre-MDT0000-mdc-ffff9404c8ca3800.batch_stats= snapshot_time: 1783969088.612434089 secs.nsecs start_time: 1783968741.643300867 secs.nsecs elapsed_time: 346.969133222 secs.nsecs subreqs per batch batches %% cum %% 1: 2 22 22 2: 0 0 22 4: 1 11 33 8: 1 11 44 16: 2 22 66 32: 1 11 77 64: 0 0 77 128: 1 11 88 256: 1 11 100 mdc.lustre-MDT0001-mdc-ffff9404c8ca3800.batch_stats= snapshot_time: 1783969088.612504223 secs.nsecs start_time: 1783968741.643452458 secs.nsecs elapsed_time: 346.969051765 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 1 11 11 8: 3 33 44 16: 0 0 44 32: 1 11 55 64: 1 11 66 128: 1 11 77 256: 2 22 100 - open/close 726 (time 1783969101.22 total 10.01 last 72.54) - open/close 1538 (time 1783969111.22 total 20.01 last 81.17) - open/close 2327 (time 1783969121.23 total 30.02 last 78.85) - open/close 3189 (time 1783969131.24 total 40.03 last 86.14) - open/close 4083 (time 1783969141.24 total 50.03 last 89.37) - open/close 4834 (time 1783969151.25 total 60.03 last 75.06) - open/close 5537 (time 1783969161.26 total 70.05 last 70.21) - open/close 6247 (time 1783969171.27 total 80.06 last 70.91) - open/close 6817 (time 1783969181.29 total 90.08 last 56.91) - open/close 7621 (time 1783969191.29 total 100.08 last 80.35) - open/close 8456 (time 1783969201.31 total 110.09 last 83.40) - open/close 9180 (time 1783969211.31 total 120.10 last 72.38) - open/close 9868 (time 1783969221.31 total 130.10 last 68.79) total: 10000 open/close in 131.87 seconds: 75.83 ops/second llite.lustre-ffff9404c8ca3800.statahead_batch_max=0 llite.lustre-ffff9404c8ca3800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9404c8ca3800.stats=clear mdc.lustre-MDT0001-mdc-ffff9404c8ca3800.stats=clear ls -l files in d123ad.sanity: 10001 real 1m9.083s user 0m0.326s sys 0m16.099s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff9404c8ca3800.statahead_batch_max=256 llite.lustre-ffff9404c8ca3800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9404c8ca3800.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff9404c8ca3800.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff9404c8ca3800.stats=clear mdc.lustre-MDT0001-mdc-ffff9404c8ca3800.stats=clear ls -l files in d123ad.sanity: 10001 real 0m59.768s user 0m0.307s sys 0m15.839s batched statahead RPC= hit count=9999 mdc.lustre-MDT0000-mdc-ffff9404c8ca3800.batch_stats= snapshot_time: 1783969384.304286251 secs.nsecs start_time: 1783969308.650551614 secs.nsecs elapsed_time: 75.653734637 secs.nsecs subreqs per batch batches %% cum %% 1: 239 51 51 2: 66 14 66 4: 53 11 77 8: 31 6 84 16: 14 3 87 32: 17 3 91 64: 7 1 92 128: 0 0 92 256: 33 7 100 mdc.lustre-MDT0001-mdc-ffff9404c8ca3800.batch_stats= snapshot_time: 1783969384.304351277 secs.nsecs start_time: 1783969308.650680202 secs.nsecs elapsed_time: 75.653671075 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second seq.cli-lustre-OST0000-super.width=65536 seq.cli-lustre-OST0001-super.width=65536 llite.lustre-ffff9404c8ca3800.statahead_batch_max=64 llite.lustre-ffff9404c8ca3800.statahead_max=128