== sanity test 123ad: Verify batching statahead works correctly ========================================================== 16:56:08 (1788468968) batch: statahead_max=32 statahead_batch_max=32 mdc.lustre-MDT0000-mdc-ffff8a31c4e53000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff8a31c4e53000.batch_stats=0 llite.lustre-ffff8a31c4e53000.statahead_max=32 llite.lustre-ffff8a31c4e53000.statahead_batch_max=32 seq.cli-lustre-OST0000-super.width=0x1ffffff seq.cli-lustre-OST0001-super.width=0x1ffffff kvm mdc.lustre-MDT0000-mdc-ffff8a31c4e53000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff8a31c4e53000.batch_stats=0 striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d123ad.sanity total: 100 open/close in 2.54 seconds: 39.33 ops/second llite.lustre-ffff8a31c4e53000.statahead_max=0 101 real 0m2.674s user 0m0.008s sys 0m0.346s 'ls -l' 100 files without statahead: 3 sec llite.lustre-ffff8a31c4e53000.statahead_max=32 llite.lustre-ffff8a31c4e53000.statahead_max=32 101 real 0m0.978s user 0m0.006s sys 0m0.191s 'ls -l' 100 files with statahead: 1 sec statahead total: 109 statahead wrong: 2 agl total: 104 list_total: 31 fname_total: 78 hit_total: 3852 miss_total: 1586 swrong=2 ewrong=2 - open/close 766 (time 1788468997.19 total 10.00 last 76.58) total: 900 open/close in 11.66 seconds: 77.19 ops/second llite.lustre-ffff8a31c4e53000.statahead_max=0 1001 real 0m31.512s user 0m0.024s sys 0m2.850s 'ls -l' 1000 files without statahead: 31 sec llite.lustre-ffff8a31c4e53000.statahead_max=32 llite.lustre-ffff8a31c4e53000.statahead_max=32 1001 real 0m10.516s user 0m0.026s sys 0m1.376s 'ls -l' 1000 files with statahead: 10 sec statahead total: 110 statahead wrong: 2 agl total: 105 list_total: 32 fname_total: 78 hit_total: 4846 miss_total: 1592 swrong=2 ewrong=2 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 28 seconds llite.lustre-ffff8a31c4e53000.statahead_stats= statahead total: 110 statahead wrong: 2 agl total: 105 list_total: 32 fname_total: 78 hit_total: 4846 miss_total: 1592 mdc.lustre-MDT0000-mdc-ffff8a31c4e53000.batch_stats= snapshot_time: 1788469084.204290159 secs.nsecs start_time: 1788468973.694157028 secs.nsecs elapsed_time: 110.510133131 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 1 2 2 4: 3 7 10 8: 1 2 12 16: 20 51 64 32: 14 35 100 mdc.lustre-MDT0001-mdc-ffff8a31c4e53000.batch_stats= snapshot_time: 1788469084.204358638 secs.nsecs start_time: 1788468973.694229398 secs.nsecs elapsed_time: 110.510129240 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 1 2 2 4: 0 0 2 8: 6 15 17 16: 20 51 69 32: 12 30 100 - open/close 568 (time 1788469098.39 total 10.02 last 56.70) - open/close 1193 (time 1788469108.40 total 20.02 last 62.48) - open/close 1908 (time 1788469118.43 total 30.06 last 71.26) - open/close 2576 (time 1788469128.43 total 40.06 last 66.79) - open/close 3221 (time 1788469138.44 total 50.07 last 64.45) - open/close 3805 (time 1788469148.45 total 60.08 last 58.32) - open/close 4477 (time 1788469158.46 total 70.09 last 67.13) - open/close 5299 (time 1788469168.47 total 80.10 last 82.12) - open/close 6089 (time 1788469178.49 total 90.11 last 78.92) - open/close 6915 (time 1788469188.49 total 100.12 last 82.54) - open/close 7749 (time 1788469198.50 total 110.12 last 83.34) - open/close 8527 (time 1788469208.50 total 120.12 last 77.80) - open/close 9243 (time 1788469218.51 total 130.13 last 71.54) - open/close 9992 (time 1788469228.52 total 140.15 last 74.80) total: 10000 open/close in 140.27 seconds: 71.29 ops/second llite.lustre-ffff8a31c4e53000.statahead_batch_max=0 llite.lustre-ffff8a31c4e53000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff8a31c4e53000.stats=clear mdc.lustre-MDT0001-mdc-ffff8a31c4e53000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m6.886s user 0m0.317s sys 0m15.228s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff8a31c4e53000.statahead_batch_max=32 llite.lustre-ffff8a31c4e53000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff8a31c4e53000.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff8a31c4e53000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff8a31c4e53000.stats=clear mdc.lustre-MDT0001-mdc-ffff8a31c4e53000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m42.354s user 0m0.307s sys 0m14.457s batched statahead RPC= hit count=9980 mdc.lustre-MDT0000-mdc-ffff8a31c4e53000.batch_stats= snapshot_time: 1788469416.567601384 secs.nsecs start_time: 1788469304.008589874 secs.nsecs elapsed_time: 112.559011510 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 0 0 0 8: 1 0 0 16: 1 0 0 32: 313 99 100 mdc.lustre-MDT0001-mdc-ffff8a31c4e53000.batch_stats= snapshot_time: 1788469416.567667944 secs.nsecs start_time: 1788469304.008717097 secs.nsecs elapsed_time: 112.558950847 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 batch: statahead_max=2048 statahead_batch_max=256 mdc.lustre-MDT0000-mdc-ffff8a31c4e53000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff8a31c4e53000.batch_stats=0 llite.lustre-ffff8a31c4e53000.statahead_max=2048 llite.lustre-ffff8a31c4e53000.statahead_batch_max=256 kvm mdc.lustre-MDT0000-mdc-ffff8a31c4e53000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff8a31c4e53000.batch_stats=0 striped dir -i1 -c2 -H all_char /mnt/lustre/d123ad.sanity total: 100 open/close in 2.66 seconds: 37.59 ops/second llite.lustre-ffff8a31c4e53000.statahead_max=0 101 real 0m2.843s user 0m0.003s sys 0m0.320s 'ls -l' 100 files without statahead: 3 sec llite.lustre-ffff8a31c4e53000.statahead_max=2048 llite.lustre-ffff8a31c4e53000.statahead_max=2048 101 real 0m1.061s user 0m0.006s sys 0m0.191s 'ls -l' 100 files with statahead: 1 sec statahead total: 2 statahead wrong: 0 agl total: 2 list_total: 2 fname_total: 0 hit_total: 10079 miss_total: 21 swrong=0 ewrong=0 - open/close 626 (time 1788469743.38 total 10.00 last 62.58) total: 900 open/close in 15.37 seconds: 58.56 ops/second llite.lustre-ffff8a31c4e53000.statahead_max=0 1001 real 0m26.134s user 0m0.038s sys 0m3.129s 'ls -l' 1000 files without statahead: 26 sec llite.lustre-ffff8a31c4e53000.statahead_max=2048 llite.lustre-ffff8a31c4e53000.statahead_max=2048 1001 real 0m6.521s user 0m0.028s sys 0m1.293s 'ls -l' 1000 files with statahead: 7 sec statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11078 miss_total: 22 swrong=0 ewrong=0 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 31 seconds llite.lustre-ffff8a31c4e53000.statahead_stats= statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11078 miss_total: 22 mdc.lustre-MDT0000-mdc-ffff8a31c4e53000.batch_stats= snapshot_time: 1788469826.366572518 secs.nsecs start_time: 1788469419.316705093 secs.nsecs elapsed_time: 407.049867425 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 2 22 22 8: 3 33 55 16: 0 0 55 32: 1 11 66 64: 1 11 77 128: 0 0 77 256: 2 22 100 mdc.lustre-MDT0001-mdc-ffff8a31c4e53000.batch_stats= snapshot_time: 1788469826.366646394 secs.nsecs start_time: 1788469419.316886857 secs.nsecs elapsed_time: 407.049759537 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 1 11 11 4: 1 11 22 8: 2 22 44 16: 1 11 55 32: 2 22 77 64: 0 0 77 128: 0 0 77 256: 2 22 100 - open/close 827 (time 1788469839.42 total 10.02 last 82.57) - open/close 1554 (time 1788469849.43 total 20.03 last 72.62) - open/close 2177 (time 1788469859.43 total 30.03 last 62.25) - open/close 2757 (time 1788469869.46 total 40.06 last 57.83) - open/close 3362 (time 1788469879.48 total 50.08 last 60.41) - open/close 4035 (time 1788469889.48 total 60.08 last 67.26) - open/close 4773 (time 1788469899.49 total 70.09 last 73.72) - open/close 5350 (time 1788469909.50 total 80.11 last 57.64) - open/close 6033 (time 1788469919.52 total 90.12 last 68.22) - open/close 6694 (time 1788469929.52 total 100.12 last 66.07) - open/close 7392 (time 1788469939.53 total 110.13 last 69.72) - open/close 8075 (time 1788469949.53 total 120.14 last 68.28) - open/close 8733 (time 1788469959.54 total 130.14 last 65.76) - open/close 9341 (time 1788469969.54 total 140.14 last 60.79) - open/close 9926 (time 1788469979.56 total 150.16 last 58.42) total: 10000 open/close in 151.26 seconds: 66.11 ops/second llite.lustre-ffff8a31c4e53000.statahead_batch_max=0 llite.lustre-ffff8a31c4e53000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff8a31c4e53000.stats=clear mdc.lustre-MDT0001-mdc-ffff8a31c4e53000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m17.447s user 0m0.316s sys 0m14.889s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff8a31c4e53000.statahead_batch_max=256 llite.lustre-ffff8a31c4e53000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff8a31c4e53000.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff8a31c4e53000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff8a31c4e53000.stats=clear mdc.lustre-MDT0001-mdc-ffff8a31c4e53000.stats=clear ls -l files in d123ad.sanity: 10001 real 0m56.596s user 0m0.289s sys 0m14.524s batched statahead RPC= hit count=9999 mdc.lustre-MDT0000-mdc-ffff8a31c4e53000.batch_stats= snapshot_time: 1788470134.649012271 secs.nsecs start_time: 1788470066.982033665 secs.nsecs elapsed_time: 67.666978606 secs.nsecs subreqs per batch batches %% cum %% 1: 108 46 46 2: 28 12 58 4: 18 7 66 8: 8 3 70 16: 11 4 74 32: 11 4 79 64: 11 4 84 128: 2 0 85 256: 34 14 100 mdc.lustre-MDT0001-mdc-ffff8a31c4e53000.batch_stats= snapshot_time: 1788470134.649085981 secs.nsecs start_time: 1788470066.982155785 secs.nsecs elapsed_time: 67.666930196 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second seq.cli-lustre-OST0000-super.width=65536 seq.cli-lustre-OST0001-super.width=65536 llite.lustre-ffff8a31c4e53000.statahead_batch_max=64 llite.lustre-ffff8a31c4e53000.statahead_max=128