== sanity test 123ad: Verify batching statahead works correctly ========================================================== 19:57:17 (1787702237) batch: statahead_max=32 statahead_batch_max=32 mdc.lustre-MDT0000-mdc-ffffa042c55ad800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffffa042c55ad800.batch_stats=0 llite.lustre-ffffa042c55ad800.statahead_max=32 llite.lustre-ffffa042c55ad800.statahead_batch_max=32 seq.cli-lustre-OST0000-super.width=0x1ffffff seq.cli-lustre-OST0001-super.width=0x1ffffff kvm mdc.lustre-MDT0000-mdc-ffffa042c55ad800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffffa042c55ad800.batch_stats=0 striped dir -i1 -c2 -H crush /mnt/lustre/d123ad.sanity total: 100 open/close in 1.99 seconds: 50.36 ops/second llite.lustre-ffffa042c55ad800.statahead_max=0 101 real 0m2.859s user 0m0.006s sys 0m0.309s 'ls -l' 100 files without statahead: 3 sec llite.lustre-ffffa042c55ad800.statahead_max=32 llite.lustre-ffffa042c55ad800.statahead_max=32 101 real 0m1.140s user 0m0.004s sys 0m0.165s 'ls -l' 100 files with statahead: 1 sec statahead total: 128 statahead wrong: 2 agl total: 122 list_total: 31 fname_total: 91 hit_total: 3835 miss_total: 1613 swrong=2 ewrong=2 - open/close 855 (time 1787702265.86 total 10.01 last 85.44) total: 900 open/close in 10.54 seconds: 85.41 ops/second llite.lustre-ffffa042c55ad800.statahead_max=0 1001 real 0m21.963s user 0m0.032s sys 0m2.814s 'ls -l' 1000 files without statahead: 22 sec llite.lustre-ffffa042c55ad800.statahead_max=32 llite.lustre-ffffa042c55ad800.statahead_max=32 1001 real 0m9.113s user 0m0.034s sys 0m1.296s 'ls -l' 1000 files with statahead: 10 sec statahead total: 129 statahead wrong: 2 agl total: 123 list_total: 32 fname_total: 91 hit_total: 4834 miss_total: 1614 swrong=2 ewrong=2 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 19 seconds llite.lustre-ffffa042c55ad800.statahead_stats= statahead total: 129 statahead wrong: 2 agl total: 123 list_total: 32 fname_total: 91 hit_total: 4834 miss_total: 1614 mdc.lustre-MDT0000-mdc-ffffa042c55ad800.batch_stats= snapshot_time: 1787702329.391404526 secs.nsecs start_time: 1787702242.748356718 secs.nsecs elapsed_time: 86.643047808 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 1 2 2 4: 3 8 11 8: 1 2 13 16: 27 75 88 32: 4 11 100 mdc.lustre-MDT0001-mdc-ffffa042c55ad800.batch_stats= snapshot_time: 1787702329.391506616 secs.nsecs start_time: 1787702242.749181261 secs.nsecs elapsed_time: 86.642325355 secs.nsecs subreqs per batch batches %% cum %% 1: 1 2 2 2: 2 5 7 4: 0 0 7 8: 3 7 15 16: 8 20 35 32: 26 65 100 - open/close 828 (time 1787702342.27 total 10.01 last 82.71) - open/close 1673 (time 1787702352.29 total 20.03 last 84.33) - open/close 2524 (time 1787702362.30 total 30.04 last 85.07) - open/close 3364 (time 1787702372.30 total 40.04 last 83.95) - open/close 4191 (time 1787702382.31 total 50.05 last 82.66) - open/close 5026 (time 1787702392.31 total 60.05 last 83.45) - open/close 5878 (time 1787702402.33 total 70.06 last 85.09) - open/close 6681 (time 1787702412.33 total 80.07 last 80.27) - open/close 7546 (time 1787702422.33 total 90.07 last 86.45) - open/close 8331 (time 1787702432.34 total 100.07 last 78.49) - open/close 9163 (time 1787702442.34 total 110.08 last 83.18) - open/close 9983 (time 1787702452.35 total 120.09 last 81.90) total: 10000 open/close in 120.35 seconds: 83.09 ops/second llite.lustre-ffffa042c55ad800.statahead_batch_max=0 llite.lustre-ffffa042c55ad800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffffa042c55ad800.stats=clear mdc.lustre-MDT0001-mdc-ffffa042c55ad800.stats=clear ls -l files in d123ad.sanity: 10001 real 1m2.867s user 0m0.325s sys 0m14.561s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffffa042c55ad800.statahead_batch_max=32 llite.lustre-ffffa042c55ad800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffffa042c55ad800.batch_stats=clear mdc.lustre-MDT0001-mdc-ffffa042c55ad800.batch_stats=clear mdc.lustre-MDT0000-mdc-ffffa042c55ad800.stats=clear mdc.lustre-MDT0001-mdc-ffffa042c55ad800.stats=clear ls -l files in d123ad.sanity: 10001 real 1m39.212s user 0m0.277s sys 0m13.508s batched statahead RPC= hit count=9977 mdc.lustre-MDT0000-mdc-ffffa042c55ad800.batch_stats= snapshot_time: 1787702634.468605052 secs.nsecs start_time: 1787702524.208143083 secs.nsecs elapsed_time: 110.260461969 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 1 0 0 8: 1 0 0 16: 1 0 0 32: 313 99 100 mdc.lustre-MDT0001-mdc-ffffa042c55ad800.batch_stats= snapshot_time: 1787702634.468685850 secs.nsecs start_time: 1787702524.208370820 secs.nsecs elapsed_time: 110.260315030 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 batch: statahead_max=2048 statahead_batch_max=256 mdc.lustre-MDT0000-mdc-ffffa042c55ad800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffffa042c55ad800.batch_stats=0 llite.lustre-ffffa042c55ad800.statahead_max=2048 llite.lustre-ffffa042c55ad800.statahead_batch_max=256 kvm mdc.lustre-MDT0000-mdc-ffffa042c55ad800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffffa042c55ad800.batch_stats=0 striped dir -i1 -c2 -H crush2 /mnt/lustre/d123ad.sanity total: 100 open/close in 2.25 seconds: 44.54 ops/second llite.lustre-ffffa042c55ad800.statahead_max=0 101 real 0m1.848s user 0m0.004s sys 0m0.306s 'ls -l' 100 files without statahead: 2 sec llite.lustre-ffffa042c55ad800.statahead_max=2048 llite.lustre-ffffa042c55ad800.statahead_max=2048 101 real 0m0.802s user 0m0.003s sys 0m0.185s 'ls -l' 100 files with statahead: 1 sec statahead total: 2 statahead wrong: 0 agl total: 2 list_total: 2 fname_total: 0 hit_total: 10076 miss_total: 24 swrong=0 ewrong=0 total: 900 open/close in 9.91 seconds: 90.80 ops/second llite.lustre-ffffa042c55ad800.statahead_max=0 1001 real 0m20.353s user 0m0.031s sys 0m2.816s 'ls -l' 1000 files without statahead: 21 sec llite.lustre-ffffa042c55ad800.statahead_max=2048 llite.lustre-ffffa042c55ad800.statahead_max=2048 1001 real 0m6.506s user 0m0.027s sys 0m1.302s 'ls -l' 1000 files with statahead: 6 sec statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11075 miss_total: 25 swrong=0 ewrong=0 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 21 seconds llite.lustre-ffffa042c55ad800.statahead_stats= statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11075 miss_total: 25 mdc.lustre-MDT0000-mdc-ffffa042c55ad800.batch_stats= snapshot_time: 1787702973.867640205 secs.nsecs start_time: 1787702637.643968677 secs.nsecs elapsed_time: 336.223671528 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 1 20 20 8: 2 40 60 16: 0 0 60 32: 0 0 60 64: 0 0 60 128: 0 0 60 256: 2 40 100 mdc.lustre-MDT0001-mdc-ffffa042c55ad800.batch_stats= snapshot_time: 1787702973.867707245 secs.nsecs start_time: 1787702637.644252957 secs.nsecs elapsed_time: 336.223454288 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 1 11 11 4: 0 0 11 8: 2 22 33 16: 1 11 44 32: 1 11 55 64: 2 22 77 128: 0 0 77 256: 2 22 100 - open/close 935 (time 1787702986.37 total 10.01 last 93.43) - open/close 1862 (time 1787702996.37 total 20.01 last 92.68) - open/close 2743 (time 1787703006.38 total 30.01 last 88.06) - open/close 3600 (time 1787703016.38 total 40.02 last 85.67) - open/close 4585 (time 1787703026.39 total 50.02 last 98.43) - open/close 5432 (time 1787703036.40 total 60.04 last 84.60) - open/close 6284 (time 1787703046.41 total 70.05 last 85.10) - open/close 7201 (time 1787703056.41 total 80.05 last 91.68) - open/close 8123 (time 1787703066.42 total 90.06 last 92.15) - open/close 8884 (time 1787703076.42 total 100.06 last 76.09) - open/close 9564 (time 1787703086.42 total 110.06 last 68.00) total: 10000 open/close in 116.57 seconds: 85.78 ops/second llite.lustre-ffffa042c55ad800.statahead_batch_max=0 llite.lustre-ffffa042c55ad800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffffa042c55ad800.stats=clear mdc.lustre-MDT0001-mdc-ffffa042c55ad800.stats=clear ls -l files in d123ad.sanity: 10001 real 1m3.252s user 0m0.289s sys 0m14.925s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffffa042c55ad800.statahead_batch_max=256 llite.lustre-ffffa042c55ad800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffffa042c55ad800.batch_stats=clear mdc.lustre-MDT0001-mdc-ffffa042c55ad800.batch_stats=clear mdc.lustre-MDT0000-mdc-ffffa042c55ad800.stats=clear mdc.lustre-MDT0001-mdc-ffffa042c55ad800.stats=clear ls -l files in d123ad.sanity: 10001 real 0m51.106s user 0m0.306s sys 0m14.618s batched statahead RPC= hit count=9999 mdc.lustre-MDT0000-mdc-ffffa042c55ad800.batch_stats= snapshot_time: 1787703226.902183645 secs.nsecs start_time: 1787703166.064206328 secs.nsecs elapsed_time: 60.837977317 secs.nsecs subreqs per batch batches %% cum %% 1: 150 46 46 2: 35 10 56 4: 44 13 70 8: 28 8 79 16: 9 2 81 32: 15 4 86 64: 10 3 89 128: 0 0 89 256: 34 10 100 mdc.lustre-MDT0001-mdc-ffffa042c55ad800.batch_stats= snapshot_time: 1787703226.902253043 secs.nsecs start_time: 1787703166.064279690 secs.nsecs elapsed_time: 60.837973353 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second seq.cli-lustre-OST0000-super.width=65536 seq.cli-lustre-OST0001-super.width=65536 llite.lustre-ffffa042c55ad800.statahead_batch_max=64 llite.lustre-ffffa042c55ad800.statahead_max=128