== sanity test 123ad: Verify batching statahead works correctly ========================================================== 09:48:56 (1788184136) batch: statahead_max=32 statahead_batch_max=32 mdc.lustre-MDT0000-mdc-ffff9c7b90e61000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9c7b90e61000.batch_stats=0 llite.lustre-ffff9c7b90e61000.statahead_max=32 llite.lustre-ffff9c7b90e61000.statahead_batch_max=32 seq.cli-lustre-OST0000-super.width=0x1ffffff seq.cli-lustre-OST0001-super.width=0x1ffffff kvm mdc.lustre-MDT0000-mdc-ffff9c7b90e61000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9c7b90e61000.batch_stats=0 striped dir -i1 -c2 -H crush /mnt/lustre/d123ad.sanity total: 100 open/close in 3.28 seconds: 30.50 ops/second llite.lustre-ffff9c7b90e61000.statahead_max=0 101 real 0m3.369s user 0m0.003s sys 0m0.362s 'ls -l' 100 files without statahead: 4 sec llite.lustre-ffff9c7b90e61000.statahead_max=32 llite.lustre-ffff9c7b90e61000.statahead_max=32 101 real 0m1.268s user 0m0.002s sys 0m0.195s 'ls -l' 100 files with statahead: 2 sec statahead total: 133 statahead wrong: 2 agl total: 126 list_total: 30 fname_total: 101 hit_total: 3980 miss_total: 1360 swrong=2 ewrong=2 - open/close 818 (time 1788184167.88 total 10.01 last 81.73) total: 900 open/close in 11.39 seconds: 79.00 ops/second llite.lustre-ffff9c7b90e61000.statahead_max=0 1001 real 0m23.408s user 0m0.025s sys 0m3.196s 'ls -l' 1000 files without statahead: 23 sec llite.lustre-ffff9c7b90e61000.statahead_max=32 llite.lustre-ffff9c7b90e61000.statahead_max=32 1001 real 0m8.718s user 0m0.024s sys 0m1.460s 'ls -l' 1000 files with statahead: 9 sec statahead total: 134 statahead wrong: 2 agl total: 127 list_total: 31 fname_total: 101 hit_total: 4976 miss_total: 1364 swrong=2 ewrong=2 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 20 seconds llite.lustre-ffff9c7b90e61000.statahead_stats= statahead total: 134 statahead wrong: 2 agl total: 127 list_total: 31 fname_total: 101 hit_total: 4976 miss_total: 1364 mdc.lustre-MDT0000-mdc-ffff9c7b90e61000.batch_stats= snapshot_time: 1788184233.804939269 secs.nsecs start_time: 1788184142.978977147 secs.nsecs elapsed_time: 90.825962122 secs.nsecs subreqs per batch batches %% cum %% 1: 3 7 7 2: 1 2 10 4: 3 7 18 8: 1 2 21 16: 25 65 86 32: 5 13 100 mdc.lustre-MDT0001-mdc-ffff9c7b90e61000.batch_stats= snapshot_time: 1788184233.805005914 secs.nsecs start_time: 1788184142.979073422 secs.nsecs elapsed_time: 90.825932492 secs.nsecs subreqs per batch batches %% cum %% 1: 1 2 2 2: 0 0 2 4: 2 5 7 8: 3 7 15 16: 10 25 40 32: 24 60 100 - open/close 933 (time 1788184246.69 total 10.00 last 93.27) - open/close 1794 (time 1788184256.69 total 20.00 last 86.09) - open/close 2661 (time 1788184266.70 total 30.02 last 86.61) - open/close 3480 (time 1788184276.70 total 40.02 last 81.90) - open/close 4220 (time 1788184286.71 total 50.03 last 73.91) - open/close 5178 (time 1788184296.72 total 60.04 last 95.69) - open/close 6190 (time 1788184306.73 total 70.04 last 101.15) - open/close 7126 (time 1788184316.73 total 80.05 last 93.55) - open/close 8132 (time 1788184326.74 total 90.05 last 100.57) - open/close 9052 (time 1788184336.74 total 100.06 last 91.95) total: 10000 open/close in 109.40 seconds: 91.41 ops/second llite.lustre-ffff9c7b90e61000.statahead_batch_max=0 llite.lustre-ffff9c7b90e61000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9c7b90e61000.stats=clear mdc.lustre-MDT0001-mdc-ffff9c7b90e61000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m0.839s user 0m0.289s sys 0m16.427s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff9c7b90e61000.statahead_batch_max=32 llite.lustre-ffff9c7b90e61000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9c7b90e61000.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff9c7b90e61000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff9c7b90e61000.stats=clear mdc.lustre-MDT0001-mdc-ffff9c7b90e61000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m37.765s user 0m0.285s sys 0m15.437s batched statahead RPC= hit count=9986 mdc.lustre-MDT0000-mdc-ffff9c7b90e61000.batch_stats= snapshot_time: 1788184527.149016950 secs.nsecs start_time: 1788184417.383648057 secs.nsecs elapsed_time: 109.765368893 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 0 0 0 8: 1 0 0 16: 1 0 0 32: 313 99 100 mdc.lustre-MDT0001-mdc-ffff9c7b90e61000.batch_stats= snapshot_time: 1788184527.149084845 secs.nsecs start_time: 1788184417.383785954 secs.nsecs elapsed_time: 109.765298891 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 batch: statahead_max=2048 statahead_batch_max=256 mdc.lustre-MDT0000-mdc-ffff9c7b90e61000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9c7b90e61000.batch_stats=0 llite.lustre-ffff9c7b90e61000.statahead_max=2048 llite.lustre-ffff9c7b90e61000.statahead_batch_max=256 kvm mdc.lustre-MDT0000-mdc-ffff9c7b90e61000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9c7b90e61000.batch_stats=0 striped dir -i1 -c2 -H all_char /mnt/lustre/d123ad.sanity total: 100 open/close in 2.12 seconds: 47.13 ops/second llite.lustre-ffff9c7b90e61000.statahead_max=0 101 real 0m2.730s user 0m0.003s sys 0m0.360s 'ls -l' 100 files without statahead: 2 sec llite.lustre-ffff9c7b90e61000.statahead_max=2048 llite.lustre-ffff9c7b90e61000.statahead_max=2048 101 real 0m1.037s user 0m0.007s sys 0m0.212s 'ls -l' 100 files with statahead: 1 sec statahead total: 2 statahead wrong: 0 agl total: 2 list_total: 2 fname_total: 0 hit_total: 10085 miss_total: 15 swrong=0 ewrong=0 - open/close 836 (time 1788184806.16 total 10.00 last 83.57) total: 900 open/close in 11.00 seconds: 81.85 ops/second llite.lustre-ffff9c7b90e61000.statahead_max=0 1001 real 0m18.225s user 0m0.033s sys 0m3.135s 'ls -l' 1000 files without statahead: 18 sec llite.lustre-ffff9c7b90e61000.statahead_max=2048 llite.lustre-ffff9c7b90e61000.statahead_max=2048 1001 real 0m7.418s user 0m0.031s sys 0m1.613s 'ls -l' 1000 files with statahead: 8 sec statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11084 miss_total: 16 swrong=0 ewrong=0 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 24 seconds llite.lustre-ffff9c7b90e61000.statahead_stats= statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11084 miss_total: 16 mdc.lustre-MDT0000-mdc-ffff9c7b90e61000.batch_stats= snapshot_time: 1788184869.071763898 secs.nsecs start_time: 1788184529.055657515 secs.nsecs elapsed_time: 340.016106383 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 1 12 12 8: 4 50 62 16: 0 0 62 32: 1 12 75 64: 0 0 75 128: 0 0 75 256: 2 25 100 mdc.lustre-MDT0001-mdc-ffff9c7b90e61000.batch_stats= snapshot_time: 1788184869.071842887 secs.nsecs start_time: 1788184529.056280273 secs.nsecs elapsed_time: 340.015562614 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 2 25 25 4: 1 12 37 8: 1 12 50 16: 1 12 62 32: 0 0 62 64: 1 12 75 128: 0 0 75 256: 2 25 100 - open/close 913 (time 1788184881.52 total 10.01 last 91.22) - open/close 1835 (time 1788184891.53 total 20.01 last 92.17) - open/close 2763 (time 1788184901.53 total 30.01 last 92.80) - open/close 3650 (time 1788184911.54 total 40.02 last 88.61) - open/close 4632 (time 1788184921.54 total 50.02 last 98.19) - open/close 5538 (time 1788184931.55 total 60.03 last 90.53) - open/close 6387 (time 1788184941.55 total 70.04 last 84.87) - open/close 7355 (time 1788184951.56 total 80.05 last 96.70) - open/close 8341 (time 1788184961.57 total 90.05 last 98.55) - open/close 9371 (time 1788184971.57 total 100.06 last 102.92) total: 10000 open/close in 106.73 seconds: 93.70 ops/second llite.lustre-ffff9c7b90e61000.statahead_batch_max=0 llite.lustre-ffff9c7b90e61000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9c7b90e61000.stats=clear mdc.lustre-MDT0001-mdc-ffff9c7b90e61000.stats=clear ls -l files in d123ad.sanity: 10001 real 0m56.036s user 0m0.324s sys 0m16.078s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff9c7b90e61000.statahead_batch_max=256 llite.lustre-ffff9c7b90e61000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9c7b90e61000.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff9c7b90e61000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff9c7b90e61000.stats=clear mdc.lustre-MDT0001-mdc-ffff9c7b90e61000.stats=clear ls -l files in d123ad.sanity: 10001 real 0m58.415s user 0m0.327s sys 0m16.676s batched statahead RPC= hit count=9999 mdc.lustre-MDT0000-mdc-ffff9c7b90e61000.batch_stats= snapshot_time: 1788185119.905475461 secs.nsecs start_time: 1788185045.217565654 secs.nsecs elapsed_time: 74.687909807 secs.nsecs subreqs per batch batches %% cum %% 1: 206 55 55 2: 45 12 67 4: 30 8 75 8: 26 6 82 16: 14 3 86 32: 13 3 89 64: 4 1 90 128: 1 0 90 256: 34 9 100 mdc.lustre-MDT0001-mdc-ffff9c7b90e61000.batch_stats= snapshot_time: 1788185119.905538054 secs.nsecs start_time: 1788185045.217645614 secs.nsecs elapsed_time: 74.687892440 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second seq.cli-lustre-OST0000-super.width=65536 seq.cli-lustre-OST0001-super.width=65536 llite.lustre-ffff9c7b90e61000.statahead_batch_max=64 llite.lustre-ffff9c7b90e61000.statahead_max=128