== sanity test 123ad: Verify batching statahead works correctly ========================================================== 09:15:18 (1789132518) batch: statahead_max=32 statahead_batch_max=32 mdc.lustre-MDT0000-mdc-ffff8a18a0c52000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff8a18a0c52000.batch_stats=0 llite.lustre-ffff8a18a0c52000.statahead_max=32 llite.lustre-ffff8a18a0c52000.statahead_batch_max=32 seq.cli-lustre-OST0000-super.width=0x1ffffff seq.cli-lustre-OST0001-super.width=0x1ffffff kvm mdc.lustre-MDT0000-mdc-ffff8a18a0c52000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff8a18a0c52000.batch_stats=0 striped dir -i1 -c2 -H all_char /mnt/lustre/d123ad.sanity total: 100 open/close in 3.47 seconds: 28.78 ops/second llite.lustre-ffff8a18a0c52000.statahead_max=0 101 real 0m3.426s user 0m0.005s sys 0m0.350s 'ls -l' 100 files without statahead: 4 sec llite.lustre-ffff8a18a0c52000.statahead_max=32 llite.lustre-ffff8a18a0c52000.statahead_max=32 101 real 0m1.061s user 0m0.004s sys 0m0.182s 'ls -l' 100 files with statahead: 1 sec statahead total: 135 statahead wrong: 2 agl total: 129 list_total: 32 fname_total: 102 hit_total: 4170 miss_total: 1291 swrong=2 ewrong=2 - open/close 749 (time 1789132550.69 total 10.02 last 74.78) total: 900 open/close in 12.17 seconds: 73.92 ops/second llite.lustre-ffff8a18a0c52000.statahead_max=0 1001 real 0m23.082s user 0m0.041s sys 0m3.058s 'ls -l' 1000 files without statahead: 23 sec llite.lustre-ffff8a18a0c52000.statahead_max=32 llite.lustre-ffff8a18a0c52000.statahead_max=32 1001 real 0m8.059s user 0m0.029s sys 0m1.541s 'ls -l' 1000 files with statahead: 8 sec statahead total: 136 statahead wrong: 2 agl total: 130 list_total: 33 fname_total: 102 hit_total: 5167 miss_total: 1294 swrong=2 ewrong=2 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 19 seconds llite.lustre-ffff8a18a0c52000.statahead_stats= statahead total: 136 statahead wrong: 2 agl total: 130 list_total: 33 fname_total: 102 hit_total: 5167 miss_total: 1294 mdc.lustre-MDT0000-mdc-ffff8a18a0c52000.batch_stats= snapshot_time: 1789132614.295389012 secs.nsecs start_time: 1789132523.328008806 secs.nsecs elapsed_time: 90.967380206 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 2 5 5 4: 3 7 12 8: 2 5 17 16: 21 52 70 32: 12 30 100 mdc.lustre-MDT0001-mdc-ffff8a18a0c52000.batch_stats= snapshot_time: 1789132614.295457397 secs.nsecs start_time: 1789132523.328168035 secs.nsecs elapsed_time: 90.967289362 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 2 5 5 4: 0 0 5 8: 5 12 17 16: 19 47 65 32: 14 35 100 - open/close 784 (time 1789132626.92 total 10.00 last 78.39) - open/close 1542 (time 1789132636.92 total 20.01 last 75.76) - open/close 2270 (time 1789132646.93 total 30.01 last 72.78) - open/close 3003 (time 1789132656.95 total 40.03 last 73.12) - open/close 3712 (time 1789132666.96 total 50.04 last 70.87) - open/close 4481 (time 1789132676.98 total 60.07 last 76.70) - open/close 5258 (time 1789132687.00 total 70.08 last 77.57) - open/close 6023 (time 1789132697.00 total 80.09 last 76.48) - open/close 6713 (time 1789132707.00 total 90.09 last 68.99) - open/close 7462 (time 1789132717.01 total 100.09 last 74.85) - open/close 8230 (time 1789132727.01 total 110.09 last 76.80) - open/close 8958 (time 1789132737.02 total 120.10 last 72.76) - open/close 9724 (time 1789132747.02 total 130.10 last 76.59) total: 10000 open/close in 133.94 seconds: 74.66 ops/second llite.lustre-ffff8a18a0c52000.statahead_batch_max=0 llite.lustre-ffff8a18a0c52000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff8a18a0c52000.stats=clear mdc.lustre-MDT0001-mdc-ffff8a18a0c52000.stats=clear ls -l files in d123ad.sanity: 10001 real 0m59.478s user 0m0.362s sys 0m17.173s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff8a18a0c52000.statahead_batch_max=32 llite.lustre-ffff8a18a0c52000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff8a18a0c52000.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff8a18a0c52000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff8a18a0c52000.stats=clear mdc.lustre-MDT0001-mdc-ffff8a18a0c52000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m43.735s user 0m0.329s sys 0m14.637s batched statahead RPC= hit count=9987 mdc.lustre-MDT0000-mdc-ffff8a18a0c52000.batch_stats= snapshot_time: 1789132933.964425053 secs.nsecs start_time: 1789132818.907823111 secs.nsecs elapsed_time: 115.056601942 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 0 0 0 8: 1 0 0 16: 1 0 0 32: 313 99 100 mdc.lustre-MDT0001-mdc-ffff8a18a0c52000.batch_stats= snapshot_time: 1789132933.964491636 secs.nsecs start_time: 1789132818.907912674 secs.nsecs elapsed_time: 115.056578962 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 batch: statahead_max=2048 statahead_batch_max=256 mdc.lustre-MDT0000-mdc-ffff8a18a0c52000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff8a18a0c52000.batch_stats=0 llite.lustre-ffff8a18a0c52000.statahead_max=2048 llite.lustre-ffff8a18a0c52000.statahead_batch_max=256 kvm mdc.lustre-MDT0000-mdc-ffff8a18a0c52000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff8a18a0c52000.batch_stats=0 striped dir -i1 -c2 -H crush2 /mnt/lustre/d123ad.sanity total: 100 open/close in 2.68 seconds: 37.34 ops/second llite.lustre-ffff8a18a0c52000.statahead_max=0 101 real 0m2.527s user 0m0.005s sys 0m0.332s 'ls -l' 100 files without statahead: 2 sec llite.lustre-ffff8a18a0c52000.statahead_max=2048 llite.lustre-ffff8a18a0c52000.statahead_max=2048 101 real 0m0.763s user 0m0.004s sys 0m0.202s 'ls -l' 100 files with statahead: 1 sec statahead total: 2 statahead wrong: 0 agl total: 2 list_total: 2 fname_total: 0 hit_total: 10086 miss_total: 14 swrong=0 ewrong=0 - open/close 716 (time 1789133249.16 total 10.01 last 71.55) total: 900 open/close in 12.27 seconds: 73.34 ops/second llite.lustre-ffff8a18a0c52000.statahead_max=0 1001 real 0m21.143s user 0m0.037s sys 0m2.939s 'ls -l' 1000 files without statahead: 21 sec llite.lustre-ffff8a18a0c52000.statahead_max=2048 llite.lustre-ffff8a18a0c52000.statahead_max=2048 1001 real 0m5.374s user 0m0.029s sys 0m1.459s 'ls -l' 1000 files with statahead: 6 sec statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11085 miss_total: 15 swrong=0 ewrong=0 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 18 seconds llite.lustre-ffff8a18a0c52000.statahead_stats= statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11085 miss_total: 15 mdc.lustre-MDT0000-mdc-ffff8a18a0c52000.batch_stats= snapshot_time: 1789133306.850507786 secs.nsecs start_time: 1789132936.161731571 secs.nsecs elapsed_time: 370.688776215 secs.nsecs subreqs per batch batches %% cum %% 1: 1 12 12 2: 2 25 37 4: 2 25 62 8: 0 0 62 16: 0 0 62 32: 1 12 75 64: 0 0 75 128: 0 0 75 256: 2 25 100 mdc.lustre-MDT0001-mdc-ffff8a18a0c52000.batch_stats= snapshot_time: 1789133306.850587643 secs.nsecs start_time: 1789132936.161887211 secs.nsecs elapsed_time: 370.688700432 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 0 0 0 8: 4 36 36 16: 1 9 45 32: 4 36 81 64: 0 0 81 128: 0 0 81 256: 2 18 100 - open/close 720 (time 1789133319.11 total 10.00 last 71.98) - open/close 1426 (time 1789133329.12 total 20.02 last 70.51) - open/close 2144 (time 1789133339.13 total 30.02 last 71.74) - open/close 2863 (time 1789133349.14 total 40.03 last 71.83) - open/close 3557 (time 1789133359.14 total 50.04 last 69.37) - open/close 4279 (time 1789133369.15 total 60.05 last 72.13) - open/close 5030 (time 1789133379.16 total 70.06 last 75.04) - open/close 5755 (time 1789133389.17 total 80.06 last 72.46) - open/close 6432 (time 1789133399.17 total 90.07 last 67.67) - open/close 7123 (time 1789133409.19 total 100.08 last 68.98) - open/close 7768 (time 1789133419.20 total 110.09 last 64.46) - open/close 8493 (time 1789133429.20 total 120.10 last 72.44) - open/close 9060 (time 1789133439.22 total 130.11 last 56.61) - open/close 9658 (time 1789133449.23 total 140.12 last 59.75) total: 10000 open/close in 144.96 seconds: 68.98 ops/second llite.lustre-ffff8a18a0c52000.statahead_batch_max=0 llite.lustre-ffff8a18a0c52000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff8a18a0c52000.stats=clear mdc.lustre-MDT0001-mdc-ffff8a18a0c52000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m9.573s user 0m0.366s sys 0m16.271s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff8a18a0c52000.statahead_batch_max=256 llite.lustre-ffff8a18a0c52000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff8a18a0c52000.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff8a18a0c52000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff8a18a0c52000.stats=clear mdc.lustre-MDT0001-mdc-ffff8a18a0c52000.stats=clear ls -l files in d123ad.sanity: 10001 real 0m52.912s user 0m0.336s sys 0m15.941s batched statahead RPC= hit count=9999 mdc.lustre-MDT0000-mdc-ffff8a18a0c52000.batch_stats= snapshot_time: 1789133597.873844120 secs.nsecs start_time: 1789133532.508868789 secs.nsecs elapsed_time: 65.364975331 secs.nsecs subreqs per batch batches %% cum %% 1: 123 46 46 2: 34 12 58 4: 34 12 71 8: 9 3 74 16: 8 2 77 32: 20 7 85 64: 3 1 86 128: 1 0 86 256: 35 13 100 mdc.lustre-MDT0001-mdc-ffff8a18a0c52000.batch_stats= snapshot_time: 1789133597.873946975 secs.nsecs start_time: 1789133532.508943876 secs.nsecs elapsed_time: 65.365003099 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second seq.cli-lustre-OST0000-super.width=65536 seq.cli-lustre-OST0001-super.width=65536 llite.lustre-ffff8a18a0c52000.statahead_batch_max=64 llite.lustre-ffff8a18a0c52000.statahead_max=128