== sanity test 123ad: Verify batching statahead works correctly ========================================================== 16:08:59 (1789589339) batch: statahead_max=32 statahead_batch_max=32 mdc.lustre-MDT0000-mdc-ffff9a086e1cc000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9a086e1cc000.batch_stats=0 llite.lustre-ffff9a086e1cc000.statahead_max=32 llite.lustre-ffff9a086e1cc000.statahead_batch_max=32 seq.cli-lustre-OST0000-super.width=0x1ffffff seq.cli-lustre-OST0001-super.width=0x1ffffff kvm mdc.lustre-MDT0000-mdc-ffff9a086e1cc000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9a086e1cc000.batch_stats=0 striped dir -i1 -c2 -H crush2 /mnt/lustre/d123ad.sanity total: 100 open/close in 2.21 seconds: 45.23 ops/second llite.lustre-ffff9a086e1cc000.statahead_max=0 101 real 0m2.246s user 0m0.005s sys 0m0.294s 'ls -l' 100 files without statahead: 3 sec llite.lustre-ffff9a086e1cc000.statahead_max=32 llite.lustre-ffff9a086e1cc000.statahead_max=32 101 real 0m0.915s user 0m0.003s sys 0m0.138s 'ls -l' 100 files with statahead: 1 sec statahead total: 135 statahead wrong: 2 agl total: 129 list_total: 30 fname_total: 88 hit_total: 3925 miss_total: 1466 swrong=2 ewrong=2 total: 900 open/close in 9.98 seconds: 90.20 ops/second llite.lustre-ffff9a086e1cc000.statahead_max=0 1001 real 0m24.784s user 0m0.035s sys 0m2.809s 'ls -l' 1000 files without statahead: 25 sec llite.lustre-ffff9a086e1cc000.statahead_max=32 llite.lustre-ffff9a086e1cc000.statahead_max=32 1001 real 0m8.752s user 0m0.030s sys 0m1.306s 'ls -l' 1000 files with statahead: 8 sec statahead total: 136 statahead wrong: 2 agl total: 130 list_total: 31 fname_total: 88 hit_total: 4920 miss_total: 1471 swrong=2 ewrong=2 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 20 seconds llite.lustre-ffff9a086e1cc000.statahead_stats= statahead total: 136 statahead wrong: 2 agl total: 130 list_total: 31 fname_total: 88 hit_total: 4920 miss_total: 1471 mdc.lustre-MDT0000-mdc-ffff9a086e1cc000.batch_stats= snapshot_time: 1789589434.062154353 secs.nsecs start_time: 1789589346.167958384 secs.nsecs elapsed_time: 87.894195969 secs.nsecs subreqs per batch batches %% cum %% 1: 2 5 5 2: 1 2 8 4: 2 5 13 8: 1 2 16 16: 29 78 94 32: 2 5 100 mdc.lustre-MDT0001-mdc-ffff9a086e1cc000.batch_stats= snapshot_time: 1789589434.062212239 secs.nsecs start_time: 1789589346.168020409 secs.nsecs elapsed_time: 87.894191830 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 1 2 2 8: 4 10 12 16: 9 23 35 32: 25 64 100 - open/close 856 (time 1789589445.93 total 10.02 last 85.42) - open/close 1666 (time 1789589455.94 total 20.03 last 80.89) - open/close 2412 (time 1789589465.95 total 30.04 last 74.54) - open/close 3270 (time 1789589475.95 total 40.05 last 85.77) - open/close 4086 (time 1789589485.96 total 50.05 last 81.54) - open/close 4930 (time 1789589495.97 total 60.06 last 84.34) - open/close 5927 (time 1789589505.98 total 70.07 last 99.58) - open/close 6800 (time 1789589515.98 total 80.07 last 87.29) - open/close 7486 (time 1789589525.98 total 90.07 last 68.59) - open/close 8074 (time 1789589536.00 total 100.09 last 58.68) - open/close 8694 (time 1789589546.03 total 110.12 last 61.84) - open/close 9264 (time 1789589556.03 total 120.13 last 56.96) total: 10000 open/close in 128.62 seconds: 77.75 ops/second llite.lustre-ffff9a086e1cc000.statahead_batch_max=0 llite.lustre-ffff9a086e1cc000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9a086e1cc000.stats=clear mdc.lustre-MDT0001-mdc-ffff9a086e1cc000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m9.670s user 0m0.289s sys 0m14.219s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff9a086e1cc000.statahead_batch_max=32 llite.lustre-ffff9a086e1cc000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9a086e1cc000.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff9a086e1cc000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff9a086e1cc000.stats=clear mdc.lustre-MDT0001-mdc-ffff9a086e1cc000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m36.876s user 0m0.264s sys 0m13.007s batched statahead RPC= hit count=9978 mdc.lustre-MDT0000-mdc-ffff9a086e1cc000.batch_stats= snapshot_time: 1789589751.663507967 secs.nsecs start_time: 1789589644.444916461 secs.nsecs elapsed_time: 107.218591506 secs.nsecs subreqs per batch batches %% cum %% 1: 2 0 0 2: 0 0 0 4: 0 0 0 8: 1 0 0 16: 0 0 0 32: 313 99 100 mdc.lustre-MDT0001-mdc-ffff9a086e1cc000.batch_stats= snapshot_time: 1789589751.663611993 secs.nsecs start_time: 1789589644.445015327 secs.nsecs elapsed_time: 107.218596666 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 batch: statahead_max=2048 statahead_batch_max=256 mdc.lustre-MDT0000-mdc-ffff9a086e1cc000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9a086e1cc000.batch_stats=0 llite.lustre-ffff9a086e1cc000.statahead_max=2048 llite.lustre-ffff9a086e1cc000.statahead_batch_max=256 kvm mdc.lustre-MDT0000-mdc-ffff9a086e1cc000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9a086e1cc000.batch_stats=0 striped dir -i1 -c2 -H all_char /mnt/lustre/d123ad.sanity total: 100 open/close in 3.29 seconds: 30.42 ops/second llite.lustre-ffff9a086e1cc000.statahead_max=0 101 real 0m2.301s user 0m0.004s sys 0m0.298s 'ls -l' 100 files without statahead: 2 sec llite.lustre-ffff9a086e1cc000.statahead_max=2048 llite.lustre-ffff9a086e1cc000.statahead_max=2048 101 real 0m1.094s user 0m0.005s sys 0m0.184s 'ls -l' 100 files with statahead: 1 sec statahead total: 2 statahead wrong: 0 agl total: 2 list_total: 2 fname_total: 0 hit_total: 10077 miss_total: 23 swrong=0 ewrong=0 - open/close 740 (time 1789590042.12 total 10.00 last 73.98) total: 900 open/close in 12.00 seconds: 74.98 ops/second llite.lustre-ffff9a086e1cc000.statahead_max=0 1001 real 0m20.121s user 0m0.031s sys 0m2.746s 'ls -l' 1000 files without statahead: 20 sec llite.lustre-ffff9a086e1cc000.statahead_max=2048 llite.lustre-ffff9a086e1cc000.statahead_max=2048 1001 real 0m7.544s user 0m0.038s sys 0m1.356s 'ls -l' 1000 files with statahead: 7 sec statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11076 miss_total: 24 swrong=0 ewrong=0 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 20 seconds llite.lustre-ffff9a086e1cc000.statahead_stats= statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11076 miss_total: 24 mdc.lustre-MDT0000-mdc-ffff9a086e1cc000.batch_stats= snapshot_time: 1789590103.936509675 secs.nsecs start_time: 1789589753.692048457 secs.nsecs elapsed_time: 350.244461218 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 3 42 42 8: 1 14 57 16: 0 0 57 32: 0 0 57 64: 1 14 71 128: 0 0 71 256: 2 28 100 mdc.lustre-MDT0001-mdc-ffff9a086e1cc000.batch_stats= snapshot_time: 1789590103.936581570 secs.nsecs start_time: 1789589753.695452256 secs.nsecs elapsed_time: 350.241129314 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 3 42 42 8: 1 14 57 16: 0 0 57 32: 0 0 57 64: 1 14 71 128: 0 0 71 256: 2 28 100 - open/close 772 (time 1789590115.88 total 10.02 last 77.03) - open/close 1556 (time 1789590125.88 total 20.02 last 78.40) - open/close 2436 (time 1789590135.88 total 30.02 last 88.00) - open/close 3263 (time 1789590145.89 total 40.03 last 82.67) - open/close 4315 (time 1789590155.89 total 50.03 last 105.13) - open/close 5408 (time 1789590165.89 total 60.03 last 109.30) - open/close 6389 (time 1789590175.90 total 70.04 last 98.05) - open/close 7388 (time 1789590185.91 total 80.05 last 99.81) - open/close 8301 (time 1789590195.92 total 90.06 last 91.22) - open/close 9174 (time 1789590205.92 total 100.06 last 87.25) - open/close 9902 (time 1789590215.94 total 110.08 last 72.68) total: 10000 open/close in 111.77 seconds: 89.47 ops/second llite.lustre-ffff9a086e1cc000.statahead_batch_max=0 llite.lustre-ffff9a086e1cc000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9a086e1cc000.stats=clear mdc.lustre-MDT0001-mdc-ffff9a086e1cc000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m4.712s user 0m0.304s sys 0m14.394s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff9a086e1cc000.statahead_batch_max=256 llite.lustre-ffff9a086e1cc000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9a086e1cc000.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff9a086e1cc000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff9a086e1cc000.stats=clear mdc.lustre-MDT0001-mdc-ffff9a086e1cc000.stats=clear ls -l files in d123ad.sanity: 10001 real 0m45.386s user 0m0.278s sys 0m13.582s batched statahead RPC= hit count=9999 mdc.lustre-MDT0000-mdc-ffff9a086e1cc000.batch_stats= snapshot_time: 1789590345.717650009 secs.nsecs start_time: 1789590290.758495160 secs.nsecs elapsed_time: 54.959154849 secs.nsecs subreqs per batch batches %% cum %% 1: 122 46 46 2: 31 11 58 4: 25 9 67 8: 14 5 73 16: 11 4 77 32: 19 7 84 64: 5 1 86 128: 1 0 87 256: 34 12 100 mdc.lustre-MDT0001-mdc-ffff9a086e1cc000.batch_stats= snapshot_time: 1789590345.717712747 secs.nsecs start_time: 1789590290.758818892 secs.nsecs elapsed_time: 54.958893855 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second seq.cli-lustre-OST0000-super.width=65536 seq.cli-lustre-OST0001-super.width=65536 llite.lustre-ffff9a086e1cc000.statahead_batch_max=64 llite.lustre-ffff9a086e1cc000.statahead_max=128