== sanity test 123ad: Verify batching statahead works correctly ========================================================== 19:42:29 (1788565349) batch: statahead_max=32 statahead_batch_max=32 mdc.lustre-MDT0000-mdc-ffffa0d683ee7000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffffa0d683ee7000.batch_stats=0 llite.lustre-ffffa0d683ee7000.statahead_max=32 llite.lustre-ffffa0d683ee7000.statahead_batch_max=32 seq.cli-lustre-OST0000-super.width=0x1ffffff seq.cli-lustre-OST0001-super.width=0x1ffffff kvm mdc.lustre-MDT0000-mdc-ffffa0d683ee7000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffffa0d683ee7000.batch_stats=0 striped dir -i1 -c2 -H crush2 /mnt/lustre/d123ad.sanity total: 100 open/close in 2.40 seconds: 41.72 ops/second llite.lustre-ffffa0d683ee7000.statahead_max=0 101 real 0m2.544s user 0m0.006s sys 0m0.315s 'ls -l' 100 files without statahead: 3 sec llite.lustre-ffffa0d683ee7000.statahead_max=32 llite.lustre-ffffa0d683ee7000.statahead_max=32 101 real 0m1.078s user 0m0.002s sys 0m0.188s 'ls -l' 100 files with statahead: 1 sec statahead total: 103 statahead wrong: 2 agl total: 97 list_total: 32 fname_total: 70 hit_total: 4399 miss_total: 1273 swrong=2 ewrong=2 total: 900 open/close in 8.65 seconds: 104.06 ops/second llite.lustre-ffffa0d683ee7000.statahead_max=0 1001 real 0m16.919s user 0m0.031s sys 0m2.743s 'ls -l' 1000 files without statahead: 17 sec llite.lustre-ffffa0d683ee7000.statahead_max=32 llite.lustre-ffffa0d683ee7000.statahead_max=32 1001 real 0m9.003s user 0m0.035s sys 0m1.378s 'ls -l' 1000 files with statahead: 9 sec statahead total: 104 statahead wrong: 2 agl total: 98 list_total: 33 fname_total: 70 hit_total: 5397 miss_total: 1275 swrong=2 ewrong=2 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 16 seconds llite.lustre-ffffa0d683ee7000.statahead_stats= statahead total: 104 statahead wrong: 2 agl total: 98 list_total: 33 fname_total: 70 hit_total: 5397 miss_total: 1275 mdc.lustre-MDT0000-mdc-ffffa0d683ee7000.batch_stats= snapshot_time: 1788565428.789575820 secs.nsecs start_time: 1788565353.893125582 secs.nsecs elapsed_time: 74.896450238 secs.nsecs subreqs per batch batches %% cum %% 1: 3 8 8 2: 0 0 8 4: 2 5 13 8: 2 5 18 16: 25 67 86 32: 5 13 100 mdc.lustre-MDT0001-mdc-ffffa0d683ee7000.batch_stats= snapshot_time: 1788565428.789640865 secs.nsecs start_time: 1788565353.893254300 secs.nsecs elapsed_time: 74.896386565 secs.nsecs subreqs per batch batches %% cum %% 1: 1 2 2 2: 1 2 5 4: 1 2 7 8: 2 5 12 16: 12 30 42 32: 23 57 100 - open/close 1027 (time 1788565441.31 total 10.00 last 102.69) - open/close 1840 (time 1788565451.32 total 20.01 last 81.24) - open/close 2702 (time 1788565461.32 total 30.01 last 86.16) - open/close 3658 (time 1788565471.33 total 40.02 last 95.58) - open/close 4578 (time 1788565481.33 total 50.02 last 91.96) - open/close 5594 (time 1788565491.34 total 60.03 last 101.48) - open/close 6504 (time 1788565501.34 total 70.03 last 91.00) - open/close 7402 (time 1788565511.36 total 80.05 last 89.68) - open/close 8106 (time 1788565521.37 total 90.06 last 70.32) - open/close 9015 (time 1788565531.37 total 100.06 last 90.84) - open/close 9926 (time 1788565541.38 total 110.06 last 91.09) total: 10000 open/close in 110.71 seconds: 90.32 ops/second llite.lustre-ffffa0d683ee7000.statahead_batch_max=0 llite.lustre-ffffa0d683ee7000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffffa0d683ee7000.stats=clear mdc.lustre-MDT0001-mdc-ffffa0d683ee7000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m4.902s user 0m0.296s sys 0m14.841s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffffa0d683ee7000.statahead_batch_max=32 llite.lustre-ffffa0d683ee7000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffffa0d683ee7000.batch_stats=clear mdc.lustre-MDT0001-mdc-ffffa0d683ee7000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffffa0d683ee7000.stats=clear mdc.lustre-MDT0001-mdc-ffffa0d683ee7000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m44.060s user 0m0.306s sys 0m13.477s batched statahead RPC= hit count=9972 mdc.lustre-MDT0000-mdc-ffffa0d683ee7000.batch_stats= snapshot_time: 1788565730.702158061 secs.nsecs start_time: 1788565614.738901334 secs.nsecs elapsed_time: 115.963256727 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 0 0 0 8: 2 0 0 16: 1 0 0 32: 313 99 100 mdc.lustre-MDT0001-mdc-ffffa0d683ee7000.batch_stats= snapshot_time: 1788565730.702224636 secs.nsecs start_time: 1788565614.739331369 secs.nsecs elapsed_time: 115.962893267 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 batch: statahead_max=2048 statahead_batch_max=256 mdc.lustre-MDT0000-mdc-ffffa0d683ee7000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffffa0d683ee7000.batch_stats=0 llite.lustre-ffffa0d683ee7000.statahead_max=2048 llite.lustre-ffffa0d683ee7000.statahead_batch_max=256 kvm mdc.lustre-MDT0000-mdc-ffffa0d683ee7000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffffa0d683ee7000.batch_stats=0 striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d123ad.sanity total: 100 open/close in 2.69 seconds: 37.14 ops/second llite.lustre-ffffa0d683ee7000.statahead_max=0 101 real 0m2.026s user 0m0.010s sys 0m0.297s 'ls -l' 100 files without statahead: 2 sec llite.lustre-ffffa0d683ee7000.statahead_max=2048 llite.lustre-ffffa0d683ee7000.statahead_max=2048 101 real 0m1.020s user 0m0.004s sys 0m0.185s 'ls -l' 100 files with statahead: 1 sec statahead total: 2 statahead wrong: 0 agl total: 2 list_total: 2 fname_total: 0 hit_total: 10071 miss_total: 29 swrong=0 ewrong=0 - open/close 659 (time 1788566037.85 total 10.01 last 65.80) total: 900 open/close in 15.02 seconds: 59.91 ops/second llite.lustre-ffffa0d683ee7000.statahead_max=0 1001 real 0m25.170s user 0m0.032s sys 0m2.853s 'ls -l' 1000 files without statahead: 25 sec llite.lustre-ffffa0d683ee7000.statahead_max=2048 llite.lustre-ffffa0d683ee7000.statahead_max=2048 1001 real 0m6.044s user 0m0.021s sys 0m1.243s 'ls -l' 1000 files with statahead: 6 sec statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11070 miss_total: 30 swrong=0 ewrong=0 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 20 seconds llite.lustre-ffffa0d683ee7000.statahead_stats= statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11070 miss_total: 30 mdc.lustre-MDT0000-mdc-ffffa0d683ee7000.batch_stats= snapshot_time: 1788566105.986492085 secs.nsecs start_time: 1788565732.793462520 secs.nsecs elapsed_time: 373.193029565 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 1 14 14 4: 1 14 28 8: 2 28 57 16: 0 0 57 32: 0 0 57 64: 1 14 71 128: 0 0 71 256: 2 28 100 mdc.lustre-MDT0001-mdc-ffffa0d683ee7000.batch_stats= snapshot_time: 1788566105.986564379 secs.nsecs start_time: 1788565732.793692447 secs.nsecs elapsed_time: 373.192871932 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 2 28 28 8: 2 28 57 16: 0 0 57 32: 0 0 57 64: 1 14 71 128: 0 0 71 256: 2 28 100 - open/close 1042 (time 1788566119.16 total 10.00 last 104.17) - open/close 1908 (time 1788566129.16 total 20.01 last 86.55) - open/close 2870 (time 1788566139.16 total 30.01 last 96.19) - open/close 3815 (time 1788566149.17 total 40.01 last 94.46) - open/close 4691 (time 1788566159.17 total 50.02 last 87.58) - open/close 5568 (time 1788566169.17 total 60.02 last 87.66) - open/close 6476 (time 1788566179.18 total 70.03 last 90.75) - open/close 7297 (time 1788566189.19 total 80.03 last 82.05) - open/close 8276 (time 1788566199.19 total 90.04 last 97.86) - open/close 9110 (time 1788566209.21 total 100.05 last 83.26) total: 10000 open/close in 110.03 seconds: 90.89 ops/second llite.lustre-ffffa0d683ee7000.statahead_batch_max=0 llite.lustre-ffffa0d683ee7000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffffa0d683ee7000.stats=clear mdc.lustre-MDT0001-mdc-ffffa0d683ee7000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m4.870s user 0m0.341s sys 0m14.563s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffffa0d683ee7000.statahead_batch_max=256 llite.lustre-ffffa0d683ee7000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffffa0d683ee7000.batch_stats=clear mdc.lustre-MDT0001-mdc-ffffa0d683ee7000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffffa0d683ee7000.stats=clear mdc.lustre-MDT0001-mdc-ffffa0d683ee7000.stats=clear ls -l files in d123ad.sanity: 10001 real 0m52.242s user 0m0.326s sys 0m14.614s batched statahead RPC= hit count=9999 mdc.lustre-MDT0000-mdc-ffffa0d683ee7000.batch_stats= snapshot_time: 1788566355.877225069 secs.nsecs start_time: 1788566293.386663567 secs.nsecs elapsed_time: 62.490561502 secs.nsecs subreqs per batch batches %% cum %% 1: 113 49 49 2: 28 12 62 4: 12 5 67 8: 10 4 71 16: 6 2 74 32: 11 4 79 64: 9 3 83 128: 3 1 84 256: 35 15 100 mdc.lustre-MDT0001-mdc-ffffa0d683ee7000.batch_stats= snapshot_time: 1788566355.877299706 secs.nsecs start_time: 1788566293.386853907 secs.nsecs elapsed_time: 62.490445799 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second seq.cli-lustre-OST0000-super.width=65536 seq.cli-lustre-OST0001-super.width=65536 llite.lustre-ffffa0d683ee7000.statahead_batch_max=64 llite.lustre-ffffa0d683ee7000.statahead_max=128