== sanity test 123ad: Verify batching statahead works correctly ========================================================== 03:44:02 (1787384642) batch: statahead_max=32 statahead_batch_max=32 mdc.lustre-MDT0000-mdc-ffff8c7e10d5f000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff8c7e10d5f000.batch_stats=0 llite.lustre-ffff8c7e10d5f000.statahead_max=32 llite.lustre-ffff8c7e10d5f000.statahead_batch_max=32 seq.cli-lustre-OST0000-super.width=0x1ffffff seq.cli-lustre-OST0001-super.width=0x1ffffff kvm mdc.lustre-MDT0000-mdc-ffff8c7e10d5f000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff8c7e10d5f000.batch_stats=0 striped dir -i1 -c2 -H all_char /mnt/lustre/d123ad.sanity total: 100 open/close in 3.24 seconds: 30.88 ops/second llite.lustre-ffff8c7e10d5f000.statahead_max=0 101 real 0m3.126s user 0m0.006s sys 0m0.354s 'ls -l' 100 files without statahead: 3 sec llite.lustre-ffff8c7e10d5f000.statahead_max=32 llite.lustre-ffff8c7e10d5f000.statahead_max=32 101 real 0m1.501s user 0m0.002s sys 0m0.209s 'ls -l' 100 files with statahead: 2 sec statahead total: 146 statahead wrong: 2 agl total: 140 list_total: 30 fname_total: 116 hit_total: 4113 miss_total: 1601 swrong=2 ewrong=2 - open/close 597 (time 1787384674.15 total 10.02 last 59.58) total: 900 open/close in 15.43 seconds: 58.34 ops/second llite.lustre-ffff8c7e10d5f000.statahead_max=0 1001 real 0m26.763s user 0m0.047s sys 0m3.652s 'ls -l' 1000 files without statahead: 27 sec llite.lustre-ffff8c7e10d5f000.statahead_max=32 llite.lustre-ffff8c7e10d5f000.statahead_max=32 1001 real 0m10.988s user 0m0.037s sys 0m1.699s 'ls -l' 1000 files with statahead: 11 sec statahead total: 147 statahead wrong: 2 agl total: 141 list_total: 31 fname_total: 116 hit_total: 5111 miss_total: 1603 swrong=2 ewrong=2 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 23 seconds llite.lustre-ffff8c7e10d5f000.statahead_stats= statahead total: 147 statahead wrong: 2 agl total: 141 list_total: 31 fname_total: 116 hit_total: 5111 miss_total: 1603 mdc.lustre-MDT0000-mdc-ffff8c7e10d5f000.batch_stats= snapshot_time: 1787384755.137348802 secs.nsecs start_time: 1787384646.873692989 secs.nsecs elapsed_time: 108.263655813 secs.nsecs subreqs per batch batches %% cum %% 1: 1 2 2 2: 0 0 2 4: 1 2 5 8: 2 5 10 16: 20 52 63 32: 14 36 100 mdc.lustre-MDT0001-mdc-ffff8c7e10d5f000.batch_stats= snapshot_time: 1787384755.138172005 secs.nsecs start_time: 1787384646.873812762 secs.nsecs elapsed_time: 108.264359243 secs.nsecs subreqs per batch batches %% cum %% 1: 1 2 2 2: 0 0 2 4: 3 7 10 8: 2 5 15 16: 19 48 64 32: 14 35 100 - open/close 674 (time 1787384768.45 total 10.01 last 67.37) - open/close 1280 (time 1787384778.46 total 20.02 last 60.53) - open/close 1958 (time 1787384788.47 total 30.02 last 67.77) - open/close 2678 (time 1787384798.48 total 40.03 last 71.94) - open/close 3368 (time 1787384808.50 total 50.06 last 68.81) - open/close 3967 (time 1787384818.51 total 60.07 last 59.84) - open/close 4540 (time 1787384828.54 total 70.09 last 57.16) - open/close 5242 (time 1787384838.55 total 80.10 last 70.12) - open/close 5925 (time 1787384848.57 total 90.12 last 68.15) - open/close 6723 (time 1787384858.57 total 100.13 last 79.77) - open/close 7431 (time 1787384868.60 total 110.15 last 70.66) - open/close 8095 (time 1787384878.60 total 120.15 last 66.37) - open/close 8774 (time 1787384888.60 total 130.16 last 67.88) - open/close 9458 (time 1787384898.61 total 140.16 last 68.35) total: 10000 open/close in 148.00 seconds: 67.57 ops/second llite.lustre-ffff8c7e10d5f000.statahead_batch_max=0 llite.lustre-ffff8c7e10d5f000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff8c7e10d5f000.stats=clear mdc.lustre-MDT0001-mdc-ffff8c7e10d5f000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m19.693s user 0m0.345s sys 0m18.101s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff8c7e10d5f000.statahead_batch_max=32 llite.lustre-ffff8c7e10d5f000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff8c7e10d5f000.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff8c7e10d5f000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff8c7e10d5f000.stats=clear mdc.lustre-MDT0001-mdc-ffff8c7e10d5f000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m57.947s user 0m0.389s sys 0m17.897s batched statahead RPC= hit count=9987 mdc.lustre-MDT0000-mdc-ffff8c7e10d5f000.batch_stats= snapshot_time: 1787385132.898150541 secs.nsecs start_time: 1787384998.107847334 secs.nsecs elapsed_time: 134.790303207 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 0 0 0 8: 1 0 0 16: 1 0 0 32: 313 99 100 mdc.lustre-MDT0001-mdc-ffff8c7e10d5f000.batch_stats= snapshot_time: 1787385132.898232686 secs.nsecs start_time: 1787384998.107995002 secs.nsecs elapsed_time: 134.790237684 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 batch: statahead_max=2048 statahead_batch_max=256 mdc.lustre-MDT0000-mdc-ffff8c7e10d5f000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff8c7e10d5f000.batch_stats=0 llite.lustre-ffff8c7e10d5f000.statahead_max=2048 llite.lustre-ffff8c7e10d5f000.statahead_batch_max=256 kvm mdc.lustre-MDT0000-mdc-ffff8c7e10d5f000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff8c7e10d5f000.batch_stats=0 striped dir -i1 -c2 -H crush /mnt/lustre/d123ad.sanity total: 100 open/close in 3.00 seconds: 33.35 ops/second llite.lustre-ffff8c7e10d5f000.statahead_max=0 101 real 0m2.911s user 0m0.010s sys 0m0.407s 'ls -l' 100 files without statahead: 3 sec llite.lustre-ffff8c7e10d5f000.statahead_max=2048 llite.lustre-ffff8c7e10d5f000.statahead_max=2048 101 real 0m1.038s user 0m0.004s sys 0m0.189s 'ls -l' 100 files with statahead: 1 sec statahead total: 2 statahead wrong: 0 agl total: 2 list_total: 2 fname_total: 0 hit_total: 10086 miss_total: 14 swrong=0 ewrong=0 - open/close 642 (time 1787385472.58 total 10.00 last 64.20) total: 900 open/close in 14.00 seconds: 64.31 ops/second llite.lustre-ffff8c7e10d5f000.statahead_max=0 1001 real 0m24.369s user 0m0.037s sys 0m3.666s 'ls -l' 1000 files without statahead: 24 sec llite.lustre-ffff8c7e10d5f000.statahead_max=2048 llite.lustre-ffff8c7e10d5f000.statahead_max=2048 1001 real 0m5.749s user 0m0.028s sys 0m1.561s 'ls -l' 1000 files with statahead: 6 sec statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11085 miss_total: 15 swrong=0 ewrong=0 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 20 seconds llite.lustre-ffff8c7e10d5f000.statahead_stats= statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11085 miss_total: 15 mdc.lustre-MDT0000-mdc-ffff8c7e10d5f000.batch_stats= snapshot_time: 1787385537.497337947 secs.nsecs start_time: 1787385135.133252211 secs.nsecs elapsed_time: 402.364085736 secs.nsecs subreqs per batch batches %% cum %% 1: 1 14 14 2: 0 0 14 4: 2 28 42 8: 2 28 71 16: 0 0 71 32: 0 0 71 64: 0 0 71 128: 0 0 71 256: 2 28 100 mdc.lustre-MDT0001-mdc-ffff8c7e10d5f000.batch_stats= snapshot_time: 1787385537.497874963 secs.nsecs start_time: 1787385135.133721341 secs.nsecs elapsed_time: 402.364153622 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 1 9 9 4: 0 0 9 8: 3 27 36 16: 1 9 45 32: 4 36 81 64: 0 0 81 128: 0 0 81 256: 2 18 100 - open/close 666 (time 1787385550.53 total 10.00 last 66.59) - open/close 1337 (time 1787385560.54 total 20.01 last 67.06) - open/close 2060 (time 1787385570.54 total 30.01 last 72.28) - open/close 2773 (time 1787385580.55 total 40.02 last 71.23) - open/close 3443 (time 1787385590.55 total 50.02 last 66.99) - open/close 4088 (time 1787385600.56 total 60.03 last 64.44) - open/close 4660 (time 1787385610.59 total 70.06 last 57.06) - open/close 5263 (time 1787385620.59 total 80.06 last 60.29) - open/close 5815 (time 1787385630.59 total 90.07 last 55.16) - open/close 6413 (time 1787385640.61 total 100.08 last 59.72) - open/close 7029 (time 1787385650.62 total 110.09 last 61.54) - open/close 7614 (time 1787385660.62 total 120.09 last 58.48) - open/close 8281 (time 1787385670.63 total 130.10 last 66.65) - open/close 8928 (time 1787385680.66 total 140.13 last 64.53) - open/close 9606 (time 1787385690.66 total 150.13 last 67.79) total: 10000 open/close in 156.43 seconds: 63.92 ops/second llite.lustre-ffff8c7e10d5f000.statahead_batch_max=0 llite.lustre-ffff8c7e10d5f000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff8c7e10d5f000.stats=clear mdc.lustre-MDT0001-mdc-ffff8c7e10d5f000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m0.080s user 0m0.338s sys 0m19.678s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff8c7e10d5f000.statahead_batch_max=256 llite.lustre-ffff8c7e10d5f000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff8c7e10d5f000.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff8c7e10d5f000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff8c7e10d5f000.stats=clear mdc.lustre-MDT0001-mdc-ffff8c7e10d5f000.stats=clear ls -l files in d123ad.sanity: 10001 real 0m52.491s user 0m0.317s sys 0m19.158s batched statahead RPC= hit count=9999 mdc.lustre-MDT0000-mdc-ffff8c7e10d5f000.batch_stats= snapshot_time: 1787385832.264002767 secs.nsecs start_time: 1787385767.596845610 secs.nsecs elapsed_time: 64.667157157 secs.nsecs subreqs per batch batches %% cum %% 1: 298 59 59 2: 62 12 72 4: 43 8 80 8: 26 5 86 16: 18 3 89 32: 11 2 91 64: 5 1 92 128: 2 0 93 256: 33 6 100 mdc.lustre-MDT0001-mdc-ffff8c7e10d5f000.batch_stats= snapshot_time: 1787385832.264067478 secs.nsecs start_time: 1787385767.597262340 secs.nsecs elapsed_time: 64.666805138 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second seq.cli-lustre-OST0000-super.width=65536 seq.cli-lustre-OST0001-super.width=65536 llite.lustre-ffff8c7e10d5f000.statahead_batch_max=64 llite.lustre-ffff8c7e10d5f000.statahead_max=128