== sanity test 123ad: Verify batching statahead works correctly ========================================================== 17:19:24 (1777583964) batch: statahead_max=32 statahead_batch_max=32 mdc.lustre-MDT0000-mdc-ffff99da8e9c9000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff99da8e9c9000.batch_stats=0 llite.lustre-ffff99da8e9c9000.statahead_max=32 llite.lustre-ffff99da8e9c9000.statahead_batch_max=32 seq.cli-lustre-OST0000-super.width=0x1ffffff seq.cli-lustre-OST0001-super.width=0x1ffffff kvm mdc.lustre-MDT0000-mdc-ffff99da8e9c9000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff99da8e9c9000.batch_stats=0 striped dir -i1 -c2 -H crush2 /mnt/lustre/d123ad.sanity total: 100 open/close in 2.70 seconds: 37.04 ops/second llite.lustre-ffff99da8e9c9000.statahead_max=0 101 real 0m2.474s user 0m0.006s sys 0m0.332s 'ls -l' 100 files without statahead: 2 sec llite.lustre-ffff99da8e9c9000.statahead_max=32 llite.lustre-ffff99da8e9c9000.statahead_max=32 101 real 0m0.956s user 0m0.003s sys 0m0.149s 'ls -l' 100 files with statahead: 1 sec statahead total: 111 statahead wrong: 2 agl total: 106 list_total: 31 fname_total: 79 hit_total: 4449 miss_total: 1291 swrong=2 ewrong=2 - open/close 827 (time 1777583991.82 total 10.01 last 82.62) total: 900 open/close in 11.40 seconds: 78.95 ops/second llite.lustre-ffff99da8e9c9000.statahead_max=0 1001 real 0m23.120s user 0m0.024s sys 0m2.967s 'ls -l' 1000 files without statahead: 23 sec llite.lustre-ffff99da8e9c9000.statahead_max=32 llite.lustre-ffff99da8e9c9000.statahead_max=32 1001 real 0m10.254s user 0m0.028s sys 0m1.532s 'ls -l' 1000 files with statahead: 10 sec statahead total: 112 statahead wrong: 2 agl total: 107 list_total: 32 fname_total: 79 hit_total: 5446 miss_total: 1294 swrong=2 ewrong=2 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 20 seconds llite.lustre-ffff99da8e9c9000.statahead_stats= statahead total: 112 statahead wrong: 2 agl total: 107 list_total: 32 fname_total: 79 hit_total: 5446 miss_total: 1294 mdc.lustre-MDT0000-mdc-ffff99da8e9c9000.batch_stats= snapshot_time: 1777584059.618818174 secs.nsecs start_time: 1777583968.994625520 secs.nsecs elapsed_time: 90.624192654 secs.nsecs subreqs per batch batches % cum % 1: 2 5 5 2: 0 0 5 4: 4 10 16 8: 2 5 21 16: 24 64 86 32: 5 13 100 mdc.lustre-MDT0001-mdc-ffff99da8e9c9000.batch_stats= snapshot_time: 1777584059.618900980 secs.nsecs start_time: 1777583968.994831009 secs.nsecs elapsed_time: 90.624069971 secs.nsecs subreqs per batch batches % cum % 1: 1 2 2 2: 0 0 2 4: 2 5 7 8: 2 5 12 16: 9 23 35 32: 25 64 100 - open/close 835 (time 1777584071.95 total 10.00 last 83.48) - open/close 1653 (time 1777584081.96 total 20.01 last 81.76) - open/close 2482 (time 1777584091.96 total 30.01 last 82.86) - open/close 3311 (time 1777584101.97 total 40.02 last 82.84) - open/close 4164 (time 1777584111.98 total 50.03 last 85.25) - open/close 4962 (time 1777584121.99 total 60.03 last 79.72) - open/close 5798 (time 1777584131.99 total 70.04 last 83.58) - open/close 6675 (time 1777584142.00 total 80.04 last 87.63) - open/close 7646 (time 1777584152.01 total 90.06 last 96.98) - open/close 8510 (time 1777584162.01 total 100.06 last 86.40) - open/close 9413 (time 1777584172.02 total 110.06 last 90.23) total: 10000 open/close in 116.75 seconds: 85.65 ops/second llite.lustre-ffff99da8e9c9000.statahead_batch_max=0 llite.lustre-ffff99da8e9c9000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff99da8e9c9000.stats=clear mdc.lustre-MDT0001-mdc-ffff99da8e9c9000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m4.411s user 0m0.313s sys 0m16.678s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff99da8e9c9000.statahead_batch_max=32 llite.lustre-ffff99da8e9c9000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff99da8e9c9000.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff99da8e9c9000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff99da8e9c9000.stats=clear mdc.lustre-MDT0001-mdc-ffff99da8e9c9000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m45.915s user 0m0.300s sys 0m14.976s batched statahead RPC= hit count=9985 mdc.lustre-MDT0000-mdc-ffff99da8e9c9000.batch_stats= snapshot_time: 1777584371.771648150 secs.nsecs start_time: 1777584251.689176331 secs.nsecs elapsed_time: 120.082471819 secs.nsecs subreqs per batch batches % cum % 1: 0 0 0 2: 0 0 0 4: 0 0 0 8: 1 0 0 16: 1 0 0 32: 313 99 100 mdc.lustre-MDT0001-mdc-ffff99da8e9c9000.batch_stats= snapshot_time: 1777584371.773076514 secs.nsecs start_time: 1777584251.689301235 secs.nsecs elapsed_time: 120.083775279 secs.nsecs subreqs per batch batches % cum % 1: 0 0 0 batch: statahead_max=2048 statahead_batch_max=256 mdc.lustre-MDT0000-mdc-ffff99da8e9c9000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff99da8e9c9000.batch_stats=0 llite.lustre-ffff99da8e9c9000.statahead_max=2048 llite.lustre-ffff99da8e9c9000.statahead_batch_max=256 kvm mdc.lustre-MDT0000-mdc-ffff99da8e9c9000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff99da8e9c9000.batch_stats=0 striped dir -i1 -c2 -H all_char /mnt/lustre/d123ad.sanity total: 100 open/close in 2.13 seconds: 46.92 ops/second llite.lustre-ffff99da8e9c9000.statahead_max=0 101 real 0m2.362s user 0m0.004s sys 0m0.328s 'ls -l' 100 files without statahead: 2 sec llite.lustre-ffff99da8e9c9000.statahead_max=2048 llite.lustre-ffff99da8e9c9000.statahead_max=2048 101 real 0m0.813s user 0m0.002s sys 0m0.175s 'ls -l' 100 files with statahead: 0 sec statahead total: 2 statahead wrong: 0 agl total: 2 list_total: 2 fname_total: 0 hit_total: 10084 miss_total: 16 swrong=0 ewrong=0 - open/close 889 (time 1777584660.55 total 10.02 last 88.75) total: 900 open/close in 10.19 seconds: 88.28 ops/second llite.lustre-ffff99da8e9c9000.statahead_max=0 1001 real 0m20.416s user 0m0.034s sys 0m2.988s 'ls -l' 1000 files without statahead: 20 sec llite.lustre-ffff99da8e9c9000.statahead_max=2048 llite.lustre-ffff99da8e9c9000.statahead_max=2048 1001 real 0m6.691s user 0m0.035s sys 0m1.581s 'ls -l' 1000 files with statahead: 6 sec statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11083 miss_total: 17 swrong=0 ewrong=0 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 18 seconds llite.lustre-ffff99da8e9c9000.statahead_stats= statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11083 miss_total: 17 mdc.lustre-MDT0000-mdc-ffff99da8e9c9000.batch_stats= snapshot_time: 1777584716.501382091 secs.nsecs start_time: 1777584374.137934313 secs.nsecs elapsed_time: 342.363447778 secs.nsecs subreqs per batch batches % cum % 1: 0 0 0 2: 0 0 0 4: 2 22 22 8: 2 22 44 16: 1 11 55 32: 2 22 77 64: 0 0 77 128: 0 0 77 256: 2 22 100 mdc.lustre-MDT0001-mdc-ffff99da8e9c9000.batch_stats= snapshot_time: 1777584716.502229716 secs.nsecs start_time: 1777584374.138065804 secs.nsecs elapsed_time: 342.364163912 secs.nsecs subreqs per batch batches % cum % 1: 0 0 0 2: 1 11 11 4: 2 22 33 8: 1 11 44 16: 0 0 44 32: 3 33 77 64: 0 0 77 128: 0 0 77 256: 2 22 100 - open/close 890 (time 1777584729.03 total 10.01 last 88.95) - open/close 1782 (time 1777584739.03 total 20.01 last 89.16) - open/close 2635 (time 1777584749.05 total 30.03 last 85.15) - open/close 3535 (time 1777584759.05 total 40.03 last 90.00) - open/close 4387 (time 1777584769.07 total 50.05 last 85.05) - open/close 5146 (time 1777584779.08 total 60.06 last 75.81) - open/close 5937 (time 1777584789.09 total 70.06 last 79.06) - open/close 6778 (time 1777584799.09 total 80.06 last 84.09) - open/close 7593 (time 1777584809.10 total 90.07 last 81.43) - open/close 8490 (time 1777584819.10 total 100.08 last 89.66) - open/close 9383 (time 1777584829.10 total 110.08 last 89.28) total: 10000 open/close in 118.21 seconds: 84.59 ops/second llite.lustre-ffff99da8e9c9000.statahead_batch_max=0 llite.lustre-ffff99da8e9c9000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff99da8e9c9000.stats=clear mdc.lustre-MDT0001-mdc-ffff99da8e9c9000.stats=clear ls -l files in d123ad.sanity: 10001 real 0m55.272s user 0m0.295s sys 0m16.486s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff99da8e9c9000.statahead_batch_max=256 llite.lustre-ffff99da8e9c9000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff99da8e9c9000.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff99da8e9c9000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff99da8e9c9000.stats=clear mdc.lustre-MDT0001-mdc-ffff99da8e9c9000.stats=clear ls -l files in d123ad.sanity: 10001 real 0m54.294s user 0m0.270s sys 0m15.827s batched statahead RPC= hit count=9999 mdc.lustre-MDT0000-mdc-ffff99da8e9c9000.batch_stats= snapshot_time: 1777584969.471095460 secs.nsecs start_time: 1777584902.986958091 secs.nsecs elapsed_time: 66.484137369 secs.nsecs subreqs per batch batches % cum % 1: 147 50 50 2: 37 12 63 4: 22 7 71 8: 15 5 76 16: 14 4 81 32: 12 4 85 64: 7 2 87 128: 1 0 87 256: 35 12 100 mdc.lustre-MDT0001-mdc-ffff99da8e9c9000.batch_stats= snapshot_time: 1777584969.471176016 secs.nsecs start_time: 1777584902.987069076 secs.nsecs elapsed_time: 66.484106940 secs.nsecs subreqs per batch batches % cum % 1: 0 0 0 unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second seq.cli-lustre-OST0000-super.width=65536 seq.cli-lustre-OST0001-super.width=65536 llite.lustre-ffff99da8e9c9000.statahead_batch_max=64 llite.lustre-ffff99da8e9c9000.statahead_max=128