== sanity test 123ad: Verify batching statahead works correctly ========================================================== 03:04:03 (1779692643) batch: statahead_max=32 statahead_batch_max=32 mdc.lustre-MDT0000-mdc-ffff98884cd12800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff98884cd12800.batch_stats=0 llite.lustre-ffff98884cd12800.statahead_max=32 llite.lustre-ffff98884cd12800.statahead_batch_max=32 seq.cli-lustre-OST0000-super.width=0x1ffffff seq.cli-lustre-OST0001-super.width=0x1ffffff kvm mdc.lustre-MDT0000-mdc-ffff98884cd12800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff98884cd12800.batch_stats=0 striped dir -i1 -c2 -H crush /mnt/lustre/d123ad.sanity total: 100 open/close in 2.42 seconds: 41.24 ops/second llite.lustre-ffff98884cd12800.statahead_max=0 101 real 0m1.992s user 0m0.009s sys 0m0.384s 'ls -l' 100 files without statahead: 2 sec llite.lustre-ffff98884cd12800.statahead_max=32 llite.lustre-ffff98884cd12800.statahead_max=32 101 real 0m1.173s user 0m0.004s sys 0m0.195s 'ls -l' 100 files with statahead: 1 sec statahead total: 119 statahead wrong: 2 agl total: 113 list_total: 30 fname_total: 88 hit_total: 3925 miss_total: 984 swrong=2 ewrong=2 - open/close 834 (time 1779692670.08 total 10.00 last 83.37) total: 900 open/close in 10.97 seconds: 82.08 ops/second llite.lustre-ffff98884cd12800.statahead_max=0 1001 real 0m20.728s user 0m0.032s sys 0m3.347s 'ls -l' 1000 files without statahead: 21 sec llite.lustre-ffff98884cd12800.statahead_max=32 llite.lustre-ffff98884cd12800.statahead_max=32 1001 real 0m8.214s user 0m0.028s sys 0m1.444s 'ls -l' 1000 files with statahead: 8 sec statahead total: 120 statahead wrong: 2 agl total: 114 list_total: 31 fname_total: 88 hit_total: 4924 miss_total: 985 swrong=2 ewrong=2 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 22 seconds llite.lustre-ffff98884cd12800.statahead_stats= statahead total: 120 statahead wrong: 2 agl total: 114 list_total: 31 fname_total: 88 hit_total: 4924 miss_total: 985 mdc.lustre-MDT0000-mdc-ffff98884cd12800.batch_stats= snapshot_time: 1779692733.223381057 secs.nsecs start_time: 1779692647.653286490 secs.nsecs elapsed_time: 85.570094567 secs.nsecs subreqs per batch batches % cum % 1: 1 2 2 2: 0 0 2 4: 3 8 11 8: 2 5 16 16: 25 69 86 32: 5 13 100 mdc.lustre-MDT0001-mdc-ffff98884cd12800.batch_stats= snapshot_time: 1779692733.223457451 secs.nsecs start_time: 1779692647.653389462 secs.nsecs elapsed_time: 85.570067989 secs.nsecs subreqs per batch batches % cum % 1: 0 0 0 2: 1 2 2 4: 1 2 5 8: 2 5 10 16: 10 26 36 32: 24 63 100 - open/close 800 (time 1779692746.08 total 10.01 last 79.92) - open/close 1587 (time 1779692756.09 total 20.02 last 78.61) - open/close 2440 (time 1779692766.10 total 30.03 last 85.24) - open/close 3364 (time 1779692776.10 total 40.03 last 92.37) - open/close 4078 (time 1779692786.10 total 50.03 last 71.39) - open/close 4819 (time 1779692796.11 total 60.05 last 74.01) - open/close 5632 (time 1779692806.12 total 70.05 last 81.23) - open/close 6468 (time 1779692816.13 total 80.06 last 83.57) - open/close 7370 (time 1779692826.13 total 90.06 last 90.20) - open/close 8229 (time 1779692836.13 total 100.06 last 85.86) - open/close 9008 (time 1779692846.13 total 110.07 last 77.88) - open/close 9801 (time 1779692856.14 total 120.07 last 79.29) total: 10000 open/close in 122.56 seconds: 81.59 ops/second llite.lustre-ffff98884cd12800.statahead_batch_max=0 llite.lustre-ffff98884cd12800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff98884cd12800.stats=clear mdc.lustre-MDT0001-mdc-ffff98884cd12800.stats=clear ls -l files in d123ad.sanity: 10001 real 1m9.023s user 0m0.336s sys 0m16.955s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff98884cd12800.statahead_batch_max=32 llite.lustre-ffff98884cd12800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff98884cd12800.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff98884cd12800.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff98884cd12800.stats=clear mdc.lustre-MDT0001-mdc-ffff98884cd12800.stats=clear ls -l files in d123ad.sanity: 10001 real 2m0.061s user 0m0.316s sys 0m15.134s batched statahead RPC= hit count=9977 mdc.lustre-MDT0000-mdc-ffff98884cd12800.batch_stats= snapshot_time: 1779693077.553148865 secs.nsecs start_time: 1779692936.583638618 secs.nsecs elapsed_time: 140.969510247 secs.nsecs subreqs per batch batches % cum % 1: 0 0 0 2: 0 0 0 4: 1 0 0 8: 1 0 0 16: 1 0 0 32: 313 99 100 mdc.lustre-MDT0001-mdc-ffff98884cd12800.batch_stats= snapshot_time: 1779693077.553220997 secs.nsecs start_time: 1779692936.583809600 secs.nsecs elapsed_time: 140.969411397 secs.nsecs subreqs per batch batches % cum % 1: 0 0 0 batch: statahead_max=2048 statahead_batch_max=256 mdc.lustre-MDT0000-mdc-ffff98884cd12800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff98884cd12800.batch_stats=0 llite.lustre-ffff98884cd12800.statahead_max=2048 llite.lustre-ffff98884cd12800.statahead_batch_max=256 kvm mdc.lustre-MDT0000-mdc-ffff98884cd12800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff98884cd12800.batch_stats=0 striped dir -i1 -c2 -H all_char /mnt/lustre/d123ad.sanity total: 100 open/close in 2.74 seconds: 36.55 ops/second llite.lustre-ffff98884cd12800.statahead_max=0 101 real 0m2.657s user 0m0.004s sys 0m0.340s 'ls -l' 100 files without statahead: 3 sec llite.lustre-ffff98884cd12800.statahead_max=2048 llite.lustre-ffff98884cd12800.statahead_max=2048 101 real 0m1.294s user 0m0.008s sys 0m0.179s 'ls -l' 100 files with statahead: 1 sec statahead total: 2 statahead wrong: 0 agl total: 2 list_total: 2 fname_total: 0 hit_total: 10076 miss_total: 24 swrong=0 ewrong=0 - open/close 505 (time 1779693412.48 total 10.01 last 50.46) total: 900 open/close in 16.69 seconds: 53.93 ops/second llite.lustre-ffff98884cd12800.statahead_max=0 1001 real 0m24.820s user 0m0.038s sys 0m3.256s 'ls -l' 1000 files without statahead: 25 sec llite.lustre-ffff98884cd12800.statahead_max=2048 llite.lustre-ffff98884cd12800.statahead_max=2048 1001 real 0m7.619s user 0m0.031s sys 0m1.707s 'ls -l' 1000 files with statahead: 8 sec statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11075 miss_total: 25 swrong=0 ewrong=0 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 21 seconds llite.lustre-ffff98884cd12800.statahead_stats= statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11075 miss_total: 25 mdc.lustre-MDT0000-mdc-ffff98884cd12800.batch_stats= snapshot_time: 1779693485.692300999 secs.nsecs start_time: 1779693080.880840264 secs.nsecs elapsed_time: 404.811460735 secs.nsecs subreqs per batch batches % cum % 1: 0 0 0 2: 1 14 14 4: 3 42 57 8: 0 0 57 16: 0 0 57 32: 0 0 57 64: 1 14 71 128: 0 0 71 256: 2 28 100 mdc.lustre-MDT0001-mdc-ffff98884cd12800.batch_stats= snapshot_time: 1779693485.692450794 secs.nsecs start_time: 1779693080.880980565 secs.nsecs elapsed_time: 404.811470229 secs.nsecs subreqs per batch batches % cum % 1: 0 0 0 2: 0 0 0 4: 1 14 14 8: 3 42 57 16: 0 0 57 32: 0 0 57 64: 1 14 71 128: 0 0 71 256: 2 28 100 - open/close 708 (time 1779693498.26 total 10.00 last 70.79) - open/close 1329 (time 1779693508.27 total 20.01 last 62.07) - open/close 2055 (time 1779693518.27 total 30.01 last 72.59) - open/close 2709 (time 1779693528.27 total 40.01 last 65.38) - open/close 3417 (time 1779693538.28 total 50.02 last 70.74) - open/close 4137 (time 1779693548.29 total 60.03 last 71.93) - open/close 4805 (time 1779693558.30 total 70.04 last 66.73) - open/close 5407 (time 1779693568.31 total 80.05 last 60.15) - open/close 6061 (time 1779693578.32 total 90.06 last 65.32) - open/close 6772 (time 1779693588.32 total 100.06 last 71.06) - open/close 7423 (time 1779693598.34 total 110.08 last 65.03) - open/close 8186 (time 1779693608.35 total 120.09 last 76.18) - open/close 8863 (time 1779693618.36 total 130.10 last 67.66) - open/close 9570 (time 1779693628.39 total 140.13 last 70.50) total: 10000 open/close in 145.95 seconds: 68.52 ops/second llite.lustre-ffff98884cd12800.statahead_batch_max=0 llite.lustre-ffff98884cd12800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff98884cd12800.stats=clear mdc.lustre-MDT0001-mdc-ffff98884cd12800.stats=clear ls -l files in d123ad.sanity: 10001 real 1m8.481s user 0m0.330s sys 0m16.031s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff98884cd12800.statahead_batch_max=256 llite.lustre-ffff98884cd12800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff98884cd12800.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff98884cd12800.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff98884cd12800.stats=clear mdc.lustre-MDT0001-mdc-ffff98884cd12800.stats=clear ls -l files in d123ad.sanity: 10001 real 0m50.476s user 0m0.311s sys 0m15.746s batched statahead RPC= hit count=9999 mdc.lustre-MDT0000-mdc-ffff98884cd12800.batch_stats= snapshot_time: 1779693778.171129117 secs.nsecs start_time: 1779693713.905437373 secs.nsecs elapsed_time: 64.265691744 secs.nsecs subreqs per batch batches % cum % 1: 117 38 38 2: 59 19 57 4: 41 13 71 8: 24 7 79 16: 9 2 82 32: 14 4 86 64: 4 1 88 128: 2 0 88 256: 34 11 100 mdc.lustre-MDT0001-mdc-ffff98884cd12800.batch_stats= snapshot_time: 1779693778.171198550 secs.nsecs start_time: 1779693713.905519113 secs.nsecs elapsed_time: 64.265679437 secs.nsecs subreqs per batch batches % cum % 1: 0 0 0 unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second seq.cli-lustre-OST0000-super.width=65536 seq.cli-lustre-OST0001-super.width=65536 llite.lustre-ffff98884cd12800.statahead_batch_max=64 llite.lustre-ffff98884cd12800.statahead_max=128