== sanity test 123ad: Verify batching statahead works correctly ========================================================== 16:10:01 (1746130201) batch: statahead_max=32 statahead_batch_max=32 mdc.lustre-MDT0000-mdc-ffff88007901d800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff88007901d800.batch_stats=0 llite.lustre-ffff88007901d800.statahead_max=32 llite.lustre-ffff88007901d800.statahead_batch_max=32 seq.cli-lustre-OST0000-super.width=0x1ffffff seq.cli-lustre-OST0001-super.width=0x1ffffff kvm mdc.lustre-MDT0000-mdc-ffff88007901d800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff88007901d800.batch_stats=0 striped dir -i1 -c2 -H crush /mnt/lustre/d123ad.sanity total: 100 open/close in 1.44 seconds: 69.41 ops/second llite.lustre-ffff88007901d800.statahead_max=0 101 real 0m1.298s user 0m0.012s sys 0m0.313s ls -l 100 files without statahead: 1 sec llite.lustre-ffff88007901d800.statahead_max=32 32 101 real 0m0.430s user 0m0.006s sys 0m0.136s ls -l 100 files with statahead: 1 sec statahead total: 144 statahead wrong: 2 agl total: 144 list_total: 28 fname_total: 116 hit_total: 5352 miss_total: 1175 total: 900 open/close in 5.82 seconds: 154.64 ops/second llite.lustre-ffff88007901d800.statahead_max=0 1001 real 0m11.204s user 0m0.061s sys 0m2.951s ls -l 1000 files without statahead: 11 sec llite.lustre-ffff88007901d800.statahead_max=32 32 1001 real 0m3.230s user 0m0.028s sys 0m1.142s ls -l 1000 files with statahead: 3 sec statahead total: 145 statahead wrong: 2 agl total: 145 list_total: 29 fname_total: 116 hit_total: 6351 miss_total: 1181 - open/close 1389 (time 1746130243.50 total 10.00 last 138.87) total: 3000 open/close in 19.28 seconds: 155.59 ops/second llite.lustre-ffff88007901d800.statahead_max=0 4001 real 0m49.834s user 0m0.403s sys 0m12.016s ls -l 4000 files without statahead: 50 sec llite.lustre-ffff88007901d800.statahead_max=32 32 4001 real 0m14.583s user 0m0.163s sys 0m4.878s ls -l 4000 files with statahead: 15 sec statahead total: 146 statahead wrong: 2 agl total: 146 list_total: 30 fname_total: 116 hit_total: 10350 miss_total: 1203 ls -l done rm -r /mnt/lustre/d123ad.sanity: 39 seconds llite.lustre-ffff88007901d800.statahead_stats= statahead total: 146 statahead wrong: 2 agl total: 146 list_total: 30 fname_total: 116 hit_total: 10350 miss_total: 1203 mdc.lustre-MDT0000-mdc-ffff88007901d800.batch_stats= snapshot_time: 1746130362.405031369 secs.nsecs start_time: 1746130203.990866241 secs.nsecs elapsed_time: 158.414165128 secs.nsecs subreqs per batch batches % cum % 1: 2 1 1 2: 2 1 2 4: 1 0 3 8: 4 2 5 16: 106 65 70 32: 48 29 100 mdc.lustre-MDT0001-mdc-ffff88007901d800.batch_stats= snapshot_time: 1746130362.405614222 secs.nsecs start_time: 1746130203.991074904 secs.nsecs elapsed_time: 158.414539318 secs.nsecs subreqs per batch batches % cum % 1: 0 0 0 2: 0 0 0 4: 3 1 1 8: 6 3 5 16: 69 41 46 32: 88 53 100 - open/close 1621 (time 1746130373.50 total 10.00 last 162.06) - open/close 3393 (time 1746130383.50 total 20.00 last 177.19) - open/close 4792 (time 1746130393.51 total 30.01 last 139.84) - open/close 5990 (time 1746130403.51 total 40.01 last 119.73) - open/close 7358 (time 1746130413.51 total 50.01 last 136.78) - open/close 8568 (time 1746130423.51 total 60.01 last 120.99) - open/close 9863 (time 1746130433.52 total 70.02 last 129.43) total: 10000 open/close in 71.05 seconds: 140.75 ops/second llite.lustre-ffff88007901d800.statahead_batch_max=0 llite.lustre-ffff88007901d800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff88007901d800.stats=clear mdc.lustre-MDT0001-mdc-ffff88007901d800.stats=clear ls -l files in d123ad.sanity: 10001 real 0m21.675s user 0m0.227s sys 0m9.917s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff88007901d800.statahead_batch_max=32 llite.lustre-ffff88007901d800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff88007901d800.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff88007901d800.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff88007901d800.stats=clear mdc.lustre-MDT0001-mdc-ffff88007901d800.stats=clear ls -l files in d123ad.sanity: 10001 real 0m31.509s user 0m0.243s sys 0m10.633s batched statahead RPC= hit count=9999 mdc.lustre-MDT0000-mdc-ffff88007901d800.batch_stats= snapshot_time: 1746130493.928565126 secs.nsecs start_time: 1746130459.189848539 secs.nsecs elapsed_time: 34.738716587 secs.nsecs subreqs per batch batches % cum % 1: 0 0 0 2: 0 0 0 4: 0 0 0 8: 1 0 0 16: 2 0 0 32: 312 99 100 mdc.lustre-MDT0001-mdc-ffff88007901d800.batch_stats= snapshot_time: 1746130493.928648934 secs.nsecs start_time: 1746130459.189913660 secs.nsecs elapsed_time: 34.738735274 secs.nsecs subreqs per batch batches % cum % 1: 0 0 0 batch: statahead_max=2048 statahead_batch_max=256 mdc.lustre-MDT0000-mdc-ffff88007901d800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff88007901d800.batch_stats=0 llite.lustre-ffff88007901d800.statahead_max=2048 llite.lustre-ffff88007901d800.statahead_batch_max=256 kvm mdc.lustre-MDT0000-mdc-ffff88007901d800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff88007901d800.batch_stats=0 striped dir -i1 -c2 -H crush2 /mnt/lustre/d123ad.sanity total: 100 open/close in 1.23 seconds: 81.20 ops/second llite.lustre-ffff88007901d800.statahead_max=0 101 real 0m0.995s user 0m0.006s sys 0m0.282s ls -l 100 files without statahead: 1 sec llite.lustre-ffff88007901d800.statahead_max=2048 2048 101 real 0m0.340s user 0m0.008s sys 0m0.131s ls -l 100 files with statahead: 0 sec statahead total: 2 statahead wrong: 0 agl total: 2 list_total: 2 fname_total: 0 hit_total: 10098 miss_total: 51 total: 900 open/close in 5.61 seconds: 160.54 ops/second llite.lustre-ffff88007901d800.statahead_max=0 1001 real 0m10.055s user 0m0.060s sys 0m2.714s ls -l 1000 files without statahead: 10 sec llite.lustre-ffff88007901d800.statahead_max=2048 2048 1001 real 0m1.513s user 0m0.020s sys 0m1.019s ls -l 1000 files with statahead: 2 sec statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11097 miss_total: 52 - open/close 1568 (time 1746130668.46 total 10.01 last 156.70) - open/close 3187 (time 1746130678.46 total 20.01 last 161.89) total: 4000 open/close in 24.96 seconds: 160.25 ops/second llite.lustre-ffff88007901d800.statahead_max=0 5001 real 0m52.726s user 0m0.391s sys 0m13.905s ls -l 5000 files without statahead: 53 sec llite.lustre-ffff88007901d800.statahead_max=2048 2048 5001 real 0m7.010s user 0m0.168s sys 0m5.149s ls -l 5000 files with statahead: 7 sec statahead total: 4 statahead wrong: 0 agl total: 4 list_total: 4 fname_total: 0 hit_total: 16096 miss_total: 53 ls -l done rm -r /mnt/lustre/d123ad.sanity: 37 seconds llite.lustre-ffff88007901d800.statahead_stats= statahead total: 4 statahead wrong: 0 agl total: 4 list_total: 4 fname_total: 0 hit_total: 16096 miss_total: 53 mdc.lustre-MDT0000-mdc-ffff88007901d800.batch_stats= snapshot_time: 1746130785.172713892 secs.nsecs start_time: 1746130494.599091782 secs.nsecs elapsed_time: 290.573622110 secs.nsecs subreqs per batch batches % cum % 1: 36 59 59 2: 3 4 63 4: 2 3 67 8: 1 1 68 16: 1 1 70 32: 2 3 73 64: 0 0 73 128: 5 8 81 256: 11 18 100 mdc.lustre-MDT0001-mdc-ffff88007901d800.batch_stats= snapshot_time: 1746130785.172796852 secs.nsecs start_time: 1746130494.599186929 secs.nsecs elapsed_time: 290.573609923 secs.nsecs subreqs per batch batches % cum % 1: 32 53 53 2: 0 0 53 4: 2 3 56 8: 6 10 66 16: 0 0 66 32: 2 3 70 64: 1 1 71 128: 1 1 73 256: 16 26 100 - open/close 2347 (time 1746130795.92 total 10.00 last 234.63) - open/close 4336 (time 1746130805.93 total 20.01 last 198.78) - open/close 6381 (time 1746130815.93 total 30.01 last 204.46) - open/close 9485 (time 1746130825.93 total 40.01 last 310.37) total: 10000 open/close in 41.65 seconds: 240.07 ops/second llite.lustre-ffff88007901d800.statahead_batch_max=0 llite.lustre-ffff88007901d800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff88007901d800.stats=clear mdc.lustre-MDT0001-mdc-ffff88007901d800.stats=clear ls -l files in d123ad.sanity: 10001 real 0m11.667s user 0m0.219s sys 0m9.733s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff88007901d800.statahead_batch_max=256 llite.lustre-ffff88007901d800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff88007901d800.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff88007901d800.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff88007901d800.stats=clear mdc.lustre-MDT0001-mdc-ffff88007901d800.stats=clear ls -l files in d123ad.sanity: 10001 real 0m12.274s user 0m0.272s sys 0m9.932s batched statahead RPC= hit count=9999 mdc.lustre-MDT0000-mdc-ffff88007901d800.batch_stats= snapshot_time: 1746130856.542373852 secs.nsecs start_time: 1746130841.409892896 secs.nsecs elapsed_time: 15.132480956 secs.nsecs subreqs per batch batches % cum % 1: 194 62 62 2: 20 6 69 4: 13 4 73 8: 17 5 78 16: 11 3 82 32: 17 5 87 64: 3 0 88 128: 0 0 88 256: 35 11 100 mdc.lustre-MDT0001-mdc-ffff88007901d800.batch_stats= snapshot_time: 1746130856.542622776 secs.nsecs start_time: 1746130841.411035146 secs.nsecs elapsed_time: 15.131587630 secs.nsecs subreqs per batch batches % cum % 1: 0 0 0 seq.cli-lustre-OST0000-super.width=65536 seq.cli-lustre-OST0001-super.width=65536 llite.lustre-ffff88007901d800.statahead_batch_max=64 llite.lustre-ffff88007901d800.statahead_max=128