== sanity test 123ad: Verify batching statahead works correctly ========================================================== 16:59:23 (1746133163) batch: statahead_max=32 statahead_batch_max=32 mdc.lustre-MDT0000-mdc-ffff88012e3f3800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff88012e3f3800.batch_stats=0 llite.lustre-ffff88012e3f3800.statahead_max=32 llite.lustre-ffff88012e3f3800.statahead_batch_max=32 seq.cli-lustre-OST0000-super.width=0x1ffffff seq.cli-lustre-OST0001-super.width=0x1ffffff kvm mdc.lustre-MDT0000-mdc-ffff88012e3f3800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff88012e3f3800.batch_stats=0 striped dir -i1 -c2 -H crush /mnt/lustre/d123ad.sanity total: 100 open/close in 1.51 seconds: 66.24 ops/second llite.lustre-ffff88012e3f3800.statahead_max=0 101 real 0m1.274s user 0m0.005s sys 0m0.303s ls -l 100 files without statahead: 1 sec llite.lustre-ffff88012e3f3800.statahead_max=32 32 101 real 0m0.427s user 0m0.004s sys 0m0.135s ls -l 100 files with statahead: 1 sec statahead total: 127 statahead wrong: 2 agl total: 127 list_total: 29 fname_total: 98 hit_total: 5947 miss_total: 1337 total: 900 open/close in 5.74 seconds: 156.69 ops/second llite.lustre-ffff88012e3f3800.statahead_max=0 1001 real 0m11.418s user 0m0.088s sys 0m2.703s ls -l 1000 files without statahead: 11 sec llite.lustre-ffff88012e3f3800.statahead_max=32 32 1001 real 0m3.376s user 0m0.021s sys 0m1.108s ls -l 1000 files with statahead: 4 sec statahead total: 128 statahead wrong: 2 agl total: 128 list_total: 30 fname_total: 98 hit_total: 6946 miss_total: 1343 - open/close 1754 (time 1746133205.95 total 10.00 last 175.39) total: 3000 open/close in 17.62 seconds: 170.31 ops/second llite.lustre-ffff88012e3f3800.statahead_max=0 4001 real 0m47.148s user 0m0.503s sys 0m11.050s ls -l 4000 files without statahead: 48 sec llite.lustre-ffff88012e3f3800.statahead_max=32 32 4001 real 0m12.680s user 0m0.130s sys 0m4.320s ls -l 4000 files with statahead: 13 sec statahead total: 129 statahead wrong: 2 agl total: 129 list_total: 31 fname_total: 98 hit_total: 10945 miss_total: 1371 ls -l done rm -r /mnt/lustre/d123ad.sanity: 38 seconds llite.lustre-ffff88012e3f3800.statahead_stats= statahead total: 129 statahead wrong: 2 agl total: 129 list_total: 31 fname_total: 98 hit_total: 10945 miss_total: 1371 mdc.lustre-MDT0000-mdc-ffff88012e3f3800.batch_stats= snapshot_time: 1746133317.618230424 secs.nsecs start_time: 1746133165.791152133 secs.nsecs elapsed_time: 151.827078291 secs.nsecs subreqs per batch batches % cum % 1: 3 1 1 2: 0 0 1 4: 2 1 3 8: 8 4 7 16: 104 63 71 32: 47 28 100 mdc.lustre-MDT0001-mdc-ffff88012e3f3800.batch_stats= snapshot_time: 1746133317.618372837 secs.nsecs start_time: 1746133165.791757165 secs.nsecs elapsed_time: 151.826615672 secs.nsecs subreqs per batch batches % cum % 1: 0 0 0 2: 3 1 1 4: 1 0 2 8: 3 1 4 16: 67 40 44 32: 92 55 100 - open/close 1549 (time 1746133328.87 total 10.00 last 154.90) - open/close 3077 (time 1746133338.87 total 20.00 last 152.77) - open/close 4640 (time 1746133348.88 total 30.01 last 156.23) - open/close 6211 (time 1746133358.88 total 40.01 last 157.02) - open/close 7701 (time 1746133368.89 total 50.02 last 148.95) - open/close 9313 (time 1746133378.89 total 60.02 last 161.20) total: 10000 open/close in 63.12 seconds: 158.44 ops/second llite.lustre-ffff88012e3f3800.statahead_batch_max=0 llite.lustre-ffff88012e3f3800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff88012e3f3800.stats=clear mdc.lustre-MDT0001-mdc-ffff88012e3f3800.stats=clear ls -l files in d123ad.sanity: 10001 real 0m17.451s user 0m0.303s sys 0m8.973s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff88012e3f3800.statahead_batch_max=32 llite.lustre-ffff88012e3f3800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff88012e3f3800.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff88012e3f3800.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff88012e3f3800.stats=clear mdc.lustre-MDT0001-mdc-ffff88012e3f3800.stats=clear ls -l files in d123ad.sanity: 10001 real 0m35.740s user 0m0.325s sys 0m10.711s batched statahead RPC= hit count=9999 mdc.lustre-MDT0000-mdc-ffff88012e3f3800.batch_stats= snapshot_time: 1746133441.100800777 secs.nsecs start_time: 1746133401.758192867 secs.nsecs elapsed_time: 39.342607910 secs.nsecs subreqs per batch batches % cum % 1: 1 0 0 2: 0 0 0 4: 0 0 0 8: 1 0 0 16: 2 0 1 32: 312 98 100 mdc.lustre-MDT0001-mdc-ffff88012e3f3800.batch_stats= snapshot_time: 1746133441.101022333 secs.nsecs start_time: 1746133401.758247514 secs.nsecs elapsed_time: 39.342774819 secs.nsecs subreqs per batch batches % cum % 1: 0 0 0 batch: statahead_max=2048 statahead_batch_max=256 mdc.lustre-MDT0000-mdc-ffff88012e3f3800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff88012e3f3800.batch_stats=0 llite.lustre-ffff88012e3f3800.statahead_max=2048 llite.lustre-ffff88012e3f3800.statahead_batch_max=256 kvm mdc.lustre-MDT0000-mdc-ffff88012e3f3800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff88012e3f3800.batch_stats=0 striped dir -i1 -c2 -H crush2 /mnt/lustre/d123ad.sanity total: 100 open/close in 1.41 seconds: 70.92 ops/second llite.lustre-ffff88012e3f3800.statahead_max=0 101 real 0m1.372s user 0m0.000s sys 0m0.308s ls -l 100 files without statahead: 1 sec llite.lustre-ffff88012e3f3800.statahead_max=2048 2048 101 real 0m0.372s user 0m0.009s sys 0m0.115s ls -l 100 files with statahead: 0 sec statahead total: 2 statahead wrong: 0 agl total: 2 list_total: 2 fname_total: 0 hit_total: 10098 miss_total: 56 total: 900 open/close in 5.36 seconds: 167.93 ops/second llite.lustre-ffff88012e3f3800.statahead_max=0 1001 real 0m12.125s user 0m0.081s sys 0m2.792s ls -l 1000 files without statahead: 12 sec llite.lustre-ffff88012e3f3800.statahead_max=2048 2048 1001 real 0m2.299s user 0m0.029s sys 0m1.143s ls -l 1000 files with statahead: 2 sec statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11097 miss_total: 57 - open/close 1561 (time 1746133626.29 total 10.00 last 156.08) total: 3000 open/close in 18.85 seconds: 159.13 ops/second llite.lustre-ffff88012e3f3800.statahead_max=0 4001 real 0m47.637s user 0m0.542s sys 0m10.978s ls -l 4000 files without statahead: 48 sec llite.lustre-ffff88012e3f3800.statahead_max=2048 2048 4001 real 0m6.858s user 0m0.123s sys 0m4.328s ls -l 4000 files with statahead: 7 sec statahead total: 4 statahead wrong: 0 agl total: 4 list_total: 4 fname_total: 0 hit_total: 15096 miss_total: 58 ls -l done rm -r /mnt/lustre/d123ad.sanity: 37 seconds llite.lustre-ffff88012e3f3800.statahead_stats= statahead total: 4 statahead wrong: 0 agl total: 4 list_total: 4 fname_total: 0 hit_total: 15096 miss_total: 58 mdc.lustre-MDT0000-mdc-ffff88012e3f3800.batch_stats= snapshot_time: 1746133731.769200222 secs.nsecs start_time: 1746133441.756190194 secs.nsecs elapsed_time: 290.013010028 secs.nsecs subreqs per batch batches % cum % 1: 24 46 46 2: 5 9 55 4: 4 7 63 8: 4 7 71 16: 2 3 75 32: 0 0 75 64: 0 0 75 128: 3 5 80 256: 10 19 100 mdc.lustre-MDT0001-mdc-ffff88012e3f3800.batch_stats= snapshot_time: 1746133731.769287395 secs.nsecs start_time: 1746133441.756283306 secs.nsecs elapsed_time: 290.013004089 secs.nsecs subreqs per batch batches % cum % 1: 28 46 46 2: 6 10 56 4: 2 3 60 8: 7 11 71 16: 1 1 73 32: 0 0 73 64: 2 3 76 128: 2 3 80 256: 12 20 100 - open/close 1516 (time 1746133742.76 total 10.01 last 151.50) - open/close 3020 (time 1746133752.76 total 20.01 last 150.37) - open/close 4538 (time 1746133762.76 total 30.01 last 151.79) - open/close 6203 (time 1746133772.76 total 40.01 last 166.45) - open/close 7785 (time 1746133782.76 total 50.01 last 158.20) - open/close 9338 (time 1746133792.76 total 60.01 last 155.30) total: 10000 open/close in 63.64 seconds: 157.13 ops/second llite.lustre-ffff88012e3f3800.statahead_batch_max=0 llite.lustre-ffff88012e3f3800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff88012e3f3800.stats=clear mdc.lustre-MDT0001-mdc-ffff88012e3f3800.stats=clear ls -l files in d123ad.sanity: 10001 real 0m26.832s user 0m0.246s sys 0m9.622s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff88012e3f3800.statahead_batch_max=256 llite.lustre-ffff88012e3f3800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff88012e3f3800.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff88012e3f3800.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff88012e3f3800.stats=clear mdc.lustre-MDT0001-mdc-ffff88012e3f3800.stats=clear ls -l files in d123ad.sanity: 10001 real 0m19.202s user 0m0.278s sys 0m10.745s batched statahead RPC= hit count=9999 mdc.lustre-MDT0000-mdc-ffff88012e3f3800.batch_stats= snapshot_time: 1746133849.566532516 secs.nsecs start_time: 1746133826.110998679 secs.nsecs elapsed_time: 23.455533837 secs.nsecs subreqs per batch batches % cum % 1: 153 59 59 2: 16 6 66 4: 8 3 69 8: 8 3 72 16: 11 4 76 32: 19 7 83 64: 6 2 86 128: 0 0 86 256: 35 13 100 mdc.lustre-MDT0001-mdc-ffff88012e3f3800.batch_stats= snapshot_time: 1746133849.566633802 secs.nsecs start_time: 1746133826.111057737 secs.nsecs elapsed_time: 23.455576065 secs.nsecs subreqs per batch batches % cum % 1: 0 0 0 seq.cli-lustre-OST0000-super.width=65536 seq.cli-lustre-OST0001-super.width=65536 llite.lustre-ffff88012e3f3800.statahead_batch_max=64 llite.lustre-ffff88012e3f3800.statahead_max=128