== sanity test 123ad: Verify batching statahead works correctly ========================================================== 19:00:02 (1786834802) batch: statahead_max=32 statahead_batch_max=32 mdc.lustre-MDT0000-mdc-ffff914cc7d1f000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff914cc7d1f000.batch_stats=0 llite.lustre-ffff914cc7d1f000.statahead_max=32 llite.lustre-ffff914cc7d1f000.statahead_batch_max=32 seq.cli-lustre-OST0000-super.width=0x1ffffff seq.cli-lustre-OST0001-super.width=0x1ffffff kvm mdc.lustre-MDT0000-mdc-ffff914cc7d1f000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff914cc7d1f000.batch_stats=0 striped dir -i1 -c2 -H crush2 /mnt/lustre/d123ad.sanity total: 100 open/close in 2.12 seconds: 47.24 ops/second llite.lustre-ffff914cc7d1f000.statahead_max=0 101 real 0m2.123s user 0m0.003s sys 0m0.333s 'ls -l' 100 files without statahead: 2 sec llite.lustre-ffff914cc7d1f000.statahead_max=32 llite.lustre-ffff914cc7d1f000.statahead_max=32 101 real 0m0.934s user 0m0.004s sys 0m0.215s 'ls -l' 100 files with statahead: 1 sec statahead total: 142 statahead wrong: 2 agl total: 136 list_total: 31 fname_total: 101 hit_total: 3727 miss_total: 1684 swrong=2 ewrong=2 - open/close 875 (time 1786834827.70 total 10.00 last 87.50) total: 900 open/close in 10.28 seconds: 87.52 ops/second llite.lustre-ffff914cc7d1f000.statahead_max=0 1001 real 0m18.319s user 0m0.033s sys 0m3.179s 'ls -l' 1000 files without statahead: 18 sec llite.lustre-ffff914cc7d1f000.statahead_max=32 llite.lustre-ffff914cc7d1f000.statahead_max=32 1001 real 0m7.760s user 0m0.026s sys 0m1.630s 'ls -l' 1000 files with statahead: 7 sec statahead total: 143 statahead wrong: 2 agl total: 137 list_total: 32 fname_total: 101 hit_total: 4726 miss_total: 1685 swrong=2 ewrong=2 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 15 seconds llite.lustre-ffff914cc7d1f000.statahead_stats= statahead total: 143 statahead wrong: 2 agl total: 137 list_total: 32 fname_total: 101 hit_total: 4726 miss_total: 1685 mdc.lustre-MDT0000-mdc-ffff914cc7d1f000.batch_stats= snapshot_time: 1786834880.165530257 secs.nsecs start_time: 1786834806.927075311 secs.nsecs elapsed_time: 73.238454946 secs.nsecs subreqs per batch batches %% cum %% 1: 4 10 10 2: 2 5 15 4: 1 2 18 8: 2 5 23 16: 23 60 84 32: 6 15 100 mdc.lustre-MDT0001-mdc-ffff914cc7d1f000.batch_stats= snapshot_time: 1786834880.165850700 secs.nsecs start_time: 1786834806.927225090 secs.nsecs elapsed_time: 73.238625610 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 2 5 5 8: 2 5 10 16: 11 28 38 32: 24 61 100 - open/close 874 (time 1786834892.38 total 10.00 last 87.39) - open/close 1792 (time 1786834902.39 total 20.01 last 91.75) - open/close 2684 (time 1786834912.39 total 30.01 last 89.16) - open/close 3620 (time 1786834922.40 total 40.01 last 93.57) - open/close 4498 (time 1786834932.40 total 50.02 last 87.76) - open/close 5411 (time 1786834942.41 total 60.02 last 91.25) - open/close 6311 (time 1786834952.41 total 70.03 last 89.95) - open/close 7243 (time 1786834962.41 total 80.03 last 93.18) - open/close 8147 (time 1786834972.42 total 90.04 last 90.31) - open/close 8968 (time 1786834982.43 total 100.05 last 82.08) - open/close 9851 (time 1786834992.43 total 110.05 last 88.25) total: 10000 open/close in 111.78 seconds: 89.46 ops/second llite.lustre-ffff914cc7d1f000.statahead_batch_max=0 llite.lustre-ffff914cc7d1f000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff914cc7d1f000.stats=clear mdc.lustre-MDT0001-mdc-ffff914cc7d1f000.stats=clear ls -l files in d123ad.sanity: 10001 real 0m53.529s user 0m0.347s sys 0m20.119s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff914cc7d1f000.statahead_batch_max=32 llite.lustre-ffff914cc7d1f000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff914cc7d1f000.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff914cc7d1f000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff914cc7d1f000.stats=clear mdc.lustre-MDT0001-mdc-ffff914cc7d1f000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m28.823s user 0m0.298s sys 0m16.549s batched statahead RPC= hit count=9987 mdc.lustre-MDT0000-mdc-ffff914cc7d1f000.batch_stats= snapshot_time: 1786835157.189384244 secs.nsecs start_time: 1786835056.854494108 secs.nsecs elapsed_time: 100.334890136 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 0 0 0 8: 1 0 0 16: 1 0 0 32: 313 99 100 mdc.lustre-MDT0001-mdc-ffff914cc7d1f000.batch_stats= snapshot_time: 1786835157.189444854 secs.nsecs start_time: 1786835056.854736596 secs.nsecs elapsed_time: 100.334708258 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 batch: statahead_max=2048 statahead_batch_max=256 mdc.lustre-MDT0000-mdc-ffff914cc7d1f000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff914cc7d1f000.batch_stats=0 llite.lustre-ffff914cc7d1f000.statahead_max=2048 llite.lustre-ffff914cc7d1f000.statahead_batch_max=256 kvm mdc.lustre-MDT0000-mdc-ffff914cc7d1f000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff914cc7d1f000.batch_stats=0 striped dir -i1 -c2 -H crush /mnt/lustre/d123ad.sanity total: 100 open/close in 1.92 seconds: 52.10 ops/second llite.lustre-ffff914cc7d1f000.statahead_max=0 101 real 0m1.835s user 0m0.003s sys 0m0.343s 'ls -l' 100 files without statahead: 2 sec llite.lustre-ffff914cc7d1f000.statahead_max=2048 llite.lustre-ffff914cc7d1f000.statahead_max=2048 101 real 0m0.672s user 0m0.006s sys 0m0.222s 'ls -l' 100 files with statahead: 0 sec statahead total: 2 statahead wrong: 0 agl total: 2 list_total: 2 fname_total: 0 hit_total: 10086 miss_total: 14 swrong=0 ewrong=0 total: 900 open/close in 9.56 seconds: 94.17 ops/second llite.lustre-ffff914cc7d1f000.statahead_max=0 1001 real 0m17.942s user 0m0.029s sys 0m3.181s 'ls -l' 1000 files without statahead: 18 sec llite.lustre-ffff914cc7d1f000.statahead_max=2048 llite.lustre-ffff914cc7d1f000.statahead_max=2048 1001 real 0m4.726s user 0m0.026s sys 0m1.671s 'ls -l' 1000 files with statahead: 5 sec statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11085 miss_total: 15 swrong=0 ewrong=0 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 14 seconds llite.lustre-ffff914cc7d1f000.statahead_stats= statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11085 miss_total: 15 mdc.lustre-MDT0000-mdc-ffff914cc7d1f000.batch_stats= snapshot_time: 1786835463.641846038 secs.nsecs start_time: 1786835159.509776824 secs.nsecs elapsed_time: 304.132069214 secs.nsecs subreqs per batch batches %% cum %% 1: 1 16 16 2: 1 16 33 4: 2 33 66 8: 0 0 66 16: 0 0 66 32: 0 0 66 64: 0 0 66 128: 0 0 66 256: 2 33 100 mdc.lustre-MDT0001-mdc-ffff914cc7d1f000.batch_stats= snapshot_time: 1786835463.641912619 secs.nsecs start_time: 1786835159.509918966 secs.nsecs elapsed_time: 304.131993653 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 1 12 12 8: 2 25 37 16: 1 12 50 32: 0 0 50 64: 1 12 62 128: 1 12 75 256: 2 25 100 - open/close 931 (time 1786835475.50 total 10.00 last 93.06) - open/close 1909 (time 1786835485.50 total 20.01 last 97.77) - open/close 2919 (time 1786835495.50 total 30.01 last 100.96) - open/close 3933 (time 1786835505.51 total 40.02 last 101.31) - open/close 4907 (time 1786835515.51 total 50.02 last 97.39) - open/close 5886 (time 1786835525.52 total 60.03 last 97.84) - open/close 6840 (time 1786835535.54 total 70.04 last 95.24) - open/close 7803 (time 1786835545.54 total 80.05 last 96.25) - open/close 8760 (time 1786835555.55 total 90.06 last 95.60) - open/close 9759 (time 1786835565.56 total 100.07 last 99.82) total: 10000 open/close in 102.58 seconds: 97.49 ops/second llite.lustre-ffff914cc7d1f000.statahead_batch_max=0 llite.lustre-ffff914cc7d1f000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff914cc7d1f000.stats=clear mdc.lustre-MDT0001-mdc-ffff914cc7d1f000.stats=clear ls -l files in d123ad.sanity: 10001 real 0m44.793s user 0m0.310s sys 0m21.553s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff914cc7d1f000.statahead_batch_max=256 llite.lustre-ffff914cc7d1f000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff914cc7d1f000.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff914cc7d1f000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff914cc7d1f000.stats=clear mdc.lustre-MDT0001-mdc-ffff914cc7d1f000.stats=clear ls -l files in d123ad.sanity: 10001 real 0m40.766s user 0m0.286s sys 0m20.431s batched statahead RPC= hit count=9999 mdc.lustre-MDT0000-mdc-ffff914cc7d1f000.batch_stats= snapshot_time: 1786835671.313280354 secs.nsecs start_time: 1786835620.266936408 secs.nsecs elapsed_time: 51.046343946 secs.nsecs subreqs per batch batches %% cum %% 1: 275 56 56 2: 68 14 70 4: 57 11 82 8: 20 4 86 16: 6 1 88 32: 17 3 91 64: 7 1 92 128: 1 0 93 256: 33 6 100 mdc.lustre-MDT0001-mdc-ffff914cc7d1f000.batch_stats= snapshot_time: 1786835671.313344799 secs.nsecs start_time: 1786835620.267024548 secs.nsecs elapsed_time: 51.046320251 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second seq.cli-lustre-OST0000-super.width=65536 seq.cli-lustre-OST0001-super.width=65536 llite.lustre-ffff914cc7d1f000.statahead_batch_max=64 llite.lustre-ffff914cc7d1f000.statahead_max=128