== sanity test 123ad: Verify batching statahead works correctly ========================================================== 18:39:53 (1785969593) batch: statahead_max=32 statahead_batch_max=32 mdc.lustre-MDT0000-mdc-ffff98bcd0ac1000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff98bcd0ac1000.batch_stats=0 llite.lustre-ffff98bcd0ac1000.statahead_max=32 llite.lustre-ffff98bcd0ac1000.statahead_batch_max=32 seq.cli-lustre-OST0000-super.width=0x1ffffff seq.cli-lustre-OST0001-super.width=0x1ffffff kvm mdc.lustre-MDT0000-mdc-ffff98bcd0ac1000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff98bcd0ac1000.batch_stats=0 striped dir -i1 -c2 -H crush /mnt/lustre/d123ad.sanity total: 100 open/close in 2.47 seconds: 40.44 ops/second llite.lustre-ffff98bcd0ac1000.statahead_max=0 101 real 0m2.702s user 0m0.003s sys 0m0.342s 'ls -l' 100 files without statahead: 3 sec llite.lustre-ffff98bcd0ac1000.statahead_max=32 llite.lustre-ffff98bcd0ac1000.statahead_max=32 101 real 0m1.370s user 0m0.004s sys 0m0.173s 'ls -l' 100 files with statahead: 1 sec statahead total: 137 statahead wrong: 2 agl total: 131 list_total: 31 fname_total: 106 hit_total: 4035 miss_total: 1494 swrong=2 ewrong=2 - open/close 723 (time 1785969624.51 total 10.00 last 72.29) total: 900 open/close in 13.79 seconds: 65.27 ops/second llite.lustre-ffff98bcd0ac1000.statahead_max=0 1001 real 0m26.898s user 0m0.036s sys 0m3.215s 'ls -l' 1000 files without statahead: 27 sec llite.lustre-ffff98bcd0ac1000.statahead_max=32 llite.lustre-ffff98bcd0ac1000.statahead_max=32 1001 real 0m11.238s user 0m0.034s sys 0m1.638s 'ls -l' 1000 files with statahead: 11 sec statahead total: 138 statahead wrong: 2 agl total: 132 list_total: 32 fname_total: 106 hit_total: 5033 miss_total: 1496 swrong=2 ewrong=2 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 20 seconds llite.lustre-ffff98bcd0ac1000.statahead_stats= statahead total: 138 statahead wrong: 2 agl total: 132 list_total: 32 fname_total: 106 hit_total: 5033 miss_total: 1496 mdc.lustre-MDT0000-mdc-ffff98bcd0ac1000.batch_stats= snapshot_time: 1785969704.099510169 secs.nsecs start_time: 1785969599.699304524 secs.nsecs elapsed_time: 104.400205645 secs.nsecs subreqs per batch batches %% cum %% 1: 2 5 5 2: 2 5 10 4: 2 5 16 8: 2 5 21 16: 23 62 83 32: 6 16 100 mdc.lustre-MDT0001-mdc-ffff98bcd0ac1000.batch_stats= snapshot_time: 1785969704.099572691 secs.nsecs start_time: 1785969599.699387215 secs.nsecs elapsed_time: 104.400185476 secs.nsecs subreqs per batch batches %% cum %% 1: 2 4 4 2: 1 2 7 4: 1 2 9 8: 4 9 19 16: 8 19 38 32: 26 61 100 - open/close 914 (time 1785969716.76 total 10.01 last 91.35) - open/close 1715 (time 1785969726.76 total 20.01 last 80.08) - open/close 2575 (time 1785969736.77 total 30.02 last 85.93) - open/close 3398 (time 1785969746.78 total 40.02 last 82.24) - open/close 4211 (time 1785969756.78 total 50.03 last 81.28) - open/close 4858 (time 1785969766.79 total 60.03 last 64.67) - open/close 5721 (time 1785969776.79 total 70.04 last 86.26) - open/close 6390 (time 1785969786.80 total 80.04 last 66.86) - open/close 7187 (time 1785969796.81 total 90.05 last 79.61) - open/close 8070 (time 1785969806.81 total 100.06 last 88.27) - open/close 8882 (time 1785969816.81 total 110.06 last 81.19) - open/close 9687 (time 1785969826.82 total 120.06 last 80.48) total: 10000 open/close in 123.92 seconds: 80.70 ops/second llite.lustre-ffff98bcd0ac1000.statahead_batch_max=0 llite.lustre-ffff98bcd0ac1000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff98bcd0ac1000.stats=clear mdc.lustre-MDT0001-mdc-ffff98bcd0ac1000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m11.758s user 0m0.310s sys 0m16.290s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff98bcd0ac1000.statahead_batch_max=32 llite.lustre-ffff98bcd0ac1000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff98bcd0ac1000.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff98bcd0ac1000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff98bcd0ac1000.stats=clear mdc.lustre-MDT0001-mdc-ffff98bcd0ac1000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m50.127s user 0m0.304s sys 0m15.483s batched statahead RPC= hit count=9984 mdc.lustre-MDT0000-mdc-ffff98bcd0ac1000.batch_stats= snapshot_time: 1785970043.570143546 secs.nsecs start_time: 1785969915.407964311 secs.nsecs elapsed_time: 128.162179235 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 0 0 0 8: 1 0 0 16: 0 0 0 32: 313 99 100 mdc.lustre-MDT0001-mdc-ffff98bcd0ac1000.batch_stats= snapshot_time: 1785970043.570212593 secs.nsecs start_time: 1785969915.408052200 secs.nsecs elapsed_time: 128.162160393 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 batch: statahead_max=2048 statahead_batch_max=256 mdc.lustre-MDT0000-mdc-ffff98bcd0ac1000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff98bcd0ac1000.batch_stats=0 llite.lustre-ffff98bcd0ac1000.statahead_max=2048 llite.lustre-ffff98bcd0ac1000.statahead_batch_max=256 kvm mdc.lustre-MDT0000-mdc-ffff98bcd0ac1000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff98bcd0ac1000.batch_stats=0 striped dir -i1 -c2 -H crush2 /mnt/lustre/d123ad.sanity total: 100 open/close in 2.45 seconds: 40.87 ops/second llite.lustre-ffff98bcd0ac1000.statahead_max=0 101 real 0m2.652s user 0m0.004s sys 0m0.339s 'ls -l' 100 files without statahead: 2 sec llite.lustre-ffff98bcd0ac1000.statahead_max=2048 llite.lustre-ffff98bcd0ac1000.statahead_max=2048 101 real 0m1.016s user 0m0.002s sys 0m0.195s 'ls -l' 100 files with statahead: 1 sec statahead total: 2 statahead wrong: 0 agl total: 2 list_total: 2 fname_total: 0 hit_total: 10083 miss_total: 17 swrong=0 ewrong=0 total: 900 open/close in 9.86 seconds: 91.28 ops/second llite.lustre-ffff98bcd0ac1000.statahead_max=0 1001 real 0m24.068s user 0m0.036s sys 0m3.253s 'ls -l' 1000 files without statahead: 24 sec llite.lustre-ffff98bcd0ac1000.statahead_max=2048 llite.lustre-ffff98bcd0ac1000.statahead_max=2048 1001 real 0m7.584s user 0m0.032s sys 0m1.470s 'ls -l' 1000 files with statahead: 8 sec statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11082 miss_total: 18 swrong=0 ewrong=0 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 16 seconds llite.lustre-ffff98bcd0ac1000.statahead_stats= statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11082 miss_total: 18 mdc.lustre-MDT0000-mdc-ffff98bcd0ac1000.batch_stats= snapshot_time: 1785970379.585571598 secs.nsecs start_time: 1785970045.934345045 secs.nsecs elapsed_time: 333.651226553 secs.nsecs subreqs per batch batches %% cum %% 1: 1 14 14 2: 0 0 14 4: 3 42 57 8: 1 14 71 16: 0 0 71 32: 0 0 71 64: 0 0 71 128: 0 0 71 256: 2 28 100 mdc.lustre-MDT0001-mdc-ffff98bcd0ac1000.batch_stats= snapshot_time: 1785970379.585667645 secs.nsecs start_time: 1785970045.936051275 secs.nsecs elapsed_time: 333.649616370 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 1 11 11 8: 3 33 44 16: 1 11 55 32: 1 11 66 64: 0 0 66 128: 1 11 77 256: 2 22 100 - open/close 1016 (time 1785970391.83 total 10.01 last 101.52) - open/close 2021 (time 1785970401.84 total 20.02 last 100.42) - open/close 3001 (time 1785970411.85 total 30.03 last 97.91) - open/close 3966 (time 1785970421.86 total 40.03 last 96.46) - open/close 5019 (time 1785970431.86 total 50.04 last 105.24) - open/close 6053 (time 1785970441.88 total 60.05 last 103.21) - open/close 6958 (time 1785970451.88 total 70.06 last 90.49) - open/close 7846 (time 1785970461.89 total 80.06 last 88.73) - open/close 8747 (time 1785970471.89 total 90.07 last 90.07) - open/close 9608 (time 1785970481.90 total 100.07 last 86.04) total: 10000 open/close in 104.19 seconds: 95.98 ops/second llite.lustre-ffff98bcd0ac1000.statahead_batch_max=0 llite.lustre-ffff98bcd0ac1000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff98bcd0ac1000.stats=clear mdc.lustre-MDT0001-mdc-ffff98bcd0ac1000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m3.116s user 0m0.325s sys 0m15.904s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff98bcd0ac1000.statahead_batch_max=256 llite.lustre-ffff98bcd0ac1000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff98bcd0ac1000.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff98bcd0ac1000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff98bcd0ac1000.stats=clear mdc.lustre-MDT0001-mdc-ffff98bcd0ac1000.stats=clear ls -l files in d123ad.sanity: 10001 real 0m57.734s user 0m0.328s sys 0m17.073s batched statahead RPC= hit count=9999 mdc.lustre-MDT0000-mdc-ffff98bcd0ac1000.batch_stats= snapshot_time: 1785970631.653819603 secs.nsecs start_time: 1785970560.076481369 secs.nsecs elapsed_time: 71.577338234 secs.nsecs subreqs per batch batches %% cum %% 1: 270 59 59 2: 64 14 73 4: 37 8 82 8: 14 3 85 16: 11 2 87 32: 19 4 91 64: 3 0 92 128: 0 0 92 256: 34 7 100 mdc.lustre-MDT0001-mdc-ffff98bcd0ac1000.batch_stats= snapshot_time: 1785970631.653883328 secs.nsecs start_time: 1785970560.076950327 secs.nsecs elapsed_time: 71.576933001 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second seq.cli-lustre-OST0000-super.width=65536 seq.cli-lustre-OST0001-super.width=65536 llite.lustre-ffff98bcd0ac1000.statahead_batch_max=64 llite.lustre-ffff98bcd0ac1000.statahead_max=128