== sanity test 123ad: Verify batching statahead works correctly ========================================================== 08:29:28 (1787833768) batch: statahead_max=32 statahead_batch_max=32 mdc.lustre-MDT0000-mdc-ffff9b0cff301800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9b0cff301800.batch_stats=0 llite.lustre-ffff9b0cff301800.statahead_max=32 llite.lustre-ffff9b0cff301800.statahead_batch_max=32 seq.cli-lustre-OST0000-super.width=0x1ffffff seq.cli-lustre-OST0001-super.width=0x1ffffff kvm mdc.lustre-MDT0000-mdc-ffff9b0cff301800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9b0cff301800.batch_stats=0 striped dir -i1 -c2 -H crush /mnt/lustre/d123ad.sanity total: 100 open/close in 2.22 seconds: 45.09 ops/second llite.lustre-ffff9b0cff301800.statahead_max=0 101 real 0m1.864s user 0m0.003s sys 0m0.294s 'ls -l' 100 files without statahead: 2 sec llite.lustre-ffff9b0cff301800.statahead_max=32 llite.lustre-ffff9b0cff301800.statahead_max=32 101 real 0m1.255s user 0m0.003s sys 0m0.163s 'ls -l' 100 files with statahead: 1 sec statahead total: 142 statahead wrong: 2 agl total: 136 list_total: 30 fname_total: 112 hit_total: 3766 miss_total: 1229 swrong=2 ewrong=2 total: 900 open/close in 9.67 seconds: 93.07 ops/second llite.lustre-ffff9b0cff301800.statahead_max=0 1001 real 0m19.864s user 0m0.032s sys 0m2.872s 'ls -l' 1000 files without statahead: 20 sec llite.lustre-ffff9b0cff301800.statahead_max=32 llite.lustre-ffff9b0cff301800.statahead_max=32 1001 real 0m7.742s user 0m0.016s sys 0m1.267s 'ls -l' 1000 files with statahead: 8 sec statahead total: 143 statahead wrong: 2 agl total: 137 list_total: 31 fname_total: 112 hit_total: 4765 miss_total: 1230 swrong=2 ewrong=2 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 17 seconds llite.lustre-ffff9b0cff301800.statahead_stats= statahead total: 143 statahead wrong: 2 agl total: 137 list_total: 31 fname_total: 112 hit_total: 4765 miss_total: 1230 mdc.lustre-MDT0000-mdc-ffff9b0cff301800.batch_stats= snapshot_time: 1787833849.370941757 secs.nsecs start_time: 1787833773.003967931 secs.nsecs elapsed_time: 76.366973826 secs.nsecs subreqs per batch batches %% cum %% 1: 1 2 2 2: 3 8 11 4: 1 2 13 8: 0 0 13 16: 25 69 83 32: 6 16 100 mdc.lustre-MDT0001-mdc-ffff9b0cff301800.batch_stats= snapshot_time: 1787833849.371484017 secs.nsecs start_time: 1787833773.004042426 secs.nsecs elapsed_time: 76.367441591 secs.nsecs subreqs per batch batches %% cum %% 1: 1 2 2 2: 0 0 2 4: 1 2 5 8: 1 2 8 16: 9 24 32 32: 25 67 100 - open/close 969 (time 1787833861.97 total 10.01 last 96.81) - open/close 1828 (time 1787833871.98 total 20.02 last 85.84) - open/close 2768 (time 1787833881.98 total 30.03 last 93.92) - open/close 3636 (time 1787833891.99 total 40.03 last 86.75) - open/close 4577 (time 1787833902.00 total 50.04 last 93.99) - open/close 5440 (time 1787833912.02 total 60.06 last 86.12) - open/close 6302 (time 1787833922.03 total 70.07 last 86.14) - open/close 7182 (time 1787833932.03 total 80.07 last 88.00) - open/close 8157 (time 1787833942.04 total 90.08 last 97.42) - open/close 9058 (time 1787833952.04 total 100.08 last 90.05) - open/close 9975 (time 1787833962.04 total 110.08 last 91.70) total: 10000 open/close in 110.38 seconds: 90.59 ops/second llite.lustre-ffff9b0cff301800.statahead_batch_max=0 llite.lustre-ffff9b0cff301800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9b0cff301800.stats=clear mdc.lustre-MDT0001-mdc-ffff9b0cff301800.stats=clear ls -l files in d123ad.sanity: 10001 real 0m55.749s user 0m0.320s sys 0m14.649s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff9b0cff301800.statahead_batch_max=32 llite.lustre-ffff9b0cff301800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9b0cff301800.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff9b0cff301800.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff9b0cff301800.stats=clear mdc.lustre-MDT0001-mdc-ffff9b0cff301800.stats=clear ls -l files in d123ad.sanity: 10001 real 1m32.917s user 0m0.304s sys 0m13.247s batched statahead RPC= hit count=9979 mdc.lustre-MDT0000-mdc-ffff9b0cff301800.batch_stats= snapshot_time: 1787834132.500751477 secs.nsecs start_time: 1787834028.456794900 secs.nsecs elapsed_time: 104.043956577 secs.nsecs subreqs per batch batches %% cum %% 1: 1 0 0 2: 0 0 0 4: 0 0 0 8: 1 0 0 16: 0 0 0 32: 313 99 100 mdc.lustre-MDT0001-mdc-ffff9b0cff301800.batch_stats= snapshot_time: 1787834132.500810161 secs.nsecs start_time: 1787834028.457157439 secs.nsecs elapsed_time: 104.043652722 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 batch: statahead_max=2048 statahead_batch_max=256 mdc.lustre-MDT0000-mdc-ffff9b0cff301800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9b0cff301800.batch_stats=0 llite.lustre-ffff9b0cff301800.statahead_max=2048 llite.lustre-ffff9b0cff301800.statahead_batch_max=256 kvm mdc.lustre-MDT0000-mdc-ffff9b0cff301800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9b0cff301800.batch_stats=0 striped dir -i1 -c2 -H crush2 /mnt/lustre/d123ad.sanity total: 100 open/close in 2.13 seconds: 47.03 ops/second llite.lustre-ffff9b0cff301800.statahead_max=0 101 real 0m2.266s user 0m0.006s sys 0m0.329s 'ls -l' 100 files without statahead: 2 sec llite.lustre-ffff9b0cff301800.statahead_max=2048 llite.lustre-ffff9b0cff301800.statahead_max=2048 101 real 0m0.968s user 0m0.004s sys 0m0.165s 'ls -l' 100 files with statahead: 1 sec statahead total: 2 statahead wrong: 0 agl total: 2 list_total: 2 fname_total: 0 hit_total: 10078 miss_total: 22 swrong=0 ewrong=0 total: 900 open/close in 9.27 seconds: 97.05 ops/second llite.lustre-ffff9b0cff301800.statahead_max=0 1001 real 0m18.436s user 0m0.032s sys 0m2.857s 'ls -l' 1000 files without statahead: 18 sec llite.lustre-ffff9b0cff301800.statahead_max=2048 llite.lustre-ffff9b0cff301800.statahead_max=2048 1001 real 0m7.045s user 0m0.022s sys 0m1.370s 'ls -l' 1000 files with statahead: 7 sec statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11077 miss_total: 23 swrong=0 ewrong=0 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 18 seconds llite.lustre-ffff9b0cff301800.statahead_stats= statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11077 miss_total: 23 mdc.lustre-MDT0000-mdc-ffff9b0cff301800.batch_stats= snapshot_time: 1787834463.740844378 secs.nsecs start_time: 1787834135.564066352 secs.nsecs elapsed_time: 328.176778026 secs.nsecs subreqs per batch batches %% cum %% 1: 1 16 16 2: 0 0 16 4: 2 33 50 8: 1 16 66 16: 0 0 66 32: 0 0 66 64: 0 0 66 128: 0 0 66 256: 2 33 100 mdc.lustre-MDT0001-mdc-ffff9b0cff301800.batch_stats= snapshot_time: 1787834463.740916433 secs.nsecs start_time: 1787834135.564207077 secs.nsecs elapsed_time: 328.176709356 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 2 25 25 8: 2 25 50 16: 0 0 50 32: 0 0 50 64: 1 12 62 128: 1 12 75 256: 2 25 100 - open/close 926 (time 1787834476.49 total 10.00 last 92.57) - open/close 1814 (time 1787834486.51 total 20.02 last 88.61) - open/close 2706 (time 1787834496.52 total 30.03 last 89.19) - open/close 3613 (time 1787834506.52 total 40.03 last 90.67) - open/close 4512 (time 1787834516.52 total 50.03 last 89.89) - open/close 5496 (time 1787834526.53 total 60.04 last 98.34) - open/close 6493 (time 1787834536.53 total 70.04 last 99.67) - open/close 7402 (time 1787834546.54 total 80.05 last 90.79) - open/close 8231 (time 1787834556.55 total 90.06 last 82.85) - open/close 9032 (time 1787834566.56 total 100.07 last 79.98) - open/close 9901 (time 1787834576.56 total 110.07 last 86.89) total: 10000 open/close in 111.14 seconds: 89.98 ops/second llite.lustre-ffff9b0cff301800.statahead_batch_max=0 llite.lustre-ffff9b0cff301800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9b0cff301800.stats=clear mdc.lustre-MDT0001-mdc-ffff9b0cff301800.stats=clear ls -l files in d123ad.sanity: 10001 real 0m56.225s user 0m0.286s sys 0m14.084s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff9b0cff301800.statahead_batch_max=256 llite.lustre-ffff9b0cff301800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9b0cff301800.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff9b0cff301800.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff9b0cff301800.stats=clear mdc.lustre-MDT0001-mdc-ffff9b0cff301800.stats=clear ls -l files in d123ad.sanity: 10001 real 0m48.703s user 0m0.276s sys 0m14.687s batched statahead RPC= hit count=9999 mdc.lustre-MDT0000-mdc-ffff9b0cff301800.batch_stats= snapshot_time: 1787834699.743594888 secs.nsecs start_time: 1787834641.405930382 secs.nsecs elapsed_time: 58.337664506 secs.nsecs subreqs per batch batches %% cum %% 1: 148 46 46 2: 40 12 58 4: 43 13 72 8: 23 7 79 16: 9 2 82 32: 18 5 88 64: 3 0 89 128: 1 0 89 256: 34 10 100 mdc.lustre-MDT0001-mdc-ffff9b0cff301800.batch_stats= snapshot_time: 1787834699.743664543 secs.nsecs start_time: 1787834641.406078251 secs.nsecs elapsed_time: 58.337586292 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second seq.cli-lustre-OST0000-super.width=65536 seq.cli-lustre-OST0001-super.width=65536 llite.lustre-ffff9b0cff301800.statahead_batch_max=64 llite.lustre-ffff9b0cff301800.statahead_max=128