== sanity test 123ad: Verify batching statahead works correctly ========================================================== 03:32:14 (1790926334) batch: statahead_max=32 statahead_batch_max=32 mdc.lustre-MDT0000-mdc-ffff9899bd0b9800.batch_stats=0 llite.lustre-ffff9899bd0b9800.statahead_max=32 llite.lustre-ffff9899bd0b9800.statahead_batch_max=32 seq.cli-lustre-OST0000-super.width=0x1ffffff seq.cli-lustre-OST0001-super.width=0x1ffffff kvm mdc.lustre-MDT0000-mdc-ffff9899bd0b9800.batch_stats=0 total: 100 open/close in 3.41 seconds: 29.32 ops/second llite.lustre-ffff9899bd0b9800.statahead_max=0 101 real 0m3.244s user 0m0.004s sys 0m0.350s 'ls -l' 100 files without statahead: 3 sec llite.lustre-ffff9899bd0b9800.statahead_max=32 llite.lustre-ffff9899bd0b9800.statahead_max=32 101 real 0m1.017s user 0m0.005s sys 0m0.173s 'ls -l' 100 files with statahead: 1 sec statahead total: 139 statahead wrong: 1 agl total: 132 list_total: 31 fname_total: 104 hit_total: 3602 miss_total: 265 swrong=1 ewrong=1 - open/close 605 (time 1790926365.39 total 10.03 last 60.35) total: 900 open/close in 15.90 seconds: 56.61 ops/second llite.lustre-ffff9899bd0b9800.statahead_max=0 1001 real 0m24.308s user 0m0.041s sys 0m3.062s 'ls -l' 1000 files without statahead: 25 sec llite.lustre-ffff9899bd0b9800.statahead_max=32 llite.lustre-ffff9899bd0b9800.statahead_max=32 1001 real 0m9.372s user 0m0.035s sys 0m1.497s 'ls -l' 1000 files with statahead: 9 sec statahead total: 140 statahead wrong: 1 agl total: 133 list_total: 32 fname_total: 104 hit_total: 4600 miss_total: 267 swrong=1 ewrong=1 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 23 seconds llite.lustre-ffff9899bd0b9800.statahead_stats= statahead total: 140 statahead wrong: 1 agl total: 133 list_total: 32 fname_total: 104 hit_total: 4600 miss_total: 267 mdc.lustre-MDT0000-mdc-ffff9899bd0b9800.batch_stats= snapshot_time: 1790926442.124665168 secs.nsecs start_time: 1790926340.679028469 secs.nsecs elapsed_time: 101.445636699 secs.nsecs subreqs per batch batches %% cum %% 1: 1 2 2 2: 0 0 2 4: 0 0 2 8: 3 7 10 16: 2 5 15 32: 34 85 100 - open/close 584 (time 1790926455.09 total 10.01 last 58.32) - open/close 1093 (time 1790926465.09 total 20.02 last 50.85) - open/close 1823 (time 1790926475.11 total 30.04 last 72.88) - open/close 2453 (time 1790926485.12 total 40.05 last 62.93) - open/close 3022 (time 1790926495.12 total 50.05 last 56.89) - open/close 3622 (time 1790926505.13 total 60.06 last 59.96) - open/close 4283 (time 1790926515.14 total 70.06 last 66.07) - open/close 4980 (time 1790926525.19 total 80.12 last 69.34) - open/close 5588 (time 1790926535.20 total 90.13 last 60.75) - open/close 6045 (time 1790926545.20 total 100.13 last 45.69) - open/close 6475 (time 1790926555.21 total 110.13 last 42.96) - open/close 6972 (time 1790926565.21 total 120.13 last 49.70) - open/close 7568 (time 1790926575.21 total 130.14 last 59.56) - open/close 8138 (time 1790926585.22 total 140.15 last 56.98) - open/close 8724 (time 1790926595.23 total 150.15 last 58.55) - open/close 9344 (time 1790926605.23 total 160.16 last 61.96) - open/close 9926 (time 1790926615.25 total 170.18 last 58.11) total: 10000 open/close in 171.38 seconds: 58.35 ops/second llite.lustre-ffff9899bd0b9800.statahead_batch_max=0 llite.lustre-ffff9899bd0b9800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9899bd0b9800.stats=clear ls -l files in d123ad.sanity: 10001 real 1m2.017s user 0m0.358s sys 0m15.913s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff9899bd0b9800.statahead_batch_max=32 llite.lustre-ffff9899bd0b9800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9899bd0b9800.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff9899bd0b9800.stats=clear ls -l files in d123ad.sanity: 10001 real 1m44.316s user 0m0.331s sys 0m14.400s batched statahead RPC= hit count=9977 mdc.lustre-MDT0000-mdc-ffff9899bd0b9800.batch_stats= snapshot_time: 1790926805.056460164 secs.nsecs start_time: 1790926687.941735684 secs.nsecs elapsed_time: 117.114724480 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 1 0 0 8: 1 0 0 16: 1 0 0 32: 313 99 100 batch: statahead_max=2048 statahead_batch_max=256 mdc.lustre-MDT0000-mdc-ffff9899bd0b9800.batch_stats=0 llite.lustre-ffff9899bd0b9800.statahead_max=2048 llite.lustre-ffff9899bd0b9800.statahead_batch_max=256 kvm mdc.lustre-MDT0000-mdc-ffff9899bd0b9800.batch_stats=0 total: 100 open/close in 3.27 seconds: 30.60 ops/second llite.lustre-ffff9899bd0b9800.statahead_max=0 101 real 0m3.120s user 0m0.005s sys 0m0.372s 'ls -l' 100 files without statahead: 3 sec llite.lustre-ffff9899bd0b9800.statahead_max=2048 llite.lustre-ffff9899bd0b9800.statahead_max=2048 101 real 0m0.995s user 0m0.006s sys 0m0.164s 'ls -l' 100 files with statahead: 1 sec statahead total: 2 statahead wrong: 0 agl total: 2 list_total: 2 fname_total: 0 hit_total: 10076 miss_total: 24 swrong=0 ewrong=0 - open/close 508 (time 1790927176.83 total 10.01 last 50.73) total: 900 open/close in 18.38 seconds: 48.97 ops/second llite.lustre-ffff9899bd0b9800.statahead_max=0 1001 real 0m26.067s user 0m0.035s sys 0m3.185s 'ls -l' 1000 files without statahead: 26 sec llite.lustre-ffff9899bd0b9800.statahead_max=2048 llite.lustre-ffff9899bd0b9800.statahead_max=2048 1001 real 0m7.640s user 0m0.036s sys 0m1.524s 'ls -l' 1000 files with statahead: 8 sec statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11075 miss_total: 25 swrong=0 ewrong=0 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 26 seconds llite.lustre-ffff9899bd0b9800.statahead_stats= statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11075 miss_total: 25 mdc.lustre-MDT0000-mdc-ffff9899bd0b9800.batch_stats= snapshot_time: 1790927260.741305067 secs.nsecs start_time: 1790926808.581261121 secs.nsecs elapsed_time: 452.160043946 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 0 0 0 8: 2 20 20 16: 2 20 40 32: 1 10 50 64: 0 0 50 128: 1 10 60 256: 4 40 100 - open/close 547 (time 1790927274.04 total 10.02 last 54.62) - open/close 1120 (time 1790927284.05 total 20.03 last 57.24) - open/close 1666 (time 1790927294.05 total 30.03 last 54.60) - open/close 2202 (time 1790927304.07 total 40.04 last 53.51) - open/close 2731 (time 1790927314.08 total 50.05 last 52.85) - open/close 3348 (time 1790927324.08 total 60.06 last 61.69) - open/close 3954 (time 1790927334.08 total 70.06 last 60.60) - open/close 4578 (time 1790927344.09 total 80.06 last 62.38) - open/close 5227 (time 1790927354.09 total 90.06 last 64.87) - open/close 5843 (time 1790927364.09 total 100.07 last 61.59) - open/close 6466 (time 1790927374.10 total 110.07 last 62.25) - open/close 7058 (time 1790927384.12 total 120.10 last 59.07) - open/close 7653 (time 1790927394.12 total 130.10 last 59.50) - open/close 8277 (time 1790927404.13 total 140.10 last 62.38) - open/close 8932 (time 1790927414.13 total 150.10 last 65.46) - open/close 9439 (time 1790927424.14 total 160.11 last 50.66) - open/close 9906 (time 1790927434.15 total 170.12 last 46.68) total: 10000 open/close in 172.25 seconds: 58.06 ops/second llite.lustre-ffff9899bd0b9800.statahead_batch_max=0 llite.lustre-ffff9899bd0b9800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9899bd0b9800.stats=clear ls -l files in d123ad.sanity: 10001 real 1m17.302s user 0m0.326s sys 0m15.549s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff9899bd0b9800.statahead_batch_max=256 llite.lustre-ffff9899bd0b9800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9899bd0b9800.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff9899bd0b9800.stats=clear ls -l files in d123ad.sanity: 10001 real 0m49.156s user 0m0.302s sys 0m15.948s batched statahead RPC= hit count=9999 mdc.lustre-MDT0000-mdc-ffff9899bd0b9800.batch_stats= snapshot_time: 1790927584.581673752 secs.nsecs start_time: 1790927524.017352829 secs.nsecs elapsed_time: 60.564320923 secs.nsecs subreqs per batch batches %% cum %% 1: 133 48 48 2: 33 12 60 4: 30 10 71 8: 13 4 76 16: 9 3 79 32: 14 5 84 64: 6 2 87 128: 0 0 87 256: 35 12 100 unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second seq.cli-lustre-OST0000-super.width=65536 seq.cli-lustre-OST0001-super.width=65536 llite.lustre-ffff9899bd0b9800.statahead_batch_max=64 llite.lustre-ffff9899bd0b9800.statahead_max=128