== sanity test 123ad: Verify batching statahead works correctly ========================================================== 05:42:36 (1786009356) batch: statahead_max=32 statahead_batch_max=32 mdc.lustre-MDT0000-mdc-ffff9829be55e000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9829be55e000.batch_stats=0 llite.lustre-ffff9829be55e000.statahead_max=32 llite.lustre-ffff9829be55e000.statahead_batch_max=32 seq.cli-lustre-OST0000-super.width=0x1ffffff seq.cli-lustre-OST0001-super.width=0x1ffffff kvm mdc.lustre-MDT0000-mdc-ffff9829be55e000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9829be55e000.batch_stats=0 striped dir -i1 -c2 -H all_char /mnt/lustre/d123ad.sanity total: 100 open/close in 4.24 seconds: 23.59 ops/second llite.lustre-ffff9829be55e000.statahead_max=0 101 real 0m3.942s user 0m0.005s sys 0m0.320s 'ls -l' 100 files without statahead: 4 sec llite.lustre-ffff9829be55e000.statahead_max=32 llite.lustre-ffff9829be55e000.statahead_max=32 101 real 0m1.899s user 0m0.005s sys 0m0.224s 'ls -l' 100 files with statahead: 1 sec statahead total: 155 statahead wrong: 2 agl total: 148 list_total: 31 fname_total: 124 hit_total: 3581 miss_total: 1526 swrong=2 ewrong=2 - open/close 569 (time 1786009392.98 total 10.01 last 56.82) total: 900 open/close in 16.53 seconds: 54.44 ops/second llite.lustre-ffff9829be55e000.statahead_max=0 1001 real 0m30.059s user 0m0.025s sys 0m2.926s 'ls -l' 1000 files without statahead: 30 sec llite.lustre-ffff9829be55e000.statahead_max=32 llite.lustre-ffff9829be55e000.statahead_max=32 1001 real 0m12.679s user 0m0.023s sys 0m1.358s 'ls -l' 1000 files with statahead: 13 sec statahead total: 156 statahead wrong: 2 agl total: 149 list_total: 32 fname_total: 124 hit_total: 4576 miss_total: 1531 swrong=2 ewrong=2 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 29 seconds llite.lustre-ffff9829be55e000.statahead_stats= statahead total: 156 statahead wrong: 2 agl total: 149 list_total: 32 fname_total: 124 hit_total: 4576 miss_total: 1531 mdc.lustre-MDT0000-mdc-ffff9829be55e000.batch_stats= snapshot_time: 1786009492.849000790 secs.nsecs start_time: 1786009362.610669368 secs.nsecs elapsed_time: 130.238331422 secs.nsecs subreqs per batch batches %% cum %% 1: 1 2 2 2: 0 0 2 4: 2 5 7 8: 3 7 15 16: 24 60 75 32: 10 25 100 mdc.lustre-MDT0001-mdc-ffff9829be55e000.batch_stats= snapshot_time: 1786009492.849059948 secs.nsecs start_time: 1786009362.611260648 secs.nsecs elapsed_time: 130.237799300 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 5 12 12 8: 2 5 17 16: 15 38 56 32: 17 43 100 - open/close 259 (time 1786009509.75 total 10.00 last 25.90) - open/close 941 (time 1786009519.76 total 20.01 last 68.14) - open/close 1482 (time 1786009529.77 total 30.02 last 54.07) - open/close 2059 (time 1786009539.78 total 40.03 last 57.64) - open/close 2520 (time 1786009549.79 total 50.04 last 46.06) - open/close 2953 (time 1786009559.80 total 60.04 last 43.26) - open/close 3408 (time 1786009569.81 total 70.06 last 45.45) - open/close 3799 (time 1786009579.82 total 80.07 last 39.06) - open/close 4133 (time 1786009589.91 total 90.16 last 33.09) - open/close 4642 (time 1786009599.92 total 100.17 last 50.86) - open/close 5243 (time 1786009609.97 total 110.22 last 59.79) - open/close 5870 (time 1786009619.97 total 120.22 last 62.67) - open/close 6501 (time 1786009629.99 total 130.24 last 63.01) - open/close 7020 (time 1786009639.99 total 140.24 last 51.88) - open/close 7636 (time 1786009650.00 total 150.24 last 61.57) - open/close 8158 (time 1786009660.00 total 160.25 last 52.16) - open/close 8750 (time 1786009670.02 total 170.27 last 59.12) - open/close 9160 (time 1786009680.03 total 180.27 last 40.96) - open/close 9704 (time 1786009690.03 total 190.28 last 54.38) total: 10000 open/close in 195.04 seconds: 51.27 ops/second llite.lustre-ffff9829be55e000.statahead_batch_max=0 llite.lustre-ffff9829be55e000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9829be55e000.stats=clear mdc.lustre-MDT0001-mdc-ffff9829be55e000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m28.700s user 0m0.268s sys 0m14.298s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff9829be55e000.statahead_batch_max=32 llite.lustre-ffff9829be55e000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9829be55e000.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff9829be55e000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff9829be55e000.stats=clear mdc.lustre-MDT0001-mdc-ffff9829be55e000.stats=clear ls -l files in d123ad.sanity: 10001 real 2m20.865s user 0m0.247s sys 0m14.019s batched statahead RPC= hit count=9987 mdc.lustre-MDT0000-mdc-ffff9829be55e000.batch_stats= snapshot_time: 1786009962.272556577 secs.nsecs start_time: 1786009797.434852671 secs.nsecs elapsed_time: 164.837703906 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 0 0 0 8: 1 0 0 16: 1 0 0 32: 313 99 100 mdc.lustre-MDT0001-mdc-ffff9829be55e000.batch_stats= snapshot_time: 1786009962.272616319 secs.nsecs start_time: 1786009797.434932247 secs.nsecs elapsed_time: 164.837684072 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 batch: statahead_max=2048 statahead_batch_max=256 mdc.lustre-MDT0000-mdc-ffff9829be55e000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9829be55e000.batch_stats=0 llite.lustre-ffff9829be55e000.statahead_max=2048 llite.lustre-ffff9829be55e000.statahead_batch_max=256 kvm mdc.lustre-MDT0000-mdc-ffff9829be55e000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9829be55e000.batch_stats=0 striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d123ad.sanity total: 100 open/close in 3.36 seconds: 29.79 ops/second llite.lustre-ffff9829be55e000.statahead_max=0 101 real 0m3.775s user 0m0.000s sys 0m0.313s 'ls -l' 100 files without statahead: 4 sec llite.lustre-ffff9829be55e000.statahead_max=2048 llite.lustre-ffff9829be55e000.statahead_max=2048 101 real 0m1.440s user 0m0.005s sys 0m0.176s 'ls -l' 100 files with statahead: 1 sec statahead total: 2 statahead wrong: 0 agl total: 2 list_total: 2 fname_total: 0 hit_total: 10086 miss_total: 14 swrong=0 ewrong=0 - open/close 633 (time 1786010387.13 total 10.01 last 63.24) total: 900 open/close in 14.61 seconds: 61.61 ops/second llite.lustre-ffff9829be55e000.statahead_max=0 1001 real 0m32.287s user 0m0.030s sys 0m3.075s 'ls -l' 1000 files without statahead: 32 sec llite.lustre-ffff9829be55e000.statahead_max=2048 llite.lustre-ffff9829be55e000.statahead_max=2048 1001 real 0m8.474s user 0m0.024s sys 0m1.362s 'ls -l' 1000 files with statahead: 9 sec statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11085 miss_total: 15 swrong=0 ewrong=0 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 30 seconds llite.lustre-ffff9829be55e000.statahead_stats= statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11085 miss_total: 15 mdc.lustre-MDT0000-mdc-ffff9829be55e000.batch_stats= snapshot_time: 1786010481.707269814 secs.nsecs start_time: 1786009965.469914959 secs.nsecs elapsed_time: 516.237354855 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 3 37 37 8: 1 12 50 16: 1 12 62 32: 0 0 62 64: 1 12 75 128: 0 0 75 256: 2 25 100 mdc.lustre-MDT0001-mdc-ffff9829be55e000.batch_stats= snapshot_time: 1786010481.707331925 secs.nsecs start_time: 1786009965.470065021 secs.nsecs elapsed_time: 516.237266904 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 2 25 25 8: 3 37 62 16: 0 0 62 32: 0 0 62 64: 1 12 75 128: 0 0 75 256: 2 25 100 - open/close 698 (time 1786010495.55 total 10.02 last 69.68) - open/close 1398 (time 1786010505.55 total 20.02 last 69.96) - open/close 2227 (time 1786010515.56 total 30.03 last 82.87) - open/close 3091 (time 1786010525.56 total 40.04 last 86.32) - open/close 3691 (time 1786010535.58 total 50.05 last 59.90) - open/close 4197 (time 1786010545.60 total 60.07 last 50.50) - open/close 4585 (time 1786010555.61 total 70.08 last 38.75) - open/close 5146 (time 1786010565.62 total 80.09 last 56.08) - open/close 5964 (time 1786010575.62 total 90.09 last 81.78) - open/close 6689 (time 1786010585.64 total 100.11 last 72.34) - open/close 7242 (time 1786010595.66 total 110.13 last 55.22) - open/close 7817 (time 1786010605.66 total 120.13 last 57.48) - open/close 8404 (time 1786010615.66 total 130.14 last 58.67) - open/close 9235 (time 1786010625.67 total 140.15 last 83.02) - open/close 9875 (time 1786010635.68 total 150.15 last 63.96) total: 10000 open/close in 151.84 seconds: 65.86 ops/second llite.lustre-ffff9829be55e000.statahead_batch_max=0 llite.lustre-ffff9829be55e000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9829be55e000.stats=clear mdc.lustre-MDT0001-mdc-ffff9829be55e000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m22.630s user 0m0.237s sys 0m14.982s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff9829be55e000.statahead_batch_max=256 llite.lustre-ffff9829be55e000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9829be55e000.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff9829be55e000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff9829be55e000.stats=clear mdc.lustre-MDT0001-mdc-ffff9829be55e000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m15.720s user 0m0.262s sys 0m14.867s batched statahead RPC= hit count=9999 mdc.lustre-MDT0000-mdc-ffff9829be55e000.batch_stats= snapshot_time: 1786010831.884770329 secs.nsecs start_time: 1786010736.477454123 secs.nsecs elapsed_time: 95.407316206 secs.nsecs subreqs per batch batches %% cum %% 1: 150 46 46 2: 37 11 57 4: 41 12 70 8: 26 8 78 16: 9 2 81 32: 22 6 88 64: 3 0 89 128: 1 0 89 256: 34 10 100 mdc.lustre-MDT0001-mdc-ffff9829be55e000.batch_stats= snapshot_time: 1786010831.884832474 secs.nsecs start_time: 1786010736.477523904 secs.nsecs elapsed_time: 95.407308570 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second seq.cli-lustre-OST0000-super.width=65536 seq.cli-lustre-OST0001-super.width=65536 llite.lustre-ffff9829be55e000.statahead_batch_max=64 llite.lustre-ffff9829be55e000.statahead_max=128