== sanity test 123ad: Verify batching statahead works correctly ========================================================== 16:48:42 (1786654122) batch: statahead_max=32 statahead_batch_max=32 mdc.lustre-MDT0000-mdc-ffff963047d1c000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff963047d1c000.batch_stats=0 llite.lustre-ffff963047d1c000.statahead_max=32 llite.lustre-ffff963047d1c000.statahead_batch_max=32 seq.cli-lustre-OST0000-super.width=0x1ffffff seq.cli-lustre-OST0001-super.width=0x1ffffff kvm mdc.lustre-MDT0000-mdc-ffff963047d1c000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff963047d1c000.batch_stats=0 striped dir -i1 -c2 -H crush2 /mnt/lustre/d123ad.sanity total: 100 open/close in 3.13 seconds: 31.90 ops/second llite.lustre-ffff963047d1c000.statahead_max=0 101 real 0m2.987s user 0m0.005s sys 0m0.423s 'ls -l' 100 files without statahead: 3 sec llite.lustre-ffff963047d1c000.statahead_max=32 llite.lustre-ffff963047d1c000.statahead_max=32 101 real 0m1.447s user 0m0.006s sys 0m0.225s 'ls -l' 100 files with statahead: 2 sec statahead total: 120 statahead wrong: 2 agl total: 113 list_total: 31 fname_total: 80 hit_total: 3581 miss_total: 1436 swrong=2 ewrong=2 - open/close 726 (time 1786654153.66 total 10.01 last 72.50) total: 900 open/close in 12.82 seconds: 70.19 ops/second llite.lustre-ffff963047d1c000.statahead_max=0 1001 real 0m24.686s user 0m0.044s sys 0m3.830s 'ls -l' 1000 files without statahead: 25 sec llite.lustre-ffff963047d1c000.statahead_max=32 llite.lustre-ffff963047d1c000.statahead_max=32 1001 real 0m11.488s user 0m0.036s sys 0m1.764s 'ls -l' 1000 files with statahead: 12 sec statahead total: 121 statahead wrong: 2 agl total: 114 list_total: 32 fname_total: 80 hit_total: 4580 miss_total: 1437 swrong=2 ewrong=2 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 30 seconds llite.lustre-ffff963047d1c000.statahead_stats= statahead total: 121 statahead wrong: 2 agl total: 114 list_total: 32 fname_total: 80 hit_total: 4580 miss_total: 1437 mdc.lustre-MDT0000-mdc-ffff963047d1c000.batch_stats= snapshot_time: 1786654238.718869614 secs.nsecs start_time: 1786654128.220093220 secs.nsecs elapsed_time: 110.498776394 secs.nsecs subreqs per batch batches %% cum %% 1: 2 5 5 2: 1 2 8 4: 3 8 16 8: 2 5 21 16: 24 64 86 32: 5 13 100 mdc.lustre-MDT0001-mdc-ffff963047d1c000.batch_stats= snapshot_time: 1786654238.718946639 secs.nsecs start_time: 1786654128.221382922 secs.nsecs elapsed_time: 110.497563717 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 2 5 5 8: 2 5 10 16: 13 33 43 32: 22 56 100 - open/close 631 (time 1786654251.91 total 10.00 last 63.09) - open/close 1386 (time 1786654261.91 total 20.01 last 75.46) - open/close 2095 (time 1786654271.93 total 30.02 last 70.78) - open/close 2828 (time 1786654281.94 total 40.03 last 73.23) - open/close 3520 (time 1786654291.95 total 50.05 last 69.09) - open/close 4239 (time 1786654301.95 total 60.05 last 71.89) - open/close 4964 (time 1786654311.96 total 70.05 last 72.49) - open/close 5681 (time 1786654321.96 total 80.05 last 71.68) - open/close 6359 (time 1786654331.96 total 90.05 last 67.79) - open/close 7028 (time 1786654341.96 total 100.06 last 66.86) - open/close 7679 (time 1786654351.97 total 110.07 last 65.06) - open/close 8346 (time 1786654361.98 total 120.08 last 66.64) - open/close 8948 (time 1786654371.98 total 130.08 last 60.17) - open/close 9599 (time 1786654381.99 total 140.08 last 65.08) total: 10000 open/close in 146.06 seconds: 68.47 ops/second llite.lustre-ffff963047d1c000.statahead_batch_max=0 llite.lustre-ffff963047d1c000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff963047d1c000.stats=clear mdc.lustre-MDT0001-mdc-ffff963047d1c000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m17.884s user 0m0.395s sys 0m19.458s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff963047d1c000.statahead_batch_max=32 llite.lustre-ffff963047d1c000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff963047d1c000.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff963047d1c000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff963047d1c000.stats=clear mdc.lustre-MDT0001-mdc-ffff963047d1c000.stats=clear ls -l files in d123ad.sanity: 10001 real 2m2.674s user 0m0.387s sys 0m18.267s batched statahead RPC= hit count=9992 mdc.lustre-MDT0000-mdc-ffff963047d1c000.batch_stats= snapshot_time: 1786654614.884411175 secs.nsecs start_time: 1786654476.510807700 secs.nsecs elapsed_time: 138.373603475 secs.nsecs subreqs per batch batches %% cum %% 1: 1 0 0 2: 0 0 0 4: 0 0 0 8: 1 0 0 16: 1 0 0 32: 313 99 100 mdc.lustre-MDT0001-mdc-ffff963047d1c000.batch_stats= snapshot_time: 1786654614.884494173 secs.nsecs start_time: 1786654476.510944346 secs.nsecs elapsed_time: 138.373549827 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 batch: statahead_max=2048 statahead_batch_max=256 mdc.lustre-MDT0000-mdc-ffff963047d1c000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff963047d1c000.batch_stats=0 llite.lustre-ffff963047d1c000.statahead_max=2048 llite.lustre-ffff963047d1c000.statahead_batch_max=256 kvm mdc.lustre-MDT0000-mdc-ffff963047d1c000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff963047d1c000.batch_stats=0 striped dir -i1 -c2 -H all_char /mnt/lustre/d123ad.sanity total: 100 open/close in 3.28 seconds: 30.53 ops/second llite.lustre-ffff963047d1c000.statahead_max=0 101 real 0m2.888s user 0m0.008s sys 0m0.391s 'ls -l' 100 files without statahead: 3 sec llite.lustre-ffff963047d1c000.statahead_max=2048 llite.lustre-ffff963047d1c000.statahead_max=2048 101 real 0m0.974s user 0m0.004s sys 0m0.184s 'ls -l' 100 files with statahead: 1 sec statahead total: 2 statahead wrong: 0 agl total: 2 list_total: 2 fname_total: 0 hit_total: 10091 miss_total: 9 swrong=0 ewrong=0 - open/close 649 (time 1786654971.57 total 10.01 last 64.81) total: 900 open/close in 13.82 seconds: 65.14 ops/second llite.lustre-ffff963047d1c000.statahead_max=0 1001 real 0m31.233s user 0m0.042s sys 0m3.717s 'ls -l' 1000 files without statahead: 31 sec llite.lustre-ffff963047d1c000.statahead_max=2048 llite.lustre-ffff963047d1c000.statahead_max=2048 1001 real 0m5.371s user 0m0.022s sys 0m1.581s 'ls -l' 1000 files with statahead: 5 sec statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11090 miss_total: 10 swrong=0 ewrong=0 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 24 seconds llite.lustre-ffff963047d1c000.statahead_stats= statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11090 miss_total: 10 mdc.lustre-MDT0000-mdc-ffff963047d1c000.batch_stats= snapshot_time: 1786655049.749953162 secs.nsecs start_time: 1786654617.408690350 secs.nsecs elapsed_time: 432.341262812 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 0 0 0 8: 5 55 55 16: 1 11 66 32: 1 11 77 64: 0 0 77 128: 0 0 77 256: 2 22 100 mdc.lustre-MDT0001-mdc-ffff963047d1c000.batch_stats= snapshot_time: 1786655049.750741964 secs.nsecs start_time: 1786654617.409193769 secs.nsecs elapsed_time: 432.341548195 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 1 10 10 4: 2 20 30 8: 1 10 40 16: 3 30 70 32: 1 10 80 64: 0 0 80 128: 0 0 80 256: 2 20 100 - open/close 545 (time 1786655063.34 total 10.02 last 54.40) - open/close 978 (time 1786655073.34 total 20.02 last 43.29) - open/close 1608 (time 1786655083.35 total 30.03 last 62.96) - open/close 2313 (time 1786655093.35 total 40.03 last 70.47) - open/close 2940 (time 1786655103.38 total 50.05 last 62.55) - open/close 3605 (time 1786655113.38 total 60.06 last 66.44) - open/close 4225 (time 1786655123.40 total 70.08 last 61.91) - open/close 4804 (time 1786655133.41 total 80.09 last 57.83) - open/close 5396 (time 1786655143.42 total 90.10 last 59.15) - open/close 6029 (time 1786655153.43 total 100.11 last 63.25) - open/close 6673 (time 1786655163.44 total 110.11 last 64.34) - open/close 7318 (time 1786655173.44 total 120.12 last 64.49) - open/close 8009 (time 1786655183.44 total 130.12 last 69.10) - open/close 8692 (time 1786655193.44 total 140.12 last 68.27) - open/close 9431 (time 1786655203.45 total 150.13 last 73.87) total: 10000 open/close in 157.45 seconds: 63.51 ops/second llite.lustre-ffff963047d1c000.statahead_batch_max=0 llite.lustre-ffff963047d1c000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff963047d1c000.stats=clear mdc.lustre-MDT0001-mdc-ffff963047d1c000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m11.640s user 0m0.356s sys 0m18.186s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff963047d1c000.statahead_batch_max=256 llite.lustre-ffff963047d1c000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff963047d1c000.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff963047d1c000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff963047d1c000.stats=clear mdc.lustre-MDT0001-mdc-ffff963047d1c000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m7.633s user 0m0.361s sys 0m18.193s batched statahead RPC= hit count=9999 mdc.lustre-MDT0000-mdc-ffff963047d1c000.batch_stats= snapshot_time: 1786655380.082077432 secs.nsecs start_time: 1786655297.008398906 secs.nsecs elapsed_time: 83.073678526 secs.nsecs subreqs per batch batches %% cum %% 1: 336 58 58 2: 78 13 71 4: 60 10 81 8: 33 5 87 16: 22 3 91 32: 11 1 93 64: 5 0 94 128: 1 0 94 256: 33 5 100 mdc.lustre-MDT0001-mdc-ffff963047d1c000.batch_stats= snapshot_time: 1786655380.082141306 secs.nsecs start_time: 1786655297.008649504 secs.nsecs elapsed_time: 83.073491802 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second seq.cli-lustre-OST0000-super.width=65536 seq.cli-lustre-OST0001-super.width=65536 llite.lustre-ffff963047d1c000.statahead_batch_max=64 llite.lustre-ffff963047d1c000.statahead_max=128