== sanity test 123ad: Verify batching statahead works correctly ========================================================== 04:43:39 (1779698619) batch: statahead_max=32 statahead_batch_max=32 mdc.lustre-MDT0000-mdc-ffff8ff893205800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff8ff893205800.batch_stats=0 llite.lustre-ffff8ff893205800.statahead_max=32 llite.lustre-ffff8ff893205800.statahead_batch_max=32 seq.cli-lustre-OST0000-super.width=0x1ffffff seq.cli-lustre-OST0001-super.width=0x1ffffff kvm mdc.lustre-MDT0000-mdc-ffff8ff893205800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff8ff893205800.batch_stats=0 striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d123ad.sanity total: 100 open/close in 2.50 seconds: 39.94 ops/second llite.lustre-ffff8ff893205800.statahead_max=0 101 real 0m2.542s user 0m0.008s sys 0m0.327s 'ls -l' 100 files without statahead: 3 sec llite.lustre-ffff8ff893205800.statahead_max=32 llite.lustre-ffff8ff893205800.statahead_max=32 101 real 0m0.996s user 0m0.002s sys 0m0.191s 'ls -l' 100 files with statahead: 1 sec statahead total: 134 statahead wrong: 2 agl total: 128 list_total: 30 fname_total: 102 hit_total: 3551 miss_total: 1184 swrong=2 ewrong=2 total: 900 open/close in 9.88 seconds: 91.10 ops/second llite.lustre-ffff8ff893205800.statahead_max=0 1001 real 0m20.291s user 0m0.030s sys 0m3.011s 'ls -l' 1000 files without statahead: 20 sec llite.lustre-ffff8ff893205800.statahead_max=32 llite.lustre-ffff8ff893205800.statahead_max=32 1001 real 0m11.381s user 0m0.034s sys 0m1.526s 'ls -l' 1000 files with statahead: 11 sec statahead total: 135 statahead wrong: 2 agl total: 129 list_total: 31 fname_total: 102 hit_total: 4550 miss_total: 1185 swrong=2 ewrong=2 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 25 seconds llite.lustre-ffff8ff893205800.statahead_stats= statahead total: 135 statahead wrong: 2 agl total: 129 list_total: 31 fname_total: 102 hit_total: 4550 miss_total: 1185 mdc.lustre-MDT0000-mdc-ffff8ff893205800.batch_stats= snapshot_time: 1779698716.946740497 secs.nsecs start_time: 1779698624.346355439 secs.nsecs elapsed_time: 92.600385058 secs.nsecs subreqs per batch batches % cum % 1: 0 0 0 2: 2 5 5 4: 2 5 10 8: 1 2 12 16: 22 56 69 32: 12 30 100 mdc.lustre-MDT0001-mdc-ffff8ff893205800.batch_stats= snapshot_time: 1779698716.946812171 secs.nsecs start_time: 1779698624.346439909 secs.nsecs elapsed_time: 92.600372262 secs.nsecs subreqs per batch batches % cum % 1: 0 0 0 2: 0 0 0 4: 2 5 5 8: 5 12 17 16: 18 46 64 32: 14 35 100 - open/close 837 (time 1779698729.97 total 10.01 last 83.64) - open/close 1731 (time 1779698739.97 total 20.01 last 89.38) - open/close 2504 (time 1779698749.97 total 30.01 last 77.29) - open/close 3406 (time 1779698759.97 total 40.01 last 90.16) - open/close 4146 (time 1779698769.97 total 50.02 last 73.99) - open/close 5086 (time 1779698779.98 total 60.02 last 93.94) - open/close 5925 (time 1779698790.00 total 70.04 last 83.77) - open/close 6798 (time 1779698800.00 total 80.04 last 87.27) - open/close 7566 (time 1779698810.00 total 90.05 last 76.76) - open/close 8288 (time 1779698820.01 total 100.05 last 72.18) - open/close 9131 (time 1779698830.02 total 110.06 last 84.19) - open/close 9851 (time 1779698840.02 total 120.06 last 71.99) total: 10000 open/close in 122.05 seconds: 81.94 ops/second llite.lustre-ffff8ff893205800.statahead_batch_max=0 llite.lustre-ffff8ff893205800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff8ff893205800.stats=clear mdc.lustre-MDT0001-mdc-ffff8ff893205800.stats=clear ls -l files in d123ad.sanity: 10001 real 1m2.072s user 0m0.322s sys 0m16.382s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff8ff893205800.statahead_batch_max=32 llite.lustre-ffff8ff893205800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff8ff893205800.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff8ff893205800.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff8ff893205800.stats=clear mdc.lustre-MDT0001-mdc-ffff8ff893205800.stats=clear ls -l files in d123ad.sanity: 10001 real 1m53.757s user 0m0.304s sys 0m15.205s batched statahead RPC= hit count=9978 mdc.lustre-MDT0000-mdc-ffff8ff893205800.batch_stats= snapshot_time: 1779699041.226729618 secs.nsecs start_time: 1779698913.244021835 secs.nsecs elapsed_time: 127.982707783 secs.nsecs subreqs per batch batches % cum % 1: 0 0 0 2: 1 0 0 4: 0 0 0 8: 1 0 0 16: 1 0 0 32: 313 99 100 mdc.lustre-MDT0001-mdc-ffff8ff893205800.batch_stats= snapshot_time: 1779699041.226793868 secs.nsecs start_time: 1779698913.244177650 secs.nsecs elapsed_time: 127.982616218 secs.nsecs subreqs per batch batches % cum % 1: 0 0 0 batch: statahead_max=2048 statahead_batch_max=256 mdc.lustre-MDT0000-mdc-ffff8ff893205800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff8ff893205800.batch_stats=0 llite.lustre-ffff8ff893205800.statahead_max=2048 llite.lustre-ffff8ff893205800.statahead_batch_max=256 kvm mdc.lustre-MDT0000-mdc-ffff8ff893205800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff8ff893205800.batch_stats=0 striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d123ad.sanity total: 100 open/close in 2.30 seconds: 43.50 ops/second llite.lustre-ffff8ff893205800.statahead_max=0 101 real 0m2.290s user 0m0.008s sys 0m0.338s 'ls -l' 100 files without statahead: 3 sec llite.lustre-ffff8ff893205800.statahead_max=2048 llite.lustre-ffff8ff893205800.statahead_max=2048 101 real 0m0.808s user 0m0.003s sys 0m0.169s 'ls -l' 100 files with statahead: 1 sec statahead total: 2 statahead wrong: 0 agl total: 2 list_total: 2 fname_total: 0 hit_total: 10077 miss_total: 23 swrong=0 ewrong=0 total: 900 open/close in 9.88 seconds: 91.13 ops/second llite.lustre-ffff8ff893205800.statahead_max=0 1001 real 0m20.791s user 0m0.034s sys 0m2.999s 'ls -l' 1000 files without statahead: 21 sec llite.lustre-ffff8ff893205800.statahead_max=2048 llite.lustre-ffff8ff893205800.statahead_max=2048 1001 real 0m6.311s user 0m0.031s sys 0m1.732s 'ls -l' 1000 files with statahead: 6 sec statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11076 miss_total: 24 swrong=0 ewrong=0 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 19 seconds llite.lustre-ffff8ff893205800.statahead_stats= statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11076 miss_total: 24 mdc.lustre-MDT0000-mdc-ffff8ff893205800.batch_stats= snapshot_time: 1779699395.304405264 secs.nsecs start_time: 1779699043.796993015 secs.nsecs elapsed_time: 351.507412249 secs.nsecs subreqs per batch batches % cum % 1: 0 0 0 2: 1 12 12 4: 2 25 37 8: 1 12 50 16: 1 12 62 32: 0 0 62 64: 1 12 75 128: 0 0 75 256: 2 25 100 mdc.lustre-MDT0001-mdc-ffff8ff893205800.batch_stats= snapshot_time: 1779699395.308819775 secs.nsecs start_time: 1779699043.797187789 secs.nsecs elapsed_time: 351.511631986 secs.nsecs subreqs per batch batches % cum % 1: 0 0 0 2: 0 0 0 4: 1 12 12 8: 4 50 62 16: 0 0 62 32: 0 0 62 64: 1 12 75 128: 0 0 75 256: 2 25 100 - open/close 900 (time 1779699407.78 total 10.01 last 89.92) - open/close 1867 (time 1779699417.78 total 20.01 last 96.64) - open/close 2862 (time 1779699427.78 total 30.01 last 99.49) - open/close 3684 (time 1779699437.79 total 40.02 last 82.17) - open/close 4565 (time 1779699447.79 total 50.02 last 88.04) - open/close 5395 (time 1779699457.80 total 60.03 last 82.93) - open/close 6277 (time 1779699467.81 total 70.04 last 88.15) - open/close 7163 (time 1779699477.81 total 80.04 last 88.55) - open/close 8009 (time 1779699487.81 total 90.05 last 84.59) - open/close 8867 (time 1779699497.83 total 100.06 last 85.63) - open/close 9759 (time 1779699507.85 total 110.08 last 89.04) total: 10000 open/close in 112.88 seconds: 88.59 ops/second llite.lustre-ffff8ff893205800.statahead_batch_max=0 llite.lustre-ffff8ff893205800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff8ff893205800.stats=clear mdc.lustre-MDT0001-mdc-ffff8ff893205800.stats=clear ls -l files in d123ad.sanity: 10001 real 1m6.203s user 0m0.317s sys 0m16.231s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff8ff893205800.statahead_batch_max=256 llite.lustre-ffff8ff893205800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff8ff893205800.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff8ff893205800.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff8ff893205800.stats=clear mdc.lustre-MDT0001-mdc-ffff8ff893205800.stats=clear ls -l files in d123ad.sanity: 10001 real 0m55.624s user 0m0.287s sys 0m16.335s batched statahead RPC= hit count=9999 mdc.lustre-MDT0000-mdc-ffff8ff893205800.batch_stats= snapshot_time: 1779699657.994889372 secs.nsecs start_time: 1779699588.426335346 secs.nsecs elapsed_time: 69.568554026 secs.nsecs subreqs per batch batches % cum % 1: 201 52 52 2: 66 17 69 4: 31 8 77 8: 22 5 83 16: 7 1 84 32: 20 5 90 64: 4 1 91 128: 0 0 91 256: 34 8 100 mdc.lustre-MDT0001-mdc-ffff8ff893205800.batch_stats= snapshot_time: 1779699657.994957492 secs.nsecs start_time: 1779699588.426413267 secs.nsecs elapsed_time: 69.568544225 secs.nsecs subreqs per batch batches % cum % 1: 0 0 0 unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second seq.cli-lustre-OST0000-super.width=65536 seq.cli-lustre-OST0001-super.width=65536 llite.lustre-ffff8ff893205800.statahead_batch_max=64 llite.lustre-ffff8ff893205800.statahead_max=128