== sanity test 123ad: Verify batching statahead works correctly ========================================================== 17:25:34 (1787865934) batch: statahead_max=32 statahead_batch_max=32 mdc.lustre-MDT0000-mdc-ffff8d4dd1be8000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff8d4dd1be8000.batch_stats=0 llite.lustre-ffff8d4dd1be8000.statahead_max=32 llite.lustre-ffff8d4dd1be8000.statahead_batch_max=32 seq.cli-lustre-OST0000-super.width=0x1ffffff seq.cli-lustre-OST0001-super.width=0x1ffffff kvm mdc.lustre-MDT0000-mdc-ffff8d4dd1be8000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff8d4dd1be8000.batch_stats=0 striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d123ad.sanity total: 100 open/close in 2.82 seconds: 35.43 ops/second llite.lustre-ffff8d4dd1be8000.statahead_max=0 101 real 0m2.153s user 0m0.004s sys 0m0.311s 'ls -l' 100 files without statahead: 2 sec llite.lustre-ffff8d4dd1be8000.statahead_max=32 llite.lustre-ffff8d4dd1be8000.statahead_max=32 101 real 0m1.100s user 0m0.005s sys 0m0.176s 'ls -l' 100 files with statahead: 1 sec statahead total: 147 statahead wrong: 2 agl total: 141 list_total: 31 fname_total: 101 hit_total: 4153 miss_total: 1287 swrong=2 ewrong=2 - open/close 735 (time 1787865961.84 total 10.01 last 73.43) total: 900 open/close in 12.32 seconds: 73.06 ops/second llite.lustre-ffff8d4dd1be8000.statahead_max=0 1001 real 0m25.466s user 0m0.033s sys 0m3.028s 'ls -l' 1000 files without statahead: 25 sec llite.lustre-ffff8d4dd1be8000.statahead_max=32 llite.lustre-ffff8d4dd1be8000.statahead_max=32 1001 real 0m10.643s user 0m0.034s sys 0m1.408s 'ls -l' 1000 files with statahead: 11 sec statahead total: 148 statahead wrong: 2 agl total: 142 list_total: 32 fname_total: 101 hit_total: 5150 miss_total: 1290 swrong=2 ewrong=2 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 21 seconds llite.lustre-ffff8d4dd1be8000.statahead_stats= statahead total: 148 statahead wrong: 2 agl total: 142 list_total: 32 fname_total: 101 hit_total: 5150 miss_total: 1290 mdc.lustre-MDT0000-mdc-ffff8d4dd1be8000.batch_stats= snapshot_time: 1787866034.922648718 secs.nsecs start_time: 1787865939.103915301 secs.nsecs elapsed_time: 95.818733417 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 1 2 2 4: 2 5 7 8: 4 10 17 16: 17 43 61 32: 15 38 100 mdc.lustre-MDT0001-mdc-ffff8d4dd1be8000.batch_stats= snapshot_time: 1787866034.922871118 secs.nsecs start_time: 1787865939.103986442 secs.nsecs elapsed_time: 95.818884676 secs.nsecs subreqs per batch batches %% cum %% 1: 1 2 2 2: 1 2 5 4: 2 5 10 8: 2 5 15 16: 24 60 75 32: 10 25 100 - open/close 782 (time 1787866047.36 total 10.01 last 78.13) - open/close 1486 (time 1787866057.37 total 20.02 last 70.34) - open/close 2224 (time 1787866067.37 total 30.02 last 73.79) - open/close 2888 (time 1787866077.38 total 40.03 last 66.34) - open/close 3599 (time 1787866087.39 total 50.04 last 71.02) - open/close 4263 (time 1787866097.41 total 60.06 last 66.24) - open/close 5012 (time 1787866107.43 total 70.07 last 74.82) - open/close 5734 (time 1787866117.43 total 80.07 last 72.19) - open/close 6530 (time 1787866127.43 total 90.08 last 79.56) - open/close 7248 (time 1787866137.44 total 100.09 last 71.73) - open/close 7939 (time 1787866147.44 total 110.09 last 69.09) - open/close 8634 (time 1787866157.45 total 120.09 last 69.47) - open/close 9355 (time 1787866167.46 total 130.10 last 72.04) total: 10000 open/close in 139.41 seconds: 71.73 ops/second llite.lustre-ffff8d4dd1be8000.statahead_batch_max=0 llite.lustre-ffff8d4dd1be8000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff8d4dd1be8000.stats=clear mdc.lustre-MDT0001-mdc-ffff8d4dd1be8000.stats=clear ls -l files in d123ad.sanity: 10001 real 0m59.055s user 0m0.323s sys 0m15.286s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff8d4dd1be8000.statahead_batch_max=32 llite.lustre-ffff8d4dd1be8000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff8d4dd1be8000.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff8d4dd1be8000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff8d4dd1be8000.stats=clear mdc.lustre-MDT0001-mdc-ffff8d4dd1be8000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m48.899s user 0m0.299s sys 0m14.150s batched statahead RPC= hit count=9978 mdc.lustre-MDT0000-mdc-ffff8d4dd1be8000.batch_stats= snapshot_time: 1787866364.016722887 secs.nsecs start_time: 1787866243.620660968 secs.nsecs elapsed_time: 120.396061919 secs.nsecs subreqs per batch batches %% cum %% 1: 1 0 0 2: 0 0 0 4: 0 0 0 8: 1 0 0 16: 1 0 0 32: 313 99 100 mdc.lustre-MDT0001-mdc-ffff8d4dd1be8000.batch_stats= snapshot_time: 1787866364.016791977 secs.nsecs start_time: 1787866243.620749648 secs.nsecs elapsed_time: 120.396042329 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 batch: statahead_max=2048 statahead_batch_max=256 mdc.lustre-MDT0000-mdc-ffff8d4dd1be8000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff8d4dd1be8000.batch_stats=0 llite.lustre-ffff8d4dd1be8000.statahead_max=2048 llite.lustre-ffff8d4dd1be8000.statahead_batch_max=256 kvm mdc.lustre-MDT0000-mdc-ffff8d4dd1be8000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff8d4dd1be8000.batch_stats=0 striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d123ad.sanity total: 100 open/close in 2.77 seconds: 36.05 ops/second llite.lustre-ffff8d4dd1be8000.statahead_max=0 101 real 0m2.172s user 0m0.003s sys 0m0.305s 'ls -l' 100 files without statahead: 2 sec llite.lustre-ffff8d4dd1be8000.statahead_max=2048 llite.lustre-ffff8d4dd1be8000.statahead_max=2048 101 real 0m0.873s user 0m0.003s sys 0m0.156s 'ls -l' 100 files with statahead: 1 sec statahead total: 2 statahead wrong: 0 agl total: 2 list_total: 2 fname_total: 0 hit_total: 10077 miss_total: 23 swrong=0 ewrong=0 - open/close 662 (time 1787866681.97 total 10.01 last 66.12) total: 900 open/close in 13.24 seconds: 67.98 ops/second llite.lustre-ffff8d4dd1be8000.statahead_max=0 1001 real 0m24.902s user 0m0.037s sys 0m2.814s 'ls -l' 1000 files without statahead: 25 sec llite.lustre-ffff8d4dd1be8000.statahead_max=2048 llite.lustre-ffff8d4dd1be8000.statahead_max=2048 1001 real 0m6.433s user 0m0.025s sys 0m1.439s 'ls -l' 1000 files with statahead: 6 sec statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11076 miss_total: 24 swrong=0 ewrong=0 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 21 seconds llite.lustre-ffff8d4dd1be8000.statahead_stats= statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11076 miss_total: 24 mdc.lustre-MDT0000-mdc-ffff8d4dd1be8000.batch_stats= snapshot_time: 1787866751.300796224 secs.nsecs start_time: 1787866366.463163716 secs.nsecs elapsed_time: 384.837632508 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 2 22 22 8: 4 44 66 16: 0 0 66 32: 0 0 66 64: 1 11 77 128: 0 0 77 256: 2 22 100 mdc.lustre-MDT0001-mdc-ffff8d4dd1be8000.batch_stats= snapshot_time: 1787866751.300860491 secs.nsecs start_time: 1787866366.463330606 secs.nsecs elapsed_time: 384.837529885 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 1 11 11 4: 2 22 33 8: 1 11 44 16: 2 22 66 32: 1 11 77 64: 0 0 77 128: 0 0 77 256: 2 22 100 - open/close 850 (time 1787866763.80 total 10.00 last 84.98) - open/close 1576 (time 1787866773.81 total 20.01 last 72.54) - open/close 2277 (time 1787866783.82 total 30.02 last 70.04) - open/close 2938 (time 1787866793.83 total 40.03 last 66.03) - open/close 3726 (time 1787866803.83 total 50.03 last 78.76) - open/close 4409 (time 1787866813.84 total 60.05 last 68.23) - open/close 5136 (time 1787866823.85 total 70.05 last 72.66) - open/close 5882 (time 1787866833.87 total 80.07 last 74.45) - open/close 6725 (time 1787866843.88 total 90.08 last 84.19) - open/close 7542 (time 1787866853.89 total 100.09 last 81.65) - open/close 8400 (time 1787866863.89 total 110.10 last 85.76) - open/close 9195 (time 1787866873.90 total 120.10 last 79.44) - open/close 9943 (time 1787866883.90 total 130.11 last 74.77) total: 10000 open/close in 130.70 seconds: 76.51 ops/second llite.lustre-ffff8d4dd1be8000.statahead_batch_max=0 llite.lustre-ffff8d4dd1be8000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff8d4dd1be8000.stats=clear mdc.lustre-MDT0001-mdc-ffff8d4dd1be8000.stats=clear ls -l files in d123ad.sanity: 10001 real 0m58.088s user 0m0.288s sys 0m15.677s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff8d4dd1be8000.statahead_batch_max=256 llite.lustre-ffff8d4dd1be8000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff8d4dd1be8000.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff8d4dd1be8000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff8d4dd1be8000.stats=clear mdc.lustre-MDT0001-mdc-ffff8d4dd1be8000.stats=clear ls -l files in d123ad.sanity: 10001 real 0m54.448s user 0m0.314s sys 0m14.896s batched statahead RPC= hit count=9999 mdc.lustre-MDT0000-mdc-ffff8d4dd1be8000.batch_stats= snapshot_time: 1787867016.393802901 secs.nsecs start_time: 1787866950.976205135 secs.nsecs elapsed_time: 65.417597766 secs.nsecs subreqs per batch batches %% cum %% 1: 169 49 49 2: 50 14 64 4: 38 11 75 8: 19 5 80 16: 6 1 82 32: 19 5 88 64: 6 1 90 128: 0 0 90 256: 34 9 100 mdc.lustre-MDT0001-mdc-ffff8d4dd1be8000.batch_stats= snapshot_time: 1787867016.393865017 secs.nsecs start_time: 1787866950.976342898 secs.nsecs elapsed_time: 65.417522119 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second seq.cli-lustre-OST0000-super.width=65536 seq.cli-lustre-OST0001-super.width=65536 llite.lustre-ffff8d4dd1be8000.statahead_batch_max=64 llite.lustre-ffff8d4dd1be8000.statahead_max=128