== sanity test 123ad: Verify batching statahead works correctly ========================================================== 19:15:50 (1786749350) batch: statahead_max=32 statahead_batch_max=32 mdc.lustre-MDT0000-mdc-ffff8fb37f7b9000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff8fb37f7b9000.batch_stats=0 llite.lustre-ffff8fb37f7b9000.statahead_max=32 llite.lustre-ffff8fb37f7b9000.statahead_batch_max=32 seq.cli-lustre-OST0000-super.width=0x1ffffff seq.cli-lustre-OST0001-super.width=0x1ffffff kvm mdc.lustre-MDT0000-mdc-ffff8fb37f7b9000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff8fb37f7b9000.batch_stats=0 striped dir -i1 -c2 -H crush2 /mnt/lustre/d123ad.sanity total: 100 open/close in 2.07 seconds: 48.42 ops/second llite.lustre-ffff8fb37f7b9000.statahead_max=0 101 real 0m2.657s user 0m0.004s sys 0m0.329s 'ls -l' 100 files without statahead: 2 sec llite.lustre-ffff8fb37f7b9000.statahead_max=32 llite.lustre-ffff8fb37f7b9000.statahead_max=32 101 real 0m1.283s user 0m0.004s sys 0m0.159s 'ls -l' 100 files with statahead: 2 sec statahead total: 123 statahead wrong: 1 agl total: 117 list_total: 30 fname_total: 84 hit_total: 3966 miss_total: 256 swrong=1 ewrong=1 total: 900 open/close in 8.90 seconds: 101.08 ops/second llite.lustre-ffff8fb37f7b9000.statahead_max=0 1001 real 0m21.176s user 0m0.023s sys 0m3.276s 'ls -l' 1000 files without statahead: 21 sec llite.lustre-ffff8fb37f7b9000.statahead_max=32 llite.lustre-ffff8fb37f7b9000.statahead_max=32 1001 real 0m9.929s user 0m0.021s sys 0m1.558s 'ls -l' 1000 files with statahead: 10 sec statahead total: 124 statahead wrong: 1 agl total: 118 list_total: 31 fname_total: 84 hit_total: 4963 miss_total: 259 swrong=1 ewrong=1 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 21 seconds llite.lustre-ffff8fb37f7b9000.statahead_stats= statahead total: 124 statahead wrong: 1 agl total: 118 list_total: 31 fname_total: 84 hit_total: 4963 miss_total: 259 mdc.lustre-MDT0000-mdc-ffff8fb37f7b9000.batch_stats= snapshot_time: 1786749446.249904343 secs.nsecs start_time: 1786749355.756057156 secs.nsecs elapsed_time: 90.493847187 secs.nsecs subreqs per batch batches %% cum %% 1: 1 2 2 2: 4 10 13 4: 0 0 13 8: 2 5 18 16: 20 54 72 32: 10 27 100 mdc.lustre-MDT0001-mdc-ffff8fb37f7b9000.batch_stats= snapshot_time: 1786749446.249985860 secs.nsecs start_time: 1786749355.756152033 secs.nsecs elapsed_time: 90.493833827 secs.nsecs subreqs per batch batches %% cum %% 1: 1 2 2 2: 0 0 2 4: 2 5 7 8: 3 7 15 16: 13 32 47 32: 21 52 100 - open/close 902 (time 1786749458.68 total 10.01 last 90.15) - open/close 1783 (time 1786749468.69 total 20.02 last 88.00) - open/close 2860 (time 1786749478.69 total 30.02 last 107.67) - open/close 3993 (time 1786749488.69 total 40.02 last 113.27) - open/close 4886 (time 1786749498.70 total 50.03 last 89.25) - open/close 5682 (time 1786749508.70 total 60.03 last 79.59) - open/close 6572 (time 1786749518.71 total 70.04 last 88.94) - open/close 7627 (time 1786749528.71 total 80.04 last 105.47) - open/close 8424 (time 1786749538.72 total 90.05 last 79.59) - open/close 9067 (time 1786749548.75 total 100.08 last 64.11) - open/close 9614 (time 1786749558.76 total 110.09 last 54.67) total: 10000 open/close in 114.49 seconds: 87.35 ops/second llite.lustre-ffff8fb37f7b9000.statahead_batch_max=0 llite.lustre-ffff8fb37f7b9000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff8fb37f7b9000.stats=clear mdc.lustre-MDT0001-mdc-ffff8fb37f7b9000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m1.236s user 0m0.291s sys 0m16.306s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff8fb37f7b9000.statahead_batch_max=32 llite.lustre-ffff8fb37f7b9000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff8fb37f7b9000.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff8fb37f7b9000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff8fb37f7b9000.stats=clear mdc.lustre-MDT0001-mdc-ffff8fb37f7b9000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m44.639s user 0m0.282s sys 0m15.968s batched statahead RPC= hit count=9982 mdc.lustre-MDT0000-mdc-ffff8fb37f7b9000.batch_stats= snapshot_time: 1786749754.896089553 secs.nsecs start_time: 1786749634.289680778 secs.nsecs elapsed_time: 120.606408775 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 0 0 0 8: 1 0 0 16: 0 0 0 32: 313 99 100 mdc.lustre-MDT0001-mdc-ffff8fb37f7b9000.batch_stats= snapshot_time: 1786749754.896158809 secs.nsecs start_time: 1786749634.289810928 secs.nsecs elapsed_time: 120.606347881 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 batch: statahead_max=2048 statahead_batch_max=256 mdc.lustre-MDT0000-mdc-ffff8fb37f7b9000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff8fb37f7b9000.batch_stats=0 llite.lustre-ffff8fb37f7b9000.statahead_max=2048 llite.lustre-ffff8fb37f7b9000.statahead_batch_max=256 kvm mdc.lustre-MDT0000-mdc-ffff8fb37f7b9000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff8fb37f7b9000.batch_stats=0 striped dir -i1 -c2 -H all_char /mnt/lustre/d123ad.sanity total: 100 open/close in 2.73 seconds: 36.67 ops/second llite.lustre-ffff8fb37f7b9000.statahead_max=0 101 real 0m2.281s user 0m0.004s sys 0m0.315s 'ls -l' 100 files without statahead: 2 sec llite.lustre-ffff8fb37f7b9000.statahead_max=2048 llite.lustre-ffff8fb37f7b9000.statahead_max=2048 101 real 0m0.858s user 0m0.002s sys 0m0.189s 'ls -l' 100 files with statahead: 1 sec statahead total: 2 statahead wrong: 0 agl total: 2 list_total: 2 fname_total: 0 hit_total: 10081 miss_total: 19 swrong=0 ewrong=0 total: 900 open/close in 9.37 seconds: 96.01 ops/second llite.lustre-ffff8fb37f7b9000.statahead_max=0 1001 real 0m20.135s user 0m0.040s sys 0m3.227s 'ls -l' 1000 files without statahead: 21 sec llite.lustre-ffff8fb37f7b9000.statahead_max=2048 llite.lustre-ffff8fb37f7b9000.statahead_max=2048 1001 real 0m7.123s user 0m0.027s sys 0m1.547s 'ls -l' 1000 files with statahead: 7 sec statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11080 miss_total: 20 swrong=0 ewrong=0 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 21 seconds llite.lustre-ffff8fb37f7b9000.statahead_stats= statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11080 miss_total: 20 mdc.lustre-MDT0000-mdc-ffff8fb37f7b9000.batch_stats= snapshot_time: 1786750086.456794995 secs.nsecs start_time: 1786749757.779671466 secs.nsecs elapsed_time: 328.677123529 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 2 22 22 4: 1 11 33 8: 2 22 55 16: 1 11 66 32: 1 11 77 64: 0 0 77 128: 0 0 77 256: 2 22 100 mdc.lustre-MDT0001-mdc-ffff8fb37f7b9000.batch_stats= snapshot_time: 1786750086.456865948 secs.nsecs start_time: 1786749757.784516880 secs.nsecs elapsed_time: 328.672349068 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 1 11 11 8: 3 33 44 16: 3 33 77 32: 0 0 77 64: 0 0 77 128: 0 0 77 256: 2 22 100 - open/close 836 (time 1786750099.45 total 10.00 last 83.57) - open/close 1618 (time 1786750109.45 total 20.01 last 78.17) - open/close 2444 (time 1786750119.46 total 30.01 last 82.58) - open/close 3428 (time 1786750129.46 total 40.02 last 98.34) - open/close 4290 (time 1786750139.48 total 50.04 last 86.01) - open/close 5304 (time 1786750149.49 total 60.04 last 101.36) - open/close 6400 (time 1786750159.50 total 70.05 last 109.50) - open/close 7445 (time 1786750169.50 total 80.05 last 104.49) - open/close 8415 (time 1786750179.50 total 90.05 last 96.98) - open/close 9460 (time 1786750189.50 total 100.06 last 104.48) total: 10000 open/close in 105.78 seconds: 94.54 ops/second llite.lustre-ffff8fb37f7b9000.statahead_batch_max=0 llite.lustre-ffff8fb37f7b9000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff8fb37f7b9000.stats=clear mdc.lustre-MDT0001-mdc-ffff8fb37f7b9000.stats=clear ls -l files in d123ad.sanity: 10001 real 0m59.590s user 0m0.335s sys 0m16.384s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff8fb37f7b9000.statahead_batch_max=256 llite.lustre-ffff8fb37f7b9000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff8fb37f7b9000.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff8fb37f7b9000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff8fb37f7b9000.stats=clear mdc.lustre-MDT0001-mdc-ffff8fb37f7b9000.stats=clear ls -l files in d123ad.sanity: 10001 real 0m58.148s user 0m0.349s sys 0m16.946s batched statahead RPC= hit count=9999 mdc.lustre-MDT0000-mdc-ffff8fb37f7b9000.batch_stats= snapshot_time: 1786750337.635038648 secs.nsecs start_time: 1786750264.355437576 secs.nsecs elapsed_time: 73.279601072 secs.nsecs subreqs per batch batches %% cum %% 1: 229 58 58 2: 48 12 70 4: 32 8 78 8: 17 4 82 16: 13 3 86 32: 16 4 90 64: 4 1 91 128: 1 0 91 256: 34 8 100 mdc.lustre-MDT0001-mdc-ffff8fb37f7b9000.batch_stats= snapshot_time: 1786750337.635656019 secs.nsecs start_time: 1786750264.355544696 secs.nsecs elapsed_time: 73.280111323 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second seq.cli-lustre-OST0000-super.width=65536 seq.cli-lustre-OST0001-super.width=65536 llite.lustre-ffff8fb37f7b9000.statahead_batch_max=64 llite.lustre-ffff8fb37f7b9000.statahead_max=128