== sanity test 123ad: Verify batching statahead works correctly ========================================================== 18:24:58 (1786832698) batch: statahead_max=32 statahead_batch_max=32 mdc.lustre-MDT0000-mdc-ffff96e0094ff000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff96e0094ff000.batch_stats=0 llite.lustre-ffff96e0094ff000.statahead_max=32 llite.lustre-ffff96e0094ff000.statahead_batch_max=32 seq.cli-lustre-OST0000-super.width=0x1ffffff seq.cli-lustre-OST0001-super.width=0x1ffffff kvm mdc.lustre-MDT0000-mdc-ffff96e0094ff000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff96e0094ff000.batch_stats=0 striped dir -i1 -c2 -H crush2 /mnt/lustre/d123ad.sanity total: 100 open/close in 2.05 seconds: 48.71 ops/second llite.lustre-ffff96e0094ff000.statahead_max=0 101 real 0m2.089s user 0m0.002s sys 0m0.325s 'ls -l' 100 files without statahead: 2 sec llite.lustre-ffff96e0094ff000.statahead_max=32 llite.lustre-ffff96e0094ff000.statahead_max=32 101 real 0m1.185s user 0m0.007s sys 0m0.161s 'ls -l' 100 files with statahead: 1 sec statahead total: 105 statahead wrong: 2 agl total: 100 list_total: 30 fname_total: 75 hit_total: 4478 miss_total: 1255 swrong=2 ewrong=2 total: 900 open/close in 9.41 seconds: 95.62 ops/second llite.lustre-ffff96e0094ff000.statahead_max=0 1001 real 0m21.868s user 0m0.032s sys 0m3.133s 'ls -l' 1000 files without statahead: 22 sec llite.lustre-ffff96e0094ff000.statahead_max=32 llite.lustre-ffff96e0094ff000.statahead_max=32 1001 real 0m7.002s user 0m0.029s sys 0m1.665s 'ls -l' 1000 files with statahead: 7 sec statahead total: 106 statahead wrong: 2 agl total: 101 list_total: 31 fname_total: 75 hit_total: 5476 miss_total: 1257 swrong=2 ewrong=2 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 15 seconds llite.lustre-ffff96e0094ff000.statahead_stats= statahead total: 106 statahead wrong: 2 agl total: 101 list_total: 31 fname_total: 75 hit_total: 5476 miss_total: 1257 mdc.lustre-MDT0000-mdc-ffff96e0094ff000.batch_stats= snapshot_time: 1786832775.851851816 secs.nsecs start_time: 1786832702.152909310 secs.nsecs elapsed_time: 73.698942506 secs.nsecs subreqs per batch batches %% cum %% 1: 2 5 5 2: 2 5 11 4: 0 0 11 8: 3 8 19 16: 20 55 75 32: 9 25 100 mdc.lustre-MDT0001-mdc-ffff96e0094ff000.batch_stats= snapshot_time: 1786832775.851917745 secs.nsecs start_time: 1786832702.153061938 secs.nsecs elapsed_time: 73.698855807 secs.nsecs subreqs per batch batches %% cum %% 1: 1 2 2 2: 1 2 5 4: 0 0 5 8: 4 10 15 16: 11 27 42 32: 23 57 100 - open/close 968 (time 1786832787.65 total 10.00 last 96.75) - open/close 1992 (time 1786832797.66 total 20.01 last 102.34) - open/close 3004 (time 1786832807.66 total 30.02 last 101.14) - open/close 3873 (time 1786832817.67 total 40.02 last 86.84) - open/close 4820 (time 1786832827.67 total 50.03 last 94.67) - open/close 5814 (time 1786832837.68 total 60.04 last 99.31) - open/close 6822 (time 1786832847.70 total 70.05 last 100.64) - open/close 7814 (time 1786832857.70 total 80.06 last 99.13) - open/close 8839 (time 1786832867.71 total 90.07 last 102.43) - open/close 9742 (time 1786832877.71 total 100.07 last 90.28) total: 10000 open/close in 102.69 seconds: 97.38 ops/second llite.lustre-ffff96e0094ff000.statahead_batch_max=0 llite.lustre-ffff96e0094ff000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff96e0094ff000.stats=clear mdc.lustre-MDT0001-mdc-ffff96e0094ff000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m1.702s user 0m0.285s sys 0m16.910s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff96e0094ff000.statahead_batch_max=32 llite.lustre-ffff96e0094ff000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff96e0094ff000.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff96e0094ff000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff96e0094ff000.stats=clear mdc.lustre-MDT0001-mdc-ffff96e0094ff000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m39.983s user 0m0.304s sys 0m16.196s batched statahead RPC= hit count=9986 mdc.lustre-MDT0000-mdc-ffff96e0094ff000.batch_stats= snapshot_time: 1786833064.494607711 secs.nsecs start_time: 1786832950.632559549 secs.nsecs elapsed_time: 113.862048162 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 0 0 0 8: 1 0 0 16: 1 0 0 32: 313 99 100 mdc.lustre-MDT0001-mdc-ffff96e0094ff000.batch_stats= snapshot_time: 1786833064.494688717 secs.nsecs start_time: 1786832950.632699590 secs.nsecs elapsed_time: 113.861989127 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 batch: statahead_max=2048 statahead_batch_max=256 mdc.lustre-MDT0000-mdc-ffff96e0094ff000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff96e0094ff000.batch_stats=0 llite.lustre-ffff96e0094ff000.statahead_max=2048 llite.lustre-ffff96e0094ff000.statahead_batch_max=256 kvm mdc.lustre-MDT0000-mdc-ffff96e0094ff000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff96e0094ff000.batch_stats=0 striped dir -i1 -c2 -H all_char /mnt/lustre/d123ad.sanity total: 100 open/close in 2.25 seconds: 44.40 ops/second llite.lustre-ffff96e0094ff000.statahead_max=0 101 real 0m2.038s user 0m0.005s sys 0m0.370s 'ls -l' 100 files without statahead: 2 sec llite.lustre-ffff96e0094ff000.statahead_max=2048 llite.lustre-ffff96e0094ff000.statahead_max=2048 101 real 0m0.826s user 0m0.005s sys 0m0.175s 'ls -l' 100 files with statahead: 1 sec statahead total: 2 statahead wrong: 0 agl total: 2 list_total: 2 fname_total: 0 hit_total: 10085 miss_total: 15 swrong=0 ewrong=0 - open/close 766 (time 1786833355.73 total 10.01 last 76.56) total: 900 open/close in 11.88 seconds: 75.78 ops/second llite.lustre-ffff96e0094ff000.statahead_max=0 1001 real 0m20.800s user 0m0.028s sys 0m3.174s 'ls -l' 1000 files without statahead: 21 sec llite.lustre-ffff96e0094ff000.statahead_max=2048 llite.lustre-ffff96e0094ff000.statahead_max=2048 1001 real 0m7.978s user 0m0.034s sys 0m1.550s 'ls -l' 1000 files with statahead: 8 sec statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11084 miss_total: 16 swrong=0 ewrong=0 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 17 seconds llite.lustre-ffff96e0094ff000.statahead_stats= statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11084 miss_total: 16 mdc.lustre-MDT0000-mdc-ffff96e0094ff000.batch_stats= snapshot_time: 1786833416.161129785 secs.nsecs start_time: 1786833066.952160493 secs.nsecs elapsed_time: 349.208969292 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 2 22 22 8: 2 22 44 16: 2 22 66 32: 1 11 77 64: 0 0 77 128: 0 0 77 256: 2 22 100 mdc.lustre-MDT0001-mdc-ffff96e0094ff000.batch_stats= snapshot_time: 1786833416.161209416 secs.nsecs start_time: 1786833066.952335440 secs.nsecs elapsed_time: 349.208873976 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 1 11 11 4: 2 22 33 8: 2 22 55 16: 0 0 55 32: 2 22 77 64: 0 0 77 128: 0 0 77 256: 2 22 100 - open/close 871 (time 1786833428.54 total 10.02 last 86.91) - open/close 1759 (time 1786833438.55 total 20.02 last 88.77) - open/close 2648 (time 1786833448.55 total 30.03 last 88.87) - open/close 3546 (time 1786833458.55 total 40.03 last 89.76) - open/close 4422 (time 1786833468.56 total 50.04 last 87.54) - open/close 5292 (time 1786833478.56 total 60.04 last 86.99) - open/close 6161 (time 1786833488.56 total 70.04 last 86.87) - open/close 6964 (time 1786833498.58 total 80.06 last 80.20) - open/close 7818 (time 1786833508.58 total 90.06 last 85.40) - open/close 8591 (time 1786833518.59 total 100.07 last 77.23) - open/close 9458 (time 1786833528.59 total 110.07 last 86.69) total: 10000 open/close in 115.86 seconds: 86.31 ops/second llite.lustre-ffff96e0094ff000.statahead_batch_max=0 llite.lustre-ffff96e0094ff000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff96e0094ff000.stats=clear mdc.lustre-MDT0001-mdc-ffff96e0094ff000.stats=clear ls -l files in d123ad.sanity: 10001 real 0m54.416s user 0m0.317s sys 0m18.827s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff96e0094ff000.statahead_batch_max=256 llite.lustre-ffff96e0094ff000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff96e0094ff000.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff96e0094ff000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff96e0094ff000.stats=clear mdc.lustre-MDT0001-mdc-ffff96e0094ff000.stats=clear ls -l files in d123ad.sanity: 10001 real 0m48.259s user 0m0.273s sys 0m17.927s batched statahead RPC= hit count=9999 mdc.lustre-MDT0000-mdc-ffff96e0094ff000.batch_stats= snapshot_time: 1786833658.338878715 secs.nsecs start_time: 1786833598.228550985 secs.nsecs elapsed_time: 60.110327730 secs.nsecs subreqs per batch batches %% cum %% 1: 236 59 59 2: 48 12 71 4: 36 9 81 8: 11 2 83 16: 6 1 85 32: 19 4 90 64: 4 1 91 128: 1 0 91 256: 34 8 100 mdc.lustre-MDT0001-mdc-ffff96e0094ff000.batch_stats= snapshot_time: 1786833658.338952469 secs.nsecs start_time: 1786833598.228648768 secs.nsecs elapsed_time: 60.110303701 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second seq.cli-lustre-OST0000-super.width=65536 seq.cli-lustre-OST0001-super.width=65536 llite.lustre-ffff96e0094ff000.statahead_batch_max=64 llite.lustre-ffff96e0094ff000.statahead_max=128