== sanity test 123ad: Verify batching statahead works correctly ========================================================== 22:19:54 (1786673994) batch: statahead_max=32 statahead_batch_max=32 mdc.lustre-MDT0000-mdc-ffff8e6173420000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff8e6173420000.batch_stats=0 llite.lustre-ffff8e6173420000.statahead_max=32 llite.lustre-ffff8e6173420000.statahead_batch_max=32 seq.cli-lustre-OST0000-super.width=0x1ffffff seq.cli-lustre-OST0001-super.width=0x1ffffff kvm mdc.lustre-MDT0000-mdc-ffff8e6173420000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff8e6173420000.batch_stats=0 striped dir -i1 -c2 -H all_char /mnt/lustre/d123ad.sanity total: 100 open/close in 3.61 seconds: 27.72 ops/second llite.lustre-ffff8e6173420000.statahead_max=0 101 real 0m3.785s user 0m0.007s sys 0m0.372s 'ls -l' 100 files without statahead: 4 sec llite.lustre-ffff8e6173420000.statahead_max=32 llite.lustre-ffff8e6173420000.statahead_max=32 101 real 0m1.776s user 0m0.005s sys 0m0.204s 'ls -l' 100 files with statahead: 2 sec statahead total: 154 statahead wrong: 2 agl total: 148 list_total: 32 fname_total: 122 hit_total: 3699 miss_total: 1481 swrong=2 ewrong=2 - open/close 634 (time 1786674027.70 total 10.02 last 63.30) total: 900 open/close in 14.00 seconds: 64.28 ops/second llite.lustre-ffff8e6173420000.statahead_max=0 1001 real 0m24.695s user 0m0.034s sys 0m3.276s 'ls -l' 1000 files without statahead: 25 sec llite.lustre-ffff8e6173420000.statahead_max=32 llite.lustre-ffff8e6173420000.statahead_max=32 1001 real 0m11.609s user 0m0.031s sys 0m1.791s 'ls -l' 1000 files with statahead: 12 sec statahead total: 155 statahead wrong: 2 agl total: 149 list_total: 33 fname_total: 122 hit_total: 4695 miss_total: 1485 swrong=2 ewrong=2 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 22 seconds llite.lustre-ffff8e6173420000.statahead_stats= statahead total: 155 statahead wrong: 2 agl total: 149 list_total: 33 fname_total: 122 hit_total: 4695 miss_total: 1485 mdc.lustre-MDT0000-mdc-ffff8e6173420000.batch_stats= snapshot_time: 1786674106.000712466 secs.nsecs start_time: 1786673999.476501616 secs.nsecs elapsed_time: 106.524210850 secs.nsecs subreqs per batch batches %% cum %% 1: 1 2 2 2: 0 0 2 4: 2 5 7 8: 3 7 15 16: 21 53 69 32: 12 30 100 mdc.lustre-MDT0001-mdc-ffff8e6173420000.batch_stats= snapshot_time: 1786674106.000785539 secs.nsecs start_time: 1786673999.476697213 secs.nsecs elapsed_time: 106.524088326 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 2 5 5 8: 4 10 15 16: 18 46 61 32: 15 38 100 - open/close 681 (time 1786674119.64 total 10.01 last 68.05) - open/close 1442 (time 1786674129.65 total 20.02 last 76.03) - open/close 2089 (time 1786674139.67 total 30.04 last 64.56) - open/close 2791 (time 1786674149.67 total 40.04 last 70.19) - open/close 3256 (time 1786674159.70 total 50.07 last 46.36) - open/close 3670 (time 1786674169.71 total 60.08 last 41.36) - open/close 4240 (time 1786674179.72 total 70.09 last 56.95) - open/close 4771 (time 1786674189.75 total 80.12 last 52.96) - open/close 5562 (time 1786674199.75 total 90.12 last 79.08) - open/close 6253 (time 1786674209.76 total 100.13 last 69.04) - open/close 6961 (time 1786674219.76 total 110.13 last 70.79) - open/close 7698 (time 1786674229.77 total 120.14 last 73.61) - open/close 8266 (time 1786674239.78 total 130.14 last 56.78) - open/close 8718 (time 1786674249.78 total 140.15 last 45.18) - open/close 9370 (time 1786674259.79 total 150.16 last 65.15) total: 10000 open/close in 159.00 seconds: 62.89 ops/second llite.lustre-ffff8e6173420000.statahead_batch_max=0 llite.lustre-ffff8e6173420000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff8e6173420000.stats=clear mdc.lustre-MDT0001-mdc-ffff8e6173420000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m14.695s user 0m0.340s sys 0m16.532s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff8e6173420000.statahead_batch_max=32 llite.lustre-ffff8e6173420000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff8e6173420000.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff8e6173420000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff8e6173420000.stats=clear mdc.lustre-MDT0001-mdc-ffff8e6173420000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m56.925s user 0m0.325s sys 0m16.507s batched statahead RPC= hit count=9981 mdc.lustre-MDT0000-mdc-ffff8e6173420000.batch_stats= snapshot_time: 1786674487.451032775 secs.nsecs start_time: 1786674355.337478582 secs.nsecs elapsed_time: 132.113554193 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 0 0 0 8: 1 0 0 16: 1 0 0 32: 313 99 100 mdc.lustre-MDT0001-mdc-ffff8e6173420000.batch_stats= snapshot_time: 1786674487.451093723 secs.nsecs start_time: 1786674355.337547668 secs.nsecs elapsed_time: 132.113546055 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 batch: statahead_max=2048 statahead_batch_max=256 mdc.lustre-MDT0000-mdc-ffff8e6173420000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff8e6173420000.batch_stats=0 llite.lustre-ffff8e6173420000.statahead_max=2048 llite.lustre-ffff8e6173420000.statahead_batch_max=256 kvm mdc.lustre-MDT0000-mdc-ffff8e6173420000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff8e6173420000.batch_stats=0 striped dir -i1 -c2 -H crush /mnt/lustre/d123ad.sanity total: 100 open/close in 2.79 seconds: 35.89 ops/second llite.lustre-ffff8e6173420000.statahead_max=0 101 real 0m2.089s user 0m0.005s sys 0m0.405s 'ls -l' 100 files without statahead: 2 sec llite.lustre-ffff8e6173420000.statahead_max=2048 llite.lustre-ffff8e6173420000.statahead_max=2048 101 real 0m1.146s user 0m0.002s sys 0m0.240s 'ls -l' 100 files with statahead: 1 sec statahead total: 2 statahead wrong: 0 agl total: 2 list_total: 2 fname_total: 0 hit_total: 10080 miss_total: 20 swrong=0 ewrong=0 - open/close 704 (time 1786674805.18 total 10.01 last 70.31) total: 900 open/close in 13.12 seconds: 68.61 ops/second llite.lustre-ffff8e6173420000.statahead_max=0 1001 real 0m24.662s user 0m0.045s sys 0m3.599s 'ls -l' 1000 files without statahead: 25 sec llite.lustre-ffff8e6173420000.statahead_max=2048 llite.lustre-ffff8e6173420000.statahead_max=2048 1001 real 0m7.659s user 0m0.034s sys 0m1.812s 'ls -l' 1000 files with statahead: 7 sec statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11079 miss_total: 21 swrong=0 ewrong=0 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 23 seconds llite.lustre-ffff8e6173420000.statahead_stats= statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11079 miss_total: 21 mdc.lustre-MDT0000-mdc-ffff8e6173420000.batch_stats= snapshot_time: 1786674878.649122530 secs.nsecs start_time: 1786674489.817537524 secs.nsecs elapsed_time: 388.831585006 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 1 14 14 4: 2 28 42 8: 2 28 71 16: 0 0 71 32: 0 0 71 64: 0 0 71 128: 0 0 71 256: 2 28 100 mdc.lustre-MDT0001-mdc-ffff8e6173420000.batch_stats= snapshot_time: 1786674878.650584370 secs.nsecs start_time: 1786674489.817727524 secs.nsecs elapsed_time: 388.832856846 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 1 10 10 4: 0 0 10 8: 2 20 30 16: 3 30 60 32: 1 10 70 64: 1 10 80 128: 0 0 80 256: 2 20 100 - open/close 714 (time 1786674891.64 total 10.02 last 71.23) - open/close 1389 (time 1786674901.64 total 20.03 last 67.47) - open/close 2159 (time 1786674911.65 total 30.03 last 76.94) - open/close 2924 (time 1786674921.66 total 40.04 last 76.44) - open/close 3688 (time 1786674931.68 total 50.07 last 76.22) - open/close 4470 (time 1786674941.68 total 60.07 last 78.20) - open/close 5240 (time 1786674951.68 total 70.07 last 76.98) - open/close 5910 (time 1786674961.68 total 80.07 last 66.99) - open/close 6598 (time 1786674971.69 total 90.07 last 68.77) - open/close 7351 (time 1786674981.70 total 100.08 last 75.23) - open/close 8132 (time 1786674991.71 total 110.09 last 78.03) - open/close 8889 (time 1786675001.72 total 120.10 last 75.62) - open/close 9597 (time 1786675011.72 total 130.11 last 70.79) total: 10000 open/close in 135.74 seconds: 73.67 ops/second llite.lustre-ffff8e6173420000.statahead_batch_max=0 llite.lustre-ffff8e6173420000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff8e6173420000.stats=clear mdc.lustre-MDT0001-mdc-ffff8e6173420000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m7.438s user 0m0.341s sys 0m16.872s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff8e6173420000.statahead_batch_max=256 llite.lustre-ffff8e6173420000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff8e6173420000.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff8e6173420000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff8e6173420000.stats=clear mdc.lustre-MDT0001-mdc-ffff8e6173420000.stats=clear ls -l files in d123ad.sanity: 10001 real 0m59.169s user 0m0.347s sys 0m16.864s batched statahead RPC= hit count=9999 mdc.lustre-MDT0000-mdc-ffff8e6173420000.batch_stats= snapshot_time: 1786675170.637017204 secs.nsecs start_time: 1786675096.148312776 secs.nsecs elapsed_time: 74.488704428 secs.nsecs subreqs per batch batches %% cum %% 1: 254 52 52 2: 92 18 70 4: 50 10 81 8: 24 4 86 16: 12 2 88 32: 17 3 92 64: 5 1 93 128: 1 0 93 256: 33 6 100 mdc.lustre-MDT0001-mdc-ffff8e6173420000.batch_stats= snapshot_time: 1786675170.637219134 secs.nsecs start_time: 1786675096.148398983 secs.nsecs elapsed_time: 74.488820151 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second seq.cli-lustre-OST0000-super.width=65536 seq.cli-lustre-OST0001-super.width=65536 llite.lustre-ffff8e6173420000.statahead_batch_max=64 llite.lustre-ffff8e6173420000.statahead_max=128