== sanity test 123ad: Verify batching statahead works correctly ========================================================== 04:01:50 (1787472110) batch: statahead_max=32 statahead_batch_max=32 mdc.lustre-MDT0000-mdc-ffff9b5e02c67800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9b5e02c67800.batch_stats=0 llite.lustre-ffff9b5e02c67800.statahead_max=32 llite.lustre-ffff9b5e02c67800.statahead_batch_max=32 seq.cli-lustre-OST0000-super.width=0x1ffffff seq.cli-lustre-OST0001-super.width=0x1ffffff kvm mdc.lustre-MDT0000-mdc-ffff9b5e02c67800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9b5e02c67800.batch_stats=0 striped dir -i1 -c2 -H crush /mnt/lustre/d123ad.sanity total: 100 open/close in 2.67 seconds: 37.52 ops/second llite.lustre-ffff9b5e02c67800.statahead_max=0 101 real 0m2.703s user 0m0.006s sys 0m0.362s 'ls -l' 100 files without statahead: 3 sec llite.lustre-ffff9b5e02c67800.statahead_max=32 llite.lustre-ffff9b5e02c67800.statahead_max=32 101 real 0m1.005s user 0m0.006s sys 0m0.197s 'ls -l' 100 files with statahead: 1 sec statahead total: 144 statahead wrong: 2 agl total: 138 list_total: 31 fname_total: 113 hit_total: 3853 miss_total: 1381 swrong=2 ewrong=2 - open/close 758 (time 1787472139.48 total 10.00 last 75.78) total: 900 open/close in 12.17 seconds: 73.93 ops/second llite.lustre-ffff9b5e02c67800.statahead_max=0 1001 real 0m24.237s user 0m0.044s sys 0m3.227s 'ls -l' 1000 files without statahead: 25 sec llite.lustre-ffff9b5e02c67800.statahead_max=32 llite.lustre-ffff9b5e02c67800.statahead_max=32 1001 real 0m9.414s user 0m0.027s sys 0m1.643s 'ls -l' 1000 files with statahead: 9 sec statahead total: 145 statahead wrong: 2 agl total: 139 list_total: 32 fname_total: 113 hit_total: 4852 miss_total: 1382 swrong=2 ewrong=2 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 21 seconds llite.lustre-ffff9b5e02c67800.statahead_stats= statahead total: 145 statahead wrong: 2 agl total: 139 list_total: 32 fname_total: 113 hit_total: 4852 miss_total: 1382 mdc.lustre-MDT0000-mdc-ffff9b5e02c67800.batch_stats= snapshot_time: 1787472209.128476049 secs.nsecs start_time: 1787472116.221405988 secs.nsecs elapsed_time: 92.907070061 secs.nsecs subreqs per batch batches %% cum %% 1: 3 8 8 2: 0 0 8 4: 2 5 13 8: 2 5 18 16: 25 67 86 32: 5 13 100 mdc.lustre-MDT0001-mdc-ffff9b5e02c67800.batch_stats= snapshot_time: 1787472209.128544884 secs.nsecs start_time: 1787472116.221482773 secs.nsecs elapsed_time: 92.907062111 secs.nsecs subreqs per batch batches %% cum %% 1: 1 2 2 2: 1 2 4 4: 2 4 9 8: 3 7 17 16: 11 26 43 32: 23 56 100 - open/close 738 (time 1787472222.18 total 10.01 last 73.76) - open/close 1444 (time 1787472232.19 total 20.01 last 70.55) - open/close 2173 (time 1787472242.20 total 30.02 last 72.84) - open/close 2889 (time 1787472252.21 total 40.04 last 71.49) - open/close 3753 (time 1787472262.22 total 50.04 last 86.37) - open/close 4564 (time 1787472272.22 total 60.04 last 81.07) - open/close 5419 (time 1787472282.23 total 70.05 last 85.45) - open/close 6144 (time 1787472292.23 total 80.06 last 72.45) - open/close 6916 (time 1787472302.24 total 90.06 last 77.14) - open/close 7796 (time 1787472312.25 total 100.07 last 87.94) - open/close 8626 (time 1787472322.25 total 110.07 last 83.00) - open/close 9417 (time 1787472332.26 total 120.09 last 78.99) total: 10000 open/close in 128.18 seconds: 78.01 ops/second llite.lustre-ffff9b5e02c67800.statahead_batch_max=0 llite.lustre-ffff9b5e02c67800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9b5e02c67800.stats=clear mdc.lustre-MDT0001-mdc-ffff9b5e02c67800.stats=clear ls -l files in d123ad.sanity: 10001 real 1m15.894s user 0m0.375s sys 0m17.422s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff9b5e02c67800.statahead_batch_max=32 llite.lustre-ffff9b5e02c67800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9b5e02c67800.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff9b5e02c67800.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff9b5e02c67800.stats=clear mdc.lustre-MDT0001-mdc-ffff9b5e02c67800.stats=clear ls -l files in d123ad.sanity: 10001 real 2m9.465s user 0m0.374s sys 0m17.032s batched statahead RPC= hit count=9988 mdc.lustre-MDT0000-mdc-ffff9b5e02c67800.batch_stats= snapshot_time: 1787472576.336197855 secs.nsecs start_time: 1787472429.287524861 secs.nsecs elapsed_time: 147.048672994 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 0 0 0 8: 1 0 0 16: 1 0 0 32: 313 99 100 mdc.lustre-MDT0001-mdc-ffff9b5e02c67800.batch_stats= snapshot_time: 1787472576.336270374 secs.nsecs start_time: 1787472429.287789569 secs.nsecs elapsed_time: 147.048480805 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 batch: statahead_max=2048 statahead_batch_max=256 mdc.lustre-MDT0000-mdc-ffff9b5e02c67800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9b5e02c67800.batch_stats=0 llite.lustre-ffff9b5e02c67800.statahead_max=2048 llite.lustre-ffff9b5e02c67800.statahead_batch_max=256 kvm mdc.lustre-MDT0000-mdc-ffff9b5e02c67800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9b5e02c67800.batch_stats=0 striped dir -i1 -c2 -H all_char /mnt/lustre/d123ad.sanity total: 100 open/close in 2.42 seconds: 41.24 ops/second llite.lustre-ffff9b5e02c67800.statahead_max=0 101 real 0m2.239s user 0m0.006s sys 0m0.341s 'ls -l' 100 files without statahead: 2 sec llite.lustre-ffff9b5e02c67800.statahead_max=2048 llite.lustre-ffff9b5e02c67800.statahead_max=2048 101 real 0m0.798s user 0m0.003s sys 0m0.202s 'ls -l' 100 files with statahead: 0 sec statahead total: 2 statahead wrong: 0 agl total: 2 list_total: 2 fname_total: 0 hit_total: 10087 miss_total: 13 swrong=0 ewrong=0 - open/close 792 (time 1787472889.57 total 10.01 last 79.09) total: 900 open/close in 11.48 seconds: 78.41 ops/second llite.lustre-ffff9b5e02c67800.statahead_max=0 1001 real 0m24.561s user 0m0.036s sys 0m3.317s 'ls -l' 1000 files without statahead: 24 sec llite.lustre-ffff9b5e02c67800.statahead_max=2048 llite.lustre-ffff9b5e02c67800.statahead_max=2048 1001 real 0m6.454s user 0m0.030s sys 0m1.663s 'ls -l' 1000 files with statahead: 6 sec statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11086 miss_total: 14 swrong=0 ewrong=0 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 21 seconds llite.lustre-ffff9b5e02c67800.statahead_stats= statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11086 miss_total: 14 mdc.lustre-MDT0000-mdc-ffff9b5e02c67800.batch_stats= snapshot_time: 1787472956.046712929 secs.nsecs start_time: 1787472579.440361049 secs.nsecs elapsed_time: 376.606351880 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 1 10 10 8: 3 30 40 16: 3 30 70 32: 1 10 80 64: 0 0 80 128: 0 0 80 256: 2 20 100 mdc.lustre-MDT0001-mdc-ffff9b5e02c67800.batch_stats= snapshot_time: 1787472956.046789527 secs.nsecs start_time: 1787472579.440512894 secs.nsecs elapsed_time: 376.606276633 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 1 10 10 4: 3 30 40 8: 2 20 60 16: 1 10 70 32: 1 10 80 64: 0 0 80 128: 0 0 80 256: 2 20 100 - open/close 840 (time 1787472968.74 total 10.00 last 83.98) - open/close 1583 (time 1787472978.74 total 20.01 last 74.28) - open/close 2342 (time 1787472988.75 total 30.01 last 75.85) - open/close 3190 (time 1787472998.75 total 40.02 last 84.78) - open/close 4080 (time 1787473008.75 total 50.02 last 89.00) - open/close 4873 (time 1787473018.75 total 60.02 last 79.28) - open/close 5730 (time 1787473028.76 total 70.02 last 85.68) - open/close 6566 (time 1787473038.77 total 80.04 last 83.44) - open/close 7473 (time 1787473048.79 total 90.05 last 90.60) - open/close 8290 (time 1787473058.79 total 100.06 last 81.65) - open/close 9060 (time 1787473068.80 total 110.06 last 76.96) - open/close 9890 (time 1787473078.80 total 120.06 last 82.98) total: 10000 open/close in 121.72 seconds: 82.16 ops/second llite.lustre-ffff9b5e02c67800.statahead_batch_max=0 llite.lustre-ffff9b5e02c67800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9b5e02c67800.stats=clear mdc.lustre-MDT0001-mdc-ffff9b5e02c67800.stats=clear ls -l files in d123ad.sanity: 10001 real 1m2.951s user 0m0.336s sys 0m18.261s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff9b5e02c67800.statahead_batch_max=256 llite.lustre-ffff9b5e02c67800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9b5e02c67800.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff9b5e02c67800.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff9b5e02c67800.stats=clear mdc.lustre-MDT0001-mdc-ffff9b5e02c67800.stats=clear ls -l files in d123ad.sanity: 10001 real 0m56.349s user 0m0.323s sys 0m17.815s batched statahead RPC= hit count=9999 mdc.lustre-MDT0000-mdc-ffff9b5e02c67800.batch_stats= snapshot_time: 1787473224.407562330 secs.nsecs start_time: 1787473154.118788443 secs.nsecs elapsed_time: 70.288773887 secs.nsecs subreqs per batch batches %% cum %% 1: 345 59 59 2: 76 13 72 4: 58 10 83 8: 28 4 87 16: 19 3 91 32: 14 2 93 64: 4 0 94 128: 0 0 94 256: 33 5 100 mdc.lustre-MDT0001-mdc-ffff9b5e02c67800.batch_stats= snapshot_time: 1787473224.408073669 secs.nsecs start_time: 1787473154.118930041 secs.nsecs elapsed_time: 70.289143628 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second seq.cli-lustre-OST0000-super.width=65536 seq.cli-lustre-OST0001-super.width=65536 llite.lustre-ffff9b5e02c67800.statahead_batch_max=64 llite.lustre-ffff9b5e02c67800.statahead_max=128