== sanity test 123ad: Verify batching statahead works correctly ========================================================== 18:48:00 (1787870880) batch: statahead_max=32 statahead_batch_max=32 mdc.lustre-MDT0000-mdc-ffff93b16b61b800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff93b16b61b800.batch_stats=0 llite.lustre-ffff93b16b61b800.statahead_max=32 llite.lustre-ffff93b16b61b800.statahead_batch_max=32 seq.cli-lustre-OST0000-super.width=0x1ffffff seq.cli-lustre-OST0001-super.width=0x1ffffff kvm mdc.lustre-MDT0000-mdc-ffff93b16b61b800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff93b16b61b800.batch_stats=0 striped dir -i1 -c2 -H all_char /mnt/lustre/d123ad.sanity total: 100 open/close in 3.65 seconds: 27.39 ops/second llite.lustre-ffff93b16b61b800.statahead_max=0 101 real 0m2.815s user 0m0.003s sys 0m0.363s 'ls -l' 100 files without statahead: 3 sec llite.lustre-ffff93b16b61b800.statahead_max=32 llite.lustre-ffff93b16b61b800.statahead_max=32 101 real 0m1.149s user 0m0.005s sys 0m0.190s 'ls -l' 100 files with statahead: 1 sec statahead total: 136 statahead wrong: 2 agl total: 130 list_total: 31 fname_total: 77 hit_total: 4062 miss_total: 1732 swrong=2 ewrong=2 - open/close 424 (time 1787870913.29 total 10.02 last 42.30) total: 900 open/close in 18.55 seconds: 48.51 ops/second llite.lustre-ffff93b16b61b800.statahead_max=0 1001 real 0m22.444s user 0m0.038s sys 0m3.197s 'ls -l' 1000 files without statahead: 22 sec llite.lustre-ffff93b16b61b800.statahead_max=32 llite.lustre-ffff93b16b61b800.statahead_max=32 1001 real 0m9.719s user 0m0.032s sys 0m1.423s 'ls -l' 1000 files with statahead: 10 sec statahead total: 137 statahead wrong: 2 agl total: 131 list_total: 32 fname_total: 77 hit_total: 5061 miss_total: 1733 swrong=2 ewrong=2 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 20 seconds llite.lustre-ffff93b16b61b800.statahead_stats= statahead total: 137 statahead wrong: 2 agl total: 131 list_total: 32 fname_total: 77 hit_total: 5061 miss_total: 1733 mdc.lustre-MDT0000-mdc-ffff93b16b61b800.batch_stats= snapshot_time: 1787870987.580394731 secs.nsecs start_time: 1787870886.089341029 secs.nsecs elapsed_time: 101.491053702 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 1 2 2 4: 2 5 7 8: 1 2 10 16: 23 60 71 32: 11 28 100 mdc.lustre-MDT0001-mdc-ffff93b16b61b800.batch_stats= snapshot_time: 1787870987.580537862 secs.nsecs start_time: 1787870886.089430858 secs.nsecs elapsed_time: 101.491107004 secs.nsecs subreqs per batch batches %% cum %% 1: 2 5 5 2: 0 0 5 4: 1 2 7 8: 3 7 15 16: 16 41 56 32: 17 43 100 - open/close 696 (time 1787871000.29 total 10.00 last 69.57) - open/close 1536 (time 1787871010.30 total 20.01 last 83.96) - open/close 2326 (time 1787871020.30 total 30.01 last 78.96) - open/close 3108 (time 1787871030.31 total 40.02 last 78.16) - open/close 3810 (time 1787871040.32 total 50.03 last 70.15) - open/close 4546 (time 1787871050.33 total 60.04 last 73.51) - open/close 5348 (time 1787871060.33 total 70.04 last 80.20) - open/close 6059 (time 1787871070.34 total 80.05 last 71.02) - open/close 6805 (time 1787871080.35 total 90.06 last 74.50) - open/close 7549 (time 1787871090.37 total 100.09 last 74.24) - open/close 8284 (time 1787871100.38 total 110.09 last 73.46) - open/close 9164 (time 1787871110.38 total 120.09 last 87.97) - open/close 9962 (time 1787871120.38 total 130.09 last 79.79) total: 10000 open/close in 130.81 seconds: 76.45 ops/second llite.lustre-ffff93b16b61b800.statahead_batch_max=0 llite.lustre-ffff93b16b61b800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff93b16b61b800.stats=clear mdc.lustre-MDT0001-mdc-ffff93b16b61b800.stats=clear ls -l files in d123ad.sanity: 10001 real 1m9.094s user 0m0.311s sys 0m14.762s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff93b16b61b800.statahead_batch_max=32 llite.lustre-ffff93b16b61b800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff93b16b61b800.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff93b16b61b800.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff93b16b61b800.stats=clear mdc.lustre-MDT0001-mdc-ffff93b16b61b800.stats=clear ls -l files in d123ad.sanity: 10001 real 1m51.946s user 0m0.312s sys 0m14.392s batched statahead RPC= hit count=9979 mdc.lustre-MDT0000-mdc-ffff93b16b61b800.batch_stats= snapshot_time: 1787871324.114230712 secs.nsecs start_time: 1787871199.914265141 secs.nsecs elapsed_time: 124.199965571 secs.nsecs subreqs per batch batches %% cum %% 1: 1 0 0 2: 0 0 0 4: 0 0 0 8: 1 0 0 16: 1 0 0 32: 313 99 100 mdc.lustre-MDT0001-mdc-ffff93b16b61b800.batch_stats= snapshot_time: 1787871324.114447004 secs.nsecs start_time: 1787871199.914365904 secs.nsecs elapsed_time: 124.200081100 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 batch: statahead_max=2048 statahead_batch_max=256 mdc.lustre-MDT0000-mdc-ffff93b16b61b800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff93b16b61b800.batch_stats=0 llite.lustre-ffff93b16b61b800.statahead_max=2048 llite.lustre-ffff93b16b61b800.statahead_batch_max=256 kvm mdc.lustre-MDT0000-mdc-ffff93b16b61b800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff93b16b61b800.batch_stats=0 striped dir -i1 -c2 -H crush /mnt/lustre/d123ad.sanity total: 100 open/close in 2.61 seconds: 38.28 ops/second llite.lustre-ffff93b16b61b800.statahead_max=0 101 real 0m2.462s user 0m0.006s sys 0m0.327s 'ls -l' 100 files without statahead: 3 sec llite.lustre-ffff93b16b61b800.statahead_max=2048 llite.lustre-ffff93b16b61b800.statahead_max=2048 101 real 0m1.255s user 0m0.003s sys 0m0.155s 'ls -l' 100 files with statahead: 1 sec statahead total: 2 statahead wrong: 0 agl total: 2 list_total: 2 fname_total: 0 hit_total: 10078 miss_total: 22 swrong=0 ewrong=0 - open/close 821 (time 1787871602.30 total 10.01 last 82.04) total: 900 open/close in 11.20 seconds: 80.36 ops/second llite.lustre-ffff93b16b61b800.statahead_max=0 1001 real 0m16.080s user 0m0.031s sys 0m2.727s 'ls -l' 1000 files without statahead: 16 sec llite.lustre-ffff93b16b61b800.statahead_max=2048 llite.lustre-ffff93b16b61b800.statahead_max=2048 1001 real 0m4.956s user 0m0.028s sys 0m1.233s 'ls -l' 1000 files with statahead: 5 sec statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11077 miss_total: 23 swrong=0 ewrong=0 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 17 seconds llite.lustre-ffff93b16b61b800.statahead_stats= statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11077 miss_total: 23 mdc.lustre-MDT0000-mdc-ffff93b16b61b800.batch_stats= snapshot_time: 1787871653.999259040 secs.nsecs start_time: 1787871326.554116892 secs.nsecs elapsed_time: 327.445142148 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 1 16 16 4: 2 33 50 8: 1 16 66 16: 0 0 66 32: 0 0 66 64: 0 0 66 128: 0 0 66 256: 2 33 100 mdc.lustre-MDT0001-mdc-ffff93b16b61b800.batch_stats= snapshot_time: 1787871653.999333520 secs.nsecs start_time: 1787871326.554286412 secs.nsecs elapsed_time: 327.445047108 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 1 12 12 8: 3 37 50 16: 0 0 50 32: 0 0 50 64: 1 12 62 128: 1 12 75 256: 2 25 100 - open/close 820 (time 1787871666.74 total 10.00 last 81.97) - open/close 1877 (time 1787871676.74 total 20.00 last 105.69) - open/close 2759 (time 1787871686.75 total 30.01 last 88.12) - open/close 3711 (time 1787871696.76 total 40.02 last 95.10) - open/close 4420 (time 1787871706.77 total 50.04 last 70.81) - open/close 5229 (time 1787871716.77 total 60.04 last 80.85) - open/close 6087 (time 1787871726.78 total 70.05 last 85.76) - open/close 6830 (time 1787871736.78 total 80.05 last 74.27) - open/close 7543 (time 1787871746.78 total 90.05 last 71.28) - open/close 8320 (time 1787871756.79 total 100.06 last 77.64) - open/close 9087 (time 1787871766.79 total 110.06 last 76.69) - open/close 9915 (time 1787871776.81 total 120.07 last 82.70) total: 10000 open/close in 120.89 seconds: 82.72 ops/second llite.lustre-ffff93b16b61b800.statahead_batch_max=0 llite.lustre-ffff93b16b61b800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff93b16b61b800.stats=clear mdc.lustre-MDT0001-mdc-ffff93b16b61b800.stats=clear ls -l files in d123ad.sanity: 10001 real 1m6.322s user 0m0.344s sys 0m14.855s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff93b16b61b800.statahead_batch_max=256 llite.lustre-ffff93b16b61b800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff93b16b61b800.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff93b16b61b800.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff93b16b61b800.stats=clear mdc.lustre-MDT0001-mdc-ffff93b16b61b800.stats=clear ls -l files in d123ad.sanity: 10001 real 0m55.720s user 0m0.305s sys 0m14.154s batched statahead RPC= hit count=9999 mdc.lustre-MDT0000-mdc-ffff93b16b61b800.batch_stats= snapshot_time: 1787871919.584224514 secs.nsecs start_time: 1787871852.236132488 secs.nsecs elapsed_time: 67.348092026 secs.nsecs subreqs per batch batches %% cum %% 1: 45 29 29 2: 16 10 39 4: 14 9 48 8: 15 9 58 16: 4 2 61 32: 19 12 73 64: 4 2 75 128: 2 1 77 256: 35 22 100 mdc.lustre-MDT0001-mdc-ffff93b16b61b800.batch_stats= snapshot_time: 1787871919.584293337 secs.nsecs start_time: 1787871852.236266407 secs.nsecs elapsed_time: 67.348026930 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second seq.cli-lustre-OST0000-super.width=65536 seq.cli-lustre-OST0001-super.width=65536 llite.lustre-ffff93b16b61b800.statahead_batch_max=64 llite.lustre-ffff93b16b61b800.statahead_max=128