== sanity test 123ad: Verify batching statahead works correctly ========================================================== 08:58:20 (1787749100) batch: statahead_max=32 statahead_batch_max=32 mdc.lustre-MDT0000-mdc-ffff9b9d7ec39000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9b9d7ec39000.batch_stats=0 llite.lustre-ffff9b9d7ec39000.statahead_max=32 llite.lustre-ffff9b9d7ec39000.statahead_batch_max=32 seq.cli-lustre-OST0000-super.width=0x1ffffff seq.cli-lustre-OST0001-super.width=0x1ffffff kvm mdc.lustre-MDT0000-mdc-ffff9b9d7ec39000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9b9d7ec39000.batch_stats=0 striped dir -i1 -c2 -H crush2 /mnt/lustre/d123ad.sanity total: 100 open/close in 2.54 seconds: 39.37 ops/second llite.lustre-ffff9b9d7ec39000.statahead_max=0 101 real 0m2.165s user 0m0.006s sys 0m0.302s 'ls -l' 100 files without statahead: 2 sec llite.lustre-ffff9b9d7ec39000.statahead_max=32 llite.lustre-ffff9b9d7ec39000.statahead_max=32 101 real 0m1.188s user 0m0.004s sys 0m0.170s 'ls -l' 100 files with statahead: 1 sec statahead total: 118 statahead wrong: 2 agl total: 112 list_total: 31 fname_total: 79 hit_total: 3991 miss_total: 1383 swrong=2 ewrong=2 total: 900 open/close in 9.94 seconds: 90.56 ops/second llite.lustre-ffff9b9d7ec39000.statahead_max=0 1001 real 0m21.188s user 0m0.038s sys 0m2.747s 'ls -l' 1000 files without statahead: 21 sec llite.lustre-ffff9b9d7ec39000.statahead_max=32 llite.lustre-ffff9b9d7ec39000.statahead_max=32 1001 real 0m9.187s user 0m0.026s sys 0m1.384s 'ls -l' 1000 files with statahead: 9 sec statahead total: 119 statahead wrong: 2 agl total: 113 list_total: 32 fname_total: 79 hit_total: 4985 miss_total: 1389 swrong=2 ewrong=2 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 19 seconds llite.lustre-ffff9b9d7ec39000.statahead_stats= statahead total: 119 statahead wrong: 2 agl total: 113 list_total: 32 fname_total: 79 hit_total: 4985 miss_total: 1389 mdc.lustre-MDT0000-mdc-ffff9b9d7ec39000.batch_stats= snapshot_time: 1787749188.768834103 secs.nsecs start_time: 1787749105.461074138 secs.nsecs elapsed_time: 83.307759965 secs.nsecs subreqs per batch batches %% cum %% 1: 4 10 10 2: 2 5 15 4: 0 0 15 8: 3 7 23 16: 23 60 84 32: 6 15 100 mdc.lustre-MDT0001-mdc-ffff9b9d7ec39000.batch_stats= snapshot_time: 1787749188.768912135 secs.nsecs start_time: 1787749105.461193076 secs.nsecs elapsed_time: 83.307719059 secs.nsecs subreqs per batch batches %% cum %% 1: 1 2 2 2: 0 0 2 4: 1 2 5 8: 3 7 12 16: 13 32 45 32: 22 55 100 - open/close 761 (time 1787749201.73 total 10.01 last 76.00) - open/close 1544 (time 1787749211.74 total 20.03 last 78.19) - open/close 2453 (time 1787749221.75 total 30.04 last 90.80) - open/close 3352 (time 1787749231.77 total 40.05 last 89.76) - open/close 4259 (time 1787749241.77 total 50.06 last 90.68) - open/close 5061 (time 1787749251.77 total 60.06 last 80.19) - open/close 5917 (time 1787749261.78 total 70.07 last 85.50) - open/close 6783 (time 1787749271.78 total 80.07 last 86.59) - open/close 7394 (time 1787749281.79 total 90.08 last 61.07) - open/close 8264 (time 1787749291.79 total 100.08 last 86.96) - open/close 9077 (time 1787749301.80 total 110.09 last 81.25) - open/close 9929 (time 1787749311.80 total 120.09 last 85.16) total: 10000 open/close in 121.02 seconds: 82.63 ops/second llite.lustre-ffff9b9d7ec39000.statahead_batch_max=0 llite.lustre-ffff9b9d7ec39000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9b9d7ec39000.stats=clear mdc.lustre-MDT0001-mdc-ffff9b9d7ec39000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m2.375s user 0m0.275s sys 0m14.267s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff9b9d7ec39000.statahead_batch_max=32 llite.lustre-ffff9b9d7ec39000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9b9d7ec39000.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff9b9d7ec39000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff9b9d7ec39000.stats=clear mdc.lustre-MDT0001-mdc-ffff9b9d7ec39000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m53.133s user 0m0.288s sys 0m13.764s batched statahead RPC= hit count=9978 mdc.lustre-MDT0000-mdc-ffff9b9d7ec39000.batch_stats= snapshot_time: 1787749508.679573820 secs.nsecs start_time: 1787749383.950621083 secs.nsecs elapsed_time: 124.728952737 secs.nsecs subreqs per batch batches %% cum %% 1: 1 0 0 2: 0 0 0 4: 0 0 0 8: 1 0 0 16: 1 0 0 32: 313 99 100 mdc.lustre-MDT0001-mdc-ffff9b9d7ec39000.batch_stats= snapshot_time: 1787749508.679636122 secs.nsecs start_time: 1787749383.950762792 secs.nsecs elapsed_time: 124.728873330 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 batch: statahead_max=2048 statahead_batch_max=256 mdc.lustre-MDT0000-mdc-ffff9b9d7ec39000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9b9d7ec39000.batch_stats=0 llite.lustre-ffff9b9d7ec39000.statahead_max=2048 llite.lustre-ffff9b9d7ec39000.statahead_batch_max=256 kvm mdc.lustre-MDT0000-mdc-ffff9b9d7ec39000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9b9d7ec39000.batch_stats=0 striped dir -i1 -c2 -H all_char /mnt/lustre/d123ad.sanity total: 100 open/close in 2.85 seconds: 35.10 ops/second llite.lustre-ffff9b9d7ec39000.statahead_max=0 101 real 0m2.240s user 0m0.005s sys 0m0.301s 'ls -l' 100 files without statahead: 2 sec llite.lustre-ffff9b9d7ec39000.statahead_max=2048 llite.lustre-ffff9b9d7ec39000.statahead_max=2048 101 real 0m0.874s user 0m0.002s sys 0m0.166s 'ls -l' 100 files with statahead: 0 sec statahead total: 2 statahead wrong: 0 agl total: 2 list_total: 2 fname_total: 0 hit_total: 10077 miss_total: 23 swrong=0 ewrong=0 - open/close 676 (time 1787749804.31 total 10.01 last 67.55) total: 900 open/close in 12.93 seconds: 69.61 ops/second llite.lustre-ffff9b9d7ec39000.statahead_max=0 1001 real 0m24.406s user 0m0.038s sys 0m2.793s 'ls -l' 1000 files without statahead: 24 sec llite.lustre-ffff9b9d7ec39000.statahead_max=2048 llite.lustre-ffff9b9d7ec39000.statahead_max=2048 1001 real 0m6.758s user 0m0.025s sys 0m1.388s 'ls -l' 1000 files with statahead: 7 sec statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11076 miss_total: 24 swrong=0 ewrong=0 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 22 seconds llite.lustre-ffff9b9d7ec39000.statahead_stats= statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11076 miss_total: 24 mdc.lustre-MDT0000-mdc-ffff9b9d7ec39000.batch_stats= snapshot_time: 1787749873.201437680 secs.nsecs start_time: 1787749511.659246762 secs.nsecs elapsed_time: 361.542190918 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 3 42 42 8: 1 14 57 16: 0 0 57 32: 0 0 57 64: 1 14 71 128: 0 0 71 256: 2 28 100 mdc.lustre-MDT0001-mdc-ffff9b9d7ec39000.batch_stats= snapshot_time: 1787749873.201501728 secs.nsecs start_time: 1787749511.659380966 secs.nsecs elapsed_time: 361.542120762 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 1 14 14 8: 3 42 57 16: 0 0 57 32: 0 0 57 64: 1 14 71 128: 0 0 71 256: 2 28 100 - open/close 796 (time 1787749886.05 total 10.00 last 79.58) - open/close 1583 (time 1787749896.06 total 20.01 last 78.62) - open/close 2529 (time 1787749906.08 total 30.04 last 94.35) - open/close 3411 (time 1787749916.10 total 40.05 last 88.07) - open/close 4395 (time 1787749926.10 total 50.06 last 98.31) - open/close 5256 (time 1787749936.11 total 60.07 last 86.05) - open/close 6183 (time 1787749946.11 total 70.07 last 92.69) - open/close 7073 (time 1787749956.12 total 80.07 last 88.97) - open/close 7978 (time 1787749966.12 total 90.07 last 90.50) - open/close 8725 (time 1787749976.12 total 100.08 last 74.69) - open/close 9432 (time 1787749986.13 total 110.08 last 70.64) total: 10000 open/close in 118.76 seconds: 84.20 ops/second llite.lustre-ffff9b9d7ec39000.statahead_batch_max=0 llite.lustre-ffff9b9d7ec39000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9b9d7ec39000.stats=clear mdc.lustre-MDT0001-mdc-ffff9b9d7ec39000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m1.541s user 0m0.290s sys 0m14.784s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff9b9d7ec39000.statahead_batch_max=256 llite.lustre-ffff9b9d7ec39000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9b9d7ec39000.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff9b9d7ec39000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff9b9d7ec39000.stats=clear mdc.lustre-MDT0001-mdc-ffff9b9d7ec39000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m0.236s user 0m0.254s sys 0m13.741s batched statahead RPC= hit count=9999 mdc.lustre-MDT0000-mdc-ffff9b9d7ec39000.batch_stats= snapshot_time: 1787750137.613027707 secs.nsecs start_time: 1787750065.514661769 secs.nsecs elapsed_time: 72.098365938 secs.nsecs subreqs per batch batches %% cum %% 1: 75 40 40 2: 24 12 53 4: 11 5 59 8: 6 3 62 16: 8 4 67 32: 17 9 76 64: 8 4 80 128: 1 0 81 256: 35 18 100 mdc.lustre-MDT0001-mdc-ffff9b9d7ec39000.batch_stats= snapshot_time: 1787750137.613498034 secs.nsecs start_time: 1787750065.514739474 secs.nsecs elapsed_time: 72.098758560 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second seq.cli-lustre-OST0000-super.width=65536 seq.cli-lustre-OST0001-super.width=65536 llite.lustre-ffff9b9d7ec39000.statahead_batch_max=64 llite.lustre-ffff9b9d7ec39000.statahead_max=128