== sanity test 123ad: Verify batching statahead works correctly ========================================================== 23:51:34 (1789271494) batch: statahead_max=32 statahead_batch_max=32 mdc.lustre-MDT0000-mdc-ffff8bd43b922000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff8bd43b922000.batch_stats=0 llite.lustre-ffff8bd43b922000.statahead_max=32 llite.lustre-ffff8bd43b922000.statahead_batch_max=32 seq.cli-lustre-OST0000-super.width=0x1ffffff seq.cli-lustre-OST0001-super.width=0x1ffffff kvm mdc.lustre-MDT0000-mdc-ffff8bd43b922000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff8bd43b922000.batch_stats=0 striped dir -i1 -c2 -H all_char /mnt/lustre/d123ad.sanity total: 100 open/close in 2.87 seconds: 34.90 ops/second llite.lustre-ffff8bd43b922000.statahead_max=0 101 real 0m2.703s user 0m0.004s sys 0m0.331s 'ls -l' 100 files without statahead: 3 sec llite.lustre-ffff8bd43b922000.statahead_max=32 llite.lustre-ffff8bd43b922000.statahead_max=32 101 real 0m1.076s user 0m0.004s sys 0m0.176s 'ls -l' 100 files with statahead: 1 sec statahead total: 121 statahead wrong: 2 agl total: 115 list_total: 31 fname_total: 80 hit_total: 3701 miss_total: 1639 swrong=2 ewrong=2 - open/close 727 (time 1789271523.59 total 10.02 last 72.55) total: 900 open/close in 12.73 seconds: 70.72 ops/second llite.lustre-ffff8bd43b922000.statahead_max=0 1001 real 0m22.236s user 0m0.045s sys 0m3.084s 'ls -l' 1000 files without statahead: 22 sec llite.lustre-ffff8bd43b922000.statahead_max=32 llite.lustre-ffff8bd43b922000.statahead_max=32 1001 real 0m10.220s user 0m0.029s sys 0m1.499s 'ls -l' 1000 files with statahead: 11 sec statahead total: 122 statahead wrong: 2 agl total: 116 list_total: 32 fname_total: 80 hit_total: 4699 miss_total: 1641 swrong=2 ewrong=2 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 25 seconds llite.lustre-ffff8bd43b922000.statahead_stats= statahead total: 122 statahead wrong: 2 agl total: 116 list_total: 32 fname_total: 80 hit_total: 4699 miss_total: 1641 mdc.lustre-MDT0000-mdc-ffff8bd43b922000.batch_stats= snapshot_time: 1789271597.285831707 secs.nsecs start_time: 1789271500.320785044 secs.nsecs elapsed_time: 96.965046663 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 3 7 7 8: 2 5 13 16: 21 55 68 32: 12 31 100 mdc.lustre-MDT0001-mdc-ffff8bd43b922000.batch_stats= snapshot_time: 1789271597.288384940 secs.nsecs start_time: 1789271500.320920502 secs.nsecs elapsed_time: 96.967464438 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 1 2 2 8: 5 12 15 16: 20 51 66 32: 13 33 100 - open/close 628 (time 1789271610.14 total 10.00 last 62.79) - open/close 1331 (time 1789271620.16 total 20.02 last 70.18) - open/close 2064 (time 1789271630.17 total 30.03 last 73.23) - open/close 2716 (time 1789271640.19 total 40.05 last 65.06) - open/close 3450 (time 1789271650.20 total 50.06 last 73.32) - open/close 4131 (time 1789271660.21 total 60.07 last 68.07) - open/close 4913 (time 1789271670.22 total 70.08 last 78.08) - open/close 5634 (time 1789271680.23 total 80.09 last 72.03) - open/close 6369 (time 1789271690.25 total 90.10 last 73.41) - open/close 7025 (time 1789271700.26 total 100.12 last 65.51) - open/close 7543 (time 1789271710.27 total 110.13 last 51.75) - open/close 8107 (time 1789271720.27 total 120.13 last 56.38) - open/close 8842 (time 1789271730.29 total 130.14 last 73.39) - open/close 9538 (time 1789271740.30 total 140.16 last 69.53) total: 10000 open/close in 146.34 seconds: 68.33 ops/second llite.lustre-ffff8bd43b922000.statahead_batch_max=0 llite.lustre-ffff8bd43b922000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff8bd43b922000.stats=clear mdc.lustre-MDT0001-mdc-ffff8bd43b922000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m9.422s user 0m0.343s sys 0m15.281s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff8bd43b922000.statahead_batch_max=32 llite.lustre-ffff8bd43b922000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff8bd43b922000.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff8bd43b922000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff8bd43b922000.stats=clear mdc.lustre-MDT0001-mdc-ffff8bd43b922000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m52.042s user 0m0.307s sys 0m14.561s batched statahead RPC= hit count=9982 mdc.lustre-MDT0000-mdc-ffff8bd43b922000.batch_stats= snapshot_time: 1789271948.430034168 secs.nsecs start_time: 1789271825.681492030 secs.nsecs elapsed_time: 122.748542138 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 0 0 0 8: 1 0 0 16: 1 0 0 32: 313 99 100 mdc.lustre-MDT0001-mdc-ffff8bd43b922000.batch_stats= snapshot_time: 1789271948.430105056 secs.nsecs start_time: 1789271825.681630176 secs.nsecs elapsed_time: 122.748474880 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 batch: statahead_max=2048 statahead_batch_max=256 mdc.lustre-MDT0000-mdc-ffff8bd43b922000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff8bd43b922000.batch_stats=0 llite.lustre-ffff8bd43b922000.statahead_max=2048 llite.lustre-ffff8bd43b922000.statahead_batch_max=256 kvm mdc.lustre-MDT0000-mdc-ffff8bd43b922000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff8bd43b922000.batch_stats=0 striped dir -i1 -c2 -H crush2 /mnt/lustre/d123ad.sanity total: 100 open/close in 3.28 seconds: 30.52 ops/second llite.lustre-ffff8bd43b922000.statahead_max=0 101 real 0m2.857s user 0m0.006s sys 0m0.342s 'ls -l' 100 files without statahead: 3 sec llite.lustre-ffff8bd43b922000.statahead_max=2048 llite.lustre-ffff8bd43b922000.statahead_max=2048 101 real 0m0.923s user 0m0.004s sys 0m0.160s 'ls -l' 100 files with statahead: 1 sec statahead total: 2 statahead wrong: 0 agl total: 2 list_total: 2 fname_total: 0 hit_total: 10081 miss_total: 19 swrong=0 ewrong=0 - open/close 768 (time 1789272285.65 total 10.01 last 76.75) total: 900 open/close in 11.72 seconds: 76.79 ops/second llite.lustre-ffff8bd43b922000.statahead_max=0 1001 real 0m23.684s user 0m0.045s sys 0m2.998s 'ls -l' 1000 files without statahead: 24 sec llite.lustre-ffff8bd43b922000.statahead_max=2048 llite.lustre-ffff8bd43b922000.statahead_max=2048 1001 real 0m5.496s user 0m0.026s sys 0m1.321s 'ls -l' 1000 files with statahead: 5 sec statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11080 miss_total: 20 swrong=0 ewrong=0 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 20 seconds llite.lustre-ffff8bd43b922000.statahead_stats= statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11080 miss_total: 20 mdc.lustre-MDT0000-mdc-ffff8bd43b922000.batch_stats= snapshot_time: 1789272348.796814026 secs.nsecs start_time: 1789271950.979755218 secs.nsecs elapsed_time: 397.817058808 secs.nsecs subreqs per batch batches %% cum %% 1: 1 20 20 2: 0 0 20 4: 2 40 60 8: 0 0 60 16: 0 0 60 32: 0 0 60 64: 0 0 60 128: 0 0 60 256: 2 40 100 mdc.lustre-MDT0001-mdc-ffff8bd43b922000.batch_stats= snapshot_time: 1789272348.796886681 secs.nsecs start_time: 1789271950.979900768 secs.nsecs elapsed_time: 397.816985913 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 0 0 0 8: 2 25 25 16: 1 12 37 32: 1 12 50 64: 2 25 75 128: 0 0 75 256: 2 25 100 - open/close 707 (time 1789272361.29 total 10.01 last 70.65) - open/close 1434 (time 1789272371.29 total 20.01 last 72.69) - open/close 2280 (time 1789272381.29 total 30.01 last 84.58) - open/close 3038 (time 1789272391.31 total 40.03 last 75.69) - open/close 3903 (time 1789272401.31 total 50.03 last 86.46) - open/close 4744 (time 1789272411.32 total 60.03 last 84.06) - open/close 5473 (time 1789272421.32 total 70.04 last 72.88) - open/close 6239 (time 1789272431.34 total 80.06 last 76.43) - open/close 7086 (time 1789272441.35 total 90.06 last 84.67) - open/close 7977 (time 1789272451.35 total 100.07 last 89.05) - open/close 8801 (time 1789272461.35 total 110.07 last 82.37) - open/close 9665 (time 1789272471.35 total 120.07 last 86.40) total: 10000 open/close in 123.74 seconds: 80.81 ops/second llite.lustre-ffff8bd43b922000.statahead_batch_max=0 llite.lustre-ffff8bd43b922000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff8bd43b922000.stats=clear mdc.lustre-MDT0001-mdc-ffff8bd43b922000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m4.023s user 0m0.332s sys 0m15.289s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff8bd43b922000.statahead_batch_max=256 llite.lustre-ffff8bd43b922000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff8bd43b922000.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff8bd43b922000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff8bd43b922000.stats=clear mdc.lustre-MDT0001-mdc-ffff8bd43b922000.stats=clear ls -l files in d123ad.sanity: 10001 real 0m51.805s user 0m0.302s sys 0m15.237s batched statahead RPC= hit count=9999 mdc.lustre-MDT0000-mdc-ffff8bd43b922000.batch_stats= snapshot_time: 1789272608.904043840 secs.nsecs start_time: 1789272548.079072110 secs.nsecs elapsed_time: 60.824971730 secs.nsecs subreqs per batch batches %% cum %% 1: 148 44 44 2: 47 14 58 4: 41 12 70 8: 27 8 78 16: 13 3 82 32: 16 4 87 64: 7 2 89 128: 3 0 90 256: 33 9 100 mdc.lustre-MDT0001-mdc-ffff8bd43b922000.batch_stats= snapshot_time: 1789272608.904139688 secs.nsecs start_time: 1789272548.079419113 secs.nsecs elapsed_time: 60.824720575 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second seq.cli-lustre-OST0000-super.width=65536 seq.cli-lustre-OST0001-super.width=65536 llite.lustre-ffff8bd43b922000.statahead_batch_max=64 llite.lustre-ffff8bd43b922000.statahead_max=128