== sanity test 123ad: Verify batching statahead works correctly ========================================================== 05:04:39 (1787389479) batch: statahead_max=32 statahead_batch_max=32 mdc.lustre-MDT0000-mdc-ffff9788c8569800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9788c8569800.batch_stats=0 llite.lustre-ffff9788c8569800.statahead_max=32 llite.lustre-ffff9788c8569800.statahead_batch_max=32 seq.cli-lustre-OST0000-super.width=0x1ffffff seq.cli-lustre-OST0001-super.width=0x1ffffff kvm mdc.lustre-MDT0000-mdc-ffff9788c8569800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9788c8569800.batch_stats=0 striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d123ad.sanity total: 100 open/close in 2.65 seconds: 37.68 ops/second llite.lustre-ffff9788c8569800.statahead_max=0 101 real 0m2.575s user 0m0.009s sys 0m0.397s 'ls -l' 100 files without statahead: 3 sec llite.lustre-ffff9788c8569800.statahead_max=32 llite.lustre-ffff9788c8569800.statahead_max=32 101 real 0m1.073s user 0m0.005s sys 0m0.218s 'ls -l' 100 files with statahead: 1 sec statahead total: 145 statahead wrong: 2 agl total: 139 list_total: 30 fname_total: 91 hit_total: 4372 miss_total: 1264 swrong=2 ewrong=2 - open/close 702 (time 1787389507.25 total 10.01 last 70.16) total: 900 open/close in 12.87 seconds: 69.91 ops/second llite.lustre-ffff9788c8569800.statahead_max=0 1001 real 0m27.719s user 0m0.045s sys 0m3.443s 'ls -l' 1000 files without statahead: 28 sec llite.lustre-ffff9788c8569800.statahead_max=32 llite.lustre-ffff9788c8569800.statahead_max=32 1001 real 0m10.158s user 0m0.036s sys 0m1.640s 'ls -l' 1000 files with statahead: 11 sec statahead total: 146 statahead wrong: 2 agl total: 140 list_total: 31 fname_total: 91 hit_total: 5370 miss_total: 1266 swrong=2 ewrong=2 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 22 seconds llite.lustre-ffff9788c8569800.statahead_stats= statahead total: 146 statahead wrong: 2 agl total: 140 list_total: 31 fname_total: 91 hit_total: 5370 miss_total: 1266 mdc.lustre-MDT0000-mdc-ffff9788c8569800.batch_stats= snapshot_time: 1787389583.973602530 secs.nsecs start_time: 1787389484.189707642 secs.nsecs elapsed_time: 99.783894888 secs.nsecs subreqs per batch batches %% cum %% 1: 2 5 5 2: 0 0 5 4: 1 2 7 8: 5 12 20 16: 15 38 58 32: 16 41 100 mdc.lustre-MDT0001-mdc-ffff9788c8569800.batch_stats= snapshot_time: 1787389583.973690033 secs.nsecs start_time: 1787389484.190154887 secs.nsecs elapsed_time: 99.783535146 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 2 5 5 4: 1 2 7 8: 2 5 12 16: 23 58 71 32: 11 28 100 - open/close 730 (time 1787389596.82 total 10.00 last 73.00) - open/close 1510 (time 1787389606.82 total 20.00 last 77.99) - open/close 2286 (time 1787389616.83 total 30.01 last 77.55) - open/close 3067 (time 1787389626.85 total 40.03 last 77.95) - open/close 3783 (time 1787389636.87 total 50.05 last 71.45) - open/close 4559 (time 1787389646.87 total 60.05 last 77.56) - open/close 5386 (time 1787389656.88 total 70.06 last 82.67) - open/close 6169 (time 1787389666.89 total 80.07 last 78.20) - open/close 6924 (time 1787389676.90 total 90.08 last 75.43) - open/close 7673 (time 1787389686.91 total 100.09 last 74.81) - open/close 8365 (time 1787389696.92 total 110.10 last 69.18) - open/close 9031 (time 1787389706.92 total 120.10 last 66.59) - open/close 9769 (time 1787389716.92 total 130.10 last 73.76) total: 10000 open/close in 133.18 seconds: 75.09 ops/second llite.lustre-ffff9788c8569800.statahead_batch_max=0 llite.lustre-ffff9788c8569800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9788c8569800.stats=clear mdc.lustre-MDT0001-mdc-ffff9788c8569800.stats=clear ls -l files in d123ad.sanity: 10001 real 0m54.411s user 0m0.347s sys 0m20.991s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff9788c8569800.statahead_batch_max=32 llite.lustre-ffff9788c8569800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9788c8569800.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff9788c8569800.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff9788c8569800.stats=clear mdc.lustre-MDT0001-mdc-ffff9788c8569800.stats=clear ls -l files in d123ad.sanity: 10001 real 1m33.390s user 0m0.338s sys 0m17.165s batched statahead RPC= hit count=9987 mdc.lustre-MDT0000-mdc-ffff9788c8569800.batch_stats= snapshot_time: 1787389886.963012402 secs.nsecs start_time: 1787389783.188140951 secs.nsecs elapsed_time: 103.774871451 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 0 0 0 8: 1 0 0 16: 1 0 0 32: 313 99 100 mdc.lustre-MDT0001-mdc-ffff9788c8569800.batch_stats= snapshot_time: 1787389886.963064719 secs.nsecs start_time: 1787389783.188372878 secs.nsecs elapsed_time: 103.774691841 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 batch: statahead_max=2048 statahead_batch_max=256 mdc.lustre-MDT0000-mdc-ffff9788c8569800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9788c8569800.batch_stats=0 llite.lustre-ffff9788c8569800.statahead_max=2048 llite.lustre-ffff9788c8569800.statahead_batch_max=256 kvm mdc.lustre-MDT0000-mdc-ffff9788c8569800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9788c8569800.batch_stats=0 striped dir -i1 -c2 -H crush /mnt/lustre/d123ad.sanity total: 100 open/close in 2.80 seconds: 35.75 ops/second llite.lustre-ffff9788c8569800.statahead_max=0 101 real 0m2.458s user 0m0.006s sys 0m0.389s 'ls -l' 100 files without statahead: 2 sec llite.lustre-ffff9788c8569800.statahead_max=2048 llite.lustre-ffff9788c8569800.statahead_max=2048 101 real 0m0.777s user 0m0.004s sys 0m0.218s 'ls -l' 100 files with statahead: 0 sec statahead total: 2 statahead wrong: 0 agl total: 2 list_total: 2 fname_total: 0 hit_total: 10086 miss_total: 14 swrong=0 ewrong=0 - open/close 782 (time 1787390199.51 total 10.01 last 78.15) total: 900 open/close in 11.44 seconds: 78.66 ops/second llite.lustre-ffff9788c8569800.statahead_max=0 1001 real 0m21.941s user 0m0.027s sys 0m3.418s 'ls -l' 1000 files without statahead: 22 sec llite.lustre-ffff9788c8569800.statahead_max=2048 llite.lustre-ffff9788c8569800.statahead_max=2048 1001 real 0m7.121s user 0m0.048s sys 0m1.772s 'ls -l' 1000 files with statahead: 7 sec statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11085 miss_total: 15 swrong=0 ewrong=0 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 18 seconds llite.lustre-ffff9788c8569800.statahead_stats= statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11085 miss_total: 15 mdc.lustre-MDT0000-mdc-ffff9788c8569800.batch_stats= snapshot_time: 1787390258.588285262 secs.nsecs start_time: 1787389888.847530776 secs.nsecs elapsed_time: 369.740754486 secs.nsecs subreqs per batch batches %% cum %% 1: 1 14 14 2: 4 57 71 4: 0 0 71 8: 0 0 71 16: 0 0 71 32: 0 0 71 64: 0 0 71 128: 0 0 71 256: 2 28 100 mdc.lustre-MDT0001-mdc-ffff9788c8569800.batch_stats= snapshot_time: 1787390258.588443854 secs.nsecs start_time: 1787389888.847700667 secs.nsecs elapsed_time: 369.740743187 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 0 0 0 8: 3 30 30 16: 1 10 40 32: 4 40 80 64: 0 0 80 128: 0 0 80 256: 2 20 100 - open/close 736 (time 1787390270.93 total 10.01 last 73.50) - open/close 1474 (time 1787390280.93 total 20.02 last 73.78) - open/close 2237 (time 1787390290.94 total 30.02 last 76.28) - open/close 2973 (time 1787390300.94 total 40.02 last 73.58) - open/close 3787 (time 1787390310.96 total 50.04 last 81.27) - open/close 4560 (time 1787390320.97 total 60.05 last 77.19) - open/close 5331 (time 1787390330.97 total 70.05 last 77.09) - open/close 6118 (time 1787390340.98 total 80.06 last 78.64) - open/close 6857 (time 1787390350.98 total 90.06 last 73.88) - open/close 7669 (time 1787390360.98 total 100.07 last 81.19) - open/close 8470 (time 1787390371.00 total 110.08 last 79.99) - open/close 9285 (time 1787390381.00 total 120.08 last 81.47) - open/close 9953 (time 1787390391.01 total 130.09 last 66.74) total: 10000 open/close in 130.99 seconds: 76.34 ops/second llite.lustre-ffff9788c8569800.statahead_batch_max=0 llite.lustre-ffff9788c8569800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9788c8569800.stats=clear mdc.lustre-MDT0001-mdc-ffff9788c8569800.stats=clear ls -l files in d123ad.sanity: 10001 real 0m51.693s user 0m0.352s sys 0m20.937s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff9788c8569800.statahead_batch_max=256 llite.lustre-ffff9788c8569800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9788c8569800.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff9788c8569800.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff9788c8569800.stats=clear mdc.lustre-MDT0001-mdc-ffff9788c8569800.stats=clear ls -l files in d123ad.sanity: 10001 real 0m56.020s user 0m0.307s sys 0m18.644s batched statahead RPC= hit count=9999 mdc.lustre-MDT0000-mdc-ffff9788c8569800.batch_stats= snapshot_time: 1787390519.599516012 secs.nsecs start_time: 1787390452.717363147 secs.nsecs elapsed_time: 66.882152865 secs.nsecs subreqs per batch batches %% cum %% 1: 306 58 58 2: 77 14 73 4: 53 10 83 8: 24 4 87 16: 6 1 88 32: 19 3 92 64: 6 1 93 128: 0 0 93 256: 33 6 100 mdc.lustre-MDT0001-mdc-ffff9788c8569800.batch_stats= snapshot_time: 1787390519.599580941 secs.nsecs start_time: 1787390452.717516274 secs.nsecs elapsed_time: 66.882064667 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second seq.cli-lustre-OST0000-super.width=65536 seq.cli-lustre-OST0001-super.width=65536 llite.lustre-ffff9788c8569800.statahead_batch_max=64 llite.lustre-ffff9788c8569800.statahead_max=128