== sanity test 123ad: Verify batching statahead works correctly ========================================================== 08:28:34 (1790684914) batch: statahead_max=32 statahead_batch_max=32 mdc.lustre-MDT0000-mdc-ffff9b46c54d7800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9b46c54d7800.batch_stats=0 llite.lustre-ffff9b46c54d7800.statahead_max=32 llite.lustre-ffff9b46c54d7800.statahead_batch_max=32 seq.cli-lustre-OST0000-super.width=0x1ffffff seq.cli-lustre-OST0001-super.width=0x1ffffff kvm mdc.lustre-MDT0000-mdc-ffff9b46c54d7800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9b46c54d7800.batch_stats=0 striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d123ad.sanity total: 100 open/close in 3.48 seconds: 28.70 ops/second llite.lustre-ffff9b46c54d7800.statahead_max=0 101 real 0m3.102s user 0m0.005s sys 0m0.354s 'ls -l' 100 files without statahead: 3 sec llite.lustre-ffff9b46c54d7800.statahead_max=32 llite.lustre-ffff9b46c54d7800.statahead_max=32 101 real 0m1.453s user 0m0.002s sys 0m0.188s 'ls -l' 100 files with statahead: 1 sec statahead total: 136 statahead wrong: 2 agl total: 130 list_total: 32 fname_total: 97 hit_total: 3830 miss_total: 1772 swrong=2 ewrong=2 - open/close 777 (time 1790684950.43 total 10.00 last 77.69) total: 900 open/close in 11.55 seconds: 77.92 ops/second llite.lustre-ffff9b46c54d7800.statahead_max=0 1001 real 0m23.997s user 0m0.034s sys 0m3.186s 'ls -l' 1000 files without statahead: 24 sec llite.lustre-ffff9b46c54d7800.statahead_max=32 llite.lustre-ffff9b46c54d7800.statahead_max=32 1001 real 0m10.355s user 0m0.029s sys 0m1.459s 'ls -l' 1000 files with statahead: 10 sec statahead total: 137 statahead wrong: 2 agl total: 131 list_total: 33 fname_total: 97 hit_total: 4824 miss_total: 1778 swrong=2 ewrong=2 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 22 seconds llite.lustre-ffff9b46c54d7800.statahead_stats= statahead total: 137 statahead wrong: 2 agl total: 131 list_total: 33 fname_total: 97 hit_total: 4824 miss_total: 1778 mdc.lustre-MDT0000-mdc-ffff9b46c54d7800.batch_stats= snapshot_time: 1790685022.257016780 secs.nsecs start_time: 1790684923.209212134 secs.nsecs elapsed_time: 99.047804646 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 1 2 2 4: 2 5 7 8: 2 5 12 16: 19 48 61 32: 15 38 100 mdc.lustre-MDT0001-mdc-ffff9b46c54d7800.batch_stats= snapshot_time: 1790685022.257102161 secs.nsecs start_time: 1790684923.209400594 secs.nsecs elapsed_time: 99.047701567 secs.nsecs subreqs per batch batches %% cum %% 1: 1 2 2 2: 1 2 5 4: 2 5 10 8: 3 7 17 16: 22 55 72 32: 11 27 100 - open/close 898 (time 1790685035.28 total 10.01 last 89.69) - open/close 1789 (time 1790685045.29 total 20.03 last 88.97) - open/close 2585 (time 1790685055.29 total 30.03 last 79.60) - open/close 3345 (time 1790685065.30 total 40.04 last 75.93) - open/close 4170 (time 1790685075.31 total 50.04 last 82.46) - open/close 4890 (time 1790685085.31 total 60.05 last 71.94) - open/close 5590 (time 1790685095.34 total 70.07 last 69.84) - open/close 6166 (time 1790685105.35 total 80.09 last 57.51) - open/close 6989 (time 1790685115.37 total 90.10 last 82.19) - open/close 7736 (time 1790685125.38 total 100.12 last 74.60) - open/close 8521 (time 1790685135.39 total 110.12 last 78.45) - open/close 9257 (time 1790685145.40 total 120.13 last 73.53) total: 10000 open/close in 128.69 seconds: 77.70 ops/second llite.lustre-ffff9b46c54d7800.statahead_batch_max=0 llite.lustre-ffff9b46c54d7800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9b46c54d7800.stats=clear mdc.lustre-MDT0001-mdc-ffff9b46c54d7800.stats=clear ls -l files in d123ad.sanity: 10001 real 1m7.368s user 0m0.360s sys 0m15.172s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff9b46c54d7800.statahead_batch_max=32 llite.lustre-ffff9b46c54d7800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9b46c54d7800.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff9b46c54d7800.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff9b46c54d7800.stats=clear mdc.lustre-MDT0001-mdc-ffff9b46c54d7800.stats=clear ls -l files in d123ad.sanity: 10001 real 1m52.664s user 0m0.397s sys 0m15.102s batched statahead RPC= hit count=9978 mdc.lustre-MDT0000-mdc-ffff9b46c54d7800.batch_stats= snapshot_time: 1790685359.672926123 secs.nsecs start_time: 1790685231.629493368 secs.nsecs elapsed_time: 128.043432755 secs.nsecs subreqs per batch batches %% cum %% 1: 1 0 0 2: 0 0 0 4: 0 0 0 8: 1 0 0 16: 1 0 0 32: 313 99 100 mdc.lustre-MDT0001-mdc-ffff9b46c54d7800.batch_stats= snapshot_time: 1790685359.673023304 secs.nsecs start_time: 1790685231.629888136 secs.nsecs elapsed_time: 128.043135168 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 batch: statahead_max=2048 statahead_batch_max=256 mdc.lustre-MDT0000-mdc-ffff9b46c54d7800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9b46c54d7800.batch_stats=0 llite.lustre-ffff9b46c54d7800.statahead_max=2048 llite.lustre-ffff9b46c54d7800.statahead_batch_max=256 kvm mdc.lustre-MDT0000-mdc-ffff9b46c54d7800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9b46c54d7800.batch_stats=0 striped dir -i1 -c2 -H crush /mnt/lustre/d123ad.sanity total: 100 open/close in 3.15 seconds: 31.76 ops/second llite.lustre-ffff9b46c54d7800.statahead_max=0 101 real 0m3.506s user 0m0.006s sys 0m0.341s 'ls -l' 100 files without statahead: 3 sec llite.lustre-ffff9b46c54d7800.statahead_max=2048 llite.lustre-ffff9b46c54d7800.statahead_max=2048 101 real 0m1.422s user 0m0.006s sys 0m0.207s 'ls -l' 100 files with statahead: 2 sec statahead total: 2 statahead wrong: 0 agl total: 2 list_total: 2 fname_total: 0 hit_total: 10077 miss_total: 23 swrong=0 ewrong=0 total: 900 open/close in 9.05 seconds: 99.48 ops/second llite.lustre-ffff9b46c54d7800.statahead_max=0 1001 real 0m20.301s user 0m0.035s sys 0m2.875s 'ls -l' 1000 files without statahead: 21 sec llite.lustre-ffff9b46c54d7800.statahead_max=2048 llite.lustre-ffff9b46c54d7800.statahead_max=2048 1001 real 0m7.476s user 0m0.035s sys 0m1.570s 'ls -l' 1000 files with statahead: 8 sec statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11076 miss_total: 24 swrong=0 ewrong=0 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 17 seconds llite.lustre-ffff9b46c54d7800.statahead_stats= statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11076 miss_total: 24 mdc.lustre-MDT0000-mdc-ffff9b46c54d7800.batch_stats= snapshot_time: 1790685736.267060522 secs.nsecs start_time: 1790685363.339063185 secs.nsecs elapsed_time: 372.927997337 secs.nsecs subreqs per batch batches %% cum %% 1: 2 25 25 2: 1 12 37 4: 2 25 62 8: 1 12 75 16: 0 0 75 32: 0 0 75 64: 0 0 75 128: 0 0 75 256: 2 25 100 mdc.lustre-MDT0001-mdc-ffff9b46c54d7800.batch_stats= snapshot_time: 1790685736.267137886 secs.nsecs start_time: 1790685363.339229501 secs.nsecs elapsed_time: 372.927908385 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 0 0 0 8: 3 30 30 16: 3 30 60 32: 1 10 70 64: 1 10 80 128: 0 0 80 256: 2 20 100 - open/close 842 (time 1790685749.36 total 10.02 last 84.01) - open/close 1718 (time 1790685759.36 total 20.02 last 87.58) - open/close 2568 (time 1790685769.37 total 30.03 last 84.99) - open/close 3444 (time 1790685779.37 total 40.03 last 87.56) - open/close 4276 (time 1790685789.38 total 50.04 last 83.14) - open/close 4923 (time 1790685799.38 total 60.04 last 64.68) - open/close 5725 (time 1790685809.39 total 70.05 last 80.15) - open/close 6486 (time 1790685819.40 total 80.06 last 75.99) - open/close 7122 (time 1790685829.41 total 90.07 last 63.53) - open/close 7824 (time 1790685839.42 total 100.08 last 70.17) - open/close 8687 (time 1790685849.42 total 110.08 last 86.26) - open/close 9587 (time 1790685859.42 total 120.08 last 90.00) total: 10000 open/close in 124.93 seconds: 80.04 ops/second llite.lustre-ffff9b46c54d7800.statahead_batch_max=0 llite.lustre-ffff9b46c54d7800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9b46c54d7800.stats=clear mdc.lustre-MDT0001-mdc-ffff9b46c54d7800.stats=clear ls -l files in d123ad.sanity: 10001 real 0m59.709s user 0m0.334s sys 0m14.251s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff9b46c54d7800.statahead_batch_max=256 llite.lustre-ffff9b46c54d7800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9b46c54d7800.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff9b46c54d7800.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff9b46c54d7800.stats=clear mdc.lustre-MDT0001-mdc-ffff9b46c54d7800.stats=clear ls -l files in d123ad.sanity: 10001 real 0m52.579s user 0m0.331s sys 0m13.944s batched statahead RPC= hit count=9999 mdc.lustre-MDT0000-mdc-ffff9b46c54d7800.batch_stats= snapshot_time: 1790685993.831834402 secs.nsecs start_time: 1790685932.665782935 secs.nsecs elapsed_time: 61.166051467 secs.nsecs subreqs per batch batches %% cum %% 1: 124 44 44 2: 36 12 56 4: 31 11 67 8: 15 5 73 16: 13 4 77 32: 21 7 85 64: 7 2 87 128: 0 0 87 256: 34 12 100 mdc.lustre-MDT0001-mdc-ffff9b46c54d7800.batch_stats= snapshot_time: 1790685993.831911220 secs.nsecs start_time: 1790685932.665872058 secs.nsecs elapsed_time: 61.166039162 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second seq.cli-lustre-OST0000-super.width=65536 seq.cli-lustre-OST0001-super.width=65536 llite.lustre-ffff9b46c54d7800.statahead_batch_max=64 llite.lustre-ffff9b46c54d7800.statahead_max=128