== sanity test 123ad: Verify batching statahead works correctly ========================================================== 10:46:52 (1787842012) batch: statahead_max=32 statahead_batch_max=32 mdc.lustre-MDT0000-mdc-ffff97d411df4000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff97d411df4000.batch_stats=0 llite.lustre-ffff97d411df4000.statahead_max=32 llite.lustre-ffff97d411df4000.statahead_batch_max=32 seq.cli-lustre-OST0000-super.width=0x1ffffff seq.cli-lustre-OST0001-super.width=0x1ffffff kvm mdc.lustre-MDT0000-mdc-ffff97d411df4000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff97d411df4000.batch_stats=0 striped dir -i1 -c2 -H crush2 /mnt/lustre/d123ad.sanity total: 100 open/close in 2.87 seconds: 34.90 ops/second llite.lustre-ffff97d411df4000.statahead_max=0 101 real 0m2.280s user 0m0.005s sys 0m0.314s 'ls -l' 100 files without statahead: 2 sec llite.lustre-ffff97d411df4000.statahead_max=32 llite.lustre-ffff97d411df4000.statahead_max=32 101 real 0m1.372s user 0m0.004s sys 0m0.162s 'ls -l' 100 files with statahead: 1 sec statahead total: 108 statahead wrong: 2 agl total: 102 list_total: 31 fname_total: 77 hit_total: 3723 miss_total: 1354 swrong=2 ewrong=2 - open/close 751 (time 1787842042.16 total 10.00 last 75.08) total: 900 open/close in 11.86 seconds: 75.86 ops/second llite.lustre-ffff97d411df4000.statahead_max=0 1001 real 0m22.268s user 0m0.040s sys 0m2.862s 'ls -l' 1000 files without statahead: 23 sec llite.lustre-ffff97d411df4000.statahead_max=32 llite.lustre-ffff97d411df4000.statahead_max=32 1001 real 0m9.084s user 0m0.027s sys 0m1.474s 'ls -l' 1000 files with statahead: 9 sec statahead total: 109 statahead wrong: 2 agl total: 103 list_total: 32 fname_total: 77 hit_total: 4720 miss_total: 1357 swrong=2 ewrong=2 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 18 seconds llite.lustre-ffff97d411df4000.statahead_stats= statahead total: 109 statahead wrong: 2 agl total: 103 list_total: 32 fname_total: 77 hit_total: 4720 miss_total: 1357 mdc.lustre-MDT0000-mdc-ffff97d411df4000.batch_stats= snapshot_time: 1787842105.325055085 secs.nsecs start_time: 1787842017.978689981 secs.nsecs elapsed_time: 87.346365104 secs.nsecs subreqs per batch batches %% cum %% 1: 3 7 7 2: 3 7 15 4: 1 2 18 8: 1 2 21 16: 26 68 89 32: 4 10 100 mdc.lustre-MDT0001-mdc-ffff97d411df4000.batch_stats= snapshot_time: 1787842105.325125424 secs.nsecs start_time: 1787842017.982598030 secs.nsecs elapsed_time: 87.342527394 secs.nsecs subreqs per batch batches %% cum %% 1: 2 4 4 2: 1 2 7 4: 0 0 7 8: 4 9 17 16: 12 29 46 32: 22 53 100 - open/close 818 (time 1787842117.94 total 10.01 last 81.74) - open/close 1619 (time 1787842127.94 total 20.01 last 80.10) - open/close 2452 (time 1787842137.96 total 30.03 last 83.15) - open/close 3255 (time 1787842147.97 total 40.03 last 80.26) - open/close 4089 (time 1787842157.98 total 50.04 last 83.29) - open/close 4989 (time 1787842167.99 total 60.05 last 89.93) - open/close 5768 (time 1787842177.99 total 70.06 last 77.84) - open/close 6609 (time 1787842188.00 total 80.06 last 84.09) - open/close 7420 (time 1787842198.01 total 90.08 last 80.96) - open/close 8226 (time 1787842208.03 total 100.10 last 80.46) - open/close 9102 (time 1787842218.03 total 110.10 last 87.57) - open/close 9904 (time 1787842228.04 total 120.10 last 80.17) total: 10000 open/close in 121.16 seconds: 82.53 ops/second llite.lustre-ffff97d411df4000.statahead_batch_max=0 llite.lustre-ffff97d411df4000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff97d411df4000.stats=clear mdc.lustre-MDT0001-mdc-ffff97d411df4000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m2.880s user 0m0.314s sys 0m15.282s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff97d411df4000.statahead_batch_max=32 llite.lustre-ffff97d411df4000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff97d411df4000.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff97d411df4000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff97d411df4000.stats=clear mdc.lustre-MDT0001-mdc-ffff97d411df4000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m40.941s user 0m0.306s sys 0m13.809s batched statahead RPC= hit count=9984 mdc.lustre-MDT0000-mdc-ffff97d411df4000.batch_stats= snapshot_time: 1787842408.709229379 secs.nsecs start_time: 1787842299.400378212 secs.nsecs elapsed_time: 109.308851167 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 0 0 0 8: 1 0 0 16: 1 0 0 32: 313 99 100 mdc.lustre-MDT0001-mdc-ffff97d411df4000.batch_stats= snapshot_time: 1787842408.709293733 secs.nsecs start_time: 1787842299.400466444 secs.nsecs elapsed_time: 109.308827289 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 batch: statahead_max=2048 statahead_batch_max=256 mdc.lustre-MDT0000-mdc-ffff97d411df4000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff97d411df4000.batch_stats=0 llite.lustre-ffff97d411df4000.statahead_max=2048 llite.lustre-ffff97d411df4000.statahead_batch_max=256 kvm mdc.lustre-MDT0000-mdc-ffff97d411df4000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff97d411df4000.batch_stats=0 striped dir -i1 -c2 -H crush2 /mnt/lustre/d123ad.sanity total: 100 open/close in 2.63 seconds: 38.00 ops/second llite.lustre-ffff97d411df4000.statahead_max=0 101 real 0m2.857s user 0m0.003s sys 0m0.316s 'ls -l' 100 files without statahead: 3 sec llite.lustre-ffff97d411df4000.statahead_max=2048 llite.lustre-ffff97d411df4000.statahead_max=2048 101 real 0m0.880s user 0m0.010s sys 0m0.144s 'ls -l' 100 files with statahead: 1 sec statahead total: 2 statahead wrong: 0 agl total: 2 list_total: 2 fname_total: 0 hit_total: 10083 miss_total: 17 swrong=0 ewrong=0 - open/close 822 (time 1787842709.88 total 10.01 last 82.12) total: 900 open/close in 10.93 seconds: 82.38 ops/second llite.lustre-ffff97d411df4000.statahead_max=0 1001 real 0m21.645s user 0m0.038s sys 0m2.878s 'ls -l' 1000 files without statahead: 21 sec llite.lustre-ffff97d411df4000.statahead_max=2048 llite.lustre-ffff97d411df4000.statahead_max=2048 1001 real 0m6.222s user 0m0.027s sys 0m1.443s 'ls -l' 1000 files with statahead: 6 sec statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11082 miss_total: 18 swrong=0 ewrong=0 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 20 seconds llite.lustre-ffff97d411df4000.statahead_stats= statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11082 miss_total: 18 mdc.lustre-MDT0000-mdc-ffff97d411df4000.batch_stats= snapshot_time: 1787842770.162233161 secs.nsecs start_time: 1787842411.036013111 secs.nsecs elapsed_time: 359.126220050 secs.nsecs subreqs per batch batches %% cum %% 1: 1 14 14 2: 1 14 28 4: 2 28 57 8: 1 14 71 16: 0 0 71 32: 0 0 71 64: 0 0 71 128: 0 0 71 256: 2 28 100 mdc.lustre-MDT0001-mdc-ffff97d411df4000.batch_stats= snapshot_time: 1787842770.164332794 secs.nsecs start_time: 1787842411.036454164 secs.nsecs elapsed_time: 359.127878630 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 0 0 0 8: 4 40 40 16: 1 10 50 32: 2 20 70 64: 1 10 80 128: 0 0 80 256: 2 20 100 - open/close 845 (time 1787842782.64 total 10.01 last 84.44) - open/close 1640 (time 1787842792.65 total 20.02 last 79.42) - open/close 2506 (time 1787842802.66 total 30.02 last 86.54) - open/close 3290 (time 1787842812.67 total 40.03 last 78.33) - open/close 4113 (time 1787842822.68 total 50.04 last 82.25) - open/close 4959 (time 1787842832.68 total 60.04 last 84.60) - open/close 5809 (time 1787842842.68 total 70.05 last 84.94) - open/close 6665 (time 1787842852.69 total 80.05 last 85.57) - open/close 7515 (time 1787842862.70 total 90.06 last 84.91) - open/close 8365 (time 1787842872.71 total 100.07 last 84.89) - open/close 9168 (time 1787842882.72 total 110.08 last 80.25) total: 10000 open/close in 119.02 seconds: 84.02 ops/second llite.lustre-ffff97d411df4000.statahead_batch_max=0 llite.lustre-ffff97d411df4000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff97d411df4000.stats=clear mdc.lustre-MDT0001-mdc-ffff97d411df4000.stats=clear ls -l files in d123ad.sanity: 10001 real 0m59.111s user 0m0.308s sys 0m15.333s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff97d411df4000.statahead_batch_max=256 llite.lustre-ffff97d411df4000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff97d411df4000.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff97d411df4000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff97d411df4000.stats=clear mdc.lustre-MDT0001-mdc-ffff97d411df4000.stats=clear ls -l files in d123ad.sanity: 10001 real 0m52.458s user 0m0.316s sys 0m15.246s batched statahead RPC= hit count=9999 mdc.lustre-MDT0000-mdc-ffff97d411df4000.batch_stats= snapshot_time: 1787843021.014564155 secs.nsecs start_time: 1787842959.122283054 secs.nsecs elapsed_time: 61.892281101 secs.nsecs subreqs per batch batches %% cum %% 1: 77 42 42 2: 19 10 52 4: 13 7 59 8: 10 5 65 16: 4 2 67 32: 9 4 72 64: 12 6 79 128: 4 2 81 256: 34 18 100 mdc.lustre-MDT0001-mdc-ffff97d411df4000.batch_stats= snapshot_time: 1787843021.014633272 secs.nsecs start_time: 1787842959.122357162 secs.nsecs elapsed_time: 61.892276110 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second seq.cli-lustre-OST0000-super.width=65536 seq.cli-lustre-OST0001-super.width=65536 llite.lustre-ffff97d411df4000.statahead_batch_max=64 llite.lustre-ffff97d411df4000.statahead_max=128