== sanity test 123ad: Verify batching statahead works correctly ========================================================== 19:56:57 (1788566217) batch: statahead_max=32 statahead_batch_max=32 mdc.lustre-MDT0000-mdc-ffff95301a1a1800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff95301a1a1800.batch_stats=0 llite.lustre-ffff95301a1a1800.statahead_max=32 llite.lustre-ffff95301a1a1800.statahead_batch_max=32 seq.cli-lustre-OST0000-super.width=0x1ffffff seq.cli-lustre-OST0001-super.width=0x1ffffff kvm mdc.lustre-MDT0000-mdc-ffff95301a1a1800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff95301a1a1800.batch_stats=0 striped dir -i1 -c2 -H all_char /mnt/lustre/d123ad.sanity total: 100 open/close in 2.76 seconds: 36.28 ops/second llite.lustre-ffff95301a1a1800.statahead_max=0 101 real 0m2.137s user 0m0.003s sys 0m0.314s 'ls -l' 100 files without statahead: 3 sec llite.lustre-ffff95301a1a1800.statahead_max=32 llite.lustre-ffff95301a1a1800.statahead_max=32 101 real 0m1.221s user 0m0.006s sys 0m0.170s 'ls -l' 100 files with statahead: 1 sec statahead total: 143 statahead wrong: 2 agl total: 136 list_total: 32 fname_total: 95 hit_total: 3748 miss_total: 1670 swrong=2 ewrong=2 - open/close 756 (time 1788566246.53 total 10.00 last 75.60) total: 900 open/close in 11.94 seconds: 75.38 ops/second llite.lustre-ffff95301a1a1800.statahead_max=0 1001 real 0m22.984s user 0m0.040s sys 0m3.010s 'ls -l' 1000 files without statahead: 23 sec llite.lustre-ffff95301a1a1800.statahead_max=32 llite.lustre-ffff95301a1a1800.statahead_max=32 1001 real 0m12.465s user 0m0.032s sys 0m1.400s 'ls -l' 1000 files with statahead: 12 sec statahead total: 144 statahead wrong: 2 agl total: 137 list_total: 33 fname_total: 95 hit_total: 4744 miss_total: 1674 swrong=2 ewrong=2 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 23 seconds llite.lustre-ffff95301a1a1800.statahead_stats= statahead total: 144 statahead wrong: 2 agl total: 137 list_total: 33 fname_total: 95 hit_total: 4744 miss_total: 1674 mdc.lustre-MDT0000-mdc-ffff95301a1a1800.batch_stats= snapshot_time: 1788566321.424364102 secs.nsecs start_time: 1788566222.560803766 secs.nsecs elapsed_time: 98.863560336 secs.nsecs subreqs per batch batches %% cum %% 1: 1 2 2 2: 0 0 2 4: 2 5 7 8: 3 7 15 16: 21 52 67 32: 13 32 100 mdc.lustre-MDT0001-mdc-ffff95301a1a1800.batch_stats= snapshot_time: 1788566321.424444398 secs.nsecs start_time: 1788566222.560918245 secs.nsecs elapsed_time: 98.863526153 secs.nsecs subreqs per batch batches %% cum %% 1: 1 2 2 2: 0 0 2 4: 2 5 7 8: 4 10 17 16: 18 46 64 32: 14 35 100 - open/close 718 (time 1788566334.46 total 10.00 last 71.79) - open/close 1482 (time 1788566344.46 total 20.01 last 76.37) - open/close 2225 (time 1788566354.48 total 30.02 last 74.20) - open/close 2996 (time 1788566364.49 total 40.03 last 77.05) - open/close 3692 (time 1788566374.49 total 50.03 last 69.60) - open/close 4442 (time 1788566384.52 total 60.06 last 74.78) - open/close 5008 (time 1788566394.53 total 70.07 last 56.51) - open/close 5741 (time 1788566404.53 total 80.08 last 73.28) - open/close 6326 (time 1788566414.54 total 90.08 last 58.46) - open/close 6994 (time 1788566424.55 total 100.09 last 66.73) - open/close 7687 (time 1788566434.56 total 110.10 last 69.28) - open/close 8528 (time 1788566444.56 total 120.10 last 84.07) - open/close 9296 (time 1788566454.57 total 130.11 last 76.75) total: 10000 open/close in 140.10 seconds: 71.38 ops/second llite.lustre-ffff95301a1a1800.statahead_batch_max=0 llite.lustre-ffff95301a1a1800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff95301a1a1800.stats=clear mdc.lustre-MDT0001-mdc-ffff95301a1a1800.stats=clear ls -l files in d123ad.sanity: 10001 real 1m4.262s user 0m0.327s sys 0m14.501s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff95301a1a1800.statahead_batch_max=32 llite.lustre-ffff95301a1a1800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff95301a1a1800.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff95301a1a1800.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff95301a1a1800.stats=clear mdc.lustre-MDT0001-mdc-ffff95301a1a1800.stats=clear ls -l files in d123ad.sanity: 10001 real 2m17.792s user 0m0.314s sys 0m14.442s batched statahead RPC= hit count=9966 mdc.lustre-MDT0000-mdc-ffff95301a1a1800.batch_stats= snapshot_time: 1788566687.131618625 secs.nsecs start_time: 1788566539.076803866 secs.nsecs elapsed_time: 148.054814759 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 0 0 0 8: 1 0 0 16: 2 0 0 32: 313 99 100 mdc.lustre-MDT0001-mdc-ffff95301a1a1800.batch_stats= snapshot_time: 1788566687.131711220 secs.nsecs start_time: 1788566539.076887486 secs.nsecs elapsed_time: 148.054823734 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 batch: statahead_max=2048 statahead_batch_max=256 mdc.lustre-MDT0000-mdc-ffff95301a1a1800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff95301a1a1800.batch_stats=0 llite.lustre-ffff95301a1a1800.statahead_max=2048 llite.lustre-ffff95301a1a1800.statahead_batch_max=256 kvm mdc.lustre-MDT0000-mdc-ffff95301a1a1800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff95301a1a1800.batch_stats=0 striped dir -i1 -c2 -H crush2 /mnt/lustre/d123ad.sanity total: 100 open/close in 2.98 seconds: 33.55 ops/second llite.lustre-ffff95301a1a1800.statahead_max=0 101 real 0m2.242s user 0m0.004s sys 0m0.331s 'ls -l' 100 files without statahead: 2 sec llite.lustre-ffff95301a1a1800.statahead_max=2048 llite.lustre-ffff95301a1a1800.statahead_max=2048 101 real 0m0.985s user 0m0.003s sys 0m0.153s 'ls -l' 100 files with statahead: 1 sec statahead total: 2 statahead wrong: 0 agl total: 2 list_total: 2 fname_total: 0 hit_total: 10065 miss_total: 35 swrong=0 ewrong=0 - open/close 650 (time 1788567016.72 total 10.00 last 64.98) total: 900 open/close in 13.52 seconds: 66.59 ops/second llite.lustre-ffff95301a1a1800.statahead_max=0 1001 real 0m25.039s user 0m0.039s sys 0m3.095s 'ls -l' 1000 files without statahead: 25 sec llite.lustre-ffff95301a1a1800.statahead_max=2048 llite.lustre-ffff95301a1a1800.statahead_max=2048 1001 real 0m8.594s user 0m0.025s sys 0m1.356s 'ls -l' 1000 files with statahead: 9 sec statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11064 miss_total: 36 swrong=0 ewrong=0 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 25 seconds llite.lustre-ffff95301a1a1800.statahead_stats= statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11064 miss_total: 36 mdc.lustre-MDT0000-mdc-ffff95301a1a1800.batch_stats= snapshot_time: 1788567096.697339591 secs.nsecs start_time: 1788566690.190055423 secs.nsecs elapsed_time: 406.507284168 secs.nsecs subreqs per batch batches %% cum %% 1: 1 14 14 2: 2 28 42 4: 1 14 57 8: 0 0 57 16: 1 14 71 32: 0 0 71 64: 0 0 71 128: 0 0 71 256: 2 28 100 mdc.lustre-MDT0001-mdc-ffff95301a1a1800.batch_stats= snapshot_time: 1788567096.697406989 secs.nsecs start_time: 1788566690.192409367 secs.nsecs elapsed_time: 406.504997622 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 0 0 0 8: 4 40 40 16: 2 20 60 32: 1 10 70 64: 1 10 80 128: 0 0 80 256: 2 20 100 - open/close 644 (time 1788567109.93 total 10.00 last 64.38) - open/close 1306 (time 1788567119.93 total 20.00 last 66.20) - open/close 1900 (time 1788567129.93 total 30.00 last 59.39) - open/close 2673 (time 1788567139.94 total 40.01 last 77.25) - open/close 3530 (time 1788567149.94 total 50.02 last 85.65) - open/close 4307 (time 1788567159.95 total 60.02 last 77.67) - open/close 5103 (time 1788567169.96 total 70.03 last 79.53) - open/close 5863 (time 1788567179.96 total 80.04 last 75.96) - open/close 6642 (time 1788567189.97 total 90.04 last 77.84) - open/close 7416 (time 1788567199.97 total 100.05 last 77.38) - open/close 8133 (time 1788567210.01 total 110.08 last 71.44) - open/close 8721 (time 1788567220.02 total 120.09 last 58.74) - open/close 9350 (time 1788567230.03 total 130.10 last 62.82) total: 10000 open/close in 139.31 seconds: 71.78 ops/second llite.lustre-ffff95301a1a1800.statahead_batch_max=0 llite.lustre-ffff95301a1a1800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff95301a1a1800.stats=clear mdc.lustre-MDT0001-mdc-ffff95301a1a1800.stats=clear ls -l files in d123ad.sanity: 10001 real 1m21.397s user 0m0.340s sys 0m14.381s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff95301a1a1800.statahead_batch_max=256 llite.lustre-ffff95301a1a1800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff95301a1a1800.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff95301a1a1800.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff95301a1a1800.stats=clear mdc.lustre-MDT0001-mdc-ffff95301a1a1800.stats=clear ls -l files in d123ad.sanity: 10001 real 1m10.870s user 0m0.338s sys 0m14.479s batched statahead RPC= hit count=9999 mdc.lustre-MDT0000-mdc-ffff95301a1a1800.batch_stats= snapshot_time: 1788567414.838742826 secs.nsecs start_time: 1788567329.883304749 secs.nsecs elapsed_time: 84.955438077 secs.nsecs subreqs per batch batches %% cum %% 1: 81 36 36 2: 31 14 50 4: 30 13 64 8: 12 5 70 16: 7 3 73 32: 20 9 82 64: 4 1 84 128: 0 0 84 256: 35 15 100 mdc.lustre-MDT0001-mdc-ffff95301a1a1800.batch_stats= snapshot_time: 1788567414.838836531 secs.nsecs start_time: 1788567329.883590280 secs.nsecs elapsed_time: 84.955246251 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second seq.cli-lustre-OST0000-super.width=65536 seq.cli-lustre-OST0001-super.width=65536 llite.lustre-ffff95301a1a1800.statahead_batch_max=64 llite.lustre-ffff95301a1a1800.statahead_max=128