== sanity test 123ad: Verify batching statahead works correctly ========================================================== 09:37:37 (1788874657) batch: statahead_max=32 statahead_batch_max=32 mdc.lustre-MDT0000-mdc-ffff8fcb4c76d000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff8fcb4c76d000.batch_stats=0 llite.lustre-ffff8fcb4c76d000.statahead_max=32 llite.lustre-ffff8fcb4c76d000.statahead_batch_max=32 seq.cli-lustre-OST0000-super.width=0x1ffffff seq.cli-lustre-OST0001-super.width=0x1ffffff kvm mdc.lustre-MDT0000-mdc-ffff8fcb4c76d000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff8fcb4c76d000.batch_stats=0 striped dir -i1 -c2 -H crush /mnt/lustre/d123ad.sanity total: 100 open/close in 2.59 seconds: 38.68 ops/second llite.lustre-ffff8fcb4c76d000.statahead_max=0 101 real 0m2.285s user 0m0.006s sys 0m0.306s 'ls -l' 100 files without statahead: 2 sec llite.lustre-ffff8fcb4c76d000.statahead_max=32 llite.lustre-ffff8fcb4c76d000.statahead_max=32 101 real 0m0.996s user 0m0.004s sys 0m0.169s 'ls -l' 100 files with statahead: 1 sec statahead total: 105 statahead wrong: 2 agl total: 99 list_total: 30 fname_total: 72 hit_total: 4039 miss_total: 1578 swrong=2 ewrong=2 - open/close 803 (time 1788874683.59 total 10.01 last 80.24) total: 900 open/close in 11.45 seconds: 78.59 ops/second llite.lustre-ffff8fcb4c76d000.statahead_max=0 1001 real 0m21.170s user 0m0.033s sys 0m2.815s 'ls -l' 1000 files without statahead: 21 sec llite.lustre-ffff8fcb4c76d000.statahead_max=32 llite.lustre-ffff8fcb4c76d000.statahead_max=32 1001 real 0m10.156s user 0m0.035s sys 0m1.379s 'ls -l' 1000 files with statahead: 10 sec statahead total: 106 statahead wrong: 2 agl total: 100 list_total: 31 fname_total: 72 hit_total: 5035 miss_total: 1582 swrong=2 ewrong=2 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 23 seconds llite.lustre-ffff8fcb4c76d000.statahead_stats= statahead total: 106 statahead wrong: 2 agl total: 100 list_total: 31 fname_total: 72 hit_total: 5035 miss_total: 1582 mdc.lustre-MDT0000-mdc-ffff8fcb4c76d000.batch_stats= snapshot_time: 1788874752.512789289 secs.nsecs start_time: 1788874661.286141484 secs.nsecs elapsed_time: 91.226647805 secs.nsecs subreqs per batch batches %% cum %% 1: 2 5 5 2: 2 5 10 4: 1 2 13 8: 3 8 21 16: 26 70 91 32: 3 8 100 mdc.lustre-MDT0001-mdc-ffff8fcb4c76d000.batch_stats= snapshot_time: 1788874752.516058127 secs.nsecs start_time: 1788874661.286305834 secs.nsecs elapsed_time: 91.229752293 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 2 5 5 8: 3 7 12 16: 11 28 41 32: 23 58 100 - open/close 865 (time 1788874765.19 total 10.00 last 86.47) - open/close 1667 (time 1788874775.19 total 20.01 last 80.18) - open/close 2453 (time 1788874785.20 total 30.01 last 78.53) - open/close 3268 (time 1788874795.21 total 40.03 last 81.41) - open/close 4001 (time 1788874805.22 total 50.04 last 73.22) - open/close 4850 (time 1788874815.23 total 60.05 last 84.83) - open/close 5675 (time 1788874825.23 total 70.05 last 82.49) - open/close 6417 (time 1788874835.24 total 80.05 last 74.15) - open/close 7262 (time 1788874845.24 total 90.06 last 84.46) - open/close 8058 (time 1788874855.25 total 100.06 last 79.57) - open/close 8950 (time 1788874865.25 total 110.07 last 89.14) - open/close 9774 (time 1788874875.26 total 120.07 last 82.35) total: 10000 open/close in 123.18 seconds: 81.18 ops/second llite.lustre-ffff8fcb4c76d000.statahead_batch_max=0 llite.lustre-ffff8fcb4c76d000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff8fcb4c76d000.stats=clear mdc.lustre-MDT0001-mdc-ffff8fcb4c76d000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m7.335s user 0m0.265s sys 0m14.183s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff8fcb4c76d000.statahead_batch_max=32 llite.lustre-ffff8fcb4c76d000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff8fcb4c76d000.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff8fcb4c76d000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff8fcb4c76d000.stats=clear mdc.lustre-MDT0001-mdc-ffff8fcb4c76d000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m46.024s user 0m0.269s sys 0m13.681s batched statahead RPC= hit count=9981 mdc.lustre-MDT0000-mdc-ffff8fcb4c76d000.batch_stats= snapshot_time: 1788875075.112912122 secs.nsecs start_time: 1788874955.817254470 secs.nsecs elapsed_time: 119.295657652 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 0 0 0 8: 1 0 0 16: 1 0 0 32: 313 99 100 mdc.lustre-MDT0001-mdc-ffff8fcb4c76d000.batch_stats= snapshot_time: 1788875075.113150423 secs.nsecs start_time: 1788874955.817561528 secs.nsecs elapsed_time: 119.295588895 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 batch: statahead_max=2048 statahead_batch_max=256 mdc.lustre-MDT0000-mdc-ffff8fcb4c76d000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff8fcb4c76d000.batch_stats=0 llite.lustre-ffff8fcb4c76d000.statahead_max=2048 llite.lustre-ffff8fcb4c76d000.statahead_batch_max=256 kvm mdc.lustre-MDT0000-mdc-ffff8fcb4c76d000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff8fcb4c76d000.batch_stats=0 striped dir -i1 -c2 -H all_char /mnt/lustre/d123ad.sanity total: 100 open/close in 2.23 seconds: 44.75 ops/second llite.lustre-ffff8fcb4c76d000.statahead_max=0 101 real 0m2.482s user 0m0.003s sys 0m0.306s 'ls -l' 100 files without statahead: 2 sec llite.lustre-ffff8fcb4c76d000.statahead_max=2048 llite.lustre-ffff8fcb4c76d000.statahead_max=2048 101 real 0m1.014s user 0m0.003s sys 0m0.148s 'ls -l' 100 files with statahead: 1 sec statahead total: 2 statahead wrong: 0 agl total: 2 list_total: 2 fname_total: 0 hit_total: 10080 miss_total: 20 swrong=0 ewrong=0 - open/close 813 (time 1788875362.70 total 10.01 last 81.18) total: 900 open/close in 11.27 seconds: 79.88 ops/second llite.lustre-ffff8fcb4c76d000.statahead_max=0 1001 real 0m24.185s user 0m0.027s sys 0m2.793s 'ls -l' 1000 files without statahead: 24 sec llite.lustre-ffff8fcb4c76d000.statahead_max=2048 llite.lustre-ffff8fcb4c76d000.statahead_max=2048 1001 real 0m7.023s user 0m0.023s sys 0m1.381s 'ls -l' 1000 files with statahead: 7 sec statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11079 miss_total: 21 swrong=0 ewrong=0 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 21 seconds llite.lustre-ffff8fcb4c76d000.statahead_stats= statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11079 miss_total: 21 mdc.lustre-MDT0000-mdc-ffff8fcb4c76d000.batch_stats= snapshot_time: 1788875428.340754608 secs.nsecs start_time: 1788875077.702436764 secs.nsecs elapsed_time: 350.638317844 secs.nsecs subreqs per batch batches %% cum %% 1: 1 11 11 2: 0 0 11 4: 0 0 11 8: 3 33 44 16: 1 11 55 32: 2 22 77 64: 0 0 77 128: 0 0 77 256: 2 22 100 mdc.lustre-MDT0001-mdc-ffff8fcb4c76d000.batch_stats= snapshot_time: 1788875428.340831966 secs.nsecs start_time: 1788875077.703541044 secs.nsecs elapsed_time: 350.637290922 secs.nsecs subreqs per batch batches %% cum %% 1: 1 11 11 2: 0 0 11 4: 2 22 33 8: 1 11 44 16: 1 11 55 32: 2 22 77 64: 0 0 77 128: 0 0 77 256: 2 22 100 - open/close 841 (time 1788875441.24 total 10.00 last 84.07) - open/close 1585 (time 1788875451.26 total 20.02 last 74.26) - open/close 2454 (time 1788875461.26 total 30.03 last 86.82) - open/close 3288 (time 1788875471.27 total 40.04 last 83.36) - open/close 4079 (time 1788875481.28 total 50.05 last 79.01) - open/close 4988 (time 1788875491.30 total 60.06 last 90.76) - open/close 5823 (time 1788875501.30 total 70.07 last 83.46) - open/close 6713 (time 1788875511.30 total 80.07 last 88.99) - open/close 7621 (time 1788875521.30 total 90.07 last 90.78) - open/close 8500 (time 1788875531.31 total 100.08 last 87.83) - open/close 9301 (time 1788875541.32 total 110.08 last 80.06) total: 10000 open/close in 118.68 seconds: 84.26 ops/second llite.lustre-ffff8fcb4c76d000.statahead_batch_max=0 llite.lustre-ffff8fcb4c76d000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff8fcb4c76d000.stats=clear mdc.lustre-MDT0001-mdc-ffff8fcb4c76d000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m4.941s user 0m0.292s sys 0m14.426s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff8fcb4c76d000.statahead_batch_max=256 llite.lustre-ffff8fcb4c76d000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff8fcb4c76d000.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff8fcb4c76d000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff8fcb4c76d000.stats=clear mdc.lustre-MDT0001-mdc-ffff8fcb4c76d000.stats=clear ls -l files in d123ad.sanity: 10001 real 0m55.123s user 0m0.292s sys 0m14.273s batched statahead RPC= hit count=9999 mdc.lustre-MDT0000-mdc-ffff8fcb4c76d000.batch_stats= snapshot_time: 1788875690.628280257 secs.nsecs start_time: 1788875623.927980638 secs.nsecs elapsed_time: 66.700299619 secs.nsecs subreqs per batch batches %% cum %% 1: 139 46 46 2: 33 11 57 4: 38 12 70 8: 20 6 77 16: 10 3 80 32: 17 5 86 64: 5 1 88 128: 1 0 88 256: 34 11 100 mdc.lustre-MDT0001-mdc-ffff8fcb4c76d000.batch_stats= snapshot_time: 1788875690.628351165 secs.nsecs start_time: 1788875623.928134569 secs.nsecs elapsed_time: 66.700216596 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second seq.cli-lustre-OST0000-super.width=65536 seq.cli-lustre-OST0001-super.width=65536 llite.lustre-ffff8fcb4c76d000.statahead_batch_max=64 llite.lustre-ffff8fcb4c76d000.statahead_max=128