== sanity test 123ad: Verify batching statahead works correctly ========================================================== 11:29:03 (1790695743) batch: statahead_max=32 statahead_batch_max=32 mdc.lustre-MDT0000-mdc-ffff8e8960bac000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff8e8960bac000.batch_stats=0 llite.lustre-ffff8e8960bac000.statahead_max=32 llite.lustre-ffff8e8960bac000.statahead_batch_max=32 seq.cli-lustre-OST0000-super.width=0x1ffffff seq.cli-lustre-OST0001-super.width=0x1ffffff kvm mdc.lustre-MDT0000-mdc-ffff8e8960bac000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff8e8960bac000.batch_stats=0 striped dir -i1 -c2 -H all_char /mnt/lustre/d123ad.sanity total: 100 open/close in 2.99 seconds: 33.48 ops/second llite.lustre-ffff8e8960bac000.statahead_max=0 101 real 0m2.644s user 0m0.006s sys 0m0.320s 'ls -l' 100 files without statahead: 3 sec llite.lustre-ffff8e8960bac000.statahead_max=32 llite.lustre-ffff8e8960bac000.statahead_max=32 101 real 0m0.977s user 0m0.005s sys 0m0.181s 'ls -l' 100 files with statahead: 1 sec statahead total: 133 statahead wrong: 2 agl total: 118 list_total: 31 fname_total: 81 hit_total: 3883 miss_total: 1297 swrong=2 ewrong=2 - open/close 720 (time 1790695771.13 total 10.01 last 71.92) total: 900 open/close in 12.99 seconds: 69.30 ops/second llite.lustre-ffff8e8960bac000.statahead_max=0 1001 real 0m26.278s user 0m0.042s sys 0m3.171s 'ls -l' 1000 files without statahead: 26 sec llite.lustre-ffff8e8960bac000.statahead_max=32 llite.lustre-ffff8e8960bac000.statahead_max=32 1001 real 0m10.290s user 0m0.032s sys 0m1.367s 'ls -l' 1000 files with statahead: 10 sec statahead total: 134 statahead wrong: 2 agl total: 119 list_total: 32 fname_total: 81 hit_total: 4881 miss_total: 1299 swrong=2 ewrong=2 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 22 seconds llite.lustre-ffff8e8960bac000.statahead_stats= statahead total: 134 statahead wrong: 2 agl total: 119 list_total: 32 fname_total: 81 hit_total: 4881 miss_total: 1299 mdc.lustre-MDT0000-mdc-ffff8e8960bac000.batch_stats= snapshot_time: 1790695846.099291848 secs.nsecs start_time: 1790695748.464003641 secs.nsecs elapsed_time: 97.635288207 secs.nsecs subreqs per batch batches %% cum %% 1: 1 2 2 2: 0 0 2 4: 3 7 10 8: 1 2 12 16: 21 53 66 32: 13 33 100 mdc.lustre-MDT0001-mdc-ffff8e8960bac000.batch_stats= snapshot_time: 1790695846.099376032 secs.nsecs start_time: 1790695748.464087522 secs.nsecs elapsed_time: 97.635288510 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 3 7 7 8: 4 10 17 16: 18 46 64 32: 14 35 100 - open/close 745 (time 1790695858.76 total 10.01 last 74.45) - open/close 1456 (time 1790695868.76 total 20.01 last 71.09) - open/close 2160 (time 1790695878.77 total 30.01 last 70.38) - open/close 2936 (time 1790695888.77 total 40.02 last 77.57) - open/close 3693 (time 1790695898.78 total 50.02 last 75.64) - open/close 4418 (time 1790695908.78 total 60.02 last 72.50) - open/close 5112 (time 1790695918.79 total 70.03 last 69.36) - open/close 5873 (time 1790695928.79 total 80.03 last 76.09) - open/close 6664 (time 1790695938.79 total 90.04 last 79.06) - open/close 7362 (time 1790695948.80 total 100.04 last 69.76) - open/close 8042 (time 1790695958.81 total 110.05 last 67.94) - open/close 8780 (time 1790695968.82 total 120.06 last 73.73) - open/close 9429 (time 1790695978.82 total 130.06 last 64.88) total: 10000 open/close in 137.77 seconds: 72.59 ops/second llite.lustre-ffff8e8960bac000.statahead_batch_max=0 llite.lustre-ffff8e8960bac000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff8e8960bac000.stats=clear mdc.lustre-MDT0001-mdc-ffff8e8960bac000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m3.414s user 0m0.295s sys 0m14.794s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff8e8960bac000.statahead_batch_max=32 llite.lustre-ffff8e8960bac000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff8e8960bac000.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff8e8960bac000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff8e8960bac000.stats=clear mdc.lustre-MDT0001-mdc-ffff8e8960bac000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m50.997s user 0m0.301s sys 0m13.778s batched statahead RPC= hit count=9967 mdc.lustre-MDT0000-mdc-ffff8e8960bac000.batch_stats= snapshot_time: 1790696181.149699714 secs.nsecs start_time: 1790696057.694616967 secs.nsecs elapsed_time: 123.455082747 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 0 0 0 8: 1 0 0 16: 1 0 0 32: 313 99 100 mdc.lustre-MDT0001-mdc-ffff8e8960bac000.batch_stats= snapshot_time: 1790696181.149763676 secs.nsecs start_time: 1790696057.694696068 secs.nsecs elapsed_time: 123.455067608 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 batch: statahead_max=2048 statahead_batch_max=256 mdc.lustre-MDT0000-mdc-ffff8e8960bac000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff8e8960bac000.batch_stats=0 llite.lustre-ffff8e8960bac000.statahead_max=2048 llite.lustre-ffff8e8960bac000.statahead_batch_max=256 kvm mdc.lustre-MDT0000-mdc-ffff8e8960bac000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff8e8960bac000.batch_stats=0 striped dir -i1 -c2 -H crush2 /mnt/lustre/d123ad.sanity total: 100 open/close in 2.97 seconds: 33.70 ops/second llite.lustre-ffff8e8960bac000.statahead_max=0 101 real 0m2.467s user 0m0.007s sys 0m0.313s 'ls -l' 100 files without statahead: 3 sec llite.lustre-ffff8e8960bac000.statahead_max=2048 llite.lustre-ffff8e8960bac000.statahead_max=2048 101 real 0m1.055s user 0m0.005s sys 0m0.194s 'ls -l' 100 files with statahead: 1 sec statahead total: 2 statahead wrong: 0 agl total: 2 list_total: 2 fname_total: 0 hit_total: 10066 miss_total: 34 swrong=0 ewrong=0 - open/close 722 (time 1790696506.22 total 10.01 last 72.14) total: 900 open/close in 12.53 seconds: 71.84 ops/second llite.lustre-ffff8e8960bac000.statahead_max=0 1001 real 0m22.491s user 0m0.040s sys 0m2.939s 'ls -l' 1000 files without statahead: 22 sec llite.lustre-ffff8e8960bac000.statahead_max=2048 llite.lustre-ffff8e8960bac000.statahead_max=2048 1001 real 0m6.632s user 0m0.031s sys 0m1.517s 'ls -l' 1000 files with statahead: 7 sec statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11065 miss_total: 35 swrong=0 ewrong=0 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 21 seconds llite.lustre-ffff8e8960bac000.statahead_stats= statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11065 miss_total: 35 mdc.lustre-MDT0000-mdc-ffff8e8960bac000.batch_stats= snapshot_time: 1790696570.781285055 secs.nsecs start_time: 1790696183.689272341 secs.nsecs elapsed_time: 387.092012714 secs.nsecs subreqs per batch batches %% cum %% 1: 1 20 20 2: 0 0 20 4: 1 20 40 8: 1 20 60 16: 0 0 60 32: 0 0 60 64: 0 0 60 128: 0 0 60 256: 2 40 100 mdc.lustre-MDT0001-mdc-ffff8e8960bac000.batch_stats= snapshot_time: 1790696570.781504475 secs.nsecs start_time: 1790696183.689410256 secs.nsecs elapsed_time: 387.092094219 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 1 12 12 4: 0 0 12 8: 2 25 37 16: 1 12 50 32: 0 0 50 64: 1 12 62 128: 1 12 75 256: 2 25 100 - open/close 825 (time 1790696583.52 total 10.01 last 82.45) - open/close 1611 (time 1790696593.53 total 20.01 last 78.54) - open/close 2378 (time 1790696603.54 total 30.02 last 76.62) - open/close 3127 (time 1790696613.55 total 40.03 last 74.84) - open/close 3959 (time 1790696623.55 total 50.03 last 83.19) - open/close 4791 (time 1790696633.55 total 60.04 last 83.15) - open/close 5526 (time 1790696643.56 total 70.05 last 73.44) - open/close 6291 (time 1790696653.57 total 80.05 last 76.44) - open/close 7029 (time 1790696663.58 total 90.06 last 73.74) - open/close 7714 (time 1790696673.58 total 100.07 last 68.48) - open/close 8451 (time 1790696683.58 total 110.07 last 73.69) - open/close 9295 (time 1790696693.59 total 120.08 last 84.31) total: 10000 open/close in 128.67 seconds: 77.72 ops/second llite.lustre-ffff8e8960bac000.statahead_batch_max=0 llite.lustre-ffff8e8960bac000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff8e8960bac000.stats=clear mdc.lustre-MDT0001-mdc-ffff8e8960bac000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m3.934s user 0m0.307s sys 0m14.563s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff8e8960bac000.statahead_batch_max=256 llite.lustre-ffff8e8960bac000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff8e8960bac000.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff8e8960bac000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff8e8960bac000.stats=clear mdc.lustre-MDT0001-mdc-ffff8e8960bac000.stats=clear ls -l files in d123ad.sanity: 10001 real 0m49.222s user 0m0.277s sys 0m15.153s batched statahead RPC= hit count=9999 mdc.lustre-MDT0000-mdc-ffff8e8960bac000.batch_stats= snapshot_time: 1790696835.261013555 secs.nsecs start_time: 1790696775.110297583 secs.nsecs elapsed_time: 60.150715972 secs.nsecs subreqs per batch batches %% cum %% 1: 173 53 53 2: 42 12 65 4: 21 6 72 8: 25 7 80 16: 4 1 81 32: 20 6 87 64: 7 2 89 128: 0 0 89 256: 34 10 100 mdc.lustre-MDT0001-mdc-ffff8e8960bac000.batch_stats= snapshot_time: 1790696835.261104941 secs.nsecs start_time: 1790696775.110590533 secs.nsecs elapsed_time: 60.150514408 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second seq.cli-lustre-OST0000-super.width=65536 seq.cli-lustre-OST0001-super.width=65536 llite.lustre-ffff8e8960bac000.statahead_batch_max=64 llite.lustre-ffff8e8960bac000.statahead_max=128