== sanity test 123ad: Verify batching statahead works correctly ========================================================== 01:32:30 (1781242350) batch: statahead_max=32 statahead_batch_max=32 mdc.lustre-MDT0000-mdc-ffff8cba86671000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff8cba86671000.batch_stats=0 llite.lustre-ffff8cba86671000.statahead_max=32 llite.lustre-ffff8cba86671000.statahead_batch_max=32 seq.cli-lustre-OST0000-super.width=0x1ffffff seq.cli-lustre-OST0001-super.width=0x1ffffff kvm mdc.lustre-MDT0000-mdc-ffff8cba86671000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff8cba86671000.batch_stats=0 striped dir -i1 -c2 -H crush /mnt/lustre/d123ad.sanity total: 100 open/close in 2.66 seconds: 37.64 ops/second llite.lustre-ffff8cba86671000.statahead_max=0 101 real 0m2.441s user 0m0.003s sys 0m0.339s 'ls -l' 100 files without statahead: 3 sec llite.lustre-ffff8cba86671000.statahead_max=32 llite.lustre-ffff8cba86671000.statahead_max=32 101 real 0m1.202s user 0m0.002s sys 0m0.197s 'ls -l' 100 files with statahead: 1 sec statahead total: 125 statahead wrong: 2 agl total: 119 list_total: 32 fname_total: 93 hit_total: 3601 miss_total: 1371 swrong=2 ewrong=2 - open/close 734 (time 1781242378.84 total 10.01 last 73.34) total: 900 open/close in 12.37 seconds: 72.78 ops/second llite.lustre-ffff8cba86671000.statahead_max=0 1001 real 0m21.520s user 0m0.035s sys 0m3.172s 'ls -l' 1000 files without statahead: 21 sec llite.lustre-ffff8cba86671000.statahead_max=32 llite.lustre-ffff8cba86671000.statahead_max=32 1001 real 0m10.203s user 0m0.024s sys 0m1.398s 'ls -l' 1000 files with statahead: 10 sec statahead total: 126 statahead wrong: 2 agl total: 120 list_total: 33 fname_total: 93 hit_total: 4598 miss_total: 1374 swrong=2 ewrong=2 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 20 seconds llite.lustre-ffff8cba86671000.statahead_stats= statahead total: 126 statahead wrong: 2 agl total: 120 list_total: 33 fname_total: 93 hit_total: 4598 miss_total: 1374 mdc.lustre-MDT0000-mdc-ffff8cba86671000.batch_stats= snapshot_time: 1781242447.374985980 secs.nsecs start_time: 1781242355.475030320 secs.nsecs elapsed_time: 91.899955660 secs.nsecs subreqs per batch batches % cum % 1: 1 2 2 2: 2 5 8 4: 1 2 11 8: 3 8 19 16: 23 63 83 32: 6 16 100 mdc.lustre-MDT0001-mdc-ffff8cba86671000.batch_stats= snapshot_time: 1781242447.375062668 secs.nsecs start_time: 1781242355.475185618 secs.nsecs elapsed_time: 91.899877050 secs.nsecs subreqs per batch batches % cum % 1: 1 2 2 2: 1 2 5 4: 1 2 7 8: 2 5 12 16: 11 28 41 32: 23 58 100 - open/close 910 (time 1781242459.63 total 10.02 last 90.81) - open/close 1804 (time 1781242469.64 total 20.03 last 89.31) - open/close 2681 (time 1781242479.65 total 30.04 last 87.63) - open/close 3574 (time 1781242489.66 total 40.06 last 89.15) - open/close 4398 (time 1781242499.69 total 50.08 last 82.20) - open/close 5197 (time 1781242509.69 total 60.08 last 79.88) - open/close 6101 (time 1781242519.72 total 70.11 last 90.18) - open/close 6860 (time 1781242529.72 total 80.11 last 75.90) - open/close 7805 (time 1781242539.72 total 90.11 last 94.49) - open/close 8770 (time 1781242549.72 total 100.11 last 96.45) - open/close 9519 (time 1781242559.73 total 110.12 last 74.86) total: 10000 open/close in 115.68 seconds: 86.45 ops/second llite.lustre-ffff8cba86671000.statahead_batch_max=0 llite.lustre-ffff8cba86671000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff8cba86671000.stats=clear mdc.lustre-MDT0001-mdc-ffff8cba86671000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m10.937s user 0m0.324s sys 0m16.144s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff8cba86671000.statahead_batch_max=32 llite.lustre-ffff8cba86671000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff8cba86671000.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff8cba86671000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff8cba86671000.stats=clear mdc.lustre-MDT0001-mdc-ffff8cba86671000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m47.200s user 0m0.313s sys 0m15.520s batched statahead RPC= hit count=9988 mdc.lustre-MDT0000-mdc-ffff8cba86671000.batch_stats= snapshot_time: 1781242769.785676882 secs.nsecs start_time: 1781242646.196181611 secs.nsecs elapsed_time: 123.589495271 secs.nsecs subreqs per batch batches % cum % 1: 0 0 0 2: 0 0 0 4: 0 0 0 8: 1 0 0 16: 1 0 0 32: 313 99 100 mdc.lustre-MDT0001-mdc-ffff8cba86671000.batch_stats= snapshot_time: 1781242769.785750525 secs.nsecs start_time: 1781242646.196254119 secs.nsecs elapsed_time: 123.589496406 secs.nsecs subreqs per batch batches % cum % 1: 0 0 0 batch: statahead_max=2048 statahead_batch_max=256 mdc.lustre-MDT0000-mdc-ffff8cba86671000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff8cba86671000.batch_stats=0 llite.lustre-ffff8cba86671000.statahead_max=2048 llite.lustre-ffff8cba86671000.statahead_batch_max=256 kvm mdc.lustre-MDT0000-mdc-ffff8cba86671000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff8cba86671000.batch_stats=0 striped dir -i1 -c2 -H all_char /mnt/lustre/d123ad.sanity total: 100 open/close in 2.02 seconds: 49.48 ops/second llite.lustre-ffff8cba86671000.statahead_max=0 101 real 0m2.308s user 0m0.004s sys 0m0.343s 'ls -l' 100 files without statahead: 3 sec llite.lustre-ffff8cba86671000.statahead_max=2048 llite.lustre-ffff8cba86671000.statahead_max=2048 101 real 0m0.742s user 0m0.006s sys 0m0.189s 'ls -l' 100 files with statahead: 0 sec statahead total: 2 statahead wrong: 0 agl total: 2 list_total: 2 fname_total: 0 hit_total: 10087 miss_total: 13 swrong=0 ewrong=0 - open/close 822 (time 1781243040.46 total 10.01 last 82.14) total: 900 open/close in 11.12 seconds: 80.91 ops/second llite.lustre-ffff8cba86671000.statahead_max=0 1001 real 0m19.860s user 0m0.037s sys 0m3.222s 'ls -l' 1000 files without statahead: 20 sec llite.lustre-ffff8cba86671000.statahead_max=2048 llite.lustre-ffff8cba86671000.statahead_max=2048 1001 real 0m7.281s user 0m0.033s sys 0m1.559s 'ls -l' 1000 files with statahead: 7 sec statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11086 miss_total: 14 swrong=0 ewrong=0 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 20 seconds llite.lustre-ffff8cba86671000.statahead_stats= statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11086 miss_total: 14 mdc.lustre-MDT0000-mdc-ffff8cba86671000.batch_stats= snapshot_time: 1781243101.491192089 secs.nsecs start_time: 1781242771.920357947 secs.nsecs elapsed_time: 329.570834142 secs.nsecs subreqs per batch batches % cum % 1: 0 0 0 2: 0 0 0 4: 3 30 30 8: 1 10 40 16: 3 30 70 32: 1 10 80 64: 0 0 80 128: 0 0 80 256: 2 20 100 mdc.lustre-MDT0001-mdc-ffff8cba86671000.batch_stats= snapshot_time: 1781243101.491248591 secs.nsecs start_time: 1781242771.920497212 secs.nsecs elapsed_time: 329.570751379 secs.nsecs subreqs per batch batches % cum % 1: 0 0 0 2: 0 0 0 4: 2 20 20 8: 4 40 60 16: 0 0 60 32: 1 10 70 64: 1 10 80 128: 0 0 80 256: 2 20 100 - open/close 1014 (time 1781243113.88 total 10.01 last 101.29) - open/close 1868 (time 1781243123.88 total 20.02 last 85.36) - open/close 2748 (time 1781243133.88 total 30.02 last 87.98) - open/close 3676 (time 1781243143.89 total 40.02 last 92.74) - open/close 4596 (time 1781243153.89 total 50.03 last 91.98) - open/close 5564 (time 1781243163.90 total 60.03 last 96.75) - open/close 6548 (time 1781243173.90 total 70.03 last 98.38) - open/close 7361 (time 1781243183.90 total 80.04 last 81.27) - open/close 8332 (time 1781243193.91 total 90.04 last 97.07) - open/close 9224 (time 1781243203.93 total 100.06 last 89.04) total: 10000 open/close in 108.95 seconds: 91.78 ops/second llite.lustre-ffff8cba86671000.statahead_batch_max=0 llite.lustre-ffff8cba86671000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff8cba86671000.stats=clear mdc.lustre-MDT0001-mdc-ffff8cba86671000.stats=clear ls -l files in d123ad.sanity: 10001 real 0m55.343s user 0m0.304s sys 0m16.402s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff8cba86671000.statahead_batch_max=256 llite.lustre-ffff8cba86671000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff8cba86671000.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff8cba86671000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff8cba86671000.stats=clear mdc.lustre-MDT0001-mdc-ffff8cba86671000.stats=clear ls -l files in d123ad.sanity: 10001 real 0m48.988s user 0m0.305s sys 0m16.875s batched statahead RPC= hit count=9999 mdc.lustre-MDT0000-mdc-ffff8cba86671000.batch_stats= snapshot_time: 1781243341.245097395 secs.nsecs start_time: 1781243279.200506720 secs.nsecs elapsed_time: 62.044590675 secs.nsecs subreqs per batch batches % cum % 1: 296 60 60 2: 64 13 73 4: 36 7 80 8: 22 4 85 16: 14 2 87 32: 19 3 91 64: 7 1 93 128: 0 0 93 256: 33 6 100 mdc.lustre-MDT0001-mdc-ffff8cba86671000.batch_stats= snapshot_time: 1781243341.245214955 secs.nsecs start_time: 1781243279.200588054 secs.nsecs elapsed_time: 62.044626901 secs.nsecs subreqs per batch batches % cum % 1: 0 0 0 unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second seq.cli-lustre-OST0000-super.width=65536 seq.cli-lustre-OST0001-super.width=65536 llite.lustre-ffff8cba86671000.statahead_batch_max=64 llite.lustre-ffff8cba86671000.statahead_max=128