== sanity test 123ad: Verify batching statahead works correctly ========================================================== 22:31:30 (1788489090) batch: statahead_max=32 statahead_batch_max=32 mdc.lustre-MDT0000-mdc-ffff92ce46e3c800.batch_stats=0 llite.lustre-ffff92ce46e3c800.statahead_max=32 llite.lustre-ffff92ce46e3c800.statahead_batch_max=32 seq.cli-lustre-OST0000-super.width=0x1ffffff seq.cli-lustre-OST0001-super.width=0x1ffffff kvm mdc.lustre-MDT0000-mdc-ffff92ce46e3c800.batch_stats=0 total: 100 open/close in 3.17 seconds: 31.51 ops/second llite.lustre-ffff92ce46e3c800.statahead_max=0 101 real 0m2.665s user 0m0.003s sys 0m0.288s 'ls -l' 100 files without statahead: 3 sec llite.lustre-ffff92ce46e3c800.statahead_max=32 llite.lustre-ffff92ce46e3c800.statahead_max=32 101 real 0m1.120s user 0m0.005s sys 0m0.154s 'ls -l' 100 files with statahead: 1 sec statahead total: 137 statahead wrong: 2 agl total: 131 list_total: 31 fname_total: 93 hit_total: 3936 miss_total: 1712 swrong=2 ewrong=2 - open/close 831 (time 1788489121.33 total 10.00 last 83.06) total: 900 open/close in 11.00 seconds: 81.80 ops/second llite.lustre-ffff92ce46e3c800.statahead_max=0 1001 real 0m24.550s user 0m0.034s sys 0m2.816s 'ls -l' 1000 files without statahead: 24 sec llite.lustre-ffff92ce46e3c800.statahead_max=32 llite.lustre-ffff92ce46e3c800.statahead_max=32 1001 real 0m10.388s user 0m0.028s sys 0m1.275s 'ls -l' 1000 files with statahead: 10 sec statahead total: 138 statahead wrong: 2 agl total: 132 list_total: 32 fname_total: 93 hit_total: 4934 miss_total: 1714 swrong=2 ewrong=2 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 21 seconds llite.lustre-ffff92ce46e3c800.statahead_stats= statahead total: 138 statahead wrong: 2 agl total: 132 list_total: 32 fname_total: 93 hit_total: 4934 miss_total: 1714 mdc.lustre-MDT0000-mdc-ffff92ce46e3c800.batch_stats= snapshot_time: 1788489191.672738236 secs.nsecs start_time: 1788489095.943826154 secs.nsecs elapsed_time: 95.728912082 secs.nsecs subreqs per batch batches %% cum %% 1: 1 2 2 2: 0 0 2 4: 0 0 2 8: 3 7 10 16: 1 2 12 32: 34 87 100 - open/close 856 (time 1788489203.82 total 10.02 last 85.46) - open/close 1577 (time 1788489213.82 total 20.02 last 72.10) - open/close 2235 (time 1788489223.84 total 30.03 last 65.71) - open/close 2807 (time 1788489233.86 total 40.05 last 57.09) - open/close 3566 (time 1788489243.86 total 50.06 last 75.84) - open/close 4407 (time 1788489253.87 total 60.07 last 84.02) - open/close 5149 (time 1788489263.88 total 70.07 last 74.18) - open/close 5894 (time 1788489273.89 total 80.08 last 74.43) - open/close 6556 (time 1788489283.90 total 90.09 last 66.12) - open/close 7473 (time 1788489293.90 total 100.10 last 91.65) - open/close 8214 (time 1788489303.90 total 110.10 last 74.08) - open/close 8975 (time 1788489313.91 total 120.10 last 76.09) - open/close 9737 (time 1788489323.91 total 130.10 last 76.17) total: 10000 open/close in 133.20 seconds: 75.08 ops/second llite.lustre-ffff92ce46e3c800.statahead_batch_max=0 llite.lustre-ffff92ce46e3c800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff92ce46e3c800.stats=clear ls -l files in d123ad.sanity: 10001 real 0m59.873s user 0m0.303s sys 0m13.339s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff92ce46e3c800.statahead_batch_max=32 llite.lustre-ffff92ce46e3c800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff92ce46e3c800.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff92ce46e3c800.stats=clear ls -l files in d123ad.sanity: 10001 real 1m37.519s user 0m0.276s sys 0m12.680s batched statahead RPC= hit count=9978 mdc.lustre-MDT0000-mdc-ffff92ce46e3c800.batch_stats= snapshot_time: 1788489506.610072766 secs.nsecs start_time: 1788489397.708791375 secs.nsecs elapsed_time: 108.901281391 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 1 0 0 4: 0 0 0 8: 1 0 0 16: 1 0 0 32: 313 99 100 batch: statahead_max=2048 statahead_batch_max=256 mdc.lustre-MDT0000-mdc-ffff92ce46e3c800.batch_stats=0 llite.lustre-ffff92ce46e3c800.statahead_max=2048 llite.lustre-ffff92ce46e3c800.statahead_batch_max=256 kvm mdc.lustre-MDT0000-mdc-ffff92ce46e3c800.batch_stats=0 total: 100 open/close in 2.55 seconds: 39.24 ops/second llite.lustre-ffff92ce46e3c800.statahead_max=0 101 real 0m2.132s user 0m0.006s sys 0m0.303s 'ls -l' 100 files without statahead: 2 sec llite.lustre-ffff92ce46e3c800.statahead_max=2048 llite.lustre-ffff92ce46e3c800.statahead_max=2048 101 real 0m1.254s user 0m0.007s sys 0m0.168s 'ls -l' 100 files with statahead: 1 sec statahead total: 2 statahead wrong: 0 agl total: 2 list_total: 2 fname_total: 0 hit_total: 10077 miss_total: 23 swrong=0 ewrong=0 - open/close 624 (time 1788489791.20 total 10.01 last 62.33) total: 900 open/close in 13.98 seconds: 64.37 ops/second llite.lustre-ffff92ce46e3c800.statahead_max=0 1001 real 0m19.083s user 0m0.035s sys 0m2.771s 'ls -l' 1000 files without statahead: 19 sec llite.lustre-ffff92ce46e3c800.statahead_max=2048 llite.lustre-ffff92ce46e3c800.statahead_max=2048 1001 real 0m7.234s user 0m0.042s sys 0m1.439s 'ls -l' 1000 files with statahead: 7 sec statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11076 miss_total: 24 swrong=0 ewrong=0 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 19 seconds llite.lustre-ffff92ce46e3c800.statahead_stats= statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11076 miss_total: 24 mdc.lustre-MDT0000-mdc-ffff92ce46e3c800.batch_stats= snapshot_time: 1788489853.018791025 secs.nsecs start_time: 1788489510.042801523 secs.nsecs elapsed_time: 342.975989502 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 0 0 0 8: 2 16 16 16: 2 16 33 32: 2 16 50 64: 1 8 58 128: 1 8 66 256: 4 33 100 - open/close 663 (time 1788489865.96 total 10.00 last 66.28) - open/close 1501 (time 1788489875.98 total 20.02 last 83.64) - open/close 2347 (time 1788489885.99 total 30.03 last 84.52) - open/close 3066 (time 1788489896.00 total 40.04 last 71.84) - open/close 3849 (time 1788489906.00 total 50.04 last 78.28) - open/close 4664 (time 1788489916.00 total 60.05 last 81.46) - open/close 5587 (time 1788489926.01 total 70.05 last 92.28) - open/close 6420 (time 1788489936.01 total 80.05 last 83.28) - open/close 7094 (time 1788489946.02 total 90.06 last 67.33) - open/close 7912 (time 1788489956.02 total 100.07 last 81.78) - open/close 8656 (time 1788489966.03 total 110.08 last 74.30) - open/close 9317 (time 1788489976.04 total 120.08 last 66.08) total: 10000 open/close in 128.87 seconds: 77.60 ops/second llite.lustre-ffff92ce46e3c800.statahead_batch_max=0 llite.lustre-ffff92ce46e3c800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff92ce46e3c800.stats=clear ls -l files in d123ad.sanity: 10001 real 1m8.399s user 0m0.321s sys 0m13.633s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff92ce46e3c800.statahead_batch_max=256 llite.lustre-ffff92ce46e3c800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff92ce46e3c800.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff92ce46e3c800.stats=clear ls -l files in d123ad.sanity: 10001 real 0m54.689s user 0m0.304s sys 0m13.019s batched statahead RPC= hit count=9999 mdc.lustre-MDT0000-mdc-ffff92ce46e3c800.batch_stats= snapshot_time: 1788490129.647778839 secs.nsecs start_time: 1788490063.373408740 secs.nsecs elapsed_time: 66.274370099 secs.nsecs subreqs per batch batches %% cum %% 1: 129 45 45 2: 41 14 59 4: 28 9 69 8: 17 5 75 16: 9 3 78 32: 20 7 85 64: 5 1 87 128: 0 0 87 256: 35 12 100 unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second seq.cli-lustre-OST0000-super.width=65536 seq.cli-lustre-OST0001-super.width=65536 llite.lustre-ffff92ce46e3c800.statahead_batch_max=64 llite.lustre-ffff92ce46e3c800.statahead_max=128