== sanity test 123ad: Verify batching statahead works correctly ========================================================== 10:21:28 (1790778088) batch: statahead_max=32 statahead_batch_max=32 mdc.lustre-MDT0000-mdc-ffff8a8769935000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff8a8769935000.batch_stats=0 llite.lustre-ffff8a8769935000.statahead_max=32 llite.lustre-ffff8a8769935000.statahead_batch_max=32 seq.cli-lustre-OST0000-super.width=0x1ffffff seq.cli-lustre-OST0001-super.width=0x1ffffff kvm mdc.lustre-MDT0000-mdc-ffff8a8769935000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff8a8769935000.batch_stats=0 striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d123ad.sanity total: 100 open/close in 2.07 seconds: 48.40 ops/second llite.lustre-ffff8a8769935000.statahead_max=0 101 real 0m2.064s user 0m0.006s sys 0m0.296s 'ls -l' 100 files without statahead: 2 sec llite.lustre-ffff8a8769935000.statahead_max=32 llite.lustre-ffff8a8769935000.statahead_max=32 101 real 0m0.962s user 0m0.008s sys 0m0.183s 'ls -l' 100 files with statahead: 1 sec statahead total: 119 statahead wrong: 2 agl total: 114 list_total: 31 fname_total: 88 hit_total: 4093 miss_total: 1292 swrong=2 ewrong=2 - open/close 859 (time 1790778114.31 total 10.01 last 85.83) total: 900 open/close in 10.57 seconds: 85.18 ops/second llite.lustre-ffff8a8769935000.statahead_max=0 1001 real 0m18.121s user 0m0.032s sys 0m2.776s 'ls -l' 1000 files without statahead: 18 sec llite.lustre-ffff8a8769935000.statahead_max=32 llite.lustre-ffff8a8769935000.statahead_max=32 1001 real 0m8.426s user 0m0.022s sys 0m1.442s 'ls -l' 1000 files with statahead: 8 sec statahead total: 120 statahead wrong: 2 agl total: 115 list_total: 32 fname_total: 88 hit_total: 5090 miss_total: 1295 swrong=2 ewrong=2 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 21 seconds llite.lustre-ffff8a8769935000.statahead_stats= statahead total: 120 statahead wrong: 2 agl total: 115 list_total: 32 fname_total: 88 hit_total: 5090 miss_total: 1295 mdc.lustre-MDT0000-mdc-ffff8a8769935000.batch_stats= snapshot_time: 1790778174.988790157 secs.nsecs start_time: 1790778092.810647217 secs.nsecs elapsed_time: 82.178142940 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 1 2 2 4: 2 5 7 8: 4 10 17 16: 16 41 58 32: 16 41 100 mdc.lustre-MDT0001-mdc-ffff8a8769935000.batch_stats= snapshot_time: 1790778174.988951906 secs.nsecs start_time: 1790778092.810726476 secs.nsecs elapsed_time: 82.178225430 secs.nsecs subreqs per batch batches %% cum %% 1: 2 5 5 2: 0 0 5 4: 1 2 7 8: 3 7 15 16: 20 50 65 32: 14 35 100 - open/close 794 (time 1790778187.14 total 10.01 last 79.36) - open/close 1725 (time 1790778197.16 total 20.02 last 92.94) - open/close 2564 (time 1790778207.16 total 30.03 last 83.88) - open/close 3486 (time 1790778217.18 total 40.05 last 91.99) - open/close 4396 (time 1790778227.19 total 50.05 last 90.98) - open/close 5244 (time 1790778237.19 total 60.05 last 84.78) - open/close 6125 (time 1790778247.19 total 70.05 last 88.09) - open/close 7104 (time 1790778257.19 total 80.06 last 97.86) - open/close 8068 (time 1790778267.20 total 90.07 last 96.32) - open/close 9009 (time 1790778277.21 total 100.07 last 94.05) - open/close 9870 (time 1790778287.22 total 110.09 last 85.98) total: 10000 open/close in 111.35 seconds: 89.80 ops/second llite.lustre-ffff8a8769935000.statahead_batch_max=0 llite.lustre-ffff8a8769935000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff8a8769935000.stats=clear mdc.lustre-MDT0001-mdc-ffff8a8769935000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m3.238s user 0m0.321s sys 0m15.595s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff8a8769935000.statahead_batch_max=32 llite.lustre-ffff8a8769935000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff8a8769935000.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff8a8769935000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff8a8769935000.stats=clear mdc.lustre-MDT0001-mdc-ffff8a8769935000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m31.148s user 0m0.296s sys 0m14.111s batched statahead RPC= hit count=9986 mdc.lustre-MDT0000-mdc-ffff8a8769935000.batch_stats= snapshot_time: 1790778460.605865187 secs.nsecs start_time: 1790778359.437174434 secs.nsecs elapsed_time: 101.168690753 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 0 0 0 8: 1 0 0 16: 1 0 0 32: 313 99 100 mdc.lustre-MDT0001-mdc-ffff8a8769935000.batch_stats= snapshot_time: 1790778460.605927858 secs.nsecs start_time: 1790778359.437835204 secs.nsecs elapsed_time: 101.168092654 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 batch: statahead_max=2048 statahead_batch_max=256 mdc.lustre-MDT0000-mdc-ffff8a8769935000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff8a8769935000.batch_stats=0 llite.lustre-ffff8a8769935000.statahead_max=2048 llite.lustre-ffff8a8769935000.statahead_batch_max=256 kvm mdc.lustre-MDT0000-mdc-ffff8a8769935000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff8a8769935000.batch_stats=0 striped dir -i1 -c2 -H crush /mnt/lustre/d123ad.sanity total: 100 open/close in 1.90 seconds: 52.50 ops/second llite.lustre-ffff8a8769935000.statahead_max=0 101 real 0m1.733s user 0m0.003s sys 0m0.285s 'ls -l' 100 files without statahead: 2 sec llite.lustre-ffff8a8769935000.statahead_max=2048 llite.lustre-ffff8a8769935000.statahead_max=2048 101 real 0m0.628s user 0m0.003s sys 0m0.183s 'ls -l' 100 files with statahead: 1 sec statahead total: 2 statahead wrong: 0 agl total: 2 list_total: 2 fname_total: 0 hit_total: 10085 miss_total: 15 swrong=0 ewrong=0 - open/close 833 (time 1790778720.79 total 10.00 last 83.29) total: 900 open/close in 10.76 seconds: 83.65 ops/second llite.lustre-ffff8a8769935000.statahead_max=0 1001 real 0m19.222s user 0m0.032s sys 0m2.730s 'ls -l' 1000 files without statahead: 19 sec llite.lustre-ffff8a8769935000.statahead_max=2048 llite.lustre-ffff8a8769935000.statahead_max=2048 1001 real 0m4.194s user 0m0.025s sys 0m1.235s 'ls -l' 1000 files with statahead: 4 sec statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11084 miss_total: 16 swrong=0 ewrong=0 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 17 seconds llite.lustre-ffff8a8769935000.statahead_stats= statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11084 miss_total: 16 mdc.lustre-MDT0000-mdc-ffff8a8769935000.batch_stats= snapshot_time: 1790778771.911788913 secs.nsecs start_time: 1790778462.543780582 secs.nsecs elapsed_time: 309.368008331 secs.nsecs subreqs per batch batches %% cum %% 1: 2 28 28 2: 0 0 28 4: 3 42 71 8: 0 0 71 16: 0 0 71 32: 0 0 71 64: 0 0 71 128: 0 0 71 256: 2 28 100 mdc.lustre-MDT0001-mdc-ffff8a8769935000.batch_stats= snapshot_time: 1790778771.911861963 secs.nsecs start_time: 1790778462.544079555 secs.nsecs elapsed_time: 309.367782408 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 0 0 0 8: 3 33 33 16: 2 22 55 32: 0 0 55 64: 2 22 77 128: 0 0 77 256: 2 22 100 - open/close 919 (time 1790778784.18 total 10.00 last 91.88) - open/close 1856 (time 1790778794.18 total 20.01 last 93.67) - open/close 2779 (time 1790778804.19 total 30.02 last 92.17) - open/close 3712 (time 1790778814.20 total 40.03 last 93.22) - open/close 4594 (time 1790778824.20 total 50.03 last 88.18) - open/close 5496 (time 1790778834.21 total 60.03 last 90.17) - open/close 6418 (time 1790778844.21 total 70.04 last 92.17) - open/close 7337 (time 1790778854.22 total 80.04 last 91.84) - open/close 8247 (time 1790778864.22 total 90.05 last 90.97) - open/close 9134 (time 1790778874.23 total 100.06 last 88.60) total: 10000 open/close in 109.33 seconds: 91.47 ops/second llite.lustre-ffff8a8769935000.statahead_batch_max=0 llite.lustre-ffff8a8769935000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff8a8769935000.stats=clear mdc.lustre-MDT0001-mdc-ffff8a8769935000.stats=clear ls -l files in d123ad.sanity: 10001 real 0m45.542s user 0m0.304s sys 0m17.232s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff8a8769935000.statahead_batch_max=256 llite.lustre-ffff8a8769935000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff8a8769935000.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff8a8769935000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff8a8769935000.stats=clear mdc.lustre-MDT0001-mdc-ffff8a8769935000.stats=clear ls -l files in d123ad.sanity: 10001 real 0m39.682s user 0m0.269s sys 0m17.170s batched statahead RPC= hit count=9999 mdc.lustre-MDT0000-mdc-ffff8a8769935000.batch_stats= snapshot_time: 1790778983.372463676 secs.nsecs start_time: 1790778934.986247906 secs.nsecs elapsed_time: 48.386215770 secs.nsecs subreqs per batch batches %% cum %% 1: 181 56 56 2: 32 9 66 4: 28 8 74 8: 15 4 79 16: 8 2 81 32: 17 5 87 64: 5 1 88 128: 2 0 89 256: 34 10 100 mdc.lustre-MDT0001-mdc-ffff8a8769935000.batch_stats= snapshot_time: 1790778983.372534654 secs.nsecs start_time: 1790778934.986493521 secs.nsecs elapsed_time: 48.386041133 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second seq.cli-lustre-OST0000-super.width=65536 seq.cli-lustre-OST0001-super.width=65536 llite.lustre-ffff8a8769935000.statahead_batch_max=64 llite.lustre-ffff8a8769935000.statahead_max=128