== sanity test 123ad: Verify batching statahead works correctly ========================================================== 01:41:29 (1783143689) batch: statahead_max=32 statahead_batch_max=32 mdc.lustre-MDT0000-mdc-ffff9f3204f00800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9f3204f00800.batch_stats=0 llite.lustre-ffff9f3204f00800.statahead_max=32 llite.lustre-ffff9f3204f00800.statahead_batch_max=32 seq.cli-lustre-OST0000-super.width=0x1ffffff seq.cli-lustre-OST0001-super.width=0x1ffffff kvm mdc.lustre-MDT0000-mdc-ffff9f3204f00800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9f3204f00800.batch_stats=0 striped dir -i1 -c2 -H crush2 /mnt/lustre/d123ad.sanity total: 100 open/close in 1.81 seconds: 55.35 ops/second llite.lustre-ffff9f3204f00800.statahead_max=0 101 real 0m1.523s user 0m0.003s sys 0m0.353s 'ls -l' 100 files without statahead: 2 sec llite.lustre-ffff9f3204f00800.statahead_max=32 llite.lustre-ffff9f3204f00800.statahead_max=32 101 real 0m0.655s user 0m0.005s sys 0m0.216s 'ls -l' 100 files with statahead: 1 sec statahead total: 126 statahead wrong: 2 agl total: 120 list_total: 32 fname_total: 92 hit_total: 3674 miss_total: 1516 swrong=2 ewrong=2 total: 900 open/close in 8.49 seconds: 105.96 ops/second llite.lustre-ffff9f3204f00800.statahead_max=0 1001 real 0m15.474s user 0m0.040s sys 0m3.386s 'ls -l' 1000 files without statahead: 15 sec llite.lustre-ffff9f3204f00800.statahead_max=32 llite.lustre-ffff9f3204f00800.statahead_max=32 1001 real 0m5.136s user 0m0.029s sys 0m1.808s 'ls -l' 1000 files with statahead: 5 sec statahead total: 127 statahead wrong: 2 agl total: 121 list_total: 33 fname_total: 92 hit_total: 4672 miss_total: 1518 swrong=2 ewrong=2 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 11 seconds llite.lustre-ffff9f3204f00800.statahead_stats= statahead total: 127 statahead wrong: 2 agl total: 121 list_total: 33 fname_total: 92 hit_total: 4672 miss_total: 1518 mdc.lustre-MDT0000-mdc-ffff9f3204f00800.batch_stats= snapshot_time: 1783143746.964884836 secs.nsecs start_time: 1783143691.834368346 secs.nsecs elapsed_time: 55.130516490 secs.nsecs subreqs per batch batches %% cum %% 1: 1 2 2 2: 4 10 13 4: 1 2 16 8: 1 2 18 16: 24 64 83 32: 6 16 100 mdc.lustre-MDT0001-mdc-ffff9f3204f00800.batch_stats= snapshot_time: 1783143746.964961976 secs.nsecs start_time: 1783143691.834545001 secs.nsecs elapsed_time: 55.130416975 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 1 2 2 4: 0 0 2 8: 3 7 10 16: 14 35 46 32: 21 53 100 - open/close 1062 (time 1783143758.48 total 10.00 last 106.17) - open/close 2087 (time 1783143768.49 total 20.01 last 102.42) - open/close 3186 (time 1783143778.49 total 30.01 last 109.88) - open/close 4345 (time 1783143788.50 total 40.02 last 115.85) - open/close 5471 (time 1783143798.50 total 50.02 last 112.57) - open/close 6569 (time 1783143808.50 total 60.02 last 109.77) - open/close 7790 (time 1783143818.51 total 70.03 last 122.04) - open/close 9020 (time 1783143828.51 total 80.03 last 122.98) total: 10000 open/close in 87.44 seconds: 114.37 ops/second llite.lustre-ffff9f3204f00800.statahead_batch_max=0 llite.lustre-ffff9f3204f00800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9f3204f00800.stats=clear mdc.lustre-MDT0001-mdc-ffff9f3204f00800.stats=clear ls -l files in d123ad.sanity: 10001 real 0m37.154s user 0m0.296s sys 0m30.888s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff9f3204f00800.statahead_batch_max=32 llite.lustre-ffff9f3204f00800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9f3204f00800.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff9f3204f00800.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff9f3204f00800.stats=clear mdc.lustre-MDT0001-mdc-ffff9f3204f00800.stats=clear ls -l files in d123ad.sanity: 10001 real 0m56.884s user 0m0.282s sys 0m20.980s batched statahead RPC= hit count=9986 mdc.lustre-MDT0000-mdc-ffff9f3204f00800.batch_stats= snapshot_time: 1783143941.961186478 secs.nsecs start_time: 1783143877.640263382 secs.nsecs elapsed_time: 64.320923096 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 0 0 0 8: 1 0 0 16: 1 0 0 32: 313 99 100 mdc.lustre-MDT0001-mdc-ffff9f3204f00800.batch_stats= snapshot_time: 1783143941.961250638 secs.nsecs start_time: 1783143877.640355419 secs.nsecs elapsed_time: 64.320895219 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 batch: statahead_max=2048 statahead_batch_max=256 mdc.lustre-MDT0000-mdc-ffff9f3204f00800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9f3204f00800.batch_stats=0 llite.lustre-ffff9f3204f00800.statahead_max=2048 llite.lustre-ffff9f3204f00800.statahead_batch_max=256 kvm mdc.lustre-MDT0000-mdc-ffff9f3204f00800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9f3204f00800.batch_stats=0 striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d123ad.sanity total: 100 open/close in 1.52 seconds: 65.94 ops/second llite.lustre-ffff9f3204f00800.statahead_max=0 101 real 0m1.559s user 0m0.007s sys 0m0.332s 'ls -l' 100 files without statahead: 1 sec llite.lustre-ffff9f3204f00800.statahead_max=2048 llite.lustre-ffff9f3204f00800.statahead_max=2048 101 real 0m0.420s user 0m0.006s sys 0m0.199s 'ls -l' 100 files with statahead: 1 sec statahead total: 2 statahead wrong: 0 agl total: 2 list_total: 2 fname_total: 0 hit_total: 10085 miss_total: 15 swrong=0 ewrong=0 total: 900 open/close in 6.24 seconds: 144.22 ops/second llite.lustre-ffff9f3204f00800.statahead_max=0 1001 real 0m13.966s user 0m0.037s sys 0m3.144s 'ls -l' 1000 files without statahead: 14 sec llite.lustre-ffff9f3204f00800.statahead_max=2048 llite.lustre-ffff9f3204f00800.statahead_max=2048 1001 real 0m4.597s user 0m0.016s sys 0m1.860s 'ls -l' 1000 files with statahead: 4 sec statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11084 miss_total: 16 swrong=0 ewrong=0 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 12 seconds llite.lustre-ffff9f3204f00800.statahead_stats= statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11084 miss_total: 16 mdc.lustre-MDT0000-mdc-ffff9f3204f00800.batch_stats= snapshot_time: 1783144177.554371992 secs.nsecs start_time: 1783143943.175292355 secs.nsecs elapsed_time: 234.379079637 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 1 11 11 4: 2 22 33 8: 2 22 55 16: 1 11 66 32: 1 11 77 64: 0 0 77 128: 0 0 77 256: 2 22 100 mdc.lustre-MDT0001-mdc-ffff9f3204f00800.batch_stats= snapshot_time: 1783144177.554445424 secs.nsecs start_time: 1783143943.175479446 secs.nsecs elapsed_time: 234.378965978 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 1 11 11 4: 0 0 11 8: 4 44 55 16: 1 11 66 32: 0 0 66 64: 1 11 77 128: 0 0 77 256: 2 22 100 - open/close 1203 (time 1783144189.05 total 10.01 last 120.23) - open/close 2417 (time 1783144199.05 total 20.01 last 121.34) - open/close 3555 (time 1783144209.05 total 30.01 last 113.79) - open/close 4701 (time 1783144219.05 total 40.01 last 114.57) - open/close 5782 (time 1783144229.06 total 50.02 last 108.05) - open/close 6941 (time 1783144239.07 total 60.03 last 115.78) - open/close 8123 (time 1783144249.07 total 70.03 last 118.18) - open/close 9225 (time 1783144259.08 total 80.04 last 110.14) total: 10000 open/close in 86.81 seconds: 115.19 ops/second llite.lustre-ffff9f3204f00800.statahead_batch_max=0 llite.lustre-ffff9f3204f00800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9f3204f00800.stats=clear mdc.lustre-MDT0001-mdc-ffff9f3204f00800.stats=clear ls -l files in d123ad.sanity: 10001 real 0m38.453s user 0m0.285s sys 0m27.431s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff9f3204f00800.statahead_batch_max=256 llite.lustre-ffff9f3204f00800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9f3204f00800.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff9f3204f00800.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff9f3204f00800.stats=clear mdc.lustre-MDT0001-mdc-ffff9f3204f00800.stats=clear ls -l files in d123ad.sanity: 10001 real 0m33.368s user 0m0.254s sys 0m21.769s batched statahead RPC= hit count=9999 mdc.lustre-MDT0000-mdc-ffff9f3204f00800.batch_stats= snapshot_time: 1783144350.958364454 secs.nsecs start_time: 1783144308.787685953 secs.nsecs elapsed_time: 42.170678501 secs.nsecs subreqs per batch batches %% cum %% 1: 326 66 66 2: 64 13 79 4: 33 6 85 8: 8 1 87 16: 2 0 88 32: 18 3 91 64: 7 1 93 128: 0 0 93 256: 34 6 100 mdc.lustre-MDT0001-mdc-ffff9f3204f00800.batch_stats= snapshot_time: 1783144350.958432539 secs.nsecs start_time: 1783144308.787860018 secs.nsecs elapsed_time: 42.170572521 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second seq.cli-lustre-OST0000-super.width=65536 seq.cli-lustre-OST0001-super.width=65536 llite.lustre-ffff9f3204f00800.statahead_batch_max=64 llite.lustre-ffff9f3204f00800.statahead_max=128