== sanity test 123ad: Verify batching statahead works correctly ========================================================== 22:20:11 (1788920411) batch: statahead_max=32 statahead_batch_max=32 mdc.lustre-MDT0000-mdc-ffff8c68d8fb6800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff8c68d8fb6800.batch_stats=0 llite.lustre-ffff8c68d8fb6800.statahead_max=32 llite.lustre-ffff8c68d8fb6800.statahead_batch_max=32 seq.cli-lustre-OST0000-super.width=0x1ffffff seq.cli-lustre-OST0001-super.width=0x1ffffff kvm mdc.lustre-MDT0000-mdc-ffff8c68d8fb6800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff8c68d8fb6800.batch_stats=0 striped dir -i1 -c2 -H all_char /mnt/lustre/d123ad.sanity total: 100 open/close in 3.30 seconds: 30.34 ops/second llite.lustre-ffff8c68d8fb6800.statahead_max=0 101 real 0m2.568s user 0m0.005s sys 0m0.352s 'ls -l' 100 files without statahead: 3 sec llite.lustre-ffff8c68d8fb6800.statahead_max=32 llite.lustre-ffff8c68d8fb6800.statahead_max=32 101 real 0m1.166s user 0m0.004s sys 0m0.184s 'ls -l' 100 files with statahead: 1 sec statahead total: 142 statahead wrong: 2 agl total: 136 list_total: 31 fname_total: 101 hit_total: 4068 miss_total: 1290 swrong=2 ewrong=2 - open/close 730 (time 1788920441.48 total 10.00 last 72.98) total: 900 open/close in 12.29 seconds: 73.23 ops/second llite.lustre-ffff8c68d8fb6800.statahead_max=0 1001 real 0m22.740s user 0m0.031s sys 0m3.346s 'ls -l' 1000 files without statahead: 23 sec llite.lustre-ffff8c68d8fb6800.statahead_max=32 llite.lustre-ffff8c68d8fb6800.statahead_max=32 1001 real 0m11.696s user 0m0.034s sys 0m1.607s 'ls -l' 1000 files with statahead: 12 sec statahead total: 143 statahead wrong: 2 agl total: 137 list_total: 32 fname_total: 101 hit_total: 5067 miss_total: 1291 swrong=2 ewrong=2 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 21 seconds llite.lustre-ffff8c68d8fb6800.statahead_stats= statahead total: 143 statahead wrong: 2 agl total: 137 list_total: 32 fname_total: 101 hit_total: 5067 miss_total: 1291 mdc.lustre-MDT0000-mdc-ffff8c68d8fb6800.batch_stats= snapshot_time: 1788920513.479261182 secs.nsecs start_time: 1788920416.955966478 secs.nsecs elapsed_time: 96.523294704 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 1 2 2 4: 1 2 5 8: 4 10 15 16: 16 41 56 32: 17 43 100 mdc.lustre-MDT0001-mdc-ffff8c68d8fb6800.batch_stats= snapshot_time: 1788920513.479330752 secs.nsecs start_time: 1788920416.956096209 secs.nsecs elapsed_time: 96.523234543 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 5 12 12 8: 0 0 12 16: 21 53 66 32: 13 33 100 - open/close 774 (time 1788920526.25 total 10.01 last 77.30) - open/close 1561 (time 1788920536.26 total 20.03 last 78.61) - open/close 2366 (time 1788920546.28 total 30.04 last 80.36) - open/close 3123 (time 1788920556.29 total 40.05 last 75.64) - open/close 3883 (time 1788920566.29 total 50.05 last 76.00) - open/close 4561 (time 1788920576.30 total 60.06 last 67.73) - open/close 5304 (time 1788920586.30 total 70.07 last 74.28) - open/close 6029 (time 1788920596.31 total 80.07 last 72.45) - open/close 6777 (time 1788920606.31 total 90.08 last 74.76) - open/close 7502 (time 1788920616.32 total 100.08 last 72.46) - open/close 8366 (time 1788920626.33 total 110.09 last 86.33) - open/close 9040 (time 1788920636.33 total 120.10 last 67.37) - open/close 9795 (time 1788920646.34 total 130.11 last 75.43) total: 10000 open/close in 132.87 seconds: 75.26 ops/second llite.lustre-ffff8c68d8fb6800.statahead_batch_max=0 llite.lustre-ffff8c68d8fb6800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff8c68d8fb6800.stats=clear mdc.lustre-MDT0001-mdc-ffff8c68d8fb6800.stats=clear ls -l files in d123ad.sanity: 10001 real 1m6.972s user 0m0.352s sys 0m17.144s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff8c68d8fb6800.statahead_batch_max=32 llite.lustre-ffff8c68d8fb6800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff8c68d8fb6800.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff8c68d8fb6800.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff8c68d8fb6800.stats=clear mdc.lustre-MDT0001-mdc-ffff8c68d8fb6800.stats=clear ls -l files in d123ad.sanity: 10001 real 1m34.252s user 0m0.289s sys 0m16.416s batched statahead RPC= hit count=9982 mdc.lustre-MDT0000-mdc-ffff8c68d8fb6800.batch_stats= snapshot_time: 1788920837.336341008 secs.nsecs start_time: 1788920729.265062036 secs.nsecs elapsed_time: 108.071278972 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 0 0 0 8: 1 0 0 16: 1 0 0 32: 313 99 100 mdc.lustre-MDT0001-mdc-ffff8c68d8fb6800.batch_stats= snapshot_time: 1788920837.336407861 secs.nsecs start_time: 1788920729.265165670 secs.nsecs elapsed_time: 108.071242191 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 batch: statahead_max=2048 statahead_batch_max=256 mdc.lustre-MDT0000-mdc-ffff8c68d8fb6800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff8c68d8fb6800.batch_stats=0 llite.lustre-ffff8c68d8fb6800.statahead_max=2048 llite.lustre-ffff8c68d8fb6800.statahead_batch_max=256 kvm mdc.lustre-MDT0000-mdc-ffff8c68d8fb6800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff8c68d8fb6800.batch_stats=0 striped dir -i1 -c2 -H crush2 /mnt/lustre/d123ad.sanity total: 100 open/close in 2.11 seconds: 47.31 ops/second llite.lustre-ffff8c68d8fb6800.statahead_max=0 101 real 0m2.130s user 0m0.005s sys 0m0.348s 'ls -l' 100 files without statahead: 2 sec llite.lustre-ffff8c68d8fb6800.statahead_max=2048 llite.lustre-ffff8c68d8fb6800.statahead_max=2048 101 real 0m0.657s user 0m0.007s sys 0m0.240s 'ls -l' 100 files with statahead: 0 sec statahead total: 2 statahead wrong: 0 agl total: 2 list_total: 2 fname_total: 0 hit_total: 10081 miss_total: 19 swrong=0 ewrong=0 total: 900 open/close in 9.67 seconds: 93.03 ops/second llite.lustre-ffff8c68d8fb6800.statahead_max=0 1001 real 0m18.041s user 0m0.036s sys 0m3.221s 'ls -l' 1000 files without statahead: 18 sec llite.lustre-ffff8c68d8fb6800.statahead_max=2048 llite.lustre-ffff8c68d8fb6800.statahead_max=2048 1001 real 0m4.218s user 0m0.024s sys 0m1.764s 'ls -l' 1000 files with statahead: 4 sec statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11080 miss_total: 20 swrong=0 ewrong=0 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 12 seconds llite.lustre-ffff8c68d8fb6800.statahead_stats= statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11080 miss_total: 20 mdc.lustre-MDT0000-mdc-ffff8c68d8fb6800.batch_stats= snapshot_time: 1788921121.191135758 secs.nsecs start_time: 1788920839.209599202 secs.nsecs elapsed_time: 281.981536556 secs.nsecs subreqs per batch batches %% cum %% 1: 1 14 14 2: 0 0 14 4: 3 42 57 8: 0 0 57 16: 0 0 57 32: 0 0 57 64: 1 14 71 128: 1 14 85 256: 1 14 100 mdc.lustre-MDT0001-mdc-ffff8c68d8fb6800.batch_stats= snapshot_time: 1788921121.191204871 secs.nsecs start_time: 1788920839.209772499 secs.nsecs elapsed_time: 281.981432372 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 1 11 11 8: 3 33 44 16: 0 0 44 32: 1 11 55 64: 2 22 77 128: 0 0 77 256: 2 22 100 - open/close 1037 (time 1788921132.80 total 10.00 last 103.65) - open/close 2120 (time 1788921142.80 total 20.01 last 108.29) - open/close 3388 (time 1788921152.81 total 30.02 last 126.66) - open/close 4482 (time 1788921162.83 total 40.04 last 109.20) - open/close 5654 (time 1788921172.83 total 50.04 last 117.13) - open/close 6799 (time 1788921182.85 total 60.05 last 114.35) - open/close 8011 (time 1788921192.85 total 70.06 last 121.15) - open/close 9178 (time 1788921202.86 total 80.07 last 116.59) total: 10000 open/close in 87.52 seconds: 114.25 ops/second llite.lustre-ffff8c68d8fb6800.statahead_batch_max=0 llite.lustre-ffff8c68d8fb6800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff8c68d8fb6800.stats=clear mdc.lustre-MDT0001-mdc-ffff8c68d8fb6800.stats=clear ls -l files in d123ad.sanity: 10001 real 0m40.254s user 0m0.289s sys 0m24.140s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff8c68d8fb6800.statahead_batch_max=256 llite.lustre-ffff8c68d8fb6800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff8c68d8fb6800.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff8c68d8fb6800.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff8c68d8fb6800.stats=clear mdc.lustre-MDT0001-mdc-ffff8c68d8fb6800.stats=clear ls -l files in d123ad.sanity: 10001 real 0m35.975s user 0m0.250s sys 0m21.172s batched statahead RPC= hit count=9999 mdc.lustre-MDT0000-mdc-ffff8c68d8fb6800.batch_stats= snapshot_time: 1788921299.277242989 secs.nsecs start_time: 1788921255.474668381 secs.nsecs elapsed_time: 43.802574608 secs.nsecs subreqs per batch batches %% cum %% 1: 375 62 62 2: 92 15 77 4: 62 10 87 8: 13 2 89 16: 6 0 90 32: 17 2 93 64: 5 0 94 128: 0 0 94 256: 33 5 100 mdc.lustre-MDT0001-mdc-ffff8c68d8fb6800.batch_stats= snapshot_time: 1788921299.277314000 secs.nsecs start_time: 1788921255.474739258 secs.nsecs elapsed_time: 43.802574742 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second seq.cli-lustre-OST0000-super.width=65536 seq.cli-lustre-OST0001-super.width=65536 llite.lustre-ffff8c68d8fb6800.statahead_batch_max=64 llite.lustre-ffff8c68d8fb6800.statahead_max=128