== sanity test 123ad: Verify batching statahead works correctly ========================================================== 16:18:14 (1789676294) batch: statahead_max=32 statahead_batch_max=32 mdc.lustre-MDT0000-mdc-ffff9d2d3aca0800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9d2d3aca0800.batch_stats=0 llite.lustre-ffff9d2d3aca0800.statahead_max=32 llite.lustre-ffff9d2d3aca0800.statahead_batch_max=32 seq.cli-lustre-OST0000-super.width=0x1ffffff seq.cli-lustre-OST0001-super.width=0x1ffffff kvm mdc.lustre-MDT0000-mdc-ffff9d2d3aca0800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9d2d3aca0800.batch_stats=0 striped dir -i1 -c2 -H crush2 /mnt/lustre/d123ad.sanity total: 100 open/close in 2.18 seconds: 45.91 ops/second llite.lustre-ffff9d2d3aca0800.statahead_max=0 101 real 0m2.173s user 0m0.006s sys 0m0.288s 'ls -l' 100 files without statahead: 3 sec llite.lustre-ffff9d2d3aca0800.statahead_max=32 llite.lustre-ffff9d2d3aca0800.statahead_max=32 101 real 0m1.030s user 0m0.005s sys 0m0.181s 'ls -l' 100 files with statahead: 1 sec statahead total: 108 statahead wrong: 2 agl total: 103 list_total: 30 fname_total: 78 hit_total: 4210 miss_total: 1282 swrong=2 ewrong=2 - open/close 826 (time 1789676319.85 total 10.01 last 82.54) total: 900 open/close in 10.98 seconds: 81.99 ops/second llite.lustre-ffff9d2d3aca0800.statahead_max=0 1001 real 0m20.943s user 0m0.032s sys 0m2.923s 'ls -l' 1000 files without statahead: 21 sec llite.lustre-ffff9d2d3aca0800.statahead_max=32 llite.lustre-ffff9d2d3aca0800.statahead_max=32 1001 real 0m8.797s user 0m0.028s sys 0m1.383s 'ls -l' 1000 files with statahead: 9 sec statahead total: 109 statahead wrong: 2 agl total: 104 list_total: 31 fname_total: 78 hit_total: 5209 miss_total: 1283 swrong=2 ewrong=2 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 21 seconds llite.lustre-ffff9d2d3aca0800.statahead_stats= statahead total: 109 statahead wrong: 2 agl total: 104 list_total: 31 fname_total: 78 hit_total: 5209 miss_total: 1283 mdc.lustre-MDT0000-mdc-ffff9d2d3aca0800.batch_stats= snapshot_time: 1789676383.648744543 secs.nsecs start_time: 1789676298.361070520 secs.nsecs elapsed_time: 85.287674023 secs.nsecs subreqs per batch batches %% cum %% 1: 1 2 2 2: 0 0 2 4: 3 8 11 8: 3 8 19 16: 22 61 80 32: 7 19 100 mdc.lustre-MDT0001-mdc-ffff9d2d3aca0800.batch_stats= snapshot_time: 1789676383.648801305 secs.nsecs start_time: 1789676298.361156754 secs.nsecs elapsed_time: 85.287644551 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 1 2 2 4: 2 5 7 8: 1 2 10 16: 12 30 41 32: 23 58 100 - open/close 684 (time 1789676396.50 total 10.00 last 68.37) - open/close 1436 (time 1789676406.51 total 20.01 last 75.17) - open/close 2363 (time 1789676416.52 total 30.02 last 92.60) - open/close 3166 (time 1789676426.52 total 40.02 last 80.29) - open/close 4053 (time 1789676436.52 total 50.02 last 88.67) - open/close 4874 (time 1789676446.53 total 60.03 last 82.07) - open/close 5732 (time 1789676456.53 total 70.03 last 85.77) - open/close 6549 (time 1789676466.54 total 80.04 last 81.64) - open/close 7148 (time 1789676476.55 total 90.05 last 59.84) - open/close 7915 (time 1789676486.55 total 100.05 last 76.66) - open/close 8863 (time 1789676496.56 total 110.06 last 94.74) - open/close 9762 (time 1789676506.56 total 120.06 last 89.86) total: 10000 open/close in 122.55 seconds: 81.60 ops/second llite.lustre-ffff9d2d3aca0800.statahead_batch_max=0 llite.lustre-ffff9d2d3aca0800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9d2d3aca0800.stats=clear mdc.lustre-MDT0001-mdc-ffff9d2d3aca0800.stats=clear ls -l files in d123ad.sanity: 10001 real 0m58.672s user 0m0.305s sys 0m14.919s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff9d2d3aca0800.statahead_batch_max=32 llite.lustre-ffff9d2d3aca0800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9d2d3aca0800.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff9d2d3aca0800.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff9d2d3aca0800.stats=clear mdc.lustre-MDT0001-mdc-ffff9d2d3aca0800.stats=clear ls -l files in d123ad.sanity: 10001 real 1m34.997s user 0m0.262s sys 0m13.696s batched statahead RPC= hit count=9983 mdc.lustre-MDT0000-mdc-ffff9d2d3aca0800.batch_stats= snapshot_time: 1789676684.307804134 secs.nsecs start_time: 1789676576.512879708 secs.nsecs elapsed_time: 107.794924426 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 0 0 0 8: 1 0 0 16: 1 0 0 32: 313 99 100 mdc.lustre-MDT0001-mdc-ffff9d2d3aca0800.batch_stats= snapshot_time: 1789676684.307872048 secs.nsecs start_time: 1789676576.513098151 secs.nsecs elapsed_time: 107.794773897 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 batch: statahead_max=2048 statahead_batch_max=256 mdc.lustre-MDT0000-mdc-ffff9d2d3aca0800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9d2d3aca0800.batch_stats=0 llite.lustre-ffff9d2d3aca0800.statahead_max=2048 llite.lustre-ffff9d2d3aca0800.statahead_batch_max=256 kvm mdc.lustre-MDT0000-mdc-ffff9d2d3aca0800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9d2d3aca0800.batch_stats=0 striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d123ad.sanity total: 100 open/close in 2.03 seconds: 49.38 ops/second llite.lustre-ffff9d2d3aca0800.statahead_max=0 101 real 0m1.853s user 0m0.003s sys 0m0.295s 'ls -l' 100 files without statahead: 2 sec llite.lustre-ffff9d2d3aca0800.statahead_max=2048 llite.lustre-ffff9d2d3aca0800.statahead_max=2048 101 real 0m0.715s user 0m0.004s sys 0m0.169s 'ls -l' 100 files with statahead: 0 sec statahead total: 2 statahead wrong: 0 agl total: 2 list_total: 2 fname_total: 0 hit_total: 10082 miss_total: 18 swrong=0 ewrong=0 - open/close 822 (time 1789676966.16 total 10.00 last 82.17) total: 900 open/close in 10.89 seconds: 82.68 ops/second llite.lustre-ffff9d2d3aca0800.statahead_max=0 1001 real 0m21.305s user 0m0.026s sys 0m2.829s 'ls -l' 1000 files without statahead: 22 sec llite.lustre-ffff9d2d3aca0800.statahead_max=2048 llite.lustre-ffff9d2d3aca0800.statahead_max=2048 1001 real 0m7.160s user 0m0.023s sys 0m1.398s 'ls -l' 1000 files with statahead: 7 sec statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11081 miss_total: 19 swrong=0 ewrong=0 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 18 seconds llite.lustre-ffff9d2d3aca0800.statahead_stats= statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11081 miss_total: 19 mdc.lustre-MDT0000-mdc-ffff9d2d3aca0800.batch_stats= snapshot_time: 1789677026.478697754 secs.nsecs start_time: 1789676686.866796412 secs.nsecs elapsed_time: 339.611901342 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 3 37 37 8: 1 12 50 16: 0 0 50 32: 2 25 75 64: 0 0 75 128: 0 0 75 256: 2 25 100 mdc.lustre-MDT0001-mdc-ffff9d2d3aca0800.batch_stats= snapshot_time: 1789677026.479064294 secs.nsecs start_time: 1789676686.866953613 secs.nsecs elapsed_time: 339.612110681 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 3 37 37 8: 1 12 50 16: 1 12 62 32: 0 0 62 64: 1 12 75 128: 0 0 75 256: 2 25 100 - open/close 848 (time 1789677038.77 total 10.00 last 84.80) - open/close 1717 (time 1789677048.79 total 20.01 last 86.80) - open/close 2619 (time 1789677058.79 total 30.01 last 90.20) - open/close 3529 (time 1789677068.79 total 40.01 last 90.99) - open/close 4427 (time 1789677078.79 total 50.01 last 89.79) - open/close 5293 (time 1789677088.79 total 60.02 last 86.55) - open/close 6237 (time 1789677098.80 total 70.03 last 94.32) - open/close 7040 (time 1789677108.81 total 80.04 last 80.24) - open/close 7894 (time 1789677118.82 total 90.05 last 85.31) - open/close 8705 (time 1789677128.83 total 100.05 last 81.03) - open/close 9501 (time 1789677138.84 total 110.07 last 79.49) total: 10000 open/close in 115.86 seconds: 86.31 ops/second llite.lustre-ffff9d2d3aca0800.statahead_batch_max=0 llite.lustre-ffff9d2d3aca0800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9d2d3aca0800.stats=clear mdc.lustre-MDT0001-mdc-ffff9d2d3aca0800.stats=clear ls -l files in d123ad.sanity: 10001 real 1m5.802s user 0m0.302s sys 0m15.128s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff9d2d3aca0800.statahead_batch_max=256 llite.lustre-ffff9d2d3aca0800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9d2d3aca0800.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff9d2d3aca0800.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff9d2d3aca0800.stats=clear mdc.lustre-MDT0001-mdc-ffff9d2d3aca0800.stats=clear ls -l files in d123ad.sanity: 10001 real 0m56.491s user 0m0.288s sys 0m15.041s batched statahead RPC= hit count=9999 mdc.lustre-MDT0000-mdc-ffff9d2d3aca0800.batch_stats= snapshot_time: 1789677283.797948238 secs.nsecs start_time: 1789677218.573685757 secs.nsecs elapsed_time: 65.224262481 secs.nsecs subreqs per batch batches %% cum %% 1: 130 44 44 2: 43 14 58 4: 30 10 68 8: 22 7 76 16: 8 2 78 32: 17 5 84 64: 7 2 87 128: 6 2 89 256: 32 10 100 mdc.lustre-MDT0001-mdc-ffff9d2d3aca0800.batch_stats= snapshot_time: 1789677283.798038734 secs.nsecs start_time: 1789677218.573822023 secs.nsecs elapsed_time: 65.224216711 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second seq.cli-lustre-OST0000-super.width=65536 seq.cli-lustre-OST0001-super.width=65536 llite.lustre-ffff9d2d3aca0800.statahead_batch_max=64 llite.lustre-ffff9d2d3aca0800.statahead_max=128