== sanity test 123ad: Verify batching statahead works correctly ========================================================== 23:12:55 (1788750775) batch: statahead_max=32 statahead_batch_max=32 mdc.lustre-MDT0000-mdc-ffff8a15d083a000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff8a15d083a000.batch_stats=0 llite.lustre-ffff8a15d083a000.statahead_max=32 llite.lustre-ffff8a15d083a000.statahead_batch_max=32 seq.cli-lustre-OST0000-super.width=0x1ffffff seq.cli-lustre-OST0001-super.width=0x1ffffff kvm mdc.lustre-MDT0000-mdc-ffff8a15d083a000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff8a15d083a000.batch_stats=0 striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d123ad.sanity total: 100 open/close in 2.52 seconds: 39.72 ops/second llite.lustre-ffff8a15d083a000.statahead_max=0 101 real 0m1.922s user 0m0.006s sys 0m0.322s 'ls -l' 100 files without statahead: 2 sec llite.lustre-ffff8a15d083a000.statahead_max=32 llite.lustre-ffff8a15d083a000.statahead_max=32 101 real 0m1.051s user 0m0.005s sys 0m0.189s 'ls -l' 100 files with statahead: 1 sec statahead total: 133 statahead wrong: 2 agl total: 127 list_total: 31 fname_total: 99 hit_total: 4103 miss_total: 1611 swrong=2 ewrong=2 - open/close 727 (time 1788750804.02 total 10.01 last 72.64) total: 900 open/close in 12.18 seconds: 73.90 ops/second llite.lustre-ffff8a15d083a000.statahead_max=0 1001 real 0m20.825s user 0m0.040s sys 0m3.253s 'ls -l' 1000 files without statahead: 21 sec llite.lustre-ffff8a15d083a000.statahead_max=32 llite.lustre-ffff8a15d083a000.statahead_max=32 1001 real 0m10.407s user 0m0.034s sys 0m1.565s 'ls -l' 1000 files with statahead: 10 sec statahead total: 134 statahead wrong: 2 agl total: 128 list_total: 32 fname_total: 99 hit_total: 5099 miss_total: 1615 swrong=2 ewrong=2 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 20 seconds llite.lustre-ffff8a15d083a000.statahead_stats= statahead total: 134 statahead wrong: 2 agl total: 128 list_total: 32 fname_total: 99 hit_total: 5099 miss_total: 1615 mdc.lustre-MDT0000-mdc-ffff8a15d083a000.batch_stats= snapshot_time: 1788750871.533167297 secs.nsecs start_time: 1788750781.347245656 secs.nsecs elapsed_time: 90.185921641 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 1 2 2 4: 1 2 5 8: 3 7 12 16: 23 58 71 32: 11 28 100 mdc.lustre-MDT0001-mdc-ffff8a15d083a000.batch_stats= snapshot_time: 1788750871.533367770 secs.nsecs start_time: 1788750781.347324411 secs.nsecs elapsed_time: 90.186043359 secs.nsecs subreqs per batch batches %% cum %% 1: 1 2 2 2: 0 0 2 4: 4 10 12 8: 3 7 20 16: 18 45 65 32: 14 35 100 - open/close 704 (time 1788750884.34 total 10.02 last 70.29) - open/close 1471 (time 1788750894.34 total 20.02 last 76.68) - open/close 2438 (time 1788750904.35 total 30.02 last 96.67) - open/close 3188 (time 1788750914.35 total 40.03 last 74.94) - open/close 4022 (time 1788750924.36 total 50.04 last 83.32) - open/close 4896 (time 1788750934.37 total 60.04 last 87.38) - open/close 5778 (time 1788750944.37 total 70.04 last 88.20) - open/close 6561 (time 1788750954.38 total 80.05 last 78.20) - open/close 7292 (time 1788750964.38 total 90.06 last 73.07) - open/close 8071 (time 1788750974.39 total 100.07 last 77.81) - open/close 9082 (time 1788750984.39 total 110.07 last 101.09) total: 10000 open/close in 119.15 seconds: 83.93 ops/second llite.lustre-ffff8a15d083a000.statahead_batch_max=0 llite.lustre-ffff8a15d083a000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff8a15d083a000.stats=clear mdc.lustre-MDT0001-mdc-ffff8a15d083a000.stats=clear ls -l files in d123ad.sanity: 10001 real 0m59.690s user 0m0.324s sys 0m14.735s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff8a15d083a000.statahead_batch_max=32 llite.lustre-ffff8a15d083a000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff8a15d083a000.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff8a15d083a000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff8a15d083a000.stats=clear mdc.lustre-MDT0001-mdc-ffff8a15d083a000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m40.927s user 0m0.287s sys 0m13.606s batched statahead RPC= hit count=9982 mdc.lustre-MDT0000-mdc-ffff8a15d083a000.batch_stats= snapshot_time: 1788751173.766861085 secs.nsecs start_time: 1788751062.643696211 secs.nsecs elapsed_time: 111.123164874 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 0 0 0 8: 1 0 0 16: 0 0 0 32: 313 99 100 mdc.lustre-MDT0001-mdc-ffff8a15d083a000.batch_stats= snapshot_time: 1788751173.766925290 secs.nsecs start_time: 1788751062.644079680 secs.nsecs elapsed_time: 111.122845610 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 batch: statahead_max=2048 statahead_batch_max=256 mdc.lustre-MDT0000-mdc-ffff8a15d083a000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff8a15d083a000.batch_stats=0 llite.lustre-ffff8a15d083a000.statahead_max=2048 llite.lustre-ffff8a15d083a000.statahead_batch_max=256 kvm mdc.lustre-MDT0000-mdc-ffff8a15d083a000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff8a15d083a000.batch_stats=0 striped dir -i1 -c2 -H crush2 /mnt/lustre/d123ad.sanity total: 100 open/close in 2.26 seconds: 44.34 ops/second llite.lustre-ffff8a15d083a000.statahead_max=0 101 real 0m1.901s user 0m0.007s sys 0m0.312s 'ls -l' 100 files without statahead: 2 sec llite.lustre-ffff8a15d083a000.statahead_max=2048 llite.lustre-ffff8a15d083a000.statahead_max=2048 101 real 0m0.860s user 0m0.007s sys 0m0.175s 'ls -l' 100 files with statahead: 1 sec statahead total: 2 statahead wrong: 0 agl total: 2 list_total: 2 fname_total: 0 hit_total: 10081 miss_total: 19 swrong=0 ewrong=0 - open/close 877 (time 1788751437.53 total 10.00 last 87.67) total: 900 open/close in 10.29 seconds: 87.49 ops/second llite.lustre-ffff8a15d083a000.statahead_max=0 1001 real 0m18.694s user 0m0.040s sys 0m2.922s 'ls -l' 1000 files without statahead: 18 sec llite.lustre-ffff8a15d083a000.statahead_max=2048 llite.lustre-ffff8a15d083a000.statahead_max=2048 1001 real 0m5.953s user 0m0.033s sys 0m1.541s 'ls -l' 1000 files with statahead: 5 sec statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11080 miss_total: 20 swrong=0 ewrong=0 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 20 seconds llite.lustre-ffff8a15d083a000.statahead_stats= statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11080 miss_total: 20 mdc.lustre-MDT0000-mdc-ffff8a15d083a000.batch_stats= snapshot_time: 1788751494.155736313 secs.nsecs start_time: 1788751176.116638068 secs.nsecs elapsed_time: 318.039098245 secs.nsecs subreqs per batch batches %% cum %% 1: 1 14 14 2: 0 0 14 4: 2 28 42 8: 1 14 57 16: 1 14 71 32: 0 0 71 64: 0 0 71 128: 0 0 71 256: 2 28 100 mdc.lustre-MDT0001-mdc-ffff8a15d083a000.batch_stats= snapshot_time: 1788751494.155805972 secs.nsecs start_time: 1788751176.116774098 secs.nsecs elapsed_time: 318.039031874 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 1 11 11 8: 4 44 55 16: 0 0 55 32: 1 11 66 64: 0 0 66 128: 1 11 77 256: 2 22 100 - open/close 809 (time 1788751506.72 total 10.02 last 80.76) - open/close 1580 (time 1788751516.75 total 20.05 last 76.83) - open/close 2472 (time 1788751526.76 total 30.06 last 89.14) - open/close 3297 (time 1788751536.77 total 40.07 last 82.41) - open/close 4242 (time 1788751546.78 total 50.08 last 94.44) - open/close 5032 (time 1788751556.78 total 60.08 last 78.97) - open/close 5763 (time 1788751566.78 total 70.08 last 73.08) - open/close 6529 (time 1788751576.80 total 80.10 last 76.47) - open/close 7309 (time 1788751586.81 total 90.11 last 77.89) - open/close 8210 (time 1788751596.82 total 100.12 last 90.06) - open/close 9046 (time 1788751606.83 total 110.13 last 83.53) - open/close 9940 (time 1788751616.83 total 120.13 last 89.38) total: 10000 open/close in 121.05 seconds: 82.61 ops/second llite.lustre-ffff8a15d083a000.statahead_batch_max=0 llite.lustre-ffff8a15d083a000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff8a15d083a000.stats=clear mdc.lustre-MDT0001-mdc-ffff8a15d083a000.stats=clear ls -l files in d123ad.sanity: 10001 real 0m53.106s user 0m0.288s sys 0m14.801s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff8a15d083a000.statahead_batch_max=256 llite.lustre-ffff8a15d083a000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff8a15d083a000.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff8a15d083a000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff8a15d083a000.stats=clear mdc.lustre-MDT0001-mdc-ffff8a15d083a000.stats=clear ls -l files in d123ad.sanity: 10001 real 0m45.000s user 0m0.323s sys 0m14.649s batched statahead RPC= hit count=9999 mdc.lustre-MDT0000-mdc-ffff8a15d083a000.batch_stats= snapshot_time: 1788751733.809953591 secs.nsecs start_time: 1788751678.776219021 secs.nsecs elapsed_time: 55.033734570 secs.nsecs subreqs per batch batches %% cum %% 1: 137 48 48 2: 35 12 60 4: 35 12 73 8: 11 3 77 16: 8 2 80 32: 13 4 84 64: 7 2 87 128: 2 0 87 256: 34 12 100 mdc.lustre-MDT0001-mdc-ffff8a15d083a000.batch_stats= snapshot_time: 1788751733.810017923 secs.nsecs start_time: 1788751678.776333215 secs.nsecs elapsed_time: 55.033684708 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second seq.cli-lustre-OST0000-super.width=65536 seq.cli-lustre-OST0001-super.width=65536 llite.lustre-ffff8a15d083a000.statahead_batch_max=64 llite.lustre-ffff8a15d083a000.statahead_max=128