== sanity test 123ad: Verify batching statahead works correctly ========================================================== 03:20:14 (1785396014) batch: statahead_max=32 statahead_batch_max=32 mdc.lustre-MDT0000-mdc-ffff9e3b61727000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9e3b61727000.batch_stats=0 llite.lustre-ffff9e3b61727000.statahead_max=32 llite.lustre-ffff9e3b61727000.statahead_batch_max=32 seq.cli-lustre-OST0000-super.width=0x1ffffff seq.cli-lustre-OST0001-super.width=0x1ffffff kvm mdc.lustre-MDT0000-mdc-ffff9e3b61727000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9e3b61727000.batch_stats=0 striped dir -i1 -c2 -H crush /mnt/lustre/d123ad.sanity total: 100 open/close in 1.95 seconds: 51.36 ops/second llite.lustre-ffff9e3b61727000.statahead_max=0 101 real 0m2.241s user 0m0.006s sys 0m0.327s 'ls -l' 100 files without statahead: 2 sec llite.lustre-ffff9e3b61727000.statahead_max=32 llite.lustre-ffff9e3b61727000.statahead_max=32 101 real 0m1.612s user 0m0.003s sys 0m0.205s 'ls -l' 100 files with statahead: 2 sec statahead total: 174 statahead wrong: 2 agl total: 166 list_total: 32 fname_total: 127 hit_total: 4120 miss_total: 1568 swrong=2 ewrong=2 - open/close 832 (time 1785396043.62 total 10.00 last 83.19) total: 900 open/close in 10.88 seconds: 82.71 ops/second llite.lustre-ffff9e3b61727000.statahead_max=0 1001 real 0m39.187s user 0m0.034s sys 0m3.140s 'ls -l' 1000 files without statahead: 39 sec llite.lustre-ffff9e3b61727000.statahead_max=32 llite.lustre-ffff9e3b61727000.statahead_max=32 1001 real 0m14.392s user 0m0.023s sys 0m1.368s 'ls -l' 1000 files with statahead: 14 sec statahead total: 175 statahead wrong: 2 agl total: 167 list_total: 33 fname_total: 127 hit_total: 5114 miss_total: 1574 swrong=2 ewrong=2 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 28 seconds llite.lustre-ffff9e3b61727000.statahead_stats= statahead total: 175 statahead wrong: 2 agl total: 167 list_total: 33 fname_total: 127 hit_total: 5114 miss_total: 1574 mdc.lustre-MDT0000-mdc-ffff9e3b61727000.batch_stats= snapshot_time: 1785396144.275767949 secs.nsecs start_time: 1785396020.581484959 secs.nsecs elapsed_time: 123.694282990 secs.nsecs subreqs per batch batches %% cum %% 1: 4 10 10 2: 1 2 13 4: 1 2 15 8: 1 2 18 16: 25 65 84 32: 6 15 100 mdc.lustre-MDT0001-mdc-ffff9e3b61727000.batch_stats= snapshot_time: 1785396144.275843184 secs.nsecs start_time: 1785396020.581684395 secs.nsecs elapsed_time: 123.694158789 secs.nsecs subreqs per batch batches %% cum %% 1: 1 2 2 2: 0 0 2 4: 2 5 7 8: 3 7 15 16: 9 22 37 32: 25 62 100 - open/close 509 (time 1785396157.31 total 10.00 last 50.88) - open/close 1150 (time 1785396167.32 total 20.02 last 64.00) - open/close 1624 (time 1785396177.33 total 30.02 last 47.38) - open/close 2041 (time 1785396187.33 total 40.03 last 41.67) - open/close 2505 (time 1785396197.34 total 50.04 last 46.36) - open/close 3288 (time 1785396207.35 total 60.05 last 78.21) - open/close 3898 (time 1785396217.37 total 70.06 last 60.91) - open/close 4426 (time 1785396227.39 total 80.09 last 52.69) - open/close 5033 (time 1785396237.40 total 90.10 last 60.63) - open/close 5576 (time 1785396247.40 total 100.10 last 54.29) - open/close 6326 (time 1785396257.40 total 110.10 last 74.99) - open/close 7408 (time 1785396267.41 total 120.11 last 108.10) - open/close 8029 (time 1785396277.42 total 130.11 last 62.08) - open/close 8492 (time 1785396287.42 total 140.11 last 46.30) - open/close 9046 (time 1785396297.43 total 150.12 last 55.34) - open/close 9683 (time 1785396307.44 total 160.13 last 63.64) total: 10000 open/close in 163.91 seconds: 61.01 ops/second llite.lustre-ffff9e3b61727000.statahead_batch_max=0 llite.lustre-ffff9e3b61727000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9e3b61727000.stats=clear mdc.lustre-MDT0001-mdc-ffff9e3b61727000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m24.664s user 0m0.279s sys 0m15.783s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff9e3b61727000.statahead_batch_max=32 llite.lustre-ffff9e3b61727000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9e3b61727000.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff9e3b61727000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff9e3b61727000.stats=clear mdc.lustre-MDT0001-mdc-ffff9e3b61727000.stats=clear ls -l files in d123ad.sanity: 10001 real 2m37.368s user 0m0.299s sys 0m15.213s batched statahead RPC= hit count=9976 mdc.lustre-MDT0000-mdc-ffff9e3b61727000.batch_stats= snapshot_time: 1785396595.046708602 secs.nsecs start_time: 1785396411.186253387 secs.nsecs elapsed_time: 183.860455215 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 1 0 0 8: 1 0 0 16: 1 0 0 32: 313 99 100 mdc.lustre-MDT0001-mdc-ffff9e3b61727000.batch_stats= snapshot_time: 1785396595.046791905 secs.nsecs start_time: 1785396411.186327829 secs.nsecs elapsed_time: 183.860464076 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 batch: statahead_max=2048 statahead_batch_max=256 mdc.lustre-MDT0000-mdc-ffff9e3b61727000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9e3b61727000.batch_stats=0 llite.lustre-ffff9e3b61727000.statahead_max=2048 llite.lustre-ffff9e3b61727000.statahead_batch_max=256 kvm mdc.lustre-MDT0000-mdc-ffff9e3b61727000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9e3b61727000.batch_stats=0 striped dir -i1 -c2 -H crush2 /mnt/lustre/d123ad.sanity total: 100 open/close in 2.24 seconds: 44.60 ops/second llite.lustre-ffff9e3b61727000.statahead_max=0 101 real 0m1.615s user 0m0.004s sys 0m0.321s 'ls -l' 100 files without statahead: 2 sec llite.lustre-ffff9e3b61727000.statahead_max=2048 llite.lustre-ffff9e3b61727000.statahead_max=2048 101 real 0m1.818s user 0m0.005s sys 0m0.198s 'ls -l' 100 files with statahead: 2 sec statahead total: 2 statahead wrong: 0 agl total: 2 list_total: 2 fname_total: 0 hit_total: 10075 miss_total: 25 swrong=0 ewrong=0 - open/close 583 (time 1785396968.56 total 10.01 last 58.21) total: 900 open/close in 15.45 seconds: 58.27 ops/second llite.lustre-ffff9e3b61727000.statahead_max=0 1001 real 0m26.414s user 0m0.029s sys 0m3.061s 'ls -l' 1000 files without statahead: 26 sec llite.lustre-ffff9e3b61727000.statahead_max=2048 llite.lustre-ffff9e3b61727000.statahead_max=2048 1001 real 0m17.779s user 0m0.031s sys 0m1.581s 'ls -l' 1000 files with statahead: 18 sec statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11074 miss_total: 26 swrong=0 ewrong=0 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 27 seconds llite.lustre-ffff9e3b61727000.statahead_stats= statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11074 miss_total: 26 mdc.lustre-MDT0000-mdc-ffff9e3b61727000.batch_stats= snapshot_time: 1785397065.067874344 secs.nsecs start_time: 1785396598.514346759 secs.nsecs elapsed_time: 466.553527585 secs.nsecs subreqs per batch batches %% cum %% 1: 3 42 42 2: 0 0 42 4: 2 28 71 8: 0 0 71 16: 0 0 71 32: 0 0 71 64: 0 0 71 128: 0 0 71 256: 2 28 100 mdc.lustre-MDT0001-mdc-ffff9e3b61727000.batch_stats= snapshot_time: 1785397065.067941525 secs.nsecs start_time: 1785396598.514488492 secs.nsecs elapsed_time: 466.553453033 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 1 11 11 8: 2 22 33 16: 2 22 55 32: 0 0 55 64: 2 22 77 128: 0 0 77 256: 2 22 100 - open/close 466 (time 1785397078.99 total 10.01 last 46.53) - open/close 1094 (time 1785397089.00 total 20.02 last 62.77) - open/close 1508 (time 1785397099.02 total 30.05 last 41.29) - open/close 2396 (time 1785397109.03 total 40.05 last 88.75) - open/close 3193 (time 1785397119.03 total 50.05 last 79.70) - open/close 4060 (time 1785397129.03 total 60.06 last 86.67) - open/close 5023 (time 1785397139.03 total 70.06 last 96.28) - open/close 5806 (time 1785397149.05 total 80.07 last 78.17) - open/close 6389 (time 1785397159.07 total 90.09 last 58.22) - open/close 7376 (time 1785397169.07 total 100.09 last 98.69) - open/close 8468 (time 1785397179.07 total 110.10 last 109.13) - open/close 9362 (time 1785397189.07 total 120.10 last 89.39) - open/close 9901 (time 1785397199.07 total 130.10 last 53.90) total: 10000 open/close in 131.64 seconds: 75.97 ops/second llite.lustre-ffff9e3b61727000.statahead_batch_max=0 llite.lustre-ffff9e3b61727000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9e3b61727000.stats=clear mdc.lustre-MDT0001-mdc-ffff9e3b61727000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m25.559s user 0m0.296s sys 0m15.698s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff9e3b61727000.statahead_batch_max=256 llite.lustre-ffff9e3b61727000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9e3b61727000.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff9e3b61727000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff9e3b61727000.stats=clear mdc.lustre-MDT0001-mdc-ffff9e3b61727000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m18.808s user 0m0.291s sys 0m15.184s batched statahead RPC= hit count=9999 mdc.lustre-MDT0000-mdc-ffff9e3b61727000.batch_stats= snapshot_time: 1785397393.482983906 secs.nsecs start_time: 1785397298.990762213 secs.nsecs elapsed_time: 94.492221693 secs.nsecs subreqs per batch batches %% cum %% 1: 165 48 48 2: 60 17 66 4: 37 10 77 8: 11 3 80 16: 6 1 82 32: 23 6 89 64: 2 0 89 128: 1 0 89 256: 34 10 100 mdc.lustre-MDT0001-mdc-ffff9e3b61727000.batch_stats= snapshot_time: 1785397393.483070936 secs.nsecs start_time: 1785397298.990900608 secs.nsecs elapsed_time: 94.492170328 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second seq.cli-lustre-OST0000-super.width=65536 seq.cli-lustre-OST0001-super.width=65536 llite.lustre-ffff9e3b61727000.statahead_batch_max=64 llite.lustre-ffff9e3b61727000.statahead_max=128