== sanity test 123ad: Verify batching statahead works correctly ========================================================== 04:05:46 (1787385946) batch: statahead_max=32 statahead_batch_max=32 mdc.lustre-MDT0000-mdc-ffff9e39c8699800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9e39c8699800.batch_stats=0 llite.lustre-ffff9e39c8699800.statahead_max=32 llite.lustre-ffff9e39c8699800.statahead_batch_max=32 seq.cli-lustre-OST0000-super.width=0x1ffffff seq.cli-lustre-OST0001-super.width=0x1ffffff kvm mdc.lustre-MDT0000-mdc-ffff9e39c8699800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9e39c8699800.batch_stats=0 striped dir -i1 -c2 -H crush2 /mnt/lustre/d123ad.sanity total: 100 open/close in 3.36 seconds: 29.72 ops/second llite.lustre-ffff9e39c8699800.statahead_max=0 101 real 0m2.784s user 0m0.004s sys 0m0.359s 'ls -l' 100 files without statahead: 3 sec llite.lustre-ffff9e39c8699800.statahead_max=32 llite.lustre-ffff9e39c8699800.statahead_max=32 101 real 0m1.145s user 0m0.004s sys 0m0.171s 'ls -l' 100 files with statahead: 1 sec statahead total: 110 statahead wrong: 2 agl total: 104 list_total: 37 fname_total: 71 hit_total: 3724 miss_total: 1498 swrong=2 ewrong=2 - open/close 639 (time 1787385975.46 total 10.01 last 63.84) total: 900 open/close in 13.56 seconds: 66.37 ops/second llite.lustre-ffff9e39c8699800.statahead_max=0 1001 real 0m26.455s user 0m0.037s sys 0m3.562s 'ls -l' 1000 files without statahead: 26 sec llite.lustre-ffff9e39c8699800.statahead_max=32 llite.lustre-ffff9e39c8699800.statahead_max=32 1001 real 0m8.223s user 0m0.026s sys 0m1.685s 'ls -l' 1000 files with statahead: 9 sec statahead total: 111 statahead wrong: 2 agl total: 105 list_total: 38 fname_total: 71 hit_total: 4723 miss_total: 1499 swrong=2 ewrong=2 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 21 seconds llite.lustre-ffff9e39c8699800.statahead_stats= statahead total: 111 statahead wrong: 2 agl total: 105 list_total: 38 fname_total: 71 hit_total: 4723 miss_total: 1499 mdc.lustre-MDT0000-mdc-ffff9e39c8699800.batch_stats= snapshot_time: 1787386047.818296658 secs.nsecs start_time: 1787385950.906100828 secs.nsecs elapsed_time: 96.912195830 secs.nsecs subreqs per batch batches %% cum %% 1: 1 2 2 2: 0 0 2 4: 2 5 8 8: 4 11 20 16: 21 60 80 32: 7 20 100 mdc.lustre-MDT0001-mdc-ffff9e39c8699800.batch_stats= snapshot_time: 1787386047.818366317 secs.nsecs start_time: 1787385950.906236977 secs.nsecs elapsed_time: 96.912129340 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 2 5 5 8: 2 5 10 16: 10 25 35 32: 25 64 100 - open/close 734 (time 1787386060.53 total 10.00 last 73.37) - open/close 1466 (time 1787386070.54 total 20.01 last 73.12) - open/close 2132 (time 1787386080.56 total 30.03 last 66.49) - open/close 2825 (time 1787386090.56 total 40.03 last 69.29) - open/close 3567 (time 1787386100.57 total 50.04 last 74.13) - open/close 4266 (time 1787386110.57 total 60.04 last 69.88) - open/close 4985 (time 1787386120.58 total 70.05 last 71.84) - open/close 5714 (time 1787386130.59 total 80.07 last 72.78) - open/close 6469 (time 1787386140.61 total 90.09 last 75.37) - open/close 7149 (time 1787386150.62 total 100.09 last 67.97) - open/close 7892 (time 1787386160.62 total 110.09 last 74.30) - open/close 8586 (time 1787386170.64 total 120.11 last 69.27) - open/close 9261 (time 1787386180.64 total 130.12 last 67.44) - open/close 9919 (time 1787386190.65 total 140.12 last 65.78) total: 10000 open/close in 141.08 seconds: 70.88 ops/second llite.lustre-ffff9e39c8699800.statahead_batch_max=0 llite.lustre-ffff9e39c8699800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9e39c8699800.stats=clear mdc.lustre-MDT0001-mdc-ffff9e39c8699800.stats=clear ls -l files in d123ad.sanity: 10001 real 1m5.363s user 0m0.349s sys 0m18.735s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff9e39c8699800.statahead_batch_max=32 llite.lustre-ffff9e39c8699800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9e39c8699800.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff9e39c8699800.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff9e39c8699800.stats=clear mdc.lustre-MDT0001-mdc-ffff9e39c8699800.stats=clear ls -l files in d123ad.sanity: 10001 real 1m51.203s user 0m0.341s sys 0m17.184s batched statahead RPC= hit count=9989 mdc.lustre-MDT0000-mdc-ffff9e39c8699800.batch_stats= snapshot_time: 1787386395.862510068 secs.nsecs start_time: 1787386269.018918178 secs.nsecs elapsed_time: 126.843591890 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 0 0 0 8: 1 0 0 16: 1 0 0 32: 313 99 100 mdc.lustre-MDT0001-mdc-ffff9e39c8699800.batch_stats= snapshot_time: 1787386395.862587383 secs.nsecs start_time: 1787386269.018994255 secs.nsecs elapsed_time: 126.843593128 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 batch: statahead_max=2048 statahead_batch_max=256 mdc.lustre-MDT0000-mdc-ffff9e39c8699800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9e39c8699800.batch_stats=0 llite.lustre-ffff9e39c8699800.statahead_max=2048 llite.lustre-ffff9e39c8699800.statahead_batch_max=256 kvm mdc.lustre-MDT0000-mdc-ffff9e39c8699800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9e39c8699800.batch_stats=0 striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d123ad.sanity total: 100 open/close in 3.29 seconds: 30.39 ops/second llite.lustre-ffff9e39c8699800.statahead_max=0 101 real 0m2.810s user 0m0.005s sys 0m0.348s 'ls -l' 100 files without statahead: 3 sec llite.lustre-ffff9e39c8699800.statahead_max=2048 llite.lustre-ffff9e39c8699800.statahead_max=2048 101 real 0m0.964s user 0m0.005s sys 0m0.222s 'ls -l' 100 files with statahead: 0 sec statahead total: 2 statahead wrong: 0 agl total: 2 list_total: 2 fname_total: 0 hit_total: 10088 miss_total: 12 swrong=0 ewrong=0 - open/close 697 (time 1787386711.43 total 10.00 last 69.70) total: 900 open/close in 13.24 seconds: 67.98 ops/second llite.lustre-ffff9e39c8699800.statahead_max=0 1001 real 0m25.193s user 0m0.041s sys 0m3.368s 'ls -l' 1000 files without statahead: 25 sec llite.lustre-ffff9e39c8699800.statahead_max=2048 llite.lustre-ffff9e39c8699800.statahead_max=2048 1001 real 0m9.136s user 0m0.024s sys 0m1.600s 'ls -l' 1000 files with statahead: 9 sec statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11087 miss_total: 13 swrong=0 ewrong=0 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 22 seconds llite.lustre-ffff9e39c8699800.statahead_stats= statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11087 miss_total: 13 mdc.lustre-MDT0000-mdc-ffff9e39c8699800.batch_stats= snapshot_time: 1787386785.293867882 secs.nsecs start_time: 1787386398.047317833 secs.nsecs elapsed_time: 387.246550049 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 1 11 11 4: 2 22 33 8: 2 22 55 16: 1 11 66 32: 0 0 66 64: 1 11 77 128: 0 0 77 256: 2 22 100 mdc.lustre-MDT0001-mdc-ffff9e39c8699800.batch_stats= snapshot_time: 1787386785.293943712 secs.nsecs start_time: 1787386398.047455939 secs.nsecs elapsed_time: 387.246487773 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 2 22 22 8: 3 33 55 16: 1 11 66 32: 1 11 77 64: 0 0 77 128: 0 0 77 256: 2 22 100 - open/close 771 (time 1787386798.08 total 10.01 last 77.04) - open/close 1439 (time 1787386808.09 total 20.02 last 66.71) - open/close 2127 (time 1787386818.11 total 30.04 last 68.66) - open/close 2799 (time 1787386828.12 total 40.05 last 67.17) - open/close 3536 (time 1787386838.12 total 50.05 last 73.67) - open/close 4292 (time 1787386848.14 total 60.06 last 75.49) - open/close 4977 (time 1787386858.15 total 70.08 last 68.40) - open/close 5726 (time 1787386868.16 total 80.09 last 74.84) - open/close 6526 (time 1787386878.18 total 90.11 last 79.81) - open/close 7238 (time 1787386888.18 total 100.11 last 71.20) - open/close 7999 (time 1787386898.19 total 110.12 last 76.05) - open/close 8785 (time 1787386908.20 total 120.13 last 78.53) - open/close 9494 (time 1787386918.20 total 130.13 last 70.86) total: 10000 open/close in 136.62 seconds: 73.20 ops/second llite.lustre-ffff9e39c8699800.statahead_batch_max=0 llite.lustre-ffff9e39c8699800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9e39c8699800.stats=clear mdc.lustre-MDT0001-mdc-ffff9e39c8699800.stats=clear ls -l files in d123ad.sanity: 10001 real 1m9.951s user 0m0.329s sys 0m18.547s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff9e39c8699800.statahead_batch_max=256 llite.lustre-ffff9e39c8699800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9e39c8699800.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff9e39c8699800.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff9e39c8699800.stats=clear mdc.lustre-MDT0001-mdc-ffff9e39c8699800.stats=clear ls -l files in d123ad.sanity: 10001 real 1m6.416s user 0m0.324s sys 0m17.824s batched statahead RPC= hit count=9999 mdc.lustre-MDT0000-mdc-ffff9e39c8699800.batch_stats= snapshot_time: 1787387088.548208023 secs.nsecs start_time: 1787387005.993336421 secs.nsecs elapsed_time: 82.554871602 secs.nsecs subreqs per batch batches %% cum %% 1: 217 52 52 2: 50 12 64 4: 50 12 76 8: 29 6 83 16: 15 3 86 32: 15 3 90 64: 4 0 91 128: 2 0 92 256: 33 7 100 mdc.lustre-MDT0001-mdc-ffff9e39c8699800.batch_stats= snapshot_time: 1787387088.548270936 secs.nsecs start_time: 1787387005.993427368 secs.nsecs elapsed_time: 82.554843568 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second seq.cli-lustre-OST0000-super.width=65536 seq.cli-lustre-OST0001-super.width=65536 llite.lustre-ffff9e39c8699800.statahead_batch_max=64 llite.lustre-ffff9e39c8699800.statahead_max=128