== sanity test 123ad: Verify batching statahead works correctly ========================================================== 23:54:45 (1787889285) batch: statahead_max=32 statahead_batch_max=32 mdc.lustre-MDT0000-mdc-ffff9dc611a49800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9dc611a49800.batch_stats=0 llite.lustre-ffff9dc611a49800.statahead_max=32 llite.lustre-ffff9dc611a49800.statahead_batch_max=32 seq.cli-lustre-OST0000-super.width=0x1ffffff seq.cli-lustre-OST0001-super.width=0x1ffffff kvm mdc.lustre-MDT0000-mdc-ffff9dc611a49800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9dc611a49800.batch_stats=0 striped dir -i1 -c2 -H crush2 /mnt/lustre/d123ad.sanity total: 100 open/close in 2.14 seconds: 46.83 ops/second llite.lustre-ffff9dc611a49800.statahead_max=0 101 real 0m1.936s user 0m0.005s sys 0m0.304s 'ls -l' 100 files without statahead: 2 sec llite.lustre-ffff9dc611a49800.statahead_max=32 llite.lustre-ffff9dc611a49800.statahead_max=32 101 real 0m0.979s user 0m0.003s sys 0m0.177s 'ls -l' 100 files with statahead: 1 sec statahead total: 143 statahead wrong: 2 agl total: 137 list_total: 30 fname_total: 111 hit_total: 3924 miss_total: 1095 swrong=2 ewrong=2 - open/close 801 (time 1787889311.08 total 10.01 last 80.02) total: 900 open/close in 11.18 seconds: 80.50 ops/second llite.lustre-ffff9dc611a49800.statahead_max=0 1001 real 0m21.814s user 0m0.032s sys 0m2.819s 'ls -l' 1000 files without statahead: 22 sec llite.lustre-ffff9dc611a49800.statahead_max=32 llite.lustre-ffff9dc611a49800.statahead_max=32 1001 real 0m9.801s user 0m0.030s sys 0m1.384s 'ls -l' 1000 files with statahead: 10 sec statahead total: 144 statahead wrong: 2 agl total: 138 list_total: 31 fname_total: 111 hit_total: 4921 miss_total: 1098 swrong=2 ewrong=2 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 23 seconds llite.lustre-ffff9dc611a49800.statahead_stats= statahead total: 144 statahead wrong: 2 agl total: 138 list_total: 31 fname_total: 111 hit_total: 4921 miss_total: 1098 mdc.lustre-MDT0000-mdc-ffff9dc611a49800.batch_stats= snapshot_time: 1787889380.021184031 secs.nsecs start_time: 1787889289.886262221 secs.nsecs elapsed_time: 90.134921810 secs.nsecs subreqs per batch batches %% cum %% 1: 3 7 7 2: 2 5 13 4: 2 5 18 8: 1 2 21 16: 25 65 86 32: 5 13 100 mdc.lustre-MDT0001-mdc-ffff9dc611a49800.batch_stats= snapshot_time: 1787889380.021251142 secs.nsecs start_time: 1787889289.886405654 secs.nsecs elapsed_time: 90.134845488 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 1 2 2 8: 4 10 12 16: 10 25 38 32: 24 61 100 - open/close 730 (time 1787889394.35 total 10.00 last 73.00) - open/close 1443 (time 1787889404.35 total 20.00 last 71.28) - open/close 2227 (time 1787889414.35 total 30.01 last 78.37) - open/close 3042 (time 1787889424.36 total 40.01 last 81.46) - open/close 3932 (time 1787889434.36 total 50.01 last 89.00) - open/close 4759 (time 1787889444.37 total 60.02 last 82.64) - open/close 5565 (time 1787889454.37 total 70.02 last 80.55) - open/close 6169 (time 1787889464.38 total 80.03 last 60.35) - open/close 6975 (time 1787889474.38 total 90.04 last 80.58) - open/close 7848 (time 1787889484.39 total 100.04 last 87.23) - open/close 8769 (time 1787889494.40 total 110.05 last 92.03) - open/close 9591 (time 1787889504.40 total 120.05 last 82.17) total: 10000 open/close in 124.17 seconds: 80.53 ops/second llite.lustre-ffff9dc611a49800.statahead_batch_max=0 llite.lustre-ffff9dc611a49800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9dc611a49800.stats=clear mdc.lustre-MDT0001-mdc-ffff9dc611a49800.stats=clear ls -l files in d123ad.sanity: 10001 real 1m1.905s user 0m0.313s sys 0m15.618s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff9dc611a49800.statahead_batch_max=32 llite.lustre-ffff9dc611a49800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9dc611a49800.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff9dc611a49800.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff9dc611a49800.stats=clear mdc.lustre-MDT0001-mdc-ffff9dc611a49800.stats=clear ls -l files in d123ad.sanity: 10001 real 1m35.835s user 0m0.270s sys 0m13.714s batched statahead RPC= hit count=9977 mdc.lustre-MDT0000-mdc-ffff9dc611a49800.batch_stats= snapshot_time: 1787889685.601074357 secs.nsecs start_time: 1787889577.620259070 secs.nsecs elapsed_time: 107.980815287 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 1 0 0 4: 0 0 0 8: 1 0 0 16: 1 0 0 32: 313 99 100 mdc.lustre-MDT0001-mdc-ffff9dc611a49800.batch_stats= snapshot_time: 1787889685.601141037 secs.nsecs start_time: 1787889577.620335110 secs.nsecs elapsed_time: 107.980805927 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 batch: statahead_max=2048 statahead_batch_max=256 mdc.lustre-MDT0000-mdc-ffff9dc611a49800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9dc611a49800.batch_stats=0 llite.lustre-ffff9dc611a49800.statahead_max=2048 llite.lustre-ffff9dc611a49800.statahead_batch_max=256 kvm mdc.lustre-MDT0000-mdc-ffff9dc611a49800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9dc611a49800.batch_stats=0 striped dir -i1 -c2 -H crush /mnt/lustre/d123ad.sanity total: 100 open/close in 2.28 seconds: 43.78 ops/second llite.lustre-ffff9dc611a49800.statahead_max=0 101 real 0m2.164s user 0m0.003s sys 0m0.304s 'ls -l' 100 files without statahead: 2 sec llite.lustre-ffff9dc611a49800.statahead_max=2048 llite.lustre-ffff9dc611a49800.statahead_max=2048 101 real 0m1.001s user 0m0.004s sys 0m0.169s 'ls -l' 100 files with statahead: 1 sec statahead total: 2 statahead wrong: 0 agl total: 2 list_total: 2 fname_total: 0 hit_total: 10076 miss_total: 24 swrong=0 ewrong=0 total: 900 open/close in 9.99 seconds: 90.05 ops/second llite.lustre-ffff9dc611a49800.statahead_max=0 1001 real 0m19.688s user 0m0.024s sys 0m2.795s 'ls -l' 1000 files without statahead: 19 sec llite.lustre-ffff9dc611a49800.statahead_max=2048 llite.lustre-ffff9dc611a49800.statahead_max=2048 1001 real 0m5.904s user 0m0.024s sys 0m1.392s 'ls -l' 1000 files with statahead: 6 sec statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11075 miss_total: 25 swrong=0 ewrong=0 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 18 seconds llite.lustre-ffff9dc611a49800.statahead_stats= statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11075 miss_total: 25 mdc.lustre-MDT0000-mdc-ffff9dc611a49800.batch_stats= snapshot_time: 1787890022.447690227 secs.nsecs start_time: 1787889689.451382943 secs.nsecs elapsed_time: 332.996307284 secs.nsecs subreqs per batch batches %% cum %% 1: 1 16 16 2: 0 0 16 4: 3 50 66 8: 0 0 66 16: 0 0 66 32: 0 0 66 64: 0 0 66 128: 0 0 66 256: 2 33 100 mdc.lustre-MDT0001-mdc-ffff9dc611a49800.batch_stats= snapshot_time: 1787890022.447776706 secs.nsecs start_time: 1787889689.451524253 secs.nsecs elapsed_time: 332.996252453 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 1 12 12 8: 2 25 37 16: 1 12 50 32: 0 0 50 64: 1 12 62 128: 1 12 75 256: 2 25 100 - open/close 721 (time 1787890035.27 total 10.02 last 71.94) - open/close 1459 (time 1787890045.28 total 20.03 last 73.73) - open/close 2268 (time 1787890055.28 total 30.03 last 80.89) - open/close 3234 (time 1787890065.29 total 40.04 last 96.50) - open/close 4171 (time 1787890075.29 total 50.05 last 93.66) - open/close 5093 (time 1787890085.30 total 60.05 last 92.14) - open/close 5995 (time 1787890095.30 total 70.06 last 90.15) - open/close 6948 (time 1787890105.30 total 80.06 last 95.29) - open/close 7862 (time 1787890115.32 total 90.07 last 91.27) - open/close 8786 (time 1787890125.34 total 100.09 last 92.24) - open/close 9744 (time 1787890135.34 total 110.10 last 95.76) total: 10000 open/close in 112.82 seconds: 88.64 ops/second llite.lustre-ffff9dc611a49800.statahead_batch_max=0 llite.lustre-ffff9dc611a49800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9dc611a49800.stats=clear mdc.lustre-MDT0001-mdc-ffff9dc611a49800.stats=clear ls -l files in d123ad.sanity: 10001 real 0m44.372s user 0m0.318s sys 0m17.799s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff9dc611a49800.statahead_batch_max=256 llite.lustre-ffff9dc611a49800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9dc611a49800.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff9dc611a49800.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff9dc611a49800.stats=clear mdc.lustre-MDT0001-mdc-ffff9dc611a49800.stats=clear ls -l files in d123ad.sanity: 10001 real 0m39.681s user 0m0.236s sys 0m16.612s batched statahead RPC= hit count=9999 mdc.lustre-MDT0000-mdc-ffff9dc611a49800.batch_stats= snapshot_time: 1787890236.077592723 secs.nsecs start_time: 1787890188.063605726 secs.nsecs elapsed_time: 48.013986997 secs.nsecs subreqs per batch batches %% cum %% 1: 141 51 51 2: 27 9 61 4: 28 10 72 8: 12 4 76 16: 9 3 79 32: 13 4 84 64: 6 2 86 128: 2 0 87 256: 34 12 100 mdc.lustre-MDT0001-mdc-ffff9dc611a49800.batch_stats= snapshot_time: 1787890236.077686281 secs.nsecs start_time: 1787890188.063856703 secs.nsecs elapsed_time: 48.013829578 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second seq.cli-lustre-OST0000-super.width=65536 seq.cli-lustre-OST0001-super.width=65536 llite.lustre-ffff9dc611a49800.statahead_batch_max=64 llite.lustre-ffff9dc611a49800.statahead_max=128