== sanity test 123ad: Verify batching statahead works correctly ========================================================== 06:24:33 (1788517473) batch: statahead_max=32 statahead_batch_max=32 mdc.lustre-MDT0000-mdc-ffff8f8205bc1000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff8f8205bc1000.batch_stats=0 llite.lustre-ffff8f8205bc1000.statahead_max=32 llite.lustre-ffff8f8205bc1000.statahead_batch_max=32 seq.cli-lustre-OST0000-super.width=0x1ffffff seq.cli-lustre-OST0001-super.width=0x1ffffff kvm mdc.lustre-MDT0000-mdc-ffff8f8205bc1000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff8f8205bc1000.batch_stats=0 striped dir -i1 -c2 -H crush2 /mnt/lustre/d123ad.sanity total: 100 open/close in 3.05 seconds: 32.78 ops/second llite.lustre-ffff8f8205bc1000.statahead_max=0 101 real 0m2.303s user 0m0.005s sys 0m0.317s 'ls -l' 100 files without statahead: 2 sec llite.lustre-ffff8f8205bc1000.statahead_max=32 llite.lustre-ffff8f8205bc1000.statahead_max=32 101 real 0m1.142s user 0m0.004s sys 0m0.173s 'ls -l' 100 files with statahead: 1 sec statahead total: 117 statahead wrong: 2 agl total: 111 list_total: 31 fname_total: 69 hit_total: 3580 miss_total: 1296 swrong=2 ewrong=2 - open/close 795 (time 1788517504.31 total 10.02 last 79.34) total: 900 open/close in 11.49 seconds: 78.34 ops/second llite.lustre-ffff8f8205bc1000.statahead_max=0 1001 real 0m20.549s user 0m0.032s sys 0m2.927s 'ls -l' 1000 files without statahead: 20 sec llite.lustre-ffff8f8205bc1000.statahead_max=32 llite.lustre-ffff8f8205bc1000.statahead_max=32 1001 real 0m10.434s user 0m0.032s sys 0m1.378s 'ls -l' 1000 files with statahead: 11 sec statahead total: 118 statahead wrong: 2 agl total: 112 list_total: 32 fname_total: 69 hit_total: 4576 miss_total: 1300 swrong=2 ewrong=2 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 22 seconds llite.lustre-ffff8f8205bc1000.statahead_stats= statahead total: 118 statahead wrong: 2 agl total: 112 list_total: 32 fname_total: 69 hit_total: 4576 miss_total: 1300 mdc.lustre-MDT0000-mdc-ffff8f8205bc1000.batch_stats= snapshot_time: 1788517572.197921600 secs.nsecs start_time: 1788517479.915873134 secs.nsecs elapsed_time: 92.282048466 secs.nsecs subreqs per batch batches %% cum %% 1: 3 8 8 2: 2 5 13 4: 1 2 16 8: 0 0 16 16: 26 70 86 32: 5 13 100 mdc.lustre-MDT0001-mdc-ffff8f8205bc1000.batch_stats= snapshot_time: 1788517572.197994317 secs.nsecs start_time: 1788517479.915966911 secs.nsecs elapsed_time: 92.282027406 secs.nsecs subreqs per batch batches %% cum %% 1: 1 2 2 2: 0 0 2 4: 0 0 2 8: 5 12 15 16: 10 25 40 32: 24 60 100 - open/close 791 (time 1788517585.34 total 10.00 last 79.07) - open/close 1715 (time 1788517595.35 total 20.01 last 92.36) - open/close 2429 (time 1788517605.36 total 30.02 last 71.32) - open/close 3169 (time 1788517615.37 total 40.03 last 73.92) - open/close 4000 (time 1788517625.38 total 50.04 last 83.02) - open/close 4824 (time 1788517635.39 total 60.06 last 82.26) - open/close 5575 (time 1788517645.40 total 70.07 last 75.01) - open/close 6395 (time 1788517655.41 total 80.07 last 81.98) - open/close 7202 (time 1788517665.42 total 90.08 last 80.63) - open/close 8158 (time 1788517675.42 total 100.08 last 95.59) - open/close 8843 (time 1788517685.44 total 110.10 last 68.36) - open/close 9731 (time 1788517695.46 total 120.12 last 88.61) total: 10000 open/close in 123.35 seconds: 81.07 ops/second llite.lustre-ffff8f8205bc1000.statahead_batch_max=0 llite.lustre-ffff8f8205bc1000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff8f8205bc1000.stats=clear mdc.lustre-MDT0001-mdc-ffff8f8205bc1000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m0.276s user 0m0.298s sys 0m13.845s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff8f8205bc1000.statahead_batch_max=32 llite.lustre-ffff8f8205bc1000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff8f8205bc1000.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff8f8205bc1000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff8f8205bc1000.stats=clear mdc.lustre-MDT0001-mdc-ffff8f8205bc1000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m43.808s user 0m0.285s sys 0m13.322s batched statahead RPC= hit count=9983 mdc.lustre-MDT0000-mdc-ffff8f8205bc1000.batch_stats= snapshot_time: 1788517879.920091340 secs.nsecs start_time: 1788517767.878052502 secs.nsecs elapsed_time: 112.042038838 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 0 0 0 8: 1 0 0 16: 0 0 0 32: 313 99 100 mdc.lustre-MDT0001-mdc-ffff8f8205bc1000.batch_stats= snapshot_time: 1788517879.920417968 secs.nsecs start_time: 1788517767.878135392 secs.nsecs elapsed_time: 112.042282576 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 batch: statahead_max=2048 statahead_batch_max=256 mdc.lustre-MDT0000-mdc-ffff8f8205bc1000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff8f8205bc1000.batch_stats=0 llite.lustre-ffff8f8205bc1000.statahead_max=2048 llite.lustre-ffff8f8205bc1000.statahead_batch_max=256 kvm mdc.lustre-MDT0000-mdc-ffff8f8205bc1000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff8f8205bc1000.batch_stats=0 striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d123ad.sanity total: 100 open/close in 1.87 seconds: 53.57 ops/second llite.lustre-ffff8f8205bc1000.statahead_max=0 101 real 0m2.070s user 0m0.005s sys 0m0.329s 'ls -l' 100 files without statahead: 2 sec llite.lustre-ffff8f8205bc1000.statahead_max=2048 llite.lustre-ffff8f8205bc1000.statahead_max=2048 101 real 0m1.090s user 0m0.008s sys 0m0.173s 'ls -l' 100 files with statahead: 1 sec statahead total: 2 statahead wrong: 0 agl total: 2 list_total: 2 fname_total: 0 hit_total: 10082 miss_total: 18 swrong=0 ewrong=0 - open/close 828 (time 1788518161.26 total 10.01 last 82.71) total: 900 open/close in 11.06 seconds: 81.39 ops/second llite.lustre-ffff8f8205bc1000.statahead_max=0 1001 real 0m23.350s user 0m0.038s sys 0m2.993s 'ls -l' 1000 files without statahead: 23 sec llite.lustre-ffff8f8205bc1000.statahead_max=2048 llite.lustre-ffff8f8205bc1000.statahead_max=2048 1001 real 0m7.036s user 0m0.026s sys 0m1.478s 'ls -l' 1000 files with statahead: 7 sec statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11081 miss_total: 19 swrong=0 ewrong=0 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 21 seconds llite.lustre-ffff8f8205bc1000.statahead_stats= statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11081 miss_total: 19 mdc.lustre-MDT0000-mdc-ffff8f8205bc1000.batch_stats= snapshot_time: 1788518226.781766218 secs.nsecs start_time: 1788517882.124227170 secs.nsecs elapsed_time: 344.657539048 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 1 12 12 4: 3 37 50 8: 0 0 50 16: 1 12 62 32: 0 0 62 64: 1 12 75 128: 0 0 75 256: 2 25 100 mdc.lustre-MDT0001-mdc-ffff8f8205bc1000.batch_stats= snapshot_time: 1788518226.781850829 secs.nsecs start_time: 1788517882.124576014 secs.nsecs elapsed_time: 344.657274815 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 1 12 12 8: 4 50 62 16: 0 0 62 32: 1 12 75 64: 0 0 75 128: 0 0 75 256: 2 25 100 - open/close 795 (time 1788518239.93 total 10.00 last 79.50) - open/close 1537 (time 1788518249.95 total 20.01 last 74.11) - open/close 2410 (time 1788518259.95 total 30.02 last 87.25) - open/close 3166 (time 1788518269.96 total 40.03 last 75.52) - open/close 4001 (time 1788518279.97 total 50.03 last 83.45) - open/close 4789 (time 1788518289.97 total 60.04 last 78.79) - open/close 5629 (time 1788518299.98 total 70.05 last 83.91) - open/close 6526 (time 1788518309.99 total 80.06 last 89.58) - open/close 7413 (time 1788518320.00 total 90.07 last 88.62) - open/close 8262 (time 1788518330.01 total 100.07 last 84.88) - open/close 9178 (time 1788518340.01 total 110.08 last 91.54) total: 10000 open/close in 118.59 seconds: 84.32 ops/second llite.lustre-ffff8f8205bc1000.statahead_batch_max=0 llite.lustre-ffff8f8205bc1000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff8f8205bc1000.stats=clear mdc.lustre-MDT0001-mdc-ffff8f8205bc1000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m3.524s user 0m0.306s sys 0m14.108s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff8f8205bc1000.statahead_batch_max=256 llite.lustre-ffff8f8205bc1000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff8f8205bc1000.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff8f8205bc1000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff8f8205bc1000.stats=clear mdc.lustre-MDT0001-mdc-ffff8f8205bc1000.stats=clear ls -l files in d123ad.sanity: 10001 real 0m52.535s user 0m0.289s sys 0m15.485s batched statahead RPC= hit count=9999 mdc.lustre-MDT0000-mdc-ffff8f8205bc1000.batch_stats= snapshot_time: 1788518483.994628017 secs.nsecs start_time: 1788518420.477468163 secs.nsecs elapsed_time: 63.517159854 secs.nsecs subreqs per batch batches %% cum %% 1: 94 41 41 2: 30 13 55 4: 20 8 64 8: 14 6 70 16: 11 4 75 32: 10 4 79 64: 9 4 83 128: 2 0 84 256: 35 15 100 mdc.lustre-MDT0001-mdc-ffff8f8205bc1000.batch_stats= snapshot_time: 1788518483.994708714 secs.nsecs start_time: 1788518420.477553723 secs.nsecs elapsed_time: 63.517154991 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 1 seconds: 0.000000 unlinks/second unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second seq.cli-lustre-OST0000-super.width=65536 seq.cli-lustre-OST0001-super.width=65536 llite.lustre-ffff8f8205bc1000.statahead_batch_max=64 llite.lustre-ffff8f8205bc1000.statahead_max=128