== sanity test 123ad: Verify batching statahead works correctly ========================================================== 11:23:30 (1789053810) batch: statahead_max=32 statahead_batch_max=32 mdc.lustre-MDT0000-mdc-ffff9265cfcbd000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9265cfcbd000.batch_stats=0 llite.lustre-ffff9265cfcbd000.statahead_max=32 llite.lustre-ffff9265cfcbd000.statahead_batch_max=32 seq.cli-lustre-OST0000-super.width=0x1ffffff seq.cli-lustre-OST0001-super.width=0x1ffffff kvm mdc.lustre-MDT0000-mdc-ffff9265cfcbd000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9265cfcbd000.batch_stats=0 striped dir -i1 -c2 -H crush /mnt/lustre/d123ad.sanity total: 100 open/close in 2.78 seconds: 36.01 ops/second llite.lustre-ffff9265cfcbd000.statahead_max=0 101 real 0m2.277s user 0m0.002s sys 0m0.306s 'ls -l' 100 files without statahead: 2 sec llite.lustre-ffff9265cfcbd000.statahead_max=32 llite.lustre-ffff9265cfcbd000.statahead_max=32 101 real 0m0.980s user 0m0.005s sys 0m0.175s 'ls -l' 100 files with statahead: 1 sec statahead total: 124 statahead wrong: 2 agl total: 116 list_total: 31 fname_total: 81 hit_total: 4008 miss_total: 1753 swrong=2 ewrong=2 - open/close 788 (time 1789053837.23 total 10.00 last 78.79) total: 900 open/close in 11.16 seconds: 80.65 ops/second llite.lustre-ffff9265cfcbd000.statahead_max=0 1001 real 0m22.199s user 0m0.033s sys 0m2.850s 'ls -l' 1000 files without statahead: 22 sec llite.lustre-ffff9265cfcbd000.statahead_max=32 llite.lustre-ffff9265cfcbd000.statahead_max=32 1001 real 0m7.122s user 0m0.024s sys 0m1.302s 'ls -l' 1000 files with statahead: 7 sec statahead total: 125 statahead wrong: 2 agl total: 117 list_total: 32 fname_total: 81 hit_total: 5006 miss_total: 1755 swrong=2 ewrong=2 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 22 seconds llite.lustre-ffff9265cfcbd000.statahead_stats= statahead total: 125 statahead wrong: 2 agl total: 117 list_total: 32 fname_total: 81 hit_total: 5006 miss_total: 1755 mdc.lustre-MDT0000-mdc-ffff9265cfcbd000.batch_stats= snapshot_time: 1789053901.099502177 secs.nsecs start_time: 1789053814.895045801 secs.nsecs elapsed_time: 86.204456376 secs.nsecs subreqs per batch batches %% cum %% 1: 2 5 5 2: 0 0 5 4: 2 5 11 8: 2 5 16 16: 23 63 80 32: 7 19 100 mdc.lustre-MDT0001-mdc-ffff9265cfcbd000.batch_stats= snapshot_time: 1789053901.099566549 secs.nsecs start_time: 1789053814.895260414 secs.nsecs elapsed_time: 86.204306135 secs.nsecs subreqs per batch batches %% cum %% 1: 1 2 2 2: 0 0 2 4: 1 2 5 8: 3 7 12 16: 13 32 45 32: 22 55 100 - open/close 698 (time 1789053914.22 total 10.00 last 69.79) - open/close 1422 (time 1789053924.22 total 20.00 last 72.37) - open/close 2266 (time 1789053934.23 total 30.02 last 84.31) - open/close 3080 (time 1789053944.24 total 40.02 last 81.36) - open/close 4001 (time 1789053954.24 total 50.02 last 92.06) - open/close 4831 (time 1789053964.26 total 60.04 last 82.85) - open/close 5554 (time 1789053974.26 total 70.05 last 72.27) - open/close 6326 (time 1789053984.27 total 80.05 last 77.18) - open/close 7140 (time 1789053994.27 total 90.06 last 81.35) - open/close 7941 (time 1789054004.28 total 100.06 last 80.07) - open/close 8741 (time 1789054014.28 total 110.07 last 79.93) - open/close 9355 (time 1789054024.30 total 120.09 last 61.28) total: 10000 open/close in 129.75 seconds: 77.07 ops/second llite.lustre-ffff9265cfcbd000.statahead_batch_max=0 llite.lustre-ffff9265cfcbd000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9265cfcbd000.stats=clear mdc.lustre-MDT0001-mdc-ffff9265cfcbd000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m9.048s user 0m0.314s sys 0m14.616s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff9265cfcbd000.statahead_batch_max=32 llite.lustre-ffff9265cfcbd000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9265cfcbd000.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff9265cfcbd000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff9265cfcbd000.stats=clear mdc.lustre-MDT0001-mdc-ffff9265cfcbd000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m40.726s user 0m0.277s sys 0m13.675s batched statahead RPC= hit count=9982 mdc.lustre-MDT0000-mdc-ffff9265cfcbd000.batch_stats= snapshot_time: 1789054224.688096229 secs.nsecs start_time: 1789054110.851621540 secs.nsecs elapsed_time: 113.836474689 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 0 0 0 8: 1 0 0 16: 1 0 0 32: 313 99 100 mdc.lustre-MDT0001-mdc-ffff9265cfcbd000.batch_stats= snapshot_time: 1789054224.688156769 secs.nsecs start_time: 1789054110.851756998 secs.nsecs elapsed_time: 113.836399771 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 batch: statahead_max=2048 statahead_batch_max=256 mdc.lustre-MDT0000-mdc-ffff9265cfcbd000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9265cfcbd000.batch_stats=0 llite.lustre-ffff9265cfcbd000.statahead_max=2048 llite.lustre-ffff9265cfcbd000.statahead_batch_max=256 kvm mdc.lustre-MDT0000-mdc-ffff9265cfcbd000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9265cfcbd000.batch_stats=0 striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d123ad.sanity total: 100 open/close in 2.55 seconds: 39.15 ops/second llite.lustre-ffff9265cfcbd000.statahead_max=0 101 real 0m2.539s user 0m0.006s sys 0m0.306s 'ls -l' 100 files without statahead: 3 sec llite.lustre-ffff9265cfcbd000.statahead_max=2048 llite.lustre-ffff9265cfcbd000.statahead_max=2048 101 real 0m1.619s user 0m0.003s sys 0m0.147s 'ls -l' 100 files with statahead: 1 sec statahead total: 2 statahead wrong: 0 agl total: 2 list_total: 2 fname_total: 0 hit_total: 10081 miss_total: 19 swrong=0 ewrong=0 - open/close 766 (time 1789054524.28 total 10.03 last 76.40) total: 900 open/close in 11.80 seconds: 76.30 ops/second llite.lustre-ffff9265cfcbd000.statahead_max=0 1001 real 0m25.846s user 0m0.033s sys 0m2.825s 'ls -l' 1000 files without statahead: 26 sec llite.lustre-ffff9265cfcbd000.statahead_max=2048 llite.lustre-ffff9265cfcbd000.statahead_max=2048 1001 real 0m6.609s user 0m0.021s sys 0m1.385s 'ls -l' 1000 files with statahead: 7 sec statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11080 miss_total: 20 swrong=0 ewrong=0 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 19 seconds llite.lustre-ffff9265cfcbd000.statahead_stats= statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11080 miss_total: 20 mdc.lustre-MDT0000-mdc-ffff9265cfcbd000.batch_stats= snapshot_time: 1789054589.239731256 secs.nsecs start_time: 1789054227.219703656 secs.nsecs elapsed_time: 362.020027600 secs.nsecs subreqs per batch batches %% cum %% 1: 1 14 14 2: 0 0 14 4: 1 14 28 8: 1 14 42 16: 1 14 57 32: 0 0 57 64: 1 14 71 128: 0 0 71 256: 2 28 100 mdc.lustre-MDT0001-mdc-ffff9265cfcbd000.batch_stats= snapshot_time: 1789054589.239799903 secs.nsecs start_time: 1789054227.219858276 secs.nsecs elapsed_time: 362.019941627 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 1 14 14 8: 2 28 42 16: 1 14 57 32: 1 14 71 64: 0 0 71 128: 0 0 71 256: 2 28 100 - open/close 851 (time 1789054601.96 total 10.01 last 85.05) - open/close 1667 (time 1789054611.97 total 20.01 last 81.56) - open/close 2509 (time 1789054621.98 total 30.03 last 84.07) - open/close 3369 (time 1789054631.98 total 40.03 last 85.99) - open/close 4273 (time 1789054641.99 total 50.03 last 90.35) - open/close 5099 (time 1789054651.99 total 60.04 last 82.55) - open/close 5910 (time 1789054662.01 total 70.05 last 81.00) - open/close 6803 (time 1789054672.01 total 80.06 last 89.24) - open/close 7661 (time 1789054682.02 total 90.06 last 85.74) - open/close 8498 (time 1789054692.05 total 100.10 last 83.41) - open/close 9330 (time 1789054702.06 total 110.10 last 83.17) total: 10000 open/close in 118.72 seconds: 84.23 ops/second llite.lustre-ffff9265cfcbd000.statahead_batch_max=0 llite.lustre-ffff9265cfcbd000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9265cfcbd000.stats=clear mdc.lustre-MDT0001-mdc-ffff9265cfcbd000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m0.417s user 0m0.287s sys 0m14.833s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff9265cfcbd000.statahead_batch_max=256 llite.lustre-ffff9265cfcbd000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9265cfcbd000.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff9265cfcbd000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff9265cfcbd000.stats=clear mdc.lustre-MDT0001-mdc-ffff9265cfcbd000.stats=clear ls -l files in d123ad.sanity: 10001 real 0m47.131s user 0m0.272s sys 0m15.731s batched statahead RPC= hit count=9999 mdc.lustre-MDT0000-mdc-ffff9265cfcbd000.batch_stats= snapshot_time: 1789054835.939918326 secs.nsecs start_time: 1789054778.800385450 secs.nsecs elapsed_time: 57.139532876 secs.nsecs subreqs per batch batches %% cum %% 1: 106 40 40 2: 45 17 58 4: 28 10 68 8: 13 5 73 16: 12 4 78 32: 9 3 81 64: 12 4 86 128: 1 0 86 256: 34 13 100 mdc.lustre-MDT0001-mdc-ffff9265cfcbd000.batch_stats= snapshot_time: 1789054835.939985568 secs.nsecs start_time: 1789054778.800538440 secs.nsecs elapsed_time: 57.139447128 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second seq.cli-lustre-OST0000-super.width=65536 seq.cli-lustre-OST0001-super.width=65536 llite.lustre-ffff9265cfcbd000.statahead_batch_max=64 llite.lustre-ffff9265cfcbd000.statahead_max=128