== sanity test 123ad: Verify batching statahead works correctly ========================================================== 03:59:25 (1787471965) batch: statahead_max=32 statahead_batch_max=32 mdc.lustre-MDT0000-mdc-ffff9ef410ebb800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9ef410ebb800.batch_stats=0 llite.lustre-ffff9ef410ebb800.statahead_max=32 llite.lustre-ffff9ef410ebb800.statahead_batch_max=32 seq.cli-lustre-OST0000-super.width=0x1ffffff seq.cli-lustre-OST0001-super.width=0x1ffffff kvm mdc.lustre-MDT0000-mdc-ffff9ef410ebb800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9ef410ebb800.batch_stats=0 striped dir -i1 -c2 -H crush /mnt/lustre/d123ad.sanity total: 100 open/close in 2.19 seconds: 45.65 ops/second llite.lustre-ffff9ef410ebb800.statahead_max=0 101 real 0m1.941s user 0m0.007s sys 0m0.329s 'ls -l' 100 files without statahead: 2 sec llite.lustre-ffff9ef410ebb800.statahead_max=32 llite.lustre-ffff9ef410ebb800.statahead_max=32 101 real 0m1.235s user 0m0.005s sys 0m0.222s 'ls -l' 100 files with statahead: 1 sec statahead total: 149 statahead wrong: 2 agl total: 143 list_total: 31 fname_total: 91 hit_total: 4002 miss_total: 1650 swrong=2 ewrong=2 total: 900 open/close in 8.99 seconds: 100.10 ops/second llite.lustre-ffff9ef410ebb800.statahead_max=0 1001 real 0m19.934s user 0m0.034s sys 0m3.098s 'ls -l' 1000 files without statahead: 20 sec llite.lustre-ffff9ef410ebb800.statahead_max=32 llite.lustre-ffff9ef410ebb800.statahead_max=32 1001 real 0m8.137s user 0m0.029s sys 0m1.469s 'ls -l' 1000 files with statahead: 8 sec statahead total: 150 statahead wrong: 2 agl total: 144 list_total: 32 fname_total: 91 hit_total: 4999 miss_total: 1653 swrong=2 ewrong=2 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 20 seconds llite.lustre-ffff9ef410ebb800.statahead_stats= statahead total: 150 statahead wrong: 2 agl total: 144 list_total: 32 fname_total: 91 hit_total: 4999 miss_total: 1653 mdc.lustre-MDT0000-mdc-ffff9ef410ebb800.batch_stats= snapshot_time: 1787472052.803537007 secs.nsecs start_time: 1787471969.957958025 secs.nsecs elapsed_time: 82.845578982 secs.nsecs subreqs per batch batches %% cum %% 1: 4 10 10 2: 2 5 15 4: 1 2 17 8: 2 5 23 16: 22 56 79 32: 8 20 100 mdc.lustre-MDT0001-mdc-ffff9ef410ebb800.batch_stats= snapshot_time: 1787472052.803607017 secs.nsecs start_time: 1787471969.958041450 secs.nsecs elapsed_time: 82.845565567 secs.nsecs subreqs per batch batches %% cum %% 1: 1 2 2 2: 0 0 2 4: 1 2 5 8: 3 7 12 16: 11 28 41 32: 23 58 100 - open/close 836 (time 1787472065.84 total 10.01 last 83.54) - open/close 1848 (time 1787472075.84 total 20.01 last 101.18) - open/close 2799 (time 1787472085.85 total 30.02 last 95.04) - open/close 3727 (time 1787472095.87 total 40.04 last 92.59) - open/close 4682 (time 1787472105.87 total 50.04 last 95.49) - open/close 5601 (time 1787472115.88 total 60.05 last 91.84) - open/close 6482 (time 1787472125.88 total 70.05 last 88.07) - open/close 7458 (time 1787472135.88 total 80.05 last 97.59) - open/close 8417 (time 1787472145.89 total 90.06 last 95.82) - open/close 9304 (time 1787472155.90 total 100.07 last 88.62) total: 10000 open/close in 109.77 seconds: 91.10 ops/second llite.lustre-ffff9ef410ebb800.statahead_batch_max=0 llite.lustre-ffff9ef410ebb800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9ef410ebb800.stats=clear mdc.lustre-MDT0001-mdc-ffff9ef410ebb800.stats=clear ls -l files in d123ad.sanity: 10001 real 1m7.932s user 0m0.329s sys 0m15.919s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff9ef410ebb800.statahead_batch_max=32 llite.lustre-ffff9ef410ebb800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9ef410ebb800.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff9ef410ebb800.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff9ef410ebb800.stats=clear mdc.lustre-MDT0001-mdc-ffff9ef410ebb800.stats=clear ls -l files in d123ad.sanity: 10001 real 1m42.293s user 0m0.318s sys 0m15.232s batched statahead RPC= hit count=9978 mdc.lustre-MDT0000-mdc-ffff9ef410ebb800.batch_stats= snapshot_time: 1787472358.824622799 secs.nsecs start_time: 1787472243.857202866 secs.nsecs elapsed_time: 114.967419933 secs.nsecs subreqs per batch batches %% cum %% 1: 1 0 0 2: 0 0 0 4: 0 0 0 8: 1 0 0 16: 1 0 0 32: 313 99 100 mdc.lustre-MDT0001-mdc-ffff9ef410ebb800.batch_stats= snapshot_time: 1787472358.824667338 secs.nsecs start_time: 1787472243.857331857 secs.nsecs elapsed_time: 114.967335481 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 batch: statahead_max=2048 statahead_batch_max=256 mdc.lustre-MDT0000-mdc-ffff9ef410ebb800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9ef410ebb800.batch_stats=0 llite.lustre-ffff9ef410ebb800.statahead_max=2048 llite.lustre-ffff9ef410ebb800.statahead_batch_max=256 kvm mdc.lustre-MDT0000-mdc-ffff9ef410ebb800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9ef410ebb800.batch_stats=0 striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d123ad.sanity total: 100 open/close in 2.46 seconds: 40.66 ops/second llite.lustre-ffff9ef410ebb800.statahead_max=0 101 real 0m2.593s user 0m0.006s sys 0m0.349s 'ls -l' 100 files without statahead: 3 sec llite.lustre-ffff9ef410ebb800.statahead_max=2048 llite.lustre-ffff9ef410ebb800.statahead_max=2048 101 real 0m1.084s user 0m0.004s sys 0m0.164s 'ls -l' 100 files with statahead: 2 sec statahead total: 2 statahead wrong: 0 agl total: 2 list_total: 2 fname_total: 0 hit_total: 10077 miss_total: 23 swrong=0 ewrong=0 - open/close 713 (time 1787472647.69 total 10.03 last 71.09) total: 900 open/close in 13.09 seconds: 68.76 ops/second llite.lustre-ffff9ef410ebb800.statahead_max=0 1001 real 0m24.103s user 0m0.039s sys 0m3.226s 'ls -l' 1000 files without statahead: 24 sec llite.lustre-ffff9ef410ebb800.statahead_max=2048 llite.lustre-ffff9ef410ebb800.statahead_max=2048 1001 real 0m6.916s user 0m0.034s sys 0m1.634s 'ls -l' 1000 files with statahead: 6 sec statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11076 miss_total: 24 swrong=0 ewrong=0 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 16 seconds llite.lustre-ffff9ef410ebb800.statahead_stats= statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11076 miss_total: 24 mdc.lustre-MDT0000-mdc-ffff9ef410ebb800.batch_stats= snapshot_time: 1787472712.739713175 secs.nsecs start_time: 1787472361.292113814 secs.nsecs elapsed_time: 351.447599361 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 3 33 33 8: 1 11 44 16: 2 22 66 32: 1 11 77 64: 0 0 77 128: 0 0 77 256: 2 22 100 mdc.lustre-MDT0001-mdc-ffff9ef410ebb800.batch_stats= snapshot_time: 1787472712.744182459 secs.nsecs start_time: 1787472361.296909294 secs.nsecs elapsed_time: 351.447273165 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 3 33 33 8: 3 33 66 16: 0 0 66 32: 0 0 66 64: 1 11 77 128: 0 0 77 256: 2 22 100 - open/close 876 (time 1787472725.27 total 10.04 last 87.29) - open/close 1839 (time 1787472735.27 total 20.04 last 96.30) - open/close 2826 (time 1787472745.27 total 30.04 last 98.64) - open/close 3789 (time 1787472755.28 total 40.05 last 96.22) - open/close 4669 (time 1787472765.29 total 50.06 last 87.90) - open/close 5388 (time 1787472775.30 total 60.07 last 71.85) - open/close 6083 (time 1787472785.32 total 70.09 last 69.38) - open/close 6746 (time 1787472795.32 total 80.09 last 66.28) - open/close 7307 (time 1787472805.34 total 90.11 last 55.97) - open/close 8046 (time 1787472815.35 total 100.12 last 73.88) - open/close 8827 (time 1787472825.35 total 110.12 last 78.04) - open/close 9471 (time 1787472835.36 total 120.13 last 64.38) total: 10000 open/close in 127.24 seconds: 78.59 ops/second llite.lustre-ffff9ef410ebb800.statahead_batch_max=0 llite.lustre-ffff9ef410ebb800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9ef410ebb800.stats=clear mdc.lustre-MDT0001-mdc-ffff9ef410ebb800.stats=clear ls -l files in d123ad.sanity: 10001 real 1m14.377s user 0m0.344s sys 0m16.506s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff9ef410ebb800.statahead_batch_max=256 llite.lustre-ffff9ef410ebb800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9ef410ebb800.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff9ef410ebb800.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff9ef410ebb800.stats=clear mdc.lustre-MDT0001-mdc-ffff9ef410ebb800.stats=clear ls -l files in d123ad.sanity: 10001 real 1m9.507s user 0m0.329s sys 0m16.229s batched statahead RPC= hit count=9999 mdc.lustre-MDT0000-mdc-ffff9ef410ebb800.batch_stats= snapshot_time: 1787473012.838269689 secs.nsecs start_time: 1787472928.019117641 secs.nsecs elapsed_time: 84.819152048 secs.nsecs subreqs per batch batches %% cum %% 1: 124 44 44 2: 41 14 58 4: 31 11 69 8: 20 7 76 16: 11 3 80 32: 16 5 86 64: 3 1 87 128: 0 0 87 256: 35 12 100 mdc.lustre-MDT0001-mdc-ffff9ef410ebb800.batch_stats= snapshot_time: 1787473012.838337378 secs.nsecs start_time: 1787472928.019210361 secs.nsecs elapsed_time: 84.819127017 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second seq.cli-lustre-OST0000-super.width=65536 seq.cli-lustre-OST0001-super.width=65536 llite.lustre-ffff9ef410ebb800.statahead_batch_max=64 llite.lustre-ffff9ef410ebb800.statahead_max=128