== sanity test 123ad: Verify batching statahead works correctly ========================================================== 04:06:17 (1787472377) batch: statahead_max=32 statahead_batch_max=32 mdc.lustre-MDT0000-mdc-ffff941b061a9800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff941b061a9800.batch_stats=0 llite.lustre-ffff941b061a9800.statahead_max=32 llite.lustre-ffff941b061a9800.statahead_batch_max=32 seq.cli-lustre-OST0000-super.width=0x1ffffff seq.cli-lustre-OST0001-super.width=0x1ffffff kvm mdc.lustre-MDT0000-mdc-ffff941b061a9800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff941b061a9800.batch_stats=0 striped dir -i1 -c2 -H crush /mnt/lustre/d123ad.sanity total: 100 open/close in 3.52 seconds: 28.38 ops/second llite.lustre-ffff941b061a9800.statahead_max=0 101 real 0m2.997s user 0m0.006s sys 0m0.364s 'ls -l' 100 files without statahead: 3 sec llite.lustre-ffff941b061a9800.statahead_max=32 llite.lustre-ffff941b061a9800.statahead_max=32 101 real 0m1.509s user 0m0.006s sys 0m0.198s 'ls -l' 100 files with statahead: 1 sec statahead total: 142 statahead wrong: 2 agl total: 136 list_total: 31 fname_total: 96 hit_total: 3912 miss_total: 1563 swrong=2 ewrong=2 - open/close 651 (time 1787472408.24 total 10.02 last 64.94) total: 900 open/close in 14.51 seconds: 62.04 ops/second llite.lustre-ffff941b061a9800.statahead_max=0 1001 real 0m25.064s user 0m0.041s sys 0m3.382s 'ls -l' 1000 files without statahead: 25 sec llite.lustre-ffff941b061a9800.statahead_max=32 llite.lustre-ffff941b061a9800.statahead_max=32 1001 real 0m9.147s user 0m0.031s sys 0m1.506s 'ls -l' 1000 files with statahead: 10 sec statahead total: 143 statahead wrong: 2 agl total: 137 list_total: 32 fname_total: 96 hit_total: 4911 miss_total: 1564 swrong=2 ewrong=2 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 20 seconds llite.lustre-ffff941b061a9800.statahead_stats= statahead total: 143 statahead wrong: 2 agl total: 137 list_total: 32 fname_total: 96 hit_total: 4911 miss_total: 1564 mdc.lustre-MDT0000-mdc-ffff941b061a9800.batch_stats= snapshot_time: 1787472481.044075811 secs.nsecs start_time: 1787472382.502904670 secs.nsecs elapsed_time: 98.541171141 secs.nsecs subreqs per batch batches %% cum %% 1: 4 10 10 2: 1 2 13 4: 1 2 15 8: 3 7 23 16: 23 60 84 32: 6 15 100 mdc.lustre-MDT0001-mdc-ffff941b061a9800.batch_stats= snapshot_time: 1787472481.044151712 secs.nsecs start_time: 1787472382.503316545 secs.nsecs elapsed_time: 98.540835167 secs.nsecs subreqs per batch batches %% cum %% 1: 1 2 2 2: 0 0 2 4: 2 5 7 8: 3 7 15 16: 11 27 42 32: 23 57 100 - open/close 694 (time 1787472494.03 total 10.01 last 69.35) - open/close 1498 (time 1787472504.06 total 20.03 last 80.19) - open/close 2289 (time 1787472514.07 total 30.04 last 79.04) - open/close 3040 (time 1787472524.09 total 40.06 last 74.94) - open/close 3835 (time 1787472534.09 total 50.06 last 79.48) - open/close 4576 (time 1787472544.09 total 60.06 last 74.09) - open/close 5321 (time 1787472554.10 total 70.07 last 74.46) - open/close 6049 (time 1787472564.10 total 80.08 last 72.75) - open/close 6681 (time 1787472574.11 total 90.08 last 63.15) - open/close 7291 (time 1787472584.12 total 100.09 last 60.94) - open/close 7938 (time 1787472594.13 total 110.10 last 64.64) - open/close 8574 (time 1787472604.15 total 120.12 last 63.48) - open/close 9181 (time 1787472614.16 total 130.13 last 60.65) - open/close 9938 (time 1787472624.16 total 140.14 last 75.65) total: 10000 open/close in 141.18 seconds: 70.83 ops/second llite.lustre-ffff941b061a9800.statahead_batch_max=0 llite.lustre-ffff941b061a9800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff941b061a9800.stats=clear mdc.lustre-MDT0001-mdc-ffff941b061a9800.stats=clear ls -l files in d123ad.sanity: 10001 real 1m9.085s user 0m0.325s sys 0m15.883s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff941b061a9800.statahead_batch_max=32 llite.lustre-ffff941b061a9800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff941b061a9800.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff941b061a9800.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff941b061a9800.stats=clear mdc.lustre-MDT0001-mdc-ffff941b061a9800.stats=clear ls -l files in d123ad.sanity: 10001 real 2m0.626s user 0m0.308s sys 0m15.452s batched statahead RPC= hit count=9970 mdc.lustre-MDT0000-mdc-ffff941b061a9800.batch_stats= snapshot_time: 1787472841.133290342 secs.nsecs start_time: 1787472705.558891029 secs.nsecs elapsed_time: 135.574399313 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 0 0 0 8: 1 0 0 16: 2 0 0 32: 313 99 100 mdc.lustre-MDT0001-mdc-ffff941b061a9800.batch_stats= snapshot_time: 1787472841.133356044 secs.nsecs start_time: 1787472705.559311824 secs.nsecs elapsed_time: 135.574044220 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 batch: statahead_max=2048 statahead_batch_max=256 mdc.lustre-MDT0000-mdc-ffff941b061a9800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff941b061a9800.batch_stats=0 llite.lustre-ffff941b061a9800.statahead_max=2048 llite.lustre-ffff941b061a9800.statahead_batch_max=256 kvm mdc.lustre-MDT0000-mdc-ffff941b061a9800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff941b061a9800.batch_stats=0 striped dir -i1 -c2 -H crush2 /mnt/lustre/d123ad.sanity total: 100 open/close in 3.07 seconds: 32.58 ops/second llite.lustre-ffff941b061a9800.statahead_max=0 101 real 0m3.221s user 0m0.005s sys 0m0.355s 'ls -l' 100 files without statahead: 4 sec llite.lustre-ffff941b061a9800.statahead_max=2048 llite.lustre-ffff941b061a9800.statahead_max=2048 101 real 0m0.771s user 0m0.002s sys 0m0.172s 'ls -l' 100 files with statahead: 1 sec statahead total: 2 statahead wrong: 0 agl total: 2 list_total: 2 fname_total: 0 hit_total: 10069 miss_total: 31 swrong=0 ewrong=0 - open/close 818 (time 1787473170.49 total 10.00 last 81.79) total: 900 open/close in 11.27 seconds: 79.85 ops/second llite.lustre-ffff941b061a9800.statahead_max=0 1001 real 0m27.206s user 0m0.046s sys 0m3.170s 'ls -l' 1000 files without statahead: 28 sec llite.lustre-ffff941b061a9800.statahead_max=2048 llite.lustre-ffff941b061a9800.statahead_max=2048 1001 real 0m7.312s user 0m0.035s sys 0m1.494s 'ls -l' 1000 files with statahead: 7 sec statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11068 miss_total: 32 swrong=0 ewrong=0 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 20 seconds llite.lustre-ffff941b061a9800.statahead_stats= statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11068 miss_total: 32 mdc.lustre-MDT0000-mdc-ffff941b061a9800.batch_stats= snapshot_time: 1787473241.033370783 secs.nsecs start_time: 1787472845.017429509 secs.nsecs elapsed_time: 396.015941274 secs.nsecs subreqs per batch batches %% cum %% 1: 3 33 33 2: 1 11 44 4: 1 11 55 8: 2 22 77 16: 0 0 77 32: 0 0 77 64: 0 0 77 128: 0 0 77 256: 2 22 100 mdc.lustre-MDT0001-mdc-ffff941b061a9800.batch_stats= snapshot_time: 1787473241.033439924 secs.nsecs start_time: 1787472845.017596431 secs.nsecs elapsed_time: 396.015843493 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 1 9 9 8: 3 27 36 16: 2 18 54 32: 3 27 81 64: 0 0 81 128: 0 0 81 256: 2 18 100 - open/close 1081 (time 1787473253.31 total 10.01 last 108.03) - open/close 1715 (time 1787473263.32 total 20.01 last 63.37) - open/close 2555 (time 1787473273.32 total 30.01 last 83.98) - open/close 3404 (time 1787473283.33 total 40.02 last 84.83) - open/close 4232 (time 1787473293.34 total 50.04 last 82.66) - open/close 5246 (time 1787473303.35 total 60.05 last 101.33) - open/close 6170 (time 1787473313.35 total 70.05 last 92.37) - open/close 7099 (time 1787473323.36 total 80.05 last 92.89) - open/close 8020 (time 1787473333.37 total 90.06 last 92.00) - open/close 8873 (time 1787473343.37 total 100.06 last 85.27) - open/close 9702 (time 1787473353.37 total 110.07 last 82.89) total: 10000 open/close in 113.55 seconds: 88.07 ops/second llite.lustre-ffff941b061a9800.statahead_batch_max=0 llite.lustre-ffff941b061a9800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff941b061a9800.stats=clear mdc.lustre-MDT0001-mdc-ffff941b061a9800.stats=clear ls -l files in d123ad.sanity: 10001 real 1m17.712s user 0m0.371s sys 0m16.825s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff941b061a9800.statahead_batch_max=256 llite.lustre-ffff941b061a9800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff941b061a9800.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff941b061a9800.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff941b061a9800.stats=clear mdc.lustre-MDT0001-mdc-ffff941b061a9800.stats=clear ls -l files in d123ad.sanity: 10001 real 1m3.710s user 0m0.337s sys 0m16.229s batched statahead RPC= hit count=9999 mdc.lustre-MDT0000-mdc-ffff941b061a9800.batch_stats= snapshot_time: 1787473525.423199977 secs.nsecs start_time: 1787473444.986862308 secs.nsecs elapsed_time: 80.436337669 secs.nsecs subreqs per batch batches %% cum %% 1: 168 45 45 2: 68 18 63 4: 46 12 76 8: 19 5 81 16: 12 3 84 32: 20 5 90 64: 3 0 90 128: 0 0 90 256: 34 9 100 mdc.lustre-MDT0001-mdc-ffff941b061a9800.batch_stats= snapshot_time: 1787473525.423263497 secs.nsecs start_time: 1787473444.987011887 secs.nsecs elapsed_time: 80.436251610 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second seq.cli-lustre-OST0000-super.width=65536 seq.cli-lustre-OST0001-super.width=65536 llite.lustre-ffff941b061a9800.statahead_batch_max=64 llite.lustre-ffff941b061a9800.statahead_max=128