== sanity test 123ad: Verify batching statahead works correctly ========================================================== 15:14:39 (1789067679) batch: statahead_max=32 statahead_batch_max=32 mdc.lustre-MDT0000-mdc-ffff948650e74000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff948650e74000.batch_stats=0 llite.lustre-ffff948650e74000.statahead_max=32 llite.lustre-ffff948650e74000.statahead_batch_max=32 seq.cli-lustre-OST0000-super.width=0x1ffffff seq.cli-lustre-OST0001-super.width=0x1ffffff kvm mdc.lustre-MDT0000-mdc-ffff948650e74000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff948650e74000.batch_stats=0 striped dir -i1 -c2 -H crush2 /mnt/lustre/d123ad.sanity total: 100 open/close in 3.16 seconds: 31.68 ops/second llite.lustre-ffff948650e74000.statahead_max=0 101 real 0m3.142s user 0m0.007s sys 0m0.370s 'ls -l' 100 files without statahead: 3 sec llite.lustre-ffff948650e74000.statahead_max=32 llite.lustre-ffff948650e74000.statahead_max=32 101 real 0m1.222s user 0m0.008s sys 0m0.167s 'ls -l' 100 files with statahead: 1 sec statahead total: 139 statahead wrong: 2 agl total: 132 list_total: 31 fname_total: 87 hit_total: 3656 miss_total: 1419 swrong=2 ewrong=2 - open/close 573 (time 1789067709.62 total 10.00 last 57.29) total: 900 open/close in 15.26 seconds: 58.98 ops/second llite.lustre-ffff948650e74000.statahead_max=0 1001 real 0m25.836s user 0m0.046s sys 0m3.180s 'ls -l' 1000 files without statahead: 26 sec llite.lustre-ffff948650e74000.statahead_max=32 llite.lustre-ffff948650e74000.statahead_max=32 1001 real 0m11.293s user 0m0.035s sys 0m1.631s 'ls -l' 1000 files with statahead: 12 sec statahead total: 140 statahead wrong: 2 agl total: 133 list_total: 32 fname_total: 87 hit_total: 4653 miss_total: 1422 swrong=2 ewrong=2 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 23 seconds llite.lustre-ffff948650e74000.statahead_stats= statahead total: 140 statahead wrong: 2 agl total: 133 list_total: 32 fname_total: 87 hit_total: 4653 miss_total: 1422 mdc.lustre-MDT0000-mdc-ffff948650e74000.batch_stats= snapshot_time: 1789067790.028170273 secs.nsecs start_time: 1789067684.829581635 secs.nsecs elapsed_time: 105.198588638 secs.nsecs subreqs per batch batches %% cum %% 1: 1 2 2 2: 2 5 8 4: 3 8 16 8: 1 2 18 16: 24 64 83 32: 6 16 100 mdc.lustre-MDT0001-mdc-ffff948650e74000.batch_stats= snapshot_time: 1789067790.028278745 secs.nsecs start_time: 1789067684.829749430 secs.nsecs elapsed_time: 105.198529315 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 1 2 2 4: 1 2 5 8: 2 5 10 16: 12 30 41 32: 23 58 100 - open/close 610 (time 1789067803.26 total 10.00 last 60.99) - open/close 1311 (time 1789067813.27 total 20.00 last 70.09) - open/close 2033 (time 1789067823.27 total 30.00 last 72.18) - open/close 2689 (time 1789067833.28 total 40.02 last 65.52) - open/close 3395 (time 1789067843.28 total 50.02 last 70.59) - open/close 4045 (time 1789067853.30 total 60.04 last 64.86) - open/close 4707 (time 1789067863.30 total 70.04 last 66.18) - open/close 5314 (time 1789067873.32 total 80.06 last 60.61) - open/close 5929 (time 1789067883.33 total 90.07 last 61.43) - open/close 6598 (time 1789067893.35 total 100.08 last 66.79) - open/close 7313 (time 1789067903.35 total 110.09 last 71.49) - open/close 7976 (time 1789067913.36 total 120.09 last 66.25) - open/close 8599 (time 1789067923.37 total 130.11 last 62.20) - open/close 9187 (time 1789067933.38 total 140.12 last 58.76) - open/close 9814 (time 1789067943.38 total 150.12 last 62.68) total: 10000 open/close in 153.83 seconds: 65.01 ops/second llite.lustre-ffff948650e74000.statahead_batch_max=0 llite.lustre-ffff948650e74000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff948650e74000.stats=clear mdc.lustre-MDT0001-mdc-ffff948650e74000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m7.826s user 0m0.396s sys 0m17.823s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff948650e74000.statahead_batch_max=32 llite.lustre-ffff948650e74000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff948650e74000.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff948650e74000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff948650e74000.stats=clear mdc.lustre-MDT0001-mdc-ffff948650e74000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m46.414s user 0m0.372s sys 0m15.635s batched statahead RPC= hit count=9983 mdc.lustre-MDT0000-mdc-ffff948650e74000.batch_stats= snapshot_time: 1789068142.708490379 secs.nsecs start_time: 1789068025.108055603 secs.nsecs elapsed_time: 117.600434776 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 0 0 0 8: 1 0 0 16: 1 0 0 32: 313 99 100 mdc.lustre-MDT0001-mdc-ffff948650e74000.batch_stats= snapshot_time: 1789068142.708767161 secs.nsecs start_time: 1789068025.108194301 secs.nsecs elapsed_time: 117.600572860 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 batch: statahead_max=2048 statahead_batch_max=256 mdc.lustre-MDT0000-mdc-ffff948650e74000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff948650e74000.batch_stats=0 llite.lustre-ffff948650e74000.statahead_max=2048 llite.lustre-ffff948650e74000.statahead_batch_max=256 kvm mdc.lustre-MDT0000-mdc-ffff948650e74000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff948650e74000.batch_stats=0 striped dir -i1 -c2 -H crush /mnt/lustre/d123ad.sanity total: 100 open/close in 3.68 seconds: 27.16 ops/second llite.lustre-ffff948650e74000.statahead_max=0 101 real 0m2.550s user 0m0.008s sys 0m0.313s 'ls -l' 100 files without statahead: 3 sec llite.lustre-ffff948650e74000.statahead_max=2048 llite.lustre-ffff948650e74000.statahead_max=2048 101 real 0m1.177s user 0m0.008s sys 0m0.193s 'ls -l' 100 files with statahead: 1 sec statahead total: 2 statahead wrong: 0 agl total: 2 list_total: 2 fname_total: 0 hit_total: 10082 miss_total: 18 swrong=0 ewrong=0 - open/close 655 (time 1789068482.90 total 10.01 last 65.41) total: 900 open/close in 13.71 seconds: 65.63 ops/second llite.lustre-ffff948650e74000.statahead_max=0 1001 real 0m23.397s user 0m0.033s sys 0m3.266s 'ls -l' 1000 files without statahead: 23 sec llite.lustre-ffff948650e74000.statahead_max=2048 llite.lustre-ffff948650e74000.statahead_max=2048 1001 real 0m7.101s user 0m0.032s sys 0m1.530s 'ls -l' 1000 files with statahead: 7 sec statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11081 miss_total: 19 swrong=0 ewrong=0 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 21 seconds llite.lustre-ffff948650e74000.statahead_stats= statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11081 miss_total: 19 mdc.lustre-MDT0000-mdc-ffff948650e74000.batch_stats= snapshot_time: 1789068551.190770373 secs.nsecs start_time: 1789068145.479797142 secs.nsecs elapsed_time: 405.710973231 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 2 33 33 8: 2 33 66 16: 0 0 66 32: 0 0 66 64: 0 0 66 128: 0 0 66 256: 2 33 100 mdc.lustre-MDT0001-mdc-ffff948650e74000.batch_stats= snapshot_time: 1789068551.190851648 secs.nsecs start_time: 1789068145.479989320 secs.nsecs elapsed_time: 405.710862328 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 1 11 11 8: 2 22 33 16: 2 22 55 32: 1 11 66 64: 0 0 66 128: 1 11 77 256: 2 22 100 - open/close 746 (time 1789068564.25 total 10.00 last 74.59) - open/close 1437 (time 1789068574.26 total 20.01 last 69.05) - open/close 2176 (time 1789068584.26 total 30.01 last 73.90) - open/close 2901 (time 1789068594.26 total 40.01 last 72.49) - open/close 3632 (time 1789068604.26 total 50.01 last 73.07) - open/close 4370 (time 1789068614.26 total 60.02 last 73.79) - open/close 5037 (time 1789068624.27 total 70.02 last 66.69) - open/close 5763 (time 1789068634.28 total 80.03 last 72.51) - open/close 6461 (time 1789068644.30 total 90.05 last 69.64) - open/close 7227 (time 1789068654.31 total 100.06 last 76.53) - open/close 7984 (time 1789068664.32 total 110.07 last 75.64) - open/close 8696 (time 1789068674.34 total 120.09 last 71.07) - open/close 9355 (time 1789068684.34 total 130.10 last 65.86) - open/close 9991 (time 1789068694.36 total 140.11 last 63.50) total: 10000 open/close in 140.20 seconds: 71.33 ops/second llite.lustre-ffff948650e74000.statahead_batch_max=0 llite.lustre-ffff948650e74000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff948650e74000.stats=clear mdc.lustre-MDT0001-mdc-ffff948650e74000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m10.776s user 0m0.369s sys 0m17.239s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff948650e74000.statahead_batch_max=256 llite.lustre-ffff948650e74000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff948650e74000.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff948650e74000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff948650e74000.stats=clear mdc.lustre-MDT0001-mdc-ffff948650e74000.stats=clear ls -l files in d123ad.sanity: 10001 real 0m56.395s user 0m0.348s sys 0m16.818s batched statahead RPC= hit count=9999 mdc.lustre-MDT0000-mdc-ffff948650e74000.batch_stats= snapshot_time: 1789068842.973424512 secs.nsecs start_time: 1789068775.278020868 secs.nsecs elapsed_time: 67.695403644 secs.nsecs subreqs per batch batches %% cum %% 1: 52 32 32 2: 20 12 44 4: 13 8 52 8: 8 4 57 16: 7 4 61 32: 22 13 75 64: 5 3 78 128: 0 0 78 256: 35 21 100 mdc.lustre-MDT0001-mdc-ffff948650e74000.batch_stats= snapshot_time: 1789068842.973496657 secs.nsecs start_time: 1789068775.278164773 secs.nsecs elapsed_time: 67.695331884 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second seq.cli-lustre-OST0000-super.width=65536 seq.cli-lustre-OST0001-super.width=65536 llite.lustre-ffff948650e74000.statahead_batch_max=64 llite.lustre-ffff948650e74000.statahead_max=128