== sanity test 123ad: Verify batching statahead works correctly ========================================================== 22:34:00 (1782614040) batch: statahead_max=32 statahead_batch_max=32 mdc.lustre-MDT0000-mdc-ffff8b7b6043a000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff8b7b6043a000.batch_stats=0 llite.lustre-ffff8b7b6043a000.statahead_max=32 llite.lustre-ffff8b7b6043a000.statahead_batch_max=32 seq.cli-lustre-OST0000-super.width=0x1ffffff seq.cli-lustre-OST0001-super.width=0x1ffffff kvm mdc.lustre-MDT0000-mdc-ffff8b7b6043a000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff8b7b6043a000.batch_stats=0 striped dir -i1 -c2 -H crush /mnt/lustre/d123ad.sanity total: 100 open/close in 2.64 seconds: 37.90 ops/second llite.lustre-ffff8b7b6043a000.statahead_max=0 101 real 0m2.246s user 0m0.007s sys 0m0.337s 'ls -l' 100 files without statahead: 2 sec llite.lustre-ffff8b7b6043a000.statahead_max=32 llite.lustre-ffff8b7b6043a000.statahead_max=32 101 real 0m1.114s user 0m0.004s sys 0m0.186s 'ls -l' 100 files with statahead: 1 sec statahead total: 129 statahead wrong: 2 agl total: 123 list_total: 36 fname_total: 93 hit_total: 3871 miss_total: 1391 swrong=2 ewrong=2 - open/close 748 (time 1782614068.36 total 10.00 last 74.78) total: 900 open/close in 12.24 seconds: 73.52 ops/second llite.lustre-ffff8b7b6043a000.statahead_max=0 1001 real 0m22.007s user 0m0.030s sys 0m3.151s 'ls -l' 1000 files without statahead: 22 sec llite.lustre-ffff8b7b6043a000.statahead_max=32 llite.lustre-ffff8b7b6043a000.statahead_max=32 1001 real 0m9.212s user 0m0.035s sys 0m1.571s 'ls -l' 1000 files with statahead: 9 sec statahead total: 130 statahead wrong: 2 agl total: 124 list_total: 37 fname_total: 93 hit_total: 4869 miss_total: 1393 swrong=2 ewrong=2 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 20 seconds llite.lustre-ffff8b7b6043a000.statahead_stats= statahead total: 130 statahead wrong: 2 agl total: 124 list_total: 37 fname_total: 93 hit_total: 4869 miss_total: 1393 mdc.lustre-MDT0000-mdc-ffff8b7b6043a000.batch_stats= snapshot_time: 1782614135.882241633 secs.nsecs start_time: 1782614045.391347090 secs.nsecs elapsed_time: 90.490894543 secs.nsecs subreqs per batch batches %% cum %% 1: 1 2 2 2: 2 5 8 4: 3 8 16 8: 1 2 18 16: 28 75 94 32: 2 5 100 mdc.lustre-MDT0001-mdc-ffff8b7b6043a000.batch_stats= snapshot_time: 1782614135.882313712 secs.nsecs start_time: 1782614045.391462070 secs.nsecs elapsed_time: 90.490851642 secs.nsecs subreqs per batch batches %% cum %% 1: 2 5 5 2: 0 0 5 4: 0 0 5 8: 4 10 15 16: 6 15 30 32: 28 70 100 - open/close 665 (time 1782614149.03 total 10.00 last 66.48) - open/close 1304 (time 1782614159.03 total 20.01 last 63.86) - open/close 2060 (time 1782614169.04 total 30.01 last 75.58) - open/close 2834 (time 1782614179.04 total 40.02 last 77.37) - open/close 3554 (time 1782614189.06 total 50.04 last 71.84) - open/close 4383 (time 1782614199.07 total 60.04 last 82.88) - open/close 5134 (time 1782614209.07 total 70.04 last 75.08) - open/close 5856 (time 1782614219.09 total 80.06 last 72.06) - open/close 6645 (time 1782614229.10 total 90.08 last 78.78) - open/close 7422 (time 1782614239.10 total 100.08 last 77.68) - open/close 8162 (time 1782614249.11 total 110.09 last 73.94) - open/close 8965 (time 1782614259.11 total 120.09 last 80.29) - open/close 9799 (time 1782614269.12 total 130.10 last 83.34) total: 10000 open/close in 132.62 seconds: 75.40 ops/second llite.lustre-ffff8b7b6043a000.statahead_batch_max=0 llite.lustre-ffff8b7b6043a000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff8b7b6043a000.stats=clear mdc.lustre-MDT0001-mdc-ffff8b7b6043a000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m3.270s user 0m0.314s sys 0m17.963s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff8b7b6043a000.statahead_batch_max=32 llite.lustre-ffff8b7b6043a000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff8b7b6043a000.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff8b7b6043a000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff8b7b6043a000.stats=clear mdc.lustre-MDT0001-mdc-ffff8b7b6043a000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m46.095s user 0m0.302s sys 0m16.018s batched statahead RPC= hit count=9983 mdc.lustre-MDT0000-mdc-ffff8b7b6043a000.batch_stats= snapshot_time: 1782614465.273689514 secs.nsecs start_time: 1782614346.023848640 secs.nsecs elapsed_time: 119.249840874 secs.nsecs subreqs per batch batches %% cum %% 1: 1 0 0 2: 0 0 0 4: 0 0 0 8: 1 0 0 16: 1 0 0 32: 313 99 100 mdc.lustre-MDT0001-mdc-ffff8b7b6043a000.batch_stats= snapshot_time: 1782614465.273769711 secs.nsecs start_time: 1782614346.024059898 secs.nsecs elapsed_time: 119.249709813 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 batch: statahead_max=2048 statahead_batch_max=256 mdc.lustre-MDT0000-mdc-ffff8b7b6043a000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff8b7b6043a000.batch_stats=0 llite.lustre-ffff8b7b6043a000.statahead_max=2048 llite.lustre-ffff8b7b6043a000.statahead_batch_max=256 kvm mdc.lustre-MDT0000-mdc-ffff8b7b6043a000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff8b7b6043a000.batch_stats=0 striped dir -i1 -c2 -H crush /mnt/lustre/d123ad.sanity total: 100 open/close in 2.94 seconds: 33.96 ops/second llite.lustre-ffff8b7b6043a000.statahead_max=0 101 real 0m2.668s user 0m0.007s sys 0m0.344s 'ls -l' 100 files without statahead: 3 sec llite.lustre-ffff8b7b6043a000.statahead_max=2048 llite.lustre-ffff8b7b6043a000.statahead_max=2048 101 real 0m1.105s user 0m0.005s sys 0m0.202s 'ls -l' 100 files with statahead: 1 sec statahead total: 2 statahead wrong: 0 agl total: 2 list_total: 2 fname_total: 0 hit_total: 10082 miss_total: 18 swrong=0 ewrong=0 - open/close 673 (time 1782614790.71 total 10.00 last 67.29) total: 900 open/close in 13.35 seconds: 67.40 ops/second llite.lustre-ffff8b7b6043a000.statahead_max=0 1001 real 0m24.193s user 0m0.028s sys 0m3.199s 'ls -l' 1000 files without statahead: 25 sec llite.lustre-ffff8b7b6043a000.statahead_max=2048 llite.lustre-ffff8b7b6043a000.statahead_max=2048 1001 real 0m6.051s user 0m0.031s sys 0m1.529s 'ls -l' 1000 files with statahead: 6 sec statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11081 miss_total: 19 swrong=0 ewrong=0 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 23 seconds llite.lustre-ffff8b7b6043a000.statahead_stats= statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11081 miss_total: 19 mdc.lustre-MDT0000-mdc-ffff8b7b6043a000.batch_stats= snapshot_time: 1782614859.727505014 secs.nsecs start_time: 1782614468.112069256 secs.nsecs elapsed_time: 391.615435758 secs.nsecs subreqs per batch batches %% cum %% 1: 1 14 14 2: 0 0 14 4: 3 42 57 8: 1 14 71 16: 0 0 71 32: 0 0 71 64: 0 0 71 128: 0 0 71 256: 2 28 100 mdc.lustre-MDT0001-mdc-ffff8b7b6043a000.batch_stats= snapshot_time: 1782614859.727699165 secs.nsecs start_time: 1782614468.112277169 secs.nsecs elapsed_time: 391.615421996 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 0 0 0 8: 4 40 40 16: 1 10 50 32: 2 20 70 64: 1 10 80 128: 0 0 80 256: 2 20 100 - open/close 821 (time 1782614872.28 total 10.00 last 82.08) - open/close 1478 (time 1782614882.28 total 20.00 last 65.69) - open/close 2195 (time 1782614892.28 total 30.00 last 71.69) - open/close 2988 (time 1782614902.29 total 40.01 last 79.24) - open/close 3662 (time 1782614912.29 total 50.02 last 67.37) - open/close 4342 (time 1782614922.30 total 60.02 last 67.94) - open/close 5037 (time 1782614932.32 total 70.04 last 69.39) - open/close 5661 (time 1782614942.32 total 80.04 last 62.38) - open/close 6382 (time 1782614952.32 total 90.05 last 72.10) - open/close 6992 (time 1782614962.33 total 100.05 last 60.97) - open/close 7595 (time 1782614972.34 total 110.07 last 60.21) - open/close 8391 (time 1782614982.35 total 120.07 last 79.55) - open/close 9085 (time 1782614992.35 total 130.07 last 69.40) - open/close 9751 (time 1782615002.35 total 140.07 last 66.60) total: 10000 open/close in 143.28 seconds: 69.80 ops/second llite.lustre-ffff8b7b6043a000.statahead_batch_max=0 llite.lustre-ffff8b7b6043a000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff8b7b6043a000.stats=clear mdc.lustre-MDT0001-mdc-ffff8b7b6043a000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m2.562s user 0m0.298s sys 0m17.397s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff8b7b6043a000.statahead_batch_max=256 llite.lustre-ffff8b7b6043a000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff8b7b6043a000.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff8b7b6043a000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff8b7b6043a000.stats=clear mdc.lustre-MDT0001-mdc-ffff8b7b6043a000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m3.720s user 0m0.307s sys 0m17.419s batched statahead RPC= hit count=9999 mdc.lustre-MDT0000-mdc-ffff8b7b6043a000.batch_stats= snapshot_time: 1782615155.889053753 secs.nsecs start_time: 1782615076.480484364 secs.nsecs elapsed_time: 79.408569389 secs.nsecs subreqs per batch batches %% cum %% 1: 253 56 56 2: 74 16 72 4: 37 8 81 8: 20 4 85 16: 11 2 88 32: 12 2 90 64: 7 1 92 128: 0 0 92 256: 34 7 100 mdc.lustre-MDT0001-mdc-ffff8b7b6043a000.batch_stats= snapshot_time: 1782615155.889112424 secs.nsecs start_time: 1782615076.480553862 secs.nsecs elapsed_time: 79.408558562 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second seq.cli-lustre-OST0000-super.width=65536 seq.cli-lustre-OST0001-super.width=65536 llite.lustre-ffff8b7b6043a000.statahead_batch_max=64 llite.lustre-ffff8b7b6043a000.statahead_max=128