== sanity test 123ad: Verify batching statahead works correctly ========================================================== 10:31:22 (1783953082) batch: statahead_max=32 statahead_batch_max=32 mdc.lustre-MDT0000-mdc-ffff8a1a475e2800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff8a1a475e2800.batch_stats=0 llite.lustre-ffff8a1a475e2800.statahead_max=32 llite.lustre-ffff8a1a475e2800.statahead_batch_max=32 seq.cli-lustre-OST0000-super.width=0x1ffffff seq.cli-lustre-OST0001-super.width=0x1ffffff kvm mdc.lustre-MDT0000-mdc-ffff8a1a475e2800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff8a1a475e2800.batch_stats=0 striped dir -i1 -c2 -H crush2 /mnt/lustre/d123ad.sanity total: 100 open/close in 2.32 seconds: 43.06 ops/second llite.lustre-ffff8a1a475e2800.statahead_max=0 101 real 0m1.933s user 0m0.004s sys 0m0.335s 'ls -l' 100 files without statahead: 2 sec llite.lustre-ffff8a1a475e2800.statahead_max=32 llite.lustre-ffff8a1a475e2800.statahead_max=32 101 real 0m1.156s user 0m0.005s sys 0m0.202s 'ls -l' 100 files with statahead: 1 sec statahead total: 121 statahead wrong: 2 agl total: 115 list_total: 31 fname_total: 76 hit_total: 3879 miss_total: 1443 swrong=2 ewrong=2 - open/close 886 (time 1783953108.21 total 10.01 last 88.52) total: 900 open/close in 10.16 seconds: 88.62 ops/second llite.lustre-ffff8a1a475e2800.statahead_max=0 1001 real 0m21.927s user 0m0.028s sys 0m3.212s 'ls -l' 1000 files without statahead: 22 sec llite.lustre-ffff8a1a475e2800.statahead_max=32 llite.lustre-ffff8a1a475e2800.statahead_max=32 1001 real 0m11.124s user 0m0.029s sys 0m1.695s 'ls -l' 1000 files with statahead: 12 sec statahead total: 122 statahead wrong: 2 agl total: 116 list_total: 32 fname_total: 76 hit_total: 4878 miss_total: 1444 swrong=2 ewrong=2 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 18 seconds llite.lustre-ffff8a1a475e2800.statahead_stats= statahead total: 122 statahead wrong: 2 agl total: 116 list_total: 32 fname_total: 76 hit_total: 4878 miss_total: 1444 mdc.lustre-MDT0000-mdc-ffff8a1a475e2800.batch_stats= snapshot_time: 1783953172.024846362 secs.nsecs start_time: 1783953086.192204621 secs.nsecs elapsed_time: 85.832641741 secs.nsecs subreqs per batch batches %% cum %% 1: 4 10 10 2: 1 2 13 4: 0 0 13 8: 3 7 21 16: 26 68 89 32: 4 10 100 mdc.lustre-MDT0001-mdc-ffff8a1a475e2800.batch_stats= snapshot_time: 1783953172.024919940 secs.nsecs start_time: 1783953086.192419704 secs.nsecs elapsed_time: 85.832500236 secs.nsecs subreqs per batch batches %% cum %% 1: 1 2 2 2: 1 2 5 4: 2 5 10 8: 3 7 17 16: 9 22 40 32: 24 60 100 - open/close 993 (time 1783953184.74 total 10.01 last 99.23) - open/close 1881 (time 1783953194.75 total 20.02 last 88.72) - open/close 2843 (time 1783953204.75 total 30.02 last 96.18) - open/close 3711 (time 1783953214.75 total 40.02 last 86.77) - open/close 4523 (time 1783953224.78 total 50.04 last 81.01) - open/close 5359 (time 1783953234.79 total 60.06 last 83.51) - open/close 6248 (time 1783953244.79 total 70.06 last 88.85) - open/close 6843 (time 1783953254.82 total 80.08 last 59.37) - open/close 7396 (time 1783953264.83 total 90.10 last 55.21) - open/close 8113 (time 1783953274.84 total 100.10 last 71.67) - open/close 8942 (time 1783953284.84 total 110.11 last 82.88) - open/close 9723 (time 1783953294.85 total 120.12 last 78.03) total: 10000 open/close in 124.05 seconds: 80.61 ops/second llite.lustre-ffff8a1a475e2800.statahead_batch_max=0 llite.lustre-ffff8a1a475e2800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff8a1a475e2800.stats=clear mdc.lustre-MDT0001-mdc-ffff8a1a475e2800.stats=clear ls -l files in d123ad.sanity: 10001 real 1m16.875s user 0m0.370s sys 0m17.575s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff8a1a475e2800.statahead_batch_max=32 llite.lustre-ffff8a1a475e2800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff8a1a475e2800.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff8a1a475e2800.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff8a1a475e2800.stats=clear mdc.lustre-MDT0001-mdc-ffff8a1a475e2800.stats=clear ls -l files in d123ad.sanity: 10001 real 1m59.864s user 0m0.365s sys 0m16.083s batched statahead RPC= hit count=9986 mdc.lustre-MDT0000-mdc-ffff8a1a475e2800.batch_stats= snapshot_time: 1783953529.520805410 secs.nsecs start_time: 1783953392.552418508 secs.nsecs elapsed_time: 136.968386902 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 0 0 0 8: 1 0 0 16: 1 0 0 32: 313 99 100 mdc.lustre-MDT0001-mdc-ffff8a1a475e2800.batch_stats= snapshot_time: 1783953529.520855934 secs.nsecs start_time: 1783953392.552490644 secs.nsecs elapsed_time: 136.968365290 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 batch: statahead_max=2048 statahead_batch_max=256 mdc.lustre-MDT0000-mdc-ffff8a1a475e2800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff8a1a475e2800.batch_stats=0 llite.lustre-ffff8a1a475e2800.statahead_max=2048 llite.lustre-ffff8a1a475e2800.statahead_batch_max=256 kvm mdc.lustre-MDT0000-mdc-ffff8a1a475e2800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff8a1a475e2800.batch_stats=0 striped dir -i1 -c2 -H crush /mnt/lustre/d123ad.sanity total: 100 open/close in 2.14 seconds: 46.76 ops/second llite.lustre-ffff8a1a475e2800.statahead_max=0 101 real 0m2.297s user 0m0.004s sys 0m0.338s 'ls -l' 100 files without statahead: 3 sec llite.lustre-ffff8a1a475e2800.statahead_max=2048 llite.lustre-ffff8a1a475e2800.statahead_max=2048 101 real 0m1.133s user 0m0.003s sys 0m0.192s 'ls -l' 100 files with statahead: 1 sec statahead total: 2 statahead wrong: 0 agl total: 2 list_total: 2 fname_total: 0 hit_total: 10085 miss_total: 15 swrong=0 ewrong=0 - open/close 797 (time 1783953857.02 total 10.00 last 79.70) total: 900 open/close in 11.41 seconds: 78.87 ops/second llite.lustre-ffff8a1a475e2800.statahead_max=0 1001 real 0m23.538s user 0m0.036s sys 0m3.374s 'ls -l' 1000 files without statahead: 23 sec llite.lustre-ffff8a1a475e2800.statahead_max=2048 llite.lustre-ffff8a1a475e2800.statahead_max=2048 1001 real 0m6.941s user 0m0.028s sys 0m1.777s 'ls -l' 1000 files with statahead: 6 sec statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11084 miss_total: 16 swrong=0 ewrong=0 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 21 seconds llite.lustre-ffff8a1a475e2800.statahead_stats= statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11084 miss_total: 16 mdc.lustre-MDT0000-mdc-ffff8a1a475e2800.batch_stats= snapshot_time: 1783953923.721209147 secs.nsecs start_time: 1783953533.144291873 secs.nsecs elapsed_time: 390.576917274 secs.nsecs subreqs per batch batches %% cum %% 1: 2 28 28 2: 0 0 28 4: 1 14 42 8: 2 28 71 16: 0 0 71 32: 0 0 71 64: 0 0 71 128: 0 0 71 256: 2 28 100 mdc.lustre-MDT0001-mdc-ffff8a1a475e2800.batch_stats= snapshot_time: 1783953923.721294909 secs.nsecs start_time: 1783953533.144463089 secs.nsecs elapsed_time: 390.576831820 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 1 11 11 4: 1 11 22 8: 2 22 44 16: 0 0 44 32: 1 11 55 64: 2 22 77 128: 0 0 77 256: 2 22 100 - open/close 798 (time 1783953936.74 total 10.00 last 79.76) - open/close 1610 (time 1783953946.75 total 20.02 last 81.11) - open/close 2344 (time 1783953956.75 total 30.02 last 73.36) - open/close 3104 (time 1783953966.76 total 40.02 last 75.99) - open/close 3736 (time 1783953976.77 total 50.03 last 63.14) - open/close 4503 (time 1783953986.77 total 60.04 last 76.67) - open/close 5388 (time 1783953996.77 total 70.04 last 88.47) - open/close 6125 (time 1783954006.78 total 80.04 last 73.68) - open/close 6715 (time 1783954016.78 total 90.05 last 58.96) - open/close 7507 (time 1783954026.79 total 100.06 last 79.13) - open/close 8231 (time 1783954036.80 total 110.07 last 72.33) - open/close 8800 (time 1783954046.82 total 120.08 last 56.82) - open/close 9442 (time 1783954056.82 total 130.08 last 64.19) total: 10000 open/close in 138.45 seconds: 72.23 ops/second llite.lustre-ffff8a1a475e2800.statahead_batch_max=0 llite.lustre-ffff8a1a475e2800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff8a1a475e2800.stats=clear mdc.lustre-MDT0001-mdc-ffff8a1a475e2800.stats=clear ls -l files in d123ad.sanity: 10001 real 1m18.655s user 0m0.385s sys 0m17.412s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff8a1a475e2800.statahead_batch_max=256 llite.lustre-ffff8a1a475e2800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff8a1a475e2800.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff8a1a475e2800.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff8a1a475e2800.stats=clear mdc.lustre-MDT0001-mdc-ffff8a1a475e2800.stats=clear ls -l files in d123ad.sanity: 10001 real 1m0.935s user 0m0.310s sys 0m16.534s batched statahead RPC= hit count=9999 mdc.lustre-MDT0000-mdc-ffff8a1a475e2800.batch_stats= snapshot_time: 1783954231.344810150 secs.nsecs start_time: 1783954152.703073927 secs.nsecs elapsed_time: 78.641736223 secs.nsecs subreqs per batch batches %% cum %% 1: 142 46 46 2: 50 16 62 4: 29 9 72 8: 19 6 78 16: 9 2 81 32: 18 5 87 64: 2 0 87 128: 3 0 88 256: 34 11 100 mdc.lustre-MDT0001-mdc-ffff8a1a475e2800.batch_stats= snapshot_time: 1783954231.344867845 secs.nsecs start_time: 1783954152.703205832 secs.nsecs elapsed_time: 78.641662013 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second seq.cli-lustre-OST0000-super.width=65536 seq.cli-lustre-OST0001-super.width=65536 llite.lustre-ffff8a1a475e2800.statahead_batch_max=64 llite.lustre-ffff8a1a475e2800.statahead_max=128