== sanity test 123ad: Verify batching statahead works correctly ========================================================== 01:06:11 (1788671171) batch: statahead_max=32 statahead_batch_max=32 mdc.lustre-MDT0000-mdc-ffffa023b6b2a000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffffa023b6b2a000.batch_stats=0 llite.lustre-ffffa023b6b2a000.statahead_max=32 llite.lustre-ffffa023b6b2a000.statahead_batch_max=32 seq.cli-lustre-OST0000-super.width=0x1ffffff seq.cli-lustre-OST0001-super.width=0x1ffffff kvm mdc.lustre-MDT0000-mdc-ffffa023b6b2a000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffffa023b6b2a000.batch_stats=0 striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d123ad.sanity total: 100 open/close in 2.83 seconds: 35.29 ops/second llite.lustre-ffffa023b6b2a000.statahead_max=0 101 real 0m2.602s user 0m0.004s sys 0m0.308s 'ls -l' 100 files without statahead: 2 sec llite.lustre-ffffa023b6b2a000.statahead_max=32 llite.lustre-ffffa023b6b2a000.statahead_max=32 101 real 0m1.471s user 0m0.003s sys 0m0.175s 'ls -l' 100 files with statahead: 1 sec statahead total: 132 statahead wrong: 2 agl total: 126 list_total: 30 fname_total: 88 hit_total: 3912 miss_total: 1391 swrong=2 ewrong=2 - open/close 790 (time 1788671200.85 total 10.01 last 78.96) total: 900 open/close in 11.79 seconds: 76.31 ops/second llite.lustre-ffffa023b6b2a000.statahead_max=0 1001 real 0m26.340s user 0m0.041s sys 0m2.933s 'ls -l' 1000 files without statahead: 26 sec llite.lustre-ffffa023b6b2a000.statahead_max=32 llite.lustre-ffffa023b6b2a000.statahead_max=32 1001 real 0m9.755s user 0m0.041s sys 0m1.321s 'ls -l' 1000 files with statahead: 10 sec statahead total: 133 statahead wrong: 2 agl total: 127 list_total: 31 fname_total: 88 hit_total: 4911 miss_total: 1392 swrong=2 ewrong=2 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 20 seconds llite.lustre-ffffa023b6b2a000.statahead_stats= statahead total: 133 statahead wrong: 2 agl total: 127 list_total: 31 fname_total: 88 hit_total: 4911 miss_total: 1392 mdc.lustre-MDT0000-mdc-ffffa023b6b2a000.batch_stats= snapshot_time: 1788671272.885074104 secs.nsecs start_time: 1788671177.016231939 secs.nsecs elapsed_time: 95.868842165 secs.nsecs subreqs per batch batches %% cum %% 1: 1 2 2 2: 1 2 5 4: 3 7 12 8: 2 5 17 16: 19 47 65 32: 14 35 100 mdc.lustre-MDT0001-mdc-ffffa023b6b2a000.batch_stats= snapshot_time: 1788671272.885142248 secs.nsecs start_time: 1788671177.016323841 secs.nsecs elapsed_time: 95.868818407 secs.nsecs subreqs per batch batches %% cum %% 1: 1 2 2 2: 0 0 2 4: 2 5 7 8: 3 7 15 16: 18 46 61 32: 15 38 100 - open/close 765 (time 1788671285.31 total 10.00 last 76.48) - open/close 1546 (time 1788671295.31 total 20.00 last 78.10) - open/close 2304 (time 1788671305.31 total 30.00 last 75.80) - open/close 3059 (time 1788671315.31 total 40.00 last 75.49) - open/close 3775 (time 1788671325.32 total 50.01 last 71.55) - open/close 4533 (time 1788671335.33 total 60.02 last 75.75) - open/close 5372 (time 1788671345.33 total 70.03 last 83.82) - open/close 6192 (time 1788671355.34 total 80.03 last 81.98) - open/close 7014 (time 1788671365.34 total 90.03 last 82.17) - open/close 7778 (time 1788671375.37 total 100.06 last 76.20) - open/close 8504 (time 1788671385.37 total 110.07 last 72.55) - open/close 9238 (time 1788671395.38 total 120.07 last 73.37) total: 10000 open/close in 129.34 seconds: 77.32 ops/second llite.lustre-ffffa023b6b2a000.statahead_batch_max=0 llite.lustre-ffffa023b6b2a000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffffa023b6b2a000.stats=clear mdc.lustre-MDT0001-mdc-ffffa023b6b2a000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m0.499s user 0m0.299s sys 0m14.767s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffffa023b6b2a000.statahead_batch_max=32 llite.lustre-ffffa023b6b2a000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffffa023b6b2a000.batch_stats=clear mdc.lustre-MDT0001-mdc-ffffa023b6b2a000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffffa023b6b2a000.stats=clear mdc.lustre-MDT0001-mdc-ffffa023b6b2a000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m47.793s user 0m0.341s sys 0m14.050s batched statahead RPC= hit count=9983 mdc.lustre-MDT0000-mdc-ffffa023b6b2a000.batch_stats= snapshot_time: 1788671592.582182876 secs.nsecs start_time: 1788671473.875407182 secs.nsecs elapsed_time: 118.706775694 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 0 0 0 8: 1 0 0 16: 1 0 0 32: 313 99 100 mdc.lustre-MDT0001-mdc-ffffa023b6b2a000.batch_stats= snapshot_time: 1788671592.582249367 secs.nsecs start_time: 1788671473.875537048 secs.nsecs elapsed_time: 118.706712319 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 batch: statahead_max=2048 statahead_batch_max=256 mdc.lustre-MDT0000-mdc-ffffa023b6b2a000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffffa023b6b2a000.batch_stats=0 llite.lustre-ffffa023b6b2a000.statahead_max=2048 llite.lustre-ffffa023b6b2a000.statahead_batch_max=256 kvm mdc.lustre-MDT0000-mdc-ffffa023b6b2a000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffffa023b6b2a000.batch_stats=0 striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d123ad.sanity total: 100 open/close in 2.51 seconds: 39.81 ops/second llite.lustre-ffffa023b6b2a000.statahead_max=0 101 real 0m2.412s user 0m0.005s sys 0m0.323s 'ls -l' 100 files without statahead: 2 sec llite.lustre-ffffa023b6b2a000.statahead_max=2048 llite.lustre-ffffa023b6b2a000.statahead_max=2048 101 real 0m0.961s user 0m0.014s sys 0m0.176s 'ls -l' 100 files with statahead: 1 sec statahead total: 2 statahead wrong: 0 agl total: 2 list_total: 2 fname_total: 0 hit_total: 10082 miss_total: 18 swrong=0 ewrong=0 - open/close 777 (time 1788671888.91 total 10.01 last 77.64) total: 900 open/close in 11.79 seconds: 76.37 ops/second llite.lustre-ffffa023b6b2a000.statahead_max=0 1001 real 0m20.335s user 0m0.041s sys 0m2.923s 'ls -l' 1000 files without statahead: 21 sec llite.lustre-ffffa023b6b2a000.statahead_max=2048 llite.lustre-ffffa023b6b2a000.statahead_max=2048 1001 real 0m6.050s user 0m0.034s sys 0m1.400s 'ls -l' 1000 files with statahead: 6 sec statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11081 miss_total: 19 swrong=0 ewrong=0 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 21 seconds llite.lustre-ffffa023b6b2a000.statahead_stats= statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11081 miss_total: 19 mdc.lustre-MDT0000-mdc-ffffa023b6b2a000.batch_stats= snapshot_time: 1788671950.384666350 secs.nsecs start_time: 1788671595.059618353 secs.nsecs elapsed_time: 355.325047997 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 1 12 12 4: 2 25 37 8: 1 12 50 16: 1 12 62 32: 1 12 75 64: 0 0 75 128: 0 0 75 256: 2 25 100 mdc.lustre-MDT0001-mdc-ffffa023b6b2a000.batch_stats= snapshot_time: 1788671950.384777634 secs.nsecs start_time: 1788671595.059925973 secs.nsecs elapsed_time: 355.324851661 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 2 25 25 8: 3 37 62 16: 0 0 62 32: 0 0 62 64: 1 12 75 128: 0 0 75 256: 2 25 100 - open/close 861 (time 1788671962.83 total 10.02 last 85.95) - open/close 1649 (time 1788671972.83 total 20.02 last 78.79) - open/close 2394 (time 1788671982.84 total 30.03 last 74.38) - open/close 3170 (time 1788671992.86 total 40.05 last 77.46) - open/close 3892 (time 1788672002.88 total 50.07 last 72.08) - open/close 4437 (time 1788672012.89 total 60.08 last 54.43) - open/close 5119 (time 1788672022.90 total 70.09 last 68.15) - open/close 5907 (time 1788672032.90 total 80.09 last 78.79) - open/close 6723 (time 1788672042.90 total 90.09 last 81.60) - open/close 7367 (time 1788672052.91 total 100.10 last 64.35) - open/close 8054 (time 1788672062.92 total 110.11 last 68.60) - open/close 8730 (time 1788672072.92 total 120.11 last 67.60) - open/close 9522 (time 1788672082.93 total 130.12 last 79.12) total: 10000 open/close in 135.53 seconds: 73.78 ops/second llite.lustre-ffffa023b6b2a000.statahead_batch_max=0 llite.lustre-ffffa023b6b2a000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffffa023b6b2a000.stats=clear mdc.lustre-MDT0001-mdc-ffffa023b6b2a000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m2.917s user 0m0.364s sys 0m14.935s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffffa023b6b2a000.statahead_batch_max=256 llite.lustre-ffffa023b6b2a000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffffa023b6b2a000.batch_stats=clear mdc.lustre-MDT0001-mdc-ffffa023b6b2a000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffffa023b6b2a000.stats=clear mdc.lustre-MDT0001-mdc-ffffa023b6b2a000.stats=clear ls -l files in d123ad.sanity: 10001 real 0m50.054s user 0m0.333s sys 0m15.100s batched statahead RPC= hit count=9999 mdc.lustre-MDT0000-mdc-ffffa023b6b2a000.batch_stats= snapshot_time: 1788672219.064892329 secs.nsecs start_time: 1788672158.520383520 secs.nsecs elapsed_time: 60.544508809 secs.nsecs subreqs per batch batches %% cum %% 1: 112 47 47 2: 25 10 58 4: 18 7 65 8: 14 5 71 16: 10 4 76 32: 17 7 83 64: 4 1 85 128: 0 0 85 256: 35 14 100 mdc.lustre-MDT0001-mdc-ffffa023b6b2a000.batch_stats= snapshot_time: 1788672219.067903210 secs.nsecs start_time: 1788672158.520471901 secs.nsecs elapsed_time: 60.547431309 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second seq.cli-lustre-OST0000-super.width=65536 seq.cli-lustre-OST0001-super.width=65536 llite.lustre-ffffa023b6b2a000.statahead_batch_max=64 llite.lustre-ffffa023b6b2a000.statahead_max=128