== sanity test 123ad: Verify batching statahead works correctly ========================================================== 02:05:37 (1788933937) batch: statahead_max=32 statahead_batch_max=32 mdc.lustre-MDT0000-mdc-ffff8a768577e800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff8a768577e800.batch_stats=0 llite.lustre-ffff8a768577e800.statahead_max=32 llite.lustre-ffff8a768577e800.statahead_batch_max=32 seq.cli-lustre-OST0000-super.width=0x1ffffff seq.cli-lustre-OST0001-super.width=0x1ffffff kvm mdc.lustre-MDT0000-mdc-ffff8a768577e800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff8a768577e800.batch_stats=0 striped dir -i1 -c2 -H crush /mnt/lustre/d123ad.sanity total: 100 open/close in 2.81 seconds: 35.62 ops/second llite.lustre-ffff8a768577e800.statahead_max=0 101 real 0m2.815s user 0m0.004s sys 0m0.320s 'ls -l' 100 files without statahead: 3 sec llite.lustre-ffff8a768577e800.statahead_max=32 llite.lustre-ffff8a768577e800.statahead_max=32 101 real 0m1.344s user 0m0.003s sys 0m0.172s 'ls -l' 100 files with statahead: 2 sec statahead total: 155 statahead wrong: 2 agl total: 148 list_total: 38 fname_total: 117 hit_total: 4430 miss_total: 778 swrong=2 ewrong=2 - open/close 689 (time 1788933968.96 total 10.00 last 68.88) total: 900 open/close in 12.59 seconds: 71.50 ops/second llite.lustre-ffff8a768577e800.statahead_max=0 1001 real 0m21.304s user 0m0.034s sys 0m2.920s 'ls -l' 1000 files without statahead: 21 sec llite.lustre-ffff8a768577e800.statahead_max=32 llite.lustre-ffff8a768577e800.statahead_max=32 1001 real 0m8.651s user 0m0.032s sys 0m1.361s 'ls -l' 1000 files with statahead: 8 sec statahead total: 156 statahead wrong: 2 agl total: 149 list_total: 39 fname_total: 117 hit_total: 5429 miss_total: 779 swrong=2 ewrong=2 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 22 seconds llite.lustre-ffff8a768577e800.statahead_stats= statahead total: 156 statahead wrong: 2 agl total: 149 list_total: 39 fname_total: 117 hit_total: 5429 miss_total: 779 mdc.lustre-MDT0000-mdc-ffff8a768577e800.batch_stats= snapshot_time: 1788934036.167125387 secs.nsecs start_time: 1788933943.324350793 secs.nsecs elapsed_time: 92.842774594 secs.nsecs subreqs per batch batches %% cum %% 1: 3 8 8 2: 2 5 13 4: 0 0 13 8: 0 0 13 16: 25 69 83 32: 6 16 100 mdc.lustre-MDT0001-mdc-ffff8a768577e800.batch_stats= snapshot_time: 1788934036.167190246 secs.nsecs start_time: 1788933943.324443881 secs.nsecs elapsed_time: 92.842746365 secs.nsecs subreqs per batch batches %% cum %% 1: 1 2 2 2: 0 0 2 4: 1 2 5 8: 3 7 12 16: 12 30 43 32: 22 56 100 - open/close 758 (time 1788934049.03 total 10.01 last 75.72) - open/close 1499 (time 1788934059.04 total 20.02 last 74.04) - open/close 2303 (time 1788934069.05 total 30.03 last 80.31) - open/close 3141 (time 1788934079.06 total 40.04 last 83.75) - open/close 3990 (time 1788934089.06 total 50.04 last 84.87) - open/close 4844 (time 1788934099.07 total 60.04 last 85.36) - open/close 5705 (time 1788934109.07 total 70.05 last 86.05) - open/close 6585 (time 1788934119.08 total 80.06 last 87.95) - open/close 7423 (time 1788934129.08 total 90.06 last 83.75) - open/close 8216 (time 1788934139.09 total 100.07 last 79.23) - open/close 8970 (time 1788934149.10 total 110.07 last 75.37) - open/close 9696 (time 1788934159.10 total 120.08 last 72.59) total: 10000 open/close in 123.49 seconds: 80.98 ops/second llite.lustre-ffff8a768577e800.statahead_batch_max=0 llite.lustre-ffff8a768577e800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff8a768577e800.stats=clear mdc.lustre-MDT0001-mdc-ffff8a768577e800.stats=clear ls -l files in d123ad.sanity: 10001 real 1m2.631s user 0m0.317s sys 0m14.337s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff8a768577e800.statahead_batch_max=32 llite.lustre-ffff8a768577e800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff8a768577e800.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff8a768577e800.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff8a768577e800.stats=clear mdc.lustre-MDT0001-mdc-ffff8a768577e800.stats=clear ls -l files in d123ad.sanity: 10001 real 1m45.165s user 0m0.309s sys 0m13.895s batched statahead RPC= hit count=9985 mdc.lustre-MDT0000-mdc-ffff8a768577e800.batch_stats= snapshot_time: 1788934350.811816961 secs.nsecs start_time: 1788934235.620036776 secs.nsecs elapsed_time: 115.191780185 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 0 0 0 8: 1 0 0 16: 1 0 0 32: 313 99 100 mdc.lustre-MDT0001-mdc-ffff8a768577e800.batch_stats= snapshot_time: 1788934350.811972538 secs.nsecs start_time: 1788934235.620143435 secs.nsecs elapsed_time: 115.191829103 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 batch: statahead_max=2048 statahead_batch_max=256 mdc.lustre-MDT0000-mdc-ffff8a768577e800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff8a768577e800.batch_stats=0 llite.lustre-ffff8a768577e800.statahead_max=2048 llite.lustre-ffff8a768577e800.statahead_batch_max=256 kvm mdc.lustre-MDT0000-mdc-ffff8a768577e800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff8a768577e800.batch_stats=0 striped dir -i1 -c2 -H crush2 /mnt/lustre/d123ad.sanity total: 100 open/close in 2.77 seconds: 36.13 ops/second llite.lustre-ffff8a768577e800.statahead_max=0 101 real 0m2.196s user 0m0.005s sys 0m0.323s 'ls -l' 100 files without statahead: 2 sec llite.lustre-ffff8a768577e800.statahead_max=2048 llite.lustre-ffff8a768577e800.statahead_max=2048 101 real 0m1.348s user 0m0.006s sys 0m0.190s 'ls -l' 100 files with statahead: 1 sec statahead total: 2 statahead wrong: 0 agl total: 2 list_total: 2 fname_total: 0 hit_total: 10084 miss_total: 16 swrong=0 ewrong=0 - open/close 875 (time 1788934652.67 total 10.00 last 87.47) total: 900 open/close in 10.32 seconds: 87.25 ops/second llite.lustre-ffff8a768577e800.statahead_max=0 1001 real 0m20.228s user 0m0.039s sys 0m2.902s 'ls -l' 1000 files without statahead: 20 sec llite.lustre-ffff8a768577e800.statahead_max=2048 llite.lustre-ffff8a768577e800.statahead_max=2048 1001 real 0m7.464s user 0m0.036s sys 0m1.470s 'ls -l' 1000 files with statahead: 7 sec statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11083 miss_total: 17 swrong=0 ewrong=0 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 18 seconds llite.lustre-ffff8a768577e800.statahead_stats= statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11083 miss_total: 17 mdc.lustre-MDT0000-mdc-ffff8a768577e800.batch_stats= snapshot_time: 1788934711.517007103 secs.nsecs start_time: 1788934354.932426273 secs.nsecs elapsed_time: 356.584580830 secs.nsecs subreqs per batch batches %% cum %% 1: 3 42 42 2: 0 0 42 4: 1 14 57 8: 0 0 57 16: 1 14 71 32: 0 0 71 64: 0 0 71 128: 0 0 71 256: 2 28 100 mdc.lustre-MDT0001-mdc-ffff8a768577e800.batch_stats= snapshot_time: 1788934711.517078120 secs.nsecs start_time: 1788934354.932583863 secs.nsecs elapsed_time: 356.584494257 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 0 0 0 8: 5 55 55 16: 0 0 55 32: 1 11 66 64: 0 0 66 128: 1 11 77 256: 2 22 100 - open/close 814 (time 1788934724.14 total 10.01 last 81.30) - open/close 1564 (time 1788934734.14 total 20.01 last 74.98) - open/close 2329 (time 1788934744.14 total 30.02 last 76.47) - open/close 3128 (time 1788934754.15 total 40.02 last 79.88) - open/close 4013 (time 1788934764.16 total 50.03 last 88.42) - open/close 4834 (time 1788934774.16 total 60.03 last 82.05) - open/close 5598 (time 1788934784.16 total 70.04 last 76.38) - open/close 6461 (time 1788934794.17 total 80.04 last 86.27) - open/close 7365 (time 1788934804.17 total 90.04 last 90.38) - open/close 8206 (time 1788934814.18 total 100.05 last 84.06) - open/close 8970 (time 1788934824.20 total 110.08 last 76.20) - open/close 9826 (time 1788934834.21 total 120.08 last 85.53) total: 10000 open/close in 121.74 seconds: 82.14 ops/second llite.lustre-ffff8a768577e800.statahead_batch_max=0 llite.lustre-ffff8a768577e800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff8a768577e800.stats=clear mdc.lustre-MDT0001-mdc-ffff8a768577e800.stats=clear ls -l files in d123ad.sanity: 10001 real 1m8.097s user 0m0.379s sys 0m14.907s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff8a768577e800.statahead_batch_max=256 llite.lustre-ffff8a768577e800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff8a768577e800.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff8a768577e800.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff8a768577e800.stats=clear mdc.lustre-MDT0001-mdc-ffff8a768577e800.stats=clear ls -l files in d123ad.sanity: 10001 real 0m57.336s user 0m0.315s sys 0m14.172s batched statahead RPC= hit count=9999 mdc.lustre-MDT0000-mdc-ffff8a768577e800.batch_stats= snapshot_time: 1788934983.230282370 secs.nsecs start_time: 1788934913.170914571 secs.nsecs elapsed_time: 70.059367799 secs.nsecs subreqs per batch batches %% cum %% 1: 140 46 46 2: 44 14 61 4: 32 10 72 8: 15 5 77 16: 12 4 81 32: 15 5 86 64: 8 2 88 128: 0 0 88 256: 34 11 100 mdc.lustre-MDT0001-mdc-ffff8a768577e800.batch_stats= snapshot_time: 1788934983.230353890 secs.nsecs start_time: 1788934913.171020988 secs.nsecs elapsed_time: 70.059332902 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second seq.cli-lustre-OST0000-super.width=65536 seq.cli-lustre-OST0001-super.width=65536 llite.lustre-ffff8a768577e800.statahead_batch_max=64 llite.lustre-ffff8a768577e800.statahead_max=128