== sanity test 123ad: Verify batching statahead works correctly ========================================================== 21:55:22 (1789523722) batch: statahead_max=32 statahead_batch_max=32 mdc.lustre-MDT0000-mdc-ffff9f51e2d9e800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9f51e2d9e800.batch_stats=0 llite.lustre-ffff9f51e2d9e800.statahead_max=32 llite.lustre-ffff9f51e2d9e800.statahead_batch_max=32 seq.cli-lustre-OST0000-super.width=0x1ffffff seq.cli-lustre-OST0001-super.width=0x1ffffff kvm mdc.lustre-MDT0000-mdc-ffff9f51e2d9e800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9f51e2d9e800.batch_stats=0 striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d123ad.sanity total: 100 open/close in 2.04 seconds: 48.94 ops/second llite.lustre-ffff9f51e2d9e800.statahead_max=0 101 real 0m2.067s user 0m0.003s sys 0m0.312s 'ls -l' 100 files without statahead: 2 sec llite.lustre-ffff9f51e2d9e800.statahead_max=32 llite.lustre-ffff9f51e2d9e800.statahead_max=32 101 real 0m0.815s user 0m0.006s sys 0m0.196s 'ls -l' 100 files with statahead: 1 sec statahead total: 128 statahead wrong: 2 agl total: 122 list_total: 31 fname_total: 97 hit_total: 3573 miss_total: 1429 swrong=2 ewrong=2 total: 900 open/close in 9.98 seconds: 90.16 ops/second llite.lustre-ffff9f51e2d9e800.statahead_max=0 1001 real 0m20.066s user 0m0.032s sys 0m3.147s 'ls -l' 1000 files without statahead: 20 sec llite.lustre-ffff9f51e2d9e800.statahead_max=32 llite.lustre-ffff9f51e2d9e800.statahead_max=32 1001 real 0m6.084s user 0m0.022s sys 0m1.490s 'ls -l' 1000 files with statahead: 6 sec statahead total: 129 statahead wrong: 2 agl total: 123 list_total: 32 fname_total: 97 hit_total: 4569 miss_total: 1433 swrong=2 ewrong=2 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 13 seconds llite.lustre-ffff9f51e2d9e800.statahead_stats= statahead total: 129 statahead wrong: 2 agl total: 123 list_total: 32 fname_total: 97 hit_total: 4569 miss_total: 1433 mdc.lustre-MDT0000-mdc-ffff9f51e2d9e800.batch_stats= snapshot_time: 1789523793.287298644 secs.nsecs start_time: 1789523725.834579275 secs.nsecs elapsed_time: 67.452719369 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 2 5 5 4: 0 0 5 8: 3 7 13 16: 19 50 63 32: 14 36 100 mdc.lustre-MDT0001-mdc-ffff9f51e2d9e800.batch_stats= snapshot_time: 1789523793.287545485 secs.nsecs start_time: 1789523725.834713677 secs.nsecs elapsed_time: 67.452831808 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 1 2 2 4: 1 2 5 8: 4 10 15 16: 21 53 69 32: 12 30 100 - open/close 825 (time 1789523805.31 total 10.01 last 82.40) - open/close 1620 (time 1789523815.32 total 20.02 last 79.44) - open/close 2593 (time 1789523825.32 total 30.03 last 97.24) - open/close 3437 (time 1789523835.33 total 40.03 last 84.35) - open/close 4335 (time 1789523845.33 total 50.03 last 89.78) - open/close 5239 (time 1789523855.34 total 60.04 last 90.36) - open/close 6065 (time 1789523865.34 total 70.04 last 82.55) - open/close 6896 (time 1789523875.35 total 80.05 last 83.01) - open/close 7725 (time 1789523885.35 total 90.06 last 82.88) - open/close 8733 (time 1789523895.36 total 100.06 last 100.75) - open/close 9728 (time 1789523905.36 total 110.07 last 99.46) total: 10000 open/close in 112.96 seconds: 88.53 ops/second llite.lustre-ffff9f51e2d9e800.statahead_batch_max=0 llite.lustre-ffff9f51e2d9e800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9f51e2d9e800.stats=clear mdc.lustre-MDT0001-mdc-ffff9f51e2d9e800.stats=clear ls -l files in d123ad.sanity: 10001 real 0m39.495s user 0m0.304s sys 0m22.204s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff9f51e2d9e800.statahead_batch_max=32 llite.lustre-ffff9f51e2d9e800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9f51e2d9e800.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff9f51e2d9e800.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff9f51e2d9e800.stats=clear mdc.lustre-MDT0001-mdc-ffff9f51e2d9e800.stats=clear ls -l files in d123ad.sanity: 10001 real 1m0.423s user 0m0.249s sys 0m16.279s batched statahead RPC= hit count=9990 mdc.lustre-MDT0000-mdc-ffff9f51e2d9e800.batch_stats= snapshot_time: 1789524020.732502384 secs.nsecs start_time: 1789523952.701558065 secs.nsecs elapsed_time: 68.030944319 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 0 0 0 8: 1 0 0 16: 1 0 0 32: 313 99 100 mdc.lustre-MDT0001-mdc-ffff9f51e2d9e800.batch_stats= snapshot_time: 1789524020.732577367 secs.nsecs start_time: 1789523952.701650170 secs.nsecs elapsed_time: 68.030927197 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 batch: statahead_max=2048 statahead_batch_max=256 mdc.lustre-MDT0000-mdc-ffff9f51e2d9e800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9f51e2d9e800.batch_stats=0 llite.lustre-ffff9f51e2d9e800.statahead_max=2048 llite.lustre-ffff9f51e2d9e800.statahead_batch_max=256 kvm mdc.lustre-MDT0000-mdc-ffff9f51e2d9e800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9f51e2d9e800.batch_stats=0 striped dir -i1 -c2 -H crush2 /mnt/lustre/d123ad.sanity total: 100 open/close in 1.83 seconds: 54.61 ops/second llite.lustre-ffff9f51e2d9e800.statahead_max=0 101 real 0m1.628s user 0m0.007s sys 0m0.319s 'ls -l' 100 files without statahead: 2 sec llite.lustre-ffff9f51e2d9e800.statahead_max=2048 llite.lustre-ffff9f51e2d9e800.statahead_max=2048 101 real 0m0.737s user 0m0.004s sys 0m0.200s 'ls -l' 100 files with statahead: 1 sec statahead total: 2 statahead wrong: 0 agl total: 2 list_total: 2 fname_total: 0 hit_total: 10089 miss_total: 11 swrong=0 ewrong=0 total: 900 open/close in 8.27 seconds: 108.84 ops/second llite.lustre-ffff9f51e2d9e800.statahead_max=0 1001 real 0m16.673s user 0m0.028s sys 0m2.913s 'ls -l' 1000 files without statahead: 17 sec llite.lustre-ffff9f51e2d9e800.statahead_max=2048 llite.lustre-ffff9f51e2d9e800.statahead_max=2048 1001 real 0m4.394s user 0m0.024s sys 0m1.425s 'ls -l' 1000 files with statahead: 4 sec statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11088 miss_total: 12 swrong=0 ewrong=0 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 12 seconds llite.lustre-ffff9f51e2d9e800.statahead_stats= statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11088 miss_total: 12 mdc.lustre-MDT0000-mdc-ffff9f51e2d9e800.batch_stats= snapshot_time: 1789524276.971540559 secs.nsecs start_time: 1789524022.223881125 secs.nsecs elapsed_time: 254.747659434 secs.nsecs subreqs per batch batches %% cum %% 1: 1 16 16 2: 1 16 33 4: 2 33 66 8: 0 0 66 16: 0 0 66 32: 0 0 66 64: 0 0 66 128: 0 0 66 256: 2 33 100 mdc.lustre-MDT0001-mdc-ffff9f51e2d9e800.batch_stats= snapshot_time: 1789524276.971607624 secs.nsecs start_time: 1789524022.224169579 secs.nsecs elapsed_time: 254.747438045 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 0 0 0 8: 4 50 50 16: 0 0 50 32: 0 0 50 64: 1 12 62 128: 1 12 75 256: 2 25 100 - open/close 1274 (time 1789524288.48 total 10.00 last 127.36) - open/close 2609 (time 1789524298.48 total 20.01 last 133.45) - open/close 4036 (time 1789524308.48 total 30.01 last 142.67) - open/close 5484 (time 1789524318.49 total 40.01 last 144.78) - open/close 6812 (time 1789524328.49 total 50.01 last 132.74) - open/close 8204 (time 1789524338.49 total 60.02 last 139.17) - open/close 9527 (time 1789524348.50 total 70.02 last 132.23) total: 10000 open/close in 73.89 seconds: 135.34 ops/second llite.lustre-ffff9f51e2d9e800.statahead_batch_max=0 llite.lustre-ffff9f51e2d9e800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9f51e2d9e800.stats=clear mdc.lustre-MDT0001-mdc-ffff9f51e2d9e800.stats=clear ls -l files in d123ad.sanity: 10001 real 0m34.846s user 0m0.265s sys 0m21.220s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff9f51e2d9e800.statahead_batch_max=256 llite.lustre-ffff9f51e2d9e800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9f51e2d9e800.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff9f51e2d9e800.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff9f51e2d9e800.stats=clear mdc.lustre-MDT0001-mdc-ffff9f51e2d9e800.stats=clear ls -l files in d123ad.sanity: 10001 real 0m27.990s user 0m0.201s sys 0m18.601s batched statahead RPC= hit count=9999 mdc.lustre-MDT0000-mdc-ffff9f51e2d9e800.batch_stats= snapshot_time: 1789524425.936837279 secs.nsecs start_time: 1789524391.990904213 secs.nsecs elapsed_time: 33.945933066 secs.nsecs subreqs per batch batches %% cum %% 1: 240 59 59 2: 62 15 74 4: 36 8 83 8: 4 0 84 16: 4 0 85 32: 14 3 88 64: 11 2 91 128: 0 0 91 256: 34 8 100 mdc.lustre-MDT0001-mdc-ffff9f51e2d9e800.batch_stats= snapshot_time: 1789524425.936904816 secs.nsecs start_time: 1789524391.991051601 secs.nsecs elapsed_time: 33.945853215 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second seq.cli-lustre-OST0000-super.width=65536 seq.cli-lustre-OST0001-super.width=65536 llite.lustre-ffff9f51e2d9e800.statahead_batch_max=64 llite.lustre-ffff9f51e2d9e800.statahead_max=128