== sanity test 123ad: Verify batching statahead works correctly ========================================================== 05:59:42 (1789293582) batch: statahead_max=32 statahead_batch_max=32 mdc.lustre-MDT0000-mdc-ffff8a48901b7800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff8a48901b7800.batch_stats=0 llite.lustre-ffff8a48901b7800.statahead_max=32 llite.lustre-ffff8a48901b7800.statahead_batch_max=32 seq.cli-lustre-OST0000-super.width=0x1ffffff seq.cli-lustre-OST0001-super.width=0x1ffffff kvm mdc.lustre-MDT0000-mdc-ffff8a48901b7800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff8a48901b7800.batch_stats=0 striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d123ad.sanity total: 100 open/close in 3.11 seconds: 32.11 ops/second llite.lustre-ffff8a48901b7800.statahead_max=0 101 real 0m3.094s user 0m0.006s sys 0m0.386s 'ls -l' 100 files without statahead: 3 sec llite.lustre-ffff8a48901b7800.statahead_max=32 llite.lustre-ffff8a48901b7800.statahead_max=32 101 real 0m1.543s user 0m0.004s sys 0m0.223s 'ls -l' 100 files with statahead: 1 sec statahead total: 111 statahead wrong: 2 agl total: 105 list_total: 30 fname_total: 75 hit_total: 3995 miss_total: 1700 swrong=2 ewrong=2 - open/close 694 (time 1789293613.74 total 10.02 last 69.24) total: 900 open/close in 12.92 seconds: 69.65 ops/second llite.lustre-ffff8a48901b7800.statahead_max=0 1001 real 0m25.910s user 0m0.045s sys 0m3.085s 'ls -l' 1000 files without statahead: 26 sec llite.lustre-ffff8a48901b7800.statahead_max=32 llite.lustre-ffff8a48901b7800.statahead_max=32 1001 real 0m8.507s user 0m0.030s sys 0m1.397s 'ls -l' 1000 files with statahead: 9 sec statahead total: 112 statahead wrong: 2 agl total: 106 list_total: 31 fname_total: 75 hit_total: 4993 miss_total: 1702 swrong=2 ewrong=2 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 22 seconds llite.lustre-ffff8a48901b7800.statahead_stats= statahead total: 112 statahead wrong: 2 agl total: 106 list_total: 31 fname_total: 75 hit_total: 4993 miss_total: 1702 mdc.lustre-MDT0000-mdc-ffff8a48901b7800.batch_stats= snapshot_time: 1789293686.751182795 secs.nsecs start_time: 1789293587.664977794 secs.nsecs elapsed_time: 99.086205001 secs.nsecs subreqs per batch batches %% cum %% 1: 3 7 7 2: 0 0 7 4: 1 2 9 8: 4 9 19 16: 21 51 70 32: 12 29 100 mdc.lustre-MDT0001-mdc-ffff8a48901b7800.batch_stats= snapshot_time: 1789293686.751268692 secs.nsecs start_time: 1789293587.665160493 secs.nsecs elapsed_time: 99.086108199 secs.nsecs subreqs per batch batches %% cum %% 1: 1 2 2 2: 2 5 7 4: 2 5 12 8: 1 2 15 16: 19 47 62 32: 15 37 100 - open/close 721 (time 1789293699.15 total 10.01 last 72.03) - open/close 1414 (time 1789293709.15 total 20.01 last 69.29) - open/close 2148 (time 1789293719.15 total 30.01 last 73.38) - open/close 2859 (time 1789293729.17 total 40.03 last 70.98) - open/close 3520 (time 1789293739.17 total 50.03 last 66.09) - open/close 4159 (time 1789293749.18 total 60.04 last 63.85) - open/close 4775 (time 1789293759.21 total 70.07 last 61.39) - open/close 5415 (time 1789293769.22 total 80.08 last 63.99) - open/close 6041 (time 1789293779.22 total 90.08 last 62.56) - open/close 6749 (time 1789293789.23 total 100.09 last 70.76) - open/close 7361 (time 1789293799.23 total 110.09 last 61.19) - open/close 7980 (time 1789293809.25 total 120.11 last 61.76) - open/close 8565 (time 1789293819.27 total 130.12 last 58.42) - open/close 9241 (time 1789293829.28 total 140.14 last 67.52) - open/close 9862 (time 1789293839.28 total 150.14 last 62.06) total: 10000 open/close in 152.31 seconds: 65.66 ops/second llite.lustre-ffff8a48901b7800.statahead_batch_max=0 llite.lustre-ffff8a48901b7800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff8a48901b7800.stats=clear mdc.lustre-MDT0001-mdc-ffff8a48901b7800.stats=clear ls -l files in d123ad.sanity: 10001 real 1m5.752s user 0m0.331s sys 0m15.774s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff8a48901b7800.statahead_batch_max=32 llite.lustre-ffff8a48901b7800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff8a48901b7800.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff8a48901b7800.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff8a48901b7800.stats=clear mdc.lustre-MDT0001-mdc-ffff8a48901b7800.stats=clear ls -l files in d123ad.sanity: 10001 real 1m34.025s user 0m0.326s sys 0m14.926s batched statahead RPC= hit count=9989 mdc.lustre-MDT0000-mdc-ffff8a48901b7800.batch_stats= snapshot_time: 1789294018.357541225 secs.nsecs start_time: 1789293914.637171979 secs.nsecs elapsed_time: 103.720369246 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 0 0 0 8: 1 0 0 16: 1 0 0 32: 313 99 100 mdc.lustre-MDT0001-mdc-ffff8a48901b7800.batch_stats= snapshot_time: 1789294018.358251771 secs.nsecs start_time: 1789293914.637346587 secs.nsecs elapsed_time: 103.720905184 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 batch: statahead_max=2048 statahead_batch_max=256 mdc.lustre-MDT0000-mdc-ffff8a48901b7800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff8a48901b7800.batch_stats=0 llite.lustre-ffff8a48901b7800.statahead_max=2048 llite.lustre-ffff8a48901b7800.statahead_batch_max=256 kvm mdc.lustre-MDT0000-mdc-ffff8a48901b7800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff8a48901b7800.batch_stats=0 striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d123ad.sanity total: 100 open/close in 3.12 seconds: 32.02 ops/second llite.lustre-ffff8a48901b7800.statahead_max=0 101 real 0m2.738s user 0m0.009s sys 0m0.331s 'ls -l' 100 files without statahead: 3 sec llite.lustre-ffff8a48901b7800.statahead_max=2048 llite.lustre-ffff8a48901b7800.statahead_max=2048 101 real 0m0.898s user 0m0.003s sys 0m0.184s 'ls -l' 100 files with statahead: 1 sec statahead total: 2 statahead wrong: 0 agl total: 2 list_total: 2 fname_total: 0 hit_total: 10088 miss_total: 12 swrong=0 ewrong=0 - open/close 697 (time 1789294369.12 total 10.01 last 69.62) total: 900 open/close in 13.92 seconds: 64.67 ops/second llite.lustre-ffff8a48901b7800.statahead_max=0 1001 real 0m26.850s user 0m0.038s sys 0m2.974s 'ls -l' 1000 files without statahead: 27 sec llite.lustre-ffff8a48901b7800.statahead_max=2048 llite.lustre-ffff8a48901b7800.statahead_max=2048 1001 real 0m6.867s user 0m0.031s sys 0m1.422s 'ls -l' 1000 files with statahead: 7 sec statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11087 miss_total: 13 swrong=0 ewrong=0 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 19 seconds llite.lustre-ffff8a48901b7800.statahead_stats= statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11087 miss_total: 13 mdc.lustre-MDT0000-mdc-ffff8a48901b7800.batch_stats= snapshot_time: 1789294439.307036356 secs.nsecs start_time: 1789294021.031420892 secs.nsecs elapsed_time: 418.275615464 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 1 14 14 8: 3 42 57 16: 0 0 57 32: 0 0 57 64: 1 14 71 128: 0 0 71 256: 2 28 100 mdc.lustre-MDT0001-mdc-ffff8a48901b7800.batch_stats= snapshot_time: 1789294439.307105903 secs.nsecs start_time: 1789294021.031577401 secs.nsecs elapsed_time: 418.275528502 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 1 14 14 4: 2 28 42 8: 1 14 57 16: 0 0 57 32: 0 0 57 64: 1 14 71 128: 0 0 71 256: 2 28 100 - open/close 721 (time 1789294451.50 total 10.01 last 72.05) - open/close 1441 (time 1789294461.50 total 20.01 last 71.99) - open/close 2168 (time 1789294471.50 total 30.01 last 72.68) - open/close 2884 (time 1789294481.52 total 40.03 last 71.46) - open/close 3591 (time 1789294491.54 total 50.05 last 70.55) - open/close 4289 (time 1789294501.55 total 60.06 last 69.73) - open/close 5000 (time 1789294511.57 total 70.07 last 71.02) - open/close 5676 (time 1789294521.57 total 80.08 last 67.58) - open/close 6363 (time 1789294531.58 total 90.09 last 68.60) - open/close 7062 (time 1789294541.58 total 100.09 last 69.90) - open/close 7836 (time 1789294551.58 total 110.09 last 77.40) - open/close 8527 (time 1789294561.59 total 120.10 last 69.04) - open/close 9252 (time 1789294571.59 total 130.10 last 72.50) - open/close 9961 (time 1789294581.60 total 140.11 last 70.85) total: 10000 open/close in 140.60 seconds: 71.12 ops/second llite.lustre-ffff8a48901b7800.statahead_batch_max=0 llite.lustre-ffff8a48901b7800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff8a48901b7800.stats=clear mdc.lustre-MDT0001-mdc-ffff8a48901b7800.stats=clear ls -l files in d123ad.sanity: 10001 real 0m48.966s user 0m0.315s sys 0m17.593s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff8a48901b7800.statahead_batch_max=256 llite.lustre-ffff8a48901b7800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff8a48901b7800.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff8a48901b7800.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff8a48901b7800.stats=clear mdc.lustre-MDT0001-mdc-ffff8a48901b7800.stats=clear ls -l files in d123ad.sanity: 10001 real 0m44.721s user 0m0.319s sys 0m16.804s batched statahead RPC= hit count=9999 mdc.lustre-MDT0000-mdc-ffff8a48901b7800.batch_stats= snapshot_time: 1789294692.888104676 secs.nsecs start_time: 1789294638.544054568 secs.nsecs elapsed_time: 54.344050108 secs.nsecs subreqs per batch batches %% cum %% 1: 169 46 46 2: 50 13 60 4: 49 13 73 8: 30 8 81 16: 8 2 84 32: 14 3 87 64: 10 2 90 128: 1 0 90 256: 33 9 100 mdc.lustre-MDT0001-mdc-ffff8a48901b7800.batch_stats= snapshot_time: 1789294692.888304948 secs.nsecs start_time: 1789294638.544151097 secs.nsecs elapsed_time: 54.344153851 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second seq.cli-lustre-OST0000-super.width=65536 seq.cli-lustre-OST0001-super.width=65536 llite.lustre-ffff8a48901b7800.statahead_batch_max=64 llite.lustre-ffff8a48901b7800.statahead_max=128