== sanity test 123ad: Verify batching statahead works correctly ========================================================== 01:53:19 (1789451599) batch: statahead_max=32 statahead_batch_max=32 mdc.lustre-MDT0000-mdc-ffff8a527f0cb800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff8a527f0cb800.batch_stats=0 llite.lustre-ffff8a527f0cb800.statahead_max=32 llite.lustre-ffff8a527f0cb800.statahead_batch_max=32 seq.cli-lustre-OST0000-super.width=0x1ffffff seq.cli-lustre-OST0001-super.width=0x1ffffff kvm mdc.lustre-MDT0000-mdc-ffff8a527f0cb800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff8a527f0cb800.batch_stats=0 striped dir -i1 -c2 -H crush2 /mnt/lustre/d123ad.sanity total: 100 open/close in 2.14 seconds: 46.75 ops/second llite.lustre-ffff8a527f0cb800.statahead_max=0 101 real 0m1.701s user 0m0.003s sys 0m0.275s 'ls -l' 100 files without statahead: 1 sec llite.lustre-ffff8a527f0cb800.statahead_max=32 llite.lustre-ffff8a527f0cb800.statahead_max=32 101 real 0m1.349s user 0m0.010s sys 0m0.208s 'ls -l' 100 files with statahead: 1 sec statahead total: 124 statahead wrong: 2 agl total: 118 list_total: 30 fname_total: 94 hit_total: 3788 miss_total: 1360 swrong=2 ewrong=2 total: 900 open/close in 9.66 seconds: 93.18 ops/second llite.lustre-ffff8a527f0cb800.statahead_max=0 1001 real 0m18.382s user 0m0.032s sys 0m2.846s 'ls -l' 1000 files without statahead: 19 sec llite.lustre-ffff8a527f0cb800.statahead_max=32 llite.lustre-ffff8a527f0cb800.statahead_max=32 1001 real 0m8.220s user 0m0.021s sys 0m1.264s 'ls -l' 1000 files with statahead: 8 sec statahead total: 125 statahead wrong: 2 agl total: 119 list_total: 31 fname_total: 94 hit_total: 4784 miss_total: 1364 swrong=2 ewrong=2 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 17 seconds llite.lustre-ffff8a527f0cb800.statahead_stats= statahead total: 125 statahead wrong: 2 agl total: 119 list_total: 31 fname_total: 94 hit_total: 4784 miss_total: 1364 mdc.lustre-MDT0000-mdc-ffff8a527f0cb800.batch_stats= snapshot_time: 1789451680.351269186 secs.nsecs start_time: 1789451604.612416362 secs.nsecs elapsed_time: 75.738852824 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 2 5 5 4: 1 2 8 8: 3 8 16 16: 26 72 88 32: 4 11 100 mdc.lustre-MDT0001-mdc-ffff8a527f0cb800.batch_stats= snapshot_time: 1789451680.351335979 secs.nsecs start_time: 1789451604.612503837 secs.nsecs elapsed_time: 75.738832142 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 1 2 2 4: 2 5 7 8: 3 7 15 16: 10 25 40 32: 24 60 100 - open/close 701 (time 1789451693.02 total 10.02 last 69.98) - open/close 1197 (time 1789451703.02 total 20.02 last 49.59) - open/close 1854 (time 1789451713.04 total 30.04 last 65.56) - open/close 2714 (time 1789451723.04 total 40.04 last 86.00) - open/close 3593 (time 1789451733.04 total 50.05 last 87.86) - open/close 4383 (time 1789451743.05 total 60.05 last 78.95) - open/close 5092 (time 1789451753.06 total 70.06 last 70.86) - open/close 6048 (time 1789451763.07 total 80.08 last 95.42) - open/close 6757 (time 1789451773.09 total 90.09 last 70.79) - open/close 7584 (time 1789451783.10 total 100.10 last 82.65) - open/close 8336 (time 1789451793.10 total 110.10 last 75.20) - open/close 9186 (time 1789451803.10 total 120.10 last 84.97) total: 10000 open/close in 128.98 seconds: 77.53 ops/second llite.lustre-ffff8a527f0cb800.statahead_batch_max=0 llite.lustre-ffff8a527f0cb800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff8a527f0cb800.stats=clear mdc.lustre-MDT0001-mdc-ffff8a527f0cb800.stats=clear ls -l files in d123ad.sanity: 10001 real 0m58.210s user 0m0.323s sys 0m13.638s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff8a527f0cb800.statahead_batch_max=32 llite.lustre-ffff8a527f0cb800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff8a527f0cb800.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff8a527f0cb800.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff8a527f0cb800.stats=clear mdc.lustre-MDT0001-mdc-ffff8a527f0cb800.stats=clear ls -l files in d123ad.sanity: 10001 real 1m45.142s user 0m0.305s sys 0m13.312s batched statahead RPC= hit count=9977 mdc.lustre-MDT0000-mdc-ffff8a527f0cb800.batch_stats= snapshot_time: 1789451994.432159162 secs.nsecs start_time: 1789451878.922661937 secs.nsecs elapsed_time: 115.509497225 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 1 0 0 8: 1 0 0 16: 0 0 0 32: 313 99 100 mdc.lustre-MDT0001-mdc-ffff8a527f0cb800.batch_stats= snapshot_time: 1789451994.432224427 secs.nsecs start_time: 1789451878.922745093 secs.nsecs elapsed_time: 115.509479334 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 batch: statahead_max=2048 statahead_batch_max=256 mdc.lustre-MDT0000-mdc-ffff8a527f0cb800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff8a527f0cb800.batch_stats=0 llite.lustre-ffff8a527f0cb800.statahead_max=2048 llite.lustre-ffff8a527f0cb800.statahead_batch_max=256 kvm mdc.lustre-MDT0000-mdc-ffff8a527f0cb800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff8a527f0cb800.batch_stats=0 striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d123ad.sanity total: 100 open/close in 2.26 seconds: 44.21 ops/second llite.lustre-ffff8a527f0cb800.statahead_max=0 101 real 0m1.911s user 0m0.004s sys 0m0.304s 'ls -l' 100 files without statahead: 2 sec llite.lustre-ffff8a527f0cb800.statahead_max=2048 llite.lustre-ffff8a527f0cb800.statahead_max=2048 101 real 0m0.670s user 0m0.002s sys 0m0.150s 'ls -l' 100 files with statahead: 1 sec statahead total: 2 statahead wrong: 0 agl total: 2 list_total: 2 fname_total: 0 hit_total: 10076 miss_total: 24 swrong=0 ewrong=0 - open/close 792 (time 1789452259.37 total 10.01 last 79.09) total: 900 open/close in 11.43 seconds: 78.75 ops/second llite.lustre-ffff8a527f0cb800.statahead_max=0 1001 real 0m21.370s user 0m0.033s sys 0m2.855s 'ls -l' 1000 files without statahead: 21 sec llite.lustre-ffff8a527f0cb800.statahead_max=2048 llite.lustre-ffff8a527f0cb800.statahead_max=2048 1001 real 0m6.171s user 0m0.028s sys 0m1.308s 'ls -l' 1000 files with statahead: 7 sec statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11075 miss_total: 25 swrong=0 ewrong=0 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 20 seconds llite.lustre-ffff8a527f0cb800.statahead_stats= statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11075 miss_total: 25 mdc.lustre-MDT0000-mdc-ffff8a527f0cb800.batch_stats= snapshot_time: 1789452320.035669451 secs.nsecs start_time: 1789451996.553472838 secs.nsecs elapsed_time: 323.482196613 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 2 28 28 8: 2 28 57 16: 0 0 57 32: 0 0 57 64: 1 14 71 128: 0 0 71 256: 2 28 100 mdc.lustre-MDT0001-mdc-ffff8a527f0cb800.batch_stats= snapshot_time: 1789452320.035741462 secs.nsecs start_time: 1789451996.553631560 secs.nsecs elapsed_time: 323.482109902 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 2 28 28 8: 2 28 57 16: 0 0 57 32: 0 0 57 64: 1 14 71 128: 0 0 71 256: 2 28 100 - open/close 865 (time 1789452332.41 total 10.00 last 86.47) - open/close 1734 (time 1789452342.42 total 20.01 last 86.83) - open/close 2509 (time 1789452352.43 total 30.02 last 77.41) - open/close 3243 (time 1789452362.43 total 40.03 last 73.38) - open/close 4042 (time 1789452372.45 total 50.04 last 79.81) - open/close 4874 (time 1789452382.45 total 60.05 last 83.13) - open/close 5703 (time 1789452392.48 total 70.07 last 82.69) - open/close 6506 (time 1789452402.48 total 80.08 last 80.27) - open/close 7348 (time 1789452412.49 total 90.08 last 84.15) - open/close 8277 (time 1789452422.50 total 100.09 last 92.82) - open/close 9163 (time 1789452432.50 total 110.10 last 88.54) - open/close 9987 (time 1789452442.52 total 120.11 last 82.31) total: 10000 open/close in 120.23 seconds: 83.17 ops/second llite.lustre-ffff8a527f0cb800.statahead_batch_max=0 llite.lustre-ffff8a527f0cb800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff8a527f0cb800.stats=clear mdc.lustre-MDT0001-mdc-ffff8a527f0cb800.stats=clear ls -l files in d123ad.sanity: 10001 real 1m2.232s user 0m0.333s sys 0m14.019s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff8a527f0cb800.statahead_batch_max=256 llite.lustre-ffff8a527f0cb800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff8a527f0cb800.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff8a527f0cb800.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff8a527f0cb800.stats=clear mdc.lustre-MDT0001-mdc-ffff8a527f0cb800.stats=clear ls -l files in d123ad.sanity: 10001 real 0m51.036s user 0m0.322s sys 0m13.774s batched statahead RPC= hit count=9999 mdc.lustre-MDT0000-mdc-ffff8a527f0cb800.batch_stats= snapshot_time: 1789452576.092401559 secs.nsecs start_time: 1789452514.074201172 secs.nsecs elapsed_time: 62.018200387 secs.nsecs subreqs per batch batches %% cum %% 1: 161 43 43 2: 57 15 58 4: 42 11 70 8: 37 10 80 16: 16 4 84 32: 16 4 88 64: 7 1 90 128: 1 0 91 256: 33 8 100 mdc.lustre-MDT0001-mdc-ffff8a527f0cb800.batch_stats= snapshot_time: 1789452576.092467559 secs.nsecs start_time: 1789452514.074274211 secs.nsecs elapsed_time: 62.018193348 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second seq.cli-lustre-OST0000-super.width=65536 seq.cli-lustre-OST0001-super.width=65536 llite.lustre-ffff8a527f0cb800.statahead_batch_max=64 llite.lustre-ffff8a527f0cb800.statahead_max=128