== sanity test 123ad: Verify batching statahead works correctly ========================================================== 14:38:26 (1783967906) batch: statahead_max=32 statahead_batch_max=32 mdc.lustre-MDT0000-mdc-ffff8d668b1ad000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff8d668b1ad000.batch_stats=0 llite.lustre-ffff8d668b1ad000.statahead_max=32 llite.lustre-ffff8d668b1ad000.statahead_batch_max=32 seq.cli-lustre-OST0000-super.width=0x1ffffff seq.cli-lustre-OST0001-super.width=0x1ffffff kvm mdc.lustre-MDT0000-mdc-ffff8d668b1ad000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff8d668b1ad000.batch_stats=0 striped dir -i1 -c2 -H crush /mnt/lustre/d123ad.sanity total: 100 open/close in 1.92 seconds: 52.06 ops/second llite.lustre-ffff8d668b1ad000.statahead_max=0 101 real 0m2.571s user 0m0.006s sys 0m0.334s 'ls -l' 100 files without statahead: 3 sec llite.lustre-ffff8d668b1ad000.statahead_max=32 llite.lustre-ffff8d668b1ad000.statahead_max=32 101 real 0m1.130s user 0m0.001s sys 0m0.186s 'ls -l' 100 files with statahead: 1 sec statahead total: 115 statahead wrong: 2 agl total: 109 list_total: 31 fname_total: 83 hit_total: 3860 miss_total: 1588 swrong=2 ewrong=2 total: 900 open/close in 9.21 seconds: 97.69 ops/second llite.lustre-ffff8d668b1ad000.statahead_max=0 1001 real 0m19.826s user 0m0.031s sys 0m3.086s 'ls -l' 1000 files without statahead: 20 sec llite.lustre-ffff8d668b1ad000.statahead_max=32 llite.lustre-ffff8d668b1ad000.statahead_max=32 1001 real 0m10.335s user 0m0.034s sys 0m1.519s 'ls -l' 1000 files with statahead: 11 sec statahead total: 116 statahead wrong: 2 agl total: 110 list_total: 32 fname_total: 83 hit_total: 4855 miss_total: 1593 swrong=2 ewrong=2 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 19 seconds llite.lustre-ffff8d668b1ad000.statahead_stats= statahead total: 116 statahead wrong: 2 agl total: 110 list_total: 32 fname_total: 83 hit_total: 4855 miss_total: 1593 mdc.lustre-MDT0000-mdc-ffff8d668b1ad000.batch_stats= snapshot_time: 1783967994.474014494 secs.nsecs start_time: 1783967911.034204310 secs.nsecs elapsed_time: 83.439810184 secs.nsecs subreqs per batch batches %% cum %% 1: 3 8 8 2: 2 5 13 4: 1 2 16 8: 1 2 18 16: 25 67 86 32: 5 13 100 mdc.lustre-MDT0001-mdc-ffff8d668b1ad000.batch_stats= snapshot_time: 1783967994.474073537 secs.nsecs start_time: 1783967911.034360256 secs.nsecs elapsed_time: 83.439713281 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 0 0 0 8: 5 12 12 16: 11 28 41 32: 23 58 100 - open/close 830 (time 1783968007.90 total 10.01 last 82.92) - open/close 1668 (time 1783968017.91 total 20.01 last 83.76) - open/close 2557 (time 1783968027.91 total 30.02 last 88.88) - open/close 3488 (time 1783968037.91 total 40.02 last 93.10) - open/close 4511 (time 1783968047.91 total 50.02 last 102.29) - open/close 5427 (time 1783968057.93 total 60.03 last 91.45) - open/close 6292 (time 1783968067.93 total 70.03 last 86.50) - open/close 7257 (time 1783968077.93 total 80.03 last 96.50) - open/close 8274 (time 1783968087.93 total 90.04 last 101.65) - open/close 9324 (time 1783968097.93 total 100.04 last 104.97) total: 10000 open/close in 106.25 seconds: 94.12 ops/second llite.lustre-ffff8d668b1ad000.statahead_batch_max=0 llite.lustre-ffff8d668b1ad000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff8d668b1ad000.stats=clear mdc.lustre-MDT0001-mdc-ffff8d668b1ad000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m3.325s user 0m0.305s sys 0m15.833s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff8d668b1ad000.statahead_batch_max=32 llite.lustre-ffff8d668b1ad000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff8d668b1ad000.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff8d668b1ad000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff8d668b1ad000.stats=clear mdc.lustre-MDT0001-mdc-ffff8d668b1ad000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m41.570s user 0m0.310s sys 0m14.943s batched statahead RPC= hit count=9985 mdc.lustre-MDT0000-mdc-ffff8d668b1ad000.batch_stats= snapshot_time: 1783968291.040076933 secs.nsecs start_time: 1783968176.276359540 secs.nsecs elapsed_time: 114.763717393 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 0 0 0 8: 1 0 0 16: 1 0 0 32: 313 99 100 mdc.lustre-MDT0001-mdc-ffff8d668b1ad000.batch_stats= snapshot_time: 1783968291.040152868 secs.nsecs start_time: 1783968176.276427656 secs.nsecs elapsed_time: 114.763725212 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 batch: statahead_max=2048 statahead_batch_max=256 mdc.lustre-MDT0000-mdc-ffff8d668b1ad000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff8d668b1ad000.batch_stats=0 llite.lustre-ffff8d668b1ad000.statahead_max=2048 llite.lustre-ffff8d668b1ad000.statahead_batch_max=256 kvm mdc.lustre-MDT0000-mdc-ffff8d668b1ad000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff8d668b1ad000.batch_stats=0 striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d123ad.sanity total: 100 open/close in 2.11 seconds: 47.49 ops/second llite.lustre-ffff8d668b1ad000.statahead_max=0 101 real 0m2.323s user 0m0.006s sys 0m0.352s 'ls -l' 100 files without statahead: 2 sec llite.lustre-ffff8d668b1ad000.statahead_max=2048 llite.lustre-ffff8d668b1ad000.statahead_max=2048 101 real 0m0.926s user 0m0.004s sys 0m0.192s 'ls -l' 100 files with statahead: 1 sec statahead total: 2 statahead wrong: 0 agl total: 2 list_total: 2 fname_total: 0 hit_total: 10084 miss_total: 16 swrong=0 ewrong=0 - open/close 897 (time 1783968564.57 total 10.00 last 89.67) total: 900 open/close in 10.02 seconds: 89.80 ops/second llite.lustre-ffff8d668b1ad000.statahead_max=0 1001 real 0m19.591s user 0m0.036s sys 0m3.327s 'ls -l' 1000 files without statahead: 20 sec llite.lustre-ffff8d668b1ad000.statahead_max=2048 llite.lustre-ffff8d668b1ad000.statahead_max=2048 1001 real 0m7.781s user 0m0.029s sys 0m1.608s 'ls -l' 1000 files with statahead: 8 sec statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11083 miss_total: 17 swrong=0 ewrong=0 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 18 seconds llite.lustre-ffff8d668b1ad000.statahead_stats= statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11083 miss_total: 17 mdc.lustre-MDT0000-mdc-ffff8d668b1ad000.batch_stats= snapshot_time: 1783968623.290282396 secs.nsecs start_time: 1783968294.657098067 secs.nsecs elapsed_time: 328.633184329 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 2 22 22 4: 0 0 22 8: 3 33 55 16: 1 11 66 32: 0 0 66 64: 1 11 77 128: 0 0 77 256: 2 22 100 mdc.lustre-MDT0001-mdc-ffff8d668b1ad000.batch_stats= snapshot_time: 1783968623.290354028 secs.nsecs start_time: 1783968294.657245434 secs.nsecs elapsed_time: 328.633108594 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 2 22 22 8: 3 33 55 16: 1 11 66 32: 1 11 77 64: 0 0 77 128: 0 0 77 256: 2 22 100 - open/close 842 (time 1783968636.27 total 10.00 last 84.17) - open/close 1712 (time 1783968646.28 total 20.02 last 86.88) - open/close 2692 (time 1783968656.30 total 30.03 last 97.89) - open/close 3666 (time 1783968666.30 total 40.04 last 97.31) - open/close 4548 (time 1783968676.31 total 50.04 last 88.19) - open/close 5380 (time 1783968686.31 total 60.04 last 83.16) - open/close 6222 (time 1783968696.32 total 70.05 last 84.15) - open/close 7179 (time 1783968706.33 total 80.06 last 95.61) - open/close 8195 (time 1783968716.34 total 90.07 last 101.50) - open/close 9261 (time 1783968726.34 total 100.07 last 106.55) total: 10000 open/close in 107.35 seconds: 93.15 ops/second llite.lustre-ffff8d668b1ad000.statahead_batch_max=0 llite.lustre-ffff8d668b1ad000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff8d668b1ad000.stats=clear mdc.lustre-MDT0001-mdc-ffff8d668b1ad000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m11.996s user 0m0.336s sys 0m15.977s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff8d668b1ad000.statahead_batch_max=256 llite.lustre-ffff8d668b1ad000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff8d668b1ad000.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff8d668b1ad000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff8d668b1ad000.stats=clear mdc.lustre-MDT0001-mdc-ffff8d668b1ad000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m6.106s user 0m0.314s sys 0m15.820s batched statahead RPC= hit count=9999 mdc.lustre-MDT0000-mdc-ffff8d668b1ad000.batch_stats= snapshot_time: 1783968897.305440267 secs.nsecs start_time: 1783968818.157257878 secs.nsecs elapsed_time: 79.148182389 secs.nsecs subreqs per batch batches %% cum %% 1: 293 54 54 2: 94 17 72 4: 47 8 80 8: 22 4 84 16: 19 3 88 32: 22 4 92 64: 8 1 94 128: 0 0 94 256: 32 5 100 mdc.lustre-MDT0001-mdc-ffff8d668b1ad000.batch_stats= snapshot_time: 1783968897.305507462 secs.nsecs start_time: 1783968818.157599214 secs.nsecs elapsed_time: 79.147908248 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second seq.cli-lustre-OST0000-super.width=65536 seq.cli-lustre-OST0001-super.width=65536 llite.lustre-ffff8d668b1ad000.statahead_batch_max=64 llite.lustre-ffff8d668b1ad000.statahead_max=128