== sanity test 123ad: Verify batching statahead works correctly ========================================================== 01:32:16 (1788586336) batch: statahead_max=32 statahead_batch_max=32 mdc.lustre-MDT0000-mdc-ffff976f05f46000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff976f05f46000.batch_stats=0 llite.lustre-ffff976f05f46000.statahead_max=32 llite.lustre-ffff976f05f46000.statahead_batch_max=32 seq.cli-lustre-OST0000-super.width=0x1ffffff seq.cli-lustre-OST0001-super.width=0x1ffffff kvm mdc.lustre-MDT0000-mdc-ffff976f05f46000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff976f05f46000.batch_stats=0 striped dir -i1 -c2 -H crush /mnt/lustre/d123ad.sanity total: 100 open/close in 3.41 seconds: 29.37 ops/second llite.lustre-ffff976f05f46000.statahead_max=0 101 real 0m3.060s user 0m0.006s sys 0m0.368s 'ls -l' 100 files without statahead: 3 sec llite.lustre-ffff976f05f46000.statahead_max=32 llite.lustre-ffff976f05f46000.statahead_max=32 101 real 0m1.407s user 0m0.005s sys 0m0.172s 'ls -l' 100 files with statahead: 2 sec statahead total: 117 statahead wrong: 2 agl total: 112 list_total: 31 fname_total: 86 hit_total: 4082 miss_total: 1293 swrong=2 ewrong=2 - open/close 628 (time 1788586368.17 total 10.01 last 62.74) total: 900 open/close in 14.32 seconds: 62.83 ops/second llite.lustre-ffff976f05f46000.statahead_max=0 1001 real 0m24.258s user 0m0.035s sys 0m3.120s 'ls -l' 1000 files without statahead: 25 sec llite.lustre-ffff976f05f46000.statahead_max=32 llite.lustre-ffff976f05f46000.statahead_max=32 1001 real 0m9.120s user 0m0.029s sys 0m1.443s 'ls -l' 1000 files with statahead: 9 sec statahead total: 118 statahead wrong: 2 agl total: 113 list_total: 32 fname_total: 86 hit_total: 5079 miss_total: 1296 swrong=2 ewrong=2 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 24 seconds llite.lustre-ffff976f05f46000.statahead_stats= statahead total: 118 statahead wrong: 2 agl total: 113 list_total: 32 fname_total: 86 hit_total: 5079 miss_total: 1296 mdc.lustre-MDT0000-mdc-ffff976f05f46000.batch_stats= snapshot_time: 1788586441.536393962 secs.nsecs start_time: 1788586342.500194004 secs.nsecs elapsed_time: 99.036199958 secs.nsecs subreqs per batch batches %% cum %% 1: 3 7 7 2: 0 0 7 4: 3 7 15 8: 4 10 26 16: 21 55 81 32: 7 18 100 mdc.lustre-MDT0001-mdc-ffff976f05f46000.batch_stats= snapshot_time: 1788586441.536458456 secs.nsecs start_time: 1788586342.500282747 secs.nsecs elapsed_time: 99.036175709 secs.nsecs subreqs per batch batches %% cum %% 1: 1 2 2 2: 1 2 5 4: 1 2 7 8: 3 7 15 16: 12 30 45 32: 22 55 100 - open/close 692 (time 1788586454.56 total 10.01 last 69.15) - open/close 1451 (time 1788586464.57 total 20.02 last 75.82) - open/close 2195 (time 1788586474.58 total 30.04 last 74.26) - open/close 2878 (time 1788586484.59 total 40.05 last 68.24) - open/close 3590 (time 1788586494.60 total 50.05 last 71.16) - open/close 4263 (time 1788586504.61 total 60.06 last 67.20) - open/close 5000 (time 1788586514.62 total 70.08 last 73.62) - open/close 5720 (time 1788586524.63 total 80.08 last 71.95) - open/close 6418 (time 1788586534.63 total 90.08 last 69.79) - open/close 7112 (time 1788586544.64 total 100.09 last 69.34) - open/close 7834 (time 1788586554.65 total 110.11 last 72.11) - open/close 8481 (time 1788586564.66 total 120.11 last 64.65) - open/close 9273 (time 1788586574.67 total 130.12 last 79.16) total: 10000 open/close in 139.18 seconds: 71.85 ops/second llite.lustre-ffff976f05f46000.statahead_batch_max=0 llite.lustre-ffff976f05f46000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff976f05f46000.stats=clear mdc.lustre-MDT0001-mdc-ffff976f05f46000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m1.749s user 0m0.304s sys 0m15.848s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff976f05f46000.statahead_batch_max=32 llite.lustre-ffff976f05f46000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff976f05f46000.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff976f05f46000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff976f05f46000.stats=clear mdc.lustre-MDT0001-mdc-ffff976f05f46000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m33.554s user 0m0.291s sys 0m14.157s batched statahead RPC= hit count=9979 mdc.lustre-MDT0000-mdc-ffff976f05f46000.batch_stats= snapshot_time: 1788586759.563473360 secs.nsecs start_time: 1788586653.957718360 secs.nsecs elapsed_time: 105.605755000 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 0 0 0 8: 1 0 0 16: 1 0 0 32: 313 99 100 mdc.lustre-MDT0001-mdc-ffff976f05f46000.batch_stats= snapshot_time: 1788586759.564124523 secs.nsecs start_time: 1788586653.957781479 secs.nsecs elapsed_time: 105.606343044 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 batch: statahead_max=2048 statahead_batch_max=256 mdc.lustre-MDT0000-mdc-ffff976f05f46000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff976f05f46000.batch_stats=0 llite.lustre-ffff976f05f46000.statahead_max=2048 llite.lustre-ffff976f05f46000.statahead_batch_max=256 kvm mdc.lustre-MDT0000-mdc-ffff976f05f46000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff976f05f46000.batch_stats=0 striped dir -i1 -c2 -H all_char /mnt/lustre/d123ad.sanity total: 100 open/close in 3.00 seconds: 33.32 ops/second llite.lustre-ffff976f05f46000.statahead_max=0 101 real 0m2.053s user 0m0.006s sys 0m0.312s 'ls -l' 100 files without statahead: 2 sec llite.lustre-ffff976f05f46000.statahead_max=2048 llite.lustre-ffff976f05f46000.statahead_max=2048 101 real 0m0.891s user 0m0.001s sys 0m0.163s 'ls -l' 100 files with statahead: 1 sec statahead total: 2 statahead wrong: 0 agl total: 2 list_total: 2 fname_total: 0 hit_total: 10078 miss_total: 22 swrong=0 ewrong=0 - open/close 790 (time 1788587074.55 total 10.00 last 79.00) total: 900 open/close in 11.53 seconds: 78.06 ops/second llite.lustre-ffff976f05f46000.statahead_max=0 1001 real 0m22.310s user 0m0.033s sys 0m2.967s 'ls -l' 1000 files without statahead: 22 sec llite.lustre-ffff976f05f46000.statahead_max=2048 llite.lustre-ffff976f05f46000.statahead_max=2048 1001 real 0m6.207s user 0m0.024s sys 0m1.404s 'ls -l' 1000 files with statahead: 6 sec statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11077 miss_total: 23 swrong=0 ewrong=0 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 20 seconds llite.lustre-ffff976f05f46000.statahead_stats= statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11077 miss_total: 23 mdc.lustre-MDT0000-mdc-ffff976f05f46000.batch_stats= snapshot_time: 1788587135.429223347 secs.nsecs start_time: 1788586762.323779592 secs.nsecs elapsed_time: 373.105443755 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 1 12 12 4: 1 12 25 8: 1 12 37 16: 2 25 62 32: 1 12 75 64: 0 0 75 128: 0 0 75 256: 2 25 100 mdc.lustre-MDT0001-mdc-ffff976f05f46000.batch_stats= snapshot_time: 1788587135.429308816 secs.nsecs start_time: 1788586762.323935882 secs.nsecs elapsed_time: 373.105372934 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 1 12 12 8: 2 25 37 16: 2 25 62 32: 1 12 75 64: 0 0 75 128: 0 0 75 256: 2 25 100 - open/close 764 (time 1788587148.08 total 10.00 last 76.39) - open/close 1518 (time 1788587158.09 total 20.01 last 75.34) - open/close 2339 (time 1788587168.10 total 30.02 last 82.00) - open/close 3153 (time 1788587178.10 total 40.02 last 81.38) - open/close 3914 (time 1788587188.11 total 50.03 last 76.04) - open/close 4572 (time 1788587198.11 total 60.03 last 65.79) - open/close 5472 (time 1788587208.13 total 70.06 last 89.79) - open/close 6213 (time 1788587218.14 total 80.07 last 74.03) - open/close 6922 (time 1788587228.15 total 90.07 last 70.88) - open/close 7711 (time 1788587238.16 total 100.08 last 78.78) - open/close 8472 (time 1788587248.16 total 110.09 last 76.07) - open/close 9332 (time 1788587258.17 total 120.09 last 85.97) total: 10000 open/close in 130.04 seconds: 76.90 ops/second llite.lustre-ffff976f05f46000.statahead_batch_max=0 llite.lustre-ffff976f05f46000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff976f05f46000.stats=clear mdc.lustre-MDT0001-mdc-ffff976f05f46000.stats=clear ls -l files in d123ad.sanity: 10001 real 0m51.735s user 0m0.330s sys 0m16.708s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff976f05f46000.statahead_batch_max=256 llite.lustre-ffff976f05f46000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff976f05f46000.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff976f05f46000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff976f05f46000.stats=clear mdc.lustre-MDT0001-mdc-ffff976f05f46000.stats=clear ls -l files in d123ad.sanity: 10001 real 0m47.585s user 0m0.281s sys 0m15.030s batched statahead RPC= hit count=9999 mdc.lustre-MDT0000-mdc-ffff976f05f46000.batch_stats= snapshot_time: 1788587384.685791441 secs.nsecs start_time: 1788587327.972330423 secs.nsecs elapsed_time: 56.713461018 secs.nsecs subreqs per batch batches %% cum %% 1: 123 48 48 2: 31 12 60 4: 19 7 68 8: 15 5 74 16: 5 1 76 32: 15 5 82 64: 10 3 86 128: 0 0 86 256: 35 13 100 mdc.lustre-MDT0001-mdc-ffff976f05f46000.batch_stats= snapshot_time: 1788587384.685880561 secs.nsecs start_time: 1788587327.972407173 secs.nsecs elapsed_time: 56.713473388 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second seq.cli-lustre-OST0000-super.width=65536 seq.cli-lustre-OST0001-super.width=65536 llite.lustre-ffff976f05f46000.statahead_batch_max=64 llite.lustre-ffff976f05f46000.statahead_max=128