== sanity test 123ad: Verify batching statahead works correctly ========================================================== 14:53:34 (1780340014) batch: statahead_max=32 statahead_batch_max=32 mdc.lustre-MDT0000-mdc-ffff93f318e17000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff93f318e17000.batch_stats=0 llite.lustre-ffff93f318e17000.statahead_max=32 llite.lustre-ffff93f318e17000.statahead_batch_max=32 seq.cli-lustre-OST0000-super.width=0x1ffffff seq.cli-lustre-OST0001-super.width=0x1ffffff kvm mdc.lustre-MDT0000-mdc-ffff93f318e17000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff93f318e17000.batch_stats=0 striped dir -i1 -c2 -H crush2 /mnt/lustre/d123ad.sanity total: 100 open/close in 2.14 seconds: 46.65 ops/second llite.lustre-ffff93f318e17000.statahead_max=0 101 real 0m2.467s user 0m0.007s sys 0m0.359s 'ls -l' 100 files without statahead: 2 sec llite.lustre-ffff93f318e17000.statahead_max=32 llite.lustre-ffff93f318e17000.statahead_max=32 101 real 0m1.102s user 0m0.004s sys 0m0.177s 'ls -l' 100 files with statahead: 1 sec statahead total: 147 statahead wrong: 2 agl total: 141 list_total: 30 fname_total: 117 hit_total: 4100 miss_total: 1094 swrong=2 ewrong=2 - open/close 796 (time 1780340043.19 total 10.01 last 79.54) total: 900 open/close in 11.62 seconds: 77.46 ops/second llite.lustre-ffff93f318e17000.statahead_max=0 1001 real 0m23.718s user 0m0.042s sys 0m3.399s 'ls -l' 1000 files without statahead: 23 sec llite.lustre-ffff93f318e17000.statahead_max=32 llite.lustre-ffff93f318e17000.statahead_max=32 1001 real 0m9.817s user 0m0.029s sys 0m1.556s 'ls -l' 1000 files with statahead: 10 sec statahead total: 148 statahead wrong: 2 agl total: 142 list_total: 31 fname_total: 117 hit_total: 5098 miss_total: 1096 swrong=2 ewrong=2 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 18 seconds llite.lustre-ffff93f318e17000.statahead_stats= statahead total: 148 statahead wrong: 2 agl total: 142 list_total: 31 fname_total: 117 hit_total: 5098 miss_total: 1096 mdc.lustre-MDT0000-mdc-ffff93f318e17000.batch_stats= snapshot_time: 1780340111.373760647 secs.nsecs start_time: 1780340020.276973778 secs.nsecs elapsed_time: 91.096786869 secs.nsecs subreqs per batch batches % cum % 1: 3 7 7 2: 1 2 10 4: 2 5 15 8: 2 5 21 16: 24 63 84 32: 6 15 100 mdc.lustre-MDT0001-mdc-ffff93f318e17000.batch_stats= snapshot_time: 1780340111.374040747 secs.nsecs start_time: 1780340020.277240451 secs.nsecs elapsed_time: 91.096800296 secs.nsecs subreqs per batch batches % cum % 1: 1 2 2 2: 1 2 5 4: 1 2 7 8: 3 7 15 16: 10 25 40 32: 24 60 100 - open/close 859 (time 1780340124.18 total 10.01 last 85.86) - open/close 1620 (time 1780340134.19 total 20.02 last 75.98) - open/close 2362 (time 1780340144.20 total 30.02 last 74.17) - open/close 3181 (time 1780340154.20 total 40.03 last 81.87) - open/close 4029 (time 1780340164.22 total 50.05 last 84.63) - open/close 4603 (time 1780340174.23 total 60.05 last 57.36) - open/close 5261 (time 1780340184.23 total 70.06 last 65.79) - open/close 5915 (time 1780340194.24 total 80.06 last 65.36) - open/close 6660 (time 1780340204.24 total 90.07 last 74.48) - open/close 7460 (time 1780340214.24 total 100.07 last 79.98) - open/close 8389 (time 1780340224.25 total 110.08 last 92.81) - open/close 9178 (time 1780340234.27 total 120.09 last 78.78) - open/close 9980 (time 1780340244.27 total 130.10 last 80.14) total: 10000 open/close in 130.28 seconds: 76.76 ops/second llite.lustre-ffff93f318e17000.statahead_batch_max=0 llite.lustre-ffff93f318e17000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff93f318e17000.stats=clear mdc.lustre-MDT0001-mdc-ffff93f318e17000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m9.935s user 0m0.326s sys 0m15.754s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff93f318e17000.statahead_batch_max=32 llite.lustre-ffff93f318e17000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff93f318e17000.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff93f318e17000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff93f318e17000.stats=clear mdc.lustre-MDT0001-mdc-ffff93f318e17000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m53.959s user 0m0.299s sys 0m14.963s batched statahead RPC= hit count=9985 mdc.lustre-MDT0000-mdc-ffff93f318e17000.batch_stats= snapshot_time: 1780340455.974298491 secs.nsecs start_time: 1780340325.825466421 secs.nsecs elapsed_time: 130.148832070 secs.nsecs subreqs per batch batches % cum % 1: 0 0 0 2: 0 0 0 4: 0 0 0 8: 1 0 0 16: 0 0 0 32: 313 99 100 mdc.lustre-MDT0001-mdc-ffff93f318e17000.batch_stats= snapshot_time: 1780340455.976198877 secs.nsecs start_time: 1780340325.825537046 secs.nsecs elapsed_time: 130.150661831 secs.nsecs subreqs per batch batches % cum % 1: 0 0 0 batch: statahead_max=2048 statahead_batch_max=256 mdc.lustre-MDT0000-mdc-ffff93f318e17000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff93f318e17000.batch_stats=0 llite.lustre-ffff93f318e17000.statahead_max=2048 llite.lustre-ffff93f318e17000.statahead_batch_max=256 kvm mdc.lustre-MDT0000-mdc-ffff93f318e17000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff93f318e17000.batch_stats=0 striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d123ad.sanity total: 100 open/close in 1.82 seconds: 55.05 ops/second llite.lustre-ffff93f318e17000.statahead_max=0 101 real 0m2.254s user 0m0.006s sys 0m0.332s 'ls -l' 100 files without statahead: 2 sec llite.lustre-ffff93f318e17000.statahead_max=2048 llite.lustre-ffff93f318e17000.statahead_max=2048 101 real 0m1.117s user 0m0.006s sys 0m0.203s 'ls -l' 100 files with statahead: 2 sec statahead total: 2 statahead wrong: 0 agl total: 2 list_total: 2 fname_total: 0 hit_total: 10084 miss_total: 16 swrong=0 ewrong=0 - open/close 836 (time 1780340759.49 total 10.01 last 83.55) total: 900 open/close in 11.03 seconds: 81.60 ops/second llite.lustre-ffff93f318e17000.statahead_max=0 1001 real 0m22.968s user 0m0.046s sys 0m3.087s 'ls -l' 1000 files without statahead: 23 sec llite.lustre-ffff93f318e17000.statahead_max=2048 llite.lustre-ffff93f318e17000.statahead_max=2048 1001 real 0m4.431s user 0m0.024s sys 0m1.199s 'ls -l' 1000 files with statahead: 4 sec statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11083 miss_total: 17 swrong=0 ewrong=0 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 20 seconds llite.lustre-ffff93f318e17000.statahead_stats= statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11083 miss_total: 17 mdc.lustre-MDT0000-mdc-ffff93f318e17000.batch_stats= snapshot_time: 1780340823.629431177 secs.nsecs start_time: 1780340459.365477429 secs.nsecs elapsed_time: 364.263953748 secs.nsecs subreqs per batch batches % cum % 1: 0 0 0 2: 1 12 12 4: 1 12 25 8: 2 25 50 16: 1 12 62 32: 1 12 75 64: 0 0 75 128: 0 0 75 256: 2 25 100 mdc.lustre-MDT0001-mdc-ffff93f318e17000.batch_stats= snapshot_time: 1780340823.629521825 secs.nsecs start_time: 1780340459.365639609 secs.nsecs elapsed_time: 364.263882216 secs.nsecs subreqs per batch batches % cum % 1: 0 0 0 2: 0 0 0 4: 4 50 50 8: 1 12 62 16: 0 0 62 32: 0 0 62 64: 1 12 75 128: 0 0 75 256: 2 25 100 - open/close 583 (time 1780340837.00 total 10.01 last 58.23) - open/close 1596 (time 1780340847.00 total 20.01 last 101.28) - open/close 2561 (time 1780340857.01 total 30.02 last 96.40) - open/close 3312 (time 1780340867.02 total 40.03 last 75.05) - open/close 4240 (time 1780340877.03 total 50.04 last 92.74) - open/close 5027 (time 1780340887.03 total 60.05 last 78.64) - open/close 5861 (time 1780340897.05 total 70.06 last 83.31) - open/close 6555 (time 1780340907.06 total 80.07 last 69.31) - open/close 7327 (time 1780340917.07 total 90.08 last 77.13) - open/close 8158 (time 1780340927.07 total 100.08 last 83.09) - open/close 8919 (time 1780340937.07 total 110.08 last 76.07) - open/close 9675 (time 1780340947.07 total 120.08 last 75.60) total: 10000 open/close in 124.02 seconds: 80.63 ops/second llite.lustre-ffff93f318e17000.statahead_batch_max=0 llite.lustre-ffff93f318e17000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff93f318e17000.stats=clear mdc.lustre-MDT0001-mdc-ffff93f318e17000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m1.834s user 0m0.315s sys 0m15.517s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff93f318e17000.statahead_batch_max=256 llite.lustre-ffff93f318e17000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff93f318e17000.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff93f318e17000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff93f318e17000.stats=clear mdc.lustre-MDT0001-mdc-ffff93f318e17000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m5.771s user 0m0.306s sys 0m16.356s batched statahead RPC= hit count=9999 mdc.lustre-MDT0000-mdc-ffff93f318e17000.batch_stats= snapshot_time: 1780341107.245022264 secs.nsecs start_time: 1780341026.686428928 secs.nsecs elapsed_time: 80.558593336 secs.nsecs subreqs per batch batches % cum % 1: 185 50 50 2: 55 14 65 4: 36 9 75 8: 20 5 80 16: 18 4 85 32: 14 3 89 64: 5 1 90 128: 1 0 90 256: 34 9 100 mdc.lustre-MDT0001-mdc-ffff93f318e17000.batch_stats= snapshot_time: 1780341107.245084134 secs.nsecs start_time: 1780341026.687010140 secs.nsecs elapsed_time: 80.558073994 secs.nsecs subreqs per batch batches % cum % 1: 0 0 0 unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second seq.cli-lustre-OST0000-super.width=65536 seq.cli-lustre-OST0001-super.width=65536 llite.lustre-ffff93f318e17000.statahead_batch_max=64 llite.lustre-ffff93f318e17000.statahead_max=128