== sanity test 123ad: Verify batching statahead works correctly ========================================================== 18:30:55 (1781217055) batch: statahead_max=32 statahead_batch_max=32 mdc.lustre-MDT0000-mdc-ffff9d374b7ed000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9d374b7ed000.batch_stats=0 llite.lustre-ffff9d374b7ed000.statahead_max=32 llite.lustre-ffff9d374b7ed000.statahead_batch_max=32 seq.cli-lustre-OST0000-super.width=0x1ffffff seq.cli-lustre-OST0001-super.width=0x1ffffff kvm mdc.lustre-MDT0000-mdc-ffff9d374b7ed000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9d374b7ed000.batch_stats=0 striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d123ad.sanity total: 100 open/close in 2.50 seconds: 40.03 ops/second llite.lustre-ffff9d374b7ed000.statahead_max=0 101 real 0m2.713s user 0m0.008s sys 0m0.394s 'ls -l' 100 files without statahead: 3 sec llite.lustre-ffff9d374b7ed000.statahead_max=32 llite.lustre-ffff9d374b7ed000.statahead_max=32 101 real 0m1.027s user 0m0.009s sys 0m0.185s 'ls -l' 100 files with statahead: 1 sec statahead total: 123 statahead wrong: 2 agl total: 117 list_total: 30 fname_total: 93 hit_total: 4281 miss_total: 1280 swrong=2 ewrong=2 - open/close 691 (time 1781217085.30 total 10.00 last 69.08) total: 900 open/close in 12.49 seconds: 72.06 ops/second llite.lustre-ffff9d374b7ed000.statahead_max=0 1001 real 0m21.129s user 0m0.039s sys 0m3.603s 'ls -l' 1000 files without statahead: 21 sec llite.lustre-ffff9d374b7ed000.statahead_max=32 llite.lustre-ffff9d374b7ed000.statahead_max=32 1001 real 0m12.115s user 0m0.042s sys 0m1.725s 'ls -l' 1000 files with statahead: 12 sec statahead total: 124 statahead wrong: 2 agl total: 118 list_total: 31 fname_total: 93 hit_total: 5280 miss_total: 1281 swrong=2 ewrong=2 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 22 seconds llite.lustre-ffff9d374b7ed000.statahead_stats= statahead total: 124 statahead wrong: 2 agl total: 118 list_total: 31 fname_total: 93 hit_total: 5280 miss_total: 1281 mdc.lustre-MDT0000-mdc-ffff9d374b7ed000.batch_stats= snapshot_time: 1781217157.460984286 secs.nsecs start_time: 1781217062.102212421 secs.nsecs elapsed_time: 95.358771865 secs.nsecs subreqs per batch batches % cum % 1: 0 0 0 2: 1 2 2 4: 3 7 10 8: 3 7 17 16: 18 46 64 32: 14 35 100 mdc.lustre-MDT0001-mdc-ffff9d374b7ed000.batch_stats= snapshot_time: 1781217157.461086464 secs.nsecs start_time: 1781217062.102344656 secs.nsecs elapsed_time: 95.358741808 secs.nsecs subreqs per batch batches % cum % 1: 1 2 2 2: 1 2 5 4: 3 7 12 8: 2 5 17 16: 21 52 70 32: 12 30 100 - open/close 722 (time 1781217170.41 total 10.01 last 72.12) - open/close 1601 (time 1781217180.42 total 20.02 last 87.83) - open/close 2559 (time 1781217190.43 total 30.02 last 95.77) - open/close 3579 (time 1781217200.43 total 40.03 last 101.94) - open/close 4499 (time 1781217210.43 total 50.03 last 91.98) - open/close 5399 (time 1781217220.44 total 60.03 last 89.97) - open/close 6154 (time 1781217230.45 total 70.04 last 75.44) - open/close 6953 (time 1781217240.46 total 80.05 last 79.82) - open/close 7707 (time 1781217250.46 total 90.06 last 75.35) - open/close 8514 (time 1781217260.46 total 100.06 last 80.69) - open/close 9279 (time 1781217270.47 total 110.06 last 76.46) total: 10000 open/close in 118.56 seconds: 84.35 ops/second llite.lustre-ffff9d374b7ed000.statahead_batch_max=0 llite.lustre-ffff9d374b7ed000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9d374b7ed000.stats=clear mdc.lustre-MDT0001-mdc-ffff9d374b7ed000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m20.684s user 0m0.360s sys 0m17.088s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff9d374b7ed000.statahead_batch_max=32 llite.lustre-ffff9d374b7ed000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9d374b7ed000.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff9d374b7ed000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff9d374b7ed000.stats=clear mdc.lustre-MDT0001-mdc-ffff9d374b7ed000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m58.015s user 0m0.347s sys 0m16.149s batched statahead RPC= hit count=9975 mdc.lustre-MDT0000-mdc-ffff9d374b7ed000.batch_stats= snapshot_time: 1781217510.655227475 secs.nsecs start_time: 1781217372.112439026 secs.nsecs elapsed_time: 138.542788449 secs.nsecs subreqs per batch batches % cum % 1: 0 0 0 2: 0 0 0 4: 0 0 0 8: 2 0 0 16: 1 0 0 32: 313 99 100 mdc.lustre-MDT0001-mdc-ffff9d374b7ed000.batch_stats= snapshot_time: 1781217510.655304320 secs.nsecs start_time: 1781217372.112573148 secs.nsecs elapsed_time: 138.542731172 secs.nsecs subreqs per batch batches % cum % 1: 0 0 0 batch: statahead_max=2048 statahead_batch_max=256 mdc.lustre-MDT0000-mdc-ffff9d374b7ed000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9d374b7ed000.batch_stats=0 llite.lustre-ffff9d374b7ed000.statahead_max=2048 llite.lustre-ffff9d374b7ed000.statahead_batch_max=256 kvm mdc.lustre-MDT0000-mdc-ffff9d374b7ed000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9d374b7ed000.batch_stats=0 striped dir -i1 -c2 -H crush2 /mnt/lustre/d123ad.sanity total: 100 open/close in 2.27 seconds: 44.04 ops/second llite.lustre-ffff9d374b7ed000.statahead_max=0 101 real 0m1.966s user 0m0.006s sys 0m0.375s 'ls -l' 100 files without statahead: 2 sec llite.lustre-ffff9d374b7ed000.statahead_max=2048 llite.lustre-ffff9d374b7ed000.statahead_max=2048 101 real 0m0.840s user 0m0.006s sys 0m0.233s 'ls -l' 100 files with statahead: 1 sec statahead total: 2 statahead wrong: 0 agl total: 2 list_total: 2 fname_total: 0 hit_total: 10074 miss_total: 26 swrong=0 ewrong=0 - open/close 801 (time 1781217796.40 total 10.01 last 80.04) total: 900 open/close in 11.28 seconds: 79.76 ops/second llite.lustre-ffff9d374b7ed000.statahead_max=0 1001 real 0m24.804s user 0m0.040s sys 0m3.396s 'ls -l' 1000 files without statahead: 25 sec llite.lustre-ffff9d374b7ed000.statahead_max=2048 llite.lustre-ffff9d374b7ed000.statahead_max=2048 1001 real 0m10.671s user 0m0.042s sys 0m1.686s 'ls -l' 1000 files with statahead: 11 sec statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11073 miss_total: 27 swrong=0 ewrong=0 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 22 seconds llite.lustre-ffff9d374b7ed000.statahead_stats= statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11073 miss_total: 27 mdc.lustre-MDT0000-mdc-ffff9d374b7ed000.batch_stats= snapshot_time: 1781217868.697727832 secs.nsecs start_time: 1781217514.527453122 secs.nsecs elapsed_time: 354.170274710 secs.nsecs subreqs per batch batches % cum % 1: 1 14 14 2: 1 14 28 4: 1 14 42 8: 2 28 71 16: 0 0 71 32: 0 0 71 64: 0 0 71 128: 0 0 71 256: 2 28 100 mdc.lustre-MDT0001-mdc-ffff9d374b7ed000.batch_stats= snapshot_time: 1781217868.697811360 secs.nsecs start_time: 1781217514.527694645 secs.nsecs elapsed_time: 354.170116715 secs.nsecs subreqs per batch batches % cum % 1: 0 0 0 2: 0 0 0 4: 1 10 10 8: 2 20 30 16: 3 30 60 32: 1 10 70 64: 1 10 80 128: 0 0 80 256: 2 20 100 - open/close 826 (time 1781217882.27 total 10.01 last 82.54) - open/close 1704 (time 1781217892.28 total 20.01 last 87.75) - open/close 2397 (time 1781217902.29 total 30.02 last 69.23) - open/close 3098 (time 1781217912.31 total 40.04 last 69.99) - open/close 3906 (time 1781217922.31 total 50.05 last 80.73) - open/close 4549 (time 1781217932.33 total 60.06 last 64.22) - open/close 5435 (time 1781217942.34 total 70.07 last 88.50) - open/close 6197 (time 1781217952.34 total 80.08 last 76.16) - open/close 7032 (time 1781217962.34 total 90.08 last 83.49) - open/close 7977 (time 1781217972.36 total 100.09 last 94.38) - open/close 8645 (time 1781217982.36 total 110.09 last 66.78) - open/close 9420 (time 1781217992.36 total 120.09 last 77.48) total: 10000 open/close in 129.48 seconds: 77.23 ops/second llite.lustre-ffff9d374b7ed000.statahead_batch_max=0 llite.lustre-ffff9d374b7ed000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9d374b7ed000.stats=clear mdc.lustre-MDT0001-mdc-ffff9d374b7ed000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m11.655s user 0m0.359s sys 0m17.230s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff9d374b7ed000.statahead_batch_max=256 llite.lustre-ffff9d374b7ed000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9d374b7ed000.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff9d374b7ed000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff9d374b7ed000.stats=clear mdc.lustre-MDT0001-mdc-ffff9d374b7ed000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m6.617s user 0m0.350s sys 0m17.051s batched statahead RPC= hit count=9999 mdc.lustre-MDT0000-mdc-ffff9d374b7ed000.batch_stats= snapshot_time: 1781218169.039122930 secs.nsecs start_time: 1781218085.302828165 secs.nsecs elapsed_time: 83.736294765 secs.nsecs subreqs per batch batches % cum % 1: 179 47 47 2: 60 15 62 4: 37 9 72 8: 30 7 80 16: 20 5 85 32: 17 4 90 64: 2 0 90 128: 1 0 91 256: 34 8 100 mdc.lustre-MDT0001-mdc-ffff9d374b7ed000.batch_stats= snapshot_time: 1781218169.040652114 secs.nsecs start_time: 1781218085.302910853 secs.nsecs elapsed_time: 83.737741261 secs.nsecs subreqs per batch batches % cum % 1: 0 0 0 unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second seq.cli-lustre-OST0000-super.width=65536 seq.cli-lustre-OST0001-super.width=65536 llite.lustre-ffff9d374b7ed000.statahead_batch_max=64 llite.lustre-ffff9d374b7ed000.statahead_max=128