== sanity test 123ad: Verify batching statahead works correctly ========================================================== 08:14:46 (1787400886) batch: statahead_max=32 statahead_batch_max=32 mdc.lustre-MDT0000-mdc-ffff908158b62800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff908158b62800.batch_stats=0 llite.lustre-ffff908158b62800.statahead_max=32 llite.lustre-ffff908158b62800.statahead_batch_max=32 seq.cli-lustre-OST0000-super.width=0x1ffffff seq.cli-lustre-OST0001-super.width=0x1ffffff kvm mdc.lustre-MDT0000-mdc-ffff908158b62800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff908158b62800.batch_stats=0 striped dir -i1 -c2 -H crush2 /mnt/lustre/d123ad.sanity total: 100 open/close in 2.44 seconds: 40.90 ops/second llite.lustre-ffff908158b62800.statahead_max=0 101 real 0m2.640s user 0m0.007s sys 0m0.325s 'ls -l' 100 files without statahead: 3 sec llite.lustre-ffff908158b62800.statahead_max=32 llite.lustre-ffff908158b62800.statahead_max=32 101 real 0m0.971s user 0m0.005s sys 0m0.205s 'ls -l' 100 files with statahead: 1 sec statahead total: 144 statahead wrong: 2 agl total: 138 list_total: 31 fname_total: 103 hit_total: 4084 miss_total: 1385 swrong=2 ewrong=2 - open/close 736 (time 1787400913.40 total 10.00 last 73.57) total: 900 open/close in 12.45 seconds: 72.31 ops/second llite.lustre-ffff908158b62800.statahead_max=0 1001 real 0m24.631s user 0m0.038s sys 0m3.537s 'ls -l' 1000 files without statahead: 25 sec llite.lustre-ffff908158b62800.statahead_max=32 llite.lustre-ffff908158b62800.statahead_max=32 1001 real 0m8.380s user 0m0.030s sys 0m1.671s 'ls -l' 1000 files with statahead: 9 sec statahead total: 145 statahead wrong: 2 agl total: 139 list_total: 32 fname_total: 103 hit_total: 5082 miss_total: 1387 swrong=2 ewrong=2 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 20 seconds llite.lustre-ffff908158b62800.statahead_stats= statahead total: 145 statahead wrong: 2 agl total: 139 list_total: 32 fname_total: 103 hit_total: 5082 miss_total: 1387 mdc.lustre-MDT0000-mdc-ffff908158b62800.batch_stats= snapshot_time: 1787400982.308234615 secs.nsecs start_time: 1787400890.747548602 secs.nsecs elapsed_time: 91.560686013 secs.nsecs subreqs per batch batches %% cum %% 1: 1 2 2 2: 4 10 13 4: 1 2 16 8: 1 2 18 16: 25 67 86 32: 5 13 100 mdc.lustre-MDT0001-mdc-ffff908158b62800.batch_stats= snapshot_time: 1787400982.310964761 secs.nsecs start_time: 1787400890.747698340 secs.nsecs elapsed_time: 91.563266421 secs.nsecs subreqs per batch batches %% cum %% 1: 1 2 2 2: 0 0 2 4: 1 2 5 8: 4 10 15 16: 10 25 40 32: 24 60 100 - open/close 776 (time 1787400995.02 total 10.00 last 77.59) - open/close 1488 (time 1787401005.04 total 20.02 last 71.05) - open/close 2182 (time 1787401015.04 total 30.03 last 69.38) - open/close 2863 (time 1787401025.05 total 40.04 last 68.03) - open/close 3526 (time 1787401035.07 total 50.06 last 66.14) - open/close 4177 (time 1787401045.08 total 60.07 last 65.05) - open/close 4785 (time 1787401055.10 total 70.08 last 60.70) - open/close 5543 (time 1787401065.10 total 80.09 last 75.75) - open/close 6213 (time 1787401075.11 total 90.10 last 66.94) - open/close 6918 (time 1787401085.14 total 100.12 last 70.34) - open/close 7623 (time 1787401095.15 total 110.13 last 70.43) - open/close 8383 (time 1787401105.15 total 120.13 last 76.00) - open/close 9044 (time 1787401115.15 total 130.14 last 66.05) - open/close 9785 (time 1787401125.16 total 140.15 last 74.06) total: 10000 open/close in 143.27 seconds: 69.80 ops/second llite.lustre-ffff908158b62800.statahead_batch_max=0 llite.lustre-ffff908158b62800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff908158b62800.stats=clear mdc.lustre-MDT0001-mdc-ffff908158b62800.stats=clear ls -l files in d123ad.sanity: 10001 real 1m9.585s user 0m0.319s sys 0m18.053s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff908158b62800.statahead_batch_max=32 llite.lustre-ffff908158b62800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff908158b62800.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff908158b62800.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff908158b62800.stats=clear mdc.lustre-MDT0001-mdc-ffff908158b62800.stats=clear ls -l files in d123ad.sanity: 10001 real 2m3.235s user 0m0.356s sys 0m16.984s batched statahead RPC= hit count=9980 mdc.lustre-MDT0000-mdc-ffff908158b62800.batch_stats= snapshot_time: 1787401347.151334493 secs.nsecs start_time: 1787401209.650592861 secs.nsecs elapsed_time: 137.500741632 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 0 0 0 8: 1 0 0 16: 1 0 0 32: 313 99 100 mdc.lustre-MDT0001-mdc-ffff908158b62800.batch_stats= snapshot_time: 1787401347.151710783 secs.nsecs start_time: 1787401209.650801259 secs.nsecs elapsed_time: 137.500909524 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 batch: statahead_max=2048 statahead_batch_max=256 mdc.lustre-MDT0000-mdc-ffff908158b62800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff908158b62800.batch_stats=0 llite.lustre-ffff908158b62800.statahead_max=2048 llite.lustre-ffff908158b62800.statahead_batch_max=256 kvm mdc.lustre-MDT0000-mdc-ffff908158b62800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff908158b62800.batch_stats=0 striped dir -i1 -c2 -H crush2 /mnt/lustre/d123ad.sanity total: 100 open/close in 4.01 seconds: 24.93 ops/second llite.lustre-ffff908158b62800.statahead_max=0 101 real 0m3.527s user 0m0.005s sys 0m0.357s 'ls -l' 100 files without statahead: 4 sec llite.lustre-ffff908158b62800.statahead_max=2048 llite.lustre-ffff908158b62800.statahead_max=2048 101 real 0m0.999s user 0m0.003s sys 0m0.198s 'ls -l' 100 files with statahead: 1 sec statahead total: 2 statahead wrong: 0 agl total: 2 list_total: 2 fname_total: 0 hit_total: 10079 miss_total: 21 swrong=0 ewrong=0 - open/close 651 (time 1787401704.38 total 10.00 last 65.08) total: 900 open/close in 14.21 seconds: 63.34 ops/second llite.lustre-ffff908158b62800.statahead_max=0 1001 real 0m34.902s user 0m0.042s sys 0m3.577s 'ls -l' 1000 files without statahead: 34 sec llite.lustre-ffff908158b62800.statahead_max=2048 llite.lustre-ffff908158b62800.statahead_max=2048 1001 real 0m5.937s user 0m0.031s sys 0m1.519s 'ls -l' 1000 files with statahead: 6 sec statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11078 miss_total: 22 swrong=0 ewrong=0 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 20 seconds llite.lustre-ffff908158b62800.statahead_stats= statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11078 miss_total: 22 mdc.lustre-MDT0000-mdc-ffff908158b62800.batch_stats= snapshot_time: 1787401783.861672847 secs.nsecs start_time: 1787401349.773512660 secs.nsecs elapsed_time: 434.088160187 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 2 25 25 4: 2 25 50 8: 1 12 62 16: 1 12 75 32: 0 0 75 64: 0 0 75 128: 0 0 75 256: 2 25 100 mdc.lustre-MDT0001-mdc-ffff908158b62800.batch_stats= snapshot_time: 1787401783.862286345 secs.nsecs start_time: 1787401349.773703883 secs.nsecs elapsed_time: 434.088582462 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 0 0 0 8: 5 50 50 16: 1 10 60 32: 1 10 70 64: 0 0 70 128: 1 10 80 256: 2 20 100 - open/close 749 (time 1787401796.39 total 10.01 last 74.81) - open/close 1399 (time 1787401806.40 total 20.03 last 64.90) - open/close 2028 (time 1787401816.42 total 30.05 last 62.79) - open/close 2615 (time 1787401826.43 total 40.06 last 58.62) - open/close 3232 (time 1787401836.43 total 50.06 last 61.69) - open/close 3917 (time 1787401846.46 total 60.08 last 68.35) - open/close 4557 (time 1787401856.46 total 70.09 last 63.96) - open/close 5166 (time 1787401866.49 total 80.11 last 60.76) - open/close 5831 (time 1787401876.50 total 90.13 last 66.39) - open/close 6490 (time 1787401886.52 total 100.14 last 65.81) - open/close 7199 (time 1787401896.53 total 110.15 last 70.81) - open/close 7955 (time 1787401906.53 total 120.16 last 75.58) - open/close 8605 (time 1787401916.54 total 130.16 last 64.96) - open/close 9258 (time 1787401926.54 total 140.17 last 65.26) - open/close 9870 (time 1787401936.55 total 150.17 last 61.18) total: 10000 open/close in 152.26 seconds: 65.68 ops/second llite.lustre-ffff908158b62800.statahead_batch_max=0 llite.lustre-ffff908158b62800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff908158b62800.stats=clear mdc.lustre-MDT0001-mdc-ffff908158b62800.stats=clear ls -l files in d123ad.sanity: 10001 real 1m12.744s user 0m0.326s sys 0m17.237s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff908158b62800.statahead_batch_max=256 llite.lustre-ffff908158b62800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff908158b62800.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff908158b62800.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff908158b62800.stats=clear mdc.lustre-MDT0001-mdc-ffff908158b62800.stats=clear ls -l files in d123ad.sanity: 10001 real 0m57.026s user 0m0.318s sys 0m17.704s batched statahead RPC= hit count=9999 mdc.lustre-MDT0000-mdc-ffff908158b62800.batch_stats= snapshot_time: 1787402093.943459465 secs.nsecs start_time: 1787402022.384895845 secs.nsecs elapsed_time: 71.558563620 secs.nsecs subreqs per batch batches %% cum %% 1: 227 52 52 2: 61 14 66 4: 53 12 78 8: 27 6 84 16: 8 1 86 32: 20 4 91 64: 4 0 92 128: 0 0 92 256: 34 7 100 mdc.lustre-MDT0001-mdc-ffff908158b62800.batch_stats= snapshot_time: 1787402093.943517772 secs.nsecs start_time: 1787402022.384976690 secs.nsecs elapsed_time: 71.558541082 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second seq.cli-lustre-OST0000-super.width=65536 seq.cli-lustre-OST0001-super.width=65536 llite.lustre-ffff908158b62800.statahead_batch_max=64 llite.lustre-ffff908158b62800.statahead_max=128