== sanity test 398b: DIO and buffer IO race ============== 12:11:56 (1730826716) /usr/bin/fio 12+0 records in 12+0 records out 12582912 bytes (13 MB) copied, 0.085923 s, 146 MB/s mix direct rw 12M to OST0 by fio with 4 jobs... mix buffer rw 12M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=14839: Tue Nov 5 12:11:58 2024 read: IOPS=369, BW=1477KiB/s (1512kB/s)(1480KiB/1002msec) clat (usec): min=3, max=28981, avg=1482.20, stdev=2417.32 lat (usec): min=4, max=28982, avg=1483.33, stdev=2417.37 clat percentiles (usec): | 1.00th=[ 5], 5.00th=[ 7], 10.00th=[ 8], 20.00th=[ 10], | 30.00th=[ 12], 40.00th=[ 17], 50.00th=[ 1090], 60.00th=[ 1713], | 70.00th=[ 2114], 80.00th=[ 2606], 90.00th=[ 3097], 95.00th=[ 3687], | 99.00th=[10290], 99.50th=[24511], 99.90th=[28967], 99.95th=[28967], | 99.99th=[28967] bw ( KiB/s): min= 1232, max= 1712, per=26.21%, avg=1472.00, stdev=339.41, samples=2 iops : min= 308, max= 428, avg=368.00, stdev=84.85, samples=2 write: IOPS=397, BW=1589KiB/s (1627kB/s)(1592KiB/1002msec) clat (usec): min=88, max=29553, avg=1102.72, stdev=2061.68 lat (usec): min=89, max=29554, avg=1104.13, stdev=2061.50 clat percentiles (usec): | 1.00th=[ 98], 5.00th=[ 106], 10.00th=[ 121], 20.00th=[ 157], | 30.00th=[ 174], 40.00th=[ 200], 50.00th=[ 351], 60.00th=[ 881], | 70.00th=[ 1123], 80.00th=[ 1729], 90.00th=[ 2606], 95.00th=[ 3720], | 99.00th=[ 7177], 99.50th=[16450], 99.90th=[29492], 99.95th=[29492], | 99.99th=[29492] bw ( KiB/s): min= 1232, max= 1944, per=26.87%, avg=1588.00, stdev=503.46, samples=2 iops : min= 308, max= 486, avg=397.00, stdev=125.87, samples=2 lat (usec) : 4=0.13%, 10=9.90%, 20=9.37%, 50=1.30%, 100=0.91% lat (usec) : 250=23.05%, 500=3.78%, 750=1.43%, 1000=6.51% lat (msec) : 2=18.49%, 4=21.48%, 10=2.86%, 20=0.39%, 50=0.39% cpu : usr=0.00%, sys=18.08%, ctx=812, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=14840: Tue Nov 5 12:11:58 2024 read: IOPS=355, BW=1420KiB/s (1454kB/s)(1436KiB/1011msec) clat (usec): min=3, max=27614, avg=1613.14, stdev=2211.25 lat (usec): min=4, max=27615, avg=1613.71, stdev=2211.27 clat percentiles (usec): | 1.00th=[ 5], 5.00th=[ 7], 10.00th=[ 8], 20.00th=[ 10], | 30.00th=[ 14], 40.00th=[ 979], 50.00th=[ 1434], 60.00th=[ 1844], | 70.00th=[ 2343], 80.00th=[ 2737], 90.00th=[ 3261], 95.00th=[ 3982], | 99.00th=[ 7111], 99.50th=[14615], 99.90th=[27657], 99.95th=[27657], | 99.99th=[27657] bw ( KiB/s): min= 1208, max= 1616, per=25.14%, avg=1412.00, stdev=288.50, samples=2 iops : min= 302, max= 404, avg=353.00, stdev=72.12, samples=2 write: IOPS=404, BW=1618KiB/s (1657kB/s)(1636KiB/1011msec) clat (usec): min=84, max=12553, avg=1028.25, stdev=1387.62 lat (usec): min=85, max=12554, avg=1028.97, stdev=1387.63 clat percentiles (usec): | 1.00th=[ 96], 5.00th=[ 109], 10.00th=[ 120], 20.00th=[ 149], | 30.00th=[ 172], 40.00th=[ 198], 50.00th=[ 412], 60.00th=[ 865], | 70.00th=[ 1090], 80.00th=[ 1663], 90.00th=[ 2671], 95.00th=[ 3654], | 99.00th=[ 6456], 99.50th=[ 7504], 99.90th=[12518], 99.95th=[12518], | 99.99th=[12518] bw ( KiB/s): min= 1488, max= 1736, per=27.28%, avg=1612.00, stdev=175.36, samples=2 iops : min= 372, max= 434, avg=403.00, stdev=43.84, samples=2 lat (usec) : 4=0.13%, 10=9.37%, 20=6.90%, 50=0.91%, 100=0.78% lat (usec) : 250=23.57%, 500=2.99%, 750=1.95%, 1000=7.94% lat (msec) : 2=18.75%, 4=22.40%, 10=3.78%, 20=0.39%, 50=0.13% cpu : usr=0.59%, sys=17.23%, ctx=828, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=14841: Tue Nov 5 12:11:58 2024 read: IOPS=366, BW=1467KiB/s (1502kB/s)(1564KiB/1066msec) clat (usec): min=3, max=18582, avg=1696.68, stdev=2142.31 lat (usec): min=3, max=18582, avg=1697.37, stdev=2142.35 clat percentiles (usec): | 1.00th=[ 4], 5.00th=[ 7], 10.00th=[ 8], 20.00th=[ 11], | 30.00th=[ 14], 40.00th=[ 816], 50.00th=[ 1516], 60.00th=[ 1942], | 70.00th=[ 2409], 80.00th=[ 2835], 90.00th=[ 3392], 95.00th=[ 5080], | 99.00th=[13435], 99.50th=[16581], 99.90th=[18482], 99.95th=[18482], | 99.99th=[18482] bw ( KiB/s): min= 1224, max= 1608, per=25.21%, avg=1416.00, stdev=271.53, samples=2 iops : min= 306, max= 402, avg=354.00, stdev=67.88, samples=2 write: IOPS=353, BW=1415KiB/s (1449kB/s)(1508KiB/1066msec) clat (usec): min=89, max=15810, avg=1029.89, stdev=1368.43 lat (usec): min=89, max=15812, avg=1030.58, stdev=1368.49 clat percentiles (usec): | 1.00th=[ 97], 5.00th=[ 111], 10.00th=[ 126], 20.00th=[ 165], | 30.00th=[ 180], 40.00th=[ 221], 50.00th=[ 734], 60.00th=[ 947], | 70.00th=[ 1123], 80.00th=[ 1549], 90.00th=[ 2540], 95.00th=[ 3425], | 99.00th=[ 5342], 99.50th=[ 7046], 99.90th=[15795], 99.95th=[15795], | 99.99th=[15795] bw ( KiB/s): min= 1160, max= 1536, per=22.81%, avg=1348.00, stdev=265.87, samples=2 iops : min= 290, max= 384, avg=337.00, stdev=66.47, samples=2 lat (usec) : 4=0.52%, 10=8.98%, 20=9.24%, 50=1.04%, 100=0.78% lat (usec) : 250=20.18%, 500=2.08%, 750=2.08%, 1000=7.68% lat (msec) : 2=20.31%, 4=21.35%, 10=5.08%, 20=0.65% cpu : usr=0.19%, sys=17.28%, ctx=950, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=14842: Tue Nov 5 12:11:58 2024 read: IOPS=369, BW=1477KiB/s (1512kB/s)(1508KiB/1021msec) clat (usec): min=3, max=22910, avg=1561.93, stdev=2363.23 lat (usec): min=3, max=22911, avg=1562.48, stdev=2363.28 clat percentiles (usec): | 1.00th=[ 4], 5.00th=[ 5], 10.00th=[ 7], 20.00th=[ 9], | 30.00th=[ 11], 40.00th=[ 15], 50.00th=[ 1037], 60.00th=[ 1762], | 70.00th=[ 2180], 80.00th=[ 2704], 90.00th=[ 3556], 95.00th=[ 4752], | 99.00th=[10552], 99.50th=[22676], 99.90th=[22938], 99.95th=[22938], | 99.99th=[22938] bw ( KiB/s): min= 1288, max= 1600, per=25.71%, avg=1444.00, stdev=220.62, samples=2 iops : min= 322, max= 400, avg=361.00, stdev=55.15, samples=2 write: IOPS=382, BW=1532KiB/s (1569kB/s)(1564KiB/1021msec) clat (usec): min=86, max=11687, avg=1077.33, stdev=1291.95 lat (usec): min=86, max=11688, avg=1077.99, stdev=1291.97 clat percentiles (usec): | 1.00th=[ 96], 5.00th=[ 108], 10.00th=[ 118], 20.00th=[ 147], | 30.00th=[ 172], 40.00th=[ 204], 50.00th=[ 717], 60.00th=[ 1020], | 70.00th=[ 1287], 80.00th=[ 1975], 90.00th=[ 2606], 95.00th=[ 3392], | 99.00th=[ 6194], 99.50th=[ 6783], 99.90th=[11731], 99.95th=[11731], | 99.99th=[11731] bw ( KiB/s): min= 1280, max= 1736, per=25.52%, avg=1508.00, stdev=322.44, samples=2 iops : min= 320, max= 434, avg=377.00, stdev=80.61, samples=2 lat (usec) : 4=1.04%, 10=13.28%, 20=7.16%, 50=0.39%, 100=1.95% lat (usec) : 250=20.96%, 500=2.60%, 750=1.17%, 1000=5.86% lat (msec) : 2=18.49%, 4=21.87%, 10=4.56%, 20=0.39%, 50=0.26% cpu : usr=0.00%, sys=17.06%, ctx=860, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=5617KiB/s (5752kB/s), 1420KiB/s-1477KiB/s (1454kB/s-1512kB/s), io=5988KiB (6132kB), run=1002-1066msec WRITE: bw=5910KiB/s (6052kB/s), 1415KiB/s-1618KiB/s (1449kB/s-1657kB/s), io=6300KiB (6451kB), run=1002-1066msec rand-rw: (groupid=0, jobs=1): err= 0: pid=14843: Tue Nov 5 12:11:59 2024 read: IOPS=226, BW=906KiB/s (927kB/s)(1480KiB/1634msec) clat (usec): min=597, max=46370, avg=1575.23, stdev=2843.53 lat (usec): min=598, max=46372, avg=1575.90, stdev=2843.59 clat percentiles (usec): | 1.00th=[ 619], 5.00th=[ 685], 10.00th=[ 701], 20.00th=[ 766], | 30.00th=[ 807], 40.00th=[ 873], 50.00th=[ 930], 60.00th=[ 1045], | 70.00th=[ 1188], 80.00th=[ 1680], 90.00th=[ 3097], 95.00th=[ 4359], | 99.00th=[ 6456], 99.50th=[23200], 99.90th=[46400], 99.95th=[46400], | 99.99th=[46400] bw ( KiB/s): min= 560, max= 1272, per=24.15%, avg=864.00, stdev=367.22, samples=3 iops : min= 140, max= 318, avg=216.00, stdev=91.80, samples=3 write: IOPS=243, BW=974KiB/s (998kB/s)(1592KiB/1634msec) clat (usec): min=1073, max=23277, avg=2620.49, stdev=1688.19 lat (usec): min=1074, max=23277, avg=2621.47, stdev=1688.10 clat percentiles (usec): | 1.00th=[ 1156], 5.00th=[ 1352], 10.00th=[ 1500], 20.00th=[ 1713], | 30.00th=[ 1844], 40.00th=[ 1975], 50.00th=[ 2089], 60.00th=[ 2278], | 70.00th=[ 2606], 80.00th=[ 3359], 90.00th=[ 4424], 95.00th=[ 5735], | 99.00th=[ 8291], 99.50th=[ 9896], 99.90th=[23200], 99.95th=[23200], | 99.99th=[23200] bw ( KiB/s): min= 584, max= 1440, per=24.94%, avg=938.67, stdev=446.45, samples=3 iops : min= 146, max= 360, avg=234.67, stdev=111.61, samples=3 lat (usec) : 750=7.94%, 1000=19.66% lat (msec) : 2=36.33%, 4=26.69%, 10=8.98%, 50=0.39% cpu : usr=0.55%, sys=5.94%, ctx=823, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=14844: Tue Nov 5 12:11:59 2024 read: IOPS=221, BW=886KiB/s (907kB/s)(1436KiB/1621msec) clat (usec): min=563, max=44955, avg=1503.47, stdev=2586.98 lat (usec): min=563, max=44957, avg=1504.08, stdev=2587.07 clat percentiles (usec): | 1.00th=[ 603], 5.00th=[ 652], 10.00th=[ 693], 20.00th=[ 758], | 30.00th=[ 807], 40.00th=[ 848], 50.00th=[ 938], 60.00th=[ 1012], | 70.00th=[ 1205], 80.00th=[ 1680], 90.00th=[ 2638], 95.00th=[ 4178], | 99.00th=[ 6783], 99.50th=[ 9110], 99.90th=[44827], 99.95th=[44827], | 99.99th=[44827] bw ( KiB/s): min= 592, max= 1352, per=24.01%, avg=858.67, stdev=427.71, samples=3 iops : min= 148, max= 338, avg=214.67, stdev=106.93, samples=3 write: IOPS=252, BW=1009KiB/s (1033kB/s)(1636KiB/1621msec) clat (usec): min=1106, max=10156, avg=2624.95, stdev=1356.95 lat (usec): min=1107, max=10158, avg=2625.66, stdev=1357.01 clat percentiles (usec): | 1.00th=[ 1221], 5.00th=[ 1450], 10.00th=[ 1582], 20.00th=[ 1745], | 30.00th=[ 1844], 40.00th=[ 1958], 50.00th=[ 2114], 60.00th=[ 2343], | 70.00th=[ 2737], 80.00th=[ 3294], 90.00th=[ 4424], 95.00th=[ 5604], | 99.00th=[ 7439], 99.50th=[ 8094], 99.90th=[10159], 99.95th=[10159], | 99.99th=[10159] bw ( KiB/s): min= 672, max= 1384, per=25.72%, avg=968.00, stdev=370.86, samples=3 iops : min= 168, max= 346, avg=242.00, stdev=92.71, samples=3 lat (usec) : 750=8.85%, 1000=17.97% lat (msec) : 2=35.29%, 4=28.12%, 10=9.51%, 20=0.13%, 50=0.13% cpu : usr=0.31%, sys=6.11%, ctx=808, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=14845: Tue Nov 5 12:11:59 2024 read: IOPS=240, BW=962KiB/s (986kB/s)(1564KiB/1625msec) clat (usec): min=514, max=11182, avg=1443.27, stdev=1344.47 lat (usec): min=514, max=11183, avg=1443.89, stdev=1344.50 clat percentiles (usec): | 1.00th=[ 586], 5.00th=[ 644], 10.00th=[ 676], 20.00th=[ 742], | 30.00th=[ 799], 40.00th=[ 848], 50.00th=[ 922], 60.00th=[ 1020], | 70.00th=[ 1205], 80.00th=[ 1795], 90.00th=[ 3261], 95.00th=[ 4228], | 99.00th=[ 7963], 99.50th=[ 8291], 99.90th=[11207], 99.95th=[11207], | 99.99th=[11207] bw ( KiB/s): min= 480, max= 1448, per=25.35%, avg=906.67, stdev=494.08, samples=3 iops : min= 120, max= 362, avg=226.67, stdev=123.52, samples=3 write: IOPS=232, BW=928KiB/s (950kB/s)(1508KiB/1625msec) clat (usec): min=1016, max=45374, avg=2790.16, stdev=2699.70 lat (usec): min=1017, max=45375, avg=2791.03, stdev=2699.71 clat percentiles (usec): | 1.00th=[ 1172], 5.00th=[ 1369], 10.00th=[ 1565], 20.00th=[ 1713], | 30.00th=[ 1827], 40.00th=[ 1942], 50.00th=[ 2114], 60.00th=[ 2311], | 70.00th=[ 2737], 80.00th=[ 3556], 90.00th=[ 4621], 95.00th=[ 5735], | 99.00th=[ 9110], 99.50th=[14877], 99.90th=[45351], 99.95th=[45351], | 99.99th=[45351] bw ( KiB/s): min= 600, max= 1408, per=23.60%, avg=888.00, stdev=451.20, samples=3 iops : min= 150, max= 352, avg=222.00, stdev=112.80, samples=3 lat (usec) : 750=11.46%, 1000=18.75% lat (msec) : 2=33.20%, 4=25.65%, 10=10.42%, 20=0.39%, 50=0.13% cpu : usr=0.49%, sys=5.85%, ctx=815, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=14846: Tue Nov 5 12:11:59 2024 read: IOPS=225, BW=901KiB/s (922kB/s)(1508KiB/1674msec) clat (usec): min=590, max=26293, avg=1559.09, stdev=1900.33 lat (usec): min=591, max=26294, avg=1559.93, stdev=1900.39 clat percentiles (usec): | 1.00th=[ 619], 5.00th=[ 676], 10.00th=[ 701], 20.00th=[ 750], | 30.00th=[ 816], 40.00th=[ 865], 50.00th=[ 930], 60.00th=[ 1057], | 70.00th=[ 1303], 80.00th=[ 1860], 90.00th=[ 3261], 95.00th=[ 4621], | 99.00th=[ 8356], 99.50th=[11994], 99.90th=[26346], 99.95th=[26346], | 99.99th=[26346] bw ( KiB/s): min= 528, max= 1256, per=23.26%, avg=832.00, stdev=378.54, samples=3 iops : min= 132, max= 314, avg=208.00, stdev=94.64, samples=3 write: IOPS=233, BW=934KiB/s (957kB/s)(1564KiB/1674msec) clat (usec): min=1092, max=43260, avg=2758.27, stdev=2544.37 lat (usec): min=1093, max=43262, avg=2759.11, stdev=2544.49 clat percentiles (usec): | 1.00th=[ 1139], 5.00th=[ 1303], 10.00th=[ 1434], 20.00th=[ 1745], | 30.00th=[ 1860], 40.00th=[ 1975], 50.00th=[ 2114], 60.00th=[ 2278], | 70.00th=[ 2606], 80.00th=[ 3392], 90.00th=[ 4686], 95.00th=[ 6063], | 99.00th=[ 8160], 99.50th=[12911], 99.90th=[43254], 99.95th=[43254], | 99.99th=[43254] bw ( KiB/s): min= 576, max= 1416, per=23.31%, avg=877.33, stdev=467.60, samples=3 iops : min= 144, max= 354, avg=219.33, stdev=116.90, samples=3 lat (usec) : 750=10.29%, 1000=16.93% lat (msec) : 2=34.51%, 4=26.95%, 10=10.81%, 20=0.26%, 50=0.26% cpu : usr=0.24%, sys=6.16%, ctx=805, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=3577KiB/s (3663kB/s), 886KiB/s-962KiB/s (907kB/s-986kB/s), io=5988KiB (6132kB), run=1621-1674msec WRITE: bw=3763KiB/s (3854kB/s), 928KiB/s-1009KiB/s (950kB/s-1033kB/s), io=6300KiB (6451kB), run=1621-1674msec