== sanity test 398b: DIO and buffer IO race ============== 23:31:51 (1723260711) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.23009 s, 40.9 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=7398: Fri Aug 9 23:32:06 2024 read: IOPS=128, BW=515KiB/s (528kB/s)(5992KiB/11628msec) clat (usec): min=50, max=43060, avg=4273.26, stdev=2690.23 lat (usec): min=50, max=43060, avg=4274.21, stdev=2689.66 clat percentiles (usec): | 1.00th=[ 60], 5.00th=[ 502], 10.00th=[ 2114], 20.00th=[ 2507], | 30.00th=[ 2835], 40.00th=[ 3294], 50.00th=[ 3818], 60.00th=[ 4359], | 70.00th=[ 5014], 80.00th=[ 5735], 90.00th=[ 6980], 95.00th=[ 8455], | 99.00th=[13698], 99.50th=[15795], 99.90th=[27395], 99.95th=[43254], | 99.99th=[43254] bw ( KiB/s): min= 232, max= 678, per=25.46%, avg=513.57, stdev=89.99, samples=23 iops : min= 58, max= 169, avg=128.35, stdev=22.46, samples=23 write: IOPS=135, BW=541KiB/s (554kB/s)(6296KiB/11628msec) clat (usec): min=394, max=123947, avg=3268.34, stdev=4154.73 lat (usec): min=395, max=123948, avg=3269.65, stdev=4154.85 clat percentiles (usec): | 1.00th=[ 490], 5.00th=[ 523], 10.00th=[ 545], 20.00th=[ 603], | 30.00th=[ 791], 40.00th=[ 1909], 50.00th=[ 2606], 60.00th=[ 3195], | 70.00th=[ 4080], 80.00th=[ 5407], 90.00th=[ 7242], 95.00th=[ 8848], | 99.00th=[ 12256], 99.50th=[ 13566], 99.90th=[ 20579], 99.95th=[124257], | 99.99th=[124257] bw ( KiB/s): min= 264, max= 728, per=25.44%, avg=537.22, stdev=83.57, samples=23 iops : min= 66, max= 182, avg=134.26, stdev=20.87, samples=23 lat (usec) : 100=2.18%, 250=0.23%, 500=0.98%, 750=13.90%, 1000=2.70% lat (msec) : 2=4.98%, 4=36.43%, 10=36.10%, 20=2.34%, 50=0.13% lat (msec) : 250=0.03% cpu : usr=0.17%, sys=23.05%, ctx=5719, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=7399: Fri Aug 9 23:32:06 2024 read: IOPS=125, BW=502KiB/s (515kB/s)(5940KiB/11821msec) clat (usec): min=53, max=28290, avg=4314.65, stdev=2427.50 lat (usec): min=54, max=28291, avg=4315.34, stdev=2427.60 clat percentiles (usec): | 1.00th=[ 59], 5.00th=[ 1860], 10.00th=[ 2147], 20.00th=[ 2540], | 30.00th=[ 2933], 40.00th=[ 3326], 50.00th=[ 3818], 60.00th=[ 4490], | 70.00th=[ 5145], 80.00th=[ 5866], 90.00th=[ 6980], 95.00th=[ 8586], | 99.00th=[12780], 99.50th=[13829], 99.90th=[25297], 99.95th=[28181], | 99.99th=[28181] bw ( KiB/s): min= 208, max= 638, per=24.56%, avg=495.39, stdev=88.35, samples=23 iops : min= 52, max= 159, avg=123.74, stdev=22.07, samples=23 write: IOPS=134, BW=537KiB/s (550kB/s)(6348KiB/11821msec) clat (usec): min=413, max=161882, avg=3358.32, stdev=4974.01 lat (usec): min=413, max=161883, avg=3360.09, stdev=4973.73 clat percentiles (usec): | 1.00th=[ 490], 5.00th=[ 523], 10.00th=[ 545], 20.00th=[ 594], | 30.00th=[ 824], 40.00th=[ 2024], 50.00th=[ 2606], 60.00th=[ 3163], | 70.00th=[ 4080], 80.00th=[ 5407], 90.00th=[ 7373], 95.00th=[ 8979], | 99.00th=[ 11994], 99.50th=[ 14091], 99.90th=[ 39584], 99.95th=[162530], | 99.99th=[162530] bw ( KiB/s): min= 264, max= 768, per=24.68%, avg=521.17, stdev=89.00, samples=23 iops : min= 66, max= 192, avg=130.17, stdev=22.31, samples=23 lat (usec) : 100=1.50%, 250=0.33%, 500=1.33%, 750=13.54%, 1000=2.15% lat (msec) : 2=5.21%, 4=36.88%, 10=36.13%, 20=2.80%, 50=0.10% lat (msec) : 250=0.03% cpu : usr=0.18%, sys=22.65%, ctx=5668, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=7400: Fri Aug 9 23:32:06 2024 read: IOPS=127, BW=510KiB/s (522kB/s)(6028KiB/11817msec) clat (usec): min=50, max=168708, avg=4473.27, stdev=4922.55 lat (usec): min=51, max=168709, avg=4474.07, stdev=4922.55 clat percentiles (usec): | 1.00th=[ 60], 5.00th=[ 1876], 10.00th=[ 2147], 20.00th=[ 2573], | 30.00th=[ 2933], 40.00th=[ 3359], 50.00th=[ 3916], 60.00th=[ 4490], | 70.00th=[ 5080], 80.00th=[ 5800], 90.00th=[ 7242], 95.00th=[ 8717], | 99.00th=[ 12518], 99.50th=[ 15139], 99.90th=[ 37487], 99.95th=[168821], | 99.99th=[168821] bw ( KiB/s): min= 192, max= 608, per=25.17%, avg=507.61, stdev=83.03, samples=23 iops : min= 48, max= 152, avg=126.83, stdev=20.76, samples=23 write: IOPS=132, BW=530KiB/s (542kB/s)(6260KiB/11817msec) clat (usec): min=399, max=22727, avg=3192.62, stdev=2831.20 lat (usec): min=400, max=22727, avg=3193.63, stdev=2831.14 clat percentiles (usec): | 1.00th=[ 482], 5.00th=[ 515], 10.00th=[ 537], 20.00th=[ 586], | 30.00th=[ 734], 40.00th=[ 1893], 50.00th=[ 2606], 60.00th=[ 3195], | 70.00th=[ 4080], 80.00th=[ 5407], 90.00th=[ 7111], 95.00th=[ 8848], | 99.00th=[11994], 99.50th=[13042], 99.90th=[18220], 99.95th=[22676], | 99.99th=[22676] bw ( KiB/s): min= 272, max= 688, per=24.86%, avg=524.96, stdev=93.07, samples=23 iops : min= 68, max= 172, avg=131.17, stdev=23.26, samples=23 lat (usec) : 100=1.66%, 250=0.13%, 500=1.82%, 750=13.74%, 1000=1.86% lat (msec) : 2=4.92%, 4=36.13%, 10=37.21%, 20=2.41%, 50=0.10% lat (msec) : 250=0.03% cpu : usr=0.14%, sys=22.69%, ctx=5695, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=7401: Fri Aug 9 23:32:06 2024 read: IOPS=127, BW=508KiB/s (521kB/s)(6052KiB/11903msec) clat (usec): min=48, max=168064, avg=4538.54, stdev=4881.28 lat (usec): min=48, max=168064, avg=4539.11, stdev=4881.27 clat percentiles (usec): | 1.00th=[ 61], 5.00th=[ 1827], 10.00th=[ 2180], 20.00th=[ 2606], | 30.00th=[ 3032], 40.00th=[ 3523], 50.00th=[ 3982], 60.00th=[ 4555], | 70.00th=[ 5211], 80.00th=[ 5997], 90.00th=[ 7111], 95.00th=[ 8717], | 99.00th=[ 13435], 99.50th=[ 16188], 99.90th=[ 26608], 99.95th=[168821], | 99.99th=[168821] bw ( KiB/s): min= 208, max= 648, per=24.81%, avg=500.43, stdev=81.99, samples=23 iops : min= 52, max= 162, avg=125.09, stdev=20.48, samples=23 write: IOPS=130, BW=524KiB/s (536kB/s)(6236KiB/11903msec) clat (usec): min=460, max=16641, avg=3179.38, stdev=2704.91 lat (usec): min=461, max=16642, avg=3180.27, stdev=2704.71 clat percentiles (usec): | 1.00th=[ 486], 5.00th=[ 515], 10.00th=[ 537], 20.00th=[ 586], | 30.00th=[ 758], 40.00th=[ 2114], 50.00th=[ 2671], 60.00th=[ 3228], | 70.00th=[ 4113], 80.00th=[ 5211], 90.00th=[ 7111], 95.00th=[ 8455], | 99.00th=[11469], 99.50th=[12125], 99.90th=[14484], 99.95th=[16581], | 99.99th=[16581] bw ( KiB/s): min= 240, max= 720, per=24.52%, avg=517.83, stdev=91.79, samples=23 iops : min= 60, max= 180, avg=129.43, stdev=22.93, samples=23 lat (usec) : 50=0.03%, 100=1.73%, 250=0.20%, 500=0.94%, 750=14.26% lat (usec) : 1000=1.63% lat (msec) : 2=4.52%, 4=36.13%, 10=37.66%, 20=2.83%, 50=0.03% lat (msec) : 250=0.03% cpu : usr=0.18%, sys=22.48%, ctx=5880, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2017KiB/s (2066kB/s), 502KiB/s-515KiB/s (515kB/s-528kB/s), io=23.4MiB (24.6MB), run=11628-11903msec WRITE: bw=2112KiB/s (2163kB/s), 524KiB/s-541KiB/s (536kB/s-554kB/s), io=24.6MiB (25.7MB), run=11628-11903msec rand-rw: (groupid=0, jobs=1): err= 0: pid=7402: Fri Aug 9 23:32:09 2024 read: IOPS=97, BW=391KiB/s (400kB/s)(5992KiB/15323msec) clat (usec): min=1376, max=188103, avg=4205.92, stdev=5612.48 lat (usec): min=1376, max=188105, avg=4206.52, stdev=5612.51 clat percentiles (usec): | 1.00th=[ 1631], 5.00th=[ 1762], 10.00th=[ 1876], 20.00th=[ 2024], | 30.00th=[ 2212], 40.00th=[ 2540], 50.00th=[ 2900], 60.00th=[ 3490], | 70.00th=[ 4228], 80.00th=[ 5604], 90.00th=[ 8717], 95.00th=[ 10421], | 99.00th=[ 15795], 99.50th=[ 17957], 99.90th=[ 23987], 99.95th=[187696], | 99.99th=[187696] bw ( KiB/s): min= 152, max= 816, per=24.79%, avg=387.90, stdev=162.06, samples=30 iops : min= 38, max= 204, avg=96.93, stdev=40.50, samples=30 write: IOPS=102, BW=411KiB/s (421kB/s)(6296KiB/15323msec) clat (usec): min=2098, max=28344, avg=5683.24, stdev=3368.51 lat (usec): min=2099, max=28344, avg=5684.07, stdev=3368.41 clat percentiles (usec): | 1.00th=[ 2376], 5.00th=[ 2671], 10.00th=[ 2868], 20.00th=[ 3130], | 30.00th=[ 3425], 40.00th=[ 3785], 50.00th=[ 4424], 60.00th=[ 5276], | 70.00th=[ 6587], 80.00th=[ 7963], 90.00th=[10159], 95.00th=[11994], | 99.00th=[17695], 99.50th=[20841], 99.90th=[27657], 99.95th=[28443], | 99.99th=[28443] bw ( KiB/s): min= 160, max= 832, per=24.87%, avg=407.37, stdev=163.59, samples=30 iops : min= 40, max= 208, avg=101.80, stdev=40.88, samples=30 lat (msec) : 2=8.76%, 4=46.55%, 10=36.33%, 20=7.94%, 50=0.39% lat (msec) : 250=0.03% cpu : usr=0.11%, sys=10.92%, ctx=3598, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=7403: Fri Aug 9 23:32:09 2024 read: IOPS=96, BW=387KiB/s (397kB/s)(5940KiB/15340msec) clat (usec): min=1460, max=167694, avg=4174.70, stdev=5201.04 lat (usec): min=1460, max=167695, avg=4175.29, stdev=5201.06 clat percentiles (usec): | 1.00th=[ 1614], 5.00th=[ 1762], 10.00th=[ 1844], 20.00th=[ 2024], | 30.00th=[ 2212], 40.00th=[ 2442], 50.00th=[ 2835], 60.00th=[ 3326], | 70.00th=[ 4080], 80.00th=[ 5538], 90.00th=[ 8717], 95.00th=[ 10945], | 99.00th=[ 14877], 99.50th=[ 17433], 99.90th=[ 20055], 99.95th=[166724], | 99.99th=[166724] bw ( KiB/s): min= 88, max= 792, per=24.71%, avg=386.63, stdev=173.98, samples=30 iops : min= 22, max= 198, avg=96.50, stdev=43.53, samples=30 write: IOPS=103, BW=414KiB/s (424kB/s)(6348KiB/15340msec) clat (usec): min=2054, max=47779, avg=5722.23, stdev=3544.39 lat (usec): min=2055, max=47781, avg=5722.95, stdev=3544.38 clat percentiles (usec): | 1.00th=[ 2409], 5.00th=[ 2737], 10.00th=[ 2933], 20.00th=[ 3163], | 30.00th=[ 3458], 40.00th=[ 3851], 50.00th=[ 4359], 60.00th=[ 5342], | 70.00th=[ 6456], 80.00th=[ 8094], 90.00th=[10028], 95.00th=[11994], | 99.00th=[18220], 99.50th=[20055], 99.90th=[35914], 99.95th=[47973], | 99.99th=[47973] bw ( KiB/s): min= 152, max= 824, per=24.88%, avg=407.47, stdev=162.23, samples=30 iops : min= 38, max= 206, avg=101.70, stdev=40.57, samples=30 lat (msec) : 2=9.11%, 4=46.61%, 10=35.68%, 20=8.24%, 50=0.33% lat (msec) : 250=0.03% cpu : usr=0.16%, sys=10.91%, ctx=3598, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=7404: Fri Aug 9 23:32:09 2024 read: IOPS=98, BW=394KiB/s (403kB/s)(6028KiB/15303msec) clat (usec): min=1553, max=24172, avg=4163.26, stdev=3070.30 lat (usec): min=1554, max=24172, avg=4163.99, stdev=3070.28 clat percentiles (usec): | 1.00th=[ 1631], 5.00th=[ 1762], 10.00th=[ 1876], 20.00th=[ 2024], | 30.00th=[ 2212], 40.00th=[ 2507], 50.00th=[ 2868], 60.00th=[ 3490], | 70.00th=[ 4424], 80.00th=[ 5800], 90.00th=[ 8848], 95.00th=[10552], | 99.00th=[15008], 99.50th=[17957], 99.90th=[23200], 99.95th=[24249], | 99.99th=[24249] bw ( KiB/s): min= 120, max= 832, per=25.01%, avg=391.33, stdev=164.41, samples=30 iops : min= 30, max= 208, avg=97.77, stdev=41.11, samples=30 write: IOPS=102, BW=409KiB/s (419kB/s)(6260KiB/15303msec) clat (msec): min=2, max=157, avg= 5.72, stdev= 5.07 lat (msec): min=2, max=157, avg= 5.72, stdev= 5.08 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 3], 20.00th=[ 4], | 30.00th=[ 4], 40.00th=[ 4], 50.00th=[ 5], 60.00th=[ 6], | 70.00th=[ 7], 80.00th=[ 8], 90.00th=[ 11], 95.00th=[ 13], | 99.00th=[ 19], 99.50th=[ 22], 99.90th=[ 27], 99.95th=[ 159], | 99.99th=[ 159] bw ( KiB/s): min= 168, max= 800, per=24.74%, avg=405.20, stdev=166.55, samples=30 iops : min= 42, max= 200, avg=101.23, stdev=41.65, samples=30 lat (msec) : 2=9.02%, 4=46.91%, 10=35.61%, 20=8.04%, 50=0.39% lat (msec) : 250=0.03% cpu : usr=0.16%, sys=10.87%, ctx=3593, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=7405: Fri Aug 9 23:32:09 2024 read: IOPS=99, BW=396KiB/s (406kB/s)(6052KiB/15276msec) clat (usec): min=1410, max=20572, avg=3983.51, stdev=2861.38 lat (usec): min=1410, max=20573, avg=3984.10, stdev=2861.37 clat percentiles (usec): | 1.00th=[ 1647], 5.00th=[ 1778], 10.00th=[ 1860], 20.00th=[ 2008], | 30.00th=[ 2212], 40.00th=[ 2442], 50.00th=[ 2802], 60.00th=[ 3326], | 70.00th=[ 4178], 80.00th=[ 5342], 90.00th=[ 8356], 95.00th=[10421], | 99.00th=[13435], 99.50th=[16057], 99.90th=[19006], 99.95th=[20579], | 99.99th=[20579] bw ( KiB/s): min= 112, max= 784, per=25.18%, avg=394.10, stdev=167.92, samples=30 iops : min= 28, max= 196, avg=98.50, stdev=41.98, samples=30 write: IOPS=102, BW=408KiB/s (418kB/s)(6236KiB/15276msec) clat (msec): min=2, max=165, avg= 5.89, stdev= 5.29 lat (msec): min=2, max=165, avg= 5.89, stdev= 5.29 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 3], 20.00th=[ 4], | 30.00th=[ 4], 40.00th=[ 4], 50.00th=[ 5], 60.00th=[ 6], | 70.00th=[ 7], 80.00th=[ 9], 90.00th=[ 11], 95.00th=[ 13], | 99.00th=[ 18], 99.50th=[ 21], 99.90th=[ 36], 99.95th=[ 167], | 99.99th=[ 167] bw ( KiB/s): min= 128, max= 776, per=24.73%, avg=405.03, stdev=165.18, samples=30 iops : min= 32, max= 194, avg=101.23, stdev=41.31, samples=30 lat (msec) : 2=9.70%, 4=45.38%, 10=36.07%, 20=8.59%, 50=0.23% lat (msec) : 250=0.03% cpu : usr=0.16%, sys=10.80%, ctx=3561, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1565KiB/s (1603kB/s), 387KiB/s-396KiB/s (397kB/s-406kB/s), io=23.4MiB (24.6MB), run=15276-15340msec WRITE: bw=1639KiB/s (1678kB/s), 408KiB/s-414KiB/s (418kB/s-424kB/s), io=24.6MiB (25.7MB), run=15276-15340msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=7434: Fri Aug 9 23:32:16 2024 read: IOPS=59, BW=945KiB/s (968kB/s)(5920KiB/6263msec) clat (usec): min=61, max=37707, avg=10946.78, stdev=4229.23 lat (usec): min=61, max=37708, avg=10947.37, stdev=4229.28 clat percentiles (usec): | 1.00th=[ 78], 5.00th=[ 7177], 10.00th=[ 7635], 20.00th=[ 8455], | 30.00th=[ 9110], 40.00th=[ 9634], 50.00th=[10159], 60.00th=[10945], | 70.00th=[11600], 80.00th=[12911], 90.00th=[14746], 95.00th=[17695], | 99.00th=[28443], 99.50th=[32375], 99.90th=[37487], 99.95th=[37487], | 99.99th=[37487] bw ( KiB/s): min= 672, max= 1184, per=24.64%, avg=938.42, stdev=155.63, samples=12 iops : min= 42, max= 74, avg=58.58, stdev= 9.69, samples=12 write: IOPS=63, BW=1017KiB/s (1041kB/s)(6368KiB/6263msec) clat (usec): min=682, max=40825, avg=5487.56, stdev=6126.33 lat (usec): min=683, max=40826, avg=5488.52, stdev=6126.33 clat percentiles (usec): | 1.00th=[ 766], 5.00th=[ 799], 10.00th=[ 840], 20.00th=[ 1319], | 30.00th=[ 2278], 40.00th=[ 2606], 50.00th=[ 3163], 60.00th=[ 4490], | 70.00th=[ 5932], 80.00th=[ 8291], 90.00th=[12780], 95.00th=[16909], | 99.00th=[34866], 99.50th=[36439], 99.90th=[40633], 99.95th=[40633], | 99.99th=[40633] bw ( KiB/s): min= 736, max= 1312, per=25.68%, avg=1029.08, stdev=179.67, samples=12 iops : min= 46, max= 82, avg=64.25, stdev=11.17, samples=12 lat (usec) : 100=0.78%, 250=0.13%, 750=0.13%, 1000=8.72% lat (msec) : 2=4.17%, 4=17.06%, 10=35.03%, 20=30.60%, 50=3.39% cpu : usr=0.19%, sys=18.19%, ctx=2820, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=7435: Fri Aug 9 23:32:16 2024 read: IOPS=57, BW=922KiB/s (944kB/s)(5744KiB/6228msec) clat (usec): min=65, max=29304, avg=11115.87, stdev=3462.97 lat (usec): min=65, max=29305, avg=11116.49, stdev=3462.99 clat percentiles (usec): | 1.00th=[ 6587], 5.00th=[ 7373], 10.00th=[ 7898], 20.00th=[ 8586], | 30.00th=[ 9241], 40.00th=[ 9896], 50.00th=[10421], 60.00th=[11076], | 70.00th=[11863], 80.00th=[12911], 90.00th=[15008], 95.00th=[17171], | 99.00th=[25297], 99.50th=[27132], 99.90th=[29230], 99.95th=[29230], | 99.99th=[29230] bw ( KiB/s): min= 512, max= 1216, per=24.36%, avg=927.75, stdev=171.85, samples=12 iops : min= 32, max= 76, avg=57.92, stdev=10.69, samples=12 write: IOPS=65, BW=1051KiB/s (1076kB/s)(6544KiB/6228msec) clat (usec): min=675, max=59983, avg=5376.90, stdev=7160.55 lat (usec): min=676, max=59985, avg=5377.89, stdev=7160.55 clat percentiles (usec): | 1.00th=[ 758], 5.00th=[ 791], 10.00th=[ 824], 20.00th=[ 1352], | 30.00th=[ 2114], 40.00th=[ 2442], 50.00th=[ 2900], 60.00th=[ 3818], | 70.00th=[ 4883], 80.00th=[ 7111], 90.00th=[12125], 95.00th=[16057], | 99.00th=[40109], 99.50th=[42730], 99.90th=[60031], 99.95th=[60031], | 99.99th=[60031] bw ( KiB/s): min= 800, max= 1536, per=26.01%, avg=1042.42, stdev=238.35, samples=12 iops : min= 50, max= 96, avg=65.08, stdev=14.87, samples=12 lat (usec) : 100=0.26%, 750=0.52%, 1000=8.07% lat (msec) : 2=5.86%, 4=18.23%, 10=31.90%, 20=31.90%, 50=2.99% lat (msec) : 100=0.26% cpu : usr=0.08%, sys=18.28%, ctx=2806, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=7436: Fri Aug 9 23:32:16 2024 read: IOPS=62, BW=995KiB/s (1019kB/s)(6256KiB/6288msec) clat (usec): min=58, max=46185, avg=10802.27, stdev=4574.18 lat (usec): min=59, max=46186, avg=10802.86, stdev=4574.21 clat percentiles (usec): | 1.00th=[ 71], 5.00th=[ 6718], 10.00th=[ 7570], 20.00th=[ 8586], | 30.00th=[ 9110], 40.00th=[ 9503], 50.00th=[10290], 60.00th=[10814], | 70.00th=[11338], 80.00th=[12125], 90.00th=[13829], 95.00th=[17695], | 99.00th=[30278], 99.50th=[36439], 99.90th=[46400], 99.95th=[46400], | 99.99th=[46400] bw ( KiB/s): min= 640, max= 1309, per=25.90%, avg=986.42, stdev=197.21, samples=12 iops : min= 40, max= 81, avg=61.58, stdev=12.21, samples=12 write: IOPS=59, BW=959KiB/s (982kB/s)(6032KiB/6288msec) clat (usec): min=622, max=44473, avg=5435.08, stdev=6177.90 lat (usec): min=623, max=44474, avg=5437.58, stdev=6177.78 clat percentiles (usec): | 1.00th=[ 758], 5.00th=[ 807], 10.00th=[ 848], 20.00th=[ 1795], | 30.00th=[ 2180], 40.00th=[ 2573], 50.00th=[ 3163], 60.00th=[ 4113], | 70.00th=[ 5538], 80.00th=[ 8225], 90.00th=[12518], 95.00th=[17957], | 99.00th=[34866], 99.50th=[39584], 99.90th=[44303], 99.95th=[44303], | 99.99th=[44303] bw ( KiB/s): min= 544, max= 1248, per=23.09%, avg=925.17, stdev=201.90, samples=12 iops : min= 34, max= 78, avg=57.75, stdev=12.72, samples=12 lat (usec) : 100=0.91%, 250=0.26%, 750=0.39%, 1000=7.16% lat (msec) : 2=4.17%, 4=17.58%, 10=35.16%, 20=31.12%, 50=3.26% cpu : usr=0.14%, sys=19.42%, ctx=2899, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=7437: Fri Aug 9 23:32:16 2024 read: IOPS=61, BW=976KiB/s (1000kB/s)(6032KiB/6178msec) clat (usec): min=59, max=33211, avg=10749.10, stdev=3929.45 lat (usec): min=59, max=33212, avg=10749.68, stdev=3929.47 clat percentiles (usec): | 1.00th=[ 72], 5.00th=[ 7111], 10.00th=[ 7701], 20.00th=[ 8455], | 30.00th=[ 8979], 40.00th=[ 9503], 50.00th=[10028], 60.00th=[10814], | 70.00th=[11731], 80.00th=[12387], 90.00th=[14484], 95.00th=[16909], | 99.00th=[26870], 99.50th=[28705], 99.90th=[33162], 99.95th=[33162], | 99.99th=[33162] bw ( KiB/s): min= 608, max= 1152, per=25.48%, avg=970.42, stdev=155.03, samples=12 iops : min= 38, max= 72, avg=60.58, stdev= 9.65, samples=12 write: IOPS=63, BW=1013KiB/s (1037kB/s)(6256KiB/6178msec) clat (usec): min=729, max=55310, avg=5318.41, stdev=6272.19 lat (usec): min=730, max=55312, avg=5319.47, stdev=6272.24 clat percentiles (usec): | 1.00th=[ 742], 5.00th=[ 783], 10.00th=[ 873], 20.00th=[ 1860], | 30.00th=[ 2180], 40.00th=[ 2540], 50.00th=[ 3097], 60.00th=[ 4146], | 70.00th=[ 5407], 80.00th=[ 7570], 90.00th=[11863], 95.00th=[16909], | 99.00th=[32375], 99.50th=[42206], 99.90th=[55313], 99.95th=[55313], | 99.99th=[55313] bw ( KiB/s): min= 768, max= 1184, per=25.35%, avg=1015.75, stdev=114.09, samples=12 iops : min= 48, max= 74, avg=63.42, stdev= 7.09, samples=12 lat (usec) : 100=0.65%, 250=0.39%, 750=0.65%, 1000=5.73% lat (msec) : 2=5.73%, 4=17.97%, 10=37.89%, 20=27.60%, 50=3.26% lat (msec) : 100=0.13% cpu : usr=0.24%, sys=18.81%, ctx=2842, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=3809KiB/s (3901kB/s), 922KiB/s-995KiB/s (944kB/s-1019kB/s), io=23.4MiB (24.5MB), run=6178-6288msec WRITE: bw=4008KiB/s (4104kB/s), 959KiB/s-1051KiB/s (982kB/s-1076kB/s), io=24.6MiB (25.8MB), run=6178-6288msec rand-rw: (groupid=0, jobs=1): err= 0: pid=7438: Fri Aug 9 23:32:16 2024 read: IOPS=59, BW=947KiB/s (969kB/s)(5920KiB/6253msec) clat (usec): min=1830, max=45623, avg=6534.24, stdev=5517.73 lat (usec): min=1831, max=45624, avg=6534.82, stdev=5517.73 clat percentiles (usec): | 1.00th=[ 1844], 5.00th=[ 1975], 10.00th=[ 2180], 20.00th=[ 2507], | 30.00th=[ 2900], 40.00th=[ 3392], 50.00th=[ 4424], 60.00th=[ 5866], | 70.00th=[ 7832], 80.00th=[ 9765], 90.00th=[13042], 95.00th=[16909], | 99.00th=[27132], 99.50th=[33817], 99.90th=[45876], 99.95th=[45876], | 99.99th=[45876] bw ( KiB/s): min= 608, max= 1120, per=24.76%, avg=938.42, stdev=166.45, samples=12 iops : min= 38, max= 70, avg=58.58, stdev=10.33, samples=12 write: IOPS=63, BW=1018KiB/s (1043kB/s)(6368KiB/6253msec) clat (usec): min=2721, max=45559, avg=9428.65, stdev=6200.38 lat (usec): min=2722, max=45560, avg=9430.03, stdev=6200.26 clat percentiles (usec): | 1.00th=[ 2868], 5.00th=[ 3294], 10.00th=[ 3752], 20.00th=[ 4817], | 30.00th=[ 5538], 40.00th=[ 6718], 50.00th=[ 7570], 60.00th=[ 8979], | 70.00th=[10683], 80.00th=[13435], 90.00th=[16319], 95.00th=[20841], | 99.00th=[36439], 99.50th=[40109], 99.90th=[45351], 99.95th=[45351], | 99.99th=[45351] bw ( KiB/s): min= 640, max= 1312, per=25.68%, avg=1023.83, stdev=196.94, samples=12 iops : min= 40, max= 82, avg=63.92, stdev=12.38, samples=12 lat (msec) : 2=2.47%, 4=25.78%, 10=44.92%, 20=22.66%, 50=4.17% cpu : usr=0.11%, sys=8.75%, ctx=1243, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=7439: Fri Aug 9 23:32:16 2024 read: IOPS=57, BW=924KiB/s (946kB/s)(5744KiB/6219msec) clat (usec): min=1742, max=43637, avg=6350.79, stdev=4825.84 lat (usec): min=1742, max=43638, avg=6351.39, stdev=4825.86 clat percentiles (usec): | 1.00th=[ 1811], 5.00th=[ 2008], 10.00th=[ 2147], 20.00th=[ 2540], | 30.00th=[ 2868], 40.00th=[ 3621], 50.00th=[ 4948], 60.00th=[ 6587], | 70.00th=[ 7963], 80.00th=[ 9765], 90.00th=[11600], 95.00th=[14615], | 99.00th=[23987], 99.50th=[26346], 99.90th=[43779], 99.95th=[43779], | 99.99th=[43779] bw ( KiB/s): min= 448, max= 1184, per=24.26%, avg=919.50, stdev=198.74, samples=12 iops : min= 28, max= 74, avg=57.33, stdev=12.30, samples=12 write: IOPS=65, BW=1052KiB/s (1078kB/s)(6544KiB/6219msec) clat (usec): min=2419, max=57267, avg=9451.93, stdev=7082.65 lat (usec): min=2420, max=57269, avg=9453.39, stdev=7082.78 clat percentiles (usec): | 1.00th=[ 2769], 5.00th=[ 3195], 10.00th=[ 3490], 20.00th=[ 4424], | 30.00th=[ 5604], 40.00th=[ 6718], 50.00th=[ 7832], 60.00th=[ 9110], | 70.00th=[10552], 80.00th=[12780], 90.00th=[15533], 95.00th=[21365], | 99.00th=[38011], 99.50th=[49546], 99.90th=[57410], 99.95th=[57410], | 99.99th=[57410] bw ( KiB/s): min= 736, max= 1408, per=26.07%, avg=1039.50, stdev=231.10, samples=12 iops : min= 46, max= 88, avg=64.83, stdev=14.38, samples=12 lat (msec) : 2=2.21%, 4=26.30%, 10=45.83%, 20=21.74%, 50=3.65% lat (msec) : 100=0.26% cpu : usr=0.10%, sys=9.12%, ctx=1248, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=7440: Fri Aug 9 23:32:16 2024 read: IOPS=61, BW=990KiB/s (1014kB/s)(6256KiB/6319msec) clat (usec): min=1685, max=43673, avg=6509.71, stdev=5243.68 lat (usec): min=1686, max=43673, avg=6510.30, stdev=5243.68 clat percentiles (usec): | 1.00th=[ 1795], 5.00th=[ 1926], 10.00th=[ 2114], 20.00th=[ 2376], | 30.00th=[ 2769], 40.00th=[ 3392], 50.00th=[ 4621], 60.00th=[ 6783], | 70.00th=[ 8455], 80.00th=[ 9896], 90.00th=[13042], 95.00th=[15401], | 99.00th=[27919], 99.50th=[31851], 99.90th=[43779], 99.95th=[43779], | 99.99th=[43779] bw ( KiB/s): min= 544, max= 1344, per=26.03%, avg=986.67, stdev=213.07, samples=12 iops : min= 34, max= 84, avg=61.67, stdev=13.32, samples=12 write: IOPS=59, BW=955KiB/s (977kB/s)(6032KiB/6319msec) clat (usec): min=2573, max=52837, avg=9927.83, stdev=6882.52 lat (usec): min=2574, max=52839, avg=9929.05, stdev=6882.62 clat percentiles (usec): | 1.00th=[ 2737], 5.00th=[ 3261], 10.00th=[ 3654], 20.00th=[ 5080], | 30.00th=[ 6128], 40.00th=[ 7308], 50.00th=[ 8291], 60.00th=[ 9503], | 70.00th=[11207], 80.00th=[12911], 90.00th=[17433], 95.00th=[22414], | 99.00th=[45351], 99.50th=[50070], 99.90th=[52691], 99.95th=[52691], | 99.99th=[52691] bw ( KiB/s): min= 640, max= 1216, per=23.28%, avg=928.00, stdev=197.26, samples=12 iops : min= 40, max= 76, avg=58.00, stdev=12.33, samples=12 lat (msec) : 2=3.78%, 4=26.17%, 10=42.06%, 20=23.57%, 50=4.17% lat (msec) : 100=0.26% cpu : usr=0.17%, sys=9.05%, ctx=1312, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=7441: Fri Aug 9 23:32:16 2024 read: IOPS=60, BW=972KiB/s (996kB/s)(6032KiB/6203msec) clat (usec): min=1677, max=48070, avg=6606.58, stdev=5595.66 lat (usec): min=1678, max=48070, avg=6607.17, stdev=5595.67 clat percentiles (usec): | 1.00th=[ 1795], 5.00th=[ 2024], 10.00th=[ 2147], 20.00th=[ 2442], | 30.00th=[ 2900], 40.00th=[ 3490], 50.00th=[ 4555], 60.00th=[ 6456], | 70.00th=[ 8094], 80.00th=[ 9765], 90.00th=[13304], 95.00th=[17433], | 99.00th=[27657], 99.50th=[35914], 99.90th=[47973], 99.95th=[47973], | 99.99th=[47973] bw ( KiB/s): min= 640, max= 1213, per=25.67%, avg=972.83, stdev=167.35, samples=12 iops : min= 40, max= 75, avg=60.67, stdev=10.31, samples=12 write: IOPS=63, BW=1009KiB/s (1033kB/s)(6256KiB/6203msec) clat (usec): min=2474, max=43209, avg=9305.08, stdev=5697.97 lat (usec): min=2475, max=43210, avg=9306.24, stdev=5698.00 clat percentiles (usec): | 1.00th=[ 2704], 5.00th=[ 3130], 10.00th=[ 3621], 20.00th=[ 4817], | 30.00th=[ 5997], 40.00th=[ 7242], 50.00th=[ 8094], 60.00th=[ 9241], | 70.00th=[10683], 80.00th=[12911], 90.00th=[16057], 95.00th=[17957], | 99.00th=[34341], 99.50th=[40633], 99.90th=[43254], 99.95th=[43254], | 99.99th=[43254] bw ( KiB/s): min= 768, max= 1280, per=25.47%, avg=1015.58, stdev=122.85, samples=12 iops : min= 48, max= 80, avg=63.33, stdev= 7.68, samples=12 lat (msec) : 2=2.08%, 4=26.56%, 10=45.31%, 20=22.92%, 50=3.12% cpu : usr=0.08%, sys=9.05%, ctx=1271, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=3790KiB/s (3881kB/s), 924KiB/s-990KiB/s (946kB/s-1014kB/s), io=23.4MiB (24.5MB), run=6203-6319msec WRITE: bw=3988KiB/s (4084kB/s), 955KiB/s-1052KiB/s (977kB/s-1078kB/s), io=24.6MiB (25.8MB), run=6203-6319msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=7452: Fri Aug 9 23:32:22 2024 read: IOPS=5, BW=5785KiB/s (5924kB/s)(7168KiB/1239msec) clat (msec): min=15, max=814, avg=138.60, stdev=298.61 lat (msec): min=15, max=814, avg=138.60, stdev=298.61 clat percentiles (msec): | 1.00th=[ 16], 5.00th=[ 16], 10.00th=[ 16], 20.00th=[ 18], | 30.00th=[ 19], 40.00th=[ 19], 50.00th=[ 20], 60.00th=[ 26], | 70.00th=[ 26], 80.00th=[ 59], 90.00th=[ 818], 95.00th=[ 818], | 99.00th=[ 818], 99.50th=[ 818], 99.90th=[ 818], 99.95th=[ 818], | 99.99th=[ 818] bw ( KiB/s): min= 4096, max= 6144, per=100.00%, avg=5120.00, stdev=1448.15, samples=2 iops : min= 4, max= 6, avg= 5.00, stdev= 1.41, samples=2 write: IOPS=4, BW=4132KiB/s (4232kB/s)(5120KiB/1239msec) clat (msec): min=17, max=106, avg=50.99, stdev=37.97 lat (msec): min=17, max=106, avg=51.05, stdev=37.97 clat percentiles (msec): | 1.00th=[ 18], 5.00th=[ 18], 10.00th=[ 18], 20.00th=[ 18], | 30.00th=[ 20], 40.00th=[ 20], 50.00th=[ 41], 60.00th=[ 41], | 70.00th=[ 71], 80.00th=[ 71], 90.00th=[ 107], 95.00th=[ 107], | 99.00th=[ 107], 99.50th=[ 107], 99.90th=[ 107], 99.95th=[ 107], | 99.99th=[ 107] bw ( KiB/s): min= 2048, max= 2048, per=31.39%, avg=2048.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 20=50.00%, 50=16.67%, 100=16.67%, 250=8.33%, 1000=8.33% cpu : usr=0.00%, sys=7.84%, ctx=48, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=7453: Fri Aug 9 23:32:22 2024 read: IOPS=0, BW=572KiB/s (586kB/s)(2048KiB/3580msec) clat (usec): min=41543, max=71864, avg=56703.99, stdev=21440.67 lat (usec): min=41543, max=71866, avg=56705.00, stdev=21441.29 clat percentiles (usec): | 1.00th=[41681], 5.00th=[41681], 10.00th=[41681], 20.00th=[41681], | 30.00th=[41681], 40.00th=[41681], 50.00th=[41681], 60.00th=[71828], | 70.00th=[71828], 80.00th=[71828], 90.00th=[71828], 95.00th=[71828], | 99.00th=[71828], 99.50th=[71828], 99.90th=[71828], 99.95th=[71828], | 99.99th=[71828] bw ( KiB/s): min= 2043, max= 2048, per=43.89%, avg=2045.50, stdev= 3.54, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 write: IOPS=2, BW=2860KiB/s (2929kB/s)(10.0MiB/3580msec) clat (msec): min=18, max=1445, avg=342.09, stdev=513.94 lat (msec): min=18, max=1445, avg=342.15, stdev=513.95 clat percentiles (msec): | 1.00th=[ 18], 5.00th=[ 18], 10.00th=[ 18], 20.00th=[ 25], | 30.00th=[ 29], 40.00th=[ 32], 50.00th=[ 53], 60.00th=[ 58], | 70.00th=[ 852], 80.00th=[ 852], 90.00th=[ 852], 95.00th=[ 1452], | 99.00th=[ 1452], 99.50th=[ 1452], 99.90th=[ 1452], 99.95th=[ 1452], | 99.99th=[ 1452] bw ( KiB/s): min= 2048, max=12263, per=94.03%, avg=6135.67, stdev=5404.33, samples=3 iops : min= 2, max= 11, avg= 5.67, stdev= 4.73, samples=3 lat (msec) : 20=8.33%, 50=33.33%, 100=33.33%, 1000=16.67% cpu : usr=0.00%, sys=3.38%, ctx=47, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=7455: Fri Aug 9 23:32:22 2024 read: IOPS=1, BW=1689KiB/s (1730kB/s)(6144KiB/3637msec) clat (msec): min=12, max=1433, avg=382.07, stdev=575.01 lat (msec): min=12, max=1433, avg=382.07, stdev=575.01 clat percentiles (msec): | 1.00th=[ 13], 5.00th=[ 13], 10.00th=[ 13], 20.00th=[ 15], | 30.00th=[ 15], 40.00th=[ 24], 50.00th=[ 24], 60.00th=[ 136], | 70.00th=[ 676], 80.00th=[ 676], 90.00th=[ 1435], 95.00th=[ 1435], | 99.00th=[ 1435], 99.50th=[ 1435], 99.90th=[ 1435], 99.95th=[ 1435], | 99.99th=[ 1435] bw ( KiB/s): min= 2048, max= 8192, per=100.00%, avg=5120.00, stdev=4344.46, samples=2 iops : min= 2, max= 8, avg= 5.00, stdev= 4.24, samples=2 write: IOPS=1, BW=1689KiB/s (1730kB/s)(6144KiB/3637msec) clat (msec): min=16, max=944, avg=213.93, stdev=359.51 lat (msec): min=16, max=944, avg=214.00, stdev=359.51 clat percentiles (msec): | 1.00th=[ 17], 5.00th=[ 17], 10.00th=[ 17], 20.00th=[ 57], | 30.00th=[ 57], 40.00th=[ 64], 50.00th=[ 64], 60.00th=[ 84], | 70.00th=[ 120], 80.00th=[ 120], 90.00th=[ 944], 95.00th=[ 944], | 99.00th=[ 944], 99.50th=[ 944], 99.90th=[ 944], 99.95th=[ 944], | 99.99th=[ 944] bw ( KiB/s): min= 2048, max= 6144, per=62.77%, avg=4096.00, stdev=2048.00, samples=3 iops : min= 2, max= 6, avg= 4.00, stdev= 2.00, samples=3 lat (msec) : 20=25.00%, 50=8.33%, 100=25.00%, 250=16.67%, 750=8.33% lat (msec) : 1000=8.33% cpu : usr=0.03%, sys=3.33%, ctx=49, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=7457: Fri Aug 9 23:32:22 2024 read: IOPS=1, BW=1165KiB/s (1193kB/s)(5120KiB/4394msec) clat (msec): min=26, max=520, avg=157.54, stdev=205.54 lat (msec): min=26, max=520, avg=157.54, stdev=205.54 clat percentiles (msec): | 1.00th=[ 27], 5.00th=[ 27], 10.00th=[ 27], 20.00th=[ 27], | 30.00th=[ 43], 40.00th=[ 43], 50.00th=[ 91], 60.00th=[ 91], | 70.00th=[ 109], 80.00th=[ 109], 90.00th=[ 523], 95.00th=[ 523], | 99.00th=[ 523], 99.50th=[ 523], 99.90th=[ 523], 99.95th=[ 523], | 99.99th=[ 523] bw ( KiB/s): min= 2048, max= 6144, per=87.90%, avg=4096.00, stdev=2896.31, samples=2 iops : min= 2, max= 6, avg= 4.00, stdev= 2.83, samples=2 write: IOPS=1, BW=1631KiB/s (1670kB/s)(7168KiB/4394msec) clat (msec): min=51, max=1419, avg=509.02, stdev=582.40 lat (msec): min=51, max=1419, avg=509.07, stdev=582.40 clat percentiles (msec): | 1.00th=[ 52], 5.00th=[ 52], 10.00th=[ 52], 20.00th=[ 55], | 30.00th=[ 57], 40.00th=[ 57], 50.00th=[ 73], 60.00th=[ 936], | 70.00th=[ 936], 80.00th=[ 969], 90.00th=[ 1418], 95.00th=[ 1418], | 99.00th=[ 1418], 99.50th=[ 1418], 99.90th=[ 1418], 99.95th=[ 1418], | 99.99th=[ 1418] bw ( KiB/s): min= 2048, max= 6144, per=62.77%, avg=4096.00, stdev=2048.00, samples=3 iops : min= 2, max= 6, avg= 4.00, stdev= 2.00, samples=3 lat (msec) : 50=16.67%, 100=41.67%, 250=8.33%, 750=8.33%, 1000=16.67% cpu : usr=0.00%, sys=3.28%, ctx=47, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=4661KiB/s (4773kB/s), 572KiB/s-5785KiB/s (586kB/s-5924kB/s), io=20.0MiB (20.0MB), run=1239-4394msec WRITE: bw=6525KiB/s (6682kB/s), 1631KiB/s-4132KiB/s (1670kB/s-4232kB/s), io=28.0MiB (29.4MB), run=1239-4394msec rand-rw: (groupid=0, jobs=1): err= 0: pid=7454: Fri Aug 9 23:32:22 2024 read: IOPS=1, BW=1407KiB/s (1441kB/s)(7168KiB/5095msec) clat (usec): min=525, max=895621, avg=597355.30, stdev=397577.23 lat (usec): min=526, max=895637, avg=597358.73, stdev=397579.25 clat percentiles (usec): | 1.00th=[ 529], 5.00th=[ 529], 10.00th=[ 529], 20.00th=[ 41681], | 30.00th=[734004], 40.00th=[734004], 50.00th=[801113], 60.00th=[826278], | 70.00th=[826278], 80.00th=[884999], 90.00th=[893387], 95.00th=[893387], | 99.00th=[893387], 99.50th=[893387], 99.90th=[893387], 99.95th=[893387], | 99.99th=[893387] bw ( KiB/s): min= 1812, max= 2048, per=49.89%, avg=2005.17, stdev=95.00, samples=6 iops : min= 1, max= 2, avg= 1.67, stdev= 0.52, samples=6 write: IOPS=0, BW=1005KiB/s (1029kB/s)(5120KiB/5095msec) clat (msec): min=25, max=686, avg=173.96, stdev=287.26 lat (msec): min=25, max=686, avg=174.03, stdev=287.24 clat percentiles (msec): | 1.00th=[ 26], 5.00th=[ 26], 10.00th=[ 26], 20.00th=[ 26], | 30.00th=[ 33], 40.00th=[ 33], 50.00th=[ 54], 60.00th=[ 54], | 70.00th=[ 71], 80.00th=[ 71], 90.00th=[ 684], 95.00th=[ 684], | 99.00th=[ 684], 99.50th=[ 684], 99.90th=[ 684], 99.95th=[ 684], | 99.99th=[ 684] bw ( KiB/s): min= 1812, max= 4055, per=46.89%, avg=2638.33, stdev=1232.53, samples=3 iops : min= 1, max= 3, avg= 2.00, stdev= 1.00, samples=3 lat (usec) : 750=8.33% lat (msec) : 50=25.00%, 100=16.67%, 750=16.67%, 1000=33.33% cpu : usr=0.06%, sys=16.71%, ctx=1435, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=7456: Fri Aug 9 23:32:22 2024 read: IOPS=0, BW=479KiB/s (490kB/s)(2048KiB/4278msec) clat (msec): min=70, max=747, avg=409.01, stdev=479.25 lat (msec): min=70, max=747, avg=409.01, stdev=479.25 clat percentiles (msec): | 1.00th=[ 70], 5.00th=[ 70], 10.00th=[ 70], 20.00th=[ 70], | 30.00th=[ 70], 40.00th=[ 70], 50.00th=[ 70], 60.00th=[ 751], | 70.00th=[ 751], 80.00th=[ 751], 90.00th=[ 751], 95.00th=[ 751], | 99.00th=[ 751], 99.50th=[ 751], 99.90th=[ 751], 99.95th=[ 751], | 99.99th=[ 751] bw ( KiB/s): min= 2048, max= 2048, per=50.96%, avg=2048.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 write: IOPS=2, BW=2394KiB/s (2451kB/s)(10.0MiB/4278msec) clat (msec): min=24, max=891, avg=336.85, stdev=373.59 lat (msec): min=24, max=891, avg=336.92, stdev=373.58 clat percentiles (msec): | 1.00th=[ 25], 5.00th=[ 25], 10.00th=[ 25], 20.00th=[ 27], | 30.00th=[ 47], 40.00th=[ 62], 50.00th=[ 73], 60.00th=[ 107], | 70.00th=[ 743], 80.00th=[ 743], 90.00th=[ 852], 95.00th=[ 894], | 99.00th=[ 894], 99.50th=[ 894], 99.90th=[ 894], 99.95th=[ 894], | 99.99th=[ 894] bw ( KiB/s): min= 2048, max=10240, per=96.49%, avg=5429.33, stdev=4278.96, samples=3 iops : min= 2, max= 10, avg= 5.00, stdev= 4.36, samples=3 lat (msec) : 50=25.00%, 100=25.00%, 250=8.33%, 750=25.00%, 1000=16.67% cpu : usr=0.02%, sys=11.08%, ctx=490, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=7458: Fri Aug 9 23:32:22 2024 read: IOPS=1, BW=1706KiB/s (1747kB/s)(6144KiB/3601msec) clat (usec): min=455, max=1511.8k, avg=417437.12, stdev=602740.09 lat (usec): min=456, max=1511.8k, avg=417437.84, stdev=602740.11 clat percentiles (usec): | 1.00th=[ 457], 5.00th=[ 457], 10.00th=[ 457], | 20.00th=[ 60556], 30.00th=[ 60556], 40.00th=[ 69731], | 50.00th=[ 69731], 60.00th=[ 117965], 70.00th=[ 742392], | 80.00th=[ 742392], 90.00th=[1518339], 95.00th=[1518339], | 99.00th=[1518339], 99.50th=[1518339], 99.90th=[1518339], | 99.95th=[1518339], 99.99th=[1518339] bw ( KiB/s): min= 2048, max= 8192, per=100.00%, avg=5120.00, stdev=4344.46, samples=2 iops : min= 2, max= 8, avg= 5.00, stdev= 4.24, samples=2 write: IOPS=1, BW=1706KiB/s (1747kB/s)(6144KiB/3601msec) clat (msec): min=25, max=908, avg=181.24, stdev=356.21 lat (msec): min=25, max=908, avg=181.31, stdev=356.21 clat percentiles (msec): | 1.00th=[ 26], 5.00th=[ 26], 10.00th=[ 26], 20.00th=[ 33], | 30.00th=[ 33], 40.00th=[ 39], 50.00th=[ 39], 60.00th=[ 39], | 70.00th=[ 45], 80.00th=[ 45], 90.00th=[ 911], 95.00th=[ 911], | 99.00th=[ 911], 99.50th=[ 911], 99.90th=[ 911], 99.95th=[ 911], | 99.99th=[ 911] bw ( KiB/s): min= 2048, max= 4096, per=54.59%, avg=3072.00, stdev=1182.41, samples=4 iops : min= 2, max= 4, avg= 3.00, stdev= 1.15, samples=4 lat (usec) : 500=8.33% lat (msec) : 50=41.67%, 100=16.67%, 250=8.33%, 750=8.33%, 1000=8.33% cpu : usr=0.00%, sys=13.83%, ctx=787, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=7459: Fri Aug 9 23:32:22 2024 read: IOPS=1, BW=1214KiB/s (1243kB/s)(5120KiB/4219msec) clat (msec): min=8, max=552, avg=232.84, stdev=290.28 lat (msec): min=8, max=552, avg=232.84, stdev=290.28 clat percentiles (msec): | 1.00th=[ 9], 5.00th=[ 9], 10.00th=[ 9], 20.00th=[ 9], | 30.00th=[ 18], 40.00th=[ 18], 50.00th=[ 37], 60.00th=[ 37], | 70.00th=[ 550], 80.00th=[ 550], 90.00th=[ 550], 95.00th=[ 550], | 99.00th=[ 550], 99.50th=[ 550], 99.90th=[ 550], 99.95th=[ 550], | 99.99th=[ 550] bw ( KiB/s): min= 2048, max= 6144, per=100.00%, avg=4096.00, stdev=2896.31, samples=2 iops : min= 2, max= 6, avg= 4.00, stdev= 2.83, samples=2 write: IOPS=1, BW=1699KiB/s (1740kB/s)(7168KiB/4219msec) clat (msec): min=36, max=942, avg=430.21, stdev=458.34 lat (msec): min=37, max=942, avg=430.27, stdev=458.34 clat percentiles (msec): | 1.00th=[ 37], 5.00th=[ 37], 10.00th=[ 37], 20.00th=[ 51], | 30.00th=[ 70], 40.00th=[ 70], 50.00th=[ 96], 60.00th=[ 885], | 70.00th=[ 885], 80.00th=[ 936], 90.00th=[ 944], 95.00th=[ 944], | 99.00th=[ 944], 99.50th=[ 944], 99.90th=[ 944], 99.95th=[ 944], | 99.99th=[ 944] bw ( KiB/s): min= 2043, max= 6144, per=63.67%, avg=3582.75, stdev=1962.12, samples=4 iops : min= 1, max= 6, avg= 3.25, stdev= 2.22, samples=4 lat (msec) : 10=8.33%, 20=8.33%, 50=16.67%, 100=25.00%, 750=16.67% lat (msec) : 1000=25.00% cpu : usr=0.00%, sys=8.46%, ctx=580, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=4020KiB/s (4116kB/s), 479KiB/s-1706KiB/s (490kB/s-1747kB/s), io=20.0MiB (20.0MB), run=3601-5095msec WRITE: bw=5627KiB/s (5763kB/s), 1005KiB/s-2394KiB/s (1029kB/s-2451kB/s), io=28.0MiB (29.4MB), run=3601-5095msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=7468: Fri Aug 9 23:32:27 2024 read: IOPS=0, BW=3698KiB/s (3787kB/s)(12.0MiB/3323msec) clat (msec): min=63, max=2107, avg=1096.45, stdev=1022.12 lat (msec): min=63, max=2107, avg=1096.46, stdev=1022.12 clat percentiles (msec): | 1.00th=[ 64], 5.00th=[ 64], 10.00th=[ 64], 20.00th=[ 64], | 30.00th=[ 64], 40.00th=[ 1116], 50.00th=[ 1116], 60.00th=[ 1116], | 70.00th=[ 2106], 80.00th=[ 2106], 90.00th=[ 2106], 95.00th=[ 2106], | 99.00th=[ 2106], 99.50th=[ 2106], 99.90th=[ 2106], 99.95th=[ 2106], | 99.99th=[ 2106] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 lat (msec) : 100=33.33% cpu : usr=0.03%, sys=21.70%, ctx=1506, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=7469: Fri Aug 9 23:32:27 2024 read: IOPS=0, BW=1739KiB/s (1781kB/s)(4096KiB/2355msec) clat (nsec): min=43717k, max=43717k, avg=43717028.00, stdev= 0.00 lat (nsec): min=43719k, max=43719k, avg=43719125.00, stdev= 0.00 clat percentiles (usec): | 1.00th=[43779], 5.00th=[43779], 10.00th=[43779], 20.00th=[43779], | 30.00th=[43779], 40.00th=[43779], 50.00th=[43779], 60.00th=[43779], | 70.00th=[43779], 80.00th=[43779], 90.00th=[43779], 95.00th=[43779], | 99.00th=[43779], 99.50th=[43779], 99.90th=[43779], 99.95th=[43779], | 99.99th=[43779] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=0, BW=3479KiB/s (3562kB/s)(8192KiB/2355msec) clat (msec): min=187, max=2069, avg=1128.29, stdev=1330.51 lat (msec): min=187, max=2069, avg=1128.47, stdev=1330.64 clat percentiles (msec): | 1.00th=[ 188], 5.00th=[ 188], 10.00th=[ 188], 20.00th=[ 188], | 30.00th=[ 188], 40.00th=[ 188], 50.00th=[ 188], 60.00th=[ 2072], | 70.00th=[ 2072], 80.00th=[ 2072], 90.00th=[ 2072], 95.00th=[ 2072], | 99.00th=[ 2072], 99.50th=[ 2072], 99.90th=[ 2072], 99.95th=[ 2072], | 99.99th=[ 2072] bw ( KiB/s): min= 8192, max= 8192, per=87.83%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 50=33.33%, 250=33.33% cpu : usr=0.00%, sys=6.50%, ctx=90, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=7470: Fri Aug 9 23:32:27 2024 write: IOPS=0, BW=3498KiB/s (3582kB/s)(12.0MiB/3513msec) clat (msec): min=184, max=2083, avg=1160.22, stdev=950.27 lat (msec): min=185, max=2083, avg=1160.58, stdev=950.11 clat percentiles (msec): | 1.00th=[ 186], 5.00th=[ 186], 10.00th=[ 186], 20.00th=[ 186], | 30.00th=[ 186], 40.00th=[ 1217], 50.00th=[ 1217], 60.00th=[ 1217], | 70.00th=[ 2089], 80.00th=[ 2089], 90.00th=[ 2089], 95.00th=[ 2089], | 99.00th=[ 2089], 99.50th=[ 2089], 99.90th=[ 2089], 99.95th=[ 2089], | 99.99th=[ 2089] bw ( KiB/s): min= 8192, max= 8192, per=87.83%, avg=8192.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 lat (msec) : 250=33.33% cpu : usr=0.00%, sys=9.14%, ctx=50, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=7472: Fri Aug 9 23:32:27 2024 write: IOPS=1, BW=5272KiB/s (5398kB/s)(12.0MiB/2331msec) clat (msec): min=132, max=1139, avg=770.73, stdev=554.83 lat (msec): min=133, max=1139, avg=771.15, stdev=554.65 clat percentiles (msec): | 1.00th=[ 133], 5.00th=[ 133], 10.00th=[ 133], 20.00th=[ 133], | 30.00th=[ 133], 40.00th=[ 1045], 50.00th=[ 1045], 60.00th=[ 1045], | 70.00th=[ 1133], 80.00th=[ 1133], 90.00th=[ 1133], 95.00th=[ 1133], | 99.00th=[ 1133], 99.50th=[ 1133], 99.90th=[ 1133], 99.95th=[ 1133], | 99.99th=[ 1133] bw ( KiB/s): min= 8192, max= 8192, per=87.83%, avg=8192.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 lat (msec) : 250=33.33% cpu : usr=0.00%, sys=10.99%, ctx=44, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=4930KiB/s (5049kB/s), 1739KiB/s-3698KiB/s (1781kB/s-3787kB/s), io=16.0MiB (16.8MB), run=2355-3323msec WRITE: bw=9328KiB/s (9552kB/s), 3479KiB/s-5272KiB/s (3562kB/s-5398kB/s), io=32.0MiB (33.6MB), run=2331-3513msec rand-rw: (groupid=0, jobs=1): err= 0: pid=7473: Fri Aug 9 23:32:27 2024 read: IOPS=0, BW=3469KiB/s (3552kB/s)(12.0MiB/3542msec) clat (msec): min=320, max=2073, avg=1150.31, stdev=880.11 lat (msec): min=320, max=2073, avg=1150.31, stdev=880.11 clat percentiles (msec): | 1.00th=[ 321], 5.00th=[ 321], 10.00th=[ 321], 20.00th=[ 321], | 30.00th=[ 321], 40.00th=[ 1053], 50.00th=[ 1053], 60.00th=[ 1053], | 70.00th=[ 2072], 80.00th=[ 2072], 90.00th=[ 2072], 95.00th=[ 2072], | 99.00th=[ 2072], 99.50th=[ 2072], 99.90th=[ 2072], 99.95th=[ 2072], | 99.99th=[ 2072] bw ( KiB/s): min=16384, max=16384, per=100.00%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 500=33.33% cpu : usr=0.00%, sys=2.46%, ctx=39, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=7474: Fri Aug 9 23:32:27 2024 read: IOPS=0, BW=1192KiB/s (1221kB/s)(4096KiB/3436msec) clat (nsec): min=1179.4M, max=1179.4M, avg=1179448930.00, stdev= 0.00 lat (nsec): min=1179.5M, max=1179.5M, avg=1179450087.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 1183], 5.00th=[ 1183], 10.00th=[ 1183], 20.00th=[ 1183], | 30.00th=[ 1183], 40.00th=[ 1183], 50.00th=[ 1183], 60.00th=[ 1183], | 70.00th=[ 1183], 80.00th=[ 1183], 90.00th=[ 1183], 95.00th=[ 1183], | 99.00th=[ 1183], 99.50th=[ 1183], 99.90th=[ 1183], 99.95th=[ 1183], | 99.99th=[ 1183] bw ( KiB/s): min= 8175, max= 8175, per=100.00%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=0, BW=2384KiB/s (2441kB/s)(8192KiB/3436msec) clat (msec): min=95, max=2117, avg=1106.71, stdev=1429.90 lat (msec): min=96, max=2118, avg=1107.11, stdev=1429.89 clat percentiles (msec): | 1.00th=[ 96], 5.00th=[ 96], 10.00th=[ 96], 20.00th=[ 96], | 30.00th=[ 96], 40.00th=[ 96], 50.00th=[ 96], 60.00th=[ 2123], | 70.00th=[ 2123], 80.00th=[ 2123], 90.00th=[ 2123], 95.00th=[ 2123], | 99.00th=[ 2123], 99.50th=[ 2123], 99.90th=[ 2123], 99.95th=[ 2123], | 99.99th=[ 2123] lat (msec) : 100=33.33% cpu : usr=0.00%, sys=4.05%, ctx=97, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=7475: Fri Aug 9 23:32:27 2024 write: IOPS=0, BW=3458KiB/s (3541kB/s)(12.0MiB/3553msec) clat (msec): min=314, max=2093, avg=1151.81, stdev=893.68 lat (msec): min=315, max=2093, avg=1152.09, stdev=893.67 clat percentiles (msec): | 1.00th=[ 317], 5.00th=[ 317], 10.00th=[ 317], 20.00th=[ 317], | 30.00th=[ 317], 40.00th=[ 1045], 50.00th=[ 1045], 60.00th=[ 1045], | 70.00th=[ 2089], 80.00th=[ 2089], 90.00th=[ 2089], 95.00th=[ 2089], | 99.00th=[ 2089], 99.50th=[ 2089], 99.90th=[ 2089], 99.95th=[ 2089], | 99.99th=[ 2089] bw ( KiB/s): min= 7571, max= 8192, per=85.46%, avg=7881.50, stdev=439.11, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 500=33.33% cpu : usr=0.06%, sys=1.86%, ctx=24, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=7476: Fri Aug 9 23:32:27 2024 write: IOPS=1, BW=4741KiB/s (4855kB/s)(12.0MiB/2592msec) clat (msec): min=67, max=1296, avg=830.47, stdev=665.85 lat (msec): min=68, max=1297, avg=830.78, stdev=665.92 clat percentiles (msec): | 1.00th=[ 68], 5.00th=[ 68], 10.00th=[ 68], 20.00th=[ 68], | 30.00th=[ 68], 40.00th=[ 1133], 50.00th=[ 1133], 60.00th=[ 1133], | 70.00th=[ 1301], 80.00th=[ 1301], 90.00th=[ 1301], 95.00th=[ 1301], | 99.00th=[ 1301], 99.50th=[ 1301], 99.90th=[ 1301], 99.95th=[ 1301], | 99.99th=[ 1301] bw ( KiB/s): min= 8192, max= 8192, per=88.83%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33% cpu : usr=0.00%, sys=5.09%, ctx=141, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=4626KiB/s (4737kB/s), 1192KiB/s-3469KiB/s (1221kB/s-3552kB/s), io=16.0MiB (16.8MB), run=3436-3542msec WRITE: bw=9223KiB/s (9444kB/s), 2384KiB/s-4741KiB/s (2441kB/s-4855kB/s), io=32.0MiB (33.6MB), run=2592-3553msec