== sanity test 398b: DIO and buffer IO race ============== 03:54:27 (1732438467) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.16909 s, 43.1 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=14604: Sun Nov 24 03:54:42 2024 read: IOPS=120, BW=480KiB/s (492kB/s)(5992KiB/12482msec) clat (usec): min=56, max=152671, avg=4710.72, stdev=4569.23 lat (usec): min=57, max=152672, avg=4712.14, stdev=4569.40 clat percentiles (usec): | 1.00th=[ 60], 5.00th=[ 1860], 10.00th=[ 2278], 20.00th=[ 2737], | 30.00th=[ 3130], 40.00th=[ 3621], 50.00th=[ 4146], 60.00th=[ 4752], | 70.00th=[ 5538], 80.00th=[ 6325], 90.00th=[ 7504], 95.00th=[ 8717], | 99.00th=[ 13960], 99.50th=[ 16188], 99.90th=[ 17695], 99.95th=[152044], | 99.99th=[152044] bw ( KiB/s): min= 304, max= 568, per=25.12%, avg=473.58, stdev=62.49, samples=24 iops : min= 76, max= 142, avg=118.33, stdev=15.68, samples=24 write: IOPS=126, BW=504KiB/s (517kB/s)(6296KiB/12482msec) clat (usec): min=439, max=29221, avg=3405.12, stdev=3150.99 lat (usec): min=440, max=29222, avg=3406.16, stdev=3151.37 clat percentiles (usec): | 1.00th=[ 502], 5.00th=[ 537], 10.00th=[ 562], 20.00th=[ 635], | 30.00th=[ 807], 40.00th=[ 1532], 50.00th=[ 2769], 60.00th=[ 3392], | 70.00th=[ 4359], 80.00th=[ 5800], 90.00th=[ 7898], 95.00th=[ 9372], | 99.00th=[13173], 99.50th=[14615], 99.90th=[22414], 99.95th=[29230], | 99.99th=[29230] bw ( KiB/s): min= 360, max= 648, per=25.11%, avg=495.50, stdev=61.65, samples=24 iops : min= 90, max= 162, avg=123.83, stdev=15.38, samples=24 lat (usec) : 100=1.73%, 250=0.42%, 500=0.55%, 750=13.48%, 1000=4.30% lat (msec) : 2=4.10%, 4=32.65%, 10=39.29%, 20=3.35%, 50=0.10% lat (msec) : 250=0.03% cpu : usr=0.19%, sys=22.82%, ctx=5726, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=14605: Sun Nov 24 03:54:42 2024 read: IOPS=117, BW=471KiB/s (483kB/s)(5940KiB/12602msec) clat (usec): min=56, max=19593, avg=4571.66, stdev=2470.89 lat (usec): min=56, max=19593, avg=4572.69, stdev=2471.50 clat percentiles (usec): | 1.00th=[ 61], 5.00th=[ 1942], 10.00th=[ 2278], 20.00th=[ 2737], | 30.00th=[ 3130], 40.00th=[ 3556], 50.00th=[ 4146], 60.00th=[ 4686], | 70.00th=[ 5407], 80.00th=[ 6259], 90.00th=[ 7504], 95.00th=[ 8586], | 99.00th=[13042], 99.50th=[15795], 99.90th=[18482], 99.95th=[19530], | 99.99th=[19530] bw ( KiB/s): min= 328, max= 592, per=24.91%, avg=469.64, stdev=53.74, samples=25 iops : min= 82, max= 148, avg=117.36, stdev=13.43, samples=25 write: IOPS=125, BW=504KiB/s (516kB/s)(6348KiB/12602msec) clat (usec): min=406, max=151282, avg=3621.29, stdev=4830.55 lat (usec): min=406, max=151283, avg=3622.46, stdev=4830.57 clat percentiles (usec): | 1.00th=[ 498], 5.00th=[ 537], 10.00th=[ 570], 20.00th=[ 676], | 30.00th=[ 898], 40.00th=[ 2147], 50.00th=[ 2900], 60.00th=[ 3523], | 70.00th=[ 4555], 80.00th=[ 5866], 90.00th=[ 7767], 95.00th=[ 9765], | 99.00th=[ 13566], 99.50th=[ 14746], 99.90th=[ 20579], 99.95th=[152044], | 99.99th=[152044] bw ( KiB/s): min= 360, max= 640, per=25.33%, avg=499.76, stdev=77.29, samples=25 iops : min= 90, max= 160, avg=124.88, stdev=19.37, samples=25 lat (usec) : 100=1.66%, 250=0.36%, 500=0.59%, 750=11.75%, 1000=4.30% lat (msec) : 2=4.26%, 4=33.56%, 10=39.91%, 20=3.55%, 50=0.03% lat (msec) : 250=0.03% cpu : usr=0.20%, sys=22.84%, ctx=5893, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=14606: Sun Nov 24 03:54:42 2024 read: IOPS=118, BW=473KiB/s (485kB/s)(6028KiB/12737msec) clat (usec): min=49, max=19834, avg=4597.68, stdev=2539.34 lat (usec): min=49, max=19835, avg=4598.77, stdev=2539.48 clat percentiles (usec): | 1.00th=[ 60], 5.00th=[ 127], 10.00th=[ 2245], 20.00th=[ 2671], | 30.00th=[ 3130], 40.00th=[ 3523], 50.00th=[ 4178], 60.00th=[ 4817], | 70.00th=[ 5604], 80.00th=[ 6456], 90.00th=[ 7701], 95.00th=[ 8848], | 99.00th=[13698], 99.50th=[14746], 99.90th=[19792], 99.95th=[19792], | 99.99th=[19792] bw ( KiB/s): min= 240, max= 592, per=24.70%, avg=465.60, stdev=70.58, samples=25 iops : min= 60, max= 148, avg=116.40, stdev=17.64, samples=25 write: IOPS=122, BW=491KiB/s (503kB/s)(6260KiB/12737msec) clat (usec): min=432, max=161969, avg=3665.93, stdev=5151.57 lat (usec): min=432, max=161970, avg=3666.96, stdev=5151.45 clat percentiles (usec): | 1.00th=[ 506], 5.00th=[ 537], 10.00th=[ 578], 20.00th=[ 676], | 30.00th=[ 914], 40.00th=[ 1811], 50.00th=[ 2769], 60.00th=[ 3458], | 70.00th=[ 4686], 80.00th=[ 6063], 90.00th=[ 7963], 95.00th=[ 10028], | 99.00th=[ 13698], 99.50th=[ 16450], 99.90th=[ 23462], 99.95th=[162530], | 99.99th=[162530] bw ( KiB/s): min= 320, max= 600, per=24.34%, avg=480.32, stdev=73.56, samples=25 iops : min= 80, max= 150, avg=120.08, stdev=18.39, samples=25 lat (usec) : 50=0.03%, 100=2.08%, 250=0.46%, 500=0.46%, 750=11.65% lat (usec) : 1000=4.72% lat (msec) : 2=4.52%, 4=32.62%, 10=39.42%, 20=3.94%, 50=0.07% lat (msec) : 250=0.03% cpu : usr=0.12%, sys=22.93%, ctx=5866, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=14608: Sun Nov 24 03:54:42 2024 read: IOPS=120, BW=481KiB/s (492kB/s)(6052KiB/12584msec) clat (usec): min=47, max=17852, avg=4538.92, stdev=2356.90 lat (usec): min=47, max=17853, avg=4539.98, stdev=2357.11 clat percentiles (usec): | 1.00th=[ 59], 5.00th=[ 1745], 10.00th=[ 2311], 20.00th=[ 2769], | 30.00th=[ 3097], 40.00th=[ 3621], 50.00th=[ 4146], 60.00th=[ 4817], | 70.00th=[ 5473], 80.00th=[ 6194], 90.00th=[ 7373], 95.00th=[ 8586], | 99.00th=[12518], 99.50th=[14353], 99.90th=[16909], 99.95th=[17957], | 99.99th=[17957] bw ( KiB/s): min= 303, max= 608, per=25.40%, avg=478.84, stdev=68.04, samples=25 iops : min= 75, max= 152, avg=119.60, stdev=17.10, samples=25 write: IOPS=123, BW=496KiB/s (507kB/s)(6236KiB/12584msec) clat (usec): min=449, max=153456, avg=3623.74, stdev=4909.55 lat (usec): min=450, max=153457, avg=3625.33, stdev=4909.38 clat percentiles (usec): | 1.00th=[ 506], 5.00th=[ 537], 10.00th=[ 578], 20.00th=[ 693], | 30.00th=[ 898], 40.00th=[ 2089], 50.00th=[ 2868], 60.00th=[ 3556], | 70.00th=[ 4555], 80.00th=[ 5932], 90.00th=[ 7898], 95.00th=[ 9634], | 99.00th=[ 13566], 99.50th=[ 16712], 99.90th=[ 21627], 99.95th=[154141], | 99.99th=[154141] bw ( KiB/s): min= 391, max= 680, per=25.05%, avg=494.20, stdev=66.99, samples=25 iops : min= 97, max= 170, avg=123.44, stdev=16.87, samples=25 lat (usec) : 50=0.03%, 100=1.92%, 250=0.29%, 500=0.49%, 750=11.72% lat (usec) : 1000=4.26% lat (msec) : 2=4.13%, 4=34.02%, 10=39.68%, 20=3.35%, 50=0.07% lat (msec) : 250=0.03% cpu : usr=0.23%, sys=22.90%, ctx=6008, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1885KiB/s (1930kB/s), 471KiB/s-481KiB/s (483kB/s-492kB/s), io=23.4MiB (24.6MB), run=12482-12737msec WRITE: bw=1974KiB/s (2021kB/s), 491KiB/s-504KiB/s (503kB/s-517kB/s), io=24.6MiB (25.7MB), run=12482-12737msec rand-rw: (groupid=0, jobs=1): err= 0: pid=14607: Sun Nov 24 03:54:46 2024 read: IOPS=93, BW=372KiB/s (381kB/s)(5992KiB/16107msec) clat (usec): min=1481, max=156079, avg=4387.41, stdev=5183.50 lat (usec): min=1482, max=156080, avg=4388.04, stdev=5183.52 clat percentiles (usec): | 1.00th=[ 1663], 5.00th=[ 1844], 10.00th=[ 1926], 20.00th=[ 2114], | 30.00th=[ 2278], 40.00th=[ 2507], 50.00th=[ 2835], 60.00th=[ 3359], | 70.00th=[ 4228], 80.00th=[ 5932], 90.00th=[ 9503], 95.00th=[ 11600], | 99.00th=[ 16712], 99.50th=[ 20317], 99.90th=[ 24773], 99.95th=[156238], | 99.99th=[156238] bw ( KiB/s): min= 183, max= 752, per=24.94%, avg=370.12, stdev=173.90, samples=32 iops : min= 45, max= 188, avg=92.44, stdev=43.52, samples=32 write: IOPS=97, BW=391KiB/s (400kB/s)(6296KiB/16107msec) clat (usec): min=2271, max=30156, avg=6021.85, stdev=3635.62 lat (usec): min=2272, max=30157, avg=6022.83, stdev=3635.54 clat percentiles (usec): | 1.00th=[ 2540], 5.00th=[ 2835], 10.00th=[ 3032], 20.00th=[ 3359], | 30.00th=[ 3589], 40.00th=[ 3884], 50.00th=[ 4490], 60.00th=[ 5473], | 70.00th=[ 6980], 80.00th=[ 8717], 90.00th=[10945], 95.00th=[13042], | 99.00th=[19006], 99.50th=[22414], 99.90th=[28443], 99.95th=[30278], | 99.99th=[30278] bw ( KiB/s): min= 199, max= 776, per=24.96%, avg=387.88, stdev=178.95, samples=32 iops : min= 49, max= 194, avg=96.88, stdev=44.81, samples=32 lat (msec) : 2=6.87%, 4=48.18%, 10=33.98%, 20=10.35%, 50=0.59% lat (msec) : 250=0.03% cpu : usr=0.15%, sys=11.11%, ctx=3547, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=14609: Sun Nov 24 03:54:46 2024 read: IOPS=92, BW=368KiB/s (377kB/s)(5940KiB/16141msec) clat (usec): min=1514, max=149833, avg=4371.49, stdev=5128.15 lat (usec): min=1515, max=149834, avg=4372.22, stdev=5128.21 clat percentiles (usec): | 1.00th=[ 1663], 5.00th=[ 1827], 10.00th=[ 1942], 20.00th=[ 2089], | 30.00th=[ 2245], 40.00th=[ 2474], 50.00th=[ 2737], 60.00th=[ 3261], | 70.00th=[ 4113], 80.00th=[ 6063], 90.00th=[ 9110], 95.00th=[ 11994], | 99.00th=[ 17957], 99.50th=[ 21627], 99.90th=[ 24249], 99.95th=[149947], | 99.99th=[149947] bw ( KiB/s): min= 168, max= 734, per=24.65%, avg=365.88, stdev=179.92, samples=32 iops : min= 42, max= 183, avg=91.41, stdev=44.98, samples=32 write: IOPS=98, BW=393KiB/s (403kB/s)(6348KiB/16141msec) clat (usec): min=2274, max=25362, avg=6042.05, stdev=3698.37 lat (usec): min=2274, max=25363, avg=6042.83, stdev=3698.33 clat percentiles (usec): | 1.00th=[ 2540], 5.00th=[ 2835], 10.00th=[ 3032], 20.00th=[ 3326], | 30.00th=[ 3556], 40.00th=[ 3916], 50.00th=[ 4424], 60.00th=[ 5276], | 70.00th=[ 7046], 80.00th=[ 8848], 90.00th=[11207], 95.00th=[13304], | 99.00th=[18744], 99.50th=[23200], 99.90th=[24773], 99.95th=[25297], | 99.99th=[25297] bw ( KiB/s): min= 216, max= 776, per=25.04%, avg=389.13, stdev=172.94, samples=32 iops : min= 54, max= 194, avg=97.22, stdev=43.22, samples=32 lat (msec) : 2=7.03%, 4=47.95%, 10=33.43%, 20=10.87%, 50=0.68% lat (msec) : 250=0.03% cpu : usr=0.14%, sys=11.00%, ctx=3572, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=14610: Sun Nov 24 03:54:46 2024 read: IOPS=93, BW=373KiB/s (382kB/s)(6028KiB/16175msec) clat (usec): min=1418, max=23423, avg=4356.66, stdev=3457.26 lat (usec): min=1419, max=23423, avg=4357.28, stdev=3457.26 clat percentiles (usec): | 1.00th=[ 1696], 5.00th=[ 1827], 10.00th=[ 1926], 20.00th=[ 2089], | 30.00th=[ 2245], 40.00th=[ 2474], 50.00th=[ 2835], 60.00th=[ 3326], | 70.00th=[ 4359], 80.00th=[ 6325], 90.00th=[ 9503], 95.00th=[11863], | 99.00th=[16909], 99.50th=[18744], 99.90th=[22938], 99.95th=[23462], | 99.99th=[23462] bw ( KiB/s): min= 160, max= 768, per=24.88%, avg=369.22, stdev=176.66, samples=32 iops : min= 40, max= 192, avg=92.28, stdev=44.18, samples=32 write: IOPS=96, BW=387KiB/s (396kB/s)(6260KiB/16175msec) clat (msec): min=2, max=158, avg= 6.10, stdev= 5.37 lat (msec): min=2, max=158, avg= 6.11, stdev= 5.37 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 4], 20.00th=[ 4], | 30.00th=[ 4], 40.00th=[ 4], 50.00th=[ 5], 60.00th=[ 6], | 70.00th=[ 7], 80.00th=[ 9], 90.00th=[ 12], 95.00th=[ 14], | 99.00th=[ 20], 99.50th=[ 24], 99.90th=[ 31], 99.95th=[ 159], | 99.99th=[ 159] bw ( KiB/s): min= 184, max= 776, per=24.61%, avg=382.47, stdev=181.87, samples=32 iops : min= 46, max= 194, avg=95.59, stdev=45.48, samples=32 lat (msec) : 2=7.00%, 4=48.34%, 10=33.04%, 20=11.00%, 50=0.59% lat (msec) : 250=0.03% cpu : usr=0.13%, sys=11.01%, ctx=3511, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=14611: Sun Nov 24 03:54:46 2024 read: IOPS=93, BW=376KiB/s (385kB/s)(6052KiB/16102msec) clat (usec): min=1493, max=24774, avg=4245.10, stdev=3420.10 lat (usec): min=1494, max=24775, avg=4245.85, stdev=3420.16 clat percentiles (usec): | 1.00th=[ 1696], 5.00th=[ 1844], 10.00th=[ 1942], 20.00th=[ 2114], | 30.00th=[ 2245], 40.00th=[ 2474], 50.00th=[ 2802], 60.00th=[ 3326], | 70.00th=[ 4047], 80.00th=[ 5669], 90.00th=[ 9110], 95.00th=[11863], | 99.00th=[16712], 99.50th=[20055], 99.90th=[23987], 99.95th=[24773], | 99.99th=[24773] bw ( KiB/s): min= 192, max= 744, per=25.13%, avg=372.88, stdev=169.36, samples=32 iops : min= 48, max= 186, avg=93.12, stdev=42.40, samples=32 write: IOPS=96, BW=387KiB/s (397kB/s)(6236KiB/16102msec) clat (msec): min=2, max=156, avg= 6.17, stdev= 5.30 lat (msec): min=2, max=156, avg= 6.17, stdev= 5.30 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 4], 20.00th=[ 4], | 30.00th=[ 4], 40.00th=[ 4], 50.00th=[ 5], 60.00th=[ 6], | 70.00th=[ 8], 80.00th=[ 9], 90.00th=[ 12], 95.00th=[ 14], | 99.00th=[ 20], 99.50th=[ 24], 99.90th=[ 29], 99.95th=[ 157], | 99.99th=[ 157] bw ( KiB/s): min= 208, max= 800, per=24.78%, avg=385.12, stdev=177.94, samples=32 iops : min= 52, max= 200, avg=96.19, stdev=44.53, samples=32 lat (msec) : 2=6.61%, 4=48.96%, 10=33.24%, 20=10.48%, 50=0.68% lat (msec) : 250=0.03% cpu : usr=0.19%, sys=11.04%, ctx=3548, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1485KiB/s (1520kB/s), 368KiB/s-376KiB/s (377kB/s-385kB/s), io=23.4MiB (24.6MB), run=16102-16175msec WRITE: bw=1554KiB/s (1592kB/s), 387KiB/s-393KiB/s (396kB/s-403kB/s), io=24.6MiB (25.7MB), run=16102-16175msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=14642: Sun Nov 24 03:54:53 2024 read: IOPS=60, BW=963KiB/s (986kB/s)(5920KiB/6146msec) clat (usec): min=1930, max=167174, avg=6481.35, stdev=9448.28 lat (usec): min=1930, max=167175, avg=6481.99, stdev=9448.33 clat percentiles (usec): | 1.00th=[ 1958], 5.00th=[ 2147], 10.00th=[ 2245], 20.00th=[ 2507], | 30.00th=[ 2868], 40.00th=[ 3556], 50.00th=[ 4359], 60.00th=[ 5604], | 70.00th=[ 7635], 80.00th=[ 9241], 90.00th=[ 11600], 95.00th=[ 14615], | 99.00th=[ 24511], 99.50th=[ 25560], 99.90th=[166724], 99.95th=[166724], | 99.99th=[166724] bw ( KiB/s): min= 480, max= 1213, per=24.35%, avg=948.83, stdev=211.33, samples=12 iops : min= 30, max= 75, avg=59.17, stdev=13.04, samples=12 write: IOPS=64, BW=1036KiB/s (1061kB/s)(6368KiB/6146msec) clat (usec): min=2519, max=43032, avg=9181.82, stdev=5016.18 lat (usec): min=2520, max=43033, avg=9183.03, stdev=5016.19 clat percentiles (usec): | 1.00th=[ 2835], 5.00th=[ 3359], 10.00th=[ 3851], 20.00th=[ 4621], | 30.00th=[ 5735], 40.00th=[ 7046], 50.00th=[ 8029], 60.00th=[ 9241], | 70.00th=[10945], 80.00th=[13435], 90.00th=[16057], 95.00th=[18744], | 99.00th=[22414], 99.50th=[24773], 99.90th=[43254], 99.95th=[43254], | 99.99th=[43254] bw ( KiB/s): min= 544, max= 1312, per=25.16%, avg=1031.58, stdev=232.99, samples=12 iops : min= 34, max= 82, avg=64.33, stdev=14.51, samples=12 lat (msec) : 2=0.65%, 4=27.47%, 10=45.70%, 20=23.83%, 50=2.21% lat (msec) : 250=0.13% cpu : usr=0.13%, sys=10.11%, ctx=1263, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=14643: Sun Nov 24 03:54:53 2024 read: IOPS=60, BW=974KiB/s (997kB/s)(5744KiB/5899msec) clat (usec): min=1760, max=41834, avg=5647.56, stdev=4534.78 lat (usec): min=1761, max=41835, avg=5648.18, stdev=4534.79 clat percentiles (usec): | 1.00th=[ 1860], 5.00th=[ 2089], 10.00th=[ 2212], 20.00th=[ 2409], | 30.00th=[ 2638], 40.00th=[ 3032], 50.00th=[ 3523], 60.00th=[ 4555], | 70.00th=[ 6980], 80.00th=[ 8848], 90.00th=[12256], 95.00th=[13829], | 99.00th=[19006], 99.50th=[28705], 99.90th=[41681], 99.95th=[41681], | 99.99th=[41681] bw ( KiB/s): min= 383, max= 1184, per=24.77%, avg=965.27, stdev=215.23, samples=11 iops : min= 23, max= 74, avg=60.09, stdev=13.69, samples=11 write: IOPS=69, BW=1109KiB/s (1136kB/s)(6544KiB/5899msec) clat (msec): min=2, max=158, avg= 9.22, stdev= 9.10 lat (msec): min=2, max=158, avg= 9.22, stdev= 9.10 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 4], 10.00th=[ 4], 20.00th=[ 5], | 30.00th=[ 6], 40.00th=[ 7], 50.00th=[ 8], 60.00th=[ 9], | 70.00th=[ 11], 80.00th=[ 14], 90.00th=[ 17], 95.00th=[ 20], | 99.00th=[ 25], 99.50th=[ 28], 99.90th=[ 159], 99.95th=[ 159], | 99.99th=[ 159] bw ( KiB/s): min= 638, max= 1440, per=26.31%, avg=1078.55, stdev=223.41, samples=11 iops : min= 39, max= 90, avg=67.18, stdev=14.11, samples=11 lat (msec) : 2=1.04%, 4=33.20%, 10=41.93%, 20=20.83%, 50=2.86% lat (msec) : 250=0.13% cpu : usr=0.14%, sys=9.43%, ctx=1081, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=14644: Sun Nov 24 03:54:53 2024 read: IOPS=72, BW=1155KiB/s (1183kB/s)(6256KiB/5416msec) clat (usec): min=1767, max=28933, avg=5557.83, stdev=4283.70 lat (usec): min=1768, max=28934, avg=5558.62, stdev=4283.64 clat percentiles (usec): | 1.00th=[ 1958], 5.00th=[ 2114], 10.00th=[ 2278], 20.00th=[ 2540], | 30.00th=[ 2835], 40.00th=[ 3097], 50.00th=[ 3621], 60.00th=[ 4424], | 70.00th=[ 6456], 80.00th=[ 8848], 90.00th=[11076], 95.00th=[13829], | 99.00th=[20841], 99.50th=[25560], 99.90th=[28967], 99.95th=[28967], | 99.99th=[28967] bw ( KiB/s): min= 448, max= 1664, per=30.21%, avg=1177.30, stdev=344.49, samples=10 iops : min= 28, max= 104, avg=73.50, stdev=21.51, samples=10 write: IOPS=69, BW=1114KiB/s (1140kB/s)(6032KiB/5416msec) clat (msec): min=2, max=214, avg= 8.55, stdev=11.87 lat (msec): min=2, max=214, avg= 8.55, stdev=11.87 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 4], 10.00th=[ 4], 20.00th=[ 4], | 30.00th=[ 5], 40.00th=[ 6], 50.00th=[ 7], 60.00th=[ 8], | 70.00th=[ 10], 80.00th=[ 12], 90.00th=[ 16], 95.00th=[ 18], | 99.00th=[ 27], 99.50th=[ 46], 99.90th=[ 215], 99.95th=[ 215], | 99.99th=[ 215] bw ( KiB/s): min= 576, max= 1500, per=27.00%, avg=1106.80, stdev=269.28, samples=10 iops : min= 36, max= 93, avg=69.10, stdev=16.71, samples=10 lat (msec) : 2=1.04%, 4=38.54%, 10=39.19%, 20=18.75%, 50=2.34% lat (msec) : 250=0.13% cpu : usr=0.17%, sys=9.51%, ctx=955, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=14645: Sun Nov 24 03:54:53 2024 read: IOPS=63, BW=1009KiB/s (1033kB/s)(6032KiB/5979msec) clat (usec): min=1841, max=29161, avg=6101.08, stdev=4551.86 lat (usec): min=1842, max=29161, avg=6101.87, stdev=4551.84 clat percentiles (usec): | 1.00th=[ 1942], 5.00th=[ 2114], 10.00th=[ 2278], 20.00th=[ 2606], | 30.00th=[ 2999], 40.00th=[ 3359], 50.00th=[ 4113], 60.00th=[ 5932], | 70.00th=[ 7242], 80.00th=[ 9372], 90.00th=[11863], 95.00th=[15270], | 99.00th=[21365], 99.50th=[27395], 99.90th=[29230], 99.95th=[29230], | 99.99th=[29230] bw ( KiB/s): min= 448, max= 1152, per=25.30%, avg=985.82, stdev=217.59, samples=11 iops : min= 28, max= 72, avg=61.45, stdev=13.68, samples=11 write: IOPS=65, BW=1046KiB/s (1071kB/s)(6256KiB/5979msec) clat (msec): min=2, max=146, avg= 9.15, stdev= 8.87 lat (msec): min=2, max=146, avg= 9.15, stdev= 8.87 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 4], 10.00th=[ 4], 20.00th=[ 5], | 30.00th=[ 6], 40.00th=[ 7], 50.00th=[ 8], 60.00th=[ 9], | 70.00th=[ 11], 80.00th=[ 13], 90.00th=[ 17], 95.00th=[ 21], | 99.00th=[ 27], 99.50th=[ 51], 99.90th=[ 148], 99.95th=[ 148], | 99.99th=[ 148] bw ( KiB/s): min= 608, max= 1373, per=25.39%, avg=1041.00, stdev=201.14, samples=11 iops : min= 38, max= 85, avg=64.91, stdev=12.48, samples=11 lat (msec) : 2=1.04%, 4=31.64%, 10=43.36%, 20=20.31%, 50=3.52% lat (msec) : 250=0.13% cpu : usr=0.12%, sys=9.92%, ctx=1178, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=3897KiB/s (3991kB/s), 963KiB/s-1155KiB/s (986kB/s-1183kB/s), io=23.4MiB (24.5MB), run=5416-6146msec WRITE: bw=4100KiB/s (4199kB/s), 1036KiB/s-1114KiB/s (1061kB/s-1140kB/s), io=24.6MiB (25.8MB), run=5416-6146msec rand-rw: (groupid=0, jobs=1): err= 0: pid=14638: Sun Nov 24 03:54:53 2024 read: IOPS=60, BW=964KiB/s (987kB/s)(5920KiB/6139msec) clat (usec): min=61, max=125225, avg=11443.26, stdev=7173.61 lat (usec): min=61, max=125226, avg=11443.92, stdev=7173.64 clat percentiles (usec): | 1.00th=[ 79], 5.00th=[ 7111], 10.00th=[ 7963], 20.00th=[ 8979], | 30.00th=[ 9503], 40.00th=[ 10159], 50.00th=[ 10814], 60.00th=[ 11338], | 70.00th=[ 12125], 80.00th=[ 13173], 90.00th=[ 14615], 95.00th=[ 16712], | 99.00th=[ 27132], 99.50th=[ 51119], 99.90th=[125305], 99.95th=[125305], | 99.99th=[125305] bw ( KiB/s): min= 544, max= 1213, per=24.65%, avg=949.08, stdev=186.87, samples=12 iops : min= 34, max= 75, avg=59.25, stdev=11.58, samples=12 write: IOPS=64, BW=1037KiB/s (1062kB/s)(6368KiB/6139msec) clat (usec): min=689, max=21681, avg=4630.34, stdev=4020.32 lat (usec): min=690, max=21681, avg=4631.35, stdev=4020.31 clat percentiles (usec): | 1.00th=[ 750], 5.00th=[ 824], 10.00th=[ 889], 20.00th=[ 1385], | 30.00th=[ 2343], 40.00th=[ 2737], 50.00th=[ 3195], 60.00th=[ 4113], | 70.00th=[ 5211], 80.00th=[ 6783], 90.00th=[10945], 95.00th=[14877], | 99.00th=[18220], 99.50th=[19530], 99.90th=[21627], 99.95th=[21627], | 99.99th=[21627] bw ( KiB/s): min= 608, max= 1280, per=25.47%, avg=1031.83, stdev=211.61, samples=12 iops : min= 38, max= 80, avg=64.42, stdev=13.30, samples=12 lat (usec) : 100=0.78%, 250=0.13%, 500=0.13%, 750=0.52%, 1000=6.25% lat (msec) : 2=5.60%, 4=18.49%, 10=31.77%, 20=35.16%, 50=0.91% lat (msec) : 100=0.13%, 250=0.13% cpu : usr=0.08%, sys=20.63%, ctx=2794, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=14639: Sun Nov 24 03:54:53 2024 read: IOPS=61, BW=990KiB/s (1014kB/s)(5744KiB/5802msec) clat (usec): min=65, max=50573, avg=11064.76, stdev=3695.32 lat (usec): min=66, max=50575, avg=11065.38, stdev=3695.33 clat percentiles (usec): | 1.00th=[ 74], 5.00th=[ 7308], 10.00th=[ 8356], 20.00th=[ 8979], | 30.00th=[ 9765], 40.00th=[10290], 50.00th=[10814], 60.00th=[11338], | 70.00th=[11994], 80.00th=[12780], 90.00th=[13960], 95.00th=[15795], | 99.00th=[20579], 99.50th=[25560], 99.90th=[50594], 99.95th=[50594], | 99.99th=[50594] bw ( KiB/s): min= 510, max= 1216, per=25.30%, avg=974.18, stdev=178.38, samples=11 iops : min= 31, max= 76, avg=60.73, stdev=11.40, samples=11 write: IOPS=70, BW=1128KiB/s (1155kB/s)(6544KiB/5802msec) clat (usec): min=621, max=153638, avg=4434.91, stdev=8162.29 lat (usec): min=622, max=153639, avg=4435.96, stdev=8162.28 clat percentiles (usec): | 1.00th=[ 766], 5.00th=[ 816], 10.00th=[ 873], 20.00th=[ 1156], | 30.00th=[ 2114], 40.00th=[ 2442], 50.00th=[ 2835], 60.00th=[ 3589], | 70.00th=[ 4817], 80.00th=[ 6325], 90.00th=[ 8848], 95.00th=[ 12387], | 99.00th=[ 15270], 99.50th=[ 16909], 99.90th=[154141], 99.95th=[154141], | 99.99th=[154141] bw ( KiB/s): min= 830, max= 1472, per=27.35%, avg=1107.91, stdev=211.19, samples=11 iops : min= 51, max= 92, avg=69.09, stdev=13.32, samples=11 lat (usec) : 100=0.65%, 250=0.26%, 500=0.13%, 750=0.39%, 1000=8.20% lat (msec) : 2=6.51%, 4=19.14%, 10=30.21%, 20=33.85%, 50=0.39% lat (msec) : 100=0.13%, 250=0.13% cpu : usr=0.07%, sys=19.46%, ctx=2657, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=14640: Sun Nov 24 03:54:53 2024 read: IOPS=62, BW=1006KiB/s (1030kB/s)(6256KiB/6220msec) clat (usec): min=59, max=26797, avg=10841.60, stdev=3221.20 lat (usec): min=59, max=26798, avg=10842.32, stdev=3221.16 clat percentiles (usec): | 1.00th=[ 76], 5.00th=[ 7046], 10.00th=[ 7767], 20.00th=[ 8717], | 30.00th=[ 9372], 40.00th=[10028], 50.00th=[10552], 60.00th=[10945], | 70.00th=[11600], 80.00th=[12649], 90.00th=[14222], 95.00th=[16712], | 99.00th=[24773], 99.50th=[25035], 99.90th=[26870], 99.95th=[26870], | 99.99th=[26870] bw ( KiB/s): min= 448, max= 1408, per=25.42%, avg=978.67, stdev=228.92, samples=12 iops : min= 28, max= 88, avg=61.17, stdev=14.31, samples=12 write: IOPS=60, BW=970KiB/s (993kB/s)(6032KiB/6220msec) clat (usec): min=761, max=174371, avg=5038.70, stdev=9957.92 lat (usec): min=762, max=174373, avg=5041.26, stdev=9957.33 clat percentiles (usec): | 1.00th=[ 775], 5.00th=[ 816], 10.00th=[ 898], 20.00th=[ 1237], | 30.00th=[ 2212], 40.00th=[ 2540], 50.00th=[ 2933], 60.00th=[ 3621], | 70.00th=[ 4948], 80.00th=[ 6390], 90.00th=[ 10290], 95.00th=[ 14091], | 99.00th=[ 26084], 99.50th=[ 43779], 99.90th=[175113], 99.95th=[175113], | 99.99th=[175113] bw ( KiB/s): min= 544, max= 1248, per=22.78%, avg=922.67, stdev=206.41, samples=12 iops : min= 34, max= 78, avg=57.67, stdev=12.90, samples=12 lat (usec) : 100=0.65%, 250=0.13%, 1000=7.03% lat (msec) : 2=5.08%, 4=18.88%, 10=32.55%, 20=33.98%, 50=1.56% lat (msec) : 250=0.13% cpu : usr=0.06%, sys=18.20%, ctx=2741, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=14641: Sun Nov 24 03:54:53 2024 read: IOPS=63, BW=1013KiB/s (1037kB/s)(6032KiB/5955msec) clat (usec): min=61, max=29134, avg=10856.82, stdev=3394.55 lat (usec): min=62, max=29134, avg=10861.24, stdev=3399.56 clat percentiles (usec): | 1.00th=[ 63], 5.00th=[ 7373], 10.00th=[ 8094], 20.00th=[ 8848], | 30.00th=[ 9503], 40.00th=[10028], 50.00th=[10552], 60.00th=[11207], | 70.00th=[11994], 80.00th=[12911], 90.00th=[14222], 95.00th=[16057], | 99.00th=[21103], 99.50th=[25822], 99.90th=[29230], 99.95th=[29230], | 99.99th=[29230] bw ( KiB/s): min= 640, max= 1248, per=25.75%, avg=991.55, stdev=168.25, samples=11 iops : min= 40, max= 78, avg=61.82, stdev=10.59, samples=11 write: IOPS=65, BW=1051KiB/s (1076kB/s)(6256KiB/5955msec) clat (usec): min=719, max=56820, avg=4569.85, stdev=5632.09 lat (usec): min=720, max=56821, avg=4570.84, stdev=5632.09 clat percentiles (usec): | 1.00th=[ 766], 5.00th=[ 799], 10.00th=[ 873], 20.00th=[ 1123], | 30.00th=[ 1614], 40.00th=[ 2442], 50.00th=[ 2737], 60.00th=[ 3490], | 70.00th=[ 4752], 80.00th=[ 6259], 90.00th=[10421], 95.00th=[14222], | 99.00th=[22152], 99.50th=[48497], 99.90th=[56886], 99.95th=[56886], | 99.99th=[56886] bw ( KiB/s): min= 768, max= 1376, per=25.70%, avg=1041.09, stdev=176.94, samples=11 iops : min= 48, max= 86, avg=64.91, stdev=11.19, samples=11 lat (usec) : 100=0.91%, 250=0.39%, 500=0.13%, 750=0.26%, 1000=8.07% lat (msec) : 2=7.81%, 4=16.67%, 10=30.73%, 20=33.33%, 50=1.56% lat (msec) : 100=0.13% cpu : usr=0.18%, sys=19.14%, ctx=2647, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=3851KiB/s (3943kB/s), 964KiB/s-1013KiB/s (987kB/s-1037kB/s), io=23.4MiB (24.5MB), run=5802-6220msec WRITE: bw=4051KiB/s (4149kB/s), 970KiB/s-1128KiB/s (993kB/s-1155kB/s), io=24.6MiB (25.8MB), run=5802-6220msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=14658: Sun Nov 24 03:54:58 2024 read: IOPS=4, BW=4241KiB/s (4343kB/s)(7168KiB/1690msec) clat (usec): min=412, max=614996, avg=218651.43, stdev=283652.81 lat (usec): min=412, max=614997, avg=218652.59, stdev=283652.99 clat percentiles (usec): | 1.00th=[ 412], 5.00th=[ 412], 10.00th=[ 412], 20.00th=[ 433], | 30.00th=[ 437], 40.00th=[ 437], 50.00th=[ 44303], 60.00th=[263193], | 70.00th=[263193], 80.00th=[608175], 90.00th=[616563], 95.00th=[616563], | 99.00th=[616563], 99.50th=[616563], 99.90th=[616563], 99.95th=[616563], | 99.99th=[616563] bw ( KiB/s): min= 2043, max= 2048, per=40.37%, avg=2046.33, stdev= 2.89, samples=3 iops : min= 1, max= 2, avg= 1.67, stdev= 0.58, samples=3 write: IOPS=2, BW=3030KiB/s (3102kB/s)(5120KiB/1690msec) clat (usec): min=20657, max=43971, avg=29953.84, stdev=10816.31 lat (usec): min=20718, max=44033, avg=29998.08, stdev=10830.68 clat percentiles (usec): | 1.00th=[20579], 5.00th=[20579], 10.00th=[20579], 20.00th=[20579], | 30.00th=[22152], 40.00th=[22152], 50.00th=[23725], 60.00th=[23725], | 70.00th=[39060], 80.00th=[39060], 90.00th=[43779], 95.00th=[43779], | 99.00th=[43779], 99.50th=[43779], 99.90th=[43779], 99.95th=[43779], | 99.99th=[43779] bw ( KiB/s): min= 2048, max= 2048, per=28.86%, avg=2048.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (usec) : 500=25.00% lat (msec) : 50=50.00%, 500=8.33%, 750=16.67% cpu : usr=0.12%, sys=27.12%, ctx=706, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=14660: Sun Nov 24 03:54:58 2024 read: IOPS=0, BW=1016KiB/s (1040kB/s)(2048KiB/2016msec) clat (msec): min=99, max=152, avg=126.33, stdev=37.58 lat (msec): min=99, max=152, avg=126.33, stdev=37.58 clat percentiles (msec): | 1.00th=[ 101], 5.00th=[ 101], 10.00th=[ 101], 20.00th=[ 101], | 30.00th=[ 101], 40.00th=[ 101], 50.00th=[ 101], 60.00th=[ 153], | 70.00th=[ 153], 80.00th=[ 153], 90.00th=[ 153], 95.00th=[ 153], | 99.00th=[ 153], 99.50th=[ 153], 99.90th=[ 153], 99.95th=[ 153], | 99.99th=[ 153] bw ( KiB/s): min= 2048, max= 2048, per=40.40%, avg=2048.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 write: IOPS=4, BW=5079KiB/s (5201kB/s)(10.0MiB/2016msec) clat (msec): min=17, max=1145, avg=170.15, stdev=344.18 lat (msec): min=18, max=1145, avg=170.21, stdev=344.17 clat percentiles (msec): | 1.00th=[ 18], 5.00th=[ 18], 10.00th=[ 18], 20.00th=[ 27], | 30.00th=[ 31], 40.00th=[ 39], 50.00th=[ 62], 60.00th=[ 82], | 70.00th=[ 95], 80.00th=[ 95], 90.00th=[ 116], 95.00th=[ 1150], | 99.00th=[ 1150], 99.50th=[ 1150], 99.90th=[ 1150], 99.95th=[ 1150], | 99.99th=[ 1150] bw ( KiB/s): min= 8192, max=10240, per=100.00%, avg=9216.00, stdev=1448.15, samples=2 iops : min= 8, max= 10, avg= 9.00, stdev= 1.41, samples=2 lat (msec) : 20=8.33%, 50=25.00%, 100=41.67%, 250=16.67% cpu : usr=0.05%, sys=19.55%, ctx=139, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=14662: Sun Nov 24 03:54:58 2024 read: IOPS=1, BW=1521KiB/s (1557kB/s)(6144KiB/4040msec) clat (msec): min=47, max=627, avg=299.83, stdev=250.55 lat (msec): min=47, max=627, avg=299.83, stdev=250.55 clat percentiles (msec): | 1.00th=[ 48], 5.00th=[ 48], 10.00th=[ 48], 20.00th=[ 79], | 30.00th=[ 79], 40.00th=[ 112], 50.00th=[ 112], 60.00th=[ 426], | 70.00th=[ 510], 80.00th=[ 510], 90.00th=[ 625], 95.00th=[ 625], | 99.00th=[ 625], 99.50th=[ 625], 99.90th=[ 625], 99.95th=[ 625], | 99.99th=[ 625] bw ( KiB/s): min= 2043, max= 4096, per=50.48%, avg=2558.75, stdev=1024.84, samples=4 iops : min= 1, max= 4, avg= 2.25, stdev= 1.26, samples=4 write: IOPS=1, BW=1521KiB/s (1557kB/s)(6144KiB/4040msec) clat (msec): min=17, max=1234, avg=362.93, stdev=533.16 lat (msec): min=17, max=1234, avg=363.00, stdev=533.16 clat percentiles (msec): | 1.00th=[ 18], 5.00th=[ 18], 10.00th=[ 18], 20.00th=[ 28], | 30.00th=[ 28], 40.00th=[ 31], 50.00th=[ 31], 60.00th=[ 41], | 70.00th=[ 827], 80.00th=[ 827], 90.00th=[ 1234], 95.00th=[ 1234], | 99.00th=[ 1234], 99.50th=[ 1234], 99.90th=[ 1234], 99.95th=[ 1234], | 99.99th=[ 1234] bw ( KiB/s): min= 2043, max= 4096, per=34.61%, avg=2456.60, stdev=916.46, samples=5 iops : min= 1, max= 4, avg= 2.20, stdev= 1.10, samples=5 lat (msec) : 20=8.33%, 50=33.33%, 100=8.33%, 250=8.33%, 500=8.33% lat (msec) : 750=16.67%, 1000=8.33% cpu : usr=0.00%, sys=19.36%, ctx=723, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=14663: Sun Nov 24 03:54:58 2024 read: IOPS=1, BW=1268KiB/s (1299kB/s)(5120KiB/4037msec) clat (usec): min=411, max=710986, avg=372720.62, stdev=344969.51 lat (usec): min=412, max=710988, avg=372721.66, stdev=344969.53 clat percentiles (usec): | 1.00th=[ 412], 5.00th=[ 412], 10.00th=[ 412], 20.00th=[ 412], | 30.00th=[ 5276], 40.00th=[ 5276], 50.00th=[513803], 60.00th=[513803], | 70.00th=[633340], 80.00th=[633340], 90.00th=[708838], 95.00th=[708838], | 99.00th=[708838], 99.50th=[708838], 99.90th=[708838], 99.95th=[708838], | 99.99th=[708838] bw ( KiB/s): min= 2048, max= 4096, per=53.87%, avg=2730.67, stdev=1182.41, samples=3 iops : min= 2, max= 4, avg= 2.67, stdev= 1.15, samples=3 write: IOPS=1, BW=1776KiB/s (1818kB/s)(7168KiB/4037msec) clat (msec): min=18, max=1266, avg=301.12, stdev=447.22 lat (msec): min=18, max=1267, avg=301.17, stdev=447.23 clat percentiles (msec): | 1.00th=[ 19], 5.00th=[ 19], 10.00th=[ 19], 20.00th=[ 22], | 30.00th=[ 83], 40.00th=[ 83], 50.00th=[ 94], 60.00th=[ 220], | 70.00th=[ 220], 80.00th=[ 409], 90.00th=[ 1267], 95.00th=[ 1267], | 99.00th=[ 1267], 99.50th=[ 1267], 99.90th=[ 1267], 99.95th=[ 1267], | 99.99th=[ 1267] bw ( KiB/s): min= 2048, max= 4096, per=40.40%, avg=2867.20, stdev=1121.74, samples=5 iops : min= 2, max= 4, avg= 2.80, stdev= 1.10, samples=5 lat (usec) : 500=8.33% lat (msec) : 10=8.33%, 20=8.33%, 50=8.33%, 100=16.67%, 250=8.33% lat (msec) : 500=8.33%, 750=25.00% cpu : usr=0.00%, sys=25.12%, ctx=587, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=5069KiB/s (5191kB/s), 1016KiB/s-4241KiB/s (1040kB/s-4343kB/s), io=20.0MiB (20.0MB), run=1690-4040msec WRITE: bw=7097KiB/s (7267kB/s), 1521KiB/s-5079KiB/s (1557kB/s-5201kB/s), io=28.0MiB (29.4MB), run=1690-4040msec rand-rw: (groupid=0, jobs=1): err= 0: pid=14656: Sun Nov 24 03:54:58 2024 read: IOPS=3, BW=3440KiB/s (3522kB/s)(7168KiB/2084msec) clat (msec): min=12, max=1253, avg=249.86, stdev=447.46 lat (msec): min=12, max=1253, avg=249.86, stdev=447.46 clat percentiles (msec): | 1.00th=[ 13], 5.00th=[ 13], 10.00th=[ 13], 20.00th=[ 46], | 30.00th=[ 59], 40.00th=[ 59], 50.00th=[ 64], 60.00th=[ 101], | 70.00th=[ 101], 80.00th=[ 215], 90.00th=[ 1250], 95.00th=[ 1250], | 99.00th=[ 1250], 99.50th=[ 1250], 99.90th=[ 1250], 99.95th=[ 1250], | 99.99th=[ 1250] bw ( KiB/s): min= 2048, max= 4096, per=68.64%, avg=3413.33, stdev=1182.41, samples=3 iops : min= 2, max= 4, avg= 3.33, stdev= 1.15, samples=3 write: IOPS=2, BW=2457KiB/s (2516kB/s)(5120KiB/2084msec) clat (msec): min=16, max=100, avg=51.60, stdev=37.48 lat (msec): min=16, max=100, avg=51.65, stdev=37.49 clat percentiles (msec): | 1.00th=[ 17], 5.00th=[ 17], 10.00th=[ 17], 20.00th=[ 17], | 30.00th=[ 27], 40.00th=[ 27], 50.00th=[ 32], 60.00th=[ 32], | 70.00th=[ 83], 80.00th=[ 83], 90.00th=[ 102], 95.00th=[ 102], | 99.00th=[ 102], 99.50th=[ 102], 99.90th=[ 102], 99.95th=[ 102], | 99.99th=[ 102] bw ( KiB/s): min= 2048, max= 6144, per=58.83%, avg=4096.00, stdev=2896.31, samples=2 iops : min= 2, max= 6, avg= 4.00, stdev= 2.83, samples=2 lat (msec) : 20=16.67%, 50=25.00%, 100=25.00%, 250=25.00% cpu : usr=0.05%, sys=3.94%, ctx=40, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=14657: Sun Nov 24 03:54:58 2024 read: IOPS=0, BW=1021KiB/s (1045kB/s)(2048KiB/2006msec) clat (usec): min=18550, max=69852, avg=44201.38, stdev=36276.03 lat (usec): min=18552, max=69853, avg=44203.20, stdev=36275.37 clat percentiles (usec): | 1.00th=[18482], 5.00th=[18482], 10.00th=[18482], 20.00th=[18482], | 30.00th=[18482], 40.00th=[18482], 50.00th=[18482], 60.00th=[69731], | 70.00th=[69731], 80.00th=[69731], 90.00th=[69731], 95.00th=[69731], | 99.00th=[69731], 99.50th=[69731], 99.90th=[69731], 99.95th=[69731], | 99.99th=[69731] bw ( KiB/s): min= 2048, max= 2048, per=41.18%, avg=2048.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 write: IOPS=4, BW=5105KiB/s (5227kB/s)(10.0MiB/2006msec) clat (msec): min=20, max=1166, avg=189.84, stdev=346.43 lat (msec): min=20, max=1166, avg=189.90, stdev=346.43 clat percentiles (msec): | 1.00th=[ 21], 5.00th=[ 21], 10.00th=[ 21], 20.00th=[ 22], | 30.00th=[ 59], 40.00th=[ 62], 50.00th=[ 84], 60.00th=[ 86], | 70.00th=[ 114], 80.00th=[ 114], 90.00th=[ 188], 95.00th=[ 1167], | 99.00th=[ 1167], 99.50th=[ 1167], 99.90th=[ 1167], 99.95th=[ 1167], | 99.99th=[ 1167] bw ( KiB/s): min= 4096, max= 8192, per=88.25%, avg=6144.00, stdev=2048.00, samples=3 iops : min= 4, max= 8, avg= 6.00, stdev= 2.00, samples=3 lat (msec) : 20=8.33%, 50=16.67%, 100=50.00%, 250=16.67% cpu : usr=0.00%, sys=7.33%, ctx=92, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=14659: Sun Nov 24 03:54:58 2024 read: IOPS=2, BW=2975KiB/s (3047kB/s)(6144KiB/2065msec) clat (msec): min=15, max=100, avg=65.73, stdev=34.00 lat (msec): min=15, max=100, avg=65.73, stdev=34.00 clat percentiles (msec): | 1.00th=[ 16], 5.00th=[ 16], 10.00th=[ 16], 20.00th=[ 42], | 30.00th=[ 42], 40.00th=[ 56], 50.00th=[ 56], 60.00th=[ 84], | 70.00th=[ 99], 80.00th=[ 99], 90.00th=[ 101], 95.00th=[ 101], | 99.00th=[ 101], 99.50th=[ 101], 99.90th=[ 101], 99.95th=[ 101], | 99.99th=[ 101] bw ( KiB/s): min= 4096, max= 6144, per=100.00%, avg=5120.00, stdev=1448.15, samples=2 iops : min= 4, max= 6, avg= 5.00, stdev= 1.41, samples=2 write: IOPS=2, BW=2975KiB/s (3047kB/s)(6144KiB/2065msec) clat (msec): min=18, max=1241, avg=273.98, stdev=480.13 lat (msec): min=19, max=1241, avg=274.05, stdev=480.13 clat percentiles (msec): | 1.00th=[ 20], 5.00th=[ 20], 10.00th=[ 20], 20.00th=[ 23], | 30.00th=[ 23], 40.00th=[ 38], 50.00th=[ 38], 60.00th=[ 106], | 70.00th=[ 218], 80.00th=[ 218], 90.00th=[ 1250], 95.00th=[ 1250], | 99.00th=[ 1250], 99.50th=[ 1250], 99.90th=[ 1250], 99.95th=[ 1250], | 99.99th=[ 1250] bw ( KiB/s): min= 2048, max= 4096, per=49.03%, avg=3413.33, stdev=1182.41, samples=3 iops : min= 2, max= 4, avg= 3.33, stdev= 1.15, samples=3 lat (msec) : 20=16.67%, 50=25.00%, 100=25.00%, 250=25.00% cpu : usr=0.00%, sys=5.62%, ctx=81, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=14661: Sun Nov 24 03:54:58 2024 read: IOPS=1, BW=1243KiB/s (1273kB/s)(5120KiB/4118msec) clat (msec): min=11, max=703, avg=282.02, stdev=334.20 lat (msec): min=11, max=703, avg=282.02, stdev=334.20 clat percentiles (msec): | 1.00th=[ 12], 5.00th=[ 12], 10.00th=[ 12], 20.00th=[ 12], | 30.00th=[ 16], 40.00th=[ 16], 50.00th=[ 97], 60.00th=[ 97], | 70.00th=[ 584], 80.00th=[ 584], 90.00th=[ 701], 95.00th=[ 701], | 99.00th=[ 701], 99.50th=[ 701], 99.90th=[ 701], 99.95th=[ 701], | 99.99th=[ 701] bw ( KiB/s): min= 2048, max= 4096, per=54.91%, avg=2730.67, stdev=1182.41, samples=3 iops : min= 2, max= 4, avg= 2.67, stdev= 1.15, samples=3 write: IOPS=1, BW=1741KiB/s (1782kB/s)(7168KiB/4118msec) clat (msec): min=31, max=1237, avg=380.32, stdev=430.38 lat (msec): min=31, max=1237, avg=380.39, stdev=430.37 clat percentiles (msec): | 1.00th=[ 32], 5.00th=[ 32], 10.00th=[ 32], 20.00th=[ 39], | 30.00th=[ 105], 40.00th=[ 105], 50.00th=[ 224], 60.00th=[ 489], | 70.00th=[ 489], 80.00th=[ 542], 90.00th=[ 1234], 95.00th=[ 1234], | 99.00th=[ 1234], 99.50th=[ 1234], 99.90th=[ 1234], 99.95th=[ 1234], | 99.99th=[ 1234] bw ( KiB/s): min= 2048, max= 4096, per=44.13%, avg=3072.00, stdev=1182.41, samples=4 iops : min= 2, max= 4, avg= 3.00, stdev= 1.15, samples=4 lat (msec) : 20=16.67%, 50=16.67%, 100=8.33%, 250=16.67%, 500=8.33% lat (msec) : 750=25.00% cpu : usr=0.00%, sys=2.96%, ctx=80, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=4973KiB/s (5093kB/s), 1021KiB/s-3440KiB/s (1045kB/s-3522kB/s), io=20.0MiB (20.0MB), run=2006-4118msec WRITE: bw=6963KiB/s (7130kB/s), 1741KiB/s-5105KiB/s (1782kB/s-5227kB/s), io=28.0MiB (29.4MB), run=2006-4118msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=14672: Sun Nov 24 03:55:00 2024 read: IOPS=3, BW=13.0MiB/s (13.6MB/s)(12.0MiB/922msec) clat (msec): min=124, max=467, avg=301.90, stdev=171.64 lat (msec): min=124, max=467, avg=301.90, stdev=171.64 clat percentiles (msec): | 1.00th=[ 126], 5.00th=[ 126], 10.00th=[ 126], 20.00th=[ 126], | 30.00th=[ 126], 40.00th=[ 313], 50.00th=[ 313], 60.00th=[ 313], | 70.00th=[ 468], 80.00th=[ 468], 90.00th=[ 468], 95.00th=[ 468], | 99.00th=[ 468], 99.50th=[ 468], 99.90th=[ 468], 99.95th=[ 468], | 99.99th=[ 468] bw ( KiB/s): min=16384, max=16384, per=92.20%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=66.67% cpu : usr=0.11%, sys=55.16%, ctx=508, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=14674: Sun Nov 24 03:55:00 2024 read: IOPS=1, BW=4882KiB/s (4999kB/s)(4096KiB/839msec) clat (nsec): min=131211k, max=131211k, avg=131211468.00, stdev= 0.00 lat (nsec): min=131213k, max=131213k, avg=131212598.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 132], 5.00th=[ 132], 10.00th=[ 132], 20.00th=[ 132], | 30.00th=[ 132], 40.00th=[ 132], 50.00th=[ 132], 60.00th=[ 132], | 70.00th=[ 132], 80.00th=[ 132], 90.00th=[ 132], 95.00th=[ 132], | 99.00th=[ 132], 99.50th=[ 132], 99.90th=[ 132], 99.95th=[ 132], | 99.99th=[ 132] bw ( KiB/s): min= 8192, max= 8192, per=46.10%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=2, BW=9764KiB/s (9998kB/s)(8192KiB/839msec) clat (msec): min=208, max=480, avg=344.35, stdev=192.00 lat (msec): min=208, max=480, avg=344.62, stdev=192.03 clat percentiles (msec): | 1.00th=[ 209], 5.00th=[ 209], 10.00th=[ 209], 20.00th=[ 209], | 30.00th=[ 209], 40.00th=[ 209], 50.00th=[ 209], 60.00th=[ 481], | 70.00th=[ 481], 80.00th=[ 481], 90.00th=[ 481], 95.00th=[ 481], | 99.00th=[ 481], 99.50th=[ 481], 99.90th=[ 481], 99.95th=[ 481], | 99.99th=[ 481] bw ( KiB/s): min= 8192, max= 8192, per=24.48%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.12%, sys=25.42%, ctx=127, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=14676: Sun Nov 24 03:55:00 2024 write: IOPS=3, BW=12.3MiB/s (12.9MB/s)(12.0MiB/979msec) clat (msec): min=196, max=477, avg=308.97, stdev=148.93 lat (msec): min=196, max=478, avg=309.31, stdev=148.86 clat percentiles (msec): | 1.00th=[ 197], 5.00th=[ 197], 10.00th=[ 197], 20.00th=[ 197], | 30.00th=[ 197], 40.00th=[ 253], 50.00th=[ 253], 60.00th=[ 253], | 70.00th=[ 477], 80.00th=[ 477], 90.00th=[ 477], 95.00th=[ 477], | 99.00th=[ 477], 99.50th=[ 477], 99.90th=[ 477], 99.95th=[ 477], | 99.99th=[ 477] bw ( KiB/s): min=16384, max=16384, per=48.95%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=66.67% cpu : usr=0.00%, sys=29.96%, ctx=59, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=14678: Sun Nov 24 03:55:00 2024 write: IOPS=3, BW=15.9MiB/s (16.6MB/s)(12.0MiB/756msec) clat (msec): min=191, max=301, avg=245.82, stdev=55.03 lat (msec): min=192, max=302, avg=246.08, stdev=55.05 clat percentiles (msec): | 1.00th=[ 192], 5.00th=[ 192], 10.00th=[ 192], 20.00th=[ 192], | 30.00th=[ 192], 40.00th=[ 245], 50.00th=[ 245], 60.00th=[ 245], | 70.00th=[ 300], 80.00th=[ 300], 90.00th=[ 300], 95.00th=[ 300], | 99.00th=[ 300], 99.50th=[ 300], 99.90th=[ 300], 99.95th=[ 300], | 99.99th=[ 300] bw ( KiB/s): min= 8175, max= 8175, per=24.42%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.13%, sys=35.23%, ctx=26, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=17.4MiB/s (18.2MB/s), 4882KiB/s-13.0MiB/s (4999kB/s-13.6MB/s), io=16.0MiB (16.8MB), run=839-922msec WRITE: bw=32.7MiB/s (34.3MB/s), 9764KiB/s-15.9MiB/s (9998kB/s-16.6MB/s), io=32.0MiB (33.6MB), run=756-979msec rand-rw: (groupid=0, jobs=1): err= 0: pid=14673: Sun Nov 24 03:55:00 2024 read: IOPS=2, BW=11.6MiB/s (12.1MB/s)(12.0MiB/1037msec) clat (msec): min=300, max=375, avg=336.82, stdev=37.86 lat (msec): min=300, max=375, avg=336.83, stdev=37.86 clat percentiles (msec): | 1.00th=[ 300], 5.00th=[ 300], 10.00th=[ 300], 20.00th=[ 300], | 30.00th=[ 300], 40.00th=[ 334], 50.00th=[ 334], 60.00th=[ 334], | 70.00th=[ 376], 80.00th=[ 376], 90.00th=[ 376], 95.00th=[ 376], | 99.00th=[ 376], 99.50th=[ 376], 99.90th=[ 376], 99.95th=[ 376], | 99.99th=[ 376] bw ( KiB/s): min=11055, max=11055, per=69.97%, avg=11055.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 500=100.00% cpu : usr=0.00%, sys=5.69%, ctx=26, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=14675: Sun Nov 24 03:55:00 2024 read: IOPS=1, BW=4214KiB/s (4315kB/s)(4096KiB/972msec) clat (nsec): min=333536k, max=333536k, avg=333536488.00, stdev= 0.00 lat (nsec): min=333538k, max=333538k, avg=333537909.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 334], 5.00th=[ 334], 10.00th=[ 334], 20.00th=[ 334], | 30.00th=[ 334], 40.00th=[ 334], 50.00th=[ 334], 60.00th=[ 334], | 70.00th=[ 334], 80.00th=[ 334], 90.00th=[ 334], 95.00th=[ 334], | 99.00th=[ 334], 99.50th=[ 334], 99.90th=[ 334], 99.95th=[ 334], | 99.99th=[ 334] bw ( KiB/s): min= 8078, max= 8078, per=51.13%, avg=8078.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=2, BW=8428KiB/s (8630kB/s)(8192KiB/972msec) clat (msec): min=143, max=452, avg=297.78, stdev=218.31 lat (msec): min=143, max=452, avg=297.99, stdev=218.42 clat percentiles (msec): | 1.00th=[ 144], 5.00th=[ 144], 10.00th=[ 144], 20.00th=[ 144], | 30.00th=[ 144], 40.00th=[ 144], 50.00th=[ 144], 60.00th=[ 451], | 70.00th=[ 451], 80.00th=[ 451], 90.00th=[ 451], 95.00th=[ 451], | 99.00th=[ 451], 99.50th=[ 451], 99.90th=[ 451], 99.95th=[ 451], | 99.99th=[ 451] lat (msec) : 250=33.33%, 500=66.67% cpu : usr=0.21%, sys=13.08%, ctx=222, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=14677: Sun Nov 24 03:55:00 2024 write: IOPS=2, BW=11.7MiB/s (12.3MB/s)(12.0MiB/1022msec) clat (msec): min=283, max=361, avg=332.88, stdev=43.06 lat (msec): min=283, max=363, avg=333.61, stdev=43.52 clat percentiles (msec): | 1.00th=[ 284], 5.00th=[ 284], 10.00th=[ 284], 20.00th=[ 284], | 30.00th=[ 284], 40.00th=[ 355], 50.00th=[ 355], 60.00th=[ 355], | 70.00th=[ 363], 80.00th=[ 363], 90.00th=[ 363], 95.00th=[ 363], | 99.00th=[ 363], 99.50th=[ 363], 99.90th=[ 363], 99.95th=[ 363], | 99.99th=[ 363] bw ( KiB/s): min= 8126, max= 8126, per=25.34%, avg=8126.00, stdev= 0.00, samples=2 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=2 lat (msec) : 500=100.00% cpu : usr=0.00%, sys=13.91%, ctx=320, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=14679: Sun Nov 24 03:55:00 2024 write: IOPS=6, BW=27.6MiB/s (28.9MB/s)(12.0MiB/435msec) clat (msec): min=84, max=239, avg=139.24, stdev=87.16 lat (msec): min=84, max=239, avg=139.57, stdev=87.11 clat percentiles (msec): | 1.00th=[ 85], 5.00th=[ 85], 10.00th=[ 85], 20.00th=[ 85], | 30.00th=[ 85], 40.00th=[ 94], 50.00th=[ 94], 60.00th=[ 94], | 70.00th=[ 241], 80.00th=[ 241], 90.00th=[ 241], 95.00th=[ 241], | 99.00th=[ 241], 99.50th=[ 241], 99.90th=[ 241], 99.95th=[ 241], | 99.99th=[ 241] lat (msec) : 100=66.67%, 250=33.33% cpu : usr=0.00%, sys=17.97%, ctx=39, majf=0, minf=28 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=15.4MiB/s (16.2MB/s), 4214KiB/s-11.6MiB/s (4315kB/s-12.1MB/s), io=16.0MiB (16.8MB), run=972-1037msec WRITE: bw=31.3MiB/s (32.8MB/s), 8428KiB/s-27.6MiB/s (8630kB/s-28.9MB/s), io=32.0MiB (33.6MB), run=435-1022msec