== sanity test 398b: DIO and buffer IO race ============== 23:05:56 (1712027156) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 2.17755 s, 23.1 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=9056: Mon Apr 1 23:06:32 2024 read: IOPS=48, BW=193KiB/s (198kB/s)(5992KiB/31059msec) clat (usec): min=39, max=379743, avg=11831.95, stdev=12346.43 lat (usec): min=39, max=379745, avg=11832.99, stdev=12346.49 clat percentiles (usec): | 1.00th=[ 76], 5.00th=[ 112], 10.00th=[ 4555], 20.00th=[ 5932], | 30.00th=[ 7242], 40.00th=[ 8586], 50.00th=[ 10028], 60.00th=[ 11731], | 70.00th=[ 13698], 80.00th=[ 16581], 90.00th=[ 20579], 95.00th=[ 24511], | 99.00th=[ 38536], 99.50th=[ 53740], 99.90th=[ 68682], 99.95th=[379585], | 99.99th=[379585] bw ( KiB/s): min= 39, max= 280, per=24.63%, avg=190.13, stdev=44.81, samples=61 iops : min= 9, max= 70, avg=47.26, stdev=11.21, samples=61 write: IOPS=50, BW=203KiB/s (208kB/s)(6296KiB/31059msec) clat (usec): min=405, max=66273, avg=8413.99, stdev=9236.00 lat (usec): min=406, max=66273, avg=8415.94, stdev=9235.74 clat percentiles (usec): | 1.00th=[ 586], 5.00th=[ 660], 10.00th=[ 701], 20.00th=[ 807], | 30.00th=[ 1139], 40.00th=[ 2343], 50.00th=[ 5473], 60.00th=[ 8094], | 70.00th=[11207], 80.00th=[15270], 90.00th=[20841], 95.00th=[26870], | 99.00th=[39584], 99.50th=[46400], 99.90th=[55313], 99.95th=[66323], | 99.99th=[66323] bw ( KiB/s): min= 15, max= 344, per=24.71%, avg=199.69, stdev=52.53, samples=61 iops : min= 3, max= 86, avg=49.66, stdev=13.15, samples=61 lat (usec) : 50=0.03%, 100=2.02%, 250=0.65%, 500=0.13%, 750=8.11% lat (usec) : 1000=5.66% lat (msec) : 2=5.79%, 4=4.85%, 10=31.25%, 20=29.85%, 50=11.17% lat (msec) : 100=0.46%, 500=0.03% cpu : usr=0.18%, sys=13.10%, ctx=5150, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9057: Mon Apr 1 23:06:32 2024 read: IOPS=47, BW=191KiB/s (196kB/s)(5940KiB/31097msec) clat (usec): min=50, max=68688, avg=11402.23, stdev=7904.40 lat (usec): min=50, max=68689, avg=11405.53, stdev=7903.74 clat percentiles (usec): | 1.00th=[ 72], 5.00th=[ 100], 10.00th=[ 4293], 20.00th=[ 5932], | 30.00th=[ 7177], 40.00th=[ 8291], 50.00th=[ 9765], 60.00th=[11469], | 70.00th=[13698], 80.00th=[16450], 90.00th=[20317], 95.00th=[23987], | 99.00th=[42206], 99.50th=[50594], 99.90th=[64750], 99.95th=[68682], | 99.99th=[68682] bw ( KiB/s): min= 16, max= 368, per=24.59%, avg=189.82, stdev=52.80, samples=61 iops : min= 4, max= 92, avg=47.11, stdev=13.22, samples=61 write: IOPS=51, BW=204KiB/s (209kB/s)(6348KiB/31097msec) clat (usec): min=422, max=375989, avg=8865.86, stdev=13307.73 lat (usec): min=422, max=375991, avg=8867.04, stdev=13307.79 clat percentiles (usec): | 1.00th=[ 578], 5.00th=[ 668], 10.00th=[ 701], 20.00th=[ 816], | 30.00th=[ 1336], 40.00th=[ 2737], 50.00th=[ 5932], 60.00th=[ 8291], | 70.00th=[ 10945], 80.00th=[ 15270], 90.00th=[ 21627], 95.00th=[ 26870], | 99.00th=[ 40633], 99.50th=[ 52167], 99.90th=[ 99091], 99.95th=[375391], | 99.99th=[375391] bw ( KiB/s): min= 56, max= 312, per=24.84%, avg=200.67, stdev=46.31, samples=61 iops : min= 14, max= 78, avg=49.84, stdev=11.68, samples=61 lat (usec) : 100=2.47%, 250=0.78%, 500=0.23%, 750=7.55%, 1000=5.70% lat (msec) : 2=5.27%, 4=4.85%, 10=32.81%, 20=28.97%, 50=10.84% lat (msec) : 100=0.49%, 500=0.03% cpu : usr=0.18%, sys=13.28%, ctx=4929, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9058: Mon Apr 1 23:06:32 2024 read: IOPS=49, BW=196KiB/s (201kB/s)(6028KiB/30705msec) clat (usec): min=52, max=412297, avg=11897.48, stdev=12973.04 lat (usec): min=52, max=412299, avg=11898.94, stdev=12973.02 clat percentiles (usec): | 1.00th=[ 76], 5.00th=[ 269], 10.00th=[ 4555], 20.00th=[ 6194], | 30.00th=[ 7373], 40.00th=[ 8455], 50.00th=[ 10028], 60.00th=[ 11600], | 70.00th=[ 13566], 80.00th=[ 16319], 90.00th=[ 21103], 95.00th=[ 25822], | 99.00th=[ 41157], 99.50th=[ 49546], 99.90th=[ 68682], 99.95th=[413139], | 99.99th=[413139] bw ( KiB/s): min= 23, max= 336, per=25.38%, avg=195.92, stdev=51.22, samples=61 iops : min= 5, max= 84, avg=48.62, stdev=12.86, samples=61 write: IOPS=50, BW=204KiB/s (209kB/s)(6260KiB/30705msec) clat (usec): min=443, max=60028, avg=8115.59, stdev=8766.64 lat (usec): min=444, max=60030, avg=8117.10, stdev=8766.64 clat percentiles (usec): | 1.00th=[ 586], 5.00th=[ 660], 10.00th=[ 701], 20.00th=[ 807], | 30.00th=[ 1287], 40.00th=[ 2671], 50.00th=[ 5800], 60.00th=[ 8225], | 70.00th=[10552], 80.00th=[13829], 90.00th=[19530], 95.00th=[25035], | 99.00th=[39584], 99.50th=[45351], 99.90th=[58983], 99.95th=[60031], | 99.99th=[60031] bw ( KiB/s): min= 31, max= 440, per=25.13%, avg=203.02, stdev=60.57, samples=61 iops : min= 7, max= 110, avg=50.39, stdev=15.26, samples=61 lat (usec) : 100=1.95%, 250=0.46%, 500=0.29%, 750=7.36%, 1000=5.86% lat (msec) : 2=5.31%, 4=5.01%, 10=33.07%, 20=30.47%, 50=9.86% lat (msec) : 100=0.33%, 500=0.03% cpu : usr=0.18%, sys=13.37%, ctx=5111, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9059: Mon Apr 1 23:06:32 2024 read: IOPS=49, BW=198KiB/s (202kB/s)(6052KiB/30643msec) clat (usec): min=49, max=58003, avg=11409.84, stdev=7473.46 lat (usec): min=50, max=58003, avg=11410.85, stdev=7473.50 clat percentiles (usec): | 1.00th=[ 76], 5.00th=[ 103], 10.00th=[ 4293], 20.00th=[ 6063], | 30.00th=[ 7046], 40.00th=[ 8586], 50.00th=[ 9896], 60.00th=[11600], | 70.00th=[13829], 80.00th=[16581], 90.00th=[20055], 95.00th=[24249], | 99.00th=[39060], 99.50th=[43779], 99.90th=[56886], 99.95th=[57934], | 99.99th=[57934] bw ( KiB/s): min= 112, max= 279, per=25.97%, avg=200.50, stdev=41.15, samples=60 iops : min= 28, max= 69, avg=49.75, stdev=10.34, samples=60 write: IOPS=50, BW=204KiB/s (208kB/s)(6236KiB/30643msec) clat (usec): min=452, max=421801, avg=8536.06, stdev=13838.56 lat (usec): min=452, max=421802, avg=8537.79, stdev=13839.37 clat percentiles (usec): | 1.00th=[ 611], 5.00th=[ 668], 10.00th=[ 709], 20.00th=[ 807], | 30.00th=[ 1221], 40.00th=[ 2343], 50.00th=[ 5735], 60.00th=[ 8291], | 70.00th=[ 10814], 80.00th=[ 14877], 90.00th=[ 20579], 95.00th=[ 25822], | 99.00th=[ 39584], 99.50th=[ 46400], 99.90th=[ 78119], 99.95th=[421528], | 99.99th=[421528] bw ( KiB/s): min= 15, max= 352, per=25.13%, avg=203.08, stdev=57.61, samples=61 iops : min= 3, max= 88, avg=50.39, stdev=14.43, samples=61 lat (usec) : 50=0.03%, 100=2.34%, 250=0.72%, 500=0.16%, 750=7.13% lat (usec) : 1000=5.96% lat (msec) : 2=6.18%, 4=4.98%, 10=31.35%, 20=30.53%, 50=10.35% lat (msec) : 100=0.23%, 500=0.03% cpu : usr=0.19%, sys=13.34%, ctx=5206, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=772KiB/s (791kB/s), 191KiB/s-198KiB/s (196kB/s-202kB/s), io=23.4MiB (24.6MB), run=30643-31097msec WRITE: bw=808KiB/s (828kB/s), 203KiB/s-204KiB/s (208kB/s-209kB/s), io=24.6MiB (25.7MB), run=30643-31097msec rand-rw: (groupid=0, jobs=1): err= 0: pid=9052: Mon Apr 1 23:06:47 2024 read: IOPS=32, BW=132KiB/s (135kB/s)(5992KiB/45408msec) clat (msec): min=2, max=387, avg=11.20, stdev=13.30 lat (msec): min=2, max=387, avg=11.20, stdev=13.30 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 4], 10.00th=[ 5], 20.00th=[ 6], | 30.00th=[ 6], 40.00th=[ 7], 50.00th=[ 8], 60.00th=[ 9], | 70.00th=[ 11], 80.00th=[ 15], 90.00th=[ 24], 95.00th=[ 31], | 99.00th=[ 47], 99.50th=[ 52], 99.90th=[ 100], 99.95th=[ 388], | 99.99th=[ 388] bw ( KiB/s): min= 23, max= 288, per=25.04%, avg=130.72, stdev=61.61, samples=90 iops : min= 5, max= 72, avg=32.46, stdev=15.44, samples=90 write: IOPS=34, BW=139KiB/s (142kB/s)(6296KiB/45408msec) clat (msec): min=4, max=100, avg=18.11, stdev=11.35 lat (msec): min=4, max=100, avg=18.11, stdev=11.35 clat percentiles (msec): | 1.00th=[ 6], 5.00th=[ 8], 10.00th=[ 9], 20.00th=[ 11], | 30.00th=[ 12], 40.00th=[ 13], 50.00th=[ 15], 60.00th=[ 17], | 70.00th=[ 20], 80.00th=[ 25], 90.00th=[ 33], 95.00th=[ 41], | 99.00th=[ 59], 99.50th=[ 77], 99.90th=[ 90], 99.95th=[ 102], | 99.99th=[ 102] bw ( KiB/s): min= 56, max= 288, per=25.28%, avg=138.30, stdev=61.13, samples=89 iops : min= 14, max= 72, avg=34.36, stdev=15.32, samples=89 lat (msec) : 4=3.22%, 10=39.00%, 20=36.65%, 50=19.63%, 100=1.43% lat (msec) : 250=0.03%, 500=0.03% cpu : usr=0.09%, sys=6.55%, ctx=3378, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9053: Mon Apr 1 23:06:47 2024 read: IOPS=32, BW=129KiB/s (132kB/s)(5940KiB/45943msec) clat (msec): min=2, max=410, avg=11.49, stdev=14.50 lat (msec): min=2, max=410, avg=11.49, stdev=14.50 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 4], 10.00th=[ 5], 20.00th=[ 6], | 30.00th=[ 6], 40.00th=[ 7], 50.00th=[ 8], 60.00th=[ 9], | 70.00th=[ 11], 80.00th=[ 15], 90.00th=[ 25], 95.00th=[ 34], | 99.00th=[ 52], 99.50th=[ 63], 99.90th=[ 100], 99.95th=[ 409], | 99.99th=[ 409] bw ( KiB/s): min= 7, max= 328, per=24.59%, avg=128.35, stdev=68.27, samples=91 iops : min= 1, max= 82, avg=31.82, stdev=17.03, samples=91 write: IOPS=34, BW=138KiB/s (141kB/s)(6348KiB/45943msec) clat (usec): min=4947, max=89973, avg=18113.94, stdev=11179.15 lat (usec): min=4948, max=89976, avg=18115.91, stdev=11180.40 clat percentiles (usec): | 1.00th=[ 5997], 5.00th=[ 7635], 10.00th=[ 8586], 20.00th=[10159], | 30.00th=[11207], 40.00th=[12649], 50.00th=[14222], 60.00th=[16581], | 70.00th=[19530], 80.00th=[24773], 90.00th=[33162], 95.00th=[40633], | 99.00th=[57934], 99.50th=[67634], 99.90th=[78119], 99.95th=[89654], | 99.99th=[89654] bw ( KiB/s): min= 56, max= 296, per=25.19%, avg=137.79, stdev=64.02, samples=90 iops : min= 14, max= 74, avg=34.19, stdev=15.97, samples=90 lat (msec) : 4=3.22%, 10=39.71%, 20=35.48%, 50=19.73%, 100=1.82% lat (msec) : 500=0.03% cpu : usr=0.10%, sys=6.33%, ctx=3393, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9054: Mon Apr 1 23:06:47 2024 read: IOPS=32, BW=132KiB/s (135kB/s)(6028KiB/45773msec) clat (usec): min=2575, max=78242, avg=10976.64, stdev=9673.73 lat (usec): min=2575, max=78243, avg=10977.99, stdev=9673.75 clat percentiles (usec): | 1.00th=[ 3163], 5.00th=[ 3851], 10.00th=[ 4359], 20.00th=[ 5080], | 30.00th=[ 5735], 40.00th=[ 6521], 50.00th=[ 7504], 60.00th=[ 8717], | 70.00th=[10421], 80.00th=[14091], 90.00th=[23462], 95.00th=[31589], | 99.00th=[50070], 99.50th=[61080], 99.90th=[76022], 99.95th=[78119], | 99.99th=[78119] bw ( KiB/s): min= 55, max= 280, per=25.37%, avg=132.43, stdev=62.45, samples=90 iops : min= 13, max= 70, avg=32.83, stdev=15.65, samples=90 write: IOPS=34, BW=137KiB/s (140kB/s)(6260KiB/45773msec) clat (msec): min=4, max=421, avg=18.59, stdev=15.07 lat (msec): min=4, max=421, avg=18.60, stdev=15.07 clat percentiles (msec): | 1.00th=[ 6], 5.00th=[ 8], 10.00th=[ 9], 20.00th=[ 11], | 30.00th=[ 12], 40.00th=[ 14], 50.00th=[ 15], 60.00th=[ 17], | 70.00th=[ 21], 80.00th=[ 26], 90.00th=[ 33], 95.00th=[ 41], | 99.00th=[ 62], 99.50th=[ 73], 99.90th=[ 88], 99.95th=[ 422], | 99.99th=[ 422] bw ( KiB/s): min= 47, max= 288, per=25.10%, avg=137.31, stdev=63.00, samples=90 iops : min= 11, max= 72, avg=34.06, stdev=15.81, samples=90 lat (msec) : 4=2.96%, 10=39.55%, 20=35.71%, 50=20.15%, 100=1.60% lat (msec) : 500=0.03% cpu : usr=0.16%, sys=6.37%, ctx=3407, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9055: Mon Apr 1 23:06:47 2024 read: IOPS=33, BW=133KiB/s (136kB/s)(6052KiB/45670msec) clat (usec): min=2354, max=99701, avg=11217.62, stdev=9746.09 lat (usec): min=2355, max=99703, avg=11220.02, stdev=9745.88 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 4], 10.00th=[ 5], 20.00th=[ 6], | 30.00th=[ 6], 40.00th=[ 7], 50.00th=[ 8], 60.00th=[ 9], | 70.00th=[ 11], 80.00th=[ 16], 90.00th=[ 25], 95.00th=[ 33], | 99.00th=[ 51], 99.50th=[ 54], 99.90th=[ 63], 99.95th=[ 101], | 99.99th=[ 101] bw ( KiB/s): min= 39, max= 352, per=25.57%, avg=133.46, stdev=65.44, samples=90 iops : min= 9, max= 88, avg=33.12, stdev=16.45, samples=90 write: IOPS=34, BW=137KiB/s (140kB/s)(6236KiB/45670msec) clat (msec): min=4, max=415, avg=18.33, stdev=14.89 lat (msec): min=4, max=415, avg=18.33, stdev=14.89 clat percentiles (msec): | 1.00th=[ 7], 5.00th=[ 8], 10.00th=[ 9], 20.00th=[ 11], | 30.00th=[ 12], 40.00th=[ 13], 50.00th=[ 15], 60.00th=[ 17], | 70.00th=[ 21], 80.00th=[ 25], 90.00th=[ 32], 95.00th=[ 40], | 99.00th=[ 57], 99.50th=[ 72], 99.90th=[ 101], 99.95th=[ 418], | 99.99th=[ 418] bw ( KiB/s): min= 8, max= 288, per=24.84%, avg=135.86, stdev=62.16, samples=91 iops : min= 2, max= 72, avg=33.73, stdev=15.61, samples=91 lat (msec) : 4=3.39%, 10=38.96%, 20=35.51%, 50=20.54%, 100=1.56% lat (msec) : 500=0.03% cpu : usr=0.13%, sys=6.36%, ctx=3387, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=523KiB/s (535kB/s), 129KiB/s-133KiB/s (132kB/s-136kB/s), io=23.4MiB (24.6MB), run=45408-45943msec WRITE: bw=547KiB/s (560kB/s), 137KiB/s-139KiB/s (140kB/s-142kB/s), io=24.6MiB (25.7MB), run=45408-45943msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=9114: Mon Apr 1 23:07:04 2024 read: IOPS=24, BW=389KiB/s (398kB/s)(5920KiB/15221msec) clat (usec): min=52, max=167659, avg=28034.56, stdev=12345.71 lat (usec): min=53, max=167661, avg=28043.76, stdev=12365.27 clat percentiles (usec): | 1.00th=[ 80], 5.00th=[ 14353], 10.00th=[ 17957], 20.00th=[ 21365], | 30.00th=[ 23200], 40.00th=[ 25297], 50.00th=[ 27132], 60.00th=[ 29230], | 70.00th=[ 31589], 80.00th=[ 33817], 90.00th=[ 38536], 95.00th=[ 45351], | 99.00th=[ 61080], 99.50th=[ 66323], 99.90th=[166724], 99.95th=[166724], | 99.99th=[166724] bw ( KiB/s): min= 256, max= 606, per=25.40%, avg=388.07, stdev=81.67, samples=30 iops : min= 16, max= 37, avg=24.10, stdev= 5.09, samples=30 write: IOPS=26, BW=418KiB/s (428kB/s)(6368KiB/15221msec) clat (usec): min=869, max=137692, avg=12121.33, stdev=15798.16 lat (usec): min=871, max=137694, avg=12128.44, stdev=15797.11 clat percentiles (usec): | 1.00th=[ 947], 5.00th=[ 1090], 10.00th=[ 1139], 20.00th=[ 1827], | 30.00th=[ 3752], 40.00th=[ 5080], 50.00th=[ 6718], 60.00th=[ 8356], | 70.00th=[ 12649], 80.00th=[ 17433], 90.00th=[ 29492], 95.00th=[ 42206], | 99.00th=[ 80217], 99.50th=[ 95945], 99.90th=[137364], 99.95th=[137364], | 99.99th=[137364] bw ( KiB/s): min= 160, max= 638, per=26.06%, avg=419.00, stdev=116.78, samples=30 iops : min= 10, max= 39, avg=26.03, stdev= 7.31, samples=30 lat (usec) : 100=1.04%, 250=0.65%, 1000=0.65% lat (msec) : 2=10.16%, 4=5.47%, 10=17.19%, 20=14.84%, 50=46.87% lat (msec) : 100=2.86%, 250=0.26% cpu : usr=0.12%, sys=11.67%, ctx=2607, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9117: Mon Apr 1 23:07:04 2024 read: IOPS=23, BW=370KiB/s (379kB/s)(5744KiB/15526msec) clat (usec): min=75, max=68900, avg=28552.40, stdev=10093.25 lat (usec): min=76, max=68902, avg=28553.77, stdev=10093.31 clat percentiles (usec): | 1.00th=[ 93], 5.00th=[15795], 10.00th=[18220], 20.00th=[21103], | 30.00th=[23200], 40.00th=[25035], 50.00th=[27657], 60.00th=[29492], | 70.00th=[31851], 80.00th=[35390], 90.00th=[41157], 95.00th=[47973], | 99.00th=[60031], 99.50th=[66847], 99.90th=[68682], 99.95th=[68682], | 99.99th=[68682] bw ( KiB/s): min= 223, max= 512, per=24.23%, avg=370.23, stdev=78.78, samples=31 iops : min= 13, max= 32, avg=22.84, stdev= 5.06, samples=31 write: IOPS=26, BW=421KiB/s (432kB/s)(6544KiB/15526msec) clat (usec): min=919, max=197433, avg=12845.00, stdev=16824.99 lat (usec): min=920, max=197435, avg=12850.37, stdev=16824.25 clat percentiles (usec): | 1.00th=[ 963], 5.00th=[ 1045], 10.00th=[ 1139], 20.00th=[ 1975], | 30.00th=[ 4424], 40.00th=[ 5604], 50.00th=[ 7177], 60.00th=[ 9634], | 70.00th=[ 14091], 80.00th=[ 21103], 90.00th=[ 28967], 95.00th=[ 40633], | 99.00th=[ 73925], 99.50th=[ 84411], 99.90th=[198181], 99.95th=[198181], | 99.99th=[198181] bw ( KiB/s): min= 191, max= 672, per=26.10%, avg=419.68, stdev=138.49, samples=31 iops : min= 11, max= 42, avg=25.94, stdev= 8.65, samples=31 lat (usec) : 100=0.52%, 250=0.13%, 1000=1.17% lat (msec) : 2=9.64%, 4=4.17%, 10=17.71%, 20=16.15%, 50=47.79% lat (msec) : 100=2.47%, 250=0.26% cpu : usr=0.26%, sys=11.71%, ctx=2607, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9118: Mon Apr 1 23:07:04 2024 read: IOPS=24, BW=399KiB/s (409kB/s)(6256KiB/15671msec) clat (usec): min=79, max=63308, avg=28053.83, stdev=9564.63 lat (usec): min=80, max=63310, avg=28055.03, stdev=9564.65 clat percentiles (usec): | 1.00th=[ 98], 5.00th=[15008], 10.00th=[18744], 20.00th=[21890], | 30.00th=[23987], 40.00th=[26084], 50.00th=[27919], 60.00th=[29492], | 70.00th=[31589], 80.00th=[34341], 90.00th=[39060], 95.00th=[43254], | 99.00th=[54789], 99.50th=[61604], 99.90th=[63177], 99.95th=[63177], | 99.99th=[63177] bw ( KiB/s): min= 288, max= 512, per=25.99%, avg=397.13, stdev=65.41, samples=31 iops : min= 18, max= 32, avg=24.55, stdev= 4.06, samples=31 write: IOPS=24, BW=385KiB/s (394kB/s)(6032KiB/15671msec) clat (usec): min=870, max=192926, avg=12419.53, stdev=16685.06 lat (usec): min=871, max=192928, avg=12421.86, stdev=16684.98 clat percentiles (usec): | 1.00th=[ 914], 5.00th=[ 1037], 10.00th=[ 1123], 20.00th=[ 1844], | 30.00th=[ 4178], 40.00th=[ 5932], 50.00th=[ 7635], 60.00th=[ 10290], | 70.00th=[ 12911], 80.00th=[ 16712], 90.00th=[ 29492], 95.00th=[ 40109], | 99.00th=[ 81265], 99.50th=[103285], 99.90th=[191890], 99.95th=[191890], | 99.99th=[191890] bw ( KiB/s): min= 160, max= 702, per=23.54%, avg=378.48, stdev=137.83, samples=31 iops : min= 10, max= 43, avg=23.39, stdev= 8.58, samples=31 lat (usec) : 100=0.52%, 250=0.91%, 500=0.26%, 1000=1.82% lat (msec) : 2=8.46%, 4=4.04%, 10=14.71%, 20=17.45%, 50=49.22% lat (msec) : 100=2.34%, 250=0.26% cpu : usr=0.16%, sys=11.50%, ctx=2660, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9120: Mon Apr 1 23:07:04 2024 read: IOPS=24, BW=387KiB/s (396kB/s)(6032KiB/15589msec) clat (usec): min=83, max=148585, avg=28903.88, stdev=12493.69 lat (usec): min=84, max=148586, avg=28906.11, stdev=12494.00 clat percentiles (usec): | 1.00th=[ 95], 5.00th=[ 16188], 10.00th=[ 19530], 20.00th=[ 21627], | 30.00th=[ 23725], 40.00th=[ 25560], 50.00th=[ 27919], 60.00th=[ 29492], | 70.00th=[ 31589], 80.00th=[ 34341], 90.00th=[ 40109], 95.00th=[ 45876], | 99.00th=[ 70779], 99.50th=[101188], 99.90th=[147850], 99.95th=[147850], | 99.99th=[147850] bw ( KiB/s): min= 223, max= 576, per=25.11%, avg=383.65, stdev=92.01, samples=31 iops : min= 13, max= 36, avg=23.68, stdev= 5.83, samples=31 write: IOPS=25, BW=401KiB/s (411kB/s)(6256KiB/15589msec) clat (usec): min=744, max=91322, avg=11947.69, stdev=12517.35 lat (usec): min=745, max=91325, avg=11950.04, stdev=12517.47 clat percentiles (usec): | 1.00th=[ 873], 5.00th=[ 1045], 10.00th=[ 1123], 20.00th=[ 1827], | 30.00th=[ 4752], 40.00th=[ 6390], 50.00th=[ 7898], 60.00th=[10290], | 70.00th=[12911], 80.00th=[19268], 90.00th=[29230], 95.00th=[37487], | 99.00th=[58459], 99.50th=[64750], 99.90th=[91751], 99.95th=[91751], | 99.99th=[91751] bw ( KiB/s): min= 160, max= 766, per=24.95%, avg=401.19, stdev=126.27, samples=31 iops : min= 10, max= 47, avg=24.77, stdev= 7.93, samples=31 lat (usec) : 100=0.65%, 250=0.52%, 750=0.13%, 1000=1.69% lat (msec) : 2=9.11%, 4=2.47%, 10=16.93%, 20=15.36%, 50=50.78% lat (msec) : 100=2.08%, 250=0.26% cpu : usr=0.06%, sys=12.45%, ctx=2679, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1528KiB/s (1565kB/s), 370KiB/s-399KiB/s (379kB/s-409kB/s), io=23.4MiB (24.5MB), run=15221-15671msec WRITE: bw=1608KiB/s (1647kB/s), 385KiB/s-421KiB/s (394kB/s-432kB/s), io=24.6MiB (25.8MB), run=15221-15671msec rand-rw: (groupid=0, jobs=1): err= 0: pid=9115: Mon Apr 1 23:07:04 2024 read: IOPS=24, BW=387KiB/s (396kB/s)(5920KiB/15295msec) clat (msec): min=3, max=138, avg=15.87, stdev=14.87 lat (msec): min=3, max=138, avg=15.88, stdev=14.87 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 5], 10.00th=[ 5], 20.00th=[ 6], | 30.00th=[ 7], 40.00th=[ 9], 50.00th=[ 11], 60.00th=[ 14], | 70.00th=[ 19], 80.00th=[ 25], 90.00th=[ 33], 95.00th=[ 44], | 99.00th=[ 70], 99.50th=[ 101], 99.90th=[ 140], 99.95th=[ 140], | 99.99th=[ 140] bw ( KiB/s): min= 224, max= 640, per=25.19%, avg=382.63, stdev=104.50, samples=30 iops : min= 14, max= 40, avg=23.70, stdev= 6.59, samples=30 write: IOPS=26, BW=416KiB/s (426kB/s)(6368KiB/15295msec) clat (usec): min=6242, max=99315, avg=23628.54, stdev=13806.31 lat (usec): min=6245, max=99317, avg=23630.67, stdev=13806.25 clat percentiles (usec): | 1.00th=[ 6849], 5.00th=[ 9110], 10.00th=[10683], 20.00th=[12911], | 30.00th=[15270], 40.00th=[17957], 50.00th=[20055], 60.00th=[22938], | 70.00th=[26870], 80.00th=[31851], 90.00th=[40109], 95.00th=[52691], | 99.00th=[77071], 99.50th=[91751], 99.90th=[99091], 99.95th=[99091], | 99.99th=[99091] bw ( KiB/s): min= 160, max= 608, per=26.01%, avg=415.63, stdev=127.84, samples=30 iops : min= 10, max= 38, avg=25.77, stdev= 8.03, samples=30 lat (msec) : 4=1.17%, 10=25.52%, 20=34.64%, 50=34.37%, 100=4.04% lat (msec) : 250=0.26% cpu : usr=0.03%, sys=6.19%, ctx=1078, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9116: Mon Apr 1 23:07:04 2024 read: IOPS=22, BW=367KiB/s (376kB/s)(5744KiB/15664msec) clat (msec): min=3, max=101, avg=14.67, stdev=12.22 lat (msec): min=3, max=101, avg=14.67, stdev=12.22 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 5], 10.00th=[ 5], 20.00th=[ 7], | 30.00th=[ 7], 40.00th=[ 8], 50.00th=[ 10], 60.00th=[ 12], | 70.00th=[ 18], 80.00th=[ 24], 90.00th=[ 31], 95.00th=[ 36], | 99.00th=[ 58], 99.50th=[ 72], 99.90th=[ 102], 99.95th=[ 102], | 99.99th=[ 102] bw ( KiB/s): min= 159, max= 576, per=23.88%, avg=362.74, stdev=102.64, samples=31 iops : min= 9, max= 36, avg=22.29, stdev= 6.48, samples=31 write: IOPS=26, BW=418KiB/s (428kB/s)(6544KiB/15664msec) clat (msec): min=6, max=195, avg=25.37, stdev=16.76 lat (msec): min=6, max=195, avg=25.38, stdev=16.76 clat percentiles (msec): | 1.00th=[ 8], 5.00th=[ 10], 10.00th=[ 12], 20.00th=[ 13], | 30.00th=[ 16], 40.00th=[ 18], 50.00th=[ 22], 60.00th=[ 26], | 70.00th=[ 31], 80.00th=[ 37], 90.00th=[ 44], 95.00th=[ 53], | 99.00th=[ 78], 99.50th=[ 84], 99.90th=[ 197], 99.95th=[ 197], | 99.99th=[ 197] bw ( KiB/s): min= 191, max= 640, per=25.86%, avg=413.23, stdev=114.16, samples=31 iops : min= 11, max= 40, avg=25.45, stdev= 7.13, samples=31 lat (msec) : 4=1.04%, 10=26.95%, 20=31.77%, 50=36.07%, 100=3.78% lat (msec) : 250=0.39% cpu : usr=0.02%, sys=6.03%, ctx=1089, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9119: Mon Apr 1 23:07:04 2024 read: IOPS=24, BW=397KiB/s (406kB/s)(6256KiB/15762msec) clat (usec): min=3042, max=97338, avg=15110.97, stdev=12216.68 lat (usec): min=3042, max=97339, avg=15112.20, stdev=12216.66 clat percentiles (usec): | 1.00th=[ 3425], 5.00th=[ 4424], 10.00th=[ 5211], 20.00th=[ 6390], | 30.00th=[ 7308], 40.00th=[ 8455], 50.00th=[10683], 60.00th=[13566], | 70.00th=[17433], 80.00th=[23200], 90.00th=[31327], 95.00th=[36963], | 99.00th=[63177], 99.50th=[82314], 99.90th=[96994], 99.95th=[96994], | 99.99th=[96994] bw ( KiB/s): min= 192, max= 576, per=26.01%, avg=395.03, stdev=98.73, samples=31 iops : min= 12, max= 36, avg=24.52, stdev= 6.09, samples=31 write: IOPS=23, BW=383KiB/s (392kB/s)(6032KiB/15762msec) clat (msec): min=5, max=121, avg=26.08, stdev=16.06 lat (msec): min=5, max=121, avg=26.08, stdev=16.06 clat percentiles (msec): | 1.00th=[ 8], 5.00th=[ 9], 10.00th=[ 11], 20.00th=[ 14], | 30.00th=[ 16], 40.00th=[ 20], 50.00th=[ 23], 60.00th=[ 26], | 70.00th=[ 30], 80.00th=[ 36], 90.00th=[ 47], 95.00th=[ 61], | 99.00th=[ 85], 99.50th=[ 95], 99.90th=[ 123], 99.95th=[ 123], | 99.99th=[ 123] bw ( KiB/s): min= 128, max= 576, per=23.56%, avg=376.52, stdev=107.61, samples=31 iops : min= 8, max= 36, avg=23.35, stdev= 6.73, samples=31 lat (msec) : 4=1.17%, 10=26.56%, 20=29.82%, 50=37.76%, 100=4.56% lat (msec) : 250=0.13% cpu : usr=0.06%, sys=6.05%, ctx=1146, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9121: Mon Apr 1 23:07:04 2024 read: IOPS=23, BW=384KiB/s (393kB/s)(6032KiB/15721msec) clat (msec): min=2, max=152, avg=16.10, stdev=15.37 lat (msec): min=2, max=152, avg=16.10, stdev=15.37 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 5], 10.00th=[ 6], 20.00th=[ 7], | 30.00th=[ 8], 40.00th=[ 9], 50.00th=[ 10], 60.00th=[ 14], | 70.00th=[ 19], 80.00th=[ 25], 90.00th=[ 34], 95.00th=[ 42], | 99.00th=[ 83], 99.50th=[ 95], 99.90th=[ 153], 99.95th=[ 153], | 99.99th=[ 153] bw ( KiB/s): min= 64, max= 576, per=24.98%, avg=379.45, stdev=116.37, samples=31 iops : min= 4, max= 36, avg=23.45, stdev= 7.21, samples=31 write: IOPS=24, BW=398KiB/s (407kB/s)(6256KiB/15721msec) clat (msec): min=6, max=127, avg=24.65, stdev=14.17 lat (msec): min=6, max=127, avg=24.65, stdev=14.17 clat percentiles (msec): | 1.00th=[ 8], 5.00th=[ 10], 10.00th=[ 12], 20.00th=[ 14], | 30.00th=[ 16], 40.00th=[ 19], 50.00th=[ 22], 60.00th=[ 24], | 70.00th=[ 28], 80.00th=[ 33], 90.00th=[ 41], 95.00th=[ 51], | 99.00th=[ 81], 99.50th=[ 102], 99.90th=[ 128], 99.95th=[ 128], | 99.99th=[ 128] bw ( KiB/s): min= 224, max= 576, per=24.65%, avg=393.87, stdev=84.01, samples=31 iops : min= 14, max= 36, avg=24.35, stdev= 5.28, samples=31 lat (msec) : 4=1.30%, 10=26.17%, 20=30.86%, 50=37.24%, 100=4.04% lat (msec) : 250=0.39% cpu : usr=0.11%, sys=5.99%, ctx=1114, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1520KiB/s (1556kB/s), 367KiB/s-397KiB/s (376kB/s-406kB/s), io=23.4MiB (24.5MB), run=15295-15762msec WRITE: bw=1599KiB/s (1637kB/s), 383KiB/s-418KiB/s (392kB/s-428kB/s), io=24.6MiB (25.8MB), run=15295-15762msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=9143: Mon Apr 1 23:07:11 2024 read: IOPS=1, BW=1154KiB/s (1181kB/s)(7168KiB/6213msec) clat (usec): min=582, max=1375.9k, avg=656830.11, stdev=600330.30 lat (usec): min=583, max=1375.9k, avg=656831.66, stdev=600330.26 clat percentiles (usec): | 1.00th=[ 586], 5.00th=[ 586], 10.00th=[ 586], | 20.00th=[ 66847], 30.00th=[ 135267], 40.00th=[ 135267], | 50.00th=[ 616563], 60.00th=[1149240], 70.00th=[1149240], | 80.00th=[1249903], 90.00th=[1384121], 95.00th=[1384121], | 99.00th=[1384121], 99.50th=[1384121], 99.90th=[1384121], | 99.95th=[1384121], 99.99th=[1384121] bw ( KiB/s): min= 2048, max= 4071, per=75.33%, avg=2452.60, stdev=904.71, samples=5 iops : min= 2, max= 3, avg= 2.20, stdev= 0.45, samples=5 write: IOPS=0, BW=824KiB/s (844kB/s)(5120KiB/6213msec) clat (msec): min=37, max=1248, avg=321.99, stdev=521.28 lat (msec): min=37, max=1248, avg=322.84, stdev=520.86 clat percentiles (msec): | 1.00th=[ 37], 5.00th=[ 37], 10.00th=[ 37], 20.00th=[ 37], | 30.00th=[ 42], 40.00th=[ 42], 50.00th=[ 100], 60.00th=[ 100], | 70.00th=[ 184], 80.00th=[ 184], 90.00th=[ 1250], 95.00th=[ 1250], | 99.00th=[ 1250], 99.50th=[ 1250], 99.90th=[ 1250], 99.95th=[ 1250], | 99.99th=[ 1250] bw ( KiB/s): min= 2048, max= 4096, per=74.87%, avg=3413.33, stdev=1182.41, samples=3 iops : min= 2, max= 4, avg= 3.33, stdev= 1.15, samples=3 lat (usec) : 750=8.33% lat (msec) : 50=16.67%, 100=16.67%, 250=16.67%, 750=8.33% cpu : usr=0.16%, sys=13.62%, ctx=1048, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9144: Mon Apr 1 23:07:11 2024 read: IOPS=0, BW=326KiB/s (334kB/s)(2048KiB/6288msec) clat (msec): min=102, max=350, avg=226.89, stdev=175.28 lat (msec): min=102, max=350, avg=226.89, stdev=175.28 clat percentiles (msec): | 1.00th=[ 104], 5.00th=[ 104], 10.00th=[ 104], 20.00th=[ 104], | 30.00th=[ 104], 40.00th=[ 104], 50.00th=[ 104], 60.00th=[ 351], | 70.00th=[ 351], 80.00th=[ 351], 90.00th=[ 351], 95.00th=[ 351], | 99.00th=[ 351], 99.50th=[ 351], 99.90th=[ 351], 99.95th=[ 351], | 99.99th=[ 351] bw ( KiB/s): min= 2039, max= 2043, per=62.68%, avg=2041.00, stdev= 2.83, samples=2 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=2 write: IOPS=1, BW=1628KiB/s (1668kB/s)(10.0MiB/6288msec) clat (msec): min=56, max=1488, avg=583.22, stdev=582.16 lat (msec): min=56, max=1488, avg=583.30, stdev=582.18 clat percentiles (msec): | 1.00th=[ 57], 5.00th=[ 57], 10.00th=[ 57], 20.00th=[ 61], | 30.00th=[ 66], 40.00th=[ 79], 50.00th=[ 262], 60.00th=[ 355], | 70.00th=[ 1200], 80.00th=[ 1200], 90.00th=[ 1217], 95.00th=[ 1485], | 99.00th=[ 1485], 99.50th=[ 1485], 99.90th=[ 1485], 99.95th=[ 1485], | 99.99th=[ 1485] bw ( KiB/s): min= 2039, max= 4096, per=59.81%, avg=2726.83, stdev=1057.08, samples=6 iops : min= 1, max= 4, avg= 2.17, stdev= 1.17, samples=6 lat (msec) : 100=33.33%, 250=8.33%, 500=25.00% cpu : usr=0.02%, sys=6.22%, ctx=129, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9146: Mon Apr 1 23:07:11 2024 read: IOPS=0, BW=1000KiB/s (1024kB/s)(6144KiB/6144msec) clat (usec): min=1678, max=1268.5k, avg=448637.11, stdev=604743.60 lat (usec): min=1679, max=1268.5k, avg=448638.42, stdev=604743.71 clat percentiles (usec): | 1.00th=[ 1680], 5.00th=[ 1680], 10.00th=[ 1680], | 20.00th=[ 22152], 30.00th=[ 22152], 40.00th=[ 73925], | 50.00th=[ 73925], 60.00th=[ 141558], 70.00th=[1182794], | 80.00th=[1182794], 90.00th=[1266680], 95.00th=[1266680], | 99.00th=[1266680], 99.50th=[1266680], 99.90th=[1266680], | 99.95th=[1266680], 99.99th=[1266680] bw ( KiB/s): min= 2035, max= 4087, per=78.39%, avg=2552.25, stdev=1023.18, samples=4 iops : min= 1, max= 3, avg= 1.75, stdev= 0.96, samples=4 write: IOPS=0, BW=1000KiB/s (1024kB/s)(6144KiB/6144msec) clat (msec): min=44, max=2398, avg=574.54, stdev=905.51 lat (msec): min=44, max=2398, avg=574.63, stdev=905.50 clat percentiles (msec): | 1.00th=[ 45], 5.00th=[ 45], 10.00th=[ 45], 20.00th=[ 70], | 30.00th=[ 70], 40.00th=[ 215], 50.00th=[ 215], 60.00th=[ 279], | 70.00th=[ 443], 80.00th=[ 443], 90.00th=[ 2400], 95.00th=[ 2400], | 99.00th=[ 2400], 99.50th=[ 2400], 99.90th=[ 2400], 99.95th=[ 2400], | 99.99th=[ 2400] bw ( KiB/s): min= 2039, max= 4087, per=67.10%, avg=3059.25, stdev=1172.93, samples=4 iops : min= 1, max= 3, avg= 2.25, stdev= 0.96, samples=4 lat (msec) : 2=8.33%, 50=16.67%, 100=16.67%, 250=16.67%, 500=16.67% cpu : usr=0.13%, sys=8.37%, ctx=637, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9147: Mon Apr 1 23:07:11 2024 read: IOPS=0, BW=837KiB/s (857kB/s)(5120KiB/6118msec) clat (usec): min=371, max=1043.8k, avg=228786.51, stdev=457589.06 lat (usec): min=372, max=1043.8k, avg=228787.83, stdev=457589.26 clat percentiles (usec): | 1.00th=[ 371], 5.00th=[ 371], 10.00th=[ 371], | 20.00th=[ 371], 30.00th=[ 396], 40.00th=[ 396], | 50.00th=[ 586], 60.00th=[ 586], 70.00th=[ 99091], | 80.00th=[ 99091], 90.00th=[1044382], 95.00th=[1044382], | 99.00th=[1044382], 99.50th=[1044382], 99.90th=[1044382], | 99.95th=[1044382], 99.99th=[1044382] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=1, BW=1172KiB/s (1200kB/s)(7168KiB/6118msec) clat (msec): min=81, max=2458, avg=707.26, stdev=853.90 lat (msec): min=81, max=2461, avg=707.80, stdev=855.04 clat percentiles (msec): | 1.00th=[ 82], 5.00th=[ 82], 10.00th=[ 82], 20.00th=[ 228], | 30.00th=[ 243], 40.00th=[ 243], 50.00th=[ 266], 60.00th=[ 493], | 70.00th=[ 493], 80.00th=[ 1183], 90.00th=[ 2467], 95.00th=[ 2467], | 99.00th=[ 2467], 99.50th=[ 2467], 99.90th=[ 2467], 99.95th=[ 2467], | 99.99th=[ 2467] bw ( KiB/s): min= 2043, max= 4096, per=62.69%, avg=2858.00, stdev=1111.52, samples=5 iops : min= 1, max= 4, avg= 2.40, stdev= 1.14, samples=5 lat (usec) : 500=16.67%, 750=8.33% lat (msec) : 100=16.67%, 250=16.67%, 500=16.67% cpu : usr=0.00%, sys=4.84%, ctx=296, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=3257KiB/s (3335kB/s), 326KiB/s-1154KiB/s (334kB/s-1181kB/s), io=20.0MiB (20.0MB), run=6118-6288msec WRITE: bw=4560KiB/s (4669kB/s), 824KiB/s-1628KiB/s (844kB/s-1668kB/s), io=28.0MiB (29.4MB), run=6118-6288msec rand-rw: (groupid=0, jobs=1): err= 0: pid=9139: Mon Apr 1 23:07:11 2024 read: IOPS=1, BW=1908KiB/s (1954kB/s)(7168KiB/3756msec) clat (msec): min=45, max=1375, avg=393.56, stdev=568.24 lat (msec): min=45, max=1375, avg=393.56, stdev=568.24 clat percentiles (msec): | 1.00th=[ 46], 5.00th=[ 46], 10.00th=[ 46], 20.00th=[ 52], | 30.00th=[ 69], 40.00th=[ 69], 50.00th=[ 82], 60.00th=[ 82], | 70.00th=[ 82], 80.00th=[ 1053], 90.00th=[ 1368], 95.00th=[ 1368], | 99.00th=[ 1368], 99.50th=[ 1368], 99.90th=[ 1368], 99.95th=[ 1368], | 99.99th=[ 1368] bw ( KiB/s): min= 2043, max= 4087, per=100.00%, avg=3403.00, stdev=1177.80, samples=3 iops : min= 1, max= 3, avg= 2.33, stdev= 1.15, samples=3 write: IOPS=1, BW=1363KiB/s (1396kB/s)(5120KiB/3756msec) clat (msec): min=71, max=583, avg=199.71, stdev=216.01 lat (msec): min=71, max=583, avg=199.81, stdev=216.01 clat percentiles (msec): | 1.00th=[ 72], 5.00th=[ 72], 10.00th=[ 72], 20.00th=[ 72], | 30.00th=[ 91], 40.00th=[ 91], 50.00th=[ 118], 60.00th=[ 118], | 70.00th=[ 136], 80.00th=[ 136], 90.00th=[ 584], 95.00th=[ 584], | 99.00th=[ 584], 99.50th=[ 584], 99.90th=[ 584], 99.95th=[ 584], | 99.99th=[ 584] bw ( KiB/s): min= 2043, max= 4079, per=67.93%, avg=3061.00, stdev=1439.67, samples=2 iops : min= 1, max= 3, avg= 2.00, stdev= 1.41, samples=2 lat (msec) : 50=8.33%, 100=50.00%, 250=16.67%, 750=8.33% cpu : usr=0.00%, sys=3.89%, ctx=55, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9140: Mon Apr 1 23:07:11 2024 read: IOPS=0, BW=409KiB/s (419kB/s)(2048KiB/5007msec) clat (msec): min=68, max=118, avg=93.61, stdev=35.33 lat (msec): min=68, max=118, avg=93.61, stdev=35.33 clat percentiles (msec): | 1.00th=[ 69], 5.00th=[ 69], 10.00th=[ 69], 20.00th=[ 69], | 30.00th=[ 69], 40.00th=[ 69], 50.00th=[ 69], 60.00th=[ 120], | 70.00th=[ 120], 80.00th=[ 120], 90.00th=[ 120], 95.00th=[ 120], | 99.00th=[ 120], 99.50th=[ 120], 99.90th=[ 120], 99.95th=[ 120], | 99.99th=[ 120] bw ( KiB/s): min= 2043, max= 2048, per=63.54%, avg=2045.50, stdev= 3.54, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 write: IOPS=1, BW=2045KiB/s (2094kB/s)(10.0MiB/5007msec) clat (msec): min=56, max=1517, avg=481.63, stdev=552.18 lat (msec): min=56, max=1517, avg=481.72, stdev=552.18 clat percentiles (msec): | 1.00th=[ 56], 5.00th=[ 56], 10.00th=[ 56], 20.00th=[ 64], | 30.00th=[ 67], 40.00th=[ 79], 50.00th=[ 178], 60.00th=[ 184], | 70.00th=[ 1045], 80.00th=[ 1045], 90.00th=[ 1183], 95.00th=[ 1519], | 99.00th=[ 1519], 99.50th=[ 1519], 99.90th=[ 1519], 99.95th=[ 1519], | 99.99th=[ 1519] bw ( KiB/s): min= 2048, max= 6144, per=100.00%, avg=4608.00, stdev=1960.81, samples=4 iops : min= 2, max= 6, avg= 4.50, stdev= 1.91, samples=4 lat (msec) : 100=41.67%, 250=25.00%, 500=8.33% cpu : usr=0.00%, sys=2.92%, ctx=60, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9141: Mon Apr 1 23:07:11 2024 read: IOPS=0, BW=966KiB/s (989kB/s)(6144KiB/6362msec) clat (msec): min=62, max=1274, avg=465.43, stdev=540.58 lat (msec): min=62, max=1274, avg=465.43, stdev=540.58 clat percentiles (msec): | 1.00th=[ 63], 5.00th=[ 63], 10.00th=[ 63], 20.00th=[ 78], | 30.00th=[ 78], 40.00th=[ 107], 50.00th=[ 107], 60.00th=[ 247], | 70.00th=[ 1028], 80.00th=[ 1028], 90.00th=[ 1267], 95.00th=[ 1267], | 99.00th=[ 1267], 99.50th=[ 1267], 99.90th=[ 1267], 99.95th=[ 1267], | 99.99th=[ 1267] bw ( KiB/s): min= 2048, max= 4087, per=79.46%, avg=2557.75, stdev=1019.50, samples=4 iops : min= 2, max= 3, avg= 2.25, stdev= 0.50, samples=4 write: IOPS=0, BW=966KiB/s (989kB/s)(6144KiB/6362msec) clat (msec): min=56, max=2406, avg=593.92, stdev=902.04 lat (msec): min=56, max=2406, avg=594.03, stdev=902.03 clat percentiles (msec): | 1.00th=[ 57], 5.00th=[ 57], 10.00th=[ 57], 20.00th=[ 118], | 30.00th=[ 118], 40.00th=[ 228], 50.00th=[ 228], 60.00th=[ 239], | 70.00th=[ 518], 80.00th=[ 518], 90.00th=[ 2400], 95.00th=[ 2400], | 99.00th=[ 2400], 99.50th=[ 2400], 99.90th=[ 2400], 99.95th=[ 2400], | 99.99th=[ 2400] bw ( KiB/s): min= 2048, max= 4096, per=68.13%, avg=3069.75, stdev=1179.82, samples=4 iops : min= 2, max= 4, avg= 2.75, stdev= 0.96, samples=4 lat (msec) : 100=25.00%, 250=41.67%, 750=8.33% cpu : usr=0.00%, sys=2.63%, ctx=64, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9142: Mon Apr 1 23:07:11 2024 read: IOPS=0, BW=809KiB/s (829kB/s)(5120KiB/6326msec) clat (msec): min=36, max=1244, avg=304.66, stdev=526.28 lat (msec): min=36, max=1244, avg=304.67, stdev=526.28 clat percentiles (msec): | 1.00th=[ 36], 5.00th=[ 36], 10.00th=[ 36], 20.00th=[ 36], | 30.00th=[ 47], 40.00th=[ 47], 50.00th=[ 83], 60.00th=[ 83], | 70.00th=[ 114], 80.00th=[ 114], 90.00th=[ 1250], 95.00th=[ 1250], | 99.00th=[ 1250], 99.50th=[ 1250], 99.90th=[ 1250], 99.95th=[ 1250], | 99.99th=[ 1250] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=1, BW=1133KiB/s (1160kB/s)(7168KiB/6326msec) clat (msec): min=99, max=2473, avg=685.72, stdev=871.09 lat (msec): min=99, max=2473, avg=685.82, stdev=871.11 clat percentiles (msec): | 1.00th=[ 101], 5.00th=[ 101], 10.00th=[ 101], 20.00th=[ 222], | 30.00th=[ 224], 40.00th=[ 224], 50.00th=[ 230], 60.00th=[ 355], | 70.00th=[ 355], 80.00th=[ 1200], 90.00th=[ 2467], 95.00th=[ 2467], | 99.00th=[ 2467], 99.50th=[ 2467], 99.90th=[ 2467], 99.95th=[ 2467], | 99.99th=[ 2467] bw ( KiB/s): min= 2043, max= 4096, per=63.57%, avg=2864.40, stdev=1120.19, samples=5 iops : min= 1, max= 4, avg= 2.40, stdev= 1.14, samples=5 lat (msec) : 50=16.67%, 100=16.67%, 250=33.33%, 500=8.33% cpu : usr=0.00%, sys=2.64%, ctx=64, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=3219KiB/s (3296kB/s), 409KiB/s-1908KiB/s (419kB/s-1954kB/s), io=20.0MiB (20.0MB), run=3756-6362msec WRITE: bw=4507KiB/s (4615kB/s), 966KiB/s-2045KiB/s (989kB/s-2094kB/s), io=28.0MiB (29.4MB), run=3756-6362msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=9158: Mon Apr 1 23:07:18 2024 read: IOPS=0, BW=2390KiB/s (2448kB/s)(12.0MiB/5141msec) clat (msec): min=295, max=2909, avg=1712.91, stdev=1320.78 lat (msec): min=295, max=2909, avg=1712.91, stdev=1320.78 clat percentiles (msec): | 1.00th=[ 296], 5.00th=[ 296], 10.00th=[ 296], 20.00th=[ 296], | 30.00th=[ 296], 40.00th=[ 1938], 50.00th=[ 1938], 60.00th=[ 1938], | 70.00th=[ 2903], 80.00th=[ 2903], 90.00th=[ 2903], 95.00th=[ 2903], | 99.00th=[ 2903], 99.50th=[ 2903], 99.90th=[ 2903], 99.95th=[ 2903], | 99.99th=[ 2903] bw ( KiB/s): min= 6400, max= 8192, per=100.00%, avg=7296.00, stdev=1267.14, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 500=33.33% cpu : usr=0.25%, sys=22.32%, ctx=1234, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9160: Mon Apr 1 23:07:18 2024 read: IOPS=0, BW=1173KiB/s (1201kB/s)(4096KiB/3493msec) clat (nsec): min=299321k, max=299321k, avg=299320949.00, stdev= 0.00 lat (nsec): min=299323k, max=299323k, avg=299322831.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 300], 5.00th=[ 300], 10.00th=[ 300], 20.00th=[ 300], | 30.00th=[ 300], 40.00th=[ 300], 50.00th=[ 300], 60.00th=[ 300], | 70.00th=[ 300], 80.00th=[ 300], 90.00th=[ 300], 95.00th=[ 300], | 99.00th=[ 300], 99.50th=[ 300], 99.90th=[ 300], 99.95th=[ 300], | 99.99th=[ 300] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=0, BW=2345KiB/s (2402kB/s)(8192KiB/3493msec) clat (msec): min=385, max=2800, avg=1592.79, stdev=1707.44 lat (msec): min=385, max=2800, avg=1593.13, stdev=1707.59 clat percentiles (msec): | 1.00th=[ 384], 5.00th=[ 384], 10.00th=[ 384], 20.00th=[ 384], | 30.00th=[ 384], 40.00th=[ 384], 50.00th=[ 384], 60.00th=[ 2802], | 70.00th=[ 2802], 80.00th=[ 2802], 90.00th=[ 2802], 95.00th=[ 2802], | 99.00th=[ 2802], 99.50th=[ 2802], 99.90th=[ 2802], 99.95th=[ 2802], | 99.99th=[ 2802] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 500=66.67% cpu : usr=0.00%, sys=7.53%, ctx=143, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9163: Mon Apr 1 23:07:18 2024 write: IOPS=0, BW=2377KiB/s (2434kB/s)(12.0MiB/5169msec) clat (msec): min=609, max=2783, avg=1722.34, stdev=1087.71 lat (msec): min=609, max=2783, avg=1722.62, stdev=1087.76 clat percentiles (msec): | 1.00th=[ 609], 5.00th=[ 609], 10.00th=[ 609], 20.00th=[ 609], | 30.00th=[ 609], 40.00th=[ 1770], 50.00th=[ 1770], 60.00th=[ 1770], | 70.00th=[ 2769], 80.00th=[ 2769], 90.00th=[ 2769], 95.00th=[ 2769], | 99.00th=[ 2769], 99.50th=[ 2769], 99.90th=[ 2769], 99.95th=[ 2769], | 99.99th=[ 2769] bw ( KiB/s): min= 6131, max= 8175, per=100.00%, avg=7153.00, stdev=1445.33, samples=2 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=2 lat (msec) : 750=33.33% cpu : usr=0.00%, sys=5.61%, ctx=228, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9165: Mon Apr 1 23:07:18 2024 write: IOPS=0, BW=3644KiB/s (3732kB/s)(12.0MiB/3372msec) clat (msec): min=175, max=2835, avg=1121.66, stdev=1487.06 lat (msec): min=175, max=2835, avg=1122.01, stdev=1487.02 clat percentiles (msec): | 1.00th=[ 176], 5.00th=[ 176], 10.00th=[ 176], 20.00th=[ 176], | 30.00th=[ 176], 40.00th=[ 355], 50.00th=[ 355], 60.00th=[ 355], | 70.00th=[ 2836], 80.00th=[ 2836], 90.00th=[ 2836], 95.00th=[ 2836], | 99.00th=[ 2836], 99.50th=[ 2836], 99.90th=[ 2836], 99.95th=[ 2836], | 99.99th=[ 2836] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=33.33% cpu : usr=0.03%, sys=9.82%, ctx=128, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=3187KiB/s (3263kB/s), 1173KiB/s-2390KiB/s (1201kB/s-2448kB/s), io=16.0MiB (16.8MB), run=3493-5141msec WRITE: bw=6339KiB/s (6491kB/s), 2345KiB/s-3644KiB/s (2402kB/s-3732kB/s), io=32.0MiB (33.6MB), run=3372-5169msec rand-rw: (groupid=0, jobs=1): err= 0: pid=9159: Mon Apr 1 23:07:18 2024 read: IOPS=0, BW=2315KiB/s (2371kB/s)(12.0MiB/5307msec) clat (msec): min=861, max=2944, avg=1768.57, stdev=1067.41 lat (msec): min=861, max=2944, avg=1768.57, stdev=1067.41 clat percentiles (msec): | 1.00th=[ 860], 5.00th=[ 860], 10.00th=[ 860], 20.00th=[ 860], | 30.00th=[ 860], 40.00th=[ 1502], 50.00th=[ 1502], 60.00th=[ 1502], | 70.00th=[ 2937], 80.00th=[ 2937], 90.00th=[ 2937], 95.00th=[ 2937], | 99.00th=[ 2937], 99.50th=[ 2937], 99.90th=[ 2937], 99.95th=[ 2937], | 99.99th=[ 2937] bw ( KiB/s): min= 6532, max= 6543, per=100.00%, avg=6537.50, stdev= 7.78, samples=2 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=2 lat (msec) : 1000=33.33% cpu : usr=0.00%, sys=3.37%, ctx=68, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9161: Mon Apr 1 23:07:18 2024 read: IOPS=0, BW=1023KiB/s (1047kB/s)(4096KiB/4005msec) clat (nsec): min=396344k, max=396344k, avg=396344489.00, stdev= 0.00 lat (nsec): min=396348k, max=396348k, avg=396347534.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 397], 5.00th=[ 397], 10.00th=[ 397], 20.00th=[ 397], | 30.00th=[ 397], 40.00th=[ 397], 50.00th=[ 397], 60.00th=[ 397], | 70.00th=[ 397], 80.00th=[ 397], 90.00th=[ 397], 95.00th=[ 397], | 99.00th=[ 397], 99.50th=[ 397], 99.90th=[ 397], 99.95th=[ 397], | 99.99th=[ 397] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=0, BW=2045KiB/s (2095kB/s)(8192KiB/4005msec) clat (msec): min=648, max=2913, avg=1780.77, stdev=1601.85 lat (msec): min=648, max=2913, avg=1780.99, stdev=1601.91 clat percentiles (msec): | 1.00th=[ 651], 5.00th=[ 651], 10.00th=[ 651], 20.00th=[ 651], | 30.00th=[ 651], 40.00th=[ 651], 50.00th=[ 651], 60.00th=[ 2903], | 70.00th=[ 2903], 80.00th=[ 2903], 90.00th=[ 2903], 95.00th=[ 2903], | 99.00th=[ 2903], 99.50th=[ 2903], 99.90th=[ 2903], 99.95th=[ 2903], | 99.99th=[ 2903] bw ( KiB/s): min= 5542, max= 5542, per=67.74%, avg=5542.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 500=33.33%, 750=33.33% cpu : usr=0.07%, sys=3.87%, ctx=64, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9162: Mon Apr 1 23:07:18 2024 write: IOPS=4, BW=18.6MiB/s (19.5MB/s)(12.0MiB/644msec) clat (msec): min=119, max=381, avg=211.59, stdev=147.63 lat (msec): min=119, max=382, avg=212.06, stdev=147.55 clat percentiles (msec): | 1.00th=[ 120], 5.00th=[ 120], 10.00th=[ 120], 20.00th=[ 120], | 30.00th=[ 120], 40.00th=[ 134], 50.00th=[ 134], 60.00th=[ 134], | 70.00th=[ 384], 80.00th=[ 384], 90.00th=[ 384], 95.00th=[ 384], | 99.00th=[ 384], 99.50th=[ 384], 99.90th=[ 384], 99.95th=[ 384], | 99.99th=[ 384] bw ( KiB/s): min=16351, max=16351, per=100.00%, avg=16351.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.78%, sys=22.24%, ctx=56, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9164: Mon Apr 1 23:07:18 2024 write: IOPS=2, BW=11.9MiB/s (12.5MB/s)(12.0MiB/1005msec) clat (msec): min=124, max=708, avg=334.15, stdev=324.87 lat (msec): min=124, max=708, avg=334.51, stdev=324.76 clat percentiles (msec): | 1.00th=[ 125], 5.00th=[ 125], 10.00th=[ 125], 20.00th=[ 125], | 30.00th=[ 125], 40.00th=[ 171], 50.00th=[ 171], 60.00th=[ 171], | 70.00th=[ 709], 80.00th=[ 709], 90.00th=[ 709], 95.00th=[ 709], | 99.00th=[ 709], 99.50th=[ 709], 99.90th=[ 709], 99.95th=[ 709], | 99.99th=[ 709] bw ( KiB/s): min=16384, max=16384, per=100.00%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 750=33.33% cpu : usr=0.00%, sys=11.65%, ctx=39, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=3087KiB/s (3161kB/s), 1023KiB/s-2315KiB/s (1047kB/s-2371kB/s), io=16.0MiB (16.8MB), run=4005-5307msec WRITE: bw=8182KiB/s (8378kB/s), 2045KiB/s-18.6MiB/s (2095kB/s-19.5MB/s), io=32.0MiB (33.6MB), run=644-4005msec