== sanity test 398b: DIO and buffer IO race ============== 13:43:39 (1750182219) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.89694 s, 26.5 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=9713: Tue Jun 17 13:44:17 2025 read: IOPS=44, BW=177KiB/s (181kB/s)(5992KiB/33946msec) clat (usec): min=59, max=245068, avg=12573.24, stdev=10168.90 lat (usec): min=59, max=245069, avg=12576.52, stdev=10170.45 clat percentiles (usec): | 1.00th=[ 89], 5.00th=[ 118], 10.00th=[ 4228], 20.00th=[ 6521], | 30.00th=[ 8094], 40.00th=[ 9503], 50.00th=[ 10945], 60.00th=[ 12649], | 70.00th=[ 14877], 80.00th=[ 17957], 90.00th=[ 22676], 95.00th=[ 26870], | 99.00th=[ 41157], 99.50th=[ 47973], 99.90th=[ 78119], 99.95th=[244319], | 99.99th=[244319] bw ( KiB/s): min= 39, max= 279, per=24.36%, avg=172.25, stdev=44.71, samples=67 iops : min= 9, max= 69, avg=42.72, stdev=11.24, samples=67 write: IOPS=46, BW=185KiB/s (190kB/s)(6296KiB/33946msec) clat (usec): min=489, max=76215, avg=9489.11, stdev=10182.53 lat (usec): min=489, max=76218, avg=9490.73, stdev=10183.03 clat percentiles (usec): | 1.00th=[ 660], 5.00th=[ 742], 10.00th=[ 791], 20.00th=[ 938], | 30.00th=[ 1631], 40.00th=[ 3130], 50.00th=[ 6390], 60.00th=[ 9110], | 70.00th=[12518], 80.00th=[16450], 90.00th=[23987], 95.00th=[29492], | 99.00th=[42730], 99.50th=[48497], 99.90th=[72877], 99.95th=[76022], | 99.99th=[76022] bw ( KiB/s): min= 31, max= 320, per=24.55%, avg=181.69, stdev=49.24, samples=67 iops : min= 7, max= 80, avg=45.07, stdev=12.37, samples=67 lat (usec) : 100=1.30%, 250=2.31%, 500=0.29%, 750=2.93%, 1000=8.24% lat (msec) : 2=6.05%, 4=5.37%, 10=26.56%, 20=32.26%, 50=14.36% lat (msec) : 100=0.29%, 250=0.03% cpu : usr=0.18%, sys=12.75%, ctx=4949, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9714: Tue Jun 17 13:44:17 2025 read: IOPS=44, BW=177KiB/s (181kB/s)(5940KiB/33581msec) clat (usec): min=69, max=79052, avg=12179.80, stdev=8497.39 lat (usec): min=70, max=79054, avg=12180.79, stdev=8497.43 clat percentiles (usec): | 1.00th=[ 91], 5.00th=[ 116], 10.00th=[ 4080], 20.00th=[ 6128], | 30.00th=[ 7570], 40.00th=[ 9110], 50.00th=[10683], 60.00th=[12387], | 70.00th=[14615], 80.00th=[17171], 90.00th=[22152], 95.00th=[26608], | 99.00th=[43779], 99.50th=[51119], 99.90th=[68682], 99.95th=[79168], | 99.99th=[79168] bw ( KiB/s): min= 55, max= 264, per=24.95%, avg=176.42, stdev=38.41, samples=67 iops : min= 13, max= 66, avg=43.76, stdev= 9.61, samples=67 write: IOPS=47, BW=189KiB/s (194kB/s)(6348KiB/33581msec) clat (usec): min=486, max=264812, avg=9671.49, stdev=11875.48 lat (usec): min=487, max=264813, avg=9674.80, stdev=11880.16 clat percentiles (usec): | 1.00th=[ 676], 5.00th=[ 750], 10.00th=[ 791], 20.00th=[ 963], | 30.00th=[ 1713], 40.00th=[ 3228], 50.00th=[ 6390], 60.00th=[ 9241], | 70.00th=[ 12518], 80.00th=[ 17171], 90.00th=[ 23462], 95.00th=[ 29492], | 99.00th=[ 42730], 99.50th=[ 50070], 99.90th=[ 68682], 99.95th=[265290], | 99.99th=[265290] bw ( KiB/s): min= 79, max= 296, per=25.45%, avg=188.33, stdev=48.96, samples=67 iops : min= 19, max= 74, avg=46.75, stdev=12.28, samples=67 lat (usec) : 100=1.24%, 250=2.21%, 500=0.20%, 750=2.96%, 1000=8.01% lat (msec) : 2=6.12%, 4=6.05%, 10=27.93%, 20=30.73%, 50=14.03% lat (msec) : 100=0.49%, 500=0.03% cpu : usr=0.11%, sys=13.13%, ctx=5034, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9715: Tue Jun 17 13:44:17 2025 read: IOPS=44, BW=178KiB/s (182kB/s)(6028KiB/33895msec) clat (usec): min=49, max=76266, avg=12798.77, stdev=8841.36 lat (usec): min=49, max=76267, avg=12799.72, stdev=8841.38 clat percentiles (usec): | 1.00th=[ 89], 5.00th=[ 118], 10.00th=[ 4178], 20.00th=[ 6456], | 30.00th=[ 7963], 40.00th=[ 9634], 50.00th=[11207], 60.00th=[13042], | 70.00th=[15533], 80.00th=[18220], 90.00th=[22938], 95.00th=[28181], | 99.00th=[43254], 99.50th=[51119], 99.90th=[69731], 99.95th=[76022], | 99.99th=[76022] bw ( KiB/s): min= 23, max= 264, per=24.91%, avg=176.10, stdev=40.35, samples=67 iops : min= 5, max= 66, avg=43.79, stdev=10.13, samples=67 write: IOPS=46, BW=185KiB/s (189kB/s)(6260KiB/33895msec) clat (usec): min=466, max=259366, avg=9231.51, stdev=11968.93 lat (usec): min=467, max=259370, avg=9233.98, stdev=11968.68 clat percentiles (usec): | 1.00th=[ 668], 5.00th=[ 742], 10.00th=[ 783], 20.00th=[ 963], | 30.00th=[ 1598], 40.00th=[ 2769], 50.00th=[ 5997], 60.00th=[ 8848], | 70.00th=[ 11600], 80.00th=[ 15270], 90.00th=[ 21890], 95.00th=[ 29492], | 99.00th=[ 46924], 99.50th=[ 57410], 99.90th=[ 79168], 99.95th=[258999], | 99.99th=[258999] bw ( KiB/s): min= 31, max= 302, per=24.70%, avg=182.75, stdev=55.05, samples=67 iops : min= 7, max= 75, avg=45.46, stdev=13.78, samples=67 lat (usec) : 50=0.07%, 100=1.20%, 250=2.34%, 500=0.36%, 750=3.19% lat (usec) : 1000=7.81% lat (msec) : 2=6.48%, 4=5.76%, 10=26.69%, 20=32.23%, 50=13.18% lat (msec) : 100=0.65%, 500=0.03% cpu : usr=0.18%, sys=12.86%, ctx=5026, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9716: Tue Jun 17 13:44:17 2025 read: IOPS=44, BW=179KiB/s (184kB/s)(6052KiB/33718msec) clat (usec): min=70, max=254599, avg=12721.59, stdev=10516.01 lat (usec): min=70, max=254601, avg=12722.79, stdev=10515.92 clat percentiles (usec): | 1.00th=[ 91], 5.00th=[ 120], 10.00th=[ 4621], 20.00th=[ 6587], | 30.00th=[ 7963], 40.00th=[ 9241], 50.00th=[ 11076], 60.00th=[ 12649], | 70.00th=[ 15139], 80.00th=[ 18220], 90.00th=[ 22676], 95.00th=[ 27395], | 99.00th=[ 40633], 99.50th=[ 49546], 99.90th=[ 80217], 99.95th=[254804], | 99.99th=[254804] bw ( KiB/s): min= 79, max= 287, per=25.25%, avg=178.54, stdev=40.01, samples=67 iops : min= 19, max= 71, avg=44.30, stdev=10.02, samples=67 write: IOPS=46, BW=185KiB/s (189kB/s)(6236KiB/33718msec) clat (usec): min=464, max=76122, avg=9197.66, stdev=9721.59 lat (usec): min=465, max=76123, avg=9199.03, stdev=9721.43 clat percentiles (usec): | 1.00th=[ 676], 5.00th=[ 742], 10.00th=[ 775], 20.00th=[ 889], | 30.00th=[ 1401], 40.00th=[ 2900], 50.00th=[ 6325], 60.00th=[ 9110], | 70.00th=[12256], 80.00th=[16319], 90.00th=[22676], 95.00th=[28705], | 99.00th=[40109], 99.50th=[44303], 99.90th=[51119], 99.95th=[76022], | 99.99th=[76022] bw ( KiB/s): min= 95, max= 288, per=24.89%, avg=184.18, stdev=44.76, samples=67 iops : min= 23, max= 72, avg=45.70, stdev=11.25, samples=67 lat (usec) : 100=1.17%, 250=2.21%, 500=0.26%, 750=3.32%, 1000=8.50% lat (msec) : 2=6.18%, 4=4.88%, 10=27.47%, 20=31.61%, 50=14.10% lat (msec) : 100=0.26%, 500=0.03% cpu : usr=0.15%, sys=13.15%, ctx=5133, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=707KiB/s (724kB/s), 177KiB/s-179KiB/s (181kB/s-184kB/s), io=23.4MiB (24.6MB), run=33581-33946msec WRITE: bw=741KiB/s (758kB/s), 185KiB/s-189KiB/s (189kB/s-194kB/s), io=24.6MiB (25.7MB), run=33581-33946msec rand-rw: (groupid=0, jobs=1): err= 0: pid=9717: Tue Jun 17 13:44:35 2025 read: IOPS=29, BW=118KiB/s (121kB/s)(5992KiB/50863msec) clat (msec): min=2, max=273, avg=12.85, stdev=13.06 lat (msec): min=2, max=273, avg=12.85, stdev=13.06 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 5], 10.00th=[ 5], 20.00th=[ 6], | 30.00th=[ 7], 40.00th=[ 8], 50.00th=[ 9], 60.00th=[ 11], | 70.00th=[ 13], 80.00th=[ 17], 90.00th=[ 27], 95.00th=[ 37], | 99.00th=[ 59], 99.50th=[ 67], 99.90th=[ 107], 99.95th=[ 275], | 99.99th=[ 275] bw ( KiB/s): min= 47, max= 264, per=25.09%, avg=116.93, stdev=53.66, samples=101 iops : min= 11, max= 66, avg=28.92, stdev=13.38, samples=101 write: IOPS=30, BW=124KiB/s (127kB/s)(6296KiB/50863msec) clat (usec): min=5419, max=87602, avg=20013.81, stdev=11556.40 lat (usec): min=5420, max=87603, avg=20015.92, stdev=11555.82 clat percentiles (usec): | 1.00th=[ 7177], 5.00th=[ 8586], 10.00th=[ 9765], 20.00th=[11469], | 30.00th=[12911], 40.00th=[14353], 50.00th=[16319], 60.00th=[18744], | 70.00th=[22414], 80.00th=[27132], 90.00th=[34866], 95.00th=[44303], | 99.00th=[62129], 99.50th=[72877], 99.90th=[82314], 99.95th=[87557], | 99.99th=[87557] bw ( KiB/s): min= 32, max= 272, per=25.13%, avg=122.64, stdev=56.97, samples=101 iops : min= 8, max= 68, avg=30.35, stdev=14.24, samples=101 lat (msec) : 4=1.86%, 10=33.20%, 20=39.19%, 50=23.37%, 100=2.31% lat (msec) : 250=0.03%, 500=0.03% cpu : usr=0.12%, sys=6.23%, ctx=3384, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9718: Tue Jun 17 13:44:35 2025 read: IOPS=28, BW=115KiB/s (118kB/s)(5940KiB/51485msec) clat (msec): min=2, max=223, avg=13.03, stdev=12.41 lat (msec): min=2, max=223, avg=13.03, stdev=12.41 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 5], 10.00th=[ 5], 20.00th=[ 6], | 30.00th=[ 7], 40.00th=[ 8], 50.00th=[ 9], 60.00th=[ 11], | 70.00th=[ 13], 80.00th=[ 18], 90.00th=[ 28], 95.00th=[ 38], | 99.00th=[ 56], 99.50th=[ 63], 99.90th=[ 104], 99.95th=[ 224], | 99.99th=[ 224] bw ( KiB/s): min= 16, max= 304, per=24.56%, avg=114.45, stdev=59.13, samples=102 iops : min= 4, max= 76, avg=28.34, stdev=14.82, samples=102 write: IOPS=30, BW=123KiB/s (126kB/s)(6348KiB/51485msec) clat (msec): min=5, max=118, avg=20.15, stdev=12.45 lat (msec): min=5, max=118, avg=20.15, stdev=12.45 clat percentiles (msec): | 1.00th=[ 7], 5.00th=[ 9], 10.00th=[ 10], 20.00th=[ 12], | 30.00th=[ 13], 40.00th=[ 15], 50.00th=[ 17], 60.00th=[ 19], | 70.00th=[ 22], 80.00th=[ 27], 90.00th=[ 37], 95.00th=[ 46], | 99.00th=[ 67], 99.50th=[ 75], 99.90th=[ 92], 99.95th=[ 118], | 99.99th=[ 118] bw ( KiB/s): min= 47, max= 248, per=24.87%, avg=121.34, stdev=55.67, samples=102 iops : min= 11, max= 62, avg=30.07, stdev=13.93, samples=102 lat (msec) : 4=1.79%, 10=32.29%, 20=39.62%, 50=23.73%, 100=2.47% lat (msec) : 250=0.10% cpu : usr=0.10%, sys=6.15%, ctx=3391, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9720: Tue Jun 17 13:44:35 2025 read: IOPS=29, BW=119KiB/s (122kB/s)(6028KiB/50636msec) clat (msec): min=2, max=104, avg=12.70, stdev=11.33 lat (msec): min=2, max=104, avg=12.70, stdev=11.33 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 5], 10.00th=[ 5], 20.00th=[ 6], | 30.00th=[ 7], 40.00th=[ 8], 50.00th=[ 9], 60.00th=[ 10], | 70.00th=[ 12], 80.00th=[ 17], 90.00th=[ 27], 95.00th=[ 38], | 99.00th=[ 57], 99.50th=[ 66], 99.90th=[ 92], 99.95th=[ 105], | 99.99th=[ 105] bw ( KiB/s): min= 39, max= 288, per=25.44%, avg=118.54, stdev=57.94, samples=101 iops : min= 9, max= 72, avg=29.39, stdev=14.49, samples=101 write: IOPS=30, BW=124KiB/s (127kB/s)(6260KiB/50636msec) clat (msec): min=4, max=273, avg=20.04, stdev=13.60 lat (msec): min=4, max=273, avg=20.05, stdev=13.60 clat percentiles (msec): | 1.00th=[ 7], 5.00th=[ 9], 10.00th=[ 10], 20.00th=[ 12], | 30.00th=[ 13], 40.00th=[ 15], 50.00th=[ 16], 60.00th=[ 19], | 70.00th=[ 22], 80.00th=[ 27], 90.00th=[ 36], 95.00th=[ 44], | 99.00th=[ 61], 99.50th=[ 77], 99.90th=[ 118], 99.95th=[ 275], | 99.99th=[ 275] bw ( KiB/s): min= 39, max= 256, per=25.18%, avg=122.90, stdev=56.69, samples=101 iops : min= 9, max= 64, avg=30.48, stdev=14.18, samples=101 lat (msec) : 4=1.33%, 10=34.02%, 20=39.68%, 50=22.53%, 100=2.31% lat (msec) : 250=0.10%, 500=0.03% cpu : usr=0.05%, sys=6.30%, ctx=3390, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9721: Tue Jun 17 13:44:35 2025 read: IOPS=29, BW=119KiB/s (121kB/s)(6052KiB/51040msec) clat (msec): min=3, max=108, avg=12.93, stdev=11.17 lat (msec): min=3, max=108, avg=12.93, stdev=11.17 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 5], 10.00th=[ 6], 20.00th=[ 6], | 30.00th=[ 7], 40.00th=[ 8], 50.00th=[ 10], 60.00th=[ 11], | 70.00th=[ 13], 80.00th=[ 18], 90.00th=[ 27], 95.00th=[ 37], | 99.00th=[ 59], 99.50th=[ 67], 99.90th=[ 93], 99.95th=[ 109], | 99.99th=[ 109] bw ( KiB/s): min= 31, max= 288, per=25.37%, avg=118.21, stdev=57.68, samples=102 iops : min= 7, max= 72, avg=29.27, stdev=14.39, samples=102 write: IOPS=30, BW=122KiB/s (125kB/s)(6236KiB/51040msec) clat (msec): min=5, max=224, avg=20.08, stdev=12.82 lat (msec): min=5, max=224, avg=20.08, stdev=12.82 clat percentiles (msec): | 1.00th=[ 7], 5.00th=[ 9], 10.00th=[ 10], 20.00th=[ 12], | 30.00th=[ 13], 40.00th=[ 15], 50.00th=[ 17], 60.00th=[ 19], | 70.00th=[ 22], 80.00th=[ 27], 90.00th=[ 37], 95.00th=[ 44], | 99.00th=[ 61], 99.50th=[ 68], 99.90th=[ 103], 99.95th=[ 226], | 99.99th=[ 226] bw ( KiB/s): min= 40, max= 256, per=24.93%, avg=121.66, stdev=54.28, samples=102 iops : min= 10, max= 64, avg=30.14, stdev=13.56, samples=102 lat (msec) : 4=1.56%, 10=32.39%, 20=40.76%, 50=22.82%, 100=2.38% lat (msec) : 250=0.10% cpu : usr=0.07%, sys=6.26%, ctx=3383, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=466KiB/s (478kB/s), 115KiB/s-119KiB/s (118kB/s-122kB/s), io=23.4MiB (24.6MB), run=50636-51485msec WRITE: bw=488KiB/s (500kB/s), 122KiB/s-124KiB/s (125kB/s-127kB/s), io=24.6MiB (25.7MB), run=50636-51485msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=9780: Tue Jun 17 13:44:46 2025 read: IOPS=38, BW=609KiB/s (624kB/s)(5920KiB/9714msec) clat (usec): min=66, max=56427, avg=14322.54, stdev=10247.73 lat (usec): min=66, max=56428, avg=14323.72, stdev=10247.85 clat percentiles (usec): | 1.00th=[ 95], 5.00th=[ 108], 10.00th=[ 125], 20.00th=[ 6849], | 30.00th=[ 8717], 40.00th=[10552], 50.00th=[12649], 60.00th=[15139], | 70.00th=[17171], 80.00th=[20841], 90.00th=[27919], 95.00th=[33817], | 99.00th=[49546], 99.50th=[55837], 99.90th=[56361], 99.95th=[56361], | 99.99th=[56361] bw ( KiB/s): min= 352, max= 864, per=24.51%, avg=586.53, stdev=163.98, samples=19 iops : min= 22, max= 54, avg=36.16, stdev=10.22, samples=19 write: IOPS=40, BW=656KiB/s (671kB/s)(6368KiB/9714msec) clat (usec): min=769, max=85968, avg=10932.34, stdev=12472.47 lat (usec): min=771, max=85970, avg=10934.02, stdev=12472.50 clat percentiles (usec): | 1.00th=[ 922], 5.00th=[ 1045], 10.00th=[ 1188], 20.00th=[ 1532], | 30.00th=[ 2343], 40.00th=[ 3884], 50.00th=[ 6456], 60.00th=[10290], | 70.00th=[13304], 80.00th=[17433], 90.00th=[26608], 95.00th=[35914], | 99.00th=[60556], 99.50th=[82314], 99.90th=[85459], 99.95th=[85459], | 99.99th=[85459] bw ( KiB/s): min= 352, max= 1024, per=25.43%, avg=640.37, stdev=216.94, samples=19 iops : min= 22, max= 64, avg=39.53, stdev=13.43, samples=19 lat (usec) : 100=1.17%, 250=4.17%, 500=0.26%, 1000=1.56% lat (msec) : 2=12.24%, 4=7.68%, 10=21.48%, 20=33.20%, 50=16.93% lat (msec) : 100=1.30% cpu : usr=0.08%, sys=12.66%, ctx=1322, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9781: Tue Jun 17 13:44:46 2025 read: IOPS=36, BW=579KiB/s (593kB/s)(5744KiB/9913msec) clat (usec): min=63, max=101213, avg=15045.01, stdev=12699.18 lat (usec): min=64, max=101214, avg=15048.05, stdev=12697.68 clat percentiles (usec): | 1.00th=[ 95], 5.00th=[ 112], 10.00th=[ 135], 20.00th=[ 6259], | 30.00th=[ 8455], 40.00th=[ 10683], 50.00th=[ 12911], 60.00th=[ 15139], | 70.00th=[ 18744], 80.00th=[ 22676], 90.00th=[ 29230], 95.00th=[ 34341], | 99.00th=[ 58459], 99.50th=[ 96994], 99.90th=[101188], 99.95th=[101188], | 99.99th=[101188] bw ( KiB/s): min= 192, max= 928, per=23.34%, avg=558.47, stdev=171.80, samples=19 iops : min= 12, max= 58, avg=34.63, stdev=10.60, samples=19 write: IOPS=41, BW=660KiB/s (676kB/s)(6544KiB/9913msec) clat (usec): min=730, max=61198, avg=10629.20, stdev=11023.44 lat (usec): min=731, max=61200, avg=10630.76, stdev=11023.48 clat percentiles (usec): | 1.00th=[ 914], 5.00th=[ 1074], 10.00th=[ 1188], 20.00th=[ 1467], | 30.00th=[ 2147], 40.00th=[ 3720], 50.00th=[ 6849], 60.00th=[10159], | 70.00th=[14091], 80.00th=[18220], 90.00th=[26608], 95.00th=[32900], | 99.00th=[46400], 99.50th=[49546], 99.90th=[61080], 99.95th=[61080], | 99.99th=[61080] bw ( KiB/s): min= 448, max= 1184, per=24.86%, avg=625.89, stdev=201.46, samples=19 iops : min= 28, max= 74, avg=38.84, stdev=12.66, samples=19 lat (usec) : 100=0.78%, 250=5.34%, 500=0.13%, 750=0.52%, 1000=0.91% lat (msec) : 2=14.45%, 4=6.51%, 10=20.57%, 20=29.82%, 50=19.79% lat (msec) : 100=1.04%, 250=0.13% cpu : usr=0.15%, sys=12.42%, ctx=1316, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9782: Tue Jun 17 13:44:46 2025 read: IOPS=39, BW=625KiB/s (640kB/s)(6256KiB/10007msec) clat (usec): min=68, max=82346, avg=14336.06, stdev=11395.56 lat (usec): min=69, max=82347, avg=14337.21, stdev=11395.63 clat percentiles (usec): | 1.00th=[ 84], 5.00th=[ 108], 10.00th=[ 125], 20.00th=[ 5145], | 30.00th=[ 7963], 40.00th=[10552], 50.00th=[12911], 60.00th=[15533], | 70.00th=[18220], 80.00th=[22152], 90.00th=[27657], 95.00th=[36439], | 99.00th=[47973], 99.50th=[59507], 99.90th=[82314], 99.95th=[82314], | 99.99th=[82314] bw ( KiB/s): min= 287, max= 1248, per=26.11%, avg=624.75, stdev=272.78, samples=20 iops : min= 17, max= 78, avg=38.55, stdev=17.25, samples=20 write: IOPS=37, BW=603KiB/s (617kB/s)(6032KiB/10007msec) clat (usec): min=869, max=89778, avg=11402.24, stdev=12619.05 lat (usec): min=871, max=89779, avg=11403.79, stdev=12619.07 clat percentiles (usec): | 1.00th=[ 1037], 5.00th=[ 1139], 10.00th=[ 1237], 20.00th=[ 1663], | 30.00th=[ 2540], 40.00th=[ 3687], 50.00th=[ 7242], 60.00th=[10421], | 70.00th=[13304], 80.00th=[19792], 90.00th=[27919], 95.00th=[39584], | 99.00th=[53216], 99.50th=[71828], 99.90th=[89654], 99.95th=[89654], | 99.99th=[89654] bw ( KiB/s): min= 287, max= 1184, per=23.73%, avg=597.45, stdev=231.37, samples=20 iops : min= 17, max= 74, avg=36.85, stdev=14.61, samples=20 lat (usec) : 100=2.08%, 250=5.86%, 500=0.26%, 750=0.13%, 1000=0.26% lat (msec) : 2=12.11%, 4=8.59%, 10=18.62%, 20=29.30%, 50=21.74% lat (msec) : 100=1.04% cpu : usr=0.23%, sys=12.36%, ctx=1356, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9783: Tue Jun 17 13:44:46 2025 read: IOPS=38, BW=608KiB/s (623kB/s)(6032KiB/9914msec) clat (usec): min=87, max=72122, avg=14130.85, stdev=11051.64 lat (usec): min=87, max=72122, avg=14134.51, stdev=11052.35 clat percentiles (usec): | 1.00th=[ 94], 5.00th=[ 113], 10.00th=[ 135], 20.00th=[ 6783], | 30.00th=[ 8586], 40.00th=[10552], 50.00th=[12387], 60.00th=[13960], | 70.00th=[16909], 80.00th=[20317], 90.00th=[27132], 95.00th=[33817], | 99.00th=[53740], 99.50th=[68682], 99.90th=[71828], 99.95th=[71828], | 99.99th=[71828] bw ( KiB/s): min= 255, max= 990, per=24.39%, avg=583.63, stdev=189.93, samples=19 iops : min= 15, max= 61, avg=36.00, stdev=12.06, samples=19 write: IOPS=39, BW=631KiB/s (646kB/s)(6256KiB/9914msec) clat (usec): min=660, max=81656, avg=11356.88, stdev=12748.81 lat (usec): min=662, max=81658, avg=11358.50, stdev=12748.83 clat percentiles (usec): | 1.00th=[ 947], 5.00th=[ 1123], 10.00th=[ 1188], 20.00th=[ 1516], | 30.00th=[ 2212], 40.00th=[ 3851], 50.00th=[ 6259], 60.00th=[ 9372], | 70.00th=[12911], 80.00th=[19792], 90.00th=[31589], 95.00th=[38011], | 99.00th=[51643], 99.50th=[70779], 99.90th=[81265], 99.95th=[81265], | 99.99th=[81265] bw ( KiB/s): min= 351, max= 1120, per=24.45%, avg=615.53, stdev=214.22, samples=19 iops : min= 21, max= 70, avg=38.00, stdev=13.58, samples=19 lat (usec) : 100=1.17%, 250=4.69%, 500=0.39%, 750=0.52%, 1000=0.65% lat (msec) : 2=13.15%, 4=6.90%, 10=22.01%, 20=30.21%, 50=18.75% lat (msec) : 100=1.56% cpu : usr=0.18%, sys=12.24%, ctx=1291, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2394KiB/s (2451kB/s), 579KiB/s-625KiB/s (593kB/s-640kB/s), io=23.4MiB (24.5MB), run=9714-10007msec WRITE: bw=2518KiB/s (2579kB/s), 603KiB/s-660KiB/s (617kB/s-676kB/s), io=24.6MiB (25.8MB), run=9714-10007msec rand-rw: (groupid=0, jobs=1): err= 0: pid=9776: Tue Jun 17 13:44:50 2025 read: IOPS=27, BW=436KiB/s (446kB/s)(5920KiB/13584msec) clat (usec): min=2984, max=95758, avg=14263.71, stdev=14434.49 lat (usec): min=3134, max=95759, avg=14265.60, stdev=14434.64 clat percentiles (usec): | 1.00th=[ 3425], 5.00th=[ 4080], 10.00th=[ 4490], 20.00th=[ 5145], | 30.00th=[ 5997], 40.00th=[ 6915], 50.00th=[ 8455], 60.00th=[10159], | 70.00th=[13698], 80.00th=[21103], 90.00th=[32637], 95.00th=[43779], | 99.00th=[78119], 99.50th=[89654], 99.90th=[95945], 99.95th=[95945], | 99.99th=[95945] bw ( KiB/s): min= 159, max= 1083, per=24.39%, avg=429.78, stdev=273.04, samples=27 iops : min= 9, max= 67, avg=26.22, stdev=17.05, samples=27 write: IOPS=29, BW=469KiB/s (480kB/s)(6368KiB/13584msec) clat (usec): min=5545, max=82588, avg=20671.68, stdev=14657.71 lat (usec): min=5546, max=82589, avg=20673.64, stdev=14657.66 clat percentiles (usec): | 1.00th=[ 6915], 5.00th=[ 8029], 10.00th=[ 8979], 20.00th=[10290], | 30.00th=[11076], 40.00th=[12125], 50.00th=[14091], 60.00th=[17433], | 70.00th=[24249], 80.00th=[29754], 90.00th=[43779], 95.00th=[52167], | 99.00th=[72877], 99.50th=[76022], 99.90th=[82314], 99.95th=[82314], | 99.99th=[82314] bw ( KiB/s): min= 127, max= 1088, per=25.10%, avg=465.33, stdev=307.12, samples=27 iops : min= 7, max= 68, avg=28.44, stdev=19.23, samples=27 lat (msec) : 4=2.08%, 10=35.68%, 20=33.20%, 50=24.35%, 100=4.69% cpu : usr=0.05%, sys=6.23%, ctx=884, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9777: Tue Jun 17 13:44:50 2025 read: IOPS=26, BW=424KiB/s (434kB/s)(5744KiB/13561msec) clat (usec): min=3003, max=74045, avg=13061.04, stdev=12354.87 lat (usec): min=3005, max=74045, avg=13062.31, stdev=12354.79 clat percentiles (usec): | 1.00th=[ 3326], 5.00th=[ 3982], 10.00th=[ 4621], 20.00th=[ 5145], | 30.00th=[ 5866], 40.00th=[ 6521], 50.00th=[ 7701], 60.00th=[ 9503], | 70.00th=[12911], 80.00th=[18482], 90.00th=[31327], 95.00th=[41157], | 99.00th=[58983], 99.50th=[72877], 99.90th=[73925], 99.95th=[73925], | 99.99th=[73925] bw ( KiB/s): min= 160, max= 1152, per=23.85%, avg=420.22, stdev=292.12, samples=27 iops : min= 10, max= 72, avg=26.04, stdev=18.26, samples=27 write: IOPS=30, BW=483KiB/s (494kB/s)(6544KiB/13561msec) clat (msec): min=6, max=111, avg=21.52, stdev=16.23 lat (msec): min=6, max=111, avg=21.52, stdev=16.23 clat percentiles (msec): | 1.00th=[ 7], 5.00th=[ 9], 10.00th=[ 9], 20.00th=[ 11], | 30.00th=[ 12], 40.00th=[ 14], 50.00th=[ 16], 60.00th=[ 19], | 70.00th=[ 23], 80.00th=[ 31], 90.00th=[ 48], 95.00th=[ 56], | 99.00th=[ 73], 99.50th=[ 99], 99.90th=[ 112], 99.95th=[ 112], | 99.99th=[ 112] bw ( KiB/s): min= 192, max= 1083, per=25.92%, avg=480.56, stdev=281.37, samples=27 iops : min= 12, max= 67, avg=29.81, stdev=17.54, samples=27 lat (msec) : 4=2.34%, 10=37.11%, 20=32.81%, 50=22.66%, 100=4.82% lat (msec) : 250=0.26% cpu : usr=0.07%, sys=6.27%, ctx=883, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9778: Tue Jun 17 13:44:50 2025 read: IOPS=28, BW=463KiB/s (474kB/s)(6256KiB/13511msec) clat (usec): min=3453, max=71041, avg=13098.00, stdev=11954.84 lat (usec): min=3453, max=71041, avg=13099.11, stdev=11954.83 clat percentiles (usec): | 1.00th=[ 3720], 5.00th=[ 4228], 10.00th=[ 4555], 20.00th=[ 5211], | 30.00th=[ 6063], 40.00th=[ 6783], 50.00th=[ 7898], 60.00th=[ 9634], | 70.00th=[12649], 80.00th=[19530], 90.00th=[30540], 95.00th=[39584], | 99.00th=[53740], 99.50th=[68682], 99.90th=[70779], 99.95th=[70779], | 99.99th=[70779] bw ( KiB/s): min= 128, max= 1312, per=26.26%, avg=462.78, stdev=328.66, samples=27 iops : min= 8, max= 82, avg=28.70, stdev=20.55, samples=27 write: IOPS=27, BW=446KiB/s (457kB/s)(6032KiB/13511msec) clat (msec): min=5, max=113, avg=22.18, stdev=16.26 lat (msec): min=5, max=113, avg=22.18, stdev=16.26 clat percentiles (msec): | 1.00th=[ 6], 5.00th=[ 8], 10.00th=[ 10], 20.00th=[ 11], | 30.00th=[ 12], 40.00th=[ 14], 50.00th=[ 16], 60.00th=[ 20], | 70.00th=[ 26], 80.00th=[ 33], 90.00th=[ 46], 95.00th=[ 56], | 99.00th=[ 78], 99.50th=[ 96], 99.90th=[ 114], 99.95th=[ 114], | 99.99th=[ 114] bw ( KiB/s): min= 160, max= 988, per=24.00%, avg=445.00, stdev=261.04, samples=27 iops : min= 10, max= 61, avg=27.59, stdev=16.27, samples=27 lat (msec) : 4=1.56%, 10=37.37%, 20=31.77%, 50=24.61%, 100=4.56% lat (msec) : 250=0.13% cpu : usr=0.00%, sys=6.31%, ctx=873, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9779: Tue Jun 17 13:44:50 2025 read: IOPS=27, BW=444KiB/s (455kB/s)(6032KiB/13587msec) clat (usec): min=3224, max=86837, avg=13768.51, stdev=13727.26 lat (usec): min=3225, max=86839, avg=13769.65, stdev=13727.37 clat percentiles (usec): | 1.00th=[ 3556], 5.00th=[ 4228], 10.00th=[ 4621], 20.00th=[ 5342], | 30.00th=[ 6259], 40.00th=[ 6980], 50.00th=[ 8225], 60.00th=[10028], | 70.00th=[12780], 80.00th=[17695], 90.00th=[32900], 95.00th=[44827], | 99.00th=[67634], 99.50th=[77071], 99.90th=[86508], 99.95th=[86508], | 99.99th=[86508] bw ( KiB/s): min= 160, max= 960, per=24.84%, avg=437.74, stdev=257.60, samples=27 iops : min= 10, max= 60, avg=27.07, stdev=16.04, samples=27 write: IOPS=28, BW=460KiB/s (471kB/s)(6256KiB/13587msec) clat (usec): min=4524, max=82711, avg=21348.49, stdev=15098.68 lat (usec): min=4526, max=82713, avg=21350.69, stdev=15098.55 clat percentiles (usec): | 1.00th=[ 5997], 5.00th=[ 8225], 10.00th=[ 9241], 20.00th=[10290], | 30.00th=[11600], 40.00th=[13042], 50.00th=[15008], 60.00th=[17957], | 70.00th=[23200], 80.00th=[31589], 90.00th=[45351], 95.00th=[52691], | 99.00th=[74974], 99.50th=[74974], 99.90th=[82314], 99.95th=[82314], | 99.99th=[82314] bw ( KiB/s): min= 160, max= 1053, per=24.69%, avg=457.74, stdev=286.60, samples=27 iops : min= 10, max= 65, avg=28.33, stdev=17.79, samples=27 lat (msec) : 4=1.56%, 10=36.98%, 20=34.77%, 50=21.61%, 100=5.08% cpu : usr=0.15%, sys=6.17%, ctx=876, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1763KiB/s (1805kB/s), 424KiB/s-463KiB/s (434kB/s-474kB/s), io=23.4MiB (24.5MB), run=13511-13587msec WRITE: bw=1855KiB/s (1899kB/s), 446KiB/s-483KiB/s (457kB/s-494kB/s), io=24.6MiB (25.8MB), run=13511-13587msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=9803: Tue Jun 17 13:44:55 2025 read: IOPS=1, BW=1827KiB/s (1871kB/s)(7168KiB/3924msec) clat (usec): min=652, max=1276.7k, avg=519351.70, stdev=605226.70 lat (usec): min=654, max=1276.7k, avg=519353.36, stdev=605226.94 clat percentiles (usec): | 1.00th=[ 652], 5.00th=[ 652], 10.00th=[ 652], | 20.00th=[ 693], 30.00th=[ 60556], 40.00th=[ 60556], | 50.00th=[ 92799], 60.00th=[1019216], 70.00th=[1019216], | 80.00th=[1182794], 90.00th=[1283458], 95.00th=[1283458], | 99.00th=[1283458], 99.50th=[1283458], 99.90th=[1283458], | 99.95th=[1283458], 99.99th=[1283458] bw ( KiB/s): min= 2043, max= 2048, per=39.19%, avg=2045.50, stdev= 3.54, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 write: IOPS=1, BW=1305KiB/s (1336kB/s)(5120KiB/3924msec) clat (usec): min=40643, max=80920, avg=51642.59, stdev=16763.92 lat (usec): min=40726, max=81006, avg=51720.90, stdev=16767.07 clat percentiles (usec): | 1.00th=[40633], 5.00th=[40633], 10.00th=[40633], 20.00th=[40633], | 30.00th=[41681], 40.00th=[41681], 50.00th=[45351], 60.00th=[45351], | 70.00th=[50070], 80.00th=[50070], 90.00th=[81265], 95.00th=[81265], | 99.00th=[81265], 99.50th=[81265], 99.90th=[81265], 99.95th=[81265], | 99.99th=[81265] lat (usec) : 750=16.67% lat (msec) : 50=33.33%, 100=25.00% cpu : usr=0.23%, sys=13.15%, ctx=998, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9805: Tue Jun 17 13:44:55 2025 read: IOPS=0, BW=542KiB/s (555kB/s)(2048KiB/3779msec) clat (usec): min=64780, max=70226, avg=67503.51, stdev=3850.71 lat (usec): min=64782, max=70228, avg=67505.30, stdev=3851.01 clat percentiles (usec): | 1.00th=[64750], 5.00th=[64750], 10.00th=[64750], 20.00th=[64750], | 30.00th=[64750], 40.00th=[64750], 50.00th=[64750], 60.00th=[69731], | 70.00th=[69731], 80.00th=[69731], 90.00th=[69731], 95.00th=[69731], | 99.00th=[69731], 99.50th=[69731], 99.90th=[69731], 99.95th=[69731], | 99.99th=[69731] bw ( KiB/s): min= 1868, max= 2048, per=37.52%, avg=1958.00, stdev=127.28, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 write: IOPS=2, BW=2710KiB/s (2775kB/s)(10.0MiB/3779msec) clat (msec): min=43, max=2516, avg=357.98, stdev=765.55 lat (msec): min=43, max=2516, avg=358.05, stdev=765.55 clat percentiles (msec): | 1.00th=[ 44], 5.00th=[ 44], 10.00th=[ 44], 20.00th=[ 46], | 30.00th=[ 48], 40.00th=[ 66], 50.00th=[ 83], 60.00th=[ 125], | 70.00th=[ 131], 80.00th=[ 131], 90.00th=[ 397], 95.00th=[ 2500], | 99.00th=[ 2500], 99.50th=[ 2500], 99.90th=[ 2500], 99.95th=[ 2500], | 99.99th=[ 2500] bw ( KiB/s): min= 5605, max=10240, per=100.00%, avg=7922.50, stdev=3277.44, samples=2 iops : min= 5, max= 10, avg= 7.50, stdev= 3.54, samples=2 lat (msec) : 50=25.00%, 100=33.33%, 250=25.00%, 500=8.33% cpu : usr=0.00%, sys=7.73%, ctx=165, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9806: Tue Jun 17 13:44:55 2025 read: IOPS=1, BW=1761KiB/s (1804kB/s)(6144KiB/3488msec) clat (usec): min=515, max=256800, avg=76967.04, stdev=94649.93 lat (usec): min=516, max=256801, avg=76968.51, stdev=94650.10 clat percentiles (usec): | 1.00th=[ 515], 5.00th=[ 515], 10.00th=[ 515], 20.00th=[ 570], | 30.00th=[ 570], 40.00th=[ 51119], 50.00th=[ 51119], 60.00th=[ 69731], | 70.00th=[ 83362], 80.00th=[ 83362], 90.00th=[256902], 95.00th=[256902], | 99.00th=[256902], 99.50th=[256902], 99.90th=[256902], 99.95th=[256902], | 99.99th=[256902] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=1, BW=1761KiB/s (1804kB/s)(6144KiB/3488msec) clat (msec): min=40, max=2596, avg=487.80, stdev=1033.22 lat (msec): min=40, max=2596, avg=487.87, stdev=1033.23 clat percentiles (msec): | 1.00th=[ 41], 5.00th=[ 41], 10.00th=[ 41], 20.00th=[ 47], | 30.00th=[ 47], 40.00th=[ 71], 50.00th=[ 71], 60.00th=[ 81], | 70.00th=[ 92], 80.00th=[ 92], 90.00th=[ 2601], 95.00th=[ 2601], | 99.00th=[ 2601], 99.50th=[ 2601], 99.90th=[ 2601], 99.95th=[ 2601], | 99.99th=[ 2601] bw ( KiB/s): min= 4096, max= 4096, per=56.06%, avg=4096.00, stdev= 0.00, samples=2 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=2 lat (usec) : 750=16.67% lat (msec) : 50=16.67%, 100=50.00%, 500=8.33% cpu : usr=0.00%, sys=6.57%, ctx=321, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9807: Tue Jun 17 13:44:55 2025 read: IOPS=1, BW=1472KiB/s (1507kB/s)(5120KiB/3478msec) clat (usec): min=448, max=157265, avg=47186.48, stdev=69713.02 lat (usec): min=449, max=157266, avg=47187.65, stdev=69713.35 clat percentiles (usec): | 1.00th=[ 449], 5.00th=[ 449], 10.00th=[ 449], 20.00th=[ 449], | 30.00th=[ 717], 40.00th=[ 717], 50.00th=[ 1090], 60.00th=[ 1090], | 70.00th=[ 76022], 80.00th=[ 76022], 90.00th=[156238], 95.00th=[156238], | 99.00th=[156238], 99.50th=[156238], 99.90th=[156238], 99.95th=[156238], | 99.99th=[156238] write: IOPS=2, BW=2061KiB/s (2110kB/s)(7168KiB/3478msec) clat (msec): min=42, max=2599, avg=450.51, stdev=949.45 lat (msec): min=42, max=2599, avg=450.58, stdev=949.46 clat percentiles (msec): | 1.00th=[ 43], 5.00th=[ 43], 10.00th=[ 43], 20.00th=[ 45], | 30.00th=[ 78], 40.00th=[ 78], 50.00th=[ 90], 60.00th=[ 90], | 70.00th=[ 90], 80.00th=[ 211], 90.00th=[ 2601], 95.00th=[ 2601], | 99.00th=[ 2601], 99.50th=[ 2601], 99.90th=[ 2601], 99.95th=[ 2601], | 99.99th=[ 2601] bw ( KiB/s): min= 2048, max= 4096, per=42.05%, avg=3072.00, stdev=1448.15, samples=2 iops : min= 2, max= 4, avg= 3.00, stdev= 1.41, samples=2 lat (usec) : 500=8.33%, 750=8.33% lat (msec) : 2=8.33%, 50=16.67%, 100=33.33%, 250=16.67% cpu : usr=0.00%, sys=5.78%, ctx=161, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=5219KiB/s (5344kB/s), 542KiB/s-1827KiB/s (555kB/s-1871kB/s), io=20.0MiB (20.0MB), run=3478-3924msec WRITE: bw=7307KiB/s (7482kB/s), 1305KiB/s-2710KiB/s (1336kB/s-2775kB/s), io=28.0MiB (29.4MB), run=3478-3924msec rand-rw: (groupid=0, jobs=1): err= 0: pid=9800: Tue Jun 17 13:44:55 2025 read: IOPS=1, BW=1682KiB/s (1723kB/s)(7168KiB/4261msec) clat (msec): min=26, max=1674, avg=517.92, stdev=673.06 lat (msec): min=26, max=1674, avg=517.92, stdev=673.06 clat percentiles (msec): | 1.00th=[ 27], 5.00th=[ 27], 10.00th=[ 27], 20.00th=[ 28], | 30.00th=[ 43], 40.00th=[ 43], 50.00th=[ 56], 60.00th=[ 609], | 70.00th=[ 609], 80.00th=[ 1183], 90.00th=[ 1670], 95.00th=[ 1670], | 99.00th=[ 1670], 99.50th=[ 1670], 99.90th=[ 1670], 99.95th=[ 1670], | 99.99th=[ 1670] bw ( KiB/s): min= 2039, max= 5296, per=65.04%, avg=3126.00, stdev=1879.28, samples=3 iops : min= 1, max= 5, avg= 2.33, stdev= 2.31, samples=3 write: IOPS=1, BW=1202KiB/s (1230kB/s)(5120KiB/4261msec) clat (msec): min=71, max=155, avg=110.98, stdev=38.62 lat (msec): min=72, max=155, avg=111.06, stdev=38.61 clat percentiles (msec): | 1.00th=[ 72], 5.00th=[ 72], 10.00th=[ 72], 20.00th=[ 72], | 30.00th=[ 86], 40.00th=[ 86], 50.00th=[ 92], 60.00th=[ 92], | 70.00th=[ 150], 80.00th=[ 150], 90.00th=[ 157], 95.00th=[ 157], | 99.00th=[ 157], 99.50th=[ 157], 99.90th=[ 157], 99.95th=[ 157], | 99.99th=[ 157] bw ( KiB/s): min= 5296, max= 5296, per=78.72%, avg=5296.00, stdev= 0.00, samples=1 iops : min= 5, max= 5, avg= 5.00, stdev= 0.00, samples=1 lat (msec) : 50=25.00%, 100=33.33%, 250=16.67%, 750=8.33% cpu : usr=0.00%, sys=3.54%, ctx=76, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9801: Tue Jun 17 13:44:55 2025 read: IOPS=0, BW=552KiB/s (565kB/s)(2048KiB/3710msec) clat (usec): min=48829, max=57186, avg=53007.85, stdev=5909.07 lat (usec): min=48832, max=57187, avg=53009.90, stdev=5908.17 clat percentiles (usec): | 1.00th=[49021], 5.00th=[49021], 10.00th=[49021], 20.00th=[49021], | 30.00th=[49021], 40.00th=[49021], 50.00th=[49021], 60.00th=[57410], | 70.00th=[57410], 80.00th=[57410], 90.00th=[57410], 95.00th=[57410], | 99.00th=[57410], 99.50th=[57410], 99.90th=[57410], 99.95th=[57410], | 99.99th=[57410] bw ( KiB/s): min= 2043, max= 2048, per=42.56%, avg=2045.50, stdev= 3.54, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 write: IOPS=2, BW=2760KiB/s (2826kB/s)(10.0MiB/3710msec) clat (msec): min=40, max=2511, avg=358.49, stdev=762.41 lat (msec): min=40, max=2511, avg=358.57, stdev=762.42 clat percentiles (msec): | 1.00th=[ 41], 5.00th=[ 41], 10.00th=[ 41], 20.00th=[ 44], | 30.00th=[ 67], 40.00th=[ 67], 50.00th=[ 94], 60.00th=[ 103], | 70.00th=[ 157], 80.00th=[ 157], 90.00th=[ 372], 95.00th=[ 2500], | 99.00th=[ 2500], 99.50th=[ 2500], 99.90th=[ 2500], 99.95th=[ 2500], | 99.99th=[ 2500] bw ( KiB/s): min= 6144, max=10240, per=100.00%, avg=8192.00, stdev=2896.31, samples=2 iops : min= 6, max= 10, avg= 8.00, stdev= 2.83, samples=2 lat (msec) : 50=25.00%, 100=33.33%, 250=25.00%, 500=8.33% cpu : usr=0.00%, sys=3.64%, ctx=77, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9802: Tue Jun 17 13:44:55 2025 read: IOPS=1, BW=1600KiB/s (1638kB/s)(6144KiB/3841msec) clat (msec): min=53, max=244, avg=120.90, stdev=74.58 lat (msec): min=53, max=244, avg=120.90, stdev=74.58 clat percentiles (msec): | 1.00th=[ 54], 5.00th=[ 54], 10.00th=[ 54], 20.00th=[ 60], | 30.00th=[ 60], 40.00th=[ 69], 50.00th=[ 69], 60.00th=[ 150], | 70.00th=[ 150], 80.00th=[ 150], 90.00th=[ 245], 95.00th=[ 245], | 99.00th=[ 245], 99.50th=[ 245], 99.90th=[ 245], 99.95th=[ 245], | 99.99th=[ 245] bw ( KiB/s): min= 2048, max= 6119, per=84.97%, avg=4083.50, stdev=2878.63, samples=2 iops : min= 2, max= 5, avg= 3.50, stdev= 2.12, samples=2 write: IOPS=1, BW=1600KiB/s (1638kB/s)(6144KiB/3841msec) clat (msec): min=51, max=2562, avg=510.42, stdev=1006.58 lat (msec): min=51, max=2563, avg=510.52, stdev=1006.57 clat percentiles (msec): | 1.00th=[ 52], 5.00th=[ 52], 10.00th=[ 52], 20.00th=[ 64], | 30.00th=[ 64], 40.00th=[ 74], 50.00th=[ 74], 60.00th=[ 148], | 70.00th=[ 163], 80.00th=[ 163], 90.00th=[ 2567], 95.00th=[ 2567], | 99.00th=[ 2567], 99.50th=[ 2567], 99.90th=[ 2567], 99.95th=[ 2567], | 99.99th=[ 2567] bw ( KiB/s): min= 2048, max= 4096, per=50.65%, avg=3407.67, stdev=1177.54, samples=3 iops : min= 2, max= 4, avg= 3.00, stdev= 1.00, samples=3 lat (msec) : 100=50.00%, 250=41.67% cpu : usr=0.00%, sys=3.96%, ctx=78, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9804: Tue Jun 17 13:44:55 2025 read: IOPS=1, BW=1386KiB/s (1420kB/s)(5120KiB/3693msec) clat (msec): min=26, max=413, avg=137.55, stdev=157.41 lat (msec): min=26, max=413, avg=137.55, stdev=157.41 clat percentiles (msec): | 1.00th=[ 27], 5.00th=[ 27], 10.00th=[ 27], 20.00th=[ 27], | 30.00th=[ 66], 40.00th=[ 66], 50.00th=[ 67], 60.00th=[ 67], | 70.00th=[ 115], 80.00th=[ 115], 90.00th=[ 414], 95.00th=[ 414], | 99.00th=[ 414], 99.50th=[ 414], 99.90th=[ 414], 99.95th=[ 414], | 99.99th=[ 414] bw ( KiB/s): min= 8175, max= 8175, per=100.00%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 7, max= 7, avg= 7.00, stdev= 0.00, samples=1 write: IOPS=1, BW=1941KiB/s (1988kB/s)(7168KiB/3693msec) clat (msec): min=40, max=2535, avg=426.19, stdev=931.43 lat (msec): min=40, max=2535, avg=426.27, stdev=931.42 clat percentiles (msec): | 1.00th=[ 41], 5.00th=[ 41], 10.00th=[ 41], 20.00th=[ 46], | 30.00th=[ 50], 40.00th=[ 50], 50.00th=[ 55], 60.00th=[ 82], | 70.00th=[ 82], 80.00th=[ 178], 90.00th=[ 2534], 95.00th=[ 2534], | 99.00th=[ 2534], 99.50th=[ 2534], 99.90th=[ 2534], 99.95th=[ 2534], | 99.99th=[ 2534] bw ( KiB/s): min= 2048, max= 6144, per=70.96%, avg=4774.33, stdev=2361.08, samples=3 iops : min= 2, max= 6, avg= 4.33, stdev= 2.08, samples=3 lat (msec) : 50=33.33%, 100=33.33%, 250=16.67%, 500=8.33% cpu : usr=0.00%, sys=2.71%, ctx=94, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=4806KiB/s (4922kB/s), 552KiB/s-1682KiB/s (565kB/s-1723kB/s), io=20.0MiB (20.0MB), run=3693-4261msec WRITE: bw=6729KiB/s (6890kB/s), 1202KiB/s-2760KiB/s (1230kB/s-2826kB/s), io=28.0MiB (29.4MB), run=3693-4261msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=9816: Tue Jun 17 13:44:59 2025 read: IOPS=3, BW=13.6MiB/s (14.3MB/s)(12.0MiB/882msec) clat (msec): min=238, max=363, avg=280.46, stdev=71.87 lat (msec): min=238, max=363, avg=280.46, stdev=71.87 clat percentiles (msec): | 1.00th=[ 239], 5.00th=[ 239], 10.00th=[ 239], 20.00th=[ 239], | 30.00th=[ 239], 40.00th=[ 241], 50.00th=[ 241], 60.00th=[ 241], | 70.00th=[ 363], 80.00th=[ 363], 90.00th=[ 363], 95.00th=[ 363], | 99.00th=[ 363], 99.50th=[ 363], 99.90th=[ 363], 99.95th=[ 363], | 99.99th=[ 363] bw ( KiB/s): min= 8175, max= 8175, per=87.32%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.00%, sys=23.61%, ctx=822, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9817: Tue Jun 17 13:44:59 2025 read: IOPS=0, BW=2341KiB/s (2397kB/s)(4096KiB/1750msec) clat (nsec): min=330354k, max=330354k, avg=330353767.00, stdev= 0.00 lat (nsec): min=330356k, max=330356k, avg=330356080.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 330], 5.00th=[ 330], 10.00th=[ 330], 20.00th=[ 330], | 30.00th=[ 330], 40.00th=[ 330], 50.00th=[ 330], 60.00th=[ 330], | 70.00th=[ 330], 80.00th=[ 330], 90.00th=[ 330], 95.00th=[ 330], | 99.00th=[ 330], 99.50th=[ 330], 99.90th=[ 330], 99.95th=[ 330], | 99.99th=[ 330] bw ( KiB/s): min= 8126, max= 8126, per=86.80%, avg=8126.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=1, BW=4681KiB/s (4793kB/s)(8192KiB/1750msec) clat (msec): min=538, max=799, avg=669.07, stdev=184.69 lat (msec): min=538, max=799, avg=669.42, stdev=184.62 clat percentiles (msec): | 1.00th=[ 542], 5.00th=[ 542], 10.00th=[ 542], 20.00th=[ 542], | 30.00th=[ 542], 40.00th=[ 542], 50.00th=[ 542], 60.00th=[ 802], | 70.00th=[ 802], 80.00th=[ 802], 90.00th=[ 802], 95.00th=[ 802], | 99.00th=[ 802], 99.50th=[ 802], 99.90th=[ 802], 99.95th=[ 802], | 99.99th=[ 802] bw ( KiB/s): min= 8192, max= 8192, per=45.83%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 500=33.33%, 750=33.33%, 1000=33.33% cpu : usr=0.00%, sys=14.29%, ctx=274, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9819: Tue Jun 17 13:44:59 2025 write: IOPS=1, BW=6704KiB/s (6865kB/s)(12.0MiB/1833msec) clat (msec): min=325, max=719, avg=575.81, stdev=217.44 lat (msec): min=325, max=719, avg=576.08, stdev=217.43 clat percentiles (msec): | 1.00th=[ 326], 5.00th=[ 326], 10.00th=[ 326], 20.00th=[ 326], | 30.00th=[ 326], 40.00th=[ 684], 50.00th=[ 684], 60.00th=[ 684], | 70.00th=[ 718], 80.00th=[ 718], 90.00th=[ 718], 95.00th=[ 718], | 99.00th=[ 718], 99.50th=[ 718], 99.90th=[ 718], 99.95th=[ 718], | 99.99th=[ 718] bw ( KiB/s): min= 6884, max= 8094, per=41.89%, avg=7489.00, stdev=855.60, samples=2 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=2 lat (msec) : 500=33.33%, 750=66.67% cpu : usr=0.22%, sys=16.87%, ctx=79, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9821: Tue Jun 17 13:44:59 2025 write: IOPS=2, BW=11.2MiB/s (11.8MB/s)(12.0MiB/1070msec) clat (msec): min=302, max=386, avg=348.62, stdev=42.48 lat (msec): min=302, max=386, avg=348.90, stdev=42.47 clat percentiles (msec): | 1.00th=[ 305], 5.00th=[ 305], 10.00th=[ 305], 20.00th=[ 305], | 30.00th=[ 305], 40.00th=[ 359], 50.00th=[ 359], 60.00th=[ 359], | 70.00th=[ 388], 80.00th=[ 388], 90.00th=[ 388], 95.00th=[ 388], | 99.00th=[ 388], 99.50th=[ 388], 99.90th=[ 388], 99.95th=[ 388], | 99.99th=[ 388] bw ( KiB/s): min= 8159, max= 8175, per=45.69%, avg=8167.00, stdev=11.31, samples=2 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=2 lat (msec) : 500=100.00% cpu : usr=0.19%, sys=25.26%, ctx=107, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=9362KiB/s (9587kB/s), 2341KiB/s-13.6MiB/s (2397kB/s-14.3MB/s), io=16.0MiB (16.8MB), run=882-1750msec WRITE: bw=17.5MiB/s (18.3MB/s), 4681KiB/s-11.2MiB/s (4793kB/s-11.8MB/s), io=32.0MiB (33.6MB), run=1070-1833msec rand-rw: (groupid=0, jobs=1): err= 0: pid=9818: Tue Jun 17 13:44:59 2025 read: IOPS=1, BW=7959KiB/s (8150kB/s)(12.0MiB/1544msec) clat (msec): min=361, max=584, avg=486.70, stdev=113.94 lat (msec): min=361, max=584, avg=486.70, stdev=113.94 clat percentiles (msec): | 1.00th=[ 363], 5.00th=[ 363], 10.00th=[ 363], 20.00th=[ 363], | 30.00th=[ 363], 40.00th=[ 514], 50.00th=[ 514], 60.00th=[ 514], | 70.00th=[ 584], 80.00th=[ 584], 90.00th=[ 584], 95.00th=[ 584], | 99.00th=[ 584], 99.50th=[ 584], 99.90th=[ 584], 99.95th=[ 584], | 99.99th=[ 584] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 500=33.33%, 750=66.67% cpu : usr=0.00%, sys=4.93%, ctx=35, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9820: Tue Jun 17 13:44:59 2025 read: IOPS=0, BW=2038KiB/s (2087kB/s)(4096KiB/2010msec) clat (nsec): min=474600k, max=474600k, avg=474600351.00, stdev= 0.00 lat (nsec): min=474603k, max=474603k, avg=474602731.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 477], 5.00th=[ 477], 10.00th=[ 477], 20.00th=[ 477], | 30.00th=[ 477], 40.00th=[ 477], 50.00th=[ 477], 60.00th=[ 477], | 70.00th=[ 477], 80.00th=[ 477], 90.00th=[ 477], 95.00th=[ 477], | 99.00th=[ 477], 99.50th=[ 477], 99.90th=[ 477], 99.95th=[ 477], | 99.99th=[ 477] bw ( KiB/s): min= 8175, max= 8175, per=100.00%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=0, BW=4076KiB/s (4173kB/s)(8192KiB/2010msec) clat (msec): min=354, max=1073, avg=713.96, stdev=507.97 lat (msec): min=355, max=1073, avg=714.32, stdev=508.05 clat percentiles (msec): | 1.00th=[ 355], 5.00th=[ 355], 10.00th=[ 355], 20.00th=[ 355], | 30.00th=[ 355], 40.00th=[ 355], 50.00th=[ 355], 60.00th=[ 1070], | 70.00th=[ 1070], 80.00th=[ 1070], 90.00th=[ 1070], 95.00th=[ 1070], | 99.00th=[ 1070], 99.50th=[ 1070], 99.90th=[ 1070], 99.95th=[ 1070], | 99.99th=[ 1070] lat (msec) : 500=66.67% cpu : usr=0.30%, sys=9.01%, ctx=91, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9822: Tue Jun 17 13:44:59 2025 write: IOPS=1, BW=8100KiB/s (8295kB/s)(12.0MiB/1517msec) clat (msec): min=344, max=646, avg=484.05, stdev=152.39 lat (msec): min=344, max=647, avg=484.37, stdev=152.37 clat percentiles (msec): | 1.00th=[ 347], 5.00th=[ 347], 10.00th=[ 347], 20.00th=[ 347], | 30.00th=[ 347], 40.00th=[ 460], 50.00th=[ 460], 60.00th=[ 460], | 70.00th=[ 651], 80.00th=[ 651], 90.00th=[ 651], 95.00th=[ 651], | 99.00th=[ 651], 99.50th=[ 651], 99.90th=[ 651], 99.95th=[ 651], | 99.99th=[ 651] bw ( KiB/s): min=16384, max=16384, per=100.00%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 500=66.67%, 750=33.33% cpu : usr=0.00%, sys=9.56%, ctx=257, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9823: Tue Jun 17 13:44:59 2025 write: IOPS=2, BW=9.83MiB/s (10.3MB/s)(12.0MiB/1221msec) clat (msec): min=173, max=451, avg=358.11, stdev=159.94 lat (msec): min=173, max=451, avg=358.55, stdev=160.18 clat percentiles (msec): | 1.00th=[ 174], 5.00th=[ 174], 10.00th=[ 174], 20.00th=[ 174], | 30.00th=[ 174], 40.00th=[ 451], 50.00th=[ 451], 60.00th=[ 451], | 70.00th=[ 451], 80.00th=[ 451], 90.00th=[ 451], 95.00th=[ 451], | 99.00th=[ 451], 99.50th=[ 451], 99.90th=[ 451], 99.95th=[ 451], | 99.99th=[ 451] bw ( KiB/s): min= 8126, max= 8126, per=49.85%, avg=8126.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=66.67% cpu : usr=0.33%, sys=12.21%, ctx=68, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=8151KiB/s (8347kB/s), 2038KiB/s-7959KiB/s (2087kB/s-8150kB/s), io=16.0MiB (16.8MB), run=1544-2010msec WRITE: bw=15.9MiB/s (16.7MB/s), 4076KiB/s-9.83MiB/s (4173kB/s-10.3MB/s), io=32.0MiB (33.6MB), run=1221-2010msec