== sanity test 398b: DIO and buffer IO race ============== 00:02:53 (1746158573) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.85275 s, 27.2 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=3335: Fri May 2 00:03:27 2025 read: IOPS=50, BW=200KiB/s (205kB/s)(5992KiB/29945msec) clat (usec): min=52, max=247836, avg=8935.79, stdev=10368.77 lat (usec): min=52, max=247838, avg=8936.88, stdev=10368.79 clat percentiles (usec): | 1.00th=[ 79], 5.00th=[ 93], 10.00th=[ 104], 20.00th=[ 141], | 30.00th=[ 4948], 40.00th=[ 6325], 50.00th=[ 7570], 60.00th=[ 9110], | 70.00th=[ 11207], 80.00th=[ 14222], 90.00th=[ 17957], 95.00th=[ 22414], | 99.00th=[ 31589], 99.50th=[ 39060], 99.90th=[133694], 99.95th=[248513], | 99.99th=[248513] bw ( KiB/s): min= 31, max= 336, per=25.03%, avg=199.53, stdev=51.77, samples=59 iops : min= 7, max= 84, avg=49.64, stdev=13.00, samples=59 write: IOPS=52, BW=210KiB/s (215kB/s)(6296KiB/29945msec) clat (usec): min=507, max=81104, avg=10366.58, stdev=10484.90 lat (usec): min=507, max=81107, avg=10368.12, stdev=10484.69 clat percentiles (usec): | 1.00th=[ 644], 5.00th=[ 734], 10.00th=[ 783], 20.00th=[ 1004], | 30.00th=[ 2147], 40.00th=[ 4948], 50.00th=[ 8029], 60.00th=[10552], | 70.00th=[13435], 80.00th=[17695], 90.00th=[24511], 95.00th=[30016], | 99.00th=[46924], 99.50th=[54789], 99.90th=[67634], 99.95th=[81265], | 99.99th=[81265] bw ( KiB/s): min= 7, max= 328, per=25.02%, avg=208.93, stdev=56.66, samples=59 iops : min= 1, max= 82, avg=51.98, stdev=14.26, samples=59 lat (usec) : 100=3.91%, 250=6.80%, 500=0.29%, 750=3.94%, 1000=6.74% lat (msec) : 2=5.05%, 4=5.05%, 10=29.13%, 20=27.47%, 50=11.10% lat (msec) : 100=0.46%, 250=0.07% cpu : usr=0.16%, sys=14.45%, ctx=4926, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=3336: Fri May 2 00:03:27 2025 read: IOPS=49, BW=197KiB/s (202kB/s)(5940KiB/30095msec) clat (usec): min=46, max=99994, avg=8404.12, stdev=7774.80 lat (usec): min=46, max=99995, avg=8405.42, stdev=7774.84 clat percentiles (usec): | 1.00th=[ 82], 5.00th=[ 96], 10.00th=[ 105], 20.00th=[ 145], | 30.00th=[ 4621], 40.00th=[ 6128], 50.00th=[ 7242], 60.00th=[ 8717], | 70.00th=[ 10683], 80.00th=[ 13435], 90.00th=[ 17433], 95.00th=[ 21890], | 99.00th=[ 32113], 99.50th=[ 39584], 99.90th=[ 65274], 99.95th=[100140], | 99.99th=[100140] bw ( KiB/s): min= 15, max= 327, per=24.68%, avg=196.67, stdev=58.63, samples=60 iops : min= 3, max= 81, avg=48.85, stdev=14.66, samples=60 write: IOPS=52, BW=211KiB/s (216kB/s)(6348KiB/30095msec) clat (usec): min=456, max=308895, avg=10998.72, stdev=13690.27 lat (usec): min=456, max=308895, avg=11000.55, stdev=13690.41 clat percentiles (usec): | 1.00th=[ 644], 5.00th=[ 725], 10.00th=[ 775], 20.00th=[ 1057], | 30.00th=[ 1991], 40.00th=[ 4948], 50.00th=[ 8291], 60.00th=[ 10945], | 70.00th=[ 13960], 80.00th=[ 18220], 90.00th=[ 24511], 95.00th=[ 32375], | 99.00th=[ 50594], 99.50th=[ 68682], 99.90th=[100140], 99.95th=[308282], | 99.99th=[308282] bw ( KiB/s): min= 39, max= 368, per=25.10%, avg=209.58, stdev=54.68, samples=60 iops : min= 9, max= 92, avg=52.08, stdev=13.69, samples=60 lat (usec) : 50=0.03%, 100=3.71%, 250=6.87%, 500=0.49%, 750=4.04% lat (usec) : 1000=6.25% lat (msec) : 2=5.86%, 4=5.31%, 10=28.78%, 20=26.76%, 50=11.26% lat (msec) : 100=0.62%, 500=0.03% cpu : usr=0.12%, sys=14.22%, ctx=4942, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=3337: Fri May 2 00:03:27 2025 read: IOPS=50, BW=202KiB/s (206kB/s)(6028KiB/29911msec) clat (usec): min=51, max=322605, avg=8929.88, stdev=11297.12 lat (usec): min=51, max=322607, avg=8934.03, stdev=11299.01 clat percentiles (usec): | 1.00th=[ 77], 5.00th=[ 92], 10.00th=[ 102], 20.00th=[ 159], | 30.00th=[ 4948], 40.00th=[ 6194], 50.00th=[ 7570], 60.00th=[ 9110], | 70.00th=[ 10945], 80.00th=[ 13829], 90.00th=[ 18220], 95.00th=[ 22676], | 99.00th=[ 34866], 99.50th=[ 45876], 99.90th=[ 69731], 99.95th=[320865], | 99.99th=[320865] bw ( KiB/s): min= 32, max= 364, per=25.12%, avg=200.17, stdev=58.50, samples=59 iops : min= 8, max= 91, avg=49.90, stdev=14.66, samples=59 write: IOPS=52, BW=209KiB/s (214kB/s)(6260KiB/29911msec) clat (usec): min=466, max=117627, avg=10402.74, stdev=11153.17 lat (usec): min=467, max=117628, avg=10412.50, stdev=11152.93 clat percentiles (usec): | 1.00th=[ 660], 5.00th=[ 734], 10.00th=[ 783], 20.00th=[ 1004], | 30.00th=[ 2057], 40.00th=[ 4555], 50.00th=[ 7963], 60.00th=[ 10421], | 70.00th=[ 13566], 80.00th=[ 17171], 90.00th=[ 24249], 95.00th=[ 30540], | 99.00th=[ 43779], 99.50th=[ 60556], 99.90th=[ 99091], 99.95th=[117965], | 99.99th=[117965] bw ( KiB/s): min= 40, max= 351, per=24.87%, avg=207.64, stdev=56.94, samples=59 iops : min= 10, max= 87, avg=51.76, stdev=14.25, samples=59 lat (usec) : 100=4.23%, 250=6.02%, 500=0.65%, 750=4.04%, 1000=6.45% lat (msec) : 2=5.14%, 4=5.83%, 10=29.33%, 20=26.76%, 50=11.00% lat (msec) : 100=0.49%, 250=0.03%, 500=0.03% cpu : usr=0.26%, sys=14.30%, ctx=4859, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=3338: Fri May 2 00:03:27 2025 read: IOPS=50, BW=202KiB/s (207kB/s)(6052KiB/29961msec) clat (usec): min=61, max=74933, avg=8649.39, stdev=7637.26 lat (usec): min=62, max=74935, avg=8650.63, stdev=7637.41 clat percentiles (usec): | 1.00th=[ 82], 5.00th=[ 94], 10.00th=[ 103], 20.00th=[ 141], | 30.00th=[ 4752], 40.00th=[ 6259], 50.00th=[ 7701], 60.00th=[ 9241], | 70.00th=[11338], 80.00th=[13829], 90.00th=[18482], 95.00th=[22414], | 99.00th=[29754], 99.50th=[34866], 99.90th=[66323], 99.95th=[74974], | 99.99th=[74974] bw ( KiB/s): min= 23, max= 328, per=25.09%, avg=199.98, stdev=48.91, samples=59 iops : min= 5, max= 82, avg=49.66, stdev=12.30, samples=59 write: IOPS=52, BW=208KiB/s (213kB/s)(6236KiB/29961msec) clat (usec): min=451, max=223409, avg=10672.83, stdev=12204.21 lat (usec): min=452, max=223412, avg=10676.53, stdev=12203.52 clat percentiles (usec): | 1.00th=[ 627], 5.00th=[ 725], 10.00th=[ 766], 20.00th=[ 1106], | 30.00th=[ 2212], 40.00th=[ 5080], 50.00th=[ 7963], 60.00th=[ 10552], | 70.00th=[ 13435], 80.00th=[ 17695], 90.00th=[ 24773], 95.00th=[ 31851], | 99.00th=[ 48497], 99.50th=[ 61080], 99.90th=[112722], 99.95th=[223347], | 99.99th=[223347] bw ( KiB/s): min= 31, max= 320, per=24.63%, avg=205.68, stdev=49.10, samples=59 iops : min= 7, max= 80, avg=51.08, stdev=12.31, samples=59 lat (usec) : 100=3.97%, 250=6.87%, 500=0.75%, 750=4.26%, 1000=5.34% lat (msec) : 2=5.37%, 4=5.44%, 10=28.97%, 20=27.02%, 50=11.39% lat (msec) : 100=0.55%, 250=0.07% cpu : usr=0.15%, sys=14.15%, ctx=4942, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=798KiB/s (817kB/s), 197KiB/s-202KiB/s (202kB/s-207kB/s), io=23.4MiB (24.6MB), run=29911-30095msec WRITE: bw=835KiB/s (855kB/s), 208KiB/s-211KiB/s (213kB/s-216kB/s), io=24.6MiB (25.7MB), run=29911-30095msec rand-rw: (groupid=0, jobs=1): err= 0: pid=3331: Fri May 2 00:05:08 2025 read: IOPS=11, BW=45.7KiB/s (46.8kB/s)(5992KiB/131159msec) clat (msec): min=2, max=316, avg=13.31, stdev=12.55 lat (msec): min=2, max=316, avg=13.31, stdev=12.55 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 5], 10.00th=[ 5], 20.00th=[ 6], | 30.00th=[ 7], 40.00th=[ 9], 50.00th=[ 10], 60.00th=[ 13], | 70.00th=[ 17], 80.00th=[ 20], 90.00th=[ 25], 95.00th=[ 29], | 99.00th=[ 46], 99.50th=[ 61], 99.90th=[ 150], 99.95th=[ 317], | 99.99th=[ 317] bw ( KiB/s): min= 8, max= 160, per=25.53%, avg=46.47, stdev=25.81, samples=256 iops : min= 2, max= 40, avg=11.36, stdev= 6.44, samples=256 write: IOPS=12, BW=48.0KiB/s (49.2kB/s)(6296KiB/131159msec) clat (msec): min=26, max=179, avg=70.54, stdev=23.69 lat (msec): min=26, max=179, avg=70.54, stdev=23.69 clat percentiles (msec): | 1.00th=[ 34], 5.00th=[ 39], 10.00th=[ 44], 20.00th=[ 51], | 30.00th=[ 56], 40.00th=[ 62], 50.00th=[ 67], 60.00th=[ 73], | 70.00th=[ 81], 80.00th=[ 89], 90.00th=[ 104], 95.00th=[ 116], | 99.00th=[ 140], 99.50th=[ 150], 99.90th=[ 171], 99.95th=[ 180], | 99.99th=[ 180] bw ( KiB/s): min= 16, max= 88, per=25.03%, avg=47.81, stdev=12.45, samples=261 iops : min= 4, max= 22, avg=11.69, stdev= 3.13, samples=261 lat (msec) : 4=1.69%, 10=22.85%, 20=14.78%, 50=19.04%, 100=35.61% lat (msec) : 250=5.99%, 500=0.03% cpu : usr=0.07%, sys=2.89%, ctx=4210, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=3332: Fri May 2 00:05:08 2025 read: IOPS=11, BW=45.2KiB/s (46.3kB/s)(5940KiB/131496msec) clat (msec): min=2, max=298, avg=13.40, stdev=12.22 lat (msec): min=2, max=298, avg=13.41, stdev=12.22 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 5], 10.00th=[ 5], 20.00th=[ 6], | 30.00th=[ 7], 40.00th=[ 9], 50.00th=[ 11], 60.00th=[ 14], | 70.00th=[ 17], 80.00th=[ 20], 90.00th=[ 25], 95.00th=[ 29], | 99.00th=[ 42], 99.50th=[ 64], 99.90th=[ 153], 99.95th=[ 300], | 99.99th=[ 300] bw ( KiB/s): min= 7, max= 120, per=24.73%, avg=45.02, stdev=25.87, samples=261 iops : min= 1, max= 30, avg=11.04, stdev= 6.49, samples=261 write: IOPS=12, BW=48.3KiB/s (49.4kB/s)(6348KiB/131496msec) clat (msec): min=23, max=161, avg=70.20, stdev=23.26 lat (msec): min=23, max=161, avg=70.20, stdev=23.26 clat percentiles (msec): | 1.00th=[ 34], 5.00th=[ 41], 10.00th=[ 44], 20.00th=[ 51], | 30.00th=[ 56], 40.00th=[ 62], 50.00th=[ 66], 60.00th=[ 73], | 70.00th=[ 80], 80.00th=[ 88], 90.00th=[ 104], 95.00th=[ 117], | 99.00th=[ 136], 99.50th=[ 142], 99.90th=[ 157], 99.95th=[ 163], | 99.99th=[ 163] bw ( KiB/s): min= 23, max= 80, per=25.14%, avg=48.02, stdev=11.76, samples=261 iops : min= 5, max= 20, avg=11.79, stdev= 2.96, samples=261 lat (msec) : 4=1.50%, 10=21.61%, 20=16.08%, 50=19.08%, 100=35.51% lat (msec) : 250=6.18%, 500=0.03% cpu : usr=0.04%, sys=2.91%, ctx=4273, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=3333: Fri May 2 00:05:08 2025 read: IOPS=11, BW=46.4KiB/s (47.6kB/s)(6028KiB/129777msec) clat (usec): min=2939, max=92941, avg=12840.72, stdev=8774.52 lat (usec): min=2940, max=92943, avg=12842.33, stdev=8774.46 clat percentiles (usec): | 1.00th=[ 3490], 5.00th=[ 4228], 10.00th=[ 4686], 20.00th=[ 5735], | 30.00th=[ 6849], 40.00th=[ 8160], 50.00th=[10028], 60.00th=[12911], | 70.00th=[16057], 80.00th=[19530], 90.00th=[23462], 95.00th=[27395], | 99.00th=[41681], 99.50th=[49546], 99.90th=[81265], 99.95th=[92799], | 99.99th=[92799] bw ( KiB/s): min= 7, max= 144, per=25.80%, avg=46.96, stdev=25.23, samples=255 iops : min= 1, max= 36, avg=11.55, stdev= 6.30, samples=255 write: IOPS=12, BW=48.2KiB/s (49.4kB/s)(6260KiB/129777msec) clat (msec): min=26, max=372, avg=70.43, stdev=24.61 lat (msec): min=26, max=372, avg=70.43, stdev=24.61 clat percentiles (msec): | 1.00th=[ 34], 5.00th=[ 41], 10.00th=[ 45], 20.00th=[ 51], | 30.00th=[ 56], 40.00th=[ 62], 50.00th=[ 67], 60.00th=[ 73], | 70.00th=[ 80], 80.00th=[ 88], 90.00th=[ 103], 95.00th=[ 115], | 99.00th=[ 144], 99.50th=[ 157], 99.90th=[ 184], 99.95th=[ 372], | 99.99th=[ 372] bw ( KiB/s): min= 23, max= 80, per=25.21%, avg=48.15, stdev=11.92, samples=258 iops : min= 5, max= 20, avg=11.85, stdev= 3.01, samples=258 lat (msec) : 4=1.56%, 10=22.82%, 20=15.27%, 50=19.04%, 100=35.58% lat (msec) : 250=5.70%, 500=0.03% cpu : usr=0.05%, sys=2.87%, ctx=4186, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=3334: Fri May 2 00:05:08 2025 read: IOPS=11, BW=46.2KiB/s (47.3kB/s)(6052KiB/130941msec) clat (usec): min=2671, max=78466, avg=13079.09, stdev=8742.52 lat (usec): min=2672, max=78468, avg=13081.03, stdev=8742.25 clat percentiles (usec): | 1.00th=[ 3654], 5.00th=[ 4424], 10.00th=[ 5014], 20.00th=[ 5997], | 30.00th=[ 6849], 40.00th=[ 8291], 50.00th=[10290], 60.00th=[13304], | 70.00th=[16581], 80.00th=[19530], 90.00th=[23987], 95.00th=[28181], | 99.00th=[44303], 99.50th=[52691], 99.90th=[67634], 99.95th=[78119], | 99.99th=[78119] bw ( KiB/s): min= 7, max= 128, per=25.71%, avg=46.79, stdev=24.44, samples=256 iops : min= 1, max= 32, avg=11.45, stdev= 6.15, samples=256 write: IOPS=11, BW=47.6KiB/s (48.8kB/s)(6236KiB/130941msec) clat (msec): min=22, max=380, avg=71.12, stdev=25.73 lat (msec): min=22, max=380, avg=71.12, stdev=25.73 clat percentiles (msec): | 1.00th=[ 34], 5.00th=[ 40], 10.00th=[ 44], 20.00th=[ 51], | 30.00th=[ 56], 40.00th=[ 62], 50.00th=[ 67], 60.00th=[ 73], | 70.00th=[ 81], 80.00th=[ 90], 90.00th=[ 104], 95.00th=[ 118], | 99.00th=[ 144], 99.50th=[ 161], 99.90th=[ 207], 99.95th=[ 380], | 99.99th=[ 380] bw ( KiB/s): min= 15, max= 88, per=24.83%, avg=47.42, stdev=12.17, samples=260 iops : min= 3, max= 22, avg=11.60, stdev= 3.11, samples=260 lat (msec) : 4=1.24%, 10=22.85%, 20=16.02%, 50=18.88%, 100=35.06% lat (msec) : 250=5.92%, 500=0.03% cpu : usr=0.05%, sys=2.95%, ctx=4295, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=183KiB/s (187kB/s), 45.2KiB/s-46.4KiB/s (46.3kB/s-47.6kB/s), io=23.4MiB (24.6MB), run=129777-131496msec WRITE: bw=191KiB/s (196kB/s), 47.6KiB/s-48.3KiB/s (48.8kB/s-49.4kB/s), io=24.6MiB (25.7MB), run=129777-131496msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=3458: Fri May 2 00:05:16 2025 read: IOPS=57, BW=921KiB/s (943kB/s)(5920KiB/6431msec) clat (usec): min=63, max=98202, avg=8136.85, stdev=8425.18 lat (usec): min=63, max=98203, avg=8138.14, stdev=8425.06 clat percentiles (usec): | 1.00th=[ 71], 5.00th=[ 94], 10.00th=[ 104], 20.00th=[ 116], | 30.00th=[ 141], 40.00th=[ 6063], 50.00th=[ 7570], 60.00th=[ 9372], | 70.00th=[11076], 80.00th=[13829], 90.00th=[17433], 95.00th=[20841], | 99.00th=[27132], 99.50th=[31065], 99.90th=[98042], 99.95th=[98042], | 99.99th=[98042] bw ( KiB/s): min= 415, max= 1312, per=25.80%, avg=923.08, stdev=266.20, samples=12 iops : min= 25, max= 82, avg=57.17, stdev=16.60, samples=12 write: IOPS=61, BW=990KiB/s (1014kB/s)(6368KiB/6431msec) clat (usec): min=731, max=83071, avg=8518.75, stdev=10753.07 lat (usec): min=731, max=83072, avg=8520.60, stdev=10753.00 clat percentiles (usec): | 1.00th=[ 848], 5.00th=[ 1012], 10.00th=[ 1123], 20.00th=[ 1303], | 30.00th=[ 1811], 40.00th=[ 2474], 50.00th=[ 3458], 60.00th=[ 6783], | 70.00th=[ 9896], 80.00th=[13960], 90.00th=[21890], 95.00th=[28967], | 99.00th=[47449], 99.50th=[64226], 99.90th=[83362], 99.95th=[83362], | 99.99th=[83362] bw ( KiB/s): min= 606, max= 1469, per=26.29%, avg=989.67, stdev=292.23, samples=12 iops : min= 37, max= 91, avg=61.33, stdev=18.12, samples=12 lat (usec) : 100=3.39%, 250=12.11%, 500=0.13%, 750=0.39%, 1000=1.95% lat (msec) : 2=14.97%, 4=10.94%, 10=23.44%, 20=23.83%, 50=8.33% lat (msec) : 100=0.52% cpu : usr=0.09%, sys=16.14%, ctx=1100, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=3459: Fri May 2 00:05:16 2025 read: IOPS=53, BW=858KiB/s (879kB/s)(5744KiB/6693msec) clat (usec): min=69, max=54359, avg=7590.64, stdev=7720.33 lat (usec): min=69, max=54361, avg=7591.64, stdev=7720.41 clat percentiles (usec): | 1.00th=[ 79], 5.00th=[ 95], 10.00th=[ 102], 20.00th=[ 121], | 30.00th=[ 141], 40.00th=[ 5014], 50.00th=[ 6980], 60.00th=[ 8586], | 70.00th=[10290], 80.00th=[14615], 90.00th=[18482], 95.00th=[20579], | 99.00th=[28705], 99.50th=[30540], 99.90th=[54264], 99.95th=[54264], | 99.99th=[54264] bw ( KiB/s): min= 288, max= 1181, per=23.69%, avg=847.77, stdev=239.99, samples=13 iops : min= 18, max= 73, avg=52.69, stdev=14.73, samples=13 write: IOPS=61, BW=978KiB/s (1001kB/s)(6544KiB/6693msec) clat (usec): min=648, max=112448, avg=9282.42, stdev=12006.66 lat (usec): min=650, max=112450, avg=9284.17, stdev=12006.76 clat percentiles (usec): | 1.00th=[ 906], 5.00th=[ 1029], 10.00th=[ 1123], 20.00th=[ 1319], | 30.00th=[ 1860], 40.00th=[ 2606], 50.00th=[ 4948], 60.00th=[ 8291], | 70.00th=[ 11338], 80.00th=[ 16450], 90.00th=[ 21890], 95.00th=[ 26608], | 99.00th=[ 46924], 99.50th=[ 92799], 99.90th=[112722], 99.95th=[112722], | 99.99th=[112722] bw ( KiB/s): min= 544, max= 1245, per=25.53%, avg=961.08, stdev=178.69, samples=13 iops : min= 34, max= 77, avg=59.77, stdev=11.06, samples=13 lat (usec) : 100=3.52%, 250=12.76%, 500=1.43%, 750=0.13%, 1000=2.08% lat (msec) : 2=14.71%, 4=8.72%, 10=24.35%, 20=22.40%, 50=9.37% lat (msec) : 100=0.39%, 250=0.13% cpu : usr=0.03%, sys=15.71%, ctx=1081, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=3460: Fri May 2 00:05:16 2025 read: IOPS=63, BW=1011KiB/s (1036kB/s)(6256KiB/6186msec) clat (usec): min=76, max=138623, avg=8455.89, stdev=10313.57 lat (usec): min=76, max=138624, avg=8458.12, stdev=10313.73 clat percentiles (usec): | 1.00th=[ 86], 5.00th=[ 94], 10.00th=[ 104], 20.00th=[ 117], | 30.00th=[ 231], 40.00th=[ 5997], 50.00th=[ 7767], 60.00th=[ 9372], | 70.00th=[ 11076], 80.00th=[ 13698], 90.00th=[ 18220], 95.00th=[ 22152], | 99.00th=[ 34341], 99.50th=[ 66323], 99.90th=[139461], 99.95th=[139461], | 99.99th=[139461] bw ( KiB/s): min= 606, max= 1498, per=28.35%, avg=1014.42, stdev=283.59, samples=12 iops : min= 37, max= 93, avg=63.08, stdev=17.73, samples=12 write: IOPS=60, BW=975KiB/s (999kB/s)(6032KiB/6186msec) clat (usec): min=753, max=56001, avg=7492.88, stdev=8640.00 lat (usec): min=754, max=56002, avg=7494.44, stdev=8639.98 clat percentiles (usec): | 1.00th=[ 832], 5.00th=[ 996], 10.00th=[ 1106], 20.00th=[ 1254], | 30.00th=[ 1532], 40.00th=[ 2376], 50.00th=[ 3425], 60.00th=[ 5276], | 70.00th=[ 8848], 80.00th=[13698], 90.00th=[20579], 95.00th=[25035], | 99.00th=[37487], 99.50th=[43254], 99.90th=[55837], 99.95th=[55837], | 99.99th=[55837] bw ( KiB/s): min= 608, max= 1721, per=25.52%, avg=960.92, stdev=306.25, samples=12 iops : min= 38, max= 107, avg=59.75, stdev=18.99, samples=12 lat (usec) : 100=4.56%, 250=10.81%, 500=0.65%, 750=0.91%, 1000=2.60% lat (msec) : 2=16.02%, 4=8.72%, 10=23.57%, 20=23.44%, 50=8.33% lat (msec) : 100=0.26%, 250=0.13% cpu : usr=0.13%, sys=16.41%, ctx=1075, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=3461: Fri May 2 00:05:16 2025 read: IOPS=57, BW=919KiB/s (941kB/s)(6032KiB/6564msec) clat (usec): min=71, max=58275, avg=8038.28, stdev=8699.50 lat (usec): min=72, max=58275, avg=8039.23, stdev=8699.47 clat percentiles (usec): | 1.00th=[ 87], 5.00th=[ 94], 10.00th=[ 102], 20.00th=[ 114], | 30.00th=[ 139], 40.00th=[ 1598], 50.00th=[ 7111], 60.00th=[ 8717], | 70.00th=[11600], 80.00th=[14746], 90.00th=[18482], 95.00th=[22414], | 99.00th=[36439], 99.50th=[52167], 99.90th=[58459], 99.95th=[58459], | 99.99th=[58459] bw ( KiB/s): min= 160, max= 1312, per=25.47%, avg=911.38, stdev=282.34, samples=13 iops : min= 10, max= 82, avg=56.54, stdev=17.56, samples=13 write: IOPS=59, BW=953KiB/s (976kB/s)(6256KiB/6564msec) clat (usec): min=585, max=111363, avg=8597.98, stdev=11891.91 lat (usec): min=585, max=111364, avg=8599.47, stdev=11891.94 clat percentiles (usec): | 1.00th=[ 840], 5.00th=[ 1012], 10.00th=[ 1106], 20.00th=[ 1221], | 30.00th=[ 1582], 40.00th=[ 2343], 50.00th=[ 3458], 60.00th=[ 5800], | 70.00th=[ 10421], 80.00th=[ 14746], 90.00th=[ 21365], 95.00th=[ 28967], | 99.00th=[ 61604], 99.50th=[ 93848], 99.90th=[111674], 99.95th=[111674], | 99.99th=[111674] bw ( KiB/s): min= 224, max= 1536, per=25.25%, avg=950.62, stdev=333.59, samples=13 iops : min= 14, max= 96, avg=59.00, stdev=20.82, samples=13 lat (usec) : 100=4.04%, 250=13.02%, 500=1.04%, 750=0.65%, 1000=2.08% lat (msec) : 2=16.54%, 4=9.77%, 10=20.70%, 20=21.74%, 50=9.51% lat (msec) : 100=0.78%, 250=0.13% cpu : usr=0.12%, sys=15.62%, ctx=1034, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=3579KiB/s (3665kB/s), 858KiB/s-1011KiB/s (879kB/s-1036kB/s), io=23.4MiB (24.5MB), run=6186-6693msec WRITE: bw=3765KiB/s (3855kB/s), 953KiB/s-990KiB/s (976kB/s-1014kB/s), io=24.6MiB (25.8MB), run=6186-6693msec rand-rw: (groupid=0, jobs=1): err= 0: pid=3454: Fri May 2 00:05:42 2025 read: IOPS=11, BW=186KiB/s (190kB/s)(5920KiB/31875msec) clat (usec): min=2884, max=61576, avg=10090.71, stdev=6944.11 lat (usec): min=2885, max=61577, avg=10092.14, stdev=6944.02 clat percentiles (usec): | 1.00th=[ 3785], 5.00th=[ 4359], 10.00th=[ 4883], 20.00th=[ 5669], | 30.00th=[ 6390], 40.00th=[ 6980], 50.00th=[ 7898], 60.00th=[ 8979], | 70.00th=[10683], 80.00th=[12911], 90.00th=[17957], 95.00th=[22414], | 99.00th=[44303], 99.50th=[44827], 99.90th=[61604], 99.95th=[61604], | 99.99th=[61604] bw ( KiB/s): min= 32, max= 512, per=25.12%, avg=184.60, stdev=102.62, samples=63 iops : min= 2, max= 32, avg=11.30, stdev= 6.40, samples=63 write: IOPS=12, BW=200KiB/s (205kB/s)(6368KiB/31875msec) clat (msec): min=29, max=173, avg=70.22, stdev=24.35 lat (msec): min=29, max=173, avg=70.23, stdev=24.35 clat percentiles (msec): | 1.00th=[ 33], 5.00th=[ 41], 10.00th=[ 44], 20.00th=[ 50], | 30.00th=[ 55], 40.00th=[ 62], 50.00th=[ 66], 60.00th=[ 72], | 70.00th=[ 80], 80.00th=[ 87], 90.00th=[ 104], 95.00th=[ 120], | 99.00th=[ 159], 99.50th=[ 167], 99.90th=[ 174], 99.95th=[ 174], | 99.99th=[ 174] bw ( KiB/s): min= 95, max= 288, per=25.79%, avg=199.35, stdev=49.34, samples=63 iops : min= 5, max= 18, avg=12.22, stdev= 3.11, samples=63 lat (msec) : 4=0.91%, 10=31.51%, 20=12.89%, 50=13.67%, 100=35.42% lat (msec) : 250=5.60% cpu : usr=0.07%, sys=2.76%, ctx=914, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=3455: Fri May 2 00:05:42 2025 read: IOPS=11, BW=176KiB/s (181kB/s)(5744KiB/32579msec) clat (usec): min=3434, max=73185, avg=10706.04, stdev=8601.50 lat (usec): min=3435, max=73187, avg=10707.67, stdev=8601.27 clat percentiles (usec): | 1.00th=[ 3654], 5.00th=[ 4293], 10.00th=[ 4817], 20.00th=[ 5407], | 30.00th=[ 6128], 40.00th=[ 7046], 50.00th=[ 7963], 60.00th=[ 8979], | 70.00th=[10945], 80.00th=[13960], 90.00th=[19268], 95.00th=[28443], | 99.00th=[46400], 99.50th=[58983], 99.90th=[72877], 99.95th=[72877], | 99.99th=[72877] bw ( KiB/s): min= 53, max= 480, per=24.64%, avg=181.10, stdev=95.12, samples=62 iops : min= 3, max= 30, avg=10.95, stdev= 6.04, samples=62 write: IOPS=12, BW=201KiB/s (206kB/s)(6544KiB/32579msec) clat (msec): min=32, max=218, avg=70.19, stdev=25.94 lat (msec): min=32, max=218, avg=70.19, stdev=25.94 clat percentiles (msec): | 1.00th=[ 36], 5.00th=[ 41], 10.00th=[ 43], 20.00th=[ 50], | 30.00th=[ 55], 40.00th=[ 61], 50.00th=[ 66], 60.00th=[ 71], | 70.00th=[ 77], 80.00th=[ 87], 90.00th=[ 102], 95.00th=[ 124], | 99.00th=[ 157], 99.50th=[ 178], 99.90th=[ 220], 99.95th=[ 220], | 99.99th=[ 220] bw ( KiB/s): min= 63, max= 288, per=25.72%, avg=198.84, stdev=51.63, samples=64 iops : min= 3, max= 18, avg=12.08, stdev= 3.29, samples=64 lat (msec) : 4=1.56%, 10=29.04%, 20=11.59%, 50=15.76%, 100=36.46% lat (msec) : 250=5.60% cpu : usr=0.02%, sys=2.74%, ctx=915, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=3456: Fri May 2 00:05:42 2025 read: IOPS=12, BW=201KiB/s (206kB/s)(6256KiB/31112msec) clat (msec): min=2, max=140, avg=10.51, stdev=11.44 lat (msec): min=2, max=140, avg=10.51, stdev=11.44 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 5], 10.00th=[ 5], 20.00th=[ 6], | 30.00th=[ 6], 40.00th=[ 7], 50.00th=[ 8], 60.00th=[ 9], | 70.00th=[ 10], 80.00th=[ 13], 90.00th=[ 18], 95.00th=[ 30], | 99.00th=[ 58], 99.50th=[ 88], 99.90th=[ 140], 99.95th=[ 140], | 99.99th=[ 140] bw ( KiB/s): min= 31, max= 542, per=27.86%, avg=204.80, stdev=114.31, samples=61 iops : min= 1, max= 33, avg=12.52, stdev= 7.13, samples=61 write: IOPS=12, BW=194KiB/s (199kB/s)(6032KiB/31112msec) clat (msec): min=27, max=177, avg=71.11, stdev=25.46 lat (msec): min=27, max=177, avg=71.12, stdev=25.46 clat percentiles (msec): | 1.00th=[ 35], 5.00th=[ 41], 10.00th=[ 45], 20.00th=[ 50], | 30.00th=[ 55], 40.00th=[ 61], 50.00th=[ 67], 60.00th=[ 72], | 70.00th=[ 79], 80.00th=[ 89], 90.00th=[ 105], 95.00th=[ 116], | 99.00th=[ 163], 99.50th=[ 167], 99.90th=[ 178], 99.95th=[ 178], | 99.99th=[ 178] bw ( KiB/s): min= 95, max= 288, per=25.00%, avg=193.26, stdev=50.98, samples=62 iops : min= 5, max= 18, avg=11.81, stdev= 3.21, samples=62 lat (msec) : 4=2.08%, 10=33.72%, 20=10.68%, 50=13.80%, 100=33.46% lat (msec) : 250=6.25% cpu : usr=0.08%, sys=2.89%, ctx=930, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=3457: Fri May 2 00:05:42 2025 read: IOPS=11, BW=192KiB/s (196kB/s)(6032KiB/31474msec) clat (usec): min=3617, max=71694, avg=10807.96, stdev=8793.38 lat (usec): min=3619, max=71695, avg=10809.54, stdev=8793.29 clat percentiles (usec): | 1.00th=[ 3818], 5.00th=[ 4490], 10.00th=[ 4948], 20.00th=[ 5669], | 30.00th=[ 6194], 40.00th=[ 7111], 50.00th=[ 7898], 60.00th=[ 9372], | 70.00th=[10945], 80.00th=[13042], 90.00th=[18482], 95.00th=[28443], | 99.00th=[47973], 99.50th=[53216], 99.90th=[71828], 99.95th=[71828], | 99.99th=[71828] bw ( KiB/s): min= 32, max= 448, per=26.29%, avg=193.21, stdev=99.06, samples=61 iops : min= 2, max= 28, avg=11.75, stdev= 6.20, samples=61 write: IOPS=12, BW=199KiB/s (204kB/s)(6256KiB/31474msec) clat (msec): min=27, max=208, avg=69.84, stdev=26.03 lat (msec): min=27, max=208, avg=69.85, stdev=26.04 clat percentiles (msec): | 1.00th=[ 33], 5.00th=[ 39], 10.00th=[ 43], 20.00th=[ 48], | 30.00th=[ 55], 40.00th=[ 61], 50.00th=[ 65], 60.00th=[ 71], | 70.00th=[ 79], 80.00th=[ 86], 90.00th=[ 102], 95.00th=[ 117], | 99.00th=[ 174], 99.50th=[ 190], 99.90th=[ 209], 99.95th=[ 209], | 99.99th=[ 209] bw ( KiB/s): min= 64, max= 320, per=25.66%, avg=198.32, stdev=52.18, samples=62 iops : min= 4, max= 20, avg=12.08, stdev= 3.30, samples=62 lat (msec) : 4=0.78%, 10=30.86%, 20=12.89%, 50=16.15%, 100=33.98% lat (msec) : 250=5.34% cpu : usr=0.03%, sys=2.89%, ctx=929, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=735KiB/s (753kB/s), 176KiB/s-201KiB/s (181kB/s-206kB/s), io=23.4MiB (24.5MB), run=31112-32579msec WRITE: bw=774KiB/s (792kB/s), 194KiB/s-201KiB/s (199kB/s-206kB/s), io=24.6MiB (25.8MB), run=31112-32579msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=3494: Fri May 2 00:05:45 2025 read: IOPS=5, BW=5626KiB/s (5761kB/s)(7168KiB/1274msec) clat (usec): min=618, max=179276, avg=50898.49, stdev=68903.25 lat (usec): min=619, max=179279, avg=50899.97, stdev=68903.57 clat percentiles (usec): | 1.00th=[ 619], 5.00th=[ 619], 10.00th=[ 619], 20.00th=[ 865], | 30.00th=[ 1074], 40.00th=[ 1074], 50.00th=[ 4555], 60.00th=[ 81265], | 70.00th=[ 81265], 80.00th=[ 88605], 90.00th=[179307], 95.00th=[179307], | 99.00th=[179307], 99.50th=[179307], 99.90th=[179307], 99.95th=[179307], | 99.99th=[179307] bw ( KiB/s): min= 4096, max= 6131, per=31.81%, avg=5113.50, stdev=1438.96, samples=2 iops : min= 4, max= 5, avg= 4.50, stdev= 0.71, samples=2 write: IOPS=3, BW=4019KiB/s (4115kB/s)(5120KiB/1274msec) clat (msec): min=61, max=317, avg=165.25, stdev=93.63 lat (msec): min=61, max=317, avg=165.30, stdev=93.64 clat percentiles (msec): | 1.00th=[ 62], 5.00th=[ 62], 10.00th=[ 62], 20.00th=[ 62], | 30.00th=[ 138], 40.00th=[ 138], 50.00th=[ 144], 60.00th=[ 144], | 70.00th=[ 167], 80.00th=[ 167], 90.00th=[ 317], 95.00th=[ 317], | 99.00th=[ 317], 99.50th=[ 317], 99.90th=[ 317], 99.95th=[ 317], | 99.99th=[ 317] bw ( KiB/s): min= 6144, max= 6144, per=27.30%, avg=6144.00, stdev= 0.00, samples=1 iops : min= 6, max= 6, avg= 6.00, stdev= 0.00, samples=1 lat (usec) : 750=8.33%, 1000=8.33% lat (msec) : 2=8.33%, 10=8.33%, 100=25.00%, 250=33.33%, 500=8.33% cpu : usr=0.00%, sys=13.28%, ctx=206, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=3495: Fri May 2 00:05:45 2025 read: IOPS=2, BW=2193KiB/s (2245kB/s)(2048KiB/934msec) clat (usec): min=60588, max=85737, avg=73163.27, stdev=17783.33 lat (usec): min=60591, max=85739, avg=73165.56, stdev=17782.45 clat percentiles (usec): | 1.00th=[60556], 5.00th=[60556], 10.00th=[60556], 20.00th=[60556], | 30.00th=[60556], 40.00th=[60556], 50.00th=[60556], 60.00th=[85459], | 70.00th=[85459], 80.00th=[85459], 90.00th=[85459], 95.00th=[85459], | 99.00th=[85459], 99.50th=[85459], 99.90th=[85459], 99.95th=[85459], | 99.99th=[85459] bw ( KiB/s): min= 2043, max= 2043, per=12.71%, avg=2043.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=10, BW=10.7MiB/s (11.2MB/s)(10.0MiB/934msec) clat (msec): min=25, max=144, avg=75.48, stdev=35.94 lat (msec): min=25, max=144, avg=75.54, stdev=35.94 clat percentiles (msec): | 1.00th=[ 26], 5.00th=[ 26], 10.00th=[ 26], 20.00th=[ 34], | 30.00th=[ 50], 40.00th=[ 64], 50.00th=[ 74], 60.00th=[ 75], | 70.00th=[ 101], 80.00th=[ 101], 90.00th=[ 110], 95.00th=[ 144], | 99.00th=[ 144], 99.50th=[ 144], 99.90th=[ 144], 99.95th=[ 144], | 99.99th=[ 144] bw ( KiB/s): min= 8175, max= 8175, per=36.33%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 7, max= 7, avg= 7.00, stdev= 0.00, samples=1 lat (msec) : 50=25.00%, 100=50.00%, 250=25.00% cpu : usr=0.00%, sys=26.47%, ctx=400, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=3496: Fri May 2 00:05:45 2025 read: IOPS=5, BW=5642KiB/s (5777kB/s)(6144KiB/1089msec) clat (usec): min=473, max=253226, avg=66304.76, stdev=97783.78 lat (usec): min=475, max=253227, avg=66306.19, stdev=97783.72 clat percentiles (usec): | 1.00th=[ 474], 5.00th=[ 474], 10.00th=[ 474], 20.00th=[ 644], | 30.00th=[ 644], 40.00th=[ 1647], 50.00th=[ 1647], 60.00th=[ 70779], | 70.00th=[ 70779], 80.00th=[ 70779], 90.00th=[252707], 95.00th=[252707], | 99.00th=[252707], 99.50th=[252707], 99.90th=[252707], 99.95th=[252707], | 99.99th=[252707] bw ( KiB/s): min= 2043, max= 6144, per=25.47%, avg=4093.50, stdev=2899.84, samples=2 iops : min= 1, max= 6, avg= 3.50, stdev= 3.54, samples=2 write: IOPS=5, BW=5642KiB/s (5777kB/s)(6144KiB/1089msec) clat (msec): min=37, max=218, avg=110.51, stdev=68.01 lat (msec): min=37, max=218, avg=110.58, stdev=68.01 clat percentiles (msec): | 1.00th=[ 39], 5.00th=[ 39], 10.00th=[ 39], 20.00th=[ 58], | 30.00th=[ 58], 40.00th=[ 70], 50.00th=[ 70], 60.00th=[ 134], | 70.00th=[ 146], 80.00th=[ 146], 90.00th=[ 218], 95.00th=[ 218], | 99.00th=[ 218], 99.50th=[ 218], 99.90th=[ 218], 99.95th=[ 218], | 99.99th=[ 218] bw ( KiB/s): min= 2043, max= 8192, per=22.74%, avg=5117.50, stdev=4348.00, samples=2 iops : min= 1, max= 8, avg= 4.50, stdev= 4.95, samples=2 lat (usec) : 500=8.33%, 750=8.33% lat (msec) : 2=8.33%, 50=8.33%, 100=33.33%, 250=25.00%, 500=8.33% cpu : usr=0.00%, sys=18.01%, ctx=369, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=3497: Fri May 2 00:05:45 2025 read: IOPS=4, BW=4217KiB/s (4319kB/s)(5120KiB/1214msec) clat (usec): min=338, max=66130, avg=19045.62, stdev=28855.94 lat (usec): min=339, max=66132, avg=19046.58, stdev=28856.15 clat percentiles (usec): | 1.00th=[ 338], 5.00th=[ 338], 10.00th=[ 338], 20.00th=[ 338], | 30.00th=[ 375], 40.00th=[ 375], 50.00th=[ 627], 60.00th=[ 627], | 70.00th=[27657], 80.00th=[27657], 90.00th=[66323], 95.00th=[66323], | 99.00th=[66323], 99.50th=[66323], 99.90th=[66323], 99.95th=[66323], | 99.99th=[66323] bw ( KiB/s): min= 8175, max= 8175, per=50.86%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 7, max= 7, avg= 7.00, stdev= 0.00, samples=1 write: IOPS=5, BW=5904KiB/s (6046kB/s)(7168KiB/1214msec) clat (msec): min=51, max=325, avg=148.21, stdev=109.69 lat (msec): min=51, max=325, avg=148.29, stdev=109.67 clat percentiles (msec): | 1.00th=[ 52], 5.00th=[ 52], 10.00th=[ 52], 20.00th=[ 57], | 30.00th=[ 81], 40.00th=[ 81], 50.00th=[ 112], 60.00th=[ 134], | 70.00th=[ 134], 80.00th=[ 279], 90.00th=[ 326], 95.00th=[ 326], | 99.00th=[ 326], 99.50th=[ 326], 99.90th=[ 326], 99.95th=[ 326], | 99.99th=[ 326] bw ( KiB/s): min= 4087, max= 8192, per=27.28%, avg=6139.50, stdev=2902.67, samples=2 iops : min= 3, max= 8, avg= 5.50, stdev= 3.54, samples=2 lat (usec) : 500=16.67%, 750=8.33% lat (msec) : 50=8.33%, 100=33.33%, 250=16.67%, 500=16.67% cpu : usr=0.00%, sys=16.24%, ctx=129, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=15.7MiB/s (16.5MB/s), 2193KiB/s-5642KiB/s (2245kB/s-5777kB/s), io=20.0MiB (20.0MB), run=934-1274msec WRITE: bw=21.0MiB/s (23.0MB/s), 4019KiB/s-10.7MiB/s (4115kB/s-11.2MB/s), io=28.0MiB (29.4MB), run=934-1274msec rand-rw: (groupid=0, jobs=1): err= 0: pid=3498: Fri May 2 00:05:45 2025 read: IOPS=6, BW=6469KiB/s (6625kB/s)(7168KiB/1108msec) clat (msec): min=19, max=120, avg=56.01, stdev=44.71 lat (msec): min=19, max=120, avg=56.01, stdev=44.71 clat percentiles (msec): | 1.00th=[ 20], 5.00th=[ 20], 10.00th=[ 20], 20.00th=[ 23], | 30.00th=[ 26], 40.00th=[ 26], 50.00th=[ 34], 60.00th=[ 52], | 70.00th=[ 52], 80.00th=[ 120], 90.00th=[ 121], 95.00th=[ 121], | 99.00th=[ 121], 99.50th=[ 121], 99.90th=[ 121], 99.95th=[ 121], | 99.99th=[ 121] bw ( KiB/s): min= 2048, max=10240, per=50.10%, avg=6144.00, stdev=5792.62, samples=2 iops : min= 2, max= 10, avg= 6.00, stdev= 5.66, samples=2 write: IOPS=4, BW=4621KiB/s (4732kB/s)(5120KiB/1108msec) clat (msec): min=87, max=229, avg=132.99, stdev=56.90 lat (msec): min=87, max=233, avg=133.95, stdev=58.58 clat percentiles (msec): | 1.00th=[ 88], 5.00th=[ 88], 10.00th=[ 88], 20.00th=[ 88], | 30.00th=[ 102], 40.00th=[ 102], 50.00th=[ 109], 60.00th=[ 109], | 70.00th=[ 140], 80.00th=[ 140], 90.00th=[ 230], 95.00th=[ 230], | 99.00th=[ 230], 99.50th=[ 230], 99.90th=[ 230], 99.95th=[ 230], | 99.99th=[ 230] bw ( KiB/s): min= 2048, max= 6144, per=23.86%, avg=4096.00, stdev=2896.31, samples=2 iops : min= 2, max= 6, avg= 4.00, stdev= 2.83, samples=2 lat (msec) : 20=8.33%, 50=25.00%, 100=16.67%, 250=50.00% cpu : usr=0.27%, sys=3.79%, ctx=25, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=3499: Fri May 2 00:05:45 2025 read: IOPS=1, BW=1226KiB/s (1256kB/s)(2048KiB/1670msec) clat (usec): min=66796, max=77140, avg=71968.65, stdev=7314.54 lat (usec): min=66798, max=77143, avg=71970.74, stdev=7315.01 clat percentiles (usec): | 1.00th=[66847], 5.00th=[66847], 10.00th=[66847], 20.00th=[66847], | 30.00th=[66847], 40.00th=[66847], 50.00th=[66847], 60.00th=[77071], | 70.00th=[77071], 80.00th=[77071], 90.00th=[77071], 95.00th=[77071], | 99.00th=[77071], 99.50th=[77071], 99.90th=[77071], 99.95th=[77071], | 99.99th=[77071] bw ( KiB/s): min= 2043, max= 2048, per=16.68%, avg=2045.50, stdev= 3.54, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 write: IOPS=5, BW=6132KiB/s (6279kB/s)(10.0MiB/1670msec) clat (msec): min=75, max=189, avg=137.23, stdev=40.07 lat (msec): min=75, max=189, avg=137.31, stdev=40.07 clat percentiles (msec): | 1.00th=[ 77], 5.00th=[ 77], 10.00th=[ 77], 20.00th=[ 81], | 30.00th=[ 118], 40.00th=[ 120], 50.00th=[ 136], 60.00th=[ 140], | 70.00th=[ 165], 80.00th=[ 165], 90.00th=[ 190], 95.00th=[ 190], | 99.00th=[ 190], 99.50th=[ 190], 99.90th=[ 190], 99.95th=[ 190], | 99.99th=[ 190] bw ( KiB/s): min= 2043, max= 8192, per=27.83%, avg=4777.00, stdev=3130.55, samples=3 iops : min= 1, max= 8, avg= 4.33, stdev= 3.51, samples=3 lat (msec) : 100=33.33%, 250=66.67% cpu : usr=0.12%, sys=3.18%, ctx=35, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=3500: Fri May 2 00:05:45 2025 read: IOPS=3, BW=4050KiB/s (4147kB/s)(6144KiB/1517msec) clat (usec): min=28270, max=91100, avg=53583.80, stdev=22967.64 lat (usec): min=28271, max=91101, avg=53585.46, stdev=22967.74 clat percentiles (usec): | 1.00th=[28181], 5.00th=[28181], 10.00th=[28181], 20.00th=[32900], | 30.00th=[32900], 40.00th=[50070], 50.00th=[50070], 60.00th=[52691], | 70.00th=[65799], 80.00th=[65799], 90.00th=[90702], 95.00th=[90702], | 99.00th=[90702], 99.50th=[90702], 99.90th=[90702], 99.95th=[90702], | 99.99th=[90702] bw ( KiB/s): min= 4096, max= 6144, per=41.75%, avg=5120.00, stdev=1448.15, samples=2 iops : min= 4, max= 6, avg= 5.00, stdev= 1.41, samples=2 write: IOPS=3, BW=4050KiB/s (4147kB/s)(6144KiB/1517msec) clat (msec): min=97, max=171, avg=148.64, stdev=27.00 lat (msec): min=97, max=171, avg=148.71, stdev=27.00 clat percentiles (msec): | 1.00th=[ 99], 5.00th=[ 99], 10.00th=[ 99], 20.00th=[ 148], | 30.00th=[ 148], 40.00th=[ 150], 50.00th=[ 150], 60.00th=[ 155], | 70.00th=[ 171], 80.00th=[ 171], 90.00th=[ 171], 95.00th=[ 171], | 99.00th=[ 171], 99.50th=[ 171], 99.90th=[ 171], 99.95th=[ 171], | 99.99th=[ 171] bw ( KiB/s): min= 2043, max= 6144, per=23.85%, avg=4094.33, stdev=2050.50, samples=3 iops : min= 1, max= 6, avg= 3.67, stdev= 2.52, samples=3 lat (msec) : 50=16.67%, 100=41.67%, 250=41.67% cpu : usr=0.33%, sys=4.02%, ctx=45, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=3501: Fri May 2 00:05:45 2025 read: IOPS=3, BW=3427KiB/s (3509kB/s)(5120KiB/1494msec) clat (usec): min=16819, max=70416, avg=50253.55, stdev=21773.86 lat (usec): min=16820, max=70418, avg=50255.18, stdev=21774.20 clat percentiles (usec): | 1.00th=[16909], 5.00th=[16909], 10.00th=[16909], 20.00th=[16909], | 30.00th=[45876], 40.00th=[45876], 50.00th=[49546], 60.00th=[49546], | 70.00th=[68682], 80.00th=[68682], 90.00th=[70779], 95.00th=[70779], | 99.00th=[70779], 99.50th=[70779], 99.90th=[70779], 99.95th=[70779], | 99.99th=[70779] bw ( KiB/s): min= 6119, max= 6119, per=49.90%, avg=6119.00, stdev= 0.00, samples=1 iops : min= 5, max= 5, avg= 5.00, stdev= 0.00, samples=1 write: IOPS=4, BW=4798KiB/s (4913kB/s)(7168KiB/1494msec) clat (msec): min=82, max=214, avg=134.39, stdev=42.20 lat (msec): min=82, max=214, avg=134.49, stdev=42.18 clat percentiles (msec): | 1.00th=[ 83], 5.00th=[ 83], 10.00th=[ 83], 20.00th=[ 101], | 30.00th=[ 126], 40.00th=[ 126], 50.00th=[ 132], 60.00th=[ 136], | 70.00th=[ 136], 80.00th=[ 153], 90.00th=[ 215], 95.00th=[ 215], | 99.00th=[ 215], 99.50th=[ 215], 99.90th=[ 215], 99.95th=[ 215], | 99.99th=[ 215] bw ( KiB/s): min= 2043, max= 6119, per=23.77%, avg=4081.00, stdev=2882.17, samples=2 iops : min= 1, max= 5, avg= 3.00, stdev= 2.83, samples=2 lat (msec) : 20=8.33%, 50=16.67%, 100=25.00%, 250=50.00% cpu : usr=0.07%, sys=4.82%, ctx=37, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=11.0MiB/s (12.6MB/s), 1226KiB/s-6469KiB/s (1256kB/s-6625kB/s), io=20.0MiB (20.0MB), run=1108-1670msec WRITE: bw=16.8MiB/s (17.6MB/s), 4050KiB/s-6132KiB/s (4147kB/s-6279kB/s), io=28.0MiB (29.4MB), run=1108-1670msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=3513: Fri May 2 00:05:48 2025 read: IOPS=2, BW=10.2MiB/s (10.7MB/s)(12.0MiB/1173msec) clat (msec): min=340, max=474, avg=385.28, stdev=77.23 lat (msec): min=340, max=474, avg=385.29, stdev=77.23 clat percentiles (msec): | 1.00th=[ 342], 5.00th=[ 342], 10.00th=[ 342], 20.00th=[ 342], | 30.00th=[ 342], 40.00th=[ 342], 50.00th=[ 342], 60.00th=[ 342], | 70.00th=[ 477], 80.00th=[ 477], 90.00th=[ 477], 95.00th=[ 477], | 99.00th=[ 477], 99.50th=[ 477], 99.90th=[ 477], 99.95th=[ 477], | 99.99th=[ 477] bw ( KiB/s): min= 8143, max= 8175, per=58.57%, avg=8159.00, stdev=22.63, samples=2 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=2 lat (msec) : 500=100.00% cpu : usr=0.00%, sys=16.04%, ctx=654, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=3514: Fri May 2 00:05:48 2025 read: IOPS=0, BW=3483KiB/s (3567kB/s)(4096KiB/1176msec) clat (nsec): min=378873k, max=378873k, avg=378872739.00, stdev= 0.00 lat (nsec): min=378875k, max=378875k, avg=378875064.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 380], 5.00th=[ 380], 10.00th=[ 380], 20.00th=[ 380], | 30.00th=[ 380], 40.00th=[ 380], 50.00th=[ 380], 60.00th=[ 380], | 70.00th=[ 380], 80.00th=[ 380], 90.00th=[ 380], 95.00th=[ 380], | 99.00th=[ 380], 99.50th=[ 380], 99.90th=[ 380], 99.95th=[ 380], | 99.99th=[ 380] bw ( KiB/s): min= 8192, max= 8192, per=58.80%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=1, BW=6966KiB/s (7133kB/s)(8192KiB/1176msec) clat (msec): min=157, max=544, avg=351.02, stdev=273.34 lat (msec): min=158, max=544, avg=351.31, stdev=273.37 clat percentiles (msec): | 1.00th=[ 159], 5.00th=[ 159], 10.00th=[ 159], 20.00th=[ 159], | 30.00th=[ 159], 40.00th=[ 159], 50.00th=[ 159], 60.00th=[ 542], | 70.00th=[ 542], 80.00th=[ 542], 90.00th=[ 542], 95.00th=[ 542], | 99.00th=[ 542], 99.50th=[ 542], 99.90th=[ 542], 99.95th=[ 542], | 99.99th=[ 542] bw ( KiB/s): min= 8175, max= 8175, per=45.51%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=33.33%, 750=33.33% cpu : usr=0.00%, sys=17.96%, ctx=190, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=3515: Fri May 2 00:05:48 2025 write: IOPS=1, BW=7465KiB/s (7645kB/s)(12.0MiB/1646msec) clat (msec): min=282, max=883, avg=513.36, stdev=323.52 lat (msec): min=283, max=883, avg=513.63, stdev=323.52 clat percentiles (msec): | 1.00th=[ 284], 5.00th=[ 284], 10.00th=[ 284], 20.00th=[ 284], | 30.00th=[ 284], 40.00th=[ 376], 50.00th=[ 376], 60.00th=[ 376], | 70.00th=[ 885], 80.00th=[ 885], 90.00th=[ 885], 95.00th=[ 885], | 99.00th=[ 885], 99.50th=[ 885], 99.90th=[ 885], 99.95th=[ 885], | 99.99th=[ 885] bw ( KiB/s): min= 8175, max= 8192, per=45.55%, avg=8183.50, stdev=12.02, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 500=66.67%, 1000=33.33% cpu : usr=0.00%, sys=14.71%, ctx=71, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=3516: Fri May 2 00:05:48 2025 write: IOPS=1, BW=6737KiB/s (6899kB/s)(12.0MiB/1824msec) clat (msec): min=429, max=846, avg=576.17, stdev=234.47 lat (msec): min=429, max=846, avg=576.45, stdev=234.47 clat percentiles (msec): | 1.00th=[ 430], 5.00th=[ 430], 10.00th=[ 430], 20.00th=[ 430], | 30.00th=[ 430], 40.00th=[ 451], 50.00th=[ 451], 60.00th=[ 451], | 70.00th=[ 844], 80.00th=[ 844], 90.00th=[ 844], 95.00th=[ 844], | 99.00th=[ 844], 99.50th=[ 844], 99.90th=[ 844], 99.95th=[ 844], | 99.99th=[ 844] bw ( KiB/s): min=16384, max=16384, per=91.20%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 500=66.67%, 1000=33.33% cpu : usr=0.11%, sys=15.74%, ctx=68, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=13.6MiB/s (14.3MB/s), 3483KiB/s-10.2MiB/s (3567kB/s-10.7MB/s), io=16.0MiB (16.8MB), run=1173-1176msec WRITE: bw=17.5MiB/s (18.4MB/s), 6737KiB/s-7465KiB/s (6899kB/s-7645kB/s), io=32.0MiB (33.6MB), run=1176-1824msec rand-rw: (groupid=0, jobs=1): err= 0: pid=3508: Fri May 2 00:05:49 2025 read: IOPS=6, BW=27.3MiB/s (28.6MB/s)(12.0MiB/440msec) clat (msec): min=52, max=295, avg=141.40, stdev=133.67 lat (msec): min=52, max=295, avg=141.40, stdev=133.67 clat percentiles (msec): | 1.00th=[ 53], 5.00th=[ 53], 10.00th=[ 53], 20.00th=[ 53], | 30.00th=[ 53], 40.00th=[ 77], 50.00th=[ 77], 60.00th=[ 77], | 70.00th=[ 296], 80.00th=[ 296], 90.00th=[ 296], 95.00th=[ 296], | 99.00th=[ 296], 99.50th=[ 296], 99.90th=[ 296], 99.95th=[ 296], | 99.99th=[ 296] lat (msec) : 100=66.67%, 500=33.33% cpu : usr=0.00%, sys=4.33%, ctx=22, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=3509: Fri May 2 00:05:49 2025 read: IOPS=0, BW=1803KiB/s (1846kB/s)(4096KiB/2272msec) clat (nsec): min=474264k, max=474264k, avg=474263631.00, stdev= 0.00 lat (nsec): min=474270k, max=474270k, avg=474270091.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 477], 5.00th=[ 477], 10.00th=[ 477], 20.00th=[ 477], | 30.00th=[ 477], 40.00th=[ 477], 50.00th=[ 477], 60.00th=[ 477], | 70.00th=[ 477], 80.00th=[ 477], 90.00th=[ 477], 95.00th=[ 477], | 99.00th=[ 477], 99.50th=[ 477], 99.90th=[ 477], 99.95th=[ 477], | 99.99th=[ 477] bw ( KiB/s): min= 8126, max= 8126, per=100.00%, avg=8126.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=0, BW=3606KiB/s (3692kB/s)(8192KiB/2272msec) clat (msec): min=701, max=931, avg=816.81, stdev=162.67 lat (msec): min=702, max=932, avg=817.09, stdev=162.68 clat percentiles (msec): | 1.00th=[ 701], 5.00th=[ 701], 10.00th=[ 701], 20.00th=[ 701], | 30.00th=[ 701], 40.00th=[ 701], 50.00th=[ 701], 60.00th=[ 936], | 70.00th=[ 936], 80.00th=[ 936], 90.00th=[ 936], 95.00th=[ 936], | 99.00th=[ 936], 99.50th=[ 936], 99.90th=[ 936], 99.95th=[ 936], | 99.99th=[ 936] bw ( KiB/s): min= 7984, max= 7984, per=55.36%, avg=7984.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 500=33.33%, 750=33.33%, 1000=33.33% cpu : usr=0.13%, sys=3.21%, ctx=60, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=3510: Fri May 2 00:05:49 2025 write: IOPS=1, BW=5493KiB/s (5625kB/s)(12.0MiB/2237msec) clat (msec): min=236, max=1512, avg=678.62, stdev=722.45 lat (msec): min=236, max=1512, avg=678.92, stdev=722.46 clat percentiles (msec): | 1.00th=[ 236], 5.00th=[ 236], 10.00th=[ 236], 20.00th=[ 236], | 30.00th=[ 236], 40.00th=[ 288], 50.00th=[ 288], 60.00th=[ 288], | 70.00th=[ 1519], 80.00th=[ 1519], 90.00th=[ 1519], 95.00th=[ 1519], | 99.00th=[ 1519], 99.50th=[ 1519], 99.90th=[ 1519], 99.95th=[ 1519], | 99.99th=[ 1519] bw ( KiB/s): min=16094, max=16094, per=100.00%, avg=16094.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=33.33% cpu : usr=0.00%, sys=3.22%, ctx=45, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=3511: Fri May 2 00:05:49 2025 write: IOPS=2, BW=8331KiB/s (8531kB/s)(12.0MiB/1475msec) clat (msec): min=196, max=856, avg=450.20, stdev=355.21 lat (msec): min=196, max=856, avg=450.47, stdev=355.22 clat percentiles (msec): | 1.00th=[ 197], 5.00th=[ 197], 10.00th=[ 197], 20.00th=[ 197], | 30.00th=[ 197], 40.00th=[ 300], 50.00th=[ 300], 60.00th=[ 300], | 70.00th=[ 860], 80.00th=[ 860], 90.00th=[ 860], 95.00th=[ 860], | 99.00th=[ 860], 99.50th=[ 860], 99.90th=[ 860], 99.95th=[ 860], | 99.99th=[ 860] bw ( KiB/s): min= 8192, max= 8192, per=56.80%, avg=8192.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 lat (msec) : 250=33.33%, 500=33.33%, 1000=33.33% cpu : usr=0.07%, sys=3.32%, ctx=35, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=7211KiB/s (7384kB/s), 1803KiB/s-27.3MiB/s (1846kB/s-28.6MB/s), io=16.0MiB (16.8MB), run=440-2272msec WRITE: bw=14.1MiB/s (14.8MB/s), 3606KiB/s-8331KiB/s (3692kB/s-8531kB/s), io=32.0MiB (33.6MB), run=1475-2272msec