== sanity test 398b: DIO and buffer IO race ============== 22:58:49 (1746154729) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 2.90635 s, 17.3 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=22126: Thu May 1 22:59:55 2025 read: IOPS=25, BW=102KiB/s (105kB/s)(5992KiB/58541msec) clat (usec): min=68, max=426816, avg=22101.41, stdev=18316.62 lat (usec): min=69, max=426818, avg=22105.41, stdev=18316.81 clat percentiles (usec): | 1.00th=[ 103], 5.00th=[ 135], 10.00th=[ 7832], 20.00th=[ 11863], | 30.00th=[ 14353], 40.00th=[ 16712], 50.00th=[ 19268], 60.00th=[ 22152], | 70.00th=[ 25822], 80.00th=[ 31065], 90.00th=[ 38536], 95.00th=[ 45876], | 99.00th=[ 76022], 99.50th=[ 85459], 99.90th=[198181], 99.95th=[425722], | 99.99th=[425722] bw ( KiB/s): min= 39, max= 176, per=25.41%, avg=102.65, stdev=28.10, samples=116 iops : min= 9, max= 44, avg=25.31, stdev= 7.09, samples=116 write: IOPS=26, BW=108KiB/s (110kB/s)(6296KiB/58541msec) clat (usec): min=497, max=102293, avg=15986.43, stdev=16926.81 lat (usec): min=498, max=102295, avg=15988.35, stdev=16926.57 clat percentiles (usec): | 1.00th=[ 742], 5.00th=[ 848], 10.00th=[ 988], 20.00th=[ 1287], | 30.00th=[ 2573], 40.00th=[ 5669], 50.00th=[ 11863], 60.00th=[ 15795], | 70.00th=[ 21103], 80.00th=[ 27919], 90.00th=[ 39060], 95.00th=[ 50070], | 99.00th=[ 71828], 99.50th=[ 79168], 99.90th=[ 98042], 99.95th=[102237], | 99.99th=[102237] bw ( KiB/s): min= 39, max= 216, per=25.47%, avg=107.74, stdev=34.71, samples=116 iops : min= 9, max= 54, avg=26.59, stdev= 8.72, samples=116 lat (usec) : 100=0.26%, 250=2.77%, 500=0.49%, 750=0.55%, 1000=4.88% lat (msec) : 2=8.69%, 4=4.49%, 10=8.66%, 20=29.85%, 50=34.93% lat (msec) : 100=4.23%, 250=0.16%, 500=0.03% cpu : usr=0.18%, sys=9.34%, ctx=5203, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22127: Thu May 1 22:59:55 2025 read: IOPS=25, BW=100KiB/s (102kB/s)(5940KiB/59374msec) clat (usec): min=75, max=342623, avg=22121.56, stdev=17071.15 lat (usec): min=75, max=342626, avg=22123.86, stdev=17071.44 clat percentiles (usec): | 1.00th=[ 102], 5.00th=[ 277], 10.00th=[ 8029], 20.00th=[ 10945], | 30.00th=[ 13698], 40.00th=[ 16188], 50.00th=[ 19006], 60.00th=[ 22152], | 70.00th=[ 26346], 80.00th=[ 32113], 90.00th=[ 39584], 95.00th=[ 47449], | 99.00th=[ 72877], 99.50th=[ 91751], 99.90th=[160433], 99.95th=[341836], | 99.99th=[341836] bw ( KiB/s): min= 15, max= 184, per=24.67%, avg=99.68, stdev=28.75, samples=117 iops : min= 3, max= 46, avg=24.60, stdev= 7.26, samples=117 write: IOPS=26, BW=107KiB/s (109kB/s)(6348KiB/59374msec) clat (usec): min=551, max=153142, avg=16523.06, stdev=18024.16 lat (usec): min=552, max=153144, avg=16525.80, stdev=18023.42 clat percentiles (usec): | 1.00th=[ 742], 5.00th=[ 832], 10.00th=[ 955], 20.00th=[ 1287], | 30.00th=[ 2704], 40.00th=[ 5080], 50.00th=[ 12125], 60.00th=[ 16188], | 70.00th=[ 21627], 80.00th=[ 28967], 90.00th=[ 41157], 95.00th=[ 51119], | 99.00th=[ 77071], 99.50th=[ 87557], 99.90th=[128451], 99.95th=[154141], | 99.99th=[154141] bw ( KiB/s): min= 23, max= 280, per=25.10%, avg=106.19, stdev=37.61, samples=117 iops : min= 5, max= 70, avg=26.22, stdev= 9.46, samples=117 lat (usec) : 100=0.39%, 250=2.02%, 500=0.62%, 750=0.75%, 1000=5.76% lat (msec) : 2=7.16%, 4=5.50%, 10=9.15%, 20=29.92%, 50=33.92% lat (msec) : 100=4.43%, 250=0.36%, 500=0.03% cpu : usr=0.15%, sys=9.25%, ctx=5255, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22128: Thu May 1 22:59:55 2025 read: IOPS=25, BW=102KiB/s (104kB/s)(6028KiB/59069msec) clat (usec): min=89, max=444674, avg=22385.07, stdev=18178.44 lat (usec): min=89, max=444676, avg=22394.38, stdev=18178.07 clat percentiles (usec): | 1.00th=[ 109], 5.00th=[ 167], 10.00th=[ 8160], 20.00th=[ 11469], | 30.00th=[ 14091], 40.00th=[ 16319], 50.00th=[ 19268], 60.00th=[ 22938], | 70.00th=[ 26870], 80.00th=[ 31851], 90.00th=[ 39584], 95.00th=[ 47973], | 99.00th=[ 72877], 99.50th=[ 81265], 99.90th=[137364], 99.95th=[446694], | 99.99th=[446694] bw ( KiB/s): min= 8, max= 192, per=25.08%, avg=101.32, stdev=31.06, samples=118 iops : min= 2, max= 48, avg=24.90, stdev= 7.82, samples=118 write: IOPS=26, BW=106KiB/s (109kB/s)(6260KiB/59069msec) clat (usec): min=571, max=120157, avg=16037.65, stdev=17207.96 lat (usec): min=572, max=120158, avg=16041.22, stdev=17207.97 clat percentiles (usec): | 1.00th=[ 758], 5.00th=[ 889], 10.00th=[ 1004], 20.00th=[ 1319], | 30.00th=[ 2737], 40.00th=[ 5735], 50.00th=[ 11600], 60.00th=[ 15533], | 70.00th=[ 19792], 80.00th=[ 27919], 90.00th=[ 41157], 95.00th=[ 51119], | 99.00th=[ 68682], 99.50th=[ 87557], 99.90th=[103285], 99.95th=[120062], | 99.99th=[120062] bw ( KiB/s): min= 24, max= 200, per=24.84%, avg=105.08, stdev=34.50, samples=118 iops : min= 6, max= 50, avg=25.85, stdev= 8.68, samples=118 lat (usec) : 100=0.20%, 250=2.51%, 500=0.36%, 750=0.49%, 1000=4.56% lat (msec) : 2=8.50%, 4=4.39%, 10=10.51%, 20=30.05%, 50=33.72% lat (msec) : 100=4.52%, 250=0.16%, 500=0.03% cpu : usr=0.13%, sys=9.56%, ctx=5285, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22129: Thu May 1 22:59:55 2025 read: IOPS=25, BW=102KiB/s (104kB/s)(6052KiB/59306msec) clat (usec): min=70, max=120471, avg=22282.90, stdev=15524.56 lat (usec): min=70, max=120472, avg=22289.25, stdev=15527.73 clat percentiles (usec): | 1.00th=[ 101], 5.00th=[ 145], 10.00th=[ 7635], 20.00th=[ 11076], | 30.00th=[ 13829], 40.00th=[ 16909], 50.00th=[ 19530], 60.00th=[ 22676], | 70.00th=[ 26346], 80.00th=[ 32113], 90.00th=[ 39584], 95.00th=[ 49546], | 99.00th=[ 83362], 99.50th=[ 95945], 99.90th=[113771], 99.95th=[120062], | 99.99th=[120062] bw ( KiB/s): min= 32, max= 200, per=25.22%, avg=101.89, stdev=29.42, samples=117 iops : min= 8, max= 50, avg=25.15, stdev= 7.39, samples=117 write: IOPS=26, BW=105KiB/s (108kB/s)(6236KiB/59306msec) clat (usec): min=580, max=361403, avg=16239.48, stdev=18629.63 lat (usec): min=581, max=361405, avg=16241.36, stdev=18629.80 clat percentiles (usec): | 1.00th=[ 758], 5.00th=[ 865], 10.00th=[ 1004], 20.00th=[ 1303], | 30.00th=[ 2933], 40.00th=[ 6521], 50.00th=[ 11731], 60.00th=[ 16057], | 70.00th=[ 20841], 80.00th=[ 28443], 90.00th=[ 39584], 95.00th=[ 49021], | 99.00th=[ 66847], 99.50th=[ 79168], 99.90th=[101188], 99.95th=[362808], | 99.99th=[362808] bw ( KiB/s): min= 32, max= 224, per=24.86%, avg=105.14, stdev=33.85, samples=117 iops : min= 8, max= 56, avg=25.97, stdev= 8.53, samples=117 lat (usec) : 100=0.46%, 250=2.44%, 500=0.65%, 750=0.49%, 1000=4.65% lat (msec) : 2=7.55%, 4=4.95%, 10=10.09%, 20=28.78%, 50=35.19% lat (msec) : 100=4.49%, 250=0.23%, 500=0.03% cpu : usr=0.13%, sys=9.41%, ctx=5244, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=404KiB/s (414kB/s), 100KiB/s-102KiB/s (102kB/s-105kB/s), io=23.4MiB (24.6MB), run=58541-59374msec WRITE: bw=423KiB/s (434kB/s), 105KiB/s-108KiB/s (108kB/s-110kB/s), io=24.6MiB (25.7MB), run=58541-59374msec rand-rw: (groupid=0, jobs=1): err= 0: pid=22121: Thu May 1 23:00:16 2025 read: IOPS=18, BW=74.3KiB/s (76.1kB/s)(5992KiB/80656msec) clat (msec): min=3, max=449, avg=19.99, stdev=21.29 lat (msec): min=3, max=449, avg=19.99, stdev=21.29 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 6], 10.00th=[ 7], 20.00th=[ 9], | 30.00th=[ 10], 40.00th=[ 12], 50.00th=[ 13], 60.00th=[ 16], | 70.00th=[ 20], 80.00th=[ 27], 90.00th=[ 44], 95.00th=[ 62], | 99.00th=[ 96], 99.50th=[ 106], 99.90th=[ 133], 99.95th=[ 451], | 99.99th=[ 451] bw ( KiB/s): min= 15, max= 208, per=25.28%, avg=74.58, stdev=46.04, samples=159 iops : min= 3, max= 52, avg=18.31, stdev=11.51, samples=159 write: IOPS=19, BW=78.1KiB/s (79.9kB/s)(6296KiB/80656msec) clat (msec): min=6, max=202, avg=32.05, stdev=22.84 lat (msec): min=6, max=202, avg=32.05, stdev=22.84 clat percentiles (msec): | 1.00th=[ 10], 5.00th=[ 12], 10.00th=[ 14], 20.00th=[ 16], | 30.00th=[ 18], 40.00th=[ 21], 50.00th=[ 23], 60.00th=[ 29], | 70.00th=[ 37], 80.00th=[ 48], 90.00th=[ 63], 95.00th=[ 77], | 99.00th=[ 118], 99.50th=[ 131], 99.90th=[ 169], 99.95th=[ 203], | 99.99th=[ 203] bw ( KiB/s): min= 15, max= 208, per=25.19%, avg=77.85, stdev=46.67, samples=160 iops : min= 3, max= 52, avg=19.13, stdev=11.70, samples=160 lat (msec) : 4=0.16%, 10=16.99%, 20=38.05%, 50=31.61%, 100=11.88% lat (msec) : 250=1.27%, 500=0.03% cpu : usr=0.10%, sys=4.67%, ctx=3522, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22122: Thu May 1 23:00:17 2025 read: IOPS=18, BW=73.1KiB/s (74.9kB/s)(5940KiB/81241msec) clat (msec): min=3, max=366, avg=20.42, stdev=21.69 lat (msec): min=3, max=366, avg=20.42, stdev=21.69 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 6], 10.00th=[ 7], 20.00th=[ 9], | 30.00th=[ 10], 40.00th=[ 11], 50.00th=[ 13], 60.00th=[ 16], | 70.00th=[ 20], 80.00th=[ 28], 90.00th=[ 47], 95.00th=[ 65], | 99.00th=[ 100], 99.50th=[ 110], 99.90th=[ 184], 99.95th=[ 368], | 99.99th=[ 368] bw ( KiB/s): min= 8, max= 216, per=24.83%, avg=73.26, stdev=47.62, samples=160 iops : min= 2, max= 54, avg=17.95, stdev=11.95, samples=160 write: IOPS=19, BW=78.1KiB/s (80.0kB/s)(6348KiB/81241msec) clat (msec): min=5, max=169, avg=31.89, stdev=22.74 lat (msec): min=5, max=169, avg=31.89, stdev=22.74 clat percentiles (msec): | 1.00th=[ 10], 5.00th=[ 12], 10.00th=[ 13], 20.00th=[ 16], | 30.00th=[ 18], 40.00th=[ 20], 50.00th=[ 23], 60.00th=[ 28], | 70.00th=[ 38], 80.00th=[ 48], 90.00th=[ 64], 95.00th=[ 80], | 99.00th=[ 110], 99.50th=[ 126], 99.90th=[ 165], 99.95th=[ 169], | 99.99th=[ 169] bw ( KiB/s): min= 16, max= 240, per=25.14%, avg=77.68, stdev=48.13, samples=161 iops : min= 4, max= 60, avg=19.06, stdev=12.10, samples=161 lat (msec) : 4=0.16%, 10=17.51%, 20=38.22%, 50=30.44%, 100=12.47% lat (msec) : 250=1.17%, 500=0.03% cpu : usr=0.07%, sys=4.50%, ctx=3524, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22123: Thu May 1 23:00:17 2025 read: IOPS=18, BW=74.5KiB/s (76.3kB/s)(6028KiB/80865msec) clat (msec): min=3, max=134, avg=20.12, stdev=18.30 lat (msec): min=3, max=134, avg=20.12, stdev=18.30 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 6], 10.00th=[ 7], 20.00th=[ 9], | 30.00th=[ 10], 40.00th=[ 12], 50.00th=[ 14], 60.00th=[ 17], | 70.00th=[ 20], 80.00th=[ 29], 90.00th=[ 45], 95.00th=[ 60], | 99.00th=[ 87], 99.50th=[ 103], 99.90th=[ 131], 99.95th=[ 136], | 99.99th=[ 136] bw ( KiB/s): min= 7, max= 207, per=25.05%, avg=73.91, stdev=47.17, samples=160 iops : min= 1, max= 51, avg=18.06, stdev=11.86, samples=160 write: IOPS=19, BW=77.4KiB/s (79.3kB/s)(6260KiB/80865msec) clat (msec): min=5, max=432, avg=32.13, stdev=25.18 lat (msec): min=5, max=432, avg=32.13, stdev=25.18 clat percentiles (msec): | 1.00th=[ 9], 5.00th=[ 12], 10.00th=[ 13], 20.00th=[ 16], | 30.00th=[ 18], 40.00th=[ 21], 50.00th=[ 23], 60.00th=[ 29], | 70.00th=[ 37], 80.00th=[ 47], 90.00th=[ 62], 95.00th=[ 77], | 99.00th=[ 122], 99.50th=[ 132], 99.90th=[ 239], 99.95th=[ 435], | 99.99th=[ 435] bw ( KiB/s): min= 7, max= 184, per=24.80%, avg=76.63, stdev=46.28, samples=160 iops : min= 1, max= 46, avg=18.73, stdev=11.68, samples=160 lat (msec) : 4=0.07%, 10=17.61%, 20=36.91%, 50=32.75%, 100=11.46% lat (msec) : 250=1.17%, 500=0.03% cpu : usr=0.08%, sys=4.63%, ctx=3567, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22124: Thu May 1 23:00:17 2025 read: IOPS=18, BW=74.6KiB/s (76.4kB/s)(6052KiB/81159msec) clat (msec): min=2, max=139, avg=20.19, stdev=19.69 lat (msec): min=2, max=139, avg=20.19, stdev=19.69 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 6], 10.00th=[ 7], 20.00th=[ 8], | 30.00th=[ 10], 40.00th=[ 11], 50.00th=[ 13], 60.00th=[ 16], | 70.00th=[ 20], 80.00th=[ 29], 90.00th=[ 45], 95.00th=[ 66], | 99.00th=[ 102], 99.50th=[ 109], 99.90th=[ 123], 99.95th=[ 140], | 99.99th=[ 140] bw ( KiB/s): min= 15, max= 216, per=25.23%, avg=74.43, stdev=48.82, samples=161 iops : min= 3, max= 54, avg=18.24, stdev=12.23, samples=161 write: IOPS=19, BW=76.8KiB/s (78.7kB/s)(6236KiB/81159msec) clat (msec): min=6, max=388, avg=32.28, stdev=24.61 lat (msec): min=6, max=388, avg=32.28, stdev=24.61 clat percentiles (msec): | 1.00th=[ 10], 5.00th=[ 12], 10.00th=[ 13], 20.00th=[ 16], | 30.00th=[ 18], 40.00th=[ 20], 50.00th=[ 23], 60.00th=[ 30], | 70.00th=[ 37], 80.00th=[ 48], 90.00th=[ 64], 95.00th=[ 78], | 99.00th=[ 117], 99.50th=[ 133], 99.90th=[ 236], 99.95th=[ 388], | 99.99th=[ 388] bw ( KiB/s): min= 16, max= 208, per=24.76%, avg=76.52, stdev=47.21, samples=161 iops : min= 4, max= 52, avg=18.76, stdev=11.86, samples=161 lat (msec) : 4=0.16%, 10=18.62%, 20=36.95%, 50=30.79%, 100=11.91% lat (msec) : 250=1.53%, 500=0.03% cpu : usr=0.10%, sys=4.60%, ctx=3541, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=296KiB/s (303kB/s), 73.1KiB/s-74.6KiB/s (74.9kB/s-76.4kB/s), io=23.4MiB (24.6MB), run=80656-81241msec WRITE: bw=309KiB/s (317kB/s), 76.8KiB/s-78.1KiB/s (78.7kB/s-80.0kB/s), io=24.6MiB (25.7MB), run=80656-81241msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=22211: Thu May 1 23:00:36 2025 read: IOPS=22, BW=361KiB/s (370kB/s)(5920KiB/16379msec) clat (usec): min=90, max=128161, avg=23089.75, stdev=17067.85 lat (usec): min=90, max=128163, avg=23090.89, stdev=17067.88 clat percentiles (usec): | 1.00th=[ 100], 5.00th=[ 129], 10.00th=[ 163], 20.00th=[ 10290], | 30.00th=[ 14484], 40.00th=[ 17957], 50.00th=[ 21365], 60.00th=[ 25297], | 70.00th=[ 29230], 80.00th=[ 34341], 90.00th=[ 40109], 95.00th=[ 51643], | 99.00th=[ 77071], 99.50th=[107480], 99.90th=[128451], 99.95th=[128451], | 99.99th=[128451] bw ( KiB/s): min= 190, max= 670, per=26.33%, avg=359.12, stdev=104.69, samples=32 iops : min= 11, max= 41, avg=21.78, stdev= 6.50, samples=32 write: IOPS=24, BW=389KiB/s (398kB/s)(6368KiB/16379msec) clat (usec): min=1007, max=117863, avg=18915.56, stdev=19417.55 lat (usec): min=1008, max=117866, avg=18920.20, stdev=19419.07 clat percentiles (usec): | 1.00th=[ 1139], 5.00th=[ 1287], 10.00th=[ 1467], 20.00th=[ 2311], | 30.00th=[ 5342], 40.00th=[ 7701], 50.00th=[ 12911], 60.00th=[ 17171], | 70.00th=[ 25035], 80.00th=[ 34866], 90.00th=[ 45876], 95.00th=[ 57410], | 99.00th=[ 88605], 99.50th=[ 99091], 99.90th=[117965], 99.95th=[117965], | 99.99th=[117965] bw ( KiB/s): min= 95, max= 766, per=27.18%, avg=389.97, stdev=140.52, samples=32 iops : min= 5, max= 47, avg=23.72, stdev= 8.73, samples=32 lat (usec) : 100=0.26%, 250=5.34%, 500=1.17%, 750=0.13%, 1000=0.13% lat (msec) : 2=9.51%, 4=3.91%, 10=12.11%, 20=23.70%, 50=37.11% lat (msec) : 100=6.25%, 250=0.39% cpu : usr=0.13%, sys=9.15%, ctx=1381, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22214: Thu May 1 23:00:36 2025 read: IOPS=21, BW=344KiB/s (353kB/s)(5744KiB/16683msec) clat (usec): min=83, max=141550, avg=24807.01, stdev=19335.17 lat (usec): min=83, max=141552, avg=24808.14, stdev=19335.24 clat percentiles (usec): | 1.00th=[ 106], 5.00th=[ 124], 10.00th=[ 147], 20.00th=[ 11469], | 30.00th=[ 15008], 40.00th=[ 18744], 50.00th=[ 22414], 60.00th=[ 25822], | 70.00th=[ 30802], 80.00th=[ 35914], 90.00th=[ 46400], 95.00th=[ 55313], | 99.00th=[ 90702], 99.50th=[135267], 99.90th=[141558], 99.95th=[141558], | 99.99th=[141558] bw ( KiB/s): min= 160, max= 510, per=24.91%, avg=339.76, stdev=100.22, samples=33 iops : min= 10, max= 31, avg=20.67, stdev= 6.17, samples=33 write: IOPS=24, BW=392KiB/s (402kB/s)(6544KiB/16683msec) clat (usec): min=1045, max=142947, avg=18752.67, stdev=20545.98 lat (usec): min=1046, max=142949, avg=18755.98, stdev=20545.74 clat percentiles (usec): | 1.00th=[ 1090], 5.00th=[ 1237], 10.00th=[ 1434], 20.00th=[ 1942], | 30.00th=[ 4228], 40.00th=[ 8455], 50.00th=[ 12518], 60.00th=[ 17171], | 70.00th=[ 23200], 80.00th=[ 32375], 90.00th=[ 45876], 95.00th=[ 58983], | 99.00th=[ 80217], 99.50th=[119014], 99.90th=[143655], 99.95th=[143655], | 99.99th=[143655] bw ( KiB/s): min= 223, max= 608, per=26.85%, avg=385.30, stdev=101.72, samples=33 iops : min= 13, max= 38, avg=23.52, stdev= 6.44, samples=33 lat (usec) : 100=0.39%, 250=4.95%, 500=0.78% lat (msec) : 2=11.20%, 4=4.43%, 10=9.64%, 20=24.09%, 50=36.59% lat (msec) : 100=7.16%, 250=0.78% cpu : usr=0.12%, sys=9.26%, ctx=1401, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22216: Thu May 1 23:00:36 2025 read: IOPS=22, BW=356KiB/s (365kB/s)(6256KiB/17559msec) clat (usec): min=73, max=100093, avg=24468.07, stdev=16708.29 lat (usec): min=74, max=100095, avg=24475.39, stdev=16717.84 clat percentiles (usec): | 1.00th=[ 106], 5.00th=[ 122], 10.00th=[ 273], 20.00th=[ 12125], | 30.00th=[ 15401], 40.00th=[ 18482], 50.00th=[ 22152], 60.00th=[ 26608], | 70.00th=[ 30540], 80.00th=[ 36439], 90.00th=[ 46924], 95.00th=[ 54264], | 99.00th=[ 79168], 99.50th=[ 88605], 99.90th=[100140], 99.95th=[100140], | 99.99th=[100140] bw ( KiB/s): min= 128, max= 638, per=25.82%, avg=352.23, stdev=120.80, samples=35 iops : min= 8, max= 39, avg=21.34, stdev= 7.50, samples=35 write: IOPS=21, BW=344KiB/s (352kB/s)(6032KiB/17559msec) clat (usec): min=971, max=108873, avg=20763.48, stdev=21136.20 lat (usec): min=972, max=108874, avg=20765.27, stdev=21136.28 clat percentiles (usec): | 1.00th=[ 1074], 5.00th=[ 1319], 10.00th=[ 1532], 20.00th=[ 2409], | 30.00th=[ 4948], 40.00th=[ 8717], 50.00th=[ 15270], 60.00th=[ 19268], | 70.00th=[ 27132], 80.00th=[ 36439], 90.00th=[ 49021], 95.00th=[ 62653], | 99.00th=[ 96994], 99.50th=[106431], 99.90th=[108528], 99.95th=[108528], | 99.99th=[108528] bw ( KiB/s): min= 128, max= 720, per=23.54%, avg=337.74, stdev=132.96, samples=35 iops : min= 8, max= 45, avg=20.46, stdev= 8.33, samples=35 lat (usec) : 100=0.39%, 250=4.69%, 500=0.91%, 750=0.26%, 1000=0.13% lat (msec) : 2=8.33%, 4=4.95%, 10=8.33%, 20=24.09%, 50=38.93% lat (msec) : 100=8.59%, 250=0.39% cpu : usr=0.09%, sys=9.08%, ctx=1481, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22217: Thu May 1 23:00:36 2025 read: IOPS=22, BW=362KiB/s (370kB/s)(6032KiB/16680msec) clat (usec): min=85, max=119852, avg=24849.14, stdev=19609.98 lat (usec): min=86, max=119853, avg=24850.86, stdev=19609.86 clat percentiles (usec): | 1.00th=[ 99], 5.00th=[ 117], 10.00th=[ 151], 20.00th=[ 10683], | 30.00th=[ 14615], 40.00th=[ 17171], 50.00th=[ 21103], 60.00th=[ 26608], | 70.00th=[ 30278], 80.00th=[ 36963], 90.00th=[ 50070], 95.00th=[ 58459], | 99.00th=[105382], 99.50th=[117965], 99.90th=[120062], 99.95th=[120062], | 99.99th=[120062] bw ( KiB/s): min= 190, max= 544, per=26.13%, avg=356.39, stdev=105.54, samples=33 iops : min= 11, max= 34, avg=21.79, stdev= 6.64, samples=33 write: IOPS=23, BW=375KiB/s (384kB/s)(6256KiB/16680msec) clat (usec): min=956, max=89015, avg=18195.14, stdev=18818.02 lat (usec): min=957, max=89016, avg=18202.55, stdev=18819.89 clat percentiles (usec): | 1.00th=[ 1123], 5.00th=[ 1287], 10.00th=[ 1483], 20.00th=[ 1942], | 30.00th=[ 4686], 40.00th=[ 7570], 50.00th=[12518], 60.00th=[16909], | 70.00th=[22938], 80.00th=[31327], 90.00th=[43779], 95.00th=[56361], | 99.00th=[83362], 99.50th=[88605], 99.90th=[88605], 99.95th=[88605], | 99.99th=[88605] bw ( KiB/s): min= 191, max= 669, per=25.78%, avg=369.94, stdev=114.09, samples=33 iops : min= 11, max= 41, avg=22.64, stdev= 7.22, samples=33 lat (usec) : 100=0.52%, 250=5.86%, 500=0.13%, 1000=0.13% lat (msec) : 2=10.42%, 4=4.56%, 10=10.29%, 20=24.35%, 50=35.55% lat (msec) : 100=7.55%, 250=0.65% cpu : usr=0.12%, sys=9.19%, ctx=1406, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1364KiB/s (1397kB/s), 344KiB/s-362KiB/s (353kB/s-370kB/s), io=23.4MiB (24.5MB), run=16379-17559msec WRITE: bw=1435KiB/s (1470kB/s), 344KiB/s-392KiB/s (352kB/s-402kB/s), io=24.6MiB (25.8MB), run=16379-17559msec rand-rw: (groupid=0, jobs=1): err= 0: pid=22212: Thu May 1 23:00:42 2025 read: IOPS=15, BW=247KiB/s (253kB/s)(5920KiB/23947msec) clat (msec): min=4, max=137, avg=22.78, stdev=20.27 lat (msec): min=4, max=137, avg=22.79, stdev=20.27 clat percentiles (msec): | 1.00th=[ 6], 5.00th=[ 8], 10.00th=[ 9], 20.00th=[ 10], | 30.00th=[ 12], 40.00th=[ 14], 50.00th=[ 16], 60.00th=[ 18], | 70.00th=[ 22], 80.00th=[ 31], 90.00th=[ 47], 95.00th=[ 67], | 99.00th=[ 104], 99.50th=[ 122], 99.90th=[ 138], 99.95th=[ 138], | 99.99th=[ 138] bw ( KiB/s): min= 64, max= 606, per=23.95%, avg=238.53, stdev=131.87, samples=47 iops : min= 4, max= 37, avg=14.43, stdev= 8.26, samples=47 write: IOPS=16, BW=266KiB/s (272kB/s)(6368KiB/23947msec) clat (msec): min=8, max=158, avg=38.30, stdev=26.13 lat (msec): min=8, max=158, avg=38.30, stdev=26.13 clat percentiles (msec): | 1.00th=[ 11], 5.00th=[ 14], 10.00th=[ 17], 20.00th=[ 19], | 30.00th=[ 21], 40.00th=[ 25], 50.00th=[ 28], 60.00th=[ 34], | 70.00th=[ 44], 80.00th=[ 57], 90.00th=[ 79], 95.00th=[ 95], | 99.00th=[ 130], 99.50th=[ 138], 99.90th=[ 159], 99.95th=[ 159], | 99.99th=[ 159] bw ( KiB/s): min= 64, max= 606, per=24.97%, avg=261.64, stdev=150.65, samples=47 iops : min= 4, max= 37, avg=15.87, stdev= 9.46, samples=47 lat (msec) : 10=10.16%, 20=35.42%, 50=37.76%, 100=14.32%, 250=2.34% cpu : usr=0.04%, sys=4.24%, ctx=904, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22213: Thu May 1 23:00:42 2025 read: IOPS=15, BW=241KiB/s (247kB/s)(5744KiB/23805msec) clat (msec): min=4, max=145, avg=23.86, stdev=21.49 lat (msec): min=4, max=145, avg=23.87, stdev=21.49 clat percentiles (msec): | 1.00th=[ 6], 5.00th=[ 8], 10.00th=[ 8], 20.00th=[ 11], | 30.00th=[ 12], 40.00th=[ 14], 50.00th=[ 17], 60.00th=[ 19], | 70.00th=[ 24], 80.00th=[ 32], 90.00th=[ 54], 95.00th=[ 72], | 99.00th=[ 111], 99.50th=[ 132], 99.90th=[ 146], 99.95th=[ 146], | 99.99th=[ 146] bw ( KiB/s): min= 63, max= 576, per=24.46%, avg=243.61, stdev=145.64, samples=46 iops : min= 3, max= 36, avg=14.72, stdev= 9.14, samples=46 write: IOPS=17, BW=275KiB/s (281kB/s)(6544KiB/23805msec) clat (msec): min=5, max=194, avg=36.72, stdev=24.73 lat (msec): min=5, max=194, avg=36.73, stdev=24.73 clat percentiles (msec): | 1.00th=[ 12], 5.00th=[ 15], 10.00th=[ 17], 20.00th=[ 19], | 30.00th=[ 22], 40.00th=[ 26], 50.00th=[ 29], 60.00th=[ 33], | 70.00th=[ 41], 80.00th=[ 55], 90.00th=[ 70], 95.00th=[ 87], | 99.00th=[ 121], 99.50th=[ 127], 99.90th=[ 194], 99.95th=[ 194], | 99.99th=[ 194] bw ( KiB/s): min= 125, max= 672, per=25.80%, avg=270.43, stdev=144.56, samples=47 iops : min= 7, max= 42, avg=16.38, stdev= 9.09, samples=47 lat (msec) : 10=8.85%, 20=35.03%, 50=39.19%, 100=14.71%, 250=2.21% cpu : usr=0.10%, sys=4.15%, ctx=918, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22215: Thu May 1 23:00:42 2025 read: IOPS=16, BW=264KiB/s (271kB/s)(6256KiB/23667msec) clat (msec): min=3, max=141, avg=23.02, stdev=21.90 lat (msec): min=3, max=141, avg=23.02, stdev=21.90 clat percentiles (msec): | 1.00th=[ 6], 5.00th=[ 7], 10.00th=[ 8], 20.00th=[ 10], | 30.00th=[ 12], 40.00th=[ 14], 50.00th=[ 16], 60.00th=[ 19], | 70.00th=[ 23], 80.00th=[ 28], 90.00th=[ 52], 95.00th=[ 74], | 99.00th=[ 120], 99.50th=[ 132], 99.90th=[ 142], 99.95th=[ 142], | 99.99th=[ 142] bw ( KiB/s): min= 61, max= 828, per=26.54%, avg=264.32, stdev=169.30, samples=47 iops : min= 3, max= 51, avg=15.96, stdev=10.60, samples=47 write: IOPS=15, BW=255KiB/s (261kB/s)(6032KiB/23667msec) clat (msec): min=7, max=192, avg=38.70, stdev=27.45 lat (msec): min=7, max=192, avg=38.71, stdev=27.45 clat percentiles (msec): | 1.00th=[ 11], 5.00th=[ 14], 10.00th=[ 17], 20.00th=[ 20], | 30.00th=[ 23], 40.00th=[ 26], 50.00th=[ 28], 60.00th=[ 34], | 70.00th=[ 42], 80.00th=[ 54], 90.00th=[ 79], 95.00th=[ 99], | 99.00th=[ 133], 99.50th=[ 163], 99.90th=[ 192], 99.95th=[ 192], | 99.99th=[ 192] bw ( KiB/s): min= 93, max= 576, per=24.00%, avg=251.47, stdev=125.94, samples=47 iops : min= 5, max= 36, avg=15.15, stdev= 7.94, samples=47 lat (msec) : 4=0.13%, 10=12.11%, 20=30.21%, 50=40.62%, 100=13.67% lat (msec) : 250=3.26% cpu : usr=0.16%, sys=4.20%, ctx=917, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22218: Thu May 1 23:00:42 2025 read: IOPS=15, BW=251KiB/s (257kB/s)(6032KiB/24033msec) clat (msec): min=5, max=164, avg=24.35, stdev=21.69 lat (msec): min=5, max=164, avg=24.35, stdev=21.70 clat percentiles (msec): | 1.00th=[ 7], 5.00th=[ 8], 10.00th=[ 9], 20.00th=[ 11], | 30.00th=[ 13], 40.00th=[ 15], 50.00th=[ 17], 60.00th=[ 20], | 70.00th=[ 25], 80.00th=[ 34], 90.00th=[ 53], 95.00th=[ 70], | 99.00th=[ 123], 99.50th=[ 142], 99.90th=[ 165], 99.95th=[ 165], | 99.99th=[ 165] bw ( KiB/s): min= 93, max= 638, per=25.39%, avg=252.87, stdev=139.98, samples=47 iops : min= 5, max= 39, avg=15.26, stdev= 8.70, samples=47 write: IOPS=16, BW=260KiB/s (267kB/s)(6256KiB/24033msec) clat (msec): min=8, max=158, avg=37.21, stdev=24.90 lat (msec): min=8, max=158, avg=37.22, stdev=24.90 clat percentiles (msec): | 1.00th=[ 10], 5.00th=[ 14], 10.00th=[ 17], 20.00th=[ 20], | 30.00th=[ 22], 40.00th=[ 25], 50.00th=[ 29], 60.00th=[ 34], | 70.00th=[ 42], 80.00th=[ 54], 90.00th=[ 71], 95.00th=[ 92], | 99.00th=[ 129], 99.50th=[ 136], 99.90th=[ 159], 99.95th=[ 159], | 99.99th=[ 159] bw ( KiB/s): min= 94, max= 608, per=24.78%, avg=259.67, stdev=141.26, samples=48 iops : min= 5, max= 38, avg=15.67, stdev= 8.87, samples=48 lat (msec) : 10=9.90%, 20=31.64%, 50=41.41%, 100=14.32%, 250=2.73% cpu : usr=0.11%, sys=4.12%, ctx=896, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=997KiB/s (1021kB/s), 241KiB/s-264KiB/s (247kB/s-271kB/s), io=23.4MiB (24.5MB), run=23667-24033msec WRITE: bw=1049KiB/s (1074kB/s), 255KiB/s-275KiB/s (261kB/s-281kB/s), io=24.6MiB (25.8MB), run=23667-24033msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=22248: Thu May 1 23:00:47 2025 read: IOPS=3, BW=4075KiB/s (4173kB/s)(7168KiB/1759msec) clat (usec): min=858, max=337467, avg=164771.07, stdev=120351.27 lat (usec): min=860, max=337468, avg=164774.89, stdev=120350.98 clat percentiles (usec): | 1.00th=[ 857], 5.00th=[ 857], 10.00th=[ 857], 20.00th=[ 82314], | 30.00th=[130548], 40.00th=[130548], 50.00th=[141558], 60.00th=[147850], | 70.00th=[147850], 80.00th=[312476], 90.00th=[337642], 95.00th=[337642], | 99.00th=[337642], 99.50th=[337642], 99.90th=[337642], 99.95th=[337642], | 99.99th=[337642] bw ( KiB/s): min= 2043, max= 6131, per=56.20%, avg=4084.33, stdev=2044.01, samples=3 iops : min= 1, max= 5, avg= 3.00, stdev= 2.00, samples=3 write: IOPS=2, BW=2911KiB/s (2981kB/s)(5120KiB/1759msec) clat (msec): min=59, max=136, avg=100.06, stdev=35.44 lat (msec): min=59, max=136, avg=103.50, stdev=37.12 clat percentiles (msec): | 1.00th=[ 60], 5.00th=[ 60], 10.00th=[ 60], 20.00th=[ 60], | 30.00th=[ 67], 40.00th=[ 67], 50.00th=[ 109], 60.00th=[ 109], | 70.00th=[ 130], 80.00th=[ 130], 90.00th=[ 138], 95.00th=[ 138], | 99.00th=[ 138], 99.50th=[ 138], 99.90th=[ 138], 99.95th=[ 138], | 99.99th=[ 138] bw ( KiB/s): min= 2039, max= 8175, per=50.20%, avg=5107.00, stdev=4338.81, samples=2 iops : min= 1, max= 7, avg= 4.00, stdev= 4.24, samples=2 lat (usec) : 1000=8.33% lat (msec) : 100=25.00%, 250=50.00%, 500=16.67% cpu : usr=0.00%, sys=18.89%, ctx=252, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22249: Thu May 1 23:00:47 2025 read: IOPS=0, BW=727KiB/s (744kB/s)(2048KiB/2818msec) clat (msec): min=86, max=110, avg=98.51, stdev=17.17 lat (msec): min=86, max=110, avg=98.52, stdev=17.17 clat percentiles (msec): | 1.00th=[ 87], 5.00th=[ 87], 10.00th=[ 87], 20.00th=[ 87], | 30.00th=[ 87], 40.00th=[ 87], 50.00th=[ 87], 60.00th=[ 111], | 70.00th=[ 111], 80.00th=[ 111], 90.00th=[ 111], 95.00th=[ 111], | 99.00th=[ 111], 99.50th=[ 111], 99.90th=[ 111], 99.95th=[ 111], | 99.99th=[ 111] bw ( KiB/s): min= 1917, max= 2048, per=27.28%, avg=1982.50, stdev=92.63, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 write: IOPS=3, BW=3634KiB/s (3721kB/s)(10.0MiB/2818msec) clat (msec): min=107, max=507, avg=225.37, stdev=123.26 lat (msec): min=107, max=507, avg=225.44, stdev=123.26 clat percentiles (msec): | 1.00th=[ 108], 5.00th=[ 108], 10.00th=[ 108], 20.00th=[ 121], | 30.00th=[ 127], 40.00th=[ 138], 50.00th=[ 190], 60.00th=[ 203], | 70.00th=[ 275], 80.00th=[ 275], 90.00th=[ 321], 95.00th=[ 506], | 99.00th=[ 506], 99.50th=[ 506], 99.90th=[ 506], 99.95th=[ 506], | 99.99th=[ 506] bw ( KiB/s): min= 2043, max= 5495, per=37.99%, avg=3865.00, stdev=1417.36, samples=4 iops : min= 1, max= 5, avg= 3.00, stdev= 1.63, samples=4 lat (msec) : 100=8.33%, 250=58.33%, 500=25.00%, 750=8.33% cpu : usr=0.00%, sys=13.67%, ctx=110, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22250: Thu May 1 23:00:47 2025 read: IOPS=3, BW=3733KiB/s (3822kB/s)(6144KiB/1646msec) clat (usec): min=771, max=211391, avg=113364.64, stdev=90860.99 lat (usec): min=772, max=211393, avg=113366.31, stdev=90861.09 clat percentiles (usec): | 1.00th=[ 775], 5.00th=[ 775], 10.00th=[ 775], 20.00th=[ 1057], | 30.00th=[ 1057], 40.00th=[141558], 50.00th=[141558], 60.00th=[145753], | 70.00th=[181404], 80.00th=[181404], 90.00th=[210764], 95.00th=[210764], | 99.00th=[210764], 99.50th=[210764], 99.90th=[210764], 99.95th=[210764], | 99.99th=[210764] bw ( KiB/s): min= 4055, max= 6144, per=70.17%, avg=5099.50, stdev=1477.15, samples=2 iops : min= 3, max= 6, avg= 4.50, stdev= 2.12, samples=2 write: IOPS=3, BW=3733KiB/s (3822kB/s)(6144KiB/1646msec) clat (msec): min=51, max=262, avg=131.63, stdev=75.57 lat (msec): min=51, max=263, avg=131.76, stdev=75.69 clat percentiles (msec): | 1.00th=[ 53], 5.00th=[ 53], 10.00th=[ 53], 20.00th=[ 82], | 30.00th=[ 82], 40.00th=[ 111], 50.00th=[ 111], 60.00th=[ 112], | 70.00th=[ 171], 80.00th=[ 171], 90.00th=[ 264], 95.00th=[ 264], | 99.00th=[ 264], 99.50th=[ 264], 99.90th=[ 264], 99.95th=[ 264], | 99.99th=[ 264] bw ( KiB/s): min= 2027, max= 6144, per=40.13%, avg=4083.33, stdev=2058.50, samples=3 iops : min= 1, max= 6, avg= 3.33, stdev= 2.52, samples=3 lat (usec) : 1000=8.33% lat (msec) : 2=8.33%, 100=16.67%, 250=58.33%, 500=8.33% cpu : usr=0.00%, sys=15.50%, ctx=335, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22251: Thu May 1 23:00:47 2025 read: IOPS=2, BW=2374KiB/s (2431kB/s)(5120KiB/2157msec) clat (usec): min=595, max=116908, avg=43972.26, stdev=59564.05 lat (usec): min=595, max=116909, avg=43973.34, stdev=59564.06 clat percentiles (usec): | 1.00th=[ 594], 5.00th=[ 594], 10.00th=[ 594], 20.00th=[ 594], | 30.00th=[ 660], 40.00th=[ 660], 50.00th=[ 758], 60.00th=[ 758], | 70.00th=[101188], 80.00th=[101188], 90.00th=[116917], 95.00th=[116917], | 99.00th=[116917], 99.50th=[116917], 99.90th=[116917], 99.95th=[116917], | 99.99th=[116917] bw ( KiB/s): min= 4063, max= 4087, per=56.08%, avg=4075.00, stdev=16.97, samples=2 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=2 write: IOPS=3, BW=3323KiB/s (3403kB/s)(7168KiB/2157msec) clat (msec): min=79, max=665, avg=229.32, stdev=214.18 lat (msec): min=79, max=665, avg=229.40, stdev=214.17 clat percentiles (msec): | 1.00th=[ 80], 5.00th=[ 80], 10.00th=[ 80], 20.00th=[ 87], | 30.00th=[ 87], 40.00th=[ 87], 50.00th=[ 111], 60.00th=[ 275], | 70.00th=[ 275], 80.00th=[ 305], 90.00th=[ 667], 95.00th=[ 667], | 99.00th=[ 667], 99.50th=[ 667], 99.90th=[ 667], 99.95th=[ 667], | 99.99th=[ 667] bw ( KiB/s): min= 2043, max= 6095, per=33.37%, avg=3395.33, stdev=2337.98, samples=3 iops : min= 1, max= 5, avg= 2.67, stdev= 2.08, samples=3 lat (usec) : 750=16.67%, 1000=8.33% lat (msec) : 100=25.00%, 250=25.00%, 500=16.67%, 750=8.33% cpu : usr=0.00%, sys=10.99%, ctx=140, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=7268KiB/s (7442kB/s), 727KiB/s-4075KiB/s (744kB/s-4173kB/s), io=20.0MiB (20.0MB), run=1646-2818msec WRITE: bw=9.94MiB/s (10.4MB/s), 2911KiB/s-3733KiB/s (2981kB/s-3822kB/s), io=28.0MiB (29.4MB), run=1646-2818msec rand-rw: (groupid=0, jobs=1): err= 0: pid=22244: Thu May 1 23:00:48 2025 read: IOPS=3, BW=3233KiB/s (3311kB/s)(7168KiB/2217msec) clat (msec): min=33, max=511, avg=194.47, stdev=196.53 lat (msec): min=33, max=511, avg=194.47, stdev=196.53 clat percentiles (msec): | 1.00th=[ 34], 5.00th=[ 34], 10.00th=[ 34], 20.00th=[ 71], | 30.00th=[ 81], 40.00th=[ 81], 50.00th=[ 86], 60.00th=[ 136], | 70.00th=[ 136], 80.00th=[ 443], 90.00th=[ 510], 95.00th=[ 510], | 99.00th=[ 510], 99.50th=[ 510], 99.90th=[ 510], 99.95th=[ 510], | 99.99th=[ 510] bw ( KiB/s): min= 2007, max= 4039, per=37.71%, avg=2535.50, stdev=1002.52, samples=4 iops : min= 1, max= 3, avg= 2.00, stdev= 0.82, samples=4 write: IOPS=2, BW=2309KiB/s (2365kB/s)(5120KiB/2217msec) clat (msec): min=33, max=206, avg=97.12, stdev=72.80 lat (msec): min=33, max=206, avg=97.66, stdev=72.44 clat percentiles (msec): | 1.00th=[ 34], 5.00th=[ 34], 10.00th=[ 34], 20.00th=[ 34], | 30.00th=[ 41], 40.00th=[ 41], 50.00th=[ 72], 60.00th=[ 72], | 70.00th=[ 134], 80.00th=[ 134], 90.00th=[ 207], 95.00th=[ 207], | 99.00th=[ 207], 99.50th=[ 207], 99.90th=[ 207], 99.95th=[ 207], | 99.99th=[ 207] bw ( KiB/s): min= 2048, max= 4039, per=32.33%, avg=3043.50, stdev=1407.85, samples=2 iops : min= 2, max= 3, avg= 2.50, stdev= 0.71, samples=2 lat (msec) : 50=25.00%, 100=33.33%, 250=25.00%, 500=8.33%, 750=8.33% cpu : usr=0.00%, sys=3.93%, ctx=41, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22245: Thu May 1 23:00:48 2025 read: IOPS=0, BW=672KiB/s (688kB/s)(2048KiB/3046msec) clat (msec): min=63, max=127, avg=95.80, stdev=45.44 lat (msec): min=63, max=127, avg=95.80, stdev=45.44 clat percentiles (msec): | 1.00th=[ 64], 5.00th=[ 64], 10.00th=[ 64], 20.00th=[ 64], | 30.00th=[ 64], 40.00th=[ 64], 50.00th=[ 64], 60.00th=[ 129], | 70.00th=[ 129], 80.00th=[ 129], 90.00th=[ 129], 95.00th=[ 129], | 99.00th=[ 129], 99.50th=[ 129], 99.90th=[ 129], 99.95th=[ 129], | 99.99th=[ 129] bw ( KiB/s): min= 1984, max= 2048, per=29.99%, avg=2016.00, stdev=45.25, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 write: IOPS=3, BW=3362KiB/s (3442kB/s)(10.0MiB/3046msec) clat (msec): min=124, max=675, avg=282.07, stdev=167.91 lat (msec): min=124, max=675, avg=282.15, stdev=167.92 clat percentiles (msec): | 1.00th=[ 125], 5.00th=[ 125], 10.00th=[ 125], 20.00th=[ 128], | 30.00th=[ 144], 40.00th=[ 201], 50.00th=[ 230], 60.00th=[ 259], | 70.00th=[ 376], 80.00th=[ 376], 90.00th=[ 393], 95.00th=[ 676], | 99.00th=[ 676], 99.50th=[ 676], 99.90th=[ 676], 99.95th=[ 676], | 99.99th=[ 676] bw ( KiB/s): min= 1984, max= 6083, per=38.36%, avg=3610.60, stdev=2183.01, samples=5 iops : min= 1, max= 5, avg= 2.80, stdev= 2.05, samples=5 lat (msec) : 100=8.33%, 250=50.00%, 500=33.33%, 750=8.33% cpu : usr=0.03%, sys=6.40%, ctx=80, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22246: Thu May 1 23:00:48 2025 read: IOPS=2, BW=2489KiB/s (2549kB/s)(6144KiB/2468msec) clat (msec): min=34, max=377, avg=136.94, stdev=131.21 lat (msec): min=34, max=377, avg=136.94, stdev=131.21 clat percentiles (msec): | 1.00th=[ 34], 5.00th=[ 34], 10.00th=[ 34], 20.00th=[ 47], | 30.00th=[ 47], 40.00th=[ 50], 50.00th=[ 50], 60.00th=[ 138], | 70.00th=[ 178], 80.00th=[ 178], 90.00th=[ 380], 95.00th=[ 380], | 99.00th=[ 380], 99.50th=[ 380], 99.90th=[ 380], 99.95th=[ 380], | 99.99th=[ 380] bw ( KiB/s): min= 4047, max= 4047, per=60.20%, avg=4047.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 write: IOPS=2, BW=2489KiB/s (2549kB/s)(6144KiB/2468msec) clat (msec): min=43, max=421, avg=213.94, stdev=152.63 lat (msec): min=44, max=421, avg=214.06, stdev=152.59 clat percentiles (msec): | 1.00th=[ 44], 5.00th=[ 44], 10.00th=[ 44], 20.00th=[ 49], | 30.00th=[ 49], 40.00th=[ 205], 50.00th=[ 205], 60.00th=[ 222], | 70.00th=[ 342], 80.00th=[ 342], 90.00th=[ 422], 95.00th=[ 422], | 99.00th=[ 422], 99.50th=[ 422], 99.90th=[ 422], 99.95th=[ 422], | 99.99th=[ 422] bw ( KiB/s): min= 2027, max= 4031, per=32.18%, avg=3029.00, stdev=1417.04, samples=2 iops : min= 1, max= 3, avg= 2.00, stdev= 1.41, samples=2 lat (msec) : 50=41.67%, 250=33.33%, 500=25.00% cpu : usr=0.00%, sys=4.78%, ctx=55, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22247: Thu May 1 23:00:48 2025 read: IOPS=2, BW=2862KiB/s (2931kB/s)(5120KiB/1789msec) clat (msec): min=58, max=169, avg=104.16, stdev=47.44 lat (msec): min=58, max=169, avg=104.16, stdev=47.44 clat percentiles (msec): | 1.00th=[ 58], 5.00th=[ 58], 10.00th=[ 58], 20.00th=[ 58], | 30.00th=[ 66], 40.00th=[ 66], 50.00th=[ 93], 60.00th=[ 93], | 70.00th=[ 136], 80.00th=[ 136], 90.00th=[ 171], 95.00th=[ 171], | 99.00th=[ 171], 99.50th=[ 171], 99.90th=[ 171], 99.95th=[ 171], | 99.99th=[ 171] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=3, BW=4007KiB/s (4103kB/s)(7168KiB/1789msec) clat (msec): min=34, max=311, avg=170.43, stdev=121.17 lat (msec): min=34, max=311, avg=170.51, stdev=121.17 clat percentiles (msec): | 1.00th=[ 35], 5.00th=[ 35], 10.00th=[ 35], 20.00th=[ 46], | 30.00th=[ 77], 40.00th=[ 77], 50.00th=[ 167], 60.00th=[ 255], | 70.00th=[ 255], 80.00th=[ 305], 90.00th=[ 313], 95.00th=[ 313], | 99.00th=[ 313], 99.50th=[ 313], 99.90th=[ 313], 99.95th=[ 313], | 99.99th=[ 313] bw ( KiB/s): min= 2048, max= 6023, per=42.87%, avg=4035.50, stdev=2810.75, samples=2 iops : min= 2, max= 5, avg= 3.50, stdev= 2.12, samples=2 lat (msec) : 50=16.67%, 100=33.33%, 250=25.00%, 500=25.00% cpu : usr=0.00%, sys=3.86%, ctx=41, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=6724KiB/s (6885kB/s), 672KiB/s-3233KiB/s (688kB/s-3311kB/s), io=20.0MiB (20.0MB), run=1789-3046msec WRITE: bw=9413KiB/s (9639kB/s), 2309KiB/s-4007KiB/s (2365kB/s-4103kB/s), io=28.0MiB (29.4MB), run=1789-3046msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=22260: Thu May 1 23:00:52 2025 read: IOPS=3, BW=12.4MiB/s (13.0MB/s)(12.0MiB/965msec) clat (msec): min=3, max=636, avg=269.85, stdev=328.14 lat (msec): min=3, max=636, avg=269.85, stdev=328.14 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 4], 10.00th=[ 4], 20.00th=[ 4], | 30.00th=[ 4], 40.00th=[ 169], 50.00th=[ 169], 60.00th=[ 169], | 70.00th=[ 634], 80.00th=[ 634], 90.00th=[ 634], 95.00th=[ 634], | 99.00th=[ 634], 99.50th=[ 634], 99.90th=[ 634], 99.95th=[ 634], | 99.99th=[ 634] bw ( KiB/s): min= 7937, max= 7937, per=100.00%, avg=7937.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 4=33.33%, 250=33.33%, 750=33.33% cpu : usr=0.00%, sys=13.07%, ctx=184, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22261: Thu May 1 23:00:52 2025 read: IOPS=0, BW=1655KiB/s (1695kB/s)(4096KiB/2475msec) clat (nsec): min=167667k, max=167667k, avg=167666704.00, stdev= 0.00 lat (nsec): min=167668k, max=167668k, avg=167668351.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 167], 5.00th=[ 167], 10.00th=[ 167], 20.00th=[ 167], | 30.00th=[ 167], 40.00th=[ 167], 50.00th=[ 167], 60.00th=[ 167], | 70.00th=[ 167], 80.00th=[ 167], 90.00th=[ 167], 95.00th=[ 167], | 99.00th=[ 167], 99.50th=[ 167], 99.90th=[ 167], 99.95th=[ 167], | 99.99th=[ 167] bw ( KiB/s): min= 8175, max= 8175, per=100.00%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=0, BW=3310KiB/s (3389kB/s)(8192KiB/2475msec) clat (msec): min=624, max=1469, avg=1046.94, stdev=597.89 lat (msec): min=626, max=1470, avg=1048.43, stdev=597.39 clat percentiles (msec): | 1.00th=[ 625], 5.00th=[ 625], 10.00th=[ 625], 20.00th=[ 625], | 30.00th=[ 625], 40.00th=[ 625], 50.00th=[ 625], 60.00th=[ 1469], | 70.00th=[ 1469], 80.00th=[ 1469], 90.00th=[ 1469], 95.00th=[ 1469], | 99.00th=[ 1469], 99.50th=[ 1469], 99.90th=[ 1469], 99.95th=[ 1469], | 99.99th=[ 1469] bw ( KiB/s): min= 8126, max= 8126, per=61.38%, avg=8126.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 750=33.33% cpu : usr=0.04%, sys=11.03%, ctx=90, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22262: Thu May 1 23:00:52 2025 write: IOPS=1, BW=6958KiB/s (7125kB/s)(12.0MiB/1766msec) clat (msec): min=380, max=775, avg=560.79, stdev=200.04 lat (msec): min=386, max=776, avg=563.31, stdev=197.48 clat percentiles (msec): | 1.00th=[ 380], 5.00th=[ 380], 10.00th=[ 380], 20.00th=[ 380], | 30.00th=[ 380], 40.00th=[ 527], 50.00th=[ 527], 60.00th=[ 527], | 70.00th=[ 776], 80.00th=[ 776], 90.00th=[ 776], 95.00th=[ 776], | 99.00th=[ 776], 99.50th=[ 776], 99.90th=[ 776], 99.95th=[ 776], | 99.99th=[ 776] bw ( KiB/s): min= 8175, max= 8192, per=61.81%, avg=8183.50, stdev=12.02, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 500=33.33%, 750=33.33%, 1000=33.33% cpu : usr=0.00%, sys=17.51%, ctx=76, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22265: Thu May 1 23:00:52 2025 write: IOPS=1, BW=5877KiB/s (6018kB/s)(12.0MiB/2091msec) clat (msec): min=358, max=1121, avg=659.08, stdev=406.12 lat (msec): min=358, max=1121, avg=659.39, stdev=406.25 clat percentiles (msec): | 1.00th=[ 359], 5.00th=[ 359], 10.00th=[ 359], 20.00th=[ 359], | 30.00th=[ 359], 40.00th=[ 498], 50.00th=[ 498], 60.00th=[ 498], | 70.00th=[ 1116], 80.00th=[ 1116], 90.00th=[ 1116], 95.00th=[ 1116], | 99.00th=[ 1116], 99.50th=[ 1116], 99.90th=[ 1116], 99.95th=[ 1116], | 99.99th=[ 1116] bw ( KiB/s): min= 8192, max= 8192, per=61.88%, avg=8192.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 lat (msec) : 500=66.67% cpu : usr=0.00%, sys=18.90%, ctx=80, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=6620KiB/s (6779kB/s), 1655KiB/s-12.4MiB/s (1695kB/s-13.0MB/s), io=16.0MiB (16.8MB), run=965-2475msec WRITE: bw=12.9MiB/s (13.6MB/s), 3310KiB/s-6958KiB/s (3389kB/s-7125kB/s), io=32.0MiB (33.6MB), run=1766-2475msec rand-rw: (groupid=0, jobs=1): err= 0: pid=22263: Thu May 1 23:00:53 2025 read: IOPS=1, BW=5804KiB/s (5944kB/s)(12.0MiB/2117msec) clat (msec): min=414, max=844, avg=665.19, stdev=223.72 lat (msec): min=414, max=844, avg=665.20, stdev=223.72 clat percentiles (msec): | 1.00th=[ 414], 5.00th=[ 414], 10.00th=[ 414], 20.00th=[ 414], | 30.00th=[ 414], 40.00th=[ 735], 50.00th=[ 735], 60.00th=[ 735], | 70.00th=[ 844], 80.00th=[ 844], 90.00th=[ 844], 95.00th=[ 844], | 99.00th=[ 844], 99.50th=[ 844], 99.90th=[ 844], 99.95th=[ 844], | 99.99th=[ 844] bw ( KiB/s): min= 5826, max= 5826, per=98.08%, avg=5826.00, stdev= 0.00, samples=2 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=2 lat (msec) : 500=33.33%, 750=33.33%, 1000=33.33% cpu : usr=0.09%, sys=3.12%, ctx=35, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22264: Thu May 1 23:00:53 2025 read: IOPS=0, BW=1485KiB/s (1521kB/s)(4096KiB/2758msec) clat (nsec): min=572598k, max=572598k, avg=572597737.00, stdev= 0.00 lat (nsec): min=572601k, max=572601k, avg=572601422.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 575], 5.00th=[ 575], 10.00th=[ 575], 20.00th=[ 575], | 30.00th=[ 575], 40.00th=[ 575], 50.00th=[ 575], 60.00th=[ 575], | 70.00th=[ 575], 80.00th=[ 575], 90.00th=[ 575], 95.00th=[ 575], | 99.00th=[ 575], 99.50th=[ 575], 99.90th=[ 575], 99.95th=[ 575], | 99.99th=[ 575] bw ( KiB/s): min= 6272, max= 6272, per=100.00%, avg=6272.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=0, BW=2970KiB/s (3042kB/s)(8192KiB/2758msec) clat (msec): min=807, max=1209, avg=1008.49, stdev=283.82 lat (msec): min=808, max=1209, avg=1009.12, stdev=283.55 clat percentiles (msec): | 1.00th=[ 810], 5.00th=[ 810], 10.00th=[ 810], 20.00th=[ 810], | 30.00th=[ 810], 40.00th=[ 810], 50.00th=[ 810], 60.00th=[ 1217], | 70.00th=[ 1217], 80.00th=[ 1217], 90.00th=[ 1217], 95.00th=[ 1217], | 99.00th=[ 1217], 99.50th=[ 1217], 99.90th=[ 1217], 99.95th=[ 1217], | 99.99th=[ 1217] bw ( KiB/s): min= 6282, max= 6282, per=56.00%, avg=6282.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 750=33.33%, 1000=33.33% cpu : usr=0.04%, sys=7.29%, ctx=80, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22266: Thu May 1 23:00:53 2025 write: IOPS=1, BW=4207KiB/s (4308kB/s)(12.0MiB/2921msec) clat (msec): min=79, max=2175, avg=906.89, stdev=1115.37 lat (msec): min=82, max=2175, avg=907.97, stdev=1114.65 clat percentiles (msec): | 1.00th=[ 81], 5.00th=[ 81], 10.00th=[ 81], 20.00th=[ 81], | 30.00th=[ 81], 40.00th=[ 464], 50.00th=[ 464], 60.00th=[ 464], | 70.00th=[ 2165], 80.00th=[ 2165], 90.00th=[ 2165], 95.00th=[ 2165], | 99.00th=[ 2165], 99.50th=[ 2165], 99.90th=[ 2165], 99.95th=[ 2165], | 99.99th=[ 2165] bw ( KiB/s): min= 5936, max= 5936, per=52.91%, avg=5936.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 500=33.33% cpu : usr=0.00%, sys=2.91%, ctx=47, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22267: Thu May 1 23:00:53 2025 write: IOPS=3, BW=12.4MiB/s (13.0MB/s)(12.0MiB/967msec) clat (msec): min=207, max=377, avg=301.61, stdev=86.72 lat (msec): min=208, max=378, avg=302.49, stdev=86.14 clat percentiles (msec): | 1.00th=[ 207], 5.00th=[ 207], 10.00th=[ 207], 20.00th=[ 207], | 30.00th=[ 207], 40.00th=[ 321], 50.00th=[ 321], 60.00th=[ 321], | 70.00th=[ 380], 80.00th=[ 380], 90.00th=[ 380], 95.00th=[ 380], | 99.00th=[ 380], 99.50th=[ 380], 99.90th=[ 380], 99.95th=[ 380], | 99.99th=[ 380] bw ( KiB/s): min= 8192, max= 8192, per=73.03%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=66.67% cpu : usr=0.31%, sys=2.69%, ctx=22, majf=0, minf=28 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=5941KiB/s (6083kB/s), 1485KiB/s-5804KiB/s (1521kB/s-5944kB/s), io=16.0MiB (16.8MB), run=2117-2758msec WRITE: bw=10.0MiB/s (11.5MB/s), 2970KiB/s-12.4MiB/s (3042kB/s-13.0MB/s), io=32.0MiB (33.6MB), run=967-2921msec