== sanity test 398b: DIO and buffer IO race ============== 01:02:12 (1746162132) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.99613 s, 25.2 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=20836: Fri May 2 01:02:48 2025 read: IOPS=48, BW=192KiB/s (197kB/s)(5992KiB/31184msec) clat (usec): min=63, max=82964, avg=11781.19, stdev=8286.40 lat (usec): min=64, max=82964, avg=11782.76, stdev=8288.95 clat percentiles (usec): | 1.00th=[ 91], 5.00th=[ 157], 10.00th=[ 4293], 20.00th=[ 5932], | 30.00th=[ 7177], 40.00th=[ 8586], 50.00th=[10028], 60.00th=[11994], | 70.00th=[13960], 80.00th=[16319], 90.00th=[21103], 95.00th=[26608], | 99.00th=[40633], 99.50th=[49546], 99.90th=[67634], 99.95th=[83362], | 99.99th=[83362] bw ( KiB/s): min= 48, max= 367, per=25.27%, avg=191.06, stdev=58.90, samples=62 iops : min= 12, max= 91, avg=47.45, stdev=14.68, samples=62 write: IOPS=50, BW=202KiB/s (207kB/s)(6296KiB/31184msec) clat (usec): min=444, max=284799, avg=8504.28, stdev=11457.82 lat (usec): min=445, max=284801, avg=8505.86, stdev=11457.93 clat percentiles (usec): | 1.00th=[ 635], 5.00th=[ 725], 10.00th=[ 783], 20.00th=[ 889], | 30.00th=[ 1336], 40.00th=[ 2606], 50.00th=[ 5735], 60.00th=[ 8029], | 70.00th=[ 10683], 80.00th=[ 14353], 90.00th=[ 21103], 95.00th=[ 26870], | 99.00th=[ 39060], 99.50th=[ 44827], 99.90th=[ 92799], 99.95th=[283116], | 99.99th=[283116] bw ( KiB/s): min= 32, max= 431, per=25.36%, avg=200.58, stdev=70.89, samples=62 iops : min= 8, max= 107, avg=49.84, stdev=17.69, samples=62 lat (usec) : 100=1.07%, 250=1.50%, 500=0.26%, 750=3.71%, 1000=8.53% lat (msec) : 2=6.54%, 4=5.27%, 10=32.52%, 20=28.97%, 50=11.30% lat (msec) : 100=0.29%, 500=0.03% cpu : usr=0.23%, sys=13.39%, ctx=5193, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20837: Fri May 2 01:02:48 2025 read: IOPS=46, BW=187KiB/s (192kB/s)(5940KiB/31743msec) clat (usec): min=55, max=84981, avg=11485.87, stdev=8146.99 lat (usec): min=56, max=84982, avg=11489.62, stdev=8146.71 clat percentiles (usec): | 1.00th=[ 90], 5.00th=[ 135], 10.00th=[ 4080], 20.00th=[ 5669], | 30.00th=[ 6783], 40.00th=[ 8291], 50.00th=[ 9896], 60.00th=[11731], | 70.00th=[13698], 80.00th=[16450], 90.00th=[20579], 95.00th=[25035], | 99.00th=[41157], 99.50th=[48497], 99.90th=[69731], 99.95th=[85459], | 99.99th=[85459] bw ( KiB/s): min= 8, max= 352, per=24.52%, avg=185.35, stdev=62.76, samples=63 iops : min= 2, max= 88, avg=46.03, stdev=15.65, samples=63 write: IOPS=49, BW=200KiB/s (205kB/s)(6348KiB/31743msec) clat (usec): min=383, max=306230, avg=9169.95, stdev=12224.04 lat (usec): min=384, max=306231, avg=9171.23, stdev=12224.20 clat percentiles (usec): | 1.00th=[ 611], 5.00th=[ 725], 10.00th=[ 775], 20.00th=[ 881], | 30.00th=[ 1516], 40.00th=[ 3195], 50.00th=[ 6456], 60.00th=[ 8717], | 70.00th=[ 11338], 80.00th=[ 16057], 90.00th=[ 21890], 95.00th=[ 27395], | 99.00th=[ 45876], 99.50th=[ 51643], 99.90th=[ 67634], 99.95th=[308282], | 99.99th=[308282] bw ( KiB/s): min= 40, max= 407, per=24.81%, avg=196.27, stdev=64.42, samples=63 iops : min= 10, max= 101, avg=48.76, stdev=16.08, samples=63 lat (usec) : 100=1.01%, 250=1.73%, 500=0.33%, 750=3.97%, 1000=8.07% lat (msec) : 2=5.73%, 4=5.40%, 10=32.06%, 20=30.01%, 50=11.13% lat (msec) : 100=0.52%, 500=0.03% cpu : usr=0.13%, sys=13.43%, ctx=5106, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20838: Fri May 2 01:02:48 2025 read: IOPS=47, BW=191KiB/s (195kB/s)(6028KiB/31608msec) clat (usec): min=69, max=305853, avg=12076.68, stdev=11559.01 lat (usec): min=70, max=305853, avg=12078.61, stdev=11559.42 clat percentiles (usec): | 1.00th=[ 94], 5.00th=[ 124], 10.00th=[ 4146], 20.00th=[ 5866], | 30.00th=[ 7111], 40.00th=[ 8717], 50.00th=[ 10159], 60.00th=[ 11863], | 70.00th=[ 14091], 80.00th=[ 16909], 90.00th=[ 21627], 95.00th=[ 27657], | 99.00th=[ 44303], 99.50th=[ 52167], 99.90th=[ 96994], 99.95th=[304088], | 99.99th=[304088] bw ( KiB/s): min= 64, max= 351, per=25.12%, avg=189.89, stdev=57.57, samples=63 iops : min= 16, max= 87, avg=47.14, stdev=14.39, samples=63 write: IOPS=49, BW=198KiB/s (203kB/s)(6260KiB/31608msec) clat (usec): min=479, max=68225, avg=8475.58, stdev=9362.05 lat (usec): min=480, max=68227, avg=8478.92, stdev=9363.27 clat percentiles (usec): | 1.00th=[ 578], 5.00th=[ 717], 10.00th=[ 766], 20.00th=[ 873], | 30.00th=[ 1237], 40.00th=[ 2540], 50.00th=[ 5800], 60.00th=[ 7963], | 70.00th=[10814], 80.00th=[14746], 90.00th=[21365], 95.00th=[27132], | 99.00th=[40633], 99.50th=[50594], 99.90th=[66323], 99.95th=[68682], | 99.99th=[68682] bw ( KiB/s): min= 63, max= 383, per=24.85%, avg=196.59, stdev=68.82, samples=63 iops : min= 15, max= 95, avg=48.83, stdev=17.17, samples=63 lat (usec) : 100=0.94%, 250=2.12%, 500=0.13%, 750=4.59%, 1000=8.37% lat (msec) : 2=6.12%, 4=4.92%, 10=31.38%, 20=29.62%, 50=11.23% lat (msec) : 100=0.55%, 500=0.03% cpu : usr=0.20%, sys=13.45%, ctx=5169, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20840: Fri May 2 01:02:48 2025 read: IOPS=47, BW=191KiB/s (196kB/s)(6052KiB/31619msec) clat (usec): min=62, max=97329, avg=11750.22, stdev=8776.71 lat (usec): min=63, max=97331, avg=11751.50, stdev=8776.87 clat percentiles (usec): | 1.00th=[ 91], 5.00th=[ 118], 10.00th=[ 4015], 20.00th=[ 5669], | 30.00th=[ 6915], 40.00th=[ 8356], 50.00th=[ 9896], 60.00th=[11731], | 70.00th=[13829], 80.00th=[16712], 90.00th=[21890], 95.00th=[27132], | 99.00th=[46924], 99.50th=[51119], 99.90th=[67634], 99.95th=[96994], | 99.99th=[96994] bw ( KiB/s): min= 88, max= 383, per=25.47%, avg=192.55, stdev=64.74, samples=62 iops : min= 22, max= 95, avg=47.87, stdev=16.16, samples=62 write: IOPS=49, BW=197KiB/s (202kB/s)(6236KiB/31619msec) clat (usec): min=406, max=335356, avg=8776.74, stdev=12428.96 lat (usec): min=407, max=335357, avg=8782.49, stdev=12428.62 clat percentiles (usec): | 1.00th=[ 594], 5.00th=[ 725], 10.00th=[ 775], 20.00th=[ 889], | 30.00th=[ 1352], 40.00th=[ 2704], 50.00th=[ 6063], 60.00th=[ 8356], | 70.00th=[ 10945], 80.00th=[ 15008], 90.00th=[ 21627], 95.00th=[ 27132], | 99.00th=[ 39060], 99.50th=[ 47449], 99.90th=[ 85459], 99.95th=[333448], | 99.99th=[333448] bw ( KiB/s): min= 16, max= 391, per=24.76%, avg=195.84, stdev=71.52, samples=63 iops : min= 4, max= 97, avg=48.70, stdev=17.81, samples=63 lat (usec) : 100=1.24%, 250=1.99%, 500=0.23%, 750=3.78%, 1000=8.59% lat (msec) : 2=6.51%, 4=4.79%, 10=31.97%, 20=28.71%, 50=11.69% lat (msec) : 100=0.49%, 500=0.03% cpu : usr=0.09%, sys=13.43%, ctx=5195, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=756KiB/s (775kB/s), 187KiB/s-192KiB/s (192kB/s-197kB/s), io=23.4MiB (24.6MB), run=31184-31743msec WRITE: bw=792KiB/s (811kB/s), 197KiB/s-202KiB/s (202kB/s-207kB/s), io=24.6MiB (25.7MB), run=31184-31743msec rand-rw: (groupid=0, jobs=1): err= 0: pid=20839: Fri May 2 01:03:02 2025 read: IOPS=33, BW=133KiB/s (136kB/s)(5992KiB/45052msec) clat (msec): min=2, max=336, avg=11.33, stdev=12.95 lat (msec): min=2, max=336, avg=11.33, stdev=12.94 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 4], 10.00th=[ 5], 20.00th=[ 5], | 30.00th=[ 6], 40.00th=[ 7], 50.00th=[ 8], 60.00th=[ 9], | 70.00th=[ 11], 80.00th=[ 16], 90.00th=[ 25], 95.00th=[ 33], | 99.00th=[ 51], 99.50th=[ 58], 99.90th=[ 67], 99.95th=[ 338], | 99.99th=[ 338] bw ( KiB/s): min= 16, max= 334, per=25.01%, avg=132.58, stdev=73.80, samples=90 iops : min= 4, max= 83, avg=32.87, stdev=18.52, samples=90 write: IOPS=34, BW=140KiB/s (143kB/s)(6296KiB/45052msec) clat (msec): min=3, max=104, avg=17.73, stdev=12.18 lat (msec): min=3, max=104, avg=17.73, stdev=12.18 clat percentiles (msec): | 1.00th=[ 6], 5.00th=[ 7], 10.00th=[ 8], 20.00th=[ 10], | 30.00th=[ 11], 40.00th=[ 12], 50.00th=[ 14], 60.00th=[ 16], | 70.00th=[ 20], 80.00th=[ 24], 90.00th=[ 33], 95.00th=[ 42], | 99.00th=[ 66], 99.50th=[ 78], 99.90th=[ 95], 99.95th=[ 105], | 99.99th=[ 105] bw ( KiB/s): min= 39, max= 376, per=25.35%, avg=140.72, stdev=74.48, samples=89 iops : min= 9, max= 94, avg=34.90, stdev=18.70, samples=89 lat (msec) : 4=4.52%, 10=40.49%, 20=33.53%, 50=19.53%, 100=1.86% lat (msec) : 250=0.03%, 500=0.03% cpu : usr=0.11%, sys=6.35%, ctx=3401, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20841: Fri May 2 01:03:02 2025 read: IOPS=32, BW=131KiB/s (134kB/s)(5940KiB/45287msec) clat (msec): min=2, max=312, avg=11.44, stdev=13.05 lat (msec): min=2, max=312, avg=11.44, stdev=13.05 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 4], 10.00th=[ 5], 20.00th=[ 5], | 30.00th=[ 6], 40.00th=[ 7], 50.00th=[ 8], 60.00th=[ 9], | 70.00th=[ 11], 80.00th=[ 15], 90.00th=[ 26], 95.00th=[ 33], | 99.00th=[ 54], 99.50th=[ 64], 99.90th=[ 101], 99.95th=[ 313], | 99.99th=[ 313] bw ( KiB/s): min= 8, max= 328, per=24.61%, avg=130.44, stdev=75.38, samples=90 iops : min= 2, max= 82, avg=32.39, stdev=18.83, samples=90 write: IOPS=35, BW=140KiB/s (144kB/s)(6348KiB/45287msec) clat (usec): min=4181, max=94300, avg=17751.99, stdev=12308.92 lat (usec): min=4182, max=94300, avg=17754.00, stdev=12308.42 clat percentiles (usec): | 1.00th=[ 5211], 5.00th=[ 6718], 10.00th=[ 7767], 20.00th=[ 9241], | 30.00th=[10421], 40.00th=[11731], 50.00th=[13304], 60.00th=[15664], | 70.00th=[19268], 80.00th=[25035], 90.00th=[33817], 95.00th=[43779], | 99.00th=[65799], 99.50th=[74974], 99.90th=[88605], 99.95th=[93848], | 99.99th=[93848] bw ( KiB/s): min= 16, max= 392, per=25.03%, avg=138.89, stdev=78.04, samples=90 iops : min= 4, max= 98, avg=34.50, stdev=19.53, samples=90 lat (msec) : 4=4.69%, 10=42.09%, 20=31.51%, 50=19.56%, 100=2.08% lat (msec) : 250=0.03%, 500=0.03% cpu : usr=0.12%, sys=6.38%, ctx=3395, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20842: Fri May 2 01:03:02 2025 read: IOPS=33, BW=134KiB/s (137kB/s)(6028KiB/45035msec) clat (usec): min=2456, max=84802, avg=11124.63, stdev=10013.05 lat (usec): min=2457, max=84803, avg=11125.79, stdev=10013.01 clat percentiles (usec): | 1.00th=[ 2999], 5.00th=[ 3621], 10.00th=[ 4015], 20.00th=[ 4817], | 30.00th=[ 5538], 40.00th=[ 6259], 50.00th=[ 7242], 60.00th=[ 8586], | 70.00th=[10683], 80.00th=[15139], 90.00th=[24773], 95.00th=[33424], | 99.00th=[48497], 99.50th=[55313], 99.90th=[72877], 99.95th=[84411], | 99.99th=[84411] bw ( KiB/s): min= 16, max= 328, per=25.17%, avg=133.40, stdev=73.35, samples=90 iops : min= 4, max= 82, avg=33.07, stdev=18.42, samples=90 write: IOPS=34, BW=139KiB/s (142kB/s)(6260KiB/45035msec) clat (msec): min=3, max=288, avg=17.96, stdev=14.34 lat (msec): min=3, max=288, avg=17.96, stdev=14.34 clat percentiles (msec): | 1.00th=[ 6], 5.00th=[ 7], 10.00th=[ 8], 20.00th=[ 10], | 30.00th=[ 11], 40.00th=[ 12], 50.00th=[ 14], 60.00th=[ 16], | 70.00th=[ 20], 80.00th=[ 26], 90.00th=[ 34], 95.00th=[ 43], | 99.00th=[ 69], 99.50th=[ 87], 99.90th=[ 117], 99.95th=[ 288], | 99.99th=[ 288] bw ( KiB/s): min= 24, max= 376, per=24.94%, avg=138.44, stdev=77.51, samples=90 iops : min= 6, max= 94, avg=34.33, stdev=19.44, samples=90 lat (msec) : 4=4.88%, 10=41.76%, 20=31.48%, 50=20.02%, 100=1.79% lat (msec) : 250=0.03%, 500=0.03% cpu : usr=0.17%, sys=6.34%, ctx=3399, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20843: Fri May 2 01:03:02 2025 read: IOPS=33, BW=135KiB/s (138kB/s)(6052KiB/44820msec) clat (msec): min=2, max=101, avg=10.83, stdev= 9.88 lat (msec): min=2, max=101, avg=10.83, stdev= 9.88 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 4], 10.00th=[ 4], 20.00th=[ 5], | 30.00th=[ 6], 40.00th=[ 7], 50.00th=[ 8], 60.00th=[ 9], | 70.00th=[ 11], 80.00th=[ 15], 90.00th=[ 24], 95.00th=[ 32], | 99.00th=[ 52], 99.50th=[ 57], 99.90th=[ 81], 99.95th=[ 102], | 99.99th=[ 102] bw ( KiB/s): min= 31, max= 400, per=25.57%, avg=135.50, stdev=76.88, samples=88 iops : min= 7, max= 100, avg=33.57, stdev=19.27, samples=88 write: IOPS=34, BW=139KiB/s (142kB/s)(6236KiB/44820msec) clat (msec): min=4, max=323, avg=18.16, stdev=14.56 lat (msec): min=4, max=323, avg=18.17, stdev=14.56 clat percentiles (msec): | 1.00th=[ 6], 5.00th=[ 7], 10.00th=[ 8], 20.00th=[ 10], | 30.00th=[ 11], 40.00th=[ 13], 50.00th=[ 14], 60.00th=[ 17], | 70.00th=[ 20], 80.00th=[ 25], 90.00th=[ 34], 95.00th=[ 43], | 99.00th=[ 67], 99.50th=[ 79], 99.90th=[ 116], 99.95th=[ 326], | 99.99th=[ 326] bw ( KiB/s): min= 24, max= 352, per=24.92%, avg=138.28, stdev=73.83, samples=89 iops : min= 6, max= 88, avg=34.27, stdev=18.47, samples=89 lat (msec) : 4=5.18%, 10=41.41%, 20=32.42%, 50=18.82%, 100=2.08% lat (msec) : 250=0.07%, 500=0.03% cpu : usr=0.03%, sys=6.56%, ctx=3384, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=530KiB/s (543kB/s), 131KiB/s-135KiB/s (134kB/s-138kB/s), io=23.4MiB (24.6MB), run=44820-45287msec WRITE: bw=555KiB/s (568kB/s), 139KiB/s-140KiB/s (142kB/s-144kB/s), io=24.6MiB (25.7MB), run=44820-45287msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=20898: Fri May 2 01:03:13 2025 read: IOPS=36, BW=581KiB/s (595kB/s)(5920KiB/10182msec) clat (usec): min=73, max=58012, avg=14482.34, stdev=10009.58 lat (usec): min=74, max=58013, avg=14485.52, stdev=10008.01 clat percentiles (usec): | 1.00th=[ 91], 5.00th=[ 116], 10.00th=[ 129], 20.00th=[ 6587], | 30.00th=[ 8979], 40.00th=[11076], 50.00th=[13566], 60.00th=[15664], | 70.00th=[18220], 80.00th=[21365], 90.00th=[27132], 95.00th=[32113], | 99.00th=[49021], 99.50th=[54789], 99.90th=[57934], 99.95th=[57934], | 99.99th=[57934] bw ( KiB/s): min= 383, max= 864, per=24.02%, avg=565.00, stdev=134.19, samples=20 iops : min= 23, max= 54, avg=34.70, stdev= 8.55, samples=20 write: IOPS=39, BW=625KiB/s (640kB/s)(6368KiB/10182msec) clat (usec): min=924, max=94631, avg=11714.55, stdev=12759.88 lat (usec): min=925, max=94632, avg=11735.40, stdev=12749.05 clat percentiles (usec): | 1.00th=[ 971], 5.00th=[ 1139], 10.00th=[ 1221], 20.00th=[ 1532], | 30.00th=[ 2409], 40.00th=[ 4424], 50.00th=[ 7701], 60.00th=[11207], | 70.00th=[14222], 80.00th=[19268], 90.00th=[28967], 95.00th=[36963], | 99.00th=[53740], 99.50th=[86508], 99.90th=[94897], 99.95th=[94897], | 99.99th=[94897] bw ( KiB/s): min= 288, max= 800, per=24.90%, avg=616.05, stdev=128.04, samples=20 iops : min= 18, max= 50, avg=37.90, stdev= 7.97, samples=20 lat (usec) : 100=0.91%, 250=4.43%, 500=0.13%, 750=0.13%, 1000=0.78% lat (msec) : 2=12.50%, 4=7.29%, 10=20.05%, 20=32.16%, 50=20.57% lat (msec) : 100=1.04% cpu : usr=0.26%, sys=12.36%, ctx=1294, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20901: Fri May 2 01:03:13 2025 read: IOPS=35, BW=567KiB/s (580kB/s)(5744KiB/10135msec) clat (usec): min=82, max=53356, avg=14731.32, stdev=10552.79 lat (usec): min=82, max=53357, avg=14732.38, stdev=10552.83 clat percentiles (usec): | 1.00th=[ 93], 5.00th=[ 108], 10.00th=[ 137], 20.00th=[ 6980], | 30.00th=[ 9110], 40.00th=[10814], 50.00th=[12911], 60.00th=[15664], | 70.00th=[17957], 80.00th=[22152], 90.00th=[29754], 95.00th=[36963], | 99.00th=[44827], 99.50th=[47449], 99.90th=[53216], 99.95th=[53216], | 99.99th=[53216] bw ( KiB/s): min= 256, max= 928, per=23.72%, avg=558.00, stdev=145.47, samples=20 iops : min= 16, max= 58, avg=34.65, stdev= 9.11, samples=20 write: IOPS=40, BW=646KiB/s (661kB/s)(6544KiB/10135msec) clat (usec): min=897, max=64004, avg=11378.35, stdev=11602.39 lat (usec): min=899, max=64006, avg=11379.99, stdev=11602.43 clat percentiles (usec): | 1.00th=[ 971], 5.00th=[ 1106], 10.00th=[ 1188], 20.00th=[ 1647], | 30.00th=[ 2900], 40.00th=[ 4883], 50.00th=[ 7767], 60.00th=[10028], | 70.00th=[13435], 80.00th=[18744], 90.00th=[29492], 95.00th=[37487], | 99.00th=[46924], 99.50th=[51643], 99.90th=[64226], 99.95th=[64226], | 99.99th=[64226] bw ( KiB/s): min= 288, max= 896, per=25.27%, avg=625.20, stdev=171.93, samples=20 iops : min= 18, max= 56, avg=38.85, stdev=10.78, samples=20 lat (usec) : 100=1.04%, 250=4.82%, 500=0.26%, 750=0.13%, 1000=1.04% lat (msec) : 2=11.07%, 4=7.94%, 10=21.74%, 20=30.47%, 50=20.96% lat (msec) : 100=0.52% cpu : usr=0.19%, sys=12.42%, ctx=1276, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20902: Fri May 2 01:03:13 2025 read: IOPS=39, BW=625KiB/s (640kB/s)(6256KiB/10002msec) clat (usec): min=87, max=57345, avg=14346.90, stdev=10393.86 lat (usec): min=88, max=57346, avg=14347.94, stdev=10393.93 clat percentiles (usec): | 1.00th=[ 97], 5.00th=[ 112], 10.00th=[ 143], 20.00th=[ 6915], | 30.00th=[ 8979], 40.00th=[10945], 50.00th=[13173], 60.00th=[14615], | 70.00th=[16909], 80.00th=[21103], 90.00th=[27395], 95.00th=[34341], | 99.00th=[54264], 99.50th=[56886], 99.90th=[57410], 99.95th=[57410], | 99.99th=[57410] bw ( KiB/s): min= 381, max= 992, per=26.55%, avg=624.45, stdev=169.68, samples=20 iops : min= 23, max= 62, avg=38.70, stdev=10.69, samples=20 write: IOPS=37, BW=603KiB/s (618kB/s)(6032KiB/10002msec) clat (usec): min=762, max=60094, avg=11525.10, stdev=12195.87 lat (usec): min=763, max=60095, avg=11526.70, stdev=12195.87 clat percentiles (usec): | 1.00th=[ 947], 5.00th=[ 1090], 10.00th=[ 1172], 20.00th=[ 1418], | 30.00th=[ 2180], 40.00th=[ 3949], 50.00th=[ 6980], 60.00th=[10814], | 70.00th=[14615], 80.00th=[20317], 90.00th=[28181], 95.00th=[37487], | 99.00th=[52691], 99.50th=[57410], 99.90th=[60031], 99.95th=[60031], | 99.99th=[60031] bw ( KiB/s): min= 288, max= 928, per=24.27%, avg=600.55, stdev=167.84, samples=20 iops : min= 18, max= 58, avg=37.20, stdev=10.60, samples=20 lat (usec) : 100=0.91%, 250=4.95%, 500=0.26%, 750=0.26%, 1000=1.17% lat (msec) : 2=13.15%, 4=5.86%, 10=20.31%, 20=31.12%, 50=20.70% lat (msec) : 100=1.30% cpu : usr=0.13%, sys=12.41%, ctx=1342, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20903: Fri May 2 01:03:13 2025 read: IOPS=37, BW=604KiB/s (619kB/s)(6032KiB/9979msec) clat (usec): min=54, max=93179, avg=15121.85, stdev=11467.99 lat (usec): min=54, max=93181, avg=15124.30, stdev=11466.18 clat percentiles (usec): | 1.00th=[ 88], 5.00th=[ 108], 10.00th=[ 131], 20.00th=[ 7570], | 30.00th=[ 9503], 40.00th=[11207], 50.00th=[13304], 60.00th=[15795], | 70.00th=[18482], 80.00th=[22414], 90.00th=[27657], 95.00th=[32113], | 99.00th=[59507], 99.50th=[66323], 99.90th=[92799], 99.95th=[92799], | 99.99th=[92799] bw ( KiB/s): min= 254, max= 864, per=24.94%, avg=586.58, stdev=144.33, samples=19 iops : min= 15, max= 54, avg=36.11, stdev= 9.14, samples=19 write: IOPS=39, BW=627KiB/s (642kB/s)(6256KiB/9979msec) clat (usec): min=838, max=130963, avg=10638.71, stdev=12630.21 lat (usec): min=840, max=130965, avg=10640.31, stdev=12630.22 clat percentiles (usec): | 1.00th=[ 947], 5.00th=[ 1139], 10.00th=[ 1221], 20.00th=[ 1532], | 30.00th=[ 2212], 40.00th=[ 3752], 50.00th=[ 5800], 60.00th=[ 9372], | 70.00th=[ 12518], 80.00th=[ 19268], 90.00th=[ 26608], 95.00th=[ 32113], | 99.00th=[ 52167], 99.50th=[ 60556], 99.90th=[130548], 99.95th=[130548], | 99.99th=[130548] bw ( KiB/s): min= 351, max= 894, per=24.93%, avg=616.74, stdev=137.50, samples=19 iops : min= 21, max= 55, avg=38.00, stdev= 8.52, samples=19 lat (usec) : 100=1.43%, 250=4.43%, 500=0.26%, 1000=0.91% lat (msec) : 2=12.89%, 4=7.94%, 10=19.79%, 20=29.82%, 50=20.70% lat (msec) : 100=1.69%, 250=0.13% cpu : usr=0.01%, sys=12.56%, ctx=1314, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2352KiB/s (2409kB/s), 567KiB/s-625KiB/s (580kB/s-640kB/s), io=23.4MiB (24.5MB), run=9979-10182msec WRITE: bw=2475KiB/s (2534kB/s), 603KiB/s-646KiB/s (618kB/s-661kB/s), io=24.6MiB (25.8MB), run=9979-10182msec rand-rw: (groupid=0, jobs=1): err= 0: pid=20896: Fri May 2 01:03:18 2025 read: IOPS=24, BW=385KiB/s (395kB/s)(5920KiB/15364msec) clat (usec): min=3489, max=80651, avg=15372.57, stdev=13424.00 lat (usec): min=3490, max=80652, avg=15376.11, stdev=13422.89 clat percentiles (usec): | 1.00th=[ 3982], 5.00th=[ 5211], 10.00th=[ 5866], 20.00th=[ 7046], | 30.00th=[ 7898], 40.00th=[ 9241], 50.00th=[10290], 60.00th=[11600], | 70.00th=[14746], 80.00th=[20841], 90.00th=[32113], 95.00th=[47973], | 99.00th=[69731], 99.50th=[72877], 99.90th=[80217], 99.95th=[80217], | 99.99th=[80217] bw ( KiB/s): min= 160, max= 828, per=24.38%, avg=376.50, stdev=159.05, samples=30 iops : min= 10, max= 51, avg=23.00, stdev= 9.92, samples=30 write: IOPS=25, BW=414KiB/s (424kB/s)(6368KiB/15364msec) clat (msec): min=6, max=102, avg=23.97, stdev=14.29 lat (msec): min=6, max=102, avg=23.97, stdev=14.29 clat percentiles (msec): | 1.00th=[ 9], 5.00th=[ 11], 10.00th=[ 13], 20.00th=[ 15], | 30.00th=[ 16], 40.00th=[ 18], 50.00th=[ 20], 60.00th=[ 23], | 70.00th=[ 27], 80.00th=[ 31], 90.00th=[ 43], 95.00th=[ 53], | 99.00th=[ 86], 99.50th=[ 99], 99.90th=[ 104], 99.95th=[ 104], | 99.99th=[ 104] bw ( KiB/s): min= 159, max= 736, per=25.28%, avg=410.73, stdev=174.00, samples=30 iops : min= 9, max= 46, avg=25.13, stdev=10.92, samples=30 lat (msec) : 4=0.52%, 10=23.44%, 20=41.02%, 50=29.95%, 100=4.95% lat (msec) : 250=0.13% cpu : usr=0.09%, sys=5.30%, ctx=882, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20897: Fri May 2 01:03:18 2025 read: IOPS=23, BW=370KiB/s (379kB/s)(5744KiB/15507msec) clat (usec): min=3409, max=82497, avg=14527.32, stdev=12429.56 lat (usec): min=3410, max=82498, avg=14534.53, stdev=12427.97 clat percentiles (usec): | 1.00th=[ 4424], 5.00th=[ 5080], 10.00th=[ 5800], 20.00th=[ 6849], | 30.00th=[ 7767], 40.00th=[ 8717], 50.00th=[10028], 60.00th=[11994], | 70.00th=[14353], 80.00th=[18744], 90.00th=[31327], 95.00th=[39060], | 99.00th=[71828], 99.50th=[79168], 99.90th=[82314], 99.95th=[82314], | 99.99th=[82314] bw ( KiB/s): min= 63, max= 736, per=23.71%, avg=366.13, stdev=196.45, samples=30 iops : min= 3, max= 46, avg=22.47, stdev=12.28, samples=30 write: IOPS=26, BW=422KiB/s (432kB/s)(6544KiB/15507msec) clat (msec): min=7, max=101, avg=24.93, stdev=15.47 lat (msec): min=7, max=101, avg=24.93, stdev=15.47 clat percentiles (msec): | 1.00th=[ 8], 5.00th=[ 11], 10.00th=[ 12], 20.00th=[ 14], | 30.00th=[ 16], 40.00th=[ 18], 50.00th=[ 21], 60.00th=[ 24], | 70.00th=[ 29], 80.00th=[ 35], 90.00th=[ 45], 95.00th=[ 55], | 99.00th=[ 89], 99.50th=[ 94], 99.90th=[ 102], 99.95th=[ 102], | 99.99th=[ 102] bw ( KiB/s): min= 190, max= 928, per=25.15%, avg=408.77, stdev=174.70, samples=30 iops : min= 11, max= 58, avg=25.13, stdev=11.04, samples=30 lat (msec) : 4=0.26%, 10=25.52%, 20=38.67%, 50=30.60%, 100=4.82% lat (msec) : 250=0.13% cpu : usr=0.17%, sys=5.20%, ctx=858, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20899: Fri May 2 01:03:18 2025 read: IOPS=25, BW=405KiB/s (414kB/s)(6256KiB/15462msec) clat (usec): min=3903, max=85857, avg=15439.15, stdev=13815.81 lat (usec): min=3904, max=85858, avg=15440.38, stdev=13815.76 clat percentiles (usec): | 1.00th=[ 4113], 5.00th=[ 5276], 10.00th=[ 5866], 20.00th=[ 6980], | 30.00th=[ 7635], 40.00th=[ 8848], 50.00th=[10028], 60.00th=[11863], | 70.00th=[14615], 80.00th=[21103], 90.00th=[32637], 95.00th=[50594], | 99.00th=[68682], 99.50th=[72877], 99.90th=[85459], 99.95th=[85459], | 99.99th=[85459] bw ( KiB/s): min= 159, max= 894, per=26.12%, avg=403.27, stdev=198.39, samples=30 iops : min= 9, max= 55, avg=24.67, stdev=12.38, samples=30 write: IOPS=24, BW=390KiB/s (399kB/s)(6032KiB/15462msec) clat (msec): min=6, max=105, avg=24.54, stdev=14.20 lat (msec): min=6, max=105, avg=24.54, stdev=14.20 clat percentiles (msec): | 1.00th=[ 9], 5.00th=[ 12], 10.00th=[ 13], 20.00th=[ 15], | 30.00th=[ 17], 40.00th=[ 18], 50.00th=[ 20], 60.00th=[ 23], | 70.00th=[ 27], 80.00th=[ 35], 90.00th=[ 44], 95.00th=[ 58], | 99.00th=[ 74], 99.50th=[ 81], 99.90th=[ 106], 99.95th=[ 106], | 99.99th=[ 106] bw ( KiB/s): min= 128, max= 736, per=23.31%, avg=378.73, stdev=166.92, samples=30 iops : min= 8, max= 46, avg=23.13, stdev=10.44, samples=30 lat (msec) : 4=0.13%, 10=26.56%, 20=38.80%, 50=28.78%, 100=5.60% lat (msec) : 250=0.13% cpu : usr=0.19%, sys=5.23%, ctx=868, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20900: Fri May 2 01:03:18 2025 read: IOPS=24, BW=396KiB/s (405kB/s)(6032KiB/15234msec) clat (usec): min=3630, max=88061, avg=14919.74, stdev=12025.52 lat (usec): min=3632, max=88063, avg=14920.89, stdev=12025.50 clat percentiles (usec): | 1.00th=[ 4228], 5.00th=[ 5276], 10.00th=[ 6325], 20.00th=[ 7373], | 30.00th=[ 8455], 40.00th=[ 9503], 50.00th=[10683], 60.00th=[12256], | 70.00th=[14353], 80.00th=[19530], 90.00th=[30016], 95.00th=[39060], | 99.00th=[72877], 99.50th=[80217], 99.90th=[87557], 99.95th=[87557], | 99.99th=[87557] bw ( KiB/s): min= 159, max= 828, per=25.45%, avg=392.90, stdev=170.02, samples=30 iops : min= 9, max= 51, avg=24.17, stdev=10.71, samples=30 write: IOPS=25, BW=411KiB/s (421kB/s)(6256KiB/15234msec) clat (msec): min=6, max=101, avg=24.44, stdev=14.57 lat (msec): min=6, max=101, avg=24.46, stdev=14.59 clat percentiles (msec): | 1.00th=[ 8], 5.00th=[ 12], 10.00th=[ 13], 20.00th=[ 14], | 30.00th=[ 16], 40.00th=[ 17], 50.00th=[ 20], 60.00th=[ 23], | 70.00th=[ 27], 80.00th=[ 34], 90.00th=[ 44], 95.00th=[ 56], | 99.00th=[ 83], 99.50th=[ 84], 99.90th=[ 102], 99.95th=[ 102], | 99.99th=[ 102] bw ( KiB/s): min= 159, max= 830, per=25.03%, avg=406.73, stdev=173.58, samples=30 iops : min= 9, max= 51, avg=25.03, stdev=10.91, samples=30 lat (msec) : 4=0.26%, 10=22.53%, 20=43.10%, 50=29.43%, 100=4.56% lat (msec) : 250=0.13% cpu : usr=0.12%, sys=5.17%, ctx=864, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1545KiB/s (1582kB/s), 370KiB/s-405KiB/s (379kB/s-414kB/s), io=23.4MiB (24.5MB), run=15234-15507msec WRITE: bw=1625KiB/s (1664kB/s), 390KiB/s-422KiB/s (399kB/s-432kB/s), io=24.6MiB (25.8MB), run=15234-15507msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=20921: Fri May 2 01:03:23 2025 read: IOPS=2, BW=2866KiB/s (2935kB/s)(7168KiB/2501msec) clat (usec): min=403, max=143716, avg=70385.90, stdev=54681.52 lat (usec): min=404, max=143717, avg=70387.35, stdev=54681.09 clat percentiles (usec): | 1.00th=[ 404], 5.00th=[ 404], 10.00th=[ 404], 20.00th=[ 2638], | 30.00th=[ 51119], 40.00th=[ 51119], 50.00th=[ 90702], 60.00th=[ 90702], | 70.00th=[ 90702], 80.00th=[113771], 90.00th=[143655], 95.00th=[143655], | 99.00th=[143655], 99.50th=[143655], 99.90th=[143655], 99.95th=[143655], | 99.99th=[143655] bw ( KiB/s): min= 4079, max= 6144, per=69.29%, avg=5111.50, stdev=1460.18, samples=2 iops : min= 3, max= 6, avg= 4.50, stdev= 2.12, samples=2 write: IOPS=1, BW=2047KiB/s (2096kB/s)(5120KiB/2501msec) clat (msec): min=52, max=1479, avg=352.54, stdev=630.12 lat (msec): min=52, max=1479, avg=352.60, stdev=630.13 clat percentiles (msec): | 1.00th=[ 53], 5.00th=[ 53], 10.00th=[ 53], 20.00th=[ 53], | 30.00th=[ 70], 40.00th=[ 70], 50.00th=[ 79], 60.00th=[ 79], | 70.00th=[ 84], 80.00th=[ 84], 90.00th=[ 1485], 95.00th=[ 1485], | 99.00th=[ 1485], 99.50th=[ 1485], 99.90th=[ 1485], 99.95th=[ 1485], | 99.99th=[ 1485] bw ( KiB/s): min= 6144, max= 6144, per=59.49%, avg=6144.00, stdev= 0.00, samples=1 iops : min= 6, max= 6, avg= 6.00, stdev= 0.00, samples=1 lat (usec) : 500=8.33% lat (msec) : 4=8.33%, 100=58.33%, 250=16.67% cpu : usr=0.00%, sys=8.92%, ctx=258, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20922: Fri May 2 01:03:23 2025 read: IOPS=0, BW=738KiB/s (755kB/s)(2048KiB/2776msec) clat (msec): min=119, max=1310, avg=714.98, stdev=842.13 lat (msec): min=119, max=1310, avg=714.98, stdev=842.13 clat percentiles (msec): | 1.00th=[ 120], 5.00th=[ 120], 10.00th=[ 120], 20.00th=[ 120], | 30.00th=[ 120], 40.00th=[ 120], 50.00th=[ 120], 60.00th=[ 1318], | 70.00th=[ 1318], 80.00th=[ 1318], 90.00th=[ 1318], 95.00th=[ 1318], | 99.00th=[ 1318], 99.50th=[ 1318], 99.90th=[ 1318], 99.95th=[ 1318], | 99.99th=[ 1318] bw ( KiB/s): min= 2048, max= 2048, per=27.76%, avg=2048.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 write: IOPS=3, BW=3689KiB/s (3777kB/s)(10.0MiB/2776msec) clat (msec): min=31, max=324, avg=131.06, stdev=80.76 lat (msec): min=31, max=324, avg=131.14, stdev=80.75 clat percentiles (msec): | 1.00th=[ 32], 5.00th=[ 32], 10.00th=[ 32], 20.00th=[ 80], | 30.00th=[ 86], 40.00th=[ 91], 50.00th=[ 103], 60.00th=[ 108], | 70.00th=[ 157], 80.00th=[ 157], 90.00th=[ 186], 95.00th=[ 326], | 99.00th=[ 326], 99.50th=[ 326], 99.90th=[ 326], 99.95th=[ 326], | 99.99th=[ 326] bw ( KiB/s): min= 2048, max= 6144, per=39.66%, avg=4096.00, stdev=2048.00, samples=3 iops : min= 2, max= 6, avg= 4.00, stdev= 2.00, samples=3 lat (msec) : 50=8.33%, 100=25.00%, 250=50.00%, 500=8.33% cpu : usr=0.00%, sys=14.05%, ctx=403, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20924: Fri May 2 01:03:23 2025 read: IOPS=2, BW=2422KiB/s (2480kB/s)(6144KiB/2537msec) clat (usec): min=416, max=269708, avg=60678.75, stdev=105267.78 lat (usec): min=417, max=269710, avg=60679.85, stdev=105267.95 clat percentiles (usec): | 1.00th=[ 416], 5.00th=[ 416], 10.00th=[ 416], 20.00th=[ 482], | 30.00th=[ 482], 40.00th=[ 1745], 50.00th=[ 1745], 60.00th=[ 29230], | 70.00th=[ 62129], 80.00th=[ 62129], 90.00th=[270533], 95.00th=[270533], | 99.00th=[270533], 99.50th=[270533], 99.90th=[270533], 99.95th=[270533], | 99.99th=[270533] bw ( KiB/s): min= 2048, max= 8192, per=69.40%, avg=5120.00, stdev=4344.46, samples=2 iops : min= 2, max= 8, avg= 5.00, stdev= 4.24, samples=2 write: IOPS=2, BW=2422KiB/s (2480kB/s)(6144KiB/2537msec) clat (msec): min=48, max=1317, avg=332.60, stdev=488.54 lat (msec): min=48, max=1317, avg=332.68, stdev=488.54 clat percentiles (msec): | 1.00th=[ 49], 5.00th=[ 49], 10.00th=[ 49], 20.00th=[ 56], | 30.00th=[ 56], 40.00th=[ 138], 50.00th=[ 138], 60.00th=[ 182], | 70.00th=[ 255], 80.00th=[ 255], 90.00th=[ 1318], 95.00th=[ 1318], | 99.00th=[ 1318], 99.50th=[ 1318], 99.90th=[ 1318], 99.95th=[ 1318], | 99.99th=[ 1318] bw ( KiB/s): min= 2048, max= 4096, per=33.05%, avg=3413.33, stdev=1182.41, samples=3 iops : min= 2, max= 4, avg= 3.33, stdev= 1.15, samples=3 lat (usec) : 500=16.67% lat (msec) : 2=8.33%, 50=16.67%, 100=16.67%, 250=16.67%, 500=16.67% cpu : usr=0.04%, sys=6.47%, ctx=217, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20926: Fri May 2 01:03:23 2025 read: IOPS=1, BW=2035KiB/s (2084kB/s)(5120KiB/2516msec) clat (usec): min=530, max=78853, avg=29926.68, stdev=36879.22 lat (usec): min=531, max=78853, avg=29927.80, stdev=36878.82 clat percentiles (usec): | 1.00th=[ 529], 5.00th=[ 529], 10.00th=[ 529], 20.00th=[ 529], | 30.00th=[ 586], 40.00th=[ 586], 50.00th=[ 9634], 60.00th=[ 9634], | 70.00th=[60031], 80.00th=[60031], 90.00th=[79168], 95.00th=[79168], | 99.00th=[79168], 99.50th=[79168], 99.90th=[79168], 99.95th=[79168], | 99.99th=[79168] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=2, BW=2849KiB/s (2917kB/s)(7168KiB/2516msec) clat (msec): min=61, max=1582, avg=328.29, stdev=554.67 lat (msec): min=61, max=1583, avg=328.35, stdev=554.66 clat percentiles (msec): | 1.00th=[ 62], 5.00th=[ 62], 10.00th=[ 62], 20.00th=[ 101], | 30.00th=[ 105], 40.00th=[ 105], 50.00th=[ 112], 60.00th=[ 157], | 70.00th=[ 157], 80.00th=[ 184], 90.00th=[ 1586], 95.00th=[ 1586], | 99.00th=[ 1586], 99.50th=[ 1586], 99.90th=[ 1586], 99.95th=[ 1586], | 99.99th=[ 1586] bw ( KiB/s): min= 6144, max= 6144, per=59.49%, avg=6144.00, stdev= 0.00, samples=2 iops : min= 6, max= 6, avg= 6.00, stdev= 0.00, samples=2 lat (usec) : 750=16.67% lat (msec) : 10=8.33%, 100=33.33%, 250=33.33% cpu : usr=0.00%, sys=8.39%, ctx=87, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=7378KiB/s (7555kB/s), 738KiB/s-2866KiB/s (755kB/s-2935kB/s), io=20.0MiB (20.0MB), run=2501-2776msec WRITE: bw=10.1MiB/s (10.6MB/s), 2047KiB/s-3689KiB/s (2096kB/s-3777kB/s), io=28.0MiB (29.4MB), run=2501-2776msec rand-rw: (groupid=0, jobs=1): err= 0: pid=20923: Fri May 2 01:03:23 2025 read: IOPS=3, BW=3152KiB/s (3228kB/s)(7168KiB/2274msec) clat (msec): min=27, max=197, avg=67.15, stdev=60.16 lat (msec): min=27, max=197, avg=67.16, stdev=60.16 clat percentiles (msec): | 1.00th=[ 28], 5.00th=[ 28], 10.00th=[ 28], 20.00th=[ 28], | 30.00th=[ 35], 40.00th=[ 35], 50.00th=[ 46], 60.00th=[ 64], | 70.00th=[ 64], 80.00th=[ 74], 90.00th=[ 199], 95.00th=[ 199], | 99.00th=[ 199], 99.50th=[ 199], 99.90th=[ 199], 99.95th=[ 199], | 99.99th=[ 199] bw ( KiB/s): min= 4087, max= 6131, per=71.83%, avg=5109.00, stdev=1445.33, samples=2 iops : min= 3, max= 5, avg= 4.00, stdev= 1.41, samples=2 write: IOPS=2, BW=2252KiB/s (2306kB/s)(5120KiB/2274msec) clat (msec): min=35, max=1351, avg=351.68, stdev=561.09 lat (msec): min=35, max=1351, avg=351.87, stdev=561.04 clat percentiles (msec): | 1.00th=[ 36], 5.00th=[ 36], 10.00th=[ 36], 20.00th=[ 36], | 30.00th=[ 86], 40.00th=[ 86], 50.00th=[ 120], 60.00th=[ 120], | 70.00th=[ 167], 80.00th=[ 167], 90.00th=[ 1351], 95.00th=[ 1351], | 99.00th=[ 1351], 99.50th=[ 1351], 99.90th=[ 1351], 99.95th=[ 1351], | 99.99th=[ 1351] bw ( KiB/s): min= 2043, max= 4087, per=30.78%, avg=3065.00, stdev=1445.33, samples=2 iops : min= 1, max= 3, avg= 2.00, stdev= 1.41, samples=2 lat (msec) : 50=41.67%, 100=25.00%, 250=25.00% cpu : usr=0.13%, sys=1.32%, ctx=61, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20925: Fri May 2 01:03:23 2025 read: IOPS=0, BW=711KiB/s (728kB/s)(2048KiB/2879msec) clat (msec): min=82, max=1116, avg=599.38, stdev=731.50 lat (msec): min=82, max=1116, avg=599.38, stdev=731.50 clat percentiles (msec): | 1.00th=[ 83], 5.00th=[ 83], 10.00th=[ 83], 20.00th=[ 83], | 30.00th=[ 83], 40.00th=[ 83], 50.00th=[ 83], 60.00th=[ 1116], | 70.00th=[ 1116], 80.00th=[ 1116], 90.00th=[ 1116], 95.00th=[ 1116], | 99.00th=[ 1116], 99.50th=[ 1116], 99.90th=[ 1116], 99.95th=[ 1116], | 99.99th=[ 1116] bw ( KiB/s): min= 2043, max= 2048, per=28.76%, avg=2045.50, stdev= 3.54, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 write: IOPS=3, BW=3557KiB/s (3642kB/s)(10.0MiB/2879msec) clat (msec): min=54, max=297, avg=133.68, stdev=72.89 lat (msec): min=54, max=297, avg=133.74, stdev=72.89 clat percentiles (msec): | 1.00th=[ 55], 5.00th=[ 55], 10.00th=[ 55], 20.00th=[ 59], | 30.00th=[ 68], 40.00th=[ 108], 50.00th=[ 130], 60.00th=[ 134], | 70.00th=[ 155], 80.00th=[ 155], 90.00th=[ 194], 95.00th=[ 296], | 99.00th=[ 296], 99.50th=[ 296], 99.90th=[ 296], 99.95th=[ 296], | 99.99th=[ 296] bw ( KiB/s): min= 2048, max= 6144, per=41.10%, avg=4093.00, stdev=2048.01, samples=3 iops : min= 2, max= 6, avg= 3.67, stdev= 2.08, samples=3 lat (msec) : 100=33.33%, 250=50.00%, 500=8.33% cpu : usr=0.00%, sys=4.48%, ctx=118, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20927: Fri May 2 01:03:23 2025 read: IOPS=2, BW=2180KiB/s (2233kB/s)(6144KiB/2818msec) clat (msec): min=48, max=252, avg=123.39, stdev=86.04 lat (msec): min=48, max=252, avg=123.39, stdev=86.04 clat percentiles (msec): | 1.00th=[ 48], 5.00th=[ 48], 10.00th=[ 48], 20.00th=[ 50], | 30.00th=[ 50], 40.00th=[ 53], 50.00th=[ 53], 60.00th=[ 163], | 70.00th=[ 176], 80.00th=[ 176], 90.00th=[ 253], 95.00th=[ 253], | 99.00th=[ 253], 99.50th=[ 253], 99.90th=[ 253], 99.95th=[ 253], | 99.99th=[ 253] bw ( KiB/s): min= 4087, max= 4096, per=57.52%, avg=4091.50, stdev= 6.36, samples=2 iops : min= 3, max= 4, avg= 3.50, stdev= 0.71, samples=2 write: IOPS=2, BW=2180KiB/s (2233kB/s)(6144KiB/2818msec) clat (msec): min=98, max=1294, avg=334.38, stdev=471.59 lat (msec): min=98, max=1295, avg=334.49, stdev=471.57 clat percentiles (msec): | 1.00th=[ 99], 5.00th=[ 99], 10.00th=[ 99], 20.00th=[ 131], | 30.00th=[ 131], 40.00th=[ 136], 50.00th=[ 136], 60.00th=[ 155], | 70.00th=[ 192], 80.00th=[ 192], 90.00th=[ 1301], 95.00th=[ 1301], | 99.00th=[ 1301], 99.50th=[ 1301], 99.90th=[ 1301], 99.95th=[ 1301], | 99.99th=[ 1301] bw ( KiB/s): min= 2048, max= 4096, per=34.24%, avg=3410.33, stdev=1179.82, samples=3 iops : min= 2, max= 4, avg= 3.00, stdev= 1.00, samples=3 lat (msec) : 50=16.67%, 100=16.67%, 250=50.00%, 500=8.33% cpu : usr=0.14%, sys=2.52%, ctx=42, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20928: Fri May 2 01:03:23 2025 read: IOPS=1, BW=1839KiB/s (1883kB/s)(5120KiB/2784msec) clat (usec): min=16822, max=85495, avg=42639.56, stdev=28017.75 lat (usec): min=16824, max=85496, avg=42641.05, stdev=28017.92 clat percentiles (usec): | 1.00th=[16909], 5.00th=[16909], 10.00th=[16909], 20.00th=[16909], | 30.00th=[17957], 40.00th=[17957], 50.00th=[45351], 60.00th=[45351], | 70.00th=[47449], 80.00th=[47449], 90.00th=[85459], 95.00th=[85459], | 99.00th=[85459], 99.50th=[85459], 99.90th=[85459], 99.95th=[85459], | 99.99th=[85459] bw ( KiB/s): min= 8175, max= 8175, per=100.00%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 7, max= 7, avg= 7.00, stdev= 0.00, samples=1 write: IOPS=2, BW=2575KiB/s (2637kB/s)(7168KiB/2784msec) clat (msec): min=25, max=1629, avg=348.79, stdev=579.71 lat (msec): min=26, max=1629, avg=348.87, stdev=579.71 clat percentiles (msec): | 1.00th=[ 26], 5.00th=[ 26], 10.00th=[ 26], 20.00th=[ 37], | 30.00th=[ 52], 40.00th=[ 52], 50.00th=[ 90], 60.00th=[ 220], | 70.00th=[ 220], 80.00th=[ 393], 90.00th=[ 1636], 95.00th=[ 1636], | 99.00th=[ 1636], 99.50th=[ 1636], 99.90th=[ 1636], 99.95th=[ 1636], | 99.99th=[ 1636] bw ( KiB/s): min= 2048, max=10219, per=61.59%, avg=6133.50, stdev=5777.77, samples=2 iops : min= 2, max= 9, avg= 5.50, stdev= 4.95, samples=2 lat (msec) : 20=16.67%, 50=33.33%, 100=25.00%, 250=8.33%, 500=8.33% cpu : usr=0.11%, sys=2.84%, ctx=52, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=7114KiB/s (7284kB/s), 711KiB/s-3152KiB/s (728kB/s-3228kB/s), io=20.0MiB (20.0MB), run=2274-2879msec WRITE: bw=9959KiB/s (10.2MB/s), 2180KiB/s-3557KiB/s (2233kB/s-3642kB/s), io=28.0MiB (29.4MB), run=2274-2879msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=20937: Fri May 2 01:03:26 2025 read: IOPS=3, BW=13.8MiB/s (14.4MB/s)(12.0MiB/872msec) clat (msec): min=80, max=415, avg=273.09, stdev=173.05 lat (msec): min=80, max=415, avg=273.09, stdev=173.05 clat percentiles (msec): | 1.00th=[ 81], 5.00th=[ 81], 10.00th=[ 81], 20.00th=[ 81], | 30.00th=[ 81], 40.00th=[ 326], 50.00th=[ 326], 60.00th=[ 326], | 70.00th=[ 414], 80.00th=[ 414], 90.00th=[ 414], 95.00th=[ 414], | 99.00th=[ 414], 99.50th=[ 414], 99.90th=[ 414], 99.95th=[ 414], | 99.99th=[ 414] bw ( KiB/s): min= 8159, max= 8159, per=87.85%, avg=8159.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 500=66.67% cpu : usr=0.00%, sys=19.75%, ctx=322, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20938: Fri May 2 01:03:26 2025 read: IOPS=0, BW=2322KiB/s (2378kB/s)(4096KiB/1764msec) clat (nsec): min=360554k, max=360554k, avg=360553769.00, stdev= 0.00 lat (nsec): min=360557k, max=360557k, avg=360556532.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 359], 5.00th=[ 359], 10.00th=[ 359], 20.00th=[ 359], | 30.00th=[ 359], 40.00th=[ 359], 50.00th=[ 359], 60.00th=[ 359], | 70.00th=[ 359], 80.00th=[ 359], 90.00th=[ 359], 95.00th=[ 359], | 99.00th=[ 359], 99.50th=[ 359], 99.90th=[ 359], 99.95th=[ 359], | 99.99th=[ 359] bw ( KiB/s): min= 8192, max= 8192, per=88.21%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=1, BW=4644KiB/s (4755kB/s)(8192KiB/1764msec) clat (msec): min=509, max=877, avg=693.70, stdev=260.05 lat (msec): min=510, max=877, avg=693.99, stdev=260.01 clat percentiles (msec): | 1.00th=[ 510], 5.00th=[ 510], 10.00th=[ 510], 20.00th=[ 510], | 30.00th=[ 510], 40.00th=[ 510], 50.00th=[ 510], 60.00th=[ 877], | 70.00th=[ 877], 80.00th=[ 877], 90.00th=[ 877], 95.00th=[ 877], | 99.00th=[ 877], 99.50th=[ 877], 99.90th=[ 877], 99.95th=[ 877], | 99.99th=[ 877] bw ( KiB/s): min= 7656, max= 7656, per=41.22%, avg=7656.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 500=33.33%, 750=33.33%, 1000=33.33% cpu : usr=0.00%, sys=13.95%, ctx=321, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20939: Fri May 2 01:03:26 2025 write: IOPS=1, BW=8143KiB/s (8339kB/s)(12.0MiB/1509msec) clat (msec): min=367, max=712, avg=486.39, stdev=196.18 lat (msec): min=367, max=713, avg=486.71, stdev=196.21 clat percentiles (msec): | 1.00th=[ 368], 5.00th=[ 368], 10.00th=[ 368], 20.00th=[ 368], | 30.00th=[ 368], 40.00th=[ 380], 50.00th=[ 380], 60.00th=[ 380], | 70.00th=[ 709], 80.00th=[ 709], 90.00th=[ 709], 95.00th=[ 709], | 99.00th=[ 709], 99.50th=[ 709], 99.90th=[ 709], 99.95th=[ 709], | 99.99th=[ 709] bw ( KiB/s): min= 8192, max= 8192, per=44.10%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 500=66.67%, 750=33.33% cpu : usr=0.00%, sys=16.64%, ctx=94, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20941: Fri May 2 01:03:26 2025 write: IOPS=2, BW=9.80MiB/s (10.3MB/s)(12.0MiB/1225msec) clat (msec): min=267, max=618, avg=397.43, stdev=192.17 lat (msec): min=267, max=618, avg=397.98, stdev=191.91 clat percentiles (msec): | 1.00th=[ 268], 5.00th=[ 268], 10.00th=[ 268], 20.00th=[ 268], | 30.00th=[ 268], 40.00th=[ 309], 50.00th=[ 309], 60.00th=[ 309], | 70.00th=[ 617], 80.00th=[ 617], 90.00th=[ 617], 95.00th=[ 617], | 99.00th=[ 617], 99.50th=[ 617], 99.90th=[ 617], 99.95th=[ 617], | 99.99th=[ 617] bw ( KiB/s): min= 8192, max= 8192, per=44.10%, avg=8192.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 lat (msec) : 500=66.67%, 750=33.33% cpu : usr=0.25%, sys=23.28%, ctx=87, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=9288KiB/s (9511kB/s), 2322KiB/s-13.8MiB/s (2378kB/s-14.4MB/s), io=16.0MiB (16.8MB), run=872-1764msec WRITE: bw=18.1MiB/s (19.0MB/s), 4644KiB/s-9.80MiB/s (4755kB/s-10.3MB/s), io=32.0MiB (33.6MB), run=1225-1764msec rand-rw: (groupid=0, jobs=1): err= 0: pid=20940: Fri May 2 01:03:26 2025 read: IOPS=1, BW=7376KiB/s (7553kB/s)(12.0MiB/1666msec) clat (msec): min=272, max=785, avg=518.16, stdev=257.22 lat (msec): min=272, max=785, avg=518.16, stdev=257.22 clat percentiles (msec): | 1.00th=[ 271], 5.00th=[ 271], 10.00th=[ 271], 20.00th=[ 271], | 30.00th=[ 271], 40.00th=[ 498], 50.00th=[ 498], 60.00th=[ 498], | 70.00th=[ 785], 80.00th=[ 785], 90.00th=[ 785], 95.00th=[ 785], | 99.00th=[ 785], 99.50th=[ 785], 99.90th=[ 785], 99.95th=[ 785], | 99.99th=[ 785] bw ( KiB/s): min= 7984, max= 8192, per=82.25%, avg=8088.00, stdev=147.08, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 500=66.67%, 1000=33.33% cpu : usr=0.06%, sys=4.26%, ctx=46, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20942: Fri May 2 01:03:26 2025 read: IOPS=0, BW=2766KiB/s (2832kB/s)(4096KiB/1481msec) clat (nsec): min=259581k, max=259581k, avg=259580918.00, stdev= 0.00 lat (nsec): min=259584k, max=259584k, avg=259583782.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 259], 5.00th=[ 259], 10.00th=[ 259], 20.00th=[ 259], | 30.00th=[ 259], 40.00th=[ 259], 50.00th=[ 259], 60.00th=[ 259], | 70.00th=[ 259], 80.00th=[ 259], 90.00th=[ 259], 95.00th=[ 259], | 99.00th=[ 259], 99.50th=[ 259], 99.90th=[ 259], 99.95th=[ 259], | 99.99th=[ 259] bw ( KiB/s): min= 8175, max= 8175, per=83.13%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=1, BW=5531KiB/s (5664kB/s)(8192KiB/1481msec) clat (msec): min=547, max=628, avg=587.82, stdev=57.68 lat (msec): min=547, max=628, avg=588.10, stdev=57.68 clat percentiles (msec): | 1.00th=[ 550], 5.00th=[ 550], 10.00th=[ 550], 20.00th=[ 550], | 30.00th=[ 550], 40.00th=[ 550], 50.00th=[ 550], 60.00th=[ 625], | 70.00th=[ 625], 80.00th=[ 625], 90.00th=[ 625], 95.00th=[ 625], | 99.00th=[ 625], 99.50th=[ 625], 99.90th=[ 625], 99.95th=[ 625], | 99.99th=[ 625] bw ( KiB/s): min= 8192, max= 8192, per=42.43%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 500=33.33%, 750=66.67% cpu : usr=0.27%, sys=2.84%, ctx=80, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20943: Fri May 2 01:03:26 2025 write: IOPS=1, BW=7241KiB/s (7415kB/s)(12.0MiB/1697msec) clat (msec): min=193, max=815, avg=525.14, stdev=313.21 lat (msec): min=194, max=816, avg=525.89, stdev=312.87 clat percentiles (msec): | 1.00th=[ 194], 5.00th=[ 194], 10.00th=[ 194], 20.00th=[ 194], | 30.00th=[ 194], 40.00th=[ 567], 50.00th=[ 567], 60.00th=[ 567], | 70.00th=[ 818], 80.00th=[ 818], 90.00th=[ 818], 95.00th=[ 818], | 99.00th=[ 818], 99.50th=[ 818], 99.90th=[ 818], 99.95th=[ 818], | 99.99th=[ 818] bw ( KiB/s): min= 8175, max= 8175, per=42.34%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 750=33.33%, 1000=33.33% cpu : usr=0.18%, sys=1.30%, ctx=17, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20944: Fri May 2 01:03:26 2025 write: IOPS=1, BW=8159KiB/s (8355kB/s)(12.0MiB/1506msec) clat (msec): min=192, max=618, avg=472.65, stdev=242.35 lat (msec): min=193, max=618, avg=472.89, stdev=242.31 clat percentiles (msec): | 1.00th=[ 192], 5.00th=[ 192], 10.00th=[ 192], 20.00th=[ 192], | 30.00th=[ 192], 40.00th=[ 609], 50.00th=[ 609], 60.00th=[ 609], | 70.00th=[ 617], 80.00th=[ 617], 90.00th=[ 617], 95.00th=[ 617], | 99.00th=[ 617], 99.50th=[ 617], 99.90th=[ 617], 99.95th=[ 617], | 99.99th=[ 617] bw ( KiB/s): min=16384, max=16384, per=84.85%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 750=66.67% cpu : usr=0.07%, sys=5.78%, ctx=65, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=9834KiB/s (10.1MB/s), 2766KiB/s-7376KiB/s (2832kB/s-7553kB/s), io=16.0MiB (16.8MB), run=1481-1666msec WRITE: bw=18.9MiB/s (19.8MB/s), 5531KiB/s-8159KiB/s (5664kB/s-8355kB/s), io=32.0MiB (33.6MB), run=1481-1697msec