== sanity test 398b: DIO and buffer IO race ============== 18:28:54 (1748471334) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.56452 s, 32.2 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=25484: Wed May 28 18:29:17 2025 read: IOPS=76, BW=305KiB/s (313kB/s)(5992KiB/19617msec) clat (usec): min=55, max=33513, avg=7322.61, stdev=4418.34 lat (usec): min=56, max=33513, avg=7323.55, stdev=4418.30 clat percentiles (usec): | 1.00th=[ 95], 5.00th=[ 149], 10.00th=[ 3228], 20.00th=[ 4047], | 30.00th=[ 4752], 40.00th=[ 5538], 50.00th=[ 6456], 60.00th=[ 7504], | 70.00th=[ 8717], 80.00th=[10421], 90.00th=[13042], 95.00th=[15139], | 99.00th=[22152], 99.50th=[25297], 99.90th=[31851], 99.95th=[33424], | 99.99th=[33424] bw ( KiB/s): min= 160, max= 448, per=25.35%, avg=304.21, stdev=62.60, samples=39 iops : min= 40, max= 112, avg=75.90, stdev=15.68, samples=39 write: IOPS=80, BW=321KiB/s (329kB/s)(6296KiB/19617msec) clat (usec): min=433, max=215639, avg=5434.94, stdev=7600.16 lat (usec): min=434, max=215639, avg=5436.15, stdev=7600.13 clat percentiles (usec): | 1.00th=[ 545], 5.00th=[ 685], 10.00th=[ 742], 20.00th=[ 832], | 30.00th=[ 1004], 40.00th=[ 1729], 50.00th=[ 3916], 60.00th=[ 5145], | 70.00th=[ 6915], 80.00th=[ 8979], 90.00th=[ 12780], 95.00th=[ 15795], | 99.00th=[ 23200], 99.50th=[ 30278], 99.90th=[ 39060], 99.95th=[214959], | 99.99th=[214959] bw ( KiB/s): min= 135, max= 512, per=25.43%, avg=319.38, stdev=72.72, samples=39 iops : min= 33, max= 128, avg=79.69, stdev=18.25, samples=39 lat (usec) : 100=0.65%, 250=2.18%, 500=0.29%, 750=5.31%, 1000=9.86% lat (msec) : 2=6.12%, 4=10.74%, 10=45.90%, 20=16.96%, 50=1.95% lat (msec) : 250=0.03% cpu : usr=0.22%, sys=18.66%, ctx=5298, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25485: Wed May 28 18:29:17 2025 read: IOPS=75, BW=300KiB/s (307kB/s)(5940KiB/19783msec) clat (usec): min=65, max=43621, avg=7265.55, stdev=4747.80 lat (usec): min=65, max=43622, avg=7266.45, stdev=4747.91 clat percentiles (usec): | 1.00th=[ 94], 5.00th=[ 190], 10.00th=[ 3130], 20.00th=[ 3949], | 30.00th=[ 4686], 40.00th=[ 5342], 50.00th=[ 6194], 60.00th=[ 7308], | 70.00th=[ 8455], 80.00th=[10159], 90.00th=[12780], 95.00th=[15664], | 99.00th=[23725], 99.50th=[31065], 99.90th=[35914], 99.95th=[43779], | 99.99th=[43779] bw ( KiB/s): min= 152, max= 472, per=24.81%, avg=297.77, stdev=62.34, samples=39 iops : min= 38, max= 118, avg=74.23, stdev=15.61, samples=39 write: IOPS=80, BW=321KiB/s (329kB/s)(6348KiB/19783msec) clat (usec): min=438, max=223063, avg=5604.75, stdev=7780.78 lat (usec): min=439, max=223064, avg=5605.86, stdev=7780.77 clat percentiles (usec): | 1.00th=[ 537], 5.00th=[ 676], 10.00th=[ 750], 20.00th=[ 832], | 30.00th=[ 1037], 40.00th=[ 1893], 50.00th=[ 4047], 60.00th=[ 5473], | 70.00th=[ 7177], 80.00th=[ 9372], 90.00th=[ 12911], 95.00th=[ 16057], | 99.00th=[ 23987], 99.50th=[ 27395], 99.90th=[ 58983], 99.95th=[223347], | 99.99th=[223347] bw ( KiB/s): min= 216, max= 480, per=25.05%, avg=314.59, stdev=61.82, samples=39 iops : min= 54, max= 120, avg=78.44, stdev=15.46, samples=39 lat (usec) : 100=0.91%, 250=1.60%, 500=0.33%, 750=5.05%, 1000=9.83% lat (msec) : 2=6.02%, 4=11.85%, 10=45.05%, 20=17.25%, 50=2.05% lat (msec) : 100=0.03%, 250=0.03% cpu : usr=0.18%, sys=18.41%, ctx=5369, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25486: Wed May 28 18:29:17 2025 read: IOPS=75, BW=301KiB/s (308kB/s)(6028KiB/20010msec) clat (usec): min=52, max=45123, avg=7050.23, stdev=4317.87 lat (usec): min=53, max=45124, avg=7051.46, stdev=4317.80 clat percentiles (usec): | 1.00th=[ 89], 5.00th=[ 121], 10.00th=[ 3097], 20.00th=[ 3982], | 30.00th=[ 4621], 40.00th=[ 5407], 50.00th=[ 6194], 60.00th=[ 7111], | 70.00th=[ 8586], 80.00th=[10028], 90.00th=[12387], 95.00th=[14484], | 99.00th=[21890], 99.50th=[25035], 99.90th=[34866], 99.95th=[45351], | 99.99th=[45351] bw ( KiB/s): min= 88, max= 424, per=24.69%, avg=296.31, stdev=63.28, samples=39 iops : min= 22, max= 106, avg=74.00, stdev=15.85, samples=39 write: IOPS=78, BW=313KiB/s (320kB/s)(6260KiB/20010msec) clat (usec): min=390, max=227093, avg=5928.20, stdev=7969.19 lat (usec): min=391, max=227095, avg=5929.28, stdev=7969.22 clat percentiles (usec): | 1.00th=[ 537], 5.00th=[ 693], 10.00th=[ 758], 20.00th=[ 848], | 30.00th=[ 1106], 40.00th=[ 2278], 50.00th=[ 4555], 60.00th=[ 5932], | 70.00th=[ 7439], 80.00th=[ 9896], 90.00th=[ 13698], 95.00th=[ 16712], | 99.00th=[ 25822], 99.50th=[ 29230], 99.90th=[ 40109], 99.95th=[227541], | 99.99th=[227541] bw ( KiB/s): min= 88, max= 416, per=24.36%, avg=305.95, stdev=69.03, samples=39 iops : min= 22, max= 104, avg=76.41, stdev=17.24, samples=39 lat (usec) : 100=1.20%, 250=1.86%, 500=0.36%, 750=4.62%, 1000=9.08% lat (msec) : 2=5.86%, 4=11.04%, 10=46.19%, 20=17.94%, 50=1.82% lat (msec) : 250=0.03% cpu : usr=0.25%, sys=18.22%, ctx=5214, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25489: Wed May 28 18:29:17 2025 read: IOPS=76, BW=306KiB/s (313kB/s)(6052KiB/19803msec) clat (usec): min=60, max=43828, avg=7201.59, stdev=4454.02 lat (usec): min=61, max=43829, avg=7203.93, stdev=4453.74 clat percentiles (usec): | 1.00th=[ 92], 5.00th=[ 717], 10.00th=[ 3294], 20.00th=[ 4047], | 30.00th=[ 4621], 40.00th=[ 5342], 50.00th=[ 6194], 60.00th=[ 7308], | 70.00th=[ 8356], 80.00th=[10028], 90.00th=[12649], 95.00th=[15139], | 99.00th=[22938], 99.50th=[25035], 99.90th=[33424], 99.95th=[43779], | 99.99th=[43779] bw ( KiB/s): min= 152, max= 432, per=25.14%, avg=301.64, stdev=64.03, samples=39 iops : min= 38, max= 108, avg=75.33, stdev=16.04, samples=39 write: IOPS=78, BW=315KiB/s (322kB/s)(6236KiB/19803msec) clat (usec): min=437, max=215069, avg=5645.77, stdev=7683.01 lat (usec): min=437, max=215070, avg=5647.12, stdev=7682.83 clat percentiles (usec): | 1.00th=[ 529], 5.00th=[ 693], 10.00th=[ 758], 20.00th=[ 824], | 30.00th=[ 996], 40.00th=[ 1844], 50.00th=[ 4047], 60.00th=[ 5473], | 70.00th=[ 7242], 80.00th=[ 9503], 90.00th=[ 13698], 95.00th=[ 16450], | 99.00th=[ 23462], 99.50th=[ 28705], 99.90th=[ 39060], 99.95th=[214959], | 99.99th=[214959] bw ( KiB/s): min= 184, max= 488, per=24.77%, avg=311.08, stdev=65.63, samples=39 iops : min= 46, max= 122, avg=77.69, stdev=16.43, samples=39 lat (usec) : 100=0.75%, 250=1.50%, 500=0.36%, 750=4.69%, 1000=10.45% lat (msec) : 2=5.66%, 4=11.30%, 10=46.00%, 20=17.25%, 50=2.02% lat (msec) : 250=0.03% cpu : usr=0.11%, sys=18.33%, ctx=5399, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1200KiB/s (1229kB/s), 300KiB/s-306KiB/s (307kB/s-313kB/s), io=23.4MiB (24.6MB), run=19617-20010msec WRITE: bw=1256KiB/s (1287kB/s), 313KiB/s-321KiB/s (320kB/s-329kB/s), io=24.6MiB (25.7MB), run=19617-20010msec rand-rw: (groupid=0, jobs=1): err= 0: pid=25487: Wed May 28 18:29:24 2025 read: IOPS=57, BW=228KiB/s (234kB/s)(5992KiB/26251msec) clat (usec): min=1711, max=228384, avg=6794.43, stdev=8357.47 lat (usec): min=1712, max=228387, avg=6795.60, stdev=8357.43 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 3], 20.00th=[ 3], | 30.00th=[ 4], 40.00th=[ 4], 50.00th=[ 5], 60.00th=[ 5], | 70.00th=[ 7], 80.00th=[ 10], 90.00th=[ 16], 95.00th=[ 20], | 99.00th=[ 32], 99.50th=[ 36], 99.90th=[ 45], 99.95th=[ 228], | 99.99th=[ 228] bw ( KiB/s): min= 79, max= 520, per=24.74%, avg=224.90, stdev=134.25, samples=52 iops : min= 19, max= 130, avg=56.10, stdev=33.58, samples=52 write: IOPS=59, BW=240KiB/s (246kB/s)(6296KiB/26251msec) clat (usec): min=2561, max=56580, avg=10144.02, stdev=7175.61 lat (usec): min=2562, max=56581, avg=10145.11, stdev=7175.63 clat percentiles (usec): | 1.00th=[ 3097], 5.00th=[ 3720], 10.00th=[ 4228], 20.00th=[ 4752], | 30.00th=[ 5407], 40.00th=[ 6194], 50.00th=[ 7439], 60.00th=[ 9110], | 70.00th=[11731], 80.00th=[15139], 90.00th=[19792], 95.00th=[24249], | 99.00th=[36963], 99.50th=[41681], 99.90th=[53216], 99.95th=[56361], | 99.99th=[56361] bw ( KiB/s): min= 64, max= 576, per=24.86%, avg=236.44, stdev=142.11, samples=52 iops : min= 16, max= 144, avg=58.98, stdev=35.55, samples=52 lat (msec) : 2=0.33%, 4=26.92%, 10=44.89%, 20=20.54%, 50=7.23% lat (msec) : 100=0.07%, 250=0.03% cpu : usr=0.21%, sys=9.06%, ctx=3623, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25488: Wed May 28 18:29:24 2025 read: IOPS=56, BW=226KiB/s (231kB/s)(5940KiB/26311msec) clat (usec): min=1802, max=227776, avg=6546.87, stdev=8357.21 lat (usec): min=1803, max=227778, avg=6547.77, stdev=8357.25 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 3], 20.00th=[ 3], | 30.00th=[ 3], 40.00th=[ 4], 50.00th=[ 4], 60.00th=[ 5], | 70.00th=[ 6], 80.00th=[ 9], 90.00th=[ 15], 95.00th=[ 20], | 99.00th=[ 32], 99.50th=[ 37], 99.90th=[ 51], 99.95th=[ 228], | 99.99th=[ 228] bw ( KiB/s): min= 31, max= 560, per=24.59%, avg=223.50, stdev=143.68, samples=52 iops : min= 7, max= 140, avg=55.73, stdev=35.98, samples=52 write: IOPS=60, BW=241KiB/s (247kB/s)(6348KiB/26311msec) clat (usec): min=2622, max=53776, avg=10399.10, stdev=7638.41 lat (usec): min=2623, max=53778, avg=10400.55, stdev=7638.29 clat percentiles (usec): | 1.00th=[ 3294], 5.00th=[ 3818], 10.00th=[ 4178], 20.00th=[ 4817], | 30.00th=[ 5407], 40.00th=[ 6128], 50.00th=[ 7177], 60.00th=[ 9110], | 70.00th=[11731], 80.00th=[15926], 90.00th=[21103], 95.00th=[24773], | 99.00th=[38011], 99.50th=[42730], 99.90th=[53740], 99.95th=[53740], | 99.99th=[53740] bw ( KiB/s): min= 87, max= 576, per=24.81%, avg=235.96, stdev=142.47, samples=52 iops : min= 21, max= 144, avg=58.85, stdev=35.67, samples=52 lat (msec) : 2=0.33%, 4=28.29%, 10=44.47%, 20=18.72%, 50=8.04% lat (msec) : 100=0.13%, 250=0.03% cpu : usr=0.20%, sys=9.26%, ctx=3602, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25490: Wed May 28 18:29:24 2025 read: IOPS=57, BW=229KiB/s (234kB/s)(6028KiB/26331msec) clat (usec): min=1802, max=45985, avg=6657.39, stdev=6115.87 lat (usec): min=1803, max=45986, avg=6658.36, stdev=6115.91 clat percentiles (usec): | 1.00th=[ 2024], 5.00th=[ 2245], 10.00th=[ 2409], 20.00th=[ 2671], | 30.00th=[ 3032], 40.00th=[ 3425], 50.00th=[ 4080], 60.00th=[ 5145], | 70.00th=[ 6456], 80.00th=[ 9503], 90.00th=[15533], 95.00th=[20055], | 99.00th=[29754], 99.50th=[32637], 99.90th=[39060], 99.95th=[45876], | 99.99th=[45876] bw ( KiB/s): min= 64, max= 552, per=24.77%, avg=225.12, stdev=136.96, samples=52 iops : min= 16, max= 138, avg=56.21, stdev=34.25, samples=52 write: IOPS=59, BW=238KiB/s (243kB/s)(6260KiB/26331msec) clat (msec): min=2, max=233, avg=10.36, stdev= 9.33 lat (msec): min=2, max=233, avg=10.36, stdev= 9.33 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 4], 10.00th=[ 5], 20.00th=[ 5], | 30.00th=[ 6], 40.00th=[ 7], 50.00th=[ 8], 60.00th=[ 9], | 70.00th=[ 12], 80.00th=[ 16], 90.00th=[ 21], 95.00th=[ 25], | 99.00th=[ 37], 99.50th=[ 42], 99.90th=[ 72], 99.95th=[ 234], | 99.99th=[ 234] bw ( KiB/s): min= 80, max= 542, per=24.48%, avg=232.81, stdev=144.41, samples=52 iops : min= 20, max= 135, avg=58.13, stdev=36.11, samples=52 lat (msec) : 2=0.36%, 4=27.44%, 10=44.89%, 20=19.66%, 50=7.49% lat (msec) : 100=0.13%, 250=0.03% cpu : usr=0.14%, sys=9.25%, ctx=3588, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25491: Wed May 28 18:29:24 2025 read: IOPS=57, BW=229KiB/s (235kB/s)(6052KiB/26413msec) clat (usec): min=1865, max=42724, avg=6682.43, stdev=6285.60 lat (usec): min=1865, max=42725, avg=6683.44, stdev=6285.63 clat percentiles (usec): | 1.00th=[ 2008], 5.00th=[ 2278], 10.00th=[ 2442], 20.00th=[ 2704], | 30.00th=[ 2999], 40.00th=[ 3425], 50.00th=[ 3982], 60.00th=[ 4883], | 70.00th=[ 6325], 80.00th=[ 9241], 90.00th=[16057], 95.00th=[20317], | 99.00th=[30016], 99.50th=[33424], 99.90th=[42206], 99.95th=[42730], | 99.99th=[42730] bw ( KiB/s): min= 56, max= 550, per=24.54%, avg=223.04, stdev=139.54, samples=52 iops : min= 14, max= 137, avg=55.63, stdev=34.91, samples=52 write: IOPS=59, BW=236KiB/s (242kB/s)(6236KiB/26413msec) clat (msec): min=2, max=211, avg=10.38, stdev= 8.63 lat (msec): min=2, max=211, avg=10.38, stdev= 8.63 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 4], 10.00th=[ 5], 20.00th=[ 5], | 30.00th=[ 6], 40.00th=[ 7], 50.00th=[ 8], 60.00th=[ 10], | 70.00th=[ 13], 80.00th=[ 16], 90.00th=[ 21], 95.00th=[ 24], | 99.00th=[ 34], 99.50th=[ 39], 99.90th=[ 70], 99.95th=[ 211], | 99.99th=[ 211] bw ( KiB/s): min= 64, max= 568, per=24.23%, avg=230.44, stdev=142.18, samples=52 iops : min= 16, max= 142, avg=57.48, stdev=35.58, samples=52 lat (msec) : 2=0.49%, 4=28.06%, 10=42.71%, 20=20.64%, 50=8.04% lat (msec) : 100=0.03%, 250=0.03% cpu : usr=0.19%, sys=8.98%, ctx=3631, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=909KiB/s (931kB/s), 226KiB/s-229KiB/s (231kB/s-235kB/s), io=23.4MiB (24.6MB), run=26251-26413msec WRITE: bw=952KiB/s (975kB/s), 236KiB/s-241KiB/s (242kB/s-247kB/s), io=24.6MiB (25.7MB), run=26251-26413msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=25528: Wed May 28 18:29:29 2025 read: IOPS=89, BW=1432KiB/s (1467kB/s)(5920KiB/4133msec) clat (usec): min=56, max=22784, avg=5856.75, stdev=3583.84 lat (usec): min=56, max=22785, avg=5857.58, stdev=3583.86 clat percentiles (usec): | 1.00th=[ 68], 5.00th=[ 115], 10.00th=[ 2835], 20.00th=[ 3720], | 30.00th=[ 4080], 40.00th=[ 4555], 50.00th=[ 5342], 60.00th=[ 6063], | 70.00th=[ 6783], 80.00th=[ 7832], 90.00th=[ 9634], 95.00th=[11731], | 99.00th=[20055], 99.50th=[22676], 99.90th=[22676], 99.95th=[22676], | 99.99th=[22676] bw ( KiB/s): min= 1248, max= 1632, per=24.23%, avg=1403.62, stdev=119.15, samples=8 iops : min= 78, max= 102, avg=87.62, stdev= 7.48, samples=8 write: IOPS=96, BW=1541KiB/s (1578kB/s)(6368KiB/4133msec) clat (usec): min=670, max=24039, avg=4781.79, stdev=4035.90 lat (usec): min=670, max=24041, avg=4784.66, stdev=4034.94 clat percentiles (usec): | 1.00th=[ 750], 5.00th=[ 955], 10.00th=[ 1057], 20.00th=[ 1188], | 30.00th=[ 1467], 40.00th=[ 2638], 50.00th=[ 3949], 60.00th=[ 4817], | 70.00th=[ 6259], 80.00th=[ 8094], 90.00th=[10028], 95.00th=[11863], | 99.00th=[19006], 99.50th=[23200], 99.90th=[23987], 99.95th=[23987], | 99.99th=[23987] bw ( KiB/s): min= 1120, max= 1888, per=25.19%, avg=1535.62, stdev=265.69, samples=8 iops : min= 70, max= 118, avg=95.88, stdev=16.65, samples=8 lat (usec) : 100=1.82%, 250=1.82%, 500=0.26%, 750=0.39%, 1000=3.65% lat (msec) : 2=15.23%, 4=16.28%, 10=50.78%, 20=8.98%, 50=0.78% cpu : usr=0.12%, sys=23.38%, ctx=1530, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25529: Wed May 28 18:29:29 2025 read: IOPS=87, BW=1395KiB/s (1429kB/s)(5744KiB/4117msec) clat (usec): min=52, max=21838, avg=5986.25, stdev=3796.31 lat (usec): min=52, max=21838, avg=5987.19, stdev=3796.26 clat percentiles (usec): | 1.00th=[ 80], 5.00th=[ 109], 10.00th=[ 2343], 20.00th=[ 3490], | 30.00th=[ 3949], 40.00th=[ 4621], 50.00th=[ 5407], 60.00th=[ 6259], | 70.00th=[ 7242], 80.00th=[ 8160], 90.00th=[10421], 95.00th=[13304], | 99.00th=[19268], 99.50th=[20055], 99.90th=[21890], 99.95th=[21890], | 99.99th=[21890] bw ( KiB/s): min= 1056, max= 1664, per=23.82%, avg=1379.63, stdev=217.84, samples=8 iops : min= 66, max= 104, avg=86.13, stdev=13.76, samples=8 write: IOPS=99, BW=1590KiB/s (1628kB/s)(6544KiB/4117msec) clat (usec): min=716, max=22493, avg=4755.43, stdev=4049.23 lat (usec): min=718, max=22494, avg=4757.85, stdev=4048.96 clat percentiles (usec): | 1.00th=[ 824], 5.00th=[ 955], 10.00th=[ 1045], 20.00th=[ 1221], | 30.00th=[ 1500], 40.00th=[ 2933], 50.00th=[ 3687], 60.00th=[ 4752], | 70.00th=[ 5800], 80.00th=[ 7504], 90.00th=[10290], 95.00th=[12780], | 99.00th=[18744], 99.50th=[20317], 99.90th=[22414], 99.95th=[22414], | 99.99th=[22414] bw ( KiB/s): min= 1344, max= 1856, per=25.32%, avg=1543.50, stdev=168.60, samples=8 iops : min= 84, max= 116, avg=96.38, stdev=10.45, samples=8 lat (usec) : 100=1.17%, 250=3.12%, 500=0.13%, 750=0.39%, 1000=3.78% lat (msec) : 2=14.84%, 4=19.01%, 10=46.74%, 20=10.16%, 50=0.65% cpu : usr=0.15%, sys=23.71%, ctx=1555, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25530: Wed May 28 18:29:29 2025 read: IOPS=94, BW=1513KiB/s (1550kB/s)(6256KiB/4134msec) clat (usec): min=57, max=23545, avg=6058.24, stdev=3807.88 lat (usec): min=57, max=23545, avg=6059.94, stdev=3807.90 clat percentiles (usec): | 1.00th=[ 87], 5.00th=[ 104], 10.00th=[ 188], 20.00th=[ 3654], | 30.00th=[ 4293], 40.00th=[ 5014], 50.00th=[ 5669], 60.00th=[ 6259], | 70.00th=[ 7046], 80.00th=[ 8225], 90.00th=[10421], 95.00th=[12911], | 99.00th=[20317], 99.50th=[22938], 99.90th=[23462], 99.95th=[23462], | 99.99th=[23462] bw ( KiB/s): min= 1152, max= 2144, per=26.10%, avg=1512.00, stdev=311.55, samples=8 iops : min= 72, max= 134, avg=94.50, stdev=19.47, samples=8 write: IOPS=91, BW=1459KiB/s (1494kB/s)(6032KiB/4134msec) clat (usec): min=744, max=19467, avg=4484.83, stdev=3897.17 lat (usec): min=745, max=19468, avg=4486.10, stdev=3897.18 clat percentiles (usec): | 1.00th=[ 807], 5.00th=[ 914], 10.00th=[ 1045], 20.00th=[ 1172], | 30.00th=[ 1319], 40.00th=[ 2040], 50.00th=[ 3654], 60.00th=[ 4293], | 70.00th=[ 5211], 80.00th=[ 7308], 90.00th=[10028], 95.00th=[12518], | 99.00th=[17433], 99.50th=[18744], 99.90th=[19530], 99.95th=[19530], | 99.99th=[19530] bw ( KiB/s): min= 1216, max= 1760, per=23.17%, avg=1412.00, stdev=188.49, samples=8 iops : min= 76, max= 110, avg=88.25, stdev=11.78, samples=8 lat (usec) : 100=1.82%, 250=3.65%, 750=0.13%, 1000=4.04% lat (msec) : 2=15.10%, 4=14.97%, 10=50.00%, 20=9.64%, 50=0.65% cpu : usr=0.24%, sys=23.13%, ctx=1546, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25531: Wed May 28 18:29:29 2025 read: IOPS=91, BW=1465KiB/s (1500kB/s)(6032KiB/4118msec) clat (usec): min=63, max=25969, avg=5713.54, stdev=3888.44 lat (usec): min=63, max=25970, avg=5714.36, stdev=3888.56 clat percentiles (usec): | 1.00th=[ 75], 5.00th=[ 96], 10.00th=[ 123], 20.00th=[ 3195], | 30.00th=[ 3818], 40.00th=[ 4424], 50.00th=[ 5145], 60.00th=[ 5932], | 70.00th=[ 6980], 80.00th=[ 8225], 90.00th=[10290], 95.00th=[12518], | 99.00th=[18220], 99.50th=[19530], 99.90th=[26084], 99.95th=[26084], | 99.99th=[26084] bw ( KiB/s): min= 1216, max= 1664, per=24.64%, avg=1427.63, stdev=153.85, samples=8 iops : min= 76, max= 104, avg=89.13, stdev= 9.61, samples=8 write: IOPS=94, BW=1519KiB/s (1556kB/s)(6256KiB/4118msec) clat (usec): min=775, max=19519, avg=4870.44, stdev=4224.36 lat (usec): min=776, max=19520, avg=4871.65, stdev=4224.38 clat percentiles (usec): | 1.00th=[ 799], 5.00th=[ 898], 10.00th=[ 1004], 20.00th=[ 1123], | 30.00th=[ 1303], 40.00th=[ 2376], 50.00th=[ 3621], 60.00th=[ 4686], | 70.00th=[ 6325], 80.00th=[ 8455], 90.00th=[11076], 95.00th=[13566], | 99.00th=[18744], 99.50th=[18744], 99.90th=[19530], 99.95th=[19530], | 99.99th=[19530] bw ( KiB/s): min= 1152, max= 1760, per=24.47%, avg=1491.63, stdev=178.89, samples=8 iops : min= 72, max= 110, avg=93.12, stdev=11.28, samples=8 lat (usec) : 100=2.60%, 250=3.52%, 500=0.13%, 750=0.13%, 1000=4.82% lat (msec) : 2=14.71%, 4=18.10%, 10=43.88%, 20=11.98%, 50=0.13% cpu : usr=0.02%, sys=22.95%, ctx=1544, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=5794KiB/s (5933kB/s), 1395KiB/s-1513KiB/s (1429kB/s-1550kB/s), io=23.4MiB (24.5MB), run=4117-4134msec WRITE: bw=6096KiB/s (6242kB/s), 1459KiB/s-1590KiB/s (1494kB/s-1628kB/s), io=24.6MiB (25.8MB), run=4117-4134msec rand-rw: (groupid=0, jobs=1): err= 0: pid=25532: Wed May 28 18:29:30 2025 read: IOPS=68, BW=1102KiB/s (1128kB/s)(5920KiB/5373msec) clat (usec): min=1932, max=21677, avg=5512.89, stdev=4335.00 lat (usec): min=1933, max=21677, avg=5513.82, stdev=4335.01 clat percentiles (usec): | 1.00th=[ 2040], 5.00th=[ 2212], 10.00th=[ 2376], 20.00th=[ 2606], | 30.00th=[ 2900], 40.00th=[ 3294], 50.00th=[ 3687], 60.00th=[ 4359], | 70.00th=[ 5080], 80.00th=[ 7570], 90.00th=[12911], 95.00th=[16188], | 99.00th=[19530], 99.50th=[21627], 99.90th=[21627], 99.95th=[21627], | 99.99th=[21627] bw ( KiB/s): min= 608, max= 2144, per=22.25%, avg=991.60, stdev=450.49, samples=10 iops : min= 38, max= 134, avg=61.80, stdev=28.25, samples=10 write: IOPS=74, BW=1185KiB/s (1214kB/s)(6368KiB/5373msec) clat (usec): min=2450, max=34119, avg=8170.83, stdev=5549.22 lat (usec): min=2451, max=34121, avg=8172.31, stdev=5549.26 clat percentiles (usec): | 1.00th=[ 2769], 5.00th=[ 3359], 10.00th=[ 3556], 20.00th=[ 4047], | 30.00th=[ 4490], 40.00th=[ 4948], 50.00th=[ 5932], 60.00th=[ 7111], | 70.00th=[ 9503], 80.00th=[12125], 90.00th=[15795], 95.00th=[19268], | 99.00th=[26870], 99.50th=[28705], 99.90th=[34341], 99.95th=[34341], | 99.99th=[34341] bw ( KiB/s): min= 576, max= 2176, per=23.26%, avg=1090.70, stdev=495.91, samples=10 iops : min= 36, max= 136, avg=68.00, stdev=31.02, samples=10 lat (msec) : 2=0.26%, 4=36.46%, 10=42.19%, 20=18.36%, 50=2.73% cpu : usr=0.13%, sys=11.32%, ctx=925, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25533: Wed May 28 18:29:30 2025 read: IOPS=67, BW=1076KiB/s (1102kB/s)(5744KiB/5337msec) clat (usec): min=1929, max=23899, avg=5504.28, stdev=4257.83 lat (usec): min=1930, max=23900, avg=5505.17, stdev=4257.84 clat percentiles (usec): | 1.00th=[ 2008], 5.00th=[ 2212], 10.00th=[ 2376], 20.00th=[ 2671], | 30.00th=[ 2999], 40.00th=[ 3392], 50.00th=[ 3785], 60.00th=[ 4228], | 70.00th=[ 5276], 80.00th=[ 7439], 90.00th=[12125], 95.00th=[16188], | 99.00th=[20317], 99.50th=[21890], 99.90th=[23987], 99.95th=[23987], | 99.99th=[23987] bw ( KiB/s): min= 608, max= 1756, per=22.61%, avg=1007.60, stdev=409.89, samples=10 iops : min= 38, max= 109, avg=62.90, stdev=25.47, samples=10 write: IOPS=76, BW=1226KiB/s (1256kB/s)(6544KiB/5337msec) clat (usec): min=2764, max=31259, avg=8167.12, stdev=5159.78 lat (usec): min=2765, max=31261, avg=8168.82, stdev=5159.71 clat percentiles (usec): | 1.00th=[ 3097], 5.00th=[ 3458], 10.00th=[ 3785], 20.00th=[ 4228], | 30.00th=[ 4752], 40.00th=[ 5276], 50.00th=[ 6194], 60.00th=[ 7570], | 70.00th=[ 9110], 80.00th=[11863], 90.00th=[15533], 95.00th=[17957], | 99.00th=[26346], 99.50th=[27657], 99.90th=[31327], 99.95th=[31327], | 99.99th=[31327] bw ( KiB/s): min= 832, max= 2331, per=24.28%, avg=1138.70, stdev=448.09, samples=10 iops : min= 52, max= 145, avg=71.10, stdev=27.80, samples=10 lat (msec) : 2=0.39%, 4=32.68%, 10=46.48%, 20=17.71%, 50=2.73% cpu : usr=0.24%, sys=11.17%, ctx=895, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25534: Wed May 28 18:29:30 2025 read: IOPS=72, BW=1166KiB/s (1194kB/s)(6256KiB/5367msec) clat (usec): min=1798, max=26042, avg=5542.89, stdev=4454.11 lat (usec): min=1799, max=26042, avg=5543.72, stdev=4454.13 clat percentiles (usec): | 1.00th=[ 1975], 5.00th=[ 2212], 10.00th=[ 2311], 20.00th=[ 2540], | 30.00th=[ 2868], 40.00th=[ 3294], 50.00th=[ 3687], 60.00th=[ 4178], | 70.00th=[ 5145], 80.00th=[ 7701], 90.00th=[13173], 95.00th=[15795], | 99.00th=[20841], 99.50th=[22938], 99.90th=[26084], 99.95th=[26084], | 99.99th=[26084] bw ( KiB/s): min= 640, max= 2011, per=23.47%, avg=1045.90, stdev=487.95, samples=10 iops : min= 40, max= 125, avg=65.30, stdev=30.35, samples=10 write: IOPS=70, BW=1124KiB/s (1151kB/s)(6032KiB/5367msec) clat (usec): min=2631, max=32290, avg=8265.05, stdev=5313.91 lat (usec): min=2632, max=32292, avg=8266.75, stdev=5314.15 clat percentiles (usec): | 1.00th=[ 3032], 5.00th=[ 3392], 10.00th=[ 3621], 20.00th=[ 4047], | 30.00th=[ 4555], 40.00th=[ 5080], 50.00th=[ 6194], 60.00th=[ 7635], | 70.00th=[10028], 80.00th=[12125], 90.00th=[15533], 95.00th=[19268], | 99.00th=[24249], 99.50th=[31327], 99.90th=[32375], 99.95th=[32375], | 99.99th=[32375] bw ( KiB/s): min= 672, max= 2235, per=22.10%, avg=1036.30, stdev=454.64, samples=10 iops : min= 42, max= 139, avg=64.70, stdev=28.21, samples=10 lat (msec) : 2=0.52%, 4=36.20%, 10=40.76%, 20=20.18%, 50=2.34% cpu : usr=0.24%, sys=11.01%, ctx=906, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25535: Wed May 28 18:29:30 2025 read: IOPS=70, BW=1123KiB/s (1150kB/s)(6032KiB/5373msec) clat (usec): min=1713, max=23586, avg=5963.40, stdev=4906.51 lat (usec): min=1714, max=23587, avg=5964.22, stdev=4906.53 clat percentiles (usec): | 1.00th=[ 2024], 5.00th=[ 2180], 10.00th=[ 2311], 20.00th=[ 2540], | 30.00th=[ 2900], 40.00th=[ 3326], 50.00th=[ 3818], 60.00th=[ 4555], | 70.00th=[ 5669], 80.00th=[ 8848], 90.00th=[14222], 95.00th=[17171], | 99.00th=[22152], 99.50th=[23462], 99.90th=[23462], 99.95th=[23462], | 99.99th=[23462] bw ( KiB/s): min= 608, max= 2112, per=22.96%, avg=1023.40, stdev=451.12, samples=10 iops : min= 38, max= 132, avg=63.70, stdev=28.34, samples=10 write: IOPS=72, BW=1164KiB/s (1192kB/s)(6256KiB/5373msec) clat (usec): min=2682, max=34293, avg=7817.06, stdev=5188.62 lat (usec): min=2683, max=34295, avg=7818.90, stdev=5188.81 clat percentiles (usec): | 1.00th=[ 2933], 5.00th=[ 3392], 10.00th=[ 3589], 20.00th=[ 4015], | 30.00th=[ 4424], 40.00th=[ 5145], 50.00th=[ 5932], 60.00th=[ 6980], | 70.00th=[ 8717], 80.00th=[10814], 90.00th=[14484], 95.00th=[19792], | 99.00th=[27657], 99.50th=[29492], 99.90th=[34341], 99.95th=[34341], | 99.99th=[34341] bw ( KiB/s): min= 736, max= 2240, per=22.71%, avg=1065.00, stdev=493.19, samples=10 iops : min= 46, max= 140, avg=66.30, stdev=30.92, samples=10 lat (msec) : 2=0.26%, 4=35.16%, 10=43.49%, 20=17.71%, 50=3.39% cpu : usr=0.35%, sys=10.95%, ctx=936, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=4458KiB/s (4565kB/s), 1076KiB/s-1166KiB/s (1102kB/s-1194kB/s), io=23.4MiB (24.5MB), run=5337-5373msec WRITE: bw=4690KiB/s (4803kB/s), 1124KiB/s-1226KiB/s (1151kB/s-1256kB/s), io=24.6MiB (25.8MB), run=5337-5373msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=25546: Wed May 28 18:29:32 2025 read: IOPS=16, BW=16.3MiB/s (17.1MB/s)(7168KiB/430msec) clat (usec): min=458, max=56123, avg=30798.87, stdev=22272.13 lat (usec): min=459, max=56125, avg=30800.57, stdev=22272.09 clat percentiles (usec): | 1.00th=[ 457], 5.00th=[ 457], 10.00th=[ 457], 20.00th=[ 644], | 30.00th=[33817], 40.00th=[33817], 50.00th=[33817], 60.00th=[40109], | 70.00th=[40109], 80.00th=[51119], 90.00th=[56361], 95.00th=[56361], | 99.00th=[56361], 99.50th=[56361], 99.90th=[56361], 99.95th=[56361], | 99.99th=[56361] write: IOPS=11, BW=11.6MiB/s (12.2MB/s)(5120KiB/430msec) clat (usec): min=28476, max=67865, avg=40318.44, stdev=16219.05 lat (usec): min=28509, max=67881, avg=40349.32, stdev=16210.96 clat percentiles (usec): | 1.00th=[28443], 5.00th=[28443], 10.00th=[28443], 20.00th=[28443], | 30.00th=[29492], 40.00th=[29492], 50.00th=[34866], 60.00th=[34866], | 70.00th=[41157], 80.00th=[41157], 90.00th=[67634], 95.00th=[67634], | 99.00th=[67634], 99.50th=[67634], 99.90th=[67634], 99.95th=[67634], | 99.99th=[67634] lat (usec) : 500=8.33%, 750=8.33% lat (msec) : 50=58.33%, 100=25.00% cpu : usr=0.23%, sys=53.15%, ctx=703, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25548: Wed May 28 18:29:32 2025 read: IOPS=2, BW=2231KiB/s (2284kB/s)(2048KiB/918msec) clat (usec): min=32214, max=34157, avg=33185.87, stdev=1374.05 lat (usec): min=32216, max=34159, avg=33187.91, stdev=1373.56 clat percentiles (usec): | 1.00th=[32113], 5.00th=[32113], 10.00th=[32113], 20.00th=[32113], | 30.00th=[32113], 40.00th=[32113], 50.00th=[32113], 60.00th=[34341], | 70.00th=[34341], 80.00th=[34341], 90.00th=[34341], 95.00th=[34341], | 99.00th=[34341], 99.50th=[34341], 99.90th=[34341], 99.95th=[34341], | 99.99th=[34341] bw ( KiB/s): min= 4096, max= 4096, per=18.36%, avg=4096.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 write: IOPS=10, BW=10.9MiB/s (11.4MB/s)(10.0MiB/918msec) clat (msec): min=31, max=264, avg=81.42, stdev=67.03 lat (msec): min=31, max=264, avg=81.50, stdev=67.03 clat percentiles (msec): | 1.00th=[ 32], 5.00th=[ 32], 10.00th=[ 32], 20.00th=[ 48], | 30.00th=[ 48], 40.00th=[ 54], 50.00th=[ 57], 60.00th=[ 66], | 70.00th=[ 79], 80.00th=[ 79], 90.00th=[ 101], 95.00th=[ 266], | 99.00th=[ 266], 99.50th=[ 266], 99.90th=[ 266], 99.95th=[ 266], | 99.99th=[ 266] bw ( KiB/s): min=10240, max=10240, per=32.79%, avg=10240.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 lat (msec) : 50=41.67%, 100=50.00%, 500=8.33% cpu : usr=0.22%, sys=35.55%, ctx=343, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25550: Wed May 28 18:29:32 2025 read: IOPS=8, BW=8629KiB/s (8836kB/s)(6144KiB/712msec) clat (usec): min=351, max=75767, avg=22962.07, stdev=35086.73 lat (usec): min=352, max=75768, avg=22963.29, stdev=35087.02 clat percentiles (usec): | 1.00th=[ 351], 5.00th=[ 351], 10.00th=[ 351], 20.00th=[ 545], | 30.00th=[ 545], 40.00th=[ 635], 50.00th=[ 635], 60.00th=[ 660], | 70.00th=[60031], 80.00th=[60031], 90.00th=[76022], 95.00th=[76022], | 99.00th=[76022], 99.50th=[76022], 99.90th=[76022], 99.95th=[76022], | 99.99th=[76022] bw ( KiB/s): min= 4096, max= 4096, per=18.36%, avg=4096.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 write: IOPS=8, BW=8629KiB/s (8836kB/s)(6144KiB/712msec) clat (msec): min=25, max=175, avg=89.69, stdev=65.71 lat (msec): min=25, max=175, avg=89.74, stdev=65.70 clat percentiles (msec): | 1.00th=[ 26], 5.00th=[ 26], 10.00th=[ 26], 20.00th=[ 35], | 30.00th=[ 35], 40.00th=[ 43], 50.00th=[ 43], 60.00th=[ 103], | 70.00th=[ 159], 80.00th=[ 159], 90.00th=[ 176], 95.00th=[ 176], | 99.00th=[ 176], 99.50th=[ 176], 99.90th=[ 176], 99.95th=[ 176], | 99.99th=[ 176] bw ( KiB/s): min= 6144, max= 6144, per=19.67%, avg=6144.00, stdev= 0.00, samples=1 iops : min= 6, max= 6, avg= 6.00, stdev= 0.00, samples=1 lat (usec) : 500=8.33%, 750=25.00% lat (msec) : 50=25.00%, 100=16.67%, 250=25.00% cpu : usr=0.00%, sys=24.33%, ctx=313, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25552: Wed May 28 18:29:32 2025 read: IOPS=5, BW=6066KiB/s (6212kB/s)(5120KiB/844msec) clat (usec): min=497, max=35459, avg=13749.97, stdev=18058.60 lat (usec): min=499, max=35460, avg=13751.17, stdev=18058.28 clat percentiles (usec): | 1.00th=[ 498], 5.00th=[ 498], 10.00th=[ 498], 20.00th=[ 498], | 30.00th=[ 519], 40.00th=[ 519], 50.00th=[ 791], 60.00th=[ 791], | 70.00th=[31589], 80.00th=[31589], 90.00th=[35390], 95.00th=[35390], | 99.00th=[35390], 99.50th=[35390], 99.90th=[35390], 99.95th=[35390], | 99.99th=[35390] write: IOPS=8, BW=8493KiB/s (8697kB/s)(7168KiB/844msec) clat (msec): min=63, max=171, avg=103.49, stdev=38.27 lat (msec): min=63, max=171, avg=103.54, stdev=38.27 clat percentiles (msec): | 1.00th=[ 64], 5.00th=[ 64], 10.00th=[ 64], 20.00th=[ 65], | 30.00th=[ 93], 40.00th=[ 93], 50.00th=[ 99], 60.00th=[ 102], | 70.00th=[ 102], 80.00th=[ 133], 90.00th=[ 171], 95.00th=[ 171], | 99.00th=[ 171], 99.50th=[ 171], 99.90th=[ 171], 99.95th=[ 171], | 99.99th=[ 171] bw ( KiB/s): min= 6131, max= 6131, per=19.63%, avg=6131.00, stdev= 0.00, samples=1 iops : min= 5, max= 5, avg= 5.00, stdev= 0.00, samples=1 lat (usec) : 500=8.33%, 750=8.33%, 1000=8.33% lat (msec) : 50=16.67%, 100=33.33%, 250=25.00% cpu : usr=0.12%, sys=30.13%, ctx=314, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=21.8MiB/s (22.8MB/s), 2231KiB/s-16.3MiB/s (2284kB/s-17.1MB/s), io=20.0MiB (20.0MB), run=430-918msec WRITE: bw=30.5MiB/s (31.0MB/s), 8493KiB/s-11.6MiB/s (8697kB/s-12.2MB/s), io=28.0MiB (29.4MB), run=430-918msec rand-rw: (groupid=0, jobs=1): err= 0: pid=25545: Wed May 28 18:29:32 2025 read: IOPS=8, BW=8688KiB/s (8897kB/s)(7168KiB/825msec) clat (msec): min=19, max=136, avg=67.57, stdev=46.85 lat (msec): min=19, max=136, avg=67.57, stdev=46.85 clat percentiles (msec): | 1.00th=[ 20], 5.00th=[ 20], 10.00th=[ 20], 20.00th=[ 27], | 30.00th=[ 43], 40.00th=[ 43], 50.00th=[ 46], 60.00th=[ 79], | 70.00th=[ 79], 80.00th=[ 125], 90.00th=[ 138], 95.00th=[ 138], | 99.00th=[ 138], 99.50th=[ 138], 99.90th=[ 138], 99.95th=[ 138], | 99.99th=[ 138] bw ( KiB/s): min= 8192, max= 8192, per=38.92%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=6, BW=6206KiB/s (6355kB/s)(5120KiB/825msec) clat (msec): min=31, max=115, avg=65.91, stdev=35.65 lat (msec): min=31, max=115, avg=65.99, stdev=35.64 clat percentiles (msec): | 1.00th=[ 32], 5.00th=[ 32], 10.00th=[ 32], 20.00th=[ 32], | 30.00th=[ 33], 40.00th=[ 33], 50.00th=[ 69], 60.00th=[ 69], | 70.00th=[ 83], 80.00th=[ 83], 90.00th=[ 116], 95.00th=[ 116], | 99.00th=[ 116], 99.50th=[ 116], 99.90th=[ 116], 99.95th=[ 116], | 99.99th=[ 116] bw ( KiB/s): min= 2048, max= 2048, per=6.95%, avg=2048.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 20=8.33%, 50=41.67%, 100=25.00%, 250=25.00% cpu : usr=0.00%, sys=15.66%, ctx=67, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25547: Wed May 28 18:29:32 2025 read: IOPS=2, BW=2365KiB/s (2422kB/s)(2048KiB/866msec) clat (usec): min=17874, max=58353, avg=38114.09, stdev=28623.00 lat (usec): min=17876, max=58355, avg=38115.91, stdev=28622.56 clat percentiles (usec): | 1.00th=[17957], 5.00th=[17957], 10.00th=[17957], 20.00th=[17957], | 30.00th=[17957], 40.00th=[17957], 50.00th=[17957], 60.00th=[58459], | 70.00th=[58459], 80.00th=[58459], 90.00th=[58459], 95.00th=[58459], | 99.00th=[58459], 99.50th=[58459], 99.90th=[58459], 99.95th=[58459], | 99.99th=[58459] bw ( KiB/s): min= 4071, max= 4071, per=19.34%, avg=4071.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 write: IOPS=11, BW=11.5MiB/s (12.1MB/s)(10.0MiB/866msec) clat (msec): min=31, max=242, avg=78.06, stdev=63.35 lat (msec): min=31, max=243, avg=78.14, stdev=63.34 clat percentiles (msec): | 1.00th=[ 32], 5.00th=[ 32], 10.00th=[ 32], 20.00th=[ 34], | 30.00th=[ 39], 40.00th=[ 41], 50.00th=[ 55], 60.00th=[ 70], | 70.00th=[ 86], 80.00th=[ 86], 90.00th=[ 111], 95.00th=[ 243], | 99.00th=[ 243], 99.50th=[ 243], 99.90th=[ 243], 99.95th=[ 243], | 99.99th=[ 243] bw ( KiB/s): min=10178, max=10178, per=34.54%, avg=10178.00, stdev= 0.00, samples=1 iops : min= 9, max= 9, avg= 9.00, stdev= 0.00, samples=1 lat (msec) : 20=8.33%, 50=33.33%, 100=41.67%, 250=16.67% cpu : usr=0.12%, sys=13.06%, ctx=67, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25549: Wed May 28 18:29:32 2025 read: IOPS=6, BW=6314KiB/s (6466kB/s)(6144KiB/973msec) clat (msec): min=19, max=117, avg=60.32, stdev=37.68 lat (msec): min=19, max=117, avg=60.32, stdev=37.68 clat percentiles (msec): | 1.00th=[ 20], 5.00th=[ 20], 10.00th=[ 20], 20.00th=[ 31], | 30.00th=[ 31], 40.00th=[ 41], 50.00th=[ 41], 60.00th=[ 65], | 70.00th=[ 90], 80.00th=[ 90], 90.00th=[ 118], 95.00th=[ 118], | 99.00th=[ 118], 99.50th=[ 118], 99.90th=[ 118], 99.95th=[ 118], | 99.99th=[ 118] bw ( KiB/s): min= 4087, max= 4087, per=19.42%, avg=4087.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 write: IOPS=6, BW=6314KiB/s (6466kB/s)(6144KiB/973msec) clat (msec): min=15, max=258, avg=96.98, stdev=103.47 lat (msec): min=15, max=258, avg=97.10, stdev=103.46 clat percentiles (msec): | 1.00th=[ 16], 5.00th=[ 16], 10.00th=[ 16], 20.00th=[ 22], | 30.00th=[ 22], 40.00th=[ 22], 50.00th=[ 22], 60.00th=[ 74], | 70.00th=[ 190], 80.00th=[ 190], 90.00th=[ 259], 95.00th=[ 259], | 99.00th=[ 259], 99.50th=[ 259], 99.90th=[ 259], 99.95th=[ 259], | 99.99th=[ 259] bw ( KiB/s): min= 6131, max= 6131, per=20.81%, avg=6131.00, stdev= 0.00, samples=1 iops : min= 5, max= 5, avg= 5.00, stdev= 0.00, samples=1 lat (msec) : 20=16.67%, 50=33.33%, 100=25.00%, 250=16.67%, 500=8.33% cpu : usr=0.10%, sys=12.35%, ctx=63, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25551: Wed May 28 18:29:32 2025 read: IOPS=6, BW=7062KiB/s (7232kB/s)(5120KiB/725msec) clat (msec): min=26, max=127, avg=58.45, stdev=41.55 lat (msec): min=26, max=127, avg=58.45, stdev=41.55 clat percentiles (msec): | 1.00th=[ 27], 5.00th=[ 27], 10.00th=[ 27], 20.00th=[ 27], | 30.00th=[ 30], 40.00th=[ 30], 50.00th=[ 44], 60.00th=[ 44], | 70.00th=[ 65], 80.00th=[ 65], 90.00th=[ 128], 95.00th=[ 128], | 99.00th=[ 128], 99.50th=[ 128], 99.90th=[ 128], 99.95th=[ 128], | 99.99th=[ 128] bw ( KiB/s): min= 8192, max= 8192, per=38.92%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=9, BW=9887KiB/s (10.1MB/s)(7168KiB/725msec) clat (msec): min=14, max=158, avg=60.31, stdev=52.38 lat (msec): min=14, max=158, avg=60.39, stdev=52.38 clat percentiles (msec): | 1.00th=[ 15], 5.00th=[ 15], 10.00th=[ 15], 20.00th=[ 18], | 30.00th=[ 21], 40.00th=[ 21], 50.00th=[ 46], 60.00th=[ 81], | 70.00th=[ 81], 80.00th=[ 85], 90.00th=[ 159], 95.00th=[ 159], | 99.00th=[ 159], 99.50th=[ 159], 99.90th=[ 159], 99.95th=[ 159], | 99.99th=[ 159] bw ( KiB/s): min=10240, max=10240, per=34.75%, avg=10240.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 lat (msec) : 20=16.67%, 50=41.67%, 100=25.00%, 250=16.67% cpu : usr=0.00%, sys=14.64%, ctx=45, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=20.6MiB/s (21.6MB/s), 2365KiB/s-8688KiB/s (2422kB/s-8897kB/s), io=20.0MiB (20.0MB), run=725-973msec WRITE: bw=28.8MiB/s (30.2MB/s), 6206KiB/s-11.5MiB/s (6355kB/s-12.1MB/s), io=28.0MiB (29.4MB), run=725-973msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=25558: Wed May 28 18:29:34 2025 read: IOPS=4, BW=18.3MiB/s (19.2MB/s)(12.0MiB/655msec) clat (msec): min=170, max=198, avg=182.43, stdev=14.49 lat (msec): min=170, max=198, avg=182.43, stdev=14.49 clat percentiles (msec): | 1.00th=[ 171], 5.00th=[ 171], 10.00th=[ 171], 20.00th=[ 171], | 30.00th=[ 171], 40.00th=[ 178], 50.00th=[ 178], 60.00th=[ 178], | 70.00th=[ 199], 80.00th=[ 199], 90.00th=[ 199], 95.00th=[ 199], | 99.00th=[ 199], 99.50th=[ 199], 99.90th=[ 199], 99.95th=[ 199], | 99.99th=[ 199] bw ( KiB/s): min=16351, max=16351, per=100.00%, avg=16351.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 lat (msec) : 250=100.00% cpu : usr=0.00%, sys=26.91%, ctx=1131, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25560: Wed May 28 18:29:34 2025 read: IOPS=0, BW=3992KiB/s (4088kB/s)(4096KiB/1026msec) clat (nsec): min=203923k, max=203923k, avg=203922633.00, stdev= 0.00 lat (nsec): min=203924k, max=203924k, avg=203923672.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 205], 5.00th=[ 205], 10.00th=[ 205], 20.00th=[ 205], | 30.00th=[ 205], 40.00th=[ 205], 50.00th=[ 205], 60.00th=[ 205], | 70.00th=[ 205], 80.00th=[ 205], 90.00th=[ 205], 95.00th=[ 205], | 99.00th=[ 205], 99.50th=[ 205], 99.90th=[ 205], 99.95th=[ 205], | 99.99th=[ 205] bw ( KiB/s): min= 8192, max= 8192, per=51.30%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=1, BW=7984KiB/s (8176kB/s)(8192KiB/1026msec) clat (msec): min=153, max=566, avg=359.62, stdev=291.91 lat (msec): min=153, max=566, avg=359.98, stdev=291.80 clat percentiles (msec): | 1.00th=[ 155], 5.00th=[ 155], 10.00th=[ 155], 20.00th=[ 155], | 30.00th=[ 155], 40.00th=[ 155], 50.00th=[ 155], 60.00th=[ 567], | 70.00th=[ 567], 80.00th=[ 567], 90.00th=[ 567], 95.00th=[ 567], | 99.00th=[ 567], 99.50th=[ 567], 99.90th=[ 567], 99.95th=[ 567], | 99.99th=[ 567] lat (msec) : 250=66.67%, 750=33.33% cpu : usr=0.00%, sys=29.27%, ctx=291, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25562: Wed May 28 18:29:34 2025 write: IOPS=2, BW=10.5MiB/s (10.0MB/s)(12.0MiB/1144msec) clat (msec): min=347, max=409, avg=373.25, stdev=32.42 lat (msec): min=347, max=410, avg=373.63, stdev=32.49 clat percentiles (msec): | 1.00th=[ 347], 5.00th=[ 347], 10.00th=[ 347], 20.00th=[ 347], | 30.00th=[ 347], 40.00th=[ 363], 50.00th=[ 363], 60.00th=[ 363], | 70.00th=[ 409], 80.00th=[ 409], 90.00th=[ 409], 95.00th=[ 409], | 99.00th=[ 409], 99.50th=[ 409], 99.90th=[ 409], 99.95th=[ 409], | 99.99th=[ 409] bw ( KiB/s): min= 8159, max= 8159, per=28.49%, avg=8159.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 500=100.00% cpu : usr=0.00%, sys=27.12%, ctx=125, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25565: Wed May 28 18:29:34 2025 write: IOPS=4, BW=16.6MiB/s (17.4MB/s)(12.0MiB/723msec) clat (msec): min=168, max=261, avg=205.90, stdev=48.71 lat (msec): min=169, max=261, avg=206.23, stdev=48.68 clat percentiles (msec): | 1.00th=[ 169], 5.00th=[ 169], 10.00th=[ 169], 20.00th=[ 169], | 30.00th=[ 169], 40.00th=[ 188], 50.00th=[ 188], 60.00th=[ 188], | 70.00th=[ 262], 80.00th=[ 262], 90.00th=[ 262], 95.00th=[ 262], | 99.00th=[ 262], 99.50th=[ 262], 99.90th=[ 262], 99.95th=[ 262], | 99.99th=[ 262] bw ( KiB/s): min= 8192, max= 8192, per=28.60%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.00%, sys=51.11%, ctx=68, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=15.6MiB/s (16.4MB/s), 3992KiB/s-18.3MiB/s (4088kB/s-19.2MB/s), io=16.0MiB (16.8MB), run=655-1026msec WRITE: bw=27.0MiB/s (29.3MB/s), 7984KiB/s-16.6MiB/s (8176kB/s-17.4MB/s), io=32.0MiB (33.6MB), run=723-1144msec rand-rw: (groupid=0, jobs=1): err= 0: pid=25559: Wed May 28 18:29:34 2025 read: IOPS=3, BW=15.7MiB/s (16.4MB/s)(12.0MiB/765msec) clat (msec): min=89, max=471, avg=249.35, stdev=198.57 lat (msec): min=89, max=471, avg=249.35, stdev=198.57 clat percentiles (msec): | 1.00th=[ 90], 5.00th=[ 90], 10.00th=[ 90], 20.00th=[ 90], | 30.00th=[ 90], 40.00th=[ 188], 50.00th=[ 188], 60.00th=[ 188], | 70.00th=[ 472], 80.00th=[ 472], 90.00th=[ 472], 95.00th=[ 472], | 99.00th=[ 472], 99.50th=[ 472], 99.90th=[ 472], 99.95th=[ 472], | 99.99th=[ 472] bw ( KiB/s): min=16286, max=16286, per=100.00%, avg=16286.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 250=33.33%, 500=33.33% cpu : usr=0.00%, sys=10.21%, ctx=46, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25561: Wed May 28 18:29:34 2025 read: IOPS=0, BW=3073KiB/s (3147kB/s)(4096KiB/1333msec) clat (nsec): min=378681k, max=378681k, avg=378680657.00, stdev= 0.00 lat (nsec): min=378683k, max=378683k, avg=378683178.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 380], 5.00th=[ 380], 10.00th=[ 380], 20.00th=[ 380], | 30.00th=[ 380], 40.00th=[ 380], 50.00th=[ 380], 60.00th=[ 380], | 70.00th=[ 380], 80.00th=[ 380], 90.00th=[ 380], 95.00th=[ 380], | 99.00th=[ 380], 99.50th=[ 380], 99.90th=[ 380], 99.95th=[ 380], | 99.99th=[ 380] bw ( KiB/s): min= 8192, max= 8192, per=66.65%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=1, BW=6146KiB/s (6293kB/s)(8192KiB/1333msec) clat (msec): min=130, max=726, avg=428.67, stdev=421.54 lat (msec): min=130, max=727, avg=429.00, stdev=421.62 clat percentiles (msec): | 1.00th=[ 131], 5.00th=[ 131], 10.00th=[ 131], 20.00th=[ 131], | 30.00th=[ 131], 40.00th=[ 131], 50.00th=[ 131], 60.00th=[ 726], | 70.00th=[ 726], 80.00th=[ 726], 90.00th=[ 726], 95.00th=[ 726], | 99.00th=[ 726], 99.50th=[ 726], 99.90th=[ 726], 99.95th=[ 726], | 99.99th=[ 726] lat (msec) : 250=33.33%, 500=33.33%, 750=33.33% cpu : usr=0.08%, sys=14.41%, ctx=76, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25563: Wed May 28 18:29:34 2025 write: IOPS=4, BW=16.1MiB/s (16.9MB/s)(12.0MiB/744msec) clat (msec): min=52, max=567, avg=235.81, stdev=287.90 lat (msec): min=52, max=567, avg=236.20, stdev=287.82 clat percentiles (msec): | 1.00th=[ 53], 5.00th=[ 53], 10.00th=[ 53], 20.00th=[ 53], | 30.00th=[ 53], 40.00th=[ 88], 50.00th=[ 88], 60.00th=[ 88], | 70.00th=[ 567], 80.00th=[ 567], 90.00th=[ 567], 95.00th=[ 567], | 99.00th=[ 567], 99.50th=[ 567], 99.90th=[ 567], 99.95th=[ 567], | 99.99th=[ 567] bw ( KiB/s): min=16351, max=16351, per=66.52%, avg=16351.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 lat (msec) : 100=66.67%, 750=33.33% cpu : usr=0.00%, sys=20.05%, ctx=87, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25564: Wed May 28 18:29:34 2025 write: IOPS=3, BW=12.5MiB/s (13.1MB/s)(12.0MiB/963msec) clat (msec): min=99, max=392, avg=294.21, stdev=168.93 lat (msec): min=99, max=393, avg=294.55, stdev=168.91 clat percentiles (msec): | 1.00th=[ 100], 5.00th=[ 100], 10.00th=[ 100], 20.00th=[ 100], | 30.00th=[ 100], 40.00th=[ 393], 50.00th=[ 393], 60.00th=[ 393], | 70.00th=[ 393], 80.00th=[ 393], 90.00th=[ 393], 95.00th=[ 393], | 99.00th=[ 393], 99.50th=[ 393], 99.90th=[ 393], 99.95th=[ 393], | 99.99th=[ 393] bw ( KiB/s): min= 8192, max= 8192, per=33.33%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 500=66.67% cpu : usr=0.10%, sys=21.21%, ctx=94, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=12.0MiB/s (12.6MB/s), 3073KiB/s-15.7MiB/s (3147kB/s-16.4MB/s), io=16.0MiB (16.8MB), run=765-1333msec WRITE: bw=24.0MiB/s (25.2MB/s), 6146KiB/s-16.1MiB/s (6293kB/s-16.9MB/s), io=32.0MiB (33.6MB), run=744-1333msec