== sanity test 398b: DIO and buffer IO race ============== 23:17:50 (1745464670) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.16382 s, 43.2 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=8120: Wed Apr 23 23:18:04 2025 read: IOPS=128, BW=514KiB/s (526kB/s)(5992KiB/11665msec) clat (usec): min=52, max=176924, avg=3448.93, stdev=5332.21 lat (usec): min=52, max=176925, avg=3450.53, stdev=5332.60 clat percentiles (usec): | 1.00th=[ 60], 5.00th=[ 69], 10.00th=[ 73], 20.00th=[ 118], | 30.00th=[ 2057], 40.00th=[ 2376], 50.00th=[ 2737], 60.00th=[ 3195], | 70.00th=[ 4015], 80.00th=[ 5538], 90.00th=[ 7111], 95.00th=[ 8455], | 99.00th=[ 12518], 99.50th=[ 17695], 99.90th=[ 25822], 99.95th=[177210], | 99.99th=[177210] bw ( KiB/s): min= 312, max= 784, per=24.72%, avg=508.17, stdev=121.42, samples=23 iops : min= 78, max= 196, avg=127.04, stdev=30.35, samples=23 write: IOPS=134, BW=540KiB/s (553kB/s)(6296KiB/11665msec) clat (usec): min=442, max=79664, avg=4050.91, stdev=4350.08 lat (usec): min=442, max=79665, avg=4051.56, stdev=4350.08 clat percentiles (usec): | 1.00th=[ 498], 5.00th=[ 529], 10.00th=[ 553], 20.00th=[ 611], | 30.00th=[ 848], 40.00th=[ 1991], 50.00th=[ 3261], 60.00th=[ 4047], | 70.00th=[ 5211], 80.00th=[ 6652], 90.00th=[ 8979], 95.00th=[11207], | 99.00th=[18220], 99.50th=[19268], 99.90th=[44827], 99.95th=[79168], | 99.99th=[79168] bw ( KiB/s): min= 368, max= 776, per=24.81%, avg=533.91, stdev=106.88, samples=23 iops : min= 92, max= 194, avg=133.48, stdev=26.72, samples=23 lat (usec) : 100=9.41%, 250=0.91%, 500=0.81%, 750=13.44%, 1000=2.34% lat (msec) : 2=7.52%, 4=29.98%, 10=30.99%, 20=4.26%, 50=0.26% lat (msec) : 100=0.03%, 250=0.03% cpu : usr=0.13%, sys=23.19%, ctx=5612, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8121: Wed Apr 23 23:18:04 2025 read: IOPS=127, BW=509KiB/s (522kB/s)(5940KiB/11661msec) clat (usec): min=53, max=38141, avg=3330.48, stdev=3167.67 lat (usec): min=53, max=38141, avg=3331.46, stdev=3167.63 clat percentiles (usec): | 1.00th=[ 60], 5.00th=[ 70], 10.00th=[ 73], 20.00th=[ 99], | 30.00th=[ 1991], 40.00th=[ 2311], 50.00th=[ 2638], 60.00th=[ 3163], | 70.00th=[ 3916], 80.00th=[ 5342], 90.00th=[ 7046], 95.00th=[ 8455], | 99.00th=[12911], 99.50th=[21103], 99.90th=[29754], 99.95th=[38011], | 99.99th=[38011] bw ( KiB/s): min= 264, max= 768, per=24.66%, avg=506.96, stdev=113.99, samples=23 iops : min= 66, max= 192, avg=126.65, stdev=28.54, samples=23 write: IOPS=136, BW=544KiB/s (557kB/s)(6348KiB/11661msec) clat (usec): min=436, max=179590, avg=4154.88, stdev=5968.85 lat (usec): min=437, max=179590, avg=4155.68, stdev=5968.78 clat percentiles (usec): | 1.00th=[ 506], 5.00th=[ 529], 10.00th=[ 553], 20.00th=[ 603], | 30.00th=[ 816], 40.00th=[ 1762], 50.00th=[ 3097], 60.00th=[ 4047], | 70.00th=[ 5211], 80.00th=[ 6849], 90.00th=[ 9241], 95.00th=[ 11338], | 99.00th=[ 18482], 99.50th=[ 20579], 99.90th=[ 34341], 99.95th=[179307], | 99.99th=[179307] bw ( KiB/s): min= 344, max= 790, per=24.96%, avg=537.17, stdev=110.14, samples=23 iops : min= 86, max= 197, avg=134.22, stdev=27.51, samples=23 lat (usec) : 100=9.77%, 250=0.78%, 500=0.36%, 750=14.32%, 1000=2.60% lat (msec) : 2=8.24%, 4=28.87%, 10=29.79%, 20=4.75%, 50=0.49% lat (msec) : 250=0.03% cpu : usr=0.17%, sys=22.85%, ctx=5591, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8122: Wed Apr 23 23:18:04 2025 read: IOPS=129, BW=516KiB/s (529kB/s)(6028KiB/11678msec) clat (usec): min=57, max=179698, avg=3429.42, stdev=5396.97 lat (usec): min=57, max=179699, avg=3430.17, stdev=5396.95 clat percentiles (usec): | 1.00th=[ 61], 5.00th=[ 71], 10.00th=[ 78], 20.00th=[ 141], | 30.00th=[ 2057], 40.00th=[ 2409], 50.00th=[ 2769], 60.00th=[ 3195], | 70.00th=[ 4047], 80.00th=[ 5276], 90.00th=[ 6783], 95.00th=[ 8160], | 99.00th=[ 11600], 99.50th=[ 14484], 99.90th=[ 47973], 99.95th=[179307], | 99.99th=[179307] bw ( KiB/s): min= 288, max= 840, per=24.98%, avg=513.57, stdev=125.21, samples=23 iops : min= 72, max= 210, avg=128.30, stdev=31.32, samples=23 write: IOPS=134, BW=536KiB/s (549kB/s)(6260KiB/11678msec) clat (usec): min=459, max=48285, avg=4082.31, stdev=4148.25 lat (usec): min=459, max=48286, avg=4083.25, stdev=4148.18 clat percentiles (usec): | 1.00th=[ 498], 5.00th=[ 537], 10.00th=[ 562], 20.00th=[ 627], | 30.00th=[ 889], 40.00th=[ 2114], 50.00th=[ 3261], 60.00th=[ 4080], | 70.00th=[ 5080], 80.00th=[ 6718], 90.00th=[ 9241], 95.00th=[11469], | 99.00th=[17171], 99.50th=[20317], 99.90th=[42730], 99.95th=[48497], | 99.99th=[48497] bw ( KiB/s): min= 352, max= 720, per=24.69%, avg=531.30, stdev=103.03, samples=23 iops : min= 88, max= 180, avg=132.74, stdev=25.78, samples=23 lat (usec) : 100=8.82%, 250=1.27%, 500=0.59%, 750=12.86%, 1000=3.22% lat (msec) : 2=7.13%, 4=30.31%, 10=30.89%, 20=4.52%, 50=0.36% lat (msec) : 250=0.03% cpu : usr=0.22%, sys=23.55%, ctx=5677, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8124: Wed Apr 23 23:18:04 2025 read: IOPS=130, BW=521KiB/s (533kB/s)(6052KiB/11626msec) clat (usec): min=53, max=48541, avg=3324.48, stdev=3066.21 lat (usec): min=53, max=48542, avg=3325.99, stdev=3065.70 clat percentiles (usec): | 1.00th=[ 61], 5.00th=[ 70], 10.00th=[ 74], 20.00th=[ 130], | 30.00th=[ 2089], 40.00th=[ 2343], 50.00th=[ 2671], 60.00th=[ 3130], | 70.00th=[ 4015], 80.00th=[ 5342], 90.00th=[ 6915], 95.00th=[ 8455], | 99.00th=[12125], 99.50th=[14877], 99.90th=[36439], 99.95th=[48497], | 99.99th=[48497] bw ( KiB/s): min= 280, max= 702, per=25.19%, avg=518.00, stdev=88.02, samples=23 iops : min= 70, max= 175, avg=129.43, stdev=21.95, samples=23 write: IOPS=134, BW=536KiB/s (549kB/s)(6236KiB/11626msec) clat (usec): min=469, max=179966, avg=4152.70, stdev=6118.79 lat (usec): min=469, max=179967, avg=4153.74, stdev=6118.74 clat percentiles (usec): | 1.00th=[ 506], 5.00th=[ 529], 10.00th=[ 545], 20.00th=[ 594], | 30.00th=[ 791], 40.00th=[ 1647], 50.00th=[ 3097], 60.00th=[ 3949], | 70.00th=[ 5080], 80.00th=[ 6652], 90.00th=[ 9634], 95.00th=[ 12256], | 99.00th=[ 17957], 99.50th=[ 20841], 99.90th=[ 46400], 99.95th=[179307], | 99.99th=[179307] bw ( KiB/s): min= 352, max= 784, per=24.87%, avg=535.13, stdev=100.60, samples=23 iops : min= 88, max= 196, avg=133.70, stdev=25.16, samples=23 lat (usec) : 100=9.05%, 250=1.14%, 500=0.36%, 750=14.19%, 1000=2.38% lat (msec) : 2=7.91%, 4=30.05%, 10=29.46%, 20=5.05%, 50=0.39% lat (msec) : 250=0.03% cpu : usr=0.16%, sys=23.29%, ctx=5472, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2056KiB/s (2106kB/s), 509KiB/s-521KiB/s (522kB/s-533kB/s), io=23.4MiB (24.6MB), run=11626-11678msec WRITE: bw=2153KiB/s (2204kB/s), 536KiB/s-544KiB/s (549kB/s-557kB/s), io=24.6MiB (25.7MB), run=11626-11678msec rand-rw: (groupid=0, jobs=1): err= 0: pid=8125: Wed Apr 23 23:18:25 2025 read: IOPS=46, BW=187KiB/s (192kB/s)(5992KiB/32024msec) clat (usec): min=1506, max=178952, avg=3639.94, stdev=5374.41 lat (usec): min=1507, max=178954, avg=3640.62, stdev=5374.42 clat percentiles (usec): | 1.00th=[ 1663], 5.00th=[ 1795], 10.00th=[ 1860], 20.00th=[ 1942], | 30.00th=[ 2008], 40.00th=[ 2114], 50.00th=[ 2343], 60.00th=[ 2933], | 70.00th=[ 3490], 80.00th=[ 4146], 90.00th=[ 6980], 95.00th=[ 9503], | 99.00th=[ 13566], 99.50th=[ 18220], 99.90th=[ 43779], 99.95th=[179307], | 99.99th=[179307] bw ( KiB/s): min= 64, max= 352, per=25.00%, avg=186.78, stdev=73.03, samples=63 iops : min= 16, max= 88, avg=46.59, stdev=18.27, samples=63 write: IOPS=49, BW=197KiB/s (201kB/s)(6296KiB/32024msec) clat (usec): min=8277, max=81576, avg=16815.46, stdev=6572.69 lat (usec): min=8278, max=81576, avg=16816.19, stdev=6572.68 clat percentiles (usec): | 1.00th=[ 8979], 5.00th=[10028], 10.00th=[10683], 20.00th=[11600], | 30.00th=[12649], 40.00th=[13829], 50.00th=[15401], 60.00th=[16712], | 70.00th=[18220], 80.00th=[20317], 90.00th=[25560], 95.00th=[29754], | 99.00th=[38011], 99.50th=[42206], 99.90th=[57934], 99.95th=[81265], | 99.99th=[81265] bw ( KiB/s): min= 64, max= 280, per=25.02%, avg=195.67, stdev=59.44, samples=63 iops : min= 16, max= 70, avg=48.81, stdev=14.87, samples=63 lat (msec) : 2=13.83%, 4=24.41%, 10=11.17%, 20=39.49%, 50=10.97% lat (msec) : 100=0.10%, 250=0.03% cpu : usr=0.11%, sys=5.43%, ctx=3639, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8126: Wed Apr 23 23:18:25 2025 read: IOPS=46, BW=185KiB/s (189kB/s)(5940KiB/32127msec) clat (usec): min=1470, max=184077, avg=3643.38, stdev=5533.51 lat (usec): min=1471, max=184078, avg=3644.04, stdev=5533.55 clat percentiles (usec): | 1.00th=[ 1647], 5.00th=[ 1778], 10.00th=[ 1844], 20.00th=[ 1926], | 30.00th=[ 2008], 40.00th=[ 2147], 50.00th=[ 2311], 60.00th=[ 2802], | 70.00th=[ 3490], 80.00th=[ 4047], 90.00th=[ 7046], 95.00th=[ 9634], | 99.00th=[ 16450], 99.50th=[ 19006], 99.90th=[ 36963], 99.95th=[183501], | 99.99th=[183501] bw ( KiB/s): min= 16, max= 352, per=24.74%, avg=184.83, stdev=78.78, samples=64 iops : min= 4, max= 88, avg=46.09, stdev=19.75, samples=64 write: IOPS=49, BW=198KiB/s (202kB/s)(6348KiB/32127msec) clat (usec): min=8225, max=58567, avg=16773.40, stdev=6615.05 lat (usec): min=8226, max=58568, avg=16774.13, stdev=6615.09 clat percentiles (usec): | 1.00th=[ 8979], 5.00th=[ 9765], 10.00th=[10421], 20.00th=[11600], | 30.00th=[12911], 40.00th=[13960], 50.00th=[15401], 60.00th=[16712], | 70.00th=[17957], 80.00th=[19792], 90.00th=[25822], 95.00th=[31851], | 99.00th=[39060], 99.50th=[42730], 99.90th=[51643], 99.95th=[58459], | 99.99th=[58459] bw ( KiB/s): min= 80, max= 272, per=25.21%, avg=197.11, stdev=60.74, samples=64 iops : min= 20, max= 68, avg=49.17, stdev=15.20, samples=64 lat (msec) : 2=14.00%, 4=24.54%, 10=10.77%, 20=40.27%, 50=10.29% lat (msec) : 100=0.10%, 250=0.03% cpu : usr=0.08%, sys=5.53%, ctx=3631, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8127: Wed Apr 23 23:18:25 2025 read: IOPS=47, BW=189KiB/s (194kB/s)(6028KiB/31831msec) clat (usec): min=1477, max=55641, avg=3661.69, stdev=3061.26 lat (usec): min=1477, max=55641, avg=3662.38, stdev=3061.24 clat percentiles (usec): | 1.00th=[ 1680], 5.00th=[ 1778], 10.00th=[ 1844], 20.00th=[ 1942], | 30.00th=[ 2024], 40.00th=[ 2147], 50.00th=[ 2442], 60.00th=[ 3163], | 70.00th=[ 3621], 80.00th=[ 4424], 90.00th=[ 7439], 95.00th=[ 9503], | 99.00th=[13960], 99.50th=[16909], 99.90th=[26870], 99.95th=[55837], | 99.99th=[55837] bw ( KiB/s): min= 56, max= 352, per=25.32%, avg=189.14, stdev=65.21, samples=63 iops : min= 14, max= 88, avg=47.24, stdev=16.33, samples=63 write: IOPS=49, BW=197KiB/s (201kB/s)(6260KiB/31831msec) clat (msec): min=8, max=157, avg=16.73, stdev= 7.32 lat (msec): min=8, max=157, avg=16.73, stdev= 7.32 clat percentiles (msec): | 1.00th=[ 9], 5.00th=[ 11], 10.00th=[ 11], 20.00th=[ 12], | 30.00th=[ 13], 40.00th=[ 14], 50.00th=[ 16], 60.00th=[ 17], | 70.00th=[ 18], 80.00th=[ 20], 90.00th=[ 26], 95.00th=[ 30], | 99.00th=[ 40], 99.50th=[ 46], 99.90th=[ 67], 99.95th=[ 159], | 99.99th=[ 159] bw ( KiB/s): min= 88, max= 280, per=25.08%, avg=196.13, stdev=61.00, samples=63 iops : min= 22, max= 70, avg=48.98, stdev=15.29, samples=63 lat (msec) : 2=13.83%, 4=23.57%, 10=12.17%, 20=40.66%, 50=9.60% lat (msec) : 100=0.13%, 250=0.03% cpu : usr=0.10%, sys=5.46%, ctx=3660, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8128: Wed Apr 23 23:18:25 2025 read: IOPS=47, BW=192KiB/s (196kB/s)(6052KiB/31585msec) clat (usec): min=1506, max=43780, avg=3501.60, stdev=2959.87 lat (usec): min=1507, max=43781, avg=3502.32, stdev=2959.96 clat percentiles (usec): | 1.00th=[ 1696], 5.00th=[ 1778], 10.00th=[ 1844], 20.00th=[ 1926], | 30.00th=[ 2008], 40.00th=[ 2114], 50.00th=[ 2311], 60.00th=[ 2835], | 70.00th=[ 3490], 80.00th=[ 4146], 90.00th=[ 6718], 95.00th=[ 8848], | 99.00th=[13698], 99.50th=[18482], 99.90th=[36963], 99.95th=[43779], | 99.99th=[43779] bw ( KiB/s): min= 56, max= 336, per=25.64%, avg=191.51, stdev=72.63, samples=63 iops : min= 14, max= 84, avg=47.79, stdev=18.16, samples=63 write: IOPS=49, BW=197KiB/s (202kB/s)(6236KiB/31585msec) clat (msec): min=8, max=162, avg=16.78, stdev= 7.75 lat (msec): min=8, max=162, avg=16.78, stdev= 7.75 clat percentiles (msec): | 1.00th=[ 9], 5.00th=[ 10], 10.00th=[ 11], 20.00th=[ 12], | 30.00th=[ 13], 40.00th=[ 14], 50.00th=[ 16], 60.00th=[ 17], | 70.00th=[ 18], 80.00th=[ 20], 90.00th=[ 27], 95.00th=[ 31], | 99.00th=[ 41], 99.50th=[ 49], 99.90th=[ 82], 99.95th=[ 163], | 99.99th=[ 163] bw ( KiB/s): min= 80, max= 280, per=25.22%, avg=197.22, stdev=60.32, samples=63 iops : min= 20, max= 70, avg=49.22, stdev=15.12, samples=63 lat (msec) : 2=14.49%, 4=24.15%, 10=12.08%, 20=39.49%, 50=9.67% lat (msec) : 100=0.10%, 250=0.03% cpu : usr=0.12%, sys=5.48%, ctx=3673, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=747KiB/s (765kB/s), 185KiB/s-192KiB/s (189kB/s-196kB/s), io=23.4MiB (24.6MB), run=31585-32127msec WRITE: bw=783KiB/s (801kB/s), 197KiB/s-198KiB/s (201kB/s-202kB/s), io=24.6MiB (25.7MB), run=31585-32127msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=8163: Wed Apr 23 23:18:28 2025 read: IOPS=150, BW=2407KiB/s (2465kB/s)(5920KiB/2459msec) clat (usec): min=60, max=24581, avg=2963.59, stdev=2675.97 lat (usec): min=60, max=24582, avg=2964.13, stdev=2675.99 clat percentiles (usec): | 1.00th=[ 63], 5.00th=[ 72], 10.00th=[ 75], 20.00th=[ 86], | 30.00th=[ 133], 40.00th=[ 2409], 50.00th=[ 2802], 60.00th=[ 3294], | 70.00th=[ 4113], 80.00th=[ 5014], 90.00th=[ 6063], 95.00th=[ 7242], | 99.00th=[10683], 99.50th=[10814], 99.90th=[24511], 99.95th=[24511], | 99.99th=[24511] bw ( KiB/s): min= 1856, max= 2976, per=24.72%, avg=2408.00, stdev=460.31, samples=4 iops : min= 116, max= 186, avg=150.50, stdev=28.77, samples=4 write: IOPS=161, BW=2590KiB/s (2652kB/s)(6368KiB/2459msec) clat (usec): min=627, max=26153, avg=3322.19, stdev=3325.39 lat (usec): min=627, max=26153, avg=3323.02, stdev=3325.39 clat percentiles (usec): | 1.00th=[ 668], 5.00th=[ 717], 10.00th=[ 766], 20.00th=[ 824], | 30.00th=[ 889], 40.00th=[ 1139], 50.00th=[ 1975], 60.00th=[ 3097], | 70.00th=[ 4293], 80.00th=[ 5866], 90.00th=[ 7373], 95.00th=[ 9634], | 99.00th=[14746], 99.50th=[18744], 99.90th=[26084], 99.95th=[26084], | 99.99th=[26084] bw ( KiB/s): min= 1984, max= 3264, per=25.68%, avg=2632.00, stdev=526.71, samples=4 iops : min= 124, max= 204, avg=164.50, stdev=32.92, samples=4 lat (usec) : 100=13.80%, 250=1.04%, 750=4.56%, 1000=14.32% lat (msec) : 2=7.55%, 4=27.08%, 10=28.78%, 20=2.60%, 50=0.26% cpu : usr=0.24%, sys=28.44%, ctx=1309, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8165: Wed Apr 23 23:18:28 2025 read: IOPS=147, BW=2368KiB/s (2425kB/s)(5744KiB/2426msec) clat (usec): min=61, max=23497, avg=3162.34, stdev=2944.50 lat (usec): min=62, max=23497, avg=3163.11, stdev=2944.35 clat percentiles (usec): | 1.00th=[ 68], 5.00th=[ 74], 10.00th=[ 77], 20.00th=[ 87], | 30.00th=[ 1631], 40.00th=[ 2540], 50.00th=[ 3032], 60.00th=[ 3523], | 70.00th=[ 4228], 80.00th=[ 5145], 90.00th=[ 6325], 95.00th=[ 7373], | 99.00th=[11994], 99.50th=[22152], 99.90th=[23462], 99.95th=[23462], | 99.99th=[23462] bw ( KiB/s): min= 1792, max= 2810, per=24.54%, avg=2390.50, stdev=437.91, samples=4 iops : min= 112, max= 175, avg=149.25, stdev=27.17, samples=4 write: IOPS=168, BW=2697KiB/s (2762kB/s)(6544KiB/2426msec) clat (usec): min=662, max=27097, avg=3118.12, stdev=3040.55 lat (usec): min=664, max=27098, avg=3118.96, stdev=3040.56 clat percentiles (usec): | 1.00th=[ 676], 5.00th=[ 725], 10.00th=[ 791], 20.00th=[ 840], | 30.00th=[ 914], 40.00th=[ 1090], 50.00th=[ 1598], 60.00th=[ 2966], | 70.00th=[ 3949], 80.00th=[ 5276], 90.00th=[ 7308], 95.00th=[ 8848], | 99.00th=[12518], 99.50th=[13304], 99.90th=[27132], 99.95th=[27132], | 99.99th=[27132] bw ( KiB/s): min= 2144, max= 3008, per=26.21%, avg=2686.50, stdev=385.51, samples=4 iops : min= 134, max= 188, avg=167.75, stdev=23.98, samples=4 lat (usec) : 100=10.94%, 250=2.08%, 500=0.13%, 750=4.04%, 1000=15.36% lat (msec) : 2=9.90%, 4=26.82%, 10=28.26%, 20=2.08%, 50=0.39% cpu : usr=0.21%, sys=28.54%, ctx=1432, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8167: Wed Apr 23 23:18:28 2025 read: IOPS=164, BW=2636KiB/s (2700kB/s)(6256KiB/2373msec) clat (usec): min=58, max=23673, avg=2828.14, stdev=2913.26 lat (usec): min=58, max=23674, avg=2828.67, stdev=2913.27 clat percentiles (usec): | 1.00th=[ 61], 5.00th=[ 71], 10.00th=[ 73], 20.00th=[ 80], | 30.00th=[ 94], 40.00th=[ 2245], 50.00th=[ 2704], 60.00th=[ 3228], | 70.00th=[ 3818], 80.00th=[ 4621], 90.00th=[ 5997], 95.00th=[ 7046], | 99.00th=[12256], 99.50th=[20055], 99.90th=[23725], 99.95th=[23725], | 99.99th=[23725] bw ( KiB/s): min= 1728, max= 3065, per=26.84%, avg=2614.25, stdev=612.35, samples=4 iops : min= 108, max= 191, avg=163.25, stdev=38.13, samples=4 write: IOPS=158, BW=2542KiB/s (2603kB/s)(6032KiB/2373msec) clat (usec): min=627, max=21170, avg=3056.87, stdev=2896.52 lat (usec): min=627, max=21170, avg=3057.74, stdev=2896.46 clat percentiles (usec): | 1.00th=[ 676], 5.00th=[ 734], 10.00th=[ 758], 20.00th=[ 824], | 30.00th=[ 906], 40.00th=[ 1123], 50.00th=[ 1614], 60.00th=[ 2900], | 70.00th=[ 4015], 80.00th=[ 5342], 90.00th=[ 7373], 95.00th=[ 8455], | 99.00th=[12387], 99.50th=[14877], 99.90th=[21103], 99.95th=[21103], | 99.99th=[21103] bw ( KiB/s): min= 1984, max= 3129, per=25.20%, avg=2582.25, stdev=482.92, samples=4 iops : min= 124, max= 195, avg=161.25, stdev=29.97, samples=4 lat (usec) : 100=16.28%, 250=1.56%, 500=0.39%, 750=4.04%, 1000=13.02% lat (msec) : 2=9.64%, 4=25.78%, 10=27.21%, 20=1.69%, 50=0.39% cpu : usr=0.25%, sys=27.82%, ctx=1365, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8169: Wed Apr 23 23:18:28 2025 read: IOPS=155, BW=2494KiB/s (2553kB/s)(6032KiB/2419msec) clat (usec): min=59, max=16171, avg=2894.34, stdev=2549.38 lat (usec): min=60, max=16172, avg=2899.76, stdev=2553.20 clat percentiles (usec): | 1.00th=[ 63], 5.00th=[ 72], 10.00th=[ 75], 20.00th=[ 87], | 30.00th=[ 131], 40.00th=[ 2376], 50.00th=[ 2802], 60.00th=[ 3261], | 70.00th=[ 4178], 80.00th=[ 4817], 90.00th=[ 5997], 95.00th=[ 7177], | 99.00th=[ 9634], 99.50th=[13698], 99.90th=[16188], 99.95th=[16188], | 99.99th=[16188] bw ( KiB/s): min= 1792, max= 2848, per=25.22%, avg=2456.00, stdev=459.57, samples=4 iops : min= 112, max= 178, avg=153.50, stdev=28.72, samples=4 write: IOPS=161, BW=2586KiB/s (2648kB/s)(6256KiB/2419msec) clat (usec): min=655, max=26440, avg=3266.84, stdev=3363.46 lat (usec): min=656, max=26440, avg=3267.67, stdev=3363.46 clat percentiles (usec): | 1.00th=[ 701], 5.00th=[ 734], 10.00th=[ 775], 20.00th=[ 832], | 30.00th=[ 898], 40.00th=[ 1172], 50.00th=[ 1893], 60.00th=[ 3097], | 70.00th=[ 3982], 80.00th=[ 5538], 90.00th=[ 7242], 95.00th=[ 9765], | 99.00th=[15795], 99.50th=[21103], 99.90th=[26346], 99.95th=[26346], | 99.99th=[26346] bw ( KiB/s): min= 1952, max= 3040, per=25.53%, avg=2616.00, stdev=474.19, samples=4 iops : min= 122, max= 190, avg=163.50, stdev=29.64, samples=4 lat (usec) : 100=14.06%, 250=1.17%, 500=0.65%, 750=3.78%, 1000=14.19% lat (msec) : 2=8.59%, 4=26.69%, 10=28.12%, 20=2.47%, 50=0.26% cpu : usr=0.17%, sys=28.04%, ctx=1344, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=9741KiB/s (9974kB/s), 2368KiB/s-2636KiB/s (2425kB/s-2700kB/s), io=23.4MiB (24.5MB), run=2373-2459msec WRITE: bw=10.0MiB/s (10.5MB/s), 2542KiB/s-2697KiB/s (2603kB/s-2762kB/s), io=24.6MiB (25.8MB), run=2373-2459msec rand-rw: (groupid=0, jobs=1): err= 0: pid=8164: Wed Apr 23 23:18:33 2025 read: IOPS=48, BW=774KiB/s (793kB/s)(5920KiB/7644msec) clat (usec): min=1735, max=28236, avg=3367.83, stdev=3038.23 lat (usec): min=1736, max=28237, avg=3368.46, stdev=3038.24 clat percentiles (usec): | 1.00th=[ 1762], 5.00th=[ 1844], 10.00th=[ 1909], 20.00th=[ 1975], | 30.00th=[ 2040], 40.00th=[ 2114], 50.00th=[ 2212], 60.00th=[ 2343], | 70.00th=[ 2606], 80.00th=[ 3523], 90.00th=[ 6325], 95.00th=[ 9503], | 99.00th=[16909], 99.50th=[20055], 99.90th=[28181], 99.95th=[28181], | 99.99th=[28181] bw ( KiB/s): min= 448, max= 1312, per=25.05%, avg=763.47, stdev=245.70, samples=15 iops : min= 28, max= 82, avg=47.60, stdev=15.33, samples=15 write: IOPS=52, BW=833KiB/s (853kB/s)(6368KiB/7644msec) clat (usec): min=8626, max=38273, avg=15935.52, stdev=5270.25 lat (usec): min=8627, max=38274, avg=15936.63, stdev=5270.28 clat percentiles (usec): | 1.00th=[ 8979], 5.00th=[ 9765], 10.00th=[10552], 20.00th=[11338], | 30.00th=[12649], 40.00th=[14222], 50.00th=[15008], 60.00th=[16057], | 70.00th=[17171], 80.00th=[18482], 90.00th=[23725], 95.00th=[26870], | 99.00th=[32900], 99.50th=[35390], 99.90th=[38536], 99.95th=[38536], | 99.99th=[38536] bw ( KiB/s): min= 448, max= 1088, per=25.86%, avg=829.47, stdev=234.57, samples=15 iops : min= 28, max= 68, avg=51.73, stdev=14.55, samples=15 lat (msec) : 2=11.20%, 4=28.12%, 10=10.03%, 20=42.71%, 50=7.94% cpu : usr=0.09%, sys=6.32%, ctx=867, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8166: Wed Apr 23 23:18:33 2025 read: IOPS=45, BW=731KiB/s (749kB/s)(5744KiB/7856msec) clat (usec): min=1692, max=30600, avg=3427.47, stdev=3279.66 lat (usec): min=1693, max=30600, avg=3428.08, stdev=3279.68 clat percentiles (usec): | 1.00th=[ 1762], 5.00th=[ 1844], 10.00th=[ 1893], 20.00th=[ 1942], | 30.00th=[ 1991], 40.00th=[ 2073], 50.00th=[ 2147], 60.00th=[ 2311], | 70.00th=[ 2573], 80.00th=[ 3851], 90.00th=[ 6652], 95.00th=[11338], | 99.00th=[18220], 99.50th=[19006], 99.90th=[30540], 99.95th=[30540], | 99.99th=[30540] bw ( KiB/s): min= 287, max= 1280, per=24.14%, avg=735.93, stdev=280.13, samples=15 iops : min= 17, max= 80, avg=45.93, stdev=17.62, samples=15 write: IOPS=52, BW=833KiB/s (853kB/s)(6544KiB/7856msec) clat (usec): min=8376, max=37338, avg=15920.08, stdev=5324.52 lat (usec): min=8377, max=37338, avg=15921.21, stdev=5324.53 clat percentiles (usec): | 1.00th=[ 9110], 5.00th=[ 9765], 10.00th=[10421], 20.00th=[11207], | 30.00th=[12256], 40.00th=[13829], 50.00th=[15008], 60.00th=[16057], | 70.00th=[17695], 80.00th=[18744], 90.00th=[23725], 95.00th=[27132], | 99.00th=[33817], 99.50th=[34866], 99.90th=[37487], 99.95th=[37487], | 99.99th=[37487] bw ( KiB/s): min= 447, max= 1120, per=25.61%, avg=821.27, stdev=233.91, samples=15 iops : min= 27, max= 70, avg=51.27, stdev=14.73, samples=15 lat (msec) : 2=14.06%, 4=23.70%, 10=9.77%, 20=44.27%, 50=8.20% cpu : usr=0.05%, sys=6.39%, ctx=877, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8168: Wed Apr 23 23:18:33 2025 read: IOPS=53, BW=853KiB/s (873kB/s)(6256KiB/7335msec) clat (usec): min=1626, max=20110, avg=3053.79, stdev=2346.94 lat (usec): min=1627, max=20110, avg=3054.41, stdev=2346.93 clat percentiles (usec): | 1.00th=[ 1729], 5.00th=[ 1827], 10.00th=[ 1876], 20.00th=[ 1942], | 30.00th=[ 1991], 40.00th=[ 2089], 50.00th=[ 2180], 60.00th=[ 2311], | 70.00th=[ 2573], 80.00th=[ 3097], 90.00th=[ 5538], 95.00th=[ 7701], | 99.00th=[14091], 99.50th=[17957], 99.90th=[20055], 99.95th=[20055], | 99.99th=[20055] bw ( KiB/s): min= 384, max= 1408, per=28.34%, avg=863.79, stdev=358.75, samples=14 iops : min= 24, max= 88, avg=53.93, stdev=22.36, samples=14 write: IOPS=51, BW=822KiB/s (842kB/s)(6032KiB/7335msec) clat (usec): min=8465, max=46239, avg=16242.28, stdev=5824.39 lat (usec): min=8466, max=46240, avg=16243.38, stdev=5824.43 clat percentiles (usec): | 1.00th=[ 8586], 5.00th=[ 9634], 10.00th=[10421], 20.00th=[11469], | 30.00th=[12780], 40.00th=[14091], 50.00th=[15270], 60.00th=[16319], | 70.00th=[17171], 80.00th=[19530], 90.00th=[23987], 95.00th=[27395], | 99.00th=[37487], 99.50th=[45351], 99.90th=[46400], 99.95th=[46400], | 99.99th=[46400] bw ( KiB/s): min= 512, max= 1088, per=25.30%, avg=811.21, stdev=218.05, samples=14 iops : min= 32, max= 68, avg=50.64, stdev=13.55, samples=14 lat (msec) : 2=15.62%, 4=27.47%, 10=9.51%, 20=38.15%, 50=9.24% cpu : usr=0.10%, sys=6.52%, ctx=882, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8170: Wed Apr 23 23:18:33 2025 read: IOPS=49, BW=786KiB/s (805kB/s)(6032KiB/7673msec) clat (usec): min=1675, max=26250, avg=3345.05, stdev=3396.67 lat (usec): min=1675, max=26251, avg=3345.65, stdev=3396.69 clat percentiles (usec): | 1.00th=[ 1745], 5.00th=[ 1811], 10.00th=[ 1876], 20.00th=[ 1958], | 30.00th=[ 2024], 40.00th=[ 2057], 50.00th=[ 2147], 60.00th=[ 2278], | 70.00th=[ 2573], 80.00th=[ 3261], 90.00th=[ 5735], 95.00th=[10028], | 99.00th=[20317], 99.50th=[25035], 99.90th=[26346], 99.95th=[26346], | 99.99th=[26346] bw ( KiB/s): min= 352, max= 1312, per=25.26%, avg=770.00, stdev=283.74, samples=15 iops : min= 22, max= 82, avg=48.07, stdev=17.72, samples=15 write: IOPS=50, BW=815KiB/s (835kB/s)(6256KiB/7673msec) clat (usec): min=8356, max=41139, avg=16155.00, stdev=5359.06 lat (usec): min=8357, max=41140, avg=16156.11, stdev=5359.13 clat percentiles (usec): | 1.00th=[ 8586], 5.00th=[10028], 10.00th=[10421], 20.00th=[11207], | 30.00th=[12649], 40.00th=[14222], 50.00th=[15533], 60.00th=[16712], | 70.00th=[17433], 80.00th=[19530], 90.00th=[23725], 95.00th=[26608], | 99.00th=[36439], 99.50th=[38011], 99.90th=[41157], 99.95th=[41157], | 99.99th=[41157] bw ( KiB/s): min= 416, max= 1056, per=25.21%, avg=808.40, stdev=232.37, samples=15 iops : min= 26, max= 66, avg=50.47, stdev=14.48, samples=15 lat (msec) : 2=13.41%, 4=27.73%, 10=8.07%, 20=41.02%, 50=9.77% cpu : usr=0.12%, sys=6.41%, ctx=892, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=3049KiB/s (3122kB/s), 731KiB/s-853KiB/s (749kB/s-873kB/s), io=23.4MiB (24.5MB), run=7335-7856msec WRITE: bw=3208KiB/s (3285kB/s), 815KiB/s-833KiB/s (835kB/s-853kB/s), io=24.6MiB (25.8MB), run=7335-7856msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=8184: Wed Apr 23 23:18:35 2025 read: IOPS=22, BW=22.4MiB/s (23.5MB/s)(7168KiB/312msec) clat (usec): min=428, max=38378, avg=22825.42, stdev=15972.93 lat (usec): min=428, max=38379, avg=22826.16, stdev=15973.01 clat percentiles (usec): | 1.00th=[ 429], 5.00th=[ 429], 10.00th=[ 429], 20.00th=[ 461], | 30.00th=[24249], 40.00th=[24249], 50.00th=[28705], 60.00th=[31327], | 70.00th=[31327], 80.00th=[35914], 90.00th=[38536], 95.00th=[38536], | 99.00th=[38536], 99.50th=[38536], 99.90th=[38536], 99.95th=[38536], | 99.99th=[38536] write: IOPS=16, BW=16.0MiB/s (16.8MB/s)(5120KiB/312msec) clat (usec): min=19218, max=42195, avg=27161.02, stdev=9317.29 lat (usec): min=19237, max=42215, avg=27187.12, stdev=9319.23 clat percentiles (usec): | 1.00th=[19268], 5.00th=[19268], 10.00th=[19268], 20.00th=[19268], | 30.00th=[21103], 40.00th=[21103], 50.00th=[23200], 60.00th=[23200], | 70.00th=[30016], 80.00th=[30016], 90.00th=[42206], 95.00th=[42206], | 99.00th=[42206], 99.50th=[42206], 99.90th=[42206], 99.95th=[42206], | 99.99th=[42206] lat (usec) : 500=16.67% lat (msec) : 20=8.33%, 50=75.00% cpu : usr=0.00%, sys=53.05%, ctx=658, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8185: Wed Apr 23 23:18:35 2025 read: IOPS=3, BW=3391KiB/s (3472kB/s)(2048KiB/604msec) clat (usec): min=33395, max=48626, avg=41010.76, stdev=10769.62 lat (usec): min=33396, max=48628, avg=41012.18, stdev=10770.68 clat percentiles (usec): | 1.00th=[33424], 5.00th=[33424], 10.00th=[33424], 20.00th=[33424], | 30.00th=[33424], 40.00th=[33424], 50.00th=[33424], 60.00th=[48497], | 70.00th=[48497], 80.00th=[48497], 90.00th=[48497], 95.00th=[48497], | 99.00th=[48497], 99.50th=[48497], 99.90th=[48497], 99.95th=[48497], | 99.99th=[48497] bw ( KiB/s): min= 3631, max= 3631, per=10.71%, avg=3631.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 write: IOPS=16, BW=16.6MiB/s (17.4MB/s)(10.0MiB/604msec) clat (msec): min=20, max=158, avg=46.50, stdev=42.29 lat (msec): min=20, max=158, avg=46.55, stdev=42.29 clat percentiles (msec): | 1.00th=[ 21], 5.00th=[ 21], 10.00th=[ 21], 20.00th=[ 22], | 30.00th=[ 23], 40.00th=[ 24], 50.00th=[ 27], 60.00th=[ 32], | 70.00th=[ 60], 80.00th=[ 60], 90.00th=[ 65], 95.00th=[ 159], | 99.00th=[ 159], 99.50th=[ 159], 99.90th=[ 159], 99.95th=[ 159], | 99.99th=[ 159] bw ( KiB/s): min=14524, max=14524, per=30.60%, avg=14524.00, stdev= 0.00, samples=1 iops : min= 14, max= 14, avg=14.00, stdev= 0.00, samples=1 lat (msec) : 50=75.00%, 100=16.67%, 250=8.33% cpu : usr=0.00%, sys=37.15%, ctx=234, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8187: Wed Apr 23 23:18:35 2025 read: IOPS=15, BW=15.7MiB/s (16.5MB/s)(6144KiB/381msec) clat (usec): min=441, max=42257, avg=25016.35, stdev=20178.32 lat (usec): min=442, max=42258, avg=25017.05, stdev=20178.30 clat percentiles (usec): | 1.00th=[ 441], 5.00th=[ 441], 10.00th=[ 441], 20.00th=[ 490], | 30.00th=[ 490], 40.00th=[24249], 50.00th=[24249], 60.00th=[40633], | 70.00th=[42206], 80.00th=[42206], 90.00th=[42206], 95.00th=[42206], | 99.00th=[42206], 99.50th=[42206], 99.90th=[42206], 99.95th=[42206], | 99.99th=[42206] write: IOPS=15, BW=15.7MiB/s (16.5MB/s)(6144KiB/381msec) clat (usec): min=15627, max=64489, avg=36747.89, stdev=20512.23 lat (usec): min=15644, max=64538, avg=36787.35, stdev=20524.45 clat percentiles (usec): | 1.00th=[15664], 5.00th=[15664], 10.00th=[15664], 20.00th=[16712], | 30.00th=[16712], 40.00th=[28705], 50.00th=[28705], 60.00th=[38011], | 70.00th=[57410], 80.00th=[57410], 90.00th=[64750], 95.00th=[64750], | 99.00th=[64750], 99.50th=[64750], 99.90th=[64750], 99.95th=[64750], | 99.99th=[64750] lat (usec) : 500=16.67% lat (msec) : 20=16.67%, 50=50.00%, 100=16.67% cpu : usr=0.00%, sys=37.11%, ctx=388, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8189: Wed Apr 23 23:18:35 2025 read: IOPS=8, BW=8843KiB/s (9055kB/s)(5120KiB/579msec) clat (usec): min=293, max=32131, avg=14140.46, stdev=14669.77 lat (usec): min=293, max=32132, avg=14141.06, stdev=14669.86 clat percentiles (usec): | 1.00th=[ 293], 5.00th=[ 293], 10.00th=[ 293], 20.00th=[ 293], | 30.00th=[ 338], 40.00th=[ 338], 50.00th=[11600], 60.00th=[11600], | 70.00th=[26346], 80.00th=[26346], 90.00th=[32113], 95.00th=[32113], | 99.00th=[32113], 99.50th=[32113], 99.90th=[32113], 99.95th=[32113], | 99.99th=[32113] bw ( KiB/s): min= 8192, max= 8192, per=24.16%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=12, BW=12.1MiB/s (12.7MB/s)(7168KiB/579msec) clat (msec): min=22, max=138, avg=68.43, stdev=51.52 lat (msec): min=22, max=138, avg=68.47, stdev=51.52 clat percentiles (msec): | 1.00th=[ 23], 5.00th=[ 23], 10.00th=[ 23], 20.00th=[ 27], | 30.00th=[ 33], 40.00th=[ 33], 50.00th=[ 35], 60.00th=[ 89], | 70.00th=[ 89], 80.00th=[ 136], 90.00th=[ 138], 95.00th=[ 138], | 99.00th=[ 138], 99.50th=[ 138], 99.90th=[ 138], 99.95th=[ 138], | 99.99th=[ 138] bw ( KiB/s): min=10240, max=10240, per=21.57%, avg=10240.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 lat (usec) : 500=16.67% lat (msec) : 20=8.33%, 50=50.00%, 100=8.33%, 250=16.67% cpu : usr=0.00%, sys=25.26%, ctx=390, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=33.1MiB/s (34.7MB/s), 3391KiB/s-22.4MiB/s (3472kB/s-23.5MB/s), io=20.0MiB (20.0MB), run=312-604msec WRITE: bw=46.4MiB/s (48.6MB/s), 12.1MiB/s-16.6MiB/s (12.7MB/s-17.4MB/s), io=28.0MiB (29.4MB), run=312-604msec rand-rw: (groupid=0, jobs=1): err= 0: pid=8183: Wed Apr 23 23:18:35 2025 read: IOPS=10, BW=10.2MiB/s (10.7MB/s)(7168KiB/686msec) clat (usec): min=12613, max=90121, avg=38826.55, stdev=26571.53 lat (usec): min=12614, max=90122, avg=38827.42, stdev=26571.57 clat percentiles (usec): | 1.00th=[12649], 5.00th=[12649], 10.00th=[12649], 20.00th=[13042], | 30.00th=[27395], 40.00th=[27395], 50.00th=[35914], 60.00th=[45351], | 70.00th=[45351], 80.00th=[47449], 90.00th=[89654], 95.00th=[89654], | 99.00th=[89654], 99.50th=[89654], 99.90th=[89654], 99.95th=[89654], | 99.99th=[89654] bw ( KiB/s): min=10240, max=10240, per=35.50%, avg=10240.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 write: IOPS=7, BW=7464KiB/s (7643kB/s)(5120KiB/686msec) clat (msec): min=37, max=137, avg=73.30, stdev=39.67 lat (msec): min=37, max=137, avg=73.35, stdev=39.66 clat percentiles (msec): | 1.00th=[ 37], 5.00th=[ 37], 10.00th=[ 37], 20.00th=[ 37], | 30.00th=[ 46], 40.00th=[ 46], 50.00th=[ 68], 60.00th=[ 68], | 70.00th=[ 79], 80.00th=[ 79], 90.00th=[ 138], 95.00th=[ 138], | 99.00th=[ 138], 99.50th=[ 138], 99.90th=[ 138], 99.95th=[ 138], | 99.99th=[ 138] bw ( KiB/s): min= 4096, max= 4096, per=10.14%, avg=4096.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 20=16.67%, 50=50.00%, 100=25.00%, 250=8.33% cpu : usr=0.00%, sys=11.53%, ctx=52, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8186: Wed Apr 23 23:18:35 2025 read: IOPS=3, BW=3471KiB/s (3554kB/s)(2048KiB/590msec) clat (usec): min=12369, max=73177, avg=42773.13, stdev=42997.84 lat (usec): min=12369, max=73178, avg=42774.07, stdev=42998.20 clat percentiles (usec): | 1.00th=[12387], 5.00th=[12387], 10.00th=[12387], 20.00th=[12387], | 30.00th=[12387], 40.00th=[12387], 50.00th=[12387], 60.00th=[72877], | 70.00th=[72877], 80.00th=[72877], 90.00th=[72877], 95.00th=[72877], | 99.00th=[72877], 99.50th=[72877], 99.90th=[72877], 99.95th=[72877], | 99.99th=[72877] bw ( KiB/s): min= 4096, max= 4096, per=14.20%, avg=4096.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 write: IOPS=16, BW=16.9MiB/s (17.8MB/s)(10.0MiB/590msec) clat (msec): min=25, max=118, avg=49.06, stdev=28.70 lat (msec): min=25, max=118, avg=49.11, stdev=28.71 clat percentiles (msec): | 1.00th=[ 26], 5.00th=[ 26], 10.00th=[ 26], 20.00th=[ 27], | 30.00th=[ 31], 40.00th=[ 32], 50.00th=[ 36], 60.00th=[ 44], | 70.00th=[ 63], 80.00th=[ 63], 90.00th=[ 71], 95.00th=[ 118], | 99.00th=[ 118], 99.50th=[ 118], 99.90th=[ 118], 99.95th=[ 118], | 99.99th=[ 118] bw ( KiB/s): min=14336, max=14336, per=35.50%, avg=14336.00, stdev= 0.00, samples=1 iops : min= 14, max= 14, avg=14.00, stdev= 0.00, samples=1 lat (msec) : 20=8.33%, 50=58.33%, 100=25.00%, 250=8.33% cpu : usr=0.00%, sys=16.64%, ctx=51, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8188: Wed Apr 23 23:18:35 2025 read: IOPS=8, BW=9102KiB/s (9321kB/s)(6144KiB/675msec) clat (usec): min=15673, max=62685, avg=34780.97, stdev=19465.29 lat (usec): min=15674, max=62685, avg=34781.77, stdev=19465.27 clat percentiles (usec): | 1.00th=[15664], 5.00th=[15664], 10.00th=[15664], 20.00th=[22152], | 30.00th=[22152], 40.00th=[22676], 50.00th=[22676], 60.00th=[30278], | 70.00th=[55313], 80.00th=[55313], 90.00th=[62653], 95.00th=[62653], | 99.00th=[62653], 99.50th=[62653], 99.90th=[62653], 99.95th=[62653], | 99.99th=[62653] bw ( KiB/s): min= 6144, max= 6144, per=21.30%, avg=6144.00, stdev= 0.00, samples=1 iops : min= 6, max= 6, avg= 6.00, stdev= 0.00, samples=1 write: IOPS=8, BW=9102KiB/s (9321kB/s)(6144KiB/675msec) clat (msec): min=26, max=154, avg=68.50, stdev=47.90 lat (msec): min=26, max=156, avg=69.01, stdev=48.80 clat percentiles (msec): | 1.00th=[ 27], 5.00th=[ 27], 10.00th=[ 27], 20.00th=[ 35], | 30.00th=[ 35], 40.00th=[ 52], 50.00th=[ 52], 60.00th=[ 52], | 70.00th=[ 93], 80.00th=[ 93], 90.00th=[ 155], 95.00th=[ 155], | 99.00th=[ 155], 99.50th=[ 155], 99.90th=[ 155], 99.95th=[ 155], | 99.99th=[ 155] bw ( KiB/s): min= 8192, max= 8192, per=20.29%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 lat (msec) : 20=8.33%, 50=41.67%, 100=41.67%, 250=8.33% cpu : usr=0.00%, sys=15.73%, ctx=70, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8190: Wed Apr 23 23:18:35 2025 read: IOPS=7, BW=7211KiB/s (7384kB/s)(5120KiB/710msec) clat (usec): min=15079, max=88885, avg=42184.85, stdev=29768.58 lat (usec): min=15080, max=88886, avg=42185.59, stdev=29768.52 clat percentiles (usec): | 1.00th=[15139], 5.00th=[15139], 10.00th=[15139], 20.00th=[15139], | 30.00th=[18744], 40.00th=[18744], 50.00th=[38011], 60.00th=[38011], | 70.00th=[50070], 80.00th=[50070], 90.00th=[88605], 95.00th=[88605], | 99.00th=[88605], 99.50th=[88605], 99.90th=[88605], 99.95th=[88605], | 99.99th=[88605] bw ( KiB/s): min= 4096, max= 4096, per=14.20%, avg=4096.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 write: IOPS=9, BW=9.86MiB/s (10.3MB/s)(7168KiB/710msec) clat (msec): min=22, max=132, avg=70.02, stdev=40.89 lat (msec): min=22, max=132, avg=70.07, stdev=40.89 clat percentiles (msec): | 1.00th=[ 23], 5.00th=[ 23], 10.00th=[ 23], 20.00th=[ 30], | 30.00th=[ 57], 40.00th=[ 57], 50.00th=[ 61], 60.00th=[ 77], | 70.00th=[ 77], 80.00th=[ 113], 90.00th=[ 133], 95.00th=[ 133], | 99.00th=[ 133], 99.50th=[ 133], 99.90th=[ 133], 99.95th=[ 133], | 99.99th=[ 133] bw ( KiB/s): min= 8192, max= 8192, per=20.29%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 lat (msec) : 20=16.67%, 50=25.00%, 100=41.67%, 250=16.67% cpu : usr=0.00%, sys=10.16%, ctx=42, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=28.2MiB/s (29.5MB/s), 3471KiB/s-10.2MiB/s (3554kB/s-10.7MB/s), io=20.0MiB (20.0MB), run=590-710msec WRITE: bw=39.4MiB/s (41.4MB/s), 7464KiB/s-16.9MiB/s (7643kB/s-17.8MB/s), io=28.0MiB (29.4MB), run=590-710msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=8197: Wed Apr 23 23:18:37 2025 read: IOPS=7, BW=29.3MiB/s (30.8MB/s)(12.0MiB/409msec) clat (msec): min=94, max=157, avg=119.40, stdev=33.30 lat (msec): min=94, max=157, avg=119.40, stdev=33.30 clat percentiles (msec): | 1.00th=[ 95], 5.00th=[ 95], 10.00th=[ 95], 20.00th=[ 95], | 30.00th=[ 95], 40.00th=[ 106], 50.00th=[ 106], 60.00th=[ 106], | 70.00th=[ 159], 80.00th=[ 159], 90.00th=[ 159], 95.00th=[ 159], | 99.00th=[ 159], 99.50th=[ 159], 99.90th=[ 159], 99.95th=[ 159], | 99.99th=[ 159] lat (msec) : 100=33.33%, 250=66.67% cpu : usr=0.00%, sys=34.80%, ctx=1815, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8200: Wed Apr 23 23:18:37 2025 read: IOPS=1, BW=6533KiB/s (6689kB/s)(4096KiB/627msec) clat (nsec): min=106504k, max=106504k, avg=106504166.00, stdev= 0.00 lat (nsec): min=106505k, max=106505k, avg=106505068.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 107], 5.00th=[ 107], 10.00th=[ 107], 20.00th=[ 107], | 30.00th=[ 107], 40.00th=[ 107], 50.00th=[ 107], 60.00th=[ 107], | 70.00th=[ 107], 80.00th=[ 107], 90.00th=[ 107], 95.00th=[ 107], | 99.00th=[ 107], 99.50th=[ 107], 99.90th=[ 107], 99.95th=[ 107], | 99.99th=[ 107] bw ( KiB/s): min= 8175, max= 8175, per=31.29%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=3, BW=12.8MiB/s (13.4MB/s)(8192KiB/627msec) clat (msec): min=173, max=324, avg=249.32, stdev=106.66 lat (msec): min=174, max=324, avg=249.58, stdev=106.57 clat percentiles (msec): | 1.00th=[ 174], 5.00th=[ 174], 10.00th=[ 174], 20.00th=[ 174], | 30.00th=[ 174], 40.00th=[ 174], 50.00th=[ 174], 60.00th=[ 326], | 70.00th=[ 326], 80.00th=[ 326], 90.00th=[ 326], 95.00th=[ 326], | 99.00th=[ 326], 99.50th=[ 326], 99.90th=[ 326], 99.95th=[ 326], | 99.99th=[ 326] bw ( KiB/s): min= 8175, max= 8175, per=19.56%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.00%, sys=23.96%, ctx=440, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8202: Wed Apr 23 23:18:37 2025 write: IOPS=3, BW=15.3MiB/s (16.0MB/s)(12.0MiB/784msec) clat (msec): min=226, max=292, avg=252.60, stdev=35.14 lat (msec): min=226, max=292, avg=252.87, stdev=35.16 clat percentiles (msec): | 1.00th=[ 228], 5.00th=[ 228], 10.00th=[ 228], 20.00th=[ 228], | 30.00th=[ 228], 40.00th=[ 239], 50.00th=[ 239], 60.00th=[ 239], | 70.00th=[ 292], 80.00th=[ 292], 90.00th=[ 292], 95.00th=[ 292], | 99.00th=[ 292], 99.50th=[ 292], 99.90th=[ 292], 99.95th=[ 292], | 99.99th=[ 292] bw ( KiB/s): min= 8192, max= 8192, per=19.60%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.00%, sys=31.67%, ctx=50, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8204: Wed Apr 23 23:18:37 2025 write: IOPS=5, BW=21.4MiB/s (22.4MB/s)(12.0MiB/562msec) clat (msec): min=99, max=253, avg=184.48, stdev=78.51 lat (msec): min=99, max=254, avg=184.68, stdev=78.55 clat percentiles (msec): | 1.00th=[ 100], 5.00th=[ 100], 10.00th=[ 100], 20.00th=[ 100], | 30.00th=[ 100], 40.00th=[ 201], 50.00th=[ 201], 60.00th=[ 201], | 70.00th=[ 255], 80.00th=[ 255], 90.00th=[ 255], 95.00th=[ 255], | 99.00th=[ 255], 99.50th=[ 255], 99.90th=[ 255], 99.95th=[ 255], | 99.99th=[ 255] bw ( KiB/s): min=16384, max=16384, per=39.20%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 250=33.33%, 500=33.33% cpu : usr=0.00%, sys=40.11%, ctx=75, majf=0, minf=28 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=25.5MiB/s (26.8MB/s), 6533KiB/s-29.3MiB/s (6689kB/s-30.8MB/s), io=16.0MiB (16.8MB), run=409-627msec WRITE: bw=40.8MiB/s (42.8MB/s), 12.8MiB/s-21.4MiB/s (13.4MB/s-22.4MB/s), io=32.0MiB (33.6MB), run=562-784msec rand-rw: (groupid=0, jobs=1): err= 0: pid=8198: Wed Apr 23 23:18:37 2025 read: IOPS=5, BW=22.5MiB/s (23.6MB/s)(12.0MiB/534msec) clat (msec): min=127, max=220, avg=167.26, stdev=47.67 lat (msec): min=127, max=220, avg=167.26, stdev=47.67 clat percentiles (msec): | 1.00th=[ 128], 5.00th=[ 128], 10.00th=[ 128], 20.00th=[ 128], | 30.00th=[ 128], 40.00th=[ 155], 50.00th=[ 155], 60.00th=[ 155], | 70.00th=[ 220], 80.00th=[ 220], 90.00th=[ 220], 95.00th=[ 220], | 99.00th=[ 220], 99.50th=[ 220], 99.90th=[ 220], 99.95th=[ 220], | 99.99th=[ 220] lat (msec) : 250=100.00% cpu : usr=0.19%, sys=12.38%, ctx=42, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8199: Wed Apr 23 23:18:37 2025 read: IOPS=1, BW=4900KiB/s (5017kB/s)(4096KiB/836msec) clat (nsec): min=116429k, max=116429k, avg=116429164.00, stdev= 0.00 lat (nsec): min=116431k, max=116431k, avg=116431050.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 117], 5.00th=[ 117], 10.00th=[ 117], 20.00th=[ 117], | 30.00th=[ 117], 40.00th=[ 117], 50.00th=[ 117], 60.00th=[ 117], | 70.00th=[ 117], 80.00th=[ 117], 90.00th=[ 117], 95.00th=[ 117], | 99.00th=[ 117], 99.50th=[ 117], 99.90th=[ 117], 99.95th=[ 117], | 99.99th=[ 117] bw ( KiB/s): min= 5535, max= 5535, per=28.24%, avg=5535.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=2, BW=9799KiB/s (10.0MB/s)(8192KiB/836msec) clat (msec): min=159, max=550, avg=355.02, stdev=276.60 lat (msec): min=159, max=550, avg=355.24, stdev=276.50 clat percentiles (msec): | 1.00th=[ 161], 5.00th=[ 161], 10.00th=[ 161], 20.00th=[ 161], | 30.00th=[ 161], 40.00th=[ 161], 50.00th=[ 161], 60.00th=[ 550], | 70.00th=[ 550], 80.00th=[ 550], 90.00th=[ 550], 95.00th=[ 550], | 99.00th=[ 550], 99.50th=[ 550], 99.90th=[ 550], 99.95th=[ 550], | 99.99th=[ 550] bw ( KiB/s): min= 5535, max= 5535, per=14.12%, avg=5535.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 750=33.33% cpu : usr=0.00%, sys=11.62%, ctx=104, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8201: Wed Apr 23 23:18:37 2025 write: IOPS=8, BW=34.9MiB/s (36.6MB/s)(12.0MiB/344msec) clat (msec): min=52, max=219, avg=108.63, stdev=96.27 lat (msec): min=52, max=220, avg=108.95, stdev=96.23 clat percentiles (msec): | 1.00th=[ 53], 5.00th=[ 53], 10.00th=[ 53], 20.00th=[ 53], | 30.00th=[ 53], 40.00th=[ 55], 50.00th=[ 55], 60.00th=[ 55], | 70.00th=[ 220], 80.00th=[ 220], 90.00th=[ 220], 95.00th=[ 220], | 99.00th=[ 220], 99.50th=[ 220], 99.90th=[ 220], 99.95th=[ 220], | 99.99th=[ 220] lat (msec) : 100=66.67%, 250=33.33% cpu : usr=0.00%, sys=19.83%, ctx=58, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8203: Wed Apr 23 23:18:37 2025 write: IOPS=8, BW=34.4MiB/s (36.1MB/s)(12.0MiB/349msec) clat (msec): min=53, max=163, avg=113.83, stdev=55.83 lat (msec): min=53, max=163, avg=114.13, stdev=55.93 clat percentiles (msec): | 1.00th=[ 54], 5.00th=[ 54], 10.00th=[ 54], 20.00th=[ 54], | 30.00th=[ 54], 40.00th=[ 126], 50.00th=[ 126], 60.00th=[ 126], | 70.00th=[ 163], 80.00th=[ 163], 90.00th=[ 163], 95.00th=[ 163], | 99.00th=[ 163], 99.50th=[ 163], 99.90th=[ 163], 99.95th=[ 163], | 99.99th=[ 163] lat (msec) : 100=33.33%, 250=66.67% cpu : usr=0.29%, sys=22.13%, ctx=51, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=19.1MiB/s (20.1MB/s), 4900KiB/s-22.5MiB/s (5017kB/s-23.6MB/s), io=16.0MiB (16.8MB), run=534-836msec WRITE: bw=38.3MiB/s (40.1MB/s), 9799KiB/s-34.9MiB/s (10.0MB/s-36.6MB/s), io=32.0MiB (33.6MB), run=344-836msec