== sanity test 398b: DIO and buffer IO race ============== 19:39:47 (1746142787) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 2.90089 s, 17.4 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=22242: Thu May 1 19:40:41 2025 read: IOPS=31, BW=127KiB/s (130kB/s)(5992KiB/47182msec) clat (usec): min=81, max=105310, avg=17548.53, stdev=11950.31 lat (usec): min=82, max=105310, avg=17550.23, stdev=11950.06 clat percentiles (usec): | 1.00th=[ 95], 5.00th=[ 126], 10.00th=[ 6063], 20.00th=[ 8848], | 30.00th=[ 11076], 40.00th=[ 12911], 50.00th=[ 15139], 60.00th=[ 17957], | 70.00th=[ 20841], 80.00th=[ 25560], 90.00th=[ 32113], 95.00th=[ 39060], | 99.00th=[ 61604], 99.50th=[ 68682], 99.90th=[ 98042], 99.95th=[105382], | 99.99th=[105382] bw ( KiB/s): min= 24, max= 208, per=24.87%, avg=125.83, stdev=35.72, samples=94 iops : min= 6, max= 52, avg=31.03, stdev= 8.92, samples=94 write: IOPS=33, BW=133KiB/s (137kB/s)(6296KiB/47182msec) clat (usec): min=628, max=325953, avg=13146.27, stdev=16349.11 lat (usec): min=629, max=325955, avg=13150.42, stdev=16348.69 clat percentiles (usec): | 1.00th=[ 693], 5.00th=[ 750], 10.00th=[ 799], 20.00th=[ 955], | 30.00th=[ 2180], 40.00th=[ 4228], 50.00th=[ 8979], 60.00th=[ 12518], | 70.00th=[ 16909], 80.00th=[ 23200], 90.00th=[ 31851], 95.00th=[ 41157], | 99.00th=[ 61080], 99.50th=[ 78119], 99.90th=[107480], 99.95th=[325059], | 99.99th=[325059] bw ( KiB/s): min= 8, max= 232, per=24.95%, avg=132.23, stdev=43.78, samples=94 iops : min= 2, max= 58, avg=32.63, stdev=10.99, samples=94 lat (usec) : 100=0.85%, 250=2.34%, 750=2.60%, 1000=8.30% lat (msec) : 2=4.33%, 4=5.44%, 10=15.69%, 20=31.77%, 50=26.60% lat (msec) : 100=1.99%, 250=0.07%, 500=0.03% cpu : usr=0.17%, sys=9.87%, ctx=4972, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22243: Thu May 1 19:40:41 2025 read: IOPS=31, BW=127KiB/s (130kB/s)(5940KiB/46755msec) clat (usec): min=87, max=106273, avg=17648.00, stdev=11584.72 lat (usec): min=88, max=106273, avg=17650.96, stdev=11584.29 clat percentiles (usec): | 1.00th=[ 97], 5.00th=[ 130], 10.00th=[ 6390], 20.00th=[ 9110], | 30.00th=[ 10945], 40.00th=[ 13304], 50.00th=[ 15533], 60.00th=[ 18220], | 70.00th=[ 21627], 80.00th=[ 25297], 90.00th=[ 31327], 95.00th=[ 39060], | 99.00th=[ 56886], 99.50th=[ 64750], 99.90th=[106431], 99.95th=[106431], | 99.99th=[106431] bw ( KiB/s): min= 7, max= 192, per=24.95%, avg=126.25, stdev=33.73, samples=93 iops : min= 1, max= 48, avg=31.18, stdev= 8.54, samples=93 write: IOPS=33, BW=136KiB/s (139kB/s)(6348KiB/46755msec) clat (usec): min=515, max=344102, avg=12813.11, stdev=16144.87 lat (usec): min=516, max=344103, avg=12816.45, stdev=16145.85 clat percentiles (usec): | 1.00th=[ 676], 5.00th=[ 742], 10.00th=[ 791], 20.00th=[ 906], | 30.00th=[ 1745], 40.00th=[ 3785], 50.00th=[ 8979], 60.00th=[ 12256], | 70.00th=[ 16188], 80.00th=[ 22414], 90.00th=[ 32375], 95.00th=[ 40109], | 99.00th=[ 61604], 99.50th=[ 70779], 99.90th=[ 88605], 99.95th=[346031], | 99.99th=[346031] bw ( KiB/s): min= 7, max= 272, per=25.40%, avg=134.63, stdev=47.02, samples=93 iops : min= 1, max= 68, avg=33.29, stdev=11.87, samples=93 lat (usec) : 100=0.75%, 250=2.28%, 500=0.10%, 750=3.03%, 1000=9.08% lat (msec) : 2=4.59%, 4=4.52%, 10=14.91%, 20=31.58%, 50=27.12% lat (msec) : 100=1.95%, 250=0.07%, 500=0.03% cpu : usr=0.18%, sys=9.89%, ctx=4976, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22244: Thu May 1 19:40:41 2025 read: IOPS=32, BW=130KiB/s (133kB/s)(6028KiB/46372msec) clat (usec): min=78, max=79884, avg=17370.56, stdev=10908.68 lat (usec): min=78, max=79885, avg=17372.16, stdev=10908.33 clat percentiles (usec): | 1.00th=[ 98], 5.00th=[ 137], 10.00th=[ 6783], 20.00th=[ 9241], | 30.00th=[11076], 40.00th=[13173], 50.00th=[15533], 60.00th=[18220], | 70.00th=[20841], 80.00th=[24773], 90.00th=[31065], 95.00th=[36439], | 99.00th=[54789], 99.50th=[63177], 99.90th=[76022], 99.95th=[80217], | 99.99th=[80217] bw ( KiB/s): min= 64, max= 200, per=25.83%, avg=130.69, stdev=31.99, samples=91 iops : min= 16, max= 50, avg=32.34, stdev= 8.00, samples=91 write: IOPS=33, BW=135KiB/s (138kB/s)(6260KiB/46372msec) clat (usec): min=507, max=380788, avg=12770.70, stdev=17234.05 lat (usec): min=508, max=380790, avg=12775.07, stdev=17233.43 clat percentiles (usec): | 1.00th=[ 668], 5.00th=[ 734], 10.00th=[ 783], 20.00th=[ 881], | 30.00th=[ 1434], 40.00th=[ 3064], 50.00th=[ 7635], 60.00th=[ 12125], | 70.00th=[ 16581], 80.00th=[ 22414], 90.00th=[ 32900], 95.00th=[ 40633], | 99.00th=[ 62129], 99.50th=[ 74974], 99.90th=[122160], 99.95th=[379585], | 99.99th=[379585] bw ( KiB/s): min= 48, max= 287, per=25.59%, avg=135.64, stdev=44.79, samples=91 iops : min= 12, max= 71, avg=33.57, stdev=11.23, samples=91 lat (usec) : 100=0.72%, 250=2.12%, 500=0.07%, 750=3.35%, 1000=9.11% lat (msec) : 2=4.82%, 4=5.24%, 10=14.55%, 20=31.97%, 50=26.20% lat (msec) : 100=1.76%, 250=0.07%, 500=0.03% cpu : usr=0.17%, sys=9.89%, ctx=4848, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22245: Thu May 1 19:40:41 2025 read: IOPS=31, BW=128KiB/s (131kB/s)(6052KiB/47373msec) clat (usec): min=71, max=112201, avg=17975.80, stdev=12439.20 lat (usec): min=72, max=112202, avg=17981.70, stdev=12454.18 clat percentiles (usec): | 1.00th=[ 95], 5.00th=[ 135], 10.00th=[ 6652], 20.00th=[ 9110], | 30.00th=[ 11076], 40.00th=[ 13304], 50.00th=[ 15795], 60.00th=[ 18220], | 70.00th=[ 21365], 80.00th=[ 25297], 90.00th=[ 32375], 95.00th=[ 38536], | 99.00th=[ 57934], 99.50th=[ 83362], 99.90th=[111674], 99.95th=[112722], | 99.99th=[112722] bw ( KiB/s): min= 55, max= 208, per=25.11%, avg=127.08, stdev=32.80, samples=93 iops : min= 13, max= 52, avg=31.41, stdev= 8.27, samples=93 write: IOPS=32, BW=132KiB/s (135kB/s)(6236KiB/47373msec) clat (usec): min=550, max=347525, avg=12787.51, stdev=16478.69 lat (usec): min=551, max=347528, avg=12791.40, stdev=16478.21 clat percentiles (usec): | 1.00th=[ 685], 5.00th=[ 734], 10.00th=[ 783], 20.00th=[ 898], | 30.00th=[ 1926], 40.00th=[ 3523], 50.00th=[ 8356], 60.00th=[ 12387], | 70.00th=[ 16319], 80.00th=[ 22414], 90.00th=[ 32375], 95.00th=[ 40633], | 99.00th=[ 62129], 99.50th=[ 72877], 99.90th=[130548], 99.95th=[346031], | 99.99th=[346031] bw ( KiB/s): min= 47, max= 261, per=24.78%, avg=131.34, stdev=42.58, samples=93 iops : min= 11, max= 65, avg=32.47, stdev=10.68, samples=93 lat (usec) : 100=0.85%, 250=1.95%, 500=0.10%, 750=3.22%, 1000=8.50% lat (msec) : 2=3.97%, 4=5.40%, 10=15.10%, 20=32.23%, 50=26.73% lat (msec) : 100=1.76%, 250=0.16%, 500=0.03% cpu : usr=0.12%, sys=9.69%, ctx=4871, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=507KiB/s (519kB/s), 127KiB/s-130KiB/s (130kB/s-133kB/s), io=23.4MiB (24.6MB), run=46372-47373msec WRITE: bw=531KiB/s (543kB/s), 132KiB/s-136KiB/s (135kB/s-139kB/s), io=24.6MiB (25.7MB), run=46372-47373msec rand-rw: (groupid=0, jobs=1): err= 0: pid=22247: Thu May 1 19:41:04 2025 read: IOPS=21, BW=84.5KiB/s (86.5kB/s)(5992KiB/70915msec) clat (msec): min=2, max=332, avg=17.97, stdev=17.80 lat (msec): min=2, max=332, avg=17.97, stdev=17.80 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 6], 10.00th=[ 7], 20.00th=[ 8], | 30.00th=[ 9], 40.00th=[ 11], 50.00th=[ 13], 60.00th=[ 15], | 70.00th=[ 18], 80.00th=[ 26], 90.00th=[ 38], 95.00th=[ 52], | 99.00th=[ 89], 99.50th=[ 100], 99.90th=[ 136], 99.95th=[ 334], | 99.99th=[ 334] bw ( KiB/s): min= 7, max= 216, per=24.65%, avg=83.33, stdev=46.17, samples=141 iops : min= 1, max= 54, avg=20.45, stdev=11.62, samples=141 write: IOPS=22, BW=88.8KiB/s (90.9kB/s)(6296KiB/70915msec) clat (msec): min=5, max=167, avg=27.80, stdev=18.03 lat (msec): min=5, max=167, avg=27.80, stdev=18.03 clat percentiles (msec): | 1.00th=[ 8], 5.00th=[ 11], 10.00th=[ 12], 20.00th=[ 15], | 30.00th=[ 17], 40.00th=[ 20], 50.00th=[ 23], 60.00th=[ 27], | 70.00th=[ 32], 80.00th=[ 39], 90.00th=[ 53], 95.00th=[ 65], | 99.00th=[ 92], 99.50th=[ 102], 99.90th=[ 144], 99.95th=[ 167], | 99.99th=[ 167] bw ( KiB/s): min= 23, max= 200, per=24.88%, avg=88.09, stdev=42.21, samples=140 iops : min= 5, max= 50, avg=21.63, stdev=10.62, samples=140 lat (msec) : 4=0.29%, 10=20.64%, 20=36.39%, 50=34.31%, 100=7.94% lat (msec) : 250=0.39%, 500=0.03% cpu : usr=0.06%, sys=4.75%, ctx=3705, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22248: Thu May 1 19:41:04 2025 read: IOPS=21, BW=84.5KiB/s (86.6kB/s)(5940KiB/70267msec) clat (msec): min=3, max=194, avg=17.49, stdev=15.52 lat (msec): min=3, max=194, avg=17.49, stdev=15.52 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 6], 10.00th=[ 7], 20.00th=[ 8], | 30.00th=[ 9], 40.00th=[ 11], 50.00th=[ 13], 60.00th=[ 15], | 70.00th=[ 19], 80.00th=[ 24], 90.00th=[ 36], 95.00th=[ 50], | 99.00th=[ 77], 99.50th=[ 89], 99.90th=[ 136], 99.95th=[ 194], | 99.99th=[ 194] bw ( KiB/s): min= 7, max= 223, per=24.86%, avg=84.04, stdev=44.06, samples=140 iops : min= 1, max= 55, avg=20.65, stdev=11.01, samples=140 write: IOPS=22, BW=90.3KiB/s (92.5kB/s)(6348KiB/70267msec) clat (msec): min=5, max=170, avg=27.72, stdev=17.87 lat (msec): min=5, max=170, avg=27.73, stdev=17.87 clat percentiles (msec): | 1.00th=[ 9], 5.00th=[ 11], 10.00th=[ 13], 20.00th=[ 15], | 30.00th=[ 17], 40.00th=[ 20], 50.00th=[ 23], 60.00th=[ 27], | 70.00th=[ 32], 80.00th=[ 39], 90.00th=[ 50], 95.00th=[ 61], | 99.00th=[ 99], 99.50th=[ 112], 99.90th=[ 161], 99.95th=[ 171], | 99.99th=[ 171] bw ( KiB/s): min= 16, max= 208, per=25.49%, avg=90.22, stdev=43.35, samples=139 iops : min= 4, max= 52, avg=22.20, stdev=10.88, samples=139 lat (msec) : 4=0.39%, 10=19.56%, 20=37.70%, 50=35.16%, 100=6.64% lat (msec) : 250=0.55% cpu : usr=0.13%, sys=4.71%, ctx=3648, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22249: Thu May 1 19:41:04 2025 read: IOPS=21, BW=85.8KiB/s (87.8kB/s)(6028KiB/70270msec) clat (msec): min=3, max=131, avg=17.80, stdev=15.23 lat (msec): min=3, max=131, avg=17.80, stdev=15.23 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 6], 10.00th=[ 7], 20.00th=[ 8], | 30.00th=[ 10], 40.00th=[ 11], 50.00th=[ 13], 60.00th=[ 16], | 70.00th=[ 19], 80.00th=[ 25], 90.00th=[ 37], 95.00th=[ 51], | 99.00th=[ 79], 99.50th=[ 93], 99.90th=[ 123], 99.95th=[ 131], | 99.99th=[ 131] bw ( KiB/s): min= 23, max= 192, per=25.40%, avg=85.85, stdev=41.37, samples=139 iops : min= 5, max= 48, avg=21.16, stdev=10.42, samples=139 write: IOPS=22, BW=89.1KiB/s (91.2kB/s)(6260KiB/70270msec) clat (msec): min=6, max=378, avg=27.66, stdev=19.18 lat (msec): min=6, max=378, avg=27.66, stdev=19.18 clat percentiles (msec): | 1.00th=[ 9], 5.00th=[ 11], 10.00th=[ 13], 20.00th=[ 15], | 30.00th=[ 17], 40.00th=[ 20], 50.00th=[ 22], 60.00th=[ 26], | 70.00th=[ 31], 80.00th=[ 40], 90.00th=[ 50], 95.00th=[ 62], | 99.00th=[ 88], 99.50th=[ 103], 99.90th=[ 146], 99.95th=[ 380], | 99.99th=[ 380] bw ( KiB/s): min= 8, max= 208, per=24.97%, avg=88.39, stdev=43.68, samples=140 iops : min= 2, max= 52, avg=21.79, stdev=10.99, samples=140 lat (msec) : 4=0.23%, 10=18.33%, 20=39.75%, 50=34.18%, 100=7.06% lat (msec) : 250=0.42%, 500=0.03% cpu : usr=0.11%, sys=4.72%, ctx=3656, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22250: Thu May 1 19:41:04 2025 read: IOPS=21, BW=86.0KiB/s (88.1kB/s)(6052KiB/70338msec) clat (msec): min=3, max=131, avg=17.23, stdev=14.70 lat (msec): min=3, max=131, avg=17.23, stdev=14.70 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 6], 10.00th=[ 6], 20.00th=[ 8], | 30.00th=[ 9], 40.00th=[ 11], 50.00th=[ 13], 60.00th=[ 15], | 70.00th=[ 19], 80.00th=[ 24], 90.00th=[ 36], 95.00th=[ 48], | 99.00th=[ 79], 99.50th=[ 88], 99.90th=[ 105], 99.95th=[ 132], | 99.99th=[ 132] bw ( KiB/s): min= 8, max= 240, per=25.41%, avg=85.88, stdev=44.93, samples=139 iops : min= 2, max= 60, avg=21.19, stdev=11.22, samples=139 write: IOPS=22, BW=88.7KiB/s (90.8kB/s)(6236KiB/70338msec) clat (msec): min=6, max=188, avg=28.17, stdev=19.36 lat (msec): min=6, max=188, avg=28.17, stdev=19.36 clat percentiles (msec): | 1.00th=[ 9], 5.00th=[ 12], 10.00th=[ 13], 20.00th=[ 15], | 30.00th=[ 17], 40.00th=[ 20], 50.00th=[ 23], 60.00th=[ 26], | 70.00th=[ 32], 80.00th=[ 39], 90.00th=[ 52], 95.00th=[ 65], | 99.00th=[ 104], 99.50th=[ 122], 99.90th=[ 161], 99.95th=[ 188], | 99.99th=[ 188] bw ( KiB/s): min= 8, max= 208, per=24.85%, avg=87.96, stdev=43.56, samples=140 iops : min= 2, max= 52, avg=21.71, stdev=10.88, samples=140 lat (msec) : 4=0.49%, 10=20.44%, 20=37.34%, 50=33.98%, 100=7.06% lat (msec) : 250=0.68% cpu : usr=0.10%, sys=4.78%, ctx=3641, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=339KiB/s (347kB/s), 84.5KiB/s-86.0KiB/s (86.5kB/s-88.1kB/s), io=23.4MiB (24.6MB), run=70267-70915msec WRITE: bw=355KiB/s (363kB/s), 88.7KiB/s-90.3KiB/s (90.8kB/s-92.5kB/s), io=24.6MiB (25.7MB), run=70267-70915msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=22321: Thu May 1 19:41:17 2025 read: IOPS=32, BW=527KiB/s (539kB/s)(5920KiB/11244msec) clat (usec): min=86, max=74844, avg=15992.87, stdev=11414.77 lat (usec): min=86, max=74845, avg=15993.88, stdev=11414.73 clat percentiles (usec): | 1.00th=[ 92], 5.00th=[ 113], 10.00th=[ 143], 20.00th=[ 7504], | 30.00th=[ 9634], 40.00th=[11994], 50.00th=[14484], 60.00th=[17695], | 70.00th=[20317], 80.00th=[23462], 90.00th=[29754], 95.00th=[39060], | 99.00th=[49546], 99.50th=[63177], 99.90th=[74974], 99.95th=[74974], | 99.99th=[74974] bw ( KiB/s): min= 191, max= 768, per=24.20%, avg=515.36, stdev=136.07, samples=22 iops : min= 11, max= 48, avg=31.95, stdev= 8.50, samples=22 write: IOPS=35, BW=566KiB/s (580kB/s)(6368KiB/11244msec) clat (usec): min=902, max=90904, avg=12755.68, stdev=13199.31 lat (usec): min=903, max=90906, avg=12757.64, stdev=13199.27 clat percentiles (usec): | 1.00th=[ 971], 5.00th=[ 1106], 10.00th=[ 1172], 20.00th=[ 1500], | 30.00th=[ 2671], 40.00th=[ 4948], 50.00th=[ 8586], 60.00th=[12256], | 70.00th=[16712], 80.00th=[23200], 90.00th=[31065], 95.00th=[37487], | 99.00th=[53740], 99.50th=[73925], 99.90th=[90702], 99.95th=[90702], | 99.99th=[90702] bw ( KiB/s): min= 127, max= 864, per=24.99%, avg=560.05, stdev=202.33, samples=22 iops : min= 7, max= 54, avg=34.73, stdev=12.68, samples=22 lat (usec) : 100=0.78%, 250=5.08%, 500=0.65%, 1000=0.65% lat (msec) : 2=12.63%, 4=6.38%, 10=17.19%, 20=29.30%, 50=26.43% lat (msec) : 100=0.91% cpu : usr=0.05%, sys=11.53%, ctx=1370, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22323: Thu May 1 19:41:17 2025 read: IOPS=33, BW=529KiB/s (541kB/s)(5744KiB/10868msec) clat (usec): min=82, max=71412, avg=15818.08, stdev=11082.02 lat (usec): min=83, max=71414, avg=15822.82, stdev=11077.04 clat percentiles (usec): | 1.00th=[ 89], 5.00th=[ 111], 10.00th=[ 131], 20.00th=[ 7767], | 30.00th=[10421], 40.00th=[12256], 50.00th=[14222], 60.00th=[16581], | 70.00th=[19792], 80.00th=[23200], 90.00th=[27919], 95.00th=[36963], | 99.00th=[55837], 99.50th=[61604], 99.90th=[71828], 99.95th=[71828], | 99.99th=[71828] bw ( KiB/s): min= 288, max= 768, per=24.65%, avg=525.10, stdev=132.32, samples=21 iops : min= 18, max= 48, avg=32.57, stdev= 8.21, samples=21 write: IOPS=37, BW=602KiB/s (617kB/s)(6544KiB/10868msec) clat (usec): min=799, max=72854, avg=12511.24, stdev=13228.36 lat (usec): min=799, max=72855, avg=12513.02, stdev=13228.35 clat percentiles (usec): | 1.00th=[ 971], 5.00th=[ 1090], 10.00th=[ 1188], 20.00th=[ 1696], | 30.00th=[ 2802], 40.00th=[ 4948], 50.00th=[ 8356], 60.00th=[11600], | 70.00th=[14877], 80.00th=[22152], 90.00th=[30802], 95.00th=[40109], | 99.00th=[55313], 99.50th=[63177], 99.90th=[72877], 99.95th=[72877], | 99.99th=[72877] bw ( KiB/s): min= 288, max= 960, per=26.56%, avg=595.14, stdev=155.39, samples=21 iops : min= 18, max= 60, avg=36.95, stdev= 9.69, samples=21 lat (usec) : 100=0.91%, 250=4.30%, 750=0.13%, 1000=0.91% lat (msec) : 2=12.24%, 4=6.12%, 10=18.62%, 20=31.12%, 50=23.70% lat (msec) : 100=1.95% cpu : usr=0.10%, sys=11.63%, ctx=1423, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22325: Thu May 1 19:41:17 2025 read: IOPS=35, BW=569KiB/s (583kB/s)(6256KiB/10991msec) clat (usec): min=69, max=79378, avg=15597.38, stdev=11439.69 lat (usec): min=70, max=79379, avg=15601.76, stdev=11440.12 clat percentiles (usec): | 1.00th=[ 93], 5.00th=[ 110], 10.00th=[ 133], 20.00th=[ 7701], | 30.00th=[ 9896], 40.00th=[12256], 50.00th=[13829], 60.00th=[16057], | 70.00th=[19268], 80.00th=[23462], 90.00th=[29230], 95.00th=[34866], | 99.00th=[62653], 99.50th=[71828], 99.90th=[79168], 99.95th=[79168], | 99.99th=[79168] bw ( KiB/s): min= 127, max= 928, per=26.48%, avg=564.05, stdev=190.83, samples=21 iops : min= 7, max= 58, avg=34.81, stdev=12.06, samples=21 write: IOPS=34, BW=549KiB/s (562kB/s)(6032KiB/10991msec) clat (usec): min=823, max=79745, avg=12533.43, stdev=13863.17 lat (usec): min=824, max=79746, avg=12535.24, stdev=13863.16 clat percentiles (usec): | 1.00th=[ 938], 5.00th=[ 1090], 10.00th=[ 1172], 20.00th=[ 1319], | 30.00th=[ 2442], 40.00th=[ 4015], 50.00th=[ 8029], 60.00th=[11207], | 70.00th=[15533], 80.00th=[21365], 90.00th=[30802], 95.00th=[42730], | 99.00th=[58983], 99.50th=[76022], 99.90th=[80217], 99.95th=[80217], | 99.99th=[80217] bw ( KiB/s): min= 159, max= 862, per=23.61%, avg=529.00, stdev=157.53, samples=21 iops : min= 9, max= 53, avg=32.62, stdev= 9.82, samples=21 lat (usec) : 100=1.30%, 250=5.21%, 500=0.13%, 1000=0.91% lat (msec) : 2=12.24%, 4=7.03%, 10=16.54%, 20=30.60%, 50=24.09% lat (msec) : 100=1.95% cpu : usr=0.18%, sys=11.32%, ctx=1324, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22328: Thu May 1 19:41:17 2025 read: IOPS=33, BW=540KiB/s (553kB/s)(6032KiB/11170msec) clat (usec): min=88, max=81921, avg=15912.29, stdev=11241.40 lat (usec): min=88, max=81922, avg=15913.33, stdev=11241.45 clat percentiles (usec): | 1.00th=[ 94], 5.00th=[ 115], 10.00th=[ 139], 20.00th=[ 8356], | 30.00th=[10421], 40.00th=[12256], 50.00th=[13960], 60.00th=[16188], | 70.00th=[18744], 80.00th=[23462], 90.00th=[30278], 95.00th=[34866], | 99.00th=[64226], 99.50th=[66323], 99.90th=[82314], 99.95th=[82314], | 99.99th=[82314] bw ( KiB/s): min= 64, max= 763, per=25.11%, avg=534.86, stdev=145.49, samples=22 iops : min= 4, max= 47, avg=33.05, stdev= 9.03, samples=22 write: IOPS=35, BW=560KiB/s (574kB/s)(6256KiB/11170msec) clat (usec): min=739, max=97582, avg=12626.94, stdev=12930.70 lat (usec): min=740, max=97585, avg=12630.21, stdev=12933.92 clat percentiles (usec): | 1.00th=[ 914], 5.00th=[ 1074], 10.00th=[ 1172], 20.00th=[ 1467], | 30.00th=[ 2933], 40.00th=[ 5538], 50.00th=[ 8848], 60.00th=[11731], | 70.00th=[15533], 80.00th=[22676], 90.00th=[30278], 95.00th=[35914], | 99.00th=[53740], 99.50th=[65274], 99.90th=[98042], 99.95th=[98042], | 99.99th=[98042] bw ( KiB/s): min= 128, max= 896, per=24.97%, avg=559.59, stdev=174.10, samples=22 iops : min= 8, max= 56, avg=34.59, stdev=10.94, samples=22 lat (usec) : 100=1.17%, 250=4.04%, 750=0.39%, 1000=1.43% lat (msec) : 2=10.94%, 4=5.08%, 10=17.84%, 20=33.72%, 50=23.83% lat (msec) : 100=1.56% cpu : usr=0.08%, sys=11.47%, ctx=1378, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2130KiB/s (2181kB/s), 527KiB/s-569KiB/s (539kB/s-583kB/s), io=23.4MiB (24.5MB), run=10868-11244msec WRITE: bw=2241KiB/s (2295kB/s), 549KiB/s-602KiB/s (562kB/s-617kB/s), io=24.6MiB (25.8MB), run=10868-11244msec rand-rw: (groupid=0, jobs=1): err= 0: pid=22322: Thu May 1 19:41:22 2025 read: IOPS=22, BW=362KiB/s (371kB/s)(5920KiB/16344msec) clat (msec): min=3, max=113, avg=15.85, stdev=14.29 lat (msec): min=3, max=113, avg=15.85, stdev=14.29 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 6], 10.00th=[ 7], 20.00th=[ 8], | 30.00th=[ 9], 40.00th=[ 10], 50.00th=[ 11], 60.00th=[ 13], | 70.00th=[ 15], 80.00th=[ 23], 90.00th=[ 34], 95.00th=[ 50], | 99.00th=[ 71], 99.50th=[ 78], 99.90th=[ 113], 99.95th=[ 113], | 99.99th=[ 113] bw ( KiB/s): min= 127, max= 800, per=24.25%, avg=352.41, stdev=175.35, samples=32 iops : min= 7, max= 50, avg=21.69, stdev=11.11, samples=32 write: IOPS=24, BW=390KiB/s (399kB/s)(6368KiB/16344msec) clat (msec): min=7, max=104, avg=26.21, stdev=17.54 lat (msec): min=7, max=104, avg=26.22, stdev=17.54 clat percentiles (msec): | 1.00th=[ 9], 5.00th=[ 11], 10.00th=[ 12], 20.00th=[ 14], | 30.00th=[ 16], 40.00th=[ 18], 50.00th=[ 20], 60.00th=[ 23], | 70.00th=[ 28], 80.00th=[ 37], 90.00th=[ 50], 95.00th=[ 70], | 99.00th=[ 89], 99.50th=[ 96], 99.90th=[ 106], 99.95th=[ 106], | 99.99th=[ 106] bw ( KiB/s): min= 159, max= 768, per=25.20%, avg=385.28, stdev=196.39, samples=32 iops : min= 9, max= 48, avg=23.75, stdev=12.40, samples=32 lat (msec) : 4=0.52%, 10=23.96%, 20=39.58%, 50=28.65%, 100=7.03% lat (msec) : 250=0.26% cpu : usr=0.10%, sys=5.13%, ctx=887, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22324: Thu May 1 19:41:22 2025 read: IOPS=21, BW=349KiB/s (357kB/s)(5744KiB/16479msec) clat (usec): min=3957, max=88911, avg=15507.29, stdev=13560.16 lat (usec): min=3959, max=88912, avg=15510.08, stdev=13559.62 clat percentiles (usec): | 1.00th=[ 4359], 5.00th=[ 5538], 10.00th=[ 6259], 20.00th=[ 7439], | 30.00th=[ 8291], 40.00th=[ 9634], 50.00th=[10945], 60.00th=[12256], | 70.00th=[14484], 80.00th=[19792], 90.00th=[31065], 95.00th=[46924], | 99.00th=[72877], 99.50th=[81265], 99.90th=[88605], 99.95th=[88605], | 99.99th=[88605] bw ( KiB/s): min= 63, max= 736, per=23.64%, avg=343.47, stdev=192.93, samples=32 iops : min= 3, max= 46, avg=21.12, stdev=12.25, samples=32 write: IOPS=24, BW=397KiB/s (407kB/s)(6544KiB/16479msec) clat (usec): min=5953, max=92260, avg=26101.89, stdev=15718.93 lat (usec): min=5955, max=92262, avg=26103.87, stdev=15718.92 clat percentiles (usec): | 1.00th=[ 7963], 5.00th=[10814], 10.00th=[12387], 20.00th=[14091], | 30.00th=[16188], 40.00th=[17957], 50.00th=[21365], 60.00th=[24249], | 70.00th=[28181], 80.00th=[36439], 90.00th=[49021], 95.00th=[59507], | 99.00th=[78119], 99.50th=[82314], 99.90th=[91751], 99.95th=[91751], | 99.99th=[91751] bw ( KiB/s): min= 126, max= 800, per=25.08%, avg=383.41, stdev=170.19, samples=32 iops : min= 7, max= 50, avg=23.62, stdev=10.81, samples=32 lat (msec) : 4=0.13%, 10=21.74%, 20=40.62%, 50=30.60%, 100=6.90% cpu : usr=0.04%, sys=5.24%, ctx=911, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22326: Thu May 1 19:41:22 2025 read: IOPS=24, BW=386KiB/s (395kB/s)(6256KiB/16224msec) clat (msec): min=3, max=112, avg=15.85, stdev=14.10 lat (msec): min=3, max=112, avg=15.85, stdev=14.10 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 6], 10.00th=[ 7], 20.00th=[ 8], | 30.00th=[ 9], 40.00th=[ 10], 50.00th=[ 11], 60.00th=[ 13], | 70.00th=[ 16], 80.00th=[ 22], 90.00th=[ 33], 95.00th=[ 52], | 99.00th=[ 72], 99.50th=[ 78], 99.90th=[ 113], 99.95th=[ 113], | 99.99th=[ 113] bw ( KiB/s): min= 127, max= 1024, per=26.44%, avg=384.13, stdev=223.02, samples=32 iops : min= 7, max= 64, avg=23.56, stdev=14.02, samples=32 write: IOPS=23, BW=372KiB/s (381kB/s)(6032KiB/16224msec) clat (msec): min=5, max=102, avg=26.17, stdev=15.43 lat (msec): min=5, max=102, avg=26.18, stdev=15.44 clat percentiles (msec): | 1.00th=[ 8], 5.00th=[ 11], 10.00th=[ 12], 20.00th=[ 15], | 30.00th=[ 16], 40.00th=[ 18], 50.00th=[ 21], 60.00th=[ 25], | 70.00th=[ 31], 80.00th=[ 39], 90.00th=[ 49], 95.00th=[ 56], | 99.00th=[ 78], 99.50th=[ 84], 99.90th=[ 103], 99.95th=[ 103], | 99.99th=[ 103] bw ( KiB/s): min= 127, max= 800, per=23.95%, avg=366.22, stdev=169.23, samples=32 iops : min= 7, max= 50, avg=22.44, stdev=10.68, samples=32 lat (msec) : 4=0.26%, 10=25.13%, 20=38.02%, 50=29.30%, 100=7.03% lat (msec) : 250=0.26% cpu : usr=0.12%, sys=5.39%, ctx=916, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22327: Thu May 1 19:41:22 2025 read: IOPS=22, BW=366KiB/s (375kB/s)(6032KiB/16472msec) clat (usec): min=3682, max=79788, avg=17139.95, stdev=14462.12 lat (usec): min=3684, max=79789, avg=17141.13, stdev=14462.09 clat percentiles (usec): | 1.00th=[ 4424], 5.00th=[ 5342], 10.00th=[ 5932], 20.00th=[ 7570], | 30.00th=[ 8586], 40.00th=[10028], 50.00th=[11731], 60.00th=[13566], | 70.00th=[17171], 80.00th=[24249], 90.00th=[39060], 95.00th=[47449], | 99.00th=[71828], 99.50th=[79168], 99.90th=[80217], 99.95th=[80217], | 99.99th=[80217] bw ( KiB/s): min= 96, max= 798, per=24.46%, avg=355.37, stdev=175.56, samples=32 iops : min= 6, max= 49, avg=21.81, stdev=10.90, samples=32 write: IOPS=23, BW=380KiB/s (389kB/s)(6256KiB/16472msec) clat (usec): min=6772, max=98048, avg=25083.02, stdev=14950.60 lat (usec): min=6774, max=98050, avg=25084.96, stdev=14950.58 clat percentiles (usec): | 1.00th=[ 7308], 5.00th=[10421], 10.00th=[11994], 20.00th=[14091], | 30.00th=[15533], 40.00th=[17171], 50.00th=[19792], 60.00th=[23200], | 70.00th=[27657], 80.00th=[35390], 90.00th=[45876], 95.00th=[55837], | 99.00th=[77071], 99.50th=[88605], 99.90th=[98042], 99.95th=[98042], | 99.99th=[98042] bw ( KiB/s): min= 128, max= 736, per=24.22%, avg=370.25, stdev=184.10, samples=32 iops : min= 8, max= 46, avg=22.75, stdev=11.48, samples=32 lat (msec) : 4=0.13%, 10=21.09%, 20=41.41%, 50=31.51%, 100=5.86% cpu : usr=0.07%, sys=5.27%, ctx=878, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1453KiB/s (1488kB/s), 349KiB/s-386KiB/s (357kB/s-395kB/s), io=23.4MiB (24.5MB), run=16224-16479msec WRITE: bw=1529KiB/s (1566kB/s), 372KiB/s-397KiB/s (381kB/s-407kB/s), io=24.6MiB (25.8MB), run=16224-16479msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=22347: Thu May 1 19:41:25 2025 read: IOPS=7, BW=7851KiB/s (8039kB/s)(7168KiB/913msec) clat (usec): min=27595, max=52880, avg=37388.00, stdev=8803.13 lat (usec): min=27595, max=52881, avg=37389.31, stdev=8803.16 clat percentiles (usec): | 1.00th=[27657], 5.00th=[27657], 10.00th=[27657], 20.00th=[28443], | 30.00th=[32637], 40.00th=[32637], 50.00th=[38011], 60.00th=[41157], | 70.00th=[41157], 80.00th=[41157], 90.00th=[52691], 95.00th=[52691], | 99.00th=[52691], 99.50th=[52691], 99.90th=[52691], 99.95th=[52691], | 99.99th=[52691] bw ( KiB/s): min= 9941, max= 9941, per=63.98%, avg=9941.00, stdev= 0.00, samples=1 iops : min= 9, max= 9, avg= 9.00, stdev= 0.00, samples=1 write: IOPS=5, BW=5608KiB/s (5742kB/s)(5120KiB/913msec) clat (msec): min=84, max=195, avg=127.12, stdev=44.80 lat (msec): min=84, max=196, avg=127.20, stdev=44.83 clat percentiles (msec): | 1.00th=[ 85], 5.00th=[ 85], 10.00th=[ 85], 20.00th=[ 85], | 30.00th=[ 99], 40.00th=[ 99], 50.00th=[ 111], 60.00th=[ 111], | 70.00th=[ 146], 80.00th=[ 146], 90.00th=[ 197], 95.00th=[ 197], | 99.00th=[ 197], 99.50th=[ 197], 99.90th=[ 197], 99.95th=[ 197], | 99.99th=[ 197] bw ( KiB/s): min= 3976, max= 3976, per=18.28%, avg=3976.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 lat (msec) : 50=50.00%, 100=25.00%, 250=25.00% cpu : usr=0.00%, sys=11.29%, ctx=61, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22349: Thu May 1 19:41:25 2025 read: IOPS=1, BW=1554KiB/s (1591kB/s)(2048KiB/1318msec) clat (usec): min=49684, max=98743, avg=74214.06, stdev=34689.62 lat (usec): min=49686, max=98745, avg=74215.96, stdev=34690.22 clat percentiles (usec): | 1.00th=[49546], 5.00th=[49546], 10.00th=[49546], 20.00th=[49546], | 30.00th=[49546], 40.00th=[49546], 50.00th=[49546], 60.00th=[99091], | 70.00th=[99091], 80.00th=[99091], 90.00th=[99091], 95.00th=[99091], | 99.00th=[99091], 99.50th=[99091], 99.90th=[99091], 99.95th=[99091], | 99.99th=[99091] bw ( KiB/s): min= 2027, max= 2048, per=13.11%, avg=2037.50, stdev=14.85, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 write: IOPS=7, BW=7769KiB/s (7956kB/s)(10.0MiB/1318msec) clat (msec): min=35, max=224, avg=112.74, stdev=70.03 lat (msec): min=35, max=224, avg=112.82, stdev=70.02 clat percentiles (msec): | 1.00th=[ 36], 5.00th=[ 36], 10.00th=[ 36], 20.00th=[ 41], | 30.00th=[ 50], 40.00th=[ 79], 50.00th=[ 99], 60.00th=[ 104], | 70.00th=[ 174], 80.00th=[ 174], 90.00th=[ 218], 95.00th=[ 226], | 99.00th=[ 226], 99.50th=[ 226], 99.90th=[ 226], 99.95th=[ 226], | 99.99th=[ 226] bw ( KiB/s): min= 4055, max= 6144, per=23.44%, avg=5099.50, stdev=1477.15, samples=2 iops : min= 3, max= 6, avg= 4.50, stdev= 2.12, samples=2 lat (msec) : 50=33.33%, 100=25.00%, 250=41.67% cpu : usr=0.08%, sys=12.15%, ctx=129, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22352: Thu May 1 19:41:25 2025 read: IOPS=4, BW=4741KiB/s (4855kB/s)(6144KiB/1296msec) clat (msec): min=24, max=225, avg=79.04, stdev=74.45 lat (msec): min=24, max=225, avg=79.04, stdev=74.45 clat percentiles (msec): | 1.00th=[ 25], 5.00th=[ 25], 10.00th=[ 25], 20.00th=[ 39], | 30.00th=[ 39], 40.00th=[ 48], 50.00th=[ 48], 60.00th=[ 54], | 70.00th=[ 86], 80.00th=[ 86], 90.00th=[ 226], 95.00th=[ 226], | 99.00th=[ 226], 99.50th=[ 226], 99.90th=[ 226], 99.95th=[ 226], | 99.99th=[ 226] bw ( KiB/s): min= 8192, max= 8192, per=52.72%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=4, BW=4741KiB/s (4855kB/s)(6144KiB/1296msec) clat (msec): min=74, max=211, avg=117.16, stdev=48.72 lat (msec): min=74, max=213, avg=117.51, stdev=49.34 clat percentiles (msec): | 1.00th=[ 74], 5.00th=[ 74], 10.00th=[ 74], 20.00th=[ 93], | 30.00th=[ 93], 40.00th=[ 99], 50.00th=[ 99], 60.00th=[ 105], | 70.00th=[ 122], 80.00th=[ 122], 90.00th=[ 211], 95.00th=[ 211], | 99.00th=[ 211], 99.50th=[ 211], 99.90th=[ 211], 99.95th=[ 211], | 99.99th=[ 211] bw ( KiB/s): min= 4096, max= 4096, per=18.83%, avg=4096.00, stdev= 0.00, samples=2 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=2 lat (msec) : 50=25.00%, 100=41.67%, 250=33.33% cpu : usr=0.00%, sys=11.81%, ctx=56, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22354: Thu May 1 19:41:25 2025 read: IOPS=3, BW=3972KiB/s (4067kB/s)(5120KiB/1289msec) clat (msec): min=22, max=181, avg=93.94, stdev=58.78 lat (msec): min=22, max=181, avg=93.94, stdev=58.78 clat percentiles (msec): | 1.00th=[ 23], 5.00th=[ 23], 10.00th=[ 23], 20.00th=[ 23], | 30.00th=[ 66], 40.00th=[ 66], 50.00th=[ 94], 60.00th=[ 94], | 70.00th=[ 109], 80.00th=[ 109], 90.00th=[ 182], 95.00th=[ 182], | 99.00th=[ 182], 99.50th=[ 182], 99.90th=[ 182], 99.95th=[ 182], | 99.99th=[ 182] bw ( KiB/s): min= 8192, max= 8192, per=52.72%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=5, BW=5561KiB/s (5694kB/s)(7168KiB/1289msec) clat (msec): min=30, max=220, avg=108.69, stdev=67.89 lat (msec): min=30, max=221, avg=108.76, stdev=67.89 clat percentiles (msec): | 1.00th=[ 31], 5.00th=[ 31], 10.00th=[ 31], 20.00th=[ 63], | 30.00th=[ 68], 40.00th=[ 68], 50.00th=[ 97], 60.00th=[ 104], | 70.00th=[ 104], 80.00th=[ 180], 90.00th=[ 222], 95.00th=[ 222], | 99.00th=[ 222], 99.50th=[ 222], 99.90th=[ 222], 99.95th=[ 222], | 99.99th=[ 222] bw ( KiB/s): min= 2048, max= 8192, per=23.54%, avg=5120.00, stdev=4344.46, samples=2 iops : min= 2, max= 8, avg= 5.00, stdev= 4.24, samples=2 lat (msec) : 50=16.67%, 100=41.67%, 250=41.67% cpu : usr=0.16%, sys=7.06%, ctx=53, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=15.2MiB/s (15.9MB/s), 1554KiB/s-7851KiB/s (1591kB/s-8039kB/s), io=20.0MiB (20.0MB), run=913-1318msec WRITE: bw=21.2MiB/s (22.3MB/s), 4741KiB/s-7769KiB/s (4855kB/s-7956kB/s), io=28.0MiB (29.4MB), run=913-1318msec rand-rw: (groupid=0, jobs=1): err= 0: pid=22348: Thu May 1 19:41:25 2025 read: IOPS=4, BW=5023KiB/s (5144kB/s)(7168KiB/1427msec) clat (usec): min=594, max=544766, avg=133637.55, stdev=188225.00 lat (usec): min=596, max=544768, avg=133638.95, stdev=188224.98 clat percentiles (usec): | 1.00th=[ 594], 5.00th=[ 594], 10.00th=[ 594], 20.00th=[ 33817], | 30.00th=[ 46924], 40.00th=[ 46924], 50.00th=[ 71828], 60.00th=[ 73925], | 70.00th=[ 73925], 80.00th=[164627], 90.00th=[541066], 95.00th=[541066], | 99.00th=[541066], 99.50th=[541066], 99.90th=[541066], 99.95th=[541066], | 99.99th=[541066] bw ( KiB/s): min= 3730, max= 4087, per=27.24%, avg=3908.50, stdev=252.44, samples=2 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=2 write: IOPS=3, BW=3588KiB/s (3674kB/s)(5120KiB/1427msec) clat (usec): min=46740, max=79667, avg=66867.61, stdev=12876.92 lat (usec): min=46765, max=79699, avg=66978.02, stdev=12888.94 clat percentiles (usec): | 1.00th=[46924], 5.00th=[46924], 10.00th=[46924], 20.00th=[46924], | 30.00th=[64226], 40.00th=[64226], 50.00th=[67634], 60.00th=[67634], | 70.00th=[76022], 80.00th=[76022], 90.00th=[79168], 95.00th=[79168], | 99.00th=[79168], 99.50th=[79168], 99.90th=[79168], 99.95th=[79168], | 99.99th=[79168] bw ( KiB/s): min= 1865, max= 1865, per=9.28%, avg=1865.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (usec) : 750=8.33% lat (msec) : 50=25.00%, 100=50.00%, 250=8.33%, 750=8.33% cpu : usr=0.00%, sys=18.23%, ctx=582, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22350: Thu May 1 19:41:25 2025 read: IOPS=1, BW=1848KiB/s (1893kB/s)(2048KiB/1108msec) clat (usec): min=30987, max=60070, avg=45528.72, stdev=20565.06 lat (usec): min=30988, max=60072, avg=45530.49, stdev=20564.96 clat percentiles (usec): | 1.00th=[31065], 5.00th=[31065], 10.00th=[31065], 20.00th=[31065], | 30.00th=[31065], 40.00th=[31065], 50.00th=[31065], 60.00th=[60031], | 70.00th=[60031], 80.00th=[60031], 90.00th=[60031], 95.00th=[60031], | 99.00th=[60031], 99.50th=[60031], 99.90th=[60031], 99.95th=[60031], | 99.99th=[60031] bw ( KiB/s): min= 2048, max= 2048, per=14.27%, avg=2048.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 write: IOPS=9, BW=9242KiB/s (9464kB/s)(10.0MiB/1108msec) clat (msec): min=39, max=370, avg=99.63, stdev=102.32 lat (msec): min=39, max=370, avg=99.84, stdev=102.28 clat percentiles (msec): | 1.00th=[ 40], 5.00th=[ 40], 10.00th=[ 40], 20.00th=[ 42], | 30.00th=[ 47], 40.00th=[ 53], 50.00th=[ 59], 60.00th=[ 63], | 70.00th=[ 79], 80.00th=[ 79], 90.00th=[ 169], 95.00th=[ 372], | 99.00th=[ 372], 99.50th=[ 372], 99.90th=[ 372], 99.95th=[ 372], | 99.99th=[ 372] bw ( KiB/s): min= 6144, max=10240, per=40.77%, avg=8192.00, stdev=2896.31, samples=2 iops : min= 6, max= 10, avg= 8.00, stdev= 2.83, samples=2 lat (msec) : 50=33.33%, 100=50.00%, 250=8.33%, 500=8.33% cpu : usr=0.00%, sys=21.05%, ctx=238, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22351: Thu May 1 19:41:25 2025 read: IOPS=6, BW=6508KiB/s (6665kB/s)(6144KiB/944msec) clat (usec): min=1683, max=173714, avg=68014.60, stdev=59736.21 lat (usec): min=1685, max=173715, avg=68015.94, stdev=59736.09 clat percentiles (usec): | 1.00th=[ 1680], 5.00th=[ 1680], 10.00th=[ 1680], 20.00th=[ 29492], | 30.00th=[ 29492], 40.00th=[ 52691], 50.00th=[ 52691], 60.00th=[ 60031], | 70.00th=[ 90702], 80.00th=[ 90702], 90.00th=[173016], 95.00th=[173016], | 99.00th=[173016], 99.50th=[173016], 99.90th=[173016], 99.95th=[173016], | 99.99th=[173016] bw ( KiB/s): min= 4039, max= 4039, per=28.14%, avg=4039.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 write: IOPS=6, BW=6508KiB/s (6665kB/s)(6144KiB/944msec) clat (msec): min=45, max=130, avg=73.02, stdev=31.61 lat (msec): min=50, max=130, avg=73.85, stdev=30.87 clat percentiles (msec): | 1.00th=[ 46], 5.00th=[ 46], 10.00th=[ 46], 20.00th=[ 53], | 30.00th=[ 53], 40.00th=[ 57], 50.00th=[ 57], 60.00th=[ 65], | 70.00th=[ 88], 80.00th=[ 88], 90.00th=[ 131], 95.00th=[ 131], | 99.00th=[ 131], 99.50th=[ 131], 99.90th=[ 131], 99.95th=[ 131], | 99.99th=[ 131] bw ( KiB/s): min= 6059, max= 6059, per=30.16%, avg=6059.00, stdev= 0.00, samples=1 iops : min= 5, max= 5, avg= 5.00, stdev= 0.00, samples=1 lat (msec) : 2=8.33%, 50=16.67%, 100=58.33%, 250=16.67% cpu : usr=0.00%, sys=24.92%, ctx=225, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22353: Thu May 1 19:41:25 2025 read: IOPS=4, BW=4853KiB/s (4970kB/s)(5120KiB/1055msec) clat (usec): min=541, max=69272, avg=15675.42, stdev=30071.68 lat (usec): min=542, max=69274, avg=15676.85, stdev=30071.66 clat percentiles (usec): | 1.00th=[ 545], 5.00th=[ 545], 10.00th=[ 545], 20.00th=[ 545], | 30.00th=[ 562], 40.00th=[ 562], 50.00th=[ 1303], 60.00th=[ 1303], | 70.00th=[ 6718], 80.00th=[ 6718], 90.00th=[69731], 95.00th=[69731], | 99.00th=[69731], 99.50th=[69731], 99.90th=[69731], 99.95th=[69731], | 99.99th=[69731] bw ( KiB/s): min= 8175, max= 8175, per=56.96%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 7, max= 7, avg= 7.00, stdev= 0.00, samples=1 write: IOPS=6, BW=6794KiB/s (6957kB/s)(7168KiB/1055msec) clat (msec): min=58, max=241, avg=117.28, stdev=61.08 lat (msec): min=58, max=241, avg=117.35, stdev=61.06 clat percentiles (msec): | 1.00th=[ 59], 5.00th=[ 59], 10.00th=[ 59], 20.00th=[ 62], | 30.00th=[ 102], 40.00th=[ 102], 50.00th=[ 114], 60.00th=[ 120], | 70.00th=[ 120], 80.00th=[ 127], 90.00th=[ 243], 95.00th=[ 243], | 99.00th=[ 243], 99.50th=[ 243], 99.90th=[ 243], 99.95th=[ 243], | 99.99th=[ 243] bw ( KiB/s): min= 4063, max= 8175, per=30.45%, avg=6119.00, stdev=2907.62, samples=2 iops : min= 3, max= 7, avg= 5.00, stdev= 2.83, samples=2 lat (usec) : 750=16.67% lat (msec) : 2=8.33%, 10=8.33%, 100=25.00%, 250=41.67% cpu : usr=0.00%, sys=17.36%, ctx=125, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=14.0MiB/s (14.7MB/s), 1848KiB/s-6508KiB/s (1893kB/s-6665kB/s), io=20.0MiB (20.0MB), run=944-1427msec WRITE: bw=19.6MiB/s (20.6MB/s), 3588KiB/s-9242KiB/s (3674kB/s-9464kB/s), io=28.0MiB (29.4MB), run=944-1427msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=22361: Thu May 1 19:41:28 2025 read: IOPS=4, BW=16.2MiB/s (17.0MB/s)(12.0MiB/739msec) clat (msec): min=109, max=384, avg=235.61, stdev=138.76 lat (msec): min=109, max=384, avg=235.61, stdev=138.76 clat percentiles (msec): | 1.00th=[ 110], 5.00th=[ 110], 10.00th=[ 110], 20.00th=[ 110], | 30.00th=[ 110], 40.00th=[ 213], 50.00th=[ 213], 60.00th=[ 213], | 70.00th=[ 384], 80.00th=[ 384], 90.00th=[ 384], 95.00th=[ 384], | 99.00th=[ 384], 99.50th=[ 384], 99.90th=[ 384], 99.95th=[ 384], | 99.99th=[ 384] bw ( KiB/s): min= 8192, max= 8192, per=72.66%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.00%, sys=25.61%, ctx=623, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22363: Thu May 1 19:41:28 2025 read: IOPS=0, BW=2819KiB/s (2887kB/s)(4096KiB/1453msec) clat (nsec): min=63891k, max=63891k, avg=63891140.00, stdev= 0.00 lat (nsec): min=63894k, max=63894k, avg=63893742.00, stdev= 0.00 clat percentiles (usec): | 1.00th=[63701], 5.00th=[63701], 10.00th=[63701], 20.00th=[63701], | 30.00th=[63701], 40.00th=[63701], 50.00th=[63701], 60.00th=[63701], | 70.00th=[63701], 80.00th=[63701], 90.00th=[63701], 95.00th=[63701], | 99.00th=[63701], 99.50th=[63701], 99.90th=[63701], 99.95th=[63701], | 99.99th=[63701] bw ( KiB/s): min= 8175, max= 8175, per=72.51%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=1, BW=5638KiB/s (5773kB/s)(8192KiB/1453msec) clat (msec): min=585, max=735, avg=660.44, stdev=106.64 lat (msec): min=585, max=736, avg=660.72, stdev=106.72 clat percentiles (msec): | 1.00th=[ 584], 5.00th=[ 584], 10.00th=[ 584], 20.00th=[ 584], | 30.00th=[ 584], 40.00th=[ 584], 50.00th=[ 584], 60.00th=[ 735], | 70.00th=[ 735], 80.00th=[ 735], 90.00th=[ 735], 95.00th=[ 735], | 99.00th=[ 735], 99.50th=[ 735], 99.90th=[ 735], 99.95th=[ 735], | 99.99th=[ 735] bw ( KiB/s): min= 6177, max= 6177, per=27.39%, avg=6177.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 750=66.67% cpu : usr=0.00%, sys=15.43%, ctx=58, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22364: Thu May 1 19:41:28 2025 write: IOPS=2, BW=10.4MiB/s (10.9MB/s)(12.0MiB/1153msec) clat (msec): min=336, max=420, avg=378.41, stdev=42.11 lat (msec): min=337, max=421, avg=378.75, stdev=42.16 clat percentiles (msec): | 1.00th=[ 338], 5.00th=[ 338], 10.00th=[ 338], 20.00th=[ 338], | 30.00th=[ 338], 40.00th=[ 380], 50.00th=[ 380], 60.00th=[ 380], | 70.00th=[ 422], 80.00th=[ 422], 90.00th=[ 422], 95.00th=[ 422], | 99.00th=[ 422], 99.50th=[ 422], 99.90th=[ 422], 99.95th=[ 422], | 99.99th=[ 422] bw ( KiB/s): min= 8175, max= 8175, per=36.25%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 500=100.00% cpu : usr=0.00%, sys=23.18%, ctx=79, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22366: Thu May 1 19:41:28 2025 write: IOPS=3, BW=14.1MiB/s (14.8MB/s)(12.0MiB/850msec) clat (msec): min=210, max=347, avg=275.42, stdev=68.83 lat (msec): min=210, max=347, avg=275.75, stdev=68.74 clat percentiles (msec): | 1.00th=[ 211], 5.00th=[ 211], 10.00th=[ 211], 20.00th=[ 211], | 30.00th=[ 211], 40.00th=[ 271], 50.00th=[ 271], 60.00th=[ 271], | 70.00th=[ 347], 80.00th=[ 347], 90.00th=[ 347], 95.00th=[ 347], | 99.00th=[ 347], 99.50th=[ 347], 99.90th=[ 347], 99.95th=[ 347], | 99.99th=[ 347] bw ( KiB/s): min= 8192, max= 8192, per=36.33%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=66.67% cpu : usr=0.00%, sys=32.86%, ctx=125, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=11.0MiB/s (11.5MB/s), 2819KiB/s-16.2MiB/s (2887kB/s-17.0MB/s), io=16.0MiB (16.8MB), run=739-1453msec WRITE: bw=22.0MiB/s (23.1MB/s), 5638KiB/s-14.1MiB/s (5773kB/s-14.8MB/s), io=32.0MiB (33.6MB), run=850-1453msec rand-rw: (groupid=0, jobs=1): err= 0: pid=22362: Thu May 1 19:41:28 2025 read: IOPS=2, BW=8802KiB/s (9014kB/s)(12.0MiB/1396msec) clat (msec): min=163, max=595, avg=407.32, stdev=220.86 lat (msec): min=163, max=595, avg=407.32, stdev=220.86 clat percentiles (msec): | 1.00th=[ 165], 5.00th=[ 165], 10.00th=[ 165], 20.00th=[ 165], | 30.00th=[ 165], 40.00th=[ 464], 50.00th=[ 464], 60.00th=[ 464], | 70.00th=[ 592], 80.00th=[ 592], 90.00th=[ 592], 95.00th=[ 592], | 99.00th=[ 592], 99.50th=[ 592], 99.90th=[ 592], 99.95th=[ 592], | 99.99th=[ 592] bw ( KiB/s): min= 8175, max= 8175, per=83.33%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=33.33%, 750=33.33% cpu : usr=0.36%, sys=5.02%, ctx=59, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22365: Thu May 1 19:41:28 2025 read: IOPS=0, BW=2453KiB/s (2512kB/s)(4096KiB/1670msec) clat (nsec): min=499904k, max=499904k, avg=499903942.00, stdev= 0.00 lat (nsec): min=499905k, max=499905k, avg=499905147.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 502], 5.00th=[ 502], 10.00th=[ 502], 20.00th=[ 502], | 30.00th=[ 502], 40.00th=[ 502], 50.00th=[ 502], 60.00th=[ 502], | 70.00th=[ 502], 80.00th=[ 502], 90.00th=[ 502], 95.00th=[ 502], | 99.00th=[ 502], 99.50th=[ 502], 99.90th=[ 502], 99.95th=[ 502], | 99.99th=[ 502] bw ( KiB/s): min= 8192, max= 8192, per=83.51%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=1, BW=4905KiB/s (5023kB/s)(8192KiB/1670msec) clat (msec): min=460, max=534, avg=497.49, stdev=51.98 lat (msec): min=460, max=534, avg=497.87, stdev=52.27 clat percentiles (msec): | 1.00th=[ 460], 5.00th=[ 460], 10.00th=[ 460], 20.00th=[ 460], | 30.00th=[ 460], 40.00th=[ 460], 50.00th=[ 460], 60.00th=[ 535], | 70.00th=[ 535], 80.00th=[ 535], 90.00th=[ 535], 95.00th=[ 535], | 99.00th=[ 535], 99.50th=[ 535], 99.90th=[ 535], 99.95th=[ 535], | 99.99th=[ 535] bw ( KiB/s): min= 6131, max= 6131, per=31.32%, avg=6131.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 500=66.67%, 750=33.33% cpu : usr=0.12%, sys=8.57%, ctx=102, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22367: Thu May 1 19:41:28 2025 write: IOPS=2, BW=9344KiB/s (9569kB/s)(12.0MiB/1315msec) clat (msec): min=217, max=528, avg=409.78, stdev=167.91 lat (msec): min=217, max=529, avg=410.11, stdev=168.08 clat percentiles (msec): | 1.00th=[ 218], 5.00th=[ 218], 10.00th=[ 218], 20.00th=[ 218], | 30.00th=[ 218], 40.00th=[ 485], 50.00th=[ 485], 60.00th=[ 485], | 70.00th=[ 531], 80.00th=[ 531], 90.00th=[ 531], 95.00th=[ 531], | 99.00th=[ 531], 99.50th=[ 531], 99.90th=[ 531], 99.95th=[ 531], | 99.99th=[ 531] bw ( KiB/s): min= 8192, max= 8192, per=41.85%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=33.33%, 750=33.33% cpu : usr=0.15%, sys=10.11%, ctx=69, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22368: Thu May 1 19:41:28 2025 write: IOPS=1, BW=7341KiB/s (7517kB/s)(12.0MiB/1674msec) clat (msec): min=129, max=1042, avg=536.36, stdev=464.71 lat (msec): min=129, max=1042, avg=536.64, stdev=464.72 clat percentiles (msec): | 1.00th=[ 130], 5.00th=[ 130], 10.00th=[ 130], 20.00th=[ 130], | 30.00th=[ 130], 40.00th=[ 439], 50.00th=[ 439], 60.00th=[ 439], | 70.00th=[ 1045], 80.00th=[ 1045], 90.00th=[ 1045], 95.00th=[ 1045], | 99.00th=[ 1045], 99.50th=[ 1045], 99.90th=[ 1045], 99.95th=[ 1045], | 99.99th=[ 1045] bw ( KiB/s): min= 8192, max= 8192, per=41.85%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=33.33% cpu : usr=0.12%, sys=13.81%, ctx=142, majf=0, minf=28 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=9811KiB/s (10.0MB/s), 2453KiB/s-8802KiB/s (2512kB/s-9014kB/s), io=16.0MiB (16.8MB), run=1396-1670msec WRITE: bw=19.1MiB/s (20.0MB/s), 4905KiB/s-9344KiB/s (5023kB/s-9569kB/s), io=32.0MiB (33.6MB), run=1315-1674msec