== sanity test 398b: DIO and buffer IO race ============== 21:49:42 (1743212982) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.25011 s, 40.3 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=27910: Fri Mar 28 21:49:56 2025 read: IOPS=134, BW=540KiB/s (553kB/s)(5992KiB/11101msec) clat (usec): min=58, max=19687, avg=4008.92, stdev=2296.91 lat (usec): min=58, max=19688, avg=4009.49, stdev=2296.92 clat percentiles (usec): | 1.00th=[ 67], 5.00th=[ 1631], 10.00th=[ 1975], 20.00th=[ 2311], | 30.00th=[ 2671], 40.00th=[ 3097], 50.00th=[ 3654], 60.00th=[ 4146], | 70.00th=[ 4752], 80.00th=[ 5407], 90.00th=[ 6456], 95.00th=[ 7767], | 99.00th=[12911], 99.50th=[14222], 99.90th=[18482], 99.95th=[19792], | 99.99th=[19792] bw ( KiB/s): min= 239, max= 710, per=24.78%, avg=536.05, stdev=91.62, samples=22 iops : min= 59, max= 177, avg=133.91, stdev=22.96, samples=22 write: IOPS=141, BW=567KiB/s (581kB/s)(6296KiB/11101msec) clat (usec): min=395, max=179841, avg=3182.48, stdev=5260.30 lat (usec): min=396, max=179842, avg=3183.39, stdev=5260.38 clat percentiles (usec): | 1.00th=[ 478], 5.00th=[ 502], 10.00th=[ 519], 20.00th=[ 562], | 30.00th=[ 685], 40.00th=[ 1762], 50.00th=[ 2442], 60.00th=[ 2966], | 70.00th=[ 3916], 80.00th=[ 5211], 90.00th=[ 7046], 95.00th=[ 8717], | 99.00th=[ 11469], 99.50th=[ 13042], 99.90th=[ 22414], 99.95th=[179307], | 99.99th=[179307] bw ( KiB/s): min= 295, max= 720, per=24.79%, avg=561.14, stdev=85.73, samples=22 iops : min= 73, max= 180, avg=140.18, stdev=21.53, samples=22 lat (usec) : 100=1.95%, 250=0.20%, 500=2.54%, 750=13.67%, 1000=1.82% lat (msec) : 2=6.93%, 4=37.17%, 10=33.37%, 20=2.25%, 50=0.07% lat (msec) : 250=0.03% cpu : usr=0.16%, sys=21.87%, ctx=5534, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=27911: Fri Mar 28 21:49:56 2025 read: IOPS=135, BW=540KiB/s (553kB/s)(5940KiB/10995msec) clat (usec): min=51, max=18439, avg=3956.28, stdev=2178.92 lat (usec): min=51, max=18440, avg=3956.91, stdev=2178.90 clat percentiles (usec): | 1.00th=[ 65], 5.00th=[ 1696], 10.00th=[ 1942], 20.00th=[ 2311], | 30.00th=[ 2638], 40.00th=[ 3032], 50.00th=[ 3589], 60.00th=[ 4113], | 70.00th=[ 4686], 80.00th=[ 5407], 90.00th=[ 6587], 95.00th=[ 7701], | 99.00th=[11600], 99.50th=[13042], 99.90th=[15926], 99.95th=[18482], | 99.99th=[18482] bw ( KiB/s): min= 239, max= 600, per=24.66%, avg=533.48, stdev=78.84, samples=21 iops : min= 59, max= 150, avg=133.29, stdev=19.84, samples=21 write: IOPS=144, BW=577KiB/s (591kB/s)(6348KiB/10995msec) clat (usec): min=420, max=180329, avg=3175.45, stdev=5172.53 lat (usec): min=420, max=180330, avg=3176.31, stdev=5172.45 clat percentiles (usec): | 1.00th=[ 469], 5.00th=[ 498], 10.00th=[ 519], 20.00th=[ 562], | 30.00th=[ 750], 40.00th=[ 1926], 50.00th=[ 2573], 60.00th=[ 3163], | 70.00th=[ 3916], 80.00th=[ 5145], 90.00th=[ 6783], 95.00th=[ 8291], | 99.00th=[ 11338], 99.50th=[ 13829], 99.90th=[ 15270], 99.95th=[179307], | 99.99th=[179307] bw ( KiB/s): min= 287, max= 680, per=24.99%, avg=565.86, stdev=83.42, samples=21 iops : min= 71, max= 170, avg=141.38, stdev=20.97, samples=21 lat (usec) : 100=1.86%, 250=0.10%, 500=2.96%, 750=12.66%, 1000=2.15% lat (msec) : 2=7.16%, 4=37.60%, 10=33.53%, 20=1.95%, 250=0.03% cpu : usr=0.16%, sys=21.97%, ctx=5369, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=27912: Fri Mar 28 21:49:56 2025 read: IOPS=136, BW=545KiB/s (558kB/s)(6028KiB/11055msec) clat (usec): min=56, max=179151, avg=4245.95, stdev=5057.66 lat (usec): min=56, max=179152, avg=4246.55, stdev=5057.65 clat percentiles (usec): | 1.00th=[ 66], 5.00th=[ 1762], 10.00th=[ 2008], 20.00th=[ 2409], | 30.00th=[ 2737], 40.00th=[ 3163], 50.00th=[ 3654], 60.00th=[ 4228], | 70.00th=[ 4817], 80.00th=[ 5669], 90.00th=[ 6652], 95.00th=[ 8160], | 99.00th=[ 13042], 99.50th=[ 14353], 99.90th=[ 16712], 99.95th=[179307], | 99.99th=[179307] bw ( KiB/s): min= 255, max= 646, per=25.14%, avg=543.68, stdev=83.22, samples=22 iops : min= 63, max= 161, avg=135.82, stdev=20.89, samples=22 write: IOPS=141, BW=566KiB/s (580kB/s)(6260KiB/11055msec) clat (usec): min=445, max=17346, avg=2920.79, stdev=2585.60 lat (usec): min=446, max=17347, avg=2921.91, stdev=2585.72 clat percentiles (usec): | 1.00th=[ 474], 5.00th=[ 498], 10.00th=[ 519], 20.00th=[ 562], | 30.00th=[ 685], 40.00th=[ 1418], 50.00th=[ 2442], 60.00th=[ 2966], | 70.00th=[ 3785], 80.00th=[ 4883], 90.00th=[ 6521], 95.00th=[ 7963], | 99.00th=[11207], 99.50th=[12256], 99.90th=[16909], 99.95th=[17433], | 99.99th=[17433] bw ( KiB/s): min= 319, max= 744, per=24.91%, avg=564.05, stdev=105.06, samples=22 iops : min= 79, max= 186, avg=140.91, stdev=26.27, samples=22 lat (usec) : 100=1.43%, 250=0.07%, 500=2.70%, 750=13.77%, 1000=2.02% lat (msec) : 2=6.58%, 4=37.63%, 10=33.56%, 20=2.21%, 250=0.03% cpu : usr=0.22%, sys=22.07%, ctx=5517, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=27913: Fri Mar 28 21:49:56 2025 read: IOPS=136, BW=547KiB/s (560kB/s)(6052KiB/11072msec) clat (usec): min=51, max=184150, avg=4083.92, stdev=5106.14 lat (usec): min=51, max=184151, avg=4085.13, stdev=5106.08 clat percentiles (usec): | 1.00th=[ 65], 5.00th=[ 1696], 10.00th=[ 2040], 20.00th=[ 2376], | 30.00th=[ 2671], 40.00th=[ 3032], 50.00th=[ 3556], 60.00th=[ 4113], | 70.00th=[ 4752], 80.00th=[ 5407], 90.00th=[ 6456], 95.00th=[ 7635], | 99.00th=[ 11338], 99.50th=[ 14484], 99.90th=[ 17695], 99.95th=[183501], | 99.99th=[183501] bw ( KiB/s): min= 231, max= 672, per=25.19%, avg=544.77, stdev=91.93, samples=22 iops : min= 57, max= 168, avg=136.09, stdev=23.07, samples=22 write: IOPS=140, BW=563KiB/s (577kB/s)(6236KiB/11072msec) clat (usec): min=412, max=18646, avg=3083.17, stdev=2713.43 lat (usec): min=413, max=18647, avg=3083.91, stdev=2713.39 clat percentiles (usec): | 1.00th=[ 474], 5.00th=[ 498], 10.00th=[ 519], 20.00th=[ 562], | 30.00th=[ 709], 40.00th=[ 1958], 50.00th=[ 2507], 60.00th=[ 3130], | 70.00th=[ 3949], 80.00th=[ 5145], 90.00th=[ 6915], 95.00th=[ 8225], | 99.00th=[11863], 99.50th=[13698], 99.90th=[16450], 99.95th=[18744], | 99.99th=[18744] bw ( KiB/s): min= 295, max= 696, per=24.83%, avg=562.23, stdev=88.00, samples=22 iops : min= 73, max= 174, avg=140.45, stdev=22.03, samples=22 lat (usec) : 100=1.86%, 250=0.03%, 500=2.73%, 750=13.12%, 1000=1.60% lat (msec) : 2=5.96%, 4=38.87%, 10=33.79%, 20=2.02%, 250=0.03% cpu : usr=0.18%, sys=22.03%, ctx=5634, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2163KiB/s (2215kB/s), 540KiB/s-547KiB/s (553kB/s-560kB/s), io=23.4MiB (24.6MB), run=10995-11101msec WRITE: bw=2265KiB/s (2319kB/s), 563KiB/s-577KiB/s (577kB/s-591kB/s), io=24.6MiB (25.7MB), run=10995-11101msec rand-rw: (groupid=0, jobs=1): err= 0: pid=27915: Fri Mar 28 21:50:02 2025 read: IOPS=87, BW=351KiB/s (359kB/s)(5992KiB/17074msec) clat (usec): min=1558, max=187075, avg=4654.21, stdev=5444.89 lat (usec): min=1558, max=187077, avg=4655.10, stdev=5444.89 clat percentiles (usec): | 1.00th=[ 1745], 5.00th=[ 1975], 10.00th=[ 2180], 20.00th=[ 2540], | 30.00th=[ 2999], 40.00th=[ 3490], 50.00th=[ 3851], 60.00th=[ 4113], | 70.00th=[ 4490], 80.00th=[ 5604], 90.00th=[ 8717], 95.00th=[ 10683], | 99.00th=[ 14484], 99.50th=[ 15664], 99.90th=[ 19792], 99.95th=[187696], | 99.99th=[187696] bw ( KiB/s): min= 176, max= 670, per=25.09%, avg=351.03, stdev=89.77, samples=34 iops : min= 44, max= 167, avg=87.68, stdev=22.34, samples=34 write: IOPS=92, BW=369KiB/s (378kB/s)(6296KiB/17074msec) clat (usec): min=2249, max=22848, avg=6373.84, stdev=2854.40 lat (usec): min=2250, max=22848, avg=6374.76, stdev=2854.41 clat percentiles (usec): | 1.00th=[ 2606], 5.00th=[ 3032], 10.00th=[ 3326], 20.00th=[ 3851], | 30.00th=[ 4752], 40.00th=[ 5473], 50.00th=[ 6128], 60.00th=[ 6521], | 70.00th=[ 6980], 80.00th=[ 7898], 90.00th=[ 9765], 95.00th=[11731], | 99.00th=[16909], 99.50th=[20055], 99.90th=[21890], 99.95th=[22938], | 99.99th=[22938] bw ( KiB/s): min= 192, max= 742, per=25.13%, avg=368.21, stdev=88.25, samples=34 iops : min= 48, max= 185, avg=91.97, stdev=22.00, samples=34 lat (msec) : 2=2.93%, 4=35.51%, 10=53.81%, 20=7.49%, 50=0.23% lat (msec) : 250=0.03% cpu : usr=0.22%, sys=12.53%, ctx=3535, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=27916: Fri Mar 28 21:50:02 2025 read: IOPS=88, BW=354KiB/s (362kB/s)(5940KiB/16783msec) clat (usec): min=1467, max=188280, avg=4563.60, stdev=5520.18 lat (usec): min=1468, max=188281, avg=4564.43, stdev=5520.19 clat percentiles (usec): | 1.00th=[ 1696], 5.00th=[ 1893], 10.00th=[ 2057], 20.00th=[ 2474], | 30.00th=[ 2900], 40.00th=[ 3425], 50.00th=[ 3785], 60.00th=[ 4047], | 70.00th=[ 4424], 80.00th=[ 5538], 90.00th=[ 8356], 95.00th=[ 10552], | 99.00th=[ 15270], 99.50th=[ 17433], 99.90th=[ 27132], 99.95th=[187696], | 99.99th=[187696] bw ( KiB/s): min= 120, max= 726, per=25.36%, avg=354.73, stdev=101.70, samples=33 iops : min= 30, max= 181, avg=88.58, stdev=25.33, samples=33 write: IOPS=94, BW=378KiB/s (387kB/s)(6348KiB/16783msec) clat (usec): min=2235, max=22728, avg=6270.47, stdev=2741.42 lat (usec): min=2236, max=22729, avg=6271.46, stdev=2741.44 clat percentiles (usec): | 1.00th=[ 2573], 5.00th=[ 3064], 10.00th=[ 3294], 20.00th=[ 3851], | 30.00th=[ 4686], 40.00th=[ 5407], 50.00th=[ 6063], 60.00th=[ 6521], | 70.00th=[ 6915], 80.00th=[ 7701], 90.00th=[ 9372], 95.00th=[11338], | 99.00th=[16712], 99.50th=[18744], 99.90th=[22152], 99.95th=[22676], | 99.99th=[22676] bw ( KiB/s): min= 200, max= 718, per=25.75%, avg=377.27, stdev=84.26, samples=33 iops : min= 50, max= 179, avg=94.21, stdev=21.01, samples=33 lat (msec) : 2=3.97%, 4=35.45%, 10=53.39%, 20=7.00%, 50=0.16% lat (msec) : 250=0.03% cpu : usr=0.18%, sys=12.64%, ctx=3559, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=27917: Fri Mar 28 21:50:02 2025 read: IOPS=90, BW=363KiB/s (372kB/s)(6028KiB/16591msec) clat (usec): min=1589, max=27790, avg=4429.83, stdev=2701.20 lat (usec): min=1590, max=27790, avg=4430.65, stdev=2701.18 clat percentiles (usec): | 1.00th=[ 1762], 5.00th=[ 1958], 10.00th=[ 2114], 20.00th=[ 2474], | 30.00th=[ 2933], 40.00th=[ 3392], 50.00th=[ 3785], 60.00th=[ 4047], | 70.00th=[ 4424], 80.00th=[ 5473], 90.00th=[ 8029], 95.00th=[10028], | 99.00th=[13698], 99.50th=[17433], 99.90th=[21627], 99.95th=[27919], | 99.99th=[27919] bw ( KiB/s): min= 160, max= 640, per=26.00%, avg=363.79, stdev=78.11, samples=33 iops : min= 40, max= 160, avg=90.88, stdev=19.53, samples=33 write: IOPS=94, BW=377KiB/s (386kB/s)(6260KiB/16591msec) clat (msec): min=2, max=190, avg= 6.30, stdev= 5.42 lat (msec): min=2, max=190, avg= 6.30, stdev= 5.42 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 4], 20.00th=[ 4], | 30.00th=[ 5], 40.00th=[ 6], 50.00th=[ 6], 60.00th=[ 7], | 70.00th=[ 7], 80.00th=[ 8], 90.00th=[ 10], 95.00th=[ 12], | 99.00th=[ 17], 99.50th=[ 20], 99.90th=[ 22], 99.95th=[ 190], | 99.99th=[ 190] bw ( KiB/s): min= 200, max= 784, per=25.73%, avg=376.88, stdev=97.97, samples=33 iops : min= 50, max= 196, avg=94.15, stdev=24.51, samples=33 lat (msec) : 2=3.06%, 4=37.30%, 10=52.64%, 20=6.67%, 50=0.29% lat (msec) : 250=0.03% cpu : usr=0.20%, sys=12.64%, ctx=3532, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=27918: Fri Mar 28 21:50:02 2025 read: IOPS=88, BW=353KiB/s (361kB/s)(6052KiB/17156msec) clat (usec): min=1502, max=21691, avg=4579.59, stdev=2723.63 lat (usec): min=1503, max=21691, avg=4580.43, stdev=2723.59 clat percentiles (usec): | 1.00th=[ 1729], 5.00th=[ 1958], 10.00th=[ 2147], 20.00th=[ 2573], | 30.00th=[ 3064], 40.00th=[ 3556], 50.00th=[ 3884], 60.00th=[ 4146], | 70.00th=[ 4555], 80.00th=[ 5800], 90.00th=[ 8455], 95.00th=[10421], | 99.00th=[14222], 99.50th=[16712], 99.90th=[20055], 99.95th=[21627], | 99.99th=[21627] bw ( KiB/s): min= 160, max= 752, per=25.14%, avg=351.71, stdev=96.96, samples=34 iops : min= 40, max= 188, avg=87.88, stdev=24.26, samples=34 write: IOPS=90, BW=363KiB/s (372kB/s)(6236KiB/17156msec) clat (msec): min=2, max=169, avg= 6.52, stdev= 5.02 lat (msec): min=2, max=169, avg= 6.52, stdev= 5.02 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 4], 20.00th=[ 4], | 30.00th=[ 5], 40.00th=[ 6], 50.00th=[ 7], 60.00th=[ 7], | 70.00th=[ 8], 80.00th=[ 8], 90.00th=[ 10], 95.00th=[ 12], | 99.00th=[ 17], 99.50th=[ 21], 99.90th=[ 23], 99.95th=[ 169], | 99.99th=[ 169] bw ( KiB/s): min= 192, max= 712, per=24.81%, avg=363.47, stdev=88.08, samples=34 iops : min= 48, max= 178, avg=90.82, stdev=22.04, samples=34 lat (msec) : 2=3.03%, 4=33.85%, 10=55.86%, 20=7.00%, 50=0.23% lat (msec) : 250=0.03% cpu : usr=0.20%, sys=12.55%, ctx=3589, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1400KiB/s (1433kB/s), 351KiB/s-363KiB/s (359kB/s-372kB/s), io=23.4MiB (24.6MB), run=16591-17156msec WRITE: bw=1465KiB/s (1501kB/s), 363KiB/s-378KiB/s (372kB/s-387kB/s), io=24.6MiB (25.7MB), run=16591-17156msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=27950: Fri Mar 28 21:50:06 2025 read: IOPS=122, BW=1967KiB/s (2014kB/s)(5920KiB/3010msec) clat (usec): min=53, max=16246, avg=4062.12, stdev=2447.22 lat (usec): min=53, max=16246, avg=4062.82, stdev=2447.28 clat percentiles (usec): | 1.00th=[ 67], 5.00th=[ 74], 10.00th=[ 88], 20.00th=[ 2343], | 30.00th=[ 2868], 40.00th=[ 3326], 50.00th=[ 3851], 60.00th=[ 4424], | 70.00th=[ 5014], 80.00th=[ 5735], 90.00th=[ 6915], 95.00th=[ 8160], | 99.00th=[11207], 99.50th=[14615], 99.90th=[16188], 99.95th=[16188], | 99.99th=[16188] bw ( KiB/s): min= 1216, max= 2336, per=25.49%, avg=1957.33, stdev=390.04, samples=6 iops : min= 76, max= 146, avg=122.33, stdev=24.38, samples=6 write: IOPS=132, BW=2116KiB/s (2166kB/s)(6368KiB/3010msec) clat (usec): min=602, max=17068, avg=3638.86, stdev=3181.12 lat (usec): min=602, max=17070, avg=3639.66, stdev=3181.14 clat percentiles (usec): | 1.00th=[ 685], 5.00th=[ 758], 10.00th=[ 799], 20.00th=[ 848], | 30.00th=[ 1106], 40.00th=[ 2089], 50.00th=[ 2671], 60.00th=[ 3425], | 70.00th=[ 4424], 80.00th=[ 5997], 90.00th=[ 8455], 95.00th=[10290], | 99.00th=[15270], 99.50th=[15664], 99.90th=[17171], 99.95th=[17171], | 99.99th=[17171] bw ( KiB/s): min= 1440, max= 2560, per=26.08%, avg=2106.67, stdev=400.41, samples=6 iops : min= 90, max= 160, avg=131.67, stdev=25.03, samples=6 lat (usec) : 100=4.95%, 250=0.39%, 500=0.13%, 750=2.34%, 1000=11.98% lat (msec) : 2=7.03%, 4=32.68%, 10=36.59%, 20=3.91% cpu : usr=0.17%, sys=24.69%, ctx=1472, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=27951: Fri Mar 28 21:50:06 2025 read: IOPS=117, BW=1888KiB/s (1933kB/s)(5744KiB/3043msec) clat (usec): min=55, max=18115, avg=4372.49, stdev=2762.67 lat (usec): min=55, max=18116, avg=4373.18, stdev=2762.70 clat percentiles (usec): | 1.00th=[ 67], 5.00th=[ 76], 10.00th=[ 102], 20.00th=[ 2540], | 30.00th=[ 2966], 40.00th=[ 3458], 50.00th=[ 4015], 60.00th=[ 4686], | 70.00th=[ 5538], 80.00th=[ 6259], 90.00th=[ 7308], 95.00th=[ 8717], | 99.00th=[13173], 99.50th=[17957], 99.90th=[18220], 99.95th=[18220], | 99.99th=[18220] bw ( KiB/s): min= 1504, max= 2368, per=24.45%, avg=1877.33, stdev=330.49, samples=6 iops : min= 94, max= 148, avg=117.33, stdev=20.66, samples=6 write: IOPS=134, BW=2151KiB/s (2202kB/s)(6544KiB/3043msec) clat (usec): min=622, max=15119, avg=3558.22, stdev=3030.30 lat (usec): min=623, max=15120, avg=3559.11, stdev=3030.30 clat percentiles (usec): | 1.00th=[ 676], 5.00th=[ 742], 10.00th=[ 791], 20.00th=[ 840], | 30.00th=[ 971], 40.00th=[ 1844], 50.00th=[ 2671], 60.00th=[ 3818], | 70.00th=[ 4686], 80.00th=[ 5866], 90.00th=[ 7898], 95.00th=[ 9634], | 99.00th=[13960], 99.50th=[14222], 99.90th=[15139], 99.95th=[15139], | 99.99th=[15139] bw ( KiB/s): min= 1568, max= 2368, per=26.21%, avg=2117.33, stdev=295.66, samples=6 iops : min= 98, max= 148, avg=132.33, stdev=18.48, samples=6 lat (usec) : 100=4.56%, 250=0.52%, 500=0.13%, 750=3.12%, 1000=13.28% lat (msec) : 2=6.25%, 4=28.39%, 10=39.97%, 20=3.78% cpu : usr=0.20%, sys=24.88%, ctx=1553, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=27952: Fri Mar 28 21:50:06 2025 read: IOPS=125, BW=2006KiB/s (2054kB/s)(6256KiB/3119msec) clat (usec): min=58, max=17526, avg=4467.79, stdev=2505.92 lat (usec): min=58, max=17527, avg=4468.34, stdev=2505.94 clat percentiles (usec): | 1.00th=[ 68], 5.00th=[ 80], 10.00th=[ 2089], 20.00th=[ 2638], | 30.00th=[ 3195], 40.00th=[ 3654], 50.00th=[ 4228], 60.00th=[ 4883], | 70.00th=[ 5473], 80.00th=[ 6128], 90.00th=[ 7111], 95.00th=[ 8717], | 99.00th=[13173], 99.50th=[15401], 99.90th=[17433], 99.95th=[17433], | 99.99th=[17433] bw ( KiB/s): min= 1280, max= 2304, per=25.62%, avg=1967.17, stdev=374.34, samples=6 iops : min= 80, max= 144, avg=122.83, stdev=23.28, samples=6 write: IOPS=120, BW=1934KiB/s (1980kB/s)(6032KiB/3119msec) clat (usec): min=613, max=29801, avg=3535.45, stdev=3346.12 lat (usec): min=614, max=29803, avg=3536.86, stdev=3345.78 clat percentiles (usec): | 1.00th=[ 652], 5.00th=[ 758], 10.00th=[ 783], 20.00th=[ 824], | 30.00th=[ 930], 40.00th=[ 1483], 50.00th=[ 2769], 60.00th=[ 3458], | 70.00th=[ 4686], 80.00th=[ 5932], 90.00th=[ 7504], 95.00th=[ 9765], | 99.00th=[14484], 99.50th=[18744], 99.90th=[29754], 99.95th=[29754], | 99.99th=[29754] bw ( KiB/s): min= 1564, max= 2304, per=22.83%, avg=1844.67, stdev=289.60, samples=6 iops : min= 97, max= 144, avg=115.17, stdev=18.25, samples=6 lat (usec) : 100=4.04%, 750=2.34%, 1000=13.80% lat (msec) : 2=5.99%, 4=29.82%, 10=40.49%, 20=3.39%, 50=0.13% cpu : usr=0.03%, sys=24.43%, ctx=1538, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=27954: Fri Mar 28 21:50:06 2025 read: IOPS=122, BW=1965KiB/s (2013kB/s)(6032KiB/3069msec) clat (usec): min=58, max=18798, avg=4333.56, stdev=2503.10 lat (usec): min=58, max=18799, avg=4334.13, stdev=2503.10 clat percentiles (usec): | 1.00th=[ 68], 5.00th=[ 78], 10.00th=[ 110], 20.00th=[ 2737], | 30.00th=[ 3261], 40.00th=[ 3752], 50.00th=[ 4146], 60.00th=[ 4621], | 70.00th=[ 5276], 80.00th=[ 5997], 90.00th=[ 7046], 95.00th=[ 7963], | 99.00th=[13042], 99.50th=[14091], 99.90th=[18744], 99.95th=[18744], | 99.99th=[18744] bw ( KiB/s): min= 1376, max= 2208, per=25.07%, avg=1925.33, stdev=301.83, samples=6 iops : min= 86, max= 138, avg=120.33, stdev=18.86, samples=6 write: IOPS=127, BW=2038KiB/s (2087kB/s)(6256KiB/3069msec) clat (usec): min=605, max=18104, avg=3542.69, stdev=3103.02 lat (usec): min=606, max=18105, avg=3545.64, stdev=3103.32 clat percentiles (usec): | 1.00th=[ 685], 5.00th=[ 758], 10.00th=[ 791], 20.00th=[ 840], | 30.00th=[ 1045], 40.00th=[ 1549], 50.00th=[ 2737], 60.00th=[ 3326], | 70.00th=[ 4424], 80.00th=[ 6194], 90.00th=[ 7898], 95.00th=[ 9503], | 99.00th=[13304], 99.50th=[16188], 99.90th=[18220], 99.95th=[18220], | 99.99th=[18220] bw ( KiB/s): min= 1664, max= 2368, per=25.02%, avg=2021.33, stdev=258.72, samples=6 iops : min= 104, max= 148, avg=126.33, stdev=16.17, samples=6 lat (usec) : 100=4.56%, 250=0.65%, 750=2.21%, 1000=12.37% lat (msec) : 2=7.03%, 4=29.95%, 10=39.58%, 20=3.65% cpu : usr=0.49%, sys=24.48%, ctx=1519, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=7679KiB/s (7864kB/s), 1888KiB/s-2006KiB/s (1933kB/s-2054kB/s), io=23.4MiB (24.5MB), run=3010-3119msec WRITE: bw=8080KiB/s (8273kB/s), 1934KiB/s-2151KiB/s (1980kB/s-2202kB/s), io=24.6MiB (25.8MB), run=3010-3119msec rand-rw: (groupid=0, jobs=1): err= 0: pid=27953: Fri Mar 28 21:50:07 2025 read: IOPS=92, BW=1484KiB/s (1519kB/s)(5920KiB/3990msec) clat (usec): min=1757, max=18737, avg=4159.93, stdev=3205.80 lat (usec): min=1758, max=18738, avg=4160.55, stdev=3205.88 clat percentiles (usec): | 1.00th=[ 1811], 5.00th=[ 1958], 10.00th=[ 2024], 20.00th=[ 2180], | 30.00th=[ 2343], 40.00th=[ 2507], 50.00th=[ 2835], 60.00th=[ 3163], | 70.00th=[ 4015], 80.00th=[ 5014], 90.00th=[ 9241], 95.00th=[11469], | 99.00th=[16450], 99.50th=[17695], 99.90th=[18744], 99.95th=[18744], | 99.99th=[18744] bw ( KiB/s): min= 928, max= 2336, per=22.10%, avg=1312.00, stdev=496.43, samples=7 iops : min= 58, max= 146, avg=82.00, stdev=31.03, samples=7 write: IOPS=99, BW=1596KiB/s (1634kB/s)(6368KiB/3990msec) clat (usec): min=2521, max=25953, avg=5992.91, stdev=3713.41 lat (usec): min=2522, max=25955, avg=5994.24, stdev=3713.28 clat percentiles (usec): | 1.00th=[ 2573], 5.00th=[ 2802], 10.00th=[ 3097], 20.00th=[ 3326], | 30.00th=[ 3556], 40.00th=[ 3818], 50.00th=[ 4490], 60.00th=[ 5407], | 70.00th=[ 6652], 80.00th=[ 8586], 90.00th=[11076], 95.00th=[12387], | 99.00th=[20841], 99.50th=[22152], 99.90th=[26084], 99.95th=[26084], | 99.99th=[26084] bw ( KiB/s): min= 1088, max= 2496, per=22.69%, avg=1417.14, stdev=481.67, samples=7 iops : min= 68, max= 156, avg=88.57, stdev=30.10, samples=7 lat (msec) : 2=3.52%, 4=53.26%, 10=32.16%, 20=10.29%, 50=0.78% cpu : usr=0.20%, sys=11.83%, ctx=931, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=27955: Fri Mar 28 21:50:07 2025 read: IOPS=88, BW=1424KiB/s (1458kB/s)(5744KiB/4034msec) clat (usec): min=1732, max=23171, avg=4163.45, stdev=3112.94 lat (usec): min=1732, max=23172, avg=4164.02, stdev=3112.95 clat percentiles (usec): | 1.00th=[ 1844], 5.00th=[ 1942], 10.00th=[ 2024], 20.00th=[ 2180], | 30.00th=[ 2343], 40.00th=[ 2507], 50.00th=[ 2737], 60.00th=[ 3261], | 70.00th=[ 3884], 80.00th=[ 5932], 90.00th=[ 8848], 95.00th=[10945], | 99.00th=[14615], 99.50th=[15795], 99.90th=[23200], 99.95th=[23200], | 99.99th=[23200] bw ( KiB/s): min= 736, max= 2496, per=23.91%, avg=1419.38, stdev=610.31, samples=8 iops : min= 46, max= 156, avg=88.62, stdev=38.02, samples=8 write: IOPS=101, BW=1622KiB/s (1661kB/s)(6544KiB/4034msec) clat (usec): min=2486, max=21271, avg=6000.24, stdev=3496.75 lat (usec): min=2487, max=21273, avg=6001.58, stdev=3496.78 clat percentiles (usec): | 1.00th=[ 2638], 5.00th=[ 2966], 10.00th=[ 3097], 20.00th=[ 3359], | 30.00th=[ 3621], 40.00th=[ 3884], 50.00th=[ 4359], 60.00th=[ 5407], | 70.00th=[ 6915], 80.00th=[ 9110], 90.00th=[11207], 95.00th=[12780], | 99.00th=[17433], 99.50th=[19792], 99.90th=[21365], 99.95th=[21365], | 99.99th=[21365] bw ( KiB/s): min= 1024, max= 3129, per=25.99%, avg=1623.12, stdev=700.76, samples=8 iops : min= 64, max= 195, avg=101.38, stdev=43.62, samples=8 lat (msec) : 2=3.91%, 4=51.95%, 10=32.55%, 20=11.20%, 50=0.39% cpu : usr=0.10%, sys=11.75%, ctx=914, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=27956: Fri Mar 28 21:50:07 2025 read: IOPS=98, BW=1578KiB/s (1616kB/s)(6256KiB/3964msec) clat (usec): min=1732, max=23254, avg=4268.44, stdev=3320.64 lat (usec): min=1733, max=23255, avg=4269.05, stdev=3320.80 clat percentiles (usec): | 1.00th=[ 1860], 5.00th=[ 1942], 10.00th=[ 2024], 20.00th=[ 2180], | 30.00th=[ 2311], 40.00th=[ 2540], 50.00th=[ 2933], 60.00th=[ 3326], | 70.00th=[ 4113], 80.00th=[ 5669], 90.00th=[ 8717], 95.00th=[11469], | 99.00th=[17957], 99.50th=[18482], 99.90th=[23200], 99.95th=[23200], | 99.99th=[23200] bw ( KiB/s): min= 768, max= 2304, per=23.02%, avg=1366.86, stdev=480.96, samples=7 iops : min= 48, max= 144, avg=85.43, stdev=30.06, samples=7 write: IOPS=95, BW=1522KiB/s (1558kB/s)(6032KiB/3964msec) clat (usec): min=2380, max=25168, avg=6029.22, stdev=3747.47 lat (usec): min=2381, max=25169, avg=6030.49, stdev=3747.46 clat percentiles (usec): | 1.00th=[ 2573], 5.00th=[ 2900], 10.00th=[ 3032], 20.00th=[ 3326], | 30.00th=[ 3523], 40.00th=[ 3785], 50.00th=[ 4424], 60.00th=[ 5276], | 70.00th=[ 6980], 80.00th=[ 8455], 90.00th=[11076], 95.00th=[14222], | 99.00th=[19530], 99.50th=[21365], 99.90th=[25297], 99.95th=[25297], | 99.99th=[25297] bw ( KiB/s): min= 1056, max= 2432, per=22.03%, avg=1376.00, stdev=487.41, samples=7 iops : min= 66, max= 152, avg=86.00, stdev=30.46, samples=7 lat (msec) : 2=4.43%, 4=52.47%, 10=32.68%, 20=9.90%, 50=0.52% cpu : usr=0.15%, sys=11.78%, ctx=886, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=27957: Fri Mar 28 21:50:07 2025 read: IOPS=95, BW=1523KiB/s (1559kB/s)(6032KiB/3961msec) clat (usec): min=1787, max=20502, avg=4228.06, stdev=3315.69 lat (usec): min=1787, max=20503, avg=4228.63, stdev=3315.69 clat percentiles (usec): | 1.00th=[ 1827], 5.00th=[ 1926], 10.00th=[ 1991], 20.00th=[ 2180], | 30.00th=[ 2343], 40.00th=[ 2507], 50.00th=[ 2769], 60.00th=[ 3261], | 70.00th=[ 4080], 80.00th=[ 5538], 90.00th=[ 9503], 95.00th=[11600], | 99.00th=[17433], 99.50th=[19006], 99.90th=[20579], 99.95th=[20579], | 99.99th=[20579] bw ( KiB/s): min= 800, max= 2112, per=22.32%, avg=1325.14, stdev=412.26, samples=7 iops : min= 50, max= 132, avg=82.71, stdev=25.70, samples=7 write: IOPS=98, BW=1579KiB/s (1617kB/s)(6256KiB/3961msec) clat (usec): min=2391, max=21453, avg=5815.50, stdev=3367.81 lat (usec): min=2392, max=21456, avg=5816.69, stdev=3367.84 clat percentiles (usec): | 1.00th=[ 2573], 5.00th=[ 2802], 10.00th=[ 3097], 20.00th=[ 3392], | 30.00th=[ 3621], 40.00th=[ 3818], 50.00th=[ 4293], 60.00th=[ 5276], | 70.00th=[ 6521], 80.00th=[ 8160], 90.00th=[10683], 95.00th=[12780], | 99.00th=[19268], 99.50th=[20579], 99.90th=[21365], 99.95th=[21365], | 99.99th=[21365] bw ( KiB/s): min= 832, max= 2624, per=22.90%, avg=1430.43, stdev=564.62, samples=7 iops : min= 52, max= 164, avg=89.29, stdev=35.31, samples=7 lat (msec) : 2=5.21%, 4=52.21%, 10=31.90%, 20=10.29%, 50=0.39% cpu : usr=0.20%, sys=11.79%, ctx=902, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=5938KiB/s (6080kB/s), 1424KiB/s-1578KiB/s (1458kB/s-1616kB/s), io=23.4MiB (24.5MB), run=3961-4034msec WRITE: bw=6247KiB/s (6397kB/s), 1522KiB/s-1622KiB/s (1558kB/s-1661kB/s), io=24.6MiB (25.8MB), run=3961-4034msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=27966: Fri Mar 28 21:50:09 2025 read: IOPS=6, BW=6801KiB/s (6964kB/s)(7168KiB/1054msec) clat (usec): min=378, max=622647, avg=108715.29, stdev=227305.57 lat (usec): min=379, max=622648, avg=108716.28, stdev=227305.67 clat percentiles (usec): | 1.00th=[ 379], 5.00th=[ 379], 10.00th=[ 379], 20.00th=[ 437], | 30.00th=[ 22152], 40.00th=[ 22152], 50.00th=[ 32113], 60.00th=[ 36439], | 70.00th=[ 36439], 80.00th=[ 46924], 90.00th=[624952], 95.00th=[624952], | 99.00th=[624952], 99.50th=[624952], 99.90th=[624952], 99.95th=[624952], | 99.99th=[624952] bw ( KiB/s): min=10240, max=10240, per=63.30%, avg=10240.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 write: IOPS=4, BW=4858KiB/s (4974kB/s)(5120KiB/1054msec) clat (usec): min=30791, max=97534, avg=53277.98, stdev=27713.70 lat (usec): min=30857, max=97564, avg=53320.42, stdev=27698.88 clat percentiles (usec): | 1.00th=[30802], 5.00th=[30802], 10.00th=[30802], 20.00th=[30802], | 30.00th=[36439], 40.00th=[36439], 50.00th=[38536], 60.00th=[38536], | 70.00th=[63177], 80.00th=[63177], 90.00th=[98042], 95.00th=[98042], | 99.00th=[98042], 99.50th=[98042], 99.90th=[98042], 99.95th=[98042], | 99.99th=[98042] bw ( KiB/s): min= 8192, max= 8192, per=36.17%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 lat (usec) : 500=16.67% lat (msec) : 50=58.33%, 100=16.67%, 750=8.33% cpu : usr=0.00%, sys=27.26%, ctx=1082, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=27967: Fri Mar 28 21:50:09 2025 read: IOPS=1, BW=1618KiB/s (1657kB/s)(2048KiB/1266msec) clat (msec): min=23, max=752, avg=388.01, stdev=515.82 lat (msec): min=23, max=752, avg=388.01, stdev=515.82 clat percentiles (msec): | 1.00th=[ 24], 5.00th=[ 24], 10.00th=[ 24], 20.00th=[ 24], | 30.00th=[ 24], 40.00th=[ 24], 50.00th=[ 24], 60.00th=[ 751], | 70.00th=[ 751], 80.00th=[ 751], 90.00th=[ 751], 95.00th=[ 751], | 99.00th=[ 751], 99.50th=[ 751], 99.90th=[ 751], 99.95th=[ 751], | 99.99th=[ 751] bw ( KiB/s): min= 4096, max= 4096, per=25.32%, avg=4096.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 write: IOPS=7, BW=8088KiB/s (8283kB/s)(10.0MiB/1266msec) clat (msec): min=17, max=123, avg=46.14, stdev=31.24 lat (msec): min=17, max=123, avg=46.19, stdev=31.23 clat percentiles (msec): | 1.00th=[ 18], 5.00th=[ 18], 10.00th=[ 18], 20.00th=[ 18], | 30.00th=[ 26], 40.00th=[ 40], 50.00th=[ 40], 60.00th=[ 40], | 70.00th=[ 46], 80.00th=[ 46], 90.00th=[ 72], 95.00th=[ 124], | 99.00th=[ 124], 99.50th=[ 124], 99.90th=[ 124], 99.95th=[ 124], | 99.99th=[ 124] bw ( KiB/s): min=10240, max=10240, per=45.22%, avg=10240.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 lat (msec) : 20=16.67%, 50=58.33%, 100=8.33%, 250=8.33%, 1000=8.33% cpu : usr=0.00%, sys=27.04%, ctx=422, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=27968: Fri Mar 28 21:50:09 2025 read: IOPS=4, BW=5003KiB/s (5123kB/s)(6144KiB/1228msec) clat (usec): min=355, max=45552, avg=25852.88, stdev=15900.72 lat (usec): min=356, max=45553, avg=25853.54, stdev=15900.79 clat percentiles (usec): | 1.00th=[ 355], 5.00th=[ 355], 10.00th=[ 355], 20.00th=[15139], | 30.00th=[15139], 40.00th=[28443], 50.00th=[28443], 60.00th=[30802], | 70.00th=[34866], 80.00th=[34866], 90.00th=[45351], 95.00th=[45351], | 99.00th=[45351], 99.50th=[45351], 99.90th=[45351], 99.95th=[45351], | 99.99th=[45351] bw ( KiB/s): min= 4096, max= 4096, per=25.32%, avg=4096.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 write: IOPS=4, BW=5003KiB/s (5123kB/s)(6144KiB/1228msec) clat (msec): min=24, max=797, avg=174.39, stdev=306.04 lat (msec): min=24, max=797, avg=174.44, stdev=306.07 clat percentiles (msec): | 1.00th=[ 24], 5.00th=[ 24], 10.00th=[ 24], 20.00th=[ 34], | 30.00th=[ 34], 40.00th=[ 42], 50.00th=[ 42], 60.00th=[ 55], | 70.00th=[ 96], 80.00th=[ 96], 90.00th=[ 802], 95.00th=[ 802], | 99.00th=[ 802], 99.50th=[ 802], 99.90th=[ 802], 99.95th=[ 802], | 99.99th=[ 802] bw ( KiB/s): min= 2048, max= 4096, per=13.56%, avg=3072.00, stdev=1448.15, samples=2 iops : min= 2, max= 4, avg= 3.00, stdev= 1.41, samples=2 lat (usec) : 500=8.33% lat (msec) : 20=8.33%, 50=58.33%, 100=16.67%, 1000=8.33% cpu : usr=0.00%, sys=17.92%, ctx=256, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=27970: Fri Mar 28 21:50:09 2025 read: IOPS=4, BW=4444KiB/s (4551kB/s)(5120KiB/1152msec) clat (usec): min=329, max=43086, avg=12979.82, stdev=18985.97 lat (usec): min=330, max=43087, avg=12980.33, stdev=18986.09 clat percentiles (usec): | 1.00th=[ 330], 5.00th=[ 330], 10.00th=[ 330], 20.00th=[ 330], | 30.00th=[ 375], 40.00th=[ 375], 50.00th=[ 437], 60.00th=[ 437], | 70.00th=[20579], 80.00th=[20579], 90.00th=[43254], 95.00th=[43254], | 99.00th=[43254], 99.50th=[43254], 99.90th=[43254], 99.95th=[43254], | 99.99th=[43254] bw ( KiB/s): min= 8192, max= 8192, per=50.64%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=6, BW=6222KiB/s (6372kB/s)(7168KiB/1152msec) clat (msec): min=24, max=749, avg=153.99, stdev=265.68 lat (msec): min=24, max=749, avg=154.06, stdev=265.69 clat percentiles (msec): | 1.00th=[ 25], 5.00th=[ 25], 10.00th=[ 25], 20.00th=[ 25], | 30.00th=[ 29], 40.00th=[ 29], 50.00th=[ 30], 60.00th=[ 109], | 70.00th=[ 109], 80.00th=[ 113], 90.00th=[ 751], 95.00th=[ 751], | 99.00th=[ 751], 99.50th=[ 751], 99.90th=[ 751], 99.95th=[ 751], | 99.99th=[ 751] bw ( KiB/s): min= 2048, max= 8192, per=22.61%, avg=5120.00, stdev=4344.46, samples=2 iops : min= 2, max= 8, avg= 5.00, stdev= 4.24, samples=2 lat (usec) : 500=25.00% lat (msec) : 50=50.00%, 250=16.67%, 750=8.33% cpu : usr=0.00%, sys=16.77%, ctx=274, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=15.8MiB/s (16.6MB/s), 1618KiB/s-6801KiB/s (1657kB/s-6964kB/s), io=20.0MiB (20.0MB), run=1054-1266msec WRITE: bw=22.1MiB/s (23.2MB/s), 4858KiB/s-8088KiB/s (4974kB/s-8283kB/s), io=28.0MiB (29.4MB), run=1054-1266msec rand-rw: (groupid=0, jobs=1): err= 0: pid=27969: Fri Mar 28 21:50:09 2025 read: IOPS=5, BW=5385KiB/s (5515kB/s)(7168KiB/1331msec) clat (msec): min=12, max=598, avg=157.64, stdev=219.22 lat (msec): min=12, max=598, avg=157.64, stdev=219.22 clat percentiles (msec): | 1.00th=[ 13], 5.00th=[ 13], 10.00th=[ 13], 20.00th=[ 14], | 30.00th=[ 19], 40.00th=[ 19], 50.00th=[ 33], 60.00th=[ 140], | 70.00th=[ 140], 80.00th=[ 288], 90.00th=[ 600], 95.00th=[ 600], | 99.00th=[ 600], 99.50th=[ 600], 99.90th=[ 600], 99.95th=[ 600], | 99.99th=[ 600] bw ( KiB/s): min= 6119, max= 6119, per=39.77%, avg=6119.00, stdev= 0.00, samples=1 iops : min= 5, max= 5, avg= 5.00, stdev= 0.00, samples=1 write: IOPS=3, BW=3847KiB/s (3939kB/s)(5120KiB/1331msec) clat (usec): min=13897, max=71546, avg=32493.51, stdev=25424.42 lat (usec): min=13937, max=71608, avg=32529.40, stdev=25434.39 clat percentiles (usec): | 1.00th=[13960], 5.00th=[13960], 10.00th=[13960], 20.00th=[13960], | 30.00th=[14353], 40.00th=[14353], 50.00th=[17433], 60.00th=[17433], | 70.00th=[45351], 80.00th=[45351], 90.00th=[71828], 95.00th=[71828], | 99.00th=[71828], 99.50th=[71828], 99.90th=[71828], 99.95th=[71828], | 99.99th=[71828] lat (msec) : 20=50.00%, 50=16.67%, 100=8.33%, 250=8.33%, 500=8.33% lat (msec) : 750=8.33% cpu : usr=0.00%, sys=8.20%, ctx=132, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=27971: Fri Mar 28 21:50:09 2025 read: IOPS=1, BW=1553KiB/s (1590kB/s)(2048KiB/1319msec) clat (msec): min=21, max=756, avg=388.70, stdev=519.59 lat (msec): min=21, max=756, avg=388.70, stdev=519.59 clat percentiles (msec): | 1.00th=[ 22], 5.00th=[ 22], 10.00th=[ 22], 20.00th=[ 22], | 30.00th=[ 22], 40.00th=[ 22], 50.00th=[ 22], 60.00th=[ 760], | 70.00th=[ 760], 80.00th=[ 760], 90.00th=[ 760], 95.00th=[ 760], | 99.00th=[ 760], 99.50th=[ 760], 99.90th=[ 760], 99.95th=[ 760], | 99.99th=[ 760] bw ( KiB/s): min= 2048, max= 2048, per=13.31%, avg=2048.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=7, BW=7763KiB/s (7950kB/s)(10.0MiB/1319msec) clat (msec): min=19, max=100, avg=51.17, stdev=30.26 lat (msec): min=19, max=100, avg=51.22, stdev=30.26 clat percentiles (msec): | 1.00th=[ 20], 5.00th=[ 20], 10.00th=[ 20], 20.00th=[ 21], | 30.00th=[ 29], 40.00th=[ 33], 50.00th=[ 37], 60.00th=[ 47], | 70.00th=[ 84], 80.00th=[ 84], 90.00th=[ 93], 95.00th=[ 101], | 99.00th=[ 101], 99.50th=[ 101], 99.90th=[ 101], 99.95th=[ 101], | 99.99th=[ 101] bw ( KiB/s): min= 4096, max= 4096, per=19.01%, avg=4096.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 20=8.33%, 50=50.00%, 100=25.00%, 250=8.33%, 1000=8.33% cpu : usr=0.08%, sys=6.75%, ctx=60, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=27972: Fri Mar 28 21:50:09 2025 read: IOPS=4, BW=4823KiB/s (4938kB/s)(6144KiB/1274msec) clat (usec): min=15385, max=76563, avg=43043.24, stdev=25942.70 lat (usec): min=15386, max=76564, avg=43043.97, stdev=25942.57 clat percentiles (usec): | 1.00th=[15401], 5.00th=[15401], 10.00th=[15401], 20.00th=[15664], | 30.00th=[15664], 40.00th=[30802], 50.00th=[30802], 60.00th=[57934], | 70.00th=[62129], 80.00th=[62129], 90.00th=[77071], 95.00th=[77071], | 99.00th=[77071], 99.50th=[77071], 99.90th=[77071], 99.95th=[77071], | 99.99th=[77071] bw ( KiB/s): min= 4096, max= 4096, per=26.62%, avg=4096.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 write: IOPS=4, BW=4823KiB/s (4938kB/s)(6144KiB/1274msec) clat (msec): min=18, max=761, avg=166.45, stdev=292.47 lat (msec): min=18, max=761, avg=166.50, stdev=292.46 clat percentiles (msec): | 1.00th=[ 19], 5.00th=[ 19], 10.00th=[ 19], 20.00th=[ 28], | 30.00th=[ 28], 40.00th=[ 44], 50.00th=[ 44], 60.00th=[ 59], | 70.00th=[ 89], 80.00th=[ 89], 90.00th=[ 760], 95.00th=[ 760], | 99.00th=[ 760], 99.50th=[ 760], 99.90th=[ 760], 99.95th=[ 760], | 99.99th=[ 760] bw ( KiB/s): min= 2048, max= 4096, per=14.26%, avg=3072.00, stdev=1448.15, samples=2 iops : min= 2, max= 4, avg= 3.00, stdev= 1.41, samples=2 lat (msec) : 20=25.00%, 50=25.00%, 100=41.67%, 1000=8.33% cpu : usr=0.00%, sys=6.52%, ctx=65, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=27973: Fri Mar 28 21:50:09 2025 read: IOPS=3, BW=4093KiB/s (4191kB/s)(5120KiB/1251msec) clat (usec): min=11626, max=38779, avg=25973.76, stdev=11887.17 lat (usec): min=11627, max=38780, avg=25974.48, stdev=11887.26 clat percentiles (usec): | 1.00th=[11600], 5.00th=[11600], 10.00th=[11600], 20.00th=[11600], | 30.00th=[16909], 40.00th=[16909], 50.00th=[26084], 60.00th=[26084], | 70.00th=[36439], 80.00th=[36439], 90.00th=[38536], 95.00th=[38536], | 99.00th=[38536], 99.50th=[38536], 99.90th=[38536], 99.95th=[38536], | 99.99th=[38536] write: IOPS=5, BW=5730KiB/s (5867kB/s)(7168KiB/1251msec) clat (msec): min=14, max=774, avg=150.94, stdev=276.83 lat (msec): min=14, max=774, avg=151.01, stdev=276.83 clat percentiles (msec): | 1.00th=[ 15], 5.00th=[ 15], 10.00th=[ 15], 20.00th=[ 20], | 30.00th=[ 20], 40.00th=[ 20], 50.00th=[ 56], 60.00th=[ 77], | 70.00th=[ 77], 80.00th=[ 99], 90.00th=[ 776], 95.00th=[ 776], | 99.00th=[ 776], 99.50th=[ 776], 99.90th=[ 776], 99.95th=[ 776], | 99.99th=[ 776] bw ( KiB/s): min= 2048, max= 6144, per=19.01%, avg=4096.00, stdev=2896.31, samples=2 iops : min= 2, max= 6, avg= 4.00, stdev= 2.83, samples=2 lat (msec) : 20=41.67%, 50=25.00%, 100=25.00%, 1000=8.33% cpu : usr=0.00%, sys=6.96%, ctx=64, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=15.0MiB/s (15.8MB/s), 1553KiB/s-5385KiB/s (1590kB/s-5515kB/s), io=20.0MiB (20.0MB), run=1251-1331msec WRITE: bw=21.0MiB/s (22.1MB/s), 3847KiB/s-7763KiB/s (3939kB/s-7950kB/s), io=28.0MiB (29.4MB), run=1251-1331msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=27980: Fri Mar 28 21:50:12 2025 read: IOPS=1, BW=7026KiB/s (7194kB/s)(12.0MiB/1749msec) clat (msec): min=111, max=1458, avg=575.15, stdev=765.14 lat (msec): min=111, max=1458, avg=575.16, stdev=765.14 clat percentiles (msec): | 1.00th=[ 112], 5.00th=[ 112], 10.00th=[ 112], 20.00th=[ 112], | 30.00th=[ 112], 40.00th=[ 157], 50.00th=[ 157], 60.00th=[ 157], | 70.00th=[ 1452], 80.00th=[ 1452], 90.00th=[ 1452], 95.00th=[ 1452], | 99.00th=[ 1452], 99.50th=[ 1452], 99.90th=[ 1452], 99.95th=[ 1452], | 99.99th=[ 1452] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67% cpu : usr=0.06%, sys=16.76%, ctx=1946, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=27981: Fri Mar 28 21:50:12 2025 read: IOPS=0, BW=1911KiB/s (1957kB/s)(4096KiB/2143msec) clat (nsec): min=644451k, max=644451k, avg=644450511.00, stdev= 0.00 lat (nsec): min=644453k, max=644453k, avg=644452991.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 642], 5.00th=[ 642], 10.00th=[ 642], 20.00th=[ 642], | 30.00th=[ 642], 40.00th=[ 642], 50.00th=[ 642], 60.00th=[ 642], | 70.00th=[ 642], 80.00th=[ 642], 90.00th=[ 642], 95.00th=[ 642], | 99.00th=[ 642], 99.50th=[ 642], 99.90th=[ 642], 99.95th=[ 642], | 99.99th=[ 642] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=0, BW=3823KiB/s (3914kB/s)(8192KiB/2143msec) clat (msec): min=307, max=1178, avg=742.86, stdev=615.75 lat (msec): min=307, max=1178, avg=743.17, stdev=615.90 clat percentiles (msec): | 1.00th=[ 309], 5.00th=[ 309], 10.00th=[ 309], 20.00th=[ 309], | 30.00th=[ 309], 40.00th=[ 309], 50.00th=[ 309], 60.00th=[ 1183], | 70.00th=[ 1183], 80.00th=[ 1183], 90.00th=[ 1183], 95.00th=[ 1183], | 99.00th=[ 1183], 99.50th=[ 1183], 99.90th=[ 1183], 99.95th=[ 1183], | 99.99th=[ 1183] bw ( KiB/s): min= 8192, max= 8192, per=53.58%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 500=33.33%, 750=33.33% cpu : usr=0.05%, sys=15.83%, ctx=937, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=27982: Fri Mar 28 21:50:12 2025 write: IOPS=1, BW=6012KiB/s (6156kB/s)(12.0MiB/2044msec) clat (msec): min=191, max=1614, avg=674.72, stdev=813.76 lat (msec): min=191, max=1614, avg=674.98, stdev=813.78 clat percentiles (msec): | 1.00th=[ 192], 5.00th=[ 192], 10.00th=[ 192], 20.00th=[ 192], | 30.00th=[ 192], 40.00th=[ 220], 50.00th=[ 220], 60.00th=[ 220], | 70.00th=[ 1620], 80.00th=[ 1620], 90.00th=[ 1620], 95.00th=[ 1620], | 99.00th=[ 1620], 99.50th=[ 1620], 99.90th=[ 1620], 99.95th=[ 1620], | 99.99th=[ 1620] bw ( KiB/s): min=15142, max=15142, per=99.03%, avg=15142.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67% cpu : usr=0.00%, sys=13.22%, ctx=47, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=27983: Fri Mar 28 21:50:12 2025 write: IOPS=1, BW=6454KiB/s (6609kB/s)(12.0MiB/1904msec) clat (msec): min=152, max=1559, avg=631.18, stdev=804.28 lat (msec): min=152, max=1559, avg=631.42, stdev=804.16 clat percentiles (msec): | 1.00th=[ 153], 5.00th=[ 153], 10.00th=[ 153], 20.00th=[ 153], | 30.00th=[ 153], 40.00th=[ 182], 50.00th=[ 182], 60.00th=[ 182], | 70.00th=[ 1552], 80.00th=[ 1552], 90.00th=[ 1552], 95.00th=[ 1552], | 99.00th=[ 1552], 99.50th=[ 1552], 99.90th=[ 1552], 99.95th=[ 1552], | 99.99th=[ 1552] lat (msec) : 250=66.67% cpu : usr=0.05%, sys=15.40%, ctx=32, majf=0, minf=28 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=7645KiB/s (7829kB/s), 1911KiB/s-7026KiB/s (1957kB/s-7194kB/s), io=16.0MiB (16.8MB), run=1749-2143msec WRITE: bw=14.9MiB/s (15.7MB/s), 3823KiB/s-6454KiB/s (3914kB/s-6609kB/s), io=32.0MiB (33.6MB), run=1904-2143msec rand-rw: (groupid=0, jobs=1): err= 0: pid=27984: Fri Mar 28 21:50:13 2025 read: IOPS=1, BW=5498KiB/s (5630kB/s)(12.0MiB/2235msec) clat (msec): min=231, max=1696, avg=738.28, stdev=830.24 lat (msec): min=231, max=1696, avg=738.28, stdev=830.24 clat percentiles (msec): | 1.00th=[ 232], 5.00th=[ 232], 10.00th=[ 232], 20.00th=[ 232], | 30.00th=[ 232], 40.00th=[ 288], 50.00th=[ 288], 60.00th=[ 288], | 70.00th=[ 1703], 80.00th=[ 1703], 90.00th=[ 1703], 95.00th=[ 1703], | 99.00th=[ 1703], 99.50th=[ 1703], 99.90th=[ 1703], 99.95th=[ 1703], | 99.99th=[ 1703] bw ( KiB/s): min=16351, max=16351, per=100.00%, avg=16351.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=33.33% cpu : usr=0.00%, sys=4.83%, ctx=36, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=27985: Fri Mar 28 21:50:13 2025 read: IOPS=0, BW=1811KiB/s (1854kB/s)(4096KiB/2262msec) clat (nsec): min=1696.4M, max=1696.4M, avg=1696367076.00, stdev= 0.00 lat (nsec): min=1696.4M, max=1696.4M, avg=1696368910.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 1703], 5.00th=[ 1703], 10.00th=[ 1703], 20.00th=[ 1703], | 30.00th=[ 1703], 40.00th=[ 1703], 50.00th=[ 1703], 60.00th=[ 1703], | 70.00th=[ 1703], 80.00th=[ 1703], 90.00th=[ 1703], 95.00th=[ 1703], | 99.00th=[ 1703], 99.50th=[ 1703], 99.90th=[ 1703], 99.95th=[ 1703], | 99.99th=[ 1703] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=0, BW=3622KiB/s (3708kB/s)(8192KiB/2262msec) clat (msec): min=66, max=485, avg=275.80, stdev=295.99 lat (msec): min=66, max=485, avg=275.98, stdev=296.07 clat percentiles (msec): | 1.00th=[ 67], 5.00th=[ 67], 10.00th=[ 67], 20.00th=[ 67], | 30.00th=[ 67], 40.00th=[ 67], 50.00th=[ 67], 60.00th=[ 485], | 70.00th=[ 485], 80.00th=[ 485], 90.00th=[ 485], 95.00th=[ 485], | 99.00th=[ 485], 99.50th=[ 485], 99.90th=[ 485], 99.95th=[ 485], | 99.99th=[ 485] lat (msec) : 100=33.33%, 500=33.33% cpu : usr=0.09%, sys=5.40%, ctx=73, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=27986: Fri Mar 28 21:50:13 2025 write: IOPS=1, BW=5324KiB/s (5452kB/s)(12.0MiB/2308msec) clat (msec): min=239, max=1770, avg=763.91, stdev=872.41 lat (msec): min=239, max=1771, avg=764.16, stdev=872.33 clat percentiles (msec): | 1.00th=[ 241], 5.00th=[ 241], 10.00th=[ 241], 20.00th=[ 241], | 30.00th=[ 241], 40.00th=[ 284], 50.00th=[ 284], 60.00th=[ 284], | 70.00th=[ 1770], 80.00th=[ 1770], 90.00th=[ 1770], 95.00th=[ 1770], | 99.00th=[ 1770], 99.50th=[ 1770], 99.90th=[ 1770], 99.95th=[ 1770], | 99.99th=[ 1770] bw ( KiB/s): min= 8192, max= 8192, per=57.70%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=33.33% cpu : usr=0.04%, sys=4.46%, ctx=103, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=27987: Fri Mar 28 21:50:13 2025 write: IOPS=1, BW=5399KiB/s (5529kB/s)(12.0MiB/2276msec) clat (msec): min=73, max=1792, avg=746.92, stdev=917.65 lat (msec): min=74, max=1792, avg=747.22, stdev=917.74 clat percentiles (msec): | 1.00th=[ 74], 5.00th=[ 74], 10.00th=[ 74], 20.00th=[ 74], | 30.00th=[ 74], 40.00th=[ 376], 50.00th=[ 376], 60.00th=[ 376], | 70.00th=[ 1787], 80.00th=[ 1787], 90.00th=[ 1787], 95.00th=[ 1787], | 99.00th=[ 1787], 99.50th=[ 1787], 99.90th=[ 1787], 99.95th=[ 1787], | 99.99th=[ 1787] bw ( KiB/s): min= 8192, max= 8192, per=57.70%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 500=33.33% cpu : usr=0.04%, sys=6.42%, ctx=151, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=7243KiB/s (7417kB/s), 1811KiB/s-5498KiB/s (1854kB/s-5630kB/s), io=16.0MiB (16.8MB), run=2235-2262msec WRITE: bw=13.9MiB/s (14.5MB/s), 3622KiB/s-5399KiB/s (3708kB/s-5529kB/s), io=32.0MiB (33.6MB), run=2262-2308msec