== sanity test 398b: DIO and buffer IO race ============== 13:22:55 (1740680575) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.48239 s, 34.0 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=8700: Thu Feb 27 13:23:09 2025 read: IOPS=127, BW=511KiB/s (523kB/s)(5992KiB/11726msec) clat (usec): min=51, max=33525, avg=3478.39, stdev=3101.70 lat (usec): min=52, max=33525, avg=3479.06, stdev=3101.68 clat percentiles (usec): | 1.00th=[ 57], 5.00th=[ 67], 10.00th=[ 76], 20.00th=[ 1696], | 30.00th=[ 2147], 40.00th=[ 2474], 50.00th=[ 2835], 60.00th=[ 3261], | 70.00th=[ 4178], 80.00th=[ 5407], 90.00th=[ 7046], 95.00th=[ 8455], | 99.00th=[12256], 99.50th=[20317], 99.90th=[32375], 99.95th=[33424], | 99.99th=[33424] bw ( KiB/s): min= 288, max= 688, per=24.63%, avg=504.26, stdev=99.99, samples=23 iops : min= 72, max= 172, avg=126.00, stdev=25.04, samples=23 write: IOPS=134, BW=537KiB/s (550kB/s)(6296KiB/11726msec) clat (usec): min=420, max=163880, avg=4069.91, stdev=5644.01 lat (usec): min=421, max=163881, avg=4071.59, stdev=5643.89 clat percentiles (usec): | 1.00th=[ 474], 5.00th=[ 510], 10.00th=[ 537], 20.00th=[ 603], | 30.00th=[ 775], 40.00th=[ 1434], 50.00th=[ 2999], 60.00th=[ 4080], | 70.00th=[ 5211], 80.00th=[ 6783], 90.00th=[ 9372], 95.00th=[ 11731], | 99.00th=[ 17433], 99.50th=[ 21365], 99.90th=[ 30016], 99.95th=[164627], | 99.99th=[164627] bw ( KiB/s): min= 360, max= 832, per=24.76%, avg=530.61, stdev=103.67, samples=23 iops : min= 90, max= 208, avg=132.61, stdev=25.89, samples=23 lat (usec) : 100=8.20%, 250=1.04%, 500=1.76%, 750=13.31%, 1000=3.32% lat (msec) : 2=6.64%, 4=29.65%, 10=30.79%, 20=4.69%, 50=0.55% lat (msec) : 250=0.03% cpu : usr=0.14%, sys=22.68%, ctx=5500, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8701: Thu Feb 27 13:23:09 2025 read: IOPS=126, BW=508KiB/s (520kB/s)(5940KiB/11700msec) clat (usec): min=51, max=31672, avg=3365.10, stdev=2928.30 lat (usec): min=51, max=31673, avg=3365.88, stdev=2928.51 clat percentiles (usec): | 1.00th=[ 58], 5.00th=[ 68], 10.00th=[ 79], 20.00th=[ 176], | 30.00th=[ 2089], 40.00th=[ 2409], 50.00th=[ 2769], 60.00th=[ 3195], | 70.00th=[ 3949], 80.00th=[ 5407], 90.00th=[ 7046], 95.00th=[ 8586], | 99.00th=[12125], 99.50th=[14091], 99.90th=[24249], 99.95th=[31589], | 99.99th=[31589] bw ( KiB/s): min= 344, max= 728, per=24.65%, avg=504.61, stdev=104.25, samples=23 iops : min= 86, max= 182, avg=126.13, stdev=26.06, samples=23 write: IOPS=135, BW=543KiB/s (556kB/s)(6348KiB/11700msec) clat (usec): min=451, max=181955, avg=4153.56, stdev=6051.89 lat (usec): min=452, max=181955, avg=4154.27, stdev=6051.88 clat percentiles (usec): | 1.00th=[ 482], 5.00th=[ 519], 10.00th=[ 545], 20.00th=[ 611], | 30.00th=[ 840], 40.00th=[ 1598], 50.00th=[ 3064], 60.00th=[ 4047], | 70.00th=[ 5145], 80.00th=[ 6587], 90.00th=[ 9503], 95.00th=[ 12125], | 99.00th=[ 17695], 99.50th=[ 21627], 99.90th=[ 37487], 99.95th=[181404], | 99.99th=[181404] bw ( KiB/s): min= 351, max= 736, per=24.88%, avg=533.17, stdev=105.79, samples=23 iops : min= 87, max= 184, avg=133.26, stdev=26.51, samples=23 lat (usec) : 100=7.88%, 250=2.02%, 500=1.50%, 750=12.60%, 1000=3.68% lat (msec) : 2=6.64%, 4=30.47%, 10=29.46%, 20=5.27%, 50=0.46% lat (msec) : 250=0.03% cpu : usr=0.16%, sys=22.78%, ctx=5458, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8703: Thu Feb 27 13:23:09 2025 read: IOPS=128, BW=514KiB/s (526kB/s)(6028KiB/11725msec) clat (usec): min=51, max=40390, avg=3380.31, stdev=3093.50 lat (usec): min=51, max=40391, avg=3381.83, stdev=3093.71 clat percentiles (usec): | 1.00th=[ 57], 5.00th=[ 67], 10.00th=[ 73], 20.00th=[ 116], | 30.00th=[ 2040], 40.00th=[ 2376], 50.00th=[ 2802], 60.00th=[ 3326], | 70.00th=[ 4080], 80.00th=[ 5473], 90.00th=[ 7111], 95.00th=[ 8586], | 99.00th=[11469], 99.50th=[14877], 99.90th=[31589], 99.95th=[40633], | 99.99th=[40633] bw ( KiB/s): min= 335, max= 728, per=24.77%, avg=507.09, stdev=102.52, samples=23 iops : min= 83, max= 182, avg=126.74, stdev=25.69, samples=23 write: IOPS=133, BW=534KiB/s (547kB/s)(6260KiB/11725msec) clat (usec): min=412, max=181731, avg=4168.32, stdev=6190.34 lat (usec): min=413, max=181732, avg=4169.31, stdev=6190.33 clat percentiles (usec): | 1.00th=[ 474], 5.00th=[ 510], 10.00th=[ 537], 20.00th=[ 611], | 30.00th=[ 840], 40.00th=[ 1582], 50.00th=[ 3163], 60.00th=[ 4080], | 70.00th=[ 5211], 80.00th=[ 6718], 90.00th=[ 9241], 95.00th=[ 11863], | 99.00th=[ 18220], 99.50th=[ 20579], 99.90th=[ 54789], 99.95th=[181404], | 99.99th=[181404] bw ( KiB/s): min= 368, max= 632, per=24.49%, avg=524.83, stdev=74.20, samples=23 iops : min= 92, max= 158, avg=131.17, stdev=18.60, samples=23 lat (usec) : 100=9.21%, 250=1.43%, 500=2.08%, 750=11.88%, 1000=4.10% lat (msec) : 2=6.74%, 4=28.52%, 10=30.50%, 20=5.11%, 50=0.36% lat (msec) : 100=0.03%, 250=0.03% cpu : usr=0.17%, sys=22.39%, ctx=5465, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8705: Thu Feb 27 13:23:09 2025 read: IOPS=130, BW=521KiB/s (534kB/s)(6052KiB/11609msec) clat (usec): min=52, max=166775, avg=3389.36, stdev=4986.48 lat (usec): min=53, max=166775, avg=3390.06, stdev=4986.47 clat percentiles (usec): | 1.00th=[ 58], 5.00th=[ 68], 10.00th=[ 77], 20.00th=[ 247], | 30.00th=[ 2114], 40.00th=[ 2442], 50.00th=[ 2769], 60.00th=[ 3261], | 70.00th=[ 4015], 80.00th=[ 5211], 90.00th=[ 6849], 95.00th=[ 8094], | 99.00th=[ 10945], 99.50th=[ 14353], 99.90th=[ 23987], 99.95th=[166724], | 99.99th=[166724] bw ( KiB/s): min= 304, max= 696, per=25.34%, avg=518.65, stdev=85.82, samples=23 iops : min= 76, max= 174, avg=129.57, stdev=21.45, samples=23 write: IOPS=134, BW=537KiB/s (550kB/s)(6236KiB/11609msec) clat (usec): min=409, max=40687, avg=4086.10, stdev=4295.21 lat (usec): min=410, max=40688, avg=4086.79, stdev=4295.21 clat percentiles (usec): | 1.00th=[ 478], 5.00th=[ 515], 10.00th=[ 545], 20.00th=[ 619], | 30.00th=[ 832], 40.00th=[ 1696], 50.00th=[ 3064], 60.00th=[ 4080], | 70.00th=[ 5080], 80.00th=[ 6652], 90.00th=[ 9503], 95.00th=[11863], | 99.00th=[19006], 99.50th=[23987], 99.90th=[34866], 99.95th=[40633], | 99.99th=[40633] bw ( KiB/s): min= 384, max= 736, per=25.03%, avg=536.35, stdev=94.12, samples=23 iops : min= 96, max= 184, avg=134.00, stdev=23.51, samples=23 lat (usec) : 100=8.63%, 250=1.24%, 500=1.66%, 750=12.50%, 1000=3.48% lat (msec) : 2=7.26%, 4=29.56%, 10=30.53%, 20=4.59%, 50=0.52% lat (msec) : 250=0.03% cpu : usr=0.15%, sys=22.86%, ctx=5477, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2048KiB/s (2097kB/s), 508KiB/s-521KiB/s (520kB/s-534kB/s), io=23.4MiB (24.6MB), run=11609-11726msec WRITE: bw=2144KiB/s (2195kB/s), 534KiB/s-543KiB/s (547kB/s-556kB/s), io=24.6MiB (25.7MB), run=11609-11726msec rand-rw: (groupid=0, jobs=1): err= 0: pid=8699: Thu Feb 27 13:23:30 2025 read: IOPS=46, BW=186KiB/s (190kB/s)(5992KiB/32234msec) clat (usec): min=1556, max=187807, avg=3494.31, stdev=5717.04 lat (usec): min=1557, max=187808, avg=3494.97, stdev=5717.05 clat percentiles (usec): | 1.00th=[ 1680], 5.00th=[ 1778], 10.00th=[ 1811], 20.00th=[ 1909], | 30.00th=[ 1975], 40.00th=[ 2057], 50.00th=[ 2147], 60.00th=[ 2311], | 70.00th=[ 2704], 80.00th=[ 4228], 90.00th=[ 6849], 95.00th=[ 9241], | 99.00th=[ 15926], 99.50th=[ 18744], 99.90th=[ 49021], 99.95th=[187696], | 99.99th=[187696] bw ( KiB/s): min= 64, max= 328, per=24.94%, avg=185.55, stdev=72.15, samples=64 iops : min= 16, max= 82, avg=46.33, stdev=18.01, samples=64 write: IOPS=48, BW=195KiB/s (200kB/s)(6296KiB/32234msec) clat (usec): min=8345, max=52837, avg=17084.92, stdev=6719.32 lat (usec): min=8345, max=52838, avg=17085.73, stdev=6719.36 clat percentiles (usec): | 1.00th=[ 8979], 5.00th=[ 9765], 10.00th=[10683], 20.00th=[11600], | 30.00th=[12911], 40.00th=[14484], 50.00th=[15795], 60.00th=[16909], | 70.00th=[18220], 80.00th=[20579], 90.00th=[26870], 95.00th=[31065], | 99.00th=[39584], 99.50th=[45876], 99.90th=[49021], 99.95th=[52691], | 99.99th=[52691] bw ( KiB/s): min= 64, max= 264, per=25.02%, avg=194.92, stdev=59.76, samples=64 iops : min= 16, max= 66, avg=48.67, stdev=14.93, samples=64 lat (msec) : 2=16.83%, 4=21.97%, 10=11.33%, 20=38.77%, 50=11.04% lat (msec) : 100=0.03%, 250=0.03% cpu : usr=0.09%, sys=5.65%, ctx=3484, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8702: Thu Feb 27 13:23:30 2025 read: IOPS=46, BW=185KiB/s (190kB/s)(5940KiB/32044msec) clat (usec): min=1595, max=171849, avg=3391.02, stdev=5192.50 lat (usec): min=1596, max=171850, avg=3391.69, stdev=5192.54 clat percentiles (usec): | 1.00th=[ 1680], 5.00th=[ 1778], 10.00th=[ 1811], 20.00th=[ 1893], | 30.00th=[ 1942], 40.00th=[ 2024], 50.00th=[ 2114], 60.00th=[ 2311], | 70.00th=[ 2638], 80.00th=[ 4015], 90.00th=[ 6652], 95.00th=[ 9110], | 99.00th=[ 13698], 99.50th=[ 17957], 99.90th=[ 38011], 99.95th=[170918], | 99.99th=[170918] bw ( KiB/s): min= 16, max= 352, per=24.88%, avg=185.14, stdev=81.13, samples=64 iops : min= 4, max= 88, avg=46.20, stdev=20.25, samples=64 write: IOPS=49, BW=198KiB/s (203kB/s)(6348KiB/32044msec) clat (usec): min=8228, max=69167, avg=16949.63, stdev=6824.39 lat (usec): min=8228, max=69168, avg=16950.40, stdev=6824.44 clat percentiles (usec): | 1.00th=[ 8979], 5.00th=[ 9896], 10.00th=[10421], 20.00th=[11600], | 30.00th=[12780], 40.00th=[14091], 50.00th=[15664], 60.00th=[16909], | 70.00th=[18220], 80.00th=[20055], 90.00th=[26084], 95.00th=[30802], | 99.00th=[40109], 99.50th=[45876], 99.90th=[67634], 99.95th=[68682], | 99.99th=[68682] bw ( KiB/s): min= 80, max= 280, per=25.40%, avg=197.89, stdev=59.96, samples=64 iops : min= 20, max= 70, avg=49.39, stdev=14.96, samples=64 lat (msec) : 2=17.84%, 4=20.77%, 10=11.17%, 20=39.75%, 50=10.29% lat (msec) : 100=0.16%, 250=0.03% cpu : usr=0.10%, sys=5.63%, ctx=3492, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8704: Thu Feb 27 13:23:30 2025 read: IOPS=47, BW=189KiB/s (193kB/s)(6028KiB/31907msec) clat (usec): min=1583, max=48814, avg=3487.32, stdev=3415.85 lat (usec): min=1584, max=48814, avg=3488.03, stdev=3415.83 clat percentiles (usec): | 1.00th=[ 1680], 5.00th=[ 1762], 10.00th=[ 1795], 20.00th=[ 1876], | 30.00th=[ 1958], 40.00th=[ 2040], 50.00th=[ 2147], 60.00th=[ 2311], | 70.00th=[ 2802], 80.00th=[ 4490], 90.00th=[ 7308], 95.00th=[ 9634], | 99.00th=[15664], 99.50th=[22152], 99.90th=[38011], 99.95th=[49021], | 99.99th=[49021] bw ( KiB/s): min= 55, max= 384, per=25.35%, avg=188.59, stdev=67.30, samples=63 iops : min= 13, max= 96, avg=47.06, stdev=16.84, samples=63 write: IOPS=49, BW=196KiB/s (201kB/s)(6260KiB/31907msec) clat (msec): min=8, max=192, avg=16.96, stdev= 7.99 lat (msec): min=8, max=192, avg=16.96, stdev= 7.99 clat percentiles (msec): | 1.00th=[ 10], 5.00th=[ 10], 10.00th=[ 11], 20.00th=[ 12], | 30.00th=[ 13], 40.00th=[ 15], 50.00th=[ 16], 60.00th=[ 17], | 70.00th=[ 18], 80.00th=[ 21], 90.00th=[ 27], 95.00th=[ 32], | 99.00th=[ 39], 99.50th=[ 43], 99.90th=[ 57], 99.95th=[ 192], | 99.99th=[ 192] bw ( KiB/s): min= 72, max= 264, per=25.04%, avg=195.06, stdev=63.00, samples=63 iops : min= 18, max= 66, avg=48.68, stdev=15.71, samples=63 lat (msec) : 2=17.35%, 4=20.83%, 10=11.69%, 20=39.71%, 50=10.32% lat (msec) : 100=0.07%, 250=0.03% cpu : usr=0.09%, sys=5.68%, ctx=3545, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8706: Thu Feb 27 13:23:30 2025 read: IOPS=47, BW=189KiB/s (194kB/s)(6052KiB/32013msec) clat (usec): min=1540, max=29821, avg=3284.74, stdev=2722.88 lat (usec): min=1540, max=29821, avg=3285.38, stdev=2722.91 clat percentiles (usec): | 1.00th=[ 1680], 5.00th=[ 1762], 10.00th=[ 1827], 20.00th=[ 1893], | 30.00th=[ 1958], 40.00th=[ 2024], 50.00th=[ 2114], 60.00th=[ 2278], | 70.00th=[ 2606], 80.00th=[ 4015], 90.00th=[ 6456], 95.00th=[ 9241], | 99.00th=[13829], 99.50th=[15664], 99.90th=[25035], 99.95th=[29754], | 99.99th=[29754] bw ( KiB/s): min= 47, max= 392, per=25.41%, avg=189.03, stdev=74.46, samples=64 iops : min= 11, max= 98, avg=47.19, stdev=18.69, samples=64 write: IOPS=48, BW=195KiB/s (199kB/s)(6236KiB/32013msec) clat (msec): min=8, max=178, avg=17.28, stdev= 7.98 lat (msec): min=8, max=178, avg=17.28, stdev= 7.98 clat percentiles (msec): | 1.00th=[ 9], 5.00th=[ 10], 10.00th=[ 11], 20.00th=[ 12], | 30.00th=[ 14], 40.00th=[ 15], 50.00th=[ 16], 60.00th=[ 18], | 70.00th=[ 19], 80.00th=[ 21], 90.00th=[ 27], 95.00th=[ 32], | 99.00th=[ 43], 99.50th=[ 47], 99.90th=[ 69], 99.95th=[ 180], | 99.99th=[ 180] bw ( KiB/s): min= 72, max= 264, per=24.99%, avg=194.66, stdev=59.51, samples=64 iops : min= 18, max= 66, avg=48.59, stdev=14.95, samples=64 lat (msec) : 2=17.90%, 4=21.48%, 10=10.77%, 20=38.74%, 50=10.97% lat (msec) : 100=0.10%, 250=0.03% cpu : usr=0.12%, sys=5.53%, ctx=3517, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=745KiB/s (763kB/s), 185KiB/s-189KiB/s (190kB/s-194kB/s), io=23.4MiB (24.6MB), run=31907-32234msec WRITE: bw=780KiB/s (799kB/s), 195KiB/s-198KiB/s (199kB/s-203kB/s), io=24.6MiB (25.7MB), run=31907-32234msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=8750: Thu Feb 27 13:23:33 2025 read: IOPS=151, BW=2428KiB/s (2486kB/s)(5920KiB/2438msec) clat (usec): min=55, max=21405, avg=2962.63, stdev=2769.08 lat (usec): min=56, max=21405, avg=2963.20, stdev=2769.11 clat percentiles (usec): | 1.00th=[ 61], 5.00th=[ 69], 10.00th=[ 72], 20.00th=[ 85], | 30.00th=[ 1106], 40.00th=[ 2343], 50.00th=[ 2769], 60.00th=[ 3195], | 70.00th=[ 3818], 80.00th=[ 4686], 90.00th=[ 6063], 95.00th=[ 7177], | 99.00th=[13435], 99.50th=[18220], 99.90th=[21365], 99.95th=[21365], | 99.99th=[21365] bw ( KiB/s): min= 1248, max= 3072, per=25.20%, avg=2408.00, stdev=839.50, samples=4 iops : min= 78, max= 192, avg=150.50, stdev=52.47, samples=4 write: IOPS=163, BW=2612KiB/s (2675kB/s)(6368KiB/2438msec) clat (usec): min=643, max=34710, avg=3179.83, stdev=3714.51 lat (usec): min=644, max=34711, avg=3180.98, stdev=3714.55 clat percentiles (usec): | 1.00th=[ 676], 5.00th=[ 734], 10.00th=[ 775], 20.00th=[ 832], | 30.00th=[ 963], 40.00th=[ 1172], 50.00th=[ 1516], 60.00th=[ 2769], | 70.00th=[ 3752], 80.00th=[ 4948], 90.00th=[ 6980], 95.00th=[ 8848], | 99.00th=[19268], 99.50th=[30016], 99.90th=[34866], 99.95th=[34866], | 99.99th=[34866] bw ( KiB/s): min= 1472, max= 4032, per=26.19%, avg=2632.00, stdev=1054.99, samples=4 iops : min= 92, max= 252, avg=164.50, stdev=65.94, samples=4 lat (usec) : 100=11.46%, 250=2.34%, 500=0.52%, 750=2.99%, 1000=13.80% lat (msec) : 2=11.59%, 4=29.56%, 10=25.13%, 20=2.08%, 50=0.52% cpu : usr=0.25%, sys=27.00%, ctx=1409, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8751: Thu Feb 27 13:23:33 2025 read: IOPS=147, BW=2359KiB/s (2416kB/s)(5744KiB/2435msec) clat (usec): min=57, max=15353, avg=2948.37, stdev=2303.45 lat (usec): min=58, max=15354, avg=2948.96, stdev=2303.42 clat percentiles (usec): | 1.00th=[ 60], 5.00th=[ 70], 10.00th=[ 74], 20.00th=[ 91], | 30.00th=[ 2089], 40.00th=[ 2540], 50.00th=[ 2900], 60.00th=[ 3261], | 70.00th=[ 3884], 80.00th=[ 4817], 90.00th=[ 5932], 95.00th=[ 6915], | 99.00th=[ 8848], 99.50th=[10159], 99.90th=[15401], 99.95th=[15401], | 99.99th=[15401] bw ( KiB/s): min= 1376, max= 3168, per=25.12%, avg=2400.00, stdev=762.65, samples=4 iops : min= 86, max= 198, avg=150.00, stdev=47.67, samples=4 write: IOPS=167, BW=2687KiB/s (2752kB/s)(6544KiB/2435msec) clat (usec): min=647, max=34374, avg=3140.73, stdev=4190.82 lat (usec): min=648, max=34375, avg=3141.58, stdev=4190.87 clat percentiles (usec): | 1.00th=[ 660], 5.00th=[ 709], 10.00th=[ 750], 20.00th=[ 816], | 30.00th=[ 873], 40.00th=[ 1012], 50.00th=[ 1385], 60.00th=[ 2573], | 70.00th=[ 3425], 80.00th=[ 4817], 90.00th=[ 6718], 95.00th=[ 8455], | 99.00th=[25560], 99.50th=[29230], 99.90th=[34341], 99.95th=[34341], | 99.99th=[34341] bw ( KiB/s): min= 1472, max= 3616, per=26.90%, avg=2704.00, stdev=928.37, samples=4 iops : min= 92, max= 226, avg=169.00, stdev=58.02, samples=4 lat (usec) : 100=10.29%, 250=2.08%, 500=0.26%, 750=5.47%, 1000=15.76% lat (msec) : 2=9.51%, 4=30.34%, 10=24.22%, 20=1.17%, 50=0.91% cpu : usr=0.12%, sys=27.49%, ctx=1323, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8752: Thu Feb 27 13:23:33 2025 read: IOPS=155, BW=2495KiB/s (2555kB/s)(6256KiB/2507msec) clat (usec): min=59, max=30460, avg=2959.95, stdev=3032.67 lat (usec): min=59, max=30460, avg=2961.31, stdev=3033.33 clat percentiles (usec): | 1.00th=[ 62], 5.00th=[ 70], 10.00th=[ 72], 20.00th=[ 83], | 30.00th=[ 449], 40.00th=[ 2409], 50.00th=[ 2769], 60.00th=[ 3195], | 70.00th=[ 3752], 80.00th=[ 4621], 90.00th=[ 5866], 95.00th=[ 7177], | 99.00th=[13435], 99.50th=[27395], 99.90th=[30540], 99.95th=[30540], | 99.99th=[30540] bw ( KiB/s): min= 1248, max= 3072, per=26.19%, avg=2502.40, stdev=720.32, samples=5 iops : min= 78, max= 192, avg=156.40, stdev=45.02, samples=5 write: IOPS=150, BW=2406KiB/s (2464kB/s)(6032KiB/2507msec) clat (usec): min=620, max=38902, avg=3361.97, stdev=3869.24 lat (usec): min=621, max=38905, avg=3362.84, stdev=3869.32 clat percentiles (usec): | 1.00th=[ 676], 5.00th=[ 725], 10.00th=[ 775], 20.00th=[ 832], | 30.00th=[ 914], 40.00th=[ 1139], 50.00th=[ 2024], 60.00th=[ 3130], | 70.00th=[ 4178], 80.00th=[ 5538], 90.00th=[ 7308], 95.00th=[ 9241], | 99.00th=[16909], 99.50th=[29754], 99.90th=[39060], 99.95th=[39060], | 99.99th=[39060] bw ( KiB/s): min= 1440, max= 2944, per=23.88%, avg=2400.00, stdev=571.09, samples=5 iops : min= 90, max= 184, avg=150.00, stdev=35.69, samples=5 lat (usec) : 100=13.54%, 250=1.17%, 500=0.65%, 750=3.39%, 1000=13.80% lat (msec) : 2=7.55%, 4=30.99%, 10=26.43%, 20=1.82%, 50=0.65% cpu : usr=0.16%, sys=26.30%, ctx=1399, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8753: Thu Feb 27 13:23:33 2025 read: IOPS=150, BW=2414KiB/s (2472kB/s)(6032KiB/2499msec) clat (usec): min=57, max=24188, avg=3033.86, stdev=2760.00 lat (usec): min=57, max=24189, avg=3034.47, stdev=2760.04 clat percentiles (usec): | 1.00th=[ 60], 5.00th=[ 69], 10.00th=[ 72], 20.00th=[ 83], | 30.00th=[ 1991], 40.00th=[ 2507], 50.00th=[ 2868], 60.00th=[ 3392], | 70.00th=[ 4080], 80.00th=[ 4883], 90.00th=[ 6128], 95.00th=[ 7373], | 99.00th=[10945], 99.50th=[19006], 99.90th=[24249], 99.95th=[24249], | 99.99th=[24249] bw ( KiB/s): min= 1600, max= 2944, per=24.79%, avg=2368.00, stdev=597.52, samples=4 iops : min= 100, max= 184, avg=148.00, stdev=37.35, samples=4 write: IOPS=156, BW=2503KiB/s (2563kB/s)(6256KiB/2499msec) clat (usec): min=622, max=44231, avg=3418.02, stdev=4455.77 lat (usec): min=623, max=44233, avg=3418.89, stdev=4455.88 clat percentiles (usec): | 1.00th=[ 676], 5.00th=[ 725], 10.00th=[ 775], 20.00th=[ 824], | 30.00th=[ 898], 40.00th=[ 1106], 50.00th=[ 1844], 60.00th=[ 2769], | 70.00th=[ 3884], 80.00th=[ 5342], 90.00th=[ 7570], 95.00th=[ 8979], | 99.00th=[27395], 99.50th=[31065], 99.90th=[44303], 99.95th=[44303], | 99.99th=[44303] bw ( KiB/s): min= 1824, max= 3296, per=25.39%, avg=2552.00, stdev=602.29, samples=4 iops : min= 114, max= 206, avg=159.50, stdev=37.64, samples=4 lat (usec) : 100=12.76%, 250=1.43%, 750=3.52%, 1000=15.36% lat (msec) : 2=7.55%, 4=29.04%, 10=27.60%, 20=1.82%, 50=0.91% cpu : usr=0.12%, sys=26.54%, ctx=1420, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=9554KiB/s (9783kB/s), 2359KiB/s-2495KiB/s (2416kB/s-2555kB/s), io=23.4MiB (24.5MB), run=2435-2507msec WRITE: bw=9.82MiB/s (10.3MB/s), 2406KiB/s-2687KiB/s (2464kB/s-2752kB/s), io=24.6MiB (25.8MB), run=2435-2507msec rand-rw: (groupid=0, jobs=1): err= 0: pid=8746: Thu Feb 27 13:23:39 2025 read: IOPS=45, BW=729KiB/s (746kB/s)(5920KiB/8122msec) clat (usec): min=1744, max=26937, avg=3689.38, stdev=3108.58 lat (usec): min=1745, max=26938, avg=3690.24, stdev=3108.49 clat percentiles (usec): | 1.00th=[ 1876], 5.00th=[ 1942], 10.00th=[ 2008], 20.00th=[ 2089], | 30.00th=[ 2147], 40.00th=[ 2245], 50.00th=[ 2376], 60.00th=[ 2540], | 70.00th=[ 3163], 80.00th=[ 4817], 90.00th=[ 6783], 95.00th=[10814], | 99.00th=[16712], 99.50th=[21365], 99.90th=[26870], 99.95th=[26870], | 99.99th=[26870] bw ( KiB/s): min= 416, max= 1216, per=24.83%, avg=717.50, stdev=237.25, samples=16 iops : min= 26, max= 76, avg=44.62, stdev=14.75, samples=16 write: IOPS=49, BW=784KiB/s (803kB/s)(6368KiB/8122msec) clat (usec): min=9217, max=44767, avg=16815.33, stdev=5824.91 lat (usec): min=9218, max=44767, avg=16816.69, stdev=5825.21 clat percentiles (usec): | 1.00th=[ 9503], 5.00th=[10290], 10.00th=[10945], 20.00th=[11994], | 30.00th=[12911], 40.00th=[14091], 50.00th=[15795], 60.00th=[17171], | 70.00th=[18482], 80.00th=[20055], 90.00th=[25035], 95.00th=[27919], | 99.00th=[35914], 99.50th=[42206], 99.90th=[44827], 99.95th=[44827], | 99.99th=[44827] bw ( KiB/s): min= 448, max= 1024, per=25.71%, avg=781.50, stdev=202.39, samples=16 iops : min= 28, max= 64, avg=48.62, stdev=12.52, samples=16 lat (msec) : 2=4.56%, 4=32.03%, 10=10.29%, 20=42.71%, 50=10.42% cpu : usr=0.09%, sys=6.50%, ctx=956, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8747: Thu Feb 27 13:23:39 2025 read: IOPS=43, BW=693KiB/s (710kB/s)(5744KiB/8287msec) clat (usec): min=1828, max=17416, avg=3621.05, stdev=2707.36 lat (usec): min=1828, max=17416, avg=3621.71, stdev=2707.38 clat percentiles (usec): | 1.00th=[ 1893], 5.00th=[ 1975], 10.00th=[ 2040], 20.00th=[ 2114], | 30.00th=[ 2180], 40.00th=[ 2278], 50.00th=[ 2409], 60.00th=[ 2671], | 70.00th=[ 3228], 80.00th=[ 4621], 90.00th=[ 6783], 95.00th=[ 9765], | 99.00th=[14615], 99.50th=[16057], 99.90th=[17433], 99.95th=[17433], | 99.99th=[17433] bw ( KiB/s): min= 288, max= 1056, per=23.88%, avg=690.00, stdev=228.67, samples=16 iops : min= 18, max= 66, avg=43.12, stdev=14.29, samples=16 write: IOPS=49, BW=790KiB/s (809kB/s)(6544KiB/8287msec) clat (usec): min=8878, max=50025, avg=16949.30, stdev=5652.84 lat (usec): min=8879, max=50027, avg=16950.47, stdev=5652.92 clat percentiles (usec): | 1.00th=[ 9634], 5.00th=[10814], 10.00th=[11207], 20.00th=[12518], | 30.00th=[13698], 40.00th=[14615], 50.00th=[16057], 60.00th=[17171], | 70.00th=[18482], 80.00th=[19792], 90.00th=[24249], 95.00th=[27657], | 99.00th=[35390], 99.50th=[41681], 99.90th=[50070], 99.95th=[50070], | 99.99th=[50070] bw ( KiB/s): min= 448, max= 992, per=25.66%, avg=780.00, stdev=191.24, samples=16 iops : min= 28, max= 62, avg=48.75, stdev=11.95, samples=16 lat (msec) : 2=2.73%, 4=32.94%, 10=10.68%, 20=44.14%, 50=9.37% lat (msec) : 100=0.13% cpu : usr=0.13%, sys=6.28%, ctx=913, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8748: Thu Feb 27 13:23:39 2025 read: IOPS=49, BW=795KiB/s (814kB/s)(6256KiB/7872msec) clat (usec): min=1775, max=19838, avg=3385.05, stdev=2584.80 lat (usec): min=1776, max=19838, avg=3385.90, stdev=2585.02 clat percentiles (usec): | 1.00th=[ 1795], 5.00th=[ 1958], 10.00th=[ 2008], 20.00th=[ 2073], | 30.00th=[ 2147], 40.00th=[ 2212], 50.00th=[ 2311], 60.00th=[ 2474], | 70.00th=[ 2835], 80.00th=[ 4080], 90.00th=[ 6325], 95.00th=[ 8094], | 99.00th=[15795], 99.50th=[18744], 99.90th=[19792], 99.95th=[19792], | 99.99th=[19792] bw ( KiB/s): min= 384, max= 1408, per=27.88%, avg=805.80, stdev=349.28, samples=15 iops : min= 24, max= 88, avg=50.13, stdev=21.69, samples=15 write: IOPS=47, BW=766KiB/s (785kB/s)(6032KiB/7872msec) clat (usec): min=9129, max=69817, avg=17177.51, stdev=6298.40 lat (usec): min=9130, max=69818, avg=17178.68, stdev=6298.46 clat percentiles (usec): | 1.00th=[ 9503], 5.00th=[10814], 10.00th=[11338], 20.00th=[12518], | 30.00th=[13435], 40.00th=[14353], 50.00th=[15795], 60.00th=[17171], | 70.00th=[18220], 80.00th=[20055], 90.00th=[25297], 95.00th=[30540], | 99.00th=[38011], 99.50th=[41681], 99.90th=[69731], 99.95th=[69731], | 99.99th=[69731] bw ( KiB/s): min= 416, max= 992, per=24.82%, avg=754.67, stdev=189.37, samples=15 iops : min= 26, max= 62, avg=46.93, stdev=11.73, samples=15 lat (msec) : 2=4.95%, 4=35.55%, 10=9.24%, 20=40.76%, 50=9.37% lat (msec) : 100=0.13% cpu : usr=0.10%, sys=6.59%, ctx=944, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8749: Thu Feb 27 13:23:39 2025 read: IOPS=47, BW=754KiB/s (772kB/s)(6032KiB/8004msec) clat (usec): min=1840, max=17597, avg=3490.25, stdev=2681.06 lat (usec): min=1840, max=17598, avg=3491.02, stdev=2681.01 clat percentiles (usec): | 1.00th=[ 1909], 5.00th=[ 1975], 10.00th=[ 2040], 20.00th=[ 2114], | 30.00th=[ 2180], 40.00th=[ 2278], 50.00th=[ 2376], 60.00th=[ 2540], | 70.00th=[ 2900], 80.00th=[ 4146], 90.00th=[ 6325], 95.00th=[10159], | 99.00th=[15139], 99.50th=[16712], 99.90th=[17695], 99.95th=[17695], | 99.99th=[17695] bw ( KiB/s): min= 351, max= 1248, per=25.95%, avg=749.94, stdev=248.24, samples=16 iops : min= 21, max= 78, avg=46.81, stdev=15.62, samples=16 write: IOPS=48, BW=782KiB/s (800kB/s)(6256KiB/8004msec) clat (usec): min=8697, max=50290, avg=17073.13, stdev=5700.47 lat (usec): min=8698, max=50291, avg=17074.33, stdev=5700.52 clat percentiles (usec): | 1.00th=[ 9634], 5.00th=[10421], 10.00th=[11076], 20.00th=[12125], | 30.00th=[13435], 40.00th=[15139], 50.00th=[16319], 60.00th=[17433], | 70.00th=[18482], 80.00th=[20055], 90.00th=[25035], 95.00th=[28181], | 99.00th=[35390], 99.50th=[45351], 99.90th=[50070], 99.95th=[50070], | 99.99th=[50070] bw ( KiB/s): min= 448, max= 1024, per=25.72%, avg=781.88, stdev=187.46, samples=16 iops : min= 28, max= 64, avg=48.81, stdev=11.78, samples=16 lat (msec) : 2=3.12%, 4=36.07%, 10=8.20%, 20=41.67%, 50=10.81% lat (msec) : 100=0.13% cpu : usr=0.11%, sys=6.55%, ctx=933, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2890KiB/s (2960kB/s), 693KiB/s-795KiB/s (710kB/s-814kB/s), io=23.4MiB (24.5MB), run=7872-8287msec WRITE: bw=3041KiB/s (3114kB/s), 766KiB/s-790KiB/s (785kB/s-809kB/s), io=24.6MiB (25.8MB), run=7872-8287msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=8765: Thu Feb 27 13:23:41 2025 read: IOPS=14, BW=14.1MiB/s (14.8MB/s)(7168KiB/496msec) clat (usec): min=11939, max=51414, avg=23704.96, stdev=14403.20 lat (usec): min=11940, max=51414, avg=23705.84, stdev=14403.13 clat percentiles (usec): | 1.00th=[11994], 5.00th=[11994], 10.00th=[11994], 20.00th=[15795], | 30.00th=[15795], 40.00th=[15795], 50.00th=[16712], 60.00th=[18744], | 70.00th=[18744], 80.00th=[35390], 90.00th=[51643], 95.00th=[51643], | 99.00th=[51643], 99.50th=[51643], 99.90th=[51643], 99.95th=[51643], | 99.99th=[51643] write: IOPS=10, BW=10.1MiB/s (10.6MB/s)(5120KiB/496msec) clat (msec): min=34, max=119, avg=55.17, stdev=36.25 lat (msec): min=35, max=119, avg=55.22, stdev=36.25 clat percentiles (msec): | 1.00th=[ 35], 5.00th=[ 35], 10.00th=[ 35], 20.00th=[ 35], | 30.00th=[ 37], 40.00th=[ 37], 50.00th=[ 41], 60.00th=[ 41], | 70.00th=[ 44], 80.00th=[ 44], 90.00th=[ 121], 95.00th=[ 121], | 99.00th=[ 121], 99.50th=[ 121], 99.90th=[ 121], 99.95th=[ 121], | 99.99th=[ 121] lat (msec) : 20=41.67%, 50=41.67%, 100=8.33%, 250=8.33% cpu : usr=0.00%, sys=13.74%, ctx=35, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8766: Thu Feb 27 13:23:41 2025 read: IOPS=2, BW=2723KiB/s (2789kB/s)(2048KiB/752msec) clat (usec): min=18383, max=25258, avg=21820.94, stdev=4861.38 lat (usec): min=18384, max=25259, avg=21821.95, stdev=4860.86 clat percentiles (usec): | 1.00th=[18482], 5.00th=[18482], 10.00th=[18482], 20.00th=[18482], | 30.00th=[18482], 40.00th=[18482], 50.00th=[18482], 60.00th=[25297], | 70.00th=[25297], 80.00th=[25297], 90.00th=[25297], 95.00th=[25297], | 99.00th=[25297], 99.50th=[25297], 99.90th=[25297], 99.95th=[25297], | 99.99th=[25297] bw ( KiB/s): min= 2043, max= 2043, per=8.16%, avg=2043.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=13, BW=13.3MiB/s (13.9MB/s)(10.0MiB/752msec) clat (msec): min=29, max=125, avg=63.13, stdev=33.63 lat (msec): min=29, max=125, avg=63.20, stdev=33.63 clat percentiles (msec): | 1.00th=[ 30], 5.00th=[ 30], 10.00th=[ 30], 20.00th=[ 36], | 30.00th=[ 38], 40.00th=[ 43], 50.00th=[ 44], 60.00th=[ 53], | 70.00th=[ 81], 80.00th=[ 81], 90.00th=[ 114], 95.00th=[ 126], | 99.00th=[ 126], 99.50th=[ 126], 99.90th=[ 126], 99.95th=[ 126], | 99.99th=[ 126] bw ( KiB/s): min=10219, max=10219, per=29.15%, avg=10219.00, stdev= 0.00, samples=1 iops : min= 9, max= 9, avg= 9.00, stdev= 0.00, samples=1 lat (msec) : 20=8.33%, 50=50.00%, 100=25.00%, 250=16.67% cpu : usr=0.00%, sys=14.78%, ctx=68, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8767: Thu Feb 27 13:23:41 2025 read: IOPS=8, BW=8428KiB/s (8630kB/s)(6144KiB/729msec) clat (usec): min=11084, max=87063, avg=43570.54, stdev=25429.44 lat (usec): min=11085, max=87064, avg=43571.48, stdev=25429.40 clat percentiles (usec): | 1.00th=[11076], 5.00th=[11076], 10.00th=[11076], 20.00th=[33424], | 30.00th=[33424], 40.00th=[36439], 50.00th=[36439], 60.00th=[39060], | 70.00th=[54264], 80.00th=[54264], 90.00th=[87557], 95.00th=[87557], | 99.00th=[87557], 99.50th=[87557], 99.90th=[87557], 99.95th=[87557], | 99.99th=[87557] bw ( KiB/s): min= 8175, max= 8175, per=32.65%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 7, max= 7, avg= 7.00, stdev= 0.00, samples=1 write: IOPS=8, BW=8428KiB/s (8630kB/s)(6144KiB/729msec) clat (msec): min=32, max=115, avg=65.25, stdev=33.73 lat (msec): min=32, max=115, avg=65.36, stdev=33.67 clat percentiles (msec): | 1.00th=[ 33], 5.00th=[ 33], 10.00th=[ 33], 20.00th=[ 36], | 30.00th=[ 36], 40.00th=[ 44], 50.00th=[ 44], 60.00th=[ 72], | 70.00th=[ 93], 80.00th=[ 93], 90.00th=[ 115], 95.00th=[ 115], | 99.00th=[ 115], 99.50th=[ 115], 99.90th=[ 115], 99.95th=[ 115], | 99.99th=[ 115] bw ( KiB/s): min= 8175, max= 8175, per=23.32%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 7, max= 7, avg= 7.00, stdev= 0.00, samples=1 lat (msec) : 20=8.33%, 50=50.00%, 100=33.33%, 250=8.33% cpu : usr=0.41%, sys=13.19%, ctx=63, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8768: Thu Feb 27 13:23:41 2025 read: IOPS=6, BW=6259KiB/s (6409kB/s)(5120KiB/818msec) clat (usec): min=23359, max=90791, avg=48628.15, stdev=26359.54 lat (usec): min=23360, max=90792, avg=48629.03, stdev=26359.66 clat percentiles (usec): | 1.00th=[23462], 5.00th=[23462], 10.00th=[23462], 20.00th=[23462], | 30.00th=[34341], 40.00th=[34341], 50.00th=[38536], 60.00th=[38536], | 70.00th=[56361], 80.00th=[56361], 90.00th=[90702], 95.00th=[90702], | 99.00th=[90702], 99.50th=[90702], 99.90th=[90702], 99.95th=[90702], | 99.99th=[90702] bw ( KiB/s): min= 8175, max= 8175, per=32.65%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 7, max= 7, avg= 7.00, stdev= 0.00, samples=1 write: IOPS=8, BW=8763KiB/s (8973kB/s)(7168KiB/818msec) clat (msec): min=38, max=144, avg=79.12, stdev=40.21 lat (msec): min=38, max=144, avg=79.20, stdev=40.21 clat percentiles (msec): | 1.00th=[ 39], 5.00th=[ 39], 10.00th=[ 39], 20.00th=[ 40], | 30.00th=[ 46], 40.00th=[ 46], 50.00th=[ 82], 60.00th=[ 101], | 70.00th=[ 101], 80.00th=[ 105], 90.00th=[ 146], 95.00th=[ 146], | 99.00th=[ 146], 99.50th=[ 146], 99.90th=[ 146], 99.95th=[ 146], | 99.99th=[ 146] bw ( KiB/s): min= 8175, max= 8175, per=23.32%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 7, max= 7, avg= 7.00, stdev= 0.00, samples=1 lat (msec) : 50=50.00%, 100=33.33%, 250=16.67% cpu : usr=0.37%, sys=8.94%, ctx=40, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=24.4MiB/s (25.6MB/s), 2723KiB/s-14.1MiB/s (2789kB/s-14.8MB/s), io=20.0MiB (20.0MB), run=496-818msec WRITE: bw=34.2MiB/s (35.9MB/s), 8428KiB/s-13.3MiB/s (8630kB/s-13.9MB/s), io=28.0MiB (29.4MB), run=496-818msec rand-rw: (groupid=0, jobs=1): err= 0: pid=8770: Thu Feb 27 13:23:41 2025 read: IOPS=8, BW=8605KiB/s (8812kB/s)(7168KiB/833msec) clat (msec): min=21, max=408, avg=87.53, stdev=141.84 lat (msec): min=21, max=408, avg=87.53, stdev=141.84 clat percentiles (msec): | 1.00th=[ 22], 5.00th=[ 22], 10.00th=[ 22], 20.00th=[ 24], | 30.00th=[ 37], 40.00th=[ 37], 50.00th=[ 39], 60.00th=[ 39], | 70.00th=[ 39], 80.00th=[ 46], 90.00th=[ 409], 95.00th=[ 409], | 99.00th=[ 409], 99.50th=[ 409], 99.90th=[ 409], 99.95th=[ 409], | 99.99th=[ 409] bw ( KiB/s): min= 4096, max= 4096, per=16.66%, avg=4096.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 write: IOPS=6, BW=6146KiB/s (6294kB/s)(5120KiB/833msec) clat (usec): min=20824, max=38479, avg=30946.49, stdev=7820.72 lat (usec): min=20842, max=38565, avg=30985.93, stdev=7834.16 clat percentiles (usec): | 1.00th=[20841], 5.00th=[20841], 10.00th=[20841], 20.00th=[20841], | 30.00th=[24773], 40.00th=[24773], 50.00th=[33162], 60.00th=[33162], | 70.00th=[37487], 80.00th=[37487], 90.00th=[38536], 95.00th=[38536], | 99.00th=[38536], 99.50th=[38536], 99.90th=[38536], 99.95th=[38536], | 99.99th=[38536] lat (msec) : 50=91.67%, 500=8.33% cpu : usr=0.00%, sys=25.48%, ctx=933, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8771: Thu Feb 27 13:23:41 2025 read: IOPS=3, BW=3215KiB/s (3292kB/s)(2048KiB/637msec) clat (usec): min=471, max=37234, avg=18853.07, stdev=25995.74 lat (usec): min=472, max=37236, avg=18854.49, stdev=25996.45 clat percentiles (usec): | 1.00th=[ 474], 5.00th=[ 474], 10.00th=[ 474], 20.00th=[ 474], | 30.00th=[ 474], 40.00th=[ 474], 50.00th=[ 474], 60.00th=[37487], | 70.00th=[37487], 80.00th=[37487], 90.00th=[37487], 95.00th=[37487], | 99.00th=[37487], 99.50th=[37487], 99.90th=[37487], 99.95th=[37487], | 99.99th=[37487] bw ( KiB/s): min= 4096, max= 4096, per=16.66%, avg=4096.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 write: IOPS=15, BW=15.7MiB/s (16.5MB/s)(10.0MiB/637msec) clat (msec): min=28, max=115, avg=58.73, stdev=33.65 lat (msec): min=28, max=115, avg=58.78, stdev=33.65 clat percentiles (msec): | 1.00th=[ 29], 5.00th=[ 29], 10.00th=[ 29], 20.00th=[ 34], | 30.00th=[ 35], 40.00th=[ 36], 50.00th=[ 40], 60.00th=[ 41], | 70.00th=[ 96], 80.00th=[ 96], 90.00th=[ 107], 95.00th=[ 115], | 99.00th=[ 115], 99.50th=[ 115], 99.90th=[ 115], 99.95th=[ 115], | 99.99th=[ 115] bw ( KiB/s): min=16384, max=16384, per=47.60%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 16, max= 16, avg=16.00, stdev= 0.00, samples=1 lat (usec) : 500=8.33% lat (msec) : 50=58.33%, 100=16.67%, 250=16.67% cpu : usr=0.00%, sys=33.02%, ctx=184, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8772: Thu Feb 27 13:23:41 2025 read: IOPS=10, BW=10.8MiB/s (11.3MB/s)(6144KiB/556msec) clat (usec): min=2509, max=77072, avg=29194.64, stdev=26314.86 lat (usec): min=2510, max=77073, avg=29195.43, stdev=26314.74 clat percentiles (usec): | 1.00th=[ 2507], 5.00th=[ 2507], 10.00th=[ 2507], 20.00th=[ 9634], | 30.00th=[ 9634], 40.00th=[23462], 50.00th=[23462], 60.00th=[26870], | 70.00th=[35390], 80.00th=[35390], 90.00th=[77071], 95.00th=[77071], | 99.00th=[77071], 99.50th=[77071], 99.90th=[77071], 99.95th=[77071], | 99.99th=[77071] bw ( KiB/s): min=10240, max=10240, per=41.65%, avg=10240.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 write: IOPS=10, BW=10.8MiB/s (11.3MB/s)(6144KiB/556msec) clat (usec): min=33047, max=89074, avg=53694.89, stdev=22309.08 lat (usec): min=33081, max=89146, avg=53741.77, stdev=22335.71 clat percentiles (usec): | 1.00th=[33162], 5.00th=[33162], 10.00th=[33162], 20.00th=[35390], | 30.00th=[35390], 40.00th=[37487], 50.00th=[37487], 60.00th=[61080], | 70.00th=[66323], 80.00th=[66323], 90.00th=[88605], 95.00th=[88605], | 99.00th=[88605], 99.50th=[88605], 99.90th=[88605], 99.95th=[88605], | 99.99th=[88605] bw ( KiB/s): min=12288, max=12288, per=35.70%, avg=12288.00, stdev= 0.00, samples=1 iops : min= 12, max= 12, avg=12.00, stdev= 0.00, samples=1 lat (msec) : 4=8.33%, 10=8.33%, 50=50.00%, 100=33.33% cpu : usr=0.00%, sys=25.41%, ctx=636, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8773: Thu Feb 27 13:23:41 2025 read: IOPS=6, BW=6799KiB/s (6963kB/s)(5120KiB/753msec) clat (usec): min=513, max=39177, avg=26701.77, stdev=15607.75 lat (usec): min=513, max=39178, avg=26702.68, stdev=15607.76 clat percentiles (usec): | 1.00th=[ 515], 5.00th=[ 515], 10.00th=[ 515], 20.00th=[ 515], | 30.00th=[24773], 40.00th=[24773], 50.00th=[32900], 60.00th=[32900], | 70.00th=[35914], 80.00th=[35914], 90.00th=[39060], 95.00th=[39060], | 99.00th=[39060], 99.50th=[39060], 99.90th=[39060], 99.95th=[39060], | 99.99th=[39060] bw ( KiB/s): min= 8192, max= 8192, per=33.32%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=9, BW=9519KiB/s (9748kB/s)(7168KiB/753msec) clat (msec): min=27, max=135, avg=83.06, stdev=41.57 lat (msec): min=27, max=135, avg=83.14, stdev=41.54 clat percentiles (msec): | 1.00th=[ 28], 5.00th=[ 28], 10.00th=[ 28], 20.00th=[ 38], | 30.00th=[ 72], 40.00th=[ 72], 50.00th=[ 81], 60.00th=[ 103], | 70.00th=[ 103], 80.00th=[ 128], 90.00th=[ 136], 95.00th=[ 136], | 99.00th=[ 136], 99.50th=[ 136], 99.90th=[ 136], 99.95th=[ 136], | 99.99th=[ 136] bw ( KiB/s): min= 8192, max= 8192, per=23.80%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 lat (usec) : 750=8.33% lat (msec) : 50=50.00%, 100=16.67%, 250=25.00% cpu : usr=0.00%, sys=26.46%, ctx=732, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=24.0MiB/s (25.2MB/s), 3215KiB/s-10.8MiB/s (3292kB/s-11.3MB/s), io=20.0MiB (20.0MB), run=556-833msec WRITE: bw=33.6MiB/s (35.2MB/s), 6146KiB/s-15.7MiB/s (6294kB/s-16.5MB/s), io=28.0MiB (29.4MB), run=556-833msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=8783: Thu Feb 27 13:23:43 2025 read: IOPS=7, BW=29.9MiB/s (31.3MB/s)(12.0MiB/402msec) clat (msec): min=56, max=148, avg=105.28, stdev=46.43 lat (msec): min=56, max=148, avg=105.28, stdev=46.43 clat percentiles (msec): | 1.00th=[ 57], 5.00th=[ 57], 10.00th=[ 57], 20.00th=[ 57], | 30.00th=[ 57], 40.00th=[ 111], 50.00th=[ 111], 60.00th=[ 111], | 70.00th=[ 148], 80.00th=[ 148], 90.00th=[ 148], 95.00th=[ 148], | 99.00th=[ 148], 99.50th=[ 148], 99.90th=[ 148], 99.95th=[ 148], | 99.99th=[ 148] lat (msec) : 100=33.33%, 250=66.67% cpu : usr=0.00%, sys=37.16%, ctx=1850, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8784: Thu Feb 27 13:23:43 2025 read: IOPS=1, BW=5689KiB/s (5825kB/s)(4096KiB/720msec) clat (nsec): min=129414k, max=129414k, avg=129414477.00, stdev= 0.00 lat (nsec): min=129416k, max=129416k, avg=129416282.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 130], 5.00th=[ 130], 10.00th=[ 130], 20.00th=[ 130], | 30.00th=[ 130], 40.00th=[ 130], 50.00th=[ 130], 60.00th=[ 130], | 70.00th=[ 130], 80.00th=[ 130], 90.00th=[ 130], 95.00th=[ 130], | 99.00th=[ 130], 99.50th=[ 130], 99.90th=[ 130], 99.95th=[ 130], | 99.99th=[ 130] bw ( KiB/s): min= 8159, max= 8159, per=35.86%, avg=8159.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=2, BW=11.1MiB/s (11.7MB/s)(8192KiB/720msec) clat (msec): min=200, max=370, avg=285.23, stdev=119.90 lat (msec): min=200, max=370, avg=285.50, stdev=119.85 clat percentiles (msec): | 1.00th=[ 201], 5.00th=[ 201], 10.00th=[ 201], 20.00th=[ 201], | 30.00th=[ 201], 40.00th=[ 201], 50.00th=[ 201], 60.00th=[ 372], | 70.00th=[ 372], 80.00th=[ 372], 90.00th=[ 372], 95.00th=[ 372], | 99.00th=[ 372], 99.50th=[ 372], 99.90th=[ 372], 99.95th=[ 372], | 99.99th=[ 372] lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.00%, sys=31.85%, ctx=668, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8785: Thu Feb 27 13:23:43 2025 write: IOPS=4, BW=18.8MiB/s (19.7MB/s)(12.0MiB/639msec) clat (msec): min=148, max=212, avg=190.78, stdev=36.57 lat (msec): min=148, max=213, avg=191.06, stdev=36.56 clat percentiles (msec): | 1.00th=[ 148], 5.00th=[ 148], 10.00th=[ 148], 20.00th=[ 148], | 30.00th=[ 148], 40.00th=[ 211], 50.00th=[ 211], 60.00th=[ 211], | 70.00th=[ 213], 80.00th=[ 213], 90.00th=[ 213], 95.00th=[ 213], | 99.00th=[ 213], 99.50th=[ 213], 99.90th=[ 213], 99.95th=[ 213], | 99.99th=[ 213] bw ( KiB/s): min=15906, max=15906, per=34.95%, avg=15906.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 lat (msec) : 250=100.00% cpu : usr=0.00%, sys=34.48%, ctx=57, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8786: Thu Feb 27 13:23:43 2025 write: IOPS=6, BW=24.8MiB/s (25.0MB/s)(12.0MiB/484msec) clat (msec): min=129, max=176, avg=152.65, stdev=23.98 lat (msec): min=129, max=177, avg=152.93, stdev=23.98 clat percentiles (msec): | 1.00th=[ 130], 5.00th=[ 130], 10.00th=[ 130], 20.00th=[ 130], | 30.00th=[ 130], 40.00th=[ 153], 50.00th=[ 153], 60.00th=[ 153], | 70.00th=[ 178], 80.00th=[ 178], 90.00th=[ 178], 95.00th=[ 178], | 99.00th=[ 178], 99.50th=[ 178], 99.90th=[ 178], 99.95th=[ 178], | 99.99th=[ 178] lat (msec) : 250=100.00% cpu : usr=0.00%, sys=51.76%, ctx=135, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=22.2MiB/s (23.3MB/s), 5689KiB/s-29.9MiB/s (5825kB/s-31.3MB/s), io=16.0MiB (16.8MB), run=402-720msec WRITE: bw=44.4MiB/s (46.6MB/s), 11.1MiB/s-24.8MiB/s (11.7MB/s-25.0MB/s), io=32.0MiB (33.6MB), run=484-720msec rand-rw: (groupid=0, jobs=1): err= 0: pid=8779: Thu Feb 27 13:23:43 2025 read: IOPS=8, BW=34.7MiB/s (36.4MB/s)(12.0MiB/346msec) clat (msec): min=27, max=239, avg=104.85, stdev=117.21 lat (msec): min=27, max=239, avg=104.85, stdev=117.21 clat percentiles (msec): | 1.00th=[ 28], 5.00th=[ 28], 10.00th=[ 28], 20.00th=[ 28], | 30.00th=[ 28], 40.00th=[ 47], 50.00th=[ 47], 60.00th=[ 47], | 70.00th=[ 241], 80.00th=[ 241], 90.00th=[ 241], 95.00th=[ 241], | 99.00th=[ 241], 99.50th=[ 241], 99.90th=[ 241], 99.95th=[ 241], | 99.99th=[ 241] lat (msec) : 50=66.67%, 250=33.33% cpu : usr=0.00%, sys=19.42%, ctx=49, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8780: Thu Feb 27 13:23:43 2025 read: IOPS=1, BW=5245KiB/s (5370kB/s)(4096KiB/781msec) clat (nsec): min=92711k, max=92711k, avg=92710684.00, stdev= 0.00 lat (nsec): min=92712k, max=92712k, avg=92711954.00, stdev= 0.00 clat percentiles (usec): | 1.00th=[92799], 5.00th=[92799], 10.00th=[92799], 20.00th=[92799], | 30.00th=[92799], 40.00th=[92799], 50.00th=[92799], 60.00th=[92799], | 70.00th=[92799], 80.00th=[92799], 90.00th=[92799], 95.00th=[92799], | 99.00th=[92799], 99.50th=[92799], 99.90th=[92799], 99.95th=[92799], | 99.99th=[92799] bw ( KiB/s): min= 8192, max= 8192, per=39.05%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=2, BW=10.2MiB/s (10.7MB/s)(8192KiB/781msec) clat (msec): min=212, max=454, avg=333.60, stdev=171.33 lat (msec): min=212, max=454, avg=333.81, stdev=171.23 clat percentiles (msec): | 1.00th=[ 213], 5.00th=[ 213], 10.00th=[ 213], 20.00th=[ 213], | 30.00th=[ 213], 40.00th=[ 213], 50.00th=[ 213], 60.00th=[ 456], | 70.00th=[ 456], 80.00th=[ 456], 90.00th=[ 456], 95.00th=[ 456], | 99.00th=[ 456], 99.50th=[ 456], 99.90th=[ 456], 99.95th=[ 456], | 99.99th=[ 456] lat (msec) : 100=33.33%, 250=33.33%, 500=33.33% cpu : usr=0.00%, sys=13.72%, ctx=173, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8781: Thu Feb 27 13:23:43 2025 write: IOPS=3, BW=15.5MiB/s (16.3MB/s)(12.0MiB/772msec) clat (msec): min=122, max=368, avg=245.18, stdev=122.78 lat (msec): min=122, max=368, avg=245.44, stdev=122.89 clat percentiles (msec): | 1.00th=[ 123], 5.00th=[ 123], 10.00th=[ 123], 20.00th=[ 123], | 30.00th=[ 123], 40.00th=[ 245], 50.00th=[ 245], 60.00th=[ 245], | 70.00th=[ 368], 80.00th=[ 368], 90.00th=[ 368], 95.00th=[ 368], | 99.00th=[ 368], 99.50th=[ 368], 99.90th=[ 368], 99.95th=[ 368], | 99.99th=[ 368] bw ( KiB/s): min= 8192, max= 8192, per=19.53%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.13%, sys=13.49%, ctx=156, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8782: Thu Feb 27 13:23:43 2025 write: IOPS=4, BW=18.6MiB/s (19.5MB/s)(12.0MiB/646msec) clat (msec): min=78, max=308, avg=199.72, stdev=115.24 lat (msec): min=79, max=308, avg=199.96, stdev=115.20 clat percentiles (msec): | 1.00th=[ 80], 5.00th=[ 80], 10.00th=[ 80], 20.00th=[ 80], | 30.00th=[ 80], 40.00th=[ 213], 50.00th=[ 213], 60.00th=[ 213], | 70.00th=[ 309], 80.00th=[ 309], 90.00th=[ 309], 95.00th=[ 309], | 99.00th=[ 309], 99.50th=[ 309], 99.90th=[ 309], 99.95th=[ 309], | 99.99th=[ 309] bw ( KiB/s): min= 8192, max= 8192, per=19.53%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 250=33.33%, 500=33.33% cpu : usr=0.31%, sys=20.31%, ctx=120, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=20.5MiB/s (21.5MB/s), 5245KiB/s-34.7MiB/s (5370kB/s-36.4MB/s), io=16.0MiB (16.8MB), run=346-781msec WRITE: bw=40.0MiB/s (42.0MB/s), 10.2MiB/s-18.6MiB/s (10.7MB/s-19.5MB/s), io=32.0MiB (33.6MB), run=646-781msec