== sanity test 398b: DIO and buffer IO race ============== 01:15:45 (1743743745) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.1296 s, 44.6 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=29731: Fri Apr 4 01:15:58 2025 read: IOPS=140, BW=563KiB/s (577kB/s)(5992KiB/10634msec) clat (usec): min=52, max=16168, avg=3803.35, stdev=2059.75 lat (usec): min=52, max=16169, avg=3804.61, stdev=2059.67 clat percentiles (usec): | 1.00th=[ 64], 5.00th=[ 81], 10.00th=[ 1876], 20.00th=[ 2245], | 30.00th=[ 2606], 40.00th=[ 3032], 50.00th=[ 3523], 60.00th=[ 4015], | 70.00th=[ 4621], 80.00th=[ 5276], 90.00th=[ 6325], 95.00th=[ 7242], | 99.00th=[10683], 99.50th=[12387], 99.90th=[15401], 99.95th=[16188], | 99.99th=[16188] bw ( KiB/s): min= 272, max= 760, per=25.19%, avg=560.57, stdev=93.07, samples=21 iops : min= 68, max= 190, avg=140.10, stdev=23.28, samples=21 write: IOPS=148, BW=592KiB/s (606kB/s)(6296KiB/10634msec) clat (usec): min=397, max=165857, avg=3088.62, stdev=4892.08 lat (usec): min=398, max=165858, avg=3089.67, stdev=4891.92 clat percentiles (usec): | 1.00th=[ 465], 5.00th=[ 494], 10.00th=[ 515], 20.00th=[ 545], | 30.00th=[ 693], 40.00th=[ 1713], 50.00th=[ 2442], 60.00th=[ 3097], | 70.00th=[ 3785], 80.00th=[ 5080], 90.00th=[ 6849], 95.00th=[ 8356], | 99.00th=[ 10945], 99.50th=[ 13042], 99.90th=[ 19530], 99.95th=[166724], | 99.99th=[166724] bw ( KiB/s): min= 352, max= 776, per=25.21%, avg=587.29, stdev=95.03, samples=21 iops : min= 88, max= 194, avg=146.76, stdev=23.77, samples=21 lat (usec) : 100=2.57%, 250=0.20%, 500=3.45%, 750=12.70%, 1000=2.54% lat (msec) : 2=6.80%, 4=38.15%, 10=32.06%, 20=1.50%, 250=0.03% cpu : usr=0.19%, sys=22.13%, ctx=5534, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29732: Fri Apr 4 01:15:58 2025 read: IOPS=139, BW=559KiB/s (573kB/s)(5940KiB/10617msec) clat (usec): min=51, max=17117, avg=3824.46, stdev=2125.04 lat (usec): min=51, max=17118, avg=3825.49, stdev=2125.21 clat percentiles (usec): | 1.00th=[ 65], 5.00th=[ 1532], 10.00th=[ 1876], 20.00th=[ 2212], | 30.00th=[ 2540], 40.00th=[ 2933], 50.00th=[ 3490], 60.00th=[ 4047], | 70.00th=[ 4621], 80.00th=[ 5211], 90.00th=[ 6325], 95.00th=[ 7439], | 99.00th=[11207], 99.50th=[12911], 99.90th=[16712], 99.95th=[17171], | 99.99th=[17171] bw ( KiB/s): min= 328, max= 704, per=25.06%, avg=557.57, stdev=81.50, samples=21 iops : min= 82, max= 176, avg=139.33, stdev=20.38, samples=21 write: IOPS=149, BW=598KiB/s (612kB/s)(6348KiB/10617msec) clat (usec): min=395, max=170750, avg=3066.18, stdev=4971.38 lat (usec): min=395, max=170750, avg=3066.84, stdev=4971.38 clat percentiles (usec): | 1.00th=[ 474], 5.00th=[ 498], 10.00th=[ 519], 20.00th=[ 562], | 30.00th=[ 685], 40.00th=[ 1582], 50.00th=[ 2442], 60.00th=[ 3032], | 70.00th=[ 3818], 80.00th=[ 4883], 90.00th=[ 6652], 95.00th=[ 8160], | 99.00th=[ 11600], 99.50th=[ 13566], 99.90th=[ 16450], 99.95th=[170918], | 99.99th=[170918] bw ( KiB/s): min= 368, max= 752, per=25.45%, avg=592.95, stdev=90.82, samples=21 iops : min= 92, max= 188, avg=148.19, stdev=22.69, samples=21 lat (usec) : 100=2.18%, 250=0.07%, 500=2.86%, 750=14.06%, 1000=1.95% lat (msec) : 2=6.74%, 4=37.99%, 10=32.32%, 20=1.79%, 250=0.03% cpu : usr=0.20%, sys=22.02%, ctx=5587, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29734: Fri Apr 4 01:15:58 2025 read: IOPS=141, BW=568KiB/s (581kB/s)(6028KiB/10619msec) clat (usec): min=52, max=167807, avg=3928.59, stdev=4710.28 lat (usec): min=52, max=167808, avg=3929.17, stdev=4710.29 clat percentiles (usec): | 1.00th=[ 64], 5.00th=[ 217], 10.00th=[ 1893], 20.00th=[ 2245], | 30.00th=[ 2573], 40.00th=[ 2999], 50.00th=[ 3490], 60.00th=[ 4015], | 70.00th=[ 4555], 80.00th=[ 5342], 90.00th=[ 6390], 95.00th=[ 7504], | 99.00th=[ 10814], 99.50th=[ 12387], 99.90th=[ 19006], 99.95th=[168821], | 99.99th=[168821] bw ( KiB/s): min= 336, max= 752, per=25.45%, avg=566.29, stdev=94.58, samples=21 iops : min= 84, max= 188, avg=141.52, stdev=23.65, samples=21 write: IOPS=147, BW=590KiB/s (604kB/s)(6260KiB/10619msec) clat (usec): min=393, max=19977, avg=2955.90, stdev=2674.75 lat (usec): min=393, max=19978, avg=2957.00, stdev=2674.58 clat percentiles (usec): | 1.00th=[ 461], 5.00th=[ 494], 10.00th=[ 515], 20.00th=[ 562], | 30.00th=[ 660], 40.00th=[ 1418], 50.00th=[ 2343], 60.00th=[ 2900], | 70.00th=[ 3752], 80.00th=[ 5080], 90.00th=[ 6849], 95.00th=[ 8356], | 99.00th=[11338], 99.50th=[11994], 99.90th=[13698], 99.95th=[20055], | 99.99th=[20055] bw ( KiB/s): min= 424, max= 768, per=25.16%, avg=586.14, stdev=84.25, samples=21 iops : min= 106, max= 192, avg=146.48, stdev=21.12, samples=21 lat (usec) : 100=2.41%, 250=0.07%, 500=3.19%, 750=13.83%, 1000=2.05% lat (msec) : 2=6.90%, 4=37.43%, 10=32.36%, 20=1.73%, 250=0.03% cpu : usr=0.26%, sys=22.17%, ctx=5520, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29736: Fri Apr 4 01:15:58 2025 read: IOPS=140, BW=561KiB/s (574kB/s)(6052KiB/10788msec) clat (usec): min=53, max=20057, avg=3893.34, stdev=2092.06 lat (usec): min=53, max=20058, avg=3893.98, stdev=2092.03 clat percentiles (usec): | 1.00th=[ 65], 5.00th=[ 1582], 10.00th=[ 1909], 20.00th=[ 2311], | 30.00th=[ 2671], 40.00th=[ 3064], 50.00th=[ 3589], 60.00th=[ 4113], | 70.00th=[ 4686], 80.00th=[ 5342], 90.00th=[ 6390], 95.00th=[ 7373], | 99.00th=[10814], 99.50th=[12125], 99.90th=[14877], 99.95th=[20055], | 99.99th=[20055] bw ( KiB/s): min= 280, max= 656, per=25.16%, avg=559.90, stdev=85.29, samples=21 iops : min= 70, max= 164, avg=139.95, stdev=21.30, samples=21 write: IOPS=144, BW=578KiB/s (592kB/s)(6236KiB/10788msec) clat (usec): min=438, max=161365, avg=3085.71, stdev=4825.23 lat (usec): min=438, max=161368, avg=3086.38, stdev=4825.25 clat percentiles (usec): | 1.00th=[ 474], 5.00th=[ 502], 10.00th=[ 523], 20.00th=[ 570], | 30.00th=[ 709], 40.00th=[ 1778], 50.00th=[ 2442], 60.00th=[ 2900], | 70.00th=[ 3687], 80.00th=[ 5014], 90.00th=[ 6849], 95.00th=[ 8586], | 99.00th=[ 11863], 99.50th=[ 12780], 99.90th=[ 16188], 99.95th=[160433], | 99.99th=[160433] bw ( KiB/s): min= 352, max= 736, per=24.77%, avg=577.05, stdev=81.18, samples=21 iops : min= 88, max= 184, avg=144.24, stdev=20.32, samples=21 lat (usec) : 100=2.15%, 250=0.23%, 500=2.38%, 750=13.44%, 1000=2.15% lat (msec) : 2=7.10%, 4=37.79%, 10=32.71%, 20=1.99%, 50=0.03% lat (msec) : 250=0.03% cpu : usr=0.19%, sys=21.81%, ctx=5692, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2226KiB/s (2279kB/s), 559KiB/s-568KiB/s (573kB/s-581kB/s), io=23.4MiB (24.6MB), run=10617-10788msec WRITE: bw=2330KiB/s (2386kB/s), 578KiB/s-598KiB/s (592kB/s-612kB/s), io=24.6MiB (25.7MB), run=10617-10788msec rand-rw: (groupid=0, jobs=1): err= 0: pid=29733: Fri Apr 4 01:16:05 2025 read: IOPS=85, BW=343KiB/s (351kB/s)(5992KiB/17475msec) clat (usec): min=1557, max=168170, avg=4698.60, stdev=4946.69 lat (usec): min=1558, max=168171, avg=4699.40, stdev=4946.69 clat percentiles (usec): | 1.00th=[ 1844], 5.00th=[ 2089], 10.00th=[ 2311], 20.00th=[ 2737], | 30.00th=[ 3261], 40.00th=[ 3687], 50.00th=[ 3916], 60.00th=[ 4113], | 70.00th=[ 4490], 80.00th=[ 5735], 90.00th=[ 8225], 95.00th=[ 10159], | 99.00th=[ 14484], 99.50th=[ 16319], 99.90th=[ 21627], 99.95th=[168821], | 99.99th=[168821] bw ( KiB/s): min= 151, max= 480, per=25.16%, avg=343.74, stdev=59.82, samples=34 iops : min= 37, max= 120, avg=85.91, stdev=15.03, samples=34 write: IOPS=90, BW=360KiB/s (369kB/s)(6296KiB/17475msec) clat (usec): min=2254, max=23555, avg=6570.13, stdev=2536.45 lat (usec): min=2254, max=23555, avg=6571.09, stdev=2536.45 clat percentiles (usec): | 1.00th=[ 2573], 5.00th=[ 3261], 10.00th=[ 3818], 20.00th=[ 4490], | 30.00th=[ 5276], 40.00th=[ 5800], 50.00th=[ 6390], 60.00th=[ 6718], | 70.00th=[ 7111], 80.00th=[ 7963], 90.00th=[ 9634], 95.00th=[11076], | 99.00th=[15926], 99.50th=[17695], 99.90th=[21890], 99.95th=[23462], | 99.99th=[23462] bw ( KiB/s): min= 175, max= 528, per=25.26%, avg=361.15, stdev=59.02, samples=34 iops : min= 43, max= 132, avg=90.26, stdev=14.83, samples=34 lat (msec) : 2=1.66%, 4=31.02%, 10=60.38%, 20=6.74%, 50=0.16% lat (msec) : 250=0.03% cpu : usr=0.29%, sys=12.17%, ctx=3541, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29735: Fri Apr 4 01:16:05 2025 read: IOPS=84, BW=338KiB/s (346kB/s)(5940KiB/17575msec) clat (usec): min=1592, max=176942, avg=4752.04, stdev=5189.02 lat (usec): min=1592, max=176943, avg=4752.89, stdev=5189.00 clat percentiles (usec): | 1.00th=[ 1893], 5.00th=[ 2114], 10.00th=[ 2376], 20.00th=[ 2737], | 30.00th=[ 3326], 40.00th=[ 3720], 50.00th=[ 3949], 60.00th=[ 4146], | 70.00th=[ 4555], 80.00th=[ 5735], 90.00th=[ 8160], 95.00th=[ 10290], | 99.00th=[ 14353], 99.50th=[ 16909], 99.90th=[ 26608], 99.95th=[177210], | 99.99th=[177210] bw ( KiB/s): min= 111, max= 496, per=24.99%, avg=341.38, stdev=66.65, samples=34 iops : min= 27, max= 124, avg=85.32, stdev=16.74, samples=34 write: IOPS=90, BW=361KiB/s (370kB/s)(6348KiB/17575msec) clat (usec): min=2317, max=25923, avg=6581.04, stdev=2657.41 lat (usec): min=2317, max=25924, avg=6582.04, stdev=2657.35 clat percentiles (usec): | 1.00th=[ 2737], 5.00th=[ 3326], 10.00th=[ 3720], 20.00th=[ 4490], | 30.00th=[ 5211], 40.00th=[ 5669], 50.00th=[ 6325], 60.00th=[ 6718], | 70.00th=[ 7046], 80.00th=[ 8094], 90.00th=[ 9896], 95.00th=[11469], | 99.00th=[16581], 99.50th=[18482], 99.90th=[23200], 99.95th=[25822], | 99.99th=[25822] bw ( KiB/s): min= 175, max= 520, per=25.16%, avg=359.74, stdev=54.70, samples=34 iops : min= 43, max= 130, avg=89.91, stdev=13.75, samples=34 lat (msec) : 2=1.33%, 4=31.74%, 10=59.60%, 20=7.10%, 50=0.20% lat (msec) : 250=0.03% cpu : usr=0.15%, sys=12.36%, ctx=3580, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29737: Fri Apr 4 01:16:05 2025 read: IOPS=86, BW=347KiB/s (355kB/s)(6028KiB/17387msec) clat (usec): min=1579, max=27532, avg=4501.45, stdev=2532.92 lat (usec): min=1579, max=27533, avg=4502.26, stdev=2532.91 clat percentiles (usec): | 1.00th=[ 1827], 5.00th=[ 2073], 10.00th=[ 2311], 20.00th=[ 2737], | 30.00th=[ 3228], 40.00th=[ 3687], 50.00th=[ 3884], 60.00th=[ 4113], | 70.00th=[ 4424], 80.00th=[ 5538], 90.00th=[ 7832], 95.00th=[ 9634], | 99.00th=[14091], 99.50th=[16450], 99.90th=[18220], 99.95th=[27657], | 99.99th=[27657] bw ( KiB/s): min= 127, max= 472, per=25.46%, avg=347.74, stdev=61.80, samples=34 iops : min= 31, max= 118, avg=86.91, stdev=15.53, samples=34 write: IOPS=90, BW=360KiB/s (369kB/s)(6260KiB/17387msec) clat (msec): min=2, max=180, avg= 6.73, stdev= 5.14 lat (msec): min=2, max=180, avg= 6.73, stdev= 5.14 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 4], 10.00th=[ 4], 20.00th=[ 5], | 30.00th=[ 6], 40.00th=[ 6], 50.00th=[ 7], 60.00th=[ 7], | 70.00th=[ 8], 80.00th=[ 8], 90.00th=[ 10], 95.00th=[ 12], | 99.00th=[ 18], 99.50th=[ 20], 99.90th=[ 27], 99.95th=[ 182], | 99.99th=[ 182] bw ( KiB/s): min= 175, max= 520, per=25.22%, avg=360.68, stdev=65.15, samples=34 iops : min= 43, max= 130, avg=90.15, stdev=16.35, samples=34 lat (msec) : 2=1.53%, 4=32.55%, 10=58.98%, 20=6.71%, 50=0.20% lat (msec) : 250=0.03% cpu : usr=0.22%, sys=12.43%, ctx=3537, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29738: Fri Apr 4 01:16:05 2025 read: IOPS=87, BW=351KiB/s (360kB/s)(6052KiB/17233msec) clat (usec): min=1648, max=19707, avg=4512.74, stdev=2518.18 lat (usec): min=1649, max=19707, avg=4513.69, stdev=2518.15 clat percentiles (usec): | 1.00th=[ 1811], 5.00th=[ 2089], 10.00th=[ 2311], 20.00th=[ 2638], | 30.00th=[ 3163], 40.00th=[ 3654], 50.00th=[ 3884], 60.00th=[ 4080], | 70.00th=[ 4424], 80.00th=[ 5604], 90.00th=[ 7898], 95.00th=[10028], | 99.00th=[13960], 99.50th=[15008], 99.90th=[18220], 99.95th=[19792], | 99.99th=[19792] bw ( KiB/s): min= 120, max= 488, per=25.69%, avg=350.91, stdev=72.14, samples=34 iops : min= 30, max= 122, avg=87.62, stdev=18.05, samples=34 write: IOPS=90, BW=362KiB/s (371kB/s)(6236KiB/17233msec) clat (msec): min=2, max=172, avg= 6.62, stdev= 4.89 lat (msec): min=2, max=172, avg= 6.62, stdev= 4.89 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 4], 10.00th=[ 4], 20.00th=[ 5], | 30.00th=[ 6], 40.00th=[ 6], 50.00th=[ 7], 60.00th=[ 7], | 70.00th=[ 8], 80.00th=[ 8], 90.00th=[ 10], 95.00th=[ 12], | 99.00th=[ 16], 99.50th=[ 17], 99.90th=[ 23], 99.95th=[ 174], | 99.99th=[ 174] bw ( KiB/s): min= 160, max= 528, per=25.35%, avg=362.44, stdev=59.53, samples=34 iops : min= 40, max= 132, avg=90.50, stdev=14.83, samples=34 lat (msec) : 2=1.76%, 4=32.75%, 10=58.89%, 20=6.48%, 50=0.10% lat (msec) : 250=0.03% cpu : usr=0.19%, sys=12.34%, ctx=3533, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1366KiB/s (1399kB/s), 338KiB/s-351KiB/s (346kB/s-360kB/s), io=23.4MiB (24.6MB), run=17233-17575msec WRITE: bw=1430KiB/s (1465kB/s), 360KiB/s-362KiB/s (369kB/s-371kB/s), io=24.6MiB (25.7MB), run=17233-17575msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=29765: Fri Apr 4 01:16:09 2025 read: IOPS=120, BW=1930KiB/s (1976kB/s)(5920KiB/3068msec) clat (usec): min=58, max=16735, avg=4294.31, stdev=2594.12 lat (usec): min=59, max=16736, avg=4295.33, stdev=2594.29 clat percentiles (usec): | 1.00th=[ 65], 5.00th=[ 74], 10.00th=[ 92], 20.00th=[ 2540], | 30.00th=[ 2966], 40.00th=[ 3490], 50.00th=[ 3982], 60.00th=[ 4686], | 70.00th=[ 5342], 80.00th=[ 5997], 90.00th=[ 7373], 95.00th=[ 8848], | 99.00th=[11994], 99.50th=[16188], 99.90th=[16712], 99.95th=[16712], | 99.99th=[16712] bw ( KiB/s): min= 800, max= 2304, per=24.45%, avg=1893.33, stdev=560.65, samples=6 iops : min= 50, max= 144, avg=118.33, stdev=35.04, samples=6 write: IOPS=129, BW=2076KiB/s (2125kB/s)(6368KiB/3068msec) clat (usec): min=659, max=25032, avg=3512.35, stdev=3461.79 lat (usec): min=660, max=25033, avg=3513.53, stdev=3461.75 clat percentiles (usec): | 1.00th=[ 685], 5.00th=[ 750], 10.00th=[ 783], 20.00th=[ 832], | 30.00th=[ 898], 40.00th=[ 1369], 50.00th=[ 2442], 60.00th=[ 3163], | 70.00th=[ 4080], 80.00th=[ 6128], 90.00th=[ 8094], 95.00th=[ 9896], | 99.00th=[15664], 99.50th=[21890], 99.90th=[25035], 99.95th=[25035], | 99.99th=[25035] bw ( KiB/s): min= 928, max= 2688, per=25.33%, avg=2064.00, stdev=657.91, samples=6 iops : min= 58, max= 168, avg=129.00, stdev=41.12, samples=6 lat (usec) : 100=4.95%, 250=0.52%, 750=2.60%, 1000=15.49% lat (msec) : 2=5.60%, 4=31.12%, 10=36.07%, 20=3.39%, 50=0.26% cpu : usr=0.03%, sys=23.83%, ctx=1423, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29766: Fri Apr 4 01:16:09 2025 read: IOPS=118, BW=1899KiB/s (1944kB/s)(5744KiB/3025msec) clat (usec): min=65, max=23009, avg=4445.36, stdev=3151.46 lat (usec): min=66, max=23010, avg=4446.21, stdev=3151.36 clat percentiles (usec): | 1.00th=[ 69], 5.00th=[ 74], 10.00th=[ 129], 20.00th=[ 2474], | 30.00th=[ 2835], 40.00th=[ 3326], 50.00th=[ 3916], 60.00th=[ 4490], | 70.00th=[ 5276], 80.00th=[ 6063], 90.00th=[ 8029], 95.00th=[10159], | 99.00th=[15664], 99.50th=[19792], 99.90th=[22938], 99.95th=[22938], | 99.99th=[22938] bw ( KiB/s): min= 1568, max= 2144, per=24.31%, avg=1882.67, stdev=200.78, samples=6 iops : min= 98, max= 134, avg=117.67, stdev=12.55, samples=6 write: IOPS=135, BW=2163KiB/s (2215kB/s)(6544KiB/3025msec) clat (usec): min=640, max=13400, avg=3414.75, stdev=2602.62 lat (usec): min=641, max=13401, avg=3419.47, stdev=2602.06 clat percentiles (usec): | 1.00th=[ 701], 5.00th=[ 766], 10.00th=[ 799], 20.00th=[ 881], | 30.00th=[ 1221], 40.00th=[ 2057], 50.00th=[ 2900], 60.00th=[ 3556], | 70.00th=[ 4424], 80.00th=[ 5538], 90.00th=[ 7046], 95.00th=[ 8717], | 99.00th=[10552], 99.50th=[11338], 99.90th=[13435], 99.95th=[13435], | 99.99th=[13435] bw ( KiB/s): min= 1792, max= 2752, per=26.38%, avg=2149.33, stdev=381.55, samples=6 iops : min= 112, max= 172, avg=134.33, stdev=23.85, samples=6 lat (usec) : 100=4.56%, 250=0.26%, 750=2.08%, 1000=11.59% lat (msec) : 2=7.81%, 4=32.94%, 10=37.37%, 20=3.26%, 50=0.13% cpu : usr=0.17%, sys=25.00%, ctx=1525, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29767: Fri Apr 4 01:16:09 2025 read: IOPS=128, BW=2061KiB/s (2110kB/s)(6256KiB/3036msec) clat (usec): min=57, max=13864, avg=4085.14, stdev=2413.77 lat (usec): min=58, max=13865, avg=4087.07, stdev=2415.38 clat percentiles (usec): | 1.00th=[ 69], 5.00th=[ 75], 10.00th=[ 93], 20.00th=[ 2507], | 30.00th=[ 2900], 40.00th=[ 3425], 50.00th=[ 3916], 60.00th=[ 4490], | 70.00th=[ 5276], 80.00th=[ 5866], 90.00th=[ 7046], 95.00th=[ 8160], | 99.00th=[12387], 99.50th=[13304], 99.90th=[13829], 99.95th=[13829], | 99.99th=[13829] bw ( KiB/s): min= 1472, max= 2720, per=26.79%, avg=2074.67, stdev=408.51, samples=6 iops : min= 92, max= 170, avg=129.67, stdev=25.53, samples=6 write: IOPS=124, BW=1987KiB/s (2035kB/s)(6032KiB/3036msec) clat (usec): min=650, max=16521, avg=3763.25, stdev=3287.48 lat (usec): min=650, max=16522, avg=3764.09, stdev=3287.52 clat percentiles (usec): | 1.00th=[ 676], 5.00th=[ 742], 10.00th=[ 783], 20.00th=[ 857], | 30.00th=[ 1188], 40.00th=[ 2114], 50.00th=[ 2835], 60.00th=[ 3392], | 70.00th=[ 4621], 80.00th=[ 6259], 90.00th=[ 8717], 95.00th=[10945], | 99.00th=[13829], 99.50th=[15139], 99.90th=[16581], 99.95th=[16581], | 99.99th=[16581] bw ( KiB/s): min= 1760, max= 2432, per=24.22%, avg=1973.33, stdev=262.06, samples=6 iops : min= 110, max= 152, avg=123.33, stdev=16.38, samples=6 lat (usec) : 100=5.60%, 250=0.78%, 500=0.26%, 750=2.60%, 1000=10.42% lat (msec) : 2=6.90%, 4=31.12%, 10=37.76%, 20=4.56% cpu : usr=0.10%, sys=24.42%, ctx=1429, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29768: Fri Apr 4 01:16:09 2025 read: IOPS=121, BW=1950KiB/s (1997kB/s)(6032KiB/3093msec) clat (usec): min=62, max=24071, avg=4280.88, stdev=2686.09 lat (usec): min=62, max=24072, avg=4281.45, stdev=2686.11 clat percentiles (usec): | 1.00th=[ 69], 5.00th=[ 77], 10.00th=[ 644], 20.00th=[ 2606], | 30.00th=[ 2999], 40.00th=[ 3556], 50.00th=[ 4015], 60.00th=[ 4686], | 70.00th=[ 5080], 80.00th=[ 5800], 90.00th=[ 7046], 95.00th=[ 8356], | 99.00th=[13435], 99.50th=[15926], 99.90th=[23987], 99.95th=[23987], | 99.99th=[23987] bw ( KiB/s): min= 1216, max= 2240, per=24.73%, avg=1914.67, stdev=367.33, samples=6 iops : min= 76, max= 140, avg=119.67, stdev=22.96, samples=6 write: IOPS=126, BW=2023KiB/s (2071kB/s)(6256KiB/3093msec) clat (usec): min=655, max=16477, avg=3550.48, stdev=3141.01 lat (usec): min=656, max=16479, avg=3551.32, stdev=3141.00 clat percentiles (usec): | 1.00th=[ 693], 5.00th=[ 766], 10.00th=[ 791], 20.00th=[ 848], | 30.00th=[ 971], 40.00th=[ 1680], 50.00th=[ 2737], 60.00th=[ 3458], | 70.00th=[ 4293], 80.00th=[ 6063], 90.00th=[ 8094], 95.00th=[ 9896], | 99.00th=[12911], 99.50th=[16188], 99.90th=[16450], 99.95th=[16450], | 99.99th=[16450] bw ( KiB/s): min= 1600, max= 2528, per=24.55%, avg=2000.00, stdev=373.59, samples=6 iops : min= 100, max= 158, avg=125.00, stdev=23.35, samples=6 lat (usec) : 100=4.43%, 250=0.26%, 750=2.47%, 1000=13.93% lat (msec) : 2=5.73%, 4=31.90%, 10=37.37%, 20=3.78%, 50=0.13% cpu : usr=0.16%, sys=23.87%, ctx=1542, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=7744KiB/s (7930kB/s), 1899KiB/s-2061KiB/s (1944kB/s-2110kB/s), io=23.4MiB (24.5MB), run=3025-3093msec WRITE: bw=8147KiB/s (8343kB/s), 1987KiB/s-2163KiB/s (2035kB/s-2215kB/s), io=24.6MiB (25.8MB), run=3025-3093msec rand-rw: (groupid=0, jobs=1): err= 0: pid=29769: Fri Apr 4 01:16:10 2025 read: IOPS=83, BW=1329KiB/s (1361kB/s)(5920KiB/4453msec) clat (usec): min=1938, max=21331, avg=4654.01, stdev=3008.87 lat (usec): min=1939, max=21332, avg=4654.80, stdev=3008.84 clat percentiles (usec): | 1.00th=[ 2089], 5.00th=[ 2311], 10.00th=[ 2442], 20.00th=[ 2704], | 30.00th=[ 2999], 40.00th=[ 3195], 50.00th=[ 3523], 60.00th=[ 4047], | 70.00th=[ 4555], 80.00th=[ 5473], 90.00th=[ 8979], 95.00th=[11731], | 99.00th=[16450], 99.50th=[18744], 99.90th=[21365], 99.95th=[21365], | 99.99th=[21365] bw ( KiB/s): min= 864, max= 2176, per=24.59%, avg=1308.00, stdev=432.19, samples=8 iops : min= 54, max= 136, avg=81.75, stdev=27.01, samples=8 write: IOPS=89, BW=1430KiB/s (1464kB/s)(6368KiB/4453msec) clat (usec): min=2838, max=29517, avg=6803.19, stdev=3549.76 lat (usec): min=2838, max=29518, avg=6804.54, stdev=3549.68 clat percentiles (usec): | 1.00th=[ 3032], 5.00th=[ 3392], 10.00th=[ 3589], 20.00th=[ 4015], | 30.00th=[ 4424], 40.00th=[ 5080], 50.00th=[ 5866], 60.00th=[ 6652], | 70.00th=[ 7373], 80.00th=[ 9241], 90.00th=[11338], 95.00th=[13173], | 99.00th=[20317], 99.50th=[22938], 99.90th=[29492], 99.95th=[29492], | 99.99th=[29492] bw ( KiB/s): min= 960, max= 2176, per=25.66%, avg=1436.00, stdev=467.32, samples=8 iops : min= 60, max= 136, avg=89.75, stdev=29.21, samples=8 lat (msec) : 2=0.26%, 4=38.41%, 10=49.35%, 20=11.33%, 50=0.65% cpu : usr=0.16%, sys=12.35%, ctx=934, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29770: Fri Apr 4 01:16:10 2025 read: IOPS=79, BW=1276KiB/s (1306kB/s)(5744KiB/4502msec) clat (usec): min=1838, max=17569, avg=4672.34, stdev=3044.96 lat (usec): min=1838, max=17570, avg=4673.01, stdev=3044.95 clat percentiles (usec): | 1.00th=[ 1958], 5.00th=[ 2212], 10.00th=[ 2376], 20.00th=[ 2573], | 30.00th=[ 2802], 40.00th=[ 3130], 50.00th=[ 3556], 60.00th=[ 4113], | 70.00th=[ 4752], 80.00th=[ 5997], 90.00th=[ 9241], 95.00th=[11600], | 99.00th=[16909], 99.50th=[16909], 99.90th=[17695], 99.95th=[17695], | 99.99th=[17695] bw ( KiB/s): min= 768, max= 2080, per=23.99%, avg=1276.44, stdev=400.46, samples=9 iops : min= 48, max= 130, avg=79.78, stdev=25.03, samples=9 write: IOPS=90, BW=1454KiB/s (1488kB/s)(6544KiB/4502msec) clat (usec): min=2568, max=29682, avg=6680.71, stdev=3430.35 lat (usec): min=2569, max=29683, avg=6682.14, stdev=3430.32 clat percentiles (usec): | 1.00th=[ 2966], 5.00th=[ 3490], 10.00th=[ 3621], 20.00th=[ 4113], | 30.00th=[ 4424], 40.00th=[ 4948], 50.00th=[ 5735], 60.00th=[ 6325], | 70.00th=[ 7373], 80.00th=[ 8356], 90.00th=[12256], 95.00th=[13698], | 99.00th=[17695], 99.50th=[19792], 99.90th=[29754], 99.95th=[29754], | 99.99th=[29754] bw ( KiB/s): min= 1088, max= 2144, per=25.92%, avg=1450.67, stdev=402.23, samples=9 iops : min= 68, max= 134, avg=90.67, stdev=25.14, samples=9 lat (msec) : 2=0.65%, 4=35.81%, 10=52.21%, 20=11.07%, 50=0.26% cpu : usr=0.31%, sys=12.11%, ctx=883, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29771: Fri Apr 4 01:16:10 2025 read: IOPS=88, BW=1414KiB/s (1448kB/s)(6256KiB/4425msec) clat (usec): min=1864, max=20779, avg=4729.82, stdev=2951.08 lat (usec): min=1864, max=20780, avg=4730.50, stdev=2951.07 clat percentiles (usec): | 1.00th=[ 2114], 5.00th=[ 2343], 10.00th=[ 2409], 20.00th=[ 2671], | 30.00th=[ 2933], 40.00th=[ 3228], 50.00th=[ 3687], 60.00th=[ 4146], | 70.00th=[ 4948], 80.00th=[ 6128], 90.00th=[ 8848], 95.00th=[11207], | 99.00th=[15664], 99.50th=[17957], 99.90th=[20841], 99.95th=[20841], | 99.99th=[20841] bw ( KiB/s): min= 832, max= 2208, per=26.84%, avg=1428.00, stdev=531.61, samples=8 iops : min= 52, max= 138, avg=89.25, stdev=33.23, samples=8 write: IOPS=85, BW=1363KiB/s (1396kB/s)(6032KiB/4425msec) clat (usec): min=2669, max=20684, avg=6668.31, stdev=3296.68 lat (usec): min=2669, max=20685, avg=6669.64, stdev=3296.73 clat percentiles (usec): | 1.00th=[ 2835], 5.00th=[ 3294], 10.00th=[ 3589], 20.00th=[ 4047], | 30.00th=[ 4359], 40.00th=[ 4817], 50.00th=[ 5735], 60.00th=[ 6783], | 70.00th=[ 7504], 80.00th=[ 8979], 90.00th=[10814], 95.00th=[13435], | 99.00th=[18220], 99.50th=[19792], 99.90th=[20579], 99.95th=[20579], | 99.99th=[20579] bw ( KiB/s): min= 928, max= 1984, per=24.30%, avg=1360.00, stdev=370.43, samples=8 iops : min= 58, max= 124, avg=85.00, stdev=23.15, samples=8 lat (msec) : 2=0.26%, 4=38.80%, 10=49.74%, 20=10.94%, 50=0.26% cpu : usr=0.18%, sys=12.23%, ctx=902, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29772: Fri Apr 4 01:16:10 2025 read: IOPS=85, BW=1363KiB/s (1396kB/s)(6032KiB/4425msec) clat (usec): min=1987, max=17200, avg=4552.15, stdev=2854.28 lat (usec): min=1987, max=17200, avg=4552.86, stdev=2854.25 clat percentiles (usec): | 1.00th=[ 2147], 5.00th=[ 2311], 10.00th=[ 2442], 20.00th=[ 2638], | 30.00th=[ 2868], 40.00th=[ 3032], 50.00th=[ 3425], 60.00th=[ 4015], | 70.00th=[ 4490], 80.00th=[ 5669], 90.00th=[ 8979], 95.00th=[11469], | 99.00th=[15795], 99.50th=[16909], 99.90th=[17171], 99.95th=[17171], | 99.99th=[17171] bw ( KiB/s): min= 832, max= 1984, per=25.49%, avg=1356.00, stdev=382.07, samples=8 iops : min= 52, max= 124, avg=84.75, stdev=23.88, samples=8 write: IOPS=88, BW=1414KiB/s (1448kB/s)(6256KiB/4425msec) clat (usec): min=2791, max=29453, avg=6724.30, stdev=3532.60 lat (usec): min=2792, max=29454, avg=6725.68, stdev=3532.55 clat percentiles (usec): | 1.00th=[ 2933], 5.00th=[ 3326], 10.00th=[ 3589], 20.00th=[ 4015], | 30.00th=[ 4359], 40.00th=[ 5014], 50.00th=[ 5669], 60.00th=[ 6652], | 70.00th=[ 7439], 80.00th=[ 8979], 90.00th=[11600], 95.00th=[13566], | 99.00th=[16712], 99.50th=[29492], 99.90th=[29492], 99.95th=[29492], | 99.99th=[29492] bw ( KiB/s): min= 928, max= 2368, per=25.51%, avg=1428.00, stdev=474.31, samples=8 iops : min= 58, max= 148, avg=89.25, stdev=29.64, samples=8 lat (msec) : 2=0.13%, 4=38.80%, 10=50.00%, 20=10.81%, 50=0.26% cpu : usr=0.07%, sys=12.50%, ctx=903, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=5320KiB/s (5448kB/s), 1276KiB/s-1414KiB/s (1306kB/s-1448kB/s), io=23.4MiB (24.5MB), run=4425-4502msec WRITE: bw=5598KiB/s (5732kB/s), 1363KiB/s-1454KiB/s (1396kB/s-1488kB/s), io=24.6MiB (25.8MB), run=4425-4502msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=29785: Fri Apr 4 01:16:12 2025 read: IOPS=16, BW=16.5MiB/s (17.4MB/s)(7168KiB/423msec) clat (usec): min=409, max=47867, avg=26494.72, stdev=14857.13 lat (usec): min=410, max=47868, avg=26495.49, stdev=14857.15 clat percentiles (usec): | 1.00th=[ 412], 5.00th=[ 412], 10.00th=[ 412], 20.00th=[21627], | 30.00th=[22414], 40.00th=[22414], 50.00th=[23200], 60.00th=[34866], | 70.00th=[34866], 80.00th=[34866], 90.00th=[47973], 95.00th=[47973], | 99.00th=[47973], 99.50th=[47973], 99.90th=[47973], 99.95th=[47973], | 99.99th=[47973] write: IOPS=11, BW=11.8MiB/s (12.4MB/s)(5120KiB/423msec) clat (usec): min=28697, max=75149, avg=46440.76, stdev=19035.91 lat (usec): min=28717, max=75214, avg=46481.65, stdev=19050.56 clat percentiles (usec): | 1.00th=[28705], 5.00th=[28705], 10.00th=[28705], 20.00th=[28705], | 30.00th=[32900], 40.00th=[32900], 50.00th=[39584], 60.00th=[39584], | 70.00th=[55837], 80.00th=[55837], 90.00th=[74974], 95.00th=[74974], | 99.00th=[74974], 99.50th=[74974], 99.90th=[74974], 99.95th=[74974], | 99.99th=[74974] lat (usec) : 500=8.33% lat (msec) : 50=75.00%, 100=16.67% cpu : usr=0.00%, sys=49.53%, ctx=821, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29786: Fri Apr 4 01:16:12 2025 read: IOPS=2, BW=2934KiB/s (3005kB/s)(2048KiB/698msec) clat (usec): min=24045, max=24783, avg=24414.41, stdev=522.24 lat (usec): min=24045, max=24784, avg=24415.27, stdev=522.54 clat percentiles (usec): | 1.00th=[23987], 5.00th=[23987], 10.00th=[23987], 20.00th=[23987], | 30.00th=[23987], 40.00th=[23987], 50.00th=[23987], 60.00th=[24773], | 70.00th=[24773], 80.00th=[24773], 90.00th=[24773], 95.00th=[24773], | 99.00th=[24773], 99.50th=[24773], 99.90th=[24773], 99.95th=[24773], | 99.99th=[24773] bw ( KiB/s): min= 3599, max= 3599, per=12.27%, avg=3599.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 write: IOPS=14, BW=14.3MiB/s (15.0MB/s)(10.0MiB/698msec) clat (msec): min=23, max=106, avg=57.06, stdev=26.64 lat (msec): min=23, max=106, avg=57.11, stdev=26.63 clat percentiles (msec): | 1.00th=[ 24], 5.00th=[ 24], 10.00th=[ 24], 20.00th=[ 32], | 30.00th=[ 34], 40.00th=[ 40], 50.00th=[ 46], 60.00th=[ 65], | 70.00th=[ 74], 80.00th=[ 74], 90.00th=[ 83], 95.00th=[ 107], | 99.00th=[ 107], 99.50th=[ 107], 99.90th=[ 107], 99.95th=[ 107], | 99.99th=[ 107] bw ( KiB/s): min=10797, max=10797, per=26.28%, avg=10797.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 lat (msec) : 50=58.33%, 100=33.33%, 250=8.33% cpu : usr=0.14%, sys=36.59%, ctx=89, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29787: Fri Apr 4 01:16:12 2025 read: IOPS=11, BW=11.7MiB/s (12.3MB/s)(6144KiB/512msec) clat (usec): min=280, max=49976, avg=13716.04, stdev=21539.44 lat (usec): min=281, max=49977, avg=13716.71, stdev=21539.38 clat percentiles (usec): | 1.00th=[ 281], 5.00th=[ 281], 10.00th=[ 281], 20.00th=[ 347], | 30.00th=[ 347], 40.00th=[ 416], 50.00th=[ 416], 60.00th=[ 453], | 70.00th=[30802], 80.00th=[30802], 90.00th=[50070], 95.00th=[50070], | 99.00th=[50070], 99.50th=[50070], 99.90th=[50070], 99.95th=[50070], | 99.99th=[50070] write: IOPS=11, BW=11.7MiB/s (12.3MB/s)(6144KiB/512msec) clat (usec): min=27934, max=94009, avg=58715.18, stdev=24401.73 lat (usec): min=27944, max=94040, avg=58751.69, stdev=24400.22 clat percentiles (usec): | 1.00th=[27919], 5.00th=[27919], 10.00th=[27919], 20.00th=[36963], | 30.00th=[36963], 40.00th=[52691], 50.00th=[52691], 60.00th=[69731], | 70.00th=[70779], 80.00th=[70779], 90.00th=[93848], 95.00th=[93848], | 99.00th=[93848], 99.50th=[93848], 99.90th=[93848], 99.95th=[93848], | 99.99th=[93848] lat (usec) : 500=33.33% lat (msec) : 50=33.33%, 100=33.33% cpu : usr=0.00%, sys=28.18%, ctx=101, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29788: Fri Apr 4 01:16:12 2025 read: IOPS=10, BW=10.3MiB/s (10.8MB/s)(5120KiB/484msec) clat (usec): min=328, max=35020, avg=13034.05, stdev=17460.14 lat (usec): min=328, max=35021, avg=13034.67, stdev=17460.05 clat percentiles (usec): | 1.00th=[ 330], 5.00th=[ 330], 10.00th=[ 330], 20.00th=[ 330], | 30.00th=[ 396], 40.00th=[ 396], 50.00th=[ 408], 60.00th=[ 408], | 70.00th=[28967], 80.00th=[28967], 90.00th=[34866], 95.00th=[34866], | 99.00th=[34866], 99.50th=[34866], 99.90th=[34866], 99.95th=[34866], | 99.99th=[34866] write: IOPS=14, BW=14.5MiB/s (15.2MB/s)(7168KiB/484msec) clat (usec): min=18732, max=90408, avg=49741.76, stdev=29169.83 lat (usec): min=19037, max=90440, avg=49837.02, stdev=29104.16 clat percentiles (usec): | 1.00th=[18744], 5.00th=[18744], 10.00th=[18744], 20.00th=[23200], | 30.00th=[32637], 40.00th=[32637], 50.00th=[38536], 60.00th=[58983], | 70.00th=[58983], 80.00th=[85459], 90.00th=[90702], 95.00th=[90702], | 99.00th=[90702], 99.50th=[90702], 99.90th=[90702], 99.95th=[90702], | 99.99th=[90702] lat (usec) : 500=25.00% lat (msec) : 20=8.33%, 50=41.67%, 100=25.00% cpu : usr=0.00%, sys=30.64%, ctx=63, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=28.7MiB/s (30.0MB/s), 2934KiB/s-16.5MiB/s (3005kB/s-17.4MB/s), io=20.0MiB (20.0MB), run=423-698msec WRITE: bw=40.1MiB/s (42.1MB/s), 11.7MiB/s-14.5MiB/s (12.3MB/s-15.2MB/s), io=28.0MiB (29.4MB), run=423-698msec rand-rw: (groupid=0, jobs=1): err= 0: pid=29781: Fri Apr 4 01:16:12 2025 read: IOPS=10, BW=10.0MiB/s (11.5MB/s)(7168KiB/638msec) clat (usec): min=15420, max=50933, avg=35448.99, stdev=14424.29 lat (usec): min=15420, max=50934, avg=35449.85, stdev=14424.33 clat percentiles (usec): | 1.00th=[15401], 5.00th=[15401], 10.00th=[15401], 20.00th=[18744], | 30.00th=[31589], 40.00th=[31589], 50.00th=[35914], 60.00th=[44827], | 70.00th=[44827], 80.00th=[50594], 90.00th=[51119], 95.00th=[51119], | 99.00th=[51119], 99.50th=[51119], 99.90th=[51119], 99.95th=[51119], | 99.99th=[51119] bw ( KiB/s): min=10219, max=10219, per=36.53%, avg=10219.00, stdev= 0.00, samples=1 iops : min= 9, max= 9, avg= 9.00, stdev= 0.00, samples=1 write: IOPS=7, BW=8025KiB/s (8218kB/s)(5120KiB/638msec) clat (msec): min=21, max=163, avg=71.80, stdev=53.61 lat (msec): min=21, max=163, avg=71.84, stdev=53.61 clat percentiles (msec): | 1.00th=[ 22], 5.00th=[ 22], 10.00th=[ 22], 20.00th=[ 22], | 30.00th=[ 57], 40.00th=[ 57], 50.00th=[ 58], 60.00th=[ 58], | 70.00th=[ 60], 80.00th=[ 60], 90.00th=[ 163], 95.00th=[ 163], | 99.00th=[ 163], 99.50th=[ 163], 99.90th=[ 163], 99.95th=[ 163], | 99.99th=[ 163] bw ( KiB/s): min= 4087, max= 4087, per=10.43%, avg=4087.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 lat (msec) : 20=16.67%, 50=33.33%, 100=41.67%, 250=8.33% cpu : usr=0.16%, sys=13.97%, ctx=71, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29782: Fri Apr 4 01:16:12 2025 read: IOPS=3, BW=3098KiB/s (3173kB/s)(2048KiB/661msec) clat (usec): min=17662, max=25324, avg=21493.63, stdev=5417.45 lat (usec): min=17664, max=25325, avg=21494.96, stdev=5416.79 clat percentiles (usec): | 1.00th=[17695], 5.00th=[17695], 10.00th=[17695], 20.00th=[17695], | 30.00th=[17695], 40.00th=[17695], 50.00th=[17695], 60.00th=[25297], | 70.00th=[25297], 80.00th=[25297], 90.00th=[25297], 95.00th=[25297], | 99.00th=[25297], 99.50th=[25297], 99.90th=[25297], 99.95th=[25297], | 99.99th=[25297] bw ( KiB/s): min= 4096, max= 4096, per=14.64%, avg=4096.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 write: IOPS=15, BW=15.1MiB/s (15.9MB/s)(10.0MiB/661msec) clat (msec): min=23, max=114, avg=56.01, stdev=30.22 lat (msec): min=23, max=114, avg=56.04, stdev=30.23 clat percentiles (msec): | 1.00th=[ 24], 5.00th=[ 24], 10.00th=[ 24], 20.00th=[ 32], | 30.00th=[ 36], 40.00th=[ 40], 50.00th=[ 42], 60.00th=[ 43], | 70.00th=[ 73], 80.00th=[ 73], 90.00th=[ 97], 95.00th=[ 115], | 99.00th=[ 115], 99.50th=[ 115], 99.90th=[ 115], 99.95th=[ 115], | 99.99th=[ 115] bw ( KiB/s): min=10240, max=10240, per=26.14%, avg=10240.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 lat (msec) : 20=8.33%, 50=58.33%, 100=25.00%, 250=8.33% cpu : usr=0.00%, sys=17.58%, ctx=77, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29783: Fri Apr 4 01:16:12 2025 read: IOPS=8, BW=8629KiB/s (8836kB/s)(6144KiB/712msec) clat (usec): min=26786, max=92471, avg=55547.96, stdev=22859.36 lat (usec): min=26786, max=92471, avg=55548.74, stdev=22859.34 clat percentiles (usec): | 1.00th=[26870], 5.00th=[26870], 10.00th=[26870], 20.00th=[41681], | 30.00th=[41681], 40.00th=[45876], 50.00th=[45876], 60.00th=[60031], | 70.00th=[66323], 80.00th=[66323], 90.00th=[92799], 95.00th=[92799], | 99.00th=[92799], 99.50th=[92799], 99.90th=[92799], 99.95th=[92799], | 99.99th=[92799] bw ( KiB/s): min= 4087, max= 4087, per=14.61%, avg=4087.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 write: IOPS=8, BW=8629KiB/s (8836kB/s)(6144KiB/712msec) clat (msec): min=16, max=122, avg=56.69, stdev=43.59 lat (msec): min=16, max=122, avg=56.76, stdev=43.61 clat percentiles (msec): | 1.00th=[ 17], 5.00th=[ 17], 10.00th=[ 17], 20.00th=[ 19], | 30.00th=[ 19], 40.00th=[ 26], 50.00th=[ 26], 60.00th=[ 78], | 70.00th=[ 82], 80.00th=[ 82], 90.00th=[ 124], 95.00th=[ 124], | 99.00th=[ 124], 99.50th=[ 124], 99.90th=[ 124], 99.95th=[ 124], | 99.99th=[ 124] bw ( KiB/s): min= 8175, max= 8175, per=20.87%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 7, max= 7, avg= 7.00, stdev= 0.00, samples=1 lat (msec) : 20=16.67%, 50=33.33%, 100=41.67%, 250=8.33% cpu : usr=0.00%, sys=15.19%, ctx=71, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29784: Fri Apr 4 01:16:12 2025 read: IOPS=6, BW=6995KiB/s (7162kB/s)(5120KiB/732msec) clat (usec): min=17862, max=91942, avg=55956.04, stdev=33234.76 lat (usec): min=17863, max=91942, avg=55956.79, stdev=33234.72 clat percentiles (usec): | 1.00th=[17957], 5.00th=[17957], 10.00th=[17957], 20.00th=[17957], | 30.00th=[24249], 40.00th=[24249], 50.00th=[66323], 60.00th=[66323], | 70.00th=[79168], 80.00th=[79168], 90.00th=[91751], 95.00th=[91751], | 99.00th=[91751], 99.50th=[91751], 99.90th=[91751], 99.95th=[91751], | 99.99th=[91751] bw ( KiB/s): min= 8175, max= 8175, per=29.22%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 7, max= 7, avg= 7.00, stdev= 0.00, samples=1 write: IOPS=9, BW=9792KiB/s (10.0MB/s)(7168KiB/732msec) clat (msec): min=20, max=133, avg=63.32, stdev=36.86 lat (msec): min=20, max=133, avg=63.36, stdev=36.86 clat percentiles (msec): | 1.00th=[ 21], 5.00th=[ 21], 10.00th=[ 21], 20.00th=[ 40], | 30.00th=[ 47], 40.00th=[ 47], 50.00th=[ 51], 60.00th=[ 72], | 70.00th=[ 72], 80.00th=[ 83], 90.00th=[ 133], 95.00th=[ 133], | 99.00th=[ 133], 99.50th=[ 133], 99.90th=[ 133], 99.95th=[ 133], | 99.99th=[ 133] bw ( KiB/s): min= 8175, max= 8175, per=20.87%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 7, max= 7, avg= 7.00, stdev= 0.00, samples=1 lat (msec) : 20=8.33%, 50=33.33%, 100=50.00%, 250=8.33% cpu : usr=0.00%, sys=13.27%, ctx=63, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=27.3MiB/s (28.6MB/s), 3098KiB/s-10.0MiB/s (3173kB/s-11.5MB/s), io=20.0MiB (20.0MB), run=638-732msec WRITE: bw=38.3MiB/s (40.1MB/s), 8025KiB/s-15.1MiB/s (8218kB/s-15.9MB/s), io=28.0MiB (29.4MB), run=638-732msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=29793: Fri Apr 4 01:16:14 2025 read: IOPS=4, BW=17.2MiB/s (18.0MB/s)(12.0MiB/698msec) clat (msec): min=76, max=358, avg=210.30, stdev=141.31 lat (msec): min=76, max=358, avg=210.30, stdev=141.31 clat percentiles (msec): | 1.00th=[ 78], 5.00th=[ 78], 10.00th=[ 78], 20.00th=[ 78], | 30.00th=[ 78], 40.00th=[ 197], 50.00th=[ 197], 60.00th=[ 197], | 70.00th=[ 359], 80.00th=[ 359], 90.00th=[ 359], 95.00th=[ 359], | 99.00th=[ 359], 99.50th=[ 359], 99.90th=[ 359], 99.95th=[ 359], | 99.99th=[ 359] bw ( KiB/s): min=16351, max=16351, per=69.66%, avg=16351.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 250=33.33%, 500=33.33% cpu : usr=0.00%, sys=11.19%, ctx=122, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29795: Fri Apr 4 01:16:14 2025 read: IOPS=4, BW=16.5MiB/s (17.3MB/s)(4096KiB/243msec) clat (nsec): min=57346k, max=57346k, avg=57345696.00, stdev= 0.00 lat (nsec): min=57348k, max=57348k, avg=57348349.00, stdev= 0.00 clat percentiles (usec): | 1.00th=[57410], 5.00th=[57410], 10.00th=[57410], 20.00th=[57410], | 30.00th=[57410], 40.00th=[57410], 50.00th=[57410], 60.00th=[57410], | 70.00th=[57410], 80.00th=[57410], 90.00th=[57410], 95.00th=[57410], | 99.00th=[57410], 99.50th=[57410], 99.90th=[57410], 99.95th=[57410], | 99.99th=[57410] write: IOPS=8, BW=32.9MiB/s (34.5MB/s)(8192KiB/243msec) clat (usec): min=43821, max=94462, avg=69142.19, stdev=35808.30 lat (usec): min=43906, max=94781, avg=69343.95, stdev=35973.91 clat percentiles (usec): | 1.00th=[43779], 5.00th=[43779], 10.00th=[43779], 20.00th=[43779], | 30.00th=[43779], 40.00th=[43779], 50.00th=[43779], 60.00th=[94897], | 70.00th=[94897], 80.00th=[94897], 90.00th=[94897], 95.00th=[94897], | 99.00th=[94897], 99.50th=[94897], 99.90th=[94897], 99.95th=[94897], | 99.99th=[94897] lat (msec) : 50=33.33%, 100=66.67% cpu : usr=0.00%, sys=22.31%, ctx=35, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29796: Fri Apr 4 01:16:14 2025 write: IOPS=6, BW=26.8MiB/s (28.1MB/s)(12.0MiB/447msec) clat (msec): min=43, max=314, avg=145.66, stdev=147.69 lat (msec): min=43, max=315, avg=145.89, stdev=147.73 clat percentiles (msec): | 1.00th=[ 44], 5.00th=[ 44], 10.00th=[ 44], 20.00th=[ 44], | 30.00th=[ 44], 40.00th=[ 80], 50.00th=[ 80], 60.00th=[ 80], | 70.00th=[ 317], 80.00th=[ 317], 90.00th=[ 317], 95.00th=[ 317], | 99.00th=[ 317], 99.50th=[ 317], 99.90th=[ 317], 99.95th=[ 317], | 99.99th=[ 317] lat (msec) : 50=33.33%, 100=33.33%, 500=33.33% cpu : usr=0.22%, sys=19.73%, ctx=48, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29797: Fri Apr 4 01:16:14 2025 write: IOPS=6, BW=26.5MiB/s (27.8MB/s)(12.0MiB/452msec) clat (msec): min=42, max=280, avg=132.26, stdev=129.43 lat (msec): min=42, max=280, avg=132.48, stdev=129.36 clat percentiles (msec): | 1.00th=[ 43], 5.00th=[ 43], 10.00th=[ 43], 20.00th=[ 43], | 30.00th=[ 43], 40.00th=[ 74], 50.00th=[ 74], 60.00th=[ 74], | 70.00th=[ 279], 80.00th=[ 279], 90.00th=[ 279], 95.00th=[ 279], | 99.00th=[ 279], 99.50th=[ 279], 99.90th=[ 279], 99.95th=[ 279], | 99.99th=[ 279] lat (msec) : 50=33.33%, 100=33.33%, 500=33.33% cpu : usr=0.00%, sys=15.30%, ctx=82, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=22.9MiB/s (24.0MB/s), 16.5MiB/s-17.2MiB/s (17.3MB/s-18.0MB/s), io=16.0MiB (16.8MB), run=243-698msec WRITE: bw=70.8MiB/s (74.2MB/s), 26.5MiB/s-32.9MiB/s (27.8MB/s-34.5MB/s), io=32.0MiB (33.6MB), run=243-452msec rand-rw: (groupid=0, jobs=1): err= 0: pid=29798: Fri Apr 4 01:16:14 2025 read: IOPS=6, BW=25.8MiB/s (27.0MB/s)(12.0MiB/466msec) clat (msec): min=101, max=165, avg=135.81, stdev=32.24 lat (msec): min=101, max=165, avg=135.81, stdev=32.24 clat percentiles (msec): | 1.00th=[ 102], 5.00th=[ 102], 10.00th=[ 102], 20.00th=[ 102], | 30.00th=[ 102], 40.00th=[ 140], 50.00th=[ 140], 60.00th=[ 140], | 70.00th=[ 165], 80.00th=[ 165], 90.00th=[ 165], 95.00th=[ 165], | 99.00th=[ 165], 99.50th=[ 165], 99.90th=[ 165], 99.95th=[ 165], | 99.99th=[ 165] lat (msec) : 250=100.00% cpu : usr=0.00%, sys=32.69%, ctx=2337, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29799: Fri Apr 4 01:16:14 2025 read: IOPS=1, BW=6244KiB/s (6394kB/s)(4096KiB/656msec) clat (nsec): min=142554k, max=142554k, avg=142554368.00, stdev= 0.00 lat (nsec): min=142555k, max=142555k, avg=142555468.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 142], 5.00th=[ 142], 10.00th=[ 142], 20.00th=[ 142], | 30.00th=[ 142], 40.00th=[ 142], 50.00th=[ 142], 60.00th=[ 142], | 70.00th=[ 142], 80.00th=[ 142], 90.00th=[ 142], 95.00th=[ 142], | 99.00th=[ 142], 99.50th=[ 142], 99.90th=[ 142], 99.95th=[ 142], | 99.99th=[ 142] bw ( KiB/s): min= 8192, max= 8192, per=32.80%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=3, BW=12.2MiB/s (12.8MB/s)(8192KiB/656msec) clat (msec): min=140, max=297, avg=219.39, stdev=111.02 lat (msec): min=140, max=298, avg=219.52, stdev=111.08 clat percentiles (msec): | 1.00th=[ 142], 5.00th=[ 142], 10.00th=[ 142], 20.00th=[ 142], | 30.00th=[ 142], 40.00th=[ 142], 50.00th=[ 142], 60.00th=[ 300], | 70.00th=[ 300], 80.00th=[ 300], 90.00th=[ 300], 95.00th=[ 300], | 99.00th=[ 300], 99.50th=[ 300], 99.90th=[ 300], 99.95th=[ 300], | 99.99th=[ 300] lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.15%, sys=32.98%, ctx=914, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29800: Fri Apr 4 01:16:14 2025 write: IOPS=3, BW=15.9MiB/s (16.7MB/s)(12.0MiB/755msec) clat (msec): min=172, max=319, avg=230.11, stdev=78.52 lat (msec): min=172, max=319, avg=230.36, stdev=78.54 clat percentiles (msec): | 1.00th=[ 174], 5.00th=[ 174], 10.00th=[ 174], 20.00th=[ 174], | 30.00th=[ 174], 40.00th=[ 199], 50.00th=[ 199], 60.00th=[ 199], | 70.00th=[ 321], 80.00th=[ 321], 90.00th=[ 321], 95.00th=[ 321], | 99.00th=[ 321], 99.50th=[ 321], 99.90th=[ 321], 99.95th=[ 321], | 99.99th=[ 321] bw ( KiB/s): min=16384, max=16384, per=37.75%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.00%, sys=38.73%, ctx=49, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29801: Fri Apr 4 01:16:14 2025 write: IOPS=5, BW=20.0MiB/s (20.0MB/s)(12.0MiB/600msec) clat (msec): min=91, max=244, avg=175.79, stdev=77.89 lat (msec): min=91, max=244, avg=176.02, stdev=77.89 clat percentiles (msec): | 1.00th=[ 91], 5.00th=[ 91], 10.00th=[ 91], 20.00th=[ 91], | 30.00th=[ 91], 40.00th=[ 192], 50.00th=[ 192], 60.00th=[ 192], | 70.00th=[ 245], 80.00th=[ 245], 90.00th=[ 245], 95.00th=[ 245], | 99.00th=[ 245], 99.50th=[ 245], 99.90th=[ 245], 99.95th=[ 245], | 99.99th=[ 245] bw ( KiB/s): min= 8192, max= 8192, per=18.88%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 250=66.67% cpu : usr=0.17%, sys=42.24%, ctx=36, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=24.4MiB/s (25.6MB/s), 6244KiB/s-25.8MiB/s (6394kB/s-27.0MB/s), io=16.0MiB (16.8MB), run=466-656msec WRITE: bw=42.4MiB/s (44.4MB/s), 12.2MiB/s-20.0MiB/s (12.8MB/s-20.0MB/s), io=32.0MiB (33.6MB), run=600-755msec