== sanity test 398b: DIO and buffer IO race ============== 23:17:25 (1708834645) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.03812 s, 48.5 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=24498: Sat Feb 24 23:17:40 2024 read: IOPS=117, BW=471KiB/s (482kB/s)(5992KiB/12725msec) clat (usec): min=42, max=155828, avg=3522.71, stdev=5289.64 lat (usec): min=42, max=155829, avg=3523.60, stdev=5289.91 clat percentiles (usec): | 1.00th=[ 47], 5.00th=[ 63], 10.00th=[ 68], 20.00th=[ 100], | 30.00th=[ 1926], 40.00th=[ 2278], 50.00th=[ 2671], 60.00th=[ 3163], | 70.00th=[ 4146], 80.00th=[ 5669], 90.00th=[ 7570], 95.00th=[ 9503], | 99.00th=[ 11994], 99.50th=[ 13829], 99.90th=[ 53740], 99.95th=[156238], | 99.99th=[156238] bw ( KiB/s): min= 288, max= 600, per=24.78%, avg=467.28, stdev=77.96, samples=25 iops : min= 72, max= 150, avg=116.72, stdev=19.52, samples=25 write: IOPS=123, BW=495KiB/s (507kB/s)(6296KiB/12725msec) clat (usec): min=370, max=32906, avg=4668.95, stdev=4455.39 lat (usec): min=371, max=32907, avg=4669.84, stdev=4455.36 clat percentiles (usec): | 1.00th=[ 445], 5.00th=[ 478], 10.00th=[ 502], 20.00th=[ 603], | 30.00th=[ 1106], 40.00th=[ 2540], 50.00th=[ 3752], 60.00th=[ 4817], | 70.00th=[ 6063], 80.00th=[ 7701], 90.00th=[10421], 95.00th=[13566], | 99.00th=[19268], 99.50th=[21890], 99.90th=[31327], 99.95th=[32900], | 99.99th=[32900] bw ( KiB/s): min= 360, max= 648, per=24.84%, avg=490.64, stdev=77.92, samples=25 iops : min= 90, max= 162, avg=122.56, stdev=19.49, samples=25 lat (usec) : 50=0.68%, 100=9.15%, 250=1.60%, 500=5.34%, 750=8.20% lat (usec) : 1000=1.73% lat (msec) : 2=7.81%, 4=25.94%, 10=32.03%, 20=6.97%, 50=0.46% lat (msec) : 100=0.07%, 250=0.03% cpu : usr=0.20%, sys=22.33%, ctx=5441, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24500: Sat Feb 24 23:17:40 2024 read: IOPS=117, BW=470KiB/s (481kB/s)(5940KiB/12636msec) clat (usec): min=39, max=27529, avg=3335.99, stdev=3016.89 lat (usec): min=39, max=27530, avg=3337.44, stdev=3016.30 clat percentiles (usec): | 1.00th=[ 47], 5.00th=[ 64], 10.00th=[ 69], 20.00th=[ 95], | 30.00th=[ 1991], 40.00th=[ 2343], 50.00th=[ 2704], 60.00th=[ 3228], | 70.00th=[ 4015], 80.00th=[ 5342], 90.00th=[ 7373], 95.00th=[ 9503], | 99.00th=[13042], 99.50th=[14746], 99.90th=[19792], 99.95th=[27657], | 99.99th=[27657] bw ( KiB/s): min= 328, max= 632, per=24.99%, avg=471.32, stdev=67.43, samples=25 iops : min= 82, max= 158, avg=117.80, stdev=16.87, samples=25 write: IOPS=125, BW=502KiB/s (514kB/s)(6348KiB/12636msec) clat (usec): min=424, max=147713, avg=4777.24, stdev=6146.70 lat (usec): min=424, max=147811, avg=4778.24, stdev=6148.11 clat percentiles (usec): | 1.00th=[ 449], 5.00th=[ 478], 10.00th=[ 502], 20.00th=[ 611], | 30.00th=[ 947], 40.00th=[ 2376], 50.00th=[ 3523], 60.00th=[ 4555], | 70.00th=[ 5997], 80.00th=[ 7504], 90.00th=[ 10552], 95.00th=[ 13304], | 99.00th=[ 22938], 99.50th=[ 30278], 99.90th=[ 51119], 99.95th=[147850], | 99.99th=[147850] bw ( KiB/s): min= 336, max= 688, per=25.42%, avg=502.00, stdev=86.94, samples=25 iops : min= 84, max= 172, avg=125.48, stdev=21.70, samples=25 lat (usec) : 50=0.78%, 100=9.28%, 250=1.43%, 500=5.05%, 750=8.69% lat (usec) : 1000=2.31% lat (msec) : 2=6.74%, 4=27.96%, 10=29.98%, 20=6.97%, 50=0.75% lat (msec) : 100=0.03%, 250=0.03% cpu : usr=0.25%, sys=22.26%, ctx=5388, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24501: Sat Feb 24 23:17:40 2024 read: IOPS=118, BW=475KiB/s (487kB/s)(6028KiB/12687msec) clat (usec): min=40, max=23536, avg=3382.40, stdev=3142.99 lat (usec): min=40, max=23537, avg=3383.36, stdev=3142.92 clat percentiles (usec): | 1.00th=[ 47], 5.00th=[ 63], 10.00th=[ 68], 20.00th=[ 95], | 30.00th=[ 1893], 40.00th=[ 2245], 50.00th=[ 2638], 60.00th=[ 3195], | 70.00th=[ 4178], 80.00th=[ 5604], 90.00th=[ 7832], 95.00th=[ 9503], | 99.00th=[13960], 99.50th=[14877], 99.90th=[22938], 99.95th=[23462], | 99.99th=[23462] bw ( KiB/s): min= 184, max= 616, per=25.09%, avg=473.24, stdev=96.27, samples=25 iops : min= 46, max= 154, avg=118.28, stdev=24.07, samples=25 write: IOPS=123, BW=493KiB/s (505kB/s)(6260KiB/12687msec) clat (usec): min=390, max=163808, avg=4783.49, stdev=6172.82 lat (usec): min=390, max=163810, avg=4784.81, stdev=6172.87 clat percentiles (usec): | 1.00th=[ 445], 5.00th=[ 474], 10.00th=[ 498], 20.00th=[ 603], | 30.00th=[ 1029], 40.00th=[ 2474], 50.00th=[ 3654], 60.00th=[ 4555], | 70.00th=[ 5997], 80.00th=[ 7767], 90.00th=[ 11076], 95.00th=[ 13435], | 99.00th=[ 19792], 99.50th=[ 25035], 99.90th=[ 53740], 99.95th=[164627], | 99.99th=[164627] bw ( KiB/s): min= 248, max= 648, per=24.85%, avg=490.84, stdev=87.18, samples=25 iops : min= 62, max= 162, avg=122.68, stdev=21.80, samples=25 lat (usec) : 50=1.04%, 100=9.11%, 250=1.95%, 500=5.47%, 750=8.04% lat (usec) : 1000=1.89% lat (msec) : 2=7.88%, 4=25.33%, 10=31.25%, 20=7.52%, 50=0.46% lat (msec) : 100=0.03%, 250=0.03% cpu : usr=0.25%, sys=22.47%, ctx=5369, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24502: Sat Feb 24 23:17:40 2024 read: IOPS=119, BW=480KiB/s (491kB/s)(6052KiB/12612msec) clat (usec): min=38, max=29905, avg=3431.71, stdev=3283.87 lat (usec): min=39, max=29905, avg=3433.15, stdev=3283.84 clat percentiles (usec): | 1.00th=[ 45], 5.00th=[ 60], 10.00th=[ 68], 20.00th=[ 92], | 30.00th=[ 1942], 40.00th=[ 2278], 50.00th=[ 2671], 60.00th=[ 3261], | 70.00th=[ 4146], 80.00th=[ 5669], 90.00th=[ 7701], 95.00th=[ 9241], | 99.00th=[13698], 99.50th=[20055], 99.90th=[23200], 99.95th=[30016], | 99.99th=[30016] bw ( KiB/s): min= 232, max= 608, per=25.39%, avg=478.80, stdev=79.08, samples=25 iops : min= 58, max= 152, avg=119.60, stdev=19.76, samples=25 write: IOPS=123, BW=494KiB/s (506kB/s)(6236KiB/12612msec) clat (usec): min=385, max=163441, avg=4692.93, stdev=6163.74 lat (usec): min=385, max=163443, avg=4694.71, stdev=6163.79 clat percentiles (usec): | 1.00th=[ 441], 5.00th=[ 474], 10.00th=[ 502], 20.00th=[ 619], | 30.00th=[ 971], 40.00th=[ 2245], 50.00th=[ 3425], 60.00th=[ 4424], | 70.00th=[ 5800], 80.00th=[ 7701], 90.00th=[ 10945], 95.00th=[ 13698], | 99.00th=[ 22152], 99.50th=[ 25035], 99.90th=[ 46924], 99.95th=[162530], | 99.99th=[162530] bw ( KiB/s): min= 296, max= 704, per=25.04%, avg=494.48, stdev=93.21, samples=25 iops : min= 74, max= 176, avg=123.52, stdev=23.34, samples=25 lat (usec) : 50=1.43%, 100=9.18%, 250=1.56%, 500=5.08%, 750=8.14% lat (usec) : 1000=2.41% lat (msec) : 2=7.45%, 4=26.73%, 10=30.11%, 20=7.06%, 50=0.81% lat (msec) : 250=0.03% cpu : usr=0.20%, sys=22.34%, ctx=5318, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1887KiB/s (1932kB/s), 470KiB/s-480KiB/s (481kB/s-491kB/s), io=23.4MiB (24.6MB), run=12612-12725msec WRITE: bw=1976KiB/s (2023kB/s), 493KiB/s-502KiB/s (505kB/s-514kB/s), io=24.6MiB (25.7MB), run=12612-12725msec rand-rw: (groupid=0, jobs=1): err= 0: pid=24495: Sat Feb 24 23:18:06 2024 read: IOPS=38, BW=155KiB/s (159kB/s)(5992KiB/38610msec) clat (usec): min=1483, max=161469, avg=3847.42, stdev=5535.65 lat (usec): min=1483, max=161471, avg=3848.51, stdev=5535.62 clat percentiles (usec): | 1.00th=[ 1680], 5.00th=[ 1778], 10.00th=[ 1876], 20.00th=[ 2008], | 30.00th=[ 2114], 40.00th=[ 2311], 50.00th=[ 2507], 60.00th=[ 2802], | 70.00th=[ 3097], 80.00th=[ 3752], 90.00th=[ 7832], 95.00th=[ 11207], | 99.00th=[ 20317], 99.50th=[ 29492], 99.90th=[ 42730], 99.95th=[160433], | 99.99th=[160433] bw ( KiB/s): min= 47, max= 304, per=24.95%, avg=154.94, stdev=64.06, samples=77 iops : min= 11, max= 76, avg=38.64, stdev=16.04, samples=77 write: IOPS=40, BW=163KiB/s (167kB/s)(6296KiB/38610msec) clat (usec): min=10486, max=80888, avg=20798.67, stdev=7964.22 lat (usec): min=10487, max=80889, avg=20799.92, stdev=7964.10 clat percentiles (usec): | 1.00th=[11469], 5.00th=[12387], 10.00th=[12911], 20.00th=[14091], | 30.00th=[15926], 40.00th=[17957], 50.00th=[19530], 60.00th=[20841], | 70.00th=[21890], 80.00th=[24249], 90.00th=[31851], 95.00th=[37487], | 99.00th=[47449], 99.50th=[50070], 99.90th=[68682], 99.95th=[81265], | 99.99th=[81265] bw ( KiB/s): min= 64, max= 232, per=24.99%, avg=162.68, stdev=49.30, samples=77 iops : min= 16, max= 58, avg=40.57, stdev=12.34, samples=77 lat (msec) : 2=9.67%, 4=30.08%, 10=6.12%, 20=29.82%, 50=24.02% lat (msec) : 100=0.26%, 250=0.03% cpu : usr=0.11%, sys=5.31%, ctx=3367, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24496: Sat Feb 24 23:18:06 2024 read: IOPS=38, BW=154KiB/s (158kB/s)(5940KiB/38488msec) clat (usec): min=1530, max=152839, avg=3662.97, stdev=4962.94 lat (usec): min=1531, max=152842, avg=3663.93, stdev=4962.97 clat percentiles (usec): | 1.00th=[ 1663], 5.00th=[ 1795], 10.00th=[ 1860], 20.00th=[ 1975], | 30.00th=[ 2114], 40.00th=[ 2278], 50.00th=[ 2507], 60.00th=[ 2769], | 70.00th=[ 3064], 80.00th=[ 3556], 90.00th=[ 7046], 95.00th=[ 10552], | 99.00th=[ 17695], 99.50th=[ 21627], 99.90th=[ 28181], 99.95th=[152044], | 99.99th=[152044] bw ( KiB/s): min= 16, max= 312, per=24.80%, avg=154.00, stdev=66.45, samples=76 iops : min= 4, max= 78, avg=38.34, stdev=16.64, samples=76 write: IOPS=41, BW=165KiB/s (169kB/s)(6348KiB/38488msec) clat (usec): min=10671, max=62518, avg=20756.73, stdev=8132.34 lat (usec): min=10672, max=62519, avg=20757.86, stdev=8132.28 clat percentiles (usec): | 1.00th=[11469], 5.00th=[12387], 10.00th=[13042], 20.00th=[13960], | 30.00th=[15533], 40.00th=[17433], 50.00th=[19006], 60.00th=[20579], | 70.00th=[22152], 80.00th=[24249], 90.00th=[32375], 95.00th=[38011], | 99.00th=[49546], 99.50th=[53216], 99.90th=[62129], 99.95th=[62653], | 99.99th=[62653] bw ( KiB/s): min= 80, max= 224, per=25.18%, avg=163.89, stdev=49.86, samples=76 iops : min= 20, max= 56, avg=40.82, stdev=12.45, samples=76 lat (msec) : 2=10.64%, 4=29.23%, 10=5.60%, 20=31.90%, 50=22.20% lat (msec) : 100=0.39%, 250=0.03% cpu : usr=0.10%, sys=5.25%, ctx=3340, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24497: Sat Feb 24 23:18:06 2024 read: IOPS=39, BW=159KiB/s (162kB/s)(6028KiB/38006msec) clat (usec): min=1526, max=36265, avg=3686.22, stdev=3284.59 lat (usec): min=1527, max=36265, avg=3687.14, stdev=3284.57 clat percentiles (usec): | 1.00th=[ 1680], 5.00th=[ 1795], 10.00th=[ 1893], 20.00th=[ 2024], | 30.00th=[ 2147], 40.00th=[ 2311], 50.00th=[ 2540], 60.00th=[ 2802], | 70.00th=[ 3097], 80.00th=[ 3982], 90.00th=[ 7570], 95.00th=[10421], | 99.00th=[16450], 99.50th=[19006], 99.90th=[32375], 99.95th=[36439], | 99.99th=[36439] bw ( KiB/s): min= 31, max= 288, per=25.51%, avg=158.42, stdev=58.20, samples=76 iops : min= 7, max= 72, avg=39.45, stdev=14.53, samples=76 write: IOPS=41, BW=165KiB/s (169kB/s)(6260KiB/38006msec) clat (msec): min=10, max=162, avg=20.67, stdev= 8.76 lat (msec): min=10, max=162, avg=20.67, stdev= 8.76 clat percentiles (msec): | 1.00th=[ 12], 5.00th=[ 13], 10.00th=[ 13], 20.00th=[ 15], | 30.00th=[ 16], 40.00th=[ 18], 50.00th=[ 20], 60.00th=[ 21], | 70.00th=[ 22], 80.00th=[ 25], 90.00th=[ 32], 95.00th=[ 39], | 99.00th=[ 49], 99.50th=[ 52], 99.90th=[ 62], 99.95th=[ 163], | 99.99th=[ 163] bw ( KiB/s): min= 72, max= 232, per=25.26%, avg=164.42, stdev=51.09, samples=76 iops : min= 18, max= 58, avg=40.95, stdev=12.77, samples=76 lat (msec) : 2=8.66%, 4=30.70%, 10=6.97%, 20=31.61%, 50=21.71% lat (msec) : 100=0.33%, 250=0.03% cpu : usr=0.11%, sys=5.30%, ctx=3344, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24499: Sat Feb 24 23:18:06 2024 read: IOPS=39, BW=158KiB/s (162kB/s)(6052KiB/38216msec) clat (usec): min=1453, max=42870, avg=3718.72, stdev=3540.40 lat (usec): min=1454, max=42871, avg=3719.71, stdev=3540.41 clat percentiles (usec): | 1.00th=[ 1680], 5.00th=[ 1795], 10.00th=[ 1876], 20.00th=[ 2008], | 30.00th=[ 2147], 40.00th=[ 2311], 50.00th=[ 2507], 60.00th=[ 2802], | 70.00th=[ 3163], 80.00th=[ 3752], 90.00th=[ 7832], 95.00th=[10421], | 99.00th=[18482], 99.50th=[22676], 99.90th=[36963], 99.95th=[42730], | 99.99th=[42730] bw ( KiB/s): min= 16, max= 304, per=25.47%, avg=158.20, stdev=64.08, samples=76 iops : min= 4, max= 76, avg=39.46, stdev=16.03, samples=76 write: IOPS=40, BW=163KiB/s (167kB/s)(6236KiB/38216msec) clat (msec): min=10, max=162, avg=20.84, stdev= 8.72 lat (msec): min=10, max=162, avg=20.84, stdev= 8.72 clat percentiles (msec): | 1.00th=[ 12], 5.00th=[ 13], 10.00th=[ 13], 20.00th=[ 15], | 30.00th=[ 16], 40.00th=[ 18], 50.00th=[ 20], 60.00th=[ 21], | 70.00th=[ 22], 80.00th=[ 25], 90.00th=[ 33], 95.00th=[ 39], | 99.00th=[ 48], 99.50th=[ 52], 99.90th=[ 71], 99.95th=[ 163], | 99.99th=[ 163] bw ( KiB/s): min= 64, max= 224, per=25.01%, avg=162.83, stdev=49.06, samples=76 iops : min= 16, max= 56, avg=40.62, stdev=12.29, samples=76 lat (msec) : 2=9.15%, 4=30.89%, 10=6.48%, 20=30.34%, 50=22.75% lat (msec) : 100=0.36%, 250=0.03% cpu : usr=0.14%, sys=5.21%, ctx=3354, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=622KiB/s (637kB/s), 154KiB/s-159KiB/s (158kB/s-162kB/s), io=23.4MiB (24.6MB), run=38006-38610msec WRITE: bw=651KiB/s (667kB/s), 163KiB/s-165KiB/s (167kB/s-169kB/s), io=24.6MiB (25.7MB), run=38006-38610msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=24548: Sat Feb 24 23:18:10 2024 read: IOPS=144, BW=2304KiB/s (2360kB/s)(5920KiB/2569msec) clat (usec): min=47, max=30722, avg=3312.88, stdev=3185.28 lat (usec): min=48, max=30723, avg=3313.55, stdev=3185.30 clat percentiles (usec): | 1.00th=[ 52], 5.00th=[ 67], 10.00th=[ 71], 20.00th=[ 85], | 30.00th=[ 2311], 40.00th=[ 2704], 50.00th=[ 2999], 60.00th=[ 3458], | 70.00th=[ 4047], 80.00th=[ 4817], 90.00th=[ 6128], 95.00th=[ 8291], | 99.00th=[16450], 99.50th=[19530], 99.90th=[30802], 99.95th=[30802], | 99.99th=[30802] bw ( KiB/s): min= 1724, max= 2880, per=25.02%, avg=2277.60, stdev=457.92, samples=5 iops : min= 107, max= 180, avg=142.20, stdev=28.85, samples=5 write: IOPS=154, BW=2479KiB/s (2538kB/s)(6368KiB/2569msec) clat (usec): min=594, max=28191, avg=3357.77, stdev=3815.21 lat (usec): min=595, max=28192, avg=3358.89, stdev=3815.93 clat percentiles (usec): | 1.00th=[ 619], 5.00th=[ 660], 10.00th=[ 701], 20.00th=[ 758], | 30.00th=[ 848], 40.00th=[ 1037], 50.00th=[ 2147], 60.00th=[ 3064], | 70.00th=[ 4113], 80.00th=[ 5538], 90.00th=[ 7373], 95.00th=[ 9634], | 99.00th=[21627], 99.50th=[27395], 99.90th=[28181], 99.95th=[28181], | 99.99th=[28181] bw ( KiB/s): min= 1852, max= 3136, per=25.85%, avg=2476.00, stdev=519.46, samples=5 iops : min= 115, max= 196, avg=154.60, stdev=32.69, samples=5 lat (usec) : 50=0.26%, 100=10.55%, 250=1.69%, 500=0.26%, 750=9.24% lat (usec) : 1000=10.68% lat (msec) : 2=5.86%, 4=29.69%, 10=28.39%, 20=2.73%, 50=0.65% cpu : usr=0.35%, sys=28.27%, ctx=1473, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24549: Sat Feb 24 23:18:10 2024 read: IOPS=136, BW=2183KiB/s (2236kB/s)(5744KiB/2631msec) clat (usec): min=48, max=10930, avg=2887.56, stdev=2359.64 lat (usec): min=49, max=10931, avg=2888.27, stdev=2359.73 clat percentiles (usec): | 1.00th=[ 50], 5.00th=[ 64], 10.00th=[ 70], 20.00th=[ 77], | 30.00th=[ 519], 40.00th=[ 2507], 50.00th=[ 2933], 60.00th=[ 3261], | 70.00th=[ 3851], 80.00th=[ 4752], 90.00th=[ 6194], 95.00th=[ 7111], | 99.00th=[ 8848], 99.50th=[10028], 99.90th=[10945], 99.95th=[10945], | 99.99th=[10945] bw ( KiB/s): min= 1373, max= 2752, per=23.55%, avg=2143.40, stdev=536.06, samples=5 iops : min= 85, max= 172, avg=133.80, stdev=33.80, samples=5 write: IOPS=155, BW=2487KiB/s (2547kB/s)(6544KiB/2631msec) clat (usec): min=603, max=44247, avg=3876.76, stdev=5412.54 lat (usec): min=603, max=44248, avg=3877.74, stdev=5412.64 clat percentiles (usec): | 1.00th=[ 627], 5.00th=[ 668], 10.00th=[ 709], 20.00th=[ 775], | 30.00th=[ 873], 40.00th=[ 1090], 50.00th=[ 2245], 60.00th=[ 2966], | 70.00th=[ 4047], 80.00th=[ 5866], 90.00th=[ 8586], 95.00th=[11731], | 99.00th=[32637], 99.50th=[33817], 99.90th=[44303], 99.95th=[44303], | 99.99th=[44303] bw ( KiB/s): min= 1469, max= 3040, per=25.25%, avg=2418.60, stdev=576.61, samples=5 iops : min= 91, max= 190, avg=151.00, stdev=36.37, samples=5 lat (usec) : 50=0.39%, 100=11.33%, 250=2.21%, 750=9.24%, 1000=10.94% lat (msec) : 2=6.77%, 4=29.56%, 10=25.78%, 20=2.60%, 50=1.17% cpu : usr=0.27%, sys=26.58%, ctx=1480, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24550: Sat Feb 24 23:18:10 2024 read: IOPS=154, BW=2476KiB/s (2535kB/s)(6256KiB/2527msec) clat (usec): min=46, max=27603, avg=3134.08, stdev=3031.23 lat (usec): min=46, max=27603, avg=3135.23, stdev=3031.35 clat percentiles (usec): | 1.00th=[ 50], 5.00th=[ 66], 10.00th=[ 72], 20.00th=[ 86], | 30.00th=[ 2147], 40.00th=[ 2540], 50.00th=[ 2900], 60.00th=[ 3228], | 70.00th=[ 4015], 80.00th=[ 4948], 90.00th=[ 6259], 95.00th=[ 7701], | 99.00th=[15139], 99.50th=[23725], 99.90th=[27657], 99.95th=[27657], | 99.99th=[27657] bw ( KiB/s): min= 1724, max= 2752, per=27.48%, avg=2501.60, stdev=436.09, samples=5 iops : min= 107, max= 172, avg=156.20, stdev=27.59, samples=5 write: IOPS=149, BW=2387KiB/s (2444kB/s)(6032KiB/2527msec) clat (usec): min=595, max=56895, avg=3430.95, stdev=4968.71 lat (usec): min=596, max=56897, avg=3432.08, stdev=4968.67 clat percentiles (usec): | 1.00th=[ 611], 5.00th=[ 660], 10.00th=[ 709], 20.00th=[ 766], | 30.00th=[ 865], 40.00th=[ 1074], 50.00th=[ 1598], 60.00th=[ 2802], | 70.00th=[ 3851], 80.00th=[ 5276], 90.00th=[ 7439], 95.00th=[10159], | 99.00th=[19792], 99.50th=[39584], 99.90th=[56886], 99.95th=[56886], | 99.99th=[56886] bw ( KiB/s): min= 2075, max= 2880, per=24.91%, avg=2386.20, stdev=359.28, samples=5 iops : min= 129, max= 180, avg=149.00, stdev=22.61, samples=5 lat (usec) : 50=0.52%, 100=11.98%, 250=1.82%, 500=0.26%, 750=9.11% lat (usec) : 1000=9.90% lat (msec) : 2=7.16%, 4=29.69%, 10=26.04%, 20=2.86%, 50=0.52% lat (msec) : 100=0.13% cpu : usr=0.12%, sys=27.91%, ctx=1520, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24551: Sat Feb 24 23:18:10 2024 read: IOPS=149, BW=2387KiB/s (2444kB/s)(6032KiB/2527msec) clat (usec): min=47, max=23701, avg=3036.73, stdev=2923.66 lat (usec): min=47, max=23702, avg=3038.26, stdev=2923.43 clat percentiles (usec): | 1.00th=[ 51], 5.00th=[ 63], 10.00th=[ 68], 20.00th=[ 72], | 30.00th=[ 100], 40.00th=[ 2409], 50.00th=[ 2835], 60.00th=[ 3458], | 70.00th=[ 4228], 80.00th=[ 5080], 90.00th=[ 6325], 95.00th=[ 7635], | 99.00th=[13698], 99.50th=[17171], 99.90th=[23725], 99.95th=[23725], | 99.99th=[23725] bw ( KiB/s): min= 1216, max= 2912, per=26.01%, avg=2368.00, stdev=694.85, samples=5 iops : min= 76, max= 182, avg=148.00, stdev=43.43, samples=5 write: IOPS=154, BW=2476KiB/s (2535kB/s)(6256KiB/2527msec) clat (usec): min=553, max=39855, avg=3516.62, stdev=4795.73 lat (usec): min=554, max=39857, avg=3517.89, stdev=4795.62 clat percentiles (usec): | 1.00th=[ 594], 5.00th=[ 676], 10.00th=[ 717], 20.00th=[ 766], | 30.00th=[ 857], 40.00th=[ 1106], 50.00th=[ 1614], 60.00th=[ 2966], | 70.00th=[ 3752], 80.00th=[ 5473], 90.00th=[ 7111], 95.00th=[ 9896], | 99.00th=[26608], 99.50th=[39060], 99.90th=[40109], 99.95th=[40109], | 99.99th=[40109] bw ( KiB/s): min= 1600, max= 2944, per=25.79%, avg=2470.40, stdev=552.13, samples=5 iops : min= 100, max= 184, avg=154.40, stdev=34.51, samples=5 lat (usec) : 50=0.39%, 100=14.58%, 250=0.78%, 500=0.39%, 750=8.72% lat (usec) : 1000=10.16% lat (msec) : 2=8.46%, 4=26.04%, 10=27.21%, 20=1.95%, 50=1.30% cpu : usr=0.28%, sys=27.20%, ctx=1451, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=9104KiB/s (9322kB/s), 2183KiB/s-2476KiB/s (2236kB/s-2535kB/s), io=23.4MiB (24.5MB), run=2527-2631msec WRITE: bw=9578KiB/s (9808kB/s), 2387KiB/s-2487KiB/s (2444kB/s-2547kB/s), io=24.6MiB (25.8MB), run=2527-2631msec rand-rw: (groupid=0, jobs=1): err= 0: pid=24543: Sat Feb 24 23:18:16 2024 read: IOPS=40, BW=651KiB/s (667kB/s)(5920KiB/9091msec) clat (usec): min=1990, max=24670, avg=3883.19, stdev=2632.45 lat (usec): min=1991, max=24670, avg=3884.12, stdev=2632.42 clat percentiles (usec): | 1.00th=[ 2057], 5.00th=[ 2180], 10.00th=[ 2278], 20.00th=[ 2376], | 30.00th=[ 2507], 40.00th=[ 2638], 50.00th=[ 2802], 60.00th=[ 3130], | 70.00th=[ 3785], 80.00th=[ 4817], 90.00th=[ 6587], 95.00th=[ 9372], | 99.00th=[15139], 99.50th=[17695], 99.90th=[24773], 99.95th=[24773], | 99.99th=[24773] bw ( KiB/s): min= 384, max= 1184, per=25.11%, avg=645.17, stdev=212.95, samples=18 iops : min= 24, max= 74, avg=40.22, stdev=13.36, samples=18 write: IOPS=43, BW=700KiB/s (717kB/s)(6368KiB/9091msec) clat (usec): min=11589, max=47918, avg=19206.31, stdev=5757.27 lat (usec): min=11591, max=47919, avg=19207.88, stdev=5757.27 clat percentiles (usec): | 1.00th=[11863], 5.00th=[12518], 10.00th=[13173], 20.00th=[14222], | 30.00th=[15270], 40.00th=[16712], 50.00th=[18482], 60.00th=[19792], | 70.00th=[21365], 80.00th=[22676], 90.00th=[26084], 95.00th=[30540], | 99.00th=[40109], 99.50th=[45351], 99.90th=[47973], 99.95th=[47973], | 99.99th=[47973] bw ( KiB/s): min= 416, max= 864, per=25.85%, avg=698.50, stdev=144.69, samples=18 iops : min= 26, max= 54, avg=43.56, stdev= 9.19, samples=18 lat (msec) : 2=0.13%, 4=34.77%, 10=11.20%, 20=33.20%, 50=20.70% cpu : usr=0.08%, sys=6.42%, ctx=936, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24544: Sat Feb 24 23:18:16 2024 read: IOPS=38, BW=616KiB/s (631kB/s)(5744KiB/9323msec) clat (usec): min=1980, max=23368, avg=3890.97, stdev=3020.70 lat (usec): min=1980, max=23369, avg=3891.89, stdev=3020.68 clat percentiles (usec): | 1.00th=[ 2040], 5.00th=[ 2147], 10.00th=[ 2212], 20.00th=[ 2343], | 30.00th=[ 2474], 40.00th=[ 2606], 50.00th=[ 2769], 60.00th=[ 2966], | 70.00th=[ 3523], 80.00th=[ 4686], 90.00th=[ 6783], 95.00th=[10028], | 99.00th=[20055], 99.50th=[22938], 99.90th=[23462], 99.95th=[23462], | 99.99th=[23462] bw ( KiB/s): min= 287, max= 1120, per=23.87%, avg=613.22, stdev=231.72, samples=18 iops : min= 17, max= 70, avg=38.22, stdev=14.59, samples=18 write: IOPS=43, BW=702KiB/s (719kB/s)(6544KiB/9323msec) clat (usec): min=10721, max=48352, avg=19357.82, stdev=6023.75 lat (usec): min=10723, max=48354, avg=19359.78, stdev=6023.90 clat percentiles (usec): | 1.00th=[11731], 5.00th=[12518], 10.00th=[13042], 20.00th=[14091], | 30.00th=[15270], 40.00th=[17171], 50.00th=[18482], 60.00th=[20055], | 70.00th=[21103], 80.00th=[22676], 90.00th=[26608], 95.00th=[31851], | 99.00th=[42206], 99.50th=[42206], 99.90th=[48497], 99.95th=[48497], | 99.99th=[48497] bw ( KiB/s): min= 416, max= 864, per=25.65%, avg=693.17, stdev=157.75, samples=18 iops : min= 26, max= 54, avg=43.22, stdev=10.00, samples=18 lat (msec) : 2=0.13%, 4=35.03%, 10=9.11%, 20=34.24%, 50=21.48% cpu : usr=0.11%, sys=6.32%, ctx=903, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24545: Sat Feb 24 23:18:16 2024 read: IOPS=43, BW=699KiB/s (716kB/s)(6256KiB/8946msec) clat (usec): min=1818, max=23543, avg=3845.30, stdev=2629.65 lat (usec): min=1819, max=23545, avg=3846.18, stdev=2629.65 clat percentiles (usec): | 1.00th=[ 1975], 5.00th=[ 2180], 10.00th=[ 2278], 20.00th=[ 2409], | 30.00th=[ 2540], 40.00th=[ 2671], 50.00th=[ 2802], 60.00th=[ 3032], | 70.00th=[ 3556], 80.00th=[ 4948], 90.00th=[ 6587], 95.00th=[ 9241], | 99.00th=[15401], 99.50th=[20841], 99.90th=[23462], 99.95th=[23462], | 99.99th=[23462] bw ( KiB/s): min= 288, max= 1216, per=27.69%, avg=711.41, stdev=284.43, samples=17 iops : min= 18, max= 76, avg=44.35, stdev=17.89, samples=17 write: IOPS=42, BW=674KiB/s (690kB/s)(6032KiB/8946msec) clat (usec): min=11496, max=54205, avg=19717.67, stdev=6050.31 lat (usec): min=11498, max=54206, avg=19719.40, stdev=6050.35 clat percentiles (usec): | 1.00th=[11994], 5.00th=[13173], 10.00th=[13698], 20.00th=[14746], | 30.00th=[16188], 40.00th=[17695], 50.00th=[19006], 60.00th=[20055], | 70.00th=[21103], 80.00th=[22152], 90.00th=[26608], 95.00th=[31327], | 99.00th=[43779], 99.50th=[50070], 99.90th=[54264], 99.95th=[54264], | 99.99th=[54264] bw ( KiB/s): min= 447, max= 864, per=24.80%, avg=670.00, stdev=127.94, samples=17 iops : min= 27, max= 54, avg=41.76, stdev= 8.19, samples=17 lat (msec) : 2=0.65%, 4=36.72%, 10=11.72%, 20=30.21%, 50=20.44% lat (msec) : 100=0.26% cpu : usr=0.13%, sys=6.61%, ctx=931, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24546: Sat Feb 24 23:18:16 2024 read: IOPS=41, BW=667KiB/s (683kB/s)(6032KiB/9043msec) clat (usec): min=1978, max=28648, avg=3718.12, stdev=2721.21 lat (usec): min=1979, max=28649, avg=3719.00, stdev=2721.18 clat percentiles (usec): | 1.00th=[ 2040], 5.00th=[ 2147], 10.00th=[ 2245], 20.00th=[ 2376], | 30.00th=[ 2507], 40.00th=[ 2638], 50.00th=[ 2737], 60.00th=[ 2933], | 70.00th=[ 3294], 80.00th=[ 4555], 90.00th=[ 6259], 95.00th=[ 8160], | 99.00th=[15795], 99.50th=[23200], 99.90th=[28705], 99.95th=[28705], | 99.99th=[28705] bw ( KiB/s): min= 288, max= 1056, per=25.81%, avg=662.94, stdev=216.20, samples=18 iops : min= 18, max= 66, avg=41.33, stdev=13.57, samples=18 write: IOPS=43, BW=692KiB/s (708kB/s)(6256KiB/9043msec) clat (usec): min=11069, max=42010, avg=19519.31, stdev=5612.42 lat (usec): min=11070, max=42012, avg=19521.20, stdev=5612.69 clat percentiles (usec): | 1.00th=[11863], 5.00th=[12911], 10.00th=[13435], 20.00th=[14615], | 30.00th=[15926], 40.00th=[17171], 50.00th=[19006], 60.00th=[20317], | 70.00th=[21365], 80.00th=[22414], 90.00th=[26084], 95.00th=[31851], | 99.00th=[40109], 99.50th=[40633], 99.90th=[42206], 99.95th=[42206], | 99.99th=[42206] bw ( KiB/s): min= 416, max= 832, per=25.59%, avg=691.33, stdev=132.10, samples=18 iops : min= 26, max= 52, avg=43.11, stdev= 8.40, samples=18 lat (msec) : 2=0.13%, 4=36.85%, 10=10.55%, 20=30.60%, 50=21.87% cpu : usr=0.17%, sys=6.31%, ctx=899, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2569KiB/s (2631kB/s), 616KiB/s-699KiB/s (631kB/s-716kB/s), io=23.4MiB (24.5MB), run=8946-9323msec WRITE: bw=2703KiB/s (2768kB/s), 674KiB/s-702KiB/s (690kB/s-719kB/s), io=24.6MiB (25.8MB), run=8946-9323msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=24565: Sat Feb 24 23:18:18 2024 read: IOPS=13, BW=13.7MiB/s (14.4MB/s)(7168KiB/511msec) clat (usec): min=9453, max=61163, avg=40968.42, stdev=16685.41 lat (usec): min=9453, max=61165, avg=40969.58, stdev=16685.55 clat percentiles (usec): | 1.00th=[ 9503], 5.00th=[ 9503], 10.00th=[ 9503], 20.00th=[33162], | 30.00th=[39584], 40.00th=[39584], 50.00th=[41681], 60.00th=[49546], | 70.00th=[49546], 80.00th=[52167], 90.00th=[61080], 95.00th=[61080], | 99.00th=[61080], 99.50th=[61080], 99.90th=[61080], 99.95th=[61080], | 99.99th=[61080] write: IOPS=9, BW=9.78MiB/s (10.3MB/s)(5120KiB/511msec) clat (usec): min=18909, max=69793, avg=44569.19, stdev=20107.03 lat (usec): min=18930, max=69842, avg=44622.30, stdev=20122.32 clat percentiles (usec): | 1.00th=[19006], 5.00th=[19006], 10.00th=[19006], 20.00th=[19006], | 30.00th=[31327], 40.00th=[31327], 50.00th=[46400], 60.00th=[46400], | 70.00th=[56361], 80.00th=[56361], 90.00th=[69731], 95.00th=[69731], | 99.00th=[69731], 99.50th=[69731], 99.90th=[69731], 99.95th=[69731], | 99.99th=[69731] lat (msec) : 10=8.33%, 20=8.33%, 50=50.00%, 100=33.33% cpu : usr=0.00%, sys=49.22%, ctx=164, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24566: Sat Feb 24 23:18:18 2024 read: IOPS=3, BW=3938KiB/s (4033kB/s)(2048KiB/520msec) clat (usec): min=36835, max=47902, avg=42368.89, stdev=7825.21 lat (usec): min=36836, max=47904, avg=42370.38, stdev=7826.13 clat percentiles (usec): | 1.00th=[36963], 5.00th=[36963], 10.00th=[36963], 20.00th=[36963], | 30.00th=[36963], 40.00th=[36963], 50.00th=[36963], 60.00th=[47973], | 70.00th=[47973], 80.00th=[47973], 90.00th=[47973], 95.00th=[47973], | 99.00th=[47973], 99.50th=[47973], 99.90th=[47973], 99.95th=[47973], | 99.99th=[47973] bw ( KiB/s): min= 4087, max= 4087, per=10.38%, avg=4087.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 write: IOPS=19, BW=19.2MiB/s (20.2MB/s)(10.0MiB/520msec) clat (msec): min=17, max=104, avg=43.33, stdev=32.51 lat (msec): min=17, max=104, avg=43.39, stdev=32.52 clat percentiles (msec): | 1.00th=[ 18], 5.00th=[ 18], 10.00th=[ 18], 20.00th=[ 18], | 30.00th=[ 22], 40.00th=[ 24], 50.00th=[ 32], 60.00th=[ 33], | 70.00th=[ 44], 80.00th=[ 44], 90.00th=[ 102], 95.00th=[ 105], | 99.00th=[ 105], 99.50th=[ 105], 99.90th=[ 105], 99.95th=[ 105], | 99.99th=[ 105] bw ( KiB/s): min=18395, max=18395, per=33.36%, avg=18395.00, stdev= 0.00, samples=1 iops : min= 17, max= 17, avg=17.00, stdev= 0.00, samples=1 lat (msec) : 20=16.67%, 50=66.67%, 250=16.67% cpu : usr=0.00%, sys=40.08%, ctx=67, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24567: Sat Feb 24 23:18:18 2024 read: IOPS=12, BW=12.3MiB/s (12.9MB/s)(6144KiB/486msec) clat (usec): min=382, max=63945, avg=23574.30, stdev=23619.53 lat (usec): min=384, max=63946, avg=23575.75, stdev=23619.63 clat percentiles (usec): | 1.00th=[ 383], 5.00th=[ 383], 10.00th=[ 383], 20.00th=[ 494], | 30.00th=[ 494], 40.00th=[18482], 50.00th=[18482], 60.00th=[28705], | 70.00th=[29754], 80.00th=[29754], 90.00th=[63701], 95.00th=[63701], | 99.00th=[63701], 99.50th=[63701], 99.90th=[63701], 99.95th=[63701], | 99.99th=[63701] write: IOPS=12, BW=12.3MiB/s (12.9MB/s)(6144KiB/486msec) clat (msec): min=24, max=126, avg=57.21, stdev=38.33 lat (msec): min=24, max=126, avg=57.25, stdev=38.35 clat percentiles (msec): | 1.00th=[ 25], 5.00th=[ 25], 10.00th=[ 25], 20.00th=[ 31], | 30.00th=[ 31], 40.00th=[ 41], 50.00th=[ 41], 60.00th=[ 46], | 70.00th=[ 78], 80.00th=[ 78], 90.00th=[ 127], 95.00th=[ 127], | 99.00th=[ 127], 99.50th=[ 127], 99.90th=[ 127], 99.95th=[ 127], | 99.99th=[ 127] lat (usec) : 500=16.67% lat (msec) : 20=8.33%, 50=50.00%, 100=16.67%, 250=8.33% cpu : usr=0.00%, sys=32.78%, ctx=81, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24568: Sat Feb 24 23:18:18 2024 read: IOPS=10, BW=10.4MiB/s (10.9MB/s)(5120KiB/481msec) clat (usec): min=386, max=50885, avg=22948.53, stdev=23110.02 lat (usec): min=387, max=50886, avg=22949.73, stdev=23110.05 clat percentiles (usec): | 1.00th=[ 388], 5.00th=[ 388], 10.00th=[ 388], 20.00th=[ 388], | 30.00th=[ 429], 40.00th=[ 429], 50.00th=[21627], 60.00th=[21627], | 70.00th=[41157], 80.00th=[41157], 90.00th=[51119], 95.00th=[51119], | 99.00th=[51119], 99.50th=[51119], 99.90th=[51119], 99.95th=[51119], | 99.99th=[51119] write: IOPS=14, BW=14.6MiB/s (15.3MB/s)(7168KiB/481msec) clat (msec): min=20, max=115, avg=52.13, stdev=32.58 lat (msec): min=20, max=115, avg=52.21, stdev=32.58 clat percentiles (msec): | 1.00th=[ 21], 5.00th=[ 21], 10.00th=[ 21], 20.00th=[ 27], | 30.00th=[ 34], 40.00th=[ 34], 50.00th=[ 46], 60.00th=[ 54], | 70.00th=[ 54], 80.00th=[ 70], 90.00th=[ 115], 95.00th=[ 115], | 99.00th=[ 115], 99.50th=[ 115], 99.90th=[ 115], 99.95th=[ 115], | 99.99th=[ 115] lat (usec) : 500=16.67% lat (msec) : 50=50.00%, 100=25.00%, 250=8.33% cpu : usr=0.00%, sys=34.10%, ctx=82, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=38.5MiB/s (40.3MB/s), 3938KiB/s-13.7MiB/s (4033kB/s-14.4MB/s), io=20.0MiB (20.0MB), run=481-520msec WRITE: bw=53.8MiB/s (56.5MB/s), 9.78MiB/s-19.2MiB/s (10.3MB/s-20.2MB/s), io=28.0MiB (29.4MB), run=481-520msec rand-rw: (groupid=0, jobs=1): err= 0: pid=24569: Sat Feb 24 23:18:18 2024 read: IOPS=8, BW=8763KiB/s (8973kB/s)(7168KiB/818msec) clat (msec): min=20, max=149, avg=70.79, stdev=57.09 lat (msec): min=20, max=149, avg=70.79, stdev=57.09 clat percentiles (msec): | 1.00th=[ 21], 5.00th=[ 21], 10.00th=[ 21], 20.00th=[ 23], | 30.00th=[ 24], 40.00th=[ 24], 50.00th=[ 64], 60.00th=[ 67], | 70.00th=[ 67], 80.00th=[ 150], 90.00th=[ 150], 95.00th=[ 150], | 99.00th=[ 150], 99.50th=[ 150], 99.90th=[ 150], 99.95th=[ 150], | 99.99th=[ 150] bw ( KiB/s): min= 6144, max= 6144, per=24.54%, avg=6144.00, stdev= 0.00, samples=1 iops : min= 6, max= 6, avg= 6.00, stdev= 0.00, samples=1 write: IOPS=6, BW=6259KiB/s (6409kB/s)(5120KiB/818msec) clat (msec): min=35, max=157, avg=64.25, stdev=52.75 lat (msec): min=35, max=157, avg=64.31, stdev=52.74 clat percentiles (msec): | 1.00th=[ 35], 5.00th=[ 35], 10.00th=[ 35], 20.00th=[ 35], | 30.00th=[ 37], 40.00th=[ 37], 50.00th=[ 37], 60.00th=[ 37], | 70.00th=[ 55], 80.00th=[ 55], 90.00th=[ 159], 95.00th=[ 159], | 99.00th=[ 159], 99.50th=[ 159], 99.90th=[ 159], 99.95th=[ 159], | 99.99th=[ 159] bw ( KiB/s): min= 2048, max= 2048, per=5.84%, avg=2048.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 50=50.00%, 100=25.00%, 250=25.00% cpu : usr=0.00%, sys=12.36%, ctx=50, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24570: Sat Feb 24 23:18:18 2024 read: IOPS=2, BW=2868KiB/s (2937kB/s)(2048KiB/714msec) clat (usec): min=40773, max=99542, avg=70158.32, stdev=41555.75 lat (usec): min=40775, max=99544, avg=70159.71, stdev=41556.01 clat percentiles (usec): | 1.00th=[40633], 5.00th=[40633], 10.00th=[40633], 20.00th=[40633], | 30.00th=[40633], 40.00th=[40633], 50.00th=[40633], 60.00th=[99091], | 70.00th=[99091], 80.00th=[99091], 90.00th=[99091], 95.00th=[99091], | 99.00th=[99091], 99.50th=[99091], 99.90th=[99091], 99.95th=[99091], | 99.99th=[99091] bw ( KiB/s): min= 4096, max= 4096, per=16.36%, avg=4096.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 write: IOPS=14, BW=14.0MiB/s (14.7MB/s)(10.0MiB/714msec) clat (msec): min=30, max=128, avg=57.21, stdev=29.36 lat (msec): min=30, max=128, avg=57.28, stdev=29.36 clat percentiles (msec): | 1.00th=[ 31], 5.00th=[ 31], 10.00th=[ 31], 20.00th=[ 35], | 30.00th=[ 37], 40.00th=[ 44], 50.00th=[ 47], 60.00th=[ 50], | 70.00th=[ 63], 80.00th=[ 63], 90.00th=[ 84], 95.00th=[ 129], | 99.00th=[ 129], 99.50th=[ 129], 99.90th=[ 129], 99.95th=[ 129], | 99.99th=[ 129] bw ( KiB/s): min=10240, max=10240, per=29.21%, avg=10240.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 lat (msec) : 50=58.33%, 100=33.33%, 250=8.33% cpu : usr=0.00%, sys=16.97%, ctx=46, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24571: Sat Feb 24 23:18:18 2024 read: IOPS=8, BW=8336KiB/s (8537kB/s)(6144KiB/737msec) clat (usec): min=20815, max=37944, avg=28331.80, stdev=6550.21 lat (usec): min=20816, max=37945, avg=28332.86, stdev=6550.27 clat percentiles (usec): | 1.00th=[20841], 5.00th=[20841], 10.00th=[20841], 20.00th=[22414], | 30.00th=[22414], 40.00th=[25297], 50.00th=[25297], 60.00th=[31065], | 70.00th=[32375], 80.00th=[32375], 90.00th=[38011], 95.00th=[38011], | 99.00th=[38011], 99.50th=[38011], 99.90th=[38011], 99.95th=[38011], | 99.99th=[38011] bw ( KiB/s): min= 4096, max= 4096, per=16.36%, avg=4096.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 write: IOPS=8, BW=8336KiB/s (8537kB/s)(6144KiB/737msec) clat (msec): min=35, max=159, avg=94.19, stdev=56.17 lat (msec): min=36, max=159, avg=94.25, stdev=56.16 clat percentiles (msec): | 1.00th=[ 36], 5.00th=[ 36], 10.00th=[ 36], 20.00th=[ 43], | 30.00th=[ 43], 40.00th=[ 54], 50.00th=[ 54], 60.00th=[ 128], | 70.00th=[ 148], 80.00th=[ 148], 90.00th=[ 159], 95.00th=[ 159], | 99.00th=[ 159], 99.50th=[ 159], 99.90th=[ 159], 99.95th=[ 159], | 99.99th=[ 159] bw ( KiB/s): min= 6144, max= 6144, per=17.53%, avg=6144.00, stdev= 0.00, samples=1 iops : min= 6, max= 6, avg= 6.00, stdev= 0.00, samples=1 lat (msec) : 50=66.67%, 100=8.33%, 250=25.00% cpu : usr=0.14%, sys=17.39%, ctx=63, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24572: Sat Feb 24 23:18:18 2024 read: IOPS=6, BW=6481KiB/s (6637kB/s)(5120KiB/790msec) clat (usec): min=18282, max=46612, avg=31969.05, stdev=13119.83 lat (usec): min=18282, max=46613, avg=31970.01, stdev=13120.03 clat percentiles (usec): | 1.00th=[18220], 5.00th=[18220], 10.00th=[18220], 20.00th=[18220], | 30.00th=[20317], 40.00th=[20317], 50.00th=[30278], 60.00th=[30278], | 70.00th=[44303], 80.00th=[44303], 90.00th=[46400], 95.00th=[46400], | 99.00th=[46400], 99.50th=[46400], 99.90th=[46400], 99.95th=[46400], | 99.99th=[46400] write: IOPS=8, BW=9073KiB/s (9291kB/s)(7168KiB/790msec) clat (msec): min=29, max=190, avg=85.21, stdev=66.96 lat (msec): min=29, max=190, avg=85.29, stdev=66.97 clat percentiles (msec): | 1.00th=[ 30], 5.00th=[ 30], 10.00th=[ 30], 20.00th=[ 41], | 30.00th=[ 44], 40.00th=[ 44], 50.00th=[ 49], 60.00th=[ 72], | 70.00th=[ 72], 80.00th=[ 171], 90.00th=[ 190], 95.00th=[ 190], | 99.00th=[ 190], 99.50th=[ 190], 99.90th=[ 190], 99.95th=[ 190], | 99.99th=[ 190] bw ( KiB/s): min= 6144, max= 6144, per=17.53%, avg=6144.00, stdev= 0.00, samples=1 iops : min= 6, max= 6, avg= 6.00, stdev= 0.00, samples=1 lat (msec) : 20=8.33%, 50=66.67%, 100=8.33%, 250=16.67% cpu : usr=0.13%, sys=16.60%, ctx=46, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=24.4MiB/s (25.6MB/s), 2868KiB/s-8763KiB/s (2937kB/s-8973kB/s), io=20.0MiB (20.0MB), run=714-818msec WRITE: bw=34.2MiB/s (35.9MB/s), 6259KiB/s-14.0MiB/s (6409kB/s-14.7MB/s), io=28.0MiB (29.4MB), run=714-818msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=24578: Sat Feb 24 23:18:20 2024 read: IOPS=10, BW=43.3MiB/s (45.4MB/s)(12.0MiB/277msec) clat (msec): min=52, max=119, avg=90.45, stdev=34.74 lat (msec): min=52, max=119, avg=90.45, stdev=34.74 clat percentiles (msec): | 1.00th=[ 53], 5.00th=[ 53], 10.00th=[ 53], 20.00th=[ 53], | 30.00th=[ 53], 40.00th=[ 101], 50.00th=[ 101], 60.00th=[ 101], | 70.00th=[ 121], 80.00th=[ 121], 90.00th=[ 121], 95.00th=[ 121], | 99.00th=[ 121], 99.50th=[ 121], 99.90th=[ 121], 99.95th=[ 121], | 99.99th=[ 121] lat (msec) : 100=66.67%, 250=33.33% cpu : usr=0.00%, sys=77.90%, ctx=147, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24579: Sat Feb 24 23:18:20 2024 read: IOPS=1, BW=5826KiB/s (5966kB/s)(4096KiB/703msec) clat (nsec): min=70681k, max=70681k, avg=70681004.00, stdev= 0.00 lat (nsec): min=70682k, max=70682k, avg=70682416.00, stdev= 0.00 clat percentiles (usec): | 1.00th=[70779], 5.00th=[70779], 10.00th=[70779], 20.00th=[70779], | 30.00th=[70779], 40.00th=[70779], 50.00th=[70779], 60.00th=[70779], | 70.00th=[70779], 80.00th=[70779], 90.00th=[70779], 95.00th=[70779], | 99.00th=[70779], 99.50th=[70779], 99.90th=[70779], 99.95th=[70779], | 99.99th=[70779] bw ( KiB/s): min= 8159, max= 8159, per=35.01%, avg=8159.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=2, BW=11.4MiB/s (11.9MB/s)(8192KiB/703msec) clat (msec): min=129, max=500, avg=314.70, stdev=262.10 lat (msec): min=129, max=500, avg=314.87, stdev=262.10 clat percentiles (msec): | 1.00th=[ 130], 5.00th=[ 130], 10.00th=[ 130], 20.00th=[ 130], | 30.00th=[ 130], 40.00th=[ 130], 50.00th=[ 130], 60.00th=[ 502], | 70.00th=[ 502], 80.00th=[ 502], 90.00th=[ 502], 95.00th=[ 502], | 99.00th=[ 502], 99.50th=[ 502], 99.90th=[ 502], 99.95th=[ 502], | 99.99th=[ 502] lat (msec) : 100=33.33%, 250=33.33%, 750=33.33% cpu : usr=0.00%, sys=25.07%, ctx=88, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24580: Sat Feb 24 23:18:20 2024 write: IOPS=4, BW=17.1MiB/s (17.9MB/s)(12.0MiB/701msec) clat (msec): min=93, max=329, avg=230.72, stdev=122.95 lat (msec): min=93, max=330, avg=231.05, stdev=122.96 clat percentiles (msec): | 1.00th=[ 93], 5.00th=[ 93], 10.00th=[ 93], 20.00th=[ 93], | 30.00th=[ 93], 40.00th=[ 271], 50.00th=[ 271], 60.00th=[ 271], | 70.00th=[ 330], 80.00th=[ 330], 90.00th=[ 330], 95.00th=[ 330], | 99.00th=[ 330], 99.50th=[ 330], 99.90th=[ 330], 99.95th=[ 330], | 99.99th=[ 330] bw ( KiB/s): min=16384, max=16384, per=35.15%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 500=66.67% cpu : usr=0.00%, sys=30.14%, ctx=51, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24581: Sat Feb 24 23:18:20 2024 write: IOPS=5, BW=22.3MiB/s (23.4MB/s)(12.0MiB/538msec) clat (msec): min=82, max=308, avg=178.85, stdev=116.30 lat (msec): min=82, max=308, avg=179.05, stdev=116.24 clat percentiles (msec): | 1.00th=[ 83], 5.00th=[ 83], 10.00th=[ 83], 20.00th=[ 83], | 30.00th=[ 83], 40.00th=[ 146], 50.00th=[ 146], 60.00th=[ 146], | 70.00th=[ 309], 80.00th=[ 309], 90.00th=[ 309], 95.00th=[ 309], | 99.00th=[ 309], 99.50th=[ 309], 99.90th=[ 309], 99.95th=[ 309], | 99.99th=[ 309] bw ( KiB/s): min=16384, max=16384, per=35.15%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 250=33.33%, 500=33.33% cpu : usr=0.19%, sys=36.31%, ctx=50, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=22.8MiB/s (23.9MB/s), 5826KiB/s-43.3MiB/s (5966kB/s-45.4MB/s), io=16.0MiB (16.8MB), run=277-703msec WRITE: bw=45.5MiB/s (47.7MB/s), 11.4MiB/s-22.3MiB/s (11.9MB/s-23.4MB/s), io=32.0MiB (33.6MB), run=538-703msec rand-rw: (groupid=0, jobs=1): err= 0: pid=24582: Sat Feb 24 23:18:20 2024 read: IOPS=3, BW=14.2MiB/s (14.9MB/s)(12.0MiB/845msec) clat (msec): min=141, max=368, avg=280.39, stdev=121.82 lat (msec): min=141, max=368, avg=280.39, stdev=121.82 clat percentiles (msec): | 1.00th=[ 142], 5.00th=[ 142], 10.00th=[ 142], 20.00th=[ 142], | 30.00th=[ 142], 40.00th=[ 334], 50.00th=[ 334], 60.00th=[ 334], | 70.00th=[ 368], 80.00th=[ 368], 90.00th=[ 368], 95.00th=[ 368], | 99.00th=[ 368], 99.50th=[ 368], 99.90th=[ 368], 99.95th=[ 368], | 99.99th=[ 368] bw ( KiB/s): min= 8192, max= 8192, per=42.25%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=66.67% cpu : usr=0.00%, sys=11.61%, ctx=48, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24583: Sat Feb 24 23:18:20 2024 read: IOPS=1, BW=5114KiB/s (5236kB/s)(4096KiB/801msec) clat (nsec): min=412274k, max=412274k, avg=412273685.00, stdev= 0.00 lat (nsec): min=412275k, max=412275k, avg=412274867.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 414], 5.00th=[ 414], 10.00th=[ 414], 20.00th=[ 414], | 30.00th=[ 414], 40.00th=[ 414], 50.00th=[ 414], 60.00th=[ 414], | 70.00th=[ 414], 80.00th=[ 414], 90.00th=[ 414], 95.00th=[ 414], | 99.00th=[ 414], 99.50th=[ 414], 99.90th=[ 414], 99.95th=[ 414], | 99.99th=[ 414] bw ( KiB/s): min= 8192, max= 8192, per=42.25%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=2, BW=9.99MiB/s (10.5MB/s)(8192KiB/801msec) clat (msec): min=86, max=293, avg=190.10, stdev=145.95 lat (msec): min=87, max=293, avg=190.35, stdev=146.08 clat percentiles (msec): | 1.00th=[ 87], 5.00th=[ 87], 10.00th=[ 87], 20.00th=[ 87], | 30.00th=[ 87], 40.00th=[ 87], 50.00th=[ 87], 60.00th=[ 292], | 70.00th=[ 292], 80.00th=[ 292], 90.00th=[ 292], 95.00th=[ 292], | 99.00th=[ 292], 99.50th=[ 292], 99.90th=[ 292], 99.95th=[ 292], | 99.99th=[ 292] lat (msec) : 100=33.33%, 500=66.67% cpu : usr=0.00%, sys=15.62%, ctx=202, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24584: Sat Feb 24 23:18:20 2024 write: IOPS=3, BW=14.0MiB/s (15.7MB/s)(12.0MiB/802msec) clat (msec): min=131, max=347, avg=264.29, stdev=116.61 lat (msec): min=131, max=348, avg=264.63, stdev=116.59 clat percentiles (msec): | 1.00th=[ 131], 5.00th=[ 131], 10.00th=[ 131], 20.00th=[ 131], | 30.00th=[ 131], 40.00th=[ 313], 50.00th=[ 313], 60.00th=[ 313], | 70.00th=[ 347], 80.00th=[ 347], 90.00th=[ 347], 95.00th=[ 347], | 99.00th=[ 347], 99.50th=[ 347], 99.90th=[ 347], 99.95th=[ 347], | 99.99th=[ 347] bw ( KiB/s): min=16384, max=16384, per=40.10%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=66.67% cpu : usr=0.25%, sys=14.98%, ctx=82, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24585: Sat Feb 24 23:18:20 2024 write: IOPS=4, BW=17.6MiB/s (18.5MB/s)(12.0MiB/682msec) clat (msec): min=97, max=415, avg=225.86, stdev=167.37 lat (msec): min=97, max=415, avg=226.20, stdev=167.34 clat percentiles (msec): | 1.00th=[ 97], 5.00th=[ 97], 10.00th=[ 97], 20.00th=[ 97], | 30.00th=[ 97], 40.00th=[ 165], 50.00th=[ 165], 60.00th=[ 165], | 70.00th=[ 414], 80.00th=[ 414], 90.00th=[ 414], 95.00th=[ 414], | 99.00th=[ 414], 99.50th=[ 414], 99.90th=[ 414], 99.95th=[ 414], | 99.99th=[ 414] bw ( KiB/s): min= 8192, max= 8192, per=20.05%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 250=33.33%, 500=33.33% cpu : usr=0.00%, sys=17.60%, ctx=47, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=18.9MiB/s (19.9MB/s), 5114KiB/s-14.2MiB/s (5236kB/s-14.9MB/s), io=16.0MiB (16.8MB), run=801-845msec WRITE: bw=39.9MiB/s (41.8MB/s), 9.99MiB/s-17.6MiB/s (10.5MB/s-18.5MB/s), io=32.0MiB (33.6MB), run=682-802msec