== sanity test 398b: DIO and buffer IO race ============== 23:26:33 (1708835193) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.39523 s, 36.1 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=25905: Sat Feb 24 23:26:47 2024 read: IOPS=126, BW=505KiB/s (518kB/s)(5992KiB/11856msec) clat (usec): min=41, max=137050, avg=3444.12, stdev=5252.60 lat (usec): min=41, max=137051, avg=3444.77, stdev=5252.61 clat percentiles (usec): | 1.00th=[ 46], 5.00th=[ 57], 10.00th=[ 65], 20.00th=[ 79], | 30.00th=[ 1893], 40.00th=[ 2212], 50.00th=[ 2573], 60.00th=[ 3032], | 70.00th=[ 3785], 80.00th=[ 5407], 90.00th=[ 7570], 95.00th=[ 8979], | 99.00th=[ 13829], 99.50th=[ 20317], 99.90th=[ 85459], 99.95th=[137364], | 99.99th=[137364] bw ( KiB/s): min= 352, max= 664, per=24.61%, avg=497.96, stdev=79.20, samples=23 iops : min= 88, max= 166, avg=124.43, stdev=19.81, samples=23 write: IOPS=132, BW=531KiB/s (544kB/s)(6296KiB/11856msec) clat (usec): min=347, max=37497, avg=4187.42, stdev=4100.83 lat (usec): min=347, max=37498, avg=4188.71, stdev=4100.74 clat percentiles (usec): | 1.00th=[ 424], 5.00th=[ 457], 10.00th=[ 478], 20.00th=[ 545], | 30.00th=[ 1037], 40.00th=[ 2278], 50.00th=[ 3294], 60.00th=[ 4228], | 70.00th=[ 5276], 80.00th=[ 6849], 90.00th=[ 9503], 95.00th=[11994], | 99.00th=[17433], 99.50th=[20317], 99.90th=[35914], 99.95th=[37487], | 99.99th=[37487] bw ( KiB/s): min= 400, max= 704, per=24.58%, avg=520.57, stdev=77.56, samples=23 iops : min= 100, max= 176, avg=130.09, stdev=19.41, samples=23 lat (usec) : 50=1.24%, 100=9.77%, 250=0.65%, 500=7.91%, 750=5.92% lat (usec) : 1000=1.56% lat (msec) : 2=8.98%, 4=28.29%, 10=29.75%, 20=5.31%, 50=0.55% lat (msec) : 100=0.03%, 250=0.03% cpu : usr=0.20%, sys=21.11%, ctx=5509, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25907: Sat Feb 24 23:26:47 2024 read: IOPS=125, BW=503KiB/s (515kB/s)(5940KiB/11820msec) clat (usec): min=38, max=39148, avg=3205.85, stdev=3151.68 lat (usec): min=39, max=39149, avg=3206.92, stdev=3151.99 clat percentiles (usec): | 1.00th=[ 44], 5.00th=[ 58], 10.00th=[ 65], 20.00th=[ 80], | 30.00th=[ 1844], 40.00th=[ 2147], 50.00th=[ 2540], 60.00th=[ 2999], | 70.00th=[ 3851], 80.00th=[ 5276], 90.00th=[ 7111], 95.00th=[ 8717], | 99.00th=[13698], 99.50th=[17695], 99.90th=[27657], 99.95th=[39060], | 99.99th=[39060] bw ( KiB/s): min= 344, max= 648, per=24.60%, avg=497.65, stdev=82.07, samples=23 iops : min= 86, max= 162, avg=124.35, stdev=20.53, samples=23 write: IOPS=134, BW=537KiB/s (550kB/s)(6348KiB/11820msec) clat (usec): min=409, max=136846, avg=4385.58, stdev=5625.40 lat (usec): min=410, max=136847, avg=4386.43, stdev=5625.51 clat percentiles (usec): | 1.00th=[ 433], 5.00th=[ 457], 10.00th=[ 478], 20.00th=[ 537], | 30.00th=[ 1012], 40.00th=[ 2507], 50.00th=[ 3458], 60.00th=[ 4293], | 70.00th=[ 5407], 80.00th=[ 7046], 90.00th=[ 9503], 95.00th=[ 11731], | 99.00th=[ 18220], 99.50th=[ 25560], 99.90th=[ 70779], 99.95th=[137364], | 99.99th=[137364] bw ( KiB/s): min= 392, max= 712, per=24.72%, avg=523.65, stdev=88.53, samples=23 iops : min= 98, max= 178, avg=130.87, stdev=22.13, samples=23 lat (usec) : 50=1.27%, 100=9.64%, 250=0.46%, 500=8.17%, 750=6.05% lat (usec) : 1000=1.43% lat (msec) : 2=9.44%, 4=27.28%, 10=30.08%, 20=5.66%, 50=0.46% lat (msec) : 100=0.03%, 250=0.03% cpu : usr=0.21%, sys=20.68%, ctx=5463, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25909: Sat Feb 24 23:26:47 2024 read: IOPS=127, BW=508KiB/s (520kB/s)(6028KiB/11865msec) clat (usec): min=40, max=57704, avg=3150.61, stdev=3582.85 lat (usec): min=41, max=57704, avg=3151.28, stdev=3582.86 clat percentiles (usec): | 1.00th=[ 46], 5.00th=[ 58], 10.00th=[ 64], 20.00th=[ 72], | 30.00th=[ 1778], 40.00th=[ 2114], 50.00th=[ 2442], 60.00th=[ 2966], | 70.00th=[ 3818], 80.00th=[ 5080], 90.00th=[ 6783], 95.00th=[ 8586], | 99.00th=[13435], 99.50th=[16909], 99.90th=[42206], 99.95th=[57934], | 99.99th=[57934] bw ( KiB/s): min= 264, max= 696, per=24.87%, avg=503.17, stdev=108.99, samples=23 iops : min= 66, max= 174, avg=125.74, stdev=27.27, samples=23 write: IOPS=131, BW=528KiB/s (540kB/s)(6260KiB/11865msec) clat (usec): min=383, max=162850, avg=4485.42, stdev=6220.36 lat (usec): min=383, max=162852, avg=4486.23, stdev=6220.38 clat percentiles (usec): | 1.00th=[ 433], 5.00th=[ 457], 10.00th=[ 474], 20.00th=[ 529], | 30.00th=[ 857], 40.00th=[ 2212], 50.00th=[ 3326], 60.00th=[ 4424], | 70.00th=[ 5538], 80.00th=[ 7242], 90.00th=[ 9765], 95.00th=[ 12387], | 99.00th=[ 20055], 99.50th=[ 24773], 99.90th=[ 67634], 99.95th=[162530], | 99.99th=[162530] bw ( KiB/s): min= 320, max= 688, per=24.50%, avg=518.83, stdev=103.16, samples=23 iops : min= 80, max= 172, avg=129.65, stdev=25.81, samples=23 lat (usec) : 50=1.56%, 100=10.06%, 250=0.98%, 500=8.63%, 750=6.32% lat (usec) : 1000=1.43% lat (msec) : 2=8.89%, 4=26.07%, 10=30.24%, 20=5.08%, 50=0.65% lat (msec) : 100=0.07%, 250=0.03% cpu : usr=0.20%, sys=20.37%, ctx=5404, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25911: Sat Feb 24 23:26:47 2024 read: IOPS=128, BW=513KiB/s (525kB/s)(6052KiB/11803msec) clat (usec): min=40, max=46048, avg=3150.74, stdev=3113.59 lat (usec): min=40, max=46049, avg=3151.48, stdev=3113.66 clat percentiles (usec): | 1.00th=[ 45], 5.00th=[ 58], 10.00th=[ 65], 20.00th=[ 82], | 30.00th=[ 1860], 40.00th=[ 2147], 50.00th=[ 2474], 60.00th=[ 2966], | 70.00th=[ 3720], 80.00th=[ 5080], 90.00th=[ 6980], 95.00th=[ 8455], | 99.00th=[12387], 99.50th=[14877], 99.90th=[34866], 99.95th=[45876], | 99.99th=[45876] bw ( KiB/s): min= 247, max= 664, per=25.12%, avg=508.09, stdev=94.52, samples=23 iops : min= 61, max= 166, avg=126.96, stdev=23.74, samples=23 write: IOPS=132, BW=528KiB/s (541kB/s)(6236KiB/11803msec) clat (usec): min=404, max=181723, avg=4448.42, stdev=6451.87 lat (usec): min=405, max=181725, avg=4449.20, stdev=6451.89 clat percentiles (usec): | 1.00th=[ 437], 5.00th=[ 457], 10.00th=[ 474], 20.00th=[ 529], | 30.00th=[ 816], 40.00th=[ 2180], 50.00th=[ 3326], 60.00th=[ 4228], | 70.00th=[ 5407], 80.00th=[ 6980], 90.00th=[ 10290], 95.00th=[ 13042], | 99.00th=[ 19792], 99.50th=[ 29492], 99.90th=[ 51119], 99.95th=[181404], | 99.99th=[181404] bw ( KiB/s): min= 311, max= 672, per=24.70%, avg=523.04, stdev=85.45, samples=23 iops : min= 77, max= 168, avg=130.70, stdev=21.47, samples=23 lat (usec) : 50=1.37%, 100=9.57%, 250=0.94%, 500=8.79%, 750=6.05% lat (usec) : 1000=1.73% lat (msec) : 2=8.69%, 4=27.86%, 10=28.22%, 20=6.22%, 50=0.49% lat (msec) : 100=0.03%, 250=0.03% cpu : usr=0.17%, sys=20.61%, ctx=5368, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2024KiB/s (2072kB/s), 503KiB/s-513KiB/s (515kB/s-525kB/s), io=23.4MiB (24.6MB), run=11803-11865msec WRITE: bw=2119KiB/s (2170kB/s), 528KiB/s-537KiB/s (540kB/s-550kB/s), io=24.6MiB (25.7MB), run=11803-11865msec rand-rw: (groupid=0, jobs=1): err= 0: pid=25906: Sat Feb 24 23:27:14 2024 read: IOPS=38, BW=156KiB/s (159kB/s)(5992KiB/38476msec) clat (usec): min=1481, max=141022, avg=3562.14, stdev=4733.02 lat (usec): min=1482, max=141024, avg=3562.89, stdev=4733.04 clat percentiles (usec): | 1.00th=[ 1614], 5.00th=[ 1745], 10.00th=[ 1811], 20.00th=[ 1893], | 30.00th=[ 1991], 40.00th=[ 2089], 50.00th=[ 2245], 60.00th=[ 2507], | 70.00th=[ 3261], 80.00th=[ 3949], 90.00th=[ 6652], 95.00th=[ 10159], | 99.00th=[ 16909], 99.50th=[ 21365], 99.90th=[ 40633], 99.95th=[141558], | 99.99th=[141558] bw ( KiB/s): min= 48, max= 272, per=25.05%, avg=155.79, stdev=62.56, samples=76 iops : min= 12, max= 68, avg=38.87, stdev=15.62, samples=76 write: IOPS=40, BW=164KiB/s (168kB/s)(6296KiB/38476msec) clat (usec): min=11422, max=58842, avg=20990.85, stdev=7817.66 lat (usec): min=11423, max=58843, avg=20991.77, stdev=7817.67 clat percentiles (usec): | 1.00th=[11994], 5.00th=[12780], 10.00th=[13304], 20.00th=[14484], | 30.00th=[15926], 40.00th=[17695], 50.00th=[19792], 60.00th=[21103], | 70.00th=[22414], 80.00th=[24773], 90.00th=[32113], 95.00th=[38011], | 99.00th=[47449], 99.50th=[51643], 99.90th=[57934], 99.95th=[58983], | 99.99th=[58983] bw ( KiB/s): min= 56, max= 216, per=25.08%, avg=163.26, stdev=46.83, samples=76 iops : min= 14, max= 54, avg=40.74, stdev=11.68, samples=76 lat (msec) : 2=15.14%, 4=24.15%, 10=6.97%, 20=28.97%, 50=24.38% lat (msec) : 100=0.36%, 250=0.03% cpu : usr=0.08%, sys=4.07%, ctx=3484, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25908: Sat Feb 24 23:27:14 2024 read: IOPS=38, BW=154KiB/s (158kB/s)(5940KiB/38579msec) clat (usec): min=1535, max=141386, avg=3430.25, stdev=4456.45 lat (usec): min=1535, max=141388, avg=3431.03, stdev=4456.47 clat percentiles (usec): | 1.00th=[ 1631], 5.00th=[ 1745], 10.00th=[ 1795], 20.00th=[ 1893], | 30.00th=[ 1975], 40.00th=[ 2089], 50.00th=[ 2245], 60.00th=[ 2540], | 70.00th=[ 3261], 80.00th=[ 3982], 90.00th=[ 6390], 95.00th=[ 8979], | 99.00th=[ 14091], 99.50th=[ 15664], 99.90th=[ 38011], 99.95th=[141558], | 99.99th=[141558] bw ( KiB/s): min= 15, max= 264, per=24.78%, avg=154.16, stdev=64.51, samples=76 iops : min= 3, max= 66, avg=38.50, stdev=16.16, samples=76 write: IOPS=41, BW=165KiB/s (168kB/s)(6348KiB/38579msec) clat (usec): min=11461, max=85959, avg=21036.80, stdev=8422.78 lat (usec): min=11462, max=85960, avg=21037.70, stdev=8422.76 clat percentiles (usec): | 1.00th=[11994], 5.00th=[12649], 10.00th=[13173], 20.00th=[14222], | 30.00th=[15664], 40.00th=[17695], 50.00th=[19268], 60.00th=[20841], | 70.00th=[22414], 80.00th=[24511], 90.00th=[32637], 95.00th=[39060], | 99.00th=[48497], 99.50th=[53740], 99.90th=[81265], 99.95th=[85459], | 99.99th=[85459] bw ( KiB/s): min= 63, max= 224, per=25.17%, avg=163.84, stdev=48.11, samples=76 iops : min= 15, max= 56, avg=40.92, stdev=12.05, samples=76 lat (msec) : 2=15.46%, 4=23.24%, 10=7.98%, 20=29.26%, 50=23.57% lat (msec) : 100=0.46%, 250=0.03% cpu : usr=0.12%, sys=4.05%, ctx=3498, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25910: Sat Feb 24 23:27:14 2024 read: IOPS=39, BW=157KiB/s (161kB/s)(6028KiB/38436msec) clat (usec): min=1422, max=39112, avg=3626.27, stdev=3233.53 lat (usec): min=1423, max=39113, avg=3627.00, stdev=3233.53 clat percentiles (usec): | 1.00th=[ 1614], 5.00th=[ 1745], 10.00th=[ 1811], 20.00th=[ 1893], | 30.00th=[ 2008], 40.00th=[ 2114], 50.00th=[ 2311], 60.00th=[ 2704], | 70.00th=[ 3458], 80.00th=[ 4424], 90.00th=[ 7373], 95.00th=[10159], | 99.00th=[16450], 99.50th=[18220], 99.90th=[36963], 99.95th=[39060], | 99.99th=[39060] bw ( KiB/s): min= 48, max= 312, per=25.28%, avg=157.21, stdev=60.32, samples=76 iops : min= 12, max= 78, avg=39.26, stdev=15.08, samples=76 write: IOPS=40, BW=163KiB/s (167kB/s)(6260KiB/38436msec) clat (msec): min=11, max=159, avg=21.01, stdev= 8.51 lat (msec): min=11, max=159, avg=21.01, stdev= 8.51 clat percentiles (msec): | 1.00th=[ 12], 5.00th=[ 13], 10.00th=[ 14], 20.00th=[ 15], | 30.00th=[ 16], 40.00th=[ 19], 50.00th=[ 20], 60.00th=[ 22], | 70.00th=[ 23], 80.00th=[ 25], 90.00th=[ 32], 95.00th=[ 38], | 99.00th=[ 48], 99.50th=[ 52], 99.90th=[ 86], 99.95th=[ 161], | 99.99th=[ 161] bw ( KiB/s): min= 64, max= 224, per=24.99%, avg=162.68, stdev=47.49, samples=76 iops : min= 16, max= 56, avg=40.63, stdev=11.87, samples=76 lat (msec) : 2=14.42%, 4=23.73%, 10=8.33%, 20=28.68%, 50=24.48% lat (msec) : 100=0.33%, 250=0.03% cpu : usr=0.11%, sys=4.12%, ctx=3535, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25912: Sat Feb 24 23:27:14 2024 read: IOPS=39, BW=157KiB/s (161kB/s)(6052KiB/38465msec) clat (usec): min=1529, max=39226, avg=3479.02, stdev=3238.99 lat (usec): min=1530, max=39227, avg=3479.86, stdev=3238.95 clat percentiles (usec): | 1.00th=[ 1598], 5.00th=[ 1713], 10.00th=[ 1778], 20.00th=[ 1876], | 30.00th=[ 1958], 40.00th=[ 2057], 50.00th=[ 2212], 60.00th=[ 2474], | 70.00th=[ 3294], 80.00th=[ 4015], 90.00th=[ 6521], 95.00th=[10290], | 99.00th=[17171], 99.50th=[19006], 99.90th=[36963], 99.95th=[39060], | 99.99th=[39060] bw ( KiB/s): min= 40, max= 280, per=25.18%, avg=156.61, stdev=62.28, samples=76 iops : min= 10, max= 70, avg=39.13, stdev=15.59, samples=76 write: IOPS=40, BW=162KiB/s (166kB/s)(6236KiB/38465msec) clat (msec): min=11, max=179, avg=21.23, stdev= 9.17 lat (msec): min=11, max=179, avg=21.24, stdev= 9.17 clat percentiles (msec): | 1.00th=[ 12], 5.00th=[ 13], 10.00th=[ 14], 20.00th=[ 15], | 30.00th=[ 16], 40.00th=[ 18], 50.00th=[ 20], 60.00th=[ 22], | 70.00th=[ 23], 80.00th=[ 25], 90.00th=[ 34], 95.00th=[ 40], | 99.00th=[ 51], 99.50th=[ 56], 99.90th=[ 82], 99.95th=[ 180], | 99.99th=[ 180] bw ( KiB/s): min= 56, max= 216, per=24.86%, avg=161.87, stdev=48.49, samples=76 iops : min= 14, max= 54, avg=40.45, stdev=12.13, samples=76 lat (msec) : 2=17.29%, 4=22.04%, 10=7.36%, 20=29.10%, 50=23.67% lat (msec) : 100=0.52%, 250=0.03% cpu : usr=0.08%, sys=4.18%, ctx=3542, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=622KiB/s (637kB/s), 154KiB/s-157KiB/s (158kB/s-161kB/s), io=23.4MiB (24.6MB), run=38436-38579msec WRITE: bw=652KiB/s (667kB/s), 162KiB/s-165KiB/s (166kB/s-168kB/s), io=24.6MiB (25.7MB), run=38436-38579msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=25961: Sat Feb 24 23:27:17 2024 read: IOPS=171, BW=2743KiB/s (2809kB/s)(5920KiB/2158msec) clat (usec): min=44, max=23961, avg=2642.26, stdev=2866.57 lat (usec): min=45, max=23962, avg=2642.97, stdev=2866.58 clat percentiles (usec): | 1.00th=[ 47], 5.00th=[ 60], 10.00th=[ 66], 20.00th=[ 71], | 30.00th=[ 84], 40.00th=[ 1188], 50.00th=[ 2507], 60.00th=[ 2966], | 70.00th=[ 3589], 80.00th=[ 4752], 90.00th=[ 5997], 95.00th=[ 6980], | 99.00th=[ 9765], 99.50th=[19006], 99.90th=[23987], 99.95th=[23987], | 99.99th=[23987] bw ( KiB/s): min= 2080, max= 3648, per=25.34%, avg=2704.00, stdev=669.46, samples=4 iops : min= 130, max= 228, avg=169.00, stdev=41.84, samples=4 write: IOPS=184, BW=2951KiB/s (3022kB/s)(6368KiB/2158msec) clat (usec): min=577, max=23681, avg=2947.71, stdev=3188.75 lat (usec): min=578, max=23682, avg=2949.22, stdev=3188.38 clat percentiles (usec): | 1.00th=[ 603], 5.00th=[ 644], 10.00th=[ 685], 20.00th=[ 758], | 30.00th=[ 848], 40.00th=[ 979], 50.00th=[ 1237], 60.00th=[ 2114], | 70.00th=[ 3720], 80.00th=[ 5080], 90.00th=[ 7046], 95.00th=[ 9110], | 99.00th=[13698], 99.50th=[17171], 99.90th=[23725], 99.95th=[23725], | 99.99th=[23725] bw ( KiB/s): min= 2368, max= 4192, per=26.09%, avg=2928.00, stdev=849.26, samples=4 iops : min= 148, max= 262, avg=183.00, stdev=53.08, samples=4 lat (usec) : 50=0.91%, 100=14.97%, 250=2.60%, 500=0.39%, 750=10.03% lat (usec) : 1000=11.72% lat (msec) : 2=8.85%, 4=23.70%, 10=24.48%, 20=2.08%, 50=0.26% cpu : usr=0.42%, sys=29.44%, ctx=1381, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25962: Sat Feb 24 23:27:17 2024 read: IOPS=161, BW=2583KiB/s (2645kB/s)(5744KiB/2224msec) clat (usec): min=45, max=11740, avg=2348.16, stdev=2328.71 lat (usec): min=45, max=11740, avg=2348.81, stdev=2328.74 clat percentiles (usec): | 1.00th=[ 48], 5.00th=[ 57], 10.00th=[ 63], 20.00th=[ 69], | 30.00th=[ 74], 40.00th=[ 225], 50.00th=[ 2474], 60.00th=[ 2737], | 70.00th=[ 3163], 80.00th=[ 4178], 90.00th=[ 5604], 95.00th=[ 6849], | 99.00th=[ 8586], 99.50th=[10552], 99.90th=[11731], 99.95th=[11731], | 99.99th=[11731] bw ( KiB/s): min= 2112, max= 3744, per=25.12%, avg=2680.00, stdev=736.87, samples=4 iops : min= 132, max= 234, avg=167.50, stdev=46.05, samples=4 write: IOPS=183, BW=2942KiB/s (3013kB/s)(6544KiB/2224msec) clat (usec): min=582, max=41797, avg=3358.70, stdev=4131.72 lat (usec): min=583, max=41798, avg=3359.61, stdev=4131.76 clat percentiles (usec): | 1.00th=[ 603], 5.00th=[ 652], 10.00th=[ 693], 20.00th=[ 766], | 30.00th=[ 865], 40.00th=[ 1106], 50.00th=[ 1696], 60.00th=[ 2966], | 70.00th=[ 4015], 80.00th=[ 5604], 90.00th=[ 7177], 95.00th=[ 9503], | 99.00th=[16712], 99.50th=[25560], 99.90th=[41681], 99.95th=[41681], | 99.99th=[41681] bw ( KiB/s): min= 2496, max= 3520, per=26.80%, avg=3008.00, stdev=555.49, samples=4 iops : min= 156, max= 220, avg=188.00, stdev=34.72, samples=4 lat (usec) : 50=1.04%, 100=16.54%, 250=1.17%, 500=0.39%, 750=9.77% lat (usec) : 1000=10.03% lat (msec) : 2=8.72%, 4=25.65%, 10=23.96%, 20=2.34%, 50=0.39% cpu : usr=0.22%, sys=28.97%, ctx=1492, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25964: Sat Feb 24 23:27:17 2024 read: IOPS=174, BW=2787KiB/s (2854kB/s)(6256KiB/2245msec) clat (usec): min=45, max=46550, avg=2800.65, stdev=3907.60 lat (usec): min=45, max=46551, avg=2801.28, stdev=3907.62 clat percentiles (usec): | 1.00th=[ 48], 5.00th=[ 54], 10.00th=[ 60], 20.00th=[ 69], | 30.00th=[ 78], 40.00th=[ 2180], 50.00th=[ 2704], 60.00th=[ 2966], | 70.00th=[ 3359], 80.00th=[ 4621], 90.00th=[ 5997], 95.00th=[ 7504], | 99.00th=[13173], 99.50th=[42730], 99.90th=[46400], 99.95th=[46400], | 99.99th=[46400] bw ( KiB/s): min= 2176, max= 3584, per=27.22%, avg=2904.00, stdev=582.70, samples=4 iops : min= 136, max= 224, avg=181.50, stdev=36.42, samples=4 write: IOPS=167, BW=2687KiB/s (2751kB/s)(6032KiB/2245msec) clat (usec): min=561, max=31758, avg=3027.64, stdev=3521.28 lat (usec): min=562, max=31759, avg=3028.53, stdev=3521.30 clat percentiles (usec): | 1.00th=[ 594], 5.00th=[ 619], 10.00th=[ 676], 20.00th=[ 750], | 30.00th=[ 824], 40.00th=[ 1020], 50.00th=[ 1369], 60.00th=[ 2606], | 70.00th=[ 3687], 80.00th=[ 5145], 90.00th=[ 6980], 95.00th=[ 8717], | 99.00th=[17171], 99.50th=[27395], 99.90th=[31851], 99.95th=[31851], | 99.99th=[31851] bw ( KiB/s): min= 2528, max= 3232, per=24.52%, avg=2752.00, stdev=327.38, samples=4 iops : min= 158, max= 202, avg=172.00, stdev=20.46, samples=4 lat (usec) : 50=1.43%, 100=15.89%, 250=1.43%, 500=0.13%, 750=10.03% lat (usec) : 1000=9.37% lat (msec) : 2=8.33%, 4=27.47%, 10=23.83%, 20=1.56%, 50=0.52% cpu : usr=0.31%, sys=28.03%, ctx=1445, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25966: Sat Feb 24 23:27:17 2024 read: IOPS=179, BW=2867KiB/s (2936kB/s)(6032KiB/2104msec) clat (usec): min=46, max=29204, avg=2788.77, stdev=3012.44 lat (usec): min=47, max=29204, avg=2789.58, stdev=3012.67 clat percentiles (usec): | 1.00th=[ 48], 5.00th=[ 59], 10.00th=[ 66], 20.00th=[ 70], | 30.00th=[ 76], 40.00th=[ 2212], 50.00th=[ 2573], 60.00th=[ 3097], | 70.00th=[ 3752], 80.00th=[ 4752], 90.00th=[ 6128], 95.00th=[ 7963], | 99.00th=[10028], 99.50th=[20055], 99.90th=[29230], 99.95th=[29230], | 99.99th=[29230] bw ( KiB/s): min= 2496, max= 3264, per=27.07%, avg=2888.00, stdev=369.85, samples=4 iops : min= 156, max= 204, avg=180.50, stdev=23.12, samples=4 write: IOPS=185, BW=2973KiB/s (3045kB/s)(6256KiB/2104msec) clat (usec): min=596, max=34565, avg=2674.60, stdev=3267.69 lat (usec): min=597, max=34566, avg=2675.53, stdev=3267.68 clat percentiles (usec): | 1.00th=[ 603], 5.00th=[ 635], 10.00th=[ 685], 20.00th=[ 734], | 30.00th=[ 783], 40.00th=[ 881], 50.00th=[ 1090], 60.00th=[ 1860], | 70.00th=[ 3032], 80.00th=[ 4228], 90.00th=[ 6194], 95.00th=[ 8848], | 99.00th=[14091], 99.50th=[17695], 99.90th=[34341], 99.95th=[34341], | 99.99th=[34341] bw ( KiB/s): min= 2464, max= 3936, per=26.87%, avg=3016.00, stdev=666.84, samples=4 iops : min= 154, max= 246, avg=188.50, stdev=41.68, samples=4 lat (usec) : 50=1.17%, 100=15.62%, 250=1.04%, 500=0.39%, 750=11.59% lat (usec) : 1000=12.76% lat (msec) : 2=6.64%, 4=25.65%, 10=22.53%, 20=2.21%, 50=0.39% cpu : usr=0.19%, sys=29.86%, ctx=1455, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=10.4MiB/s (10.9MB/s), 2583KiB/s-2867KiB/s (2645kB/s-2936kB/s), io=23.4MiB (24.5MB), run=2104-2245msec WRITE: bw=10.0MiB/s (11.5MB/s), 2687KiB/s-2973KiB/s (2751kB/s-3045kB/s), io=24.6MiB (25.8MB), run=2104-2245msec rand-rw: (groupid=0, jobs=1): err= 0: pid=25963: Sat Feb 24 23:27:25 2024 read: IOPS=38, BW=612KiB/s (627kB/s)(5920KiB/9673msec) clat (usec): min=1682, max=24275, avg=3561.88, stdev=3152.31 lat (usec): min=1683, max=24276, avg=3562.80, stdev=3152.45 clat percentiles (usec): | 1.00th=[ 1909], 5.00th=[ 2040], 10.00th=[ 2114], 20.00th=[ 2180], | 30.00th=[ 2245], 40.00th=[ 2343], 50.00th=[ 2442], 60.00th=[ 2573], | 70.00th=[ 2835], 80.00th=[ 3425], 90.00th=[ 6325], 95.00th=[11076], | 99.00th=[19006], 99.50th=[23725], 99.90th=[24249], 99.95th=[24249], | 99.99th=[24249] bw ( KiB/s): min= 287, max= 1088, per=24.49%, avg=602.47, stdev=200.10, samples=19 iops : min= 17, max= 68, avg=37.47, stdev=12.56, samples=19 write: IOPS=41, BW=658KiB/s (674kB/s)(6368KiB/9673msec) clat (usec): min=11419, max=87534, avg=20974.68, stdev=7170.66 lat (usec): min=11420, max=87536, avg=20976.25, stdev=7170.84 clat percentiles (usec): | 1.00th=[12387], 5.00th=[12911], 10.00th=[13698], 20.00th=[14746], | 30.00th=[16712], 40.00th=[19006], 50.00th=[20579], 60.00th=[22152], | 70.00th=[23200], 80.00th=[24773], 90.00th=[27919], 95.00th=[31589], | 99.00th=[46924], 99.50th=[63701], 99.90th=[87557], 99.95th=[87557], | 99.99th=[87557] bw ( KiB/s): min= 384, max= 768, per=25.29%, avg=654.68, stdev=131.80, samples=19 iops : min= 24, max= 48, avg=40.74, stdev= 8.20, samples=19 lat (msec) : 2=1.56%, 4=38.15%, 10=5.73%, 20=26.43%, 50=27.86% lat (msec) : 100=0.26% cpu : usr=0.07%, sys=5.05%, ctx=847, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25965: Sat Feb 24 23:27:25 2024 read: IOPS=36, BW=590KiB/s (604kB/s)(5744KiB/9733msec) clat (usec): min=1805, max=16982, avg=3307.24, stdev=2483.09 lat (usec): min=1806, max=16983, avg=3308.01, stdev=2483.10 clat percentiles (usec): | 1.00th=[ 1942], 5.00th=[ 2040], 10.00th=[ 2089], 20.00th=[ 2180], | 30.00th=[ 2245], 40.00th=[ 2343], 50.00th=[ 2409], 60.00th=[ 2540], | 70.00th=[ 2737], 80.00th=[ 3228], 90.00th=[ 5866], 95.00th=[ 8717], | 99.00th=[14746], 99.50th=[15795], 99.90th=[16909], 99.95th=[16909], | 99.99th=[16909] bw ( KiB/s): min= 192, max= 960, per=24.09%, avg=592.74, stdev=192.75, samples=19 iops : min= 12, max= 60, avg=37.00, stdev=12.07, samples=19 write: IOPS=42, BW=672KiB/s (688kB/s)(6544KiB/9733msec) clat (usec): min=12145, max=59429, avg=20871.23, stdev=6640.51 lat (usec): min=12147, max=59431, avg=20872.63, stdev=6640.50 clat percentiles (usec): | 1.00th=[12780], 5.00th=[13435], 10.00th=[13960], 20.00th=[14877], | 30.00th=[16319], 40.00th=[18482], 50.00th=[20579], 60.00th=[21627], | 70.00th=[22938], 80.00th=[24511], 90.00th=[28443], 95.00th=[33162], | 99.00th=[44303], 99.50th=[47449], 99.90th=[59507], 99.95th=[59507], | 99.99th=[59507] bw ( KiB/s): min= 384, max= 832, per=25.82%, avg=668.53, stdev=140.63, samples=19 iops : min= 24, max= 52, avg=41.74, stdev= 8.78, samples=19 lat (msec) : 2=1.30%, 4=38.28%, 10=5.34%, 20=26.69%, 50=28.12% lat (msec) : 100=0.26% cpu : usr=0.03%, sys=5.01%, ctx=842, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25967: Sat Feb 24 23:27:25 2024 read: IOPS=42, BW=680KiB/s (696kB/s)(6256KiB/9205msec) clat (usec): min=1862, max=36977, avg=3206.16, stdev=2831.10 lat (usec): min=1863, max=36978, avg=3206.90, stdev=2831.11 clat percentiles (usec): | 1.00th=[ 1893], 5.00th=[ 1991], 10.00th=[ 2057], 20.00th=[ 2147], | 30.00th=[ 2212], 40.00th=[ 2278], 50.00th=[ 2376], 60.00th=[ 2474], | 70.00th=[ 2671], 80.00th=[ 2999], 90.00th=[ 5407], 95.00th=[ 8094], | 99.00th=[14353], 99.50th=[19006], 99.90th=[36963], 99.95th=[36963], | 99.99th=[36963] bw ( KiB/s): min= 320, max= 1184, per=27.82%, avg=684.33, stdev=284.95, samples=18 iops : min= 20, max= 74, avg=42.72, stdev=17.80, samples=18 write: IOPS=40, BW=655KiB/s (671kB/s)(6032KiB/9205msec) clat (usec): min=11520, max=68306, avg=21065.50, stdev=6438.20 lat (usec): min=11521, max=68309, avg=21066.87, stdev=6438.21 clat percentiles (usec): | 1.00th=[12649], 5.00th=[13435], 10.00th=[14091], 20.00th=[15401], | 30.00th=[17433], 40.00th=[19006], 50.00th=[20579], 60.00th=[21890], | 70.00th=[23200], 80.00th=[24249], 90.00th=[28181], 95.00th=[33817], | 99.00th=[41681], 99.50th=[44827], 99.90th=[68682], 99.95th=[68682], | 99.99th=[68682] bw ( KiB/s): min= 416, max= 800, per=25.13%, avg=650.56, stdev=125.52, samples=18 iops : min= 26, max= 50, avg=40.61, stdev= 7.80, samples=18 lat (msec) : 2=2.73%, 4=41.28%, 10=5.21%, 20=23.96%, 50=26.69% lat (msec) : 100=0.13% cpu : usr=0.08%, sys=5.25%, ctx=867, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25968: Sat Feb 24 23:27:25 2024 read: IOPS=39, BW=629KiB/s (644kB/s)(6032KiB/9597msec) clat (usec): min=1936, max=22015, avg=3387.58, stdev=2715.71 lat (usec): min=1936, max=22015, avg=3388.32, stdev=2715.70 clat percentiles (usec): | 1.00th=[ 1958], 5.00th=[ 2073], 10.00th=[ 2114], 20.00th=[ 2212], | 30.00th=[ 2278], 40.00th=[ 2376], 50.00th=[ 2474], 60.00th=[ 2573], | 70.00th=[ 2835], 80.00th=[ 3228], 90.00th=[ 5604], 95.00th=[ 9634], | 99.00th=[16581], 99.50th=[18744], 99.90th=[21890], 99.95th=[21890], | 99.99th=[21890] bw ( KiB/s): min= 160, max= 1024, per=25.32%, avg=622.89, stdev=225.73, samples=19 iops : min= 10, max= 64, avg=38.79, stdev=14.07, samples=19 write: IOPS=40, BW=652KiB/s (668kB/s)(6256KiB/9597msec) clat (usec): min=12423, max=53964, avg=21254.04, stdev=6357.35 lat (usec): min=12424, max=53967, avg=21255.42, stdev=6357.43 clat percentiles (usec): | 1.00th=[12911], 5.00th=[13435], 10.00th=[13960], 20.00th=[15139], | 30.00th=[16909], 40.00th=[19530], 50.00th=[21103], 60.00th=[22152], | 70.00th=[23462], 80.00th=[24773], 90.00th=[29230], 95.00th=[33424], | 99.00th=[42206], 99.50th=[46400], 99.90th=[53740], 99.95th=[53740], | 99.99th=[53740] bw ( KiB/s): min= 416, max= 800, per=25.10%, avg=649.79, stdev=131.34, samples=19 iops : min= 26, max= 50, avg=40.47, stdev= 8.13, samples=19 lat (msec) : 2=0.91%, 4=41.15%, 10=4.95%, 20=24.22%, 50=28.65% lat (msec) : 100=0.13% cpu : usr=0.04%, sys=5.20%, ctx=858, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2461KiB/s (2520kB/s), 590KiB/s-680KiB/s (604kB/s-696kB/s), io=23.4MiB (24.5MB), run=9205-9733msec WRITE: bw=2589KiB/s (2651kB/s), 652KiB/s-672KiB/s (668kB/s-688kB/s), io=24.6MiB (25.8MB), run=9205-9733msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=25981: Sat Feb 24 23:27:26 2024 read: IOPS=15, BW=15.5MiB/s (16.3MB/s)(7168KiB/451msec) clat (usec): min=364, max=57587, avg=36754.74, stdev=17598.79 lat (usec): min=365, max=57588, avg=36755.69, stdev=17598.84 clat percentiles (usec): | 1.00th=[ 363], 5.00th=[ 363], 10.00th=[ 363], 20.00th=[34866], | 30.00th=[38536], 40.00th=[38536], 50.00th=[39584], 60.00th=[42206], | 70.00th=[42206], 80.00th=[44303], 90.00th=[57410], 95.00th=[57410], | 99.00th=[57410], 99.50th=[57410], 99.90th=[57410], 99.95th=[57410], | 99.99th=[57410] write: IOPS=11, BW=11.1MiB/s (11.6MB/s)(5120KiB/451msec) clat (usec): min=12665, max=60421, avg=36004.75, stdev=21177.48 lat (usec): min=12693, max=60490, avg=36051.91, stdev=21192.79 clat percentiles (usec): | 1.00th=[12649], 5.00th=[12649], 10.00th=[12649], 20.00th=[12649], | 30.00th=[17957], 40.00th=[17957], 50.00th=[34866], 60.00th=[34866], | 70.00th=[54264], 80.00th=[54264], 90.00th=[60556], 95.00th=[60556], | 99.00th=[60556], 99.50th=[60556], 99.90th=[60556], 99.95th=[60556], | 99.99th=[60556] lat (usec) : 500=8.33% lat (msec) : 20=16.67%, 50=50.00%, 100=25.00% cpu : usr=0.00%, sys=49.33%, ctx=93, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25982: Sat Feb 24 23:27:26 2024 read: IOPS=3, BW=3319KiB/s (3399kB/s)(2048KiB/617msec) clat (usec): min=48460, max=55982, avg=52221.38, stdev=5318.94 lat (usec): min=48461, max=55983, avg=52222.64, stdev=5318.77 clat percentiles (usec): | 1.00th=[48497], 5.00th=[48497], 10.00th=[48497], 20.00th=[48497], | 30.00th=[48497], 40.00th=[48497], 50.00th=[48497], 60.00th=[55837], | 70.00th=[55837], 80.00th=[55837], 90.00th=[55837], 95.00th=[55837], | 99.00th=[55837], 99.50th=[55837], 99.90th=[55837], 99.95th=[55837], | 99.99th=[55837] bw ( KiB/s): min= 4087, max= 4087, per=12.31%, avg=4087.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 write: IOPS=16, BW=16.2MiB/s (16.0MB/s)(10.0MiB/617msec) clat (msec): min=12, max=134, avg=50.87, stdev=36.20 lat (msec): min=12, max=134, avg=50.93, stdev=36.19 clat percentiles (msec): | 1.00th=[ 13], 5.00th=[ 13], 10.00th=[ 13], 20.00th=[ 24], | 30.00th=[ 27], 40.00th=[ 30], 50.00th=[ 34], 60.00th=[ 48], | 70.00th=[ 68], 80.00th=[ 68], 90.00th=[ 83], 95.00th=[ 136], | 99.00th=[ 136], 99.50th=[ 136], 99.90th=[ 136], 99.95th=[ 136], | 99.99th=[ 136] bw ( KiB/s): min=12263, max=12263, per=26.39%, avg=12263.00, stdev= 0.00, samples=1 iops : min= 11, max= 11, avg=11.00, stdev= 0.00, samples=1 lat (msec) : 20=8.33%, 50=58.33%, 100=25.00%, 250=8.33% cpu : usr=0.00%, sys=35.06%, ctx=77, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25983: Sat Feb 24 23:27:26 2024 read: IOPS=10, BW=10.6MiB/s (11.1MB/s)(6144KiB/565msec) clat (usec): min=444, max=62193, avg=31756.70, stdev=26743.24 lat (usec): min=445, max=62194, avg=31757.46, stdev=26743.23 clat percentiles (usec): | 1.00th=[ 445], 5.00th=[ 445], 10.00th=[ 445], 20.00th=[ 611], | 30.00th=[ 611], 40.00th=[27919], 50.00th=[27919], 60.00th=[45351], | 70.00th=[54264], 80.00th=[54264], 90.00th=[62129], 95.00th=[62129], | 99.00th=[62129], 99.50th=[62129], 99.90th=[62129], 99.95th=[62129], | 99.99th=[62129] bw ( KiB/s): min= 6144, max= 6144, per=18.51%, avg=6144.00, stdev= 0.00, samples=1 iops : min= 6, max= 6, avg= 6.00, stdev= 0.00, samples=1 write: IOPS=10, BW=10.6MiB/s (11.1MB/s)(6144KiB/565msec) clat (msec): min=14, max=104, avg=59.38, stdev=35.56 lat (msec): min=14, max=104, avg=59.42, stdev=35.57 clat percentiles (msec): | 1.00th=[ 15], 5.00th=[ 15], 10.00th=[ 15], 20.00th=[ 43], | 30.00th=[ 43], 40.00th=[ 47], 50.00th=[ 47], 60.00th=[ 49], | 70.00th=[ 101], 80.00th=[ 101], 90.00th=[ 105], 95.00th=[ 105], | 99.00th=[ 105], 99.50th=[ 105], 99.90th=[ 105], 99.95th=[ 105], | 99.99th=[ 105] bw ( KiB/s): min= 8192, max= 8192, per=17.63%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 lat (usec) : 500=8.33%, 750=8.33% lat (msec) : 20=8.33%, 50=41.67%, 100=16.67%, 250=16.67% cpu : usr=0.00%, sys=32.62%, ctx=93, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25984: Sat Feb 24 23:27:26 2024 read: IOPS=9, BW=9259KiB/s (9481kB/s)(5120KiB/553msec) clat (usec): min=375, max=44279, avg=17819.09, stdev=22650.76 lat (usec): min=376, max=44279, avg=17819.87, stdev=22650.60 clat percentiles (usec): | 1.00th=[ 375], 5.00th=[ 375], 10.00th=[ 375], 20.00th=[ 375], | 30.00th=[ 742], 40.00th=[ 742], 50.00th=[ 2835], 60.00th=[ 2835], | 70.00th=[40633], 80.00th=[40633], 90.00th=[44303], 95.00th=[44303], | 99.00th=[44303], 99.50th=[44303], 99.90th=[44303], 99.95th=[44303], | 99.99th=[44303] bw ( KiB/s): min= 8192, max= 8192, per=24.68%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=12, BW=12.7MiB/s (13.3MB/s)(7168KiB/553msec) clat (usec): min=32927, max=88451, avg=64182.18, stdev=18608.51 lat (usec): min=32995, max=88508, avg=64242.27, stdev=18601.28 clat percentiles (usec): | 1.00th=[32900], 5.00th=[32900], 10.00th=[32900], 20.00th=[47973], | 30.00th=[62653], 40.00th=[62653], 50.00th=[67634], 60.00th=[71828], | 70.00th=[71828], 80.00th=[77071], 90.00th=[88605], 95.00th=[88605], | 99.00th=[88605], 99.50th=[88605], 99.90th=[88605], 99.95th=[88605], | 99.99th=[88605] bw ( KiB/s): min=12288, max=12288, per=26.44%, avg=12288.00, stdev= 0.00, samples=1 iops : min= 12, max= 12, avg=12.00, stdev= 0.00, samples=1 lat (usec) : 500=8.33%, 750=8.33% lat (msec) : 4=8.33%, 50=33.33%, 100=41.67% cpu : usr=0.00%, sys=31.52%, ctx=86, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=32.4MiB/s (33.0MB/s), 3319KiB/s-15.5MiB/s (3399kB/s-16.3MB/s), io=20.0MiB (20.0MB), run=451-617msec WRITE: bw=45.4MiB/s (47.6MB/s), 10.6MiB/s-16.2MiB/s (11.1MB/s-16.0MB/s), io=28.0MiB (29.4MB), run=451-617msec rand-rw: (groupid=0, jobs=1): err= 0: pid=25985: Sat Feb 24 23:27:26 2024 read: IOPS=10, BW=10.4MiB/s (10.9MB/s)(7168KiB/674msec) clat (usec): min=27583, max=60726, avg=41877.58, stdev=12543.58 lat (usec): min=27584, max=60727, avg=41878.61, stdev=12543.54 clat percentiles (usec): | 1.00th=[27657], 5.00th=[27657], 10.00th=[27657], 20.00th=[30540], | 30.00th=[32637], 40.00th=[32637], 50.00th=[42730], 60.00th=[44303], | 70.00th=[44303], 80.00th=[54789], 90.00th=[60556], 95.00th=[60556], | 99.00th=[60556], 99.50th=[60556], 99.90th=[60556], 99.95th=[60556], | 99.99th=[60556] bw ( KiB/s): min=10240, max=10240, per=41.90%, avg=10240.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 write: IOPS=7, BW=7596KiB/s (7779kB/s)(5120KiB/674msec) clat (msec): min=51, max=102, avg=75.79, stdev=21.79 lat (msec): min=51, max=102, avg=75.84, stdev=21.79 clat percentiles (msec): | 1.00th=[ 52], 5.00th=[ 52], 10.00th=[ 52], 20.00th=[ 52], | 30.00th=[ 59], 40.00th=[ 59], 50.00th=[ 73], 60.00th=[ 73], | 70.00th=[ 93], 80.00th=[ 93], 90.00th=[ 104], 95.00th=[ 104], | 99.00th=[ 104], 99.50th=[ 104], 99.90th=[ 104], 99.95th=[ 104], | 99.99th=[ 104] bw ( KiB/s): min= 8192, max= 8192, per=23.94%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 lat (msec) : 50=41.67%, 100=50.00%, 250=8.33% cpu : usr=0.00%, sys=17.98%, ctx=81, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25986: Sat Feb 24 23:27:26 2024 read: IOPS=2, BW=2444KiB/s (2503kB/s)(2048KiB/838msec) clat (usec): min=28442, max=56704, avg=42573.45, stdev=19984.46 lat (usec): min=28443, max=56705, avg=42574.56, stdev=19984.14 clat percentiles (usec): | 1.00th=[28443], 5.00th=[28443], 10.00th=[28443], 20.00th=[28443], | 30.00th=[28443], 40.00th=[28443], 50.00th=[28443], 60.00th=[56886], | 70.00th=[56886], 80.00th=[56886], 90.00th=[56886], 95.00th=[56886], | 99.00th=[56886], 99.50th=[56886], 99.90th=[56886], 99.95th=[56886], | 99.99th=[56886] bw ( KiB/s): min= 2048, max= 2048, per=8.38%, avg=2048.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=11, BW=11.9MiB/s (12.5MB/s)(10.0MiB/838msec) clat (msec): min=31, max=183, avg=74.63, stdev=52.67 lat (msec): min=31, max=183, avg=74.70, stdev=52.67 clat percentiles (msec): | 1.00th=[ 32], 5.00th=[ 32], 10.00th=[ 32], 20.00th=[ 37], | 30.00th=[ 39], 40.00th=[ 41], 50.00th=[ 54], 60.00th=[ 54], | 70.00th=[ 95], 80.00th=[ 95], 90.00th=[ 153], 95.00th=[ 184], | 99.00th=[ 184], 99.50th=[ 184], 99.90th=[ 184], 99.95th=[ 184], | 99.99th=[ 184] bw ( KiB/s): min= 6144, max= 6144, per=17.96%, avg=6144.00, stdev= 0.00, samples=1 iops : min= 6, max= 6, avg= 6.00, stdev= 0.00, samples=1 lat (msec) : 50=41.67%, 100=41.67%, 250=16.67% cpu : usr=0.00%, sys=14.58%, ctx=47, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25987: Sat Feb 24 23:27:26 2024 read: IOPS=8, BW=8853KiB/s (9065kB/s)(6144KiB/694msec) clat (msec): min=23, max=110, avg=55.38, stdev=31.35 lat (msec): min=23, max=110, avg=55.38, stdev=31.35 clat percentiles (msec): | 1.00th=[ 23], 5.00th=[ 23], 10.00th=[ 23], 20.00th=[ 31], | 30.00th=[ 31], 40.00th=[ 45], 50.00th=[ 45], 60.00th=[ 62], | 70.00th=[ 63], 80.00th=[ 63], 90.00th=[ 111], 95.00th=[ 111], | 99.00th=[ 111], 99.50th=[ 111], 99.90th=[ 111], 99.95th=[ 111], | 99.99th=[ 111] bw ( KiB/s): min= 6144, max= 6144, per=25.14%, avg=6144.00, stdev= 0.00, samples=1 iops : min= 6, max= 6, avg= 6.00, stdev= 0.00, samples=1 write: IOPS=8, BW=8853KiB/s (9065kB/s)(6144KiB/694msec) clat (usec): min=35349, max=93032, avg=59988.25, stdev=22234.51 lat (usec): min=35423, max=93105, avg=60055.54, stdev=22224.69 clat percentiles (usec): | 1.00th=[35390], 5.00th=[35390], 10.00th=[35390], 20.00th=[43254], | 30.00th=[43254], 40.00th=[53216], 50.00th=[53216], 60.00th=[54789], | 70.00th=[80217], 80.00th=[80217], 90.00th=[92799], 95.00th=[92799], | 99.00th=[92799], 99.50th=[92799], 99.90th=[92799], 99.95th=[92799], | 99.99th=[92799] bw ( KiB/s): min= 8192, max= 8192, per=23.94%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 lat (msec) : 50=41.67%, 100=50.00%, 250=8.33% cpu : usr=0.14%, sys=18.47%, ctx=46, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25988: Sat Feb 24 23:27:26 2024 read: IOPS=6, BW=6282KiB/s (6433kB/s)(5120KiB/815msec) clat (usec): min=25012, max=64990, avg=40815.33, stdev=15995.05 lat (usec): min=25013, max=64990, avg=40816.19, stdev=15994.90 clat percentiles (usec): | 1.00th=[25035], 5.00th=[25035], 10.00th=[25035], 20.00th=[25035], | 30.00th=[27657], 40.00th=[27657], 50.00th=[42206], 60.00th=[42206], | 70.00th=[44303], 80.00th=[44303], 90.00th=[64750], 95.00th=[64750], | 99.00th=[64750], 99.50th=[64750], 99.90th=[64750], 99.95th=[64750], | 99.99th=[64750] write: IOPS=8, BW=8795KiB/s (9006kB/s)(7168KiB/815msec) clat (msec): min=34, max=192, avg=86.10, stdev=60.10 lat (msec): min=34, max=192, avg=86.18, stdev=60.09 clat percentiles (msec): | 1.00th=[ 35], 5.00th=[ 35], 10.00th=[ 35], 20.00th=[ 36], | 30.00th=[ 51], 40.00th=[ 51], 50.00th=[ 56], 60.00th=[ 94], | 70.00th=[ 94], 80.00th=[ 140], 90.00th=[ 192], 95.00th=[ 192], | 99.00th=[ 192], 99.50th=[ 192], 99.90th=[ 192], 99.95th=[ 192], | 99.99th=[ 192] bw ( KiB/s): min= 8192, max= 8192, per=23.94%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 lat (msec) : 50=50.00%, 100=33.33%, 250=16.67% cpu : usr=0.12%, sys=13.02%, ctx=79, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=23.9MiB/s (25.0MB/s), 2444KiB/s-10.4MiB/s (2503kB/s-10.9MB/s), io=20.0MiB (20.0MB), run=674-838msec WRITE: bw=33.4MiB/s (35.0MB/s), 7596KiB/s-11.9MiB/s (7779kB/s-12.5MB/s), io=28.0MiB (29.4MB), run=674-838msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=25994: Sat Feb 24 23:27:28 2024 read: IOPS=7, BW=30.8MiB/s (32.3MB/s)(12.0MiB/389msec) clat (msec): min=88, max=197, avg=129.46, stdev=59.47 lat (msec): min=88, max=197, avg=129.46, stdev=59.47 clat percentiles (msec): | 1.00th=[ 89], 5.00th=[ 89], 10.00th=[ 89], 20.00th=[ 89], | 30.00th=[ 89], 40.00th=[ 103], 50.00th=[ 103], 60.00th=[ 103], | 70.00th=[ 199], 80.00th=[ 199], 90.00th=[ 199], 95.00th=[ 199], | 99.00th=[ 199], 99.50th=[ 199], 99.90th=[ 199], 99.95th=[ 199], | 99.99th=[ 199] lat (msec) : 100=33.33%, 250=66.67% cpu : usr=0.00%, sys=63.92%, ctx=139, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25995: Sat Feb 24 23:27:28 2024 read: IOPS=1, BW=5919KiB/s (6061kB/s)(4096KiB/692msec) clat (nsec): min=166976k, max=166976k, avg=166975921.00, stdev= 0.00 lat (nsec): min=166978k, max=166978k, avg=166977811.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 167], 5.00th=[ 167], 10.00th=[ 167], 20.00th=[ 167], | 30.00th=[ 167], 40.00th=[ 167], 50.00th=[ 167], 60.00th=[ 167], | 70.00th=[ 167], 80.00th=[ 167], 90.00th=[ 167], 95.00th=[ 167], | 99.00th=[ 167], 99.50th=[ 167], 99.90th=[ 167], 99.95th=[ 167], | 99.99th=[ 167] bw ( KiB/s): min= 8192, max= 8192, per=34.60%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=2, BW=11.6MiB/s (12.1MB/s)(8192KiB/692msec) clat (msec): min=210, max=312, avg=261.69, stdev=72.19 lat (msec): min=210, max=312, avg=261.89, stdev=72.07 clat percentiles (msec): | 1.00th=[ 211], 5.00th=[ 211], 10.00th=[ 211], 20.00th=[ 211], | 30.00th=[ 211], 40.00th=[ 211], 50.00th=[ 211], 60.00th=[ 313], | 70.00th=[ 313], 80.00th=[ 313], 90.00th=[ 313], 95.00th=[ 313], | 99.00th=[ 313], 99.50th=[ 313], 99.90th=[ 313], 99.95th=[ 313], | 99.99th=[ 313] bw ( KiB/s): min= 8192, max= 8192, per=17.30%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.00%, sys=28.80%, ctx=91, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25997: Sat Feb 24 23:27:28 2024 write: IOPS=4, BW=18.2MiB/s (19.1MB/s)(12.0MiB/660msec) clat (msec): min=95, max=283, avg=218.26, stdev=106.55 lat (msec): min=95, max=284, avg=218.51, stdev=106.54 clat percentiles (msec): | 1.00th=[ 95], 5.00th=[ 95], 10.00th=[ 95], 20.00th=[ 95], | 30.00th=[ 95], 40.00th=[ 275], 50.00th=[ 275], 60.00th=[ 275], | 70.00th=[ 284], 80.00th=[ 284], 90.00th=[ 284], 95.00th=[ 284], | 99.00th=[ 284], 99.50th=[ 284], 99.90th=[ 284], 99.95th=[ 284], | 99.99th=[ 284] bw ( KiB/s): min= 8192, max= 8192, per=17.30%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 500=66.67% cpu : usr=0.30%, sys=30.35%, ctx=115, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25999: Sat Feb 24 23:27:28 2024 write: IOPS=4, BW=19.2MiB/s (20.1MB/s)(12.0MiB/626msec) clat (msec): min=193, max=223, avg=208.07, stdev=15.40 lat (msec): min=193, max=224, avg=208.26, stdev=15.39 clat percentiles (msec): | 1.00th=[ 194], 5.00th=[ 194], 10.00th=[ 194], 20.00th=[ 194], | 30.00th=[ 194], 40.00th=[ 207], 50.00th=[ 207], 60.00th=[ 207], | 70.00th=[ 224], 80.00th=[ 224], 90.00th=[ 224], 95.00th=[ 224], | 99.00th=[ 224], 99.50th=[ 224], 99.90th=[ 224], 99.95th=[ 224], | 99.99th=[ 224] bw ( KiB/s): min=16384, max=16384, per=34.60%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 250=100.00% cpu : usr=0.00%, sys=30.72%, ctx=78, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=23.1MiB/s (24.2MB/s), 5919KiB/s-30.8MiB/s (6061kB/s-32.3MB/s), io=16.0MiB (16.8MB), run=389-692msec WRITE: bw=46.2MiB/s (48.5MB/s), 11.6MiB/s-19.2MiB/s (12.1MB/s-20.1MB/s), io=32.0MiB (33.6MB), run=626-692msec rand-rw: (groupid=0, jobs=1): err= 0: pid=25996: Sat Feb 24 23:27:28 2024 read: IOPS=5, BW=21.9MiB/s (22.0MB/s)(12.0MiB/548msec) clat (msec): min=131, max=235, avg=182.11, stdev=52.02 lat (msec): min=131, max=235, avg=182.11, stdev=52.02 clat percentiles (msec): | 1.00th=[ 132], 5.00th=[ 132], 10.00th=[ 132], 20.00th=[ 132], | 30.00th=[ 132], 40.00th=[ 180], 50.00th=[ 180], 60.00th=[ 180], | 70.00th=[ 236], 80.00th=[ 236], 90.00th=[ 236], 95.00th=[ 236], | 99.00th=[ 236], 99.50th=[ 236], 99.90th=[ 236], 99.95th=[ 236], | 99.99th=[ 236] bw ( KiB/s): min=16351, max=16351, per=74.45%, avg=16351.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 lat (msec) : 250=100.00% cpu : usr=0.00%, sys=14.63%, ctx=36, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25998: Sat Feb 24 23:27:28 2024 read: IOPS=1, BW=5491KiB/s (5622kB/s)(4096KiB/746msec) clat (nsec): min=296884k, max=296884k, avg=296883719.00, stdev= 0.00 lat (nsec): min=296886k, max=296886k, avg=296885549.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 296], 5.00th=[ 296], 10.00th=[ 296], 20.00th=[ 296], | 30.00th=[ 296], 40.00th=[ 296], 50.00th=[ 296], 60.00th=[ 296], | 70.00th=[ 296], 80.00th=[ 296], 90.00th=[ 296], 95.00th=[ 296], | 99.00th=[ 296], 99.50th=[ 296], 99.90th=[ 296], 99.95th=[ 296], | 99.99th=[ 296] bw ( KiB/s): min= 8192, max= 8192, per=37.30%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=2, BW=10.7MiB/s (11.2MB/s)(8192KiB/746msec) clat (msec): min=153, max=276, avg=214.77, stdev=86.72 lat (msec): min=153, max=276, avg=215.00, stdev=86.76 clat percentiles (msec): | 1.00th=[ 155], 5.00th=[ 155], 10.00th=[ 155], 20.00th=[ 155], | 30.00th=[ 155], 40.00th=[ 155], 50.00th=[ 155], 60.00th=[ 275], | 70.00th=[ 275], 80.00th=[ 275], 90.00th=[ 275], 95.00th=[ 275], | 99.00th=[ 275], 99.50th=[ 275], 99.90th=[ 275], 99.95th=[ 275], | 99.99th=[ 275] bw ( KiB/s): min= 8192, max= 8192, per=20.63%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=66.67% cpu : usr=0.00%, sys=15.70%, ctx=48, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=26000: Sat Feb 24 23:27:28 2024 write: IOPS=3, BW=14.5MiB/s (15.3MB/s)(12.0MiB/825msec) clat (msec): min=85, max=508, avg=273.85, stdev=214.86 lat (msec): min=85, max=508, avg=274.02, stdev=214.93 clat percentiles (msec): | 1.00th=[ 86], 5.00th=[ 86], 10.00th=[ 86], 20.00th=[ 86], | 30.00th=[ 86], 40.00th=[ 228], 50.00th=[ 228], 60.00th=[ 228], | 70.00th=[ 510], 80.00th=[ 510], 90.00th=[ 510], 95.00th=[ 510], | 99.00th=[ 510], 99.50th=[ 510], 99.90th=[ 510], 99.95th=[ 510], | 99.99th=[ 510] lat (msec) : 100=33.33%, 250=33.33%, 750=33.33% cpu : usr=0.00%, sys=13.47%, ctx=49, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=26001: Sat Feb 24 23:27:28 2024 write: IOPS=4, BW=17.8MiB/s (18.6MB/s)(12.0MiB/676msec) clat (msec): min=175, max=253, avg=218.43, stdev=39.87 lat (msec): min=175, max=253, avg=218.72, stdev=39.86 clat percentiles (msec): | 1.00th=[ 176], 5.00th=[ 176], 10.00th=[ 176], 20.00th=[ 176], | 30.00th=[ 176], 40.00th=[ 228], 50.00th=[ 228], 60.00th=[ 228], | 70.00th=[ 253], 80.00th=[ 253], 90.00th=[ 253], 95.00th=[ 253], | 99.00th=[ 253], 99.50th=[ 253], 99.90th=[ 253], 99.95th=[ 253], | 99.99th=[ 253] bw ( KiB/s): min=16384, max=16384, per=41.25%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.00%, sys=16.00%, ctx=46, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=21.4MiB/s (22.5MB/s), 5491KiB/s-21.9MiB/s (5622kB/s-22.0MB/s), io=16.0MiB (16.8MB), run=548-746msec WRITE: bw=38.8MiB/s (40.7MB/s), 10.7MiB/s-17.8MiB/s (11.2MB/s-18.6MB/s), io=32.0MiB (33.6MB), run=676-825msec