== sanity test 398b: DIO and buffer IO race ============== 23:26:05 (1708835165) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 0.978631 s, 51.4 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=25040: Sat Feb 24 23:26:20 2024 read: IOPS=115, BW=463KiB/s (474kB/s)(5992KiB/12934msec) clat (usec): min=41, max=144344, avg=3642.02, stdev=4996.43 lat (usec): min=41, max=144345, avg=3642.81, stdev=4996.47 clat percentiles (usec): | 1.00th=[ 49], 5.00th=[ 66], 10.00th=[ 71], 20.00th=[ 115], | 30.00th=[ 1991], 40.00th=[ 2376], 50.00th=[ 2769], 60.00th=[ 3359], | 70.00th=[ 4293], 80.00th=[ 5800], 90.00th=[ 7963], 95.00th=[ 9503], | 99.00th=[ 14484], 99.50th=[ 18482], 99.90th=[ 40109], 99.95th=[143655], | 99.99th=[143655] bw ( KiB/s): min= 280, max= 704, per=25.22%, avg=460.08, stdev=97.19, samples=25 iops : min= 70, max= 176, avg=114.96, stdev=24.32, samples=25 write: IOPS=121, BW=487KiB/s (498kB/s)(6296KiB/12934msec) clat (usec): min=391, max=39397, avg=4681.61, stdev=4459.18 lat (usec): min=392, max=39398, avg=4682.53, stdev=4459.21 clat percentiles (usec): | 1.00th=[ 453], 5.00th=[ 486], 10.00th=[ 529], 20.00th=[ 685], | 30.00th=[ 1090], 40.00th=[ 2573], 50.00th=[ 3621], 60.00th=[ 4817], | 70.00th=[ 5997], 80.00th=[ 7767], 90.00th=[10814], 95.00th=[13435], | 99.00th=[18220], 99.50th=[21890], 99.90th=[29230], 99.95th=[39584], | 99.99th=[39584] bw ( KiB/s): min= 328, max= 728, per=25.21%, avg=481.52, stdev=97.81, samples=25 iops : min= 82, max= 182, avg=120.32, stdev=24.47, samples=25 lat (usec) : 50=0.52%, 100=8.24%, 250=2.51%, 500=3.71%, 750=8.07% lat (usec) : 1000=2.90% lat (msec) : 2=8.04%, 4=26.24%, 10=31.58%, 20=7.62%, 50=0.55% lat (msec) : 250=0.03% cpu : usr=0.20%, sys=23.05%, ctx=5923, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25041: Sat Feb 24 23:26:20 2024 read: IOPS=114, BW=459KiB/s (470kB/s)(5940KiB/12939msec) clat (usec): min=41, max=28674, avg=3511.09, stdev=3173.43 lat (usec): min=41, max=28675, avg=3512.30, stdev=3173.12 clat percentiles (usec): | 1.00th=[ 47], 5.00th=[ 64], 10.00th=[ 71], 20.00th=[ 115], | 30.00th=[ 2040], 40.00th=[ 2409], 50.00th=[ 2802], 60.00th=[ 3326], | 70.00th=[ 4178], 80.00th=[ 5604], 90.00th=[ 7767], 95.00th=[ 9634], | 99.00th=[13960], 99.50th=[15401], 99.90th=[20841], 99.95th=[28705], | 99.99th=[28705] bw ( KiB/s): min= 240, max= 712, per=25.06%, avg=457.04, stdev=97.11, samples=25 iops : min= 60, max= 178, avg=114.16, stdev=24.26, samples=25 write: IOPS=122, BW=491KiB/s (502kB/s)(6348KiB/12939msec) clat (usec): min=419, max=145719, avg=4801.84, stdev=6251.32 lat (usec): min=420, max=145721, avg=4803.65, stdev=6251.32 clat percentiles (usec): | 1.00th=[ 449], 5.00th=[ 494], 10.00th=[ 529], 20.00th=[ 652], | 30.00th=[ 988], 40.00th=[ 2180], 50.00th=[ 3654], 60.00th=[ 4752], | 70.00th=[ 5932], 80.00th=[ 7767], 90.00th=[ 10945], 95.00th=[ 13566], | 99.00th=[ 19792], 99.50th=[ 24773], 99.90th=[ 81265], 99.95th=[145753], | 99.99th=[145753] bw ( KiB/s): min= 288, max= 800, per=25.22%, avg=481.68, stdev=118.90, samples=25 iops : min= 72, max= 200, avg=120.32, stdev=29.71, samples=25 lat (usec) : 50=0.72%, 100=7.78%, 250=2.44%, 500=3.42%, 750=9.67% lat (usec) : 1000=2.86% lat (msec) : 2=7.26%, 4=26.20%, 10=31.25%, 20=7.85%, 50=0.46% lat (msec) : 100=0.07%, 250=0.03% cpu : usr=0.19%, sys=22.85%, ctx=5985, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25042: Sat Feb 24 23:26:20 2024 read: IOPS=114, BW=458KiB/s (469kB/s)(6028KiB/13159msec) clat (usec): min=42, max=145107, avg=3669.53, stdev=5389.81 lat (usec): min=42, max=145108, avg=3670.98, stdev=5389.60 clat percentiles (usec): | 1.00th=[ 49], 5.00th=[ 66], 10.00th=[ 71], 20.00th=[ 98], | 30.00th=[ 1926], 40.00th=[ 2343], 50.00th=[ 2737], 60.00th=[ 3228], | 70.00th=[ 4146], 80.00th=[ 5800], 90.00th=[ 8029], 95.00th=[ 10159], | 99.00th=[ 15008], 99.50th=[ 22152], 99.90th=[ 70779], 99.95th=[145753], | 99.99th=[145753] bw ( KiB/s): min= 328, max= 794, per=24.83%, avg=452.85, stdev=110.18, samples=26 iops : min= 82, max= 198, avg=113.12, stdev=27.52, samples=26 write: IOPS=118, BW=476KiB/s (487kB/s)(6260KiB/13159msec) clat (usec): min=416, max=30687, avg=4806.29, stdev=4640.29 lat (usec): min=417, max=30688, avg=4807.28, stdev=4640.18 clat percentiles (usec): | 1.00th=[ 441], 5.00th=[ 478], 10.00th=[ 510], 20.00th=[ 652], | 30.00th=[ 1074], 40.00th=[ 2474], 50.00th=[ 3752], 60.00th=[ 4817], | 70.00th=[ 6063], 80.00th=[ 8094], 90.00th=[11338], 95.00th=[13960], | 99.00th=[20317], 99.50th=[22414], 99.90th=[30540], 99.95th=[30802], | 99.99th=[30802] bw ( KiB/s): min= 271, max= 846, per=24.59%, avg=469.62, stdev=115.89, samples=26 iops : min= 67, max= 211, avg=117.31, stdev=28.96, samples=26 lat (usec) : 50=0.68%, 100=9.31%, 250=2.28%, 500=4.49%, 750=7.91% lat (usec) : 1000=2.67% lat (msec) : 2=7.39%, 4=25.68%, 10=30.40%, 20=8.30%, 50=0.81% lat (msec) : 100=0.03%, 250=0.03% cpu : usr=0.27%, sys=22.17%, ctx=5927, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25043: Sat Feb 24 23:26:20 2024 read: IOPS=116, BW=466KiB/s (478kB/s)(6052KiB/12977msec) clat (usec): min=39, max=39219, avg=3513.80, stdev=3415.96 lat (usec): min=39, max=39220, avg=3515.39, stdev=3415.64 clat percentiles (usec): | 1.00th=[ 46], 5.00th=[ 63], 10.00th=[ 69], 20.00th=[ 95], | 30.00th=[ 1876], 40.00th=[ 2311], 50.00th=[ 2737], 60.00th=[ 3359], | 70.00th=[ 4293], 80.00th=[ 5866], 90.00th=[ 8094], 95.00th=[ 9634], | 99.00th=[13566], 99.50th=[15664], 99.90th=[29230], 99.95th=[39060], | 99.99th=[39060] bw ( KiB/s): min= 224, max= 640, per=25.05%, avg=456.96, stdev=95.07, samples=25 iops : min= 56, max= 160, avg=114.08, stdev=23.73, samples=25 write: IOPS=120, BW=481KiB/s (492kB/s)(6236KiB/12977msec) clat (usec): min=426, max=210956, avg=4847.41, stdev=7084.54 lat (usec): min=427, max=210958, avg=4848.45, stdev=7084.82 clat percentiles (usec): | 1.00th=[ 449], 5.00th=[ 486], 10.00th=[ 529], 20.00th=[ 660], | 30.00th=[ 930], 40.00th=[ 1991], 50.00th=[ 3556], 60.00th=[ 4621], | 70.00th=[ 6128], 80.00th=[ 7963], 90.00th=[ 11076], 95.00th=[ 13829], | 99.00th=[ 20055], 99.50th=[ 27919], 99.90th=[ 42730], 99.95th=[210764], | 99.99th=[210764] bw ( KiB/s): min= 256, max= 654, per=24.63%, avg=470.40, stdev=98.69, samples=25 iops : min= 64, max= 163, avg=117.44, stdev=24.69, samples=25 lat (usec) : 50=1.27%, 100=8.92%, 250=2.41%, 500=3.58%, 750=9.21% lat (usec) : 1000=3.39% lat (msec) : 2=7.58%, 4=24.58%, 10=30.24%, 20=8.20%, 50=0.59% lat (msec) : 250=0.03% cpu : usr=0.18%, sys=22.36%, ctx=5867, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1825KiB/s (1869kB/s), 458KiB/s-466KiB/s (469kB/s-478kB/s), io=23.4MiB (24.6MB), run=12934-13159msec WRITE: bw=1910KiB/s (1956kB/s), 476KiB/s-491KiB/s (487kB/s-502kB/s), io=24.6MiB (25.7MB), run=12934-13159msec rand-rw: (groupid=0, jobs=1): err= 0: pid=25036: Sat Feb 24 23:26:45 2024 read: IOPS=39, BW=158KiB/s (162kB/s)(5992KiB/37870msec) clat (usec): min=1549, max=150754, avg=3493.49, stdev=4884.04 lat (usec): min=1550, max=150756, avg=3494.37, stdev=4884.08 clat percentiles (usec): | 1.00th=[ 1680], 5.00th=[ 1795], 10.00th=[ 1844], 20.00th=[ 1926], | 30.00th=[ 2008], 40.00th=[ 2089], 50.00th=[ 2180], 60.00th=[ 2311], | 70.00th=[ 2606], 80.00th=[ 3425], 90.00th=[ 7767], 95.00th=[ 10552], | 99.00th=[ 15401], 99.50th=[ 17171], 99.90th=[ 40109], 99.95th=[149947], | 99.99th=[149947] bw ( KiB/s): min= 47, max= 280, per=24.96%, avg=157.76, stdev=64.27, samples=75 iops : min= 11, max= 70, avg=39.36, stdev=16.07, samples=75 write: IOPS=41, BW=166KiB/s (170kB/s)(6296KiB/37870msec) clat (msec): min=10, max=101, avg=20.66, stdev= 8.22 lat (msec): min=10, max=101, avg=20.66, stdev= 8.22 clat percentiles (msec): | 1.00th=[ 12], 5.00th=[ 13], 10.00th=[ 13], 20.00th=[ 14], | 30.00th=[ 16], 40.00th=[ 18], 50.00th=[ 20], 60.00th=[ 21], | 70.00th=[ 22], 80.00th=[ 24], 90.00th=[ 33], 95.00th=[ 39], | 99.00th=[ 48], 99.50th=[ 53], 99.90th=[ 88], 99.95th=[ 102], | 99.99th=[ 102] bw ( KiB/s): min= 48, max= 248, per=25.03%, avg=165.72, stdev=52.26, samples=75 iops : min= 12, max= 62, avg=41.35, stdev=13.07, samples=75 lat (msec) : 2=14.32%, 4=25.65%, 10=5.83%, 20=32.00%, 50=21.91% lat (msec) : 100=0.23%, 250=0.07% cpu : usr=0.14%, sys=4.52%, ctx=3361, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25037: Sat Feb 24 23:26:45 2024 read: IOPS=39, BW=157KiB/s (160kB/s)(5940KiB/37938msec) clat (usec): min=1500, max=170646, avg=3578.40, stdev=5764.98 lat (usec): min=1500, max=170648, avg=3579.25, stdev=5765.01 clat percentiles (usec): | 1.00th=[ 1663], 5.00th=[ 1762], 10.00th=[ 1827], 20.00th=[ 1926], | 30.00th=[ 2008], 40.00th=[ 2089], 50.00th=[ 2180], 60.00th=[ 2311], | 70.00th=[ 2573], 80.00th=[ 3326], 90.00th=[ 7504], 95.00th=[ 10683], | 99.00th=[ 17433], 99.50th=[ 22414], 99.90th=[ 60031], 99.95th=[170918], | 99.99th=[170918] bw ( KiB/s): min= 16, max= 312, per=24.76%, avg=156.47, stdev=68.32, samples=75 iops : min= 4, max= 78, avg=39.03, stdev=17.03, samples=75 write: IOPS=41, BW=167KiB/s (171kB/s)(6348KiB/37938msec) clat (usec): min=10938, max=59922, avg=20484.22, stdev=7850.84 lat (usec): min=10939, max=59923, avg=20485.19, stdev=7850.88 clat percentiles (usec): | 1.00th=[11338], 5.00th=[12125], 10.00th=[12780], 20.00th=[13960], | 30.00th=[15533], 40.00th=[17433], 50.00th=[19006], 60.00th=[20317], | 70.00th=[21627], 80.00th=[23987], 90.00th=[31851], 95.00th=[37487], | 99.00th=[49021], 99.50th=[50594], 99.90th=[57934], 99.95th=[60031], | 99.99th=[60031] bw ( KiB/s): min= 64, max= 224, per=25.13%, avg=166.35, stdev=51.91, samples=75 iops : min= 16, max= 56, avg=41.49, stdev=13.00, samples=75 lat (msec) : 2=14.29%, 4=25.94%, 10=5.08%, 20=32.42%, 50=21.87% lat (msec) : 100=0.36%, 250=0.03% cpu : usr=0.13%, sys=4.52%, ctx=3359, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25038: Sat Feb 24 23:26:45 2024 read: IOPS=39, BW=159KiB/s (163kB/s)(6028KiB/37899msec) clat (usec): min=1570, max=35821, avg=3480.05, stdev=3255.12 lat (usec): min=1570, max=35822, avg=3481.08, stdev=3255.13 clat percentiles (usec): | 1.00th=[ 1663], 5.00th=[ 1778], 10.00th=[ 1827], 20.00th=[ 1926], | 30.00th=[ 2008], 40.00th=[ 2089], 50.00th=[ 2180], 60.00th=[ 2311], | 70.00th=[ 2573], 80.00th=[ 3982], 90.00th=[ 7635], 95.00th=[10421], | 99.00th=[16909], 99.50th=[19530], 99.90th=[25822], 99.95th=[35914], | 99.99th=[35914] bw ( KiB/s): min= 32, max= 320, per=25.11%, avg=158.72, stdev=66.23, samples=75 iops : min= 8, max= 80, avg=39.63, stdev=16.53, samples=75 write: IOPS=41, BW=165KiB/s (169kB/s)(6260KiB/37899msec) clat (msec): min=10, max=186, avg=20.79, stdev= 9.19 lat (msec): min=10, max=186, avg=20.79, stdev= 9.19 clat percentiles (msec): | 1.00th=[ 12], 5.00th=[ 13], 10.00th=[ 13], 20.00th=[ 15], | 30.00th=[ 17], 40.00th=[ 18], 50.00th=[ 20], 60.00th=[ 21], | 70.00th=[ 22], 80.00th=[ 24], 90.00th=[ 33], 95.00th=[ 39], | 99.00th=[ 48], 99.50th=[ 55], 99.90th=[ 102], 99.95th=[ 188], | 99.99th=[ 188] bw ( KiB/s): min= 56, max= 223, per=24.83%, avg=164.37, stdev=53.18, samples=75 iops : min= 14, max= 55, avg=41.04, stdev=13.25, samples=75 lat (msec) : 2=14.10%, 4=25.20%, 10=7.06%, 20=31.41%, 50=21.84% lat (msec) : 100=0.33%, 250=0.07% cpu : usr=0.12%, sys=4.56%, ctx=3337, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25039: Sat Feb 24 23:26:45 2024 read: IOPS=40, BW=162KiB/s (166kB/s)(6052KiB/37357msec) clat (usec): min=1541, max=39085, avg=3460.54, stdev=3418.89 lat (usec): min=1541, max=39085, avg=3461.37, stdev=3418.90 clat percentiles (usec): | 1.00th=[ 1680], 5.00th=[ 1795], 10.00th=[ 1844], 20.00th=[ 1926], | 30.00th=[ 2008], 40.00th=[ 2089], 50.00th=[ 2180], 60.00th=[ 2311], | 70.00th=[ 2540], 80.00th=[ 3556], 90.00th=[ 7439], 95.00th=[10421], | 99.00th=[16581], 99.50th=[21627], 99.90th=[35914], 99.95th=[39060], | 99.99th=[39060] bw ( KiB/s): min= 16, max= 320, per=25.44%, avg=160.80, stdev=63.91, samples=74 iops : min= 4, max= 80, avg=40.15, stdev=15.96, samples=74 write: IOPS=41, BW=167KiB/s (171kB/s)(6236KiB/37357msec) clat (msec): min=10, max=208, avg=20.53, stdev= 9.34 lat (msec): min=10, max=209, avg=20.53, stdev= 9.34 clat percentiles (msec): | 1.00th=[ 12], 5.00th=[ 13], 10.00th=[ 13], 20.00th=[ 14], | 30.00th=[ 16], 40.00th=[ 18], 50.00th=[ 20], 60.00th=[ 21], | 70.00th=[ 22], 80.00th=[ 24], 90.00th=[ 32], 95.00th=[ 37], | 99.00th=[ 47], 99.50th=[ 55], 99.90th=[ 104], 99.95th=[ 209], | 99.99th=[ 209] bw ( KiB/s): min= 56, max= 232, per=25.11%, avg=166.20, stdev=51.90, samples=74 iops : min= 14, max= 58, avg=41.50, stdev=12.94, samples=74 lat (msec) : 2=14.78%, 4=25.49%, 10=6.18%, 20=31.45%, 50=21.68% lat (msec) : 100=0.36%, 250=0.07% cpu : usr=0.11%, sys=4.61%, ctx=3354, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=633KiB/s (648kB/s), 157KiB/s-162KiB/s (160kB/s-166kB/s), io=23.4MiB (24.6MB), run=37357-37938msec WRITE: bw=663KiB/s (679kB/s), 165KiB/s-167KiB/s (169kB/s-171kB/s), io=24.6MiB (25.7MB), run=37357-37938msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=25085: Sat Feb 24 23:26:48 2024 read: IOPS=161, BW=2591KiB/s (2653kB/s)(5920KiB/2285msec) clat (usec): min=43, max=11310, avg=2570.30, stdev=2495.89 lat (usec): min=43, max=11311, avg=2571.28, stdev=2495.57 clat percentiles (usec): | 1.00th=[ 48], 5.00th=[ 57], 10.00th=[ 64], 20.00th=[ 72], | 30.00th=[ 92], 40.00th=[ 1975], 50.00th=[ 2704], 60.00th=[ 3064], | 70.00th=[ 3490], 80.00th=[ 4293], 90.00th=[ 5866], 95.00th=[ 7570], | 99.00th=[10159], 99.50th=[10814], 99.90th=[11338], 99.95th=[11338], | 99.99th=[11338] bw ( KiB/s): min= 2208, max= 2970, per=26.54%, avg=2645.00, stdev=359.13, samples=4 iops : min= 138, max= 185, avg=165.00, stdev=22.11, samples=4 write: IOPS=174, BW=2787KiB/s (2854kB/s)(6368KiB/2285msec) clat (usec): min=582, max=33001, avg=3335.54, stdev=3636.69 lat (usec): min=583, max=33002, avg=3336.46, stdev=3636.70 clat percentiles (usec): | 1.00th=[ 619], 5.00th=[ 668], 10.00th=[ 701], 20.00th=[ 775], | 30.00th=[ 881], 40.00th=[ 1106], 50.00th=[ 1614], 60.00th=[ 2802], | 70.00th=[ 4178], 80.00th=[ 5932], 90.00th=[ 7832], 95.00th=[ 9503], | 99.00th=[16188], 99.50th=[23725], 99.90th=[32900], 99.95th=[32900], | 99.99th=[32900] bw ( KiB/s): min= 2368, max= 3640, per=27.51%, avg=2884.25, stdev=571.03, samples=4 iops : min= 148, max= 227, avg=180.00, stdev=35.43, samples=4 lat (usec) : 50=0.91%, 100=14.84%, 250=2.86%, 500=0.39%, 750=8.46% lat (usec) : 1000=9.51% lat (msec) : 2=9.77%, 4=26.04%, 10=24.48%, 20=2.47%, 50=0.26% cpu : usr=0.04%, sys=29.33%, ctx=1578, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25086: Sat Feb 24 23:26:48 2024 read: IOPS=149, BW=2390KiB/s (2448kB/s)(5744KiB/2403msec) clat (usec): min=47, max=17088, avg=2769.31, stdev=2835.54 lat (usec): min=47, max=17089, avg=2770.41, stdev=2836.21 clat percentiles (usec): | 1.00th=[ 51], 5.00th=[ 59], 10.00th=[ 67], 20.00th=[ 72], | 30.00th=[ 90], 40.00th=[ 306], 50.00th=[ 2671], 60.00th=[ 3228], | 70.00th=[ 3916], 80.00th=[ 5080], 90.00th=[ 6259], 95.00th=[ 7439], | 99.00th=[ 9896], 99.50th=[17171], 99.90th=[17171], 99.95th=[17171], | 99.99th=[17171] bw ( KiB/s): min= 1920, max= 3040, per=23.90%, avg=2382.50, stdev=519.60, samples=4 iops : min= 120, max= 190, avg=148.75, stdev=32.41, samples=4 write: IOPS=170, BW=2723KiB/s (2789kB/s)(6544KiB/2403msec) clat (usec): min=565, max=38131, avg=3421.75, stdev=3749.82 lat (usec): min=566, max=38132, avg=3422.69, stdev=3749.83 clat percentiles (usec): | 1.00th=[ 611], 5.00th=[ 668], 10.00th=[ 709], 20.00th=[ 783], | 30.00th=[ 914], 40.00th=[ 1172], 50.00th=[ 1811], 60.00th=[ 3097], | 70.00th=[ 4178], 80.00th=[ 5997], 90.00th=[ 7701], 95.00th=[ 9372], | 99.00th=[15008], 99.50th=[19268], 99.90th=[38011], 99.95th=[38011], | 99.99th=[38011] bw ( KiB/s): min= 2080, max= 3321, per=25.39%, avg=2662.25, stdev=560.29, samples=4 iops : min= 130, max= 207, avg=166.25, stdev=34.80, samples=4 lat (usec) : 50=0.26%, 100=15.23%, 250=3.12%, 500=0.26%, 750=8.33% lat (usec) : 1000=10.29% lat (msec) : 2=9.11%, 4=23.05%, 10=27.34%, 20=2.73%, 50=0.26% cpu : usr=0.08%, sys=29.56%, ctx=1493, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25087: Sat Feb 24 23:26:48 2024 read: IOPS=170, BW=2735KiB/s (2801kB/s)(6256KiB/2287msec) clat (usec): min=45, max=21521, avg=2702.26, stdev=3094.33 lat (usec): min=46, max=21521, avg=2703.90, stdev=3094.30 clat percentiles (usec): | 1.00th=[ 49], 5.00th=[ 52], 10.00th=[ 60], 20.00th=[ 70], | 30.00th=[ 78], 40.00th=[ 119], 50.00th=[ 2474], 60.00th=[ 2966], | 70.00th=[ 3916], 80.00th=[ 5014], 90.00th=[ 6652], 95.00th=[ 8094], | 99.00th=[14353], 99.50th=[15926], 99.90th=[21627], 99.95th=[21627], | 99.99th=[21627] bw ( KiB/s): min= 2080, max= 3161, per=27.90%, avg=2780.75, stdev=479.13, samples=4 iops : min= 130, max= 197, avg=173.50, stdev=29.72, samples=4 write: IOPS=164, BW=2638KiB/s (2701kB/s)(6032KiB/2287msec) clat (usec): min=570, max=19710, avg=3246.26, stdev=3169.94 lat (usec): min=571, max=19711, avg=3247.31, stdev=3169.89 clat percentiles (usec): | 1.00th=[ 586], 5.00th=[ 652], 10.00th=[ 693], 20.00th=[ 791], | 30.00th=[ 938], 40.00th=[ 1139], 50.00th=[ 1647], 60.00th=[ 3032], | 70.00th=[ 4080], 80.00th=[ 6063], 90.00th=[ 7767], 95.00th=[ 9372], | 99.00th=[13435], 99.50th=[15795], 99.90th=[19792], 99.95th=[19792], | 99.99th=[19792] bw ( KiB/s): min= 2363, max= 3065, per=25.61%, avg=2685.00, stdev=299.00, samples=4 iops : min= 147, max= 191, avg=167.50, stdev=18.70, samples=4 lat (usec) : 50=1.56%, 100=16.67%, 250=4.30%, 500=0.39%, 750=8.33% lat (usec) : 1000=8.07% lat (msec) : 2=10.68%, 4=20.31%, 10=26.56%, 20=2.99%, 50=0.13% cpu : usr=0.26%, sys=28.61%, ctx=1486, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25088: Sat Feb 24 23:26:48 2024 read: IOPS=161, BW=2590KiB/s (2652kB/s)(6032KiB/2329msec) clat (usec): min=45, max=37549, avg=2810.29, stdev=3393.39 lat (usec): min=45, max=37550, avg=2810.93, stdev=3393.40 clat percentiles (usec): | 1.00th=[ 50], 5.00th=[ 58], 10.00th=[ 67], 20.00th=[ 70], | 30.00th=[ 86], 40.00th=[ 149], 50.00th=[ 2671], 60.00th=[ 3261], | 70.00th=[ 4080], 80.00th=[ 4948], 90.00th=[ 6390], 95.00th=[ 7504], | 99.00th=[11207], 99.50th=[20579], 99.90th=[37487], 99.95th=[37487], | 99.99th=[37487] bw ( KiB/s): min= 1984, max= 3289, per=26.55%, avg=2646.25, stdev=572.41, samples=4 iops : min= 124, max= 205, avg=165.25, stdev=35.57, samples=4 write: IOPS=167, BW=2686KiB/s (2751kB/s)(6256KiB/2329msec) clat (usec): min=573, max=28749, avg=3229.56, stdev=3271.73 lat (usec): min=574, max=28750, avg=3231.16, stdev=3271.27 clat percentiles (usec): | 1.00th=[ 586], 5.00th=[ 619], 10.00th=[ 693], 20.00th=[ 775], | 30.00th=[ 922], 40.00th=[ 1074], 50.00th=[ 1483], 60.00th=[ 2868], | 70.00th=[ 4359], 80.00th=[ 5735], 90.00th=[ 8029], 95.00th=[ 9110], | 99.00th=[12256], 99.50th=[15664], 99.90th=[28705], 99.95th=[28705], | 99.99th=[28705] bw ( KiB/s): min= 2144, max= 3328, per=26.46%, avg=2774.50, stdev=502.05, samples=4 iops : min= 134, max= 208, avg=173.25, stdev=31.30, samples=4 lat (usec) : 50=0.52%, 100=15.49%, 250=4.30%, 500=0.39%, 750=8.59% lat (usec) : 1000=9.77% lat (msec) : 2=10.03%, 4=18.62%, 10=29.69%, 20=2.21%, 50=0.39% cpu : usr=0.17%, sys=28.74%, ctx=1545, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=9968KiB/s (10.2MB/s), 2390KiB/s-2735KiB/s (2448kB/s-2801kB/s), io=23.4MiB (24.5MB), run=2285-2403msec WRITE: bw=10.2MiB/s (10.7MB/s), 2638KiB/s-2787KiB/s (2701kB/s-2854kB/s), io=24.6MiB (25.8MB), run=2285-2403msec rand-rw: (groupid=0, jobs=1): err= 0: pid=25089: Sat Feb 24 23:26:56 2024 read: IOPS=38, BW=608KiB/s (623kB/s)(5920KiB/9730msec) clat (usec): min=1890, max=21087, avg=3816.52, stdev=2790.45 lat (usec): min=1890, max=21088, avg=3817.35, stdev=2790.45 clat percentiles (usec): | 1.00th=[ 1958], 5.00th=[ 2089], 10.00th=[ 2147], 20.00th=[ 2311], | 30.00th=[ 2409], 40.00th=[ 2507], 50.00th=[ 2606], 60.00th=[ 2868], | 70.00th=[ 3458], 80.00th=[ 5014], 90.00th=[ 6849], 95.00th=[ 9503], | 99.00th=[16450], 99.50th=[16909], 99.90th=[21103], 99.95th=[21103], | 99.99th=[21103] bw ( KiB/s): min= 352, max= 1056, per=24.28%, avg=597.53, stdev=187.68, samples=19 iops : min= 22, max= 66, avg=37.32, stdev=11.72, samples=19 write: IOPS=40, BW=654KiB/s (670kB/s)(6368KiB/9730msec) clat (usec): min=12374, max=79038, avg=20879.75, stdev=6523.29 lat (usec): min=12375, max=79039, avg=20881.38, stdev=6523.29 clat percentiles (usec): | 1.00th=[12518], 5.00th=[13566], 10.00th=[14222], 20.00th=[15795], | 30.00th=[16909], 40.00th=[18744], 50.00th=[20579], 60.00th=[21627], | 70.00th=[22676], 80.00th=[24249], 90.00th=[27395], 95.00th=[32113], | 99.00th=[41157], 99.50th=[51119], 99.90th=[79168], 99.95th=[79168], | 99.99th=[79168] bw ( KiB/s): min= 352, max= 768, per=25.09%, avg=649.68, stdev=127.29, samples=19 iops : min= 22, max= 48, avg=40.58, stdev= 7.94, samples=19 lat (msec) : 2=1.17%, 4=35.42%, 10=9.51%, 20=26.04%, 50=27.60% lat (msec) : 100=0.26% cpu : usr=0.20%, sys=5.50%, ctx=914, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25090: Sat Feb 24 23:26:56 2024 read: IOPS=37, BW=592KiB/s (606kB/s)(5744KiB/9699msec) clat (usec): min=1843, max=20279, avg=3712.93, stdev=2805.17 lat (usec): min=1844, max=20280, avg=3713.72, stdev=2805.17 clat percentiles (usec): | 1.00th=[ 1991], 5.00th=[ 2114], 10.00th=[ 2212], 20.00th=[ 2278], | 30.00th=[ 2376], 40.00th=[ 2474], 50.00th=[ 2638], 60.00th=[ 2802], | 70.00th=[ 3097], 80.00th=[ 4621], 90.00th=[ 6390], 95.00th=[ 8848], | 99.00th=[16712], 99.50th=[18744], 99.90th=[20317], 99.95th=[20317], | 99.99th=[20317] bw ( KiB/s): min= 320, max= 990, per=24.07%, avg=592.37, stdev=193.48, samples=19 iops : min= 20, max= 61, avg=36.89, stdev=12.03, samples=19 write: IOPS=42, BW=675KiB/s (691kB/s)(6544KiB/9699msec) clat (usec): min=11567, max=57584, avg=20433.97, stdev=6625.95 lat (usec): min=11568, max=57586, avg=20435.67, stdev=6625.93 clat percentiles (usec): | 1.00th=[12387], 5.00th=[13435], 10.00th=[14091], 20.00th=[15139], | 30.00th=[16057], 40.00th=[17957], 50.00th=[19006], 60.00th=[21103], | 70.00th=[22152], 80.00th=[23725], 90.00th=[28443], 95.00th=[34866], | 99.00th=[44827], 99.50th=[47449], 99.90th=[57410], 99.95th=[57410], | 99.99th=[57410] bw ( KiB/s): min= 352, max= 832, per=25.87%, avg=669.79, stdev=149.32, samples=19 iops : min= 22, max= 52, avg=41.74, stdev= 9.25, samples=19 lat (msec) : 2=0.52%, 4=36.07%, 10=8.07%, 20=31.12%, 50=23.96% lat (msec) : 100=0.26% cpu : usr=0.10%, sys=5.69%, ctx=887, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25091: Sat Feb 24 23:26:56 2024 read: IOPS=41, BW=671KiB/s (687kB/s)(6256KiB/9319msec) clat (usec): min=1791, max=49373, avg=3829.86, stdev=3688.11 lat (usec): min=1791, max=49374, avg=3830.71, stdev=3688.10 clat percentiles (usec): | 1.00th=[ 1926], 5.00th=[ 2114], 10.00th=[ 2180], 20.00th=[ 2311], | 30.00th=[ 2409], 40.00th=[ 2507], 50.00th=[ 2638], 60.00th=[ 2835], | 70.00th=[ 3163], 80.00th=[ 4293], 90.00th=[ 6456], 95.00th=[10683], | 99.00th=[21890], 99.50th=[22152], 99.90th=[49546], 99.95th=[49546], | 99.99th=[49546] bw ( KiB/s): min= 256, max= 1184, per=27.59%, avg=679.00, stdev=289.28, samples=18 iops : min= 16, max= 74, avg=42.39, stdev=18.09, samples=18 write: IOPS=40, BW=647KiB/s (663kB/s)(6032KiB/9319msec) clat (usec): min=12009, max=52724, avg=20682.94, stdev=6011.70 lat (usec): min=12011, max=52726, avg=20684.43, stdev=6011.71 clat percentiles (usec): | 1.00th=[12256], 5.00th=[13304], 10.00th=[14091], 20.00th=[15401], | 30.00th=[16581], 40.00th=[18220], 50.00th=[20055], 60.00th=[21627], | 70.00th=[22938], 80.00th=[24511], 90.00th=[27395], 95.00th=[32637], | 99.00th=[41681], 99.50th=[44827], 99.90th=[52691], 99.95th=[52691], | 99.99th=[52691] bw ( KiB/s): min= 416, max= 864, per=24.85%, avg=643.44, stdev=128.33, samples=18 iops : min= 26, max= 54, avg=40.17, stdev= 7.99, samples=18 lat (msec) : 2=1.04%, 4=38.67%, 10=8.33%, 20=26.43%, 50=25.39% lat (msec) : 100=0.13% cpu : usr=0.12%, sys=5.92%, ctx=900, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25092: Sat Feb 24 23:26:56 2024 read: IOPS=39, BW=629KiB/s (644kB/s)(6032KiB/9590msec) clat (usec): min=1862, max=18730, avg=3745.85, stdev=2671.48 lat (usec): min=1862, max=18731, avg=3746.66, stdev=2671.48 clat percentiles (usec): | 1.00th=[ 1975], 5.00th=[ 2114], 10.00th=[ 2180], 20.00th=[ 2311], | 30.00th=[ 2442], 40.00th=[ 2540], 50.00th=[ 2671], 60.00th=[ 2868], | 70.00th=[ 3195], 80.00th=[ 4686], 90.00th=[ 6652], 95.00th=[ 8979], | 99.00th=[16057], 99.50th=[17171], 99.90th=[18744], 99.95th=[18744], | 99.99th=[18744] bw ( KiB/s): min= 288, max= 928, per=25.38%, avg=624.63, stdev=169.05, samples=19 iops : min= 18, max= 58, avg=38.95, stdev=10.54, samples=19 write: IOPS=40, BW=652KiB/s (668kB/s)(6256KiB/9590msec) clat (usec): min=12173, max=51971, avg=20854.00, stdev=6196.24 lat (usec): min=12174, max=51972, avg=20855.50, stdev=6196.28 clat percentiles (usec): | 1.00th=[12649], 5.00th=[13304], 10.00th=[14222], 20.00th=[15401], | 30.00th=[16909], 40.00th=[18744], 50.00th=[20317], 60.00th=[21627], | 70.00th=[22938], 80.00th=[24249], 90.00th=[27132], 95.00th=[32637], | 99.00th=[44827], 99.50th=[46400], 99.90th=[52167], 99.95th=[52167], | 99.99th=[52167] bw ( KiB/s): min= 384, max= 800, per=25.17%, avg=651.58, stdev=126.13, samples=19 iops : min= 24, max= 50, avg=40.63, stdev= 7.81, samples=19 lat (msec) : 2=0.65%, 4=36.46%, 10=9.77%, 20=26.43%, 50=26.56% lat (msec) : 100=0.13% cpu : usr=0.09%, sys=5.65%, ctx=903, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2462KiB/s (2521kB/s), 592KiB/s-671KiB/s (606kB/s-687kB/s), io=23.4MiB (24.5MB), run=9319-9730msec WRITE: bw=2590KiB/s (2652kB/s), 647KiB/s-675KiB/s (663kB/s-691kB/s), io=24.6MiB (25.8MB), run=9319-9730msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=25109: Sat Feb 24 23:26:57 2024 read: IOPS=10, BW=10.9MiB/s (11.4MB/s)(7168KiB/643msec) clat (usec): min=35019, max=61310, avg=49410.24, stdev=9003.71 lat (usec): min=35020, max=61311, avg=49411.47, stdev=9003.79 clat percentiles (usec): | 1.00th=[34866], 5.00th=[34866], 10.00th=[34866], 20.00th=[41681], | 30.00th=[47973], 40.00th=[47973], 50.00th=[49546], 60.00th=[53740], | 70.00th=[53740], 80.00th=[56886], 90.00th=[61080], 95.00th=[61080], | 99.00th=[61080], 99.50th=[61080], 99.90th=[61080], 99.95th=[61080], | 99.99th=[61080] bw ( KiB/s): min=12263, max=12263, per=39.94%, avg=12263.00, stdev= 0.00, samples=1 iops : min= 11, max= 11, avg=11.00, stdev= 0.00, samples=1 write: IOPS=7, BW=7963KiB/s (8154kB/s)(5120KiB/643msec) clat (msec): min=28, max=108, avg=59.02, stdev=32.45 lat (msec): min=28, max=108, avg=59.30, stdev=32.49 clat percentiles (msec): | 1.00th=[ 29], 5.00th=[ 29], 10.00th=[ 29], 20.00th=[ 29], | 30.00th=[ 31], 40.00th=[ 31], 50.00th=[ 63], 60.00th=[ 63], | 70.00th=[ 65], 80.00th=[ 65], 90.00th=[ 109], 95.00th=[ 109], | 99.00th=[ 109], 99.50th=[ 109], 99.90th=[ 109], 99.95th=[ 109], | 99.99th=[ 109] bw ( KiB/s): min= 8175, max= 8175, per=19.02%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 7, max= 7, avg= 7.00, stdev= 0.00, samples=1 lat (msec) : 50=50.00%, 100=41.67%, 250=8.33% cpu : usr=0.16%, sys=43.39%, ctx=133, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25110: Sat Feb 24 23:26:57 2024 read: IOPS=2, BW=3070KiB/s (3144kB/s)(2048KiB/667msec) clat (usec): min=38130, max=60422, avg=49276.47, stdev=15763.18 lat (usec): min=38131, max=60423, avg=49277.74, stdev=15762.78 clat percentiles (usec): | 1.00th=[38011], 5.00th=[38011], 10.00th=[38011], 20.00th=[38011], | 30.00th=[38011], 40.00th=[38011], 50.00th=[38011], 60.00th=[60556], | 70.00th=[60556], 80.00th=[60556], 90.00th=[60556], 95.00th=[60556], | 99.00th=[60556], 99.50th=[60556], 99.90th=[60556], 99.95th=[60556], | 99.99th=[60556] bw ( KiB/s): min= 4079, max= 4079, per=13.28%, avg=4079.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 write: IOPS=14, BW=14.0MiB/s (15.7MB/s)(10.0MiB/667msec) clat (msec): min=18, max=107, avg=55.29, stdev=31.81 lat (msec): min=18, max=107, avg=55.37, stdev=31.82 clat percentiles (msec): | 1.00th=[ 19], 5.00th=[ 19], 10.00th=[ 19], 20.00th=[ 24], | 30.00th=[ 28], 40.00th=[ 37], 50.00th=[ 41], 60.00th=[ 45], | 70.00th=[ 86], 80.00th=[ 86], 90.00th=[ 87], 95.00th=[ 108], | 99.00th=[ 108], 99.50th=[ 108], 99.90th=[ 108], 99.95th=[ 108], | 99.99th=[ 108] bw ( KiB/s): min=14278, max=14278, per=33.22%, avg=14278.00, stdev= 0.00, samples=1 iops : min= 13, max= 13, avg=13.00, stdev= 0.00, samples=1 lat (msec) : 20=8.33%, 50=50.00%, 100=33.33%, 250=8.33% cpu : usr=0.00%, sys=33.78%, ctx=120, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25111: Sat Feb 24 23:26:57 2024 read: IOPS=11, BW=11.0MiB/s (11.6MB/s)(6144KiB/544msec) clat (usec): min=356, max=59620, avg=28481.16, stdev=23913.46 lat (usec): min=357, max=59621, avg=28482.50, stdev=23913.39 clat percentiles (usec): | 1.00th=[ 355], 5.00th=[ 355], 10.00th=[ 355], 20.00th=[ 562], | 30.00th=[ 562], 40.00th=[31589], 50.00th=[31589], 60.00th=[33424], | 70.00th=[45351], 80.00th=[45351], 90.00th=[59507], 95.00th=[59507], | 99.00th=[59507], 99.50th=[59507], 99.90th=[59507], 99.95th=[59507], | 99.99th=[59507] bw ( KiB/s): min=10178, max=10178, per=33.15%, avg=10178.00, stdev= 0.00, samples=1 iops : min= 9, max= 9, avg= 9.00, stdev= 0.00, samples=1 write: IOPS=11, BW=11.0MiB/s (11.6MB/s)(6144KiB/544msec) clat (msec): min=30, max=104, avg=60.51, stdev=27.97 lat (msec): min=30, max=104, avg=60.57, stdev=27.98 clat percentiles (msec): | 1.00th=[ 31], 5.00th=[ 31], 10.00th=[ 31], 20.00th=[ 40], | 30.00th=[ 40], 40.00th=[ 47], 50.00th=[ 47], 60.00th=[ 62], | 70.00th=[ 82], 80.00th=[ 82], 90.00th=[ 105], 95.00th=[ 105], | 99.00th=[ 105], 99.50th=[ 105], 99.90th=[ 105], 99.95th=[ 105], | 99.99th=[ 105] bw ( KiB/s): min=10178, max=10178, per=23.68%, avg=10178.00, stdev= 0.00, samples=1 iops : min= 9, max= 9, avg= 9.00, stdev= 0.00, samples=1 lat (usec) : 500=8.33%, 750=8.33% lat (msec) : 50=50.00%, 100=25.00%, 250=8.33% cpu : usr=0.00%, sys=29.47%, ctx=90, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25112: Sat Feb 24 23:26:57 2024 read: IOPS=8, BW=8366KiB/s (8567kB/s)(5120KiB/612msec) clat (usec): min=498, max=82155, avg=38917.20, stdev=30189.59 lat (usec): min=500, max=82156, avg=38918.49, stdev=30189.51 clat percentiles (usec): | 1.00th=[ 498], 5.00th=[ 498], 10.00th=[ 498], 20.00th=[ 498], | 30.00th=[29230], 40.00th=[29230], 50.00th=[31327], 60.00th=[31327], | 70.00th=[51119], 80.00th=[51119], 90.00th=[82314], 95.00th=[82314], | 99.00th=[82314], 99.50th=[82314], 99.90th=[82314], 99.95th=[82314], | 99.99th=[82314] bw ( KiB/s): min= 8126, max= 8126, per=26.47%, avg=8126.00, stdev= 0.00, samples=1 iops : min= 7, max= 7, avg= 7.00, stdev= 0.00, samples=1 write: IOPS=11, BW=11.4MiB/s (11.0MB/s)(7168KiB/612msec) clat (msec): min=14, max=140, avg=59.32, stdev=47.48 lat (msec): min=14, max=140, avg=59.39, stdev=47.50 clat percentiles (msec): | 1.00th=[ 15], 5.00th=[ 15], 10.00th=[ 15], 20.00th=[ 24], | 30.00th=[ 29], 40.00th=[ 29], 50.00th=[ 40], 60.00th=[ 62], | 70.00th=[ 62], 80.00th=[ 108], 90.00th=[ 140], 95.00th=[ 140], | 99.00th=[ 140], 99.50th=[ 140], 99.90th=[ 140], 99.95th=[ 140], | 99.99th=[ 140] bw ( KiB/s): min=10158, max=10158, per=23.63%, avg=10158.00, stdev= 0.00, samples=1 iops : min= 9, max= 9, avg= 9.00, stdev= 0.00, samples=1 lat (usec) : 500=8.33% lat (msec) : 20=8.33%, 50=41.67%, 100=25.00%, 250=16.67% cpu : usr=0.16%, sys=30.93%, ctx=119, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=29.0MiB/s (31.4MB/s), 3070KiB/s-11.0MiB/s (3144kB/s-11.6MB/s), io=20.0MiB (20.0MB), run=544-667msec WRITE: bw=41.0MiB/s (44.0MB/s), 7963KiB/s-14.0MiB/s (8154kB/s-15.7MB/s), io=28.0MiB (29.4MB), run=544-667msec rand-rw: (groupid=0, jobs=1): err= 0: pid=25105: Sat Feb 24 23:26:57 2024 read: IOPS=9, BW=9901KiB/s (10.1MB/s)(7168KiB/724msec) clat (msec): min=19, max=116, avg=51.86, stdev=31.09 lat (msec): min=19, max=116, avg=51.86, stdev=31.09 clat percentiles (msec): | 1.00th=[ 20], 5.00th=[ 20], 10.00th=[ 20], 20.00th=[ 36], | 30.00th=[ 39], 40.00th=[ 39], 50.00th=[ 42], 60.00th=[ 55], | 70.00th=[ 55], 80.00th=[ 58], 90.00th=[ 117], 95.00th=[ 117], | 99.00th=[ 117], 99.50th=[ 117], 99.90th=[ 117], 99.95th=[ 117], | 99.99th=[ 117] bw ( KiB/s): min=10240, max=10240, per=41.05%, avg=10240.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 write: IOPS=6, BW=7072KiB/s (7242kB/s)(5120KiB/724msec) clat (usec): min=36682, max=98553, avg=70054.60, stdev=25649.66 lat (usec): min=36759, max=98650, avg=70127.74, stdev=25647.45 clat percentiles (usec): | 1.00th=[36439], 5.00th=[36439], 10.00th=[36439], 20.00th=[36439], | 30.00th=[50594], 40.00th=[50594], 50.00th=[79168], 60.00th=[79168], | 70.00th=[85459], 80.00th=[85459], 90.00th=[98042], 95.00th=[98042], | 99.00th=[98042], 99.50th=[98042], 99.90th=[98042], 99.95th=[98042], | 99.99th=[98042] bw ( KiB/s): min= 6144, max= 6144, per=17.59%, avg=6144.00, stdev= 0.00, samples=1 iops : min= 6, max= 6, avg= 6.00, stdev= 0.00, samples=1 lat (msec) : 20=8.33%, 50=33.33%, 100=50.00%, 250=8.33% cpu : usr=0.00%, sys=15.77%, ctx=228, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25106: Sat Feb 24 23:26:57 2024 read: IOPS=2, BW=2495KiB/s (2554kB/s)(2048KiB/821msec) clat (usec): min=42184, max=84793, avg=63488.76, stdev=30129.35 lat (usec): min=42185, max=84794, avg=63490.21, stdev=30129.45 clat percentiles (usec): | 1.00th=[42206], 5.00th=[42206], 10.00th=[42206], 20.00th=[42206], | 30.00th=[42206], 40.00th=[42206], 50.00th=[42206], 60.00th=[84411], | 70.00th=[84411], 80.00th=[84411], 90.00th=[84411], 95.00th=[84411], | 99.00th=[84411], 99.50th=[84411], 99.90th=[84411], 99.95th=[84411], | 99.99th=[84411] bw ( KiB/s): min= 2048, max= 2048, per=8.21%, avg=2048.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=12, BW=12.2MiB/s (12.8MB/s)(10.0MiB/821msec) clat (msec): min=30, max=151, avg=67.81, stdev=47.21 lat (msec): min=30, max=151, avg=67.90, stdev=47.21 clat percentiles (msec): | 1.00th=[ 31], 5.00th=[ 31], 10.00th=[ 31], 20.00th=[ 33], | 30.00th=[ 33], 40.00th=[ 34], 50.00th=[ 44], 60.00th=[ 50], | 70.00th=[ 102], 80.00th=[ 102], 90.00th=[ 146], 95.00th=[ 153], | 99.00th=[ 153], 99.50th=[ 153], 99.90th=[ 153], 99.95th=[ 153], | 99.99th=[ 153] bw ( KiB/s): min=10240, max=10240, per=29.32%, avg=10240.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 lat (msec) : 50=58.33%, 100=16.67%, 250=25.00% cpu : usr=0.49%, sys=14.15%, ctx=199, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25107: Sat Feb 24 23:26:57 2024 read: IOPS=8, BW=8225KiB/s (8422kB/s)(6144KiB/747msec) clat (usec): min=22452, max=69839, avg=37345.00, stdev=17671.19 lat (usec): min=22453, max=69840, avg=37346.09, stdev=17671.11 clat percentiles (usec): | 1.00th=[22414], 5.00th=[22414], 10.00th=[22414], 20.00th=[24249], | 30.00th=[24249], 40.00th=[27395], 50.00th=[27395], 60.00th=[39060], | 70.00th=[41157], 80.00th=[41157], 90.00th=[69731], 95.00th=[69731], | 99.00th=[69731], 99.50th=[69731], 99.90th=[69731], 99.95th=[69731], | 99.99th=[69731] bw ( KiB/s): min= 4096, max= 4096, per=16.42%, avg=4096.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 write: IOPS=8, BW=8225KiB/s (8422kB/s)(6144KiB/747msec) clat (msec): min=35, max=201, avg=83.56, stdev=62.14 lat (msec): min=35, max=201, avg=83.65, stdev=62.14 clat percentiles (msec): | 1.00th=[ 36], 5.00th=[ 36], 10.00th=[ 36], 20.00th=[ 43], | 30.00th=[ 43], 40.00th=[ 52], 50.00th=[ 52], 60.00th=[ 71], | 70.00th=[ 102], 80.00th=[ 102], 90.00th=[ 201], 95.00th=[ 201], | 99.00th=[ 201], 99.50th=[ 201], 99.90th=[ 201], 99.95th=[ 201], | 99.99th=[ 201] bw ( KiB/s): min= 6144, max= 6144, per=17.59%, avg=6144.00, stdev= 0.00, samples=1 iops : min= 6, max= 6, avg= 6.00, stdev= 0.00, samples=1 lat (msec) : 50=58.33%, 100=25.00%, 250=16.67% cpu : usr=0.13%, sys=16.22%, ctx=267, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25108: Sat Feb 24 23:26:57 2024 read: IOPS=6, BW=6456KiB/s (6611kB/s)(5120KiB/793msec) clat (usec): min=19565, max=73932, avg=42950.72, stdev=22130.98 lat (usec): min=19567, max=73934, avg=42952.20, stdev=22131.10 clat percentiles (usec): | 1.00th=[19530], 5.00th=[19530], 10.00th=[19530], 20.00th=[19530], | 30.00th=[28705], 40.00th=[28705], 50.00th=[35390], 60.00th=[35390], | 70.00th=[56886], 80.00th=[56886], 90.00th=[73925], 95.00th=[73925], | 99.00th=[73925], 99.50th=[73925], 99.90th=[73925], 99.95th=[73925], | 99.99th=[73925] bw ( KiB/s): min= 2048, max= 2048, per=8.21%, avg=2048.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=8, BW=9039KiB/s (9256kB/s)(7168KiB/793msec) clat (msec): min=33, max=157, avg=82.31, stdev=47.28 lat (msec): min=33, max=157, avg=82.38, stdev=47.29 clat percentiles (msec): | 1.00th=[ 34], 5.00th=[ 34], 10.00th=[ 34], 20.00th=[ 46], | 30.00th=[ 47], 40.00th=[ 47], 50.00th=[ 64], 60.00th=[ 101], | 70.00th=[ 101], 80.00th=[ 129], 90.00th=[ 159], 95.00th=[ 159], | 99.00th=[ 159], 99.50th=[ 159], 99.90th=[ 159], 99.95th=[ 159], | 99.99th=[ 159] bw ( KiB/s): min= 8192, max= 8192, per=23.46%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 lat (msec) : 20=8.33%, 50=41.67%, 100=25.00%, 250=25.00% cpu : usr=0.13%, sys=13.76%, ctx=38, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=24.4MiB/s (25.5MB/s), 2495KiB/s-9901KiB/s (2554kB/s-10.1MB/s), io=20.0MiB (20.0MB), run=724-821msec WRITE: bw=34.1MiB/s (35.8MB/s), 7072KiB/s-12.2MiB/s (7242kB/s-12.8MB/s), io=28.0MiB (29.4MB), run=724-821msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=25119: Sat Feb 24 23:26:59 2024 read: IOPS=7, BW=31.0MiB/s (32.5MB/s)(12.0MiB/387msec) clat (msec): min=82, max=163, avg=121.58, stdev=40.43 lat (msec): min=82, max=163, avg=121.58, stdev=40.43 clat percentiles (msec): | 1.00th=[ 83], 5.00th=[ 83], 10.00th=[ 83], 20.00th=[ 83], | 30.00th=[ 83], 40.00th=[ 118], 50.00th=[ 118], 60.00th=[ 118], | 70.00th=[ 163], 80.00th=[ 163], 90.00th=[ 163], 95.00th=[ 163], | 99.00th=[ 163], 99.50th=[ 163], 99.90th=[ 163], 99.95th=[ 163], | 99.99th=[ 163] lat (msec) : 100=33.33%, 250=66.67% cpu : usr=0.00%, sys=50.26%, ctx=155, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25121: Sat Feb 24 23:26:59 2024 read: IOPS=1, BW=6215KiB/s (6365kB/s)(4096KiB/659msec) clat (nsec): min=247842k, max=247842k, avg=247841715.00, stdev= 0.00 lat (nsec): min=247844k, max=247844k, avg=247843817.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 249], 5.00th=[ 249], 10.00th=[ 249], 20.00th=[ 249], | 30.00th=[ 249], 40.00th=[ 249], 50.00th=[ 249], 60.00th=[ 249], | 70.00th=[ 249], 80.00th=[ 249], 90.00th=[ 249], 95.00th=[ 249], | 99.00th=[ 249], 99.50th=[ 249], 99.90th=[ 249], 99.95th=[ 249], | 99.99th=[ 249] bw ( KiB/s): min= 7922, max= 7922, per=31.87%, avg=7922.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=3, BW=12.1MiB/s (12.7MB/s)(8192KiB/659msec) clat (msec): min=122, max=262, avg=192.40, stdev=99.44 lat (msec): min=122, max=263, avg=192.62, stdev=99.55 clat percentiles (msec): | 1.00th=[ 123], 5.00th=[ 123], 10.00th=[ 123], 20.00th=[ 123], | 30.00th=[ 123], 40.00th=[ 123], 50.00th=[ 123], 60.00th=[ 264], | 70.00th=[ 264], 80.00th=[ 264], 90.00th=[ 264], 95.00th=[ 264], | 99.00th=[ 264], 99.50th=[ 264], 99.90th=[ 264], 99.95th=[ 264], | 99.99th=[ 264] bw ( KiB/s): min= 7922, max= 7922, per=15.93%, avg=7922.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.00%, sys=39.36%, ctx=106, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25122: Sat Feb 24 23:26:59 2024 write: IOPS=6, BW=24.2MiB/s (25.4MB/s)(12.0MiB/495msec) clat (msec): min=73, max=310, avg=162.02, stdev=129.33 lat (msec): min=73, max=310, avg=162.29, stdev=129.31 clat percentiles (msec): | 1.00th=[ 74], 5.00th=[ 74], 10.00th=[ 74], 20.00th=[ 74], | 30.00th=[ 74], 40.00th=[ 103], 50.00th=[ 103], 60.00th=[ 103], | 70.00th=[ 313], 80.00th=[ 313], 90.00th=[ 313], 95.00th=[ 313], | 99.00th=[ 313], 99.50th=[ 313], 99.90th=[ 313], 99.95th=[ 313], | 99.99th=[ 313] lat (msec) : 100=33.33%, 250=33.33%, 500=33.33% cpu : usr=0.00%, sys=36.44%, ctx=75, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25124: Sat Feb 24 23:26:59 2024 write: IOPS=5, BW=20.3MiB/s (21.3MB/s)(12.0MiB/592msec) clat (msec): min=111, max=300, avg=195.98, stdev=95.88 lat (msec): min=111, max=300, avg=196.93, stdev=95.70 clat percentiles (msec): | 1.00th=[ 112], 5.00th=[ 112], 10.00th=[ 112], 20.00th=[ 112], | 30.00th=[ 112], 40.00th=[ 176], 50.00th=[ 176], 60.00th=[ 176], | 70.00th=[ 300], 80.00th=[ 300], 90.00th=[ 300], 95.00th=[ 300], | 99.00th=[ 300], 99.50th=[ 300], 99.90th=[ 300], 99.95th=[ 300], | 99.99th=[ 300] bw ( KiB/s): min=16384, max=16384, per=32.95%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.00%, sys=33.67%, ctx=65, majf=0, minf=28 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=24.3MiB/s (25.5MB/s), 6215KiB/s-31.0MiB/s (6365kB/s-32.5MB/s), io=16.0MiB (16.8MB), run=387-659msec WRITE: bw=48.6MiB/s (50.9MB/s), 12.1MiB/s-24.2MiB/s (12.7MB/s-25.4MB/s), io=32.0MiB (33.6MB), run=495-659msec rand-rw: (groupid=0, jobs=1): err= 0: pid=25120: Sat Feb 24 23:26:59 2024 read: IOPS=4, BW=16.1MiB/s (16.9MB/s)(12.0MiB/744msec) clat (msec): min=51, max=397, avg=232.22, stdev=173.69 lat (msec): min=51, max=397, avg=232.22, stdev=173.69 clat percentiles (msec): | 1.00th=[ 52], 5.00th=[ 52], 10.00th=[ 52], 20.00th=[ 52], | 30.00th=[ 52], 40.00th=[ 247], 50.00th=[ 247], 60.00th=[ 247], | 70.00th=[ 397], 80.00th=[ 397], 90.00th=[ 397], 95.00th=[ 397], | 99.00th=[ 397], 99.50th=[ 397], 99.90th=[ 397], 99.95th=[ 397], | 99.99th=[ 397] bw ( KiB/s): min= 8159, max= 8159, per=38.15%, avg=8159.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 250=33.33%, 500=33.33% cpu : usr=0.00%, sys=13.19%, ctx=60, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25123: Sat Feb 24 23:26:59 2024 read: IOPS=1, BW=5347KiB/s (5476kB/s)(4096KiB/766msec) clat (nsec): min=394605k, max=394605k, avg=394605337.00, stdev= 0.00 lat (nsec): min=394608k, max=394608k, avg=394607769.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 397], 5.00th=[ 397], 10.00th=[ 397], 20.00th=[ 397], | 30.00th=[ 397], 40.00th=[ 397], 50.00th=[ 397], 60.00th=[ 397], | 70.00th=[ 397], 80.00th=[ 397], 90.00th=[ 397], 95.00th=[ 397], | 99.00th=[ 397], 99.50th=[ 397], 99.90th=[ 397], 99.95th=[ 397], | 99.99th=[ 397] bw ( KiB/s): min= 8126, max= 8126, per=37.99%, avg=8126.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=2, BW=10.4MiB/s (10.0MB/s)(8192KiB/766msec) clat (msec): min=167, max=197, avg=182.29, stdev=21.27 lat (msec): min=167, max=197, avg=182.51, stdev=21.41 clat percentiles (msec): | 1.00th=[ 167], 5.00th=[ 167], 10.00th=[ 167], 20.00th=[ 167], | 30.00th=[ 167], 40.00th=[ 167], 50.00th=[ 167], 60.00th=[ 199], | 70.00th=[ 199], 80.00th=[ 199], 90.00th=[ 199], 95.00th=[ 199], | 99.00th=[ 199], 99.50th=[ 199], 99.90th=[ 199], 99.95th=[ 199], | 99.99th=[ 199] lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.13%, sys=14.64%, ctx=48, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25125: Sat Feb 24 23:26:59 2024 write: IOPS=4, BW=17.0MiB/s (18.9MB/s)(12.0MiB/667msec) clat (msec): min=82, max=356, avg=221.87, stdev=136.95 lat (msec): min=83, max=356, avg=222.09, stdev=136.84 clat percentiles (msec): | 1.00th=[ 84], 5.00th=[ 84], 10.00th=[ 84], 20.00th=[ 84], | 30.00th=[ 84], 40.00th=[ 226], 50.00th=[ 226], 60.00th=[ 226], | 70.00th=[ 359], 80.00th=[ 359], 90.00th=[ 359], 95.00th=[ 359], | 99.00th=[ 359], 99.50th=[ 359], 99.90th=[ 359], 99.95th=[ 359], | 99.99th=[ 359] bw ( KiB/s): min= 8192, max= 8192, per=19.15%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 250=33.33%, 500=33.33% cpu : usr=0.00%, sys=16.52%, ctx=40, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25126: Sat Feb 24 23:26:59 2024 write: IOPS=4, BW=17.7MiB/s (18.6MB/s)(12.0MiB/677msec) clat (msec): min=147, max=265, avg=225.01, stdev=67.20 lat (msec): min=147, max=265, avg=225.24, stdev=67.17 clat percentiles (msec): | 1.00th=[ 148], 5.00th=[ 148], 10.00th=[ 148], 20.00th=[ 148], | 30.00th=[ 148], 40.00th=[ 262], 50.00th=[ 262], 60.00th=[ 262], | 70.00th=[ 266], 80.00th=[ 266], 90.00th=[ 266], 95.00th=[ 266], | 99.00th=[ 266], 99.50th=[ 266], 99.90th=[ 266], 99.95th=[ 266], | 99.99th=[ 266] bw ( KiB/s): min= 8192, max= 8192, per=19.15%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=66.67% cpu : usr=0.00%, sys=15.53%, ctx=69, majf=0, minf=28 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=20.9MiB/s (21.9MB/s), 5347KiB/s-16.1MiB/s (5476kB/s-16.9MB/s), io=16.0MiB (16.8MB), run=744-766msec WRITE: bw=41.8MiB/s (43.8MB/s), 10.4MiB/s-17.0MiB/s (10.0MB/s-18.9MB/s), io=32.0MiB (33.6MB), run=667-766msec