== sanity test 398b: DIO and buffer IO race ============== 04:48:53 (1733651333) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.30865 s, 38.5 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=22188: Sun Dec 8 04:49:10 2024 read: IOPS=103, BW=414KiB/s (423kB/s)(5992KiB/14490msec) clat (usec): min=49, max=189500, avg=5370.77, stdev=5558.83 lat (usec): min=50, max=189501, avg=5371.84, stdev=5558.75 clat percentiles (usec): | 1.00th=[ 73], 5.00th=[ 367], 10.00th=[ 2606], 20.00th=[ 3130], | 30.00th=[ 3589], 40.00th=[ 4113], 50.00th=[ 4752], 60.00th=[ 5473], | 70.00th=[ 6259], 80.00th=[ 7177], 90.00th=[ 8586], 95.00th=[ 10159], | 99.00th=[ 15270], 99.50th=[ 17433], 99.90th=[ 24511], 99.95th=[189793], | 99.99th=[189793] bw ( KiB/s): min= 232, max= 560, per=24.56%, avg=406.93, stdev=69.54, samples=28 iops : min= 58, max= 140, avg=101.61, stdev=17.31, samples=28 write: IOPS=108, BW=435KiB/s (445kB/s)(6296KiB/14490msec) clat (usec): min=502, max=20641, avg=4052.20, stdev=3349.86 lat (usec): min=503, max=20642, avg=4053.28, stdev=3349.77 clat percentiles (usec): | 1.00th=[ 553], 5.00th=[ 644], 10.00th=[ 734], 20.00th=[ 865], | 30.00th=[ 1172], 40.00th=[ 2671], 50.00th=[ 3392], 60.00th=[ 4080], | 70.00th=[ 5145], 80.00th=[ 6849], 90.00th=[ 8848], 95.00th=[10421], | 99.00th=[14091], 99.50th=[16712], 99.90th=[19530], 99.95th=[20579], | 99.99th=[20579] bw ( KiB/s): min= 288, max= 536, per=24.54%, avg=425.54, stdev=59.50, samples=28 iops : min= 72, max= 134, avg=106.25, stdev=14.95, samples=28 lat (usec) : 50=0.03%, 100=1.46%, 250=0.91%, 500=0.07%, 750=5.60% lat (usec) : 1000=7.98% lat (msec) : 2=4.79%, 4=27.93%, 10=45.67%, 20=5.47%, 50=0.07% lat (msec) : 250=0.03% cpu : usr=0.21%, sys=23.59%, ctx=6259, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22189: Sun Dec 8 04:49:10 2024 read: IOPS=102, BW=412KiB/s (421kB/s)(5940KiB/14431msec) clat (usec): min=49, max=180449, avg=5335.64, stdev=5287.81 lat (usec): min=49, max=180451, avg=5336.80, stdev=5287.43 clat percentiles (usec): | 1.00th=[ 71], 5.00th=[ 2147], 10.00th=[ 2638], 20.00th=[ 3163], | 30.00th=[ 3621], 40.00th=[ 4178], 50.00th=[ 4817], 60.00th=[ 5604], | 70.00th=[ 6259], 80.00th=[ 7242], 90.00th=[ 8455], 95.00th=[ 9765], | 99.00th=[ 14353], 99.50th=[ 15664], 99.90th=[ 26346], 99.95th=[181404], | 99.99th=[181404] bw ( KiB/s): min= 200, max= 504, per=24.49%, avg=405.86, stdev=62.67, samples=28 iops : min= 50, max= 126, avg=101.36, stdev=15.67, samples=28 write: IOPS=109, BW=440KiB/s (450kB/s)(6348KiB/14431msec) clat (usec): min=509, max=20277, avg=4051.97, stdev=3437.47 lat (usec): min=509, max=20278, avg=4054.11, stdev=3437.51 clat percentiles (usec): | 1.00th=[ 562], 5.00th=[ 644], 10.00th=[ 709], 20.00th=[ 832], | 30.00th=[ 1090], 40.00th=[ 2442], 50.00th=[ 3392], 60.00th=[ 4178], | 70.00th=[ 5211], 80.00th=[ 6849], 90.00th=[ 8848], 95.00th=[10814], | 99.00th=[14484], 99.50th=[15926], 99.90th=[18744], 99.95th=[20317], | 99.99th=[20317] bw ( KiB/s): min= 256, max= 550, per=24.66%, avg=427.54, stdev=61.86, samples=28 iops : min= 64, max= 137, avg=106.79, stdev=15.42, samples=28 lat (usec) : 50=0.03%, 100=1.27%, 250=0.88%, 500=0.07%, 750=6.54% lat (usec) : 1000=7.91% lat (msec) : 2=5.31%, 4=25.94%, 10=46.26%, 20=5.70%, 50=0.07% lat (msec) : 250=0.03% cpu : usr=0.23%, sys=23.41%, ctx=6166, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22190: Sun Dec 8 04:49:10 2024 read: IOPS=104, BW=418KiB/s (428kB/s)(6028KiB/14406msec) clat (usec): min=56, max=31806, avg=5312.71, stdev=3226.67 lat (usec): min=57, max=31808, avg=5314.92, stdev=3227.94 clat percentiles (usec): | 1.00th=[ 70], 5.00th=[ 133], 10.00th=[ 2573], 20.00th=[ 3064], | 30.00th=[ 3556], 40.00th=[ 4080], 50.00th=[ 4686], 60.00th=[ 5473], | 70.00th=[ 6259], 80.00th=[ 7111], 90.00th=[ 8979], 95.00th=[10421], | 99.00th=[17957], 99.50th=[19530], 99.90th=[28967], 99.95th=[31851], | 99.99th=[31851] bw ( KiB/s): min= 264, max= 520, per=24.96%, avg=413.57, stdev=51.13, samples=28 iops : min= 66, max= 130, avg=103.29, stdev=12.77, samples=28 write: IOPS=108, BW=435KiB/s (445kB/s)(6260KiB/14406msec) clat (usec): min=495, max=178509, avg=4033.79, stdev=5562.04 lat (usec): min=496, max=178510, avg=4035.01, stdev=5561.84 clat percentiles (usec): | 1.00th=[ 545], 5.00th=[ 644], 10.00th=[ 717], 20.00th=[ 840], | 30.00th=[ 1074], 40.00th=[ 2278], 50.00th=[ 3261], 60.00th=[ 4015], | 70.00th=[ 5145], 80.00th=[ 6521], 90.00th=[ 8717], 95.00th=[ 10290], | 99.00th=[ 14091], 99.50th=[ 16909], 99.90th=[ 31065], 99.95th=[179307], | 99.99th=[179307] bw ( KiB/s): min= 280, max= 560, per=24.63%, avg=427.00, stdev=68.39, samples=28 iops : min= 70, max= 140, avg=106.64, stdev=17.08, samples=28 lat (usec) : 100=1.66%, 250=0.98%, 500=0.07%, 750=6.51%, 1000=7.78% lat (msec) : 2=5.60%, 4=26.95%, 10=44.69%, 20=5.44%, 50=0.29% lat (msec) : 250=0.03% cpu : usr=0.23%, sys=23.38%, ctx=6213, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22191: Sun Dec 8 04:49:10 2024 read: IOPS=105, BW=422KiB/s (432kB/s)(6052KiB/14354msec) clat (usec): min=49, max=21757, avg=5212.05, stdev=2781.61 lat (usec): min=49, max=21757, avg=5214.55, stdev=2780.63 clat percentiles (usec): | 1.00th=[ 71], 5.00th=[ 2024], 10.00th=[ 2573], 20.00th=[ 3130], | 30.00th=[ 3556], 40.00th=[ 4146], 50.00th=[ 4817], 60.00th=[ 5538], | 70.00th=[ 6259], 80.00th=[ 7111], 90.00th=[ 8455], 95.00th=[ 9765], | 99.00th=[14877], 99.50th=[17171], 99.90th=[20841], 99.95th=[21627], | 99.99th=[21627] bw ( KiB/s): min= 208, max= 552, per=25.19%, avg=417.43, stdev=65.93, samples=28 iops : min= 52, max= 138, avg=104.18, stdev=16.47, samples=28 write: IOPS=108, BW=434KiB/s (445kB/s)(6236KiB/14354msec) clat (usec): min=500, max=174962, avg=4088.00, stdev=5490.87 lat (usec): min=500, max=174964, avg=4089.77, stdev=5490.78 clat percentiles (usec): | 1.00th=[ 545], 5.00th=[ 627], 10.00th=[ 693], 20.00th=[ 840], | 30.00th=[ 1037], 40.00th=[ 2474], 50.00th=[ 3359], 60.00th=[ 4228], | 70.00th=[ 5145], 80.00th=[ 6718], 90.00th=[ 8717], 95.00th=[ 10290], | 99.00th=[ 13960], 99.50th=[ 16712], 99.90th=[ 26870], 99.95th=[175113], | 99.99th=[175113] bw ( KiB/s): min= 240, max= 592, per=24.85%, avg=430.89, stdev=66.25, samples=28 iops : min= 60, max= 148, avg=107.54, stdev=16.61, samples=28 lat (usec) : 50=0.03%, 100=1.20%, 250=1.04%, 500=0.03%, 750=7.19% lat (usec) : 1000=7.62% lat (msec) : 2=4.62%, 4=26.01%, 10=47.04%, 20=4.98%, 50=0.20% lat (msec) : 250=0.03% cpu : usr=0.17%, sys=23.38%, ctx=6264, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1657KiB/s (1697kB/s), 412KiB/s-422KiB/s (421kB/s-432kB/s), io=23.4MiB (24.6MB), run=14354-14490msec WRITE: bw=1735KiB/s (1777kB/s), 434KiB/s-440KiB/s (445kB/s-450kB/s), io=24.6MiB (25.7MB), run=14354-14490msec rand-rw: (groupid=0, jobs=1): err= 0: pid=22184: Sun Dec 8 04:49:14 2024 read: IOPS=81, BW=325KiB/s (333kB/s)(5992KiB/18434msec) clat (usec): min=1733, max=189801, avg=5076.12, stdev=6025.10 lat (usec): min=1734, max=189802, avg=5076.86, stdev=6025.12 clat percentiles (usec): | 1.00th=[ 1844], 5.00th=[ 2024], 10.00th=[ 2147], 20.00th=[ 2376], | 30.00th=[ 2606], 40.00th=[ 2999], 50.00th=[ 3458], 60.00th=[ 4113], | 70.00th=[ 5080], 80.00th=[ 7177], 90.00th=[ 10421], 95.00th=[ 13042], | 99.00th=[ 18220], 99.50th=[ 21365], 99.90th=[ 23200], 99.95th=[189793], | 99.99th=[189793] bw ( KiB/s): min= 152, max= 704, per=24.71%, avg=320.06, stdev=145.65, samples=36 iops : min= 38, max= 176, avg=79.89, stdev=36.45, samples=36 write: IOPS=85, BW=342KiB/s (350kB/s)(6296KiB/18434msec) clat (usec): min=2500, max=32036, avg=6837.76, stdev=3995.36 lat (usec): min=2500, max=32036, avg=6838.65, stdev=3995.37 clat percentiles (usec): | 1.00th=[ 2835], 5.00th=[ 3097], 10.00th=[ 3326], 20.00th=[ 3687], | 30.00th=[ 4047], 40.00th=[ 4555], 50.00th=[ 5342], 60.00th=[ 6325], | 70.00th=[ 7963], 80.00th=[ 9896], 90.00th=[12518], 95.00th=[14746], | 99.00th=[20055], 99.50th=[23200], 99.90th=[27919], 99.95th=[32113], | 99.99th=[32113] bw ( KiB/s): min= 160, max= 752, per=24.83%, avg=336.72, stdev=152.39, samples=36 iops : min= 40, max= 188, avg=84.06, stdev=38.16, samples=36 lat (msec) : 2=1.99%, 4=40.69%, 10=42.35%, 20=14.19%, 50=0.75% lat (msec) : 250=0.03% cpu : usr=0.15%, sys=11.23%, ctx=3699, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22185: Sun Dec 8 04:49:14 2024 read: IOPS=80, BW=320KiB/s (328kB/s)(5940KiB/18535msec) clat (usec): min=1714, max=182996, avg=4926.88, stdev=5851.35 lat (usec): min=1714, max=182998, avg=4927.71, stdev=5851.38 clat percentiles (usec): | 1.00th=[ 1844], 5.00th=[ 2024], 10.00th=[ 2147], 20.00th=[ 2343], | 30.00th=[ 2606], 40.00th=[ 2966], 50.00th=[ 3458], 60.00th=[ 3982], | 70.00th=[ 4817], 80.00th=[ 6325], 90.00th=[ 10552], 95.00th=[ 13304], | 99.00th=[ 17171], 99.50th=[ 18744], 99.90th=[ 24511], 99.95th=[183501], | 99.99th=[183501] bw ( KiB/s): min= 112, max= 672, per=24.47%, avg=316.89, stdev=153.43, samples=36 iops : min= 28, max= 168, avg=79.22, stdev=38.36, samples=36 write: IOPS=85, BW=342KiB/s (351kB/s)(6348KiB/18535msec) clat (usec): min=2479, max=53699, avg=7025.74, stdev=4418.77 lat (usec): min=2480, max=53700, avg=7026.64, stdev=4418.81 clat percentiles (usec): | 1.00th=[ 2671], 5.00th=[ 3097], 10.00th=[ 3359], 20.00th=[ 3687], | 30.00th=[ 4080], 40.00th=[ 4686], 50.00th=[ 5538], 60.00th=[ 6521], | 70.00th=[ 8094], 80.00th=[ 9896], 90.00th=[12911], 95.00th=[15139], | 99.00th=[22938], 99.50th=[26346], 99.90th=[32113], 99.95th=[53740], | 99.99th=[53740] bw ( KiB/s): min= 168, max= 680, per=24.61%, avg=333.78, stdev=142.92, samples=36 iops : min= 42, max= 170, avg=83.44, stdev=35.73, samples=36 lat (msec) : 2=1.89%, 4=41.63%, 10=40.89%, 20=14.55%, 50=0.98% lat (msec) : 100=0.03%, 250=0.03% cpu : usr=0.23%, sys=10.98%, ctx=3727, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22186: Sun Dec 8 04:49:14 2024 read: IOPS=82, BW=328KiB/s (336kB/s)(6028KiB/18352msec) clat (usec): min=1642, max=24145, avg=5005.12, stdev=3836.71 lat (usec): min=1643, max=24145, avg=5005.89, stdev=3836.73 clat percentiles (usec): | 1.00th=[ 1876], 5.00th=[ 2024], 10.00th=[ 2147], 20.00th=[ 2376], | 30.00th=[ 2638], 40.00th=[ 2966], 50.00th=[ 3458], 60.00th=[ 4113], | 70.00th=[ 4948], 80.00th=[ 6915], 90.00th=[11207], 95.00th=[13435], | 99.00th=[18482], 99.50th=[21365], 99.90th=[23987], 99.95th=[24249], | 99.99th=[24249] bw ( KiB/s): min= 160, max= 776, per=25.11%, avg=325.17, stdev=151.34, samples=36 iops : min= 40, max= 194, avg=81.17, stdev=37.85, samples=36 write: IOPS=85, BW=341KiB/s (349kB/s)(6260KiB/18352msec) clat (msec): min=2, max=185, avg= 6.86, stdev= 6.03 lat (msec): min=2, max=185, avg= 6.86, stdev= 6.03 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 4], 10.00th=[ 4], 20.00th=[ 4], | 30.00th=[ 4], 40.00th=[ 5], 50.00th=[ 6], 60.00th=[ 7], | 70.00th=[ 8], 80.00th=[ 10], 90.00th=[ 13], 95.00th=[ 15], | 99.00th=[ 22], 99.50th=[ 24], 99.90th=[ 29], 99.95th=[ 186], | 99.99th=[ 186] bw ( KiB/s): min= 184, max= 728, per=24.80%, avg=336.28, stdev=152.13, samples=36 iops : min= 46, max= 182, avg=83.94, stdev=38.10, samples=36 lat (msec) : 2=2.02%, 4=42.51%, 10=39.68%, 20=14.81%, 50=0.94% lat (msec) : 250=0.03% cpu : usr=0.15%, sys=11.27%, ctx=3744, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22187: Sun Dec 8 04:49:14 2024 read: IOPS=82, BW=330KiB/s (338kB/s)(6052KiB/18357msec) clat (usec): min=1607, max=22937, avg=4896.75, stdev=3729.90 lat (usec): min=1608, max=22938, avg=4897.51, stdev=3729.91 clat percentiles (usec): | 1.00th=[ 1844], 5.00th=[ 2008], 10.00th=[ 2147], 20.00th=[ 2343], | 30.00th=[ 2573], 40.00th=[ 2966], 50.00th=[ 3425], 60.00th=[ 4015], | 70.00th=[ 4883], 80.00th=[ 6718], 90.00th=[10814], 95.00th=[13435], | 99.00th=[17957], 99.50th=[19792], 99.90th=[22938], 99.95th=[22938], | 99.99th=[22938] bw ( KiB/s): min= 152, max= 688, per=25.18%, avg=326.03, stdev=148.96, samples=36 iops : min= 38, max= 172, avg=81.36, stdev=37.31, samples=36 write: IOPS=84, BW=340KiB/s (348kB/s)(6236KiB/18357msec) clat (msec): min=2, max=187, avg= 6.98, stdev= 6.20 lat (msec): min=2, max=187, avg= 6.98, stdev= 6.20 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 4], 10.00th=[ 4], 20.00th=[ 4], | 30.00th=[ 5], 40.00th=[ 5], 50.00th=[ 6], 60.00th=[ 7], | 70.00th=[ 8], 80.00th=[ 11], 90.00th=[ 13], 95.00th=[ 15], | 99.00th=[ 24], 99.50th=[ 27], 99.90th=[ 33], 99.95th=[ 188], | 99.99th=[ 188] bw ( KiB/s): min= 176, max= 664, per=24.78%, avg=336.03, stdev=147.27, samples=36 iops : min= 44, max= 166, avg=83.86, stdev=36.88, samples=36 lat (msec) : 2=2.08%, 4=41.89%, 10=40.14%, 20=14.84%, 50=1.01% lat (msec) : 250=0.03% cpu : usr=0.19%, sys=11.37%, ctx=3692, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1295KiB/s (1327kB/s), 320KiB/s-330KiB/s (328kB/s-338kB/s), io=23.4MiB (24.6MB), run=18352-18535msec WRITE: bw=1356KiB/s (1389kB/s), 340KiB/s-342KiB/s (348kB/s-351kB/s), io=24.6MiB (25.7MB), run=18352-18535msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=22222: Sun Dec 8 04:49:23 2024 read: IOPS=49, BW=795KiB/s (814kB/s)(5920KiB/7444msec) clat (usec): min=72, max=37395, avg=13036.96, stdev=4171.94 lat (usec): min=72, max=37396, avg=13037.77, stdev=4171.97 clat percentiles (usec): | 1.00th=[ 100], 5.00th=[ 8848], 10.00th=[ 9634], 20.00th=[10552], | 30.00th=[11207], 40.00th=[11863], 50.00th=[12780], 60.00th=[13304], | 70.00th=[14091], 80.00th=[15008], 90.00th=[16450], 95.00th=[19006], | 99.00th=[32637], 99.50th=[33817], 99.90th=[37487], 99.95th=[37487], | 99.99th=[37487] bw ( KiB/s): min= 640, max= 928, per=25.38%, avg=795.14, stdev=96.33, samples=14 iops : min= 40, max= 58, avg=49.57, stdev= 6.05, samples=14 write: IOPS=53, BW=855KiB/s (876kB/s)(6368KiB/7444msec) clat (usec): min=696, max=60502, avg=6472.53, stdev=7366.92 lat (usec): min=697, max=60504, avg=6475.07, stdev=7366.26 clat percentiles (usec): | 1.00th=[ 848], 5.00th=[ 971], 10.00th=[ 1172], 20.00th=[ 1827], | 30.00th=[ 2900], 40.00th=[ 3359], 50.00th=[ 3884], 60.00th=[ 5014], | 70.00th=[ 6587], 80.00th=[ 9110], 90.00th=[14353], 95.00th=[19792], | 99.00th=[42206], 99.50th=[45876], 99.90th=[60556], 99.95th=[60556], | 99.99th=[60556] bw ( KiB/s): min= 512, max= 1120, per=25.86%, avg=852.21, stdev=176.18, samples=14 iops : min= 32, max= 70, avg=53.14, stdev=10.95, samples=14 lat (usec) : 100=0.52%, 250=0.26%, 500=0.13%, 750=0.26%, 1000=2.73% lat (msec) : 2=8.20%, 4=15.49%, 10=21.35%, 20=46.48%, 50=4.43% lat (msec) : 100=0.13% cpu : usr=0.15%, sys=19.59%, ctx=2819, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22223: Sun Dec 8 04:49:23 2024 read: IOPS=47, BW=762KiB/s (780kB/s)(5744KiB/7543msec) clat (usec): min=64, max=38725, avg=12977.44, stdev=4102.07 lat (usec): min=64, max=38726, avg=12978.19, stdev=4102.07 clat percentiles (usec): | 1.00th=[ 93], 5.00th=[ 8586], 10.00th=[ 9765], 20.00th=[10552], | 30.00th=[11338], 40.00th=[11863], 50.00th=[12518], 60.00th=[13042], | 70.00th=[13829], 80.00th=[15008], 90.00th=[16909], 95.00th=[19268], | 99.00th=[27919], 99.50th=[32375], 99.90th=[38536], 99.95th=[38536], | 99.99th=[38536] bw ( KiB/s): min= 512, max= 990, per=24.16%, avg=756.93, stdev=141.17, samples=15 iops : min= 32, max= 61, avg=47.13, stdev= 8.73, samples=15 write: IOPS=54, BW=868KiB/s (888kB/s)(6544KiB/7543msec) clat (usec): min=777, max=95477, avg=6926.77, stdev=9548.45 lat (usec): min=778, max=95479, avg=6928.12, stdev=9548.46 clat percentiles (usec): | 1.00th=[ 930], 5.00th=[ 1045], 10.00th=[ 1156], 20.00th=[ 1500], | 30.00th=[ 2802], 40.00th=[ 3195], 50.00th=[ 3785], 60.00th=[ 5145], | 70.00th=[ 6587], 80.00th=[ 9634], 90.00th=[15926], 95.00th=[19268], | 99.00th=[49546], 99.50th=[52691], 99.90th=[95945], 99.95th=[95945], | 99.99th=[95945] bw ( KiB/s): min= 544, max= 1312, per=26.33%, avg=867.80, stdev=200.43, samples=15 iops : min= 34, max= 82, avg=54.07, stdev=12.54, samples=15 lat (usec) : 100=0.78%, 1000=1.95% lat (msec) : 2=9.64%, 4=16.41%, 10=20.44%, 20=46.48%, 50=3.78% lat (msec) : 100=0.52% cpu : usr=0.07%, sys=18.55%, ctx=2783, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22224: Sun Dec 8 04:49:23 2024 read: IOPS=51, BW=827KiB/s (847kB/s)(6256KiB/7562msec) clat (usec): min=63, max=89354, avg=13173.77, stdev=5828.37 lat (usec): min=63, max=89356, avg=13174.55, stdev=5828.40 clat percentiles (usec): | 1.00th=[ 67], 5.00th=[ 8586], 10.00th=[ 9372], 20.00th=[10552], | 30.00th=[11338], 40.00th=[12125], 50.00th=[12780], 60.00th=[13435], | 70.00th=[13960], 80.00th=[15008], 90.00th=[16712], 95.00th=[18482], | 99.00th=[36439], 99.50th=[43779], 99.90th=[89654], 99.95th=[89654], | 99.99th=[89654] bw ( KiB/s): min= 576, max= 1056, per=26.49%, avg=829.87, stdev=132.75, samples=15 iops : min= 36, max= 66, avg=51.87, stdev= 8.30, samples=15 write: IOPS=49, BW=798KiB/s (817kB/s)(6032KiB/7562msec) clat (usec): min=648, max=52294, avg=6357.03, stdev=6508.85 lat (usec): min=649, max=52296, avg=6358.93, stdev=6508.86 clat percentiles (usec): | 1.00th=[ 840], 5.00th=[ 996], 10.00th=[ 1156], 20.00th=[ 1745], | 30.00th=[ 2835], 40.00th=[ 3294], 50.00th=[ 3818], 60.00th=[ 5145], | 70.00th=[ 6587], 80.00th=[ 9503], 90.00th=[15926], 95.00th=[18744], | 99.00th=[31065], 99.50th=[37487], 99.90th=[52167], 99.95th=[52167], | 99.99th=[52167] bw ( KiB/s): min= 512, max= 992, per=23.82%, avg=785.07, stdev=167.10, samples=15 iops : min= 32, max= 62, avg=49.07, stdev=10.44, samples=15 lat (usec) : 100=0.65%, 250=0.52%, 750=0.13%, 1000=2.47% lat (msec) : 2=7.94%, 4=14.84%, 10=21.22%, 20=48.31%, 50=3.65% lat (msec) : 100=0.26% cpu : usr=0.12%, sys=20.37%, ctx=2937, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22225: Sun Dec 8 04:49:23 2024 read: IOPS=49, BW=789KiB/s (808kB/s)(6032KiB/7644msec) clat (usec): min=69, max=59778, avg=12946.59, stdev=4509.92 lat (usec): min=70, max=59779, avg=12947.58, stdev=4509.79 clat percentiles (usec): | 1.00th=[ 101], 5.00th=[ 8356], 10.00th=[ 9241], 20.00th=[10421], | 30.00th=[10945], 40.00th=[11863], 50.00th=[12518], 60.00th=[13435], | 70.00th=[14353], 80.00th=[15139], 90.00th=[16909], 95.00th=[18220], | 99.00th=[27657], 99.50th=[31327], 99.90th=[60031], 99.95th=[60031], | 99.99th=[60031] bw ( KiB/s): min= 608, max= 960, per=24.85%, avg=778.67, stdev=126.08, samples=15 iops : min= 38, max= 60, avg=48.67, stdev= 7.88, samples=15 write: IOPS=51, BW=818KiB/s (838kB/s)(6256KiB/7644msec) clat (usec): min=778, max=99853, avg=6942.92, stdev=8789.46 lat (usec): min=780, max=99855, avg=6946.09, stdev=8789.33 clat percentiles (usec): | 1.00th=[ 898], 5.00th=[ 1020], 10.00th=[ 1188], 20.00th=[ 2245], | 30.00th=[ 2868], 40.00th=[ 3359], 50.00th=[ 4047], 60.00th=[ 5407], | 70.00th=[ 7242], 80.00th=[ 9896], 90.00th=[ 15401], 95.00th=[ 19268], | 99.00th=[ 51119], 99.50th=[ 60556], 99.90th=[100140], 99.95th=[100140], | 99.99th=[100140] bw ( KiB/s): min= 608, max= 1120, per=24.66%, avg=812.80, stdev=148.53, samples=15 iops : min= 38, max= 70, avg=50.80, stdev= 9.28, samples=15 lat (usec) : 100=0.39%, 250=0.52%, 1000=2.08% lat (msec) : 2=7.94%, 4=15.23%, 10=22.79%, 20=47.01%, 50=3.39% lat (msec) : 100=0.65% cpu : usr=0.16%, sys=19.14%, ctx=2809, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=3133KiB/s (3209kB/s), 762KiB/s-827KiB/s (780kB/s-847kB/s), io=23.4MiB (24.5MB), run=7444-7644msec WRITE: bw=3297KiB/s (3376kB/s), 798KiB/s-868KiB/s (817kB/s-888kB/s), io=24.6MiB (25.8MB), run=7444-7644msec rand-rw: (groupid=0, jobs=1): err= 0: pid=22226: Sun Dec 8 04:49:23 2024 read: IOPS=49, BW=795KiB/s (814kB/s)(5920KiB/7449msec) clat (usec): min=2158, max=78811, avg=7420.76, stdev=6976.05 lat (usec): min=2159, max=78813, avg=7421.63, stdev=6976.07 clat percentiles (usec): | 1.00th=[ 2278], 5.00th=[ 2540], 10.00th=[ 2769], 20.00th=[ 3163], | 30.00th=[ 3523], 40.00th=[ 4015], 50.00th=[ 4883], 60.00th=[ 6456], | 70.00th=[ 8586], 80.00th=[10814], 90.00th=[13304], 95.00th=[17695], | 99.00th=[32113], 99.50th=[51643], 99.90th=[79168], 99.95th=[79168], | 99.99th=[79168] bw ( KiB/s): min= 608, max= 1024, per=25.46%, avg=797.29, stdev=137.11, samples=14 iops : min= 38, max= 64, avg=49.64, stdev= 8.54, samples=14 write: IOPS=53, BW=855KiB/s (875kB/s)(6368KiB/7449msec) clat (usec): min=2784, max=62397, avg=11778.24, stdev=7305.09 lat (usec): min=2786, max=62400, avg=11779.83, stdev=7305.09 clat percentiles (usec): | 1.00th=[ 3326], 5.00th=[ 4178], 10.00th=[ 4752], 20.00th=[ 6194], | 30.00th=[ 7439], 40.00th=[ 8848], 50.00th=[10290], 60.00th=[11469], | 70.00th=[13435], 80.00th=[15926], 90.00th=[19530], 95.00th=[25035], | 99.00th=[44303], 99.50th=[48497], 99.90th=[62653], 99.95th=[62653], | 99.99th=[62653] bw ( KiB/s): min= 544, max= 1120, per=25.86%, avg=852.07, stdev=163.71, samples=14 iops : min= 34, max= 70, avg=53.07, stdev=10.19, samples=14 lat (msec) : 4=20.70%, 10=40.23%, 20=32.29%, 50=6.38%, 100=0.39% cpu : usr=0.13%, sys=9.67%, ctx=1246, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22227: Sun Dec 8 04:49:23 2024 read: IOPS=47, BW=758KiB/s (776kB/s)(5744KiB/7580msec) clat (usec): min=2100, max=36419, avg=7562.45, stdev=5649.69 lat (usec): min=2101, max=36420, avg=7563.25, stdev=5649.72 clat percentiles (usec): | 1.00th=[ 2212], 5.00th=[ 2507], 10.00th=[ 2769], 20.00th=[ 3097], | 30.00th=[ 3523], 40.00th=[ 4228], 50.00th=[ 5145], 60.00th=[ 6783], | 70.00th=[ 9896], 80.00th=[11994], 90.00th=[14877], 95.00th=[17957], | 99.00th=[30540], 99.50th=[31065], 99.90th=[36439], 99.95th=[36439], | 99.99th=[36439] bw ( KiB/s): min= 480, max= 992, per=24.10%, avg=754.93, stdev=162.64, samples=15 iops : min= 30, max= 62, avg=47.07, stdev=10.17, samples=15 write: IOPS=53, BW=863KiB/s (884kB/s)(6544KiB/7580msec) clat (usec): min=3039, max=91877, avg=11751.52, stdev=9296.90 lat (usec): min=3040, max=91879, avg=11753.39, stdev=9296.85 clat percentiles (usec): | 1.00th=[ 3490], 5.00th=[ 3982], 10.00th=[ 4293], 20.00th=[ 5342], | 30.00th=[ 6980], 40.00th=[ 8160], 50.00th=[ 9634], 60.00th=[11076], | 70.00th=[12911], 80.00th=[15139], 90.00th=[20841], 95.00th=[26084], | 99.00th=[49546], 99.50th=[51119], 99.90th=[91751], 99.95th=[91751], | 99.99th=[91751] bw ( KiB/s): min= 608, max= 1216, per=26.15%, avg=861.53, stdev=170.88, samples=15 iops : min= 38, max= 76, avg=53.73, stdev=10.57, samples=15 lat (msec) : 4=19.92%, 10=41.41%, 20=31.51%, 50=6.64%, 100=0.52% cpu : usr=0.16%, sys=9.68%, ctx=1271, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22228: Sun Dec 8 04:49:23 2024 read: IOPS=51, BW=824KiB/s (844kB/s)(6256KiB/7592msec) clat (msec): min=2, max=104, avg= 8.17, stdev= 7.80 lat (msec): min=2, max=104, avg= 8.18, stdev= 7.80 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 3], 20.00th=[ 4], | 30.00th=[ 4], 40.00th=[ 5], 50.00th=[ 6], 60.00th=[ 9], | 70.00th=[ 11], 80.00th=[ 13], 90.00th=[ 15], 95.00th=[ 21], | 99.00th=[ 34], 99.50th=[ 45], 99.90th=[ 105], 99.95th=[ 105], | 99.99th=[ 105] bw ( KiB/s): min= 544, max= 1120, per=26.22%, avg=821.07, stdev=165.81, samples=15 iops : min= 34, max= 70, avg=51.20, stdev=10.42, samples=15 write: IOPS=49, BW=795KiB/s (814kB/s)(6032KiB/7592msec) clat (usec): min=3095, max=55425, avg=11499.07, stdev=6978.09 lat (usec): min=3096, max=55427, avg=11500.54, stdev=6978.13 clat percentiles (usec): | 1.00th=[ 3425], 5.00th=[ 4113], 10.00th=[ 4621], 20.00th=[ 6128], | 30.00th=[ 7242], 40.00th=[ 8455], 50.00th=[ 9896], 60.00th=[11600], | 70.00th=[13173], 80.00th=[15795], 90.00th=[20055], 95.00th=[23725], | 99.00th=[36963], 99.50th=[51643], 99.90th=[55313], 99.95th=[55313], | 99.99th=[55313] bw ( KiB/s): min= 544, max= 992, per=23.69%, avg=780.53, stdev=155.62, samples=15 iops : min= 34, max= 62, avg=48.67, stdev= 9.68, samples=15 lat (msec) : 4=21.74%, 10=37.63%, 20=33.20%, 50=7.03%, 100=0.26% lat (msec) : 250=0.13% cpu : usr=0.08%, sys=9.45%, ctx=1279, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22229: Sun Dec 8 04:49:23 2024 read: IOPS=49, BW=789KiB/s (808kB/s)(6032KiB/7646msec) clat (usec): min=1864, max=102178, avg=8067.25, stdev=7903.10 lat (usec): min=1864, max=102178, avg=8068.08, stdev=7903.10 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 3], 20.00th=[ 3], | 30.00th=[ 4], 40.00th=[ 4], 50.00th=[ 6], 60.00th=[ 7], | 70.00th=[ 11], 80.00th=[ 13], 90.00th=[ 17], 95.00th=[ 21], | 99.00th=[ 31], 99.50th=[ 40], 99.90th=[ 103], 99.95th=[ 103], | 99.99th=[ 103] bw ( KiB/s): min= 576, max= 1120, per=24.58%, avg=769.87, stdev=163.63, samples=15 iops : min= 36, max= 70, avg=48.00, stdev=10.34, samples=15 write: IOPS=51, BW=818KiB/s (838kB/s)(6256KiB/7646msec) clat (usec): min=3102, max=66856, avg=11551.36, stdev=7653.29 lat (usec): min=3103, max=66859, avg=11552.90, stdev=7653.31 clat percentiles (usec): | 1.00th=[ 3621], 5.00th=[ 4146], 10.00th=[ 4621], 20.00th=[ 6063], | 30.00th=[ 7308], 40.00th=[ 8586], 50.00th=[ 9896], 60.00th=[11338], | 70.00th=[12911], 80.00th=[15139], 90.00th=[19792], 95.00th=[25035], | 99.00th=[44827], 99.50th=[60031], 99.90th=[66847], 99.95th=[66847], | 99.99th=[66847] bw ( KiB/s): min= 544, max= 1152, per=24.47%, avg=806.13, stdev=147.63, samples=15 iops : min= 34, max= 72, avg=50.27, stdev= 9.18, samples=15 lat (msec) : 2=0.13%, 4=21.61%, 10=38.28%, 20=32.55%, 50=6.90% lat (msec) : 100=0.39%, 250=0.13% cpu : usr=0.13%, sys=9.35%, ctx=1236, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=3133KiB/s (3208kB/s), 758KiB/s-824KiB/s (776kB/s-844kB/s), io=23.4MiB (24.5MB), run=7449-7646msec WRITE: bw=3296KiB/s (3375kB/s), 795KiB/s-863KiB/s (814kB/s-884kB/s), io=24.6MiB (25.8MB), run=7449-7646msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=22245: Sun Dec 8 04:49:26 2024 read: IOPS=3, BW=3444KiB/s (3527kB/s)(7168KiB/2081msec) clat (msec): min=4, max=631, avg=219.91, stdev=248.12 lat (msec): min=4, max=631, avg=219.91, stdev=248.12 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 5], 10.00th=[ 5], 20.00th=[ 46], | 30.00th=[ 51], 40.00th=[ 51], 50.00th=[ 131], 60.00th=[ 167], | 70.00th=[ 167], 80.00th=[ 510], 90.00th=[ 634], 95.00th=[ 634], | 99.00th=[ 634], 99.50th=[ 634], 99.90th=[ 634], 99.95th=[ 634], | 99.99th=[ 634] bw ( KiB/s): min= 2043, max= 4096, per=31.20%, avg=3070.75, stdev=1183.86, samples=4 iops : min= 1, max= 4, avg= 2.75, stdev= 1.50, samples=4 write: IOPS=2, BW=2460KiB/s (2519kB/s)(5120KiB/2081msec) clat (msec): min=24, max=328, avg=105.99, stdev=130.62 lat (msec): min=24, max=328, avg=106.05, stdev=130.63 clat percentiles (msec): | 1.00th=[ 25], 5.00th=[ 25], 10.00th=[ 25], 20.00th=[ 25], | 30.00th=[ 29], 40.00th=[ 29], 50.00th=[ 30], 60.00th=[ 30], | 70.00th=[ 118], 80.00th=[ 118], 90.00th=[ 330], 95.00th=[ 330], | 99.00th=[ 330], 99.50th=[ 330], 99.90th=[ 330], 99.95th=[ 330], | 99.99th=[ 330] bw ( KiB/s): min= 4096, max= 6144, per=37.16%, avg=5120.00, stdev=1448.15, samples=2 iops : min= 4, max= 6, avg= 5.00, stdev= 1.41, samples=2 lat (msec) : 10=8.33%, 50=33.33%, 100=8.33%, 250=25.00%, 500=8.33% lat (msec) : 750=16.67% cpu : usr=0.10%, sys=23.51%, ctx=676, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22246: Sun Dec 8 04:49:26 2024 read: IOPS=0, BW=985KiB/s (1009kB/s)(2048KiB/2079msec) clat (usec): min=5974, max=41893, avg=23933.95, stdev=25398.11 lat (usec): min=5975, max=41895, avg=23935.43, stdev=25399.01 clat percentiles (usec): | 1.00th=[ 5997], 5.00th=[ 5997], 10.00th=[ 5997], 20.00th=[ 5997], | 30.00th=[ 5997], 40.00th=[ 5997], 50.00th=[ 5997], 60.00th=[41681], | 70.00th=[41681], 80.00th=[41681], 90.00th=[41681], 95.00th=[41681], | 99.00th=[41681], 99.50th=[41681], 99.90th=[41681], 99.95th=[41681], | 99.99th=[41681] bw ( KiB/s): min= 2048, max= 2048, per=20.81%, avg=2048.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 write: IOPS=4, BW=4925KiB/s (5044kB/s)(10.0MiB/2079msec) clat (msec): min=23, max=679, avg=196.83, stdev=193.00 lat (msec): min=23, max=679, avg=196.89, stdev=193.01 clat percentiles (msec): | 1.00th=[ 24], 5.00th=[ 24], 10.00th=[ 24], 20.00th=[ 25], | 30.00th=[ 44], 40.00th=[ 110], 50.00th=[ 180], 60.00th=[ 184], | 70.00th=[ 224], 80.00th=[ 224], 90.00th=[ 300], 95.00th=[ 676], | 99.00th=[ 676], 99.50th=[ 676], 99.90th=[ 676], 99.95th=[ 676], | 99.99th=[ 676] bw ( KiB/s): min= 4096, max= 7148, per=42.07%, avg=5796.00, stdev=1555.48, samples=3 iops : min= 4, max= 6, avg= 5.33, stdev= 1.15, samples=3 lat (msec) : 10=8.33%, 50=33.33%, 250=41.67%, 500=8.33%, 750=8.33% cpu : usr=0.00%, sys=15.98%, ctx=334, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22247: Sun Dec 8 04:49:26 2024 read: IOPS=3, BW=3247KiB/s (3325kB/s)(6144KiB/1892msec) clat (usec): min=503, max=516036, avg=201008.50, stdev=200369.65 lat (usec): min=505, max=516037, avg=201009.49, stdev=200369.50 clat percentiles (usec): | 1.00th=[ 506], 5.00th=[ 506], 10.00th=[ 506], 20.00th=[ 54264], | 30.00th=[ 54264], 40.00th=[ 74974], 50.00th=[ 74974], 60.00th=[206570], | 70.00th=[354419], 80.00th=[354419], 90.00th=[517997], 95.00th=[517997], | 99.00th=[517997], 99.50th=[517997], 99.90th=[517997], 99.95th=[517997], | 99.99th=[517997] bw ( KiB/s): min= 4096, max= 6144, per=52.03%, avg=5120.00, stdev=1448.15, samples=2 iops : min= 4, max= 6, avg= 5.00, stdev= 1.41, samples=2 write: IOPS=3, BW=3247KiB/s (3325kB/s)(6144KiB/1892msec) clat (msec): min=30, max=202, avg=103.58, stdev=80.86 lat (msec): min=30, max=202, avg=103.64, stdev=80.87 clat percentiles (msec): | 1.00th=[ 31], 5.00th=[ 31], 10.00th=[ 31], 20.00th=[ 32], | 30.00th=[ 32], 40.00th=[ 35], 50.00th=[ 35], 60.00th=[ 140], | 70.00th=[ 184], 80.00th=[ 184], 90.00th=[ 203], 95.00th=[ 203], | 99.00th=[ 203], 99.50th=[ 203], 99.90th=[ 203], 99.95th=[ 203], | 99.99th=[ 203] bw ( KiB/s): min= 2048, max= 6144, per=24.78%, avg=3413.33, stdev=2364.83, samples=3 iops : min= 2, max= 6, avg= 3.33, stdev= 2.31, samples=3 lat (usec) : 750=8.33% lat (msec) : 50=25.00%, 100=16.67%, 250=33.33%, 500=8.33%, 750=8.33% cpu : usr=0.00%, sys=23.00%, ctx=615, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22248: Sun Dec 8 04:49:26 2024 read: IOPS=2, BW=2710KiB/s (2775kB/s)(5120KiB/1889msec) clat (msec): min=7, max=557, avg=243.57, stdev=224.63 lat (msec): min=7, max=557, avg=243.58, stdev=224.63 clat percentiles (msec): | 1.00th=[ 8], 5.00th=[ 8], 10.00th=[ 8], 20.00th=[ 8], | 30.00th=[ 44], 40.00th=[ 44], 50.00th=[ 300], 60.00th=[ 300], | 70.00th=[ 309], 80.00th=[ 309], 90.00th=[ 558], 95.00th=[ 558], | 99.00th=[ 558], 99.50th=[ 558], 99.90th=[ 558], 99.95th=[ 558], | 99.99th=[ 558] bw ( KiB/s): min= 2048, max= 6144, per=41.62%, avg=4096.00, stdev=2896.31, samples=2 iops : min= 2, max= 6, avg= 4.00, stdev= 2.83, samples=2 write: IOPS=3, BW=3795KiB/s (3886kB/s)(7168KiB/1889msec) clat (msec): min=24, max=198, avg=86.50, stdev=67.46 lat (msec): min=24, max=198, avg=86.56, stdev=67.47 clat percentiles (msec): | 1.00th=[ 24], 5.00th=[ 24], 10.00th=[ 24], 20.00th=[ 36], | 30.00th=[ 37], 40.00th=[ 37], 50.00th=[ 56], 60.00th=[ 100], | 70.00th=[ 100], 80.00th=[ 157], 90.00th=[ 199], 95.00th=[ 199], | 99.00th=[ 199], 99.50th=[ 199], 99.90th=[ 199], 99.95th=[ 199], | 99.99th=[ 199] bw ( KiB/s): min= 2048, max= 6144, per=29.73%, avg=4096.00, stdev=2896.31, samples=2 iops : min= 2, max= 6, avg= 4.00, stdev= 2.83, samples=2 lat (msec) : 10=8.33%, 50=33.33%, 100=16.67%, 250=16.67%, 500=16.67% lat (msec) : 750=8.33% cpu : usr=0.05%, sys=33.21%, ctx=605, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=9841KiB/s (10.1MB/s), 985KiB/s-3444KiB/s (1009kB/s-3527kB/s), io=20.0MiB (20.0MB), run=1889-2081msec WRITE: bw=13.5MiB/s (14.1MB/s), 2460KiB/s-4925KiB/s (2519kB/s-5044kB/s), io=28.0MiB (29.4MB), run=1889-2081msec rand-rw: (groupid=0, jobs=1): err= 0: pid=22241: Sun Dec 8 04:49:26 2024 read: IOPS=3, BW=3254KiB/s (3332kB/s)(7168KiB/2203msec) clat (msec): min=17, max=486, avg=190.35, stdev=144.41 lat (msec): min=17, max=486, avg=190.35, stdev=144.41 clat percentiles (msec): | 1.00th=[ 18], 5.00th=[ 18], 10.00th=[ 18], 20.00th=[ 122], | 30.00th=[ 148], 40.00th=[ 148], 50.00th=[ 174], 60.00th=[ 178], | 70.00th=[ 178], 80.00th=[ 209], 90.00th=[ 489], 95.00th=[ 489], | 99.00th=[ 489], 99.50th=[ 489], 99.90th=[ 489], 99.95th=[ 489], | 99.99th=[ 489] bw ( KiB/s): min= 1753, max= 4096, per=25.16%, avg=2338.75, stdev=1171.50, samples=4 iops : min= 1, max= 4, avg= 1.75, stdev= 1.50, samples=4 write: IOPS=2, BW=2324KiB/s (2380kB/s)(5120KiB/2203msec) clat (msec): min=18, max=470, avg=160.68, stdev=196.43 lat (msec): min=18, max=470, avg=160.73, stdev=196.44 clat percentiles (msec): | 1.00th=[ 19], 5.00th=[ 19], 10.00th=[ 19], 20.00th=[ 19], | 30.00th=[ 19], 40.00th=[ 19], 50.00th=[ 55], 60.00th=[ 55], | 70.00th=[ 243], 80.00th=[ 243], 90.00th=[ 472], 95.00th=[ 472], | 99.00th=[ 472], 99.50th=[ 472], 99.90th=[ 472], 99.95th=[ 472], | 99.99th=[ 472] bw ( KiB/s): min= 1753, max= 5260, per=26.94%, avg=3506.50, stdev=2479.82, samples=2 iops : min= 1, max= 5, avg= 3.00, stdev= 2.83, samples=2 lat (msec) : 20=25.00%, 100=8.33%, 250=50.00%, 500=16.67% cpu : usr=0.00%, sys=6.13%, ctx=101, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22242: Sun Dec 8 04:49:26 2024 read: IOPS=2, BW=2133KiB/s (2185kB/s)(2048KiB/960msec) clat (usec): min=16268, max=73049, avg=44659.12, stdev=40150.45 lat (usec): min=16270, max=73050, avg=44660.75, stdev=40149.26 clat percentiles (usec): | 1.00th=[16319], 5.00th=[16319], 10.00th=[16319], 20.00th=[16319], | 30.00th=[16319], 40.00th=[16319], 50.00th=[16319], 60.00th=[72877], | 70.00th=[72877], 80.00th=[72877], 90.00th=[72877], 95.00th=[72877], | 99.00th=[72877], 99.50th=[72877], 99.90th=[72877], 99.95th=[72877], | 99.99th=[72877] bw ( KiB/s): min= 2048, max= 2048, per=22.03%, avg=2048.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=10, BW=10.4MiB/s (10.9MB/s)(10.0MiB/960msec) clat (msec): min=18, max=270, avg=85.70, stdev=94.77 lat (msec): min=18, max=270, avg=85.78, stdev=94.77 clat percentiles (msec): | 1.00th=[ 19], 5.00th=[ 19], 10.00th=[ 19], 20.00th=[ 19], | 30.00th=[ 20], 40.00th=[ 26], 50.00th=[ 29], 60.00th=[ 65], | 70.00th=[ 99], 80.00th=[ 99], 90.00th=[ 245], 95.00th=[ 271], | 99.00th=[ 271], 99.50th=[ 271], 99.90th=[ 271], 99.95th=[ 271], | 99.99th=[ 271] bw ( KiB/s): min= 6144, max= 6144, per=47.21%, avg=6144.00, stdev= 0.00, samples=1 iops : min= 6, max= 6, avg= 6.00, stdev= 0.00, samples=1 lat (msec) : 20=33.33%, 50=16.67%, 100=33.33%, 250=8.33%, 500=8.33% cpu : usr=0.00%, sys=12.93%, ctx=68, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22243: Sun Dec 8 04:49:26 2024 read: IOPS=6, BW=6729KiB/s (6891kB/s)(6144KiB/913msec) clat (msec): min=14, max=262, avg=77.39, stdev=94.15 lat (msec): min=14, max=262, avg=77.39, stdev=94.15 clat percentiles (msec): | 1.00th=[ 15], 5.00th=[ 15], 10.00th=[ 15], 20.00th=[ 20], | 30.00th=[ 20], 40.00th=[ 38], 50.00th=[ 38], 60.00th=[ 47], | 70.00th=[ 84], 80.00th=[ 84], 90.00th=[ 264], 95.00th=[ 264], | 99.00th=[ 264], 99.50th=[ 264], 99.90th=[ 264], 99.95th=[ 264], | 99.99th=[ 264] bw ( KiB/s): min= 4096, max= 4096, per=44.06%, avg=4096.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 write: IOPS=6, BW=6729KiB/s (6891kB/s)(6144KiB/913msec) clat (msec): min=17, max=296, avg=67.39, stdev=112.34 lat (msec): min=17, max=297, avg=67.52, stdev=112.51 clat percentiles (msec): | 1.00th=[ 18], 5.00th=[ 18], 10.00th=[ 18], 20.00th=[ 18], | 30.00th=[ 18], 40.00th=[ 22], 50.00th=[ 22], 60.00th=[ 23], | 70.00th=[ 29], 80.00th=[ 29], 90.00th=[ 296], 95.00th=[ 296], | 99.00th=[ 296], 99.50th=[ 296], 99.90th=[ 296], 99.95th=[ 296], | 99.99th=[ 296] bw ( KiB/s): min= 8192, max= 8192, per=62.95%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 lat (msec) : 20=33.33%, 50=41.67%, 100=8.33%, 500=16.67% cpu : usr=0.22%, sys=9.10%, ctx=52, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22244: Sun Dec 8 04:49:26 2024 read: IOPS=5, BW=5695KiB/s (5832kB/s)(5120KiB/899msec) clat (msec): min=13, max=385, avg=89.72, stdev=165.24 lat (msec): min=13, max=385, avg=89.72, stdev=165.24 clat percentiles (msec): | 1.00th=[ 14], 5.00th=[ 14], 10.00th=[ 14], 20.00th=[ 14], | 30.00th=[ 15], 40.00th=[ 15], 50.00th=[ 16], 60.00th=[ 16], | 70.00th=[ 21], 80.00th=[ 21], 90.00th=[ 384], 95.00th=[ 384], | 99.00th=[ 384], 99.50th=[ 384], 99.90th=[ 384], 99.95th=[ 384], | 99.99th=[ 384] bw ( KiB/s): min= 8192, max= 8192, per=88.12%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=7, BW=7973KiB/s (8165kB/s)(7168KiB/899msec) clat (msec): min=15, max=249, avg=59.51, stdev=85.30 lat (msec): min=15, max=249, avg=59.57, stdev=85.31 clat percentiles (msec): | 1.00th=[ 16], 5.00th=[ 16], 10.00th=[ 16], 20.00th=[ 17], | 30.00th=[ 22], 40.00th=[ 22], 50.00th=[ 26], 60.00th=[ 26], | 70.00th=[ 26], 80.00th=[ 63], 90.00th=[ 251], 95.00th=[ 251], | 99.00th=[ 251], 99.50th=[ 251], 99.90th=[ 251], 99.95th=[ 251], | 99.99th=[ 251] bw ( KiB/s): min=12288, max=12288, per=94.42%, avg=12288.00, stdev= 0.00, samples=1 iops : min= 12, max= 12, avg=12.00, stdev= 0.00, samples=1 lat (msec) : 20=50.00%, 50=25.00%, 100=8.33%, 250=8.33%, 500=8.33% cpu : usr=0.00%, sys=8.35%, ctx=33, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=9296KiB/s (9520kB/s), 2133KiB/s-6729KiB/s (2185kB/s-6891kB/s), io=20.0MiB (20.0MB), run=899-2203msec WRITE: bw=12.7MiB/s (13.3MB/s), 2324KiB/s-10.4MiB/s (2380kB/s-10.9MB/s), io=28.0MiB (29.4MB), run=899-2203msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=22255: Sun Dec 8 04:49:29 2024 read: IOPS=3, BW=15.1MiB/s (15.8MB/s)(12.0MiB/794msec) clat (msec): min=71, max=495, avg=248.78, stdev=220.35 lat (msec): min=71, max=495, avg=248.78, stdev=220.35 clat percentiles (msec): | 1.00th=[ 72], 5.00th=[ 72], 10.00th=[ 72], 20.00th=[ 72], | 30.00th=[ 72], 40.00th=[ 180], 50.00th=[ 180], 60.00th=[ 180], | 70.00th=[ 498], 80.00th=[ 498], 90.00th=[ 498], 95.00th=[ 498], | 99.00th=[ 498], 99.50th=[ 498], 99.90th=[ 498], 99.95th=[ 498], | 99.99th=[ 498] bw ( KiB/s): min=16384, max=16384, per=94.20%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 250=33.33%, 500=33.33% cpu : usr=0.25%, sys=29.26%, ctx=353, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22256: Sun Dec 8 04:49:29 2024 read: IOPS=1, BW=4348KiB/s (4453kB/s)(4096KiB/942msec) clat (nsec): min=235352k, max=235352k, avg=235351703.00, stdev= 0.00 lat (nsec): min=235354k, max=235354k, avg=235353816.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 236], 5.00th=[ 236], 10.00th=[ 236], 20.00th=[ 236], | 30.00th=[ 236], 40.00th=[ 236], 50.00th=[ 236], 60.00th=[ 236], | 70.00th=[ 236], 80.00th=[ 236], 90.00th=[ 236], 95.00th=[ 236], | 99.00th=[ 236], 99.50th=[ 236], 99.90th=[ 236], 99.95th=[ 236], | 99.99th=[ 236] bw ( KiB/s): min= 8192, max= 8192, per=47.10%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=2, BW=8696KiB/s (8905kB/s)(8192KiB/942msec) clat (msec): min=105, max=566, avg=336.00, stdev=326.52 lat (msec): min=105, max=567, avg=336.25, stdev=326.52 clat percentiles (msec): | 1.00th=[ 106], 5.00th=[ 106], 10.00th=[ 106], 20.00th=[ 106], | 30.00th=[ 106], 40.00th=[ 106], 50.00th=[ 106], 60.00th=[ 567], | 70.00th=[ 567], 80.00th=[ 567], 90.00th=[ 567], 95.00th=[ 567], | 99.00th=[ 567], 99.50th=[ 567], 99.90th=[ 567], 99.95th=[ 567], | 99.99th=[ 567] lat (msec) : 250=66.67%, 750=33.33% cpu : usr=0.00%, sys=39.32%, ctx=123, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22257: Sun Dec 8 04:49:29 2024 write: IOPS=2, BW=10.5MiB/s (10.0MB/s)(12.0MiB/1148msec) clat (msec): min=211, max=632, avg=361.52, stdev=235.09 lat (msec): min=212, max=632, avg=361.75, stdev=235.13 clat percentiles (msec): | 1.00th=[ 213], 5.00th=[ 213], 10.00th=[ 213], 20.00th=[ 213], | 30.00th=[ 213], 40.00th=[ 241], 50.00th=[ 241], 60.00th=[ 241], | 70.00th=[ 634], 80.00th=[ 634], 90.00th=[ 634], 95.00th=[ 634], | 99.00th=[ 634], 99.50th=[ 634], 99.90th=[ 634], 99.95th=[ 634], | 99.99th=[ 634] bw ( KiB/s): min= 7937, max= 8192, per=28.25%, avg=8064.50, stdev=180.31, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 250=66.67%, 750=33.33% cpu : usr=0.00%, sys=29.03%, ctx=69, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22259: Sun Dec 8 04:49:29 2024 write: IOPS=4, BW=18.4MiB/s (19.3MB/s)(12.0MiB/651msec) clat (msec): min=126, max=336, avg=211.19, stdev=110.33 lat (msec): min=127, max=336, avg=211.47, stdev=110.32 clat percentiles (msec): | 1.00th=[ 128], 5.00th=[ 128], 10.00th=[ 128], 20.00th=[ 128], | 30.00th=[ 128], 40.00th=[ 171], 50.00th=[ 171], 60.00th=[ 171], | 70.00th=[ 338], 80.00th=[ 338], 90.00th=[ 338], 95.00th=[ 338], | 99.00th=[ 338], 99.50th=[ 338], 99.90th=[ 338], 99.95th=[ 338], | 99.99th=[ 338] bw ( KiB/s): min=16351, max=16351, per=57.29%, avg=16351.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.15%, sys=55.69%, ctx=28, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=16.0MiB/s (17.8MB/s), 4348KiB/s-15.1MiB/s (4453kB/s-15.8MB/s), io=16.0MiB (16.8MB), run=794-942msec WRITE: bw=27.9MiB/s (29.2MB/s), 8696KiB/s-18.4MiB/s (8905kB/s-19.3MB/s), io=32.0MiB (33.6MB), run=651-1148msec rand-rw: (groupid=0, jobs=1): err= 0: pid=22258: Sun Dec 8 04:49:29 2024 read: IOPS=2, BW=9.83MiB/s (10.3MB/s)(12.0MiB/1221msec) clat (msec): min=175, max=619, avg=389.50, stdev=222.31 lat (msec): min=175, max=619, avg=389.50, stdev=222.31 clat percentiles (msec): | 1.00th=[ 176], 5.00th=[ 176], 10.00th=[ 176], 20.00th=[ 176], | 30.00th=[ 176], 40.00th=[ 372], 50.00th=[ 372], 60.00th=[ 372], | 70.00th=[ 617], 80.00th=[ 617], 90.00th=[ 617], 95.00th=[ 617], | 99.00th=[ 617], 99.50th=[ 617], 99.90th=[ 617], 99.95th=[ 617], | 99.99th=[ 617] bw ( KiB/s): min= 8192, max= 8192, per=61.05%, avg=8192.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 lat (msec) : 250=33.33%, 500=33.33%, 750=33.33% cpu : usr=0.00%, sys=9.67%, ctx=109, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22260: Sun Dec 8 04:49:29 2024 read: IOPS=0, BW=3430KiB/s (3513kB/s)(4096KiB/1194msec) clat (nsec): min=378761k, max=378761k, avg=378760672.00, stdev= 0.00 lat (nsec): min=378763k, max=378763k, avg=378763059.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 380], 5.00th=[ 380], 10.00th=[ 380], 20.00th=[ 380], | 30.00th=[ 380], 40.00th=[ 380], 50.00th=[ 380], 60.00th=[ 380], | 70.00th=[ 380], 80.00th=[ 380], 90.00th=[ 380], 95.00th=[ 380], | 99.00th=[ 380], 99.50th=[ 380], 99.90th=[ 380], 99.95th=[ 380], | 99.99th=[ 380] bw ( KiB/s): min= 8192, max= 8192, per=61.05%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=1, BW=6861KiB/s (7026kB/s)(8192KiB/1194msec) clat (msec): min=220, max=494, avg=357.45, stdev=193.77 lat (msec): min=220, max=494, avg=357.71, stdev=193.76 clat percentiles (msec): | 1.00th=[ 222], 5.00th=[ 222], 10.00th=[ 222], 20.00th=[ 222], | 30.00th=[ 222], 40.00th=[ 222], 50.00th=[ 222], 60.00th=[ 493], | 70.00th=[ 493], 80.00th=[ 493], 90.00th=[ 493], 95.00th=[ 493], | 99.00th=[ 493], 99.50th=[ 493], 99.90th=[ 493], 99.95th=[ 493], | 99.99th=[ 493] bw ( KiB/s): min= 8192, max= 8192, per=30.83%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=66.67% cpu : usr=0.08%, sys=11.06%, ctx=64, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22261: Sun Dec 8 04:49:29 2024 write: IOPS=2, BW=9966KiB/s (10.2MB/s)(12.0MiB/1233msec) clat (msec): min=174, max=616, avg=387.98, stdev=221.33 lat (msec): min=174, max=616, avg=388.25, stdev=221.30 clat percentiles (msec): | 1.00th=[ 176], 5.00th=[ 176], 10.00th=[ 176], 20.00th=[ 176], | 30.00th=[ 176], 40.00th=[ 372], 50.00th=[ 372], 60.00th=[ 372], | 70.00th=[ 617], 80.00th=[ 617], 90.00th=[ 617], 95.00th=[ 617], | 99.00th=[ 617], 99.50th=[ 617], 99.90th=[ 617], 99.95th=[ 617], | 99.99th=[ 617] bw ( KiB/s): min= 8192, max= 8192, per=30.83%, avg=8192.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 lat (msec) : 250=33.33%, 500=33.33%, 750=33.33% cpu : usr=0.08%, sys=6.41%, ctx=35, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22262: Sun Dec 8 04:49:29 2024 write: IOPS=2, BW=9.96MiB/s (10.4MB/s)(12.0MiB/1205msec) clat (msec): min=108, max=820, avg=386.91, stdev=380.88 lat (msec): min=108, max=821, avg=387.19, stdev=380.88 clat percentiles (msec): | 1.00th=[ 109], 5.00th=[ 109], 10.00th=[ 109], 20.00th=[ 109], | 30.00th=[ 109], 40.00th=[ 232], 50.00th=[ 232], 60.00th=[ 232], | 70.00th=[ 818], 80.00th=[ 818], 90.00th=[ 818], 95.00th=[ 818], | 99.00th=[ 818], 99.50th=[ 818], 99.90th=[ 818], 99.95th=[ 818], | 99.99th=[ 818] bw ( KiB/s): min= 8192, max= 8192, per=30.83%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 1000=33.33% cpu : usr=0.17%, sys=14.70%, ctx=125, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=13.1MiB/s (13.7MB/s), 3430KiB/s-9.83MiB/s (3513kB/s-10.3MB/s), io=16.0MiB (16.8MB), run=1194-1221msec WRITE: bw=25.0MiB/s (27.2MB/s), 6861KiB/s-9.96MiB/s (7026kB/s-10.4MB/s), io=32.0MiB (33.6MB), run=1194-1233msec