== sanity test 398b: DIO and buffer IO race ============== 05:39:33 (1727775573) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.31137 s, 38.4 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=5923: Tue Oct 1 05:39:49 2024 read: IOPS=107, BW=430KiB/s (440kB/s)(5992KiB/13935msec) clat (usec): min=51, max=170682, avg=5172.32, stdev=5179.04 lat (usec): min=51, max=170683, avg=5173.76, stdev=5178.98 clat percentiles (usec): | 1.00th=[ 63], 5.00th=[ 2008], 10.00th=[ 2474], 20.00th=[ 2999], | 30.00th=[ 3425], 40.00th=[ 3884], 50.00th=[ 4490], 60.00th=[ 5211], | 70.00th=[ 5997], 80.00th=[ 6849], 90.00th=[ 8455], 95.00th=[ 9896], | 99.00th=[ 15926], 99.50th=[ 18220], 99.90th=[ 30540], 99.95th=[170918], | 99.99th=[170918] bw ( KiB/s): min= 184, max= 560, per=24.66%, avg=424.48, stdev=71.50, samples=27 iops : min= 46, max= 140, avg=106.04, stdev=17.84, samples=27 write: IOPS=112, BW=452KiB/s (463kB/s)(6296KiB/13935msec) clat (usec): min=483, max=23458, avg=3872.04, stdev=3359.66 lat (usec): min=484, max=23459, avg=3874.13, stdev=3359.05 clat percentiles (usec): | 1.00th=[ 537], 5.00th=[ 578], 10.00th=[ 627], 20.00th=[ 758], | 30.00th=[ 1012], 40.00th=[ 2343], 50.00th=[ 3228], 60.00th=[ 3818], | 70.00th=[ 4883], 80.00th=[ 6652], 90.00th=[ 8717], 95.00th=[10421], | 99.00th=[13304], 99.50th=[15270], 99.90th=[21627], 99.95th=[23462], | 99.99th=[23462] bw ( KiB/s): min= 224, max= 600, per=24.62%, avg=443.74, stdev=68.53, samples=27 iops : min= 56, max= 150, avg=110.85, stdev=17.20, samples=27 lat (usec) : 100=1.40%, 250=0.85%, 500=0.13%, 750=9.83%, 1000=5.34% lat (msec) : 2=4.13%, 4=30.89%, 10=41.89%, 20=5.27%, 50=0.23% lat (msec) : 250=0.03% cpu : usr=0.23%, sys=23.59%, ctx=6064, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5924: Tue Oct 1 05:39:49 2024 read: IOPS=106, BW=427KiB/s (437kB/s)(5940KiB/13912msec) clat (usec): min=51, max=25202, avg=5178.19, stdev=2810.94 lat (usec): min=52, max=25203, avg=5179.26, stdev=2810.86 clat percentiles (usec): | 1.00th=[ 63], 5.00th=[ 1975], 10.00th=[ 2474], 20.00th=[ 3032], | 30.00th=[ 3523], 40.00th=[ 4080], 50.00th=[ 4817], 60.00th=[ 5473], | 70.00th=[ 6194], 80.00th=[ 7111], 90.00th=[ 8717], 95.00th=[ 9896], | 99.00th=[14877], 99.50th=[16712], 99.90th=[21103], 99.95th=[25297], | 99.99th=[25297] bw ( KiB/s): min= 240, max= 576, per=24.55%, avg=422.44, stdev=65.81, samples=27 iops : min= 60, max= 144, avg=105.56, stdev=16.47, samples=27 write: IOPS=114, BW=456KiB/s (467kB/s)(6348KiB/13912msec) clat (usec): min=402, max=168980, avg=3863.79, stdev=5301.11 lat (usec): min=402, max=168981, avg=3864.70, stdev=5301.10 clat percentiles (usec): | 1.00th=[ 537], 5.00th=[ 570], 10.00th=[ 619], 20.00th=[ 742], | 30.00th=[ 988], 40.00th=[ 2057], 50.00th=[ 3130], 60.00th=[ 3818], | 70.00th=[ 4883], 80.00th=[ 6390], 90.00th=[ 8225], 95.00th=[ 10028], | 99.00th=[ 14877], 99.50th=[ 17433], 99.90th=[ 26608], 99.95th=[168821], | 99.99th=[168821] bw ( KiB/s): min= 288, max= 576, per=24.66%, avg=444.37, stdev=73.04, samples=27 iops : min= 72, max= 144, avg=111.04, stdev=18.29, samples=27 lat (usec) : 100=1.40%, 250=0.81%, 500=0.10%, 750=10.55%, 1000=5.11% lat (msec) : 2=4.92%, 4=27.73%, 10=44.40%, 20=4.85%, 50=0.10% lat (msec) : 250=0.03% cpu : usr=0.17%, sys=23.51%, ctx=6218, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5925: Tue Oct 1 05:39:49 2024 read: IOPS=108, BW=432KiB/s (442kB/s)(6028KiB/13950msec) clat (usec): min=50, max=27401, avg=5120.81, stdev=2913.41 lat (usec): min=50, max=27401, avg=5122.43, stdev=2913.95 clat percentiles (usec): | 1.00th=[ 62], 5.00th=[ 137], 10.00th=[ 2376], 20.00th=[ 2900], | 30.00th=[ 3425], 40.00th=[ 3982], 50.00th=[ 4686], 60.00th=[ 5407], | 70.00th=[ 6259], 80.00th=[ 7177], 90.00th=[ 8586], 95.00th=[10159], | 99.00th=[14091], 99.50th=[16581], 99.90th=[25560], 99.95th=[27395], | 99.99th=[27395] bw ( KiB/s): min= 192, max= 552, per=24.65%, avg=424.22, stdev=70.11, samples=27 iops : min= 48, max= 138, avg=106.00, stdev=17.53, samples=27 write: IOPS=112, BW=449KiB/s (460kB/s)(6260KiB/13950msec) clat (usec): min=432, max=190335, avg=3922.88, stdev=5794.40 lat (usec): min=433, max=190337, avg=3923.86, stdev=5794.45 clat percentiles (usec): | 1.00th=[ 523], 5.00th=[ 562], 10.00th=[ 603], 20.00th=[ 758], | 30.00th=[ 988], 40.00th=[ 2147], 50.00th=[ 3097], 60.00th=[ 3851], | 70.00th=[ 4883], 80.00th=[ 6325], 90.00th=[ 8455], 95.00th=[ 10552], | 99.00th=[ 13960], 99.50th=[ 15795], 99.90th=[ 28181], 99.95th=[189793], | 99.99th=[189793] bw ( KiB/s): min= 264, max= 584, per=24.35%, avg=438.74, stdev=75.08, samples=27 iops : min= 66, max= 146, avg=109.63, stdev=18.80, samples=27 lat (usec) : 100=1.82%, 250=0.91%, 500=0.26%, 750=9.96%, 1000=5.34% lat (msec) : 2=4.79%, 4=28.09%, 10=43.13%, 20=5.47%, 50=0.20% lat (msec) : 250=0.03% cpu : usr=0.26%, sys=23.59%, ctx=6170, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5926: Tue Oct 1 05:39:49 2024 read: IOPS=108, BW=435KiB/s (446kB/s)(6052KiB/13899msec) clat (usec): min=57, max=22382, avg=5103.13, stdev=2770.60 lat (usec): min=58, max=22383, avg=5105.66, stdev=2771.19 clat percentiles (usec): | 1.00th=[ 70], 5.00th=[ 2057], 10.00th=[ 2442], 20.00th=[ 2999], | 30.00th=[ 3490], 40.00th=[ 3982], 50.00th=[ 4621], 60.00th=[ 5342], | 70.00th=[ 6063], 80.00th=[ 6915], 90.00th=[ 8356], 95.00th=[10028], | 99.00th=[13698], 99.50th=[16712], 99.90th=[21890], 99.95th=[22414], | 99.99th=[22414] bw ( KiB/s): min= 224, max= 552, per=25.01%, avg=430.44, stdev=63.04, samples=27 iops : min= 56, max= 138, avg=107.56, stdev=15.80, samples=27 write: IOPS=112, BW=449KiB/s (459kB/s)(6236KiB/13899msec) clat (usec): min=459, max=174993, avg=3892.92, stdev=5408.39 lat (usec): min=460, max=174995, avg=3893.78, stdev=5408.43 clat percentiles (usec): | 1.00th=[ 529], 5.00th=[ 570], 10.00th=[ 627], 20.00th=[ 775], | 30.00th=[ 1012], 40.00th=[ 2311], 50.00th=[ 3130], 60.00th=[ 3884], | 70.00th=[ 4948], 80.00th=[ 6390], 90.00th=[ 8455], 95.00th=[ 10159], | 99.00th=[ 13304], 99.50th=[ 15008], 99.90th=[ 23725], 99.95th=[175113], | 99.99th=[175113] bw ( KiB/s): min= 272, max= 624, per=24.68%, avg=444.67, stdev=68.19, samples=27 iops : min= 68, max= 156, avg=111.11, stdev=17.08, samples=27 lat (usec) : 100=1.53%, 250=0.46%, 500=0.20%, 750=9.37%, 1000=5.60% lat (msec) : 2=4.65%, 4=29.26%, 10=43.72%, 20=5.05%, 50=0.13% lat (msec) : 250=0.03% cpu : usr=0.24%, sys=23.69%, ctx=6181, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1721KiB/s (1763kB/s), 427KiB/s-435KiB/s (437kB/s-446kB/s), io=23.4MiB (24.6MB), run=13899-13950msec WRITE: bw=1802KiB/s (1845kB/s), 449KiB/s-456KiB/s (459kB/s-467kB/s), io=24.6MiB (25.7MB), run=13899-13950msec rand-rw: (groupid=0, jobs=1): err= 0: pid=5927: Tue Oct 1 05:39:53 2024 read: IOPS=85, BW=343KiB/s (351kB/s)(5992KiB/17479msec) clat (usec): min=1574, max=173588, avg=4638.40, stdev=5538.49 lat (usec): min=1574, max=173590, avg=4639.18, stdev=5538.52 clat percentiles (usec): | 1.00th=[ 1795], 5.00th=[ 1926], 10.00th=[ 2024], 20.00th=[ 2180], | 30.00th=[ 2376], 40.00th=[ 2638], 50.00th=[ 3064], 60.00th=[ 3687], | 70.00th=[ 4621], 80.00th=[ 6390], 90.00th=[ 10028], 95.00th=[ 12387], | 99.00th=[ 16581], 99.50th=[ 17433], 99.90th=[ 23200], 99.95th=[173016], | 99.99th=[173016] bw ( KiB/s): min= 136, max= 768, per=24.46%, avg=335.82, stdev=170.54, samples=34 iops : min= 34, max= 192, avg=83.85, stdev=42.62, samples=34 write: IOPS=90, BW=360KiB/s (369kB/s)(6296KiB/17479msec) clat (usec): min=2264, max=29238, avg=6638.32, stdev=4222.52 lat (usec): min=2265, max=29239, avg=6639.25, stdev=4222.55 clat percentiles (usec): | 1.00th=[ 2573], 5.00th=[ 2900], 10.00th=[ 3097], 20.00th=[ 3392], | 30.00th=[ 3654], 40.00th=[ 4113], 50.00th=[ 4883], 60.00th=[ 6194], | 70.00th=[ 7898], 80.00th=[ 9765], 90.00th=[12518], 95.00th=[14746], | 99.00th=[22414], 99.50th=[25035], 99.90th=[28967], 99.95th=[29230], | 99.99th=[29230] bw ( KiB/s): min= 160, max= 824, per=24.42%, avg=351.12, stdev=172.42, samples=34 iops : min= 40, max= 206, avg=87.68, stdev=43.11, samples=34 lat (msec) : 2=4.23%, 4=45.90%, 10=35.38%, 20=13.61%, 50=0.85% lat (msec) : 250=0.03% cpu : usr=0.23%, sys=11.56%, ctx=3768, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5928: Tue Oct 1 05:39:53 2024 read: IOPS=85, BW=341KiB/s (349kB/s)(5940KiB/17416msec) clat (usec): min=1590, max=172252, avg=4727.31, stdev=5645.45 lat (usec): min=1590, max=172254, avg=4728.08, stdev=5645.47 clat percentiles (usec): | 1.00th=[ 1778], 5.00th=[ 1909], 10.00th=[ 2008], 20.00th=[ 2180], | 30.00th=[ 2409], 40.00th=[ 2671], 50.00th=[ 3032], 60.00th=[ 3687], | 70.00th=[ 4686], 80.00th=[ 6456], 90.00th=[ 10159], 95.00th=[ 12518], | 99.00th=[ 16712], 99.50th=[ 19006], 99.90th=[ 31589], 99.95th=[173016], | 99.99th=[173016] bw ( KiB/s): min= 104, max= 784, per=24.58%, avg=337.53, stdev=175.84, samples=34 iops : min= 26, max= 196, avg=84.29, stdev=43.99, samples=34 write: IOPS=91, BW=364KiB/s (373kB/s)(6348KiB/17416msec) clat (usec): min=2328, max=29505, avg=6490.10, stdev=3886.85 lat (usec): min=2329, max=29506, avg=6491.04, stdev=3886.88 clat percentiles (usec): | 1.00th=[ 2606], 5.00th=[ 2933], 10.00th=[ 3130], 20.00th=[ 3425], | 30.00th=[ 3720], 40.00th=[ 4113], 50.00th=[ 4752], 60.00th=[ 6063], | 70.00th=[ 7898], 80.00th=[ 9503], 90.00th=[12125], 95.00th=[14222], | 99.00th=[19268], 99.50th=[22676], 99.90th=[26608], 99.95th=[29492], | 99.99th=[29492] bw ( KiB/s): min= 168, max= 800, per=24.78%, avg=356.35, stdev=166.53, samples=34 iops : min= 42, max= 200, avg=89.00, stdev=41.68, samples=34 lat (msec) : 2=4.49%, 4=46.00%, 10=35.45%, 20=13.44%, 50=0.59% lat (msec) : 250=0.03% cpu : usr=0.21%, sys=11.36%, ctx=3687, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5929: Tue Oct 1 05:39:53 2024 read: IOPS=86, BW=346KiB/s (354kB/s)(6028KiB/17417msec) clat (usec): min=1619, max=23269, avg=4557.37, stdev=3572.83 lat (usec): min=1619, max=23270, avg=4558.12, stdev=3572.86 clat percentiles (usec): | 1.00th=[ 1778], 5.00th=[ 1926], 10.00th=[ 2057], 20.00th=[ 2212], | 30.00th=[ 2376], 40.00th=[ 2638], 50.00th=[ 2999], 60.00th=[ 3621], | 70.00th=[ 4621], 80.00th=[ 6456], 90.00th=[ 9765], 95.00th=[12387], | 99.00th=[18220], 99.50th=[21365], 99.90th=[22938], 99.95th=[23200], | 99.99th=[23200] bw ( KiB/s): min= 112, max= 784, per=24.81%, avg=340.59, stdev=173.29, samples=34 iops : min= 28, max= 196, avg=85.06, stdev=43.37, samples=34 write: IOPS=89, BW=359KiB/s (368kB/s)(6260KiB/17417msec) clat (msec): min=2, max=188, avg= 6.69, stdev= 6.15 lat (msec): min=2, max=188, avg= 6.69, stdev= 6.15 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 4], 20.00th=[ 4], | 30.00th=[ 4], 40.00th=[ 5], 50.00th=[ 5], 60.00th=[ 7], | 70.00th=[ 8], 80.00th=[ 10], 90.00th=[ 13], 95.00th=[ 15], | 99.00th=[ 22], 99.50th=[ 25], 99.90th=[ 31], 99.95th=[ 190], | 99.99th=[ 190] bw ( KiB/s): min= 144, max= 744, per=24.47%, avg=351.88, stdev=172.93, samples=34 iops : min= 36, max= 186, avg=87.88, stdev=43.27, samples=34 lat (msec) : 2=3.87%, 4=47.27%, 10=34.90%, 20=12.86%, 50=1.07% lat (msec) : 250=0.03% cpu : usr=0.20%, sys=11.52%, ctx=3744, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5930: Tue Oct 1 05:39:53 2024 read: IOPS=86, BW=347KiB/s (355kB/s)(6052KiB/17456msec) clat (usec): min=1649, max=23309, avg=4590.49, stdev=3525.04 lat (usec): min=1650, max=23310, avg=4591.24, stdev=3525.06 clat percentiles (usec): | 1.00th=[ 1811], 5.00th=[ 1942], 10.00th=[ 2040], 20.00th=[ 2212], | 30.00th=[ 2409], 40.00th=[ 2638], 50.00th=[ 2999], 60.00th=[ 3589], | 70.00th=[ 4686], 80.00th=[ 6456], 90.00th=[10028], 95.00th=[12518], | 99.00th=[16581], 99.50th=[19006], 99.90th=[22938], 99.95th=[23200], | 99.99th=[23200] bw ( KiB/s): min= 112, max= 768, per=24.63%, avg=338.24, stdev=170.04, samples=34 iops : min= 28, max= 192, avg=84.47, stdev=42.55, samples=34 write: IOPS=89, BW=357KiB/s (366kB/s)(6236KiB/17456msec) clat (msec): min=2, max=178, avg= 6.68, stdev= 6.11 lat (msec): min=2, max=178, avg= 6.68, stdev= 6.11 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 4], 20.00th=[ 4], | 30.00th=[ 4], 40.00th=[ 5], 50.00th=[ 5], 60.00th=[ 6], | 70.00th=[ 8], 80.00th=[ 10], 90.00th=[ 13], 95.00th=[ 15], | 99.00th=[ 24], 99.50th=[ 26], 99.90th=[ 37], 99.95th=[ 180], | 99.99th=[ 180] bw ( KiB/s): min= 144, max= 784, per=24.34%, avg=350.00, stdev=170.95, samples=34 iops : min= 36, max= 196, avg=87.41, stdev=42.77, samples=34 lat (msec) : 2=4.07%, 4=47.07%, 10=34.96%, 20=12.76%, 50=1.11% lat (msec) : 250=0.03% cpu : usr=0.19%, sys=11.40%, ctx=3718, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1374KiB/s (1407kB/s), 341KiB/s-347KiB/s (349kB/s-355kB/s), io=23.4MiB (24.6MB), run=17416-17479msec WRITE: bw=1438KiB/s (1473kB/s), 357KiB/s-364KiB/s (366kB/s-373kB/s), io=24.6MiB (25.7MB), run=17416-17479msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=5962: Tue Oct 1 05:40:00 2024 read: IOPS=54, BW=877KiB/s (898kB/s)(5920KiB/6752msec) clat (usec): min=61, max=27636, avg=12235.80, stdev=3855.27 lat (usec): min=62, max=27637, avg=12236.76, stdev=3855.29 clat percentiles (usec): | 1.00th=[ 70], 5.00th=[ 6849], 10.00th=[ 8717], 20.00th=[10028], | 30.00th=[10814], 40.00th=[11600], 50.00th=[12125], 60.00th=[13173], | 70.00th=[14091], 80.00th=[14877], 90.00th=[16319], 95.00th=[18220], | 99.00th=[20317], 99.50th=[21890], 99.90th=[27657], 99.95th=[27657], | 99.99th=[27657] bw ( KiB/s): min= 736, max= 1024, per=24.66%, avg=873.38, stdev=98.91, samples=13 iops : min= 46, max= 64, avg=54.38, stdev= 6.17, samples=13 write: IOPS=58, BW=943KiB/s (966kB/s)(6368KiB/6752msec) clat (usec): min=728, max=45129, avg=5521.89, stdev=5523.28 lat (usec): min=729, max=45130, avg=5523.46, stdev=5523.51 clat percentiles (usec): | 1.00th=[ 791], 5.00th=[ 898], 10.00th=[ 988], 20.00th=[ 1336], | 30.00th=[ 2474], 40.00th=[ 3064], 50.00th=[ 3621], 60.00th=[ 4621], | 70.00th=[ 6194], 80.00th=[ 8848], 90.00th=[12911], 95.00th=[15008], | 99.00th=[26346], 99.50th=[44827], 99.90th=[45351], 99.95th=[45351], | 99.99th=[45351] bw ( KiB/s): min= 704, max= 1184, per=25.55%, avg=952.08, stdev=144.62, samples=13 iops : min= 44, max= 74, avg=59.31, stdev= 9.17, samples=13 lat (usec) : 100=1.95%, 250=0.13%, 750=0.13%, 1000=5.47% lat (msec) : 2=8.72%, 4=13.54%, 10=23.44%, 20=45.18%, 50=1.43% cpu : usr=0.16%, sys=19.06%, ctx=2689, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5964: Tue Oct 1 05:40:00 2024 read: IOPS=55, BW=890KiB/s (911kB/s)(5744KiB/6453msec) clat (usec): min=77, max=30775, avg=12576.54, stdev=3757.59 lat (usec): min=78, max=30776, avg=12578.10, stdev=3758.15 clat percentiles (usec): | 1.00th=[ 101], 5.00th=[ 8356], 10.00th=[ 9110], 20.00th=[10028], | 30.00th=[10945], 40.00th=[11600], 50.00th=[12518], 60.00th=[13173], | 70.00th=[14091], 80.00th=[14877], 90.00th=[16581], 95.00th=[18744], | 99.00th=[22152], 99.50th=[29230], 99.90th=[30802], 99.95th=[30802], | 99.99th=[30802] bw ( KiB/s): min= 768, max= 1088, per=25.15%, avg=890.50, stdev=86.18, samples=12 iops : min= 48, max= 68, avg=55.58, stdev= 5.42, samples=12 write: IOPS=63, BW=1014KiB/s (1038kB/s)(6544KiB/6453msec) clat (usec): min=709, max=34987, avg=4695.49, stdev=4371.73 lat (usec): min=710, max=34988, avg=4697.08, stdev=4371.67 clat percentiles (usec): | 1.00th=[ 816], 5.00th=[ 889], 10.00th=[ 988], 20.00th=[ 1352], | 30.00th=[ 2245], 40.00th=[ 2966], 50.00th=[ 3326], 60.00th=[ 3916], | 70.00th=[ 5211], 80.00th=[ 6849], 90.00th=[10421], 95.00th=[15008], | 99.00th=[19792], 99.50th=[20055], 99.90th=[34866], 99.95th=[34866], | 99.99th=[34866] bw ( KiB/s): min= 798, max= 1312, per=26.76%, avg=997.17, stdev=180.62, samples=12 iops : min= 49, max= 82, avg=62.25, stdev=11.38, samples=12 lat (usec) : 100=0.39%, 250=0.65%, 750=0.13%, 1000=5.34% lat (msec) : 2=9.51%, 4=17.84%, 10=23.18%, 20=41.02%, 50=1.95% cpu : usr=0.17%, sys=20.01%, ctx=2589, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5966: Tue Oct 1 05:40:00 2024 read: IOPS=57, BW=925KiB/s (947kB/s)(6256KiB/6763msec) clat (usec): min=105, max=28615, avg=12510.61, stdev=3327.84 lat (usec): min=105, max=28616, avg=12511.43, stdev=3327.86 clat percentiles (usec): | 1.00th=[ 3064], 5.00th=[ 8225], 10.00th=[ 8848], 20.00th=[10028], | 30.00th=[10814], 40.00th=[11600], 50.00th=[12387], 60.00th=[13042], | 70.00th=[13829], 80.00th=[14746], 90.00th=[16450], 95.00th=[18220], | 99.00th=[22414], 99.50th=[23200], 99.90th=[28705], 99.95th=[28705], | 99.99th=[28705] bw ( KiB/s): min= 768, max= 1152, per=26.13%, avg=925.38, stdev=116.63, samples=13 iops : min= 48, max= 72, avg=57.77, stdev= 7.37, samples=13 write: IOPS=55, BW=892KiB/s (913kB/s)(6032KiB/6763msec) clat (usec): min=779, max=41207, avg=4844.83, stdev=5062.43 lat (usec): min=780, max=41208, avg=4848.06, stdev=5062.05 clat percentiles (usec): | 1.00th=[ 807], 5.00th=[ 889], 10.00th=[ 988], 20.00th=[ 1172], | 30.00th=[ 2343], 40.00th=[ 2769], 50.00th=[ 3130], 60.00th=[ 3720], | 70.00th=[ 4883], 80.00th=[ 6587], 90.00th=[11600], 95.00th=[14877], | 99.00th=[24511], 99.50th=[31065], 99.90th=[41157], 99.95th=[41157], | 99.99th=[41157] bw ( KiB/s): min= 608, max= 1152, per=23.32%, avg=868.77, stdev=169.16, samples=13 iops : min= 38, max= 72, avg=54.23, stdev=10.54, samples=13 lat (usec) : 250=0.39%, 1000=5.08% lat (msec) : 2=7.55%, 4=18.10%, 10=22.53%, 20=44.27%, 50=2.08% cpu : usr=0.13%, sys=19.46%, ctx=2795, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5968: Tue Oct 1 05:40:00 2024 read: IOPS=56, BW=908KiB/s (930kB/s)(6032KiB/6641msec) clat (usec): min=52, max=29232, avg=12376.68, stdev=4010.79 lat (usec): min=52, max=29233, avg=12377.60, stdev=4010.82 clat percentiles (usec): | 1.00th=[ 63], 5.00th=[ 7439], 10.00th=[ 9110], 20.00th=[10159], | 30.00th=[10945], 40.00th=[11731], 50.00th=[12387], 60.00th=[13042], | 70.00th=[13960], 80.00th=[14877], 90.00th=[16581], 95.00th=[17695], | 99.00th=[23462], 99.50th=[26346], 99.90th=[29230], 99.95th=[29230], | 99.99th=[29230] bw ( KiB/s): min= 768, max= 1024, per=25.44%, avg=900.92, stdev=78.22, samples=13 iops : min= 48, max= 64, avg=56.31, stdev= 4.89, samples=13 write: IOPS=58, BW=942KiB/s (965kB/s)(6256KiB/6641msec) clat (usec): min=703, max=31463, avg=4966.72, stdev=4740.27 lat (usec): min=705, max=31464, avg=4968.23, stdev=4740.14 clat percentiles (usec): | 1.00th=[ 783], 5.00th=[ 914], 10.00th=[ 1037], 20.00th=[ 1319], | 30.00th=[ 2409], 40.00th=[ 2900], 50.00th=[ 3458], 60.00th=[ 4047], | 70.00th=[ 5407], 80.00th=[ 6980], 90.00th=[11469], 95.00th=[15926], | 99.00th=[22938], 99.50th=[24773], 99.90th=[31589], 99.95th=[31589], | 99.99th=[31589] bw ( KiB/s): min= 608, max= 1344, per=25.24%, avg=940.31, stdev=183.61, samples=13 iops : min= 38, max= 84, avg=58.77, stdev=11.48, samples=13 lat (usec) : 100=1.56%, 250=0.52%, 750=0.13%, 1000=3.91% lat (msec) : 2=9.11%, 4=17.06%, 10=21.48%, 20=43.88%, 50=2.34% cpu : usr=0.12%, sys=19.86%, ctx=2723, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=3542KiB/s (3627kB/s), 877KiB/s-925KiB/s (898kB/s-947kB/s), io=23.4MiB (24.5MB), run=6453-6763msec WRITE: bw=3726KiB/s (3816kB/s), 892KiB/s-1014KiB/s (913kB/s-1038kB/s), io=24.6MiB (25.8MB), run=6453-6763msec rand-rw: (groupid=0, jobs=1): err= 0: pid=5961: Tue Oct 1 05:40:00 2024 read: IOPS=54, BW=875KiB/s (896kB/s)(5920KiB/6767msec) clat (usec): min=1869, max=30835, avg=6514.49, stdev=4522.21 lat (usec): min=1870, max=30836, avg=6515.31, stdev=4522.21 clat percentiles (usec): | 1.00th=[ 2114], 5.00th=[ 2409], 10.00th=[ 2540], 20.00th=[ 2835], | 30.00th=[ 3228], 40.00th=[ 3785], 50.00th=[ 4359], 60.00th=[ 5997], | 70.00th=[ 7898], 80.00th=[10421], 90.00th=[13435], 95.00th=[15401], | 99.00th=[20317], 99.50th=[20841], 99.90th=[30802], 99.95th=[30802], | 99.99th=[30802] bw ( KiB/s): min= 704, max= 1088, per=24.68%, avg=873.38, stdev=122.08, samples=13 iops : min= 44, max= 68, avg=54.38, stdev= 7.62, samples=13 write: IOPS=58, BW=941KiB/s (964kB/s)(6368KiB/6767msec) clat (usec): min=3060, max=34766, avg=10823.63, stdev=5813.06 lat (usec): min=3061, max=34767, avg=10825.18, stdev=5812.98 clat percentiles (usec): | 1.00th=[ 3326], 5.00th=[ 3916], 10.00th=[ 4424], 20.00th=[ 5669], | 30.00th=[ 6915], 40.00th=[ 8291], 50.00th=[ 9765], 60.00th=[11076], | 70.00th=[12911], 80.00th=[14877], 90.00th=[19792], 95.00th=[22676], | 99.00th=[27395], 99.50th=[32375], 99.90th=[34866], 99.95th=[34866], | 99.99th=[34866] bw ( KiB/s): min= 574, max= 1152, per=25.51%, avg=949.69, stdev=175.83, samples=13 iops : min= 35, max= 72, avg=59.15, stdev=11.16, samples=13 lat (msec) : 2=0.26%, 4=23.70%, 10=40.62%, 20=30.21%, 50=5.21% cpu : usr=0.16%, sys=10.35%, ctx=1170, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5963: Tue Oct 1 05:40:00 2024 read: IOPS=55, BW=888KiB/s (910kB/s)(5744KiB/6467msec) clat (usec): min=1932, max=29313, avg=6555.53, stdev=4936.22 lat (usec): min=1932, max=29313, avg=6556.35, stdev=4936.24 clat percentiles (usec): | 1.00th=[ 2089], 5.00th=[ 2311], 10.00th=[ 2540], 20.00th=[ 2868], | 30.00th=[ 3195], 40.00th=[ 3621], 50.00th=[ 4228], 60.00th=[ 5800], | 70.00th=[ 8356], 80.00th=[10290], 90.00th=[13566], 95.00th=[16909], | 99.00th=[25560], 99.50th=[27132], 99.90th=[29230], 99.95th=[29230], | 99.99th=[29230] bw ( KiB/s): min= 576, max= 1056, per=25.02%, avg=885.33, stdev=135.31, samples=12 iops : min= 36, max= 66, avg=55.33, stdev= 8.46, samples=12 write: IOPS=63, BW=1012KiB/s (1036kB/s)(6544KiB/6467msec) clat (usec): min=2981, max=44753, avg=9926.15, stdev=5446.27 lat (usec): min=2982, max=44754, avg=9927.64, stdev=5446.20 clat percentiles (usec): | 1.00th=[ 3097], 5.00th=[ 3916], 10.00th=[ 4293], 20.00th=[ 5276], | 30.00th=[ 6456], 40.00th=[ 7439], 50.00th=[ 9110], 60.00th=[10028], | 70.00th=[11731], 80.00th=[13304], 90.00th=[17433], 95.00th=[21365], | 99.00th=[26346], 99.50th=[27395], 99.90th=[44827], 99.95th=[44827], | 99.99th=[44827] bw ( KiB/s): min= 800, max= 1344, per=26.57%, avg=989.33, stdev=161.71, samples=12 iops : min= 50, max= 84, avg=61.83, stdev=10.11, samples=12 lat (msec) : 2=0.13%, 4=24.87%, 10=44.27%, 20=26.30%, 50=4.43% cpu : usr=0.15%, sys=10.55%, ctx=1142, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5965: Tue Oct 1 05:40:00 2024 read: IOPS=64, BW=1031KiB/s (1056kB/s)(6256KiB/6066msec) clat (usec): min=1813, max=35338, avg=6434.68, stdev=4605.03 lat (usec): min=1814, max=35338, avg=6435.49, stdev=4605.00 clat percentiles (usec): | 1.00th=[ 2180], 5.00th=[ 2409], 10.00th=[ 2638], 20.00th=[ 2966], | 30.00th=[ 3294], 40.00th=[ 3654], 50.00th=[ 4359], 60.00th=[ 5932], | 70.00th=[ 7767], 80.00th=[10159], 90.00th=[12649], 95.00th=[14484], | 99.00th=[22414], 99.50th=[29492], 99.90th=[35390], 99.95th=[35390], | 99.99th=[35390] bw ( KiB/s): min= 766, max= 1344, per=29.30%, avg=1036.92, stdev=211.11, samples=12 iops : min= 47, max= 84, avg=64.67, stdev=13.23, samples=12 write: IOPS=62, BW=994KiB/s (1018kB/s)(6032KiB/6066msec) clat (usec): min=2715, max=34058, avg=9384.15, stdev=5644.98 lat (usec): min=2716, max=34059, avg=9385.79, stdev=5645.04 clat percentiles (usec): | 1.00th=[ 3032], 5.00th=[ 3818], 10.00th=[ 4113], 20.00th=[ 4555], | 30.00th=[ 5276], 40.00th=[ 6259], 50.00th=[ 7570], 60.00th=[ 9372], | 70.00th=[10945], 80.00th=[14091], 90.00th=[17433], 95.00th=[20055], | 99.00th=[28181], 99.50th=[30278], 99.90th=[33817], 99.95th=[33817], | 99.99th=[33817] bw ( KiB/s): min= 832, max= 1152, per=26.42%, avg=983.58, stdev=88.62, samples=12 iops : min= 52, max= 72, avg=61.33, stdev= 5.60, samples=12 lat (msec) : 2=0.13%, 4=27.08%, 10=44.27%, 20=24.87%, 50=3.65% cpu : usr=0.20%, sys=9.63%, ctx=985, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5967: Tue Oct 1 05:40:00 2024 read: IOPS=56, BW=905KiB/s (926kB/s)(6032KiB/6668msec) clat (usec): min=2040, max=34058, avg=7103.96, stdev=4997.76 lat (usec): min=2041, max=34059, avg=7104.95, stdev=4997.98 clat percentiles (usec): | 1.00th=[ 2180], 5.00th=[ 2442], 10.00th=[ 2704], 20.00th=[ 3163], | 30.00th=[ 3556], 40.00th=[ 4047], 50.00th=[ 5014], 60.00th=[ 6915], | 70.00th=[ 8979], 80.00th=[10814], 90.00th=[13960], 95.00th=[16712], | 99.00th=[26608], 99.50th=[28181], 99.90th=[33817], 99.95th=[33817], | 99.99th=[33817] bw ( KiB/s): min= 640, max= 1120, per=25.10%, avg=888.46, stdev=114.13, samples=13 iops : min= 40, max= 70, avg=55.46, stdev= 7.20, samples=13 write: IOPS=58, BW=938KiB/s (961kB/s)(6256KiB/6668msec) clat (usec): min=3001, max=33296, avg=10095.11, stdev=5776.93 lat (usec): min=3003, max=33298, avg=10096.75, stdev=5776.91 clat percentiles (usec): | 1.00th=[ 3163], 5.00th=[ 3654], 10.00th=[ 4293], 20.00th=[ 5014], | 30.00th=[ 6325], 40.00th=[ 7373], 50.00th=[ 8586], 60.00th=[ 9896], | 70.00th=[11600], 80.00th=[14353], 90.00th=[18744], 95.00th=[21627], | 99.00th=[29492], 99.50th=[32900], 99.90th=[33424], 99.95th=[33424], | 99.99th=[33424] bw ( KiB/s): min= 736, max= 1216, per=25.19%, avg=937.69, stdev=126.39, samples=13 iops : min= 46, max= 76, avg=58.54, stdev= 7.96, samples=13 lat (msec) : 4=23.57%, 10=44.40%, 20=27.21%, 50=4.82% cpu : usr=0.13%, sys=10.23%, ctx=1209, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=3540KiB/s (3624kB/s), 875KiB/s-1031KiB/s (896kB/s-1056kB/s), io=23.4MiB (24.5MB), run=6066-6767msec WRITE: bw=3724KiB/s (3813kB/s), 938KiB/s-1012KiB/s (961kB/s-1036kB/s), io=24.6MiB (25.8MB), run=6066-6767msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=5982: Tue Oct 1 05:40:04 2024 read: IOPS=2, BW=3016KiB/s (3088kB/s)(7168KiB/2377msec) clat (usec): min=477, max=718838, avg=241382.20, stdev=292304.00 lat (usec): min=478, max=718840, avg=241383.51, stdev=292303.97 clat percentiles (usec): | 1.00th=[ 478], 5.00th=[ 478], 10.00th=[ 478], 20.00th=[ 494], | 30.00th=[ 5407], 40.00th=[ 5407], 50.00th=[ 72877], 60.00th=[425722], | 70.00th=[425722], 80.00th=[463471], 90.00th=[717226], 95.00th=[717226], | 99.00th=[717226], 99.50th=[717226], 99.90th=[717226], 99.95th=[717226], | 99.99th=[717226] bw ( KiB/s): min= 1812, max= 2048, per=22.41%, avg=1930.75, stdev=135.39, samples=4 iops : min= 1, max= 2, avg= 1.50, stdev= 0.58, samples=4 write: IOPS=2, BW=2154KiB/s (2206kB/s)(5120KiB/2377msec) clat (msec): min=27, max=316, avg=120.63, stdev=125.81 lat (msec): min=27, max=316, avg=120.69, stdev=125.82 clat percentiles (msec): | 1.00th=[ 28], 5.00th=[ 28], 10.00th=[ 28], 20.00th=[ 28], | 30.00th=[ 29], 40.00th=[ 29], 50.00th=[ 54], 60.00th=[ 54], | 70.00th=[ 178], 80.00th=[ 178], 90.00th=[ 317], 95.00th=[ 317], | 99.00th=[ 317], 99.50th=[ 317], 99.90th=[ 317], 99.95th=[ 317], | 99.99th=[ 317] bw ( KiB/s): min= 1815, max= 1815, per=15.05%, avg=1815.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (usec) : 500=16.67% lat (msec) : 10=8.33%, 50=16.67%, 100=16.67%, 250=8.33%, 500=25.00% lat (msec) : 750=8.33% cpu : usr=0.08%, sys=28.87%, ctx=776, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5984: Tue Oct 1 05:40:04 2024 read: IOPS=0, BW=1016KiB/s (1040kB/s)(2048KiB/2016msec) clat (msec): min=104, max=333, avg=219.14, stdev=162.17 lat (msec): min=104, max=333, avg=219.14, stdev=162.17 clat percentiles (msec): | 1.00th=[ 105], 5.00th=[ 105], 10.00th=[ 105], 20.00th=[ 105], | 30.00th=[ 105], 40.00th=[ 105], 50.00th=[ 105], 60.00th=[ 334], | 70.00th=[ 334], 80.00th=[ 334], 90.00th=[ 334], 95.00th=[ 334], | 99.00th=[ 334], 99.50th=[ 334], 99.90th=[ 334], 99.95th=[ 334], | 99.99th=[ 334] bw ( KiB/s): min= 2048, max= 2048, per=23.77%, avg=2048.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=4, BW=5079KiB/s (5201kB/s)(10.0MiB/2016msec) clat (msec): min=21, max=645, avg=149.31, stdev=227.72 lat (msec): min=21, max=645, avg=149.38, stdev=227.72 clat percentiles (msec): | 1.00th=[ 22], 5.00th=[ 22], 10.00th=[ 22], 20.00th=[ 22], | 30.00th=[ 23], 40.00th=[ 23], 50.00th=[ 26], 60.00th=[ 44], | 70.00th=[ 103], 80.00th=[ 103], 90.00th=[ 502], 95.00th=[ 642], | 99.00th=[ 642], 99.50th=[ 642], 99.90th=[ 642], 99.95th=[ 642], | 99.99th=[ 642] bw ( KiB/s): min= 1809, max= 6144, per=32.97%, avg=3976.50, stdev=3065.31, samples=2 iops : min= 1, max= 6, avg= 3.50, stdev= 3.54, samples=2 lat (msec) : 50=50.00%, 100=8.33%, 250=16.67%, 500=8.33%, 750=16.67% cpu : usr=0.00%, sys=22.78%, ctx=163, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5985: Tue Oct 1 05:40:04 2024 read: IOPS=3, BW=3084KiB/s (3158kB/s)(6144KiB/1992msec) clat (usec): min=519, max=493817, avg=198355.52, stdev=193385.64 lat (usec): min=521, max=493819, avg=198356.93, stdev=193385.72 clat percentiles (usec): | 1.00th=[ 519], 5.00th=[ 519], 10.00th=[ 519], 20.00th=[ 47973], | 30.00th=[ 47973], 40.00th=[132645], 50.00th=[132645], 60.00th=[141558], | 70.00th=[371196], 80.00th=[371196], 90.00th=[492831], 95.00th=[492831], | 99.00th=[492831], 99.50th=[492831], 99.90th=[492831], 99.95th=[492831], | 99.99th=[492831] bw ( KiB/s): min= 1809, max= 6144, per=46.16%, avg=3976.50, stdev=3065.31, samples=2 iops : min= 1, max= 6, avg= 3.50, stdev= 3.54, samples=2 write: IOPS=3, BW=3084KiB/s (3158kB/s)(6144KiB/1992msec) clat (msec): min=33, max=533, avg=119.38, stdev=202.96 lat (msec): min=33, max=533, avg=119.44, stdev=202.97 clat percentiles (msec): | 1.00th=[ 34], 5.00th=[ 34], 10.00th=[ 34], 20.00th=[ 35], | 30.00th=[ 35], 40.00th=[ 36], 50.00th=[ 36], 60.00th=[ 38], | 70.00th=[ 42], 80.00th=[ 42], 90.00th=[ 535], 95.00th=[ 535], | 99.00th=[ 535], 99.50th=[ 535], 99.90th=[ 535], 99.95th=[ 535], | 99.99th=[ 535] bw ( KiB/s): min= 1809, max= 4096, per=27.64%, avg=3333.67, stdev=1320.40, samples=3 iops : min= 1, max= 4, avg= 3.00, stdev= 1.73, samples=3 lat (usec) : 750=8.33% lat (msec) : 50=50.00%, 250=16.67%, 500=16.67%, 750=8.33% cpu : usr=0.00%, sys=39.83%, ctx=585, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5986: Tue Oct 1 05:40:04 2024 read: IOPS=3, BW=3810KiB/s (3901kB/s)(5120KiB/1344msec) clat (usec): min=306, max=322175, avg=98409.09, stdev=144726.29 lat (usec): min=306, max=322175, avg=98409.69, stdev=144726.29 clat percentiles (usec): | 1.00th=[ 306], 5.00th=[ 306], 10.00th=[ 306], 20.00th=[ 306], | 30.00th=[ 490], 40.00th=[ 490], 50.00th=[ 529], 60.00th=[ 529], | 70.00th=[168821], 80.00th=[168821], 90.00th=[320865], 95.00th=[320865], | 99.00th=[320865], 99.50th=[320865], 99.90th=[320865], 99.95th=[320865], | 99.99th=[320865] bw ( KiB/s): min= 8192, max= 8192, per=95.09%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=5, BW=5333KiB/s (5461kB/s)(7168KiB/1344msec) clat (msec): min=26, max=566, avg=119.74, stdev=197.93 lat (msec): min=26, max=566, avg=119.89, stdev=197.90 clat percentiles (msec): | 1.00th=[ 27], 5.00th=[ 27], 10.00th=[ 27], 20.00th=[ 36], | 30.00th=[ 39], 40.00th=[ 39], 50.00th=[ 45], 60.00th=[ 46], | 70.00th=[ 46], 80.00th=[ 82], 90.00th=[ 567], 95.00th=[ 567], | 99.00th=[ 567], 99.50th=[ 567], 99.90th=[ 567], 99.95th=[ 567], | 99.99th=[ 567] bw ( KiB/s): min= 6144, max= 6144, per=50.94%, avg=6144.00, stdev= 0.00, samples=2 iops : min= 6, max= 6, avg= 6.00, stdev= 0.00, samples=2 lat (usec) : 500=16.67%, 750=8.33% lat (msec) : 50=41.67%, 100=8.33%, 250=8.33%, 500=8.33%, 750=8.33% cpu : usr=0.00%, sys=21.74%, ctx=293, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=8616KiB/s (8823kB/s), 1016KiB/s-3810KiB/s (1040kB/s-3901kB/s), io=20.0MiB (20.0MB), run=1344-2377msec WRITE: bw=11.8MiB/s (12.4MB/s), 2154KiB/s-5333KiB/s (2206kB/s-5461kB/s), io=28.0MiB (29.4MB), run=1344-2377msec rand-rw: (groupid=0, jobs=1): err= 0: pid=5979: Tue Oct 1 05:40:04 2024 read: IOPS=4, BW=4833KiB/s (4949kB/s)(7168KiB/1483msec) clat (msec): min=15, max=458, avg=151.20, stdev=209.39 lat (msec): min=15, max=458, avg=151.21, stdev=209.39 clat percentiles (msec): | 1.00th=[ 16], 5.00th=[ 16], 10.00th=[ 16], 20.00th=[ 17], | 30.00th=[ 18], 40.00th=[ 18], 50.00th=[ 27], 60.00th=[ 71], | 70.00th=[ 71], 80.00th=[ 456], 90.00th=[ 460], 95.00th=[ 460], | 99.00th=[ 460], 99.50th=[ 460], 99.90th=[ 460], 99.95th=[ 460], | 99.99th=[ 460] bw ( KiB/s): min= 4096, max= 6144, per=64.26%, avg=5120.00, stdev=1448.15, samples=2 iops : min= 4, max= 6, avg= 5.00, stdev= 1.41, samples=2 write: IOPS=3, BW=3452KiB/s (3535kB/s)(5120KiB/1483msec) clat (msec): min=16, max=158, avg=82.63, stdev=51.20 lat (msec): min=16, max=158, avg=82.70, stdev=51.20 clat percentiles (msec): | 1.00th=[ 17], 5.00th=[ 17], 10.00th=[ 17], 20.00th=[ 17], | 30.00th=[ 64], 40.00th=[ 64], 50.00th=[ 86], 60.00th=[ 86], | 70.00th=[ 91], 80.00th=[ 91], 90.00th=[ 159], 95.00th=[ 159], | 99.00th=[ 159], 99.50th=[ 159], 99.90th=[ 159], 99.95th=[ 159], | 99.99th=[ 159] bw ( KiB/s): min= 2048, max= 6144, per=36.72%, avg=4096.00, stdev=2896.31, samples=2 iops : min= 2, max= 6, avg= 4.00, stdev= 2.83, samples=2 lat (msec) : 20=33.33%, 50=8.33%, 100=33.33%, 250=8.33%, 500=16.67% cpu : usr=0.07%, sys=7.76%, ctx=79, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5980: Tue Oct 1 05:40:04 2024 read: IOPS=1, BW=1398KiB/s (1432kB/s)(2048KiB/1465msec) clat (usec): min=15200, max=33034, avg=24117.59, stdev=12610.25 lat (usec): min=15202, max=33035, avg=24119.03, stdev=12609.90 clat percentiles (usec): | 1.00th=[15139], 5.00th=[15139], 10.00th=[15139], 20.00th=[15139], | 30.00th=[15139], 40.00th=[15139], 50.00th=[15139], 60.00th=[33162], | 70.00th=[33162], 80.00th=[33162], 90.00th=[33162], 95.00th=[33162], | 99.00th=[33162], 99.50th=[33162], 99.90th=[33162], 99.95th=[33162], | 99.99th=[33162] bw ( KiB/s): min= 2048, max= 2048, per=25.70%, avg=2048.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 write: IOPS=6, BW=6990KiB/s (7158kB/s)(10.0MiB/1465msec) clat (msec): min=19, max=570, avg=139.93, stdev=173.77 lat (msec): min=19, max=570, avg=140.00, stdev=173.78 clat percentiles (msec): | 1.00th=[ 20], 5.00th=[ 20], 10.00th=[ 20], 20.00th=[ 22], | 30.00th=[ 47], 40.00th=[ 52], 50.00th=[ 58], 60.00th=[ 74], | 70.00th=[ 190], 80.00th=[ 190], 90.00th=[ 296], 95.00th=[ 567], | 99.00th=[ 567], 99.50th=[ 567], 99.90th=[ 567], 99.95th=[ 567], | 99.99th=[ 567] bw ( KiB/s): min= 4096, max= 6144, per=45.89%, avg=5120.00, stdev=1448.15, samples=2 iops : min= 4, max= 6, avg= 5.00, stdev= 1.41, samples=2 lat (msec) : 20=16.67%, 50=25.00%, 100=33.33%, 250=8.33%, 500=8.33% lat (msec) : 750=8.33% cpu : usr=0.00%, sys=10.04%, ctx=125, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5981: Tue Oct 1 05:40:04 2024 read: IOPS=2, BW=2391KiB/s (2448kB/s)(6144KiB/2570msec) clat (msec): min=15, max=762, avg=252.49, stdev=337.19 lat (msec): min=15, max=762, avg=252.49, stdev=337.19 clat percentiles (msec): | 1.00th=[ 16], 5.00th=[ 16], 10.00th=[ 16], 20.00th=[ 23], | 30.00th=[ 23], 40.00th=[ 42], 50.00th=[ 42], 60.00th=[ 72], | 70.00th=[ 600], 80.00th=[ 600], 90.00th=[ 760], 95.00th=[ 760], | 99.00th=[ 760], 99.50th=[ 760], 99.90th=[ 760], 99.95th=[ 760], | 99.99th=[ 760] bw ( KiB/s): min= 1812, max= 3618, per=29.87%, avg=2380.25, stdev=832.48, samples=4 iops : min= 1, max= 3, avg= 1.75, stdev= 0.96, samples=4 write: IOPS=2, BW=2391KiB/s (2448kB/s)(6144KiB/2570msec) clat (msec): min=19, max=594, avg=166.14, stdev=221.32 lat (msec): min=19, max=594, avg=166.22, stdev=221.32 clat percentiles (msec): | 1.00th=[ 20], 5.00th=[ 20], 10.00th=[ 20], 20.00th=[ 36], | 30.00th=[ 36], 40.00th=[ 38], 50.00th=[ 38], 60.00th=[ 101], | 70.00th=[ 209], 80.00th=[ 209], 90.00th=[ 592], 95.00th=[ 592], | 99.00th=[ 592], 99.50th=[ 592], 99.90th=[ 592], 99.95th=[ 592], | 99.99th=[ 592] bw ( KiB/s): min= 2043, max= 6144, per=35.29%, avg=3937.00, stdev=2068.34, samples=3 iops : min= 1, max= 6, avg= 3.33, stdev= 2.52, samples=3 lat (msec) : 20=16.67%, 50=33.33%, 100=8.33%, 250=16.67%, 750=16.67% lat (msec) : 1000=8.33% cpu : usr=0.08%, sys=5.64%, ctx=192, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5983: Tue Oct 1 05:40:04 2024 read: IOPS=3, BW=3798KiB/s (3889kB/s)(5120KiB/1348msec) clat (msec): min=13, max=274, avg=90.85, stdev=106.81 lat (msec): min=13, max=274, avg=90.86, stdev=106.81 clat percentiles (msec): | 1.00th=[ 14], 5.00th=[ 14], 10.00th=[ 14], 20.00th=[ 14], | 30.00th=[ 18], 40.00th=[ 18], 50.00th=[ 70], 60.00th=[ 70], | 70.00th=[ 81], 80.00th=[ 81], 90.00th=[ 275], 95.00th=[ 275], | 99.00th=[ 275], 99.50th=[ 275], 99.90th=[ 275], 99.95th=[ 275], | 99.99th=[ 275] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=5, BW=5318KiB/s (5445kB/s)(7168KiB/1348msec) clat (msec): min=18, max=535, avg=121.76, stdev=184.35 lat (msec): min=18, max=535, avg=121.83, stdev=184.36 clat percentiles (msec): | 1.00th=[ 19], 5.00th=[ 19], 10.00th=[ 19], 20.00th=[ 29], | 30.00th=[ 46], 40.00th=[ 46], 50.00th=[ 52], 60.00th=[ 80], | 70.00th=[ 80], 80.00th=[ 93], 90.00th=[ 535], 95.00th=[ 535], | 99.00th=[ 535], 99.50th=[ 535], 99.90th=[ 535], 99.95th=[ 535], | 99.99th=[ 535] bw ( KiB/s): min= 2048, max= 6144, per=36.72%, avg=4096.00, stdev=2896.31, samples=2 iops : min= 2, max= 6, avg= 4.00, stdev= 2.83, samples=2 lat (msec) : 20=25.00%, 50=16.67%, 100=41.67%, 500=8.33%, 750=8.33% cpu : usr=0.00%, sys=6.16%, ctx=52, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=7969KiB/s (8160kB/s), 1398KiB/s-4833KiB/s (1432kB/s-4949kB/s), io=20.0MiB (20.0MB), run=1348-2570msec WRITE: bw=10.9MiB/s (11.4MB/s), 2391KiB/s-6990KiB/s (2448kB/s-7158kB/s), io=28.0MiB (29.4MB), run=1348-2570msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=5994: Tue Oct 1 05:40:08 2024 read: IOPS=0, BW=3848KiB/s (3941kB/s)(12.0MiB/3193msec) clat (msec): min=974, max=1113, avg=1055.73, stdev=72.30 lat (msec): min=974, max=1113, avg=1055.74, stdev=72.30 clat percentiles (msec): | 1.00th=[ 978], 5.00th=[ 978], 10.00th=[ 978], 20.00th=[ 978], | 30.00th=[ 978], 40.00th=[ 1083], 50.00th=[ 1083], 60.00th=[ 1083], | 70.00th=[ 1116], 80.00th=[ 1116], 90.00th=[ 1116], 95.00th=[ 1116], | 99.00th=[ 1116], 99.50th=[ 1116], 99.90th=[ 1116], 99.95th=[ 1116], | 99.99th=[ 1116] bw ( KiB/s): min= 7684, max= 7684, per=100.00%, avg=7684.00, stdev= 0.00, samples=2 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=2 lat (msec) : 1000=33.33% cpu : usr=0.00%, sys=2.44%, ctx=25, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5995: Tue Oct 1 05:40:08 2024 read: IOPS=0, BW=1895KiB/s (1940kB/s)(4096KiB/2162msec) clat (nsec): min=56661k, max=56661k, avg=56661390.00, stdev= 0.00 lat (nsec): min=56663k, max=56663k, avg=56663230.00, stdev= 0.00 clat percentiles (usec): | 1.00th=[56886], 5.00th=[56886], 10.00th=[56886], 20.00th=[56886], | 30.00th=[56886], 40.00th=[56886], 50.00th=[56886], 60.00th=[56886], | 70.00th=[56886], 80.00th=[56886], 90.00th=[56886], 95.00th=[56886], | 99.00th=[56886], 99.50th=[56886], 99.90th=[56886], 99.95th=[56886], | 99.99th=[56886] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=0, BW=3789KiB/s (3880kB/s)(8192KiB/2162msec) clat (msec): min=160, max=1932, avg=1046.54, stdev=1253.23 lat (msec): min=160, max=1933, avg=1046.82, stdev=1253.27 clat percentiles (msec): | 1.00th=[ 161], 5.00th=[ 161], 10.00th=[ 161], 20.00th=[ 161], | 30.00th=[ 161], 40.00th=[ 161], 50.00th=[ 161], 60.00th=[ 1938], | 70.00th=[ 1938], 80.00th=[ 1938], 90.00th=[ 1938], 95.00th=[ 1938], | 99.00th=[ 1938], 99.50th=[ 1938], 99.90th=[ 1938], 99.95th=[ 1938], | 99.99th=[ 1938] bw ( KiB/s): min= 8192, max= 8192, per=79.46%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 250=33.33% cpu : usr=0.05%, sys=4.58%, ctx=45, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5996: Tue Oct 1 05:40:08 2024 write: IOPS=0, BW=3867KiB/s (3959kB/s)(12.0MiB/3178msec) clat (msec): min=219, max=1966, avg=1050.11, stdev=876.79 lat (msec): min=219, max=1967, avg=1050.43, stdev=876.78 clat percentiles (msec): | 1.00th=[ 220], 5.00th=[ 220], 10.00th=[ 220], 20.00th=[ 220], | 30.00th=[ 220], 40.00th=[ 961], 50.00th=[ 961], 60.00th=[ 961], | 70.00th=[ 1972], 80.00th=[ 1972], 90.00th=[ 1972], 95.00th=[ 1972], | 99.00th=[ 1972], 99.50th=[ 1972], 99.90th=[ 1972], 99.95th=[ 1972], | 99.99th=[ 1972] bw ( KiB/s): min= 7314, max= 8175, per=75.12%, avg=7744.50, stdev=608.82, samples=2 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=2 lat (msec) : 250=33.33%, 1000=33.33% cpu : usr=0.09%, sys=4.12%, ctx=71, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5997: Tue Oct 1 05:40:08 2024 write: IOPS=2, BW=8885KiB/s (9098kB/s)(12.0MiB/1383msec) clat (msec): min=46, max=1159, avg=457.51, stdev=610.54 lat (msec): min=46, max=1159, avg=457.89, stdev=610.47 clat percentiles (msec): | 1.00th=[ 47], 5.00th=[ 47], 10.00th=[ 47], 20.00th=[ 47], | 30.00th=[ 47], 40.00th=[ 167], 50.00th=[ 167], 60.00th=[ 167], | 70.00th=[ 1167], 80.00th=[ 1167], 90.00th=[ 1167], 95.00th=[ 1167], | 99.00th=[ 1167], 99.50th=[ 1167], 99.90th=[ 1167], 99.95th=[ 1167], | 99.99th=[ 1167] bw ( KiB/s): min=16384, max=16384, per=100.00%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 50=33.33%, 250=33.33% cpu : usr=0.00%, sys=7.53%, ctx=54, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=5131KiB/s (5254kB/s), 1895KiB/s-3848KiB/s (1940kB/s-3941kB/s), io=16.0MiB (16.8MB), run=2162-3193msec WRITE: bw=10.1MiB/s (10.6MB/s), 3789KiB/s-8885KiB/s (3880kB/s-9098kB/s), io=32.0MiB (33.6MB), run=1383-3178msec rand-rw: (groupid=0, jobs=1): err= 0: pid=5998: Tue Oct 1 05:40:08 2024 read: IOPS=0, BW=3977KiB/s (4072kB/s)(12.0MiB/3090msec) clat (msec): min=152, max=1908, avg=1026.05, stdev=878.04 lat (msec): min=152, max=1908, avg=1026.06, stdev=878.04 clat percentiles (msec): | 1.00th=[ 153], 5.00th=[ 153], 10.00th=[ 153], 20.00th=[ 153], | 30.00th=[ 153], 40.00th=[ 1020], 50.00th=[ 1020], 60.00th=[ 1020], | 70.00th=[ 1905], 80.00th=[ 1905], 90.00th=[ 1905], 95.00th=[ 1905], | 99.00th=[ 1905], 99.50th=[ 1905], 99.90th=[ 1905], 99.95th=[ 1905], | 99.99th=[ 1905] bw ( KiB/s): min= 8175, max= 8192, per=100.00%, avg=8183.50, stdev=12.02, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 250=33.33% cpu : usr=0.06%, sys=38.22%, ctx=1637, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5999: Tue Oct 1 05:40:08 2024 read: IOPS=0, BW=1646KiB/s (1685kB/s)(4096KiB/2489msec) clat (nsec): min=131560k, max=131560k, avg=131560038.00, stdev= 0.00 lat (nsec): min=131561k, max=131561k, avg=131561053.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 132], 5.00th=[ 132], 10.00th=[ 132], 20.00th=[ 132], | 30.00th=[ 132], 40.00th=[ 132], 50.00th=[ 132], 60.00th=[ 132], | 70.00th=[ 132], 80.00th=[ 132], 90.00th=[ 132], 95.00th=[ 132], | 99.00th=[ 132], 99.50th=[ 132], 99.90th=[ 132], 99.95th=[ 132], | 99.99th=[ 132] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=0, BW=3291KiB/s (3370kB/s)(8192KiB/2489msec) clat (msec): min=1141, max=1204, avg=1173.38, stdev=44.38 lat (msec): min=1142, max=1205, avg=1173.63, stdev=44.38 clat percentiles (msec): | 1.00th=[ 1150], 5.00th=[ 1150], 10.00th=[ 1150], 20.00th=[ 1150], | 30.00th=[ 1150], 40.00th=[ 1150], 50.00th=[ 1150], 60.00th=[ 1200], | 70.00th=[ 1200], 80.00th=[ 1200], 90.00th=[ 1200], 95.00th=[ 1200], | 99.00th=[ 1200], 99.50th=[ 1200], 99.90th=[ 1200], 99.95th=[ 1200], | 99.99th=[ 1200] bw ( KiB/s): min= 8192, max= 8192, per=81.81%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33% cpu : usr=0.00%, sys=10.17%, ctx=127, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6000: Tue Oct 1 05:40:08 2024 write: IOPS=0, BW=3756KiB/s (3846kB/s)(12.0MiB/3272msec) clat (msec): min=861, max=1195, avg=1083.70, stdev=192.55 lat (msec): min=861, max=1196, avg=1084.26, stdev=192.80 clat percentiles (msec): | 1.00th=[ 860], 5.00th=[ 860], 10.00th=[ 860], 20.00th=[ 860], | 30.00th=[ 860], 40.00th=[ 1200], 50.00th=[ 1200], 60.00th=[ 1200], | 70.00th=[ 1200], 80.00th=[ 1200], 90.00th=[ 1200], 95.00th=[ 1200], | 99.00th=[ 1200], 99.50th=[ 1200], 99.90th=[ 1200], 99.95th=[ 1200], | 99.99th=[ 1200] bw ( KiB/s): min= 7742, max= 8192, per=79.56%, avg=7967.00, stdev=318.20, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 1000=33.33% cpu : usr=0.09%, sys=9.51%, ctx=53, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6001: Tue Oct 1 05:40:08 2024 write: IOPS=1, BW=4985KiB/s (5105kB/s)(12.0MiB/2465msec) clat (msec): min=161, max=1152, avg=810.06, stdev=562.26 lat (msec): min=161, max=1153, avg=810.34, stdev=562.23 clat percentiles (msec): | 1.00th=[ 161], 5.00th=[ 161], 10.00th=[ 161], 20.00th=[ 161], | 30.00th=[ 161], 40.00th=[ 1116], 50.00th=[ 1116], 60.00th=[ 1116], | 70.00th=[ 1150], 80.00th=[ 1150], 90.00th=[ 1150], 95.00th=[ 1150], | 99.00th=[ 1150], 99.50th=[ 1150], 99.90th=[ 1150], 99.95th=[ 1150], | 99.99th=[ 1150] bw ( KiB/s): min= 8192, max= 8192, per=81.81%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33% cpu : usr=0.00%, sys=14.08%, ctx=40, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=5302KiB/s (5430kB/s), 1646KiB/s-3977KiB/s (1685kB/s-4072kB/s), io=16.0MiB (16.8MB), run=2489-3090msec WRITE: bw=9.78MiB/s (10.3MB/s), 3291KiB/s-4985KiB/s (3370kB/s-5105kB/s), io=32.0MiB (33.6MB), run=2465-3272msec