== sanity test 398b: DIO and buffer IO race ============== 01:08:49 (1718860129) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.18737 s, 42.4 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=8562: Thu Jun 20 01:09:05 2024 read: IOPS=117, BW=469KiB/s (480kB/s)(5992KiB/12779msec) clat (usec): min=49, max=174521, avg=4727.94, stdev=5080.44 lat (usec): min=49, max=174522, avg=4728.78, stdev=5080.52 clat percentiles (usec): | 1.00th=[ 60], 5.00th=[ 1844], 10.00th=[ 2311], 20.00th=[ 2704], | 30.00th=[ 3097], 40.00th=[ 3589], 50.00th=[ 4146], 60.00th=[ 4752], | 70.00th=[ 5407], 80.00th=[ 6325], 90.00th=[ 7767], 95.00th=[ 9110], | 99.00th=[ 13435], 99.50th=[ 15533], 99.90th=[ 21627], 99.95th=[175113], | 99.99th=[175113] bw ( KiB/s): min= 255, max= 608, per=24.88%, avg=465.08, stdev=77.70, samples=25 iops : min= 63, max= 152, avg=116.20, stdev=19.47, samples=25 write: IOPS=123, BW=493KiB/s (505kB/s)(6296KiB/12779msec) clat (usec): min=410, max=22304, avg=3574.26, stdev=3206.62 lat (usec): min=410, max=22304, avg=3575.46, stdev=3207.33 clat percentiles (usec): | 1.00th=[ 498], 5.00th=[ 537], 10.00th=[ 578], 20.00th=[ 668], | 30.00th=[ 840], 40.00th=[ 1926], 50.00th=[ 2802], 60.00th=[ 3556], | 70.00th=[ 4621], 80.00th=[ 6128], 90.00th=[ 8225], 95.00th=[10159], | 99.00th=[13304], 99.50th=[14484], 99.90th=[15926], 99.95th=[22414], | 99.99th=[22414] bw ( KiB/s): min= 303, max= 656, per=24.99%, avg=488.80, stdev=79.36, samples=25 iops : min= 75, max= 164, avg=122.12, stdev=19.90, samples=25 lat (usec) : 50=0.03%, 100=1.92%, 250=0.29%, 500=0.75%, 750=12.24% lat (usec) : 1000=4.65% lat (msec) : 2=3.35%, 4=33.27%, 10=39.13%, 20=4.26%, 50=0.07% lat (msec) : 250=0.03% cpu : usr=0.24%, sys=22.54%, ctx=5805, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8563: Thu Jun 20 01:09:05 2024 read: IOPS=118, BW=473KiB/s (484kB/s)(5940KiB/12570msec) clat (usec): min=52, max=20553, avg=4638.33, stdev=2567.08 lat (usec): min=52, max=20553, avg=4639.20, stdev=2567.27 clat percentiles (usec): | 1.00th=[ 62], 5.00th=[ 2008], 10.00th=[ 2311], 20.00th=[ 2704], | 30.00th=[ 3064], 40.00th=[ 3490], 50.00th=[ 4113], 60.00th=[ 4752], | 70.00th=[ 5538], 80.00th=[ 6390], 90.00th=[ 7767], 95.00th=[ 9110], | 99.00th=[13698], 99.50th=[15139], 99.90th=[19792], 99.95th=[20579], | 99.99th=[20579] bw ( KiB/s): min= 256, max= 592, per=25.21%, avg=471.12, stdev=70.58, samples=25 iops : min= 64, max= 148, avg=117.68, stdev=17.63, samples=25 write: IOPS=126, BW=505KiB/s (517kB/s)(6348KiB/12570msec) clat (usec): min=403, max=170024, avg=3533.58, stdev=5196.63 lat (usec): min=403, max=170025, avg=3535.17, stdev=5196.81 clat percentiles (usec): | 1.00th=[ 506], 5.00th=[ 545], 10.00th=[ 586], 20.00th=[ 676], | 30.00th=[ 865], 40.00th=[ 1778], 50.00th=[ 2671], 60.00th=[ 3359], | 70.00th=[ 4293], 80.00th=[ 6063], 90.00th=[ 7898], 95.00th=[ 9503], | 99.00th=[ 13566], 99.50th=[ 14615], 99.90th=[ 20055], 99.95th=[170918], | 99.99th=[170918] bw ( KiB/s): min= 336, max= 648, per=25.67%, avg=502.12, stdev=79.53, samples=25 iops : min= 84, max= 162, avg=125.44, stdev=19.87, samples=25 lat (usec) : 100=1.53%, 250=0.29%, 500=0.36%, 750=12.24%, 1000=5.21% lat (msec) : 2=4.07%, 4=34.80%, 10=37.99%, 20=3.42%, 50=0.07% lat (msec) : 250=0.03% cpu : usr=0.23%, sys=22.97%, ctx=5546, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8564: Thu Jun 20 01:09:05 2024 read: IOPS=117, BW=469KiB/s (480kB/s)(6028KiB/12847msec) clat (usec): min=54, max=18953, avg=4696.62, stdev=2577.98 lat (usec): min=55, max=18953, avg=4697.76, stdev=2578.13 clat percentiles (usec): | 1.00th=[ 59], 5.00th=[ 1745], 10.00th=[ 2278], 20.00th=[ 2704], | 30.00th=[ 3130], 40.00th=[ 3621], 50.00th=[ 4293], 60.00th=[ 4948], | 70.00th=[ 5604], 80.00th=[ 6390], 90.00th=[ 7898], 95.00th=[ 9241], | 99.00th=[13698], 99.50th=[14746], 99.90th=[17957], 99.95th=[19006], | 99.99th=[19006] bw ( KiB/s): min= 176, max= 616, per=24.75%, avg=462.60, stdev=87.59, samples=25 iops : min= 44, max= 154, avg=115.60, stdev=21.90, samples=25 write: IOPS=121, BW=487KiB/s (499kB/s)(6260KiB/12847msec) clat (usec): min=468, max=172103, avg=3641.56, stdev=5364.60 lat (usec): min=469, max=172105, avg=3642.70, stdev=5364.43 clat percentiles (usec): | 1.00th=[ 498], 5.00th=[ 537], 10.00th=[ 562], 20.00th=[ 652], | 30.00th=[ 832], 40.00th=[ 1614], 50.00th=[ 2769], 60.00th=[ 3458], | 70.00th=[ 4555], 80.00th=[ 5997], 90.00th=[ 8160], 95.00th=[ 10028], | 99.00th=[ 14222], 99.50th=[ 15664], 99.90th=[ 19792], 99.95th=[173016], | 99.99th=[173016] bw ( KiB/s): min= 240, max= 648, per=24.40%, avg=477.32, stdev=88.85, samples=25 iops : min= 60, max= 162, avg=119.28, stdev=22.21, samples=25 lat (usec) : 100=1.86%, 250=0.46%, 500=0.72%, 750=12.66%, 1000=4.56% lat (msec) : 2=3.61%, 4=32.06%, 10=39.94%, 20=4.10%, 250=0.03% cpu : usr=0.13%, sys=22.78%, ctx=5876, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8565: Thu Jun 20 01:09:05 2024 read: IOPS=118, BW=475KiB/s (486kB/s)(6052KiB/12751msec) clat (usec): min=56, max=24872, avg=4610.86, stdev=2609.58 lat (usec): min=56, max=24873, avg=4612.01, stdev=2609.43 clat percentiles (usec): | 1.00th=[ 60], 5.00th=[ 1876], 10.00th=[ 2278], 20.00th=[ 2671], | 30.00th=[ 3032], 40.00th=[ 3523], 50.00th=[ 4080], 60.00th=[ 4817], | 70.00th=[ 5538], 80.00th=[ 6325], 90.00th=[ 7635], 95.00th=[ 9110], | 99.00th=[13435], 99.50th=[15926], 99.90th=[20055], 99.95th=[24773], | 99.99th=[24773] bw ( KiB/s): min= 288, max= 597, per=25.23%, avg=471.48, stdev=74.16, samples=25 iops : min= 72, max= 149, avg=117.80, stdev=18.54, samples=25 write: IOPS=122, BW=489KiB/s (501kB/s)(6236KiB/12751msec) clat (usec): min=433, max=165066, avg=3657.51, stdev=5136.73 lat (usec): min=434, max=165067, avg=3660.30, stdev=5137.88 clat percentiles (usec): | 1.00th=[ 498], 5.00th=[ 545], 10.00th=[ 578], 20.00th=[ 685], | 30.00th=[ 898], 40.00th=[ 2040], 50.00th=[ 2868], 60.00th=[ 3589], | 70.00th=[ 4686], 80.00th=[ 6128], 90.00th=[ 8029], 95.00th=[ 9503], | 99.00th=[ 13042], 99.50th=[ 15664], 99.90th=[ 23200], 99.95th=[164627], | 99.99th=[164627] bw ( KiB/s): min= 360, max= 616, per=24.83%, avg=485.60, stdev=72.11, samples=25 iops : min= 90, max= 154, avg=121.32, stdev=18.08, samples=25 lat (usec) : 100=1.89%, 250=0.33%, 500=0.65%, 750=11.72%, 1000=4.04% lat (msec) : 2=4.36%, 4=33.63%, 10=39.55%, 20=3.71%, 50=0.10% lat (msec) : 250=0.03% cpu : usr=0.19%, sys=22.68%, ctx=5891, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1869KiB/s (1914kB/s), 469KiB/s-475KiB/s (480kB/s-486kB/s), io=23.4MiB (24.6MB), run=12570-12847msec WRITE: bw=1957KiB/s (2004kB/s), 487KiB/s-505KiB/s (499kB/s-517kB/s), io=24.6MiB (25.7MB), run=12570-12847msec rand-rw: (groupid=0, jobs=1): err= 0: pid=8558: Thu Jun 20 01:09:08 2024 read: IOPS=92, BW=370KiB/s (379kB/s)(5992KiB/16184msec) clat (usec): min=1604, max=175114, avg=4388.96, stdev=5568.14 lat (usec): min=1605, max=175116, avg=4389.68, stdev=5568.17 clat percentiles (usec): | 1.00th=[ 1729], 5.00th=[ 1844], 10.00th=[ 1909], 20.00th=[ 2057], | 30.00th=[ 2212], 40.00th=[ 2474], 50.00th=[ 2769], 60.00th=[ 3326], | 70.00th=[ 4293], 80.00th=[ 5997], 90.00th=[ 9372], 95.00th=[ 11731], | 99.00th=[ 16909], 99.50th=[ 18744], 99.90th=[ 25035], 99.95th=[175113], | 99.99th=[175113] bw ( KiB/s): min= 176, max= 840, per=24.79%, avg=366.38, stdev=184.02, samples=32 iops : min= 44, max= 210, avg=91.53, stdev=46.01, samples=32 write: IOPS=97, BW=389KiB/s (398kB/s)(6296KiB/16184msec) clat (usec): min=2249, max=28413, avg=6061.74, stdev=3848.04 lat (usec): min=2250, max=28414, avg=6062.54, stdev=3848.06 clat percentiles (usec): | 1.00th=[ 2474], 5.00th=[ 2802], 10.00th=[ 2966], 20.00th=[ 3195], | 30.00th=[ 3425], 40.00th=[ 3752], 50.00th=[ 4424], 60.00th=[ 5342], | 70.00th=[ 7046], 80.00th=[ 8848], 90.00th=[11600], 95.00th=[13698], | 99.00th=[19530], 99.50th=[22414], 99.90th=[26608], 99.95th=[28443], | 99.99th=[28443] bw ( KiB/s): min= 184, max= 814, per=24.85%, avg=384.38, stdev=186.90, samples=32 iops : min= 46, max= 203, avg=96.03, stdev=46.72, samples=32 lat (msec) : 2=7.88%, 4=47.88%, 10=31.93%, 20=11.62%, 50=0.65% lat (msec) : 250=0.03% cpu : usr=0.19%, sys=11.29%, ctx=3589, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8559: Thu Jun 20 01:09:08 2024 read: IOPS=91, BW=367KiB/s (376kB/s)(5940KiB/16196msec) clat (usec): min=1588, max=169212, avg=4301.76, stdev=5410.09 lat (usec): min=1589, max=169215, avg=4302.40, stdev=5410.12 clat percentiles (usec): | 1.00th=[ 1729], 5.00th=[ 1827], 10.00th=[ 1893], 20.00th=[ 2024], | 30.00th=[ 2180], 40.00th=[ 2442], 50.00th=[ 2802], 60.00th=[ 3359], | 70.00th=[ 4113], 80.00th=[ 5866], 90.00th=[ 9110], 95.00th=[ 11207], | 99.00th=[ 17433], 99.50th=[ 19268], 99.90th=[ 27395], 99.95th=[168821], | 99.99th=[168821] bw ( KiB/s): min= 104, max= 784, per=24.64%, avg=364.16, stdev=194.60, samples=32 iops : min= 26, max= 196, avg=90.97, stdev=48.70, samples=32 write: IOPS=97, BW=392KiB/s (401kB/s)(6348KiB/16196msec) clat (usec): min=2231, max=30200, avg=6137.09, stdev=3939.55 lat (usec): min=2232, max=30201, avg=6137.86, stdev=3939.54 clat percentiles (usec): | 1.00th=[ 2409], 5.00th=[ 2769], 10.00th=[ 2933], 20.00th=[ 3195], | 30.00th=[ 3425], 40.00th=[ 3818], 50.00th=[ 4555], 60.00th=[ 5538], | 70.00th=[ 7177], 80.00th=[ 8979], 90.00th=[11731], 95.00th=[13960], | 99.00th=[19792], 99.50th=[23987], 99.90th=[28967], 99.95th=[30278], | 99.99th=[30278] bw ( KiB/s): min= 160, max= 832, per=24.93%, avg=385.66, stdev=185.69, samples=32 iops : min= 40, max= 208, avg=96.34, stdev=46.45, samples=32 lat (msec) : 2=8.85%, 4=46.81%, 10=33.01%, 20=10.61%, 50=0.68% lat (msec) : 250=0.03% cpu : usr=0.16%, sys=11.34%, ctx=3588, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8560: Thu Jun 20 01:09:08 2024 read: IOPS=92, BW=371KiB/s (380kB/s)(6028KiB/16246msec) clat (usec): min=1536, max=27066, avg=4388.04, stdev=3552.01 lat (usec): min=1537, max=27066, avg=4388.75, stdev=3552.02 clat percentiles (usec): | 1.00th=[ 1696], 5.00th=[ 1827], 10.00th=[ 1893], 20.00th=[ 2024], | 30.00th=[ 2180], 40.00th=[ 2442], 50.00th=[ 2802], 60.00th=[ 3359], | 70.00th=[ 4293], 80.00th=[ 6390], 90.00th=[ 9765], 95.00th=[12387], | 99.00th=[16712], 99.50th=[19268], 99.90th=[22152], 99.95th=[27132], | 99.99th=[27132] bw ( KiB/s): min= 128, max= 864, per=24.79%, avg=366.41, stdev=190.44, samples=32 iops : min= 32, max= 216, avg=91.53, stdev=47.63, samples=32 write: IOPS=96, BW=385KiB/s (395kB/s)(6260KiB/16246msec) clat (msec): min=2, max=168, avg= 6.11, stdev= 5.62 lat (msec): min=2, max=168, avg= 6.11, stdev= 5.62 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 3], 20.00th=[ 4], | 30.00th=[ 4], 40.00th=[ 4], 50.00th=[ 5], 60.00th=[ 6], | 70.00th=[ 8], 80.00th=[ 9], 90.00th=[ 12], 95.00th=[ 14], | 99.00th=[ 21], 99.50th=[ 24], 99.90th=[ 27], 99.95th=[ 169], | 99.99th=[ 169] bw ( KiB/s): min= 176, max= 808, per=24.49%, avg=378.91, stdev=190.95, samples=32 iops : min= 44, max= 202, avg=94.66, stdev=47.78, samples=32 lat (msec) : 2=8.59%, 4=48.08%, 10=31.05%, 20=11.49%, 50=0.75% lat (msec) : 250=0.03% cpu : usr=0.16%, sys=11.54%, ctx=3605, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8561: Thu Jun 20 01:09:08 2024 read: IOPS=93, BW=373KiB/s (382kB/s)(6052KiB/16208msec) clat (usec): min=1541, max=25278, avg=4316.87, stdev=3507.51 lat (usec): min=1542, max=25279, avg=4317.55, stdev=3507.50 clat percentiles (usec): | 1.00th=[ 1713], 5.00th=[ 1811], 10.00th=[ 1893], 20.00th=[ 2024], | 30.00th=[ 2180], 40.00th=[ 2409], 50.00th=[ 2802], 60.00th=[ 3294], | 70.00th=[ 4359], 80.00th=[ 6063], 90.00th=[ 9372], 95.00th=[12256], | 99.00th=[17957], 99.50th=[19006], 99.90th=[22938], 99.95th=[25297], | 99.99th=[25297] bw ( KiB/s): min= 136, max= 792, per=24.93%, avg=368.41, stdev=185.83, samples=32 iops : min= 34, max= 198, avg=92.03, stdev=46.47, samples=32 write: IOPS=96, BW=385KiB/s (394kB/s)(6236KiB/16208msec) clat (msec): min=2, max=168, avg= 6.16, stdev= 5.63 lat (msec): min=2, max=168, avg= 6.16, stdev= 5.63 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 3], 20.00th=[ 4], | 30.00th=[ 4], 40.00th=[ 4], 50.00th=[ 5], 60.00th=[ 6], | 70.00th=[ 8], 80.00th=[ 9], 90.00th=[ 12], 95.00th=[ 14], | 99.00th=[ 21], 99.50th=[ 24], 99.90th=[ 27], 99.95th=[ 169], | 99.99th=[ 169] bw ( KiB/s): min= 160, max= 808, per=24.54%, avg=379.66, stdev=189.93, samples=32 iops : min= 40, max= 202, avg=94.84, stdev=47.53, samples=32 lat (msec) : 2=8.85%, 4=46.81%, 10=32.58%, 20=11.00%, 50=0.72% lat (msec) : 250=0.03% cpu : usr=0.17%, sys=11.32%, ctx=3606, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1478KiB/s (1513kB/s), 367KiB/s-373KiB/s (376kB/s-382kB/s), io=23.4MiB (24.6MB), run=16184-16246msec WRITE: bw=1547KiB/s (1585kB/s), 385KiB/s-392KiB/s (394kB/s-401kB/s), io=24.6MiB (25.7MB), run=16184-16246msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=8590: Thu Jun 20 01:09:15 2024 read: IOPS=56, BW=902KiB/s (924kB/s)(5920KiB/6561msec) clat (usec): min=59, max=48130, avg=12123.11, stdev=5241.71 lat (usec): min=59, max=48131, avg=12123.77, stdev=5241.67 clat percentiles (usec): | 1.00th=[ 118], 5.00th=[ 7373], 10.00th=[ 8291], 20.00th=[ 9110], | 30.00th=[10028], 40.00th=[10552], 50.00th=[11207], 60.00th=[11994], | 70.00th=[12649], 80.00th=[14222], 90.00th=[16581], 95.00th=[20055], | 99.00th=[35914], 99.50th=[40109], 99.90th=[47973], 99.95th=[47973], | 99.99th=[47973] bw ( KiB/s): min= 640, max= 1152, per=24.68%, avg=900.92, stdev=149.06, samples=12 iops : min= 40, max= 72, avg=56.17, stdev= 9.25, samples=12 write: IOPS=60, BW=971KiB/s (994kB/s)(6368KiB/6561msec) clat (usec): min=741, max=33316, avg=5198.91, stdev=5044.94 lat (usec): min=742, max=33317, avg=5199.95, stdev=5045.00 clat percentiles (usec): | 1.00th=[ 766], 5.00th=[ 824], 10.00th=[ 889], 20.00th=[ 1336], | 30.00th=[ 2343], 40.00th=[ 2802], 50.00th=[ 3359], 60.00th=[ 4555], | 70.00th=[ 5538], 80.00th=[ 7963], 90.00th=[11994], 95.00th=[15926], | 99.00th=[25822], 99.50th=[26346], 99.90th=[33424], 99.95th=[33424], | 99.99th=[33424] bw ( KiB/s): min= 638, max= 1408, per=25.48%, avg=978.25, stdev=259.37, samples=12 iops : min= 39, max= 88, avg=61.00, stdev=16.24, samples=12 lat (usec) : 100=0.39%, 250=0.65%, 500=0.13%, 750=0.26%, 1000=7.81% lat (msec) : 2=4.43%, 4=15.89%, 10=29.69%, 20=36.85%, 50=3.91% cpu : usr=0.12%, sys=20.35%, ctx=2888, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8591: Thu Jun 20 01:09:15 2024 read: IOPS=56, BW=912KiB/s (933kB/s)(5744KiB/6301msec) clat (usec): min=61, max=29388, avg=11606.71, stdev=3764.46 lat (usec): min=61, max=29389, avg=11607.33, stdev=3764.46 clat percentiles (usec): | 1.00th=[ 139], 5.00th=[ 7635], 10.00th=[ 8291], 20.00th=[ 9110], | 30.00th=[ 9634], 40.00th=[10159], 50.00th=[11076], 60.00th=[11994], | 70.00th=[12780], 80.00th=[13960], 90.00th=[16057], 95.00th=[18220], | 99.00th=[23462], 99.50th=[28443], 99.90th=[29492], 99.95th=[29492], | 99.99th=[29492] bw ( KiB/s): min= 704, max= 1120, per=24.91%, avg=909.17, stdev=119.38, samples=12 iops : min= 44, max= 70, avg=56.75, stdev= 7.50, samples=12 write: IOPS=64, BW=1039KiB/s (1063kB/s)(6544KiB/6301msec) clat (usec): min=692, max=73721, avg=5197.11, stdev=6200.63 lat (usec): min=694, max=73721, avg=5198.09, stdev=6200.64 clat percentiles (usec): | 1.00th=[ 766], 5.00th=[ 807], 10.00th=[ 881], 20.00th=[ 1172], | 30.00th=[ 2376], 40.00th=[ 2737], 50.00th=[ 3130], 60.00th=[ 3884], | 70.00th=[ 5342], 80.00th=[ 6980], 90.00th=[12518], 95.00th=[16450], | 99.00th=[25035], 99.50th=[28443], 99.90th=[73925], 99.95th=[73925], | 99.99th=[73925] bw ( KiB/s): min= 640, max= 1376, per=26.59%, avg=1021.17, stdev=231.62, samples=12 iops : min= 40, max= 86, avg=63.75, stdev=14.60, samples=12 lat (usec) : 100=0.39%, 250=0.26%, 500=0.13%, 750=0.26%, 1000=8.59% lat (msec) : 2=4.43%, 4=19.66%, 10=29.43%, 20=34.24%, 50=2.47% lat (msec) : 100=0.13% cpu : usr=0.11%, sys=18.81%, ctx=2791, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8592: Thu Jun 20 01:09:15 2024 read: IOPS=59, BW=955KiB/s (978kB/s)(6256KiB/6553msec) clat (usec): min=61, max=29342, avg=11258.14, stdev=3548.75 lat (usec): min=62, max=29343, avg=11260.56, stdev=3549.67 clat percentiles (usec): | 1.00th=[ 65], 5.00th=[ 7504], 10.00th=[ 8291], 20.00th=[ 8979], | 30.00th=[ 9634], 40.00th=[10290], 50.00th=[10945], 60.00th=[11600], | 70.00th=[12387], 80.00th=[13304], 90.00th=[15008], 95.00th=[17171], | 99.00th=[23725], 99.50th=[25297], 99.90th=[29230], 99.95th=[29230], | 99.99th=[29230] bw ( KiB/s): min= 640, max= 1248, per=25.93%, avg=946.50, stdev=180.89, samples=12 iops : min= 40, max= 78, avg=59.08, stdev=11.37, samples=12 write: IOPS=57, BW=920KiB/s (943kB/s)(6032KiB/6553msec) clat (usec): min=682, max=67522, avg=5680.81, stdev=6869.65 lat (usec): min=683, max=67523, avg=5681.76, stdev=6869.65 clat percentiles (usec): | 1.00th=[ 775], 5.00th=[ 807], 10.00th=[ 930], 20.00th=[ 1336], | 30.00th=[ 2409], 40.00th=[ 2802], 50.00th=[ 3392], 60.00th=[ 4621], | 70.00th=[ 5735], 80.00th=[ 7767], 90.00th=[13042], 95.00th=[16188], | 99.00th=[33817], 99.50th=[43779], 99.90th=[67634], 99.95th=[67634], | 99.99th=[67634] bw ( KiB/s): min= 640, max= 1280, per=23.54%, avg=903.83, stdev=185.69, samples=12 iops : min= 40, max= 80, avg=56.42, stdev=11.63, samples=12 lat (usec) : 100=1.17%, 250=0.13%, 500=0.13%, 750=0.13%, 1000=6.25% lat (msec) : 2=4.69%, 4=16.54%, 10=30.86%, 20=37.76%, 50=2.21% lat (msec) : 100=0.13% cpu : usr=0.08%, sys=18.56%, ctx=2815, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8593: Thu Jun 20 01:09:15 2024 read: IOPS=59, BW=959KiB/s (982kB/s)(6032KiB/6293msec) clat (usec): min=61, max=32658, avg=11509.18, stdev=3644.14 lat (usec): min=61, max=32659, avg=11509.83, stdev=3644.13 clat percentiles (usec): | 1.00th=[ 70], 5.00th=[ 7373], 10.00th=[ 8356], 20.00th=[ 9241], | 30.00th=[ 9896], 40.00th=[10421], 50.00th=[10945], 60.00th=[11600], | 70.00th=[12518], 80.00th=[13304], 90.00th=[15008], 95.00th=[17171], | 99.00th=[27132], 99.50th=[31065], 99.90th=[32637], 99.95th=[32637], | 99.99th=[32637] bw ( KiB/s): min= 672, max= 1184, per=25.92%, avg=946.17, stdev=144.93, samples=12 iops : min= 42, max= 74, avg=59.00, stdev= 8.95, samples=12 write: IOPS=62, BW=994KiB/s (1018kB/s)(6256KiB/6293msec) clat (usec): min=664, max=69095, avg=4974.24, stdev=5843.95 lat (usec): min=665, max=69097, avg=4975.23, stdev=5843.97 clat percentiles (usec): | 1.00th=[ 758], 5.00th=[ 791], 10.00th=[ 848], 20.00th=[ 1074], | 30.00th=[ 2212], 40.00th=[ 2638], 50.00th=[ 3228], 60.00th=[ 4293], | 70.00th=[ 5669], 80.00th=[ 7177], 90.00th=[ 9634], 95.00th=[14353], | 99.00th=[31851], 99.50th=[42206], 99.90th=[68682], 99.95th=[68682], | 99.99th=[68682] bw ( KiB/s): min= 736, max= 1344, per=25.68%, avg=986.17, stdev=181.10, samples=12 iops : min= 46, max= 84, avg=61.50, stdev=11.23, samples=12 lat (usec) : 100=0.52%, 250=0.13%, 750=0.26%, 1000=8.98% lat (msec) : 2=3.65%, 4=16.41%, 10=32.03%, 20=35.81%, 50=2.08% lat (msec) : 100=0.13% cpu : usr=0.03%, sys=19.06%, ctx=2819, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=3651KiB/s (3738kB/s), 902KiB/s-959KiB/s (924kB/s-982kB/s), io=23.4MiB (24.5MB), run=6293-6561msec WRITE: bw=3841KiB/s (3933kB/s), 920KiB/s-1039KiB/s (943kB/s-1063kB/s), io=24.6MiB (25.8MB), run=6293-6561msec rand-rw: (groupid=0, jobs=1): err= 0: pid=8594: Thu Jun 20 01:09:16 2024 read: IOPS=56, BW=901KiB/s (923kB/s)(5920KiB/6569msec) clat (usec): min=1906, max=41293, avg=6949.42, stdev=6048.95 lat (usec): min=1906, max=41294, avg=6950.09, stdev=6049.09 clat percentiles (usec): | 1.00th=[ 1942], 5.00th=[ 2180], 10.00th=[ 2376], 20.00th=[ 2769], | 30.00th=[ 3195], 40.00th=[ 3589], 50.00th=[ 4359], 60.00th=[ 6063], | 70.00th=[ 8291], 80.00th=[10683], 90.00th=[14091], 95.00th=[18744], | 99.00th=[33162], 99.50th=[41157], 99.90th=[41157], 99.95th=[41157], | 99.99th=[41157] bw ( KiB/s): min= 640, max= 1184, per=24.84%, avg=901.17, stdev=169.28, samples=12 iops : min= 40, max= 74, avg=56.25, stdev=10.60, samples=12 write: IOPS=60, BW=969KiB/s (993kB/s)(6368KiB/6569msec) clat (usec): min=3087, max=43282, avg=10027.11, stdev=6203.37 lat (usec): min=3088, max=43283, avg=10028.15, stdev=6203.40 clat percentiles (usec): | 1.00th=[ 3228], 5.00th=[ 3687], 10.00th=[ 4080], 20.00th=[ 4883], | 30.00th=[ 6128], 40.00th=[ 7242], 50.00th=[ 8455], 60.00th=[ 9765], | 70.00th=[11469], 80.00th=[14091], 90.00th=[18482], 95.00th=[22414], | 99.00th=[33817], 99.50th=[43254], 99.90th=[43254], 99.95th=[43254], | 99.99th=[43254] bw ( KiB/s): min= 576, max= 1344, per=25.63%, avg=978.42, stdev=234.02, samples=12 iops : min= 36, max= 84, avg=61.08, stdev=14.59, samples=12 lat (msec) : 2=0.65%, 4=26.95%, 10=41.67%, 20=25.13%, 50=5.60% cpu : usr=0.14%, sys=9.35%, ctx=1231, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8595: Thu Jun 20 01:09:16 2024 read: IOPS=56, BW=909KiB/s (930kB/s)(5744KiB/6322msec) clat (usec): min=1868, max=45344, avg=6805.53, stdev=5573.21 lat (usec): min=1868, max=45345, avg=6806.17, stdev=5573.22 clat percentiles (usec): | 1.00th=[ 1942], 5.00th=[ 2212], 10.00th=[ 2409], 20.00th=[ 2704], | 30.00th=[ 3064], 40.00th=[ 3589], 50.00th=[ 4424], 60.00th=[ 5932], | 70.00th=[ 8717], 80.00th=[10814], 90.00th=[13566], 95.00th=[16319], | 99.00th=[26346], 99.50th=[39060], 99.90th=[45351], 99.95th=[45351], | 99.99th=[45351] bw ( KiB/s): min= 640, max= 1056, per=24.62%, avg=893.17, stdev=124.00, samples=12 iops : min= 40, max= 66, avg=55.75, stdev= 7.79, samples=12 write: IOPS=64, BW=1035KiB/s (1060kB/s)(6544KiB/6322msec) clat (usec): min=2730, max=66684, avg=9463.88, stdev=6090.21 lat (usec): min=2731, max=66687, avg=9465.24, stdev=6090.34 clat percentiles (usec): | 1.00th=[ 2933], 5.00th=[ 3458], 10.00th=[ 3818], 20.00th=[ 4621], | 30.00th=[ 5735], 40.00th=[ 7177], 50.00th=[ 8094], 60.00th=[ 9372], | 70.00th=[10814], 80.00th=[13304], 90.00th=[16712], 95.00th=[20317], | 99.00th=[30016], 99.50th=[31327], 99.90th=[66847], 99.95th=[66847], | 99.99th=[66847] bw ( KiB/s): min= 576, max= 1376, per=26.54%, avg=1013.17, stdev=231.13, samples=12 iops : min= 36, max= 86, avg=63.25, stdev=14.55, samples=12 lat (msec) : 2=0.65%, 4=27.34%, 10=41.28%, 20=26.69%, 50=3.91% lat (msec) : 100=0.13% cpu : usr=0.14%, sys=9.32%, ctx=1164, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8596: Thu Jun 20 01:09:16 2024 read: IOPS=59, BW=948KiB/s (970kB/s)(6256KiB/6601msec) clat (usec): min=1788, max=41110, avg=6590.27, stdev=4876.76 lat (usec): min=1789, max=41111, avg=6590.94, stdev=4876.79 clat percentiles (usec): | 1.00th=[ 1975], 5.00th=[ 2212], 10.00th=[ 2442], 20.00th=[ 2737], | 30.00th=[ 3032], 40.00th=[ 3556], 50.00th=[ 4490], 60.00th=[ 6652], | 70.00th=[ 8717], 80.00th=[10683], 90.00th=[12387], 95.00th=[15139], | 99.00th=[24249], 99.50th=[29492], 99.90th=[41157], 99.95th=[41157], | 99.99th=[41157] bw ( KiB/s): min= 640, max= 1280, per=26.16%, avg=949.08, stdev=176.87, samples=12 iops : min= 40, max= 80, avg=59.25, stdev=11.01, samples=12 write: IOPS=57, BW=914KiB/s (936kB/s)(6032KiB/6601msec) clat (usec): min=2757, max=75408, avg=10655.98, stdev=7348.57 lat (usec): min=2758, max=75410, avg=10657.01, stdev=7348.62 clat percentiles (usec): | 1.00th=[ 3130], 5.00th=[ 3523], 10.00th=[ 4080], 20.00th=[ 5014], | 30.00th=[ 6652], 40.00th=[ 7504], 50.00th=[ 8717], 60.00th=[10159], | 70.00th=[12256], 80.00th=[14615], 90.00th=[19006], 95.00th=[24773], | 99.00th=[36439], 99.50th=[39584], 99.90th=[74974], 99.95th=[74974], | 99.99th=[74974] bw ( KiB/s): min= 608, max= 1280, per=23.68%, avg=903.75, stdev=182.95, samples=12 iops : min= 38, max= 80, avg=56.42, stdev=11.39, samples=12 lat (msec) : 2=0.52%, 4=26.69%, 10=41.02%, 20=26.82%, 50=4.82% lat (msec) : 100=0.13% cpu : usr=0.09%, sys=9.26%, ctx=1216, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8597: Thu Jun 20 01:09:16 2024 read: IOPS=59, BW=958KiB/s (981kB/s)(6032KiB/6296msec) clat (usec): min=1945, max=55145, avg=6420.79, stdev=5451.26 lat (usec): min=1946, max=55145, avg=6421.41, stdev=5451.26 clat percentiles (usec): | 1.00th=[ 2008], 5.00th=[ 2180], 10.00th=[ 2376], 20.00th=[ 2704], | 30.00th=[ 3032], 40.00th=[ 3392], 50.00th=[ 4146], 60.00th=[ 5342], | 70.00th=[ 7832], 80.00th=[ 9896], 90.00th=[13042], 95.00th=[15139], | 99.00th=[26870], 99.50th=[31589], 99.90th=[55313], 99.95th=[55313], | 99.99th=[55313] bw ( KiB/s): min= 608, max= 1312, per=26.02%, avg=944.00, stdev=185.84, samples=12 iops : min= 38, max= 82, avg=59.00, stdev=11.62, samples=12 write: IOPS=62, BW=994KiB/s (1017kB/s)(6256KiB/6296msec) clat (usec): min=2580, max=86174, avg=9891.63, stdev=7008.03 lat (usec): min=2581, max=86176, avg=9892.74, stdev=7008.09 clat percentiles (usec): | 1.00th=[ 2737], 5.00th=[ 3490], 10.00th=[ 3916], 20.00th=[ 5080], | 30.00th=[ 6063], 40.00th=[ 7046], 50.00th=[ 8356], 60.00th=[ 9503], | 70.00th=[11207], 80.00th=[13435], 90.00th=[17695], 95.00th=[19792], | 99.00th=[32375], 99.50th=[45876], 99.90th=[86508], 99.95th=[86508], | 99.99th=[86508] bw ( KiB/s): min= 704, max= 1312, per=25.78%, avg=984.00, stdev=171.58, samples=12 iops : min= 44, max= 82, avg=61.50, stdev=10.72, samples=12 lat (msec) : 2=0.39%, 4=28.91%, 10=42.06%, 20=25.13%, 50=3.26% lat (msec) : 100=0.26% cpu : usr=0.14%, sys=9.37%, ctx=1169, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=3629KiB/s (3716kB/s), 901KiB/s-958KiB/s (923kB/s-981kB/s), io=23.4MiB (24.5MB), run=6296-6601msec WRITE: bw=3818KiB/s (3909kB/s), 914KiB/s-1035KiB/s (936kB/s-1060kB/s), io=24.6MiB (25.8MB), run=6296-6601msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=8608: Thu Jun 20 01:09:19 2024 read: IOPS=3, BW=3556KiB/s (3641kB/s)(7168KiB/2016msec) clat (usec): min=432, max=706496, avg=179091.57, stdev=260275.58 lat (usec): min=432, max=706496, avg=179092.62, stdev=260275.40 clat percentiles (usec): | 1.00th=[ 433], 5.00th=[ 433], 10.00th=[ 433], 20.00th=[ 498], | 30.00th=[ 9372], 40.00th=[ 9372], 50.00th=[ 40633], 60.00th=[187696], | 70.00th=[187696], 80.00th=[308282], 90.00th=[708838], 95.00th=[708838], | 99.00th=[708838], 99.50th=[708838], 99.90th=[708838], 99.95th=[708838], | 99.99th=[708838] bw ( KiB/s): min= 4096, max= 6131, per=74.72%, avg=5113.50, stdev=1438.96, samples=2 iops : min= 4, max= 5, avg= 4.50, stdev= 0.71, samples=2 write: IOPS=2, BW=2540KiB/s (2601kB/s)(5120KiB/2016msec) clat (msec): min=22, max=610, avg=149.52, stdev=257.65 lat (msec): min=22, max=610, avg=149.57, stdev=257.65 clat percentiles (msec): | 1.00th=[ 23], 5.00th=[ 23], 10.00th=[ 23], 20.00th=[ 23], | 30.00th=[ 35], 40.00th=[ 35], 50.00th=[ 37], 60.00th=[ 37], | 70.00th=[ 44], 80.00th=[ 44], 90.00th=[ 609], 95.00th=[ 609], | 99.00th=[ 609], 99.50th=[ 609], 99.90th=[ 609], 99.95th=[ 609], | 99.99th=[ 609] bw ( KiB/s): min= 2043, max= 3488, per=26.37%, avg=2526.33, stdev=832.83, samples=3 iops : min= 1, max= 3, avg= 2.00, stdev= 1.00, samples=3 lat (usec) : 500=16.67% lat (msec) : 10=8.33%, 50=41.67%, 250=8.33%, 500=8.33%, 750=16.67% cpu : usr=0.00%, sys=24.57%, ctx=474, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8609: Thu Jun 20 01:09:19 2024 read: IOPS=1, BW=1549KiB/s (1586kB/s)(2048KiB/1322msec) clat (msec): min=46, max=517, avg=281.98, stdev=333.03 lat (msec): min=46, max=517, avg=281.98, stdev=333.03 clat percentiles (msec): | 1.00th=[ 47], 5.00th=[ 47], 10.00th=[ 47], 20.00th=[ 47], | 30.00th=[ 47], 40.00th=[ 47], 50.00th=[ 47], 60.00th=[ 518], | 70.00th=[ 518], 80.00th=[ 518], 90.00th=[ 518], 95.00th=[ 518], | 99.00th=[ 518], 99.50th=[ 518], 99.90th=[ 518], 99.95th=[ 518], | 99.99th=[ 518] bw ( KiB/s): min= 2043, max= 2043, per=29.85%, avg=2043.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=7, BW=7746KiB/s (7932kB/s)(10.0MiB/1322msec) clat (msec): min=24, max=288, avg=75.52, stdev=83.48 lat (msec): min=24, max=288, avg=75.58, stdev=83.47 clat percentiles (msec): | 1.00th=[ 25], 5.00th=[ 25], 10.00th=[ 25], 20.00th=[ 27], | 30.00th=[ 28], 40.00th=[ 30], 50.00th=[ 35], 60.00th=[ 39], | 70.00th=[ 78], 80.00th=[ 78], 90.00th=[ 144], 95.00th=[ 288], | 99.00th=[ 288], 99.50th=[ 288], 99.90th=[ 288], 99.95th=[ 288], | 99.99th=[ 288] bw ( KiB/s): min= 4096, max= 6131, per=53.37%, avg=5113.50, stdev=1438.96, samples=2 iops : min= 4, max= 5, avg= 4.50, stdev= 0.71, samples=2 lat (msec) : 50=58.33%, 100=16.67%, 250=8.33%, 500=8.33%, 750=8.33% cpu : usr=0.08%, sys=26.27%, ctx=348, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8610: Thu Jun 20 01:09:19 2024 read: IOPS=2, BW=2053KiB/s (2103kB/s)(6144KiB/2992msec) clat (usec): min=451, max=951548, avg=264892.98, stdev=395487.22 lat (usec): min=451, max=951549, avg=264893.91, stdev=395487.31 clat percentiles (usec): | 1.00th=[ 453], 5.00th=[ 453], 10.00th=[ 453], 20.00th=[ 570], | 30.00th=[ 570], 40.00th=[ 41681], 50.00th=[ 41681], 60.00th=[ 53740], | 70.00th=[541066], 80.00th=[541066], 90.00th=[952108], 95.00th=[952108], | 99.00th=[952108], 99.50th=[952108], 99.90th=[952108], 99.95th=[952108], | 99.99th=[952108] bw ( KiB/s): min= 4096, max= 5278, per=68.48%, avg=4687.00, stdev=835.80, samples=2 iops : min= 4, max= 5, avg= 4.50, stdev= 0.71, samples=2 write: IOPS=2, BW=2053KiB/s (2103kB/s)(6144KiB/2992msec) clat (msec): min=49, max=736, avg=232.30, stdev=253.43 lat (msec): min=49, max=736, avg=232.34, stdev=253.42 clat percentiles (msec): | 1.00th=[ 50], 5.00th=[ 50], 10.00th=[ 50], 20.00th=[ 79], | 30.00th=[ 79], 40.00th=[ 165], 50.00th=[ 165], 60.00th=[ 184], | 70.00th=[ 184], 80.00th=[ 184], 90.00th=[ 735], 95.00th=[ 735], | 99.00th=[ 735], 99.50th=[ 735], 99.90th=[ 735], 99.95th=[ 735], | 99.99th=[ 735] bw ( KiB/s): min= 1759, max= 6144, per=30.56%, avg=2928.25, stdev=2148.11, samples=4 iops : min= 1, max= 6, avg= 2.50, stdev= 2.38, samples=4 lat (usec) : 500=8.33%, 750=8.33% lat (msec) : 50=16.67%, 100=16.67%, 250=25.00%, 750=16.67%, 1000=8.33% cpu : usr=0.00%, sys=13.74%, ctx=610, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8612: Thu Jun 20 01:09:19 2024 read: IOPS=3, BW=3966KiB/s (4061kB/s)(5120KiB/1291msec) clat (usec): min=527, max=556180, avg=155890.78, stdev=236895.24 lat (usec): min=527, max=556181, avg=155891.91, stdev=236895.11 clat percentiles (usec): | 1.00th=[ 529], 5.00th=[ 529], 10.00th=[ 529], 20.00th=[ 529], | 30.00th=[ 553], 40.00th=[ 553], 50.00th=[ 33817], 60.00th=[ 33817], | 70.00th=[187696], 80.00th=[187696], 90.00th=[557843], 95.00th=[557843], | 99.00th=[557843], 99.50th=[557843], 99.90th=[557843], 99.95th=[557843], | 99.99th=[557843] bw ( KiB/s): min= 2048, max= 2048, per=29.92%, avg=2048.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 write: IOPS=5, BW=5552KiB/s (5686kB/s)(7168KiB/1291msec) clat (msec): min=24, max=237, avg=72.27, stdev=75.25 lat (msec): min=24, max=237, avg=72.33, stdev=75.25 clat percentiles (msec): | 1.00th=[ 25], 5.00th=[ 25], 10.00th=[ 25], 20.00th=[ 35], | 30.00th=[ 37], 40.00th=[ 37], 50.00th=[ 37], 60.00th=[ 51], | 70.00th=[ 51], 80.00th=[ 85], 90.00th=[ 239], 95.00th=[ 239], | 99.00th=[ 239], 99.50th=[ 239], 99.90th=[ 239], 99.95th=[ 239], | 99.99th=[ 239] bw ( KiB/s): min= 8192, max= 8192, per=85.49%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 lat (usec) : 750=16.67% lat (msec) : 50=41.67%, 100=16.67%, 250=16.67%, 750=8.33% cpu : usr=0.08%, sys=23.95%, ctx=419, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=6845KiB/s (7009kB/s), 1549KiB/s-3966KiB/s (1586kB/s-4061kB/s), io=20.0MiB (20.0MB), run=1291-2992msec WRITE: bw=9583KiB/s (9813kB/s), 2053KiB/s-7746KiB/s (2103kB/s-7932kB/s), io=28.0MiB (29.4MB), run=1291-2992msec rand-rw: (groupid=0, jobs=1): err= 0: pid=8611: Thu Jun 20 01:09:19 2024 read: IOPS=3, BW=3238KiB/s (3315kB/s)(7168KiB/2214msec) clat (msec): min=16, max=786, avg=200.39, stdev=275.52 lat (msec): min=16, max=786, avg=200.40, stdev=275.52 clat percentiles (msec): | 1.00th=[ 17], 5.00th=[ 17], 10.00th=[ 17], 20.00th=[ 34], | 30.00th=[ 47], 40.00th=[ 47], 50.00th=[ 55], 60.00th=[ 190], | 70.00th=[ 190], 80.00th=[ 275], 90.00th=[ 785], 95.00th=[ 785], | 99.00th=[ 785], 99.50th=[ 785], 99.90th=[ 785], 99.95th=[ 785], | 99.99th=[ 785] bw ( KiB/s): min= 4096, max= 6131, per=75.21%, avg=5113.50, stdev=1438.96, samples=2 iops : min= 4, max= 5, avg= 4.50, stdev= 0.71, samples=2 write: IOPS=2, BW=2313KiB/s (2368kB/s)(5120KiB/2214msec) clat (msec): min=41, max=598, avg=161.67, stdev=244.30 lat (msec): min=41, max=598, avg=161.72, stdev=244.31 clat percentiles (msec): | 1.00th=[ 42], 5.00th=[ 42], 10.00th=[ 42], 20.00th=[ 42], | 30.00th=[ 43], 40.00th=[ 43], 50.00th=[ 55], 60.00th=[ 55], | 70.00th=[ 71], 80.00th=[ 71], 90.00th=[ 600], 95.00th=[ 600], | 99.00th=[ 600], 99.50th=[ 600], 99.90th=[ 600], 99.95th=[ 600], | 99.99th=[ 600] bw ( KiB/s): min= 2043, max= 4087, per=28.64%, avg=2726.00, stdev=1178.66, samples=3 iops : min= 1, max= 3, avg= 2.00, stdev= 1.00, samples=3 lat (msec) : 20=8.33%, 50=33.33%, 100=25.00%, 250=8.33%, 500=8.33% lat (msec) : 750=8.33%, 1000=8.33% cpu : usr=0.00%, sys=7.41%, ctx=59, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8613: Thu Jun 20 01:09:19 2024 read: IOPS=1, BW=1589KiB/s (1627kB/s)(2048KiB/1289msec) clat (usec): min=17106, max=19031, avg=18068.63, stdev=1361.30 lat (usec): min=17106, max=19031, avg=18069.33, stdev=1361.19 clat percentiles (usec): | 1.00th=[17171], 5.00th=[17171], 10.00th=[17171], 20.00th=[17171], | 30.00th=[17171], 40.00th=[17171], 50.00th=[17171], 60.00th=[19006], | 70.00th=[19006], 80.00th=[19006], 90.00th=[19006], 95.00th=[19006], | 99.00th=[19006], 99.50th=[19006], 99.90th=[19006], 99.95th=[19006], | 99.99th=[19006] bw ( KiB/s): min= 4087, max= 4087, per=60.11%, avg=4087.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 write: IOPS=7, BW=7944KiB/s (8135kB/s)(10.0MiB/1289msec) clat (msec): min=16, max=514, avg=125.13, stdev=164.05 lat (msec): min=16, max=514, avg=125.18, stdev=164.04 clat percentiles (msec): | 1.00th=[ 17], 5.00th=[ 17], 10.00th=[ 17], 20.00th=[ 17], | 30.00th=[ 17], 40.00th=[ 23], 50.00th=[ 36], 60.00th=[ 43], | 70.00th=[ 184], 80.00th=[ 184], 90.00th=[ 288], 95.00th=[ 514], | 99.00th=[ 514], 99.50th=[ 514], 99.90th=[ 514], 99.95th=[ 514], | 99.99th=[ 514] bw ( KiB/s): min= 4096, max=10219, per=75.19%, avg=7157.50, stdev=4329.61, samples=2 iops : min= 4, max= 9, avg= 6.50, stdev= 3.54, samples=2 lat (msec) : 20=41.67%, 50=25.00%, 250=16.67%, 500=8.33%, 750=8.33% cpu : usr=0.23%, sys=8.46%, ctx=45, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8614: Thu Jun 20 01:09:19 2024 read: IOPS=1, BW=2040KiB/s (2089kB/s)(6144KiB/3012msec) clat (msec): min=14, max=854, avg=254.71, stdev=345.86 lat (msec): min=14, max=854, avg=254.71, stdev=345.86 clat percentiles (msec): | 1.00th=[ 15], 5.00th=[ 15], 10.00th=[ 15], 20.00th=[ 26], | 30.00th=[ 26], 40.00th=[ 41], 50.00th=[ 41], 60.00th=[ 100], | 70.00th=[ 498], 80.00th=[ 498], 90.00th=[ 852], 95.00th=[ 852], | 99.00th=[ 852], 99.50th=[ 852], 99.90th=[ 852], 99.95th=[ 852], | 99.99th=[ 852] bw ( KiB/s): min= 2048, max= 6144, per=50.20%, avg=3413.33, stdev=2364.83, samples=3 iops : min= 2, max= 6, avg= 3.33, stdev= 2.31, samples=3 write: IOPS=1, BW=2040KiB/s (2089kB/s)(6144KiB/3012msec) clat (msec): min=37, max=790, avg=247.01, stdev=277.61 lat (msec): min=37, max=790, avg=247.06, stdev=277.59 clat percentiles (msec): | 1.00th=[ 38], 5.00th=[ 38], 10.00th=[ 38], 20.00th=[ 57], | 30.00th=[ 57], 40.00th=[ 169], 50.00th=[ 169], 60.00th=[ 182], | 70.00th=[ 247], 80.00th=[ 247], 90.00th=[ 793], 95.00th=[ 793], | 99.00th=[ 793], 99.50th=[ 793], 99.90th=[ 793], 99.95th=[ 793], | 99.99th=[ 793] bw ( KiB/s): min= 1793, max= 6144, per=31.60%, avg=3008.25, stdev=2093.95, samples=4 iops : min= 1, max= 6, avg= 2.75, stdev= 2.22, samples=4 lat (msec) : 20=8.33%, 50=25.00%, 100=16.67%, 250=25.00%, 500=8.33% lat (msec) : 1000=16.67% cpu : usr=0.03%, sys=4.95%, ctx=60, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8615: Thu Jun 20 01:09:19 2024 read: IOPS=3, BW=3699KiB/s (3788kB/s)(5120KiB/1384msec) clat (msec): min=17, max=571, avg=142.04, stdev=240.56 lat (msec): min=17, max=571, avg=142.04, stdev=240.56 clat percentiles (msec): | 1.00th=[ 18], 5.00th=[ 18], 10.00th=[ 18], 20.00th=[ 18], | 30.00th=[ 31], 40.00th=[ 31], 50.00th=[ 43], 60.00th=[ 43], | 70.00th=[ 48], 80.00th=[ 48], 90.00th=[ 575], 95.00th=[ 575], | 99.00th=[ 575], 99.50th=[ 575], 99.90th=[ 575], 99.95th=[ 575], | 99.99th=[ 575] bw ( KiB/s): min= 4096, max= 4096, per=60.24%, avg=4096.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 write: IOPS=5, BW=5179KiB/s (5303kB/s)(7168KiB/1384msec) clat (msec): min=14, max=198, avg=95.99, stdev=66.26 lat (msec): min=14, max=198, avg=96.06, stdev=66.25 clat percentiles (msec): | 1.00th=[ 15], 5.00th=[ 15], 10.00th=[ 15], 20.00th=[ 50], | 30.00th=[ 64], 40.00th=[ 64], 50.00th=[ 69], 60.00th=[ 109], | 70.00th=[ 109], 80.00th=[ 169], 90.00th=[ 199], 95.00th=[ 199], | 99.00th=[ 199], 99.50th=[ 199], 99.90th=[ 199], 99.95th=[ 199], | 99.99th=[ 199] bw ( KiB/s): min= 2048, max= 6144, per=43.03%, avg=4096.00, stdev=2896.31, samples=2 iops : min= 2, max= 6, avg= 4.00, stdev= 2.83, samples=2 lat (msec) : 20=16.67%, 50=33.33%, 100=16.67%, 250=25.00%, 750=8.33% cpu : usr=0.14%, sys=8.46%, ctx=109, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=6799KiB/s (6963kB/s), 1589KiB/s-3699KiB/s (1627kB/s-3788kB/s), io=20.0MiB (20.0MB), run=1289-3012msec WRITE: bw=9519KiB/s (9748kB/s), 2040KiB/s-7944KiB/s (2089kB/s-8135kB/s), io=28.0MiB (29.4MB), run=1289-3012msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=8622: Thu Jun 20 01:09:23 2024 read: IOPS=1, BW=5218KiB/s (5343kB/s)(12.0MiB/2355msec) clat (msec): min=76, max=2061, avg=781.66, stdev=1110.58 lat (msec): min=76, max=2061, avg=781.66, stdev=1110.58 clat percentiles (msec): | 1.00th=[ 77], 5.00th=[ 77], 10.00th=[ 77], 20.00th=[ 77], | 30.00th=[ 77], 40.00th=[ 207], 50.00th=[ 207], 60.00th=[ 207], | 70.00th=[ 2056], 80.00th=[ 2056], 90.00th=[ 2056], 95.00th=[ 2056], | 99.00th=[ 2056], 99.50th=[ 2056], 99.90th=[ 2056], 99.95th=[ 2056], | 99.99th=[ 2056] bw ( KiB/s): min= 7275, max= 7275, per=100.00%, avg=7275.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 250=33.33% cpu : usr=0.00%, sys=3.70%, ctx=35, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8623: Thu Jun 20 01:09:23 2024 read: IOPS=0, BW=1391KiB/s (1424kB/s)(4096KiB/2945msec) clat (nsec): min=198636k, max=198636k, avg=198635588.00, stdev= 0.00 lat (nsec): min=198637k, max=198637k, avg=198636751.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 199], 5.00th=[ 199], 10.00th=[ 199], 20.00th=[ 199], | 30.00th=[ 199], 40.00th=[ 199], 50.00th=[ 199], 60.00th=[ 199], | 70.00th=[ 199], 80.00th=[ 199], 90.00th=[ 199], 95.00th=[ 199], | 99.00th=[ 199], 99.50th=[ 199], 99.90th=[ 199], 99.95th=[ 199], | 99.99th=[ 199] bw ( KiB/s): min= 7892, max= 7892, per=100.00%, avg=7892.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=0, BW=2782KiB/s (2848kB/s)(8192KiB/2945msec) clat (msec): min=250, max=2485, avg=1368.16, stdev=1580.73 lat (msec): min=250, max=2486, avg=1368.37, stdev=1580.82 clat percentiles (msec): | 1.00th=[ 251], 5.00th=[ 251], 10.00th=[ 251], 20.00th=[ 251], | 30.00th=[ 251], 40.00th=[ 251], 50.00th=[ 251], 60.00th=[ 2500], | 70.00th=[ 2500], 80.00th=[ 2500], 90.00th=[ 2500], 95.00th=[ 2500], | 99.00th=[ 2500], 99.50th=[ 2500], 99.90th=[ 2500], 99.95th=[ 2500], | 99.99th=[ 2500] bw ( KiB/s): min= 7892, max= 7892, per=70.93%, avg=7892.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=33.33% cpu : usr=0.03%, sys=5.47%, ctx=166, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8624: Thu Jun 20 01:09:23 2024 write: IOPS=6, BW=24.2MiB/s (25.4MB/s)(12.0MiB/496msec) clat (msec): min=60, max=282, avg=161.79, stdev=112.57 lat (msec): min=60, max=283, avg=162.15, stdev=112.52 clat percentiles (msec): | 1.00th=[ 61], 5.00th=[ 61], 10.00th=[ 61], 20.00th=[ 61], | 30.00th=[ 61], 40.00th=[ 142], 50.00th=[ 142], 60.00th=[ 142], | 70.00th=[ 284], 80.00th=[ 284], 90.00th=[ 284], 95.00th=[ 284], | 99.00th=[ 284], 99.50th=[ 284], 99.90th=[ 284], 99.95th=[ 284], | 99.99th=[ 284] lat (msec) : 100=33.33%, 250=33.33%, 500=33.33% cpu : usr=0.20%, sys=14.55%, ctx=30, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8625: Thu Jun 20 01:09:23 2024 write: IOPS=4, BW=17.0MiB/s (17.9MB/s)(12.0MiB/704msec) clat (msec): min=86, max=414, avg=234.20, stdev=166.27 lat (msec): min=86, max=414, avg=234.40, stdev=166.23 clat percentiles (msec): | 1.00th=[ 87], 5.00th=[ 87], 10.00th=[ 87], 20.00th=[ 87], | 30.00th=[ 87], 40.00th=[ 203], 50.00th=[ 203], 60.00th=[ 203], | 70.00th=[ 414], 80.00th=[ 414], 90.00th=[ 414], 95.00th=[ 414], | 99.00th=[ 414], 99.50th=[ 414], 99.90th=[ 414], 99.95th=[ 414], | 99.99th=[ 414] bw ( KiB/s): min= 8175, max= 8175, per=73.48%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 250=33.33%, 500=33.33% cpu : usr=0.00%, sys=15.93%, ctx=63, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=5563KiB/s (5697kB/s), 1391KiB/s-5218KiB/s (1424kB/s-5343kB/s), io=16.0MiB (16.8MB), run=2355-2945msec WRITE: bw=10.9MiB/s (11.4MB/s), 2782KiB/s-24.2MiB/s (2848kB/s-25.4MB/s), io=32.0MiB (33.6MB), run=496-2945msec rand-rw: (groupid=0, jobs=1): err= 0: pid=8627: Thu Jun 20 01:09:23 2024 read: IOPS=1, BW=6065KiB/s (6211kB/s)(12.0MiB/2026msec) clat (msec): min=92, max=1603, avg=675.10, stdev=812.99 lat (msec): min=92, max=1603, avg=675.10, stdev=812.99 clat percentiles (msec): | 1.00th=[ 92], 5.00th=[ 92], 10.00th=[ 92], 20.00th=[ 92], | 30.00th=[ 92], 40.00th=[ 330], 50.00th=[ 330], 60.00th=[ 330], | 70.00th=[ 1603], 80.00th=[ 1603], 90.00th=[ 1603], 95.00th=[ 1603], | 99.00th=[ 1603], 99.50th=[ 1603], 99.90th=[ 1603], 99.95th=[ 1603], | 99.99th=[ 1603] bw ( KiB/s): min=16351, max=16351, per=100.00%, avg=16351.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 500=33.33% cpu : usr=0.00%, sys=52.99%, ctx=425, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8628: Thu Jun 20 01:09:23 2024 read: IOPS=0, BW=1541KiB/s (1578kB/s)(4096KiB/2658msec) clat (nsec): min=111804k, max=111804k, avg=111803951.00, stdev= 0.00 lat (nsec): min=111806k, max=111806k, avg=111805535.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 112], 5.00th=[ 112], 10.00th=[ 112], 20.00th=[ 112], | 30.00th=[ 112], 40.00th=[ 112], 50.00th=[ 112], 60.00th=[ 112], | 70.00th=[ 112], 80.00th=[ 112], 90.00th=[ 112], 95.00th=[ 112], | 99.00th=[ 112], 99.50th=[ 112], 99.90th=[ 112], 99.95th=[ 112], | 99.99th=[ 112] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=0, BW=3082KiB/s (3156kB/s)(8192KiB/2658msec) clat (msec): min=167, max=2377, avg=1272.62, stdev=1563.07 lat (msec): min=167, max=2378, avg=1272.81, stdev=1562.99 clat percentiles (msec): | 1.00th=[ 167], 5.00th=[ 167], 10.00th=[ 167], 20.00th=[ 167], | 30.00th=[ 167], 40.00th=[ 167], 50.00th=[ 167], 60.00th=[ 2366], | 70.00th=[ 2366], 80.00th=[ 2366], 90.00th=[ 2366], 95.00th=[ 2366], | 99.00th=[ 2366], 99.50th=[ 2366], 99.90th=[ 2366], 99.95th=[ 2366], | 99.99th=[ 2366] bw ( KiB/s): min= 8192, max= 8192, per=72.95%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67% cpu : usr=0.00%, sys=8.96%, ctx=353, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8629: Thu Jun 20 01:09:23 2024 write: IOPS=1, BW=4211KiB/s (4312kB/s)(12.0MiB/2918msec) clat (msec): min=178, max=2388, avg=971.01, stdev=1230.83 lat (msec): min=179, max=2389, avg=971.23, stdev=1230.90 clat percentiles (msec): | 1.00th=[ 180], 5.00th=[ 180], 10.00th=[ 180], 20.00th=[ 180], | 30.00th=[ 180], 40.00th=[ 347], 50.00th=[ 347], 60.00th=[ 347], | 70.00th=[ 2400], 80.00th=[ 2400], 90.00th=[ 2400], 95.00th=[ 2400], | 99.00th=[ 2400], 99.50th=[ 2400], 99.90th=[ 2400], 99.95th=[ 2400], | 99.99th=[ 2400] bw ( KiB/s): min= 8175, max= 8175, per=72.80%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=33.33% cpu : usr=0.00%, sys=9.15%, ctx=24, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8630: Thu Jun 20 01:09:23 2024 write: IOPS=1, BW=4627KiB/s (4738kB/s)(12.0MiB/2656msec) clat (msec): min=91, max=2374, avg=883.96, stdev=1291.95 lat (msec): min=91, max=2375, avg=884.22, stdev=1291.96 clat percentiles (msec): | 1.00th=[ 91], 5.00th=[ 91], 10.00th=[ 91], 20.00th=[ 91], | 30.00th=[ 91], 40.00th=[ 186], 50.00th=[ 186], 60.00th=[ 186], | 70.00th=[ 2366], 80.00th=[ 2366], 90.00th=[ 2366], 95.00th=[ 2366], | 99.00th=[ 2366], 99.50th=[ 2366], 99.90th=[ 2366], 99.95th=[ 2366], | 99.99th=[ 2366] bw ( KiB/s): min= 8192, max= 8192, per=72.95%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 250=33.33% cpu : usr=0.00%, sys=11.71%, ctx=25, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=6164KiB/s (6312kB/s), 1541KiB/s-6065KiB/s (1578kB/s-6211kB/s), io=16.0MiB (16.8MB), run=2026-2658msec WRITE: bw=10.0MiB/s (11.5MB/s), 3082KiB/s-4627KiB/s (3156kB/s-4738kB/s), io=32.0MiB (33.6MB), run=2656-2918msec