== sanity test 398b: DIO and buffer IO race ============== 10:22:59 (1724768579) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.02788 s, 49.0 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=22588: Tue Aug 27 10:23:15 2024 read: IOPS=111, BW=445KiB/s (455kB/s)(5992KiB/13475msec) clat (usec): min=46, max=155629, avg=4267.00, stdev=5277.75 lat (usec): min=47, max=155630, avg=4267.74, stdev=5277.75 clat percentiles (usec): | 1.00th=[ 58], 5.00th=[ 67], 10.00th=[ 90], 20.00th=[ 2180], | 30.00th=[ 2671], 40.00th=[ 3097], 50.00th=[ 3589], 60.00th=[ 4293], | 70.00th=[ 5080], 80.00th=[ 6325], 90.00th=[ 7832], 95.00th=[ 9372], | 99.00th=[ 14615], 99.50th=[ 25822], 99.90th=[ 46924], 99.95th=[156238], | 99.99th=[156238] bw ( KiB/s): min= 240, max= 654, per=24.51%, avg=436.50, stdev=87.08, samples=26 iops : min= 60, max= 163, avg=109.08, stdev=21.71, samples=26 write: IOPS=116, BW=467KiB/s (478kB/s)(6296KiB/13475msec) clat (usec): min=455, max=37088, avg=4429.47, stdev=4025.00 lat (usec): min=455, max=37088, avg=4430.55, stdev=4024.97 clat percentiles (usec): | 1.00th=[ 502], 5.00th=[ 553], 10.00th=[ 603], 20.00th=[ 807], | 30.00th=[ 1205], 40.00th=[ 2606], 50.00th=[ 3720], 60.00th=[ 4490], | 70.00th=[ 5669], 80.00th=[ 7308], 90.00th=[ 9634], 95.00th=[11994], | 99.00th=[16909], 99.50th=[19006], 99.90th=[31851], 99.95th=[36963], | 99.99th=[36963] bw ( KiB/s): min= 296, max= 614, per=24.49%, avg=456.81, stdev=63.89, samples=26 iops : min= 74, max= 153, avg=114.15, stdev=15.95, samples=26 lat (usec) : 50=0.29%, 100=5.89%, 250=2.18%, 500=0.59%, 750=8.50% lat (usec) : 1000=4.88% lat (msec) : 2=5.66%, 4=27.12%, 10=38.51%, 20=5.83%, 50=0.52% lat (msec) : 250=0.03% cpu : usr=0.27%, sys=24.30%, ctx=6314, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22589: Tue Aug 27 10:23:15 2024 read: IOPS=110, BW=441KiB/s (452kB/s)(5940KiB/13458msec) clat (usec): min=47, max=50858, avg=4038.76, stdev=3307.95 lat (usec): min=47, max=50859, avg=4039.51, stdev=3307.96 clat percentiles (usec): | 1.00th=[ 58], 5.00th=[ 71], 10.00th=[ 91], 20.00th=[ 2147], | 30.00th=[ 2606], 40.00th=[ 2999], 50.00th=[ 3425], 60.00th=[ 4047], | 70.00th=[ 4948], 80.00th=[ 6194], 90.00th=[ 7832], 95.00th=[ 9110], | 99.00th=[15401], 99.50th=[17695], 99.90th=[26346], 99.95th=[51119], | 99.99th=[51119] bw ( KiB/s): min= 232, max= 616, per=24.78%, avg=441.27, stdev=83.89, samples=26 iops : min= 58, max= 154, avg=110.19, stdev=20.97, samples=26 write: IOPS=117, BW=472KiB/s (483kB/s)(6348KiB/13458msec) clat (usec): min=426, max=154557, avg=4630.25, stdev=5674.23 lat (usec): min=427, max=154558, avg=4631.39, stdev=5674.18 clat percentiles (usec): | 1.00th=[ 498], 5.00th=[ 553], 10.00th=[ 603], 20.00th=[ 816], | 30.00th=[ 1221], 40.00th=[ 2573], 50.00th=[ 3752], 60.00th=[ 4621], | 70.00th=[ 5800], 80.00th=[ 7308], 90.00th=[ 9896], 95.00th=[ 12518], | 99.00th=[ 19268], 99.50th=[ 23200], 99.90th=[ 44303], 99.95th=[154141], | 99.99th=[154141] bw ( KiB/s): min= 288, max= 584, per=24.97%, avg=465.62, stdev=61.51, samples=26 iops : min= 72, max= 146, avg=116.27, stdev=15.40, samples=26 lat (usec) : 50=0.13%, 100=5.79%, 250=2.02%, 500=0.81%, 750=7.88% lat (usec) : 1000=5.66% lat (msec) : 2=5.50%, 4=27.99%, 10=37.50%, 20=6.12%, 50=0.52% lat (msec) : 100=0.03%, 250=0.03% cpu : usr=0.28%, sys=24.24%, ctx=6424, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22590: Tue Aug 27 10:23:15 2024 read: IOPS=112, BW=448KiB/s (459kB/s)(6028KiB/13450msec) clat (usec): min=48, max=156321, avg=4355.05, stdev=5188.41 lat (usec): min=48, max=156321, avg=4356.29, stdev=5188.35 clat percentiles (usec): | 1.00th=[ 58], 5.00th=[ 72], 10.00th=[ 92], 20.00th=[ 2212], | 30.00th=[ 2704], 40.00th=[ 3064], 50.00th=[ 3621], 60.00th=[ 4359], | 70.00th=[ 5276], 80.00th=[ 6456], 90.00th=[ 8029], 95.00th=[ 9634], | 99.00th=[ 17433], 99.50th=[ 22938], 99.90th=[ 34341], 99.95th=[156238], | 99.99th=[156238] bw ( KiB/s): min= 264, max= 592, per=25.00%, avg=445.19, stdev=72.03, samples=26 iops : min= 66, max= 148, avg=111.27, stdev=17.99, samples=26 write: IOPS=116, BW=465KiB/s (477kB/s)(6260KiB/13450msec) clat (usec): min=479, max=32431, avg=4328.49, stdev=3912.70 lat (usec): min=480, max=32433, avg=4329.74, stdev=3912.38 clat percentiles (usec): | 1.00th=[ 506], 5.00th=[ 545], 10.00th=[ 594], 20.00th=[ 758], | 30.00th=[ 1020], 40.00th=[ 2311], 50.00th=[ 3589], 60.00th=[ 4555], | 70.00th=[ 5604], 80.00th=[ 7242], 90.00th=[ 9634], 95.00th=[11731], | 99.00th=[16319], 99.50th=[19530], 99.90th=[26346], 99.95th=[32375], | 99.99th=[32375] bw ( KiB/s): min= 328, max= 568, per=24.61%, avg=459.04, stdev=64.63, samples=26 iops : min= 82, max= 142, avg=114.73, stdev=16.18, samples=26 lat (usec) : 50=0.10%, 100=5.50%, 250=2.31%, 500=0.29%, 750=9.83% lat (usec) : 1000=5.11% lat (msec) : 2=4.82%, 4=27.15%, 10=38.12%, 20=6.22%, 50=0.52% lat (msec) : 250=0.03% cpu : usr=0.20%, sys=24.37%, ctx=6441, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22591: Tue Aug 27 10:23:15 2024 read: IOPS=113, BW=453KiB/s (464kB/s)(6052KiB/13355msec) clat (usec): min=49, max=172853, avg=4205.69, stdev=5484.10 lat (usec): min=49, max=172855, avg=4206.39, stdev=5484.11 clat percentiles (usec): | 1.00th=[ 57], 5.00th=[ 69], 10.00th=[ 89], 20.00th=[ 2245], | 30.00th=[ 2671], 40.00th=[ 3032], 50.00th=[ 3490], 60.00th=[ 4080], | 70.00th=[ 5014], 80.00th=[ 6063], 90.00th=[ 7701], 95.00th=[ 9110], | 99.00th=[ 15008], 99.50th=[ 22938], 99.90th=[ 44827], 99.95th=[173016], | 99.99th=[173016] bw ( KiB/s): min= 288, max= 568, per=25.26%, avg=449.81, stdev=58.00, samples=26 iops : min= 72, max= 142, avg=112.42, stdev=14.49, samples=26 write: IOPS=116, BW=467KiB/s (478kB/s)(6236KiB/13355msec) clat (usec): min=449, max=31952, avg=4414.41, stdev=4084.89 lat (usec): min=450, max=31952, avg=4415.92, stdev=4084.37 clat percentiles (usec): | 1.00th=[ 506], 5.00th=[ 545], 10.00th=[ 586], 20.00th=[ 783], | 30.00th=[ 1057], 40.00th=[ 2507], 50.00th=[ 3654], 60.00th=[ 4621], | 70.00th=[ 5669], 80.00th=[ 7177], 90.00th=[ 9634], 95.00th=[11994], | 99.00th=[18482], 99.50th=[22152], 99.90th=[30016], 99.95th=[31851], | 99.99th=[31851] bw ( KiB/s): min= 336, max= 648, per=24.89%, avg=464.27, stdev=68.61, samples=26 iops : min= 84, max= 162, avg=116.04, stdev=17.15, samples=26 lat (usec) : 50=0.20%, 100=5.40%, 250=1.86%, 500=0.55%, 750=8.82% lat (usec) : 1000=5.57% lat (msec) : 2=4.95%, 4=28.84%, 10=37.63%, 20=5.60%, 50=0.55% lat (msec) : 250=0.03% cpu : usr=0.15%, sys=24.69%, ctx=6356, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1782KiB/s (1825kB/s), 441KiB/s-453KiB/s (452kB/s-464kB/s), io=23.4MiB (24.6MB), run=13355-13475msec WRITE: bw=1866KiB/s (1910kB/s), 465KiB/s-472KiB/s (477kB/s-483kB/s), io=24.6MiB (25.7MB), run=13355-13475msec rand-rw: (groupid=0, jobs=1): err= 0: pid=22593: Tue Aug 27 10:23:38 2024 read: IOPS=41, BW=167KiB/s (171kB/s)(5992KiB/35903msec) clat (usec): min=1506, max=159207, avg=3738.40, stdev=5156.64 lat (usec): min=1506, max=159208, avg=3739.25, stdev=5156.65 clat percentiles (usec): | 1.00th=[ 1795], 5.00th=[ 1926], 10.00th=[ 2008], 20.00th=[ 2114], | 30.00th=[ 2212], 40.00th=[ 2311], 50.00th=[ 2442], 60.00th=[ 2638], | 70.00th=[ 2933], 80.00th=[ 4080], 90.00th=[ 7373], 95.00th=[ 9634], | 99.00th=[ 16450], 99.50th=[ 23725], 99.90th=[ 38011], 99.95th=[158335], | 99.99th=[158335] bw ( KiB/s): min= 64, max= 304, per=25.10%, avg=166.93, stdev=64.21, samples=71 iops : min= 16, max= 76, avg=41.69, stdev=16.07, samples=71 write: IOPS=43, BW=175KiB/s (180kB/s)(6296KiB/35903msec) clat (usec): min=9884, max=54410, avg=19211.32, stdev=6297.12 lat (usec): min=9885, max=54411, avg=19212.49, stdev=6297.10 clat percentiles (usec): | 1.00th=[10814], 5.00th=[11731], 10.00th=[12518], 20.00th=[13566], | 30.00th=[14877], 40.00th=[16909], 50.00th=[18482], 60.00th=[19792], | 70.00th=[21103], 80.00th=[23200], 90.00th=[27919], 95.00th=[31065], | 99.00th=[39584], 99.50th=[41681], 99.90th=[54264], 99.95th=[54264], | 99.99th=[54264] bw ( KiB/s): min= 72, max= 240, per=25.18%, avg=175.27, stdev=45.74, samples=71 iops : min= 18, max= 60, avg=43.77, stdev=11.45, samples=71 lat (msec) : 2=4.65%, 4=34.08%, 10=7.91%, 20=33.43%, 50=19.82% lat (msec) : 100=0.07%, 250=0.03% cpu : usr=0.14%, sys=5.68%, ctx=3525, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22594: Tue Aug 27 10:23:38 2024 read: IOPS=41, BW=166KiB/s (170kB/s)(5940KiB/35881msec) clat (usec): min=1695, max=153924, avg=3667.48, stdev=4903.71 lat (usec): min=1696, max=153926, avg=3668.46, stdev=4903.74 clat percentiles (usec): | 1.00th=[ 1827], 5.00th=[ 1926], 10.00th=[ 2008], 20.00th=[ 2114], | 30.00th=[ 2212], 40.00th=[ 2343], 50.00th=[ 2442], 60.00th=[ 2638], | 70.00th=[ 2966], 80.00th=[ 4015], 90.00th=[ 7177], 95.00th=[ 9765], | 99.00th=[ 14615], 99.50th=[ 16909], 99.90th=[ 40109], 99.95th=[154141], | 99.99th=[154141] bw ( KiB/s): min= 39, max= 328, per=24.98%, avg=166.11, stdev=69.29, samples=71 iops : min= 9, max= 82, avg=41.46, stdev=17.34, samples=71 write: IOPS=44, BW=177KiB/s (181kB/s)(6348KiB/35881msec) clat (usec): min=10186, max=54201, avg=19119.91, stdev=6307.37 lat (usec): min=10188, max=54202, avg=19121.00, stdev=6307.32 clat percentiles (usec): | 1.00th=[10945], 5.00th=[11863], 10.00th=[12387], 20.00th=[13566], | 30.00th=[14877], 40.00th=[16712], 50.00th=[18482], 60.00th=[19530], | 70.00th=[21103], 80.00th=[22938], 90.00th=[27657], 95.00th=[31065], | 99.00th=[40109], 99.50th=[45351], 99.90th=[53740], 99.95th=[54264], | 99.99th=[54264] bw ( KiB/s): min= 87, max= 240, per=25.40%, avg=176.82, stdev=44.45, samples=71 iops : min= 21, max= 60, avg=44.14, stdev=11.14, samples=71 lat (msec) : 2=4.52%, 4=34.02%, 10=7.49%, 20=34.44%, 50=19.37% lat (msec) : 100=0.13%, 250=0.03% cpu : usr=0.11%, sys=5.72%, ctx=3481, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22595: Tue Aug 27 10:23:38 2024 read: IOPS=41, BW=167KiB/s (171kB/s)(6028KiB/36086msec) clat (usec): min=1610, max=37665, avg=3833.91, stdev=3299.24 lat (usec): min=1611, max=37665, avg=3834.85, stdev=3299.24 clat percentiles (usec): | 1.00th=[ 1844], 5.00th=[ 1958], 10.00th=[ 2024], 20.00th=[ 2147], | 30.00th=[ 2245], 40.00th=[ 2343], 50.00th=[ 2474], 60.00th=[ 2671], | 70.00th=[ 3163], 80.00th=[ 5080], 90.00th=[ 7701], 95.00th=[10159], | 99.00th=[16450], 99.50th=[19530], 99.90th=[33817], 99.95th=[37487], | 99.99th=[37487] bw ( KiB/s): min= 55, max= 296, per=25.01%, avg=166.32, stdev=57.87, samples=71 iops : min= 13, max= 74, avg=41.51, stdev=14.50, samples=71 write: IOPS=43, BW=173KiB/s (178kB/s)(6260KiB/36086msec) clat (msec): min=9, max=180, avg=19.32, stdev= 7.52 lat (msec): min=9, max=180, avg=19.33, stdev= 7.52 clat percentiles (msec): | 1.00th=[ 11], 5.00th=[ 12], 10.00th=[ 13], 20.00th=[ 14], | 30.00th=[ 16], 40.00th=[ 17], 50.00th=[ 19], 60.00th=[ 20], | 70.00th=[ 22], 80.00th=[ 24], 90.00th=[ 28], 95.00th=[ 32], | 99.00th=[ 40], 99.50th=[ 46], 99.90th=[ 55], 99.95th=[ 182], | 99.99th=[ 182] bw ( KiB/s): min= 88, max= 240, per=24.79%, avg=172.52, stdev=46.09, samples=71 iops : min= 22, max= 60, avg=43.06, stdev=11.54, samples=71 lat (msec) : 2=4.10%, 4=33.17%, 10=9.34%, 20=33.89%, 50=19.34% lat (msec) : 100=0.13%, 250=0.03% cpu : usr=0.13%, sys=5.66%, ctx=3583, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22596: Tue Aug 27 10:23:38 2024 read: IOPS=42, BW=169KiB/s (173kB/s)(6052KiB/35736msec) clat (usec): min=1634, max=52296, avg=3727.66, stdev=3583.62 lat (usec): min=1634, max=52297, avg=3728.63, stdev=3583.97 clat percentiles (usec): | 1.00th=[ 1795], 5.00th=[ 1942], 10.00th=[ 2008], 20.00th=[ 2114], | 30.00th=[ 2212], 40.00th=[ 2311], 50.00th=[ 2442], 60.00th=[ 2606], | 70.00th=[ 2966], 80.00th=[ 4146], 90.00th=[ 7177], 95.00th=[10028], | 99.00th=[16909], 99.50th=[25035], 99.90th=[38011], 99.95th=[52167], | 99.99th=[52167] bw ( KiB/s): min= 56, max= 320, per=25.51%, avg=169.66, stdev=62.83, samples=71 iops : min= 14, max= 80, avg=42.39, stdev=15.71, samples=71 write: IOPS=43, BW=175KiB/s (179kB/s)(6236KiB/35736msec) clat (msec): min=9, max=178, avg=19.26, stdev= 7.29 lat (msec): min=9, max=178, avg=19.26, stdev= 7.29 clat percentiles (msec): | 1.00th=[ 11], 5.00th=[ 12], 10.00th=[ 13], 20.00th=[ 14], | 30.00th=[ 16], 40.00th=[ 17], 50.00th=[ 19], 60.00th=[ 20], | 70.00th=[ 21], 80.00th=[ 23], 90.00th=[ 28], 95.00th=[ 32], | 99.00th=[ 40], 99.50th=[ 42], 99.90th=[ 60], 99.95th=[ 180], | 99.99th=[ 180] bw ( KiB/s): min= 80, max= 232, per=25.07%, avg=174.51, stdev=45.11, samples=71 iops : min= 20, max= 58, avg=43.61, stdev=11.28, samples=71 lat (msec) : 2=4.62%, 4=34.41%, 10=7.78%, 20=34.08%, 50=18.98% lat (msec) : 100=0.10%, 250=0.03% cpu : usr=0.10%, sys=5.77%, ctx=3535, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=665KiB/s (681kB/s), 166KiB/s-169KiB/s (170kB/s-173kB/s), io=23.4MiB (24.6MB), run=35736-36086msec WRITE: bw=697KiB/s (713kB/s), 173KiB/s-177KiB/s (178kB/s-181kB/s), io=24.6MiB (25.7MB), run=35736-36086msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=22634: Tue Aug 27 10:23:44 2024 read: IOPS=74, BW=1199KiB/s (1227kB/s)(5920KiB/4939msec) clat (usec): min=51, max=27696, avg=9425.28, stdev=6318.74 lat (usec): min=51, max=27697, avg=9426.06, stdev=6318.74 clat percentiles (usec): | 1.00th=[ 56], 5.00th=[ 65], 10.00th=[ 79], 20.00th=[ 105], | 30.00th=[ 8586], 40.00th=[ 9896], 50.00th=[11207], 60.00th=[12125], | 70.00th=[13042], 80.00th=[14353], 90.00th=[16057], 95.00th=[17695], | 99.00th=[22938], 99.50th=[26346], 99.90th=[27657], 99.95th=[27657], | 99.99th=[27657] bw ( KiB/s): min= 864, max= 1568, per=24.97%, avg=1154.67, stdev=242.67, samples=9 iops : min= 54, max= 98, avg=71.89, stdev=15.29, samples=9 write: IOPS=80, BW=1289KiB/s (1320kB/s)(6368KiB/4939msec) clat (usec): min=648, max=24298, avg=3447.69, stdev=3521.53 lat (usec): min=649, max=24300, avg=3448.93, stdev=3521.54 clat percentiles (usec): | 1.00th=[ 717], 5.00th=[ 807], 10.00th=[ 865], 20.00th=[ 996], | 30.00th=[ 1172], 40.00th=[ 1319], 50.00th=[ 1614], 60.00th=[ 3294], | 70.00th=[ 4113], 80.00th=[ 5407], 90.00th=[ 7832], 95.00th=[10683], | 99.00th=[17433], 99.50th=[19792], 99.90th=[24249], 99.95th=[24249], | 99.99th=[24249] bw ( KiB/s): min= 830, max= 1984, per=25.86%, avg=1257.89, stdev=391.25, samples=9 iops : min= 51, max= 124, avg=78.33, stdev=24.70, samples=9 lat (usec) : 100=8.33%, 250=4.56%, 750=1.30%, 1000=9.37% lat (msec) : 2=17.19%, 4=7.94%, 10=19.92%, 20=30.21%, 50=1.17% cpu : usr=0.26%, sys=23.09%, ctx=1964, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22635: Tue Aug 27 10:23:44 2024 read: IOPS=72, BW=1161KiB/s (1189kB/s)(5744KiB/4947msec) clat (usec): min=51, max=39551, avg=9895.92, stdev=6495.11 lat (usec): min=52, max=39552, avg=9897.04, stdev=6495.23 clat percentiles (usec): | 1.00th=[ 57], 5.00th=[ 66], 10.00th=[ 85], 20.00th=[ 121], | 30.00th=[ 9241], 40.00th=[10421], 50.00th=[11338], 60.00th=[12256], | 70.00th=[13304], 80.00th=[14615], 90.00th=[16712], 95.00th=[19006], | 99.00th=[22414], 99.50th=[25560], 99.90th=[39584], 99.95th=[39584], | 99.99th=[39584] bw ( KiB/s): min= 896, max= 1440, per=24.83%, avg=1148.11, stdev=236.17, samples=9 iops : min= 56, max= 90, avg=71.67, stdev=14.68, samples=9 write: IOPS=82, BW=1323KiB/s (1355kB/s)(6544KiB/4947msec) clat (usec): min=668, max=29174, avg=3372.69, stdev=3715.11 lat (usec): min=669, max=29175, avg=3374.73, stdev=3714.60 clat percentiles (usec): | 1.00th=[ 717], 5.00th=[ 791], 10.00th=[ 889], 20.00th=[ 1037], | 30.00th=[ 1172], 40.00th=[ 1352], 50.00th=[ 1811], 60.00th=[ 2966], | 70.00th=[ 3851], 80.00th=[ 4883], 90.00th=[ 7177], 95.00th=[ 9765], | 99.00th=[20055], 99.50th=[21890], 99.90th=[29230], 99.95th=[29230], | 99.99th=[29230] bw ( KiB/s): min= 1056, max= 1760, per=26.67%, avg=1297.44, stdev=209.56, samples=9 iops : min= 66, max= 110, avg=81.00, stdev=13.17, samples=9 lat (usec) : 100=7.03%, 250=4.17%, 500=0.26%, 750=1.69%, 1000=7.42% lat (msec) : 2=18.88%, 4=10.81%, 10=17.58%, 20=29.82%, 50=2.34% cpu : usr=0.16%, sys=22.99%, ctx=2064, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22636: Tue Aug 27 10:23:44 2024 read: IOPS=75, BW=1208KiB/s (1237kB/s)(6256KiB/5179msec) clat (usec): min=51, max=36024, avg=9812.36, stdev=6182.78 lat (usec): min=52, max=36025, avg=9813.11, stdev=6182.79 clat percentiles (usec): | 1.00th=[ 53], 5.00th=[ 71], 10.00th=[ 93], 20.00th=[ 130], | 30.00th=[ 9110], 40.00th=[10290], 50.00th=[11338], 60.00th=[12387], | 70.00th=[13173], 80.00th=[14353], 90.00th=[15926], 95.00th=[17433], | 99.00th=[23725], 99.50th=[28967], 99.90th=[35914], 99.95th=[35914], | 99.99th=[35914] bw ( KiB/s): min= 768, max= 1568, per=25.95%, avg=1200.00, stdev=237.20, samples=10 iops : min= 48, max= 98, avg=75.00, stdev=14.82, samples=10 write: IOPS=72, BW=1165KiB/s (1193kB/s)(6032KiB/5179msec) clat (usec): min=662, max=25317, avg=3386.00, stdev=3466.84 lat (usec): min=663, max=25318, avg=3387.49, stdev=3467.01 clat percentiles (usec): | 1.00th=[ 701], 5.00th=[ 799], 10.00th=[ 857], 20.00th=[ 1004], | 30.00th=[ 1172], 40.00th=[ 1303], 50.00th=[ 1762], 60.00th=[ 2966], | 70.00th=[ 3949], 80.00th=[ 5473], 90.00th=[ 7963], 95.00th=[ 9765], | 99.00th=[17695], 99.50th=[23725], 99.90th=[25297], 99.95th=[25297], | 99.99th=[25297] bw ( KiB/s): min= 768, max= 1632, per=23.22%, avg=1129.60, stdev=272.39, samples=10 iops : min= 48, max= 102, avg=70.60, stdev=17.02, samples=10 lat (usec) : 100=6.64%, 250=4.82%, 500=0.52%, 750=1.17%, 1000=8.59% lat (msec) : 2=15.23%, 4=9.64%, 10=19.01%, 20=32.94%, 50=1.43% cpu : usr=0.31%, sys=22.05%, ctx=2168, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22638: Tue Aug 27 10:23:44 2024 read: IOPS=75, BW=1203KiB/s (1232kB/s)(6032KiB/5015msec) clat (usec): min=52, max=32263, avg=9492.08, stdev=6263.82 lat (usec): min=52, max=32264, avg=9492.96, stdev=6263.81 clat percentiles (usec): | 1.00th=[ 60], 5.00th=[ 71], 10.00th=[ 81], 20.00th=[ 110], | 30.00th=[ 9110], 40.00th=[10159], 50.00th=[11207], 60.00th=[12387], | 70.00th=[13304], 80.00th=[14222], 90.00th=[16188], 95.00th=[17695], | 99.00th=[20055], 99.50th=[25297], 99.90th=[32375], 99.95th=[32375], | 99.99th=[32375] bw ( KiB/s): min= 928, max= 1472, per=25.81%, avg=1193.60, stdev=207.96, samples=10 iops : min= 58, max= 92, avg=74.60, stdev=13.00, samples=10 write: IOPS=77, BW=1247KiB/s (1277kB/s)(6256KiB/5015msec) clat (usec): min=661, max=20503, avg=3560.04, stdev=3552.14 lat (usec): min=662, max=20505, avg=3561.49, stdev=3552.00 clat percentiles (usec): | 1.00th=[ 685], 5.00th=[ 783], 10.00th=[ 857], 20.00th=[ 963], | 30.00th=[ 1106], 40.00th=[ 1254], 50.00th=[ 1582], 60.00th=[ 3294], | 70.00th=[ 4359], 80.00th=[ 6259], 90.00th=[ 8225], 95.00th=[10814], | 99.00th=[16909], 99.50th=[20317], 99.90th=[20579], 99.95th=[20579], | 99.99th=[20579] bw ( KiB/s): min= 864, max= 1664, per=25.65%, avg=1248.00, stdev=265.17, samples=10 iops : min= 54, max= 104, avg=78.00, stdev=16.57, samples=10 lat (usec) : 100=8.72%, 250=4.04%, 500=0.13%, 750=1.17%, 1000=10.55% lat (msec) : 2=15.36%, 4=6.90%, 10=19.92%, 20=32.42%, 50=0.78% cpu : usr=0.16%, sys=22.24%, ctx=2026, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=4625KiB/s (4736kB/s), 1161KiB/s-1208KiB/s (1189kB/s-1237kB/s), io=23.4MiB (24.5MB), run=4939-5179msec WRITE: bw=4866KiB/s (4983kB/s), 1165KiB/s-1323KiB/s (1193kB/s-1355kB/s), io=24.6MiB (25.8MB), run=4939-5179msec rand-rw: (groupid=0, jobs=1): err= 0: pid=22637: Tue Aug 27 10:23:49 2024 read: IOPS=36, BW=577KiB/s (591kB/s)(5920KiB/10252msec) clat (usec): min=1797, max=36078, avg=4380.55, stdev=3984.97 lat (usec): min=1797, max=36079, avg=4381.43, stdev=3984.95 clat percentiles (usec): | 1.00th=[ 1975], 5.00th=[ 2089], 10.00th=[ 2212], 20.00th=[ 2409], | 30.00th=[ 2507], 40.00th=[ 2638], 50.00th=[ 2900], 60.00th=[ 3195], | 70.00th=[ 3687], 80.00th=[ 5145], 90.00th=[ 9110], 95.00th=[12256], | 99.00th=[25035], 99.50th=[30016], 99.90th=[35914], 99.95th=[35914], | 99.99th=[35914] bw ( KiB/s): min= 352, max= 1024, per=24.11%, avg=563.15, stdev=182.66, samples=20 iops : min= 22, max= 64, avg=35.15, stdev=11.46, samples=20 write: IOPS=38, BW=621KiB/s (636kB/s)(6368KiB/10252msec) clat (usec): min=11511, max=48216, avg=21491.88, stdev=6709.71 lat (usec): min=11513, max=48218, avg=21493.60, stdev=6709.79 clat percentiles (usec): | 1.00th=[12125], 5.00th=[13042], 10.00th=[14091], 20.00th=[15533], | 30.00th=[17433], 40.00th=[19268], 50.00th=[20579], 60.00th=[22152], | 70.00th=[23462], 80.00th=[25822], 90.00th=[30278], 95.00th=[34341], | 99.00th=[44827], 99.50th=[46924], 99.90th=[47973], 99.95th=[47973], | 99.99th=[47973] bw ( KiB/s): min= 384, max= 832, per=24.99%, avg=614.30, stdev=159.43, samples=20 iops : min= 24, max= 52, avg=38.35, stdev=10.00, samples=20 lat (msec) : 2=1.04%, 4=33.85%, 10=9.37%, 20=25.91%, 50=29.82% cpu : usr=0.10%, sys=5.99%, ctx=876, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22639: Tue Aug 27 10:23:49 2024 read: IOPS=35, BW=563KiB/s (577kB/s)(5744KiB/10200msec) clat (usec): min=1805, max=24919, avg=4300.39, stdev=3517.39 lat (usec): min=1805, max=24920, avg=4301.23, stdev=3517.38 clat percentiles (usec): | 1.00th=[ 2024], 5.00th=[ 2114], 10.00th=[ 2245], 20.00th=[ 2376], | 30.00th=[ 2507], 40.00th=[ 2638], 50.00th=[ 2835], 60.00th=[ 3130], | 70.00th=[ 3621], 80.00th=[ 5473], 90.00th=[ 8160], 95.00th=[12911], | 99.00th=[17957], 99.50th=[23462], 99.90th=[25035], 99.95th=[25035], | 99.99th=[25035] bw ( KiB/s): min= 191, max= 1024, per=23.96%, avg=559.75, stdev=224.55, samples=20 iops : min= 11, max= 64, avg=34.85, stdev=14.11, samples=20 write: IOPS=40, BW=642KiB/s (657kB/s)(6544KiB/10200msec) clat (usec): min=11526, max=48636, avg=20940.01, stdev=6646.88 lat (usec): min=11528, max=48638, avg=20941.69, stdev=6646.91 clat percentiles (usec): | 1.00th=[11863], 5.00th=[12911], 10.00th=[13698], 20.00th=[15008], | 30.00th=[16581], 40.00th=[18220], 50.00th=[20055], 60.00th=[21627], | 70.00th=[22938], 80.00th=[25560], 90.00th=[29754], 95.00th=[33817], | 99.00th=[43254], 99.50th=[44827], 99.90th=[48497], 99.95th=[48497], | 99.99th=[48497] bw ( KiB/s): min= 415, max= 832, per=25.77%, avg=633.40, stdev=144.32, samples=20 iops : min= 25, max= 52, avg=39.45, stdev= 9.20, samples=20 lat (msec) : 2=0.26%, 4=33.98%, 10=8.85%, 20=30.21%, 50=26.69% cpu : usr=0.13%, sys=5.89%, ctx=878, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22640: Tue Aug 27 10:23:49 2024 read: IOPS=39, BW=635KiB/s (650kB/s)(6256KiB/9849msec) clat (usec): min=1913, max=36127, avg=4577.38, stdev=4032.08 lat (usec): min=1914, max=36128, avg=4578.26, stdev=4032.07 clat percentiles (usec): | 1.00th=[ 2008], 5.00th=[ 2147], 10.00th=[ 2245], 20.00th=[ 2409], | 30.00th=[ 2540], 40.00th=[ 2704], 50.00th=[ 2900], 60.00th=[ 3228], | 70.00th=[ 3916], 80.00th=[ 6194], 90.00th=[ 9241], 95.00th=[12387], | 99.00th=[20579], 99.50th=[28705], 99.90th=[35914], 99.95th=[35914], | 99.99th=[35914] bw ( KiB/s): min= 352, max= 1245, per=27.39%, avg=639.74, stdev=235.76, samples=19 iops : min= 22, max= 77, avg=39.89, stdev=14.65, samples=19 write: IOPS=38, BW=612KiB/s (627kB/s)(6032KiB/9849msec) clat (usec): min=10376, max=48181, avg=21325.93, stdev=6515.24 lat (usec): min=10377, max=48183, avg=21327.66, stdev=6515.33 clat percentiles (usec): | 1.00th=[12387], 5.00th=[13304], 10.00th=[14091], 20.00th=[15664], | 30.00th=[17433], 40.00th=[19006], 50.00th=[20579], 60.00th=[21627], | 70.00th=[23200], 80.00th=[24773], 90.00th=[30802], 95.00th=[35914], | 99.00th=[41157], 99.50th=[42730], 99.90th=[47973], 99.95th=[47973], | 99.99th=[47973] bw ( KiB/s): min= 320, max= 800, per=24.66%, avg=606.11, stdev=152.18, samples=19 iops : min= 20, max= 50, avg=37.79, stdev= 9.53, samples=19 lat (msec) : 2=0.26%, 4=35.94%, 10=9.90%, 20=27.34%, 50=26.56% cpu : usr=0.10%, sys=6.26%, ctx=888, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22641: Tue Aug 27 10:23:49 2024 read: IOPS=37, BW=600KiB/s (614kB/s)(6032KiB/10055msec) clat (usec): min=1915, max=28950, avg=4334.18, stdev=3776.34 lat (usec): min=1916, max=28951, avg=4335.04, stdev=3776.30 clat percentiles (usec): | 1.00th=[ 2024], 5.00th=[ 2147], 10.00th=[ 2245], 20.00th=[ 2409], | 30.00th=[ 2507], 40.00th=[ 2671], 50.00th=[ 2835], 60.00th=[ 3228], | 70.00th=[ 3785], 80.00th=[ 5080], 90.00th=[ 7832], 95.00th=[13698], | 99.00th=[19792], 99.50th=[24511], 99.90th=[28967], 99.95th=[28967], | 99.99th=[28967] bw ( KiB/s): min= 288, max= 1056, per=25.40%, avg=593.45, stdev=212.90, samples=20 iops : min= 18, max= 66, avg=37.00, stdev=13.33, samples=20 write: IOPS=38, BW=622KiB/s (637kB/s)(6256KiB/10055msec) clat (usec): min=11822, max=47910, avg=21368.98, stdev=6070.34 lat (usec): min=11823, max=47911, avg=21370.71, stdev=6070.42 clat percentiles (usec): | 1.00th=[12649], 5.00th=[13173], 10.00th=[14222], 20.00th=[15795], | 30.00th=[17957], 40.00th=[19268], 50.00th=[20579], 60.00th=[22152], | 70.00th=[23462], 80.00th=[25560], 90.00th=[28705], 95.00th=[32113], | 99.00th=[40633], 99.50th=[44827], 99.90th=[47973], 99.95th=[47973], | 99.99th=[47973] bw ( KiB/s): min= 416, max= 832, per=25.25%, avg=620.70, stdev=150.00, samples=20 iops : min= 26, max= 52, avg=38.70, stdev= 9.49, samples=20 lat (msec) : 2=0.26%, 4=35.42%, 10=9.90%, 20=26.17%, 50=28.26% cpu : usr=0.09%, sys=6.05%, ctx=867, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2336KiB/s (2392kB/s), 563KiB/s-635KiB/s (577kB/s-650kB/s), io=23.4MiB (24.5MB), run=9849-10252msec WRITE: bw=2458KiB/s (2517kB/s), 612KiB/s-642KiB/s (627kB/s-657kB/s), io=24.6MiB (25.8MB), run=9849-10252msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=22660: Tue Aug 27 10:23:53 2024 read: IOPS=2, BW=2491KiB/s (2551kB/s)(7168KiB/2877msec) clat (msec): min=38, max=561, avg=375.57, stdev=215.82 lat (msec): min=38, max=561, avg=375.57, stdev=215.82 clat percentiles (msec): | 1.00th=[ 39], 5.00th=[ 39], 10.00th=[ 39], 20.00th=[ 92], | 30.00th=[ 456], 40.00th=[ 456], 50.00th=[ 464], 60.00th=[ 498], | 70.00th=[ 498], 80.00th=[ 523], 90.00th=[ 558], 95.00th=[ 558], | 99.00th=[ 558], 99.50th=[ 558], 99.90th=[ 558], 99.95th=[ 558], | 99.99th=[ 558] bw ( KiB/s): min= 1700, max= 3407, per=31.71%, avg=2249.20, stdev=664.38, samples=5 iops : min= 1, max= 3, avg= 1.80, stdev= 0.84, samples=5 write: IOPS=1, BW=1780KiB/s (1822kB/s)(5120KiB/2877msec) clat (usec): min=21989, max=61830, avg=45545.56, stdev=16094.42 lat (usec): min=22021, max=61850, avg=45598.18, stdev=16096.44 clat percentiles (usec): | 1.00th=[21890], 5.00th=[21890], 10.00th=[21890], 20.00th=[21890], | 30.00th=[36963], 40.00th=[36963], 50.00th=[50594], 60.00th=[50594], | 70.00th=[56361], 80.00th=[56361], 90.00th=[61604], 95.00th=[61604], | 99.00th=[61604], 99.50th=[61604], 99.90th=[61604], 99.95th=[61604], | 99.99th=[61604] bw ( KiB/s): min= 1700, max= 5111, per=29.74%, avg=2953.00, stdev=1876.97, samples=3 iops : min= 1, max= 4, avg= 2.33, stdev= 1.53, samples=3 lat (msec) : 50=25.00%, 100=33.33%, 500=25.00%, 750=16.67% cpu : usr=0.03%, sys=19.40%, ctx=1410, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22661: Tue Aug 27 10:23:53 2024 read: IOPS=1, BW=1125KiB/s (1152kB/s)(2048KiB/1821msec) clat (msec): min=36, max=371, avg=203.79, stdev=236.88 lat (msec): min=36, max=371, avg=203.79, stdev=236.88 clat percentiles (msec): | 1.00th=[ 37], 5.00th=[ 37], 10.00th=[ 37], 20.00th=[ 37], | 30.00th=[ 37], 40.00th=[ 37], 50.00th=[ 37], 60.00th=[ 372], | 70.00th=[ 372], 80.00th=[ 372], 90.00th=[ 372], 95.00th=[ 372], | 99.00th=[ 372], 99.50th=[ 372], 99.90th=[ 372], 99.95th=[ 372], | 99.99th=[ 372] bw ( KiB/s): min= 2048, max= 2048, per=28.87%, avg=2048.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 write: IOPS=5, BW=5623KiB/s (5758kB/s)(10.0MiB/1821msec) clat (msec): min=25, max=1044, avg=140.02, stdev=317.95 lat (msec): min=25, max=1044, avg=140.09, stdev=317.93 clat percentiles (msec): | 1.00th=[ 26], 5.00th=[ 26], 10.00th=[ 26], 20.00th=[ 27], | 30.00th=[ 28], 40.00th=[ 30], 50.00th=[ 33], 60.00th=[ 45], | 70.00th=[ 59], 80.00th=[ 59], 90.00th=[ 62], 95.00th=[ 1045], | 99.00th=[ 1045], 99.50th=[ 1045], 99.90th=[ 1045], 99.95th=[ 1045], | 99.99th=[ 1045] bw ( KiB/s): min=10240, max=10240, per=100.00%, avg=10240.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 lat (msec) : 50=66.67%, 100=16.67%, 500=8.33% cpu : usr=0.05%, sys=16.15%, ctx=302, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22662: Tue Aug 27 10:23:53 2024 read: IOPS=2, BW=2128KiB/s (2179kB/s)(6144KiB/2887msec) clat (usec): min=423, max=521197, avg=168098.94, stdev=243383.21 lat (usec): min=424, max=521198, avg=168100.19, stdev=243383.21 clat percentiles (usec): | 1.00th=[ 424], 5.00th=[ 424], 10.00th=[ 424], 20.00th=[ 437], | 30.00th=[ 437], 40.00th=[ 465], 50.00th=[ 465], 60.00th=[ 47973], | 70.00th=[438305], 80.00th=[438305], 90.00th=[522191], 95.00th=[522191], | 99.00th=[522191], 99.50th=[522191], 99.90th=[522191], 99.95th=[522191], | 99.99th=[522191] bw ( KiB/s): min= 4096, max= 6144, per=72.18%, avg=5120.00, stdev=1448.15, samples=2 iops : min= 4, max= 6, avg= 5.00, stdev= 1.41, samples=2 write: IOPS=2, BW=2128KiB/s (2179kB/s)(6144KiB/2887msec) clat (msec): min=17, max=1048, avg=311.37, stdev=418.60 lat (msec): min=17, max=1048, avg=311.41, stdev=418.60 clat percentiles (msec): | 1.00th=[ 18], 5.00th=[ 18], 10.00th=[ 18], 20.00th=[ 46], | 30.00th=[ 46], 40.00th=[ 82], 50.00th=[ 82], 60.00th=[ 93], | 70.00th=[ 584], 80.00th=[ 584], 90.00th=[ 1053], 95.00th=[ 1053], | 99.00th=[ 1053], 99.50th=[ 1053], 99.90th=[ 1053], 99.95th=[ 1053], | 99.99th=[ 1053] bw ( KiB/s): min= 2048, max= 4096, per=30.93%, avg=3072.00, stdev=1182.41, samples=4 iops : min= 2, max= 4, avg= 3.00, stdev= 1.15, samples=4 lat (usec) : 500=25.00% lat (msec) : 20=8.33%, 50=16.67%, 100=16.67%, 500=8.33%, 750=16.67% cpu : usr=0.03%, sys=10.88%, ctx=608, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22663: Tue Aug 27 10:23:53 2024 read: IOPS=2, BW=2150KiB/s (2202kB/s)(5120KiB/2381msec) clat (usec): min=526, max=457648, avg=189488.09, stdev=240127.01 lat (usec): min=527, max=457649, avg=189489.55, stdev=240127.05 clat percentiles (usec): | 1.00th=[ 529], 5.00th=[ 529], 10.00th=[ 529], 20.00th=[ 529], | 30.00th=[ 7635], 40.00th=[ 7635], 50.00th=[ 34866], 60.00th=[ 34866], | 70.00th=[446694], 80.00th=[446694], 90.00th=[459277], 95.00th=[459277], | 99.00th=[459277], 99.50th=[459277], 99.90th=[459277], 99.95th=[459277], | 99.99th=[459277] bw ( KiB/s): min= 2048, max= 6144, per=57.75%, avg=4096.00, stdev=2896.31, samples=2 iops : min= 2, max= 6, avg= 4.00, stdev= 2.83, samples=2 write: IOPS=2, BW=3010KiB/s (3083kB/s)(7168KiB/2381msec) clat (msec): min=25, max=1019, avg=199.91, stdev=363.61 lat (msec): min=25, max=1019, avg=199.97, stdev=363.62 clat percentiles (msec): | 1.00th=[ 26], 5.00th=[ 26], 10.00th=[ 26], 20.00th=[ 27], | 30.00th=[ 47], 40.00th=[ 47], 50.00th=[ 50], 60.00th=[ 109], | 70.00th=[ 109], 80.00th=[ 124], 90.00th=[ 1020], 95.00th=[ 1020], | 99.00th=[ 1020], 99.50th=[ 1020], 99.90th=[ 1020], 99.95th=[ 1020], | 99.99th=[ 1020] bw ( KiB/s): min= 2048, max= 6144, per=48.12%, avg=4778.67, stdev=2364.83, samples=3 iops : min= 2, max= 6, avg= 4.67, stdev= 2.31, samples=3 lat (usec) : 750=8.33% lat (msec) : 10=8.33%, 50=41.67%, 250=16.67%, 500=16.67% cpu : usr=0.00%, sys=17.48%, ctx=580, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=7094KiB/s (7264kB/s), 1125KiB/s-2491KiB/s (1152kB/s-2551kB/s), io=20.0MiB (20.0MB), run=1821-2887msec WRITE: bw=9931KiB/s (10.2MB/s), 1780KiB/s-5623KiB/s (1822kB/s-5758kB/s), io=28.0MiB (29.4MB), run=1821-2887msec rand-rw: (groupid=0, jobs=1): err= 0: pid=22655: Tue Aug 27 10:23:53 2024 read: IOPS=5, BW=5762KiB/s (5900kB/s)(7168KiB/1244msec) clat (msec): min=13, max=545, avg=94.79, stdev=198.75 lat (msec): min=13, max=545, avg=94.79, stdev=198.75 clat percentiles (msec): | 1.00th=[ 14], 5.00th=[ 14], 10.00th=[ 14], 20.00th=[ 16], | 30.00th=[ 17], 40.00th=[ 17], 50.00th=[ 22], 60.00th=[ 26], | 70.00th=[ 26], 80.00th=[ 27], 90.00th=[ 550], 95.00th=[ 550], | 99.00th=[ 550], 99.50th=[ 550], 99.90th=[ 550], 99.95th=[ 550], | 99.99th=[ 550] bw ( KiB/s): min= 4096, max= 6144, per=73.41%, avg=5120.00, stdev=1448.15, samples=2 iops : min= 4, max= 6, avg= 5.00, stdev= 1.41, samples=2 write: IOPS=4, BW=4116KiB/s (4215kB/s)(5120KiB/1244msec) clat (msec): min=25, max=426, avg=114.79, stdev=175.12 lat (msec): min=25, max=426, avg=114.88, stdev=175.18 clat percentiles (msec): | 1.00th=[ 26], 5.00th=[ 26], 10.00th=[ 26], 20.00th=[ 26], | 30.00th=[ 27], 40.00th=[ 27], 50.00th=[ 30], 60.00th=[ 30], | 70.00th=[ 66], 80.00th=[ 66], 90.00th=[ 426], 95.00th=[ 426], | 99.00th=[ 426], 99.50th=[ 426], 99.90th=[ 426], 99.95th=[ 426], | 99.99th=[ 426] bw ( KiB/s): min= 2048, max= 4096, per=31.46%, avg=3072.00, stdev=1448.15, samples=2 iops : min= 2, max= 4, avg= 3.00, stdev= 1.41, samples=2 lat (msec) : 20=25.00%, 50=50.00%, 100=8.33%, 500=8.33%, 750=8.33% cpu : usr=0.08%, sys=5.79%, ctx=50, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22656: Tue Aug 27 10:23:53 2024 read: IOPS=0, BW=820KiB/s (839kB/s)(2048KiB/2499msec) clat (msec): min=14, max=1038, avg=526.72, stdev=724.47 lat (msec): min=14, max=1038, avg=526.72, stdev=724.47 clat percentiles (msec): | 1.00th=[ 15], 5.00th=[ 15], 10.00th=[ 15], 20.00th=[ 15], | 30.00th=[ 15], 40.00th=[ 15], 50.00th=[ 15], 60.00th=[ 1036], | 70.00th=[ 1036], 80.00th=[ 1036], 90.00th=[ 1036], 95.00th=[ 1036], | 99.00th=[ 1036], 99.50th=[ 1036], 99.90th=[ 1036], 99.95th=[ 1036], | 99.99th=[ 1036] bw ( KiB/s): min= 1689, max= 2043, per=26.75%, avg=1866.00, stdev=250.32, samples=2 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=2 write: IOPS=4, BW=4098KiB/s (4196kB/s)(10.0MiB/2499msec) clat (msec): min=26, max=555, avg=143.85, stdev=202.80 lat (msec): min=26, max=555, avg=143.92, stdev=202.79 clat percentiles (msec): | 1.00th=[ 27], 5.00th=[ 27], 10.00th=[ 27], 20.00th=[ 28], | 30.00th=[ 40], 40.00th=[ 44], 50.00th=[ 44], 60.00th=[ 47], | 70.00th=[ 82], 80.00th=[ 82], 90.00th=[ 498], 95.00th=[ 558], | 99.00th=[ 558], 99.50th=[ 558], 99.90th=[ 558], 99.95th=[ 558], | 99.99th=[ 558] bw ( KiB/s): min= 5060, max=10219, per=78.23%, avg=7639.50, stdev=3647.96, samples=2 iops : min= 4, max= 9, avg= 6.50, stdev= 3.54, samples=2 lat (msec) : 20=8.33%, 50=50.00%, 100=16.67%, 500=8.33%, 750=8.33% cpu : usr=0.00%, sys=4.00%, ctx=49, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22657: Tue Aug 27 10:23:53 2024 read: IOPS=2, BW=2093KiB/s (2143kB/s)(6144KiB/2936msec) clat (msec): min=20, max=627, avg=187.85, stdev=254.49 lat (msec): min=20, max=627, avg=187.85, stdev=254.49 clat percentiles (msec): | 1.00th=[ 21], 5.00th=[ 21], 10.00th=[ 21], 20.00th=[ 23], | 30.00th=[ 23], 40.00th=[ 41], 50.00th=[ 41], 60.00th=[ 45], | 70.00th=[ 372], 80.00th=[ 372], 90.00th=[ 625], 95.00th=[ 625], | 99.00th=[ 625], 99.50th=[ 625], 99.90th=[ 625], 99.95th=[ 625], | 99.99th=[ 625] bw ( KiB/s): min= 1692, max= 3385, per=40.44%, avg=2820.67, stdev=977.45, samples=3 iops : min= 1, max= 3, avg= 2.33, stdev= 1.15, samples=3 write: IOPS=2, BW=2093KiB/s (2143kB/s)(6144KiB/2936msec) clat (msec): min=31, max=1019, avg=297.83, stdev=409.25 lat (msec): min=31, max=1019, avg=297.90, stdev=409.28 clat percentiles (msec): | 1.00th=[ 32], 5.00th=[ 32], 10.00th=[ 32], 20.00th=[ 40], | 30.00th=[ 40], 40.00th=[ 42], 50.00th=[ 42], 60.00th=[ 92], | 70.00th=[ 567], 80.00th=[ 567], 90.00th=[ 1020], 95.00th=[ 1020], | 99.00th=[ 1020], 99.50th=[ 1020], 99.90th=[ 1020], 99.95th=[ 1020], | 99.99th=[ 1020] bw ( KiB/s): min= 1692, max= 4096, per=27.82%, avg=2716.25, stdev=1217.80, samples=4 iops : min= 1, max= 4, avg= 2.25, stdev= 1.50, samples=4 lat (msec) : 50=58.33%, 100=8.33%, 500=8.33%, 750=16.67% cpu : usr=0.00%, sys=3.27%, ctx=37, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22658: Tue Aug 27 10:23:53 2024 read: IOPS=3, BW=4013KiB/s (4109kB/s)(5120KiB/1276msec) clat (usec): min=12495, max=68993, avg=43103.40, stdev=28064.67 lat (usec): min=12496, max=68994, avg=43104.86, stdev=28064.50 clat percentiles (usec): | 1.00th=[12518], 5.00th=[12518], 10.00th=[12518], 20.00th=[12518], | 30.00th=[12780], 40.00th=[12780], 50.00th=[58983], 60.00th=[58983], | 70.00th=[62129], 80.00th=[62129], 90.00th=[68682], 95.00th=[68682], | 99.00th=[68682], 99.50th=[68682], 99.90th=[68682], 99.95th=[68682], | 99.99th=[68682] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=5, BW=5618KiB/s (5752kB/s)(7168KiB/1276msec) clat (msec): min=31, max=774, avg=148.98, stdev=276.57 lat (msec): min=31, max=774, avg=149.03, stdev=276.57 clat percentiles (msec): | 1.00th=[ 32], 5.00th=[ 32], 10.00th=[ 32], 20.00th=[ 34], | 30.00th=[ 35], 40.00th=[ 35], 50.00th=[ 42], 60.00th=[ 44], | 70.00th=[ 44], 80.00th=[ 85], 90.00th=[ 776], 95.00th=[ 776], | 99.00th=[ 776], 99.50th=[ 776], 99.90th=[ 776], 99.95th=[ 776], | 99.99th=[ 776] bw ( KiB/s): min=10240, max=10240, per=100.00%, avg=10240.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 lat (msec) : 20=16.67%, 50=41.67%, 100=33.33%, 1000=8.33% cpu : usr=0.00%, sys=6.35%, ctx=62, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=6975KiB/s (7143kB/s), 820KiB/s-5762KiB/s (839kB/s-5900kB/s), io=20.0MiB (20.0MB), run=1244-2936msec WRITE: bw=9766KiB/s (10.0MB/s), 2093KiB/s-5618KiB/s (2143kB/s-5752kB/s), io=28.0MiB (29.4MB), run=1244-2936msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=22671: Tue Aug 27 10:23:56 2024 read: IOPS=1, BW=7119KiB/s (7290kB/s)(12.0MiB/1726msec) clat (msec): min=115, max=845, avg=564.54, stdev=392.76 lat (msec): min=115, max=845, avg=564.54, stdev=392.76 clat percentiles (msec): | 1.00th=[ 116], 5.00th=[ 116], 10.00th=[ 116], 20.00th=[ 116], | 30.00th=[ 116], 40.00th=[ 735], 50.00th=[ 735], 60.00th=[ 735], | 70.00th=[ 844], 80.00th=[ 844], 90.00th=[ 844], 95.00th=[ 844], | 99.00th=[ 844], 99.50th=[ 844], 99.90th=[ 844], 99.95th=[ 844], | 99.99th=[ 844] bw ( KiB/s): min= 8192, max= 8192, per=86.30%, avg=8192.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 lat (msec) : 250=33.33%, 750=33.33%, 1000=33.33% cpu : usr=0.00%, sys=61.97%, ctx=647, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22672: Tue Aug 27 10:23:56 2024 read: IOPS=0, BW=2466KiB/s (2525kB/s)(4096KiB/1661msec) clat (nsec): min=124031k, max=124031k, avg=124030582.00, stdev= 0.00 lat (nsec): min=124032k, max=124032k, avg=124032290.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 125], 5.00th=[ 125], 10.00th=[ 125], 20.00th=[ 125], | 30.00th=[ 125], 40.00th=[ 125], 50.00th=[ 125], 60.00th=[ 125], | 70.00th=[ 125], 80.00th=[ 125], 90.00th=[ 125], 95.00th=[ 125], | 99.00th=[ 125], 99.50th=[ 125], 99.90th=[ 125], 99.95th=[ 125], | 99.99th=[ 125] bw ( KiB/s): min= 8192, max= 8192, per=86.30%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=1, BW=4932KiB/s (5050kB/s)(8192KiB/1661msec) clat (msec): min=300, max=1221, avg=760.85, stdev=650.99 lat (msec): min=300, max=1221, avg=761.15, stdev=651.01 clat percentiles (msec): | 1.00th=[ 300], 5.00th=[ 300], 10.00th=[ 300], 20.00th=[ 300], | 30.00th=[ 300], 40.00th=[ 300], 50.00th=[ 300], 60.00th=[ 1217], | 70.00th=[ 1217], 80.00th=[ 1217], 90.00th=[ 1217], 95.00th=[ 1217], | 99.00th=[ 1217], 99.50th=[ 1217], 99.90th=[ 1217], 99.95th=[ 1217], | 99.99th=[ 1217] bw ( KiB/s): min= 8192, max= 8192, per=43.58%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=33.33% cpu : usr=0.00%, sys=14.34%, ctx=128, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22673: Tue Aug 27 10:23:56 2024 write: IOPS=1, BW=7050KiB/s (7219kB/s)(12.0MiB/1743msec) clat (msec): min=240, max=1157, avg=567.93, stdev=511.74 lat (msec): min=240, max=1157, avg=568.19, stdev=511.76 clat percentiles (msec): | 1.00th=[ 241], 5.00th=[ 241], 10.00th=[ 241], 20.00th=[ 241], | 30.00th=[ 241], 40.00th=[ 305], 50.00th=[ 305], 60.00th=[ 305], | 70.00th=[ 1150], 80.00th=[ 1150], 90.00th=[ 1150], 95.00th=[ 1150], | 99.00th=[ 1150], 99.50th=[ 1150], 99.90th=[ 1150], 99.95th=[ 1150], | 99.99th=[ 1150] bw ( KiB/s): min= 8175, max= 8175, per=43.49%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=33.33% cpu : usr=0.00%, sys=13.38%, ctx=78, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22674: Tue Aug 27 10:23:56 2024 write: IOPS=1, BW=8111KiB/s (8306kB/s)(12.0MiB/1515msec) clat (msec): min=187, max=1099, avg=501.93, stdev=517.60 lat (msec): min=187, max=1099, avg=502.23, stdev=517.55 clat percentiles (msec): | 1.00th=[ 188], 5.00th=[ 188], 10.00th=[ 188], 20.00th=[ 188], | 30.00th=[ 188], 40.00th=[ 220], 50.00th=[ 220], 60.00th=[ 220], | 70.00th=[ 1099], 80.00th=[ 1099], 90.00th=[ 1099], 95.00th=[ 1099], | 99.00th=[ 1099], 99.50th=[ 1099], 99.90th=[ 1099], 99.95th=[ 1099], | 99.99th=[ 1099] bw ( KiB/s): min= 8175, max= 8192, per=43.53%, avg=8183.50, stdev=12.02, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 250=66.67% cpu : usr=0.00%, sys=16.51%, ctx=61, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=9492KiB/s (9720kB/s), 2466KiB/s-7119KiB/s (2525kB/s-7290kB/s), io=16.0MiB (16.8MB), run=1661-1726msec WRITE: bw=18.4MiB/s (19.2MB/s), 4932KiB/s-8111KiB/s (5050kB/s-8306kB/s), io=32.0MiB (33.6MB), run=1515-1743msec rand-rw: (groupid=0, jobs=1): err= 0: pid=22675: Tue Aug 27 10:23:56 2024 read: IOPS=1, BW=7022KiB/s (7190kB/s)(12.0MiB/1750msec) clat (msec): min=222, max=1157, avg=579.94, stdev=504.81 lat (msec): min=222, max=1157, avg=579.94, stdev=504.81 clat percentiles (msec): | 1.00th=[ 222], 5.00th=[ 222], 10.00th=[ 222], 20.00th=[ 222], | 30.00th=[ 222], 40.00th=[ 359], 50.00th=[ 359], 60.00th=[ 359], | 70.00th=[ 1150], 80.00th=[ 1150], 90.00th=[ 1150], 95.00th=[ 1150], | 99.00th=[ 1150], 99.50th=[ 1150], 99.90th=[ 1150], 99.95th=[ 1150], | 99.99th=[ 1150] bw ( KiB/s): min= 8192, max= 8192, per=88.41%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=33.33% cpu : usr=0.06%, sys=3.32%, ctx=40, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22676: Tue Aug 27 10:23:56 2024 read: IOPS=0, BW=2317KiB/s (2372kB/s)(4096KiB/1768msec) clat (nsec): min=423283k, max=423283k, avg=423283494.00, stdev= 0.00 lat (nsec): min=423284k, max=423284k, avg=423284338.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 422], 5.00th=[ 422], 10.00th=[ 422], 20.00th=[ 422], | 30.00th=[ 422], 40.00th=[ 422], 50.00th=[ 422], 60.00th=[ 422], | 70.00th=[ 422], 80.00th=[ 422], 90.00th=[ 422], 95.00th=[ 422], | 99.00th=[ 422], 99.50th=[ 422], 99.90th=[ 422], 99.95th=[ 422], | 99.99th=[ 422] bw ( KiB/s): min= 8192, max= 8192, per=88.41%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=1, BW=4633KiB/s (4745kB/s)(8192KiB/1768msec) clat (msec): min=78, max=1251, avg=664.79, stdev=829.21 lat (msec): min=78, max=1251, avg=665.04, stdev=829.23 clat percentiles (msec): | 1.00th=[ 79], 5.00th=[ 79], 10.00th=[ 79], 20.00th=[ 79], | 30.00th=[ 79], 40.00th=[ 79], 50.00th=[ 79], 60.00th=[ 1250], | 70.00th=[ 1250], 80.00th=[ 1250], 90.00th=[ 1250], 95.00th=[ 1250], | 99.00th=[ 1250], 99.50th=[ 1250], 99.90th=[ 1250], 99.95th=[ 1250], | 99.99th=[ 1250] bw ( KiB/s): min= 8192, max= 8192, per=46.75%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 500=33.33% cpu : usr=0.00%, sys=4.02%, ctx=132, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22677: Tue Aug 27 10:23:56 2024 write: IOPS=1, BW=6571KiB/s (6729kB/s)(12.0MiB/1870msec) clat (msec): min=40, max=1415, avg=620.50, stdev=712.23 lat (msec): min=40, max=1415, avg=620.76, stdev=712.15 clat percentiles (msec): | 1.00th=[ 41], 5.00th=[ 41], 10.00th=[ 41], 20.00th=[ 41], | 30.00th=[ 41], 40.00th=[ 405], 50.00th=[ 405], 60.00th=[ 405], | 70.00th=[ 1418], 80.00th=[ 1418], 90.00th=[ 1418], 95.00th=[ 1418], | 99.00th=[ 1418], 99.50th=[ 1418], 99.90th=[ 1418], 99.95th=[ 1418], | 99.99th=[ 1418] bw ( KiB/s): min= 8192, max= 8192, per=46.75%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 50=33.33%, 500=33.33% cpu : usr=0.05%, sys=4.44%, ctx=47, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22678: Tue Aug 27 10:23:56 2024 write: IOPS=1, BW=7792KiB/s (7979kB/s)(12.0MiB/1577msec) clat (msec): min=234, max=1063, avg=519.39, stdev=471.37 lat (msec): min=234, max=1063, avg=519.64, stdev=471.43 clat percentiles (msec): | 1.00th=[ 234], 5.00th=[ 234], 10.00th=[ 234], 20.00th=[ 234], | 30.00th=[ 234], 40.00th=[ 259], 50.00th=[ 259], 60.00th=[ 259], | 70.00th=[ 1062], 80.00th=[ 1062], 90.00th=[ 1062], 95.00th=[ 1062], | 99.00th=[ 1062], 99.50th=[ 1062], 99.90th=[ 1062], 99.95th=[ 1062], | 99.99th=[ 1062] bw ( KiB/s): min= 8159, max= 8192, per=46.66%, avg=8175.50, stdev=23.33, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 250=33.33%, 500=33.33% cpu : usr=0.06%, sys=7.11%, ctx=68, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=9267KiB/s (9489kB/s), 2317KiB/s-7022KiB/s (2372kB/s-7190kB/s), io=16.0MiB (16.8MB), run=1750-1768msec WRITE: bw=17.1MiB/s (17.9MB/s), 4633KiB/s-7792KiB/s (4745kB/s-7979kB/s), io=32.0MiB (33.6MB), run=1577-1870msec