== sanity test 398b: DIO and buffer IO race ============== 07:19:24 (1730809164) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.33014 s, 37.8 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=27126: Tue Nov 5 07:19:39 2024 read: IOPS=123, BW=493KiB/s (505kB/s)(5992KiB/12155msec) clat (usec): min=48, max=173911, avg=3780.58, stdev=5444.42 lat (usec): min=48, max=173912, avg=3781.26, stdev=5444.46 clat percentiles (usec): | 1.00th=[ 57], 5.00th=[ 61], 10.00th=[ 72], 20.00th=[ 1860], | 30.00th=[ 2245], 40.00th=[ 2606], 50.00th=[ 3032], 60.00th=[ 3556], | 70.00th=[ 4490], 80.00th=[ 5669], 90.00th=[ 7242], 95.00th=[ 8848], | 99.00th=[ 15664], 99.50th=[ 20579], 99.90th=[ 32375], 99.95th=[173016], | 99.99th=[173016] bw ( KiB/s): min= 232, max= 734, per=24.68%, avg=487.42, stdev=105.13, samples=24 iops : min= 58, max= 183, avg=121.79, stdev=26.20, samples=24 write: IOPS=129, BW=518KiB/s (530kB/s)(6296KiB/12155msec) clat (usec): min=465, max=26456, avg=4037.45, stdev=3749.60 lat (usec): min=465, max=26456, avg=4038.72, stdev=3749.44 clat percentiles (usec): | 1.00th=[ 486], 5.00th=[ 523], 10.00th=[ 545], 20.00th=[ 619], | 30.00th=[ 857], 40.00th=[ 1893], 50.00th=[ 3392], 60.00th=[ 4359], | 70.00th=[ 5342], 80.00th=[ 6783], 90.00th=[ 8979], 95.00th=[11207], | 99.00th=[16909], 99.50th=[19006], 99.90th=[23725], 99.95th=[26346], | 99.99th=[26346] bw ( KiB/s): min= 280, max= 704, per=24.75%, avg=511.75, stdev=99.10, samples=24 iops : min= 70, max= 176, avg=127.88, stdev=24.73, samples=24 lat (usec) : 50=0.07%, 100=7.88%, 250=1.04%, 500=1.17%, 750=12.89% lat (usec) : 1000=2.73% lat (msec) : 2=5.99%, 4=28.94%, 10=34.05%, 20=4.79%, 50=0.42% lat (msec) : 250=0.03% cpu : usr=0.15%, sys=24.03%, ctx=5765, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=27127: Tue Nov 5 07:19:39 2024 read: IOPS=124, BW=497KiB/s (508kB/s)(5940KiB/11962msec) clat (usec): min=48, max=28392, avg=3552.90, stdev=2750.66 lat (usec): min=48, max=28392, avg=3554.17, stdev=2750.82 clat percentiles (usec): | 1.00th=[ 55], 5.00th=[ 61], 10.00th=[ 72], 20.00th=[ 1909], | 30.00th=[ 2311], 40.00th=[ 2638], 50.00th=[ 3032], 60.00th=[ 3556], | 70.00th=[ 4424], 80.00th=[ 5538], 90.00th=[ 7046], 95.00th=[ 8225], | 99.00th=[11863], 99.50th=[14746], 99.90th=[25035], 99.95th=[28443], | 99.99th=[28443] bw ( KiB/s): min= 248, max= 694, per=25.01%, avg=493.96, stdev=109.45, samples=23 iops : min= 62, max= 173, avg=123.39, stdev=27.31, samples=23 write: IOPS=132, BW=531KiB/s (543kB/s)(6348KiB/11962msec) clat (usec): min=421, max=196014, avg=4126.34, stdev=6260.53 lat (usec): min=422, max=196015, avg=4127.80, stdev=6261.34 clat percentiles (usec): | 1.00th=[ 490], 5.00th=[ 523], 10.00th=[ 545], 20.00th=[ 611], | 30.00th=[ 824], 40.00th=[ 1663], 50.00th=[ 3130], 60.00th=[ 4113], | 70.00th=[ 5145], 80.00th=[ 6652], 90.00th=[ 8979], 95.00th=[ 11469], | 99.00th=[ 17957], 99.50th=[ 21890], 99.90th=[ 36439], 99.95th=[196084], | 99.99th=[196084] bw ( KiB/s): min= 288, max= 862, per=25.15%, avg=520.04, stdev=106.29, samples=23 iops : min= 72, max= 215, avg=129.91, stdev=26.51, samples=23 lat (usec) : 50=0.10%, 100=7.75%, 250=0.75%, 500=0.85%, 750=13.35% lat (usec) : 1000=3.26% lat (msec) : 2=5.86%, 4=30.37%, 10=32.85%, 20=4.49%, 50=0.36% lat (msec) : 250=0.03% cpu : usr=0.21%, sys=24.49%, ctx=5728, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=27128: Tue Nov 5 07:19:39 2024 read: IOPS=125, BW=502KiB/s (514kB/s)(6028KiB/11999msec) clat (usec): min=46, max=29857, avg=3667.85, stdev=2929.23 lat (usec): min=46, max=29858, avg=3668.51, stdev=2929.37 clat percentiles (usec): | 1.00th=[ 51], 5.00th=[ 62], 10.00th=[ 73], 20.00th=[ 1860], | 30.00th=[ 2376], 40.00th=[ 2737], 50.00th=[ 3097], 60.00th=[ 3621], | 70.00th=[ 4490], 80.00th=[ 5800], 90.00th=[ 7373], 95.00th=[ 8717], | 99.00th=[12387], 99.50th=[13829], 99.90th=[27657], 99.95th=[29754], | 99.99th=[29754] bw ( KiB/s): min= 264, max= 758, per=25.33%, avg=500.26, stdev=104.36, samples=23 iops : min= 66, max= 189, avg=124.96, stdev=26.04, samples=23 write: IOPS=130, BW=522KiB/s (534kB/s)(6260KiB/11999msec) clat (usec): min=428, max=168377, avg=4049.07, stdev=5664.06 lat (usec): min=428, max=168378, avg=4049.86, stdev=5664.05 clat percentiles (usec): | 1.00th=[ 486], 5.00th=[ 523], 10.00th=[ 545], 20.00th=[ 603], | 30.00th=[ 799], 40.00th=[ 1713], 50.00th=[ 3163], 60.00th=[ 4015], | 70.00th=[ 5145], 80.00th=[ 6783], 90.00th=[ 9110], 95.00th=[ 10945], | 99.00th=[ 16909], 99.50th=[ 20055], 99.90th=[ 35390], 99.95th=[168821], | 99.99th=[168821] bw ( KiB/s): min= 328, max= 694, per=25.01%, avg=517.26, stdev=88.57, samples=23 iops : min= 82, max= 173, avg=129.22, stdev=22.08, samples=23 lat (usec) : 50=0.36%, 100=7.68%, 250=1.01%, 500=1.04%, 750=13.44% lat (usec) : 1000=2.77% lat (msec) : 2=6.41%, 4=29.36%, 10=32.81%, 20=4.75%, 50=0.33% lat (msec) : 250=0.03% cpu : usr=0.18%, sys=24.05%, ctx=5715, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=27129: Tue Nov 5 07:19:39 2024 read: IOPS=125, BW=503KiB/s (515kB/s)(6052KiB/12027msec) clat (usec): min=46, max=173422, avg=3719.48, stdev=5234.75 lat (usec): min=47, max=173422, avg=3720.29, stdev=5234.75 clat percentiles (usec): | 1.00th=[ 51], 5.00th=[ 60], 10.00th=[ 70], 20.00th=[ 1860], | 30.00th=[ 2278], 40.00th=[ 2671], 50.00th=[ 3097], 60.00th=[ 3720], | 70.00th=[ 4555], 80.00th=[ 5669], 90.00th=[ 7111], 95.00th=[ 8356], | 99.00th=[ 11076], 99.50th=[ 18220], 99.90th=[ 34341], 99.95th=[173016], | 99.99th=[173016] bw ( KiB/s): min= 272, max= 798, per=25.49%, avg=503.33, stdev=106.81, samples=24 iops : min= 68, max= 199, avg=125.75, stdev=26.61, samples=24 write: IOPS=129, BW=519KiB/s (531kB/s)(6236KiB/12027msec) clat (usec): min=450, max=33337, avg=4016.76, stdev=3957.61 lat (usec): min=451, max=33337, avg=4018.43, stdev=3957.01 clat percentiles (usec): | 1.00th=[ 486], 5.00th=[ 519], 10.00th=[ 545], 20.00th=[ 611], | 30.00th=[ 807], 40.00th=[ 1598], 50.00th=[ 3097], 60.00th=[ 4113], | 70.00th=[ 5211], 80.00th=[ 6915], 90.00th=[ 9241], 95.00th=[11731], | 99.00th=[16712], 99.50th=[18744], 99.90th=[33162], 99.95th=[33424], | 99.99th=[33424] bw ( KiB/s): min= 344, max= 710, per=25.03%, avg=517.71, stdev=106.01, samples=24 iops : min= 86, max= 177, avg=129.33, stdev=26.46, samples=24 lat (usec) : 50=0.42%, 100=7.75%, 250=0.98%, 500=1.01%, 750=13.44% lat (usec) : 1000=3.26% lat (msec) : 2=5.53%, 4=28.61%, 10=33.89%, 20=4.75%, 50=0.33% lat (msec) : 250=0.03% cpu : usr=0.10%, sys=24.33%, ctx=5668, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1975KiB/s (2023kB/s), 493KiB/s-503KiB/s (505kB/s-515kB/s), io=23.4MiB (24.6MB), run=11962-12155msec WRITE: bw=2068KiB/s (2118kB/s), 518KiB/s-531KiB/s (530kB/s-543kB/s), io=24.6MiB (25.7MB), run=11962-12155msec rand-rw: (groupid=0, jobs=1): err= 0: pid=27130: Tue Nov 5 07:19:59 2024 read: IOPS=47, BW=189KiB/s (193kB/s)(5992KiB/31744msec) clat (usec): min=1554, max=203951, avg=3541.54, stdev=6071.84 lat (usec): min=1555, max=203952, avg=3542.25, stdev=6071.85 clat percentiles (usec): | 1.00th=[ 1713], 5.00th=[ 1795], 10.00th=[ 1860], 20.00th=[ 1975], | 30.00th=[ 2089], 40.00th=[ 2212], 50.00th=[ 2343], 60.00th=[ 2507], | 70.00th=[ 2769], 80.00th=[ 3654], 90.00th=[ 6652], 95.00th=[ 9241], | 99.00th=[ 16450], 99.50th=[ 26084], 99.90th=[ 41681], 99.95th=[204473], | 99.99th=[204473] bw ( KiB/s): min= 56, max= 320, per=25.13%, avg=188.44, stdev=73.16, samples=63 iops : min= 14, max= 80, avg=47.02, stdev=18.32, samples=63 write: IOPS=49, BW=198KiB/s (203kB/s)(6296KiB/31744msec) clat (usec): min=8374, max=57312, avg=16715.01, stdev=6288.90 lat (usec): min=8374, max=57313, avg=16715.94, stdev=6289.00 clat percentiles (usec): | 1.00th=[ 8848], 5.00th=[ 9896], 10.00th=[10552], 20.00th=[11338], | 30.00th=[12518], 40.00th=[14091], 50.00th=[15533], 60.00th=[16909], | 70.00th=[18220], 80.00th=[20317], 90.00th=[25035], 95.00th=[29492], | 99.00th=[38011], 99.50th=[40633], 99.90th=[50070], 99.95th=[57410], | 99.99th=[57410] bw ( KiB/s): min= 47, max= 296, per=25.19%, avg=197.97, stdev=60.67, samples=63 iops : min= 11, max= 74, avg=49.40, stdev=15.19, samples=63 lat (msec) : 2=10.38%, 4=29.43%, 10=9.86%, 20=39.45%, 50=10.81% lat (msec) : 100=0.03%, 250=0.03% cpu : usr=0.14%, sys=5.98%, ctx=3422, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=27131: Tue Nov 5 07:19:59 2024 read: IOPS=46, BW=186KiB/s (190kB/s)(5940KiB/31978msec) clat (usec): min=1621, max=177943, avg=3481.58, stdev=5510.38 lat (usec): min=1621, max=177945, avg=3482.31, stdev=5510.39 clat percentiles (usec): | 1.00th=[ 1680], 5.00th=[ 1795], 10.00th=[ 1876], 20.00th=[ 1991], | 30.00th=[ 2073], 40.00th=[ 2180], 50.00th=[ 2311], 60.00th=[ 2442], | 70.00th=[ 2671], 80.00th=[ 3458], 90.00th=[ 6849], 95.00th=[ 9634], | 99.00th=[ 14877], 99.50th=[ 26346], 99.90th=[ 37487], 99.95th=[177210], | 99.99th=[177210] bw ( KiB/s): min= 16, max= 344, per=24.78%, avg=185.87, stdev=76.21, samples=63 iops : min= 4, max= 86, avg=46.44, stdev=19.05, samples=63 write: IOPS=49, BW=199KiB/s (203kB/s)(6348KiB/31978msec) clat (usec): min=8180, max=65026, avg=16806.87, stdev=6176.50 lat (usec): min=8180, max=65027, avg=16807.76, stdev=6176.53 clat percentiles (usec): | 1.00th=[ 9110], 5.00th=[10028], 10.00th=[10683], 20.00th=[11600], | 30.00th=[12780], 40.00th=[14353], 50.00th=[15795], 60.00th=[16909], | 70.00th=[18220], 80.00th=[20317], 90.00th=[25297], 95.00th=[29230], | 99.00th=[38011], 99.50th=[40633], 99.90th=[54789], 99.95th=[65274], | 99.99th=[65274] bw ( KiB/s): min= 64, max= 280, per=25.13%, avg=197.56, stdev=57.67, samples=63 iops : min= 16, max= 70, avg=49.37, stdev=14.42, samples=63 lat (msec) : 2=10.61%, 4=29.72%, 10=8.43%, 20=40.14%, 50=11.00% lat (msec) : 100=0.07%, 250=0.03% cpu : usr=0.14%, sys=5.79%, ctx=3440, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=27132: Tue Nov 5 07:19:59 2024 read: IOPS=47, BW=189KiB/s (194kB/s)(6028KiB/31896msec) clat (usec): min=1614, max=35723, avg=3580.51, stdev=3135.12 lat (usec): min=1615, max=35724, avg=3581.39, stdev=3135.21 clat percentiles (usec): | 1.00th=[ 1680], 5.00th=[ 1778], 10.00th=[ 1860], 20.00th=[ 1958], | 30.00th=[ 2073], 40.00th=[ 2180], 50.00th=[ 2343], 60.00th=[ 2507], | 70.00th=[ 2802], 80.00th=[ 4490], 90.00th=[ 7898], 95.00th=[ 9896], | 99.00th=[15401], 99.50th=[18482], 99.90th=[28967], 99.95th=[35914], | 99.99th=[35914] bw ( KiB/s): min= 56, max= 368, per=25.17%, avg=188.79, stdev=67.94, samples=63 iops : min= 14, max= 92, avg=47.17, stdev=16.99, samples=63 write: IOPS=49, BW=196KiB/s (201kB/s)(6260KiB/31896msec) clat (msec): min=7, max=197, avg=16.85, stdev= 7.67 lat (msec): min=7, max=197, avg=16.85, stdev= 7.67 clat percentiles (msec): | 1.00th=[ 9], 5.00th=[ 11], 10.00th=[ 11], 20.00th=[ 12], | 30.00th=[ 13], 40.00th=[ 15], 50.00th=[ 16], 60.00th=[ 17], | 70.00th=[ 19], 80.00th=[ 21], 90.00th=[ 26], 95.00th=[ 30], | 99.00th=[ 37], 99.50th=[ 42], 99.90th=[ 58], 99.95th=[ 199], | 99.99th=[ 199] bw ( KiB/s): min= 64, max= 280, per=24.88%, avg=195.52, stdev=60.25, samples=63 iops : min= 16, max= 70, avg=48.86, stdev=15.06, samples=63 lat (msec) : 2=11.49%, 4=26.89%, 10=10.90%, 20=40.10%, 50=10.51% lat (msec) : 100=0.07%, 250=0.03% cpu : usr=0.11%, sys=5.84%, ctx=3497, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=27133: Tue Nov 5 07:19:59 2024 read: IOPS=47, BW=191KiB/s (196kB/s)(6052KiB/31627msec) clat (usec): min=1548, max=41502, avg=3390.76, stdev=3132.98 lat (usec): min=1549, max=41502, avg=3391.48, stdev=3132.96 clat percentiles (usec): | 1.00th=[ 1696], 5.00th=[ 1778], 10.00th=[ 1860], 20.00th=[ 1975], | 30.00th=[ 2073], 40.00th=[ 2147], 50.00th=[ 2311], 60.00th=[ 2474], | 70.00th=[ 2737], 80.00th=[ 3818], 90.00th=[ 6652], 95.00th=[ 8848], | 99.00th=[15270], 99.50th=[21365], 99.90th=[35390], 99.95th=[41681], | 99.99th=[41681] bw ( KiB/s): min= 56, max= 344, per=25.52%, avg=191.41, stdev=71.32, samples=63 iops : min= 14, max= 86, avg=47.79, stdev=17.84, samples=63 write: IOPS=49, BW=197KiB/s (202kB/s)(6236KiB/31627msec) clat (msec): min=8, max=198, avg=16.91, stdev= 7.51 lat (msec): min=8, max=198, avg=16.91, stdev= 7.51 clat percentiles (msec): | 1.00th=[ 9], 5.00th=[ 11], 10.00th=[ 11], 20.00th=[ 12], | 30.00th=[ 13], 40.00th=[ 15], 50.00th=[ 16], 60.00th=[ 18], | 70.00th=[ 19], 80.00th=[ 21], 90.00th=[ 26], 95.00th=[ 29], | 99.00th=[ 37], 99.50th=[ 41], 99.90th=[ 52], 99.95th=[ 199], | 99.99th=[ 199] bw ( KiB/s): min= 56, max= 280, per=25.03%, avg=196.75, stdev=58.41, samples=63 iops : min= 14, max= 70, avg=49.13, stdev=14.60, samples=63 lat (msec) : 2=11.30%, 4=28.74%, 10=9.73%, 20=38.93%, 50=11.23% lat (msec) : 100=0.03%, 250=0.03% cpu : usr=0.10%, sys=5.90%, ctx=3482, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=751KiB/s (769kB/s), 186KiB/s-191KiB/s (190kB/s-196kB/s), io=23.4MiB (24.6MB), run=31627-31978msec WRITE: bw=786KiB/s (805kB/s), 196KiB/s-199KiB/s (201kB/s-203kB/s), io=24.6MiB (25.7MB), run=31627-31978msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=27171: Tue Nov 5 07:20:05 2024 read: IOPS=80, BW=1289KiB/s (1320kB/s)(5920KiB/4594msec) clat (usec): min=54, max=29246, avg=9121.42, stdev=4500.17 lat (usec): min=55, max=29248, avg=9122.02, stdev=4500.18 clat percentiles (usec): | 1.00th=[ 62], 5.00th=[ 65], 10.00th=[ 78], 20.00th=[ 7504], | 30.00th=[ 8455], 40.00th=[ 9110], 50.00th=[ 9765], 60.00th=[10552], | 70.00th=[11207], 80.00th=[11994], 90.00th=[13566], 95.00th=[15008], | 99.00th=[18482], 99.50th=[24511], 99.90th=[29230], 99.95th=[29230], | 99.99th=[29230] bw ( KiB/s): min= 960, max= 1536, per=25.08%, avg=1272.56, stdev=176.03, samples=9 iops : min= 60, max= 96, avg=79.44, stdev=11.06, samples=9 write: IOPS=86, BW=1386KiB/s (1419kB/s)(6368KiB/4594msec) clat (usec): min=656, max=20232, avg=2911.90, stdev=3067.03 lat (usec): min=657, max=20233, avg=2912.99, stdev=3067.02 clat percentiles (usec): | 1.00th=[ 660], 5.00th=[ 725], 10.00th=[ 766], 20.00th=[ 816], | 30.00th=[ 881], 40.00th=[ 1090], 50.00th=[ 2147], 60.00th=[ 2704], | 70.00th=[ 3195], 80.00th=[ 4359], 90.00th=[ 5997], 95.00th=[ 8356], | 99.00th=[17433], 99.50th=[20055], 99.90th=[20317], 99.95th=[20317], | 99.99th=[20317] bw ( KiB/s): min= 1152, max= 1920, per=25.84%, avg=1379.22, stdev=262.38, samples=9 iops : min= 72, max= 120, avg=86.11, stdev=16.47, samples=9 lat (usec) : 100=6.38%, 250=0.65%, 750=4.17%, 1000=15.36% lat (msec) : 2=5.47%, 4=15.49%, 10=28.12%, 20=23.70%, 50=0.65% cpu : usr=0.09%, sys=22.27%, ctx=2140, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=27173: Tue Nov 5 07:20:05 2024 read: IOPS=77, BW=1235KiB/s (1264kB/s)(5744KiB/4652msec) clat (usec): min=53, max=37225, avg=8714.01, stdev=5129.77 lat (usec): min=53, max=37226, avg=8714.61, stdev=5129.78 clat percentiles (usec): | 1.00th=[ 56], 5.00th=[ 65], 10.00th=[ 69], 20.00th=[ 4146], | 30.00th=[ 8455], 40.00th=[ 8979], 50.00th=[ 9765], 60.00th=[10552], | 70.00th=[11207], 80.00th=[12125], 90.00th=[13173], 95.00th=[14484], | 99.00th=[20317], 99.50th=[33817], 99.90th=[37487], 99.95th=[37487], | 99.99th=[37487] bw ( KiB/s): min= 448, max= 1632, per=24.17%, avg=1226.33, stdev=363.59, samples=9 iops : min= 28, max= 102, avg=76.56, stdev=22.77, samples=9 write: IOPS=87, BW=1407KiB/s (1440kB/s)(6544KiB/4652msec) clat (usec): min=653, max=29915, avg=3540.64, stdev=4502.32 lat (usec): min=654, max=29916, avg=3541.66, stdev=4502.50 clat percentiles (usec): | 1.00th=[ 685], 5.00th=[ 742], 10.00th=[ 775], 20.00th=[ 840], | 30.00th=[ 930], 40.00th=[ 1139], 50.00th=[ 2343], 60.00th=[ 2802], | 70.00th=[ 3589], 80.00th=[ 5211], 90.00th=[ 6915], 95.00th=[12256], | 99.00th=[25560], 99.50th=[28443], 99.90th=[30016], 99.95th=[30016], | 99.99th=[30016] bw ( KiB/s): min= 800, max= 2016, per=25.71%, avg=1372.11, stdev=370.43, samples=9 iops : min= 50, max= 126, avg=85.67, stdev=23.21, samples=9 lat (usec) : 100=8.07%, 250=1.17%, 750=3.39%, 1000=15.23% lat (msec) : 2=5.99%, 4=14.71%, 10=26.43%, 20=23.31%, 50=1.69% cpu : usr=0.09%, sys=21.61%, ctx=2069, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=27175: Tue Nov 5 07:20:05 2024 read: IOPS=82, BW=1325KiB/s (1357kB/s)(6256KiB/4721msec) clat (usec): min=52, max=39197, avg=8857.26, stdev=5223.95 lat (usec): min=52, max=39198, avg=8857.89, stdev=5223.95 clat percentiles (usec): | 1.00th=[ 57], 5.00th=[ 64], 10.00th=[ 77], 20.00th=[ 6456], | 30.00th=[ 8356], 40.00th=[ 9110], 50.00th=[ 9896], 60.00th=[10290], | 70.00th=[11207], 80.00th=[11994], 90.00th=[13698], 95.00th=[15401], | 99.00th=[22152], 99.50th=[31589], 99.90th=[39060], 99.95th=[39060], | 99.99th=[39060] bw ( KiB/s): min= 896, max= 1568, per=25.51%, avg=1293.89, stdev=225.96, samples=9 iops : min= 56, max= 98, avg=80.78, stdev=14.18, samples=9 write: IOPS=79, BW=1278KiB/s (1308kB/s)(6032KiB/4721msec) clat (usec): min=649, max=36353, avg=3156.41, stdev=3942.49 lat (usec): min=650, max=36355, avg=3157.76, stdev=3942.39 clat percentiles (usec): | 1.00th=[ 668], 5.00th=[ 725], 10.00th=[ 758], 20.00th=[ 799], | 30.00th=[ 857], 40.00th=[ 1012], 50.00th=[ 2073], 60.00th=[ 2573], | 70.00th=[ 3130], 80.00th=[ 4883], 90.00th=[ 6915], 95.00th=[10421], | 99.00th=[17695], 99.50th=[33424], 99.90th=[36439], 99.95th=[36439], | 99.99th=[36439] bw ( KiB/s): min= 768, max= 1664, per=22.98%, avg=1226.33, stdev=280.62, samples=9 iops : min= 48, max= 104, avg=76.56, stdev=17.61, samples=9 lat (usec) : 100=8.33%, 250=0.91%, 500=0.13%, 750=4.56%, 1000=14.97% lat (msec) : 2=4.95%, 4=13.93%, 10=26.04%, 20=25.26%, 50=0.91% cpu : usr=0.25%, sys=20.91%, ctx=2124, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=27176: Tue Nov 5 07:20:05 2024 read: IOPS=82, BW=1320KiB/s (1352kB/s)(6032KiB/4569msec) clat (usec): min=52, max=36210, avg=8715.93, stdev=5253.30 lat (usec): min=52, max=36211, avg=8716.55, stdev=5253.25 clat percentiles (usec): | 1.00th=[ 55], 5.00th=[ 64], 10.00th=[ 76], 20.00th=[ 151], | 30.00th=[ 8225], 40.00th=[ 8979], 50.00th=[ 9765], 60.00th=[10552], | 70.00th=[11207], 80.00th=[12125], 90.00th=[13829], 95.00th=[15533], | 99.00th=[23987], 99.50th=[24773], 99.90th=[36439], 99.95th=[36439], | 99.99th=[36439] bw ( KiB/s): min= 800, max= 1856, per=25.86%, avg=1311.67, stdev=272.65, samples=9 iops : min= 50, max= 116, avg=81.89, stdev=17.09, samples=9 write: IOPS=85, BW=1369KiB/s (1402kB/s)(6256KiB/4569msec) clat (usec): min=649, max=32448, avg=3229.02, stdev=4070.62 lat (usec): min=650, max=32474, avg=3230.05, stdev=4071.12 clat percentiles (usec): | 1.00th=[ 668], 5.00th=[ 717], 10.00th=[ 766], 20.00th=[ 824], | 30.00th=[ 881], 40.00th=[ 1074], 50.00th=[ 1991], 60.00th=[ 2671], | 70.00th=[ 3228], 80.00th=[ 4948], 90.00th=[ 6652], 95.00th=[11076], | 99.00th=[24249], 99.50th=[29492], 99.90th=[32375], 99.95th=[32375], | 99.99th=[32375] bw ( KiB/s): min= 864, max= 1824, per=25.44%, avg=1357.89, stdev=300.88, samples=9 iops : min= 54, max= 114, avg=84.78, stdev=18.87, samples=9 lat (usec) : 100=8.46%, 250=1.43%, 500=0.13%, 750=4.17%, 1000=15.62% lat (msec) : 2=5.99%, 4=13.02%, 10=25.65%, 20=24.22%, 50=1.30% cpu : usr=0.07%, sys=21.58%, ctx=2057, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=5074KiB/s (5195kB/s), 1235KiB/s-1325KiB/s (1264kB/s-1357kB/s), io=23.4MiB (24.5MB), run=4569-4721msec WRITE: bw=5338KiB/s (5466kB/s), 1278KiB/s-1407KiB/s (1308kB/s-1440kB/s), io=24.6MiB (25.8MB), run=4569-4721msec rand-rw: (groupid=0, jobs=1): err= 0: pid=27169: Tue Nov 5 07:20:09 2024 read: IOPS=42, BW=682KiB/s (699kB/s)(5920KiB/8676msec) clat (usec): min=1853, max=21020, avg=4110.24, stdev=3372.10 lat (usec): min=1854, max=21020, avg=4110.95, stdev=3372.27 clat percentiles (usec): | 1.00th=[ 1926], 5.00th=[ 2008], 10.00th=[ 2073], 20.00th=[ 2147], | 30.00th=[ 2245], 40.00th=[ 2343], 50.00th=[ 2573], 60.00th=[ 2835], | 70.00th=[ 3785], 80.00th=[ 5407], 90.00th=[ 8717], 95.00th=[12125], | 99.00th=[18220], 99.50th=[19792], 99.90th=[21103], 99.95th=[21103], | 99.99th=[21103] bw ( KiB/s): min= 416, max= 1248, per=24.45%, avg=673.88, stdev=218.64, samples=17 iops : min= 26, max= 78, avg=42.12, stdev=13.67, samples=17 write: IOPS=45, BW=734KiB/s (752kB/s)(6368KiB/8676msec) clat (usec): min=9143, max=42843, avg=17884.82, stdev=5501.54 lat (usec): min=9145, max=42844, avg=17886.27, stdev=5501.52 clat percentiles (usec): | 1.00th=[ 9634], 5.00th=[11076], 10.00th=[11600], 20.00th=[12649], | 30.00th=[14746], 40.00th=[16188], 50.00th=[16909], 60.00th=[18220], | 70.00th=[19530], 80.00th=[21103], 90.00th=[25560], 95.00th=[27919], | 99.00th=[36439], 99.50th=[37487], 99.90th=[42730], 99.95th=[42730], | 99.99th=[42730] bw ( KiB/s): min= 512, max= 960, per=25.18%, avg=730.35, stdev=171.67, samples=17 iops : min= 32, max= 60, avg=45.65, stdev=10.73, samples=17 lat (msec) : 2=2.34%, 4=32.29%, 10=10.42%, 20=41.41%, 50=13.54% cpu : usr=0.06%, sys=6.03%, ctx=891, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=27170: Tue Nov 5 07:20:09 2024 read: IOPS=41, BW=661KiB/s (677kB/s)(5744KiB/8688msec) clat (usec): min=1826, max=28814, avg=3899.71, stdev=3235.38 lat (usec): min=1826, max=28814, avg=3900.51, stdev=3235.34 clat percentiles (usec): | 1.00th=[ 1909], 5.00th=[ 2024], 10.00th=[ 2073], 20.00th=[ 2114], | 30.00th=[ 2212], 40.00th=[ 2311], 50.00th=[ 2474], 60.00th=[ 2835], | 70.00th=[ 3589], 80.00th=[ 5145], 90.00th=[ 7570], 95.00th=[10683], | 99.00th=[16909], 99.50th=[21365], 99.90th=[28705], 99.95th=[28705], | 99.99th=[28705] bw ( KiB/s): min= 288, max= 1152, per=24.04%, avg=662.59, stdev=256.44, samples=17 iops : min= 18, max= 72, avg=41.41, stdev=16.03, samples=17 write: IOPS=47, BW=753KiB/s (771kB/s)(6544KiB/8688msec) clat (usec): min=9893, max=47861, avg=17690.11, stdev=5271.67 lat (usec): min=9894, max=47863, avg=17691.67, stdev=5271.60 clat percentiles (usec): | 1.00th=[10421], 5.00th=[11207], 10.00th=[11731], 20.00th=[13173], | 30.00th=[14877], 40.00th=[15926], 50.00th=[17171], 60.00th=[17957], | 70.00th=[19530], 80.00th=[21365], 90.00th=[23987], 95.00th=[27919], | 99.00th=[34341], 99.50th=[36439], 99.90th=[47973], 99.95th=[47973], | 99.99th=[47973] bw ( KiB/s): min= 544, max= 960, per=25.77%, avg=747.29, stdev=148.78, samples=17 iops : min= 34, max= 60, avg=46.71, stdev= 9.30, samples=17 lat (msec) : 2=1.30%, 4=32.68%, 10=10.03%, 20=42.45%, 50=13.54% cpu : usr=0.17%, sys=5.77%, ctx=881, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=27172: Tue Nov 5 07:20:09 2024 read: IOPS=45, BW=735KiB/s (752kB/s)(6256KiB/8517msec) clat (usec): min=1857, max=24107, avg=3924.92, stdev=3332.23 lat (usec): min=1858, max=24107, avg=3925.58, stdev=3332.23 clat percentiles (usec): | 1.00th=[ 1876], 5.00th=[ 1991], 10.00th=[ 2040], 20.00th=[ 2114], | 30.00th=[ 2212], 40.00th=[ 2278], 50.00th=[ 2540], 60.00th=[ 2769], | 70.00th=[ 3326], 80.00th=[ 4948], 90.00th=[ 7701], 95.00th=[11076], | 99.00th=[19268], 99.50th=[22414], 99.90th=[23987], 99.95th=[23987], | 99.99th=[23987] bw ( KiB/s): min= 384, max= 1440, per=26.71%, avg=736.00, stdev=289.11, samples=17 iops : min= 24, max= 90, avg=46.00, stdev=18.07, samples=17 write: IOPS=44, BW=708KiB/s (725kB/s)(6032KiB/8517msec) clat (usec): min=8999, max=52675, avg=18398.17, stdev=5770.78 lat (usec): min=9000, max=52676, avg=18399.60, stdev=5770.76 clat percentiles (usec): | 1.00th=[ 9896], 5.00th=[11207], 10.00th=[12125], 20.00th=[13960], | 30.00th=[15401], 40.00th=[16909], 50.00th=[17695], 60.00th=[18482], | 70.00th=[19530], 80.00th=[21890], 90.00th=[24511], 95.00th=[26870], | 99.00th=[42206], 99.50th=[50594], 99.90th=[52691], 99.95th=[52691], | 99.99th=[52691] bw ( KiB/s): min= 512, max= 928, per=24.34%, avg=705.88, stdev=152.83, samples=17 iops : min= 32, max= 58, avg=44.12, stdev= 9.55, samples=17 lat (msec) : 2=3.39%, 4=35.16%, 10=9.11%, 20=38.67%, 50=13.41% lat (msec) : 100=0.26% cpu : usr=0.14%, sys=6.02%, ctx=893, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=27174: Tue Nov 5 07:20:09 2024 read: IOPS=43, BW=695KiB/s (712kB/s)(6032KiB/8675msec) clat (usec): min=1885, max=26160, avg=4155.93, stdev=3595.71 lat (usec): min=1886, max=26160, avg=4156.66, stdev=3595.76 clat percentiles (usec): | 1.00th=[ 1942], 5.00th=[ 2024], 10.00th=[ 2089], 20.00th=[ 2180], | 30.00th=[ 2245], 40.00th=[ 2343], 50.00th=[ 2573], 60.00th=[ 2769], | 70.00th=[ 3818], 80.00th=[ 5342], 90.00th=[ 8717], 95.00th=[12780], | 99.00th=[19792], 99.50th=[25560], 99.90th=[26084], 99.95th=[26084], | 99.99th=[26084] bw ( KiB/s): min= 416, max= 1024, per=25.00%, avg=688.94, stdev=160.05, samples=17 iops : min= 26, max= 64, avg=43.06, stdev=10.00, samples=17 write: IOPS=45, BW=721KiB/s (738kB/s)(6256KiB/8675msec) clat (usec): min=9949, max=39414, avg=18143.36, stdev=5084.72 lat (usec): min=9951, max=39416, avg=18144.90, stdev=5084.64 clat percentiles (usec): | 1.00th=[10290], 5.00th=[11207], 10.00th=[11863], 20.00th=[13829], | 30.00th=[15270], 40.00th=[16450], 50.00th=[17433], 60.00th=[18744], | 70.00th=[19792], 80.00th=[21890], 90.00th=[25035], 95.00th=[27395], | 99.00th=[34866], 99.50th=[39060], 99.90th=[39584], 99.95th=[39584], | 99.99th=[39584] bw ( KiB/s): min= 480, max= 992, per=24.73%, avg=717.18, stdev=156.79, samples=17 iops : min= 30, max= 62, avg=44.82, stdev= 9.80, samples=17 lat (msec) : 2=1.43%, 4=33.20%, 10=10.55%, 20=39.97%, 50=14.84% cpu : usr=0.22%, sys=5.75%, ctx=906, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2757KiB/s (2823kB/s), 661KiB/s-735KiB/s (677kB/s-752kB/s), io=23.4MiB (24.5MB), run=8517-8688msec WRITE: bw=2901KiB/s (2970kB/s), 708KiB/s-753KiB/s (725kB/s-771kB/s), io=24.6MiB (25.8MB), run=8517-8688msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=27187: Tue Nov 5 07:20:14 2024 read: IOPS=1, BW=1796KiB/s (1839kB/s)(7168KiB/3991msec) clat (usec): min=366, max=926874, avg=404133.64, stdev=481464.24 lat (usec): min=367, max=926874, avg=404135.00, stdev=481464.34 clat percentiles (usec): | 1.00th=[ 367], 5.00th=[ 367], 10.00th=[ 367], 20.00th=[ 469], | 30.00th=[ 938], 40.00th=[ 938], 50.00th=[ 72877], 60.00th=[910164], | 70.00th=[910164], 80.00th=[918553], 90.00th=[926942], 95.00th=[926942], | 99.00th=[926942], 99.50th=[926942], 99.90th=[926942], 99.95th=[926942], | 99.99th=[926942] bw ( KiB/s): min= 2048, max= 4096, per=49.89%, avg=2560.00, stdev=1024.00, samples=4 iops : min= 2, max= 4, avg= 2.50, stdev= 1.00, samples=4 write: IOPS=1, BW=1283KiB/s (1314kB/s)(5120KiB/3991msec) clat (msec): min=23, max=961, avg=224.60, stdev=412.14 lat (msec): min=24, max=961, avg=224.66, stdev=412.13 clat percentiles (msec): | 1.00th=[ 24], 5.00th=[ 24], 10.00th=[ 24], 20.00th=[ 24], | 30.00th=[ 31], 40.00th=[ 31], 50.00th=[ 43], 60.00th=[ 43], | 70.00th=[ 66], 80.00th=[ 66], 90.00th=[ 961], 95.00th=[ 961], | 99.00th=[ 961], 99.50th=[ 961], 99.90th=[ 961], 99.95th=[ 961], | 99.99th=[ 961] bw ( KiB/s): min= 2048, max= 2048, per=28.51%, avg=2048.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 lat (usec) : 500=16.67%, 1000=8.33% lat (msec) : 50=25.00%, 100=16.67%, 1000=33.33% cpu : usr=0.00%, sys=14.51%, ctx=921, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=27188: Tue Nov 5 07:20:14 2024 read: IOPS=0, BW=515KiB/s (527kB/s)(2048KiB/3978msec) clat (msec): min=109, max=913, avg=511.44, stdev=568.47 lat (msec): min=109, max=913, avg=511.44, stdev=568.47 clat percentiles (msec): | 1.00th=[ 110], 5.00th=[ 110], 10.00th=[ 110], 20.00th=[ 110], | 30.00th=[ 110], 40.00th=[ 110], 50.00th=[ 110], 60.00th=[ 911], | 70.00th=[ 911], 80.00th=[ 911], 90.00th=[ 911], 95.00th=[ 911], | 99.00th=[ 911], 99.50th=[ 911], 99.90th=[ 911], 99.95th=[ 911], | 99.99th=[ 911] bw ( KiB/s): min= 2048, max= 2048, per=39.91%, avg=2048.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 write: IOPS=2, BW=2574KiB/s (2636kB/s)(10.0MiB/3978msec) clat (msec): min=17, max=1005, avg=291.18, stdev=418.78 lat (msec): min=17, max=1005, avg=291.23, stdev=418.80 clat percentiles (msec): | 1.00th=[ 18], 5.00th=[ 18], 10.00th=[ 18], 20.00th=[ 21], | 30.00th=[ 26], 40.00th=[ 28], 50.00th=[ 42], 60.00th=[ 43], | 70.00th=[ 751], 80.00th=[ 751], 90.00th=[ 919], 95.00th=[ 1003], | 99.00th=[ 1003], 99.50th=[ 1003], 99.90th=[ 1003], 99.95th=[ 1003], | 99.99th=[ 1003] bw ( KiB/s): min= 4096, max=10240, per=99.78%, avg=7168.00, stdev=4344.46, samples=2 iops : min= 4, max= 10, avg= 7.00, stdev= 4.24, samples=2 lat (msec) : 20=8.33%, 50=41.67%, 100=8.33%, 250=8.33%, 750=8.33% lat (msec) : 1000=16.67% cpu : usr=0.05%, sys=10.94%, ctx=384, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=27189: Tue Nov 5 07:20:14 2024 read: IOPS=1, BW=1555KiB/s (1593kB/s)(6144KiB/3950msec) clat (usec): min=361, max=950362, avg=296473.54, stdev=437549.05 lat (usec): min=362, max=950362, avg=296474.46, stdev=437548.85 clat percentiles (usec): | 1.00th=[ 363], 5.00th=[ 363], 10.00th=[ 363], 20.00th=[ 420], | 30.00th=[ 420], 40.00th=[ 22938], 50.00th=[ 22938], 60.00th=[ 43779], | 70.00th=[759170], 80.00th=[759170], 90.00th=[952108], 95.00th=[952108], | 99.00th=[952108], 99.50th=[952108], 99.90th=[952108], 99.95th=[952108], | 99.99th=[952108] bw ( KiB/s): min= 2048, max= 6131, per=66.44%, avg=3409.00, stdev=2357.32, samples=3 iops : min= 2, max= 5, avg= 3.00, stdev= 1.73, samples=3 write: IOPS=1, BW=1555KiB/s (1593kB/s)(6144KiB/3950msec) clat (msec): min=34, max=1095, avg=360.86, stdev=496.08 lat (msec): min=34, max=1096, avg=360.91, stdev=496.08 clat percentiles (msec): | 1.00th=[ 35], 5.00th=[ 35], 10.00th=[ 35], 20.00th=[ 37], | 30.00th=[ 37], 40.00th=[ 40], 50.00th=[ 40], 60.00th=[ 63], | 70.00th=[ 894], 80.00th=[ 894], 90.00th=[ 1099], 95.00th=[ 1099], | 99.00th=[ 1099], 99.50th=[ 1099], 99.90th=[ 1099], 99.95th=[ 1099], | 99.99th=[ 1099] bw ( KiB/s): min= 2048, max= 4096, per=47.47%, avg=3410.33, stdev=1179.82, samples=3 iops : min= 2, max= 4, avg= 3.00, stdev= 1.00, samples=3 lat (usec) : 500=16.67% lat (msec) : 50=41.67%, 100=8.33%, 1000=25.00% cpu : usr=0.03%, sys=10.28%, ctx=616, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=27190: Tue Nov 5 07:20:14 2024 read: IOPS=1, BW=1295KiB/s (1326kB/s)(5120KiB/3955msec) clat (usec): min=404, max=907012, avg=522101.95, stdev=459184.70 lat (usec): min=405, max=907014, avg=522103.22, stdev=459185.27 clat percentiles (usec): | 1.00th=[ 404], 5.00th=[ 404], 10.00th=[ 404], 20.00th=[ 404], | 30.00th=[ 43254], 40.00th=[ 43254], 50.00th=[784335], 60.00th=[784335], | 70.00th=[876610], 80.00th=[876610], 90.00th=[910164], 95.00th=[910164], | 99.00th=[910164], 99.50th=[910164], 99.90th=[910164], 99.95th=[910164], | 99.99th=[910164] bw ( KiB/s): min= 2043, max= 2048, per=39.88%, avg=2046.33, stdev= 2.89, samples=3 iops : min= 1, max= 2, avg= 1.67, stdev= 0.58, samples=3 write: IOPS=1, BW=1812KiB/s (1856kB/s)(7168KiB/3955msec) clat (msec): min=26, max=1074, avg=188.19, stdev=390.89 lat (msec): min=26, max=1074, avg=188.27, stdev=390.87 clat percentiles (msec): | 1.00th=[ 27], 5.00th=[ 27], 10.00th=[ 27], 20.00th=[ 33], | 30.00th=[ 42], 40.00th=[ 42], 50.00th=[ 46], 60.00th=[ 47], | 70.00th=[ 47], 80.00th=[ 51], 90.00th=[ 1083], 95.00th=[ 1083], | 99.00th=[ 1083], 99.50th=[ 1083], 99.90th=[ 1083], 99.95th=[ 1083], | 99.99th=[ 1083] bw ( KiB/s): min= 2048, max= 6144, per=57.02%, avg=4096.00, stdev=2896.31, samples=2 iops : min= 2, max= 6, avg= 4.00, stdev= 2.83, samples=2 lat (usec) : 500=8.33% lat (msec) : 50=50.00%, 100=8.33%, 1000=25.00% cpu : usr=0.00%, sys=12.54%, ctx=881, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=5132KiB/s (5255kB/s), 515KiB/s-1796KiB/s (527kB/s-1839kB/s), io=20.0MiB (20.0MB), run=3950-3991msec WRITE: bw=7184KiB/s (7357kB/s), 1283KiB/s-2574KiB/s (1314kB/s-2636kB/s), io=28.0MiB (29.4MB), run=3950-3991msec rand-rw: (groupid=0, jobs=1): err= 0: pid=27192: Tue Nov 5 07:20:14 2024 read: IOPS=1, BW=1759KiB/s (1801kB/s)(7168KiB/4075msec) clat (msec): min=14, max=1138, avg=525.61, stdev=494.50 lat (msec): min=14, max=1138, avg=525.61, stdev=494.50 clat percentiles (msec): | 1.00th=[ 16], 5.00th=[ 16], 10.00th=[ 16], 20.00th=[ 19], | 30.00th=[ 24], 40.00th=[ 24], 50.00th=[ 718], 60.00th=[ 768], | 70.00th=[ 768], 80.00th=[ 1003], 90.00th=[ 1133], 95.00th=[ 1133], | 99.00th=[ 1133], 99.50th=[ 1133], 99.90th=[ 1133], 99.95th=[ 1133], | 99.99th=[ 1133] bw ( KiB/s): min= 2048, max= 2048, per=41.03%, avg=2048.00, stdev= 0.00, samples=4 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=4 write: IOPS=1, BW=1256KiB/s (1287kB/s)(5120KiB/4075msec) clat (msec): min=30, max=110, avg=65.95, stdev=37.08 lat (msec): min=30, max=110, avg=66.02, stdev=37.10 clat percentiles (msec): | 1.00th=[ 31], 5.00th=[ 31], 10.00th=[ 31], 20.00th=[ 31], | 30.00th=[ 36], 40.00th=[ 36], 50.00th=[ 54], 60.00th=[ 54], | 70.00th=[ 101], 80.00th=[ 101], 90.00th=[ 110], 95.00th=[ 110], | 99.00th=[ 110], 99.50th=[ 110], 99.90th=[ 110], 99.95th=[ 110], | 99.99th=[ 110] bw ( KiB/s): min= 2043, max= 2043, per=29.23%, avg=2043.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 20=16.67%, 50=25.00%, 100=8.33%, 250=16.67%, 750=8.33% lat (msec) : 1000=8.33% cpu : usr=0.00%, sys=3.19%, ctx=61, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=27193: Tue Nov 5 07:20:14 2024 read: IOPS=0, BW=516KiB/s (528kB/s)(2048KiB/3969msec) clat (msec): min=117, max=897, avg=507.56, stdev=551.84 lat (msec): min=117, max=897, avg=507.56, stdev=551.84 clat percentiles (msec): | 1.00th=[ 117], 5.00th=[ 117], 10.00th=[ 117], 20.00th=[ 117], | 30.00th=[ 117], 40.00th=[ 117], 50.00th=[ 117], 60.00th=[ 902], | 70.00th=[ 902], 80.00th=[ 902], 90.00th=[ 902], 95.00th=[ 902], | 99.00th=[ 902], 99.50th=[ 902], 99.90th=[ 902], 99.95th=[ 902], | 99.99th=[ 902] bw ( KiB/s): min= 2048, max= 2048, per=41.03%, avg=2048.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 write: IOPS=2, BW=2580KiB/s (2642kB/s)(10.0MiB/3969msec) clat (msec): min=27, max=1023, avg=293.17, stdev=412.11 lat (msec): min=27, max=1023, avg=293.24, stdev=412.12 clat percentiles (msec): | 1.00th=[ 28], 5.00th=[ 28], 10.00th=[ 28], 20.00th=[ 31], | 30.00th=[ 34], 40.00th=[ 34], 50.00th=[ 35], 60.00th=[ 64], | 70.00th=[ 667], 80.00th=[ 667], 90.00th=[ 936], 95.00th=[ 1020], | 99.00th=[ 1020], 99.50th=[ 1020], 99.90th=[ 1020], 99.95th=[ 1020], | 99.99th=[ 1020] bw ( KiB/s): min= 4096, max=10240, per=100.00%, avg=7168.00, stdev=4344.46, samples=2 iops : min= 4, max= 10, avg= 7.00, stdev= 4.24, samples=2 lat (msec) : 50=41.67%, 100=16.67%, 250=8.33%, 750=8.33%, 1000=16.67% cpu : usr=0.00%, sys=3.20%, ctx=62, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=27194: Tue Nov 5 07:20:14 2024 read: IOPS=1, BW=1498KiB/s (1534kB/s)(6144KiB/4102msec) clat (msec): min=23, max=581, avg=132.50, stdev=221.24 lat (msec): min=23, max=581, avg=132.50, stdev=221.24 clat percentiles (msec): | 1.00th=[ 24], 5.00th=[ 24], 10.00th=[ 24], 20.00th=[ 26], | 30.00th=[ 26], 40.00th=[ 35], 50.00th=[ 35], 60.00th=[ 44], | 70.00th=[ 88], 80.00th=[ 88], 90.00th=[ 584], 95.00th=[ 584], | 99.00th=[ 584], 99.50th=[ 584], 99.90th=[ 584], 99.95th=[ 584], | 99.99th=[ 584] bw ( KiB/s): min= 4096, max= 4096, per=82.05%, avg=4096.00, stdev= 0.00, samples=2 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=2 write: IOPS=1, BW=1498KiB/s (1534kB/s)(6144KiB/4102msec) clat (msec): min=31, max=1989, avg=547.59, stdev=791.78 lat (msec): min=31, max=1989, avg=547.66, stdev=791.77 clat percentiles (msec): | 1.00th=[ 32], 5.00th=[ 32], 10.00th=[ 32], 20.00th=[ 42], | 30.00th=[ 42], 40.00th=[ 70], 50.00th=[ 70], 60.00th=[ 188], | 70.00th=[ 969], 80.00th=[ 969], 90.00th=[ 1989], 95.00th=[ 1989], | 99.00th=[ 1989], 99.50th=[ 1989], 99.90th=[ 1989], 99.95th=[ 1989], | 99.99th=[ 1989] bw ( KiB/s): min= 2048, max= 4096, per=36.63%, avg=2560.00, stdev=1024.00, samples=4 iops : min= 2, max= 4, avg= 2.50, stdev= 1.00, samples=4 lat (msec) : 50=50.00%, 100=16.67%, 250=8.33%, 750=8.33%, 1000=8.33% cpu : usr=0.00%, sys=2.34%, ctx=168, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=27195: Tue Nov 5 07:20:14 2024 read: IOPS=1, BW=1255KiB/s (1285kB/s)(5120KiB/4081msec) clat (msec): min=12, max=912, avg=501.74, stdev=451.78 lat (msec): min=12, max=912, avg=501.74, stdev=451.78 clat percentiles (msec): | 1.00th=[ 13], 5.00th=[ 13], 10.00th=[ 13], 20.00th=[ 13], | 30.00th=[ 20], 40.00th=[ 20], 50.00th=[ 684], 60.00th=[ 684], | 70.00th=[ 877], 80.00th=[ 877], 90.00th=[ 911], 95.00th=[ 911], | 99.00th=[ 911], 99.50th=[ 911], 99.90th=[ 911], 99.95th=[ 911], | 99.99th=[ 911] bw ( KiB/s): min= 2048, max= 2048, per=41.03%, avg=2048.00, stdev= 0.00, samples=4 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=4 write: IOPS=1, BW=1756KiB/s (1799kB/s)(7168KiB/4081msec) clat (msec): min=24, max=1079, avg=218.11, stdev=382.88 lat (msec): min=24, max=1079, avg=218.18, stdev=382.88 clat percentiles (msec): | 1.00th=[ 25], 5.00th=[ 25], 10.00th=[ 25], 20.00th=[ 33], | 30.00th=[ 47], 40.00th=[ 47], 50.00th=[ 58], 60.00th=[ 140], | 70.00th=[ 140], 80.00th=[ 148], 90.00th=[ 1083], 95.00th=[ 1083], | 99.00th=[ 1083], 99.50th=[ 1083], 99.90th=[ 1083], 99.95th=[ 1083], | 99.99th=[ 1083] bw ( KiB/s): min= 2048, max= 6144, per=48.84%, avg=3413.33, stdev=2364.83, samples=3 iops : min= 2, max= 6, avg= 3.33, stdev= 2.31, samples=3 lat (msec) : 20=16.67%, 50=25.00%, 100=8.33%, 250=16.67%, 750=8.33% lat (msec) : 1000=16.67% cpu : usr=0.02%, sys=2.33%, ctx=46, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=4993KiB/s (5113kB/s), 516KiB/s-1759KiB/s (528kB/s-1801kB/s), io=20.0MiB (20.0MB), run=3969-4102msec WRITE: bw=6990KiB/s (7158kB/s), 1256KiB/s-2580KiB/s (1287kB/s-2642kB/s), io=28.0MiB (29.4MB), run=3969-4102msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=27204: Tue Nov 5 07:20:20 2024 read: IOPS=0, BW=2405KiB/s (2462kB/s)(12.0MiB/5110msec) clat (msec): min=155, max=3702, avg=1697.35, stdev=1818.58 lat (msec): min=155, max=3702, avg=1697.35, stdev=1818.58 clat percentiles (msec): | 1.00th=[ 155], 5.00th=[ 155], 10.00th=[ 155], 20.00th=[ 155], | 30.00th=[ 155], 40.00th=[ 1234], 50.00th=[ 1234], 60.00th=[ 1234], | 70.00th=[ 3708], 80.00th=[ 3708], 90.00th=[ 3708], 95.00th=[ 3708], | 99.00th=[ 3708], 99.50th=[ 3708], 99.90th=[ 3708], 99.95th=[ 3708], | 99.99th=[ 3708] bw ( KiB/s): min= 8175, max= 8192, per=100.00%, avg=8183.50, stdev=12.02, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 250=33.33% cpu : usr=0.04%, sys=18.44%, ctx=1684, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=27205: Tue Nov 5 07:20:20 2024 read: IOPS=0, BW=774KiB/s (792kB/s)(4096KiB/5293msec) clat (nsec): min=143519k, max=143519k, avg=143519247.00, stdev= 0.00 lat (nsec): min=143521k, max=143521k, avg=143521277.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 144], 5.00th=[ 144], 10.00th=[ 144], 20.00th=[ 144], | 30.00th=[ 144], 40.00th=[ 144], 50.00th=[ 144], 60.00th=[ 144], | 70.00th=[ 144], 80.00th=[ 144], 90.00th=[ 144], 95.00th=[ 144], | 99.00th=[ 144], 99.50th=[ 144], 99.90th=[ 144], 99.95th=[ 144], | 99.99th=[ 144] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=0, BW=1548KiB/s (1585kB/s)(8192KiB/5293msec) clat (msec): min=1940, max=3202, avg=2571.43, stdev=892.06 lat (msec): min=1941, max=3202, avg=2571.79, stdev=892.06 clat percentiles (msec): | 1.00th=[ 1938], 5.00th=[ 1938], 10.00th=[ 1938], 20.00th=[ 1938], | 30.00th=[ 1938], 40.00th=[ 1938], 50.00th=[ 1938], 60.00th=[ 3205], | 70.00th=[ 3205], 80.00th=[ 3205], 90.00th=[ 3205], 95.00th=[ 3205], | 99.00th=[ 3205], 99.50th=[ 3205], 99.90th=[ 3205], 99.95th=[ 3205], | 99.99th=[ 3205] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33% cpu : usr=0.00%, sys=6.12%, ctx=136, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=27206: Tue Nov 5 07:20:20 2024 write: IOPS=0, BW=2381KiB/s (2439kB/s)(12.0MiB/5160msec) clat (msec): min=200, max=3667, avg=1717.29, stdev=1773.66 lat (msec): min=200, max=3667, avg=1717.52, stdev=1773.58 clat percentiles (msec): | 1.00th=[ 201], 5.00th=[ 201], 10.00th=[ 201], 20.00th=[ 201], | 30.00th=[ 201], 40.00th=[ 1284], 50.00th=[ 1284], 60.00th=[ 1284], | 70.00th=[ 3675], 80.00th=[ 3675], 90.00th=[ 3675], 95.00th=[ 3675], | 99.00th=[ 3675], 99.50th=[ 3675], 99.90th=[ 3675], 99.95th=[ 3675], | 99.99th=[ 3675] bw ( KiB/s): min= 8175, max= 8192, per=100.00%, avg=8183.50, stdev=12.02, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 250=33.33% cpu : usr=0.02%, sys=6.75%, ctx=320, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=27207: Tue Nov 5 07:20:20 2024 write: IOPS=0, BW=3007KiB/s (3079kB/s)(12.0MiB/4087msec) clat (msec): min=172, max=3718, avg=1356.74, stdev=2045.03 lat (msec): min=173, max=3718, avg=1356.97, stdev=2045.10 clat percentiles (msec): | 1.00th=[ 174], 5.00th=[ 174], 10.00th=[ 174], 20.00th=[ 174], | 30.00th=[ 174], 40.00th=[ 180], 50.00th=[ 180], 60.00th=[ 180], | 70.00th=[ 3708], 80.00th=[ 3708], 90.00th=[ 3708], 95.00th=[ 3708], | 99.00th=[ 3708], 99.50th=[ 3708], 99.90th=[ 3708], 99.95th=[ 3708], | 99.99th=[ 3708] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 lat (msec) : 250=66.67% cpu : usr=0.00%, sys=6.00%, ctx=70, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=3095KiB/s (3170kB/s), 774KiB/s-2405KiB/s (792kB/s-2462kB/s), io=16.0MiB (16.8MB), run=5110-5293msec WRITE: bw=6191KiB/s (6339kB/s), 1548KiB/s-3007KiB/s (1585kB/s-3079kB/s), io=32.0MiB (33.6MB), run=4087-5293msec rand-rw: (groupid=0, jobs=1): err= 0: pid=27208: Tue Nov 5 07:20:20 2024 read: IOPS=0, BW=2367KiB/s (2424kB/s)(12.0MiB/5192msec) clat (msec): min=184, max=3909, avg=1722.29, stdev=1945.60 lat (msec): min=184, max=3909, avg=1722.29, stdev=1945.60 clat percentiles (msec): | 1.00th=[ 184], 5.00th=[ 184], 10.00th=[ 184], 20.00th=[ 184], | 30.00th=[ 184], 40.00th=[ 1070], 50.00th=[ 1070], 60.00th=[ 1070], | 70.00th=[ 3910], 80.00th=[ 3910], 90.00th=[ 3910], 95.00th=[ 3910], | 99.00th=[ 3910], 99.50th=[ 3910], 99.90th=[ 3910], 99.95th=[ 3910], | 99.99th=[ 3910] bw ( KiB/s): min= 7922, max= 8192, per=100.00%, avg=8057.00, stdev=190.92, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 250=33.33% cpu : usr=0.00%, sys=1.31%, ctx=28, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=27209: Tue Nov 5 07:20:20 2024 read: IOPS=0, BW=761KiB/s (779kB/s)(4096KiB/5381msec) clat (nsec): min=252146k, max=252146k, avg=252146071.00, stdev= 0.00 lat (nsec): min=252147k, max=252147k, avg=252147064.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 253], 5.00th=[ 253], 10.00th=[ 253], 20.00th=[ 253], | 30.00th=[ 253], 40.00th=[ 253], 50.00th=[ 253], 60.00th=[ 253], | 70.00th=[ 253], 80.00th=[ 253], 90.00th=[ 253], 95.00th=[ 253], | 99.00th=[ 253], 99.50th=[ 253], 99.90th=[ 253], 99.95th=[ 253], | 99.99th=[ 253] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=0, BW=1522KiB/s (1559kB/s)(8192KiB/5381msec) clat (msec): min=1850, max=3267, avg=2558.85, stdev=1001.57 lat (msec): min=1850, max=3267, avg=2559.12, stdev=1001.74 clat percentiles (msec): | 1.00th=[ 1854], 5.00th=[ 1854], 10.00th=[ 1854], 20.00th=[ 1854], | 30.00th=[ 1854], 40.00th=[ 1854], 50.00th=[ 1854], 60.00th=[ 3272], | 70.00th=[ 3272], 80.00th=[ 3272], 90.00th=[ 3272], 95.00th=[ 3272], | 99.00th=[ 3272], 99.50th=[ 3272], 99.90th=[ 3272], 99.95th=[ 3272], | 99.99th=[ 3272] bw ( KiB/s): min= 5490, max= 5490, per=90.16%, avg=5490.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 500=33.33% cpu : usr=0.00%, sys=1.80%, ctx=72, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=27210: Tue Nov 5 07:20:20 2024 write: IOPS=0, BW=2341KiB/s (2397kB/s)(12.0MiB/5250msec) clat (msec): min=128, max=4872, avg=1742.32, stdev=2711.27 lat (msec): min=129, max=4872, avg=1742.70, stdev=2711.34 clat percentiles (msec): | 1.00th=[ 129], 5.00th=[ 129], 10.00th=[ 129], 20.00th=[ 129], | 30.00th=[ 129], 40.00th=[ 226], 50.00th=[ 226], 60.00th=[ 226], | 70.00th=[ 4866], 80.00th=[ 4866], 90.00th=[ 4866], 95.00th=[ 4866], | 99.00th=[ 4866], 99.50th=[ 4866], 99.90th=[ 4866], 99.95th=[ 4866], | 99.99th=[ 4866] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67% cpu : usr=0.04%, sys=1.73%, ctx=26, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=27211: Tue Nov 5 07:20:20 2024 write: IOPS=1, BW=5939KiB/s (6082kB/s)(12.0MiB/2069msec) clat (msec): min=82, max=1881, avg=683.89, stdev=1037.16 lat (msec): min=82, max=1882, avg=684.56, stdev=1037.45 clat percentiles (msec): | 1.00th=[ 83], 5.00th=[ 83], 10.00th=[ 83], 20.00th=[ 83], | 30.00th=[ 83], 40.00th=[ 88], 50.00th=[ 88], 60.00th=[ 88], | 70.00th=[ 1888], 80.00th=[ 1888], 90.00th=[ 1888], 95.00th=[ 1888], | 99.00th=[ 1888], 99.50th=[ 1888], 99.90th=[ 1888], 99.95th=[ 1888], | 99.99th=[ 1888] bw ( KiB/s): min= 8159, max= 8192, per=100.00%, avg=8175.50, stdev=23.33, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 100=66.67% cpu : usr=0.10%, sys=4.26%, ctx=77, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=3045KiB/s (3118kB/s), 761KiB/s-2367KiB/s (779kB/s-2424kB/s), io=16.0MiB (16.8MB), run=5192-5381msec WRITE: bw=6090KiB/s (6236kB/s), 1522KiB/s-5939KiB/s (1559kB/s-6082kB/s), io=32.0MiB (33.6MB), run=2069-5381msec