== sanity test 398b: DIO and buffer IO race ============== 02:00:48 (1744092048) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.0593 s, 47.5 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=9116: Tue Apr 8 02:01:03 2025 read: IOPS=122, BW=490KiB/s (501kB/s)(5992KiB/12236msec) clat (usec): min=54, max=181433, avg=3880.53, stdev=5742.03 lat (usec): min=55, max=181433, avg=3881.18, stdev=5742.01 clat percentiles (usec): | 1.00th=[ 66], 5.00th=[ 79], 10.00th=[ 95], 20.00th=[ 1975], | 30.00th=[ 2278], 40.00th=[ 2606], 50.00th=[ 3032], 60.00th=[ 3556], | 70.00th=[ 4424], 80.00th=[ 5669], 90.00th=[ 7177], 95.00th=[ 8848], | 99.00th=[ 15270], 99.50th=[ 17957], 99.90th=[ 54789], 99.95th=[181404], | 99.99th=[181404] bw ( KiB/s): min= 272, max= 624, per=24.47%, avg=480.17, stdev=79.58, samples=24 iops : min= 68, max= 156, avg=119.96, stdev=19.95, samples=24 write: IOPS=128, BW=515KiB/s (527kB/s)(6296KiB/12236msec) clat (usec): min=437, max=25151, avg=4020.93, stdev=3613.00 lat (usec): min=438, max=25152, avg=4021.68, stdev=3612.99 clat percentiles (usec): | 1.00th=[ 502], 5.00th=[ 537], 10.00th=[ 562], 20.00th=[ 644], | 30.00th=[ 947], 40.00th=[ 2278], 50.00th=[ 3326], 60.00th=[ 4228], | 70.00th=[ 5276], 80.00th=[ 6718], 90.00th=[ 8848], 95.00th=[10945], | 99.00th=[15533], 99.50th=[17957], 99.90th=[25035], 99.95th=[25035], | 99.99th=[25035] bw ( KiB/s): min= 328, max= 688, per=24.59%, avg=505.17, stdev=88.67, samples=24 iops : min= 82, max= 172, avg=126.21, stdev=22.26, samples=24 lat (usec) : 100=5.11%, 250=1.89%, 500=0.59%, 750=12.34%, 1000=3.16% lat (msec) : 2=6.97%, 4=31.77%, 10=33.04%, 20=4.85%, 50=0.23% lat (msec) : 100=0.03%, 250=0.03% cpu : usr=0.11%, sys=23.31%, ctx=5761, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9117: Tue Apr 8 02:01:03 2025 read: IOPS=122, BW=489KiB/s (501kB/s)(5940KiB/12146msec) clat (usec): min=51, max=29860, avg=3595.15, stdev=2792.49 lat (usec): min=52, max=29860, avg=3596.34, stdev=2792.32 clat percentiles (usec): | 1.00th=[ 66], 5.00th=[ 75], 10.00th=[ 96], 20.00th=[ 1958], | 30.00th=[ 2311], 40.00th=[ 2606], 50.00th=[ 2999], 60.00th=[ 3490], | 70.00th=[ 4228], 80.00th=[ 5538], 90.00th=[ 6980], 95.00th=[ 8455], | 99.00th=[11731], 99.50th=[17433], 99.90th=[23462], 99.95th=[29754], | 99.99th=[29754] bw ( KiB/s): min= 304, max= 600, per=24.80%, avg=486.54, stdev=74.13, samples=24 iops : min= 76, max= 150, avg=121.54, stdev=18.61, samples=24 write: IOPS=130, BW=523KiB/s (535kB/s)(6348KiB/12146msec) clat (usec): min=411, max=180299, avg=4228.86, stdev=6041.50 lat (usec): min=412, max=180300, avg=4230.06, stdev=6041.73 clat percentiles (usec): | 1.00th=[ 502], 5.00th=[ 537], 10.00th=[ 570], 20.00th=[ 660], | 30.00th=[ 922], 40.00th=[ 2057], 50.00th=[ 3392], 60.00th=[ 4228], | 70.00th=[ 5342], 80.00th=[ 6652], 90.00th=[ 8848], 95.00th=[ 11338], | 99.00th=[ 16909], 99.50th=[ 30016], 99.90th=[ 50070], 99.95th=[179307], | 99.99th=[179307] bw ( KiB/s): min= 327, max= 664, per=25.10%, avg=515.50, stdev=81.71, samples=24 iops : min= 81, max= 166, avg=128.79, stdev=20.52, samples=24 lat (usec) : 100=5.24%, 250=1.89%, 500=0.49%, 750=12.14%, 1000=3.87% lat (msec) : 2=6.84%, 4=31.87%, 10=32.85%, 20=4.36%, 50=0.39% lat (msec) : 100=0.03%, 250=0.03% cpu : usr=0.13%, sys=23.45%, ctx=5718, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9118: Tue Apr 8 02:01:03 2025 read: IOPS=124, BW=500KiB/s (512kB/s)(6028KiB/12064msec) clat (usec): min=53, max=179721, avg=3761.34, stdev=5515.80 lat (usec): min=54, max=179722, avg=3761.98, stdev=5515.80 clat percentiles (usec): | 1.00th=[ 63], 5.00th=[ 73], 10.00th=[ 84], 20.00th=[ 1893], | 30.00th=[ 2311], 40.00th=[ 2638], 50.00th=[ 3032], 60.00th=[ 3523], | 70.00th=[ 4424], 80.00th=[ 5538], 90.00th=[ 7111], 95.00th=[ 8586], | 99.00th=[ 13304], 99.50th=[ 20317], 99.90th=[ 40109], 99.95th=[179307], | 99.99th=[179307] bw ( KiB/s): min= 304, max= 712, per=25.38%, avg=497.96, stdev=99.09, samples=24 iops : min= 76, max= 178, avg=124.46, stdev=24.79, samples=24 write: IOPS=129, BW=519KiB/s (531kB/s)(6260KiB/12064msec) clat (usec): min=454, max=41073, avg=4026.07, stdev=4007.94 lat (usec): min=455, max=41074, avg=4026.93, stdev=4008.07 clat percentiles (usec): | 1.00th=[ 506], 5.00th=[ 537], 10.00th=[ 562], 20.00th=[ 652], | 30.00th=[ 898], 40.00th=[ 1762], 50.00th=[ 3261], 60.00th=[ 4146], | 70.00th=[ 5145], 80.00th=[ 6652], 90.00th=[ 8979], 95.00th=[11207], | 99.00th=[16319], 99.50th=[21890], 99.90th=[38011], 99.95th=[41157], | 99.99th=[41157] bw ( KiB/s): min= 312, max= 680, per=25.12%, avg=515.96, stdev=84.94, samples=24 iops : min= 78, max= 170, avg=128.96, stdev=21.30, samples=24 lat (usec) : 100=6.28%, 250=2.05%, 500=0.65%, 750=12.17%, 1000=3.78% lat (msec) : 2=7.03%, 4=30.27%, 10=32.88%, 20=4.33%, 50=0.52% lat (msec) : 250=0.03% cpu : usr=0.21%, sys=23.49%, ctx=5761, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9120: Tue Apr 8 02:01:03 2025 read: IOPS=125, BW=503KiB/s (515kB/s)(6052KiB/12034msec) clat (usec): min=53, max=29800, avg=3595.66, stdev=2882.23 lat (usec): min=54, max=29801, avg=3596.95, stdev=2882.41 clat percentiles (usec): | 1.00th=[ 64], 5.00th=[ 72], 10.00th=[ 83], 20.00th=[ 1893], | 30.00th=[ 2311], 40.00th=[ 2638], 50.00th=[ 3032], 60.00th=[ 3556], | 70.00th=[ 4490], 80.00th=[ 5473], 90.00th=[ 7111], 95.00th=[ 8356], | 99.00th=[13173], 99.50th=[16319], 99.90th=[27132], 99.95th=[29754], | 99.99th=[29754] bw ( KiB/s): min= 280, max= 720, per=25.61%, avg=502.46, stdev=88.87, samples=24 iops : min= 70, max= 180, avg=125.54, stdev=22.25, samples=24 write: IOPS=129, BW=518KiB/s (531kB/s)(6236KiB/12034msec) clat (usec): min=425, max=176041, avg=4166.95, stdev=5827.70 lat (usec): min=426, max=176042, avg=4172.11, stdev=5836.94 clat percentiles (usec): | 1.00th=[ 502], 5.00th=[ 537], 10.00th=[ 570], 20.00th=[ 676], | 30.00th=[ 938], 40.00th=[ 2089], 50.00th=[ 3326], 60.00th=[ 4146], | 70.00th=[ 5211], 80.00th=[ 6652], 90.00th=[ 8848], 95.00th=[ 10945], | 99.00th=[ 17171], 99.50th=[ 21627], 99.90th=[ 37487], 99.95th=[175113], | 99.99th=[175113] bw ( KiB/s): min= 303, max= 752, per=25.16%, avg=516.83, stdev=97.11, samples=24 iops : min= 75, max= 188, avg=129.12, stdev=24.36, samples=24 lat (usec) : 100=6.84%, 250=1.76%, 500=0.46%, 750=11.72%, 1000=3.87% lat (msec) : 2=6.09%, 4=30.76%, 10=34.05%, 20=4.00%, 50=0.42% lat (msec) : 250=0.03% cpu : usr=0.14%, sys=23.47%, ctx=5614, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1962KiB/s (2010kB/s), 489KiB/s-503KiB/s (501kB/s-515kB/s), io=23.4MiB (24.6MB), run=12034-12236msec WRITE: bw=2055KiB/s (2104kB/s), 515KiB/s-523KiB/s (527kB/s-535kB/s), io=24.6MiB (25.7MB), run=12034-12236msec rand-rw: (groupid=0, jobs=1): err= 0: pid=9121: Tue Apr 8 02:01:24 2025 read: IOPS=45, BW=182KiB/s (186kB/s)(5992KiB/32985msec) clat (usec): min=1476, max=183452, avg=3708.68, stdev=5455.79 lat (usec): min=1477, max=183453, avg=3709.38, stdev=5455.82 clat percentiles (usec): | 1.00th=[ 1631], 5.00th=[ 1795], 10.00th=[ 1860], 20.00th=[ 1958], | 30.00th=[ 2040], 40.00th=[ 2180], 50.00th=[ 2376], 60.00th=[ 2900], | 70.00th=[ 3556], 80.00th=[ 4424], 90.00th=[ 7046], 95.00th=[ 9503], | 99.00th=[ 15926], 99.50th=[ 19530], 99.90th=[ 27395], 99.95th=[183501], | 99.99th=[183501] bw ( KiB/s): min= 72, max= 312, per=24.91%, avg=181.08, stdev=66.13, samples=65 iops : min= 18, max= 78, avg=45.20, stdev=16.59, samples=65 write: IOPS=47, BW=191KiB/s (195kB/s)(6296KiB/32985msec) clat (usec): min=9203, max=58415, avg=17384.10, stdev=6018.34 lat (usec): min=9204, max=58416, avg=17384.94, stdev=6018.36 clat percentiles (usec): | 1.00th=[10028], 5.00th=[10814], 10.00th=[11338], 20.00th=[12256], | 30.00th=[13435], 40.00th=[15139], 50.00th=[16581], 60.00th=[17695], | 70.00th=[18744], 80.00th=[20579], 90.00th=[25297], 95.00th=[28967], | 99.00th=[36963], 99.50th=[42730], 99.90th=[57410], 99.95th=[58459], | 99.99th=[58459] bw ( KiB/s): min= 72, max= 248, per=24.91%, avg=189.57, stdev=50.77, samples=65 iops : min= 18, max= 62, avg=47.32, stdev=12.70, samples=65 lat (msec) : 2=12.73%, 4=24.87%, 10=9.80%, 20=41.18%, 50=11.26% lat (msec) : 100=0.13%, 250=0.03% cpu : usr=0.12%, sys=5.55%, ctx=3735, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9122: Tue Apr 8 02:01:24 2025 read: IOPS=44, BW=180KiB/s (184kB/s)(5940KiB/33005msec) clat (usec): min=1441, max=145677, avg=3700.89, stdev=4826.92 lat (usec): min=1441, max=145679, avg=3701.55, stdev=4826.94 clat percentiles (usec): | 1.00th=[ 1631], 5.00th=[ 1762], 10.00th=[ 1844], 20.00th=[ 1942], | 30.00th=[ 2040], 40.00th=[ 2180], 50.00th=[ 2376], 60.00th=[ 2999], | 70.00th=[ 3556], 80.00th=[ 4293], 90.00th=[ 6783], 95.00th=[ 9241], | 99.00th=[ 15795], 99.50th=[ 22414], 99.90th=[ 40633], 99.95th=[145753], | 99.99th=[145753] bw ( KiB/s): min= 48, max= 328, per=24.78%, avg=180.12, stdev=69.71, samples=65 iops : min= 12, max= 82, avg=44.98, stdev=17.42, samples=65 write: IOPS=48, BW=192KiB/s (197kB/s)(6348KiB/33005msec) clat (usec): min=8465, max=65354, avg=17282.45, stdev=5817.79 lat (usec): min=8466, max=65355, avg=17283.29, stdev=5817.77 clat percentiles (usec): | 1.00th=[10028], 5.00th=[10683], 10.00th=[11338], 20.00th=[12256], | 30.00th=[13435], 40.00th=[15008], 50.00th=[16319], 60.00th=[17433], | 70.00th=[18744], 80.00th=[20841], 90.00th=[25297], 95.00th=[28967], | 99.00th=[34866], 99.50th=[40109], 99.90th=[51119], 99.95th=[65274], | 99.99th=[65274] bw ( KiB/s): min= 96, max= 248, per=25.14%, avg=191.32, stdev=49.95, samples=65 iops : min= 24, max= 62, avg=47.78, stdev=12.51, samples=65 lat (msec) : 2=12.24%, 4=24.93%, 10=9.70%, 20=41.08%, 50=11.95% lat (msec) : 100=0.07%, 250=0.03% cpu : usr=0.12%, sys=5.52%, ctx=3704, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9123: Tue Apr 8 02:01:24 2025 read: IOPS=45, BW=183KiB/s (187kB/s)(6028KiB/33021msec) clat (usec): min=1499, max=36820, avg=3841.19, stdev=3405.49 lat (usec): min=1499, max=36820, avg=3841.87, stdev=3405.52 clat percentiles (usec): | 1.00th=[ 1647], 5.00th=[ 1795], 10.00th=[ 1860], 20.00th=[ 1958], | 30.00th=[ 2057], 40.00th=[ 2180], 50.00th=[ 2474], 60.00th=[ 3228], | 70.00th=[ 3752], 80.00th=[ 4752], 90.00th=[ 7570], 95.00th=[ 9896], | 99.00th=[18482], 99.50th=[23987], 99.90th=[32113], 99.95th=[36963], | 99.99th=[36963] bw ( KiB/s): min= 80, max= 328, per=24.99%, avg=181.69, stdev=61.27, samples=65 iops : min= 20, max= 82, avg=45.35, stdev=15.34, samples=65 write: IOPS=47, BW=190KiB/s (194kB/s)(6260KiB/33021msec) clat (msec): min=9, max=148, avg=17.35, stdev= 6.73 lat (msec): min=9, max=148, avg=17.35, stdev= 6.73 clat percentiles (msec): | 1.00th=[ 10], 5.00th=[ 11], 10.00th=[ 12], 20.00th=[ 13], | 30.00th=[ 14], 40.00th=[ 15], 50.00th=[ 17], 60.00th=[ 18], | 70.00th=[ 19], 80.00th=[ 21], 90.00th=[ 26], 95.00th=[ 29], | 99.00th=[ 37], 99.50th=[ 42], 99.90th=[ 59], 99.95th=[ 148], | 99.99th=[ 148] bw ( KiB/s): min= 88, max= 264, per=24.76%, avg=188.46, stdev=52.79, samples=65 iops : min= 22, max= 66, avg=47.05, stdev=13.21, samples=65 lat (msec) : 2=12.24%, 4=23.96%, 10=11.17%, 20=40.92%, 50=11.65% lat (msec) : 100=0.03%, 250=0.03% cpu : usr=0.10%, sys=5.58%, ctx=3739, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9124: Tue Apr 8 02:01:24 2025 read: IOPS=46, BW=185KiB/s (190kB/s)(6052KiB/32634msec) clat (usec): min=1453, max=35912, avg=3601.14, stdev=2976.97 lat (usec): min=1453, max=35912, avg=3601.81, stdev=2976.98 clat percentiles (usec): | 1.00th=[ 1663], 5.00th=[ 1795], 10.00th=[ 1876], 20.00th=[ 1958], | 30.00th=[ 2040], 40.00th=[ 2180], 50.00th=[ 2409], 60.00th=[ 3032], | 70.00th=[ 3589], 80.00th=[ 4424], 90.00th=[ 6456], 95.00th=[ 9110], | 99.00th=[16319], 99.50th=[21365], 99.90th=[28181], 99.95th=[35914], | 99.99th=[35914] bw ( KiB/s): min= 64, max= 368, per=25.51%, avg=185.49, stdev=66.33, samples=65 iops : min= 16, max= 92, avg=46.29, stdev=16.59, samples=65 write: IOPS=47, BW=191KiB/s (196kB/s)(6236KiB/32634msec) clat (msec): min=8, max=174, avg=17.40, stdev= 7.01 lat (msec): min=8, max=174, avg=17.40, stdev= 7.01 clat percentiles (msec): | 1.00th=[ 11], 5.00th=[ 11], 10.00th=[ 12], 20.00th=[ 13], | 30.00th=[ 14], 40.00th=[ 16], 50.00th=[ 17], 60.00th=[ 18], | 70.00th=[ 19], 80.00th=[ 21], 90.00th=[ 26], 95.00th=[ 29], | 99.00th=[ 37], 99.50th=[ 40], 99.90th=[ 65], 99.95th=[ 176], | 99.99th=[ 176] bw ( KiB/s): min= 88, max= 256, per=25.05%, avg=190.66, stdev=49.61, samples=65 iops : min= 22, max= 64, avg=47.58, stdev=12.42, samples=65 lat (msec) : 2=12.73%, 4=24.77%, 10=10.42%, 20=40.14%, 50=11.85% lat (msec) : 100=0.07%, 250=0.03% cpu : usr=0.09%, sys=5.64%, ctx=3770, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=727KiB/s (745kB/s), 180KiB/s-185KiB/s (184kB/s-190kB/s), io=23.4MiB (24.6MB), run=32634-33021msec WRITE: bw=761KiB/s (780kB/s), 190KiB/s-192KiB/s (194kB/s-197kB/s), io=24.6MiB (25.7MB), run=32634-33021msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=9167: Tue Apr 8 02:01:27 2025 read: IOPS=151, BW=2427KiB/s (2485kB/s)(5920KiB/2439msec) clat (usec): min=58, max=14562, avg=3205.18, stdev=2337.24 lat (usec): min=58, max=14562, avg=3205.76, stdev=2337.27 clat percentiles (usec): | 1.00th=[ 63], 5.00th=[ 72], 10.00th=[ 82], 20.00th=[ 125], | 30.00th=[ 2343], 40.00th=[ 2737], 50.00th=[ 3130], 60.00th=[ 3523], | 70.00th=[ 4113], 80.00th=[ 5014], 90.00th=[ 6128], 95.00th=[ 7046], | 99.00th=[ 9241], 99.50th=[11600], 99.90th=[14615], 99.95th=[14615], | 99.99th=[14615] bw ( KiB/s): min= 1792, max= 3040, per=25.33%, avg=2400.00, stdev=517.97, samples=4 iops : min= 112, max= 190, avg=150.00, stdev=32.37, samples=4 write: IOPS=163, BW=2611KiB/s (2674kB/s)(6368KiB/2439msec) clat (usec): min=639, max=19514, avg=3086.64, stdev=2914.65 lat (usec): min=640, max=19515, avg=3087.51, stdev=2914.66 clat percentiles (usec): | 1.00th=[ 693], 5.00th=[ 758], 10.00th=[ 807], 20.00th=[ 922], | 30.00th=[ 1012], 40.00th=[ 1237], 50.00th=[ 1614], 60.00th=[ 2737], | 70.00th=[ 3884], 80.00th=[ 5211], 90.00th=[ 6915], 95.00th=[ 8848], | 99.00th=[15270], 99.50th=[15926], 99.90th=[19530], 99.95th=[19530], | 99.99th=[19530] bw ( KiB/s): min= 1952, max= 3392, per=26.24%, avg=2616.00, stdev=611.85, samples=4 iops : min= 122, max= 212, avg=163.50, stdev=38.24, samples=4 lat (usec) : 100=8.33%, 250=2.60%, 500=0.13%, 750=2.60%, 1000=12.89% lat (msec) : 2=12.50%, 4=30.34%, 10=28.39%, 20=2.21% cpu : usr=0.12%, sys=29.00%, ctx=1729, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9169: Tue Apr 8 02:01:27 2025 read: IOPS=146, BW=2350KiB/s (2407kB/s)(5744KiB/2444msec) clat (usec): min=62, max=17588, avg=2918.59, stdev=2493.93 lat (usec): min=62, max=17589, avg=2920.70, stdev=2492.34 clat percentiles (usec): | 1.00th=[ 69], 5.00th=[ 73], 10.00th=[ 77], 20.00th=[ 91], | 30.00th=[ 1893], 40.00th=[ 2474], 50.00th=[ 2769], 60.00th=[ 3326], | 70.00th=[ 3785], 80.00th=[ 4621], 90.00th=[ 5997], 95.00th=[ 7242], | 99.00th=[11076], 99.50th=[15533], 99.90th=[17695], 99.95th=[17695], | 99.99th=[17695] bw ( KiB/s): min= 1696, max= 2816, per=25.16%, avg=2384.00, stdev=482.13, samples=4 iops : min= 106, max= 176, avg=149.00, stdev=30.13, samples=4 write: IOPS=167, BW=2678KiB/s (2742kB/s)(6544KiB/2444msec) clat (usec): min=664, max=24815, avg=3150.25, stdev=3015.12 lat (usec): min=664, max=24817, avg=3151.13, stdev=3015.12 clat percentiles (usec): | 1.00th=[ 717], 5.00th=[ 775], 10.00th=[ 807], 20.00th=[ 881], | 30.00th=[ 1020], 40.00th=[ 1254], 50.00th=[ 1991], 60.00th=[ 3130], | 70.00th=[ 3949], 80.00th=[ 4883], 90.00th=[ 7111], 95.00th=[ 8455], | 99.00th=[12387], 99.50th=[13566], 99.90th=[24773], 99.95th=[24773], | 99.99th=[24773] bw ( KiB/s): min= 1952, max= 3200, per=26.57%, avg=2648.00, stdev=539.51, samples=4 iops : min= 122, max= 200, avg=165.50, stdev=33.72, samples=4 lat (usec) : 100=10.03%, 250=2.73%, 500=0.65%, 750=1.69%, 1000=13.54% lat (msec) : 2=12.50%, 4=30.73%, 10=25.52%, 20=2.34%, 50=0.26% cpu : usr=0.20%, sys=28.08%, ctx=1750, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9170: Tue Apr 8 02:01:27 2025 read: IOPS=154, BW=2475KiB/s (2534kB/s)(6256KiB/2528msec) clat (usec): min=61, max=17040, avg=2930.95, stdev=2609.18 lat (usec): min=62, max=17041, avg=2931.72, stdev=2609.21 clat percentiles (usec): | 1.00th=[ 63], 5.00th=[ 72], 10.00th=[ 81], 20.00th=[ 98], | 30.00th=[ 277], 40.00th=[ 2442], 50.00th=[ 2737], 60.00th=[ 3195], | 70.00th=[ 3949], 80.00th=[ 4817], 90.00th=[ 5866], 95.00th=[ 6718], | 99.00th=[12387], 99.50th=[13960], 99.90th=[17171], 99.95th=[17171], | 99.99th=[17171] bw ( KiB/s): min= 1472, max= 3136, per=26.28%, avg=2489.60, stdev=667.18, samples=5 iops : min= 92, max= 196, avg=155.60, stdev=41.70, samples=5 write: IOPS=149, BW=2386KiB/s (2443kB/s)(6032KiB/2528msec) clat (usec): min=658, max=30799, avg=3384.31, stdev=3446.77 lat (usec): min=659, max=30800, avg=3385.70, stdev=3446.93 clat percentiles (usec): | 1.00th=[ 676], 5.00th=[ 758], 10.00th=[ 824], 20.00th=[ 930], | 30.00th=[ 1057], 40.00th=[ 1287], 50.00th=[ 1860], 60.00th=[ 3195], | 70.00th=[ 4228], 80.00th=[ 5538], 90.00th=[ 7242], 95.00th=[10028], | 99.00th=[16712], 99.50th=[20317], 99.90th=[30802], 99.95th=[30802], | 99.99th=[30802] bw ( KiB/s): min= 1792, max= 2720, per=23.63%, avg=2355.20, stdev=348.05, samples=5 iops : min= 112, max= 170, avg=147.20, stdev=21.75, samples=5 lat (usec) : 100=10.81%, 250=4.43%, 500=0.26%, 750=2.21%, 1000=10.81% lat (msec) : 2=13.54%, 4=27.34%, 10=27.34%, 20=2.99%, 50=0.26% cpu : usr=0.08%, sys=27.74%, ctx=1708, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9171: Tue Apr 8 02:01:27 2025 read: IOPS=156, BW=2504KiB/s (2564kB/s)(6032KiB/2409msec) clat (usec): min=60, max=20284, avg=2918.77, stdev=2576.17 lat (usec): min=60, max=20285, avg=2919.34, stdev=2576.17 clat percentiles (usec): | 1.00th=[ 64], 5.00th=[ 73], 10.00th=[ 80], 20.00th=[ 106], | 30.00th=[ 515], 40.00th=[ 2442], 50.00th=[ 2802], 60.00th=[ 3261], | 70.00th=[ 4015], 80.00th=[ 4817], 90.00th=[ 5997], 95.00th=[ 6915], | 99.00th=[10421], 99.50th=[15139], 99.90th=[20317], 99.95th=[20317], | 99.99th=[20317] bw ( KiB/s): min= 1504, max= 3040, per=25.84%, avg=2448.00, stdev=659.70, samples=4 iops : min= 94, max= 190, avg=153.00, stdev=41.23, samples=4 write: IOPS=162, BW=2597KiB/s (2659kB/s)(6256KiB/2409msec) clat (usec): min=640, max=33713, avg=3182.94, stdev=3539.25 lat (usec): min=641, max=33715, avg=3184.61, stdev=3541.29 clat percentiles (usec): | 1.00th=[ 676], 5.00th=[ 766], 10.00th=[ 840], 20.00th=[ 955], | 30.00th=[ 1057], 40.00th=[ 1287], 50.00th=[ 1729], 60.00th=[ 2900], | 70.00th=[ 3589], 80.00th=[ 4817], 90.00th=[ 7046], 95.00th=[ 8225], | 99.00th=[21627], 99.50th=[23987], 99.90th=[33817], 99.95th=[33817], | 99.99th=[33817] bw ( KiB/s): min= 1760, max= 3008, per=26.00%, avg=2592.00, stdev=570.64, samples=4 iops : min= 110, max= 188, avg=162.00, stdev=35.67, samples=4 lat (usec) : 100=9.37%, 250=4.43%, 500=0.91%, 750=1.69%, 1000=11.72% lat (msec) : 2=13.93%, 4=29.95%, 10=26.04%, 20=1.17%, 50=0.78% cpu : usr=0.29%, sys=28.45%, ctx=1673, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=9475KiB/s (9702kB/s), 2350KiB/s-2504KiB/s (2407kB/s-2564kB/s), io=23.4MiB (24.5MB), run=2409-2528msec WRITE: bw=9968KiB/s (10.2MB/s), 2386KiB/s-2678KiB/s (2443kB/s-2742kB/s), io=24.6MiB (25.8MB), run=2409-2528msec rand-rw: (groupid=0, jobs=1): err= 0: pid=9164: Tue Apr 8 02:01:32 2025 read: IOPS=48, BW=780KiB/s (799kB/s)(5920KiB/7590msec) clat (usec): min=1831, max=27697, avg=3359.55, stdev=2770.82 lat (usec): min=1831, max=27697, avg=3360.41, stdev=2770.77 clat percentiles (usec): | 1.00th=[ 1844], 5.00th=[ 1958], 10.00th=[ 2008], 20.00th=[ 2089], | 30.00th=[ 2180], 40.00th=[ 2278], 50.00th=[ 2376], 60.00th=[ 2540], | 70.00th=[ 2835], 80.00th=[ 3359], 90.00th=[ 5866], 95.00th=[ 9634], | 99.00th=[14615], 99.50th=[16909], 99.90th=[27657], 99.95th=[27657], | 99.99th=[27657] bw ( KiB/s): min= 447, max= 1312, per=25.31%, avg=774.33, stdev=254.43, samples=15 iops : min= 27, max= 82, avg=48.33, stdev=15.99, samples=15 write: IOPS=52, BW=839KiB/s (859kB/s)(6368KiB/7590msec) clat (usec): min=8693, max=40786, avg=15806.79, stdev=5442.83 lat (usec): min=8695, max=40787, avg=15808.07, stdev=5442.81 clat percentiles (usec): | 1.00th=[ 9503], 5.00th=[10159], 10.00th=[10552], 20.00th=[11469], | 30.00th=[12256], 40.00th=[13435], 50.00th=[14484], 60.00th=[16057], | 70.00th=[17171], 80.00th=[18744], 90.00th=[22676], 95.00th=[26346], | 99.00th=[36963], 99.50th=[38011], 99.90th=[40633], 99.95th=[40633], | 99.99th=[40633] bw ( KiB/s): min= 510, max= 1056, per=25.97%, avg=836.13, stdev=224.16, samples=15 iops : min= 31, max= 66, avg=52.20, stdev=14.10, samples=15 lat (msec) : 2=4.30%, 4=36.07%, 10=7.55%, 20=43.75%, 50=8.33% cpu : usr=0.12%, sys=7.26%, ctx=902, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9165: Tue Apr 8 02:01:32 2025 read: IOPS=45, BW=734KiB/s (751kB/s)(5744KiB/7828msec) clat (usec): min=1843, max=27766, avg=3516.90, stdev=3077.22 lat (usec): min=1843, max=27766, avg=3517.60, stdev=3077.22 clat percentiles (usec): | 1.00th=[ 1876], 5.00th=[ 1975], 10.00th=[ 2040], 20.00th=[ 2114], | 30.00th=[ 2212], 40.00th=[ 2278], 50.00th=[ 2409], 60.00th=[ 2573], | 70.00th=[ 2835], 80.00th=[ 3589], 90.00th=[ 6587], 95.00th=[10421], | 99.00th=[15270], 99.50th=[25035], 99.90th=[27657], 99.95th=[27657], | 99.99th=[27657] bw ( KiB/s): min= 319, max= 1248, per=24.05%, avg=735.80, stdev=243.47, samples=15 iops : min= 19, max= 78, avg=45.87, stdev=15.31, samples=15 write: IOPS=52, BW=836KiB/s (856kB/s)(6544KiB/7828msec) clat (usec): min=8710, max=41027, avg=16013.33, stdev=4984.53 lat (usec): min=8711, max=41029, avg=16014.67, stdev=4984.56 clat percentiles (usec): | 1.00th=[ 9634], 5.00th=[10552], 10.00th=[10945], 20.00th=[11731], | 30.00th=[12780], 40.00th=[13829], 50.00th=[15008], 60.00th=[16581], | 70.00th=[17433], 80.00th=[18744], 90.00th=[22414], 95.00th=[26084], | 99.00th=[32375], 99.50th=[35390], 99.90th=[41157], 99.95th=[41157], | 99.99th=[41157] bw ( KiB/s): min= 512, max= 1056, per=25.71%, avg=827.47, stdev=206.38, samples=15 iops : min= 32, max= 66, avg=51.60, stdev=12.95, samples=15 lat (msec) : 2=3.26%, 4=34.90%, 10=7.16%, 20=46.74%, 50=7.94% cpu : usr=0.13%, sys=6.80%, ctx=901, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9166: Tue Apr 8 02:01:32 2025 read: IOPS=51, BW=823KiB/s (843kB/s)(6256KiB/7597msec) clat (usec): min=1714, max=27371, avg=3328.45, stdev=3239.10 lat (usec): min=1714, max=27372, avg=3329.25, stdev=3239.04 clat percentiles (usec): | 1.00th=[ 1860], 5.00th=[ 1958], 10.00th=[ 2024], 20.00th=[ 2114], | 30.00th=[ 2180], 40.00th=[ 2278], 50.00th=[ 2343], 60.00th=[ 2442], | 70.00th=[ 2606], 80.00th=[ 2868], 90.00th=[ 5342], 95.00th=[ 9765], | 99.00th=[21890], 99.50th=[24773], 99.90th=[27395], 99.95th=[27395], | 99.99th=[27395] bw ( KiB/s): min= 352, max= 1536, per=27.19%, avg=831.80, stdev=367.51, samples=15 iops : min= 22, max= 96, avg=51.87, stdev=23.04, samples=15 write: IOPS=49, BW=794KiB/s (813kB/s)(6032KiB/7597msec) clat (usec): min=9319, max=54468, avg=16574.19, stdev=5833.27 lat (usec): min=9320, max=54469, avg=16575.42, stdev=5833.38 clat percentiles (usec): | 1.00th=[ 9896], 5.00th=[10552], 10.00th=[10945], 20.00th=[11863], | 30.00th=[13173], 40.00th=[14353], 50.00th=[15664], 60.00th=[16712], | 70.00th=[17695], 80.00th=[19006], 90.00th=[23987], 95.00th=[27657], | 99.00th=[43779], 99.50th=[44827], 99.90th=[54264], 99.95th=[54264], | 99.99th=[54264] bw ( KiB/s): min= 480, max= 1024, per=24.51%, avg=789.07, stdev=200.18, samples=15 iops : min= 30, max= 64, avg=49.20, stdev=12.56, samples=15 lat (msec) : 2=4.04%, 4=40.10%, 10=5.21%, 20=41.80%, 50=8.72% lat (msec) : 100=0.13% cpu : usr=0.07%, sys=7.04%, ctx=909, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9168: Tue Apr 8 02:01:32 2025 read: IOPS=49, BW=790KiB/s (809kB/s)(6032KiB/7635msec) clat (usec): min=1761, max=27887, avg=3436.90, stdev=3109.38 lat (usec): min=1761, max=27888, avg=3437.65, stdev=3109.36 clat percentiles (usec): | 1.00th=[ 1844], 5.00th=[ 1958], 10.00th=[ 2040], 20.00th=[ 2114], | 30.00th=[ 2212], 40.00th=[ 2278], 50.00th=[ 2376], 60.00th=[ 2507], | 70.00th=[ 2704], 80.00th=[ 3163], 90.00th=[ 6390], 95.00th=[ 9634], | 99.00th=[16581], 99.50th=[21890], 99.90th=[27919], 99.95th=[27919], | 99.99th=[27919] bw ( KiB/s): min= 447, max= 1152, per=25.59%, avg=782.87, stdev=254.34, samples=15 iops : min= 27, max= 72, avg=48.87, stdev=15.99, samples=15 write: IOPS=51, BW=819KiB/s (839kB/s)(6256KiB/7635msec) clat (usec): min=8311, max=39355, avg=16111.89, stdev=4873.04 lat (usec): min=8313, max=39356, avg=16113.20, stdev=4873.10 clat percentiles (usec): | 1.00th=[ 9634], 5.00th=[10290], 10.00th=[10683], 20.00th=[11600], | 30.00th=[12911], 40.00th=[14353], 50.00th=[15795], 60.00th=[16712], | 70.00th=[17957], 80.00th=[19006], 90.00th=[22152], 95.00th=[24249], | 99.00th=[32375], 99.50th=[35914], 99.90th=[39584], 99.95th=[39584], | 99.99th=[39584] bw ( KiB/s): min= 510, max= 1024, per=25.31%, avg=814.80, stdev=189.11, samples=15 iops : min= 31, max= 64, avg=50.87, stdev=11.92, samples=15 lat (msec) : 2=3.78%, 4=37.50%, 10=6.64%, 20=43.49%, 50=8.59% cpu : usr=0.12%, sys=7.18%, ctx=894, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=3060KiB/s (3133kB/s), 734KiB/s-823KiB/s (751kB/s-843kB/s), io=23.4MiB (24.5MB), run=7590-7828msec WRITE: bw=3219KiB/s (3296kB/s), 794KiB/s-839KiB/s (813kB/s-859kB/s), io=24.6MiB (25.8MB), run=7590-7828msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=9187: Tue Apr 8 02:01:35 2025 read: IOPS=7, BW=8109KiB/s (8303kB/s)(7168KiB/884msec) clat (usec): min=405, max=36278, avg=20899.17, stdev=14359.25 lat (usec): min=406, max=36279, avg=20900.08, stdev=14359.39 clat percentiles (usec): | 1.00th=[ 408], 5.00th=[ 408], 10.00th=[ 408], 20.00th=[ 586], | 30.00th=[25822], 40.00th=[25822], 50.00th=[26084], 60.00th=[28181], | 70.00th=[28181], 80.00th=[28967], 90.00th=[36439], 95.00th=[36439], | 99.00th=[36439], 99.50th=[36439], 99.90th=[36439], 99.95th=[36439], | 99.99th=[36439] bw ( KiB/s): min= 6144, max= 6144, per=37.68%, avg=6144.00, stdev= 0.00, samples=1 iops : min= 6, max= 6, avg= 6.00, stdev= 0.00, samples=1 write: IOPS=5, BW=5792KiB/s (5931kB/s)(5120KiB/884msec) clat (msec): min=22, max=615, avg=146.33, stdev=262.06 lat (msec): min=22, max=615, avg=146.38, stdev=262.05 clat percentiles (msec): | 1.00th=[ 23], 5.00th=[ 23], 10.00th=[ 23], 20.00th=[ 23], | 30.00th=[ 31], 40.00th=[ 31], 50.00th=[ 32], 60.00th=[ 32], | 70.00th=[ 33], 80.00th=[ 33], 90.00th=[ 617], 95.00th=[ 617], | 99.00th=[ 617], 99.50th=[ 617], 99.90th=[ 617], 99.95th=[ 617], | 99.99th=[ 617] lat (usec) : 500=8.33%, 750=8.33% lat (msec) : 50=75.00%, 750=8.33% cpu : usr=0.00%, sys=20.95%, ctx=972, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9188: Tue Apr 8 02:01:35 2025 read: IOPS=1, BW=1684KiB/s (1725kB/s)(2048KiB/1216msec) clat (msec): min=35, max=619, avg=327.71, stdev=413.19 lat (msec): min=35, max=619, avg=327.71, stdev=413.19 clat percentiles (msec): | 1.00th=[ 36], 5.00th=[ 36], 10.00th=[ 36], 20.00th=[ 36], | 30.00th=[ 36], 40.00th=[ 36], 50.00th=[ 36], 60.00th=[ 617], | 70.00th=[ 617], 80.00th=[ 617], 90.00th=[ 617], 95.00th=[ 617], | 99.00th=[ 617], 99.50th=[ 617], 99.90th=[ 617], 99.95th=[ 617], | 99.99th=[ 617] bw ( KiB/s): min= 4079, max= 4079, per=25.02%, avg=4079.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 write: IOPS=8, BW=8421KiB/s (8623kB/s)(10.0MiB/1216msec) clat (msec): min=25, max=113, avg=53.61, stdev=26.69 lat (msec): min=25, max=113, avg=53.69, stdev=26.70 clat percentiles (msec): | 1.00th=[ 26], 5.00th=[ 26], 10.00th=[ 26], 20.00th=[ 28], | 30.00th=[ 36], 40.00th=[ 37], 50.00th=[ 46], 60.00th=[ 53], | 70.00th=[ 69], 80.00th=[ 69], 90.00th=[ 77], 95.00th=[ 113], | 99.00th=[ 113], 99.50th=[ 113], 99.90th=[ 113], 99.95th=[ 113], | 99.99th=[ 113] bw ( KiB/s): min=10199, max=10199, per=44.68%, avg=10199.00, stdev= 0.00, samples=1 iops : min= 9, max= 9, avg= 9.00, stdev= 0.00, samples=1 lat (msec) : 50=50.00%, 100=33.33%, 250=8.33%, 750=8.33% cpu : usr=0.00%, sys=25.02%, ctx=489, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9189: Tue Apr 8 02:01:35 2025 read: IOPS=5, BW=5560KiB/s (5694kB/s)(6144KiB/1105msec) clat (usec): min=363, max=35061, avg=20269.33, stdev=15741.90 lat (usec): min=363, max=35062, avg=20270.04, stdev=15741.95 clat percentiles (usec): | 1.00th=[ 363], 5.00th=[ 363], 10.00th=[ 363], 20.00th=[ 465], | 30.00th=[ 465], 40.00th=[24773], 50.00th=[24773], 60.00th=[29230], | 70.00th=[31589], 80.00th=[31589], 90.00th=[34866], 95.00th=[34866], | 99.00th=[34866], 99.50th=[34866], 99.90th=[34866], 99.95th=[34866], | 99.99th=[34866] bw ( KiB/s): min= 8175, max= 8175, per=50.14%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 7, max= 7, avg= 7.00, stdev= 0.00, samples=1 write: IOPS=5, BW=5560KiB/s (5694kB/s)(6144KiB/1105msec) clat (msec): min=24, max=641, avg=153.82, stdev=243.90 lat (msec): min=24, max=642, avg=153.87, stdev=243.91 clat percentiles (msec): | 1.00th=[ 25], 5.00th=[ 25], 10.00th=[ 25], 20.00th=[ 29], | 30.00th=[ 29], 40.00th=[ 37], 50.00th=[ 37], 60.00th=[ 40], | 70.00th=[ 153], 80.00th=[ 153], 90.00th=[ 642], 95.00th=[ 642], | 99.00th=[ 642], 99.50th=[ 642], 99.90th=[ 642], 99.95th=[ 642], | 99.99th=[ 642] bw ( KiB/s): min= 2048, max= 6131, per=17.91%, avg=4089.50, stdev=2887.12, samples=2 iops : min= 2, max= 5, avg= 3.50, stdev= 2.12, samples=2 lat (usec) : 500=16.67% lat (msec) : 50=66.67%, 250=8.33%, 750=8.33% cpu : usr=0.00%, sys=15.22%, ctx=343, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9190: Tue Apr 8 02:01:35 2025 read: IOPS=3, BW=4076KiB/s (4174kB/s)(5120KiB/1256msec) clat (usec): min=342, max=40552, avg=25574.56, stdev=15127.84 lat (usec): min=343, max=40553, avg=25575.53, stdev=15127.74 clat percentiles (usec): | 1.00th=[ 343], 5.00th=[ 343], 10.00th=[ 343], 20.00th=[ 343], | 30.00th=[27132], 40.00th=[27132], 50.00th=[27395], 60.00th=[27395], | 70.00th=[32637], 80.00th=[32637], 90.00th=[40633], 95.00th=[40633], | 99.00th=[40633], 99.50th=[40633], 99.90th=[40633], 99.95th=[40633], | 99.99th=[40633] write: IOPS=5, BW=5707KiB/s (5844kB/s)(7168KiB/1256msec) clat (msec): min=28, max=669, avg=152.46, stdev=231.23 lat (msec): min=28, max=669, avg=152.52, stdev=231.24 clat percentiles (msec): | 1.00th=[ 29], 5.00th=[ 29], 10.00th=[ 29], 20.00th=[ 31], | 30.00th=[ 32], 40.00th=[ 32], 50.00th=[ 71], 60.00th=[ 113], | 70.00th=[ 113], 80.00th=[ 125], 90.00th=[ 667], 95.00th=[ 667], | 99.00th=[ 667], 99.50th=[ 667], 99.90th=[ 667], 99.95th=[ 667], | 99.99th=[ 667] bw ( KiB/s): min= 8175, max= 8175, per=35.81%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 7, max= 7, avg= 7.00, stdev= 0.00, samples=1 lat (usec) : 500=8.33% lat (msec) : 50=58.33%, 100=8.33%, 250=16.67%, 750=8.33% cpu : usr=0.00%, sys=16.41%, ctx=406, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=15.9MiB/s (16.7MB/s), 1684KiB/s-8109KiB/s (1725kB/s-8303kB/s), io=20.0MiB (20.0MB), run=884-1256msec WRITE: bw=22.3MiB/s (23.4MB/s), 5560KiB/s-8421KiB/s (5694kB/s-8623kB/s), io=28.0MiB (29.4MB), run=884-1256msec rand-rw: (groupid=0, jobs=1): err= 0: pid=9183: Tue Apr 8 02:01:35 2025 read: IOPS=6, BW=6406KiB/s (6559kB/s)(7168KiB/1119msec) clat (usec): min=11260, max=50893, avg=25706.12, stdev=12984.22 lat (usec): min=11261, max=50908, avg=25721.56, stdev=12994.76 clat percentiles (usec): | 1.00th=[11207], 5.00th=[11207], 10.00th=[11207], 20.00th=[14877], | 30.00th=[21103], 40.00th=[21103], 50.00th=[25035], 60.00th=[25822], | 70.00th=[25822], 80.00th=[31065], 90.00th=[51119], 95.00th=[51119], | 99.00th=[51119], 99.50th=[51119], 99.90th=[51119], 99.95th=[51119], | 99.99th=[51119] bw ( KiB/s): min= 4096, max= 6144, per=32.13%, avg=5120.00, stdev=1448.15, samples=2 iops : min= 4, max= 6, avg= 5.00, stdev= 1.41, samples=2 write: IOPS=4, BW=4576KiB/s (4685kB/s)(5120KiB/1119msec) clat (msec): min=44, max=663, avg=172.72, stdev=274.12 lat (msec): min=44, max=663, avg=172.77, stdev=274.11 clat percentiles (msec): | 1.00th=[ 45], 5.00th=[ 45], 10.00th=[ 45], 20.00th=[ 45], | 30.00th=[ 48], 40.00th=[ 48], 50.00th=[ 52], 60.00th=[ 52], | 70.00th=[ 57], 80.00th=[ 57], 90.00th=[ 667], 95.00th=[ 667], | 99.00th=[ 667], 99.50th=[ 667], 99.90th=[ 667], 99.95th=[ 667], | 99.99th=[ 667] bw ( KiB/s): min= 8192, max= 8192, per=36.72%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 lat (msec) : 20=16.67%, 50=50.00%, 100=25.00%, 750=8.33% cpu : usr=0.09%, sys=9.39%, ctx=52, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9184: Tue Apr 8 02:01:35 2025 read: IOPS=1, BW=1698KiB/s (1739kB/s)(2048KiB/1206msec) clat (usec): min=14709, max=25272, avg=19990.89, stdev=7468.92 lat (usec): min=14711, max=25273, avg=19992.68, stdev=7468.75 clat percentiles (usec): | 1.00th=[14746], 5.00th=[14746], 10.00th=[14746], 20.00th=[14746], | 30.00th=[14746], 40.00th=[14746], 50.00th=[14746], 60.00th=[25297], | 70.00th=[25297], 80.00th=[25297], 90.00th=[25297], 95.00th=[25297], | 99.00th=[25297], 99.50th=[25297], 99.90th=[25297], 99.95th=[25297], | 99.99th=[25297] bw ( KiB/s): min= 2048, max= 2048, per=12.85%, avg=2048.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 write: IOPS=8, BW=8491KiB/s (8695kB/s)(10.0MiB/1206msec) clat (msec): min=25, max=704, avg=114.91, stdev=208.37 lat (msec): min=25, max=704, avg=114.97, stdev=208.37 clat percentiles (msec): | 1.00th=[ 26], 5.00th=[ 26], 10.00th=[ 26], 20.00th=[ 33], | 30.00th=[ 33], 40.00th=[ 34], 50.00th=[ 43], 60.00th=[ 46], | 70.00th=[ 74], 80.00th=[ 74], 90.00th=[ 97], 95.00th=[ 701], | 99.00th=[ 701], 99.50th=[ 701], 99.90th=[ 701], 99.95th=[ 701], | 99.99th=[ 701] bw ( KiB/s): min= 2048, max= 8192, per=22.95%, avg=5120.00, stdev=4344.46, samples=2 iops : min= 2, max= 8, avg= 5.00, stdev= 4.24, samples=2 lat (msec) : 20=8.33%, 50=58.33%, 100=25.00%, 750=8.33% cpu : usr=0.17%, sys=6.22%, ctx=58, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9185: Tue Apr 8 02:01:35 2025 read: IOPS=4, BW=4781KiB/s (4896kB/s)(6144KiB/1285msec) clat (usec): min=21323, max=88529, avg=47115.45, stdev=25538.58 lat (usec): min=21324, max=88530, avg=47116.16, stdev=25538.53 clat percentiles (usec): | 1.00th=[21365], 5.00th=[21365], 10.00th=[21365], 20.00th=[30016], | 30.00th=[30016], 40.00th=[30278], 50.00th=[30278], 60.00th=[47973], | 70.00th=[64750], 80.00th=[64750], 90.00th=[88605], 95.00th=[88605], | 99.00th=[88605], 99.50th=[88605], 99.90th=[88605], 99.95th=[88605], | 99.99th=[88605] bw ( KiB/s): min= 4096, max= 4096, per=25.70%, avg=4096.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 write: IOPS=4, BW=4781KiB/s (4896kB/s)(6144KiB/1285msec) clat (msec): min=26, max=667, avg=162.92, stdev=249.67 lat (msec): min=27, max=667, avg=162.98, stdev=249.68 clat percentiles (msec): | 1.00th=[ 27], 5.00th=[ 27], 10.00th=[ 27], 20.00th=[ 31], | 30.00th=[ 31], 40.00th=[ 65], 50.00th=[ 65], 60.00th=[ 68], | 70.00th=[ 121], 80.00th=[ 121], 90.00th=[ 667], 95.00th=[ 667], | 99.00th=[ 667], 99.50th=[ 667], 99.90th=[ 667], 99.95th=[ 667], | 99.99th=[ 667] bw ( KiB/s): min= 2048, max= 4096, per=13.77%, avg=3072.00, stdev=1448.15, samples=2 iops : min= 2, max= 4, avg= 3.00, stdev= 1.41, samples=2 lat (msec) : 50=50.00%, 100=33.33%, 250=8.33%, 750=8.33% cpu : usr=0.08%, sys=6.15%, ctx=53, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9186: Tue Apr 8 02:01:35 2025 read: IOPS=4, BW=4109KiB/s (4208kB/s)(5120KiB/1246msec) clat (msec): min=12, max=111, avg=39.63, stdev=41.03 lat (msec): min=12, max=111, avg=39.63, stdev=41.03 clat percentiles (msec): | 1.00th=[ 14], 5.00th=[ 14], 10.00th=[ 14], 20.00th=[ 14], | 30.00th=[ 17], 40.00th=[ 17], 50.00th=[ 28], 60.00th=[ 28], | 70.00th=[ 30], 80.00th=[ 30], 90.00th=[ 112], 95.00th=[ 112], | 99.00th=[ 112], 99.50th=[ 112], 99.90th=[ 112], 99.95th=[ 112], | 99.99th=[ 112] bw ( KiB/s): min= 2048, max= 2048, per=12.85%, avg=2048.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=5, BW=5753KiB/s (5891kB/s)(7168KiB/1246msec) clat (msec): min=35, max=633, avg=138.92, stdev=218.90 lat (msec): min=35, max=633, avg=139.00, stdev=218.90 clat percentiles (msec): | 1.00th=[ 36], 5.00th=[ 36], 10.00th=[ 36], 20.00th=[ 39], | 30.00th=[ 47], 40.00th=[ 47], 50.00th=[ 50], 60.00th=[ 78], | 70.00th=[ 78], 80.00th=[ 92], 90.00th=[ 634], 95.00th=[ 634], | 99.00th=[ 634], 99.50th=[ 634], 99.90th=[ 634], 99.95th=[ 634], | 99.99th=[ 634] bw ( KiB/s): min= 8192, max= 8192, per=36.72%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 lat (msec) : 20=16.67%, 50=50.00%, 100=16.67%, 250=8.33%, 750=8.33% cpu : usr=0.16%, sys=6.83%, ctx=48, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=15.6MiB/s (16.3MB/s), 1698KiB/s-6406KiB/s (1739kB/s-6559kB/s), io=20.0MiB (20.0MB), run=1119-1285msec WRITE: bw=21.8MiB/s (22.8MB/s), 4576KiB/s-8491KiB/s (4685kB/s-8695kB/s), io=28.0MiB (29.4MB), run=1119-1285msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=9198: Tue Apr 8 02:01:36 2025 read: IOPS=8, BW=32.6MiB/s (34.2MB/s)(12.0MiB/368msec) clat (msec): min=110, max=128, avg=120.42, stdev= 9.23 lat (msec): min=110, max=128, avg=120.42, stdev= 9.23 clat percentiles (msec): | 1.00th=[ 111], 5.00th=[ 111], 10.00th=[ 111], 20.00th=[ 111], | 30.00th=[ 111], 40.00th=[ 124], 50.00th=[ 124], 60.00th=[ 124], | 70.00th=[ 129], 80.00th=[ 129], 90.00th=[ 129], 95.00th=[ 129], | 99.00th=[ 129], 99.50th=[ 129], 99.90th=[ 129], 99.95th=[ 129], | 99.99th=[ 129] lat (msec) : 250=100.00% cpu : usr=0.00%, sys=40.60%, ctx=1341, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9200: Tue Apr 8 02:01:36 2025 read: IOPS=1, BW=6585KiB/s (6743kB/s)(4096KiB/622msec) clat (nsec): min=96358k, max=96358k, avg=96358102.00, stdev= 0.00 lat (nsec): min=96359k, max=96359k, avg=96359288.00, stdev= 0.00 clat percentiles (usec): | 1.00th=[95945], 5.00th=[95945], 10.00th=[95945], 20.00th=[95945], | 30.00th=[95945], 40.00th=[95945], 50.00th=[95945], 60.00th=[95945], | 70.00th=[95945], 80.00th=[95945], 90.00th=[95945], 95.00th=[95945], | 99.00th=[95945], 99.50th=[95945], 99.90th=[95945], 99.95th=[95945], | 99.99th=[95945] bw ( KiB/s): min= 8192, max= 8192, per=31.10%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=3, BW=12.9MiB/s (13.5MB/s)(8192KiB/622msec) clat (msec): min=156, max=358, avg=257.77, stdev=142.85 lat (msec): min=157, max=359, avg=258.05, stdev=142.81 clat percentiles (msec): | 1.00th=[ 157], 5.00th=[ 157], 10.00th=[ 157], 20.00th=[ 157], | 30.00th=[ 157], 40.00th=[ 157], 50.00th=[ 157], 60.00th=[ 359], | 70.00th=[ 359], 80.00th=[ 359], 90.00th=[ 359], 95.00th=[ 359], | 99.00th=[ 359], 99.50th=[ 359], 99.90th=[ 359], 99.95th=[ 359], | 99.99th=[ 359] bw ( KiB/s): min= 8192, max= 8192, per=15.90%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 250=33.33%, 500=33.33% cpu : usr=0.00%, sys=25.28%, ctx=347, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9202: Tue Apr 8 02:01:36 2025 write: IOPS=4, BW=18.9MiB/s (19.8MB/s)(12.0MiB/636msec) clat (msec): min=110, max=352, avg=205.52, stdev=128.78 lat (msec): min=111, max=352, avg=205.75, stdev=128.83 clat percentiles (msec): | 1.00th=[ 111], 5.00th=[ 111], 10.00th=[ 111], 20.00th=[ 111], | 30.00th=[ 111], 40.00th=[ 155], 50.00th=[ 155], 60.00th=[ 155], | 70.00th=[ 351], 80.00th=[ 351], 90.00th=[ 351], 95.00th=[ 351], | 99.00th=[ 351], 99.50th=[ 351], 99.90th=[ 351], 99.95th=[ 351], | 99.99th=[ 351] bw ( KiB/s): min=16384, max=16384, per=31.80%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.00%, sys=32.13%, ctx=85, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9204: Tue Apr 8 02:01:36 2025 write: IOPS=4, BW=19.8MiB/s (20.8MB/s)(12.0MiB/606msec) clat (msec): min=176, max=221, avg=192.46, stdev=25.28 lat (msec): min=176, max=221, avg=192.70, stdev=25.33 clat percentiles (msec): | 1.00th=[ 178], 5.00th=[ 178], 10.00th=[ 178], 20.00th=[ 178], | 30.00th=[ 178], 40.00th=[ 180], 50.00th=[ 180], 60.00th=[ 180], | 70.00th=[ 222], 80.00th=[ 222], 90.00th=[ 222], 95.00th=[ 222], | 99.00th=[ 222], 99.50th=[ 222], 99.90th=[ 222], 99.95th=[ 222], | 99.99th=[ 222] bw ( KiB/s): min=16318, max=16318, per=31.67%, avg=16318.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 lat (msec) : 250=100.00% cpu : usr=0.00%, sys=38.02%, ctx=163, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=25.7MiB/s (26.0MB/s), 6585KiB/s-32.6MiB/s (6743kB/s-34.2MB/s), io=16.0MiB (16.8MB), run=368-622msec WRITE: bw=50.3MiB/s (52.8MB/s), 12.9MiB/s-19.8MiB/s (13.5MB/s-20.8MB/s), io=32.0MiB (33.6MB), run=606-636msec rand-rw: (groupid=0, jobs=1): err= 0: pid=9197: Tue Apr 8 02:01:36 2025 read: IOPS=7, BW=31.8MiB/s (33.4MB/s)(12.0MiB/377msec) clat (msec): min=41, max=238, avg=109.00, stdev=111.90 lat (msec): min=41, max=238, avg=109.00, stdev=111.90 clat percentiles (msec): | 1.00th=[ 42], 5.00th=[ 42], 10.00th=[ 42], 20.00th=[ 42], | 30.00th=[ 42], 40.00th=[ 48], 50.00th=[ 48], 60.00th=[ 48], | 70.00th=[ 239], 80.00th=[ 239], 90.00th=[ 239], 95.00th=[ 239], | 99.00th=[ 239], 99.50th=[ 239], 99.90th=[ 239], 99.95th=[ 239], | 99.99th=[ 239] lat (msec) : 50=66.67%, 250=33.33% cpu : usr=0.27%, sys=23.67%, ctx=90, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9199: Tue Apr 8 02:01:36 2025 read: IOPS=1, BW=5013KiB/s (5134kB/s)(4096KiB/817msec) clat (nsec): min=131122k, max=131122k, avg=131122182.00, stdev= 0.00 lat (nsec): min=131123k, max=131123k, avg=131123185.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 132], 5.00th=[ 132], 10.00th=[ 132], 20.00th=[ 132], | 30.00th=[ 132], 40.00th=[ 132], 50.00th=[ 132], 60.00th=[ 132], | 70.00th=[ 132], 80.00th=[ 132], 90.00th=[ 132], 95.00th=[ 132], | 99.00th=[ 132], 99.50th=[ 132], 99.90th=[ 132], 99.95th=[ 132], | 99.99th=[ 132] bw ( KiB/s): min= 8192, max= 8192, per=40.85%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=2, BW=9.79MiB/s (10.3MB/s)(8192KiB/817msec) clat (msec): min=227, max=434, avg=331.20, stdev=146.41 lat (msec): min=227, max=434, avg=331.46, stdev=146.35 clat percentiles (msec): | 1.00th=[ 228], 5.00th=[ 228], 10.00th=[ 228], 20.00th=[ 228], | 30.00th=[ 228], 40.00th=[ 228], 50.00th=[ 228], 60.00th=[ 435], | 70.00th=[ 435], 80.00th=[ 435], 90.00th=[ 435], 95.00th=[ 435], | 99.00th=[ 435], 99.50th=[ 435], 99.90th=[ 435], 99.95th=[ 435], | 99.99th=[ 435] lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.12%, sys=13.48%, ctx=63, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9201: Tue Apr 8 02:01:36 2025 write: IOPS=3, BW=15.1MiB/s (15.8MB/s)(12.0MiB/795msec) clat (msec): min=74, max=436, avg=259.10, stdev=181.34 lat (msec): min=74, max=437, avg=259.55, stdev=181.58 clat percentiles (msec): | 1.00th=[ 74], 5.00th=[ 74], 10.00th=[ 74], 20.00th=[ 74], | 30.00th=[ 74], 40.00th=[ 268], 50.00th=[ 268], 60.00th=[ 268], | 70.00th=[ 439], 80.00th=[ 439], 90.00th=[ 439], 95.00th=[ 439], | 99.00th=[ 439], 99.50th=[ 439], 99.90th=[ 439], 99.95th=[ 439], | 99.99th=[ 439] bw ( KiB/s): min= 8192, max= 8192, per=20.43%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 500=66.67% cpu : usr=0.00%, sys=18.01%, ctx=66, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9203: Tue Apr 8 02:01:36 2025 write: IOPS=5, BW=21.4MiB/s (22.4MB/s)(12.0MiB/562msec) clat (msec): min=59, max=285, avg=173.43, stdev=113.01 lat (msec): min=59, max=285, avg=173.75, stdev=113.00 clat percentiles (msec): | 1.00th=[ 60], 5.00th=[ 60], 10.00th=[ 60], 20.00th=[ 60], | 30.00th=[ 60], 40.00th=[ 176], 50.00th=[ 176], 60.00th=[ 176], | 70.00th=[ 288], 80.00th=[ 288], 90.00th=[ 288], 95.00th=[ 288], | 99.00th=[ 288], 99.50th=[ 288], 99.90th=[ 288], 99.95th=[ 288], | 99.99th=[ 288] bw ( KiB/s): min=16384, max=16384, per=40.85%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 250=33.33%, 500=33.33% cpu : usr=0.00%, sys=15.15%, ctx=100, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=19.6MiB/s (20.5MB/s), 5013KiB/s-31.8MiB/s (5134kB/s-33.4MB/s), io=16.0MiB (16.8MB), run=377-817msec WRITE: bw=39.2MiB/s (41.1MB/s), 9.79MiB/s-21.4MiB/s (10.3MB/s-22.4MB/s), io=32.0MiB (33.6MB), run=562-817msec