== sanity test 398b: DIO and buffer IO race ============== 18:55:56 (1743461756) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.29475 s, 38.9 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=9958: Mon Mar 31 18:56:15 2025 read: IOPS=93, BW=373KiB/s (382kB/s)(5992KiB/16056msec) clat (usec): min=50, max=284817, avg=5002.72, stdev=8219.89 lat (usec): min=50, max=284819, avg=5005.18, stdev=8219.88 clat percentiles (usec): | 1.00th=[ 73], 5.00th=[ 104], 10.00th=[ 130], 20.00th=[ 2704], | 30.00th=[ 3064], 40.00th=[ 3523], 50.00th=[ 3949], 60.00th=[ 4686], | 70.00th=[ 5800], 80.00th=[ 7308], 90.00th=[ 9241], 95.00th=[ 10683], | 99.00th=[ 18220], 99.50th=[ 25297], 99.90th=[ 55837], 99.95th=[283116], | 99.99th=[283116] bw ( KiB/s): min= 96, max= 638, per=24.82%, avg=371.09, stdev=94.16, samples=32 iops : min= 24, max= 159, avg=92.72, stdev=23.46, samples=32 write: IOPS=98, BW=392KiB/s (402kB/s)(6296KiB/16056msec) clat (usec): min=438, max=46839, avg=5356.56, stdev=4912.76 lat (usec): min=439, max=46840, avg=5358.01, stdev=4912.56 clat percentiles (usec): | 1.00th=[ 578], 5.00th=[ 676], 10.00th=[ 766], 20.00th=[ 922], | 30.00th=[ 1287], 40.00th=[ 3032], 50.00th=[ 4490], 60.00th=[ 5538], | 70.00th=[ 6783], 80.00th=[ 8979], 90.00th=[11863], 95.00th=[14222], | 99.00th=[20317], 99.50th=[22938], 99.90th=[42730], 99.95th=[46924], | 99.99th=[46924] bw ( KiB/s): min= 64, max= 662, per=24.93%, avg=390.09, stdev=102.31, samples=32 iops : min= 16, max= 165, avg=97.47, stdev=25.51, samples=32 lat (usec) : 100=1.99%, 250=4.98%, 500=0.23%, 750=4.69%, 1000=7.49% lat (msec) : 2=6.15%, 4=22.72%, 10=40.46%, 20=10.35%, 50=0.88% lat (msec) : 100=0.03%, 500=0.03% cpu : usr=0.21%, sys=23.03%, ctx=6178, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9959: Mon Mar 31 18:56:15 2025 read: IOPS=92, BW=370KiB/s (379kB/s)(5940KiB/16061msec) clat (usec): min=53, max=37350, avg=4672.95, stdev=3781.76 lat (usec): min=54, max=37351, avg=4674.07, stdev=3781.82 clat percentiles (usec): | 1.00th=[ 74], 5.00th=[ 102], 10.00th=[ 126], 20.00th=[ 2606], | 30.00th=[ 2999], 40.00th=[ 3425], 50.00th=[ 3949], 60.00th=[ 4490], | 70.00th=[ 5538], 80.00th=[ 6980], 90.00th=[ 9110], 95.00th=[10814], | 99.00th=[16712], 99.50th=[23987], 99.90th=[36963], 99.95th=[37487], | 99.99th=[37487] bw ( KiB/s): min= 88, max= 542, per=24.74%, avg=369.81, stdev=92.61, samples=32 iops : min= 22, max= 135, avg=92.38, stdev=23.09, samples=32 write: IOPS=98, BW=395KiB/s (405kB/s)(6348KiB/16061msec) clat (usec): min=418, max=202619, avg=5664.65, stdev=7459.88 lat (usec): min=419, max=202620, avg=5665.81, stdev=7459.88 clat percentiles (usec): | 1.00th=[ 553], 5.00th=[ 676], 10.00th=[ 758], 20.00th=[ 955], | 30.00th=[ 1500], 40.00th=[ 3294], 50.00th=[ 4621], 60.00th=[ 5669], | 70.00th=[ 6980], 80.00th=[ 9110], 90.00th=[ 11731], 95.00th=[ 14484], | 99.00th=[ 22676], 99.50th=[ 30540], 99.90th=[ 82314], 99.95th=[202376], | 99.99th=[202376] bw ( KiB/s): min= 152, max= 670, per=25.16%, avg=393.81, stdev=96.80, samples=32 iops : min= 38, max= 167, avg=98.37, stdev=24.16, samples=32 lat (usec) : 100=2.31%, 250=5.01%, 500=0.26%, 750=4.69%, 1000=6.71% lat (msec) : 2=5.76%, 4=23.54%, 10=40.20%, 20=10.38%, 50=1.04% lat (msec) : 100=0.07%, 250=0.03% cpu : usr=0.25%, sys=22.98%, ctx=6093, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9960: Mon Mar 31 18:56:15 2025 read: IOPS=94, BW=376KiB/s (385kB/s)(6028KiB/16026msec) clat (usec): min=46, max=40328, avg=4483.77, stdev=3397.06 lat (usec): min=46, max=40328, avg=4485.56, stdev=3397.28 clat percentiles (usec): | 1.00th=[ 73], 5.00th=[ 98], 10.00th=[ 120], 20.00th=[ 2540], | 30.00th=[ 2999], 40.00th=[ 3425], 50.00th=[ 3851], 60.00th=[ 4490], | 70.00th=[ 5473], 80.00th=[ 6849], 90.00th=[ 8717], 95.00th=[10421], | 99.00th=[14222], 99.50th=[16319], 99.90th=[28705], 99.95th=[40109], | 99.99th=[40109] bw ( KiB/s): min= 112, max= 630, per=25.14%, avg=375.88, stdev=94.97, samples=32 iops : min= 28, max= 157, avg=93.91, stdev=23.68, samples=32 write: IOPS=97, BW=391KiB/s (400kB/s)(6260KiB/16026msec) clat (usec): min=473, max=227639, avg=5836.69, stdev=7880.64 lat (usec): min=474, max=227641, avg=5838.19, stdev=7880.49 clat percentiles (usec): | 1.00th=[ 562], 5.00th=[ 685], 10.00th=[ 775], 20.00th=[ 938], | 30.00th=[ 1418], 40.00th=[ 3130], 50.00th=[ 4621], 60.00th=[ 5866], | 70.00th=[ 7373], 80.00th=[ 9503], 90.00th=[ 12387], 95.00th=[ 15270], | 99.00th=[ 22152], 99.50th=[ 35914], 99.90th=[ 58983], 99.95th=[227541], | 99.99th=[227541] bw ( KiB/s): min= 144, max= 686, per=24.91%, avg=389.84, stdev=91.92, samples=32 iops : min= 36, max= 171, avg=97.41, stdev=22.89, samples=32 lat (usec) : 50=0.03%, 100=2.54%, 250=5.53%, 500=0.29%, 750=4.30% lat (usec) : 1000=7.16% lat (msec) : 2=6.38%, 4=22.92%, 10=38.93%, 20=10.97%, 50=0.85% lat (msec) : 100=0.07%, 250=0.03% cpu : usr=0.17%, sys=22.64%, ctx=5987, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9961: Mon Mar 31 18:56:15 2025 read: IOPS=94, BW=380KiB/s (389kB/s)(6052KiB/15938msec) clat (usec): min=48, max=43543, avg=4788.59, stdev=3853.06 lat (usec): min=48, max=43544, avg=4789.63, stdev=3853.07 clat percentiles (usec): | 1.00th=[ 71], 5.00th=[ 103], 10.00th=[ 128], 20.00th=[ 2638], | 30.00th=[ 3097], 40.00th=[ 3589], 50.00th=[ 4080], 60.00th=[ 4686], | 70.00th=[ 5669], 80.00th=[ 7111], 90.00th=[ 9241], 95.00th=[11076], | 99.00th=[16319], 99.50th=[24249], 99.90th=[40633], 99.95th=[43779], | 99.99th=[43779] bw ( KiB/s): min= 136, max= 526, per=24.90%, avg=372.32, stdev=80.58, samples=31 iops : min= 34, max= 131, avg=93.06, stdev=20.11, samples=31 write: IOPS=97, BW=391KiB/s (401kB/s)(6236KiB/15938msec) clat (usec): min=488, max=205484, avg=5491.89, stdev=7279.79 lat (usec): min=489, max=205486, avg=5493.57, stdev=7280.34 clat percentiles (usec): | 1.00th=[ 570], 5.00th=[ 668], 10.00th=[ 758], 20.00th=[ 898], | 30.00th=[ 1254], 40.00th=[ 3130], 50.00th=[ 4359], 60.00th=[ 5473], | 70.00th=[ 6783], 80.00th=[ 8717], 90.00th=[ 11863], 95.00th=[ 15139], | 99.00th=[ 21365], 99.50th=[ 26084], 99.90th=[ 81265], 99.95th=[204473], | 99.99th=[204473] bw ( KiB/s): min= 160, max= 552, per=24.65%, avg=385.74, stdev=79.08, samples=31 iops : min= 40, max= 138, avg=96.42, stdev=19.74, samples=31 lat (usec) : 50=0.03%, 100=2.21%, 250=5.11%, 500=0.20%, 750=4.95% lat (usec) : 1000=7.78% lat (msec) : 2=5.70%, 4=21.84%, 10=41.08%, 20=10.12%, 50=0.91% lat (msec) : 100=0.03%, 250=0.03% cpu : usr=0.23%, sys=22.91%, ctx=6079, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1495KiB/s (1531kB/s), 370KiB/s-380KiB/s (379kB/s-389kB/s), io=23.4MiB (24.6MB), run=15938-16061msec WRITE: bw=1565KiB/s (1603kB/s), 391KiB/s-395KiB/s (400kB/s-405kB/s), io=24.6MiB (25.7MB), run=15938-16061msec rand-rw: (groupid=0, jobs=1): err= 0: pid=9962: Mon Mar 31 18:56:41 2025 read: IOPS=35, BW=140KiB/s (143kB/s)(5992KiB/42782msec) clat (usec): min=1673, max=289260, avg=4573.47, stdev=8438.49 lat (usec): min=1674, max=289262, avg=4574.88, stdev=8438.47 clat percentiles (usec): | 1.00th=[ 1844], 5.00th=[ 2008], 10.00th=[ 2147], 20.00th=[ 2311], | 30.00th=[ 2474], 40.00th=[ 2638], 50.00th=[ 2802], 60.00th=[ 3064], | 70.00th=[ 3621], 80.00th=[ 5538], 90.00th=[ 8848], 95.00th=[ 12256], | 99.00th=[ 20317], 99.50th=[ 30540], 99.90th=[ 45351], 99.95th=[287310], | 99.99th=[287310] bw ( KiB/s): min= 39, max= 272, per=24.83%, avg=139.32, stdev=61.35, samples=85 iops : min= 9, max= 68, avg=34.72, stdev=15.35, samples=85 write: IOPS=36, BW=147KiB/s (151kB/s)(6296KiB/42782msec) clat (usec): min=9830, max=80382, avg=22749.32, stdev=9128.89 lat (usec): min=9832, max=80383, avg=22750.73, stdev=9128.89 clat percentiles (usec): | 1.00th=[10814], 5.00th=[12518], 10.00th=[13435], 20.00th=[15139], | 30.00th=[16909], 40.00th=[19006], 50.00th=[20841], 60.00th=[22676], | 70.00th=[25035], 80.00th=[28181], 90.00th=[35914], 95.00th=[41681], | 99.00th=[54264], 99.50th=[56886], 99.90th=[62653], 99.95th=[80217], | 99.99th=[80217] bw ( KiB/s): min= 24, max= 216, per=24.93%, avg=146.33, stdev=47.59, samples=85 iops : min= 6, max= 54, avg=36.47, stdev=11.91, samples=85 lat (msec) : 2=2.28%, 4=33.40%, 10=9.21%, 20=26.86%, 50=27.47% lat (msec) : 100=0.75%, 500=0.03% cpu : usr=0.14%, sys=5.52%, ctx=3617, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9963: Mon Mar 31 18:56:41 2025 read: IOPS=34, BW=139KiB/s (142kB/s)(5940KiB/42713msec) clat (usec): min=1619, max=207355, avg=4434.85, stdev=6527.04 lat (usec): min=1619, max=207357, avg=4436.09, stdev=6527.05 clat percentiles (usec): | 1.00th=[ 1860], 5.00th=[ 2040], 10.00th=[ 2147], 20.00th=[ 2343], | 30.00th=[ 2474], 40.00th=[ 2638], 50.00th=[ 2835], 60.00th=[ 3097], | 70.00th=[ 3556], 80.00th=[ 5211], 90.00th=[ 9372], 95.00th=[ 11994], | 99.00th=[ 17957], 99.50th=[ 26870], 99.90th=[ 45351], 99.95th=[206570], | 99.99th=[206570] bw ( KiB/s): min= 8, max= 319, per=24.76%, avg=138.91, stdev=65.39, samples=85 iops : min= 2, max= 79, avg=34.65, stdev=16.37, samples=85 write: IOPS=37, BW=149KiB/s (152kB/s)(6348KiB/42713msec) clat (usec): min=9483, max=95471, avg=22685.32, stdev=9656.03 lat (usec): min=9484, max=95472, avg=22686.78, stdev=9656.00 clat percentiles (usec): | 1.00th=[11076], 5.00th=[12518], 10.00th=[13566], 20.00th=[15139], | 30.00th=[16909], 40.00th=[18482], 50.00th=[20317], 60.00th=[22152], | 70.00th=[24511], 80.00th=[28181], 90.00th=[36439], 95.00th=[43254], | 99.00th=[54264], 99.50th=[58459], 99.90th=[85459], 99.95th=[95945], | 99.99th=[95945] bw ( KiB/s): min= 32, max= 216, per=25.23%, avg=148.13, stdev=48.94, samples=85 iops : min= 8, max= 54, avg=36.95, stdev=12.26, samples=85 lat (msec) : 2=1.82%, 4=33.72%, 10=8.72%, 20=28.87%, 50=25.88% lat (msec) : 100=0.94%, 250=0.03% cpu : usr=0.14%, sys=5.54%, ctx=3593, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9964: Mon Mar 31 18:56:41 2025 read: IOPS=35, BW=141KiB/s (145kB/s)(6028KiB/42645msec) clat (usec): min=1666, max=46845, avg=4465.17, stdev=4148.26 lat (usec): min=1666, max=46846, avg=4466.37, stdev=4148.25 clat percentiles (usec): | 1.00th=[ 1844], 5.00th=[ 2008], 10.00th=[ 2114], 20.00th=[ 2278], | 30.00th=[ 2442], 40.00th=[ 2638], 50.00th=[ 2835], 60.00th=[ 3163], | 70.00th=[ 3851], 80.00th=[ 5800], 90.00th=[ 9372], 95.00th=[12387], | 99.00th=[20317], 99.50th=[29754], 99.90th=[39584], 99.95th=[46924], | 99.99th=[46924] bw ( KiB/s): min= 40, max= 320, per=25.18%, avg=141.28, stdev=59.66, samples=85 iops : min= 10, max= 80, avg=35.29, stdev=14.90, samples=85 write: IOPS=36, BW=147KiB/s (150kB/s)(6260KiB/42645msec) clat (msec): min=9, max=225, avg=22.87, stdev=10.58 lat (msec): min=9, max=225, avg=22.87, stdev=10.58 clat percentiles (msec): | 1.00th=[ 12], 5.00th=[ 13], 10.00th=[ 14], 20.00th=[ 16], | 30.00th=[ 17], 40.00th=[ 19], 50.00th=[ 21], 60.00th=[ 23], | 70.00th=[ 25], 80.00th=[ 29], 90.00th=[ 36], 95.00th=[ 43], | 99.00th=[ 55], 99.50th=[ 58], 99.90th=[ 81], 99.95th=[ 226], | 99.99th=[ 226] bw ( KiB/s): min= 40, max= 224, per=24.95%, avg=146.46, stdev=49.31, samples=85 iops : min= 10, max= 56, avg=36.59, stdev=12.33, samples=85 lat (msec) : 2=2.41%, 4=32.45%, 10=10.25%, 20=27.25%, 50=26.89% lat (msec) : 100=0.72%, 250=0.03% cpu : usr=0.11%, sys=5.53%, ctx=3654, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9965: Mon Mar 31 18:56:41 2025 read: IOPS=35, BW=143KiB/s (146kB/s)(6052KiB/42374msec) clat (usec): min=1643, max=48654, avg=4364.33, stdev=4208.33 lat (usec): min=1644, max=48655, avg=4365.52, stdev=4208.30 clat percentiles (usec): | 1.00th=[ 1860], 5.00th=[ 2040], 10.00th=[ 2147], 20.00th=[ 2311], | 30.00th=[ 2474], 40.00th=[ 2606], 50.00th=[ 2802], 60.00th=[ 3097], | 70.00th=[ 3621], 80.00th=[ 5342], 90.00th=[ 8979], 95.00th=[11863], | 99.00th=[20841], 99.50th=[29492], 99.90th=[46924], 99.95th=[48497], | 99.99th=[48497] bw ( KiB/s): min= 16, max= 280, per=25.32%, avg=142.06, stdev=62.75, samples=84 iops : min= 4, max= 70, avg=35.49, stdev=15.69, samples=84 write: IOPS=36, BW=147KiB/s (151kB/s)(6236KiB/42374msec) clat (msec): min=9, max=232, avg=22.86, stdev=10.82 lat (msec): min=9, max=232, avg=22.86, stdev=10.82 clat percentiles (msec): | 1.00th=[ 11], 5.00th=[ 13], 10.00th=[ 14], 20.00th=[ 16], | 30.00th=[ 17], 40.00th=[ 19], 50.00th=[ 21], 60.00th=[ 23], | 70.00th=[ 25], 80.00th=[ 29], 90.00th=[ 37], 95.00th=[ 43], | 99.00th=[ 53], 99.50th=[ 58], 99.90th=[ 86], 99.95th=[ 232], | 99.99th=[ 232] bw ( KiB/s): min= 32, max= 224, per=25.00%, avg=146.73, stdev=48.48, samples=84 iops : min= 8, max= 56, avg=36.65, stdev=12.12, samples=84 lat (msec) : 2=1.76%, 4=34.60%, 10=9.15%, 20=27.51%, 50=26.27% lat (msec) : 100=0.68%, 250=0.03% cpu : usr=0.14%, sys=5.52%, ctx=3601, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=561KiB/s (575kB/s), 139KiB/s-143KiB/s (142kB/s-146kB/s), io=23.4MiB (24.6MB), run=42374-42782msec WRITE: bw=588KiB/s (602kB/s), 147KiB/s-149KiB/s (150kB/s-152kB/s), io=24.6MiB (25.7MB), run=42374-42782msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=10011: Mon Mar 31 18:56:45 2025 read: IOPS=136, BW=2182KiB/s (2234kB/s)(5920KiB/2713msec) clat (usec): min=50, max=13331, avg=3157.71, stdev=2893.54 lat (usec): min=51, max=13332, avg=3161.93, stdev=2890.64 clat percentiles (usec): | 1.00th=[ 56], 5.00th=[ 67], 10.00th=[ 80], 20.00th=[ 104], | 30.00th=[ 139], 40.00th=[ 2540], 50.00th=[ 3064], 60.00th=[ 3458], | 70.00th=[ 4178], 80.00th=[ 5669], 90.00th=[ 7242], 95.00th=[ 8225], | 99.00th=[12125], 99.50th=[12911], 99.90th=[13304], 99.95th=[13304], | 99.99th=[13304] bw ( KiB/s): min= 1408, max= 2522, per=24.74%, avg=2180.20, stdev=455.69, samples=5 iops : min= 88, max= 157, avg=136.00, stdev=28.38, samples=5 write: IOPS=146, BW=2347KiB/s (2404kB/s)(6368KiB/2713msec) clat (usec): min=605, max=34315, avg=3826.32, stdev=4149.40 lat (usec): min=605, max=34316, avg=3827.40, stdev=4149.46 clat percentiles (usec): | 1.00th=[ 652], 5.00th=[ 816], 10.00th=[ 889], 20.00th=[ 1012], | 30.00th=[ 1123], 40.00th=[ 1319], 50.00th=[ 1680], 60.00th=[ 3064], | 70.00th=[ 4621], 80.00th=[ 6456], 90.00th=[ 9372], 95.00th=[11076], | 99.00th=[18744], 99.50th=[26870], 99.90th=[34341], 99.95th=[34341], | 99.99th=[34341] bw ( KiB/s): min= 1440, max= 3488, per=25.38%, avg=2353.20, stdev=734.92, samples=5 iops : min= 90, max= 218, avg=146.80, stdev=46.00, samples=5 lat (usec) : 100=8.85%, 250=7.42%, 500=0.65%, 750=1.30%, 1000=8.46% lat (msec) : 2=18.36%, 4=21.61%, 10=27.86%, 20=5.08%, 50=0.39% cpu : usr=0.18%, sys=28.02%, ctx=1605, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10012: Mon Mar 31 18:56:45 2025 read: IOPS=132, BW=2113KiB/s (2164kB/s)(5744KiB/2718msec) clat (usec): min=52, max=40394, avg=3109.54, stdev=3618.92 lat (usec): min=52, max=40394, avg=3110.24, stdev=3618.91 clat percentiles (usec): | 1.00th=[ 58], 5.00th=[ 66], 10.00th=[ 80], 20.00th=[ 98], | 30.00th=[ 125], 40.00th=[ 2343], 50.00th=[ 3097], 60.00th=[ 3490], | 70.00th=[ 4146], 80.00th=[ 5342], 90.00th=[ 7046], 95.00th=[ 8029], | 99.00th=[11207], 99.50th=[28967], 99.90th=[40633], 99.95th=[40633], | 99.99th=[40633] bw ( KiB/s): min= 1824, max= 2400, per=24.02%, avg=2116.60, stdev=221.12, samples=5 iops : min= 114, max= 150, avg=132.00, stdev=13.87, samples=5 write: IOPS=150, BW=2408KiB/s (2465kB/s)(6544KiB/2718msec) clat (usec): min=601, max=34297, avg=3885.00, stdev=3911.92 lat (usec): min=602, max=34297, avg=3886.10, stdev=3912.00 clat percentiles (usec): | 1.00th=[ 652], 5.00th=[ 734], 10.00th=[ 816], 20.00th=[ 963], | 30.00th=[ 1106], 40.00th=[ 1401], 50.00th=[ 2147], 60.00th=[ 3720], | 70.00th=[ 5211], 80.00th=[ 6652], 90.00th=[ 8717], 95.00th=[10028], | 99.00th=[16057], 99.50th=[20579], 99.90th=[34341], 99.95th=[34341], | 99.99th=[34341] bw ( KiB/s): min= 2048, max= 2906, per=25.45%, avg=2359.40, stdev=330.07, samples=5 iops : min= 128, max= 181, avg=147.20, stdev=20.44, samples=5 lat (usec) : 100=9.90%, 250=7.29%, 500=0.39%, 750=3.26%, 1000=8.98% lat (msec) : 2=13.93%, 4=21.35%, 10=31.38%, 20=2.86%, 50=0.65% cpu : usr=0.26%, sys=28.23%, ctx=1497, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10013: Mon Mar 31 18:56:45 2025 read: IOPS=147, BW=2355KiB/s (2411kB/s)(6256KiB/2657msec) clat (usec): min=51, max=19310, avg=3122.56, stdev=3097.79 lat (usec): min=51, max=19311, avg=3123.41, stdev=3097.65 clat percentiles (usec): | 1.00th=[ 55], 5.00th=[ 65], 10.00th=[ 74], 20.00th=[ 100], | 30.00th=[ 117], 40.00th=[ 2245], 50.00th=[ 2999], 60.00th=[ 3556], | 70.00th=[ 4359], 80.00th=[ 5800], 90.00th=[ 7308], 95.00th=[ 8455], | 99.00th=[12125], 99.50th=[13435], 99.90th=[19268], 99.95th=[19268], | 99.99th=[19268] bw ( KiB/s): min= 1856, max= 2880, per=27.38%, avg=2412.80, stdev=378.36, samples=5 iops : min= 116, max= 180, avg=150.80, stdev=23.65, samples=5 write: IOPS=141, BW=2270KiB/s (2325kB/s)(6032KiB/2657msec) clat (usec): min=573, max=28276, avg=3734.84, stdev=4015.21 lat (usec): min=573, max=28279, avg=3742.95, stdev=4013.17 clat percentiles (usec): | 1.00th=[ 668], 5.00th=[ 807], 10.00th=[ 881], 20.00th=[ 996], | 30.00th=[ 1139], 40.00th=[ 1287], 50.00th=[ 1729], 60.00th=[ 3326], | 70.00th=[ 4752], 80.00th=[ 6128], 90.00th=[ 8586], 95.00th=[11731], | 99.00th=[20579], 99.50th=[24249], 99.90th=[28181], 99.95th=[28181], | 99.99th=[28181] bw ( KiB/s): min= 1568, max= 2976, per=24.37%, avg=2259.20, stdev=547.56, samples=5 iops : min= 98, max= 186, avg=141.20, stdev=34.22, samples=5 lat (usec) : 100=10.55%, 250=9.11%, 500=0.52%, 750=1.43%, 1000=8.72% lat (msec) : 2=15.76%, 4=19.53%, 10=29.82%, 20=4.04%, 50=0.52% cpu : usr=0.23%, sys=28.24%, ctx=1555, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10014: Mon Mar 31 18:56:45 2025 read: IOPS=139, BW=2230KiB/s (2283kB/s)(6032KiB/2705msec) clat (usec): min=51, max=29035, avg=3166.16, stdev=3510.54 lat (usec): min=51, max=29035, avg=3166.84, stdev=3510.54 clat percentiles (usec): | 1.00th=[ 60], 5.00th=[ 65], 10.00th=[ 78], 20.00th=[ 104], | 30.00th=[ 131], 40.00th=[ 906], 50.00th=[ 3064], 60.00th=[ 3621], | 70.00th=[ 4359], 80.00th=[ 5407], 90.00th=[ 7177], 95.00th=[ 8455], | 99.00th=[16057], 99.50th=[23462], 99.90th=[28967], 99.95th=[28967], | 99.99th=[28967] bw ( KiB/s): min= 1952, max= 2656, per=25.42%, avg=2240.00, stdev=352.00, samples=5 iops : min= 122, max= 166, avg=140.00, stdev=22.00, samples=5 write: IOPS=144, BW=2313KiB/s (2368kB/s)(6256KiB/2705msec) clat (usec): min=595, max=35893, avg=3783.95, stdev=3839.17 lat (usec): min=596, max=35894, avg=3785.33, stdev=3839.81 clat percentiles (usec): | 1.00th=[ 627], 5.00th=[ 766], 10.00th=[ 857], 20.00th=[ 1045], | 30.00th=[ 1221], 40.00th=[ 1450], 50.00th=[ 2024], 60.00th=[ 3490], | 70.00th=[ 4621], 80.00th=[ 6587], 90.00th=[ 8979], 95.00th=[10683], | 99.00th=[16909], 99.50th=[20317], 99.90th=[35914], 99.95th=[35914], | 99.99th=[35914] bw ( KiB/s): min= 1760, max= 2944, per=25.33%, avg=2348.80, stdev=436.77, samples=5 iops : min= 110, max= 184, avg=146.80, stdev=27.30, samples=5 lat (usec) : 100=9.11%, 250=9.37%, 500=1.04%, 750=1.95%, 1000=7.16% lat (msec) : 2=16.54%, 4=19.92%, 10=30.73%, 20=3.52%, 50=0.65% cpu : usr=0.18%, sys=28.03%, ctx=1617, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=8812KiB/s (9024kB/s), 2113KiB/s-2355KiB/s (2164kB/s-2411kB/s), io=23.4MiB (24.5MB), run=2657-2718msec WRITE: bw=9272KiB/s (9494kB/s), 2270KiB/s-2408KiB/s (2325kB/s-2465kB/s), io=24.6MiB (25.8MB), run=2657-2718msec rand-rw: (groupid=0, jobs=1): err= 0: pid=10015: Mon Mar 31 18:56:53 2025 read: IOPS=36, BW=582KiB/s (596kB/s)(5920KiB/10166msec) clat (usec): min=2029, max=24636, avg=4189.70, stdev=3092.96 lat (usec): min=2029, max=24637, avg=4190.81, stdev=3092.88 clat percentiles (usec): | 1.00th=[ 2114], 5.00th=[ 2343], 10.00th=[ 2409], 20.00th=[ 2573], | 30.00th=[ 2737], 40.00th=[ 2900], 50.00th=[ 3064], 60.00th=[ 3326], | 70.00th=[ 3818], 80.00th=[ 4555], 90.00th=[ 7046], 95.00th=[11600], | 99.00th=[17957], 99.50th=[23200], 99.90th=[24511], 99.95th=[24511], | 99.99th=[24511] bw ( KiB/s): min= 351, max= 1024, per=25.27%, avg=574.20, stdev=183.62, samples=20 iops : min= 21, max= 64, avg=35.75, stdev=11.57, samples=20 write: IOPS=39, BW=626KiB/s (641kB/s)(6368KiB/10166msec) clat (usec): min=10581, max=50306, avg=21588.05, stdev=6817.59 lat (usec): min=10582, max=50307, avg=21590.13, stdev=6817.41 clat percentiles (usec): | 1.00th=[11731], 5.00th=[13698], 10.00th=[14353], 20.00th=[15533], | 30.00th=[16909], 40.00th=[19006], 50.00th=[20579], 60.00th=[21890], | 70.00th=[23725], 80.00th=[26346], 90.00th=[30016], 95.00th=[35914], | 99.00th=[43254], 99.50th=[45351], 99.90th=[50070], 99.95th=[50070], | 99.99th=[50070] bw ( KiB/s): min= 352, max= 800, per=26.10%, avg=623.75, stdev=135.64, samples=20 iops : min= 22, max= 50, avg=38.85, stdev= 8.55, samples=20 lat (msec) : 4=34.51%, 10=10.81%, 20=26.82%, 50=27.73%, 100=0.13% cpu : usr=0.09%, sys=6.12%, ctx=930, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10016: Mon Mar 31 18:56:53 2025 read: IOPS=34, BW=545KiB/s (558kB/s)(5744KiB/10540msec) clat (usec): min=1875, max=39225, avg=4261.44, stdev=3770.08 lat (usec): min=1875, max=39226, avg=4262.54, stdev=3770.06 clat percentiles (usec): | 1.00th=[ 2008], 5.00th=[ 2180], 10.00th=[ 2343], 20.00th=[ 2540], | 30.00th=[ 2671], 40.00th=[ 2835], 50.00th=[ 2999], 60.00th=[ 3294], | 70.00th=[ 3752], 80.00th=[ 4817], 90.00th=[ 7308], 95.00th=[10421], | 99.00th=[21365], 99.50th=[22938], 99.90th=[39060], 99.95th=[39060], | 99.99th=[39060] bw ( KiB/s): min= 192, max= 864, per=23.80%, avg=540.81, stdev=194.56, samples=21 iops : min= 12, max= 54, avg=33.71, stdev=12.14, samples=21 write: IOPS=38, BW=621KiB/s (636kB/s)(6544KiB/10540msec) clat (usec): min=11256, max=67451, avg=21859.70, stdev=7554.54 lat (usec): min=11258, max=67454, avg=21861.64, stdev=7554.46 clat percentiles (usec): | 1.00th=[11600], 5.00th=[13304], 10.00th=[14353], 20.00th=[15664], | 30.00th=[17171], 40.00th=[19006], 50.00th=[20841], 60.00th=[22414], | 70.00th=[23725], 80.00th=[25560], 90.00th=[32375], 95.00th=[36963], | 99.00th=[49021], 99.50th=[53740], 99.90th=[67634], 99.95th=[67634], | 99.99th=[67634] bw ( KiB/s): min= 319, max= 800, per=25.88%, avg=618.52, stdev=154.68, samples=21 iops : min= 19, max= 50, avg=38.57, stdev= 9.77, samples=21 lat (msec) : 2=0.39%, 4=34.24%, 10=9.51%, 20=25.00%, 50=30.47% lat (msec) : 100=0.39% cpu : usr=0.13%, sys=5.93%, ctx=929, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10017: Mon Mar 31 18:56:53 2025 read: IOPS=39, BW=629KiB/s (644kB/s)(6256KiB/9941msec) clat (usec): min=1926, max=28542, avg=4240.71, stdev=3500.16 lat (usec): min=1927, max=28542, avg=4244.46, stdev=3499.72 clat percentiles (usec): | 1.00th=[ 2040], 5.00th=[ 2245], 10.00th=[ 2376], 20.00th=[ 2507], | 30.00th=[ 2671], 40.00th=[ 2802], 50.00th=[ 2966], 60.00th=[ 3163], | 70.00th=[ 3556], 80.00th=[ 4817], 90.00th=[ 7439], 95.00th=[11994], | 99.00th=[20841], 99.50th=[25035], 99.90th=[28443], 99.95th=[28443], | 99.99th=[28443] bw ( KiB/s): min= 288, max= 1184, per=28.16%, avg=639.89, stdev=293.48, samples=19 iops : min= 18, max= 74, avg=39.95, stdev=18.34, samples=19 write: IOPS=37, BW=607KiB/s (621kB/s)(6032KiB/9941msec) clat (usec): min=11098, max=64865, avg=21740.76, stdev=7352.28 lat (usec): min=11099, max=64866, avg=21742.64, stdev=7352.20 clat percentiles (usec): | 1.00th=[12125], 5.00th=[13566], 10.00th=[14615], 20.00th=[16057], | 30.00th=[17433], 40.00th=[19006], 50.00th=[20317], 60.00th=[21890], | 70.00th=[23462], 80.00th=[25560], 90.00th=[30540], 95.00th=[36963], | 99.00th=[51119], 99.50th=[55837], 99.90th=[64750], 99.95th=[64750], | 99.99th=[64750] bw ( KiB/s): min= 352, max= 768, per=25.22%, avg=602.84, stdev=138.23, samples=19 iops : min= 22, max= 48, avg=37.63, stdev= 8.58, samples=19 lat (msec) : 2=0.26%, 4=38.02%, 10=9.24%, 20=26.69%, 50=25.26% lat (msec) : 100=0.52% cpu : usr=0.14%, sys=6.44%, ctx=956, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10018: Mon Mar 31 18:56:53 2025 read: IOPS=36, BW=580KiB/s (594kB/s)(6032KiB/10401msec) clat (usec): min=2017, max=36064, avg=4332.71, stdev=3449.45 lat (usec): min=2018, max=36065, avg=4333.69, stdev=3449.43 clat percentiles (usec): | 1.00th=[ 2057], 5.00th=[ 2245], 10.00th=[ 2376], 20.00th=[ 2540], | 30.00th=[ 2671], 40.00th=[ 2835], 50.00th=[ 3032], 60.00th=[ 3261], | 70.00th=[ 3851], 80.00th=[ 5342], 90.00th=[ 7898], 95.00th=[10290], | 99.00th=[19006], 99.50th=[26346], 99.90th=[35914], 99.95th=[35914], | 99.99th=[35914] bw ( KiB/s): min= 224, max= 960, per=25.06%, avg=569.45, stdev=216.02, samples=20 iops : min= 14, max= 60, avg=35.50, stdev=13.57, samples=20 write: IOPS=37, BW=601KiB/s (616kB/s)(6256KiB/10401msec) clat (usec): min=11098, max=64657, avg=22210.24, stdev=7341.45 lat (usec): min=11099, max=64658, avg=22212.17, stdev=7341.44 clat percentiles (usec): | 1.00th=[11600], 5.00th=[13698], 10.00th=[14615], 20.00th=[16057], | 30.00th=[17957], 40.00th=[19268], 50.00th=[21365], 60.00th=[22938], | 70.00th=[24511], 80.00th=[26084], 90.00th=[30278], 95.00th=[36439], | 99.00th=[49021], 99.50th=[55837], 99.90th=[64750], 99.95th=[64750], | 99.99th=[64750] bw ( KiB/s): min= 351, max= 768, per=24.90%, avg=595.05, stdev=139.48, samples=20 iops : min= 21, max= 48, avg=37.10, stdev= 8.78, samples=20 lat (msec) : 4=35.03%, 10=11.07%, 20=24.61%, 50=28.91%, 100=0.39% cpu : usr=0.12%, sys=5.82%, ctx=921, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2272KiB/s (2327kB/s), 545KiB/s-629KiB/s (558kB/s-644kB/s), io=23.4MiB (24.5MB), run=9941-10540msec WRITE: bw=2391KiB/s (2448kB/s), 601KiB/s-626KiB/s (616kB/s-641kB/s), io=24.6MiB (25.8MB), run=9941-10540msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=10032: Mon Mar 31 18:56:55 2025 read: IOPS=7, BW=8018KiB/s (8210kB/s)(7168KiB/894msec) clat (usec): min=462, max=538888, avg=92445.45, stdev=197411.26 lat (usec): min=463, max=538890, avg=92446.95, stdev=197411.35 clat percentiles (usec): | 1.00th=[ 461], 5.00th=[ 461], 10.00th=[ 461], 20.00th=[ 537], | 30.00th=[ 11207], 40.00th=[ 11207], 50.00th=[ 26346], 60.00th=[ 34341], | 70.00th=[ 34341], 80.00th=[ 35390], 90.00th=[541066], 95.00th=[541066], | 99.00th=[541066], 99.50th=[541066], 99.90th=[541066], 99.95th=[541066], | 99.99th=[541066] bw ( KiB/s): min= 4096, max= 4096, per=17.88%, avg=4096.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 write: IOPS=5, BW=5727KiB/s (5865kB/s)(5120KiB/894msec) clat (usec): min=18551, max=56816, avg=34930.68, stdev=16730.35 lat (usec): min=18588, max=56914, avg=35002.08, stdev=16758.03 clat percentiles (usec): | 1.00th=[18482], 5.00th=[18482], 10.00th=[18482], 20.00th=[18482], | 30.00th=[24511], 40.00th=[24511], 50.00th=[26084], 60.00th=[26084], | 70.00th=[48497], 80.00th=[48497], 90.00th=[56886], 95.00th=[56886], | 99.00th=[56886], 99.50th=[56886], 99.90th=[56886], 99.95th=[56886], | 99.99th=[56886] lat (usec) : 500=8.33%, 750=8.33% lat (msec) : 20=16.67%, 50=50.00%, 100=8.33%, 750=8.33% cpu : usr=0.00%, sys=22.73%, ctx=967, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10033: Mon Mar 31 18:56:55 2025 read: IOPS=2, BW=2395KiB/s (2453kB/s)(2048KiB/855msec) clat (usec): min=29300, max=48419, avg=38860.25, stdev=13519.26 lat (usec): min=29301, max=48421, avg=38861.42, stdev=13519.43 clat percentiles (usec): | 1.00th=[29230], 5.00th=[29230], 10.00th=[29230], 20.00th=[29230], | 30.00th=[29230], 40.00th=[29230], 50.00th=[29230], 60.00th=[48497], | 70.00th=[48497], 80.00th=[48497], 90.00th=[48497], 95.00th=[48497], | 99.00th=[48497], 99.50th=[48497], 99.90th=[48497], 99.95th=[48497], | 99.99th=[48497] bw ( KiB/s): min= 2039, max= 2039, per=8.90%, avg=2039.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=11, BW=11.7MiB/s (12.3MB/s)(10.0MiB/855msec) clat (msec): min=28, max=169, avg=71.11, stdev=47.56 lat (msec): min=28, max=169, avg=71.18, stdev=47.56 clat percentiles (msec): | 1.00th=[ 29], 5.00th=[ 29], 10.00th=[ 29], 20.00th=[ 31], | 30.00th=[ 33], 40.00th=[ 39], 50.00th=[ 41], 60.00th=[ 60], | 70.00th=[ 102], 80.00th=[ 102], 90.00th=[ 120], 95.00th=[ 169], | 99.00th=[ 169], 99.50th=[ 169], 99.90th=[ 169], 99.95th=[ 169], | 99.99th=[ 169] bw ( KiB/s): min= 8159, max= 8159, per=25.44%, avg=8159.00, stdev= 0.00, samples=1 iops : min= 7, max= 7, avg= 7.00, stdev= 0.00, samples=1 lat (msec) : 50=58.33%, 100=16.67%, 250=25.00% cpu : usr=0.00%, sys=34.07%, ctx=322, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10034: Mon Mar 31 18:56:55 2025 read: IOPS=8, BW=9170KiB/s (9390kB/s)(6144KiB/670msec) clat (usec): min=628, max=289827, avg=68364.59, stdev=109131.99 lat (usec): min=629, max=289828, avg=68365.70, stdev=109132.10 clat percentiles (usec): | 1.00th=[ 627], 5.00th=[ 627], 10.00th=[ 627], 20.00th=[ 27919], | 30.00th=[ 27919], 40.00th=[ 29754], 50.00th=[ 29754], 60.00th=[ 30802], | 70.00th=[ 31327], 80.00th=[ 31327], 90.00th=[291505], 95.00th=[291505], | 99.00th=[291505], 99.50th=[291505], 99.90th=[291505], 99.95th=[291505], | 99.99th=[291505] bw ( KiB/s): min=10240, max=10240, per=44.70%, avg=10240.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 write: IOPS=8, BW=9170KiB/s (9390kB/s)(6144KiB/670msec) clat (msec): min=16, max=121, avg=40.64, stdev=40.11 lat (msec): min=16, max=122, avg=40.83, stdev=40.44 clat percentiles (msec): | 1.00th=[ 17], 5.00th=[ 17], 10.00th=[ 17], 20.00th=[ 22], | 30.00th=[ 22], 40.00th=[ 25], 50.00th=[ 25], 60.00th=[ 26], | 70.00th=[ 34], 80.00th=[ 34], 90.00th=[ 123], 95.00th=[ 123], | 99.00th=[ 123], 99.50th=[ 123], 99.90th=[ 123], 99.95th=[ 123], | 99.99th=[ 123] bw ( KiB/s): min=12288, max=12288, per=38.31%, avg=12288.00, stdev= 0.00, samples=1 iops : min= 12, max= 12, avg=12.00, stdev= 0.00, samples=1 lat (usec) : 750=8.33% lat (msec) : 20=8.33%, 50=66.67%, 250=8.33%, 500=8.33% cpu : usr=0.00%, sys=29.00%, ctx=953, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10035: Mon Mar 31 18:56:55 2025 read: IOPS=6, BW=6267KiB/s (6417kB/s)(5120KiB/817msec) clat (usec): min=474, max=33322, avg=16666.26, stdev=16305.65 lat (usec): min=475, max=33323, avg=16667.29, stdev=16305.76 clat percentiles (usec): | 1.00th=[ 474], 5.00th=[ 474], 10.00th=[ 474], 20.00th=[ 474], | 30.00th=[ 1004], 40.00th=[ 1004], 50.00th=[15270], 60.00th=[15270], | 70.00th=[33424], 80.00th=[33424], 90.00th=[33424], 95.00th=[33424], | 99.00th=[33424], 99.50th=[33424], 99.90th=[33424], 99.95th=[33424], | 99.99th=[33424] bw ( KiB/s): min= 2043, max= 2043, per=8.92%, avg=2043.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=8, BW=8774KiB/s (8984kB/s)(7168KiB/817msec) clat (msec): min=26, max=184, avg=99.33, stdev=49.64 lat (msec): min=26, max=184, avg=99.39, stdev=49.64 clat percentiles (msec): | 1.00th=[ 27], 5.00th=[ 27], 10.00th=[ 27], 20.00th=[ 64], | 30.00th=[ 92], 40.00th=[ 92], 50.00th=[ 96], 60.00th=[ 109], | 70.00th=[ 109], 80.00th=[ 127], 90.00th=[ 184], 95.00th=[ 184], | 99.00th=[ 184], 99.50th=[ 184], 99.90th=[ 184], 99.95th=[ 184], | 99.99th=[ 184] bw ( KiB/s): min= 8175, max= 8175, per=25.49%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 7, max= 7, avg= 7.00, stdev= 0.00, samples=1 lat (usec) : 500=8.33% lat (msec) : 2=8.33%, 20=8.33%, 50=25.00%, 100=25.00%, 250=25.00% cpu : usr=0.00%, sys=21.20%, ctx=360, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=22.4MiB/s (23.5MB/s), 2395KiB/s-9170KiB/s (2453kB/s-9390kB/s), io=20.0MiB (20.0MB), run=670-894msec WRITE: bw=31.3MiB/s (32.8MB/s), 5727KiB/s-11.7MiB/s (5865kB/s-12.3MB/s), io=28.0MiB (29.4MB), run=670-894msec rand-rw: (groupid=0, jobs=1): err= 0: pid=10036: Mon Mar 31 18:56:55 2025 read: IOPS=7, BW=7233KiB/s (7407kB/s)(7168KiB/991msec) clat (msec): min=13, max=449, avg=101.81, stdev=161.45 lat (msec): min=13, max=449, avg=101.81, stdev=161.45 clat percentiles (msec): | 1.00th=[ 14], 5.00th=[ 14], 10.00th=[ 14], 20.00th=[ 14], | 30.00th=[ 16], 40.00th=[ 16], 50.00th=[ 29], 60.00th=[ 37], | 70.00th=[ 37], 80.00th=[ 157], 90.00th=[ 451], 95.00th=[ 451], | 99.00th=[ 451], 99.50th=[ 451], 99.90th=[ 451], 99.95th=[ 451], | 99.99th=[ 451] bw ( KiB/s): min= 4087, max= 4087, per=19.78%, avg=4087.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 write: IOPS=5, BW=5166KiB/s (5290kB/s)(5120KiB/991msec) clat (usec): min=30936, max=74786, avg=44780.78, stdev=17899.05 lat (usec): min=30994, max=74921, avg=44883.40, stdev=17918.28 clat percentiles (usec): | 1.00th=[31065], 5.00th=[31065], 10.00th=[31065], 20.00th=[31065], | 30.00th=[33817], 40.00th=[33817], 50.00th=[36439], 60.00th=[36439], | 70.00th=[47449], 80.00th=[47449], 90.00th=[74974], 95.00th=[74974], | 99.00th=[74974], 99.50th=[74974], 99.90th=[74974], 99.95th=[74974], | 99.99th=[74974] lat (msec) : 20=25.00%, 50=50.00%, 100=8.33%, 250=8.33%, 500=8.33% cpu : usr=0.00%, sys=8.79%, ctx=49, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10037: Mon Mar 31 18:56:55 2025 read: IOPS=2, BW=2513KiB/s (2573kB/s)(2048KiB/815msec) clat (usec): min=25479, max=54376, avg=39928.31, stdev=20433.04 lat (usec): min=25481, max=54380, avg=39930.59, stdev=20434.73 clat percentiles (usec): | 1.00th=[25560], 5.00th=[25560], 10.00th=[25560], 20.00th=[25560], | 30.00th=[25560], 40.00th=[25560], 50.00th=[25560], 60.00th=[54264], | 70.00th=[54264], 80.00th=[54264], 90.00th=[54264], 95.00th=[54264], | 99.00th=[54264], 99.50th=[54264], 99.90th=[54264], 99.95th=[54264], | 99.99th=[54264] bw ( KiB/s): min= 4071, max= 4071, per=19.70%, avg=4071.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 write: IOPS=12, BW=12.3MiB/s (12.9MB/s)(10.0MiB/815msec) clat (msec): min=25, max=216, avg=70.38, stdev=55.98 lat (msec): min=26, max=216, avg=70.45, stdev=55.97 clat percentiles (msec): | 1.00th=[ 27], 5.00th=[ 27], 10.00th=[ 27], 20.00th=[ 28], | 30.00th=[ 41], 40.00th=[ 45], 50.00th=[ 47], 60.00th=[ 51], | 70.00th=[ 84], 80.00th=[ 84], 90.00th=[ 89], 95.00th=[ 218], | 99.00th=[ 218], 99.50th=[ 218], 99.90th=[ 218], 99.95th=[ 218], | 99.99th=[ 218] bw ( KiB/s): min=14250, max=14250, per=49.25%, avg=14250.00, stdev= 0.00, samples=1 iops : min= 13, max= 13, avg=13.00, stdev= 0.00, samples=1 lat (msec) : 50=50.00%, 100=41.67%, 250=8.33% cpu : usr=0.00%, sys=14.13%, ctx=52, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10038: Mon Mar 31 18:56:55 2025 read: IOPS=6, BW=7111KiB/s (7282kB/s)(6144KiB/864msec) clat (usec): min=18236, max=95406, avg=46314.78, stdev=32432.39 lat (usec): min=18237, max=95407, avg=46316.31, stdev=32432.10 clat percentiles (usec): | 1.00th=[18220], 5.00th=[18220], 10.00th=[18220], 20.00th=[19792], | 30.00th=[19792], 40.00th=[19792], 50.00th=[19792], 60.00th=[54264], | 70.00th=[70779], 80.00th=[70779], 90.00th=[94897], 95.00th=[94897], | 99.00th=[94897], 99.50th=[94897], 99.90th=[94897], 99.95th=[94897], | 99.99th=[94897] bw ( KiB/s): min= 6144, max= 6144, per=29.73%, avg=6144.00, stdev= 0.00, samples=1 iops : min= 6, max= 6, avg= 6.00, stdev= 0.00, samples=1 write: IOPS=6, BW=7111KiB/s (7282kB/s)(6144KiB/864msec) clat (msec): min=26, max=158, avg=88.74, stdev=48.97 lat (msec): min=26, max=158, avg=88.81, stdev=48.98 clat percentiles (msec): | 1.00th=[ 27], 5.00th=[ 27], 10.00th=[ 27], 20.00th=[ 48], | 30.00th=[ 48], 40.00th=[ 72], 50.00th=[ 72], 60.00th=[ 112], | 70.00th=[ 117], 80.00th=[ 117], 90.00th=[ 159], 95.00th=[ 159], | 99.00th=[ 159], 99.50th=[ 159], 99.90th=[ 159], 99.95th=[ 159], | 99.99th=[ 159] bw ( KiB/s): min= 8192, max= 8192, per=28.31%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 lat (msec) : 20=25.00%, 50=16.67%, 100=33.33%, 250=25.00% cpu : usr=0.35%, sys=10.31%, ctx=51, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10039: Mon Mar 31 18:56:55 2025 read: IOPS=5, BW=6038KiB/s (6183kB/s)(5120KiB/848msec) clat (usec): min=14078, max=96977, avg=37121.87, stdev=34058.16 lat (usec): min=14078, max=96978, avg=37122.74, stdev=34058.18 clat percentiles (usec): | 1.00th=[14091], 5.00th=[14091], 10.00th=[14091], 20.00th=[14091], | 30.00th=[18220], 40.00th=[18220], 50.00th=[26346], 60.00th=[26346], | 70.00th=[30016], 80.00th=[30016], 90.00th=[96994], 95.00th=[96994], | 99.00th=[96994], 99.50th=[96994], 99.90th=[96994], 99.95th=[96994], | 99.99th=[96994] bw ( KiB/s): min= 8192, max= 8192, per=39.64%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=8, BW=8453KiB/s (8656kB/s)(7168KiB/848msec) clat (msec): min=53, max=148, avg=92.94, stdev=33.76 lat (msec): min=53, max=148, avg=93.03, stdev=33.76 clat percentiles (msec): | 1.00th=[ 54], 5.00th=[ 54], 10.00th=[ 54], 20.00th=[ 58], | 30.00th=[ 83], 40.00th=[ 83], 50.00th=[ 89], 60.00th=[ 102], | 70.00th=[ 102], 80.00th=[ 120], 90.00th=[ 148], 95.00th=[ 148], | 99.00th=[ 148], 99.50th=[ 148], 99.90th=[ 148], 99.95th=[ 148], | 99.99th=[ 148] bw ( KiB/s): min= 8192, max= 8192, per=28.31%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 lat (msec) : 20=16.67%, 50=16.67%, 100=41.67%, 250=25.00% cpu : usr=0.12%, sys=8.26%, ctx=42, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=20.2MiB/s (21.2MB/s), 2513KiB/s-7233KiB/s (2573kB/s-7407kB/s), io=20.0MiB (20.0MB), run=815-991msec WRITE: bw=28.3MiB/s (29.6MB/s), 5166KiB/s-12.3MiB/s (5290kB/s-12.9MB/s), io=28.0MiB (29.4MB), run=815-991msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=10045: Mon Mar 31 18:56:56 2025 read: IOPS=8, BW=34.6MiB/s (36.3MB/s)(12.0MiB/347msec) clat (msec): min=22, max=168, avg=107.89, stdev=76.16 lat (msec): min=22, max=168, avg=107.89, stdev=76.16 clat percentiles (msec): | 1.00th=[ 23], 5.00th=[ 23], 10.00th=[ 23], 20.00th=[ 23], | 30.00th=[ 23], 40.00th=[ 133], 50.00th=[ 133], 60.00th=[ 133], | 70.00th=[ 169], 80.00th=[ 169], 90.00th=[ 169], 95.00th=[ 169], | 99.00th=[ 169], 99.50th=[ 169], 99.90th=[ 169], 99.95th=[ 169], | 99.99th=[ 169] lat (msec) : 50=33.33%, 250=66.67% cpu : usr=0.00%, sys=28.24%, ctx=1309, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10046: Mon Mar 31 18:56:56 2025 read: IOPS=1, BW=7275KiB/s (7450kB/s)(4096KiB/563msec) clat (nsec): min=220128k, max=220128k, avg=220127568.00, stdev= 0.00 lat (nsec): min=220129k, max=220129k, avg=220129098.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 220], 5.00th=[ 220], 10.00th=[ 220], 20.00th=[ 220], | 30.00th=[ 220], 40.00th=[ 220], 50.00th=[ 220], 60.00th=[ 220], | 70.00th=[ 220], 80.00th=[ 220], 90.00th=[ 220], 95.00th=[ 220], | 99.00th=[ 220], 99.50th=[ 220], 99.90th=[ 220], 99.95th=[ 220], | 99.99th=[ 220] bw ( KiB/s): min= 8192, max= 8192, per=28.15%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=3, BW=14.2MiB/s (14.9MB/s)(8192KiB/563msec) clat (msec): min=133, max=182, avg=157.82, stdev=35.04 lat (msec): min=133, max=182, avg=158.09, stdev=35.00 clat percentiles (msec): | 1.00th=[ 133], 5.00th=[ 133], 10.00th=[ 133], 20.00th=[ 133], | 30.00th=[ 133], 40.00th=[ 133], 50.00th=[ 133], 60.00th=[ 184], | 70.00th=[ 184], 80.00th=[ 184], 90.00th=[ 184], 95.00th=[ 184], | 99.00th=[ 184], 99.50th=[ 184], 99.90th=[ 184], 99.95th=[ 184], | 99.99th=[ 184] bw ( KiB/s): min= 8192, max= 8192, per=18.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=100.00% cpu : usr=0.00%, sys=41.81%, ctx=878, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10048: Mon Mar 31 18:56:56 2025 write: IOPS=4, BW=16.7MiB/s (17.5MB/s)(12.0MiB/720msec) clat (msec): min=127, max=402, avg=234.74, stdev=146.97 lat (msec): min=128, max=402, avg=235.06, stdev=147.02 clat percentiles (msec): | 1.00th=[ 128], 5.00th=[ 128], 10.00th=[ 128], 20.00th=[ 128], | 30.00th=[ 128], 40.00th=[ 174], 50.00th=[ 174], 60.00th=[ 174], | 70.00th=[ 401], 80.00th=[ 401], 90.00th=[ 401], 95.00th=[ 401], | 99.00th=[ 401], 99.50th=[ 401], 99.90th=[ 401], 99.95th=[ 401], | 99.99th=[ 401] bw ( KiB/s): min= 8192, max= 8192, per=18.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.00%, sys=32.27%, ctx=65, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10050: Mon Mar 31 18:56:56 2025 write: IOPS=4, BW=17.9MiB/s (18.8MB/s)(12.0MiB/671msec) clat (msec): min=114, max=401, avg=218.58, stdev=159.12 lat (msec): min=114, max=401, avg=219.18, stdev=158.78 clat percentiles (msec): | 1.00th=[ 114], 5.00th=[ 114], 10.00th=[ 114], 20.00th=[ 114], | 30.00th=[ 114], 40.00th=[ 140], 50.00th=[ 140], 60.00th=[ 140], | 70.00th=[ 401], 80.00th=[ 401], 90.00th=[ 401], 95.00th=[ 401], | 99.00th=[ 401], 99.50th=[ 401], 99.90th=[ 401], 99.95th=[ 401], | 99.99th=[ 401] bw ( KiB/s): min=16384, max=16384, per=36.00%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.00%, sys=35.67%, ctx=91, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=28.4MiB/s (29.8MB/s), 7275KiB/s-34.6MiB/s (7450kB/s-36.3MB/s), io=16.0MiB (16.8MB), run=347-563msec WRITE: bw=44.4MiB/s (46.6MB/s), 14.2MiB/s-17.9MiB/s (14.9MB/s-18.8MB/s), io=32.0MiB (33.6MB), run=563-720msec rand-rw: (groupid=0, jobs=1): err= 0: pid=10047: Mon Mar 31 18:56:56 2025 read: IOPS=5, BW=22.7MiB/s (23.8MB/s)(12.0MiB/528msec) clat (msec): min=63, max=231, avg=173.24, stdev=95.14 lat (msec): min=63, max=231, avg=173.24, stdev=95.14 clat percentiles (msec): | 1.00th=[ 64], 5.00th=[ 64], 10.00th=[ 64], 20.00th=[ 64], | 30.00th=[ 64], 40.00th=[ 224], 50.00th=[ 224], 60.00th=[ 224], | 70.00th=[ 232], 80.00th=[ 232], 90.00th=[ 232], 95.00th=[ 232], | 99.00th=[ 232], 99.50th=[ 232], 99.90th=[ 232], 99.95th=[ 232], | 99.99th=[ 232] bw ( KiB/s): min=16384, max=16384, per=81.80%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 250=66.67% cpu : usr=0.00%, sys=10.82%, ctx=38, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10049: Mon Mar 31 18:56:56 2025 read: IOPS=1, BW=5007KiB/s (5128kB/s)(4096KiB/818msec) clat (nsec): min=233514k, max=233514k, avg=233513764.00, stdev= 0.00 lat (nsec): min=233517k, max=233517k, avg=233516804.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 234], 5.00th=[ 234], 10.00th=[ 234], 20.00th=[ 234], | 30.00th=[ 234], 40.00th=[ 234], 50.00th=[ 234], 60.00th=[ 234], | 70.00th=[ 234], 80.00th=[ 234], 90.00th=[ 234], 95.00th=[ 234], | 99.00th=[ 234], 99.50th=[ 234], 99.90th=[ 234], 99.95th=[ 234], | 99.99th=[ 234] bw ( KiB/s): min= 8126, max= 8126, per=40.57%, avg=8126.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=2, BW=9.78MiB/s (10.3MB/s)(8192KiB/818msec) clat (msec): min=220, max=340, avg=280.37, stdev=85.18 lat (msec): min=220, max=340, avg=280.63, stdev=85.19 clat percentiles (msec): | 1.00th=[ 220], 5.00th=[ 220], 10.00th=[ 220], 20.00th=[ 220], | 30.00th=[ 220], 40.00th=[ 220], 50.00th=[ 220], 60.00th=[ 342], | 70.00th=[ 342], 80.00th=[ 342], 90.00th=[ 342], 95.00th=[ 342], | 99.00th=[ 342], 99.50th=[ 342], 99.90th=[ 342], 99.95th=[ 342], | 99.99th=[ 342] bw ( KiB/s): min= 8126, max= 8126, per=20.29%, avg=8126.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.12%, sys=11.51%, ctx=54, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10051: Mon Mar 31 18:56:56 2025 write: IOPS=3, BW=14.0MiB/s (15.7MB/s)(12.0MiB/802msec) clat (msec): min=46, max=454, avg=259.02, stdev=204.45 lat (msec): min=46, max=455, avg=259.31, stdev=204.56 clat percentiles (msec): | 1.00th=[ 47], 5.00th=[ 47], 10.00th=[ 47], 20.00th=[ 47], | 30.00th=[ 47], 40.00th=[ 275], 50.00th=[ 275], 60.00th=[ 275], | 70.00th=[ 456], 80.00th=[ 456], 90.00th=[ 456], 95.00th=[ 456], | 99.00th=[ 456], 99.50th=[ 456], 99.90th=[ 456], 99.95th=[ 456], | 99.99th=[ 456] bw ( KiB/s): min= 8192, max= 8192, per=20.45%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 50=33.33%, 500=66.67% cpu : usr=0.25%, sys=9.86%, ctx=59, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10052: Mon Mar 31 18:56:56 2025 write: IOPS=4, BW=16.7MiB/s (17.5MB/s)(12.0MiB/719msec) clat (msec): min=120, max=403, avg=235.94, stdev=148.10 lat (msec): min=121, max=403, avg=236.21, stdev=148.11 clat percentiles (msec): | 1.00th=[ 122], 5.00th=[ 122], 10.00th=[ 122], 20.00th=[ 122], | 30.00th=[ 122], 40.00th=[ 184], 50.00th=[ 184], 60.00th=[ 184], | 70.00th=[ 405], 80.00th=[ 405], 90.00th=[ 405], 95.00th=[ 405], | 99.00th=[ 405], 99.50th=[ 405], 99.90th=[ 405], 99.95th=[ 405], | 99.99th=[ 405] bw ( KiB/s): min=16384, max=16384, per=40.90%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.14%, sys=12.67%, ctx=45, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=19.6MiB/s (20.5MB/s), 5007KiB/s-22.7MiB/s (5128kB/s-23.8MB/s), io=16.0MiB (16.8MB), run=528-818msec WRITE: bw=39.1MiB/s (41.0MB/s), 9.78MiB/s-16.7MiB/s (10.3MB/s-17.5MB/s), io=32.0MiB (33.6MB), run=719-818msec