== sanity test 398b: DIO and buffer IO race ============== 14:21:39 (1744136499) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.27845 s, 39.4 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=11644: Tue Apr 8 14:21:59 2025 read: IOPS=88, BW=355KiB/s (364kB/s)(5992KiB/16877msec) clat (usec): min=49, max=200891, avg=4856.37, stdev=6637.42 lat (usec): min=49, max=200892, avg=4857.25, stdev=6637.40 clat percentiles (usec): | 1.00th=[ 62], 5.00th=[ 87], 10.00th=[ 99], 20.00th=[ 129], | 30.00th=[ 2704], 40.00th=[ 3228], 50.00th=[ 3752], 60.00th=[ 4555], | 70.00th=[ 5866], 80.00th=[ 7570], 90.00th=[ 10552], 95.00th=[ 13042], | 99.00th=[ 19006], 99.50th=[ 22414], 99.90th=[ 32900], 99.95th=[200279], | 99.99th=[200279] bw ( KiB/s): min= 192, max= 536, per=24.54%, avg=348.97, stdev=82.03, samples=33 iops : min= 48, max= 134, avg=87.18, stdev=20.57, samples=33 write: IOPS=93, BW=373KiB/s (382kB/s)(6296KiB/16877msec) clat (usec): min=428, max=71534, avg=6029.73, stdev=6079.73 lat (usec): min=429, max=71535, avg=6031.73, stdev=6079.72 clat percentiles (usec): | 1.00th=[ 515], 5.00th=[ 627], 10.00th=[ 693], 20.00th=[ 807], | 30.00th=[ 1237], 40.00th=[ 3228], 50.00th=[ 4817], 60.00th=[ 6063], | 70.00th=[ 7701], 80.00th=[ 9765], 90.00th=[13698], 95.00th=[17171], | 99.00th=[26608], 99.50th=[28443], 99.90th=[47973], 99.95th=[71828], | 99.99th=[71828] bw ( KiB/s): min= 223, max= 576, per=24.59%, avg=366.21, stdev=86.83, samples=33 iops : min= 55, max= 144, avg=91.48, stdev=21.78, samples=33 lat (usec) : 50=0.07%, 100=5.27%, 250=5.70%, 500=0.59%, 750=7.68% lat (usec) : 1000=5.79% lat (msec) : 2=4.13%, 4=19.86%, 10=35.61%, 20=13.18%, 50=2.05% lat (msec) : 100=0.03%, 250=0.03% cpu : usr=0.14%, sys=21.62%, ctx=5687, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=11645: Tue Apr 8 14:21:59 2025 read: IOPS=88, BW=355KiB/s (363kB/s)(5940KiB/16736msec) clat (usec): min=50, max=41084, avg=4627.60, stdev=4195.45 lat (usec): min=51, max=41086, avg=4628.60, stdev=4195.62 clat percentiles (usec): | 1.00th=[ 69], 5.00th=[ 90], 10.00th=[ 100], 20.00th=[ 135], | 30.00th=[ 2671], 40.00th=[ 3163], 50.00th=[ 3752], 60.00th=[ 4490], | 70.00th=[ 5735], 80.00th=[ 7504], 90.00th=[10028], 95.00th=[12256], | 99.00th=[18220], 99.50th=[22414], 99.90th=[31851], 99.95th=[41157], | 99.99th=[41157] bw ( KiB/s): min= 184, max= 616, per=24.81%, avg=352.85, stdev=87.99, samples=33 iops : min= 46, max= 154, avg=88.15, stdev=22.02, samples=33 write: IOPS=94, BW=379KiB/s (388kB/s)(6348KiB/16736msec) clat (usec): min=422, max=198320, avg=6142.54, stdev=7575.05 lat (usec): min=423, max=198321, avg=6144.75, stdev=7575.17 clat percentiles (usec): | 1.00th=[ 537], 5.00th=[ 635], 10.00th=[ 701], 20.00th=[ 840], | 30.00th=[ 1221], 40.00th=[ 2999], 50.00th=[ 4817], 60.00th=[ 6259], | 70.00th=[ 7701], 80.00th=[ 10028], 90.00th=[ 13829], 95.00th=[ 17433], | 99.00th=[ 25560], 99.50th=[ 28181], 99.90th=[ 53740], 99.95th=[198181], | 99.99th=[198181] bw ( KiB/s): min= 239, max= 664, per=25.06%, avg=373.21, stdev=91.76, samples=33 iops : min= 59, max= 166, avg=93.24, stdev=22.99, samples=33 lat (usec) : 100=4.88%, 250=5.86%, 500=0.55%, 750=7.26%, 1000=6.05% lat (msec) : 2=5.14%, 4=19.27%, 10=35.87%, 20=13.05%, 50=1.99% lat (msec) : 100=0.03%, 250=0.03% cpu : usr=0.24%, sys=22.20%, ctx=5660, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=11646: Tue Apr 8 14:21:59 2025 read: IOPS=90, BW=363KiB/s (371kB/s)(6028KiB/16617msec) clat (usec): min=44, max=45416, avg=4665.57, stdev=4387.65 lat (usec): min=44, max=45417, avg=4666.78, stdev=4387.83 clat percentiles (usec): | 1.00th=[ 67], 5.00th=[ 88], 10.00th=[ 99], 20.00th=[ 128], | 30.00th=[ 2638], 40.00th=[ 3228], 50.00th=[ 3818], 60.00th=[ 4555], | 70.00th=[ 5604], 80.00th=[ 7504], 90.00th=[10159], 95.00th=[12387], | 99.00th=[17433], 99.50th=[22676], 99.90th=[43779], 99.95th=[45351], | 99.99th=[45351] bw ( KiB/s): min= 176, max= 600, per=25.39%, avg=361.03, stdev=86.88, samples=33 iops : min= 44, max= 150, avg=90.15, stdev=21.73, samples=33 write: IOPS=94, BW=377KiB/s (386kB/s)(6260KiB/16617msec) clat (usec): min=443, max=199468, avg=6055.21, stdev=7980.84 lat (usec): min=444, max=199468, avg=6056.41, stdev=7980.79 clat percentiles (usec): | 1.00th=[ 529], 5.00th=[ 644], 10.00th=[ 709], 20.00th=[ 832], | 30.00th=[ 1237], 40.00th=[ 2835], 50.00th=[ 4752], 60.00th=[ 6128], | 70.00th=[ 7504], 80.00th=[ 9503], 90.00th=[ 13173], 95.00th=[ 16909], | 99.00th=[ 25822], 99.50th=[ 34341], 99.90th=[ 74974], 99.95th=[200279], | 99.99th=[200279] bw ( KiB/s): min= 224, max= 624, per=25.13%, avg=374.12, stdev=88.24, samples=33 iops : min= 56, max= 156, avg=93.42, stdev=22.11, samples=33 lat (usec) : 50=0.07%, 100=5.40%, 250=5.83%, 500=0.55%, 750=6.97% lat (usec) : 1000=6.12% lat (msec) : 2=5.37%, 4=18.85%, 10=36.23%, 20=12.76%, 50=1.69% lat (msec) : 100=0.13%, 250=0.03% cpu : usr=0.17%, sys=21.71%, ctx=5532, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=11647: Tue Apr 8 14:21:59 2025 read: IOPS=91, BW=366KiB/s (375kB/s)(6052KiB/16524msec) clat (usec): min=46, max=216881, avg=4856.04, stdev=7096.73 lat (usec): min=46, max=216881, avg=4857.44, stdev=7096.62 clat percentiles (usec): | 1.00th=[ 67], 5.00th=[ 88], 10.00th=[ 101], 20.00th=[ 133], | 30.00th=[ 2638], 40.00th=[ 3228], 50.00th=[ 3818], 60.00th=[ 4555], | 70.00th=[ 5735], 80.00th=[ 7635], 90.00th=[ 10290], 95.00th=[ 12387], | 99.00th=[ 19006], 99.50th=[ 22152], 99.90th=[ 60556], 99.95th=[217056], | 99.99th=[217056] bw ( KiB/s): min= 200, max= 552, per=25.75%, avg=366.18, stdev=79.87, samples=33 iops : min= 50, max= 138, avg=91.48, stdev=20.03, samples=33 write: IOPS=94, BW=377KiB/s (386kB/s)(6236KiB/16524msec) clat (usec): min=443, max=44378, avg=5814.75, stdev=5687.11 lat (usec): min=444, max=44379, avg=5815.70, stdev=5687.13 clat percentiles (usec): | 1.00th=[ 510], 5.00th=[ 619], 10.00th=[ 701], 20.00th=[ 807], | 30.00th=[ 1172], 40.00th=[ 2769], 50.00th=[ 4555], 60.00th=[ 5997], | 70.00th=[ 7373], 80.00th=[ 9634], 90.00th=[13829], 95.00th=[16712], | 99.00th=[23462], 99.50th=[28705], 99.90th=[38536], 99.95th=[44303], | 99.99th=[44303] bw ( KiB/s): min= 200, max= 592, per=25.31%, avg=376.85, stdev=81.56, samples=33 iops : min= 50, max= 148, avg=94.15, stdev=20.42, samples=33 lat (usec) : 50=0.07%, 100=4.72%, 250=6.51%, 500=0.65%, 750=7.45% lat (usec) : 1000=6.45% lat (msec) : 2=4.52%, 4=19.21%, 10=35.68%, 20=13.31%, 50=1.37% lat (msec) : 100=0.03%, 250=0.03% cpu : usr=0.10%, sys=21.64%, ctx=5579, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1423KiB/s (1457kB/s), 355KiB/s-366KiB/s (363kB/s-375kB/s), io=23.4MiB (24.6MB), run=16524-16877msec WRITE: bw=1490KiB/s (1525kB/s), 373KiB/s-379KiB/s (382kB/s-388kB/s), io=24.6MiB (25.7MB), run=16524-16877msec rand-rw: (groupid=0, jobs=1): err= 0: pid=11648: Tue Apr 8 14:22:22 2025 read: IOPS=37, BW=150KiB/s (154kB/s)(5992KiB/39866msec) clat (usec): min=1438, max=203124, avg=4181.50, stdev=6853.00 lat (usec): min=1439, max=203125, avg=4182.43, stdev=6853.05 clat percentiles (usec): | 1.00th=[ 1582], 5.00th=[ 1762], 10.00th=[ 1876], 20.00th=[ 2008], | 30.00th=[ 2147], 40.00th=[ 2278], 50.00th=[ 2409], 60.00th=[ 2606], | 70.00th=[ 2999], 80.00th=[ 4424], 90.00th=[ 9241], 95.00th=[ 13304], | 99.00th=[ 22152], 99.50th=[ 27395], 99.90th=[ 69731], 99.95th=[202376], | 99.99th=[202376] bw ( KiB/s): min= 31, max= 320, per=24.91%, avg=149.70, stdev=78.13, samples=79 iops : min= 7, max= 80, avg=37.29, stdev=19.55, samples=79 write: IOPS=39, BW=158KiB/s (162kB/s)(6296KiB/39866msec) clat (usec): min=8372, max=93357, avg=21286.72, stdev=12264.50 lat (usec): min=8373, max=93358, avg=21287.90, stdev=12264.57 clat percentiles (usec): | 1.00th=[ 9634], 5.00th=[10814], 10.00th=[11469], 20.00th=[12780], | 30.00th=[14222], 40.00th=[15664], 50.00th=[17171], 60.00th=[18744], | 70.00th=[21103], 80.00th=[26870], 90.00th=[39584], 95.00th=[49021], | 99.00th=[66323], 99.50th=[70779], 99.90th=[84411], 99.95th=[93848], | 99.99th=[93848] bw ( KiB/s): min= 48, max= 280, per=24.89%, avg=156.58, stdev=70.21, samples=79 iops : min= 12, max= 70, avg=39.01, stdev=17.56, samples=79 lat (msec) : 2=9.51%, 4=28.78%, 10=7.10%, 20=37.01%, 50=15.36% lat (msec) : 100=2.21%, 250=0.03% cpu : usr=0.13%, sys=5.28%, ctx=3469, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=11649: Tue Apr 8 14:22:22 2025 read: IOPS=37, BW=149KiB/s (152kB/s)(5940KiB/39922msec) clat (usec): min=1426, max=229478, avg=4136.48, stdev=7270.58 lat (usec): min=1426, max=229479, avg=4137.41, stdev=7270.59 clat percentiles (usec): | 1.00th=[ 1614], 5.00th=[ 1745], 10.00th=[ 1844], 20.00th=[ 1991], | 30.00th=[ 2114], 40.00th=[ 2245], 50.00th=[ 2409], 60.00th=[ 2606], | 70.00th=[ 3032], 80.00th=[ 4228], 90.00th=[ 9372], 95.00th=[ 12911], | 99.00th=[ 20055], 99.50th=[ 25822], 99.90th=[ 71828], 99.95th=[229639], | 99.99th=[229639] bw ( KiB/s): min= 16, max= 344, per=24.67%, avg=148.24, stdev=82.10, samples=79 iops : min= 4, max= 86, avg=36.97, stdev=20.56, samples=79 write: IOPS=39, BW=159KiB/s (163kB/s)(6348KiB/39922msec) clat (usec): min=8654, max=96459, avg=21212.84, stdev=12213.55 lat (usec): min=8654, max=96460, avg=21213.98, stdev=12213.64 clat percentiles (usec): | 1.00th=[ 9372], 5.00th=[10683], 10.00th=[11469], 20.00th=[12911], | 30.00th=[14353], 40.00th=[15664], 50.00th=[17171], 60.00th=[19006], | 70.00th=[20841], 80.00th=[26608], 90.00th=[39060], 95.00th=[49021], | 99.00th=[63701], 99.50th=[69731], 99.90th=[90702], 99.95th=[95945], | 99.99th=[95945] bw ( KiB/s): min= 48, max= 256, per=25.05%, avg=157.56, stdev=69.92, samples=79 iops : min= 12, max= 64, avg=39.30, stdev=17.49, samples=79 lat (msec) : 2=9.86%, 4=28.03%, 10=7.36%, 20=36.72%, 50=15.69% lat (msec) : 100=2.31%, 250=0.03% cpu : usr=0.14%, sys=5.24%, ctx=3420, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=11650: Tue Apr 8 14:22:22 2025 read: IOPS=37, BW=151KiB/s (155kB/s)(6028KiB/39904msec) clat (usec): min=1342, max=72424, avg=4252.34, stdev=4802.21 lat (usec): min=1343, max=72425, avg=4253.27, stdev=4802.24 clat percentiles (usec): | 1.00th=[ 1631], 5.00th=[ 1778], 10.00th=[ 1876], 20.00th=[ 2008], | 30.00th=[ 2147], 40.00th=[ 2278], 50.00th=[ 2474], 60.00th=[ 2704], | 70.00th=[ 3228], 80.00th=[ 5080], 90.00th=[ 9896], 95.00th=[13435], | 99.00th=[20317], 99.50th=[27395], 99.90th=[69731], 99.95th=[72877], | 99.99th=[72877] bw ( KiB/s): min= 24, max= 304, per=25.01%, avg=150.30, stdev=72.97, samples=79 iops : min= 6, max= 76, avg=37.44, stdev=18.25, samples=79 write: IOPS=39, BW=157KiB/s (161kB/s)(6260KiB/39904msec) clat (msec): min=8, max=242, avg=21.34, stdev=13.08 lat (msec): min=8, max=242, avg=21.34, stdev=13.08 clat percentiles (msec): | 1.00th=[ 10], 5.00th=[ 11], 10.00th=[ 12], 20.00th=[ 13], | 30.00th=[ 15], 40.00th=[ 16], 50.00th=[ 18], 60.00th=[ 20], | 70.00th=[ 22], 80.00th=[ 27], 90.00th=[ 40], 95.00th=[ 48], | 99.00th=[ 66], 99.50th=[ 70], 99.90th=[ 85], 99.95th=[ 243], | 99.99th=[ 243] bw ( KiB/s): min= 40, max= 256, per=24.72%, avg=155.47, stdev=69.20, samples=79 iops : min= 10, max= 64, avg=38.73, stdev=17.31, samples=79 lat (msec) : 2=9.34%, 4=27.67%, 10=8.43%, 20=36.17%, 50=16.18% lat (msec) : 100=2.18%, 250=0.03% cpu : usr=0.12%, sys=5.30%, ctx=3453, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=11651: Tue Apr 8 14:22:22 2025 read: IOPS=38, BW=153KiB/s (157kB/s)(6052KiB/39523msec) clat (usec): min=1447, max=43595, avg=3980.00, stdev=3907.01 lat (usec): min=1448, max=43597, avg=3980.90, stdev=3907.09 clat percentiles (usec): | 1.00th=[ 1614], 5.00th=[ 1762], 10.00th=[ 1860], 20.00th=[ 1991], | 30.00th=[ 2114], 40.00th=[ 2245], 50.00th=[ 2409], 60.00th=[ 2638], | 70.00th=[ 3032], 80.00th=[ 4621], 90.00th=[ 9634], 95.00th=[12125], | 99.00th=[18220], 99.50th=[21627], 99.90th=[41681], 99.95th=[43779], | 99.99th=[43779] bw ( KiB/s): min= 8, max= 367, per=25.45%, avg=152.94, stdev=81.29, samples=79 iops : min= 2, max= 91, avg=38.10, stdev=20.33, samples=79 write: IOPS=39, BW=158KiB/s (162kB/s)(6236KiB/39523msec) clat (msec): min=8, max=243, avg=21.43, stdev=13.56 lat (msec): min=8, max=243, avg=21.43, stdev=13.56 clat percentiles (msec): | 1.00th=[ 10], 5.00th=[ 11], 10.00th=[ 12], 20.00th=[ 13], | 30.00th=[ 15], 40.00th=[ 16], 50.00th=[ 18], 60.00th=[ 19], | 70.00th=[ 22], 80.00th=[ 27], 90.00th=[ 40], 95.00th=[ 51], | 99.00th=[ 65], 99.50th=[ 72], 99.90th=[ 97], 99.95th=[ 245], | 99.99th=[ 245] bw ( KiB/s): min= 47, max= 264, per=25.04%, avg=157.49, stdev=70.28, samples=79 iops : min= 11, max= 66, avg=39.24, stdev=17.58, samples=79 lat (msec) : 2=9.99%, 4=28.45%, 10=7.52%, 20=36.52%, 50=14.94% lat (msec) : 100=2.54%, 250=0.03% cpu : usr=0.06%, sys=5.48%, ctx=3447, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=601KiB/s (616kB/s), 149KiB/s-153KiB/s (152kB/s-157kB/s), io=23.4MiB (24.6MB), run=39523-39922msec WRITE: bw=630KiB/s (645kB/s), 157KiB/s-159KiB/s (161kB/s-163kB/s), io=24.6MiB (25.7MB), run=39523-39922msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=11697: Tue Apr 8 14:22:26 2025 read: IOPS=127, BW=2036KiB/s (2085kB/s)(5920KiB/2907msec) clat (usec): min=62, max=21811, avg=3709.52, stdev=3222.08 lat (usec): min=63, max=21812, avg=3710.21, stdev=3222.08 clat percentiles (usec): | 1.00th=[ 70], 5.00th=[ 76], 10.00th=[ 91], 20.00th=[ 123], | 30.00th=[ 2147], 40.00th=[ 2999], 50.00th=[ 3490], 60.00th=[ 4015], | 70.00th=[ 4817], 80.00th=[ 6128], 90.00th=[ 7635], 95.00th=[ 9110], | 99.00th=[13042], 99.50th=[19268], 99.90th=[21890], 99.95th=[21890], | 99.99th=[21890] bw ( KiB/s): min= 1888, max= 2240, per=25.82%, avg=2048.00, stdev=148.38, samples=5 iops : min= 118, max= 140, avg=128.00, stdev= 9.27, samples=5 write: IOPS=136, BW=2191KiB/s (2243kB/s)(6368KiB/2907msec) clat (usec): min=674, max=21197, avg=3817.19, stdev=3654.44 lat (usec): min=675, max=21198, avg=3818.23, stdev=3654.46 clat percentiles (usec): | 1.00th=[ 725], 5.00th=[ 807], 10.00th=[ 906], 20.00th=[ 1090], | 30.00th=[ 1270], 40.00th=[ 1483], 50.00th=[ 2024], 60.00th=[ 3163], | 70.00th=[ 4555], 80.00th=[ 6259], 90.00th=[ 9372], 95.00th=[11207], | 99.00th=[16581], 99.50th=[19792], 99.90th=[21103], 99.95th=[21103], | 99.99th=[21103] bw ( KiB/s): min= 1760, max= 2816, per=26.69%, avg=2227.20, stdev=381.73, samples=5 iops : min= 110, max= 176, avg=139.20, stdev=23.86, samples=5 lat (usec) : 100=5.60%, 250=7.81%, 500=0.39%, 750=0.91%, 1000=7.29% lat (msec) : 2=17.97%, 4=22.27%, 10=31.51%, 20=5.99%, 50=0.26% cpu : usr=0.31%, sys=29.56%, ctx=1463, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=11699: Tue Apr 8 14:22:26 2025 read: IOPS=119, BW=1915KiB/s (1961kB/s)(5744KiB/2999msec) clat (usec): min=60, max=24240, avg=3503.49, stdev=3333.68 lat (usec): min=60, max=24241, avg=3504.43, stdev=3333.44 clat percentiles (usec): | 1.00th=[ 62], 5.00th=[ 73], 10.00th=[ 81], 20.00th=[ 102], | 30.00th=[ 141], 40.00th=[ 2737], 50.00th=[ 3326], 60.00th=[ 3884], | 70.00th=[ 4752], 80.00th=[ 6259], 90.00th=[ 7898], 95.00th=[ 9110], | 99.00th=[14353], 99.50th=[15533], 99.90th=[24249], 99.95th=[24249], | 99.99th=[24249] bw ( KiB/s): min= 1536, max= 2560, per=24.13%, avg=1913.60, stdev=413.16, samples=5 iops : min= 96, max= 160, avg=119.60, stdev=25.82, samples=5 write: IOPS=136, BW=2182KiB/s (2234kB/s)(6544KiB/2999msec) clat (usec): min=643, max=29587, avg=4168.86, stdev=4258.01 lat (usec): min=645, max=29588, avg=4170.58, stdev=4257.69 clat percentiles (usec): | 1.00th=[ 734], 5.00th=[ 816], 10.00th=[ 865], 20.00th=[ 1074], | 30.00th=[ 1221], 40.00th=[ 1500], 50.00th=[ 2442], 60.00th=[ 3884], | 70.00th=[ 5276], 80.00th=[ 7046], 90.00th=[ 9765], 95.00th=[11600], | 99.00th=[19792], 99.50th=[23725], 99.90th=[29492], 99.95th=[29492], | 99.99th=[29492] bw ( KiB/s): min= 1600, max= 2368, per=25.85%, avg=2156.80, stdev=317.27, samples=5 iops : min= 100, max= 148, avg=134.80, stdev=19.83, samples=5 lat (usec) : 100=9.11%, 250=6.25%, 500=0.65%, 750=1.17%, 1000=8.33% lat (msec) : 2=15.89%, 4=20.31%, 10=32.03%, 20=5.60%, 50=0.65% cpu : usr=0.17%, sys=28.49%, ctx=1486, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=11701: Tue Apr 8 14:22:26 2025 read: IOPS=132, BW=2115KiB/s (2166kB/s)(6256KiB/2958msec) clat (usec): min=59, max=15199, avg=3496.44, stdev=3268.79 lat (usec): min=59, max=15200, avg=3497.22, stdev=3268.87 clat percentiles (usec): | 1.00th=[ 62], 5.00th=[ 74], 10.00th=[ 88], 20.00th=[ 114], | 30.00th=[ 145], 40.00th=[ 2606], 50.00th=[ 3326], 60.00th=[ 3916], | 70.00th=[ 4948], 80.00th=[ 6259], 90.00th=[ 7701], 95.00th=[ 9634], | 99.00th=[13960], 99.50th=[14615], 99.90th=[15139], 99.95th=[15139], | 99.99th=[15139] bw ( KiB/s): min= 1728, max= 2272, per=25.82%, avg=2048.00, stdev=206.15, samples=5 iops : min= 108, max= 142, avg=128.00, stdev=12.88, samples=5 write: IOPS=127, BW=2039KiB/s (2088kB/s)(6032KiB/2958msec) clat (usec): min=662, max=23787, avg=4095.80, stdev=3819.05 lat (usec): min=663, max=23788, avg=4097.63, stdev=3819.56 clat percentiles (usec): | 1.00th=[ 725], 5.00th=[ 807], 10.00th=[ 873], 20.00th=[ 1090], | 30.00th=[ 1287], 40.00th=[ 1663], 50.00th=[ 2704], 60.00th=[ 4047], | 70.00th=[ 5080], 80.00th=[ 6718], 90.00th=[ 9241], 95.00th=[11863], | 99.00th=[17957], 99.50th=[18482], 99.90th=[23725], 99.95th=[23725], | 99.99th=[23725] bw ( KiB/s): min= 1600, max= 2528, per=24.39%, avg=2035.20, stdev=329.93, samples=5 iops : min= 100, max= 158, avg=127.20, stdev=20.62, samples=5 lat (usec) : 100=7.03%, 250=10.29%, 500=0.91%, 750=1.04%, 1000=7.29% lat (msec) : 2=13.28%, 4=20.96%, 10=33.59%, 20=5.47%, 50=0.13% cpu : usr=0.20%, sys=28.31%, ctx=1416, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=11702: Tue Apr 8 14:22:26 2025 read: IOPS=124, BW=1997KiB/s (2045kB/s)(6032KiB/3020msec) clat (usec): min=57, max=21231, avg=3454.49, stdev=3388.60 lat (usec): min=57, max=21232, avg=3456.43, stdev=3387.51 clat percentiles (usec): | 1.00th=[ 63], 5.00th=[ 74], 10.00th=[ 82], 20.00th=[ 106], | 30.00th=[ 135], 40.00th=[ 2573], 50.00th=[ 3195], 60.00th=[ 3916], | 70.00th=[ 4752], 80.00th=[ 6194], 90.00th=[ 8291], 95.00th=[ 9503], | 99.00th=[12911], 99.50th=[18744], 99.90th=[21103], 99.95th=[21103], | 99.99th=[21103] bw ( KiB/s): min= 1216, max= 2496, per=24.94%, avg=1977.83, stdev=430.32, samples=6 iops : min= 76, max= 156, avg=123.50, stdev=26.88, samples=6 write: IOPS=129, BW=2072KiB/s (2121kB/s)(6256KiB/3020msec) clat (usec): min=672, max=34136, avg=4246.62, stdev=4536.78 lat (usec): min=673, max=34136, avg=4247.74, stdev=4536.86 clat percentiles (usec): | 1.00th=[ 734], 5.00th=[ 791], 10.00th=[ 898], 20.00th=[ 1090], | 30.00th=[ 1270], 40.00th=[ 1614], 50.00th=[ 2040], 60.00th=[ 3884], | 70.00th=[ 5145], 80.00th=[ 6783], 90.00th=[ 9634], 95.00th=[12518], | 99.00th=[21103], 99.50th=[27395], 99.90th=[34341], 99.95th=[34341], | 99.99th=[34341] bw ( KiB/s): min= 1568, max= 2368, per=24.73%, avg=2063.33, stdev=303.79, samples=6 iops : min= 98, max= 148, avg=128.83, stdev=19.06, samples=6 lat (usec) : 100=8.85%, 250=8.46%, 500=0.65%, 750=1.30%, 1000=6.64% lat (msec) : 2=17.84%, 4=17.71%, 10=32.16%, 20=5.60%, 50=0.78% cpu : usr=0.20%, sys=28.06%, ctx=1404, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=7931KiB/s (8121kB/s), 1915KiB/s-2115KiB/s (1961kB/s-2166kB/s), io=23.4MiB (24.5MB), run=2907-3020msec WRITE: bw=8344KiB/s (8545kB/s), 2039KiB/s-2191KiB/s (2088kB/s-2243kB/s), io=24.6MiB (25.8MB), run=2907-3020msec rand-rw: (groupid=0, jobs=1): err= 0: pid=11695: Tue Apr 8 14:22:32 2025 read: IOPS=40, BW=646KiB/s (661kB/s)(5920KiB/9166msec) clat (usec): min=1933, max=25367, avg=4312.24, stdev=3577.84 lat (usec): min=1934, max=25367, avg=4313.14, stdev=3577.81 clat percentiles (usec): | 1.00th=[ 1991], 5.00th=[ 2147], 10.00th=[ 2245], 20.00th=[ 2409], | 30.00th=[ 2573], 40.00th=[ 2704], 50.00th=[ 2835], 60.00th=[ 3195], | 70.00th=[ 3916], 80.00th=[ 5145], 90.00th=[ 8225], 95.00th=[11469], | 99.00th=[21890], 99.50th=[22938], 99.90th=[25297], 99.95th=[25297], | 99.99th=[25297] bw ( KiB/s): min= 288, max= 1120, per=24.98%, avg=638.11, stdev=209.20, samples=18 iops : min= 18, max= 70, avg=39.83, stdev=13.05, samples=18 write: IOPS=43, BW=695KiB/s (711kB/s)(6368KiB/9166msec) clat (usec): min=9937, max=45560, avg=18939.18, stdev=5906.77 lat (usec): min=9939, max=45562, avg=18940.88, stdev=5906.78 clat percentiles (usec): | 1.00th=[10945], 5.00th=[11994], 10.00th=[12780], 20.00th=[13960], | 30.00th=[15401], 40.00th=[16450], 50.00th=[17695], 60.00th=[19006], | 70.00th=[20579], 80.00th=[22414], 90.00th=[26870], 95.00th=[31589], | 99.00th=[39584], 99.50th=[43779], 99.90th=[45351], 99.95th=[45351], | 99.99th=[45351] bw ( KiB/s): min= 416, max= 928, per=25.80%, avg=693.22, stdev=184.29, samples=18 iops : min= 26, max= 58, avg=43.28, stdev=11.51, samples=18 lat (msec) : 2=0.52%, 4=33.59%, 10=11.07%, 20=37.50%, 50=17.32% cpu : usr=0.13%, sys=6.94%, ctx=930, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=11696: Tue Apr 8 14:22:32 2025 read: IOPS=38, BW=612KiB/s (627kB/s)(5744KiB/9378msec) clat (usec): min=1842, max=21118, avg=4296.10, stdev=3399.20 lat (usec): min=1842, max=21118, avg=4297.28, stdev=3399.06 clat percentiles (usec): | 1.00th=[ 2008], 5.00th=[ 2180], 10.00th=[ 2245], 20.00th=[ 2409], | 30.00th=[ 2540], 40.00th=[ 2638], 50.00th=[ 2802], 60.00th=[ 3097], | 70.00th=[ 3982], 80.00th=[ 5014], 90.00th=[ 8848], 95.00th=[11994], | 99.00th=[17957], 99.50th=[20317], 99.90th=[21103], 99.95th=[21103], | 99.99th=[21103] bw ( KiB/s): min= 288, max= 1152, per=24.01%, avg=613.22, stdev=235.20, samples=18 iops : min= 18, max= 72, avg=38.28, stdev=14.70, samples=18 write: IOPS=43, BW=698KiB/s (715kB/s)(6544KiB/9378msec) clat (usec): min=9036, max=51329, avg=19087.19, stdev=6694.44 lat (usec): min=9036, max=51330, avg=19088.83, stdev=6694.37 clat percentiles (usec): | 1.00th=[11076], 5.00th=[11863], 10.00th=[12387], 20.00th=[13698], | 30.00th=[14877], 40.00th=[16319], 50.00th=[17957], 60.00th=[19006], | 70.00th=[20841], 80.00th=[22676], 90.00th=[26608], 95.00th=[33817], | 99.00th=[43779], 99.50th=[44827], 99.90th=[51119], 99.95th=[51119], | 99.99th=[51119] bw ( KiB/s): min= 416, max= 928, per=25.47%, avg=684.33, stdev=186.58, samples=18 iops : min= 26, max= 58, avg=42.72, stdev=11.64, samples=18 lat (msec) : 2=0.39%, 4=32.42%, 10=10.29%, 20=37.76%, 50=19.01% lat (msec) : 100=0.13% cpu : usr=0.14%, sys=6.70%, ctx=913, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=11698: Tue Apr 8 14:22:32 2025 read: IOPS=43, BW=691KiB/s (707kB/s)(6256KiB/9058msec) clat (usec): min=1870, max=26794, avg=4124.27, stdev=3313.92 lat (usec): min=1871, max=26795, avg=4125.16, stdev=3313.92 clat percentiles (usec): | 1.00th=[ 1975], 5.00th=[ 2180], 10.00th=[ 2278], 20.00th=[ 2409], | 30.00th=[ 2507], 40.00th=[ 2638], 50.00th=[ 2835], 60.00th=[ 3097], | 70.00th=[ 3720], 80.00th=[ 5014], 90.00th=[ 7308], 95.00th=[10814], | 99.00th=[19268], 99.50th=[22938], 99.90th=[26870], 99.95th=[26870], | 99.99th=[26870] bw ( KiB/s): min= 288, max= 1216, per=27.22%, avg=695.11, stdev=277.85, samples=18 iops : min= 18, max= 76, avg=43.44, stdev=17.37, samples=18 write: IOPS=41, BW=666KiB/s (682kB/s)(6032KiB/9058msec) clat (usec): min=10006, max=52102, avg=19711.02, stdev=6982.16 lat (usec): min=10008, max=52103, avg=19712.61, stdev=6982.14 clat percentiles (usec): | 1.00th=[10945], 5.00th=[11994], 10.00th=[12780], 20.00th=[13960], | 30.00th=[15270], 40.00th=[16712], 50.00th=[18482], 60.00th=[19792], | 70.00th=[21103], 80.00th=[23725], 90.00th=[28705], 95.00th=[33817], | 99.00th=[44303], 99.50th=[47449], 99.90th=[52167], 99.95th=[52167], | 99.99th=[52167] bw ( KiB/s): min= 384, max= 864, per=24.68%, avg=663.11, stdev=168.55, samples=18 iops : min= 24, max= 54, avg=41.44, stdev=10.53, samples=18 lat (msec) : 2=0.52%, 4=36.59%, 10=10.81%, 20=33.07%, 50=18.88% lat (msec) : 100=0.13% cpu : usr=0.17%, sys=7.01%, ctx=943, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=11700: Tue Apr 8 14:22:32 2025 read: IOPS=40, BW=655KiB/s (671kB/s)(6032KiB/9203msec) clat (usec): min=2003, max=19454, avg=4159.95, stdev=3216.93 lat (usec): min=2004, max=19455, avg=4160.84, stdev=3216.93 clat percentiles (usec): | 1.00th=[ 2057], 5.00th=[ 2180], 10.00th=[ 2278], 20.00th=[ 2409], | 30.00th=[ 2573], 40.00th=[ 2704], 50.00th=[ 2835], 60.00th=[ 3064], | 70.00th=[ 3687], 80.00th=[ 4555], 90.00th=[ 8717], 95.00th=[11469], | 99.00th=[18220], 99.50th=[19268], 99.90th=[19530], 99.95th=[19530], | 99.99th=[19530] bw ( KiB/s): min= 288, max= 1088, per=25.41%, avg=648.89, stdev=214.47, samples=18 iops : min= 18, max= 68, avg=40.56, stdev=13.40, samples=18 write: IOPS=42, BW=680KiB/s (696kB/s)(6256KiB/9203msec) clat (usec): min=9949, max=48108, avg=19411.42, stdev=6353.38 lat (usec): min=9951, max=48110, avg=19413.01, stdev=6353.43 clat percentiles (usec): | 1.00th=[11076], 5.00th=[12125], 10.00th=[12911], 20.00th=[14091], | 30.00th=[15533], 40.00th=[16909], 50.00th=[18220], 60.00th=[19268], | 70.00th=[21103], 80.00th=[22938], 90.00th=[28443], 95.00th=[32113], | 99.00th=[40109], 99.50th=[47973], 99.90th=[47973], 99.95th=[47973], | 99.99th=[47973] bw ( KiB/s): min= 416, max= 960, per=25.21%, avg=677.33, stdev=179.10, samples=18 iops : min= 26, max= 60, avg=42.33, stdev=11.19, samples=18 lat (msec) : 4=36.46%, 10=8.72%, 20=36.46%, 50=18.36% cpu : usr=0.18%, sys=6.79%, ctx=919, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2554KiB/s (2615kB/s), 612KiB/s-691KiB/s (627kB/s-707kB/s), io=23.4MiB (24.5MB), run=9058-9378msec WRITE: bw=2687KiB/s (2752kB/s), 666KiB/s-698KiB/s (682kB/s-715kB/s), io=24.6MiB (25.8MB), run=9058-9378msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=11719: Tue Apr 8 14:22:34 2025 read: IOPS=9, BW=9.83MiB/s (10.3MB/s)(7168KiB/712msec) clat (usec): min=504, max=64520, avg=34213.33, stdev=25239.15 lat (usec): min=505, max=64522, avg=34214.69, stdev=25239.04 clat percentiles (usec): | 1.00th=[ 506], 5.00th=[ 506], 10.00th=[ 506], 20.00th=[ 611], | 30.00th=[35390], 40.00th=[35390], 50.00th=[35914], 60.00th=[45876], | 70.00th=[45876], 80.00th=[56361], 90.00th=[64750], 95.00th=[64750], | 99.00th=[64750], 99.50th=[64750], 99.90th=[64750], 99.95th=[64750], | 99.99th=[64750] bw ( KiB/s): min=10118, max=10118, per=44.46%, avg=10118.00, stdev= 0.00, samples=1 iops : min= 9, max= 9, avg= 9.00, stdev= 0.00, samples=1 write: IOPS=7, BW=7191KiB/s (7364kB/s)(5120KiB/712msec) clat (msec): min=30, max=102, avg=77.43, stdev=30.46 lat (msec): min=30, max=102, avg=77.51, stdev=30.41 clat percentiles (msec): | 1.00th=[ 31], 5.00th=[ 31], 10.00th=[ 31], 20.00th=[ 31], | 30.00th=[ 63], 40.00th=[ 63], 50.00th=[ 95], 60.00th=[ 95], | 70.00th=[ 96], 80.00th=[ 96], 90.00th=[ 103], 95.00th=[ 103], | 99.00th=[ 103], 99.50th=[ 103], 99.90th=[ 103], 99.95th=[ 103], | 99.99th=[ 103] bw ( KiB/s): min= 4047, max= 4047, per=12.70%, avg=4047.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 lat (usec) : 750=16.67% lat (msec) : 50=33.33%, 100=41.67%, 250=8.33% cpu : usr=0.00%, sys=29.82%, ctx=1102, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=11720: Tue Apr 8 14:22:34 2025 read: IOPS=2, BW=2276KiB/s (2330kB/s)(2048KiB/900msec) clat (usec): min=48199, max=53704, avg=50952.09, stdev=3892.81 lat (usec): min=48200, max=53707, avg=50954.20, stdev=3893.96 clat percentiles (usec): | 1.00th=[47973], 5.00th=[47973], 10.00th=[47973], 20.00th=[47973], | 30.00th=[47973], 40.00th=[47973], 50.00th=[47973], 60.00th=[53740], | 70.00th=[53740], 80.00th=[53740], 90.00th=[53740], 95.00th=[53740], | 99.00th=[53740], 99.50th=[53740], 99.90th=[53740], 99.95th=[53740], | 99.99th=[53740] bw ( KiB/s): min= 2031, max= 2031, per=8.93%, avg=2031.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=11, BW=11.1MiB/s (11.7MB/s)(10.0MiB/900msec) clat (msec): min=34, max=147, avg=71.48, stdev=37.44 lat (msec): min=35, max=147, avg=71.54, stdev=37.43 clat percentiles (msec): | 1.00th=[ 35], 5.00th=[ 35], 10.00th=[ 35], 20.00th=[ 37], | 30.00th=[ 43], 40.00th=[ 49], 50.00th=[ 56], 60.00th=[ 57], | 70.00th=[ 99], 80.00th=[ 99], 90.00th=[ 112], 95.00th=[ 148], | 99.00th=[ 148], 99.50th=[ 148], 99.90th=[ 148], 99.95th=[ 148], | 99.99th=[ 148] bw ( KiB/s): min= 6095, max= 6095, per=19.13%, avg=6095.00, stdev= 0.00, samples=1 iops : min= 5, max= 5, avg= 5.00, stdev= 0.00, samples=1 lat (msec) : 50=41.67%, 100=41.67%, 250=16.67% cpu : usr=0.00%, sys=32.48%, ctx=272, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=11721: Tue Apr 8 14:22:34 2025 read: IOPS=9, BW=9481KiB/s (9709kB/s)(6144KiB/648msec) clat (usec): min=490, max=62194, avg=40275.46, stdev=21750.59 lat (usec): min=491, max=62195, avg=40276.54, stdev=21750.52 clat percentiles (usec): | 1.00th=[ 490], 5.00th=[ 490], 10.00th=[ 490], 20.00th=[32113], | 30.00th=[32113], 40.00th=[46924], 50.00th=[46924], 60.00th=[49546], | 70.00th=[50594], 80.00th=[50594], 90.00th=[62129], 95.00th=[62129], | 99.00th=[62129], 99.50th=[62129], 99.90th=[62129], 99.95th=[62129], | 99.99th=[62129] bw ( KiB/s): min= 8078, max= 8078, per=35.50%, avg=8078.00, stdev= 0.00, samples=1 iops : min= 7, max= 7, avg= 7.00, stdev= 0.00, samples=1 write: IOPS=9, BW=9481KiB/s (9709kB/s)(6144KiB/648msec) clat (msec): min=27, max=154, avg=66.27, stdev=47.95 lat (msec): min=27, max=154, avg=66.34, stdev=47.95 clat percentiles (msec): | 1.00th=[ 28], 5.00th=[ 28], 10.00th=[ 28], 20.00th=[ 31], | 30.00th=[ 31], 40.00th=[ 46], 50.00th=[ 46], 60.00th=[ 56], | 70.00th=[ 85], 80.00th=[ 85], 90.00th=[ 155], 95.00th=[ 155], | 99.00th=[ 155], 99.50th=[ 155], 99.90th=[ 155], 99.95th=[ 155], | 99.99th=[ 155] bw ( KiB/s): min=10098, max=10098, per=31.70%, avg=10098.00, stdev= 0.00, samples=1 iops : min= 9, max= 9, avg= 9.00, stdev= 0.00, samples=1 lat (usec) : 500=8.33% lat (msec) : 50=50.00%, 100=33.33%, 250=8.33% cpu : usr=0.15%, sys=37.09%, ctx=614, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=11722: Tue Apr 8 14:22:34 2025 read: IOPS=6, BW=6872KiB/s (7037kB/s)(5120KiB/745msec) clat (usec): min=499, max=126490, avg=35491.17, stdev=55065.83 lat (usec): min=500, max=126490, avg=35492.11, stdev=55065.60 clat percentiles (usec): | 1.00th=[ 498], 5.00th=[ 498], 10.00th=[ 498], 20.00th=[ 498], | 30.00th=[ 562], 40.00th=[ 562], 50.00th=[ 652], 60.00th=[ 652], | 70.00th=[ 49021], 80.00th=[ 49021], 90.00th=[126354], 95.00th=[126354], | 99.00th=[126354], 99.50th=[126354], 99.90th=[126354], 99.95th=[126354], | 99.99th=[126354] write: IOPS=9, BW=9621KiB/s (9852kB/s)(7168KiB/745msec) clat (msec): min=20, max=200, avg=76.22, stdev=71.51 lat (msec): min=20, max=200, avg=76.28, stdev=71.52 clat percentiles (msec): | 1.00th=[ 21], 5.00th=[ 21], 10.00th=[ 21], 20.00th=[ 28], | 30.00th=[ 43], 40.00th=[ 43], 50.00th=[ 43], 60.00th=[ 45], | 70.00th=[ 45], 80.00th=[ 157], 90.00th=[ 201], 95.00th=[ 201], | 99.00th=[ 201], 99.50th=[ 201], 99.90th=[ 201], 99.95th=[ 201], | 99.99th=[ 201] bw ( KiB/s): min= 8143, max= 8143, per=25.56%, avg=8143.00, stdev= 0.00, samples=1 iops : min= 7, max= 7, avg= 7.00, stdev= 0.00, samples=1 lat (usec) : 500=8.33%, 750=16.67% lat (msec) : 50=50.00%, 250=25.00% cpu : usr=0.00%, sys=20.83%, ctx=333, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=22.2MiB/s (23.3MB/s), 2276KiB/s-9.83MiB/s (2330kB/s-10.3MB/s), io=20.0MiB (20.0MB), run=648-900msec WRITE: bw=31.1MiB/s (32.6MB/s), 7191KiB/s-11.1MiB/s (7364kB/s-11.7MB/s), io=28.0MiB (29.4MB), run=648-900msec rand-rw: (groupid=0, jobs=1): err= 0: pid=11715: Tue Apr 8 14:22:34 2025 read: IOPS=9, BW=9.84MiB/s (10.3MB/s)(7168KiB/711msec) clat (usec): min=13950, max=88066, avg=34668.13, stdev=25353.43 lat (usec): min=13952, max=88068, avg=34669.75, stdev=25353.44 clat percentiles (usec): | 1.00th=[13960], 5.00th=[13960], 10.00th=[13960], 20.00th=[15270], | 30.00th=[24773], 40.00th=[24773], 50.00th=[25822], 60.00th=[35390], | 70.00th=[35390], 80.00th=[39060], 90.00th=[87557], 95.00th=[87557], | 99.00th=[87557], 99.50th=[87557], 99.90th=[87557], 99.95th=[87557], | 99.99th=[87557] bw ( KiB/s): min=10240, max=10240, per=46.50%, avg=10240.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 write: IOPS=7, BW=7201KiB/s (7374kB/s)(5120KiB/711msec) clat (msec): min=34, max=133, avg=82.95, stdev=38.60 lat (msec): min=34, max=133, avg=83.01, stdev=38.60 clat percentiles (msec): | 1.00th=[ 35], 5.00th=[ 35], 10.00th=[ 35], 20.00th=[ 35], | 30.00th=[ 60], 40.00th=[ 60], 50.00th=[ 82], 60.00th=[ 82], | 70.00th=[ 107], 80.00th=[ 107], 90.00th=[ 133], 95.00th=[ 133], | 99.00th=[ 133], 99.50th=[ 133], 99.90th=[ 133], 99.95th=[ 133], | 99.99th=[ 133] bw ( KiB/s): min= 6144, max= 6144, per=19.93%, avg=6144.00, stdev= 0.00, samples=1 iops : min= 6, max= 6, avg= 6.00, stdev= 0.00, samples=1 lat (msec) : 20=16.67%, 50=41.67%, 100=25.00%, 250=16.67% cpu : usr=0.28%, sys=13.80%, ctx=36, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=11716: Tue Apr 8 14:22:34 2025 read: IOPS=2, BW=2304KiB/s (2359kB/s)(2048KiB/889msec) clat (usec): min=33087, max=61241, avg=47164.52, stdev=19907.50 lat (usec): min=33088, max=61242, avg=47165.56, stdev=19907.54 clat percentiles (usec): | 1.00th=[33162], 5.00th=[33162], 10.00th=[33162], 20.00th=[33162], | 30.00th=[33162], 40.00th=[33162], 50.00th=[33162], 60.00th=[61080], | 70.00th=[61080], 80.00th=[61080], 90.00th=[61080], 95.00th=[61080], | 99.00th=[61080], 99.50th=[61080], 99.90th=[61080], 99.95th=[61080], | 99.99th=[61080] bw ( KiB/s): min= 2048, max= 2048, per=9.30%, avg=2048.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=11, BW=11.2MiB/s (11.8MB/s)(10.0MiB/889msec) clat (msec): min=29, max=141, avg=73.73, stdev=34.71 lat (msec): min=30, max=141, avg=73.80, stdev=34.70 clat percentiles (msec): | 1.00th=[ 31], 5.00th=[ 31], 10.00th=[ 31], 20.00th=[ 38], | 30.00th=[ 55], 40.00th=[ 56], 50.00th=[ 59], 60.00th=[ 66], | 70.00th=[ 92], 80.00th=[ 92], 90.00th=[ 113], 95.00th=[ 142], | 99.00th=[ 142], 99.50th=[ 142], 99.90th=[ 142], 99.95th=[ 142], | 99.99th=[ 142] bw ( KiB/s): min= 8192, max= 8192, per=26.57%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 lat (msec) : 50=25.00%, 100=58.33%, 250=16.67% cpu : usr=0.11%, sys=17.34%, ctx=106, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=11717: Tue Apr 8 14:22:34 2025 read: IOPS=6, BW=6606KiB/s (6765kB/s)(6144KiB/930msec) clat (msec): min=18, max=183, avg=68.43, stdev=59.21 lat (msec): min=18, max=183, avg=68.43, stdev=59.21 clat percentiles (msec): | 1.00th=[ 19], 5.00th=[ 19], 10.00th=[ 19], 20.00th=[ 41], | 30.00th=[ 41], 40.00th=[ 43], 50.00th=[ 43], 60.00th=[ 50], | 70.00th=[ 78], 80.00th=[ 78], 90.00th=[ 184], 95.00th=[ 184], | 99.00th=[ 184], 99.50th=[ 184], 99.90th=[ 184], 99.95th=[ 184], | 99.99th=[ 184] bw ( KiB/s): min= 2048, max= 2048, per=9.30%, avg=2048.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=6, BW=6606KiB/s (6765kB/s)(6144KiB/930msec) clat (msec): min=34, max=202, avg=81.22, stdev=66.07 lat (msec): min=34, max=202, avg=81.29, stdev=66.08 clat percentiles (msec): | 1.00th=[ 35], 5.00th=[ 35], 10.00th=[ 35], 20.00th=[ 39], | 30.00th=[ 39], 40.00th=[ 48], 50.00th=[ 48], 60.00th=[ 52], | 70.00th=[ 114], 80.00th=[ 114], 90.00th=[ 203], 95.00th=[ 203], | 99.00th=[ 203], 99.50th=[ 203], 99.90th=[ 203], 99.95th=[ 203], | 99.99th=[ 203] bw ( KiB/s): min= 6144, max= 6144, per=19.93%, avg=6144.00, stdev= 0.00, samples=1 iops : min= 6, max= 6, avg= 6.00, stdev= 0.00, samples=1 lat (msec) : 20=8.33%, 50=50.00%, 100=16.67%, 250=25.00% cpu : usr=0.00%, sys=14.21%, ctx=73, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=11718: Tue Apr 8 14:22:34 2025 read: IOPS=8, BW=8285KiB/s (8484kB/s)(5120KiB/618msec) clat (usec): min=25103, max=31000, avg=27751.81, stdev=2569.13 lat (usec): min=25104, max=31001, avg=27753.37, stdev=2569.15 clat percentiles (usec): | 1.00th=[25035], 5.00th=[25035], 10.00th=[25035], 20.00th=[25035], | 30.00th=[25297], 40.00th=[25297], 50.00th=[27919], 60.00th=[27919], | 70.00th=[29492], 80.00th=[29492], 90.00th=[31065], 95.00th=[31065], | 99.00th=[31065], 99.50th=[31065], 99.90th=[31065], 99.95th=[31065], | 99.99th=[31065] bw ( KiB/s): min= 8159, max= 8159, per=37.05%, avg=8159.00, stdev= 0.00, samples=1 iops : min= 7, max= 7, avg= 7.00, stdev= 0.00, samples=1 write: IOPS=11, BW=11.3MiB/s (11.9MB/s)(7168KiB/618msec) clat (msec): min=33, max=108, avg=67.37, stdev=33.02 lat (msec): min=33, max=108, avg=67.44, stdev=33.02 clat percentiles (msec): | 1.00th=[ 34], 5.00th=[ 34], 10.00th=[ 34], 20.00th=[ 43], | 30.00th=[ 44], 40.00th=[ 44], 50.00th=[ 46], 60.00th=[ 99], | 70.00th=[ 99], 80.00th=[ 102], 90.00th=[ 109], 95.00th=[ 109], | 99.00th=[ 109], 99.50th=[ 109], 99.90th=[ 109], 99.95th=[ 109], | 99.99th=[ 109] bw ( KiB/s): min=12239, max=12239, per=39.70%, avg=12239.00, stdev= 0.00, samples=1 iops : min= 11, max= 11, avg=11.00, stdev= 0.00, samples=1 lat (msec) : 50=75.00%, 100=8.33%, 250=16.67% cpu : usr=0.00%, sys=16.37%, ctx=54, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=21.5MiB/s (22.6MB/s), 2304KiB/s-9.84MiB/s (2359kB/s-10.3MB/s), io=20.0MiB (20.0MB), run=618-930msec WRITE: bw=30.1MiB/s (31.6MB/s), 6606KiB/s-11.3MiB/s (6765kB/s-11.9MB/s), io=28.0MiB (29.4MB), run=618-930msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=11729: Tue Apr 8 14:22:36 2025 read: IOPS=3, BW=14.4MiB/s (15.1MB/s)(12.0MiB/833msec) clat (usec): min=1784, max=753200, avg=265839.15, stdev=422558.77 lat (usec): min=1785, max=753203, avg=265840.76, stdev=422559.54 clat percentiles (usec): | 1.00th=[ 1778], 5.00th=[ 1778], 10.00th=[ 1778], 20.00th=[ 1778], | 30.00th=[ 1778], 40.00th=[ 42730], 50.00th=[ 42730], 60.00th=[ 42730], | 70.00th=[750781], 80.00th=[750781], 90.00th=[750781], 95.00th=[750781], | 99.00th=[750781], 99.50th=[750781], 99.90th=[750781], 99.95th=[750781], | 99.99th=[750781] lat (msec) : 2=33.33%, 50=33.33%, 1000=33.33% cpu : usr=0.12%, sys=19.83%, ctx=975, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=11730: Tue Apr 8 14:22:36 2025 read: IOPS=1, BW=5425KiB/s (5555kB/s)(4096KiB/755msec) clat (nsec): min=204586k, max=204586k, avg=204586458.00, stdev= 0.00 lat (nsec): min=204589k, max=204589k, avg=204588720.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 205], 5.00th=[ 205], 10.00th=[ 205], 20.00th=[ 205], | 30.00th=[ 205], 40.00th=[ 205], 50.00th=[ 205], 60.00th=[ 205], | 70.00th=[ 205], 80.00th=[ 205], 90.00th=[ 205], 95.00th=[ 205], | 99.00th=[ 205], 99.50th=[ 205], 99.90th=[ 205], 99.95th=[ 205], | 99.99th=[ 205] bw ( KiB/s): min= 8192, max= 8192, per=41.65%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=2, BW=10.6MiB/s (11.1MB/s)(8192KiB/755msec) clat (msec): min=103, max=416, avg=260.07, stdev=221.17 lat (msec): min=103, max=416, avg=260.27, stdev=221.28 clat percentiles (msec): | 1.00th=[ 104], 5.00th=[ 104], 10.00th=[ 104], 20.00th=[ 104], | 30.00th=[ 104], 40.00th=[ 104], 50.00th=[ 104], 60.00th=[ 418], | 70.00th=[ 418], 80.00th=[ 418], 90.00th=[ 418], 95.00th=[ 418], | 99.00th=[ 418], 99.50th=[ 418], 99.90th=[ 418], 99.95th=[ 418], | 99.99th=[ 418] lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.00%, sys=40.19%, ctx=375, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=11731: Tue Apr 8 14:22:36 2025 write: IOPS=2, BW=9630KiB/s (9861kB/s)(12.0MiB/1276msec) clat (msec): min=127, max=940, avg=407.61, stdev=461.97 lat (msec): min=127, max=941, avg=407.91, stdev=461.95 clat percentiles (msec): | 1.00th=[ 128], 5.00th=[ 128], 10.00th=[ 128], 20.00th=[ 128], | 30.00th=[ 128], 40.00th=[ 155], 50.00th=[ 155], 60.00th=[ 155], | 70.00th=[ 944], 80.00th=[ 944], 90.00th=[ 944], 95.00th=[ 944], | 99.00th=[ 944], 99.50th=[ 944], 99.90th=[ 944], 99.95th=[ 944], | 99.99th=[ 944] bw ( KiB/s): min= 8192, max= 8192, per=31.90%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 1000=33.33% cpu : usr=0.00%, sys=23.76%, ctx=85, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=11733: Tue Apr 8 14:22:36 2025 write: IOPS=2, BW=9934KiB/s (10.2MB/s)(12.0MiB/1237msec) clat (msec): min=134, max=695, avg=393.90, stdev=283.03 lat (msec): min=134, max=696, avg=394.14, stdev=283.06 clat percentiles (msec): | 1.00th=[ 136], 5.00th=[ 136], 10.00th=[ 136], 20.00th=[ 136], | 30.00th=[ 136], 40.00th=[ 351], 50.00th=[ 351], 60.00th=[ 351], | 70.00th=[ 693], 80.00th=[ 693], 90.00th=[ 693], 95.00th=[ 693], | 99.00th=[ 693], 99.50th=[ 693], 99.90th=[ 693], 99.95th=[ 693], | 99.99th=[ 693] bw ( KiB/s): min= 8192, max= 8192, per=31.90%, avg=8192.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 lat (msec) : 250=33.33%, 500=33.33%, 750=33.33% cpu : usr=0.00%, sys=24.11%, ctx=143, majf=0, minf=28 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=19.2MiB/s (20.1MB/s), 5425KiB/s-14.4MiB/s (5555kB/s-15.1MB/s), io=16.0MiB (16.8MB), run=755-833msec WRITE: bw=25.1MiB/s (26.3MB/s), 9630KiB/s-10.6MiB/s (9861kB/s-11.1MB/s), io=32.0MiB (33.6MB), run=755-1276msec rand-rw: (groupid=0, jobs=1): err= 0: pid=11732: Tue Apr 8 14:22:36 2025 read: IOPS=2, BW=11.6MiB/s (12.2MB/s)(12.0MiB/1035msec) clat (msec): min=74, max=801, avg=335.24, stdev=404.88 lat (msec): min=74, max=801, avg=335.24, stdev=404.88 clat percentiles (msec): | 1.00th=[ 75], 5.00th=[ 75], 10.00th=[ 75], 20.00th=[ 75], | 30.00th=[ 75], 40.00th=[ 130], 50.00th=[ 130], 60.00th=[ 130], | 70.00th=[ 802], 80.00th=[ 802], 90.00th=[ 802], 95.00th=[ 802], | 99.00th=[ 802], 99.50th=[ 802], 99.90th=[ 802], 99.95th=[ 802], | 99.99th=[ 802] bw ( KiB/s): min=16384, max=16384, per=100.00%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 250=33.33%, 1000=33.33% cpu : usr=0.00%, sys=8.12%, ctx=34, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=11734: Tue Apr 8 14:22:36 2025 read: IOPS=0, BW=2999KiB/s (3071kB/s)(4096KiB/1366msec) clat (nsec): min=782028k, max=782028k, avg=782028009.00, stdev= 0.00 lat (nsec): min=782030k, max=782030k, avg=782030129.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 785], 5.00th=[ 785], 10.00th=[ 785], 20.00th=[ 785], | 30.00th=[ 785], 40.00th=[ 785], 50.00th=[ 785], 60.00th=[ 785], | 70.00th=[ 785], 80.00th=[ 785], 90.00th=[ 785], 95.00th=[ 785], | 99.00th=[ 785], 99.50th=[ 785], 99.90th=[ 785], 99.95th=[ 785], | 99.99th=[ 785] bw ( KiB/s): min= 8192, max= 8192, per=68.30%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=1, BW=5997KiB/s (6141kB/s)(8192KiB/1366msec) clat (msec): min=248, max=309, avg=278.99, stdev=42.55 lat (msec): min=249, max=309, avg=279.26, stdev=42.58 clat percentiles (msec): | 1.00th=[ 249], 5.00th=[ 249], 10.00th=[ 249], 20.00th=[ 249], | 30.00th=[ 249], 40.00th=[ 249], 50.00th=[ 249], 60.00th=[ 309], | 70.00th=[ 309], 80.00th=[ 309], 90.00th=[ 309], 95.00th=[ 309], | 99.00th=[ 309], 99.50th=[ 309], 99.90th=[ 309], 99.95th=[ 309], | 99.99th=[ 309] lat (msec) : 250=33.33%, 500=33.33%, 1000=33.33% cpu : usr=0.00%, sys=7.11%, ctx=48, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=11735: Tue Apr 8 14:22:36 2025 write: IOPS=2, BW=8879KiB/s (9092kB/s)(12.0MiB/1384msec) clat (msec): min=71, max=1067, avg=456.36, stdev=535.02 lat (msec): min=72, max=1067, avg=456.69, stdev=535.07 clat percentiles (msec): | 1.00th=[ 72], 5.00th=[ 72], 10.00th=[ 72], 20.00th=[ 72], | 30.00th=[ 72], 40.00th=[ 230], 50.00th=[ 230], 60.00th=[ 230], | 70.00th=[ 1070], 80.00th=[ 1070], 90.00th=[ 1070], 95.00th=[ 1070], | 99.00th=[ 1070], 99.50th=[ 1070], 99.90th=[ 1070], 99.95th=[ 1070], | 99.99th=[ 1070] lat (msec) : 100=33.33%, 250=33.33% cpu : usr=0.22%, sys=9.76%, ctx=68, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=11736: Tue Apr 8 14:22:36 2025 write: IOPS=2, BW=9615KiB/s (9846kB/s)(12.0MiB/1278msec) clat (msec): min=118, max=710, avg=413.49, stdev=295.99 lat (msec): min=119, max=711, avg=413.77, stdev=295.99 clat percentiles (msec): | 1.00th=[ 120], 5.00th=[ 120], 10.00th=[ 120], 20.00th=[ 120], | 30.00th=[ 120], 40.00th=[ 409], 50.00th=[ 409], 60.00th=[ 409], | 70.00th=[ 709], 80.00th=[ 709], 90.00th=[ 709], 95.00th=[ 709], | 99.00th=[ 709], 99.50th=[ 709], 99.90th=[ 709], 99.95th=[ 709], | 99.99th=[ 709] bw ( KiB/s): min= 8192, max= 8192, per=34.60%, avg=8192.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 lat (msec) : 250=33.33%, 500=33.33%, 750=33.33% cpu : usr=0.16%, sys=12.14%, ctx=104, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=11.7MiB/s (12.3MB/s), 2999KiB/s-11.6MiB/s (3071kB/s-12.2MB/s), io=16.0MiB (16.8MB), run=1035-1366msec WRITE: bw=23.1MiB/s (24.2MB/s), 5997KiB/s-9615KiB/s (6141kB/s-9846kB/s), io=32.0MiB (33.6MB), run=1278-1384msec