== sanity test 398b: DIO and buffer IO race ============== 16:06:14 (1738962374) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.20974 s, 41.6 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=6817: Fri Feb 7 16:06:28 2025 read: IOPS=131, BW=524KiB/s (537kB/s)(5992KiB/11431msec) clat (usec): min=54, max=179273, avg=3530.21, stdev=5402.61 lat (usec): min=54, max=179274, avg=3531.38, stdev=5402.52 clat percentiles (usec): | 1.00th=[ 60], 5.00th=[ 70], 10.00th=[ 80], 20.00th=[ 1844], | 30.00th=[ 2180], 40.00th=[ 2442], 50.00th=[ 2835], 60.00th=[ 3294], | 70.00th=[ 4047], 80.00th=[ 5342], 90.00th=[ 6783], 95.00th=[ 8291], | 99.00th=[ 12256], 99.50th=[ 16450], 99.90th=[ 38536], 99.95th=[179307], | 99.99th=[179307] bw ( KiB/s): min= 248, max= 760, per=25.00%, avg=520.73, stdev=105.78, samples=22 iops : min= 62, max= 190, avg=130.18, stdev=26.45, samples=22 write: IOPS=137, BW=551KiB/s (564kB/s)(6296KiB/11431msec) clat (usec): min=412, max=28040, avg=3831.75, stdev=3667.70 lat (usec): min=412, max=28040, avg=3832.51, stdev=3667.61 clat percentiles (usec): | 1.00th=[ 490], 5.00th=[ 523], 10.00th=[ 545], 20.00th=[ 603], | 30.00th=[ 840], 40.00th=[ 1614], 50.00th=[ 3064], 60.00th=[ 3949], | 70.00th=[ 4883], 80.00th=[ 6390], 90.00th=[ 8848], 95.00th=[11076], | 99.00th=[16319], 99.50th=[19006], 99.90th=[26084], 99.95th=[27919], | 99.99th=[27919] bw ( KiB/s): min= 296, max= 744, per=24.99%, avg=544.73, stdev=104.13, samples=22 iops : min= 74, max= 186, avg=136.18, stdev=26.03, samples=22 lat (usec) : 100=7.91%, 250=0.72%, 500=1.01%, 750=13.38%, 1000=2.77% lat (msec) : 2=7.06%, 4=32.19%, 10=30.44%, 20=4.17%, 50=0.33% lat (msec) : 250=0.03% cpu : usr=0.15%, sys=23.45%, ctx=5576, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6819: Fri Feb 7 16:06:28 2025 read: IOPS=129, BW=519KiB/s (532kB/s)(5940KiB/11442msec) clat (usec): min=52, max=179663, avg=3453.88, stdev=5600.70 lat (usec): min=52, max=179665, avg=3454.71, stdev=5600.57 clat percentiles (usec): | 1.00th=[ 59], 5.00th=[ 69], 10.00th=[ 77], 20.00th=[ 1713], | 30.00th=[ 2114], 40.00th=[ 2409], 50.00th=[ 2737], 60.00th=[ 3195], | 70.00th=[ 3884], 80.00th=[ 5080], 90.00th=[ 6652], 95.00th=[ 8029], | 99.00th=[ 13042], 99.50th=[ 17433], 99.90th=[ 52691], 99.95th=[179307], | 99.99th=[179307] bw ( KiB/s): min= 135, max= 736, per=24.84%, avg=517.41, stdev=120.15, samples=22 iops : min= 33, max= 184, avg=129.32, stdev=30.15, samples=22 write: IOPS=138, BW=555KiB/s (568kB/s)(6348KiB/11442msec) clat (usec): min=432, max=41970, avg=3904.28, stdev=3979.13 lat (usec): min=432, max=41970, avg=3905.02, stdev=3979.12 clat percentiles (usec): | 1.00th=[ 502], 5.00th=[ 529], 10.00th=[ 545], 20.00th=[ 603], | 30.00th=[ 783], 40.00th=[ 1549], 50.00th=[ 3032], 60.00th=[ 4047], | 70.00th=[ 5014], 80.00th=[ 6390], 90.00th=[ 8979], 95.00th=[11207], | 99.00th=[16450], 99.50th=[20579], 99.90th=[38011], 99.95th=[42206], | 99.99th=[42206] bw ( KiB/s): min= 207, max= 752, per=24.99%, avg=544.68, stdev=112.36, samples=22 iops : min= 51, max= 188, avg=136.14, stdev=28.20, samples=22 lat (usec) : 100=8.33%, 250=0.62%, 500=0.62%, 750=14.49%, 1000=2.96% lat (msec) : 2=8.01%, 4=30.05%, 10=30.08%, 20=4.33%, 50=0.46% lat (msec) : 100=0.03%, 250=0.03% cpu : usr=0.18%, sys=23.45%, ctx=5441, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6820: Fri Feb 7 16:06:28 2025 read: IOPS=132, BW=531KiB/s (544kB/s)(6028KiB/11346msec) clat (usec): min=51, max=178299, avg=3494.46, stdev=5492.06 lat (usec): min=52, max=178300, avg=3495.14, stdev=5492.10 clat percentiles (usec): | 1.00th=[ 58], 5.00th=[ 69], 10.00th=[ 78], 20.00th=[ 277], | 30.00th=[ 2114], 40.00th=[ 2409], 50.00th=[ 2802], 60.00th=[ 3261], | 70.00th=[ 4080], 80.00th=[ 5276], 90.00th=[ 6980], 95.00th=[ 8029], | 99.00th=[ 12256], 99.50th=[ 14222], 99.90th=[ 51643], 99.95th=[179307], | 99.99th=[179307] bw ( KiB/s): min= 256, max= 664, per=25.24%, avg=525.73, stdev=115.70, samples=22 iops : min= 64, max= 166, avg=131.41, stdev=28.91, samples=22 write: IOPS=137, BW=552KiB/s (565kB/s)(6260KiB/11346msec) clat (usec): min=414, max=44343, avg=3814.04, stdev=3791.18 lat (usec): min=415, max=44344, avg=3814.77, stdev=3791.12 clat percentiles (usec): | 1.00th=[ 490], 5.00th=[ 523], 10.00th=[ 553], 20.00th=[ 611], | 30.00th=[ 824], 40.00th=[ 1663], 50.00th=[ 3032], 60.00th=[ 3884], | 70.00th=[ 4817], 80.00th=[ 6259], 90.00th=[ 8586], 95.00th=[11076], | 99.00th=[17171], 99.50th=[19268], 99.90th=[35914], 99.95th=[44303], | 99.99th=[44303] bw ( KiB/s): min= 320, max= 712, per=24.89%, avg=542.50, stdev=92.66, samples=22 iops : min= 80, max= 178, avg=135.59, stdev=23.19, samples=22 lat (usec) : 100=8.66%, 250=1.14%, 500=0.91%, 750=13.48%, 1000=2.44% lat (msec) : 2=7.75%, 4=30.60%, 10=30.40%, 20=4.36%, 50=0.20% lat (msec) : 100=0.03%, 250=0.03% cpu : usr=0.19%, sys=23.68%, ctx=5485, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6821: Fri Feb 7 16:06:28 2025 read: IOPS=131, BW=525KiB/s (538kB/s)(6052KiB/11527msec) clat (usec): min=51, max=57653, avg=3236.56, stdev=3075.35 lat (usec): min=52, max=57653, avg=3237.52, stdev=3075.69 clat percentiles (usec): | 1.00th=[ 58], 5.00th=[ 68], 10.00th=[ 72], 20.00th=[ 101], | 30.00th=[ 1991], 40.00th=[ 2311], 50.00th=[ 2638], 60.00th=[ 3163], | 70.00th=[ 3949], 80.00th=[ 5276], 90.00th=[ 6652], 95.00th=[ 8225], | 99.00th=[10814], 99.50th=[15008], 99.90th=[24249], 99.95th=[57410], | 99.99th=[57410] bw ( KiB/s): min= 56, max= 712, per=25.10%, avg=522.87, stdev=140.53, samples=23 iops : min= 14, max= 178, avg=130.65, stdev=35.11, samples=23 write: IOPS=135, BW=541KiB/s (554kB/s)(6236KiB/11527msec) clat (usec): min=403, max=203943, avg=4180.36, stdev=6590.13 lat (usec): min=404, max=203944, avg=4181.03, stdev=6590.16 clat percentiles (usec): | 1.00th=[ 494], 5.00th=[ 523], 10.00th=[ 545], 20.00th=[ 611], | 30.00th=[ 848], 40.00th=[ 1893], 50.00th=[ 3195], 60.00th=[ 4015], | 70.00th=[ 4948], 80.00th=[ 6456], 90.00th=[ 8979], 95.00th=[ 11731], | 99.00th=[ 22938], 99.50th=[ 26346], 99.90th=[ 34866], 99.95th=[204473], | 99.99th=[204473] bw ( KiB/s): min= 72, max= 808, per=24.71%, avg=538.61, stdev=144.11, samples=23 iops : min= 18, max= 202, avg=134.57, stdev=36.05, samples=23 lat (usec) : 100=9.83%, 250=1.07%, 500=0.98%, 750=13.02%, 1000=2.93% lat (msec) : 2=7.49%, 4=29.82%, 10=30.34%, 20=3.74%, 50=0.72% lat (msec) : 100=0.03%, 250=0.03% cpu : usr=0.20%, sys=23.06%, ctx=5400, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2083KiB/s (2133kB/s), 519KiB/s-531KiB/s (532kB/s-544kB/s), io=23.4MiB (24.6MB), run=11346-11527msec WRITE: bw=2181KiB/s (2233kB/s), 541KiB/s-555KiB/s (554kB/s-568kB/s), io=24.6MiB (25.7MB), run=11346-11527msec rand-rw: (groupid=0, jobs=1): err= 0: pid=6814: Fri Feb 7 16:06:49 2025 read: IOPS=47, BW=190KiB/s (194kB/s)(5992KiB/31585msec) clat (usec): min=1548, max=179649, avg=3309.80, stdev=5408.66 lat (usec): min=1548, max=179651, avg=3310.48, stdev=5408.69 clat percentiles (usec): | 1.00th=[ 1713], 5.00th=[ 1795], 10.00th=[ 1844], 20.00th=[ 1909], | 30.00th=[ 1975], 40.00th=[ 2040], 50.00th=[ 2114], 60.00th=[ 2245], | 70.00th=[ 2507], 80.00th=[ 3294], 90.00th=[ 6521], 95.00th=[ 8979], | 99.00th=[ 14484], 99.50th=[ 18220], 99.90th=[ 48497], 99.95th=[179307], | 99.99th=[179307] bw ( KiB/s): min= 71, max= 328, per=25.12%, avg=189.62, stdev=70.65, samples=63 iops : min= 17, max= 82, avg=47.33, stdev=17.69, samples=63 write: IOPS=49, BW=199KiB/s (204kB/s)(6296KiB/31585msec) clat (usec): min=8713, max=67757, avg=16842.16, stdev=6002.69 lat (usec): min=8714, max=67758, avg=16842.90, stdev=6002.69 clat percentiles (usec): | 1.00th=[ 9765], 5.00th=[10290], 10.00th=[10683], 20.00th=[11600], | 30.00th=[13042], 40.00th=[14746], 50.00th=[16188], 60.00th=[17171], | 70.00th=[18220], 80.00th=[19792], 90.00th=[24773], 95.00th=[28967], | 99.00th=[35914], 99.50th=[38536], 99.90th=[62129], 99.95th=[67634], | 99.99th=[67634] bw ( KiB/s): min= 48, max= 272, per=25.21%, avg=199.14, stdev=57.62, samples=63 iops : min= 12, max= 68, avg=49.71, stdev=14.45, samples=63 lat (msec) : 2=16.99%, 4=23.40%, 10=7.58%, 20=41.80%, 50=10.12% lat (msec) : 100=0.07%, 250=0.03% cpu : usr=0.09%, sys=5.64%, ctx=3441, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6815: Fri Feb 7 16:06:49 2025 read: IOPS=46, BW=188KiB/s (192kB/s)(5940KiB/31674msec) clat (usec): min=1582, max=184183, avg=3267.50, stdev=5375.08 lat (usec): min=1583, max=184186, avg=3268.13, stdev=5375.14 clat percentiles (usec): | 1.00th=[ 1696], 5.00th=[ 1778], 10.00th=[ 1844], 20.00th=[ 1909], | 30.00th=[ 1958], 40.00th=[ 2024], 50.00th=[ 2114], 60.00th=[ 2212], | 70.00th=[ 2474], 80.00th=[ 3294], 90.00th=[ 6783], 95.00th=[ 8717], | 99.00th=[ 13829], 99.50th=[ 17957], 99.90th=[ 28181], 99.95th=[183501], | 99.99th=[183501] bw ( KiB/s): min= 16, max= 344, per=24.89%, avg=187.90, stdev=75.42, samples=63 iops : min= 4, max= 86, avg=46.95, stdev=18.87, samples=63 write: IOPS=50, BW=200KiB/s (205kB/s)(6348KiB/31674msec) clat (usec): min=8405, max=64034, avg=16828.64, stdev=5923.34 lat (usec): min=8406, max=64035, avg=16829.40, stdev=5923.35 clat percentiles (usec): | 1.00th=[ 9634], 5.00th=[10290], 10.00th=[10683], 20.00th=[11731], | 30.00th=[13173], 40.00th=[14615], 50.00th=[16057], 60.00th=[16909], | 70.00th=[17957], 80.00th=[19530], 90.00th=[25560], 95.00th=[29230], | 99.00th=[37487], 99.50th=[39584], 99.90th=[52167], 99.95th=[64226], | 99.99th=[64226] bw ( KiB/s): min= 56, max= 264, per=25.33%, avg=200.10, stdev=55.46, samples=63 iops : min= 14, max= 66, avg=50.00, stdev=13.89, samples=63 lat (msec) : 2=17.19%, 4=22.88%, 10=8.14%, 20=42.19%, 50=9.51% lat (msec) : 100=0.07%, 250=0.03% cpu : usr=0.05%, sys=5.64%, ctx=3436, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6816: Fri Feb 7 16:06:49 2025 read: IOPS=47, BW=190KiB/s (194kB/s)(6028KiB/31791msec) clat (usec): min=1623, max=48459, avg=3408.01, stdev=3283.16 lat (usec): min=1624, max=48459, avg=3408.71, stdev=3283.18 clat percentiles (usec): | 1.00th=[ 1696], 5.00th=[ 1795], 10.00th=[ 1844], 20.00th=[ 1909], | 30.00th=[ 1975], 40.00th=[ 2057], 50.00th=[ 2147], 60.00th=[ 2278], | 70.00th=[ 2606], 80.00th=[ 3949], 90.00th=[ 7242], 95.00th=[ 9765], | 99.00th=[17695], 99.50th=[19530], 99.90th=[31851], 99.95th=[48497], | 99.99th=[48497] bw ( KiB/s): min= 48, max= 384, per=25.12%, avg=189.62, stdev=67.68, samples=63 iops : min= 12, max= 96, avg=47.33, stdev=16.92, samples=63 write: IOPS=49, BW=197KiB/s (202kB/s)(6260KiB/31791msec) clat (msec): min=8, max=201, avg=16.96, stdev= 7.31 lat (msec): min=8, max=201, avg=16.96, stdev= 7.31 clat percentiles (msec): | 1.00th=[ 10], 5.00th=[ 11], 10.00th=[ 11], 20.00th=[ 12], | 30.00th=[ 14], 40.00th=[ 15], 50.00th=[ 17], 60.00th=[ 18], | 70.00th=[ 19], 80.00th=[ 20], 90.00th=[ 25], 95.00th=[ 29], | 99.00th=[ 36], 99.50th=[ 38], 99.90th=[ 50], 99.95th=[ 203], | 99.99th=[ 203] bw ( KiB/s): min= 56, max= 264, per=24.87%, avg=196.48, stdev=57.03, samples=63 iops : min= 14, max= 66, avg=49.05, stdev=14.30, samples=63 lat (msec) : 2=16.67%, 4=22.66%, 10=8.53%, 20=42.15%, 50=9.96% lat (msec) : 250=0.03% cpu : usr=0.10%, sys=5.61%, ctx=3427, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6818: Fri Feb 7 16:06:49 2025 read: IOPS=48, BW=192KiB/s (197kB/s)(6052KiB/31492msec) clat (usec): min=1594, max=31868, avg=3143.51, stdev=2622.41 lat (usec): min=1595, max=31869, avg=3144.15, stdev=2622.44 clat percentiles (usec): | 1.00th=[ 1696], 5.00th=[ 1795], 10.00th=[ 1844], 20.00th=[ 1909], | 30.00th=[ 1958], 40.00th=[ 2024], 50.00th=[ 2114], 60.00th=[ 2245], | 70.00th=[ 2474], 80.00th=[ 3294], 90.00th=[ 6521], 95.00th=[ 9110], | 99.00th=[13698], 99.50th=[15270], 99.90th=[23200], 99.95th=[31851], | 99.99th=[31851] bw ( KiB/s): min= 31, max= 328, per=25.33%, avg=191.26, stdev=70.08, samples=62 iops : min= 7, max= 82, avg=47.74, stdev=17.58, samples=62 write: IOPS=49, BW=198KiB/s (203kB/s)(6236KiB/31492msec) clat (msec): min=8, max=201, avg=17.08, stdev= 7.78 lat (msec): min=8, max=201, avg=17.08, stdev= 7.78 clat percentiles (msec): | 1.00th=[ 10], 5.00th=[ 11], 10.00th=[ 11], 20.00th=[ 12], | 30.00th=[ 14], 40.00th=[ 15], 50.00th=[ 17], 60.00th=[ 18], | 70.00th=[ 19], 80.00th=[ 20], 90.00th=[ 26], 95.00th=[ 31], | 99.00th=[ 38], 99.50th=[ 44], 99.90th=[ 62], 99.95th=[ 203], | 99.99th=[ 203] bw ( KiB/s): min= 56, max= 272, per=24.96%, avg=197.19, stdev=56.96, samples=62 iops : min= 14, max= 68, avg=49.23, stdev=14.28, samples=62 lat (msec) : 2=17.74%, 4=23.27%, 10=8.14%, 20=40.92%, 50=9.77% lat (msec) : 100=0.13%, 250=0.03% cpu : usr=0.10%, sys=5.53%, ctx=3395, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=755KiB/s (773kB/s), 188KiB/s-192KiB/s (192kB/s-197kB/s), io=23.4MiB (24.6MB), run=31492-31791msec WRITE: bw=791KiB/s (810kB/s), 197KiB/s-200KiB/s (202kB/s-205kB/s), io=24.6MiB (25.7MB), run=31492-31791msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=6861: Fri Feb 7 16:06:52 2025 read: IOPS=142, BW=2287KiB/s (2341kB/s)(5920KiB/2589msec) clat (usec): min=53, max=19706, avg=2870.41, stdev=2584.63 lat (usec): min=54, max=19706, avg=2871.15, stdev=2584.83 clat percentiles (usec): | 1.00th=[ 62], 5.00th=[ 69], 10.00th=[ 73], 20.00th=[ 85], | 30.00th=[ 151], 40.00th=[ 2409], 50.00th=[ 2802], 60.00th=[ 3294], | 70.00th=[ 3785], 80.00th=[ 4621], 90.00th=[ 5932], 95.00th=[ 7177], | 99.00th=[10683], 99.50th=[16188], 99.90th=[19792], 99.95th=[19792], | 99.99th=[19792] bw ( KiB/s): min= 768, max= 3040, per=24.32%, avg=2233.60, stdev=884.90, samples=5 iops : min= 48, max= 190, avg=139.60, stdev=55.31, samples=5 write: IOPS=153, BW=2460KiB/s (2519kB/s)(6368KiB/2589msec) clat (usec): min=624, max=32302, avg=3675.14, stdev=4587.89 lat (usec): min=625, max=32303, avg=3676.08, stdev=4587.85 clat percentiles (usec): | 1.00th=[ 668], 5.00th=[ 734], 10.00th=[ 783], 20.00th=[ 857], | 30.00th=[ 971], 40.00th=[ 1205], 50.00th=[ 1893], 60.00th=[ 2868], | 70.00th=[ 3982], 80.00th=[ 5604], 90.00th=[ 8356], 95.00th=[13042], | 99.00th=[25297], 99.50th=[28705], 99.90th=[32375], 99.95th=[32375], | 99.99th=[32375] bw ( KiB/s): min= 896, max= 3392, per=24.77%, avg=2393.60, stdev=928.39, samples=5 iops : min= 56, max= 212, avg=149.60, stdev=58.02, samples=5 lat (usec) : 100=12.63%, 250=2.47%, 500=0.13%, 750=3.52%, 1000=12.24% lat (msec) : 2=10.42%, 4=29.69%, 10=24.35%, 20=3.52%, 50=1.04% cpu : usr=0.23%, sys=25.85%, ctx=1388, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6862: Fri Feb 7 16:06:52 2025 read: IOPS=149, BW=2392KiB/s (2450kB/s)(5744KiB/2401msec) clat (usec): min=56, max=24738, avg=2967.11, stdev=2521.63 lat (usec): min=57, max=24738, avg=2967.65, stdev=2521.63 clat percentiles (usec): | 1.00th=[ 61], 5.00th=[ 69], 10.00th=[ 75], 20.00th=[ 95], | 30.00th=[ 2180], 40.00th=[ 2474], 50.00th=[ 2737], 60.00th=[ 3228], | 70.00th=[ 3687], 80.00th=[ 4359], 90.00th=[ 5735], 95.00th=[ 7439], | 99.00th=[ 9241], 99.50th=[16188], 99.90th=[24773], 99.95th=[24773], | 99.99th=[24773] bw ( KiB/s): min= 1984, max= 2816, per=26.48%, avg=2432.00, stdev=365.79, samples=4 iops : min= 124, max= 176, avg=152.00, stdev=22.86, samples=4 write: IOPS=170, BW=2726KiB/s (2791kB/s)(6544KiB/2401msec) clat (usec): min=628, max=31977, avg=3088.27, stdev=3624.59 lat (usec): min=629, max=31978, avg=3089.10, stdev=3624.60 clat percentiles (usec): | 1.00th=[ 668], 5.00th=[ 709], 10.00th=[ 766], 20.00th=[ 824], | 30.00th=[ 889], 40.00th=[ 1074], 50.00th=[ 1467], 60.00th=[ 2671], | 70.00th=[ 3556], 80.00th=[ 5014], 90.00th=[ 6980], 95.00th=[ 8717], | 99.00th=[21627], 99.50th=[23725], 99.90th=[31851], 99.95th=[31851], | 99.99th=[31851] bw ( KiB/s): min= 2336, max= 3200, per=28.48%, avg=2752.00, stdev=372.27, samples=4 iops : min= 146, max= 200, avg=172.00, stdev=23.27, samples=4 lat (usec) : 100=9.37%, 250=1.82%, 500=0.26%, 750=3.65%, 1000=16.02% lat (msec) : 2=8.46%, 4=34.90%, 10=24.09%, 20=0.52%, 50=0.91% cpu : usr=0.46%, sys=28.42%, ctx=1343, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6863: Fri Feb 7 16:06:52 2025 read: IOPS=153, BW=2457KiB/s (2516kB/s)(6256KiB/2546msec) clat (usec): min=58, max=22719, avg=2958.65, stdev=2631.14 lat (usec): min=58, max=22720, avg=2959.24, stdev=2631.15 clat percentiles (usec): | 1.00th=[ 60], 5.00th=[ 70], 10.00th=[ 73], 20.00th=[ 85], | 30.00th=[ 1958], 40.00th=[ 2409], 50.00th=[ 2835], 60.00th=[ 3326], | 70.00th=[ 3884], 80.00th=[ 4817], 90.00th=[ 6063], 95.00th=[ 7177], | 99.00th=[11338], 99.50th=[13698], 99.90th=[22676], 99.95th=[22676], | 99.99th=[22676] bw ( KiB/s): min= 1248, max= 3104, per=27.11%, avg=2489.60, stdev=746.16, samples=5 iops : min= 78, max= 194, avg=155.60, stdev=46.63, samples=5 write: IOPS=148, BW=2369KiB/s (2426kB/s)(6032KiB/2546msec) clat (usec): min=649, max=41903, avg=3607.07, stdev=4595.29 lat (usec): min=650, max=41905, avg=3608.11, stdev=4595.28 clat percentiles (usec): | 1.00th=[ 668], 5.00th=[ 717], 10.00th=[ 766], 20.00th=[ 816], | 30.00th=[ 922], 40.00th=[ 1139], 50.00th=[ 2089], 60.00th=[ 3064], | 70.00th=[ 3982], 80.00th=[ 5538], 90.00th=[ 7832], 95.00th=[10683], | 99.00th=[22152], 99.50th=[33424], 99.90th=[41681], 99.95th=[41681], | 99.99th=[41681] bw ( KiB/s): min= 1472, max= 2976, per=24.44%, avg=2361.60, stdev=569.02, samples=5 iops : min= 92, max= 186, avg=147.60, stdev=35.56, samples=5 lat (usec) : 100=13.15%, 250=1.69%, 500=0.26%, 750=3.78%, 1000=13.54% lat (msec) : 2=7.55%, 4=30.47%, 10=25.78%, 20=2.86%, 50=0.91% cpu : usr=0.20%, sys=26.13%, ctx=1338, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6864: Fri Feb 7 16:06:52 2025 read: IOPS=144, BW=2313KiB/s (2368kB/s)(6032KiB/2608msec) clat (usec): min=56, max=24927, avg=2947.52, stdev=2799.21 lat (usec): min=56, max=24928, avg=2948.04, stdev=2799.21 clat percentiles (usec): | 1.00th=[ 62], 5.00th=[ 69], 10.00th=[ 74], 20.00th=[ 83], | 30.00th=[ 105], 40.00th=[ 2343], 50.00th=[ 2769], 60.00th=[ 3294], | 70.00th=[ 4047], 80.00th=[ 4948], 90.00th=[ 6259], 95.00th=[ 7111], | 99.00th=[13042], 99.50th=[17433], 99.90th=[25035], 99.95th=[25035], | 99.99th=[25035] bw ( KiB/s): min= 864, max= 2592, per=24.18%, avg=2220.80, stdev=759.49, samples=5 iops : min= 54, max= 162, avg=138.80, stdev=47.47, samples=5 write: IOPS=149, BW=2399KiB/s (2456kB/s)(6256KiB/2608msec) clat (usec): min=621, max=35894, avg=3616.31, stdev=4636.24 lat (usec): min=621, max=35894, avg=3617.16, stdev=4636.26 clat percentiles (usec): | 1.00th=[ 644], 5.00th=[ 701], 10.00th=[ 742], 20.00th=[ 816], | 30.00th=[ 889], 40.00th=[ 1004], 50.00th=[ 1598], 60.00th=[ 2900], | 70.00th=[ 3949], 80.00th=[ 5669], 90.00th=[ 7570], 95.00th=[11469], | 99.00th=[23200], 99.50th=[28705], 99.90th=[35914], 99.95th=[35914], | 99.99th=[35914] bw ( KiB/s): min= 992, max= 3520, per=24.31%, avg=2348.80, stdev=905.27, samples=5 iops : min= 62, max= 220, avg=146.80, stdev=56.58, samples=5 lat (usec) : 100=14.58%, 250=1.04%, 500=0.13%, 750=6.12%, 1000=14.45% lat (msec) : 2=7.16%, 4=26.56%, 10=26.17%, 20=2.47%, 50=1.30% cpu : usr=0.08%, sys=25.78%, ctx=1361, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=9184KiB/s (9404kB/s), 2287KiB/s-2457KiB/s (2341kB/s-2516kB/s), io=23.4MiB (24.5MB), run=2401-2608msec WRITE: bw=9663KiB/s (9894kB/s), 2369KiB/s-2726KiB/s (2426kB/s-2791kB/s), io=24.6MiB (25.8MB), run=2401-2608msec rand-rw: (groupid=0, jobs=1): err= 0: pid=6856: Fri Feb 7 16:06:58 2025 read: IOPS=45, BW=722KiB/s (739kB/s)(5920KiB/8198msec) clat (usec): min=1825, max=29846, avg=4023.87, stdev=3399.66 lat (usec): min=1825, max=29847, avg=4024.60, stdev=3399.63 clat percentiles (usec): | 1.00th=[ 1893], 5.00th=[ 1975], 10.00th=[ 2040], 20.00th=[ 2147], | 30.00th=[ 2245], 40.00th=[ 2409], 50.00th=[ 2671], 60.00th=[ 3359], | 70.00th=[ 4293], 80.00th=[ 5342], 90.00th=[ 6194], 95.00th=[ 9241], | 99.00th=[20579], 99.50th=[27657], 99.90th=[29754], 99.95th=[29754], | 99.99th=[29754] bw ( KiB/s): min= 416, max= 1152, per=24.80%, avg=713.12, stdev=186.39, samples=16 iops : min= 26, max= 72, avg=44.44, stdev=11.64, samples=16 write: IOPS=48, BW=777KiB/s (795kB/s)(6368KiB/8198msec) clat (usec): min=9883, max=44515, avg=16795.60, stdev=4814.29 lat (usec): min=9884, max=44517, avg=16797.02, stdev=4814.32 clat percentiles (usec): | 1.00th=[10290], 5.00th=[10814], 10.00th=[11600], 20.00th=[12387], | 30.00th=[13566], 40.00th=[15139], 50.00th=[16581], 60.00th=[17695], | 70.00th=[18482], 80.00th=[19792], 90.00th=[21890], 95.00th=[25560], | 99.00th=[31851], 99.50th=[44303], 99.90th=[44303], 99.95th=[44303], | 99.99th=[44303] bw ( KiB/s): min= 512, max= 1053, per=25.49%, avg=771.44, stdev=169.08, samples=16 iops : min= 32, max= 65, avg=48.06, stdev=10.43, samples=16 lat (msec) : 2=3.12%, 4=28.78%, 10=14.06%, 20=44.53%, 50=9.51% cpu : usr=0.11%, sys=6.88%, ctx=1017, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6857: Fri Feb 7 16:06:58 2025 read: IOPS=43, BW=690KiB/s (706kB/s)(5744KiB/8326msec) clat (usec): min=1790, max=28284, avg=3819.35, stdev=3130.02 lat (usec): min=1790, max=28285, avg=3819.99, stdev=3130.01 clat percentiles (usec): | 1.00th=[ 1860], 5.00th=[ 1991], 10.00th=[ 2073], 20.00th=[ 2180], | 30.00th=[ 2245], 40.00th=[ 2343], 50.00th=[ 2507], 60.00th=[ 2868], | 70.00th=[ 3851], 80.00th=[ 5211], 90.00th=[ 6521], 95.00th=[10028], | 99.00th=[19006], 99.50th=[23987], 99.90th=[28181], 99.95th=[28181], | 99.99th=[28181] bw ( KiB/s): min= 384, max= 1152, per=23.99%, avg=689.88, stdev=203.77, samples=16 iops : min= 24, max= 72, avg=43.06, stdev=12.76, samples=16 write: IOPS=49, BW=786KiB/s (805kB/s)(6544KiB/8326msec) clat (usec): min=9988, max=42585, avg=16969.89, stdev=5092.22 lat (usec): min=9989, max=42586, avg=16971.23, stdev=5092.24 clat percentiles (usec): | 1.00th=[10290], 5.00th=[11076], 10.00th=[11863], 20.00th=[12780], | 30.00th=[13435], 40.00th=[14746], 50.00th=[16319], 60.00th=[17695], | 70.00th=[18744], 80.00th=[19792], 90.00th=[23200], 95.00th=[27395], | 99.00th=[34866], 99.50th=[35914], 99.90th=[42730], 99.95th=[42730], | 99.99th=[42730] bw ( KiB/s): min= 448, max= 960, per=25.71%, avg=777.87, stdev=172.22, samples=16 iops : min= 28, max= 60, avg=48.56, stdev=10.86, samples=16 lat (msec) : 2=2.34%, 4=32.03%, 10=10.42%, 20=44.66%, 50=10.55% cpu : usr=0.06%, sys=6.69%, ctx=967, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6858: Fri Feb 7 16:06:58 2025 read: IOPS=50, BW=804KiB/s (823kB/s)(6256KiB/7780msec) clat (usec): min=1823, max=23970, avg=3767.55, stdev=2871.22 lat (usec): min=1823, max=23985, avg=3768.22, stdev=2871.48 clat percentiles (usec): | 1.00th=[ 1926], 5.00th=[ 2008], 10.00th=[ 2057], 20.00th=[ 2147], | 30.00th=[ 2245], 40.00th=[ 2376], 50.00th=[ 2507], 60.00th=[ 2802], | 70.00th=[ 3785], 80.00th=[ 5211], 90.00th=[ 6390], 95.00th=[ 9110], | 99.00th=[16581], 99.50th=[20055], 99.90th=[23987], 99.95th=[23987], | 99.99th=[23987] bw ( KiB/s): min= 352, max= 1312, per=28.18%, avg=810.60, stdev=292.54, samples=15 iops : min= 22, max= 82, avg=50.60, stdev=18.37, samples=15 write: IOPS=48, BW=775KiB/s (794kB/s)(6032KiB/7780msec) clat (usec): min=10020, max=42295, avg=16615.11, stdev=5322.62 lat (usec): min=10021, max=42296, avg=16616.48, stdev=5322.50 clat percentiles (usec): | 1.00th=[10290], 5.00th=[10814], 10.00th=[11469], 20.00th=[12256], | 30.00th=[13173], 40.00th=[14091], 50.00th=[15270], 60.00th=[16909], | 70.00th=[18220], 80.00th=[19792], 90.00th=[23200], 95.00th=[26870], | 99.00th=[36963], 99.50th=[39584], 99.90th=[42206], 99.95th=[42206], | 99.99th=[42206] bw ( KiB/s): min= 542, max= 992, per=25.38%, avg=767.87, stdev=165.59, samples=15 iops : min= 33, max= 62, avg=47.93, stdev=10.44, samples=15 lat (msec) : 2=2.21%, 4=33.98%, 10=12.24%, 20=41.67%, 50=9.90% cpu : usr=0.13%, sys=6.98%, ctx=997, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6859: Fri Feb 7 16:06:58 2025 read: IOPS=46, BW=741KiB/s (759kB/s)(6032KiB/8139msec) clat (usec): min=1875, max=29948, avg=4001.14, stdev=3349.04 lat (usec): min=1876, max=29948, avg=4002.29, stdev=3349.06 clat percentiles (usec): | 1.00th=[ 1909], 5.00th=[ 2024], 10.00th=[ 2073], 20.00th=[ 2180], | 30.00th=[ 2278], 40.00th=[ 2409], 50.00th=[ 2573], 60.00th=[ 2900], | 70.00th=[ 4080], 80.00th=[ 5538], 90.00th=[ 6390], 95.00th=[10683], | 99.00th=[19268], 99.50th=[28443], 99.90th=[30016], 99.95th=[30016], | 99.99th=[30016] bw ( KiB/s): min= 448, max= 1216, per=25.66%, avg=738.00, stdev=209.50, samples=16 iops : min= 28, max= 76, avg=46.12, stdev=13.09, samples=16 write: IOPS=48, BW=769KiB/s (787kB/s)(6256KiB/8139msec) clat (usec): min=10138, max=37332, avg=16867.64, stdev=5103.50 lat (usec): min=10140, max=37333, avg=16868.82, stdev=5103.55 clat percentiles (usec): | 1.00th=[10421], 5.00th=[11207], 10.00th=[11469], 20.00th=[12387], | 30.00th=[13435], 40.00th=[14746], 50.00th=[15795], 60.00th=[17695], | 70.00th=[18482], 80.00th=[20055], 90.00th=[23462], 95.00th=[27657], | 99.00th=[34866], 99.50th=[36963], 99.90th=[37487], 99.95th=[37487], | 99.99th=[37487] bw ( KiB/s): min= 448, max= 928, per=25.31%, avg=766.00, stdev=168.30, samples=16 iops : min= 28, max= 58, avg=47.88, stdev=10.52, samples=16 lat (msec) : 2=1.95%, 4=32.16%, 10=12.37%, 20=43.10%, 50=10.42% cpu : usr=0.04%, sys=7.08%, ctx=997, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2877KiB/s (2946kB/s), 690KiB/s-804KiB/s (706kB/s-823kB/s), io=23.4MiB (24.5MB), run=7780-8326msec WRITE: bw=3027KiB/s (3099kB/s), 769KiB/s-786KiB/s (787kB/s-805kB/s), io=24.6MiB (25.8MB), run=7780-8326msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=6885: Fri Feb 7 16:07:00 2025 read: IOPS=12, BW=12.0MiB/s (13.6MB/s)(7168KiB/539msec) clat (usec): min=358, max=81232, avg=33461.42, stdev=24113.77 lat (usec): min=359, max=81233, avg=33462.22, stdev=24113.77 clat percentiles (usec): | 1.00th=[ 359], 5.00th=[ 359], 10.00th=[ 359], 20.00th=[25822], | 30.00th=[27395], 40.00th=[27395], 50.00th=[31065], 60.00th=[31851], | 70.00th=[31851], 80.00th=[36439], 90.00th=[81265], 95.00th=[81265], | 99.00th=[81265], 99.50th=[81265], 99.90th=[81265], 99.95th=[81265], | 99.99th=[81265] bw ( KiB/s): min=12288, max=12288, per=41.10%, avg=12288.00, stdev= 0.00, samples=1 iops : min= 12, max= 12, avg=12.00, stdev= 0.00, samples=1 write: IOPS=9, BW=9499KiB/s (9727kB/s)(5120KiB/539msec) clat (usec): min=28584, max=75016, avg=50023.31, stdev=17782.59 lat (usec): min=28603, max=75057, avg=50063.68, stdev=17785.73 clat percentiles (usec): | 1.00th=[28705], 5.00th=[28705], 10.00th=[28705], 20.00th=[28705], | 30.00th=[40109], 40.00th=[40109], 50.00th=[47973], 60.00th=[47973], | 70.00th=[58459], 80.00th=[58459], 90.00th=[74974], 95.00th=[74974], | 99.00th=[74974], 99.50th=[74974], 99.90th=[74974], 99.95th=[74974], | 99.99th=[74974] bw ( KiB/s): min= 8192, max= 8192, per=19.57%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 lat (usec) : 500=8.33% lat (msec) : 50=66.67%, 100=25.00% cpu : usr=0.00%, sys=33.27%, ctx=691, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6886: Fri Feb 7 16:07:00 2025 read: IOPS=2, BW=2990KiB/s (3062kB/s)(2048KiB/685msec) clat (usec): min=28928, max=38453, avg=33691.02, stdev=6735.64 lat (usec): min=28928, max=38455, avg=33692.18, stdev=6736.32 clat percentiles (usec): | 1.00th=[28967], 5.00th=[28967], 10.00th=[28967], 20.00th=[28967], | 30.00th=[28967], 40.00th=[28967], 50.00th=[28967], 60.00th=[38536], | 70.00th=[38536], 80.00th=[38536], 90.00th=[38536], 95.00th=[38536], | 99.00th=[38536], 99.50th=[38536], 99.90th=[38536], 99.95th=[38536], | 99.99th=[38536] bw ( KiB/s): min= 4096, max= 4096, per=13.70%, avg=4096.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 write: IOPS=14, BW=14.6MiB/s (15.3MB/s)(10.0MiB/685msec) clat (msec): min=16, max=135, avg=56.96, stdev=35.78 lat (msec): min=16, max=135, avg=57.03, stdev=35.76 clat percentiles (msec): | 1.00th=[ 17], 5.00th=[ 17], 10.00th=[ 17], 20.00th=[ 23], | 30.00th=[ 28], 40.00th=[ 45], 50.00th=[ 54], 60.00th=[ 54], | 70.00th=[ 62], 80.00th=[ 62], 90.00th=[ 96], 95.00th=[ 136], | 99.00th=[ 136], 99.50th=[ 136], 99.90th=[ 136], 99.95th=[ 136], | 99.99th=[ 136] bw ( KiB/s): min=10240, max=10240, per=24.46%, avg=10240.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 lat (msec) : 20=8.33%, 50=41.67%, 100=41.67%, 250=8.33% cpu : usr=0.29%, sys=32.46%, ctx=395, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6887: Fri Feb 7 16:07:00 2025 read: IOPS=11, BW=11.4MiB/s (11.9MB/s)(6144KiB/527msec) clat (usec): min=308, max=31195, avg=16797.41, stdev=13308.88 lat (usec): min=309, max=31196, avg=16798.18, stdev=13308.90 clat percentiles (usec): | 1.00th=[ 310], 5.00th=[ 310], 10.00th=[ 310], 20.00th=[ 445], | 30.00th=[ 445], 40.00th=[20579], 50.00th=[20579], 60.00th=[21365], | 70.00th=[27132], 80.00th=[27132], 90.00th=[31327], 95.00th=[31327], | 99.00th=[31327], 99.50th=[31327], 99.90th=[31327], 99.95th=[31327], | 99.99th=[31327] bw ( KiB/s): min=10240, max=10240, per=34.25%, avg=10240.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 write: IOPS=11, BW=11.4MiB/s (11.9MB/s)(6144KiB/527msec) clat (msec): min=29, max=108, avg=62.03, stdev=37.39 lat (msec): min=29, max=108, avg=62.07, stdev=37.39 clat percentiles (msec): | 1.00th=[ 30], 5.00th=[ 30], 10.00th=[ 30], 20.00th=[ 31], | 30.00th=[ 31], 40.00th=[ 40], 50.00th=[ 40], 60.00th=[ 56], | 70.00th=[ 109], 80.00th=[ 109], 90.00th=[ 109], 95.00th=[ 109], | 99.00th=[ 109], 99.50th=[ 109], 99.90th=[ 109], 99.95th=[ 109], | 99.99th=[ 109] bw ( KiB/s): min=12288, max=12288, per=29.36%, avg=12288.00, stdev= 0.00, samples=1 iops : min= 12, max= 12, avg=12.00, stdev= 0.00, samples=1 lat (usec) : 500=16.67% lat (msec) : 50=58.33%, 100=8.33%, 250=16.67% cpu : usr=0.00%, sys=27.57%, ctx=241, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6888: Fri Feb 7 16:07:00 2025 read: IOPS=8, BW=8366KiB/s (8567kB/s)(5120KiB/612msec) clat (usec): min=506, max=58361, avg=23470.92, stdev=24148.56 lat (usec): min=507, max=58362, avg=23471.62, stdev=24148.56 clat percentiles (usec): | 1.00th=[ 506], 5.00th=[ 506], 10.00th=[ 506], 20.00th=[ 506], | 30.00th=[ 529], 40.00th=[ 529], 50.00th=[28443], 60.00th=[28443], | 70.00th=[29492], 80.00th=[29492], 90.00th=[58459], 95.00th=[58459], | 99.00th=[58459], 99.50th=[58459], 99.90th=[58459], 99.95th=[58459], | 99.99th=[58459] bw ( KiB/s): min= 8192, max= 8192, per=27.40%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=11, BW=11.4MiB/s (11.0MB/s)(7168KiB/612msec) clat (msec): min=15, max=149, avg=68.01, stdev=46.60 lat (msec): min=15, max=149, avg=68.07, stdev=46.59 clat percentiles (msec): | 1.00th=[ 16], 5.00th=[ 16], 10.00th=[ 16], 20.00th=[ 25], | 30.00th=[ 42], 40.00th=[ 42], 50.00th=[ 70], 60.00th=[ 77], | 70.00th=[ 77], 80.00th=[ 100], 90.00th=[ 150], 95.00th=[ 150], | 99.00th=[ 150], 99.50th=[ 150], 99.90th=[ 150], 99.95th=[ 150], | 99.99th=[ 150] bw ( KiB/s): min=12288, max=12288, per=29.36%, avg=12288.00, stdev= 0.00, samples=1 iops : min= 12, max= 12, avg=12.00, stdev= 0.00, samples=1 lat (usec) : 750=16.67% lat (msec) : 20=8.33%, 50=33.33%, 100=33.33%, 250=8.33% cpu : usr=0.00%, sys=29.62%, ctx=404, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=29.2MiB/s (30.6MB/s), 2990KiB/s-12.0MiB/s (3062kB/s-13.6MB/s), io=20.0MiB (20.0MB), run=527-685msec WRITE: bw=40.9MiB/s (42.9MB/s), 9499KiB/s-14.6MiB/s (9727kB/s-15.3MB/s), io=28.0MiB (29.4MB), run=527-685msec rand-rw: (groupid=0, jobs=1): err= 0: pid=6880: Fri Feb 7 16:07:00 2025 read: IOPS=15, BW=15.0MiB/s (15.8MB/s)(7168KiB/466msec) clat (usec): min=11614, max=29582, avg=19983.69, stdev=6654.87 lat (usec): min=11615, max=29582, avg=19984.43, stdev=6654.91 clat percentiles (usec): | 1.00th=[11600], 5.00th=[11600], 10.00th=[11600], 20.00th=[12518], | 30.00th=[17171], 40.00th=[17171], 50.00th=[21103], 60.00th=[22152], | 70.00th=[22152], 80.00th=[25822], 90.00th=[29492], 95.00th=[29492], | 99.00th=[29492], 99.50th=[29492], 99.90th=[29492], 99.95th=[29492], | 99.99th=[29492] write: IOPS=10, BW=10.7MiB/s (11.2MB/s)(5120KiB/466msec) clat (usec): min=31601, max=88320, avg=56153.16, stdev=22415.88 lat (usec): min=31665, max=88347, avg=56195.18, stdev=22408.70 clat percentiles (usec): | 1.00th=[31589], 5.00th=[31589], 10.00th=[31589], 20.00th=[31589], | 30.00th=[38536], 40.00th=[38536], 50.00th=[58459], 60.00th=[58459], | 70.00th=[63701], 80.00th=[63701], 90.00th=[88605], 95.00th=[88605], | 99.00th=[88605], 99.50th=[88605], 99.90th=[88605], 99.95th=[88605], | 99.99th=[88605] lat (msec) : 20=25.00%, 50=50.00%, 100=25.00% cpu : usr=0.00%, sys=17.20%, ctx=77, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6881: Fri Feb 7 16:07:00 2025 read: IOPS=2, BW=3003KiB/s (3075kB/s)(2048KiB/682msec) clat (usec): min=18424, max=26957, avg=22691.16, stdev=6033.26 lat (usec): min=18427, max=26958, avg=22693.10, stdev=6031.93 clat percentiles (usec): | 1.00th=[18482], 5.00th=[18482], 10.00th=[18482], 20.00th=[18482], | 30.00th=[18482], 40.00th=[18482], 50.00th=[18482], 60.00th=[26870], | 70.00th=[26870], 80.00th=[26870], 90.00th=[26870], 95.00th=[26870], | 99.00th=[26870], 99.50th=[26870], 99.90th=[26870], 99.95th=[26870], | 99.99th=[26870] bw ( KiB/s): min= 4096, max= 4096, per=15.04%, avg=4096.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 write: IOPS=14, BW=14.7MiB/s (15.4MB/s)(10.0MiB/682msec) clat (usec): min=31823, max=87611, avg=58998.05, stdev=18003.93 lat (usec): min=31896, max=87657, avg=59091.53, stdev=18035.30 clat percentiles (usec): | 1.00th=[31851], 5.00th=[31851], 10.00th=[31851], 20.00th=[40633], | 30.00th=[43254], 40.00th=[46924], 50.00th=[61080], 60.00th=[62653], | 70.00th=[74974], 80.00th=[74974], 90.00th=[77071], 95.00th=[87557], | 99.00th=[87557], 99.50th=[87557], 99.90th=[87557], 99.95th=[87557], | 99.99th=[87557] bw ( KiB/s): min=14336, max=14336, per=37.60%, avg=14336.00, stdev= 0.00, samples=1 iops : min= 14, max= 14, avg=14.00, stdev= 0.00, samples=1 lat (msec) : 20=8.33%, 50=41.67%, 100=50.00% cpu : usr=0.29%, sys=13.95%, ctx=55, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6882: Fri Feb 7 16:07:00 2025 read: IOPS=7, BW=8170KiB/s (8366kB/s)(6144KiB/752msec) clat (msec): min=33, max=111, avg=61.29, stdev=31.49 lat (msec): min=33, max=111, avg=61.29, stdev=31.49 clat percentiles (msec): | 1.00th=[ 34], 5.00th=[ 34], 10.00th=[ 34], 20.00th=[ 41], | 30.00th=[ 41], 40.00th=[ 41], 50.00th=[ 41], 60.00th=[ 56], | 70.00th=[ 88], 80.00th=[ 88], 90.00th=[ 112], 95.00th=[ 112], | 99.00th=[ 112], 99.50th=[ 112], 99.90th=[ 112], 99.95th=[ 112], | 99.99th=[ 112] bw ( KiB/s): min= 8192, max= 8192, per=30.08%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=7, BW=8170KiB/s (8366kB/s)(6144KiB/752msec) clat (msec): min=29, max=121, avg=62.71, stdev=36.77 lat (msec): min=29, max=121, avg=62.75, stdev=36.78 clat percentiles (msec): | 1.00th=[ 30], 5.00th=[ 30], 10.00th=[ 30], 20.00th=[ 39], | 30.00th=[ 39], 40.00th=[ 43], 50.00th=[ 43], 60.00th=[ 50], | 70.00th=[ 95], 80.00th=[ 95], 90.00th=[ 122], 95.00th=[ 122], | 99.00th=[ 122], 99.50th=[ 122], 99.90th=[ 122], 99.95th=[ 122], | 99.99th=[ 122] bw ( KiB/s): min= 8192, max= 8192, per=21.49%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 lat (msec) : 50=58.33%, 100=25.00%, 250=16.67% cpu : usr=0.00%, sys=11.32%, ctx=43, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6883: Fri Feb 7 16:07:00 2025 read: IOPS=7, BW=7711KiB/s (7896kB/s)(5120KiB/664msec) clat (usec): min=12802, max=95271, avg=34244.05, stdev=34498.88 lat (usec): min=12802, max=95272, avg=34244.70, stdev=34498.92 clat percentiles (usec): | 1.00th=[12780], 5.00th=[12780], 10.00th=[12780], 20.00th=[12780], | 30.00th=[15926], 40.00th=[15926], 50.00th=[20841], 60.00th=[20841], | 70.00th=[26346], 80.00th=[26346], 90.00th=[94897], 95.00th=[94897], | 99.00th=[94897], 99.50th=[94897], 99.90th=[94897], 99.95th=[94897], | 99.99th=[94897] bw ( KiB/s): min= 8192, max= 8192, per=30.08%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=10, BW=10.5MiB/s (11.1MB/s)(7168KiB/664msec) clat (usec): min=52673, max=86626, avg=68001.20, stdev=11105.23 lat (usec): min=52711, max=86656, avg=68064.27, stdev=11112.92 clat percentiles (usec): | 1.00th=[52691], 5.00th=[52691], 10.00th=[52691], 20.00th=[62653], | 30.00th=[63701], 40.00th=[63701], 50.00th=[64226], 60.00th=[68682], | 70.00th=[68682], 80.00th=[78119], 90.00th=[86508], 95.00th=[86508], | 99.00th=[86508], 99.50th=[86508], 99.90th=[86508], 99.95th=[86508], | 99.99th=[86508] bw ( KiB/s): min=12288, max=12288, per=32.23%, avg=12288.00, stdev= 0.00, samples=1 iops : min= 12, max= 12, avg=12.00, stdev= 0.00, samples=1 lat (msec) : 20=16.67%, 50=16.67%, 100=66.67% cpu : usr=0.00%, sys=10.71%, ctx=33, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=26.6MiB/s (27.9MB/s), 3003KiB/s-15.0MiB/s (3075kB/s-15.8MB/s), io=20.0MiB (20.0MB), run=466-752msec WRITE: bw=37.2MiB/s (39.0MB/s), 8170KiB/s-14.7MiB/s (8366kB/s-15.4MB/s), io=28.0MiB (29.4MB), run=466-752msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=6893: Fri Feb 7 16:07:01 2025 read: IOPS=6, BW=26.3MiB/s (27.5MB/s)(12.0MiB/457msec) clat (msec): min=59, max=208, avg=133.16, stdev=74.34 lat (msec): min=59, max=208, avg=133.16, stdev=74.34 clat percentiles (msec): | 1.00th=[ 60], 5.00th=[ 60], 10.00th=[ 60], 20.00th=[ 60], | 30.00th=[ 60], 40.00th=[ 132], 50.00th=[ 132], 60.00th=[ 132], | 70.00th=[ 209], 80.00th=[ 209], 90.00th=[ 209], 95.00th=[ 209], | 99.00th=[ 209], 99.50th=[ 209], 99.90th=[ 209], 99.95th=[ 209], | 99.99th=[ 209] lat (msec) : 100=33.33%, 250=66.67% cpu : usr=0.00%, sys=12.28%, ctx=59, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6894: Fri Feb 7 16:07:01 2025 read: IOPS=2, BW=10.6MiB/s (11.1MB/s)(4096KiB/379msec) clat (nsec): min=52923k, max=52923k, avg=52922902.00, stdev= 0.00 lat (nsec): min=52925k, max=52925k, avg=52925444.00, stdev= 0.00 clat percentiles (usec): | 1.00th=[52691], 5.00th=[52691], 10.00th=[52691], 20.00th=[52691], | 30.00th=[52691], 40.00th=[52691], 50.00th=[52691], 60.00th=[52691], | 70.00th=[52691], 80.00th=[52691], 90.00th=[52691], 95.00th=[52691], | 99.00th=[52691], 99.50th=[52691], 99.90th=[52691], 99.95th=[52691], | 99.99th=[52691] write: IOPS=5, BW=21.1MiB/s (22.1MB/s)(8192KiB/379msec) clat (msec): min=122, max=145, avg=133.97, stdev=15.87 lat (msec): min=123, max=145, avg=134.28, stdev=15.80 clat percentiles (msec): | 1.00th=[ 124], 5.00th=[ 124], 10.00th=[ 124], 20.00th=[ 124], | 30.00th=[ 124], 40.00th=[ 124], 50.00th=[ 124], 60.00th=[ 146], | 70.00th=[ 146], 80.00th=[ 146], 90.00th=[ 146], 95.00th=[ 146], | 99.00th=[ 146], 99.50th=[ 146], 99.90th=[ 146], 99.95th=[ 146], | 99.99th=[ 146] lat (msec) : 100=33.33%, 250=66.67% cpu : usr=0.00%, sys=28.31%, ctx=48, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6895: Fri Feb 7 16:07:01 2025 write: IOPS=5, BW=22.3MiB/s (23.4MB/s)(12.0MiB/537msec) clat (msec): min=102, max=207, avg=159.18, stdev=53.14 lat (msec): min=102, max=208, avg=159.45, stdev=53.27 clat percentiles (msec): | 1.00th=[ 103], 5.00th=[ 103], 10.00th=[ 103], 20.00th=[ 103], | 30.00th=[ 103], 40.00th=[ 167], 50.00th=[ 167], 60.00th=[ 167], | 70.00th=[ 209], 80.00th=[ 209], 90.00th=[ 209], 95.00th=[ 209], | 99.00th=[ 209], 99.50th=[ 209], 99.90th=[ 209], 99.95th=[ 209], | 99.99th=[ 209] lat (msec) : 250=100.00% cpu : usr=0.00%, sys=17.91%, ctx=47, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6896: Fri Feb 7 16:07:01 2025 write: IOPS=6, BW=27.2MiB/s (28.5MB/s)(12.0MiB/441msec) clat (msec): min=49, max=262, avg=125.97, stdev=118.88 lat (msec): min=50, max=263, avg=126.33, stdev=118.89 clat percentiles (msec): | 1.00th=[ 50], 5.00th=[ 50], 10.00th=[ 50], 20.00th=[ 50], | 30.00th=[ 50], 40.00th=[ 66], 50.00th=[ 66], 60.00th=[ 66], | 70.00th=[ 264], 80.00th=[ 264], 90.00th=[ 264], 95.00th=[ 264], | 99.00th=[ 264], 99.50th=[ 264], 99.90th=[ 264], 99.95th=[ 264], | 99.99th=[ 264] lat (msec) : 50=33.33%, 100=33.33%, 500=33.33% cpu : usr=0.68%, sys=17.05%, ctx=78, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=35.0MiB/s (36.7MB/s), 10.6MiB/s-26.3MiB/s (11.1MB/s-27.5MB/s), io=16.0MiB (16.8MB), run=379-457msec WRITE: bw=59.6MiB/s (62.5MB/s), 21.1MiB/s-27.2MiB/s (22.1MB/s-28.5MB/s), io=32.0MiB (33.6MB), run=379-537msec rand-rw: (groupid=0, jobs=1): err= 0: pid=6898: Fri Feb 7 16:07:01 2025 read: IOPS=11, BW=47.4MiB/s (49.7MB/s)(12.0MiB/253msec) clat (msec): min=8, max=122, avg=80.44, stdev=62.20 lat (msec): min=8, max=122, avg=80.44, stdev=62.20 clat percentiles (msec): | 1.00th=[ 9], 5.00th=[ 9], 10.00th=[ 9], 20.00th=[ 9], | 30.00th=[ 9], 40.00th=[ 110], 50.00th=[ 110], 60.00th=[ 110], | 70.00th=[ 123], 80.00th=[ 123], 90.00th=[ 123], 95.00th=[ 123], | 99.00th=[ 123], 99.50th=[ 123], 99.90th=[ 123], 99.95th=[ 123], | 99.99th=[ 123] lat (msec) : 10=33.33%, 250=66.67% cpu : usr=0.00%, sys=35.97%, ctx=1449, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6899: Fri Feb 7 16:07:01 2025 read: IOPS=1, BW=6896KiB/s (7061kB/s)(4096KiB/594msec) clat (nsec): min=117005k, max=117005k, avg=117005164.00, stdev= 0.00 lat (nsec): min=117006k, max=117006k, avg=117006477.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 117], 5.00th=[ 117], 10.00th=[ 117], 20.00th=[ 117], | 30.00th=[ 117], 40.00th=[ 117], 50.00th=[ 117], 60.00th=[ 117], | 70.00th=[ 117], 80.00th=[ 117], 90.00th=[ 117], 95.00th=[ 117], | 99.00th=[ 117], 99.50th=[ 117], 99.90th=[ 117], 99.95th=[ 117], | 99.99th=[ 117] bw ( KiB/s): min= 8159, max= 8159, per=29.58%, avg=8159.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=3, BW=13.5MiB/s (14.1MB/s)(8192KiB/594msec) clat (msec): min=182, max=271, avg=226.98, stdev=62.78 lat (msec): min=182, max=271, avg=227.15, stdev=62.70 clat percentiles (msec): | 1.00th=[ 184], 5.00th=[ 184], 10.00th=[ 184], 20.00th=[ 184], | 30.00th=[ 184], 40.00th=[ 184], 50.00th=[ 184], 60.00th=[ 271], | 70.00th=[ 271], 80.00th=[ 271], 90.00th=[ 271], 95.00th=[ 271], | 99.00th=[ 271], 99.50th=[ 271], 99.90th=[ 271], 99.95th=[ 271], | 99.99th=[ 271] bw ( KiB/s): min= 8159, max= 8159, per=14.94%, avg=8159.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.17%, sys=27.15%, ctx=776, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6900: Fri Feb 7 16:07:01 2025 write: IOPS=5, BW=20.0MiB/s (20.0MB/s)(12.0MiB/600msec) clat (msec): min=81, max=285, avg=197.56, stdev=104.75 lat (msec): min=81, max=285, avg=197.79, stdev=104.74 clat percentiles (msec): | 1.00th=[ 82], 5.00th=[ 82], 10.00th=[ 82], 20.00th=[ 82], | 30.00th=[ 82], 40.00th=[ 226], 50.00th=[ 226], 60.00th=[ 226], | 70.00th=[ 288], 80.00th=[ 288], 90.00th=[ 288], 95.00th=[ 288], | 99.00th=[ 288], 99.50th=[ 288], 99.90th=[ 288], 99.95th=[ 288], | 99.99th=[ 288] bw ( KiB/s): min= 8192, max= 8192, per=15.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 250=33.33%, 500=33.33% cpu : usr=0.00%, sys=33.22%, ctx=54, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6901: Fri Feb 7 16:07:01 2025 write: IOPS=5, BW=23.5MiB/s (24.7MB/s)(12.0MiB/510msec) clat (msec): min=98, max=211, avg=167.10, stdev=59.81 lat (msec): min=99, max=211, avg=167.37, stdev=59.83 clat percentiles (msec): | 1.00th=[ 100], 5.00th=[ 100], 10.00th=[ 100], 20.00th=[ 100], | 30.00th=[ 100], 40.00th=[ 192], 50.00th=[ 192], 60.00th=[ 192], | 70.00th=[ 211], 80.00th=[ 211], 90.00th=[ 211], 95.00th=[ 211], | 99.00th=[ 211], 99.50th=[ 211], 99.90th=[ 211], 99.95th=[ 211], | 99.99th=[ 211] bw ( KiB/s): min=16384, max=16384, per=30.00%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 250=66.67% cpu : usr=0.00%, sys=51.67%, ctx=68, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=26.9MiB/s (28.2MB/s), 6896KiB/s-47.4MiB/s (7061kB/s-49.7MB/s), io=16.0MiB (16.8MB), run=253-594msec WRITE: bw=53.3MiB/s (55.9MB/s), 13.5MiB/s-23.5MiB/s (14.1MB/s-24.7MB/s), io=32.0MiB (33.6MB), run=510-600msec