== sanity test 398b: DIO and buffer IO race ============== 02:16:41 (1744179401) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.10877 s, 45.4 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=31865: Wed Apr 9 02:16:55 2025 read: IOPS=134, BW=537KiB/s (550kB/s)(5992KiB/11160msec) clat (usec): min=57, max=20536, avg=4091.56, stdev=2340.64 lat (usec): min=58, max=20536, avg=4092.20, stdev=2340.66 clat percentiles (usec): | 1.00th=[ 67], 5.00th=[ 1582], 10.00th=[ 1975], 20.00th=[ 2343], | 30.00th=[ 2704], 40.00th=[ 3130], 50.00th=[ 3654], 60.00th=[ 4293], | 70.00th=[ 4817], 80.00th=[ 5538], 90.00th=[ 6783], 95.00th=[ 8455], | 99.00th=[12256], 99.50th=[13173], 99.90th=[17171], 99.95th=[20579], | 99.99th=[20579] bw ( KiB/s): min= 256, max= 704, per=25.07%, avg=532.91, stdev=90.81, samples=22 iops : min= 64, max= 176, avg=133.18, stdev=22.67, samples=22 write: IOPS=141, BW=564KiB/s (578kB/s)(6296KiB/11160msec) clat (usec): min=414, max=170758, avg=3145.75, stdev=5012.48 lat (usec): min=415, max=170759, avg=3146.75, stdev=5013.00 clat percentiles (usec): | 1.00th=[ 490], 5.00th=[ 519], 10.00th=[ 529], 20.00th=[ 562], | 30.00th=[ 701], 40.00th=[ 1926], 50.00th=[ 2540], 60.00th=[ 3064], | 70.00th=[ 3752], 80.00th=[ 5080], 90.00th=[ 6915], 95.00th=[ 8586], | 99.00th=[ 11731], 99.50th=[ 12649], 99.90th=[ 16450], 99.95th=[170918], | 99.99th=[170918] bw ( KiB/s): min= 304, max= 720, per=25.12%, avg=559.09, stdev=89.13, samples=22 iops : min= 76, max= 180, avg=139.73, stdev=22.28, samples=22 lat (usec) : 100=2.18%, 250=0.16%, 500=1.11%, 750=14.94%, 1000=1.99% lat (msec) : 2=5.63%, 4=37.83%, 10=33.72%, 20=2.38%, 50=0.03% lat (msec) : 250=0.03% cpu : usr=0.14%, sys=22.41%, ctx=5458, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31866: Wed Apr 9 02:16:55 2025 read: IOPS=131, BW=526KiB/s (539kB/s)(5940KiB/11290msec) clat (usec): min=62, max=21273, avg=4063.30, stdev=2271.66 lat (usec): min=62, max=21274, avg=4064.29, stdev=2271.44 clat percentiles (usec): | 1.00th=[ 68], 5.00th=[ 1713], 10.00th=[ 1991], 20.00th=[ 2376], | 30.00th=[ 2704], 40.00th=[ 3097], 50.00th=[ 3589], 60.00th=[ 4228], | 70.00th=[ 4817], 80.00th=[ 5604], 90.00th=[ 6783], 95.00th=[ 8160], | 99.00th=[11731], 99.50th=[13304], 99.90th=[19006], 99.95th=[21365], | 99.99th=[21365] bw ( KiB/s): min= 288, max= 718, per=24.54%, avg=521.64, stdev=84.28, samples=22 iops : min= 72, max= 179, avg=130.32, stdev=21.06, samples=22 write: IOPS=140, BW=562KiB/s (576kB/s)(6348KiB/11290msec) clat (usec): min=418, max=176539, avg=3259.56, stdev=5140.83 lat (usec): min=419, max=176540, avg=3261.00, stdev=5140.60 clat percentiles (usec): | 1.00th=[ 486], 5.00th=[ 515], 10.00th=[ 537], 20.00th=[ 570], | 30.00th=[ 725], 40.00th=[ 2040], 50.00th=[ 2638], 60.00th=[ 3195], | 70.00th=[ 4047], 80.00th=[ 5276], 90.00th=[ 6915], 95.00th=[ 8356], | 99.00th=[ 12256], 99.50th=[ 13173], 99.90th=[ 18220], 99.95th=[177210], | 99.99th=[177210] bw ( KiB/s): min= 352, max= 728, per=24.75%, avg=551.05, stdev=86.75, samples=22 iops : min= 88, max= 182, avg=137.68, stdev=21.70, samples=22 lat (usec) : 100=1.86%, 250=0.20%, 500=1.27%, 750=14.58%, 1000=1.99% lat (msec) : 2=5.47%, 4=37.86%, 10=34.60%, 20=2.12%, 50=0.03% lat (msec) : 250=0.03% cpu : usr=0.19%, sys=22.03%, ctx=5633, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31867: Wed Apr 9 02:16:55 2025 read: IOPS=135, BW=542KiB/s (555kB/s)(6028KiB/11120msec) clat (usec): min=57, max=174461, avg=4154.21, stdev=4958.96 lat (usec): min=57, max=174462, avg=4155.51, stdev=4959.17 clat percentiles (usec): | 1.00th=[ 68], 5.00th=[ 1696], 10.00th=[ 1975], 20.00th=[ 2311], | 30.00th=[ 2638], 40.00th=[ 3097], 50.00th=[ 3621], 60.00th=[ 4228], | 70.00th=[ 4817], 80.00th=[ 5473], 90.00th=[ 6652], 95.00th=[ 8029], | 99.00th=[ 12256], 99.50th=[ 14615], 99.90th=[ 21365], 99.95th=[175113], | 99.99th=[175113] bw ( KiB/s): min= 288, max= 688, per=25.43%, avg=540.55, stdev=85.67, samples=22 iops : min= 72, max= 172, avg=135.09, stdev=21.39, samples=22 write: IOPS=140, BW=563KiB/s (576kB/s)(6260KiB/11120msec) clat (usec): min=428, max=22760, avg=3056.12, stdev=2755.09 lat (usec): min=429, max=22760, avg=3057.63, stdev=2754.88 clat percentiles (usec): | 1.00th=[ 486], 5.00th=[ 515], 10.00th=[ 529], 20.00th=[ 570], | 30.00th=[ 693], 40.00th=[ 1500], 50.00th=[ 2507], 60.00th=[ 3097], | 70.00th=[ 3949], 80.00th=[ 5211], 90.00th=[ 6849], 95.00th=[ 8291], | 99.00th=[11863], 99.50th=[14484], 99.90th=[16909], 99.95th=[22676], | 99.99th=[22676] bw ( KiB/s): min= 352, max= 704, per=25.12%, avg=559.14, stdev=84.96, samples=22 iops : min= 88, max= 176, avg=139.73, stdev=21.29, samples=22 lat (usec) : 100=1.86%, 250=0.16%, 500=1.37%, 750=15.04%, 1000=1.92% lat (msec) : 2=6.28%, 4=36.95%, 10=34.21%, 20=2.12%, 50=0.07% lat (msec) : 250=0.03% cpu : usr=0.25%, sys=22.34%, ctx=5607, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31868: Wed Apr 9 02:16:55 2025 read: IOPS=136, BW=544KiB/s (557kB/s)(6052KiB/11117msec) clat (usec): min=56, max=177149, avg=4143.65, stdev=4980.61 lat (usec): min=56, max=177149, avg=4144.42, stdev=4980.60 clat percentiles (usec): | 1.00th=[ 68], 5.00th=[ 1549], 10.00th=[ 2008], 20.00th=[ 2409], | 30.00th=[ 2671], 40.00th=[ 3130], 50.00th=[ 3589], 60.00th=[ 4146], | 70.00th=[ 4817], 80.00th=[ 5538], 90.00th=[ 6783], 95.00th=[ 7963], | 99.00th=[ 12387], 99.50th=[ 13960], 99.90th=[ 17695], 99.95th=[177210], | 99.99th=[177210] bw ( KiB/s): min= 248, max= 648, per=25.51%, avg=542.36, stdev=85.30, samples=22 iops : min= 62, max= 162, avg=135.55, stdev=21.29, samples=22 write: IOPS=140, BW=561KiB/s (574kB/s)(6236KiB/11117msec) clat (usec): min=400, max=19583, avg=3059.97, stdev=2761.30 lat (usec): min=400, max=19584, avg=3061.44, stdev=2760.98 clat percentiles (usec): | 1.00th=[ 482], 5.00th=[ 510], 10.00th=[ 537], 20.00th=[ 570], | 30.00th=[ 652], 40.00th=[ 1582], 50.00th=[ 2474], 60.00th=[ 3130], | 70.00th=[ 3916], 80.00th=[ 5276], 90.00th=[ 6980], 95.00th=[ 8225], | 99.00th=[11600], 99.50th=[12911], 99.90th=[18744], 99.95th=[19530], | 99.99th=[19530] bw ( KiB/s): min= 312, max= 736, per=25.17%, avg=560.23, stdev=92.97, samples=22 iops : min= 78, max= 184, avg=140.00, stdev=23.26, samples=22 lat (usec) : 100=2.02%, 250=0.23%, 500=1.56%, 750=15.33%, 1000=1.63% lat (msec) : 2=5.76%, 4=37.43%, 10=33.69%, 20=2.31%, 250=0.03% cpu : usr=0.24%, sys=21.92%, ctx=5504, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2127KiB/s (2178kB/s), 526KiB/s-544KiB/s (539kB/s-557kB/s), io=23.4MiB (24.6MB), run=11117-11290msec WRITE: bw=2227KiB/s (2280kB/s), 561KiB/s-564KiB/s (574kB/s-578kB/s), io=24.6MiB (25.7MB), run=11117-11290msec rand-rw: (groupid=0, jobs=1): err= 0: pid=31869: Wed Apr 9 02:17:01 2025 read: IOPS=88, BW=355KiB/s (363kB/s)(5992KiB/16892msec) clat (usec): min=1489, max=155242, avg=4441.08, stdev=4735.07 lat (usec): min=1489, max=155243, avg=4442.07, stdev=4735.06 clat percentiles (usec): | 1.00th=[ 1713], 5.00th=[ 1942], 10.00th=[ 2073], 20.00th=[ 2442], | 30.00th=[ 2769], 40.00th=[ 3195], 50.00th=[ 3589], 60.00th=[ 3949], | 70.00th=[ 4359], 80.00th=[ 5604], 90.00th=[ 8029], 95.00th=[ 10290], | 99.00th=[ 14091], 99.50th=[ 16057], 99.90th=[ 23987], 99.95th=[156238], | 99.99th=[156238] bw ( KiB/s): min= 160, max= 688, per=25.12%, avg=353.18, stdev=114.78, samples=33 iops : min= 40, max= 172, avg=88.27, stdev=28.70, samples=33 write: IOPS=93, BW=373KiB/s (382kB/s)(6296KiB/16892msec) clat (usec): min=2257, max=23346, avg=6450.74, stdev=3006.54 lat (usec): min=2258, max=23346, avg=6451.72, stdev=3006.54 clat percentiles (usec): | 1.00th=[ 2573], 5.00th=[ 3032], 10.00th=[ 3294], 20.00th=[ 3884], | 30.00th=[ 4555], 40.00th=[ 5211], 50.00th=[ 5932], 60.00th=[ 6652], | 70.00th=[ 7242], 80.00th=[ 8291], 90.00th=[10290], 95.00th=[12125], | 99.00th=[17433], 99.50th=[19268], 99.90th=[22152], 99.95th=[23462], | 99.99th=[23462] bw ( KiB/s): min= 176, max= 752, per=25.18%, avg=370.64, stdev=104.49, samples=33 iops : min= 44, max= 188, avg=92.64, stdev=26.14, samples=33 lat (msec) : 2=3.65%, 4=37.11%, 10=50.68%, 20=8.27%, 50=0.26% lat (msec) : 250=0.03% cpu : usr=0.17%, sys=12.18%, ctx=3571, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31870: Wed Apr 9 02:17:01 2025 read: IOPS=87, BW=352KiB/s (360kB/s)(5940KiB/16886msec) clat (usec): min=1560, max=181908, avg=4459.40, stdev=5373.84 lat (usec): min=1561, max=181909, avg=4460.35, stdev=5373.83 clat percentiles (usec): | 1.00th=[ 1713], 5.00th=[ 1926], 10.00th=[ 2089], 20.00th=[ 2376], | 30.00th=[ 2737], 40.00th=[ 3097], 50.00th=[ 3589], 60.00th=[ 3949], | 70.00th=[ 4359], 80.00th=[ 5407], 90.00th=[ 8455], 95.00th=[ 10421], | 99.00th=[ 14091], 99.50th=[ 16909], 99.90th=[ 24511], 99.95th=[181404], | 99.99th=[181404] bw ( KiB/s): min= 112, max= 720, per=25.06%, avg=352.30, stdev=118.35, samples=33 iops : min= 28, max= 180, avg=87.97, stdev=29.55, samples=33 write: IOPS=93, BW=376KiB/s (385kB/s)(6348KiB/16886msec) clat (usec): min=2237, max=25074, avg=6425.04, stdev=3042.60 lat (usec): min=2238, max=25074, avg=6425.96, stdev=3042.54 clat percentiles (usec): | 1.00th=[ 2671], 5.00th=[ 3097], 10.00th=[ 3392], 20.00th=[ 3916], | 30.00th=[ 4621], 40.00th=[ 5276], 50.00th=[ 5866], 60.00th=[ 6456], | 70.00th=[ 7046], 80.00th=[ 8160], 90.00th=[10552], 95.00th=[12387], | 99.00th=[17433], 99.50th=[21365], 99.90th=[22938], 99.95th=[25035], | 99.99th=[25035] bw ( KiB/s): min= 168, max= 696, per=25.30%, avg=372.42, stdev=104.56, samples=33 iops : min= 42, max= 174, avg=93.00, stdev=26.12, samples=33 lat (msec) : 2=3.58%, 4=37.14%, 10=50.36%, 20=8.53%, 50=0.36% lat (msec) : 250=0.03% cpu : usr=0.15%, sys=12.29%, ctx=3541, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31871: Wed Apr 9 02:17:01 2025 read: IOPS=88, BW=353KiB/s (361kB/s)(6028KiB/17076msec) clat (usec): min=1615, max=24499, avg=4517.41, stdev=2850.00 lat (usec): min=1615, max=24500, avg=4518.16, stdev=2849.97 clat percentiles (usec): | 1.00th=[ 1795], 5.00th=[ 1975], 10.00th=[ 2147], 20.00th=[ 2540], | 30.00th=[ 2802], 40.00th=[ 3195], 50.00th=[ 3654], 60.00th=[ 4047], | 70.00th=[ 4490], 80.00th=[ 5735], 90.00th=[ 8979], 95.00th=[10683], | 99.00th=[13829], 99.50th=[16909], 99.90th=[20317], 99.95th=[24511], | 99.99th=[24511] bw ( KiB/s): min= 144, max= 680, per=25.07%, avg=352.52, stdev=96.36, samples=33 iops : min= 36, max= 170, avg=87.97, stdev=24.01, samples=33 write: IOPS=91, BW=367KiB/s (375kB/s)(6260KiB/17076msec) clat (msec): min=2, max=164, avg= 6.50, stdev= 5.05 lat (msec): min=2, max=164, avg= 6.51, stdev= 5.05 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 4], 10.00th=[ 4], 20.00th=[ 4], | 30.00th=[ 5], 40.00th=[ 6], 50.00th=[ 6], 60.00th=[ 7], | 70.00th=[ 8], 80.00th=[ 9], 90.00th=[ 11], 95.00th=[ 13], | 99.00th=[ 18], 99.50th=[ 22], 99.90th=[ 27], 99.95th=[ 165], | 99.99th=[ 165] bw ( KiB/s): min= 176, max= 728, per=24.69%, avg=363.39, stdev=121.00, samples=33 iops : min= 44, max= 182, avg=90.70, stdev=30.23, samples=33 lat (msec) : 2=2.83%, 4=37.47%, 10=50.98%, 20=8.30%, 50=0.39% lat (msec) : 250=0.03% cpu : usr=0.22%, sys=12.11%, ctx=3596, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31886: Wed Apr 9 02:17:01 2025 read: IOPS=90, BW=360KiB/s (369kB/s)(6052KiB/16796msec) clat (usec): min=1513, max=21924, avg=4324.13, stdev=2709.43 lat (usec): min=1514, max=21925, avg=4324.92, stdev=2709.42 clat percentiles (usec): | 1.00th=[ 1696], 5.00th=[ 1926], 10.00th=[ 2089], 20.00th=[ 2376], | 30.00th=[ 2704], 40.00th=[ 3064], 50.00th=[ 3556], 60.00th=[ 3949], | 70.00th=[ 4359], 80.00th=[ 5407], 90.00th=[ 8455], 95.00th=[10159], | 99.00th=[13698], 99.50th=[14746], 99.90th=[19530], 99.95th=[21890], | 99.99th=[21890] bw ( KiB/s): min= 192, max= 672, per=25.51%, avg=358.67, stdev=104.04, samples=33 iops : min= 48, max= 168, avg=89.58, stdev=26.05, samples=33 write: IOPS=92, BW=371KiB/s (380kB/s)(6236KiB/16796msec) clat (msec): min=2, max=182, avg= 6.53, stdev= 5.44 lat (msec): min=2, max=182, avg= 6.53, stdev= 5.44 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 4], 10.00th=[ 4], 20.00th=[ 4], | 30.00th=[ 5], 40.00th=[ 6], 50.00th=[ 6], 60.00th=[ 7], | 70.00th=[ 8], 80.00th=[ 9], 90.00th=[ 11], 95.00th=[ 13], | 99.00th=[ 19], 99.50th=[ 21], 99.90th=[ 24], 99.95th=[ 184], | 99.99th=[ 184] bw ( KiB/s): min= 208, max= 704, per=25.11%, avg=369.58, stdev=103.26, samples=33 iops : min= 52, max= 176, avg=92.30, stdev=25.84, samples=33 lat (msec) : 2=3.65%, 4=38.15%, 10=49.84%, 20=8.07%, 50=0.26% lat (msec) : 250=0.03% cpu : usr=0.22%, sys=12.04%, ctx=3577, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1406KiB/s (1440kB/s), 352KiB/s-360KiB/s (360kB/s-369kB/s), io=23.4MiB (24.6MB), run=16796-17076msec WRITE: bw=1472KiB/s (1508kB/s), 367KiB/s-376KiB/s (375kB/s-385kB/s), io=24.6MiB (25.7MB), run=16796-17076msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=31920: Wed Apr 9 02:17:06 2025 read: IOPS=114, BW=1832KiB/s (1876kB/s)(5920KiB/3231msec) clat (usec): min=65, max=19565, avg=4540.56, stdev=2661.95 lat (usec): min=66, max=19565, avg=4541.29, stdev=2662.25 clat percentiles (usec): | 1.00th=[ 71], 5.00th=[ 87], 10.00th=[ 2343], 20.00th=[ 2769], | 30.00th=[ 3097], 40.00th=[ 3523], 50.00th=[ 4047], 60.00th=[ 4621], | 70.00th=[ 5473], 80.00th=[ 6128], 90.00th=[ 7504], 95.00th=[ 9241], | 99.00th=[13829], 99.50th=[16909], 99.90th=[19530], 99.95th=[19530], | 99.99th=[19530] bw ( KiB/s): min= 1312, max= 2176, per=25.46%, avg=1781.33, stdev=299.28, samples=6 iops : min= 82, max= 136, avg=111.33, stdev=18.70, samples=6 write: IOPS=123, BW=1971KiB/s (2018kB/s)(6368KiB/3231msec) clat (usec): min=696, max=14942, avg=3755.65, stdev=3125.08 lat (usec): min=697, max=14943, avg=3756.49, stdev=3125.09 clat percentiles (usec): | 1.00th=[ 717], 5.00th=[ 766], 10.00th=[ 816], 20.00th=[ 873], | 30.00th=[ 1090], 40.00th=[ 2114], 50.00th=[ 2999], 60.00th=[ 3621], | 70.00th=[ 4752], 80.00th=[ 6456], 90.00th=[ 8586], 95.00th=[ 9896], | 99.00th=[13435], 99.50th=[14353], 99.90th=[15008], 99.95th=[15008], | 99.99th=[15008] bw ( KiB/s): min= 1504, max= 2304, per=26.30%, avg=1936.00, stdev=373.59, samples=6 iops : min= 94, max= 144, avg=121.00, stdev=23.35, samples=6 lat (usec) : 100=2.60%, 250=0.52%, 500=0.26%, 750=1.43%, 1000=13.02% lat (msec) : 2=6.12%, 4=32.68%, 10=39.19%, 20=4.17% cpu : usr=0.09%, sys=24.49%, ctx=1473, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31921: Wed Apr 9 02:17:06 2025 read: IOPS=104, BW=1678KiB/s (1718kB/s)(5744KiB/3423msec) clat (usec): min=68, max=22080, avg=4802.59, stdev=2938.86 lat (usec): min=68, max=22081, avg=4805.21, stdev=2938.93 clat percentiles (usec): | 1.00th=[ 73], 5.00th=[ 84], 10.00th=[ 2278], 20.00th=[ 2802], | 30.00th=[ 3294], 40.00th=[ 3851], 50.00th=[ 4490], 60.00th=[ 5014], | 70.00th=[ 5604], 80.00th=[ 6718], 90.00th=[ 7832], 95.00th=[ 9241], | 99.00th=[17433], 99.50th=[18744], 99.90th=[22152], 99.95th=[22152], | 99.99th=[22152] bw ( KiB/s): min= 1312, max= 1888, per=23.01%, avg=1610.00, stdev=255.07, samples=6 iops : min= 82, max= 118, avg=100.50, stdev=15.97, samples=6 write: IOPS=119, BW=1912KiB/s (1958kB/s)(6544KiB/3423msec) clat (usec): min=664, max=18808, avg=4002.27, stdev=3173.11 lat (usec): min=665, max=18809, avg=4003.11, stdev=3173.14 clat percentiles (usec): | 1.00th=[ 717], 5.00th=[ 791], 10.00th=[ 824], 20.00th=[ 906], | 30.00th=[ 1729], 40.00th=[ 2737], 50.00th=[ 3228], 60.00th=[ 4015], | 70.00th=[ 4883], 80.00th=[ 6259], 90.00th=[ 8356], 95.00th=[10421], | 99.00th=[14091], 99.50th=[15795], 99.90th=[18744], 99.95th=[18744], | 99.99th=[18744] bw ( KiB/s): min= 1440, max= 2080, per=24.63%, avg=1812.67, stdev=211.06, samples=6 iops : min= 90, max= 130, avg=113.17, stdev=13.21, samples=6 lat (usec) : 100=3.39%, 250=0.52%, 500=0.13%, 750=1.04%, 1000=10.55% lat (msec) : 2=5.47%, 4=30.21%, 10=43.62%, 20=4.95%, 50=0.13% cpu : usr=0.12%, sys=23.63%, ctx=1595, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31922: Wed Apr 9 02:17:06 2025 read: IOPS=118, BW=1894KiB/s (1939kB/s)(6256KiB/3303msec) clat (usec): min=61, max=28378, avg=4691.44, stdev=3011.98 lat (usec): min=61, max=28379, avg=4692.04, stdev=3011.95 clat percentiles (usec): | 1.00th=[ 70], 5.00th=[ 93], 10.00th=[ 2343], 20.00th=[ 2737], | 30.00th=[ 3130], 40.00th=[ 3621], 50.00th=[ 4080], 60.00th=[ 4621], | 70.00th=[ 5473], 80.00th=[ 6325], 90.00th=[ 7832], 95.00th=[ 9372], | 99.00th=[17695], 99.50th=[20841], 99.90th=[28443], 99.95th=[28443], | 99.99th=[28443] bw ( KiB/s): min= 1312, max= 2043, per=25.83%, avg=1807.17, stdev=266.65, samples=6 iops : min= 82, max= 127, avg=112.83, stdev=16.55, samples=6 write: IOPS=114, BW=1826KiB/s (1870kB/s)(6032KiB/3303msec) clat (usec): min=665, max=16505, avg=3854.41, stdev=3087.75 lat (usec): min=665, max=16506, avg=3858.50, stdev=3092.79 clat percentiles (usec): | 1.00th=[ 685], 5.00th=[ 791], 10.00th=[ 824], 20.00th=[ 898], | 30.00th=[ 1467], 40.00th=[ 2540], 50.00th=[ 3097], 60.00th=[ 3785], | 70.00th=[ 4752], 80.00th=[ 6325], 90.00th=[ 8455], 95.00th=[10159], | 99.00th=[13435], 99.50th=[15533], 99.90th=[16450], 99.95th=[16450], | 99.99th=[16450] bw ( KiB/s): min= 1568, max= 2080, per=24.12%, avg=1775.33, stdev=177.24, samples=6 iops : min= 98, max= 130, avg=110.83, stdev=11.07, samples=6 lat (usec) : 100=2.86%, 250=0.39%, 500=0.13%, 750=1.30%, 1000=10.55% lat (msec) : 2=4.69%, 4=35.81%, 10=39.71%, 20=4.17%, 50=0.39% cpu : usr=0.09%, sys=24.35%, ctx=1521, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31923: Wed Apr 9 02:17:06 2025 read: IOPS=115, BW=1846KiB/s (1891kB/s)(6032KiB/3267msec) clat (usec): min=66, max=20265, avg=4491.30, stdev=2957.51 lat (usec): min=66, max=20266, avg=4491.92, stdev=2957.49 clat percentiles (usec): | 1.00th=[ 71], 5.00th=[ 80], 10.00th=[ 95], 20.00th=[ 2573], | 30.00th=[ 3064], 40.00th=[ 3621], 50.00th=[ 4146], 60.00th=[ 4752], | 70.00th=[ 5342], 80.00th=[ 5997], 90.00th=[ 7832], 95.00th=[ 9241], | 99.00th=[16712], 99.50th=[19530], 99.90th=[20317], 99.95th=[20317], | 99.99th=[20317] bw ( KiB/s): min= 1184, max= 2112, per=25.84%, avg=1808.00, stdev=323.03, samples=6 iops : min= 74, max= 132, avg=113.00, stdev=20.19, samples=6 write: IOPS=119, BW=1915KiB/s (1961kB/s)(6256KiB/3267msec) clat (usec): min=646, max=15478, avg=3932.07, stdev=3244.42 lat (usec): min=647, max=15479, avg=3932.93, stdev=3244.43 clat percentiles (usec): | 1.00th=[ 693], 5.00th=[ 775], 10.00th=[ 816], 20.00th=[ 881], | 30.00th=[ 1237], 40.00th=[ 2343], 50.00th=[ 3097], 60.00th=[ 3851], | 70.00th=[ 4948], 80.00th=[ 6390], 90.00th=[ 8979], 95.00th=[10814], | 99.00th=[14091], 99.50th=[14746], 99.90th=[15533], 99.95th=[15533], | 99.99th=[15533] bw ( KiB/s): min= 1504, max= 2464, per=25.87%, avg=1904.00, stdev=340.92, samples=6 iops : min= 94, max= 154, avg=119.00, stdev=21.31, samples=6 lat (usec) : 100=5.21%, 250=0.26%, 750=1.69%, 1000=11.20% lat (msec) : 2=6.25%, 4=29.56%, 10=40.36%, 20=5.34%, 50=0.13% cpu : usr=0.34%, sys=23.64%, ctx=1525, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=6997KiB/s (7165kB/s), 1678KiB/s-1894KiB/s (1718kB/s-1939kB/s), io=23.4MiB (24.5MB), run=3231-3423msec WRITE: bw=7362KiB/s (7539kB/s), 1826KiB/s-1971KiB/s (1870kB/s-2018kB/s), io=24.6MiB (25.8MB), run=3231-3423msec rand-rw: (groupid=0, jobs=1): err= 0: pid=31915: Wed Apr 9 02:17:07 2025 read: IOPS=81, BW=1300KiB/s (1331kB/s)(5920KiB/4554msec) clat (usec): min=1948, max=18707, avg=4861.33, stdev=3453.52 lat (usec): min=1948, max=18708, avg=4862.00, stdev=3453.54 clat percentiles (usec): | 1.00th=[ 2073], 5.00th=[ 2278], 10.00th=[ 2376], 20.00th=[ 2606], | 30.00th=[ 2835], 40.00th=[ 3097], 50.00th=[ 3425], 60.00th=[ 3785], | 70.00th=[ 4621], 80.00th=[ 6521], 90.00th=[ 9896], 95.00th=[13435], | 99.00th=[16712], 99.50th=[18220], 99.90th=[18744], 99.95th=[18744], | 99.99th=[18744] bw ( KiB/s): min= 864, max= 1984, per=25.12%, avg=1297.11, stdev=403.61, samples=9 iops : min= 54, max= 124, avg=80.89, stdev=25.33, samples=9 write: IOPS=87, BW=1398KiB/s (1432kB/s)(6368KiB/4554msec) clat (usec): min=2597, max=25766, avg=6858.19, stdev=3496.62 lat (usec): min=2598, max=25768, avg=6859.38, stdev=3496.63 clat percentiles (usec): | 1.00th=[ 3130], 5.00th=[ 3523], 10.00th=[ 3752], 20.00th=[ 4228], | 30.00th=[ 4686], 40.00th=[ 5080], 50.00th=[ 5735], 60.00th=[ 6325], | 70.00th=[ 7635], 80.00th=[ 9241], 90.00th=[11600], 95.00th=[13698], | 99.00th=[19530], 99.50th=[25035], 99.90th=[25822], 99.95th=[25822], | 99.99th=[25822] bw ( KiB/s): min= 960, max= 2016, per=25.77%, avg=1400.11, stdev=371.50, samples=9 iops : min= 60, max= 126, avg=87.33, stdev=23.18, samples=9 lat (msec) : 2=0.26%, 4=37.11%, 10=48.96%, 20=13.28%, 50=0.39% cpu : usr=0.13%, sys=11.77%, ctx=938, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31916: Wed Apr 9 02:17:07 2025 read: IOPS=77, BW=1238KiB/s (1268kB/s)(5744KiB/4638msec) clat (usec): min=1927, max=20099, avg=5011.67, stdev=3559.89 lat (usec): min=1928, max=20100, avg=5012.32, stdev=3559.88 clat percentiles (usec): | 1.00th=[ 2008], 5.00th=[ 2180], 10.00th=[ 2311], 20.00th=[ 2573], | 30.00th=[ 2835], 40.00th=[ 3097], 50.00th=[ 3523], 60.00th=[ 4015], | 70.00th=[ 5014], 80.00th=[ 7439], 90.00th=[10159], 95.00th=[13042], | 99.00th=[18220], 99.50th=[19006], 99.90th=[20055], 99.95th=[20055], | 99.99th=[20055] bw ( KiB/s): min= 832, max= 1952, per=24.08%, avg=1243.67, stdev=360.16, samples=9 iops : min= 52, max= 122, avg=77.56, stdev=22.39, samples=9 write: IOPS=88, BW=1411KiB/s (1445kB/s)(6544KiB/4638msec) clat (usec): min=3030, max=25773, avg=6900.29, stdev=3521.68 lat (usec): min=3031, max=25775, avg=6901.72, stdev=3521.55 clat percentiles (usec): | 1.00th=[ 3163], 5.00th=[ 3556], 10.00th=[ 3851], 20.00th=[ 4293], | 30.00th=[ 4686], 40.00th=[ 5080], 50.00th=[ 5604], 60.00th=[ 6325], | 70.00th=[ 7635], 80.00th=[ 9241], 90.00th=[11994], 95.00th=[13960], | 99.00th=[19268], 99.50th=[19530], 99.90th=[25822], 99.95th=[25822], | 99.99th=[25822] bw ( KiB/s): min= 1021, max= 2107, per=25.90%, avg=1407.11, stdev=396.37, samples=9 iops : min= 63, max= 131, avg=87.78, stdev=24.72, samples=9 lat (msec) : 2=0.52%, 4=34.24%, 10=51.30%, 20=13.54%, 50=0.39% cpu : usr=0.24%, sys=11.49%, ctx=903, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31917: Wed Apr 9 02:17:07 2025 read: IOPS=88, BW=1419KiB/s (1453kB/s)(6256KiB/4410msec) clat (usec): min=1824, max=24276, avg=4669.64, stdev=3259.94 lat (usec): min=1825, max=24276, avg=4670.55, stdev=3259.85 clat percentiles (usec): | 1.00th=[ 1860], 5.00th=[ 2147], 10.00th=[ 2311], 20.00th=[ 2573], | 30.00th=[ 2737], 40.00th=[ 2966], 50.00th=[ 3294], 60.00th=[ 3818], | 70.00th=[ 4752], 80.00th=[ 6521], 90.00th=[ 9372], 95.00th=[11469], | 99.00th=[17957], 99.50th=[19530], 99.90th=[24249], 99.95th=[24249], | 99.99th=[24249] bw ( KiB/s): min= 768, max= 1952, per=25.70%, avg=1327.25, stdev=369.73, samples=8 iops : min= 48, max= 122, avg=82.75, stdev=23.24, samples=8 write: IOPS=85, BW=1368KiB/s (1401kB/s)(6032KiB/4410msec) clat (usec): min=2666, max=21784, avg=6749.67, stdev=3536.24 lat (usec): min=2667, max=21785, avg=6750.98, stdev=3536.18 clat percentiles (usec): | 1.00th=[ 2769], 5.00th=[ 3425], 10.00th=[ 3654], 20.00th=[ 4146], | 30.00th=[ 4490], 40.00th=[ 4817], 50.00th=[ 5211], 60.00th=[ 6259], | 70.00th=[ 7439], 80.00th=[ 9241], 90.00th=[11994], 95.00th=[14484], | 99.00th=[18220], 99.50th=[21365], 99.90th=[21890], 99.95th=[21890], | 99.99th=[21890] bw ( KiB/s): min= 928, max= 2048, per=24.21%, avg=1315.25, stdev=336.26, samples=8 iops : min= 58, max= 128, avg=82.00, stdev=21.02, samples=8 lat (msec) : 2=1.04%, 4=39.58%, 10=47.79%, 20=11.20%, 50=0.39% cpu : usr=0.25%, sys=11.70%, ctx=890, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31918: Wed Apr 9 02:17:07 2025 read: IOPS=84, BW=1344KiB/s (1377kB/s)(6032KiB/4487msec) clat (usec): min=1940, max=20139, avg=4662.78, stdev=3137.91 lat (usec): min=1940, max=20140, avg=4663.49, stdev=3137.89 clat percentiles (usec): | 1.00th=[ 2057], 5.00th=[ 2212], 10.00th=[ 2343], 20.00th=[ 2606], | 30.00th=[ 2868], 40.00th=[ 3064], 50.00th=[ 3425], 60.00th=[ 3785], | 70.00th=[ 4621], 80.00th=[ 6063], 90.00th=[ 9503], 95.00th=[11863], | 99.00th=[17957], 99.50th=[20055], 99.90th=[20055], 99.95th=[20055], | 99.99th=[20055] bw ( KiB/s): min= 928, max= 1920, per=24.77%, avg=1279.25, stdev=284.31, samples=8 iops : min= 58, max= 120, avg=79.75, stdev=17.81, samples=8 write: IOPS=87, BW=1394KiB/s (1428kB/s)(6256KiB/4487msec) clat (usec): min=2685, max=25582, avg=6895.75, stdev=3568.07 lat (usec): min=2686, max=25583, avg=6897.14, stdev=3568.00 clat percentiles (usec): | 1.00th=[ 2900], 5.00th=[ 3523], 10.00th=[ 3752], 20.00th=[ 4178], | 30.00th=[ 4555], 40.00th=[ 5014], 50.00th=[ 5604], 60.00th=[ 6390], | 70.00th=[ 7570], 80.00th=[ 9503], 90.00th=[12125], 95.00th=[14353], | 99.00th=[19006], 99.50th=[22152], 99.90th=[25560], 99.95th=[25560], | 99.99th=[25560] bw ( KiB/s): min= 1085, max= 2080, per=24.43%, avg=1327.25, stdev=325.25, samples=8 iops : min= 67, max= 130, avg=82.75, stdev=20.40, samples=8 lat (msec) : 2=0.26%, 4=38.28%, 10=48.05%, 20=13.02%, 50=0.39% cpu : usr=0.31%, sys=11.95%, ctx=902, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=5164KiB/s (5288kB/s), 1238KiB/s-1419KiB/s (1268kB/s-1453kB/s), io=23.4MiB (24.5MB), run=4410-4638msec WRITE: bw=5433KiB/s (5564kB/s), 1368KiB/s-1411KiB/s (1401kB/s-1445kB/s), io=24.6MiB (25.8MB), run=4410-4638msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=31937: Wed Apr 9 02:17:08 2025 read: IOPS=16, BW=16.2MiB/s (16.0MB/s)(7168KiB/433msec) clat (usec): min=508, max=38863, avg=28747.36, stdev=12924.97 lat (usec): min=509, max=38864, avg=28748.18, stdev=12925.03 clat percentiles (usec): | 1.00th=[ 510], 5.00th=[ 510], 10.00th=[ 510], 20.00th=[27919], | 30.00th=[31065], 40.00th=[31065], 50.00th=[32637], 60.00th=[34866], | 70.00th=[34866], 80.00th=[35390], 90.00th=[39060], 95.00th=[39060], | 99.00th=[39060], 99.50th=[39060], 99.90th=[39060], 99.95th=[39060], | 99.99th=[39060] write: IOPS=11, BW=11.5MiB/s (12.1MB/s)(5120KiB/433msec) clat (usec): min=31401, max=71968, avg=44953.14, stdev=16303.42 lat (usec): min=31498, max=72087, avg=45013.01, stdev=16329.44 clat percentiles (usec): | 1.00th=[31327], 5.00th=[31327], 10.00th=[31327], 20.00th=[31327], | 30.00th=[34341], 40.00th=[34341], 50.00th=[39584], 60.00th=[39584], | 70.00th=[47449], 80.00th=[47449], 90.00th=[71828], 95.00th=[71828], | 99.00th=[71828], 99.50th=[71828], 99.90th=[71828], 99.95th=[71828], | 99.99th=[71828] lat (usec) : 750=8.33% lat (msec) : 50=83.33%, 100=8.33% cpu : usr=0.00%, sys=45.37%, ctx=1189, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31938: Wed Apr 9 02:17:08 2025 read: IOPS=3, BW=3638KiB/s (3725kB/s)(2048KiB/563msec) clat (usec): min=34139, max=44856, avg=39497.72, stdev=7577.84 lat (usec): min=34139, max=44858, avg=39499.05, stdev=7578.91 clat percentiles (usec): | 1.00th=[34341], 5.00th=[34341], 10.00th=[34341], 20.00th=[34341], | 30.00th=[34341], 40.00th=[34341], 50.00th=[34341], 60.00th=[44827], | 70.00th=[44827], 80.00th=[44827], 90.00th=[44827], 95.00th=[44827], | 99.00th=[44827], 99.50th=[44827], 99.90th=[44827], 99.95th=[44827], | 99.99th=[44827] bw ( KiB/s): min= 4096, max= 4096, per=12.82%, avg=4096.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 write: IOPS=17, BW=17.8MiB/s (18.6MB/s)(10.0MiB/563msec) clat (usec): min=23715, max=97827, avg=40981.20, stdev=23010.03 lat (usec): min=23738, max=97875, avg=41024.63, stdev=23011.27 clat percentiles (usec): | 1.00th=[23725], 5.00th=[23725], 10.00th=[23725], 20.00th=[24249], | 30.00th=[24511], 40.00th=[28443], 50.00th=[30540], 60.00th=[35914], | 70.00th=[47973], 80.00th=[47973], 90.00th=[59507], 95.00th=[98042], | 99.00th=[98042], 99.50th=[98042], 99.90th=[98042], 99.95th=[98042], | 99.99th=[98042] bw ( KiB/s): min=16384, max=16384, per=36.63%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 16, max= 16, avg=16.00, stdev= 0.00, samples=1 lat (msec) : 50=83.33%, 100=16.67% cpu : usr=0.18%, sys=41.28%, ctx=374, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31939: Wed Apr 9 02:17:08 2025 read: IOPS=10, BW=10.7MiB/s (11.2MB/s)(6144KiB/560msec) clat (usec): min=335, max=40822, avg=17819.14, stdev=19844.56 lat (usec): min=336, max=40823, avg=17820.03, stdev=19844.45 clat percentiles (usec): | 1.00th=[ 334], 5.00th=[ 334], 10.00th=[ 334], 20.00th=[ 355], | 30.00th=[ 355], 40.00th=[ 603], 50.00th=[ 603], 60.00th=[25035], | 70.00th=[39584], 80.00th=[39584], 90.00th=[40633], 95.00th=[40633], | 99.00th=[40633], 99.50th=[40633], 99.90th=[40633], 99.95th=[40633], | 99.99th=[40633] bw ( KiB/s): min= 8192, max= 8192, per=25.64%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=10, BW=10.7MiB/s (11.2MB/s)(6144KiB/560msec) clat (msec): min=30, max=137, avg=62.43, stdev=39.47 lat (msec): min=30, max=137, avg=62.48, stdev=39.47 clat percentiles (msec): | 1.00th=[ 31], 5.00th=[ 31], 10.00th=[ 31], 20.00th=[ 33], | 30.00th=[ 33], 40.00th=[ 47], 50.00th=[ 47], 60.00th=[ 64], | 70.00th=[ 65], 80.00th=[ 65], 90.00th=[ 138], 95.00th=[ 138], | 99.00th=[ 138], 99.50th=[ 138], 99.90th=[ 138], 99.95th=[ 138], | 99.99th=[ 138] bw ( KiB/s): min= 8192, max= 8192, per=18.31%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 lat (usec) : 500=16.67%, 750=8.33% lat (msec) : 50=50.00%, 100=16.67%, 250=8.33% cpu : usr=0.00%, sys=32.20%, ctx=80, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31940: Wed Apr 9 02:17:08 2025 read: IOPS=7, BW=7988KiB/s (8179kB/s)(5120KiB/641msec) clat (usec): min=452, max=38266, avg=19883.98, stdev=18197.76 lat (usec): min=453, max=38267, avg=19884.79, stdev=18197.54 clat percentiles (usec): | 1.00th=[ 453], 5.00th=[ 453], 10.00th=[ 453], 20.00th=[ 453], | 30.00th=[ 1037], 40.00th=[ 1037], 50.00th=[24511], 60.00th=[24511], | 70.00th=[34866], 80.00th=[34866], 90.00th=[38011], 95.00th=[38011], | 99.00th=[38011], 99.50th=[38011], 99.90th=[38011], 99.95th=[38011], | 99.99th=[38011] bw ( KiB/s): min= 8192, max= 8192, per=25.64%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=10, BW=10.9MiB/s (11.5MB/s)(7168KiB/641msec) clat (msec): min=28, max=161, avg=70.88, stdev=53.88 lat (msec): min=28, max=161, avg=70.95, stdev=53.89 clat percentiles (msec): | 1.00th=[ 29], 5.00th=[ 29], 10.00th=[ 29], 20.00th=[ 29], | 30.00th=[ 40], 40.00th=[ 40], 50.00th=[ 47], 60.00th=[ 58], | 70.00th=[ 58], 80.00th=[ 134], 90.00th=[ 161], 95.00th=[ 161], | 99.00th=[ 161], 99.50th=[ 161], 99.90th=[ 161], 99.95th=[ 161], | 99.99th=[ 161] bw ( KiB/s): min= 8192, max= 8192, per=18.31%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 lat (usec) : 500=8.33% lat (msec) : 2=8.33%, 50=58.33%, 100=8.33%, 250=16.67% cpu : usr=0.16%, sys=35.62%, ctx=398, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=31.2MiB/s (32.7MB/s), 3638KiB/s-16.2MiB/s (3725kB/s-16.0MB/s), io=20.0MiB (20.0MB), run=433-641msec WRITE: bw=43.7MiB/s (45.8MB/s), 10.7MiB/s-17.8MiB/s (11.2MB/s-18.6MB/s), io=28.0MiB (29.4MB), run=433-641msec rand-rw: (groupid=0, jobs=1): err= 0: pid=31933: Wed Apr 9 02:17:08 2025 read: IOPS=8, BW=9108KiB/s (9327kB/s)(7168KiB/787msec) clat (msec): min=13, max=130, avg=72.94, stdev=48.44 lat (msec): min=13, max=130, avg=72.94, stdev=48.44 clat percentiles (msec): | 1.00th=[ 14], 5.00th=[ 14], 10.00th=[ 14], 20.00th=[ 28], | 30.00th=[ 41], 40.00th=[ 41], 50.00th=[ 64], 60.00th=[ 117], | 70.00th=[ 117], 80.00th=[ 120], 90.00th=[ 131], 95.00th=[ 131], | 99.00th=[ 131], 99.50th=[ 131], 99.90th=[ 131], 99.95th=[ 131], | 99.99th=[ 131] bw ( KiB/s): min= 6144, max= 6144, per=24.09%, avg=6144.00, stdev= 0.00, samples=1 iops : min= 6, max= 6, avg= 6.00, stdev= 0.00, samples=1 write: IOPS=6, BW=6506KiB/s (6662kB/s)(5120KiB/787msec) clat (usec): min=15843, max=65127, avg=42702.86, stdev=22873.23 lat (usec): min=15879, max=65195, avg=42739.22, stdev=22877.42 clat percentiles (usec): | 1.00th=[15795], 5.00th=[15795], 10.00th=[15795], 20.00th=[15795], | 30.00th=[20317], 40.00th=[20317], 50.00th=[54264], 60.00th=[54264], | 70.00th=[57934], 80.00th=[57934], 90.00th=[65274], 95.00th=[65274], | 99.00th=[65274], 99.50th=[65274], 99.90th=[65274], 99.95th=[65274], | 99.99th=[65274] bw ( KiB/s): min= 2048, max= 2048, per=5.74%, avg=2048.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 20=16.67%, 50=25.00%, 100=33.33%, 250=25.00% cpu : usr=0.00%, sys=14.89%, ctx=60, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31934: Wed Apr 9 02:17:08 2025 read: IOPS=2, BW=2550KiB/s (2612kB/s)(2048KiB/803msec) clat (msec): min=20, max=111, avg=65.96, stdev=64.74 lat (msec): min=20, max=111, avg=65.96, stdev=64.74 clat percentiles (msec): | 1.00th=[ 21], 5.00th=[ 21], 10.00th=[ 21], 20.00th=[ 21], | 30.00th=[ 21], 40.00th=[ 21], 50.00th=[ 21], 60.00th=[ 112], | 70.00th=[ 112], 80.00th=[ 112], 90.00th=[ 112], 95.00th=[ 112], | 99.00th=[ 112], 99.50th=[ 112], 99.90th=[ 112], 99.95th=[ 112], | 99.99th=[ 112] bw ( KiB/s): min= 1744, max= 1744, per=6.84%, avg=1744.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=12, BW=12.5MiB/s (13.1MB/s)(10.0MiB/803msec) clat (msec): min=15, max=181, avg=60.14, stdev=46.36 lat (msec): min=15, max=181, avg=60.19, stdev=46.37 clat percentiles (msec): | 1.00th=[ 16], 5.00th=[ 16], 10.00th=[ 16], 20.00th=[ 26], | 30.00th=[ 36], 40.00th=[ 50], 50.00th=[ 51], 60.00th=[ 51], | 70.00th=[ 62], 80.00th=[ 62], 90.00th=[ 81], 95.00th=[ 182], | 99.00th=[ 182], 99.50th=[ 182], 99.90th=[ 182], 99.95th=[ 182], | 99.99th=[ 182] bw ( KiB/s): min= 8722, max= 8722, per=24.43%, avg=8722.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 lat (msec) : 20=8.33%, 50=33.33%, 100=41.67%, 250=16.67% cpu : usr=0.00%, sys=14.84%, ctx=59, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31935: Wed Apr 9 02:17:08 2025 read: IOPS=12, BW=12.7MiB/s (13.3MB/s)(6144KiB/472msec) clat (usec): min=13746, max=57097, avg=30170.35, stdev=16604.21 lat (usec): min=13746, max=57098, avg=30171.19, stdev=16604.37 clat percentiles (usec): | 1.00th=[13698], 5.00th=[13698], 10.00th=[13698], 20.00th=[18220], | 30.00th=[18220], 40.00th=[21627], 50.00th=[21627], 60.00th=[27395], | 70.00th=[42730], 80.00th=[42730], 90.00th=[56886], 95.00th=[56886], | 99.00th=[56886], 99.50th=[56886], 99.90th=[56886], 99.95th=[56886], | 99.99th=[56886] write: IOPS=12, BW=12.7MiB/s (13.3MB/s)(6144KiB/472msec) clat (usec): min=19779, max=85766, avg=46703.48, stdev=25667.94 lat (usec): min=19872, max=85865, avg=46759.63, stdev=25662.44 clat percentiles (usec): | 1.00th=[19792], 5.00th=[19792], 10.00th=[19792], 20.00th=[21890], | 30.00th=[21890], 40.00th=[37487], 50.00th=[37487], 60.00th=[50070], | 70.00th=[65274], 80.00th=[65274], 90.00th=[85459], 95.00th=[85459], | 99.00th=[85459], 99.50th=[85459], 99.90th=[85459], 99.95th=[85459], | 99.99th=[85459] lat (msec) : 20=25.00%, 50=50.00%, 100=25.00% cpu : usr=0.21%, sys=16.74%, ctx=145, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31936: Wed Apr 9 02:17:08 2025 read: IOPS=7, BW=8127KiB/s (8322kB/s)(5120KiB/630msec) clat (usec): min=13210, max=58985, avg=30325.55, stdev=17059.10 lat (usec): min=13211, max=58986, avg=30326.36, stdev=17059.06 clat percentiles (usec): | 1.00th=[13173], 5.00th=[13173], 10.00th=[13173], 20.00th=[13173], | 30.00th=[25035], 40.00th=[25035], 50.00th=[26346], 60.00th=[26346], | 70.00th=[28181], 80.00th=[28181], 90.00th=[58983], 95.00th=[58983], | 99.00th=[58983], 99.50th=[58983], 99.90th=[58983], 99.95th=[58983], | 99.99th=[58983] bw ( KiB/s): min= 8192, max= 8192, per=32.12%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=11, BW=11.1MiB/s (11.7MB/s)(7168KiB/630msec) clat (msec): min=13, max=148, avg=62.01, stdev=48.05 lat (msec): min=13, max=148, avg=62.08, stdev=48.07 clat percentiles (msec): | 1.00th=[ 14], 5.00th=[ 14], 10.00th=[ 14], 20.00th=[ 24], | 30.00th=[ 39], 40.00th=[ 39], 50.00th=[ 47], 60.00th=[ 59], | 70.00th=[ 59], 80.00th=[ 105], 90.00th=[ 148], 95.00th=[ 148], | 99.00th=[ 148], 99.50th=[ 148], 99.90th=[ 148], 99.95th=[ 148], | 99.99th=[ 148] bw ( KiB/s): min= 8192, max= 8192, per=22.94%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 lat (msec) : 20=16.67%, 50=50.00%, 100=16.67%, 250=16.67% cpu : usr=0.16%, sys=14.79%, ctx=162, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=24.9MiB/s (26.1MB/s), 2550KiB/s-12.7MiB/s (2612kB/s-13.3MB/s), io=20.0MiB (20.0MB), run=472-803msec WRITE: bw=34.9MiB/s (36.6MB/s), 6506KiB/s-12.7MiB/s (6662kB/s-13.3MB/s), io=28.0MiB (29.4MB), run=472-803msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=31945: Wed Apr 9 02:17:10 2025 read: IOPS=4, BW=19.0MiB/s (19.0MB/s)(12.0MiB/630msec) clat (msec): min=86, max=435, avg=203.41, stdev=200.77 lat (msec): min=86, max=435, avg=203.41, stdev=200.77 clat percentiles (msec): | 1.00th=[ 87], 5.00th=[ 87], 10.00th=[ 87], 20.00th=[ 87], | 30.00th=[ 87], 40.00th=[ 89], 50.00th=[ 89], 60.00th=[ 89], | 70.00th=[ 435], 80.00th=[ 435], 90.00th=[ 435], 95.00th=[ 435], | 99.00th=[ 435], 99.50th=[ 435], 99.90th=[ 435], 99.95th=[ 435], | 99.99th=[ 435] bw ( KiB/s): min=13979, max=13979, per=53.75%, avg=13979.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 lat (msec) : 100=66.67%, 500=33.33% cpu : usr=0.00%, sys=11.61%, ctx=45, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31946: Wed Apr 9 02:17:10 2025 read: IOPS=7, BW=31.0MiB/s (32.5MB/s)(4096KiB/129msec) clat (nsec): min=42905k, max=42905k, avg=42904515.00, stdev= 0.00 lat (nsec): min=42906k, max=42906k, avg=42906182.00, stdev= 0.00 clat percentiles (usec): | 1.00th=[42730], 5.00th=[42730], 10.00th=[42730], 20.00th=[42730], | 30.00th=[42730], 40.00th=[42730], 50.00th=[42730], 60.00th=[42730], | 70.00th=[42730], 80.00th=[42730], 90.00th=[42730], 95.00th=[42730], | 99.00th=[42730], 99.50th=[42730], 99.90th=[42730], 99.95th=[42730], | 99.99th=[42730] write: IOPS=15, BW=62.0MiB/s (65.0MB/s)(8192KiB/129msec) clat (usec): min=38633, max=45734, avg=42183.91, stdev=5021.65 lat (usec): min=38889, max=46081, avg=42485.40, stdev=5085.44 clat percentiles (usec): | 1.00th=[38536], 5.00th=[38536], 10.00th=[38536], 20.00th=[38536], | 30.00th=[38536], 40.00th=[38536], 50.00th=[38536], 60.00th=[45876], | 70.00th=[45876], 80.00th=[45876], 90.00th=[45876], 95.00th=[45876], | 99.00th=[45876], 99.50th=[45876], 99.90th=[45876], 99.95th=[45876], | 99.99th=[45876] lat (msec) : 50=100.00% cpu : usr=0.00%, sys=43.75%, ctx=21, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31947: Wed Apr 9 02:17:10 2025 write: IOPS=4, BW=16.6MiB/s (17.4MB/s)(12.0MiB/725msec) clat (msec): min=88, max=473, avg=233.30, stdev=209.63 lat (msec): min=88, max=474, avg=233.61, stdev=209.72 clat percentiles (msec): | 1.00th=[ 89], 5.00th=[ 89], 10.00th=[ 89], 20.00th=[ 89], | 30.00th=[ 89], 40.00th=[ 138], 50.00th=[ 138], 60.00th=[ 138], | 70.00th=[ 472], 80.00th=[ 472], 90.00th=[ 472], 95.00th=[ 472], | 99.00th=[ 472], 99.50th=[ 472], 99.90th=[ 472], 99.95th=[ 472], | 99.99th=[ 472] bw ( KiB/s): min= 8192, max= 8192, per=18.13%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 250=33.33%, 500=33.33% cpu : usr=0.14%, sys=10.08%, ctx=47, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31948: Wed Apr 9 02:17:10 2025 write: IOPS=5, BW=21.4MiB/s (22.4MB/s)(12.0MiB/561msec) clat (msec): min=35, max=482, avg=184.80, stdev=258.03 lat (msec): min=35, max=483, avg=185.03, stdev=258.13 clat percentiles (msec): | 1.00th=[ 36], 5.00th=[ 36], 10.00th=[ 36], 20.00th=[ 36], | 30.00th=[ 36], 40.00th=[ 36], 50.00th=[ 36], 60.00th=[ 36], | 70.00th=[ 485], 80.00th=[ 485], 90.00th=[ 485], 95.00th=[ 485], | 99.00th=[ 485], 99.50th=[ 485], 99.90th=[ 485], 99.95th=[ 485], | 99.99th=[ 485] bw ( KiB/s): min=16384, max=16384, per=36.25%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 50=66.67%, 500=33.33% cpu : usr=0.36%, sys=9.64%, ctx=69, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=25.4MiB/s (26.6MB/s), 19.0MiB/s-31.0MiB/s (19.0MB/s-32.5MB/s), io=16.0MiB (16.8MB), run=129-630msec WRITE: bw=44.1MiB/s (46.3MB/s), 16.6MiB/s-62.0MiB/s (17.4MB/s-65.0MB/s), io=32.0MiB (33.6MB), run=129-725msec rand-rw: (groupid=0, jobs=1): err= 0: pid=31950: Wed Apr 9 02:17:10 2025 read: IOPS=4, BW=17.7MiB/s (18.6MB/s)(12.0MiB/677msec) clat (msec): min=96, max=471, avg=223.25, stdev=214.87 lat (msec): min=96, max=471, avg=223.26, stdev=214.87 clat percentiles (msec): | 1.00th=[ 96], 5.00th=[ 96], 10.00th=[ 96], 20.00th=[ 96], | 30.00th=[ 96], 40.00th=[ 103], 50.00th=[ 103], 60.00th=[ 103], | 70.00th=[ 472], 80.00th=[ 472], 90.00th=[ 472], 95.00th=[ 472], | 99.00th=[ 472], 99.50th=[ 472], 99.90th=[ 472], 99.95th=[ 472], | 99.99th=[ 472] bw ( KiB/s): min= 8175, max= 8175, per=37.17%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 250=33.33%, 500=33.33% cpu : usr=0.00%, sys=24.70%, ctx=1306, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31951: Wed Apr 9 02:17:10 2025 read: IOPS=1, BW=5498KiB/s (5630kB/s)(4096KiB/745msec) clat (nsec): min=444376k, max=444376k, avg=444376167.00, stdev= 0.00 lat (nsec): min=444378k, max=444378k, avg=444377865.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 443], 5.00th=[ 443], 10.00th=[ 443], 20.00th=[ 443], | 30.00th=[ 443], 40.00th=[ 443], 50.00th=[ 443], 60.00th=[ 443], | 70.00th=[ 443], 80.00th=[ 443], 90.00th=[ 443], 95.00th=[ 443], | 99.00th=[ 443], 99.50th=[ 443], 99.90th=[ 443], 99.95th=[ 443], | 99.99th=[ 443] bw ( KiB/s): min= 8192, max= 8192, per=37.25%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=2, BW=10.7MiB/s (11.3MB/s)(8192KiB/745msec) clat (msec): min=103, max=178, avg=140.69, stdev=53.02 lat (msec): min=103, max=178, avg=140.99, stdev=53.06 clat percentiles (msec): | 1.00th=[ 104], 5.00th=[ 104], 10.00th=[ 104], 20.00th=[ 104], | 30.00th=[ 104], 40.00th=[ 104], 50.00th=[ 104], 60.00th=[ 178], | 70.00th=[ 178], 80.00th=[ 178], 90.00th=[ 178], 95.00th=[ 178], | 99.00th=[ 178], 99.50th=[ 178], 99.90th=[ 178], 99.95th=[ 178], | 99.99th=[ 178] lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.00%, sys=25.94%, ctx=798, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31952: Wed Apr 9 02:17:10 2025 write: IOPS=3, BW=15.9MiB/s (16.7MB/s)(12.0MiB/755msec) clat (msec): min=85, max=539, avg=250.20, stdev=251.17 lat (msec): min=85, max=539, avg=250.38, stdev=251.09 clat percentiles (msec): | 1.00th=[ 86], 5.00th=[ 86], 10.00th=[ 86], 20.00th=[ 86], | 30.00th=[ 86], 40.00th=[ 127], 50.00th=[ 127], 60.00th=[ 127], | 70.00th=[ 542], 80.00th=[ 542], 90.00th=[ 542], 95.00th=[ 542], | 99.00th=[ 542], 99.50th=[ 542], 99.90th=[ 542], 99.95th=[ 542], | 99.99th=[ 542] lat (msec) : 100=33.33%, 250=33.33%, 750=33.33% cpu : usr=0.00%, sys=30.64%, ctx=52, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31953: Wed Apr 9 02:17:10 2025 write: IOPS=4, BW=17.1MiB/s (17.9MB/s)(12.0MiB/702msec) clat (msec): min=76, max=509, avg=233.39, stdev=239.80 lat (msec): min=76, max=509, avg=233.60, stdev=239.70 clat percentiles (msec): | 1.00th=[ 77], 5.00th=[ 77], 10.00th=[ 77], 20.00th=[ 77], | 30.00th=[ 77], 40.00th=[ 115], 50.00th=[ 115], 60.00th=[ 115], | 70.00th=[ 510], 80.00th=[ 510], 90.00th=[ 510], 95.00th=[ 510], | 99.00th=[ 510], 99.50th=[ 510], 99.90th=[ 510], 99.95th=[ 510], | 99.99th=[ 510] bw ( KiB/s): min= 8143, max= 8143, per=18.76%, avg=8143.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 250=33.33%, 750=33.33% cpu : usr=0.00%, sys=33.95%, ctx=60, majf=0, minf=28 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=21.5MiB/s (22.5MB/s), 5498KiB/s-17.7MiB/s (5630kB/s-18.6MB/s), io=16.0MiB (16.8MB), run=677-745msec WRITE: bw=42.4MiB/s (44.4MB/s), 10.7MiB/s-17.1MiB/s (11.3MB/s-17.9MB/s), io=32.0MiB (33.6MB), run=702-755msec