== sanity test 398b: DIO and buffer IO race ============== 01:03:30 (1743743010) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.13188 s, 44.5 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=7230: Fri Apr 4 01:03:43 2025 read: IOPS=135, BW=543KiB/s (556kB/s)(5992KiB/11029msec) clat (usec): min=52, max=167982, avg=3301.93, stdev=5184.58 lat (usec): min=53, max=167983, avg=3302.48, stdev=5184.58 clat percentiles (usec): | 1.00th=[ 63], 5.00th=[ 69], 10.00th=[ 78], 20.00th=[ 1647], | 30.00th=[ 2040], 40.00th=[ 2311], 50.00th=[ 2606], 60.00th=[ 3064], | 70.00th=[ 3720], 80.00th=[ 4948], 90.00th=[ 6456], 95.00th=[ 7635], | 99.00th=[ 12780], 99.50th=[ 16057], 99.90th=[ 46400], 99.95th=[168821], | 99.99th=[168821] bw ( KiB/s): min= 216, max= 792, per=25.22%, avg=541.68, stdev=121.17, samples=22 iops : min= 54, max= 198, avg=135.36, stdev=30.30, samples=22 write: IOPS=142, BW=571KiB/s (585kB/s)(6296KiB/11029msec) clat (usec): min=443, max=55397, avg=3771.66, stdev=3710.98 lat (usec): min=444, max=55397, avg=3772.99, stdev=3710.86 clat percentiles (usec): | 1.00th=[ 486], 5.00th=[ 510], 10.00th=[ 537], 20.00th=[ 586], | 30.00th=[ 799], 40.00th=[ 1942], 50.00th=[ 3064], 60.00th=[ 3884], | 70.00th=[ 4817], 80.00th=[ 6325], 90.00th=[ 8455], 95.00th=[10421], | 99.00th=[14746], 99.50th=[17695], 99.90th=[25297], 99.95th=[55313], | 99.99th=[55313] bw ( KiB/s): min= 248, max= 792, per=25.31%, avg=569.32, stdev=107.72, samples=22 iops : min= 62, max= 198, avg=142.27, stdev=26.96, samples=22 lat (usec) : 100=8.33%, 250=0.94%, 500=1.69%, 750=13.09%, 1000=2.77% lat (msec) : 2=7.65%, 4=32.58%, 10=29.07%, 20=3.52%, 50=0.29% lat (msec) : 100=0.03%, 250=0.03% cpu : usr=0.24%, sys=22.65%, ctx=5518, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=7232: Fri Apr 4 01:03:43 2025 read: IOPS=134, BW=536KiB/s (549kB/s)(5940KiB/11073msec) clat (usec): min=50, max=57310, avg=3208.07, stdev=3152.69 lat (usec): min=51, max=57310, avg=3209.48, stdev=3152.58 clat percentiles (usec): | 1.00th=[ 59], 5.00th=[ 68], 10.00th=[ 79], 20.00th=[ 1745], | 30.00th=[ 2073], 40.00th=[ 2311], 50.00th=[ 2671], 60.00th=[ 3130], | 70.00th=[ 3818], 80.00th=[ 4817], 90.00th=[ 6128], 95.00th=[ 7439], | 99.00th=[10945], 99.50th=[16450], 99.90th=[44827], 99.95th=[57410], | 99.99th=[57410] bw ( KiB/s): min= 224, max= 704, per=24.97%, avg=536.27, stdev=108.85, samples=22 iops : min= 56, max= 176, avg=134.05, stdev=27.21, samples=22 write: IOPS=143, BW=573KiB/s (587kB/s)(6348KiB/11073msec) clat (usec): min=415, max=166854, avg=3880.24, stdev=5522.95 lat (usec): min=416, max=166856, avg=3881.12, stdev=5522.84 clat percentiles (usec): | 1.00th=[ 482], 5.00th=[ 515], 10.00th=[ 537], 20.00th=[ 578], | 30.00th=[ 758], 40.00th=[ 2008], 50.00th=[ 3064], 60.00th=[ 3916], | 70.00th=[ 4817], 80.00th=[ 6194], 90.00th=[ 8586], 95.00th=[ 10552], | 99.00th=[ 15533], 99.50th=[ 25297], 99.90th=[ 32375], 99.95th=[166724], | 99.99th=[166724] bw ( KiB/s): min= 288, max= 832, per=25.38%, avg=570.82, stdev=121.25, samples=22 iops : min= 72, max= 208, avg=142.68, stdev=30.32, samples=22 lat (usec) : 100=7.55%, 250=0.98%, 500=1.56%, 750=14.00%, 1000=2.15% lat (msec) : 2=7.36%, 4=32.78%, 10=29.98%, 20=3.09%, 50=0.49% lat (msec) : 100=0.03%, 250=0.03% cpu : usr=0.23%, sys=22.62%, ctx=5459, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=7234: Fri Apr 4 01:03:43 2025 read: IOPS=134, BW=539KiB/s (552kB/s)(6028KiB/11177msec) clat (usec): min=51, max=164083, avg=3340.40, stdev=4922.89 lat (usec): min=52, max=164084, avg=3340.99, stdev=4922.91 clat percentiles (usec): | 1.00th=[ 59], 5.00th=[ 69], 10.00th=[ 79], 20.00th=[ 1696], | 30.00th=[ 2040], 40.00th=[ 2311], 50.00th=[ 2671], 60.00th=[ 3130], | 70.00th=[ 4047], 80.00th=[ 5080], 90.00th=[ 6456], 95.00th=[ 7963], | 99.00th=[ 11338], 99.50th=[ 15664], 99.90th=[ 26870], 99.95th=[164627], | 99.99th=[164627] bw ( KiB/s): min= 192, max= 776, per=24.90%, avg=534.86, stdev=136.22, samples=22 iops : min= 48, max= 194, avg=133.68, stdev=34.07, samples=22 write: IOPS=140, BW=560KiB/s (574kB/s)(6260KiB/11177msec) clat (usec): min=466, max=58256, avg=3830.90, stdev=4036.70 lat (usec): min=466, max=58257, avg=3832.22, stdev=4036.22 clat percentiles (usec): | 1.00th=[ 482], 5.00th=[ 510], 10.00th=[ 529], 20.00th=[ 594], | 30.00th=[ 848], 40.00th=[ 1893], 50.00th=[ 3195], 60.00th=[ 3818], | 70.00th=[ 4817], 80.00th=[ 6128], 90.00th=[ 8455], 95.00th=[10552], | 99.00th=[15926], 99.50th=[21890], 99.90th=[45876], 99.95th=[58459], | 99.99th=[58459] bw ( KiB/s): min= 256, max= 744, per=24.69%, avg=555.18, stdev=113.62, samples=22 iops : min= 64, max= 186, avg=138.77, stdev=28.40, samples=22 lat (usec) : 100=8.04%, 250=1.04%, 500=1.92%, 750=12.40%, 1000=2.90% lat (msec) : 2=8.30%, 4=31.12%, 10=30.24%, 20=3.65%, 50=0.33% lat (msec) : 100=0.03%, 250=0.03% cpu : usr=0.15%, sys=22.60%, ctx=5498, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=7236: Fri Apr 4 01:03:43 2025 read: IOPS=135, BW=543KiB/s (556kB/s)(6052KiB/11148msec) clat (usec): min=51, max=170325, avg=3367.54, stdev=5014.47 lat (usec): min=52, max=170326, avg=3368.16, stdev=5014.48 clat percentiles (usec): | 1.00th=[ 59], 5.00th=[ 69], 10.00th=[ 79], 20.00th=[ 1795], | 30.00th=[ 2114], 40.00th=[ 2376], 50.00th=[ 2704], 60.00th=[ 3261], | 70.00th=[ 3949], 80.00th=[ 5014], 90.00th=[ 6390], 95.00th=[ 7701], | 99.00th=[ 11469], 99.50th=[ 14877], 99.90th=[ 29754], 99.95th=[170918], | 99.99th=[170918] bw ( KiB/s): min= 232, max= 792, per=25.10%, avg=539.23, stdev=115.48, samples=22 iops : min= 58, max= 198, avg=134.77, stdev=28.89, samples=22 write: IOPS=139, BW=559KiB/s (573kB/s)(6236KiB/11148msec) clat (usec): min=432, max=34151, avg=3788.81, stdev=3782.49 lat (usec): min=433, max=34152, avg=3789.58, stdev=3782.44 clat percentiles (usec): | 1.00th=[ 482], 5.00th=[ 510], 10.00th=[ 529], 20.00th=[ 594], | 30.00th=[ 783], 40.00th=[ 1680], 50.00th=[ 3064], 60.00th=[ 3851], | 70.00th=[ 4817], 80.00th=[ 6259], 90.00th=[ 8717], 95.00th=[10683], | 99.00th=[16319], 99.50th=[20579], 99.90th=[33162], 99.95th=[34341], | 99.99th=[34341] bw ( KiB/s): min= 288, max= 752, per=24.70%, avg=555.55, stdev=116.30, samples=22 iops : min= 72, max= 188, avg=138.86, stdev=29.06, samples=22 lat (usec) : 100=7.65%, 250=0.59%, 500=1.46%, 750=13.22%, 1000=2.51% lat (msec) : 2=8.85%, 4=31.90%, 10=29.88%, 20=3.52%, 50=0.39% lat (msec) : 250=0.03% cpu : usr=0.19%, sys=22.68%, ctx=5690, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2148KiB/s (2200kB/s), 536KiB/s-543KiB/s (549kB/s-556kB/s), io=23.4MiB (24.6MB), run=11029-11177msec WRITE: bw=2249KiB/s (2303kB/s), 559KiB/s-573KiB/s (573kB/s-587kB/s), io=24.6MiB (25.7MB), run=11029-11177msec rand-rw: (groupid=0, jobs=1): err= 0: pid=7231: Fri Apr 4 01:04:07 2025 read: IOPS=43, BW=176KiB/s (180kB/s)(5992KiB/34076msec) clat (usec): min=1617, max=170857, avg=3432.57, stdev=5259.93 lat (usec): min=1617, max=170859, avg=3433.22, stdev=5259.94 clat percentiles (usec): | 1.00th=[ 1745], 5.00th=[ 1844], 10.00th=[ 1909], 20.00th=[ 1991], | 30.00th=[ 2057], 40.00th=[ 2147], 50.00th=[ 2245], 60.00th=[ 2442], | 70.00th=[ 2802], 80.00th=[ 3621], 90.00th=[ 6390], 95.00th=[ 8455], | 99.00th=[ 15008], 99.50th=[ 21365], 99.90th=[ 40109], 99.95th=[170918], | 99.99th=[170918] bw ( KiB/s): min= 48, max= 288, per=25.37%, avg=175.82, stdev=64.76, samples=68 iops : min= 12, max= 72, avg=43.91, stdev=16.20, samples=68 write: IOPS=46, BW=185KiB/s (189kB/s)(6296KiB/34076msec) clat (usec): min=8923, max=66421, avg=18288.84, stdev=6511.14 lat (usec): min=8923, max=66422, avg=18289.63, stdev=6511.12 clat percentiles (usec): | 1.00th=[10290], 5.00th=[10945], 10.00th=[11600], 20.00th=[12911], | 30.00th=[14222], 40.00th=[15795], 50.00th=[17433], 60.00th=[18482], | 70.00th=[19792], 80.00th=[21627], 90.00th=[26346], 95.00th=[31065], | 99.00th=[40633], 99.50th=[47973], 99.90th=[58983], 99.95th=[66323], | 99.99th=[66323] bw ( KiB/s): min= 32, max= 240, per=25.42%, avg=184.53, stdev=50.32, samples=68 iops : min= 8, max= 60, avg=46.09, stdev=12.57, samples=68 lat (msec) : 2=10.06%, 4=29.92%, 10=7.26%, 20=37.63%, 50=14.91% lat (msec) : 100=0.20%, 250=0.03% cpu : usr=0.10%, sys=5.41%, ctx=3494, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=7233: Fri Apr 4 01:04:07 2025 read: IOPS=42, BW=172KiB/s (176kB/s)(5940KiB/34611msec) clat (usec): min=1556, max=173990, avg=3485.81, stdev=5358.35 lat (usec): min=1557, max=173991, avg=3486.50, stdev=5358.37 clat percentiles (usec): | 1.00th=[ 1713], 5.00th=[ 1827], 10.00th=[ 1909], 20.00th=[ 1991], | 30.00th=[ 2057], 40.00th=[ 2114], 50.00th=[ 2245], 60.00th=[ 2442], | 70.00th=[ 2802], 80.00th=[ 3818], 90.00th=[ 6521], 95.00th=[ 9110], | 99.00th=[ 14746], 99.50th=[ 18744], 99.90th=[ 42730], 99.95th=[173016], | 99.99th=[173016] bw ( KiB/s): min= 8, max= 312, per=24.78%, avg=171.75, stdev=70.80, samples=69 iops : min= 2, max= 78, avg=42.91, stdev=17.69, samples=69 write: IOPS=45, BW=183KiB/s (188kB/s)(6348KiB/34611msec) clat (usec): min=8990, max=64965, avg=18452.49, stdev=6590.75 lat (usec): min=8990, max=64966, avg=18453.48, stdev=6590.67 clat percentiles (usec): | 1.00th=[10159], 5.00th=[11076], 10.00th=[11600], 20.00th=[12911], | 30.00th=[14222], 40.00th=[15926], 50.00th=[17695], 60.00th=[18744], | 70.00th=[20055], 80.00th=[21890], 90.00th=[27395], 95.00th=[31851], | 99.00th=[40633], 99.50th=[44303], 99.90th=[57934], 99.95th=[64750], | 99.99th=[64750] bw ( KiB/s): min= 40, max= 240, per=25.25%, avg=183.32, stdev=50.56, samples=69 iops : min= 10, max= 60, avg=45.80, stdev=12.62, samples=69 lat (msec) : 2=10.61%, 4=28.65%, 10=7.62%, 20=37.08%, 50=15.92% lat (msec) : 100=0.10%, 250=0.03% cpu : usr=0.11%, sys=5.35%, ctx=3534, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=7235: Fri Apr 4 01:04:07 2025 read: IOPS=43, BW=176KiB/s (180kB/s)(6028KiB/34268msec) clat (usec): min=1584, max=40266, avg=3418.65, stdev=2891.66 lat (usec): min=1585, max=40267, avg=3419.40, stdev=2891.82 clat percentiles (usec): | 1.00th=[ 1729], 5.00th=[ 1844], 10.00th=[ 1893], 20.00th=[ 1975], | 30.00th=[ 2073], 40.00th=[ 2147], 50.00th=[ 2278], 60.00th=[ 2507], | 70.00th=[ 2900], 80.00th=[ 4080], 90.00th=[ 6718], 95.00th=[ 8979], | 99.00th=[14353], 99.50th=[17171], 99.90th=[38011], 99.95th=[40109], | 99.99th=[40109] bw ( KiB/s): min= 24, max= 384, per=25.36%, avg=175.75, stdev=67.54, samples=68 iops : min= 6, max= 96, avg=43.84, stdev=16.87, samples=68 write: IOPS=45, BW=183KiB/s (187kB/s)(6260KiB/34268msec) clat (msec): min=9, max=214, avg=18.51, stdev= 8.47 lat (msec): min=9, max=214, avg=18.51, stdev= 8.47 clat percentiles (msec): | 1.00th=[ 11], 5.00th=[ 12], 10.00th=[ 12], 20.00th=[ 13], | 30.00th=[ 15], 40.00th=[ 17], 50.00th=[ 18], 60.00th=[ 19], | 70.00th=[ 20], 80.00th=[ 22], 90.00th=[ 28], 95.00th=[ 33], | 99.00th=[ 43], 99.50th=[ 47], 99.90th=[ 86], 99.95th=[ 215], | 99.99th=[ 215] bw ( KiB/s): min= 32, max= 240, per=25.10%, avg=182.22, stdev=51.62, samples=68 iops : min= 8, max= 60, avg=45.46, stdev=12.89, samples=68 lat (msec) : 2=11.20%, 4=27.83%, 10=8.37%, 20=37.47%, 50=14.91% lat (msec) : 100=0.20%, 250=0.03% cpu : usr=0.09%, sys=5.37%, ctx=3546, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=7237: Fri Apr 4 01:04:07 2025 read: IOPS=44, BW=177KiB/s (182kB/s)(6052KiB/34139msec) clat (usec): min=1557, max=43588, avg=3361.30, stdev=2883.63 lat (usec): min=1557, max=43589, avg=3361.99, stdev=2883.62 clat percentiles (usec): | 1.00th=[ 1713], 5.00th=[ 1844], 10.00th=[ 1909], 20.00th=[ 1991], | 30.00th=[ 2057], 40.00th=[ 2147], 50.00th=[ 2245], 60.00th=[ 2442], | 70.00th=[ 2835], 80.00th=[ 3949], 90.00th=[ 6325], 95.00th=[ 8717], | 99.00th=[14353], 99.50th=[16319], 99.90th=[35390], 99.95th=[43779], | 99.99th=[43779] bw ( KiB/s): min= 24, max= 320, per=25.59%, avg=177.35, stdev=68.45, samples=68 iops : min= 6, max= 80, avg=44.29, stdev=17.11, samples=68 write: IOPS=45, BW=183KiB/s (187kB/s)(6236KiB/34139msec) clat (msec): min=9, max=243, avg=18.54, stdev= 8.57 lat (msec): min=9, max=243, avg=18.54, stdev= 8.57 clat percentiles (msec): | 1.00th=[ 11], 5.00th=[ 12], 10.00th=[ 12], 20.00th=[ 13], | 30.00th=[ 15], 40.00th=[ 16], 50.00th=[ 18], 60.00th=[ 19], | 70.00th=[ 20], 80.00th=[ 22], 90.00th=[ 28], 95.00th=[ 33], | 99.00th=[ 40], 99.50th=[ 42], 99.90th=[ 61], 99.95th=[ 245], | 99.99th=[ 245] bw ( KiB/s): min= 32, max= 248, per=25.14%, avg=182.53, stdev=50.62, samples=68 iops : min= 8, max= 62, avg=45.59, stdev=12.65, samples=68 lat (msec) : 2=11.00%, 4=28.61%, 10=8.04%, 20=37.50%, 50=14.71% lat (msec) : 100=0.10%, 250=0.03% cpu : usr=0.09%, sys=5.42%, ctx=3521, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=694KiB/s (710kB/s), 172KiB/s-177KiB/s (176kB/s-182kB/s), io=23.4MiB (24.6MB), run=34076-34611msec WRITE: bw=726KiB/s (744kB/s), 183KiB/s-185KiB/s (187kB/s-189kB/s), io=24.6MiB (25.7MB), run=34076-34611msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=7275: Fri Apr 4 01:04:10 2025 read: IOPS=156, BW=2502KiB/s (2562kB/s)(5920KiB/2366msec) clat (usec): min=58, max=17946, avg=2860.17, stdev=2573.10 lat (usec): min=59, max=17946, avg=2860.76, stdev=2573.19 clat percentiles (usec): | 1.00th=[ 63], 5.00th=[ 71], 10.00th=[ 75], 20.00th=[ 87], | 30.00th=[ 133], 40.00th=[ 2409], 50.00th=[ 2704], 60.00th=[ 3261], | 70.00th=[ 3884], 80.00th=[ 4817], 90.00th=[ 5866], 95.00th=[ 7177], | 99.00th=[10683], 99.50th=[11994], 99.90th=[17957], 99.95th=[17957], | 99.99th=[17957] bw ( KiB/s): min= 1504, max= 2976, per=25.15%, avg=2512.00, stdev=695.95, samples=4 iops : min= 94, max= 186, avg=157.00, stdev=43.50, samples=4 write: IOPS=168, BW=2691KiB/s (2756kB/s)(6368KiB/2366msec) clat (usec): min=624, max=32979, avg=3189.02, stdev=3458.79 lat (usec): min=625, max=32980, avg=3189.94, stdev=3458.83 clat percentiles (usec): | 1.00th=[ 668], 5.00th=[ 758], 10.00th=[ 807], 20.00th=[ 889], | 30.00th=[ 1029], 40.00th=[ 1287], 50.00th=[ 1663], 60.00th=[ 2933], | 70.00th=[ 3589], 80.00th=[ 5145], 90.00th=[ 7046], 95.00th=[ 8979], | 99.00th=[17433], 99.50th=[21627], 99.90th=[32900], 99.95th=[32900], | 99.99th=[32900] bw ( KiB/s): min= 1760, max= 4096, per=25.96%, avg=2728.00, stdev=982.62, samples=4 iops : min= 110, max= 256, avg=170.50, stdev=61.41, samples=4 lat (usec) : 100=13.15%, 250=2.47%, 500=0.39%, 750=2.34%, 1000=12.50% lat (msec) : 2=13.41%, 4=27.73%, 10=25.78%, 20=1.95%, 50=0.26% cpu : usr=0.25%, sys=29.05%, ctx=1544, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=7277: Fri Apr 4 01:04:10 2025 read: IOPS=149, BW=2395KiB/s (2453kB/s)(5744KiB/2398msec) clat (usec): min=58, max=18728, avg=2772.59, stdev=2751.83 lat (usec): min=58, max=18729, avg=2776.51, stdev=2752.20 clat percentiles (usec): | 1.00th=[ 62], 5.00th=[ 70], 10.00th=[ 73], 20.00th=[ 82], | 30.00th=[ 94], 40.00th=[ 2245], 50.00th=[ 2671], 60.00th=[ 3163], | 70.00th=[ 3949], 80.00th=[ 4752], 90.00th=[ 5735], 95.00th=[ 7046], | 99.00th=[15664], 99.50th=[17433], 99.90th=[18744], 99.95th=[18744], | 99.99th=[18744] bw ( KiB/s): min= 1405, max= 3104, per=23.94%, avg=2391.25, stdev=713.28, samples=4 iops : min= 87, max= 194, avg=149.25, stdev=44.95, samples=4 write: IOPS=170, BW=2729KiB/s (2794kB/s)(6544KiB/2398msec) clat (usec): min=668, max=31424, avg=3308.89, stdev=3592.16 lat (usec): min=669, max=31425, avg=3309.84, stdev=3592.26 clat percentiles (usec): | 1.00th=[ 701], 5.00th=[ 750], 10.00th=[ 783], 20.00th=[ 857], | 30.00th=[ 971], 40.00th=[ 1123], 50.00th=[ 1778], 60.00th=[ 3064], | 70.00th=[ 4047], 80.00th=[ 5407], 90.00th=[ 7308], 95.00th=[ 9241], | 99.00th=[16450], 99.50th=[23462], 99.90th=[31327], 99.95th=[31327], | 99.99th=[31327] bw ( KiB/s): min= 1500, max= 3296, per=25.65%, avg=2695.00, stdev=839.37, samples=4 iops : min= 93, max= 206, avg=168.25, stdev=52.82, samples=4 lat (usec) : 100=14.58%, 250=1.69%, 500=0.26%, 750=3.26%, 1000=14.19% lat (msec) : 2=10.29%, 4=26.04%, 10=27.08%, 20=2.21%, 50=0.39% cpu : usr=0.04%, sys=28.74%, ctx=1438, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=7279: Fri Apr 4 01:04:10 2025 read: IOPS=165, BW=2650KiB/s (2713kB/s)(6256KiB/2361msec) clat (usec): min=55, max=17951, avg=2617.26, stdev=2558.45 lat (usec): min=55, max=17952, avg=2620.36, stdev=2557.50 clat percentiles (usec): | 1.00th=[ 59], 5.00th=[ 64], 10.00th=[ 72], 20.00th=[ 78], | 30.00th=[ 94], 40.00th=[ 2089], 50.00th=[ 2540], 60.00th=[ 3064], | 70.00th=[ 3589], 80.00th=[ 4555], 90.00th=[ 5800], 95.00th=[ 6849], | 99.00th=[ 9110], 99.50th=[17957], 99.90th=[17957], 99.95th=[17957], | 99.99th=[17957] bw ( KiB/s): min= 1213, max= 3232, per=25.70%, avg=2567.25, stdev=915.72, samples=4 iops : min= 75, max= 202, avg=160.25, stdev=57.63, samples=4 write: IOPS=159, BW=2555KiB/s (2616kB/s)(6032KiB/2361msec) clat (usec): min=642, max=40485, avg=3257.31, stdev=3892.87 lat (usec): min=643, max=40486, avg=3258.18, stdev=3892.87 clat percentiles (usec): | 1.00th=[ 668], 5.00th=[ 725], 10.00th=[ 783], 20.00th=[ 865], | 30.00th=[ 963], 40.00th=[ 1106], 50.00th=[ 1582], 60.00th=[ 2868], | 70.00th=[ 3687], 80.00th=[ 5211], 90.00th=[ 7242], 95.00th=[ 9110], | 99.00th=[19792], 99.50th=[27132], 99.90th=[40633], 99.95th=[40633], | 99.99th=[40633] bw ( KiB/s): min= 1437, max= 3104, per=24.28%, avg=2551.25, stdev=767.39, samples=4 iops : min= 89, max= 194, avg=159.25, stdev=48.36, samples=4 lat (usec) : 100=16.28%, 250=2.34%, 500=0.39%, 750=4.04%, 1000=12.63% lat (msec) : 2=9.24%, 4=28.91%, 10=24.35%, 20=1.43%, 50=0.39% cpu : usr=0.21%, sys=27.63%, ctx=1484, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=7281: Fri Apr 4 01:04:10 2025 read: IOPS=162, BW=2598KiB/s (2660kB/s)(6032KiB/2322msec) clat (usec): min=57, max=27838, avg=2957.01, stdev=2932.49 lat (usec): min=57, max=27839, avg=2957.55, stdev=2932.52 clat percentiles (usec): | 1.00th=[ 63], 5.00th=[ 71], 10.00th=[ 74], 20.00th=[ 86], | 30.00th=[ 112], 40.00th=[ 2212], 50.00th=[ 2704], 60.00th=[ 3359], | 70.00th=[ 4293], 80.00th=[ 5211], 90.00th=[ 6259], 95.00th=[ 7242], | 99.00th=[10945], 99.50th=[17695], 99.90th=[27919], 99.95th=[27919], | 99.99th=[27919] bw ( KiB/s): min= 1728, max= 2976, per=26.19%, avg=2616.00, stdev=596.02, samples=4 iops : min= 108, max= 186, avg=163.50, stdev=37.25, samples=4 write: IOPS=168, BW=2694KiB/s (2759kB/s)(6256KiB/2322msec) clat (usec): min=617, max=17713, avg=3039.02, stdev=2820.23 lat (usec): min=618, max=17714, avg=3039.91, stdev=2820.29 clat percentiles (usec): | 1.00th=[ 668], 5.00th=[ 725], 10.00th=[ 791], 20.00th=[ 873], | 30.00th=[ 971], 40.00th=[ 1172], 50.00th=[ 1713], 60.00th=[ 2933], | 70.00th=[ 3949], 80.00th=[ 5211], 90.00th=[ 6915], 95.00th=[ 8094], | 99.00th=[13698], 99.50th=[16319], 99.90th=[17695], 99.95th=[17695], | 99.99th=[17695] bw ( KiB/s): min= 1856, max= 3552, per=25.89%, avg=2720.00, stdev=713.15, samples=4 iops : min= 116, max= 222, avg=170.00, stdev=44.57, samples=4 lat (usec) : 100=12.37%, 250=4.30%, 500=0.52%, 750=3.39%, 1000=12.63% lat (msec) : 2=11.46%, 4=24.09%, 10=29.30%, 20=1.82%, 50=0.13% cpu : usr=0.26%, sys=29.90%, ctx=1441, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=9988KiB/s (10.2MB/s), 2395KiB/s-2650KiB/s (2453kB/s-2713kB/s), io=23.4MiB (24.5MB), run=2322-2398msec WRITE: bw=10.3MiB/s (10.8MB/s), 2555KiB/s-2729KiB/s (2616kB/s-2794kB/s), io=24.6MiB (25.8MB), run=2322-2398msec rand-rw: (groupid=0, jobs=1): err= 0: pid=7274: Fri Apr 4 01:04:17 2025 read: IOPS=42, BW=684KiB/s (701kB/s)(5920KiB/8653msec) clat (usec): min=1853, max=27011, avg=3595.27, stdev=2858.78 lat (usec): min=1853, max=27011, avg=3595.92, stdev=2858.81 clat percentiles (usec): | 1.00th=[ 1958], 5.00th=[ 2057], 10.00th=[ 2089], 20.00th=[ 2180], | 30.00th=[ 2245], 40.00th=[ 2343], 50.00th=[ 2474], 60.00th=[ 2671], | 70.00th=[ 3097], 80.00th=[ 4228], 90.00th=[ 6325], 95.00th=[ 8848], | 99.00th=[17433], 99.50th=[18744], 99.90th=[27132], 99.95th=[27132], | 99.99th=[27132] bw ( KiB/s): min= 384, max= 1216, per=25.06%, avg=675.65, stdev=222.21, samples=17 iops : min= 24, max= 76, avg=42.18, stdev=13.88, samples=17 write: IOPS=45, BW=736KiB/s (754kB/s)(6368KiB/8653msec) clat (usec): min=10510, max=52076, avg=18223.51, stdev=5805.25 lat (usec): min=10511, max=52078, avg=18224.83, stdev=5805.41 clat percentiles (usec): | 1.00th=[10814], 5.00th=[11469], 10.00th=[12125], 20.00th=[13173], | 30.00th=[14484], 40.00th=[15926], 50.00th=[17695], 60.00th=[18744], | 70.00th=[20055], 80.00th=[21365], 90.00th=[25035], 95.00th=[28967], | 99.00th=[38536], 99.50th=[44303], 99.90th=[52167], 99.95th=[52167], | 99.99th=[52167] bw ( KiB/s): min= 384, max= 896, per=25.87%, avg=734.00, stdev=170.16, samples=17 iops : min= 24, max= 56, avg=45.82, stdev=10.60, samples=17 lat (msec) : 2=1.04%, 4=36.20%, 10=8.98%, 20=38.28%, 50=15.36% lat (msec) : 100=0.13% cpu : usr=0.08%, sys=6.07%, ctx=893, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=7276: Fri Apr 4 01:04:17 2025 read: IOPS=40, BW=647KiB/s (662kB/s)(5744KiB/8882msec) clat (usec): min=1767, max=31916, avg=3682.93, stdev=3024.25 lat (usec): min=1768, max=31916, avg=3683.59, stdev=3024.24 clat percentiles (usec): | 1.00th=[ 1942], 5.00th=[ 2057], 10.00th=[ 2089], 20.00th=[ 2180], | 30.00th=[ 2245], 40.00th=[ 2343], 50.00th=[ 2474], 60.00th=[ 2671], | 70.00th=[ 3458], 80.00th=[ 4293], 90.00th=[ 6587], 95.00th=[10159], | 99.00th=[16319], 99.50th=[16581], 99.90th=[31851], 99.95th=[31851], | 99.99th=[31851] bw ( KiB/s): min= 192, max= 1184, per=24.08%, avg=649.18, stdev=241.39, samples=17 iops : min= 12, max= 74, avg=40.47, stdev=15.15, samples=17 write: IOPS=46, BW=737KiB/s (754kB/s)(6544KiB/8882msec) clat (usec): min=10515, max=47611, avg=18274.73, stdev=5650.82 lat (usec): min=10516, max=47612, avg=18275.83, stdev=5650.88 clat percentiles (usec): | 1.00th=[10945], 5.00th=[11600], 10.00th=[12125], 20.00th=[13173], | 30.00th=[14615], 40.00th=[16319], 50.00th=[17433], 60.00th=[19006], | 70.00th=[20055], 80.00th=[21365], 90.00th=[25560], 95.00th=[29230], | 99.00th=[34341], 99.50th=[42206], 99.90th=[47449], 99.95th=[47449], | 99.99th=[47449] bw ( KiB/s): min= 384, max= 928, per=25.80%, avg=732.06, stdev=171.17, samples=17 iops : min= 24, max= 58, avg=45.65, stdev=10.76, samples=17 lat (msec) : 2=0.91%, 4=34.37%, 10=9.11%, 20=39.19%, 50=16.41% cpu : usr=0.09%, sys=6.05%, ctx=885, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=7278: Fri Apr 4 01:04:17 2025 read: IOPS=46, BW=747KiB/s (764kB/s)(6256KiB/8380msec) clat (usec): min=1798, max=20236, avg=3295.40, stdev=2353.81 lat (usec): min=1798, max=20237, avg=3296.09, stdev=2353.88 clat percentiles (usec): | 1.00th=[ 1926], 5.00th=[ 2024], 10.00th=[ 2089], 20.00th=[ 2180], | 30.00th=[ 2245], 40.00th=[ 2311], 50.00th=[ 2442], 60.00th=[ 2606], | 70.00th=[ 2835], 80.00th=[ 3720], 90.00th=[ 5538], 95.00th=[ 7308], | 99.00th=[15795], 99.50th=[18482], 99.90th=[20317], 99.95th=[20317], | 99.99th=[20317] bw ( KiB/s): min= 352, max= 1344, per=28.18%, avg=759.81, stdev=296.29, samples=16 iops : min= 22, max= 84, avg=47.38, stdev=18.60, samples=16 write: IOPS=44, BW=720KiB/s (737kB/s)(6032KiB/8380msec) clat (usec): min=10269, max=47660, avg=18773.67, stdev=6461.18 lat (usec): min=10270, max=47662, avg=18774.81, stdev=6461.19 clat percentiles (usec): | 1.00th=[10945], 5.00th=[11731], 10.00th=[12125], 20.00th=[13173], | 30.00th=[14615], 40.00th=[16188], 50.00th=[17957], 60.00th=[19006], | 70.00th=[20579], 80.00th=[22414], 90.00th=[26608], 95.00th=[31851], | 99.00th=[43779], 99.50th=[47449], 99.90th=[47449], 99.95th=[47449], | 99.99th=[47449] bw ( KiB/s): min= 448, max= 896, per=25.30%, avg=717.75, stdev=145.13, samples=16 iops : min= 28, max= 56, avg=44.75, stdev= 9.14, samples=16 lat (msec) : 2=1.82%, 4=39.58%, 10=8.07%, 20=33.85%, 50=16.67% cpu : usr=0.05%, sys=6.16%, ctx=908, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=7280: Fri Apr 4 01:04:17 2025 read: IOPS=43, BW=690KiB/s (707kB/s)(6032KiB/8742msec) clat (usec): min=1885, max=32020, avg=3537.97, stdev=2924.85 lat (usec): min=1886, max=32021, avg=3538.60, stdev=2924.86 clat percentiles (usec): | 1.00th=[ 1909], 5.00th=[ 2024], 10.00th=[ 2089], 20.00th=[ 2180], | 30.00th=[ 2245], 40.00th=[ 2311], 50.00th=[ 2442], 60.00th=[ 2638], | 70.00th=[ 3130], 80.00th=[ 4047], 90.00th=[ 5800], 95.00th=[ 9503], | 99.00th=[16581], 99.50th=[19530], 99.90th=[32113], 99.95th=[32113], | 99.99th=[32113] bw ( KiB/s): min= 288, max= 992, per=25.20%, avg=679.41, stdev=195.09, samples=17 iops : min= 18, max= 62, avg=42.41, stdev=12.17, samples=17 write: IOPS=44, BW=716KiB/s (733kB/s)(6256KiB/8742msec) clat (usec): min=10365, max=48894, avg=18668.19, stdev=5667.87 lat (usec): min=10366, max=48896, avg=18669.28, stdev=5667.93 clat percentiles (usec): | 1.00th=[11076], 5.00th=[11731], 10.00th=[12387], 20.00th=[13960], | 30.00th=[15401], 40.00th=[16450], 50.00th=[17957], 60.00th=[19268], | 70.00th=[20317], 80.00th=[21627], 90.00th=[25297], 95.00th=[30278], | 99.00th=[39060], 99.50th=[43254], 99.90th=[49021], 99.95th=[49021], | 99.99th=[49021] bw ( KiB/s): min= 352, max= 896, per=25.14%, avg=713.29, stdev=165.35, samples=17 iops : min= 22, max= 56, avg=44.53, stdev=10.31, samples=17 lat (msec) : 2=2.08%, 4=36.85%, 10=8.20%, 20=35.81%, 50=17.06% cpu : usr=0.06%, sys=5.87%, ctx=941, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2697KiB/s (2761kB/s), 647KiB/s-747KiB/s (662kB/s-764kB/s), io=23.4MiB (24.5MB), run=8380-8882msec WRITE: bw=2837KiB/s (2905kB/s), 716KiB/s-737KiB/s (733kB/s-754kB/s), io=24.6MiB (25.8MB), run=8380-8882msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=7297: Fri Apr 4 01:04:19 2025 read: IOPS=7, BW=7443KiB/s (7622kB/s)(7168KiB/963msec) clat (usec): min=427, max=469581, avg=96694.80, stdev=165278.65 lat (usec): min=428, max=469582, avg=96695.88, stdev=165278.82 clat percentiles (usec): | 1.00th=[ 429], 5.00th=[ 429], 10.00th=[ 429], 20.00th=[ 28443], | 30.00th=[ 38011], 40.00th=[ 38011], 50.00th=[ 43779], 60.00th=[ 47449], | 70.00th=[ 47449], 80.00th=[ 49021], 90.00th=[467665], 95.00th=[467665], | 99.00th=[467665], 99.50th=[467665], 99.90th=[467665], 99.95th=[467665], | 99.99th=[467665] bw ( KiB/s): min= 2048, max= 2048, per=11.73%, avg=2048.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=5, BW=5317KiB/s (5444kB/s)(5120KiB/963msec) clat (usec): min=34738, max=67282, avg=52384.89, stdev=12530.83 lat (usec): min=34764, max=67301, avg=52423.52, stdev=12536.55 clat percentiles (usec): | 1.00th=[34866], 5.00th=[34866], 10.00th=[34866], 20.00th=[34866], | 30.00th=[45876], 40.00th=[45876], 50.00th=[55837], 60.00th=[55837], | 70.00th=[58459], 80.00th=[58459], 90.00th=[67634], 95.00th=[67634], | 99.00th=[67634], 99.50th=[67634], 99.90th=[67634], 99.95th=[67634], | 99.99th=[67634] lat (usec) : 500=8.33% lat (msec) : 50=58.33%, 100=25.00%, 500=8.33% cpu : usr=0.00%, sys=27.96%, ctx=806, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=7299: Fri Apr 4 01:04:19 2025 read: IOPS=1, BW=1746KiB/s (1788kB/s)(2048KiB/1173msec) clat (usec): min=26294, max=37821, avg=32058.31, stdev=8150.74 lat (usec): min=26295, max=37823, avg=32059.31, stdev=8151.20 clat percentiles (usec): | 1.00th=[26346], 5.00th=[26346], 10.00th=[26346], 20.00th=[26346], | 30.00th=[26346], 40.00th=[26346], 50.00th=[26346], 60.00th=[38011], | 70.00th=[38011], 80.00th=[38011], 90.00th=[38011], 95.00th=[38011], | 99.00th=[38011], 99.50th=[38011], 99.90th=[38011], 99.95th=[38011], | 99.99th=[38011] bw ( KiB/s): min= 1793, max= 2048, per=11.00%, avg=1920.50, stdev=180.31, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 write: IOPS=8, BW=8730KiB/s (8939kB/s)(10.0MiB/1173msec) clat (msec): min=24, max=567, avg=104.71, stdev=164.00 lat (msec): min=24, max=567, avg=104.75, stdev=163.99 clat percentiles (msec): | 1.00th=[ 25], 5.00th=[ 25], 10.00th=[ 25], 20.00th=[ 27], | 30.00th=[ 43], 40.00th=[ 45], 50.00th=[ 56], 60.00th=[ 56], | 70.00th=[ 77], 80.00th=[ 77], 90.00th=[ 92], 95.00th=[ 567], | 99.00th=[ 567], 99.50th=[ 567], 99.90th=[ 567], 99.95th=[ 567], | 99.99th=[ 567] bw ( KiB/s): min=14346, max=14346, per=58.69%, avg=14346.00, stdev= 0.00, samples=1 iops : min= 14, max= 14, avg=14.00, stdev= 0.00, samples=1 lat (msec) : 50=50.00%, 100=41.67%, 750=8.33% cpu : usr=0.00%, sys=18.69%, ctx=356, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=7300: Fri Apr 4 01:04:19 2025 read: IOPS=6, BW=6302KiB/s (6453kB/s)(6144KiB/975msec) clat (usec): min=340, max=37159, avg=15447.36, stdev=14698.78 lat (usec): min=340, max=37160, avg=15447.97, stdev=14698.75 clat percentiles (usec): | 1.00th=[ 343], 5.00th=[ 343], 10.00th=[ 343], 20.00th=[ 355], | 30.00th=[ 355], 40.00th=[11207], 50.00th=[11207], 60.00th=[16581], | 70.00th=[27132], 80.00th=[27132], 90.00th=[36963], 95.00th=[36963], | 99.00th=[36963], 99.50th=[36963], 99.90th=[36963], 99.95th=[36963], | 99.99th=[36963] write: IOPS=6, BW=6302KiB/s (6453kB/s)(6144KiB/975msec) clat (msec): min=15, max=595, avg=129.05, stdev=228.99 lat (msec): min=15, max=595, avg=129.08, stdev=228.98 clat percentiles (msec): | 1.00th=[ 16], 5.00th=[ 16], 10.00th=[ 16], 20.00th=[ 29], | 30.00th=[ 29], 40.00th=[ 32], 50.00th=[ 32], 60.00th=[ 48], | 70.00th=[ 57], 80.00th=[ 57], 90.00th=[ 592], 95.00th=[ 592], | 99.00th=[ 592], 99.50th=[ 592], 99.90th=[ 592], 99.95th=[ 592], | 99.99th=[ 592] bw ( KiB/s): min= 4096, max= 4096, per=16.76%, avg=4096.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (usec) : 500=16.67% lat (msec) : 20=25.00%, 50=41.67%, 100=8.33%, 750=8.33% cpu : usr=0.00%, sys=12.94%, ctx=489, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=7301: Fri Apr 4 01:04:19 2025 read: IOPS=5, BW=5214KiB/s (5339kB/s)(5120KiB/982msec) clat (usec): min=536, max=33281, avg=17613.00, stdev=15818.17 lat (usec): min=537, max=33282, avg=17613.74, stdev=15817.97 clat percentiles (usec): | 1.00th=[ 537], 5.00th=[ 537], 10.00th=[ 537], 20.00th=[ 537], | 30.00th=[ 578], 40.00th=[ 578], 50.00th=[25560], 60.00th=[25560], | 70.00th=[28181], 80.00th=[28181], 90.00th=[33162], 95.00th=[33162], | 99.00th=[33162], 99.50th=[33162], 99.90th=[33162], 99.95th=[33162], | 99.99th=[33162] write: IOPS=7, BW=7299KiB/s (7475kB/s)(7168KiB/982msec) clat (msec): min=32, max=588, avg=126.10, stdev=205.82 lat (msec): min=32, max=588, avg=126.15, stdev=205.82 clat percentiles (msec): | 1.00th=[ 33], 5.00th=[ 33], 10.00th=[ 33], 20.00th=[ 35], | 30.00th=[ 37], 40.00th=[ 37], 50.00th=[ 41], 60.00th=[ 43], | 70.00th=[ 43], 80.00th=[ 109], 90.00th=[ 592], 95.00th=[ 592], | 99.00th=[ 592], 99.50th=[ 592], 99.90th=[ 592], 99.95th=[ 592], | 99.99th=[ 592] bw ( KiB/s): min= 2048, max= 2048, per=8.38%, avg=2048.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (usec) : 750=16.67% lat (msec) : 50=66.67%, 250=8.33%, 750=8.33% cpu : usr=0.00%, sys=17.94%, ctx=402, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=17.0MiB/s (17.9MB/s), 1746KiB/s-7443KiB/s (1788kB/s-7622kB/s), io=20.0MiB (20.0MB), run=963-1173msec WRITE: bw=23.9MiB/s (25.0MB/s), 5317KiB/s-8730KiB/s (5444kB/s-8939kB/s), io=28.0MiB (29.4MB), run=963-1173msec rand-rw: (groupid=0, jobs=1): err= 0: pid=7294: Fri Apr 4 01:04:19 2025 read: IOPS=7, BW=7666KiB/s (7850kB/s)(7168KiB/935msec) clat (msec): min=15, max=481, avg=88.94, stdev=173.17 lat (msec): min=15, max=481, avg=88.94, stdev=173.17 clat percentiles (msec): | 1.00th=[ 16], 5.00th=[ 16], 10.00th=[ 16], 20.00th=[ 19], | 30.00th=[ 19], 40.00th=[ 19], 50.00th=[ 24], 60.00th=[ 24], | 70.00th=[ 24], 80.00th=[ 42], 90.00th=[ 481], 95.00th=[ 481], | 99.00th=[ 481], 99.50th=[ 481], 99.90th=[ 481], 99.95th=[ 481], | 99.99th=[ 481] bw ( KiB/s): min= 4096, max= 4096, per=24.14%, avg=4096.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 write: IOPS=5, BW=5476KiB/s (5607kB/s)(5120KiB/935msec) clat (usec): min=46034, max=96719, avg=59420.63, stdev=21054.40 lat (usec): min=46057, max=96759, avg=59467.45, stdev=21051.83 clat percentiles (usec): | 1.00th=[45876], 5.00th=[45876], 10.00th=[45876], 20.00th=[45876], | 30.00th=[48497], 40.00th=[48497], 50.00th=[52691], 60.00th=[52691], | 70.00th=[53216], 80.00th=[53216], 90.00th=[96994], 95.00th=[96994], | 99.00th=[96994], 99.50th=[96994], 99.90th=[96994], 99.95th=[96994], | 99.99th=[96994] lat (msec) : 20=25.00%, 50=41.67%, 100=25.00%, 500=8.33% cpu : usr=0.00%, sys=9.85%, ctx=35, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=7295: Fri Apr 4 01:04:19 2025 read: IOPS=1, BW=1761KiB/s (1803kB/s)(2048KiB/1163msec) clat (usec): min=51697, max=52242, avg=51970.11, stdev=385.29 lat (usec): min=51698, max=52243, avg=51971.11, stdev=385.69 clat percentiles (usec): | 1.00th=[51643], 5.00th=[51643], 10.00th=[51643], 20.00th=[51643], | 30.00th=[51643], 40.00th=[51643], 50.00th=[51643], 60.00th=[52167], | 70.00th=[52167], 80.00th=[52167], 90.00th=[52167], 95.00th=[52167], | 99.00th=[52167], 99.50th=[52167], 99.90th=[52167], 99.95th=[52167], | 99.99th=[52167] bw ( KiB/s): min= 2048, max= 2048, per=12.07%, avg=2048.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 write: IOPS=8, BW=8805KiB/s (9016kB/s)(10.0MiB/1163msec) clat (msec): min=29, max=550, avg=101.09, stdev=159.26 lat (msec): min=29, max=550, avg=101.16, stdev=159.24 clat percentiles (msec): | 1.00th=[ 30], 5.00th=[ 30], 10.00th=[ 30], 20.00th=[ 35], | 30.00th=[ 39], 40.00th=[ 43], 50.00th=[ 43], 60.00th=[ 54], | 70.00th=[ 69], 80.00th=[ 69], 90.00th=[ 97], 95.00th=[ 550], | 99.00th=[ 550], 99.50th=[ 550], 99.90th=[ 550], 99.95th=[ 550], | 99.99th=[ 550] bw ( KiB/s): min=12288, max=12288, per=51.73%, avg=12288.00, stdev= 0.00, samples=1 iops : min= 12, max= 12, avg=12.00, stdev= 0.00, samples=1 lat (msec) : 50=41.67%, 100=50.00%, 750=8.33% cpu : usr=0.09%, sys=10.67%, ctx=48, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=7296: Fri Apr 4 01:04:19 2025 read: IOPS=4, BW=5090KiB/s (5212kB/s)(6144KiB/1207msec) clat (msec): min=14, max=122, avg=42.61, stdev=41.88 lat (msec): min=14, max=122, avg=42.61, stdev=41.88 clat percentiles (msec): | 1.00th=[ 15], 5.00th=[ 15], 10.00th=[ 15], 20.00th=[ 17], | 30.00th=[ 17], 40.00th=[ 19], 50.00th=[ 19], 60.00th=[ 29], | 70.00th=[ 55], 80.00th=[ 55], 90.00th=[ 124], 95.00th=[ 124], | 99.00th=[ 124], 99.50th=[ 124], 99.90th=[ 124], 99.95th=[ 124], | 99.99th=[ 124] bw ( KiB/s): min= 4087, max= 4087, per=24.09%, avg=4087.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 write: IOPS=4, BW=5090KiB/s (5212kB/s)(6144KiB/1207msec) clat (msec): min=35, max=585, avg=150.21, stdev=215.38 lat (msec): min=35, max=586, avg=150.27, stdev=215.38 clat percentiles (msec): | 1.00th=[ 36], 5.00th=[ 36], 10.00th=[ 36], 20.00th=[ 41], | 30.00th=[ 41], 40.00th=[ 49], 50.00th=[ 49], 60.00th=[ 80], | 70.00th=[ 112], 80.00th=[ 112], 90.00th=[ 584], 95.00th=[ 584], | 99.00th=[ 584], 99.50th=[ 584], 99.90th=[ 584], 99.95th=[ 584], | 99.99th=[ 584] bw ( KiB/s): min= 2043, max= 4096, per=12.92%, avg=3069.50, stdev=1451.69, samples=2 iops : min= 1, max= 4, avg= 2.50, stdev= 2.12, samples=2 lat (msec) : 20=25.00%, 50=33.33%, 100=16.67%, 250=16.67%, 750=8.33% cpu : usr=0.00%, sys=6.72%, ctx=39, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=7298: Fri Apr 4 01:04:19 2025 read: IOPS=4, BW=4285KiB/s (4387kB/s)(5120KiB/1195msec) clat (usec): min=13325, max=64867, avg=41346.14, stdev=22614.67 lat (usec): min=13325, max=64868, avg=41347.01, stdev=22614.72 clat percentiles (usec): | 1.00th=[13304], 5.00th=[13304], 10.00th=[13304], 20.00th=[13304], | 30.00th=[21627], 40.00th=[21627], 50.00th=[49546], 60.00th=[49546], | 70.00th=[57410], 80.00th=[57410], 90.00th=[64750], 95.00th=[64750], | 99.00th=[64750], 99.50th=[64750], 99.90th=[64750], 99.95th=[64750], | 99.99th=[64750] bw ( KiB/s): min= 8192, max= 8192, per=48.28%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=5, BW=5998KiB/s (6142kB/s)(7168KiB/1195msec) clat (msec): min=36, max=560, avg=123.38, stdev=193.01 lat (msec): min=36, max=560, avg=123.45, stdev=193.01 clat percentiles (msec): | 1.00th=[ 37], 5.00th=[ 37], 10.00th=[ 37], 20.00th=[ 43], | 30.00th=[ 47], 40.00th=[ 47], 50.00th=[ 53], 60.00th=[ 58], | 70.00th=[ 58], 80.00th=[ 68], 90.00th=[ 558], 95.00th=[ 558], | 99.00th=[ 558], 99.50th=[ 558], 99.90th=[ 558], 99.95th=[ 558], | 99.99th=[ 558] bw ( KiB/s): min= 2048, max= 6144, per=17.24%, avg=4096.00, stdev=2896.31, samples=2 iops : min= 2, max= 6, avg= 4.00, stdev= 2.83, samples=2 lat (msec) : 20=8.33%, 50=41.67%, 100=41.67%, 750=8.33% cpu : usr=0.00%, sys=8.71%, ctx=55, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=16.6MiB/s (17.4MB/s), 1761KiB/s-7666KiB/s (1803kB/s-7850kB/s), io=20.0MiB (20.0MB), run=935-1207msec WRITE: bw=23.2MiB/s (24.3MB/s), 5090KiB/s-8805KiB/s (5212kB/s-9016kB/s), io=28.0MiB (29.4MB), run=935-1207msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=7308: Fri Apr 4 01:04:21 2025 read: IOPS=9, BW=37.0MiB/s (39.8MB/s)(12.0MiB/316msec) clat (msec): min=55, max=128, avg=100.06, stdev=39.02 lat (msec): min=55, max=128, avg=100.06, stdev=39.02 clat percentiles (msec): | 1.00th=[ 56], 5.00th=[ 56], 10.00th=[ 56], 20.00th=[ 56], | 30.00th=[ 56], 40.00th=[ 117], 50.00th=[ 117], 60.00th=[ 117], | 70.00th=[ 129], 80.00th=[ 129], 90.00th=[ 129], 95.00th=[ 129], | 99.00th=[ 129], 99.50th=[ 129], 99.90th=[ 129], 99.95th=[ 129], | 99.99th=[ 129] lat (msec) : 100=33.33%, 250=66.67% cpu : usr=0.00%, sys=23.81%, ctx=644, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=7310: Fri Apr 4 01:04:21 2025 read: IOPS=1, BW=5729KiB/s (5866kB/s)(4096KiB/715msec) clat (nsec): min=96699k, max=96699k, avg=96698906.00, stdev= 0.00 lat (nsec): min=96700k, max=96700k, avg=96699797.00, stdev= 0.00 clat percentiles (usec): | 1.00th=[96994], 5.00th=[96994], 10.00th=[96994], 20.00th=[96994], | 30.00th=[96994], 40.00th=[96994], 50.00th=[96994], 60.00th=[96994], | 70.00th=[96994], 80.00th=[96994], 90.00th=[96994], 95.00th=[96994], | 99.00th=[96994], 99.50th=[96994], 99.90th=[96994], 99.95th=[96994], | 99.99th=[96994] bw ( KiB/s): min= 5876, max= 5876, per=25.64%, avg=5876.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=2, BW=11.2MiB/s (11.7MB/s)(8192KiB/715msec) clat (msec): min=175, max=406, avg=291.05, stdev=163.77 lat (msec): min=175, max=407, avg=291.29, stdev=163.79 clat percentiles (msec): | 1.00th=[ 176], 5.00th=[ 176], 10.00th=[ 176], 20.00th=[ 176], | 30.00th=[ 176], 40.00th=[ 176], 50.00th=[ 176], 60.00th=[ 409], | 70.00th=[ 409], 80.00th=[ 409], 90.00th=[ 409], 95.00th=[ 409], | 99.00th=[ 409], 99.50th=[ 409], 99.90th=[ 409], 99.95th=[ 409], | 99.99th=[ 409] bw ( KiB/s): min= 5876, max= 5876, per=12.82%, avg=5876.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 250=33.33%, 500=33.33% cpu : usr=0.00%, sys=31.37%, ctx=170, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=7312: Fri Apr 4 01:04:21 2025 write: IOPS=5, BW=23.0MiB/s (25.1MB/s)(12.0MiB/501msec) clat (msec): min=131, max=187, avg=164.14, stdev=28.81 lat (msec): min=132, max=187, avg=164.38, stdev=28.78 clat percentiles (msec): | 1.00th=[ 132], 5.00th=[ 132], 10.00th=[ 132], 20.00th=[ 132], | 30.00th=[ 132], 40.00th=[ 174], 50.00th=[ 174], 60.00th=[ 174], | 70.00th=[ 188], 80.00th=[ 188], 90.00th=[ 188], 95.00th=[ 188], | 99.00th=[ 188], 99.50th=[ 188], 99.90th=[ 188], 99.95th=[ 188], | 99.99th=[ 188] lat (msec) : 250=100.00% cpu : usr=0.00%, sys=41.80%, ctx=85, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=7314: Fri Apr 4 01:04:21 2025 write: IOPS=6, BW=26.1MiB/s (27.4MB/s)(12.0MiB/459msec) clat (msec): min=105, max=202, avg=139.44, stdev=54.35 lat (msec): min=105, max=202, avg=139.66, stdev=54.39 clat percentiles (msec): | 1.00th=[ 106], 5.00th=[ 106], 10.00th=[ 106], 20.00th=[ 106], | 30.00th=[ 106], 40.00th=[ 111], 50.00th=[ 111], 60.00th=[ 111], | 70.00th=[ 203], 80.00th=[ 203], 90.00th=[ 203], 95.00th=[ 203], | 99.00th=[ 203], 99.50th=[ 203], 99.90th=[ 203], 99.95th=[ 203], | 99.99th=[ 203] lat (msec) : 250=100.00% cpu : usr=0.00%, sys=50.87%, ctx=81, majf=0, minf=28 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=22.4MiB/s (23.5MB/s), 5729KiB/s-37.0MiB/s (5866kB/s-39.8MB/s), io=16.0MiB (16.8MB), run=316-715msec WRITE: bw=44.8MiB/s (46.9MB/s), 11.2MiB/s-26.1MiB/s (11.7MB/s-27.4MB/s), io=32.0MiB (33.6MB), run=459-715msec rand-rw: (groupid=0, jobs=1): err= 0: pid=7309: Fri Apr 4 01:04:21 2025 read: IOPS=5, BW=20.2MiB/s (21.1MB/s)(12.0MiB/595msec) clat (msec): min=136, max=226, avg=194.72, stdev=50.31 lat (msec): min=136, max=226, avg=194.72, stdev=50.31 clat percentiles (msec): | 1.00th=[ 138], 5.00th=[ 138], 10.00th=[ 138], 20.00th=[ 138], | 30.00th=[ 138], 40.00th=[ 222], 50.00th=[ 222], 60.00th=[ 222], | 70.00th=[ 228], 80.00th=[ 228], 90.00th=[ 228], 95.00th=[ 228], | 99.00th=[ 228], 99.50th=[ 228], 99.90th=[ 228], 99.95th=[ 228], | 99.99th=[ 228] lat (msec) : 250=100.00% cpu : usr=0.00%, sys=10.44%, ctx=33, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=7311: Fri Apr 4 01:04:21 2025 read: IOPS=1, BW=5063KiB/s (5185kB/s)(4096KiB/809msec) clat (nsec): min=228156k, max=228156k, avg=228155578.00, stdev= 0.00 lat (nsec): min=228157k, max=228157k, avg=228157426.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 228], 5.00th=[ 228], 10.00th=[ 228], 20.00th=[ 228], | 30.00th=[ 228], 40.00th=[ 228], 50.00th=[ 228], 60.00th=[ 228], | 70.00th=[ 228], 80.00th=[ 228], 90.00th=[ 228], 95.00th=[ 228], | 99.00th=[ 228], 99.50th=[ 228], 99.90th=[ 228], 99.95th=[ 228], | 99.99th=[ 228] bw ( KiB/s): min= 5673, max= 5673, per=28.01%, avg=5673.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=2, BW=9.89MiB/s (10.4MB/s)(8192KiB/809msec) clat (msec): min=276, max=291, avg=284.02, stdev=10.14 lat (msec): min=277, max=291, avg=284.32, stdev=10.11 clat percentiles (msec): | 1.00th=[ 279], 5.00th=[ 279], 10.00th=[ 279], 20.00th=[ 279], | 30.00th=[ 279], 40.00th=[ 279], 50.00th=[ 279], 60.00th=[ 292], | 70.00th=[ 292], 80.00th=[ 292], 90.00th=[ 292], 95.00th=[ 292], | 99.00th=[ 292], 99.50th=[ 292], 99.90th=[ 292], 99.95th=[ 292], | 99.99th=[ 292] bw ( KiB/s): min= 5673, max= 5673, per=14.01%, avg=5673.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=66.67% cpu : usr=0.12%, sys=11.39%, ctx=31, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=7313: Fri Apr 4 01:04:21 2025 write: IOPS=4, BW=16.6MiB/s (17.4MB/s)(12.0MiB/725msec) clat (msec): min=144, max=289, avg=235.33, stdev=78.86 lat (msec): min=145, max=289, avg=235.68, stdev=78.76 clat percentiles (msec): | 1.00th=[ 146], 5.00th=[ 146], 10.00th=[ 146], 20.00th=[ 146], | 30.00th=[ 146], 40.00th=[ 271], 50.00th=[ 271], 60.00th=[ 271], | 70.00th=[ 292], 80.00th=[ 292], 90.00th=[ 292], 95.00th=[ 292], | 99.00th=[ 292], 99.50th=[ 292], 99.90th=[ 292], 99.95th=[ 292], | 99.99th=[ 292] bw ( KiB/s): min=11538, max=11538, per=28.49%, avg=11538.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=66.67% cpu : usr=0.14%, sys=8.98%, ctx=37, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=7315: Fri Apr 4 01:04:21 2025 write: IOPS=6, BW=27.5MiB/s (28.9MB/s)(12.0MiB/436msec) clat (msec): min=95, max=186, avg=136.54, stdev=46.16 lat (msec): min=95, max=186, avg=136.81, stdev=46.27 clat percentiles (msec): | 1.00th=[ 96], 5.00th=[ 96], 10.00th=[ 96], 20.00th=[ 96], | 30.00th=[ 96], 40.00th=[ 128], 50.00th=[ 128], 60.00th=[ 128], | 70.00th=[ 186], 80.00th=[ 186], 90.00th=[ 186], 95.00th=[ 186], | 99.00th=[ 186], 99.50th=[ 186], 99.90th=[ 186], 99.95th=[ 186], | 99.99th=[ 186] lat (msec) : 100=33.33%, 250=66.67% cpu : usr=0.46%, sys=24.60%, ctx=86, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=19.8MiB/s (20.7MB/s), 5063KiB/s-20.2MiB/s (5185kB/s-21.1MB/s), io=16.0MiB (16.8MB), run=595-809msec WRITE: bw=39.6MiB/s (41.5MB/s), 9.89MiB/s-27.5MiB/s (10.4MB/s-28.9MB/s), io=32.0MiB (33.6MB), run=436-809msec