== sanity test 398b: DIO and buffer IO race ============== 09:53:09 (1742910789) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.33649 s, 37.7 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=28842: Tue Mar 25 09:53:24 2025 read: IOPS=120, BW=484KiB/s (496kB/s)(5992KiB/12381msec) clat (usec): min=63, max=210651, avg=4650.23, stdev=5902.12 lat (usec): min=64, max=210651, avg=4651.58, stdev=5902.03 clat percentiles (usec): | 1.00th=[ 68], 5.00th=[ 139], 10.00th=[ 2180], 20.00th=[ 2638], | 30.00th=[ 2999], 40.00th=[ 3589], 50.00th=[ 4047], 60.00th=[ 4752], | 70.00th=[ 5407], 80.00th=[ 6259], 90.00th=[ 7635], 95.00th=[ 8848], | 99.00th=[ 12125], 99.50th=[ 15008], 99.90th=[ 23462], 99.95th=[210764], | 99.99th=[210764] bw ( KiB/s): min= 247, max= 584, per=25.11%, avg=474.83, stdev=68.38, samples=24 iops : min= 61, max= 146, avg=118.62, stdev=17.20, samples=24 write: IOPS=127, BW=509KiB/s (521kB/s)(6296KiB/12381msec) clat (usec): min=464, max=28036, avg=3395.93, stdev=3025.25 lat (usec): min=464, max=28037, avg=3396.78, stdev=3025.17 clat percentiles (usec): | 1.00th=[ 515], 5.00th=[ 562], 10.00th=[ 594], 20.00th=[ 676], | 30.00th=[ 840], 40.00th=[ 1614], 50.00th=[ 2802], 60.00th=[ 3458], | 70.00th=[ 4424], 80.00th=[ 5669], 90.00th=[ 7570], 95.00th=[ 9241], | 99.00th=[13042], 99.50th=[14353], 99.90th=[17171], 99.95th=[27919], | 99.99th=[27919] bw ( KiB/s): min= 295, max= 632, per=25.08%, avg=496.54, stdev=69.72, samples=24 iops : min= 73, max= 158, avg=124.04, stdev=17.56, samples=24 lat (usec) : 100=1.86%, 250=0.65%, 500=0.29%, 750=13.31%, 1000=4.62% lat (msec) : 2=4.00%, 4=32.85%, 10=39.06%, 20=3.19%, 50=0.13% lat (msec) : 250=0.03% cpu : usr=0.22%, sys=23.00%, ctx=5550, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28843: Tue Mar 25 09:53:24 2025 read: IOPS=117, BW=470KiB/s (482kB/s)(5940KiB/12625msec) clat (usec): min=55, max=171986, avg=4561.85, stdev=5000.94 lat (usec): min=55, max=171988, avg=4562.55, stdev=5000.94 clat percentiles (usec): | 1.00th=[ 68], 5.00th=[ 121], 10.00th=[ 2147], 20.00th=[ 2573], | 30.00th=[ 2966], 40.00th=[ 3425], 50.00th=[ 4047], 60.00th=[ 4621], | 70.00th=[ 5407], 80.00th=[ 6259], 90.00th=[ 7504], 95.00th=[ 8848], | 99.00th=[ 12518], 99.50th=[ 14615], 99.90th=[ 19268], 99.95th=[173016], | 99.99th=[173016] bw ( KiB/s): min= 191, max= 712, per=24.75%, avg=468.04, stdev=91.68, samples=25 iops : min= 47, max= 178, avg=116.92, stdev=23.02, samples=25 write: IOPS=125, BW=503KiB/s (515kB/s)(6348KiB/12625msec) clat (usec): min=463, max=30824, avg=3635.46, stdev=3313.23 lat (usec): min=464, max=30825, avg=3636.40, stdev=3313.55 clat percentiles (usec): | 1.00th=[ 519], 5.00th=[ 562], 10.00th=[ 594], 20.00th=[ 676], | 30.00th=[ 873], 40.00th=[ 2040], 50.00th=[ 2966], 60.00th=[ 3589], | 70.00th=[ 4424], 80.00th=[ 6194], 90.00th=[ 8455], 95.00th=[10028], | 99.00th=[13304], 99.50th=[15139], 99.90th=[23462], 99.95th=[30802], | 99.99th=[30802] bw ( KiB/s): min= 255, max= 704, per=25.04%, avg=495.88, stdev=84.10, samples=25 iops : min= 63, max= 176, avg=123.88, stdev=21.15, samples=25 lat (usec) : 100=1.92%, 250=0.81%, 500=0.26%, 750=12.73%, 1000=4.17% lat (msec) : 2=4.26%, 4=33.11%, 10=38.74%, 20=3.87%, 50=0.10% lat (msec) : 250=0.03% cpu : usr=0.25%, sys=22.31%, ctx=5456, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28844: Tue Mar 25 09:53:24 2025 read: IOPS=120, BW=481KiB/s (493kB/s)(6028KiB/12521msec) clat (usec): min=58, max=20170, avg=4521.59, stdev=2593.00 lat (usec): min=58, max=20171, avg=4522.69, stdev=2592.84 clat percentiles (usec): | 1.00th=[ 73], 5.00th=[ 379], 10.00th=[ 2245], 20.00th=[ 2606], | 30.00th=[ 2999], 40.00th=[ 3458], 50.00th=[ 3982], 60.00th=[ 4752], | 70.00th=[ 5407], 80.00th=[ 6194], 90.00th=[ 7504], 95.00th=[ 8979], | 99.00th=[14353], 99.50th=[15926], 99.90th=[19792], 99.95th=[20055], | 99.99th=[20055] bw ( KiB/s): min= 208, max= 640, per=25.39%, avg=480.20, stdev=85.69, samples=25 iops : min= 52, max= 160, avg=119.96, stdev=21.48, samples=25 write: IOPS=124, BW=500KiB/s (512kB/s)(6260KiB/12521msec) clat (usec): min=449, max=207320, avg=3600.76, stdev=6019.20 lat (usec): min=449, max=207321, avg=3601.49, stdev=6019.19 clat percentiles (usec): | 1.00th=[ 515], 5.00th=[ 562], 10.00th=[ 594], 20.00th=[ 676], | 30.00th=[ 832], 40.00th=[ 1450], 50.00th=[ 2835], 60.00th=[ 3556], | 70.00th=[ 4424], 80.00th=[ 6063], 90.00th=[ 7898], 95.00th=[ 9503], | 99.00th=[ 13698], 99.50th=[ 14877], 99.90th=[ 19006], 99.95th=[206570], | 99.99th=[206570] bw ( KiB/s): min= 304, max= 728, per=25.19%, avg=498.72, stdev=105.25, samples=25 iops : min= 76, max= 182, avg=124.60, stdev=26.30, samples=25 lat (usec) : 100=1.66%, 250=0.72%, 500=0.26%, 750=13.18%, 1000=4.82% lat (msec) : 2=4.04%, 4=33.46%, 10=38.25%, 20=3.55%, 50=0.03% lat (msec) : 250=0.03% cpu : usr=0.26%, sys=22.37%, ctx=5698, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28845: Tue Mar 25 09:53:24 2025 read: IOPS=119, BW=477KiB/s (488kB/s)(6052KiB/12693msec) clat (usec): min=62, max=21111, avg=4544.30, stdev=2607.91 lat (usec): min=63, max=21112, avg=4545.33, stdev=2607.98 clat percentiles (usec): | 1.00th=[ 68], 5.00th=[ 120], 10.00th=[ 2147], 20.00th=[ 2638], | 30.00th=[ 2999], 40.00th=[ 3458], 50.00th=[ 4080], 60.00th=[ 4817], | 70.00th=[ 5473], 80.00th=[ 6390], 90.00th=[ 7570], 95.00th=[ 8979], | 99.00th=[13304], 99.50th=[15795], 99.90th=[18482], 99.95th=[21103], | 99.99th=[21103] bw ( KiB/s): min= 208, max= 648, per=24.89%, avg=470.64, stdev=86.30, samples=25 iops : min= 52, max= 162, avg=117.60, stdev=21.62, samples=25 write: IOPS=122, BW=491KiB/s (503kB/s)(6236KiB/12693msec) clat (usec): min=470, max=190167, avg=3690.41, stdev=5770.44 lat (usec): min=470, max=190168, avg=3691.39, stdev=5770.37 clat percentiles (usec): | 1.00th=[ 519], 5.00th=[ 553], 10.00th=[ 586], 20.00th=[ 660], | 30.00th=[ 807], 40.00th=[ 1565], 50.00th=[ 2835], 60.00th=[ 3490], | 70.00th=[ 4621], 80.00th=[ 6128], 90.00th=[ 8356], 95.00th=[ 9765], | 99.00th=[ 13698], 99.50th=[ 15533], 99.90th=[ 31065], 99.95th=[189793], | 99.99th=[189793] bw ( KiB/s): min= 224, max= 744, per=24.51%, avg=485.28, stdev=108.29, samples=25 iops : min= 56, max= 186, avg=121.28, stdev=27.04, samples=25 lat (usec) : 100=2.18%, 250=0.68%, 500=0.13%, 750=13.44%, 1000=4.23% lat (msec) : 2=4.17%, 4=32.10%, 10=38.93%, 20=3.97%, 50=0.13% lat (msec) : 250=0.03% cpu : usr=0.15%, sys=22.20%, ctx=5541, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1892KiB/s (1937kB/s), 470KiB/s-484KiB/s (482kB/s-496kB/s), io=23.4MiB (24.6MB), run=12381-12693msec WRITE: bw=1981KiB/s (2028kB/s), 491KiB/s-509KiB/s (503kB/s-521kB/s), io=24.6MiB (25.7MB), run=12381-12693msec rand-rw: (groupid=0, jobs=1): err= 0: pid=28838: Tue Mar 25 09:53:30 2025 read: IOPS=81, BW=325KiB/s (333kB/s)(5992KiB/18412msec) clat (usec): min=1604, max=214652, avg=5012.60, stdev=6287.38 lat (usec): min=1605, max=214655, avg=5013.56, stdev=6287.36 clat percentiles (usec): | 1.00th=[ 1860], 5.00th=[ 2089], 10.00th=[ 2311], 20.00th=[ 2737], | 30.00th=[ 3163], 40.00th=[ 3523], 50.00th=[ 3818], 60.00th=[ 4113], | 70.00th=[ 4686], 80.00th=[ 6325], 90.00th=[ 9372], 95.00th=[ 12387], | 99.00th=[ 16712], 99.50th=[ 19268], 99.90th=[ 25560], 99.95th=[214959], | 99.99th=[214959] bw ( KiB/s): min= 120, max= 600, per=24.75%, avg=319.53, stdev=103.20, samples=36 iops : min= 30, max= 150, avg=79.86, stdev=25.82, samples=36 write: IOPS=85, BW=342KiB/s (350kB/s)(6296KiB/18412msec) clat (usec): min=2351, max=41575, avg=6877.96, stdev=3355.78 lat (usec): min=2352, max=41576, avg=6879.02, stdev=3355.73 clat percentiles (usec): | 1.00th=[ 2835], 5.00th=[ 3261], 10.00th=[ 3556], 20.00th=[ 4228], | 30.00th=[ 4883], 40.00th=[ 5604], 50.00th=[ 6128], 60.00th=[ 6718], | 70.00th=[ 7439], 80.00th=[ 9110], 90.00th=[11207], 95.00th=[13173], | 99.00th=[18482], 99.50th=[20841], 99.90th=[27132], 99.95th=[41681], | 99.99th=[41681] bw ( KiB/s): min= 136, max= 712, per=24.72%, avg=334.19, stdev=101.96, samples=36 iops : min= 34, max= 178, avg=83.53, stdev=25.49, samples=36 lat (msec) : 2=1.46%, 4=34.80%, 10=51.69%, 20=11.59%, 50=0.42% lat (msec) : 250=0.03% cpu : usr=0.20%, sys=12.44%, ctx=3570, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28839: Tue Mar 25 09:53:30 2025 read: IOPS=80, BW=322KiB/s (330kB/s)(5940KiB/18459msec) clat (usec): min=1665, max=185354, avg=4915.04, stdev=5628.84 lat (usec): min=1665, max=185356, avg=4915.99, stdev=5628.79 clat percentiles (usec): | 1.00th=[ 1893], 5.00th=[ 2073], 10.00th=[ 2311], 20.00th=[ 2704], | 30.00th=[ 3097], 40.00th=[ 3458], 50.00th=[ 3851], 60.00th=[ 4146], | 70.00th=[ 4686], 80.00th=[ 5997], 90.00th=[ 9241], 95.00th=[ 11600], | 99.00th=[ 17695], 99.50th=[ 18482], 99.90th=[ 24511], 99.95th=[185598], | 99.99th=[185598] bw ( KiB/s): min= 95, max= 680, per=24.55%, avg=316.97, stdev=108.67, samples=36 iops : min= 23, max= 170, avg=79.14, stdev=27.22, samples=36 write: IOPS=85, BW=344KiB/s (352kB/s)(6348KiB/18459msec) clat (usec): min=2443, max=31352, avg=6982.84, stdev=3507.87 lat (usec): min=2444, max=31353, avg=6984.07, stdev=3507.89 clat percentiles (usec): | 1.00th=[ 2737], 5.00th=[ 3228], 10.00th=[ 3556], 20.00th=[ 4228], | 30.00th=[ 4883], 40.00th=[ 5538], 50.00th=[ 6259], 60.00th=[ 6849], | 70.00th=[ 7701], 80.00th=[ 9110], 90.00th=[11338], 95.00th=[13304], | 99.00th=[20055], 99.50th=[22676], 99.90th=[30802], 99.95th=[31327], | 99.99th=[31327] bw ( KiB/s): min= 159, max= 656, per=24.69%, avg=333.86, stdev=94.01, samples=36 iops : min= 39, max= 164, avg=83.36, stdev=23.61, samples=36 lat (msec) : 2=1.33%, 4=33.95%, 10=52.77%, 20=11.23%, 50=0.68% lat (msec) : 250=0.03% cpu : usr=0.22%, sys=12.55%, ctx=3621, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28840: Tue Mar 25 09:53:30 2025 read: IOPS=81, BW=324KiB/s (332kB/s)(6028KiB/18589msec) clat (usec): min=1703, max=28558, avg=4904.44, stdev=3288.26 lat (usec): min=1704, max=28559, avg=4905.35, stdev=3288.22 clat percentiles (usec): | 1.00th=[ 1860], 5.00th=[ 2114], 10.00th=[ 2278], 20.00th=[ 2704], | 30.00th=[ 3130], 40.00th=[ 3490], 50.00th=[ 3785], 60.00th=[ 4146], | 70.00th=[ 4817], 80.00th=[ 6325], 90.00th=[ 9503], 95.00th=[12125], | 99.00th=[16581], 99.50th=[20317], 99.90th=[25560], 99.95th=[28443], | 99.99th=[28443] bw ( KiB/s): min= 160, max= 616, per=25.14%, avg=324.51, stdev=107.28, samples=37 iops : min= 40, max= 154, avg=81.11, stdev=26.84, samples=37 write: IOPS=84, BW=337KiB/s (345kB/s)(6260KiB/18589msec) clat (msec): min=2, max=173, avg= 7.10, stdev= 5.54 lat (msec): min=2, max=173, avg= 7.10, stdev= 5.54 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 4], 10.00th=[ 4], 20.00th=[ 5], | 30.00th=[ 5], 40.00th=[ 6], 50.00th=[ 7], 60.00th=[ 7], | 70.00th=[ 8], 80.00th=[ 10], 90.00th=[ 12], 95.00th=[ 14], | 99.00th=[ 21], 99.50th=[ 22], 99.90th=[ 39], 99.95th=[ 174], | 99.99th=[ 174] bw ( KiB/s): min= 176, max= 680, per=24.88%, avg=336.41, stdev=121.52, samples=37 iops : min= 44, max= 170, avg=84.08, stdev=30.40, samples=37 lat (msec) : 2=1.27%, 4=33.98%, 10=52.12%, 20=11.69%, 50=0.91% lat (msec) : 250=0.03% cpu : usr=0.20%, sys=12.26%, ctx=3617, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28841: Tue Mar 25 09:53:30 2025 read: IOPS=82, BW=330KiB/s (338kB/s)(6052KiB/18353msec) clat (usec): min=1635, max=28658, avg=4773.38, stdev=3041.39 lat (usec): min=1635, max=28659, avg=4774.24, stdev=3041.38 clat percentiles (usec): | 1.00th=[ 1860], 5.00th=[ 2089], 10.00th=[ 2343], 20.00th=[ 2802], | 30.00th=[ 3130], 40.00th=[ 3458], 50.00th=[ 3785], 60.00th=[ 4113], | 70.00th=[ 4621], 80.00th=[ 6128], 90.00th=[ 9110], 95.00th=[11338], | 99.00th=[15139], 99.50th=[20055], 99.90th=[23200], 99.95th=[28705], | 99.99th=[28705] bw ( KiB/s): min= 144, max= 648, per=25.03%, avg=323.08, stdev=100.36, samples=36 iops : min= 36, max= 162, avg=80.75, stdev=25.10, samples=36 write: IOPS=84, BW=340KiB/s (348kB/s)(6236KiB/18353msec) clat (msec): min=2, max=189, avg= 7.09, stdev= 5.88 lat (msec): min=2, max=189, avg= 7.09, stdev= 5.88 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 4], 10.00th=[ 4], 20.00th=[ 5], | 30.00th=[ 6], 40.00th=[ 6], 50.00th=[ 7], 60.00th=[ 7], | 70.00th=[ 8], 80.00th=[ 9], 90.00th=[ 12], 95.00th=[ 14], | 99.00th=[ 21], 99.50th=[ 24], 99.90th=[ 42], 99.95th=[ 190], | 99.99th=[ 190] bw ( KiB/s): min= 160, max= 680, per=24.69%, avg=333.75, stdev=103.38, samples=36 iops : min= 40, max= 170, avg=83.42, stdev=25.86, samples=36 lat (msec) : 2=1.30%, 4=34.86%, 10=52.12%, 20=10.94%, 50=0.75% lat (msec) : 250=0.03% cpu : usr=0.19%, sys=12.53%, ctx=3561, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1292KiB/s (1323kB/s), 322KiB/s-330KiB/s (330kB/s-338kB/s), io=23.4MiB (24.6MB), run=18353-18589msec WRITE: bw=1352KiB/s (1385kB/s), 337KiB/s-344KiB/s (345kB/s-352kB/s), io=24.6MiB (25.7MB), run=18353-18589msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=28879: Tue Mar 25 09:53:35 2025 read: IOPS=108, BW=1732KiB/s (1773kB/s)(5920KiB/3419msec) clat (usec): min=67, max=18398, avg=4762.09, stdev=2951.45 lat (usec): min=68, max=18399, avg=4762.71, stdev=2951.47 clat percentiles (usec): | 1.00th=[ 70], 5.00th=[ 83], 10.00th=[ 131], 20.00th=[ 2737], | 30.00th=[ 3163], 40.00th=[ 3818], 50.00th=[ 4490], 60.00th=[ 5211], | 70.00th=[ 5800], 80.00th=[ 6718], 90.00th=[ 8094], 95.00th=[ 9896], | 99.00th=[13829], 99.50th=[16057], 99.90th=[18482], 99.95th=[18482], | 99.99th=[18482] bw ( KiB/s): min= 1277, max= 1820, per=25.16%, avg=1689.50, stdev=209.45, samples=6 iops : min= 79, max= 113, avg=105.33, stdev=13.32, samples=6 write: IOPS=116, BW=1863KiB/s (1907kB/s)(6368KiB/3419msec) clat (usec): min=636, max=18060, avg=4115.24, stdev=3386.57 lat (usec): min=637, max=18060, avg=4116.16, stdev=3386.60 clat percentiles (usec): | 1.00th=[ 676], 5.00th=[ 824], 10.00th=[ 865], 20.00th=[ 971], | 30.00th=[ 1139], 40.00th=[ 2245], 50.00th=[ 3425], 60.00th=[ 4359], | 70.00th=[ 5473], 80.00th=[ 6783], 90.00th=[ 8848], 95.00th=[11207], | 99.00th=[13960], 99.50th=[17957], 99.90th=[17957], 99.95th=[17957], | 99.99th=[17957] bw ( KiB/s): min= 1532, max= 2240, per=26.02%, avg=1838.50, stdev=304.00, samples=6 iops : min= 95, max= 140, avg=114.67, stdev=19.03, samples=6 lat (usec) : 100=4.04%, 250=1.30%, 750=1.30%, 1000=10.55% lat (msec) : 2=8.72%, 4=23.05%, 10=45.31%, 20=5.73% cpu : usr=0.18%, sys=24.43%, ctx=1518, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28880: Tue Mar 25 09:53:35 2025 read: IOPS=104, BW=1677KiB/s (1717kB/s)(5744KiB/3426msec) clat (usec): min=68, max=19571, avg=4761.84, stdev=2706.69 lat (usec): min=68, max=19572, avg=4762.61, stdev=2706.86 clat percentiles (usec): | 1.00th=[ 72], 5.00th=[ 87], 10.00th=[ 652], 20.00th=[ 2900], | 30.00th=[ 3425], 40.00th=[ 3884], 50.00th=[ 4490], 60.00th=[ 5276], | 70.00th=[ 5866], 80.00th=[ 6652], 90.00th=[ 7898], 95.00th=[ 9503], | 99.00th=[13173], 99.50th=[14746], 99.90th=[19530], 99.95th=[19530], | 99.99th=[19530] bw ( KiB/s): min= 1280, max= 1984, per=24.93%, avg=1674.00, stdev=256.07, samples=6 iops : min= 80, max= 124, avg=104.50, stdev=16.07, samples=6 write: IOPS=119, BW=1910KiB/s (1956kB/s)(6544KiB/3426msec) clat (usec): min=640, max=16090, avg=4027.34, stdev=3286.59 lat (usec): min=641, max=16090, avg=4028.27, stdev=3286.62 clat percentiles (usec): | 1.00th=[ 725], 5.00th=[ 807], 10.00th=[ 865], 20.00th=[ 1004], | 30.00th=[ 1450], 40.00th=[ 2540], 50.00th=[ 3130], 60.00th=[ 3851], | 70.00th=[ 4948], 80.00th=[ 6652], 90.00th=[ 8848], 95.00th=[10683], | 99.00th=[14615], 99.50th=[15533], 99.90th=[16057], 99.95th=[16057], | 99.99th=[16057] bw ( KiB/s): min= 1440, max= 2272, per=26.71%, avg=1887.17, stdev=302.64, samples=6 iops : min= 90, max= 142, avg=117.83, stdev=18.79, samples=6 lat (usec) : 100=2.86%, 250=1.43%, 500=0.13%, 750=1.04%, 1000=9.77% lat (msec) : 2=7.81%, 4=29.56%, 10=42.32%, 20=5.08% cpu : usr=0.23%, sys=24.99%, ctx=1513, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28881: Tue Mar 25 09:53:35 2025 read: IOPS=109, BW=1754KiB/s (1796kB/s)(6256KiB/3566msec) clat (usec): min=61, max=17111, avg=5074.40, stdev=3049.15 lat (usec): min=61, max=17112, avg=5075.26, stdev=3049.03 clat percentiles (usec): | 1.00th=[ 72], 5.00th=[ 90], 10.00th=[ 2376], 20.00th=[ 2999], | 30.00th=[ 3458], 40.00th=[ 4047], 50.00th=[ 4555], 60.00th=[ 5276], | 70.00th=[ 5800], 80.00th=[ 6718], 90.00th=[ 8586], 95.00th=[11207], | 99.00th=[16450], 99.50th=[17171], 99.90th=[17171], 99.95th=[17171], | 99.99th=[17171] bw ( KiB/s): min= 1184, max= 2080, per=25.07%, avg=1684.00, stdev=348.91, samples=6 iops : min= 74, max= 130, avg=105.00, stdev=21.65, samples=6 write: IOPS=105, BW=1692KiB/s (1732kB/s)(6032KiB/3566msec) clat (usec): min=664, max=17625, avg=3980.79, stdev=3155.86 lat (usec): min=664, max=17626, avg=3981.67, stdev=3155.86 clat percentiles (usec): | 1.00th=[ 717], 5.00th=[ 791], 10.00th=[ 840], 20.00th=[ 938], | 30.00th=[ 1352], 40.00th=[ 2573], 50.00th=[ 3294], 60.00th=[ 4047], | 70.00th=[ 5211], 80.00th=[ 6652], 90.00th=[ 8717], 95.00th=[10159], | 99.00th=[12256], 99.50th=[16188], 99.90th=[17695], 99.95th=[17695], | 99.99th=[17695] bw ( KiB/s): min= 1245, max= 1980, per=23.61%, avg=1668.17, stdev=313.16, samples=6 iops : min= 77, max= 123, avg=104.00, stdev=19.65, samples=6 lat (usec) : 100=2.86%, 250=0.52%, 500=0.26%, 750=1.04%, 1000=10.81% lat (msec) : 2=6.38%, 4=26.56%, 10=45.31%, 20=6.25% cpu : usr=0.11%, sys=23.70%, ctx=1639, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28882: Tue Mar 25 09:53:35 2025 read: IOPS=109, BW=1752KiB/s (1794kB/s)(6032KiB/3443msec) clat (usec): min=54, max=19262, avg=4815.11, stdev=3118.01 lat (usec): min=55, max=19262, avg=4815.77, stdev=3117.97 clat percentiles (usec): | 1.00th=[ 68], 5.00th=[ 81], 10.00th=[ 110], 20.00th=[ 2671], | 30.00th=[ 3294], 40.00th=[ 3785], 50.00th=[ 4424], 60.00th=[ 5211], | 70.00th=[ 6128], 80.00th=[ 6849], 90.00th=[ 8356], 95.00th=[10552], | 99.00th=[14615], 99.50th=[18482], 99.90th=[19268], 99.95th=[19268], | 99.99th=[19268] bw ( KiB/s): min= 1120, max= 2016, per=25.33%, avg=1701.33, stdev=310.53, samples=6 iops : min= 70, max= 126, avg=106.33, stdev=19.41, samples=6 write: IOPS=113, BW=1817KiB/s (1861kB/s)(6256KiB/3443msec) clat (usec): min=611, max=21442, avg=3942.35, stdev=3320.17 lat (usec): min=612, max=21443, avg=3943.24, stdev=3320.31 clat percentiles (usec): | 1.00th=[ 701], 5.00th=[ 799], 10.00th=[ 840], 20.00th=[ 971], | 30.00th=[ 1303], 40.00th=[ 2245], 50.00th=[ 3064], 60.00th=[ 3949], | 70.00th=[ 5080], 80.00th=[ 6718], 90.00th=[ 8291], 95.00th=[ 9765], | 99.00th=[15533], 99.50th=[16909], 99.90th=[21365], 99.95th=[21365], | 99.99th=[21365] bw ( KiB/s): min= 1376, max= 1984, per=25.06%, avg=1770.67, stdev=225.06, samples=6 iops : min= 86, max= 124, avg=110.67, stdev=14.07, samples=6 lat (usec) : 100=4.43%, 250=1.69%, 500=0.13%, 750=1.43%, 1000=10.03% lat (msec) : 2=8.33%, 4=26.04%, 10=42.97%, 20=4.82%, 50=0.13% cpu : usr=0.12%, sys=24.14%, ctx=1519, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=6717KiB/s (6878kB/s), 1677KiB/s-1754KiB/s (1717kB/s-1796kB/s), io=23.4MiB (24.5MB), run=3419-3566msec WRITE: bw=7067KiB/s (7236kB/s), 1692KiB/s-1910KiB/s (1732kB/s-1956kB/s), io=24.6MiB (25.8MB), run=3419-3566msec rand-rw: (groupid=0, jobs=1): err= 0: pid=28875: Tue Mar 25 09:53:36 2025 read: IOPS=83, BW=1340KiB/s (1372kB/s)(5920KiB/4418msec) clat (usec): min=1825, max=19491, avg=4784.68, stdev=3688.01 lat (usec): min=1826, max=19491, avg=4785.31, stdev=3688.02 clat percentiles (usec): | 1.00th=[ 1926], 5.00th=[ 2057], 10.00th=[ 2147], 20.00th=[ 2311], | 30.00th=[ 2474], 40.00th=[ 2802], 50.00th=[ 3195], 60.00th=[ 3687], | 70.00th=[ 4686], 80.00th=[ 6456], 90.00th=[10421], 95.00th=[13435], | 99.00th=[18482], 99.50th=[19268], 99.90th=[19530], 99.95th=[19530], | 99.99th=[19530] bw ( KiB/s): min= 798, max= 2464, per=22.66%, avg=1207.38, stdev=545.69, samples=8 iops : min= 49, max= 154, avg=75.25, stdev=34.22, samples=8 write: IOPS=90, BW=1441KiB/s (1476kB/s)(6368KiB/4418msec) clat (usec): min=2469, max=23537, avg=6489.47, stdev=3960.98 lat (usec): min=2470, max=23537, avg=6490.81, stdev=3960.88 clat percentiles (usec): | 1.00th=[ 2671], 5.00th=[ 2999], 10.00th=[ 3261], 20.00th=[ 3458], | 30.00th=[ 3818], 40.00th=[ 4146], 50.00th=[ 4686], 60.00th=[ 5866], | 70.00th=[ 7570], 80.00th=[ 9241], 90.00th=[12518], 95.00th=[14353], | 99.00th=[21365], 99.50th=[22676], 99.90th=[23462], 99.95th=[23462], | 99.99th=[23462] bw ( KiB/s): min= 926, max= 2528, per=23.61%, avg=1323.38, stdev=512.63, samples=8 iops : min= 57, max= 158, avg=82.50, stdev=32.18, samples=8 lat (msec) : 2=1.56%, 4=47.14%, 10=36.85%, 20=13.93%, 50=0.52% cpu : usr=0.18%, sys=11.84%, ctx=947, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28876: Tue Mar 25 09:53:36 2025 read: IOPS=79, BW=1278KiB/s (1309kB/s)(5744KiB/4495msec) clat (usec): min=1841, max=19515, avg=4951.15, stdev=3782.45 lat (usec): min=1841, max=19515, avg=4951.78, stdev=3782.45 clat percentiles (usec): | 1.00th=[ 1926], 5.00th=[ 2073], 10.00th=[ 2180], 20.00th=[ 2376], | 30.00th=[ 2573], 40.00th=[ 2737], 50.00th=[ 3195], 60.00th=[ 3752], | 70.00th=[ 4948], 80.00th=[ 7308], 90.00th=[10945], 95.00th=[13566], | 99.00th=[16909], 99.50th=[18482], 99.90th=[19530], 99.95th=[19530], | 99.99th=[19530] bw ( KiB/s): min= 832, max= 2528, per=22.22%, avg=1184.00, stdev=555.57, samples=8 iops : min= 52, max= 158, avg=74.00, stdev=34.72, samples=8 write: IOPS=90, BW=1456KiB/s (1491kB/s)(6544KiB/4495msec) clat (usec): min=2635, max=26311, avg=6502.73, stdev=3973.54 lat (usec): min=2637, max=26313, avg=6503.96, stdev=3973.55 clat percentiles (usec): | 1.00th=[ 2704], 5.00th=[ 2999], 10.00th=[ 3163], 20.00th=[ 3425], | 30.00th=[ 3687], 40.00th=[ 4113], 50.00th=[ 5014], 60.00th=[ 6128], | 70.00th=[ 7504], 80.00th=[ 9241], 90.00th=[12125], 95.00th=[14353], | 99.00th=[20841], 99.50th=[21627], 99.90th=[26346], 99.95th=[26346], | 99.99th=[26346] bw ( KiB/s): min= 832, max= 2496, per=23.26%, avg=1304.00, stdev=502.99, samples=8 iops : min= 52, max= 156, avg=81.50, stdev=31.44, samples=8 lat (msec) : 2=1.17%, 4=48.05%, 10=36.46%, 20=13.54%, 50=0.78% cpu : usr=0.11%, sys=11.95%, ctx=948, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28877: Tue Mar 25 09:53:36 2025 read: IOPS=88, BW=1420KiB/s (1454kB/s)(6256KiB/4405msec) clat (usec): min=1853, max=19891, avg=4624.47, stdev=3380.81 lat (usec): min=1854, max=19892, avg=4625.14, stdev=3380.80 clat percentiles (usec): | 1.00th=[ 1942], 5.00th=[ 2040], 10.00th=[ 2180], 20.00th=[ 2376], | 30.00th=[ 2540], 40.00th=[ 2769], 50.00th=[ 3228], 60.00th=[ 3818], | 70.00th=[ 4817], 80.00th=[ 6194], 90.00th=[10028], 95.00th=[12387], | 99.00th=[16909], 99.50th=[19530], 99.90th=[19792], 99.95th=[19792], | 99.99th=[19792] bw ( KiB/s): min= 800, max= 2272, per=23.72%, avg=1264.00, stdev=510.57, samples=8 iops : min= 50, max= 142, avg=79.00, stdev=31.91, samples=8 write: IOPS=85, BW=1369KiB/s (1402kB/s)(6032KiB/4405msec) clat (usec): min=2414, max=27427, avg=6840.13, stdev=4404.03 lat (usec): min=2415, max=27428, avg=6841.44, stdev=4404.08 clat percentiles (usec): | 1.00th=[ 2573], 5.00th=[ 2966], 10.00th=[ 3195], 20.00th=[ 3392], | 30.00th=[ 3654], 40.00th=[ 4080], 50.00th=[ 5145], 60.00th=[ 6390], | 70.00th=[ 8160], 80.00th=[10421], 90.00th=[13042], 95.00th=[15270], | 99.00th=[21890], 99.50th=[24249], 99.90th=[27395], 99.95th=[27395], | 99.99th=[27395] bw ( KiB/s): min= 992, max= 2560, per=22.62%, avg=1268.00, stdev=527.46, samples=8 iops : min= 62, max= 160, avg=79.25, stdev=32.97, samples=8 lat (msec) : 2=1.04%, 4=49.09%, 10=34.37%, 20=14.58%, 50=0.91% cpu : usr=0.16%, sys=12.31%, ctx=935, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28878: Tue Mar 25 09:53:36 2025 read: IOPS=85, BW=1367KiB/s (1400kB/s)(6032KiB/4413msec) clat (usec): min=1752, max=19487, avg=4789.97, stdev=3614.89 lat (usec): min=1752, max=19487, avg=4790.60, stdev=3614.91 clat percentiles (usec): | 1.00th=[ 1876], 5.00th=[ 2040], 10.00th=[ 2147], 20.00th=[ 2311], | 30.00th=[ 2507], 40.00th=[ 2769], 50.00th=[ 3195], 60.00th=[ 3720], | 70.00th=[ 4948], 80.00th=[ 6652], 90.00th=[11076], 95.00th=[13173], | 99.00th=[16581], 99.50th=[19268], 99.90th=[19530], 99.95th=[19530], | 99.99th=[19530] bw ( KiB/s): min= 768, max= 2368, per=22.97%, avg=1224.00, stdev=490.92, samples=8 iops : min= 48, max= 148, avg=76.50, stdev=30.68, samples=8 write: IOPS=88, BW=1418KiB/s (1452kB/s)(6256KiB/4413msec) clat (usec): min=2452, max=24351, avg=6488.60, stdev=4322.18 lat (usec): min=2453, max=24352, avg=6489.96, stdev=4322.36 clat percentiles (usec): | 1.00th=[ 2573], 5.00th=[ 2933], 10.00th=[ 3130], 20.00th=[ 3392], | 30.00th=[ 3621], 40.00th=[ 3949], 50.00th=[ 4424], 60.00th=[ 5604], | 70.00th=[ 7439], 80.00th=[ 9634], 90.00th=[12518], 95.00th=[15401], | 99.00th=[21627], 99.50th=[23462], 99.90th=[24249], 99.95th=[24249], | 99.99th=[24249] bw ( KiB/s): min= 800, max= 2592, per=23.19%, avg=1300.00, stdev=550.53, samples=8 iops : min= 50, max= 162, avg=81.25, stdev=34.41, samples=8 lat (msec) : 2=1.30%, 4=51.17%, 10=32.29%, 20=14.32%, 50=0.91% cpu : usr=0.09%, sys=12.33%, ctx=923, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=5329KiB/s (5456kB/s), 1278KiB/s-1420KiB/s (1309kB/s-1454kB/s), io=23.4MiB (24.5MB), run=4405-4495msec WRITE: bw=5606KiB/s (5741kB/s), 1369KiB/s-1456KiB/s (1402kB/s-1491kB/s), io=24.6MiB (25.8MB), run=4405-4495msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=28891: Tue Mar 25 09:53:38 2025 read: IOPS=12, BW=12.3MiB/s (12.9MB/s)(7168KiB/569msec) clat (usec): min=15557, max=59413, avg=31532.49, stdev=15226.41 lat (usec): min=15559, max=59414, avg=31533.64, stdev=15226.18 clat percentiles (usec): | 1.00th=[15533], 5.00th=[15533], 10.00th=[15533], 20.00th=[17695], | 30.00th=[22938], 40.00th=[22938], 50.00th=[30540], 60.00th=[33817], | 70.00th=[33817], 80.00th=[40633], 90.00th=[59507], 95.00th=[59507], | 99.00th=[59507], 99.50th=[59507], 99.90th=[59507], 99.95th=[59507], | 99.99th=[59507] bw ( KiB/s): min=12288, max=12288, per=43.62%, avg=12288.00, stdev= 0.00, samples=1 iops : min= 12, max= 12, avg=12.00, stdev= 0.00, samples=1 write: IOPS=8, BW=8998KiB/s (9214kB/s)(5120KiB/569msec) clat (usec): min=14657, max=86233, avg=61750.91, stdev=31755.55 lat (usec): min=14719, max=86305, avg=61809.77, stdev=31752.05 clat percentiles (usec): | 1.00th=[14615], 5.00th=[14615], 10.00th=[14615], 20.00th=[14615], | 30.00th=[43254], 40.00th=[43254], 50.00th=[80217], 60.00th=[80217], | 70.00th=[84411], 80.00th=[84411], 90.00th=[86508], 95.00th=[86508], | 99.00th=[86508], 99.50th=[86508], 99.90th=[86508], 99.95th=[86508], | 99.99th=[86508] bw ( KiB/s): min= 8192, max= 8192, per=20.77%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 lat (msec) : 20=25.00%, 50=41.67%, 100=33.33% cpu : usr=0.00%, sys=16.02%, ctx=51, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28892: Tue Mar 25 09:53:38 2025 read: IOPS=2, BW=2817KiB/s (2885kB/s)(2048KiB/727msec) clat (usec): min=25794, max=69256, avg=47525.76, stdev=30732.55 lat (usec): min=25796, max=69257, avg=47526.99, stdev=30731.79 clat percentiles (usec): | 1.00th=[25822], 5.00th=[25822], 10.00th=[25822], 20.00th=[25822], | 30.00th=[25822], 40.00th=[25822], 50.00th=[25822], 60.00th=[69731], | 70.00th=[69731], 80.00th=[69731], 90.00th=[69731], 95.00th=[69731], | 99.00th=[69731], 99.50th=[69731], 99.90th=[69731], 99.95th=[69731], | 99.99th=[69731] bw ( KiB/s): min= 4096, max= 4096, per=14.54%, avg=4096.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 write: IOPS=13, BW=13.8MiB/s (14.4MB/s)(10.0MiB/727msec) clat (msec): min=22, max=168, avg=61.58, stdev=48.63 lat (msec): min=22, max=168, avg=61.63, stdev=48.63 clat percentiles (msec): | 1.00th=[ 23], 5.00th=[ 23], 10.00th=[ 23], 20.00th=[ 24], | 30.00th=[ 27], 40.00th=[ 31], 50.00th=[ 37], 60.00th=[ 46], | 70.00th=[ 85], 80.00th=[ 85], 90.00th=[ 120], 95.00th=[ 169], | 99.00th=[ 169], 99.50th=[ 169], 99.90th=[ 169], 99.95th=[ 169], | 99.99th=[ 169] bw ( KiB/s): min=10240, max=10240, per=25.96%, avg=10240.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 lat (msec) : 50=58.33%, 100=25.00%, 250=16.67% cpu : usr=0.28%, sys=19.28%, ctx=77, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28893: Tue Mar 25 09:53:38 2025 read: IOPS=8, BW=8533KiB/s (8738kB/s)(6144KiB/720msec) clat (msec): min=13, max=104, avg=47.87, stdev=39.09 lat (msec): min=13, max=104, avg=47.87, stdev=39.09 clat percentiles (msec): | 1.00th=[ 14], 5.00th=[ 14], 10.00th=[ 14], 20.00th=[ 16], | 30.00th=[ 16], 40.00th=[ 24], 50.00th=[ 24], 60.00th=[ 43], | 70.00th=[ 88], 80.00th=[ 88], 90.00th=[ 105], 95.00th=[ 105], | 99.00th=[ 105], 99.50th=[ 105], 99.90th=[ 105], 99.95th=[ 105], | 99.99th=[ 105] bw ( KiB/s): min= 4096, max= 4096, per=14.54%, avg=4096.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 write: IOPS=8, BW=8533KiB/s (8738kB/s)(6144KiB/720msec) clat (usec): min=30132, max=88626, avg=69463.45, stdev=23245.45 lat (usec): min=30172, max=88699, avg=69525.92, stdev=23266.92 clat percentiles (usec): | 1.00th=[30016], 5.00th=[30016], 10.00th=[30016], 20.00th=[52691], | 30.00th=[52691], 40.00th=[76022], 50.00th=[76022], 60.00th=[83362], | 70.00th=[86508], 80.00th=[86508], 90.00th=[88605], 95.00th=[88605], | 99.00th=[88605], 99.50th=[88605], 99.90th=[88605], 99.95th=[88605], | 99.99th=[88605] bw ( KiB/s): min= 8192, max= 8192, per=20.77%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 lat (msec) : 20=16.67%, 50=25.00%, 100=50.00%, 250=8.33% cpu : usr=0.00%, sys=12.38%, ctx=69, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28894: Tue Mar 25 09:53:38 2025 read: IOPS=7, BW=7630KiB/s (7814kB/s)(5120KiB/671msec) clat (usec): min=14884, max=83351, avg=38428.91, stdev=27528.97 lat (usec): min=14885, max=83352, avg=38429.89, stdev=27529.00 clat percentiles (usec): | 1.00th=[14877], 5.00th=[14877], 10.00th=[14877], 20.00th=[14877], | 30.00th=[19530], 40.00th=[19530], 50.00th=[30016], 60.00th=[30016], | 70.00th=[44303], 80.00th=[44303], 90.00th=[83362], 95.00th=[83362], | 99.00th=[83362], 99.50th=[83362], 99.90th=[83362], 99.95th=[83362], | 99.99th=[83362] bw ( KiB/s): min= 8192, max= 8192, per=29.08%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=10, BW=10.4MiB/s (10.9MB/s)(7168KiB/671msec) clat (msec): min=24, max=118, avg=65.68, stdev=36.00 lat (msec): min=24, max=118, avg=65.75, stdev=36.00 clat percentiles (msec): | 1.00th=[ 25], 5.00th=[ 25], 10.00th=[ 25], 20.00th=[ 28], | 30.00th=[ 52], 40.00th=[ 52], 50.00th=[ 58], 60.00th=[ 77], | 70.00th=[ 77], 80.00th=[ 105], 90.00th=[ 120], 95.00th=[ 120], | 99.00th=[ 120], 99.50th=[ 120], 99.90th=[ 120], 99.95th=[ 120], | 99.99th=[ 120] bw ( KiB/s): min=10240, max=10240, per=25.96%, avg=10240.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 lat (msec) : 20=16.67%, 50=33.33%, 100=33.33%, 250=16.67% cpu : usr=0.30%, sys=10.00%, ctx=51, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=27.5MiB/s (28.8MB/s), 2817KiB/s-12.3MiB/s (2885kB/s-12.9MB/s), io=20.0MiB (20.0MB), run=569-727msec WRITE: bw=38.5MiB/s (40.4MB/s), 8533KiB/s-13.8MiB/s (8738kB/s-14.4MB/s), io=28.0MiB (29.4MB), run=569-727msec rand-rw: (groupid=0, jobs=1): err= 0: pid=28895: Tue Mar 25 09:53:38 2025 read: IOPS=12, BW=12.4MiB/s (13.0MB/s)(7168KiB/564msec) clat (usec): min=420, max=41184, avg=29533.62, stdev=13757.44 lat (usec): min=421, max=41186, avg=29534.61, stdev=13757.66 clat percentiles (usec): | 1.00th=[ 420], 5.00th=[ 420], 10.00th=[ 420], 20.00th=[25560], | 30.00th=[31589], 40.00th=[31589], 50.00th=[34341], 60.00th=[35390], | 70.00th=[35390], 80.00th=[38011], 90.00th=[41157], 95.00th=[41157], | 99.00th=[41157], 99.50th=[41157], 99.90th=[41157], 99.95th=[41157], | 99.99th=[41157] bw ( KiB/s): min=10240, max=10240, per=39.60%, avg=10240.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 write: IOPS=8, BW=9078KiB/s (9296kB/s)(5120KiB/564msec) clat (msec): min=27, max=100, avg=58.27, stdev=28.16 lat (msec): min=28, max=100, avg=58.32, stdev=28.18 clat percentiles (msec): | 1.00th=[ 28], 5.00th=[ 28], 10.00th=[ 28], 20.00th=[ 28], | 30.00th=[ 42], 40.00th=[ 42], 50.00th=[ 52], 60.00th=[ 52], | 70.00th=[ 70], 80.00th=[ 70], 90.00th=[ 101], 95.00th=[ 101], | 99.00th=[ 101], 99.50th=[ 101], 99.90th=[ 101], 99.95th=[ 101], | 99.99th=[ 101] bw ( KiB/s): min= 8192, max= 8192, per=22.63%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 lat (usec) : 500=8.33% lat (msec) : 50=66.67%, 100=16.67%, 250=8.33% cpu : usr=0.00%, sys=35.88%, ctx=920, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28896: Tue Mar 25 09:53:38 2025 read: IOPS=2, BW=2586KiB/s (2648kB/s)(2048KiB/792msec) clat (usec): min=68667, max=71904, avg=70286.35, stdev=2288.97 lat (usec): min=68668, max=71905, avg=70287.14, stdev=2288.75 clat percentiles (usec): | 1.00th=[68682], 5.00th=[68682], 10.00th=[68682], 20.00th=[68682], | 30.00th=[68682], 40.00th=[68682], 50.00th=[68682], 60.00th=[71828], | 70.00th=[71828], 80.00th=[71828], 90.00th=[71828], 95.00th=[71828], | 99.00th=[71828], 99.50th=[71828], 99.90th=[71828], 99.95th=[71828], | 99.99th=[71828] bw ( KiB/s): min= 4096, max= 4096, per=15.84%, avg=4096.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 write: IOPS=12, BW=12.6MiB/s (13.2MB/s)(10.0MiB/792msec) clat (msec): min=23, max=135, avg=58.33, stdev=34.84 lat (msec): min=23, max=135, avg=58.39, stdev=34.83 clat percentiles (msec): | 1.00th=[ 24], 5.00th=[ 24], 10.00th=[ 24], 20.00th=[ 29], | 30.00th=[ 29], 40.00th=[ 34], 50.00th=[ 46], 60.00th=[ 58], | 70.00th=[ 77], 80.00th=[ 77], 90.00th=[ 89], 95.00th=[ 136], | 99.00th=[ 136], 99.50th=[ 136], 99.90th=[ 136], 99.95th=[ 136], | 99.99th=[ 136] bw ( KiB/s): min=10240, max=10240, per=28.29%, avg=10240.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 lat (msec) : 50=41.67%, 100=50.00%, 250=8.33% cpu : usr=0.13%, sys=33.25%, ctx=154, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28897: Tue Mar 25 09:53:38 2025 read: IOPS=10, BW=10.2MiB/s (10.7MB/s)(6144KiB/590msec) clat (usec): min=351, max=109362, avg=35297.71, stdev=46318.36 lat (usec): min=352, max=109362, avg=35298.44, stdev=46318.27 clat percentiles (usec): | 1.00th=[ 351], 5.00th=[ 351], 10.00th=[ 351], 20.00th=[ 433], | 30.00th=[ 433], 40.00th=[ 562], 50.00th=[ 562], 60.00th=[ 26870], | 70.00th=[ 73925], 80.00th=[ 73925], 90.00th=[109577], 95.00th=[109577], | 99.00th=[109577], 99.50th=[109577], 99.90th=[109577], 99.95th=[109577], | 99.99th=[109577] bw ( KiB/s): min= 8192, max= 8192, per=31.68%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=10, BW=10.2MiB/s (10.7MB/s)(6144KiB/590msec) clat (msec): min=31, max=106, avg=58.88, stdev=28.65 lat (msec): min=31, max=106, avg=58.93, stdev=28.67 clat percentiles (msec): | 1.00th=[ 32], 5.00th=[ 32], 10.00th=[ 32], 20.00th=[ 37], | 30.00th=[ 37], 40.00th=[ 51], 50.00th=[ 51], 60.00th=[ 51], | 70.00th=[ 79], 80.00th=[ 79], 90.00th=[ 107], 95.00th=[ 107], | 99.00th=[ 107], 99.50th=[ 107], 99.90th=[ 107], 99.95th=[ 107], | 99.99th=[ 107] bw ( KiB/s): min= 8192, max= 8192, per=22.63%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 lat (usec) : 500=16.67%, 750=8.33% lat (msec) : 50=25.00%, 100=33.33%, 250=16.67% cpu : usr=0.00%, sys=29.88%, ctx=516, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28898: Tue Mar 25 09:53:38 2025 read: IOPS=6, BW=7082KiB/s (7252kB/s)(5120KiB/723msec) clat (usec): min=355, max=54057, avg=17179.74, stdev=24395.46 lat (usec): min=355, max=54058, avg=17180.42, stdev=24395.63 clat percentiles (usec): | 1.00th=[ 355], 5.00th=[ 355], 10.00th=[ 355], 20.00th=[ 355], | 30.00th=[ 420], 40.00th=[ 420], 50.00th=[ 515], 60.00th=[ 515], | 70.00th=[30540], 80.00th=[30540], 90.00th=[54264], 95.00th=[54264], | 99.00th=[54264], 99.50th=[54264], 99.90th=[54264], 99.95th=[54264], | 99.99th=[54264] bw ( KiB/s): min= 8192, max= 8192, per=31.68%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=9, BW=9914KiB/s (10.2MB/s)(7168KiB/723msec) clat (msec): min=59, max=117, avg=81.21, stdev=21.27 lat (msec): min=59, max=117, avg=81.26, stdev=21.28 clat percentiles (msec): | 1.00th=[ 60], 5.00th=[ 60], 10.00th=[ 60], 20.00th=[ 62], | 30.00th=[ 65], 40.00th=[ 65], 50.00th=[ 81], 60.00th=[ 91], | 70.00th=[ 91], 80.00th=[ 94], 90.00th=[ 118], 95.00th=[ 118], | 99.00th=[ 118], 99.50th=[ 118], 99.90th=[ 118], 99.95th=[ 118], | 99.99th=[ 118] bw ( KiB/s): min= 8192, max= 8192, per=22.63%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 lat (usec) : 500=16.67%, 750=8.33% lat (msec) : 50=8.33%, 100=58.33%, 250=8.33% cpu : usr=0.00%, sys=26.18%, ctx=494, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=25.3MiB/s (26.5MB/s), 2586KiB/s-12.4MiB/s (2648kB/s-13.0MB/s), io=20.0MiB (20.0MB), run=564-792msec WRITE: bw=35.4MiB/s (37.1MB/s), 9078KiB/s-12.6MiB/s (9296kB/s-13.2MB/s), io=28.0MiB (29.4MB), run=564-792msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=28904: Tue Mar 25 09:53:39 2025 read: IOPS=19, BW=76.9MiB/s (80.7MB/s)(12.0MiB/156msec) clat (usec): min=1475, max=111702, avg=47395.01, stdev=57367.85 lat (usec): min=1477, max=111703, avg=47396.39, stdev=57367.45 clat percentiles (usec): | 1.00th=[ 1483], 5.00th=[ 1483], 10.00th=[ 1483], 20.00th=[ 1483], | 30.00th=[ 1483], 40.00th=[ 28967], 50.00th=[ 28967], 60.00th=[ 28967], | 70.00th=[111674], 80.00th=[111674], 90.00th=[111674], 95.00th=[111674], | 99.00th=[111674], 99.50th=[111674], 99.90th=[111674], 99.95th=[111674], | 99.99th=[111674] lat (msec) : 2=33.33%, 50=33.33%, 250=33.33% cpu : usr=0.00%, sys=41.94%, ctx=721, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28906: Tue Mar 25 09:53:39 2025 read: IOPS=1, BW=5185KiB/s (5309kB/s)(4096KiB/790msec) clat (nsec): min=173042k, max=173042k, avg=173041744.00, stdev= 0.00 lat (nsec): min=173044k, max=173044k, avg=173044248.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 174], 5.00th=[ 174], 10.00th=[ 174], 20.00th=[ 174], | 30.00th=[ 174], 40.00th=[ 174], 50.00th=[ 174], 60.00th=[ 174], | 70.00th=[ 174], 80.00th=[ 174], 90.00th=[ 174], 95.00th=[ 174], | 99.00th=[ 174], 99.50th=[ 174], 99.90th=[ 174], 99.95th=[ 174], | 99.99th=[ 174] bw ( KiB/s): min= 8192, max= 8192, per=39.50%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=2, BW=10.1MiB/s (10.6MB/s)(8192KiB/790msec) clat (msec): min=203, max=377, avg=290.49, stdev=122.97 lat (msec): min=204, max=377, avg=290.87, stdev=122.85 clat percentiles (msec): | 1.00th=[ 205], 5.00th=[ 205], 10.00th=[ 205], 20.00th=[ 205], | 30.00th=[ 205], 40.00th=[ 205], 50.00th=[ 205], 60.00th=[ 376], | 70.00th=[ 376], 80.00th=[ 376], 90.00th=[ 376], 95.00th=[ 376], | 99.00th=[ 376], 99.50th=[ 376], 99.90th=[ 376], 99.95th=[ 376], | 99.99th=[ 376] bw ( KiB/s): min= 8192, max= 8192, per=19.75%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.00%, sys=31.05%, ctx=381, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28907: Tue Mar 25 09:53:39 2025 write: IOPS=4, BW=17.3MiB/s (18.2MB/s)(12.0MiB/693msec) clat (msec): min=191, max=252, avg=217.45, stdev=31.91 lat (msec): min=191, max=253, avg=217.71, stdev=31.91 clat percentiles (msec): | 1.00th=[ 192], 5.00th=[ 192], 10.00th=[ 192], 20.00th=[ 192], | 30.00th=[ 192], 40.00th=[ 209], 50.00th=[ 209], 60.00th=[ 209], | 70.00th=[ 253], 80.00th=[ 253], 90.00th=[ 253], 95.00th=[ 253], | 99.00th=[ 253], 99.50th=[ 253], 99.90th=[ 253], 99.95th=[ 253], | 99.99th=[ 253] bw ( KiB/s): min=16384, max=16384, per=39.50%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.00%, sys=43.35%, ctx=76, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28908: Tue Mar 25 09:53:39 2025 write: IOPS=5, BW=22.3MiB/s (23.3MB/s)(12.0MiB/539msec) clat (msec): min=113, max=196, avg=166.66, stdev=46.32 lat (msec): min=116, max=196, avg=167.79, stdev=44.79 clat percentiles (msec): | 1.00th=[ 114], 5.00th=[ 114], 10.00th=[ 114], 20.00th=[ 114], | 30.00th=[ 114], 40.00th=[ 190], 50.00th=[ 190], 60.00th=[ 190], | 70.00th=[ 197], 80.00th=[ 197], 90.00th=[ 197], 95.00th=[ 197], | 99.00th=[ 197], 99.50th=[ 197], 99.90th=[ 197], 99.95th=[ 197], | 99.99th=[ 197] bw ( KiB/s): min=16384, max=16384, per=39.50%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 250=100.00% cpu : usr=0.00%, sys=48.33%, ctx=119, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=20.3MiB/s (21.2MB/s), 5185KiB/s-76.9MiB/s (5309kB/s-80.7MB/s), io=16.0MiB (16.8MB), run=156-790msec WRITE: bw=40.5MiB/s (42.5MB/s), 10.1MiB/s-22.3MiB/s (10.6MB/s-23.3MB/s), io=32.0MiB (33.6MB), run=539-790msec rand-rw: (groupid=0, jobs=1): err= 0: pid=28909: Tue Mar 25 09:53:40 2025 read: IOPS=4, BW=16.3MiB/s (17.1MB/s)(12.0MiB/734msec) clat (msec): min=137, max=281, avg=226.93, stdev=78.34 lat (msec): min=137, max=281, avg=226.93, stdev=78.34 clat percentiles (msec): | 1.00th=[ 138], 5.00th=[ 138], 10.00th=[ 138], 20.00th=[ 138], | 30.00th=[ 138], 40.00th=[ 264], 50.00th=[ 264], 60.00th=[ 264], | 70.00th=[ 284], 80.00th=[ 284], 90.00th=[ 284], 95.00th=[ 284], | 99.00th=[ 284], 99.50th=[ 284], 99.90th=[ 284], 99.95th=[ 284], | 99.99th=[ 284] bw ( KiB/s): min= 8192, max= 8192, per=45.75%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=66.67% cpu : usr=0.00%, sys=9.55%, ctx=41, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28910: Tue Mar 25 09:53:40 2025 read: IOPS=1, BW=4477KiB/s (4584kB/s)(4096KiB/915msec) clat (nsec): min=272790k, max=272790k, avg=272790079.00, stdev= 0.00 lat (nsec): min=272792k, max=272792k, avg=272791854.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 275], 5.00th=[ 275], 10.00th=[ 275], 20.00th=[ 275], | 30.00th=[ 275], 40.00th=[ 275], 50.00th=[ 275], 60.00th=[ 275], | 70.00th=[ 275], 80.00th=[ 275], 90.00th=[ 275], 95.00th=[ 275], | 99.00th=[ 275], 99.50th=[ 275], 99.90th=[ 275], 99.95th=[ 275], | 99.99th=[ 275] bw ( KiB/s): min= 8192, max= 8192, per=45.75%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=2, BW=8953KiB/s (9168kB/s)(8192KiB/915msec) clat (msec): min=215, max=390, avg=302.76, stdev=124.09 lat (msec): min=215, max=390, avg=303.02, stdev=124.05 clat percentiles (msec): | 1.00th=[ 215], 5.00th=[ 215], 10.00th=[ 215], 20.00th=[ 215], | 30.00th=[ 215], 40.00th=[ 215], 50.00th=[ 215], 60.00th=[ 393], | 70.00th=[ 393], 80.00th=[ 393], 90.00th=[ 393], 95.00th=[ 393], | 99.00th=[ 393], 99.50th=[ 393], 99.90th=[ 393], 99.95th=[ 393], | 99.99th=[ 393] lat (msec) : 250=33.33%, 500=66.67% cpu : usr=0.11%, sys=15.54%, ctx=76, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28911: Tue Mar 25 09:53:40 2025 write: IOPS=3, BW=13.4MiB/s (14.1MB/s)(12.0MiB/894msec) clat (msec): min=46, max=492, avg=281.19, stdev=223.87 lat (msec): min=46, max=492, avg=281.50, stdev=223.85 clat percentiles (msec): | 1.00th=[ 47], 5.00th=[ 47], 10.00th=[ 47], 20.00th=[ 47], | 30.00th=[ 47], 40.00th=[ 305], 50.00th=[ 305], 60.00th=[ 305], | 70.00th=[ 493], 80.00th=[ 493], 90.00th=[ 493], 95.00th=[ 493], | 99.00th=[ 493], 99.50th=[ 493], 99.90th=[ 493], 99.95th=[ 493], | 99.99th=[ 493] bw ( KiB/s): min= 8192, max= 8192, per=22.88%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 50=33.33%, 500=66.67% cpu : usr=0.11%, sys=13.10%, ctx=187, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28912: Tue Mar 25 09:53:40 2025 write: IOPS=4, BW=18.1MiB/s (19.0MB/s)(12.0MiB/662msec) clat (msec): min=138, max=265, avg=199.21, stdev=63.66 lat (msec): min=138, max=265, avg=199.51, stdev=63.66 clat percentiles (msec): | 1.00th=[ 138], 5.00th=[ 138], 10.00th=[ 138], 20.00th=[ 138], | 30.00th=[ 138], 40.00th=[ 194], 50.00th=[ 194], 60.00th=[ 194], | 70.00th=[ 266], 80.00th=[ 266], 90.00th=[ 266], 95.00th=[ 266], | 99.00th=[ 266], 99.50th=[ 266], 99.90th=[ 266], 99.95th=[ 266], | 99.99th=[ 266] bw ( KiB/s): min= 8192, max= 8192, per=22.88%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.15%, sys=26.48%, ctx=152, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=17.5MiB/s (18.3MB/s), 4477KiB/s-16.3MiB/s (4584kB/s-17.1MB/s), io=16.0MiB (16.8MB), run=734-915msec WRITE: bw=34.0MiB/s (36.7MB/s), 8953KiB/s-18.1MiB/s (9168kB/s-19.0MB/s), io=32.0MiB (33.6MB), run=662-915msec