== sanity test 398b: DIO and buffer IO race ============== 03:38:03 (1746085083) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.53148 s, 32.9 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=25686: Thu May 1 03:38:26 2025 read: IOPS=75, BW=301KiB/s (309kB/s)(5992KiB/19877msec) clat (usec): min=47, max=223710, avg=7247.73, stdev=7290.99 lat (usec): min=48, max=223711, avg=7251.07, stdev=7295.52 clat percentiles (usec): | 1.00th=[ 88], 5.00th=[ 128], 10.00th=[ 2900], 20.00th=[ 3720], | 30.00th=[ 4490], 40.00th=[ 5342], 50.00th=[ 6259], 60.00th=[ 7242], | 70.00th=[ 8455], 80.00th=[ 10028], 90.00th=[ 12387], 95.00th=[ 14877], | 99.00th=[ 23462], 99.50th=[ 30802], 99.90th=[ 46924], 99.95th=[223347], | 99.99th=[223347] bw ( KiB/s): min= 168, max= 432, per=24.88%, avg=296.51, stdev=61.01, samples=39 iops : min= 42, max= 108, avg=74.05, stdev=15.31, samples=39 write: IOPS=79, BW=317KiB/s (324kB/s)(6296KiB/19877msec) clat (usec): min=445, max=33143, avg=5675.83, stdev=5536.07 lat (usec): min=446, max=33144, avg=5677.05, stdev=5536.33 clat percentiles (usec): | 1.00th=[ 562], 5.00th=[ 701], 10.00th=[ 791], 20.00th=[ 963], | 30.00th=[ 1188], 40.00th=[ 2180], 50.00th=[ 4146], 60.00th=[ 5604], | 70.00th=[ 7308], 80.00th=[ 9765], 90.00th=[13698], 95.00th=[16712], | 99.00th=[22938], 99.50th=[26608], 99.90th=[30802], 99.95th=[33162], | 99.99th=[33162] bw ( KiB/s): min= 176, max= 464, per=25.02%, avg=312.31, stdev=60.74, samples=39 iops : min= 44, max= 116, avg=78.00, stdev=15.18, samples=39 lat (usec) : 50=0.03%, 100=1.27%, 250=1.73%, 500=0.46%, 750=3.74% lat (usec) : 1000=7.29% lat (msec) : 2=8.85%, 4=12.83%, 10=44.30%, 20=17.45%, 50=2.02% lat (msec) : 250=0.03% cpu : usr=0.28%, sys=19.56%, ctx=5638, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25688: Thu May 1 03:38:26 2025 read: IOPS=73, BW=295KiB/s (302kB/s)(5940KiB/20112msec) clat (usec): min=55, max=213424, avg=7392.54, stdev=7140.19 lat (usec): min=55, max=213427, avg=7393.32, stdev=7140.23 clat percentiles (usec): | 1.00th=[ 89], 5.00th=[ 153], 10.00th=[ 3032], 20.00th=[ 3818], | 30.00th=[ 4555], 40.00th=[ 5407], 50.00th=[ 6259], 60.00th=[ 7373], | 70.00th=[ 8717], 80.00th=[ 10421], 90.00th=[ 12649], 95.00th=[ 15139], | 99.00th=[ 24511], 99.50th=[ 28967], 99.90th=[ 48497], 99.95th=[212861], | 99.99th=[212861] bw ( KiB/s): min= 80, max= 447, per=24.63%, avg=293.58, stdev=67.94, samples=40 iops : min= 20, max= 111, avg=73.23, stdev=16.96, samples=40 write: IOPS=78, BW=316KiB/s (323kB/s)(6348KiB/20112msec) clat (usec): min=436, max=28808, avg=5691.99, stdev=5242.78 lat (usec): min=437, max=28809, avg=5693.47, stdev=5242.87 clat percentiles (usec): | 1.00th=[ 586], 5.00th=[ 717], 10.00th=[ 807], 20.00th=[ 979], | 30.00th=[ 1254], 40.00th=[ 2474], 50.00th=[ 4424], 60.00th=[ 5997], | 70.00th=[ 7439], 80.00th=[ 9634], 90.00th=[13566], 95.00th=[16188], | 99.00th=[21627], 99.50th=[26084], 99.90th=[27919], 99.95th=[28705], | 99.99th=[28705] bw ( KiB/s): min= 136, max= 566, per=25.03%, avg=312.35, stdev=72.23, samples=40 iops : min= 34, max= 141, avg=77.93, stdev=18.05, samples=40 lat (usec) : 100=0.91%, 250=1.63%, 500=0.29%, 750=3.26%, 1000=7.68% lat (msec) : 2=8.89%, 4=12.66%, 10=44.37%, 20=18.59%, 50=1.69% lat (msec) : 250=0.03% cpu : usr=0.25%, sys=19.71%, ctx=5696, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25690: Thu May 1 03:38:26 2025 read: IOPS=74, BW=299KiB/s (307kB/s)(6028KiB/20137msec) clat (usec): min=53, max=221755, avg=7485.54, stdev=7174.29 lat (usec): min=53, max=221756, avg=7488.29, stdev=7175.41 clat percentiles (usec): | 1.00th=[ 93], 5.00th=[ 293], 10.00th=[ 3097], 20.00th=[ 3949], | 30.00th=[ 4686], 40.00th=[ 5538], 50.00th=[ 6456], 60.00th=[ 7570], | 70.00th=[ 8848], 80.00th=[ 10290], 90.00th=[ 13042], 95.00th=[ 16057], | 99.00th=[ 23725], 99.50th=[ 27132], 99.90th=[ 35914], 99.95th=[221250], | 99.99th=[221250] bw ( KiB/s): min= 184, max= 464, per=24.99%, avg=297.85, stdev=52.76, samples=40 iops : min= 46, max= 116, avg=74.35, stdev=13.20, samples=40 write: IOPS=77, BW=311KiB/s (318kB/s)(6260KiB/20137msec) clat (usec): min=394, max=43306, avg=5604.07, stdev=5335.59 lat (usec): min=394, max=43306, avg=5605.53, stdev=5335.70 clat percentiles (usec): | 1.00th=[ 553], 5.00th=[ 709], 10.00th=[ 816], 20.00th=[ 1004], | 30.00th=[ 1287], 40.00th=[ 2278], 50.00th=[ 4228], 60.00th=[ 5604], | 70.00th=[ 7242], 80.00th=[ 9765], 90.00th=[13304], 95.00th=[16188], | 99.00th=[22152], 99.50th=[24773], 99.90th=[33817], 99.95th=[43254], | 99.99th=[43254] bw ( KiB/s): min= 192, max= 472, per=24.73%, avg=308.65, stdev=67.96, samples=40 iops : min= 48, max= 118, avg=77.05, stdev=17.02, samples=40 lat (usec) : 100=0.91%, 250=1.46%, 500=0.52%, 750=3.32%, 1000=6.74% lat (msec) : 2=9.64%, 4=12.34%, 10=44.95%, 20=18.16%, 50=1.92% lat (msec) : 250=0.03% cpu : usr=0.24%, sys=19.85%, ctx=5823, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25706: Thu May 1 03:38:26 2025 read: IOPS=75, BW=302KiB/s (309kB/s)(6052KiB/20037msec) clat (usec): min=57, max=219662, avg=7472.84, stdev=7314.02 lat (usec): min=58, max=219663, avg=7474.53, stdev=7314.61 clat percentiles (usec): | 1.00th=[ 87], 5.00th=[ 176], 10.00th=[ 3163], 20.00th=[ 3982], | 30.00th=[ 4686], 40.00th=[ 5473], 50.00th=[ 6259], 60.00th=[ 7439], | 70.00th=[ 8586], 80.00th=[ 10421], 90.00th=[ 12649], 95.00th=[ 16188], | 99.00th=[ 22938], 99.50th=[ 26870], 99.90th=[ 73925], 99.95th=[219153], | 99.99th=[219153] bw ( KiB/s): min= 200, max= 496, per=25.25%, avg=301.03, stdev=62.38, samples=40 iops : min= 50, max= 124, avg=75.12, stdev=15.60, samples=40 write: IOPS=77, BW=311KiB/s (319kB/s)(6236KiB/20037msec) clat (usec): min=446, max=38395, avg=5547.63, stdev=5239.82 lat (usec): min=447, max=38396, avg=5548.61, stdev=5239.82 clat percentiles (usec): | 1.00th=[ 594], 5.00th=[ 717], 10.00th=[ 807], 20.00th=[ 996], | 30.00th=[ 1254], 40.00th=[ 2311], 50.00th=[ 4178], 60.00th=[ 5604], | 70.00th=[ 7373], 80.00th=[ 9634], 90.00th=[12780], 95.00th=[15401], | 99.00th=[21890], 99.50th=[24511], 99.90th=[35390], 99.95th=[38536], | 99.99th=[38536] bw ( KiB/s): min= 199, max= 528, per=24.81%, avg=309.62, stdev=72.42, samples=40 iops : min= 49, max= 132, avg=77.28, stdev=18.07, samples=40 lat (usec) : 100=0.98%, 250=1.60%, 500=0.26%, 750=3.29%, 1000=6.93% lat (msec) : 2=9.28%, 4=12.47%, 10=45.08%, 20=17.90%, 50=2.15% lat (msec) : 100=0.03%, 250=0.03% cpu : usr=0.23%, sys=19.90%, ctx=5956, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1192KiB/s (1221kB/s), 295KiB/s-302KiB/s (302kB/s-309kB/s), io=23.4MiB (24.6MB), run=19877-20137msec WRITE: bw=1248KiB/s (1278kB/s), 311KiB/s-317KiB/s (318kB/s-324kB/s), io=24.6MiB (25.7MB), run=19877-20137msec rand-rw: (groupid=0, jobs=1): err= 0: pid=25685: Thu May 1 03:38:31 2025 read: IOPS=58, BW=236KiB/s (241kB/s)(5992KiB/25436msec) clat (usec): min=1693, max=226101, avg=6561.28, stdev=8112.41 lat (usec): min=1694, max=226102, avg=6562.56, stdev=8112.29 clat percentiles (usec): | 1.00th=[ 1909], 5.00th=[ 2147], 10.00th=[ 2343], 20.00th=[ 2638], | 30.00th=[ 2966], 40.00th=[ 3392], 50.00th=[ 3982], 60.00th=[ 5014], | 70.00th=[ 6456], 80.00th=[ 8979], 90.00th=[ 14746], 95.00th=[ 19268], | 99.00th=[ 28443], 99.50th=[ 32637], 99.90th=[ 41681], 99.95th=[225444], | 99.99th=[225444] bw ( KiB/s): min= 80, max= 624, per=24.47%, avg=230.28, stdev=146.79, samples=50 iops : min= 20, max= 156, avg=57.48, stdev=36.74, samples=50 write: IOPS=61, BW=248KiB/s (253kB/s)(6296KiB/25436msec) clat (usec): min=2506, max=68032, avg=9865.26, stdev=7213.08 lat (usec): min=2507, max=68033, avg=9866.62, stdev=7213.27 clat percentiles (usec): | 1.00th=[ 2868], 5.00th=[ 3392], 10.00th=[ 3720], 20.00th=[ 4293], | 30.00th=[ 4817], 40.00th=[ 5604], 50.00th=[ 6980], 60.00th=[ 9241], | 70.00th=[11863], 80.00th=[15139], 90.00th=[20055], 95.00th=[24249], | 99.00th=[33424], 99.50th=[39060], 99.90th=[48497], 99.95th=[67634], | 99.99th=[67634] bw ( KiB/s): min= 111, max= 648, per=24.47%, avg=241.00, stdev=149.06, samples=50 iops : min= 27, max= 162, avg=60.16, stdev=37.30, samples=50 lat (msec) : 2=0.91%, 4=31.28%, 10=40.62%, 20=19.73%, 50=7.39% lat (msec) : 100=0.03%, 250=0.03% cpu : usr=0.19%, sys=9.65%, ctx=3756, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25687: Thu May 1 03:38:31 2025 read: IOPS=58, BW=233KiB/s (239kB/s)(5940KiB/25441msec) clat (usec): min=1725, max=223026, avg=6595.43, stdev=8250.70 lat (usec): min=1726, max=223029, avg=6596.47, stdev=8250.90 clat percentiles (usec): | 1.00th=[ 1876], 5.00th=[ 2089], 10.00th=[ 2278], 20.00th=[ 2606], | 30.00th=[ 2868], 40.00th=[ 3326], 50.00th=[ 3916], 60.00th=[ 4883], | 70.00th=[ 6325], 80.00th=[ 9110], 90.00th=[ 14484], 95.00th=[ 20317], | 99.00th=[ 30278], 99.50th=[ 33162], 99.90th=[ 48497], 99.95th=[223347], | 99.99th=[223347] bw ( KiB/s): min= 80, max= 648, per=24.36%, avg=229.24, stdev=153.68, samples=50 iops : min= 20, max= 162, avg=57.20, stdev=38.42, samples=50 write: IOPS=62, BW=250KiB/s (256kB/s)(6348KiB/25441msec) clat (usec): min=2541, max=71336, avg=9810.67, stdev=7199.95 lat (usec): min=2541, max=71337, avg=9811.92, stdev=7199.85 clat percentiles (usec): | 1.00th=[ 2999], 5.00th=[ 3458], 10.00th=[ 3752], 20.00th=[ 4359], | 30.00th=[ 4817], 40.00th=[ 5538], 50.00th=[ 6849], 60.00th=[ 8979], | 70.00th=[11863], 80.00th=[15270], 90.00th=[19792], 95.00th=[23725], | 99.00th=[33424], 99.50th=[37487], 99.90th=[49546], 99.95th=[71828], | 99.99th=[71828] bw ( KiB/s): min= 104, max= 598, per=24.57%, avg=242.04, stdev=147.32, samples=50 iops : min= 26, max= 149, avg=60.40, stdev=36.83, samples=50 lat (msec) : 2=1.40%, 4=30.37%, 10=40.72%, 20=20.05%, 50=7.39% lat (msec) : 100=0.03%, 250=0.03% cpu : usr=0.17%, sys=9.71%, ctx=3729, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25689: Thu May 1 03:38:31 2025 read: IOPS=59, BW=238KiB/s (244kB/s)(6028KiB/25343msec) clat (usec): min=1726, max=44310, avg=6599.70, stdev=5746.38 lat (usec): min=1727, max=44311, avg=6600.53, stdev=5746.39 clat percentiles (usec): | 1.00th=[ 1926], 5.00th=[ 2180], 10.00th=[ 2343], 20.00th=[ 2671], | 30.00th=[ 2999], 40.00th=[ 3490], 50.00th=[ 4228], 60.00th=[ 5276], | 70.00th=[ 6980], 80.00th=[ 9765], 90.00th=[14353], 95.00th=[19268], | 99.00th=[27657], 99.50th=[31065], 99.90th=[37487], 99.95th=[44303], | 99.99th=[44303] bw ( KiB/s): min= 80, max= 648, per=24.82%, avg=233.56, stdev=143.33, samples=50 iops : min= 20, max= 162, avg=58.28, stdev=35.82, samples=50 write: IOPS=61, BW=247KiB/s (253kB/s)(6260KiB/25343msec) clat (msec): min=2, max=225, avg= 9.78, stdev= 8.87 lat (msec): min=2, max=225, avg= 9.79, stdev= 8.87 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 4], 10.00th=[ 4], 20.00th=[ 5], | 30.00th=[ 5], 40.00th=[ 6], 50.00th=[ 7], 60.00th=[ 9], | 70.00th=[ 12], 80.00th=[ 15], 90.00th=[ 20], 95.00th=[ 24], | 99.00th=[ 35], 99.50th=[ 39], 99.90th=[ 49], 99.95th=[ 226], | 99.99th=[ 226] bw ( KiB/s): min= 104, max= 632, per=24.51%, avg=241.42, stdev=153.00, samples=50 iops : min= 26, max= 158, avg=60.24, stdev=38.25, samples=50 lat (msec) : 2=0.78%, 4=30.24%, 10=41.50%, 20=20.80%, 50=6.64% lat (msec) : 250=0.03% cpu : usr=0.17%, sys=9.76%, ctx=3651, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25691: Thu May 1 03:38:31 2025 read: IOPS=59, BW=237KiB/s (243kB/s)(6052KiB/25510msec) clat (usec): min=1643, max=48761, avg=6642.22, stdev=6090.25 lat (usec): min=1644, max=48762, avg=6643.01, stdev=6090.27 clat percentiles (usec): | 1.00th=[ 1876], 5.00th=[ 2147], 10.00th=[ 2311], 20.00th=[ 2606], | 30.00th=[ 2933], 40.00th=[ 3425], 50.00th=[ 4080], 60.00th=[ 5145], | 70.00th=[ 6718], 80.00th=[ 9634], 90.00th=[15139], 95.00th=[20055], | 99.00th=[29230], 99.50th=[34341], 99.90th=[38536], 99.95th=[49021], | 99.99th=[49021] bw ( KiB/s): min= 88, max= 624, per=25.19%, avg=237.02, stdev=157.44, samples=51 iops : min= 22, max= 156, avg=59.16, stdev=39.38, samples=51 write: IOPS=61, BW=244KiB/s (250kB/s)(6236KiB/25510msec) clat (msec): min=2, max=225, avg= 9.86, stdev= 9.15 lat (msec): min=2, max=225, avg= 9.86, stdev= 9.15 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 4], 10.00th=[ 4], 20.00th=[ 5], | 30.00th=[ 5], 40.00th=[ 6], 50.00th=[ 7], 60.00th=[ 9], | 70.00th=[ 12], 80.00th=[ 15], 90.00th=[ 21], 95.00th=[ 26], | 99.00th=[ 37], 99.50th=[ 41], 99.90th=[ 50], 99.95th=[ 226], | 99.99th=[ 226] bw ( KiB/s): min= 104, max= 632, per=24.78%, avg=244.08, stdev=157.42, samples=51 iops : min= 26, max= 158, avg=60.92, stdev=39.37, samples=51 lat (msec) : 2=1.30%, 4=30.47%, 10=40.82%, 20=19.86%, 50=7.52% lat (msec) : 250=0.03% cpu : usr=0.18%, sys=9.76%, ctx=3774, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=941KiB/s (964kB/s), 233KiB/s-238KiB/s (239kB/s-244kB/s), io=23.4MiB (24.6MB), run=25343-25510msec WRITE: bw=985KiB/s (1009kB/s), 244KiB/s-250KiB/s (250kB/s-256kB/s), io=24.6MiB (25.7MB), run=25343-25510msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=25742: Thu May 1 03:38:38 2025 read: IOPS=72, BW=1163KiB/s (1191kB/s)(5920KiB/5091msec) clat (usec): min=61, max=31110, avg=6854.99, stdev=4725.35 lat (usec): min=62, max=31110, avg=6855.72, stdev=4725.37 clat percentiles (usec): | 1.00th=[ 78], 5.00th=[ 109], 10.00th=[ 135], 20.00th=[ 3949], | 30.00th=[ 4817], 40.00th=[ 5604], 50.00th=[ 6194], 60.00th=[ 7111], | 70.00th=[ 8094], 80.00th=[ 9503], 90.00th=[11469], 95.00th=[15401], | 99.00th=[23462], 99.50th=[30802], 99.90th=[31065], 99.95th=[31065], | 99.99th=[31065] bw ( KiB/s): min= 736, max= 1532, per=25.45%, avg=1148.40, stdev=233.18, samples=10 iops : min= 46, max= 95, avg=71.70, stdev=14.44, samples=10 write: IOPS=78, BW=1251KiB/s (1281kB/s)(6368KiB/5091msec) clat (usec): min=830, max=31124, avg=6244.80, stdev=5483.50 lat (usec): min=830, max=31124, avg=6246.00, stdev=5483.52 clat percentiles (usec): | 1.00th=[ 865], 5.00th=[ 1090], 10.00th=[ 1221], 20.00th=[ 1500], | 30.00th=[ 1942], 40.00th=[ 3359], 50.00th=[ 4621], 60.00th=[ 6194], | 70.00th=[ 7963], 80.00th=[10159], 90.00th=[14222], 95.00th=[16909], | 99.00th=[24249], 99.50th=[24773], 99.90th=[31065], 99.95th=[31065], | 99.99th=[31065] bw ( KiB/s): min= 1088, max= 1504, per=26.35%, avg=1250.90, stdev=140.93, samples=10 iops : min= 68, max= 94, avg=78.10, stdev= 8.77, samples=10 lat (usec) : 100=1.43%, 250=4.04%, 500=0.65%, 750=0.13%, 1000=1.30% lat (msec) : 2=15.49%, 4=9.77%, 10=48.57%, 20=15.76%, 50=2.86% cpu : usr=0.16%, sys=22.08%, ctx=1582, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25744: Thu May 1 03:38:38 2025 read: IOPS=67, BW=1082KiB/s (1108kB/s)(5744KiB/5308msec) clat (usec): min=62, max=26169, avg=7247.42, stdev=4699.97 lat (usec): min=62, max=26170, avg=7248.19, stdev=4699.99 clat percentiles (usec): | 1.00th=[ 76], 5.00th=[ 106], 10.00th=[ 245], 20.00th=[ 3982], | 30.00th=[ 4621], 40.00th=[ 5604], 50.00th=[ 6652], 60.00th=[ 7898], | 70.00th=[ 8848], 80.00th=[10290], 90.00th=[13173], 95.00th=[15664], | 99.00th=[23725], 99.50th=[25560], 99.90th=[26084], 99.95th=[26084], | 99.99th=[26084] bw ( KiB/s): min= 544, max= 1405, per=23.54%, avg=1062.10, stdev=237.46, samples=10 iops : min= 34, max= 87, avg=66.30, stdev=14.71, samples=10 write: IOPS=77, BW=1233KiB/s (1262kB/s)(6544KiB/5308msec) clat (usec): min=705, max=56500, avg=6413.89, stdev=5918.26 lat (usec): min=706, max=56501, avg=6415.68, stdev=5919.89 clat percentiles (usec): | 1.00th=[ 865], 5.00th=[ 1090], 10.00th=[ 1221], 20.00th=[ 1516], | 30.00th=[ 2343], 40.00th=[ 4113], 50.00th=[ 5014], 60.00th=[ 6456], | 70.00th=[ 7701], 80.00th=[ 9896], 90.00th=[13829], 95.00th=[17171], | 99.00th=[25035], 99.50th=[30016], 99.90th=[56361], 99.95th=[56361], | 99.99th=[56361] bw ( KiB/s): min= 864, max= 1664, per=25.07%, avg=1190.10, stdev=264.00, samples=10 iops : min= 54, max= 104, avg=74.30, stdev=16.56, samples=10 lat (usec) : 100=1.43%, 250=3.26%, 500=0.13%, 750=0.39%, 1000=1.30% lat (msec) : 2=13.54%, 4=10.68%, 10=48.83%, 20=18.10%, 50=2.21% lat (msec) : 100=0.13% cpu : usr=0.19%, sys=21.69%, ctx=1716, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25745: Thu May 1 03:38:38 2025 read: IOPS=75, BW=1203KiB/s (1232kB/s)(6256KiB/5200msec) clat (usec): min=71, max=30689, avg=7368.65, stdev=4563.87 lat (usec): min=72, max=30702, avg=7369.46, stdev=4564.07 clat percentiles (usec): | 1.00th=[ 79], 5.00th=[ 105], 10.00th=[ 149], 20.00th=[ 4015], | 30.00th=[ 4948], 40.00th=[ 6063], 50.00th=[ 6849], 60.00th=[ 8029], | 70.00th=[ 8979], 80.00th=[10683], 90.00th=[12911], 95.00th=[14746], | 99.00th=[21890], 99.50th=[25822], 99.90th=[30802], 99.95th=[30802], | 99.99th=[30802] bw ( KiB/s): min= 832, max= 1408, per=26.59%, avg=1199.70, stdev=178.96, samples=10 iops : min= 52, max= 88, avg=74.90, stdev=11.10, samples=10 write: IOPS=72, BW=1160KiB/s (1188kB/s)(6032KiB/5200msec) clat (usec): min=764, max=34665, avg=6096.61, stdev=5417.01 lat (usec): min=765, max=34666, avg=6097.78, stdev=5417.06 clat percentiles (usec): | 1.00th=[ 898], 5.00th=[ 1074], 10.00th=[ 1188], 20.00th=[ 1385], | 30.00th=[ 1811], 40.00th=[ 3032], 50.00th=[ 4555], 60.00th=[ 5997], | 70.00th=[ 7898], 80.00th=[10290], 90.00th=[13173], 95.00th=[17433], | 99.00th=[21627], 99.50th=[26346], 99.90th=[34866], 99.95th=[34866], | 99.99th=[34866] bw ( KiB/s): min= 704, max= 1596, per=23.79%, avg=1129.20, stdev=283.11, samples=10 iops : min= 44, max= 99, avg=70.50, stdev=17.56, samples=10 lat (usec) : 100=1.56%, 250=3.78%, 500=0.26%, 1000=1.30% lat (msec) : 2=14.71%, 4=10.68%, 10=44.66%, 20=21.35%, 50=1.69% cpu : usr=0.21%, sys=21.43%, ctx=1665, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25746: Thu May 1 03:38:38 2025 read: IOPS=73, BW=1171KiB/s (1200kB/s)(6032KiB/5149msec) clat (usec): min=65, max=52464, avg=7272.55, stdev=5011.22 lat (usec): min=65, max=52465, avg=7273.33, stdev=5011.29 clat percentiles (usec): | 1.00th=[ 92], 5.00th=[ 118], 10.00th=[ 347], 20.00th=[ 4015], | 30.00th=[ 4948], 40.00th=[ 5604], 50.00th=[ 6718], 60.00th=[ 7635], | 70.00th=[ 8586], 80.00th=[10290], 90.00th=[13304], 95.00th=[14484], | 99.00th=[23462], 99.50th=[29230], 99.90th=[52691], 99.95th=[52691], | 99.99th=[52691] bw ( KiB/s): min= 862, max= 1472, per=25.51%, avg=1150.90, stdev=185.39, samples=10 iops : min= 53, max= 92, avg=71.60, stdev=11.82, samples=10 write: IOPS=75, BW=1215KiB/s (1244kB/s)(6256KiB/5149msec) clat (usec): min=728, max=40195, avg=5936.01, stdev=5390.60 lat (usec): min=729, max=40197, avg=5937.86, stdev=5390.22 clat percentiles (usec): | 1.00th=[ 824], 5.00th=[ 1106], 10.00th=[ 1205], 20.00th=[ 1434], | 30.00th=[ 1762], 40.00th=[ 3064], 50.00th=[ 4178], 60.00th=[ 5669], | 70.00th=[ 7308], 80.00th=[10290], 90.00th=[13304], 95.00th=[16581], | 99.00th=[22414], 99.50th=[30802], 99.90th=[40109], 99.95th=[40109], | 99.99th=[40109] bw ( KiB/s): min= 958, max= 1756, per=25.39%, avg=1205.20, stdev=234.78, samples=10 iops : min= 59, max= 109, avg=75.00, stdev=14.57, samples=10 lat (usec) : 100=0.91%, 250=3.78%, 500=0.26%, 750=0.13%, 1000=1.17% lat (msec) : 2=16.28%, 4=11.59%, 10=44.53%, 20=19.66%, 50=1.56% lat (msec) : 100=0.13% cpu : usr=0.12%, sys=22.14%, ctx=1673, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=4512KiB/s (4621kB/s), 1082KiB/s-1203KiB/s (1108kB/s-1232kB/s), io=23.4MiB (24.5MB), run=5091-5308msec WRITE: bw=4748KiB/s (4861kB/s), 1160KiB/s-1251KiB/s (1188kB/s-1281kB/s), io=24.6MiB (25.8MB), run=5091-5308msec rand-rw: (groupid=0, jobs=1): err= 0: pid=25739: Thu May 1 03:38:39 2025 read: IOPS=56, BW=901KiB/s (922kB/s)(5920KiB/6574msec) clat (usec): min=1848, max=39310, avg=6706.07, stdev=5726.19 lat (usec): min=1849, max=39311, avg=6706.88, stdev=5726.22 clat percentiles (usec): | 1.00th=[ 2147], 5.00th=[ 2311], 10.00th=[ 2507], 20.00th=[ 2868], | 30.00th=[ 3163], 40.00th=[ 3654], 50.00th=[ 4424], 60.00th=[ 5407], | 70.00th=[ 6783], 80.00th=[ 9372], 90.00th=[14615], 95.00th=[20317], | 99.00th=[26870], 99.50th=[31327], 99.90th=[39060], 99.95th=[39060], | 99.99th=[39060] bw ( KiB/s): min= 352, max= 1980, per=24.26%, avg=880.77, stdev=515.91, samples=13 iops : min= 22, max= 123, avg=54.92, stdev=32.13, samples=13 write: IOPS=60, BW=969KiB/s (992kB/s)(6368KiB/6574msec) clat (usec): min=2857, max=84347, avg=10149.27, stdev=7816.33 lat (usec): min=2859, max=84349, avg=10151.59, stdev=7816.81 clat percentiles (usec): | 1.00th=[ 3097], 5.00th=[ 3720], 10.00th=[ 4113], 20.00th=[ 4555], | 30.00th=[ 5211], 40.00th=[ 5866], 50.00th=[ 7439], 60.00th=[ 9372], | 70.00th=[11863], 80.00th=[15139], 90.00th=[19530], 95.00th=[25035], | 99.00th=[35390], 99.50th=[41157], 99.90th=[84411], 99.95th=[84411], | 99.99th=[84411] bw ( KiB/s): min= 415, max= 2144, per=25.13%, avg=959.62, stdev=537.62, samples=13 iops : min= 25, max= 134, avg=59.85, stdev=33.56, samples=13 lat (msec) : 2=0.13%, 4=25.91%, 10=45.96%, 20=20.70%, 50=7.16% lat (msec) : 100=0.13% cpu : usr=0.26%, sys=10.19%, ctx=946, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25740: Thu May 1 03:38:39 2025 read: IOPS=54, BW=871KiB/s (892kB/s)(5744KiB/6597msec) clat (usec): min=1997, max=45954, avg=7058.00, stdev=6136.32 lat (usec): min=1998, max=45954, avg=7059.40, stdev=6136.05 clat percentiles (usec): | 1.00th=[ 2073], 5.00th=[ 2311], 10.00th=[ 2573], 20.00th=[ 2868], | 30.00th=[ 3163], 40.00th=[ 3752], 50.00th=[ 4555], 60.00th=[ 5538], | 70.00th=[ 6915], 80.00th=[10814], 90.00th=[16909], 95.00th=[20841], | 99.00th=[27657], 99.50th=[29230], 99.90th=[45876], 99.95th=[45876], | 99.99th=[45876] bw ( KiB/s): min= 352, max= 2144, per=23.72%, avg=860.92, stdev=530.24, samples=13 iops : min= 22, max= 134, avg=53.62, stdev=33.07, samples=13 write: IOPS=61, BW=992KiB/s (1016kB/s)(6544KiB/6597msec) clat (usec): min=2740, max=41446, avg=9795.24, stdev=6929.13 lat (usec): min=2741, max=41448, avg=9796.70, stdev=6929.20 clat percentiles (usec): | 1.00th=[ 3163], 5.00th=[ 3621], 10.00th=[ 4015], 20.00th=[ 4621], | 30.00th=[ 5080], 40.00th=[ 5735], 50.00th=[ 6980], 60.00th=[ 9110], | 70.00th=[11469], 80.00th=[14091], 90.00th=[19268], 95.00th=[24773], | 99.00th=[34866], 99.50th=[36963], 99.90th=[41681], 99.95th=[41681], | 99.99th=[41681] bw ( KiB/s): min= 416, max= 2331, per=25.38%, avg=969.15, stdev=571.80, samples=13 iops : min= 26, max= 145, avg=60.38, stdev=35.67, samples=13 lat (msec) : 2=0.13%, 4=24.87%, 10=45.83%, 20=21.74%, 50=7.42% cpu : usr=0.14%, sys=9.99%, ctx=951, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25741: Thu May 1 03:38:39 2025 read: IOPS=59, BW=953KiB/s (976kB/s)(6256KiB/6563msec) clat (usec): min=1980, max=37229, avg=6685.84, stdev=5703.33 lat (usec): min=1980, max=37230, avg=6686.57, stdev=5703.35 clat percentiles (usec): | 1.00th=[ 2114], 5.00th=[ 2376], 10.00th=[ 2573], 20.00th=[ 2835], | 30.00th=[ 3163], 40.00th=[ 3523], 50.00th=[ 4228], 60.00th=[ 5473], | 70.00th=[ 7308], 80.00th=[ 9110], 90.00th=[15139], 95.00th=[18220], | 99.00th=[27657], 99.50th=[36963], 99.90th=[37487], 99.95th=[37487], | 99.99th=[37487] bw ( KiB/s): min= 448, max= 2400, per=26.10%, avg=947.38, stdev=585.02, samples=13 iops : min= 28, max= 150, avg=59.08, stdev=36.62, samples=13 write: IOPS=57, BW=919KiB/s (941kB/s)(6032KiB/6563msec) clat (usec): min=2627, max=84075, avg=10431.24, stdev=7812.93 lat (usec): min=2628, max=84077, avg=10432.75, stdev=7813.17 clat percentiles (usec): | 1.00th=[ 2966], 5.00th=[ 3589], 10.00th=[ 4047], 20.00th=[ 4686], | 30.00th=[ 5407], 40.00th=[ 6259], 50.00th=[ 7963], 60.00th=[ 9896], | 70.00th=[12125], 80.00th=[15926], 90.00th=[20317], 95.00th=[24773], | 99.00th=[34341], 99.50th=[39060], 99.90th=[84411], 99.95th=[84411], | 99.99th=[84411] bw ( KiB/s): min= 352, max= 2048, per=23.65%, avg=903.08, stdev=499.70, samples=13 iops : min= 22, max= 128, avg=56.31, stdev=31.30, samples=13 lat (msec) : 2=0.13%, 4=28.26%, 10=43.62%, 20=20.70%, 50=7.16% lat (msec) : 100=0.13% cpu : usr=0.15%, sys=10.04%, ctx=943, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25743: Thu May 1 03:38:39 2025 read: IOPS=57, BW=919KiB/s (941kB/s)(6032KiB/6561msec) clat (usec): min=1857, max=39092, avg=6605.87, stdev=5908.84 lat (usec): min=1858, max=39093, avg=6606.62, stdev=5908.87 clat percentiles (usec): | 1.00th=[ 2114], 5.00th=[ 2311], 10.00th=[ 2573], 20.00th=[ 2900], | 30.00th=[ 3228], 40.00th=[ 3687], 50.00th=[ 4359], 60.00th=[ 4883], | 70.00th=[ 5997], 80.00th=[ 9241], 90.00th=[14746], 95.00th=[19268], | 99.00th=[31851], 99.50th=[37487], 99.90th=[39060], 99.95th=[39060], | 99.99th=[39060] bw ( KiB/s): min= 416, max= 2235, per=25.00%, avg=907.62, stdev=526.13, samples=13 iops : min= 26, max= 139, avg=56.54, stdev=32.81, samples=13 write: IOPS=59, BW=954KiB/s (976kB/s)(6256KiB/6561msec) clat (usec): min=2773, max=47395, avg=10248.03, stdev=7179.30 lat (usec): min=2774, max=47397, avg=10251.21, stdev=7181.04 clat percentiles (usec): | 1.00th=[ 2999], 5.00th=[ 3687], 10.00th=[ 4047], 20.00th=[ 4555], | 30.00th=[ 5276], 40.00th=[ 6128], 50.00th=[ 7898], 60.00th=[ 9896], | 70.00th=[11338], 80.00th=[15795], 90.00th=[20317], 95.00th=[23725], | 99.00th=[36439], 99.50th=[44303], 99.90th=[47449], 99.95th=[47449], | 99.99th=[47449] bw ( KiB/s): min= 576, max= 2112, per=24.67%, avg=942.15, stdev=526.83, samples=13 iops : min= 36, max= 132, avg=58.69, stdev=32.85, samples=13 lat (msec) : 2=0.26%, 4=26.69%, 10=44.92%, 20=20.05%, 50=8.07% cpu : usr=0.11%, sys=10.06%, ctx=954, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=3631KiB/s (3718kB/s), 871KiB/s-953KiB/s (892kB/s-976kB/s), io=23.4MiB (24.5MB), run=6561-6597msec WRITE: bw=3820KiB/s (3912kB/s), 919KiB/s-992KiB/s (941kB/s-1016kB/s), io=24.6MiB (25.8MB), run=6561-6597msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=25757: Thu May 1 03:38:41 2025 read: IOPS=8, BW=8374KiB/s (8575kB/s)(7168KiB/856msec) clat (usec): min=423, max=405806, avg=84657.54, stdev=143289.35 lat (usec): min=424, max=405807, avg=84658.74, stdev=143289.29 clat percentiles (usec): | 1.00th=[ 424], 5.00th=[ 424], 10.00th=[ 424], 20.00th=[ 465], | 30.00th=[ 42730], 40.00th=[ 42730], 50.00th=[ 44303], 60.00th=[ 49021], | 70.00th=[ 49021], 80.00th=[ 50070], 90.00th=[404751], 95.00th=[404751], | 99.00th=[404751], 99.50th=[404751], 99.90th=[404751], 99.95th=[404751], | 99.99th=[404751] bw ( KiB/s): min= 2048, max= 2048, per=11.69%, avg=2048.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=5, BW=5981KiB/s (6125kB/s)(5120KiB/856msec) clat (usec): min=24468, max=81594, avg=41688.43, stdev=22900.89 lat (usec): min=24509, max=81661, avg=41742.48, stdev=22912.72 clat percentiles (usec): | 1.00th=[24511], 5.00th=[24511], 10.00th=[24511], 20.00th=[24511], | 30.00th=[29492], 40.00th=[29492], 50.00th=[34866], 60.00th=[34866], | 70.00th=[38011], 80.00th=[38011], 90.00th=[81265], 95.00th=[81265], | 99.00th=[81265], 99.50th=[81265], 99.90th=[81265], 99.95th=[81265], | 99.99th=[81265] lat (usec) : 500=16.67% lat (msec) : 50=58.33%, 100=16.67%, 500=8.33% cpu : usr=0.23%, sys=31.11%, ctx=839, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25758: Thu May 1 03:38:41 2025 read: IOPS=1, BW=1916KiB/s (1962kB/s)(2048KiB/1069msec) clat (usec): min=1088, max=42942, avg=22015.40, stdev=29595.00 lat (usec): min=1089, max=42943, avg=22016.32, stdev=29595.01 clat percentiles (usec): | 1.00th=[ 1090], 5.00th=[ 1090], 10.00th=[ 1090], 20.00th=[ 1090], | 30.00th=[ 1090], 40.00th=[ 1090], 50.00th=[ 1090], 60.00th=[42730], | 70.00th=[42730], 80.00th=[42730], 90.00th=[42730], 95.00th=[42730], | 99.00th=[42730], 99.50th=[42730], 99.90th=[42730], 99.95th=[42730], | 99.99th=[42730] bw ( KiB/s): min= 2048, max= 2048, per=11.69%, avg=2048.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 write: IOPS=9, BW=9579KiB/s (9809kB/s)(10.0MiB/1069msec) clat (msec): min=30, max=410, avg=97.10, stdev=113.18 lat (msec): min=30, max=410, avg=97.16, stdev=113.17 clat percentiles (msec): | 1.00th=[ 31], 5.00th=[ 31], 10.00th=[ 31], 20.00th=[ 37], | 30.00th=[ 41], 40.00th=[ 58], 50.00th=[ 59], 60.00th=[ 59], | 70.00th=[ 94], 80.00th=[ 94], 90.00th=[ 118], 95.00th=[ 409], | 99.00th=[ 409], 99.50th=[ 409], 99.90th=[ 409], 99.95th=[ 409], | 99.99th=[ 409] bw ( KiB/s): min=16384, max=16384, per=66.80%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 16, max= 16, avg=16.00, stdev= 0.00, samples=1 lat (msec) : 2=8.33%, 50=33.33%, 100=41.67%, 250=8.33%, 500=8.33% cpu : usr=0.00%, sys=27.62%, ctx=137, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25759: Thu May 1 03:38:41 2025 read: IOPS=5, BW=5256KiB/s (5382kB/s)(6144KiB/1169msec) clat (usec): min=949, max=86011, avg=41049.83, stdev=27295.42 lat (usec): min=951, max=86012, avg=41089.66, stdev=27283.40 clat percentiles (usec): | 1.00th=[ 947], 5.00th=[ 947], 10.00th=[ 947], 20.00th=[33817], | 30.00th=[33817], 40.00th=[36963], 50.00th=[36963], 60.00th=[42206], | 70.00th=[46400], 80.00th=[46400], 90.00th=[86508], 95.00th=[86508], | 99.00th=[86508], 99.50th=[86508], 99.90th=[86508], 99.95th=[86508], | 99.99th=[86508] bw ( KiB/s): min= 6930, max= 6930, per=39.56%, avg=6930.00, stdev= 0.00, samples=1 iops : min= 6, max= 6, avg= 6.00, stdev= 0.00, samples=1 write: IOPS=5, BW=5256KiB/s (5382kB/s)(6144KiB/1169msec) clat (msec): min=34, max=537, avg=144.94, stdev=195.45 lat (msec): min=34, max=537, avg=145.00, stdev=195.45 clat percentiles (msec): | 1.00th=[ 35], 5.00th=[ 35], 10.00th=[ 35], 20.00th=[ 53], | 30.00th=[ 53], 40.00th=[ 54], 50.00th=[ 54], 60.00th=[ 58], | 70.00th=[ 133], 80.00th=[ 133], 90.00th=[ 542], 95.00th=[ 542], | 99.00th=[ 542], 99.50th=[ 542], 99.90th=[ 542], 99.95th=[ 542], | 99.99th=[ 542] bw ( KiB/s): min= 4087, max= 5197, per=18.93%, avg=4642.00, stdev=784.89, samples=2 iops : min= 3, max= 5, avg= 4.00, stdev= 1.41, samples=2 lat (usec) : 1000=8.33% lat (msec) : 50=41.67%, 100=33.33%, 250=8.33%, 750=8.33% cpu : usr=0.00%, sys=25.60%, ctx=774, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25760: Thu May 1 03:38:41 2025 read: IOPS=5, BW=5350KiB/s (5478kB/s)(5120KiB/957msec) clat (usec): min=328, max=49700, avg=27869.46, stdev=25248.71 lat (usec): min=330, max=49701, avg=27870.39, stdev=25248.27 clat percentiles (usec): | 1.00th=[ 330], 5.00th=[ 330], 10.00th=[ 330], 20.00th=[ 330], | 30.00th=[ 490], 40.00th=[ 490], 50.00th=[41681], 60.00th=[41681], | 70.00th=[47449], 80.00th=[47449], 90.00th=[49546], 95.00th=[49546], | 99.00th=[49546], 99.50th=[49546], 99.90th=[49546], 99.95th=[49546], | 99.99th=[49546] write: IOPS=7, BW=7490KiB/s (7670kB/s)(7168KiB/957msec) clat (msec): min=37, max=451, avg=114.63, stdev=149.36 lat (msec): min=37, max=451, avg=114.75, stdev=149.33 clat percentiles (msec): | 1.00th=[ 38], 5.00th=[ 38], 10.00th=[ 38], 20.00th=[ 42], | 30.00th=[ 59], 40.00th=[ 59], 50.00th=[ 66], 60.00th=[ 67], | 70.00th=[ 67], 80.00th=[ 81], 90.00th=[ 451], 95.00th=[ 451], | 99.00th=[ 451], 99.50th=[ 451], 99.90th=[ 451], 99.95th=[ 451], | 99.99th=[ 451] bw ( KiB/s): min= 2048, max= 2048, per=8.35%, avg=2048.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (usec) : 500=16.67% lat (msec) : 50=41.67%, 100=33.33%, 500=8.33% cpu : usr=0.00%, sys=23.74%, ctx=214, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=17.1MiB/s (17.9MB/s), 1916KiB/s-8374KiB/s (1962kB/s-8575kB/s), io=20.0MiB (20.0MB), run=856-1169msec WRITE: bw=23.0MiB/s (25.1MB/s), 5256KiB/s-9579KiB/s (5382kB/s-9809kB/s), io=28.0MiB (29.4MB), run=856-1169msec rand-rw: (groupid=0, jobs=1): err= 0: pid=25762: Thu May 1 03:38:41 2025 read: IOPS=5, BW=5776KiB/s (5915kB/s)(7168KiB/1241msec) clat (msec): min=15, max=410, avg=136.26, stdev=157.91 lat (msec): min=15, max=410, avg=136.26, stdev=157.91 clat percentiles (msec): | 1.00th=[ 16], 5.00th=[ 16], 10.00th=[ 16], 20.00th=[ 17], | 30.00th=[ 22], 40.00th=[ 22], 50.00th=[ 59], 60.00th=[ 134], | 70.00th=[ 134], 80.00th=[ 296], 90.00th=[ 409], 95.00th=[ 409], | 99.00th=[ 409], 99.50th=[ 409], 99.90th=[ 409], 99.95th=[ 409], | 99.99th=[ 409] bw ( KiB/s): min= 2048, max= 8192, per=31.03%, avg=5120.00, stdev=4344.46, samples=2 iops : min= 2, max= 8, avg= 5.00, stdev= 4.24, samples=2 write: IOPS=4, BW=4126KiB/s (4225kB/s)(5120KiB/1241msec) clat (msec): min=16, max=122, avg=48.93, stdev=42.95 lat (msec): min=16, max=122, avg=49.04, stdev=42.93 clat percentiles (msec): | 1.00th=[ 17], 5.00th=[ 17], 10.00th=[ 17], 20.00th=[ 17], | 30.00th=[ 19], 40.00th=[ 19], 50.00th=[ 43], 60.00th=[ 43], | 70.00th=[ 45], 80.00th=[ 45], 90.00th=[ 123], 95.00th=[ 123], | 99.00th=[ 123], 99.50th=[ 123], 99.90th=[ 123], 99.95th=[ 123], | 99.99th=[ 123] bw ( KiB/s): min= 2048, max= 2048, per=8.86%, avg=2048.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 20=33.33%, 50=25.00%, 100=8.33%, 250=16.67%, 500=16.67% cpu : usr=0.24%, sys=8.23%, ctx=85, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25763: Thu May 1 03:38:41 2025 read: IOPS=1, BW=1672KiB/s (1712kB/s)(2048KiB/1225msec) clat (usec): min=14177, max=45623, avg=29900.65, stdev=22235.78 lat (usec): min=14179, max=45626, avg=29902.60, stdev=22236.33 clat percentiles (usec): | 1.00th=[14222], 5.00th=[14222], 10.00th=[14222], 20.00th=[14222], | 30.00th=[14222], 40.00th=[14222], 50.00th=[14222], 60.00th=[45876], | 70.00th=[45876], 80.00th=[45876], 90.00th=[45876], 95.00th=[45876], | 99.00th=[45876], 99.50th=[45876], 99.90th=[45876], 99.95th=[45876], | 99.99th=[45876] bw ( KiB/s): min= 2043, max= 2048, per=12.40%, avg=2045.50, stdev= 3.54, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 write: IOPS=8, BW=8359KiB/s (8560kB/s)(10.0MiB/1225msec) clat (msec): min=27, max=466, avg=111.59, stdev=130.26 lat (msec): min=27, max=466, avg=111.68, stdev=130.26 clat percentiles (msec): | 1.00th=[ 28], 5.00th=[ 28], 10.00th=[ 28], 20.00th=[ 30], | 30.00th=[ 32], 40.00th=[ 55], 50.00th=[ 58], 60.00th=[ 104], | 70.00th=[ 107], 80.00th=[ 107], 90.00th=[ 136], 95.00th=[ 468], | 99.00th=[ 468], 99.50th=[ 468], 99.90th=[ 468], 99.95th=[ 468], | 99.99th=[ 468] bw ( KiB/s): min=10240, max=10240, per=44.32%, avg=10240.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 lat (msec) : 20=8.33%, 50=33.33%, 100=16.67%, 250=33.33%, 500=8.33% cpu : usr=0.16%, sys=14.71%, ctx=108, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25764: Thu May 1 03:38:41 2025 read: IOPS=5, BW=5596KiB/s (5730kB/s)(6144KiB/1098msec) clat (msec): min=21, max=102, avg=63.14, stdev=35.35 lat (msec): min=21, max=102, avg=63.14, stdev=35.35 clat percentiles (msec): | 1.00th=[ 22], 5.00th=[ 22], 10.00th=[ 22], 20.00th=[ 30], | 30.00th=[ 30], 40.00th=[ 48], 50.00th=[ 48], 60.00th=[ 77], | 70.00th=[ 102], 80.00th=[ 102], 90.00th=[ 104], 95.00th=[ 104], | 99.00th=[ 104], 99.50th=[ 104], 99.90th=[ 104], 99.95th=[ 104], | 99.99th=[ 104] bw ( KiB/s): min= 8192, max= 8192, per=49.64%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=5, BW=5596KiB/s (5730kB/s)(6144KiB/1098msec) clat (msec): min=18, max=453, avg=114.05, stdev=169.92 lat (msec): min=18, max=453, avg=114.11, stdev=169.90 clat percentiles (msec): | 1.00th=[ 18], 5.00th=[ 18], 10.00th=[ 18], 20.00th=[ 20], | 30.00th=[ 20], 40.00th=[ 26], 50.00th=[ 26], 60.00th=[ 57], | 70.00th=[ 113], 80.00th=[ 113], 90.00th=[ 456], 95.00th=[ 456], | 99.00th=[ 456], 99.50th=[ 456], 99.90th=[ 456], 99.95th=[ 456], | 99.99th=[ 456] bw ( KiB/s): min= 4096, max= 6144, per=22.16%, avg=5120.00, stdev=1448.15, samples=2 iops : min= 4, max= 6, avg= 5.00, stdev= 1.41, samples=2 lat (msec) : 20=16.67%, 50=33.33%, 100=16.67%, 250=25.00%, 500=8.33% cpu : usr=0.00%, sys=9.21%, ctx=56, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25765: Thu May 1 03:38:41 2025 read: IOPS=4, BW=4414KiB/s (4520kB/s)(5120KiB/1160msec) clat (usec): min=13231, max=81404, avg=35064.03, stdev=27189.96 lat (usec): min=13232, max=81405, avg=35065.07, stdev=27189.83 clat percentiles (usec): | 1.00th=[13173], 5.00th=[13173], 10.00th=[13173], 20.00th=[13173], | 30.00th=[19792], 40.00th=[19792], 50.00th=[25297], 60.00th=[25297], | 70.00th=[35914], 80.00th=[35914], 90.00th=[81265], 95.00th=[81265], | 99.00th=[81265], 99.50th=[81265], 99.90th=[81265], 99.95th=[81265], | 99.99th=[81265] bw ( KiB/s): min= 8192, max= 8192, per=49.64%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=6, BW=6179KiB/s (6328kB/s)(7168KiB/1160msec) clat (msec): min=16, max=519, avg=127.40, stdev=176.74 lat (msec): min=16, max=519, avg=127.48, stdev=176.74 clat percentiles (msec): | 1.00th=[ 17], 5.00th=[ 17], 10.00th=[ 17], 20.00th=[ 23], | 30.00th=[ 44], 40.00th=[ 44], 50.00th=[ 88], 60.00th=[ 90], | 70.00th=[ 90], 80.00th=[ 113], 90.00th=[ 518], 95.00th=[ 518], | 99.00th=[ 518], 99.50th=[ 518], 99.90th=[ 518], 99.95th=[ 518], | 99.99th=[ 518] bw ( KiB/s): min= 2048, max= 6144, per=17.73%, avg=4096.00, stdev=2896.31, samples=2 iops : min= 2, max= 6, avg= 4.00, stdev= 2.83, samples=2 lat (msec) : 20=25.00%, 50=33.33%, 100=25.00%, 250=8.33%, 750=8.33% cpu : usr=0.09%, sys=8.80%, ctx=112, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=16.1MiB/s (16.9MB/s), 1672KiB/s-5776KiB/s (1712kB/s-5915kB/s), io=20.0MiB (20.0MB), run=1098-1241msec WRITE: bw=22.6MiB/s (23.7MB/s), 4126KiB/s-8359KiB/s (4225kB/s-8560kB/s), io=28.0MiB (29.4MB), run=1098-1241msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=25771: Thu May 1 03:38:43 2025 read: IOPS=8, BW=33.8MiB/s (35.4MB/s)(12.0MiB/355msec) clat (msec): min=2, max=195, avg=115.47, stdev=100.93 lat (msec): min=2, max=195, avg=115.48, stdev=100.93 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 3], 20.00th=[ 3], | 30.00th=[ 3], 40.00th=[ 148], 50.00th=[ 148], 60.00th=[ 148], | 70.00th=[ 197], 80.00th=[ 197], 90.00th=[ 197], 95.00th=[ 197], | 99.00th=[ 197], 99.50th=[ 197], 99.90th=[ 197], 99.95th=[ 197], | 99.99th=[ 197] lat (msec) : 4=33.33%, 250=66.67% cpu : usr=0.00%, sys=43.50%, ctx=1100, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25774: Thu May 1 03:38:43 2025 read: IOPS=1, BW=4334KiB/s (4438kB/s)(4096KiB/945msec) clat (nsec): min=177344k, max=177344k, avg=177344393.00, stdev= 0.00 lat (nsec): min=177347k, max=177347k, avg=177346778.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 178], 5.00th=[ 178], 10.00th=[ 178], 20.00th=[ 178], | 30.00th=[ 178], 40.00th=[ 178], 50.00th=[ 178], 60.00th=[ 178], | 70.00th=[ 178], 80.00th=[ 178], 90.00th=[ 178], 95.00th=[ 178], | 99.00th=[ 178], 99.50th=[ 178], 99.90th=[ 178], 99.95th=[ 178], | 99.99th=[ 178] bw ( KiB/s): min= 8192, max= 8192, per=47.25%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=2, BW=8669KiB/s (8877kB/s)(8192KiB/945msec) clat (msec): min=363, max=366, avg=365.05, stdev= 2.38 lat (msec): min=363, max=367, avg=365.41, stdev= 2.53 clat percentiles (msec): | 1.00th=[ 363], 5.00th=[ 363], 10.00th=[ 363], 20.00th=[ 363], | 30.00th=[ 363], 40.00th=[ 363], 50.00th=[ 363], 60.00th=[ 368], | 70.00th=[ 368], 80.00th=[ 368], 90.00th=[ 368], 95.00th=[ 368], | 99.00th=[ 368], 99.50th=[ 368], 99.90th=[ 368], 99.95th=[ 368], | 99.99th=[ 368] lat (msec) : 250=33.33%, 500=66.67% cpu : usr=0.00%, sys=36.23%, ctx=337, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25775: Thu May 1 03:38:43 2025 write: IOPS=3, BW=14.5MiB/s (15.3MB/s)(12.0MiB/825msec) clat (msec): min=183, max=310, avg=266.44, stdev=71.49 lat (msec): min=184, max=311, avg=266.79, stdev=71.43 clat percentiles (msec): | 1.00th=[ 184], 5.00th=[ 184], 10.00th=[ 184], 20.00th=[ 184], | 30.00th=[ 184], 40.00th=[ 305], 50.00th=[ 305], 60.00th=[ 305], | 70.00th=[ 313], 80.00th=[ 313], 90.00th=[ 313], 95.00th=[ 313], | 99.00th=[ 313], 99.50th=[ 313], 99.90th=[ 313], 99.95th=[ 313], | 99.99th=[ 313] bw ( KiB/s): min= 8192, max= 8192, per=23.63%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=66.67% cpu : usr=0.00%, sys=42.96%, ctx=42, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25777: Thu May 1 03:38:43 2025 write: IOPS=3, BW=15.4MiB/s (16.2MB/s)(12.0MiB/777msec) clat (msec): min=201, max=293, avg=242.90, stdev=46.46 lat (msec): min=201, max=293, avg=243.15, stdev=46.51 clat percentiles (msec): | 1.00th=[ 203], 5.00th=[ 203], 10.00th=[ 203], 20.00th=[ 203], | 30.00th=[ 203], 40.00th=[ 234], 50.00th=[ 234], 60.00th=[ 234], | 70.00th=[ 292], 80.00th=[ 292], 90.00th=[ 292], 95.00th=[ 292], | 99.00th=[ 292], 99.50th=[ 292], 99.90th=[ 292], 99.95th=[ 292], | 99.99th=[ 292] bw ( KiB/s): min=16384, max=16384, per=47.25%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.00%, sys=47.81%, ctx=64, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=16.9MiB/s (17.8MB/s), 4334KiB/s-33.8MiB/s (4438kB/s-35.4MB/s), io=16.0MiB (16.8MB), run=355-945msec WRITE: bw=33.9MiB/s (35.5MB/s), 8669KiB/s-15.4MiB/s (8877kB/s-16.2MB/s), io=32.0MiB (33.6MB), run=777-945msec rand-rw: (groupid=0, jobs=1): err= 0: pid=25772: Thu May 1 03:38:43 2025 read: IOPS=3, BW=12.0MiB/s (12.6MB/s)(12.0MiB/997msec) clat (msec): min=232, max=459, avg=321.94, stdev=120.87 lat (msec): min=232, max=459, avg=321.95, stdev=120.87 clat percentiles (msec): | 1.00th=[ 232], 5.00th=[ 232], 10.00th=[ 232], 20.00th=[ 232], | 30.00th=[ 232], 40.00th=[ 275], 50.00th=[ 275], 60.00th=[ 275], | 70.00th=[ 460], 80.00th=[ 460], 90.00th=[ 460], 95.00th=[ 460], | 99.00th=[ 460], 99.50th=[ 460], 99.90th=[ 460], 99.95th=[ 460], | 99.99th=[ 460] bw ( KiB/s): min= 8192, max= 8192, per=54.15%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=66.67% cpu : usr=0.10%, sys=17.17%, ctx=99, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25773: Thu May 1 03:38:43 2025 read: IOPS=0, BW=3782KiB/s (3873kB/s)(4096KiB/1083msec) clat (nsec): min=74020k, max=74020k, avg=74019564.00, stdev= 0.00 lat (nsec): min=74022k, max=74022k, avg=74021610.00, stdev= 0.00 clat percentiles (usec): | 1.00th=[73925], 5.00th=[73925], 10.00th=[73925], 20.00th=[73925], | 30.00th=[73925], 40.00th=[73925], 50.00th=[73925], 60.00th=[73925], | 70.00th=[73925], 80.00th=[73925], 90.00th=[73925], 95.00th=[73925], | 99.00th=[73925], 99.50th=[73925], 99.90th=[73925], 99.95th=[73925], | 99.99th=[73925] bw ( KiB/s): min= 8192, max= 8192, per=54.15%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=1, BW=7564KiB/s (7746kB/s)(8192KiB/1083msec) clat (msec): min=411, max=564, avg=487.79, stdev=108.25 lat (msec): min=411, max=564, avg=488.28, stdev=108.23 clat percentiles (msec): | 1.00th=[ 414], 5.00th=[ 414], 10.00th=[ 414], 20.00th=[ 414], | 30.00th=[ 414], 40.00th=[ 414], 50.00th=[ 414], 60.00th=[ 567], | 70.00th=[ 567], 80.00th=[ 567], 90.00th=[ 567], 95.00th=[ 567], | 99.00th=[ 567], 99.50th=[ 567], 99.90th=[ 567], 99.95th=[ 567], | 99.99th=[ 567] lat (msec) : 100=33.33%, 500=33.33%, 750=33.33% cpu : usr=0.18%, sys=13.86%, ctx=75, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25776: Thu May 1 03:38:43 2025 write: IOPS=2, BW=10.7MiB/s (11.2MB/s)(12.0MiB/1123msec) clat (msec): min=35, max=547, avg=366.09, stdev=287.03 lat (msec): min=35, max=547, avg=366.36, stdev=287.02 clat percentiles (msec): | 1.00th=[ 36], 5.00th=[ 36], 10.00th=[ 36], 20.00th=[ 36], | 30.00th=[ 36], 40.00th=[ 514], 50.00th=[ 514], 60.00th=[ 514], | 70.00th=[ 550], 80.00th=[ 550], 90.00th=[ 550], 95.00th=[ 550], | 99.00th=[ 550], 99.50th=[ 550], 99.90th=[ 550], 99.95th=[ 550], | 99.99th=[ 550] lat (msec) : 50=33.33%, 750=66.67% cpu : usr=0.18%, sys=9.08%, ctx=79, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25778: Thu May 1 03:38:43 2025 write: IOPS=4, BW=18.5MiB/s (19.4MB/s)(12.0MiB/650msec) clat (msec): min=54, max=433, avg=209.26, stdev=198.93 lat (msec): min=55, max=433, avg=209.52, stdev=198.92 clat percentiles (msec): | 1.00th=[ 55], 5.00th=[ 55], 10.00th=[ 55], 20.00th=[ 55], | 30.00th=[ 55], 40.00th=[ 140], 50.00th=[ 140], 60.00th=[ 140], | 70.00th=[ 435], 80.00th=[ 435], 90.00th=[ 435], 95.00th=[ 435], | 99.00th=[ 435], 99.50th=[ 435], 99.90th=[ 435], 99.95th=[ 435], | 99.99th=[ 435] bw ( KiB/s): min=16384, max=16384, per=56.15%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 250=33.33%, 500=33.33% cpu : usr=0.31%, sys=11.71%, ctx=63, majf=0, minf=28 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=14.8MiB/s (15.5MB/s), 3782KiB/s-12.0MiB/s (3873kB/s-12.6MB/s), io=16.0MiB (16.8MB), run=997-1083msec WRITE: bw=28.5MiB/s (29.9MB/s), 7564KiB/s-18.5MiB/s (7746kB/s-19.4MB/s), io=32.0MiB (33.6MB), run=650-1123msec