== sanity test 398b: DIO and buffer IO race ============== 08:23:48 (1733405028) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.39527 s, 36.1 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=20969: Thu Dec 5 08:24:06 2024 read: IOPS=99, BW=398KiB/s (408kB/s)(5992KiB/15045msec) clat (usec): min=48, max=51793, avg=5550.22, stdev=3504.69 lat (usec): min=48, max=51794, avg=5551.16, stdev=3505.09 clat percentiles (usec): | 1.00th=[ 70], 5.00th=[ 155], 10.00th=[ 2704], 20.00th=[ 3195], | 30.00th=[ 3654], 40.00th=[ 4178], 50.00th=[ 4883], 60.00th=[ 5669], | 70.00th=[ 6521], 80.00th=[ 7570], 90.00th=[ 9241], 95.00th=[10945], | 99.00th=[17957], 99.50th=[21365], 99.90th=[32637], 99.95th=[51643], | 99.99th=[51643] bw ( KiB/s): min= 264, max= 544, per=25.13%, avg=396.77, stdev=63.66, samples=30 iops : min= 66, max= 136, avg=99.17, stdev=15.96, samples=30 write: IOPS=104, BW=418KiB/s (429kB/s)(6296KiB/15045msec) clat (usec): min=423, max=144243, avg=4234.42, stdev=5148.15 lat (usec): min=423, max=144245, avg=4236.35, stdev=5149.25 clat percentiles (usec): | 1.00th=[ 537], 5.00th=[ 611], 10.00th=[ 676], 20.00th=[ 816], | 30.00th=[ 1012], 40.00th=[ 2343], 50.00th=[ 3392], 60.00th=[ 4293], | 70.00th=[ 5211], 80.00th=[ 6849], 90.00th=[ 9372], 95.00th=[ 11338], | 99.00th=[ 16450], 99.50th=[ 20841], 99.90th=[ 28967], 99.95th=[143655], | 99.99th=[143655] bw ( KiB/s): min= 280, max= 632, per=25.26%, avg=417.57, stdev=78.63, samples=30 iops : min= 70, max= 158, avg=104.37, stdev=19.65, samples=30 lat (usec) : 50=0.07%, 100=1.60%, 250=0.85%, 500=0.33%, 750=7.55% lat (usec) : 1000=7.39% lat (msec) : 2=4.49%, 4=24.67%, 10=45.83%, 20=6.67%, 50=0.49% lat (msec) : 100=0.03%, 250=0.03% cpu : usr=0.30%, sys=23.88%, ctx=6261, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20970: Thu Dec 5 08:24:06 2024 read: IOPS=98, BW=394KiB/s (403kB/s)(5940KiB/15089msec) clat (usec): min=50, max=30510, avg=5495.92, stdev=3180.20 lat (usec): min=50, max=30510, avg=5498.60, stdev=3181.30 clat percentiles (usec): | 1.00th=[ 72], 5.00th=[ 119], 10.00th=[ 2638], 20.00th=[ 3195], | 30.00th=[ 3720], 40.00th=[ 4228], 50.00th=[ 5014], 60.00th=[ 5735], | 70.00th=[ 6521], 80.00th=[ 7635], 90.00th=[ 9241], 95.00th=[10683], | 99.00th=[16057], 99.50th=[17695], 99.90th=[26084], 99.95th=[30540], | 99.99th=[30540] bw ( KiB/s): min= 256, max= 544, per=24.81%, avg=391.73, stdev=57.71, samples=30 iops : min= 64, max= 136, avg=97.93, stdev=14.43, samples=30 write: IOPS=105, BW=421KiB/s (431kB/s)(6348KiB/15089msec) clat (usec): min=436, max=180416, avg=4317.74, stdev=5871.33 lat (usec): min=437, max=180417, avg=4319.00, stdev=5871.30 clat percentiles (usec): | 1.00th=[ 529], 5.00th=[ 619], 10.00th=[ 701], 20.00th=[ 840], | 30.00th=[ 1057], 40.00th=[ 2311], 50.00th=[ 3392], 60.00th=[ 4228], | 70.00th=[ 5407], 80.00th=[ 7111], 90.00th=[ 9503], 95.00th=[ 11469], | 99.00th=[ 15926], 99.50th=[ 18744], 99.90th=[ 47449], 99.95th=[181404], | 99.99th=[181404] bw ( KiB/s): min= 288, max= 576, per=25.21%, avg=416.80, stdev=79.48, samples=30 iops : min= 72, max= 144, avg=104.20, stdev=19.87, samples=30 lat (usec) : 100=1.79%, 250=0.98%, 500=0.23%, 750=6.58%, 1000=7.91% lat (msec) : 2=5.47%, 4=23.99%, 10=45.54%, 20=7.13%, 50=0.36% lat (msec) : 250=0.03% cpu : usr=0.19%, sys=23.89%, ctx=6328, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20971: Thu Dec 5 08:24:06 2024 read: IOPS=99, BW=397KiB/s (406kB/s)(6028KiB/15200msec) clat (usec): min=50, max=180443, avg=5793.52, stdev=5620.99 lat (usec): min=50, max=180443, avg=5794.43, stdev=5620.95 clat percentiles (usec): | 1.00th=[ 69], 5.00th=[ 139], 10.00th=[ 2671], 20.00th=[ 3261], | 30.00th=[ 3785], 40.00th=[ 4490], 50.00th=[ 5211], 60.00th=[ 5932], | 70.00th=[ 6783], 80.00th=[ 7898], 90.00th=[ 9372], 95.00th=[ 10814], | 99.00th=[ 17695], 99.50th=[ 19792], 99.90th=[ 49021], 99.95th=[181404], | 99.99th=[181404] bw ( KiB/s): min= 200, max= 592, per=24.81%, avg=391.73, stdev=74.05, samples=30 iops : min= 50, max= 148, avg=97.93, stdev=18.51, samples=30 write: IOPS=102, BW=412KiB/s (422kB/s)(6260KiB/15200msec) clat (usec): min=367, max=24724, avg=4090.99, stdev=3609.32 lat (usec): min=368, max=24725, avg=4092.55, stdev=3609.60 clat percentiles (usec): | 1.00th=[ 529], 5.00th=[ 594], 10.00th=[ 668], 20.00th=[ 832], | 30.00th=[ 996], 40.00th=[ 2180], 50.00th=[ 3359], 60.00th=[ 4228], | 70.00th=[ 5342], 80.00th=[ 7046], 90.00th=[ 9110], 95.00th=[11207], | 99.00th=[15401], 99.50th=[16450], 99.90th=[24249], 99.95th=[24773], | 99.99th=[24773] bw ( KiB/s): min= 240, max= 544, per=24.57%, avg=406.13, stdev=70.11, samples=30 iops : min= 60, max= 136, avg=101.53, stdev=17.53, samples=30 lat (usec) : 100=1.82%, 250=0.85%, 500=0.10%, 750=7.29%, 1000=8.01% lat (msec) : 2=4.88%, 4=22.59%, 10=46.94%, 20=7.19%, 50=0.29% lat (msec) : 250=0.03% cpu : usr=0.16%, sys=23.85%, ctx=6336, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20972: Thu Dec 5 08:24:06 2024 read: IOPS=100, BW=404KiB/s (414kB/s)(6052KiB/14981msec) clat (usec): min=52, max=182069, avg=5679.28, stdev=5519.91 lat (usec): min=52, max=182070, avg=5682.12, stdev=5519.96 clat percentiles (usec): | 1.00th=[ 71], 5.00th=[ 2180], 10.00th=[ 2769], 20.00th=[ 3228], | 30.00th=[ 3720], 40.00th=[ 4228], 50.00th=[ 4948], 60.00th=[ 5800], | 70.00th=[ 6652], 80.00th=[ 7635], 90.00th=[ 9241], 95.00th=[ 11076], | 99.00th=[ 16581], 99.50th=[ 19268], 99.90th=[ 26870], 99.95th=[181404], | 99.99th=[181404] bw ( KiB/s): min= 248, max= 536, per=25.35%, avg=400.24, stdev=62.93, samples=29 iops : min= 62, max= 134, avg=100.03, stdev=15.75, samples=29 write: IOPS=104, BW=416KiB/s (426kB/s)(6236KiB/14981msec) clat (usec): min=469, max=33421, avg=4054.24, stdev=3589.25 lat (usec): min=469, max=33421, avg=4055.11, stdev=3589.24 clat percentiles (usec): | 1.00th=[ 529], 5.00th=[ 611], 10.00th=[ 693], 20.00th=[ 832], | 30.00th=[ 1029], 40.00th=[ 2278], 50.00th=[ 3326], 60.00th=[ 4080], | 70.00th=[ 5211], 80.00th=[ 6783], 90.00th=[ 8848], 95.00th=[10683], | 99.00th=[15139], 99.50th=[16909], 99.90th=[23987], 99.95th=[33424], | 99.99th=[33424] bw ( KiB/s): min= 272, max= 616, per=25.01%, avg=413.48, stdev=83.10, samples=29 iops : min= 68, max= 154, avg=103.34, stdev=20.77, samples=29 lat (usec) : 100=1.40%, 250=0.91%, 500=0.20%, 750=6.71%, 1000=7.94% lat (msec) : 2=5.05%, 4=25.20%, 10=45.83%, 20=6.58%, 50=0.16% lat (msec) : 250=0.03% cpu : usr=0.17%, sys=24.25%, ctx=6200, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1580KiB/s (1618kB/s), 394KiB/s-404KiB/s (403kB/s-414kB/s), io=23.4MiB (24.6MB), run=14981-15200msec WRITE: bw=1654KiB/s (1694kB/s), 412KiB/s-421KiB/s (422kB/s-431kB/s), io=24.6MiB (25.7MB), run=14981-15200msec rand-rw: (groupid=0, jobs=1): err= 0: pid=20973: Thu Dec 5 08:24:10 2024 read: IOPS=78, BW=315KiB/s (323kB/s)(5992KiB/19009msec) clat (usec): min=1501, max=204790, avg=5132.56, stdev=6532.08 lat (usec): min=1502, max=204792, avg=5133.29, stdev=6532.10 clat percentiles (usec): | 1.00th=[ 1663], 5.00th=[ 1909], 10.00th=[ 2057], 20.00th=[ 2245], | 30.00th=[ 2507], 40.00th=[ 2900], 50.00th=[ 3458], 60.00th=[ 4015], | 70.00th=[ 5080], 80.00th=[ 7046], 90.00th=[ 11076], 95.00th=[ 13566], | 99.00th=[ 19792], 99.50th=[ 21890], 99.90th=[ 33817], 99.95th=[204473], | 99.99th=[204473] bw ( KiB/s): min= 152, max= 736, per=24.56%, avg=309.97, stdev=158.81, samples=37 iops : min= 38, max= 184, avg=77.43, stdev=39.73, samples=37 write: IOPS=82, BW=331KiB/s (339kB/s)(6296KiB/19009msec) clat (usec): min=2272, max=32485, avg=7156.84, stdev=4602.00 lat (usec): min=2273, max=32486, avg=7157.83, stdev=4602.03 clat percentiles (usec): | 1.00th=[ 2540], 5.00th=[ 2999], 10.00th=[ 3228], 20.00th=[ 3589], | 30.00th=[ 4015], 40.00th=[ 4621], 50.00th=[ 5407], 60.00th=[ 6521], | 70.00th=[ 8455], 80.00th=[10421], 90.00th=[13435], 95.00th=[15795], | 99.00th=[23987], 99.50th=[28181], 99.90th=[32113], 99.95th=[32375], | 99.99th=[32375] bw ( KiB/s): min= 160, max= 760, per=24.53%, avg=324.03, stdev=161.16, samples=37 iops : min= 40, max= 190, avg=80.95, stdev=40.31, samples=37 lat (msec) : 2=3.71%, 4=40.59%, 10=38.48%, 20=15.79%, 50=1.40% lat (msec) : 250=0.03% cpu : usr=0.19%, sys=10.83%, ctx=3605, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20974: Thu Dec 5 08:24:10 2024 read: IOPS=78, BW=313KiB/s (320kB/s)(5940KiB/18994msec) clat (usec): min=1576, max=183781, avg=5172.20, stdev=6229.77 lat (usec): min=1577, max=183782, avg=5172.90, stdev=6229.80 clat percentiles (usec): | 1.00th=[ 1713], 5.00th=[ 1893], 10.00th=[ 2024], 20.00th=[ 2245], | 30.00th=[ 2507], 40.00th=[ 2868], 50.00th=[ 3392], 60.00th=[ 4015], | 70.00th=[ 5014], 80.00th=[ 7177], 90.00th=[ 11338], 95.00th=[ 14353], | 99.00th=[ 20579], 99.50th=[ 21627], 99.90th=[ 27657], 99.95th=[183501], | 99.99th=[183501] bw ( KiB/s): min= 112, max= 672, per=24.58%, avg=310.19, stdev=165.02, samples=37 iops : min= 28, max= 168, avg=77.49, stdev=41.28, samples=37 write: IOPS=83, BW=334KiB/s (342kB/s)(6348KiB/18994msec) clat (usec): min=2249, max=52025, avg=7109.97, stdev=4482.44 lat (usec): min=2249, max=52026, avg=7110.89, stdev=4482.52 clat percentiles (usec): | 1.00th=[ 2638], 5.00th=[ 3032], 10.00th=[ 3261], 20.00th=[ 3687], | 30.00th=[ 4146], 40.00th=[ 4621], 50.00th=[ 5473], 60.00th=[ 6718], | 70.00th=[ 8291], 80.00th=[10159], 90.00th=[12911], 95.00th=[15926], | 99.00th=[22938], 99.50th=[26608], 99.90th=[31851], 99.95th=[52167], | 99.99th=[52167] bw ( KiB/s): min= 184, max= 776, per=24.77%, avg=327.27, stdev=155.79, samples=37 iops : min= 46, max= 194, avg=81.76, stdev=38.97, samples=37 lat (msec) : 2=4.49%, 4=38.38%, 10=40.43%, 20=15.40%, 50=1.24% lat (msec) : 100=0.03%, 250=0.03% cpu : usr=0.16%, sys=10.89%, ctx=3608, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20975: Thu Dec 5 08:24:10 2024 read: IOPS=79, BW=317KiB/s (324kB/s)(6028KiB/19023msec) clat (usec): min=1499, max=30706, avg=5127.78, stdev=4087.85 lat (usec): min=1499, max=30707, avg=5128.56, stdev=4087.91 clat percentiles (usec): | 1.00th=[ 1680], 5.00th=[ 1909], 10.00th=[ 2040], 20.00th=[ 2245], | 30.00th=[ 2507], 40.00th=[ 2966], 50.00th=[ 3425], 60.00th=[ 4080], | 70.00th=[ 5276], 80.00th=[ 7635], 90.00th=[11207], 95.00th=[14222], | 99.00th=[19530], 99.50th=[21627], 99.90th=[28443], 99.95th=[30802], | 99.99th=[30802] bw ( KiB/s): min= 136, max= 760, per=25.08%, avg=316.53, stdev=160.59, samples=38 iops : min= 34, max= 190, avg=79.05, stdev=40.18, samples=38 write: IOPS=82, BW=329KiB/s (337kB/s)(6260KiB/19023msec) clat (msec): min=2, max=167, avg= 7.19, stdev= 6.08 lat (msec): min=2, max=167, avg= 7.19, stdev= 6.08 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 4], 20.00th=[ 4], | 30.00th=[ 4], 40.00th=[ 5], 50.00th=[ 6], 60.00th=[ 7], | 70.00th=[ 9], 80.00th=[ 11], 90.00th=[ 14], 95.00th=[ 16], | 99.00th=[ 24], 99.50th=[ 28], 99.90th=[ 41], 99.95th=[ 167], | 99.99th=[ 167] bw ( KiB/s): min= 176, max= 696, per=24.85%, avg=328.32, stdev=166.15, samples=38 iops : min= 44, max= 174, avg=82.00, stdev=41.56, samples=38 lat (msec) : 2=4.04%, 4=40.27%, 10=38.87%, 20=15.40%, 50=1.40% lat (msec) : 250=0.03% cpu : usr=0.24%, sys=10.77%, ctx=3601, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20976: Thu Dec 5 08:24:10 2024 read: IOPS=79, BW=319KiB/s (327kB/s)(6052KiB/18945msec) clat (usec): min=1438, max=31936, avg=5020.37, stdev=3963.39 lat (usec): min=1439, max=31936, avg=5021.12, stdev=3963.40 clat percentiles (usec): | 1.00th=[ 1729], 5.00th=[ 1926], 10.00th=[ 2057], 20.00th=[ 2245], | 30.00th=[ 2540], 40.00th=[ 2966], 50.00th=[ 3458], 60.00th=[ 4080], | 70.00th=[ 5211], 80.00th=[ 7046], 90.00th=[10814], 95.00th=[13173], | 99.00th=[19268], 99.50th=[22152], 99.90th=[30278], 99.95th=[31851], | 99.99th=[31851] bw ( KiB/s): min= 152, max= 712, per=24.90%, avg=314.19, stdev=152.84, samples=37 iops : min= 38, max= 178, avg=78.43, stdev=38.24, samples=37 write: IOPS=82, BW=329KiB/s (337kB/s)(6236KiB/18945msec) clat (msec): min=2, max=166, avg= 7.25, stdev= 6.11 lat (msec): min=2, max=166, avg= 7.25, stdev= 6.11 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 4], 10.00th=[ 4], 20.00th=[ 4], | 30.00th=[ 5], 40.00th=[ 5], 50.00th=[ 6], 60.00th=[ 7], | 70.00th=[ 9], 80.00th=[ 11], 90.00th=[ 14], 95.00th=[ 16], | 99.00th=[ 26], 99.50th=[ 30], 99.90th=[ 46], 99.95th=[ 167], | 99.99th=[ 167] bw ( KiB/s): min= 168, max= 728, per=24.57%, avg=324.57, stdev=158.85, samples=37 iops : min= 42, max= 182, avg=81.03, stdev=39.74, samples=37 lat (msec) : 2=3.91%, 4=39.75%, 10=39.52%, 20=15.33%, 50=1.46% lat (msec) : 250=0.03% cpu : usr=0.28%, sys=10.81%, ctx=3609, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1262KiB/s (1293kB/s), 313KiB/s-319KiB/s (320kB/s-327kB/s), io=23.4MiB (24.6MB), run=18945-19023msec WRITE: bw=1322KiB/s (1353kB/s), 329KiB/s-334KiB/s (337kB/s-342kB/s), io=24.6MiB (25.7MB), run=18945-19023msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=21008: Thu Dec 5 08:24:18 2024 read: IOPS=55, BW=881KiB/s (903kB/s)(5920KiB/6716msec) clat (usec): min=53, max=44860, avg=12739.08, stdev=4542.17 lat (usec): min=54, max=44861, avg=12739.82, stdev=4542.21 clat percentiles (usec): | 1.00th=[ 100], 5.00th=[ 8291], 10.00th=[ 9110], 20.00th=[10028], | 30.00th=[10683], 40.00th=[11338], 50.00th=[11994], 60.00th=[12911], | 70.00th=[13960], 80.00th=[15008], 90.00th=[16909], 95.00th=[19792], | 99.00th=[29230], 99.50th=[34866], 99.90th=[44827], 99.95th=[44827], | 99.99th=[44827] bw ( KiB/s): min= 576, max= 1088, per=24.54%, avg=873.54, stdev=155.29, samples=13 iops : min= 36, max= 68, avg=54.46, stdev= 9.69, samples=13 write: IOPS=59, BW=948KiB/s (971kB/s)(6368KiB/6716msec) clat (usec): min=656, max=41535, avg=4999.13, stdev=5161.41 lat (usec): min=657, max=41537, avg=5000.37, stdev=5161.43 clat percentiles (usec): | 1.00th=[ 717], 5.00th=[ 873], 10.00th=[ 971], 20.00th=[ 1385], | 30.00th=[ 2442], 40.00th=[ 2900], 50.00th=[ 3392], 60.00th=[ 4113], | 70.00th=[ 5342], 80.00th=[ 6718], 90.00th=[11600], 95.00th=[15926], | 99.00th=[24773], 99.50th=[36439], 99.90th=[41681], 99.95th=[41681], | 99.99th=[41681] bw ( KiB/s): min= 576, max= 1309, per=25.36%, avg=949.77, stdev=234.01, samples=13 iops : min= 36, max= 81, avg=59.23, stdev=14.61, samples=13 lat (usec) : 100=0.52%, 250=0.52%, 750=0.91%, 1000=4.69% lat (msec) : 2=7.42%, 4=17.71%, 10=23.83%, 20=40.89%, 50=3.52% cpu : usr=0.18%, sys=20.24%, ctx=2816, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=21009: Thu Dec 5 08:24:18 2024 read: IOPS=54, BW=866KiB/s (887kB/s)(5744KiB/6629msec) clat (usec): min=61, max=32458, avg=12688.31, stdev=4203.17 lat (usec): min=62, max=32459, avg=12689.24, stdev=4203.13 clat percentiles (usec): | 1.00th=[ 106], 5.00th=[ 7963], 10.00th=[ 8586], 20.00th=[ 9896], | 30.00th=[10552], 40.00th=[11207], 50.00th=[11994], 60.00th=[12911], | 70.00th=[14091], 80.00th=[15401], 90.00th=[17695], 95.00th=[20055], | 99.00th=[25560], 99.50th=[31851], 99.90th=[32375], 99.95th=[32375], | 99.99th=[32375] bw ( KiB/s): min= 479, max= 1088, per=24.26%, avg=863.77, stdev=175.94, samples=13 iops : min= 29, max= 68, avg=53.85, stdev=11.17, samples=13 write: IOPS=61, BW=987KiB/s (1011kB/s)(6544KiB/6629msec) clat (usec): min=680, max=52978, avg=4938.11, stdev=5131.33 lat (usec): min=681, max=52980, avg=4939.50, stdev=5131.28 clat percentiles (usec): | 1.00th=[ 717], 5.00th=[ 865], 10.00th=[ 1004], 20.00th=[ 1352], | 30.00th=[ 2409], 40.00th=[ 2868], 50.00th=[ 3195], 60.00th=[ 4015], | 70.00th=[ 5145], 80.00th=[ 6783], 90.00th=[11338], 95.00th=[14353], | 99.00th=[22152], 99.50th=[26084], 99.90th=[53216], 99.95th=[53216], | 99.99th=[53216] bw ( KiB/s): min= 640, max= 1341, per=26.08%, avg=976.85, stdev=257.96, samples=13 iops : min= 40, max= 83, avg=60.92, stdev=16.08, samples=13 lat (usec) : 100=0.39%, 250=0.39%, 750=1.17%, 1000=4.04% lat (msec) : 2=7.68%, 4=19.01%, 10=23.70%, 20=40.23%, 50=3.26% lat (msec) : 100=0.13% cpu : usr=0.05%, sys=20.37%, ctx=2815, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=21010: Thu Dec 5 08:24:18 2024 read: IOPS=59, BW=945KiB/s (968kB/s)(6256KiB/6620msec) clat (usec): min=53, max=35751, avg=12348.89, stdev=3758.34 lat (usec): min=54, max=35752, avg=12349.60, stdev=3758.36 clat percentiles (usec): | 1.00th=[ 85], 5.00th=[ 7963], 10.00th=[ 8848], 20.00th=[ 9896], | 30.00th=[10552], 40.00th=[11469], 50.00th=[12125], 60.00th=[12780], | 70.00th=[13698], 80.00th=[14746], 90.00th=[16319], 95.00th=[17695], | 99.00th=[23725], 99.50th=[29492], 99.90th=[35914], 99.95th=[35914], | 99.99th=[35914] bw ( KiB/s): min= 702, max= 1152, per=26.40%, avg=940.00, stdev=160.11, samples=13 iops : min= 43, max= 72, avg=58.62, stdev=10.17, samples=13 write: IOPS=56, BW=911KiB/s (933kB/s)(6032KiB/6620msec) clat (usec): min=694, max=74427, avg=4697.96, stdev=5524.20 lat (usec): min=695, max=74428, avg=4699.29, stdev=5524.18 clat percentiles (usec): | 1.00th=[ 799], 5.00th=[ 930], 10.00th=[ 1020], 20.00th=[ 1172], | 30.00th=[ 2089], 40.00th=[ 2704], 50.00th=[ 3097], 60.00th=[ 3752], | 70.00th=[ 5211], 80.00th=[ 6915], 90.00th=[10421], 95.00th=[14484], | 99.00th=[20841], 99.50th=[24249], 99.90th=[73925], 99.95th=[73925], | 99.99th=[73925] bw ( KiB/s): min= 640, max= 1312, per=23.91%, avg=895.62, stdev=197.05, samples=13 iops : min= 40, max= 82, avg=55.85, stdev=12.27, samples=13 lat (usec) : 100=0.78%, 250=0.26%, 750=0.39%, 1000=4.17% lat (msec) : 2=10.03%, 4=15.89%, 10=23.57%, 20=43.36%, 50=1.43% lat (msec) : 100=0.13% cpu : usr=0.14%, sys=20.02%, ctx=2887, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=21011: Thu Dec 5 08:24:18 2024 read: IOPS=56, BW=897KiB/s (918kB/s)(6032KiB/6728msec) clat (usec): min=53, max=33764, avg=12386.26, stdev=4271.44 lat (usec): min=53, max=33764, avg=12388.46, stdev=4271.04 clat percentiles (usec): | 1.00th=[ 85], 5.00th=[ 7111], 10.00th=[ 8717], 20.00th=[ 9765], | 30.00th=[10552], 40.00th=[11338], 50.00th=[12125], 60.00th=[12911], | 70.00th=[13960], 80.00th=[15139], 90.00th=[17171], 95.00th=[19006], | 99.00th=[25822], 99.50th=[27657], 99.90th=[33817], 99.95th=[33817], | 99.99th=[33817] bw ( KiB/s): min= 576, max= 1056, per=24.67%, avg=878.31, stdev=148.42, samples=13 iops : min= 36, max= 66, avg=54.69, stdev= 9.40, samples=13 write: IOPS=58, BW=930KiB/s (952kB/s)(6256KiB/6728msec) clat (usec): min=667, max=42323, avg=5130.69, stdev=5258.23 lat (usec): min=668, max=42324, avg=5131.93, stdev=5258.23 clat percentiles (usec): | 1.00th=[ 717], 5.00th=[ 865], 10.00th=[ 979], 20.00th=[ 1516], | 30.00th=[ 2442], 40.00th=[ 2868], 50.00th=[ 3294], 60.00th=[ 3949], | 70.00th=[ 5407], 80.00th=[ 6980], 90.00th=[11863], 95.00th=[15664], | 99.00th=[24249], 99.50th=[29754], 99.90th=[42206], 99.95th=[42206], | 99.99th=[42206] bw ( KiB/s): min= 606, max= 1312, per=24.50%, avg=917.69, stdev=237.01, samples=13 iops : min= 37, max= 82, avg=57.15, stdev=14.94, samples=13 lat (usec) : 100=0.91%, 250=0.78%, 750=0.78%, 1000=4.95% lat (msec) : 2=6.64%, 4=18.49%, 10=22.79%, 20=41.67%, 50=2.99% cpu : usr=0.25%, sys=19.69%, ctx=2759, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=3560KiB/s (3645kB/s), 866KiB/s-945KiB/s (887kB/s-968kB/s), io=23.4MiB (24.5MB), run=6620-6728msec WRITE: bw=3746KiB/s (3835kB/s), 911KiB/s-987KiB/s (933kB/s-1011kB/s), io=24.6MiB (25.8MB), run=6620-6728msec rand-rw: (groupid=0, jobs=1): err= 0: pid=21012: Thu Dec 5 08:24:18 2024 read: IOPS=55, BW=881KiB/s (902kB/s)(5920KiB/6722msec) clat (usec): min=1753, max=28900, avg=6609.60, stdev=5154.65 lat (usec): min=1754, max=28901, avg=6610.34, stdev=5154.68 clat percentiles (usec): | 1.00th=[ 1909], 5.00th=[ 2278], 10.00th=[ 2474], 20.00th=[ 2868], | 30.00th=[ 3326], 40.00th=[ 3556], 50.00th=[ 4178], 60.00th=[ 5080], | 70.00th=[ 8160], 80.00th=[10159], 90.00th=[13566], 95.00th=[17171], | 99.00th=[25822], 99.50th=[27919], 99.90th=[28967], 99.95th=[28967], | 99.99th=[28967] bw ( KiB/s): min= 574, max= 1088, per=24.39%, avg=865.92, stdev=155.99, samples=13 iops : min= 35, max= 68, avg=53.92, stdev= 9.83, samples=13 write: IOPS=59, BW=947KiB/s (970kB/s)(6368KiB/6722msec) clat (usec): min=2863, max=47543, avg=10614.19, stdev=6388.26 lat (usec): min=2865, max=47545, avg=10615.70, stdev=6388.22 clat percentiles (usec): | 1.00th=[ 3228], 5.00th=[ 3785], 10.00th=[ 4293], 20.00th=[ 5276], | 30.00th=[ 6259], 40.00th=[ 7570], 50.00th=[ 9110], 60.00th=[10683], | 70.00th=[12518], 80.00th=[15139], 90.00th=[19792], 95.00th=[23200], | 99.00th=[32375], 99.50th=[39584], 99.90th=[47449], 99.95th=[47449], | 99.99th=[47449] bw ( KiB/s): min= 640, max= 1248, per=25.35%, avg=947.08, stdev=227.33, samples=13 iops : min= 40, max= 78, avg=59.00, stdev=14.16, samples=13 lat (msec) : 2=0.65%, 4=26.17%, 10=40.49%, 20=26.43%, 50=6.25% cpu : usr=0.07%, sys=9.72%, ctx=1144, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=21013: Thu Dec 5 08:24:18 2024 read: IOPS=60, BW=960KiB/s (983kB/s)(5744KiB/5983msec) clat (usec): min=1983, max=27631, avg=6099.45, stdev=4471.67 lat (usec): min=1983, max=27632, avg=6100.28, stdev=4471.59 clat percentiles (usec): | 1.00th=[ 2040], 5.00th=[ 2245], 10.00th=[ 2409], 20.00th=[ 2704], | 30.00th=[ 3032], 40.00th=[ 3490], 50.00th=[ 4293], 60.00th=[ 5538], | 70.00th=[ 7177], 80.00th=[ 9110], 90.00th=[11994], 95.00th=[15664], | 99.00th=[21890], 99.50th=[26346], 99.90th=[27657], 99.95th=[27657], | 99.99th=[27657] bw ( KiB/s): min= 736, max= 1245, per=26.93%, avg=956.36, stdev=185.68, samples=11 iops : min= 46, max= 77, avg=59.55, stdev=11.46, samples=11 write: IOPS=68, BW=1094KiB/s (1120kB/s)(6544KiB/5983msec) clat (usec): min=2727, max=68024, avg=9232.88, stdev=6660.41 lat (usec): min=2728, max=68026, avg=9234.37, stdev=6660.56 clat percentiles (usec): | 1.00th=[ 2966], 5.00th=[ 3490], 10.00th=[ 3818], 20.00th=[ 4424], | 30.00th=[ 5014], 40.00th=[ 5997], 50.00th=[ 7046], 60.00th=[ 8586], | 70.00th=[10683], 80.00th=[13566], 90.00th=[17171], 95.00th=[20055], | 99.00th=[32900], 99.50th=[40109], 99.90th=[67634], 99.95th=[67634], | 99.99th=[67634] bw ( KiB/s): min= 768, max= 1500, per=28.63%, avg=1069.73, stdev=204.38, samples=11 iops : min= 48, max= 93, avg=66.64, stdev=12.69, samples=11 lat (msec) : 2=0.26%, 4=28.39%, 10=45.96%, 20=21.87%, 50=3.39% lat (msec) : 100=0.13% cpu : usr=0.17%, sys=9.55%, ctx=985, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=21014: Thu Dec 5 08:24:18 2024 read: IOPS=58, BW=931KiB/s (953kB/s)(6256KiB/6720msec) clat (usec): min=1840, max=28771, avg=7026.42, stdev=5032.24 lat (usec): min=1841, max=28772, avg=7027.34, stdev=5032.41 clat percentiles (usec): | 1.00th=[ 1991], 5.00th=[ 2278], 10.00th=[ 2573], 20.00th=[ 2933], | 30.00th=[ 3294], 40.00th=[ 4015], 50.00th=[ 4752], 60.00th=[ 6849], | 70.00th=[ 9110], 80.00th=[11469], 90.00th=[13566], 95.00th=[17171], | 99.00th=[23462], 99.50th=[28705], 99.90th=[28705], 99.95th=[28705], | 99.99th=[28705] bw ( KiB/s): min= 640, max= 1248, per=26.33%, avg=934.92, stdev=172.12, samples=13 iops : min= 40, max= 78, avg=58.23, stdev=10.86, samples=13 write: IOPS=56, BW=898KiB/s (919kB/s)(6032KiB/6720msec) clat (usec): min=2965, max=47028, avg=10409.13, stdev=6178.03 lat (usec): min=2966, max=47030, avg=10410.59, stdev=6178.08 clat percentiles (usec): | 1.00th=[ 3326], 5.00th=[ 3916], 10.00th=[ 4359], 20.00th=[ 4948], | 30.00th=[ 6063], 40.00th=[ 7570], 50.00th=[ 8979], 60.00th=[10552], | 70.00th=[12387], 80.00th=[15008], 90.00th=[18482], 95.00th=[22152], | 99.00th=[25822], 99.50th=[46400], 99.90th=[46924], 99.95th=[46924], | 99.99th=[46924] bw ( KiB/s): min= 640, max= 1213, per=23.51%, avg=878.15, stdev=196.73, samples=13 iops : min= 40, max= 75, avg=54.69, stdev=12.07, samples=13 lat (msec) : 2=0.52%, 4=22.92%, 10=42.45%, 20=28.91%, 50=5.21% cpu : usr=0.15%, sys=9.87%, ctx=1182, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=21015: Thu Dec 5 08:24:18 2024 read: IOPS=55, BW=894KiB/s (916kB/s)(6032KiB/6744msec) clat (usec): min=1893, max=28767, avg=6635.29, stdev=4878.47 lat (usec): min=1894, max=28767, avg=6635.99, stdev=4878.48 clat percentiles (usec): | 1.00th=[ 2114], 5.00th=[ 2343], 10.00th=[ 2507], 20.00th=[ 2868], | 30.00th=[ 3261], 40.00th=[ 3621], 50.00th=[ 4293], 60.00th=[ 5800], | 70.00th=[ 8455], 80.00th=[10421], 90.00th=[12649], 95.00th=[16712], | 99.00th=[22676], 99.50th=[27657], 99.90th=[28705], 99.95th=[28705], | 99.99th=[28705] bw ( KiB/s): min= 608, max= 1216, per=24.67%, avg=876.15, stdev=179.33, samples=13 iops : min= 38, max= 76, avg=54.69, stdev=11.19, samples=13 write: IOPS=57, BW=928KiB/s (950kB/s)(6256KiB/6744msec) clat (usec): min=2502, max=50952, avg=10703.78, stdev=6248.07 lat (usec): min=2504, max=50954, avg=10705.14, stdev=6248.06 clat percentiles (usec): | 1.00th=[ 2835], 5.00th=[ 3720], 10.00th=[ 4359], 20.00th=[ 5604], | 30.00th=[ 6718], 40.00th=[ 8094], 50.00th=[ 9372], 60.00th=[10683], | 70.00th=[12256], 80.00th=[15664], 90.00th=[19006], 95.00th=[22152], | 99.00th=[33817], 99.50th=[36963], 99.90th=[51119], 99.95th=[51119], | 99.99th=[51119] bw ( KiB/s): min= 574, max= 1280, per=24.44%, avg=913.08, stdev=234.16, samples=13 iops : min= 35, max= 80, avg=57.00, stdev=14.74, samples=13 lat (msec) : 2=0.26%, 4=26.17%, 10=39.06%, 20=29.17%, 50=5.21% lat (msec) : 100=0.13% cpu : usr=0.15%, sys=9.95%, ctx=1162, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=3552KiB/s (3637kB/s), 881KiB/s-960KiB/s (902kB/s-983kB/s), io=23.4MiB (24.5MB), run=5983-6744msec WRITE: bw=3737KiB/s (3826kB/s), 898KiB/s-1094KiB/s (919kB/s-1120kB/s), io=24.6MiB (25.8MB), run=5983-6744msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=21026: Thu Dec 5 08:24:21 2024 read: IOPS=2, BW=2758KiB/s (2824kB/s)(7168KiB/2599msec) clat (usec): min=420, max=847937, avg=264777.65, stdev=333853.60 lat (usec): min=421, max=847937, avg=264778.71, stdev=333853.41 clat percentiles (usec): | 1.00th=[ 420], 5.00th=[ 420], 10.00th=[ 420], 20.00th=[ 486], | 30.00th=[ 10814], 40.00th=[ 10814], 50.00th=[ 58983], 60.00th=[459277], | 70.00th=[459277], 80.00th=[476054], 90.00th=[851444], 95.00th=[851444], | 99.00th=[851444], 99.50th=[851444], 99.90th=[851444], 99.95th=[851444], | 99.99th=[851444] bw ( KiB/s): min= 1799, max= 4096, per=29.37%, avg=2312.20, stdev=1002.69, samples=5 iops : min= 1, max= 4, avg= 1.80, stdev= 1.30, samples=5 write: IOPS=1, BW=1970KiB/s (2017kB/s)(5120KiB/2599msec) clat (msec): min=32, max=480, avg=136.89, stdev=192.85 lat (msec): min=32, max=480, avg=136.98, stdev=192.83 clat percentiles (msec): | 1.00th=[ 33], 5.00th=[ 33], 10.00th=[ 33], 20.00th=[ 33], | 30.00th=[ 43], 40.00th=[ 43], 50.00th=[ 56], 60.00th=[ 56], | 70.00th=[ 74], 80.00th=[ 74], 90.00th=[ 481], 95.00th=[ 481], | 99.00th=[ 481], 99.50th=[ 481], 99.90th=[ 481], 99.95th=[ 481], | 99.99th=[ 481] bw ( KiB/s): min= 1799, max= 3618, per=21.85%, avg=2408.67, stdev=1047.33, samples=3 iops : min= 1, max= 3, avg= 1.67, stdev= 1.15, samples=3 lat (usec) : 500=16.67% lat (msec) : 20=8.33%, 50=16.67%, 100=25.00%, 500=25.00%, 1000=8.33% cpu : usr=0.00%, sys=36.80%, ctx=719, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=21028: Thu Dec 5 08:24:21 2024 read: IOPS=0, BW=787KiB/s (806kB/s)(2048KiB/2601msec) clat (msec): min=57, max=479, avg=268.13, stdev=298.37 lat (msec): min=57, max=479, avg=268.13, stdev=298.37 clat percentiles (msec): | 1.00th=[ 58], 5.00th=[ 58], 10.00th=[ 58], 20.00th=[ 58], | 30.00th=[ 58], 40.00th=[ 58], 50.00th=[ 58], 60.00th=[ 481], | 70.00th=[ 481], 80.00th=[ 481], 90.00th=[ 481], 95.00th=[ 481], | 99.00th=[ 481], 99.50th=[ 481], 99.90th=[ 481], 99.95th=[ 481], | 99.99th=[ 481] bw ( KiB/s): min= 1802, max= 2048, per=24.45%, avg=1925.00, stdev=173.95, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 write: IOPS=3, BW=3937KiB/s (4031kB/s)(10.0MiB/2601msec) clat (msec): min=25, max=1117, avg=200.08, stdev=344.11 lat (msec): min=25, max=1117, avg=200.16, stdev=344.11 clat percentiles (msec): | 1.00th=[ 26], 5.00th=[ 26], 10.00th=[ 26], 20.00th=[ 27], | 30.00th=[ 34], 40.00th=[ 43], 50.00th=[ 54], 60.00th=[ 65], | 70.00th=[ 109], 80.00th=[ 109], 90.00th=[ 430], 95.00th=[ 1116], | 99.00th=[ 1116], 99.50th=[ 1116], 99.90th=[ 1116], 99.95th=[ 1116], | 99.99th=[ 1116] bw ( KiB/s): min= 3586, max= 6144, per=38.43%, avg=4236.50, stdev=1271.71, samples=4 iops : min= 3, max= 6, avg= 3.75, stdev= 1.50, samples=4 lat (msec) : 50=33.33%, 100=33.33%, 250=8.33%, 500=16.67% cpu : usr=0.04%, sys=14.81%, ctx=371, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=21030: Thu Dec 5 08:24:21 2024 read: IOPS=2, BW=2930KiB/s (3000kB/s)(6144KiB/2097msec) clat (msec): min=4, max=732, avg=173.54, stdev=276.80 lat (msec): min=4, max=732, avg=173.55, stdev=276.80 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 5], 10.00th=[ 5], 20.00th=[ 35], | 30.00th=[ 35], 40.00th=[ 56], 50.00th=[ 56], 60.00th=[ 95], | 70.00th=[ 121], 80.00th=[ 121], 90.00th=[ 735], 95.00th=[ 735], | 99.00th=[ 735], 99.50th=[ 735], 99.90th=[ 735], 99.95th=[ 735], | 99.99th=[ 735] bw ( KiB/s): min=10240, max=10240, per=100.00%, avg=10240.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 write: IOPS=2, BW=2930KiB/s (3000kB/s)(6144KiB/2097msec) clat (msec): min=22, max=875, avg=174.30, stdev=343.70 lat (msec): min=22, max=875, avg=174.36, stdev=343.69 clat percentiles (msec): | 1.00th=[ 23], 5.00th=[ 23], 10.00th=[ 23], 20.00th=[ 26], | 30.00th=[ 26], 40.00th=[ 27], 50.00th=[ 27], 60.00th=[ 33], | 70.00th=[ 64], 80.00th=[ 64], 90.00th=[ 877], 95.00th=[ 877], | 99.00th=[ 877], 99.50th=[ 877], 99.90th=[ 877], 99.95th=[ 877], | 99.99th=[ 877] bw ( KiB/s): min= 1793, max=10240, per=54.58%, avg=6016.50, stdev=5972.93, samples=2 iops : min= 1, max= 10, avg= 5.50, stdev= 6.36, samples=2 lat (msec) : 10=8.33%, 50=41.67%, 100=25.00%, 250=8.33%, 750=8.33% lat (msec) : 1000=8.33% cpu : usr=0.05%, sys=15.03%, ctx=393, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=21032: Thu Dec 5 08:24:21 2024 read: IOPS=3, BW=3807KiB/s (3898kB/s)(5120KiB/1345msec) clat (usec): min=414, max=619670, avg=212332.75, stdev=280720.03 lat (usec): min=416, max=619671, avg=212334.02, stdev=280720.03 clat percentiles (usec): | 1.00th=[ 416], 5.00th=[ 416], 10.00th=[ 416], 20.00th=[ 416], | 30.00th=[ 453], 40.00th=[ 453], 50.00th=[ 48497], 60.00th=[ 48497], | 70.00th=[392168], 80.00th=[392168], 90.00th=[616563], 95.00th=[616563], | 99.00th=[616563], 99.50th=[616563], 99.90th=[616563], 99.95th=[616563], | 99.99th=[616563] bw ( KiB/s): min= 2048, max= 2048, per=26.01%, avg=2048.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 write: IOPS=5, BW=5329KiB/s (5457kB/s)(7168KiB/1345msec) clat (usec): min=19509, max=77358, avg=34445.07, stdev=21031.80 lat (usec): min=19541, max=77433, avg=34504.24, stdev=21042.93 clat percentiles (usec): | 1.00th=[19530], 5.00th=[19530], 10.00th=[19530], 20.00th=[19792], | 30.00th=[21365], 40.00th=[21365], 50.00th=[26084], 60.00th=[31327], | 70.00th=[31327], 80.00th=[45351], 90.00th=[77071], 95.00th=[77071], | 99.00th=[77071], 99.50th=[77071], 99.90th=[77071], 99.95th=[77071], | 99.99th=[77071] bw ( KiB/s): min= 8192, max= 8192, per=74.32%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 lat (usec) : 500=16.67% lat (msec) : 20=16.67%, 50=41.67%, 100=8.33%, 500=8.33%, 750=8.33% cpu : usr=0.00%, sys=22.84%, ctx=602, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=7874KiB/s (8063kB/s), 787KiB/s-3807KiB/s (806kB/s-3898kB/s), io=20.0MiB (20.0MB), run=1345-2601msec WRITE: bw=10.8MiB/s (11.3MB/s), 1970KiB/s-5329KiB/s (2017kB/s-5457kB/s), io=28.0MiB (29.4MB), run=1345-2601msec rand-rw: (groupid=0, jobs=1): err= 0: pid=21027: Thu Dec 5 08:24:21 2024 read: IOPS=2, BW=2680KiB/s (2744kB/s)(7168KiB/2675msec) clat (msec): min=17, max=798, avg=262.18, stdev=298.00 lat (msec): min=17, max=798, avg=262.18, stdev=298.00 clat percentiles (msec): | 1.00th=[ 18], 5.00th=[ 18], 10.00th=[ 18], 20.00th=[ 21], | 30.00th=[ 46], 40.00th=[ 46], 50.00th=[ 102], 60.00th=[ 418], | 70.00th=[ 418], 80.00th=[ 439], 90.00th=[ 802], 95.00th=[ 802], | 99.00th=[ 802], 99.50th=[ 802], 99.90th=[ 802], 99.95th=[ 802], | 99.99th=[ 802] bw ( KiB/s): min= 1818, max= 4096, per=39.34%, avg=3012.25, stdev=1249.75, samples=4 iops : min= 1, max= 4, avg= 2.50, stdev= 1.29, samples=4 write: IOPS=1, BW=1914KiB/s (1960kB/s)(5120KiB/2675msec) clat (msec): min=45, max=503, avg=156.48, stdev=194.55 lat (msec): min=45, max=503, avg=156.56, stdev=194.53 clat percentiles (msec): | 1.00th=[ 46], 5.00th=[ 46], 10.00th=[ 46], 20.00th=[ 46], | 30.00th=[ 68], 40.00th=[ 68], 50.00th=[ 77], 60.00th=[ 77], | 70.00th=[ 90], 80.00th=[ 90], 90.00th=[ 502], 95.00th=[ 502], | 99.00th=[ 502], 99.50th=[ 502], 99.90th=[ 502], 99.95th=[ 502], | 99.99th=[ 502] bw ( KiB/s): min= 2048, max= 5456, per=35.01%, avg=3752.00, stdev=2409.82, samples=2 iops : min= 2, max= 5, avg= 3.50, stdev= 2.12, samples=2 lat (msec) : 20=8.33%, 50=25.00%, 100=25.00%, 250=8.33%, 500=16.67% lat (msec) : 750=8.33%, 1000=8.33% cpu : usr=0.07%, sys=5.27%, ctx=56, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=21029: Thu Dec 5 08:24:21 2024 read: IOPS=1, BW=1167KiB/s (1195kB/s)(2048KiB/1755msec) clat (usec): min=15009, max=91678, avg=53343.70, stdev=54213.27 lat (usec): min=15009, max=91679, avg=53344.73, stdev=54213.51 clat percentiles (usec): | 1.00th=[15008], 5.00th=[15008], 10.00th=[15008], 20.00th=[15008], | 30.00th=[15008], 40.00th=[15008], 50.00th=[15008], 60.00th=[91751], | 70.00th=[91751], 80.00th=[91751], 90.00th=[91751], 95.00th=[91751], | 99.00th=[91751], 99.50th=[91751], 99.90th=[91751], 99.95th=[91751], | 99.99th=[91751] bw ( KiB/s): min= 4096, max= 4096, per=53.50%, avg=4096.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 write: IOPS=5, BW=5835KiB/s (5975kB/s)(10.0MiB/1755msec) clat (msec): min=15, max=770, avg=163.98, stdev=226.32 lat (msec): min=16, max=770, avg=164.20, stdev=226.21 clat percentiles (msec): | 1.00th=[ 16], 5.00th=[ 16], 10.00th=[ 16], 20.00th=[ 19], | 30.00th=[ 20], 40.00th=[ 48], 50.00th=[ 89], 60.00th=[ 94], | 70.00th=[ 192], 80.00th=[ 192], 90.00th=[ 228], 95.00th=[ 768], | 99.00th=[ 768], 99.50th=[ 768], 99.90th=[ 768], 99.95th=[ 768], | 99.99th=[ 768] bw ( KiB/s): min= 2048, max=10240, per=50.95%, avg=5461.33, stdev=4263.25, samples=3 iops : min= 2, max= 10, avg= 5.33, stdev= 4.16, samples=3 lat (msec) : 20=33.33%, 50=8.33%, 100=25.00%, 250=25.00%, 1000=8.33% cpu : usr=0.00%, sys=5.53%, ctx=62, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=21031: Thu Dec 5 08:24:21 2024 read: IOPS=2, BW=2916KiB/s (2986kB/s)(6144KiB/2107msec) clat (msec): min=46, max=1072, avg=285.79, stdev=401.88 lat (msec): min=46, max=1072, avg=285.79, stdev=401.88 clat percentiles (msec): | 1.00th=[ 47], 5.00th=[ 47], 10.00th=[ 47], 20.00th=[ 50], | 30.00th=[ 50], 40.00th=[ 57], 50.00th=[ 57], 60.00th=[ 148], | 70.00th=[ 342], 80.00th=[ 342], 90.00th=[ 1070], 95.00th=[ 1070], | 99.00th=[ 1070], 99.50th=[ 1070], 99.90th=[ 1070], 99.95th=[ 1070], | 99.99th=[ 1070] bw ( KiB/s): min= 2048, max= 4096, per=44.58%, avg=3413.33, stdev=1182.41, samples=3 iops : min= 2, max= 4, avg= 3.33, stdev= 1.15, samples=3 write: IOPS=2, BW=2916KiB/s (2986kB/s)(6144KiB/2107msec) clat (msec): min=22, max=117, avg=55.72, stdev=34.13 lat (msec): min=22, max=117, avg=55.83, stdev=34.09 clat percentiles (msec): | 1.00th=[ 23], 5.00th=[ 23], 10.00th=[ 23], 20.00th=[ 36], | 30.00th=[ 36], 40.00th=[ 39], 50.00th=[ 39], 60.00th=[ 53], | 70.00th=[ 69], 80.00th=[ 69], 90.00th=[ 118], 95.00th=[ 118], | 99.00th=[ 118], 99.50th=[ 118], 99.90th=[ 118], 99.95th=[ 118], | 99.99th=[ 118] bw ( KiB/s): min= 4096, max= 8192, per=57.32%, avg=6144.00, stdev=2896.31, samples=2 iops : min= 4, max= 8, avg= 6.00, stdev= 2.83, samples=2 lat (msec) : 50=41.67%, 100=25.00%, 250=16.67%, 500=8.33% cpu : usr=0.00%, sys=4.99%, ctx=87, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=21033: Thu Dec 5 08:24:21 2024 read: IOPS=2, BW=2423KiB/s (2481kB/s)(5120KiB/2113msec) clat (msec): min=13, max=465, avg=204.23, stdev=218.42 lat (msec): min=13, max=465, avg=204.23, stdev=218.42 clat percentiles (msec): | 1.00th=[ 14], 5.00th=[ 14], 10.00th=[ 14], 20.00th=[ 14], | 30.00th=[ 20], 40.00th=[ 20], 50.00th=[ 111], 60.00th=[ 111], | 70.00th=[ 414], 80.00th=[ 414], 90.00th=[ 464], 95.00th=[ 464], | 99.00th=[ 464], 99.50th=[ 464], 99.90th=[ 464], 99.95th=[ 464], | 99.99th=[ 464] bw ( KiB/s): min= 2048, max= 6144, per=53.50%, avg=4096.00, stdev=2896.31, samples=2 iops : min= 2, max= 6, avg= 4.00, stdev= 2.83, samples=2 write: IOPS=3, BW=3392KiB/s (3474kB/s)(7168KiB/2113msec) clat (msec): min=13, max=371, avg=152.81, stdev=124.39 lat (msec): min=13, max=371, avg=152.88, stdev=124.39 clat percentiles (msec): | 1.00th=[ 14], 5.00th=[ 14], 10.00th=[ 14], 20.00th=[ 75], | 30.00th=[ 81], 40.00th=[ 81], 50.00th=[ 87], 60.00th=[ 215], | 70.00th=[ 215], 80.00th=[ 230], 90.00th=[ 372], 95.00th=[ 372], | 99.00th=[ 372], 99.50th=[ 372], 99.90th=[ 372], 99.95th=[ 372], | 99.99th=[ 372] bw ( KiB/s): min= 4087, max= 8192, per=57.28%, avg=6139.50, stdev=2902.67, samples=2 iops : min= 3, max= 8, avg= 5.50, stdev= 3.54, samples=2 lat (msec) : 20=25.00%, 100=25.00%, 250=25.00%, 500=25.00% cpu : usr=0.05%, sys=4.78%, ctx=52, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=7656KiB/s (7840kB/s), 1167KiB/s-2916KiB/s (1195kB/s-2986kB/s), io=20.0MiB (20.0MB), run=1755-2675msec WRITE: bw=10.5MiB/s (10.0MB/s), 1914KiB/s-5835KiB/s (1960kB/s-5975kB/s), io=28.0MiB (29.4MB), run=1755-2675msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=21044: Thu Dec 5 08:24:25 2024 read: IOPS=1, BW=6337KiB/s (6489kB/s)(12.0MiB/1939msec) clat (msec): min=260, max=1204, avg=611.91, stdev=516.49 lat (msec): min=260, max=1204, avg=611.91, stdev=516.49 clat percentiles (msec): | 1.00th=[ 262], 5.00th=[ 262], 10.00th=[ 262], 20.00th=[ 262], | 30.00th=[ 262], 40.00th=[ 372], 50.00th=[ 372], 60.00th=[ 372], | 70.00th=[ 1200], 80.00th=[ 1200], 90.00th=[ 1200], 95.00th=[ 1200], | 99.00th=[ 1200], 99.50th=[ 1200], 99.90th=[ 1200], 99.95th=[ 1200], | 99.99th=[ 1200] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 500=66.67% cpu : usr=0.15%, sys=33.02%, ctx=581, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=21046: Thu Dec 5 08:24:25 2024 read: IOPS=0, BW=1880KiB/s (1925kB/s)(4096KiB/2179msec) clat (nsec): min=265233k, max=265233k, avg=265232802.00, stdev= 0.00 lat (nsec): min=265235k, max=265235k, avg=265235477.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 266], 5.00th=[ 266], 10.00th=[ 266], 20.00th=[ 266], | 30.00th=[ 266], 40.00th=[ 266], 50.00th=[ 266], 60.00th=[ 266], | 70.00th=[ 266], 80.00th=[ 266], 90.00th=[ 266], 95.00th=[ 266], | 99.00th=[ 266], 99.50th=[ 266], 99.90th=[ 266], 99.95th=[ 266], | 99.99th=[ 266] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=0, BW=3760KiB/s (3850kB/s)(8192KiB/2179msec) clat (msec): min=708, max=1093, avg=900.96, stdev=272.48 lat (msec): min=708, max=1093, avg=901.28, stdev=272.42 clat percentiles (msec): | 1.00th=[ 709], 5.00th=[ 709], 10.00th=[ 709], 20.00th=[ 709], | 30.00th=[ 709], 40.00th=[ 709], 50.00th=[ 709], 60.00th=[ 1099], | 70.00th=[ 1099], 80.00th=[ 1099], 90.00th=[ 1099], 95.00th=[ 1099], | 99.00th=[ 1099], 99.50th=[ 1099], 99.90th=[ 1099], 99.95th=[ 1099], | 99.99th=[ 1099] bw ( KiB/s): min= 8192, max= 8192, per=54.48%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 500=33.33%, 750=33.33% cpu : usr=0.14%, sys=19.93%, ctx=306, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=21047: Thu Dec 5 08:24:25 2024 write: IOPS=1, BW=5699KiB/s (5836kB/s)(12.0MiB/2156msec) clat (msec): min=306, max=1102, avg=681.69, stdev=399.94 lat (msec): min=306, max=1102, avg=681.95, stdev=399.89 clat percentiles (msec): | 1.00th=[ 309], 5.00th=[ 309], 10.00th=[ 309], 20.00th=[ 309], | 30.00th=[ 309], 40.00th=[ 634], 50.00th=[ 634], 60.00th=[ 634], | 70.00th=[ 1099], 80.00th=[ 1099], 90.00th=[ 1099], 95.00th=[ 1099], | 99.00th=[ 1099], 99.50th=[ 1099], 99.90th=[ 1099], 99.95th=[ 1099], | 99.99th=[ 1099] bw ( KiB/s): min= 8062, max= 8192, per=54.04%, avg=8127.00, stdev=91.92, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 500=33.33%, 750=33.33% cpu : usr=0.14%, sys=18.52%, ctx=43, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=21048: Thu Dec 5 08:24:25 2024 write: IOPS=1, BW=7153KiB/s (7324kB/s)(12.0MiB/1718msec) clat (msec): min=134, max=1196, avg=562.34, stdev=560.33 lat (msec): min=134, max=1196, avg=562.71, stdev=560.34 clat percentiles (msec): | 1.00th=[ 136], 5.00th=[ 136], 10.00th=[ 136], 20.00th=[ 136], | 30.00th=[ 136], 40.00th=[ 355], 50.00th=[ 355], 60.00th=[ 355], | 70.00th=[ 1200], 80.00th=[ 1200], 90.00th=[ 1200], 95.00th=[ 1200], | 99.00th=[ 1200], 99.50th=[ 1200], 99.90th=[ 1200], 99.95th=[ 1200], | 99.99th=[ 1200] bw ( KiB/s): min= 8192, max= 8192, per=54.48%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=33.33% cpu : usr=0.06%, sys=18.99%, ctx=127, majf=0, minf=28 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=7519KiB/s (7700kB/s), 1880KiB/s-6337KiB/s (1925kB/s-6489kB/s), io=16.0MiB (16.8MB), run=1939-2179msec WRITE: bw=14.7MiB/s (15.4MB/s), 3760KiB/s-7153KiB/s (3850kB/s-7324kB/s), io=32.0MiB (33.6MB), run=1718-2179msec rand-rw: (groupid=0, jobs=1): err= 0: pid=21041: Thu Dec 5 08:24:25 2024 read: IOPS=1, BW=6219KiB/s (6368kB/s)(12.0MiB/1976msec) clat (msec): min=277, max=1297, avg=643.28, stdev=567.74 lat (msec): min=277, max=1297, avg=643.28, stdev=567.74 clat percentiles (msec): | 1.00th=[ 279], 5.00th=[ 279], 10.00th=[ 279], 20.00th=[ 279], | 30.00th=[ 279], 40.00th=[ 355], 50.00th=[ 355], 60.00th=[ 355], | 70.00th=[ 1301], 80.00th=[ 1301], 90.00th=[ 1301], 95.00th=[ 1301], | 99.00th=[ 1301], 99.50th=[ 1301], 99.90th=[ 1301], 99.95th=[ 1301], | 99.99th=[ 1301] bw ( KiB/s): min= 5535, max= 8192, per=95.43%, avg=6863.50, stdev=1878.78, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 500=66.67% cpu : usr=0.15%, sys=3.44%, ctx=39, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=21042: Thu Dec 5 08:24:25 2024 read: IOPS=0, BW=1798KiB/s (1841kB/s)(4096KiB/2278msec) clat (nsec): min=48381k, max=48381k, avg=48381061.00, stdev= 0.00 lat (nsec): min=48383k, max=48383k, avg=48383210.00, stdev= 0.00 clat percentiles (usec): | 1.00th=[48497], 5.00th=[48497], 10.00th=[48497], 20.00th=[48497], | 30.00th=[48497], 40.00th=[48497], 50.00th=[48497], 60.00th=[48497], | 70.00th=[48497], 80.00th=[48497], 90.00th=[48497], 95.00th=[48497], | 99.00th=[48497], 99.50th=[48497], 99.90th=[48497], 99.95th=[48497], | 99.99th=[48497] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=0, BW=3596KiB/s (3682kB/s)(8192KiB/2278msec) clat (msec): min=985, max=1213, avg=1099.26, stdev=161.10 lat (msec): min=985, max=1213, avg=1099.58, stdev=161.10 clat percentiles (msec): | 1.00th=[ 986], 5.00th=[ 986], 10.00th=[ 986], 20.00th=[ 986], | 30.00th=[ 986], 40.00th=[ 986], 50.00th=[ 986], 60.00th=[ 1217], | 70.00th=[ 1217], 80.00th=[ 1217], 90.00th=[ 1217], 95.00th=[ 1217], | 99.00th=[ 1217], 99.50th=[ 1217], 99.90th=[ 1217], 99.95th=[ 1217], | 99.99th=[ 1217] bw ( KiB/s): min= 8175, max= 8175, per=57.51%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 50=33.33%, 1000=33.33% cpu : usr=0.00%, sys=7.64%, ctx=69, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=21043: Thu Dec 5 08:24:25 2024 write: IOPS=1, BW=5331KiB/s (5459kB/s)(12.0MiB/2305msec) clat (msec): min=255, max=1343, avg=757.01, stdev=548.72 lat (msec): min=255, max=1343, avg=757.34, stdev=548.81 clat percentiles (msec): | 1.00th=[ 255], 5.00th=[ 255], 10.00th=[ 255], 20.00th=[ 255], | 30.00th=[ 255], 40.00th=[ 676], 50.00th=[ 676], 60.00th=[ 676], | 70.00th=[ 1351], 80.00th=[ 1351], 90.00th=[ 1351], 95.00th=[ 1351], | 99.00th=[ 1351], 99.50th=[ 1351], 99.90th=[ 1351], 99.95th=[ 1351], | 99.99th=[ 1351] bw ( KiB/s): min= 5520, max= 5520, per=38.83%, avg=5520.00, stdev= 0.00, samples=2 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=2 lat (msec) : 500=33.33%, 750=33.33% cpu : usr=0.09%, sys=7.03%, ctx=72, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=21045: Thu Dec 5 08:24:25 2024 write: IOPS=1, BW=8159KiB/s (8355kB/s)(12.0MiB/1506msec) clat (msec): min=38, max=1231, avg=494.61, stdev=643.92 lat (msec): min=39, max=1231, avg=494.94, stdev=644.03 clat percentiles (msec): | 1.00th=[ 40], 5.00th=[ 40], 10.00th=[ 40], 20.00th=[ 40], | 30.00th=[ 40], 40.00th=[ 213], 50.00th=[ 213], 60.00th=[ 213], | 70.00th=[ 1234], 80.00th=[ 1234], 90.00th=[ 1234], 95.00th=[ 1234], | 99.00th=[ 1234], 99.50th=[ 1234], 99.90th=[ 1234], 99.95th=[ 1234], | 99.99th=[ 1234] bw ( KiB/s): min=16384, max=16384, per=100.00%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 50=33.33%, 250=33.33% cpu : usr=0.07%, sys=6.64%, ctx=34, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=7192KiB/s (7365kB/s), 1798KiB/s-6219KiB/s (1841kB/s-6368kB/s), io=16.0MiB (16.8MB), run=1976-2278msec WRITE: bw=13.9MiB/s (14.6MB/s), 3596KiB/s-8159KiB/s (3682kB/s-8355kB/s), io=32.0MiB (33.6MB), run=1506-2305msec