== sanity test 398b: DIO and buffer IO race ============== 04:48:20 (1738489700) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.50752 s, 33.4 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=27713: Sun Feb 2 04:48:39 2025 read: IOPS=94, BW=379KiB/s (388kB/s)(5992KiB/15828msec) clat (usec): min=61, max=190157, avg=5849.11, stdev=5807.35 lat (usec): min=61, max=190160, avg=5851.94, stdev=5807.86 clat percentiles (usec): | 1.00th=[ 86], 5.00th=[ 128], 10.00th=[ 2769], 20.00th=[ 3359], | 30.00th=[ 3818], 40.00th=[ 4424], 50.00th=[ 5080], 60.00th=[ 5932], | 70.00th=[ 6980], 80.00th=[ 7963], 90.00th=[ 9634], 95.00th=[ 11338], | 99.00th=[ 17695], 99.50th=[ 20055], 99.90th=[ 32637], 99.95th=[189793], | 99.99th=[189793] bw ( KiB/s): min= 184, max= 512, per=24.71%, avg=374.84, stdev=65.57, samples=31 iops : min= 46, max= 128, avg=93.61, stdev=16.41, samples=31 write: IOPS=99, BW=398KiB/s (407kB/s)(6296KiB/15828msec) clat (usec): min=474, max=22585, avg=4420.02, stdev=4018.38 lat (usec): min=474, max=22587, avg=4421.60, stdev=4018.02 clat percentiles (usec): | 1.00th=[ 570], 5.00th=[ 668], 10.00th=[ 717], 20.00th=[ 799], | 30.00th=[ 963], 40.00th=[ 1942], 50.00th=[ 3589], 60.00th=[ 4490], | 70.00th=[ 5604], 80.00th=[ 7635], 90.00th=[10290], 95.00th=[12387], | 99.00th=[16450], 99.50th=[19530], 99.90th=[22152], 99.95th=[22676], | 99.99th=[22676] bw ( KiB/s): min= 224, max= 552, per=24.81%, avg=393.94, stdev=65.64, samples=31 iops : min= 56, max= 138, avg=98.39, stdev=16.41, samples=31 lat (usec) : 100=1.27%, 250=1.56%, 500=0.10%, 750=7.00%, 1000=9.15% lat (msec) : 2=4.49%, 4=20.74%, 10=45.83%, 20=9.41%, 50=0.42% lat (msec) : 250=0.03% cpu : usr=0.26%, sys=20.95%, ctx=5679, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=27714: Sun Feb 2 04:48:39 2025 read: IOPS=94, BW=378KiB/s (387kB/s)(5940KiB/15733msec) clat (usec): min=59, max=30397, avg=5607.46, stdev=3171.28 lat (usec): min=60, max=30398, avg=5609.40, stdev=3170.70 clat percentiles (usec): | 1.00th=[ 84], 5.00th=[ 120], 10.00th=[ 2737], 20.00th=[ 3294], | 30.00th=[ 3818], 40.00th=[ 4359], 50.00th=[ 5014], 60.00th=[ 5932], | 70.00th=[ 6783], 80.00th=[ 7832], 90.00th=[ 9372], 95.00th=[10945], | 99.00th=[16581], 99.50th=[17957], 99.90th=[24511], 99.95th=[30278], | 99.99th=[30278] bw ( KiB/s): min= 208, max= 496, per=24.79%, avg=376.10, stdev=56.82, samples=31 iops : min= 52, max= 124, avg=93.94, stdev=14.20, samples=31 write: IOPS=100, BW=403KiB/s (413kB/s)(6348KiB/15733msec) clat (usec): min=475, max=204050, avg=4612.68, stdev=6590.41 lat (usec): min=476, max=204051, avg=4614.96, stdev=6589.97 clat percentiles (usec): | 1.00th=[ 586], 5.00th=[ 668], 10.00th=[ 717], 20.00th=[ 783], | 30.00th=[ 947], 40.00th=[ 1909], 50.00th=[ 3556], 60.00th=[ 4424], | 70.00th=[ 5538], 80.00th=[ 7635], 90.00th=[ 10159], 95.00th=[ 13042], | 99.00th=[ 19792], 99.50th=[ 21890], 99.90th=[ 26870], 99.95th=[204473], | 99.99th=[204473] bw ( KiB/s): min= 264, max= 536, per=25.10%, avg=398.55, stdev=58.20, samples=31 iops : min= 66, max= 134, avg=99.55, stdev=14.64, samples=31 lat (usec) : 100=1.37%, 250=1.50%, 500=0.16%, 750=7.62%, 1000=8.46% lat (msec) : 2=4.65%, 4=21.00%, 10=45.93%, 20=8.79%, 50=0.49% lat (msec) : 250=0.03% cpu : usr=0.24%, sys=21.06%, ctx=5564, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=27715: Sun Feb 2 04:48:39 2025 read: IOPS=96, BW=384KiB/s (393kB/s)(6028KiB/15694msec) clat (usec): min=57, max=26751, avg=5766.55, stdev=3230.21 lat (usec): min=58, max=26752, avg=5767.89, stdev=3230.21 clat percentiles (usec): | 1.00th=[ 87], 5.00th=[ 157], 10.00th=[ 2769], 20.00th=[ 3359], | 30.00th=[ 3884], 40.00th=[ 4424], 50.00th=[ 5145], 60.00th=[ 6063], | 70.00th=[ 6980], 80.00th=[ 8160], 90.00th=[ 9765], 95.00th=[11076], | 99.00th=[16319], 99.50th=[20055], 99.90th=[24773], 99.95th=[26870], | 99.99th=[26870] bw ( KiB/s): min= 200, max= 464, per=25.17%, avg=381.77, stdev=56.54, samples=31 iops : min= 50, max= 116, avg=95.35, stdev=14.24, samples=31 write: IOPS=99, BW=399KiB/s (408kB/s)(6260KiB/15694msec) clat (usec): min=475, max=204066, avg=4420.12, stdev=6381.70 lat (usec): min=476, max=204066, avg=4421.26, stdev=6381.83 clat percentiles (usec): | 1.00th=[ 537], 5.00th=[ 660], 10.00th=[ 709], 20.00th=[ 783], | 30.00th=[ 930], 40.00th=[ 1926], 50.00th=[ 3621], 60.00th=[ 4359], | 70.00th=[ 5407], 80.00th=[ 7308], 90.00th=[ 9503], 95.00th=[ 11994], | 99.00th=[ 16909], 99.50th=[ 18744], 99.90th=[ 26084], 99.95th=[204473], | 99.99th=[204473] bw ( KiB/s): min= 240, max= 520, per=24.93%, avg=395.97, stdev=76.09, samples=31 iops : min= 60, max= 130, avg=98.90, stdev=19.04, samples=31 lat (usec) : 100=1.30%, 250=1.27%, 500=0.20%, 750=7.36%, 1000=8.95% lat (msec) : 2=4.23%, 4=21.29%, 10=46.48%, 20=8.56%, 50=0.33% lat (msec) : 250=0.03% cpu : usr=0.23%, sys=21.42%, ctx=5740, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=27716: Sun Feb 2 04:48:39 2025 read: IOPS=95, BW=384KiB/s (393kB/s)(6052KiB/15776msec) clat (usec): min=59, max=206080, avg=5896.96, stdev=6135.20 lat (usec): min=59, max=206081, avg=5898.00, stdev=6135.28 clat percentiles (usec): | 1.00th=[ 90], 5.00th=[ 131], 10.00th=[ 2835], 20.00th=[ 3359], | 30.00th=[ 3818], 40.00th=[ 4359], 50.00th=[ 5145], 60.00th=[ 5932], | 70.00th=[ 6980], 80.00th=[ 8094], 90.00th=[ 9765], 95.00th=[ 11338], | 99.00th=[ 18220], 99.50th=[ 19792], 99.90th=[ 27395], 99.95th=[206570], | 99.99th=[206570] bw ( KiB/s): min= 232, max= 480, per=25.10%, avg=380.74, stdev=55.37, samples=31 iops : min= 58, max= 120, avg=95.10, stdev=13.92, samples=31 write: IOPS=98, BW=395KiB/s (405kB/s)(6236KiB/15776msec) clat (usec): min=472, max=33742, avg=4345.47, stdev=4148.19 lat (usec): min=473, max=33742, avg=4347.36, stdev=4148.95 clat percentiles (usec): | 1.00th=[ 586], 5.00th=[ 668], 10.00th=[ 717], 20.00th=[ 791], | 30.00th=[ 938], 40.00th=[ 1713], 50.00th=[ 3490], 60.00th=[ 4359], | 70.00th=[ 5669], 80.00th=[ 7570], 90.00th=[ 9634], 95.00th=[11994], | 99.00th=[18220], 99.50th=[22414], 99.90th=[33424], 99.95th=[33817], | 99.99th=[33817] bw ( KiB/s): min= 280, max= 600, per=24.67%, avg=391.84, stdev=71.38, samples=31 iops : min= 70, max= 150, avg=97.87, stdev=17.85, samples=31 lat (usec) : 100=1.24%, 250=1.40%, 500=0.10%, 750=7.29%, 1000=9.08% lat (msec) : 2=4.46%, 4=21.55%, 10=46.06%, 20=8.20%, 50=0.59% lat (msec) : 250=0.03% cpu : usr=0.25%, sys=20.99%, ctx=5751, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1517KiB/s (1553kB/s), 378KiB/s-384KiB/s (387kB/s-393kB/s), io=23.4MiB (24.6MB), run=15694-15828msec WRITE: bw=1588KiB/s (1626kB/s), 395KiB/s-403KiB/s (405kB/s-413kB/s), io=24.6MiB (25.7MB), run=15694-15828msec rand-rw: (groupid=0, jobs=1): err= 0: pid=27717: Sun Feb 2 04:48:45 2025 read: IOPS=67, BW=271KiB/s (278kB/s)(5992KiB/22076msec) clat (usec): min=1804, max=206403, avg=5640.43, stdev=6591.42 lat (usec): min=1805, max=206404, avg=5641.52, stdev=6591.51 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 3], 20.00th=[ 3], | 30.00th=[ 4], 40.00th=[ 4], 50.00th=[ 4], 60.00th=[ 5], | 70.00th=[ 6], 80.00th=[ 8], 90.00th=[ 12], 95.00th=[ 15], | 99.00th=[ 22], 99.50th=[ 24], 99.90th=[ 30], 99.95th=[ 207], | 99.99th=[ 207] bw ( KiB/s): min= 112, max= 552, per=24.98%, avg=270.32, stdev=114.02, samples=44 iops : min= 28, max= 138, avg=67.55, stdev=28.49, samples=44 write: IOPS=71, BW=285KiB/s (292kB/s)(6296KiB/22076msec) clat (usec): min=2721, max=46298, avg=8613.11, stdev=4937.04 lat (usec): min=2722, max=46298, avg=8614.34, stdev=4936.98 clat percentiles (usec): | 1.00th=[ 3490], 5.00th=[ 4228], 10.00th=[ 4621], 20.00th=[ 5145], | 30.00th=[ 5669], 40.00th=[ 6128], 50.00th=[ 6652], 60.00th=[ 7635], | 70.00th=[ 9503], 80.00th=[11863], 90.00th=[15139], 95.00th=[18482], | 99.00th=[26346], 99.50th=[30802], 99.90th=[42730], 99.95th=[46400], | 99.99th=[46400] bw ( KiB/s): min= 128, max= 544, per=25.09%, avg=284.32, stdev=117.80, samples=44 iops : min= 32, max= 136, avg=71.05, stdev=29.44, samples=44 lat (msec) : 2=0.10%, 4=27.54%, 10=52.21%, 20=17.58%, 50=2.54% lat (msec) : 250=0.03% cpu : usr=0.16%, sys=10.40%, ctx=3589, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=27718: Sun Feb 2 04:48:45 2025 read: IOPS=66, BW=268KiB/s (274kB/s)(5940KiB/22181msec) clat (msec): min=2, max=211, avg= 5.70, stdev= 6.85 lat (msec): min=2, max=211, avg= 5.70, stdev= 6.85 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 3], 20.00th=[ 3], | 30.00th=[ 4], 40.00th=[ 4], 50.00th=[ 4], 60.00th=[ 5], | 70.00th=[ 6], 80.00th=[ 8], 90.00th=[ 12], 95.00th=[ 16], | 99.00th=[ 22], 99.50th=[ 24], 99.90th=[ 34], 99.95th=[ 211], | 99.99th=[ 211] bw ( KiB/s): min= 88, max= 544, per=24.66%, avg=266.84, stdev=119.21, samples=44 iops : min= 22, max= 136, avg=66.66, stdev=29.80, samples=44 write: IOPS=71, BW=286KiB/s (293kB/s)(6348KiB/22181msec) clat (usec): min=2854, max=34365, avg=8596.33, stdev=4743.00 lat (usec): min=2855, max=34366, avg=8597.50, stdev=4743.07 clat percentiles (usec): | 1.00th=[ 3523], 5.00th=[ 4146], 10.00th=[ 4555], 20.00th=[ 5145], | 30.00th=[ 5604], 40.00th=[ 6128], 50.00th=[ 6783], 60.00th=[ 7767], | 70.00th=[ 9503], 80.00th=[11731], 90.00th=[15139], 95.00th=[17957], | 99.00th=[26084], 99.50th=[27919], 99.90th=[34341], 99.95th=[34341], | 99.99th=[34341] bw ( KiB/s): min= 152, max= 560, per=25.03%, avg=283.57, stdev=114.77, samples=44 iops : min= 38, max= 140, avg=70.84, stdev=28.68, samples=44 lat (msec) : 4=27.90%, 10=51.53%, 20=17.94%, 50=2.60%, 250=0.03% cpu : usr=0.23%, sys=10.39%, ctx=3563, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=27719: Sun Feb 2 04:48:45 2025 read: IOPS=68, BW=273KiB/s (279kB/s)(6028KiB/22119msec) clat (usec): min=2065, max=30507, avg=5607.65, stdev=4332.00 lat (usec): min=2066, max=30508, avg=5608.71, stdev=4331.96 clat percentiles (usec): | 1.00th=[ 2212], 5.00th=[ 2409], 10.00th=[ 2606], 20.00th=[ 2868], | 30.00th=[ 3097], 40.00th=[ 3425], 50.00th=[ 3818], 60.00th=[ 4293], | 70.00th=[ 5342], 80.00th=[ 7635], 90.00th=[11994], 95.00th=[15008], | 99.00th=[22414], 99.50th=[23987], 99.90th=[29754], 99.95th=[30540], | 99.99th=[30540] bw ( KiB/s): min= 112, max= 504, per=25.09%, avg=271.52, stdev=108.15, samples=44 iops : min= 28, max= 126, avg=67.80, stdev=27.06, samples=44 write: IOPS=70, BW=283KiB/s (290kB/s)(6260KiB/22119msec) clat (msec): min=3, max=195, avg= 8.68, stdev= 6.67 lat (msec): min=3, max=195, avg= 8.68, stdev= 6.67 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 5], 10.00th=[ 5], 20.00th=[ 6], | 30.00th=[ 6], 40.00th=[ 7], 50.00th=[ 7], 60.00th=[ 8], | 70.00th=[ 10], 80.00th=[ 12], 90.00th=[ 16], 95.00th=[ 19], | 99.00th=[ 26], 99.50th=[ 30], 99.90th=[ 39], 99.95th=[ 197], | 99.99th=[ 197] bw ( KiB/s): min= 136, max= 616, per=24.82%, avg=281.16, stdev=121.61, samples=44 iops : min= 34, max= 154, avg=70.20, stdev=30.42, samples=44 lat (msec) : 4=28.48%, 10=50.49%, 20=18.55%, 50=2.44%, 250=0.03% cpu : usr=0.31%, sys=10.25%, ctx=3586, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=27720: Sun Feb 2 04:48:45 2025 read: IOPS=68, BW=273KiB/s (280kB/s)(6052KiB/22153msec) clat (usec): min=1985, max=29438, avg=5516.33, stdev=4323.84 lat (usec): min=1986, max=29439, avg=5517.41, stdev=4323.77 clat percentiles (usec): | 1.00th=[ 2212], 5.00th=[ 2409], 10.00th=[ 2573], 20.00th=[ 2835], | 30.00th=[ 3064], 40.00th=[ 3359], 50.00th=[ 3720], 60.00th=[ 4293], | 70.00th=[ 5211], 80.00th=[ 7308], 90.00th=[11863], 95.00th=[15008], | 99.00th=[22676], 99.50th=[25560], 99.90th=[29230], 99.95th=[29492], | 99.99th=[29492] bw ( KiB/s): min= 112, max= 552, per=25.06%, avg=271.16, stdev=116.31, samples=44 iops : min= 28, max= 138, avg=67.70, stdev=29.09, samples=44 write: IOPS=70, BW=281KiB/s (288kB/s)(6236KiB/22153msec) clat (msec): min=3, max=190, avg= 8.79, stdev= 6.73 lat (msec): min=3, max=190, avg= 8.80, stdev= 6.73 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 5], 10.00th=[ 5], 20.00th=[ 6], | 30.00th=[ 6], 40.00th=[ 7], 50.00th=[ 7], 60.00th=[ 8], | 70.00th=[ 10], 80.00th=[ 12], 90.00th=[ 16], 95.00th=[ 19], | 99.00th=[ 26], 99.50th=[ 32], 99.90th=[ 48], 99.95th=[ 190], | 99.99th=[ 190] bw ( KiB/s): min= 152, max= 544, per=24.69%, avg=279.70, stdev=116.52, samples=44 iops : min= 38, max= 136, avg=69.84, stdev=29.14, samples=44 lat (msec) : 2=0.03%, 4=29.26%, 10=49.97%, 20=18.03%, 50=2.67% lat (msec) : 250=0.03% cpu : usr=0.23%, sys=10.32%, ctx=3594, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1083KiB/s (1109kB/s), 268KiB/s-273KiB/s (274kB/s-280kB/s), io=23.4MiB (24.6MB), run=22076-22181msec WRITE: bw=1133KiB/s (1161kB/s), 281KiB/s-286KiB/s (288kB/s-293kB/s), io=24.6MiB (25.7MB), run=22076-22181msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=27755: Sun Feb 2 04:48:50 2025 read: IOPS=90, BW=1456KiB/s (1491kB/s)(5920KiB/4066msec) clat (usec): min=61, max=20540, avg=5738.66, stdev=3670.67 lat (usec): min=62, max=20541, avg=5739.47, stdev=3670.72 clat percentiles (usec): | 1.00th=[ 65], 5.00th=[ 99], 10.00th=[ 114], 20.00th=[ 3556], | 30.00th=[ 4080], 40.00th=[ 4686], 50.00th=[ 5211], 60.00th=[ 6128], | 70.00th=[ 7177], 80.00th=[ 8291], 90.00th=[10159], 95.00th=[12125], | 99.00th=[17957], 99.50th=[19792], 99.90th=[20579], 99.95th=[20579], | 99.99th=[20579] bw ( KiB/s): min= 736, max= 1920, per=25.04%, avg=1423.50, stdev=371.14, samples=8 iops : min= 46, max= 120, avg=88.88, stdev=23.11, samples=8 write: IOPS=97, BW=1566KiB/s (1604kB/s)(6368KiB/4066msec) clat (usec): min=711, max=26880, avg=4697.27, stdev=4502.15 lat (usec): min=712, max=26882, avg=4698.50, stdev=4502.19 clat percentiles (usec): | 1.00th=[ 857], 5.00th=[ 979], 10.00th=[ 1057], 20.00th=[ 1172], | 30.00th=[ 1287], 40.00th=[ 1795], 50.00th=[ 3556], 60.00th=[ 4293], | 70.00th=[ 5604], 80.00th=[ 7832], 90.00th=[11207], 95.00th=[13829], | 99.00th=[21365], 99.50th=[25035], 99.90th=[26870], 99.95th=[26870], | 99.99th=[26870] bw ( KiB/s): min= 864, max= 1952, per=25.87%, avg=1547.50, stdev=393.36, samples=8 iops : min= 54, max= 122, avg=96.62, stdev=24.51, samples=8 lat (usec) : 100=2.73%, 250=3.91%, 750=0.13%, 1000=2.73% lat (msec) : 2=19.53%, 4=12.89%, 10=46.87%, 20=10.42%, 50=0.78% cpu : usr=0.30%, sys=23.81%, ctx=1442, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=27756: Sun Feb 2 04:48:50 2025 read: IOPS=86, BW=1385KiB/s (1418kB/s)(5744KiB/4147msec) clat (usec): min=48, max=22269, avg=5820.91, stdev=3720.98 lat (usec): min=48, max=22270, avg=5821.73, stdev=3720.98 clat percentiles (usec): | 1.00th=[ 70], 5.00th=[ 102], 10.00th=[ 125], 20.00th=[ 3523], | 30.00th=[ 4113], 40.00th=[ 4686], 50.00th=[ 5342], 60.00th=[ 6063], | 70.00th=[ 7177], 80.00th=[ 8455], 90.00th=[10421], 95.00th=[12649], | 99.00th=[17695], 99.50th=[21365], 99.90th=[22152], 99.95th=[22152], | 99.99th=[22152] bw ( KiB/s): min= 1024, max= 1856, per=23.99%, avg=1364.00, stdev=272.57, samples=8 iops : min= 64, max= 116, avg=85.25, stdev=17.04, samples=8 write: IOPS=98, BW=1578KiB/s (1616kB/s)(6544KiB/4147msec) clat (usec): min=713, max=24245, avg=4986.90, stdev=4158.46 lat (usec): min=714, max=24247, avg=4988.82, stdev=4158.17 clat percentiles (usec): | 1.00th=[ 783], 5.00th=[ 955], 10.00th=[ 1045], 20.00th=[ 1188], | 30.00th=[ 1565], 40.00th=[ 2966], 50.00th=[ 3916], 60.00th=[ 4817], | 70.00th=[ 6652], 80.00th=[ 8455], 90.00th=[11076], 95.00th=[12780], | 99.00th=[17695], 99.50th=[20055], 99.90th=[24249], 99.95th=[24249], | 99.99th=[24249] bw ( KiB/s): min= 1248, max= 1920, per=25.55%, avg=1528.00, stdev=235.00, samples=8 iops : min= 78, max= 120, avg=95.50, stdev=14.69, samples=8 lat (usec) : 50=0.13%, 100=2.08%, 250=3.78%, 750=0.26%, 1000=3.91% lat (msec) : 2=14.06%, 4=16.15%, 10=47.53%, 20=11.46%, 50=0.65% cpu : usr=0.22%, sys=23.25%, ctx=1563, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=27757: Sun Feb 2 04:48:50 2025 read: IOPS=92, BW=1485KiB/s (1521kB/s)(6256KiB/4213msec) clat (usec): min=62, max=23610, avg=5729.95, stdev=3383.47 lat (usec): min=62, max=23610, avg=5730.83, stdev=3383.46 clat percentiles (usec): | 1.00th=[ 77], 5.00th=[ 118], 10.00th=[ 424], 20.00th=[ 3458], | 30.00th=[ 4015], 40.00th=[ 4490], 50.00th=[ 5276], 60.00th=[ 6390], | 70.00th=[ 7111], 80.00th=[ 8225], 90.00th=[ 9634], 95.00th=[11338], | 99.00th=[17171], 99.50th=[21103], 99.90th=[23725], 99.95th=[23725], | 99.99th=[23725] bw ( KiB/s): min= 928, max= 1792, per=25.46%, avg=1447.62, stdev=290.12, samples=8 iops : min= 58, max= 112, avg=90.38, stdev=18.13, samples=8 write: IOPS=89, BW=1432KiB/s (1466kB/s)(6032KiB/4213msec) clat (usec): min=801, max=27863, avg=5012.68, stdev=4318.19 lat (usec): min=802, max=27865, avg=5015.11, stdev=4319.01 clat percentiles (usec): | 1.00th=[ 857], 5.00th=[ 996], 10.00th=[ 1090], 20.00th=[ 1205], | 30.00th=[ 1401], 40.00th=[ 2704], 50.00th=[ 3851], 60.00th=[ 4948], | 70.00th=[ 6652], 80.00th=[ 8291], 90.00th=[10421], 95.00th=[12649], | 99.00th=[19530], 99.50th=[24511], 99.90th=[27919], 99.95th=[27919], | 99.99th=[27919] bw ( KiB/s): min= 1216, max= 1628, per=22.93%, avg=1371.50, stdev=147.57, samples=8 iops : min= 76, max= 101, avg=85.62, stdev= 9.04, samples=8 lat (usec) : 100=1.43%, 250=3.26%, 500=0.52%, 750=0.13%, 1000=2.73% lat (msec) : 2=15.76%, 4=15.89%, 10=49.87%, 20=9.77%, 50=0.65% cpu : usr=0.38%, sys=22.98%, ctx=1644, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=27758: Sun Feb 2 04:48:50 2025 read: IOPS=92, BW=1477KiB/s (1512kB/s)(6032KiB/4084msec) clat (usec): min=64, max=20194, avg=5587.54, stdev=3465.56 lat (usec): min=65, max=20195, avg=5588.35, stdev=3465.57 clat percentiles (usec): | 1.00th=[ 76], 5.00th=[ 101], 10.00th=[ 123], 20.00th=[ 3490], | 30.00th=[ 3982], 40.00th=[ 4555], 50.00th=[ 5211], 60.00th=[ 5997], | 70.00th=[ 6718], 80.00th=[ 8029], 90.00th=[ 9503], 95.00th=[11600], | 99.00th=[17171], 99.50th=[18744], 99.90th=[20317], 99.95th=[20317], | 99.99th=[20317] bw ( KiB/s): min= 1056, max= 1692, per=25.32%, avg=1439.50, stdev=195.03, samples=8 iops : min= 66, max= 105, avg=89.88, stdev=12.05, samples=8 write: IOPS=95, BW=1532KiB/s (1569kB/s)(6256KiB/4084msec) clat (usec): min=713, max=25771, avg=4845.54, stdev=4316.57 lat (usec): min=714, max=25772, avg=4847.05, stdev=4316.68 clat percentiles (usec): | 1.00th=[ 922], 5.00th=[ 1020], 10.00th=[ 1074], 20.00th=[ 1172], | 30.00th=[ 1319], 40.00th=[ 1844], 50.00th=[ 3720], 60.00th=[ 4621], | 70.00th=[ 6390], 80.00th=[ 8455], 90.00th=[11076], 95.00th=[13435], | 99.00th=[19268], 99.50th=[22938], 99.90th=[25822], 99.95th=[25822], | 99.99th=[25822] bw ( KiB/s): min= 1184, max= 1920, per=25.20%, avg=1507.50, stdev=252.60, samples=8 iops : min= 74, max= 120, avg=94.12, stdev=15.75, samples=8 lat (usec) : 100=2.21%, 250=3.65%, 500=0.13%, 750=0.39%, 1000=1.56% lat (msec) : 2=19.27%, 4=14.45%, 10=47.66%, 20=10.16%, 50=0.52% cpu : usr=0.22%, sys=23.51%, ctx=1555, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=5685KiB/s (5822kB/s), 1385KiB/s-1485KiB/s (1418kB/s-1521kB/s), io=23.4MiB (24.5MB), run=4066-4213msec WRITE: bw=5981KiB/s (6125kB/s), 1432KiB/s-1578KiB/s (1466kB/s-1616kB/s), io=24.6MiB (25.8MB), run=4066-4213msec rand-rw: (groupid=0, jobs=1): err= 0: pid=27751: Sun Feb 2 04:48:52 2025 read: IOPS=65, BW=1042KiB/s (1067kB/s)(5920KiB/5681msec) clat (usec): min=2128, max=28614, avg=5585.24, stdev=4102.72 lat (usec): min=2128, max=28615, avg=5586.72, stdev=4102.41 clat percentiles (usec): | 1.00th=[ 2311], 5.00th=[ 2606], 10.00th=[ 2769], 20.00th=[ 2999], | 30.00th=[ 3195], 40.00th=[ 3523], 50.00th=[ 3916], 60.00th=[ 4490], | 70.00th=[ 5211], 80.00th=[ 7373], 90.00th=[11994], 95.00th=[15795], | 99.00th=[19792], 99.50th=[20055], 99.90th=[28705], 99.95th=[28705], | 99.99th=[28705] bw ( KiB/s): min= 640, max= 1920, per=24.45%, avg=1015.27, stdev=434.09, samples=11 iops : min= 40, max= 120, avg=63.45, stdev=27.13, samples=11 write: IOPS=70, BW=1121KiB/s (1148kB/s)(6368KiB/5681msec) clat (usec): min=3070, max=32621, avg=8910.18, stdev=5213.60 lat (usec): min=3072, max=32623, avg=8911.91, stdev=5213.63 clat percentiles (usec): | 1.00th=[ 3687], 5.00th=[ 4228], 10.00th=[ 4621], 20.00th=[ 5145], | 30.00th=[ 5604], 40.00th=[ 6063], 50.00th=[ 6783], 60.00th=[ 7832], | 70.00th=[ 9765], 80.00th=[12125], 90.00th=[16581], 95.00th=[19792], | 99.00th=[26870], 99.50th=[32375], 99.90th=[32637], 99.95th=[32637], | 99.99th=[32637] bw ( KiB/s): min= 704, max= 1984, per=25.37%, avg=1108.36, stdev=454.19, samples=11 iops : min= 44, max= 124, avg=69.27, stdev=28.39, samples=11 lat (msec) : 4=26.82%, 10=51.82%, 20=18.62%, 50=2.73% cpu : usr=0.23%, sys=11.02%, ctx=942, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=27752: Sun Feb 2 04:48:52 2025 read: IOPS=62, BW=996KiB/s (1020kB/s)(5744KiB/5767msec) clat (usec): min=2367, max=29905, avg=6206.13, stdev=5221.37 lat (usec): min=2368, max=29905, avg=6208.05, stdev=5220.78 clat percentiles (usec): | 1.00th=[ 2474], 5.00th=[ 2671], 10.00th=[ 2802], 20.00th=[ 2999], | 30.00th=[ 3228], 40.00th=[ 3523], 50.00th=[ 3916], 60.00th=[ 4424], | 70.00th=[ 5538], 80.00th=[ 8291], 90.00th=[14484], 95.00th=[18744], | 99.00th=[27132], 99.50th=[28967], 99.90th=[30016], 99.95th=[30016], | 99.99th=[30016] bw ( KiB/s): min= 576, max= 2048, per=23.45%, avg=974.00, stdev=464.97, samples=11 iops : min= 36, max= 128, avg=60.64, stdev=29.19, samples=11 write: IOPS=70, BW=1135KiB/s (1162kB/s)(6544KiB/5767msec) clat (usec): min=3439, max=26026, avg=8602.64, stdev=4760.04 lat (usec): min=3441, max=26028, avg=8604.90, stdev=4759.84 clat percentiles (usec): | 1.00th=[ 4015], 5.00th=[ 4359], 10.00th=[ 4621], 20.00th=[ 5014], | 30.00th=[ 5407], 40.00th=[ 5735], 50.00th=[ 6587], 60.00th=[ 7373], | 70.00th=[ 9634], 80.00th=[12518], 90.00th=[16057], 95.00th=[19006], | 99.00th=[23200], 99.50th=[24249], 99.90th=[26084], 99.95th=[26084], | 99.99th=[26084] bw ( KiB/s): min= 640, max= 2144, per=25.09%, avg=1096.09, stdev=483.98, samples=11 iops : min= 40, max= 134, avg=68.27, stdev=30.39, samples=11 lat (msec) : 4=25.26%, 10=51.82%, 20=19.53%, 50=3.39% cpu : usr=0.23%, sys=10.77%, ctx=942, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=27753: Sun Feb 2 04:48:52 2025 read: IOPS=68, BW=1101KiB/s (1127kB/s)(6256KiB/5682msec) clat (usec): min=2335, max=35209, avg=5631.39, stdev=4572.34 lat (usec): min=2336, max=35209, avg=5632.25, stdev=4572.34 clat percentiles (usec): | 1.00th=[ 2474], 5.00th=[ 2638], 10.00th=[ 2769], 20.00th=[ 2933], | 30.00th=[ 3195], 40.00th=[ 3425], 50.00th=[ 3785], 60.00th=[ 4293], | 70.00th=[ 4948], 80.00th=[ 6783], 90.00th=[11600], 95.00th=[16909], | 99.00th=[23725], 99.50th=[23987], 99.90th=[35390], 99.95th=[35390], | 99.99th=[35390] bw ( KiB/s): min= 606, max= 2336, per=26.46%, avg=1098.73, stdev=567.62, samples=11 iops : min= 37, max= 146, avg=68.36, stdev=35.57, samples=11 write: IOPS=66, BW=1062KiB/s (1087kB/s)(6032KiB/5682msec) clat (usec): min=3348, max=27526, avg=9078.60, stdev=5072.33 lat (usec): min=3349, max=27528, avg=9080.45, stdev=5072.80 clat percentiles (usec): | 1.00th=[ 3589], 5.00th=[ 4146], 10.00th=[ 4555], 20.00th=[ 5145], | 30.00th=[ 5538], 40.00th=[ 6194], 50.00th=[ 7046], 60.00th=[ 8717], | 70.00th=[10683], 80.00th=[12911], 90.00th=[16450], 95.00th=[20317], | 99.00th=[25297], 99.50th=[27132], 99.90th=[27657], 99.95th=[27657], | 99.99th=[27657] bw ( KiB/s): min= 736, max= 1952, per=23.55%, avg=1029.00, stdev=398.52, samples=11 iops : min= 46, max= 122, avg=64.00, stdev=25.00, samples=11 lat (msec) : 4=29.17%, 10=47.66%, 20=19.27%, 50=3.91% cpu : usr=0.12%, sys=11.05%, ctx=904, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=27754: Sun Feb 2 04:48:52 2025 read: IOPS=65, BW=1056KiB/s (1081kB/s)(6032KiB/5714msec) clat (usec): min=2158, max=34925, avg=6159.88, stdev=5208.23 lat (usec): min=2158, max=34926, avg=6160.82, stdev=5208.20 clat percentiles (usec): | 1.00th=[ 2311], 5.00th=[ 2573], 10.00th=[ 2769], 20.00th=[ 2999], | 30.00th=[ 3195], 40.00th=[ 3490], 50.00th=[ 3949], 60.00th=[ 4555], | 70.00th=[ 5669], 80.00th=[ 8225], 90.00th=[14091], 95.00th=[18220], | 99.00th=[24773], 99.50th=[30016], 99.90th=[34866], 99.95th=[34866], | 99.99th=[34866] bw ( KiB/s): min= 638, max= 2048, per=24.50%, avg=1017.36, stdev=438.29, samples=11 iops : min= 39, max= 128, avg=63.27, stdev=27.54, samples=11 write: IOPS=68, BW=1095KiB/s (1121kB/s)(6256KiB/5714msec) clat (usec): min=3368, max=35071, avg=8512.80, stdev=5268.52 lat (usec): min=3369, max=35073, avg=8514.63, stdev=5268.59 clat percentiles (usec): | 1.00th=[ 3621], 5.00th=[ 4178], 10.00th=[ 4555], 20.00th=[ 4948], | 30.00th=[ 5407], 40.00th=[ 5735], 50.00th=[ 6390], 60.00th=[ 7111], | 70.00th=[ 8979], 80.00th=[11469], 90.00th=[16057], 95.00th=[19268], | 99.00th=[30016], 99.50th=[32637], 99.90th=[34866], 99.95th=[34866], | 99.99th=[34866] bw ( KiB/s): min= 734, max= 2048, per=24.55%, avg=1072.55, stdev=489.14, samples=11 iops : min= 45, max= 128, avg=66.73, stdev=30.66, samples=11 lat (msec) : 4=26.95%, 10=52.21%, 20=17.06%, 50=3.78% cpu : usr=0.26%, sys=11.12%, ctx=936, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=4153KiB/s (4253kB/s), 996KiB/s-1101KiB/s (1020kB/s-1127kB/s), io=23.4MiB (24.5MB), run=5681-5767msec WRITE: bw=4370KiB/s (4475kB/s), 1062KiB/s-1135KiB/s (1087kB/s-1162kB/s), io=24.6MiB (25.8MB), run=5681-5767msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=27768: Sun Feb 2 04:48:55 2025 read: IOPS=5, BW=5666KiB/s (5802kB/s)(7168KiB/1265msec) clat (msec): min=31, max=618, avg=140.91, stdev=214.30 lat (msec): min=31, max=618, avg=140.91, stdev=214.30 clat percentiles (msec): | 1.00th=[ 32], 5.00th=[ 32], 10.00th=[ 32], 20.00th=[ 34], | 30.00th=[ 35], 40.00th=[ 35], 50.00th=[ 39], 60.00th=[ 99], | 70.00th=[ 99], 80.00th=[ 131], 90.00th=[ 617], 95.00th=[ 617], | 99.00th=[ 617], 99.50th=[ 617], 99.90th=[ 617], 99.95th=[ 617], | 99.99th=[ 617] bw ( KiB/s): min= 4087, max= 6119, per=38.50%, avg=5103.00, stdev=1436.84, samples=2 iops : min= 3, max= 5, avg= 4.00, stdev= 1.41, samples=2 write: IOPS=3, BW=4047KiB/s (4145kB/s)(5120KiB/1265msec) clat (usec): min=36535, max=98994, avg=53559.37, stdev=25858.92 lat (usec): min=36585, max=99048, avg=53643.08, stdev=25845.45 clat percentiles (usec): | 1.00th=[36439], 5.00th=[36439], 10.00th=[36439], 20.00th=[36439], | 30.00th=[40633], 40.00th=[40633], 50.00th=[41681], 60.00th=[41681], | 70.00th=[50070], 80.00th=[50070], 90.00th=[99091], 95.00th=[99091], | 99.00th=[99091], 99.50th=[99091], 99.90th=[99091], 99.95th=[99091], | 99.99th=[99091] bw ( KiB/s): min= 8159, max= 8159, per=43.97%, avg=8159.00, stdev= 0.00, samples=1 iops : min= 7, max= 7, avg= 7.00, stdev= 0.00, samples=1 lat (msec) : 50=66.67%, 100=16.67%, 250=8.33%, 750=8.33% cpu : usr=0.00%, sys=27.45%, ctx=1253, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=27769: Sun Feb 2 04:48:55 2025 read: IOPS=1, BW=1517KiB/s (1553kB/s)(2048KiB/1350msec) clat (usec): min=37042, max=65081, avg=51062.06, stdev=19826.68 lat (usec): min=37044, max=65083, avg=51063.99, stdev=19826.82 clat percentiles (usec): | 1.00th=[36963], 5.00th=[36963], 10.00th=[36963], 20.00th=[36963], | 30.00th=[36963], 40.00th=[36963], 50.00th=[36963], 60.00th=[65274], | 70.00th=[65274], 80.00th=[65274], 90.00th=[65274], 95.00th=[65274], | 99.00th=[65274], 99.50th=[65274], 99.90th=[65274], 99.95th=[65274], | 99.99th=[65274] bw ( KiB/s): min= 2048, max= 2048, per=15.45%, avg=2048.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=7, BW=7585KiB/s (7767kB/s)(10.0MiB/1350msec) clat (msec): min=37, max=629, avg=116.59, stdev=182.10 lat (msec): min=37, max=629, avg=116.68, stdev=182.09 clat percentiles (msec): | 1.00th=[ 38], 5.00th=[ 38], 10.00th=[ 38], 20.00th=[ 41], | 30.00th=[ 42], 40.00th=[ 51], 50.00th=[ 51], 60.00th=[ 53], | 70.00th=[ 72], 80.00th=[ 72], 90.00th=[ 128], 95.00th=[ 634], | 99.00th=[ 634], 99.50th=[ 634], 99.90th=[ 634], 99.95th=[ 634], | 99.99th=[ 634] bw ( KiB/s): min=10219, max=10219, per=55.07%, avg=10219.00, stdev= 0.00, samples=1 iops : min= 9, max= 9, avg= 9.00, stdev= 0.00, samples=1 lat (msec) : 50=33.33%, 100=50.00%, 250=8.33%, 750=8.33% cpu : usr=0.07%, sys=21.13%, ctx=253, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=27770: Sun Feb 2 04:48:55 2025 read: IOPS=5, BW=5678KiB/s (5815kB/s)(6144KiB/1082msec) clat (usec): min=559, max=46296, avg=16518.75, stdev=17606.71 lat (usec): min=561, max=46298, avg=16519.93, stdev=17606.84 clat percentiles (usec): | 1.00th=[ 562], 5.00th=[ 562], 10.00th=[ 562], 20.00th=[ 5342], | 30.00th=[ 5342], 40.00th=[ 6325], 50.00th=[ 6325], 60.00th=[11600], | 70.00th=[28967], 80.00th=[28967], 90.00th=[46400], 95.00th=[46400], | 99.00th=[46400], 99.50th=[46400], 99.90th=[46400], 99.95th=[46400], | 99.99th=[46400] bw ( KiB/s): min=10118, max=10118, per=76.33%, avg=10118.00, stdev= 0.00, samples=1 iops : min= 9, max= 9, avg= 9.00, stdev= 0.00, samples=1 write: IOPS=5, BW=5678KiB/s (5815kB/s)(6144KiB/1082msec) clat (msec): min=28, max=706, avg=159.34, stdev=268.45 lat (msec): min=28, max=706, avg=159.42, stdev=268.44 clat percentiles (msec): | 1.00th=[ 29], 5.00th=[ 29], 10.00th=[ 29], 20.00th=[ 42], | 30.00th=[ 42], 40.00th=[ 50], 50.00th=[ 50], 60.00th=[ 64], | 70.00th=[ 67], 80.00th=[ 67], 90.00th=[ 709], 95.00th=[ 709], | 99.00th=[ 709], 99.50th=[ 709], 99.90th=[ 709], 99.95th=[ 709], | 99.99th=[ 709] bw ( KiB/s): min= 4096, max= 6071, per=27.39%, avg=5083.50, stdev=1396.54, samples=2 iops : min= 4, max= 5, avg= 4.50, stdev= 0.71, samples=2 lat (usec) : 750=8.33% lat (msec) : 10=16.67%, 20=8.33%, 50=41.67%, 100=16.67%, 750=8.33% cpu : usr=0.00%, sys=19.52%, ctx=187, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=27772: Sun Feb 2 04:48:55 2025 read: IOPS=3, BW=3314KiB/s (3393kB/s)(5120KiB/1545msec) clat (usec): min=22181, max=41378, avg=33361.78, stdev=8123.99 lat (usec): min=22181, max=41380, avg=33363.14, stdev=8124.35 clat percentiles (usec): | 1.00th=[22152], 5.00th=[22152], 10.00th=[22152], 20.00th=[22152], | 30.00th=[28443], 40.00th=[28443], 50.00th=[34341], 60.00th=[34341], | 70.00th=[40109], 80.00th=[40109], 90.00th=[41157], 95.00th=[41157], | 99.00th=[41157], 99.50th=[41157], 99.90th=[41157], 99.95th=[41157], | 99.99th=[41157] bw ( KiB/s): min= 8192, max= 8192, per=61.80%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=4, BW=4639KiB/s (4751kB/s)(7168KiB/1545msec) clat (msec): min=28, max=733, avg=184.91, stdev=244.58 lat (msec): min=28, max=733, avg=184.99, stdev=244.56 clat percentiles (msec): | 1.00th=[ 29], 5.00th=[ 29], 10.00th=[ 29], 20.00th=[ 59], | 30.00th=[ 107], 40.00th=[ 107], 50.00th=[ 112], 60.00th=[ 125], | 70.00th=[ 125], 80.00th=[ 132], 90.00th=[ 735], 95.00th=[ 735], | 99.00th=[ 735], 99.50th=[ 735], 99.90th=[ 735], 99.95th=[ 735], | 99.99th=[ 735] bw ( KiB/s): min= 2048, max= 6144, per=22.07%, avg=4096.00, stdev=2048.00, samples=3 iops : min= 2, max= 6, avg= 4.00, stdev= 2.00, samples=3 lat (msec) : 50=50.00%, 100=8.33%, 250=33.33%, 750=8.33% cpu : usr=0.00%, sys=18.85%, ctx=1243, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=12.9MiB/s (13.6MB/s), 1517KiB/s-5678KiB/s (1553kB/s-5815kB/s), io=20.0MiB (20.0MB), run=1082-1545msec WRITE: bw=18.1MiB/s (19.0MB/s), 4047KiB/s-7585KiB/s (4145kB/s-7767kB/s), io=28.0MiB (29.4MB), run=1082-1545msec rand-rw: (groupid=0, jobs=1): err= 0: pid=27771: Sun Feb 2 04:48:55 2025 read: IOPS=4, BW=4710KiB/s (4823kB/s)(7168KiB/1522msec) clat (msec): min=16, max=604, avg=162.58, stdev=225.77 lat (msec): min=16, max=604, avg=162.58, stdev=225.77 clat percentiles (msec): | 1.00th=[ 17], 5.00th=[ 17], 10.00th=[ 17], 20.00th=[ 21], | 30.00th=[ 39], 40.00th=[ 39], 50.00th=[ 54], 60.00th=[ 65], | 70.00th=[ 65], 80.00th=[ 342], 90.00th=[ 609], 95.00th=[ 609], | 99.00th=[ 609], 99.50th=[ 609], 99.90th=[ 609], 99.95th=[ 609], | 99.99th=[ 609] bw ( KiB/s): min= 2048, max= 2048, per=15.84%, avg=2048.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=3, BW=3364KiB/s (3445kB/s)(5120KiB/1522msec) clat (msec): min=22, max=152, avg=57.32, stdev=54.21 lat (msec): min=22, max=152, avg=57.41, stdev=54.21 clat percentiles (msec): | 1.00th=[ 23], 5.00th=[ 23], 10.00th=[ 23], 20.00th=[ 23], | 30.00th=[ 25], 40.00th=[ 25], 50.00th=[ 40], 60.00th=[ 40], | 70.00th=[ 50], 80.00th=[ 50], 90.00th=[ 153], 95.00th=[ 153], | 99.00th=[ 153], 99.50th=[ 153], 99.90th=[ 153], 99.95th=[ 153], | 99.99th=[ 153] lat (msec) : 20=8.33%, 50=50.00%, 100=16.67%, 250=8.33%, 500=8.33% lat (msec) : 750=8.33% cpu : usr=0.13%, sys=6.77%, ctx=48, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=27773: Sun Feb 2 04:48:55 2025 read: IOPS=1, BW=1454KiB/s (1488kB/s)(2048KiB/1409msec) clat (msec): min=85, max=250, avg=167.77, stdev=117.01 lat (msec): min=85, max=250, avg=167.77, stdev=117.01 clat percentiles (msec): | 1.00th=[ 86], 5.00th=[ 86], 10.00th=[ 86], 20.00th=[ 86], | 30.00th=[ 86], 40.00th=[ 86], 50.00th=[ 86], 60.00th=[ 251], | 70.00th=[ 251], 80.00th=[ 251], 90.00th=[ 251], 95.00th=[ 251], | 99.00th=[ 251], 99.50th=[ 251], 99.90th=[ 251], 99.95th=[ 251], | 99.99th=[ 251] bw ( KiB/s): min= 2043, max= 2043, per=15.80%, avg=2043.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=7, BW=7268KiB/s (7442kB/s)(10.0MiB/1409msec) clat (msec): min=22, max=613, avg=104.16, stdev=180.00 lat (msec): min=22, max=613, avg=104.26, stdev=180.00 clat percentiles (msec): | 1.00th=[ 23], 5.00th=[ 23], 10.00th=[ 23], 20.00th=[ 24], | 30.00th=[ 28], 40.00th=[ 42], 50.00th=[ 51], 60.00th=[ 53], | 70.00th=[ 68], 80.00th=[ 68], 90.00th=[ 82], 95.00th=[ 617], | 99.00th=[ 617], 99.50th=[ 617], 99.90th=[ 617], 99.95th=[ 617], | 99.99th=[ 617] bw ( KiB/s): min=10240, max=10240, per=56.57%, avg=10240.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 lat (msec) : 50=41.67%, 100=41.67%, 500=8.33%, 750=8.33% cpu : usr=0.07%, sys=8.66%, ctx=166, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=27774: Sun Feb 2 04:48:55 2025 read: IOPS=3, BW=3879KiB/s (3972kB/s)(6144KiB/1584msec) clat (msec): min=27, max=137, avg=70.87, stdev=40.62 lat (msec): min=27, max=137, avg=70.87, stdev=40.62 clat percentiles (msec): | 1.00th=[ 28], 5.00th=[ 28], 10.00th=[ 28], 20.00th=[ 42], | 30.00th=[ 42], 40.00th=[ 48], 50.00th=[ 48], 60.00th=[ 78], | 70.00th=[ 94], 80.00th=[ 94], 90.00th=[ 138], 95.00th=[ 138], | 99.00th=[ 138], 99.50th=[ 138], 99.90th=[ 138], 99.95th=[ 138], | 99.99th=[ 138] bw ( KiB/s): min= 2048, max= 7433, per=36.67%, avg=4740.50, stdev=3807.77, samples=2 iops : min= 2, max= 7, avg= 4.50, stdev= 3.54, samples=2 write: IOPS=3, BW=3879KiB/s (3972kB/s)(6144KiB/1584msec) clat (msec): min=22, max=749, avg=178.48, stdev=281.83 lat (msec): min=22, max=749, avg=178.57, stdev=281.82 clat percentiles (msec): | 1.00th=[ 23], 5.00th=[ 23], 10.00th=[ 23], 20.00th=[ 43], | 30.00th=[ 43], 40.00th=[ 47], 50.00th=[ 47], 60.00th=[ 103], | 70.00th=[ 108], 80.00th=[ 108], 90.00th=[ 751], 95.00th=[ 751], | 99.00th=[ 751], 99.50th=[ 751], 99.90th=[ 751], 99.95th=[ 751], | 99.99th=[ 751] bw ( KiB/s): min= 2048, max= 5575, per=21.58%, avg=3906.33, stdev=1771.13, samples=3 iops : min= 2, max= 5, avg= 3.67, stdev= 1.53, samples=3 lat (msec) : 50=50.00%, 100=16.67%, 250=25.00%, 750=8.33% cpu : usr=0.25%, sys=7.96%, ctx=119, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=27775: Sun Feb 2 04:48:55 2025 read: IOPS=3, BW=3514KiB/s (3598kB/s)(5120KiB/1457msec) clat (msec): min=35, max=164, avg=90.33, stdev=49.82 lat (msec): min=35, max=164, avg=90.33, stdev=49.81 clat percentiles (msec): | 1.00th=[ 36], 5.00th=[ 36], 10.00th=[ 36], 20.00th=[ 36], | 30.00th=[ 59], 40.00th=[ 59], 50.00th=[ 84], 60.00th=[ 84], | 70.00th=[ 111], 80.00th=[ 111], 90.00th=[ 165], 95.00th=[ 165], | 99.00th=[ 165], 99.50th=[ 165], 99.90th=[ 165], 99.95th=[ 165], | 99.99th=[ 165] bw ( KiB/s): min= 2048, max= 2048, per=15.84%, avg=2048.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=4, BW=4920KiB/s (5038kB/s)(7168KiB/1457msec) clat (msec): min=32, max=616, avg=133.74, stdev=213.49 lat (msec): min=33, max=616, avg=133.82, stdev=213.50 clat percentiles (msec): | 1.00th=[ 33], 5.00th=[ 33], 10.00th=[ 33], 20.00th=[ 40], | 30.00th=[ 47], 40.00th=[ 47], 50.00th=[ 53], 60.00th=[ 58], | 70.00th=[ 58], 80.00th=[ 91], 90.00th=[ 617], 95.00th=[ 617], | 99.00th=[ 617], 99.50th=[ 617], 99.90th=[ 617], 99.95th=[ 617], | 99.99th=[ 617] bw ( KiB/s): min= 2048, max= 6144, per=22.63%, avg=4096.00, stdev=2896.31, samples=2 iops : min= 2, max= 6, avg= 4.00, stdev= 2.83, samples=2 lat (msec) : 50=33.33%, 100=41.67%, 250=16.67%, 750=8.33% cpu : usr=0.07%, sys=7.42%, ctx=52, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=12.6MiB/s (13.2MB/s), 1454KiB/s-4710KiB/s (1488kB/s-4823kB/s), io=20.0MiB (20.0MB), run=1409-1584msec WRITE: bw=17.7MiB/s (18.5MB/s), 3364KiB/s-7268KiB/s (3445kB/s-7442kB/s), io=28.0MiB (29.4MB), run=1409-1584msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=27782: Sun Feb 2 04:48:56 2025 read: IOPS=6, BW=26.1MiB/s (27.4MB/s)(12.0MiB/460msec) clat (msec): min=90, max=169, avg=136.84, stdev=41.51 lat (msec): min=90, max=169, avg=136.84, stdev=41.51 clat percentiles (msec): | 1.00th=[ 90], 5.00th=[ 90], 10.00th=[ 90], 20.00th=[ 90], | 30.00th=[ 90], 40.00th=[ 153], 50.00th=[ 153], 60.00th=[ 153], | 70.00th=[ 169], 80.00th=[ 169], 90.00th=[ 169], 95.00th=[ 169], | 99.00th=[ 169], 99.50th=[ 169], 99.90th=[ 169], 99.95th=[ 169], | 99.99th=[ 169] lat (msec) : 100=33.33%, 250=66.67% cpu : usr=0.00%, sys=37.04%, ctx=2064, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=27784: Sun Feb 2 04:48:56 2025 read: IOPS=1, BW=4842KiB/s (4958kB/s)(4096KiB/846msec) clat (nsec): min=80703k, max=80703k, avg=80703267.00, stdev= 0.00 lat (nsec): min=80706k, max=80706k, avg=80705869.00, stdev= 0.00 clat percentiles (usec): | 1.00th=[80217], 5.00th=[80217], 10.00th=[80217], 20.00th=[80217], | 30.00th=[80217], 40.00th=[80217], 50.00th=[80217], 60.00th=[80217], | 70.00th=[80217], 80.00th=[80217], 90.00th=[80217], 95.00th=[80217], | 99.00th=[80217], 99.50th=[80217], 99.90th=[80217], 99.95th=[80217], | 99.99th=[80217] bw ( KiB/s): min= 8192, max= 8192, per=42.30%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=2, BW=9683KiB/s (9916kB/s)(8192KiB/846msec) clat (msec): min=326, max=401, avg=363.79, stdev=52.72 lat (msec): min=326, max=401, avg=364.10, stdev=52.72 clat percentiles (msec): | 1.00th=[ 326], 5.00th=[ 326], 10.00th=[ 326], 20.00th=[ 326], | 30.00th=[ 326], 40.00th=[ 326], 50.00th=[ 326], 60.00th=[ 401], | 70.00th=[ 401], 80.00th=[ 401], 90.00th=[ 401], 95.00th=[ 401], | 99.00th=[ 401], 99.50th=[ 401], 99.90th=[ 401], 99.95th=[ 401], | 99.99th=[ 401] lat (msec) : 100=33.33%, 500=66.67% cpu : usr=0.12%, sys=30.06%, ctx=538, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=27786: Sun Feb 2 04:48:56 2025 write: IOPS=3, BW=14.8MiB/s (15.5MB/s)(12.0MiB/810msec) clat (msec): min=188, max=411, avg=264.71, stdev=127.28 lat (msec): min=189, max=411, avg=265.05, stdev=127.18 clat percentiles (msec): | 1.00th=[ 190], 5.00th=[ 190], 10.00th=[ 190], 20.00th=[ 190], | 30.00th=[ 190], 40.00th=[ 194], 50.00th=[ 194], 60.00th=[ 194], | 70.00th=[ 414], 80.00th=[ 414], 90.00th=[ 414], 95.00th=[ 414], | 99.00th=[ 414], 99.50th=[ 414], 99.90th=[ 414], 99.95th=[ 414], | 99.99th=[ 414] bw ( KiB/s): min=16384, max=16384, per=42.30%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.00%, sys=36.34%, ctx=140, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=27788: Sun Feb 2 04:48:56 2025 write: IOPS=4, BW=19.2MiB/s (20.2MB/s)(12.0MiB/624msec) clat (msec): min=131, max=331, avg=200.39, stdev=113.48 lat (msec): min=131, max=331, avg=200.92, stdev=113.36 clat percentiles (msec): | 1.00th=[ 131], 5.00th=[ 131], 10.00th=[ 131], 20.00th=[ 131], | 30.00th=[ 131], 40.00th=[ 140], 50.00th=[ 140], 60.00th=[ 140], | 70.00th=[ 330], 80.00th=[ 330], 90.00th=[ 330], 95.00th=[ 330], | 99.00th=[ 330], 99.50th=[ 330], 99.90th=[ 330], 99.95th=[ 330], | 99.99th=[ 330] bw ( KiB/s): min=16384, max=16384, per=42.30%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.00%, sys=54.90%, ctx=39, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=18.9MiB/s (19.8MB/s), 4842KiB/s-26.1MiB/s (4958kB/s-27.4MB/s), io=16.0MiB (16.8MB), run=460-846msec WRITE: bw=37.8MiB/s (39.7MB/s), 9683KiB/s-19.2MiB/s (9916kB/s-20.2MB/s), io=32.0MiB (33.6MB), run=624-846msec rand-rw: (groupid=0, jobs=1): err= 0: pid=27783: Sun Feb 2 04:48:57 2025 read: IOPS=3, BW=12.4MiB/s (13.0MB/s)(12.0MiB/967msec) clat (msec): min=245, max=337, avg=304.97, stdev=51.50 lat (msec): min=245, max=337, avg=304.97, stdev=51.50 clat percentiles (msec): | 1.00th=[ 247], 5.00th=[ 247], 10.00th=[ 247], 20.00th=[ 247], | 30.00th=[ 247], 40.00th=[ 334], 50.00th=[ 334], 60.00th=[ 334], | 70.00th=[ 338], 80.00th=[ 338], 90.00th=[ 338], 95.00th=[ 338], | 99.00th=[ 338], 99.50th=[ 338], 99.90th=[ 338], 99.95th=[ 338], | 99.99th=[ 338] bw ( KiB/s): min= 8192, max= 8192, per=49.95%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=66.67% cpu : usr=0.10%, sys=11.39%, ctx=55, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=27785: Sun Feb 2 04:48:57 2025 read: IOPS=1, BW=4100KiB/s (4199kB/s)(4096KiB/999msec) clat (nsec): min=331549k, max=331549k, avg=331548696.00, stdev= 0.00 lat (nsec): min=331552k, max=331552k, avg=331551657.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 334], 5.00th=[ 334], 10.00th=[ 334], 20.00th=[ 334], | 30.00th=[ 334], 40.00th=[ 334], 50.00th=[ 334], 60.00th=[ 334], | 70.00th=[ 334], 80.00th=[ 334], 90.00th=[ 334], 95.00th=[ 334], | 99.00th=[ 334], 99.50th=[ 334], 99.90th=[ 334], 99.95th=[ 334], | 99.99th=[ 334] bw ( KiB/s): min= 8192, max= 8192, per=49.95%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=2, BW=8200KiB/s (8397kB/s)(8192KiB/999msec) clat (msec): min=77, max=496, avg=287.31, stdev=296.45 lat (msec): min=77, max=497, avg=287.59, stdev=296.45 clat percentiles (msec): | 1.00th=[ 79], 5.00th=[ 79], 10.00th=[ 79], 20.00th=[ 79], | 30.00th=[ 79], 40.00th=[ 79], 50.00th=[ 79], 60.00th=[ 498], | 70.00th=[ 498], 80.00th=[ 498], 90.00th=[ 498], 95.00th=[ 498], | 99.00th=[ 498], 99.50th=[ 498], 99.90th=[ 498], 99.95th=[ 498], | 99.99th=[ 498] lat (msec) : 100=33.33%, 500=66.67% cpu : usr=0.20%, sys=13.23%, ctx=64, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=27787: Sun Feb 2 04:48:57 2025 write: IOPS=2, BW=11.7MiB/s (12.3MB/s)(12.0MiB/1026msec) clat (msec): min=265, max=424, avg=330.50, stdev=83.42 lat (msec): min=266, max=425, avg=330.85, stdev=83.45 clat percentiles (msec): | 1.00th=[ 266], 5.00th=[ 266], 10.00th=[ 266], 20.00th=[ 266], | 30.00th=[ 266], 40.00th=[ 300], 50.00th=[ 300], 60.00th=[ 300], | 70.00th=[ 426], 80.00th=[ 426], 90.00th=[ 426], 95.00th=[ 426], | 99.00th=[ 426], 99.50th=[ 426], 99.90th=[ 426], 99.95th=[ 426], | 99.99th=[ 426] bw ( KiB/s): min= 8192, max= 8192, per=27.18%, avg=8192.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 lat (msec) : 500=100.00% cpu : usr=0.20%, sys=8.39%, ctx=99, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=27789: Sun Feb 2 04:48:57 2025 write: IOPS=2, BW=11.0MiB/s (11.6MB/s)(12.0MiB/1087msec) clat (msec): min=60, max=555, avg=341.29, stdev=253.92 lat (msec): min=61, max=556, avg=341.65, stdev=254.03 clat percentiles (msec): | 1.00th=[ 61], 5.00th=[ 61], 10.00th=[ 61], 20.00th=[ 61], | 30.00th=[ 61], 40.00th=[ 409], 50.00th=[ 409], 60.00th=[ 409], | 70.00th=[ 558], 80.00th=[ 558], 90.00th=[ 558], 95.00th=[ 558], | 99.00th=[ 558], 99.50th=[ 558], 99.90th=[ 558], 99.95th=[ 558], | 99.99th=[ 558] bw ( KiB/s): min= 8192, max= 8192, per=27.18%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 500=33.33%, 750=33.33% cpu : usr=0.55%, sys=17.50%, ctx=236, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=16.0MiB/s (16.8MB/s), 4100KiB/s-12.4MiB/s (4199kB/s-13.0MB/s), io=16.0MiB (16.8MB), run=967-999msec WRITE: bw=29.4MiB/s (30.9MB/s), 8200KiB/s-11.7MiB/s (8397kB/s-12.3MB/s), io=32.0MiB (33.6MB), run=999-1087msec