== sanity test 398b: DIO and buffer IO race ============== 21:50:51 (1746150651) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 2.42272 s, 20.8 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=24078: Thu May 1 21:51:32 2025 read: IOPS=42, BW=169KiB/s (173kB/s)(5992KiB/35437msec) clat (usec): min=62, max=285759, avg=13133.33, stdev=10748.08 lat (usec): min=62, max=285760, avg=13134.37, stdev=10748.07 clat percentiles (usec): | 1.00th=[ 96], 5.00th=[ 128], 10.00th=[ 4817], 20.00th=[ 6718], | 30.00th=[ 8455], 40.00th=[ 9896], 50.00th=[ 11600], 60.00th=[ 13829], | 70.00th=[ 15926], 80.00th=[ 18744], 90.00th=[ 22414], 95.00th=[ 27132], | 99.00th=[ 40109], 99.50th=[ 46400], 99.90th=[ 64226], 99.95th=[287310], | 99.99th=[287310] bw ( KiB/s): min= 48, max= 248, per=24.71%, avg=167.27, stdev=39.00, samples=70 iops : min= 12, max= 62, avg=41.53, stdev= 9.76, samples=70 write: IOPS=44, BW=178KiB/s (182kB/s)(6296KiB/35437msec) clat (usec): min=397, max=73093, avg=9924.57, stdev=10516.03 lat (usec): min=398, max=73094, avg=9929.32, stdev=10515.36 clat percentiles (usec): | 1.00th=[ 685], 5.00th=[ 742], 10.00th=[ 791], 20.00th=[ 1020], | 30.00th=[ 1713], 40.00th=[ 3326], 50.00th=[ 7111], 60.00th=[ 9765], | 70.00th=[12780], 80.00th=[17433], 90.00th=[24511], 95.00th=[31065], | 99.00th=[44827], 99.50th=[50070], 99.90th=[72877], 99.95th=[72877], | 99.99th=[72877] bw ( KiB/s): min= 40, max= 280, per=24.71%, avg=175.19, stdev=45.00, samples=70 iops : min= 10, max= 70, avg=43.51, stdev=11.21, samples=70 lat (usec) : 100=0.75%, 250=2.34%, 500=0.20%, 750=3.06%, 1000=7.16% lat (msec) : 2=6.80%, 4=4.72%, 10=26.50%, 20=32.85%, 50=15.20% lat (msec) : 100=0.39%, 500=0.03% cpu : usr=0.11%, sys=12.65%, ctx=5199, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24079: Thu May 1 21:51:32 2025 read: IOPS=42, BW=168KiB/s (172kB/s)(5940KiB/35270msec) clat (usec): min=70, max=78272, avg=12876.18, stdev=8305.87 lat (usec): min=71, max=78273, avg=12880.02, stdev=8306.87 clat percentiles (usec): | 1.00th=[ 91], 5.00th=[ 125], 10.00th=[ 5145], 20.00th=[ 6718], | 30.00th=[ 8094], 40.00th=[ 9896], 50.00th=[11469], 60.00th=[13304], | 70.00th=[15664], 80.00th=[18220], 90.00th=[22676], 95.00th=[27132], | 99.00th=[41157], 99.50th=[51119], 99.90th=[69731], 99.95th=[78119], | 99.99th=[78119] bw ( KiB/s): min= 40, max= 272, per=24.71%, avg=167.31, stdev=39.85, samples=70 iops : min= 10, max= 68, avg=41.59, stdev= 9.97, samples=70 write: IOPS=44, BW=180KiB/s (184kB/s)(6348KiB/35270msec) clat (usec): min=424, max=272426, avg=10093.27, stdev=12406.34 lat (usec): min=425, max=272427, avg=10094.54, stdev=12406.30 clat percentiles (usec): | 1.00th=[ 635], 5.00th=[ 742], 10.00th=[ 791], 20.00th=[ 979], | 30.00th=[ 1647], 40.00th=[ 3490], 50.00th=[ 7111], 60.00th=[ 9765], | 70.00th=[ 12911], 80.00th=[ 17695], 90.00th=[ 24249], 95.00th=[ 31327], | 99.00th=[ 43254], 99.50th=[ 52167], 99.90th=[ 82314], 99.95th=[270533], | 99.99th=[270533] bw ( KiB/s): min= 96, max= 384, per=25.12%, avg=178.10, stdev=52.51, samples=70 iops : min= 24, max= 96, avg=44.27, stdev=13.13, samples=70 lat (usec) : 100=1.11%, 250=2.02%, 500=0.13%, 750=3.06%, 1000=7.75% lat (msec) : 2=6.58%, 4=4.56%, 10=26.30%, 20=32.91%, 50=14.97% lat (msec) : 100=0.59%, 500=0.03% cpu : usr=0.19%, sys=12.49%, ctx=5121, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24080: Thu May 1 21:51:32 2025 read: IOPS=42, BW=170KiB/s (174kB/s)(6028KiB/35406msec) clat (usec): min=70, max=282574, avg=13130.61, stdev=10927.15 lat (usec): min=70, max=282575, avg=13138.23, stdev=10932.31 clat percentiles (usec): | 1.00th=[ 97], 5.00th=[ 135], 10.00th=[ 4817], 20.00th=[ 6718], | 30.00th=[ 8029], 40.00th=[ 9372], 50.00th=[ 11207], 60.00th=[ 13304], | 70.00th=[ 15795], 80.00th=[ 18744], 90.00th=[ 23462], 95.00th=[ 28705], | 99.00th=[ 39584], 99.50th=[ 45876], 99.90th=[ 72877], 99.95th=[283116], | 99.99th=[283116] bw ( KiB/s): min= 40, max= 288, per=24.93%, avg=168.74, stdev=44.64, samples=70 iops : min= 10, max= 72, avg=41.91, stdev=11.15, samples=70 write: IOPS=44, BW=177KiB/s (181kB/s)(6260KiB/35406msec) clat (usec): min=425, max=86789, avg=9891.28, stdev=10759.36 lat (usec): min=426, max=86791, avg=9892.49, stdev=10759.41 clat percentiles (usec): | 1.00th=[ 652], 5.00th=[ 742], 10.00th=[ 783], 20.00th=[ 938], | 30.00th=[ 1500], 40.00th=[ 2900], 50.00th=[ 6849], 60.00th=[ 9634], | 70.00th=[12780], 80.00th=[17695], 90.00th=[24773], 95.00th=[30278], | 99.00th=[44303], 99.50th=[54264], 99.90th=[76022], 99.95th=[86508], | 99.99th=[86508] bw ( KiB/s): min= 48, max= 280, per=24.63%, avg=174.66, stdev=47.97, samples=70 iops : min= 12, max= 70, avg=43.39, stdev=12.04, samples=70 lat (usec) : 100=0.75%, 250=2.18%, 500=0.33%, 750=3.03%, 1000=8.50% lat (msec) : 2=6.15%, 4=4.69%, 10=26.95%, 20=30.53%, 50=16.41% lat (msec) : 100=0.46%, 500=0.03% cpu : usr=0.18%, sys=12.65%, ctx=5159, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24081: Thu May 1 21:51:32 2025 read: IOPS=43, BW=175KiB/s (179kB/s)(6052KiB/34656msec) clat (usec): min=59, max=90458, avg=13036.55, stdev=8252.34 lat (usec): min=60, max=90459, avg=13037.92, stdev=8252.22 clat percentiles (usec): | 1.00th=[ 94], 5.00th=[ 742], 10.00th=[ 5604], 20.00th=[ 7177], | 30.00th=[ 8455], 40.00th=[ 9765], 50.00th=[11338], 60.00th=[13173], | 70.00th=[15401], 80.00th=[18482], 90.00th=[22676], 95.00th=[26870], | 99.00th=[40109], 99.50th=[51643], 99.90th=[79168], 99.95th=[90702], | 99.99th=[90702] bw ( KiB/s): min= 31, max= 271, per=25.72%, avg=174.10, stdev=45.04, samples=69 iops : min= 7, max= 67, avg=43.25, stdev=11.26, samples=69 write: IOPS=44, BW=180KiB/s (184kB/s)(6236KiB/34656msec) clat (usec): min=438, max=291458, avg=9474.62, stdev=12506.56 lat (usec): min=439, max=291461, avg=9476.12, stdev=12506.69 clat percentiles (usec): | 1.00th=[ 685], 5.00th=[ 750], 10.00th=[ 783], 20.00th=[ 914], | 30.00th=[ 1434], 40.00th=[ 2474], 50.00th=[ 6128], 60.00th=[ 8848], | 70.00th=[ 11994], 80.00th=[ 16581], 90.00th=[ 23200], 95.00th=[ 30540], | 99.00th=[ 43779], 99.50th=[ 51119], 99.90th=[ 72877], 99.95th=[291505], | 99.99th=[291505] bw ( KiB/s): min= 31, max= 319, per=25.27%, avg=179.19, stdev=49.54, samples=69 iops : min= 7, max= 79, avg=44.52, stdev=12.38, samples=69 lat (usec) : 100=0.88%, 250=1.50%, 500=0.07%, 750=2.54%, 1000=9.24% lat (msec) : 2=6.77%, 4=4.49%, 10=27.47%, 20=31.61%, 50=14.88% lat (msec) : 100=0.52%, 500=0.03% cpu : usr=0.16%, sys=12.77%, ctx=5129, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=678KiB/s (694kB/s), 168KiB/s-175KiB/s (172kB/s-179kB/s), io=23.4MiB (24.6MB), run=34656-35437msec WRITE: bw=709KiB/s (726kB/s), 177KiB/s-180KiB/s (181kB/s-184kB/s), io=24.6MiB (25.7MB), run=34656-35437msec rand-rw: (groupid=0, jobs=1): err= 0: pid=24082: Thu May 1 21:51:50 2025 read: IOPS=28, BW=112KiB/s (115kB/s)(5992KiB/53350msec) clat (msec): min=3, max=280, avg=13.26, stdev=12.91 lat (msec): min=3, max=280, avg=13.26, stdev=12.91 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 5], 10.00th=[ 6], 20.00th=[ 7], | 30.00th=[ 8], 40.00th=[ 9], 50.00th=[ 10], 60.00th=[ 11], | 70.00th=[ 13], 80.00th=[ 17], 90.00th=[ 29], 95.00th=[ 37], | 99.00th=[ 57], 99.50th=[ 63], 99.90th=[ 88], 99.95th=[ 279], | 99.99th=[ 279] bw ( KiB/s): min= 32, max= 247, per=25.05%, avg=111.71, stdev=49.59, samples=106 iops : min= 8, max= 61, avg=27.72, stdev=12.44, samples=106 write: IOPS=29, BW=118KiB/s (121kB/s)(6296KiB/53350msec) clat (msec): min=6, max=138, avg=21.19, stdev=12.30 lat (msec): min=6, max=138, avg=21.20, stdev=12.30 clat percentiles (msec): | 1.00th=[ 8], 5.00th=[ 10], 10.00th=[ 12], 20.00th=[ 13], | 30.00th=[ 14], 40.00th=[ 16], 50.00th=[ 18], 60.00th=[ 21], | 70.00th=[ 23], 80.00th=[ 28], 90.00th=[ 37], 95.00th=[ 45], | 99.00th=[ 69], 99.50th=[ 78], 99.90th=[ 107], 99.95th=[ 140], | 99.99th=[ 140] bw ( KiB/s): min= 15, max= 232, per=25.15%, avg=117.20, stdev=48.75, samples=106 iops : min= 3, max= 58, avg=29.09, stdev=12.23, samples=106 lat (msec) : 4=0.55%, 10=30.60%, 20=40.53%, 50=25.88%, 100=2.34% lat (msec) : 250=0.07%, 500=0.03% cpu : usr=0.10%, sys=5.70%, ctx=3446, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24083: Thu May 1 21:51:50 2025 read: IOPS=27, BW=110KiB/s (113kB/s)(5940KiB/53833msec) clat (msec): min=2, max=331, avg=13.30, stdev=13.72 lat (msec): min=2, max=331, avg=13.30, stdev=13.72 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 5], 10.00th=[ 6], 20.00th=[ 7], | 30.00th=[ 8], 40.00th=[ 9], 50.00th=[ 10], 60.00th=[ 11], | 70.00th=[ 13], 80.00th=[ 18], 90.00th=[ 28], 95.00th=[ 37], | 99.00th=[ 55], 99.50th=[ 64], 99.90th=[ 103], 99.95th=[ 334], | 99.99th=[ 334] bw ( KiB/s): min= 8, max= 256, per=24.59%, avg=109.65, stdev=52.51, samples=107 iops : min= 2, max= 64, avg=27.19, stdev=13.16, samples=107 write: IOPS=29, BW=118KiB/s (121kB/s)(6348KiB/53833msec) clat (msec): min=5, max=139, avg=21.39, stdev=12.47 lat (msec): min=5, max=139, avg=21.40, stdev=12.48 clat percentiles (msec): | 1.00th=[ 8], 5.00th=[ 10], 10.00th=[ 11], 20.00th=[ 13], | 30.00th=[ 14], 40.00th=[ 16], 50.00th=[ 18], 60.00th=[ 20], | 70.00th=[ 24], 80.00th=[ 29], 90.00th=[ 39], 95.00th=[ 46], | 99.00th=[ 69], 99.50th=[ 75], 99.90th=[ 93], 99.95th=[ 140], | 99.99th=[ 140] bw ( KiB/s): min= 40, max= 248, per=25.03%, avg=116.62, stdev=48.80, samples=107 iops : min= 10, max= 62, avg=28.93, stdev=12.20, samples=107 lat (msec) : 4=0.98%, 10=29.85%, 20=40.92%, 50=25.68%, 100=2.47% lat (msec) : 250=0.07%, 500=0.03% cpu : usr=0.08%, sys=5.68%, ctx=3432, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24084: Thu May 1 21:51:50 2025 read: IOPS=28, BW=113KiB/s (115kB/s)(6028KiB/53512msec) clat (usec): min=2640, max=99146, avg=13309.33, stdev=10789.68 lat (usec): min=2641, max=99147, avg=13310.59, stdev=10789.60 clat percentiles (usec): | 1.00th=[ 3818], 5.00th=[ 4817], 10.00th=[ 5473], 20.00th=[ 6456], | 30.00th=[ 7308], 40.00th=[ 8356], 50.00th=[ 9634], 60.00th=[10945], | 70.00th=[13304], 80.00th=[17695], 90.00th=[27395], 95.00th=[35914], | 99.00th=[54264], 99.50th=[66847], 99.90th=[82314], 99.95th=[99091], | 99.99th=[99091] bw ( KiB/s): min= 8, max= 248, per=25.20%, avg=112.39, stdev=49.94, samples=107 iops : min= 2, max= 62, avg=27.90, stdev=12.43, samples=107 write: IOPS=29, BW=117KiB/s (120kB/s)(6260KiB/53512msec) clat (msec): min=5, max=330, avg=21.29, stdev=14.08 lat (msec): min=5, max=330, avg=21.29, stdev=14.08 clat percentiles (msec): | 1.00th=[ 8], 5.00th=[ 10], 10.00th=[ 11], 20.00th=[ 13], | 30.00th=[ 15], 40.00th=[ 16], 50.00th=[ 18], 60.00th=[ 20], | 70.00th=[ 23], 80.00th=[ 29], 90.00th=[ 38], 95.00th=[ 44], | 99.00th=[ 69], 99.50th=[ 74], 99.90th=[ 92], 99.95th=[ 330], | 99.99th=[ 330] bw ( KiB/s): min= 24, max= 223, per=25.02%, avg=116.59, stdev=49.50, samples=107 iops : min= 6, max= 55, avg=28.94, stdev=12.35, samples=107 lat (msec) : 4=0.65%, 10=28.48%, 20=43.49%, 50=24.80%, 100=2.54% lat (msec) : 500=0.03% cpu : usr=0.14%, sys=5.61%, ctx=3432, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24085: Thu May 1 21:51:50 2025 read: IOPS=28, BW=113KiB/s (116kB/s)(6052KiB/53535msec) clat (msec): min=2, max=102, avg=13.30, stdev=10.92 lat (msec): min=2, max=102, avg=13.30, stdev=10.92 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 5], 10.00th=[ 6], 20.00th=[ 7], | 30.00th=[ 8], 40.00th=[ 9], 50.00th=[ 10], 60.00th=[ 11], | 70.00th=[ 14], 80.00th=[ 18], 90.00th=[ 29], 95.00th=[ 37], | 99.00th=[ 52], 99.50th=[ 60], 99.90th=[ 85], 99.95th=[ 104], | 99.99th=[ 104] bw ( KiB/s): min= 16, max= 240, per=25.49%, avg=113.70, stdev=50.24, samples=106 iops : min= 4, max= 60, avg=28.14, stdev=12.58, samples=106 write: IOPS=29, BW=116KiB/s (119kB/s)(6236KiB/53535msec) clat (msec): min=5, max=320, avg=21.32, stdev=13.87 lat (msec): min=5, max=320, avg=21.32, stdev=13.87 clat percentiles (msec): | 1.00th=[ 8], 5.00th=[ 10], 10.00th=[ 11], 20.00th=[ 13], | 30.00th=[ 15], 40.00th=[ 16], 50.00th=[ 18], 60.00th=[ 21], | 70.00th=[ 24], 80.00th=[ 29], 90.00th=[ 37], 95.00th=[ 44], | 99.00th=[ 64], 99.50th=[ 70], 99.90th=[ 106], 99.95th=[ 321], | 99.99th=[ 321] bw ( KiB/s): min= 23, max= 256, per=24.87%, avg=115.92, stdev=48.71, samples=107 iops : min= 5, max= 64, avg=28.69, stdev=12.24, samples=107 lat (msec) : 4=0.78%, 10=30.50%, 20=39.71%, 50=26.92%, 100=1.99% lat (msec) : 250=0.07%, 500=0.03% cpu : usr=0.18%, sys=5.57%, ctx=3450, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=446KiB/s (457kB/s), 110KiB/s-113KiB/s (113kB/s-116kB/s), io=23.4MiB (24.6MB), run=53350-53833msec WRITE: bw=467KiB/s (478kB/s), 116KiB/s-118KiB/s (119kB/s-121kB/s), io=24.6MiB (25.7MB), run=53350-53833msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=24149: Thu May 1 21:52:01 2025 read: IOPS=41, BW=666KiB/s (682kB/s)(5920KiB/8892msec) clat (usec): min=84, max=59810, avg=12777.35, stdev=9301.36 lat (usec): min=84, max=59811, avg=12778.43, stdev=9301.40 clat percentiles (usec): | 1.00th=[ 89], 5.00th=[ 109], 10.00th=[ 125], 20.00th=[ 5604], | 30.00th=[ 7832], 40.00th=[ 9896], 50.00th=[11863], 60.00th=[13829], | 70.00th=[16909], 80.00th=[19268], 90.00th=[23987], 95.00th=[26870], | 99.00th=[44827], 99.50th=[57410], 99.90th=[60031], 99.95th=[60031], | 99.99th=[60031] bw ( KiB/s): min= 382, max= 960, per=25.75%, avg=656.12, stdev=191.97, samples=17 iops : min= 23, max= 60, avg=40.65, stdev=12.10, samples=17 write: IOPS=44, BW=716KiB/s (733kB/s)(6368KiB/8892msec) clat (usec): min=822, max=56344, avg=9932.00, stdev=10516.08 lat (usec): min=823, max=56345, avg=9934.14, stdev=10515.76 clat percentiles (usec): | 1.00th=[ 922], 5.00th=[ 1057], 10.00th=[ 1188], 20.00th=[ 1401], | 30.00th=[ 2245], 40.00th=[ 3130], 50.00th=[ 4883], 60.00th=[ 9503], | 70.00th=[12387], 80.00th=[17433], 90.00th=[24511], 95.00th=[32375], | 99.00th=[44827], 99.50th=[51643], 99.90th=[56361], 99.95th=[56361], | 99.99th=[56361] bw ( KiB/s): min= 286, max= 1056, per=26.16%, avg=701.35, stdev=239.17, samples=17 iops : min= 17, max= 66, avg=43.47, stdev=15.15, samples=17 lat (usec) : 100=1.30%, 250=5.73%, 500=0.39%, 1000=1.30% lat (msec) : 2=13.15%, 4=8.98%, 10=20.31%, 20=31.90%, 50=16.41% lat (msec) : 100=0.52% cpu : usr=0.09%, sys=12.95%, ctx=1243, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24150: Thu May 1 21:52:01 2025 read: IOPS=39, BW=631KiB/s (646kB/s)(5744KiB/9105msec) clat (usec): min=60, max=72501, avg=12839.34, stdev=10190.73 lat (usec): min=61, max=72502, avg=12841.76, stdev=10195.27 clat percentiles (usec): | 1.00th=[ 81], 5.00th=[ 101], 10.00th=[ 115], 20.00th=[ 5211], | 30.00th=[ 7832], 40.00th=[ 9765], 50.00th=[11338], 60.00th=[13698], | 70.00th=[15926], 80.00th=[18744], 90.00th=[25560], 95.00th=[30802], | 99.00th=[46400], 99.50th=[55313], 99.90th=[72877], 99.95th=[72877], | 99.99th=[72877] bw ( KiB/s): min= 319, max= 1053, per=24.53%, avg=625.11, stdev=211.28, samples=18 iops : min= 19, max= 65, avg=38.72, stdev=13.30, samples=18 write: IOPS=44, BW=719KiB/s (736kB/s)(6544KiB/9105msec) clat (usec): min=830, max=64768, avg=10731.03, stdev=11632.97 lat (usec): min=831, max=64770, avg=10735.78, stdev=11631.38 clat percentiles (usec): | 1.00th=[ 898], 5.00th=[ 1037], 10.00th=[ 1139], 20.00th=[ 1401], | 30.00th=[ 2343], 40.00th=[ 3785], 50.00th=[ 6915], 60.00th=[ 9372], | 70.00th=[12911], 80.00th=[18220], 90.00th=[26084], 95.00th=[37487], | 99.00th=[47449], 99.50th=[59507], 99.90th=[64750], 99.95th=[64750], | 99.99th=[64750] bw ( KiB/s): min= 448, max= 1280, per=26.30%, avg=705.00, stdev=202.70, samples=18 iops : min= 28, max= 80, avg=43.72, stdev=12.75, samples=18 lat (usec) : 100=1.82%, 250=6.12%, 500=0.26%, 750=0.13%, 1000=1.95% lat (msec) : 2=11.85%, 4=8.33%, 10=21.74%, 20=29.95%, 50=17.06% lat (msec) : 100=0.78% cpu : usr=0.05%, sys=13.38%, ctx=1275, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24151: Thu May 1 21:52:01 2025 read: IOPS=41, BW=666KiB/s (682kB/s)(6256KiB/9397msec) clat (usec): min=65, max=79044, avg=13293.09, stdev=9690.91 lat (usec): min=66, max=79045, avg=13294.20, stdev=9690.99 clat percentiles (usec): | 1.00th=[ 87], 5.00th=[ 105], 10.00th=[ 123], 20.00th=[ 6325], | 30.00th=[ 8717], 40.00th=[10552], 50.00th=[12387], 60.00th=[14091], | 70.00th=[16909], 80.00th=[19792], 90.00th=[24773], 95.00th=[28443], | 99.00th=[43254], 99.50th=[57934], 99.90th=[79168], 99.95th=[79168], | 99.99th=[79168] bw ( KiB/s): min= 384, max= 1120, per=26.07%, avg=664.28, stdev=179.07, samples=18 iops : min= 24, max= 70, avg=41.22, stdev=11.32, samples=18 write: IOPS=40, BW=642KiB/s (657kB/s)(6032KiB/9397msec) clat (usec): min=917, max=54755, avg=11024.83, stdev=10643.67 lat (usec): min=918, max=54755, avg=11026.56, stdev=10643.69 clat percentiles (usec): | 1.00th=[ 1004], 5.00th=[ 1106], 10.00th=[ 1205], 20.00th=[ 1614], | 30.00th=[ 2671], 40.00th=[ 4555], 50.00th=[ 7898], 60.00th=[11600], | 70.00th=[14353], 80.00th=[19530], 90.00th=[26346], 95.00th=[34341], | 99.00th=[45351], 99.50th=[51119], 99.90th=[54789], 99.95th=[54789], | 99.99th=[54789] bw ( KiB/s): min= 320, max= 928, per=23.18%, avg=621.50, stdev=179.56, samples=18 iops : min= 20, max= 58, avg=38.56, stdev=11.14, samples=18 lat (usec) : 100=1.56%, 250=5.60%, 500=0.52%, 750=0.13%, 1000=0.39% lat (msec) : 2=11.72%, 4=7.03%, 10=19.66%, 20=34.51%, 50=18.10% lat (msec) : 100=0.78% cpu : usr=0.12%, sys=12.97%, ctx=1330, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24152: Thu May 1 21:52:01 2025 read: IOPS=41, BW=656KiB/s (672kB/s)(6032KiB/9192msec) clat (usec): min=78, max=58958, avg=13085.18, stdev=9865.35 lat (usec): min=78, max=58959, avg=13086.19, stdev=9865.40 clat percentiles (usec): | 1.00th=[ 89], 5.00th=[ 104], 10.00th=[ 118], 20.00th=[ 5473], | 30.00th=[ 8160], 40.00th=[ 9634], 50.00th=[11469], 60.00th=[13698], | 70.00th=[16712], 80.00th=[20579], 90.00th=[25560], 95.00th=[31851], | 99.00th=[41681], 99.50th=[53740], 99.90th=[58983], 99.95th=[58983], | 99.99th=[58983] bw ( KiB/s): min= 351, max= 894, per=25.16%, avg=641.06, stdev=154.75, samples=18 iops : min= 21, max= 55, avg=39.72, stdev= 9.62, samples=18 write: IOPS=42, BW=681KiB/s (697kB/s)(6256KiB/9192msec) clat (usec): min=770, max=76666, avg=10354.12, stdev=11913.27 lat (usec): min=771, max=76667, avg=10356.87, stdev=11912.63 clat percentiles (usec): | 1.00th=[ 930], 5.00th=[ 1074], 10.00th=[ 1172], 20.00th=[ 1385], | 30.00th=[ 1975], 40.00th=[ 3326], 50.00th=[ 5342], 60.00th=[ 8455], | 70.00th=[12125], 80.00th=[18744], 90.00th=[27395], 95.00th=[32375], | 99.00th=[56361], 99.50th=[57410], 99.90th=[77071], 99.95th=[77071], | 99.99th=[77071] bw ( KiB/s): min= 288, max= 1213, per=24.97%, avg=669.44, stdev=221.93, samples=18 iops : min= 18, max= 75, avg=41.50, stdev=13.79, samples=18 lat (usec) : 100=1.69%, 250=6.12%, 500=0.26%, 1000=0.91% lat (msec) : 2=14.84%, 4=7.42%, 10=21.87%, 20=26.56%, 50=19.27% lat (msec) : 100=1.04% cpu : usr=0.12%, sys=12.96%, ctx=1304, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2549KiB/s (2610kB/s), 631KiB/s-666KiB/s (646kB/s-682kB/s), io=23.4MiB (24.5MB), run=8892-9397msec WRITE: bw=2682KiB/s (2746kB/s), 642KiB/s-719KiB/s (657kB/s-736kB/s), io=24.6MiB (25.8MB), run=8892-9397msec rand-rw: (groupid=0, jobs=1): err= 0: pid=24145: Thu May 1 21:52:05 2025 read: IOPS=27, BW=444KiB/s (454kB/s)(5920KiB/13345msec) clat (usec): min=3032, max=97511, avg=13284.91, stdev=12398.28 lat (usec): min=3033, max=97512, avg=13287.54, stdev=12397.46 clat percentiles (usec): | 1.00th=[ 3097], 5.00th=[ 4047], 10.00th=[ 4686], 20.00th=[ 5538], | 30.00th=[ 6259], 40.00th=[ 7439], 50.00th=[ 8717], 60.00th=[10290], | 70.00th=[12387], 80.00th=[17695], 90.00th=[29754], 95.00th=[41157], | 99.00th=[59507], 99.50th=[67634], 99.90th=[96994], 99.95th=[96994], | 99.99th=[96994] bw ( KiB/s): min= 160, max= 1024, per=24.23%, avg=434.62, stdev=240.57, samples=26 iops : min= 10, max= 64, avg=26.85, stdev=15.09, samples=26 write: IOPS=29, BW=477KiB/s (489kB/s)(6368KiB/13345msec) clat (usec): min=5883, max=81549, avg=20765.87, stdev=14380.73 lat (usec): min=5885, max=81551, avg=20767.94, stdev=14380.66 clat percentiles (usec): | 1.00th=[ 6325], 5.00th=[ 8029], 10.00th=[ 8979], 20.00th=[10683], | 30.00th=[11994], 40.00th=[13304], 50.00th=[15401], 60.00th=[18220], | 70.00th=[22414], 80.00th=[29492], 90.00th=[40633], 95.00th=[54264], | 99.00th=[78119], 99.50th=[80217], 99.90th=[81265], 99.95th=[81265], | 99.99th=[81265] bw ( KiB/s): min= 128, max= 1088, per=25.11%, avg=474.08, stdev=273.38, samples=26 iops : min= 8, max= 68, avg=29.31, stdev=17.16, samples=26 lat (msec) : 4=2.21%, 10=33.72%, 20=37.89%, 50=21.74%, 100=4.43% cpu : usr=0.12%, sys=5.82%, ctx=877, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24146: Thu May 1 21:52:05 2025 read: IOPS=27, BW=437KiB/s (448kB/s)(5744KiB/13137msec) clat (usec): min=3004, max=62456, avg=12353.71, stdev=11087.98 lat (usec): min=3005, max=62457, avg=12354.95, stdev=11087.93 clat percentiles (usec): | 1.00th=[ 3195], 5.00th=[ 3916], 10.00th=[ 4752], 20.00th=[ 5407], | 30.00th=[ 6456], 40.00th=[ 7177], 50.00th=[ 8029], 60.00th=[ 9372], | 70.00th=[11338], 80.00th=[15926], 90.00th=[27395], 95.00th=[38011], | 99.00th=[55313], 99.50th=[59507], 99.90th=[62653], 99.95th=[62653], | 99.99th=[62653] bw ( KiB/s): min= 64, max= 1120, per=24.18%, avg=433.81, stdev=257.74, samples=26 iops : min= 4, max= 70, avg=26.73, stdev=16.21, samples=26 write: IOPS=31, BW=498KiB/s (510kB/s)(6544KiB/13137msec) clat (usec): min=5104, max=84538, avg=20945.56, stdev=13910.72 lat (usec): min=5106, max=84541, avg=20947.94, stdev=13910.63 clat percentiles (usec): | 1.00th=[ 5932], 5.00th=[ 7570], 10.00th=[ 8979], 20.00th=[10421], | 30.00th=[11863], 40.00th=[14222], 50.00th=[16712], 60.00th=[19268], | 70.00th=[22676], 80.00th=[29230], 90.00th=[42730], 95.00th=[47449], | 99.00th=[72877], 99.50th=[77071], 99.90th=[84411], 99.95th=[84411], | 99.99th=[84411] bw ( KiB/s): min= 191, max= 1056, per=26.17%, avg=494.12, stdev=254.16, samples=26 iops : min= 11, max= 66, avg=30.50, stdev=15.99, samples=26 lat (msec) : 4=2.34%, 10=36.59%, 20=33.72%, 50=24.09%, 100=3.26% cpu : usr=0.18%, sys=5.83%, ctx=863, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24147: Thu May 1 21:52:05 2025 read: IOPS=29, BW=472KiB/s (483kB/s)(6256KiB/13268msec) clat (usec): min=3117, max=99911, avg=13376.44, stdev=12445.67 lat (usec): min=3118, max=99912, avg=13378.43, stdev=12445.58 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 5], 10.00th=[ 5], 20.00th=[ 6], | 30.00th=[ 7], 40.00th=[ 8], 50.00th=[ 9], 60.00th=[ 10], | 70.00th=[ 14], 80.00th=[ 20], 90.00th=[ 29], 95.00th=[ 41], | 99.00th=[ 62], 99.50th=[ 79], 99.90th=[ 101], 99.95th=[ 101], | 99.99th=[ 101] bw ( KiB/s): min= 127, max= 1248, per=26.17%, avg=469.58, stdev=309.21, samples=26 iops : min= 7, max= 78, avg=29.04, stdev=19.40, samples=26 write: IOPS=28, BW=455KiB/s (466kB/s)(6032KiB/13268msec) clat (usec): min=5263, max=69214, avg=20882.82, stdev=13476.86 lat (usec): min=5265, max=69217, avg=20886.34, stdev=13480.12 clat percentiles (usec): | 1.00th=[ 5997], 5.00th=[ 8356], 10.00th=[ 9372], 20.00th=[10814], | 30.00th=[12256], 40.00th=[13829], 50.00th=[15664], 60.00th=[18482], | 70.00th=[22676], 80.00th=[30802], 90.00th=[40109], 95.00th=[52691], | 99.00th=[63177], 99.50th=[67634], 99.90th=[69731], 99.95th=[69731], | 99.99th=[69731] bw ( KiB/s): min= 192, max= 992, per=23.70%, avg=447.46, stdev=240.25, samples=26 iops : min= 12, max= 62, avg=27.65, stdev=15.08, samples=26 lat (msec) : 4=2.08%, 10=35.81%, 20=33.98%, 50=24.09%, 100=4.04% cpu : usr=0.00%, sys=5.86%, ctx=870, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24148: Thu May 1 21:52:05 2025 read: IOPS=28, BW=454KiB/s (465kB/s)(6032KiB/13278msec) clat (usec): min=3006, max=66103, avg=13177.79, stdev=11510.79 lat (usec): min=3007, max=66103, avg=13183.92, stdev=11507.93 clat percentiles (usec): | 1.00th=[ 3425], 5.00th=[ 4080], 10.00th=[ 4621], 20.00th=[ 5669], | 30.00th=[ 6456], 40.00th=[ 7373], 50.00th=[ 8848], 60.00th=[10290], | 70.00th=[12518], 80.00th=[19006], 90.00th=[29492], 95.00th=[37487], | 99.00th=[56361], 99.50th=[62129], 99.90th=[66323], 99.95th=[66323], | 99.99th=[66323] bw ( KiB/s): min= 160, max= 1021, per=24.79%, avg=444.81, stdev=229.22, samples=26 iops : min= 10, max= 63, avg=27.42, stdev=14.39, samples=26 write: IOPS=29, BW=471KiB/s (482kB/s)(6256KiB/13278msec) clat (usec): min=5499, max=86116, avg=21180.96, stdev=13839.76 lat (usec): min=5500, max=86118, avg=21182.93, stdev=13839.70 clat percentiles (usec): | 1.00th=[ 6259], 5.00th=[ 8094], 10.00th=[ 8979], 20.00th=[10683], | 30.00th=[11994], 40.00th=[14091], 50.00th=[16188], 60.00th=[19792], | 70.00th=[23462], 80.00th=[30802], 90.00th=[40633], 95.00th=[50070], | 99.00th=[69731], 99.50th=[77071], 99.90th=[86508], 99.95th=[86508], | 99.99th=[86508] bw ( KiB/s): min= 192, max= 1115, per=24.80%, avg=468.23, stdev=265.86, samples=26 iops : min= 12, max= 69, avg=28.88, stdev=16.70, samples=26 lat (msec) : 4=1.95%, 10=34.77%, 20=34.64%, 50=25.00%, 100=3.65% cpu : usr=0.18%, sys=5.66%, ctx=884, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1795KiB/s (1838kB/s), 437KiB/s-472KiB/s (448kB/s-483kB/s), io=23.4MiB (24.5MB), run=13137-13345msec WRITE: bw=1888KiB/s (1934kB/s), 455KiB/s-498KiB/s (466kB/s-510kB/s), io=24.6MiB (25.8MB), run=13137-13345msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=24168: Thu May 1 21:52:08 2025 read: IOPS=6, BW=6255KiB/s (6405kB/s)(7168KiB/1146msec) clat (usec): min=540, max=99077, avg=70630.08, stdev=32608.25 lat (usec): min=541, max=99078, avg=70631.78, stdev=32608.32 clat percentiles (usec): | 1.00th=[ 537], 5.00th=[ 537], 10.00th=[ 537], 20.00th=[65274], | 30.00th=[76022], 40.00th=[76022], 50.00th=[81265], 60.00th=[84411], | 70.00th=[84411], 80.00th=[87557], 90.00th=[99091], 95.00th=[99091], | 99.00th=[99091], 99.50th=[99091], 99.90th=[99091], 99.95th=[99091], | 99.99th=[99091] bw ( KiB/s): min= 6095, max= 6131, per=50.71%, avg=6113.00, stdev=25.46, samples=2 iops : min= 5, max= 5, avg= 5.00, stdev= 0.00, samples=2 write: IOPS=4, BW=4468KiB/s (4575kB/s)(5120KiB/1146msec) clat (msec): min=56, max=195, avg=115.92, stdev=58.04 lat (msec): min=56, max=195, avg=115.98, stdev=58.04 clat percentiles (msec): | 1.00th=[ 57], 5.00th=[ 57], 10.00th=[ 57], 20.00th=[ 57], | 30.00th=[ 64], 40.00th=[ 64], 50.00th=[ 120], 60.00th=[ 120], | 70.00th=[ 146], 80.00th=[ 146], 90.00th=[ 197], 95.00th=[ 197], | 99.00th=[ 197], 99.50th=[ 197], 99.90th=[ 197], 99.95th=[ 197], | 99.99th=[ 197] bw ( KiB/s): min= 2031, max= 6131, per=24.18%, avg=4081.00, stdev=2899.14, samples=2 iops : min= 1, max= 5, avg= 3.00, stdev= 2.83, samples=2 lat (usec) : 750=8.33% lat (msec) : 100=66.67%, 250=25.00% cpu : usr=0.00%, sys=21.48%, ctx=271, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24169: Thu May 1 21:52:08 2025 read: IOPS=1, BW=1205KiB/s (1234kB/s)(2048KiB/1699msec) clat (usec): min=58079, max=70750, avg=64414.89, stdev=8959.89 lat (usec): min=58081, max=70753, avg=64417.18, stdev=8960.47 clat percentiles (usec): | 1.00th=[57934], 5.00th=[57934], 10.00th=[57934], 20.00th=[57934], | 30.00th=[57934], 40.00th=[57934], 50.00th=[57934], 60.00th=[70779], | 70.00th=[70779], 80.00th=[70779], 90.00th=[70779], 95.00th=[70779], | 99.00th=[70779], 99.50th=[70779], 99.90th=[70779], 99.95th=[70779], | 99.99th=[70779] bw ( KiB/s): min= 1984, max= 2027, per=16.64%, avg=2005.50, stdev=30.41, samples=2 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=2 write: IOPS=5, BW=6027KiB/s (6172kB/s)(10.0MiB/1699msec) clat (msec): min=26, max=373, avg=150.47, stdev=93.95 lat (msec): min=26, max=373, avg=150.55, stdev=93.95 clat percentiles (msec): | 1.00th=[ 27], 5.00th=[ 27], 10.00th=[ 27], 20.00th=[ 88], | 30.00th=[ 103], 40.00th=[ 116], 50.00th=[ 118], 60.00th=[ 140], | 70.00th=[ 176], 80.00th=[ 176], 90.00th=[ 222], 95.00th=[ 376], | 99.00th=[ 376], 99.50th=[ 376], 99.90th=[ 376], 99.95th=[ 376], | 99.99th=[ 376] bw ( KiB/s): min= 2027, max= 9922, per=31.68%, avg=5345.33, stdev=4095.16, samples=3 iops : min= 1, max= 9, avg= 4.33, stdev= 4.16, samples=3 lat (msec) : 50=8.33%, 100=25.00%, 250=58.33%, 500=8.33% cpu : usr=0.00%, sys=17.43%, ctx=231, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24170: Thu May 1 21:52:08 2025 read: IOPS=4, BW=4880KiB/s (4997kB/s)(6144KiB/1259msec) clat (usec): min=575, max=273882, avg=97907.73, stdev=93171.83 lat (usec): min=576, max=273884, avg=97909.11, stdev=93171.86 clat percentiles (usec): | 1.00th=[ 578], 5.00th=[ 578], 10.00th=[ 578], 20.00th=[ 58459], | 30.00th=[ 58459], 40.00th=[ 63177], 50.00th=[ 63177], 60.00th=[ 86508], | 70.00th=[105382], 80.00th=[105382], 90.00th=[274727], 95.00th=[274727], | 99.00th=[274727], 99.50th=[274727], 99.90th=[274727], 99.95th=[274727], | 99.99th=[274727] bw ( KiB/s): min= 2048, max= 4079, per=25.41%, avg=3063.50, stdev=1436.13, samples=2 iops : min= 2, max= 3, avg= 2.50, stdev= 0.71, samples=2 write: IOPS=4, BW=4880KiB/s (4997kB/s)(6144KiB/1259msec) clat (msec): min=48, max=250, avg=107.91, stdev=73.48 lat (msec): min=48, max=250, avg=107.97, stdev=73.46 clat percentiles (msec): | 1.00th=[ 49], 5.00th=[ 49], 10.00th=[ 49], 20.00th=[ 65], | 30.00th=[ 65], 40.00th=[ 77], 50.00th=[ 77], 60.00th=[ 91], | 70.00th=[ 118], 80.00th=[ 118], 90.00th=[ 251], 95.00th=[ 251], | 99.00th=[ 251], 99.50th=[ 251], 99.90th=[ 251], 99.95th=[ 251], | 99.99th=[ 251] bw ( KiB/s): min= 2039, max= 6144, per=24.25%, avg=4091.50, stdev=2902.67, samples=2 iops : min= 1, max= 6, avg= 3.50, stdev= 3.54, samples=2 lat (usec) : 750=8.33% lat (msec) : 50=8.33%, 100=50.00%, 250=16.67%, 500=16.67% cpu : usr=0.00%, sys=15.90%, ctx=419, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24171: Thu May 1 21:52:08 2025 read: IOPS=3, BW=4035KiB/s (4132kB/s)(5120KiB/1269msec) clat (usec): min=502, max=113659, avg=58472.35, stdev=54004.51 lat (usec): min=502, max=113661, avg=58473.54, stdev=54004.82 clat percentiles (usec): | 1.00th=[ 502], 5.00th=[ 502], 10.00th=[ 502], 20.00th=[ 502], | 30.00th=[ 506], 40.00th=[ 506], 50.00th=[ 83362], 60.00th=[ 83362], | 70.00th=[ 93848], 80.00th=[ 93848], 90.00th=[113771], 95.00th=[113771], | 99.00th=[113771], 99.50th=[113771], 99.90th=[113771], 99.95th=[113771], | 99.99th=[113771] bw ( KiB/s): min= 4047, max= 4047, per=33.57%, avg=4047.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 write: IOPS=5, BW=5649KiB/s (5784kB/s)(7168KiB/1269msec) clat (msec): min=44, max=299, avg=124.60, stdev=103.95 lat (msec): min=44, max=299, avg=124.68, stdev=103.96 clat percentiles (msec): | 1.00th=[ 45], 5.00th=[ 45], 10.00th=[ 45], 20.00th=[ 57], | 30.00th=[ 58], 40.00th=[ 58], 50.00th=[ 63], 60.00th=[ 105], | 70.00th=[ 105], 80.00th=[ 247], 90.00th=[ 300], 95.00th=[ 300], | 99.00th=[ 300], 99.50th=[ 300], 99.90th=[ 300], 99.95th=[ 300], | 99.99th=[ 300] bw ( KiB/s): min= 4047, max= 4087, per=24.10%, avg=4067.00, stdev=28.28, samples=2 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=2 lat (usec) : 750=16.67% lat (msec) : 50=8.33%, 100=41.67%, 250=25.00%, 500=8.33% cpu : usr=0.08%, sys=17.43%, ctx=90, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=11.8MiB/s (12.3MB/s), 1205KiB/s-6255KiB/s (1234kB/s-6405kB/s), io=20.0MiB (20.0MB), run=1146-1699msec WRITE: bw=16.5MiB/s (17.3MB/s), 4468KiB/s-6027KiB/s (4575kB/s-6172kB/s), io=28.0MiB (29.4MB), run=1146-1699msec rand-rw: (groupid=0, jobs=1): err= 0: pid=24173: Thu May 1 21:52:08 2025 read: IOPS=5, BW=5689KiB/s (5825kB/s)(7168KiB/1260msec) clat (msec): min=25, max=299, avg=93.31, stdev=100.42 lat (msec): min=25, max=299, avg=93.31, stdev=100.42 clat percentiles (msec): | 1.00th=[ 26], 5.00th=[ 26], 10.00th=[ 26], 20.00th=[ 31], | 30.00th=[ 33], 40.00th=[ 33], 50.00th=[ 53], 60.00th=[ 63], | 70.00th=[ 63], 80.00th=[ 150], 90.00th=[ 300], 95.00th=[ 300], | 99.00th=[ 300], 99.50th=[ 300], 99.90th=[ 300], 99.95th=[ 300], | 99.99th=[ 300] bw ( KiB/s): min= 4096, max= 6144, per=39.85%, avg=5120.00, stdev=1448.15, samples=2 iops : min= 4, max= 6, avg= 5.00, stdev= 1.41, samples=2 write: IOPS=3, BW=4063KiB/s (4161kB/s)(5120KiB/1260msec) clat (msec): min=40, max=186, avg=110.44, stdev=60.60 lat (msec): min=40, max=186, avg=110.53, stdev=60.59 clat percentiles (msec): | 1.00th=[ 41], 5.00th=[ 41], 10.00th=[ 41], 20.00th=[ 41], | 30.00th=[ 82], 40.00th=[ 82], 50.00th=[ 85], 60.00th=[ 85], | 70.00th=[ 159], 80.00th=[ 159], 90.00th=[ 188], 95.00th=[ 188], | 99.00th=[ 188], 99.50th=[ 188], 99.90th=[ 188], 99.95th=[ 188], | 99.99th=[ 188] bw ( KiB/s): min= 2048, max= 4096, per=17.08%, avg=3072.00, stdev=1448.15, samples=2 iops : min= 2, max= 4, avg= 3.00, stdev= 1.41, samples=2 lat (msec) : 50=33.33%, 100=33.33%, 250=25.00%, 500=8.33% cpu : usr=0.00%, sys=6.59%, ctx=48, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24174: Thu May 1 21:52:08 2025 read: IOPS=1, BW=1285KiB/s (1316kB/s)(2048KiB/1594msec) clat (msec): min=112, max=213, avg=162.83, stdev=71.31 lat (msec): min=112, max=213, avg=162.83, stdev=71.31 clat percentiles (msec): | 1.00th=[ 113], 5.00th=[ 113], 10.00th=[ 113], 20.00th=[ 113], | 30.00th=[ 113], 40.00th=[ 113], 50.00th=[ 113], 60.00th=[ 213], | 70.00th=[ 213], 80.00th=[ 213], 90.00th=[ 213], 95.00th=[ 213], | 99.00th=[ 213], 99.50th=[ 213], 99.90th=[ 213], 99.95th=[ 213], | 99.99th=[ 213] bw ( KiB/s): min= 2048, max= 2048, per=15.94%, avg=2048.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=6, BW=6424KiB/s (6578kB/s)(10.0MiB/1594msec) clat (msec): min=21, max=300, avg=125.92, stdev=93.83 lat (msec): min=21, max=300, avg=126.02, stdev=93.82 clat percentiles (msec): | 1.00th=[ 22], 5.00th=[ 22], 10.00th=[ 22], 20.00th=[ 46], | 30.00th=[ 54], 40.00th=[ 63], 50.00th=[ 69], 60.00th=[ 117], | 70.00th=[ 205], 80.00th=[ 205], 90.00th=[ 236], 95.00th=[ 300], | 99.00th=[ 300], 99.50th=[ 300], 99.90th=[ 300], 99.95th=[ 300], | 99.99th=[ 300] bw ( KiB/s): min= 2048, max= 4096, per=17.08%, avg=3072.00, stdev=1448.15, samples=2 iops : min= 2, max= 4, avg= 3.00, stdev= 1.41, samples=2 lat (msec) : 50=16.67%, 100=25.00%, 250=50.00%, 500=8.33% cpu : usr=0.31%, sys=7.53%, ctx=62, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24175: Thu May 1 21:52:08 2025 read: IOPS=4, BW=4171KiB/s (4271kB/s)(6144KiB/1473msec) clat (msec): min=21, max=281, avg=125.85, stdev=105.87 lat (msec): min=21, max=281, avg=125.85, stdev=105.87 clat percentiles (msec): | 1.00th=[ 22], 5.00th=[ 22], 10.00th=[ 22], 20.00th=[ 31], | 30.00th=[ 31], 40.00th=[ 64], 50.00th=[ 64], 60.00th=[ 148], | 70.00th=[ 211], 80.00th=[ 211], 90.00th=[ 284], 95.00th=[ 284], | 99.00th=[ 284], 99.50th=[ 284], 99.90th=[ 284], 99.95th=[ 284], | 99.99th=[ 284] bw ( KiB/s): min= 4096, max= 4096, per=31.88%, avg=4096.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 write: IOPS=4, BW=4171KiB/s (4271kB/s)(6144KiB/1473msec) clat (msec): min=23, max=262, avg=115.03, stdev=101.10 lat (msec): min=23, max=262, avg=115.09, stdev=101.10 clat percentiles (msec): | 1.00th=[ 24], 5.00th=[ 24], 10.00th=[ 24], 20.00th=[ 27], | 30.00th=[ 27], 40.00th=[ 40], 50.00th=[ 40], 60.00th=[ 142], | 70.00th=[ 199], 80.00th=[ 199], 90.00th=[ 264], 95.00th=[ 264], | 99.00th=[ 264], 99.50th=[ 264], 99.90th=[ 264], 99.95th=[ 264], | 99.99th=[ 264] bw ( KiB/s): min= 4071, max= 4096, per=22.70%, avg=4083.50, stdev=17.68, samples=2 iops : min= 3, max= 4, avg= 3.50, stdev= 0.71, samples=2 lat (msec) : 50=41.67%, 100=8.33%, 250=33.33%, 500=16.67% cpu : usr=0.14%, sys=4.76%, ctx=38, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24176: Thu May 1 21:52:08 2025 read: IOPS=3, BW=3382KiB/s (3463kB/s)(5120KiB/1514msec) clat (msec): min=16, max=176, avg=109.77, stdev=76.68 lat (msec): min=16, max=176, avg=109.77, stdev=76.68 clat percentiles (msec): | 1.00th=[ 17], 5.00th=[ 17], 10.00th=[ 17], 20.00th=[ 17], | 30.00th=[ 37], 40.00th=[ 37], 50.00th=[ 159], 60.00th=[ 159], | 70.00th=[ 161], 80.00th=[ 161], 90.00th=[ 176], 95.00th=[ 176], | 99.00th=[ 176], 99.50th=[ 176], 99.90th=[ 176], 99.95th=[ 176], | 99.99th=[ 176] bw ( KiB/s): min= 8015, max= 8015, per=62.38%, avg=8015.00, stdev= 0.00, samples=1 iops : min= 7, max= 7, avg= 7.00, stdev= 0.00, samples=1 write: IOPS=4, BW=4734KiB/s (4848kB/s)(7168KiB/1514msec) clat (msec): min=34, max=350, avg=131.50, stdev=123.74 lat (msec): min=34, max=350, avg=131.58, stdev=123.75 clat percentiles (msec): | 1.00th=[ 35], 5.00th=[ 35], 10.00th=[ 35], 20.00th=[ 39], | 30.00th=[ 40], 40.00th=[ 40], 50.00th=[ 57], 60.00th=[ 169], | 70.00th=[ 169], 80.00th=[ 234], 90.00th=[ 351], 95.00th=[ 351], | 99.00th=[ 351], 99.50th=[ 351], 99.90th=[ 351], 99.95th=[ 351], | 99.99th=[ 351] bw ( KiB/s): min= 4087, max= 6011, per=26.30%, avg=4731.33, stdev=1108.23, samples=3 iops : min= 3, max= 5, avg= 4.00, stdev= 1.00, samples=3 lat (msec) : 20=8.33%, 50=33.33%, 100=8.33%, 250=41.67%, 500=8.33% cpu : usr=0.20%, sys=3.17%, ctx=32, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=12.5MiB/s (13.2MB/s), 1285KiB/s-5689KiB/s (1316kB/s-5825kB/s), io=20.0MiB (20.0MB), run=1260-1594msec WRITE: bw=17.6MiB/s (18.4MB/s), 4063KiB/s-6424KiB/s (4161kB/s-6578kB/s), io=28.0MiB (29.4MB), run=1260-1594msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=24183: Thu May 1 21:52:10 2025 read: IOPS=14, BW=57.4MiB/s (60.2MB/s)(12.0MiB/209msec) clat (msec): min=35, max=121, avg=64.29, stdev=49.40 lat (msec): min=35, max=121, avg=64.29, stdev=49.40 clat percentiles (msec): | 1.00th=[ 36], 5.00th=[ 36], 10.00th=[ 36], 20.00th=[ 36], | 30.00th=[ 36], 40.00th=[ 37], 50.00th=[ 37], 60.00th=[ 37], | 70.00th=[ 122], 80.00th=[ 122], 90.00th=[ 122], 95.00th=[ 122], | 99.00th=[ 122], 99.50th=[ 122], 99.90th=[ 122], 99.95th=[ 122], | 99.99th=[ 122] lat (msec) : 50=66.67%, 250=33.33% cpu : usr=0.96%, sys=6.73%, ctx=16, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24184: Thu May 1 21:52:10 2025 read: IOPS=1, BW=7892KiB/s (8082kB/s)(4096KiB/519msec) clat (nsec): min=132420k, max=132420k, avg=132420305.00, stdev= 0.00 lat (nsec): min=132423k, max=132423k, avg=132422542.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 133], 5.00th=[ 133], 10.00th=[ 133], 20.00th=[ 133], | 30.00th=[ 133], 40.00th=[ 133], 50.00th=[ 133], 60.00th=[ 133], | 70.00th=[ 133], 80.00th=[ 133], 90.00th=[ 133], 95.00th=[ 133], | 99.00th=[ 133], 99.50th=[ 133], 99.90th=[ 133], 99.95th=[ 133], | 99.99th=[ 133] bw ( KiB/s): min= 8192, max= 8192, per=25.95%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=3, BW=15.4MiB/s (16.2MB/s)(8192KiB/519msec) clat (msec): min=90, max=236, avg=163.43, stdev=102.75 lat (msec): min=91, max=236, avg=163.77, stdev=102.76 clat percentiles (msec): | 1.00th=[ 91], 5.00th=[ 91], 10.00th=[ 91], 20.00th=[ 91], | 30.00th=[ 91], 40.00th=[ 91], 50.00th=[ 91], 60.00th=[ 236], | 70.00th=[ 236], 80.00th=[ 236], 90.00th=[ 236], 95.00th=[ 236], | 99.00th=[ 236], 99.50th=[ 236], 99.90th=[ 236], 99.95th=[ 236], | 99.99th=[ 236] bw ( KiB/s): min= 8192, max= 8192, per=23.15%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 250=66.67% cpu : usr=0.39%, sys=4.44%, ctx=25, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24185: Thu May 1 21:52:10 2025 write: IOPS=5, BW=23.2MiB/s (24.3MB/s)(12.0MiB/518msec) clat (msec): min=64, max=336, avg=162.38, stdev=151.09 lat (msec): min=65, max=336, avg=162.80, stdev=151.04 clat percentiles (msec): | 1.00th=[ 65], 5.00th=[ 65], 10.00th=[ 65], 20.00th=[ 65], | 30.00th=[ 65], 40.00th=[ 86], 50.00th=[ 86], 60.00th=[ 86], | 70.00th=[ 338], 80.00th=[ 338], 90.00th=[ 338], 95.00th=[ 338], | 99.00th=[ 338], 99.50th=[ 338], 99.90th=[ 338], 99.95th=[ 338], | 99.99th=[ 338] bw ( KiB/s): min=16351, max=16351, per=46.21%, avg=16351.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 lat (msec) : 100=66.67%, 500=33.33% cpu : usr=0.58%, sys=3.66%, ctx=13, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24186: Thu May 1 21:52:10 2025 write: IOPS=3, BW=12.0MiB/s (13.6MB/s)(12.0MiB/926msec) clat (msec): min=88, max=516, avg=287.05, stdev=215.51 lat (msec): min=88, max=516, avg=287.54, stdev=215.33 clat percentiles (msec): | 1.00th=[ 89], 5.00th=[ 89], 10.00th=[ 89], 20.00th=[ 89], | 30.00th=[ 89], 40.00th=[ 257], 50.00th=[ 257], 60.00th=[ 257], | 70.00th=[ 518], 80.00th=[ 518], 90.00th=[ 518], 95.00th=[ 518], | 99.00th=[ 518], 99.50th=[ 518], 99.90th=[ 518], 99.95th=[ 518], | 99.99th=[ 518] bw ( KiB/s): min=16318, max=16318, per=46.11%, avg=16318.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 500=33.33%, 750=33.33% cpu : usr=0.00%, sys=7.35%, ctx=44, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=30.8MiB/s (32.3MB/s), 7892KiB/s-57.4MiB/s (8082kB/s-60.2MB/s), io=16.0MiB (16.8MB), run=209-519msec WRITE: bw=34.6MiB/s (36.2MB/s), 12.0MiB/s-23.2MiB/s (13.6MB/s-24.3MB/s), io=32.0MiB (33.6MB), run=518-926msec rand-rw: (groupid=0, jobs=1): err= 0: pid=24188: Thu May 1 21:52:11 2025 read: IOPS=5, BW=20.1MiB/s (21.0MB/s)(12.0MiB/598msec) clat (msec): min=80, max=262, avg=191.17, stdev=97.28 lat (msec): min=80, max=262, avg=191.17, stdev=97.28 clat percentiles (msec): | 1.00th=[ 81], 5.00th=[ 81], 10.00th=[ 81], 20.00th=[ 81], | 30.00th=[ 81], 40.00th=[ 230], 50.00th=[ 230], 60.00th=[ 230], | 70.00th=[ 264], 80.00th=[ 264], 90.00th=[ 264], 95.00th=[ 264], | 99.00th=[ 264], 99.50th=[ 264], 99.90th=[ 264], 99.95th=[ 264], | 99.99th=[ 264] bw ( KiB/s): min= 8192, max= 8192, per=52.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 250=33.33%, 500=33.33% cpu : usr=0.50%, sys=26.13%, ctx=370, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24189: Thu May 1 21:52:11 2025 read: IOPS=0, BW=3938KiB/s (4033kB/s)(4096KiB/1040msec) clat (nsec): min=201651k, max=201651k, avg=201650947.00, stdev= 0.00 lat (nsec): min=201653k, max=201653k, avg=201653127.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 203], 5.00th=[ 203], 10.00th=[ 203], 20.00th=[ 203], | 30.00th=[ 203], 40.00th=[ 203], 50.00th=[ 203], 60.00th=[ 203], | 70.00th=[ 203], 80.00th=[ 203], 90.00th=[ 203], 95.00th=[ 203], | 99.00th=[ 203], 99.50th=[ 203], 99.90th=[ 203], 99.95th=[ 203], | 99.99th=[ 203] bw ( KiB/s): min= 8192, max= 8192, per=52.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=1, BW=7877KiB/s (8066kB/s)(8192KiB/1040msec) clat (msec): min=288, max=493, avg=391.13, stdev=144.59 lat (msec): min=289, max=493, avg=391.38, stdev=144.70 clat percentiles (msec): | 1.00th=[ 288], 5.00th=[ 288], 10.00th=[ 288], 20.00th=[ 288], | 30.00th=[ 288], 40.00th=[ 288], 50.00th=[ 288], 60.00th=[ 493], | 70.00th=[ 493], 80.00th=[ 493], 90.00th=[ 493], 95.00th=[ 493], | 99.00th=[ 493], 99.50th=[ 493], 99.90th=[ 493], 99.95th=[ 493], | 99.99th=[ 493] lat (msec) : 250=33.33%, 500=66.67% cpu : usr=0.00%, sys=24.90%, ctx=384, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24190: Thu May 1 21:52:11 2025 write: IOPS=3, BW=12.2MiB/s (12.8MB/s)(12.0MiB/983msec) clat (msec): min=211, max=489, avg=317.47, stdev=150.63 lat (msec): min=211, max=490, avg=317.83, stdev=150.69 clat percentiles (msec): | 1.00th=[ 211], 5.00th=[ 211], 10.00th=[ 211], 20.00th=[ 211], | 30.00th=[ 211], 40.00th=[ 251], 50.00th=[ 251], 60.00th=[ 251], | 70.00th=[ 489], 80.00th=[ 489], 90.00th=[ 489], 95.00th=[ 489], | 99.00th=[ 489], 99.50th=[ 489], 99.90th=[ 489], 99.95th=[ 489], | 99.99th=[ 489] lat (msec) : 250=33.33%, 500=66.67% cpu : usr=0.00%, sys=27.49%, ctx=102, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24191: Thu May 1 21:52:11 2025 write: IOPS=2, BW=10.8MiB/s (11.3MB/s)(12.0MiB/1112msec) clat (msec): min=158, max=553, avg=346.00, stdev=198.12 lat (msec): min=159, max=553, avg=346.30, stdev=198.15 clat percentiles (msec): | 1.00th=[ 159], 5.00th=[ 159], 10.00th=[ 159], 20.00th=[ 159], | 30.00th=[ 159], 40.00th=[ 326], 50.00th=[ 326], 60.00th=[ 326], | 70.00th=[ 550], 80.00th=[ 550], 90.00th=[ 550], 95.00th=[ 550], | 99.00th=[ 550], 99.50th=[ 550], 99.90th=[ 550], 99.95th=[ 550], | 99.99th=[ 550] bw ( KiB/s): min=16253, max=16253, per=55.16%, avg=16253.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=33.33%, 750=33.33% cpu : usr=0.00%, sys=27.99%, ctx=54, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=15.4MiB/s (16.1MB/s), 3938KiB/s-20.1MiB/s (4033kB/s-21.0MB/s), io=16.0MiB (16.8MB), run=598-1040msec WRITE: bw=28.8MiB/s (30.2MB/s), 7877KiB/s-12.2MiB/s (8066kB/s-12.8MB/s), io=32.0MiB (33.6MB), run=983-1112msec