== sanity test 398b: DIO and buffer IO race ============== 19:56:43 (1746143803) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.55348 s, 32.4 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=22615: Thu May 1 19:57:15 2025 read: IOPS=51, BW=205KiB/s (209kB/s)(5992KiB/29294msec) clat (usec): min=72, max=69088, avg=10908.73, stdev=7316.57 lat (usec): min=72, max=69089, avg=10910.44, stdev=7316.48 clat percentiles (usec): | 1.00th=[ 93], 5.00th=[ 125], 10.00th=[ 4228], 20.00th=[ 5669], | 30.00th=[ 6980], 40.00th=[ 8094], 50.00th=[ 9372], 60.00th=[10945], | 70.00th=[12780], 80.00th=[15533], 90.00th=[20055], 95.00th=[23987], | 99.00th=[35390], 99.50th=[41157], 99.90th=[60031], 99.95th=[68682], | 99.99th=[68682] bw ( KiB/s): min= 103, max= 328, per=24.74%, avg=202.60, stdev=52.59, samples=58 iops : min= 25, max= 82, avg=50.40, stdev=13.14, samples=58 write: IOPS=53, BW=215KiB/s (220kB/s)(6296KiB/29294msec) clat (usec): min=460, max=211690, avg=8141.93, stdev=10090.26 lat (usec): min=461, max=211692, avg=8143.17, stdev=10090.29 clat percentiles (usec): | 1.00th=[ 619], 5.00th=[ 734], 10.00th=[ 791], 20.00th=[ 914], | 30.00th=[ 1369], 40.00th=[ 2409], 50.00th=[ 5604], 60.00th=[ 7570], | 70.00th=[ 10159], 80.00th=[ 13960], 90.00th=[ 19530], 95.00th=[ 25297], | 99.00th=[ 38536], 99.50th=[ 42730], 99.90th=[ 62653], 99.95th=[210764], | 99.99th=[210764] bw ( KiB/s): min= 95, max= 327, per=24.70%, avg=211.97, stdev=54.59, samples=58 iops : min= 23, max= 81, avg=52.72, stdev=13.69, samples=58 lat (usec) : 100=0.94%, 250=1.99%, 500=0.13%, 750=3.55%, 1000=8.50% lat (msec) : 2=7.03%, 4=5.31%, 10=34.83%, 20=27.73%, 50=9.70% lat (msec) : 100=0.26%, 250=0.03% cpu : usr=0.19%, sys=14.38%, ctx=5291, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22616: Thu May 1 19:57:15 2025 read: IOPS=51, BW=204KiB/s (209kB/s)(5940KiB/29112msec) clat (usec): min=65, max=67975, avg=10694.89, stdev=7310.71 lat (usec): min=65, max=67976, avg=10696.39, stdev=7310.82 clat percentiles (usec): | 1.00th=[ 88], 5.00th=[ 121], 10.00th=[ 3982], 20.00th=[ 5473], | 30.00th=[ 6587], 40.00th=[ 7898], 50.00th=[ 9241], 60.00th=[10814], | 70.00th=[13042], 80.00th=[15270], 90.00th=[19006], 95.00th=[22938], | 99.00th=[36963], 99.50th=[39584], 99.90th=[64750], 99.95th=[67634], | 99.99th=[67634] bw ( KiB/s): min= 80, max= 296, per=24.82%, avg=203.28, stdev=48.81, samples=58 iops : min= 20, max= 74, avg=50.62, stdev=12.20, samples=58 write: IOPS=54, BW=218KiB/s (223kB/s)(6348KiB/29112msec) clat (usec): min=431, max=225347, avg=8255.92, stdev=10161.98 lat (usec): min=432, max=225348, avg=8258.69, stdev=10161.19 clat percentiles (usec): | 1.00th=[ 619], 5.00th=[ 734], 10.00th=[ 791], 20.00th=[ 963], | 30.00th=[ 1467], 40.00th=[ 2769], 50.00th=[ 5604], 60.00th=[ 7832], | 70.00th=[ 10552], 80.00th=[ 14615], 90.00th=[ 20055], 95.00th=[ 24773], | 99.00th=[ 37487], 99.50th=[ 47973], 99.90th=[ 68682], 99.95th=[225444], | 99.99th=[225444] bw ( KiB/s): min= 112, max= 404, per=25.29%, avg=217.00, stdev=60.06, samples=58 iops : min= 28, max= 101, avg=54.05, stdev=14.98, samples=58 lat (usec) : 100=1.11%, 250=2.08%, 500=0.26%, 750=3.16%, 1000=7.68% lat (msec) : 2=7.78%, 4=5.57%, 10=33.53%, 20=29.65%, 50=8.85% lat (msec) : 100=0.29%, 250=0.03% cpu : usr=0.17%, sys=14.54%, ctx=5213, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22618: Thu May 1 19:57:15 2025 read: IOPS=52, BW=208KiB/s (213kB/s)(6028KiB/28965msec) clat (usec): min=61, max=54874, avg=10537.81, stdev=6527.60 lat (usec): min=62, max=54875, avg=10540.31, stdev=6526.56 clat percentiles (usec): | 1.00th=[ 89], 5.00th=[ 129], 10.00th=[ 4178], 20.00th=[ 5669], | 30.00th=[ 6849], 40.00th=[ 8094], 50.00th=[ 9372], 60.00th=[10683], | 70.00th=[12649], 80.00th=[14877], 90.00th=[19006], 95.00th=[22152], | 99.00th=[31589], 99.50th=[36963], 99.90th=[47973], 99.95th=[54789], | 99.99th=[54789] bw ( KiB/s): min= 48, max= 328, per=25.20%, avg=206.42, stdev=53.97, samples=57 iops : min= 12, max= 82, avg=51.40, stdev=13.51, samples=57 write: IOPS=54, BW=216KiB/s (221kB/s)(6260KiB/28965msec) clat (usec): min=428, max=217578, avg=8260.67, stdev=10057.20 lat (usec): min=429, max=217581, avg=8262.46, stdev=10057.80 clat percentiles (usec): | 1.00th=[ 594], 5.00th=[ 717], 10.00th=[ 783], 20.00th=[ 889], | 30.00th=[ 1303], 40.00th=[ 2606], 50.00th=[ 5997], 60.00th=[ 8225], | 70.00th=[ 10552], 80.00th=[ 14091], 90.00th=[ 19792], 95.00th=[ 24511], | 99.00th=[ 37487], 99.50th=[ 41681], 99.90th=[ 62129], 99.95th=[217056], | 99.99th=[217056] bw ( KiB/s): min= 56, max= 408, per=24.94%, avg=214.00, stdev=54.96, samples=57 iops : min= 14, max= 102, avg=53.30, stdev=13.74, samples=57 lat (usec) : 100=1.30%, 250=1.63%, 500=0.26%, 750=3.74%, 1000=8.27% lat (msec) : 2=6.77%, 4=4.65%, 10=35.09%, 20=29.43%, 50=8.69% lat (msec) : 100=0.13%, 250=0.03% cpu : usr=0.19%, sys=14.56%, ctx=5276, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22620: Thu May 1 19:57:15 2025 read: IOPS=52, BW=211KiB/s (216kB/s)(6052KiB/28663msec) clat (usec): min=49, max=59550, avg=10324.50, stdev=7117.90 lat (usec): min=49, max=59573, avg=10325.83, stdev=7117.84 clat percentiles (usec): | 1.00th=[ 94], 5.00th=[ 126], 10.00th=[ 3785], 20.00th=[ 5276], | 30.00th=[ 6456], 40.00th=[ 7570], 50.00th=[ 8848], 60.00th=[10552], | 70.00th=[12387], 80.00th=[14746], 90.00th=[18482], 95.00th=[22676], | 99.00th=[38011], 99.50th=[42206], 99.90th=[58983], 99.95th=[59507], | 99.99th=[59507] bw ( KiB/s): min= 120, max= 311, per=25.75%, avg=210.91, stdev=45.56, samples=57 iops : min= 30, max= 77, avg=52.53, stdev=11.38, samples=57 write: IOPS=54, BW=218KiB/s (223kB/s)(6236KiB/28663msec) clat (usec): min=436, max=226987, avg=8280.85, stdev=10337.71 lat (usec): min=436, max=226988, avg=8284.02, stdev=10338.41 clat percentiles (usec): | 1.00th=[ 603], 5.00th=[ 734], 10.00th=[ 783], 20.00th=[ 914], | 30.00th=[ 1287], 40.00th=[ 2442], 50.00th=[ 5800], 60.00th=[ 8094], | 70.00th=[ 10290], 80.00th=[ 14222], 90.00th=[ 20055], 95.00th=[ 26346], | 99.00th=[ 37487], 99.50th=[ 42206], 99.90th=[ 67634], 99.95th=[227541], | 99.99th=[227541] bw ( KiB/s): min= 120, max= 343, per=25.30%, avg=217.11, stdev=53.78, samples=57 iops : min= 30, max= 85, avg=54.07, stdev=13.45, samples=57 lat (usec) : 50=0.03%, 100=0.91%, 250=2.28%, 500=0.20%, 750=3.42% lat (usec) : 1000=8.20% lat (msec) : 2=7.49%, 4=5.31%, 10=35.38%, 20=27.54%, 50=9.02% lat (msec) : 100=0.20%, 250=0.03% cpu : usr=0.21%, sys=14.66%, ctx=5039, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=820KiB/s (839kB/s), 204KiB/s-211KiB/s (209kB/s-216kB/s), io=23.4MiB (24.6MB), run=28663-29294msec WRITE: bw=858KiB/s (879kB/s), 215KiB/s-218KiB/s (220kB/s-223kB/s), io=24.6MiB (25.7MB), run=28663-29294msec rand-rw: (groupid=0, jobs=1): err= 0: pid=22617: Thu May 1 19:57:30 2025 read: IOPS=34, BW=138KiB/s (141kB/s)(5992KiB/43556msec) clat (msec): min=2, max=239, avg=11.10, stdev=11.08 lat (msec): min=2, max=239, avg=11.11, stdev=11.08 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 4], 10.00th=[ 5], 20.00th=[ 6], | 30.00th=[ 6], 40.00th=[ 7], 50.00th=[ 8], 60.00th=[ 9], | 70.00th=[ 11], 80.00th=[ 15], 90.00th=[ 24], 95.00th=[ 31], | 99.00th=[ 51], 99.50th=[ 63], 99.90th=[ 73], 99.95th=[ 241], | 99.99th=[ 241] bw ( KiB/s): min= 47, max= 304, per=25.07%, avg=137.38, stdev=65.99, samples=87 iops : min= 11, max= 76, avg=34.14, stdev=16.55, samples=87 write: IOPS=36, BW=145KiB/s (148kB/s)(6296KiB/43556msec) clat (usec): min=4399, max=83746, avg=17001.57, stdev=10189.12 lat (usec): min=4400, max=83747, avg=17002.97, stdev=10189.11 clat percentiles (usec): | 1.00th=[ 5604], 5.00th=[ 7504], 10.00th=[ 8356], 20.00th=[ 9634], | 30.00th=[10814], 40.00th=[12125], 50.00th=[13566], 60.00th=[15664], | 70.00th=[18482], 80.00th=[22938], 90.00th=[30278], 95.00th=[37487], | 99.00th=[54789], 99.50th=[57934], 99.90th=[76022], 99.95th=[83362], | 99.99th=[83362] bw ( KiB/s): min= 39, max= 312, per=25.10%, avg=144.08, stdev=64.53, samples=87 iops : min= 9, max= 78, avg=35.82, stdev=16.16, samples=87 lat (msec) : 4=3.65%, 10=40.46%, 20=36.46%, 50=18.13%, 100=1.27% lat (msec) : 250=0.03% cpu : usr=0.14%, sys=6.54%, ctx=3543, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22619: Thu May 1 19:57:30 2025 read: IOPS=33, BW=136KiB/s (139kB/s)(5940KiB/43702msec) clat (msec): min=2, max=208, avg=10.72, stdev=10.00 lat (msec): min=2, max=208, avg=10.73, stdev=10.00 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 4], 10.00th=[ 5], 20.00th=[ 6], | 30.00th=[ 6], 40.00th=[ 7], 50.00th=[ 8], 60.00th=[ 9], | 70.00th=[ 11], 80.00th=[ 15], 90.00th=[ 22], 95.00th=[ 29], | 99.00th=[ 46], 99.50th=[ 54], 99.90th=[ 78], 99.95th=[ 209], | 99.99th=[ 209] bw ( KiB/s): min= 16, max= 344, per=24.71%, avg=135.43, stdev=67.50, samples=87 iops : min= 4, max= 86, avg=33.63, stdev=16.94, samples=87 write: IOPS=36, BW=145KiB/s (149kB/s)(6348KiB/43702msec) clat (usec): min=3470, max=97655, avg=17412.33, stdev=10789.04 lat (usec): min=3471, max=97657, avg=17414.46, stdev=10789.08 clat percentiles (usec): | 1.00th=[ 5669], 5.00th=[ 7242], 10.00th=[ 8225], 20.00th=[ 9765], | 30.00th=[10945], 40.00th=[12256], 50.00th=[13698], 60.00th=[15926], | 70.00th=[18744], 80.00th=[23200], 90.00th=[31851], 95.00th=[40109], | 99.00th=[57410], 99.50th=[65274], 99.90th=[79168], 99.95th=[98042], | 99.99th=[98042] bw ( KiB/s): min= 48, max= 311, per=25.16%, avg=144.43, stdev=64.76, samples=87 iops : min= 12, max= 77, avg=35.89, stdev=16.21, samples=87 lat (msec) : 4=3.58%, 10=39.97%, 20=36.85%, 50=18.26%, 100=1.30% lat (msec) : 250=0.03% cpu : usr=0.08%, sys=6.61%, ctx=3575, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22621: Thu May 1 19:57:30 2025 read: IOPS=34, BW=139KiB/s (142kB/s)(6028KiB/43506msec) clat (usec): min=2505, max=77669, avg=10779.29, stdev=9143.43 lat (usec): min=2506, max=77670, avg=10780.59, stdev=9143.37 clat percentiles (usec): | 1.00th=[ 3064], 5.00th=[ 3752], 10.00th=[ 4293], 20.00th=[ 5145], | 30.00th=[ 5866], 40.00th=[ 6652], 50.00th=[ 7570], 60.00th=[ 8586], | 70.00th=[10552], 80.00th=[14222], 90.00th=[22414], 95.00th=[30016], | 99.00th=[45351], 99.50th=[56361], 99.90th=[71828], 99.95th=[78119], | 99.99th=[78119] bw ( KiB/s): min= 48, max= 320, per=25.23%, avg=138.28, stdev=63.40, samples=87 iops : min= 12, max= 80, avg=34.36, stdev=15.87, samples=87 write: IOPS=35, BW=144KiB/s (147kB/s)(6260KiB/43506msec) clat (msec): min=4, max=240, avg=17.34, stdev=11.97 lat (msec): min=4, max=240, avg=17.34, stdev=11.97 clat percentiles (msec): | 1.00th=[ 6], 5.00th=[ 8], 10.00th=[ 9], 20.00th=[ 10], | 30.00th=[ 11], 40.00th=[ 13], 50.00th=[ 14], 60.00th=[ 16], | 70.00th=[ 19], 80.00th=[ 24], 90.00th=[ 32], 95.00th=[ 39], | 99.00th=[ 56], 99.50th=[ 72], 99.90th=[ 111], 99.95th=[ 241], | 99.99th=[ 241] bw ( KiB/s): min= 31, max= 311, per=25.01%, avg=143.53, stdev=65.52, samples=87 iops : min= 7, max= 77, avg=35.67, stdev=16.43, samples=87 lat (msec) : 4=3.35%, 10=41.73%, 20=35.19%, 50=18.55%, 100=1.11% lat (msec) : 250=0.07% cpu : usr=0.04%, sys=6.77%, ctx=3576, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22622: Thu May 1 19:57:30 2025 read: IOPS=34, BW=138KiB/s (142kB/s)(6052KiB/43771msec) clat (usec): min=2492, max=72358, avg=10824.23, stdev=9233.41 lat (usec): min=2492, max=72359, avg=10826.26, stdev=9234.08 clat percentiles (usec): | 1.00th=[ 3064], 5.00th=[ 3851], 10.00th=[ 4359], 20.00th=[ 5211], | 30.00th=[ 5866], 40.00th=[ 6652], 50.00th=[ 7439], 60.00th=[ 8586], | 70.00th=[10421], 80.00th=[13829], 90.00th=[23725], 95.00th=[30802], | 99.00th=[46924], 99.50th=[56886], 99.90th=[67634], 99.95th=[72877], | 99.99th=[72877] bw ( KiB/s): min= 32, max= 319, per=25.04%, avg=137.24, stdev=66.78, samples=87 iops : min= 8, max= 79, avg=34.07, stdev=16.72, samples=87 write: IOPS=35, BW=142KiB/s (146kB/s)(6236KiB/43771msec) clat (msec): min=3, max=235, avg=17.49, stdev=12.24 lat (msec): min=3, max=235, avg=17.49, stdev=12.25 clat percentiles (msec): | 1.00th=[ 6], 5.00th=[ 8], 10.00th=[ 9], 20.00th=[ 10], | 30.00th=[ 11], 40.00th=[ 13], 50.00th=[ 14], 60.00th=[ 17], | 70.00th=[ 20], 80.00th=[ 24], 90.00th=[ 32], 95.00th=[ 40], | 99.00th=[ 60], 99.50th=[ 66], 99.90th=[ 113], 99.95th=[ 236], | 99.99th=[ 236] bw ( KiB/s): min= 55, max= 296, per=24.69%, avg=141.75, stdev=62.71, samples=87 iops : min= 13, max= 74, avg=35.20, stdev=15.70, samples=87 lat (msec) : 4=3.12%, 10=42.77%, 20=34.44%, 50=18.23%, 100=1.37% lat (msec) : 250=0.07% cpu : usr=0.16%, sys=6.54%, ctx=3575, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=549KiB/s (562kB/s), 136KiB/s-139KiB/s (139kB/s-142kB/s), io=23.4MiB (24.6MB), run=43506-43771msec WRITE: bw=574KiB/s (588kB/s), 142KiB/s-145KiB/s (146kB/s-149kB/s), io=24.6MiB (25.7MB), run=43506-43771msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=22673: Thu May 1 19:57:39 2025 read: IOPS=45, BW=724KiB/s (741kB/s)(5920KiB/8180msec) clat (usec): min=84, max=59003, avg=10913.76, stdev=8066.87 lat (usec): min=84, max=59004, avg=10915.77, stdev=8066.38 clat percentiles (usec): | 1.00th=[ 93], 5.00th=[ 109], 10.00th=[ 128], 20.00th=[ 5407], | 30.00th=[ 6915], 40.00th=[ 8455], 50.00th=[ 9503], 60.00th=[11600], | 70.00th=[13829], 80.00th=[16319], 90.00th=[21365], 95.00th=[25297], | 99.00th=[34341], 99.50th=[55837], 99.90th=[58983], 99.95th=[58983], | 99.99th=[58983] bw ( KiB/s): min= 224, max= 1104, per=24.75%, avg=722.19, stdev=222.65, samples=16 iops : min= 14, max= 69, avg=44.88, stdev=13.87, samples=16 write: IOPS=48, BW=778KiB/s (797kB/s)(6368KiB/8180msec) clat (usec): min=784, max=71554, avg=10082.18, stdev=10978.01 lat (usec): min=784, max=71556, avg=10084.91, stdev=10978.83 clat percentiles (usec): | 1.00th=[ 938], 5.00th=[ 1057], 10.00th=[ 1205], 20.00th=[ 1598], | 30.00th=[ 2212], 40.00th=[ 4146], 50.00th=[ 6390], 60.00th=[ 9241], | 70.00th=[12649], 80.00th=[16319], 90.00th=[22938], 95.00th=[31589], | 99.00th=[53216], 99.50th=[65274], 99.90th=[71828], 99.95th=[71828], | 99.99th=[71828] bw ( KiB/s): min= 352, max= 1120, per=25.35%, avg=778.25, stdev=237.51, samples=16 iops : min= 22, max= 70, avg=48.38, stdev=14.72, samples=16 lat (usec) : 100=1.17%, 250=5.86%, 500=0.13%, 750=0.13%, 1000=1.30% lat (msec) : 2=13.67%, 4=6.12%, 10=28.65%, 20=29.82%, 50=12.24% lat (msec) : 100=0.91% cpu : usr=0.32%, sys=14.67%, ctx=1401, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22674: Thu May 1 19:57:39 2025 read: IOPS=47, BW=760KiB/s (778kB/s)(5744KiB/7562msec) clat (usec): min=68, max=65658, avg=11070.44, stdev=7663.02 lat (usec): min=69, max=65658, avg=11072.29, stdev=7663.36 clat percentiles (usec): | 1.00th=[ 90], 5.00th=[ 108], 10.00th=[ 127], 20.00th=[ 5800], | 30.00th=[ 7111], 40.00th=[ 8455], 50.00th=[10159], 60.00th=[11731], | 70.00th=[13960], 80.00th=[17433], 90.00th=[20579], 95.00th=[24249], | 99.00th=[32113], 99.50th=[34341], 99.90th=[65799], 99.95th=[65799], | 99.99th=[65799] bw ( KiB/s): min= 446, max= 992, per=26.16%, avg=763.33, stdev=161.64, samples=15 iops : min= 27, max= 62, avg=47.60, stdev=10.15, samples=15 write: IOPS=54, BW=865KiB/s (886kB/s)(6544KiB/7562msec) clat (usec): min=783, max=85567, avg=8653.56, stdev=9477.22 lat (usec): min=784, max=85568, avg=8656.50, stdev=9476.32 clat percentiles (usec): | 1.00th=[ 930], 5.00th=[ 1074], 10.00th=[ 1188], 20.00th=[ 1369], | 30.00th=[ 1778], 40.00th=[ 2933], 50.00th=[ 5407], 60.00th=[ 7373], | 70.00th=[10683], 80.00th=[15008], 90.00th=[22414], 95.00th=[27919], | 99.00th=[35914], 99.50th=[38536], 99.90th=[85459], 99.95th=[85459], | 99.99th=[85459] bw ( KiB/s): min= 576, max= 1280, per=28.27%, avg=867.80, stdev=234.46, samples=15 iops : min= 36, max= 80, avg=54.13, stdev=14.73, samples=15 lat (usec) : 100=1.95%, 250=3.91%, 500=0.52%, 750=0.13%, 1000=1.04% lat (msec) : 2=16.15%, 4=7.03%, 10=28.12%, 20=29.69%, 50=11.20% lat (msec) : 100=0.26% cpu : usr=0.19%, sys=15.47%, ctx=1406, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22675: Thu May 1 19:57:39 2025 read: IOPS=47, BW=762KiB/s (781kB/s)(6256KiB/8206msec) clat (usec): min=87, max=67087, avg=12142.29, stdev=9071.84 lat (usec): min=88, max=67089, avg=12143.38, stdev=9071.87 clat percentiles (usec): | 1.00th=[ 92], 5.00th=[ 108], 10.00th=[ 125], 20.00th=[ 5669], | 30.00th=[ 7767], 40.00th=[ 9372], 50.00th=[11076], 60.00th=[12780], | 70.00th=[15008], 80.00th=[17695], 90.00th=[22414], 95.00th=[27395], | 99.00th=[48497], 99.50th=[60031], 99.90th=[66847], 99.95th=[66847], | 99.99th=[66847] bw ( KiB/s): min= 480, max= 1088, per=26.21%, avg=764.81, stdev=189.20, samples=16 iops : min= 30, max= 68, avg=47.56, stdev=11.79, samples=16 write: IOPS=45, BW=735KiB/s (753kB/s)(6032KiB/8206msec) clat (usec): min=756, max=46298, avg=8936.98, stdev=9382.31 lat (usec): min=756, max=46299, avg=8938.81, stdev=9382.34 clat percentiles (usec): | 1.00th=[ 971], 5.00th=[ 1139], 10.00th=[ 1205], 20.00th=[ 1614], | 30.00th=[ 2180], 40.00th=[ 3294], 50.00th=[ 5866], 60.00th=[ 8225], | 70.00th=[10552], 80.00th=[14091], 90.00th=[21365], 95.00th=[29754], | 99.00th=[44827], 99.50th=[46400], 99.90th=[46400], 99.95th=[46400], | 99.99th=[46400] bw ( KiB/s): min= 512, max= 1152, per=23.59%, avg=724.19, stdev=172.38, samples=16 iops : min= 32, max= 72, avg=45.00, stdev=10.75, samples=16 lat (usec) : 100=1.56%, 250=5.60%, 500=0.39%, 750=0.13%, 1000=0.65% lat (msec) : 2=12.76%, 4=7.68%, 10=27.21%, 20=30.60%, 50=13.02% lat (msec) : 100=0.39% cpu : usr=0.16%, sys=14.86%, ctx=1436, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22676: Thu May 1 19:57:39 2025 read: IOPS=49, BW=793KiB/s (812kB/s)(6032KiB/7611msec) clat (usec): min=80, max=54864, avg=11294.97, stdev=8782.26 lat (usec): min=80, max=54866, avg=11296.06, stdev=8782.31 clat percentiles (usec): | 1.00th=[ 93], 5.00th=[ 106], 10.00th=[ 119], 20.00th=[ 5604], | 30.00th=[ 7242], 40.00th=[ 8455], 50.00th=[10028], 60.00th=[11863], | 70.00th=[13960], 80.00th=[16319], 90.00th=[20841], 95.00th=[27657], | 99.00th=[47449], 99.50th=[53740], 99.90th=[54789], 99.95th=[54789], | 99.99th=[54789] bw ( KiB/s): min= 509, max= 992, per=26.82%, avg=782.47, stdev=124.59, samples=15 iops : min= 31, max= 62, avg=48.80, stdev= 7.86, samples=15 write: IOPS=51, BW=822KiB/s (842kB/s)(6256KiB/7611msec) clat (usec): min=696, max=53499, avg=8513.12, stdev=9119.19 lat (usec): min=697, max=53502, avg=8516.84, stdev=9119.13 clat percentiles (usec): | 1.00th=[ 947], 5.00th=[ 1106], 10.00th=[ 1172], 20.00th=[ 1303], | 30.00th=[ 1745], 40.00th=[ 2540], 50.00th=[ 5538], 60.00th=[ 7963], | 70.00th=[10945], 80.00th=[14615], 90.00th=[20317], 95.00th=[26346], | 99.00th=[44827], 99.50th=[52167], 99.90th=[53740], 99.95th=[53740], | 99.99th=[53740] bw ( KiB/s): min= 608, max= 1056, per=26.53%, avg=814.40, stdev=112.41, samples=15 iops : min= 38, max= 66, avg=50.80, stdev= 6.97, samples=15 lat (usec) : 100=1.43%, 250=5.86%, 750=0.26%, 1000=0.91% lat (msec) : 2=16.54%, 4=6.77%, 10=26.69%, 20=30.86%, 50=9.90% lat (msec) : 100=0.78% cpu : usr=0.14%, sys=15.32%, ctx=1339, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2919KiB/s (2989kB/s), 724KiB/s-793KiB/s (741kB/s-812kB/s), io=23.4MiB (24.5MB), run=7562-8206msec WRITE: bw=3071KiB/s (3145kB/s), 735KiB/s-865KiB/s (753kB/s-886kB/s), io=24.6MiB (25.8MB), run=7562-8206msec rand-rw: (groupid=0, jobs=1): err= 0: pid=22677: Thu May 1 19:57:44 2025 read: IOPS=28, BW=448KiB/s (459kB/s)(5920KiB/13201msec) clat (usec): min=3902, max=66086, avg=13606.40, stdev=9979.96 lat (usec): min=3904, max=66087, avg=13607.63, stdev=9979.95 clat percentiles (usec): | 1.00th=[ 4146], 5.00th=[ 4817], 10.00th=[ 5538], 20.00th=[ 6587], | 30.00th=[ 7635], 40.00th=[ 8717], 50.00th=[10159], 60.00th=[12125], | 70.00th=[14877], 80.00th=[18220], 90.00th=[26346], 95.00th=[33817], | 99.00th=[55313], 99.50th=[57934], 99.90th=[66323], 99.95th=[66323], | 99.99th=[66323] bw ( KiB/s): min= 254, max= 766, per=24.72%, avg=440.92, stdev=137.69, samples=26 iops : min= 15, max= 47, avg=27.08, stdev= 8.70, samples=26 write: IOPS=30, BW=482KiB/s (494kB/s)(6368KiB/13201msec) clat (usec): min=6832, max=82006, avg=20436.61, stdev=10951.89 lat (usec): min=6833, max=82008, avg=20438.87, stdev=10951.79 clat percentiles (usec): | 1.00th=[ 8029], 5.00th=[ 9503], 10.00th=[10683], 20.00th=[12518], | 30.00th=[14091], 40.00th=[15795], 50.00th=[17433], 60.00th=[19268], | 70.00th=[22152], 80.00th=[26870], 90.00th=[33817], 95.00th=[42206], | 99.00th=[70779], 99.50th=[76022], 99.90th=[82314], 99.95th=[82314], | 99.99th=[82314] bw ( KiB/s): min= 256, max= 800, per=25.59%, avg=480.27, stdev=142.45, samples=26 iops : min= 16, max= 50, avg=29.54, stdev= 9.10, samples=26 lat (msec) : 4=0.39%, 10=26.69%, 20=45.70%, 50=24.87%, 100=2.34% cpu : usr=0.00%, sys=6.02%, ctx=950, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22678: Thu May 1 19:57:44 2025 read: IOPS=26, BW=428KiB/s (438kB/s)(5744KiB/13424msec) clat (usec): min=3446, max=53711, avg=13149.89, stdev=9198.73 lat (usec): min=3448, max=53712, avg=13151.18, stdev=9198.68 clat percentiles (usec): | 1.00th=[ 4146], 5.00th=[ 5145], 10.00th=[ 5800], 20.00th=[ 6915], | 30.00th=[ 7635], 40.00th=[ 8848], 50.00th=[10159], 60.00th=[11731], | 70.00th=[14222], 80.00th=[17171], 90.00th=[25297], 95.00th=[34341], | 99.00th=[50594], 99.50th=[52167], 99.90th=[53740], 99.95th=[53740], | 99.99th=[53740] bw ( KiB/s): min= 190, max= 736, per=23.76%, avg=423.85, stdev=158.10, samples=26 iops : min= 11, max= 46, avg=26.12, stdev= 9.98, samples=26 write: IOPS=30, BW=487KiB/s (499kB/s)(6544KiB/13424msec) clat (usec): min=4797, max=84853, avg=21123.34, stdev=11468.53 lat (usec): min=4799, max=84874, avg=21126.74, stdev=11469.55 clat percentiles (usec): | 1.00th=[ 6587], 5.00th=[ 9634], 10.00th=[10683], 20.00th=[12780], | 30.00th=[14091], 40.00th=[15795], 50.00th=[17171], 60.00th=[20579], | 70.00th=[24249], 80.00th=[28705], 90.00th=[35914], 95.00th=[43779], | 99.00th=[58983], 99.50th=[74974], 99.90th=[84411], 99.95th=[84411], | 99.99th=[84411] bw ( KiB/s): min= 255, max= 768, per=25.40%, avg=476.77, stdev=139.72, samples=26 iops : min= 15, max= 48, avg=29.42, stdev= 8.95, samples=26 lat (msec) : 4=0.39%, 10=26.17%, 20=45.18%, 50=26.30%, 100=1.95% cpu : usr=0.04%, sys=5.79%, ctx=944, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22679: Thu May 1 19:57:44 2025 read: IOPS=29, BW=467KiB/s (479kB/s)(6256KiB/13383msec) clat (usec): min=3182, max=96061, avg=13469.93, stdev=10136.36 lat (usec): min=3184, max=96063, avg=13471.27, stdev=10136.54 clat percentiles (usec): | 1.00th=[ 3425], 5.00th=[ 4883], 10.00th=[ 5604], 20.00th=[ 6521], | 30.00th=[ 7504], 40.00th=[ 8717], 50.00th=[10552], 60.00th=[12256], | 70.00th=[14615], 80.00th=[18220], 90.00th=[25035], 95.00th=[33817], | 99.00th=[53216], 99.50th=[59507], 99.90th=[95945], 99.95th=[95945], | 99.99th=[95945] bw ( KiB/s): min= 159, max= 928, per=26.12%, avg=466.00, stdev=187.92, samples=26 iops : min= 9, max= 58, avg=28.85, stdev=11.91, samples=26 write: IOPS=28, BW=451KiB/s (462kB/s)(6032KiB/13383msec) clat (usec): min=6473, max=95769, avg=21181.44, stdev=12094.20 lat (usec): min=6476, max=95771, avg=21185.05, stdev=12093.45 clat percentiles (usec): | 1.00th=[ 7111], 5.00th=[ 9241], 10.00th=[10552], 20.00th=[12649], | 30.00th=[13829], 40.00th=[15664], 50.00th=[17433], 60.00th=[20317], | 70.00th=[23462], 80.00th=[27657], 90.00th=[34866], 95.00th=[42730], | 99.00th=[70779], 99.50th=[77071], 99.90th=[95945], 99.95th=[95945], | 99.99th=[95945] bw ( KiB/s): min= 224, max= 640, per=23.38%, avg=438.92, stdev=128.16, samples=26 iops : min= 14, max= 40, avg=27.15, stdev= 8.01, samples=26 lat (msec) : 4=1.30%, 10=25.52%, 20=44.79%, 50=25.91%, 100=2.47% cpu : usr=0.08%, sys=5.94%, ctx=949, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22680: Thu May 1 19:57:44 2025 read: IOPS=28, BW=455KiB/s (466kB/s)(6032KiB/13247msec) clat (usec): min=3859, max=66567, avg=13789.31, stdev=10071.49 lat (usec): min=3860, max=66569, avg=13790.64, stdev=10071.37 clat percentiles (usec): | 1.00th=[ 4178], 5.00th=[ 4883], 10.00th=[ 5538], 20.00th=[ 6521], | 30.00th=[ 7635], 40.00th=[ 9110], 50.00th=[10683], 60.00th=[12649], | 70.00th=[14877], 80.00th=[17957], 90.00th=[27657], 95.00th=[35390], | 99.00th=[52691], 99.50th=[63177], 99.90th=[66323], 99.95th=[66323], | 99.99th=[66323] bw ( KiB/s): min= 286, max= 704, per=25.22%, avg=450.00, stdev=125.17, samples=26 iops : min= 17, max= 44, avg=27.88, stdev= 7.91, samples=26 write: IOPS=29, BW=472KiB/s (484kB/s)(6256KiB/13247msec) clat (usec): min=5740, max=96084, avg=20224.10, stdev=10755.33 lat (usec): min=5742, max=96085, avg=20227.42, stdev=10755.04 clat percentiles (usec): | 1.00th=[ 6849], 5.00th=[ 8586], 10.00th=[10028], 20.00th=[12518], | 30.00th=[14091], 40.00th=[15795], 50.00th=[17433], 60.00th=[19792], | 70.00th=[22414], 80.00th=[26870], 90.00th=[33817], 95.00th=[41157], | 99.00th=[62653], 99.50th=[69731], 99.90th=[95945], 99.95th=[95945], | 99.99th=[95945] bw ( KiB/s): min= 190, max= 800, per=24.96%, avg=468.50, stdev=155.08, samples=26 iops : min= 11, max= 50, avg=29.04, stdev= 9.83, samples=26 lat (msec) : 4=0.13%, 10=27.99%, 20=44.01%, 50=26.30%, 100=1.56% cpu : usr=0.00%, sys=6.02%, ctx=936, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1784KiB/s (1827kB/s), 428KiB/s-467KiB/s (438kB/s-479kB/s), io=23.4MiB (24.5MB), run=13201-13424msec WRITE: bw=1877KiB/s (1922kB/s), 451KiB/s-487KiB/s (462kB/s-499kB/s), io=24.6MiB (25.8MB), run=13201-13424msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=22699: Thu May 1 19:57:47 2025 read: IOPS=8, BW=8533KiB/s (8738kB/s)(7168KiB/840msec) clat (usec): min=506, max=67807, avg=32068.13, stdev=30586.68 lat (usec): min=507, max=67808, avg=32069.68, stdev=30586.52 clat percentiles (usec): | 1.00th=[ 506], 5.00th=[ 506], 10.00th=[ 506], 20.00th=[ 553], | 30.00th=[ 586], 40.00th=[ 586], 50.00th=[41681], 60.00th=[51119], | 70.00th=[51119], 80.00th=[62129], 90.00th=[67634], 95.00th=[67634], | 99.00th=[67634], 99.50th=[67634], 99.90th=[67634], 99.95th=[67634], | 99.99th=[67634] bw ( KiB/s): min=10199, max=10199, per=59.26%, avg=10199.00, stdev= 0.00, samples=1 iops : min= 9, max= 9, avg= 9.00, stdev= 0.00, samples=1 write: IOPS=5, BW=6088KiB/s (6234kB/s)(5120KiB/841msec) clat (msec): min=47, max=117, avg=92.15, stdev=28.90 lat (msec): min=47, max=118, avg=92.21, stdev=28.92 clat percentiles (msec): | 1.00th=[ 48], 5.00th=[ 48], 10.00th=[ 48], 20.00th=[ 48], | 30.00th=[ 80], 40.00th=[ 80], 50.00th=[ 106], 60.00th=[ 106], | 70.00th=[ 110], 80.00th=[ 110], 90.00th=[ 118], 95.00th=[ 118], | 99.00th=[ 118], 99.50th=[ 118], 99.90th=[ 118], 99.95th=[ 118], | 99.99th=[ 118] bw ( KiB/s): min= 2039, max= 2039, per=8.46%, avg=2039.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (usec) : 750=25.00% lat (msec) : 50=16.67%, 100=33.33%, 250=25.00% cpu : usr=0.00%, sys=27.50%, ctx=291, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22700: Thu May 1 19:57:47 2025 read: IOPS=1, BW=1721KiB/s (1762kB/s)(2048KiB/1190msec) clat (usec): min=44177, max=93726, avg=68951.96, stdev=35037.03 lat (usec): min=44179, max=93728, avg=68953.89, stdev=35036.67 clat percentiles (usec): | 1.00th=[44303], 5.00th=[44303], 10.00th=[44303], 20.00th=[44303], | 30.00th=[44303], 40.00th=[44303], 50.00th=[44303], 60.00th=[93848], | 70.00th=[93848], 80.00th=[93848], 90.00th=[93848], 95.00th=[93848], | 99.00th=[93848], 99.50th=[93848], 99.90th=[93848], 99.95th=[93848], | 99.99th=[93848] bw ( KiB/s): min= 1878, max= 2048, per=11.41%, avg=1963.00, stdev=120.21, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 write: IOPS=8, BW=8605KiB/s (8812kB/s)(10.0MiB/1190msec) clat (msec): min=40, max=157, avg=88.31, stdev=43.03 lat (msec): min=40, max=157, avg=88.42, stdev=43.03 clat percentiles (msec): | 1.00th=[ 41], 5.00th=[ 41], 10.00th=[ 41], 20.00th=[ 45], | 30.00th=[ 59], 40.00th=[ 62], 50.00th=[ 64], 60.00th=[ 73], | 70.00th=[ 131], 80.00th=[ 131], 90.00th=[ 142], 95.00th=[ 159], | 99.00th=[ 159], 99.50th=[ 159], 99.90th=[ 159], 99.95th=[ 159], | 99.99th=[ 159] bw ( KiB/s): min= 4096, max= 9394, per=27.99%, avg=6745.00, stdev=3746.25, samples=2 iops : min= 4, max= 9, avg= 6.50, stdev= 3.54, samples=2 lat (msec) : 50=25.00%, 100=41.67%, 250=33.33% cpu : usr=0.00%, sys=24.22%, ctx=144, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22703: Thu May 1 19:57:47 2025 read: IOPS=6, BW=6621KiB/s (6780kB/s)(6144KiB/928msec) clat (usec): min=559, max=224952, avg=73376.68, stdev=77479.57 lat (usec): min=560, max=224954, avg=73378.20, stdev=77479.78 clat percentiles (usec): | 1.00th=[ 562], 5.00th=[ 562], 10.00th=[ 562], 20.00th=[ 44303], | 30.00th=[ 44303], 40.00th=[ 51119], 50.00th=[ 51119], 60.00th=[ 56361], | 70.00th=[ 63177], 80.00th=[ 63177], 90.00th=[225444], 95.00th=[225444], | 99.00th=[225444], 99.50th=[225444], 99.90th=[225444], 99.95th=[225444], | 99.99th=[225444] bw ( KiB/s): min= 6119, max= 6119, per=35.55%, avg=6119.00, stdev= 0.00, samples=1 iops : min= 5, max= 5, avg= 5.00, stdev= 0.00, samples=1 write: IOPS=6, BW=6621KiB/s (6780kB/s)(6144KiB/928msec) clat (msec): min=34, max=201, avg=77.66, stdev=62.12 lat (msec): min=34, max=201, avg=77.72, stdev=62.10 clat percentiles (msec): | 1.00th=[ 35], 5.00th=[ 35], 10.00th=[ 35], 20.00th=[ 44], | 30.00th=[ 44], 40.00th=[ 55], 50.00th=[ 55], 60.00th=[ 66], | 70.00th=[ 66], 80.00th=[ 66], 90.00th=[ 203], 95.00th=[ 203], | 99.00th=[ 203], 99.50th=[ 203], 99.90th=[ 203], 99.95th=[ 203], | 99.99th=[ 203] bw ( KiB/s): min= 8159, max= 8159, per=33.86%, avg=8159.00, stdev= 0.00, samples=1 iops : min= 7, max= 7, avg= 7.00, stdev= 0.00, samples=1 lat (usec) : 750=8.33% lat (msec) : 50=25.00%, 100=50.00%, 250=16.67% cpu : usr=0.00%, sys=24.27%, ctx=498, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22704: Thu May 1 19:57:47 2025 read: IOPS=5, BW=5639KiB/s (5774kB/s)(5120KiB/908msec) clat (usec): min=677, max=71641, avg=30653.70, stdev=33626.29 lat (usec): min=677, max=71642, avg=30655.08, stdev=33626.51 clat percentiles (usec): | 1.00th=[ 676], 5.00th=[ 676], 10.00th=[ 676], 20.00th=[ 676], | 30.00th=[ 2540], 40.00th=[ 2540], 50.00th=[16712], 60.00th=[16712], | 70.00th=[61604], 80.00th=[61604], 90.00th=[71828], 95.00th=[71828], | 99.00th=[71828], 99.50th=[71828], 99.90th=[71828], 99.95th=[71828], | 99.99th=[71828] write: IOPS=7, BW=7894KiB/s (8084kB/s)(7168KiB/908msec) clat (msec): min=31, max=162, avg=85.06, stdev=48.87 lat (msec): min=31, max=162, avg=85.15, stdev=48.88 clat percentiles (msec): | 1.00th=[ 32], 5.00th=[ 32], 10.00th=[ 32], 20.00th=[ 36], | 30.00th=[ 51], 40.00th=[ 51], 50.00th=[ 89], 60.00th=[ 106], | 70.00th=[ 106], 80.00th=[ 122], 90.00th=[ 163], 95.00th=[ 163], | 99.00th=[ 163], 99.50th=[ 163], 99.90th=[ 163], 99.95th=[ 163], | 99.99th=[ 163] bw ( KiB/s): min= 4087, max= 4087, per=16.96%, avg=4087.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 lat (usec) : 750=8.33% lat (msec) : 4=8.33%, 20=8.33%, 50=16.67%, 100=33.33%, 250=25.00% cpu : usr=0.00%, sys=21.37%, ctx=376, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=16.8MiB/s (17.6MB/s), 1721KiB/s-8533KiB/s (1762kB/s-8738kB/s), io=20.0MiB (20.0MB), run=840-1190msec WRITE: bw=23.5MiB/s (24.7MB/s), 6088KiB/s-8605KiB/s (6234kB/s-8812kB/s), io=28.0MiB (29.4MB), run=841-1190msec rand-rw: (groupid=0, jobs=1): err= 0: pid=22697: Thu May 1 19:57:47 2025 read: IOPS=7, BW=7506KiB/s (7686kB/s)(7168KiB/955msec) clat (msec): min=11, max=130, avg=44.30, stdev=49.17 lat (msec): min=11, max=130, avg=44.31, stdev=49.17 clat percentiles (msec): | 1.00th=[ 12], 5.00th=[ 12], 10.00th=[ 12], 20.00th=[ 16], | 30.00th=[ 16], 40.00th=[ 16], 50.00th=[ 17], 60.00th=[ 22], | 70.00th=[ 22], 80.00th=[ 100], 90.00th=[ 131], 95.00th=[ 131], | 99.00th=[ 131], 99.50th=[ 131], 99.90th=[ 131], 99.95th=[ 131], | 99.99th=[ 131] bw ( KiB/s): min=10078, max=10078, per=58.51%, avg=10078.00, stdev= 0.00, samples=1 iops : min= 9, max= 9, avg= 9.00, stdev= 0.00, samples=1 write: IOPS=5, BW=5361KiB/s (5490kB/s)(5120KiB/955msec) clat (msec): min=94, max=155, avg=126.09, stdev=24.48 lat (msec): min=95, max=155, avg=126.17, stdev=24.48 clat percentiles (msec): | 1.00th=[ 95], 5.00th=[ 95], 10.00th=[ 95], 20.00th=[ 95], | 30.00th=[ 108], 40.00th=[ 108], 50.00th=[ 133], 60.00th=[ 133], | 70.00th=[ 140], 80.00th=[ 140], 90.00th=[ 157], 95.00th=[ 157], | 99.00th=[ 157], 99.50th=[ 157], 99.90th=[ 157], 99.95th=[ 157], | 99.99th=[ 157] bw ( KiB/s): min= 4031, max= 4031, per=16.72%, avg=4031.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 lat (msec) : 20=33.33%, 50=8.33%, 100=16.67%, 250=41.67% cpu : usr=0.00%, sys=6.07%, ctx=45, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22698: Thu May 1 19:57:47 2025 read: IOPS=1, BW=1830KiB/s (1874kB/s)(2048KiB/1119msec) clat (usec): min=46474, max=97094, avg=71784.81, stdev=35793.55 lat (usec): min=46475, max=97097, avg=71786.40, stdev=35794.77 clat percentiles (usec): | 1.00th=[46400], 5.00th=[46400], 10.00th=[46400], 20.00th=[46400], | 30.00th=[46400], 40.00th=[46400], 50.00th=[46400], 60.00th=[96994], | 70.00th=[96994], 80.00th=[96994], 90.00th=[96994], 95.00th=[96994], | 99.00th=[96994], 99.50th=[96994], 99.90th=[96994], 99.95th=[96994], | 99.99th=[96994] bw ( KiB/s): min= 2039, max= 2048, per=11.86%, avg=2043.50, stdev= 6.36, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 write: IOPS=8, BW=9151KiB/s (9371kB/s)(10.0MiB/1119msec) clat (msec): min=19, max=250, avg=90.12, stdev=78.84 lat (msec): min=19, max=251, avg=90.22, stdev=78.86 clat percentiles (msec): | 1.00th=[ 20], 5.00th=[ 20], 10.00th=[ 20], 20.00th=[ 24], | 30.00th=[ 28], 40.00th=[ 33], 50.00th=[ 45], 60.00th=[ 79], | 70.00th=[ 155], 80.00th=[ 155], 90.00th=[ 176], 95.00th=[ 251], | 99.00th=[ 251], 99.50th=[ 251], 99.90th=[ 251], 99.95th=[ 251], | 99.99th=[ 251] bw ( KiB/s): min= 6119, max=10240, per=33.92%, avg=8179.50, stdev=2913.99, samples=2 iops : min= 5, max= 10, avg= 7.50, stdev= 3.54, samples=2 lat (msec) : 20=8.33%, 50=41.67%, 100=25.00%, 250=16.67%, 500=8.33% cpu : usr=0.09%, sys=7.07%, ctx=43, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22701: Thu May 1 19:57:47 2025 read: IOPS=5, BW=5510KiB/s (5643kB/s)(6144KiB/1115msec) clat (msec): min=24, max=284, avg=84.57, stdev=99.98 lat (msec): min=24, max=284, avg=84.57, stdev=99.98 clat percentiles (msec): | 1.00th=[ 25], 5.00th=[ 25], 10.00th=[ 25], 20.00th=[ 30], | 30.00th=[ 30], 40.00th=[ 37], 50.00th=[ 37], 60.00th=[ 56], | 70.00th=[ 77], 80.00th=[ 77], 90.00th=[ 284], 95.00th=[ 284], | 99.00th=[ 284], 99.50th=[ 284], 99.90th=[ 284], 99.95th=[ 284], | 99.99th=[ 284] bw ( KiB/s): min= 4096, max= 4096, per=23.78%, avg=4096.00, stdev= 0.00, samples=2 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=2 write: IOPS=5, BW=5510KiB/s (5643kB/s)(6144KiB/1115msec) clat (msec): min=32, max=146, avg=85.34, stdev=46.61 lat (msec): min=32, max=146, avg=85.41, stdev=46.61 clat percentiles (msec): | 1.00th=[ 33], 5.00th=[ 33], 10.00th=[ 33], 20.00th=[ 45], | 30.00th=[ 45], 40.00th=[ 57], 50.00th=[ 57], 60.00th=[ 111], | 70.00th=[ 121], 80.00th=[ 121], 90.00th=[ 146], 95.00th=[ 146], | 99.00th=[ 146], 99.50th=[ 146], 99.90th=[ 146], 99.95th=[ 146], | 99.99th=[ 146] bw ( KiB/s): min= 4096, max= 6144, per=21.23%, avg=5120.00, stdev=1448.15, samples=2 iops : min= 4, max= 6, avg= 5.00, stdev= 1.41, samples=2 lat (msec) : 50=41.67%, 100=25.00%, 250=25.00%, 500=8.33% cpu : usr=0.00%, sys=6.19%, ctx=45, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22702: Thu May 1 19:57:47 2025 read: IOPS=4, BW=4306KiB/s (4409kB/s)(5120KiB/1189msec) clat (msec): min=30, max=165, avg=97.08, stdev=63.95 lat (msec): min=30, max=165, avg=97.08, stdev=63.95 clat percentiles (msec): | 1.00th=[ 31], 5.00th=[ 31], 10.00th=[ 31], 20.00th=[ 31], | 30.00th=[ 50], 40.00th=[ 50], 50.00th=[ 78], 60.00th=[ 78], | 70.00th=[ 165], 80.00th=[ 165], 90.00th=[ 165], 95.00th=[ 165], | 99.00th=[ 165], 99.50th=[ 165], 99.90th=[ 165], 99.95th=[ 165], | 99.99th=[ 165] bw ( KiB/s): min= 8192, max= 8192, per=47.56%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=5, BW=6029KiB/s (6173kB/s)(7168KiB/1189msec) clat (msec): min=34, max=132, avg=86.41, stdev=33.87 lat (msec): min=34, max=132, avg=86.51, stdev=33.90 clat percentiles (msec): | 1.00th=[ 34], 5.00th=[ 34], 10.00th=[ 34], 20.00th=[ 64], | 30.00th=[ 65], 40.00th=[ 65], 50.00th=[ 93], 60.00th=[ 107], | 70.00th=[ 107], 80.00th=[ 112], 90.00th=[ 132], 95.00th=[ 132], | 99.00th=[ 132], 99.50th=[ 132], 99.90th=[ 132], 99.95th=[ 132], | 99.99th=[ 132] bw ( KiB/s): min= 2048, max= 8192, per=21.23%, avg=5120.00, stdev=4344.46, samples=2 iops : min= 2, max= 8, avg= 5.00, stdev= 4.24, samples=2 lat (msec) : 50=25.00%, 100=33.33%, 250=41.67% cpu : usr=0.17%, sys=7.24%, ctx=52, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=16.8MiB/s (17.6MB/s), 1830KiB/s-7506KiB/s (1874kB/s-7686kB/s), io=20.0MiB (20.0MB), run=955-1189msec WRITE: bw=23.5MiB/s (24.7MB/s), 5361KiB/s-9151KiB/s (5490kB/s-9371kB/s), io=28.0MiB (29.4MB), run=955-1189msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=22711: Thu May 1 19:57:49 2025 read: IOPS=4, BW=18.5MiB/s (19.4MB/s)(12.0MiB/650msec) clat (msec): min=153, max=256, avg=202.57, stdev=51.78 lat (msec): min=153, max=256, avg=202.57, stdev=51.78 clat percentiles (msec): | 1.00th=[ 155], 5.00th=[ 155], 10.00th=[ 155], 20.00th=[ 155], | 30.00th=[ 155], 40.00th=[ 199], 50.00th=[ 199], 60.00th=[ 199], | 70.00th=[ 257], 80.00th=[ 257], 90.00th=[ 257], 95.00th=[ 257], | 99.00th=[ 257], 99.50th=[ 257], 99.90th=[ 257], 99.95th=[ 257], | 99.99th=[ 257] bw ( KiB/s): min=16384, max=16384, per=100.00%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.00%, sys=26.66%, ctx=642, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22712: Thu May 1 19:57:49 2025 read: IOPS=0, BW=3683KiB/s (3772kB/s)(4096KiB/1112msec) clat (nsec): min=192744k, max=192744k, avg=192743880.00, stdev= 0.00 lat (nsec): min=192747k, max=192747k, avg=192746620.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 192], 5.00th=[ 192], 10.00th=[ 192], 20.00th=[ 192], | 30.00th=[ 192], 40.00th=[ 192], 50.00th=[ 192], 60.00th=[ 192], | 70.00th=[ 192], 80.00th=[ 192], 90.00th=[ 192], 95.00th=[ 192], | 99.00th=[ 192], 99.50th=[ 192], 99.90th=[ 192], 99.95th=[ 192], | 99.99th=[ 192] bw ( KiB/s): min= 8192, max= 8192, per=55.60%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=1, BW=7367KiB/s (7544kB/s)(8192KiB/1112msec) clat (msec): min=433, max=465, avg=449.38, stdev=23.01 lat (msec): min=433, max=465, avg=449.62, stdev=22.92 clat percentiles (msec): | 1.00th=[ 435], 5.00th=[ 435], 10.00th=[ 435], 20.00th=[ 435], | 30.00th=[ 435], 40.00th=[ 435], 50.00th=[ 435], 60.00th=[ 468], | 70.00th=[ 468], 80.00th=[ 468], 90.00th=[ 468], 95.00th=[ 468], | 99.00th=[ 468], 99.50th=[ 468], 99.90th=[ 468], 99.95th=[ 468], | 99.99th=[ 468] lat (msec) : 250=33.33%, 500=66.67% cpu : usr=0.09%, sys=21.06%, ctx=343, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22714: Thu May 1 19:57:49 2025 write: IOPS=3, BW=13.1MiB/s (13.7MB/s)(12.0MiB/919msec) clat (msec): min=122, max=469, avg=284.09, stdev=174.96 lat (msec): min=122, max=470, avg=284.48, stdev=174.94 clat percentiles (msec): | 1.00th=[ 123], 5.00th=[ 123], 10.00th=[ 123], 20.00th=[ 123], | 30.00th=[ 123], 40.00th=[ 259], 50.00th=[ 259], 60.00th=[ 259], | 70.00th=[ 472], 80.00th=[ 472], 90.00th=[ 472], 95.00th=[ 472], | 99.00th=[ 472], 99.50th=[ 472], 99.90th=[ 472], 99.95th=[ 472], | 99.99th=[ 472] bw ( KiB/s): min=16384, max=16384, per=55.60%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=66.67% cpu : usr=0.00%, sys=22.11%, ctx=77, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22716: Thu May 1 19:57:49 2025 write: IOPS=3, BW=15.6MiB/s (16.3MB/s)(12.0MiB/770msec) clat (msec): min=155, max=339, avg=237.31, stdev=93.97 lat (msec): min=156, max=340, avg=237.67, stdev=93.87 clat percentiles (msec): | 1.00th=[ 157], 5.00th=[ 157], 10.00th=[ 157], 20.00th=[ 157], | 30.00th=[ 157], 40.00th=[ 218], 50.00th=[ 218], 60.00th=[ 218], | 70.00th=[ 342], 80.00th=[ 342], 90.00th=[ 342], 95.00th=[ 342], | 99.00th=[ 342], 99.50th=[ 342], 99.90th=[ 342], 99.95th=[ 342], | 99.99th=[ 342] bw ( KiB/s): min= 8192, max= 8192, per=27.80%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.00%, sys=42.26%, ctx=72, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=14.4MiB/s (15.1MB/s), 3683KiB/s-18.5MiB/s (3772kB/s-19.4MB/s), io=16.0MiB (16.8MB), run=650-1112msec WRITE: bw=28.8MiB/s (30.2MB/s), 7367KiB/s-15.6MiB/s (7544kB/s-16.3MB/s), io=32.0MiB (33.6MB), run=770-1112msec rand-rw: (groupid=0, jobs=1): err= 0: pid=22713: Thu May 1 19:57:50 2025 read: IOPS=2, BW=10.5MiB/s (11.0MB/s)(12.0MiB/1139msec) clat (msec): min=299, max=391, avg=356.22, stdev=49.31 lat (msec): min=299, max=391, avg=356.22, stdev=49.31 clat percentiles (msec): | 1.00th=[ 300], 5.00th=[ 300], 10.00th=[ 300], 20.00th=[ 300], | 30.00th=[ 300], 40.00th=[ 376], 50.00th=[ 376], 60.00th=[ 376], | 70.00th=[ 393], 80.00th=[ 393], 90.00th=[ 393], 95.00th=[ 393], | 99.00th=[ 393], 99.50th=[ 393], 99.90th=[ 393], 99.95th=[ 393], | 99.99th=[ 393] bw ( KiB/s): min= 8175, max= 8192, per=61.59%, avg=8183.50, stdev=12.02, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 500=100.00% cpu : usr=0.09%, sys=7.47%, ctx=23, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22715: Thu May 1 19:57:50 2025 read: IOPS=0, BW=3322KiB/s (3402kB/s)(4096KiB/1233msec) clat (nsec): min=389786k, max=389786k, avg=389786240.00, stdev= 0.00 lat (nsec): min=389789k, max=389789k, avg=389788668.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 388], 5.00th=[ 388], 10.00th=[ 388], 20.00th=[ 388], | 30.00th=[ 388], 40.00th=[ 388], 50.00th=[ 388], 60.00th=[ 388], | 70.00th=[ 388], 80.00th=[ 388], 90.00th=[ 388], 95.00th=[ 388], | 99.00th=[ 388], 99.50th=[ 388], 99.90th=[ 388], 99.95th=[ 388], | 99.99th=[ 388] bw ( KiB/s): min= 8192, max= 8192, per=61.65%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=1, BW=6644KiB/s (6803kB/s)(8192KiB/1233msec) clat (msec): min=176, max=536, avg=356.59, stdev=254.58 lat (msec): min=176, max=536, avg=356.85, stdev=254.59 clat percentiles (msec): | 1.00th=[ 178], 5.00th=[ 178], 10.00th=[ 178], 20.00th=[ 178], | 30.00th=[ 178], 40.00th=[ 178], 50.00th=[ 178], 60.00th=[ 535], | 70.00th=[ 535], 80.00th=[ 535], 90.00th=[ 535], 95.00th=[ 535], | 99.00th=[ 535], 99.50th=[ 535], 99.90th=[ 535], 99.95th=[ 535], | 99.99th=[ 535] lat (msec) : 250=33.33%, 500=33.33%, 750=33.33% cpu : usr=0.08%, sys=9.98%, ctx=100, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22717: Thu May 1 19:57:50 2025 write: IOPS=2, BW=10.4MiB/s (10.9MB/s)(12.0MiB/1158msec) clat (msec): min=268, max=442, avg=371.06, stdev=90.81 lat (msec): min=269, max=443, avg=371.40, stdev=90.83 clat percentiles (msec): | 1.00th=[ 271], 5.00th=[ 271], 10.00th=[ 271], 20.00th=[ 271], | 30.00th=[ 271], 40.00th=[ 401], 50.00th=[ 401], 60.00th=[ 401], | 70.00th=[ 443], 80.00th=[ 443], 90.00th=[ 443], 95.00th=[ 443], | 99.00th=[ 443], 99.50th=[ 443], 99.90th=[ 443], 99.95th=[ 443], | 99.99th=[ 443] bw ( KiB/s): min= 8175, max= 8192, per=30.79%, avg=8183.50, stdev=12.02, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 500=100.00% cpu : usr=0.00%, sys=2.07%, ctx=20, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22718: Thu May 1 19:57:50 2025 write: IOPS=2, BW=10.9MiB/s (11.4MB/s)(12.0MiB/1101msec) clat (msec): min=242, max=400, avg=329.13, stdev=79.82 lat (msec): min=243, max=400, avg=329.82, stdev=80.00 clat percentiles (msec): | 1.00th=[ 243], 5.00th=[ 243], 10.00th=[ 243], 20.00th=[ 243], | 30.00th=[ 243], 40.00th=[ 347], 50.00th=[ 347], 60.00th=[ 347], | 70.00th=[ 401], 80.00th=[ 401], 90.00th=[ 401], 95.00th=[ 401], | 99.00th=[ 401], 99.50th=[ 401], 99.90th=[ 401], 99.95th=[ 401], | 99.99th=[ 401] bw ( KiB/s): min= 8126, max= 8143, per=30.61%, avg=8134.50, stdev=12.02, samples=2 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=2 lat (msec) : 250=33.33%, 500=66.67% cpu : usr=0.00%, sys=6.91%, ctx=60, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=12.0MiB/s (13.6MB/s), 3322KiB/s-10.5MiB/s (3402kB/s-11.0MB/s), io=16.0MiB (16.8MB), run=1139-1233msec WRITE: bw=25.0MiB/s (27.2MB/s), 6644KiB/s-10.9MiB/s (6803kB/s-11.4MB/s), io=32.0MiB (33.6MB), run=1101-1233msec