== sanity test 398b: DIO and buffer IO race ============== 02:33:28 (1746167608) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 2.29913 s, 21.9 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=2688: Fri May 2 02:34:23 2025 read: IOPS=30, BW=121KiB/s (124kB/s)(5992KiB/49493msec) clat (usec): min=71, max=221685, avg=14499.53, stdev=14702.95 lat (usec): min=72, max=221686, avg=14505.14, stdev=14702.85 clat percentiles (usec): | 1.00th=[ 79], 5.00th=[ 94], 10.00th=[ 105], 20.00th=[ 133], | 30.00th=[ 6915], 40.00th=[ 9634], 50.00th=[ 12125], 60.00th=[ 15008], | 70.00th=[ 19006], 80.00th=[ 23725], 90.00th=[ 31589], 95.00th=[ 39584], | 99.00th=[ 55313], 99.50th=[ 67634], 99.90th=[135267], 99.95th=[221250], | 99.99th=[221250] bw ( KiB/s): min= 47, max= 223, per=25.06%, avg=119.79, stdev=37.66, samples=98 iops : min= 11, max= 55, avg=29.58, stdev= 9.49, samples=98 write: IOPS=31, BW=127KiB/s (130kB/s)(6296KiB/49493msec) clat (usec): min=596, max=306184, avg=17493.81, stdev=19521.21 lat (usec): min=597, max=306185, avg=17496.58, stdev=19521.19 clat percentiles (usec): | 1.00th=[ 676], 5.00th=[ 734], 10.00th=[ 783], 20.00th=[ 971], | 30.00th=[ 3130], 40.00th=[ 7111], 50.00th=[ 12911], 60.00th=[ 17695], | 70.00th=[ 22938], 80.00th=[ 30540], 90.00th=[ 43254], 95.00th=[ 51119], | 99.00th=[ 73925], 99.50th=[ 89654], 99.90th=[141558], 99.95th=[308282], | 99.99th=[308282] bw ( KiB/s): min= 39, max= 240, per=25.03%, avg=125.40, stdev=40.26, samples=98 iops : min= 9, max= 60, avg=30.99, stdev=10.13, samples=98 lat (usec) : 100=3.74%, 250=8.04%, 500=0.36%, 750=3.55%, 1000=7.36% lat (msec) : 2=3.61%, 4=3.78%, 10=12.40%, 20=25.55%, 50=27.86% lat (msec) : 100=3.39%, 250=0.33%, 500=0.03% cpu : usr=0.11%, sys=9.68%, ctx=4627, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2689: Fri May 2 02:34:23 2025 read: IOPS=29, BW=119KiB/s (122kB/s)(5940KiB/49986msec) clat (usec): min=68, max=324217, avg=14421.56, stdev=16551.55 lat (usec): min=68, max=324217, avg=14427.07, stdev=16556.94 clat percentiles (usec): | 1.00th=[ 79], 5.00th=[ 93], 10.00th=[ 104], 20.00th=[ 129], | 30.00th=[ 6587], 40.00th=[ 9634], 50.00th=[ 11863], 60.00th=[ 14615], | 70.00th=[ 18220], 80.00th=[ 23200], 90.00th=[ 31065], 95.00th=[ 40109], | 99.00th=[ 58983], 99.50th=[ 70779], 99.90th=[193987], 99.95th=[325059], | 99.99th=[325059] bw ( KiB/s): min= 32, max= 285, per=24.58%, avg=117.51, stdev=41.78, samples=99 iops : min= 8, max= 71, avg=28.96, stdev=10.49, samples=99 write: IOPS=31, BW=127KiB/s (130kB/s)(6348KiB/49986msec) clat (usec): min=556, max=180735, avg=17854.90, stdev=19709.39 lat (usec): min=558, max=180737, avg=17857.45, stdev=19710.25 clat percentiles (usec): | 1.00th=[ 668], 5.00th=[ 742], 10.00th=[ 791], 20.00th=[ 996], | 30.00th=[ 2868], 40.00th=[ 7242], 50.00th=[ 13173], 60.00th=[ 18482], | 70.00th=[ 23200], 80.00th=[ 30540], 90.00th=[ 42206], 95.00th=[ 53740], | 99.00th=[ 90702], 99.50th=[102237], 99.90th=[168821], 99.95th=[181404], | 99.99th=[181404] bw ( KiB/s): min= 40, max= 231, per=24.89%, avg=124.71, stdev=39.15, samples=99 iops : min= 10, max= 57, avg=30.76, stdev= 9.78, samples=99 lat (usec) : 100=3.97%, 250=8.20%, 500=0.20%, 750=3.39%, 1000=7.19% lat (msec) : 2=3.12%, 4=4.46%, 10=12.14%, 20=26.33%, 50=26.99% lat (msec) : 100=3.55%, 250=0.42%, 500=0.03% cpu : usr=0.11%, sys=9.75%, ctx=4638, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2690: Fri May 2 02:34:23 2025 read: IOPS=30, BW=120KiB/s (123kB/s)(6028KiB/50144msec) clat (usec): min=63, max=322027, avg=14738.33, stdev=16887.04 lat (usec): min=63, max=322029, avg=14742.68, stdev=16885.67 clat percentiles (usec): | 1.00th=[ 81], 5.00th=[ 95], 10.00th=[ 110], 20.00th=[ 135], | 30.00th=[ 7242], 40.00th=[ 9765], 50.00th=[ 11994], 60.00th=[ 14746], | 70.00th=[ 18220], 80.00th=[ 23200], 90.00th=[ 30802], 95.00th=[ 39060], | 99.00th=[ 63177], 99.50th=[ 96994], 99.90th=[166724], 99.95th=[320865], | 99.99th=[320865] bw ( KiB/s): min= 24, max= 210, per=24.90%, avg=119.01, stdev=41.88, samples=100 iops : min= 6, max= 52, avg=29.40, stdev=10.55, samples=100 write: IOPS=31, BW=125KiB/s (128kB/s)(6260KiB/50144msec) clat (usec): min=550, max=180131, avg=17694.75, stdev=18496.26 lat (usec): min=551, max=180132, avg=17698.25, stdev=18497.07 clat percentiles (usec): | 1.00th=[ 685], 5.00th=[ 734], 10.00th=[ 791], 20.00th=[ 1029], | 30.00th=[ 2802], 40.00th=[ 8586], 50.00th=[ 13829], 60.00th=[ 18220], | 70.00th=[ 23462], 80.00th=[ 30802], 90.00th=[ 42730], 95.00th=[ 51119], | 99.00th=[ 77071], 99.50th=[ 89654], 99.90th=[162530], 99.95th=[179307], | 99.99th=[179307] bw ( KiB/s): min= 48, max= 217, per=24.62%, avg=123.35, stdev=36.43, samples=100 iops : min= 12, max= 54, avg=30.49, stdev= 9.20, samples=100 lat (usec) : 100=3.29%, 250=8.50%, 500=0.23%, 750=3.39%, 1000=6.80% lat (msec) : 2=3.71%, 4=3.91%, 10=12.14%, 20=26.50%, 50=27.67% lat (msec) : 100=3.45%, 250=0.39%, 500=0.03% cpu : usr=0.09%, sys=9.75%, ctx=4807, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2691: Fri May 2 02:34:23 2025 read: IOPS=30, BW=123KiB/s (126kB/s)(6052KiB/49016msec) clat (usec): min=64, max=128635, avg=14266.52, stdev=13256.73 lat (usec): min=65, max=128636, avg=14267.78, stdev=13256.81 clat percentiles (usec): | 1.00th=[ 82], 5.00th=[ 95], 10.00th=[ 106], 20.00th=[ 133], | 30.00th=[ 7177], 40.00th=[ 9896], 50.00th=[ 12387], 60.00th=[ 15139], | 70.00th=[ 18482], 80.00th=[ 22676], 90.00th=[ 31589], 95.00th=[ 39060], | 99.00th=[ 54264], 99.50th=[ 69731], 99.90th=[ 83362], 99.95th=[128451], | 99.99th=[128451] bw ( KiB/s): min= 15, max= 216, per=25.71%, avg=122.88, stdev=36.64, samples=98 iops : min= 3, max= 54, avg=30.31, stdev= 9.24, samples=98 write: IOPS=31, BW=127KiB/s (130kB/s)(6236KiB/49016msec) clat (usec): min=548, max=280914, avg=17449.65, stdev=19886.82 lat (usec): min=549, max=280915, avg=17453.25, stdev=19885.91 clat percentiles (usec): | 1.00th=[ 676], 5.00th=[ 742], 10.00th=[ 783], 20.00th=[ 1004], | 30.00th=[ 2769], 40.00th=[ 6456], 50.00th=[ 12518], 60.00th=[ 17695], | 70.00th=[ 22676], 80.00th=[ 30540], 90.00th=[ 41681], 95.00th=[ 52167], | 99.00th=[ 85459], 99.50th=[ 99091], 99.90th=[166724], 99.95th=[278922], | 99.99th=[278922] bw ( KiB/s): min= 48, max= 255, per=25.26%, avg=126.55, stdev=39.36, samples=98 iops : min= 12, max= 63, avg=31.22, stdev= 9.86, samples=98 lat (usec) : 100=3.74%, 250=7.81%, 500=0.36%, 750=3.26%, 1000=7.16% lat (msec) : 2=3.29%, 4=4.69%, 10=12.70%, 20=26.95%, 50=26.43% lat (msec) : 100=3.35%, 250=0.23%, 500=0.03% cpu : usr=0.11%, sys=9.57%, ctx=4638, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=479KiB/s (490kB/s), 119KiB/s-123KiB/s (122kB/s-126kB/s), io=23.4MiB (24.6MB), run=49016-50144msec WRITE: bw=501KiB/s (513kB/s), 125KiB/s-127KiB/s (128kB/s-130kB/s), io=24.6MiB (25.7MB), run=49016-50144msec rand-rw: (groupid=0, jobs=1): err= 0: pid=2693: Fri May 2 02:36:18 2025 read: IOPS=9, BW=36.5KiB/s (37.3kB/s)(5992KiB/164353msec) clat (msec): min=3, max=312, avg=13.69, stdev=14.75 lat (msec): min=3, max=312, avg=13.69, stdev=14.75 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 5], 10.00th=[ 6], 20.00th=[ 7], | 30.00th=[ 8], 40.00th=[ 9], 50.00th=[ 10], 60.00th=[ 12], | 70.00th=[ 14], 80.00th=[ 18], 90.00th=[ 26], 95.00th=[ 38], | 99.00th=[ 64], 99.50th=[ 70], 99.90th=[ 184], 99.95th=[ 313], | 99.99th=[ 313] bw ( KiB/s): min= 7, max= 152, per=26.62%, avg=38.60, stdev=23.85, samples=307 iops : min= 1, max= 38, avg= 9.36, stdev= 5.97, samples=307 write: IOPS=9, BW=38.3KiB/s (39.2kB/s)(6296KiB/164353msec) clat (msec): min=29, max=332, avg=91.34, stdev=42.29 lat (msec): min=29, max=332, avg=91.34, stdev=42.29 clat percentiles (msec): | 1.00th=[ 38], 5.00th=[ 45], 10.00th=[ 52], 20.00th=[ 60], | 30.00th=[ 67], 40.00th=[ 74], 50.00th=[ 82], 60.00th=[ 89], | 70.00th=[ 100], 80.00th=[ 114], 90.00th=[ 148], 95.00th=[ 180], | 99.00th=[ 239], 99.50th=[ 266], 99.90th=[ 326], 99.95th=[ 334], | 99.99th=[ 334] bw ( KiB/s): min= 7, max= 64, per=24.93%, avg=37.89, stdev=13.75, samples=328 iops : min= 1, max= 16, avg= 9.18, stdev= 3.53, samples=328 lat (msec) : 4=0.33%, 10=25.75%, 20=14.94%, 50=11.39%, 100=32.49% lat (msec) : 250=14.71%, 500=0.39% cpu : usr=0.05%, sys=2.21%, ctx=3583, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2694: Fri May 2 02:36:18 2025 read: IOPS=9, BW=36.1KiB/s (36.9kB/s)(5940KiB/164680msec) clat (msec): min=2, max=342, avg=14.45, stdev=17.08 lat (msec): min=2, max=342, avg=14.45, stdev=17.08 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 6], 10.00th=[ 6], 20.00th=[ 7], | 30.00th=[ 8], 40.00th=[ 9], 50.00th=[ 10], 60.00th=[ 12], | 70.00th=[ 14], 80.00th=[ 18], 90.00th=[ 28], 95.00th=[ 41], | 99.00th=[ 68], 99.50th=[ 87], 99.90th=[ 226], 99.95th=[ 342], | 99.99th=[ 342] bw ( KiB/s): min= 7, max= 119, per=26.08%, avg=37.82, stdev=23.38, samples=310 iops : min= 1, max= 29, avg= 9.15, stdev= 5.86, samples=310 write: IOPS=9, BW=38.5KiB/s (39.5kB/s)(6348KiB/164680msec) clat (msec): min=30, max=425, avg=90.20, stdev=42.64 lat (msec): min=30, max=425, avg=90.20, stdev=42.64 clat percentiles (msec): | 1.00th=[ 39], 5.00th=[ 46], 10.00th=[ 51], 20.00th=[ 60], | 30.00th=[ 66], 40.00th=[ 73], 50.00th=[ 81], 60.00th=[ 87], | 70.00th=[ 97], 80.00th=[ 112], 90.00th=[ 146], 95.00th=[ 180], | 99.00th=[ 247], 99.50th=[ 271], 99.90th=[ 351], 99.95th=[ 426], | 99.99th=[ 426] bw ( KiB/s): min= 7, max= 72, per=25.05%, avg=38.07, stdev=14.08, samples=329 iops : min= 1, max= 18, avg= 9.21, stdev= 3.56, samples=329 lat (msec) : 4=0.33%, 10=25.20%, 20=14.49%, 50=11.78%, 100=33.40% lat (msec) : 250=14.29%, 500=0.52% cpu : usr=0.08%, sys=2.18%, ctx=3581, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2695: Fri May 2 02:36:18 2025 read: IOPS=9, BW=37.0KiB/s (37.9kB/s)(6028KiB/162794msec) clat (msec): min=3, max=194, avg=13.98, stdev=12.53 lat (msec): min=3, max=194, avg=13.99, stdev=12.53 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 6], 10.00th=[ 6], 20.00th=[ 7], | 30.00th=[ 8], 40.00th=[ 9], 50.00th=[ 10], 60.00th=[ 12], | 70.00th=[ 14], 80.00th=[ 18], 90.00th=[ 28], 95.00th=[ 39], | 99.00th=[ 64], 99.50th=[ 73], 99.90th=[ 108], 99.95th=[ 194], | 99.99th=[ 194] bw ( KiB/s): min= 7, max= 120, per=25.96%, avg=37.64, stdev=23.72, samples=316 iops : min= 1, max= 30, avg= 9.11, stdev= 6.00, samples=316 write: IOPS=9, BW=38.5KiB/s (39.4kB/s)(6260KiB/162794msec) clat (msec): min=31, max=543, avg=90.50, stdev=42.70 lat (msec): min=31, max=543, avg=90.50, stdev=42.70 clat percentiles (msec): | 1.00th=[ 41], 5.00th=[ 47], 10.00th=[ 52], 20.00th=[ 59], | 30.00th=[ 66], 40.00th=[ 73], 50.00th=[ 80], 60.00th=[ 88], | 70.00th=[ 99], 80.00th=[ 114], 90.00th=[ 140], 95.00th=[ 180], | 99.00th=[ 239], 99.50th=[ 271], 99.90th=[ 326], 99.95th=[ 542], | 99.99th=[ 542] bw ( KiB/s): min= 7, max= 71, per=25.08%, avg=38.11, stdev=13.64, samples=324 iops : min= 1, max= 17, avg= 9.23, stdev= 3.48, samples=324 lat (msec) : 4=0.26%, 10=25.68%, 20=14.81%, 50=11.36%, 100=33.37% lat (msec) : 250=14.13%, 500=0.36%, 750=0.03% cpu : usr=0.06%, sys=2.24%, ctx=3592, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2696: Fri May 2 02:36:18 2025 read: IOPS=9, BW=37.0KiB/s (37.9kB/s)(6052KiB/163502msec) clat (msec): min=2, max=209, avg=14.00, stdev=13.79 lat (msec): min=2, max=209, avg=14.00, stdev=13.79 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 5], 10.00th=[ 6], 20.00th=[ 7], | 30.00th=[ 8], 40.00th=[ 9], 50.00th=[ 10], 60.00th=[ 11], | 70.00th=[ 14], 80.00th=[ 18], 90.00th=[ 29], 95.00th=[ 40], | 99.00th=[ 65], 99.50th=[ 75], 99.90th=[ 190], 99.95th=[ 209], | 99.99th=[ 209] bw ( KiB/s): min= 7, max= 120, per=26.83%, avg=38.91, stdev=23.32, samples=306 iops : min= 1, max= 30, avg= 9.34, stdev= 5.84, samples=306 write: IOPS=9, BW=38.1KiB/s (39.1kB/s)(6236KiB/163502msec) clat (msec): min=31, max=445, avg=91.23, stdev=42.95 lat (msec): min=31, max=445, avg=91.24, stdev=42.95 clat percentiles (msec): | 1.00th=[ 39], 5.00th=[ 46], 10.00th=[ 52], 20.00th=[ 60], | 30.00th=[ 68], 40.00th=[ 73], 50.00th=[ 80], 60.00th=[ 89], | 70.00th=[ 97], 80.00th=[ 114], 90.00th=[ 148], 95.00th=[ 182], | 99.00th=[ 251], 99.50th=[ 271], 99.90th=[ 305], 99.95th=[ 447], | 99.99th=[ 447] bw ( KiB/s): min= 7, max= 71, per=24.71%, avg=37.56, stdev=13.85, samples=326 iops : min= 1, max= 17, avg= 9.00, stdev= 3.51, samples=326 lat (msec) : 4=0.46%, 10=26.24%, 20=14.55%, 50=11.33%, 100=32.68% lat (msec) : 250=14.23%, 500=0.52% cpu : usr=0.04%, sys=2.25%, ctx=3603, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=146KiB/s (149kB/s), 36.1KiB/s-37.0KiB/s (36.9kB/s-37.9kB/s), io=23.4MiB (24.6MB), run=162794-164680msec WRITE: bw=153KiB/s (156kB/s), 38.1KiB/s-38.5KiB/s (39.1kB/s-39.5kB/s), io=24.6MiB (25.7MB), run=162794-164680msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=2837: Fri May 2 02:36:28 2025 read: IOPS=43, BW=701KiB/s (717kB/s)(5920KiB/8450msec) clat (usec): min=69, max=82915, avg=10371.97, stdev=10310.30 lat (usec): min=69, max=82917, avg=10373.74, stdev=10310.35 clat percentiles (usec): | 1.00th=[ 82], 5.00th=[ 94], 10.00th=[ 100], 20.00th=[ 114], | 30.00th=[ 133], 40.00th=[ 7177], 50.00th=[10028], 60.00th=[11863], | 70.00th=[15139], 80.00th=[18220], 90.00th=[22938], 95.00th=[27919], | 99.00th=[39060], 99.50th=[42730], 99.90th=[83362], 99.95th=[83362], | 99.99th=[83362] bw ( KiB/s): min= 448, max= 1248, per=25.35%, avg=696.88, stdev=234.03, samples=16 iops : min= 28, max= 78, avg=43.13, stdev=14.53, samples=16 write: IOPS=47, BW=754KiB/s (772kB/s)(6368KiB/8450msec) clat (usec): min=928, max=90407, avg=11485.66, stdev=13288.65 lat (usec): min=928, max=90408, avg=11494.31, stdev=13284.88 clat percentiles (usec): | 1.00th=[ 955], 5.00th=[ 1057], 10.00th=[ 1139], 20.00th=[ 1303], | 30.00th=[ 1958], 40.00th=[ 3359], 50.00th=[ 5604], 60.00th=[10421], | 70.00th=[14877], 80.00th=[21365], 90.00th=[28705], 95.00th=[35914], | 99.00th=[69731], 99.50th=[84411], 99.90th=[90702], 99.95th=[90702], | 99.99th=[90702] bw ( KiB/s): min= 448, max= 1149, per=25.75%, avg=744.75, stdev=204.38, samples=16 iops : min= 28, max= 71, avg=46.13, stdev=12.66, samples=16 lat (usec) : 100=4.69%, 250=12.24%, 500=0.13%, 750=0.39%, 1000=1.69% lat (msec) : 2=14.58%, 4=6.77%, 10=13.80%, 20=26.69%, 50=18.10% lat (msec) : 100=0.91% cpu : usr=0.08%, sys=12.79%, ctx=1062, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2838: Fri May 2 02:36:28 2025 read: IOPS=41, BW=665KiB/s (681kB/s)(5744KiB/8643msec) clat (usec): min=74, max=102910, avg=10831.22, stdev=11357.01 lat (usec): min=75, max=102910, avg=10833.96, stdev=11355.47 clat percentiles (usec): | 1.00th=[ 87], 5.00th=[ 94], 10.00th=[ 103], 20.00th=[ 123], | 30.00th=[ 167], 40.00th=[ 7046], 50.00th=[ 9896], 60.00th=[ 12125], | 70.00th=[ 15008], 80.00th=[ 19268], 90.00th=[ 23725], 95.00th=[ 31851], | 99.00th=[ 45876], 99.50th=[ 52691], 99.90th=[103285], 99.95th=[103285], | 99.99th=[103285] bw ( KiB/s): min= 191, max= 1024, per=23.92%, avg=657.59, stdev=207.35, samples=17 iops : min= 11, max= 64, avg=40.82, stdev=13.15, samples=17 write: IOPS=47, BW=757KiB/s (775kB/s)(6544KiB/8643msec) clat (usec): min=855, max=67130, avg=11294.84, stdev=12495.12 lat (usec): min=856, max=67131, avg=11296.87, stdev=12495.04 clat percentiles (usec): | 1.00th=[ 930], 5.00th=[ 1029], 10.00th=[ 1106], 20.00th=[ 1287], | 30.00th=[ 2147], 40.00th=[ 3195], 50.00th=[ 5735], 60.00th=[10683], | 70.00th=[14877], 80.00th=[21365], 90.00th=[28181], 95.00th=[35914], | 99.00th=[52691], 99.50th=[61080], 99.90th=[67634], 99.95th=[67634], | 99.99th=[67634] bw ( KiB/s): min= 383, max= 1088, per=25.46%, avg=736.41, stdev=192.22, samples=17 iops : min= 23, max= 68, avg=45.71, stdev=12.23, samples=17 lat (usec) : 100=3.91%, 250=10.42%, 500=1.04%, 750=0.52%, 1000=2.21% lat (msec) : 2=13.80%, 4=8.20%, 10=15.62%, 20=24.74%, 50=18.36% lat (msec) : 100=1.04%, 250=0.13% cpu : usr=0.10%, sys=13.06%, ctx=1109, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2839: Fri May 2 02:36:28 2025 read: IOPS=45, BW=723KiB/s (740kB/s)(6256KiB/8654msec) clat (usec): min=78, max=70940, avg=10606.73, stdev=11378.03 lat (usec): min=78, max=70941, avg=10607.89, stdev=11377.92 clat percentiles (usec): | 1.00th=[ 84], 5.00th=[ 90], 10.00th=[ 99], 20.00th=[ 117], | 30.00th=[ 133], 40.00th=[ 6063], 50.00th=[ 8979], 60.00th=[11469], | 70.00th=[15270], 80.00th=[18482], 90.00th=[25822], 95.00th=[32113], | 99.00th=[45351], 99.50th=[61604], 99.90th=[70779], 99.95th=[70779], | 99.99th=[70779] bw ( KiB/s): min= 255, max= 1206, per=26.09%, avg=717.29, stdev=224.06, samples=17 iops : min= 15, max= 75, avg=44.35, stdev=14.12, samples=17 write: IOPS=43, BW=697KiB/s (714kB/s)(6032KiB/8654msec) clat (usec): min=840, max=83689, avg=11674.36, stdev=13573.98 lat (usec): min=842, max=83691, avg=11684.36, stdev=13570.53 clat percentiles (usec): | 1.00th=[ 938], 5.00th=[ 1012], 10.00th=[ 1090], 20.00th=[ 1270], | 30.00th=[ 1975], 40.00th=[ 3359], 50.00th=[ 5538], 60.00th=[ 9110], | 70.00th=[14484], 80.00th=[22414], 90.00th=[32375], 95.00th=[39060], | 99.00th=[53216], 99.50th=[80217], 99.90th=[83362], 99.95th=[83362], | 99.99th=[83362] bw ( KiB/s): min= 319, max= 960, per=23.58%, avg=681.82, stdev=174.44, samples=17 iops : min= 19, max= 60, avg=42.12, stdev=11.17, samples=17 lat (usec) : 100=5.21%, 250=13.93%, 500=0.39%, 1000=2.34% lat (msec) : 2=12.89%, 4=6.25%, 10=16.67%, 20=21.74%, 50=19.53% lat (msec) : 100=1.04% cpu : usr=0.13%, sys=12.65%, ctx=1073, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2840: Fri May 2 02:36:28 2025 read: IOPS=43, BW=692KiB/s (709kB/s)(6032KiB/8711msec) clat (usec): min=70, max=64388, avg=10607.96, stdev=10739.79 lat (usec): min=71, max=64388, avg=10609.07, stdev=10739.68 clat percentiles (usec): | 1.00th=[ 83], 5.00th=[ 93], 10.00th=[ 101], 20.00th=[ 117], | 30.00th=[ 137], 40.00th=[ 5080], 50.00th=[ 9241], 60.00th=[12780], | 70.00th=[15926], 80.00th=[19792], 90.00th=[25035], 95.00th=[29230], | 99.00th=[42730], 99.50th=[50594], 99.90th=[64226], 99.95th=[64226], | 99.99th=[64226] bw ( KiB/s): min= 224, max= 960, per=24.38%, avg=670.29, stdev=196.05, samples=17 iops : min= 14, max= 60, avg=41.41, stdev=12.27, samples=17 write: IOPS=44, BW=718KiB/s (735kB/s)(6256KiB/8711msec) clat (usec): min=878, max=84976, avg=11683.86, stdev=13712.36 lat (usec): min=879, max=84978, avg=11685.48, stdev=13712.42 clat percentiles (usec): | 1.00th=[ 947], 5.00th=[ 1012], 10.00th=[ 1090], 20.00th=[ 1270], | 30.00th=[ 1942], 40.00th=[ 3326], 50.00th=[ 5473], 60.00th=[ 8979], | 70.00th=[13566], 80.00th=[21627], 90.00th=[32113], 95.00th=[39060], | 99.00th=[60031], 99.50th=[69731], 99.90th=[85459], 99.95th=[85459], | 99.99th=[85459] bw ( KiB/s): min= 352, max= 1021, per=24.21%, avg=700.29, stdev=166.08, samples=17 iops : min= 22, max= 63, avg=43.29, stdev=10.23, samples=17 lat (usec) : 100=4.69%, 250=12.63%, 500=1.04%, 750=0.26%, 1000=1.95% lat (msec) : 2=13.93%, 4=8.07%, 10=14.45%, 20=21.61%, 50=20.18% lat (msec) : 100=1.17% cpu : usr=0.18%, sys=12.41%, ctx=1102, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2750KiB/s (2816kB/s), 665KiB/s-723KiB/s (681kB/s-740kB/s), io=23.4MiB (24.5MB), run=8450-8711msec WRITE: bw=2893KiB/s (2962kB/s), 697KiB/s-757KiB/s (714kB/s-775kB/s), io=24.6MiB (25.8MB), run=8450-8711msec rand-rw: (groupid=0, jobs=1): err= 0: pid=2842: Fri May 2 02:36:58 2025 read: IOPS=9, BW=160KiB/s (164kB/s)(5920KiB/37027msec) clat (msec): min=3, max=121, avg=12.85, stdev=14.45 lat (msec): min=3, max=121, avg=12.85, stdev=14.44 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 6], 10.00th=[ 6], 20.00th=[ 7], | 30.00th=[ 8], 40.00th=[ 8], 50.00th=[ 9], 60.00th=[ 10], | 70.00th=[ 12], 80.00th=[ 15], 90.00th=[ 22], 95.00th=[ 36], | 99.00th=[ 91], 99.50th=[ 117], 99.90th=[ 123], 99.95th=[ 123], | 99.99th=[ 123] bw ( KiB/s): min= 31, max= 416, per=25.47%, avg=161.51, stdev=89.21, samples=73 iops : min= 1, max= 26, avg= 9.64, stdev= 5.65, samples=73 write: IOPS=10, BW=172KiB/s (176kB/s)(6368KiB/37027msec) clat (msec): min=33, max=206, avg=80.49, stdev=27.52 lat (msec): min=33, max=206, avg=80.50, stdev=27.52 clat percentiles (msec): | 1.00th=[ 40], 5.00th=[ 46], 10.00th=[ 51], 20.00th=[ 57], | 30.00th=[ 65], 40.00th=[ 70], 50.00th=[ 77], 60.00th=[ 83], | 70.00th=[ 89], 80.00th=[ 100], 90.00th=[ 117], 95.00th=[ 130], | 99.00th=[ 178], 99.50th=[ 192], 99.90th=[ 207], 99.95th=[ 207], | 99.99th=[ 207] bw ( KiB/s): min= 63, max= 256, per=25.99%, avg=173.33, stdev=43.96, samples=73 iops : min= 3, max= 16, avg=10.38, stdev= 2.84, samples=73 lat (msec) : 4=0.13%, 10=30.21%, 20=12.63%, 50=9.11%, 100=37.37% lat (msec) : 250=10.55% cpu : usr=0.05%, sys=2.57%, ctx=883, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2843: Fri May 2 02:36:58 2025 read: IOPS=9, BW=152KiB/s (156kB/s)(5744KiB/37775msec) clat (msec): min=3, max=130, avg=12.47, stdev=13.76 lat (msec): min=3, max=130, avg=12.47, stdev=13.76 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 6], 10.00th=[ 6], 20.00th=[ 7], | 30.00th=[ 7], 40.00th=[ 8], 50.00th=[ 9], 60.00th=[ 10], | 70.00th=[ 11], 80.00th=[ 14], 90.00th=[ 23], 95.00th=[ 38], | 99.00th=[ 66], 99.50th=[ 113], 99.90th=[ 131], 99.95th=[ 131], | 99.99th=[ 131] bw ( KiB/s): min= 31, max= 479, per=25.09%, avg=159.07, stdev=97.38, samples=71 iops : min= 1, max= 29, avg= 9.48, stdev= 6.11, samples=71 write: IOPS=10, BW=173KiB/s (177kB/s)(6544KiB/37775msec) clat (msec): min=27, max=250, avg=80.78, stdev=29.62 lat (msec): min=27, max=250, avg=80.78, stdev=29.62 clat percentiles (msec): | 1.00th=[ 40], 5.00th=[ 44], 10.00th=[ 48], 20.00th=[ 56], | 30.00th=[ 62], 40.00th=[ 70], 50.00th=[ 78], 60.00th=[ 85], | 70.00th=[ 91], 80.00th=[ 102], 90.00th=[ 118], 95.00th=[ 136], | 99.00th=[ 174], 99.50th=[ 199], 99.90th=[ 251], 99.95th=[ 251], | 99.99th=[ 251] bw ( KiB/s): min= 32, max= 293, per=25.77%, avg=171.91, stdev=52.02, samples=75 iops : min= 2, max= 18, avg=10.28, stdev= 3.38, samples=75 lat (msec) : 4=0.26%, 10=30.99%, 20=10.29%, 50=10.55%, 100=36.72% lat (msec) : 250=11.07%, 500=0.13% cpu : usr=0.03%, sys=2.43%, ctx=866, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2844: Fri May 2 02:36:58 2025 read: IOPS=11, BW=177KiB/s (182kB/s)(6256KiB/35292msec) clat (usec): min=4128, max=68897, avg=11964.67, stdev=9796.98 lat (usec): min=4130, max=68898, avg=11966.40, stdev=9796.83 clat percentiles (usec): | 1.00th=[ 4490], 5.00th=[ 5145], 10.00th=[ 5800], 20.00th=[ 6652], | 30.00th=[ 7570], 40.00th=[ 8455], 50.00th=[ 9372], 60.00th=[10028], | 70.00th=[11076], 80.00th=[13173], 90.00th=[19530], 95.00th=[34866], | 99.00th=[61080], 99.50th=[68682], 99.90th=[68682], 99.95th=[68682], | 99.99th=[68682] bw ( KiB/s): min= 31, max= 479, per=28.54%, avg=180.91, stdev=111.58, samples=69 iops : min= 1, max= 29, avg=10.94, stdev= 6.97, samples=69 write: IOPS=10, BW=171KiB/s (175kB/s)(6032KiB/35292msec) clat (msec): min=37, max=231, avg=80.27, stdev=30.07 lat (msec): min=37, max=231, avg=80.27, stdev=30.07 clat percentiles (msec): | 1.00th=[ 39], 5.00th=[ 46], 10.00th=[ 51], 20.00th=[ 57], | 30.00th=[ 63], 40.00th=[ 68], 50.00th=[ 73], 60.00th=[ 82], | 70.00th=[ 89], 80.00th=[ 99], 90.00th=[ 118], 95.00th=[ 138], | 99.00th=[ 207], 99.50th=[ 226], 99.90th=[ 232], 99.95th=[ 232], | 99.99th=[ 232] bw ( KiB/s): min= 31, max= 256, per=25.50%, avg=170.07, stdev=49.32, samples=70 iops : min= 1, max= 16, avg=10.26, stdev= 3.21, samples=70 lat (msec) : 10=30.34%, 20=15.89%, 50=8.33%, 100=36.59%, 250=8.85% cpu : usr=0.05%, sys=2.70%, ctx=902, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2845: Fri May 2 02:36:58 2025 read: IOPS=10, BW=163KiB/s (167kB/s)(6032KiB/36996msec) clat (msec): min=3, max=110, avg=12.83, stdev=12.03 lat (msec): min=3, max=110, avg=12.84, stdev=12.03 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 6], 10.00th=[ 6], 20.00th=[ 7], | 30.00th=[ 8], 40.00th=[ 9], 50.00th=[ 10], 60.00th=[ 11], | 70.00th=[ 12], 80.00th=[ 15], 90.00th=[ 24], 95.00th=[ 36], | 99.00th=[ 68], 99.50th=[ 89], 99.90th=[ 111], 99.95th=[ 111], | 99.99th=[ 111] bw ( KiB/s): min= 31, max= 415, per=26.25%, avg=166.45, stdev=92.59, samples=71 iops : min= 1, max= 25, avg=10.15, stdev= 5.84, samples=71 write: IOPS=10, BW=169KiB/s (173kB/s)(6256KiB/36996msec) clat (msec): min=38, max=245, avg=81.34, stdev=29.70 lat (msec): min=38, max=245, avg=81.34, stdev=29.70 clat percentiles (msec): | 1.00th=[ 41], 5.00th=[ 47], 10.00th=[ 51], 20.00th=[ 58], | 30.00th=[ 65], 40.00th=[ 72], 50.00th=[ 77], 60.00th=[ 83], | 70.00th=[ 89], 80.00th=[ 100], 90.00th=[ 116], 95.00th=[ 142], | 99.00th=[ 182], 99.50th=[ 241], 99.90th=[ 245], 99.95th=[ 245], | 99.99th=[ 245] bw ( KiB/s): min= 63, max= 256, per=25.67%, avg=171.25, stdev=46.40, samples=72 iops : min= 3, max= 16, avg=10.46, stdev= 3.04, samples=72 lat (msec) : 4=0.26%, 10=28.26%, 20=14.32%, 50=10.03%, 100=37.11% lat (msec) : 250=10.03% cpu : usr=0.04%, sys=2.48%, ctx=891, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=634KiB/s (649kB/s), 152KiB/s-177KiB/s (156kB/s-182kB/s), io=23.4MiB (24.5MB), run=35292-37775msec WRITE: bw=667KiB/s (683kB/s), 169KiB/s-173KiB/s (173kB/s-177kB/s), io=24.6MiB (25.8MB), run=35292-37775msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=2882: Fri May 2 02:37:03 2025 read: IOPS=2, BW=3002KiB/s (3074kB/s)(7168KiB/2388msec) clat (msec): min=16, max=185, avg=87.25, stdev=65.11 lat (msec): min=16, max=185, avg=87.26, stdev=65.11 clat percentiles (msec): | 1.00th=[ 17], 5.00th=[ 17], 10.00th=[ 17], 20.00th=[ 53], | 30.00th=[ 57], 40.00th=[ 57], 50.00th=[ 62], 60.00th=[ 65], | 70.00th=[ 65], 80.00th=[ 174], 90.00th=[ 186], 95.00th=[ 186], | 99.00th=[ 186], 99.50th=[ 186], 99.90th=[ 186], 99.95th=[ 186], | 99.99th=[ 186] bw ( KiB/s): min= 4079, max= 6144, per=100.00%, avg=5111.50, stdev=1460.18, samples=2 iops : min= 3, max= 6, avg= 4.50, stdev= 2.12, samples=2 write: IOPS=2, BW=2144KiB/s (2196kB/s)(5120KiB/2388msec) clat (msec): min=38, max=1405, avg=346.11, stdev=593.64 lat (msec): min=38, max=1405, avg=346.18, stdev=593.63 clat percentiles (msec): | 1.00th=[ 40], 5.00th=[ 40], 10.00th=[ 40], 20.00th=[ 40], | 30.00th=[ 58], 40.00th=[ 58], 50.00th=[ 88], 60.00th=[ 88], | 70.00th=[ 142], 80.00th=[ 142], 90.00th=[ 1401], 95.00th=[ 1401], | 99.00th=[ 1401], 99.50th=[ 1401], 99.90th=[ 1401], 99.95th=[ 1401], | 99.99th=[ 1401] bw ( KiB/s): min= 4079, max= 4079, per=63.03%, avg=4079.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 lat (msec) : 20=8.33%, 50=8.33%, 100=50.00%, 250=25.00% cpu : usr=0.04%, sys=9.80%, ctx=485, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2883: Fri May 2 02:37:03 2025 read: IOPS=0, BW=743KiB/s (761kB/s)(2048KiB/2757msec) clat (msec): min=46, max=1568, avg=807.32, stdev=1076.12 lat (msec): min=46, max=1568, avg=807.32, stdev=1076.12 clat percentiles (msec): | 1.00th=[ 47], 5.00th=[ 47], 10.00th=[ 47], 20.00th=[ 47], | 30.00th=[ 47], 40.00th=[ 47], 50.00th=[ 47], 60.00th=[ 1569], | 70.00th=[ 1569], 80.00th=[ 1569], 90.00th=[ 1569], 95.00th=[ 1569], | 99.00th=[ 1569], 99.50th=[ 1569], 99.90th=[ 1569], 99.95th=[ 1569], | 99.99th=[ 1569] bw ( KiB/s): min= 1988, max= 2039, per=43.55%, avg=2013.50, stdev=36.06, samples=2 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=2 write: IOPS=3, BW=3714KiB/s (3803kB/s)(10.0MiB/2757msec) clat (msec): min=35, max=177, avg=112.32, stdev=50.17 lat (msec): min=35, max=177, avg=112.40, stdev=50.17 clat percentiles (msec): | 1.00th=[ 36], 5.00th=[ 36], 10.00th=[ 36], 20.00th=[ 66], | 30.00th=[ 66], 40.00th=[ 77], 50.00th=[ 114], 60.00th=[ 116], | 70.00th=[ 167], 80.00th=[ 167], 90.00th=[ 169], 95.00th=[ 178], | 99.00th=[ 178], 99.50th=[ 178], 99.90th=[ 178], 99.95th=[ 178], | 99.99th=[ 178] bw ( KiB/s): min= 5965, max= 8159, per=100.00%, avg=7062.00, stdev=1551.39, samples=2 iops : min= 5, max= 7, avg= 6.00, stdev= 1.41, samples=2 lat (msec) : 50=16.67%, 100=25.00%, 250=50.00% cpu : usr=0.00%, sys=13.75%, ctx=371, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2885: Fri May 2 02:37:03 2025 read: IOPS=1, BW=1444KiB/s (1479kB/s)(6144KiB/4254msec) clat (usec): min=441, max=1522.3k, avg=289062.55, stdev=606262.06 lat (usec): min=443, max=1522.3k, avg=289063.81, stdev=606262.30 clat percentiles (usec): | 1.00th=[ 441], 5.00th=[ 441], 10.00th=[ 441], | 20.00th=[ 502], 30.00th=[ 502], 40.00th=[ 3523], | 50.00th=[ 3523], 60.00th=[ 100140], 70.00th=[ 107480], | 80.00th=[ 107480], 90.00th=[1518339], 95.00th=[1518339], | 99.00th=[1518339], 99.50th=[1518339], 99.90th=[1518339], | 99.95th=[1518339], 99.99th=[1518339] bw ( KiB/s): min= 2043, max= 6144, per=88.55%, avg=4093.50, stdev=2899.84, samples=2 iops : min= 1, max= 6, avg= 3.50, stdev= 3.54, samples=2 write: IOPS=1, BW=1444KiB/s (1479kB/s)(6144KiB/4254msec) clat (msec): min=48, max=1840, avg=401.15, stdev=709.25 lat (msec): min=48, max=1840, avg=401.22, stdev=709.25 clat percentiles (msec): | 1.00th=[ 49], 5.00th=[ 49], 10.00th=[ 49], 20.00th=[ 68], | 30.00th=[ 68], 40.00th=[ 79], 50.00th=[ 79], 60.00th=[ 108], | 70.00th=[ 264], 80.00th=[ 264], 90.00th=[ 1838], 95.00th=[ 1838], | 99.00th=[ 1838], 99.50th=[ 1838], 99.90th=[ 1838], 99.95th=[ 1838], | 99.99th=[ 1838] bw ( KiB/s): min= 2011, max= 4087, per=41.96%, avg=2715.33, stdev=1188.04, samples=3 iops : min= 1, max= 3, avg= 2.00, stdev= 1.00, samples=3 lat (usec) : 500=8.33%, 750=8.33% lat (msec) : 4=8.33%, 50=8.33%, 100=16.67%, 250=25.00%, 500=8.33% cpu : usr=0.00%, sys=7.50%, ctx=521, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2886: Fri May 2 02:37:03 2025 read: IOPS=1, BW=1156KiB/s (1183kB/s)(5120KiB/4430msec) clat (msec): min=62, max=196, avg=99.34, stdev=54.99 lat (msec): min=62, max=196, avg=99.34, stdev=54.99 clat percentiles (msec): | 1.00th=[ 63], 5.00th=[ 63], 10.00th=[ 63], 20.00th=[ 63], | 30.00th=[ 75], 40.00th=[ 75], 50.00th=[ 78], 60.00th=[ 78], | 70.00th=[ 86], 80.00th=[ 86], 90.00th=[ 197], 95.00th=[ 197], | 99.00th=[ 197], 99.50th=[ 197], 99.90th=[ 197], 99.95th=[ 197], | 99.99th=[ 197] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=1, BW=1618KiB/s (1657kB/s)(7168KiB/4430msec) clat (msec): min=45, max=1716, avg=551.36, stdev=725.52 lat (msec): min=45, max=1716, avg=551.44, stdev=725.52 clat percentiles (msec): | 1.00th=[ 46], 5.00th=[ 46], 10.00th=[ 46], 20.00th=[ 68], | 30.00th=[ 114], 40.00th=[ 114], 50.00th=[ 142], 60.00th=[ 284], | 70.00th=[ 284], 80.00th=[ 1485], 90.00th=[ 1720], 95.00th=[ 1720], | 99.00th=[ 1720], 99.50th=[ 1720], 99.90th=[ 1720], 99.95th=[ 1720], | 99.99th=[ 1720] bw ( KiB/s): min= 4031, max= 4096, per=62.95%, avg=4074.33, stdev=37.53, samples=3 iops : min= 3, max= 4, avg= 3.67, stdev= 0.58, samples=3 lat (msec) : 50=8.33%, 100=41.67%, 250=25.00%, 500=8.33% cpu : usr=0.00%, sys=5.35%, ctx=701, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=4623KiB/s (4734kB/s), 743KiB/s-3002KiB/s (761kB/s-3074kB/s), io=20.0MiB (20.0MB), run=2388-4430msec WRITE: bw=6472KiB/s (6628kB/s), 1444KiB/s-3714KiB/s (1479kB/s-3803kB/s), io=28.0MiB (29.4MB), run=2388-4430msec rand-rw: (groupid=0, jobs=1): err= 0: pid=2887: Fri May 2 02:37:04 2025 read: IOPS=1, BW=1676KiB/s (1716kB/s)(7168KiB/4277msec) clat (msec): min=20, max=267, avg=75.91, stdev=85.30 lat (msec): min=20, max=267, avg=75.91, stdev=85.30 clat percentiles (msec): | 1.00th=[ 21], 5.00th=[ 21], 10.00th=[ 21], 20.00th=[ 32], | 30.00th=[ 51], 40.00th=[ 51], 50.00th=[ 52], 60.00th=[ 53], | 70.00th=[ 53], 80.00th=[ 57], 90.00th=[ 268], 95.00th=[ 268], | 99.00th=[ 268], 99.50th=[ 268], 99.90th=[ 268], 99.95th=[ 268], | 99.99th=[ 268] bw ( KiB/s): min= 2043, max= 6144, per=77.95%, avg=3411.67, stdev=2366.27, samples=3 iops : min= 1, max= 6, avg= 3.00, stdev= 2.65, samples=3 write: IOPS=1, BW=1197KiB/s (1226kB/s)(5120KiB/4277msec) clat (msec): min=130, max=1642, avg=733.62, stdev=813.44 lat (msec): min=130, max=1642, avg=733.70, stdev=813.43 clat percentiles (msec): | 1.00th=[ 131], 5.00th=[ 131], 10.00th=[ 131], 20.00th=[ 131], | 30.00th=[ 136], 40.00th=[ 136], 50.00th=[ 155], 60.00th=[ 155], | 70.00th=[ 1603], 80.00th=[ 1603], 90.00th=[ 1636], 95.00th=[ 1636], | 99.00th=[ 1636], 99.50th=[ 1636], 99.90th=[ 1636], 99.95th=[ 1636], | 99.99th=[ 1636] bw ( KiB/s): min= 2043, max= 4096, per=50.08%, avg=3069.50, stdev=1451.69, samples=2 iops : min= 1, max= 4, avg= 2.50, stdev= 2.12, samples=2 lat (msec) : 50=16.67%, 100=33.33%, 250=25.00%, 500=8.33% cpu : usr=0.00%, sys=1.50%, ctx=43, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2888: Fri May 2 02:37:04 2025 read: IOPS=0, BW=438KiB/s (448kB/s)(2048KiB/4678msec) clat (msec): min=52, max=1511, avg=782.12, stdev=1032.06 lat (msec): min=52, max=1511, avg=782.12, stdev=1032.06 clat percentiles (msec): | 1.00th=[ 53], 5.00th=[ 53], 10.00th=[ 53], 20.00th=[ 53], | 30.00th=[ 53], 40.00th=[ 53], 50.00th=[ 53], 60.00th=[ 1519], | 70.00th=[ 1519], 80.00th=[ 1519], 90.00th=[ 1519], 95.00th=[ 1519], | 99.00th=[ 1519], 99.50th=[ 1519], 99.90th=[ 1519], 99.95th=[ 1519], | 99.99th=[ 1519] bw ( KiB/s): min= 1896, max= 2035, per=44.91%, avg=1965.50, stdev=98.29, samples=2 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=2 write: IOPS=2, BW=2189KiB/s (2242kB/s)(10.0MiB/4678msec) clat (msec): min=46, max=1855, avg=302.89, stdev=552.51 lat (msec): min=46, max=1855, avg=303.77, stdev=552.13 clat percentiles (msec): | 1.00th=[ 47], 5.00th=[ 47], 10.00th=[ 47], 20.00th=[ 60], | 30.00th=[ 69], 40.00th=[ 77], 50.00th=[ 81], 60.00th=[ 103], | 70.00th=[ 259], 80.00th=[ 259], 90.00th=[ 300], 95.00th=[ 1854], | 99.00th=[ 1854], 99.50th=[ 1854], 99.90th=[ 1854], 99.95th=[ 1854], | 99.99th=[ 1854] bw ( KiB/s): min= 2035, max= 7585, per=55.91%, avg=3426.50, stdev=2772.34, samples=4 iops : min= 1, max= 7, avg= 2.50, stdev= 3.00, samples=4 lat (msec) : 50=8.33%, 100=41.67%, 250=16.67%, 500=16.67% cpu : usr=0.09%, sys=1.75%, ctx=43, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2889: Fri May 2 02:37:04 2025 read: IOPS=1, BW=1360KiB/s (1393kB/s)(6144KiB/4516msec) clat (msec): min=25, max=1787, avg=346.21, stdev=706.90 lat (msec): min=25, max=1787, avg=346.21, stdev=706.90 clat percentiles (msec): | 1.00th=[ 26], 5.00th=[ 26], 10.00th=[ 26], 20.00th=[ 39], | 30.00th=[ 39], 40.00th=[ 42], 50.00th=[ 42], 60.00th=[ 73], | 70.00th=[ 111], 80.00th=[ 111], 90.00th=[ 1787], 95.00th=[ 1787], | 99.00th=[ 1787], 99.50th=[ 1787], 99.90th=[ 1787], 99.95th=[ 1787], | 99.99th=[ 1787] bw ( KiB/s): min= 2043, max= 2043, per=46.68%, avg=2043.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=1, BW=1360KiB/s (1393kB/s)(6144KiB/4516msec) clat (msec): min=69, max=1749, avg=400.61, stdev=664.52 lat (msec): min=69, max=1749, avg=400.69, stdev=664.52 clat percentiles (msec): | 1.00th=[ 70], 5.00th=[ 70], 10.00th=[ 70], 20.00th=[ 82], | 30.00th=[ 82], 40.00th=[ 111], 50.00th=[ 111], 60.00th=[ 133], | 70.00th=[ 262], 80.00th=[ 262], 90.00th=[ 1754], 95.00th=[ 1754], | 99.00th=[ 1754], 99.50th=[ 1754], 99.90th=[ 1754], 99.95th=[ 1754], | 99.99th=[ 1754] bw ( KiB/s): min= 2043, max= 4087, per=50.01%, avg=3065.00, stdev=1445.33, samples=2 iops : min= 1, max= 3, avg= 2.00, stdev= 1.41, samples=2 lat (msec) : 50=25.00%, 100=25.00%, 250=25.00%, 500=8.33% cpu : usr=0.11%, sys=1.42%, ctx=37, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2890: Fri May 2 02:37:04 2025 read: IOPS=1, BW=1173KiB/s (1201kB/s)(5120KiB/4366msec) clat (msec): min=25, max=155, avg=76.27, stdev=48.58 lat (msec): min=25, max=155, avg=76.27, stdev=48.58 clat percentiles (msec): | 1.00th=[ 26], 5.00th=[ 26], 10.00th=[ 26], 20.00th=[ 26], | 30.00th=[ 52], 40.00th=[ 52], 50.00th=[ 71], 60.00th=[ 71], | 70.00th=[ 79], 80.00th=[ 79], 90.00th=[ 155], 95.00th=[ 155], | 99.00th=[ 155], 99.50th=[ 155], 99.90th=[ 155], 99.95th=[ 155], | 99.99th=[ 155] bw ( KiB/s): min= 2048, max= 6144, per=93.58%, avg=4096.00, stdev=2896.31, samples=2 iops : min= 2, max= 6, avg= 4.00, stdev= 2.83, samples=2 write: IOPS=1, BW=1642KiB/s (1681kB/s)(7168KiB/4366msec) clat (msec): min=80, max=1715, avg=566.08, stdev=738.13 lat (msec): min=80, max=1715, avg=566.17, stdev=738.14 clat percentiles (msec): | 1.00th=[ 81], 5.00th=[ 81], 10.00th=[ 81], 20.00th=[ 97], | 30.00th=[ 108], 40.00th=[ 108], 50.00th=[ 115], 60.00th=[ 284], | 70.00th=[ 284], 80.00th=[ 1569], 90.00th=[ 1720], 95.00th=[ 1720], | 99.00th=[ 1720], 99.50th=[ 1720], 99.90th=[ 1720], 99.95th=[ 1720], | 99.99th=[ 1720] bw ( KiB/s): min= 2048, max= 6144, per=66.83%, avg=4096.00, stdev=2048.00, samples=3 iops : min= 2, max= 6, avg= 4.00, stdev= 2.00, samples=3 lat (msec) : 50=8.33%, 100=41.67%, 250=25.00%, 500=8.33% cpu : usr=0.02%, sys=0.92%, ctx=26, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=4378KiB/s (4483kB/s), 438KiB/s-1676KiB/s (448kB/s-1716kB/s), io=20.0MiB (20.0MB), run=4277-4678msec WRITE: bw=6129KiB/s (6276kB/s), 1197KiB/s-2189KiB/s (1226kB/s-2242kB/s), io=28.0MiB (29.4MB), run=4277-4678msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=2899: Fri May 2 02:37:07 2025 read: IOPS=4, BW=16.4MiB/s (17.2MB/s)(12.0MiB/731msec) clat (msec): min=15, max=399, avg=176.75, stdev=199.21 lat (msec): min=15, max=399, avg=176.75, stdev=199.21 clat percentiles (msec): | 1.00th=[ 16], 5.00th=[ 16], 10.00th=[ 16], 20.00th=[ 16], | 30.00th=[ 16], 40.00th=[ 116], 50.00th=[ 116], 60.00th=[ 116], | 70.00th=[ 401], 80.00th=[ 401], 90.00th=[ 401], 95.00th=[ 401], | 99.00th=[ 401], 99.50th=[ 401], 99.90th=[ 401], 99.95th=[ 401], | 99.99th=[ 401] bw ( KiB/s): min= 8192, max= 8192, per=64.35%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 20=33.33%, 250=33.33%, 500=33.33% cpu : usr=0.27%, sys=10.96%, ctx=82, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2900: Fri May 2 02:37:07 2025 read: IOPS=0, BW=3183KiB/s (3259kB/s)(4096KiB/1287msec) clat (nsec): min=171452k, max=171452k, avg=171451639.00, stdev= 0.00 lat (nsec): min=171454k, max=171454k, avg=171454100.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 171], 5.00th=[ 171], 10.00th=[ 171], 20.00th=[ 171], | 30.00th=[ 171], 40.00th=[ 171], 50.00th=[ 171], 60.00th=[ 171], | 70.00th=[ 171], 80.00th=[ 171], 90.00th=[ 171], 95.00th=[ 171], | 99.00th=[ 171], 99.50th=[ 171], 99.90th=[ 171], 99.95th=[ 171], | 99.99th=[ 171] bw ( KiB/s): min= 8192, max= 8192, per=64.35%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=1, BW=6365KiB/s (6518kB/s)(8192KiB/1287msec) clat (msec): min=391, max=572, avg=482.26, stdev=127.66 lat (msec): min=392, max=572, avg=482.63, stdev=127.71 clat percentiles (msec): | 1.00th=[ 393], 5.00th=[ 393], 10.00th=[ 393], 20.00th=[ 393], | 30.00th=[ 393], 40.00th=[ 393], 50.00th=[ 393], 60.00th=[ 575], | 70.00th=[ 575], 80.00th=[ 575], 90.00th=[ 575], 95.00th=[ 575], | 99.00th=[ 575], 99.50th=[ 575], 99.90th=[ 575], 99.95th=[ 575], | 99.99th=[ 575] bw ( KiB/s): min= 8192, max= 8192, per=33.48%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=33.33%, 750=33.33% cpu : usr=0.00%, sys=17.03%, ctx=88, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2901: Fri May 2 02:37:07 2025 write: IOPS=2, BW=9177KiB/s (9397kB/s)(12.0MiB/1339msec) clat (msec): min=313, max=501, avg=438.79, stdev=108.83 lat (msec): min=313, max=502, avg=439.14, stdev=108.82 clat percentiles (msec): | 1.00th=[ 313], 5.00th=[ 313], 10.00th=[ 313], 20.00th=[ 313], | 30.00th=[ 313], 40.00th=[ 502], 50.00th=[ 502], 60.00th=[ 502], | 70.00th=[ 502], 80.00th=[ 502], 90.00th=[ 502], 95.00th=[ 502], | 99.00th=[ 502], 99.50th=[ 502], 99.90th=[ 502], 99.95th=[ 502], | 99.99th=[ 502] bw ( KiB/s): min= 8175, max= 8192, per=33.44%, avg=8183.50, stdev=12.02, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 500=33.33%, 750=66.67% cpu : usr=0.07%, sys=21.00%, ctx=93, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2902: Fri May 2 02:37:07 2025 write: IOPS=2, BW=9691KiB/s (9923kB/s)(12.0MiB/1268msec) clat (msec): min=265, max=559, avg=367.01, stdev=166.43 lat (msec): min=265, max=559, avg=367.31, stdev=166.44 clat percentiles (msec): | 1.00th=[ 266], 5.00th=[ 266], 10.00th=[ 266], 20.00th=[ 266], | 30.00th=[ 266], 40.00th=[ 279], 50.00th=[ 279], 60.00th=[ 279], | 70.00th=[ 558], 80.00th=[ 558], 90.00th=[ 558], 95.00th=[ 558], | 99.00th=[ 558], 99.50th=[ 558], 99.90th=[ 558], 99.95th=[ 558], | 99.99th=[ 558] bw ( KiB/s): min= 8175, max= 8175, per=33.41%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 500=66.67%, 750=33.33% cpu : usr=0.00%, sys=21.39%, ctx=87, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=12.4MiB/s (13.0MB/s), 3183KiB/s-16.4MiB/s (3259kB/s-17.2MB/s), io=16.0MiB (16.8MB), run=731-1287msec WRITE: bw=23.9MiB/s (25.1MB/s), 6365KiB/s-9691KiB/s (6518kB/s-9923kB/s), io=32.0MiB (33.6MB), run=1268-1339msec rand-rw: (groupid=0, jobs=1): err= 0: pid=2904: Fri May 2 02:37:08 2025 read: IOPS=1, BW=7642KiB/s (7825kB/s)(12.0MiB/1608msec) clat (msec): min=388, max=627, avg=502.31, stdev=120.08 lat (msec): min=388, max=627, avg=502.31, stdev=120.08 clat percentiles (msec): | 1.00th=[ 388], 5.00th=[ 388], 10.00th=[ 388], 20.00th=[ 388], | 30.00th=[ 388], 40.00th=[ 493], 50.00th=[ 493], 60.00th=[ 493], | 70.00th=[ 625], 80.00th=[ 625], 90.00th=[ 625], 95.00th=[ 625], | 99.00th=[ 625], 99.50th=[ 625], 99.90th=[ 625], 99.95th=[ 625], | 99.99th=[ 625] bw ( KiB/s): min= 8110, max= 8175, per=83.50%, avg=8142.50, stdev=45.96, samples=2 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=2 lat (msec) : 500=66.67%, 750=33.33% cpu : usr=0.00%, sys=0.87%, ctx=16, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2905: Fri May 2 02:37:08 2025 read: IOPS=0, BW=2438KiB/s (2497kB/s)(4096KiB/1680msec) clat (nsec): min=257041k, max=257041k, avg=257041224.00, stdev= 0.00 lat (nsec): min=257042k, max=257042k, avg=257042392.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 257], 5.00th=[ 257], 10.00th=[ 257], 20.00th=[ 257], | 30.00th=[ 257], 40.00th=[ 257], 50.00th=[ 257], 60.00th=[ 257], | 70.00th=[ 257], 80.00th=[ 257], 90.00th=[ 257], 95.00th=[ 257], | 99.00th=[ 257], 99.50th=[ 257], 99.90th=[ 257], 99.95th=[ 257], | 99.99th=[ 257] bw ( KiB/s): min= 8192, max= 8192, per=84.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=1, BW=4876KiB/s (4993kB/s)(8192KiB/1680msec) clat (msec): min=390, max=744, avg=567.67, stdev=250.38 lat (msec): min=391, max=745, avg=568.05, stdev=250.30 clat percentiles (msec): | 1.00th=[ 393], 5.00th=[ 393], 10.00th=[ 393], 20.00th=[ 393], | 30.00th=[ 393], 40.00th=[ 393], 50.00th=[ 393], 60.00th=[ 743], | 70.00th=[ 743], 80.00th=[ 743], 90.00th=[ 743], 95.00th=[ 743], | 99.00th=[ 743], 99.50th=[ 743], 99.90th=[ 743], 99.95th=[ 743], | 99.99th=[ 743] bw ( KiB/s): min= 8192, max= 8192, per=48.05%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 500=66.67%, 750=33.33% cpu : usr=0.00%, sys=2.32%, ctx=27, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2906: Fri May 2 02:37:08 2025 write: IOPS=1, BW=7030KiB/s (7198kB/s)(12.0MiB/1748msec) clat (msec): min=292, max=718, avg=492.68, stdev=214.24 lat (msec): min=293, max=719, avg=493.49, stdev=214.22 clat percentiles (msec): | 1.00th=[ 292], 5.00th=[ 292], 10.00th=[ 292], 20.00th=[ 292], | 30.00th=[ 292], 40.00th=[ 468], 50.00th=[ 468], 60.00th=[ 468], | 70.00th=[ 718], 80.00th=[ 718], 90.00th=[ 718], 95.00th=[ 718], | 99.00th=[ 718], 99.50th=[ 718], 99.90th=[ 718], 99.95th=[ 718], | 99.99th=[ 718] bw ( KiB/s): min= 8110, max= 8126, per=47.62%, avg=8118.00, stdev=11.31, samples=2 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=2 lat (msec) : 500=66.67%, 750=33.33% cpu : usr=0.34%, sys=3.09%, ctx=44, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2907: Fri May 2 02:37:08 2025 write: IOPS=1, BW=6393KiB/s (6547kB/s)(12.0MiB/1922msec) clat (msec): min=133, max=1173, avg=560.16, stdev=544.73 lat (msec): min=133, max=1174, avg=560.49, stdev=544.81 clat percentiles (msec): | 1.00th=[ 134], 5.00th=[ 134], 10.00th=[ 134], 20.00th=[ 134], | 30.00th=[ 134], 40.00th=[ 372], 50.00th=[ 372], 60.00th=[ 372], | 70.00th=[ 1167], 80.00th=[ 1167], 90.00th=[ 1167], 95.00th=[ 1167], | 99.00th=[ 1167], 99.50th=[ 1167], 99.90th=[ 1167], 99.95th=[ 1167], | 99.99th=[ 1167] bw ( KiB/s): min= 8192, max= 8192, per=48.05%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=33.33% cpu : usr=0.00%, sys=4.58%, ctx=40, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=9752KiB/s (9986kB/s), 2438KiB/s-7642KiB/s (2497kB/s-7825kB/s), io=16.0MiB (16.8MB), run=1608-1680msec WRITE: bw=16.6MiB/s (17.5MB/s), 4876KiB/s-7030KiB/s (4993kB/s-7198kB/s), io=32.0MiB (33.6MB), run=1680-1922msec