== sanity test 398b: DIO and buffer IO race ============== 18:55:51 (1746140151) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 2.8687 s, 17.5 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=23542: Thu May 1 18:56:51 2025 read: IOPS=28, BW=112KiB/s (115kB/s)(5992KiB/53316msec) clat (usec): min=71, max=106742, avg=19754.45, stdev=12566.61 lat (usec): min=71, max=106743, avg=19758.61, stdev=12566.81 clat percentiles (usec): | 1.00th=[ 99], 5.00th=[ 128], 10.00th=[ 6915], 20.00th=[ 10421], | 30.00th=[ 12780], 40.00th=[ 15139], 50.00th=[ 17695], 60.00th=[ 20579], | 70.00th=[ 24249], 80.00th=[ 28705], 90.00th=[ 35914], 95.00th=[ 41681], | 99.00th=[ 61604], 99.50th=[ 72877], 99.90th=[ 94897], 99.95th=[106431], | 99.99th=[106431] bw ( KiB/s): min= 23, max= 224, per=25.06%, avg=111.53, stdev=30.90, samples=106 iops : min= 5, max= 56, avg=27.42, stdev= 7.79, samples=106 write: IOPS=29, BW=118KiB/s (121kB/s)(6296KiB/53316msec) clat (usec): min=520, max=389187, avg=14937.34, stdev=18741.94 lat (usec): min=520, max=389189, avg=14940.78, stdev=18743.42 clat percentiles (usec): | 1.00th=[ 693], 5.00th=[ 750], 10.00th=[ 791], 20.00th=[ 930], | 30.00th=[ 2409], 40.00th=[ 4883], 50.00th=[ 10159], 60.00th=[ 14353], | 70.00th=[ 19268], 80.00th=[ 26346], 90.00th=[ 36963], 95.00th=[ 45876], | 99.00th=[ 64750], 99.50th=[ 77071], 99.90th=[185598], 99.95th=[387974], | 99.99th=[387974] bw ( KiB/s): min= 32, max= 207, per=25.41%, avg=118.40, stdev=36.38, samples=105 iops : min= 8, max= 51, avg=29.14, stdev= 9.11, samples=105 lat (usec) : 100=0.62%, 250=2.54%, 500=0.10%, 750=2.90%, 1000=8.14% lat (msec) : 2=3.39%, 4=4.79%, 10=11.95%, 20=30.50%, 50=32.19% lat (msec) : 100=2.73%, 250=0.13%, 500=0.03% cpu : usr=0.20%, sys=8.88%, ctx=4949, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23543: Thu May 1 18:56:51 2025 read: IOPS=27, BW=110KiB/s (113kB/s)(5940KiB/53927msec) clat (usec): min=71, max=281380, avg=19686.64, stdev=14231.77 lat (usec): min=71, max=281382, avg=19689.03, stdev=14231.40 clat percentiles (usec): | 1.00th=[ 97], 5.00th=[ 128], 10.00th=[ 7242], 20.00th=[ 10159], | 30.00th=[ 12649], 40.00th=[ 14877], 50.00th=[ 17171], 60.00th=[ 20055], | 70.00th=[ 23462], 80.00th=[ 28443], 90.00th=[ 34866], 95.00th=[ 41681], | 99.00th=[ 58983], 99.50th=[ 74974], 99.90th=[107480], 99.95th=[283116], | 99.99th=[283116] bw ( KiB/s): min= 31, max= 175, per=24.74%, avg=110.08, stdev=29.13, samples=106 iops : min= 7, max= 43, avg=27.15, stdev= 7.24, samples=106 write: IOPS=29, BW=118KiB/s (121kB/s)(6348KiB/53927msec) clat (usec): min=525, max=154253, avg=15349.39, stdev=16818.98 lat (usec): min=526, max=154254, avg=15350.81, stdev=16818.99 clat percentiles (usec): | 1.00th=[ 685], 5.00th=[ 742], 10.00th=[ 783], 20.00th=[ 971], | 30.00th=[ 2409], 40.00th=[ 5407], 50.00th=[ 10945], 60.00th=[ 15270], | 70.00th=[ 20055], 80.00th=[ 26608], 90.00th=[ 37487], 95.00th=[ 48497], | 99.00th=[ 71828], 99.50th=[ 88605], 99.90th=[114820], 99.95th=[154141], | 99.99th=[154141] bw ( KiB/s): min= 48, max= 255, per=25.12%, avg=117.04, stdev=40.95, samples=106 iops : min= 12, max= 63, avg=28.89, stdev=10.21, samples=106 lat (usec) : 100=0.78%, 250=2.31%, 500=0.07%, 750=3.03%, 1000=7.62% lat (msec) : 2=3.87%, 4=4.23%, 10=12.14%, 20=31.09%, 50=31.45% lat (msec) : 100=3.26%, 250=0.13%, 500=0.03% cpu : usr=0.16%, sys=8.99%, ctx=4953, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23544: Thu May 1 18:56:51 2025 read: IOPS=28, BW=113KiB/s (116kB/s)(6028KiB/53354msec) clat (usec): min=65, max=117436, avg=20093.28, stdev=14102.90 lat (usec): min=66, max=117437, avg=20094.51, stdev=14102.85 clat percentiles (usec): | 1.00th=[ 92], 5.00th=[ 118], 10.00th=[ 6456], 20.00th=[ 10552], | 30.00th=[ 12780], 40.00th=[ 15139], 50.00th=[ 17433], 60.00th=[ 20841], | 70.00th=[ 23987], 80.00th=[ 28443], 90.00th=[ 35914], 95.00th=[ 43779], | 99.00th=[ 72877], 99.50th=[ 83362], 99.90th=[113771], 99.95th=[116917], | 99.99th=[116917] bw ( KiB/s): min= 63, max= 224, per=25.40%, avg=113.02, stdev=28.95, samples=105 iops : min= 15, max= 56, avg=27.89, stdev= 7.26, samples=105 write: IOPS=29, BW=117KiB/s (120kB/s)(6260KiB/53354msec) clat (usec): min=531, max=329855, avg=14540.91, stdev=17858.56 lat (usec): min=532, max=329858, avg=14544.88, stdev=17857.84 clat percentiles (usec): | 1.00th=[ 693], 5.00th=[ 750], 10.00th=[ 799], 20.00th=[ 889], | 30.00th=[ 1860], 40.00th=[ 4228], 50.00th=[ 9503], 60.00th=[ 13566], | 70.00th=[ 18482], 80.00th=[ 26346], 90.00th=[ 36963], 95.00th=[ 46924], | 99.00th=[ 65799], 99.50th=[ 86508], 99.90th=[109577], 99.95th=[329253], | 99.99th=[329253] bw ( KiB/s): min= 55, max= 216, per=25.14%, avg=117.13, stdev=37.33, samples=105 iops : min= 13, max= 54, avg=28.91, stdev= 9.40, samples=105 lat (usec) : 100=1.11%, 250=2.77%, 500=0.10%, 750=2.51%, 1000=9.64% lat (msec) : 2=3.97%, 4=4.23%, 10=10.74%, 20=30.08%, 50=31.48% lat (msec) : 100=3.19%, 250=0.16%, 500=0.03% cpu : usr=0.14%, sys=8.85%, ctx=4871, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23545: Thu May 1 18:56:51 2025 read: IOPS=28, BW=114KiB/s (116kB/s)(6052KiB/53274msec) clat (usec): min=78, max=121920, avg=20106.08, stdev=13295.50 lat (usec): min=79, max=121921, avg=20112.04, stdev=13289.90 clat percentiles (usec): | 1.00th=[ 93], 5.00th=[ 125], 10.00th=[ 6915], 20.00th=[ 10290], | 30.00th=[ 12911], 40.00th=[ 15401], 50.00th=[ 17695], 60.00th=[ 20841], | 70.00th=[ 24511], 80.00th=[ 28443], 90.00th=[ 35390], 95.00th=[ 43254], | 99.00th=[ 65274], 99.50th=[ 73925], 99.90th=[113771], 99.95th=[122160], | 99.99th=[122160] bw ( KiB/s): min= 31, max= 205, per=25.56%, avg=113.74, stdev=32.67, samples=105 iops : min= 7, max= 51, avg=27.99, stdev= 8.18, samples=105 write: IOPS=29, BW=117KiB/s (120kB/s)(6236KiB/53274msec) clat (usec): min=530, max=446490, avg=14517.02, stdev=19117.34 lat (usec): min=530, max=446492, avg=14521.01, stdev=19116.13 clat percentiles (usec): | 1.00th=[ 676], 5.00th=[ 750], 10.00th=[ 791], 20.00th=[ 906], | 30.00th=[ 2278], 40.00th=[ 4752], 50.00th=[ 9896], 60.00th=[ 14091], | 70.00th=[ 18744], 80.00th=[ 25822], 90.00th=[ 34866], 95.00th=[ 43779], | 99.00th=[ 64226], 99.50th=[ 84411], 99.90th=[135267], 99.95th=[446694], | 99.99th=[446694] bw ( KiB/s): min= 39, max= 216, per=25.16%, avg=117.26, stdev=33.82, samples=105 iops : min= 9, max= 54, avg=28.87, stdev= 8.54, samples=105 lat (usec) : 100=0.88%, 250=2.51%, 500=0.13%, 750=2.60%, 1000=9.15% lat (msec) : 2=3.16%, 4=4.82%, 10=11.59%, 20=30.08%, 50=31.84% lat (msec) : 100=3.03%, 250=0.20%, 500=0.03% cpu : usr=0.12%, sys=9.06%, ctx=4917, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=445KiB/s (456kB/s), 110KiB/s-114KiB/s (113kB/s-116kB/s), io=23.4MiB (24.6MB), run=53274-53927msec WRITE: bw=466KiB/s (477kB/s), 117KiB/s-118KiB/s (120kB/s-121kB/s), io=24.6MiB (25.7MB), run=53274-53927msec rand-rw: (groupid=0, jobs=1): err= 0: pid=23537: Thu May 1 18:57:16 2025 read: IOPS=19, BW=76.8KiB/s (78.6kB/s)(5992KiB/78058msec) clat (msec): min=3, max=365, avg=18.68, stdev=18.88 lat (msec): min=3, max=365, avg=18.68, stdev=18.88 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 6], 10.00th=[ 7], 20.00th=[ 9], | 30.00th=[ 10], 40.00th=[ 11], 50.00th=[ 13], 60.00th=[ 15], | 70.00th=[ 18], 80.00th=[ 25], 90.00th=[ 42], 95.00th=[ 55], | 99.00th=[ 86], 99.50th=[ 105], 99.90th=[ 163], 99.95th=[ 368], | 99.99th=[ 368] bw ( KiB/s): min= 8, max= 224, per=24.90%, avg=76.44, stdev=44.98, samples=154 iops : min= 2, max= 56, avg=18.79, stdev=11.26, samples=154 write: IOPS=20, BW=80.7KiB/s (82.6kB/s)(6296KiB/78058msec) clat (msec): min=6, max=192, avg=31.61, stdev=20.20 lat (msec): min=6, max=192, avg=31.61, stdev=20.20 clat percentiles (msec): | 1.00th=[ 10], 5.00th=[ 13], 10.00th=[ 15], 20.00th=[ 17], | 30.00th=[ 20], 40.00th=[ 22], 50.00th=[ 25], 60.00th=[ 29], | 70.00th=[ 36], 80.00th=[ 46], 90.00th=[ 60], 95.00th=[ 72], | 99.00th=[ 101], 99.50th=[ 116], 99.90th=[ 161], 99.95th=[ 192], | 99.99th=[ 192] bw ( KiB/s): min= 7, max= 184, per=24.89%, avg=80.15, stdev=41.33, samples=154 iops : min= 1, max= 46, avg=19.72, stdev=10.40, samples=154 lat (msec) : 4=0.13%, 10=17.06%, 20=36.52%, 50=35.06%, 100=10.48% lat (msec) : 250=0.72%, 500=0.03% cpu : usr=0.07%, sys=4.15%, ctx=3416, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23539: Thu May 1 18:57:16 2025 read: IOPS=19, BW=76.1KiB/s (77.9kB/s)(5940KiB/78064msec) clat (msec): min=3, max=360, avg=18.90, stdev=19.29 lat (msec): min=3, max=360, avg=18.90, stdev=19.29 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 6], 10.00th=[ 7], 20.00th=[ 8], | 30.00th=[ 10], 40.00th=[ 11], 50.00th=[ 13], 60.00th=[ 15], | 70.00th=[ 18], 80.00th=[ 26], 90.00th=[ 43], 95.00th=[ 57], | 99.00th=[ 82], 99.50th=[ 99], 99.90th=[ 155], 99.95th=[ 363], | 99.99th=[ 363] bw ( KiB/s): min= 8, max= 224, per=24.93%, avg=76.54, stdev=45.55, samples=154 iops : min= 2, max= 56, avg=18.81, stdev=11.48, samples=154 write: IOPS=20, BW=81.3KiB/s (83.3kB/s)(6348KiB/78064msec) clat (msec): min=6, max=204, avg=31.31, stdev=20.33 lat (msec): min=6, max=204, avg=31.32, stdev=20.33 clat percentiles (msec): | 1.00th=[ 10], 5.00th=[ 13], 10.00th=[ 15], 20.00th=[ 17], | 30.00th=[ 19], 40.00th=[ 22], 50.00th=[ 24], 60.00th=[ 28], | 70.00th=[ 35], 80.00th=[ 45], 90.00th=[ 62], 95.00th=[ 73], | 99.00th=[ 104], 99.50th=[ 120], 99.90th=[ 157], 99.95th=[ 205], | 99.99th=[ 205] bw ( KiB/s): min= 16, max= 176, per=25.21%, avg=81.17, stdev=42.85, samples=155 iops : min= 4, max= 44, avg=19.97, stdev=10.76, samples=155 lat (msec) : 4=0.10%, 10=18.10%, 20=35.58%, 50=34.37%, 100=11.07% lat (msec) : 250=0.75%, 500=0.03% cpu : usr=0.05%, sys=4.26%, ctx=3442, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23540: Thu May 1 18:57:16 2025 read: IOPS=19, BW=77.8KiB/s (79.7kB/s)(6028KiB/77445msec) clat (msec): min=3, max=152, avg=18.75, stdev=17.21 lat (msec): min=3, max=152, avg=18.75, stdev=17.21 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 6], 10.00th=[ 7], 20.00th=[ 9], | 30.00th=[ 10], 40.00th=[ 11], 50.00th=[ 13], 60.00th=[ 16], | 70.00th=[ 19], 80.00th=[ 26], 90.00th=[ 39], 95.00th=[ 57], | 99.00th=[ 87], 99.50th=[ 97], 99.90th=[ 136], 99.95th=[ 153], | 99.99th=[ 153] bw ( KiB/s): min= 16, max= 224, per=25.28%, avg=77.62, stdev=43.97, samples=153 iops : min= 4, max= 56, avg=19.05, stdev=11.03, samples=153 write: IOPS=20, BW=80.8KiB/s (82.8kB/s)(6260KiB/77445msec) clat (msec): min=6, max=435, avg=31.27, stdev=22.25 lat (msec): min=6, max=435, avg=31.27, stdev=22.25 clat percentiles (msec): | 1.00th=[ 10], 5.00th=[ 12], 10.00th=[ 14], 20.00th=[ 17], | 30.00th=[ 19], 40.00th=[ 21], 50.00th=[ 24], 60.00th=[ 29], | 70.00th=[ 36], 80.00th=[ 44], 90.00th=[ 59], 95.00th=[ 70], | 99.00th=[ 103], 99.50th=[ 121], 99.90th=[ 174], 99.95th=[ 435], | 99.99th=[ 435] bw ( KiB/s): min= 15, max= 200, per=24.98%, avg=80.44, stdev=42.67, samples=153 iops : min= 3, max= 50, avg=19.75, stdev=10.71, samples=153 lat (msec) : 4=0.20%, 10=17.64%, 20=36.39%, 50=34.47%, 100=10.51% lat (msec) : 250=0.75%, 500=0.03% cpu : usr=0.08%, sys=4.22%, ctx=3416, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23541: Thu May 1 18:57:16 2025 read: IOPS=19, BW=77.9KiB/s (79.8kB/s)(6052KiB/77695msec) clat (msec): min=2, max=128, avg=18.89, stdev=16.76 lat (msec): min=2, max=128, avg=18.90, stdev=16.76 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 6], 10.00th=[ 7], 20.00th=[ 9], | 30.00th=[ 10], 40.00th=[ 11], 50.00th=[ 13], 60.00th=[ 15], | 70.00th=[ 19], 80.00th=[ 27], 90.00th=[ 43], 95.00th=[ 57], | 99.00th=[ 84], 99.50th=[ 92], 99.90th=[ 102], 99.95th=[ 129], | 99.99th=[ 129] bw ( KiB/s): min= 7, max= 216, per=25.35%, avg=77.81, stdev=43.90, samples=154 iops : min= 1, max= 54, avg=19.07, stdev=10.98, samples=154 write: IOPS=20, BW=80.3KiB/s (82.2kB/s)(6236KiB/77695msec) clat (msec): min=7, max=437, avg=31.35, stdev=22.95 lat (msec): min=7, max=437, avg=31.35, stdev=22.95 clat percentiles (msec): | 1.00th=[ 10], 5.00th=[ 12], 10.00th=[ 14], 20.00th=[ 17], | 30.00th=[ 19], 40.00th=[ 22], 50.00th=[ 25], 60.00th=[ 29], | 70.00th=[ 34], 80.00th=[ 44], 90.00th=[ 57], 95.00th=[ 73], | 99.00th=[ 112], 99.50th=[ 129], 99.90th=[ 207], 99.95th=[ 439], | 99.99th=[ 439] bw ( KiB/s): min= 16, max= 183, per=24.88%, avg=80.10, stdev=41.01, samples=154 iops : min= 4, max= 45, avg=19.64, stdev=10.26, samples=154 lat (msec) : 4=0.23%, 10=17.61%, 20=35.97%, 50=34.96%, 100=10.42% lat (msec) : 250=0.78%, 500=0.03% cpu : usr=0.13%, sys=4.25%, ctx=3436, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=308KiB/s (315kB/s), 76.1KiB/s-77.9KiB/s (77.9kB/s-79.8kB/s), io=23.4MiB (24.6MB), run=77445-78064msec WRITE: bw=322KiB/s (330kB/s), 80.3KiB/s-81.3KiB/s (82.2kB/s-83.3kB/s), io=24.6MiB (25.7MB), run=77445-78064msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=23625: Thu May 1 18:57:32 2025 read: IOPS=25, BW=410KiB/s (420kB/s)(5920KiB/14431msec) clat (usec): min=85, max=106460, avg=20922.71, stdev=16870.03 lat (usec): min=85, max=106461, avg=20924.04, stdev=16870.09 clat percentiles (usec): | 1.00th=[ 90], 5.00th=[ 113], 10.00th=[ 124], 20.00th=[ 6325], | 30.00th=[ 11994], 40.00th=[ 15270], 50.00th=[ 19268], 60.00th=[ 22152], | 70.00th=[ 26346], 80.00th=[ 32113], 90.00th=[ 41157], 95.00th=[ 51119], | 99.00th=[ 82314], 99.50th=[ 98042], 99.90th=[106431], 99.95th=[106431], | 99.99th=[106431] bw ( KiB/s): min= 192, max= 736, per=25.49%, avg=400.89, stdev=117.35, samples=28 iops : min= 12, max= 46, avg=24.46, stdev= 7.45, samples=28 write: IOPS=27, BW=441KiB/s (452kB/s)(6368KiB/14431msec) clat (usec): min=930, max=155119, avg=16237.86, stdev=18722.88 lat (usec): min=931, max=155121, avg=16239.78, stdev=18722.82 clat percentiles (usec): | 1.00th=[ 1029], 5.00th=[ 1188], 10.00th=[ 1270], 20.00th=[ 1893], | 30.00th=[ 3818], 40.00th=[ 7046], 50.00th=[ 10421], 60.00th=[ 14615], | 70.00th=[ 19268], 80.00th=[ 26870], 90.00th=[ 39584], 95.00th=[ 54264], | 99.00th=[100140], 99.50th=[112722], 99.90th=[154141], 99.95th=[154141], | 99.99th=[154141] bw ( KiB/s): min= 128, max= 670, per=26.56%, avg=439.61, stdev=131.50, samples=28 iops : min= 8, max= 41, avg=26.89, stdev= 8.05, samples=28 lat (usec) : 100=1.17%, 250=6.90%, 500=0.26%, 750=0.13%, 1000=0.39% lat (msec) : 2=10.55%, 4=5.60%, 10=11.46%, 20=26.04%, 50=31.90% lat (msec) : 100=5.08%, 250=0.52% cpu : usr=0.08%, sys=9.00%, ctx=1238, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23626: Thu May 1 18:57:32 2025 read: IOPS=24, BW=396KiB/s (405kB/s)(5744KiB/14507msec) clat (usec): min=83, max=97615, avg=20847.69, stdev=16593.96 lat (usec): min=84, max=97616, avg=20849.50, stdev=16593.93 clat percentiles (usec): | 1.00th=[ 91], 5.00th=[ 105], 10.00th=[ 122], 20.00th=[ 8094], | 30.00th=[12256], 40.00th=[15008], 50.00th=[19268], 60.00th=[22938], | 70.00th=[25822], 80.00th=[31327], 90.00th=[40109], 95.00th=[49021], | 99.00th=[78119], 99.50th=[91751], 99.90th=[98042], 99.95th=[98042], | 99.99th=[98042] bw ( KiB/s): min= 159, max= 633, per=24.81%, avg=390.25, stdev=104.66, samples=28 iops : min= 9, max= 39, avg=23.75, stdev= 6.73, samples=28 write: IOPS=28, BW=451KiB/s (462kB/s)(6544KiB/14507msec) clat (usec): min=913, max=107835, avg=17036.78, stdev=18196.54 lat (usec): min=914, max=107837, avg=17038.72, stdev=18196.59 clat percentiles (usec): | 1.00th=[ 1020], 5.00th=[ 1123], 10.00th=[ 1172], 20.00th=[ 1352], | 30.00th=[ 3490], 40.00th=[ 7308], 50.00th=[ 12125], 60.00th=[ 16188], | 70.00th=[ 20841], 80.00th=[ 28443], 90.00th=[ 41681], 95.00th=[ 54789], | 99.00th=[ 80217], 99.50th=[ 85459], 99.90th=[107480], 99.95th=[107480], | 99.99th=[107480] bw ( KiB/s): min= 222, max= 766, per=26.39%, avg=436.68, stdev=122.75, samples=28 iops : min= 13, max= 47, avg=26.68, stdev= 7.75, samples=28 lat (usec) : 100=1.43%, 250=6.25%, 500=0.13%, 750=0.13%, 1000=0.26% lat (msec) : 2=11.85%, 4=5.47%, 10=8.85%, 20=26.30%, 50=33.59% lat (msec) : 100=5.47%, 250=0.26% cpu : usr=0.08%, sys=8.83%, ctx=1236, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23627: Thu May 1 18:57:32 2025 read: IOPS=25, BW=411KiB/s (421kB/s)(6256KiB/15225msec) clat (usec): min=86, max=98438, avg=21121.16, stdev=15409.46 lat (usec): min=87, max=98440, avg=21122.43, stdev=15409.54 clat percentiles (usec): | 1.00th=[ 90], 5.00th=[ 108], 10.00th=[ 125], 20.00th=[ 8455], | 30.00th=[13042], 40.00th=[16450], 50.00th=[20317], 60.00th=[23462], | 70.00th=[27132], 80.00th=[31589], 90.00th=[41157], 95.00th=[50594], | 99.00th=[62129], 99.50th=[69731], 99.90th=[98042], 99.95th=[98042], | 99.99th=[98042] bw ( KiB/s): min= 96, max= 638, per=25.71%, avg=404.47, stdev=133.72, samples=30 iops : min= 6, max= 39, avg=24.90, stdev= 8.29, samples=30 write: IOPS=24, BW=396KiB/s (406kB/s)(6032KiB/15225msec) clat (usec): min=893, max=100483, avg=17731.19, stdev=19121.88 lat (usec): min=894, max=100484, avg=17733.07, stdev=19121.93 clat percentiles (usec): | 1.00th=[ 971], 5.00th=[ 1106], 10.00th=[ 1156], 20.00th=[ 1401], | 30.00th=[ 3654], 40.00th=[ 5735], 50.00th=[ 11338], 60.00th=[ 15533], | 70.00th=[ 23200], 80.00th=[ 32113], 90.00th=[ 45876], 95.00th=[ 55837], | 99.00th=[ 81265], 99.50th=[ 84411], 99.90th=[100140], 99.95th=[100140], | 99.99th=[100140] bw ( KiB/s): min= 128, max= 766, per=23.28%, avg=385.33, stdev=155.16, samples=30 iops : min= 8, max= 47, avg=23.67, stdev= 9.60, samples=30 lat (usec) : 100=1.56%, 250=6.25%, 500=0.39%, 1000=0.78% lat (msec) : 2=10.94%, 4=4.43%, 10=10.94%, 20=21.74%, 50=36.59% lat (msec) : 100=6.25%, 250=0.13% cpu : usr=0.14%, sys=8.66%, ctx=1310, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23628: Thu May 1 18:57:32 2025 read: IOPS=26, BW=431KiB/s (442kB/s)(6032KiB/13985msec) clat (usec): min=74, max=101213, avg=20560.02, stdev=15410.82 lat (usec): min=74, max=101218, avg=20561.28, stdev=15410.92 clat percentiles (usec): | 1.00th=[ 83], 5.00th=[ 102], 10.00th=[ 120], 20.00th=[ 8356], | 30.00th=[ 12911], 40.00th=[ 15139], 50.00th=[ 18744], 60.00th=[ 21365], | 70.00th=[ 26608], 80.00th=[ 32375], 90.00th=[ 39584], 95.00th=[ 46924], | 99.00th=[ 69731], 99.50th=[ 82314], 99.90th=[101188], 99.95th=[101188], | 99.99th=[101188] bw ( KiB/s): min= 128, max= 670, per=26.89%, avg=422.93, stdev=126.84, samples=27 iops : min= 8, max= 41, avg=26.04, stdev= 7.95, samples=27 write: IOPS=27, BW=447KiB/s (458kB/s)(6256KiB/13985msec) clat (usec): min=890, max=147585, avg=15255.86, stdev=17670.98 lat (usec): min=890, max=147587, avg=15257.81, stdev=17670.95 clat percentiles (usec): | 1.00th=[ 1045], 5.00th=[ 1123], 10.00th=[ 1172], 20.00th=[ 1401], | 30.00th=[ 3294], 40.00th=[ 5538], 50.00th=[ 8848], 60.00th=[ 12649], | 70.00th=[ 17957], 80.00th=[ 25822], 90.00th=[ 39584], 95.00th=[ 50070], | 99.00th=[ 76022], 99.50th=[ 86508], 99.90th=[147850], 99.95th=[147850], | 99.99th=[147850] bw ( KiB/s): min= 160, max= 862, per=26.77%, avg=443.07, stdev=153.30, samples=27 iops : min= 10, max= 53, avg=27.30, stdev= 9.49, samples=27 lat (usec) : 100=2.21%, 250=4.82%, 500=0.26%, 1000=0.78% lat (msec) : 2=11.72%, 4=5.86%, 10=13.02%, 20=24.74%, 50=32.29% lat (msec) : 100=4.04%, 250=0.26% cpu : usr=0.08%, sys=8.85%, ctx=1242, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1573KiB/s (1611kB/s), 396KiB/s-431KiB/s (405kB/s-442kB/s), io=23.4MiB (24.5MB), run=13985-15225msec WRITE: bw=1655KiB/s (1695kB/s), 396KiB/s-451KiB/s (406kB/s-462kB/s), io=24.6MiB (25.8MB), run=13985-15225msec rand-rw: (groupid=0, jobs=1): err= 0: pid=23629: Thu May 1 18:57:39 2025 read: IOPS=17, BW=274KiB/s (281kB/s)(5920KiB/21586msec) clat (msec): min=3, max=105, avg=20.90, stdev=18.50 lat (msec): min=3, max=105, avg=20.90, stdev=18.50 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 7], 10.00th=[ 8], 20.00th=[ 10], | 30.00th=[ 11], 40.00th=[ 13], 50.00th=[ 14], 60.00th=[ 17], | 70.00th=[ 21], 80.00th=[ 28], 90.00th=[ 48], 95.00th=[ 65], | 99.00th=[ 97], 99.50th=[ 105], 99.90th=[ 107], 99.95th=[ 107], | 99.99th=[ 107] bw ( KiB/s): min= 31, max= 608, per=25.19%, avg=272.77, stdev=141.71, samples=43 iops : min= 1, max= 38, avg=16.60, stdev= 8.96, samples=43 write: IOPS=18, BW=295KiB/s (302kB/s)(6368KiB/21586msec) clat (msec): min=5, max=159, avg=34.50, stdev=23.39 lat (msec): min=5, max=159, avg=34.51, stdev=23.39 clat percentiles (msec): | 1.00th=[ 8], 5.00th=[ 13], 10.00th=[ 14], 20.00th=[ 18], | 30.00th=[ 21], 40.00th=[ 23], 50.00th=[ 26], 60.00th=[ 31], | 70.00th=[ 39], 80.00th=[ 51], 90.00th=[ 71], 95.00th=[ 87], | 99.00th=[ 107], 99.50th=[ 155], 99.90th=[ 159], 99.95th=[ 159], | 99.99th=[ 159] bw ( KiB/s): min= 64, max= 736, per=25.69%, avg=292.84, stdev=159.92, samples=43 iops : min= 4, max= 46, avg=17.86, stdev=10.09, samples=43 lat (msec) : 4=0.13%, 10=13.54%, 20=34.90%, 50=36.59%, 100=13.80% lat (msec) : 250=1.04% cpu : usr=0.09%, sys=4.14%, ctx=885, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23630: Thu May 1 18:57:39 2025 read: IOPS=16, BW=260KiB/s (266kB/s)(5744KiB/22083msec) clat (msec): min=3, max=117, avg=21.98, stdev=20.63 lat (msec): min=3, max=117, avg=21.98, stdev=20.63 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 7], 10.00th=[ 8], 20.00th=[ 9], | 30.00th=[ 11], 40.00th=[ 12], 50.00th=[ 14], 60.00th=[ 17], | 70.00th=[ 22], 80.00th=[ 32], 90.00th=[ 50], 95.00th=[ 74], | 99.00th=[ 96], 99.50th=[ 113], 99.90th=[ 117], 99.95th=[ 117], | 99.99th=[ 117] bw ( KiB/s): min= 31, max= 672, per=24.30%, avg=263.16, stdev=165.50, samples=43 iops : min= 1, max= 42, avg=15.86, stdev=10.48, samples=43 write: IOPS=18, BW=296KiB/s (303kB/s)(6544KiB/22083msec) clat (msec): min=6, max=188, avg=34.06, stdev=25.19 lat (msec): min=6, max=188, avg=34.06, stdev=25.19 clat percentiles (msec): | 1.00th=[ 9], 5.00th=[ 13], 10.00th=[ 15], 20.00th=[ 18], | 30.00th=[ 20], 40.00th=[ 23], 50.00th=[ 26], 60.00th=[ 30], | 70.00th=[ 36], 80.00th=[ 46], 90.00th=[ 68], 95.00th=[ 88], | 99.00th=[ 130], 99.50th=[ 155], 99.90th=[ 190], 99.95th=[ 190], | 99.99th=[ 190] bw ( KiB/s): min= 95, max= 702, per=25.74%, avg=293.41, stdev=155.28, samples=44 iops : min= 5, max= 43, avg=17.75, stdev= 9.83, samples=44 lat (msec) : 4=0.13%, 10=14.06%, 20=34.51%, 50=37.24%, 100=12.24% lat (msec) : 250=1.82% cpu : usr=0.10%, sys=4.18%, ctx=876, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23631: Thu May 1 18:57:39 2025 read: IOPS=18, BW=288KiB/s (295kB/s)(6256KiB/21714msec) clat (msec): min=4, max=118, avg=20.47, stdev=18.89 lat (msec): min=4, max=118, avg=20.47, stdev=18.89 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 6], 10.00th=[ 7], 20.00th=[ 9], | 30.00th=[ 11], 40.00th=[ 12], 50.00th=[ 14], 60.00th=[ 17], | 70.00th=[ 20], 80.00th=[ 28], 90.00th=[ 44], 95.00th=[ 63], | 99.00th=[ 95], 99.50th=[ 118], 99.90th=[ 120], 99.95th=[ 120], | 99.99th=[ 120] bw ( KiB/s): min= 63, max= 736, per=26.57%, avg=287.79, stdev=180.46, samples=43 iops : min= 3, max= 46, avg=17.47, stdev=11.38, samples=43 write: IOPS=17, BW=278KiB/s (284kB/s)(6032KiB/21714msec) clat (msec): min=8, max=187, avg=35.87, stdev=25.95 lat (msec): min=8, max=187, avg=35.88, stdev=25.95 clat percentiles (msec): | 1.00th=[ 10], 5.00th=[ 12], 10.00th=[ 15], 20.00th=[ 18], | 30.00th=[ 21], 40.00th=[ 23], 50.00th=[ 27], 60.00th=[ 32], | 70.00th=[ 39], 80.00th=[ 52], 90.00th=[ 69], 95.00th=[ 88], | 99.00th=[ 142], 99.50th=[ 159], 99.90th=[ 188], 99.95th=[ 188], | 99.99th=[ 188] bw ( KiB/s): min= 63, max= 638, per=23.87%, avg=272.14, stdev=153.64, samples=43 iops : min= 3, max= 39, avg=16.49, stdev= 9.67, samples=43 lat (msec) : 10=14.19%, 20=36.46%, 50=35.55%, 100=11.85%, 250=1.95% cpu : usr=0.05%, sys=4.26%, ctx=887, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23632: Thu May 1 18:57:39 2025 read: IOPS=17, BW=273KiB/s (280kB/s)(6032KiB/22096msec) clat (msec): min=4, max=126, avg=22.43, stdev=19.94 lat (msec): min=4, max=126, avg=22.43, stdev=19.94 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 7], 10.00th=[ 8], 20.00th=[ 9], | 30.00th=[ 11], 40.00th=[ 13], 50.00th=[ 16], 60.00th=[ 18], | 70.00th=[ 22], 80.00th=[ 32], 90.00th=[ 54], 95.00th=[ 66], | 99.00th=[ 92], 99.50th=[ 120], 99.90th=[ 127], 99.95th=[ 127], | 99.99th=[ 127] bw ( KiB/s): min= 95, max= 638, per=24.82%, avg=268.77, stdev=142.15, samples=44 iops : min= 5, max= 39, avg=16.18, stdev= 9.01, samples=44 write: IOPS=17, BW=283KiB/s (290kB/s)(6256KiB/22096msec) clat (msec): min=8, max=186, avg=34.74, stdev=23.98 lat (msec): min=8, max=186, avg=34.75, stdev=23.98 clat percentiles (msec): | 1.00th=[ 11], 5.00th=[ 14], 10.00th=[ 16], 20.00th=[ 19], | 30.00th=[ 22], 40.00th=[ 24], 50.00th=[ 27], 60.00th=[ 30], | 70.00th=[ 37], 80.00th=[ 48], 90.00th=[ 66], 95.00th=[ 86], | 99.00th=[ 122], 99.50th=[ 176], 99.90th=[ 186], 99.95th=[ 186], | 99.99th=[ 186] bw ( KiB/s): min= 95, max= 638, per=24.65%, avg=281.00, stdev=155.48, samples=44 iops : min= 5, max= 39, avg=16.95, stdev= 9.84, samples=44 lat (msec) : 10=13.54%, 20=31.90%, 50=39.97%, 100=13.41%, 250=1.17% cpu : usr=0.13%, sys=4.07%, ctx=882, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1084KiB/s (1110kB/s), 260KiB/s-288KiB/s (266kB/s-295kB/s), io=23.4MiB (24.5MB), run=21586-22096msec WRITE: bw=1140KiB/s (1168kB/s), 278KiB/s-296KiB/s (284kB/s-303kB/s), io=24.6MiB (25.8MB), run=21586-22096msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=23655: Thu May 1 18:57:44 2025 read: IOPS=4, BW=4334KiB/s (4438kB/s)(7168KiB/1654msec) clat (usec): min=401, max=119642, avg=71136.10, stdev=50556.87 lat (usec): min=402, max=119643, avg=71137.45, stdev=50556.78 clat percentiles (usec): | 1.00th=[ 400], 5.00th=[ 400], 10.00th=[ 400], 20.00th=[ 437], | 30.00th=[ 73925], 40.00th=[ 73925], 50.00th=[ 87557], 60.00th=[106431], | 70.00th=[106431], 80.00th=[109577], 90.00th=[120062], 95.00th=[120062], | 99.00th=[120062], 99.50th=[120062], 99.90th=[120062], 99.95th=[120062], | 99.99th=[120062] bw ( KiB/s): min= 1976, max=10219, per=66.96%, avg=6097.50, stdev=5828.68, samples=2 iops : min= 1, max= 9, avg= 5.00, stdev= 5.66, samples=2 write: IOPS=3, BW=3096KiB/s (3170kB/s)(5120KiB/1654msec) clat (msec): min=46, max=672, avg=205.01, stdev=262.80 lat (msec): min=46, max=672, avg=205.06, stdev=262.82 clat percentiles (msec): | 1.00th=[ 47], 5.00th=[ 47], 10.00th=[ 47], 20.00th=[ 47], | 30.00th=[ 84], 40.00th=[ 84], 50.00th=[ 105], 60.00th=[ 105], | 70.00th=[ 120], 80.00th=[ 120], 90.00th=[ 676], 95.00th=[ 676], | 99.00th=[ 676], 99.50th=[ 676], 99.90th=[ 676], 99.95th=[ 676], | 99.99th=[ 676] bw ( KiB/s): min= 2043, max= 3953, per=21.03%, avg=2681.33, stdev=1101.30, samples=3 iops : min= 1, max= 3, avg= 2.00, stdev= 1.00, samples=3 lat (usec) : 500=16.67% lat (msec) : 50=8.33%, 100=25.00%, 250=41.67%, 750=8.33% cpu : usr=0.00%, sys=11.49%, ctx=129, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23656: Thu May 1 18:57:44 2025 read: IOPS=0, BW=911KiB/s (932kB/s)(2048KiB/2249msec) clat (msec): min=146, max=534, avg=340.62, stdev=273.93 lat (msec): min=146, max=534, avg=340.63, stdev=273.93 clat percentiles (msec): | 1.00th=[ 148], 5.00th=[ 148], 10.00th=[ 148], 20.00th=[ 148], | 30.00th=[ 148], 40.00th=[ 148], 50.00th=[ 148], 60.00th=[ 535], | 70.00th=[ 535], 80.00th=[ 535], 90.00th=[ 535], 95.00th=[ 535], | 99.00th=[ 535], 99.50th=[ 535], 99.90th=[ 535], 99.95th=[ 535], | 99.99th=[ 535] bw ( KiB/s): min= 1442, max= 2048, per=19.16%, avg=1745.00, stdev=428.51, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 write: IOPS=4, BW=4553KiB/s (4662kB/s)(10.0MiB/2249msec) clat (msec): min=38, max=616, avg=152.66, stdev=166.46 lat (msec): min=38, max=616, avg=152.74, stdev=166.46 clat percentiles (msec): | 1.00th=[ 39], 5.00th=[ 39], 10.00th=[ 39], 20.00th=[ 73], | 30.00th=[ 78], 40.00th=[ 100], 50.00th=[ 101], 60.00th=[ 112], | 70.00th=[ 144], 80.00th=[ 144], 90.00th=[ 153], 95.00th=[ 617], | 99.00th=[ 617], 99.50th=[ 617], 99.90th=[ 617], 99.95th=[ 617], | 99.99th=[ 617] bw ( KiB/s): min= 2048, max= 7211, per=40.15%, avg=5118.00, stdev=2716.62, samples=3 iops : min= 2, max= 7, avg= 4.67, stdev= 2.52, samples=3 lat (msec) : 50=8.33%, 100=33.33%, 250=41.67%, 750=16.67% cpu : usr=0.00%, sys=15.78%, ctx=116, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23657: Thu May 1 18:57:44 2025 read: IOPS=3, BW=3633KiB/s (3721kB/s)(6144KiB/1691msec) clat (usec): min=540, max=250649, avg=103606.69, stdev=102308.54 lat (usec): min=542, max=250651, avg=103608.16, stdev=102308.24 clat percentiles (usec): | 1.00th=[ 545], 5.00th=[ 545], 10.00th=[ 545], 20.00th=[ 578], | 30.00th=[ 578], 40.00th=[ 49546], 50.00th=[ 49546], 60.00th=[152044], | 70.00th=[168821], 80.00th=[168821], 90.00th=[250610], 95.00th=[250610], | 99.00th=[250610], 99.50th=[250610], 99.90th=[250610], 99.95th=[250610], | 99.99th=[250610] bw ( KiB/s): min= 4096, max= 6119, per=56.09%, avg=5107.50, stdev=1430.48, samples=2 iops : min= 4, max= 5, avg= 4.50, stdev= 0.71, samples=2 write: IOPS=3, BW=3633KiB/s (3721kB/s)(6144KiB/1691msec) clat (msec): min=50, max=505, avg=168.18, stdev=168.95 lat (msec): min=51, max=505, avg=168.24, stdev=168.94 clat percentiles (msec): | 1.00th=[ 52], 5.00th=[ 52], 10.00th=[ 52], 20.00th=[ 88], | 30.00th=[ 88], 40.00th=[ 97], 50.00th=[ 97], 60.00th=[ 109], | 70.00th=[ 159], 80.00th=[ 159], 90.00th=[ 506], 95.00th=[ 506], | 99.00th=[ 506], 99.50th=[ 506], 99.90th=[ 506], 99.95th=[ 506], | 99.99th=[ 506] bw ( KiB/s): min= 2011, max= 6144, per=31.99%, avg=4078.00, stdev=2066.50, samples=3 iops : min= 1, max= 6, avg= 3.33, stdev= 2.52, samples=3 lat (usec) : 750=16.67% lat (msec) : 50=8.33%, 100=25.00%, 250=33.33%, 500=8.33%, 750=8.33% cpu : usr=0.00%, sys=12.60%, ctx=330, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23658: Thu May 1 18:57:44 2025 read: IOPS=2, BW=3001KiB/s (3073kB/s)(5120KiB/1706msec) clat (usec): min=547, max=364013, avg=118408.44, stdev=146279.81 lat (usec): min=549, max=364014, avg=118409.96, stdev=146279.34 clat percentiles (usec): | 1.00th=[ 545], 5.00th=[ 545], 10.00th=[ 545], 20.00th=[ 545], | 30.00th=[ 27657], 40.00th=[ 27657], 50.00th=[ 65274], 60.00th=[ 65274], | 70.00th=[135267], 80.00th=[135267], 90.00th=[362808], 95.00th=[362808], | 99.00th=[362808], 99.50th=[362808], 99.90th=[362808], 99.95th=[362808], | 99.99th=[362808] bw ( KiB/s): min= 2027, max= 6131, per=44.79%, avg=4079.00, stdev=2901.97, samples=2 iops : min= 1, max= 5, avg= 3.00, stdev= 2.83, samples=2 write: IOPS=4, BW=4202KiB/s (4302kB/s)(7168KiB/1706msec) clat (msec): min=27, max=372, avg=156.95, stdev=149.26 lat (msec): min=27, max=372, avg=157.12, stdev=149.38 clat percentiles (msec): | 1.00th=[ 28], 5.00th=[ 28], 10.00th=[ 28], 20.00th=[ 58], | 30.00th=[ 58], 40.00th=[ 58], 50.00th=[ 99], 60.00th=[ 116], | 70.00th=[ 116], 80.00th=[ 372], 90.00th=[ 372], 95.00th=[ 372], | 99.00th=[ 372], 99.50th=[ 372], 99.90th=[ 372], 99.95th=[ 372], | 99.99th=[ 372] bw ( KiB/s): min= 3961, max= 8110, per=47.34%, avg=6035.50, stdev=2933.79, samples=2 iops : min= 3, max= 7, avg= 5.00, stdev= 2.83, samples=2 lat (usec) : 750=8.33% lat (msec) : 50=16.67%, 100=33.33%, 250=16.67%, 500=25.00% cpu : usr=0.06%, sys=12.67%, ctx=86, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=9106KiB/s (9325kB/s), 911KiB/s-4334KiB/s (932kB/s-4438kB/s), io=20.0MiB (20.0MB), run=1654-2249msec WRITE: bw=12.4MiB/s (13.1MB/s), 3096KiB/s-4553KiB/s (3170kB/s-4662kB/s), io=28.0MiB (29.4MB), run=1654-2249msec rand-rw: (groupid=0, jobs=1): err= 0: pid=23660: Thu May 1 18:57:44 2025 read: IOPS=2, BW=2683KiB/s (2747kB/s)(7168KiB/2672msec) clat (msec): min=44, max=740, avg=268.68, stdev=298.36 lat (msec): min=44, max=740, avg=268.68, stdev=298.36 clat percentiles (msec): | 1.00th=[ 44], 5.00th=[ 44], 10.00th=[ 44], 20.00th=[ 56], | 30.00th=[ 58], 40.00th=[ 58], 50.00th=[ 150], 60.00th=[ 178], | 70.00th=[ 178], 80.00th=[ 659], 90.00th=[ 743], 95.00th=[ 743], | 99.00th=[ 743], 99.50th=[ 743], 99.90th=[ 743], 99.95th=[ 743], | 99.99th=[ 743] bw ( KiB/s): min= 2043, max= 6035, per=55.03%, avg=4058.00, stdev=1996.27, samples=3 iops : min= 1, max= 5, avg= 3.33, stdev= 2.08, samples=3 write: IOPS=1, BW=1916KiB/s (1962kB/s)(5120KiB/2672msec) clat (msec): min=41, max=219, avg=108.21, stdev=74.70 lat (msec): min=41, max=219, avg=108.29, stdev=74.71 clat percentiles (msec): | 1.00th=[ 42], 5.00th=[ 42], 10.00th=[ 42], 20.00th=[ 42], | 30.00th=[ 48], 40.00th=[ 48], 50.00th=[ 87], 60.00th=[ 87], | 70.00th=[ 146], 80.00th=[ 146], 90.00th=[ 220], 95.00th=[ 220], | 99.00th=[ 220], 99.50th=[ 220], 99.90th=[ 220], 99.95th=[ 220], | 99.99th=[ 220] bw ( KiB/s): min= 2011, max= 6144, per=39.50%, avg=4077.50, stdev=2922.47, samples=2 iops : min= 1, max= 6, avg= 3.50, stdev= 3.54, samples=2 lat (msec) : 50=25.00%, 100=25.00%, 250=33.33%, 750=16.67% cpu : usr=0.04%, sys=4.68%, ctx=58, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23661: Thu May 1 18:57:44 2025 read: IOPS=0, BW=743KiB/s (760kB/s)(2048KiB/2758msec) clat (msec): min=44, max=729, avg=386.64, stdev=484.42 lat (msec): min=44, max=729, avg=386.65, stdev=484.42 clat percentiles (msec): | 1.00th=[ 45], 5.00th=[ 45], 10.00th=[ 45], 20.00th=[ 45], | 30.00th=[ 45], 40.00th=[ 45], 50.00th=[ 45], 60.00th=[ 726], | 70.00th=[ 726], 80.00th=[ 726], 90.00th=[ 726], 95.00th=[ 726], | 99.00th=[ 726], 99.50th=[ 726], 99.90th=[ 726], 99.95th=[ 726], | 99.99th=[ 726] bw ( KiB/s): min= 2039, max= 2043, per=27.68%, avg=2041.00, stdev= 2.83, samples=2 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=2 write: IOPS=3, BW=3713KiB/s (3802kB/s)(10.0MiB/2758msec) clat (msec): min=47, max=480, avg=174.56, stdev=130.76 lat (msec): min=47, max=480, avg=174.66, stdev=130.76 clat percentiles (msec): | 1.00th=[ 47], 5.00th=[ 47], 10.00th=[ 47], 20.00th=[ 50], | 30.00th=[ 83], 40.00th=[ 84], 50.00th=[ 140], 60.00th=[ 184], | 70.00th=[ 220], 80.00th=[ 220], 90.00th=[ 262], 95.00th=[ 481], | 99.00th=[ 481], 99.50th=[ 481], 99.90th=[ 481], 99.95th=[ 481], | 99.99th=[ 481] bw ( KiB/s): min= 2048, max= 6131, per=46.16%, avg=4766.00, stdev=2353.86, samples=3 iops : min= 2, max= 5, avg= 4.00, stdev= 1.73, samples=3 lat (msec) : 50=25.00%, 100=16.67%, 250=33.33%, 500=16.67%, 750=8.33% cpu : usr=0.07%, sys=5.48%, ctx=63, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23662: Thu May 1 18:57:44 2025 read: IOPS=2, BW=2240KiB/s (2294kB/s)(6144KiB/2743msec) clat (msec): min=46, max=304, avg=125.51, stdev=95.28 lat (msec): min=46, max=304, avg=125.51, stdev=95.28 clat percentiles (msec): | 1.00th=[ 46], 5.00th=[ 46], 10.00th=[ 46], 20.00th=[ 58], | 30.00th=[ 58], 40.00th=[ 78], 50.00th=[ 78], 60.00th=[ 128], | 70.00th=[ 140], 80.00th=[ 140], 90.00th=[ 305], 95.00th=[ 305], | 99.00th=[ 305], 99.50th=[ 305], 99.90th=[ 305], 99.95th=[ 305], | 99.99th=[ 305] bw ( KiB/s): min= 2048, max= 6119, per=55.38%, avg=4083.50, stdev=2878.63, samples=2 iops : min= 2, max= 5, avg= 3.50, stdev= 2.12, samples=2 write: IOPS=2, BW=2240KiB/s (2294kB/s)(6144KiB/2743msec) clat (msec): min=57, max=817, avg=298.65, stdev=301.61 lat (msec): min=57, max=817, avg=298.72, stdev=301.59 clat percentiles (msec): | 1.00th=[ 57], 5.00th=[ 57], 10.00th=[ 57], 20.00th=[ 72], | 30.00th=[ 72], 40.00th=[ 100], 50.00th=[ 100], 60.00th=[ 262], | 70.00th=[ 485], 80.00th=[ 485], 90.00th=[ 818], 95.00th=[ 818], | 99.00th=[ 818], 99.50th=[ 818], 99.90th=[ 818], 99.95th=[ 818], | 99.99th=[ 818] bw ( KiB/s): min= 2048, max= 4079, per=32.95%, avg=3402.00, stdev=1172.60, samples=3 iops : min= 2, max= 3, avg= 2.67, stdev= 0.58, samples=3 lat (msec) : 50=8.33%, 100=41.67%, 250=16.67%, 500=25.00%, 1000=8.33% cpu : usr=0.04%, sys=3.17%, ctx=49, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23663: Thu May 1 18:57:44 2025 read: IOPS=1, BW=1844KiB/s (1888kB/s)(5120KiB/2777msec) clat (msec): min=41, max=163, avg=98.11, stdev=55.41 lat (msec): min=41, max=163, avg=98.11, stdev=55.41 clat percentiles (msec): | 1.00th=[ 42], 5.00th=[ 42], 10.00th=[ 42], 20.00th=[ 42], | 30.00th=[ 46], 40.00th=[ 46], 50.00th=[ 96], 60.00th=[ 96], | 70.00th=[ 144], 80.00th=[ 144], 90.00th=[ 163], 95.00th=[ 163], | 99.00th=[ 163], 99.50th=[ 163], 99.90th=[ 163], 99.95th=[ 163], | 99.99th=[ 163] bw ( KiB/s): min= 8159, max= 8159, per=100.00%, avg=8159.00, stdev= 0.00, samples=1 iops : min= 7, max= 7, avg= 7.00, stdev= 0.00, samples=1 write: IOPS=2, BW=2581KiB/s (2643kB/s)(7168KiB/2777msec) clat (msec): min=61, max=902, avg=311.16, stdev=295.35 lat (msec): min=61, max=902, avg=311.27, stdev=295.37 clat percentiles (msec): | 1.00th=[ 62], 5.00th=[ 62], 10.00th=[ 62], 20.00th=[ 68], | 30.00th=[ 190], 40.00th=[ 190], 50.00th=[ 218], 60.00th=[ 268], | 70.00th=[ 268], 80.00th=[ 477], 90.00th=[ 902], 95.00th=[ 902], | 99.00th=[ 902], 99.50th=[ 902], 99.90th=[ 902], 99.95th=[ 902], | 99.99th=[ 902] bw ( KiB/s): min= 4071, max= 4096, per=39.54%, avg=4082.00, stdev=12.77, samples=3 iops : min= 3, max= 4, avg= 3.33, stdev= 0.58, samples=3 lat (msec) : 50=16.67%, 100=25.00%, 250=33.33%, 500=16.67%, 1000=8.33% cpu : usr=0.14%, sys=4.72%, ctx=70, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=7375KiB/s (7552kB/s), 743KiB/s-2683KiB/s (760kB/s-2747kB/s), io=20.0MiB (20.0MB), run=2672-2777msec WRITE: bw=10.1MiB/s (10.6MB/s), 1916KiB/s-3713KiB/s (1962kB/s-3802kB/s), io=28.0MiB (29.4MB), run=2672-2777msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=23671: Thu May 1 18:57:49 2025 read: IOPS=2, BW=8225KiB/s (8422kB/s)(12.0MiB/1494msec) clat (msec): min=166, max=795, avg=487.90, stdev=314.65 lat (msec): min=166, max=795, avg=487.91, stdev=314.65 clat percentiles (msec): | 1.00th=[ 167], 5.00th=[ 167], 10.00th=[ 167], 20.00th=[ 167], | 30.00th=[ 167], 40.00th=[ 502], 50.00th=[ 502], 60.00th=[ 502], | 70.00th=[ 793], 80.00th=[ 793], 90.00th=[ 793], 95.00th=[ 793], | 99.00th=[ 793], 99.50th=[ 793], 99.90th=[ 793], 99.95th=[ 793], | 99.99th=[ 793] bw ( KiB/s): min=16384, max=16384, per=100.00%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 750=33.33%, 1000=33.33% cpu : usr=0.00%, sys=9.24%, ctx=108, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23672: Thu May 1 18:57:49 2025 read: IOPS=0, BW=1736KiB/s (1777kB/s)(4096KiB/2360msec) clat (nsec): min=566152k, max=566152k, avg=566151894.00, stdev= 0.00 lat (nsec): min=566155k, max=566155k, avg=566154880.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 567], 5.00th=[ 567], 10.00th=[ 567], 20.00th=[ 567], | 30.00th=[ 567], 40.00th=[ 567], 50.00th=[ 567], 60.00th=[ 567], | 70.00th=[ 567], 80.00th=[ 567], 90.00th=[ 567], 95.00th=[ 567], | 99.00th=[ 567], 99.50th=[ 567], 99.90th=[ 567], 99.95th=[ 567], | 99.99th=[ 567] bw ( KiB/s): min= 8159, max= 8159, per=100.00%, avg=8159.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=0, BW=3471KiB/s (3554kB/s)(8192KiB/2360msec) clat (msec): min=738, max=1033, avg=885.76, stdev=208.26 lat (msec): min=738, max=1038, avg=888.77, stdev=211.92 clat percentiles (msec): | 1.00th=[ 743], 5.00th=[ 743], 10.00th=[ 743], 20.00th=[ 743], | 30.00th=[ 743], 40.00th=[ 743], 50.00th=[ 743], 60.00th=[ 1036], | 70.00th=[ 1036], 80.00th=[ 1036], 90.00th=[ 1036], 95.00th=[ 1036], | 99.00th=[ 1036], 99.50th=[ 1036], 99.90th=[ 1036], 99.95th=[ 1036], | 99.99th=[ 1036] bw ( KiB/s): min= 6282, max= 6282, per=47.39%, avg=6282.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 750=66.67% cpu : usr=0.08%, sys=10.17%, ctx=83, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23673: Thu May 1 18:57:49 2025 write: IOPS=1, BW=4971KiB/s (5090kB/s)(12.0MiB/2472msec) clat (msec): min=648, max=1073, avg=811.22, stdev=229.27 lat (msec): min=648, max=1073, avg=816.13, stdev=226.37 clat percentiles (msec): | 1.00th=[ 651], 5.00th=[ 651], 10.00th=[ 651], 20.00th=[ 651], | 30.00th=[ 651], 40.00th=[ 709], 50.00th=[ 709], 60.00th=[ 709], | 70.00th=[ 1070], 80.00th=[ 1070], 90.00th=[ 1070], 95.00th=[ 1070], | 99.00th=[ 1070], 99.50th=[ 1070], 99.90th=[ 1070], 99.95th=[ 1070], | 99.99th=[ 1070] bw ( KiB/s): min= 8078, max= 8143, per=61.19%, avg=8110.50, stdev=45.96, samples=2 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=2 lat (msec) : 750=66.67% cpu : usr=0.00%, sys=11.57%, ctx=88, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23674: Thu May 1 18:57:49 2025 write: IOPS=1, BW=6132KiB/s (6279kB/s)(12.0MiB/2004msec) clat (msec): min=396, max=1099, avg=652.58, stdev=388.12 lat (msec): min=396, max=1099, avg=653.03, stdev=388.12 clat percentiles (msec): | 1.00th=[ 397], 5.00th=[ 397], 10.00th=[ 397], 20.00th=[ 397], | 30.00th=[ 397], 40.00th=[ 464], 50.00th=[ 464], 60.00th=[ 464], | 70.00th=[ 1099], 80.00th=[ 1099], 90.00th=[ 1099], 95.00th=[ 1099], | 99.00th=[ 1099], 99.50th=[ 1099], 99.90th=[ 1099], 99.95th=[ 1099], | 99.99th=[ 1099] bw ( KiB/s): min=16351, max=16351, per=100.00%, avg=16351.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 lat (msec) : 500=66.67% cpu : usr=0.00%, sys=15.98%, ctx=58, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=6942KiB/s (7109kB/s), 1736KiB/s-8225KiB/s (1777kB/s-8422kB/s), io=16.0MiB (16.8MB), run=1494-2360msec WRITE: bw=12.9MiB/s (13.6MB/s), 3471KiB/s-6132KiB/s (3554kB/s-6279kB/s), io=32.0MiB (33.6MB), run=2004-2472msec rand-rw: (groupid=0, jobs=1): err= 0: pid=23676: Thu May 1 18:57:50 2025 read: IOPS=1, BW=4496KiB/s (4604kB/s)(12.0MiB/2733msec) clat (msec): min=788, max=1041, avg=879.54, stdev=140.24 lat (msec): min=788, max=1041, avg=879.54, stdev=140.24 clat percentiles (msec): | 1.00th=[ 793], 5.00th=[ 793], 10.00th=[ 793], 20.00th=[ 793], | 30.00th=[ 793], 40.00th=[ 810], 50.00th=[ 810], 60.00th=[ 810], | 70.00th=[ 1045], 80.00th=[ 1045], 90.00th=[ 1045], 95.00th=[ 1045], | 99.00th=[ 1045], 99.50th=[ 1045], 99.90th=[ 1045], 99.95th=[ 1045], | 99.99th=[ 1045] bw ( KiB/s): min= 8094, max= 8159, per=100.00%, avg=8126.50, stdev=45.96, samples=2 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=2 lat (msec) : 1000=66.67% cpu : usr=0.11%, sys=2.49%, ctx=34, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23678: Thu May 1 18:57:50 2025 read: IOPS=0, BW=1557KiB/s (1595kB/s)(4096KiB/2630msec) clat (nsec): min=992798k, max=992798k, avg=992798138.00, stdev= 0.00 lat (nsec): min=992801k, max=992801k, avg=992800672.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 995], 5.00th=[ 995], 10.00th=[ 995], 20.00th=[ 995], | 30.00th=[ 995], 40.00th=[ 995], 50.00th=[ 995], 60.00th=[ 995], | 70.00th=[ 995], 80.00th=[ 995], 90.00th=[ 995], 95.00th=[ 995], | 99.00th=[ 995], 99.50th=[ 995], 99.90th=[ 995], 99.95th=[ 995], | 99.99th=[ 995] bw ( KiB/s): min= 8110, max= 8110, per=100.00%, avg=8110.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=0, BW=3115KiB/s (3190kB/s)(8192KiB/2630msec) clat (msec): min=251, max=1354, avg=802.85, stdev=779.47 lat (msec): min=251, max=1373, avg=812.82, stdev=793.25 clat percentiles (msec): | 1.00th=[ 253], 5.00th=[ 253], 10.00th=[ 253], 20.00th=[ 253], | 30.00th=[ 253], 40.00th=[ 253], 50.00th=[ 253], 60.00th=[ 1351], | 70.00th=[ 1351], 80.00th=[ 1351], 90.00th=[ 1351], 95.00th=[ 1351], | 99.00th=[ 1351], 99.50th=[ 1351], 99.90th=[ 1351], 99.95th=[ 1351], | 99.99th=[ 1351] bw ( KiB/s): min= 8159, max= 8159, per=67.83%, avg=8159.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 500=33.33%, 1000=33.33% cpu : usr=0.08%, sys=3.77%, ctx=39, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23679: Thu May 1 18:57:50 2025 write: IOPS=1, BW=4614KiB/s (4725kB/s)(12.0MiB/2663msec) clat (msec): min=748, max=1113, avg=880.62, stdev=201.86 lat (msec): min=749, max=1113, avg=881.12, stdev=201.80 clat percentiles (msec): | 1.00th=[ 751], 5.00th=[ 751], 10.00th=[ 751], 20.00th=[ 751], | 30.00th=[ 751], 40.00th=[ 776], 50.00th=[ 776], 60.00th=[ 776], | 70.00th=[ 1116], 80.00th=[ 1116], 90.00th=[ 1116], 95.00th=[ 1116], | 99.00th=[ 1116], 99.50th=[ 1116], 99.90th=[ 1116], 99.95th=[ 1116], | 99.99th=[ 1116] bw ( KiB/s): min= 8175, max= 8192, per=68.03%, avg=8183.50, stdev=12.02, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 750=33.33%, 1000=33.33% cpu : usr=0.34%, sys=5.18%, ctx=46, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23680: Thu May 1 18:57:50 2025 write: IOPS=1, BW=4511KiB/s (4619kB/s)(12.0MiB/2724msec) clat (msec): min=183, max=1624, avg=875.55, stdev=721.90 lat (msec): min=184, max=1624, avg=875.94, stdev=721.95 clat percentiles (msec): | 1.00th=[ 184], 5.00th=[ 184], 10.00th=[ 184], 20.00th=[ 184], | 30.00th=[ 184], 40.00th=[ 818], 50.00th=[ 818], 60.00th=[ 818], | 70.00th=[ 1620], 80.00th=[ 1620], 90.00th=[ 1620], 95.00th=[ 1620], | 99.00th=[ 1620], 99.50th=[ 1620], 99.90th=[ 1620], 99.95th=[ 1620], | 99.99th=[ 1620] bw ( KiB/s): min= 8175, max= 8175, per=67.96%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 1000=33.33% cpu : usr=0.00%, sys=7.05%, ctx=76, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=5995KiB/s (6139kB/s), 1557KiB/s-4496KiB/s (1595kB/s-4604kB/s), io=16.0MiB (16.8MB), run=2630-2733msec WRITE: bw=11.7MiB/s (12.3MB/s), 3115KiB/s-4614KiB/s (3190kB/s-4725kB/s), io=32.0MiB (33.6MB), run=2630-2724msec