== sanity test 398b: DIO and buffer IO race ============== 01:49:28 (1746164968) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 4.27186 s, 11.8 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=23473: Fri May 2 01:50:44 2025 read: IOPS=21, BW=87.4KiB/s (89.5kB/s)(5992KiB/68571msec) clat (usec): min=77, max=680187, avg=26107.63, stdev=24306.26 lat (usec): min=78, max=680188, avg=26112.44, stdev=24309.74 clat percentiles (usec): | 1.00th=[ 109], 5.00th=[ 139], 10.00th=[ 8455], 20.00th=[ 13042], | 30.00th=[ 16188], 40.00th=[ 19268], 50.00th=[ 22414], 60.00th=[ 26346], | 70.00th=[ 30540], 80.00th=[ 36439], 90.00th=[ 46400], 95.00th=[ 58459], | 99.00th=[ 93848], 99.50th=[ 99091], 99.90th=[119014], 99.95th=[683672], | 99.99th=[683672] bw ( KiB/s): min= 7, max= 241, per=24.92%, avg=87.24, stdev=32.05, samples=136 iops : min= 1, max= 60, avg=21.39, stdev= 8.04, samples=136 write: IOPS=22, BW=91.8KiB/s (94.0kB/s)(6296KiB/68571msec) clat (usec): min=639, max=149052, avg=18546.81, stdev=19488.62 lat (usec): min=640, max=149054, avg=18551.82, stdev=19486.64 clat percentiles (usec): | 1.00th=[ 775], 5.00th=[ 857], 10.00th=[ 979], 20.00th=[ 1319], | 30.00th=[ 3261], 40.00th=[ 7701], 50.00th=[ 13960], 60.00th=[ 18482], | 70.00th=[ 23462], 80.00th=[ 32113], 90.00th=[ 45351], 95.00th=[ 57410], | 99.00th=[ 84411], 99.50th=[ 94897], 99.90th=[119014], 99.95th=[149947], | 99.99th=[149947] bw ( KiB/s): min= 16, max= 241, per=25.02%, avg=91.59, stdev=33.22, samples=136 iops : min= 4, max= 60, avg=22.48, stdev= 8.30, samples=136 lat (usec) : 100=0.20%, 250=3.03%, 500=0.49%, 750=0.39%, 1000=5.14% lat (msec) : 2=6.67%, 4=4.69%, 10=7.26%, 20=25.07%, 50=39.10% lat (msec) : 100=7.58%, 250=0.36%, 750=0.03% cpu : usr=0.11%, sys=8.42%, ctx=5295, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23474: Fri May 2 01:50:44 2025 read: IOPS=21, BW=86.9KiB/s (88.9kB/s)(5940KiB/68387msec) clat (usec): min=81, max=137499, avg=25147.20, stdev=16698.74 lat (usec): min=82, max=137501, avg=25151.66, stdev=16698.10 clat percentiles (usec): | 1.00th=[ 104], 5.00th=[ 157], 10.00th=[ 9503], 20.00th=[ 12780], | 30.00th=[ 15664], 40.00th=[ 18744], 50.00th=[ 22414], 60.00th=[ 25822], | 70.00th=[ 30278], 80.00th=[ 35390], 90.00th=[ 44827], 95.00th=[ 55837], | 99.00th=[ 81265], 99.50th=[ 92799], 99.90th=[133694], 99.95th=[137364], | 99.99th=[137364] bw ( KiB/s): min= 23, max= 192, per=24.72%, avg=86.53, stdev=28.01, samples=135 iops : min= 5, max= 48, avg=21.20, stdev= 7.04, samples=135 write: IOPS=23, BW=92.8KiB/s (95.1kB/s)(6348KiB/68387msec) clat (usec): min=612, max=554678, avg=19400.99, stdev=25021.93 lat (usec): min=613, max=554678, avg=19403.05, stdev=25021.95 clat percentiles (usec): | 1.00th=[ 750], 5.00th=[ 857], 10.00th=[ 979], 20.00th=[ 1287], | 30.00th=[ 2999], 40.00th=[ 6390], 50.00th=[ 13173], 60.00th=[ 18220], | 70.00th=[ 24511], 80.00th=[ 33817], 90.00th=[ 48497], 95.00th=[ 62653], | 99.00th=[ 94897], 99.50th=[109577], 99.90th=[147850], 99.95th=[557843], | 99.99th=[557843] bw ( KiB/s): min= 7, max= 272, per=24.96%, avg=91.34, stdev=35.80, samples=136 iops : min= 1, max= 68, avg=22.40, stdev= 9.00, samples=136 lat (usec) : 100=0.33%, 250=2.31%, 500=0.36%, 750=0.52%, 1000=5.01% lat (msec) : 2=8.27%, 4=4.17%, 10=7.62%, 20=25.13%, 50=38.05% lat (msec) : 100=7.65%, 250=0.55%, 750=0.03% cpu : usr=0.12%, sys=8.31%, ctx=5192, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23475: Fri May 2 01:50:44 2025 read: IOPS=22, BW=88.3KiB/s (90.4kB/s)(6028KiB/68269msec) clat (usec): min=80, max=171777, avg=25117.07, stdev=16717.95 lat (usec): min=80, max=171778, avg=25121.08, stdev=16719.64 clat percentiles (usec): | 1.00th=[ 101], 5.00th=[ 141], 10.00th=[ 8029], 20.00th=[ 12518], | 30.00th=[ 15664], 40.00th=[ 18744], 50.00th=[ 21890], 60.00th=[ 26346], | 70.00th=[ 30540], 80.00th=[ 36963], 90.00th=[ 46400], 95.00th=[ 54264], | 99.00th=[ 77071], 99.50th=[ 91751], 99.90th=[109577], 99.95th=[170918], | 99.99th=[170918] bw ( KiB/s): min= 7, max= 176, per=24.94%, avg=87.29, stdev=28.18, samples=136 iops : min= 1, max= 44, avg=21.35, stdev= 7.11, samples=136 write: IOPS=22, BW=91.7KiB/s (93.9kB/s)(6260KiB/68269msec) clat (usec): min=635, max=554948, avg=19278.25, stdev=24792.50 lat (usec): min=635, max=554949, avg=19284.26, stdev=24791.42 clat percentiles (usec): | 1.00th=[ 783], 5.00th=[ 865], 10.00th=[ 971], 20.00th=[ 1336], | 30.00th=[ 3195], 40.00th=[ 6325], 50.00th=[ 13042], 60.00th=[ 18482], | 70.00th=[ 24773], 80.00th=[ 33817], 90.00th=[ 46924], 95.00th=[ 60031], | 99.00th=[ 86508], 99.50th=[107480], 99.90th=[179307], 99.95th=[557843], | 99.99th=[557843] bw ( KiB/s): min= 23, max= 192, per=24.75%, avg=90.57, stdev=30.32, samples=136 iops : min= 5, max= 48, avg=22.18, stdev= 7.64, samples=136 lat (usec) : 100=0.46%, 250=2.47%, 500=0.36%, 750=0.23%, 1000=5.66% lat (msec) : 2=7.10%, 4=4.56%, 10=8.66%, 20=23.86%, 50=38.80% lat (msec) : 100=7.39%, 250=0.42%, 750=0.03% cpu : usr=0.15%, sys=8.48%, ctx=5328, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23476: Fri May 2 01:50:44 2025 read: IOPS=22, BW=88.8KiB/s (90.0kB/s)(6052KiB/68138msec) clat (usec): min=70, max=567755, avg=25888.83, stdev=22364.83 lat (usec): min=71, max=567755, avg=25890.92, stdev=22364.36 clat percentiles (usec): | 1.00th=[ 103], 5.00th=[ 159], 10.00th=[ 9372], 20.00th=[ 13173], | 30.00th=[ 16188], 40.00th=[ 19006], 50.00th=[ 21890], 60.00th=[ 25822], | 70.00th=[ 30278], 80.00th=[ 36439], 90.00th=[ 46924], 95.00th=[ 55837], | 99.00th=[ 85459], 99.50th=[105382], 99.90th=[214959], 99.95th=[566232], | 99.99th=[566232] bw ( KiB/s): min= 16, max= 167, per=25.14%, avg=87.99, stdev=26.75, samples=136 iops : min= 4, max= 41, avg=21.54, stdev= 6.69, samples=136 write: IOPS=22, BW=91.5KiB/s (93.7kB/s)(6236KiB/68138msec) clat (usec): min=686, max=165533, avg=18429.77, stdev=20007.98 lat (usec): min=686, max=165535, avg=18432.22, stdev=20008.70 clat percentiles (usec): | 1.00th=[ 783], 5.00th=[ 873], 10.00th=[ 996], 20.00th=[ 1303], | 30.00th=[ 2933], 40.00th=[ 5800], 50.00th=[ 13042], 60.00th=[ 17433], | 70.00th=[ 24249], 80.00th=[ 33162], 90.00th=[ 44303], 95.00th=[ 60556], | 99.00th=[ 84411], 99.50th=[ 96994], 99.90th=[120062], 99.95th=[164627], | 99.99th=[164627] bw ( KiB/s): min= 8, max= 232, per=24.75%, avg=90.57, stdev=33.99, samples=136 iops : min= 2, max= 58, avg=22.18, stdev= 8.53, samples=136 lat (usec) : 100=0.36%, 250=2.28%, 500=0.42%, 750=0.29%, 1000=4.85% lat (msec) : 2=8.11%, 4=4.39%, 10=7.49%, 20=25.98%, 50=38.28% lat (msec) : 100=7.06%, 250=0.46%, 750=0.03% cpu : usr=0.12%, sys=8.36%, ctx=5212, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=350KiB/s (359kB/s), 86.9KiB/s-88.8KiB/s (88.9kB/s-90.0kB/s), io=23.4MiB (24.6MB), run=68138-68571msec WRITE: bw=367KiB/s (375kB/s), 91.5KiB/s-92.8KiB/s (93.7kB/s-95.1kB/s), io=24.6MiB (25.7MB), run=68138-68571msec rand-rw: (groupid=0, jobs=1): err= 0: pid=23478: Fri May 2 01:51:09 2025 read: IOPS=16, BW=65.2KiB/s (66.7kB/s)(5992KiB/91960msec) clat (msec): min=3, max=579, avg=23.82, stdev=27.25 lat (msec): min=3, max=579, avg=23.82, stdev=27.25 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 7], 10.00th=[ 8], 20.00th=[ 9], | 30.00th=[ 11], 40.00th=[ 12], 50.00th=[ 15], 60.00th=[ 19], | 70.00th=[ 23], 80.00th=[ 34], 90.00th=[ 53], 95.00th=[ 74], | 99.00th=[ 117], 99.50th=[ 128], 99.90th=[ 199], 99.95th=[ 584], | 99.99th=[ 584] bw ( KiB/s): min= 8, max= 207, per=24.86%, avg=64.63, stdev=43.93, samples=182 iops : min= 2, max= 51, avg=15.71, stdev=11.02, samples=182 write: IOPS=17, BW=68.5KiB/s (70.1kB/s)(6296KiB/91960msec) clat (msec): min=7, max=251, avg=35.70, stdev=26.92 lat (msec): min=7, max=251, avg=35.71, stdev=26.92 clat percentiles (msec): | 1.00th=[ 10], 5.00th=[ 12], 10.00th=[ 14], 20.00th=[ 17], | 30.00th=[ 19], 40.00th=[ 22], 50.00th=[ 25], 60.00th=[ 32], | 70.00th=[ 43], 80.00th=[ 55], 90.00th=[ 70], 95.00th=[ 87], | 99.00th=[ 130], 99.50th=[ 159], 99.90th=[ 243], 99.95th=[ 253], | 99.99th=[ 253] bw ( KiB/s): min= 8, max= 207, per=24.96%, avg=67.88, stdev=45.64, samples=182 iops : min= 2, max= 51, avg=16.53, stdev=11.43, samples=182 lat (msec) : 4=0.20%, 10=14.55%, 20=35.09%, 50=32.39%, 100=15.53% lat (msec) : 250=2.18%, 500=0.03%, 750=0.03% cpu : usr=0.09%, sys=4.10%, ctx=3555, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23479: Fri May 2 01:51:09 2025 read: IOPS=16, BW=64.7KiB/s (66.2kB/s)(5940KiB/91843msec) clat (msec): min=3, max=690, avg=23.26, stdev=28.36 lat (msec): min=3, max=690, avg=23.27, stdev=28.36 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 6], 10.00th=[ 7], 20.00th=[ 9], | 30.00th=[ 11], 40.00th=[ 12], 50.00th=[ 15], 60.00th=[ 18], | 70.00th=[ 23], 80.00th=[ 33], 90.00th=[ 52], 95.00th=[ 70], | 99.00th=[ 111], 99.50th=[ 129], 99.90th=[ 192], 99.95th=[ 693], | 99.99th=[ 693] bw ( KiB/s): min= 7, max= 239, per=24.70%, avg=64.22, stdev=45.61, samples=182 iops : min= 1, max= 59, avg=15.62, stdev=11.44, samples=182 write: IOPS=17, BW=69.1KiB/s (70.8kB/s)(6348KiB/91843msec) clat (msec): min=6, max=243, avg=36.04, stdev=26.88 lat (msec): min=6, max=243, avg=36.05, stdev=26.88 clat percentiles (msec): | 1.00th=[ 10], 5.00th=[ 13], 10.00th=[ 14], 20.00th=[ 17], | 30.00th=[ 19], 40.00th=[ 22], 50.00th=[ 26], 60.00th=[ 32], | 70.00th=[ 42], 80.00th=[ 55], 90.00th=[ 74], 95.00th=[ 88], | 99.00th=[ 130], 99.50th=[ 155], 99.90th=[ 209], 99.95th=[ 243], | 99.99th=[ 243] bw ( KiB/s): min= 15, max= 192, per=25.16%, avg=68.42, stdev=44.78, samples=182 iops : min= 3, max= 48, avg=16.67, stdev=11.24, samples=182 lat (msec) : 4=0.20%, 10=14.84%, 20=34.41%, 50=33.72%, 100=14.68% lat (msec) : 250=2.12%, 750=0.03% cpu : usr=0.11%, sys=4.01%, ctx=3477, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23480: Fri May 2 01:51:09 2025 read: IOPS=16, BW=65.3KiB/s (66.9kB/s)(6028KiB/92241msec) clat (msec): min=3, max=183, avg=23.21, stdev=23.32 lat (msec): min=3, max=183, avg=23.21, stdev=23.32 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 6], 10.00th=[ 7], 20.00th=[ 9], | 30.00th=[ 11], 40.00th=[ 12], 50.00th=[ 15], 60.00th=[ 18], | 70.00th=[ 24], 80.00th=[ 34], 90.00th=[ 54], 95.00th=[ 73], | 99.00th=[ 115], 99.50th=[ 131], 99.90th=[ 182], 99.95th=[ 184], | 99.99th=[ 184] bw ( KiB/s): min= 7, max= 192, per=24.99%, avg=64.96, stdev=45.52, samples=183 iops : min= 1, max= 48, avg=15.82, stdev=11.41, samples=183 write: IOPS=16, BW=67.9KiB/s (69.5kB/s)(6260KiB/92241msec) clat (msec): min=6, max=700, avg=36.52, stdev=31.59 lat (msec): min=6, max=700, avg=36.52, stdev=31.58 clat percentiles (msec): | 1.00th=[ 10], 5.00th=[ 12], 10.00th=[ 14], 20.00th=[ 16], | 30.00th=[ 19], 40.00th=[ 22], 50.00th=[ 26], 60.00th=[ 33], | 70.00th=[ 43], 80.00th=[ 56], 90.00th=[ 73], 95.00th=[ 88], | 99.00th=[ 129], 99.50th=[ 148], 99.90th=[ 228], 99.95th=[ 701], | 99.99th=[ 701] bw ( KiB/s): min= 7, max= 192, per=24.75%, avg=67.31, stdev=45.62, samples=183 iops : min= 1, max= 48, avg=16.41, stdev=11.49, samples=183 lat (msec) : 4=0.10%, 10=14.97%, 20=35.68%, 50=32.03%, 100=14.94% lat (msec) : 250=2.25%, 750=0.03% cpu : usr=0.11%, sys=4.04%, ctx=3539, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23481: Fri May 2 01:51:09 2025 read: IOPS=16, BW=65.5KiB/s (67.1kB/s)(6052KiB/92327msec) clat (msec): min=2, max=179, avg=22.95, stdev=22.48 lat (msec): min=2, max=179, avg=22.95, stdev=22.48 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 6], 10.00th=[ 7], 20.00th=[ 9], | 30.00th=[ 11], 40.00th=[ 12], 50.00th=[ 14], 60.00th=[ 17], | 70.00th=[ 23], 80.00th=[ 33], 90.00th=[ 54], 95.00th=[ 75], | 99.00th=[ 107], 99.50th=[ 124], 99.90th=[ 140], 99.95th=[ 180], | 99.99th=[ 180] bw ( KiB/s): min= 7, max= 200, per=24.95%, avg=64.87, stdev=45.95, samples=183 iops : min= 1, max= 50, avg=15.81, stdev=11.51, samples=183 write: IOPS=16, BW=67.5KiB/s (69.2kB/s)(6236KiB/92327msec) clat (msec): min=6, max=703, avg=36.88, stdev=32.74 lat (msec): min=6, max=703, avg=36.88, stdev=32.74 clat percentiles (msec): | 1.00th=[ 10], 5.00th=[ 12], 10.00th=[ 14], 20.00th=[ 16], | 30.00th=[ 19], 40.00th=[ 22], 50.00th=[ 26], 60.00th=[ 32], | 70.00th=[ 43], 80.00th=[ 56], 90.00th=[ 77], 95.00th=[ 91], | 99.00th=[ 138], 99.50th=[ 169], 99.90th=[ 243], 99.95th=[ 701], | 99.99th=[ 701] bw ( KiB/s): min= 15, max= 207, per=24.62%, avg=66.97, stdev=45.24, samples=183 iops : min= 3, max= 51, avg=16.34, stdev=11.36, samples=183 lat (msec) : 4=0.20%, 10=14.94%, 20=35.45%, 50=31.64%, 100=15.46% lat (msec) : 250=2.28%, 750=0.03% cpu : usr=0.08%, sys=4.04%, ctx=3565, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=260KiB/s (266kB/s), 64.7KiB/s-65.5KiB/s (66.2kB/s-67.1kB/s), io=23.4MiB (24.6MB), run=91843-92327msec WRITE: bw=272KiB/s (279kB/s), 67.5KiB/s-69.1KiB/s (69.2kB/s-70.8kB/s), io=24.6MiB (25.7MB), run=91843-92327msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=23581: Fri May 2 01:51:23 2025 read: IOPS=29, BW=465KiB/s (477kB/s)(5920KiB/12722msec) clat (usec): min=67, max=96484, avg=17738.03, stdev=13287.15 lat (usec): min=67, max=96485, avg=17739.13, stdev=13287.30 clat percentiles (usec): | 1.00th=[ 75], 5.00th=[ 109], 10.00th=[ 131], 20.00th=[ 7308], | 30.00th=[10683], 40.00th=[13829], 50.00th=[16712], 60.00th=[19268], | 70.00th=[23200], 80.00th=[26608], 90.00th=[32375], 95.00th=[39060], | 99.00th=[53740], 99.50th=[82314], 99.90th=[96994], 99.95th=[96994], | 99.99th=[96994] bw ( KiB/s): min= 255, max= 795, per=24.45%, avg=456.52, stdev=130.22, samples=25 iops : min= 15, max= 49, avg=28.00, stdev= 8.16, samples=25 write: IOPS=31, BW=501KiB/s (513kB/s)(6368KiB/12722msec) clat (usec): min=1004, max=64037, avg=15081.62, stdev=14670.00 lat (usec): min=1004, max=64039, avg=15083.51, stdev=14670.20 clat percentiles (usec): | 1.00th=[ 1106], 5.00th=[ 1336], 10.00th=[ 1500], 20.00th=[ 2040], | 30.00th=[ 3818], 40.00th=[ 7308], 50.00th=[10814], 60.00th=[14091], | 70.00th=[19530], 80.00th=[25822], 90.00th=[39584], 95.00th=[47449], | 99.00th=[57410], 99.50th=[63177], 99.90th=[64226], 99.95th=[64226], | 99.99th=[64226] bw ( KiB/s): min= 191, max= 824, per=25.19%, avg=494.76, stdev=147.37, samples=25 iops : min= 11, max= 51, avg=30.40, stdev= 9.26, samples=25 lat (usec) : 100=1.04%, 250=5.73%, 500=0.78%, 1000=0.13% lat (msec) : 2=10.29%, 4=6.64%, 10=12.76%, 20=29.30%, 50=30.08% lat (msec) : 100=3.26% cpu : usr=0.17%, sys=11.82%, ctx=1429, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23582: Fri May 2 01:51:23 2025 read: IOPS=28, BW=464KiB/s (475kB/s)(5744KiB/12389msec) clat (usec): min=89, max=94841, avg=17866.78, stdev=13255.12 lat (usec): min=90, max=94841, avg=17867.84, stdev=13255.13 clat percentiles (usec): | 1.00th=[ 99], 5.00th=[ 123], 10.00th=[ 153], 20.00th=[ 8029], | 30.00th=[11469], 40.00th=[13698], 50.00th=[16581], 60.00th=[19006], | 70.00th=[21890], 80.00th=[26608], 90.00th=[33817], 95.00th=[39584], | 99.00th=[69731], 99.50th=[77071], 99.90th=[94897], 99.95th=[94897], | 99.99th=[94897] bw ( KiB/s): min= 191, max= 672, per=24.49%, avg=457.17, stdev=120.97, samples=24 iops : min= 11, max= 42, avg=27.96, stdev= 7.68, samples=24 write: IOPS=33, BW=528KiB/s (541kB/s)(6544KiB/12389msec) clat (usec): min=847, max=68706, avg=14187.20, stdev=13977.19 lat (usec): min=848, max=68707, avg=14189.53, stdev=13977.05 clat percentiles (usec): | 1.00th=[ 971], 5.00th=[ 1254], 10.00th=[ 1418], 20.00th=[ 1958], | 30.00th=[ 3359], 40.00th=[ 5407], 50.00th=[ 9765], 60.00th=[13566], | 70.00th=[18482], 80.00th=[25035], 90.00th=[35390], 95.00th=[44303], | 99.00th=[54789], 99.50th=[57934], 99.90th=[68682], 99.95th=[68682], | 99.99th=[68682] bw ( KiB/s): min= 319, max= 948, per=25.97%, avg=510.00, stdev=166.31, samples=24 iops : min= 19, max= 59, avg=31.29, stdev=10.49, samples=24 lat (usec) : 100=0.52%, 250=4.43%, 500=1.43%, 750=0.13%, 1000=1.04% lat (msec) : 2=10.55%, 4=7.29%, 10=13.41%, 20=29.82%, 50=29.17% lat (msec) : 100=2.21% cpu : usr=0.03%, sys=11.98%, ctx=1402, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23583: Fri May 2 01:51:23 2025 read: IOPS=30, BW=488KiB/s (499kB/s)(6256KiB/12826msec) clat (usec): min=81, max=88299, avg=17819.41, stdev=13391.17 lat (usec): min=81, max=88300, avg=17821.59, stdev=13391.77 clat percentiles (usec): | 1.00th=[ 95], 5.00th=[ 114], 10.00th=[ 141], 20.00th=[ 8094], | 30.00th=[10683], 40.00th=[13829], 50.00th=[16057], 60.00th=[19006], | 70.00th=[22938], 80.00th=[25822], 90.00th=[33817], 95.00th=[43254], | 99.00th=[63701], 99.50th=[73925], 99.90th=[88605], 99.95th=[88605], | 99.99th=[88605] bw ( KiB/s): min= 160, max= 928, per=25.66%, avg=479.00, stdev=156.55, samples=25 iops : min= 10, max= 58, avg=29.52, stdev= 9.81, samples=25 write: IOPS=29, BW=470KiB/s (482kB/s)(6032KiB/12826msec) clat (usec): min=1093, max=87708, avg=15389.04, stdev=14626.96 lat (usec): min=1094, max=87709, avg=15391.97, stdev=14627.92 clat percentiles (usec): | 1.00th=[ 1172], 5.00th=[ 1385], 10.00th=[ 1614], 20.00th=[ 2540], | 30.00th=[ 4047], 40.00th=[ 7767], 50.00th=[12518], 60.00th=[15795], | 70.00th=[19006], 80.00th=[25297], 90.00th=[35914], 95.00th=[44303], | 99.00th=[66847], 99.50th=[74974], 99.90th=[87557], 99.95th=[87557], | 99.99th=[87557] bw ( KiB/s): min= 256, max= 768, per=22.75%, avg=446.88, stdev=121.87, samples=25 iops : min= 16, max= 48, avg=27.52, stdev= 7.63, samples=25 lat (usec) : 100=0.65%, 250=5.73%, 500=0.91%, 750=0.26%, 1000=0.26% lat (msec) : 2=8.33%, 4=6.77%, 10=12.76%, 20=31.51%, 50=29.56% lat (msec) : 100=3.26% cpu : usr=0.15%, sys=11.70%, ctx=1413, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23584: Fri May 2 01:51:23 2025 read: IOPS=30, BW=484KiB/s (495kB/s)(6032KiB/12474msec) clat (usec): min=67, max=80386, avg=17316.64, stdev=12129.60 lat (usec): min=68, max=80387, avg=17317.66, stdev=12129.65 clat percentiles (usec): | 1.00th=[ 95], 5.00th=[ 116], 10.00th=[ 128], 20.00th=[ 8029], | 30.00th=[11338], 40.00th=[13829], 50.00th=[16450], 60.00th=[19006], | 70.00th=[22152], 80.00th=[26346], 90.00th=[32637], 95.00th=[37487], | 99.00th=[54789], 99.50th=[62129], 99.90th=[80217], 99.95th=[80217], | 99.99th=[80217] bw ( KiB/s): min= 192, max= 796, per=24.93%, avg=465.46, stdev=149.79, samples=24 iops : min= 12, max= 49, avg=28.71, stdev= 9.36, samples=24 write: IOPS=31, BW=502KiB/s (514kB/s)(6256KiB/12474msec) clat (usec): min=944, max=85282, avg=14680.05, stdev=15414.86 lat (usec): min=945, max=85285, avg=14682.89, stdev=15414.08 clat percentiles (usec): | 1.00th=[ 1057], 5.00th=[ 1287], 10.00th=[ 1500], 20.00th=[ 2311], | 30.00th=[ 3556], 40.00th=[ 5145], 50.00th=[ 8848], 60.00th=[13698], | 70.00th=[18220], 80.00th=[26346], 90.00th=[34866], 95.00th=[46924], | 99.00th=[69731], 99.50th=[78119], 99.90th=[85459], 99.95th=[85459], | 99.99th=[85459] bw ( KiB/s): min= 222, max= 734, per=24.99%, avg=490.79, stdev=155.40, samples=24 iops : min= 13, max= 45, avg=30.29, stdev= 9.70, samples=24 lat (usec) : 100=0.78%, 250=5.86%, 500=1.04%, 750=0.26%, 1000=0.13% lat (msec) : 2=8.72%, 4=7.94%, 10=13.80%, 20=28.91%, 50=29.69% lat (msec) : 100=2.86% cpu : usr=0.15%, sys=11.47%, ctx=1418, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1867KiB/s (1912kB/s), 464KiB/s-488KiB/s (475kB/s-499kB/s), io=23.4MiB (24.5MB), run=12389-12826msec WRITE: bw=1965KiB/s (2012kB/s), 470KiB/s-528KiB/s (482kB/s-541kB/s), io=24.6MiB (25.8MB), run=12389-12826msec rand-rw: (groupid=0, jobs=1): err= 0: pid=23576: Fri May 2 01:51:28 2025 read: IOPS=20, BW=333KiB/s (341kB/s)(5920KiB/17800msec) clat (usec): min=2763, max=81843, avg=17974.67, stdev=15166.69 lat (usec): min=2763, max=81843, avg=17975.93, stdev=15166.60 clat percentiles (usec): | 1.00th=[ 4490], 5.00th=[ 5735], 10.00th=[ 6325], 20.00th=[ 7963], | 30.00th=[ 9110], 40.00th=[10552], 50.00th=[11994], 60.00th=[14091], | 70.00th=[18744], 80.00th=[25035], 90.00th=[38011], 95.00th=[51119], | 99.00th=[74974], 99.50th=[78119], 99.90th=[82314], 99.95th=[82314], | 99.99th=[82314] bw ( KiB/s): min= 95, max= 704, per=24.44%, avg=325.80, stdev=173.22, samples=35 iops : min= 5, max= 44, avg=20.03, stdev=10.99, samples=35 write: IOPS=22, BW=358KiB/s (366kB/s)(6368KiB/17800msec) clat (msec): min=6, max=119, avg=27.73, stdev=18.82 lat (msec): min=6, max=119, avg=27.73, stdev=18.82 clat percentiles (msec): | 1.00th=[ 9], 5.00th=[ 11], 10.00th=[ 13], 20.00th=[ 14], | 30.00th=[ 16], 40.00th=[ 19], 50.00th=[ 22], 60.00th=[ 26], | 70.00th=[ 31], 80.00th=[ 40], 90.00th=[ 52], 95.00th=[ 68], | 99.00th=[ 106], 99.50th=[ 113], 99.90th=[ 120], 99.95th=[ 120], | 99.99th=[ 120] bw ( KiB/s): min= 191, max= 702, per=25.17%, avg=353.14, stdev=182.41, samples=35 iops : min= 11, max= 43, avg=21.74, stdev=11.46, samples=35 lat (msec) : 4=0.39%, 10=18.36%, 20=40.76%, 50=32.03%, 100=7.81% lat (msec) : 250=0.65% cpu : usr=0.04%, sys=5.42%, ctx=902, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23577: Fri May 2 01:51:28 2025 read: IOPS=19, BW=320KiB/s (328kB/s)(5744KiB/17959msec) clat (usec): min=3099, max=88622, avg=17312.45, stdev=15401.90 lat (usec): min=3100, max=88623, avg=17318.20, stdev=15401.73 clat percentiles (usec): | 1.00th=[ 4359], 5.00th=[ 5407], 10.00th=[ 6390], 20.00th=[ 7767], | 30.00th=[ 8586], 40.00th=[10028], 50.00th=[11469], 60.00th=[13173], | 70.00th=[16581], 80.00th=[22938], 90.00th=[39584], 95.00th=[56886], | 99.00th=[74974], 99.50th=[84411], 99.90th=[88605], 99.95th=[88605], | 99.99th=[88605] bw ( KiB/s): min= 95, max= 736, per=23.47%, avg=312.80, stdev=204.71, samples=35 iops : min= 5, max= 46, avg=19.17, stdev=12.76, samples=35 write: IOPS=22, BW=364KiB/s (373kB/s)(6544KiB/17959msec) clat (msec): min=7, max=114, avg=28.49, stdev=20.47 lat (msec): min=7, max=114, avg=28.49, stdev=20.47 clat percentiles (msec): | 1.00th=[ 9], 5.00th=[ 10], 10.00th=[ 12], 20.00th=[ 14], | 30.00th=[ 16], 40.00th=[ 18], 50.00th=[ 21], 60.00th=[ 25], | 70.00th=[ 33], 80.00th=[ 43], 90.00th=[ 57], 95.00th=[ 72], | 99.00th=[ 102], 99.50th=[ 104], 99.90th=[ 114], 99.95th=[ 114], | 99.99th=[ 114] bw ( KiB/s): min= 96, max= 864, per=24.96%, avg=350.26, stdev=189.43, samples=35 iops : min= 6, max= 54, avg=21.51, stdev=11.87, samples=35 lat (msec) : 4=0.39%, 10=21.87%, 20=40.49%, 50=27.21%, 100=9.24% lat (msec) : 250=0.78% cpu : usr=0.11%, sys=5.34%, ctx=927, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23578: Fri May 2 01:51:28 2025 read: IOPS=21, BW=350KiB/s (359kB/s)(6256KiB/17860msec) clat (usec): min=3715, max=96985, avg=18329.78, stdev=16545.66 lat (usec): min=3716, max=96985, avg=18331.32, stdev=16545.36 clat percentiles (usec): | 1.00th=[ 4178], 5.00th=[ 5342], 10.00th=[ 6325], 20.00th=[ 7570], | 30.00th=[ 8586], 40.00th=[10028], 50.00th=[11863], 60.00th=[14484], | 70.00th=[17433], 80.00th=[26346], 90.00th=[41681], 95.00th=[53216], | 99.00th=[83362], 99.50th=[87557], 99.90th=[96994], 99.95th=[96994], | 99.99th=[96994] bw ( KiB/s): min= 96, max= 1053, per=26.02%, avg=346.86, stdev=226.27, samples=35 iops : min= 6, max= 65, avg=21.40, stdev=14.21, samples=35 write: IOPS=21, BW=338KiB/s (346kB/s)(6032KiB/17860msec) clat (msec): min=6, max=100, avg=28.25, stdev=18.42 lat (msec): min=6, max=100, avg=28.25, stdev=18.42 clat percentiles (msec): | 1.00th=[ 9], 5.00th=[ 11], 10.00th=[ 13], 20.00th=[ 15], | 30.00th=[ 17], 40.00th=[ 18], 50.00th=[ 21], 60.00th=[ 26], | 70.00th=[ 32], 80.00th=[ 44], 90.00th=[ 58], 95.00th=[ 65], | 99.00th=[ 91], 99.50th=[ 99], 99.90th=[ 101], 99.95th=[ 101], | 99.99th=[ 101] bw ( KiB/s): min= 128, max= 768, per=23.42%, avg=328.51, stdev=172.10, samples=35 iops : min= 8, max= 48, avg=20.26, stdev=10.85, samples=35 lat (msec) : 4=0.39%, 10=22.01%, 20=38.15%, 50=29.30%, 100=10.03% lat (msec) : 250=0.13% cpu : usr=0.02%, sys=5.23%, ctx=891, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23579: Fri May 2 01:51:28 2025 read: IOPS=21, BW=339KiB/s (347kB/s)(6032KiB/17816msec) clat (msec): min=3, max=109, avg=18.55, stdev=16.49 lat (msec): min=3, max=109, avg=18.55, stdev=16.48 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 6], 10.00th=[ 7], 20.00th=[ 8], | 30.00th=[ 10], 40.00th=[ 11], 50.00th=[ 13], 60.00th=[ 15], | 70.00th=[ 18], 80.00th=[ 26], 90.00th=[ 43], 95.00th=[ 56], | 99.00th=[ 75], 99.50th=[ 97], 99.90th=[ 110], 99.95th=[ 110], | 99.99th=[ 110] bw ( KiB/s): min= 128, max= 736, per=24.70%, avg=329.20, stdev=161.41, samples=35 iops : min= 8, max= 46, avg=20.11, stdev=10.18, samples=35 write: IOPS=21, BW=351KiB/s (360kB/s)(6256KiB/17816msec) clat (msec): min=8, max=113, avg=27.49, stdev=17.15 lat (msec): min=8, max=113, avg=27.49, stdev=17.15 clat percentiles (msec): | 1.00th=[ 10], 5.00th=[ 12], 10.00th=[ 14], 20.00th=[ 15], | 30.00th=[ 17], 40.00th=[ 19], 50.00th=[ 21], 60.00th=[ 25], | 70.00th=[ 32], 80.00th=[ 39], 90.00th=[ 52], 95.00th=[ 65], | 99.00th=[ 82], 99.50th=[ 104], 99.90th=[ 114], 99.95th=[ 114], | 99.99th=[ 114] bw ( KiB/s): min= 158, max= 768, per=24.70%, avg=346.54, stdev=181.95, samples=35 iops : min= 9, max= 48, avg=21.20, stdev=11.47, samples=35 lat (msec) : 4=0.26%, 10=17.19%, 20=44.40%, 50=28.26%, 100=9.51% lat (msec) : 250=0.39% cpu : usr=0.11%, sys=5.27%, ctx=913, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1334KiB/s (1366kB/s), 320KiB/s-350KiB/s (328kB/s-359kB/s), io=23.4MiB (24.5MB), run=17800-17959msec WRITE: bw=1403KiB/s (1437kB/s), 338KiB/s-364KiB/s (346kB/s-373kB/s), io=24.6MiB (25.8MB), run=17800-17959msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=23603: Fri May 2 01:51:32 2025 read: IOPS=4, BW=4628KiB/s (4739kB/s)(7168KiB/1549msec) clat (msec): min=74, max=120, avg=101.21, stdev=19.87 lat (msec): min=74, max=120, avg=101.21, stdev=19.87 clat percentiles (msec): | 1.00th=[ 74], 5.00th=[ 74], 10.00th=[ 74], 20.00th=[ 80], | 30.00th=[ 94], 40.00th=[ 94], 50.00th=[ 101], 60.00th=[ 120], | 70.00th=[ 120], 80.00th=[ 122], 90.00th=[ 122], 95.00th=[ 122], | 99.00th=[ 122], 99.50th=[ 122], 99.90th=[ 122], 99.95th=[ 122], | 99.99th=[ 122] bw ( KiB/s): min= 2043, max= 6071, per=35.59%, avg=4067.00, stdev=2014.07, samples=3 iops : min= 1, max= 5, avg= 3.00, stdev= 2.00, samples=3 write: IOPS=3, BW=3305KiB/s (3385kB/s)(5120KiB/1549msec) clat (msec): min=62, max=228, avg=160.62, stdev=63.22 lat (msec): min=62, max=228, avg=160.77, stdev=63.32 clat percentiles (msec): | 1.00th=[ 63], 5.00th=[ 63], 10.00th=[ 63], 20.00th=[ 63], | 30.00th=[ 148], 40.00th=[ 148], 50.00th=[ 163], 60.00th=[ 163], | 70.00th=[ 201], 80.00th=[ 201], 90.00th=[ 228], 95.00th=[ 228], | 99.00th=[ 228], 99.50th=[ 228], 99.90th=[ 228], 99.95th=[ 228], | 99.99th=[ 228] bw ( KiB/s): min= 2023, max= 6131, per=21.24%, avg=3399.00, stdev=2366.00, samples=3 iops : min= 1, max= 5, avg= 2.33, stdev= 2.31, samples=3 lat (msec) : 100=33.33%, 250=66.67% cpu : usr=0.00%, sys=21.12%, ctx=472, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23604: Fri May 2 01:51:32 2025 read: IOPS=1, BW=1143KiB/s (1170kB/s)(2048KiB/1792msec) clat (usec): min=76825, max=92248, avg=84536.63, stdev=10905.63 lat (usec): min=76827, max=92250, avg=84538.90, stdev=10906.27 clat percentiles (usec): | 1.00th=[77071], 5.00th=[77071], 10.00th=[77071], 20.00th=[77071], | 30.00th=[77071], 40.00th=[77071], 50.00th=[77071], 60.00th=[91751], | 70.00th=[91751], 80.00th=[91751], 90.00th=[91751], 95.00th=[91751], | 99.00th=[91751], 99.50th=[91751], 99.90th=[91751], 99.95th=[91751], | 99.99th=[91751] bw ( KiB/s): min= 2048, max= 2048, per=17.92%, avg=2048.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 write: IOPS=5, BW=5714KiB/s (5851kB/s)(10.0MiB/1792msec) clat (msec): min=42, max=292, avg=155.47, stdev=88.12 lat (msec): min=42, max=292, avg=155.75, stdev=88.29 clat percentiles (msec): | 1.00th=[ 43], 5.00th=[ 43], 10.00th=[ 43], 20.00th=[ 54], | 30.00th=[ 61], 40.00th=[ 134], 50.00th=[ 140], 60.00th=[ 167], | 70.00th=[ 226], 80.00th=[ 226], 90.00th=[ 268], 95.00th=[ 292], | 99.00th=[ 292], 99.50th=[ 292], 99.90th=[ 292], 99.95th=[ 292], | 99.99th=[ 292] bw ( KiB/s): min= 2048, max= 8192, per=29.87%, avg=4778.67, stdev=3128.37, samples=3 iops : min= 2, max= 8, avg= 4.67, stdev= 3.06, samples=3 lat (msec) : 50=8.33%, 100=33.33%, 250=41.67%, 500=16.67% cpu : usr=0.00%, sys=19.93%, ctx=328, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23605: Fri May 2 01:51:32 2025 read: IOPS=4, BW=4140KiB/s (4240kB/s)(6144KiB/1484msec) clat (msec): min=2, max=122, avg=47.34, stdev=45.71 lat (msec): min=2, max=122, avg=47.34, stdev=45.71 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 3], 20.00th=[ 14], | 30.00th=[ 14], 40.00th=[ 18], 50.00th=[ 18], 60.00th=[ 64], | 70.00th=[ 66], 80.00th=[ 66], 90.00th=[ 124], 95.00th=[ 124], | 99.00th=[ 124], 99.50th=[ 124], 99.90th=[ 124], 99.95th=[ 124], | 99.99th=[ 124] bw ( KiB/s): min= 4079, max= 4079, per=35.69%, avg=4079.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 write: IOPS=4, BW=4140KiB/s (4240kB/s)(6144KiB/1484msec) clat (msec): min=68, max=356, avg=192.22, stdev=99.04 lat (msec): min=68, max=356, avg=192.28, stdev=99.04 clat percentiles (msec): | 1.00th=[ 69], 5.00th=[ 69], 10.00th=[ 69], 20.00th=[ 126], | 30.00th=[ 126], 40.00th=[ 163], 50.00th=[ 163], 60.00th=[ 215], | 70.00th=[ 226], 80.00th=[ 226], 90.00th=[ 355], 95.00th=[ 355], | 99.00th=[ 355], 99.50th=[ 355], 99.90th=[ 355], 99.95th=[ 355], | 99.99th=[ 355] bw ( KiB/s): min= 4079, max= 4096, per=25.55%, avg=4087.50, stdev=12.02, samples=2 iops : min= 3, max= 4, avg= 3.50, stdev= 0.71, samples=2 lat (msec) : 4=8.33%, 20=16.67%, 100=25.00%, 250=41.67%, 500=8.33% cpu : usr=0.00%, sys=15.85%, ctx=102, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23606: Fri May 2 01:51:32 2025 read: IOPS=2, BW=3021KiB/s (3093kB/s)(5120KiB/1695msec) clat (usec): min=616, max=158752, avg=65175.00, stdev=78035.53 lat (usec): min=616, max=158754, avg=65176.42, stdev=78035.87 clat percentiles (usec): | 1.00th=[ 619], 5.00th=[ 619], 10.00th=[ 619], 20.00th=[ 619], | 30.00th=[ 881], 40.00th=[ 881], 50.00th=[ 25035], 60.00th=[ 25035], | 70.00th=[141558], 80.00th=[141558], 90.00th=[158335], 95.00th=[158335], | 99.00th=[158335], 99.50th=[158335], 99.90th=[158335], 99.95th=[158335], | 99.99th=[158335] bw ( KiB/s): min= 8192, max= 8192, per=71.68%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=4, BW=4229KiB/s (4330kB/s)(7168KiB/1695msec) clat (msec): min=87, max=283, avg=180.32, stdev=73.70 lat (msec): min=87, max=283, avg=180.43, stdev=73.68 clat percentiles (msec): | 1.00th=[ 88], 5.00th=[ 88], 10.00th=[ 88], 20.00th=[ 93], | 30.00th=[ 150], 40.00th=[ 150], 50.00th=[ 194], 60.00th=[ 220], | 70.00th=[ 220], 80.00th=[ 236], 90.00th=[ 284], 95.00th=[ 284], | 99.00th=[ 284], 99.50th=[ 284], 99.90th=[ 284], 99.95th=[ 284], | 99.99th=[ 284] bw ( KiB/s): min= 4055, max= 4096, per=25.48%, avg=4076.67, stdev=20.60, samples=3 iops : min= 3, max= 4, avg= 3.33, stdev= 0.58, samples=3 lat (usec) : 750=8.33%, 1000=8.33% lat (msec) : 50=8.33%, 100=16.67%, 250=50.00%, 500=8.33% cpu : usr=0.00%, sys=16.88%, ctx=109, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=11.2MiB/s (11.7MB/s), 1143KiB/s-4628KiB/s (1170kB/s-4739kB/s), io=20.0MiB (20.0MB), run=1484-1792msec WRITE: bw=15.6MiB/s (16.4MB/s), 3305KiB/s-5714KiB/s (3385kB/s-5851kB/s), io=28.0MiB (29.4MB), run=1484-1792msec rand-rw: (groupid=0, jobs=1): err= 0: pid=23608: Fri May 2 01:51:32 2025 read: IOPS=3, BW=3789KiB/s (3880kB/s)(7168KiB/1892msec) clat (msec): min=32, max=382, avg=160.39, stdev=137.75 lat (msec): min=32, max=382, avg=160.39, stdev=137.75 clat percentiles (msec): | 1.00th=[ 33], 5.00th=[ 33], 10.00th=[ 33], 20.00th=[ 49], | 30.00th=[ 63], 40.00th=[ 63], 50.00th=[ 86], 60.00th=[ 222], | 70.00th=[ 222], 80.00th=[ 288], 90.00th=[ 384], 95.00th=[ 384], | 99.00th=[ 384], 99.50th=[ 384], 99.90th=[ 384], 99.95th=[ 384], | 99.99th=[ 384] bw ( KiB/s): min= 4096, max= 6131, per=47.59%, avg=5113.50, stdev=1438.96, samples=2 iops : min= 4, max= 5, avg= 4.50, stdev= 0.71, samples=2 write: IOPS=2, BW=2706KiB/s (2771kB/s)(5120KiB/1892msec) clat (msec): min=66, max=258, avg=137.05, stdev=95.22 lat (msec): min=66, max=258, avg=137.13, stdev=95.22 clat percentiles (msec): | 1.00th=[ 66], 5.00th=[ 66], 10.00th=[ 66], 20.00th=[ 66], | 30.00th=[ 67], 40.00th=[ 67], 50.00th=[ 72], 60.00th=[ 72], | 70.00th=[ 224], 80.00th=[ 224], 90.00th=[ 259], 95.00th=[ 259], | 99.00th=[ 259], 99.50th=[ 259], 99.90th=[ 259], 99.95th=[ 259], | 99.99th=[ 259] bw ( KiB/s): min= 2048, max= 2048, per=13.61%, avg=2048.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 50=16.67%, 100=41.67%, 250=16.67%, 500=25.00% cpu : usr=0.05%, sys=6.66%, ctx=58, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23610: Fri May 2 01:51:32 2025 read: IOPS=1, BW=1137KiB/s (1164kB/s)(2048KiB/1802msec) clat (msec): min=45, max=163, avg=104.53, stdev=82.85 lat (msec): min=45, max=163, avg=104.53, stdev=82.85 clat percentiles (msec): | 1.00th=[ 46], 5.00th=[ 46], 10.00th=[ 46], 20.00th=[ 46], | 30.00th=[ 46], 40.00th=[ 46], 50.00th=[ 46], 60.00th=[ 163], | 70.00th=[ 163], 80.00th=[ 163], 90.00th=[ 163], 95.00th=[ 163], | 99.00th=[ 163], 99.50th=[ 163], 99.90th=[ 163], 99.95th=[ 163], | 99.99th=[ 163] bw ( KiB/s): min= 2019, max= 2048, per=18.93%, avg=2033.50, stdev=20.51, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 write: IOPS=5, BW=5683KiB/s (5819kB/s)(10.0MiB/1802msec) clat (msec): min=31, max=425, avg=158.96, stdev=130.96 lat (msec): min=31, max=425, avg=159.05, stdev=130.95 clat percentiles (msec): | 1.00th=[ 32], 5.00th=[ 32], 10.00th=[ 32], 20.00th=[ 40], | 30.00th=[ 50], 40.00th=[ 56], 50.00th=[ 131], 60.00th=[ 153], | 70.00th=[ 199], 80.00th=[ 199], 90.00th=[ 326], 95.00th=[ 426], | 99.00th=[ 426], 99.50th=[ 426], 99.90th=[ 426], 99.95th=[ 426], | 99.99th=[ 426] bw ( KiB/s): min= 4000, max= 6059, per=33.43%, avg=5029.50, stdev=1455.93, samples=2 iops : min= 3, max= 5, avg= 4.00, stdev= 1.41, samples=2 lat (msec) : 50=33.33%, 100=8.33%, 250=41.67%, 500=16.67% cpu : usr=0.00%, sys=4.72%, ctx=37, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23611: Fri May 2 01:51:32 2025 read: IOPS=3, BW=3323KiB/s (3403kB/s)(6144KiB/1849msec) clat (msec): min=28, max=319, avg=92.82, stdev=113.20 lat (msec): min=28, max=319, avg=92.82, stdev=113.20 clat percentiles (msec): | 1.00th=[ 29], 5.00th=[ 29], 10.00th=[ 29], 20.00th=[ 30], | 30.00th=[ 30], 40.00th=[ 43], 50.00th=[ 43], 60.00th=[ 45], | 70.00th=[ 92], 80.00th=[ 92], 90.00th=[ 321], 95.00th=[ 321], | 99.00th=[ 321], 99.50th=[ 321], 99.90th=[ 321], 99.95th=[ 321], | 99.99th=[ 321] bw ( KiB/s): min= 2043, max= 2048, per=19.04%, avg=2045.50, stdev= 3.54, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 write: IOPS=3, BW=3323KiB/s (3403kB/s)(6144KiB/1849msec) clat (msec): min=38, max=450, avg=209.20, stdev=181.19 lat (msec): min=38, max=451, avg=209.27, stdev=181.19 clat percentiles (msec): | 1.00th=[ 39], 5.00th=[ 39], 10.00th=[ 39], 20.00th=[ 68], | 30.00th=[ 68], 40.00th=[ 73], 50.00th=[ 73], 60.00th=[ 222], | 70.00th=[ 405], 80.00th=[ 405], 90.00th=[ 451], 95.00th=[ 451], | 99.00th=[ 451], 99.50th=[ 451], 99.90th=[ 451], 99.95th=[ 451], | 99.99th=[ 451] bw ( KiB/s): min= 2048, max= 4096, per=20.42%, avg=3072.00, stdev=1448.15, samples=2 iops : min= 2, max= 4, avg= 3.00, stdev= 1.41, samples=2 lat (msec) : 50=41.67%, 100=25.00%, 250=8.33%, 500=25.00% cpu : usr=0.00%, sys=3.95%, ctx=43, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23612: Fri May 2 01:51:32 2025 read: IOPS=2, BW=2686KiB/s (2751kB/s)(5120KiB/1906msec) clat (msec): min=21, max=197, avg=110.83, stdev=64.40 lat (msec): min=21, max=197, avg=110.83, stdev=64.40 clat percentiles (msec): | 1.00th=[ 22], 5.00th=[ 22], 10.00th=[ 22], 20.00th=[ 22], | 30.00th=[ 92], 40.00th=[ 92], 50.00th=[ 106], 60.00th=[ 106], | 70.00th=[ 138], 80.00th=[ 138], 90.00th=[ 199], 95.00th=[ 199], | 99.00th=[ 199], 99.50th=[ 199], 99.90th=[ 199], 99.95th=[ 199], | 99.99th=[ 199] bw ( KiB/s): min= 2043, max= 2043, per=19.01%, avg=2043.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=3, BW=3761KiB/s (3851kB/s)(7168KiB/1906msec) clat (msec): min=35, max=388, avg=191.20, stdev=146.08 lat (msec): min=35, max=388, avg=191.27, stdev=146.09 clat percentiles (msec): | 1.00th=[ 36], 5.00th=[ 36], 10.00th=[ 36], 20.00th=[ 41], | 30.00th=[ 68], 40.00th=[ 68], 50.00th=[ 199], 60.00th=[ 284], | 70.00th=[ 284], 80.00th=[ 330], 90.00th=[ 388], 95.00th=[ 388], | 99.00th=[ 388], 99.50th=[ 388], 99.90th=[ 388], 99.95th=[ 388], | 99.99th=[ 388] bw ( KiB/s): min= 2043, max= 4096, per=18.13%, avg=2727.33, stdev=1185.30, samples=3 iops : min= 1, max= 4, avg= 2.00, stdev= 1.73, samples=3 lat (msec) : 50=25.00%, 100=16.67%, 250=33.33%, 500=25.00% cpu : usr=0.00%, sys=6.30%, ctx=51, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=10.5MiB/s (11.0MB/s), 1137KiB/s-3789KiB/s (1164kB/s-3880kB/s), io=20.0MiB (20.0MB), run=1802-1906msec WRITE: bw=14.7MiB/s (15.4MB/s), 2706KiB/s-5683KiB/s (2771kB/s-5819kB/s), io=28.0MiB (29.4MB), run=1802-1906msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=23619: Fri May 2 01:51:36 2025 read: IOPS=2, BW=10.1MiB/s (10.6MB/s)(12.0MiB/1185msec) clat (msec): min=174, max=660, avg=379.71, stdev=252.03 lat (msec): min=174, max=661, avg=379.71, stdev=252.03 clat percentiles (msec): | 1.00th=[ 176], 5.00th=[ 176], 10.00th=[ 176], 20.00th=[ 176], | 30.00th=[ 176], 40.00th=[ 305], 50.00th=[ 305], 60.00th=[ 305], | 70.00th=[ 659], 80.00th=[ 659], 90.00th=[ 659], 95.00th=[ 659], | 99.00th=[ 659], 99.50th=[ 659], 99.90th=[ 659], 99.95th=[ 659], | 99.99th=[ 659] bw ( KiB/s): min= 8175, max= 8192, per=96.86%, avg=8183.50, stdev=12.02, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 250=33.33%, 500=33.33%, 750=33.33% cpu : usr=0.00%, sys=20.27%, ctx=625, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23620: Fri May 2 01:51:36 2025 read: IOPS=0, BW=2112KiB/s (2163kB/s)(4096KiB/1939msec) clat (nsec): min=305093k, max=305093k, avg=305093275.00, stdev= 0.00 lat (nsec): min=305096k, max=305096k, avg=305095860.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 305], 5.00th=[ 305], 10.00th=[ 305], 20.00th=[ 305], | 30.00th=[ 305], 40.00th=[ 305], 50.00th=[ 305], 60.00th=[ 305], | 70.00th=[ 305], 80.00th=[ 305], 90.00th=[ 305], 95.00th=[ 305], | 99.00th=[ 305], 99.50th=[ 305], 99.90th=[ 305], 99.95th=[ 305], | 99.99th=[ 305] bw ( KiB/s): min= 8192, max= 8192, per=96.96%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=1, BW=4225KiB/s (4326kB/s)(8192KiB/1939msec) clat (msec): min=465, max=1143, avg=804.61, stdev=479.78 lat (msec): min=465, max=1144, avg=805.04, stdev=479.60 clat percentiles (msec): | 1.00th=[ 464], 5.00th=[ 464], 10.00th=[ 464], 20.00th=[ 464], | 30.00th=[ 464], 40.00th=[ 464], 50.00th=[ 464], 60.00th=[ 1150], | 70.00th=[ 1150], 80.00th=[ 1150], 90.00th=[ 1150], 95.00th=[ 1150], | 99.00th=[ 1150], 99.50th=[ 1150], 99.90th=[ 1150], 99.95th=[ 1150], | 99.99th=[ 1150] bw ( KiB/s): min= 5490, max= 5490, per=36.56%, avg=5490.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 500=66.67% cpu : usr=0.00%, sys=16.00%, ctx=272, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23622: Fri May 2 01:51:36 2025 write: IOPS=1, BW=5632KiB/s (5767kB/s)(12.0MiB/2182msec) clat (msec): min=340, max=1030, avg=709.65, stdev=347.69 lat (msec): min=340, max=1031, avg=710.01, stdev=347.78 clat percentiles (msec): | 1.00th=[ 342], 5.00th=[ 342], 10.00th=[ 342], 20.00th=[ 342], | 30.00th=[ 342], 40.00th=[ 760], 50.00th=[ 760], 60.00th=[ 760], | 70.00th=[ 1028], 80.00th=[ 1028], 90.00th=[ 1028], 95.00th=[ 1028], | 99.00th=[ 1028], 99.50th=[ 1028], 99.90th=[ 1028], 99.95th=[ 1028], | 99.99th=[ 1028] bw ( KiB/s): min= 5843, max= 8175, per=46.67%, avg=7009.00, stdev=1648.97, samples=2 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=2 lat (msec) : 500=33.33%, 1000=33.33% cpu : usr=0.00%, sys=15.54%, ctx=89, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23623: Fri May 2 01:51:36 2025 write: IOPS=2, BW=8733KiB/s (8943kB/s)(12.0MiB/1407msec) clat (msec): min=228, max=781, avg=459.12, stdev=287.37 lat (msec): min=228, max=781, avg=459.57, stdev=287.43 clat percentiles (msec): | 1.00th=[ 228], 5.00th=[ 228], 10.00th=[ 228], 20.00th=[ 228], | 30.00th=[ 228], 40.00th=[ 368], 50.00th=[ 368], 60.00th=[ 368], | 70.00th=[ 785], 80.00th=[ 785], 90.00th=[ 785], 95.00th=[ 785], | 99.00th=[ 785], 99.50th=[ 785], 99.90th=[ 785], 99.95th=[ 785], | 99.99th=[ 785] bw ( KiB/s): min= 8192, max= 8192, per=54.55%, avg=8192.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 lat (msec) : 250=33.33%, 500=33.33%, 1000=33.33% cpu : usr=0.00%, sys=23.97%, ctx=213, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=8450KiB/s (8653kB/s), 2112KiB/s-10.1MiB/s (2163kB/s-10.6MB/s), io=16.0MiB (16.8MB), run=1185-1939msec WRITE: bw=14.7MiB/s (15.4MB/s), 4225KiB/s-8733KiB/s (4326kB/s-8943kB/s), io=32.0MiB (33.6MB), run=1407-2182msec rand-rw: (groupid=0, jobs=1): err= 0: pid=23624: Fri May 2 01:51:37 2025 read: IOPS=1, BW=5107KiB/s (5230kB/s)(12.0MiB/2406msec) clat (msec): min=412, max=943, avg=760.01, stdev=300.83 lat (msec): min=412, max=943, avg=760.01, stdev=300.83 clat percentiles (msec): | 1.00th=[ 414], 5.00th=[ 414], 10.00th=[ 414], 20.00th=[ 414], | 30.00th=[ 414], 40.00th=[ 927], 50.00th=[ 927], 60.00th=[ 927], | 70.00th=[ 944], 80.00th=[ 944], 90.00th=[ 944], 95.00th=[ 944], | 99.00th=[ 944], 99.50th=[ 944], 99.90th=[ 944], 99.95th=[ 944], | 99.99th=[ 944] bw ( KiB/s): min=16318, max=16318, per=100.00%, avg=16318.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 lat (msec) : 500=33.33%, 1000=66.67% cpu : usr=0.00%, sys=1.79%, ctx=29, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23625: Fri May 2 01:51:37 2025 read: IOPS=0, BW=1658KiB/s (1697kB/s)(4096KiB/2471msec) clat (nsec): min=859268k, max=859268k, avg=859267580.00, stdev= 0.00 lat (nsec): min=859270k, max=859270k, avg=859270063.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 860], 5.00th=[ 860], 10.00th=[ 860], 20.00th=[ 860], | 30.00th=[ 860], 40.00th=[ 860], 50.00th=[ 860], 60.00th=[ 860], | 70.00th=[ 860], 80.00th=[ 860], 90.00th=[ 860], 95.00th=[ 860], | 99.00th=[ 860], 99.50th=[ 860], 99.90th=[ 860], 99.95th=[ 860], | 99.99th=[ 860] bw ( KiB/s): min= 8031, max= 8031, per=100.00%, avg=8031.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=0, BW=3315KiB/s (3395kB/s)(8192KiB/2471msec) clat (msec): min=735, max=792, avg=763.80, stdev=40.00 lat (msec): min=735, max=792, avg=764.20, stdev=40.08 clat percentiles (msec): | 1.00th=[ 735], 5.00th=[ 735], 10.00th=[ 735], 20.00th=[ 735], | 30.00th=[ 735], 40.00th=[ 735], 50.00th=[ 735], 60.00th=[ 793], | 70.00th=[ 793], 80.00th=[ 793], 90.00th=[ 793], 95.00th=[ 793], | 99.00th=[ 793], 99.50th=[ 793], 99.90th=[ 793], 99.95th=[ 793], | 99.99th=[ 793] bw ( KiB/s): min= 8047, max= 8047, per=64.10%, avg=8047.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 750=33.33%, 1000=66.67% cpu : usr=0.00%, sys=2.31%, ctx=25, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23626: Fri May 2 01:51:37 2025 write: IOPS=1, BW=4730KiB/s (4843kB/s)(12.0MiB/2598msec) clat (msec): min=598, max=1093, avg=858.02, stdev=248.36 lat (msec): min=598, max=1093, avg=858.36, stdev=248.37 clat percentiles (msec): | 1.00th=[ 600], 5.00th=[ 600], 10.00th=[ 600], 20.00th=[ 600], | 30.00th=[ 600], 40.00th=[ 885], 50.00th=[ 885], 60.00th=[ 885], | 70.00th=[ 1099], 80.00th=[ 1099], 90.00th=[ 1099], 95.00th=[ 1099], | 99.00th=[ 1099], 99.50th=[ 1099], 99.90th=[ 1099], 99.95th=[ 1099], | 99.99th=[ 1099] bw ( KiB/s): min= 8126, max= 8192, per=64.99%, avg=8159.00, stdev=46.67, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 750=33.33%, 1000=33.33% cpu : usr=0.00%, sys=7.39%, ctx=110, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23627: Fri May 2 01:51:37 2025 write: IOPS=1, BW=4708KiB/s (4821kB/s)(12.0MiB/2610msec) clat (msec): min=236, max=1501, avg=835.12, stdev=635.05 lat (msec): min=239, max=1501, avg=836.38, stdev=633.83 clat percentiles (msec): | 1.00th=[ 236], 5.00th=[ 236], 10.00th=[ 236], 20.00th=[ 236], | 30.00th=[ 236], 40.00th=[ 768], 50.00th=[ 768], 60.00th=[ 768], | 70.00th=[ 1502], 80.00th=[ 1502], 90.00th=[ 1502], 95.00th=[ 1502], | 99.00th=[ 1502], 99.50th=[ 1502], 99.90th=[ 1502], 99.95th=[ 1502], | 99.99th=[ 1502] bw ( KiB/s): min= 8192, max= 8192, per=65.25%, avg=8192.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 lat (msec) : 250=33.33%, 1000=33.33% cpu : usr=0.00%, sys=6.67%, ctx=78, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=6631KiB/s (6790kB/s), 1658KiB/s-5107KiB/s (1697kB/s-5230kB/s), io=16.0MiB (16.8MB), run=2406-2471msec WRITE: bw=12.3MiB/s (12.9MB/s), 3315KiB/s-4730KiB/s (3395kB/s-4843kB/s), io=32.0MiB (33.6MB), run=2471-2610msec