== sanity test 398b: DIO and buffer IO race ============== 20:37:55 (1746146275) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.45962 s, 34.5 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=1240: Thu May 1 20:38:24 2025 read: IOPS=59, BW=237KiB/s (242kB/s)(5992KiB/25333msec) clat (usec): min=50, max=249289, avg=8021.52, stdev=9317.28 lat (usec): min=50, max=249290, avg=8024.10, stdev=9316.09 clat percentiles (usec): | 1.00th=[ 74], 5.00th=[ 95], 10.00th=[ 111], 20.00th=[ 2933], | 30.00th=[ 4424], 40.00th=[ 5473], 50.00th=[ 6521], 60.00th=[ 7898], | 70.00th=[ 9765], 80.00th=[ 12518], 90.00th=[ 16450], 95.00th=[ 19530], | 99.00th=[ 29230], 99.50th=[ 33817], 99.90th=[ 93848], 99.95th=[248513], | 99.99th=[248513] bw ( KiB/s): min= 71, max= 392, per=25.63%, avg=237.32, stdev=63.02, samples=50 iops : min= 17, max= 98, avg=59.12, stdev=15.74, samples=50 write: IOPS=62, BW=249KiB/s (254kB/s)(6296KiB/25333msec) clat (usec): min=533, max=70114, avg=8349.48, stdev=8504.60 lat (usec): min=533, max=70116, avg=8351.59, stdev=8504.79 clat percentiles (usec): | 1.00th=[ 603], 5.00th=[ 717], 10.00th=[ 766], 20.00th=[ 938], | 30.00th=[ 1483], 40.00th=[ 3359], 50.00th=[ 6390], 60.00th=[ 8586], | 70.00th=[11076], 80.00th=[14484], 90.00th=[19530], 95.00th=[25035], | 99.00th=[36963], 99.50th=[41681], 99.90th=[61080], 99.95th=[69731], | 99.99th=[69731] bw ( KiB/s): min= 47, max= 368, per=25.67%, avg=248.98, stdev=62.70, samples=50 iops : min= 11, max= 92, avg=62.04, stdev=15.73, samples=50 lat (usec) : 100=3.29%, 250=5.57%, 500=0.26%, 750=4.30%, 1000=7.13% lat (msec) : 2=6.84%, 4=6.74%, 10=34.34%, 20=24.58%, 50=6.77% lat (msec) : 100=0.16%, 250=0.03% cpu : usr=0.09%, sys=16.22%, ctx=5124, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1242: Thu May 1 20:38:24 2025 read: IOPS=58, BW=233KiB/s (239kB/s)(5940KiB/25499msec) clat (usec): min=59, max=46647, avg=7304.74, stdev=6123.47 lat (usec): min=60, max=46649, avg=7305.84, stdev=6123.59 clat percentiles (usec): | 1.00th=[ 79], 5.00th=[ 94], 10.00th=[ 105], 20.00th=[ 433], | 30.00th=[ 4293], 40.00th=[ 5276], 50.00th=[ 6325], 60.00th=[ 7570], | 70.00th=[ 9110], 80.00th=[11469], 90.00th=[15008], 95.00th=[18744], | 99.00th=[27657], 99.50th=[30278], 99.90th=[44303], 99.95th=[46400], | 99.99th=[46400] bw ( KiB/s): min= 56, max= 360, per=25.25%, avg=233.82, stdev=61.06, samples=50 iops : min= 14, max= 90, avg=58.18, stdev=15.36, samples=50 write: IOPS=62, BW=249KiB/s (255kB/s)(6348KiB/25499msec) clat (usec): min=385, max=240550, avg=9122.61, stdev=11054.90 lat (usec): min=386, max=240552, avg=9123.79, stdev=11054.92 clat percentiles (usec): | 1.00th=[ 611], 5.00th=[ 709], 10.00th=[ 758], 20.00th=[ 914], | 30.00th=[ 1647], 40.00th=[ 3785], 50.00th=[ 6783], 60.00th=[ 9372], | 70.00th=[ 11731], 80.00th=[ 15533], 90.00th=[ 21103], 95.00th=[ 25822], | 99.00th=[ 38011], 99.50th=[ 49021], 99.90th=[ 99091], 99.95th=[240124], | 99.99th=[240124] bw ( KiB/s): min= 112, max= 424, per=25.54%, avg=247.74, stdev=55.15, samples=50 iops : min= 28, max= 106, avg=61.66, stdev=13.86, samples=50 lat (usec) : 100=3.94%, 250=5.24%, 500=0.78%, 750=4.65%, 1000=6.80% lat (msec) : 2=5.89%, 4=6.93%, 10=33.63%, 20=24.41%, 50=7.49% lat (msec) : 100=0.20%, 250=0.03% cpu : usr=0.25%, sys=15.54%, ctx=5225, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1244: Thu May 1 20:38:24 2025 read: IOPS=58, BW=235KiB/s (240kB/s)(6028KiB/25682msec) clat (usec): min=49, max=243139, avg=7944.86, stdev=9032.30 lat (usec): min=49, max=243140, avg=7945.95, stdev=9032.38 clat percentiles (usec): | 1.00th=[ 78], 5.00th=[ 94], 10.00th=[ 109], 20.00th=[ 2606], | 30.00th=[ 4490], 40.00th=[ 5538], 50.00th=[ 6521], 60.00th=[ 7832], | 70.00th=[ 9765], 80.00th=[ 12387], 90.00th=[ 16188], 95.00th=[ 19530], | 99.00th=[ 28181], 99.50th=[ 33162], 99.90th=[ 91751], 99.95th=[242222], | 99.99th=[242222] bw ( KiB/s): min= 56, max= 392, per=25.26%, avg=233.92, stdev=60.84, samples=51 iops : min= 14, max= 98, avg=58.29, stdev=15.33, samples=51 write: IOPS=60, BW=244KiB/s (250kB/s)(6260KiB/25682msec) clat (usec): min=442, max=111984, avg=8653.57, stdev=8809.87 lat (usec): min=443, max=111984, avg=8655.24, stdev=8810.01 clat percentiles (usec): | 1.00th=[ 562], 5.00th=[ 701], 10.00th=[ 758], 20.00th=[ 955], | 30.00th=[ 1811], 40.00th=[ 4178], 50.00th=[ 6915], 60.00th=[ 9110], | 70.00th=[ 11600], 80.00th=[ 14615], 90.00th=[ 19268], 95.00th=[ 23725], | 99.00th=[ 35390], 99.50th=[ 45876], 99.90th=[ 80217], 99.95th=[111674], | 99.99th=[111674] bw ( KiB/s): min= 64, max= 376, per=24.95%, avg=242.04, stdev=57.09, samples=51 iops : min= 16, max= 94, avg=60.33, stdev=14.29, samples=51 lat (usec) : 50=0.03%, 100=3.65%, 250=5.47%, 500=0.39%, 750=4.65% lat (usec) : 1000=6.28% lat (msec) : 2=5.18%, 4=7.06%, 10=34.47%, 20=25.91%, 50=6.67% lat (msec) : 100=0.16%, 250=0.07% cpu : usr=0.20%, sys=15.77%, ctx=5210, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1246: Thu May 1 20:38:24 2025 read: IOPS=58, BW=234KiB/s (239kB/s)(6052KiB/25907msec) clat (usec): min=51, max=85569, avg=7509.01, stdev=6660.48 lat (usec): min=51, max=85571, avg=7510.17, stdev=6660.34 clat percentiles (usec): | 1.00th=[ 74], 5.00th=[ 93], 10.00th=[ 102], 20.00th=[ 208], | 30.00th=[ 4228], 40.00th=[ 5407], 50.00th=[ 6521], 60.00th=[ 7898], | 70.00th=[ 9634], 80.00th=[11994], 90.00th=[15533], 95.00th=[18744], | 99.00th=[27657], 99.50th=[32113], 99.90th=[62129], 99.95th=[85459], | 99.99th=[85459] bw ( KiB/s): min= 144, max= 352, per=25.52%, avg=236.34, stdev=50.34, samples=50 iops : min= 36, max= 88, avg=58.86, stdev=12.55, samples=50 write: IOPS=60, BW=241KiB/s (246kB/s)(6236KiB/25907msec) clat (usec): min=471, max=378687, avg=9217.40, stdev=13565.16 lat (usec): min=471, max=378689, avg=9220.15, stdev=13565.69 clat percentiles (usec): | 1.00th=[ 611], 5.00th=[ 709], 10.00th=[ 766], 20.00th=[ 947], | 30.00th=[ 1729], 40.00th=[ 3916], 50.00th=[ 6849], 60.00th=[ 8979], | 70.00th=[ 11469], 80.00th=[ 15139], 90.00th=[ 21103], 95.00th=[ 25822], | 99.00th=[ 38011], 99.50th=[ 64226], 99.90th=[101188], 99.95th=[379585], | 99.99th=[379585] bw ( KiB/s): min= 128, max= 416, per=25.07%, avg=243.22, stdev=59.37, samples=50 iops : min= 32, max= 104, avg=60.58, stdev=14.81, samples=50 lat (usec) : 100=4.52%, 250=5.53%, 500=0.55%, 750=4.56%, 1000=6.61% lat (msec) : 2=5.96%, 4=6.64%, 10=33.53%, 20=24.28%, 50=7.45% lat (msec) : 100=0.29%, 250=0.03%, 500=0.03% cpu : usr=0.20%, sys=15.49%, ctx=5095, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=927KiB/s (949kB/s), 233KiB/s-237KiB/s (239kB/s-242kB/s), io=23.4MiB (24.6MB), run=25333-25907msec WRITE: bw=970KiB/s (994kB/s), 241KiB/s-249KiB/s (246kB/s-255kB/s), io=24.6MiB (25.7MB), run=25333-25907msec rand-rw: (groupid=0, jobs=1): err= 0: pid=1239: Thu May 1 20:39:43 2025 read: IOPS=14, BW=56.8KiB/s (58.1kB/s)(5992KiB/105542msec) clat (msec): min=2, max=250, avg= 8.95, stdev= 9.25 lat (msec): min=2, max=250, avg= 8.95, stdev= 9.25 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 4], 10.00th=[ 5], 20.00th=[ 5], | 30.00th=[ 6], 40.00th=[ 7], 50.00th=[ 7], 60.00th=[ 8], | 70.00th=[ 10], 80.00th=[ 12], 90.00th=[ 16], 95.00th=[ 21], | 99.00th=[ 37], 99.50th=[ 44], 99.90th=[ 94], 99.95th=[ 251], | 99.99th=[ 251] bw ( KiB/s): min= 7, max= 160, per=25.05%, avg=56.87, stdev=29.15, samples=208 iops : min= 1, max= 40, avg=13.95, stdev= 7.32, samples=208 write: IOPS=14, BW=59.7KiB/s (61.1kB/s)(6296KiB/105542msec) clat (msec): min=20, max=151, avg=58.43, stdev=21.10 lat (msec): min=20, max=151, avg=58.43, stdev=21.10 clat percentiles (msec): | 1.00th=[ 25], 5.00th=[ 31], 10.00th=[ 35], 20.00th=[ 41], | 30.00th=[ 46], 40.00th=[ 51], 50.00th=[ 55], 60.00th=[ 61], | 70.00th=[ 66], 80.00th=[ 74], 90.00th=[ 87], 95.00th=[ 100], | 99.00th=[ 124], 99.50th=[ 133], 99.90th=[ 146], 99.95th=[ 153], | 99.99th=[ 153] bw ( KiB/s): min= 24, max= 111, per=25.02%, avg=59.54, stdev=14.85, samples=209 iops : min= 6, max= 27, avg=14.61, stdev= 3.77, samples=209 lat (msec) : 4=4.00%, 10=32.78%, 20=9.37%, 50=22.49%, 100=28.84% lat (msec) : 250=2.47%, 500=0.03% cpu : usr=0.07%, sys=3.00%, ctx=3515, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1241: Thu May 1 20:39:43 2025 read: IOPS=14, BW=56.2KiB/s (57.6kB/s)(5940KiB/105606msec) clat (msec): min=2, max=224, avg= 9.44, stdev= 9.61 lat (msec): min=2, max=224, avg= 9.44, stdev= 9.61 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 4], 10.00th=[ 5], 20.00th=[ 5], | 30.00th=[ 6], 40.00th=[ 7], 50.00th=[ 8], 60.00th=[ 9], | 70.00th=[ 10], 80.00th=[ 12], 90.00th=[ 16], 95.00th=[ 21], | 99.00th=[ 44], 99.50th=[ 56], 99.90th=[ 108], 99.95th=[ 226], | 99.99th=[ 226] bw ( KiB/s): min= 7, max= 143, per=24.72%, avg=56.12, stdev=27.11, samples=210 iops : min= 1, max= 35, avg=13.78, stdev= 6.81, samples=210 write: IOPS=15, BW=60.1KiB/s (61.6kB/s)(6348KiB/105606msec) clat (msec): min=19, max=162, avg=57.57, stdev=20.78 lat (msec): min=19, max=162, avg=57.57, stdev=20.78 clat percentiles (msec): | 1.00th=[ 26], 5.00th=[ 32], 10.00th=[ 35], 20.00th=[ 41], | 30.00th=[ 45], 40.00th=[ 50], 50.00th=[ 55], 60.00th=[ 60], | 70.00th=[ 65], 80.00th=[ 72], 90.00th=[ 85], 95.00th=[ 97], | 99.00th=[ 126], 99.50th=[ 132], 99.90th=[ 161], 99.95th=[ 163], | 99.99th=[ 163] bw ( KiB/s): min= 8, max= 112, per=25.07%, avg=59.66, stdev=16.03, samples=211 iops : min= 2, max= 28, avg=14.66, stdev= 4.03, samples=211 lat (msec) : 4=3.55%, 10=31.64%, 20=10.68%, 50=23.70%, 100=28.29% lat (msec) : 250=2.15% cpu : usr=0.05%, sys=3.05%, ctx=3515, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1243: Thu May 1 20:39:43 2025 read: IOPS=14, BW=57.5KiB/s (58.8kB/s)(6028KiB/104896msec) clat (usec): min=2399, max=89243, avg=8967.52, stdev=7165.57 lat (usec): min=2400, max=89244, avg=8968.97, stdev=7165.66 clat percentiles (usec): | 1.00th=[ 2966], 5.00th=[ 3654], 10.00th=[ 4113], 20.00th=[ 4817], | 30.00th=[ 5407], 40.00th=[ 6128], 50.00th=[ 6980], 60.00th=[ 8029], | 70.00th=[ 9372], 80.00th=[11338], 90.00th=[15008], 95.00th=[20317], | 99.00th=[43254], 99.50th=[53740], 99.90th=[65799], 99.95th=[89654], | 99.99th=[89654] bw ( KiB/s): min= 7, max= 159, per=25.44%, avg=57.74, stdev=27.39, samples=207 iops : min= 1, max= 39, avg=14.24, stdev= 6.88, samples=207 write: IOPS=14, BW=59.7KiB/s (61.1kB/s)(6260KiB/104896msec) clat (msec): min=16, max=327, avg=58.27, stdev=21.39 lat (msec): min=16, max=327, avg=58.28, stdev=21.39 clat percentiles (msec): | 1.00th=[ 26], 5.00th=[ 32], 10.00th=[ 36], 20.00th=[ 42], | 30.00th=[ 46], 40.00th=[ 51], 50.00th=[ 55], 60.00th=[ 60], | 70.00th=[ 66], 80.00th=[ 74], 90.00th=[ 85], 95.00th=[ 96], | 99.00th=[ 124], 99.50th=[ 132], 99.90th=[ 157], 99.95th=[ 330], | 99.99th=[ 330] bw ( KiB/s): min= 8, max= 104, per=24.93%, avg=59.33, stdev=15.19, samples=209 iops : min= 2, max= 26, avg=14.63, stdev= 3.86, samples=209 lat (msec) : 4=4.07%, 10=32.03%, 20=10.55%, 50=22.27%, 100=29.07% lat (msec) : 250=1.99%, 500=0.03% cpu : usr=0.05%, sys=3.09%, ctx=3544, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1245: Thu May 1 20:39:43 2025 read: IOPS=14, BW=57.7KiB/s (59.0kB/s)(6052KiB/104965msec) clat (msec): min=2, max=107, avg= 9.09, stdev= 7.53 lat (msec): min=2, max=107, avg= 9.09, stdev= 7.53 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 4], 10.00th=[ 5], 20.00th=[ 5], | 30.00th=[ 6], 40.00th=[ 6], 50.00th=[ 7], 60.00th=[ 8], | 70.00th=[ 10], 80.00th=[ 12], 90.00th=[ 17], 95.00th=[ 21], | 99.00th=[ 39], 99.50th=[ 48], 99.90th=[ 100], 99.95th=[ 108], | 99.99th=[ 108] bw ( KiB/s): min= 7, max= 135, per=25.66%, avg=58.26, stdev=29.29, samples=206 iops : min= 1, max= 33, avg=14.33, stdev= 7.31, samples=206 write: IOPS=14, BW=59.4KiB/s (60.8kB/s)(6236KiB/104965msec) clat (msec): min=17, max=375, avg=58.39, stdev=21.99 lat (msec): min=17, max=375, avg=58.39, stdev=22.00 clat percentiles (msec): | 1.00th=[ 25], 5.00th=[ 32], 10.00th=[ 36], 20.00th=[ 42], | 30.00th=[ 46], 40.00th=[ 51], 50.00th=[ 55], 60.00th=[ 61], | 70.00th=[ 66], 80.00th=[ 73], 90.00th=[ 86], 95.00th=[ 97], | 99.00th=[ 125], 99.50th=[ 133], 99.90th=[ 150], 99.95th=[ 376], | 99.99th=[ 376] bw ( KiB/s): min= 8, max= 104, per=24.79%, avg=58.99, stdev=15.13, samples=209 iops : min= 2, max= 26, avg=14.52, stdev= 3.80, samples=209 lat (msec) : 4=4.75%, 10=31.32%, 20=10.51%, 50=22.27%, 100=29.07% lat (msec) : 250=2.05%, 500=0.03% cpu : usr=0.05%, sys=3.12%, ctx=3568, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=227KiB/s (233kB/s), 56.2KiB/s-57.7KiB/s (57.6kB/s-59.0kB/s), io=23.4MiB (24.6MB), run=104896-105606msec WRITE: bw=238KiB/s (244kB/s), 59.4KiB/s-60.1KiB/s (60.8kB/s-61.6kB/s), io=24.6MiB (25.7MB), run=104896-105606msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=1341: Thu May 1 20:39:50 2025 read: IOPS=73, BW=1182KiB/s (1210kB/s)(5920KiB/5009msec) clat (usec): min=50, max=30099, avg=5502.05, stdev=5726.57 lat (usec): min=50, max=30101, avg=5503.35, stdev=5726.59 clat percentiles (usec): | 1.00th=[ 56], 5.00th=[ 85], 10.00th=[ 95], 20.00th=[ 108], | 30.00th=[ 122], 40.00th=[ 322], 50.00th=[ 4948], 60.00th=[ 6390], | 70.00th=[ 8029], 80.00th=[10159], 90.00th=[13566], 95.00th=[15795], | 99.00th=[23987], 99.50th=[28181], 99.90th=[30016], 99.95th=[30016], | 99.99th=[30016] bw ( KiB/s): min= 192, max= 1664, per=26.82%, avg=1150.89, stdev=434.22, samples=9 iops : min= 12, max= 104, avg=71.67, stdev=27.06, samples=9 write: IOPS=79, BW=1271KiB/s (1302kB/s)(6368KiB/5009msec) clat (usec): min=709, max=69565, avg=6816.59, stdev=8634.70 lat (usec): min=710, max=69566, avg=6818.61, stdev=8634.60 clat percentiles (usec): | 1.00th=[ 816], 5.00th=[ 955], 10.00th=[ 1012], 20.00th=[ 1123], | 30.00th=[ 1336], 40.00th=[ 1762], 50.00th=[ 3228], 60.00th=[ 5669], | 70.00th=[ 8356], 80.00th=[11600], 90.00th=[16450], 95.00th=[20579], | 99.00th=[42206], 99.50th=[65799], 99.90th=[69731], 99.95th=[69731], | 99.99th=[69731] bw ( KiB/s): min= 160, max= 2011, per=27.77%, avg=1253.89, stdev=490.60, samples=9 iops : min= 10, max= 125, avg=78.11, stdev=30.47, samples=9 lat (usec) : 100=6.12%, 250=12.89%, 500=0.78%, 750=0.39%, 1000=4.69% lat (msec) : 2=17.32%, 4=6.77%, 10=28.12%, 20=19.66%, 50=2.86% lat (msec) : 100=0.39% cpu : usr=0.12%, sys=18.43%, ctx=1129, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1343: Thu May 1 20:39:50 2025 read: IOPS=64, BW=1029KiB/s (1054kB/s)(5744KiB/5581msec) clat (usec): min=50, max=51601, avg=5251.78, stdev=7083.50 lat (usec): min=51, max=51602, avg=5252.66, stdev=7083.48 clat percentiles (usec): | 1.00th=[ 59], 5.00th=[ 84], 10.00th=[ 94], 20.00th=[ 105], | 30.00th=[ 118], 40.00th=[ 139], 50.00th=[ 3851], 60.00th=[ 5211], | 70.00th=[ 7701], 80.00th=[10028], 90.00th=[13042], 95.00th=[15926], | 99.00th=[41681], 99.50th=[50070], 99.90th=[51643], 99.95th=[51643], | 99.99th=[51643] bw ( KiB/s): min= 31, max= 1792, per=23.55%, avg=1010.45, stdev=573.78, samples=11 iops : min= 1, max= 112, avg=62.73, stdev=36.07, samples=11 write: IOPS=73, BW=1173KiB/s (1201kB/s)(6544KiB/5581msec) clat (usec): min=716, max=126201, avg=8396.85, stdev=14974.96 lat (usec): min=717, max=126204, avg=8401.36, stdev=14975.00 clat percentiles (usec): | 1.00th=[ 766], 5.00th=[ 930], 10.00th=[ 1004], 20.00th=[ 1156], | 30.00th=[ 1369], 40.00th=[ 1975], 50.00th=[ 4555], 60.00th=[ 6587], | 70.00th=[ 8848], 80.00th=[ 11994], 90.00th=[ 16712], 95.00th=[ 21365], | 99.00th=[ 95945], 99.50th=[108528], 99.90th=[126354], 99.95th=[126354], | 99.99th=[126354] bw ( KiB/s): min= 95, max= 2272, per=25.57%, avg=1154.45, stdev=645.08, samples=11 iops : min= 5, max= 142, avg=71.73, stdev=40.50, samples=11 lat (usec) : 100=7.29%, 250=13.41%, 500=1.30%, 750=0.78%, 1000=4.43% lat (msec) : 2=16.80%, 4=5.99%, 10=27.86%, 20=18.23%, 50=2.34% lat (msec) : 100=1.04%, 250=0.52% cpu : usr=0.18%, sys=16.51%, ctx=1142, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1344: Thu May 1 20:39:50 2025 read: IOPS=81, BW=1305KiB/s (1336kB/s)(6256KiB/4794msec) clat (usec): min=62, max=40198, avg=5777.81, stdev=6292.71 lat (usec): min=62, max=40199, avg=5778.68, stdev=6292.74 clat percentiles (usec): | 1.00th=[ 84], 5.00th=[ 88], 10.00th=[ 94], 20.00th=[ 106], | 30.00th=[ 121], 40.00th=[ 277], 50.00th=[ 5276], 60.00th=[ 6521], | 70.00th=[ 8586], 80.00th=[10552], 90.00th=[13829], 95.00th=[16450], | 99.00th=[29230], 99.50th=[33817], 99.90th=[40109], 99.95th=[40109], | 99.99th=[40109] bw ( KiB/s): min= 640, max= 2011, per=30.21%, avg=1296.33, stdev=496.62, samples=9 iops : min= 40, max= 125, avg=80.78, stdev=30.81, samples=9 write: IOPS=78, BW=1258KiB/s (1288kB/s)(6032KiB/4794msec) clat (usec): min=612, max=42740, avg=6516.45, stdev=6963.43 lat (usec): min=614, max=42741, avg=6517.96, stdev=6963.41 clat percentiles (usec): | 1.00th=[ 840], 5.00th=[ 955], 10.00th=[ 1037], 20.00th=[ 1156], | 30.00th=[ 1385], 40.00th=[ 1958], 50.00th=[ 3326], 60.00th=[ 6390], | 70.00th=[ 7963], 80.00th=[11207], 90.00th=[17433], 95.00th=[21365], | 99.00th=[28443], 99.50th=[34866], 99.90th=[42730], 99.95th=[42730], | 99.99th=[42730] bw ( KiB/s): min= 768, max= 1724, per=27.14%, avg=1225.56, stdev=317.04, samples=9 iops : min= 48, max= 107, avg=76.33, stdev=19.63, samples=9 lat (usec) : 100=7.42%, 250=12.76%, 500=0.78%, 750=0.39%, 1000=3.52% lat (msec) : 2=16.54%, 4=7.42%, 10=28.52%, 20=18.36%, 50=4.30% cpu : usr=0.25%, sys=18.49%, ctx=1105, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1345: Thu May 1 20:39:50 2025 read: IOPS=79, BW=1271KiB/s (1301kB/s)(6032KiB/4747msec) clat (usec): min=67, max=30331, avg=6042.70, stdev=5614.24 lat (usec): min=67, max=30333, avg=6043.69, stdev=5614.29 clat percentiles (usec): | 1.00th=[ 83], 5.00th=[ 91], 10.00th=[ 104], 20.00th=[ 115], | 30.00th=[ 178], 40.00th=[ 4555], 50.00th=[ 5669], 60.00th=[ 6783], | 70.00th=[ 8455], 80.00th=[10290], 90.00th=[13435], 95.00th=[15926], | 99.00th=[25560], 99.50th=[25822], 99.90th=[30278], 99.95th=[30278], | 99.99th=[30278] bw ( KiB/s): min= 800, max= 1788, per=29.48%, avg=1265.00, stdev=316.97, samples=9 iops : min= 50, max= 111, avg=78.89, stdev=19.59, samples=9 write: IOPS=82, BW=1318KiB/s (1350kB/s)(6256KiB/4747msec) clat (usec): min=796, max=36267, avg=6209.56, stdev=6724.10 lat (usec): min=798, max=36268, avg=6210.90, stdev=6724.20 clat percentiles (usec): | 1.00th=[ 857], 5.00th=[ 1020], 10.00th=[ 1090], 20.00th=[ 1188], | 30.00th=[ 1352], 40.00th=[ 1844], 50.00th=[ 2638], 60.00th=[ 5276], | 70.00th=[ 7701], 80.00th=[11076], 90.00th=[15533], 95.00th=[19792], | 99.00th=[31327], 99.50th=[32637], 99.90th=[36439], 99.95th=[36439], | 99.99th=[36439] bw ( KiB/s): min= 864, max= 2043, per=29.27%, avg=1321.67, stdev=388.56, samples=9 iops : min= 54, max= 127, avg=82.44, stdev=24.08, samples=9 lat (usec) : 100=3.52%, 250=11.85%, 500=0.65%, 1000=2.60% lat (msec) : 2=19.92%, 4=7.94%, 10=30.47%, 20=19.66%, 50=3.39% cpu : usr=0.23%, sys=19.83%, ctx=1144, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=4292KiB/s (4395kB/s), 1029KiB/s-1305KiB/s (1054kB/s-1336kB/s), io=23.4MiB (24.5MB), run=4747-5581msec WRITE: bw=4515KiB/s (4624kB/s), 1173KiB/s-1318KiB/s (1201kB/s-1350kB/s), io=24.6MiB (25.8MB), run=4747-5581msec rand-rw: (groupid=0, jobs=1): err= 0: pid=1338: Thu May 1 20:40:11 2025 read: IOPS=14, BW=225KiB/s (230kB/s)(5920KiB/26337msec) clat (usec): min=3119, max=83365, avg=8778.69, stdev=7441.14 lat (usec): min=3120, max=83366, avg=8780.93, stdev=7441.00 clat percentiles (usec): | 1.00th=[ 3261], 5.00th=[ 4113], 10.00th=[ 4555], 20.00th=[ 5080], | 30.00th=[ 5538], 40.00th=[ 6063], 50.00th=[ 6652], 60.00th=[ 7308], | 70.00th=[ 8356], 80.00th=[10159], 90.00th=[13698], 95.00th=[20055], | 99.00th=[38536], 99.50th=[45351], 99.90th=[83362], 99.95th=[83362], | 99.99th=[83362] bw ( KiB/s): min= 64, max= 672, per=25.09%, avg=221.29, stdev=111.59, samples=52 iops : min= 4, max= 42, avg=13.62, stdev= 7.04, samples=52 write: IOPS=15, BW=242KiB/s (248kB/s)(6368KiB/26337msec) clat (msec): min=27, max=117, avg=57.93, stdev=17.17 lat (msec): min=27, max=117, avg=57.94, stdev=17.17 clat percentiles (msec): | 1.00th=[ 30], 5.00th=[ 35], 10.00th=[ 39], 20.00th=[ 43], | 30.00th=[ 47], 40.00th=[ 52], 50.00th=[ 56], 60.00th=[ 61], | 70.00th=[ 66], 80.00th=[ 71], 90.00th=[ 82], 95.00th=[ 90], | 99.00th=[ 111], 99.50th=[ 118], 99.90th=[ 118], 99.95th=[ 118], | 99.99th=[ 118] bw ( KiB/s): min= 96, max= 288, per=25.90%, avg=240.37, stdev=39.77, samples=52 iops : min= 6, max= 18, avg=14.81, stdev= 2.48, samples=52 lat (msec) : 4=2.08%, 10=36.33%, 20=7.29%, 50=20.96%, 100=31.90% lat (msec) : 250=1.43% cpu : usr=0.02%, sys=3.27%, ctx=871, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1339: Thu May 1 20:40:11 2025 read: IOPS=13, BW=212KiB/s (217kB/s)(5744KiB/27143msec) clat (usec): min=3183, max=83309, avg=9074.83, stdev=8016.67 lat (usec): min=3184, max=83310, avg=9076.21, stdev=8016.79 clat percentiles (usec): | 1.00th=[ 3392], 5.00th=[ 4080], 10.00th=[ 4555], 20.00th=[ 5211], | 30.00th=[ 5735], 40.00th=[ 6259], 50.00th=[ 6783], 60.00th=[ 7373], | 70.00th=[ 8029], 80.00th=[10028], 90.00th=[15139], 95.00th=[25822], | 99.00th=[47973], 99.50th=[51643], 99.90th=[83362], 99.95th=[83362], | 99.99th=[83362] bw ( KiB/s): min= 31, max= 447, per=23.94%, avg=211.15, stdev=100.14, samples=54 iops : min= 1, max= 27, avg=12.85, stdev= 6.16, samples=54 write: IOPS=15, BW=241KiB/s (247kB/s)(6544KiB/27143msec) clat (msec): min=24, max=136, avg=58.06, stdev=18.51 lat (msec): min=24, max=136, avg=58.06, stdev=18.51 clat percentiles (msec): | 1.00th=[ 29], 5.00th=[ 33], 10.00th=[ 36], 20.00th=[ 43], | 30.00th=[ 48], 40.00th=[ 51], 50.00th=[ 56], 60.00th=[ 62], | 70.00th=[ 67], 80.00th=[ 72], 90.00th=[ 81], 95.00th=[ 90], | 99.00th=[ 117], 99.50th=[ 134], 99.90th=[ 138], 99.95th=[ 138], | 99.99th=[ 138] bw ( KiB/s): min= 95, max= 368, per=25.77%, avg=239.11, stdev=53.29, samples=54 iops : min= 5, max= 23, avg=14.63, stdev= 3.44, samples=54 lat (msec) : 4=1.95%, 10=35.16%, 20=6.64%, 50=22.92%, 100=32.29% lat (msec) : 250=1.04% cpu : usr=0.03%, sys=3.15%, ctx=862, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1340: Thu May 1 20:40:11 2025 read: IOPS=14, BW=239KiB/s (244kB/s)(6256KiB/26203msec) clat (usec): min=3041, max=82736, avg=9458.02, stdev=8087.34 lat (usec): min=3042, max=82737, avg=9459.33, stdev=8087.30 clat percentiles (usec): | 1.00th=[ 3589], 5.00th=[ 4113], 10.00th=[ 4555], 20.00th=[ 5342], | 30.00th=[ 5800], 40.00th=[ 6325], 50.00th=[ 6980], 60.00th=[ 7898], | 70.00th=[ 8848], 80.00th=[10683], 90.00th=[16712], 95.00th=[23987], | 99.00th=[47973], 99.50th=[49546], 99.90th=[82314], 99.95th=[82314], | 99.99th=[82314] bw ( KiB/s): min= 64, max= 542, per=27.25%, avg=240.37, stdev=118.92, samples=52 iops : min= 4, max= 33, avg=14.81, stdev= 7.44, samples=52 write: IOPS=14, BW=230KiB/s (236kB/s)(6032KiB/26203msec) clat (msec): min=25, max=151, avg=59.36, stdev=18.11 lat (msec): min=25, max=151, avg=59.36, stdev=18.10 clat percentiles (msec): | 1.00th=[ 29], 5.00th=[ 35], 10.00th=[ 39], 20.00th=[ 44], | 30.00th=[ 48], 40.00th=[ 53], 50.00th=[ 58], 60.00th=[ 63], | 70.00th=[ 67], 80.00th=[ 72], 90.00th=[ 82], 95.00th=[ 92], | 99.00th=[ 114], 99.50th=[ 131], 99.90th=[ 153], 99.95th=[ 153], | 99.99th=[ 153] bw ( KiB/s): min= 96, max= 320, per=24.71%, avg=229.31, stdev=44.72, samples=52 iops : min= 6, max= 20, avg=14.12, stdev= 2.85, samples=52 lat (msec) : 4=2.21%, 10=36.98%, 20=8.46%, 50=19.40%, 100=31.38% lat (msec) : 250=1.56% cpu : usr=0.00%, sys=3.40%, ctx=881, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1342: Thu May 1 20:40:11 2025 read: IOPS=14, BW=230KiB/s (236kB/s)(6032KiB/26187msec) clat (usec): min=3388, max=40647, avg=8209.19, stdev=5144.55 lat (usec): min=3389, max=40649, avg=8210.54, stdev=5144.52 clat percentiles (usec): | 1.00th=[ 3589], 5.00th=[ 4113], 10.00th=[ 4621], 20.00th=[ 5211], | 30.00th=[ 5669], 40.00th=[ 6325], 50.00th=[ 6718], 60.00th=[ 7373], | 70.00th=[ 8094], 80.00th=[ 9896], 90.00th=[13042], 95.00th=[16188], | 99.00th=[33162], 99.50th=[37487], 99.90th=[40633], 99.95th=[40633], | 99.99th=[40633] bw ( KiB/s): min= 31, max= 480, per=26.35%, avg=232.39, stdev=102.57, samples=51 iops : min= 1, max= 30, avg=14.18, stdev= 6.41, samples=51 write: IOPS=14, BW=239KiB/s (245kB/s)(6256KiB/26187msec) clat (msec): min=26, max=163, avg=58.44, stdev=17.79 lat (msec): min=26, max=163, avg=58.44, stdev=17.79 clat percentiles (msec): | 1.00th=[ 31], 5.00th=[ 36], 10.00th=[ 39], 20.00th=[ 43], | 30.00th=[ 47], 40.00th=[ 53], 50.00th=[ 57], 60.00th=[ 63], | 70.00th=[ 67], 80.00th=[ 72], 90.00th=[ 79], 95.00th=[ 87], | 99.00th=[ 113], 99.50th=[ 142], 99.90th=[ 163], 99.95th=[ 163], | 99.99th=[ 163] bw ( KiB/s): min= 95, max= 352, per=25.75%, avg=238.96, stdev=45.03, samples=52 iops : min= 5, max= 22, avg=14.58, stdev= 2.88, samples=52 lat (msec) : 4=2.08%, 10=37.63%, 20=7.29%, 50=20.31%, 100=31.90% lat (msec) : 250=0.78% cpu : usr=0.10%, sys=3.22%, ctx=871, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=882KiB/s (904kB/s), 212KiB/s-239KiB/s (217kB/s-244kB/s), io=23.4MiB (24.5MB), run=26187-27143msec WRITE: bw=928KiB/s (951kB/s), 230KiB/s-242KiB/s (236kB/s-248kB/s), io=24.6MiB (25.8MB), run=26187-27143msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=1375: Thu May 1 20:40:14 2025 read: IOPS=10, BW=10.3MiB/s (10.8MB/s)(7168KiB/679msec) clat (usec): min=622, max=76105, avg=48362.31, stdev=23396.63 lat (usec): min=623, max=76106, avg=48363.72, stdev=23396.60 clat percentiles (usec): | 1.00th=[ 619], 5.00th=[ 619], 10.00th=[ 619], 20.00th=[44303], | 30.00th=[50594], 40.00th=[50594], 50.00th=[52167], 60.00th=[52691], | 70.00th=[52691], 80.00th=[62129], 90.00th=[76022], 95.00th=[76022], | 99.00th=[76022], 99.50th=[76022], 99.90th=[76022], 99.95th=[76022], | 99.99th=[76022] bw ( KiB/s): min=10240, max=10240, per=56.80%, avg=10240.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 write: IOPS=7, BW=7541KiB/s (7721kB/s)(5120KiB/679msec) clat (usec): min=33376, max=76912, avg=51463.86, stdev=17730.44 lat (usec): min=33409, max=76951, avg=51504.62, stdev=17733.23 clat percentiles (usec): | 1.00th=[33424], 5.00th=[33424], 10.00th=[33424], 20.00th=[33424], | 30.00th=[35914], 40.00th=[35914], 50.00th=[52691], 60.00th=[52691], | 70.00th=[57934], 80.00th=[57934], 90.00th=[77071], 95.00th=[77071], | 99.00th=[77071], 99.50th=[77071], 99.90th=[77071], 99.95th=[77071], | 99.99th=[77071] bw ( KiB/s): min= 4096, max= 4096, per=16.23%, avg=4096.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (usec) : 750=8.33% lat (msec) : 50=25.00%, 100=66.67% cpu : usr=0.15%, sys=32.74%, ctx=622, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1376: Thu May 1 20:40:14 2025 read: IOPS=2, BW=2479KiB/s (2539kB/s)(2048KiB/826msec) clat (usec): min=48040, max=58505, avg=53273.14, stdev=7399.68 lat (usec): min=48042, max=58507, avg=53275.01, stdev=7399.47 clat percentiles (usec): | 1.00th=[47973], 5.00th=[47973], 10.00th=[47973], 20.00th=[47973], | 30.00th=[47973], 40.00th=[47973], 50.00th=[47973], 60.00th=[58459], | 70.00th=[58459], 80.00th=[58459], 90.00th=[58459], 95.00th=[58459], | 99.00th=[58459], 99.50th=[58459], 99.90th=[58459], 99.95th=[58459], | 99.99th=[58459] bw ( KiB/s): min= 2048, max= 2048, per=11.36%, avg=2048.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=12, BW=12.1MiB/s (12.7MB/s)(10.0MiB/826msec) clat (msec): min=25, max=120, avg=57.32, stdev=31.15 lat (msec): min=25, max=121, avg=57.38, stdev=31.15 clat percentiles (msec): | 1.00th=[ 26], 5.00th=[ 26], 10.00th=[ 26], 20.00th=[ 33], | 30.00th=[ 38], 40.00th=[ 39], 50.00th=[ 41], 60.00th=[ 50], | 70.00th=[ 72], 80.00th=[ 72], 90.00th=[ 100], 95.00th=[ 122], | 99.00th=[ 122], 99.50th=[ 122], 99.90th=[ 122], 99.95th=[ 122], | 99.99th=[ 122] bw ( KiB/s): min=10240, max=10240, per=40.57%, avg=10240.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 lat (msec) : 50=58.33%, 100=33.33%, 250=8.33% cpu : usr=0.12%, sys=29.33%, ctx=440, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1378: Thu May 1 20:40:14 2025 read: IOPS=9, BW=9511KiB/s (9739kB/s)(6144KiB/646msec) clat (usec): min=532, max=118281, avg=39215.61, stdev=47764.88 lat (usec): min=533, max=118283, avg=39217.28, stdev=47765.01 clat percentiles (usec): | 1.00th=[ 529], 5.00th=[ 529], 10.00th=[ 529], 20.00th=[ 611], | 30.00th=[ 611], 40.00th=[ 709], 50.00th=[ 709], 60.00th=[ 55313], | 70.00th=[ 60031], 80.00th=[ 60031], 90.00th=[117965], 95.00th=[117965], | 99.00th=[117965], 99.50th=[117965], 99.90th=[117965], 99.95th=[117965], | 99.99th=[117965] bw ( KiB/s): min= 4096, max= 4096, per=22.72%, avg=4096.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 write: IOPS=9, BW=9511KiB/s (9739kB/s)(6144KiB/646msec) clat (msec): min=27, max=148, avg=65.21, stdev=44.55 lat (msec): min=27, max=149, avg=65.26, stdev=44.57 clat percentiles (msec): | 1.00th=[ 28], 5.00th=[ 28], 10.00th=[ 28], 20.00th=[ 35], | 30.00th=[ 35], 40.00th=[ 43], 50.00th=[ 43], 60.00th=[ 68], | 70.00th=[ 71], 80.00th=[ 71], 90.00th=[ 150], 95.00th=[ 150], | 99.00th=[ 150], 99.50th=[ 150], 99.90th=[ 150], 99.95th=[ 150], | 99.99th=[ 150] bw ( KiB/s): min= 8192, max= 8192, per=32.46%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 lat (usec) : 750=25.00% lat (msec) : 50=25.00%, 100=33.33%, 250=16.67% cpu : usr=0.31%, sys=28.68%, ctx=117, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1380: Thu May 1 20:40:14 2025 read: IOPS=4, BW=4507KiB/s (4615kB/s)(5120KiB/1136msec) clat (usec): min=493, max=112883, avg=41338.21, stdev=45998.94 lat (usec): min=494, max=112885, avg=41339.42, stdev=45998.85 clat percentiles (usec): | 1.00th=[ 494], 5.00th=[ 494], 10.00th=[ 494], 20.00th=[ 494], | 30.00th=[ 1020], 40.00th=[ 1020], 50.00th=[ 44827], 60.00th=[ 44827], | 70.00th=[ 47449], 80.00th=[ 47449], 90.00th=[112722], 95.00th=[112722], | 99.00th=[112722], 99.50th=[112722], 99.90th=[112722], 99.95th=[112722], | 99.99th=[112722] bw ( KiB/s): min= 8192, max= 8192, per=45.44%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=6, BW=6310KiB/s (6461kB/s)(7168KiB/1136msec) clat (msec): min=30, max=375, avg=104.94, stdev=122.24 lat (msec): min=30, max=375, avg=104.99, stdev=122.24 clat percentiles (msec): | 1.00th=[ 31], 5.00th=[ 31], 10.00th=[ 31], 20.00th=[ 48], | 30.00th=[ 51], 40.00th=[ 51], 50.00th=[ 52], 60.00th=[ 66], | 70.00th=[ 66], 80.00th=[ 114], 90.00th=[ 376], 95.00th=[ 376], | 99.00th=[ 376], 99.50th=[ 376], 99.90th=[ 376], 99.95th=[ 376], | 99.99th=[ 376] bw ( KiB/s): min= 2048, max=10240, per=24.34%, avg=6144.00, stdev=5792.62, samples=2 iops : min= 2, max= 10, avg= 6.00, stdev= 5.66, samples=2 lat (usec) : 500=8.33% lat (msec) : 2=8.33%, 50=33.33%, 100=25.00%, 250=16.67%, 500=8.33% cpu : usr=0.00%, sys=19.03%, ctx=767, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=17.6MiB/s (18.5MB/s), 2479KiB/s-10.3MiB/s (2539kB/s-10.8MB/s), io=20.0MiB (20.0MB), run=646-1136msec WRITE: bw=24.6MiB/s (25.8MB/s), 6310KiB/s-12.1MiB/s (6461kB/s-12.7MB/s), io=28.0MiB (29.4MB), run=646-1136msec rand-rw: (groupid=0, jobs=1): err= 0: pid=1377: Thu May 1 20:40:14 2025 read: IOPS=6, BW=6558KiB/s (6715kB/s)(7168KiB/1093msec) clat (msec): min=34, max=116, avg=64.49, stdev=31.38 lat (msec): min=34, max=116, avg=64.49, stdev=31.38 clat percentiles (msec): | 1.00th=[ 35], 5.00th=[ 35], 10.00th=[ 35], 20.00th=[ 37], | 30.00th=[ 39], 40.00th=[ 39], 50.00th=[ 63], 60.00th=[ 72], | 70.00th=[ 72], 80.00th=[ 93], 90.00th=[ 116], 95.00th=[ 116], | 99.00th=[ 116], 99.50th=[ 116], 99.90th=[ 116], 99.95th=[ 116], | 99.99th=[ 116] bw ( KiB/s): min= 5495, max= 6131, per=32.10%, avg=5813.00, stdev=449.72, samples=2 iops : min= 5, max= 5, avg= 5.00, stdev= 0.00, samples=2 write: IOPS=4, BW=4684KiB/s (4797kB/s)(5120KiB/1093msec) clat (msec): min=41, max=207, avg=92.67, stdev=67.43 lat (msec): min=41, max=207, avg=93.24, stdev=67.39 clat percentiles (msec): | 1.00th=[ 42], 5.00th=[ 42], 10.00th=[ 42], 20.00th=[ 42], | 30.00th=[ 49], 40.00th=[ 49], 50.00th=[ 73], 60.00th=[ 73], | 70.00th=[ 93], 80.00th=[ 93], 90.00th=[ 207], 95.00th=[ 207], | 99.00th=[ 207], 99.50th=[ 207], 99.90th=[ 207], 99.95th=[ 207], | 99.99th=[ 207] bw ( KiB/s): min= 9159, max= 9159, per=36.13%, avg=9159.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 lat (msec) : 50=41.67%, 100=41.67%, 250=16.67% cpu : usr=0.37%, sys=3.75%, ctx=38, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1379: Thu May 1 20:40:14 2025 read: IOPS=1, BW=1822KiB/s (1866kB/s)(2048KiB/1124msec) clat (usec): min=43462, max=59614, avg=51538.14, stdev=11421.22 lat (usec): min=43463, max=59615, avg=51539.57, stdev=11421.42 clat percentiles (usec): | 1.00th=[43254], 5.00th=[43254], 10.00th=[43254], 20.00th=[43254], | 30.00th=[43254], 40.00th=[43254], 50.00th=[43254], 60.00th=[59507], | 70.00th=[59507], 80.00th=[59507], 90.00th=[59507], 95.00th=[59507], | 99.00th=[59507], 99.50th=[59507], 99.90th=[59507], 99.95th=[59507], | 99.99th=[59507] bw ( KiB/s): min= 1828, max= 2048, per=10.70%, avg=1938.00, stdev=155.56, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 write: IOPS=8, BW=9110KiB/s (9329kB/s)(10.0MiB/1124msec) clat (msec): min=39, max=178, avg=84.56, stdev=41.49 lat (msec): min=39, max=178, avg=84.76, stdev=41.43 clat percentiles (msec): | 1.00th=[ 40], 5.00th=[ 40], 10.00th=[ 40], 20.00th=[ 51], | 30.00th=[ 55], 40.00th=[ 61], 50.00th=[ 68], 60.00th=[ 78], | 70.00th=[ 111], 80.00th=[ 111], 90.00th=[ 116], 95.00th=[ 178], | 99.00th=[ 178], 99.50th=[ 178], 99.90th=[ 178], 99.95th=[ 178], | 99.99th=[ 178] bw ( KiB/s): min= 6144, max= 9142, per=30.15%, avg=7643.00, stdev=2119.91, samples=2 iops : min= 6, max= 8, avg= 7.00, stdev= 1.41, samples=2 lat (msec) : 50=25.00%, 100=50.00%, 250=25.00% cpu : usr=0.27%, sys=4.80%, ctx=40, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1381: Thu May 1 20:40:14 2025 read: IOPS=5, BW=5432KiB/s (5563kB/s)(6144KiB/1131msec) clat (msec): min=26, max=130, avg=72.06, stdev=40.40 lat (msec): min=26, max=130, avg=72.06, stdev=40.40 clat percentiles (msec): | 1.00th=[ 27], 5.00th=[ 27], 10.00th=[ 27], 20.00th=[ 39], | 30.00th=[ 39], 40.00th=[ 51], 50.00th=[ 51], 60.00th=[ 85], | 70.00th=[ 104], 80.00th=[ 104], 90.00th=[ 131], 95.00th=[ 131], | 99.00th=[ 131], 99.50th=[ 131], 99.90th=[ 131], 99.95th=[ 131], | 99.99th=[ 131] bw ( KiB/s): min= 9061, max= 9061, per=50.04%, avg=9061.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=5, BW=5432KiB/s (5563kB/s)(6144KiB/1131msec) clat (msec): min=40, max=264, avg=110.93, stdev=82.85 lat (msec): min=40, max=264, avg=111.00, stdev=82.85 clat percentiles (msec): | 1.00th=[ 41], 5.00th=[ 41], 10.00th=[ 41], 20.00th=[ 59], | 30.00th=[ 59], 40.00th=[ 73], 50.00th=[ 73], 60.00th=[ 87], | 70.00th=[ 144], 80.00th=[ 144], 90.00th=[ 264], 95.00th=[ 264], | 99.00th=[ 264], 99.50th=[ 264], 99.90th=[ 264], 99.95th=[ 264], | 99.99th=[ 264] bw ( KiB/s): min= 5437, max= 6144, per=22.84%, avg=5790.50, stdev=499.92, samples=2 iops : min= 5, max= 6, avg= 5.50, stdev= 0.71, samples=2 lat (msec) : 50=25.00%, 100=41.67%, 250=25.00%, 500=8.33% cpu : usr=0.18%, sys=4.51%, ctx=46, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1382: Thu May 1 20:40:14 2025 read: IOPS=5, BW=5708KiB/s (5845kB/s)(5120KiB/897msec) clat (usec): min=15071, max=61773, avg=34241.72, stdev=19916.91 lat (usec): min=15072, max=61774, avg=34242.85, stdev=19917.19 clat percentiles (usec): | 1.00th=[15008], 5.00th=[15008], 10.00th=[15008], 20.00th=[15008], | 30.00th=[17695], 40.00th=[17695], 50.00th=[29492], 60.00th=[29492], | 70.00th=[46924], 80.00th=[46924], 90.00th=[61604], 95.00th=[61604], | 99.00th=[61604], 99.50th=[61604], 99.90th=[61604], 99.95th=[61604], | 99.99th=[61604] write: IOPS=7, BW=7991KiB/s (8183kB/s)(7168KiB/897msec) clat (msec): min=45, max=156, avg=99.54, stdev=49.09 lat (msec): min=45, max=156, avg=99.61, stdev=49.09 clat percentiles (msec): | 1.00th=[ 46], 5.00th=[ 46], 10.00th=[ 46], 20.00th=[ 46], | 30.00th=[ 74], 40.00th=[ 74], 50.00th=[ 84], 60.00th=[ 136], | 70.00th=[ 136], 80.00th=[ 157], 90.00th=[ 157], 95.00th=[ 157], | 99.00th=[ 157], 99.50th=[ 157], 99.90th=[ 157], 99.95th=[ 157], | 99.99th=[ 157] bw ( KiB/s): min= 6144, max= 6144, per=24.24%, avg=6144.00, stdev= 0.00, samples=1 iops : min= 6, max= 6, avg= 6.00, stdev= 0.00, samples=1 lat (msec) : 20=16.67%, 50=33.33%, 100=25.00%, 250=25.00% cpu : usr=0.22%, sys=5.92%, ctx=38, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=17.7MiB/s (18.5MB/s), 1822KiB/s-6558KiB/s (1866kB/s-6715kB/s), io=20.0MiB (20.0MB), run=897-1131msec WRITE: bw=24.8MiB/s (25.0MB/s), 4684KiB/s-9110KiB/s (4797kB/s-9329kB/s), io=28.0MiB (29.4MB), run=897-1131msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=1391: Thu May 1 20:40:16 2025 read: IOPS=4, BW=18.1MiB/s (19.0MB/s)(12.0MiB/662msec) clat (msec): min=3, max=297, avg=190.97, stdev=163.03 lat (msec): min=3, max=297, avg=190.97, stdev=163.03 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 4], 10.00th=[ 4], 20.00th=[ 4], | 30.00th=[ 4], 40.00th=[ 271], 50.00th=[ 271], 60.00th=[ 271], | 70.00th=[ 300], 80.00th=[ 300], 90.00th=[ 300], 95.00th=[ 300], | 99.00th=[ 300], 99.50th=[ 300], 99.90th=[ 300], 99.95th=[ 300], | 99.99th=[ 300] bw ( KiB/s): min= 8192, max= 8192, per=56.25%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 4=33.33%, 500=66.67% cpu : usr=0.91%, sys=23.45%, ctx=702, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1393: Thu May 1 20:40:16 2025 read: IOPS=0, BW=3641KiB/s (3728kB/s)(4096KiB/1125msec) clat (nsec): min=232963k, max=232963k, avg=232962827.00, stdev= 0.00 lat (nsec): min=232965k, max=232965k, avg=232964914.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 234], 5.00th=[ 234], 10.00th=[ 234], 20.00th=[ 234], | 30.00th=[ 234], 40.00th=[ 234], 50.00th=[ 234], 60.00th=[ 234], | 70.00th=[ 234], 80.00th=[ 234], 90.00th=[ 234], 95.00th=[ 234], | 99.00th=[ 234], 99.50th=[ 234], 99.90th=[ 234], 99.95th=[ 234], | 99.99th=[ 234] bw ( KiB/s): min= 8192, max= 8192, per=56.25%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=1, BW=7282KiB/s (7457kB/s)(8192KiB/1125msec) clat (msec): min=425, max=442, avg=433.80, stdev=11.88 lat (msec): min=425, max=442, avg=434.00, stdev=11.82 clat percentiles (msec): | 1.00th=[ 426], 5.00th=[ 426], 10.00th=[ 426], 20.00th=[ 426], | 30.00th=[ 426], 40.00th=[ 426], 50.00th=[ 426], 60.00th=[ 443], | 70.00th=[ 443], 80.00th=[ 443], 90.00th=[ 443], 95.00th=[ 443], | 99.00th=[ 443], 99.50th=[ 443], 99.90th=[ 443], 99.95th=[ 443], | 99.99th=[ 443] bw ( KiB/s): min= 8192, max= 8192, per=30.45%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=66.67% cpu : usr=0.18%, sys=19.75%, ctx=612, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1394: Thu May 1 20:40:16 2025 write: IOPS=2, BW=9.85MiB/s (10.3MB/s)(12.0MiB/1218msec) clat (msec): min=221, max=521, avg=373.86, stdev=150.10 lat (msec): min=221, max=521, avg=374.12, stdev=150.11 clat percentiles (msec): | 1.00th=[ 222], 5.00th=[ 222], 10.00th=[ 222], 20.00th=[ 222], | 30.00th=[ 222], 40.00th=[ 380], 50.00th=[ 380], 60.00th=[ 380], | 70.00th=[ 523], 80.00th=[ 523], 90.00th=[ 523], 95.00th=[ 523], | 99.00th=[ 523], 99.50th=[ 523], 99.90th=[ 523], 99.95th=[ 523], | 99.99th=[ 523] bw ( KiB/s): min=16384, max=16384, per=60.90%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=33.33%, 750=33.33% cpu : usr=0.08%, sys=21.61%, ctx=67, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1395: Thu May 1 20:40:16 2025 write: IOPS=2, BW=11.2MiB/s (11.7MB/s)(12.0MiB/1074msec) clat (msec): min=218, max=449, avg=311.17, stdev=122.31 lat (msec): min=218, max=450, avg=311.76, stdev=122.15 clat percentiles (msec): | 1.00th=[ 220], 5.00th=[ 220], 10.00th=[ 220], 20.00th=[ 220], | 30.00th=[ 220], 40.00th=[ 266], 50.00th=[ 266], 60.00th=[ 266], | 70.00th=[ 451], 80.00th=[ 451], 90.00th=[ 451], 95.00th=[ 451], | 99.00th=[ 451], 99.50th=[ 451], 99.90th=[ 451], 99.95th=[ 451], | 99.99th=[ 451] bw ( KiB/s): min= 8159, max= 8192, per=30.39%, avg=8175.50, stdev=23.33, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 250=33.33%, 500=66.67% cpu : usr=0.00%, sys=24.88%, ctx=139, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=14.2MiB/s (14.9MB/s), 3641KiB/s-18.1MiB/s (3728kB/s-19.0MB/s), io=16.0MiB (16.8MB), run=662-1125msec WRITE: bw=26.3MiB/s (27.5MB/s), 7282KiB/s-11.2MiB/s (7457kB/s-11.7MB/s), io=32.0MiB (33.6MB), run=1074-1218msec rand-rw: (groupid=0, jobs=1): err= 0: pid=1388: Thu May 1 20:40:16 2025 read: IOPS=12, BW=51.1MiB/s (53.5MB/s)(12.0MiB/235msec) clat (msec): min=33, max=104, avg=65.57, stdev=36.00 lat (msec): min=33, max=104, avg=65.57, stdev=36.00 clat percentiles (msec): | 1.00th=[ 34], 5.00th=[ 34], 10.00th=[ 34], 20.00th=[ 34], | 30.00th=[ 34], 40.00th=[ 59], 50.00th=[ 59], 60.00th=[ 59], | 70.00th=[ 105], 80.00th=[ 105], 90.00th=[ 105], 95.00th=[ 105], | 99.00th=[ 105], 99.50th=[ 105], 99.90th=[ 105], 99.95th=[ 105], | 99.99th=[ 105] lat (msec) : 50=33.33%, 100=33.33%, 250=33.33% cpu : usr=0.00%, sys=11.06%, ctx=21, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1389: Thu May 1 20:40:16 2025 read: IOPS=0, BW=3306KiB/s (3385kB/s)(4096KiB/1239msec) clat (nsec): min=338655k, max=338655k, avg=338654973.00, stdev= 0.00 lat (nsec): min=338657k, max=338657k, avg=338657331.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 338], 5.00th=[ 338], 10.00th=[ 338], 20.00th=[ 338], | 30.00th=[ 338], 40.00th=[ 338], 50.00th=[ 338], 60.00th=[ 338], | 70.00th=[ 338], 80.00th=[ 338], 90.00th=[ 338], 95.00th=[ 338], | 99.00th=[ 338], 99.50th=[ 338], 99.90th=[ 338], 99.95th=[ 338], | 99.99th=[ 338] bw ( KiB/s): min= 8192, max= 8192, per=61.95%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=1, BW=6612KiB/s (6770kB/s)(8192KiB/1239msec) clat (msec): min=168, max=690, avg=429.60, stdev=368.69 lat (msec): min=169, max=690, avg=429.86, stdev=368.70 clat percentiles (msec): | 1.00th=[ 169], 5.00th=[ 169], 10.00th=[ 169], 20.00th=[ 169], | 30.00th=[ 169], 40.00th=[ 169], 50.00th=[ 169], 60.00th=[ 693], | 70.00th=[ 693], 80.00th=[ 693], 90.00th=[ 693], 95.00th=[ 693], | 99.00th=[ 693], 99.50th=[ 693], 99.90th=[ 693], 99.95th=[ 693], | 99.99th=[ 693] bw ( KiB/s): min= 5665, max= 5665, per=21.42%, avg=5665.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=33.33%, 750=33.33% cpu : usr=0.24%, sys=5.57%, ctx=39, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1390: Thu May 1 20:40:16 2025 write: IOPS=3, BW=12.7MiB/s (13.3MB/s)(12.0MiB/944msec) clat (msec): min=290, max=345, avg=308.79, stdev=32.05 lat (msec): min=290, max=346, avg=309.62, stdev=32.06 clat percentiles (msec): | 1.00th=[ 292], 5.00th=[ 292], 10.00th=[ 292], 20.00th=[ 292], | 30.00th=[ 292], 40.00th=[ 292], 50.00th=[ 292], 60.00th=[ 292], | 70.00th=[ 347], 80.00th=[ 347], 90.00th=[ 347], 95.00th=[ 347], | 99.00th=[ 347], 99.50th=[ 347], 99.90th=[ 347], 99.95th=[ 347], | 99.99th=[ 347] bw ( KiB/s): min= 8192, max= 8192, per=30.98%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 500=100.00% cpu : usr=0.21%, sys=4.77%, ctx=212, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1392: Thu May 1 20:40:16 2025 write: IOPS=3, BW=14.1MiB/s (14.8MB/s)(12.0MiB/850msec) clat (msec): min=113, max=558, avg=270.20, stdev=249.78 lat (msec): min=113, max=558, avg=270.43, stdev=249.82 clat percentiles (msec): | 1.00th=[ 113], 5.00th=[ 113], 10.00th=[ 113], 20.00th=[ 113], | 30.00th=[ 113], 40.00th=[ 140], 50.00th=[ 140], 60.00th=[ 140], | 70.00th=[ 558], 80.00th=[ 558], 90.00th=[ 558], 95.00th=[ 558], | 99.00th=[ 558], 99.50th=[ 558], 99.90th=[ 558], 99.95th=[ 558], | 99.99th=[ 558] bw ( KiB/s): min=16318, max=16318, per=61.70%, avg=16318.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 750=33.33% cpu : usr=0.24%, sys=7.42%, ctx=24, majf=0, minf=28 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=12.9MiB/s (13.5MB/s), 3306KiB/s-51.1MiB/s (3385kB/s-53.5MB/s), io=16.0MiB (16.8MB), run=235-1239msec WRITE: bw=25.8MiB/s (27.1MB/s), 6612KiB/s-14.1MiB/s (6770kB/s-14.8MB/s), io=32.0MiB (33.6MB), run=850-1239msec