== sanity test 398b: DIO and buffer IO race ============== 22:39:02 (1746153542) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 2.59797 s, 19.4 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=20856: Thu May 1 22:39:50 2025 read: IOPS=35, BW=142KiB/s (145kB/s)(5992KiB/42334msec) clat (usec): min=59, max=281768, avg=15652.04, stdev=12183.30 lat (usec): min=60, max=281771, avg=15655.40, stdev=12184.79 clat percentiles (usec): | 1.00th=[ 97], 5.00th=[ 127], 10.00th=[ 5669], 20.00th=[ 8029], | 30.00th=[ 9896], 40.00th=[ 11863], 50.00th=[ 13698], 60.00th=[ 16057], | 70.00th=[ 18744], 80.00th=[ 22414], 90.00th=[ 27395], 95.00th=[ 32113], | 99.00th=[ 53740], 99.50th=[ 63701], 99.90th=[ 82314], 99.95th=[283116], | 99.99th=[283116] bw ( KiB/s): min= 15, max= 208, per=24.73%, avg=140.21, stdev=32.52, samples=84 iops : min= 3, max= 52, avg=34.76, stdev= 8.14, samples=84 write: IOPS=37, BW=149KiB/s (152kB/s)(6296KiB/42334msec) clat (usec): min=452, max=87081, avg=11818.72, stdev=12408.63 lat (usec): min=453, max=87083, avg=11820.49, stdev=12408.71 clat percentiles (usec): | 1.00th=[ 676], 5.00th=[ 734], 10.00th=[ 783], 20.00th=[ 971], | 30.00th=[ 1909], 40.00th=[ 4555], 50.00th=[ 8717], 60.00th=[11600], | 70.00th=[14877], 80.00th=[20841], 90.00th=[29492], 95.00th=[36439], | 99.00th=[51119], 99.50th=[56361], 99.90th=[80217], 99.95th=[87557], | 99.99th=[87557] bw ( KiB/s): min= 79, max= 240, per=25.10%, avg=148.82, stdev=40.02, samples=83 iops : min= 19, max= 60, avg=36.93, stdev= 9.98, samples=83 lat (usec) : 100=0.78%, 250=2.18%, 500=0.16%, 750=3.45%, 1000=6.97% lat (msec) : 2=5.50%, 4=4.17%, 10=19.86%, 20=33.24%, 50=22.59% lat (msec) : 100=1.07%, 500=0.03% cpu : usr=0.15%, sys=10.91%, ctx=5134, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20857: Thu May 1 22:39:50 2025 read: IOPS=35, BW=143KiB/s (147kB/s)(5940KiB/41504msec) clat (usec): min=64, max=105433, avg=15124.71, stdev=10446.00 lat (usec): min=65, max=105434, avg=15126.27, stdev=10445.41 clat percentiles (usec): | 1.00th=[ 94], 5.00th=[ 143], 10.00th=[ 5800], 20.00th=[ 7832], | 30.00th=[ 9503], 40.00th=[ 11338], 50.00th=[ 12911], 60.00th=[ 14877], | 70.00th=[ 17433], 80.00th=[ 21365], 90.00th=[ 26870], 95.00th=[ 33817], | 99.00th=[ 53740], 99.50th=[ 62129], 99.90th=[ 94897], 99.95th=[105382], | 99.99th=[105382] bw ( KiB/s): min= 7, max= 239, per=25.02%, avg=141.89, stdev=36.12, samples=82 iops : min= 1, max= 59, avg=35.00, stdev= 9.07, samples=82 write: IOPS=38, BW=153KiB/s (157kB/s)(6348KiB/41504msec) clat (usec): min=498, max=338992, avg=11874.79, stdev=14609.24 lat (usec): min=499, max=338993, avg=11876.76, stdev=14609.08 clat percentiles (usec): | 1.00th=[ 693], 5.00th=[ 750], 10.00th=[ 799], 20.00th=[ 1020], | 30.00th=[ 1975], 40.00th=[ 4359], 50.00th=[ 8455], 60.00th=[ 11338], | 70.00th=[ 15401], 80.00th=[ 21103], 90.00th=[ 28705], 95.00th=[ 35390], | 99.00th=[ 51643], 99.50th=[ 57934], 99.90th=[ 83362], 99.95th=[337642], | 99.99th=[337642] bw ( KiB/s): min= 23, max= 319, per=25.36%, avg=150.37, stdev=44.58, samples=82 iops : min= 5, max= 79, avg=37.12, stdev=11.18, samples=82 lat (usec) : 100=1.01%, 250=1.73%, 500=0.13%, 750=2.38%, 1000=7.88% lat (msec) : 2=5.44%, 4=4.82%, 10=20.77%, 20=33.59%, 50=21.03% lat (msec) : 100=1.17%, 250=0.03%, 500=0.03% cpu : usr=0.20%, sys=11.12%, ctx=5012, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20858: Thu May 1 22:39:50 2025 read: IOPS=36, BW=147KiB/s (151kB/s)(6028KiB/40940msec) clat (usec): min=76, max=346170, avg=15526.42, stdev=13042.74 lat (usec): min=77, max=346170, avg=15527.46, stdev=13042.79 clat percentiles (usec): | 1.00th=[ 93], 5.00th=[ 125], 10.00th=[ 5473], 20.00th=[ 8160], | 30.00th=[ 9896], 40.00th=[ 11469], 50.00th=[ 13698], 60.00th=[ 15926], | 70.00th=[ 18744], 80.00th=[ 21890], 90.00th=[ 27395], 95.00th=[ 32113], | 99.00th=[ 52691], 99.50th=[ 59507], 99.90th=[ 78119], 99.95th=[346031], | 99.99th=[346031] bw ( KiB/s): min= 15, max= 232, per=25.86%, avg=146.63, stdev=39.50, samples=81 iops : min= 3, max= 58, avg=36.36, stdev= 9.94, samples=81 write: IOPS=38, BW=153KiB/s (157kB/s)(6260KiB/40940msec) clat (usec): min=560, max=95016, avg=11088.13, stdev=12204.66 lat (usec): min=560, max=95018, avg=11090.03, stdev=12204.90 clat percentiles (usec): | 1.00th=[ 676], 5.00th=[ 742], 10.00th=[ 791], 20.00th=[ 898], | 30.00th=[ 1680], 40.00th=[ 3064], 50.00th=[ 7373], 60.00th=[10945], | 70.00th=[14484], 80.00th=[19792], 90.00th=[28181], 95.00th=[35390], | 99.00th=[51643], 99.50th=[57934], 99.90th=[83362], 99.95th=[94897], | 99.99th=[94897] bw ( KiB/s): min= 31, max= 256, per=25.61%, avg=151.85, stdev=42.26, samples=81 iops : min= 7, max= 64, avg=37.67, stdev=10.64, samples=81 lat (usec) : 100=1.07%, 250=2.21%, 500=0.07%, 750=2.99%, 1000=8.53% lat (msec) : 2=5.44%, 4=5.18%, 10=18.82%, 20=33.04%, 50=21.55% lat (msec) : 100=1.07%, 500=0.03% cpu : usr=0.17%, sys=11.22%, ctx=5047, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20859: Thu May 1 22:39:50 2025 read: IOPS=36, BW=145KiB/s (148kB/s)(6052KiB/41740msec) clat (usec): min=66, max=105356, avg=15776.90, stdev=10351.13 lat (usec): min=67, max=105357, avg=15779.16, stdev=10351.38 clat percentiles (usec): | 1.00th=[ 95], 5.00th=[ 127], 10.00th=[ 5866], 20.00th=[ 8225], | 30.00th=[ 10028], 40.00th=[ 11994], 50.00th=[ 13960], 60.00th=[ 16188], | 70.00th=[ 19006], 80.00th=[ 22414], 90.00th=[ 28181], 95.00th=[ 33162], | 99.00th=[ 50594], 99.50th=[ 58983], 99.90th=[ 84411], 99.95th=[105382], | 99.99th=[105382] bw ( KiB/s): min= 56, max= 224, per=25.70%, avg=145.74, stdev=34.92, samples=82 iops : min= 14, max= 56, avg=36.05, stdev= 8.75, samples=82 write: IOPS=37, BW=149KiB/s (153kB/s)(6236KiB/41740msec) clat (usec): min=522, max=378872, avg=11305.83, stdev=15315.69 lat (usec): min=524, max=378875, avg=11307.13, stdev=15315.76 clat percentiles (usec): | 1.00th=[ 660], 5.00th=[ 734], 10.00th=[ 783], 20.00th=[ 922], | 30.00th=[ 1729], 40.00th=[ 3556], 50.00th=[ 7570], 60.00th=[ 10421], | 70.00th=[ 14746], 80.00th=[ 19792], 90.00th=[ 26870], 95.00th=[ 35390], | 99.00th=[ 53740], 99.50th=[ 63177], 99.90th=[ 88605], 99.95th=[379585], | 99.99th=[379585] bw ( KiB/s): min= 71, max= 280, per=25.35%, avg=150.34, stdev=42.83, samples=82 iops : min= 17, max= 70, avg=37.20, stdev=10.65, samples=82 lat (usec) : 100=1.01%, 250=2.02%, 500=0.07%, 750=3.48%, 1000=7.94% lat (msec) : 2=5.44%, 4=4.23%, 10=20.15%, 20=32.65%, 50=21.74% lat (msec) : 100=1.20%, 250=0.03%, 500=0.03% cpu : usr=0.13%, sys=11.03%, ctx=5017, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=567KiB/s (581kB/s), 142KiB/s-147KiB/s (145kB/s-151kB/s), io=23.4MiB (24.6MB), run=40940-42334msec WRITE: bw=594KiB/s (608kB/s), 149KiB/s-153KiB/s (152kB/s-157kB/s), io=24.6MiB (25.7MB), run=40940-42334msec rand-rw: (groupid=0, jobs=1): err= 0: pid=20860: Thu May 1 22:40:09 2025 read: IOPS=24, BW=99.6KiB/s (102kB/s)(5992KiB/60186msec) clat (msec): min=3, max=334, avg=15.00, stdev=14.76 lat (msec): min=3, max=334, avg=15.00, stdev=14.76 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 5], 10.00th=[ 6], 20.00th=[ 7], | 30.00th=[ 8], 40.00th=[ 10], 50.00th=[ 11], 60.00th=[ 13], | 70.00th=[ 16], 80.00th=[ 21], 90.00th=[ 31], 95.00th=[ 40], | 99.00th=[ 67], 99.50th=[ 82], 99.90th=[ 103], 99.95th=[ 334], | 99.99th=[ 334] bw ( KiB/s): min= 23, max= 272, per=25.46%, avg=99.02, stdev=51.74, samples=120 iops : min= 5, max= 68, avg=24.38, stdev=13.01, samples=120 write: IOPS=26, BW=105KiB/s (107kB/s)(6296KiB/60186msec) clat (msec): min=4, max=119, avg=23.87, stdev=15.39 lat (msec): min=4, max=119, avg=23.87, stdev=15.39 clat percentiles (msec): | 1.00th=[ 7], 5.00th=[ 10], 10.00th=[ 11], 20.00th=[ 13], | 30.00th=[ 15], 40.00th=[ 17], 50.00th=[ 19], 60.00th=[ 23], | 70.00th=[ 27], 80.00th=[ 34], 90.00th=[ 45], 95.00th=[ 56], | 99.00th=[ 80], 99.50th=[ 89], 99.90th=[ 108], 99.95th=[ 121], | 99.99th=[ 121] bw ( KiB/s): min= 31, max= 256, per=25.74%, avg=104.77, stdev=50.42, samples=119 iops : min= 7, max= 64, avg=25.82, stdev=12.66, samples=119 lat (msec) : 4=0.81%, 10=25.46%, 20=40.40%, 50=28.52%, 100=4.62% lat (msec) : 250=0.16%, 500=0.03% cpu : usr=0.12%, sys=5.15%, ctx=3604, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20861: Thu May 1 22:40:09 2025 read: IOPS=24, BW=96.7KiB/s (98.0kB/s)(5940KiB/61444msec) clat (msec): min=2, max=310, avg=15.35, stdev=14.82 lat (msec): min=2, max=310, avg=15.35, stdev=14.82 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 5], 10.00th=[ 6], 20.00th=[ 7], | 30.00th=[ 8], 40.00th=[ 10], 50.00th=[ 12], 60.00th=[ 14], | 70.00th=[ 16], 80.00th=[ 20], 90.00th=[ 31], 95.00th=[ 45], | 99.00th=[ 66], 99.50th=[ 81], 99.90th=[ 92], 99.95th=[ 313], | 99.99th=[ 313] bw ( KiB/s): min= 8, max= 271, per=24.68%, avg=96.00, stdev=51.63, samples=122 iops : min= 2, max= 67, avg=23.66, stdev=12.96, samples=122 write: IOPS=25, BW=103KiB/s (106kB/s)(6348KiB/61444msec) clat (msec): min=5, max=117, avg=24.19, stdev=15.61 lat (msec): min=5, max=117, avg=24.20, stdev=15.61 clat percentiles (msec): | 1.00th=[ 8], 5.00th=[ 10], 10.00th=[ 11], 20.00th=[ 13], | 30.00th=[ 15], 40.00th=[ 17], 50.00th=[ 19], 60.00th=[ 23], | 70.00th=[ 28], 80.00th=[ 35], 90.00th=[ 45], 95.00th=[ 56], | 99.00th=[ 81], 99.50th=[ 95], 99.90th=[ 118], 99.95th=[ 118], | 99.99th=[ 118] bw ( KiB/s): min= 39, max= 240, per=25.34%, avg=103.13, stdev=49.12, samples=121 iops : min= 9, max= 60, avg=25.45, stdev=12.37, samples=121 lat (msec) : 4=1.04%, 10=23.21%, 20=42.71%, 50=27.77%, 100=5.11% lat (msec) : 250=0.13%, 500=0.03% cpu : usr=0.15%, sys=4.95%, ctx=3623, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20862: Thu May 1 22:40:09 2025 read: IOPS=24, BW=99.3KiB/s (102kB/s)(6028KiB/60724msec) clat (usec): min=2873, max=92195, avg=15276.10, stdev=12399.81 lat (usec): min=2874, max=92197, avg=15277.25, stdev=12399.78 clat percentiles (usec): | 1.00th=[ 3589], 5.00th=[ 4621], 10.00th=[ 5342], 20.00th=[ 6652], | 30.00th=[ 8029], 40.00th=[ 9372], 50.00th=[11076], 60.00th=[13304], | 70.00th=[15926], 80.00th=[21103], 90.00th=[32375], 95.00th=[41157], | 99.00th=[63701], 99.50th=[76022], 99.90th=[84411], 99.95th=[91751], | 99.99th=[91751] bw ( KiB/s): min= 15, max= 263, per=25.60%, avg=99.60, stdev=49.95, samples=120 iops : min= 3, max= 65, avg=24.66, stdev=12.48, samples=120 write: IOPS=25, BW=103KiB/s (106kB/s)(6260KiB/60724msec) clat (msec): min=5, max=357, avg=23.95, stdev=16.77 lat (msec): min=5, max=357, avg=23.96, stdev=16.77 clat percentiles (msec): | 1.00th=[ 8], 5.00th=[ 10], 10.00th=[ 11], 20.00th=[ 13], | 30.00th=[ 15], 40.00th=[ 17], 50.00th=[ 20], 60.00th=[ 23], | 70.00th=[ 28], 80.00th=[ 34], 90.00th=[ 43], 95.00th=[ 53], | 99.00th=[ 75], 99.50th=[ 87], 99.90th=[ 144], 99.95th=[ 359], | 99.99th=[ 359] bw ( KiB/s): min= 39, max= 224, per=25.39%, avg=103.34, stdev=48.77, samples=120 iops : min= 9, max= 56, avg=25.59, stdev=12.19, samples=120 lat (msec) : 4=1.11%, 10=23.93%, 20=40.14%, 50=30.44%, 100=4.26% lat (msec) : 250=0.10%, 500=0.03% cpu : usr=0.12%, sys=4.98%, ctx=3633, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20863: Thu May 1 22:40:09 2025 read: IOPS=24, BW=98.1KiB/s (100kB/s)(6052KiB/61702msec) clat (msec): min=2, max=101, avg=15.60, stdev=13.03 lat (msec): min=2, max=101, avg=15.60, stdev=13.03 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 5], 10.00th=[ 6], 20.00th=[ 7], | 30.00th=[ 8], 40.00th=[ 10], 50.00th=[ 12], 60.00th=[ 14], | 70.00th=[ 17], 80.00th=[ 22], 90.00th=[ 33], 95.00th=[ 44], | 99.00th=[ 63], 99.50th=[ 77], 99.90th=[ 97], 99.95th=[ 103], | 99.99th=[ 103] bw ( KiB/s): min= 31, max= 240, per=25.21%, avg=98.07, stdev=51.79, samples=122 iops : min= 7, max= 60, avg=24.16, stdev=13.01, samples=122 write: IOPS=25, BW=101KiB/s (103kB/s)(6236KiB/61702msec) clat (msec): min=4, max=406, avg=24.33, stdev=18.36 lat (msec): min=4, max=406, avg=24.34, stdev=18.36 clat percentiles (msec): | 1.00th=[ 8], 5.00th=[ 10], 10.00th=[ 11], 20.00th=[ 13], | 30.00th=[ 15], 40.00th=[ 17], 50.00th=[ 19], 60.00th=[ 23], | 70.00th=[ 27], 80.00th=[ 34], 90.00th=[ 45], 95.00th=[ 56], | 99.00th=[ 81], 99.50th=[ 88], 99.90th=[ 144], 99.95th=[ 405], | 99.99th=[ 405] bw ( KiB/s): min= 40, max= 256, per=24.82%, avg=101.02, stdev=49.79, samples=122 iops : min= 10, max= 64, avg=24.89, stdev=12.48, samples=122 lat (msec) : 4=1.07%, 10=24.25%, 20=40.36%, 50=29.26%, 100=4.85% lat (msec) : 250=0.16%, 500=0.03% cpu : usr=0.10%, sys=5.04%, ctx=3654, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=389KiB/s (399kB/s), 96.7KiB/s-99.6KiB/s (98.0kB/s-102kB/s), io=23.4MiB (24.6MB), run=60186-61702msec WRITE: bw=407KiB/s (417kB/s), 101KiB/s-105KiB/s (103kB/s-107kB/s), io=24.6MiB (25.7MB), run=60186-61702msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=20928: Thu May 1 22:40:23 2025 read: IOPS=30, BW=489KiB/s (501kB/s)(5920KiB/12101msec) clat (usec): min=78, max=88635, avg=17075.76, stdev=13360.97 lat (usec): min=78, max=88637, avg=17084.07, stdev=13352.46 clat percentiles (usec): | 1.00th=[ 99], 5.00th=[ 109], 10.00th=[ 133], 20.00th=[ 7701], | 30.00th=[10028], 40.00th=[12780], 50.00th=[14484], 60.00th=[17957], | 70.00th=[21103], 80.00th=[25297], 90.00th=[32375], 95.00th=[40633], | 99.00th=[62129], 99.50th=[77071], 99.90th=[88605], 99.95th=[88605], | 99.99th=[88605] bw ( KiB/s): min= 288, max= 932, per=24.34%, avg=481.67, stdev=147.26, samples=24 iops : min= 18, max= 58, avg=29.71, stdev= 9.19, samples=24 write: IOPS=32, BW=526KiB/s (539kB/s)(6368KiB/12101msec) clat (usec): min=801, max=121947, avg=14212.83, stdev=16001.01 lat (usec): min=802, max=121949, avg=14214.49, stdev=16001.01 clat percentiles (usec): | 1.00th=[ 955], 5.00th=[ 1106], 10.00th=[ 1205], 20.00th=[ 1401], | 30.00th=[ 2737], 40.00th=[ 4817], 50.00th=[ 8356], 60.00th=[ 12256], | 70.00th=[ 17695], 80.00th=[ 25035], 90.00th=[ 37487], 95.00th=[ 46400], | 99.00th=[ 60556], 99.50th=[ 85459], 99.90th=[122160], 99.95th=[122160], | 99.99th=[122160] bw ( KiB/s): min= 221, max= 932, per=25.00%, avg=520.46, stdev=173.32, samples=24 iops : min= 13, max= 58, avg=32.12, stdev=10.82, samples=24 lat (usec) : 100=0.65%, 250=5.47%, 500=0.26%, 750=0.13%, 1000=1.30% lat (msec) : 2=13.28%, 4=5.99%, 10=15.23%, 20=27.08%, 50=27.60% lat (msec) : 100=2.86%, 250=0.13% cpu : usr=0.07%, sys=10.35%, ctx=1302, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20930: Thu May 1 22:40:23 2025 read: IOPS=30, BW=489KiB/s (501kB/s)(5744KiB/11740msec) clat (usec): min=72, max=84699, avg=17680.14, stdev=12730.84 lat (usec): min=72, max=84700, avg=17681.27, stdev=12730.80 clat percentiles (usec): | 1.00th=[ 87], 5.00th=[ 109], 10.00th=[ 127], 20.00th=[ 8979], | 30.00th=[11469], 40.00th=[13566], 50.00th=[15270], 60.00th=[17957], | 70.00th=[21365], 80.00th=[26346], 90.00th=[33424], 95.00th=[39584], | 99.00th=[63177], 99.50th=[71828], 99.90th=[84411], 99.95th=[84411], | 99.99th=[84411] bw ( KiB/s): min= 317, max= 766, per=24.67%, avg=488.22, stdev=121.76, samples=23 iops : min= 19, max= 47, avg=30.00, stdev= 7.69, samples=23 write: IOPS=34, BW=557KiB/s (571kB/s)(6544KiB/11740msec) clat (usec): min=931, max=74697, avg=13101.54, stdev=13712.44 lat (usec): min=933, max=74698, avg=13106.17, stdev=13717.82 clat percentiles (usec): | 1.00th=[ 979], 5.00th=[ 1090], 10.00th=[ 1172], 20.00th=[ 1434], | 30.00th=[ 2704], 40.00th=[ 4752], 50.00th=[ 9110], 60.00th=[12387], | 70.00th=[16712], 80.00th=[22938], 90.00th=[32375], 95.00th=[41157], | 99.00th=[55313], 99.50th=[70779], 99.90th=[74974], 99.95th=[74974], | 99.99th=[74974] bw ( KiB/s): min= 288, max= 992, per=26.57%, avg=553.22, stdev=181.05, samples=23 iops : min= 18, max= 62, avg=34.09, stdev=11.31, samples=23 lat (usec) : 100=1.30%, 250=4.43%, 500=0.13%, 1000=0.91% lat (msec) : 2=11.72%, 4=7.03%, 10=12.89%, 20=32.42%, 50=26.82% lat (msec) : 100=2.34% cpu : usr=0.13%, sys=10.75%, ctx=1263, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20932: Thu May 1 22:40:23 2025 read: IOPS=32, BW=517KiB/s (530kB/s)(6256KiB/12097msec) clat (usec): min=70, max=78254, avg=17576.56, stdev=12825.44 lat (usec): min=71, max=78255, avg=17579.43, stdev=12824.21 clat percentiles (usec): | 1.00th=[ 89], 5.00th=[ 106], 10.00th=[ 131], 20.00th=[ 8717], | 30.00th=[11600], 40.00th=[13435], 50.00th=[15533], 60.00th=[18744], | 70.00th=[21103], 80.00th=[25560], 90.00th=[32375], 95.00th=[39060], | 99.00th=[65274], 99.50th=[68682], 99.90th=[78119], 99.95th=[78119], | 99.99th=[78119] bw ( KiB/s): min= 192, max= 832, per=26.00%, avg=514.50, stdev=159.73, samples=24 iops : min= 12, max= 52, avg=31.67, stdev=10.15, samples=24 write: IOPS=31, BW=499KiB/s (511kB/s)(6032KiB/12097msec) clat (usec): min=758, max=82024, avg=13247.51, stdev=14331.20 lat (usec): min=759, max=82025, avg=13249.23, stdev=14331.24 clat percentiles (usec): | 1.00th=[ 889], 5.00th=[ 1074], 10.00th=[ 1172], 20.00th=[ 1434], | 30.00th=[ 2769], 40.00th=[ 4752], 50.00th=[ 9503], 60.00th=[12780], | 70.00th=[16188], 80.00th=[21627], 90.00th=[33817], 95.00th=[41157], | 99.00th=[73925], 99.50th=[76022], 99.90th=[82314], 99.95th=[82314], | 99.99th=[82314] bw ( KiB/s): min= 255, max= 851, per=23.61%, avg=491.46, stdev=156.82, samples=24 iops : min= 15, max= 53, avg=30.25, stdev= 9.78, samples=24 lat (usec) : 100=1.69%, 250=4.69%, 500=0.39%, 1000=1.43% lat (msec) : 2=10.94%, 4=6.51%, 10=10.81%, 20=34.77%, 50=26.17% lat (msec) : 100=2.60% cpu : usr=0.15%, sys=10.51%, ctx=1319, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20934: Thu May 1 22:40:23 2025 read: IOPS=32, BW=513KiB/s (526kB/s)(6032KiB/11747msec) clat (usec): min=75, max=81793, avg=17014.12, stdev=12844.75 lat (usec): min=76, max=81793, avg=17018.93, stdev=12843.03 clat percentiles (usec): | 1.00th=[ 91], 5.00th=[ 106], 10.00th=[ 125], 20.00th=[ 7832], | 30.00th=[10683], 40.00th=[12649], 50.00th=[15401], 60.00th=[17957], | 70.00th=[20841], 80.00th=[25297], 90.00th=[33424], 95.00th=[40633], | 99.00th=[55837], 99.50th=[74974], 99.90th=[82314], 99.95th=[82314], | 99.99th=[82314] bw ( KiB/s): min= 191, max= 832, per=25.65%, avg=507.70, stdev=146.64, samples=23 iops : min= 11, max= 52, avg=31.26, stdev= 9.26, samples=23 write: IOPS=33, BW=533KiB/s (545kB/s)(6256KiB/11747msec) clat (usec): min=847, max=80661, avg=13374.75, stdev=15829.79 lat (usec): min=848, max=80663, avg=13376.40, stdev=15829.88 clat percentiles (usec): | 1.00th=[ 938], 5.00th=[ 1057], 10.00th=[ 1156], 20.00th=[ 1336], | 30.00th=[ 1926], 40.00th=[ 2933], 50.00th=[ 5669], 60.00th=[10814], | 70.00th=[18220], 80.00th=[24773], 90.00th=[37487], 95.00th=[46400], | 99.00th=[74974], 99.50th=[79168], 99.90th=[80217], 99.95th=[80217], | 99.99th=[80217] bw ( KiB/s): min= 287, max= 896, per=25.38%, avg=528.48, stdev=154.34, samples=23 iops : min= 17, max= 56, avg=32.57, stdev= 9.64, samples=23 lat (usec) : 100=1.43%, 250=5.99%, 500=0.13%, 1000=0.91% lat (msec) : 2=14.84%, 4=7.16%, 10=13.54%, 20=26.30%, 50=26.69% lat (msec) : 100=2.99% cpu : usr=0.12%, sys=10.46%, ctx=1241, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1979KiB/s (2027kB/s), 489KiB/s-517KiB/s (501kB/s-530kB/s), io=23.4MiB (24.5MB), run=11740-12101msec WRITE: bw=2082KiB/s (2132kB/s), 499KiB/s-557KiB/s (511kB/s-571kB/s), io=24.6MiB (25.8MB), run=11740-12101msec rand-rw: (groupid=0, jobs=1): err= 0: pid=20927: Thu May 1 22:40:29 2025 read: IOPS=21, BW=347KiB/s (355kB/s)(5920KiB/17075msec) clat (usec): min=3822, max=75435, avg=16087.67, stdev=13263.30 lat (usec): min=3823, max=75436, avg=16088.88, stdev=13263.26 clat percentiles (usec): | 1.00th=[ 4146], 5.00th=[ 5473], 10.00th=[ 6325], 20.00th=[ 7439], | 30.00th=[ 8717], 40.00th=[ 9896], 50.00th=[11338], 60.00th=[13173], | 70.00th=[16057], 80.00th=[20841], 90.00th=[32637], 95.00th=[46400], | 99.00th=[69731], 99.50th=[73925], 99.90th=[74974], 99.95th=[74974], | 99.99th=[74974] bw ( KiB/s): min= 127, max= 736, per=25.45%, avg=344.59, stdev=167.05, samples=34 iops : min= 7, max= 46, avg=21.26, stdev=10.46, samples=34 write: IOPS=23, BW=373KiB/s (382kB/s)(6368KiB/17075msec) clat (msec): min=5, max=143, avg=27.85, stdev=18.97 lat (msec): min=5, max=143, avg=27.85, stdev=18.97 clat percentiles (msec): | 1.00th=[ 9], 5.00th=[ 11], 10.00th=[ 12], 20.00th=[ 14], | 30.00th=[ 16], 40.00th=[ 19], 50.00th=[ 22], 60.00th=[ 25], | 70.00th=[ 32], 80.00th=[ 39], 90.00th=[ 56], 95.00th=[ 70], | 99.00th=[ 87], 99.50th=[ 113], 99.90th=[ 144], 99.95th=[ 144], | 99.99th=[ 144] bw ( KiB/s): min= 96, max= 736, per=26.05%, avg=370.88, stdev=189.79, samples=34 iops : min= 6, max= 46, avg=22.91, stdev=11.93, samples=34 lat (msec) : 4=0.26%, 10=20.57%, 20=41.02%, 50=29.30%, 100=8.59% lat (msec) : 250=0.26% cpu : usr=0.08%, sys=4.64%, ctx=864, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20929: Thu May 1 22:40:29 2025 read: IOPS=20, BW=326KiB/s (334kB/s)(5744KiB/17613msec) clat (usec): min=3252, max=84985, avg=17074.47, stdev=15310.41 lat (usec): min=3253, max=84986, avg=17075.70, stdev=15310.39 clat percentiles (usec): | 1.00th=[ 3720], 5.00th=[ 4948], 10.00th=[ 5669], 20.00th=[ 6718], | 30.00th=[ 7963], 40.00th=[ 9372], 50.00th=[11207], 60.00th=[13566], | 70.00th=[17433], 80.00th=[22938], 90.00th=[42730], 95.00th=[54264], | 99.00th=[68682], 99.50th=[77071], 99.90th=[85459], 99.95th=[85459], | 99.99th=[85459] bw ( KiB/s): min= 63, max= 830, per=23.82%, avg=322.46, stdev=206.05, samples=35 iops : min= 3, max= 51, avg=19.57, stdev=12.85, samples=35 write: IOPS=23, BW=372KiB/s (380kB/s)(6544KiB/17613msec) clat (msec): min=7, max=151, avg=27.57, stdev=19.20 lat (msec): min=7, max=151, avg=27.57, stdev=19.20 clat percentiles (msec): | 1.00th=[ 9], 5.00th=[ 11], 10.00th=[ 12], 20.00th=[ 14], | 30.00th=[ 17], 40.00th=[ 19], 50.00th=[ 22], 60.00th=[ 25], | 70.00th=[ 31], 80.00th=[ 38], 90.00th=[ 55], 95.00th=[ 65], | 99.00th=[ 90], 99.50th=[ 117], 99.90th=[ 153], 99.95th=[ 153], | 99.99th=[ 153] bw ( KiB/s): min= 159, max= 800, per=25.79%, avg=367.23, stdev=193.75, samples=35 iops : min= 9, max= 50, avg=22.37, stdev=12.12, samples=35 lat (msec) : 4=0.91%, 10=21.87%, 20=36.59%, 50=31.25%, 100=8.85% lat (msec) : 250=0.52% cpu : usr=0.18%, sys=4.42%, ctx=897, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20931: Thu May 1 22:40:29 2025 read: IOPS=22, BW=361KiB/s (370kB/s)(6256KiB/17315msec) clat (msec): min=3, max=104, avg=16.93, stdev=16.47 lat (msec): min=3, max=104, avg=16.94, stdev=16.47 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 6], 10.00th=[ 7], 20.00th=[ 8], | 30.00th=[ 9], 40.00th=[ 10], 50.00th=[ 11], 60.00th=[ 13], | 70.00th=[ 17], 80.00th=[ 22], 90.00th=[ 36], 95.00th=[ 56], | 99.00th=[ 90], 99.50th=[ 104], 99.90th=[ 105], 99.95th=[ 105], | 99.99th=[ 105] bw ( KiB/s): min= 96, max= 1024, per=26.47%, avg=358.44, stdev=224.64, samples=34 iops : min= 6, max= 64, avg=21.94, stdev=14.18, samples=34 write: IOPS=21, BW=348KiB/s (357kB/s)(6032KiB/17315msec) clat (msec): min=5, max=111, avg=27.98, stdev=17.96 lat (msec): min=5, max=111, avg=27.98, stdev=17.96 clat percentiles (msec): | 1.00th=[ 9], 5.00th=[ 11], 10.00th=[ 12], 20.00th=[ 15], | 30.00th=[ 17], 40.00th=[ 19], 50.00th=[ 22], 60.00th=[ 25], | 70.00th=[ 33], 80.00th=[ 44], 90.00th=[ 55], 95.00th=[ 67], | 99.00th=[ 85], 99.50th=[ 96], 99.90th=[ 112], 99.95th=[ 112], | 99.99th=[ 112] bw ( KiB/s): min= 127, max= 800, per=23.98%, avg=341.53, stdev=181.62, samples=34 iops : min= 7, max= 50, avg=20.88, stdev=11.43, samples=34 lat (msec) : 4=0.39%, 10=24.74%, 20=36.46%, 50=28.78%, 100=9.24% lat (msec) : 250=0.39% cpu : usr=0.05%, sys=4.75%, ctx=874, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20933: Thu May 1 22:40:29 2025 read: IOPS=21, BW=341KiB/s (349kB/s)(6032KiB/17686msec) clat (usec): min=2943, max=86585, avg=17061.88, stdev=15524.52 lat (usec): min=2944, max=86586, avg=17065.07, stdev=15523.42 clat percentiles (usec): | 1.00th=[ 3654], 5.00th=[ 4883], 10.00th=[ 5538], 20.00th=[ 6652], | 30.00th=[ 7701], 40.00th=[ 9110], 50.00th=[10552], 60.00th=[13042], | 70.00th=[16909], 80.00th=[23462], 90.00th=[42206], 95.00th=[54789], | 99.00th=[67634], 99.50th=[84411], 99.90th=[86508], 99.95th=[86508], | 99.99th=[86508] bw ( KiB/s): min= 94, max= 894, per=24.77%, avg=335.43, stdev=198.85, samples=35 iops : min= 5, max= 55, avg=20.37, stdev=12.45, samples=35 write: IOPS=22, BW=354KiB/s (362kB/s)(6256KiB/17686msec) clat (msec): min=5, max=154, avg=28.35, stdev=20.23 lat (msec): min=5, max=154, avg=28.35, stdev=20.23 clat percentiles (msec): | 1.00th=[ 7], 5.00th=[ 10], 10.00th=[ 12], 20.00th=[ 14], | 30.00th=[ 16], 40.00th=[ 18], 50.00th=[ 21], 60.00th=[ 25], | 70.00th=[ 32], 80.00th=[ 43], 90.00th=[ 57], 95.00th=[ 68], | 99.00th=[ 105], 99.50th=[ 117], 99.90th=[ 155], 99.95th=[ 155], | 99.99th=[ 155] bw ( KiB/s): min= 127, max= 832, per=24.44%, avg=348.09, stdev=207.40, samples=35 iops : min= 7, max= 52, avg=21.17, stdev=13.04, samples=35 lat (msec) : 4=0.78%, 10=25.00%, 20=35.42%, 50=28.65%, 100=9.64% lat (msec) : 250=0.52% cpu : usr=0.08%, sys=4.57%, ctx=888, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1354KiB/s (1387kB/s), 326KiB/s-361KiB/s (334kB/s-370kB/s), io=23.4MiB (24.5MB), run=17075-17686msec WRITE: bw=1425KiB/s (1459kB/s), 348KiB/s-373KiB/s (357kB/s-382kB/s), io=24.6MiB (25.8MB), run=17075-17686msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=20959: Thu May 1 22:40:32 2025 read: IOPS=4, BW=4534KiB/s (4643kB/s)(7168KiB/1581msec) clat (usec): min=593, max=184919, avg=102085.08, stdev=66816.28 lat (usec): min=594, max=184920, avg=102086.59, stdev=66816.26 clat percentiles (usec): | 1.00th=[ 594], 5.00th=[ 594], 10.00th=[ 594], 20.00th=[ 34866], | 30.00th=[ 99091], 40.00th=[ 99091], 50.00th=[100140], 60.00th=[126354], | 70.00th=[126354], 80.00th=[168821], 90.00th=[185598], 95.00th=[185598], | 99.00th=[185598], 99.50th=[185598], 99.90th=[185598], 99.95th=[185598], | 99.99th=[185598] bw ( KiB/s): min= 2011, max= 5976, per=41.01%, avg=4027.67, stdev=1983.38, samples=3 iops : min= 1, max= 5, avg= 3.33, stdev= 2.08, samples=3 write: IOPS=3, BW=3238KiB/s (3316kB/s)(5120KiB/1581msec) clat (msec): min=61, max=222, avg=114.93, stdev=67.06 lat (msec): min=61, max=222, avg=114.98, stdev=67.05 clat percentiles (msec): | 1.00th=[ 62], 5.00th=[ 62], 10.00th=[ 62], 20.00th=[ 62], | 30.00th=[ 72], 40.00th=[ 72], 50.00th=[ 81], 60.00th=[ 81], | 70.00th=[ 140], 80.00th=[ 140], 90.00th=[ 222], 95.00th=[ 222], | 99.00th=[ 222], 99.50th=[ 222], 99.90th=[ 222], 99.95th=[ 222], | 99.99th=[ 222] bw ( KiB/s): min= 2048, max= 5976, per=29.18%, avg=4012.00, stdev=2777.52, samples=2 iops : min= 2, max= 5, avg= 3.50, stdev= 2.12, samples=2 lat (usec) : 750=8.33% lat (msec) : 50=8.33%, 100=41.67%, 250=41.67% cpu : usr=0.13%, sys=14.24%, ctx=180, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20961: Thu May 1 22:40:32 2025 read: IOPS=1, BW=1298KiB/s (1329kB/s)(2048KiB/1578msec) clat (usec): min=1831, max=81743, avg=41787.58, stdev=56506.32 lat (usec): min=1832, max=81746, avg=41789.28, stdev=56507.72 clat percentiles (usec): | 1.00th=[ 1827], 5.00th=[ 1827], 10.00th=[ 1827], 20.00th=[ 1827], | 30.00th=[ 1827], 40.00th=[ 1827], 50.00th=[ 1827], 60.00th=[81265], | 70.00th=[81265], 80.00th=[81265], 90.00th=[81265], 95.00th=[81265], | 99.00th=[81265], 99.50th=[81265], 99.90th=[81265], 99.95th=[81265], | 99.99th=[81265] bw ( KiB/s): min= 2000, max= 2039, per=20.56%, avg=2019.50, stdev=27.58, samples=2 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=2 write: IOPS=6, BW=6489KiB/s (6645kB/s)(10.0MiB/1578msec) clat (msec): min=68, max=308, avg=147.83, stdev=89.73 lat (msec): min=68, max=308, avg=147.89, stdev=89.73 clat percentiles (msec): | 1.00th=[ 69], 5.00th=[ 69], 10.00th=[ 69], 20.00th=[ 69], | 30.00th=[ 91], 40.00th=[ 91], 50.00th=[ 96], 60.00th=[ 97], | 70.00th=[ 247], 80.00th=[ 247], 90.00th=[ 259], 95.00th=[ 309], | 99.00th=[ 309], 99.50th=[ 309], 99.90th=[ 309], 99.95th=[ 309], | 99.99th=[ 309] bw ( KiB/s): min= 4000, max= 7937, per=43.77%, avg=6018.67, stdev=1970.42, samples=3 iops : min= 3, max= 7, avg= 5.00, stdev= 2.00, samples=3 lat (msec) : 2=8.33%, 100=58.33%, 250=16.67%, 500=16.67% cpu : usr=0.00%, sys=13.38%, ctx=114, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20962: Thu May 1 22:40:32 2025 read: IOPS=2, BW=2947KiB/s (3017kB/s)(6144KiB/2085msec) clat (usec): min=516, max=120217, avg=70572.85, stdev=57531.40 lat (usec): min=518, max=120219, avg=70574.54, stdev=57531.46 clat percentiles (usec): | 1.00th=[ 519], 5.00th=[ 519], 10.00th=[ 519], 20.00th=[ 529], | 30.00th=[ 529], 40.00th=[ 68682], 50.00th=[ 68682], 60.00th=[115868], | 70.00th=[117965], 80.00th=[117965], 90.00th=[120062], 95.00th=[120062], | 99.00th=[120062], 99.50th=[120062], 99.90th=[120062], 99.95th=[120062], | 99.99th=[120062] bw ( KiB/s): min= 1976, max= 5731, per=33.11%, avg=3251.67, stdev=2147.47, samples=3 iops : min= 1, max= 5, avg= 2.67, stdev= 2.08, samples=3 write: IOPS=2, BW=2947KiB/s (3017kB/s)(6144KiB/2085msec) clat (msec): min=72, max=578, avg=252.21, stdev=203.65 lat (msec): min=72, max=578, avg=252.26, stdev=203.67 clat percentiles (msec): | 1.00th=[ 73], 5.00th=[ 73], 10.00th=[ 73], 20.00th=[ 109], | 30.00th=[ 109], 40.00th=[ 146], 50.00th=[ 146], 60.00th=[ 180], | 70.00th=[ 430], 80.00th=[ 430], 90.00th=[ 575], 95.00th=[ 575], | 99.00th=[ 575], 99.50th=[ 575], 99.90th=[ 575], 99.95th=[ 575], | 99.99th=[ 575] bw ( KiB/s): min= 2048, max= 5731, per=28.51%, avg=3921.00, stdev=1842.31, samples=3 iops : min= 2, max= 5, avg= 3.33, stdev= 1.53, samples=3 lat (usec) : 750=16.67% lat (msec) : 100=16.67%, 250=50.00%, 500=8.33%, 750=8.33% cpu : usr=0.05%, sys=11.32%, ctx=290, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20963: Thu May 1 22:40:32 2025 read: IOPS=2, BW=2843KiB/s (2911kB/s)(5120KiB/1801msec) clat (usec): min=542, max=122507, avg=64557.24, stdev=49731.71 lat (usec): min=544, max=122509, avg=64558.96, stdev=49731.54 clat percentiles (usec): | 1.00th=[ 545], 5.00th=[ 545], 10.00th=[ 545], 20.00th=[ 545], | 30.00th=[ 43779], 40.00th=[ 43779], 50.00th=[ 49546], 60.00th=[ 49546], | 70.00th=[106431], 80.00th=[106431], 90.00th=[122160], 95.00th=[122160], | 99.00th=[122160], 99.50th=[122160], 99.90th=[122160], 99.95th=[122160], | 99.99th=[122160] bw ( KiB/s): min= 8143, max= 8143, per=82.91%, avg=8143.00, stdev= 0.00, samples=1 iops : min= 7, max= 7, avg= 7.00, stdev= 0.00, samples=1 write: IOPS=3, BW=3980KiB/s (4076kB/s)(7168KiB/1801msec) clat (msec): min=44, max=567, avg=200.53, stdev=188.90 lat (msec): min=44, max=567, avg=200.61, stdev=188.90 clat percentiles (msec): | 1.00th=[ 44], 5.00th=[ 44], 10.00th=[ 44], 20.00th=[ 55], | 30.00th=[ 67], 40.00th=[ 67], 50.00th=[ 144], 60.00th=[ 218], | 70.00th=[ 218], 80.00th=[ 309], 90.00th=[ 567], 95.00th=[ 567], | 99.00th=[ 567], 99.50th=[ 567], 99.90th=[ 567], 99.95th=[ 567], | 99.99th=[ 567] bw ( KiB/s): min= 2019, max= 6107, per=24.63%, avg=3387.00, stdev=2355.60, samples=3 iops : min= 1, max= 5, avg= 2.33, stdev= 2.31, samples=3 lat (usec) : 750=8.33% lat (msec) : 50=25.00%, 100=16.67%, 250=33.33%, 500=8.33%, 750=8.33% cpu : usr=0.00%, sys=12.00%, ctx=97, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=9823KiB/s (10.1MB/s), 1298KiB/s-4534KiB/s (1329kB/s-4643kB/s), io=20.0MiB (20.0MB), run=1578-2085msec WRITE: bw=13.4MiB/s (14.1MB/s), 2947KiB/s-6489KiB/s (3017kB/s-6645kB/s), io=28.0MiB (29.4MB), run=1578-2085msec rand-rw: (groupid=0, jobs=1): err= 0: pid=20956: Thu May 1 22:40:33 2025 read: IOPS=3, BW=3104KiB/s (3179kB/s)(7168KiB/2309msec) clat (msec): min=42, max=547, avg=208.66, stdev=174.03 lat (msec): min=42, max=547, avg=208.66, stdev=174.03 clat percentiles (msec): | 1.00th=[ 43], 5.00th=[ 43], 10.00th=[ 43], 20.00th=[ 77], | 30.00th=[ 109], 40.00th=[ 109], 50.00th=[ 148], 60.00th=[ 245], | 70.00th=[ 245], 80.00th=[ 292], 90.00th=[ 550], 95.00th=[ 550], | 99.00th=[ 550], 99.50th=[ 550], 99.90th=[ 550], 99.95th=[ 550], | 99.99th=[ 550] bw ( KiB/s): min= 2048, max= 4096, per=38.15%, avg=3383.67, stdev=1157.58, samples=3 iops : min= 2, max= 4, avg= 3.00, stdev= 1.00, samples=3 write: IOPS=2, BW=2217KiB/s (2271kB/s)(5120KiB/2309msec) clat (msec): min=40, max=211, avg=128.71, stdev=73.62 lat (msec): min=40, max=211, avg=128.79, stdev=73.62 clat percentiles (msec): | 1.00th=[ 41], 5.00th=[ 41], 10.00th=[ 41], 20.00th=[ 41], | 30.00th=[ 63], 40.00th=[ 63], 50.00th=[ 161], 60.00th=[ 161], | 70.00th=[ 169], 80.00th=[ 169], 90.00th=[ 211], 95.00th=[ 211], | 99.00th=[ 211], 99.50th=[ 211], 99.90th=[ 211], 99.95th=[ 211], | 99.99th=[ 211] bw ( KiB/s): min= 2019, max= 4096, per=24.62%, avg=3057.50, stdev=1468.66, samples=2 iops : min= 1, max= 4, avg= 2.50, stdev= 2.12, samples=2 lat (msec) : 50=16.67%, 100=16.67%, 250=50.00%, 500=8.33%, 750=8.33% cpu : usr=0.39%, sys=2.77%, ctx=47, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20957: Thu May 1 22:40:33 2025 read: IOPS=0, BW=901KiB/s (923kB/s)(2048KiB/2272msec) clat (msec): min=109, max=203, avg=156.36, stdev=66.16 lat (msec): min=109, max=203, avg=156.36, stdev=66.16 clat percentiles (msec): | 1.00th=[ 110], 5.00th=[ 110], 10.00th=[ 110], 20.00th=[ 110], | 30.00th=[ 110], 40.00th=[ 110], 50.00th=[ 110], 60.00th=[ 203], | 70.00th=[ 203], 80.00th=[ 203], 90.00th=[ 203], 95.00th=[ 203], | 99.00th=[ 203], 99.50th=[ 203], 99.90th=[ 203], 99.95th=[ 203], | 99.99th=[ 203] bw ( KiB/s): min= 2003, max= 2048, per=22.84%, avg=2025.50, stdev=31.82, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 write: IOPS=4, BW=4507KiB/s (4615kB/s)(10.0MiB/2272msec) clat (msec): min=42, max=639, avg=190.69, stdev=171.65 lat (msec): min=42, max=640, avg=190.81, stdev=171.66 clat percentiles (msec): | 1.00th=[ 43], 5.00th=[ 43], 10.00th=[ 43], 20.00th=[ 73], | 30.00th=[ 87], 40.00th=[ 97], 50.00th=[ 142], 60.00th=[ 157], | 70.00th=[ 211], 80.00th=[ 211], 90.00th=[ 251], 95.00th=[ 642], | 99.00th=[ 642], 99.50th=[ 642], 99.90th=[ 642], 99.95th=[ 642], | 99.99th=[ 642] bw ( KiB/s): min= 2003, max= 6107, per=24.52%, avg=3044.25, stdev=2041.92, samples=4 iops : min= 1, max= 5, avg= 2.25, stdev= 1.89, samples=4 lat (msec) : 50=8.33%, 100=25.00%, 250=50.00%, 500=8.33%, 750=8.33% cpu : usr=0.00%, sys=5.20%, ctx=65, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20958: Thu May 1 22:40:33 2025 read: IOPS=2, BW=3060KiB/s (3133kB/s)(6144KiB/2008msec) clat (msec): min=23, max=233, avg=125.85, stdev=73.55 lat (msec): min=23, max=233, avg=125.86, stdev=73.55 clat percentiles (msec): | 1.00th=[ 24], 5.00th=[ 24], 10.00th=[ 24], 20.00th=[ 72], | 30.00th=[ 72], 40.00th=[ 115], 50.00th=[ 115], 60.00th=[ 150], | 70.00th=[ 161], 80.00th=[ 161], 90.00th=[ 234], 95.00th=[ 234], | 99.00th=[ 234], 99.50th=[ 234], 99.90th=[ 234], 99.95th=[ 234], | 99.99th=[ 234] bw ( KiB/s): min= 2023, max= 4079, per=38.02%, avg=3372.33, stdev=1169.00, samples=3 iops : min= 1, max= 3, avg= 2.33, stdev= 1.15, samples=3 write: IOPS=2, BW=3060KiB/s (3133kB/s)(6144KiB/2008msec) clat (msec): min=71, max=450, avg=186.14, stdev=140.22 lat (msec): min=71, max=450, avg=186.21, stdev=140.22 clat percentiles (msec): | 1.00th=[ 72], 5.00th=[ 72], 10.00th=[ 72], 20.00th=[ 100], | 30.00th=[ 100], 40.00th=[ 128], 50.00th=[ 128], 60.00th=[ 138], | 70.00th=[ 232], 80.00th=[ 232], 90.00th=[ 451], 95.00th=[ 451], | 99.00th=[ 451], 99.50th=[ 451], 99.90th=[ 451], 99.95th=[ 451], | 99.99th=[ 451] bw ( KiB/s): min= 2039, max= 6131, per=32.80%, avg=4072.33, stdev=2046.12, samples=3 iops : min= 1, max= 5, avg= 3.00, stdev= 2.00, samples=3 lat (msec) : 50=8.33%, 100=25.00%, 250=58.33%, 500=8.33% cpu : usr=0.35%, sys=2.64%, ctx=41, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20960: Thu May 1 22:40:33 2025 read: IOPS=2, BW=2381KiB/s (2439kB/s)(5120KiB/2150msec) clat (usec): min=33437, max=85410, avg=56241.05, stdev=25188.65 lat (usec): min=33438, max=85412, avg=56242.47, stdev=25188.42 clat percentiles (usec): | 1.00th=[33424], 5.00th=[33424], 10.00th=[33424], 20.00th=[33424], | 30.00th=[40109], 40.00th=[40109], 50.00th=[40633], 60.00th=[40633], | 70.00th=[82314], 80.00th=[82314], 90.00th=[85459], 95.00th=[85459], | 99.00th=[85459], 99.50th=[85459], 99.90th=[85459], 99.95th=[85459], | 99.99th=[85459] bw ( KiB/s): min= 8159, max= 8159, per=91.99%, avg=8159.00, stdev= 0.00, samples=1 iops : min= 7, max= 7, avg= 7.00, stdev= 0.00, samples=1 write: IOPS=3, BW=3334KiB/s (3414kB/s)(7168KiB/2150msec) clat (msec): min=94, max=468, avg=226.48, stdev=121.84 lat (msec): min=94, max=469, avg=226.56, stdev=121.84 clat percentiles (msec): | 1.00th=[ 94], 5.00th=[ 94], 10.00th=[ 94], 20.00th=[ 144], | 30.00th=[ 180], 40.00th=[ 180], 50.00th=[ 184], 60.00th=[ 253], | 70.00th=[ 253], 80.00th=[ 264], 90.00th=[ 468], 95.00th=[ 468], | 99.00th=[ 468], 99.50th=[ 468], 99.90th=[ 468], 99.95th=[ 468], | 99.99th=[ 468] bw ( KiB/s): min= 2048, max= 6144, per=32.92%, avg=4087.67, stdev=2048.05, samples=3 iops : min= 2, max= 6, avg= 3.67, stdev= 2.08, samples=3 lat (msec) : 50=25.00%, 100=25.00%, 250=25.00%, 500=25.00% cpu : usr=0.09%, sys=1.91%, ctx=39, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=8870KiB/s (9083kB/s), 901KiB/s-3104KiB/s (923kB/s-3179kB/s), io=20.0MiB (20.0MB), run=2008-2309msec WRITE: bw=12.1MiB/s (12.7MB/s), 2217KiB/s-4507KiB/s (2271kB/s-4615kB/s), io=28.0MiB (29.4MB), run=2008-2309msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=20976: Thu May 1 22:40:37 2025 read: IOPS=2, BW=10.5MiB/s (11.1MB/s)(12.0MiB/1138msec) clat (msec): min=231, max=499, avg=366.48, stdev=133.64 lat (msec): min=231, max=499, avg=366.48, stdev=133.64 clat percentiles (msec): | 1.00th=[ 232], 5.00th=[ 232], 10.00th=[ 232], 20.00th=[ 232], | 30.00th=[ 232], 40.00th=[ 368], 50.00th=[ 368], 60.00th=[ 368], | 70.00th=[ 502], 80.00th=[ 502], 90.00th=[ 502], 95.00th=[ 502], | 99.00th=[ 502], 99.50th=[ 502], 99.90th=[ 502], 99.95th=[ 502], | 99.99th=[ 502] bw ( KiB/s): min= 8175, max= 8175, per=92.47%, avg=8175.00, stdev= 0.00, samples=2 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=2 lat (msec) : 250=33.33%, 500=66.67% cpu : usr=0.44%, sys=11.96%, ctx=375, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20977: Thu May 1 22:40:37 2025 read: IOPS=0, BW=2210KiB/s (2264kB/s)(4096KiB/1853msec) clat (nsec): min=337005k, max=337005k, avg=337004973.00, stdev= 0.00 lat (nsec): min=337007k, max=337007k, avg=337007275.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 338], 5.00th=[ 338], 10.00th=[ 338], 20.00th=[ 338], | 30.00th=[ 338], 40.00th=[ 338], 50.00th=[ 338], 60.00th=[ 338], | 70.00th=[ 338], 80.00th=[ 338], 90.00th=[ 338], 95.00th=[ 338], | 99.00th=[ 338], 99.50th=[ 338], 99.90th=[ 338], 99.95th=[ 338], | 99.99th=[ 338] bw ( KiB/s): min= 8192, max= 8192, per=92.66%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=1, BW=4421KiB/s (4527kB/s)(8192KiB/1853msec) clat (msec): min=682, max=764, avg=723.75, stdev=57.76 lat (msec): min=683, max=765, avg=724.23, stdev=57.91 clat percentiles (msec): | 1.00th=[ 684], 5.00th=[ 684], 10.00th=[ 684], 20.00th=[ 684], | 30.00th=[ 684], 40.00th=[ 684], 50.00th=[ 684], 60.00th=[ 768], | 70.00th=[ 768], 80.00th=[ 768], 90.00th=[ 768], 95.00th=[ 768], | 99.00th=[ 768], 99.50th=[ 768], 99.90th=[ 768], 99.95th=[ 768], | 99.99th=[ 768] bw ( KiB/s): min= 6860, max= 6860, per=41.41%, avg=6860.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 500=33.33%, 750=33.33%, 1000=33.33% cpu : usr=0.00%, sys=12.80%, ctx=122, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20978: Thu May 1 22:40:37 2025 write: IOPS=1, BW=6212KiB/s (6361kB/s)(12.0MiB/1978msec) clat (msec): min=417, max=963, avg=650.06, stdev=281.97 lat (msec): min=417, max=964, avg=650.46, stdev=282.01 clat percentiles (msec): | 1.00th=[ 418], 5.00th=[ 418], 10.00th=[ 418], 20.00th=[ 418], | 30.00th=[ 418], 40.00th=[ 567], 50.00th=[ 567], 60.00th=[ 567], | 70.00th=[ 961], 80.00th=[ 961], 90.00th=[ 961], 95.00th=[ 961], | 99.00th=[ 961], 99.50th=[ 961], 99.90th=[ 961], 99.95th=[ 961], | 99.99th=[ 961] bw ( KiB/s): min= 8159, max= 8175, per=49.30%, avg=8167.00, stdev=11.31, samples=2 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=2 lat (msec) : 500=33.33%, 750=33.33%, 1000=33.33% cpu : usr=0.10%, sys=14.16%, ctx=65, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20979: Thu May 1 22:40:37 2025 write: IOPS=2, BW=8758KiB/s (8969kB/s)(12.0MiB/1403msec) clat (msec): min=259, max=555, avg=439.53, stdev=157.96 lat (msec): min=259, max=555, avg=439.84, stdev=157.99 clat percentiles (msec): | 1.00th=[ 259], 5.00th=[ 259], 10.00th=[ 259], 20.00th=[ 259], | 30.00th=[ 259], 40.00th=[ 506], 50.00th=[ 506], 60.00th=[ 506], | 70.00th=[ 558], 80.00th=[ 558], 90.00th=[ 558], 95.00th=[ 558], | 99.00th=[ 558], 99.50th=[ 558], 99.90th=[ 558], 99.95th=[ 558], | 99.99th=[ 558] bw ( KiB/s): min= 8175, max= 8175, per=49.35%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 500=33.33%, 750=66.67% cpu : usr=0.07%, sys=22.33%, ctx=90, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=8842KiB/s (9054kB/s), 2210KiB/s-10.5MiB/s (2264kB/s-11.1MB/s), io=16.0MiB (16.8MB), run=1138-1853msec WRITE: bw=16.2MiB/s (16.0MB/s), 4421KiB/s-8758KiB/s (4527kB/s-8969kB/s), io=32.0MiB (33.6MB), run=1403-1978msec rand-rw: (groupid=0, jobs=1): err= 0: pid=20972: Thu May 1 22:40:37 2025 read: IOPS=3, BW=12.8MiB/s (13.4MB/s)(12.0MiB/941msec) clat (msec): min=63, max=705, avg=299.58, stdev=352.94 lat (msec): min=63, max=705, avg=299.58, stdev=352.94 clat percentiles (msec): | 1.00th=[ 64], 5.00th=[ 64], 10.00th=[ 64], 20.00th=[ 64], | 30.00th=[ 64], 40.00th=[ 130], 50.00th=[ 130], 60.00th=[ 130], | 70.00th=[ 709], 80.00th=[ 709], 90.00th=[ 709], 95.00th=[ 709], | 99.00th=[ 709], 99.50th=[ 709], 99.90th=[ 709], 99.95th=[ 709], | 99.99th=[ 709] bw ( KiB/s): min=16221, max=16221, per=100.00%, avg=16221.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 250=33.33%, 750=33.33% cpu : usr=0.32%, sys=4.57%, ctx=21, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20973: Thu May 1 22:40:37 2025 read: IOPS=0, BW=1858KiB/s (1902kB/s)(4096KiB/2205msec) clat (nsec): min=613106k, max=613106k, avg=613106172.00, stdev= 0.00 lat (nsec): min=613109k, max=613109k, avg=613108620.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 617], 5.00th=[ 617], 10.00th=[ 617], 20.00th=[ 617], | 30.00th=[ 617], 40.00th=[ 617], 50.00th=[ 617], 60.00th=[ 617], | 70.00th=[ 617], 80.00th=[ 617], 90.00th=[ 617], 95.00th=[ 617], | 99.00th=[ 617], 99.50th=[ 617], 99.90th=[ 617], 99.95th=[ 617], | 99.99th=[ 617] bw ( KiB/s): min= 7340, max= 7340, per=98.79%, avg=7340.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=0, BW=3715KiB/s (3804kB/s)(8192KiB/2205msec) clat (msec): min=610, max=955, avg=783.10, stdev=244.24 lat (msec): min=610, max=956, avg=783.43, stdev=244.21 clat percentiles (msec): | 1.00th=[ 609], 5.00th=[ 609], 10.00th=[ 609], 20.00th=[ 609], | 30.00th=[ 609], 40.00th=[ 609], 50.00th=[ 609], 60.00th=[ 953], | 70.00th=[ 953], 80.00th=[ 953], 90.00th=[ 953], 95.00th=[ 953], | 99.00th=[ 953], 99.50th=[ 953], 99.90th=[ 953], 99.95th=[ 953], | 99.99th=[ 953] bw ( KiB/s): min= 7327, max= 7327, per=50.06%, avg=7327.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 750=66.67%, 1000=33.33% cpu : usr=0.09%, sys=6.17%, ctx=69, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20974: Thu May 1 22:40:37 2025 write: IOPS=1, BW=5488KiB/s (5620kB/s)(12.0MiB/2239msec) clat (msec): min=566, max=896, avg=712.13, stdev=168.23 lat (msec): min=567, max=896, avg=712.37, stdev=168.15 clat percentiles (msec): | 1.00th=[ 567], 5.00th=[ 567], 10.00th=[ 567], 20.00th=[ 567], | 30.00th=[ 567], 40.00th=[ 676], 50.00th=[ 676], 60.00th=[ 676], | 70.00th=[ 894], 80.00th=[ 894], 90.00th=[ 894], 95.00th=[ 894], | 99.00th=[ 894], 99.50th=[ 894], 99.90th=[ 894], 99.95th=[ 894], | 99.99th=[ 894] bw ( KiB/s): min= 7892, max= 7922, per=54.03%, avg=7907.00, stdev=21.21, samples=2 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=2 lat (msec) : 750=66.67%, 1000=33.33% cpu : usr=0.13%, sys=3.35%, ctx=37, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20975: Thu May 1 22:40:37 2025 write: IOPS=3, BW=14.1MiB/s (14.8MB/s)(12.0MiB/849msec) clat (msec): min=228, max=260, avg=246.05, stdev=16.08 lat (msec): min=228, max=260, avg=246.38, stdev=16.04 clat percentiles (msec): | 1.00th=[ 228], 5.00th=[ 228], 10.00th=[ 228], 20.00th=[ 228], | 30.00th=[ 228], 40.00th=[ 249], 50.00th=[ 249], 60.00th=[ 249], | 70.00th=[ 262], 80.00th=[ 262], 90.00th=[ 262], 95.00th=[ 262], | 99.00th=[ 262], 99.50th=[ 262], 99.90th=[ 262], 99.95th=[ 262], | 99.99th=[ 262] bw ( KiB/s): min= 8192, max= 8192, per=55.98%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.47%, sys=2.36%, ctx=23, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=7430KiB/s (7609kB/s), 1858KiB/s-12.8MiB/s (1902kB/s-13.4MB/s), io=16.0MiB (16.8MB), run=941-2205msec WRITE: bw=14.3MiB/s (14.0MB/s), 3715KiB/s-14.1MiB/s (3804kB/s-14.8MB/s), io=32.0MiB (33.6MB), run=849-2239msec