== sanity test 398b: DIO and buffer IO race ============== 22:08:17 (1749607697) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.52348 s, 33.0 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=22280: Tue Jun 10 22:08:40 2025 read: IOPS=79, BW=317KiB/s (325kB/s)(5992KiB/18894msec) clat (usec): min=60, max=41202, avg=6957.74, stdev=4471.68 lat (usec): min=60, max=41203, avg=6962.50, stdev=4470.41 clat percentiles (usec): | 1.00th=[ 87], 5.00th=[ 116], 10.00th=[ 3032], 20.00th=[ 3851], | 30.00th=[ 4490], 40.00th=[ 5342], 50.00th=[ 6194], 60.00th=[ 7242], | 70.00th=[ 8455], 80.00th=[ 9634], 90.00th=[12125], 95.00th=[14746], | 99.00th=[22152], 99.50th=[24511], 99.90th=[36963], 99.95th=[41157], | 99.99th=[41157] bw ( KiB/s): min= 160, max= 432, per=25.64%, avg=315.89, stdev=58.88, samples=37 iops : min= 40, max= 108, avg=78.81, stdev=14.74, samples=37 write: IOPS=83, BW=333KiB/s (341kB/s)(6296KiB/18894msec) clat (usec): min=417, max=205144, avg=5322.97, stdev=7279.24 lat (usec): min=417, max=205145, avg=5324.09, stdev=7279.19 clat percentiles (usec): | 1.00th=[ 562], 5.00th=[ 693], 10.00th=[ 750], 20.00th=[ 824], | 30.00th=[ 996], 40.00th=[ 1598], 50.00th=[ 3785], 60.00th=[ 5145], | 70.00th=[ 6652], 80.00th=[ 9110], 90.00th=[ 12518], 95.00th=[ 15795], | 99.00th=[ 22414], 99.50th=[ 26608], 99.90th=[ 37487], 99.95th=[204473], | 99.99th=[204473] bw ( KiB/s): min= 184, max= 440, per=25.71%, avg=331.68, stdev=53.21, samples=37 iops : min= 46, max= 110, avg=82.76, stdev=13.25, samples=37 lat (usec) : 100=1.43%, 250=2.51%, 500=0.13%, 750=5.40%, 1000=10.25% lat (msec) : 2=6.25%, 4=10.94%, 10=45.87%, 20=15.59%, 50=1.60% lat (msec) : 250=0.03% cpu : usr=0.24%, sys=19.08%, ctx=5228, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22281: Tue Jun 10 22:08:40 2025 read: IOPS=77, BW=309KiB/s (317kB/s)(5940KiB/19200msec) clat (usec): min=63, max=42735, avg=7078.39, stdev=4731.93 lat (usec): min=63, max=42735, avg=7079.69, stdev=4732.76 clat percentiles (usec): | 1.00th=[ 90], 5.00th=[ 118], 10.00th=[ 3130], 20.00th=[ 3851], | 30.00th=[ 4490], 40.00th=[ 5211], 50.00th=[ 6194], 60.00th=[ 7242], | 70.00th=[ 8291], 80.00th=[ 9896], 90.00th=[12518], 95.00th=[15008], | 99.00th=[22676], 99.50th=[26870], 99.90th=[42206], 99.95th=[42730], | 99.99th=[42730] bw ( KiB/s): min= 144, max= 440, per=25.00%, avg=307.95, stdev=55.25, samples=38 iops : min= 36, max= 110, avg=76.79, stdev=13.81, samples=38 write: IOPS=82, BW=331KiB/s (339kB/s)(6348KiB/19200msec) clat (usec): min=435, max=212489, avg=5421.49, stdev=7452.85 lat (usec): min=435, max=212489, avg=5423.04, stdev=7453.41 clat percentiles (usec): | 1.00th=[ 553], 5.00th=[ 709], 10.00th=[ 758], 20.00th=[ 840], | 30.00th=[ 1037], 40.00th=[ 1729], 50.00th=[ 3884], 60.00th=[ 5276], | 70.00th=[ 6849], 80.00th=[ 9110], 90.00th=[ 12387], 95.00th=[ 16319], | 99.00th=[ 23200], 99.50th=[ 27132], 99.90th=[ 37487], 99.95th=[212861], | 99.99th=[212861] bw ( KiB/s): min= 152, max= 472, per=25.39%, avg=327.53, stdev=72.96, samples=38 iops : min= 38, max= 118, avg=81.68, stdev=18.23, samples=38 lat (usec) : 100=1.17%, 250=2.21%, 500=0.23%, 750=4.62%, 1000=10.25% lat (msec) : 2=6.71%, 4=11.46%, 10=45.51%, 20=15.66%, 50=2.15% lat (msec) : 250=0.03% cpu : usr=0.27%, sys=18.78%, ctx=5350, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22283: Tue Jun 10 22:08:40 2025 read: IOPS=78, BW=316KiB/s (324kB/s)(6028KiB/19080msec) clat (usec): min=49, max=209941, avg=7179.23, stdev=6960.35 lat (usec): min=49, max=209942, avg=7181.02, stdev=6960.91 clat percentiles (usec): | 1.00th=[ 88], 5.00th=[ 122], 10.00th=[ 2999], 20.00th=[ 3884], | 30.00th=[ 4555], 40.00th=[ 5342], 50.00th=[ 6194], 60.00th=[ 7308], | 70.00th=[ 8455], 80.00th=[ 9765], 90.00th=[ 11994], 95.00th=[ 14746], | 99.00th=[ 23987], 99.50th=[ 30802], 99.90th=[ 39060], 99.95th=[210764], | 99.99th=[210764] bw ( KiB/s): min= 208, max= 448, per=25.63%, avg=315.79, stdev=53.85, samples=38 iops : min= 52, max= 112, avg=78.79, stdev=13.51, samples=38 write: IOPS=82, BW=328KiB/s (336kB/s)(6260KiB/19080msec) clat (usec): min=462, max=34105, avg=5223.51, stdev=5123.57 lat (usec): min=463, max=34106, avg=5225.79, stdev=5123.83 clat percentiles (usec): | 1.00th=[ 578], 5.00th=[ 693], 10.00th=[ 766], 20.00th=[ 848], | 30.00th=[ 1020], 40.00th=[ 1893], 50.00th=[ 3916], 60.00th=[ 5211], | 70.00th=[ 6718], 80.00th=[ 8979], 90.00th=[12387], 95.00th=[15926], | 99.00th=[21627], 99.50th=[23725], 99.90th=[28443], 99.95th=[34341], | 99.99th=[34341] bw ( KiB/s): min= 216, max= 512, per=25.36%, avg=327.16, stdev=65.65, samples=38 iops : min= 54, max= 128, avg=81.63, stdev=16.38, samples=38 lat (usec) : 50=0.03%, 100=1.11%, 250=2.41%, 500=0.16%, 750=4.49% lat (usec) : 1000=10.51% lat (msec) : 2=5.79%, 4=12.11%, 10=45.57%, 20=15.92%, 50=1.86% lat (msec) : 250=0.03% cpu : usr=0.26%, sys=19.02%, ctx=5312, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22285: Tue Jun 10 22:08:40 2025 read: IOPS=77, BW=311KiB/s (318kB/s)(6052KiB/19477msec) clat (usec): min=52, max=56727, avg=6976.06, stdev=4659.46 lat (usec): min=53, max=56729, avg=6977.10, stdev=4659.33 clat percentiles (usec): | 1.00th=[ 89], 5.00th=[ 115], 10.00th=[ 2868], 20.00th=[ 3785], | 30.00th=[ 4490], 40.00th=[ 5276], 50.00th=[ 6194], 60.00th=[ 7177], | 70.00th=[ 8225], 80.00th=[ 9765], 90.00th=[12387], 95.00th=[15270], | 99.00th=[22938], 99.50th=[24773], 99.90th=[32637], 99.95th=[56886], | 99.99th=[56886] bw ( KiB/s): min= 128, max= 408, per=25.10%, avg=309.26, stdev=54.59, samples=38 iops : min= 32, max= 102, avg=77.16, stdev=13.64, samples=38 write: IOPS=80, BW=320KiB/s (328kB/s)(6236KiB/19477msec) clat (usec): min=392, max=201612, avg=5656.83, stdev=7456.03 lat (usec): min=393, max=201614, avg=5657.92, stdev=7456.24 clat percentiles (usec): | 1.00th=[ 553], 5.00th=[ 701], 10.00th=[ 758], 20.00th=[ 832], | 30.00th=[ 1020], 40.00th=[ 1762], 50.00th=[ 4047], 60.00th=[ 5604], | 70.00th=[ 7242], 80.00th=[ 9765], 90.00th=[ 13698], 95.00th=[ 16188], | 99.00th=[ 24773], 99.50th=[ 27657], 99.90th=[ 38011], 99.95th=[202376], | 99.99th=[202376] bw ( KiB/s): min= 160, max= 440, per=24.68%, avg=318.32, stdev=56.53, samples=38 iops : min= 40, max= 110, avg=79.42, stdev=14.18, samples=38 lat (usec) : 100=1.33%, 250=2.60%, 500=0.42%, 750=4.59%, 1000=10.09% lat (msec) : 2=6.12%, 4=11.59%, 10=44.14%, 20=16.76%, 50=2.28% lat (msec) : 100=0.03%, 250=0.03% cpu : usr=0.34%, sys=18.66%, ctx=5233, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1233KiB/s (1262kB/s), 309KiB/s-317KiB/s (317kB/s-325kB/s), io=23.4MiB (24.6MB), run=18894-19477msec WRITE: bw=1291KiB/s (1322kB/s), 320KiB/s-333KiB/s (328kB/s-341kB/s), io=24.6MiB (25.7MB), run=18894-19477msec rand-rw: (groupid=0, jobs=1): err= 0: pid=22282: Tue Jun 10 22:08:51 2025 read: IOPS=49, BW=198KiB/s (202kB/s)(5992KiB/30329msec) clat (msec): min=2, max=217, avg= 7.63, stdev= 7.77 lat (msec): min=2, max=217, avg= 7.64, stdev= 7.77 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 4], 10.00th=[ 4], 20.00th=[ 5], | 30.00th=[ 5], 40.00th=[ 5], 50.00th=[ 6], 60.00th=[ 7], | 70.00th=[ 8], 80.00th=[ 10], 90.00th=[ 15], 95.00th=[ 20], | 99.00th=[ 32], 99.50th=[ 36], 99.90th=[ 46], 99.95th=[ 218], | 99.99th=[ 218] bw ( KiB/s): min= 88, max= 359, per=25.14%, avg=197.07, stdev=68.26, samples=60 iops : min= 22, max= 89, avg=49.07, stdev=17.04, samples=60 write: IOPS=51, BW=208KiB/s (213kB/s)(6296KiB/30329msec) clat (usec): min=3796, max=76942, avg=11931.07, stdev=6231.37 lat (usec): min=3797, max=76943, avg=11933.13, stdev=6231.77 clat percentiles (usec): | 1.00th=[ 5145], 5.00th=[ 5997], 10.00th=[ 6783], 20.00th=[ 7701], | 30.00th=[ 8455], 40.00th=[ 9241], 50.00th=[10159], 60.00th=[11076], | 70.00th=[12649], 80.00th=[14877], 90.00th=[19530], 95.00th=[23987], | 99.00th=[38011], 99.50th=[42206], 99.90th=[52691], 99.95th=[77071], | 99.99th=[77071] bw ( KiB/s): min= 80, max= 352, per=25.11%, avg=206.13, stdev=67.29, samples=60 iops : min= 20, max= 88, avg=51.33, stdev=16.81, samples=60 lat (msec) : 4=8.98%, 10=55.86%, 20=28.52%, 50=6.54%, 100=0.07% lat (msec) : 250=0.03% cpu : usr=0.26%, sys=8.83%, ctx=3442, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22284: Tue Jun 10 22:08:51 2025 read: IOPS=48, BW=195KiB/s (200kB/s)(5940KiB/30464msec) clat (msec): min=2, max=203, avg= 7.51, stdev= 7.39 lat (msec): min=2, max=203, avg= 7.51, stdev= 7.39 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 4], 10.00th=[ 4], 20.00th=[ 5], | 30.00th=[ 5], 40.00th=[ 5], 50.00th=[ 6], 60.00th=[ 7], | 70.00th=[ 8], 80.00th=[ 10], 90.00th=[ 15], 95.00th=[ 20], | 99.00th=[ 31], 99.50th=[ 34], 99.90th=[ 47], 99.95th=[ 203], | 99.99th=[ 203] bw ( KiB/s): min= 64, max= 344, per=24.87%, avg=195.00, stdev=70.51, samples=60 iops : min= 16, max= 86, avg=48.60, stdev=17.64, samples=60 write: IOPS=52, BW=208KiB/s (213kB/s)(6348KiB/30464msec) clat (usec): min=4390, max=57122, avg=12099.36, stdev=6241.28 lat (usec): min=4390, max=57123, avg=12101.51, stdev=6241.01 clat percentiles (usec): | 1.00th=[ 5145], 5.00th=[ 6390], 10.00th=[ 6915], 20.00th=[ 7832], | 30.00th=[ 8586], 40.00th=[ 9241], 50.00th=[10290], 60.00th=[11338], | 70.00th=[12911], 80.00th=[15139], 90.00th=[19792], 95.00th=[23725], | 99.00th=[38536], 99.50th=[42206], 99.90th=[51643], 99.95th=[56886], | 99.99th=[56886] bw ( KiB/s): min= 80, max= 336, per=25.15%, avg=206.47, stdev=66.54, samples=60 iops : min= 20, max= 84, avg=51.47, stdev=16.64, samples=60 lat (msec) : 4=9.54%, 10=54.43%, 20=29.17%, 50=6.74%, 100=0.10% lat (msec) : 250=0.03% cpu : usr=0.09%, sys=9.01%, ctx=3459, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22286: Tue Jun 10 22:08:51 2025 read: IOPS=49, BW=199KiB/s (204kB/s)(6028KiB/30234msec) clat (usec): min=2429, max=62174, avg=7496.84, stdev=5373.16 lat (usec): min=2430, max=62174, avg=7498.20, stdev=5373.15 clat percentiles (usec): | 1.00th=[ 2966], 5.00th=[ 3359], 10.00th=[ 3621], 20.00th=[ 4113], | 30.00th=[ 4555], 40.00th=[ 4948], 50.00th=[ 5473], 60.00th=[ 6194], | 70.00th=[ 7373], 80.00th=[ 9634], 90.00th=[14615], 95.00th=[19268], | 99.00th=[26608], 99.50th=[30016], 99.90th=[42206], 99.95th=[62129], | 99.99th=[62129] bw ( KiB/s): min= 88, max= 376, per=25.37%, avg=198.93, stdev=67.79, samples=60 iops : min= 22, max= 94, avg=49.53, stdev=16.97, samples=60 write: IOPS=51, BW=207KiB/s (212kB/s)(6260KiB/30234msec) clat (msec): min=4, max=223, avg=12.04, stdev= 8.01 lat (msec): min=4, max=223, avg=12.04, stdev= 8.01 clat percentiles (msec): | 1.00th=[ 6], 5.00th=[ 6], 10.00th=[ 7], 20.00th=[ 8], | 30.00th=[ 9], 40.00th=[ 10], 50.00th=[ 11], 60.00th=[ 12], | 70.00th=[ 13], 80.00th=[ 16], 90.00th=[ 20], 95.00th=[ 25], | 99.00th=[ 35], 99.50th=[ 41], 99.90th=[ 58], 99.95th=[ 224], | 99.99th=[ 224] bw ( KiB/s): min= 104, max= 391, per=25.09%, avg=206.00, stdev=70.17, samples=60 iops : min= 26, max= 97, avg=51.30, stdev=17.52, samples=60 lat (msec) : 4=8.40%, 10=55.60%, 20=29.39%, 50=6.51%, 100=0.07% lat (msec) : 250=0.03% cpu : usr=0.15%, sys=8.93%, ctx=3470, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22287: Tue Jun 10 22:08:51 2025 read: IOPS=49, BW=198KiB/s (202kB/s)(6052KiB/30605msec) clat (usec): min=2432, max=37778, avg=7497.24, stdev=5495.38 lat (usec): min=2433, max=37779, avg=7498.39, stdev=5495.27 clat percentiles (usec): | 1.00th=[ 2900], 5.00th=[ 3392], 10.00th=[ 3654], 20.00th=[ 4080], | 30.00th=[ 4490], 40.00th=[ 4948], 50.00th=[ 5407], 60.00th=[ 6063], | 70.00th=[ 7177], 80.00th=[ 9503], 90.00th=[15139], 95.00th=[19530], | 99.00th=[29492], 99.50th=[32900], 99.90th=[36963], 99.95th=[38011], | 99.99th=[38011] bw ( KiB/s): min= 72, max= 359, per=25.17%, avg=197.36, stdev=71.57, samples=61 iops : min= 18, max= 89, avg=49.13, stdev=17.86, samples=61 write: IOPS=50, BW=204KiB/s (209kB/s)(6236KiB/30605msec) clat (msec): min=3, max=216, avg=12.29, stdev= 8.19 lat (msec): min=3, max=216, avg=12.30, stdev= 8.19 clat percentiles (msec): | 1.00th=[ 6], 5.00th=[ 7], 10.00th=[ 7], 20.00th=[ 8], | 30.00th=[ 9], 40.00th=[ 10], 50.00th=[ 11], 60.00th=[ 12], | 70.00th=[ 14], 80.00th=[ 16], 90.00th=[ 21], 95.00th=[ 26], | 99.00th=[ 36], 99.50th=[ 42], 99.90th=[ 77], 99.95th=[ 218], | 99.99th=[ 218] bw ( KiB/s): min= 80, max= 336, per=24.71%, avg=202.87, stdev=68.08, samples=61 iops : min= 20, max= 84, avg=50.51, stdev=17.02, samples=61 lat (msec) : 4=9.08%, 10=55.31%, 20=28.29%, 50=7.26%, 100=0.03% lat (msec) : 250=0.03% cpu : usr=0.17%, sys=9.05%, ctx=3490, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=785KiB/s (803kB/s), 195KiB/s-199KiB/s (200kB/s-204kB/s), io=23.4MiB (24.6MB), run=30234-30605msec WRITE: bw=821KiB/s (841kB/s), 204KiB/s-208KiB/s (209kB/s-213kB/s), io=24.6MiB (25.7MB), run=30234-30605msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=22324: Tue Jun 10 22:08:58 2025 read: IOPS=57, BW=925KiB/s (947kB/s)(5920KiB/6403msec) clat (usec): min=67, max=44661, avg=9060.78, stdev=6632.02 lat (usec): min=68, max=44662, avg=9061.71, stdev=6632.03 clat percentiles (usec): | 1.00th=[ 89], 5.00th=[ 111], 10.00th=[ 128], 20.00th=[ 4752], | 30.00th=[ 5997], 40.00th=[ 7046], 50.00th=[ 8094], 60.00th=[ 9503], | 70.00th=[11207], 80.00th=[13173], 90.00th=[16581], 95.00th=[20317], | 99.00th=[33162], 99.50th=[40633], 99.90th=[44827], 99.95th=[44827], | 99.99th=[44827] bw ( KiB/s): min= 608, max= 1280, per=24.42%, avg=892.75, stdev=190.00, samples=12 iops : min= 38, max= 80, avg=55.67, stdev=11.79, samples=12 write: IOPS=62, BW=995KiB/s (1018kB/s)(6368KiB/6403msec) clat (usec): min=716, max=53951, avg=7314.94, stdev=7353.76 lat (usec): min=717, max=53952, avg=7323.88, stdev=7374.03 clat percentiles (usec): | 1.00th=[ 783], 5.00th=[ 1090], 10.00th=[ 1188], 20.00th=[ 1352], | 30.00th=[ 1647], 40.00th=[ 2704], 50.00th=[ 5014], 60.00th=[ 6718], | 70.00th=[ 9241], 80.00th=[12518], 90.00th=[17957], 95.00th=[20841], | 99.00th=[32113], 99.50th=[34866], 99.90th=[53740], 99.95th=[53740], | 99.99th=[53740] bw ( KiB/s): min= 512, max= 1469, per=25.35%, avg=975.33, stdev=292.28, samples=12 iops : min= 32, max= 91, avg=60.83, stdev=18.14, samples=12 lat (usec) : 100=1.04%, 250=5.86%, 500=0.26%, 750=0.13%, 1000=1.17% lat (msec) : 2=16.41%, 4=6.77%, 10=36.72%, 20=25.39%, 50=6.12% lat (msec) : 100=0.13% cpu : usr=0.12%, sys=17.59%, ctx=1360, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22325: Tue Jun 10 22:08:58 2025 read: IOPS=57, BW=922KiB/s (944kB/s)(5744KiB/6228msec) clat (usec): min=61, max=42911, avg=9009.55, stdev=6689.35 lat (usec): min=62, max=42912, avg=9010.88, stdev=6689.36 clat percentiles (usec): | 1.00th=[ 75], 5.00th=[ 104], 10.00th=[ 125], 20.00th=[ 4080], | 30.00th=[ 5604], 40.00th=[ 6980], 50.00th=[ 8356], 60.00th=[ 9896], | 70.00th=[11469], 80.00th=[13566], 90.00th=[16188], 95.00th=[19530], | 99.00th=[32637], 99.50th=[42206], 99.90th=[42730], 99.95th=[42730], | 99.99th=[42730] bw ( KiB/s): min= 576, max= 1408, per=25.14%, avg=919.00, stdev=246.97, samples=12 iops : min= 36, max= 88, avg=57.17, stdev=15.49, samples=12 write: IOPS=65, BW=1051KiB/s (1076kB/s)(6544KiB/6228msec) clat (usec): min=704, max=35920, avg=7235.36, stdev=6837.54 lat (usec): min=705, max=35921, avg=7236.95, stdev=6837.61 clat percentiles (usec): | 1.00th=[ 914], 5.00th=[ 1106], 10.00th=[ 1221], 20.00th=[ 1418], | 30.00th=[ 1909], 40.00th=[ 3163], 50.00th=[ 5080], 60.00th=[ 7046], | 70.00th=[ 9110], 80.00th=[12518], 90.00th=[16712], 95.00th=[22152], | 99.00th=[29492], 99.50th=[31589], 99.90th=[35914], 99.95th=[35914], | 99.99th=[35914] bw ( KiB/s): min= 766, max= 1408, per=27.08%, avg=1041.75, stdev=224.91, samples=12 iops : min= 47, max= 88, avg=64.83, stdev=14.32, samples=12 lat (usec) : 100=1.69%, 250=5.21%, 500=0.39%, 750=0.26%, 1000=1.04% lat (msec) : 2=15.76%, 4=8.98%, 10=34.64%, 20=26.69%, 50=5.34% cpu : usr=0.16%, sys=18.21%, ctx=1377, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22326: Tue Jun 10 22:08:58 2025 read: IOPS=59, BW=955KiB/s (978kB/s)(6256KiB/6550msec) clat (usec): min=65, max=45640, avg=9240.25, stdev=6466.71 lat (usec): min=65, max=45642, avg=9241.36, stdev=6466.93 clat percentiles (usec): | 1.00th=[ 95], 5.00th=[ 115], 10.00th=[ 141], 20.00th=[ 4948], | 30.00th=[ 6063], 40.00th=[ 7046], 50.00th=[ 8356], 60.00th=[ 9896], | 70.00th=[11076], 80.00th=[13435], 90.00th=[17433], 95.00th=[19792], | 99.00th=[32113], 99.50th=[36439], 99.90th=[45876], 99.95th=[45876], | 99.99th=[45876] bw ( KiB/s): min= 480, max= 1440, per=26.04%, avg=952.00, stdev=243.33, samples=13 iops : min= 30, max= 90, avg=59.31, stdev=15.29, samples=13 write: IOPS=57, BW=921KiB/s (943kB/s)(6032KiB/6550msec) clat (usec): min=687, max=58820, avg=7417.21, stdev=7225.15 lat (usec): min=687, max=58821, avg=7421.79, stdev=7222.91 clat percentiles (usec): | 1.00th=[ 799], 5.00th=[ 1090], 10.00th=[ 1156], 20.00th=[ 1352], | 30.00th=[ 1696], 40.00th=[ 3032], 50.00th=[ 5866], 60.00th=[ 7701], | 70.00th=[ 9372], 80.00th=[12518], 90.00th=[16450], 95.00th=[22152], | 99.00th=[32637], 99.50th=[33162], 99.90th=[58983], 99.95th=[58983], | 99.99th=[58983] bw ( KiB/s): min= 576, max= 1312, per=23.46%, avg=902.69, stdev=228.80, samples=13 iops : min= 36, max= 82, avg=56.23, stdev=14.29, samples=13 lat (usec) : 100=0.78%, 250=5.47%, 500=0.65%, 750=0.26%, 1000=1.82% lat (msec) : 2=14.58%, 4=4.95%, 10=37.50%, 20=28.39%, 50=5.47% lat (msec) : 100=0.13% cpu : usr=0.14%, sys=17.67%, ctx=1379, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22327: Tue Jun 10 22:08:58 2025 read: IOPS=58, BW=939KiB/s (962kB/s)(6032KiB/6424msec) clat (usec): min=59, max=39323, avg=9362.47, stdev=6592.53 lat (usec): min=59, max=39324, avg=9363.76, stdev=6592.50 clat percentiles (usec): | 1.00th=[ 68], 5.00th=[ 105], 10.00th=[ 129], 20.00th=[ 4948], | 30.00th=[ 6194], 40.00th=[ 7373], 50.00th=[ 8455], 60.00th=[ 9634], | 70.00th=[11338], 80.00th=[13566], 90.00th=[16909], 95.00th=[22152], | 99.00th=[34341], 99.50th=[37487], 99.90th=[39584], 99.95th=[39584], | 99.99th=[39584] bw ( KiB/s): min= 608, max= 1312, per=25.00%, avg=914.00, stdev=172.56, samples=12 iops : min= 38, max= 82, avg=57.00, stdev=10.85, samples=12 write: IOPS=60, BW=974KiB/s (997kB/s)(6256KiB/6424msec) clat (usec): min=665, max=46981, avg=7008.52, stdev=7475.21 lat (usec): min=665, max=46982, avg=7010.13, stdev=7475.14 clat percentiles (usec): | 1.00th=[ 742], 5.00th=[ 1045], 10.00th=[ 1139], 20.00th=[ 1254], | 30.00th=[ 1500], 40.00th=[ 1958], 50.00th=[ 3392], 60.00th=[ 6390], | 70.00th=[ 9110], 80.00th=[13042], 90.00th=[17433], 95.00th=[21103], | 99.00th=[31851], 99.50th=[40109], 99.90th=[46924], 99.95th=[46924], | 99.99th=[46924] bw ( KiB/s): min= 766, max= 1440, per=25.15%, avg=967.33, stdev=225.77, samples=12 iops : min= 47, max= 90, avg=60.33, stdev=14.20, samples=12 lat (usec) : 100=2.21%, 250=4.30%, 500=0.26%, 750=0.65%, 1000=1.43% lat (msec) : 2=18.49%, 4=6.38%, 10=33.59%, 20=26.04%, 50=6.64% cpu : usr=0.02%, sys=17.84%, ctx=1341, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=3657KiB/s (3745kB/s), 922KiB/s-955KiB/s (944kB/s-978kB/s), io=23.4MiB (24.5MB), run=6228-6550msec WRITE: bw=3847KiB/s (3940kB/s), 921KiB/s-1051KiB/s (943kB/s-1076kB/s), io=24.6MiB (25.8MB), run=6228-6550msec rand-rw: (groupid=0, jobs=1): err= 0: pid=22328: Tue Jun 10 22:09:01 2025 read: IOPS=39, BW=634KiB/s (650kB/s)(5920KiB/9331msec) clat (usec): min=2772, max=70295, avg=9092.61, stdev=8193.34 lat (usec): min=2772, max=70296, avg=9095.05, stdev=8194.75 clat percentiles (usec): | 1.00th=[ 2900], 5.00th=[ 3392], 10.00th=[ 3654], 20.00th=[ 4015], | 30.00th=[ 4555], 40.00th=[ 5145], 50.00th=[ 5735], 60.00th=[ 6849], | 70.00th=[ 8848], 80.00th=[11731], 90.00th=[21103], 95.00th=[27132], | 99.00th=[35914], 99.50th=[53216], 99.90th=[70779], 99.95th=[70779], | 99.99th=[70779] bw ( KiB/s): min= 288, max= 1309, per=23.61%, avg=605.78, stdev=324.79, samples=18 iops : min= 18, max= 81, avg=37.67, stdev=20.24, samples=18 write: IOPS=42, BW=682KiB/s (699kB/s)(6368KiB/9331msec) clat (usec): min=3981, max=61559, avg=14564.16, stdev=9684.19 lat (usec): min=3983, max=61560, avg=14566.37, stdev=9684.03 clat percentiles (usec): | 1.00th=[ 5014], 5.00th=[ 6128], 10.00th=[ 6915], 20.00th=[ 7898], | 30.00th=[ 8586], 40.00th=[ 9372], 50.00th=[10421], 60.00th=[12518], | 70.00th=[15270], 80.00th=[21103], 90.00th=[28443], 95.00th=[34866], | 99.00th=[50594], 99.50th=[55837], 99.90th=[61604], 99.95th=[61604], | 99.99th=[61604] bw ( KiB/s): min= 288, max= 1280, per=24.21%, avg=653.78, stdev=353.56, samples=18 iops : min= 18, max= 80, avg=40.67, stdev=22.11, samples=18 lat (msec) : 4=9.51%, 10=50.26%, 20=23.96%, 50=15.49%, 100=0.78% cpu : usr=0.08%, sys=8.65%, ctx=885, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22329: Tue Jun 10 22:09:01 2025 read: IOPS=38, BW=623KiB/s (638kB/s)(5744KiB/9226msec) clat (usec): min=2600, max=56167, avg=8666.66, stdev=7537.33 lat (usec): min=2601, max=56168, avg=8667.94, stdev=7537.45 clat percentiles (usec): | 1.00th=[ 2966], 5.00th=[ 3359], 10.00th=[ 3752], 20.00th=[ 4113], | 30.00th=[ 4490], 40.00th=[ 5014], 50.00th=[ 5669], 60.00th=[ 6390], | 70.00th=[ 7701], 80.00th=[10945], 90.00th=[20055], 95.00th=[26608], | 99.00th=[35914], 99.50th=[36439], 99.90th=[56361], 99.95th=[56361], | 99.99th=[56361] bw ( KiB/s): min= 224, max= 1312, per=23.90%, avg=613.22, stdev=357.61, samples=18 iops : min= 14, max= 82, avg=38.22, stdev=22.40, samples=18 write: IOPS=44, BW=709KiB/s (726kB/s)(6544KiB/9226msec) clat (usec): min=4472, max=59329, avg=14740.02, stdev=9275.18 lat (usec): min=4474, max=59331, avg=14742.98, stdev=9274.89 clat percentiles (usec): | 1.00th=[ 5145], 5.00th=[ 6063], 10.00th=[ 6980], 20.00th=[ 7898], | 30.00th=[ 8848], 40.00th=[ 9896], 50.00th=[11207], 60.00th=[13304], | 70.00th=[16057], 80.00th=[21365], 90.00th=[27657], 95.00th=[34341], | 99.00th=[47449], 99.50th=[51643], 99.90th=[59507], 99.95th=[59507], | 99.99th=[59507] bw ( KiB/s): min= 416, max= 1312, per=25.34%, avg=684.22, stdev=308.63, samples=18 iops : min= 26, max= 82, avg=42.67, stdev=19.34, samples=18 lat (msec) : 4=8.33%, 10=50.39%, 20=24.61%, 50=16.02%, 100=0.65% cpu : usr=0.21%, sys=8.62%, ctx=889, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22330: Tue Jun 10 22:09:01 2025 read: IOPS=42, BW=687KiB/s (704kB/s)(6256KiB/9101msec) clat (usec): min=2841, max=50231, avg=8903.18, stdev=7419.61 lat (usec): min=2842, max=50232, avg=8904.27, stdev=7419.57 clat percentiles (usec): | 1.00th=[ 3032], 5.00th=[ 3523], 10.00th=[ 3884], 20.00th=[ 4424], | 30.00th=[ 4752], 40.00th=[ 5211], 50.00th=[ 5932], 60.00th=[ 6718], | 70.00th=[ 8356], 80.00th=[11600], 90.00th=[19530], 95.00th=[26346], | 99.00th=[37487], 99.50th=[39060], 99.90th=[50070], 99.95th=[50070], | 99.99th=[50070] bw ( KiB/s): min= 320, max= 1472, per=26.65%, avg=683.72, stdev=369.20, samples=18 iops : min= 20, max= 92, avg=42.50, stdev=22.98, samples=18 write: IOPS=41, BW=663KiB/s (679kB/s)(6032KiB/9101msec) clat (usec): min=4730, max=55001, avg=14785.59, stdev=8767.06 lat (usec): min=4732, max=55003, avg=14787.71, stdev=8766.96 clat percentiles (usec): | 1.00th=[ 5080], 5.00th=[ 6521], 10.00th=[ 7242], 20.00th=[ 8029], | 30.00th=[ 8848], 40.00th=[10028], 50.00th=[11469], 60.00th=[13698], | 70.00th=[16581], 80.00th=[21627], 90.00th=[26608], 95.00th=[32637], | 99.00th=[48497], 99.50th=[52691], 99.90th=[54789], 99.95th=[54789], | 99.99th=[54789] bw ( KiB/s): min= 320, max= 1216, per=24.01%, avg=648.33, stdev=280.36, samples=18 iops : min= 20, max= 76, avg=40.28, stdev=17.51, samples=18 lat (msec) : 4=6.51%, 10=52.34%, 20=24.74%, 50=16.02%, 100=0.39% cpu : usr=0.08%, sys=8.78%, ctx=881, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22331: Tue Jun 10 22:09:01 2025 read: IOPS=41, BW=659KiB/s (674kB/s)(6032KiB/9160msec) clat (usec): min=2519, max=82542, avg=9254.27, stdev=8547.25 lat (usec): min=2520, max=82543, avg=9255.48, stdev=8547.14 clat percentiles (usec): | 1.00th=[ 3064], 5.00th=[ 3425], 10.00th=[ 3621], 20.00th=[ 4228], | 30.00th=[ 4686], 40.00th=[ 5407], 50.00th=[ 6259], 60.00th=[ 7111], | 70.00th=[ 8848], 80.00th=[11863], 90.00th=[20317], 95.00th=[28967], | 99.00th=[38011], 99.50th=[45876], 99.90th=[82314], 99.95th=[82314], | 99.99th=[82314] bw ( KiB/s): min= 320, max= 1248, per=25.01%, avg=641.67, stdev=295.47, samples=18 iops : min= 20, max= 78, avg=40.06, stdev=18.49, samples=18 write: IOPS=42, BW=683KiB/s (699kB/s)(6256KiB/9160msec) clat (usec): min=4810, max=72120, avg=14057.75, stdev=8612.13 lat (usec): min=4811, max=72122, avg=14059.99, stdev=8612.09 clat percentiles (usec): | 1.00th=[ 5276], 5.00th=[ 6194], 10.00th=[ 7111], 20.00th=[ 7963], | 30.00th=[ 8848], 40.00th=[ 9634], 50.00th=[10683], 60.00th=[11994], | 70.00th=[15270], 80.00th=[20579], 90.00th=[25822], 95.00th=[29492], | 99.00th=[44303], 99.50th=[51119], 99.90th=[71828], 99.95th=[71828], | 99.99th=[71828] bw ( KiB/s): min= 288, max= 1344, per=24.82%, avg=670.11, stdev=340.52, samples=18 iops : min= 18, max= 84, avg=41.83, stdev=21.29, samples=18 lat (msec) : 4=8.46%, 10=50.78%, 20=25.13%, 50=15.23%, 100=0.39% cpu : usr=0.35%, sys=8.29%, ctx=910, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2567KiB/s (2629kB/s), 623KiB/s-687KiB/s (638kB/s-704kB/s), io=23.4MiB (24.5MB), run=9101-9331msec WRITE: bw=2701KiB/s (2765kB/s), 663KiB/s-709KiB/s (679kB/s-726kB/s), io=24.6MiB (25.8MB), run=9101-9331msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=22345: Tue Jun 10 22:09:04 2025 read: IOPS=3, BW=4002KiB/s (4098kB/s)(7168KiB/1791msec) clat (usec): min=556, max=1197.1k, avg=197040.16, stdev=441662.43 lat (usec): min=558, max=1197.1k, avg=197042.00, stdev=441662.57 clat percentiles (usec): | 1.00th=[ 553], 5.00th=[ 553], 10.00th=[ 553], | 20.00th=[ 586], 30.00th=[ 31589], 40.00th=[ 31589], | 50.00th=[ 38536], 60.00th=[ 38536], 70.00th=[ 38536], | 80.00th=[ 71828], 90.00th=[1199571], 95.00th=[1199571], | 99.00th=[1199571], 99.50th=[1199571], 99.90th=[1199571], | 99.95th=[1199571], 99.99th=[1199571] bw ( KiB/s): min= 4096, max= 6131, per=44.72%, avg=5113.50, stdev=1438.96, samples=2 iops : min= 4, max= 5, avg= 4.50, stdev= 0.71, samples=2 write: IOPS=2, BW=2859KiB/s (2927kB/s)(5120KiB/1791msec) clat (msec): min=45, max=111, avg=76.24, stdev=28.75 lat (msec): min=45, max=111, avg=76.30, stdev=28.75 clat percentiles (msec): | 1.00th=[ 46], 5.00th=[ 46], 10.00th=[ 46], 20.00th=[ 46], | 30.00th=[ 48], 40.00th=[ 48], 50.00th=[ 87], 60.00th=[ 87], | 70.00th=[ 90], 80.00th=[ 90], 90.00th=[ 112], 95.00th=[ 112], | 99.00th=[ 112], 99.50th=[ 112], 99.90th=[ 112], 99.95th=[ 112], | 99.99th=[ 112] bw ( KiB/s): min= 4087, max= 4087, per=25.53%, avg=4087.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 lat (usec) : 750=16.67% lat (msec) : 50=41.67%, 100=25.00%, 250=8.33% cpu : usr=0.00%, sys=17.32%, ctx=674, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22346: Tue Jun 10 22:09:04 2025 read: IOPS=1, BW=1290KiB/s (1321kB/s)(2048KiB/1587msec) clat (usec): min=38754, max=45241, avg=41998.04, stdev=4586.66 lat (usec): min=38757, max=45243, avg=42000.60, stdev=4586.15 clat percentiles (usec): | 1.00th=[38536], 5.00th=[38536], 10.00th=[38536], 20.00th=[38536], | 30.00th=[38536], 40.00th=[38536], 50.00th=[38536], 60.00th=[45351], | 70.00th=[45351], 80.00th=[45351], 90.00th=[45351], 95.00th=[45351], | 99.00th=[45351], 99.50th=[45351], 99.90th=[45351], 99.95th=[45351], | 99.99th=[45351] bw ( KiB/s): min= 2048, max= 2048, per=17.91%, avg=2048.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 write: IOPS=6, BW=6452KiB/s (6607kB/s)(10.0MiB/1587msec) clat (msec): min=35, max=856, avg=149.14, stdev=252.75 lat (msec): min=35, max=857, avg=149.29, stdev=252.75 clat percentiles (msec): | 1.00th=[ 36], 5.00th=[ 36], 10.00th=[ 36], 20.00th=[ 36], | 30.00th=[ 40], 40.00th=[ 49], 50.00th=[ 49], 60.00th=[ 53], | 70.00th=[ 148], 80.00th=[ 148], 90.00th=[ 157], 95.00th=[ 860], | 99.00th=[ 860], 99.50th=[ 860], 99.90th=[ 860], 99.95th=[ 860], | 99.99th=[ 860] bw ( KiB/s): min= 6144, max=10240, per=51.17%, avg=8192.00, stdev=2896.31, samples=2 iops : min= 6, max= 10, avg= 8.00, stdev= 2.83, samples=2 lat (msec) : 50=58.33%, 100=16.67%, 250=16.67%, 1000=8.33% cpu : usr=0.06%, sys=22.51%, ctx=143, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22347: Tue Jun 10 22:09:04 2025 read: IOPS=3, BW=3847KiB/s (3940kB/s)(6144KiB/1597msec) clat (usec): min=469, max=53820, avg=9436.96, stdev=21743.75 lat (usec): min=470, max=53822, avg=9438.74, stdev=21743.77 clat percentiles (usec): | 1.00th=[ 469], 5.00th=[ 469], 10.00th=[ 469], 20.00th=[ 498], | 30.00th=[ 498], 40.00th=[ 537], 50.00th=[ 537], 60.00th=[ 594], | 70.00th=[ 701], 80.00th=[ 701], 90.00th=[53740], 95.00th=[53740], | 99.00th=[53740], 99.50th=[53740], 99.90th=[53740], 99.95th=[53740], | 99.99th=[53740] bw ( KiB/s): min= 4071, max= 4071, per=35.60%, avg=4071.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 write: IOPS=3, BW=3847KiB/s (3940kB/s)(6144KiB/1597msec) clat (msec): min=25, max=1272, avg=252.15, stdev=500.34 lat (msec): min=25, max=1272, avg=252.21, stdev=500.37 clat percentiles (msec): | 1.00th=[ 26], 5.00th=[ 26], 10.00th=[ 26], 20.00th=[ 35], | 30.00th=[ 35], 40.00th=[ 43], 50.00th=[ 43], 60.00th=[ 49], | 70.00th=[ 90], 80.00th=[ 90], 90.00th=[ 1267], 95.00th=[ 1267], | 99.00th=[ 1267], 99.50th=[ 1267], 99.90th=[ 1267], 99.95th=[ 1267], | 99.99th=[ 1267] bw ( KiB/s): min= 2035, max= 4096, per=19.15%, avg=3065.50, stdev=1457.35, samples=2 iops : min= 1, max= 4, avg= 2.50, stdev= 2.12, samples=2 lat (usec) : 500=16.67%, 750=25.00% lat (msec) : 50=33.33%, 100=16.67% cpu : usr=0.00%, sys=13.60%, ctx=163, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22348: Tue Jun 10 22:09:04 2025 read: IOPS=2, BW=2880KiB/s (2949kB/s)(5120KiB/1778msec) clat (usec): min=533, max=54734, avg=30660.92, stdev=27630.73 lat (usec): min=535, max=54735, avg=30662.33, stdev=27630.29 clat percentiles (usec): | 1.00th=[ 537], 5.00th=[ 537], 10.00th=[ 537], 20.00th=[ 537], | 30.00th=[ 553], 40.00th=[ 553], 50.00th=[46924], 60.00th=[46924], | 70.00th=[50594], 80.00th=[50594], 90.00th=[54789], 95.00th=[54789], | 99.00th=[54789], 99.50th=[54789], 99.90th=[54789], 99.95th=[54789], | 99.99th=[54789] bw ( KiB/s): min= 2043, max= 2043, per=17.87%, avg=2043.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=3, BW=4031KiB/s (4128kB/s)(7168KiB/1778msec) clat (msec): min=28, max=1263, avg=227.70, stdev=457.05 lat (msec): min=28, max=1263, avg=227.80, stdev=457.06 clat percentiles (msec): | 1.00th=[ 29], 5.00th=[ 29], 10.00th=[ 29], 20.00th=[ 45], | 30.00th=[ 45], 40.00th=[ 45], 50.00th=[ 56], 60.00th=[ 70], | 70.00th=[ 70], 80.00th=[ 89], 90.00th=[ 1267], 95.00th=[ 1267], | 99.00th=[ 1267], 99.50th=[ 1267], 99.90th=[ 1267], 99.95th=[ 1267], | 99.99th=[ 1267] bw ( KiB/s): min= 2048, max= 6131, per=25.55%, avg=4089.50, stdev=2887.12, samples=2 iops : min= 2, max= 5, avg= 3.50, stdev= 2.12, samples=2 lat (usec) : 750=16.67% lat (msec) : 50=33.33%, 100=41.67% cpu : usr=0.00%, sys=13.11%, ctx=465, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=11.2MiB/s (11.7MB/s), 1290KiB/s-4002KiB/s (1321kB/s-4098kB/s), io=20.0MiB (20.0MB), run=1587-1791msec WRITE: bw=15.6MiB/s (16.4MB/s), 2859KiB/s-6452KiB/s (2927kB/s-6607kB/s), io=28.0MiB (29.4MB), run=1587-1791msec rand-rw: (groupid=0, jobs=1): err= 0: pid=22350: Tue Jun 10 22:09:05 2025 read: IOPS=3, BW=3561KiB/s (3646kB/s)(7168KiB/2013msec) clat (msec): min=23, max=1458, avg=248.72, stdev=534.44 lat (msec): min=23, max=1458, avg=248.72, stdev=534.44 clat percentiles (msec): | 1.00th=[ 24], 5.00th=[ 24], 10.00th=[ 24], 20.00th=[ 27], | 30.00th=[ 28], 40.00th=[ 28], 50.00th=[ 33], 60.00th=[ 50], | 70.00th=[ 50], 80.00th=[ 124], 90.00th=[ 1452], 95.00th=[ 1452], | 99.00th=[ 1452], 99.50th=[ 1452], 99.90th=[ 1452], 99.95th=[ 1452], | 99.99th=[ 1452] bw ( KiB/s): min= 2027, max= 8094, per=41.75%, avg=4056.33, stdev=3496.74, samples=3 iops : min= 1, max= 7, avg= 3.33, stdev= 3.21, samples=3 write: IOPS=2, BW=2543KiB/s (2605kB/s)(5120KiB/2013msec) clat (msec): min=26, max=110, avg=52.34, stdev=33.68 lat (msec): min=26, max=110, avg=52.44, stdev=33.68 clat percentiles (msec): | 1.00th=[ 27], 5.00th=[ 27], 10.00th=[ 27], 20.00th=[ 27], | 30.00th=[ 36], 40.00th=[ 36], 50.00th=[ 39], 60.00th=[ 39], | 70.00th=[ 52], 80.00th=[ 52], 90.00th=[ 111], 95.00th=[ 111], | 99.00th=[ 111], 99.50th=[ 111], 99.90th=[ 111], 99.95th=[ 111], | 99.99th=[ 111] bw ( KiB/s): min=10118, max=10118, per=74.39%, avg=10118.00, stdev= 0.00, samples=1 iops : min= 9, max= 9, avg= 9.00, stdev= 0.00, samples=1 lat (msec) : 50=66.67%, 100=8.33%, 250=16.67% cpu : usr=0.00%, sys=6.71%, ctx=58, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22351: Tue Jun 10 22:09:05 2025 read: IOPS=1, BW=1057KiB/s (1083kB/s)(2048KiB/1937msec) clat (msec): min=89, max=1143, avg=616.70, stdev=745.33 lat (msec): min=89, max=1143, avg=616.71, stdev=745.34 clat percentiles (msec): | 1.00th=[ 90], 5.00th=[ 90], 10.00th=[ 90], 20.00th=[ 90], | 30.00th=[ 90], 40.00th=[ 90], 50.00th=[ 90], 60.00th=[ 1150], | 70.00th=[ 1150], 80.00th=[ 1150], 90.00th=[ 1150], 95.00th=[ 1150], | 99.00th=[ 1150], 99.50th=[ 1150], 99.90th=[ 1150], 99.95th=[ 1150], | 99.99th=[ 1150] bw ( KiB/s): min= 2048, max= 2048, per=21.08%, avg=2048.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=5, BW=5287KiB/s (5413kB/s)(10.0MiB/1937msec) clat (msec): min=32, max=104, avg=70.11, stdev=22.51 lat (msec): min=32, max=104, avg=70.22, stdev=22.53 clat percentiles (msec): | 1.00th=[ 33], 5.00th=[ 33], 10.00th=[ 33], 20.00th=[ 50], | 30.00th=[ 53], 40.00th=[ 58], 50.00th=[ 67], 60.00th=[ 75], | 70.00th=[ 88], 80.00th=[ 88], 90.00th=[ 95], 95.00th=[ 105], | 99.00th=[ 105], 99.50th=[ 105], 99.90th=[ 105], 99.95th=[ 105], | 99.99th=[ 105] bw ( KiB/s): min=10240, max=10240, per=75.29%, avg=10240.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 lat (msec) : 50=16.67%, 100=66.67%, 250=8.33% cpu : usr=0.15%, sys=9.30%, ctx=97, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22352: Tue Jun 10 22:09:05 2025 read: IOPS=2, BW=3001KiB/s (3074kB/s)(6144KiB/2047msec) clat (usec): min=21148, max=90170, avg=56971.85, stdev=28834.31 lat (usec): min=21150, max=90172, avg=56973.62, stdev=28834.51 clat percentiles (usec): | 1.00th=[21103], 5.00th=[21103], 10.00th=[21103], 20.00th=[31327], | 30.00th=[31327], 40.00th=[44303], 50.00th=[44303], 60.00th=[69731], | 70.00th=[85459], 80.00th=[85459], 90.00th=[89654], 95.00th=[89654], | 99.00th=[89654], 99.50th=[89654], 99.90th=[89654], 99.95th=[89654], | 99.99th=[89654] bw ( KiB/s): min=10240, max=10240, per=100.00%, avg=10240.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 write: IOPS=2, BW=3001KiB/s (3074kB/s)(6144KiB/2047msec) clat (msec): min=37, max=1315, avg=280.16, stdev=508.48 lat (msec): min=37, max=1315, avg=280.26, stdev=508.48 clat percentiles (msec): | 1.00th=[ 37], 5.00th=[ 37], 10.00th=[ 37], 20.00th=[ 40], | 30.00th=[ 40], 40.00th=[ 58], 50.00th=[ 58], 60.00th=[ 101], | 70.00th=[ 133], 80.00th=[ 133], 90.00th=[ 1318], 95.00th=[ 1318], | 99.00th=[ 1318], 99.50th=[ 1318], 99.90th=[ 1318], 99.95th=[ 1318], | 99.99th=[ 1318] bw ( KiB/s): min= 4096, max= 6144, per=37.64%, avg=5120.00, stdev=1448.15, samples=2 iops : min= 4, max= 6, avg= 5.00, stdev= 1.41, samples=2 lat (msec) : 50=41.67%, 100=33.33%, 250=16.67% cpu : usr=0.10%, sys=4.30%, ctx=44, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22353: Tue Jun 10 22:09:05 2025 read: IOPS=2, BW=2429KiB/s (2487kB/s)(5120KiB/2108msec) clat (msec): min=23, max=100, avg=47.99, stdev=32.48 lat (msec): min=23, max=100, avg=47.99, stdev=32.49 clat percentiles (msec): | 1.00th=[ 24], 5.00th=[ 24], 10.00th=[ 24], 20.00th=[ 24], | 30.00th=[ 25], 40.00th=[ 25], 50.00th=[ 32], 60.00th=[ 32], | 70.00th=[ 59], 80.00th=[ 59], 90.00th=[ 101], 95.00th=[ 101], | 99.00th=[ 101], 99.50th=[ 101], 99.90th=[ 101], 99.95th=[ 101], | 99.99th=[ 101] bw ( KiB/s): min= 8192, max= 8192, per=84.32%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=3, BW=3400KiB/s (3482kB/s)(7168KiB/2108msec) clat (msec): min=23, max=1401, avg=265.36, stdev=502.95 lat (msec): min=23, max=1401, avg=265.50, stdev=502.93 clat percentiles (msec): | 1.00th=[ 24], 5.00th=[ 24], 10.00th=[ 24], 20.00th=[ 36], | 30.00th=[ 37], 40.00th=[ 37], 50.00th=[ 99], 60.00th=[ 132], | 70.00th=[ 132], 80.00th=[ 132], 90.00th=[ 1401], 95.00th=[ 1401], | 99.00th=[ 1401], 99.50th=[ 1401], 99.90th=[ 1401], 99.95th=[ 1401], | 99.99th=[ 1401] bw ( KiB/s): min= 2048, max= 6144, per=30.12%, avg=4096.00, stdev=2896.31, samples=2 iops : min= 2, max= 6, avg= 4.00, stdev= 2.83, samples=2 lat (msec) : 50=50.00%, 100=16.67%, 250=25.00% cpu : usr=0.05%, sys=4.98%, ctx=47, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=9715KiB/s (9949kB/s), 1057KiB/s-3561KiB/s (1083kB/s-3646kB/s), io=20.0MiB (20.0MB), run=1937-2108msec WRITE: bw=13.3MiB/s (13.9MB/s), 2543KiB/s-5287KiB/s (2605kB/s-5413kB/s), io=28.0MiB (29.4MB), run=1937-2108msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=22360: Tue Jun 10 22:09:07 2025 read: IOPS=3, BW=15.3MiB/s (16.1MB/s)(12.0MiB/783msec) clat (msec): min=90, max=540, avg=245.71, stdev=255.41 lat (msec): min=90, max=540, avg=245.71, stdev=255.41 clat percentiles (msec): | 1.00th=[ 91], 5.00th=[ 91], 10.00th=[ 91], 20.00th=[ 91], | 30.00th=[ 91], 40.00th=[ 106], 50.00th=[ 106], 60.00th=[ 106], | 70.00th=[ 542], 80.00th=[ 542], 90.00th=[ 542], 95.00th=[ 542], | 99.00th=[ 542], 99.50th=[ 542], 99.90th=[ 542], 99.95th=[ 542], | 99.99th=[ 542] bw ( KiB/s): min=16157, max=16157, per=87.08%, avg=16157.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 250=33.33%, 750=33.33% cpu : usr=0.00%, sys=9.97%, ctx=54, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22361: Tue Jun 10 22:09:07 2025 read: IOPS=1, BW=4639KiB/s (4750kB/s)(4096KiB/883msec) clat (nsec): min=143555k, max=143555k, avg=143555374.00, stdev= 0.00 lat (nsec): min=143558k, max=143558k, avg=143558389.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 144], 5.00th=[ 144], 10.00th=[ 144], 20.00th=[ 144], | 30.00th=[ 144], 40.00th=[ 144], 50.00th=[ 144], 60.00th=[ 144], | 70.00th=[ 144], 80.00th=[ 144], 90.00th=[ 144], 95.00th=[ 144], | 99.00th=[ 144], 99.50th=[ 144], 99.90th=[ 144], 99.95th=[ 144], | 99.99th=[ 144] bw ( KiB/s): min= 7380, max= 7380, per=39.78%, avg=7380.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=2, BW=9277KiB/s (9500kB/s)(8192KiB/883msec) clat (msec): min=286, max=405, avg=346.35, stdev=84.19 lat (msec): min=287, max=406, avg=346.74, stdev=84.23 clat percentiles (msec): | 1.00th=[ 288], 5.00th=[ 288], 10.00th=[ 288], 20.00th=[ 288], | 30.00th=[ 288], 40.00th=[ 288], 50.00th=[ 288], 60.00th=[ 405], | 70.00th=[ 405], 80.00th=[ 405], 90.00th=[ 405], 95.00th=[ 405], | 99.00th=[ 405], 99.50th=[ 405], 99.90th=[ 405], 99.95th=[ 405], | 99.99th=[ 405] lat (msec) : 250=33.33%, 500=66.67% cpu : usr=0.11%, sys=17.80%, ctx=99, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22362: Tue Jun 10 22:09:07 2025 write: IOPS=9, BW=38.5MiB/s (40.3MB/s)(12.0MiB/312msec) clat (msec): min=65, max=162, avg=99.49, stdev=54.68 lat (msec): min=66, max=162, avg=99.94, stdev=54.65 clat percentiles (msec): | 1.00th=[ 66], 5.00th=[ 66], 10.00th=[ 66], 20.00th=[ 66], | 30.00th=[ 66], 40.00th=[ 70], 50.00th=[ 70], 60.00th=[ 70], | 70.00th=[ 163], 80.00th=[ 163], 90.00th=[ 163], 95.00th=[ 163], | 99.00th=[ 163], 99.50th=[ 163], 99.90th=[ 163], 99.95th=[ 163], | 99.99th=[ 163] lat (msec) : 100=66.67%, 250=33.33% cpu : usr=0.32%, sys=22.83%, ctx=38, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22363: Tue Jun 10 22:09:07 2025 write: IOPS=4, BW=19.1MiB/s (20.0MB/s)(12.0MiB/629msec) clat (msec): min=92, max=309, avg=191.36, stdev=109.71 lat (msec): min=92, max=309, avg=191.68, stdev=109.63 clat percentiles (msec): | 1.00th=[ 92], 5.00th=[ 92], 10.00th=[ 92], 20.00th=[ 92], | 30.00th=[ 92], 40.00th=[ 174], 50.00th=[ 174], 60.00th=[ 174], | 70.00th=[ 309], 80.00th=[ 309], 90.00th=[ 309], 95.00th=[ 309], | 99.00th=[ 309], 99.50th=[ 309], 99.90th=[ 309], 99.95th=[ 309], | 99.99th=[ 309] bw ( KiB/s): min=16384, max=16384, per=44.15%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 250=33.33%, 500=33.33% cpu : usr=0.00%, sys=20.70%, ctx=56, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=18.1MiB/s (19.0MB/s), 4639KiB/s-15.3MiB/s (4750kB/s-16.1MB/s), io=16.0MiB (16.8MB), run=783-883msec WRITE: bw=36.2MiB/s (38.0MB/s), 9277KiB/s-38.5MiB/s (9500kB/s-40.3MB/s), io=32.0MiB (33.6MB), run=312-883msec rand-rw: (groupid=0, jobs=1): err= 0: pid=22365: Tue Jun 10 22:09:07 2025 read: IOPS=4, BW=18.9MiB/s (19.8MB/s)(12.0MiB/635msec) clat (msec): min=156, max=258, avg=197.11, stdev=54.29 lat (msec): min=156, max=258, avg=197.11, stdev=54.29 clat percentiles (msec): | 1.00th=[ 157], 5.00th=[ 157], 10.00th=[ 157], 20.00th=[ 157], | 30.00th=[ 157], 40.00th=[ 178], 50.00th=[ 178], 60.00th=[ 178], | 70.00th=[ 259], 80.00th=[ 259], 90.00th=[ 259], 95.00th=[ 259], | 99.00th=[ 259], 99.50th=[ 259], 99.90th=[ 259], 99.95th=[ 259], | 99.99th=[ 259] bw ( KiB/s): min=16384, max=16384, per=100.00%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.16%, sys=27.76%, ctx=1933, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22366: Tue Jun 10 22:09:07 2025 read: IOPS=0, BW=4035KiB/s (4132kB/s)(4096KiB/1015msec) clat (nsec): min=175585k, max=175585k, avg=175585143.00, stdev= 0.00 lat (nsec): min=175588k, max=175588k, avg=175588247.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 176], 5.00th=[ 176], 10.00th=[ 176], 20.00th=[ 176], | 30.00th=[ 176], 40.00th=[ 176], 50.00th=[ 176], 60.00th=[ 176], | 70.00th=[ 176], 80.00th=[ 176], 90.00th=[ 176], 95.00th=[ 176], | 99.00th=[ 176], 99.50th=[ 176], 99.90th=[ 176], 99.95th=[ 176], | 99.99th=[ 176] bw ( KiB/s): min= 8175, max= 8175, per=50.65%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=1, BW=8071KiB/s (8265kB/s)(8192KiB/1015msec) clat (msec): min=141, max=626, avg=384.10, stdev=343.40 lat (msec): min=141, max=627, avg=384.46, stdev=343.39 clat percentiles (msec): | 1.00th=[ 142], 5.00th=[ 142], 10.00th=[ 142], 20.00th=[ 142], | 30.00th=[ 142], 40.00th=[ 142], 50.00th=[ 142], 60.00th=[ 625], | 70.00th=[ 625], 80.00th=[ 625], 90.00th=[ 625], 95.00th=[ 625], | 99.00th=[ 625], 99.50th=[ 625], 99.90th=[ 625], 99.95th=[ 625], | 99.99th=[ 625] lat (msec) : 250=66.67%, 750=33.33% cpu : usr=0.00%, sys=32.74%, ctx=631, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22367: Tue Jun 10 22:09:07 2025 write: IOPS=3, BW=12.1MiB/s (12.7MB/s)(12.0MiB/993msec) clat (msec): min=162, max=542, avg=323.90, stdev=196.26 lat (msec): min=162, max=542, avg=324.27, stdev=196.27 clat percentiles (msec): | 1.00th=[ 163], 5.00th=[ 163], 10.00th=[ 163], 20.00th=[ 163], | 30.00th=[ 163], 40.00th=[ 268], 50.00th=[ 268], 60.00th=[ 268], | 70.00th=[ 542], 80.00th=[ 542], 90.00th=[ 542], 95.00th=[ 542], | 99.00th=[ 542], 99.50th=[ 542], 99.90th=[ 542], 99.95th=[ 542], | 99.99th=[ 542] bw ( KiB/s): min=16384, max=16384, per=50.75%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=33.33%, 750=33.33% cpu : usr=0.00%, sys=30.44%, ctx=78, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22368: Tue Jun 10 22:09:07 2025 write: IOPS=3, BW=15.4MiB/s (16.2MB/s)(12.0MiB/777msec) clat (msec): min=164, max=275, avg=233.11, stdev=60.32 lat (msec): min=164, max=276, avg=233.46, stdev=60.35 clat percentiles (msec): | 1.00th=[ 165], 5.00th=[ 165], 10.00th=[ 165], 20.00th=[ 165], | 30.00th=[ 165], 40.00th=[ 259], 50.00th=[ 259], 60.00th=[ 259], | 70.00th=[ 275], 80.00th=[ 275], 90.00th=[ 275], 95.00th=[ 275], | 99.00th=[ 275], 99.50th=[ 275], 99.90th=[ 275], 99.95th=[ 275], | 99.99th=[ 275] bw ( KiB/s): min= 8192, max= 8192, per=25.38%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=66.67% cpu : usr=0.26%, sys=42.01%, ctx=50, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=15.8MiB/s (16.5MB/s), 4035KiB/s-18.9MiB/s (4132kB/s-19.8MB/s), io=16.0MiB (16.8MB), run=635-1015msec WRITE: bw=31.5MiB/s (33.1MB/s), 8071KiB/s-15.4MiB/s (8265kB/s-16.2MB/s), io=32.0MiB (33.6MB), run=777-1015msec