== sanity test 398b: DIO and buffer IO race ============== 02:59:09 (1746169149) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.30049 s, 38.7 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=32661: Fri May 2 02:59:25 2025 read: IOPS=111, BW=448KiB/s (458kB/s)(5992KiB/13383msec) clat (usec): min=57, max=23769, avg=4846.46, stdev=2688.17 lat (usec): min=58, max=23770, avg=4847.33, stdev=2688.17 clat percentiles (usec): | 1.00th=[ 79], 5.00th=[ 285], 10.00th=[ 2376], 20.00th=[ 2900], | 30.00th=[ 3294], 40.00th=[ 3752], 50.00th=[ 4359], 60.00th=[ 5080], | 70.00th=[ 5866], 80.00th=[ 6587], 90.00th=[ 7898], 95.00th=[ 9241], | 99.00th=[14615], 99.50th=[16057], 99.90th=[22676], 99.95th=[23725], | 99.99th=[23725] bw ( KiB/s): min= 256, max= 536, per=24.91%, avg=442.46, stdev=64.98, samples=26 iops : min= 64, max= 134, avg=110.62, stdev=16.25, samples=26 write: IOPS=117, BW=470KiB/s (482kB/s)(6296KiB/13383msec) clat (usec): min=440, max=182287, avg=3842.98, stdev=5526.91 lat (usec): min=440, max=182288, avg=3843.96, stdev=5527.28 clat percentiles (usec): | 1.00th=[ 529], 5.00th=[ 594], 10.00th=[ 652], 20.00th=[ 791], | 30.00th=[ 947], 40.00th=[ 2311], 50.00th=[ 3130], 60.00th=[ 3785], | 70.00th=[ 4752], 80.00th=[ 6194], 90.00th=[ 8225], 95.00th=[ 10159], | 99.00th=[ 13566], 99.50th=[ 14746], 99.90th=[ 19006], 99.95th=[181404], | 99.99th=[181404] bw ( KiB/s): min= 304, max= 648, per=24.94%, avg=463.69, stdev=66.91, samples=26 iops : min= 76, max= 162, avg=115.92, stdev=16.73, samples=26 lat (usec) : 100=1.07%, 250=1.33%, 500=0.16%, 750=9.15%, 1000=6.74% lat (msec) : 2=3.68%, 4=32.03%, 10=41.31%, 20=4.43%, 50=0.07% lat (msec) : 250=0.03% cpu : usr=0.18%, sys=23.09%, ctx=5672, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32662: Fri May 2 02:59:25 2025 read: IOPS=111, BW=445KiB/s (455kB/s)(5940KiB/13361msec) clat (usec): min=59, max=19791, avg=4867.76, stdev=2649.51 lat (usec): min=59, max=19792, avg=4868.93, stdev=2649.34 clat percentiles (usec): | 1.00th=[ 76], 5.00th=[ 2024], 10.00th=[ 2409], 20.00th=[ 2835], | 30.00th=[ 3261], 40.00th=[ 3785], 50.00th=[ 4359], 60.00th=[ 5145], | 70.00th=[ 5866], 80.00th=[ 6587], 90.00th=[ 8094], 95.00th=[ 9372], | 99.00th=[15008], 99.50th=[16450], 99.90th=[19530], 99.95th=[19792], | 99.99th=[19792] bw ( KiB/s): min= 224, max= 528, per=24.84%, avg=441.23, stdev=67.53, samples=26 iops : min= 56, max= 132, avg=110.31, stdev=16.88, samples=26 write: IOPS=118, BW=475KiB/s (487kB/s)(6348KiB/13361msec) clat (usec): min=454, max=186110, avg=3819.16, stdev=5678.15 lat (usec): min=455, max=186111, avg=3819.96, stdev=5678.16 clat percentiles (usec): | 1.00th=[ 529], 5.00th=[ 594], 10.00th=[ 644], 20.00th=[ 766], | 30.00th=[ 979], 40.00th=[ 2040], 50.00th=[ 2999], 60.00th=[ 3687], | 70.00th=[ 4621], 80.00th=[ 6128], 90.00th=[ 8160], 95.00th=[ 10159], | 99.00th=[ 14746], 99.50th=[ 16450], 99.90th=[ 24773], 99.95th=[185598], | 99.99th=[185598] bw ( KiB/s): min= 288, max= 584, per=25.06%, avg=465.85, stdev=70.34, samples=26 iops : min= 72, max= 146, avg=116.46, stdev=17.58, samples=26 lat (usec) : 100=1.24%, 250=0.81%, 500=0.20%, 750=9.70%, 1000=6.15% lat (msec) : 2=4.72%, 4=31.67%, 10=40.89%, 20=4.43%, 50=0.16% lat (msec) : 250=0.03% cpu : usr=0.15%, sys=23.23%, ctx=5761, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32663: Fri May 2 02:59:25 2025 read: IOPS=111, BW=447KiB/s (458kB/s)(6028KiB/13485msec) clat (usec): min=58, max=19751, avg=4845.66, stdev=2594.00 lat (usec): min=58, max=19752, avg=4847.23, stdev=2593.98 clat percentiles (usec): | 1.00th=[ 79], 5.00th=[ 2008], 10.00th=[ 2409], 20.00th=[ 2933], | 30.00th=[ 3261], 40.00th=[ 3785], 50.00th=[ 4424], 60.00th=[ 5080], | 70.00th=[ 5800], 80.00th=[ 6652], 90.00th=[ 7963], 95.00th=[ 9241], | 99.00th=[14091], 99.50th=[16188], 99.90th=[17957], 99.95th=[19792], | 99.99th=[19792] bw ( KiB/s): min= 200, max= 552, per=24.95%, avg=443.08, stdev=71.63, samples=26 iops : min= 50, max= 138, avg=110.77, stdev=17.91, samples=26 write: IOPS=116, BW=464KiB/s (475kB/s)(6260KiB/13485msec) clat (usec): min=480, max=185729, avg=3894.86, stdev=5713.35 lat (usec): min=481, max=185731, avg=3896.27, stdev=5713.08 clat percentiles (usec): | 1.00th=[ 529], 5.00th=[ 586], 10.00th=[ 644], 20.00th=[ 766], | 30.00th=[ 914], 40.00th=[ 2245], 50.00th=[ 3097], 60.00th=[ 3785], | 70.00th=[ 4817], 80.00th=[ 6325], 90.00th=[ 8586], 95.00th=[ 10421], | 99.00th=[ 14091], 99.50th=[ 16450], 99.90th=[ 32637], 99.95th=[185598], | 99.99th=[185598] bw ( KiB/s): min= 288, max= 640, per=24.66%, avg=458.46, stdev=80.25, samples=26 iops : min= 72, max= 160, avg=114.62, stdev=20.06, samples=26 lat (usec) : 100=1.30%, 250=0.85%, 500=0.29%, 750=9.41%, 1000=6.48% lat (msec) : 2=3.81%, 4=31.84%, 10=41.11%, 20=4.79%, 50=0.10% lat (msec) : 250=0.03% cpu : usr=0.26%, sys=22.61%, ctx=5935, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32665: Fri May 2 02:59:25 2025 read: IOPS=111, BW=448KiB/s (458kB/s)(6052KiB/13517msec) clat (usec): min=54, max=29705, avg=4914.31, stdev=2820.39 lat (usec): min=55, max=29706, avg=4915.01, stdev=2820.40 clat percentiles (usec): | 1.00th=[ 77], 5.00th=[ 1876], 10.00th=[ 2442], 20.00th=[ 2868], | 30.00th=[ 3326], 40.00th=[ 3785], 50.00th=[ 4424], 60.00th=[ 5080], | 70.00th=[ 5932], 80.00th=[ 6718], 90.00th=[ 7898], 95.00th=[ 9372], | 99.00th=[14877], 99.50th=[16909], 99.90th=[24773], 99.95th=[29754], | 99.99th=[29754] bw ( KiB/s): min= 224, max= 656, per=25.18%, avg=447.11, stdev=75.65, samples=27 iops : min= 56, max= 164, avg=111.78, stdev=18.91, samples=27 write: IOPS=115, BW=461KiB/s (472kB/s)(6236KiB/13517msec) clat (usec): min=481, max=194544, avg=3856.34, stdev=5910.51 lat (usec): min=482, max=194546, avg=3857.09, stdev=5910.53 clat percentiles (usec): | 1.00th=[ 529], 5.00th=[ 586], 10.00th=[ 644], 20.00th=[ 775], | 30.00th=[ 930], 40.00th=[ 1860], 50.00th=[ 2966], 60.00th=[ 3720], | 70.00th=[ 4752], 80.00th=[ 6390], 90.00th=[ 8455], 95.00th=[ 10290], | 99.00th=[ 14222], 99.50th=[ 17695], 99.90th=[ 28967], 99.95th=[193987], | 99.99th=[193987] bw ( KiB/s): min= 256, max= 664, per=24.75%, avg=460.15, stdev=87.45, samples=27 iops : min= 64, max= 166, avg=115.04, stdev=21.86, samples=27 lat (usec) : 100=1.07%, 250=1.11%, 500=0.20%, 750=9.11%, 1000=7.19% lat (msec) : 2=4.39%, 4=30.66%, 10=41.54%, 20=4.43%, 50=0.26% lat (msec) : 250=0.03% cpu : usr=0.19%, sys=22.58%, ctx=5905, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1776KiB/s (1819kB/s), 445KiB/s-448KiB/s (455kB/s-458kB/s), io=23.4MiB (24.6MB), run=13361-13517msec WRITE: bw=1860KiB/s (1905kB/s), 461KiB/s-475KiB/s (472kB/s-487kB/s), io=24.6MiB (25.7MB), run=13361-13517msec rand-rw: (groupid=0, jobs=1): err= 0: pid=32664: Fri May 2 02:59:29 2025 read: IOPS=87, BW=350KiB/s (359kB/s)(5992KiB/17106msec) clat (usec): min=1693, max=202750, avg=4608.47, stdev=6155.17 lat (usec): min=1693, max=202751, avg=4609.31, stdev=6155.12 clat percentiles (usec): | 1.00th=[ 1811], 5.00th=[ 1942], 10.00th=[ 2040], 20.00th=[ 2180], | 30.00th=[ 2343], 40.00th=[ 2606], 50.00th=[ 2933], 60.00th=[ 3490], | 70.00th=[ 4555], 80.00th=[ 6456], 90.00th=[ 9896], 95.00th=[ 12256], | 99.00th=[ 16319], 99.50th=[ 19006], 99.90th=[ 23462], 99.95th=[202376], | 99.99th=[202376] bw ( KiB/s): min= 128, max= 728, per=24.89%, avg=348.68, stdev=176.19, samples=34 iops : min= 32, max= 182, avg=87.15, stdev=44.06, samples=34 write: IOPS=92, BW=368KiB/s (377kB/s)(6296KiB/17106msec) clat (usec): min=2298, max=34748, avg=6435.53, stdev=4104.81 lat (usec): min=2299, max=34749, avg=6436.34, stdev=4104.82 clat percentiles (usec): | 1.00th=[ 2638], 5.00th=[ 2966], 10.00th=[ 3163], 20.00th=[ 3392], | 30.00th=[ 3654], 40.00th=[ 3982], 50.00th=[ 4555], 60.00th=[ 5866], | 70.00th=[ 7701], 80.00th=[ 9503], 90.00th=[11994], 95.00th=[13960], | 99.00th=[20579], 99.50th=[24773], 99.90th=[34341], 99.95th=[34866], | 99.99th=[34866] bw ( KiB/s): min= 136, max= 816, per=24.91%, avg=365.38, stdev=179.90, samples=34 iops : min= 34, max= 204, avg=91.32, stdev=44.98, samples=34 lat (msec) : 2=3.91%, 4=49.06%, 10=33.20%, 20=13.02%, 50=0.78% lat (msec) : 250=0.03% cpu : usr=0.13%, sys=11.35%, ctx=3584, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32666: Fri May 2 02:59:29 2025 read: IOPS=86, BW=347KiB/s (355kB/s)(5940KiB/17136msec) clat (usec): min=1616, max=189914, avg=4607.82, stdev=5989.59 lat (usec): min=1616, max=189916, avg=4608.56, stdev=5989.63 clat percentiles (usec): | 1.00th=[ 1811], 5.00th=[ 1942], 10.00th=[ 2040], 20.00th=[ 2180], | 30.00th=[ 2311], 40.00th=[ 2573], 50.00th=[ 2900], 60.00th=[ 3425], | 70.00th=[ 4359], 80.00th=[ 6325], 90.00th=[ 9896], 95.00th=[ 12518], | 99.00th=[ 17695], 99.50th=[ 19006], 99.90th=[ 26870], 99.95th=[189793], | 99.99th=[189793] bw ( KiB/s): min= 104, max= 760, per=24.60%, avg=344.68, stdev=184.72, samples=34 iops : min= 26, max= 190, avg=86.15, stdev=46.19, samples=34 write: IOPS=92, BW=370KiB/s (379kB/s)(6348KiB/17136msec) clat (usec): min=2143, max=34555, avg=6445.11, stdev=4212.26 lat (usec): min=2143, max=34556, avg=6446.02, stdev=4212.26 clat percentiles (usec): | 1.00th=[ 2638], 5.00th=[ 2900], 10.00th=[ 3097], 20.00th=[ 3326], | 30.00th=[ 3556], 40.00th=[ 3916], 50.00th=[ 4621], 60.00th=[ 5866], | 70.00th=[ 7635], 80.00th=[ 9503], 90.00th=[11863], 95.00th=[13960], | 99.00th=[22938], 99.50th=[26870], 99.90th=[32375], 99.95th=[34341], | 99.99th=[34341] bw ( KiB/s): min= 160, max= 776, per=25.00%, avg=366.79, stdev=178.68, samples=34 iops : min= 40, max= 194, avg=91.68, stdev=44.68, samples=34 lat (msec) : 2=4.07%, 4=49.48%, 10=32.71%, 20=12.76%, 50=0.94% lat (msec) : 250=0.03% cpu : usr=0.22%, sys=11.40%, ctx=3607, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32667: Fri May 2 02:59:29 2025 read: IOPS=88, BW=353KiB/s (362kB/s)(6028KiB/17060msec) clat (usec): min=1617, max=26902, avg=4568.75, stdev=3608.24 lat (usec): min=1617, max=26903, avg=4569.54, stdev=3608.20 clat percentiles (usec): | 1.00th=[ 1762], 5.00th=[ 1926], 10.00th=[ 2008], 20.00th=[ 2180], | 30.00th=[ 2343], 40.00th=[ 2638], 50.00th=[ 2999], 60.00th=[ 3556], | 70.00th=[ 4490], 80.00th=[ 6587], 90.00th=[10159], 95.00th=[12256], | 99.00th=[17695], 99.50th=[19268], 99.90th=[25822], 99.95th=[26870], | 99.99th=[26870] bw ( KiB/s): min= 120, max= 776, per=25.17%, avg=352.68, stdev=176.09, samples=34 iops : min= 30, max= 194, avg=88.15, stdev=44.02, samples=34 write: IOPS=91, BW=367KiB/s (376kB/s)(6260KiB/17060msec) clat (msec): min=2, max=184, avg= 6.45, stdev= 6.02 lat (msec): min=2, max=184, avg= 6.45, stdev= 6.02 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 4], 20.00th=[ 4], | 30.00th=[ 4], 40.00th=[ 4], 50.00th=[ 5], 60.00th=[ 6], | 70.00th=[ 8], 80.00th=[ 10], 90.00th=[ 12], 95.00th=[ 15], | 99.00th=[ 22], 99.50th=[ 24], 99.90th=[ 33], 99.95th=[ 184], | 99.99th=[ 184] bw ( KiB/s): min= 160, max= 808, per=24.91%, avg=365.38, stdev=182.14, samples=34 iops : min= 40, max= 202, avg=91.32, stdev=45.54, samples=34 lat (msec) : 2=4.46%, 4=48.54%, 10=33.66%, 20=12.47%, 50=0.85% lat (msec) : 250=0.03% cpu : usr=0.14%, sys=11.40%, ctx=3580, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32668: Fri May 2 02:59:29 2025 read: IOPS=88, BW=354KiB/s (363kB/s)(6052KiB/17076msec) clat (usec): min=1660, max=24639, avg=4468.62, stdev=3486.12 lat (usec): min=1661, max=24640, avg=4469.39, stdev=3486.15 clat percentiles (usec): | 1.00th=[ 1778], 5.00th=[ 1926], 10.00th=[ 2024], 20.00th=[ 2180], | 30.00th=[ 2376], 40.00th=[ 2606], 50.00th=[ 3032], 60.00th=[ 3523], | 70.00th=[ 4359], 80.00th=[ 6128], 90.00th=[ 9503], 95.00th=[12125], | 99.00th=[17695], 99.50th=[19530], 99.90th=[23987], 99.95th=[24511], | 99.99th=[24511] bw ( KiB/s): min= 151, max= 696, per=25.16%, avg=352.56, stdev=173.99, samples=34 iops : min= 37, max= 174, avg=88.03, stdev=43.57, samples=34 write: IOPS=91, BW=365KiB/s (374kB/s)(6236KiB/17076msec) clat (msec): min=2, max=192, avg= 6.58, stdev= 6.22 lat (msec): min=2, max=192, avg= 6.58, stdev= 6.22 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 4], 20.00th=[ 4], | 30.00th=[ 4], 40.00th=[ 5], 50.00th=[ 5], 60.00th=[ 6], | 70.00th=[ 8], 80.00th=[ 10], 90.00th=[ 12], 95.00th=[ 15], | 99.00th=[ 21], 99.50th=[ 24], 99.90th=[ 35], 99.95th=[ 192], | 99.99th=[ 192] bw ( KiB/s): min= 167, max= 752, per=24.78%, avg=363.59, stdev=175.99, samples=34 iops : min= 41, max= 188, avg=90.79, stdev=44.06, samples=34 lat (msec) : 2=4.13%, 4=48.73%, 10=33.72%, 20=12.57%, 50=0.81% lat (msec) : 250=0.03% cpu : usr=0.21%, sys=11.39%, ctx=3608, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1401KiB/s (1435kB/s), 347KiB/s-354KiB/s (355kB/s-363kB/s), io=23.4MiB (24.6MB), run=17060-17136msec WRITE: bw=1467KiB/s (1502kB/s), 365KiB/s-370KiB/s (374kB/s-379kB/s), io=24.6MiB (25.7MB), run=17060-17136msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=32693: Fri May 2 02:59:33 2025 read: IOPS=106, BW=1696KiB/s (1737kB/s)(5920KiB/3490msec) clat (usec): min=64, max=19060, avg=4936.42, stdev=3010.47 lat (usec): min=64, max=19061, avg=4937.15, stdev=3010.50 clat percentiles (usec): | 1.00th=[ 71], 5.00th=[ 108], 10.00th=[ 174], 20.00th=[ 2900], | 30.00th=[ 3490], 40.00th=[ 4146], 50.00th=[ 4621], 60.00th=[ 5145], | 70.00th=[ 6128], 80.00th=[ 6783], 90.00th=[ 8455], 95.00th=[10028], | 99.00th=[15139], 99.50th=[16319], 99.90th=[19006], 99.95th=[19006], | 99.99th=[19006] bw ( KiB/s): min= 1408, max= 1824, per=24.38%, avg=1615.33, stdev=196.16, samples=6 iops : min= 88, max= 114, avg=100.83, stdev=12.11, samples=6 write: IOPS=114, BW=1825KiB/s (1868kB/s)(6368KiB/3490msec) clat (usec): min=741, max=21441, avg=4126.70, stdev=3585.37 lat (usec): min=742, max=21442, avg=4127.72, stdev=3585.34 clat percentiles (usec): | 1.00th=[ 766], 5.00th=[ 881], 10.00th=[ 955], 20.00th=[ 1123], | 30.00th=[ 1319], 40.00th=[ 1991], 50.00th=[ 3130], 60.00th=[ 3982], | 70.00th=[ 5211], 80.00th=[ 6915], 90.00th=[ 9110], 95.00th=[10814], | 99.00th=[15795], 99.50th=[21103], 99.90th=[21365], 99.95th=[21365], | 99.99th=[21365] bw ( KiB/s): min= 1504, max= 1984, per=25.47%, avg=1775.33, stdev=165.44, samples=6 iops : min= 94, max= 124, avg=110.83, stdev=10.36, samples=6 lat (usec) : 100=1.95%, 250=3.12%, 500=0.13%, 750=0.26%, 1000=7.03% lat (msec) : 2=13.80%, 4=22.66%, 10=45.31%, 20=5.47%, 50=0.26% cpu : usr=0.23%, sys=25.59%, ctx=1596, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32694: Fri May 2 02:59:33 2025 read: IOPS=100, BW=1609KiB/s (1648kB/s)(5744KiB/3570msec) clat (usec): min=62, max=25164, avg=5217.45, stdev=3142.06 lat (usec): min=62, max=25165, avg=5219.25, stdev=3142.78 clat percentiles (usec): | 1.00th=[ 73], 5.00th=[ 109], 10.00th=[ 165], 20.00th=[ 3064], | 30.00th=[ 3785], 40.00th=[ 4293], 50.00th=[ 4948], 60.00th=[ 5604], | 70.00th=[ 6390], 80.00th=[ 7242], 90.00th=[ 8586], 95.00th=[10028], | 99.00th=[14484], 99.50th=[21365], 99.90th=[25035], 99.95th=[25035], | 99.99th=[25035] bw ( KiB/s): min= 1341, max= 1728, per=24.28%, avg=1608.71, stdev=131.28, samples=7 iops : min= 83, max= 108, avg=100.43, stdev= 8.48, samples=7 write: IOPS=114, BW=1833KiB/s (1877kB/s)(6544KiB/3570msec) clat (usec): min=723, max=19907, avg=4115.62, stdev=3258.74 lat (usec): min=724, max=19908, avg=4116.62, stdev=3258.72 clat percentiles (usec): | 1.00th=[ 799], 5.00th=[ 889], 10.00th=[ 1012], 20.00th=[ 1172], | 30.00th=[ 1516], 40.00th=[ 2704], 50.00th=[ 3392], 60.00th=[ 3982], | 70.00th=[ 4948], 80.00th=[ 6783], 90.00th=[ 8848], 95.00th=[10814], | 99.00th=[14746], 99.50th=[15139], 99.90th=[19792], 99.95th=[19792], | 99.99th=[19792] bw ( KiB/s): min= 1568, max= 2176, per=26.03%, avg=1814.14, stdev=226.60, samples=7 iops : min= 98, max= 136, avg=113.29, stdev=14.03, samples=7 lat (usec) : 100=2.21%, 250=2.60%, 500=0.13%, 750=0.39%, 1000=4.43% lat (msec) : 2=14.19%, 4=23.83%, 10=46.61%, 20=5.34%, 50=0.26% cpu : usr=0.22%, sys=26.08%, ctx=1652, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32695: Fri May 2 02:59:33 2025 read: IOPS=108, BW=1731KiB/s (1772kB/s)(6256KiB/3615msec) clat (usec): min=62, max=18347, avg=5141.01, stdev=3214.23 lat (usec): min=62, max=18348, avg=5141.70, stdev=3214.26 clat percentiles (usec): | 1.00th=[ 73], 5.00th=[ 111], 10.00th=[ 135], 20.00th=[ 3032], | 30.00th=[ 3458], 40.00th=[ 4015], 50.00th=[ 4817], 60.00th=[ 5604], | 70.00th=[ 6456], 80.00th=[ 7177], 90.00th=[ 8717], 95.00th=[10552], | 99.00th=[16581], 99.50th=[17433], 99.90th=[18220], 99.95th=[18220], | 99.99th=[18220] bw ( KiB/s): min= 1248, max= 2528, per=26.07%, avg=1727.43, stdev=415.23, samples=7 iops : min= 78, max= 158, avg=107.86, stdev=25.89, samples=7 write: IOPS=104, BW=1669KiB/s (1709kB/s)(6032KiB/3615msec) clat (usec): min=751, max=20975, avg=4176.59, stdev=3348.97 lat (usec): min=752, max=20976, avg=4178.53, stdev=3348.33 clat percentiles (usec): | 1.00th=[ 816], 5.00th=[ 881], 10.00th=[ 979], 20.00th=[ 1139], | 30.00th=[ 1418], 40.00th=[ 2442], 50.00th=[ 3359], 60.00th=[ 4146], | 70.00th=[ 5211], 80.00th=[ 6849], 90.00th=[ 9241], 95.00th=[10814], | 99.00th=[12911], 99.50th=[16319], 99.90th=[21103], 99.95th=[21103], | 99.99th=[21103] bw ( KiB/s): min= 1248, max= 1984, per=23.15%, avg=1613.29, stdev=231.70, samples=7 iops : min= 78, max= 124, avg=100.71, stdev=14.57, samples=7 lat (usec) : 100=1.82%, 250=4.04%, 1000=5.86% lat (msec) : 2=11.85%, 4=25.39%, 10=44.01%, 20=6.90%, 50=0.13% cpu : usr=0.14%, sys=25.35%, ctx=1692, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32697: Fri May 2 02:59:33 2025 read: IOPS=107, BW=1726KiB/s (1767kB/s)(6032KiB/3495msec) clat (usec): min=71, max=18229, avg=5115.97, stdev=3184.30 lat (usec): min=71, max=18229, avg=5116.86, stdev=3183.94 clat percentiles (usec): | 1.00th=[ 72], 5.00th=[ 89], 10.00th=[ 139], 20.00th=[ 3064], | 30.00th=[ 3589], 40.00th=[ 4047], 50.00th=[ 4555], 60.00th=[ 5342], | 70.00th=[ 6456], 80.00th=[ 7373], 90.00th=[ 8848], 95.00th=[11076], | 99.00th=[15795], 99.50th=[17695], 99.90th=[18220], 99.95th=[18220], | 99.99th=[18220] bw ( KiB/s): min= 1312, max= 1916, per=25.11%, avg=1663.33, stdev=208.38, samples=6 iops : min= 82, max= 119, avg=103.83, stdev=12.84, samples=6 write: IOPS=111, BW=1790KiB/s (1833kB/s)(6256KiB/3495msec) clat (usec): min=738, max=25212, avg=3941.41, stdev=3744.42 lat (usec): min=739, max=25214, avg=3942.43, stdev=3744.50 clat percentiles (usec): | 1.00th=[ 783], 5.00th=[ 873], 10.00th=[ 938], 20.00th=[ 1074], | 30.00th=[ 1237], 40.00th=[ 1713], 50.00th=[ 2835], 60.00th=[ 3523], | 70.00th=[ 4555], 80.00th=[ 6325], 90.00th=[ 9241], 95.00th=[11338], | 99.00th=[17695], 99.50th=[19792], 99.90th=[25297], 99.95th=[25297], | 99.99th=[25297] bw ( KiB/s): min= 1504, max= 1984, per=25.32%, avg=1764.67, stdev=200.68, samples=6 iops : min= 94, max= 124, avg=110.17, stdev=12.72, samples=6 lat (usec) : 100=2.73%, 250=2.86%, 500=0.13%, 750=0.52%, 1000=7.03% lat (msec) : 2=14.06%, 4=24.22%, 10=41.54%, 20=6.77%, 50=0.13% cpu : usr=0.26%, sys=25.33%, ctx=1613, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=6626KiB/s (6785kB/s), 1609KiB/s-1731KiB/s (1648kB/s-1772kB/s), io=23.4MiB (24.5MB), run=3490-3615msec WRITE: bw=6971KiB/s (7138kB/s), 1669KiB/s-1833KiB/s (1709kB/s-1877kB/s), io=24.6MiB (25.8MB), run=3490-3615msec rand-rw: (groupid=0, jobs=1): err= 0: pid=32698: Fri May 2 02:59:34 2025 read: IOPS=82, BW=1319KiB/s (1350kB/s)(5920KiB/4489msec) clat (usec): min=1786, max=19881, avg=4644.19, stdev=3529.86 lat (usec): min=1787, max=19882, avg=4644.89, stdev=3529.88 clat percentiles (usec): | 1.00th=[ 1893], 5.00th=[ 2057], 10.00th=[ 2147], 20.00th=[ 2311], | 30.00th=[ 2573], 40.00th=[ 2769], 50.00th=[ 3130], 60.00th=[ 3621], | 70.00th=[ 4490], 80.00th=[ 6325], 90.00th=[10159], 95.00th=[12911], | 99.00th=[17433], 99.50th=[19792], 99.90th=[19792], 99.95th=[19792], | 99.99th=[19792] bw ( KiB/s): min= 800, max= 2240, per=22.23%, avg=1168.00, stdev=471.23, samples=8 iops : min= 50, max= 140, avg=73.00, stdev=29.45, samples=8 write: IOPS=88, BW=1419KiB/s (1453kB/s)(6368KiB/4489msec) clat (usec): min=2295, max=30299, avg=6928.53, stdev=4541.45 lat (usec): min=2296, max=30301, avg=6929.74, stdev=4541.50 clat percentiles (usec): | 1.00th=[ 2704], 5.00th=[ 3097], 10.00th=[ 3359], 20.00th=[ 3621], | 30.00th=[ 3916], 40.00th=[ 4359], 50.00th=[ 4883], 60.00th=[ 5997], | 70.00th=[ 7701], 80.00th=[10159], 90.00th=[14091], 95.00th=[15926], | 99.00th=[24511], 99.50th=[29492], 99.90th=[30278], 99.95th=[30278], | 99.99th=[30278] bw ( KiB/s): min= 928, max= 2272, per=22.79%, avg=1260.00, stdev=423.82, samples=8 iops : min= 58, max= 142, avg=78.75, stdev=26.49, samples=8 lat (msec) : 2=1.43%, 4=46.74%, 10=36.20%, 20=14.84%, 50=0.78% cpu : usr=0.18%, sys=11.34%, ctx=934, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32699: Fri May 2 02:59:34 2025 read: IOPS=78, BW=1260KiB/s (1290kB/s)(5744KiB/4558msec) clat (usec): min=1839, max=22928, avg=4874.97, stdev=3975.92 lat (usec): min=1840, max=22929, avg=4875.99, stdev=3975.79 clat percentiles (usec): | 1.00th=[ 1926], 5.00th=[ 2057], 10.00th=[ 2114], 20.00th=[ 2278], | 30.00th=[ 2507], 40.00th=[ 2737], 50.00th=[ 3195], 60.00th=[ 3621], | 70.00th=[ 4555], 80.00th=[ 6980], 90.00th=[11338], 95.00th=[13960], | 99.00th=[20317], 99.50th=[22938], 99.90th=[22938], 99.95th=[22938], | 99.99th=[22938] bw ( KiB/s): min= 768, max= 2336, per=21.39%, avg=1123.75, stdev=505.95, samples=8 iops : min= 48, max= 146, avg=70.12, stdev=31.69, samples=8 write: IOPS=89, BW=1436KiB/s (1470kB/s)(6544KiB/4558msec) clat (usec): min=2514, max=32261, avg=6770.89, stdev=4392.48 lat (usec): min=2515, max=32263, avg=6772.17, stdev=4392.62 clat percentiles (usec): | 1.00th=[ 2704], 5.00th=[ 3032], 10.00th=[ 3294], 20.00th=[ 3556], | 30.00th=[ 3818], 40.00th=[ 4228], 50.00th=[ 4686], 60.00th=[ 5997], | 70.00th=[ 7963], 80.00th=[10028], 90.00th=[13042], 95.00th=[14877], | 99.00th=[20055], 99.50th=[29492], 99.90th=[32375], 99.95th=[32375], | 99.99th=[32375] bw ( KiB/s): min= 960, max= 2112, per=22.06%, avg=1219.62, stdev=369.93, samples=8 iops : min= 60, max= 132, avg=76.12, stdev=23.15, samples=8 lat (msec) : 2=1.17%, 4=47.53%, 10=34.64%, 20=15.49%, 50=1.17% cpu : usr=0.35%, sys=11.10%, ctx=898, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32700: Fri May 2 02:59:34 2025 read: IOPS=86, BW=1387KiB/s (1420kB/s)(6256KiB/4511msec) clat (usec): min=1783, max=29185, avg=5080.29, stdev=4232.51 lat (usec): min=1783, max=29186, avg=5080.99, stdev=4232.51 clat percentiles (usec): | 1.00th=[ 1893], 5.00th=[ 2040], 10.00th=[ 2114], 20.00th=[ 2311], | 30.00th=[ 2507], 40.00th=[ 2671], 50.00th=[ 3097], 60.00th=[ 3687], | 70.00th=[ 5145], 80.00th=[ 8029], 90.00th=[11207], 95.00th=[14746], | 99.00th=[18220], 99.50th=[23462], 99.90th=[29230], 99.95th=[29230], | 99.99th=[29230] bw ( KiB/s): min= 832, max= 2048, per=22.61%, avg=1188.00, stdev=374.92, samples=8 iops : min= 52, max= 128, avg=74.25, stdev=23.43, samples=8 write: IOPS=83, BW=1337KiB/s (1369kB/s)(6032KiB/4511msec) clat (usec): min=2588, max=22516, avg=6568.14, stdev=3875.36 lat (usec): min=2590, max=22518, avg=6569.53, stdev=3875.61 clat percentiles (usec): | 1.00th=[ 2769], 5.00th=[ 2999], 10.00th=[ 3294], 20.00th=[ 3556], | 30.00th=[ 3818], 40.00th=[ 4178], 50.00th=[ 4817], 60.00th=[ 5997], | 70.00th=[ 7504], 80.00th=[ 9503], 90.00th=[12649], 95.00th=[14615], | 99.00th=[18482], 99.50th=[21890], 99.90th=[22414], 99.95th=[22414], | 99.99th=[22414] bw ( KiB/s): min= 896, max= 2304, per=21.49%, avg=1188.00, stdev=461.65, samples=8 iops : min= 56, max= 144, avg=74.25, stdev=28.85, samples=8 lat (msec) : 2=2.21%, 4=47.53%, 10=33.33%, 20=16.28%, 50=0.65% cpu : usr=0.18%, sys=11.33%, ctx=942, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32701: Fri May 2 02:59:34 2025 read: IOPS=84, BW=1352KiB/s (1384kB/s)(6032KiB/4462msec) clat (usec): min=1861, max=17813, avg=4810.86, stdev=3613.73 lat (usec): min=1861, max=17814, avg=4811.63, stdev=3613.68 clat percentiles (usec): | 1.00th=[ 1926], 5.00th=[ 2040], 10.00th=[ 2147], 20.00th=[ 2311], | 30.00th=[ 2540], 40.00th=[ 2769], 50.00th=[ 3228], 60.00th=[ 3818], | 70.00th=[ 4621], 80.00th=[ 6718], 90.00th=[11207], 95.00th=[13173], | 99.00th=[16188], 99.50th=[16712], 99.90th=[17695], 99.95th=[17695], | 99.99th=[17695] bw ( KiB/s): min= 896, max= 2112, per=22.30%, avg=1171.75, stdev=398.00, samples=8 iops : min= 56, max= 132, avg=73.12, stdev=24.95, samples=8 write: IOPS=87, BW=1402KiB/s (1436kB/s)(6256KiB/4462msec) clat (usec): min=2654, max=32452, avg=6688.92, stdev=4211.27 lat (usec): min=2655, max=32453, avg=6690.30, stdev=4211.30 clat percentiles (usec): | 1.00th=[ 2900], 5.00th=[ 3261], 10.00th=[ 3425], 20.00th=[ 3654], | 30.00th=[ 3949], 40.00th=[ 4424], 50.00th=[ 4883], 60.00th=[ 5997], | 70.00th=[ 7439], 80.00th=[ 9372], 90.00th=[11994], 95.00th=[15401], | 99.00th=[22676], 99.50th=[25297], 99.90th=[32375], 99.95th=[32375], | 99.99th=[32375] bw ( KiB/s): min= 896, max= 2464, per=23.00%, avg=1271.62, stdev=507.18, samples=8 iops : min= 56, max= 154, avg=79.38, stdev=31.74, samples=8 lat (msec) : 2=1.17%, 4=45.57%, 10=38.15%, 20=14.32%, 50=0.78% cpu : usr=0.07%, sys=11.54%, ctx=907, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=5255KiB/s (5381kB/s), 1260KiB/s-1387KiB/s (1290kB/s-1420kB/s), io=23.4MiB (24.5MB), run=4462-4558msec WRITE: bw=5529KiB/s (5661kB/s), 1337KiB/s-1436KiB/s (1369kB/s-1470kB/s), io=24.6MiB (25.8MB), run=4462-4558msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=32711: Fri May 2 02:59:36 2025 read: IOPS=15, BW=15.6MiB/s (16.3MB/s)(7168KiB/450msec) clat (usec): min=338, max=49374, avg=26700.92, stdev=19746.11 lat (usec): min=338, max=49375, avg=26701.95, stdev=19746.34 clat percentiles (usec): | 1.00th=[ 338], 5.00th=[ 338], 10.00th=[ 338], 20.00th=[ 453], | 30.00th=[25297], 40.00th=[25297], 50.00th=[31327], 60.00th=[34866], | 70.00th=[34866], 80.00th=[45351], 90.00th=[49546], 95.00th=[49546], | 99.00th=[49546], 99.50th=[49546], 99.90th=[49546], 99.95th=[49546], | 99.99th=[49546] write: IOPS=11, BW=11.1MiB/s (11.7MB/s)(5120KiB/450msec) clat (usec): min=34568, max=84828, avg=50819.92, stdev=20804.63 lat (usec): min=34636, max=84908, avg=50862.02, stdev=20819.85 clat percentiles (usec): | 1.00th=[34341], 5.00th=[34341], 10.00th=[34341], 20.00th=[34341], | 30.00th=[35390], 40.00th=[35390], 50.00th=[43254], 60.00th=[43254], | 70.00th=[55837], 80.00th=[55837], 90.00th=[84411], 95.00th=[84411], | 99.00th=[84411], 99.50th=[84411], 99.90th=[84411], 99.95th=[84411], | 99.99th=[84411] lat (usec) : 500=16.67% lat (msec) : 50=66.67%, 100=16.67% cpu : usr=0.00%, sys=50.56%, ctx=549, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32713: Fri May 2 02:59:36 2025 read: IOPS=3, BW=3185KiB/s (3262kB/s)(2048KiB/643msec) clat (usec): min=31065, max=34639, avg=32852.44, stdev=2526.77 lat (usec): min=31067, max=34639, avg=32853.51, stdev=2526.35 clat percentiles (usec): | 1.00th=[31065], 5.00th=[31065], 10.00th=[31065], 20.00th=[31065], | 30.00th=[31065], 40.00th=[31065], 50.00th=[31065], 60.00th=[34866], | 70.00th=[34866], 80.00th=[34866], 90.00th=[34866], 95.00th=[34866], | 99.00th=[34866], 99.50th=[34866], 99.90th=[34866], 99.95th=[34866], | 99.99th=[34866] bw ( KiB/s): min= 4096, max= 4096, per=12.86%, avg=4096.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 write: IOPS=15, BW=15.6MiB/s (16.3MB/s)(10.0MiB/643msec) clat (usec): min=32679, max=90638, avg=55918.51, stdev=21853.23 lat (usec): min=32747, max=90702, avg=55975.58, stdev=21854.56 clat percentiles (usec): | 1.00th=[32637], 5.00th=[32637], 10.00th=[32637], 20.00th=[34341], | 30.00th=[35914], 40.00th=[39060], 50.00th=[39060], 60.00th=[65799], | 70.00th=[73925], 80.00th=[73925], 90.00th=[78119], 95.00th=[90702], | 99.00th=[90702], 99.50th=[90702], 99.90th=[90702], 99.95th=[90702], | 99.99th=[90702] bw ( KiB/s): min=14336, max=14336, per=32.15%, avg=14336.00, stdev= 0.00, samples=1 iops : min= 14, max= 14, avg=14.00, stdev= 0.00, samples=1 lat (msec) : 50=58.33%, 100=41.67% cpu : usr=0.00%, sys=40.81%, ctx=287, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32715: Fri May 2 02:59:36 2025 read: IOPS=11, BW=11.7MiB/s (12.3MB/s)(6144KiB/513msec) clat (usec): min=423, max=39278, avg=23226.08, stdev=17905.03 lat (usec): min=424, max=39279, avg=23226.89, stdev=17905.05 clat percentiles (usec): | 1.00th=[ 424], 5.00th=[ 424], 10.00th=[ 424], 20.00th=[ 498], | 30.00th=[ 498], 40.00th=[30016], 50.00th=[30016], 60.00th=[32900], | 70.00th=[36439], 80.00th=[36439], 90.00th=[39060], 95.00th=[39060], | 99.00th=[39060], 99.50th=[39060], 99.90th=[39060], 99.95th=[39060], | 99.99th=[39060] bw ( KiB/s): min=10240, max=10240, per=32.15%, avg=10240.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 write: IOPS=11, BW=11.7MiB/s (12.3MB/s)(6144KiB/513msec) clat (msec): min=26, max=126, avg=57.72, stdev=38.62 lat (msec): min=26, max=126, avg=57.77, stdev=38.62 clat percentiles (msec): | 1.00th=[ 27], 5.00th=[ 27], 10.00th=[ 27], 20.00th=[ 29], | 30.00th=[ 29], 40.00th=[ 34], 50.00th=[ 34], 60.00th=[ 58], | 70.00th=[ 74], 80.00th=[ 74], 90.00th=[ 127], 95.00th=[ 127], | 99.00th=[ 127], 99.50th=[ 127], 99.90th=[ 127], 99.95th=[ 127], | 99.99th=[ 127] bw ( KiB/s): min=12288, max=12288, per=27.56%, avg=12288.00, stdev= 0.00, samples=1 iops : min= 12, max= 12, avg=12.00, stdev= 0.00, samples=1 lat (usec) : 500=16.67% lat (msec) : 50=58.33%, 100=16.67%, 250=8.33% cpu : usr=0.00%, sys=37.89%, ctx=260, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32717: Fri May 2 02:59:36 2025 read: IOPS=8, BW=9094KiB/s (9312kB/s)(5120KiB/563msec) clat (usec): min=410, max=70022, avg=22881.68, stdev=31833.79 lat (usec): min=410, max=70023, avg=22882.54, stdev=31833.92 clat percentiles (usec): | 1.00th=[ 412], 5.00th=[ 412], 10.00th=[ 412], 20.00th=[ 412], | 30.00th=[ 412], 40.00th=[ 412], 50.00th=[ 1549], 60.00th=[ 1549], | 70.00th=[42206], 80.00th=[42206], 90.00th=[69731], 95.00th=[69731], | 99.00th=[69731], 99.50th=[69731], 99.90th=[69731], 99.95th=[69731], | 99.99th=[69731] bw ( KiB/s): min= 8192, max= 8192, per=25.72%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=12, BW=12.4MiB/s (13.0MB/s)(7168KiB/563msec) clat (msec): min=20, max=139, avg=60.15, stdev=40.74 lat (msec): min=20, max=139, avg=60.21, stdev=40.74 clat percentiles (msec): | 1.00th=[ 21], 5.00th=[ 21], 10.00th=[ 21], 20.00th=[ 31], | 30.00th=[ 41], 40.00th=[ 41], 50.00th=[ 42], 60.00th=[ 72], | 70.00th=[ 72], 80.00th=[ 79], 90.00th=[ 140], 95.00th=[ 140], | 99.00th=[ 140], 99.50th=[ 140], 99.90th=[ 140], 99.95th=[ 140], | 99.99th=[ 140] bw ( KiB/s): min=14336, max=14336, per=32.15%, avg=14336.00, stdev= 0.00, samples=1 iops : min= 14, max= 14, avg=14.00, stdev= 0.00, samples=1 lat (usec) : 500=16.67% lat (msec) : 2=8.33%, 50=41.67%, 100=25.00%, 250=8.33% cpu : usr=0.00%, sys=35.05%, ctx=168, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=31.1MiB/s (32.6MB/s), 3185KiB/s-15.6MiB/s (3262kB/s-16.3MB/s), io=20.0MiB (20.0MB), run=450-643msec WRITE: bw=43.5MiB/s (45.7MB/s), 11.1MiB/s-15.6MiB/s (11.7MB/s-16.3MB/s), io=28.0MiB (29.4MB), run=450-643msec rand-rw: (groupid=0, jobs=1): err= 0: pid=32710: Fri May 2 02:59:36 2025 read: IOPS=13, BW=13.0MiB/s (14.7MB/s)(7168KiB/501msec) clat (msec): min=7, max=130, avg=31.84, stdev=44.54 lat (msec): min=7, max=130, avg=31.84, stdev=44.54 clat percentiles (msec): | 1.00th=[ 8], 5.00th=[ 8], 10.00th=[ 8], 20.00th=[ 9], | 30.00th=[ 10], 40.00th=[ 10], 50.00th=[ 13], 60.00th=[ 25], | 70.00th=[ 25], 80.00th=[ 30], 90.00th=[ 131], 95.00th=[ 131], | 99.00th=[ 131], 99.50th=[ 131], 99.90th=[ 131], 99.95th=[ 131], | 99.99th=[ 131] bw ( KiB/s): min=12288, max=12288, per=39.18%, avg=12288.00, stdev= 0.00, samples=1 iops : min= 12, max= 12, avg=12.00, stdev= 0.00, samples=1 write: IOPS=9, BW=9.98MiB/s (10.5MB/s)(5120KiB/501msec) clat (usec): min=17258, max=81815, avg=53403.71, stdev=30144.18 lat (usec): min=17307, max=81842, avg=53448.28, stdev=30140.07 clat percentiles (usec): | 1.00th=[17171], 5.00th=[17171], 10.00th=[17171], 20.00th=[17171], | 30.00th=[24511], 40.00th=[24511], 50.00th=[70779], 60.00th=[70779], | 70.00th=[72877], 80.00th=[72877], 90.00th=[82314], 95.00th=[82314], | 99.00th=[82314], 99.50th=[82314], 99.90th=[82314], 99.95th=[82314], | 99.99th=[82314] bw ( KiB/s): min=10240, max=10240, per=23.32%, avg=10240.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 lat (msec) : 10=25.00%, 20=16.67%, 50=25.00%, 100=25.00%, 250=8.33% cpu : usr=0.00%, sys=9.60%, ctx=48, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32712: Fri May 2 02:59:36 2025 read: IOPS=3, BW=3562KiB/s (3647kB/s)(2048KiB/575msec) clat (usec): min=8113, max=63972, avg=36042.82, stdev=39498.15 lat (usec): min=8115, max=63972, avg=36044.15, stdev=39497.26 clat percentiles (usec): | 1.00th=[ 8094], 5.00th=[ 8094], 10.00th=[ 8094], 20.00th=[ 8094], | 30.00th=[ 8094], 40.00th=[ 8094], 50.00th=[ 8094], 60.00th=[64226], | 70.00th=[64226], 80.00th=[64226], 90.00th=[64226], 95.00th=[64226], | 99.00th=[64226], 99.50th=[64226], 99.90th=[64226], 99.95th=[64226], | 99.99th=[64226] bw ( KiB/s): min= 4096, max= 4096, per=13.06%, avg=4096.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 write: IOPS=17, BW=17.4MiB/s (18.2MB/s)(10.0MiB/575msec) clat (usec): min=12012, max=92588, avg=49466.90, stdev=29703.49 lat (usec): min=12076, max=92660, avg=49526.67, stdev=29708.11 clat percentiles (usec): | 1.00th=[11994], 5.00th=[11994], 10.00th=[11994], 20.00th=[14877], | 30.00th=[21627], 40.00th=[31589], 50.00th=[38536], 60.00th=[61080], | 70.00th=[72877], 80.00th=[72877], 90.00th=[86508], 95.00th=[92799], | 99.00th=[92799], 99.50th=[92799], 99.90th=[92799], 99.95th=[92799], | 99.99th=[92799] bw ( KiB/s): min=16384, max=16384, per=37.31%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 16, max= 16, avg=16.00, stdev= 0.00, samples=1 lat (msec) : 10=8.33%, 20=16.67%, 50=25.00%, 100=50.00% cpu : usr=0.00%, sys=13.07%, ctx=57, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32714: Fri May 2 02:59:36 2025 read: IOPS=9, BW=9423KiB/s (9649kB/s)(6144KiB/652msec) clat (usec): min=12295, max=64034, avg=49046.27, stdev=21521.52 lat (usec): min=12297, max=64035, avg=49047.54, stdev=21521.37 clat percentiles (usec): | 1.00th=[12256], 5.00th=[12256], 10.00th=[12256], 20.00th=[33424], | 30.00th=[33424], 40.00th=[56886], 50.00th=[56886], 60.00th=[63701], | 70.00th=[63701], 80.00th=[63701], 90.00th=[64226], 95.00th=[64226], | 99.00th=[64226], 99.50th=[64226], 99.90th=[64226], 99.95th=[64226], | 99.99th=[64226] bw ( KiB/s): min= 8192, max= 8192, per=26.12%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=9, BW=9423KiB/s (9649kB/s)(6144KiB/652msec) clat (msec): min=8, max=128, avg=57.04, stdev=45.86 lat (msec): min=8, max=128, avg=57.10, stdev=45.86 clat percentiles (msec): | 1.00th=[ 9], 5.00th=[ 9], 10.00th=[ 9], 20.00th=[ 11], | 30.00th=[ 11], 40.00th=[ 42], 50.00th=[ 42], 60.00th=[ 75], | 70.00th=[ 78], 80.00th=[ 78], 90.00th=[ 129], 95.00th=[ 129], | 99.00th=[ 129], 99.50th=[ 129], 99.90th=[ 129], 99.95th=[ 129], | 99.99th=[ 129] bw ( KiB/s): min=10240, max=10240, per=23.32%, avg=10240.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 lat (msec) : 10=8.33%, 20=16.67%, 50=16.67%, 100=50.00%, 250=8.33% cpu : usr=0.15%, sys=13.21%, ctx=81, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32716: Fri May 2 02:59:36 2025 read: IOPS=7, BW=7841KiB/s (8029kB/s)(5120KiB/653msec) clat (usec): min=11636, max=88613, avg=59567.68, stdev=34239.76 lat (usec): min=11636, max=88613, avg=59568.58, stdev=34239.96 clat percentiles (usec): | 1.00th=[11600], 5.00th=[11600], 10.00th=[11600], 20.00th=[11600], | 30.00th=[35390], 40.00th=[35390], 50.00th=[77071], 60.00th=[77071], | 70.00th=[84411], 80.00th=[84411], 90.00th=[88605], 95.00th=[88605], | 99.00th=[88605], 99.50th=[88605], 99.90th=[88605], 99.95th=[88605], | 99.99th=[88605] bw ( KiB/s): min= 8192, max= 8192, per=26.12%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=10, BW=10.7MiB/s (11.2MB/s)(7168KiB/653msec) clat (usec): min=8863, max=88356, avg=46643.24, stdev=33091.05 lat (usec): min=8931, max=88402, avg=46701.43, stdev=33092.17 clat percentiles (usec): | 1.00th=[ 8848], 5.00th=[ 8848], 10.00th=[ 8848], 20.00th=[11600], | 30.00th=[17957], 40.00th=[17957], 50.00th=[55837], 60.00th=[71828], | 70.00th=[71828], 80.00th=[71828], 90.00th=[88605], 95.00th=[88605], | 99.00th=[88605], 99.50th=[88605], 99.90th=[88605], 99.95th=[88605], | 99.99th=[88605] bw ( KiB/s): min= 8192, max= 8192, per=18.66%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 lat (msec) : 10=8.33%, 20=25.00%, 50=8.33%, 100=58.33% cpu : usr=0.00%, sys=10.89%, ctx=73, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=30.6MiB/s (32.1MB/s), 3562KiB/s-13.0MiB/s (3647kB/s-14.7MB/s), io=20.0MiB (20.0MB), run=501-653msec WRITE: bw=42.9MiB/s (44.0MB/s), 9423KiB/s-17.4MiB/s (9649kB/s-18.2MB/s), io=28.0MiB (29.4MB), run=501-653msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=32723: Fri May 2 02:59:38 2025 read: IOPS=4, BW=18.8MiB/s (19.7MB/s)(12.0MiB/640msec) clat (msec): min=144, max=272, avg=207.07, stdev=64.10 lat (msec): min=144, max=272, avg=207.07, stdev=64.10 clat percentiles (msec): | 1.00th=[ 146], 5.00th=[ 146], 10.00th=[ 146], 20.00th=[ 146], | 30.00th=[ 146], 40.00th=[ 203], 50.00th=[ 203], 60.00th=[ 203], | 70.00th=[ 275], 80.00th=[ 275], 90.00th=[ 275], 95.00th=[ 275], | 99.00th=[ 275], 99.50th=[ 275], 99.90th=[ 275], 99.95th=[ 275], | 99.99th=[ 275] bw ( KiB/s): min=15968, max=15968, per=62.38%, avg=15968.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.00%, sys=2.50%, ctx=17, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32724: Fri May 2 02:59:38 2025 read: IOPS=4, BW=17.9MiB/s (18.8MB/s)(4096KiB/223msec) clat (nsec): min=15592k, max=15592k, avg=15592282.00, stdev= 0.00 lat (nsec): min=15594k, max=15594k, avg=15594367.00, stdev= 0.00 clat percentiles (usec): | 1.00th=[15533], 5.00th=[15533], 10.00th=[15533], 20.00th=[15533], | 30.00th=[15533], 40.00th=[15533], 50.00th=[15533], 60.00th=[15533], | 70.00th=[15533], 80.00th=[15533], 90.00th=[15533], 95.00th=[15533], | 99.00th=[15533], 99.50th=[15533], 99.90th=[15533], 99.95th=[15533], | 99.99th=[15533] write: IOPS=8, BW=35.9MiB/s (37.6MB/s)(8192KiB/223msec) clat (msec): min=47, max=119, avg=83.17, stdev=50.75 lat (msec): min=47, max=119, avg=83.36, stdev=50.85 clat percentiles (msec): | 1.00th=[ 48], 5.00th=[ 48], 10.00th=[ 48], 20.00th=[ 48], | 30.00th=[ 48], 40.00th=[ 48], 50.00th=[ 48], 60.00th=[ 120], | 70.00th=[ 120], 80.00th=[ 120], 90.00th=[ 120], 95.00th=[ 120], | 99.00th=[ 120], 99.50th=[ 120], 99.90th=[ 120], 99.95th=[ 120], | 99.99th=[ 120] lat (msec) : 20=33.33%, 50=33.33%, 250=33.33% cpu : usr=0.00%, sys=13.96%, ctx=29, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32725: Fri May 2 02:59:38 2025 write: IOPS=4, BW=17.4MiB/s (18.3MB/s)(12.0MiB/688msec) clat (msec): min=139, max=317, avg=214.95, stdev=92.01 lat (msec): min=140, max=317, avg=215.23, stdev=92.02 clat percentiles (msec): | 1.00th=[ 140], 5.00th=[ 140], 10.00th=[ 140], 20.00th=[ 140], | 30.00th=[ 140], 40.00th=[ 188], 50.00th=[ 188], 60.00th=[ 188], | 70.00th=[ 317], 80.00th=[ 317], 90.00th=[ 317], 95.00th=[ 317], | 99.00th=[ 317], 99.50th=[ 317], 99.90th=[ 317], 99.95th=[ 317], | 99.99th=[ 317] bw ( KiB/s): min=16384, max=16384, per=38.00%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.29%, sys=12.08%, ctx=187, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32726: Fri May 2 02:59:38 2025 write: IOPS=3, BW=15.8MiB/s (16.6MB/s)(12.0MiB/760msec) clat (msec): min=149, max=378, avg=246.65, stdev=118.50 lat (msec): min=149, max=378, avg=246.91, stdev=118.49 clat percentiles (msec): | 1.00th=[ 150], 5.00th=[ 150], 10.00th=[ 150], 20.00th=[ 150], | 30.00th=[ 150], 40.00th=[ 211], 50.00th=[ 211], 60.00th=[ 211], | 70.00th=[ 380], 80.00th=[ 380], 90.00th=[ 380], 95.00th=[ 380], | 99.00th=[ 380], 99.50th=[ 380], 99.90th=[ 380], 99.95th=[ 380], | 99.99th=[ 380] bw ( KiB/s): min=16351, max=16351, per=37.92%, avg=16351.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.00%, sys=11.20%, ctx=150, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=25.0MiB/s (26.2MB/s), 17.9MiB/s-18.8MiB/s (18.8MB/s-19.7MB/s), io=16.0MiB (16.8MB), run=223-640msec WRITE: bw=42.1MiB/s (44.1MB/s), 15.8MiB/s-35.9MiB/s (16.6MB/s-37.6MB/s), io=32.0MiB (33.6MB), run=223-760msec rand-rw: (groupid=0, jobs=1): err= 0: pid=32727: Fri May 2 02:59:38 2025 read: IOPS=9, BW=38.5MiB/s (40.3MB/s)(12.0MiB/312msec) clat (msec): min=30, max=148, avg=99.99, stdev=61.80 lat (msec): min=30, max=148, avg=99.99, stdev=61.80 clat percentiles (msec): | 1.00th=[ 31], 5.00th=[ 31], 10.00th=[ 31], 20.00th=[ 31], | 30.00th=[ 31], 40.00th=[ 121], 50.00th=[ 121], 60.00th=[ 121], | 70.00th=[ 148], 80.00th=[ 148], 90.00th=[ 148], 95.00th=[ 148], | 99.00th=[ 148], 99.50th=[ 148], 99.90th=[ 148], 99.95th=[ 148], | 99.99th=[ 148] lat (msec) : 50=33.33%, 250=66.67% cpu : usr=0.00%, sys=51.45%, ctx=1387, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32728: Fri May 2 02:59:38 2025 read: IOPS=1, BW=5988KiB/s (6132kB/s)(4096KiB/684msec) clat (nsec): min=119136k, max=119136k, avg=119136315.00, stdev= 0.00 lat (nsec): min=119138k, max=119138k, avg=119138495.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 120], 5.00th=[ 120], 10.00th=[ 120], 20.00th=[ 120], | 30.00th=[ 120], 40.00th=[ 120], 50.00th=[ 120], 60.00th=[ 120], | 70.00th=[ 120], 80.00th=[ 120], 90.00th=[ 120], 95.00th=[ 120], | 99.00th=[ 120], 99.50th=[ 120], 99.90th=[ 120], 99.95th=[ 120], | 99.99th=[ 120] bw ( KiB/s): min= 8192, max= 8192, per=34.20%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=2, BW=11.7MiB/s (12.3MB/s)(8192KiB/684msec) clat (msec): min=210, max=339, avg=274.85, stdev=91.16 lat (msec): min=210, max=339, avg=275.10, stdev=91.13 clat percentiles (msec): | 1.00th=[ 211], 5.00th=[ 211], 10.00th=[ 211], 20.00th=[ 211], | 30.00th=[ 211], 40.00th=[ 211], 50.00th=[ 211], 60.00th=[ 338], | 70.00th=[ 338], 80.00th=[ 338], 90.00th=[ 338], 95.00th=[ 338], | 99.00th=[ 338], 99.50th=[ 338], 99.90th=[ 338], 99.95th=[ 338], | 99.99th=[ 338] bw ( KiB/s): min= 8192, max= 8192, per=18.78%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.15%, sys=32.80%, ctx=644, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32730: Fri May 2 02:59:38 2025 write: IOPS=3, BW=15.0MiB/s (16.8MB/s)(12.0MiB/751msec) clat (msec): min=167, max=349, avg=236.65, stdev=98.80 lat (msec): min=167, max=350, avg=240.28, stdev=96.71 clat percentiles (msec): | 1.00th=[ 167], 5.00th=[ 167], 10.00th=[ 167], 20.00th=[ 167], | 30.00th=[ 167], 40.00th=[ 192], 50.00th=[ 192], 60.00th=[ 192], | 70.00th=[ 351], 80.00th=[ 351], 90.00th=[ 351], 95.00th=[ 351], | 99.00th=[ 351], 99.50th=[ 351], 99.90th=[ 351], 99.95th=[ 351], | 99.99th=[ 351] bw ( KiB/s): min= 8143, max= 8143, per=18.66%, avg=8143.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.00%, sys=39.60%, ctx=28, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32731: Fri May 2 02:59:38 2025 write: IOPS=4, BW=18.8MiB/s (19.7MB/s)(12.0MiB/638msec) clat (msec): min=158, max=291, avg=207.99, stdev=72.96 lat (msec): min=158, max=291, avg=208.21, stdev=72.90 clat percentiles (msec): | 1.00th=[ 159], 5.00th=[ 159], 10.00th=[ 159], 20.00th=[ 159], | 30.00th=[ 159], 40.00th=[ 174], 50.00th=[ 174], 60.00th=[ 174], | 70.00th=[ 292], 80.00th=[ 292], 90.00th=[ 292], 95.00th=[ 292], | 99.00th=[ 292], 99.50th=[ 292], 99.90th=[ 292], 99.95th=[ 292], | 99.99th=[ 292] bw ( KiB/s): min=16384, max=16384, per=37.55%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.00%, sys=51.41%, ctx=56, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=23.4MiB/s (24.5MB/s), 5988KiB/s-38.5MiB/s (6132kB/s-40.3MB/s), io=16.0MiB (16.8MB), run=312-684msec WRITE: bw=42.6MiB/s (44.7MB/s), 11.7MiB/s-18.8MiB/s (12.3MB/s-19.7MB/s), io=32.0MiB (33.6MB), run=638-751msec