== sanity test 398b: DIO and buffer IO race ============== 03:06:07 (1747811167) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.2545 s, 40.1 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=20206: Wed May 21 03:06:22 2025 read: IOPS=116, BW=467KiB/s (478kB/s)(5992KiB/12841msec) clat (usec): min=57, max=173765, avg=4721.94, stdev=5096.69 lat (usec): min=57, max=173767, avg=4724.57, stdev=5096.77 clat percentiles (usec): | 1.00th=[ 71], 5.00th=[ 326], 10.00th=[ 2245], 20.00th=[ 2671], | 30.00th=[ 3097], 40.00th=[ 3589], 50.00th=[ 4228], 60.00th=[ 4817], | 70.00th=[ 5538], 80.00th=[ 6259], 90.00th=[ 7570], 95.00th=[ 9110], | 99.00th=[ 14746], 99.50th=[ 16581], 99.90th=[ 20317], 99.95th=[173016], | 99.99th=[173016] bw ( KiB/s): min= 215, max= 640, per=25.18%, avg=464.76, stdev=90.83, samples=25 iops : min= 53, max= 160, avg=116.08, stdev=22.76, samples=25 write: IOPS=122, BW=490KiB/s (502kB/s)(6296KiB/12841msec) clat (usec): min=465, max=20380, avg=3598.52, stdev=3281.82 lat (usec): min=465, max=20380, avg=3601.23, stdev=3280.85 clat percentiles (usec): | 1.00th=[ 519], 5.00th=[ 553], 10.00th=[ 586], 20.00th=[ 676], | 30.00th=[ 881], 40.00th=[ 1876], 50.00th=[ 2868], 60.00th=[ 3556], | 70.00th=[ 4490], 80.00th=[ 6194], 90.00th=[ 8225], 95.00th=[10159], | 99.00th=[13829], 99.50th=[16581], 99.90th=[19792], 99.95th=[20317], | 99.99th=[20317] bw ( KiB/s): min= 239, max= 744, per=25.28%, avg=488.72, stdev=92.48, samples=25 iops : min= 59, max= 186, avg=122.08, stdev=23.16, samples=25 lat (usec) : 100=1.82%, 250=0.59%, 500=0.33%, 750=12.50%, 1000=4.52% lat (msec) : 2=4.30%, 4=32.03%, 10=39.42%, 20=4.39%, 50=0.07% lat (msec) : 250=0.03% cpu : usr=0.16%, sys=22.66%, ctx=5651, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20207: Wed May 21 03:06:22 2025 read: IOPS=114, BW=457KiB/s (468kB/s)(5940KiB/13004msec) clat (usec): min=53, max=23516, avg=4605.22, stdev=2720.57 lat (usec): min=53, max=23516, avg=4607.09, stdev=2720.27 clat percentiles (usec): | 1.00th=[ 69], 5.00th=[ 119], 10.00th=[ 2180], 20.00th=[ 2638], | 30.00th=[ 3064], 40.00th=[ 3589], 50.00th=[ 4113], 60.00th=[ 4817], | 70.00th=[ 5473], 80.00th=[ 6390], 90.00th=[ 7767], 95.00th=[ 9241], | 99.00th=[14222], 99.50th=[16909], 99.90th=[22152], 99.95th=[23462], | 99.99th=[23462] bw ( KiB/s): min= 256, max= 592, per=24.39%, avg=450.20, stdev=65.61, samples=25 iops : min= 64, max= 148, avg=112.52, stdev=16.42, samples=25 write: IOPS=122, BW=488KiB/s (500kB/s)(6348KiB/13004msec) clat (usec): min=497, max=181963, avg=3840.26, stdev=5472.32 lat (usec): min=497, max=181964, avg=3841.15, stdev=5472.34 clat percentiles (usec): | 1.00th=[ 523], 5.00th=[ 553], 10.00th=[ 594], 20.00th=[ 709], | 30.00th=[ 1020], 40.00th=[ 2376], 50.00th=[ 3097], 60.00th=[ 3916], | 70.00th=[ 4883], 80.00th=[ 6194], 90.00th=[ 8029], 95.00th=[ 9503], | 99.00th=[ 14222], 99.50th=[ 15533], 99.90th=[ 22676], 99.95th=[181404], | 99.99th=[181404] bw ( KiB/s): min= 336, max= 600, per=24.63%, avg=476.08, stdev=69.33, samples=25 iops : min= 84, max= 150, avg=119.00, stdev=17.33, samples=25 lat (usec) : 100=2.18%, 250=0.68%, 500=0.07%, 750=11.62%, 1000=3.71% lat (msec) : 2=4.20%, 4=32.03%, 10=41.63%, 20=3.71%, 50=0.13% lat (msec) : 250=0.03% cpu : usr=0.22%, sys=22.46%, ctx=5612, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20208: Wed May 21 03:06:22 2025 read: IOPS=117, BW=469KiB/s (480kB/s)(6028KiB/12847msec) clat (usec): min=53, max=19661, avg=4655.39, stdev=2560.52 lat (usec): min=54, max=19661, avg=4656.03, stdev=2560.51 clat percentiles (usec): | 1.00th=[ 74], 5.00th=[ 1004], 10.00th=[ 2311], 20.00th=[ 2769], | 30.00th=[ 3163], 40.00th=[ 3621], 50.00th=[ 4146], 60.00th=[ 4817], | 70.00th=[ 5538], 80.00th=[ 6390], 90.00th=[ 7701], 95.00th=[ 9241], | 99.00th=[13566], 99.50th=[14746], 99.90th=[18744], 99.95th=[19792], | 99.99th=[19792] bw ( KiB/s): min= 240, max= 616, per=25.24%, avg=465.84, stdev=77.66, samples=25 iops : min= 60, max= 154, avg=116.44, stdev=19.40, samples=25 write: IOPS=121, BW=487KiB/s (499kB/s)(6260KiB/12847msec) clat (usec): min=422, max=170154, avg=3667.29, stdev=5225.84 lat (usec): min=422, max=170155, avg=3668.46, stdev=5225.74 clat percentiles (usec): | 1.00th=[ 515], 5.00th=[ 562], 10.00th=[ 594], 20.00th=[ 676], | 30.00th=[ 922], 40.00th=[ 2180], 50.00th=[ 3032], 60.00th=[ 3589], | 70.00th=[ 4555], 80.00th=[ 5997], 90.00th=[ 7832], 95.00th=[ 9503], | 99.00th=[ 13435], 99.50th=[ 15926], 99.90th=[ 19792], 99.95th=[170918], | 99.99th=[170918] bw ( KiB/s): min= 320, max= 632, per=24.91%, avg=481.56, stdev=75.95, samples=25 iops : min= 80, max= 158, avg=120.36, stdev=19.03, samples=25 lat (usec) : 100=1.37%, 250=0.98%, 500=0.16%, 750=12.11%, 1000=4.13% lat (msec) : 2=4.04%, 4=33.50%, 10=39.97%, 20=3.71%, 250=0.03% cpu : usr=0.26%, sys=22.75%, ctx=5689, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20209: Wed May 21 03:06:22 2025 read: IOPS=118, BW=473KiB/s (484kB/s)(6052KiB/12794msec) clat (usec): min=58, max=20159, avg=4649.68, stdev=2624.14 lat (usec): min=58, max=20160, avg=4651.58, stdev=2623.59 clat percentiles (usec): | 1.00th=[ 73], 5.00th=[ 1827], 10.00th=[ 2245], 20.00th=[ 2737], | 30.00th=[ 3097], 40.00th=[ 3589], 50.00th=[ 4146], 60.00th=[ 4817], | 70.00th=[ 5604], 80.00th=[ 6390], 90.00th=[ 7635], 95.00th=[ 8979], | 99.00th=[14353], 99.50th=[16188], 99.90th=[19792], 99.95th=[20055], | 99.99th=[20055] bw ( KiB/s): min= 256, max= 600, per=25.39%, avg=468.76, stdev=78.63, samples=25 iops : min= 64, max= 150, avg=117.16, stdev=19.67, samples=25 write: IOPS=121, BW=487KiB/s (499kB/s)(6236KiB/12794msec) clat (usec): min=450, max=168380, avg=3629.32, stdev=5206.16 lat (usec): min=450, max=168381, avg=3630.37, stdev=5206.11 clat percentiles (usec): | 1.00th=[ 529], 5.00th=[ 562], 10.00th=[ 586], 20.00th=[ 676], | 30.00th=[ 889], 40.00th=[ 1926], 50.00th=[ 2802], 60.00th=[ 3523], | 70.00th=[ 4686], 80.00th=[ 5997], 90.00th=[ 8029], 95.00th=[ 9634], | 99.00th=[ 12911], 99.50th=[ 14091], 99.90th=[ 22938], 99.95th=[168821], | 99.99th=[168821] bw ( KiB/s): min= 336, max= 728, per=25.01%, avg=483.48, stdev=85.06, samples=25 iops : min= 84, max= 182, avg=120.84, stdev=21.28, samples=25 lat (usec) : 100=1.50%, 250=0.68%, 500=0.23%, 750=11.98%, 1000=4.79% lat (msec) : 2=4.33%, 4=32.52%, 10=40.04%, 20=3.81%, 50=0.10% lat (msec) : 250=0.03% cpu : usr=0.30%, sys=22.26%, ctx=5589, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1847KiB/s (1891kB/s), 457KiB/s-473KiB/s (468kB/s-484kB/s), io=23.4MiB (24.6MB), run=12794-13004msec WRITE: bw=1933KiB/s (1980kB/s), 487KiB/s-490KiB/s (499kB/s-502kB/s), io=24.6MiB (25.7MB), run=12794-13004msec rand-rw: (groupid=0, jobs=1): err= 0: pid=20210: Wed May 21 03:06:26 2025 read: IOPS=87, BW=351KiB/s (359kB/s)(5992KiB/17072msec) clat (usec): min=1655, max=193493, avg=4779.89, stdev=5979.80 lat (usec): min=1656, max=193496, avg=4780.57, stdev=5979.84 clat percentiles (usec): | 1.00th=[ 1778], 5.00th=[ 1926], 10.00th=[ 2024], 20.00th=[ 2180], | 30.00th=[ 2376], 40.00th=[ 2671], 50.00th=[ 3163], 60.00th=[ 3818], | 70.00th=[ 4948], 80.00th=[ 6915], 90.00th=[ 9896], 95.00th=[ 12518], | 99.00th=[ 16057], 99.50th=[ 17433], 99.90th=[ 25822], 99.95th=[193987], | 99.99th=[193987] bw ( KiB/s): min= 143, max= 776, per=24.88%, avg=349.76, stdev=161.40, samples=34 iops : min= 35, max= 194, avg=87.38, stdev=40.38, samples=34 write: IOPS=92, BW=369KiB/s (378kB/s)(6296KiB/17072msec) clat (usec): min=2391, max=32524, avg=6260.18, stdev=3697.07 lat (usec): min=2391, max=32524, avg=6260.99, stdev=3697.13 clat percentiles (usec): | 1.00th=[ 2638], 5.00th=[ 2933], 10.00th=[ 3163], 20.00th=[ 3425], | 30.00th=[ 3720], 40.00th=[ 4228], 50.00th=[ 4817], 60.00th=[ 5932], | 70.00th=[ 7177], 80.00th=[ 8979], 90.00th=[11207], 95.00th=[13435], | 99.00th=[19792], 99.50th=[22414], 99.90th=[24511], 99.95th=[32637], | 99.99th=[32637] bw ( KiB/s): min= 159, max= 816, per=24.99%, avg=367.91, stdev=166.30, samples=34 iops : min= 39, max= 204, avg=91.91, stdev=41.60, samples=34 lat (msec) : 2=3.91%, 4=45.25%, 10=38.44%, 20=11.78%, 50=0.59% lat (msec) : 250=0.03% cpu : usr=0.19%, sys=11.15%, ctx=3747, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20211: Wed May 21 03:06:26 2025 read: IOPS=88, BW=352KiB/s (361kB/s)(5940KiB/16861msec) clat (usec): min=1704, max=174743, avg=4617.71, stdev=5464.27 lat (usec): min=1705, max=174744, avg=4618.61, stdev=5464.22 clat percentiles (usec): | 1.00th=[ 1827], 5.00th=[ 1958], 10.00th=[ 2040], 20.00th=[ 2212], | 30.00th=[ 2409], 40.00th=[ 2704], 50.00th=[ 3130], 60.00th=[ 3785], | 70.00th=[ 4752], 80.00th=[ 6390], 90.00th=[ 9372], 95.00th=[ 11600], | 99.00th=[ 16188], 99.50th=[ 17695], 99.90th=[ 23987], 99.95th=[175113], | 99.99th=[175113] bw ( KiB/s): min= 112, max= 784, per=25.06%, avg=352.36, stdev=165.32, samples=33 iops : min= 28, max= 196, avg=88.03, stdev=41.34, samples=33 write: IOPS=94, BW=376KiB/s (386kB/s)(6348KiB/16861msec) clat (usec): min=2368, max=34128, avg=6252.24, stdev=3600.74 lat (usec): min=2369, max=34128, avg=6253.15, stdev=3600.70 clat percentiles (usec): | 1.00th=[ 2638], 5.00th=[ 2999], 10.00th=[ 3195], 20.00th=[ 3425], | 30.00th=[ 3720], 40.00th=[ 4146], 50.00th=[ 4948], 60.00th=[ 6063], | 70.00th=[ 7373], 80.00th=[ 8717], 90.00th=[11076], 95.00th=[13304], | 99.00th=[19006], 99.50th=[21627], 99.90th=[24249], 99.95th=[34341], | 99.99th=[34341] bw ( KiB/s): min= 200, max= 744, per=25.27%, avg=372.00, stdev=153.51, samples=33 iops : min= 50, max= 186, avg=92.94, stdev=38.36, samples=33 lat (msec) : 2=3.61%, 4=45.87%, 10=39.39%, 20=10.64%, 50=0.46% lat (msec) : 250=0.03% cpu : usr=0.27%, sys=11.11%, ctx=3752, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20212: Wed May 21 03:06:26 2025 read: IOPS=88, BW=353KiB/s (362kB/s)(6028KiB/17068msec) clat (usec): min=1654, max=25815, avg=4680.31, stdev=3500.33 lat (usec): min=1654, max=25816, avg=4681.05, stdev=3500.30 clat percentiles (usec): | 1.00th=[ 1795], 5.00th=[ 1942], 10.00th=[ 2057], 20.00th=[ 2212], | 30.00th=[ 2409], 40.00th=[ 2769], 50.00th=[ 3163], 60.00th=[ 3949], | 70.00th=[ 4817], 80.00th=[ 6652], 90.00th=[10028], 95.00th=[12256], | 99.00th=[16712], 99.50th=[18482], 99.90th=[24249], 99.95th=[25822], | 99.99th=[25822] bw ( KiB/s): min= 160, max= 792, per=24.89%, avg=349.97, stdev=158.91, samples=33 iops : min= 40, max= 198, avg=87.42, stdev=39.77, samples=33 write: IOPS=91, BW=367KiB/s (376kB/s)(6260KiB/17068msec) clat (msec): min=2, max=166, avg= 6.34, stdev= 5.48 lat (msec): min=2, max=166, avg= 6.34, stdev= 5.48 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 4], 20.00th=[ 4], | 30.00th=[ 4], 40.00th=[ 5], 50.00th=[ 5], 60.00th=[ 6], | 70.00th=[ 8], 80.00th=[ 9], 90.00th=[ 12], 95.00th=[ 14], | 99.00th=[ 21], 99.50th=[ 23], 99.90th=[ 33], 99.95th=[ 167], | 99.99th=[ 167] bw ( KiB/s): min= 184, max= 760, per=24.57%, avg=361.61, stdev=163.00, samples=33 iops : min= 46, max= 190, avg=90.33, stdev=40.78, samples=33 lat (msec) : 2=3.65%, 4=44.95%, 10=39.29%, 20=11.46%, 50=0.62% lat (msec) : 250=0.03% cpu : usr=0.20%, sys=11.11%, ctx=3675, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20213: Wed May 21 03:06:26 2025 read: IOPS=89, BW=356KiB/s (365kB/s)(6052KiB/16998msec) clat (usec): min=1678, max=22762, avg=4439.10, stdev=3238.85 lat (usec): min=1682, max=22763, avg=4439.89, stdev=3238.76 clat percentiles (usec): | 1.00th=[ 1827], 5.00th=[ 1975], 10.00th=[ 2089], 20.00th=[ 2245], | 30.00th=[ 2442], 40.00th=[ 2704], 50.00th=[ 3097], 60.00th=[ 3621], | 70.00th=[ 4490], 80.00th=[ 6063], 90.00th=[ 9110], 95.00th=[11731], | 99.00th=[15926], 99.50th=[18220], 99.90th=[20579], 99.95th=[22676], | 99.99th=[22676] bw ( KiB/s): min= 152, max= 712, per=25.04%, avg=352.09, stdev=158.33, samples=33 iops : min= 38, max= 178, avg=87.91, stdev=39.62, samples=33 write: IOPS=91, BW=367KiB/s (376kB/s)(6236KiB/16998msec) clat (msec): min=2, max=175, avg= 6.54, stdev= 5.73 lat (msec): min=2, max=175, avg= 6.54, stdev= 5.73 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 4], 20.00th=[ 4], | 30.00th=[ 4], 40.00th=[ 5], 50.00th=[ 6], 60.00th=[ 7], | 70.00th=[ 8], 80.00th=[ 10], 90.00th=[ 12], 95.00th=[ 14], | 99.00th=[ 21], 99.50th=[ 22], 99.90th=[ 35], 99.95th=[ 176], | 99.99th=[ 176] bw ( KiB/s): min= 184, max= 752, per=24.71%, avg=363.73, stdev=157.63, samples=33 iops : min= 46, max= 188, avg=90.82, stdev=39.47, samples=33 lat (msec) : 2=3.22%, 4=46.71%, 10=37.83%, 20=11.62%, 50=0.59% lat (msec) : 250=0.03% cpu : usr=0.13%, sys=11.32%, ctx=3695, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1407KiB/s (1440kB/s), 351KiB/s-356KiB/s (359kB/s-365kB/s), io=23.4MiB (24.6MB), run=16861-17072msec WRITE: bw=1473KiB/s (1508kB/s), 367KiB/s-376KiB/s (376kB/s-386kB/s), io=24.6MiB (25.7MB), run=16861-17072msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=20246: Wed May 21 03:06:31 2025 read: IOPS=105, BW=1684KiB/s (1725kB/s)(5920KiB/3515msec) clat (usec): min=69, max=23288, avg=5066.55, stdev=2962.29 lat (usec): min=69, max=23289, avg=5067.65, stdev=2961.96 clat percentiles (usec): | 1.00th=[ 74], 5.00th=[ 88], 10.00th=[ 265], 20.00th=[ 3064], | 30.00th=[ 3556], 40.00th=[ 4113], 50.00th=[ 4817], 60.00th=[ 5538], | 70.00th=[ 6194], 80.00th=[ 7046], 90.00th=[ 8291], 95.00th=[ 9896], | 99.00th=[14091], 99.50th=[16581], 99.90th=[23200], 99.95th=[23200], | 99.99th=[23200] bw ( KiB/s): min= 1408, max= 2048, per=25.97%, avg=1681.29, stdev=244.96, samples=7 iops : min= 88, max= 128, avg=104.86, stdev=15.35, samples=7 write: IOPS=113, BW=1812KiB/s (1855kB/s)(6368KiB/3515msec) clat (usec): min=656, max=15593, avg=4085.34, stdev=3301.51 lat (usec): min=657, max=15594, avg=4086.31, stdev=3301.51 clat percentiles (usec): | 1.00th=[ 734], 5.00th=[ 857], 10.00th=[ 906], 20.00th=[ 1057], | 30.00th=[ 1385], 40.00th=[ 2212], 50.00th=[ 3130], 60.00th=[ 4047], | 70.00th=[ 5342], 80.00th=[ 6915], 90.00th=[ 9241], 95.00th=[10552], | 99.00th=[13304], 99.50th=[15401], 99.90th=[15533], 99.95th=[15533], | 99.99th=[15533] bw ( KiB/s): min= 1440, max= 2272, per=26.50%, avg=1804.57, stdev=280.10, samples=7 iops : min= 90, max= 142, avg=112.57, stdev=17.62, samples=7 lat (usec) : 100=3.26%, 250=1.43%, 500=0.26%, 750=0.52%, 1000=8.59% lat (msec) : 2=11.33%, 4=22.79%, 10=45.83%, 20=5.86%, 50=0.13% cpu : usr=0.11%, sys=25.41%, ctx=1612, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20247: Wed May 21 03:06:31 2025 read: IOPS=97, BW=1552KiB/s (1590kB/s)(5744KiB/3700msec) clat (usec): min=73, max=17245, avg=5195.29, stdev=2763.83 lat (usec): min=74, max=17246, avg=5201.85, stdev=2769.47 clat percentiles (usec): | 1.00th=[ 77], 5.00th=[ 105], 10.00th=[ 2474], 20.00th=[ 3228], | 30.00th=[ 3752], 40.00th=[ 4555], 50.00th=[ 5145], 60.00th=[ 5735], | 70.00th=[ 6390], 80.00th=[ 7111], 90.00th=[ 8225], 95.00th=[10159], | 99.00th=[14877], 99.50th=[16057], 99.90th=[17171], 99.95th=[17171], | 99.99th=[17171] bw ( KiB/s): min= 1021, max= 1888, per=23.65%, avg=1530.57, stdev=301.13, samples=7 iops : min= 63, max= 118, avg=95.43, stdev=19.16, samples=7 write: IOPS=110, BW=1769KiB/s (1811kB/s)(6544KiB/3700msec) clat (usec): min=710, max=22380, avg=4362.65, stdev=3553.82 lat (usec): min=711, max=22382, avg=4363.79, stdev=3553.88 clat percentiles (usec): | 1.00th=[ 766], 5.00th=[ 881], 10.00th=[ 971], 20.00th=[ 1205], | 30.00th=[ 1729], 40.00th=[ 2933], 50.00th=[ 3556], 60.00th=[ 4146], | 70.00th=[ 5276], 80.00th=[ 6587], 90.00th=[ 9110], 95.00th=[11338], | 99.00th=[15795], 99.50th=[16909], 99.90th=[22414], 99.95th=[22414], | 99.99th=[22414] bw ( KiB/s): min= 1341, max= 2304, per=25.36%, avg=1727.00, stdev=332.69, samples=7 iops : min= 83, max= 144, avg=107.71, stdev=20.87, samples=7 lat (usec) : 100=2.08%, 250=1.82%, 500=0.26%, 750=0.65%, 1000=5.60% lat (msec) : 2=11.46%, 4=25.26%, 10=46.22%, 20=6.51%, 50=0.13% cpu : usr=0.38%, sys=25.09%, ctx=1772, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20248: Wed May 21 03:06:31 2025 read: IOPS=106, BW=1703KiB/s (1744kB/s)(6256KiB/3673msec) clat (usec): min=69, max=14858, avg=5170.81, stdev=2726.59 lat (usec): min=69, max=14859, avg=5171.54, stdev=2726.77 clat percentiles (usec): | 1.00th=[ 75], 5.00th=[ 117], 10.00th=[ 2409], 20.00th=[ 3130], | 30.00th=[ 3687], 40.00th=[ 4228], 50.00th=[ 4948], 60.00th=[ 5669], | 70.00th=[ 6587], 80.00th=[ 7177], 90.00th=[ 8586], 95.00th=[10159], | 99.00th=[13960], 99.50th=[14222], 99.90th=[14877], 99.95th=[14877], | 99.99th=[14877] bw ( KiB/s): min= 1280, max= 2048, per=26.27%, avg=1700.57, stdev=279.23, samples=7 iops : min= 80, max= 128, avg=106.29, stdev=17.45, samples=7 write: IOPS=102, BW=1642KiB/s (1682kB/s)(6032KiB/3673msec) clat (usec): min=713, max=17022, avg=4224.56, stdev=3440.15 lat (usec): min=714, max=17023, avg=4228.68, stdev=3438.03 clat percentiles (usec): | 1.00th=[ 766], 5.00th=[ 848], 10.00th=[ 922], 20.00th=[ 1057], | 30.00th=[ 1401], 40.00th=[ 2311], 50.00th=[ 3195], 60.00th=[ 4047], | 70.00th=[ 5866], 80.00th=[ 7177], 90.00th=[ 9241], 95.00th=[10945], | 99.00th=[14877], 99.50th=[16450], 99.90th=[16909], 99.95th=[16909], | 99.99th=[16909] bw ( KiB/s): min= 1312, max= 1920, per=23.49%, avg=1600.00, stdev=220.93, samples=7 iops : min= 82, max= 120, avg=100.00, stdev=13.81, samples=7 lat (usec) : 100=2.21%, 250=1.56%, 500=0.13%, 750=0.26%, 1000=7.81% lat (msec) : 2=10.03%, 4=25.39%, 10=46.22%, 20=6.38% cpu : usr=0.27%, sys=24.81%, ctx=1697, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20249: Wed May 21 03:06:31 2025 read: IOPS=104, BW=1677KiB/s (1717kB/s)(6032KiB/3597msec) clat (usec): min=69, max=16592, avg=5203.01, stdev=3046.50 lat (usec): min=70, max=16593, avg=5203.83, stdev=3046.66 clat percentiles (usec): | 1.00th=[ 74], 5.00th=[ 92], 10.00th=[ 379], 20.00th=[ 3064], | 30.00th=[ 3556], 40.00th=[ 4228], 50.00th=[ 4948], 60.00th=[ 5735], | 70.00th=[ 6521], 80.00th=[ 7308], 90.00th=[ 8979], 95.00th=[10814], | 99.00th=[15533], 99.50th=[16057], 99.90th=[16581], 99.95th=[16581], | 99.99th=[16581] bw ( KiB/s): min= 1280, max= 1856, per=25.78%, avg=1668.57, stdev=195.02, samples=7 iops : min= 80, max= 116, avg=104.29, stdev=12.19, samples=7 write: IOPS=108, BW=1739KiB/s (1781kB/s)(6256KiB/3597msec) clat (usec): min=728, max=15611, avg=4028.08, stdev=3239.70 lat (usec): min=729, max=15612, avg=4030.02, stdev=3239.38 clat percentiles (usec): | 1.00th=[ 783], 5.00th=[ 857], 10.00th=[ 914], 20.00th=[ 1074], | 30.00th=[ 1401], 40.00th=[ 2212], 50.00th=[ 3294], 60.00th=[ 4015], | 70.00th=[ 5145], 80.00th=[ 6521], 90.00th=[ 8455], 95.00th=[10814], | 99.00th=[14222], 99.50th=[15008], 99.90th=[15664], 99.95th=[15664], | 99.99th=[15664] bw ( KiB/s): min= 1408, max= 2144, per=25.58%, avg=1741.71, stdev=230.65, samples=7 iops : min= 88, max= 134, avg=108.86, stdev=14.42, samples=7 lat (usec) : 100=2.86%, 250=1.69%, 500=0.39%, 750=0.26%, 1000=7.68% lat (msec) : 2=11.72%, 4=24.09%, 10=44.79%, 20=6.51% cpu : usr=0.25%, sys=24.89%, ctx=1712, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=6474KiB/s (6629kB/s), 1552KiB/s-1703KiB/s (1590kB/s-1744kB/s), io=23.4MiB (24.5MB), run=3515-3700msec WRITE: bw=6811KiB/s (6974kB/s), 1642KiB/s-1812KiB/s (1682kB/s-1855kB/s), io=24.6MiB (25.8MB), run=3515-3700msec rand-rw: (groupid=0, jobs=1): err= 0: pid=20242: Wed May 21 03:06:32 2025 read: IOPS=80, BW=1282KiB/s (1313kB/s)(5920KiB/4616msec) clat (usec): min=2065, max=19271, avg=4967.74, stdev=3780.38 lat (usec): min=2065, max=19272, avg=4968.51, stdev=3780.35 clat percentiles (usec): | 1.00th=[ 2114], 5.00th=[ 2245], 10.00th=[ 2376], 20.00th=[ 2573], | 30.00th=[ 2769], 40.00th=[ 3032], 50.00th=[ 3359], 60.00th=[ 3752], | 70.00th=[ 4555], 80.00th=[ 6063], 90.00th=[11600], 95.00th=[13698], | 99.00th=[18744], 99.50th=[19268], 99.90th=[19268], 99.95th=[19268], | 99.99th=[19268] bw ( KiB/s): min= 928, max= 2176, per=24.97%, avg=1251.00, stdev=426.80, samples=9 iops : min= 58, max= 136, avg=78.11, stdev=26.72, samples=9 write: IOPS=86, BW=1380KiB/s (1413kB/s)(6368KiB/4616msec) clat (usec): min=2820, max=24126, avg=6872.95, stdev=3881.68 lat (usec): min=2821, max=24127, avg=6874.35, stdev=3881.69 clat percentiles (usec): | 1.00th=[ 2966], 5.00th=[ 3326], 10.00th=[ 3556], 20.00th=[ 3949], | 30.00th=[ 4228], 40.00th=[ 4621], 50.00th=[ 5407], 60.00th=[ 6390], | 70.00th=[ 7832], 80.00th=[ 9372], 90.00th=[12911], 95.00th=[15795], | 99.00th=[20579], 99.50th=[23200], 99.90th=[24249], 99.95th=[24249], | 99.99th=[24249] bw ( KiB/s): min= 832, max= 2592, per=25.95%, avg=1368.22, stdev=516.91, samples=9 iops : min= 52, max= 162, avg=85.44, stdev=32.31, samples=9 lat (msec) : 4=41.15%, 10=43.10%, 20=15.23%, 50=0.52% cpu : usr=0.13%, sys=12.39%, ctx=910, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20243: Wed May 21 03:06:32 2025 read: IOPS=75, BW=1208KiB/s (1238kB/s)(5744KiB/4753msec) clat (usec): min=1973, max=20770, avg=4983.35, stdev=3752.62 lat (usec): min=1974, max=20771, avg=4984.08, stdev=3752.61 clat percentiles (usec): | 1.00th=[ 2008], 5.00th=[ 2245], 10.00th=[ 2409], 20.00th=[ 2573], | 30.00th=[ 2737], 40.00th=[ 2999], 50.00th=[ 3458], 60.00th=[ 3884], | 70.00th=[ 4883], 80.00th=[ 6718], 90.00th=[10159], 95.00th=[13960], | 99.00th=[19530], 99.50th=[20579], 99.90th=[20841], 99.95th=[20841], | 99.99th=[20841] bw ( KiB/s): min= 768, max= 2112, per=23.62%, avg=1183.56, stdev=485.01, samples=9 iops : min= 48, max= 132, avg=73.89, stdev=30.36, samples=9 write: IOPS=86, BW=1377KiB/s (1410kB/s)(6544KiB/4753msec) clat (usec): min=2790, max=34056, avg=7130.26, stdev=4428.35 lat (usec): min=2791, max=34057, avg=7131.58, stdev=4428.33 clat percentiles (usec): | 1.00th=[ 2999], 5.00th=[ 3294], 10.00th=[ 3556], 20.00th=[ 3851], | 30.00th=[ 4178], 40.00th=[ 4621], 50.00th=[ 5211], 60.00th=[ 6194], | 70.00th=[ 8094], 80.00th=[10159], 90.00th=[14222], 95.00th=[16712], | 99.00th=[21103], 99.50th=[22414], 99.90th=[33817], 99.95th=[33817], | 99.99th=[33817] bw ( KiB/s): min= 1019, max= 2592, per=25.07%, avg=1322.11, stdev=514.86, samples=9 iops : min= 63, max= 162, avg=82.56, stdev=32.23, samples=9 lat (msec) : 2=0.26%, 4=41.54%, 10=42.45%, 20=14.58%, 50=1.17% cpu : usr=0.08%, sys=12.04%, ctx=913, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20244: Wed May 21 03:06:32 2025 read: IOPS=82, BW=1314KiB/s (1345kB/s)(6256KiB/4762msec) clat (usec): min=1972, max=20805, avg=5075.22, stdev=3755.95 lat (usec): min=1973, max=20806, avg=5075.96, stdev=3755.95 clat percentiles (usec): | 1.00th=[ 2114], 5.00th=[ 2212], 10.00th=[ 2376], 20.00th=[ 2540], | 30.00th=[ 2802], 40.00th=[ 3097], 50.00th=[ 3490], 60.00th=[ 4113], | 70.00th=[ 5014], 80.00th=[ 6718], 90.00th=[10945], 95.00th=[14091], | 99.00th=[17695], 99.50th=[19006], 99.90th=[20841], 99.95th=[20841], | 99.99th=[20841] bw ( KiB/s): min= 768, max= 2496, per=25.67%, avg=1286.56, stdev=552.89, samples=9 iops : min= 48, max= 156, avg=80.33, stdev=34.60, samples=9 write: IOPS=79, BW=1267KiB/s (1297kB/s)(6032KiB/4762msec) clat (usec): min=2626, max=26649, avg=7280.77, stdev=4363.01 lat (usec): min=2628, max=26650, avg=7282.34, stdev=4363.22 clat percentiles (usec): | 1.00th=[ 2966], 5.00th=[ 3359], 10.00th=[ 3654], 20.00th=[ 4015], | 30.00th=[ 4293], 40.00th=[ 4948], 50.00th=[ 5669], 60.00th=[ 6587], | 70.00th=[ 8455], 80.00th=[10290], 90.00th=[13304], 95.00th=[17433], | 99.00th=[22938], 99.50th=[25035], 99.90th=[26608], 99.95th=[26608], | 99.99th=[26608] bw ( KiB/s): min= 864, max= 2304, per=23.12%, avg=1219.11, stdev=449.02, samples=9 iops : min= 54, max= 144, avg=76.11, stdev=28.13, samples=9 lat (msec) : 2=0.13%, 4=39.58%, 10=44.14%, 20=14.97%, 50=1.17% cpu : usr=0.17%, sys=12.12%, ctx=954, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20245: Wed May 21 03:06:32 2025 read: IOPS=78, BW=1262KiB/s (1292kB/s)(6032KiB/4779msec) clat (usec): min=1941, max=24216, avg=5298.92, stdev=4153.95 lat (usec): min=1941, max=24216, avg=5299.81, stdev=4153.88 clat percentiles (usec): | 1.00th=[ 1991], 5.00th=[ 2245], 10.00th=[ 2409], 20.00th=[ 2573], | 30.00th=[ 2737], 40.00th=[ 3064], 50.00th=[ 3458], 60.00th=[ 4047], | 70.00th=[ 4948], 80.00th=[ 7439], 90.00th=[12387], 95.00th=[14877], | 99.00th=[19006], 99.50th=[20579], 99.90th=[24249], 99.95th=[24249], | 99.99th=[24249] bw ( KiB/s): min= 864, max= 2464, per=24.04%, avg=1204.89, stdev=500.90, samples=9 iops : min= 54, max= 154, avg=75.22, stdev=31.36, samples=9 write: IOPS=81, BW=1309KiB/s (1340kB/s)(6256KiB/4779msec) clat (usec): min=2845, max=22800, avg=7078.22, stdev=4097.06 lat (usec): min=2845, max=22801, avg=7079.63, stdev=4097.12 clat percentiles (usec): | 1.00th=[ 2999], 5.00th=[ 3326], 10.00th=[ 3523], 20.00th=[ 3884], | 30.00th=[ 4293], 40.00th=[ 4686], 50.00th=[ 5276], 60.00th=[ 6456], | 70.00th=[ 8160], 80.00th=[10421], 90.00th=[13829], 95.00th=[15533], | 99.00th=[20317], 99.50th=[21103], 99.90th=[22676], 99.95th=[22676], | 99.99th=[22676] bw ( KiB/s): min= 896, max= 2400, per=24.13%, avg=1272.33, stdev=535.43, samples=9 iops : min= 56, max= 150, avg=79.44, stdev=33.50, samples=9 lat (msec) : 2=0.52%, 4=40.23%, 10=41.41%, 20=17.06%, 50=0.78% cpu : usr=0.17%, sys=11.85%, ctx=983, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=5012KiB/s (5132kB/s), 1208KiB/s-1314KiB/s (1238kB/s-1345kB/s), io=23.4MiB (24.5MB), run=4616-4779msec WRITE: bw=5273KiB/s (5400kB/s), 1267KiB/s-1380KiB/s (1297kB/s-1413kB/s), io=24.6MiB (25.8MB), run=4616-4779msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=20263: Wed May 21 03:06:34 2025 read: IOPS=11, BW=11.5MiB/s (12.0MB/s)(7168KiB/611msec) clat (usec): min=669, max=36255, avg=28873.26, stdev=12606.54 lat (usec): min=670, max=36256, avg=28876.26, stdev=12607.42 clat percentiles (usec): | 1.00th=[ 668], 5.00th=[ 668], 10.00th=[ 668], 20.00th=[30278], | 30.00th=[31589], 40.00th=[31589], 50.00th=[33424], 60.00th=[34341], | 70.00th=[34341], 80.00th=[35390], 90.00th=[36439], 95.00th=[36439], | 99.00th=[36439], 99.50th=[36439], 99.90th=[36439], 99.95th=[36439], | 99.99th=[36439] bw ( KiB/s): min=10240, max=10240, per=41.40%, avg=10240.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 write: IOPS=8, BW=8380KiB/s (8581kB/s)(5120KiB/611msec) clat (msec): min=44, max=164, avg=77.67, stdev=50.13 lat (msec): min=44, max=164, avg=77.83, stdev=50.11 clat percentiles (msec): | 1.00th=[ 44], 5.00th=[ 44], 10.00th=[ 44], 20.00th=[ 44], | 30.00th=[ 50], 40.00th=[ 50], 50.00th=[ 53], 60.00th=[ 53], | 70.00th=[ 79], 80.00th=[ 79], 90.00th=[ 165], 95.00th=[ 165], | 99.00th=[ 165], 99.50th=[ 165], 99.90th=[ 165], 99.95th=[ 165], | 99.99th=[ 165] bw ( KiB/s): min= 6144, max= 6144, per=17.74%, avg=6144.00, stdev= 0.00, samples=1 iops : min= 6, max= 6, avg= 6.00, stdev= 0.00, samples=1 lat (usec) : 750=8.33% lat (msec) : 50=66.67%, 100=16.67%, 250=8.33% cpu : usr=0.49%, sys=40.49%, ctx=805, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20264: Wed May 21 03:06:34 2025 read: IOPS=2, BW=2473KiB/s (2533kB/s)(2048KiB/828msec) clat (usec): min=35032, max=41634, avg=38333.53, stdev=4668.43 lat (usec): min=35033, max=41636, avg=38334.94, stdev=4669.31 clat percentiles (usec): | 1.00th=[34866], 5.00th=[34866], 10.00th=[34866], 20.00th=[34866], | 30.00th=[34866], 40.00th=[34866], 50.00th=[34866], 60.00th=[41681], | 70.00th=[41681], 80.00th=[41681], 90.00th=[41681], 95.00th=[41681], | 99.00th=[41681], 99.50th=[41681], 99.90th=[41681], 99.95th=[41681], | 99.99th=[41681] bw ( KiB/s): min= 2031, max= 2031, per=8.21%, avg=2031.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=12, BW=12.1MiB/s (12.7MB/s)(10.0MiB/828msec) clat (msec): min=19, max=292, avg=65.39, stdev=81.04 lat (msec): min=19, max=292, avg=65.46, stdev=81.05 clat percentiles (msec): | 1.00th=[ 20], 5.00th=[ 20], 10.00th=[ 20], 20.00th=[ 34], | 30.00th=[ 35], 40.00th=[ 35], 50.00th=[ 38], 60.00th=[ 40], | 70.00th=[ 46], 80.00th=[ 46], 90.00th=[ 72], 95.00th=[ 292], | 99.00th=[ 292], 99.50th=[ 292], 99.90th=[ 292], 99.95th=[ 292], | 99.99th=[ 292] bw ( KiB/s): min= 6095, max= 6095, per=17.60%, avg=6095.00, stdev= 0.00, samples=1 iops : min= 5, max= 5, avg= 5.00, stdev= 0.00, samples=1 lat (msec) : 20=8.33%, 50=75.00%, 100=8.33%, 500=8.33% cpu : usr=0.00%, sys=32.29%, ctx=430, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20265: Wed May 21 03:06:34 2025 read: IOPS=10, BW=10.1MiB/s (10.6MB/s)(6144KiB/595msec) clat (usec): min=369, max=46490, avg=16805.54, stdev=18709.33 lat (usec): min=371, max=46491, avg=16807.04, stdev=18709.36 clat percentiles (usec): | 1.00th=[ 371], 5.00th=[ 371], 10.00th=[ 371], 20.00th=[ 1696], | 30.00th=[ 1696], 40.00th=[ 2114], 50.00th=[ 2114], 60.00th=[22152], | 70.00th=[28181], 80.00th=[28181], 90.00th=[46400], 95.00th=[46400], | 99.00th=[46400], 99.50th=[46400], 99.90th=[46400], 99.95th=[46400], | 99.99th=[46400] bw ( KiB/s): min= 8126, max= 8126, per=32.85%, avg=8126.00, stdev= 0.00, samples=1 iops : min= 7, max= 7, avg= 7.00, stdev= 0.00, samples=1 write: IOPS=10, BW=10.1MiB/s (10.6MB/s)(6144KiB/595msec) clat (msec): min=29, max=153, avg=66.11, stdev=46.11 lat (msec): min=29, max=153, avg=66.23, stdev=46.28 clat percentiles (msec): | 1.00th=[ 30], 5.00th=[ 30], 10.00th=[ 30], 20.00th=[ 30], | 30.00th=[ 30], 40.00th=[ 52], 50.00th=[ 52], 60.00th=[ 58], | 70.00th=[ 77], 80.00th=[ 77], 90.00th=[ 153], 95.00th=[ 153], | 99.00th=[ 153], 99.50th=[ 153], 99.90th=[ 153], 99.95th=[ 153], | 99.99th=[ 153] bw ( KiB/s): min= 8126, max= 8126, per=23.47%, avg=8126.00, stdev= 0.00, samples=1 iops : min= 7, max= 7, avg= 7.00, stdev= 0.00, samples=1 lat (usec) : 500=8.33% lat (msec) : 2=8.33%, 4=8.33%, 50=41.67%, 100=25.00%, 250=8.33% cpu : usr=0.00%, sys=32.66%, ctx=83, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20266: Wed May 21 03:06:34 2025 read: IOPS=7, BW=7399KiB/s (7576kB/s)(5120KiB/692msec) clat (usec): min=308, max=55386, avg=23961.60, stdev=23616.40 lat (usec): min=309, max=55387, avg=23962.57, stdev=23616.21 clat percentiles (usec): | 1.00th=[ 310], 5.00th=[ 310], 10.00th=[ 310], 20.00th=[ 310], | 30.00th=[ 562], 40.00th=[ 562], 50.00th=[29230], 60.00th=[29230], | 70.00th=[34341], 80.00th=[34341], 90.00th=[55313], 95.00th=[55313], | 99.00th=[55313], 99.50th=[55313], 99.90th=[55313], 99.95th=[55313], | 99.99th=[55313] bw ( KiB/s): min= 6095, max= 6095, per=24.64%, avg=6095.00, stdev= 0.00, samples=1 iops : min= 5, max= 5, avg= 5.00, stdev= 0.00, samples=1 write: IOPS=10, BW=10.1MiB/s (10.6MB/s)(7168KiB/692msec) clat (msec): min=37, max=188, avg=69.21, stdev=55.03 lat (msec): min=37, max=188, avg=69.27, stdev=55.04 clat percentiles (msec): | 1.00th=[ 38], 5.00th=[ 38], 10.00th=[ 38], 20.00th=[ 40], | 30.00th=[ 41], 40.00th=[ 41], 50.00th=[ 43], 60.00th=[ 54], | 70.00th=[ 54], 80.00th=[ 83], 90.00th=[ 190], 95.00th=[ 190], | 99.00th=[ 190], 99.50th=[ 190], 99.90th=[ 190], 99.95th=[ 190], | 99.99th=[ 190] bw ( KiB/s): min= 8126, max= 8126, per=23.47%, avg=8126.00, stdev= 0.00, samples=1 iops : min= 7, max= 7, avg= 7.00, stdev= 0.00, samples=1 lat (usec) : 500=8.33%, 750=8.33% lat (msec) : 50=50.00%, 100=25.00%, 250=8.33% cpu : usr=0.00%, sys=33.86%, ctx=190, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=24.2MiB/s (25.3MB/s), 2473KiB/s-11.5MiB/s (2533kB/s-12.0MB/s), io=20.0MiB (20.0MB), run=595-828msec WRITE: bw=33.8MiB/s (35.5MB/s), 8380KiB/s-12.1MiB/s (8581kB/s-12.7MB/s), io=28.0MiB (29.4MB), run=595-828msec rand-rw: (groupid=0, jobs=1): err= 0: pid=20259: Wed May 21 03:06:34 2025 read: IOPS=11, BW=11.5MiB/s (12.1MB/s)(7168KiB/609msec) clat (usec): min=18913, max=82675, avg=39149.58, stdev=23063.94 lat (usec): min=18914, max=82678, avg=39151.56, stdev=23064.11 clat percentiles (usec): | 1.00th=[19006], 5.00th=[19006], 10.00th=[19006], 20.00th=[22938], | 30.00th=[25560], 40.00th=[25560], 50.00th=[26346], 60.00th=[42206], | 70.00th=[42206], 80.00th=[55313], 90.00th=[82314], 95.00th=[82314], | 99.00th=[82314], 99.50th=[82314], 99.90th=[82314], 99.95th=[82314], | 99.99th=[82314] bw ( KiB/s): min=10178, max=10178, per=44.83%, avg=10178.00, stdev= 0.00, samples=1 iops : min= 9, max= 9, avg= 9.00, stdev= 0.00, samples=1 write: IOPS=8, BW=8407KiB/s (8609kB/s)(5120KiB/609msec) clat (msec): min=32, max=128, avg=64.88, stdev=41.13 lat (msec): min=32, max=128, avg=64.97, stdev=41.14 clat percentiles (msec): | 1.00th=[ 33], 5.00th=[ 33], 10.00th=[ 33], 20.00th=[ 33], | 30.00th=[ 36], 40.00th=[ 36], 50.00th=[ 45], 60.00th=[ 45], | 70.00th=[ 85], 80.00th=[ 85], 90.00th=[ 129], 95.00th=[ 129], | 99.00th=[ 129], 99.50th=[ 129], 99.90th=[ 129], 99.95th=[ 129], | 99.99th=[ 129] bw ( KiB/s): min= 8143, max= 8143, per=25.62%, avg=8143.00, stdev= 0.00, samples=1 iops : min= 7, max= 7, avg= 7.00, stdev= 0.00, samples=1 lat (msec) : 20=8.33%, 50=58.33%, 100=25.00%, 250=8.33% cpu : usr=0.00%, sys=13.32%, ctx=57, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20260: Wed May 21 03:06:34 2025 read: IOPS=2, BW=2532KiB/s (2592kB/s)(2048KiB/809msec) clat (msec): min=18, max=114, avg=66.57, stdev=67.48 lat (msec): min=18, max=114, avg=66.57, stdev=67.48 clat percentiles (msec): | 1.00th=[ 19], 5.00th=[ 19], 10.00th=[ 19], 20.00th=[ 19], | 30.00th=[ 19], 40.00th=[ 19], 50.00th=[ 19], 60.00th=[ 114], | 70.00th=[ 114], 80.00th=[ 114], 90.00th=[ 114], 95.00th=[ 114], | 99.00th=[ 114], 99.50th=[ 114], 99.90th=[ 114], 99.95th=[ 114], | 99.99th=[ 114] bw ( KiB/s): min= 2048, max= 2048, per=9.02%, avg=2048.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=12, BW=12.4MiB/s (12.0MB/s)(10.0MiB/809msec) clat (msec): min=18, max=240, avg=65.09, stdev=65.73 lat (msec): min=18, max=240, avg=65.18, stdev=65.73 clat percentiles (msec): | 1.00th=[ 19], 5.00th=[ 19], 10.00th=[ 19], 20.00th=[ 23], | 30.00th=[ 29], 40.00th=[ 39], 50.00th=[ 44], 60.00th=[ 51], | 70.00th=[ 56], 80.00th=[ 56], 90.00th=[ 100], 95.00th=[ 241], | 99.00th=[ 241], 99.50th=[ 241], 99.90th=[ 241], 99.95th=[ 241], | 99.99th=[ 241] bw ( KiB/s): min=10240, max=10240, per=32.21%, avg=10240.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 lat (msec) : 20=16.67%, 50=33.33%, 100=33.33%, 250=16.67% cpu : usr=0.12%, sys=13.61%, ctx=65, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20261: Wed May 21 03:06:34 2025 read: IOPS=6, BW=7070KiB/s (7240kB/s)(6144KiB/869msec) clat (usec): min=27536, max=87818, avg=57279.40, stdev=27190.67 lat (usec): min=27537, max=87819, avg=57280.76, stdev=27190.64 clat percentiles (usec): | 1.00th=[27657], 5.00th=[27657], 10.00th=[27657], 20.00th=[28705], | 30.00th=[28705], 40.00th=[43779], 50.00th=[43779], 60.00th=[74974], | 70.00th=[81265], 80.00th=[81265], 90.00th=[87557], 95.00th=[87557], | 99.00th=[87557], 99.50th=[87557], 99.90th=[87557], 99.95th=[87557], | 99.99th=[87557] bw ( KiB/s): min= 4096, max= 4096, per=18.04%, avg=4096.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 write: IOPS=6, BW=7070KiB/s (7240kB/s)(6144KiB/869msec) clat (msec): min=17, max=189, avg=83.74, stdev=66.44 lat (msec): min=17, max=189, avg=83.81, stdev=66.42 clat percentiles (msec): | 1.00th=[ 18], 5.00th=[ 18], 10.00th=[ 18], 20.00th=[ 32], | 30.00th=[ 32], 40.00th=[ 61], 50.00th=[ 61], 60.00th=[ 66], | 70.00th=[ 140], 80.00th=[ 140], 90.00th=[ 190], 95.00th=[ 190], | 99.00th=[ 190], 99.50th=[ 190], 99.90th=[ 190], 99.95th=[ 190], | 99.99th=[ 190] bw ( KiB/s): min= 6144, max= 6144, per=19.33%, avg=6144.00, stdev= 0.00, samples=1 iops : min= 6, max= 6, avg= 6.00, stdev= 0.00, samples=1 lat (msec) : 20=8.33%, 50=33.33%, 100=41.67%, 250=16.67% cpu : usr=0.12%, sys=17.74%, ctx=68, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20262: Wed May 21 03:06:34 2025 read: IOPS=5, BW=5676KiB/s (5813kB/s)(5120KiB/902msec) clat (usec): min=15619, max=43027, avg=31999.86, stdev=10592.40 lat (usec): min=15620, max=43029, avg=32001.23, stdev=10592.65 clat percentiles (usec): | 1.00th=[15664], 5.00th=[15664], 10.00th=[15664], 20.00th=[15664], | 30.00th=[30540], 40.00th=[30540], 50.00th=[31327], 60.00th=[31327], | 70.00th=[39584], 80.00th=[39584], 90.00th=[43254], 95.00th=[43254], | 99.00th=[43254], 99.50th=[43254], 99.90th=[43254], 99.95th=[43254], | 99.99th=[43254] write: IOPS=7, BW=7947KiB/s (8138kB/s)(7168KiB/902msec) clat (msec): min=16, max=484, avg=103.01, stdev=171.26 lat (msec): min=16, max=484, avg=103.10, stdev=171.26 clat percentiles (msec): | 1.00th=[ 17], 5.00th=[ 17], 10.00th=[ 17], 20.00th=[ 23], | 30.00th=[ 25], 40.00th=[ 25], 50.00th=[ 27], 60.00th=[ 37], | 70.00th=[ 37], 80.00th=[ 110], 90.00th=[ 485], 95.00th=[ 485], | 99.00th=[ 485], 99.50th=[ 485], 99.90th=[ 485], 99.95th=[ 485], | 99.99th=[ 485] bw ( KiB/s): min= 4087, max= 4087, per=12.86%, avg=4087.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 lat (msec) : 20=16.67%, 50=66.67%, 250=8.33%, 500=8.33% cpu : usr=0.22%, sys=12.21%, ctx=71, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=22.2MiB/s (23.2MB/s), 2532KiB/s-11.5MiB/s (2592kB/s-12.1MB/s), io=20.0MiB (20.0MB), run=609-902msec WRITE: bw=31.0MiB/s (32.6MB/s), 7070KiB/s-12.4MiB/s (7240kB/s-12.0MB/s), io=28.0MiB (29.4MB), run=609-902msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=20274: Wed May 21 03:06:36 2025 read: IOPS=7, BW=29.4MiB/s (30.8MB/s)(12.0MiB/408msec) clat (msec): min=8, max=235, avg=124.83, stdev=113.62 lat (msec): min=8, max=235, avg=124.83, stdev=113.62 clat percentiles (msec): | 1.00th=[ 9], 5.00th=[ 9], 10.00th=[ 9], 20.00th=[ 9], | 30.00th=[ 9], 40.00th=[ 130], 50.00th=[ 130], 60.00th=[ 130], | 70.00th=[ 236], 80.00th=[ 236], 90.00th=[ 236], 95.00th=[ 236], | 99.00th=[ 236], 99.50th=[ 236], 99.90th=[ 236], 99.95th=[ 236], | 99.99th=[ 236] lat (msec) : 10=33.33%, 250=66.67% cpu : usr=0.00%, sys=29.24%, ctx=997, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20275: Wed May 21 03:06:36 2025 read: IOPS=1, BW=5038KiB/s (5159kB/s)(4096KiB/813msec) clat (nsec): min=116296k, max=116296k, avg=116295850.00, stdev= 0.00 lat (nsec): min=116298k, max=116298k, avg=116298447.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 116], 5.00th=[ 116], 10.00th=[ 116], 20.00th=[ 116], | 30.00th=[ 116], 40.00th=[ 116], 50.00th=[ 116], 60.00th=[ 116], | 70.00th=[ 116], 80.00th=[ 116], 90.00th=[ 116], 95.00th=[ 116], | 99.00th=[ 116], 99.50th=[ 116], 99.90th=[ 116], 99.95th=[ 116], | 99.99th=[ 116] bw ( KiB/s): min= 8159, max= 8159, per=40.49%, avg=8159.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=2, BW=9.84MiB/s (10.3MB/s)(8192KiB/813msec) clat (msec): min=286, max=354, avg=320.65, stdev=47.73 lat (msec): min=287, max=354, avg=320.90, stdev=47.73 clat percentiles (msec): | 1.00th=[ 288], 5.00th=[ 288], 10.00th=[ 288], 20.00th=[ 288], | 30.00th=[ 288], 40.00th=[ 288], 50.00th=[ 288], 60.00th=[ 355], | 70.00th=[ 355], 80.00th=[ 355], 90.00th=[ 355], 95.00th=[ 355], | 99.00th=[ 355], 99.50th=[ 355], 99.90th=[ 355], 99.95th=[ 355], | 99.99th=[ 355] bw ( KiB/s): min= 8159, max= 8159, per=20.24%, avg=8159.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=66.67% cpu : usr=0.00%, sys=25.86%, ctx=514, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20277: Wed May 21 03:06:36 2025 write: IOPS=4, BW=16.9MiB/s (17.8MB/s)(12.0MiB/708msec) clat (msec): min=164, max=322, avg=232.11, stdev=81.25 lat (msec): min=164, max=322, avg=232.44, stdev=81.22 clat percentiles (msec): | 1.00th=[ 165], 5.00th=[ 165], 10.00th=[ 165], 20.00th=[ 165], | 30.00th=[ 165], 40.00th=[ 211], 50.00th=[ 211], 60.00th=[ 211], | 70.00th=[ 321], 80.00th=[ 321], 90.00th=[ 321], 95.00th=[ 321], | 99.00th=[ 321], 99.50th=[ 321], 99.90th=[ 321], 99.95th=[ 321], | 99.99th=[ 321] bw ( KiB/s): min= 8126, max= 8126, per=20.16%, avg=8126.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.00%, sys=46.25%, ctx=30, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20279: Wed May 21 03:06:36 2025 write: IOPS=3, BW=14.8MiB/s (15.5MB/s)(12.0MiB/811msec) clat (msec): min=103, max=382, avg=260.16, stdev=142.94 lat (msec): min=103, max=383, avg=260.43, stdev=142.96 clat percentiles (msec): | 1.00th=[ 104], 5.00th=[ 104], 10.00th=[ 104], 20.00th=[ 104], | 30.00th=[ 104], 40.00th=[ 296], 50.00th=[ 296], 60.00th=[ 296], | 70.00th=[ 384], 80.00th=[ 384], 90.00th=[ 384], 95.00th=[ 384], | 99.00th=[ 384], 99.50th=[ 384], 99.90th=[ 384], 99.95th=[ 384], | 99.99th=[ 384] bw ( KiB/s): min= 8192, max= 8192, per=20.33%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=66.67% cpu : usr=0.12%, sys=40.37%, ctx=35, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=19.7MiB/s (20.6MB/s), 5038KiB/s-29.4MiB/s (5159kB/s-30.8MB/s), io=16.0MiB (16.8MB), run=408-813msec WRITE: bw=39.4MiB/s (41.3MB/s), 9.84MiB/s-16.9MiB/s (10.3MB/s-17.8MB/s), io=32.0MiB (33.6MB), run=708-813msec rand-rw: (groupid=0, jobs=1): err= 0: pid=20273: Wed May 21 03:06:36 2025 read: IOPS=5, BW=20.0MiB/s (20.0MB/s)(12.0MiB/600msec) clat (msec): min=71, max=292, avg=197.70, stdev=113.74 lat (msec): min=71, max=292, avg=197.70, stdev=113.74 clat percentiles (msec): | 1.00th=[ 72], 5.00th=[ 72], 10.00th=[ 72], 20.00th=[ 72], | 30.00th=[ 72], 40.00th=[ 228], 50.00th=[ 228], 60.00th=[ 228], | 70.00th=[ 292], 80.00th=[ 292], 90.00th=[ 292], 95.00th=[ 292], | 99.00th=[ 292], 99.50th=[ 292], 99.90th=[ 292], 99.95th=[ 292], | 99.99th=[ 292] lat (msec) : 100=33.33%, 250=33.33%, 500=33.33% cpu : usr=0.17%, sys=17.36%, ctx=62, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20276: Wed May 21 03:06:36 2025 read: IOPS=2, BW=8569KiB/s (8775kB/s)(4096KiB/478msec) clat (nsec): min=54613k, max=54613k, avg=54613166.00, stdev= 0.00 lat (nsec): min=54616k, max=54616k, avg=54616206.00, stdev= 0.00 clat percentiles (usec): | 1.00th=[54789], 5.00th=[54789], 10.00th=[54789], 20.00th=[54789], | 30.00th=[54789], 40.00th=[54789], 50.00th=[54789], 60.00th=[54789], | 70.00th=[54789], 80.00th=[54789], 90.00th=[54789], 95.00th=[54789], | 99.00th=[54789], 99.50th=[54789], 99.90th=[54789], 99.95th=[54789], | 99.99th=[54789] write: IOPS=4, BW=16.7MiB/s (17.5MB/s)(8192KiB/478msec) clat (msec): min=141, max=265, avg=203.86, stdev=87.81 lat (msec): min=142, max=266, avg=204.13, stdev=87.62 clat percentiles (msec): | 1.00th=[ 142], 5.00th=[ 142], 10.00th=[ 142], 20.00th=[ 142], | 30.00th=[ 142], 40.00th=[ 142], 50.00th=[ 142], 60.00th=[ 266], | 70.00th=[ 266], 80.00th=[ 266], 90.00th=[ 266], 95.00th=[ 266], | 99.00th=[ 266], 99.50th=[ 266], 99.90th=[ 266], 99.95th=[ 266], | 99.99th=[ 266] lat (msec) : 100=33.33%, 250=33.33%, 500=33.33% cpu : usr=0.00%, sys=20.34%, ctx=37, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20278: Wed May 21 03:06:36 2025 write: IOPS=2, BW=11.1MiB/s (11.6MB/s)(12.0MiB/1083msec) clat (msec): min=128, max=624, avg=353.38, stdev=251.59 lat (msec): min=128, max=625, avg=353.72, stdev=251.59 clat percentiles (msec): | 1.00th=[ 129], 5.00th=[ 129], 10.00th=[ 129], 20.00th=[ 129], | 30.00th=[ 129], 40.00th=[ 309], 50.00th=[ 309], 60.00th=[ 309], | 70.00th=[ 625], 80.00th=[ 625], 90.00th=[ 625], 95.00th=[ 625], | 99.00th=[ 625], 99.50th=[ 625], 99.90th=[ 625], 99.95th=[ 625], | 99.99th=[ 625] bw ( KiB/s): min= 5809, max= 5809, per=19.20%, avg=5809.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=33.33%, 750=33.33% cpu : usr=0.18%, sys=17.19%, ctx=75, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=20280: Wed May 21 03:06:36 2025 write: IOPS=7, BW=31.9MiB/s (33.5MB/s)(12.0MiB/376msec) clat (msec): min=43, max=198, avg=117.25, stdev=77.73 lat (msec): min=43, max=198, avg=117.65, stdev=77.69 clat percentiles (msec): | 1.00th=[ 44], 5.00th=[ 44], 10.00th=[ 44], 20.00th=[ 44], | 30.00th=[ 44], 40.00th=[ 111], 50.00th=[ 111], 60.00th=[ 111], | 70.00th=[ 199], 80.00th=[ 199], 90.00th=[ 199], 95.00th=[ 199], | 99.00th=[ 199], 99.50th=[ 199], 99.90th=[ 199], 99.95th=[ 199], | 99.99th=[ 199] lat (msec) : 50=33.33%, 250=66.67% cpu : usr=0.00%, sys=26.40%, ctx=90, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=26.7MiB/s (27.0MB/s), 8569KiB/s-20.0MiB/s (8775kB/s-20.0MB/s), io=16.0MiB (16.8MB), run=478-600msec WRITE: bw=29.5MiB/s (30.0MB/s), 11.1MiB/s-31.9MiB/s (11.6MB/s-33.5MB/s), io=32.0MiB (33.6MB), run=376-1083msec