== sanity test 398b: DIO and buffer IO race ============== 12:15:51 (1746634551) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.44233 s, 34.9 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=27969: Wed May 7 12:16:08 2025 read: IOPS=108, BW=433KiB/s (443kB/s)(5992KiB/13850msec) clat (usec): min=58, max=21282, avg=4970.27, stdev=2787.56 lat (usec): min=59, max=21282, avg=4971.89, stdev=2787.10 clat percentiles (usec): | 1.00th=[ 84], 5.00th=[ 1795], 10.00th=[ 2409], 20.00th=[ 2900], | 30.00th=[ 3326], 40.00th=[ 3818], 50.00th=[ 4490], 60.00th=[ 5211], | 70.00th=[ 5866], 80.00th=[ 6915], 90.00th=[ 8094], 95.00th=[ 9896], | 99.00th=[14746], 99.50th=[16581], 99.90th=[18220], 99.95th=[21365], | 99.99th=[21365] bw ( KiB/s): min= 216, max= 536, per=24.69%, avg=425.15, stdev=69.63, samples=27 iops : min= 54, max= 134, avg=106.26, stdev=17.44, samples=27 write: IOPS=113, BW=455KiB/s (465kB/s)(6296KiB/13850msec) clat (usec): min=442, max=204702, avg=4019.60, stdev=6069.31 lat (usec): min=443, max=204703, avg=4021.21, stdev=6069.34 clat percentiles (usec): | 1.00th=[ 510], 5.00th=[ 603], 10.00th=[ 676], 20.00th=[ 775], | 30.00th=[ 955], 40.00th=[ 2540], 50.00th=[ 3261], 60.00th=[ 3949], | 70.00th=[ 4883], 80.00th=[ 6587], 90.00th=[ 8848], 95.00th=[ 10421], | 99.00th=[ 13960], 99.50th=[ 14746], 99.90th=[ 26608], 99.95th=[204473], | 99.99th=[204473] bw ( KiB/s): min= 248, max= 608, per=24.68%, avg=444.96, stdev=78.50, samples=27 iops : min= 62, max= 152, avg=111.22, stdev=19.61, samples=27 lat (usec) : 100=1.30%, 250=0.88%, 500=0.46%, 750=8.69%, 1000=6.90% lat (msec) : 2=3.94%, 4=29.82%, 10=42.38%, 20=5.50%, 50=0.10% lat (msec) : 250=0.03% cpu : usr=0.30%, sys=22.49%, ctx=5836, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=27970: Wed May 7 12:16:08 2025 read: IOPS=111, BW=445KiB/s (456kB/s)(5940KiB/13336msec) clat (usec): min=56, max=20790, avg=4799.22, stdev=2735.66 lat (usec): min=57, max=20791, avg=4801.16, stdev=2736.69 clat percentiles (usec): | 1.00th=[ 74], 5.00th=[ 359], 10.00th=[ 2311], 20.00th=[ 2802], | 30.00th=[ 3228], 40.00th=[ 3720], 50.00th=[ 4293], 60.00th=[ 4948], | 70.00th=[ 5735], 80.00th=[ 6587], 90.00th=[ 7963], 95.00th=[ 9503], | 99.00th=[14877], 99.50th=[16581], 99.90th=[17957], 99.95th=[20841], | 99.99th=[20841] bw ( KiB/s): min= 184, max= 600, per=25.69%, avg=442.42, stdev=80.81, samples=26 iops : min= 46, max= 150, avg=110.58, stdev=20.22, samples=26 write: IOPS=119, BW=476KiB/s (487kB/s)(6348KiB/13336msec) clat (usec): min=453, max=209010, avg=3863.50, stdev=6160.05 lat (usec): min=453, max=209011, avg=3864.52, stdev=6159.94 clat percentiles (usec): | 1.00th=[ 510], 5.00th=[ 594], 10.00th=[ 660], 20.00th=[ 750], | 30.00th=[ 898], 40.00th=[ 1631], 50.00th=[ 2933], 60.00th=[ 3654], | 70.00th=[ 4686], 80.00th=[ 6521], 90.00th=[ 8717], 95.00th=[ 10814], | 99.00th=[ 13829], 99.50th=[ 15270], 99.90th=[ 17433], 99.95th=[208667], | 99.99th=[208667] bw ( KiB/s): min= 256, max= 584, per=25.95%, avg=467.96, stdev=72.08, samples=26 iops : min= 64, max= 146, avg=116.96, stdev=18.06, samples=26 lat (usec) : 100=1.30%, 250=1.07%, 500=0.52%, 750=9.73%, 1000=6.80% lat (msec) : 2=5.08%, 4=30.37%, 10=39.45%, 20=5.60%, 50=0.03% lat (msec) : 250=0.03% cpu : usr=0.21%, sys=23.17%, ctx=5772, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=27971: Wed May 7 12:16:08 2025 read: IOPS=110, BW=441KiB/s (452kB/s)(6028KiB/13664msec) clat (usec): min=58, max=21743, avg=4949.37, stdev=2736.61 lat (usec): min=58, max=21743, avg=4950.13, stdev=2736.68 clat percentiles (usec): | 1.00th=[ 79], 5.00th=[ 914], 10.00th=[ 2442], 20.00th=[ 2900], | 30.00th=[ 3392], 40.00th=[ 3884], 50.00th=[ 4490], 60.00th=[ 5145], | 70.00th=[ 5932], 80.00th=[ 6783], 90.00th=[ 8029], 95.00th=[ 9634], | 99.00th=[15008], 99.50th=[16581], 99.90th=[19792], 99.95th=[21627], | 99.99th=[21627] bw ( KiB/s): min= 168, max= 584, per=25.53%, avg=439.59, stdev=78.30, samples=27 iops : min= 42, max= 146, avg=109.81, stdev=19.55, samples=27 write: IOPS=114, BW=458KiB/s (469kB/s)(6260KiB/13664msec) clat (usec): min=411, max=206838, avg=3907.97, stdev=6130.64 lat (usec): min=412, max=206839, avg=3908.81, stdev=6130.67 clat percentiles (usec): | 1.00th=[ 486], 5.00th=[ 586], 10.00th=[ 652], 20.00th=[ 758], | 30.00th=[ 906], 40.00th=[ 2212], 50.00th=[ 3130], 60.00th=[ 3916], | 70.00th=[ 4883], 80.00th=[ 6259], 90.00th=[ 8225], 95.00th=[ 10421], | 99.00th=[ 15008], 99.50th=[ 17171], 99.90th=[ 24249], 99.95th=[206570], | 99.99th=[206570] bw ( KiB/s): min= 216, max= 616, per=25.25%, avg=455.30, stdev=84.84, samples=27 iops : min= 54, max= 154, avg=113.74, stdev=21.24, samples=27 lat (usec) : 100=1.37%, 250=0.98%, 500=0.72%, 750=9.28%, 1000=6.64% lat (msec) : 2=3.87%, 4=28.55%, 10=43.49%, 20=4.98%, 50=0.10% lat (msec) : 250=0.03% cpu : usr=0.15%, sys=22.70%, ctx=5786, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=27973: Wed May 7 12:16:08 2025 read: IOPS=108, BW=434KiB/s (445kB/s)(6052KiB/13939msec) clat (usec): min=58, max=21821, avg=4923.70, stdev=2653.80 lat (usec): min=59, max=21822, avg=4924.43, stdev=2653.80 clat percentiles (usec): | 1.00th=[ 92], 5.00th=[ 2114], 10.00th=[ 2474], 20.00th=[ 2933], | 30.00th=[ 3326], 40.00th=[ 3752], 50.00th=[ 4293], 60.00th=[ 5145], | 70.00th=[ 5866], 80.00th=[ 6783], 90.00th=[ 8160], 95.00th=[ 9634], | 99.00th=[14484], 99.50th=[16188], 99.90th=[20579], 99.95th=[21890], | 99.99th=[21890] bw ( KiB/s): min= 200, max= 512, per=24.60%, avg=423.67, stdev=68.73, samples=27 iops : min= 50, max= 128, avg=105.89, stdev=17.21, samples=27 write: IOPS=111, BW=447KiB/s (458kB/s)(6236KiB/13939msec) clat (usec): min=395, max=200916, avg=4101.72, stdev=6015.18 lat (usec): min=395, max=200917, avg=4103.95, stdev=6015.40 clat percentiles (usec): | 1.00th=[ 498], 5.00th=[ 627], 10.00th=[ 685], 20.00th=[ 783], | 30.00th=[ 1139], 40.00th=[ 2606], 50.00th=[ 3326], 60.00th=[ 4015], | 70.00th=[ 5014], 80.00th=[ 6587], 90.00th=[ 8717], 95.00th=[ 10814], | 99.00th=[ 14615], 99.50th=[ 16319], 99.90th=[ 22676], 99.95th=[200279], | 99.99th=[200279] bw ( KiB/s): min= 216, max= 688, per=24.30%, avg=438.19, stdev=93.54, samples=27 iops : min= 54, max= 172, avg=109.52, stdev=23.38, samples=27 lat (usec) : 100=0.85%, 250=0.88%, 500=0.52%, 750=8.07%, 1000=5.53% lat (msec) : 2=4.52%, 4=32.19%, 10=42.22%, 20=5.08%, 50=0.10% lat (msec) : 250=0.03% cpu : usr=0.22%, sys=22.64%, ctx=5684, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1723KiB/s (1764kB/s), 433KiB/s-445KiB/s (443kB/s-456kB/s), io=23.4MiB (24.6MB), run=13336-13939msec WRITE: bw=1804KiB/s (1847kB/s), 447KiB/s-476KiB/s (458kB/s-487kB/s), io=24.6MiB (25.7MB), run=13336-13939msec rand-rw: (groupid=0, jobs=1): err= 0: pid=27972: Wed May 7 12:16:12 2025 read: IOPS=84, BW=337KiB/s (345kB/s)(5992KiB/17794msec) clat (usec): min=1593, max=210321, avg=4780.89, stdev=6484.21 lat (usec): min=1593, max=210322, avg=4781.62, stdev=6484.27 clat percentiles (usec): | 1.00th=[ 1762], 5.00th=[ 1909], 10.00th=[ 2008], 20.00th=[ 2212], | 30.00th=[ 2442], 40.00th=[ 2737], 50.00th=[ 3163], 60.00th=[ 3687], | 70.00th=[ 4555], 80.00th=[ 6325], 90.00th=[ 10159], 95.00th=[ 12649], | 99.00th=[ 19268], 99.50th=[ 21890], 99.90th=[ 25297], 99.95th=[210764], | 99.99th=[210764] bw ( KiB/s): min= 111, max= 744, per=24.57%, avg=331.00, stdev=165.63, samples=35 iops : min= 27, max= 186, avg=82.63, stdev=41.45, samples=35 write: IOPS=88, BW=354KiB/s (362kB/s)(6296KiB/17794msec) clat (usec): min=2265, max=30522, avg=6700.66, stdev=4059.71 lat (usec): min=2265, max=30541, avg=6701.63, stdev=4059.81 clat percentiles (usec): | 1.00th=[ 2638], 5.00th=[ 2966], 10.00th=[ 3195], 20.00th=[ 3523], | 30.00th=[ 3884], 40.00th=[ 4359], 50.00th=[ 5014], 60.00th=[ 6063], | 70.00th=[ 7898], 80.00th=[ 9765], 90.00th=[12518], 95.00th=[15008], | 99.00th=[19792], 99.50th=[22676], 99.90th=[27132], 99.95th=[30540], | 99.99th=[30540] bw ( KiB/s): min= 127, max= 726, per=24.72%, avg=348.60, stdev=169.03, samples=35 iops : min= 31, max= 181, avg=87.03, stdev=42.31, samples=35 lat (msec) : 2=4.59%, 4=43.36%, 10=37.53%, 20=13.57%, 50=0.91% lat (msec) : 250=0.03% cpu : usr=0.19%, sys=11.38%, ctx=3674, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=27974: Wed May 7 12:16:12 2025 read: IOPS=83, BW=333KiB/s (341kB/s)(5940KiB/17825msec) clat (usec): min=1442, max=212271, avg=4693.80, stdev=6453.40 lat (usec): min=1443, max=212273, avg=4694.71, stdev=6453.40 clat percentiles (usec): | 1.00th=[ 1729], 5.00th=[ 1860], 10.00th=[ 1975], 20.00th=[ 2180], | 30.00th=[ 2409], 40.00th=[ 2704], 50.00th=[ 3032], 60.00th=[ 3621], | 70.00th=[ 4555], 80.00th=[ 6390], 90.00th=[ 9896], 95.00th=[ 12125], | 99.00th=[ 18220], 99.50th=[ 19006], 99.90th=[ 25297], 99.95th=[212861], | 99.99th=[212861] bw ( KiB/s): min= 80, max= 720, per=24.47%, avg=329.60, stdev=172.72, samples=35 iops : min= 20, max= 180, avg=82.23, stdev=43.23, samples=35 write: IOPS=89, BW=356KiB/s (365kB/s)(6348KiB/17825msec) clat (usec): min=2114, max=35695, avg=6795.73, stdev=4138.08 lat (usec): min=2114, max=35696, avg=6796.69, stdev=4138.14 clat percentiles (usec): | 1.00th=[ 2474], 5.00th=[ 2868], 10.00th=[ 3163], 20.00th=[ 3523], | 30.00th=[ 3916], 40.00th=[ 4424], 50.00th=[ 5211], 60.00th=[ 6390], | 70.00th=[ 8160], 80.00th=[10028], 90.00th=[12649], 95.00th=[14615], | 99.00th=[21103], 99.50th=[22414], 99.90th=[34341], 99.95th=[35914], | 99.99th=[35914] bw ( KiB/s): min= 152, max= 752, per=24.71%, avg=348.34, stdev=162.02, samples=35 iops : min= 38, max= 188, avg=86.91, stdev=40.57, samples=35 lat (msec) : 2=5.21%, 4=42.87%, 10=36.82%, 20=14.36%, 50=0.72% lat (msec) : 250=0.03% cpu : usr=0.22%, sys=11.46%, ctx=3720, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=27975: Wed May 7 12:16:12 2025 read: IOPS=85, BW=340KiB/s (348kB/s)(6028KiB/17716msec) clat (usec): min=1577, max=26787, avg=4657.41, stdev=3661.99 lat (usec): min=1578, max=26788, avg=4658.30, stdev=3661.91 clat percentiles (usec): | 1.00th=[ 1745], 5.00th=[ 1926], 10.00th=[ 2008], 20.00th=[ 2245], | 30.00th=[ 2442], 40.00th=[ 2769], 50.00th=[ 3163], 60.00th=[ 3687], | 70.00th=[ 4555], 80.00th=[ 6521], 90.00th=[10028], 95.00th=[12780], | 99.00th=[17695], 99.50th=[20317], 99.90th=[25035], 99.95th=[26870], | 99.99th=[26870] bw ( KiB/s): min= 104, max= 712, per=25.03%, avg=337.20, stdev=160.73, samples=35 iops : min= 26, max= 178, avg=84.20, stdev=40.19, samples=35 write: IOPS=88, BW=353KiB/s (362kB/s)(6260KiB/17716msec) clat (msec): min=2, max=217, avg= 6.79, stdev= 6.77 lat (msec): min=2, max=217, avg= 6.79, stdev= 6.77 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 4], 20.00th=[ 4], | 30.00th=[ 4], 40.00th=[ 5], 50.00th=[ 5], 60.00th=[ 6], | 70.00th=[ 8], 80.00th=[ 10], 90.00th=[ 13], 95.00th=[ 15], | 99.00th=[ 21], 99.50th=[ 25], 99.90th=[ 36], 99.95th=[ 218], | 99.99th=[ 218] bw ( KiB/s): min= 128, max= 728, per=24.76%, avg=349.09, stdev=166.70, samples=35 iops : min= 32, max= 182, avg=87.17, stdev=41.69, samples=35 lat (msec) : 2=4.79%, 4=43.33%, 10=37.01%, 20=14.06%, 50=0.78% lat (msec) : 250=0.03% cpu : usr=0.23%, sys=11.54%, ctx=3764, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=27976: Wed May 7 12:16:12 2025 read: IOPS=85, BW=342KiB/s (350kB/s)(6052KiB/17702msec) clat (usec): min=1539, max=30097, avg=4637.26, stdev=3655.00 lat (usec): min=1540, max=30098, avg=4638.07, stdev=3655.03 clat percentiles (usec): | 1.00th=[ 1745], 5.00th=[ 1926], 10.00th=[ 2057], 20.00th=[ 2278], | 30.00th=[ 2507], 40.00th=[ 2835], 50.00th=[ 3163], 60.00th=[ 3687], | 70.00th=[ 4490], 80.00th=[ 6128], 90.00th=[ 9896], 95.00th=[12649], | 99.00th=[18482], 99.50th=[21103], 99.90th=[24249], 99.95th=[30016], | 99.99th=[30016] bw ( KiB/s): min= 112, max= 720, per=25.14%, avg=338.60, stdev=159.56, samples=35 iops : min= 28, max= 180, avg=84.57, stdev=39.89, samples=35 write: IOPS=88, BW=352KiB/s (361kB/s)(6236KiB/17702msec) clat (msec): min=2, max=208, avg= 6.80, stdev= 6.49 lat (msec): min=2, max=208, avg= 6.80, stdev= 6.49 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 4], 20.00th=[ 4], | 30.00th=[ 4], 40.00th=[ 5], 50.00th=[ 6], 60.00th=[ 7], | 70.00th=[ 8], 80.00th=[ 10], 90.00th=[ 13], 95.00th=[ 15], | 99.00th=[ 22], 99.50th=[ 23], 99.90th=[ 32], 99.95th=[ 209], | 99.99th=[ 209] bw ( KiB/s): min= 128, max= 728, per=24.76%, avg=349.11, stdev=163.32, samples=35 iops : min= 32, max= 182, avg=87.20, stdev=40.85, samples=35 lat (msec) : 2=3.68%, 4=44.17%, 10=37.70%, 20=13.57%, 50=0.85% lat (msec) : 250=0.03% cpu : usr=0.26%, sys=11.38%, ctx=3704, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1347KiB/s (1379kB/s), 333KiB/s-342KiB/s (341kB/s-350kB/s), io=23.4MiB (24.6MB), run=17702-17825msec WRITE: bw=1410KiB/s (1444kB/s), 352KiB/s-356KiB/s (361kB/s-365kB/s), io=24.6MiB (25.7MB), run=17702-17825msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=28003: Wed May 7 12:16:16 2025 read: IOPS=106, BW=1707KiB/s (1748kB/s)(5920KiB/3468msec) clat (usec): min=51, max=17291, avg=4868.09, stdev=2792.02 lat (usec): min=52, max=17291, avg=4868.98, stdev=2792.43 clat percentiles (usec): | 1.00th=[ 70], 5.00th=[ 103], 10.00th=[ 2409], 20.00th=[ 2900], | 30.00th=[ 3425], 40.00th=[ 3851], 50.00th=[ 4293], 60.00th=[ 5080], | 70.00th=[ 5866], 80.00th=[ 6587], 90.00th=[ 8356], 95.00th=[10028], | 99.00th=[14222], 99.50th=[16057], 99.90th=[17171], 99.95th=[17171], | 99.99th=[17171] bw ( KiB/s): min= 1472, max= 1920, per=26.16%, avg=1653.33, stdev=184.01, samples=6 iops : min= 92, max= 120, avg=103.33, stdev=11.50, samples=6 write: IOPS=114, BW=1836KiB/s (1880kB/s)(6368KiB/3468msec) clat (usec): min=678, max=15827, avg=4135.45, stdev=3321.55 lat (usec): min=679, max=15828, avg=4137.73, stdev=3321.27 clat percentiles (usec): | 1.00th=[ 717], 5.00th=[ 865], 10.00th=[ 930], 20.00th=[ 1057], | 30.00th=[ 1319], 40.00th=[ 2278], 50.00th=[ 3359], 60.00th=[ 4228], | 70.00th=[ 5473], 80.00th=[ 7111], 90.00th=[ 8848], 95.00th=[10683], | 99.00th=[13698], 99.50th=[14484], 99.90th=[15795], 99.95th=[15795], | 99.99th=[15795] bw ( KiB/s): min= 1504, max= 2208, per=27.35%, avg=1818.67, stdev=251.49, samples=6 iops : min= 94, max= 138, avg=113.67, stdev=15.72, samples=6 lat (usec) : 100=2.21%, 250=1.43%, 750=0.78%, 1000=6.51% lat (msec) : 2=12.76%, 4=27.60%, 10=42.71%, 20=5.99% cpu : usr=0.20%, sys=25.41%, ctx=1675, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28005: Wed May 7 12:16:16 2025 read: IOPS=97, BW=1568KiB/s (1605kB/s)(5744KiB/3664msec) clat (usec): min=58, max=21137, avg=4974.29, stdev=2953.44 lat (usec): min=59, max=21138, avg=4975.31, stdev=2952.99 clat percentiles (usec): | 1.00th=[ 83], 5.00th=[ 103], 10.00th=[ 627], 20.00th=[ 3064], | 30.00th=[ 3523], 40.00th=[ 4113], 50.00th=[ 4555], 60.00th=[ 5342], | 70.00th=[ 6063], 80.00th=[ 7111], 90.00th=[ 8455], 95.00th=[ 9634], | 99.00th=[14746], 99.50th=[17695], 99.90th=[21103], 99.95th=[21103], | 99.99th=[21103] bw ( KiB/s): min= 896, max= 1792, per=24.44%, avg=1545.14, stdev=318.24, samples=7 iops : min= 56, max= 112, avg=96.57, stdev=19.89, samples=7 write: IOPS=111, BW=1786KiB/s (1829kB/s)(6544KiB/3664msec) clat (usec): min=568, max=22591, avg=4449.40, stdev=3469.41 lat (usec): min=569, max=22592, avg=4450.43, stdev=3469.45 clat percentiles (usec): | 1.00th=[ 742], 5.00th=[ 840], 10.00th=[ 971], 20.00th=[ 1156], | 30.00th=[ 1549], 40.00th=[ 2933], 50.00th=[ 3752], 60.00th=[ 4621], | 70.00th=[ 5800], 80.00th=[ 7177], 90.00th=[ 9372], 95.00th=[10945], | 99.00th=[14615], 99.50th=[15533], 99.90th=[22676], 99.95th=[22676], | 99.99th=[22676] bw ( KiB/s): min= 1152, max= 2272, per=26.26%, avg=1746.29, stdev=394.89, samples=7 iops : min= 72, max= 142, avg=109.14, stdev=24.68, samples=7 lat (usec) : 100=1.82%, 250=2.60%, 500=0.13%, 750=0.78%, 1000=6.12% lat (msec) : 2=11.20%, 4=24.09%, 10=47.27%, 20=5.73%, 50=0.26% cpu : usr=0.19%, sys=24.16%, ctx=1815, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28007: Wed May 7 12:16:16 2025 read: IOPS=103, BW=1651KiB/s (1691kB/s)(6256KiB/3789msec) clat (usec): min=60, max=19127, avg=5074.83, stdev=3044.87 lat (usec): min=61, max=19128, avg=5077.82, stdev=3043.82 clat percentiles (usec): | 1.00th=[ 72], 5.00th=[ 110], 10.00th=[ 2245], 20.00th=[ 2900], | 30.00th=[ 3392], 40.00th=[ 3916], 50.00th=[ 4555], 60.00th=[ 5538], | 70.00th=[ 6259], 80.00th=[ 7308], 90.00th=[ 8848], 95.00th=[10290], | 99.00th=[14877], 99.50th=[16909], 99.90th=[19006], 99.95th=[19006], | 99.99th=[19006] bw ( KiB/s): min= 800, max= 1888, per=24.73%, avg=1563.43, stdev=372.00, samples=7 iops : min= 50, max= 118, avg=97.71, stdev=23.25, samples=7 write: IOPS=99, BW=1592KiB/s (1630kB/s)(6032KiB/3789msec) clat (usec): min=687, max=18022, avg=4604.19, stdev=3529.00 lat (usec): min=688, max=18024, avg=4605.39, stdev=3529.22 clat percentiles (usec): | 1.00th=[ 775], 5.00th=[ 898], 10.00th=[ 1004], 20.00th=[ 1221], | 30.00th=[ 2147], 40.00th=[ 2999], 50.00th=[ 3621], 60.00th=[ 4490], | 70.00th=[ 5735], 80.00th=[ 7439], 90.00th=[10028], 95.00th=[11994], | 99.00th=[15270], 99.50th=[15401], 99.90th=[17957], 99.95th=[17957], | 99.99th=[17957] bw ( KiB/s): min= 1184, max= 1760, per=22.96%, avg=1526.86, stdev=191.75, samples=7 iops : min= 74, max= 110, avg=95.43, stdev=11.98, samples=7 lat (usec) : 100=1.82%, 250=2.60%, 500=0.13%, 750=0.26%, 1000=4.30% lat (msec) : 2=10.29%, 4=29.82%, 10=42.71%, 20=8.07% cpu : usr=0.37%, sys=23.94%, ctx=1772, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28009: Wed May 7 12:16:16 2025 read: IOPS=105, BW=1684KiB/s (1724kB/s)(6032KiB/3583msec) clat (usec): min=53, max=22725, avg=5295.32, stdev=3022.07 lat (usec): min=53, max=22725, avg=5296.29, stdev=3022.33 clat percentiles (usec): | 1.00th=[ 68], 5.00th=[ 106], 10.00th=[ 2474], 20.00th=[ 3261], | 30.00th=[ 3818], 40.00th=[ 4293], 50.00th=[ 5014], 60.00th=[ 5669], | 70.00th=[ 6390], 80.00th=[ 7111], 90.00th=[ 8586], 95.00th=[10159], | 99.00th=[16057], 99.50th=[18482], 99.90th=[22676], 99.95th=[22676], | 99.99th=[22676] bw ( KiB/s): min= 1216, max= 1920, per=26.47%, avg=1673.14, stdev=248.36, samples=7 iops : min= 76, max= 120, avg=104.57, stdev=15.52, samples=7 write: IOPS=109, BW=1746KiB/s (1788kB/s)(6256KiB/3583msec) clat (usec): min=642, max=18537, avg=3923.94, stdev=3267.34 lat (usec): min=643, max=18538, avg=3925.64, stdev=3266.65 clat percentiles (usec): | 1.00th=[ 701], 5.00th=[ 824], 10.00th=[ 914], 20.00th=[ 1074], | 30.00th=[ 1319], 40.00th=[ 2114], 50.00th=[ 3130], 60.00th=[ 3982], | 70.00th=[ 5014], 80.00th=[ 6194], 90.00th=[ 8455], 95.00th=[10028], | 99.00th=[14877], 99.50th=[17957], 99.90th=[18482], 99.95th=[18482], | 99.99th=[18482] bw ( KiB/s): min= 1440, max= 2240, per=26.26%, avg=1746.29, stdev=253.22, samples=7 iops : min= 90, max= 140, avg=109.14, stdev=15.83, samples=7 lat (usec) : 100=1.69%, 250=2.08%, 500=0.26%, 750=1.17%, 1000=7.68% lat (msec) : 2=11.07%, 4=23.18%, 10=47.66%, 20=5.08%, 50=0.13% cpu : usr=0.22%, sys=24.96%, ctx=1727, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=6321KiB/s (6473kB/s), 1568KiB/s-1707KiB/s (1605kB/s-1748kB/s), io=23.4MiB (24.5MB), run=3468-3789msec WRITE: bw=6651KiB/s (6810kB/s), 1592KiB/s-1836KiB/s (1630kB/s-1880kB/s), io=24.6MiB (25.8MB), run=3468-3789msec rand-rw: (groupid=0, jobs=1): err= 0: pid=28004: Wed May 7 12:16:17 2025 read: IOPS=82, BW=1314KiB/s (1346kB/s)(5920KiB/4504msec) clat (usec): min=1583, max=17771, avg=4808.99, stdev=3354.42 lat (usec): min=1583, max=17772, avg=4809.63, stdev=3354.44 clat percentiles (usec): | 1.00th=[ 1795], 5.00th=[ 1975], 10.00th=[ 2114], 20.00th=[ 2474], | 30.00th=[ 2737], 40.00th=[ 2999], 50.00th=[ 3294], 60.00th=[ 4015], | 70.00th=[ 5080], 80.00th=[ 6783], 90.00th=[ 9896], 95.00th=[12387], | 99.00th=[17171], 99.50th=[17433], 99.90th=[17695], 99.95th=[17695], | 99.99th=[17695] bw ( KiB/s): min= 862, max= 2624, per=24.87%, avg=1315.33, stdev=568.02, samples=9 iops : min= 53, max= 164, avg=82.11, stdev=35.59, samples=9 write: IOPS=88, BW=1414KiB/s (1448kB/s)(6368KiB/4504msec) clat (usec): min=2502, max=24789, avg=6738.68, stdev=3669.12 lat (usec): min=2503, max=24791, avg=6739.94, stdev=3669.16 clat percentiles (usec): | 1.00th=[ 2769], 5.00th=[ 2966], 10.00th=[ 3228], 20.00th=[ 3654], | 30.00th=[ 4146], 40.00th=[ 4621], 50.00th=[ 5342], 60.00th=[ 6718], | 70.00th=[ 7963], 80.00th=[ 9634], 90.00th=[11863], 95.00th=[13960], | 99.00th=[19268], 99.50th=[20841], 99.90th=[24773], 99.95th=[24773], | 99.99th=[24773] bw ( KiB/s): min= 960, max= 2752, per=25.36%, avg=1411.22, stdev=558.01, samples=9 iops : min= 60, max= 172, avg=88.11, stdev=34.93, samples=9 lat (msec) : 2=2.47%, 4=40.10%, 10=42.97%, 20=14.06%, 50=0.39% cpu : usr=0.18%, sys=12.21%, ctx=933, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28006: Wed May 7 12:16:17 2025 read: IOPS=79, BW=1279KiB/s (1309kB/s)(5744KiB/4492msec) clat (usec): min=1778, max=17392, avg=4722.15, stdev=3213.53 lat (usec): min=1778, max=17393, avg=4722.83, stdev=3213.56 clat percentiles (usec): | 1.00th=[ 1860], 5.00th=[ 2057], 10.00th=[ 2180], 20.00th=[ 2474], | 30.00th=[ 2737], 40.00th=[ 3163], 50.00th=[ 3556], 60.00th=[ 3916], | 70.00th=[ 4817], 80.00th=[ 6259], 90.00th=[ 9765], 95.00th=[12649], | 99.00th=[16057], 99.50th=[16581], 99.90th=[17433], 99.95th=[17433], | 99.99th=[17433] bw ( KiB/s): min= 800, max= 1920, per=21.93%, avg=1159.62, stdev=350.79, samples=8 iops : min= 50, max= 120, avg=72.38, stdev=21.91, samples=8 write: IOPS=91, BW=1457KiB/s (1492kB/s)(6544KiB/4492msec) clat (usec): min=2323, max=28738, avg=6756.98, stdev=4133.75 lat (usec): min=2324, max=28739, avg=6758.27, stdev=4133.82 clat percentiles (usec): | 1.00th=[ 2474], 5.00th=[ 2966], 10.00th=[ 3195], 20.00th=[ 3556], | 30.00th=[ 3982], 40.00th=[ 4621], 50.00th=[ 5276], 60.00th=[ 6390], | 70.00th=[ 7767], 80.00th=[ 9503], 90.00th=[12256], 95.00th=[14484], | 99.00th=[22152], 99.50th=[22676], 99.90th=[28705], 99.95th=[28705], | 99.99th=[28705] bw ( KiB/s): min= 928, max= 1664, per=22.49%, avg=1251.62, stdev=219.95, samples=8 iops : min= 58, max= 104, avg=78.12, stdev=13.74, samples=8 lat (msec) : 2=1.82%, 4=42.84%, 10=41.15%, 20=13.28%, 50=0.91% cpu : usr=0.27%, sys=12.09%, ctx=928, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28008: Wed May 7 12:16:17 2025 read: IOPS=86, BW=1382KiB/s (1415kB/s)(6256KiB/4528msec) clat (usec): min=1798, max=24443, avg=4906.07, stdev=3617.18 lat (usec): min=1798, max=24443, avg=4906.89, stdev=3617.13 clat percentiles (usec): | 1.00th=[ 1893], 5.00th=[ 2024], 10.00th=[ 2212], 20.00th=[ 2409], | 30.00th=[ 2606], 40.00th=[ 2966], 50.00th=[ 3359], 60.00th=[ 3884], | 70.00th=[ 5014], 80.00th=[ 7111], 90.00th=[10683], 95.00th=[13173], | 99.00th=[16057], 99.50th=[21890], 99.90th=[24511], 99.95th=[24511], | 99.99th=[24511] bw ( KiB/s): min= 832, max= 3072, per=26.22%, avg=1386.67, stdev=707.81, samples=9 iops : min= 52, max= 192, avg=86.67, stdev=44.24, samples=9 write: IOPS=83, BW=1332KiB/s (1364kB/s)(6032KiB/4528msec) clat (usec): min=2311, max=29485, avg=6891.75, stdev=4177.23 lat (usec): min=2312, max=29486, avg=6893.04, stdev=4177.28 clat percentiles (usec): | 1.00th=[ 2474], 5.00th=[ 2933], 10.00th=[ 3195], 20.00th=[ 3523], | 30.00th=[ 3982], 40.00th=[ 4555], 50.00th=[ 5276], 60.00th=[ 6390], | 70.00th=[ 8225], 80.00th=[10028], 90.00th=[12911], 95.00th=[14746], | 99.00th=[22152], 99.50th=[26608], 99.90th=[29492], 99.95th=[29492], | 99.99th=[29492] bw ( KiB/s): min= 832, max= 2496, per=23.64%, avg=1315.56, stdev=543.87, samples=9 iops : min= 52, max= 156, avg=82.22, stdev=33.99, samples=9 lat (msec) : 2=1.95%, 4=44.14%, 10=38.15%, 20=14.84%, 50=0.91% cpu : usr=0.09%, sys=12.17%, ctx=944, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28010: Wed May 7 12:16:17 2025 read: IOPS=83, BW=1335KiB/s (1367kB/s)(6032KiB/4520msec) clat (usec): min=1697, max=24681, avg=4929.32, stdev=3759.93 lat (usec): min=1698, max=24682, avg=4930.33, stdev=3759.71 clat percentiles (usec): | 1.00th=[ 1827], 5.00th=[ 1975], 10.00th=[ 2114], 20.00th=[ 2376], | 30.00th=[ 2606], 40.00th=[ 2966], 50.00th=[ 3359], 60.00th=[ 3916], | 70.00th=[ 4817], 80.00th=[ 6783], 90.00th=[11207], 95.00th=[13173], | 99.00th=[17171], 99.50th=[21890], 99.90th=[24773], 99.95th=[24773], | 99.99th=[24773] bw ( KiB/s): min= 768, max= 2816, per=25.01%, avg=1322.67, stdev=639.80, samples=9 iops : min= 48, max= 176, avg=82.67, stdev=39.99, samples=9 write: IOPS=86, BW=1384KiB/s (1417kB/s)(6256KiB/4520msec) clat (usec): min=2255, max=24897, avg=6735.18, stdev=3787.25 lat (usec): min=2256, max=24898, avg=6736.47, stdev=3787.30 clat percentiles (usec): | 1.00th=[ 2540], 5.00th=[ 2966], 10.00th=[ 3261], 20.00th=[ 3523], | 30.00th=[ 3982], 40.00th=[ 4555], 50.00th=[ 5145], 60.00th=[ 6456], | 70.00th=[ 8291], 80.00th=[ 9634], 90.00th=[12518], 95.00th=[14091], | 99.00th=[19268], 99.50th=[21365], 99.90th=[24773], 99.95th=[24773], | 99.99th=[24773] bw ( KiB/s): min= 928, max= 2656, per=24.85%, avg=1383.11, stdev=572.83, samples=9 iops : min= 58, max= 166, avg=86.44, stdev=35.80, samples=9 lat (msec) : 2=2.73%, 4=43.75%, 10=37.63%, 20=15.36%, 50=0.52% cpu : usr=0.13%, sys=12.02%, ctx=967, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=5290KiB/s (5417kB/s), 1279KiB/s-1382KiB/s (1309kB/s-1415kB/s), io=23.4MiB (24.5MB), run=4492-4528msec WRITE: bw=5565KiB/s (5699kB/s), 1332KiB/s-1457KiB/s (1364kB/s-1492kB/s), io=24.6MiB (25.8MB), run=4492-4528msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=28019: Wed May 7 12:16:19 2025 read: IOPS=8, BW=8795KiB/s (9006kB/s)(7168KiB/815msec) clat (msec): min=16, max=102, avg=57.53, stdev=30.67 lat (msec): min=16, max=102, avg=57.53, stdev=30.67 clat percentiles (msec): | 1.00th=[ 17], 5.00th=[ 17], 10.00th=[ 17], 20.00th=[ 21], | 30.00th=[ 55], 40.00th=[ 55], 50.00th=[ 65], 60.00th=[ 65], | 70.00th=[ 65], 80.00th=[ 80], 90.00th=[ 103], 95.00th=[ 103], | 99.00th=[ 103], 99.50th=[ 103], 99.90th=[ 103], 99.95th=[ 103], | 99.99th=[ 103] bw ( KiB/s): min=10240, max=10240, per=40.75%, avg=10240.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 write: IOPS=6, BW=6282KiB/s (6433kB/s)(5120KiB/815msec) clat (msec): min=35, max=113, avg=74.41, stdev=33.79 lat (msec): min=35, max=113, avg=74.49, stdev=33.80 clat percentiles (msec): | 1.00th=[ 36], 5.00th=[ 36], 10.00th=[ 36], 20.00th=[ 36], | 30.00th=[ 43], 40.00th=[ 43], 50.00th=[ 89], 60.00th=[ 89], | 70.00th=[ 93], 80.00th=[ 93], 90.00th=[ 114], 95.00th=[ 114], | 99.00th=[ 114], 99.50th=[ 114], 99.90th=[ 114], 99.95th=[ 114], | 99.99th=[ 114] bw ( KiB/s): min= 6144, max= 6144, per=17.46%, avg=6144.00, stdev= 0.00, samples=1 iops : min= 6, max= 6, avg= 6.00, stdev= 0.00, samples=1 lat (msec) : 20=8.33%, 50=25.00%, 100=50.00%, 250=16.67% cpu : usr=0.00%, sys=16.95%, ctx=78, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28020: Wed May 7 12:16:19 2025 read: IOPS=2, BW=2872KiB/s (2941kB/s)(2048KiB/713msec) clat (usec): min=14560, max=37822, avg=26191.31, stdev=16448.71 lat (usec): min=14562, max=37823, avg=26193.43, stdev=16448.04 clat percentiles (usec): | 1.00th=[14615], 5.00th=[14615], 10.00th=[14615], 20.00th=[14615], | 30.00th=[14615], 40.00th=[14615], 50.00th=[14615], 60.00th=[38011], | 70.00th=[38011], 80.00th=[38011], 90.00th=[38011], 95.00th=[38011], | 99.00th=[38011], 99.50th=[38011], 99.90th=[38011], 99.95th=[38011], | 99.99th=[38011] bw ( KiB/s): min= 4087, max= 4087, per=16.26%, avg=4087.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 write: IOPS=14, BW=14.0MiB/s (14.7MB/s)(10.0MiB/713msec) clat (msec): min=21, max=103, avg=65.20, stdev=26.72 lat (msec): min=21, max=103, avg=65.27, stdev=26.71 clat percentiles (msec): | 1.00th=[ 22], 5.00th=[ 22], 10.00th=[ 22], 20.00th=[ 50], | 30.00th=[ 54], 40.00th=[ 55], 50.00th=[ 57], 60.00th=[ 58], | 70.00th=[ 100], 80.00th=[ 100], 90.00th=[ 101], 95.00th=[ 104], | 99.00th=[ 104], 99.50th=[ 104], 99.90th=[ 104], 99.95th=[ 104], | 99.99th=[ 104] bw ( KiB/s): min=12263, max=12263, per=34.86%, avg=12263.00, stdev= 0.00, samples=1 iops : min= 11, max= 11, avg=11.00, stdev= 0.00, samples=1 lat (msec) : 20=8.33%, 50=25.00%, 100=58.33%, 250=8.33% cpu : usr=0.00%, sys=14.04%, ctx=207, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28021: Wed May 7 12:16:19 2025 read: IOPS=7, BW=8181KiB/s (8377kB/s)(6144KiB/751msec) clat (msec): min=14, max=150, avg=70.70, stdev=48.40 lat (msec): min=14, max=150, avg=70.70, stdev=48.40 clat percentiles (msec): | 1.00th=[ 15], 5.00th=[ 15], 10.00th=[ 15], 20.00th=[ 31], | 30.00th=[ 31], 40.00th=[ 59], 50.00th=[ 59], 60.00th=[ 81], | 70.00th=[ 89], 80.00th=[ 89], 90.00th=[ 150], 95.00th=[ 150], | 99.00th=[ 150], 99.50th=[ 150], 99.90th=[ 150], 99.95th=[ 150], | 99.99th=[ 150] bw ( KiB/s): min= 8192, max= 8192, per=32.60%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=7, BW=8181KiB/s (8377kB/s)(6144KiB/751msec) clat (usec): min=18112, max=80186, avg=50597.00, stdev=26295.10 lat (usec): min=18210, max=80295, avg=50708.26, stdev=26236.11 clat percentiles (usec): | 1.00th=[18220], 5.00th=[18220], 10.00th=[18220], 20.00th=[19530], | 30.00th=[19530], 40.00th=[51643], 50.00th=[51643], 60.00th=[65274], | 70.00th=[69731], 80.00th=[69731], 90.00th=[80217], 95.00th=[80217], | 99.00th=[80217], 99.50th=[80217], 99.90th=[80217], 99.95th=[80217], | 99.99th=[80217] bw ( KiB/s): min= 8192, max= 8192, per=23.29%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 lat (msec) : 20=25.00%, 50=8.33%, 100=58.33%, 250=8.33% cpu : usr=0.27%, sys=18.40%, ctx=76, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28022: Wed May 7 12:16:19 2025 read: IOPS=6, BW=6606KiB/s (6765kB/s)(5120KiB/775msec) clat (usec): min=13308, max=28063, avg=17987.65, stdev=5936.86 lat (usec): min=13309, max=28064, avg=17988.62, stdev=5936.95 clat percentiles (usec): | 1.00th=[13304], 5.00th=[13304], 10.00th=[13304], 20.00th=[13304], | 30.00th=[14222], 40.00th=[14222], 50.00th=[16188], 60.00th=[16188], | 70.00th=[18220], 80.00th=[18220], 90.00th=[28181], 95.00th=[28181], | 99.00th=[28181], 99.50th=[28181], 99.90th=[28181], 99.95th=[28181], | 99.99th=[28181] bw ( KiB/s): min= 8192, max= 8192, per=32.60%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=9, BW=9249KiB/s (9471kB/s)(7168KiB/775msec) clat (msec): min=24, max=154, avg=89.28, stdev=42.90 lat (msec): min=24, max=154, avg=89.37, stdev=42.89 clat percentiles (msec): | 1.00th=[ 25], 5.00th=[ 25], 10.00th=[ 25], 20.00th=[ 63], | 30.00th=[ 70], 40.00th=[ 70], 50.00th=[ 82], 60.00th=[ 114], | 70.00th=[ 114], 80.00th=[ 120], 90.00th=[ 155], 95.00th=[ 155], | 99.00th=[ 155], 99.50th=[ 155], 99.90th=[ 155], 99.95th=[ 155], | 99.99th=[ 155] bw ( KiB/s): min=10240, max=10240, per=29.11%, avg=10240.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 lat (msec) : 20=33.33%, 50=16.67%, 100=25.00%, 250=25.00% cpu : usr=0.52%, sys=11.24%, ctx=114, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=24.5MiB/s (25.7MB/s), 2872KiB/s-8795KiB/s (2941kB/s-9006kB/s), io=20.0MiB (20.0MB), run=713-815msec WRITE: bw=34.4MiB/s (36.0MB/s), 6282KiB/s-14.0MiB/s (6433kB/s-14.7MB/s), io=28.0MiB (29.4MB), run=713-815msec rand-rw: (groupid=0, jobs=1): err= 0: pid=28023: Wed May 7 12:16:19 2025 read: IOPS=8, BW=8710KiB/s (8919kB/s)(7168KiB/823msec) clat (usec): min=882, max=78415, avg=39921.30, stdev=23503.63 lat (usec): min=883, max=78416, avg=39922.37, stdev=23503.72 clat percentiles (usec): | 1.00th=[ 881], 5.00th=[ 881], 10.00th=[ 881], 20.00th=[33817], | 30.00th=[33817], 40.00th=[33817], 50.00th=[36439], 60.00th=[41157], | 70.00th=[41157], 80.00th=[54789], 90.00th=[78119], 95.00th=[78119], | 99.00th=[78119], 99.50th=[78119], 99.90th=[78119], 99.95th=[78119], | 99.99th=[78119] bw ( KiB/s): min=10240, max=10240, per=43.25%, avg=10240.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 write: IOPS=6, BW=6221KiB/s (6370kB/s)(5120KiB/823msec) clat (msec): min=26, max=186, avg=94.77, stdev=59.81 lat (msec): min=26, max=186, avg=94.80, stdev=59.81 clat percentiles (msec): | 1.00th=[ 27], 5.00th=[ 27], 10.00th=[ 27], 20.00th=[ 27], | 30.00th=[ 64], 40.00th=[ 64], 50.00th=[ 91], 60.00th=[ 91], | 70.00th=[ 107], 80.00th=[ 107], 90.00th=[ 188], 95.00th=[ 188], | 99.00th=[ 188], 99.50th=[ 188], 99.90th=[ 188], 99.95th=[ 188], | 99.99th=[ 188] bw ( KiB/s): min= 2048, max= 2048, per=6.18%, avg=2048.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (usec) : 1000=8.33% lat (msec) : 50=41.67%, 100=33.33%, 250=16.67% cpu : usr=0.12%, sys=24.70%, ctx=604, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28024: Wed May 7 12:16:19 2025 read: IOPS=2, BW=2368KiB/s (2424kB/s)(2048KiB/865msec) clat (usec): min=27933, max=32814, avg=30374.15, stdev=3451.70 lat (usec): min=27934, max=32815, avg=30375.30, stdev=3451.51 clat percentiles (usec): | 1.00th=[27919], 5.00th=[27919], 10.00th=[27919], 20.00th=[27919], | 30.00th=[27919], 40.00th=[27919], 50.00th=[27919], 60.00th=[32900], | 70.00th=[32900], 80.00th=[32900], 90.00th=[32900], 95.00th=[32900], | 99.00th=[32900], 99.50th=[32900], 99.90th=[32900], 99.95th=[32900], | 99.99th=[32900] bw ( KiB/s): min= 4096, max= 4096, per=17.30%, avg=4096.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 write: IOPS=11, BW=11.6MiB/s (12.1MB/s)(10.0MiB/865msec) clat (msec): min=22, max=181, avg=75.09, stdev=44.85 lat (msec): min=22, max=181, avg=75.16, stdev=44.84 clat percentiles (msec): | 1.00th=[ 23], 5.00th=[ 23], 10.00th=[ 23], 20.00th=[ 37], | 30.00th=[ 43], 40.00th=[ 58], 50.00th=[ 67], 60.00th=[ 72], | 70.00th=[ 84], 80.00th=[ 84], 90.00th=[ 109], 95.00th=[ 182], | 99.00th=[ 182], 99.50th=[ 182], 99.90th=[ 182], 99.95th=[ 182], | 99.99th=[ 182] bw ( KiB/s): min=10240, max=10240, per=30.89%, avg=10240.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 lat (msec) : 50=41.67%, 100=41.67%, 250=16.67% cpu : usr=0.12%, sys=36.00%, ctx=308, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28025: Wed May 7 12:16:19 2025 read: IOPS=10, BW=10.3MiB/s (10.8MB/s)(6144KiB/584msec) clat (usec): min=472, max=63930, avg=28505.21, stdev=24474.52 lat (usec): min=473, max=63931, avg=28506.54, stdev=24474.20 clat percentiles (usec): | 1.00th=[ 474], 5.00th=[ 474], 10.00th=[ 474], 20.00th=[ 840], | 30.00th=[ 840], 40.00th=[31589], 50.00th=[31589], 60.00th=[33162], | 70.00th=[41157], 80.00th=[41157], 90.00th=[63701], 95.00th=[63701], | 99.00th=[63701], 99.50th=[63701], 99.90th=[63701], 99.95th=[63701], | 99.99th=[63701] bw ( KiB/s): min= 8192, max= 8192, per=34.60%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=10, BW=10.3MiB/s (10.8MB/s)(6144KiB/584msec) clat (usec): min=21658, max=88832, avg=60639.68, stdev=28797.42 lat (usec): min=21748, max=88850, avg=60705.81, stdev=28781.79 clat percentiles (usec): | 1.00th=[21627], 5.00th=[21627], 10.00th=[21627], 20.00th=[35390], | 30.00th=[35390], 40.00th=[50070], 50.00th=[50070], 60.00th=[83362], | 70.00th=[84411], 80.00th=[84411], 90.00th=[88605], 95.00th=[88605], | 99.00th=[88605], 99.50th=[88605], 99.90th=[88605], 99.95th=[88605], | 99.99th=[88605] bw ( KiB/s): min=10240, max=10240, per=30.89%, avg=10240.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 lat (usec) : 500=8.33%, 1000=8.33% lat (msec) : 50=41.67%, 100=41.67% cpu : usr=0.17%, sys=37.05%, ctx=503, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28026: Wed May 7 12:16:19 2025 read: IOPS=7, BW=7688KiB/s (7872kB/s)(5120KiB/666msec) clat (usec): min=760, max=52686, avg=36257.69, stdev=21148.06 lat (usec): min=762, max=52688, avg=36259.27, stdev=21147.96 clat percentiles (usec): | 1.00th=[ 758], 5.00th=[ 758], 10.00th=[ 758], 20.00th=[ 758], | 30.00th=[36439], 40.00th=[36439], 50.00th=[39584], 60.00th=[39584], | 70.00th=[52167], 80.00th=[52167], 90.00th=[52691], 95.00th=[52691], | 99.00th=[52691], 99.50th=[52691], 99.90th=[52691], 99.95th=[52691], | 99.99th=[52691] bw ( KiB/s): min= 8192, max= 8192, per=34.60%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=10, BW=10.5MiB/s (11.0MB/s)(7168KiB/666msec) clat (msec): min=27, max=143, avg=67.29, stdev=43.53 lat (msec): min=27, max=143, avg=67.36, stdev=43.54 clat percentiles (msec): | 1.00th=[ 28], 5.00th=[ 28], 10.00th=[ 28], 20.00th=[ 36], | 30.00th=[ 40], 40.00th=[ 40], 50.00th=[ 55], 60.00th=[ 61], | 70.00th=[ 61], 80.00th=[ 111], 90.00th=[ 144], 95.00th=[ 144], | 99.00th=[ 144], 99.50th=[ 144], 99.90th=[ 144], 99.95th=[ 144], | 99.99th=[ 144] bw ( KiB/s): min=12288, max=12288, per=37.07%, avg=12288.00, stdev= 0.00, samples=1 iops : min= 12, max= 12, avg=12.00, stdev= 0.00, samples=1 lat (usec) : 1000=8.33% lat (msec) : 50=41.67%, 100=33.33%, 250=16.67% cpu : usr=0.00%, sys=36.39%, ctx=780, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=23.1MiB/s (24.2MB/s), 2368KiB/s-10.3MiB/s (2424kB/s-10.8MB/s), io=20.0MiB (20.0MB), run=584-865msec WRITE: bw=32.4MiB/s (33.9MB/s), 6221KiB/s-11.6MiB/s (6370kB/s-12.1MB/s), io=28.0MiB (29.4MB), run=584-865msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=28034: Wed May 7 12:16:21 2025 read: IOPS=5, BW=20.2MiB/s (21.1MB/s)(12.0MiB/595msec) clat (usec): min=1526, max=537426, avg=190890.22, stdev=300540.79 lat (usec): min=1527, max=537427, avg=190891.38, stdev=300540.75 clat percentiles (usec): | 1.00th=[ 1532], 5.00th=[ 1532], 10.00th=[ 1532], 20.00th=[ 1532], | 30.00th=[ 1532], 40.00th=[ 33817], 50.00th=[ 33817], 60.00th=[ 33817], | 70.00th=[541066], 80.00th=[541066], 90.00th=[541066], 95.00th=[541066], | 99.00th=[541066], 99.50th=[541066], 99.90th=[541066], 99.95th=[541066], | 99.99th=[541066] lat (msec) : 2=33.33%, 50=33.33%, 750=33.33% cpu : usr=0.17%, sys=21.55%, ctx=1278, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28036: Wed May 7 12:16:21 2025 read: IOPS=1, BW=4774KiB/s (4888kB/s)(4096KiB/858msec) clat (nsec): min=503023k, max=503023k, avg=503022837.00, stdev= 0.00 lat (nsec): min=503025k, max=503025k, avg=503025387.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 502], 5.00th=[ 502], 10.00th=[ 502], 20.00th=[ 502], | 30.00th=[ 502], 40.00th=[ 502], 50.00th=[ 502], 60.00th=[ 502], | 70.00th=[ 502], 80.00th=[ 502], 90.00th=[ 502], 95.00th=[ 502], | 99.00th=[ 502], 99.50th=[ 502], 99.90th=[ 502], 99.95th=[ 502], | 99.99th=[ 502] write: IOPS=2, BW=9548KiB/s (9777kB/s)(8192KiB/858msec) clat (msec): min=160, max=171, avg=166.16, stdev= 7.46 lat (msec): min=161, max=171, avg=166.45, stdev= 7.51 clat percentiles (msec): | 1.00th=[ 161], 5.00th=[ 161], 10.00th=[ 161], 20.00th=[ 161], | 30.00th=[ 161], 40.00th=[ 161], 50.00th=[ 161], 60.00th=[ 171], | 70.00th=[ 171], 80.00th=[ 171], 90.00th=[ 171], 95.00th=[ 171], | 99.00th=[ 171], 99.50th=[ 171], 99.90th=[ 171], 99.95th=[ 171], | 99.99th=[ 171] lat (msec) : 250=66.67%, 750=33.33% cpu : usr=0.00%, sys=24.97%, ctx=1026, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28038: Wed May 7 12:16:21 2025 write: IOPS=2, BW=10.2MiB/s (10.7MB/s)(12.0MiB/1174msec) clat (msec): min=109, max=749, avg=377.67, stdev=332.46 lat (msec): min=109, max=749, avg=377.97, stdev=332.52 clat percentiles (msec): | 1.00th=[ 110], 5.00th=[ 110], 10.00th=[ 110], 20.00th=[ 110], | 30.00th=[ 110], 40.00th=[ 275], 50.00th=[ 275], 60.00th=[ 275], | 70.00th=[ 751], 80.00th=[ 751], 90.00th=[ 751], 95.00th=[ 751], | 99.00th=[ 751], 99.50th=[ 751], 99.90th=[ 751], 99.95th=[ 751], | 99.99th=[ 751] lat (msec) : 250=33.33%, 500=33.33%, 750=33.33% cpu : usr=0.00%, sys=28.90%, ctx=43, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28040: Wed May 7 12:16:21 2025 write: IOPS=3, BW=12.5MiB/s (13.1MB/s)(12.0MiB/960msec) clat (msec): min=111, max=686, avg=305.57, stdev=329.97 lat (msec): min=111, max=686, avg=305.89, stdev=329.96 clat percentiles (msec): | 1.00th=[ 112], 5.00th=[ 112], 10.00th=[ 112], 20.00th=[ 112], | 30.00th=[ 112], 40.00th=[ 120], 50.00th=[ 120], 60.00th=[ 120], | 70.00th=[ 684], 80.00th=[ 684], 90.00th=[ 684], 95.00th=[ 684], | 99.00th=[ 684], 99.50th=[ 684], 99.90th=[ 684], 99.95th=[ 684], | 99.99th=[ 684] bw ( KiB/s): min=16384, max=16384, per=58.70%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 750=33.33% cpu : usr=0.21%, sys=33.79%, ctx=29, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=18.6MiB/s (19.6MB/s), 4774KiB/s-20.2MiB/s (4888kB/s-21.1MB/s), io=16.0MiB (16.8MB), run=595-858msec WRITE: bw=27.3MiB/s (28.6MB/s), 9548KiB/s-12.5MiB/s (9777kB/s-13.1MB/s), io=32.0MiB (33.6MB), run=858-1174msec rand-rw: (groupid=0, jobs=1): err= 0: pid=28033: Wed May 7 12:16:21 2025 read: IOPS=3, BW=14.4MiB/s (15.1MB/s)(12.0MiB/834msec) clat (msec): min=57, max=572, avg=267.81, stdev=269.87 lat (msec): min=57, max=572, avg=267.81, stdev=269.87 clat percentiles (msec): | 1.00th=[ 58], 5.00th=[ 58], 10.00th=[ 58], 20.00th=[ 58], | 30.00th=[ 58], 40.00th=[ 174], 50.00th=[ 174], 60.00th=[ 174], | 70.00th=[ 575], 80.00th=[ 575], 90.00th=[ 575], 95.00th=[ 575], | 99.00th=[ 575], 99.50th=[ 575], 99.90th=[ 575], 99.95th=[ 575], | 99.99th=[ 575] lat (msec) : 100=33.33%, 250=33.33%, 750=33.33% cpu : usr=0.00%, sys=11.28%, ctx=58, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28035: Wed May 7 12:16:21 2025 read: IOPS=0, BW=3360KiB/s (3441kB/s)(4096KiB/1219msec) clat (nsec): min=551358k, max=551358k, avg=551357828.00, stdev= 0.00 lat (nsec): min=551360k, max=551360k, avg=551360030.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 550], 5.00th=[ 550], 10.00th=[ 550], 20.00th=[ 550], | 30.00th=[ 550], 40.00th=[ 550], 50.00th=[ 550], 60.00th=[ 550], | 70.00th=[ 550], 80.00th=[ 550], 90.00th=[ 550], 95.00th=[ 550], | 99.00th=[ 550], 99.50th=[ 550], 99.90th=[ 550], 99.95th=[ 550], | 99.99th=[ 550] bw ( KiB/s): min= 8192, max= 8192, per=60.95%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=1, BW=6720KiB/s (6882kB/s)(8192KiB/1219msec) clat (msec): min=230, max=385, avg=307.85, stdev=109.95 lat (msec): min=230, max=385, avg=308.08, stdev=110.04 clat percentiles (msec): | 1.00th=[ 230], 5.00th=[ 230], 10.00th=[ 230], 20.00th=[ 230], | 30.00th=[ 230], 40.00th=[ 230], 50.00th=[ 230], 60.00th=[ 384], | 70.00th=[ 384], 80.00th=[ 384], 90.00th=[ 384], 95.00th=[ 384], | 99.00th=[ 384], 99.50th=[ 384], 99.90th=[ 384], 99.95th=[ 384], | 99.99th=[ 384] bw ( KiB/s): min= 8192, max= 8192, per=33.18%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=33.33%, 750=33.33% cpu : usr=0.00%, sys=13.71%, ctx=97, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28037: Wed May 7 12:16:21 2025 write: IOPS=2, BW=9260KiB/s (9482kB/s)(12.0MiB/1327msec) clat (msec): min=118, max=1010, avg=435.03, stdev=499.25 lat (msec): min=118, max=1010, avg=435.25, stdev=499.18 clat percentiles (msec): | 1.00th=[ 120], 5.00th=[ 120], 10.00th=[ 120], 20.00th=[ 120], | 30.00th=[ 120], 40.00th=[ 176], 50.00th=[ 176], 60.00th=[ 176], | 70.00th=[ 1011], 80.00th=[ 1011], 90.00th=[ 1011], 95.00th=[ 1011], | 99.00th=[ 1011], 99.50th=[ 1011], 99.90th=[ 1011], 99.95th=[ 1011], | 99.99th=[ 1011] lat (msec) : 250=66.67% cpu : usr=0.08%, sys=10.41%, ctx=111, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28039: Wed May 7 12:16:21 2025 write: IOPS=2, BW=10.0MiB/s (11.5MB/s)(12.0MiB/1091msec) clat (msec): min=41, max=952, avg=356.01, stdev=516.48 lat (msec): min=41, max=952, avg=356.32, stdev=516.46 clat percentiles (msec): | 1.00th=[ 42], 5.00th=[ 42], 10.00th=[ 42], 20.00th=[ 42], | 30.00th=[ 42], 40.00th=[ 75], 50.00th=[ 75], 60.00th=[ 75], | 70.00th=[ 953], 80.00th=[ 953], 90.00th=[ 953], 95.00th=[ 953], | 99.00th=[ 953], 99.50th=[ 953], 99.90th=[ 953], 99.95th=[ 953], | 99.99th=[ 953] bw ( KiB/s): min=16384, max=16384, per=66.35%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 50=33.33%, 100=33.33%, 1000=33.33% cpu : usr=0.00%, sys=10.46%, ctx=189, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=13.1MiB/s (13.8MB/s), 3360KiB/s-14.4MiB/s (3441kB/s-15.1MB/s), io=16.0MiB (16.8MB), run=834-1219msec WRITE: bw=24.1MiB/s (25.3MB/s), 6720KiB/s-10.0MiB/s (6882kB/s-11.5MB/s), io=32.0MiB (33.6MB), run=1091-1327msec