== sanity test 398b: DIO and buffer IO race ============== 07:46:04 (1744544764) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.19357 s, 42.2 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=32502: Sun Apr 13 07:46:20 2025 read: IOPS=112, BW=450KiB/s (461kB/s)(5992KiB/13320msec) clat (usec): min=65, max=159440, avg=5017.68, stdev=4788.67 lat (usec): min=66, max=159440, avg=5018.49, stdev=4788.63 clat percentiles (usec): | 1.00th=[ 79], 5.00th=[ 2180], 10.00th=[ 2442], 20.00th=[ 2933], | 30.00th=[ 3326], 40.00th=[ 3752], 50.00th=[ 4359], 60.00th=[ 5145], | 70.00th=[ 5800], 80.00th=[ 6718], 90.00th=[ 8029], 95.00th=[ 9503], | 99.00th=[ 14877], 99.50th=[ 16319], 99.90th=[ 21890], 99.95th=[160433], | 99.99th=[160433] bw ( KiB/s): min= 272, max= 568, per=24.71%, avg=444.58, stdev=68.50, samples=26 iops : min= 68, max= 142, avg=111.12, stdev=17.11, samples=26 write: IOPS=118, BW=473KiB/s (484kB/s)(6296KiB/13320msec) clat (usec): min=432, max=25087, avg=3644.07, stdev=3147.16 lat (usec): min=433, max=25088, avg=3644.83, stdev=3147.14 clat percentiles (usec): | 1.00th=[ 529], 5.00th=[ 611], 10.00th=[ 660], 20.00th=[ 775], | 30.00th=[ 1020], 40.00th=[ 2311], 50.00th=[ 3032], 60.00th=[ 3720], | 70.00th=[ 4490], 80.00th=[ 6063], 90.00th=[ 8094], 95.00th=[ 9896], | 99.00th=[13173], 99.50th=[14353], 99.90th=[24773], 99.95th=[25035], | 99.99th=[25035] bw ( KiB/s): min= 344, max= 672, per=24.88%, avg=468.54, stdev=73.00, samples=26 iops : min= 86, max= 168, avg=117.12, stdev=18.23, samples=26 lat (usec) : 100=0.68%, 250=0.98%, 500=0.10%, 750=9.44%, 1000=5.66% lat (msec) : 2=4.62%, 4=33.04%, 10=41.11%, 20=4.20%, 50=0.13% lat (msec) : 250=0.03% cpu : usr=0.18%, sys=22.90%, ctx=5911, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32503: Sun Apr 13 07:46:20 2025 read: IOPS=111, BW=447KiB/s (458kB/s)(5940KiB/13287msec) clat (usec): min=53, max=167602, avg=4882.85, stdev=5027.56 lat (usec): min=54, max=167604, avg=4883.47, stdev=5027.59 clat percentiles (usec): | 1.00th=[ 69], 5.00th=[ 465], 10.00th=[ 2376], 20.00th=[ 2835], | 30.00th=[ 3261], 40.00th=[ 3752], 50.00th=[ 4359], 60.00th=[ 4948], | 70.00th=[ 5669], 80.00th=[ 6390], 90.00th=[ 7832], 95.00th=[ 9241], | 99.00th=[ 15401], 99.50th=[ 17695], 99.90th=[ 24773], 99.95th=[166724], | 99.99th=[166724] bw ( KiB/s): min= 232, max= 544, per=24.66%, avg=443.58, stdev=66.49, samples=26 iops : min= 58, max= 136, avg=110.85, stdev=16.59, samples=26 write: IOPS=119, BW=478KiB/s (489kB/s)(6348KiB/13287msec) clat (usec): min=470, max=20482, avg=3752.75, stdev=3092.80 lat (usec): min=470, max=20483, avg=3753.52, stdev=3092.79 clat percentiles (usec): | 1.00th=[ 545], 5.00th=[ 603], 10.00th=[ 652], 20.00th=[ 775], | 30.00th=[ 1012], 40.00th=[ 2442], 50.00th=[ 3163], 60.00th=[ 3916], | 70.00th=[ 4752], 80.00th=[ 6325], 90.00th=[ 8160], 95.00th=[ 9896], | 99.00th=[13304], 99.50th=[14091], 99.90th=[17695], 99.95th=[20579], | 99.99th=[20579] bw ( KiB/s): min= 288, max= 544, per=24.91%, avg=469.12, stdev=59.98, samples=26 iops : min= 72, max= 136, avg=117.23, stdev=14.98, samples=26 lat (usec) : 100=1.30%, 250=1.01%, 500=0.20%, 750=9.60%, 1000=5.76% lat (msec) : 2=3.71%, 4=31.54%, 10=42.61%, 20=4.13%, 50=0.10% lat (msec) : 250=0.03% cpu : usr=0.18%, sys=22.84%, ctx=5798, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32504: Sun Apr 13 07:46:20 2025 read: IOPS=113, BW=453KiB/s (464kB/s)(6028KiB/13314msec) clat (usec): min=53, max=22704, avg=4731.71, stdev=2629.81 lat (usec): min=53, max=22705, avg=4732.39, stdev=2629.79 clat percentiles (usec): | 1.00th=[ 79], 5.00th=[ 367], 10.00th=[ 2376], 20.00th=[ 2802], | 30.00th=[ 3261], 40.00th=[ 3687], 50.00th=[ 4228], 60.00th=[ 4948], | 70.00th=[ 5800], 80.00th=[ 6456], 90.00th=[ 7701], 95.00th=[ 9110], | 99.00th=[13829], 99.50th=[16188], 99.90th=[21365], 99.95th=[22676], | 99.99th=[22676] bw ( KiB/s): min= 192, max= 568, per=24.79%, avg=445.92, stdev=77.36, samples=26 iops : min= 48, max= 142, avg=111.35, stdev=19.31, samples=26 write: IOPS=117, BW=470KiB/s (481kB/s)(6260KiB/13314msec) clat (usec): min=433, max=183180, avg=3905.69, stdev=5563.48 lat (usec): min=434, max=183181, avg=3907.49, stdev=5563.16 clat percentiles (usec): | 1.00th=[ 537], 5.00th=[ 603], 10.00th=[ 660], 20.00th=[ 783], | 30.00th=[ 1012], 40.00th=[ 2442], 50.00th=[ 3130], 60.00th=[ 3851], | 70.00th=[ 4817], 80.00th=[ 6390], 90.00th=[ 8291], 95.00th=[ 10290], | 99.00th=[ 13435], 99.50th=[ 15664], 99.90th=[ 21627], 99.95th=[183501], | 99.99th=[183501] bw ( KiB/s): min= 264, max= 584, per=24.45%, avg=460.42, stdev=76.27, samples=26 iops : min= 66, max= 146, avg=114.96, stdev=19.19, samples=26 lat (usec) : 100=1.33%, 250=1.11%, 500=0.10%, 750=9.15%, 1000=6.09% lat (msec) : 2=4.46%, 4=32.19%, 10=40.72%, 20=4.72%, 50=0.10% lat (msec) : 250=0.03% cpu : usr=0.18%, sys=22.90%, ctx=5803, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32505: Sun Apr 13 07:46:20 2025 read: IOPS=113, BW=453KiB/s (464kB/s)(6052KiB/13347msec) clat (usec): min=57, max=23340, avg=4851.44, stdev=2671.71 lat (usec): min=58, max=23340, avg=4852.99, stdev=2671.26 clat percentiles (usec): | 1.00th=[ 73], 5.00th=[ 2008], 10.00th=[ 2343], 20.00th=[ 2802], | 30.00th=[ 3261], 40.00th=[ 3720], 50.00th=[ 4424], 60.00th=[ 5014], | 70.00th=[ 5866], 80.00th=[ 6718], 90.00th=[ 7963], 95.00th=[ 9372], | 99.00th=[14091], 99.50th=[16909], 99.90th=[21365], 99.95th=[23462], | 99.99th=[23462] bw ( KiB/s): min= 280, max= 568, per=24.81%, avg=446.27, stdev=59.07, samples=26 iops : min= 70, max= 142, avg=111.42, stdev=14.79, samples=26 write: IOPS=116, BW=467KiB/s (478kB/s)(6236KiB/13347msec) clat (usec): min=453, max=176625, avg=3809.02, stdev=5469.55 lat (usec): min=454, max=176626, avg=3809.81, stdev=5469.53 clat percentiles (usec): | 1.00th=[ 537], 5.00th=[ 586], 10.00th=[ 635], 20.00th=[ 725], | 30.00th=[ 889], 40.00th=[ 1827], 50.00th=[ 2999], 60.00th=[ 3785], | 70.00th=[ 4752], 80.00th=[ 6194], 90.00th=[ 8455], 95.00th=[ 10290], | 99.00th=[ 13829], 99.50th=[ 15533], 99.90th=[ 18744], 99.95th=[177210], | 99.99th=[177210] bw ( KiB/s): min= 343, max= 736, per=24.50%, avg=461.35, stdev=82.64, samples=26 iops : min= 85, max= 184, avg=115.19, stdev=20.69, samples=26 lat (usec) : 100=1.33%, 250=0.75%, 500=0.13%, 750=11.04%, 1000=5.50% lat (msec) : 2=4.43%, 4=30.05%, 10=42.29%, 20=4.39%, 50=0.07% lat (msec) : 250=0.03% cpu : usr=0.12%, sys=22.82%, ctx=5907, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1799KiB/s (1842kB/s), 447KiB/s-453KiB/s (458kB/s-464kB/s), io=23.4MiB (24.6MB), run=13287-13347msec WRITE: bw=1884KiB/s (1929kB/s), 467KiB/s-478KiB/s (478kB/s-489kB/s), io=24.6MiB (25.7MB), run=13287-13347msec rand-rw: (groupid=0, jobs=1): err= 0: pid=32506: Sun Apr 13 07:46:23 2025 read: IOPS=88, BW=354KiB/s (363kB/s)(5992KiB/16915msec) clat (usec): min=1646, max=161657, avg=4693.39, stdev=5408.01 lat (usec): min=1647, max=161658, avg=4694.03, stdev=5408.03 clat percentiles (usec): | 1.00th=[ 1811], 5.00th=[ 1926], 10.00th=[ 2008], 20.00th=[ 2147], | 30.00th=[ 2311], 40.00th=[ 2573], 50.00th=[ 3032], 60.00th=[ 3589], | 70.00th=[ 4490], 80.00th=[ 6718], 90.00th=[ 10421], 95.00th=[ 12649], | 99.00th=[ 17171], 99.50th=[ 19006], 99.90th=[ 23725], 99.95th=[162530], | 99.99th=[162530] bw ( KiB/s): min= 176, max= 800, per=24.59%, avg=348.48, stdev=173.03, samples=33 iops : min= 44, max= 200, avg=87.03, stdev=43.30, samples=33 write: IOPS=93, BW=372KiB/s (381kB/s)(6296KiB/16915msec) clat (usec): min=2347, max=27343, avg=6244.66, stdev=3894.51 lat (usec): min=2348, max=27343, avg=6245.46, stdev=3894.53 clat percentiles (usec): | 1.00th=[ 2540], 5.00th=[ 2900], 10.00th=[ 3097], 20.00th=[ 3326], | 30.00th=[ 3556], 40.00th=[ 3949], 50.00th=[ 4621], 60.00th=[ 5800], | 70.00th=[ 7308], 80.00th=[ 8848], 90.00th=[11207], 95.00th=[13960], | 99.00th=[20841], 99.50th=[23462], 99.90th=[26870], 99.95th=[27395], | 99.99th=[27395] bw ( KiB/s): min= 192, max= 784, per=24.53%, avg=364.00, stdev=174.18, samples=33 iops : min= 48, max= 196, avg=90.91, stdev=43.59, samples=33 lat (msec) : 2=4.59%, 4=48.24%, 10=34.37%, 20=11.95%, 50=0.81% lat (msec) : 250=0.03% cpu : usr=0.08%, sys=11.95%, ctx=3596, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32507: Sun Apr 13 07:46:23 2025 read: IOPS=88, BW=354KiB/s (363kB/s)(5940KiB/16767msec) clat (usec): min=1616, max=169769, avg=4531.58, stdev=5466.39 lat (usec): min=1616, max=169770, avg=4532.25, stdev=5466.41 clat percentiles (usec): | 1.00th=[ 1778], 5.00th=[ 1926], 10.00th=[ 1991], 20.00th=[ 2147], | 30.00th=[ 2278], 40.00th=[ 2606], 50.00th=[ 2999], 60.00th=[ 3556], | 70.00th=[ 4359], 80.00th=[ 6390], 90.00th=[ 9241], 95.00th=[ 11863], | 99.00th=[ 15664], 99.50th=[ 20579], 99.90th=[ 25297], 99.95th=[168821], | 99.99th=[168821] bw ( KiB/s): min= 136, max= 736, per=24.87%, avg=352.39, stdev=179.02, samples=33 iops : min= 34, max= 184, avg=88.03, stdev=44.79, samples=33 write: IOPS=94, BW=379KiB/s (388kB/s)(6348KiB/16767msec) clat (usec): min=2289, max=29329, avg=6281.46, stdev=3806.47 lat (usec): min=2290, max=29330, avg=6282.30, stdev=3806.47 clat percentiles (usec): | 1.00th=[ 2606], 5.00th=[ 2868], 10.00th=[ 3097], 20.00th=[ 3326], | 30.00th=[ 3621], 40.00th=[ 4047], 50.00th=[ 4817], 60.00th=[ 5735], | 70.00th=[ 7504], 80.00th=[ 9110], 90.00th=[11731], 95.00th=[13566], | 99.00th=[20579], 99.50th=[22152], 99.90th=[26870], 99.95th=[29230], | 99.99th=[29230] bw ( KiB/s): min= 216, max= 752, per=25.07%, avg=372.03, stdev=166.54, samples=33 iops : min= 54, max= 188, avg=92.94, stdev=41.66, samples=33 lat (msec) : 2=4.95%, 4=47.23%, 10=35.87%, 20=11.13%, 50=0.78% lat (msec) : 250=0.03% cpu : usr=0.21%, sys=11.77%, ctx=3600, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32508: Sun Apr 13 07:46:23 2025 read: IOPS=89, BW=358KiB/s (367kB/s)(6028KiB/16830msec) clat (usec): min=1623, max=22526, avg=4515.91, stdev=3454.73 lat (usec): min=1624, max=22527, avg=4516.59, stdev=3454.76 clat percentiles (usec): | 1.00th=[ 1795], 5.00th=[ 1909], 10.00th=[ 2008], 20.00th=[ 2147], | 30.00th=[ 2343], 40.00th=[ 2638], 50.00th=[ 3064], 60.00th=[ 3654], | 70.00th=[ 4424], 80.00th=[ 6390], 90.00th=[ 9896], 95.00th=[12256], | 99.00th=[16909], 99.50th=[19006], 99.90th=[22152], 99.95th=[22414], | 99.99th=[22414] bw ( KiB/s): min= 120, max= 784, per=24.95%, avg=353.58, stdev=170.96, samples=33 iops : min= 30, max= 196, avg=88.33, stdev=42.74, samples=33 write: IOPS=92, BW=372KiB/s (381kB/s)(6260KiB/16830msec) clat (msec): min=2, max=182, avg= 6.37, stdev= 5.83 lat (msec): min=2, max=182, avg= 6.37, stdev= 5.83 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 4], 20.00th=[ 4], | 30.00th=[ 4], 40.00th=[ 5], 50.00th=[ 5], 60.00th=[ 6], | 70.00th=[ 8], 80.00th=[ 10], 90.00th=[ 12], 95.00th=[ 14], | 99.00th=[ 20], 99.50th=[ 23], 99.90th=[ 32], 99.95th=[ 182], | 99.99th=[ 182] bw ( KiB/s): min= 176, max= 766, per=24.64%, avg=365.70, stdev=174.14, samples=33 iops : min= 44, max= 191, avg=91.36, stdev=43.53, samples=33 lat (msec) : 2=4.88%, 4=46.78%, 10=35.58%, 20=12.04%, 50=0.68% lat (msec) : 250=0.03% cpu : usr=0.16%, sys=11.87%, ctx=3613, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32509: Sun Apr 13 07:46:23 2025 read: IOPS=89, BW=357KiB/s (366kB/s)(6052KiB/16934msec) clat (usec): min=1675, max=24024, avg=4502.34, stdev=3457.01 lat (usec): min=1675, max=24025, avg=4503.20, stdev=3456.90 clat percentiles (usec): | 1.00th=[ 1795], 5.00th=[ 1909], 10.00th=[ 2008], 20.00th=[ 2147], | 30.00th=[ 2311], 40.00th=[ 2573], 50.00th=[ 3064], 60.00th=[ 3621], | 70.00th=[ 4490], 80.00th=[ 6456], 90.00th=[10028], 95.00th=[12256], | 99.00th=[16319], 99.50th=[19268], 99.90th=[23725], 99.95th=[23987], | 99.99th=[23987] bw ( KiB/s): min= 152, max= 744, per=24.65%, avg=349.30, stdev=171.25, samples=33 iops : min= 38, max= 186, avg=87.30, stdev=42.83, samples=33 write: IOPS=92, BW=368KiB/s (377kB/s)(6236KiB/16934msec) clat (msec): min=2, max=167, avg= 6.45, stdev= 5.68 lat (msec): min=2, max=167, avg= 6.45, stdev= 5.68 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 4], 20.00th=[ 4], | 30.00th=[ 4], 40.00th=[ 5], 50.00th=[ 5], 60.00th=[ 6], | 70.00th=[ 8], 80.00th=[ 10], 90.00th=[ 12], 95.00th=[ 14], | 99.00th=[ 22], 99.50th=[ 24], 99.90th=[ 32], 99.95th=[ 167], | 99.99th=[ 167] bw ( KiB/s): min= 184, max= 752, per=24.34%, avg=361.18, stdev=175.74, samples=33 iops : min= 46, max= 188, avg=90.27, stdev=43.95, samples=33 lat (msec) : 2=4.75%, 4=47.46%, 10=34.57%, 20=12.34%, 50=0.85% lat (msec) : 250=0.03% cpu : usr=0.14%, sys=11.70%, ctx=3666, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1418KiB/s (1452kB/s), 354KiB/s-358KiB/s (363kB/s-367kB/s), io=23.4MiB (24.6MB), run=16767-16934msec WRITE: bw=1485KiB/s (1520kB/s), 368KiB/s-379KiB/s (377kB/s-388kB/s), io=24.6MiB (25.7MB), run=16767-16934msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=32538: Sun Apr 13 07:46:28 2025 read: IOPS=107, BW=1726KiB/s (1767kB/s)(5920KiB/3430msec) clat (usec): min=70, max=28924, avg=4707.40, stdev=2943.98 lat (usec): min=71, max=28924, avg=4708.05, stdev=2943.96 clat percentiles (usec): | 1.00th=[ 74], 5.00th=[ 112], 10.00th=[ 167], 20.00th=[ 2835], | 30.00th=[ 3359], 40.00th=[ 3818], 50.00th=[ 4424], 60.00th=[ 5080], | 70.00th=[ 5669], 80.00th=[ 6587], 90.00th=[ 8225], 95.00th=[ 9241], | 99.00th=[12780], 99.50th=[14746], 99.90th=[28967], 99.95th=[28967], | 99.99th=[28967] bw ( KiB/s): min= 1437, max= 1824, per=24.77%, avg=1662.83, stdev=134.12, samples=6 iops : min= 89, max= 114, avg=103.67, stdev= 8.73, samples=6 write: IOPS=116, BW=1857KiB/s (1901kB/s)(6368KiB/3430msec) clat (usec): min=754, max=20264, avg=4199.51, stdev=3523.95 lat (usec): min=755, max=20265, avg=4210.14, stdev=3520.15 clat percentiles (usec): | 1.00th=[ 791], 5.00th=[ 889], 10.00th=[ 979], 20.00th=[ 1205], | 30.00th=[ 1483], 40.00th=[ 2409], 50.00th=[ 3195], 60.00th=[ 4047], | 70.00th=[ 5014], 80.00th=[ 6849], 90.00th=[ 9372], 95.00th=[11076], | 99.00th=[17171], 99.50th=[18220], 99.90th=[20317], 99.95th=[20317], | 99.99th=[20317] bw ( KiB/s): min= 1600, max= 2144, per=25.73%, avg=1817.33, stdev=205.91, samples=6 iops : min= 100, max= 134, avg=113.33, stdev=12.89, samples=6 lat (usec) : 100=1.82%, 250=3.39%, 500=0.26%, 1000=5.86% lat (msec) : 2=13.80%, 4=27.08%, 10=42.32%, 20=5.21%, 50=0.26% cpu : usr=0.38%, sys=25.81%, ctx=1564, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32539: Sun Apr 13 07:46:28 2025 read: IOPS=100, BW=1610KiB/s (1649kB/s)(5744KiB/3567msec) clat (usec): min=70, max=17422, avg=5251.64, stdev=2830.55 lat (usec): min=70, max=17422, avg=5252.28, stdev=2830.56 clat percentiles (usec): | 1.00th=[ 74], 5.00th=[ 121], 10.00th=[ 2474], 20.00th=[ 3228], | 30.00th=[ 3752], 40.00th=[ 4293], 50.00th=[ 4948], 60.00th=[ 5604], | 70.00th=[ 6521], 80.00th=[ 7308], 90.00th=[ 8717], 95.00th=[10028], | 99.00th=[13698], 99.50th=[16057], 99.90th=[17433], 99.95th=[17433], | 99.99th=[17433] bw ( KiB/s): min= 1312, max= 1728, per=23.81%, avg=1598.86, stdev=132.76, samples=7 iops : min= 82, max= 108, avg=99.71, stdev= 8.22, samples=7 write: IOPS=114, BW=1835KiB/s (1879kB/s)(6544KiB/3567msec) clat (usec): min=676, max=19738, avg=4050.98, stdev=3199.13 lat (usec): min=677, max=19738, avg=4052.77, stdev=3199.54 clat percentiles (usec): | 1.00th=[ 758], 5.00th=[ 881], 10.00th=[ 938], 20.00th=[ 1188], | 30.00th=[ 1483], 40.00th=[ 2507], 50.00th=[ 3261], 60.00th=[ 4015], | 70.00th=[ 5080], 80.00th=[ 6652], 90.00th=[ 8455], 95.00th=[10421], | 99.00th=[13566], 99.50th=[17171], 99.90th=[19792], 99.95th=[19792], | 99.99th=[19792] bw ( KiB/s): min= 1405, max= 2304, per=25.68%, avg=1813.86, stdev=298.19, samples=7 iops : min= 87, max= 144, avg=113.14, stdev=18.89, samples=7 lat (usec) : 100=1.43%, 250=2.34%, 750=0.65%, 1000=6.12% lat (msec) : 2=12.11%, 4=25.00%, 10=47.01%, 20=5.34% cpu : usr=0.22%, sys=25.94%, ctx=1700, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32540: Sun Apr 13 07:46:28 2025 read: IOPS=112, BW=1807KiB/s (1850kB/s)(6256KiB/3462msec) clat (usec): min=68, max=24176, avg=4816.95, stdev=3047.12 lat (usec): min=68, max=24177, avg=4817.57, stdev=3047.12 clat percentiles (usec): | 1.00th=[ 72], 5.00th=[ 99], 10.00th=[ 253], 20.00th=[ 2999], | 30.00th=[ 3359], 40.00th=[ 3884], 50.00th=[ 4359], 60.00th=[ 5080], | 70.00th=[ 5866], 80.00th=[ 6521], 90.00th=[ 8029], 95.00th=[ 9634], | 99.00th=[15139], 99.50th=[23725], 99.90th=[24249], 99.95th=[24249], | 99.99th=[24249] bw ( KiB/s): min= 1568, max= 2144, per=25.88%, avg=1737.33, stdev=225.15, samples=6 iops : min= 98, max= 134, avg=108.33, stdev=14.26, samples=6 write: IOPS=108, BW=1742KiB/s (1784kB/s)(6032KiB/3462msec) clat (usec): min=755, max=23326, avg=4127.34, stdev=3422.24 lat (usec): min=756, max=23327, avg=4128.31, stdev=3422.31 clat percentiles (usec): | 1.00th=[ 799], 5.00th=[ 881], 10.00th=[ 979], 20.00th=[ 1156], | 30.00th=[ 1418], 40.00th=[ 2343], 50.00th=[ 3228], 60.00th=[ 4080], | 70.00th=[ 5211], 80.00th=[ 6783], 90.00th=[ 8586], 95.00th=[11207], | 99.00th=[13698], 99.50th=[18482], 99.90th=[23200], 99.95th=[23200], | 99.99th=[23200] bw ( KiB/s): min= 1440, max= 2208, per=24.37%, avg=1721.33, stdev=298.48, samples=6 iops : min= 90, max= 138, avg=107.33, stdev=18.54, samples=6 lat (usec) : 100=2.60%, 250=2.47%, 500=0.13%, 1000=5.99% lat (msec) : 2=12.50%, 4=27.08%, 10=44.01%, 20=4.82%, 50=0.39% cpu : usr=0.17%, sys=25.57%, ctx=1656, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32541: Sun Apr 13 07:46:28 2025 read: IOPS=110, BW=1771KiB/s (1813kB/s)(6032KiB/3406msec) clat (usec): min=71, max=17811, avg=4840.93, stdev=2974.81 lat (usec): min=72, max=17811, avg=4841.59, stdev=2974.80 clat percentiles (usec): | 1.00th=[ 83], 5.00th=[ 101], 10.00th=[ 128], 20.00th=[ 2900], | 30.00th=[ 3490], 40.00th=[ 3884], 50.00th=[ 4490], 60.00th=[ 5211], | 70.00th=[ 5800], 80.00th=[ 6915], 90.00th=[ 8455], 95.00th=[10945], | 99.00th=[13566], 99.50th=[16188], 99.90th=[17695], 99.95th=[17695], | 99.99th=[17695] bw ( KiB/s): min= 1472, max= 1856, per=25.80%, avg=1732.00, stdev=148.16, samples=6 iops : min= 92, max= 116, avg=108.00, stdev= 9.06, samples=6 write: IOPS=114, BW=1837KiB/s (1881kB/s)(6256KiB/3406msec) clat (usec): min=727, max=26141, avg=3965.02, stdev=3448.08 lat (usec): min=727, max=26142, avg=3965.96, stdev=3448.11 clat percentiles (usec): | 1.00th=[ 758], 5.00th=[ 889], 10.00th=[ 955], 20.00th=[ 1074], | 30.00th=[ 1254], 40.00th=[ 1844], 50.00th=[ 3195], 60.00th=[ 3982], | 70.00th=[ 4883], 80.00th=[ 6456], 90.00th=[ 8356], 95.00th=[10814], | 99.00th=[16057], 99.50th=[19006], 99.90th=[26084], 99.95th=[26084], | 99.99th=[26084] bw ( KiB/s): min= 1568, max= 2240, per=25.58%, avg=1806.67, stdev=248.70, samples=6 iops : min= 98, max= 140, avg=112.67, stdev=15.59, samples=6 lat (usec) : 100=2.34%, 250=3.26%, 500=0.39%, 750=0.52%, 1000=7.03% lat (msec) : 2=13.54%, 4=24.61%, 10=42.06%, 20=6.12%, 50=0.13% cpu : usr=0.12%, sys=25.90%, ctx=1619, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=6715KiB/s (6876kB/s), 1610KiB/s-1807KiB/s (1649kB/s-1850kB/s), io=23.4MiB (24.5MB), run=3406-3567msec WRITE: bw=7065KiB/s (7234kB/s), 1742KiB/s-1857KiB/s (1784kB/s-1901kB/s), io=24.6MiB (25.8MB), run=3406-3567msec rand-rw: (groupid=0, jobs=1): err= 0: pid=32543: Sun Apr 13 07:46:29 2025 read: IOPS=83, BW=1340KiB/s (1372kB/s)(5920KiB/4418msec) clat (usec): min=1975, max=30241, avg=4682.10, stdev=3705.25 lat (usec): min=1976, max=30242, avg=4683.04, stdev=3705.17 clat percentiles (usec): | 1.00th=[ 2040], 5.00th=[ 2114], 10.00th=[ 2245], 20.00th=[ 2409], | 30.00th=[ 2606], 40.00th=[ 2835], 50.00th=[ 3228], 60.00th=[ 3687], | 70.00th=[ 4621], 80.00th=[ 6128], 90.00th=[ 9110], 95.00th=[13042], | 99.00th=[20317], 99.50th=[22938], 99.90th=[30278], 99.95th=[30278], | 99.99th=[30278] bw ( KiB/s): min= 800, max= 2368, per=22.37%, avg=1203.75, stdev=501.71, samples=8 iops : min= 50, max= 148, avg=75.12, stdev=31.41, samples=8 write: IOPS=90, BW=1441KiB/s (1476kB/s)(6368KiB/4418msec) clat (usec): min=2717, max=26565, avg=6641.48, stdev=4143.70 lat (usec): min=2718, max=26567, avg=6642.72, stdev=4143.80 clat percentiles (usec): | 1.00th=[ 2835], 5.00th=[ 3097], 10.00th=[ 3326], 20.00th=[ 3589], | 30.00th=[ 3851], 40.00th=[ 4228], 50.00th=[ 4817], 60.00th=[ 5669], | 70.00th=[ 7570], 80.00th=[ 9765], 90.00th=[12518], 95.00th=[14877], | 99.00th=[22414], 99.50th=[25822], 99.90th=[26608], 99.95th=[26608], | 99.99th=[26608] bw ( KiB/s): min= 992, max= 2496, per=23.30%, avg=1319.50, stdev=520.65, samples=8 iops : min= 62, max= 156, avg=82.38, stdev=32.48, samples=8 lat (msec) : 2=0.39%, 4=48.57%, 10=36.85%, 20=13.15%, 50=1.04% cpu : usr=0.11%, sys=12.88%, ctx=951, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32544: Sun Apr 13 07:46:29 2025 read: IOPS=81, BW=1302KiB/s (1333kB/s)(5744KiB/4413msec) clat (usec): min=1841, max=25378, avg=4700.25, stdev=3716.35 lat (usec): min=1842, max=25379, avg=4700.87, stdev=3716.35 clat percentiles (usec): | 1.00th=[ 1975], 5.00th=[ 2114], 10.00th=[ 2212], 20.00th=[ 2343], | 30.00th=[ 2573], 40.00th=[ 2933], 50.00th=[ 3195], 60.00th=[ 3687], | 70.00th=[ 4359], 80.00th=[ 6194], 90.00th=[ 9765], 95.00th=[13435], | 99.00th=[19268], 99.50th=[23725], 99.90th=[25297], 99.95th=[25297], | 99.99th=[25297] bw ( KiB/s): min= 896, max= 2368, per=22.19%, avg=1194.38, stdev=482.98, samples=8 iops : min= 56, max= 148, avg=74.25, stdev=30.31, samples=8 write: IOPS=92, BW=1483KiB/s (1518kB/s)(6544KiB/4413msec) clat (usec): min=2609, max=26713, avg=6611.54, stdev=4123.69 lat (usec): min=2610, max=26714, avg=6612.74, stdev=4123.78 clat percentiles (usec): | 1.00th=[ 2802], 5.00th=[ 3097], 10.00th=[ 3294], 20.00th=[ 3556], | 30.00th=[ 3785], 40.00th=[ 4113], 50.00th=[ 4883], 60.00th=[ 5800], | 70.00th=[ 7570], 80.00th=[ 9634], 90.00th=[12780], 95.00th=[15139], | 99.00th=[21627], 99.50th=[22152], 99.90th=[26608], 99.95th=[26608], | 99.99th=[26608] bw ( KiB/s): min= 926, max= 2592, per=23.78%, avg=1346.38, stdev=517.16, samples=8 iops : min= 57, max= 162, avg=83.75, stdev=32.48, samples=8 lat (msec) : 2=0.52%, 4=49.87%, 10=35.55%, 20=13.02%, 50=1.04% cpu : usr=0.09%, sys=12.87%, ctx=926, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32545: Sun Apr 13 07:46:29 2025 read: IOPS=89, BW=1431KiB/s (1465kB/s)(6256KiB/4372msec) clat (usec): min=1970, max=22395, avg=4558.88, stdev=3490.51 lat (usec): min=1971, max=22395, avg=4559.55, stdev=3490.54 clat percentiles (usec): | 1.00th=[ 2024], 5.00th=[ 2147], 10.00th=[ 2212], 20.00th=[ 2343], | 30.00th=[ 2540], 40.00th=[ 2737], 50.00th=[ 3097], 60.00th=[ 3621], | 70.00th=[ 4490], 80.00th=[ 5997], 90.00th=[ 9503], 95.00th=[12256], | 99.00th=[17695], 99.50th=[21627], 99.90th=[22414], 99.95th=[22414], | 99.99th=[22414] bw ( KiB/s): min= 800, max= 2400, per=23.99%, avg=1291.00, stdev=510.96, samples=8 iops : min= 50, max= 150, avg=80.50, stdev=31.98, samples=8 write: IOPS=86, BW=1380KiB/s (1413kB/s)(6032KiB/4372msec) clat (usec): min=2537, max=31628, avg=6830.48, stdev=4266.07 lat (usec): min=2539, max=31629, avg=6831.93, stdev=4266.15 clat percentiles (usec): | 1.00th=[ 2835], 5.00th=[ 3163], 10.00th=[ 3359], 20.00th=[ 3621], | 30.00th=[ 3916], 40.00th=[ 4293], 50.00th=[ 4817], 60.00th=[ 6652], | 70.00th=[ 7963], 80.00th=[ 9765], 90.00th=[13173], 95.00th=[14615], | 99.00th=[23200], 99.50th=[28705], 99.90th=[31589], 99.95th=[31589], | 99.99th=[31589] bw ( KiB/s): min= 864, max= 2592, per=22.59%, avg=1279.12, stdev=545.37, samples=8 iops : min= 54, max= 162, avg=79.75, stdev=34.18, samples=8 lat (msec) : 2=0.39%, 4=50.78%, 10=35.16%, 20=12.50%, 50=1.17% cpu : usr=0.11%, sys=12.97%, ctx=911, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32546: Sun Apr 13 07:46:29 2025 read: IOPS=84, BW=1356KiB/s (1388kB/s)(6032KiB/4450msec) clat (usec): min=1926, max=22918, avg=4863.17, stdev=3743.50 lat (usec): min=1927, max=22919, avg=4863.82, stdev=3743.50 clat percentiles (usec): | 1.00th=[ 1991], 5.00th=[ 2147], 10.00th=[ 2212], 20.00th=[ 2376], | 30.00th=[ 2573], 40.00th=[ 2868], 50.00th=[ 3261], 60.00th=[ 3687], | 70.00th=[ 4621], 80.00th=[ 7046], 90.00th=[10683], 95.00th=[13173], | 99.00th=[18220], 99.50th=[19792], 99.90th=[22938], 99.95th=[22938], | 99.99th=[22938] bw ( KiB/s): min= 864, max= 2240, per=22.06%, avg=1187.50, stdev=441.53, samples=8 iops : min= 54, max= 140, avg=74.00, stdev=27.73, samples=8 write: IOPS=87, BW=1406KiB/s (1440kB/s)(6256KiB/4450msec) clat (usec): min=2536, max=31511, avg=6594.13, stdev=4299.40 lat (usec): min=2537, max=31512, avg=6595.33, stdev=4299.44 clat percentiles (usec): | 1.00th=[ 2802], 5.00th=[ 3064], 10.00th=[ 3326], 20.00th=[ 3589], | 30.00th=[ 3818], 40.00th=[ 4178], 50.00th=[ 4817], 60.00th=[ 5669], | 70.00th=[ 7373], 80.00th=[ 9503], 90.00th=[12256], 95.00th=[14746], | 99.00th=[25822], 99.50th=[28443], 99.90th=[31589], 99.95th=[31589], | 99.99th=[31589] bw ( KiB/s): min= 960, max= 2496, per=22.59%, avg=1279.13, stdev=525.47, samples=8 iops : min= 60, max= 156, avg=79.75, stdev=32.85, samples=8 lat (msec) : 2=0.65%, 4=48.44%, 10=35.55%, 20=14.58%, 50=0.78% cpu : usr=0.09%, sys=12.99%, ctx=938, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=5382KiB/s (5512kB/s), 1302KiB/s-1431KiB/s (1333kB/s-1465kB/s), io=23.4MiB (24.5MB), run=4372-4450msec WRITE: bw=5663KiB/s (5799kB/s), 1380KiB/s-1483KiB/s (1413kB/s-1518kB/s), io=24.6MiB (25.8MB), run=4372-4450msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=32555: Sun Apr 13 07:46:31 2025 read: IOPS=7, BW=7175KiB/s (7347kB/s)(7168KiB/999msec) clat (usec): min=689, max=78773, avg=40535.16, stdev=28800.74 lat (usec): min=690, max=78774, avg=40536.26, stdev=28800.68 clat percentiles (usec): | 1.00th=[ 693], 5.00th=[ 693], 10.00th=[ 693], 20.00th=[24511], | 30.00th=[27657], 40.00th=[27657], 50.00th=[30540], 60.00th=[43779], | 70.00th=[43779], 80.00th=[78119], 90.00th=[79168], 95.00th=[79168], | 99.00th=[79168], 99.50th=[79168], 99.90th=[79168], 99.95th=[79168], | 99.99th=[79168] bw ( KiB/s): min=10240, max=10240, per=54.30%, avg=10240.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 write: IOPS=5, BW=5125KiB/s (5248kB/s)(5120KiB/999msec) clat (msec): min=26, max=450, avg=132.90, stdev=178.75 lat (msec): min=26, max=450, avg=132.95, stdev=178.74 clat percentiles (msec): | 1.00th=[ 27], 5.00th=[ 27], 10.00th=[ 27], 20.00th=[ 27], | 30.00th=[ 41], 40.00th=[ 41], 50.00th=[ 71], 60.00th=[ 71], | 70.00th=[ 78], 80.00th=[ 78], 90.00th=[ 451], 95.00th=[ 451], | 99.00th=[ 451], 99.50th=[ 451], 99.90th=[ 451], 99.95th=[ 451], | 99.99th=[ 451] bw ( KiB/s): min= 6144, max= 6144, per=23.27%, avg=6144.00, stdev= 0.00, samples=1 iops : min= 6, max= 6, avg= 6.00, stdev= 0.00, samples=1 lat (usec) : 750=8.33% lat (msec) : 50=50.00%, 100=33.33%, 500=8.33% cpu : usr=0.00%, sys=18.64%, ctx=562, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32556: Sun Apr 13 07:46:31 2025 read: IOPS=1, BW=1886KiB/s (1931kB/s)(2048KiB/1086msec) clat (usec): min=32885, max=39588, avg=36237.22, stdev=4739.73 lat (usec): min=32886, max=39590, avg=36238.73, stdev=4740.52 clat percentiles (usec): | 1.00th=[32900], 5.00th=[32900], 10.00th=[32900], 20.00th=[32900], | 30.00th=[32900], 40.00th=[32900], 50.00th=[32900], 60.00th=[39584], | 70.00th=[39584], 80.00th=[39584], 90.00th=[39584], 95.00th=[39584], | 99.00th=[39584], 99.50th=[39584], 99.90th=[39584], 99.95th=[39584], | 99.99th=[39584] bw ( KiB/s): min= 4096, max= 4096, per=21.72%, avg=4096.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 write: IOPS=9, BW=9429KiB/s (9655kB/s)(10.0MiB/1086msec) clat (msec): min=30, max=446, avg=96.06, stdev=125.83 lat (msec): min=30, max=446, avg=96.12, stdev=125.84 clat percentiles (msec): | 1.00th=[ 31], 5.00th=[ 31], 10.00th=[ 31], 20.00th=[ 32], | 30.00th=[ 35], 40.00th=[ 44], 50.00th=[ 52], 60.00th=[ 62], | 70.00th=[ 84], 80.00th=[ 84], 90.00th=[ 113], 95.00th=[ 447], | 99.00th=[ 447], 99.50th=[ 447], 99.90th=[ 447], 99.95th=[ 447], | 99.99th=[ 447] bw ( KiB/s): min= 6144, max=10240, per=31.03%, avg=8192.00, stdev=2896.31, samples=2 iops : min= 6, max= 10, avg= 8.00, stdev= 2.83, samples=2 lat (msec) : 50=50.00%, 100=33.33%, 250=8.33%, 500=8.33% cpu : usr=0.09%, sys=24.42%, ctx=391, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32557: Sun Apr 13 07:46:31 2025 read: IOPS=11, BW=11.7MiB/s (12.2MB/s)(6144KiB/514msec) clat (usec): min=348, max=60407, avg=28053.88, stdev=24613.12 lat (usec): min=349, max=60408, avg=28054.89, stdev=24613.27 clat percentiles (usec): | 1.00th=[ 351], 5.00th=[ 351], 10.00th=[ 351], 20.00th=[ 570], | 30.00th=[ 570], 40.00th=[21890], 50.00th=[21890], 60.00th=[41681], | 70.00th=[43254], 80.00th=[43254], 90.00th=[60556], 95.00th=[60556], | 99.00th=[60556], 99.50th=[60556], 99.90th=[60556], 99.95th=[60556], | 99.99th=[60556] bw ( KiB/s): min=10240, max=10240, per=54.30%, avg=10240.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 write: IOPS=11, BW=11.7MiB/s (12.2MB/s)(6144KiB/514msec) clat (usec): min=22000, max=89779, avg=49027.64, stdev=24490.91 lat (usec): min=22032, max=89847, avg=49079.57, stdev=24504.46 clat percentiles (usec): | 1.00th=[21890], 5.00th=[21890], 10.00th=[21890], 20.00th=[32113], | 30.00th=[32113], 40.00th=[38536], 50.00th=[38536], 60.00th=[48497], | 70.00th=[63701], 80.00th=[63701], 90.00th=[89654], 95.00th=[89654], | 99.00th=[89654], 99.50th=[89654], 99.90th=[89654], 99.95th=[89654], | 99.99th=[89654] bw ( KiB/s): min=10240, max=10240, per=38.79%, avg=10240.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 lat (usec) : 500=8.33%, 750=8.33% lat (msec) : 50=58.33%, 100=25.00% cpu : usr=0.00%, sys=38.79%, ctx=616, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32558: Sun Apr 13 07:46:31 2025 read: IOPS=4, BW=4872KiB/s (4988kB/s)(5120KiB/1051msec) clat (msec): min=6, max=597, avg=137.15, stdev=257.55 lat (msec): min=6, max=597, avg=137.15, stdev=257.55 clat percentiles (msec): | 1.00th=[ 7], 5.00th=[ 7], 10.00th=[ 7], 20.00th=[ 7], | 30.00th=[ 26], 40.00th=[ 26], 50.00th=[ 28], 60.00th=[ 28], | 70.00th=[ 29], 80.00th=[ 29], 90.00th=[ 600], 95.00th=[ 600], | 99.00th=[ 600], 99.50th=[ 600], 99.90th=[ 600], 99.95th=[ 600], | 99.99th=[ 600] bw ( KiB/s): min= 8192, max= 8192, per=43.44%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=6, BW=6820KiB/s (6984kB/s)(7168KiB/1051msec) clat (usec): min=31253, max=87143, avg=49161.79, stdev=22664.70 lat (usec): min=31288, max=87209, avg=49234.60, stdev=22658.02 clat percentiles (usec): | 1.00th=[31327], 5.00th=[31327], 10.00th=[31327], 20.00th=[31327], | 30.00th=[32637], 40.00th=[32637], 50.00th=[33817], 60.00th=[58459], | 70.00th=[58459], 80.00th=[69731], 90.00th=[87557], 95.00th=[87557], | 99.00th=[87557], 99.50th=[87557], 99.90th=[87557], 99.95th=[87557], | 99.99th=[87557] bw ( KiB/s): min= 4096, max= 8192, per=23.27%, avg=6144.00, stdev=2896.31, samples=2 iops : min= 4, max= 8, avg= 6.00, stdev= 2.83, samples=2 lat (msec) : 10=8.33%, 50=58.33%, 100=25.00%, 750=8.33% cpu : usr=0.00%, sys=27.52%, ctx=980, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=18.4MiB/s (19.3MB/s), 1886KiB/s-11.7MiB/s (1931kB/s-12.2MB/s), io=20.0MiB (20.0MB), run=514-1086msec WRITE: bw=25.8MiB/s (27.0MB/s), 5125KiB/s-11.7MiB/s (5248kB/s-12.2MB/s), io=28.0MiB (29.4MB), run=514-1086msec rand-rw: (groupid=0, jobs=1): err= 0: pid=32559: Sun Apr 13 07:46:31 2025 read: IOPS=6, BW=6434KiB/s (6589kB/s)(7168KiB/1114msec) clat (usec): min=38169, max=96707, avg=66196.98, stdev=23408.98 lat (usec): min=38170, max=96708, avg=66197.94, stdev=23409.00 clat percentiles (usec): | 1.00th=[38011], 5.00th=[38011], 10.00th=[38011], 20.00th=[39060], | 30.00th=[62129], 40.00th=[62129], 50.00th=[65799], 60.00th=[66847], | 70.00th=[66847], 80.00th=[94897], 90.00th=[96994], 95.00th=[96994], | 99.00th=[96994], 99.50th=[96994], 99.90th=[96994], 99.95th=[96994], | 99.99th=[96994] bw ( KiB/s): min= 2048, max=10178, per=33.67%, avg=6113.00, stdev=5748.78, samples=2 iops : min= 2, max= 9, avg= 5.50, stdev= 4.95, samples=2 write: IOPS=4, BW=4596KiB/s (4706kB/s)(5120KiB/1114msec) clat (msec): min=18, max=432, avg=123.57, stdev=176.58 lat (msec): min=18, max=432, avg=123.62, stdev=176.57 clat percentiles (msec): | 1.00th=[ 19], 5.00th=[ 19], 10.00th=[ 19], 20.00th=[ 19], | 30.00th=[ 23], 40.00th=[ 23], 50.00th=[ 37], 60.00th=[ 37], | 70.00th=[ 108], 80.00th=[ 108], 90.00th=[ 435], 95.00th=[ 435], | 99.00th=[ 435], 99.50th=[ 435], 99.90th=[ 435], 99.95th=[ 435], | 99.99th=[ 435] bw ( KiB/s): min= 2035, max= 6144, per=16.09%, avg=4089.50, stdev=2905.50, samples=2 iops : min= 1, max= 6, avg= 3.50, stdev= 3.54, samples=2 lat (msec) : 20=8.33%, 50=33.33%, 100=41.67%, 250=8.33%, 500=8.33% cpu : usr=0.00%, sys=9.34%, ctx=72, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32560: Sun Apr 13 07:46:31 2025 read: IOPS=1, BW=1816KiB/s (1859kB/s)(2048KiB/1128msec) clat (usec): min=14019, max=20555, avg=17287.49, stdev=4621.58 lat (usec): min=14020, max=20556, avg=17288.47, stdev=4621.79 clat percentiles (usec): | 1.00th=[13960], 5.00th=[13960], 10.00th=[13960], 20.00th=[13960], | 30.00th=[13960], 40.00th=[13960], 50.00th=[13960], 60.00th=[20579], | 70.00th=[20579], 80.00th=[20579], 90.00th=[20579], 95.00th=[20579], | 99.00th=[20579], 99.50th=[20579], 99.90th=[20579], 99.95th=[20579], | 99.99th=[20579] bw ( KiB/s): min= 2039, max= 2048, per=11.26%, avg=2043.50, stdev= 6.36, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 write: IOPS=8, BW=9078KiB/s (9296kB/s)(10.0MiB/1128msec) clat (msec): min=19, max=440, avg=100.71, stdev=124.74 lat (msec): min=19, max=440, avg=100.83, stdev=124.71 clat percentiles (msec): | 1.00th=[ 21], 5.00th=[ 21], 10.00th=[ 21], 20.00th=[ 22], | 30.00th=[ 44], 40.00th=[ 48], 50.00th=[ 50], 60.00th=[ 56], | 70.00th=[ 101], 80.00th=[ 101], 90.00th=[ 132], 95.00th=[ 443], | 99.00th=[ 443], 99.50th=[ 443], 99.90th=[ 443], 99.95th=[ 443], | 99.99th=[ 443] bw ( KiB/s): min= 6144, max= 8159, per=28.14%, avg=7151.50, stdev=1424.82, samples=2 iops : min= 6, max= 7, avg= 6.50, stdev= 0.71, samples=2 lat (msec) : 20=16.67%, 50=41.67%, 100=25.00%, 250=8.33%, 500=8.33% cpu : usr=0.00%, sys=12.33%, ctx=143, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32561: Sun Apr 13 07:46:31 2025 read: IOPS=5, BW=5500KiB/s (5632kB/s)(6144KiB/1117msec) clat (msec): min=13, max=551, avg=116.18, stdev=213.94 lat (msec): min=13, max=551, avg=116.18, stdev=213.94 clat percentiles (msec): | 1.00th=[ 14], 5.00th=[ 14], 10.00th=[ 14], 20.00th=[ 16], | 30.00th=[ 16], 40.00th=[ 25], 50.00th=[ 25], 60.00th=[ 28], | 70.00th=[ 66], 80.00th=[ 66], 90.00th=[ 550], 95.00th=[ 550], | 99.00th=[ 550], 99.50th=[ 550], 99.90th=[ 550], 99.95th=[ 550], | 99.99th=[ 550] bw ( KiB/s): min= 8192, max= 8192, per=45.12%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=5, BW=5500KiB/s (5632kB/s)(6144KiB/1117msec) clat (usec): min=15277, max=99951, avg=55796.39, stdev=31107.10 lat (msec): min=15, max=100, avg=55.85, stdev=31.12 clat percentiles (msec): | 1.00th=[ 16], 5.00th=[ 16], 10.00th=[ 16], 20.00th=[ 29], | 30.00th=[ 29], 40.00th=[ 50], 50.00th=[ 50], 60.00th=[ 68], | 70.00th=[ 74], 80.00th=[ 74], 90.00th=[ 101], 95.00th=[ 101], | 99.00th=[ 101], 99.50th=[ 101], 99.90th=[ 101], 99.95th=[ 101], | 99.99th=[ 101] bw ( KiB/s): min= 4096, max= 6119, per=20.09%, avg=5107.50, stdev=1430.48, samples=2 iops : min= 4, max= 5, avg= 4.50, stdev= 0.71, samples=2 lat (msec) : 20=25.00%, 50=33.33%, 100=33.33%, 750=8.33% cpu : usr=0.00%, sys=10.84%, ctx=85, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32562: Sun Apr 13 07:46:31 2025 read: IOPS=4, BW=4650KiB/s (4762kB/s)(5120KiB/1101msec) clat (msec): min=13, max=396, avg=97.30, stdev=167.20 lat (msec): min=13, max=396, avg=97.30, stdev=167.20 clat percentiles (msec): | 1.00th=[ 14], 5.00th=[ 14], 10.00th=[ 14], 20.00th=[ 14], | 30.00th=[ 15], 40.00th=[ 15], 50.00th=[ 31], 60.00th=[ 31], | 70.00th=[ 32], 80.00th=[ 32], 90.00th=[ 397], 95.00th=[ 397], | 99.00th=[ 397], 99.50th=[ 397], 99.90th=[ 397], 99.95th=[ 397], | 99.99th=[ 397] bw ( KiB/s): min= 8192, max= 8192, per=45.12%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=6, BW=6510KiB/s (6667kB/s)(7168KiB/1101msec) clat (msec): min=17, max=126, avg=75.95, stdev=40.67 lat (msec): min=17, max=126, avg=76.00, stdev=40.67 clat percentiles (msec): | 1.00th=[ 18], 5.00th=[ 18], 10.00th=[ 18], 20.00th=[ 46], | 30.00th=[ 54], 40.00th=[ 54], 50.00th=[ 74], 60.00th=[ 92], | 70.00th=[ 92], 80.00th=[ 124], 90.00th=[ 127], 95.00th=[ 127], | 99.00th=[ 127], 99.50th=[ 127], 99.90th=[ 127], 99.95th=[ 127], | 99.99th=[ 127] bw ( KiB/s): min= 8175, max= 8175, per=32.16%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 7, max= 7, avg= 7.00, stdev= 0.00, samples=1 lat (msec) : 20=25.00%, 50=25.00%, 100=25.00%, 250=16.67%, 500=8.33% cpu : usr=0.00%, sys=10.00%, ctx=43, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=17.7MiB/s (18.6MB/s), 1816KiB/s-6434KiB/s (1859kB/s-6589kB/s), io=20.0MiB (20.0MB), run=1101-1128msec WRITE: bw=24.8MiB/s (26.0MB/s), 4596KiB/s-9078KiB/s (4706kB/s-9296kB/s), io=28.0MiB (29.4MB), run=1101-1128msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=32569: Sun Apr 13 07:46:33 2025 read: IOPS=9, BW=39.1MiB/s (40.0MB/s)(12.0MiB/307msec) clat (msec): min=38, max=148, avg=98.67, stdev=55.74 lat (msec): min=38, max=148, avg=98.67, stdev=55.74 clat percentiles (msec): | 1.00th=[ 39], 5.00th=[ 39], 10.00th=[ 39], 20.00th=[ 39], | 30.00th=[ 39], 40.00th=[ 110], 50.00th=[ 110], 60.00th=[ 110], | 70.00th=[ 148], 80.00th=[ 148], 90.00th=[ 148], 95.00th=[ 148], | 99.00th=[ 148], 99.50th=[ 148], 99.90th=[ 148], 99.95th=[ 148], | 99.99th=[ 148] lat (msec) : 50=33.33%, 250=66.67% cpu : usr=0.00%, sys=38.24%, ctx=1132, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32570: Sun Apr 13 07:46:33 2025 read: IOPS=1, BW=4607KiB/s (4718kB/s)(4096KiB/889msec) clat (nsec): min=80871k, max=80871k, avg=80870562.00, stdev= 0.00 lat (nsec): min=80873k, max=80873k, avg=80873245.00, stdev= 0.00 clat percentiles (usec): | 1.00th=[81265], 5.00th=[81265], 10.00th=[81265], 20.00th=[81265], | 30.00th=[81265], 40.00th=[81265], 50.00th=[81265], 60.00th=[81265], | 70.00th=[81265], 80.00th=[81265], 90.00th=[81265], 95.00th=[81265], | 99.00th=[81265], 99.50th=[81265], 99.90th=[81265], 99.95th=[81265], | 99.99th=[81265] bw ( KiB/s): min= 8192, max= 8192, per=44.45%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=2, BW=9215KiB/s (9436kB/s)(8192KiB/889msec) clat (msec): min=286, max=511, avg=398.90, stdev=158.57 lat (msec): min=286, max=511, avg=399.09, stdev=158.68 clat percentiles (msec): | 1.00th=[ 288], 5.00th=[ 288], 10.00th=[ 288], 20.00th=[ 288], | 30.00th=[ 288], 40.00th=[ 288], 50.00th=[ 288], 60.00th=[ 510], | 70.00th=[ 510], 80.00th=[ 510], 90.00th=[ 510], 95.00th=[ 510], | 99.00th=[ 510], 99.50th=[ 510], 99.90th=[ 510], 99.95th=[ 510], | 99.99th=[ 510] bw ( KiB/s): min= 8192, max= 8192, per=22.23%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 500=33.33%, 750=33.33% cpu : usr=0.00%, sys=24.77%, ctx=529, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32571: Sun Apr 13 07:46:33 2025 write: IOPS=4, BW=19.0MiB/s (19.0MB/s)(12.0MiB/630msec) clat (msec): min=187, max=235, avg=205.46, stdev=26.43 lat (msec): min=187, max=236, avg=205.79, stdev=26.42 clat percentiles (msec): | 1.00th=[ 188], 5.00th=[ 188], 10.00th=[ 188], 20.00th=[ 188], | 30.00th=[ 188], 40.00th=[ 194], 50.00th=[ 194], 60.00th=[ 194], | 70.00th=[ 236], 80.00th=[ 236], 90.00th=[ 236], 95.00th=[ 236], | 99.00th=[ 236], 99.50th=[ 236], 99.90th=[ 236], 99.95th=[ 236], | 99.99th=[ 236] bw ( KiB/s): min=16384, max=16384, per=44.45%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 250=100.00% cpu : usr=0.00%, sys=48.17%, ctx=72, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32572: Sun Apr 13 07:46:33 2025 write: IOPS=4, BW=17.8MiB/s (18.7MB/s)(12.0MiB/674msec) clat (msec): min=111, max=303, avg=215.09, stdev=97.09 lat (msec): min=111, max=303, avg=215.34, stdev=97.16 clat percentiles (msec): | 1.00th=[ 112], 5.00th=[ 112], 10.00th=[ 112], 20.00th=[ 112], | 30.00th=[ 112], 40.00th=[ 230], 50.00th=[ 230], 60.00th=[ 230], | 70.00th=[ 305], 80.00th=[ 305], 90.00th=[ 305], 95.00th=[ 305], | 99.00th=[ 305], 99.50th=[ 305], 99.90th=[ 305], 99.95th=[ 305], | 99.99th=[ 305] bw ( KiB/s): min= 8192, max= 8192, per=22.23%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.15%, sys=43.68%, ctx=31, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=17.0MiB/s (18.9MB/s), 4607KiB/s-39.1MiB/s (4718kB/s-40.0MB/s), io=16.0MiB (16.8MB), run=307-889msec WRITE: bw=35.0MiB/s (37.7MB/s), 9215KiB/s-19.0MiB/s (9436kB/s-19.0MB/s), io=32.0MiB (33.6MB), run=630-889msec rand-rw: (groupid=0, jobs=1): err= 0: pid=32573: Sun Apr 13 07:46:33 2025 read: IOPS=3, BW=15.0MiB/s (15.7MB/s)(12.0MiB/799msec) clat (msec): min=38, max=447, avg=263.28, stdev=207.57 lat (msec): min=38, max=447, avg=263.28, stdev=207.57 clat percentiles (msec): | 1.00th=[ 39], 5.00th=[ 39], 10.00th=[ 39], 20.00th=[ 39], | 30.00th=[ 39], 40.00th=[ 305], 50.00th=[ 305], 60.00th=[ 305], | 70.00th=[ 447], 80.00th=[ 447], 90.00th=[ 447], 95.00th=[ 447], | 99.00th=[ 447], 99.50th=[ 447], 99.90th=[ 447], 99.95th=[ 447], | 99.99th=[ 447] bw ( KiB/s): min= 8192, max= 8192, per=49.95%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 50=33.33%, 500=66.67% cpu : usr=0.00%, sys=7.89%, ctx=60, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32574: Sun Apr 13 07:46:33 2025 read: IOPS=1, BW=4100KiB/s (4199kB/s)(4096KiB/999msec) clat (nsec): min=306163k, max=306163k, avg=306162878.00, stdev= 0.00 lat (nsec): min=306164k, max=306164k, avg=306163913.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 305], 5.00th=[ 305], 10.00th=[ 305], 20.00th=[ 305], | 30.00th=[ 305], 40.00th=[ 305], 50.00th=[ 305], 60.00th=[ 305], | 70.00th=[ 305], 80.00th=[ 305], 90.00th=[ 305], 95.00th=[ 305], | 99.00th=[ 305], 99.50th=[ 305], 99.90th=[ 305], 99.95th=[ 305], | 99.99th=[ 305] bw ( KiB/s): min= 8192, max= 8192, per=49.95%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=2, BW=8200KiB/s (8397kB/s)(8192KiB/999msec) clat (msec): min=136, max=532, avg=334.67, stdev=279.98 lat (msec): min=136, max=532, avg=334.94, stdev=279.98 clat percentiles (msec): | 1.00th=[ 138], 5.00th=[ 138], 10.00th=[ 138], 20.00th=[ 138], | 30.00th=[ 138], 40.00th=[ 138], 50.00th=[ 138], 60.00th=[ 531], | 70.00th=[ 531], 80.00th=[ 531], 90.00th=[ 531], 95.00th=[ 531], | 99.00th=[ 531], 99.50th=[ 531], 99.90th=[ 531], 99.95th=[ 531], | 99.99th=[ 531] bw ( KiB/s): min= 8192, max= 8192, per=24.98%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=33.33%, 750=33.33% cpu : usr=0.10%, sys=14.03%, ctx=121, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32575: Sun Apr 13 07:46:33 2025 write: IOPS=3, BW=15.1MiB/s (15.9MB/s)(12.0MiB/793msec) clat (msec): min=53, max=439, avg=261.88, stdev=194.93 lat (msec): min=53, max=439, avg=262.09, stdev=194.95 clat percentiles (msec): | 1.00th=[ 54], 5.00th=[ 54], 10.00th=[ 54], 20.00th=[ 54], | 30.00th=[ 54], 40.00th=[ 292], 50.00th=[ 292], 60.00th=[ 292], | 70.00th=[ 439], 80.00th=[ 439], 90.00th=[ 439], 95.00th=[ 439], | 99.00th=[ 439], 99.50th=[ 439], 99.90th=[ 439], 99.95th=[ 439], | 99.99th=[ 439] bw ( KiB/s): min= 8192, max= 8192, per=24.98%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 500=66.67% cpu : usr=0.13%, sys=20.96%, ctx=60, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32576: Sun Apr 13 07:46:33 2025 write: IOPS=5, BW=23.5MiB/s (24.7MB/s)(12.0MiB/510msec) clat (msec): min=34, max=339, avg=165.23, stdev=157.48 lat (msec): min=34, max=340, avg=165.52, stdev=157.50 clat percentiles (msec): | 1.00th=[ 35], 5.00th=[ 35], 10.00th=[ 35], 20.00th=[ 35], | 30.00th=[ 35], 40.00th=[ 122], 50.00th=[ 122], 60.00th=[ 122], | 70.00th=[ 342], 80.00th=[ 342], 90.00th=[ 342], 95.00th=[ 342], | 99.00th=[ 342], 99.50th=[ 342], 99.90th=[ 342], 99.95th=[ 342], | 99.99th=[ 342] bw ( KiB/s): min=16384, max=16384, per=49.95%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 50=33.33%, 250=33.33%, 500=33.33% cpu : usr=0.59%, sys=21.02%, ctx=145, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=16.0MiB/s (16.8MB/s), 4100KiB/s-15.0MiB/s (4199kB/s-15.7MB/s), io=16.0MiB (16.8MB), run=799-999msec WRITE: bw=32.0MiB/s (33.6MB/s), 8200KiB/s-23.5MiB/s (8397kB/s-24.7MB/s), io=32.0MiB (33.6MB), run=510-999msec