== sanity test 398b: DIO and buffer IO race ============== 00:52:36 (1747803156) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.34256 s, 37.5 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=15475: Wed May 21 00:52:51 2025 read: IOPS=127, BW=512KiB/s (524kB/s)(5992KiB/11710msec) clat (usec): min=51, max=167094, avg=3582.12, stdev=5146.06 lat (usec): min=51, max=167095, avg=3582.80, stdev=5146.05 clat percentiles (usec): | 1.00th=[ 62], 5.00th=[ 70], 10.00th=[ 81], 20.00th=[ 1795], | 30.00th=[ 2147], 40.00th=[ 2507], 50.00th=[ 2868], 60.00th=[ 3490], | 70.00th=[ 4293], 80.00th=[ 5342], 90.00th=[ 6652], 95.00th=[ 8094], | 99.00th=[ 12125], 99.50th=[ 15926], 99.90th=[ 36963], 99.95th=[166724], | 99.99th=[166724] bw ( KiB/s): min= 255, max= 776, per=24.61%, avg=504.52, stdev=124.46, samples=23 iops : min= 63, max= 194, avg=126.04, stdev=31.23, samples=23 write: IOPS=134, BW=538KiB/s (551kB/s)(6296KiB/11710msec) clat (usec): min=446, max=29018, avg=3957.43, stdev=3684.11 lat (usec): min=446, max=29019, avg=3958.11, stdev=3684.11 clat percentiles (usec): | 1.00th=[ 510], 5.00th=[ 537], 10.00th=[ 562], 20.00th=[ 660], | 30.00th=[ 906], 40.00th=[ 1876], 50.00th=[ 3130], 60.00th=[ 4047], | 70.00th=[ 5211], 80.00th=[ 6652], 90.00th=[ 8848], 95.00th=[11076], | 99.00th=[15533], 99.50th=[18220], 99.90th=[27395], 99.95th=[28967], | 99.99th=[28967] bw ( KiB/s): min= 295, max= 808, per=24.73%, avg=530.61, stdev=128.86, samples=23 iops : min= 73, max= 202, avg=132.57, stdev=32.27, samples=23 lat (usec) : 100=7.00%, 250=1.43%, 500=0.49%, 750=12.70%, 1000=3.26% lat (msec) : 2=7.62%, 4=30.08%, 10=32.71%, 20=4.43%, 50=0.26% lat (msec) : 250=0.03% cpu : usr=0.12%, sys=24.01%, ctx=5954, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=15477: Wed May 21 00:52:51 2025 read: IOPS=128, BW=515KiB/s (527kB/s)(5940KiB/11545msec) clat (usec): min=53, max=25749, avg=3515.21, stdev=2679.53 lat (usec): min=53, max=25750, avg=3515.84, stdev=2679.53 clat percentiles (usec): | 1.00th=[ 66], 5.00th=[ 71], 10.00th=[ 84], 20.00th=[ 1926], | 30.00th=[ 2245], 40.00th=[ 2540], 50.00th=[ 2900], 60.00th=[ 3490], | 70.00th=[ 4293], 80.00th=[ 5342], 90.00th=[ 6980], 95.00th=[ 8225], | 99.00th=[11994], 99.50th=[15270], 99.90th=[19268], 99.95th=[25822], | 99.99th=[25822] bw ( KiB/s): min= 256, max= 816, per=25.10%, avg=514.65, stdev=121.01, samples=23 iops : min= 64, max= 204, avg=128.61, stdev=30.26, samples=23 write: IOPS=137, BW=550KiB/s (563kB/s)(6348KiB/11545msec) clat (usec): min=455, max=173557, avg=3911.93, stdev=5607.22 lat (usec): min=456, max=173558, avg=3913.15, stdev=5607.31 clat percentiles (usec): | 1.00th=[ 510], 5.00th=[ 537], 10.00th=[ 570], 20.00th=[ 676], | 30.00th=[ 898], 40.00th=[ 1942], 50.00th=[ 3163], 60.00th=[ 3884], | 70.00th=[ 4752], 80.00th=[ 6128], 90.00th=[ 8586], 95.00th=[ 10552], | 99.00th=[ 15926], 99.50th=[ 22414], 99.90th=[ 31327], 99.95th=[173016], | 99.99th=[173016] bw ( KiB/s): min= 304, max= 856, per=25.54%, avg=548.04, stdev=130.09, samples=23 iops : min= 76, max= 214, avg=136.96, stdev=32.56, samples=23 lat (usec) : 100=6.51%, 250=1.04%, 500=0.23%, 750=12.79%, 1000=3.42% lat (msec) : 2=7.52%, 4=32.71%, 10=31.67%, 20=3.78%, 50=0.29% lat (msec) : 250=0.03% cpu : usr=0.20%, sys=23.79%, ctx=5867, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=15478: Wed May 21 00:52:51 2025 read: IOPS=128, BW=515KiB/s (528kB/s)(6028KiB/11697msec) clat (usec): min=55, max=40741, avg=3416.79, stdev=2893.94 lat (usec): min=55, max=40741, avg=3417.66, stdev=2893.89 clat percentiles (usec): | 1.00th=[ 65], 5.00th=[ 71], 10.00th=[ 82], 20.00th=[ 1811], | 30.00th=[ 2147], 40.00th=[ 2474], 50.00th=[ 2835], 60.00th=[ 3261], | 70.00th=[ 4113], 80.00th=[ 5276], 90.00th=[ 6849], 95.00th=[ 8225], | 99.00th=[11863], 99.50th=[14615], 99.90th=[27657], 99.95th=[40633], | 99.99th=[40633] bw ( KiB/s): min= 120, max= 768, per=24.88%, avg=510.09, stdev=142.57, samples=23 iops : min= 30, max= 192, avg=127.43, stdev=35.68, samples=23 write: IOPS=133, BW=535KiB/s (548kB/s)(6260KiB/11697msec) clat (usec): min=412, max=226454, avg=4108.09, stdev=6841.60 lat (usec): min=412, max=226457, avg=4108.95, stdev=6841.69 clat percentiles (usec): | 1.00th=[ 502], 5.00th=[ 529], 10.00th=[ 562], 20.00th=[ 668], | 30.00th=[ 1020], 40.00th=[ 1926], 50.00th=[ 3228], 60.00th=[ 4146], | 70.00th=[ 5014], 80.00th=[ 6456], 90.00th=[ 8717], 95.00th=[ 10814], | 99.00th=[ 15795], 99.50th=[ 22152], 99.90th=[ 44827], 99.95th=[225444], | 99.99th=[225444] bw ( KiB/s): min= 152, max= 776, per=24.59%, avg=527.78, stdev=139.81, samples=23 iops : min= 38, max= 194, avg=131.87, stdev=34.95, samples=23 lat (usec) : 100=6.87%, 250=1.60%, 500=0.59%, 750=11.72%, 1000=3.12% lat (msec) : 2=8.66%, 4=30.96%, 10=32.10%, 20=3.91%, 50=0.46% lat (msec) : 250=0.03% cpu : usr=0.15%, sys=23.57%, ctx=5996, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=15480: Wed May 21 00:52:51 2025 read: IOPS=129, BW=520KiB/s (532kB/s)(6052KiB/11645msec) clat (usec): min=51, max=49027, avg=3476.66, stdev=3256.59 lat (usec): min=52, max=49027, avg=3477.42, stdev=3256.69 clat percentiles (usec): | 1.00th=[ 63], 5.00th=[ 72], 10.00th=[ 83], 20.00th=[ 1762], | 30.00th=[ 2180], 40.00th=[ 2474], 50.00th=[ 2933], 60.00th=[ 3490], | 70.00th=[ 4228], 80.00th=[ 5211], 90.00th=[ 6652], 95.00th=[ 8225], | 99.00th=[13173], 99.50th=[15401], 99.90th=[42730], 99.95th=[49021], | 99.99th=[49021] bw ( KiB/s): min= 104, max= 728, per=25.10%, avg=514.52, stdev=136.16, samples=23 iops : min= 26, max= 182, avg=128.52, stdev=34.09, samples=23 write: IOPS=133, BW=536KiB/s (548kB/s)(6236KiB/11645msec) clat (usec): min=440, max=200974, avg=4018.90, stdev=6317.29 lat (usec): min=440, max=200975, avg=4019.76, stdev=6317.30 clat percentiles (usec): | 1.00th=[ 502], 5.00th=[ 529], 10.00th=[ 553], 20.00th=[ 635], | 30.00th=[ 848], 40.00th=[ 1893], 50.00th=[ 3032], 60.00th=[ 3982], | 70.00th=[ 4948], 80.00th=[ 6521], 90.00th=[ 8848], 95.00th=[ 11207], | 99.00th=[ 17171], 99.50th=[ 21103], 99.90th=[ 42206], 99.95th=[200279], | 99.99th=[200279] bw ( KiB/s): min= 120, max= 776, per=24.72%, avg=530.48, stdev=154.33, samples=23 iops : min= 30, max= 194, avg=132.52, stdev=38.60, samples=23 lat (usec) : 100=7.52%, 250=1.37%, 500=0.59%, 750=13.57%, 1000=3.19% lat (msec) : 2=6.77%, 4=31.02%, 10=31.15%, 20=4.36%, 50=0.42% lat (msec) : 250=0.03% cpu : usr=0.21%, sys=23.23%, ctx=5805, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2051KiB/s (2100kB/s), 512KiB/s-520KiB/s (524kB/s-532kB/s), io=23.4MiB (24.6MB), run=11545-11710msec WRITE: bw=2147KiB/s (2198kB/s), 535KiB/s-550KiB/s (548kB/s-563kB/s), io=24.6MiB (25.7MB), run=11545-11710msec rand-rw: (groupid=0, jobs=1): err= 0: pid=15476: Wed May 21 00:53:12 2025 read: IOPS=46, BW=185KiB/s (189kB/s)(5992KiB/32391msec) clat (usec): min=1530, max=171301, avg=3679.81, stdev=5045.84 lat (usec): min=1530, max=171302, avg=3680.54, stdev=5045.85 clat percentiles (usec): | 1.00th=[ 1663], 5.00th=[ 1811], 10.00th=[ 1893], 20.00th=[ 1991], | 30.00th=[ 2089], 40.00th=[ 2278], 50.00th=[ 2737], 60.00th=[ 3261], | 70.00th=[ 3654], 80.00th=[ 4228], 90.00th=[ 6587], 95.00th=[ 8848], | 99.00th=[ 13304], 99.50th=[ 14746], 99.90th=[ 35914], 99.95th=[170918], | 99.99th=[170918] bw ( KiB/s): min= 71, max= 352, per=25.17%, avg=185.02, stdev=65.55, samples=64 iops : min= 17, max= 88, avg=46.17, stdev=16.41, samples=64 write: IOPS=48, BW=194KiB/s (199kB/s)(6296KiB/32391msec) clat (usec): min=8296, max=83694, avg=17002.39, stdev=5955.93 lat (usec): min=8297, max=83695, avg=17003.24, stdev=5955.88 clat percentiles (usec): | 1.00th=[ 9503], 5.00th=[10552], 10.00th=[11076], 20.00th=[12125], | 30.00th=[13173], 40.00th=[14484], 50.00th=[15926], 60.00th=[17171], | 70.00th=[18482], 80.00th=[20317], 90.00th=[25035], 95.00th=[28967], | 99.00th=[34866], 99.50th=[40109], 99.90th=[46924], 99.95th=[83362], | 99.99th=[83362] bw ( KiB/s): min= 56, max= 264, per=25.18%, avg=193.89, stdev=50.71, samples=64 iops : min= 14, max= 66, avg=48.39, stdev=12.66, samples=64 lat (msec) : 2=10.06%, 4=27.77%, 10=10.64%, 20=40.69%, 50=10.77% lat (msec) : 100=0.03%, 250=0.03% cpu : usr=0.11%, sys=6.01%, ctx=3630, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=15479: Wed May 21 00:53:12 2025 read: IOPS=45, BW=182KiB/s (186kB/s)(5940KiB/32636msec) clat (usec): min=1555, max=178446, avg=3727.86, stdev=5364.29 lat (usec): min=1555, max=178448, avg=3728.56, stdev=5364.30 clat percentiles (usec): | 1.00th=[ 1663], 5.00th=[ 1795], 10.00th=[ 1860], 20.00th=[ 1958], | 30.00th=[ 2057], 40.00th=[ 2278], 50.00th=[ 2802], 60.00th=[ 3228], | 70.00th=[ 3589], 80.00th=[ 4293], 90.00th=[ 6390], 95.00th=[ 8979], | 99.00th=[ 14746], 99.50th=[ 19006], 99.90th=[ 42206], 99.95th=[179307], | 99.99th=[179307] bw ( KiB/s): min= 16, max= 328, per=24.78%, avg=182.11, stdev=70.36, samples=65 iops : min= 4, max= 82, avg=45.49, stdev=17.57, samples=65 write: IOPS=48, BW=195KiB/s (199kB/s)(6348KiB/32636msec) clat (usec): min=8499, max=69920, avg=17003.19, stdev=5877.29 lat (usec): min=8500, max=69921, avg=17004.00, stdev=5877.29 clat percentiles (usec): | 1.00th=[ 9372], 5.00th=[10421], 10.00th=[10945], 20.00th=[12125], | 30.00th=[13173], 40.00th=[14615], 50.00th=[16188], 60.00th=[17433], | 70.00th=[18482], 80.00th=[20317], 90.00th=[24773], 95.00th=[28705], | 99.00th=[36439], 99.50th=[40109], 99.90th=[46924], 99.95th=[69731], | 99.99th=[69731] bw ( KiB/s): min= 64, max= 256, per=25.22%, avg=194.17, stdev=50.89, samples=65 iops : min= 16, max= 64, avg=48.51, stdev=12.70, samples=65 lat (msec) : 2=11.59%, 4=25.85%, 10=11.07%, 20=40.53%, 50=10.90% lat (msec) : 100=0.03%, 250=0.03% cpu : usr=0.13%, sys=5.89%, ctx=3650, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=15481: Wed May 21 00:53:12 2025 read: IOPS=46, BW=185KiB/s (190kB/s)(6028KiB/32569msec) clat (usec): min=1471, max=36213, avg=3859.95, stdev=3122.11 lat (usec): min=1472, max=36214, avg=3860.68, stdev=3122.08 clat percentiles (usec): | 1.00th=[ 1713], 5.00th=[ 1827], 10.00th=[ 1909], 20.00th=[ 2008], | 30.00th=[ 2147], 40.00th=[ 2376], 50.00th=[ 2999], 60.00th=[ 3359], | 70.00th=[ 3785], 80.00th=[ 4948], 90.00th=[ 7046], 95.00th=[ 9634], | 99.00th=[15926], 99.50th=[23725], 99.90th=[30278], 99.95th=[36439], | 99.99th=[36439] bw ( KiB/s): min= 48, max= 312, per=25.19%, avg=185.14, stdev=57.59, samples=65 iops : min= 12, max= 78, avg=46.22, stdev=14.40, samples=65 write: IOPS=48, BW=192KiB/s (197kB/s)(6260KiB/32569msec) clat (msec): min=8, max=222, avg=17.02, stdev= 7.75 lat (msec): min=8, max=222, avg=17.02, stdev= 7.75 clat percentiles (msec): | 1.00th=[ 10], 5.00th=[ 11], 10.00th=[ 11], 20.00th=[ 13], | 30.00th=[ 14], 40.00th=[ 15], 50.00th=[ 17], 60.00th=[ 18], | 70.00th=[ 19], 80.00th=[ 21], 90.00th=[ 25], 95.00th=[ 29], | 99.00th=[ 37], 99.50th=[ 41], 99.90th=[ 56], 99.95th=[ 224], | 99.99th=[ 224] bw ( KiB/s): min= 48, max= 264, per=24.92%, avg=191.91, stdev=53.08, samples=65 iops : min= 12, max= 66, avg=47.91, stdev=13.26, samples=65 lat (msec) : 2=9.60%, 4=26.69%, 10=12.24%, 20=40.69%, 50=10.68% lat (msec) : 100=0.07%, 250=0.03% cpu : usr=0.05%, sys=6.09%, ctx=3703, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=15482: Wed May 21 00:53:12 2025 read: IOPS=46, BW=187KiB/s (191kB/s)(6052KiB/32443msec) clat (usec): min=1440, max=42282, avg=3687.14, stdev=2902.84 lat (usec): min=1441, max=42282, avg=3687.85, stdev=2902.86 clat percentiles (usec): | 1.00th=[ 1663], 5.00th=[ 1795], 10.00th=[ 1876], 20.00th=[ 1991], | 30.00th=[ 2114], 40.00th=[ 2311], 50.00th=[ 2835], 60.00th=[ 3294], | 70.00th=[ 3654], 80.00th=[ 4359], 90.00th=[ 6521], 95.00th=[ 9503], | 99.00th=[14484], 99.50th=[16581], 99.90th=[35914], 99.95th=[42206], | 99.99th=[42206] bw ( KiB/s): min= 40, max= 383, per=25.25%, avg=185.56, stdev=70.67, samples=64 iops : min= 10, max= 95, avg=46.34, stdev=17.64, samples=64 write: IOPS=48, BW=192KiB/s (197kB/s)(6236KiB/32443msec) clat (msec): min=8, max=197, avg=17.16, stdev= 7.35 lat (msec): min=8, max=197, avg=17.16, stdev= 7.35 clat percentiles (msec): | 1.00th=[ 10], 5.00th=[ 11], 10.00th=[ 12], 20.00th=[ 12], | 30.00th=[ 14], 40.00th=[ 15], 50.00th=[ 17], 60.00th=[ 18], | 70.00th=[ 19], 80.00th=[ 21], 90.00th=[ 25], 95.00th=[ 29], | 99.00th=[ 35], 99.50th=[ 39], 99.90th=[ 70], 99.95th=[ 199], | 99.99th=[ 199] bw ( KiB/s): min= 56, max= 272, per=24.93%, avg=191.94, stdev=50.79, samples=64 iops : min= 14, max= 68, avg=47.94, stdev=12.72, samples=64 lat (msec) : 2=10.19%, 4=27.70%, 10=10.71%, 20=39.68%, 50=11.65% lat (msec) : 100=0.03%, 250=0.03% cpu : usr=0.10%, sys=6.05%, ctx=3668, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=736KiB/s (753kB/s), 182KiB/s-187KiB/s (186kB/s-191kB/s), io=23.4MiB (24.6MB), run=32391-32636msec WRITE: bw=770KiB/s (789kB/s), 192KiB/s-195KiB/s (197kB/s-199kB/s), io=24.6MiB (25.7MB), run=32391-32636msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=15519: Wed May 21 00:53:15 2025 read: IOPS=164, BW=2631KiB/s (2694kB/s)(5920KiB/2250msec) clat (usec): min=55, max=14142, avg=2710.31, stdev=2477.42 lat (usec): min=55, max=14143, avg=2711.25, stdev=2477.67 clat percentiles (usec): | 1.00th=[ 60], 5.00th=[ 69], 10.00th=[ 72], 20.00th=[ 83], | 30.00th=[ 99], 40.00th=[ 2114], 50.00th=[ 2540], 60.00th=[ 2999], | 70.00th=[ 3752], 80.00th=[ 4817], 90.00th=[ 6063], 95.00th=[ 7046], | 99.00th=[ 9241], 99.50th=[11076], 99.90th=[14091], 99.95th=[14091], | 99.99th=[14091] bw ( KiB/s): min= 2299, max= 2976, per=26.05%, avg=2670.75, stdev=340.97, samples=4 iops : min= 143, max= 186, avg=166.75, stdev=21.56, samples=4 write: IOPS=176, BW=2830KiB/s (2898kB/s)(6368KiB/2250msec) clat (usec): min=625, max=14901, avg=3078.26, stdev=2846.17 lat (usec): min=626, max=14902, avg=3081.69, stdev=2844.97 clat percentiles (usec): | 1.00th=[ 676], 5.00th=[ 742], 10.00th=[ 791], 20.00th=[ 865], | 30.00th=[ 979], 40.00th=[ 1156], 50.00th=[ 1631], 60.00th=[ 3032], | 70.00th=[ 3720], 80.00th=[ 5473], 90.00th=[ 7046], 95.00th=[ 8848], | 99.00th=[12518], 99.50th=[13698], 99.90th=[14877], 99.95th=[14877], | 99.99th=[14877] bw ( KiB/s): min= 2331, max= 3392, per=26.84%, avg=2894.75, stdev=439.73, samples=4 iops : min= 145, max= 212, avg=180.75, stdev=27.78, samples=4 lat (usec) : 100=14.71%, 250=1.95%, 750=3.12%, 1000=13.41% lat (msec) : 2=12.11%, 4=26.43%, 10=26.30%, 20=1.95% cpu : usr=0.31%, sys=29.48%, ctx=1538, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=15521: Wed May 21 00:53:15 2025 read: IOPS=154, BW=2477KiB/s (2536kB/s)(5744KiB/2319msec) clat (usec): min=61, max=15843, avg=2750.03, stdev=2580.37 lat (usec): min=61, max=15843, avg=2751.63, stdev=2580.99 clat percentiles (usec): | 1.00th=[ 63], 5.00th=[ 70], 10.00th=[ 72], 20.00th=[ 83], | 30.00th=[ 115], 40.00th=[ 2278], 50.00th=[ 2606], 60.00th=[ 3195], | 70.00th=[ 3621], 80.00th=[ 4555], 90.00th=[ 6390], 95.00th=[ 7504], | 99.00th=[ 9503], 99.50th=[13042], 99.90th=[15795], 99.95th=[15795], | 99.99th=[15795] bw ( KiB/s): min= 1756, max= 3104, per=24.26%, avg=2487.00, stdev=561.73, samples=4 iops : min= 109, max= 194, avg=155.25, stdev=35.43, samples=4 write: IOPS=176, BW=2822KiB/s (2890kB/s)(6544KiB/2319msec) clat (usec): min=621, max=46972, avg=3102.32, stdev=3945.48 lat (usec): min=622, max=46974, avg=3103.32, stdev=3945.47 clat percentiles (usec): | 1.00th=[ 660], 5.00th=[ 717], 10.00th=[ 766], 20.00th=[ 840], | 30.00th=[ 947], 40.00th=[ 1106], 50.00th=[ 1418], 60.00th=[ 2638], | 70.00th=[ 3523], 80.00th=[ 4817], 90.00th=[ 6980], 95.00th=[ 8848], | 99.00th=[14091], 99.50th=[24511], 99.90th=[46924], 99.95th=[46924], | 99.99th=[46924] bw ( KiB/s): min= 1948, max= 3200, per=26.17%, avg=2823.00, stdev=591.66, samples=4 iops : min= 121, max= 200, avg=176.25, stdev=37.35, samples=4 lat (usec) : 100=13.15%, 250=2.86%, 500=0.26%, 750=4.69%, 1000=14.06% lat (msec) : 2=11.85%, 4=28.65%, 10=22.66%, 20=1.30%, 50=0.52% cpu : usr=0.13%, sys=27.91%, ctx=1447, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=15523: Wed May 21 00:53:15 2025 read: IOPS=167, BW=2678KiB/s (2742kB/s)(6256KiB/2336msec) clat (usec): min=56, max=17433, avg=2810.20, stdev=2724.92 lat (usec): min=57, max=17434, avg=2810.74, stdev=2724.93 clat percentiles (usec): | 1.00th=[ 59], 5.00th=[ 64], 10.00th=[ 70], 20.00th=[ 82], | 30.00th=[ 97], 40.00th=[ 2180], 50.00th=[ 2606], 60.00th=[ 3195], | 70.00th=[ 3884], 80.00th=[ 5014], 90.00th=[ 6063], 95.00th=[ 7111], | 99.00th=[11469], 99.50th=[16188], 99.90th=[17433], 99.95th=[17433], | 99.99th=[17433] bw ( KiB/s): min= 1984, max= 2880, per=25.51%, avg=2616.00, stdev=424.83, samples=4 iops : min= 124, max= 180, avg=163.50, stdev=26.55, samples=4 write: IOPS=161, BW=2582KiB/s (2644kB/s)(6032KiB/2336msec) clat (usec): min=663, max=34699, avg=3058.65, stdev=3261.83 lat (usec): min=664, max=34700, avg=3059.49, stdev=3261.82 clat percentiles (usec): | 1.00th=[ 676], 5.00th=[ 717], 10.00th=[ 758], 20.00th=[ 840], | 30.00th=[ 947], 40.00th=[ 1106], 50.00th=[ 1532], 60.00th=[ 2474], | 70.00th=[ 3720], 80.00th=[ 5211], 90.00th=[ 7242], 95.00th=[ 8455], | 99.00th=[13566], 99.50th=[16188], 99.90th=[34866], 99.95th=[34866], | 99.99th=[34866] bw ( KiB/s): min= 2080, max= 3008, per=24.10%, avg=2600.00, stdev=398.29, samples=4 iops : min= 130, max= 188, avg=162.50, stdev=24.89, samples=4 lat (usec) : 100=15.62%, 250=2.34%, 500=0.65%, 750=4.56%, 1000=12.50% lat (msec) : 2=10.03%, 4=25.91%, 10=25.78%, 20=2.47%, 50=0.13% cpu : usr=0.13%, sys=27.11%, ctx=1564, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=15525: Wed May 21 00:53:15 2025 read: IOPS=163, BW=2609KiB/s (2672kB/s)(6032KiB/2312msec) clat (usec): min=58, max=14839, avg=2652.68, stdev=2533.73 lat (usec): min=58, max=14840, avg=2653.83, stdev=2533.48 clat percentiles (usec): | 1.00th=[ 61], 5.00th=[ 64], 10.00th=[ 69], 20.00th=[ 76], | 30.00th=[ 86], 40.00th=[ 1958], 50.00th=[ 2540], 60.00th=[ 3032], | 70.00th=[ 3916], 80.00th=[ 4948], 90.00th=[ 6194], 95.00th=[ 6849], | 99.00th=[ 9110], 99.50th=[10028], 99.90th=[14877], 99.95th=[14877], | 99.99th=[14877] bw ( KiB/s): min= 1888, max= 3008, per=25.05%, avg=2568.00, stdev=489.07, samples=4 iops : min= 118, max= 188, avg=160.50, stdev=30.57, samples=4 write: IOPS=169, BW=2706KiB/s (2771kB/s)(6256KiB/2312msec) clat (usec): min=639, max=32737, avg=3163.37, stdev=3585.73 lat (usec): min=640, max=32739, avg=3165.13, stdev=3585.81 clat percentiles (usec): | 1.00th=[ 668], 5.00th=[ 734], 10.00th=[ 783], 20.00th=[ 865], | 30.00th=[ 963], 40.00th=[ 1123], 50.00th=[ 1401], 60.00th=[ 2638], | 70.00th=[ 3556], 80.00th=[ 5276], 90.00th=[ 7504], 95.00th=[ 9896], | 99.00th=[13304], 99.50th=[30016], 99.90th=[32637], 99.95th=[32637], | 99.99th=[32637] bw ( KiB/s): min= 1984, max= 3296, per=24.77%, avg=2672.00, stdev=555.79, samples=4 iops : min= 124, max= 206, avg=167.00, stdev=34.74, samples=4 lat (usec) : 100=16.80%, 250=1.95%, 500=0.13%, 750=3.26%, 1000=14.45% lat (msec) : 2=11.59%, 4=23.70%, 10=25.39%, 20=2.47%, 50=0.26% cpu : usr=0.00%, sys=28.52%, ctx=1427, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=10.0MiB/s (10.5MB/s), 2477KiB/s-2678KiB/s (2536kB/s-2742kB/s), io=23.4MiB (24.5MB), run=2250-2336msec WRITE: bw=10.5MiB/s (11.0MB/s), 2582KiB/s-2830KiB/s (2644kB/s-2898kB/s), io=24.6MiB (25.8MB), run=2250-2336msec rand-rw: (groupid=0, jobs=1): err= 0: pid=15518: Wed May 21 00:53:21 2025 read: IOPS=44, BW=717KiB/s (734kB/s)(5920KiB/8260msec) clat (usec): min=1642, max=25649, avg=3764.90, stdev=3019.53 lat (usec): min=1643, max=25650, avg=3765.52, stdev=3019.53 clat percentiles (usec): | 1.00th=[ 1876], 5.00th=[ 2008], 10.00th=[ 2057], 20.00th=[ 2147], | 30.00th=[ 2245], 40.00th=[ 2376], 50.00th=[ 2507], 60.00th=[ 2802], | 70.00th=[ 3556], 80.00th=[ 5080], 90.00th=[ 6390], 95.00th=[ 9372], | 99.00th=[17433], 99.50th=[18482], 99.90th=[25560], 99.95th=[25560], | 99.99th=[25560] bw ( KiB/s): min= 384, max= 1152, per=24.45%, avg=701.69, stdev=201.80, samples=16 iops : min= 24, max= 72, avg=43.75, stdev=12.50, samples=16 write: IOPS=48, BW=771KiB/s (789kB/s)(6368KiB/8260msec) clat (usec): min=10212, max=48746, avg=17115.52, stdev=4937.31 lat (usec): min=10213, max=48748, avg=17116.70, stdev=4937.25 clat percentiles (usec): | 1.00th=[10683], 5.00th=[11338], 10.00th=[11863], 20.00th=[12911], | 30.00th=[14222], 40.00th=[15401], 50.00th=[16581], 60.00th=[17433], | 70.00th=[18482], 80.00th=[19792], 90.00th=[23200], 95.00th=[27132], | 99.00th=[34341], 99.50th=[35914], 99.90th=[48497], 99.95th=[48497], | 99.99th=[48497] bw ( KiB/s): min= 480, max= 960, per=25.36%, avg=765.75, stdev=178.04, samples=16 iops : min= 30, max= 60, avg=47.75, stdev=11.08, samples=16 lat (msec) : 2=2.34%, 4=33.98%, 10=9.77%, 20=44.66%, 50=9.24% cpu : usr=0.10%, sys=6.32%, ctx=951, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=15520: Wed May 21 00:53:21 2025 read: IOPS=43, BW=688KiB/s (705kB/s)(5744KiB/8345msec) clat (usec): min=1827, max=21209, avg=3661.79, stdev=2781.70 lat (usec): min=1828, max=21210, avg=3662.43, stdev=2781.69 clat percentiles (usec): | 1.00th=[ 1893], 5.00th=[ 1975], 10.00th=[ 2040], 20.00th=[ 2147], | 30.00th=[ 2212], 40.00th=[ 2311], 50.00th=[ 2442], 60.00th=[ 2704], | 70.00th=[ 3425], 80.00th=[ 4817], 90.00th=[ 6259], 95.00th=[ 9372], | 99.00th=[14222], 99.50th=[20317], 99.90th=[21103], 99.95th=[21103], | 99.99th=[21103] bw ( KiB/s): min= 288, max= 1152, per=24.03%, avg=689.69, stdev=240.34, samples=16 iops : min= 18, max= 72, avg=43.00, stdev=14.97, samples=16 write: IOPS=49, BW=784KiB/s (803kB/s)(6544KiB/8345msec) clat (usec): min=10030, max=49259, avg=17075.52, stdev=5218.64 lat (usec): min=10030, max=49261, avg=17076.53, stdev=5218.67 clat percentiles (usec): | 1.00th=[10552], 5.00th=[10945], 10.00th=[11469], 20.00th=[12780], | 30.00th=[13566], 40.00th=[15008], 50.00th=[16319], 60.00th=[17433], | 70.00th=[18744], 80.00th=[20317], 90.00th=[23462], 95.00th=[27395], | 99.00th=[34341], 99.50th=[36439], 99.90th=[49021], 99.95th=[49021], | 99.99th=[49021] bw ( KiB/s): min= 512, max= 960, per=25.63%, avg=773.75, stdev=179.07, samples=16 iops : min= 32, max= 60, avg=48.25, stdev=11.14, samples=16 lat (msec) : 2=2.99%, 4=31.77%, 10=9.90%, 20=43.88%, 50=11.46% cpu : usr=0.08%, sys=6.17%, ctx=903, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=15522: Wed May 21 00:53:21 2025 read: IOPS=49, BW=795KiB/s (814kB/s)(6256KiB/7871msec) clat (usec): min=1794, max=19183, avg=3509.28, stdev=2646.57 lat (usec): min=1795, max=19183, avg=3509.90, stdev=2646.57 clat percentiles (usec): | 1.00th=[ 1876], 5.00th=[ 2008], 10.00th=[ 2057], 20.00th=[ 2147], | 30.00th=[ 2212], 40.00th=[ 2278], 50.00th=[ 2409], 60.00th=[ 2606], | 70.00th=[ 2966], 80.00th=[ 4424], 90.00th=[ 6128], 95.00th=[ 8455], | 99.00th=[15533], 99.50th=[19006], 99.90th=[19268], 99.95th=[19268], | 99.99th=[19268] bw ( KiB/s): min= 352, max= 1504, per=28.16%, avg=808.20, stdev=331.74, samples=15 iops : min= 22, max= 94, avg=50.40, stdev=20.75, samples=15 write: IOPS=47, BW=766KiB/s (785kB/s)(6032KiB/7871msec) clat (usec): min=9367, max=38001, avg=17135.61, stdev=5219.54 lat (usec): min=9368, max=38002, avg=17136.66, stdev=5219.59 clat percentiles (usec): | 1.00th=[10552], 5.00th=[11076], 10.00th=[11600], 20.00th=[12518], | 30.00th=[13435], 40.00th=[15008], 50.00th=[16450], 60.00th=[17695], | 70.00th=[18482], 80.00th=[20317], 90.00th=[24773], 95.00th=[27919], | 99.00th=[33817], 99.50th=[35914], 99.90th=[38011], 99.95th=[38011], | 99.99th=[38011] bw ( KiB/s): min= 480, max= 928, per=25.22%, avg=761.33, stdev=168.62, samples=15 iops : min= 30, max= 58, avg=47.47, stdev=10.58, samples=15 lat (msec) : 2=2.47%, 4=36.72%, 10=10.03%, 20=39.71%, 50=11.07% cpu : usr=0.13%, sys=6.65%, ctx=932, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=15524: Wed May 21 00:53:21 2025 read: IOPS=46, BW=750KiB/s (767kB/s)(6032KiB/8048msec) clat (usec): min=1769, max=26578, avg=3668.75, stdev=3117.56 lat (usec): min=1770, max=26578, avg=3669.41, stdev=3117.55 clat percentiles (usec): | 1.00th=[ 1909], 5.00th=[ 2008], 10.00th=[ 2057], 20.00th=[ 2114], | 30.00th=[ 2180], 40.00th=[ 2278], 50.00th=[ 2376], 60.00th=[ 2638], | 70.00th=[ 3228], 80.00th=[ 4883], 90.00th=[ 6259], 95.00th=[ 9765], | 99.00th=[18482], 99.50th=[19006], 99.90th=[26608], 99.95th=[26608], | 99.99th=[26608] bw ( KiB/s): min= 416, max= 1248, per=25.84%, avg=741.75, stdev=218.03, samples=16 iops : min= 26, max= 78, avg=46.25, stdev=13.63, samples=16 write: IOPS=48, BW=777KiB/s (796kB/s)(6256KiB/8048msec) clat (usec): min=9808, max=38431, avg=17010.30, stdev=4860.91 lat (usec): min=9809, max=38434, avg=17011.50, stdev=4861.01 clat percentiles (usec): | 1.00th=[10421], 5.00th=[11076], 10.00th=[11600], 20.00th=[12518], | 30.00th=[13829], 40.00th=[15139], 50.00th=[16581], 60.00th=[17957], | 70.00th=[18744], 80.00th=[20055], 90.00th=[22938], 95.00th=[26870], | 99.00th=[33424], 99.50th=[34341], 99.90th=[38536], 99.95th=[38536], | 99.99th=[38536] bw ( KiB/s): min= 480, max= 992, per=25.70%, avg=775.75, stdev=167.72, samples=16 iops : min= 30, max= 62, avg=48.37, stdev=10.40, samples=16 lat (msec) : 2=1.95%, 4=35.68%, 10=9.51%, 20=42.58%, 50=10.29% cpu : usr=0.09%, sys=6.38%, ctx=923, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2870KiB/s (2939kB/s), 688KiB/s-795KiB/s (705kB/s-814kB/s), io=23.4MiB (24.5MB), run=7871-8345msec WRITE: bw=3020KiB/s (3092kB/s), 766KiB/s-784KiB/s (785kB/s-803kB/s), io=24.6MiB (25.8MB), run=7871-8345msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=15537: Wed May 21 00:53:23 2025 read: IOPS=15, BW=15.7MiB/s (16.5MB/s)(7168KiB/446msec) clat (usec): min=276, max=35359, avg=15845.74, stdev=15114.08 lat (usec): min=277, max=35360, avg=15846.46, stdev=15114.08 clat percentiles (usec): | 1.00th=[ 277], 5.00th=[ 277], 10.00th=[ 277], 20.00th=[ 429], | 30.00th=[ 433], 40.00th=[ 433], 50.00th=[20841], 60.00th=[25035], | 70.00th=[25035], 80.00th=[28705], 90.00th=[35390], 95.00th=[35390], | 99.00th=[35390], 99.50th=[35390], 99.90th=[35390], 99.95th=[35390], | 99.99th=[35390] write: IOPS=11, BW=11.2MiB/s (11.8MB/s)(5120KiB/446msec) clat (usec): min=27938, max=77580, avg=44341.17, stdev=20151.76 lat (usec): min=27950, max=77599, avg=44373.18, stdev=20145.09 clat percentiles (usec): | 1.00th=[27919], 5.00th=[27919], 10.00th=[27919], 20.00th=[27919], | 30.00th=[28705], 40.00th=[28705], 50.00th=[42730], 60.00th=[42730], | 70.00th=[44827], 80.00th=[44827], 90.00th=[77071], 95.00th=[77071], | 99.00th=[77071], 99.50th=[77071], 99.90th=[77071], 99.95th=[77071], | 99.99th=[77071] lat (usec) : 500=25.00% lat (msec) : 50=66.67%, 100=8.33% cpu : usr=0.00%, sys=28.76%, ctx=409, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=15538: Wed May 21 00:53:23 2025 read: IOPS=3, BW=3513KiB/s (3597kB/s)(2048KiB/583msec) clat (usec): min=37156, max=45010, avg=41083.22, stdev=5553.54 lat (usec): min=37157, max=45011, avg=41084.07, stdev=5553.65 clat percentiles (usec): | 1.00th=[36963], 5.00th=[36963], 10.00th=[36963], 20.00th=[36963], | 30.00th=[36963], 40.00th=[36963], 50.00th=[36963], 60.00th=[44827], | 70.00th=[44827], 80.00th=[44827], 90.00th=[44827], 95.00th=[44827], | 99.00th=[44827], 99.50th=[44827], 99.90th=[44827], 99.95th=[44827], | 99.99th=[44827] bw ( KiB/s): min= 3531, max= 3531, per=10.05%, avg=3531.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 write: IOPS=17, BW=17.2MiB/s (17.0MB/s)(10.0MiB/583msec) clat (usec): min=19212, max=90484, avg=45993.05, stdev=20179.84 lat (usec): min=19273, max=90545, avg=46100.66, stdev=20156.06 clat percentiles (usec): | 1.00th=[19268], 5.00th=[19268], 10.00th=[19268], 20.00th=[36439], | 30.00th=[36963], 40.00th=[38011], 50.00th=[38011], 60.00th=[40633], | 70.00th=[46400], 80.00th=[46400], 90.00th=[70779], 95.00th=[90702], | 99.00th=[90702], 99.50th=[90702], 99.90th=[90702], 99.95th=[90702], | 99.99th=[90702] bw ( KiB/s): min=15889, max=15889, per=32.31%, avg=15889.00, stdev= 0.00, samples=1 iops : min= 15, max= 15, avg=15.00, stdev= 0.00, samples=1 lat (msec) : 20=8.33%, 50=75.00%, 100=16.67% cpu : usr=0.17%, sys=33.33%, ctx=213, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=15539: Wed May 21 00:53:23 2025 read: IOPS=16, BW=16.4MiB/s (17.2MB/s)(6144KiB/365msec) clat (usec): min=271, max=58310, avg=25671.08, stdev=22197.30 lat (usec): min=271, max=58310, avg=25671.81, stdev=22197.28 clat percentiles (usec): | 1.00th=[ 273], 5.00th=[ 273], 10.00th=[ 273], 20.00th=[ 416], | 30.00th=[ 416], 40.00th=[29492], 50.00th=[29492], 60.00th=[32113], | 70.00th=[33424], 80.00th=[33424], 90.00th=[58459], 95.00th=[58459], | 99.00th=[58459], 99.50th=[58459], 99.90th=[58459], 99.95th=[58459], | 99.99th=[58459] write: IOPS=16, BW=16.4MiB/s (17.2MB/s)(6144KiB/365msec) clat (usec): min=22822, max=61844, avg=33323.35, stdev=14306.59 lat (usec): min=22854, max=61899, avg=33359.67, stdev=14317.24 clat percentiles (usec): | 1.00th=[22938], 5.00th=[22938], 10.00th=[22938], 20.00th=[25560], | 30.00th=[25560], 40.00th=[28443], 50.00th=[28443], 60.00th=[30540], | 70.00th=[30802], 80.00th=[30802], 90.00th=[61604], 95.00th=[61604], | 99.00th=[61604], 99.50th=[61604], 99.90th=[61604], 99.95th=[61604], | 99.99th=[61604] lat (usec) : 500=16.67% lat (msec) : 50=66.67%, 100=16.67% cpu : usr=0.00%, sys=40.66%, ctx=484, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=15541: Wed May 21 00:53:23 2025 read: IOPS=11, BW=11.9MiB/s (12.5MB/s)(5120KiB/421msec) clat (usec): min=418, max=52888, avg=13797.83, stdev=22210.88 lat (usec): min=419, max=52889, avg=13798.58, stdev=22210.91 clat percentiles (usec): | 1.00th=[ 420], 5.00th=[ 420], 10.00th=[ 420], 20.00th=[ 420], | 30.00th=[ 453], 40.00th=[ 453], 50.00th=[ 5211], 60.00th=[ 5211], | 70.00th=[10028], 80.00th=[10028], 90.00th=[52691], 95.00th=[52691], | 99.00th=[52691], 99.50th=[52691], 99.90th=[52691], 99.95th=[52691], | 99.99th=[52691] write: IOPS=16, BW=16.6MiB/s (17.4MB/s)(7168KiB/421msec) clat (msec): min=21, max=108, avg=48.02, stdev=29.31 lat (msec): min=22, max=108, avg=48.20, stdev=29.17 clat percentiles (msec): | 1.00th=[ 22], 5.00th=[ 22], 10.00th=[ 22], 20.00th=[ 26], | 30.00th=[ 37], 40.00th=[ 37], 50.00th=[ 40], 60.00th=[ 50], | 70.00th=[ 50], 80.00th=[ 55], 90.00th=[ 109], 95.00th=[ 109], | 99.00th=[ 109], 99.50th=[ 109], 99.90th=[ 109], 99.95th=[ 109], | 99.99th=[ 109] lat (usec) : 500=16.67% lat (msec) : 10=8.33%, 20=8.33%, 50=41.67%, 100=16.67%, 250=8.33% cpu : usr=0.00%, sys=35.00%, ctx=260, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=34.3MiB/s (35.0MB/s), 3513KiB/s-16.4MiB/s (3597kB/s-17.2MB/s), io=20.0MiB (20.0MB), run=365-583msec WRITE: bw=48.0MiB/s (50.4MB/s), 11.2MiB/s-17.2MiB/s (11.8MB/s-17.0MB/s), io=28.0MiB (29.4MB), run=365-583msec rand-rw: (groupid=0, jobs=1): err= 0: pid=15540: Wed May 21 00:53:23 2025 read: IOPS=10, BW=10.5MiB/s (11.0MB/s)(7168KiB/667msec) clat (usec): min=23981, max=71807, avg=43569.92, stdev=18979.52 lat (usec): min=23982, max=71808, avg=43570.72, stdev=18979.51 clat percentiles (usec): | 1.00th=[23987], 5.00th=[23987], 10.00th=[23987], 20.00th=[27132], | 30.00th=[29492], 40.00th=[29492], 50.00th=[35914], 60.00th=[56361], | 70.00th=[56361], 80.00th=[60556], 90.00th=[71828], 95.00th=[71828], | 99.00th=[71828], 99.50th=[71828], 99.90th=[71828], 99.95th=[71828], | 99.99th=[71828] bw ( KiB/s): min=10240, max=10240, per=36.90%, avg=10240.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 write: IOPS=7, BW=7676KiB/s (7860kB/s)(5120KiB/667msec) clat (usec): min=49027, max=65108, avg=56933.50, stdev=7326.88 lat (usec): min=49056, max=65145, avg=56979.69, stdev=7336.55 clat percentiles (usec): | 1.00th=[49021], 5.00th=[49021], 10.00th=[49021], 20.00th=[49021], | 30.00th=[49546], 40.00th=[49546], 50.00th=[58983], 60.00th=[58983], | 70.00th=[62129], 80.00th=[62129], 90.00th=[65274], 95.00th=[65274], | 99.00th=[65274], 99.50th=[65274], 99.90th=[65274], 99.95th=[65274], | 99.99th=[65274] bw ( KiB/s): min= 6144, max= 6144, per=15.81%, avg=6144.00, stdev= 0.00, samples=1 iops : min= 6, max= 6, avg= 6.00, stdev= 0.00, samples=1 lat (msec) : 50=50.00%, 100=50.00% cpu : usr=0.15%, sys=14.56%, ctx=50, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=15542: Wed May 21 00:53:23 2025 read: IOPS=2, BW=2775KiB/s (2842kB/s)(2048KiB/738msec) clat (usec): min=15678, max=21648, avg=18663.31, stdev=4221.41 lat (usec): min=15679, max=21649, avg=18664.14, stdev=4221.58 clat percentiles (usec): | 1.00th=[15664], 5.00th=[15664], 10.00th=[15664], 20.00th=[15664], | 30.00th=[15664], 40.00th=[15664], 50.00th=[15664], 60.00th=[21627], | 70.00th=[21627], 80.00th=[21627], 90.00th=[21627], 95.00th=[21627], | 99.00th=[21627], 99.50th=[21627], 99.90th=[21627], 99.95th=[21627], | 99.99th=[21627] bw ( KiB/s): min= 4087, max= 4087, per=14.73%, avg=4087.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 write: IOPS=13, BW=13.5MiB/s (14.2MB/s)(10.0MiB/738msec) clat (msec): min=27, max=106, avg=62.17, stdev=28.92 lat (msec): min=27, max=106, avg=62.22, stdev=28.91 clat percentiles (msec): | 1.00th=[ 28], 5.00th=[ 28], 10.00th=[ 28], 20.00th=[ 34], | 30.00th=[ 36], 40.00th=[ 40], 50.00th=[ 56], 60.00th=[ 57], | 70.00th=[ 88], 80.00th=[ 88], 90.00th=[ 96], 95.00th=[ 107], | 99.00th=[ 107], 99.50th=[ 107], 99.90th=[ 107], 99.95th=[ 107], | 99.99th=[ 107] bw ( KiB/s): min=10219, max=10219, per=26.30%, avg=10219.00, stdev= 0.00, samples=1 iops : min= 9, max= 9, avg= 9.00, stdev= 0.00, samples=1 lat (msec) : 20=8.33%, 50=41.67%, 100=41.67%, 250=8.33% cpu : usr=0.27%, sys=12.35%, ctx=65, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=15543: Wed May 21 00:53:23 2025 read: IOPS=9, BW=9323KiB/s (9547kB/s)(6144KiB/659msec) clat (usec): min=15234, max=74991, avg=42352.62, stdev=24032.47 lat (usec): min=15235, max=74992, avg=42353.34, stdev=24032.42 clat percentiles (usec): | 1.00th=[15270], 5.00th=[15270], 10.00th=[15270], 20.00th=[27919], | 30.00th=[27919], 40.00th=[29492], 50.00th=[29492], 60.00th=[38011], | 70.00th=[68682], 80.00th=[68682], 90.00th=[74974], 95.00th=[74974], | 99.00th=[74974], 99.50th=[74974], 99.90th=[74974], 99.95th=[74974], | 99.99th=[74974] bw ( KiB/s): min= 8192, max= 8192, per=29.52%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=9, BW=9323KiB/s (9547kB/s)(6144KiB/659msec) clat (msec): min=39, max=106, avg=65.81, stdev=23.88 lat (msec): min=39, max=107, avg=65.86, stdev=23.88 clat percentiles (msec): | 1.00th=[ 40], 5.00th=[ 40], 10.00th=[ 40], 20.00th=[ 52], | 30.00th=[ 52], 40.00th=[ 56], 50.00th=[ 56], 60.00th=[ 63], | 70.00th=[ 79], 80.00th=[ 79], 90.00th=[ 108], 95.00th=[ 108], | 99.00th=[ 108], 99.50th=[ 108], 99.90th=[ 108], 99.95th=[ 108], | 99.99th=[ 108] bw ( KiB/s): min= 8192, max= 8192, per=21.09%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 lat (msec) : 20=8.33%, 50=33.33%, 100=50.00%, 250=8.33% cpu : usr=0.00%, sys=11.55%, ctx=48, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=15544: Wed May 21 00:53:23 2025 read: IOPS=7, BW=7507KiB/s (7688kB/s)(5120KiB/682msec) clat (usec): min=12946, max=39368, avg=24768.73, stdev=13497.67 lat (usec): min=12947, max=39369, avg=24769.45, stdev=13497.67 clat percentiles (usec): | 1.00th=[12911], 5.00th=[12911], 10.00th=[12911], 20.00th=[12911], | 30.00th=[13566], 40.00th=[13566], 50.00th=[18744], 60.00th=[18744], | 70.00th=[39584], 80.00th=[39584], 90.00th=[39584], 95.00th=[39584], | 99.00th=[39584], 99.50th=[39584], 99.90th=[39584], 99.95th=[39584], | 99.99th=[39584] bw ( KiB/s): min= 2048, max= 2048, per=7.38%, avg=2048.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=10, BW=10.3MiB/s (10.8MB/s)(7168KiB/682msec) clat (msec): min=30, max=105, avg=67.70, stdev=30.86 lat (msec): min=30, max=105, avg=67.76, stdev=30.88 clat percentiles (msec): | 1.00th=[ 31], 5.00th=[ 31], 10.00th=[ 31], 20.00th=[ 33], | 30.00th=[ 53], 40.00th=[ 53], 50.00th=[ 65], 60.00th=[ 88], | 70.00th=[ 88], 80.00th=[ 101], 90.00th=[ 106], 95.00th=[ 106], | 99.00th=[ 106], 99.50th=[ 106], 99.90th=[ 106], 99.95th=[ 106], | 99.99th=[ 106] bw ( KiB/s): min= 8192, max= 8192, per=21.09%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 lat (msec) : 20=25.00%, 50=33.33%, 100=25.00%, 250=16.67% cpu : usr=0.00%, sys=10.13%, ctx=41, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=27.1MiB/s (28.4MB/s), 2775KiB/s-10.5MiB/s (2842kB/s-11.0MB/s), io=20.0MiB (20.0MB), run=659-738msec WRITE: bw=37.9MiB/s (39.8MB/s), 7676KiB/s-13.5MiB/s (7860kB/s-14.2MB/s), io=28.0MiB (29.4MB), run=659-738msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=15551: Wed May 21 00:53:25 2025 read: IOPS=2, BW=9075KiB/s (9293kB/s)(12.0MiB/1354msec) clat (msec): min=138, max=1026, avg=448.66, stdev=500.63 lat (msec): min=138, max=1026, avg=448.66, stdev=500.63 clat percentiles (msec): | 1.00th=[ 138], 5.00th=[ 138], 10.00th=[ 138], 20.00th=[ 138], | 30.00th=[ 138], 40.00th=[ 182], 50.00th=[ 182], 60.00th=[ 182], | 70.00th=[ 1028], 80.00th=[ 1028], 90.00th=[ 1028], 95.00th=[ 1028], | 99.00th=[ 1028], 99.50th=[ 1028], 99.90th=[ 1028], 99.95th=[ 1028], | 99.99th=[ 1028] lat (msec) : 250=66.67% cpu : usr=0.07%, sys=24.09%, ctx=1533, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=15552: Wed May 21 00:53:25 2025 read: IOPS=0, BW=2538KiB/s (2599kB/s)(4096KiB/1614msec) clat (nsec): min=951343k, max=951343k, avg=951343113.00, stdev= 0.00 lat (nsec): min=951344k, max=951344k, avg=951344271.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 953], 5.00th=[ 953], 10.00th=[ 953], 20.00th=[ 953], | 30.00th=[ 953], 40.00th=[ 953], 50.00th=[ 953], 60.00th=[ 953], | 70.00th=[ 953], 80.00th=[ 953], 90.00th=[ 953], 95.00th=[ 953], | 99.00th=[ 953], 99.50th=[ 953], 99.90th=[ 953], 99.95th=[ 953], | 99.99th=[ 953] bw ( KiB/s): min= 8192, max= 8192, per=80.70%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=1, BW=5076KiB/s (5197kB/s)(8192KiB/1614msec) clat (msec): min=292, max=339, avg=316.18, stdev=33.14 lat (msec): min=293, max=339, avg=316.46, stdev=32.99 clat percentiles (msec): | 1.00th=[ 292], 5.00th=[ 292], 10.00th=[ 292], 20.00th=[ 292], | 30.00th=[ 292], 40.00th=[ 292], 50.00th=[ 292], 60.00th=[ 338], | 70.00th=[ 338], 80.00th=[ 338], 90.00th=[ 338], 95.00th=[ 338], | 99.00th=[ 338], 99.50th=[ 338], 99.90th=[ 338], 99.95th=[ 338], | 99.99th=[ 338] bw ( KiB/s): min= 7831, max= 7831, per=38.57%, avg=7831.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 500=66.67%, 1000=33.33% cpu : usr=0.19%, sys=12.15%, ctx=398, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=15553: Wed May 21 00:53:25 2025 write: IOPS=1, BW=7959KiB/s (8150kB/s)(12.0MiB/1544msec) clat (msec): min=128, max=1098, avg=508.65, stdev=518.17 lat (msec): min=128, max=1099, avg=508.89, stdev=518.20 clat percentiles (msec): | 1.00th=[ 129], 5.00th=[ 129], 10.00th=[ 129], 20.00th=[ 129], | 30.00th=[ 129], 40.00th=[ 300], 50.00th=[ 300], 60.00th=[ 300], | 70.00th=[ 1099], 80.00th=[ 1099], 90.00th=[ 1099], 95.00th=[ 1099], | 99.00th=[ 1099], 99.50th=[ 1099], 99.90th=[ 1099], 99.95th=[ 1099], | 99.99th=[ 1099] lat (msec) : 250=33.33%, 500=33.33% cpu : usr=0.00%, sys=14.13%, ctx=81, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=15554: Wed May 21 00:53:25 2025 write: IOPS=2, BW=8557KiB/s (8762kB/s)(12.0MiB/1436msec) clat (msec): min=102, max=1089, avg=465.23, stdev=542.80 lat (msec): min=102, max=1089, avg=467.40, stdev=541.36 clat percentiles (msec): | 1.00th=[ 103], 5.00th=[ 103], 10.00th=[ 103], 20.00th=[ 103], | 30.00th=[ 103], 40.00th=[ 205], 50.00th=[ 205], 60.00th=[ 205], | 70.00th=[ 1083], 80.00th=[ 1083], 90.00th=[ 1083], 95.00th=[ 1083], | 99.00th=[ 1083], 99.50th=[ 1083], 99.90th=[ 1083], 99.95th=[ 1083], | 99.99th=[ 1083] lat (msec) : 250=66.67% cpu : usr=0.00%, sys=15.89%, ctx=86, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=9.91MiB/s (10.4MB/s), 2538KiB/s-9075KiB/s (2599kB/s-9293kB/s), io=16.0MiB (16.8MB), run=1354-1614msec WRITE: bw=19.8MiB/s (20.8MB/s), 5076KiB/s-8557KiB/s (5197kB/s-8762kB/s), io=32.0MiB (33.6MB), run=1436-1614msec rand-rw: (groupid=0, jobs=1): err= 0: pid=15555: Wed May 21 00:53:26 2025 read: IOPS=1, BW=7897KiB/s (8087kB/s)(12.0MiB/1556msec) clat (msec): min=171, max=1174, avg=510.32, stdev=575.29 lat (msec): min=171, max=1174, avg=510.32, stdev=575.29 clat percentiles (msec): | 1.00th=[ 171], 5.00th=[ 171], 10.00th=[ 171], 20.00th=[ 171], | 30.00th=[ 171], 40.00th=[ 186], 50.00th=[ 186], 60.00th=[ 186], | 70.00th=[ 1183], 80.00th=[ 1183], 90.00th=[ 1183], 95.00th=[ 1183], | 99.00th=[ 1183], 99.50th=[ 1183], 99.90th=[ 1183], 99.95th=[ 1183], | 99.99th=[ 1183] bw ( KiB/s): min=16384, max=16384, per=100.00%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67% cpu : usr=0.00%, sys=3.60%, ctx=41, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=15556: Wed May 21 00:53:26 2025 read: IOPS=0, BW=2310KiB/s (2366kB/s)(4096KiB/1773msec) clat (nsec): min=1166.5M, max=1166.5M, avg=1166538651.00, stdev= 0.00 lat (nsec): min=1166.5M, max=1166.5M, avg=1166540897.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 1167], 5.00th=[ 1167], 10.00th=[ 1167], 20.00th=[ 1167], | 30.00th=[ 1167], 40.00th=[ 1167], 50.00th=[ 1167], 60.00th=[ 1167], | 70.00th=[ 1167], 80.00th=[ 1167], 90.00th=[ 1167], 95.00th=[ 1167], | 99.00th=[ 1167], 99.50th=[ 1167], 99.90th=[ 1167], 99.95th=[ 1167], | 99.99th=[ 1167] bw ( KiB/s): min= 8192, max= 8192, per=88.66%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=1, BW=4620KiB/s (4731kB/s)(8192KiB/1773msec) clat (msec): min=131, max=457, avg=294.99, stdev=230.51 lat (msec): min=132, max=458, avg=295.25, stdev=230.46 clat percentiles (msec): | 1.00th=[ 132], 5.00th=[ 132], 10.00th=[ 132], 20.00th=[ 132], | 30.00th=[ 132], 40.00th=[ 132], 50.00th=[ 132], 60.00th=[ 460], | 70.00th=[ 460], 80.00th=[ 460], 90.00th=[ 460], 95.00th=[ 460], | 99.00th=[ 460], 99.50th=[ 460], 99.90th=[ 460], 99.95th=[ 460], | 99.99th=[ 460] lat (msec) : 250=33.33%, 500=33.33% cpu : usr=0.06%, sys=4.63%, ctx=93, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=15557: Wed May 21 00:53:26 2025 write: IOPS=1, BW=7293KiB/s (7468kB/s)(12.0MiB/1685msec) clat (msec): min=170, max=1194, avg=542.20, stdev=566.44 lat (msec): min=170, max=1194, avg=542.88, stdev=566.02 clat percentiles (msec): | 1.00th=[ 171], 5.00th=[ 171], 10.00th=[ 171], 20.00th=[ 171], | 30.00th=[ 171], 40.00th=[ 264], 50.00th=[ 264], 60.00th=[ 264], | 70.00th=[ 1200], 80.00th=[ 1200], 90.00th=[ 1200], 95.00th=[ 1200], | 99.00th=[ 1200], 99.50th=[ 1200], 99.90th=[ 1200], 99.95th=[ 1200], | 99.99th=[ 1200] bw ( KiB/s): min= 8192, max= 8192, per=44.33%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=33.33% cpu : usr=0.18%, sys=3.98%, ctx=26, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=15558: Wed May 21 00:53:26 2025 write: IOPS=1, BW=7822KiB/s (8009kB/s)(12.0MiB/1571msec) clat (msec): min=129, max=1189, avg=513.57, stdev=587.32 lat (msec): min=129, max=1189, avg=513.85, stdev=587.32 clat percentiles (msec): | 1.00th=[ 130], 5.00th=[ 130], 10.00th=[ 130], 20.00th=[ 130], | 30.00th=[ 130], 40.00th=[ 222], 50.00th=[ 222], 60.00th=[ 222], | 70.00th=[ 1183], 80.00th=[ 1183], 90.00th=[ 1183], 95.00th=[ 1183], | 99.00th=[ 1183], 99.50th=[ 1183], 99.90th=[ 1183], 99.95th=[ 1183], | 99.99th=[ 1183] bw ( KiB/s): min=16351, max=16351, per=88.47%, avg=16351.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67% cpu : usr=0.13%, sys=4.52%, ctx=182, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=9241KiB/s (9463kB/s), 2310KiB/s-7897KiB/s (2366kB/s-8087kB/s), io=16.0MiB (16.8MB), run=1556-1773msec WRITE: bw=18.0MiB/s (18.9MB/s), 4620KiB/s-7822KiB/s (4731kB/s-8009kB/s), io=32.0MiB (33.6MB), run=1571-1773msec