== sanity test 398b: DIO and buffer IO race ============== 23:27:47 (1708835267) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 0.84788 s, 59.4 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=23144: Sat Feb 24 23:28:00 2024 read: IOPS=128, BW=512KiB/s (524kB/s)(5992KiB/11703msec) clat (usec): min=37, max=127415, avg=3306.43, stdev=4495.93 lat (usec): min=38, max=127416, avg=3307.25, stdev=4495.87 clat percentiles (usec): | 1.00th=[ 46], 5.00th=[ 62], 10.00th=[ 66], 20.00th=[ 92], | 30.00th=[ 1860], 40.00th=[ 2147], 50.00th=[ 2507], 60.00th=[ 2933], | 70.00th=[ 3785], 80.00th=[ 5342], 90.00th=[ 7308], 95.00th=[ 8979], | 99.00th=[ 13173], 99.50th=[ 15664], 99.90th=[ 49021], 99.95th=[127402], | 99.99th=[127402] bw ( KiB/s): min= 320, max= 744, per=24.66%, avg=505.74, stdev=94.48, samples=23 iops : min= 80, max= 186, avg=126.30, stdev=23.66, samples=23 write: IOPS=134, BW=538KiB/s (551kB/s)(6296KiB/11703msec) clat (usec): min=384, max=46265, avg=4248.36, stdev=4177.73 lat (usec): min=384, max=46266, avg=4249.15, stdev=4177.71 clat percentiles (usec): | 1.00th=[ 429], 5.00th=[ 461], 10.00th=[ 478], 20.00th=[ 537], | 30.00th=[ 947], 40.00th=[ 2311], 50.00th=[ 3425], 60.00th=[ 4359], | 70.00th=[ 5342], 80.00th=[ 6915], 90.00th=[ 9634], 95.00th=[12387], | 99.00th=[17695], 99.50th=[20317], 99.90th=[28967], 99.95th=[46400], | 99.99th=[46400] bw ( KiB/s): min= 376, max= 712, per=24.73%, avg=531.13, stdev=93.13, samples=23 iops : min= 94, max= 178, avg=132.65, stdev=23.35, samples=23 lat (usec) : 50=1.04%, 100=8.89%, 250=1.11%, 500=7.78%, 750=6.35% lat (usec) : 1000=1.76% lat (msec) : 2=9.70%, 4=27.41%, 10=29.92%, 20=5.66%, 50=0.36% lat (msec) : 250=0.03% cpu : usr=0.13%, sys=21.88%, ctx=5492, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23146: Sat Feb 24 23:28:00 2024 read: IOPS=127, BW=508KiB/s (520kB/s)(5940KiB/11691msec) clat (usec): min=38, max=50922, avg=3108.69, stdev=3096.37 lat (usec): min=38, max=50923, avg=3110.00, stdev=3096.54 clat percentiles (usec): | 1.00th=[ 46], 5.00th=[ 60], 10.00th=[ 65], 20.00th=[ 93], | 30.00th=[ 1876], 40.00th=[ 2212], 50.00th=[ 2540], 60.00th=[ 3032], | 70.00th=[ 3589], 80.00th=[ 4817], 90.00th=[ 6652], 95.00th=[ 8094], | 99.00th=[11863], 99.50th=[13698], 99.90th=[36439], 99.95th=[51119], | 99.99th=[51119] bw ( KiB/s): min= 336, max= 720, per=24.69%, avg=506.35, stdev=94.11, samples=23 iops : min= 84, max= 180, avg=126.52, stdev=23.55, samples=23 write: IOPS=135, BW=543KiB/s (556kB/s)(6348KiB/11691msec) clat (usec): min=392, max=124855, avg=4415.55, stdev=5568.52 lat (usec): min=393, max=124856, avg=4416.92, stdev=5568.75 clat percentiles (usec): | 1.00th=[ 437], 5.00th=[ 457], 10.00th=[ 478], 20.00th=[ 523], | 30.00th=[ 857], 40.00th=[ 2114], 50.00th=[ 3326], 60.00th=[ 4228], | 70.00th=[ 5407], 80.00th=[ 7111], 90.00th=[ 9765], 95.00th=[ 12387], | 99.00th=[ 19530], 99.50th=[ 27395], 99.90th=[ 47449], 99.95th=[125305], | 99.99th=[125305] bw ( KiB/s): min= 360, max= 808, per=24.98%, avg=536.57, stdev=98.37, samples=23 iops : min= 90, max= 202, avg=134.09, stdev=24.63, samples=23 lat (usec) : 50=1.11%, 100=8.66%, 250=0.75%, 500=8.20%, 750=6.71% lat (usec) : 1000=1.63% lat (msec) : 2=9.47%, 4=29.17%, 10=28.32%, 20=5.44%, 50=0.49% lat (msec) : 100=0.03%, 250=0.03% cpu : usr=0.23%, sys=21.70%, ctx=5386, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23148: Sat Feb 24 23:28:00 2024 read: IOPS=130, BW=523KiB/s (536kB/s)(6028KiB/11519msec) clat (usec): min=42, max=131278, avg=3329.43, stdev=4569.47 lat (usec): min=42, max=131279, avg=3330.08, stdev=4569.47 clat percentiles (usec): | 1.00th=[ 47], 5.00th=[ 62], 10.00th=[ 68], 20.00th=[ 117], | 30.00th=[ 1926], 40.00th=[ 2180], 50.00th=[ 2573], 60.00th=[ 2999], | 70.00th=[ 3752], 80.00th=[ 5080], 90.00th=[ 7177], 95.00th=[ 8455], | 99.00th=[ 11469], 99.50th=[ 17433], 99.90th=[ 43779], 99.95th=[131597], | 99.99th=[131597] bw ( KiB/s): min= 376, max= 688, per=25.51%, avg=523.17, stdev=87.37, samples=23 iops : min= 94, max= 172, avg=130.65, stdev=21.90, samples=23 write: IOPS=135, BW=543KiB/s (556kB/s)(6260KiB/11519msec) clat (usec): min=385, max=34761, avg=4113.00, stdev=3966.35 lat (usec): min=386, max=34761, avg=4113.90, stdev=3966.31 clat percentiles (usec): | 1.00th=[ 437], 5.00th=[ 461], 10.00th=[ 482], 20.00th=[ 537], | 30.00th=[ 857], 40.00th=[ 2089], 50.00th=[ 3228], 60.00th=[ 4178], | 70.00th=[ 5342], 80.00th=[ 6915], 90.00th=[ 9634], 95.00th=[11731], | 99.00th=[17433], 99.50th=[19268], 99.90th=[24773], 99.95th=[34866], | 99.99th=[34866] bw ( KiB/s): min= 392, max= 712, per=25.28%, avg=542.96, stdev=82.30, samples=23 iops : min= 98, max= 178, avg=135.61, stdev=20.63, samples=23 lat (usec) : 50=0.85%, 100=8.33%, 250=1.07%, 500=7.16%, 750=7.58% lat (usec) : 1000=1.46% lat (msec) : 2=9.37%, 4=29.49%, 10=29.23%, 20=5.11%, 50=0.29% lat (msec) : 250=0.03% cpu : usr=0.16%, sys=21.97%, ctx=5437, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23149: Sat Feb 24 23:28:00 2024 read: IOPS=130, BW=522KiB/s (535kB/s)(6052KiB/11588msec) clat (usec): min=37, max=30780, avg=3020.17, stdev=2856.74 lat (usec): min=37, max=30780, avg=3021.27, stdev=2856.71 clat percentiles (usec): | 1.00th=[ 45], 5.00th=[ 57], 10.00th=[ 65], 20.00th=[ 75], | 30.00th=[ 1762], 40.00th=[ 2073], 50.00th=[ 2409], 60.00th=[ 2868], | 70.00th=[ 3556], 80.00th=[ 5080], 90.00th=[ 6980], 95.00th=[ 8455], | 99.00th=[11994], 99.50th=[13435], 99.90th=[19006], 99.95th=[30802], | 99.99th=[30802] bw ( KiB/s): min= 176, max= 704, per=25.30%, avg=518.96, stdev=119.76, samples=23 iops : min= 44, max= 176, avg=129.74, stdev=29.94, samples=23 write: IOPS=134, BW=538KiB/s (551kB/s)(6236KiB/11588msec) clat (usec): min=367, max=180704, avg=4457.82, stdev=6516.58 lat (usec): min=368, max=180706, avg=4458.79, stdev=6516.63 clat percentiles (usec): | 1.00th=[ 433], 5.00th=[ 457], 10.00th=[ 482], 20.00th=[ 529], | 30.00th=[ 857], 40.00th=[ 2180], 50.00th=[ 3359], 60.00th=[ 4228], | 70.00th=[ 5407], 80.00th=[ 6980], 90.00th=[ 9634], 95.00th=[ 12518], | 99.00th=[ 21627], 99.50th=[ 31065], 99.90th=[ 44827], 99.95th=[181404], | 99.99th=[181404] bw ( KiB/s): min= 192, max= 856, per=24.94%, avg=535.65, stdev=132.39, samples=23 iops : min= 48, max= 214, avg=133.91, stdev=33.10, samples=23 lat (usec) : 50=1.69%, 100=9.90%, 250=1.04%, 500=7.55%, 750=7.26% lat (usec) : 1000=1.46% lat (msec) : 2=9.24%, 4=27.05%, 10=29.10%, 20=5.05%, 50=0.62% lat (msec) : 250=0.03% cpu : usr=0.20%, sys=21.37%, ctx=5377, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2052KiB/s (2101kB/s), 508KiB/s-523KiB/s (520kB/s-536kB/s), io=23.4MiB (24.6MB), run=11519-11703msec WRITE: bw=2148KiB/s (2200kB/s), 538KiB/s-543KiB/s (551kB/s-556kB/s), io=24.6MiB (25.7MB), run=11519-11703msec rand-rw: (groupid=0, jobs=1): err= 0: pid=23142: Sat Feb 24 23:28:24 2024 read: IOPS=42, BW=169KiB/s (173kB/s)(5992KiB/35518msec) clat (usec): min=1327, max=131028, avg=3441.10, stdev=4662.85 lat (usec): min=1327, max=131029, avg=3441.79, stdev=4662.86 clat percentiles (usec): | 1.00th=[ 1500], 5.00th=[ 1614], 10.00th=[ 1663], 20.00th=[ 1762], | 30.00th=[ 1860], 40.00th=[ 1975], 50.00th=[ 2245], 60.00th=[ 3032], | 70.00th=[ 3359], 80.00th=[ 3818], 90.00th=[ 5932], 95.00th=[ 9241], | 99.00th=[ 14877], 99.50th=[ 20579], 99.90th=[ 55313], 99.95th=[130548], | 99.99th=[130548] bw ( KiB/s): min= 56, max= 304, per=25.06%, avg=168.65, stdev=67.05, samples=71 iops : min= 14, max= 76, avg=42.06, stdev=16.69, samples=71 write: IOPS=44, BW=177KiB/s (182kB/s)(6296KiB/35518msec) clat (usec): min=8797, max=63583, avg=19249.43, stdev=7903.49 lat (usec): min=8798, max=63584, avg=19250.24, stdev=7903.50 clat percentiles (usec): | 1.00th=[10159], 5.00th=[11076], 10.00th=[11731], 20.00th=[13042], | 30.00th=[14353], 40.00th=[15926], 50.00th=[17695], 60.00th=[18744], | 70.00th=[20317], 80.00th=[22676], 90.00th=[31327], 95.00th=[36963], | 99.00th=[45876], 99.50th=[46924], 99.90th=[59507], 99.95th=[63701], | 99.99th=[63701] bw ( KiB/s): min= 72, max= 256, per=25.10%, avg=176.99, stdev=55.71, samples=71 iops : min= 18, max= 64, avg=44.14, stdev=13.90, samples=71 lat (msec) : 2=20.41%, 4=20.08%, 10=6.74%, 20=35.90%, 50=16.67% lat (msec) : 100=0.16%, 250=0.03% cpu : usr=0.11%, sys=4.29%, ctx=3647, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23143: Sat Feb 24 23:28:24 2024 read: IOPS=41, BW=167KiB/s (171kB/s)(5940KiB/35645msec) clat (usec): min=1321, max=128721, avg=3449.11, stdev=4443.85 lat (usec): min=1322, max=128723, avg=3449.79, stdev=4443.88 clat percentiles (usec): | 1.00th=[ 1500], 5.00th=[ 1582], 10.00th=[ 1663], 20.00th=[ 1778], | 30.00th=[ 1860], 40.00th=[ 2008], 50.00th=[ 2343], 60.00th=[ 3032], | 70.00th=[ 3392], 80.00th=[ 3818], 90.00th=[ 6128], 95.00th=[ 9241], | 99.00th=[ 15401], 99.50th=[ 20317], 99.90th=[ 35390], 99.95th=[128451], | 99.99th=[128451] bw ( KiB/s): min= 32, max= 328, per=24.72%, avg=166.35, stdev=73.30, samples=71 iops : min= 8, max= 82, avg=41.54, stdev=18.38, samples=71 write: IOPS=44, BW=178KiB/s (182kB/s)(6348KiB/35645msec) clat (usec): min=8867, max=63620, avg=19191.10, stdev=7976.45 lat (usec): min=8868, max=63621, avg=19191.92, stdev=7976.47 clat percentiles (usec): | 1.00th=[10028], 5.00th=[11076], 10.00th=[11731], 20.00th=[12911], | 30.00th=[14222], 40.00th=[15795], 50.00th=[17433], 60.00th=[19006], | 70.00th=[20317], 80.00th=[22414], 90.00th=[30802], 95.00th=[36963], | 99.00th=[47449], 99.50th=[48497], 99.90th=[53740], 99.95th=[63701], | 99.99th=[63701] bw ( KiB/s): min= 72, max= 264, per=25.18%, avg=177.51, stdev=56.51, samples=71 iops : min= 18, max= 66, avg=44.32, stdev=14.15, samples=71 lat (msec) : 2=19.34%, 4=20.51%, 10=6.90%, 20=36.17%, 50=16.89% lat (msec) : 100=0.16%, 250=0.03% cpu : usr=0.08%, sys=4.25%, ctx=3647, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23145: Sat Feb 24 23:28:24 2024 read: IOPS=42, BW=171KiB/s (175kB/s)(6028KiB/35288msec) clat (usec): min=1323, max=54483, avg=3489.11, stdev=3248.66 lat (usec): min=1323, max=54484, avg=3489.84, stdev=3248.67 clat percentiles (usec): | 1.00th=[ 1467], 5.00th=[ 1598], 10.00th=[ 1680], 20.00th=[ 1762], | 30.00th=[ 1860], 40.00th=[ 1991], 50.00th=[ 2343], 60.00th=[ 3097], | 70.00th=[ 3425], 80.00th=[ 3949], 90.00th=[ 6783], 95.00th=[ 9503], | 99.00th=[15533], 99.50th=[20317], 99.90th=[35914], 99.95th=[54264], | 99.99th=[54264] bw ( KiB/s): min= 56, max= 399, per=25.36%, avg=170.66, stdev=69.17, samples=70 iops : min= 14, max= 99, avg=42.60, stdev=17.28, samples=70 write: IOPS=44, BW=177KiB/s (182kB/s)(6260KiB/35288msec) clat (msec): min=9, max=152, avg=19.14, stdev= 8.36 lat (msec): min=9, max=152, avg=19.15, stdev= 8.36 clat percentiles (msec): | 1.00th=[ 11], 5.00th=[ 12], 10.00th=[ 12], 20.00th=[ 14], | 30.00th=[ 15], 40.00th=[ 16], 50.00th=[ 18], 60.00th=[ 20], | 70.00th=[ 21], 80.00th=[ 23], 90.00th=[ 30], 95.00th=[ 36], | 99.00th=[ 47], 99.50th=[ 51], 99.90th=[ 75], 99.95th=[ 153], | 99.99th=[ 153] bw ( KiB/s): min= 72, max= 256, per=25.10%, avg=176.94, stdev=54.37, samples=70 iops : min= 18, max= 64, avg=44.17, stdev=13.61, samples=70 lat (msec) : 2=19.73%, 4=19.66%, 10=7.91%, 20=35.51%, 50=16.86% lat (msec) : 100=0.29%, 250=0.03% cpu : usr=0.10%, sys=4.28%, ctx=3601, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23147: Sat Feb 24 23:28:24 2024 read: IOPS=42, BW=171KiB/s (175kB/s)(6052KiB/35470msec) clat (usec): min=1336, max=29300, avg=3350.99, stdev=2776.46 lat (usec): min=1337, max=29301, avg=3351.70, stdev=2776.48 clat percentiles (usec): | 1.00th=[ 1467], 5.00th=[ 1598], 10.00th=[ 1680], 20.00th=[ 1762], | 30.00th=[ 1844], 40.00th=[ 1975], 50.00th=[ 2245], 60.00th=[ 3032], | 70.00th=[ 3359], 80.00th=[ 3851], 90.00th=[ 6390], 95.00th=[ 9372], | 99.00th=[15008], 99.50th=[17171], 99.90th=[24773], 99.95th=[29230], | 99.99th=[29230] bw ( KiB/s): min= 16, max= 328, per=25.25%, avg=169.90, stdev=72.59, samples=70 iops : min= 4, max= 82, avg=42.36, stdev=18.12, samples=70 write: IOPS=43, BW=176KiB/s (180kB/s)(6236KiB/35470msec) clat (msec): min=9, max=178, avg=19.46, stdev= 8.72 lat (msec): min=9, max=178, avg=19.46, stdev= 8.72 clat percentiles (msec): | 1.00th=[ 10], 5.00th=[ 12], 10.00th=[ 12], 20.00th=[ 14], | 30.00th=[ 15], 40.00th=[ 17], 50.00th=[ 18], 60.00th=[ 20], | 70.00th=[ 21], 80.00th=[ 23], 90.00th=[ 31], 95.00th=[ 37], | 99.00th=[ 48], 99.50th=[ 50], 99.90th=[ 55], 99.95th=[ 180], | 99.99th=[ 180] bw ( KiB/s): min= 80, max= 248, per=24.88%, avg=175.39, stdev=53.49, samples=70 iops : min= 20, max= 62, avg=43.73, stdev=13.38, samples=70 lat (msec) : 2=20.35%, 4=19.63%, 10=7.88%, 20=34.99%, 50=16.93% lat (msec) : 100=0.20%, 250=0.03% cpu : usr=0.08%, sys=4.30%, ctx=3645, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=674KiB/s (690kB/s), 167KiB/s-171KiB/s (171kB/s-175kB/s), io=23.4MiB (24.6MB), run=35288-35645msec WRITE: bw=705KiB/s (722kB/s), 176KiB/s-178KiB/s (180kB/s-182kB/s), io=24.6MiB (25.7MB), run=35288-35645msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=23192: Sat Feb 24 23:28:27 2024 read: IOPS=159, BW=2550KiB/s (2611kB/s)(5920KiB/2322msec) clat (usec): min=44, max=16623, avg=2810.07, stdev=2821.29 lat (usec): min=45, max=16623, avg=2810.68, stdev=2821.29 clat percentiles (usec): | 1.00th=[ 49], 5.00th=[ 60], 10.00th=[ 67], 20.00th=[ 70], | 30.00th=[ 78], 40.00th=[ 2114], 50.00th=[ 2638], 60.00th=[ 3032], | 70.00th=[ 3949], 80.00th=[ 5211], 90.00th=[ 6587], 95.00th=[ 7635], | 99.00th=[13042], 99.50th=[15139], 99.90th=[16581], 99.95th=[16581], | 99.99th=[16581] bw ( KiB/s): min= 2368, max= 2656, per=24.57%, avg=2534.50, stdev=130.31, samples=4 iops : min= 148, max= 166, avg=158.25, stdev= 8.02, samples=4 write: IOPS=171, BW=2742KiB/s (2808kB/s)(6368KiB/2322msec) clat (usec): min=590, max=35649, avg=3207.00, stdev=3327.73 lat (usec): min=591, max=35650, avg=3207.92, stdev=3327.82 clat percentiles (usec): | 1.00th=[ 603], 5.00th=[ 627], 10.00th=[ 685], 20.00th=[ 775], | 30.00th=[ 873], 40.00th=[ 1106], 50.00th=[ 1565], 60.00th=[ 3163], | 70.00th=[ 4424], 80.00th=[ 5669], 90.00th=[ 7177], 95.00th=[ 8029], | 99.00th=[12911], 99.50th=[19530], 99.90th=[35390], 99.95th=[35390], | 99.99th=[35390] bw ( KiB/s): min= 2464, max= 3417, per=25.56%, avg=2774.25, stdev=438.73, samples=4 iops : min= 154, max= 213, avg=173.25, stdev=27.15, samples=4 lat (usec) : 50=1.30%, 100=15.10%, 250=1.56%, 500=0.26%, 750=9.64% lat (usec) : 1000=9.37% lat (msec) : 2=8.33%, 4=23.05%, 10=29.43%, 20=1.82%, 50=0.13% cpu : usr=0.26%, sys=28.29%, ctx=1452, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23193: Sat Feb 24 23:28:27 2024 read: IOPS=156, BW=2507KiB/s (2567kB/s)(5744KiB/2291msec) clat (usec): min=47, max=18767, avg=2717.87, stdev=2634.31 lat (usec): min=48, max=18767, avg=2718.84, stdev=2634.19 clat percentiles (usec): | 1.00th=[ 49], 5.00th=[ 59], 10.00th=[ 67], 20.00th=[ 70], | 30.00th=[ 80], 40.00th=[ 2245], 50.00th=[ 2704], 60.00th=[ 3097], | 70.00th=[ 3687], 80.00th=[ 4817], 90.00th=[ 6063], 95.00th=[ 7570], | 99.00th=[ 8848], 99.50th=[16188], 99.90th=[18744], 99.95th=[18744], | 99.99th=[18744] bw ( KiB/s): min= 2011, max= 3360, per=24.79%, avg=2557.50, stdev=617.58, samples=4 iops : min= 125, max= 210, avg=159.50, stdev=38.96, samples=4 write: IOPS=178, BW=2856KiB/s (2925kB/s)(6544KiB/2291msec) clat (usec): min=570, max=19212, avg=3199.99, stdev=3279.29 lat (usec): min=571, max=19213, avg=3201.02, stdev=3279.21 clat percentiles (usec): | 1.00th=[ 594], 5.00th=[ 644], 10.00th=[ 685], 20.00th=[ 750], | 30.00th=[ 832], 40.00th=[ 955], 50.00th=[ 1418], 60.00th=[ 2966], | 70.00th=[ 4146], 80.00th=[ 5604], 90.00th=[ 7898], 95.00th=[10159], | 99.00th=[14615], 99.50th=[15139], 99.90th=[19268], 99.95th=[19268], | 99.99th=[19268] bw ( KiB/s): min= 2618, max= 3104, per=26.73%, avg=2900.75, stdev=227.61, samples=4 iops : min= 163, max= 194, avg=181.00, stdev=14.35, samples=4 lat (usec) : 50=1.04%, 100=14.45%, 250=1.17%, 500=0.26%, 750=10.81% lat (usec) : 1000=12.76% lat (msec) : 2=6.38%, 4=23.44%, 10=26.43%, 20=3.26% cpu : usr=0.39%, sys=28.47%, ctx=1447, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23194: Sat Feb 24 23:28:27 2024 read: IOPS=176, BW=2818KiB/s (2886kB/s)(6256KiB/2220msec) clat (usec): min=46, max=15474, avg=2698.37, stdev=2556.05 lat (usec): min=46, max=15475, avg=2698.98, stdev=2556.06 clat percentiles (usec): | 1.00th=[ 49], 5.00th=[ 61], 10.00th=[ 67], 20.00th=[ 71], | 30.00th=[ 84], 40.00th=[ 2245], 50.00th=[ 2737], 60.00th=[ 3163], | 70.00th=[ 3785], 80.00th=[ 4686], 90.00th=[ 5997], 95.00th=[ 7242], | 99.00th=[ 9372], 99.50th=[11731], 99.90th=[15533], 99.95th=[15533], | 99.99th=[15533] bw ( KiB/s): min= 2336, max= 3648, per=27.81%, avg=2869.00, stdev=559.65, samples=4 iops : min= 146, max= 228, avg=179.00, stdev=35.07, samples=4 write: IOPS=169, BW=2717KiB/s (2782kB/s)(6032KiB/2220msec) clat (usec): min=577, max=15989, avg=3071.42, stdev=3032.22 lat (usec): min=578, max=15990, avg=3072.28, stdev=3032.23 clat percentiles (usec): | 1.00th=[ 603], 5.00th=[ 652], 10.00th=[ 693], 20.00th=[ 775], | 30.00th=[ 914], 40.00th=[ 1074], 50.00th=[ 1614], 60.00th=[ 2835], | 70.00th=[ 3752], 80.00th=[ 5211], 90.00th=[ 7373], 95.00th=[ 9634], | 99.00th=[13435], 99.50th=[14615], 99.90th=[15926], 99.95th=[15926], | 99.99th=[15926] bw ( KiB/s): min= 2395, max= 3232, per=25.04%, avg=2717.50, stdev=382.95, samples=4 iops : min= 149, max= 202, avg=169.50, stdev=24.28, samples=4 lat (usec) : 50=0.78%, 100=15.76%, 250=2.34%, 500=0.39%, 750=8.59% lat (usec) : 1000=9.77% lat (msec) : 2=8.46%, 4=26.04%, 10=25.39%, 20=2.47% cpu : usr=0.18%, sys=29.25%, ctx=1437, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23195: Sat Feb 24 23:28:27 2024 read: IOPS=166, BW=2661KiB/s (2725kB/s)(6032KiB/2267msec) clat (usec): min=43, max=13308, avg=2819.54, stdev=2652.53 lat (usec): min=43, max=13309, avg=2820.19, stdev=2652.50 clat percentiles (usec): | 1.00th=[ 47], 5.00th=[ 53], 10.00th=[ 65], 20.00th=[ 71], | 30.00th=[ 86], 40.00th=[ 2343], 50.00th=[ 2737], 60.00th=[ 3228], | 70.00th=[ 3884], 80.00th=[ 5014], 90.00th=[ 6259], 95.00th=[ 7570], | 99.00th=[10421], 99.50th=[12649], 99.90th=[13304], 99.95th=[13304], | 99.99th=[13304] bw ( KiB/s): min= 2272, max= 3033, per=25.73%, avg=2654.25, stdev=315.05, samples=4 iops : min= 142, max= 189, avg=165.75, stdev=19.47, samples=4 write: IOPS=172, BW=2760KiB/s (2826kB/s)(6256KiB/2267msec) clat (usec): min=552, max=14272, avg=3059.43, stdev=2959.70 lat (usec): min=552, max=14273, avg=3061.07, stdev=2959.21 clat percentiles (usec): | 1.00th=[ 611], 5.00th=[ 660], 10.00th=[ 693], 20.00th=[ 775], | 30.00th=[ 865], 40.00th=[ 988], 50.00th=[ 1319], 60.00th=[ 2704], | 70.00th=[ 4113], 80.00th=[ 5604], 90.00th=[ 7898], 95.00th=[ 8848], | 99.00th=[12518], 99.50th=[13435], 99.90th=[14222], 99.95th=[14222], | 99.99th=[14222] bw ( KiB/s): min= 2400, max= 3640, per=25.86%, avg=2806.00, stdev=565.74, samples=4 iops : min= 150, max= 227, avg=175.25, stdev=35.11, samples=4 lat (usec) : 50=1.04%, 100=14.97%, 250=1.04%, 500=0.78%, 750=9.24% lat (usec) : 1000=12.37% lat (msec) : 2=7.42%, 4=23.31%, 10=27.73%, 20=2.08% cpu : usr=0.18%, sys=29.13%, ctx=1509, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=10.1MiB/s (10.6MB/s), 2507KiB/s-2818KiB/s (2567kB/s-2886kB/s), io=23.4MiB (24.5MB), run=2220-2322msec WRITE: bw=10.6MiB/s (11.1MB/s), 2717KiB/s-2856KiB/s (2782kB/s-2925kB/s), io=24.6MiB (25.8MB), run=2220-2322msec rand-rw: (groupid=0, jobs=1): err= 0: pid=23197: Sat Feb 24 23:28:35 2024 read: IOPS=38, BW=623KiB/s (638kB/s)(5920KiB/9500msec) clat (usec): min=1825, max=22959, avg=3260.53, stdev=2522.64 lat (usec): min=1826, max=22960, avg=3261.20, stdev=2522.66 clat percentiles (usec): | 1.00th=[ 1876], 5.00th=[ 2008], 10.00th=[ 2057], 20.00th=[ 2147], | 30.00th=[ 2212], 40.00th=[ 2278], 50.00th=[ 2376], 60.00th=[ 2507], | 70.00th=[ 2737], 80.00th=[ 3163], 90.00th=[ 5473], 95.00th=[ 8029], | 99.00th=[14615], 99.50th=[20841], 99.90th=[22938], 99.95th=[22938], | 99.99th=[22938] bw ( KiB/s): min= 288, max= 960, per=25.13%, avg=620.44, stdev=188.56, samples=18 iops : min= 18, max= 60, avg=38.78, stdev=11.79, samples=18 write: IOPS=41, BW=670KiB/s (686kB/s)(6368KiB/9500msec) clat (usec): min=12081, max=48854, avg=20762.61, stdev=6626.53 lat (usec): min=12082, max=48856, avg=20763.92, stdev=6626.57 clat percentiles (usec): | 1.00th=[12256], 5.00th=[12911], 10.00th=[13566], 20.00th=[14746], | 30.00th=[16319], 40.00th=[17957], 50.00th=[20055], 60.00th=[21627], | 70.00th=[23200], 80.00th=[24773], 90.00th=[28967], 95.00th=[34866], | 99.00th=[44303], 99.50th=[45351], 99.90th=[49021], 99.95th=[49021], | 99.99th=[49021] bw ( KiB/s): min= 384, max= 800, per=25.60%, avg=664.89, stdev=144.18, samples=18 iops : min= 24, max= 50, avg=41.56, stdev= 9.01, samples=18 lat (msec) : 2=2.21%, 4=38.28%, 10=6.25%, 20=26.43%, 50=26.82% cpu : usr=0.11%, sys=4.88%, ctx=831, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23198: Sat Feb 24 23:28:35 2024 read: IOPS=37, BW=592KiB/s (606kB/s)(5744KiB/9701msec) clat (usec): min=1710, max=21322, avg=3544.01, stdev=3141.54 lat (usec): min=1710, max=21323, avg=3544.70, stdev=3141.54 clat percentiles (usec): | 1.00th=[ 1876], 5.00th=[ 1975], 10.00th=[ 2057], 20.00th=[ 2147], | 30.00th=[ 2245], 40.00th=[ 2311], 50.00th=[ 2376], 60.00th=[ 2540], | 70.00th=[ 2671], 80.00th=[ 3195], 90.00th=[ 6980], 95.00th=[11731], | 99.00th=[18220], 99.50th=[20841], 99.90th=[21365], 99.95th=[21365], | 99.99th=[21365] bw ( KiB/s): min= 288, max= 960, per=23.87%, avg=589.37, stdev=183.29, samples=19 iops : min= 18, max= 60, avg=36.79, stdev=11.45, samples=19 write: IOPS=42, BW=675KiB/s (691kB/s)(6544KiB/9701msec) clat (usec): min=11407, max=51815, avg=20534.69, stdev=6271.55 lat (usec): min=11408, max=51816, avg=20535.95, stdev=6271.60 clat percentiles (usec): | 1.00th=[11994], 5.00th=[12780], 10.00th=[13304], 20.00th=[14615], | 30.00th=[16581], 40.00th=[18744], 50.00th=[20055], 60.00th=[21365], | 70.00th=[22676], 80.00th=[25035], 90.00th=[27657], 95.00th=[32113], | 99.00th=[41681], 99.50th=[44827], 99.90th=[51643], 99.95th=[51643], | 99.99th=[51643] bw ( KiB/s): min= 384, max= 832, per=25.61%, avg=665.16, stdev=140.00, samples=19 iops : min= 24, max= 52, avg=41.53, stdev= 8.69, samples=19 lat (msec) : 2=2.60%, 4=36.46%, 10=4.82%, 20=29.04%, 50=26.82% lat (msec) : 100=0.26% cpu : usr=0.11%, sys=4.67%, ctx=834, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23199: Sat Feb 24 23:28:35 2024 read: IOPS=42, BW=678KiB/s (694kB/s)(6256KiB/9226msec) clat (usec): min=1862, max=24795, avg=3469.53, stdev=3094.35 lat (usec): min=1863, max=24796, avg=3470.21, stdev=3094.37 clat percentiles (usec): | 1.00th=[ 1909], 5.00th=[ 1975], 10.00th=[ 2057], 20.00th=[ 2147], | 30.00th=[ 2212], 40.00th=[ 2278], 50.00th=[ 2376], 60.00th=[ 2474], | 70.00th=[ 2704], 80.00th=[ 3032], 90.00th=[ 6718], 95.00th=[10421], | 99.00th=[17695], 99.50th=[20317], 99.90th=[24773], 99.95th=[24773], | 99.99th=[24773] bw ( KiB/s): min= 319, max= 1245, per=27.56%, avg=680.44, stdev=290.05, samples=18 iops : min= 19, max= 77, avg=42.33, stdev=18.08, samples=18 write: IOPS=40, BW=654KiB/s (669kB/s)(6032KiB/9226msec) clat (usec): min=12130, max=63724, avg=20796.49, stdev=6507.58 lat (usec): min=12131, max=63725, avg=20797.72, stdev=6507.62 clat percentiles (usec): | 1.00th=[12387], 5.00th=[13173], 10.00th=[13829], 20.00th=[15008], | 30.00th=[16712], 40.00th=[18482], 50.00th=[20055], 60.00th=[21627], | 70.00th=[22938], 80.00th=[24773], 90.00th=[27919], 95.00th=[32900], | 99.00th=[41681], 99.50th=[50070], 99.90th=[63701], 99.95th=[63701], | 99.99th=[63701] bw ( KiB/s): min= 384, max= 800, per=24.97%, avg=648.56, stdev=126.53, samples=18 iops : min= 24, max= 50, avg=40.33, stdev= 8.02, samples=18 lat (msec) : 2=3.26%, 4=39.71%, 10=5.21%, 20=26.56%, 50=25.00% lat (msec) : 100=0.26% cpu : usr=0.08%, sys=4.99%, ctx=834, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23200: Sat Feb 24 23:28:35 2024 read: IOPS=39, BW=636KiB/s (651kB/s)(6032KiB/9485msec) clat (usec): min=1852, max=24599, avg=3399.93, stdev=2940.10 lat (usec): min=1853, max=24600, avg=3400.60, stdev=2940.10 clat percentiles (usec): | 1.00th=[ 1942], 5.00th=[ 2040], 10.00th=[ 2089], 20.00th=[ 2180], | 30.00th=[ 2245], 40.00th=[ 2311], 50.00th=[ 2409], 60.00th=[ 2540], | 70.00th=[ 2769], 80.00th=[ 3261], 90.00th=[ 5800], 95.00th=[ 9110], | 99.00th=[19792], 99.50th=[22938], 99.90th=[24511], 99.95th=[24511], | 99.99th=[24511] bw ( KiB/s): min= 288, max= 928, per=24.98%, avg=616.78, stdev=183.87, samples=18 iops : min= 18, max= 58, avg=38.50, stdev=11.47, samples=18 write: IOPS=41, BW=660KiB/s (675kB/s)(6256KiB/9485msec) clat (usec): min=11828, max=45758, avg=20904.73, stdev=5930.27 lat (usec): min=11829, max=45759, avg=20906.19, stdev=5930.31 clat percentiles (usec): | 1.00th=[12518], 5.00th=[13042], 10.00th=[13960], 20.00th=[15401], | 30.00th=[16909], 40.00th=[18744], 50.00th=[20317], 60.00th=[21890], | 70.00th=[23200], 80.00th=[25035], 90.00th=[28443], 95.00th=[32113], | 99.00th=[39060], 99.50th=[44827], 99.90th=[45876], 99.95th=[45876], | 99.99th=[45876] bw ( KiB/s): min= 416, max= 768, per=25.05%, avg=650.56, stdev=126.08, samples=18 iops : min= 26, max= 48, avg=40.61, stdev= 7.88, samples=18 lat (msec) : 2=1.17%, 4=40.49%, 10=5.60%, 20=25.78%, 50=26.95% cpu : usr=0.11%, sys=4.82%, ctx=838, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2469KiB/s (2528kB/s), 592KiB/s-678KiB/s (606kB/s-694kB/s), io=23.4MiB (24.5MB), run=9226-9701msec WRITE: bw=2598KiB/s (2660kB/s), 654KiB/s-675KiB/s (669kB/s-691kB/s), io=24.6MiB (25.8MB), run=9226-9701msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=23213: Sat Feb 24 23:28:36 2024 read: IOPS=16, BW=16.7MiB/s (17.5MB/s)(7168KiB/419msec) clat (usec): min=367, max=52578, avg=37877.21, stdev=17135.10 lat (usec): min=367, max=52579, avg=37878.05, stdev=17135.14 clat percentiles (usec): | 1.00th=[ 367], 5.00th=[ 367], 10.00th=[ 367], 20.00th=[38011], | 30.00th=[40633], 40.00th=[40633], 50.00th=[43254], 60.00th=[44827], | 70.00th=[44827], 80.00th=[45351], 90.00th=[52691], 95.00th=[52691], | 99.00th=[52691], 99.50th=[52691], 99.90th=[52691], 99.95th=[52691], | 99.99th=[52691] write: IOPS=11, BW=11.9MiB/s (12.5MB/s)(5120KiB/419msec) clat (usec): min=18607, max=48571, avg=30497.21, stdev=12865.96 lat (usec): min=18657, max=48592, avg=30535.65, stdev=12852.42 clat percentiles (usec): | 1.00th=[18482], 5.00th=[18482], 10.00th=[18482], 20.00th=[18482], | 30.00th=[21890], 40.00th=[21890], 50.00th=[23987], 60.00th=[23987], | 70.00th=[39584], 80.00th=[39584], 90.00th=[48497], 95.00th=[48497], | 99.00th=[48497], 99.50th=[48497], 99.90th=[48497], 99.95th=[48497], | 99.99th=[48497] lat (usec) : 500=8.33% lat (msec) : 20=8.33%, 50=75.00%, 100=8.33% cpu : usr=0.00%, sys=49.04%, ctx=115, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23214: Sat Feb 24 23:28:36 2024 read: IOPS=3, BW=3724KiB/s (3813kB/s)(2048KiB/550msec) clat (usec): min=27580, max=50248, avg=38914.63, stdev=16028.33 lat (usec): min=27582, max=50249, avg=38915.78, stdev=16027.88 clat percentiles (usec): | 1.00th=[27657], 5.00th=[27657], 10.00th=[27657], 20.00th=[27657], | 30.00th=[27657], 40.00th=[27657], 50.00th=[27657], 60.00th=[50070], | 70.00th=[50070], 80.00th=[50070], 90.00th=[50070], 95.00th=[50070], | 99.00th=[50070], 99.50th=[50070], 99.90th=[50070], 99.95th=[50070], | 99.99th=[50070] bw ( KiB/s): min= 4087, max= 4087, per=10.98%, avg=4087.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 write: IOPS=18, BW=18.2MiB/s (19.1MB/s)(10.0MiB/550msec) clat (msec): min=20, max=107, avg=47.07, stdev=29.07 lat (msec): min=20, max=107, avg=47.12, stdev=29.06 clat percentiles (msec): | 1.00th=[ 21], 5.00th=[ 21], 10.00th=[ 21], 20.00th=[ 23], | 30.00th=[ 26], 40.00th=[ 29], 50.00th=[ 34], 60.00th=[ 39], | 70.00th=[ 66], 80.00th=[ 66], 90.00th=[ 83], 95.00th=[ 108], | 99.00th=[ 108], 99.50th=[ 108], 99.90th=[ 108], 99.95th=[ 108], | 99.99th=[ 108] bw ( KiB/s): min=16351, max=16351, per=31.37%, avg=16351.00, stdev= 0.00, samples=1 iops : min= 15, max= 15, avg=15.00, stdev= 0.00, samples=1 lat (msec) : 50=66.67%, 100=25.00%, 250=8.33% cpu : usr=0.00%, sys=34.43%, ctx=89, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23215: Sat Feb 24 23:28:36 2024 read: IOPS=12, BW=12.8MiB/s (13.4MB/s)(6144KiB/468msec) clat (usec): min=333, max=72321, avg=31393.81, stdev=30943.74 lat (usec): min=334, max=72322, avg=31394.68, stdev=30943.66 clat percentiles (usec): | 1.00th=[ 334], 5.00th=[ 334], 10.00th=[ 334], 20.00th=[ 453], | 30.00th=[ 453], 40.00th=[12649], 50.00th=[12649], 60.00th=[46400], | 70.00th=[56361], 80.00th=[56361], 90.00th=[71828], 95.00th=[71828], | 99.00th=[71828], 99.50th=[71828], 99.90th=[71828], 99.95th=[71828], | 99.99th=[71828] write: IOPS=12, BW=12.8MiB/s (13.4MB/s)(6144KiB/468msec) clat (msec): min=16, max=112, avg=46.24, stdev=36.52 lat (msec): min=16, max=112, avg=46.45, stdev=36.42 clat percentiles (msec): | 1.00th=[ 17], 5.00th=[ 17], 10.00th=[ 17], 20.00th=[ 20], | 30.00th=[ 20], 40.00th=[ 26], 50.00th=[ 26], 60.00th=[ 45], | 70.00th=[ 60], 80.00th=[ 60], 90.00th=[ 113], 95.00th=[ 113], | 99.00th=[ 113], 99.50th=[ 113], 99.90th=[ 113], 99.95th=[ 113], | 99.99th=[ 113] lat (usec) : 500=16.67% lat (msec) : 20=25.00%, 50=25.00%, 100=25.00%, 250=8.33% cpu : usr=0.00%, sys=30.19%, ctx=87, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23216: Sat Feb 24 23:28:36 2024 read: IOPS=10, BW=10.6MiB/s (11.1MB/s)(5120KiB/471msec) clat (usec): min=338, max=50039, avg=24912.68, stdev=23315.86 lat (usec): min=338, max=50040, avg=24913.57, stdev=23315.95 clat percentiles (usec): | 1.00th=[ 338], 5.00th=[ 338], 10.00th=[ 338], 20.00th=[ 338], | 30.00th=[ 420], 40.00th=[ 420], 50.00th=[31851], 60.00th=[31851], | 70.00th=[42206], 80.00th=[42206], 90.00th=[50070], 95.00th=[50070], | 99.00th=[50070], 99.50th=[50070], 99.90th=[50070], 99.95th=[50070], | 99.99th=[50070] write: IOPS=14, BW=14.9MiB/s (15.6MB/s)(7168KiB/471msec) clat (msec): min=14, max=113, avg=49.30, stdev=31.87 lat (msec): min=14, max=113, avg=49.35, stdev=31.87 clat percentiles (msec): | 1.00th=[ 15], 5.00th=[ 15], 10.00th=[ 15], 20.00th=[ 31], | 30.00th=[ 35], 40.00th=[ 35], 50.00th=[ 40], 60.00th=[ 55], | 70.00th=[ 55], 80.00th=[ 59], 90.00th=[ 114], 95.00th=[ 114], | 99.00th=[ 114], 99.50th=[ 114], 99.90th=[ 114], 99.95th=[ 114], | 99.99th=[ 114] lat (usec) : 500=16.67% lat (msec) : 20=8.33%, 50=41.67%, 100=25.00%, 250=8.33% cpu : usr=0.00%, sys=27.66%, ctx=89, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=36.4MiB/s (38.1MB/s), 3724KiB/s-16.7MiB/s (3813kB/s-17.5MB/s), io=20.0MiB (20.0MB), run=419-550msec WRITE: bw=50.9MiB/s (53.4MB/s), 11.9MiB/s-18.2MiB/s (12.5MB/s-19.1MB/s), io=28.0MiB (29.4MB), run=419-550msec rand-rw: (groupid=0, jobs=1): err= 0: pid=23218: Sat Feb 24 23:28:36 2024 read: IOPS=11, BW=11.7MiB/s (12.3MB/s)(7168KiB/597msec) clat (msec): min=25, max=150, avg=52.59, stdev=43.77 lat (msec): min=25, max=150, avg=52.59, stdev=43.77 clat percentiles (msec): | 1.00th=[ 26], 5.00th=[ 26], 10.00th=[ 26], 20.00th=[ 30], | 30.00th=[ 37], 40.00th=[ 37], 50.00th=[ 42], 60.00th=[ 43], | 70.00th=[ 43], 80.00th=[ 43], 90.00th=[ 150], 95.00th=[ 150], | 99.00th=[ 150], 99.50th=[ 150], 99.90th=[ 150], 99.95th=[ 150], | 99.99th=[ 150] bw ( KiB/s): min=10219, max=10219, per=34.68%, avg=10219.00, stdev= 0.00, samples=1 iops : min= 9, max= 9, avg= 9.00, stdev= 0.00, samples=1 write: IOPS=8, BW=8576KiB/s (8782kB/s)(5120KiB/597msec) clat (usec): min=30959, max=66483, avg=45250.73, stdev=15898.59 lat (usec): min=31020, max=66556, avg=45317.14, stdev=15900.64 clat percentiles (usec): | 1.00th=[31065], 5.00th=[31065], 10.00th=[31065], 20.00th=[31065], | 30.00th=[33162], 40.00th=[33162], 50.00th=[38011], 60.00th=[38011], | 70.00th=[57934], 80.00th=[57934], 90.00th=[66323], 95.00th=[66323], | 99.00th=[66323], 99.50th=[66323], 99.90th=[66323], 99.95th=[66323], | 99.99th=[66323] bw ( KiB/s): min= 8175, max= 8175, per=19.82%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 7, max= 7, avg= 7.00, stdev= 0.00, samples=1 lat (msec) : 50=75.00%, 100=16.67%, 250=8.33% cpu : usr=0.17%, sys=18.12%, ctx=51, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23219: Sat Feb 24 23:28:36 2024 read: IOPS=3, BW=3156KiB/s (3231kB/s)(2048KiB/649msec) clat (usec): min=18710, max=40327, avg=29518.75, stdev=15285.36 lat (usec): min=18711, max=40328, avg=29519.62, stdev=15285.36 clat percentiles (usec): | 1.00th=[18744], 5.00th=[18744], 10.00th=[18744], 20.00th=[18744], | 30.00th=[18744], 40.00th=[18744], 50.00th=[18744], 60.00th=[40109], | 70.00th=[40109], 80.00th=[40109], 90.00th=[40109], 95.00th=[40109], | 99.00th=[40109], 99.50th=[40109], 99.90th=[40109], 99.95th=[40109], | 99.99th=[40109] bw ( KiB/s): min= 4087, max= 4087, per=13.87%, avg=4087.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 write: IOPS=15, BW=15.4MiB/s (16.2MB/s)(10.0MiB/649msec) clat (msec): min=30, max=123, avg=58.79, stdev=32.51 lat (msec): min=30, max=123, avg=58.85, stdev=32.51 clat percentiles (msec): | 1.00th=[ 31], 5.00th=[ 31], 10.00th=[ 31], 20.00th=[ 33], | 30.00th=[ 36], 40.00th=[ 41], 50.00th=[ 43], 60.00th=[ 48], | 70.00th=[ 78], 80.00th=[ 78], 90.00th=[ 106], 95.00th=[ 125], | 99.00th=[ 125], 99.50th=[ 125], 99.90th=[ 125], 99.95th=[ 125], | 99.99th=[ 125] bw ( KiB/s): min=12263, max=12263, per=29.73%, avg=12263.00, stdev= 0.00, samples=1 iops : min= 11, max= 11, avg=11.00, stdev= 0.00, samples=1 lat (msec) : 20=8.33%, 50=58.33%, 100=16.67%, 250=16.67% cpu : usr=0.00%, sys=15.74%, ctx=46, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23220: Sat Feb 24 23:28:36 2024 read: IOPS=9, BW=9496KiB/s (9724kB/s)(6144KiB/647msec) clat (usec): min=21393, max=67836, avg=36226.55, stdev=17468.02 lat (usec): min=21393, max=67837, avg=36227.44, stdev=17468.10 clat percentiles (usec): | 1.00th=[21365], 5.00th=[21365], 10.00th=[21365], 20.00th=[22676], | 30.00th=[22676], 40.00th=[30278], 50.00th=[30278], 60.00th=[31327], | 70.00th=[43779], 80.00th=[43779], 90.00th=[67634], 95.00th=[67634], | 99.00th=[67634], 99.50th=[67634], 99.90th=[67634], 99.95th=[67634], | 99.99th=[67634] bw ( KiB/s): min= 6144, max= 6144, per=20.85%, avg=6144.00, stdev= 0.00, samples=1 iops : min= 6, max= 6, avg= 6.00, stdev= 0.00, samples=1 write: IOPS=9, BW=9496KiB/s (9724kB/s)(6144KiB/647msec) clat (msec): min=32, max=155, avg=70.20, stdev=46.44 lat (msec): min=32, max=155, avg=70.25, stdev=46.46 clat percentiles (msec): | 1.00th=[ 33], 5.00th=[ 33], 10.00th=[ 33], 20.00th=[ 39], | 30.00th=[ 39], 40.00th=[ 41], 50.00th=[ 41], 60.00th=[ 70], | 70.00th=[ 85], 80.00th=[ 85], 90.00th=[ 157], 95.00th=[ 157], | 99.00th=[ 157], 99.50th=[ 157], 99.90th=[ 157], 99.95th=[ 157], | 99.99th=[ 157] bw ( KiB/s): min= 8192, max= 8192, per=19.86%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 lat (msec) : 50=66.67%, 100=25.00%, 250=8.33% cpu : usr=0.00%, sys=15.79%, ctx=99, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23221: Sat Feb 24 23:28:36 2024 read: IOPS=7, BW=7367KiB/s (7544kB/s)(5120KiB/695msec) clat (usec): min=19355, max=69387, avg=38211.52, stdev=20282.41 lat (usec): min=19356, max=69388, avg=38212.41, stdev=20282.29 clat percentiles (usec): | 1.00th=[19268], 5.00th=[19268], 10.00th=[19268], 20.00th=[19268], | 30.00th=[21890], 40.00th=[21890], 50.00th=[35914], 60.00th=[35914], | 70.00th=[44827], 80.00th=[44827], 90.00th=[69731], 95.00th=[69731], | 99.00th=[69731], 99.50th=[69731], 99.90th=[69731], 99.95th=[69731], | 99.99th=[69731] bw ( KiB/s): min= 2039, max= 2039, per=6.92%, avg=2039.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=10, BW=10.1MiB/s (10.6MB/s)(7168KiB/695msec) clat (msec): min=29, max=143, avg=71.69, stdev=51.54 lat (msec): min=29, max=143, avg=71.76, stdev=51.54 clat percentiles (msec): | 1.00th=[ 31], 5.00th=[ 31], 10.00th=[ 31], 20.00th=[ 33], | 30.00th=[ 34], 40.00th=[ 34], 50.00th=[ 44], 60.00th=[ 77], | 70.00th=[ 77], 80.00th=[ 144], 90.00th=[ 144], 95.00th=[ 144], | 99.00th=[ 144], 99.50th=[ 144], 99.90th=[ 144], 99.95th=[ 144], | 99.99th=[ 144] bw ( KiB/s): min= 8159, max= 8159, per=19.78%, avg=8159.00, stdev= 0.00, samples=1 iops : min= 7, max= 7, avg= 7.00, stdev= 0.00, samples=1 lat (msec) : 20=8.33%, 50=58.33%, 100=16.67%, 250=16.67% cpu : usr=0.00%, sys=14.41%, ctx=39, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=28.8MiB/s (30.2MB/s), 3156KiB/s-11.7MiB/s (3231kB/s-12.3MB/s), io=20.0MiB (20.0MB), run=597-695msec WRITE: bw=40.3MiB/s (42.2MB/s), 8576KiB/s-15.4MiB/s (8782kB/s-16.2MB/s), io=28.0MiB (29.4MB), run=597-695msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=23227: Sat Feb 24 23:28:38 2024 read: IOPS=8, BW=33.1MiB/s (34.8MB/s)(12.0MiB/362msec) clat (msec): min=74, max=199, avg=117.35, stdev=71.14 lat (msec): min=74, max=199, avg=117.35, stdev=71.14 clat percentiles (msec): | 1.00th=[ 75], 5.00th=[ 75], 10.00th=[ 75], 20.00th=[ 75], | 30.00th=[ 75], 40.00th=[ 79], 50.00th=[ 79], 60.00th=[ 79], | 70.00th=[ 201], 80.00th=[ 201], 90.00th=[ 201], 95.00th=[ 201], | 99.00th=[ 201], 99.50th=[ 201], 99.90th=[ 201], 99.95th=[ 201], | 99.99th=[ 201] lat (msec) : 100=66.67%, 250=33.33% cpu : usr=0.00%, sys=49.31%, ctx=182, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23229: Sat Feb 24 23:28:38 2024 read: IOPS=2, BW=8463KiB/s (8666kB/s)(4096KiB/484msec) clat (nsec): min=172971k, max=172971k, avg=172971391.00, stdev= 0.00 lat (nsec): min=172974k, max=172974k, avg=172973723.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 174], 5.00th=[ 174], 10.00th=[ 174], 20.00th=[ 174], | 30.00th=[ 174], 40.00th=[ 174], 50.00th=[ 174], 60.00th=[ 174], | 70.00th=[ 174], 80.00th=[ 174], 90.00th=[ 174], 95.00th=[ 174], | 99.00th=[ 174], 99.50th=[ 174], 99.90th=[ 174], 99.95th=[ 174], | 99.99th=[ 174] write: IOPS=4, BW=16.5MiB/s (17.3MB/s)(8192KiB/484msec) clat (msec): min=103, max=175, avg=139.60, stdev=50.91 lat (msec): min=103, max=175, avg=139.85, stdev=50.95 clat percentiles (msec): | 1.00th=[ 104], 5.00th=[ 104], 10.00th=[ 104], 20.00th=[ 104], | 30.00th=[ 104], 40.00th=[ 104], 50.00th=[ 104], 60.00th=[ 176], | 70.00th=[ 176], 80.00th=[ 176], 90.00th=[ 176], 95.00th=[ 176], | 99.00th=[ 176], 99.50th=[ 176], 99.90th=[ 176], 99.95th=[ 176], | 99.99th=[ 176] lat (msec) : 250=100.00% cpu : usr=0.00%, sys=37.68%, ctx=146, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23231: Sat Feb 24 23:28:38 2024 write: IOPS=5, BW=21.7MiB/s (22.7MB/s)(12.0MiB/554msec) clat (msec): min=159, max=192, avg=178.98, stdev=17.12 lat (msec): min=160, max=192, avg=179.25, stdev=17.16 clat percentiles (msec): | 1.00th=[ 161], 5.00th=[ 161], 10.00th=[ 161], 20.00th=[ 161], | 30.00th=[ 161], 40.00th=[ 184], 50.00th=[ 184], 60.00th=[ 184], | 70.00th=[ 192], 80.00th=[ 192], 90.00th=[ 192], 95.00th=[ 192], | 99.00th=[ 192], 99.50th=[ 192], 99.90th=[ 192], 99.95th=[ 192], | 99.99th=[ 192] bw ( KiB/s): min=15875, max=15875, per=26.84%, avg=15875.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 lat (msec) : 250=100.00% cpu : usr=0.00%, sys=37.97%, ctx=62, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23233: Sat Feb 24 23:28:38 2024 write: IOPS=5, BW=22.8MiB/s (23.9MB/s)(12.0MiB/526msec) clat (msec): min=102, max=282, avg=174.85, stdev=95.36 lat (msec): min=102, max=283, avg=175.02, stdev=95.37 clat percentiles (msec): | 1.00th=[ 104], 5.00th=[ 104], 10.00th=[ 104], 20.00th=[ 104], | 30.00th=[ 104], 40.00th=[ 140], 50.00th=[ 140], 60.00th=[ 140], | 70.00th=[ 284], 80.00th=[ 284], 90.00th=[ 284], 95.00th=[ 284], | 99.00th=[ 284], 99.50th=[ 284], 99.90th=[ 284], 99.95th=[ 284], | 99.99th=[ 284] bw ( KiB/s): min=16384, max=16384, per=27.70%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.00%, sys=35.05%, ctx=75, majf=0, minf=28 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=33.1MiB/s (34.7MB/s), 8463KiB/s-33.1MiB/s (8666kB/s-34.8MB/s), io=16.0MiB (16.8MB), run=362-484msec WRITE: bw=57.8MiB/s (60.6MB/s), 16.5MiB/s-22.8MiB/s (17.3MB/s-23.9MB/s), io=32.0MiB (33.6MB), run=484-554msec rand-rw: (groupid=0, jobs=1): err= 0: pid=23228: Sat Feb 24 23:28:38 2024 read: IOPS=4, BW=18.8MiB/s (19.7MB/s)(12.0MiB/640msec) clat (msec): min=62, max=307, avg=203.59, stdev=126.84 lat (msec): min=62, max=307, avg=203.60, stdev=126.84 clat percentiles (msec): | 1.00th=[ 63], 5.00th=[ 63], 10.00th=[ 63], 20.00th=[ 63], | 30.00th=[ 63], 40.00th=[ 241], 50.00th=[ 241], 60.00th=[ 241], | 70.00th=[ 309], 80.00th=[ 309], 90.00th=[ 309], 95.00th=[ 309], | 99.00th=[ 309], 99.50th=[ 309], 99.90th=[ 309], 99.95th=[ 309], | 99.99th=[ 309] bw ( KiB/s): min= 8192, max= 8192, per=32.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 250=33.33%, 500=33.33% cpu : usr=0.00%, sys=15.49%, ctx=42, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23230: Sat Feb 24 23:28:38 2024 read: IOPS=1, BW=6543KiB/s (6700kB/s)(4096KiB/626msec) clat (nsec): min=350987k, max=350987k, avg=350986671.00, stdev= 0.00 lat (nsec): min=350988k, max=350988k, avg=350987586.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 351], 5.00th=[ 351], 10.00th=[ 351], 20.00th=[ 351], | 30.00th=[ 351], 40.00th=[ 351], 50.00th=[ 351], 60.00th=[ 351], | 70.00th=[ 351], 80.00th=[ 351], 90.00th=[ 351], 95.00th=[ 351], | 99.00th=[ 351], 99.50th=[ 351], 99.90th=[ 351], 99.95th=[ 351], | 99.99th=[ 351] bw ( KiB/s): min= 8192, max= 8192, per=32.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=3, BW=12.8MiB/s (13.4MB/s)(8192KiB/626msec) clat (msec): min=81, max=165, avg=123.31, stdev=59.62 lat (msec): min=81, max=172, avg=126.99, stdev=64.51 clat percentiles (msec): | 1.00th=[ 82], 5.00th=[ 82], 10.00th=[ 82], 20.00th=[ 82], | 30.00th=[ 82], 40.00th=[ 82], 50.00th=[ 82], 60.00th=[ 165], | 70.00th=[ 165], 80.00th=[ 165], 90.00th=[ 165], 95.00th=[ 165], | 99.00th=[ 165], 99.50th=[ 165], 99.90th=[ 165], 99.95th=[ 165], | 99.99th=[ 165] bw ( KiB/s): min= 8192, max= 8192, per=18.38%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 250=33.33%, 500=33.33% cpu : usr=0.00%, sys=17.28%, ctx=36, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23232: Sat Feb 24 23:28:38 2024 write: IOPS=4, BW=16.3MiB/s (17.1MB/s)(12.0MiB/735msec) clat (msec): min=50, max=375, avg=235.93, stdev=167.46 lat (msec): min=50, max=375, avg=236.22, stdev=167.44 clat percentiles (msec): | 1.00th=[ 51], 5.00th=[ 51], 10.00th=[ 51], 20.00th=[ 51], | 30.00th=[ 51], 40.00th=[ 284], 50.00th=[ 284], 60.00th=[ 284], | 70.00th=[ 376], 80.00th=[ 376], 90.00th=[ 376], 95.00th=[ 376], | 99.00th=[ 376], 99.50th=[ 376], 99.90th=[ 376], 99.95th=[ 376], | 99.99th=[ 376] bw ( KiB/s): min= 8175, max= 8175, per=18.34%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 500=66.67% cpu : usr=0.14%, sys=13.22%, ctx=47, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23234: Sat Feb 24 23:28:38 2024 write: IOPS=5, BW=20.6MiB/s (21.6MB/s)(12.0MiB/582msec) clat (msec): min=137, max=262, avg=184.18, stdev=68.08 lat (msec): min=137, max=262, avg=184.56, stdev=68.00 clat percentiles (msec): | 1.00th=[ 138], 5.00th=[ 138], 10.00th=[ 138], 20.00th=[ 138], | 30.00th=[ 138], 40.00th=[ 155], 50.00th=[ 155], 60.00th=[ 155], | 70.00th=[ 264], 80.00th=[ 264], 90.00th=[ 264], 95.00th=[ 264], | 99.00th=[ 264], 99.50th=[ 264], 99.90th=[ 264], 99.95th=[ 264], | 99.99th=[ 264] bw ( KiB/s): min=16384, max=16384, per=36.75%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.17%, sys=21.86%, ctx=158, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=25.0MiB/s (26.2MB/s), 6543KiB/s-18.8MiB/s (6700kB/s-19.7MB/s), io=16.0MiB (16.8MB), run=626-640msec WRITE: bw=43.5MiB/s (45.7MB/s), 12.8MiB/s-20.6MiB/s (13.4MB/s-21.6MB/s), io=32.0MiB (33.6MB), run=582-735msec