== sanity test 398b: DIO and buffer IO race ============== 01:57:13 (1746079033) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 2.34789 s, 21.4 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=25965: Thu May 1 01:57:54 2025 read: IOPS=43, BW=173KiB/s (177kB/s)(5992KiB/34632msec) clat (usec): min=82, max=359945, avg=12919.71, stdev=12008.88 lat (usec): min=83, max=359948, avg=12921.34, stdev=12009.15 clat percentiles (usec): | 1.00th=[ 98], 5.00th=[ 161], 10.00th=[ 4948], 20.00th=[ 6652], | 30.00th=[ 8160], 40.00th=[ 9634], 50.00th=[ 11338], 60.00th=[ 13173], | 70.00th=[ 15270], 80.00th=[ 17957], 90.00th=[ 22676], 95.00th=[ 27132], | 99.00th=[ 38011], 99.50th=[ 47449], 99.90th=[ 69731], 99.95th=[358613], | 99.99th=[358613] bw ( KiB/s): min= 80, max= 328, per=25.98%, avg=174.56, stdev=47.25, samples=68 iops : min= 20, max= 82, avg=43.37, stdev=11.93, samples=68 write: IOPS=45, BW=182KiB/s (186kB/s)(6296KiB/34632msec) clat (usec): min=473, max=65556, avg=9558.30, stdev=10163.63 lat (usec): min=474, max=65557, avg=9561.57, stdev=10165.42 clat percentiles (usec): | 1.00th=[ 685], 5.00th=[ 750], 10.00th=[ 816], 20.00th=[ 938], | 30.00th=[ 1598], 40.00th=[ 3294], 50.00th=[ 6718], 60.00th=[ 9241], | 70.00th=[12125], 80.00th=[16188], 90.00th=[24249], 95.00th=[29754], | 99.00th=[44303], 99.50th=[47973], 99.90th=[64226], 99.95th=[65799], | 99.99th=[65799] bw ( KiB/s): min= 63, max= 344, per=26.08%, avg=183.60, stdev=55.01, samples=68 iops : min= 15, max= 86, avg=45.63, stdev=13.83, samples=68 lat (usec) : 100=0.59%, 250=1.95%, 500=0.26%, 750=2.54%, 1000=8.92% lat (msec) : 2=5.92%, 4=4.92%, 10=27.70%, 20=32.26%, 50=14.58% lat (msec) : 100=0.33%, 500=0.03% cpu : usr=0.15%, sys=12.52%, ctx=5166, majf=0, minf=37 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25966: Thu May 1 01:57:54 2025 read: IOPS=41, BW=166KiB/s (170kB/s)(5940KiB/35691msec) clat (usec): min=68, max=389925, avg=13252.98, stdev=12996.64 lat (usec): min=69, max=389928, avg=13254.17, stdev=12996.71 clat percentiles (usec): | 1.00th=[ 92], 5.00th=[ 135], 10.00th=[ 4948], 20.00th=[ 6652], | 30.00th=[ 8094], 40.00th=[ 9634], 50.00th=[ 11338], 60.00th=[ 13304], | 70.00th=[ 15533], 80.00th=[ 18744], 90.00th=[ 23987], 95.00th=[ 28705], | 99.00th=[ 43254], 99.50th=[ 47449], 99.90th=[ 80217], 99.95th=[387974], | 99.99th=[387974] bw ( KiB/s): min= 32, max= 287, per=24.93%, avg=167.51, stdev=50.11, samples=70 iops : min= 8, max= 71, avg=41.56, stdev=12.50, samples=70 write: IOPS=44, BW=178KiB/s (182kB/s)(6348KiB/35691msec) clat (usec): min=485, max=59637, avg=9930.15, stdev=10369.10 lat (usec): min=485, max=59639, avg=9931.67, stdev=10369.08 clat percentiles (usec): | 1.00th=[ 668], 5.00th=[ 750], 10.00th=[ 807], 20.00th=[ 955], | 30.00th=[ 1696], 40.00th=[ 3818], 50.00th=[ 7177], 60.00th=[ 9765], | 70.00th=[12780], 80.00th=[17171], 90.00th=[24249], 95.00th=[31327], | 99.00th=[44827], 99.50th=[53216], 99.90th=[57934], 99.95th=[59507], | 99.99th=[59507] bw ( KiB/s): min= 80, max= 363, per=25.36%, avg=178.51, stdev=54.08, samples=70 iops : min= 20, max= 90, avg=44.30, stdev=13.49, samples=70 lat (usec) : 100=0.94%, 250=1.86%, 500=0.23%, 750=2.60%, 1000=8.82% lat (msec) : 2=5.57%, 4=4.62%, 10=27.31%, 20=31.84%, 50=15.69% lat (msec) : 100=0.49%, 500=0.03% cpu : usr=0.16%, sys=12.45%, ctx=5180, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25967: Thu May 1 01:57:54 2025 read: IOPS=42, BW=171KiB/s (176kB/s)(6028KiB/35166msec) clat (usec): min=66, max=435923, avg=13308.50, stdev=13961.87 lat (usec): min=67, max=435925, avg=13311.10, stdev=13962.24 clat percentiles (usec): | 1.00th=[ 95], 5.00th=[ 135], 10.00th=[ 4752], 20.00th=[ 6652], | 30.00th=[ 8225], 40.00th=[ 9896], 50.00th=[ 11469], 60.00th=[ 13304], | 70.00th=[ 15401], 80.00th=[ 18482], 90.00th=[ 23462], 95.00th=[ 28181], | 99.00th=[ 42730], 99.50th=[ 53216], 99.90th=[ 80217], 99.95th=[434111], | 99.99th=[434111] bw ( KiB/s): min= 8, max= 328, per=25.40%, avg=170.66, stdev=51.42, samples=70 iops : min= 2, max= 82, avg=42.39, stdev=12.94, samples=70 write: IOPS=44, BW=178KiB/s (182kB/s)(6260KiB/35166msec) clat (usec): min=485, max=69905, avg=9569.85, stdev=10412.83 lat (usec): min=486, max=69906, avg=9571.58, stdev=10413.09 clat percentiles (usec): | 1.00th=[ 668], 5.00th=[ 750], 10.00th=[ 799], 20.00th=[ 930], | 30.00th=[ 1434], 40.00th=[ 2868], 50.00th=[ 6587], 60.00th=[ 8979], | 70.00th=[12125], 80.00th=[16909], 90.00th=[24249], 95.00th=[31065], | 99.00th=[44303], 99.50th=[47973], 99.90th=[66323], 99.95th=[69731], | 99.99th=[69731] bw ( KiB/s): min= 24, max= 376, per=25.13%, avg=176.93, stdev=60.57, samples=70 iops : min= 6, max= 94, avg=43.96, stdev=15.20, samples=70 lat (usec) : 100=0.85%, 250=2.21%, 500=0.20%, 750=2.47%, 1000=8.95% lat (msec) : 2=6.93%, 4=4.10%, 10=26.56%, 20=31.90%, 50=15.30% lat (msec) : 100=0.49%, 500=0.03% cpu : usr=0.16%, sys=12.52%, ctx=5096, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25968: Thu May 1 01:57:54 2025 read: IOPS=43, BW=173KiB/s (177kB/s)(6052KiB/35066msec) clat (usec): min=81, max=100216, avg=12795.21, stdev=8494.59 lat (usec): min=81, max=100218, avg=12797.47, stdev=8494.41 clat percentiles (usec): | 1.00th=[ 99], 5.00th=[ 392], 10.00th=[ 5014], 20.00th=[ 6718], | 30.00th=[ 8160], 40.00th=[ 9503], 50.00th=[ 11338], 60.00th=[ 13042], | 70.00th=[ 15139], 80.00th=[ 18220], 90.00th=[ 22152], 95.00th=[ 27395], | 99.00th=[ 39060], 99.50th=[ 49021], 99.90th=[ 99091], 99.95th=[100140], | 99.99th=[100140] bw ( KiB/s): min= 55, max= 327, per=25.96%, avg=174.48, stdev=49.01, samples=69 iops : min= 13, max= 81, avg=43.28, stdev=12.30, samples=69 write: IOPS=44, BW=178KiB/s (182kB/s)(6236KiB/35066msec) clat (usec): min=484, max=387507, avg=9892.19, stdev=14205.32 lat (usec): min=485, max=387510, avg=9893.80, stdev=14205.25 clat percentiles (usec): | 1.00th=[ 660], 5.00th=[ 750], 10.00th=[ 799], 20.00th=[ 914], | 30.00th=[ 1385], 40.00th=[ 2900], 50.00th=[ 6718], 60.00th=[ 9372], | 70.00th=[ 12649], 80.00th=[ 16909], 90.00th=[ 23987], 95.00th=[ 30540], | 99.00th=[ 46400], 99.50th=[ 54264], 99.90th=[ 73925], 99.95th=[387974], | 99.99th=[387974] bw ( KiB/s): min= 79, max= 376, per=25.52%, avg=179.67, stdev=56.92, samples=69 iops : min= 19, max= 94, avg=44.58, stdev=14.33, samples=69 lat (usec) : 100=0.59%, 250=1.86%, 500=0.13%, 750=2.25%, 1000=9.70% lat (msec) : 2=6.41%, 4=4.23%, 10=27.28%, 20=32.85%, 50=14.10% lat (msec) : 100=0.55%, 250=0.03%, 500=0.03% cpu : usr=0.19%, sys=12.39%, ctx=5149, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=673KiB/s (689kB/s), 166KiB/s-173KiB/s (170kB/s-177kB/s), io=23.4MiB (24.6MB), run=34632-35691msec WRITE: bw=704KiB/s (721kB/s), 178KiB/s-182KiB/s (182kB/s-186kB/s), io=24.6MiB (25.7MB), run=34632-35691msec rand-rw: (groupid=0, jobs=1): err= 0: pid=25960: Thu May 1 01:58:08 2025 read: IOPS=30, BW=121KiB/s (123kB/s)(5992KiB/49703msec) clat (msec): min=2, max=447, avg=12.67, stdev=15.80 lat (msec): min=2, max=447, avg=12.67, stdev=15.80 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 4], 10.00th=[ 5], 20.00th=[ 6], | 30.00th=[ 7], 40.00th=[ 7], 50.00th=[ 8], 60.00th=[ 10], | 70.00th=[ 13], 80.00th=[ 18], 90.00th=[ 28], 95.00th=[ 37], | 99.00th=[ 55], 99.50th=[ 63], 99.90th=[ 97], 99.95th=[ 447], | 99.99th=[ 447] bw ( KiB/s): min= 39, max= 296, per=25.31%, avg=120.96, stdev=63.93, samples=98 iops : min= 9, max= 74, avg=29.93, stdev=16.03, samples=98 write: IOPS=31, BW=127KiB/s (130kB/s)(6296KiB/49703msec) clat (usec): min=5109, max=98749, avg=19410.43, stdev=12278.26 lat (usec): min=5111, max=98752, avg=19412.88, stdev=12278.30 clat percentiles (usec): | 1.00th=[ 6128], 5.00th=[ 7832], 10.00th=[ 8717], 20.00th=[10159], | 30.00th=[11600], 40.00th=[13173], 50.00th=[15139], 60.00th=[17695], | 70.00th=[21627], 80.00th=[26870], 90.00th=[36439], 95.00th=[45351], | 99.00th=[62129], 99.50th=[68682], 99.90th=[77071], 99.95th=[99091], | 99.99th=[99091] bw ( KiB/s): min= 31, max= 296, per=25.42%, avg=127.09, stdev=66.20, samples=98 iops : min= 7, max= 74, avg=31.46, stdev=16.60, samples=98 lat (msec) : 4=2.47%, 10=37.27%, 20=35.03%, 50=22.79%, 100=2.41% lat (msec) : 500=0.03% cpu : usr=0.10%, sys=6.16%, ctx=3477, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25961: Thu May 1 01:58:08 2025 read: IOPS=29, BW=119KiB/s (122kB/s)(5940KiB/50027msec) clat (msec): min=2, max=494, avg=12.55, stdev=16.80 lat (msec): min=2, max=494, avg=12.55, stdev=16.80 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 4], 10.00th=[ 5], 20.00th=[ 6], | 30.00th=[ 7], 40.00th=[ 7], 50.00th=[ 9], 60.00th=[ 10], | 70.00th=[ 12], 80.00th=[ 17], 90.00th=[ 27], 95.00th=[ 37], | 99.00th=[ 59], 99.50th=[ 64], 99.90th=[ 91], 99.95th=[ 493], | 99.99th=[ 493] bw ( KiB/s): min= 16, max= 335, per=25.01%, avg=119.54, stdev=68.76, samples=99 iops : min= 4, max= 83, avg=29.60, stdev=17.25, samples=99 write: IOPS=31, BW=127KiB/s (130kB/s)(6348KiB/50027msec) clat (msec): min=4, max=114, avg=19.66, stdev=13.10 lat (msec): min=4, max=114, avg=19.66, stdev=13.10 clat percentiles (msec): | 1.00th=[ 6], 5.00th=[ 8], 10.00th=[ 9], 20.00th=[ 11], | 30.00th=[ 12], 40.00th=[ 14], 50.00th=[ 15], 60.00th=[ 18], | 70.00th=[ 22], 80.00th=[ 28], 90.00th=[ 39], 95.00th=[ 46], | 99.00th=[ 65], 99.50th=[ 79], 99.90th=[ 109], 99.95th=[ 115], | 99.99th=[ 115] bw ( KiB/s): min= 48, max= 272, per=25.54%, avg=127.69, stdev=67.12, samples=99 iops : min= 12, max= 68, avg=31.64, stdev=16.83, samples=99 lat (msec) : 4=2.83%, 10=37.08%, 20=35.32%, 50=21.94%, 100=2.73% lat (msec) : 250=0.07%, 500=0.03% cpu : usr=0.15%, sys=6.09%, ctx=3399, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25962: Thu May 1 01:58:08 2025 read: IOPS=30, BW=120KiB/s (123kB/s)(6028KiB/50097msec) clat (usec): min=2813, max=76037, avg=12069.95, stdev=10861.88 lat (usec): min=2814, max=76038, avg=12071.31, stdev=10861.93 clat percentiles (usec): | 1.00th=[ 3392], 5.00th=[ 3982], 10.00th=[ 4424], 20.00th=[ 5145], | 30.00th=[ 5997], 40.00th=[ 6915], 50.00th=[ 7963], 60.00th=[ 9241], | 70.00th=[11863], 80.00th=[16319], 90.00th=[26346], 95.00th=[35914], | 99.00th=[55837], 99.50th=[63701], 99.90th=[74974], 99.95th=[76022], | 99.99th=[76022] bw ( KiB/s): min= 39, max= 304, per=25.29%, avg=120.91, stdev=66.41, samples=99 iops : min= 9, max= 76, avg=29.95, stdev=16.64, samples=99 write: IOPS=31, BW=125KiB/s (128kB/s)(6260KiB/50097msec) clat (msec): min=4, max=520, avg=20.27, stdev=18.44 lat (msec): min=4, max=520, avg=20.27, stdev=18.44 clat percentiles (msec): | 1.00th=[ 7], 5.00th=[ 8], 10.00th=[ 9], 20.00th=[ 11], | 30.00th=[ 12], 40.00th=[ 14], 50.00th=[ 16], 60.00th=[ 18], | 70.00th=[ 23], 80.00th=[ 29], 90.00th=[ 38], 95.00th=[ 48], | 99.00th=[ 67], 99.50th=[ 82], 99.90th=[ 116], 99.95th=[ 523], | 99.99th=[ 523] bw ( KiB/s): min= 32, max= 296, per=25.11%, avg=125.53, stdev=67.76, samples=99 iops : min= 8, max= 74, avg=31.10, stdev=16.95, samples=99 lat (msec) : 4=2.54%, 10=37.86%, 20=34.77%, 50=21.91%, 100=2.83% lat (msec) : 250=0.07%, 750=0.03% cpu : usr=0.14%, sys=6.05%, ctx=3450, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25963: Thu May 1 01:58:08 2025 read: IOPS=30, BW=121KiB/s (123kB/s)(6052KiB/50195msec) clat (usec): min=2576, max=90560, avg=12764.42, stdev=11918.99 lat (usec): min=2576, max=90562, avg=12767.19, stdev=11921.95 clat percentiles (usec): | 1.00th=[ 3228], 5.00th=[ 3785], 10.00th=[ 4359], 20.00th=[ 5145], | 30.00th=[ 5932], 40.00th=[ 6915], 50.00th=[ 8094], 60.00th=[ 9765], | 70.00th=[11994], 80.00th=[17695], 90.00th=[29230], 95.00th=[40633], | 99.00th=[57934], 99.50th=[63177], 99.90th=[78119], 99.95th=[90702], | 99.99th=[90702] bw ( KiB/s): min= 24, max= 304, per=25.20%, avg=120.45, stdev=70.03, samples=99 iops : min= 6, max= 76, avg=29.79, stdev=17.52, samples=99 write: IOPS=31, BW=124KiB/s (127kB/s)(6236KiB/50195msec) clat (msec): min=5, max=516, avg=19.69, stdev=17.85 lat (msec): min=5, max=516, avg=19.69, stdev=17.85 clat percentiles (msec): | 1.00th=[ 7], 5.00th=[ 8], 10.00th=[ 9], 20.00th=[ 10], | 30.00th=[ 12], 40.00th=[ 13], 50.00th=[ 15], 60.00th=[ 18], | 70.00th=[ 22], 80.00th=[ 28], 90.00th=[ 38], 95.00th=[ 45], | 99.00th=[ 65], 99.50th=[ 75], 99.90th=[ 109], 99.95th=[ 518], | 99.99th=[ 518] bw ( KiB/s): min= 39, max= 288, per=24.88%, avg=124.40, stdev=67.34, samples=99 iops : min= 9, max= 72, avg=30.78, stdev=16.90, samples=99 lat (msec) : 4=3.19%, 10=37.01%, 20=33.92%, 50=23.21%, 100=2.57% lat (msec) : 250=0.07%, 750=0.03% cpu : usr=0.08%, sys=6.22%, ctx=3432, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=478KiB/s (490kB/s), 119KiB/s-121KiB/s (122kB/s-123kB/s), io=23.4MiB (24.6MB), run=49703-50195msec WRITE: bw=501KiB/s (513kB/s), 124KiB/s-127KiB/s (127kB/s-130kB/s), io=24.6MiB (25.7MB), run=49703-50195msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=26031: Thu May 1 01:58:20 2025 read: IOPS=38, BW=620KiB/s (635kB/s)(5920KiB/9552msec) clat (usec): min=76, max=77757, avg=13893.84, stdev=10518.27 lat (usec): min=77, max=77758, avg=13895.13, stdev=10518.33 clat percentiles (usec): | 1.00th=[ 96], 5.00th=[ 106], 10.00th=[ 130], 20.00th=[ 6652], | 30.00th=[ 8717], 40.00th=[10814], 50.00th=[12649], 60.00th=[14484], | 70.00th=[16909], 80.00th=[20841], 90.00th=[26084], 95.00th=[30278], | 99.00th=[50594], 99.50th=[72877], 99.90th=[78119], 99.95th=[78119], | 99.99th=[78119] bw ( KiB/s): min= 352, max= 736, per=24.60%, avg=592.94, stdev=111.69, samples=18 iops : min= 22, max= 46, avg=36.72, stdev= 6.83, samples=18 write: IOPS=41, BW=667KiB/s (683kB/s)(6368KiB/9552msec) clat (usec): min=931, max=66419, avg=10871.75, stdev=12047.17 lat (usec): min=933, max=66420, avg=10882.12, stdev=12042.77 clat percentiles (usec): | 1.00th=[ 1020], 5.00th=[ 1139], 10.00th=[ 1221], 20.00th=[ 1500], | 30.00th=[ 2442], 40.00th=[ 3687], 50.00th=[ 5997], 60.00th=[ 8979], | 70.00th=[12125], 80.00th=[19268], 90.00th=[29754], 95.00th=[35914], | 99.00th=[53216], 99.50th=[54789], 99.90th=[66323], 99.95th=[66323], | 99.99th=[66323] bw ( KiB/s): min= 384, max= 958, per=25.42%, avg=644.50, stdev=143.81, samples=18 iops : min= 24, max= 59, avg=39.94, stdev= 8.89, samples=18 lat (usec) : 100=1.43%, 250=5.08%, 500=0.26%, 750=0.26%, 1000=0.39% lat (msec) : 2=13.80%, 4=7.42%, 10=21.61%, 20=28.91%, 50=19.40% lat (msec) : 100=1.43% cpu : usr=0.17%, sys=12.74%, ctx=1244, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=26032: Thu May 1 01:58:20 2025 read: IOPS=38, BW=611KiB/s (625kB/s)(5744KiB/9407msec) clat (usec): min=74, max=86840, avg=14408.65, stdev=11316.38 lat (usec): min=75, max=86842, avg=14412.25, stdev=11317.69 clat percentiles (usec): | 1.00th=[ 93], 5.00th=[ 117], 10.00th=[ 151], 20.00th=[ 6521], | 30.00th=[ 9110], 40.00th=[10945], 50.00th=[12780], 60.00th=[14877], | 70.00th=[17695], 80.00th=[21365], 90.00th=[25560], 95.00th=[33162], | 99.00th=[57934], 99.50th=[81265], 99.90th=[86508], 99.95th=[86508], | 99.99th=[86508] bw ( KiB/s): min= 384, max= 832, per=24.89%, avg=599.83, stdev=138.51, samples=18 iops : min= 24, max= 52, avg=37.00, stdev= 8.62, samples=18 write: IOPS=43, BW=696KiB/s (712kB/s)(6544KiB/9407msec) clat (usec): min=798, max=61573, avg=10282.90, stdev=11172.19 lat (usec): min=799, max=61574, avg=10284.94, stdev=11172.27 clat percentiles (usec): | 1.00th=[ 914], 5.00th=[ 1090], 10.00th=[ 1188], 20.00th=[ 1385], | 30.00th=[ 1893], 40.00th=[ 3228], 50.00th=[ 5604], 60.00th=[ 9372], | 70.00th=[13435], 80.00th=[18482], 90.00th=[27395], 95.00th=[32900], | 99.00th=[44303], 99.50th=[51643], 99.90th=[61604], 99.95th=[61604], | 99.99th=[61604] bw ( KiB/s): min= 384, max= 1181, per=26.74%, avg=677.94, stdev=207.87, samples=18 iops : min= 24, max= 73, avg=41.89, stdev=12.97, samples=18 lat (usec) : 100=0.78%, 250=4.43%, 500=0.13%, 750=0.39%, 1000=1.43% lat (msec) : 2=15.49%, 4=7.81%, 10=18.75%, 20=31.12%, 50=18.49% lat (msec) : 100=1.17% cpu : usr=0.16%, sys=12.90%, ctx=1317, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=26033: Thu May 1 01:58:20 2025 read: IOPS=39, BW=630KiB/s (645kB/s)(6256KiB/9937msec) clat (usec): min=82, max=54335, avg=13947.44, stdev=10186.46 lat (usec): min=83, max=54337, avg=13949.85, stdev=10186.33 clat percentiles (usec): | 1.00th=[ 88], 5.00th=[ 111], 10.00th=[ 124], 20.00th=[ 6456], | 30.00th=[ 8979], 40.00th=[10814], 50.00th=[12387], 60.00th=[14222], | 70.00th=[17171], 80.00th=[21627], 90.00th=[27919], 95.00th=[33162], | 99.00th=[44303], 99.50th=[49546], 99.90th=[54264], 99.95th=[54264], | 99.99th=[54264] bw ( KiB/s): min= 415, max= 928, per=25.86%, avg=623.16, stdev=146.59, samples=19 iops : min= 25, max= 58, avg=38.58, stdev= 9.15, samples=19 write: IOPS=37, BW=607KiB/s (622kB/s)(6032KiB/9937msec) clat (usec): min=855, max=70862, avg=11739.26, stdev=12851.84 lat (usec): min=857, max=70863, avg=11741.41, stdev=12851.87 clat percentiles (usec): | 1.00th=[ 988], 5.00th=[ 1123], 10.00th=[ 1221], 20.00th=[ 1467], | 30.00th=[ 2606], 40.00th=[ 3785], 50.00th=[ 7373], 60.00th=[ 9896], | 70.00th=[14353], 80.00th=[20055], 90.00th=[31589], 95.00th=[39584], | 99.00th=[57934], 99.50th=[68682], 99.90th=[70779], 99.95th=[70779], | 99.99th=[70779] bw ( KiB/s): min= 319, max= 928, per=23.23%, avg=588.95, stdev=150.68, samples=19 iops : min= 19, max= 58, avg=36.47, stdev= 9.44, samples=19 lat (usec) : 100=1.43%, 250=6.51%, 500=0.13%, 1000=0.65% lat (msec) : 2=11.59%, 4=7.94%, 10=19.40%, 20=30.21%, 50=21.35% lat (msec) : 100=0.78% cpu : usr=0.14%, sys=12.37%, ctx=1314, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=26034: Thu May 1 01:58:20 2025 read: IOPS=40, BW=650KiB/s (666kB/s)(6032KiB/9274msec) clat (usec): min=83, max=69491, avg=13833.76, stdev=10875.79 lat (usec): min=83, max=69493, avg=13835.73, stdev=10876.05 clat percentiles (usec): | 1.00th=[ 92], 5.00th=[ 112], 10.00th=[ 125], 20.00th=[ 5997], | 30.00th=[ 8848], 40.00th=[10290], 50.00th=[11863], 60.00th=[13960], | 70.00th=[17171], 80.00th=[21365], 90.00th=[27657], 95.00th=[32900], | 99.00th=[51643], 99.50th=[57934], 99.90th=[69731], 99.95th=[69731], | 99.99th=[69731] bw ( KiB/s): min= 447, max= 896, per=26.50%, avg=638.56, stdev=133.08, samples=18 iops : min= 27, max= 56, avg=39.50, stdev= 8.43, samples=18 write: IOPS=42, BW=675KiB/s (691kB/s)(6256KiB/9274msec) clat (usec): min=870, max=67086, avg=10135.87, stdev=11320.16 lat (usec): min=872, max=67088, avg=10137.79, stdev=11320.16 clat percentiles (usec): | 1.00th=[ 1029], 5.00th=[ 1156], 10.00th=[ 1254], 20.00th=[ 1450], | 30.00th=[ 1958], 40.00th=[ 3228], 50.00th=[ 5473], 60.00th=[ 8291], | 70.00th=[11994], 80.00th=[17171], 90.00th=[26608], 95.00th=[32637], | 99.00th=[49021], 99.50th=[59507], 99.90th=[66847], 99.95th=[66847], | 99.99th=[66847] bw ( KiB/s): min= 412, max= 990, per=26.38%, avg=668.83, stdev=161.42, samples=18 iops : min= 25, max= 61, avg=41.39, stdev=10.18, samples=18 lat (usec) : 100=1.17%, 250=6.38%, 750=0.26%, 1000=0.26% lat (msec) : 2=15.89%, 4=7.16%, 10=20.44%, 20=28.26%, 50=19.27% lat (msec) : 100=0.91% cpu : usr=0.11%, sys=13.09%, ctx=1263, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2410KiB/s (2468kB/s), 611KiB/s-650KiB/s (625kB/s-666kB/s), io=23.4MiB (24.5MB), run=9274-9937msec WRITE: bw=2536KiB/s (2597kB/s), 607KiB/s-696KiB/s (622kB/s-712kB/s), io=24.6MiB (25.8MB), run=9274-9937msec rand-rw: (groupid=0, jobs=1): err= 0: pid=26036: Thu May 1 01:58:25 2025 read: IOPS=24, BW=386KiB/s (395kB/s)(5920KiB/15347msec) clat (msec): min=3, max=111, avg=14.95, stdev=12.81 lat (msec): min=3, max=111, avg=14.95, stdev=12.81 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 6], 10.00th=[ 7], 20.00th=[ 8], | 30.00th=[ 9], 40.00th=[ 10], 50.00th=[ 11], 60.00th=[ 12], | 70.00th=[ 14], 80.00th=[ 18], 90.00th=[ 30], 95.00th=[ 46], | 99.00th=[ 65], 99.50th=[ 75], 99.90th=[ 112], 99.95th=[ 112], | 99.99th=[ 112] bw ( KiB/s): min= 96, max= 800, per=24.49%, avg=374.70, stdev=155.50, samples=30 iops : min= 6, max= 50, avg=23.00, stdev= 9.83, samples=30 write: IOPS=25, BW=415KiB/s (425kB/s)(6368KiB/15347msec) clat (msec): min=5, max=100, avg=24.53, stdev=13.63 lat (msec): min=5, max=100, avg=24.53, stdev=13.63 clat percentiles (msec): | 1.00th=[ 8], 5.00th=[ 11], 10.00th=[ 13], 20.00th=[ 15], | 30.00th=[ 17], 40.00th=[ 19], 50.00th=[ 21], 60.00th=[ 23], | 70.00th=[ 27], 80.00th=[ 34], 90.00th=[ 45], 95.00th=[ 52], | 99.00th=[ 77], 99.50th=[ 83], 99.90th=[ 101], 99.95th=[ 101], | 99.99th=[ 101] bw ( KiB/s): min= 128, max= 736, per=25.52%, avg=410.90, stdev=168.61, samples=30 iops : min= 8, max= 46, avg=25.27, stdev=10.63, samples=30 lat (msec) : 4=0.39%, 10=23.31%, 20=41.02%, 50=31.12%, 100=3.91% lat (msec) : 250=0.26% cpu : usr=0.14%, sys=5.66%, ctx=857, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=26037: Thu May 1 01:58:25 2025 read: IOPS=22, BW=367KiB/s (376kB/s)(5744KiB/15647msec) clat (msec): min=3, max=108, avg=15.17, stdev=12.76 lat (msec): min=3, max=108, avg=15.17, stdev=12.76 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 6], 10.00th=[ 7], 20.00th=[ 8], | 30.00th=[ 9], 40.00th=[ 10], 50.00th=[ 11], 60.00th=[ 12], | 70.00th=[ 15], 80.00th=[ 20], 90.00th=[ 32], 95.00th=[ 45], | 99.00th=[ 61], 99.50th=[ 79], 99.90th=[ 109], 99.95th=[ 109], | 99.99th=[ 109] bw ( KiB/s): min= 64, max= 763, per=23.86%, avg=365.10, stdev=187.76, samples=31 iops : min= 4, max= 47, avg=22.45, stdev=11.85, samples=31 write: IOPS=26, BW=418KiB/s (428kB/s)(6544KiB/15647msec) clat (usec): min=6150, max=79271, avg=24544.34, stdev=12575.88 lat (usec): min=6153, max=79273, avg=24546.70, stdev=12575.86 clat percentiles (usec): | 1.00th=[ 8356], 5.00th=[11469], 10.00th=[12780], 20.00th=[14877], | 30.00th=[16909], 40.00th=[18744], 50.00th=[20841], 60.00th=[23200], | 70.00th=[26870], 80.00th=[32637], 90.00th=[42730], 95.00th=[50070], | 99.00th=[66847], 99.50th=[71828], 99.90th=[79168], 99.95th=[79168], | 99.99th=[79168] bw ( KiB/s): min= 159, max= 722, per=25.77%, avg=414.97, stdev=150.81, samples=31 iops : min= 9, max= 45, avg=25.58, stdev= 9.63, samples=31 lat (msec) : 4=0.26%, 10=20.44%, 20=41.67%, 50=33.59%, 100=3.91% lat (msec) : 250=0.13% cpu : usr=0.06%, sys=5.56%, ctx=877, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=26038: Thu May 1 01:58:25 2025 read: IOPS=25, BW=407KiB/s (416kB/s)(6256KiB/15388msec) clat (usec): min=3763, max=89959, avg=14991.06, stdev=12484.53 lat (usec): min=3765, max=89961, avg=14992.48, stdev=12484.46 clat percentiles (usec): | 1.00th=[ 4424], 5.00th=[ 5342], 10.00th=[ 6063], 20.00th=[ 7177], | 30.00th=[ 8455], 40.00th=[ 9634], 50.00th=[10814], 60.00th=[12256], | 70.00th=[14222], 80.00th=[19268], 90.00th=[31327], 95.00th=[44827], | 99.00th=[67634], 99.50th=[80217], 99.90th=[89654], 99.95th=[89654], | 99.99th=[89654] bw ( KiB/s): min= 95, max= 832, per=26.38%, avg=403.60, stdev=193.86, samples=30 iops : min= 5, max= 52, avg=24.90, stdev=12.13, samples=30 write: IOPS=24, BW=392KiB/s (401kB/s)(6032KiB/15388msec) clat (msec): min=7, max=105, avg=24.82, stdev=14.18 lat (msec): min=7, max=105, avg=24.83, stdev=14.18 clat percentiles (msec): | 1.00th=[ 9], 5.00th=[ 12], 10.00th=[ 13], 20.00th=[ 15], | 30.00th=[ 17], 40.00th=[ 19], 50.00th=[ 21], 60.00th=[ 24], | 70.00th=[ 27], 80.00th=[ 34], 90.00th=[ 44], 95.00th=[ 56], | 99.00th=[ 81], 99.50th=[ 84], 99.90th=[ 106], 99.95th=[ 106], | 99.99th=[ 106] bw ( KiB/s): min= 160, max= 800, per=23.54%, avg=379.03, stdev=147.96, samples=30 iops : min= 10, max= 50, avg=23.37, stdev= 9.25, samples=30 lat (msec) : 4=0.13%, 10=23.18%, 20=42.19%, 50=30.08%, 100=4.30% lat (msec) : 250=0.13% cpu : usr=0.02%, sys=5.73%, ctx=879, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=26039: Thu May 1 01:58:25 2025 read: IOPS=24, BW=390KiB/s (399kB/s)(6032KiB/15472msec) clat (msec): min=3, max=108, avg=16.22, stdev=13.44 lat (msec): min=3, max=108, avg=16.22, stdev=13.44 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 6], 10.00th=[ 7], 20.00th=[ 8], | 30.00th=[ 9], 40.00th=[ 11], 50.00th=[ 12], 60.00th=[ 13], | 70.00th=[ 16], 80.00th=[ 23], 90.00th=[ 34], 95.00th=[ 44], | 99.00th=[ 62], 99.50th=[ 104], 99.90th=[ 109], 99.95th=[ 109], | 99.99th=[ 109] bw ( KiB/s): min= 191, max= 670, per=24.77%, avg=379.00, stdev=130.64, samples=30 iops : min= 11, max= 41, avg=23.27, stdev= 8.21, samples=30 write: IOPS=25, BW=404KiB/s (414kB/s)(6256KiB/15472msec) clat (usec): min=6438, max=94821, avg=23612.56, stdev=12497.99 lat (usec): min=6440, max=94823, avg=23615.81, stdev=12497.06 clat percentiles (usec): | 1.00th=[ 7898], 5.00th=[10159], 10.00th=[11994], 20.00th=[14615], | 30.00th=[16319], 40.00th=[18220], 50.00th=[20055], 60.00th=[22414], | 70.00th=[25560], 80.00th=[31589], 90.00th=[39584], 95.00th=[47449], | 99.00th=[71828], 99.50th=[79168], 99.90th=[94897], 99.95th=[94897], | 99.99th=[94897] bw ( KiB/s): min= 96, max= 800, per=24.53%, avg=394.97, stdev=167.65, samples=30 iops : min= 6, max= 50, avg=24.27, stdev=10.60, samples=30 lat (msec) : 4=0.26%, 10=21.22%, 20=41.93%, 50=33.07%, 100=3.26% lat (msec) : 250=0.26% cpu : usr=0.08%, sys=5.71%, ctx=870, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1531KiB/s (1568kB/s), 367KiB/s-407KiB/s (376kB/s-416kB/s), io=23.4MiB (24.5MB), run=15347-15647msec WRITE: bw=1611KiB/s (1649kB/s), 392KiB/s-418KiB/s (401kB/s-428kB/s), io=24.6MiB (25.8MB), run=15347-15647msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=26059: Thu May 1 01:58:29 2025 read: IOPS=4, BW=5055KiB/s (5176kB/s)(7168KiB/1418msec) clat (usec): min=642, max=229938, avg=107003.52, stdev=87746.40 lat (usec): min=644, max=229940, avg=107005.26, stdev=87746.74 clat percentiles (usec): | 1.00th=[ 644], 5.00th=[ 644], 10.00th=[ 644], 20.00th=[ 15008], | 30.00th=[ 66323], 40.00th=[ 66323], 50.00th=[ 95945], 60.00th=[143655], | 70.00th=[143655], 80.00th=[198181], 90.00th=[229639], 95.00th=[229639], | 99.00th=[229639], 99.50th=[229639], 99.90th=[229639], 99.95th=[229639], | 99.99th=[229639] bw ( KiB/s): min= 4079, max= 6131, per=38.49%, avg=5105.00, stdev=1450.98, samples=2 iops : min= 3, max= 5, avg= 4.00, stdev= 1.41, samples=2 write: IOPS=3, BW=3611KiB/s (3697kB/s)(5120KiB/1418msec) clat (msec): min=37, max=178, avg=95.85, stdev=52.96 lat (msec): min=37, max=178, avg=95.91, stdev=52.99 clat percentiles (msec): | 1.00th=[ 37], 5.00th=[ 37], 10.00th=[ 37], 20.00th=[ 37], | 30.00th=[ 74], 40.00th=[ 74], 50.00th=[ 80], 60.00th=[ 80], | 70.00th=[ 111], 80.00th=[ 111], 90.00th=[ 180], 95.00th=[ 180], | 99.00th=[ 180], 99.50th=[ 180], 99.90th=[ 180], 99.95th=[ 180], | 99.99th=[ 180] bw ( KiB/s): min= 2043, max= 2043, per=11.00%, avg=2043.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (usec) : 750=8.33% lat (msec) : 20=8.33%, 50=8.33%, 100=33.33%, 250=41.67% cpu : usr=0.00%, sys=16.43%, ctx=240, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=26060: Thu May 1 01:58:29 2025 read: IOPS=1, BW=1326KiB/s (1358kB/s)(2048KiB/1544msec) clat (usec): min=80979, max=86473, avg=83726.57, stdev=3885.39 lat (usec): min=80981, max=86476, avg=83728.60, stdev=3885.57 clat percentiles (usec): | 1.00th=[81265], 5.00th=[81265], 10.00th=[81265], 20.00th=[81265], | 30.00th=[81265], 40.00th=[81265], 50.00th=[81265], 60.00th=[86508], | 70.00th=[86508], 80.00th=[86508], 90.00th=[86508], 95.00th=[86508], | 99.00th=[86508], 99.50th=[86508], 99.90th=[86508], 99.95th=[86508], | 99.99th=[86508] bw ( KiB/s): min= 2039, max= 2048, per=15.41%, avg=2043.50, stdev= 6.36, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 write: IOPS=6, BW=6632KiB/s (6791kB/s)(10.0MiB/1544msec) clat (msec): min=33, max=335, avg=112.95, stdev=86.68 lat (msec): min=33, max=335, avg=113.08, stdev=86.69 clat percentiles (msec): | 1.00th=[ 34], 5.00th=[ 34], 10.00th=[ 34], 20.00th=[ 46], | 30.00th=[ 56], 40.00th=[ 85], 50.00th=[ 92], 60.00th=[ 103], | 70.00th=[ 112], 80.00th=[ 112], 90.00th=[ 165], 95.00th=[ 334], | 99.00th=[ 334], 99.50th=[ 334], 99.90th=[ 334], 99.95th=[ 334], | 99.99th=[ 334] bw ( KiB/s): min= 2039, max=10240, per=32.94%, avg=6116.67, stdev=4100.69, samples=3 iops : min= 1, max= 10, avg= 5.33, stdev= 4.51, samples=3 lat (msec) : 50=16.67%, 100=41.67%, 250=33.33%, 500=8.33% cpu : usr=0.00%, sys=19.44%, ctx=126, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=26061: Thu May 1 01:58:29 2025 read: IOPS=4, BW=5107KiB/s (5230kB/s)(6144KiB/1203msec) clat (usec): min=864, max=97284, avg=69764.42, stdev=35111.33 lat (usec): min=866, max=97285, avg=69765.91, stdev=35111.17 clat percentiles (usec): | 1.00th=[ 865], 5.00th=[ 865], 10.00th=[ 865], 20.00th=[71828], | 30.00th=[71828], 40.00th=[74974], 50.00th=[74974], 60.00th=[80217], | 70.00th=[91751], 80.00th=[91751], 90.00th=[96994], 95.00th=[96994], | 99.00th=[96994], 99.50th=[96994], 99.90th=[96994], 99.95th=[96994], | 99.99th=[96994] bw ( KiB/s): min= 2043, max= 6107, per=30.72%, avg=4075.00, stdev=2873.68, samples=2 iops : min= 1, max= 5, avg= 3.00, stdev= 2.83, samples=2 write: IOPS=4, BW=5107KiB/s (5230kB/s)(6144KiB/1203msec) clat (msec): min=69, max=257, avg=116.06, stdev=75.26 lat (msec): min=69, max=257, avg=116.15, stdev=75.28 clat percentiles (msec): | 1.00th=[ 69], 5.00th=[ 69], 10.00th=[ 69], 20.00th=[ 71], | 30.00th=[ 71], 40.00th=[ 77], 50.00th=[ 77], 60.00th=[ 78], | 70.00th=[ 146], 80.00th=[ 146], 90.00th=[ 257], 95.00th=[ 257], | 99.00th=[ 257], 99.50th=[ 257], 99.90th=[ 257], 99.95th=[ 257], | 99.99th=[ 257] bw ( KiB/s): min= 2035, max= 6131, per=21.99%, avg=4083.00, stdev=2896.31, samples=2 iops : min= 1, max= 5, avg= 3.00, stdev= 2.83, samples=2 lat (usec) : 1000=8.33% lat (msec) : 100=75.00%, 250=8.33%, 500=8.33% cpu : usr=0.00%, sys=17.89%, ctx=277, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=26062: Thu May 1 01:58:29 2025 read: IOPS=3, BW=3908KiB/s (4002kB/s)(5120KiB/1310msec) clat (usec): min=440, max=143861, avg=47961.51, stdev=66886.93 lat (usec): min=442, max=143862, avg=47963.03, stdev=66886.81 clat percentiles (usec): | 1.00th=[ 441], 5.00th=[ 441], 10.00th=[ 441], 20.00th=[ 441], | 30.00th=[ 502], 40.00th=[ 502], 50.00th=[ 1745], 60.00th=[ 1745], | 70.00th=[ 92799], 80.00th=[ 92799], 90.00th=[143655], 95.00th=[143655], | 99.00th=[143655], 99.50th=[143655], 99.90th=[143655], 99.95th=[143655], | 99.99th=[143655] bw ( KiB/s): min= 8175, max= 8175, per=61.63%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 7, max= 7, avg= 7.00, stdev= 0.00, samples=1 write: IOPS=5, BW=5472KiB/s (5603kB/s)(7168KiB/1310msec) clat (msec): min=52, max=348, avg=144.36, stdev=113.16 lat (msec): min=52, max=348, avg=144.44, stdev=113.17 clat percentiles (msec): | 1.00th=[ 53], 5.00th=[ 53], 10.00th=[ 53], 20.00th=[ 73], | 30.00th=[ 77], 40.00th=[ 77], 50.00th=[ 89], 60.00th=[ 113], | 70.00th=[ 113], 80.00th=[ 259], 90.00th=[ 351], 95.00th=[ 351], | 99.00th=[ 351], 99.50th=[ 351], 99.90th=[ 351], 99.95th=[ 351], | 99.99th=[ 351] bw ( KiB/s): min= 6119, max= 6131, per=32.99%, avg=6125.00, stdev= 8.49, samples=2 iops : min= 5, max= 5, avg= 5.00, stdev= 0.00, samples=2 lat (usec) : 500=8.33%, 750=8.33% lat (msec) : 2=8.33%, 100=41.67%, 250=16.67%, 500=16.67% cpu : usr=0.08%, sys=13.83%, ctx=124, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=12.0MiB/s (13.6MB/s), 1326KiB/s-5107KiB/s (1358kB/s-5230kB/s), io=20.0MiB (20.0MB), run=1203-1544msec WRITE: bw=18.1MiB/s (19.0MB/s), 3611KiB/s-6632KiB/s (3697kB/s-6791kB/s), io=28.0MiB (29.4MB), run=1203-1544msec rand-rw: (groupid=0, jobs=1): err= 0: pid=26063: Thu May 1 01:58:29 2025 read: IOPS=5, BW=6049KiB/s (6194kB/s)(7168KiB/1185msec) clat (msec): min=38, max=171, avg=89.04, stdev=50.34 lat (msec): min=38, max=171, avg=89.05, stdev=50.34 clat percentiles (msec): | 1.00th=[ 39], 5.00th=[ 39], 10.00th=[ 39], 20.00th=[ 53], | 30.00th=[ 64], 40.00th=[ 64], 50.00th=[ 67], 60.00th=[ 85], | 70.00th=[ 85], 80.00th=[ 146], 90.00th=[ 171], 95.00th=[ 171], | 99.00th=[ 171], 99.50th=[ 171], 99.90th=[ 171], 99.95th=[ 171], | 99.99th=[ 171] bw ( KiB/s): min= 3984, max= 6131, per=44.11%, avg=5057.50, stdev=1518.16, samples=2 iops : min= 3, max= 5, avg= 4.00, stdev= 1.41, samples=2 write: IOPS=4, BW=4321KiB/s (4424kB/s)(5120KiB/1185msec) clat (msec): min=42, max=175, avg=94.01, stdev=51.67 lat (msec): min=42, max=175, avg=94.09, stdev=51.68 clat percentiles (msec): | 1.00th=[ 43], 5.00th=[ 43], 10.00th=[ 43], 20.00th=[ 43], | 30.00th=[ 59], 40.00th=[ 59], 50.00th=[ 94], 60.00th=[ 94], | 70.00th=[ 101], 80.00th=[ 101], 90.00th=[ 176], 95.00th=[ 176], | 99.00th=[ 176], 99.50th=[ 176], 99.90th=[ 176], 99.95th=[ 176], | 99.99th=[ 176] bw ( KiB/s): min= 5976, max= 5976, per=37.23%, avg=5976.00, stdev= 0.00, samples=1 iops : min= 5, max= 5, avg= 5.00, stdev= 0.00, samples=1 lat (msec) : 50=16.67%, 100=58.33%, 250=25.00% cpu : usr=0.08%, sys=7.09%, ctx=41, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=26064: Thu May 1 01:58:29 2025 read: IOPS=1, BW=1147KiB/s (1174kB/s)(2048KiB/1786msec) clat (msec): min=134, max=208, avg=171.70, stdev=51.96 lat (msec): min=134, max=208, avg=171.71, stdev=51.96 clat percentiles (msec): | 1.00th=[ 136], 5.00th=[ 136], 10.00th=[ 136], 20.00th=[ 136], | 30.00th=[ 136], 40.00th=[ 136], 50.00th=[ 136], 60.00th=[ 209], | 70.00th=[ 209], 80.00th=[ 209], 90.00th=[ 209], 95.00th=[ 209], | 99.00th=[ 209], 99.50th=[ 209], 99.90th=[ 209], 99.95th=[ 209], | 99.99th=[ 209] bw ( KiB/s): min= 2048, max= 2048, per=17.86%, avg=2048.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 write: IOPS=5, BW=5733KiB/s (5871kB/s)(10.0MiB/1786msec) clat (msec): min=44, max=248, avg=116.66, stdev=67.55 lat (msec): min=44, max=248, avg=117.04, stdev=67.25 clat percentiles (msec): | 1.00th=[ 45], 5.00th=[ 45], 10.00th=[ 45], 20.00th=[ 55], | 30.00th=[ 57], 40.00th=[ 64], 50.00th=[ 75], 60.00th=[ 148], | 70.00th=[ 161], 80.00th=[ 161], 90.00th=[ 163], 95.00th=[ 249], | 99.00th=[ 249], 99.50th=[ 249], 99.90th=[ 249], 99.95th=[ 249], | 99.99th=[ 249] bw ( KiB/s): min= 4096, max=10240, per=44.65%, avg=7168.00, stdev=4344.46, samples=2 iops : min= 4, max= 10, avg= 7.00, stdev= 4.24, samples=2 lat (msec) : 50=8.33%, 100=33.33%, 250=58.33% cpu : usr=0.11%, sys=9.30%, ctx=89, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=26065: Thu May 1 01:58:29 2025 read: IOPS=3, BW=3690KiB/s (3779kB/s)(6144KiB/1665msec) clat (msec): min=29, max=128, avg=61.41, stdev=38.04 lat (msec): min=29, max=128, avg=61.41, stdev=38.04 clat percentiles (msec): | 1.00th=[ 30], 5.00th=[ 30], 10.00th=[ 30], 20.00th=[ 32], | 30.00th=[ 32], 40.00th=[ 39], 50.00th=[ 39], 60.00th=[ 62], | 70.00th=[ 79], 80.00th=[ 79], 90.00th=[ 129], 95.00th=[ 129], | 99.00th=[ 129], 99.50th=[ 129], 99.90th=[ 129], 99.95th=[ 129], | 99.99th=[ 129] bw ( KiB/s): min=10219, max=10219, per=89.12%, avg=10219.00, stdev= 0.00, samples=1 iops : min= 9, max= 9, avg= 9.00, stdev= 0.00, samples=1 write: IOPS=3, BW=3690KiB/s (3779kB/s)(6144KiB/1665msec) clat (msec): min=45, max=384, avg=170.09, stdev=121.64 lat (msec): min=45, max=384, avg=170.16, stdev=121.64 clat percentiles (msec): | 1.00th=[ 46], 5.00th=[ 46], 10.00th=[ 46], 20.00th=[ 86], | 30.00th=[ 86], 40.00th=[ 138], 50.00th=[ 138], 60.00th=[ 140], | 70.00th=[ 230], 80.00th=[ 230], 90.00th=[ 384], 95.00th=[ 384], | 99.00th=[ 384], 99.50th=[ 384], 99.90th=[ 384], 99.95th=[ 384], | 99.99th=[ 384] bw ( KiB/s): min= 4071, max= 6131, per=31.78%, avg=5101.00, stdev=1456.64, samples=2 iops : min= 3, max= 5, avg= 4.00, stdev= 1.41, samples=2 lat (msec) : 50=33.33%, 100=25.00%, 250=33.33%, 500=8.33% cpu : usr=0.12%, sys=7.87%, ctx=82, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=26066: Thu May 1 01:58:29 2025 read: IOPS=2, BW=3062KiB/s (3136kB/s)(5120KiB/1672msec) clat (msec): min=26, max=180, avg=75.09, stdev=66.75 lat (msec): min=26, max=180, avg=75.09, stdev=66.75 clat percentiles (msec): | 1.00th=[ 27], 5.00th=[ 27], 10.00th=[ 27], 20.00th=[ 27], | 30.00th=[ 29], 40.00th=[ 29], 50.00th=[ 38], 60.00th=[ 38], | 70.00th=[ 104], 80.00th=[ 104], 90.00th=[ 182], 95.00th=[ 182], | 99.00th=[ 182], 99.50th=[ 182], 99.90th=[ 182], 99.95th=[ 182], | 99.99th=[ 182] bw ( KiB/s): min= 8159, max= 8159, per=71.16%, avg=8159.00, stdev= 0.00, samples=1 iops : min= 7, max= 7, avg= 7.00, stdev= 0.00, samples=1 write: IOPS=4, BW=4287KiB/s (4390kB/s)(7168KiB/1672msec) clat (msec): min=44, max=324, avg=149.11, stdev=93.76 lat (msec): min=44, max=324, avg=149.30, stdev=93.66 clat percentiles (msec): | 1.00th=[ 45], 5.00th=[ 45], 10.00th=[ 45], 20.00th=[ 68], | 30.00th=[ 124], 40.00th=[ 124], 50.00th=[ 138], 60.00th=[ 138], | 70.00th=[ 138], 80.00th=[ 207], 90.00th=[ 326], 95.00th=[ 326], | 99.00th=[ 326], 99.50th=[ 326], 99.90th=[ 326], 99.95th=[ 326], | 99.99th=[ 326] bw ( KiB/s): min= 2023, max= 6119, per=25.39%, avg=4076.33, stdev=2048.02, samples=3 iops : min= 1, max= 5, avg= 3.00, stdev= 2.00, samples=3 lat (msec) : 50=33.33%, 100=8.33%, 250=50.00%, 500=8.33% cpu : usr=0.00%, sys=7.00%, ctx=70, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=11.2MiB/s (11.7MB/s), 1147KiB/s-6049KiB/s (1174kB/s-6194kB/s), io=20.0MiB (20.0MB), run=1185-1786msec WRITE: bw=15.7MiB/s (16.4MB/s), 3690KiB/s-5733KiB/s (3779kB/s-5871kB/s), io=28.0MiB (29.4MB), run=1185-1786msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=26074: Thu May 1 01:58:33 2025 read: IOPS=5, BW=22.3MiB/s (23.4MB/s)(12.0MiB/538msec) clat (msec): min=9, max=291, avg=171.56, stdev=145.40 lat (msec): min=9, max=291, avg=171.56, stdev=145.40 clat percentiles (msec): | 1.00th=[ 10], 5.00th=[ 10], 10.00th=[ 10], 20.00th=[ 10], | 30.00th=[ 10], 40.00th=[ 213], 50.00th=[ 213], 60.00th=[ 213], | 70.00th=[ 292], 80.00th=[ 292], 90.00th=[ 292], 95.00th=[ 292], | 99.00th=[ 292], 99.50th=[ 292], 99.90th=[ 292], 99.95th=[ 292], | 99.99th=[ 292] bw ( KiB/s): min= 8159, max= 8159, per=100.00%, avg=8159.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 10=33.33%, 250=33.33%, 500=33.33% cpu : usr=0.56%, sys=24.21%, ctx=657, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=26075: Thu May 1 01:58:33 2025 read: IOPS=0, BW=1578KiB/s (1616kB/s)(4096KiB/2595msec) clat (nsec): min=1525.3M, max=1525.3M, avg=1525292616.00, stdev= 0.00 lat (nsec): min=1525.3M, max=1525.3M, avg=1525295376.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 1519], 5.00th=[ 1519], 10.00th=[ 1519], 20.00th=[ 1519], | 30.00th=[ 1519], 40.00th=[ 1519], 50.00th=[ 1519], 60.00th=[ 1519], | 70.00th=[ 1519], 80.00th=[ 1519], 90.00th=[ 1519], 95.00th=[ 1519], | 99.00th=[ 1519], 99.50th=[ 1519], 99.90th=[ 1519], 99.95th=[ 1519], | 99.99th=[ 1519] bw ( KiB/s): min= 8175, max= 8175, per=100.00%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=0, BW=3157KiB/s (3233kB/s)(8192KiB/2595msec) clat (msec): min=402, max=635, avg=519.13, stdev=165.20 lat (msec): min=402, max=636, avg=519.60, stdev=165.02 clat percentiles (msec): | 1.00th=[ 401], 5.00th=[ 401], 10.00th=[ 401], 20.00th=[ 401], | 30.00th=[ 401], 40.00th=[ 401], 50.00th=[ 401], 60.00th=[ 634], | 70.00th=[ 634], 80.00th=[ 634], 90.00th=[ 634], 95.00th=[ 634], | 99.00th=[ 634], 99.50th=[ 634], 99.90th=[ 634], 99.95th=[ 634], | 99.99th=[ 634] bw ( KiB/s): min= 8175, max= 8175, per=64.74%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 500=33.33%, 750=33.33% cpu : usr=0.15%, sys=13.18%, ctx=289, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=26078: Thu May 1 01:58:33 2025 write: IOPS=1, BW=4882KiB/s (4999kB/s)(12.0MiB/2517msec) clat (msec): min=325, max=1716, avg=831.17, stdev=769.57 lat (msec): min=325, max=1717, avg=832.16, stdev=769.29 clat percentiles (msec): | 1.00th=[ 326], 5.00th=[ 326], 10.00th=[ 326], 20.00th=[ 326], | 30.00th=[ 326], 40.00th=[ 451], 50.00th=[ 451], 60.00th=[ 451], | 70.00th=[ 1720], 80.00th=[ 1720], 90.00th=[ 1720], 95.00th=[ 1720], | 99.00th=[ 1720], 99.50th=[ 1720], 99.90th=[ 1720], 99.95th=[ 1720], | 99.99th=[ 1720] bw ( KiB/s): min= 8175, max= 8175, per=64.74%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 500=66.67% cpu : usr=0.00%, sys=13.59%, ctx=67, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=26080: Thu May 1 01:58:33 2025 write: IOPS=1, BW=5508KiB/s (5640kB/s)(12.0MiB/2231msec) clat (msec): min=213, max=1647, avg=735.95, stdev=792.15 lat (msec): min=213, max=1647, avg=736.41, stdev=792.06 clat percentiles (msec): | 1.00th=[ 213], 5.00th=[ 213], 10.00th=[ 213], 20.00th=[ 213], | 30.00th=[ 213], 40.00th=[ 347], 50.00th=[ 347], 60.00th=[ 347], | 70.00th=[ 1653], 80.00th=[ 1653], 90.00th=[ 1653], 95.00th=[ 1653], | 99.00th=[ 1653], 99.50th=[ 1653], 99.90th=[ 1653], 99.95th=[ 1653], | 99.99th=[ 1653] bw ( KiB/s): min= 7846, max= 8192, per=63.51%, avg=8019.00, stdev=244.66, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 250=33.33%, 500=33.33% cpu : usr=0.00%, sys=11.52%, ctx=68, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=6314KiB/s (6465kB/s), 1578KiB/s-22.3MiB/s (1616kB/s-23.4MB/s), io=16.0MiB (16.8MB), run=538-2595msec WRITE: bw=12.3MiB/s (12.9MB/s), 3157KiB/s-5508KiB/s (3233kB/s-5640kB/s), io=32.0MiB (33.6MB), run=2231-2595msec rand-rw: (groupid=0, jobs=1): err= 0: pid=26076: Thu May 1 01:58:33 2025 read: IOPS=0, BW=4088KiB/s (4186kB/s)(12.0MiB/3006msec) clat (msec): min=401, max=1792, avg=971.11, stdev=728.78 lat (msec): min=401, max=1792, avg=971.11, stdev=728.78 clat percentiles (msec): | 1.00th=[ 401], 5.00th=[ 401], 10.00th=[ 401], 20.00th=[ 401], | 30.00th=[ 401], 40.00th=[ 718], 50.00th=[ 718], 60.00th=[ 718], | 70.00th=[ 1787], 80.00th=[ 1787], 90.00th=[ 1787], 95.00th=[ 1787], | 99.00th=[ 1787], 99.50th=[ 1787], 99.90th=[ 1787], 99.95th=[ 1787], | 99.99th=[ 1787] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 lat (msec) : 500=33.33%, 750=33.33% cpu : usr=0.00%, sys=2.86%, ctx=41, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=26077: Thu May 1 01:58:33 2025 read: IOPS=0, BW=1331KiB/s (1363kB/s)(4096KiB/3078msec) clat (nsec): min=1837.5M, max=1837.5M, avg=1837476766.00, stdev= 0.00 lat (nsec): min=1837.5M, max=1837.5M, avg=1837479958.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 1838], 5.00th=[ 1838], 10.00th=[ 1838], 20.00th=[ 1838], | 30.00th=[ 1838], 40.00th=[ 1838], 50.00th=[ 1838], 60.00th=[ 1838], | 70.00th=[ 1838], 80.00th=[ 1838], 90.00th=[ 1838], 95.00th=[ 1838], | 99.00th=[ 1838], 99.50th=[ 1838], 99.90th=[ 1838], 99.95th=[ 1838], | 99.99th=[ 1838] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=0, BW=2661KiB/s (2725kB/s)(8192KiB/3078msec) clat (msec): min=170, max=940, avg=555.80, stdev=544.40 lat (msec): min=171, max=941, avg=556.27, stdev=544.45 clat percentiles (msec): | 1.00th=[ 171], 5.00th=[ 171], 10.00th=[ 171], 20.00th=[ 171], | 30.00th=[ 171], 40.00th=[ 171], 50.00th=[ 171], 60.00th=[ 944], | 70.00th=[ 944], 80.00th=[ 944], 90.00th=[ 944], 95.00th=[ 944], | 99.00th=[ 944], 99.50th=[ 944], 99.90th=[ 944], 99.95th=[ 944], | 99.99th=[ 944] bw ( KiB/s): min= 8175, max= 8175, per=77.86%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 1000=33.33% cpu : usr=0.03%, sys=4.65%, ctx=83, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=26079: Thu May 1 01:58:33 2025 write: IOPS=0, BW=3937KiB/s (4032kB/s)(12.0MiB/3121msec) clat (msec): min=201, max=2280, avg=1029.01, stdev=1102.17 lat (msec): min=202, max=2280, avg=1029.38, stdev=1102.33 clat percentiles (msec): | 1.00th=[ 203], 5.00th=[ 203], 10.00th=[ 203], 20.00th=[ 203], | 30.00th=[ 203], 40.00th=[ 609], 50.00th=[ 609], 60.00th=[ 609], | 70.00th=[ 2265], 80.00th=[ 2265], 90.00th=[ 2265], 95.00th=[ 2265], | 99.00th=[ 2265], 99.50th=[ 2265], 99.90th=[ 2265], 99.95th=[ 2265], | 99.99th=[ 2265] bw ( KiB/s): min= 8047, max= 8192, per=77.34%, avg=8119.50, stdev=102.53, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 250=33.33%, 750=33.33% cpu : usr=0.00%, sys=3.04%, ctx=42, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=26081: Thu May 1 01:58:33 2025 write: IOPS=0, BW=3997KiB/s (4093kB/s)(12.0MiB/3074msec) clat (msec): min=103, max=1965, avg=988.63, stdev=934.71 lat (msec): min=103, max=1966, avg=988.99, stdev=934.72 clat percentiles (msec): | 1.00th=[ 104], 5.00th=[ 104], 10.00th=[ 104], 20.00th=[ 104], | 30.00th=[ 104], 40.00th=[ 894], 50.00th=[ 894], 60.00th=[ 894], | 70.00th=[ 1972], 80.00th=[ 1972], 90.00th=[ 1972], 95.00th=[ 1972], | 99.00th=[ 1972], 99.50th=[ 1972], 99.90th=[ 1972], 99.95th=[ 1972], | 99.99th=[ 1972] bw ( KiB/s): min= 8192, max= 8192, per=78.03%, avg=8192.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 lat (msec) : 250=33.33%, 1000=33.33% cpu : usr=0.03%, sys=6.25%, ctx=85, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=5323KiB/s (5451kB/s), 1331KiB/s-4088KiB/s (1363kB/s-4186kB/s), io=16.0MiB (16.8MB), run=3006-3078msec WRITE: bw=10.3MiB/s (10.8MB/s), 2661KiB/s-3997KiB/s (2725kB/s-4093kB/s), io=32.0MiB (33.6MB), run=3074-3121msec