== sanity test 398b: DIO and buffer IO race ============== 00:57:23 (1746161843) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 3.36469 s, 15.0 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=22031: Fri May 2 00:58:31 2025 read: IOPS=24, BW=98.7KiB/s (101kB/s)(5992KiB/60685msec) clat (usec): min=79, max=523374, avg=23343.82, stdev=20882.98 lat (usec): min=79, max=523377, avg=23350.70, stdev=20885.44 clat percentiles (usec): | 1.00th=[ 100], 5.00th=[ 139], 10.00th=[ 7242], 20.00th=[ 11338], | 30.00th=[ 13960], 40.00th=[ 16712], 50.00th=[ 20055], 60.00th=[ 23462], | 70.00th=[ 27657], 80.00th=[ 33817], 90.00th=[ 41681], 95.00th=[ 50594], | 99.00th=[ 85459], 99.50th=[102237], 99.90th=[179307], 99.95th=[522191], | 99.99th=[522191] bw ( KiB/s): min= 40, max= 192, per=25.26%, avg=98.52, stdev=25.89, samples=120 iops : min= 10, max= 48, avg=24.24, stdev= 6.53, samples=120 write: IOPS=25, BW=104KiB/s (106kB/s)(6296KiB/60685msec) clat (usec): min=497, max=167179, avg=16201.11, stdev=18112.96 lat (usec): min=497, max=167182, avg=16204.00, stdev=18116.39 clat percentiles (usec): | 1.00th=[ 750], 5.00th=[ 840], 10.00th=[ 947], 20.00th=[ 1287], | 30.00th=[ 2409], 40.00th=[ 4686], 50.00th=[ 10421], 60.00th=[ 15270], | 70.00th=[ 21103], 80.00th=[ 28967], 90.00th=[ 41157], 95.00th=[ 51643], | 99.00th=[ 77071], 99.50th=[ 86508], 99.90th=[106431], 99.95th=[166724], | 99.99th=[166724] bw ( KiB/s): min= 32, max= 192, per=25.36%, avg=103.48, stdev=33.45, samples=120 iops : min= 8, max= 48, avg=25.49, stdev= 8.34, samples=120 lat (usec) : 100=0.49%, 250=2.70%, 500=0.42%, 750=0.52%, 1000=5.63% lat (msec) : 2=8.11%, 4=5.44%, 10=9.67%, 20=25.85%, 50=35.77% lat (msec) : 100=4.95%, 250=0.42%, 750=0.03% cpu : usr=0.13%, sys=9.29%, ctx=5178, majf=0, minf=37 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22032: Fri May 2 00:58:31 2025 read: IOPS=24, BW=98.2KiB/s (101kB/s)(5940KiB/60466msec) clat (usec): min=80, max=158249, avg=22564.12, stdev=15727.54 lat (usec): min=80, max=158249, avg=22568.21, stdev=15729.07 clat percentiles (usec): | 1.00th=[ 100], 5.00th=[ 139], 10.00th=[ 7701], 20.00th=[ 11207], | 30.00th=[ 13960], 40.00th=[ 16581], 50.00th=[ 19530], 60.00th=[ 23200], | 70.00th=[ 27395], 80.00th=[ 32637], 90.00th=[ 41157], 95.00th=[ 49021], | 99.00th=[ 78119], 99.50th=[ 94897], 99.90th=[121111], 99.95th=[158335], | 99.99th=[158335] bw ( KiB/s): min= 8, max= 176, per=24.87%, avg=96.98, stdev=26.94, samples=120 iops : min= 2, max= 44, avg=23.76, stdev= 6.77, samples=120 write: IOPS=26, BW=105KiB/s (108kB/s)(6348KiB/60466msec) clat (usec): min=459, max=439987, avg=16898.79, stdev=20869.92 lat (usec): min=460, max=439988, avg=16900.95, stdev=20869.92 clat percentiles (usec): | 1.00th=[ 734], 5.00th=[ 848], 10.00th=[ 988], 20.00th=[ 1369], | 30.00th=[ 2900], 40.00th=[ 5735], 50.00th=[ 11076], 60.00th=[ 16450], | 70.00th=[ 21890], 80.00th=[ 29230], 90.00th=[ 41681], 95.00th=[ 51119], | 99.00th=[ 74974], 99.50th=[ 99091], 99.90th=[125305], 99.95th=[438305], | 99.99th=[438305] bw ( KiB/s): min= 8, max= 223, per=25.29%, avg=103.20, stdev=36.76, samples=120 iops : min= 2, max= 55, avg=25.32, stdev= 9.20, samples=120 lat (usec) : 100=0.46%, 250=2.57%, 500=0.52%, 750=0.68%, 1000=4.98% lat (msec) : 2=7.65%, 4=5.34%, 10=10.06%, 20=27.38%, 50=35.22% lat (msec) : 100=4.82%, 250=0.29%, 500=0.03% cpu : usr=0.15%, sys=9.42%, ctx=5186, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22033: Fri May 2 00:58:31 2025 read: IOPS=24, BW=98.0KiB/s (100kB/s)(6028KiB/61508msec) clat (usec): min=68, max=119109, avg=22171.58, stdev=14807.10 lat (usec): min=68, max=119110, avg=22172.88, stdev=14807.12 clat percentiles (usec): | 1.00th=[ 100], 5.00th=[ 133], 10.00th=[ 7570], 20.00th=[ 11469], | 30.00th=[ 14222], 40.00th=[ 16909], 50.00th=[ 19792], 60.00th=[ 22938], | 70.00th=[ 27132], 80.00th=[ 32113], 90.00th=[ 40109], 95.00th=[ 46924], | 99.00th=[ 72877], 99.50th=[ 93848], 99.90th=[109577], 99.95th=[119014], | 99.99th=[119014] bw ( KiB/s): min= 39, max= 208, per=24.93%, avg=97.21, stdev=29.84, samples=121 iops : min= 9, max= 52, avg=23.88, stdev= 7.49, samples=121 write: IOPS=25, BW=102KiB/s (104kB/s)(6260KiB/61508msec) clat (usec): min=531, max=393949, avg=17784.80, stdev=20895.62 lat (usec): min=532, max=393952, avg=17786.36, stdev=20895.50 clat percentiles (usec): | 1.00th=[ 750], 5.00th=[ 857], 10.00th=[ 1020], 20.00th=[ 1532], | 30.00th=[ 3359], 40.00th=[ 6521], 50.00th=[ 12387], 60.00th=[ 16712], | 70.00th=[ 22414], 80.00th=[ 31065], 90.00th=[ 42730], 95.00th=[ 55313], | 99.00th=[ 80217], 99.50th=[ 93848], 99.90th=[120062], 99.95th=[392168], | 99.99th=[392168] bw ( KiB/s): min= 31, max= 248, per=24.71%, avg=100.83, stdev=34.20, samples=121 iops : min= 7, max= 62, avg=24.77, stdev= 8.57, samples=121 lat (usec) : 100=0.49%, 250=2.90%, 500=0.42%, 750=0.62%, 1000=4.20% lat (msec) : 2=7.32%, 4=5.18%, 10=9.60%, 20=27.44%, 50=36.52% lat (msec) : 100=5.01%, 250=0.26%, 500=0.03% cpu : usr=0.13%, sys=9.37%, ctx=5257, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22034: Fri May 2 00:58:31 2025 read: IOPS=24, BW=98.0KiB/s (101kB/s)(6052KiB/61133msec) clat (usec): min=73, max=166380, avg=22227.41, stdev=15521.80 lat (usec): min=73, max=166381, avg=22228.69, stdev=15521.95 clat percentiles (usec): | 1.00th=[ 102], 5.00th=[ 145], 10.00th=[ 7570], 20.00th=[ 11207], | 30.00th=[ 13829], 40.00th=[ 16450], 50.00th=[ 19530], 60.00th=[ 22676], | 70.00th=[ 27132], 80.00th=[ 32113], 90.00th=[ 39584], 95.00th=[ 47973], | 99.00th=[ 78119], 99.50th=[ 98042], 99.90th=[132645], 99.95th=[166724], | 99.99th=[166724] bw ( KiB/s): min= 40, max= 216, per=25.34%, avg=98.82, stdev=31.45, samples=121 iops : min= 10, max= 54, avg=24.36, stdev= 7.86, samples=121 write: IOPS=25, BW=102KiB/s (104kB/s)(6236KiB/61133msec) clat (usec): min=599, max=366532, avg=17454.08, stdev=21151.05 lat (usec): min=601, max=366535, avg=17458.21, stdev=21150.64 clat percentiles (usec): | 1.00th=[ 766], 5.00th=[ 865], 10.00th=[ 1004], 20.00th=[ 1352], | 30.00th=[ 2737], 40.00th=[ 5604], 50.00th=[ 11731], 60.00th=[ 16712], | 70.00th=[ 22152], 80.00th=[ 31327], 90.00th=[ 43254], 95.00th=[ 53216], | 99.00th=[ 80217], 99.50th=[109577], 99.90th=[162530], 99.95th=[367002], | 99.99th=[367002] bw ( KiB/s): min= 39, max= 232, per=24.94%, avg=101.77, stdev=35.50, samples=121 iops : min= 9, max= 58, avg=25.10, stdev= 8.95, samples=121 lat (usec) : 100=0.39%, 250=2.47%, 500=0.42%, 750=0.42%, 1000=4.62% lat (msec) : 2=8.01%, 4=5.47%, 10=9.70%, 20=27.93%, 50=35.12% lat (msec) : 100=4.95%, 250=0.46%, 500=0.03% cpu : usr=0.13%, sys=9.26%, ctx=5195, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=390KiB/s (400kB/s), 98.0KiB/s-98.0KiB/s (100kB/s-101kB/s), io=23.4MiB (24.6MB), run=60466-61508msec WRITE: bw=409KiB/s (419kB/s), 102KiB/s-105KiB/s (104kB/s-108kB/s), io=24.6MiB (25.7MB), run=60466-61508msec rand-rw: (groupid=0, jobs=1): err= 0: pid=22026: Fri May 2 00:58:57 2025 read: IOPS=17, BW=68.6KiB/s (70.2kB/s)(5992KiB/87378msec) clat (msec): min=3, max=572, avg=21.78, stdev=24.20 lat (msec): min=3, max=572, avg=21.78, stdev=24.20 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 7], 10.00th=[ 8], 20.00th=[ 10], | 30.00th=[ 11], 40.00th=[ 13], 50.00th=[ 15], 60.00th=[ 17], | 70.00th=[ 22], 80.00th=[ 29], 90.00th=[ 46], 95.00th=[ 65], | 99.00th=[ 105], 99.50th=[ 114], 99.90th=[ 159], 99.95th=[ 575], | 99.99th=[ 575] bw ( KiB/s): min= 7, max= 223, per=25.12%, avg=68.84, stdev=40.41, samples=171 iops : min= 1, max= 55, avg=16.87, stdev=10.16, samples=171 write: IOPS=18, BW=72.1KiB/s (73.8kB/s)(6296KiB/87378msec) clat (msec): min=6, max=237, avg=34.64, stdev=23.34 lat (msec): min=6, max=237, avg=34.64, stdev=23.34 clat percentiles (msec): | 1.00th=[ 10], 5.00th=[ 14], 10.00th=[ 16], 20.00th=[ 18], | 30.00th=[ 21], 40.00th=[ 23], 50.00th=[ 27], 60.00th=[ 31], | 70.00th=[ 40], 80.00th=[ 51], 90.00th=[ 65], 95.00th=[ 79], | 99.00th=[ 122], 99.50th=[ 146], 99.90th=[ 199], 99.95th=[ 239], | 99.99th=[ 239] bw ( KiB/s): min= 15, max= 175, per=24.88%, avg=71.40, stdev=37.71, samples=173 iops : min= 3, max= 43, avg=17.50, stdev= 9.46, samples=173 lat (msec) : 4=0.03%, 10=12.57%, 20=35.29%, 50=38.02%, 100=12.70% lat (msec) : 250=1.37%, 750=0.03% cpu : usr=0.09%, sys=4.40%, ctx=3563, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22027: Fri May 2 00:58:57 2025 read: IOPS=17, BW=68.1KiB/s (69.8kB/s)(5940KiB/87173msec) clat (msec): min=4, max=466, avg=21.77, stdev=22.30 lat (msec): min=4, max=466, avg=21.77, stdev=22.30 clat percentiles (msec): | 1.00th=[ 6], 5.00th=[ 7], 10.00th=[ 8], 20.00th=[ 10], | 30.00th=[ 11], 40.00th=[ 13], 50.00th=[ 14], 60.00th=[ 17], | 70.00th=[ 22], 80.00th=[ 31], 90.00th=[ 48], 95.00th=[ 65], | 99.00th=[ 92], 99.50th=[ 109], 99.90th=[ 155], 99.95th=[ 468], | 99.99th=[ 468] bw ( KiB/s): min= 7, max= 191, per=24.82%, avg=68.01, stdev=41.15, samples=173 iops : min= 1, max= 47, avg=16.63, stdev=10.34, samples=173 write: IOPS=18, BW=72.8KiB/s (74.6kB/s)(6348KiB/87173msec) clat (msec): min=6, max=240, avg=34.41, stdev=23.01 lat (msec): min=6, max=240, avg=34.42, stdev=23.01 clat percentiles (msec): | 1.00th=[ 11], 5.00th=[ 14], 10.00th=[ 16], 20.00th=[ 18], | 30.00th=[ 21], 40.00th=[ 24], 50.00th=[ 27], 60.00th=[ 31], | 70.00th=[ 39], 80.00th=[ 50], 90.00th=[ 67], 95.00th=[ 80], | 99.00th=[ 112], 99.50th=[ 128], 99.90th=[ 182], 99.95th=[ 241], | 99.99th=[ 241] bw ( KiB/s): min= 15, max= 192, per=25.26%, avg=72.49, stdev=39.05, samples=173 iops : min= 3, max= 48, avg=17.75, stdev= 9.81, samples=173 lat (msec) : 10=12.92%, 20=34.77%, 50=37.79%, 100=13.41%, 250=1.07% lat (msec) : 500=0.03% cpu : usr=0.07%, sys=4.34%, ctx=3472, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22028: Fri May 2 00:58:57 2025 read: IOPS=17, BW=69.5KiB/s (71.2kB/s)(6028KiB/86704msec) clat (msec): min=4, max=131, avg=21.21, stdev=18.66 lat (msec): min=4, max=131, avg=21.21, stdev=18.66 clat percentiles (msec): | 1.00th=[ 6], 5.00th=[ 7], 10.00th=[ 8], 20.00th=[ 9], | 30.00th=[ 11], 40.00th=[ 12], 50.00th=[ 14], 60.00th=[ 17], | 70.00th=[ 21], 80.00th=[ 30], 90.00th=[ 48], 95.00th=[ 64], | 99.00th=[ 90], 99.50th=[ 103], 99.90th=[ 127], 99.95th=[ 132], | 99.99th=[ 132] bw ( KiB/s): min= 8, max= 223, per=25.23%, avg=69.14, stdev=39.40, samples=172 iops : min= 2, max= 55, avg=16.81, stdev= 9.87, samples=172 write: IOPS=18, BW=72.2KiB/s (73.9kB/s)(6260KiB/86704msec) clat (msec): min=7, max=476, avg=34.82, stdev=24.87 lat (msec): min=7, max=476, avg=34.83, stdev=24.87 clat percentiles (msec): | 1.00th=[ 11], 5.00th=[ 14], 10.00th=[ 16], 20.00th=[ 19], | 30.00th=[ 21], 40.00th=[ 24], 50.00th=[ 27], 60.00th=[ 32], | 70.00th=[ 39], 80.00th=[ 51], 90.00th=[ 64], 95.00th=[ 80], | 99.00th=[ 113], 99.50th=[ 129], 99.90th=[ 197], 99.95th=[ 477], | 99.99th=[ 477] bw ( KiB/s): min= 15, max= 168, per=24.96%, avg=71.64, stdev=38.86, samples=172 iops : min= 3, max= 42, avg=17.44, stdev= 9.75, samples=172 lat (msec) : 10=13.38%, 20=33.79%, 50=38.31%, 100=13.31%, 250=1.17% lat (msec) : 500=0.03% cpu : usr=0.08%, sys=4.50%, ctx=3476, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22029: Fri May 2 00:58:57 2025 read: IOPS=17, BW=69.7KiB/s (71.4kB/s)(6052KiB/86850msec) clat (msec): min=3, max=154, avg=21.65, stdev=19.56 lat (msec): min=3, max=154, avg=21.65, stdev=19.56 clat percentiles (msec): | 1.00th=[ 6], 5.00th=[ 7], 10.00th=[ 8], 20.00th=[ 10], | 30.00th=[ 11], 40.00th=[ 13], 50.00th=[ 15], 60.00th=[ 17], | 70.00th=[ 21], 80.00th=[ 30], 90.00th=[ 48], 95.00th=[ 67], | 99.00th=[ 93], 99.50th=[ 106], 99.90th=[ 155], 99.95th=[ 155], | 99.99th=[ 155] bw ( KiB/s): min= 7, max= 184, per=25.25%, avg=69.19, stdev=39.57, samples=172 iops : min= 1, max= 46, avg=16.95, stdev= 9.96, samples=172 write: IOPS=17, BW=71.8KiB/s (73.5kB/s)(6236KiB/86850msec) clat (msec): min=6, max=461, avg=34.54, stdev=24.53 lat (msec): min=6, max=461, avg=34.55, stdev=24.53 clat percentiles (msec): | 1.00th=[ 11], 5.00th=[ 14], 10.00th=[ 16], 20.00th=[ 19], | 30.00th=[ 21], 40.00th=[ 23], 50.00th=[ 27], 60.00th=[ 32], | 70.00th=[ 40], 80.00th=[ 51], 90.00th=[ 65], 95.00th=[ 77], | 99.00th=[ 111], 99.50th=[ 132], 99.90th=[ 207], 99.95th=[ 464], | 99.99th=[ 464] bw ( KiB/s): min= 15, max= 160, per=24.88%, avg=71.40, stdev=37.86, samples=172 iops : min= 3, max= 40, avg=17.51, stdev= 9.52, samples=172 lat (msec) : 4=0.07%, 10=12.89%, 20=35.06%, 50=37.43%, 100=13.51% lat (msec) : 250=1.01%, 500=0.03% cpu : usr=0.09%, sys=4.33%, ctx=3515, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=275KiB/s (281kB/s), 68.1KiB/s-69.7KiB/s (69.8kB/s-71.4kB/s), io=23.4MiB (24.6MB), run=86704-87378msec WRITE: bw=288KiB/s (295kB/s), 71.8KiB/s-72.8KiB/s (73.5kB/s-74.6kB/s), io=24.6MiB (25.7MB), run=86704-87378msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=22124: Fri May 2 00:59:15 2025 read: IOPS=24, BW=388KiB/s (397kB/s)(5920KiB/15269msec) clat (usec): min=81, max=86066, avg=22356.53, stdev=14940.66 lat (usec): min=81, max=86067, avg=22368.60, stdev=14926.09 clat percentiles (usec): | 1.00th=[ 95], 5.00th=[ 122], 10.00th=[ 586], 20.00th=[11600], | 30.00th=[14615], 40.00th=[16909], 50.00th=[20317], 60.00th=[23987], | 70.00th=[27657], 80.00th=[33162], 90.00th=[40633], 95.00th=[49021], | 99.00th=[76022], 99.50th=[79168], 99.90th=[86508], 99.95th=[86508], | 99.99th=[86508] bw ( KiB/s): min= 192, max= 697, per=26.38%, avg=384.37, stdev=123.66, samples=30 iops : min= 12, max= 43, avg=23.43, stdev= 7.65, samples=30 write: IOPS=26, BW=417KiB/s (427kB/s)(6368KiB/15269msec) clat (usec): min=1049, max=82236, avg=17344.53, stdev=17802.15 lat (usec): min=1050, max=82238, avg=17346.37, stdev=17802.27 clat percentiles (usec): | 1.00th=[ 1139], 5.00th=[ 1287], 10.00th=[ 1483], 20.00th=[ 2409], | 30.00th=[ 4228], 40.00th=[ 6915], 50.00th=[10421], 60.00th=[16057], | 70.00th=[21365], 80.00th=[31851], 90.00th=[42206], 95.00th=[55313], | 99.00th=[77071], 99.50th=[81265], 99.90th=[82314], 99.95th=[82314], | 99.99th=[82314] bw ( KiB/s): min= 190, max= 640, per=27.17%, avg=416.50, stdev=122.11, samples=30 iops : min= 11, max= 40, avg=25.43, stdev= 7.64, samples=30 lat (usec) : 100=0.65%, 250=3.52%, 500=0.39%, 750=0.39%, 1000=0.13% lat (msec) : 2=9.51%, 4=6.25%, 10=12.24%, 20=25.13%, 50=35.55% lat (msec) : 100=6.25% cpu : usr=0.12%, sys=9.98%, ctx=1397, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22125: Fri May 2 00:59:15 2025 read: IOPS=22, BW=363KiB/s (372kB/s)(5744KiB/15819msec) clat (usec): min=86, max=98774, avg=23280.07, stdev=16812.31 lat (usec): min=86, max=98775, avg=23287.79, stdev=16819.82 clat percentiles (usec): | 1.00th=[ 99], 5.00th=[ 125], 10.00th=[ 163], 20.00th=[11076], | 30.00th=[14353], 40.00th=[17433], 50.00th=[21103], 60.00th=[25035], | 70.00th=[28967], 80.00th=[34341], 90.00th=[43254], 95.00th=[53216], | 99.00th=[82314], 99.50th=[98042], 99.90th=[99091], 99.95th=[99091], | 99.99th=[99091] bw ( KiB/s): min= 63, max= 565, per=24.48%, avg=356.65, stdev=110.31, samples=31 iops : min= 3, max= 35, avg=21.84, stdev= 7.00, samples=31 write: IOPS=25, BW=414KiB/s (424kB/s)(6544KiB/15819msec) clat (usec): min=1026, max=126497, avg=17686.92, stdev=20073.35 lat (usec): min=1027, max=126499, avg=17703.96, stdev=20066.12 clat percentiles (usec): | 1.00th=[ 1156], 5.00th=[ 1352], 10.00th=[ 1516], 20.00th=[ 2180], | 30.00th=[ 3589], 40.00th=[ 5800], 50.00th=[ 10552], 60.00th=[ 15270], | 70.00th=[ 21890], 80.00th=[ 30802], 90.00th=[ 44827], 95.00th=[ 61080], | 99.00th=[ 84411], 99.50th=[ 94897], 99.90th=[126354], 99.95th=[126354], | 99.99th=[126354] bw ( KiB/s): min= 192, max= 860, per=26.12%, avg=400.42, stdev=145.93, samples=31 iops : min= 12, max= 53, avg=24.58, stdev= 9.04, samples=31 lat (usec) : 100=0.52%, 250=4.43%, 500=0.65% lat (msec) : 2=10.55%, 4=8.46%, 10=9.90%, 20=22.92%, 50=35.16% lat (msec) : 100=7.16%, 250=0.26% cpu : usr=0.15%, sys=9.91%, ctx=1356, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22126: Fri May 2 00:59:15 2025 read: IOPS=23, BW=381KiB/s (390kB/s)(6256KiB/16434msec) clat (usec): min=96, max=123580, avg=23400.21, stdev=18911.66 lat (usec): min=97, max=123582, avg=23416.13, stdev=18927.65 clat percentiles (usec): | 1.00th=[ 100], 5.00th=[ 123], 10.00th=[ 141], 20.00th=[ 11076], | 30.00th=[ 14091], 40.00th=[ 16188], 50.00th=[ 20841], 60.00th=[ 24249], | 70.00th=[ 27919], 80.00th=[ 33424], 90.00th=[ 44303], 95.00th=[ 58983], | 99.00th=[100140], 99.50th=[104334], 99.90th=[123208], 99.95th=[123208], | 99.99th=[123208] bw ( KiB/s): min= 220, max= 743, per=25.63%, avg=373.47, stdev=123.69, samples=32 iops : min= 13, max= 46, avg=22.97, stdev= 7.82, samples=32 write: IOPS=22, BW=367KiB/s (376kB/s)(6032KiB/16434msec) clat (usec): min=1060, max=111243, avg=18946.68, stdev=19661.12 lat (usec): min=1062, max=111244, avg=18949.61, stdev=19662.26 clat percentiles (usec): | 1.00th=[ 1139], 5.00th=[ 1352], 10.00th=[ 1549], 20.00th=[ 2737], | 30.00th=[ 5014], 40.00th=[ 8029], 50.00th=[ 13042], 60.00th=[ 16909], | 70.00th=[ 23462], 80.00th=[ 32637], 90.00th=[ 45876], 95.00th=[ 57410], | 99.00th=[ 85459], 99.50th=[110625], 99.90th=[111674], 99.95th=[111674], | 99.99th=[111674] bw ( KiB/s): min= 160, max= 540, per=22.94%, avg=351.66, stdev=108.71, samples=32 iops : min= 10, max= 33, avg=21.59, stdev= 6.76, samples=32 lat (usec) : 100=0.39%, 250=5.73%, 500=0.39% lat (msec) : 2=8.72%, 4=4.04%, 10=11.20%, 20=26.69%, 50=35.29% lat (msec) : 100=6.90%, 250=0.65% cpu : usr=0.18%, sys=9.58%, ctx=1462, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22127: Fri May 2 00:59:15 2025 read: IOPS=25, BW=400KiB/s (410kB/s)(6032KiB/15072msec) clat (usec): min=90, max=94683, avg=22190.96, stdev=15700.81 lat (usec): min=91, max=94684, avg=22192.68, stdev=15700.82 clat percentiles (usec): | 1.00th=[ 100], 5.00th=[ 121], 10.00th=[ 155], 20.00th=[10290], | 30.00th=[14484], 40.00th=[17433], 50.00th=[20055], 60.00th=[23725], | 70.00th=[27395], 80.00th=[32900], 90.00th=[41157], 95.00th=[47973], | 99.00th=[81265], 99.50th=[92799], 99.90th=[94897], 99.95th=[94897], | 99.99th=[94897] bw ( KiB/s): min= 158, max= 576, per=26.85%, avg=391.23, stdev=109.15, samples=30 iops : min= 9, max= 36, avg=24.00, stdev= 6.90, samples=30 write: IOPS=25, BW=415KiB/s (425kB/s)(6256KiB/15072msec) clat (usec): min=929, max=96701, avg=16518.20, stdev=16675.64 lat (usec): min=931, max=96703, avg=16525.76, stdev=16671.49 clat percentiles (usec): | 1.00th=[ 1012], 5.00th=[ 1303], 10.00th=[ 1516], 20.00th=[ 2376], | 30.00th=[ 4178], 40.00th=[ 6980], 50.00th=[11600], 60.00th=[16188], | 70.00th=[21103], 80.00th=[28181], 90.00th=[40109], 95.00th=[50594], | 99.00th=[72877], 99.50th=[87557], 99.90th=[96994], 99.95th=[96994], | 99.99th=[96994] bw ( KiB/s): min= 160, max= 734, per=26.48%, avg=405.93, stdev=126.75, samples=30 iops : min= 10, max= 45, avg=24.93, stdev= 7.86, samples=30 lat (usec) : 100=0.52%, 250=4.95%, 500=0.52%, 750=0.39%, 1000=0.52% lat (msec) : 2=8.59%, 4=6.12%, 10=11.20%, 20=25.78%, 50=36.59% lat (msec) : 100=4.82% cpu : usr=0.09%, sys=10.23%, ctx=1338, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1457KiB/s (1492kB/s), 363KiB/s-400KiB/s (372kB/s-410kB/s), io=23.4MiB (24.5MB), run=15072-16434msec WRITE: bw=1533KiB/s (1570kB/s), 367KiB/s-417KiB/s (376kB/s-427kB/s), io=24.6MiB (25.8MB), run=15072-16434msec rand-rw: (groupid=0, jobs=1): err= 0: pid=22120: Fri May 2 00:59:22 2025 read: IOPS=15, BW=255KiB/s (261kB/s)(5920KiB/23183msec) clat (msec): min=4, max=158, avg=22.96, stdev=21.02 lat (msec): min=4, max=158, avg=22.96, stdev=21.02 clat percentiles (msec): | 1.00th=[ 6], 5.00th=[ 7], 10.00th=[ 8], 20.00th=[ 10], | 30.00th=[ 12], 40.00th=[ 14], 50.00th=[ 16], 60.00th=[ 19], | 70.00th=[ 23], 80.00th=[ 30], 90.00th=[ 52], 95.00th=[ 69], | 99.00th=[ 115], 99.50th=[ 136], 99.90th=[ 159], 99.95th=[ 159], | 99.99th=[ 159] bw ( KiB/s): min= 63, max= 640, per=24.82%, avg=252.20, stdev=130.32, samples=46 iops : min= 3, max= 40, avg=15.20, stdev= 8.29, samples=46 write: IOPS=17, BW=275KiB/s (281kB/s)(6368KiB/23183msec) clat (msec): min=6, max=177, avg=36.18, stdev=22.88 lat (msec): min=6, max=177, avg=36.19, stdev=22.88 clat percentiles (msec): | 1.00th=[ 11], 5.00th=[ 14], 10.00th=[ 16], 20.00th=[ 20], | 30.00th=[ 23], 40.00th=[ 26], 50.00th=[ 29], 60.00th=[ 34], | 70.00th=[ 41], 80.00th=[ 52], 90.00th=[ 66], 95.00th=[ 83], | 99.00th=[ 118], 99.50th=[ 150], 99.90th=[ 178], 99.95th=[ 178], | 99.99th=[ 178] bw ( KiB/s): min= 63, max= 603, per=25.53%, avg=272.93, stdev=141.29, samples=46 iops : min= 3, max= 37, avg=16.50, stdev= 8.90, samples=46 lat (msec) : 10=10.29%, 20=31.90%, 50=41.67%, 100=14.32%, 250=1.82% cpu : usr=0.15%, sys=4.11%, ctx=870, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22121: Fri May 2 00:59:22 2025 read: IOPS=15, BW=246KiB/s (252kB/s)(5744KiB/23317msec) clat (msec): min=3, max=126, avg=22.38, stdev=20.94 lat (msec): min=3, max=126, avg=22.38, stdev=20.94 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 6], 10.00th=[ 7], 20.00th=[ 9], | 30.00th=[ 11], 40.00th=[ 13], 50.00th=[ 15], 60.00th=[ 17], | 70.00th=[ 22], 80.00th=[ 31], 90.00th=[ 53], 95.00th=[ 70], | 99.00th=[ 99], 99.50th=[ 121], 99.90th=[ 127], 99.95th=[ 127], | 99.99th=[ 127] bw ( KiB/s): min= 31, max= 672, per=24.55%, avg=249.40, stdev=154.48, samples=45 iops : min= 1, max= 42, avg=15.07, stdev= 9.75, samples=45 write: IOPS=17, BW=281KiB/s (287kB/s)(6544KiB/23317msec) clat (msec): min=10, max=177, avg=36.46, stdev=23.68 lat (msec): min=10, max=177, avg=36.47, stdev=23.68 clat percentiles (msec): | 1.00th=[ 12], 5.00th=[ 15], 10.00th=[ 16], 20.00th=[ 19], | 30.00th=[ 23], 40.00th=[ 25], 50.00th=[ 28], 60.00th=[ 35], | 70.00th=[ 42], 80.00th=[ 51], 90.00th=[ 70], 95.00th=[ 87], | 99.00th=[ 124], 99.50th=[ 142], 99.90th=[ 178], 99.95th=[ 178], | 99.99th=[ 178] bw ( KiB/s): min= 31, max= 608, per=25.94%, avg=277.26, stdev=143.45, samples=46 iops : min= 1, max= 38, avg=16.80, stdev= 9.03, samples=46 lat (msec) : 4=0.13%, 10=12.50%, 20=31.25%, 50=40.10%, 100=14.45% lat (msec) : 250=1.56% cpu : usr=0.18%, sys=4.10%, ctx=879, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22122: Fri May 2 00:59:22 2025 read: IOPS=17, BW=272KiB/s (279kB/s)(6256KiB/22975msec) clat (msec): min=4, max=159, avg=22.49, stdev=21.68 lat (msec): min=4, max=159, avg=22.49, stdev=21.68 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 7], 10.00th=[ 8], 20.00th=[ 10], | 30.00th=[ 12], 40.00th=[ 13], 50.00th=[ 16], 60.00th=[ 18], | 70.00th=[ 21], 80.00th=[ 30], 90.00th=[ 50], 95.00th=[ 70], | 99.00th=[ 116], 99.50th=[ 157], 99.90th=[ 159], 99.95th=[ 159], | 99.99th=[ 159] bw ( KiB/s): min= 63, max= 796, per=26.66%, avg=270.91, stdev=170.52, samples=45 iops : min= 3, max= 49, avg=16.56, stdev=10.61, samples=45 write: IOPS=16, BW=263KiB/s (269kB/s)(6032KiB/22975msec) clat (msec): min=8, max=148, avg=37.51, stdev=24.23 lat (msec): min=8, max=148, avg=37.51, stdev=24.23 clat percentiles (msec): | 1.00th=[ 11], 5.00th=[ 14], 10.00th=[ 17], 20.00th=[ 20], | 30.00th=[ 23], 40.00th=[ 26], 50.00th=[ 29], 60.00th=[ 35], | 70.00th=[ 43], 80.00th=[ 51], 90.00th=[ 74], 95.00th=[ 91], | 99.00th=[ 115], 99.50th=[ 148], 99.90th=[ 148], 99.95th=[ 148], | 99.99th=[ 148] bw ( KiB/s): min= 95, max= 608, per=24.15%, avg=258.11, stdev=125.31, samples=45 iops : min= 5, max= 38, avg=15.76, stdev= 7.79, samples=45 lat (msec) : 10=11.72%, 20=33.98%, 50=38.80%, 100=12.89%, 250=2.60% cpu : usr=0.14%, sys=4.26%, ctx=890, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22123: Fri May 2 00:59:22 2025 read: IOPS=16, BW=256KiB/s (262kB/s)(6032KiB/23559msec) clat (msec): min=4, max=168, avg=24.10, stdev=23.54 lat (msec): min=4, max=168, avg=24.11, stdev=23.54 clat percentiles (msec): | 1.00th=[ 6], 5.00th=[ 7], 10.00th=[ 9], 20.00th=[ 10], | 30.00th=[ 12], 40.00th=[ 14], 50.00th=[ 16], 60.00th=[ 19], | 70.00th=[ 23], 80.00th=[ 32], 90.00th=[ 53], 95.00th=[ 80], | 99.00th=[ 123], 99.50th=[ 157], 99.90th=[ 169], 99.95th=[ 169], | 99.99th=[ 169] bw ( KiB/s): min= 64, max= 606, per=25.44%, avg=258.46, stdev=135.84, samples=46 iops : min= 4, max= 37, avg=15.65, stdev= 8.46, samples=46 write: IOPS=16, BW=266KiB/s (272kB/s)(6256KiB/23559msec) clat (msec): min=9, max=150, avg=36.09, stdev=22.27 lat (msec): min=9, max=150, avg=36.10, stdev=22.27 clat percentiles (msec): | 1.00th=[ 11], 5.00th=[ 15], 10.00th=[ 17], 20.00th=[ 20], | 30.00th=[ 23], 40.00th=[ 26], 50.00th=[ 29], 60.00th=[ 33], | 70.00th=[ 41], 80.00th=[ 51], 90.00th=[ 71], 95.00th=[ 81], | 99.00th=[ 118], 99.50th=[ 130], 99.90th=[ 150], 99.95th=[ 150], | 99.99th=[ 150] bw ( KiB/s): min= 63, max= 768, per=24.75%, avg=264.55, stdev=151.67, samples=47 iops : min= 3, max= 48, avg=16.02, stdev= 9.56, samples=47 lat (msec) : 10=10.68%, 20=31.51%, 50=42.45%, 100=13.41%, 250=1.95% cpu : usr=0.06%, sys=4.30%, ctx=918, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1017KiB/s (1041kB/s), 246KiB/s-272KiB/s (252kB/s-279kB/s), io=23.4MiB (24.5MB), run=22975-23559msec WRITE: bw=1070KiB/s (1095kB/s), 263KiB/s-281KiB/s (269kB/s-287kB/s), io=24.6MiB (25.8MB), run=22975-23559msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=22157: Fri May 2 00:59:28 2025 read: IOPS=2, BW=2780KiB/s (2847kB/s)(7168KiB/2578msec) clat (usec): min=524, max=228195, avg=138581.84, stdev=89686.38 lat (usec): min=525, max=228196, avg=138583.41, stdev=89686.30 clat percentiles (usec): | 1.00th=[ 523], 5.00th=[ 523], 10.00th=[ 523], 20.00th=[ 36963], | 30.00th=[135267], 40.00th=[135267], 50.00th=[147850], 60.00th=[196084], | 70.00th=[196084], 80.00th=[225444], 90.00th=[227541], 95.00th=[227541], | 99.00th=[227541], 99.50th=[227541], 99.90th=[227541], 99.95th=[227541], | 99.99th=[227541] bw ( KiB/s): min= 1408, max= 6131, per=60.21%, avg=3875.33, stdev=2368.60, samples=3 iops : min= 1, max= 5, avg= 3.00, stdev= 2.00, samples=3 write: IOPS=1, BW=1986KiB/s (2034kB/s)(5120KiB/2578msec) clat (msec): min=136, max=547, avg=258.49, stdev=164.60 lat (msec): min=136, max=547, avg=258.55, stdev=164.58 clat percentiles (msec): | 1.00th=[ 136], 5.00th=[ 136], 10.00th=[ 136], 20.00th=[ 136], | 30.00th=[ 180], 40.00th=[ 180], 50.00th=[ 215], 60.00th=[ 215], | 70.00th=[ 215], 80.00th=[ 215], 90.00th=[ 550], 95.00th=[ 550], | 99.00th=[ 550], 99.50th=[ 550], 99.90th=[ 550], 99.95th=[ 550], | 99.99th=[ 550] bw ( KiB/s): min= 2817, max= 4087, per=38.31%, avg=3452.00, stdev=898.03, samples=2 iops : min= 2, max= 3, avg= 2.50, stdev= 0.71, samples=2 lat (usec) : 750=8.33% lat (msec) : 50=8.33%, 250=75.00%, 750=8.33% cpu : usr=0.00%, sys=10.63%, ctx=143, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22158: Fri May 2 00:59:28 2025 read: IOPS=0, BW=644KiB/s (659kB/s)(2048KiB/3182msec) clat (usec): min=1111, max=221464, avg=111287.86, stdev=155812.76 lat (usec): min=1113, max=221466, avg=111289.71, stdev=155813.10 clat percentiles (usec): | 1.00th=[ 1106], 5.00th=[ 1106], 10.00th=[ 1106], 20.00th=[ 1106], | 30.00th=[ 1106], 40.00th=[ 1106], 50.00th=[ 1106], 60.00th=[221250], | 70.00th=[221250], 80.00th=[221250], 90.00th=[221250], 95.00th=[221250], | 99.00th=[221250], 99.50th=[221250], 99.90th=[221250], 99.95th=[221250], | 99.99th=[221250] bw ( KiB/s): min= 1473, max= 1906, per=26.25%, avg=1689.50, stdev=306.18, samples=2 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=2 write: IOPS=3, BW=3218KiB/s (3295kB/s)(10.0MiB/3182msec) clat (msec): min=57, max=630, avg=266.82, stdev=178.46 lat (msec): min=57, max=630, avg=266.91, stdev=178.43 clat percentiles (msec): | 1.00th=[ 58], 5.00th=[ 58], 10.00th=[ 58], 20.00th=[ 136], | 30.00th=[ 142], 40.00th=[ 171], 50.00th=[ 186], 60.00th=[ 234], | 70.00th=[ 313], 80.00th=[ 313], 90.00th=[ 510], 95.00th=[ 634], | 99.00th=[ 634], 99.50th=[ 634], 99.90th=[ 634], 99.95th=[ 634], | 99.99th=[ 634] bw ( KiB/s): min= 1906, max= 4096, per=33.00%, avg=2973.50, stdev=894.63, samples=4 iops : min= 1, max= 4, avg= 2.25, stdev= 1.26, samples=4 lat (msec) : 2=8.33%, 100=8.33%, 250=50.00%, 500=16.67%, 750=16.67% cpu : usr=0.00%, sys=10.85%, ctx=107, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22159: Fri May 2 00:59:28 2025 read: IOPS=3, BW=3442KiB/s (3525kB/s)(6144KiB/1785msec) clat (usec): min=590, max=208705, avg=137481.76, stdev=77994.54 lat (usec): min=591, max=208707, avg=137483.24, stdev=77994.39 clat percentiles (usec): | 1.00th=[ 594], 5.00th=[ 594], 10.00th=[ 594], 20.00th=[ 99091], | 30.00th=[ 99091], 40.00th=[152044], 50.00th=[152044], 60.00th=[160433], | 70.00th=[204473], 80.00th=[204473], 90.00th=[208667], 95.00th=[208667], | 99.00th=[208667], 99.50th=[208667], 99.90th=[208667], 99.95th=[208667], | 99.99th=[208667] bw ( KiB/s): min= 4047, max= 4087, per=63.19%, avg=4067.00, stdev=28.28, samples=2 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=2 write: IOPS=3, BW=3442KiB/s (3525kB/s)(6144KiB/1785msec) clat (msec): min=58, max=367, avg=147.81, stdev=112.73 lat (msec): min=58, max=367, avg=148.42, stdev=112.78 clat percentiles (msec): | 1.00th=[ 58], 5.00th=[ 58], 10.00th=[ 58], 20.00th=[ 77], | 30.00th=[ 77], 40.00th=[ 109], 50.00th=[ 109], 60.00th=[ 121], | 70.00th=[ 157], 80.00th=[ 157], 90.00th=[ 368], 95.00th=[ 368], | 99.00th=[ 368], 99.50th=[ 368], 99.90th=[ 368], 99.95th=[ 368], | 99.99th=[ 368] bw ( KiB/s): min= 4087, max= 6035, per=56.17%, avg=5061.00, stdev=1377.44, samples=2 iops : min= 3, max= 5, avg= 4.00, stdev= 1.41, samples=2 lat (usec) : 750=8.33% lat (msec) : 100=25.00%, 250=58.33%, 500=8.33% cpu : usr=0.00%, sys=13.34%, ctx=129, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22160: Fri May 2 00:59:28 2025 read: IOPS=1, BW=1682KiB/s (1722kB/s)(5120KiB/3044msec) clat (usec): min=485, max=291269, avg=139882.31, stdev=126410.10 lat (usec): min=487, max=291271, avg=139883.68, stdev=126410.02 clat percentiles (usec): | 1.00th=[ 486], 5.00th=[ 486], 10.00th=[ 486], 20.00th=[ 486], | 30.00th=[ 20841], 40.00th=[ 20841], 50.00th=[164627], 60.00th=[164627], | 70.00th=[221250], 80.00th=[221250], 90.00th=[291505], 95.00th=[291505], | 99.00th=[291505], 99.50th=[291505], 99.90th=[291505], 99.95th=[291505], | 99.99th=[291505] bw ( KiB/s): min= 2035, max= 6131, per=63.44%, avg=4083.00, stdev=2896.31, samples=2 iops : min= 1, max= 5, avg= 3.00, stdev= 2.83, samples=2 write: IOPS=2, BW=2355KiB/s (2411kB/s)(7168KiB/3044msec) clat (msec): min=127, max=695, avg=288.43, stdev=219.20 lat (msec): min=127, max=695, avg=288.53, stdev=219.20 clat percentiles (msec): | 1.00th=[ 128], 5.00th=[ 128], 10.00th=[ 128], 20.00th=[ 129], | 30.00th=[ 136], 40.00th=[ 136], 50.00th=[ 176], 60.00th=[ 284], | 70.00th=[ 284], 80.00th=[ 477], 90.00th=[ 693], 95.00th=[ 693], | 99.00th=[ 693], 99.50th=[ 693], 99.90th=[ 693], 99.95th=[ 693], | 99.99th=[ 693] bw ( KiB/s): min= 2031, max= 4096, per=33.94%, avg=3058.25, stdev=1183.90, samples=4 iops : min= 1, max= 4, avg= 2.25, stdev= 1.50, samples=4 lat (usec) : 500=8.33% lat (msec) : 50=8.33%, 250=50.00%, 500=25.00%, 750=8.33% cpu : usr=0.00%, sys=9.92%, ctx=201, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=6436KiB/s (6591kB/s), 644KiB/s-3442KiB/s (659kB/s-3525kB/s), io=20.0MiB (20.0MB), run=1785-3182msec WRITE: bw=9011KiB/s (9227kB/s), 1986KiB/s-3442KiB/s (2034kB/s-3525kB/s), io=28.0MiB (29.4MB), run=1785-3182msec rand-rw: (groupid=0, jobs=1): err= 0: pid=22152: Fri May 2 00:59:28 2025 read: IOPS=2, BW=2631KiB/s (2695kB/s)(7168KiB/2724msec) clat (msec): min=71, max=503, avg=185.13, stdev=147.29 lat (msec): min=71, max=503, avg=185.13, stdev=147.29 clat percentiles (msec): | 1.00th=[ 71], 5.00th=[ 71], 10.00th=[ 71], 20.00th=[ 81], | 30.00th=[ 127], 40.00th=[ 127], 50.00th=[ 155], 60.00th=[ 167], | 70.00th=[ 167], 80.00th=[ 192], 90.00th=[ 506], 95.00th=[ 506], | 99.00th=[ 506], 99.50th=[ 506], 99.90th=[ 506], 99.95th=[ 506], | 99.99th=[ 506] bw ( KiB/s): min= 2007, max= 4087, per=50.48%, avg=3051.00, stdev=1187.12, samples=4 iops : min= 1, max= 3, avg= 2.00, stdev= 1.15, samples=4 write: IOPS=1, BW=1880KiB/s (1925kB/s)(5120KiB/2724msec) clat (msec): min=95, max=438, avg=219.42, stdev=133.87 lat (msec): min=95, max=438, avg=219.49, stdev=133.86 clat percentiles (msec): | 1.00th=[ 96], 5.00th=[ 96], 10.00th=[ 96], 20.00th=[ 96], | 30.00th=[ 142], 40.00th=[ 142], 50.00th=[ 178], 60.00th=[ 178], | 70.00th=[ 245], 80.00th=[ 245], 90.00th=[ 439], 95.00th=[ 439], | 99.00th=[ 439], 99.50th=[ 439], 99.90th=[ 439], 99.95th=[ 439], | 99.99th=[ 439] bw ( KiB/s): min= 2000, max= 4015, per=31.74%, avg=2686.00, stdev=1151.15, samples=3 iops : min= 1, max= 3, avg= 1.67, stdev= 1.15, samples=3 lat (msec) : 100=25.00%, 250=58.33%, 500=8.33%, 750=8.33% cpu : usr=0.00%, sys=3.20%, ctx=54, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22153: Fri May 2 00:59:28 2025 read: IOPS=0, BW=783KiB/s (801kB/s)(2048KiB/2617msec) clat (msec): min=97, max=567, avg=332.90, stdev=332.35 lat (msec): min=97, max=567, avg=332.90, stdev=332.35 clat percentiles (msec): | 1.00th=[ 99], 5.00th=[ 99], 10.00th=[ 99], 20.00th=[ 99], | 30.00th=[ 99], 40.00th=[ 99], 50.00th=[ 99], 60.00th=[ 567], | 70.00th=[ 567], 80.00th=[ 567], 90.00th=[ 567], 95.00th=[ 567], | 99.00th=[ 567], 99.50th=[ 567], 99.90th=[ 567], 99.95th=[ 567], | 99.99th=[ 567] bw ( KiB/s): min= 2007, max= 2048, per=33.55%, avg=2027.50, stdev=28.99, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 write: IOPS=3, BW=3913KiB/s (4007kB/s)(10.0MiB/2617msec) clat (msec): min=79, max=363, avg=186.06, stdev=94.83 lat (msec): min=79, max=363, avg=186.14, stdev=94.83 clat percentiles (msec): | 1.00th=[ 80], 5.00th=[ 80], 10.00th=[ 80], 20.00th=[ 90], | 30.00th=[ 101], 40.00th=[ 103], 50.00th=[ 161], 60.00th=[ 215], | 70.00th=[ 251], 80.00th=[ 251], 90.00th=[ 268], 95.00th=[ 363], | 99.00th=[ 363], 99.50th=[ 363], 99.90th=[ 363], 99.95th=[ 363], | 99.99th=[ 363] bw ( KiB/s): min= 3984, max= 6035, per=53.56%, avg=4532.50, stdev=1002.78, samples=4 iops : min= 3, max= 5, avg= 3.75, stdev= 0.96, samples=4 lat (msec) : 100=25.00%, 250=41.67%, 500=25.00%, 750=8.33% cpu : usr=0.00%, sys=2.18%, ctx=27, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22154: Fri May 2 00:59:28 2025 read: IOPS=1, BW=2043KiB/s (2092kB/s)(6144KiB/3007msec) clat (msec): min=43, max=318, avg=221.21, stdev=126.49 lat (msec): min=43, max=318, avg=221.21, stdev=126.49 clat percentiles (msec): | 1.00th=[ 44], 5.00th=[ 44], 10.00th=[ 44], 20.00th=[ 75], | 30.00th=[ 75], 40.00th=[ 284], 50.00th=[ 284], 60.00th=[ 292], | 70.00th=[ 313], 80.00th=[ 313], 90.00th=[ 317], 95.00th=[ 317], | 99.00th=[ 317], 99.50th=[ 317], 99.90th=[ 317], 99.95th=[ 317], | 99.99th=[ 317] bw ( KiB/s): min= 2031, max= 6131, per=56.26%, avg=3400.33, stdev=2364.83, samples=3 iops : min= 1, max= 5, avg= 2.33, stdev= 2.31, samples=3 write: IOPS=1, BW=2043KiB/s (2092kB/s)(6144KiB/3007msec) clat (msec): min=122, max=425, avg=240.93, stdev=110.16 lat (msec): min=122, max=425, avg=241.05, stdev=110.17 clat percentiles (msec): | 1.00th=[ 123], 5.00th=[ 123], 10.00th=[ 123], 20.00th=[ 148], | 30.00th=[ 148], 40.00th=[ 222], 50.00th=[ 222], 60.00th=[ 230], | 70.00th=[ 300], 80.00th=[ 300], 90.00th=[ 426], 95.00th=[ 426], | 99.00th=[ 426], 99.50th=[ 426], 99.90th=[ 426], 99.95th=[ 426], | 99.99th=[ 426] bw ( KiB/s): min= 2031, max= 3984, per=28.70%, avg=2429.00, stdev=869.29, samples=5 iops : min= 1, max= 3, avg= 1.60, stdev= 0.89, samples=5 lat (msec) : 50=8.33%, 100=8.33%, 250=33.33%, 500=50.00% cpu : usr=0.07%, sys=2.69%, ctx=37, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22155: Fri May 2 00:59:28 2025 read: IOPS=1, BW=1511KiB/s (1547kB/s)(5120KiB/3388msec) clat (msec): min=54, max=703, avg=200.50, stdev=281.81 lat (msec): min=54, max=703, avg=200.50, stdev=281.81 clat percentiles (msec): | 1.00th=[ 55], 5.00th=[ 55], 10.00th=[ 55], 20.00th=[ 55], | 30.00th=[ 69], 40.00th=[ 69], 50.00th=[ 69], 60.00th=[ 69], | 70.00th=[ 107], 80.00th=[ 107], 90.00th=[ 701], 95.00th=[ 701], | 99.00th=[ 701], 99.50th=[ 701], 99.90th=[ 701], 99.95th=[ 701], | 99.99th=[ 701] bw ( KiB/s): min= 8110, max= 8110, per=100.00%, avg=8110.00, stdev= 0.00, samples=1 iops : min= 7, max= 7, avg= 7.00, stdev= 0.00, samples=1 write: IOPS=2, BW=2116KiB/s (2166kB/s)(7168KiB/3388msec) clat (msec): min=147, max=559, avg=305.67, stdev=144.69 lat (msec): min=147, max=559, avg=305.80, stdev=144.67 clat percentiles (msec): | 1.00th=[ 148], 5.00th=[ 148], 10.00th=[ 148], 20.00th=[ 209], | 30.00th=[ 222], 40.00th=[ 222], 50.00th=[ 230], 60.00th=[ 368], | 70.00th=[ 368], 80.00th=[ 405], 90.00th=[ 558], 95.00th=[ 558], | 99.00th=[ 558], 99.50th=[ 558], 99.90th=[ 558], 99.95th=[ 558], | 99.99th=[ 558] bw ( KiB/s): min= 1996, max= 3938, per=29.57%, avg=2502.25, stdev=957.40, samples=4 iops : min= 1, max= 3, avg= 1.75, stdev= 0.96, samples=4 lat (msec) : 100=25.00%, 250=41.67%, 500=16.67%, 750=16.67% cpu : usr=0.00%, sys=2.80%, ctx=48, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=6045KiB/s (6190kB/s), 783KiB/s-2631KiB/s (801kB/s-2695kB/s), io=20.0MiB (20.0MB), run=2617-3388msec WRITE: bw=8463KiB/s (8666kB/s), 1880KiB/s-3913KiB/s (1925kB/s-4007kB/s), io=28.0MiB (29.4MB), run=2617-3388msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=22174: Fri May 2 00:59:33 2025 read: IOPS=1, BW=7604KiB/s (7786kB/s)(12.0MiB/1616msec) clat (msec): min=482, max=572, avg=529.38, stdev=45.23 lat (msec): min=482, max=572, avg=529.39, stdev=45.23 clat percentiles (msec): | 1.00th=[ 481], 5.00th=[ 481], 10.00th=[ 481], 20.00th=[ 481], | 30.00th=[ 481], 40.00th=[ 535], 50.00th=[ 535], 60.00th=[ 535], | 70.00th=[ 575], 80.00th=[ 575], 90.00th=[ 575], 95.00th=[ 575], | 99.00th=[ 575], 99.50th=[ 575], 99.90th=[ 575], 99.95th=[ 575], | 99.99th=[ 575] bw ( KiB/s): min= 8175, max= 8192, per=100.00%, avg=8183.50, stdev=12.02, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 500=33.33%, 750=66.67% cpu : usr=0.00%, sys=15.85%, ctx=271, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22175: Fri May 2 00:59:33 2025 read: IOPS=0, BW=1820KiB/s (1864kB/s)(4096KiB/2250msec) clat (nsec): min=432119k, max=432119k, avg=432119177.00, stdev= 0.00 lat (nsec): min=432122k, max=432122k, avg=432121815.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 435], 5.00th=[ 435], 10.00th=[ 435], 20.00th=[ 435], | 30.00th=[ 435], 40.00th=[ 435], 50.00th=[ 435], 60.00th=[ 435], | 70.00th=[ 435], 80.00th=[ 435], 90.00th=[ 435], 95.00th=[ 435], | 99.00th=[ 435], 99.50th=[ 435], 99.90th=[ 435], 99.95th=[ 435], | 99.99th=[ 435] bw ( KiB/s): min= 7968, max= 7968, per=100.00%, avg=7968.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=0, BW=3641KiB/s (3728kB/s)(8192KiB/2250msec) clat (msec): min=596, max=1185, avg=891.22, stdev=416.17 lat (msec): min=598, max=1185, avg=892.40, stdev=415.00 clat percentiles (msec): | 1.00th=[ 600], 5.00th=[ 600], 10.00th=[ 600], 20.00th=[ 600], | 30.00th=[ 600], 40.00th=[ 600], 50.00th=[ 600], 60.00th=[ 1183], | 70.00th=[ 1183], 80.00th=[ 1183], 90.00th=[ 1183], 95.00th=[ 1183], | 99.00th=[ 1183], 99.50th=[ 1183], 99.90th=[ 1183], 99.95th=[ 1183], | 99.99th=[ 1183] bw ( KiB/s): min= 8094, max= 8094, per=60.59%, avg=8094.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 500=33.33%, 750=33.33% cpu : usr=0.09%, sys=11.43%, ctx=100, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22176: Fri May 2 00:59:33 2025 write: IOPS=1, BW=5009KiB/s (5130kB/s)(12.0MiB/2453msec) clat (msec): min=431, max=1317, avg=796.98, stdev=463.28 lat (msec): min=431, max=1318, avg=797.28, stdev=463.35 clat percentiles (msec): | 1.00th=[ 430], 5.00th=[ 430], 10.00th=[ 430], 20.00th=[ 430], | 30.00th=[ 430], 40.00th=[ 642], 50.00th=[ 642], 60.00th=[ 642], | 70.00th=[ 1318], 80.00th=[ 1318], 90.00th=[ 1318], 95.00th=[ 1318], | 99.00th=[ 1318], 99.50th=[ 1318], 99.90th=[ 1318], 99.95th=[ 1318], | 99.99th=[ 1318] bw ( KiB/s): min= 8159, max= 8159, per=61.08%, avg=8159.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 500=33.33%, 750=33.33% cpu : usr=0.00%, sys=13.42%, ctx=78, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22177: Fri May 2 00:59:33 2025 write: IOPS=1, BW=5813KiB/s (5952kB/s)(12.0MiB/2114msec) clat (msec): min=311, max=827, avg=649.23, stdev=292.21 lat (msec): min=312, max=827, avg=649.61, stdev=292.17 clat percentiles (msec): | 1.00th=[ 313], 5.00th=[ 313], 10.00th=[ 313], 20.00th=[ 313], | 30.00th=[ 313], 40.00th=[ 810], 50.00th=[ 810], 60.00th=[ 810], | 70.00th=[ 827], 80.00th=[ 827], 90.00th=[ 827], 95.00th=[ 827], | 99.00th=[ 827], 99.50th=[ 827], 99.90th=[ 827], 99.95th=[ 827], | 99.99th=[ 827] bw ( KiB/s): min= 7968, max= 8159, per=60.36%, avg=8063.50, stdev=135.06, samples=2 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=2 lat (msec) : 500=33.33%, 1000=66.67% cpu : usr=0.00%, sys=18.03%, ctx=108, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=7282KiB/s (7457kB/s), 1820KiB/s-7604KiB/s (1864kB/s-7786kB/s), io=16.0MiB (16.8MB), run=1616-2250msec WRITE: bw=13.0MiB/s (13.7MB/s), 3641KiB/s-5813KiB/s (3728kB/s-5952kB/s), io=32.0MiB (33.6MB), run=2114-2453msec rand-rw: (groupid=0, jobs=1): err= 0: pid=22170: Fri May 2 00:59:34 2025 read: IOPS=1, BW=5568KiB/s (5701kB/s)(12.0MiB/2207msec) clat (msec): min=365, max=980, avg=711.44, stdev=314.86 lat (msec): min=365, max=980, avg=711.44, stdev=314.86 clat percentiles (msec): | 1.00th=[ 368], 5.00th=[ 368], 10.00th=[ 368], 20.00th=[ 368], | 30.00th=[ 368], 40.00th=[ 793], 50.00th=[ 793], 60.00th=[ 793], | 70.00th=[ 978], 80.00th=[ 978], 90.00th=[ 978], 95.00th=[ 978], | 99.00th=[ 978], 99.50th=[ 978], 99.90th=[ 978], 99.95th=[ 978], | 99.99th=[ 978] bw ( KiB/s): min= 6420, max= 8143, per=100.00%, avg=7281.50, stdev=1218.34, samples=2 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=2 lat (msec) : 500=33.33%, 1000=66.67% cpu : usr=0.14%, sys=0.86%, ctx=19, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22171: Fri May 2 00:59:34 2025 read: IOPS=0, BW=1424KiB/s (1458kB/s)(4096KiB/2877msec) clat (nsec): min=534409k, max=534409k, avg=534409113.00, stdev= 0.00 lat (nsec): min=534412k, max=534412k, avg=534411850.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 535], 5.00th=[ 535], 10.00th=[ 535], 20.00th=[ 535], | 30.00th=[ 535], 40.00th=[ 535], 50.00th=[ 535], 60.00th=[ 535], | 70.00th=[ 535], 80.00th=[ 535], 90.00th=[ 535], 95.00th=[ 535], | 99.00th=[ 535], 99.50th=[ 535], 99.90th=[ 535], 99.95th=[ 535], | 99.99th=[ 535] bw ( KiB/s): min= 8015, max= 8015, per=100.00%, avg=8015.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=0, BW=2847KiB/s (2916kB/s)(8192KiB/2877msec) clat (msec): min=656, max=1637, avg=1146.84, stdev=694.02 lat (msec): min=657, max=1637, avg=1147.76, stdev=693.20 clat percentiles (msec): | 1.00th=[ 659], 5.00th=[ 659], 10.00th=[ 659], 20.00th=[ 659], | 30.00th=[ 659], 40.00th=[ 659], 50.00th=[ 659], 60.00th=[ 1636], | 70.00th=[ 1636], 80.00th=[ 1636], 90.00th=[ 1636], 95.00th=[ 1636], | 99.00th=[ 1636], 99.50th=[ 1636], 99.90th=[ 1636], 99.95th=[ 1636], | 99.99th=[ 1636] bw ( KiB/s): min= 6977, max= 6977, per=61.26%, avg=6977.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 750=66.67% cpu : usr=0.24%, sys=5.60%, ctx=92, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22172: Fri May 2 00:59:34 2025 write: IOPS=2, BW=8557KiB/s (8762kB/s)(12.0MiB/1436msec) clat (msec): min=167, max=967, avg=463.01, stdev=438.89 lat (msec): min=167, max=968, avg=463.52, stdev=439.10 clat percentiles (msec): | 1.00th=[ 167], 5.00th=[ 167], 10.00th=[ 167], 20.00th=[ 167], | 30.00th=[ 167], 40.00th=[ 255], 50.00th=[ 255], 60.00th=[ 255], | 70.00th=[ 969], 80.00th=[ 969], 90.00th=[ 969], 95.00th=[ 969], | 99.00th=[ 969], 99.50th=[ 969], 99.90th=[ 969], 99.95th=[ 969], | 99.99th=[ 969] bw ( KiB/s): min=16253, max=16253, per=100.00%, avg=16253.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=33.33%, 1000=33.33% cpu : usr=0.35%, sys=5.57%, ctx=35, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22173: Fri May 2 00:59:34 2025 write: IOPS=1, BW=6819KiB/s (6983kB/s)(12.0MiB/1802msec) clat (msec): min=99, max=1336, avg=586.36, stdev=659.26 lat (msec): min=99, max=1337, avg=586.79, stdev=659.38 clat percentiles (msec): | 1.00th=[ 100], 5.00th=[ 100], 10.00th=[ 100], 20.00th=[ 100], | 30.00th=[ 100], 40.00th=[ 326], 50.00th=[ 326], 60.00th=[ 326], | 70.00th=[ 1334], 80.00th=[ 1334], 90.00th=[ 1334], 95.00th=[ 1334], | 99.00th=[ 1334], 99.50th=[ 1334], 99.90th=[ 1334], 99.95th=[ 1334], | 99.99th=[ 1334] bw ( KiB/s): min=16351, max=16351, per=100.00%, avg=16351.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 500=33.33% cpu : usr=0.00%, sys=5.11%, ctx=31, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=5695KiB/s (5831kB/s), 1424KiB/s-5568KiB/s (1458kB/s-5701kB/s), io=16.0MiB (16.8MB), run=2207-2877msec WRITE: bw=11.1MiB/s (11.7MB/s), 2847KiB/s-8557KiB/s (2916kB/s-8762kB/s), io=32.0MiB (33.6MB), run=1436-2877msec