== sanity test 398b: DIO and buffer IO race ============== 16:27:13 (1738963633) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.14486 s, 44.0 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=26436: Fri Feb 7 16:27:26 2025 read: IOPS=141, BW=568KiB/s (581kB/s)(5992KiB/10555msec) clat (usec): min=47, max=171239, avg=3986.50, stdev=4857.09 lat (usec): min=47, max=171240, avg=3987.05, stdev=4857.10 clat percentiles (usec): | 1.00th=[ 63], 5.00th=[ 1549], 10.00th=[ 1876], 20.00th=[ 2212], | 30.00th=[ 2573], 40.00th=[ 2999], 50.00th=[ 3523], 60.00th=[ 4015], | 70.00th=[ 4686], 80.00th=[ 5342], 90.00th=[ 6390], 95.00th=[ 7701], | 99.00th=[ 11731], 99.50th=[ 12649], 99.90th=[ 23725], 99.95th=[170918], | 99.99th=[170918] bw ( KiB/s): min= 264, max= 736, per=25.16%, avg=564.81, stdev=91.52, samples=21 iops : min= 66, max= 184, avg=141.05, stdev=22.93, samples=21 write: IOPS=149, BW=596KiB/s (611kB/s)(6296KiB/10555msec) clat (usec): min=358, max=15680, avg=2870.29, stdev=2523.37 lat (usec): min=358, max=15681, avg=2871.34, stdev=2523.29 clat percentiles (usec): | 1.00th=[ 416], 5.00th=[ 478], 10.00th=[ 515], 20.00th=[ 553], | 30.00th=[ 644], 40.00th=[ 1418], 50.00th=[ 2311], 60.00th=[ 2835], | 70.00th=[ 3720], 80.00th=[ 5014], 90.00th=[ 6587], 95.00th=[ 7898], | 99.00th=[10028], 99.50th=[10814], 99.90th=[15533], 99.95th=[15664], | 99.99th=[15664] bw ( KiB/s): min= 304, max= 816, per=25.22%, avg=592.57, stdev=99.66, samples=21 iops : min= 76, max= 204, avg=148.00, stdev=24.93, samples=21 lat (usec) : 50=0.03%, 100=2.15%, 250=0.13%, 500=3.68%, 750=13.44% lat (usec) : 1000=1.99% lat (msec) : 2=7.58%, 4=37.21%, 10=32.42%, 20=1.30%, 50=0.03% lat (msec) : 250=0.03% cpu : usr=0.23%, sys=22.11%, ctx=5608, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=26438: Fri Feb 7 16:27:26 2025 read: IOPS=141, BW=565KiB/s (579kB/s)(5940KiB/10505msec) clat (usec): min=49, max=20131, avg=3802.79, stdev=2094.27 lat (usec): min=50, max=20131, avg=3804.00, stdev=2094.07 clat percentiles (usec): | 1.00th=[ 64], 5.00th=[ 1565], 10.00th=[ 1909], 20.00th=[ 2278], | 30.00th=[ 2606], 40.00th=[ 2900], 50.00th=[ 3392], 60.00th=[ 4015], | 70.00th=[ 4555], 80.00th=[ 5211], 90.00th=[ 6259], 95.00th=[ 7373], | 99.00th=[10945], 99.50th=[12649], 99.90th=[16712], 99.95th=[20055], | 99.99th=[20055] bw ( KiB/s): min= 272, max= 720, per=25.16%, avg=564.81, stdev=85.94, samples=21 iops : min= 68, max= 180, avg=141.05, stdev=21.54, samples=21 write: IOPS=151, BW=604KiB/s (619kB/s)(6348KiB/10505msec) clat (usec): min=351, max=142298, avg=3017.89, stdev=4413.16 lat (usec): min=352, max=142299, avg=3018.83, stdev=4413.03 clat percentiles (usec): | 1.00th=[ 429], 5.00th=[ 474], 10.00th=[ 515], 20.00th=[ 553], | 30.00th=[ 644], 40.00th=[ 1745], 50.00th=[ 2442], 60.00th=[ 2966], | 70.00th=[ 3687], 80.00th=[ 4883], 90.00th=[ 6587], 95.00th=[ 8029], | 99.00th=[ 10683], 99.50th=[ 14091], 99.90th=[ 29230], 99.95th=[141558], | 99.99th=[141558] bw ( KiB/s): min= 352, max= 824, per=25.67%, avg=603.24, stdev=100.93, samples=21 iops : min= 88, max= 206, avg=150.67, stdev=25.23, samples=21 lat (usec) : 50=0.03%, 100=1.92%, 250=0.13%, 500=4.20%, 750=12.99% lat (usec) : 1000=1.43% lat (msec) : 2=7.03%, 4=39.00%, 10=31.77%, 20=1.40%, 50=0.07% lat (msec) : 250=0.03% cpu : usr=0.25%, sys=21.89%, ctx=5483, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=26440: Fri Feb 7 16:27:26 2025 read: IOPS=140, BW=564KiB/s (577kB/s)(6028KiB/10695msec) clat (usec): min=54, max=17455, avg=3850.20, stdev=2202.14 lat (usec): min=55, max=17456, avg=3850.73, stdev=2202.15 clat percentiles (usec): | 1.00th=[ 64], 5.00th=[ 1516], 10.00th=[ 1876], 20.00th=[ 2245], | 30.00th=[ 2540], 40.00th=[ 2900], 50.00th=[ 3425], 60.00th=[ 4047], | 70.00th=[ 4621], 80.00th=[ 5276], 90.00th=[ 6456], 95.00th=[ 7570], | 99.00th=[11731], 99.50th=[12911], 99.90th=[15664], 99.95th=[17433], | 99.99th=[17433] bw ( KiB/s): min= 312, max= 672, per=24.72%, avg=554.86, stdev=80.54, samples=21 iops : min= 78, max= 168, avg=138.67, stdev=20.14, samples=21 write: IOPS=146, BW=585KiB/s (599kB/s)(6260KiB/10695msec) clat (usec): min=384, max=134298, avg=3074.99, stdev=4260.41 lat (usec): min=385, max=134300, avg=3075.99, stdev=4260.27 clat percentiles (usec): | 1.00th=[ 429], 5.00th=[ 465], 10.00th=[ 510], 20.00th=[ 562], | 30.00th=[ 734], 40.00th=[ 1926], 50.00th=[ 2409], 60.00th=[ 2966], | 70.00th=[ 3720], 80.00th=[ 4883], 90.00th=[ 6783], 95.00th=[ 8160], | 99.00th=[ 11731], 99.50th=[ 13042], 99.90th=[ 26346], 99.95th=[135267], | 99.99th=[135267] bw ( KiB/s): min= 368, max= 728, per=24.42%, avg=573.90, stdev=80.26, samples=21 iops : min= 92, max= 182, avg=143.43, stdev=20.08, samples=21 lat (usec) : 100=2.12%, 250=0.13%, 500=4.30%, 750=11.17%, 1000=1.63% lat (msec) : 2=8.14%, 4=38.57%, 10=31.77%, 20=2.12%, 50=0.03% lat (msec) : 250=0.03% cpu : usr=0.23%, sys=21.59%, ctx=5536, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=26442: Fri Feb 7 16:27:26 2025 read: IOPS=144, BW=577KiB/s (591kB/s)(6052KiB/10484msec) clat (usec): min=48, max=17615, avg=3785.82, stdev=2020.89 lat (usec): min=49, max=17615, avg=3787.32, stdev=2020.16 clat percentiles (usec): | 1.00th=[ 65], 5.00th=[ 1663], 10.00th=[ 1860], 20.00th=[ 2212], | 30.00th=[ 2540], 40.00th=[ 2966], 50.00th=[ 3490], 60.00th=[ 4015], | 70.00th=[ 4555], 80.00th=[ 5080], 90.00th=[ 6063], 95.00th=[ 7177], | 99.00th=[11469], 99.50th=[12649], 99.90th=[15533], 99.95th=[17695], | 99.99th=[17695] bw ( KiB/s): min= 312, max= 712, per=25.31%, avg=568.20, stdev=85.23, samples=20 iops : min= 78, max= 178, avg=142.00, stdev=21.31, samples=20 write: IOPS=148, BW=595KiB/s (609kB/s)(6236KiB/10484msec) clat (usec): min=333, max=143154, avg=3004.27, stdev=4462.65 lat (usec): min=334, max=143155, avg=3005.21, stdev=4462.52 clat percentiles (usec): | 1.00th=[ 420], 5.00th=[ 469], 10.00th=[ 510], 20.00th=[ 553], | 30.00th=[ 644], 40.00th=[ 1500], 50.00th=[ 2278], 60.00th=[ 2835], | 70.00th=[ 3589], 80.00th=[ 5014], 90.00th=[ 6849], 95.00th=[ 8225], | 99.00th=[ 11469], 99.50th=[ 13173], 99.90th=[ 20055], 99.95th=[143655], | 99.99th=[143655] bw ( KiB/s): min= 408, max= 768, per=24.83%, avg=583.40, stdev=72.62, samples=20 iops : min= 102, max= 192, avg=145.80, stdev=18.21, samples=20 lat (usec) : 50=0.07%, 100=1.73%, 250=0.07%, 500=4.13%, 750=12.99% lat (usec) : 1000=1.60% lat (msec) : 2=9.08%, 4=37.04%, 10=31.54%, 20=1.73%, 250=0.03% cpu : usr=0.16%, sys=22.03%, ctx=5465, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2245KiB/s (2299kB/s), 564KiB/s-577KiB/s (577kB/s-591kB/s), io=23.4MiB (24.6MB), run=10484-10695msec WRITE: bw=2351KiB/s (2407kB/s), 585KiB/s-604KiB/s (599kB/s-619kB/s), io=24.6MiB (25.7MB), run=10484-10695msec rand-rw: (groupid=0, jobs=1): err= 0: pid=26435: Fri Feb 7 16:27:30 2025 read: IOPS=106, BW=427KiB/s (438kB/s)(5992KiB/14021msec) clat (usec): min=1408, max=175015, avg=3870.79, stdev=5246.34 lat (usec): min=1409, max=175016, avg=3871.39, stdev=5246.34 clat percentiles (usec): | 1.00th=[ 1516], 5.00th=[ 1647], 10.00th=[ 1729], 20.00th=[ 1860], | 30.00th=[ 2024], 40.00th=[ 2278], 50.00th=[ 2638], 60.00th=[ 3064], | 70.00th=[ 3785], 80.00th=[ 5014], 90.00th=[ 7832], 95.00th=[ 10159], | 99.00th=[ 14222], 99.50th=[ 16581], 99.90th=[ 19792], 99.95th=[175113], | 99.99th=[175113] bw ( KiB/s): min= 184, max= 880, per=25.00%, avg=426.93, stdev=184.30, samples=28 iops : min= 46, max= 220, avg=106.61, stdev=46.08, samples=28 write: IOPS=112, BW=449KiB/s (460kB/s)(6296KiB/14021msec) clat (usec): min=1961, max=23211, avg=5182.60, stdev=3137.18 lat (usec): min=1962, max=23212, avg=5183.27, stdev=3137.19 clat percentiles (usec): | 1.00th=[ 2212], 5.00th=[ 2474], 10.00th=[ 2704], 20.00th=[ 2868], | 30.00th=[ 3130], 40.00th=[ 3425], 50.00th=[ 3982], 60.00th=[ 4752], | 70.00th=[ 5866], 80.00th=[ 7111], 90.00th=[ 9241], 95.00th=[11207], | 99.00th=[18220], 99.50th=[19792], 99.90th=[22676], 99.95th=[23200], | 99.99th=[23200] bw ( KiB/s): min= 208, max= 856, per=25.08%, avg=448.36, stdev=186.59, samples=28 iops : min= 52, max= 214, avg=111.96, stdev=46.66, samples=28 lat (msec) : 2=14.36%, 4=46.32%, 10=33.04%, 20=6.02%, 50=0.23% lat (msec) : 250=0.03% cpu : usr=0.20%, sys=10.84%, ctx=3602, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=26437: Fri Feb 7 16:27:30 2025 read: IOPS=105, BW=423KiB/s (433kB/s)(5940KiB/14057msec) clat (usec): min=1386, max=146615, avg=3780.59, stdev=4561.43 lat (usec): min=1387, max=146616, avg=3781.15, stdev=4561.45 clat percentiles (usec): | 1.00th=[ 1532], 5.00th=[ 1631], 10.00th=[ 1713], 20.00th=[ 1844], | 30.00th=[ 2008], 40.00th=[ 2212], 50.00th=[ 2573], 60.00th=[ 3064], | 70.00th=[ 3752], 80.00th=[ 5080], 90.00th=[ 7963], 95.00th=[ 9765], | 99.00th=[ 12518], 99.50th=[ 14615], 99.90th=[ 17957], 99.95th=[145753], | 99.99th=[145753] bw ( KiB/s): min= 152, max= 910, per=24.71%, avg=422.07, stdev=195.61, samples=28 iops : min= 38, max= 227, avg=105.39, stdev=48.90, samples=28 write: IOPS=112, BW=452KiB/s (462kB/s)(6348KiB/14057msec) clat (usec): min=2000, max=32676, avg=5279.85, stdev=3238.31 lat (usec): min=2001, max=32677, avg=5280.57, stdev=3238.31 clat percentiles (usec): | 1.00th=[ 2180], 5.00th=[ 2474], 10.00th=[ 2671], 20.00th=[ 2868], | 30.00th=[ 3130], 40.00th=[ 3458], 50.00th=[ 3982], 60.00th=[ 4686], | 70.00th=[ 6063], 80.00th=[ 7570], 90.00th=[ 9634], 95.00th=[11338], | 99.00th=[17695], 99.50th=[18744], 99.90th=[25035], 99.95th=[32637], | 99.99th=[32637] bw ( KiB/s): min= 216, max= 864, per=25.17%, avg=450.07, stdev=183.02, samples=28 iops : min= 54, max= 216, avg=112.39, stdev=45.77, samples=28 lat (msec) : 2=14.13%, 4=47.27%, 10=32.19%, 20=6.18%, 50=0.20% lat (msec) : 250=0.03% cpu : usr=0.12%, sys=10.92%, ctx=3602, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=26439: Fri Feb 7 16:27:30 2025 read: IOPS=107, BW=431KiB/s (441kB/s)(6028KiB/14000msec) clat (usec): min=1400, max=20362, avg=3792.41, stdev=2866.70 lat (usec): min=1400, max=20363, avg=3792.98, stdev=2866.71 clat percentiles (usec): | 1.00th=[ 1532], 5.00th=[ 1647], 10.00th=[ 1745], 20.00th=[ 1876], | 30.00th=[ 2057], 40.00th=[ 2278], 50.00th=[ 2638], 60.00th=[ 3130], | 70.00th=[ 3818], 80.00th=[ 5080], 90.00th=[ 7898], 95.00th=[ 9896], | 99.00th=[14615], 99.50th=[16712], 99.90th=[19792], 99.95th=[20317], | 99.99th=[20317] bw ( KiB/s): min= 160, max= 896, per=24.95%, avg=426.15, stdev=184.50, samples=27 iops : min= 40, max= 224, avg=106.41, stdev=46.13, samples=27 write: IOPS=111, BW=447KiB/s (458kB/s)(6260KiB/14000msec) clat (msec): min=2, max=147, avg= 5.25, stdev= 4.72 lat (msec): min=2, max=147, avg= 5.25, stdev= 4.72 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 3], 20.00th=[ 3], | 30.00th=[ 4], 40.00th=[ 4], 50.00th=[ 4], 60.00th=[ 5], | 70.00th=[ 6], 80.00th=[ 8], 90.00th=[ 10], 95.00th=[ 12], | 99.00th=[ 17], 99.50th=[ 20], 99.90th=[ 27], 99.95th=[ 148], | 99.99th=[ 148] bw ( KiB/s): min= 176, max= 920, per=24.61%, avg=440.07, stdev=192.05, samples=27 iops : min= 44, max= 230, avg=109.89, stdev=48.01, samples=27 lat (msec) : 2=13.38%, 4=48.08%, 10=32.16%, 20=6.15%, 50=0.20% lat (msec) : 250=0.03% cpu : usr=0.16%, sys=11.03%, ctx=3600, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=26441: Fri Feb 7 16:27:30 2025 read: IOPS=107, BW=431KiB/s (441kB/s)(6052KiB/14040msec) clat (usec): min=1397, max=19451, avg=3753.83, stdev=2791.05 lat (usec): min=1398, max=19451, avg=3754.42, stdev=2791.14 clat percentiles (usec): | 1.00th=[ 1516], 5.00th=[ 1631], 10.00th=[ 1729], 20.00th=[ 1860], | 30.00th=[ 2040], 40.00th=[ 2311], 50.00th=[ 2638], 60.00th=[ 3097], | 70.00th=[ 3720], 80.00th=[ 5145], 90.00th=[ 7832], 95.00th=[ 9896], | 99.00th=[13960], 99.50th=[16319], 99.90th=[18220], 99.95th=[19530], | 99.99th=[19530] bw ( KiB/s): min= 176, max= 830, per=25.21%, avg=430.64, stdev=190.87, samples=28 iops : min= 44, max= 207, avg=107.54, stdev=47.75, samples=28 write: IOPS=111, BW=444KiB/s (455kB/s)(6236KiB/14040msec) clat (msec): min=2, max=140, avg= 5.32, stdev= 4.68 lat (msec): min=2, max=140, avg= 5.32, stdev= 4.68 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 3], 20.00th=[ 3], | 30.00th=[ 4], 40.00th=[ 4], 50.00th=[ 4], 60.00th=[ 5], | 70.00th=[ 6], 80.00th=[ 8], 90.00th=[ 10], 95.00th=[ 12], | 99.00th=[ 17], 99.50th=[ 21], 99.90th=[ 32], 99.95th=[ 142], | 99.99th=[ 142] bw ( KiB/s): min= 200, max= 840, per=24.76%, avg=442.64, stdev=185.14, samples=28 iops : min= 50, max= 210, avg=110.54, stdev=46.29, samples=28 lat (msec) : 2=14.03%, 4=47.88%, 10=31.18%, 20=6.64%, 50=0.23% lat (msec) : 250=0.03% cpu : usr=0.18%, sys=10.69%, ctx=3624, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1708KiB/s (1749kB/s), 423KiB/s-431KiB/s (433kB/s-441kB/s), io=23.4MiB (24.6MB), run=14000-14057msec WRITE: bw=1788KiB/s (1831kB/s), 444KiB/s-452KiB/s (455kB/s-462kB/s), io=24.6MiB (25.7MB), run=14000-14057msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=26467: Fri Feb 7 16:27:33 2025 read: IOPS=122, BW=1957KiB/s (2004kB/s)(5920KiB/3025msec) clat (usec): min=64, max=16291, avg=4289.40, stdev=2793.15 lat (usec): min=64, max=16291, avg=4290.07, stdev=2793.23 clat percentiles (usec): | 1.00th=[ 67], 5.00th=[ 77], 10.00th=[ 143], 20.00th=[ 2442], | 30.00th=[ 2835], 40.00th=[ 3294], 50.00th=[ 3851], 60.00th=[ 4490], | 70.00th=[ 5211], 80.00th=[ 5997], 90.00th=[ 7504], 95.00th=[ 8979], | 99.00th=[15664], 99.50th=[16057], 99.90th=[16319], 99.95th=[16319], | 99.99th=[16319] bw ( KiB/s): min= 1760, max= 2272, per=24.53%, avg=1941.33, stdev=186.22, samples=6 iops : min= 110, max= 142, avg=121.33, stdev=11.64, samples=6 write: IOPS=131, BW=2105KiB/s (2156kB/s)(6368KiB/3025msec) clat (usec): min=642, max=16943, avg=3550.89, stdev=3002.53 lat (usec): min=643, max=16943, avg=3551.67, stdev=3002.52 clat percentiles (usec): | 1.00th=[ 693], 5.00th=[ 766], 10.00th=[ 791], 20.00th=[ 848], | 30.00th=[ 1139], 40.00th=[ 2147], 50.00th=[ 2769], 60.00th=[ 3261], | 70.00th=[ 4686], 80.00th=[ 5800], 90.00th=[ 7832], 95.00th=[ 9372], | 99.00th=[14484], 99.50th=[15401], 99.90th=[16909], 99.95th=[16909], | 99.99th=[16909] bw ( KiB/s): min= 1696, max= 2432, per=25.17%, avg=2096.00, stdev=276.57, samples=6 iops : min= 106, max= 152, avg=131.00, stdev=17.29, samples=6 lat (usec) : 100=4.69%, 250=0.39%, 750=1.69%, 1000=13.02% lat (msec) : 2=6.25%, 4=33.46%, 10=37.11%, 20=3.39% cpu : usr=0.20%, sys=24.21%, ctx=1500, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=26469: Fri Feb 7 16:27:33 2025 read: IOPS=119, BW=1911KiB/s (1957kB/s)(5744KiB/3005msec) clat (usec): min=59, max=15197, avg=4093.50, stdev=2467.20 lat (usec): min=60, max=15198, avg=4094.04, stdev=2467.20 clat percentiles (usec): | 1.00th=[ 68], 5.00th=[ 78], 10.00th=[ 96], 20.00th=[ 2507], | 30.00th=[ 2933], 40.00th=[ 3326], 50.00th=[ 3851], 60.00th=[ 4359], | 70.00th=[ 4817], 80.00th=[ 5669], 90.00th=[ 6980], 95.00th=[ 8455], | 99.00th=[12125], 99.50th=[13042], 99.90th=[15139], 99.95th=[15139], | 99.99th=[15139] bw ( KiB/s): min= 1532, max= 2240, per=24.17%, avg=1913.17, stdev=271.61, samples=6 iops : min= 95, max= 140, avg=119.33, stdev=17.11, samples=6 write: IOPS=136, BW=2178KiB/s (2230kB/s)(6544KiB/3005msec) clat (usec): min=613, max=17128, avg=3685.40, stdev=2952.01 lat (usec): min=613, max=17129, avg=3686.21, stdev=2952.04 clat percentiles (usec): | 1.00th=[ 693], 5.00th=[ 775], 10.00th=[ 799], 20.00th=[ 873], | 30.00th=[ 1369], 40.00th=[ 2540], 50.00th=[ 3032], 60.00th=[ 3490], | 70.00th=[ 4490], 80.00th=[ 5800], 90.00th=[ 7832], 95.00th=[ 9896], | 99.00th=[12518], 99.50th=[13698], 99.90th=[17171], 99.95th=[17171], | 99.99th=[17171] bw ( KiB/s): min= 1756, max= 3136, per=26.12%, avg=2174.67, stdev=527.42, samples=6 iops : min= 109, max= 196, avg=135.67, stdev=33.19, samples=6 lat (usec) : 100=4.82%, 250=0.26%, 750=1.69%, 1000=11.46% lat (msec) : 2=5.47%, 4=34.64%, 10=37.63%, 20=4.04% cpu : usr=0.13%, sys=23.90%, ctx=1530, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=26471: Fri Feb 7 16:27:33 2025 read: IOPS=129, BW=2067KiB/s (2117kB/s)(6256KiB/3026msec) clat (usec): min=56, max=21907, avg=4322.36, stdev=2716.10 lat (usec): min=57, max=21908, avg=4322.90, stdev=2716.10 clat percentiles (usec): | 1.00th=[ 67], 5.00th=[ 78], 10.00th=[ 2057], 20.00th=[ 2573], | 30.00th=[ 2933], 40.00th=[ 3359], 50.00th=[ 3884], 60.00th=[ 4490], | 70.00th=[ 5145], 80.00th=[ 5866], 90.00th=[ 7046], 95.00th=[ 9110], | 99.00th=[13698], 99.50th=[18744], 99.90th=[21890], 99.95th=[21890], | 99.99th=[21890] bw ( KiB/s): min= 1724, max= 2528, per=25.92%, avg=2051.83, stdev=299.32, samples=6 iops : min= 107, max= 158, avg=128.00, stdev=18.88, samples=6 write: IOPS=124, BW=1993KiB/s (2041kB/s)(6032KiB/3026msec) clat (usec): min=629, max=14296, avg=3502.99, stdev=2790.46 lat (usec): min=629, max=14297, avg=3503.82, stdev=2790.42 clat percentiles (usec): | 1.00th=[ 668], 5.00th=[ 766], 10.00th=[ 799], 20.00th=[ 865], | 30.00th=[ 1188], 40.00th=[ 2245], 50.00th=[ 2868], 60.00th=[ 3687], | 70.00th=[ 4490], 80.00th=[ 5473], 90.00th=[ 7308], 95.00th=[ 9241], | 99.00th=[13042], 99.50th=[14091], 99.90th=[14353], 99.95th=[14353], | 99.99th=[14353] bw ( KiB/s): min= 1632, max= 2299, per=23.42%, avg=1950.33, stdev=222.61, samples=6 iops : min= 102, max= 143, avg=121.67, stdev=13.66, samples=6 lat (usec) : 100=3.91%, 250=0.26%, 750=2.21%, 1000=10.94% lat (msec) : 2=6.38%, 4=35.29%, 10=37.50%, 20=3.39%, 50=0.13% cpu : usr=0.10%, sys=24.33%, ctx=1568, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=26473: Fri Feb 7 16:27:33 2025 read: IOPS=126, BW=2019KiB/s (2067kB/s)(6032KiB/2988msec) clat (usec): min=59, max=16610, avg=4252.87, stdev=2837.22 lat (usec): min=60, max=16611, avg=4253.45, stdev=2837.16 clat percentiles (usec): | 1.00th=[ 68], 5.00th=[ 71], 10.00th=[ 87], 20.00th=[ 2474], | 30.00th=[ 2966], 40.00th=[ 3392], 50.00th=[ 3916], 60.00th=[ 4555], | 70.00th=[ 5211], 80.00th=[ 6128], 90.00th=[ 7635], 95.00th=[ 8848], | 99.00th=[14877], 99.50th=[16319], 99.90th=[16581], 99.95th=[16581], | 99.99th=[16581] bw ( KiB/s): min= 1568, max= 2048, per=23.68%, avg=1874.40, stdev=198.30, samples=5 iops : min= 98, max= 128, avg=117.00, stdev=12.45, samples=5 write: IOPS=130, BW=2094KiB/s (2144kB/s)(6256KiB/2988msec) clat (usec): min=644, max=15897, avg=3439.54, stdev=2843.45 lat (usec): min=645, max=15898, avg=3440.32, stdev=2843.45 clat percentiles (usec): | 1.00th=[ 701], 5.00th=[ 758], 10.00th=[ 791], 20.00th=[ 857], | 30.00th=[ 1156], 40.00th=[ 2245], 50.00th=[ 2671], 60.00th=[ 3261], | 70.00th=[ 4080], 80.00th=[ 5473], 90.00th=[ 7701], 95.00th=[ 9634], | 99.00th=[12649], 99.50th=[13829], 99.90th=[15926], 99.95th=[15926], | 99.99th=[15926] bw ( KiB/s): min= 1724, max= 2432, per=24.35%, avg=2028.00, stdev=309.24, samples=5 iops : min= 107, max= 152, avg=126.60, stdev=19.51, samples=5 lat (usec) : 100=6.38%, 250=0.26%, 750=2.47%, 1000=11.72% lat (msec) : 2=5.47%, 4=34.24%, 10=36.07%, 20=3.39% cpu : usr=0.10%, sys=24.10%, ctx=1485, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=7915KiB/s (8105kB/s), 1911KiB/s-2067KiB/s (1957kB/s-2117kB/s), io=23.4MiB (24.5MB), run=2988-3026msec WRITE: bw=8328KiB/s (8528kB/s), 1993KiB/s-2178KiB/s (2041kB/s-2230kB/s), io=24.6MiB (25.8MB), run=2988-3026msec rand-rw: (groupid=0, jobs=1): err= 0: pid=26468: Fri Feb 7 16:27:34 2025 read: IOPS=96, BW=1537KiB/s (1574kB/s)(5920KiB/3851msec) clat (usec): min=1664, max=15859, avg=4013.10, stdev=2846.78 lat (usec): min=1665, max=15859, avg=4013.81, stdev=2847.03 clat percentiles (usec): | 1.00th=[ 1745], 5.00th=[ 1876], 10.00th=[ 1975], 20.00th=[ 2089], | 30.00th=[ 2212], 40.00th=[ 2376], 50.00th=[ 2606], 60.00th=[ 2999], | 70.00th=[ 4015], 80.00th=[ 6128], 90.00th=[ 8586], 95.00th=[10290], | 99.00th=[13042], 99.50th=[14484], 99.90th=[15795], 99.95th=[15795], | 99.99th=[15795] bw ( KiB/s): min= 992, max= 2400, per=22.37%, avg=1385.14, stdev=478.83, samples=7 iops : min= 62, max= 150, avg=86.57, stdev=29.93, samples=7 write: IOPS=103, BW=1654KiB/s (1693kB/s)(6368KiB/3851msec) clat (usec): min=2263, max=25516, avg=5861.07, stdev=3685.20 lat (usec): min=2264, max=25517, avg=5862.06, stdev=3685.18 clat percentiles (usec): | 1.00th=[ 2376], 5.00th=[ 2737], 10.00th=[ 2933], 20.00th=[ 3195], | 30.00th=[ 3359], 40.00th=[ 3687], 50.00th=[ 4178], 60.00th=[ 5080], | 70.00th=[ 6849], 80.00th=[ 8586], 90.00th=[10683], 95.00th=[12649], | 99.00th=[19792], 99.50th=[21103], 99.90th=[25560], 99.95th=[25560], | 99.99th=[25560] bw ( KiB/s): min= 1120, max= 2592, per=23.36%, avg=1522.29, stdev=492.93, samples=7 iops : min= 70, max= 162, avg=95.14, stdev=30.81, samples=7 lat (msec) : 2=5.60%, 4=51.30%, 10=32.42%, 20=10.29%, 50=0.39% cpu : usr=0.10%, sys=11.71%, ctx=923, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=26470: Fri Feb 7 16:27:34 2025 read: IOPS=92, BW=1485KiB/s (1521kB/s)(5744KiB/3867msec) clat (usec): min=1631, max=22051, avg=4154.79, stdev=3312.38 lat (usec): min=1631, max=22051, avg=4155.33, stdev=3312.38 clat percentiles (usec): | 1.00th=[ 1745], 5.00th=[ 1876], 10.00th=[ 1975], 20.00th=[ 2073], | 30.00th=[ 2212], 40.00th=[ 2376], 50.00th=[ 2671], 60.00th=[ 3032], | 70.00th=[ 3982], 80.00th=[ 5735], 90.00th=[ 9634], 95.00th=[11731], | 99.00th=[15664], 99.50th=[16712], 99.90th=[22152], 99.95th=[22152], | 99.99th=[22152] bw ( KiB/s): min= 958, max= 2560, per=21.99%, avg=1361.57, stdev=551.81, samples=7 iops : min= 59, max= 160, avg=84.86, stdev=34.63, samples=7 write: IOPS=105, BW=1692KiB/s (1733kB/s)(6544KiB/3867msec) clat (usec): min=2401, max=24590, avg=5780.98, stdev=3532.27 lat (usec): min=2402, max=24591, avg=5781.94, stdev=3532.24 clat percentiles (usec): | 1.00th=[ 2442], 5.00th=[ 2671], 10.00th=[ 2802], 20.00th=[ 3064], | 30.00th=[ 3294], 40.00th=[ 3785], 50.00th=[ 4424], 60.00th=[ 5473], | 70.00th=[ 6587], 80.00th=[ 8029], 90.00th=[10683], 95.00th=[13173], | 99.00th=[17695], 99.50th=[18482], 99.90th=[24511], 99.95th=[24511], | 99.99th=[24511] bw ( KiB/s): min= 1152, max= 2656, per=23.42%, avg=1526.00, stdev=506.75, samples=7 iops : min= 72, max= 166, avg=95.14, stdev=31.73, samples=7 lat (msec) : 2=6.25%, 4=49.48%, 10=33.98%, 20=9.90%, 50=0.39% cpu : usr=0.16%, sys=11.51%, ctx=888, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=26472: Fri Feb 7 16:27:34 2025 read: IOPS=102, BW=1642KiB/s (1681kB/s)(6256KiB/3810msec) clat (usec): min=1700, max=22645, avg=4252.92, stdev=3315.56 lat (usec): min=1701, max=22645, avg=4253.53, stdev=3315.54 clat percentiles (usec): | 1.00th=[ 1745], 5.00th=[ 1860], 10.00th=[ 1942], 20.00th=[ 2114], | 30.00th=[ 2278], 40.00th=[ 2442], 50.00th=[ 2671], 60.00th=[ 3228], | 70.00th=[ 4424], 80.00th=[ 5932], 90.00th=[ 9372], 95.00th=[11469], | 99.00th=[16319], 99.50th=[19792], 99.90th=[22676], 99.95th=[22676], | 99.99th=[22676] bw ( KiB/s): min= 1024, max= 2560, per=24.05%, avg=1489.29, stdev=516.87, samples=7 iops : min= 64, max= 160, avg=92.86, stdev=32.39, samples=7 write: IOPS=98, BW=1583KiB/s (1621kB/s)(6032KiB/3810msec) clat (usec): min=2295, max=21028, avg=5571.44, stdev=3316.96 lat (usec): min=2296, max=21029, avg=5572.48, stdev=3316.93 clat percentiles (usec): | 1.00th=[ 2507], 5.00th=[ 2835], 10.00th=[ 2933], 20.00th=[ 3130], | 30.00th=[ 3392], 40.00th=[ 3720], 50.00th=[ 4146], 60.00th=[ 4948], | 70.00th=[ 6325], 80.00th=[ 7635], 90.00th=[10552], 95.00th=[12649], | 99.00th=[17957], 99.50th=[19792], 99.90th=[21103], 99.95th=[21103], | 99.99th=[21103] bw ( KiB/s): min= 1120, max= 2688, per=22.65%, avg=1475.71, stdev=546.91, samples=7 iops : min= 70, max= 168, avg=92.00, stdev=34.26, samples=7 lat (msec) : 2=6.64%, 4=50.65%, 10=32.81%, 20=9.64%, 50=0.26% cpu : usr=0.08%, sys=11.97%, ctx=918, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=26474: Fri Feb 7 16:27:34 2025 read: IOPS=98, BW=1574KiB/s (1611kB/s)(6032KiB/3833msec) clat (usec): min=1734, max=21585, avg=4204.15, stdev=3201.32 lat (usec): min=1735, max=21585, avg=4204.86, stdev=3201.48 clat percentiles (usec): | 1.00th=[ 1778], 5.00th=[ 1893], 10.00th=[ 1942], 20.00th=[ 2073], | 30.00th=[ 2180], 40.00th=[ 2409], 50.00th=[ 2769], 60.00th=[ 3163], | 70.00th=[ 4178], 80.00th=[ 5932], 90.00th=[ 9110], 95.00th=[11076], | 99.00th=[15664], 99.50th=[19006], 99.90th=[21627], 99.95th=[21627], | 99.99th=[21627] bw ( KiB/s): min= 960, max= 2624, per=23.02%, avg=1425.57, stdev=572.39, samples=7 iops : min= 60, max= 164, avg=88.86, stdev=35.90, samples=7 write: IOPS=102, BW=1632KiB/s (1671kB/s)(6256KiB/3833msec) clat (usec): min=2382, max=19371, avg=5643.87, stdev=3181.17 lat (usec): min=2383, max=19372, avg=5645.05, stdev=3181.22 clat percentiles (usec): | 1.00th=[ 2442], 5.00th=[ 2704], 10.00th=[ 2900], 20.00th=[ 3130], | 30.00th=[ 3359], 40.00th=[ 3720], 50.00th=[ 4293], 60.00th=[ 5407], | 70.00th=[ 6456], 80.00th=[ 8225], 90.00th=[10290], 95.00th=[12256], | 99.00th=[15795], 99.50th=[18220], 99.90th=[19268], 99.95th=[19268], | 99.99th=[19268] bw ( KiB/s): min= 1213, max= 2752, per=23.07%, avg=1503.14, stdev=554.08, samples=7 iops : min= 75, max= 172, avg=93.71, stdev=34.75, samples=7 lat (msec) : 2=6.12%, 4=50.91%, 10=34.24%, 20=8.59%, 50=0.13% cpu : usr=0.23%, sys=11.72%, ctx=920, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=6194KiB/s (6343kB/s), 1485KiB/s-1642KiB/s (1521kB/s-1681kB/s), io=23.4MiB (24.5MB), run=3810-3867msec WRITE: bw=6517KiB/s (6673kB/s), 1583KiB/s-1692KiB/s (1621kB/s-1733kB/s), io=24.6MiB (25.8MB), run=3810-3867msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=26483: Fri Feb 7 16:27:37 2025 read: IOPS=5, BW=5164KiB/s (5288kB/s)(7168KiB/1388msec) clat (usec): min=392, max=923739, avg=154638.59, stdev=340432.33 lat (usec): min=393, max=923741, avg=154639.97, stdev=340432.76 clat percentiles (usec): | 1.00th=[ 392], 5.00th=[ 392], 10.00th=[ 392], 20.00th=[ 416], | 30.00th=[ 515], 40.00th=[ 515], 50.00th=[ 30016], 60.00th=[ 49021], | 70.00th=[ 49021], 80.00th=[ 78119], 90.00th=[926942], 95.00th=[926942], | 99.00th=[926942], 99.50th=[926942], 99.90th=[926942], 99.95th=[926942], | 99.99th=[926942] bw ( KiB/s): min= 2048, max= 2048, per=14.69%, avg=2048.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=3, BW=3689KiB/s (3777kB/s)(5120KiB/1388msec) clat (usec): min=32942, max=61515, avg=45695.25, stdev=11754.08 lat (usec): min=32990, max=61599, avg=45738.78, stdev=11761.55 clat percentiles (usec): | 1.00th=[32900], 5.00th=[32900], 10.00th=[32900], 20.00th=[32900], | 30.00th=[35914], 40.00th=[35914], 50.00th=[45876], 60.00th=[45876], | 70.00th=[52167], 80.00th=[52167], 90.00th=[61604], 95.00th=[61604], | 99.00th=[61604], 99.50th=[61604], 99.90th=[61604], 99.95th=[61604], | 99.99th=[61604] lat (usec) : 500=16.67%, 750=8.33% lat (msec) : 50=41.67%, 100=25.00%, 1000=8.33% cpu : usr=0.00%, sys=20.48%, ctx=602, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=26484: Fri Feb 7 16:27:37 2025 read: IOPS=1, BW=1394KiB/s (1428kB/s)(2048KiB/1469msec) clat (usec): min=28636, max=34979, avg=31808.21, stdev=4485.17 lat (usec): min=28637, max=34980, avg=31809.14, stdev=4484.70 clat percentiles (usec): | 1.00th=[28705], 5.00th=[28705], 10.00th=[28705], 20.00th=[28705], | 30.00th=[28705], 40.00th=[28705], 50.00th=[28705], 60.00th=[34866], | 70.00th=[34866], 80.00th=[34866], 90.00th=[34866], 95.00th=[34866], | 99.00th=[34866], 99.50th=[34866], 99.90th=[34866], 99.95th=[34866], | 99.99th=[34866] bw ( KiB/s): min= 2048, max= 2048, per=14.69%, avg=2048.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=6, BW=6971KiB/s (7138kB/s)(10.0MiB/1469msec) clat (msec): min=17, max=975, avg=133.65, stdev=296.42 lat (msec): min=17, max=975, avg=133.69, stdev=296.43 clat percentiles (msec): | 1.00th=[ 18], 5.00th=[ 18], 10.00th=[ 18], 20.00th=[ 23], | 30.00th=[ 33], 40.00th=[ 34], 50.00th=[ 35], 60.00th=[ 38], | 70.00th=[ 66], 80.00th=[ 66], 90.00th=[ 73], 95.00th=[ 978], | 99.00th=[ 978], 99.50th=[ 978], 99.90th=[ 978], 99.95th=[ 978], | 99.99th=[ 978] lat (msec) : 20=8.33%, 50=66.67%, 100=16.67%, 1000=8.33% cpu : usr=0.00%, sys=17.78%, ctx=132, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=26485: Fri Feb 7 16:27:37 2025 read: IOPS=4, BW=4651KiB/s (4763kB/s)(6144KiB/1321msec) clat (usec): min=384, max=40438, avg=25605.48, stdev=13329.38 lat (usec): min=385, max=40439, avg=25606.30, stdev=13329.44 clat percentiles (usec): | 1.00th=[ 383], 5.00th=[ 383], 10.00th=[ 383], 20.00th=[26608], | 30.00th=[26608], 40.00th=[27657], 50.00th=[27657], 60.00th=[29230], | 70.00th=[29230], 80.00th=[29230], 90.00th=[40633], 95.00th=[40633], | 99.00th=[40633], 99.50th=[40633], 99.90th=[40633], 99.95th=[40633], | 99.99th=[40633] write: IOPS=4, BW=4651KiB/s (4763kB/s)(6144KiB/1321msec) clat (msec): min=26, max=1002, avg=192.55, stdev=396.97 lat (msec): min=26, max=1002, avg=192.58, stdev=396.96 clat percentiles (msec): | 1.00th=[ 27], 5.00th=[ 27], 10.00th=[ 27], 20.00th=[ 27], | 30.00th=[ 27], 40.00th=[ 32], 50.00th=[ 32], 60.00th=[ 33], | 70.00th=[ 35], 80.00th=[ 35], 90.00th=[ 1003], 95.00th=[ 1003], | 99.00th=[ 1003], 99.50th=[ 1003], 99.90th=[ 1003], 99.95th=[ 1003], | 99.99th=[ 1003] bw ( KiB/s): min= 4096, max= 4096, per=20.99%, avg=4096.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (usec) : 500=8.33% lat (msec) : 50=83.33% cpu : usr=0.00%, sys=15.15%, ctx=206, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=26487: Fri Feb 7 16:27:37 2025 read: IOPS=3, BW=3551KiB/s (3636kB/s)(5120KiB/1442msec) clat (usec): min=514, max=46400, avg=18978.91, stdev=24142.92 lat (usec): min=515, max=46400, avg=18979.67, stdev=24142.88 clat percentiles (usec): | 1.00th=[ 515], 5.00th=[ 515], 10.00th=[ 515], 20.00th=[ 515], | 30.00th=[ 865], 40.00th=[ 865], 50.00th=[ 2737], 60.00th=[ 2737], | 70.00th=[44303], 80.00th=[44303], 90.00th=[46400], 95.00th=[46400], | 99.00th=[46400], 99.50th=[46400], 99.90th=[46400], 99.95th=[46400], | 99.99th=[46400] write: IOPS=4, BW=4971KiB/s (5090kB/s)(7168KiB/1442msec) clat (msec): min=18, max=1009, avg=181.45, stdev=366.16 lat (msec): min=18, max=1009, avg=181.51, stdev=366.16 clat percentiles (msec): | 1.00th=[ 19], 5.00th=[ 19], 10.00th=[ 19], 20.00th=[ 24], | 30.00th=[ 36], 40.00th=[ 36], 50.00th=[ 38], 60.00th=[ 46], | 70.00th=[ 46], 80.00th=[ 101], 90.00th=[ 1011], 95.00th=[ 1011], | 99.00th=[ 1011], 99.50th=[ 1011], 99.90th=[ 1011], 99.95th=[ 1011], | 99.99th=[ 1011] bw ( KiB/s): min= 2048, max= 2048, per=10.49%, avg=2048.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (usec) : 750=8.33%, 1000=8.33% lat (msec) : 4=8.33%, 20=8.33%, 50=50.00%, 100=8.33% cpu : usr=0.00%, sys=12.70%, ctx=175, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=13.6MiB/s (14.3MB/s), 1394KiB/s-5164KiB/s (1428kB/s-5288kB/s), io=20.0MiB (20.0MB), run=1321-1469msec WRITE: bw=19.1MiB/s (19.0MB/s), 3689KiB/s-6971KiB/s (3777kB/s-7138kB/s), io=28.0MiB (29.4MB), run=1321-1469msec rand-rw: (groupid=0, jobs=1): err= 0: pid=26488: Fri Feb 7 16:27:37 2025 read: IOPS=4, BW=4607KiB/s (4717kB/s)(7168KiB/1556msec) clat (msec): min=14, max=939, avg=182.06, stdev=338.79 lat (msec): min=14, max=939, avg=182.06, stdev=338.79 clat percentiles (msec): | 1.00th=[ 16], 5.00th=[ 16], 10.00th=[ 16], 20.00th=[ 19], | 30.00th=[ 24], 40.00th=[ 24], 50.00th=[ 37], 60.00th=[ 61], | 70.00th=[ 61], 80.00th=[ 182], 90.00th=[ 936], 95.00th=[ 936], | 99.00th=[ 936], 99.50th=[ 936], 99.90th=[ 936], 99.95th=[ 936], | 99.99th=[ 936] bw ( KiB/s): min=10240, max=10240, per=78.75%, avg=10240.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 write: IOPS=3, BW=3290KiB/s (3369kB/s)(5120KiB/1556msec) clat (usec): min=14341, max=72611, avg=43811.37, stdev=23919.93 lat (usec): min=14391, max=72675, avg=43867.56, stdev=23918.19 clat percentiles (usec): | 1.00th=[14353], 5.00th=[14353], 10.00th=[14353], 20.00th=[14353], | 30.00th=[25297], 40.00th=[25297], 50.00th=[47973], 60.00th=[47973], | 70.00th=[58983], 80.00th=[58983], 90.00th=[72877], 95.00th=[72877], | 99.00th=[72877], 99.50th=[72877], 99.90th=[72877], 99.95th=[72877], | 99.99th=[72877] bw ( KiB/s): min= 8192, max= 8192, per=45.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 lat (msec) : 20=25.00%, 50=33.33%, 100=25.00%, 250=8.33%, 1000=8.33% cpu : usr=0.00%, sys=3.99%, ctx=37, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=26489: Fri Feb 7 16:27:37 2025 read: IOPS=1, BW=1300KiB/s (1332kB/s)(2048KiB/1575msec) clat (msec): min=13, max=105, avg=59.70, stdev=65.06 lat (msec): min=13, max=105, avg=59.70, stdev=65.06 clat percentiles (msec): | 1.00th=[ 14], 5.00th=[ 14], 10.00th=[ 14], 20.00th=[ 14], | 30.00th=[ 14], 40.00th=[ 14], 50.00th=[ 14], 60.00th=[ 106], | 70.00th=[ 106], 80.00th=[ 106], 90.00th=[ 106], 95.00th=[ 106], | 99.00th=[ 106], 99.50th=[ 106], 99.90th=[ 106], 99.95th=[ 106], | 99.99th=[ 106] bw ( KiB/s): min= 2048, max= 2048, per=15.75%, avg=2048.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 write: IOPS=6, BW=6502KiB/s (6658kB/s)(10.0MiB/1575msec) clat (msec): min=16, max=995, avg=142.83, stdev=299.86 lat (msec): min=16, max=995, avg=142.89, stdev=299.86 clat percentiles (msec): | 1.00th=[ 17], 5.00th=[ 17], 10.00th=[ 17], 20.00th=[ 36], | 30.00th=[ 43], 40.00th=[ 44], 50.00th=[ 47], 60.00th=[ 55], | 70.00th=[ 64], 80.00th=[ 64], 90.00th=[ 74], 95.00th=[ 995], | 99.00th=[ 995], 99.50th=[ 995], 99.90th=[ 995], 99.95th=[ 995], | 99.99th=[ 995] bw ( KiB/s): min=14336, max=14336, per=78.75%, avg=14336.00, stdev= 0.00, samples=1 iops : min= 14, max= 14, avg=14.00, stdev= 0.00, samples=1 lat (msec) : 20=16.67%, 50=33.33%, 100=33.33%, 250=8.33%, 1000=8.33% cpu : usr=0.00%, sys=8.58%, ctx=210, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=26490: Fri Feb 7 16:27:37 2025 read: IOPS=3, BW=4021KiB/s (4117kB/s)(6144KiB/1528msec) clat (usec): min=19211, max=50562, avg=35305.09, stdev=11657.14 lat (usec): min=19211, max=50563, avg=35305.86, stdev=11657.06 clat percentiles (usec): | 1.00th=[19268], 5.00th=[19268], 10.00th=[19268], 20.00th=[31065], | 30.00th=[31065], 40.00th=[31327], 50.00th=[31327], 60.00th=[32113], | 70.00th=[47449], 80.00th=[47449], 90.00th=[50594], 95.00th=[50594], | 99.00th=[50594], 99.50th=[50594], 99.90th=[50594], 99.95th=[50594], | 99.99th=[50594] bw ( KiB/s): min=10240, max=10240, per=78.75%, avg=10240.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 write: IOPS=3, BW=4021KiB/s (4117kB/s)(6144KiB/1528msec) clat (msec): min=20, max=1020, avg=208.88, stdev=398.27 lat (msec): min=20, max=1020, avg=208.93, stdev=398.29 clat percentiles (msec): | 1.00th=[ 21], 5.00th=[ 21], 10.00th=[ 21], 20.00th=[ 21], | 30.00th=[ 21], 40.00th=[ 56], 50.00th=[ 56], 60.00th=[ 63], | 70.00th=[ 74], 80.00th=[ 74], 90.00th=[ 1020], 95.00th=[ 1020], | 99.00th=[ 1020], 99.50th=[ 1020], 99.90th=[ 1020], 99.95th=[ 1020], | 99.99th=[ 1020] bw ( KiB/s): min= 4096, max= 8192, per=33.75%, avg=6144.00, stdev=2896.31, samples=2 iops : min= 4, max= 8, avg= 6.00, stdev= 2.83, samples=2 lat (msec) : 20=8.33%, 50=50.00%, 100=33.33% cpu : usr=0.00%, sys=6.48%, ctx=76, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=26491: Fri Feb 7 16:27:37 2025 read: IOPS=3, BW=3351KiB/s (3431kB/s)(5120KiB/1528msec) clat (usec): min=11709, max=54277, avg=29695.83, stdev=15369.47 lat (usec): min=11710, max=54277, avg=29696.55, stdev=15369.44 clat percentiles (usec): | 1.00th=[11731], 5.00th=[11731], 10.00th=[11731], 20.00th=[11731], | 30.00th=[26084], 40.00th=[26084], 50.00th=[27919], 60.00th=[27919], | 70.00th=[28181], 80.00th=[28181], 90.00th=[54264], 95.00th=[54264], | 99.00th=[54264], 99.50th=[54264], 99.90th=[54264], 99.95th=[54264], | 99.99th=[54264] bw ( KiB/s): min= 8192, max= 8192, per=63.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=4, BW=4691KiB/s (4804kB/s)(7168KiB/1528msec) clat (msec): min=13, max=1023, avg=182.13, stdev=372.04 lat (msec): min=13, max=1023, avg=182.19, stdev=372.03 clat percentiles (msec): | 1.00th=[ 14], 5.00th=[ 14], 10.00th=[ 14], 20.00th=[ 15], | 30.00th=[ 21], 40.00th=[ 21], 50.00th=[ 56], 60.00th=[ 64], | 70.00th=[ 64], 80.00th=[ 84], 90.00th=[ 1028], 95.00th=[ 1028], | 99.00th=[ 1028], 99.50th=[ 1028], 99.90th=[ 1028], 99.95th=[ 1028], | 99.99th=[ 1028] bw ( KiB/s): min= 2048, max=12288, per=39.38%, avg=7168.00, stdev=7240.77, samples=2 iops : min= 2, max= 12, avg= 7.00, stdev= 7.07, samples=2 lat (msec) : 20=25.00%, 50=33.33%, 100=33.33% cpu : usr=0.00%, sys=6.61%, ctx=58, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=12.7MiB/s (13.3MB/s), 1300KiB/s-4607KiB/s (1332kB/s-4717kB/s), io=20.0MiB (20.0MB), run=1528-1575msec WRITE: bw=17.8MiB/s (18.6MB/s), 3290KiB/s-6502KiB/s (3369kB/s-6658kB/s), io=28.0MiB (29.4MB), run=1528-1575msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=26502: Fri Feb 7 16:27:39 2025 read: IOPS=2, BW=9878KiB/s (10.1MB/s)(12.0MiB/1244msec) clat (msec): min=86, max=923, avg=405.33, stdev=452.87 lat (msec): min=86, max=923, avg=405.34, stdev=452.87 clat percentiles (msec): | 1.00th=[ 87], 5.00th=[ 87], 10.00th=[ 87], 20.00th=[ 87], | 30.00th=[ 87], 40.00th=[ 207], 50.00th=[ 207], 60.00th=[ 207], | 70.00th=[ 927], 80.00th=[ 927], 90.00th=[ 927], 95.00th=[ 927], | 99.00th=[ 927], 99.50th=[ 927], 99.90th=[ 927], 99.95th=[ 927], | 99.99th=[ 927] bw ( KiB/s): min= 8175, max= 8175, per=78.64%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 250=33.33%, 1000=33.33% cpu : usr=0.16%, sys=16.09%, ctx=1985, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=26503: Fri Feb 7 16:27:39 2025 read: IOPS=0, BW=2599KiB/s (2661kB/s)(4096KiB/1576msec) clat (nsec): min=923913k, max=923913k, avg=923912693.00, stdev= 0.00 lat (nsec): min=923915k, max=923915k, avg=923914561.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 927], 5.00th=[ 927], 10.00th=[ 927], 20.00th=[ 927], | 30.00th=[ 927], 40.00th=[ 927], 50.00th=[ 927], 60.00th=[ 927], | 70.00th=[ 927], 80.00th=[ 927], 90.00th=[ 927], 95.00th=[ 927], | 99.00th=[ 927], 99.50th=[ 927], 99.90th=[ 927], 99.95th=[ 927], | 99.99th=[ 927] bw ( KiB/s): min= 8192, max= 8192, per=78.81%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=1, BW=5198KiB/s (5323kB/s)(8192KiB/1576msec) clat (msec): min=297, max=321, avg=309.62, stdev=17.26 lat (msec): min=297, max=322, avg=309.86, stdev=17.23 clat percentiles (msec): | 1.00th=[ 296], 5.00th=[ 296], 10.00th=[ 296], 20.00th=[ 296], | 30.00th=[ 296], 40.00th=[ 296], 50.00th=[ 296], 60.00th=[ 321], | 70.00th=[ 321], 80.00th=[ 321], 90.00th=[ 321], 95.00th=[ 321], | 99.00th=[ 321], 99.50th=[ 321], 99.90th=[ 321], 99.95th=[ 321], | 99.99th=[ 321] bw ( KiB/s): min= 8192, max= 8192, per=40.23%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 500=66.67%, 1000=33.33% cpu : usr=0.06%, sys=17.21%, ctx=438, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=26504: Fri Feb 7 16:27:39 2025 write: IOPS=1, BW=7637KiB/s (7820kB/s)(12.0MiB/1609msec) clat (msec): min=188, max=1015, avg=531.03, stdev=431.49 lat (msec): min=188, max=1015, avg=531.27, stdev=431.52 clat percentiles (msec): | 1.00th=[ 188], 5.00th=[ 188], 10.00th=[ 188], 20.00th=[ 188], | 30.00th=[ 188], 40.00th=[ 388], 50.00th=[ 388], 60.00th=[ 388], | 70.00th=[ 1020], 80.00th=[ 1020], 90.00th=[ 1020], 95.00th=[ 1020], | 99.00th=[ 1020], 99.50th=[ 1020], 99.90th=[ 1020], 99.95th=[ 1020], | 99.99th=[ 1020] bw ( KiB/s): min=16351, max=16351, per=80.29%, avg=16351.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=33.33% cpu : usr=0.06%, sys=18.47%, ctx=89, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=26505: Fri Feb 7 16:27:39 2025 write: IOPS=1, BW=8187KiB/s (8383kB/s)(12.0MiB/1501msec) clat (msec): min=152, max=984, avg=494.05, stdev=435.72 lat (msec): min=152, max=984, avg=494.28, stdev=435.62 clat percentiles (msec): | 1.00th=[ 153], 5.00th=[ 153], 10.00th=[ 153], 20.00th=[ 153], | 30.00th=[ 153], 40.00th=[ 347], 50.00th=[ 347], 60.00th=[ 347], | 70.00th=[ 986], 80.00th=[ 986], 90.00th=[ 986], 95.00th=[ 986], | 99.00th=[ 986], 99.50th=[ 986], 99.90th=[ 986], 99.95th=[ 986], | 99.99th=[ 986] lat (msec) : 250=33.33%, 500=33.33%, 1000=33.33% cpu : usr=0.00%, sys=21.53%, ctx=58, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=10.2MiB/s (10.6MB/s), 2599KiB/s-9878KiB/s (2661kB/s-10.1MB/s), io=16.0MiB (16.8MB), run=1244-1576msec WRITE: bw=19.9MiB/s (20.9MB/s), 5198KiB/s-8187KiB/s (5323kB/s-8383kB/s), io=32.0MiB (33.6MB), run=1501-1609msec rand-rw: (groupid=0, jobs=1): err= 0: pid=26498: Fri Feb 7 16:27:40 2025 read: IOPS=2, BW=10.1MiB/s (10.6MB/s)(12.0MiB/1184msec) clat (msec): min=47, max=1016, avg=392.60, stdev=541.44 lat (msec): min=47, max=1016, avg=392.60, stdev=541.44 clat percentiles (msec): | 1.00th=[ 48], 5.00th=[ 48], 10.00th=[ 48], 20.00th=[ 48], | 30.00th=[ 48], 40.00th=[ 114], 50.00th=[ 114], 60.00th=[ 114], | 70.00th=[ 1020], 80.00th=[ 1020], 90.00th=[ 1020], 95.00th=[ 1020], | 99.00th=[ 1020], 99.50th=[ 1020], 99.90th=[ 1020], 99.95th=[ 1020], | 99.99th=[ 1020] bw ( KiB/s): min=16384, max=16384, per=100.00%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 50=33.33%, 250=33.33% cpu : usr=0.00%, sys=4.31%, ctx=43, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=26499: Fri Feb 7 16:27:40 2025 read: IOPS=0, BW=2355KiB/s (2412kB/s)(4096KiB/1739msec) clat (nsec): min=1075.5M, max=1075.5M, avg=1075516043.00, stdev= 0.00 lat (nsec): min=1075.5M, max=1075.5M, avg=1075516850.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 1083], 5.00th=[ 1083], 10.00th=[ 1083], 20.00th=[ 1083], | 30.00th=[ 1083], 40.00th=[ 1083], 50.00th=[ 1083], 60.00th=[ 1083], | 70.00th=[ 1083], 80.00th=[ 1083], 90.00th=[ 1083], 95.00th=[ 1083], | 99.00th=[ 1083], 99.50th=[ 1083], 99.90th=[ 1083], 99.95th=[ 1083], | 99.99th=[ 1083] bw ( KiB/s): min= 5704, max= 5704, per=60.55%, avg=5704.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=1, BW=4711KiB/s (4824kB/s)(8192KiB/1739msec) clat (msec): min=257, max=383, avg=320.91, stdev=89.10 lat (msec): min=258, max=384, avg=321.19, stdev=89.07 clat percentiles (msec): | 1.00th=[ 257], 5.00th=[ 257], 10.00th=[ 257], 20.00th=[ 257], | 30.00th=[ 257], 40.00th=[ 257], 50.00th=[ 257], 60.00th=[ 384], | 70.00th=[ 384], 80.00th=[ 384], 90.00th=[ 384], 95.00th=[ 384], | 99.00th=[ 384], 99.50th=[ 384], 99.90th=[ 384], 99.95th=[ 384], | 99.99th=[ 384] bw ( KiB/s): min= 5704, max= 5704, per=30.81%, avg=5704.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 500=66.67% cpu : usr=0.06%, sys=7.13%, ctx=288, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=26500: Fri Feb 7 16:27:40 2025 write: IOPS=1, BW=6942KiB/s (7109kB/s)(12.0MiB/1770msec) clat (msec): min=114, max=1165, avg=583.03, stdev=535.02 lat (msec): min=114, max=1165, avg=583.31, stdev=534.85 clat percentiles (msec): | 1.00th=[ 114], 5.00th=[ 114], 10.00th=[ 114], 20.00th=[ 114], | 30.00th=[ 114], 40.00th=[ 468], 50.00th=[ 468], 60.00th=[ 468], | 70.00th=[ 1167], 80.00th=[ 1167], 90.00th=[ 1167], 95.00th=[ 1167], | 99.00th=[ 1167], 99.50th=[ 1167], 99.90th=[ 1167], 99.95th=[ 1167], | 99.99th=[ 1167] bw ( KiB/s): min=11457, max=11457, per=61.89%, avg=11457.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=33.33% cpu : usr=0.06%, sys=10.06%, ctx=48, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=26501: Fri Feb 7 16:27:40 2025 write: IOPS=2, BW=9548KiB/s (9777kB/s)(12.0MiB/1287msec) clat (msec): min=49, max=980, avg=422.52, stdev=492.61 lat (msec): min=49, max=981, avg=422.91, stdev=492.60 clat percentiles (msec): | 1.00th=[ 50], 5.00th=[ 50], 10.00th=[ 50], 20.00th=[ 50], | 30.00th=[ 50], 40.00th=[ 236], 50.00th=[ 236], 60.00th=[ 236], | 70.00th=[ 978], 80.00th=[ 978], 90.00th=[ 978], 95.00th=[ 978], | 99.00th=[ 978], 99.50th=[ 978], 99.90th=[ 978], 99.95th=[ 978], | 99.99th=[ 978] bw ( KiB/s): min= 8192, max= 8192, per=44.25%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 50=33.33%, 250=33.33%, 1000=33.33% cpu : usr=0.08%, sys=4.28%, ctx=32, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=9422KiB/s (9648kB/s), 2355KiB/s-10.1MiB/s (2412kB/s-10.6MB/s), io=16.0MiB (16.8MB), run=1184-1739msec WRITE: bw=18.1MiB/s (18.0MB/s), 4711KiB/s-9548KiB/s (4824kB/s-9777kB/s), io=32.0MiB (33.6MB), run=1287-1770msec