== sanity test 398b: DIO and buffer IO race ============== 03:10:01 (1715584201) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.28619 s, 39.1 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=17102: Mon May 13 03:10:15 2024 read: IOPS=142, BW=569KiB/s (582kB/s)(5992KiB/10538msec) clat (usec): min=43, max=156162, avg=3973.79, stdev=4536.26 lat (usec): min=44, max=156162, avg=3974.41, stdev=4536.28 clat percentiles (usec): | 1.00th=[ 55], 5.00th=[ 112], 10.00th=[ 1827], 20.00th=[ 2245], | 30.00th=[ 2606], 40.00th=[ 2966], 50.00th=[ 3490], 60.00th=[ 4015], | 70.00th=[ 4621], 80.00th=[ 5407], 90.00th=[ 6456], 95.00th=[ 7570], | 99.00th=[ 12649], 99.50th=[ 13566], 99.90th=[ 22938], 99.95th=[156238], | 99.99th=[156238] bw ( KiB/s): min= 328, max= 840, per=25.28%, avg=567.24, stdev=111.40, samples=21 iops : min= 82, max= 210, avg=141.81, stdev=27.85, samples=21 write: IOPS=149, BW=597KiB/s (612kB/s)(6296KiB/10538msec) clat (usec): min=363, max=19914, avg=2873.51, stdev=2568.77 lat (usec): min=363, max=19915, avg=2874.20, stdev=2568.71 clat percentiles (usec): | 1.00th=[ 396], 5.00th=[ 424], 10.00th=[ 465], 20.00th=[ 545], | 30.00th=[ 693], 40.00th=[ 1598], 50.00th=[ 2409], 60.00th=[ 2900], | 70.00th=[ 3752], 80.00th=[ 4883], 90.00th=[ 6325], 95.00th=[ 7635], | 99.00th=[10814], 99.50th=[13173], 99.90th=[16909], 99.95th=[19792], | 99.99th=[19792] bw ( KiB/s): min= 384, max= 912, per=25.39%, avg=596.57, stdev=117.15, samples=21 iops : min= 96, max= 228, avg=149.14, stdev=29.29, samples=21 lat (usec) : 50=0.29%, 100=2.05%, 250=0.13%, 500=6.84%, 750=9.64% lat (usec) : 1000=2.38% lat (msec) : 2=7.68%, 4=37.11%, 10=31.90%, 20=1.92%, 50=0.03% lat (msec) : 250=0.03% cpu : usr=0.23%, sys=22.79%, ctx=6055, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17103: Mon May 13 03:10:15 2024 read: IOPS=139, BW=556KiB/s (570kB/s)(5940KiB/10677msec) clat (usec): min=37, max=22290, avg=3888.81, stdev=2250.36 lat (usec): min=37, max=22290, avg=3889.34, stdev=2250.38 clat percentiles (usec): | 1.00th=[ 52], 5.00th=[ 1467], 10.00th=[ 1844], 20.00th=[ 2180], | 30.00th=[ 2573], 40.00th=[ 2933], 50.00th=[ 3425], 60.00th=[ 4015], | 70.00th=[ 4621], 80.00th=[ 5342], 90.00th=[ 6521], 95.00th=[ 7635], | 99.00th=[11600], 99.50th=[13566], 99.90th=[21103], 99.95th=[22414], | 99.99th=[22414] bw ( KiB/s): min= 360, max= 792, per=24.56%, avg=551.10, stdev=97.30, samples=21 iops : min= 90, max= 198, avg=137.71, stdev=24.37, samples=21 write: IOPS=148, BW=595KiB/s (609kB/s)(6348KiB/10677msec) clat (usec): min=371, max=153980, avg=3050.87, stdev=4597.27 lat (usec): min=371, max=153981, avg=3051.46, stdev=4597.28 clat percentiles (usec): | 1.00th=[ 400], 5.00th=[ 437], 10.00th=[ 482], 20.00th=[ 553], | 30.00th=[ 701], 40.00th=[ 1844], 50.00th=[ 2442], 60.00th=[ 3032], | 70.00th=[ 3785], 80.00th=[ 5014], 90.00th=[ 6652], 95.00th=[ 8029], | 99.00th=[ 11338], 99.50th=[ 12518], 99.90th=[ 19530], 99.95th=[154141], | 99.99th=[154141] bw ( KiB/s): min= 392, max= 864, per=24.73%, avg=581.14, stdev=100.88, samples=21 iops : min= 98, max= 216, avg=145.24, stdev=25.22, samples=21 lat (usec) : 50=0.42%, 100=1.33%, 250=0.13%, 500=6.02%, 750=10.32% lat (usec) : 1000=1.99% lat (msec) : 2=9.02%, 4=36.91%, 10=31.77%, 20=1.99%, 50=0.07% lat (msec) : 250=0.03% cpu : usr=0.24%, sys=22.70%, ctx=6066, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17104: Mon May 13 03:10:15 2024 read: IOPS=140, BW=564KiB/s (577kB/s)(6028KiB/10697msec) clat (usec): min=36, max=20806, avg=3903.57, stdev=2374.82 lat (usec): min=37, max=20807, avg=3905.30, stdev=2374.23 clat percentiles (usec): | 1.00th=[ 47], 5.00th=[ 108], 10.00th=[ 1778], 20.00th=[ 2147], | 30.00th=[ 2507], 40.00th=[ 2868], 50.00th=[ 3458], 60.00th=[ 4047], | 70.00th=[ 4686], 80.00th=[ 5538], 90.00th=[ 6652], 95.00th=[ 7963], | 99.00th=[11731], 99.50th=[13698], 99.90th=[20579], 99.95th=[20841], | 99.99th=[20841] bw ( KiB/s): min= 240, max= 752, per=24.90%, avg=558.76, stdev=105.35, samples=21 iops : min= 60, max= 188, avg=139.67, stdev=26.33, samples=21 write: IOPS=146, BW=585KiB/s (599kB/s)(6260KiB/10697msec) clat (usec): min=342, max=163861, avg=3033.00, stdev=4858.41 lat (usec): min=342, max=163863, avg=3035.19, stdev=4859.79 clat percentiles (usec): | 1.00th=[ 392], 5.00th=[ 429], 10.00th=[ 478], 20.00th=[ 562], | 30.00th=[ 685], 40.00th=[ 1401], 50.00th=[ 2376], 60.00th=[ 2966], | 70.00th=[ 3818], 80.00th=[ 4948], 90.00th=[ 6718], 95.00th=[ 8160], | 99.00th=[ 11207], 99.50th=[ 12649], 99.90th=[ 18482], 99.95th=[164627], | 99.99th=[164627] bw ( KiB/s): min= 320, max= 736, per=24.60%, avg=578.19, stdev=107.73, samples=21 iops : min= 80, max= 184, avg=144.52, stdev=26.94, samples=21 lat (usec) : 50=0.68%, 100=1.76%, 250=0.07%, 500=6.28%, 750=10.58% lat (usec) : 1000=2.31% lat (msec) : 2=8.69%, 4=35.19%, 10=32.36%, 20=1.99%, 50=0.07% lat (msec) : 250=0.03% cpu : usr=0.15%, sys=22.55%, ctx=5986, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17105: Mon May 13 03:10:15 2024 read: IOPS=141, BW=567KiB/s (581kB/s)(6052KiB/10665msec) clat (usec): min=42, max=17913, avg=3832.30, stdev=2058.46 lat (usec): min=42, max=17914, avg=3832.84, stdev=2058.46 clat percentiles (usec): | 1.00th=[ 56], 5.00th=[ 1483], 10.00th=[ 1844], 20.00th=[ 2212], | 30.00th=[ 2606], 40.00th=[ 3032], 50.00th=[ 3523], 60.00th=[ 4047], | 70.00th=[ 4621], 80.00th=[ 5276], 90.00th=[ 6456], 95.00th=[ 7373], | 99.00th=[10421], 99.50th=[11731], 99.90th=[16188], 99.95th=[17957], | 99.99th=[17957] bw ( KiB/s): min= 304, max= 736, per=25.06%, avg=562.24, stdev=98.35, samples=21 iops : min= 76, max= 184, avg=140.52, stdev=24.62, samples=21 write: IOPS=146, BW=585KiB/s (599kB/s)(6236KiB/10665msec) clat (usec): min=365, max=163344, avg=3081.17, stdev=4866.13 lat (usec): min=365, max=163346, avg=3081.97, stdev=4866.21 clat percentiles (usec): | 1.00th=[ 388], 5.00th=[ 429], 10.00th=[ 478], 20.00th=[ 553], | 30.00th=[ 693], 40.00th=[ 1827], 50.00th=[ 2409], 60.00th=[ 3032], | 70.00th=[ 3884], 80.00th=[ 5080], 90.00th=[ 6587], 95.00th=[ 8094], | 99.00th=[ 11731], 99.50th=[ 13435], 99.90th=[ 20055], 99.95th=[162530], | 99.99th=[162530] bw ( KiB/s): min= 384, max= 824, per=24.62%, avg=578.57, stdev=98.02, samples=21 iops : min= 96, max= 206, avg=144.62, stdev=24.52, samples=21 lat (usec) : 50=0.33%, 100=1.66%, 250=0.23%, 500=6.32%, 750=10.19% lat (usec) : 1000=1.76% lat (msec) : 2=7.85%, 4=36.88%, 10=33.01%, 20=1.76%, 250=0.03% cpu : usr=0.20%, sys=22.53%, ctx=6207, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2245KiB/s (2299kB/s), 556KiB/s-569KiB/s (570kB/s-582kB/s), io=23.4MiB (24.6MB), run=10538-10697msec WRITE: bw=2350KiB/s (2407kB/s), 585KiB/s-597KiB/s (599kB/s-612kB/s), io=24.6MiB (25.7MB), run=10538-10697msec rand-rw: (groupid=0, jobs=1): err= 0: pid=17098: Mon May 13 03:10:22 2024 read: IOPS=85, BW=343KiB/s (351kB/s)(5992KiB/17494msec) clat (usec): min=1433, max=158019, avg=4744.04, stdev=4816.03 lat (usec): min=1433, max=158021, avg=4744.80, stdev=4816.06 clat percentiles (usec): | 1.00th=[ 1631], 5.00th=[ 1844], 10.00th=[ 2073], 20.00th=[ 2474], | 30.00th=[ 3064], 40.00th=[ 3621], 50.00th=[ 3916], 60.00th=[ 4228], | 70.00th=[ 4752], 80.00th=[ 6325], 90.00th=[ 8717], 95.00th=[ 10159], | 99.00th=[ 14222], 99.50th=[ 16057], 99.90th=[ 21627], 99.95th=[158335], | 99.99th=[158335] bw ( KiB/s): min= 200, max= 576, per=25.32%, avg=344.79, stdev=69.68, samples=34 iops : min= 50, max= 144, avg=86.09, stdev=17.42, samples=34 write: IOPS=89, BW=360KiB/s (369kB/s)(6296KiB/17494msec) clat (usec): min=1949, max=22444, avg=6555.18, stdev=2827.39 lat (usec): min=1950, max=22445, avg=6556.07, stdev=2827.50 clat percentiles (usec): | 1.00th=[ 2638], 5.00th=[ 2966], 10.00th=[ 3359], 20.00th=[ 4113], | 30.00th=[ 5014], 40.00th=[ 5604], 50.00th=[ 6259], 60.00th=[ 6718], | 70.00th=[ 7177], 80.00th=[ 8225], 90.00th=[10159], 95.00th=[11863], | 99.00th=[16188], 99.50th=[17695], 99.90th=[22152], 99.95th=[22414], | 99.99th=[22414] bw ( KiB/s): min= 224, max= 608, per=25.37%, avg=361.71, stdev=73.87, samples=34 iops : min= 56, max= 152, avg=90.32, stdev=18.44, samples=34 lat (msec) : 2=4.07%, 4=31.15%, 10=56.61%, 20=7.98%, 50=0.16% lat (msec) : 250=0.03% cpu : usr=0.13%, sys=12.15%, ctx=3642, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17099: Mon May 13 03:10:22 2024 read: IOPS=84, BW=340KiB/s (348kB/s)(5940KiB/17490msec) clat (usec): min=1414, max=157461, avg=4638.78, stdev=4746.85 lat (usec): min=1415, max=157462, avg=4639.54, stdev=4746.87 clat percentiles (usec): | 1.00th=[ 1598], 5.00th=[ 1827], 10.00th=[ 2040], 20.00th=[ 2442], | 30.00th=[ 2966], 40.00th=[ 3490], 50.00th=[ 3851], 60.00th=[ 4146], | 70.00th=[ 4686], 80.00th=[ 6259], 90.00th=[ 8455], 95.00th=[ 10028], | 99.00th=[ 12780], 99.50th=[ 14877], 99.90th=[ 17957], 99.95th=[158335], | 99.99th=[158335] bw ( KiB/s): min= 176, max= 504, per=25.19%, avg=343.15, stdev=73.18, samples=34 iops : min= 44, max= 126, avg=85.68, stdev=18.24, samples=34 write: IOPS=90, BW=363KiB/s (372kB/s)(6348KiB/17490msec) clat (usec): min=2073, max=22756, avg=6636.46, stdev=2924.79 lat (usec): min=2073, max=22757, avg=6637.39, stdev=2924.81 clat percentiles (usec): | 1.00th=[ 2507], 5.00th=[ 3064], 10.00th=[ 3425], 20.00th=[ 4228], | 30.00th=[ 5145], 40.00th=[ 5669], 50.00th=[ 6325], 60.00th=[ 6652], | 70.00th=[ 7242], 80.00th=[ 8356], 90.00th=[10421], 95.00th=[11863], | 99.00th=[17171], 99.50th=[20579], 99.90th=[22414], 99.95th=[22676], | 99.99th=[22676] bw ( KiB/s): min= 216, max= 568, per=25.52%, avg=363.85, stdev=64.87, samples=34 iops : min= 54, max= 142, avg=90.85, stdev=16.21, samples=34 lat (msec) : 2=4.46%, 4=31.09%, 10=56.18%, 20=7.91%, 50=0.33% lat (msec) : 250=0.03% cpu : usr=0.17%, sys=12.13%, ctx=3646, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17100: Mon May 13 03:10:22 2024 read: IOPS=86, BW=344KiB/s (352kB/s)(6028KiB/17523msec) clat (usec): min=1412, max=17195, avg=4577.12, stdev=2643.52 lat (usec): min=1413, max=17195, avg=4577.86, stdev=2643.57 clat percentiles (usec): | 1.00th=[ 1598], 5.00th=[ 1876], 10.00th=[ 2040], 20.00th=[ 2474], | 30.00th=[ 2999], 40.00th=[ 3523], 50.00th=[ 3884], 60.00th=[ 4228], | 70.00th=[ 4752], 80.00th=[ 6194], 90.00th=[ 8586], 95.00th=[10028], | 99.00th=[13698], 99.50th=[14484], 99.90th=[16057], 99.95th=[17171], | 99.99th=[17171] bw ( KiB/s): min= 160, max= 544, per=25.25%, avg=343.91, stdev=75.29, samples=35 iops : min= 40, max= 136, avg=85.91, stdev=18.84, samples=35 write: IOPS=89, BW=357KiB/s (366kB/s)(6260KiB/17523msec) clat (msec): min=2, max=162, avg= 6.74, stdev= 4.86 lat (msec): min=2, max=162, avg= 6.74, stdev= 4.86 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 4], 10.00th=[ 4], 20.00th=[ 5], | 30.00th=[ 6], 40.00th=[ 6], 50.00th=[ 7], 60.00th=[ 7], | 70.00th=[ 8], 80.00th=[ 9], 90.00th=[ 11], 95.00th=[ 12], | 99.00th=[ 17], 99.50th=[ 18], 99.90th=[ 23], 99.95th=[ 163], | 99.99th=[ 163] bw ( KiB/s): min= 200, max= 576, per=25.05%, avg=357.17, stdev=71.82, samples=35 iops : min= 50, max= 144, avg=89.23, stdev=17.96, samples=35 lat (msec) : 2=4.30%, 4=30.96%, 10=56.22%, 20=8.37%, 50=0.13% lat (msec) : 250=0.03% cpu : usr=0.15%, sys=12.09%, ctx=3640, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17101: Mon May 13 03:10:22 2024 read: IOPS=85, BW=343KiB/s (352kB/s)(6052KiB/17629msec) clat (usec): min=1400, max=20593, avg=4559.88, stdev=2577.92 lat (usec): min=1400, max=20594, avg=4560.65, stdev=2577.98 clat percentiles (usec): | 1.00th=[ 1663], 5.00th=[ 1909], 10.00th=[ 2114], 20.00th=[ 2540], | 30.00th=[ 2966], 40.00th=[ 3523], 50.00th=[ 3884], 60.00th=[ 4146], | 70.00th=[ 4686], 80.00th=[ 6194], 90.00th=[ 8586], 95.00th=[ 9896], | 99.00th=[13042], 99.50th=[13829], 99.90th=[14877], 99.95th=[20579], | 99.99th=[20579] bw ( KiB/s): min= 207, max= 552, per=25.20%, avg=343.29, stdev=70.83, samples=35 iops : min= 51, max= 138, avg=85.80, stdev=17.75, samples=35 write: IOPS=88, BW=354KiB/s (362kB/s)(6236KiB/17629msec) clat (msec): min=2, max=163, avg= 6.84, stdev= 4.92 lat (msec): min=2, max=163, avg= 6.84, stdev= 4.92 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 4], 10.00th=[ 4], 20.00th=[ 5], | 30.00th=[ 6], 40.00th=[ 6], 50.00th=[ 7], 60.00th=[ 7], | 70.00th=[ 8], 80.00th=[ 9], 90.00th=[ 11], 95.00th=[ 13], | 99.00th=[ 18], 99.50th=[ 20], 99.90th=[ 23], 99.95th=[ 165], | 99.99th=[ 165] bw ( KiB/s): min= 232, max= 600, per=24.81%, avg=353.80, stdev=67.53, samples=35 iops : min= 58, max= 150, avg=88.43, stdev=16.89, samples=35 lat (msec) : 2=3.78%, 4=31.02%, 10=56.74%, 20=8.24%, 50=0.20% lat (msec) : 250=0.03% cpu : usr=0.16%, sys=12.16%, ctx=3604, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1362KiB/s (1395kB/s), 340KiB/s-344KiB/s (348kB/s-352kB/s), io=23.4MiB (24.6MB), run=17490-17629msec WRITE: bw=1426KiB/s (1460kB/s), 354KiB/s-363KiB/s (362kB/s-372kB/s), io=24.6MiB (25.7MB), run=17490-17629msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=17132: Mon May 13 03:10:28 2024 read: IOPS=72, BW=1160KiB/s (1187kB/s)(5920KiB/5105msec) clat (usec): min=1840, max=43590, avg=5374.51, stdev=4255.59 lat (usec): min=1840, max=43592, avg=5375.19, stdev=4255.56 clat percentiles (usec): | 1.00th=[ 1876], 5.00th=[ 2073], 10.00th=[ 2180], 20.00th=[ 2409], | 30.00th=[ 2671], 40.00th=[ 3032], 50.00th=[ 3523], 60.00th=[ 4490], | 70.00th=[ 6128], 80.00th=[ 8291], 90.00th=[10945], 95.00th=[12911], | 99.00th=[19530], 99.50th=[21627], 99.90th=[43779], 99.95th=[43779], | 99.99th=[43779] bw ( KiB/s): min= 704, max= 1472, per=24.37%, avg=1142.40, stdev=237.58, samples=10 iops : min= 44, max= 92, avg=71.40, stdev=14.85, samples=10 write: IOPS=77, BW=1247KiB/s (1277kB/s)(6368KiB/5105msec) clat (usec): min=2516, max=47525, avg=7812.81, stdev=5259.02 lat (usec): min=2517, max=47527, avg=7814.19, stdev=5259.07 clat percentiles (usec): | 1.00th=[ 2671], 5.00th=[ 3163], 10.00th=[ 3326], 20.00th=[ 3752], | 30.00th=[ 4228], 40.00th=[ 5145], 50.00th=[ 6063], 60.00th=[ 7242], | 70.00th=[ 9110], 80.00th=[11469], 90.00th=[14615], 95.00th=[17171], | 99.00th=[26608], 99.50th=[33817], 99.90th=[47449], 99.95th=[47449], | 99.99th=[47449] bw ( KiB/s): min= 768, max= 1504, per=25.11%, avg=1238.40, stdev=234.21, samples=10 iops : min= 48, max= 94, avg=77.40, stdev=14.64, samples=10 lat (msec) : 2=1.30%, 4=38.80%, 10=39.97%, 20=18.10%, 50=1.82% cpu : usr=0.25%, sys=9.29%, ctx=908, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17133: Mon May 13 03:10:28 2024 read: IOPS=73, BW=1177KiB/s (1205kB/s)(5744KiB/4882msec) clat (usec): min=1662, max=30242, avg=5134.85, stdev=3914.51 lat (usec): min=1662, max=30243, avg=5135.42, stdev=3914.52 clat percentiles (usec): | 1.00th=[ 1876], 5.00th=[ 2057], 10.00th=[ 2212], 20.00th=[ 2474], | 30.00th=[ 2769], 40.00th=[ 3064], 50.00th=[ 3523], 60.00th=[ 4293], | 70.00th=[ 5276], 80.00th=[ 7439], 90.00th=[10290], 95.00th=[13566], | 99.00th=[18220], 99.50th=[23200], 99.90th=[30278], 99.95th=[30278], | 99.99th=[30278] bw ( KiB/s): min= 544, max= 1536, per=25.17%, avg=1180.11, stdev=283.14, samples=9 iops : min= 34, max= 96, avg=73.67, stdev=17.66, samples=9 write: IOPS=83, BW=1340KiB/s (1373kB/s)(6544KiB/4882msec) clat (usec): min=2559, max=91707, avg=7411.89, stdev=6089.27 lat (usec): min=2560, max=91710, avg=7413.36, stdev=6089.40 clat percentiles (usec): | 1.00th=[ 2868], 5.00th=[ 3294], 10.00th=[ 3458], 20.00th=[ 3785], | 30.00th=[ 4047], 40.00th=[ 4686], 50.00th=[ 5735], 60.00th=[ 6652], | 70.00th=[ 8160], 80.00th=[10159], 90.00th=[14091], 95.00th=[17433], | 99.00th=[21103], 99.50th=[22676], 99.90th=[91751], 99.95th=[91751], | 99.99th=[91751] bw ( KiB/s): min= 832, max= 1568, per=26.74%, avg=1318.78, stdev=210.82, samples=9 iops : min= 52, max= 98, avg=82.33, stdev=13.15, samples=9 lat (msec) : 2=1.69%, 4=39.58%, 10=42.45%, 20=15.10%, 50=1.04% lat (msec) : 100=0.13% cpu : usr=0.10%, sys=9.85%, ctx=919, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17134: Mon May 13 03:10:28 2024 read: IOPS=77, BW=1243KiB/s (1272kB/s)(6256KiB/5035msec) clat (usec): min=1730, max=31041, avg=5382.12, stdev=4160.36 lat (usec): min=1730, max=31042, avg=5382.70, stdev=4160.39 clat percentiles (usec): | 1.00th=[ 1860], 5.00th=[ 2114], 10.00th=[ 2212], 20.00th=[ 2507], | 30.00th=[ 2802], 40.00th=[ 3228], 50.00th=[ 3687], 60.00th=[ 4293], | 70.00th=[ 6128], 80.00th=[ 8094], 90.00th=[10552], 95.00th=[13304], | 99.00th=[22938], 99.50th=[23725], 99.90th=[31065], 99.95th=[31065], | 99.99th=[31065] bw ( KiB/s): min= 704, max= 1632, per=26.61%, avg=1247.70, stdev=274.72, samples=10 iops : min= 44, max= 102, avg=77.90, stdev=17.25, samples=10 write: IOPS=74, BW=1198KiB/s (1227kB/s)(6032KiB/5035msec) clat (usec): min=2562, max=55727, avg=7749.49, stdev=5206.51 lat (usec): min=2563, max=55729, avg=7750.82, stdev=5206.47 clat percentiles (usec): | 1.00th=[ 2769], 5.00th=[ 3228], 10.00th=[ 3359], 20.00th=[ 3818], | 30.00th=[ 4228], 40.00th=[ 5014], 50.00th=[ 6063], 60.00th=[ 7177], | 70.00th=[ 9110], 80.00th=[11207], 90.00th=[15008], 95.00th=[17695], | 99.00th=[21627], 99.50th=[23725], 99.90th=[55837], 99.95th=[55837], | 99.99th=[55837] bw ( KiB/s): min= 864, max= 1408, per=24.00%, avg=1183.70, stdev=185.37, samples=10 iops : min= 54, max= 88, avg=73.90, stdev=11.59, samples=10 lat (msec) : 2=1.17%, 4=40.36%, 10=39.45%, 20=17.06%, 50=1.82% lat (msec) : 100=0.13% cpu : usr=0.18%, sys=9.65%, ctx=941, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17135: Mon May 13 03:10:28 2024 read: IOPS=73, BW=1181KiB/s (1209kB/s)(6032KiB/5109msec) clat (usec): min=1817, max=24130, avg=5590.75, stdev=4017.21 lat (usec): min=1818, max=24130, avg=5591.39, stdev=4017.38 clat percentiles (usec): | 1.00th=[ 1909], 5.00th=[ 2024], 10.00th=[ 2245], 20.00th=[ 2540], | 30.00th=[ 2737], 40.00th=[ 3195], 50.00th=[ 3884], 60.00th=[ 5014], | 70.00th=[ 6849], 80.00th=[ 8717], 90.00th=[10945], 95.00th=[14222], | 99.00th=[19792], 99.50th=[20841], 99.90th=[24249], 99.95th=[24249], | 99.99th=[24249] bw ( KiB/s): min= 672, max= 1440, per=25.12%, avg=1177.60, stdev=243.14, samples=10 iops : min= 42, max= 90, avg=73.60, stdev=15.20, samples=10 write: IOPS=76, BW=1225KiB/s (1254kB/s)(6256KiB/5109msec) clat (usec): min=2437, max=93914, avg=7660.73, stdev=6355.72 lat (usec): min=2438, max=93916, avg=7662.06, stdev=6355.70 clat percentiles (usec): | 1.00th=[ 2540], 5.00th=[ 3130], 10.00th=[ 3326], 20.00th=[ 3752], | 30.00th=[ 4080], 40.00th=[ 4883], 50.00th=[ 6194], 60.00th=[ 7111], | 70.00th=[ 8848], 80.00th=[10421], 90.00th=[14091], 95.00th=[17171], | 99.00th=[22938], 99.50th=[33817], 99.90th=[93848], 99.95th=[93848], | 99.99th=[93848] bw ( KiB/s): min= 768, max= 1504, per=24.79%, avg=1222.40, stdev=257.24, samples=10 iops : min= 48, max= 94, avg=76.40, stdev=16.08, samples=10 lat (msec) : 2=1.82%, 4=37.50%, 10=42.71%, 20=16.41%, 50=1.43% lat (msec) : 100=0.13% cpu : usr=0.06%, sys=9.44%, ctx=940, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=4688KiB/s (4801kB/s), 1160KiB/s-1243KiB/s (1187kB/s-1272kB/s), io=23.4MiB (24.5MB), run=4882-5109msec WRITE: bw=4932KiB/s (5051kB/s), 1198KiB/s-1340KiB/s (1227kB/s-1373kB/s), io=24.6MiB (25.8MB), run=4882-5109msec rand-rw: (groupid=0, jobs=1): err= 0: pid=17137: Mon May 13 03:10:28 2024 read: IOPS=69, BW=1116KiB/s (1143kB/s)(5920KiB/5304msec) clat (usec): min=51, max=47774, avg=10645.44, stdev=4815.67 lat (usec): min=51, max=47776, avg=10646.01, stdev=4815.71 clat percentiles (usec): | 1.00th=[ 59], 5.00th=[ 77], 10.00th=[ 7177], 20.00th=[ 8291], | 30.00th=[ 8979], 40.00th=[ 9765], 50.00th=[10421], 60.00th=[11207], | 70.00th=[12125], 80.00th=[13173], 90.00th=[15270], 95.00th=[17433], | 99.00th=[22414], 99.50th=[39584], 99.90th=[47973], 99.95th=[47973], | 99.99th=[47973] bw ( KiB/s): min= 640, max= 1248, per=23.52%, avg=1049.10, stdev=186.53, samples=10 iops : min= 40, max= 78, avg=65.40, stdev=11.69, samples=10 write: IOPS=75, BW=1201KiB/s (1229kB/s)(6368KiB/5304msec) clat (usec): min=591, max=20704, avg=3412.29, stdev=3407.13 lat (usec): min=592, max=20705, avg=3413.23, stdev=3407.15 clat percentiles (usec): | 1.00th=[ 660], 5.00th=[ 750], 10.00th=[ 783], 20.00th=[ 840], | 30.00th=[ 1037], 40.00th=[ 2114], 50.00th=[ 2474], 60.00th=[ 2933], | 70.00th=[ 3621], 80.00th=[ 4686], 90.00th=[ 7308], 95.00th=[11207], | 99.00th=[16909], 99.50th=[19006], 99.90th=[20579], 99.95th=[20579], | 99.99th=[20579] bw ( KiB/s): min= 736, max= 1660, per=24.61%, avg=1154.50, stdev=278.20, samples=10 iops : min= 46, max= 103, avg=72.00, stdev=17.20, samples=10 lat (usec) : 100=3.12%, 250=0.26%, 750=2.47%, 1000=12.50% lat (msec) : 2=4.69%, 4=18.62%, 10=27.34%, 20=29.95%, 50=1.04% cpu : usr=0.04%, sys=20.71%, ctx=2550, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17138: Mon May 13 03:10:28 2024 read: IOPS=66, BW=1072KiB/s (1098kB/s)(5744KiB/5359msec) clat (usec): min=53, max=28445, avg=10416.61, stdev=4198.55 lat (usec): min=53, max=28445, avg=10417.18, stdev=4198.59 clat percentiles (usec): | 1.00th=[ 58], 5.00th=[ 76], 10.00th=[ 7111], 20.00th=[ 8094], | 30.00th=[ 8979], 40.00th=[ 9896], 50.00th=[10552], 60.00th=[11338], | 70.00th=[12125], 80.00th=[13042], 90.00th=[14746], 95.00th=[17433], | 99.00th=[21103], 99.50th=[23987], 99.90th=[28443], 99.95th=[28443], | 99.99th=[28443] bw ( KiB/s): min= 640, max= 1184, per=22.60%, avg=1008.00, stdev=154.76, samples=10 iops : min= 40, max= 74, avg=63.00, stdev= 9.67, samples=10 write: IOPS=76, BW=1221KiB/s (1250kB/s)(6544KiB/5359msec) clat (usec): min=584, max=86380, avg=3940.42, stdev=5540.39 lat (usec): min=585, max=86381, avg=3941.94, stdev=5541.13 clat percentiles (usec): | 1.00th=[ 644], 5.00th=[ 725], 10.00th=[ 783], 20.00th=[ 865], | 30.00th=[ 1156], 40.00th=[ 2180], 50.00th=[ 2573], 60.00th=[ 2999], | 70.00th=[ 3949], 80.00th=[ 5407], 90.00th=[ 9372], 95.00th=[12911], | 99.00th=[17433], 99.50th=[19530], 99.90th=[86508], 99.95th=[86508], | 99.99th=[86508] bw ( KiB/s): min= 896, max= 1664, per=24.28%, avg=1139.20, stdev=271.61, samples=10 iops : min= 56, max= 104, avg=71.20, stdev=16.98, samples=10 lat (usec) : 100=2.99%, 250=0.39%, 750=3.39%, 1000=9.37% lat (msec) : 2=6.64%, 4=18.36%, 10=27.08%, 20=30.60%, 50=1.04% lat (msec) : 100=0.13% cpu : usr=0.22%, sys=20.16%, ctx=2515, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17139: Mon May 13 03:10:28 2024 read: IOPS=73, BW=1168KiB/s (1196kB/s)(6256KiB/5355msec) clat (usec): min=50, max=31624, avg=10180.74, stdev=4219.24 lat (usec): min=50, max=31625, avg=10181.34, stdev=4219.28 clat percentiles (usec): | 1.00th=[ 59], 5.00th=[ 72], 10.00th=[ 6718], 20.00th=[ 8225], | 30.00th=[ 8979], 40.00th=[ 9634], 50.00th=[10421], 60.00th=[11076], | 70.00th=[11994], 80.00th=[12649], 90.00th=[14353], 95.00th=[15926], | 99.00th=[21365], 99.50th=[24773], 99.90th=[31589], 99.95th=[31589], | 99.99th=[31589] bw ( KiB/s): min= 640, max= 1472, per=24.68%, avg=1100.80, stdev=222.32, samples=10 iops : min= 40, max= 92, avg=68.80, stdev=13.89, samples=10 write: IOPS=70, BW=1126KiB/s (1153kB/s)(6032KiB/5355msec) clat (usec): min=616, max=89125, avg=3627.55, stdev=5629.89 lat (usec): min=617, max=89127, avg=3628.47, stdev=5629.93 clat percentiles (usec): | 1.00th=[ 660], 5.00th=[ 766], 10.00th=[ 791], 20.00th=[ 881], | 30.00th=[ 1188], 40.00th=[ 2057], 50.00th=[ 2442], 60.00th=[ 2966], | 70.00th=[ 3556], 80.00th=[ 4752], 90.00th=[ 6849], 95.00th=[11994], | 99.00th=[18482], 99.50th=[24511], 99.90th=[88605], 99.95th=[88605], | 99.99th=[88605] bw ( KiB/s): min= 832, max= 1312, per=22.78%, avg=1068.80, stdev=161.91, samples=10 iops : min= 52, max= 82, avg=66.80, stdev=10.12, samples=10 lat (usec) : 100=3.91%, 250=0.39%, 750=1.69%, 1000=10.81% lat (msec) : 2=6.25%, 4=17.71%, 10=27.21%, 20=30.86%, 50=1.04% lat (msec) : 100=0.13% cpu : usr=0.09%, sys=19.72%, ctx=2485, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17140: Mon May 13 03:10:28 2024 read: IOPS=70, BW=1123KiB/s (1150kB/s)(6032KiB/5370msec) clat (usec): min=49, max=30121, avg=10283.94, stdev=4182.71 lat (usec): min=49, max=30122, avg=10284.50, stdev=4182.73 clat percentiles (usec): | 1.00th=[ 59], 5.00th=[ 73], 10.00th=[ 6980], 20.00th=[ 8356], | 30.00th=[ 8979], 40.00th=[ 9634], 50.00th=[10290], 60.00th=[11076], | 70.00th=[11863], 80.00th=[12780], 90.00th=[14877], 95.00th=[16319], | 99.00th=[21890], 99.50th=[24249], 99.90th=[30016], 99.95th=[30016], | 99.99th=[30016] bw ( KiB/s): min= 576, max= 1248, per=23.46%, avg=1046.20, stdev=199.07, samples=10 iops : min= 36, max= 78, avg=65.30, stdev=12.47, samples=10 write: IOPS=72, BW=1165KiB/s (1193kB/s)(6256KiB/5370msec) clat (usec): min=596, max=89145, avg=3798.90, stdev=5782.25 lat (usec): min=597, max=89146, avg=3799.92, stdev=5782.29 clat percentiles (usec): | 1.00th=[ 685], 5.00th=[ 758], 10.00th=[ 783], 20.00th=[ 857], | 30.00th=[ 1172], 40.00th=[ 2212], 50.00th=[ 2540], 60.00th=[ 2966], | 70.00th=[ 3589], 80.00th=[ 4752], 90.00th=[ 7373], 95.00th=[12518], | 99.00th=[20055], 99.50th=[29754], 99.90th=[89654], 99.95th=[89654], | 99.99th=[89654] bw ( KiB/s): min= 672, max= 1440, per=23.39%, avg=1097.30, stdev=202.94, samples=10 iops : min= 42, max= 90, avg=68.50, stdev=12.68, samples=10 lat (usec) : 50=0.13%, 100=3.12%, 250=0.52%, 750=1.82%, 1000=10.81% lat (msec) : 2=5.34%, 4=19.53%, 10=28.12%, 20=29.04%, 50=1.43% lat (msec) : 100=0.13% cpu : usr=0.06%, sys=20.21%, ctx=2599, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=4460KiB/s (4567kB/s), 1072KiB/s-1168KiB/s (1098kB/s-1196kB/s), io=23.4MiB (24.5MB), run=5304-5370msec WRITE: bw=4693KiB/s (4805kB/s), 1126KiB/s-1221KiB/s (1153kB/s-1250kB/s), io=24.6MiB (25.8MB), run=5304-5370msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=17150: Mon May 13 03:10:33 2024 read: IOPS=17, BW=17.2MiB/s (18.1MB/s)(7168KiB/406msec) clat (usec): min=20346, max=55458, avg=36911.64, stdev=14091.80 lat (usec): min=20347, max=55458, avg=36912.52, stdev=14091.87 clat percentiles (usec): | 1.00th=[20317], 5.00th=[20317], 10.00th=[20317], 20.00th=[24249], | 30.00th=[27395], 40.00th=[27395], 50.00th=[34866], 60.00th=[42206], | 70.00th=[42206], 80.00th=[53740], 90.00th=[55313], 95.00th=[55313], | 99.00th=[55313], 99.50th=[55313], 99.90th=[55313], 99.95th=[55313], | 99.99th=[55313] write: IOPS=12, BW=12.3MiB/s (12.9MB/s)(5120KiB/406msec) clat (usec): min=21724, max=31734, avg=28753.22, stdev=4088.90 lat (usec): min=21794, max=31757, avg=28807.40, stdev=4081.30 clat percentiles (usec): | 1.00th=[21627], 5.00th=[21627], 10.00th=[21627], 20.00th=[21627], | 30.00th=[28967], 40.00th=[28967], 50.00th=[29754], 60.00th=[29754], | 70.00th=[31589], 80.00th=[31589], 90.00th=[31851], 95.00th=[31851], | 99.00th=[31851], 99.50th=[31851], 99.90th=[31851], 99.95th=[31851], | 99.99th=[31851] lat (msec) : 50=83.33%, 100=16.67% cpu : usr=0.25%, sys=29.38%, ctx=51, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17151: Mon May 13 03:10:33 2024 read: IOPS=0, BW=1007KiB/s (1031kB/s)(2048KiB/2034msec) clat (msec): min=103, max=834, avg=469.15, stdev=516.60 lat (msec): min=103, max=834, avg=469.15, stdev=516.60 clat percentiles (msec): | 1.00th=[ 105], 5.00th=[ 105], 10.00th=[ 105], 20.00th=[ 105], | 30.00th=[ 105], 40.00th=[ 105], 50.00th=[ 105], 60.00th=[ 835], | 70.00th=[ 835], 80.00th=[ 835], 90.00th=[ 835], 95.00th=[ 835], | 99.00th=[ 835], 99.50th=[ 835], 99.90th=[ 835], 99.95th=[ 835], | 99.99th=[ 835] bw ( KiB/s): min= 2048, max= 2048, per=40.69%, avg=2048.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 write: IOPS=4, BW=5034KiB/s (5155kB/s)(10.0MiB/2034msec) clat (msec): min=28, max=644, avg=109.34, stdev=188.36 lat (msec): min=28, max=644, avg=109.40, stdev=188.35 clat percentiles (msec): | 1.00th=[ 29], 5.00th=[ 29], 10.00th=[ 29], 20.00th=[ 36], | 30.00th=[ 42], 40.00th=[ 46], 50.00th=[ 47], 60.00th=[ 54], | 70.00th=[ 67], 80.00th=[ 67], 90.00th=[ 71], 95.00th=[ 642], | 99.00th=[ 642], 99.50th=[ 642], 99.90th=[ 642], 99.95th=[ 642], | 99.99th=[ 642] bw ( KiB/s): min= 2035, max=10240, per=87.14%, avg=6139.67, stdev=4102.50, samples=3 iops : min= 1, max= 10, avg= 5.67, stdev= 4.51, samples=3 lat (msec) : 50=41.67%, 100=33.33%, 250=8.33%, 750=8.33%, 1000=8.33% cpu : usr=0.00%, sys=7.92%, ctx=73, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17152: Mon May 13 03:10:33 2024 read: IOPS=1, BW=1510KiB/s (1546kB/s)(6144KiB/4069msec) clat (msec): min=19, max=745, avg=267.00, stdev=341.67 lat (msec): min=19, max=745, avg=267.00, stdev=341.67 clat percentiles (msec): | 1.00th=[ 20], 5.00th=[ 20], 10.00th=[ 20], 20.00th=[ 29], | 30.00th=[ 29], 40.00th=[ 35], 50.00th=[ 35], 60.00th=[ 110], | 70.00th=[ 667], 80.00th=[ 667], 90.00th=[ 743], 95.00th=[ 743], | 99.00th=[ 743], 99.50th=[ 743], 99.90th=[ 743], 99.95th=[ 743], | 99.99th=[ 743] bw ( KiB/s): min= 1872, max= 6144, per=65.49%, avg=3296.00, stdev=2466.44, samples=3 iops : min= 1, max= 6, avg= 2.67, stdev= 2.89, samples=3 write: IOPS=1, BW=1510KiB/s (1546kB/s)(6144KiB/4069msec) clat (msec): min=42, max=1205, avg=408.98, stdev=535.90 lat (msec): min=42, max=1205, avg=409.06, stdev=535.92 clat percentiles (msec): | 1.00th=[ 43], 5.00th=[ 43], 10.00th=[ 43], 20.00th=[ 48], | 30.00th=[ 48], 40.00th=[ 86], 50.00th=[ 86], 60.00th=[ 91], | 70.00th=[ 986], 80.00th=[ 986], 90.00th=[ 1200], 95.00th=[ 1200], | 99.00th=[ 1200], 99.50th=[ 1200], 99.90th=[ 1200], 99.95th=[ 1200], | 99.99th=[ 1200] bw ( KiB/s): min= 3744, max= 4096, per=56.47%, avg=3978.67, stdev=203.23, samples=3 iops : min= 3, max= 4, avg= 3.67, stdev= 0.58, samples=3 lat (msec) : 20=8.33%, 50=33.33%, 100=16.67%, 250=8.33%, 750=16.67% lat (msec) : 1000=8.33% cpu : usr=0.02%, sys=5.19%, ctx=63, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17154: Mon May 13 03:10:33 2024 read: IOPS=2, BW=2509KiB/s (2569kB/s)(5120KiB/2041msec) clat (msec): min=17, max=510, avg=161.55, stdev=203.53 lat (msec): min=17, max=510, avg=161.55, stdev=203.53 clat percentiles (msec): | 1.00th=[ 18], 5.00th=[ 18], 10.00th=[ 18], 20.00th=[ 18], | 30.00th=[ 24], 40.00th=[ 24], 50.00th=[ 99], 60.00th=[ 99], | 70.00th=[ 161], 80.00th=[ 161], 90.00th=[ 510], 95.00th=[ 510], | 99.00th=[ 510], 99.50th=[ 510], 99.90th=[ 510], 99.95th=[ 510], | 99.99th=[ 510] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=3, BW=3512KiB/s (3596kB/s)(7168KiB/2041msec) clat (msec): min=21, max=905, avg=171.76, stdev=324.71 lat (msec): min=21, max=905, avg=171.82, stdev=324.70 clat percentiles (msec): | 1.00th=[ 22], 5.00th=[ 22], 10.00th=[ 22], 20.00th=[ 23], | 30.00th=[ 31], 40.00th=[ 31], 50.00th=[ 68], 60.00th=[ 71], | 70.00th=[ 71], 80.00th=[ 85], 90.00th=[ 902], 95.00th=[ 902], | 99.00th=[ 902], 99.50th=[ 902], 99.90th=[ 902], 99.95th=[ 902], | 99.99th=[ 902] bw ( KiB/s): min= 2043, max= 8192, per=67.75%, avg=4774.00, stdev=3131.54, samples=3 iops : min= 1, max= 8, avg= 4.00, stdev= 3.61, samples=3 lat (msec) : 20=8.33%, 50=33.33%, 100=33.33%, 250=8.33%, 750=8.33% lat (msec) : 1000=8.33% cpu : usr=0.10%, sys=7.84%, ctx=68, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=5033KiB/s (5154kB/s), 1007KiB/s-17.2MiB/s (1031kB/s-18.1MB/s), io=20.0MiB (20.0MB), run=406-4069msec WRITE: bw=7046KiB/s (7216kB/s), 1510KiB/s-12.3MiB/s (1546kB/s-12.9MB/s), io=28.0MiB (29.4MB), run=406-4069msec rand-rw: (groupid=0, jobs=1): err= 0: pid=17153: Mon May 13 03:10:35 2024 read: IOPS=1, BW=1218KiB/s (1247kB/s)(7168KiB/5886msec) clat (msec): min=58, max=1094, avg=789.30, stdev=332.93 lat (msec): min=58, max=1094, avg=789.30, stdev=332.93 clat percentiles (msec): | 1.00th=[ 59], 5.00th=[ 59], 10.00th=[ 59], 20.00th=[ 852], | 30.00th=[ 860], 40.00th=[ 860], 50.00th=[ 860], 60.00th=[ 885], | 70.00th=[ 885], 80.00th=[ 911], 90.00th=[ 1099], 95.00th=[ 1099], | 99.00th=[ 1099], 99.50th=[ 1099], 99.90th=[ 1099], 99.95th=[ 1099], | 99.99th=[ 1099] bw ( KiB/s): min= 1984, max= 2048, per=57.64%, avg=2005.33, stdev=33.05, samples=6 iops : min= 1, max= 2, avg= 1.33, stdev= 0.52, samples=6 write: IOPS=0, BW=870KiB/s (891kB/s)(5120KiB/5886msec) clat (msec): min=25, max=184, avg=71.58, stdev=64.30 lat (msec): min=25, max=184, avg=71.67, stdev=64.30 clat percentiles (msec): | 1.00th=[ 26], 5.00th=[ 26], 10.00th=[ 26], 20.00th=[ 26], | 30.00th=[ 41], 40.00th=[ 41], 50.00th=[ 46], 60.00th=[ 46], | 70.00th=[ 63], 80.00th=[ 63], 90.00th=[ 184], 95.00th=[ 184], | 99.00th=[ 184], 99.50th=[ 184], 99.90th=[ 184], 99.95th=[ 184], | 99.99th=[ 184] bw ( KiB/s): min= 1984, max= 3968, per=50.91%, avg=2480.00, stdev=992.00, samples=4 iops : min= 1, max= 3, avg= 1.50, stdev= 1.00, samples=4 lat (msec) : 50=25.00%, 100=16.67%, 250=8.33%, 1000=41.67% cpu : usr=0.02%, sys=17.04%, ctx=1848, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17155: Mon May 13 03:10:35 2024 read: IOPS=1, BW=1050KiB/s (1075kB/s)(2048KiB/1950msec) clat (msec): min=96, max=669, avg=383.24, stdev=404.95 lat (msec): min=96, max=669, avg=383.24, stdev=404.95 clat percentiles (msec): | 1.00th=[ 97], 5.00th=[ 97], 10.00th=[ 97], 20.00th=[ 97], | 30.00th=[ 97], 40.00th=[ 97], 50.00th=[ 97], 60.00th=[ 667], | 70.00th=[ 667], 80.00th=[ 667], 90.00th=[ 667], 95.00th=[ 667], | 99.00th=[ 667], 99.50th=[ 667], 99.90th=[ 667], 99.95th=[ 667], | 99.99th=[ 667] bw ( KiB/s): min= 2048, max= 2048, per=58.87%, avg=2048.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 write: IOPS=5, BW=5251KiB/s (5377kB/s)(10.0MiB/1950msec) clat (msec): min=17, max=848, avg=117.77, stdev=256.91 lat (msec): min=17, max=848, avg=117.80, stdev=256.91 clat percentiles (msec): | 1.00th=[ 18], 5.00th=[ 18], 10.00th=[ 18], 20.00th=[ 27], | 30.00th=[ 34], 40.00th=[ 35], 50.00th=[ 39], 60.00th=[ 42], | 70.00th=[ 44], 80.00th=[ 44], 90.00th=[ 52], 95.00th=[ 852], | 99.00th=[ 852], 99.50th=[ 852], 99.90th=[ 852], 99.95th=[ 852], | 99.99th=[ 852] bw ( KiB/s): min= 2048, max=10240, per=100.00%, avg=5461.33, stdev=4263.25, samples=3 iops : min= 2, max= 10, avg= 5.33, stdev= 4.16, samples=3 lat (msec) : 20=8.33%, 50=58.33%, 100=16.67%, 750=8.33%, 1000=8.33% cpu : usr=0.00%, sys=19.19%, ctx=300, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17156: Mon May 13 03:10:35 2024 read: IOPS=1, BW=1532KiB/s (1569kB/s)(6144KiB/4010msec) clat (msec): min=3, max=1098, avg=457.03, stdev=474.27 lat (msec): min=3, max=1098, avg=457.03, stdev=474.26 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 4], 10.00th=[ 4], 20.00th=[ 36], | 30.00th=[ 36], 40.00th=[ 95], 50.00th=[ 95], 60.00th=[ 651], | 70.00th=[ 860], 80.00th=[ 860], 90.00th=[ 1099], 95.00th=[ 1099], | 99.00th=[ 1099], 99.50th=[ 1099], 99.90th=[ 1099], 99.95th=[ 1099], | 99.99th=[ 1099] bw ( KiB/s): min= 1973, max= 6144, per=96.68%, avg=3363.33, stdev=2408.13, samples=3 iops : min= 1, max= 6, avg= 2.67, stdev= 2.89, samples=3 write: IOPS=1, BW=1532KiB/s (1569kB/s)(6144KiB/4010msec) clat (msec): min=32, max=987, avg=210.92, stdev=381.75 lat (msec): min=32, max=987, avg=210.97, stdev=381.74 clat percentiles (msec): | 1.00th=[ 33], 5.00th=[ 33], 10.00th=[ 33], 20.00th=[ 35], | 30.00th=[ 35], 40.00th=[ 48], 50.00th=[ 48], 60.00th=[ 52], | 70.00th=[ 112], 80.00th=[ 112], 90.00th=[ 986], 95.00th=[ 986], | 99.00th=[ 986], 99.50th=[ 986], 99.90th=[ 986], 99.95th=[ 986], | 99.99th=[ 986] bw ( KiB/s): min= 1973, max= 4096, per=62.30%, avg=3034.50, stdev=1225.71, samples=4 iops : min= 1, max= 4, avg= 2.50, stdev= 1.73, samples=4 lat (msec) : 4=8.33%, 50=33.33%, 100=16.67%, 250=8.33%, 750=8.33% lat (msec) : 1000=16.67% cpu : usr=0.02%, sys=16.46%, ctx=834, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17157: Mon May 13 03:10:35 2024 read: IOPS=2, BW=2796KiB/s (2863kB/s)(5120KiB/1831msec) clat (usec): min=459, max=428997, avg=120337.62, stdev=185019.37 lat (usec): min=459, max=428997, avg=120338.52, stdev=185019.10 clat percentiles (usec): | 1.00th=[ 461], 5.00th=[ 461], 10.00th=[ 461], 20.00th=[ 461], | 30.00th=[ 742], 40.00th=[ 742], 50.00th=[ 12911], 60.00th=[ 12911], | 70.00th=[158335], 80.00th=[158335], 90.00th=[429917], 95.00th=[429917], | 99.00th=[429917], 99.50th=[429917], 99.90th=[429917], 99.95th=[429917], | 99.99th=[429917] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=3, BW=3915KiB/s (4009kB/s)(7168KiB/1831msec) clat (msec): min=22, max=933, avg=175.29, stdev=335.35 lat (msec): min=22, max=933, avg=175.32, stdev=335.35 clat percentiles (msec): | 1.00th=[ 23], 5.00th=[ 23], 10.00th=[ 23], 20.00th=[ 27], | 30.00th=[ 37], 40.00th=[ 37], 50.00th=[ 46], 60.00th=[ 71], | 70.00th=[ 71], 80.00th=[ 92], 90.00th=[ 936], 95.00th=[ 936], | 99.00th=[ 936], 99.50th=[ 936], 99.90th=[ 936], 99.95th=[ 936], | 99.99th=[ 936] bw ( KiB/s): min= 2048, max=12288, per=100.00%, avg=7168.00, stdev=7240.77, samples=2 iops : min= 2, max= 12, avg= 7.00, stdev= 7.07, samples=2 lat (usec) : 500=8.33%, 750=8.33% lat (msec) : 20=8.33%, 50=33.33%, 100=16.67%, 250=8.33%, 500=8.33% lat (msec) : 1000=8.33% cpu : usr=0.00%, sys=15.52%, ctx=228, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=3479KiB/s (3563kB/s), 1050KiB/s-2796KiB/s (1075kB/s-2863kB/s), io=20.0MiB (20.0MB), run=1831-5886msec WRITE: bw=4871KiB/s (4988kB/s), 870KiB/s-5251KiB/s (891kB/s-5377kB/s), io=28.0MiB (29.4MB), run=1831-5886msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=17172: Mon May 13 03:10:37 2024 read: IOPS=2, BW=10.1MiB/s (10.6MB/s)(12.0MiB/1187msec) clat (msec): min=173, max=827, avg=393.77, stdev=375.46 lat (msec): min=173, max=827, avg=393.78, stdev=375.46 clat percentiles (msec): | 1.00th=[ 174], 5.00th=[ 174], 10.00th=[ 174], 20.00th=[ 174], | 30.00th=[ 174], 40.00th=[ 180], 50.00th=[ 180], 60.00th=[ 180], | 70.00th=[ 827], 80.00th=[ 827], 90.00th=[ 827], 95.00th=[ 827], | 99.00th=[ 827], 99.50th=[ 827], 99.90th=[ 827], 99.95th=[ 827], | 99.99th=[ 827] bw ( KiB/s): min= 8192, max= 8192, per=59.60%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 1000=33.33% cpu : usr=0.08%, sys=67.71%, ctx=291, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17173: Mon May 13 03:10:37 2024 read: IOPS=0, BW=3436KiB/s (3519kB/s)(4096KiB/1192msec) clat (nsec): min=185463k, max=185463k, avg=185463489.00, stdev= 0.00 lat (nsec): min=185465k, max=185465k, avg=185464976.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 186], 5.00th=[ 186], 10.00th=[ 186], 20.00th=[ 186], | 30.00th=[ 186], 40.00th=[ 186], 50.00th=[ 186], 60.00th=[ 186], | 70.00th=[ 186], 80.00th=[ 186], 90.00th=[ 186], 95.00th=[ 186], | 99.00th=[ 186], 99.50th=[ 186], 99.90th=[ 186], 99.95th=[ 186], | 99.99th=[ 186] bw ( KiB/s): min= 8192, max= 8192, per=59.60%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=1, BW=6872KiB/s (7037kB/s)(8192KiB/1192msec) clat (msec): min=245, max=731, avg=488.80, stdev=343.41 lat (msec): min=246, max=731, avg=488.97, stdev=343.50 clat percentiles (msec): | 1.00th=[ 247], 5.00th=[ 247], 10.00th=[ 247], 20.00th=[ 247], | 30.00th=[ 247], 40.00th=[ 247], 50.00th=[ 247], 60.00th=[ 735], | 70.00th=[ 735], 80.00th=[ 735], 90.00th=[ 735], 95.00th=[ 735], | 99.00th=[ 735], 99.50th=[ 735], 99.90th=[ 735], 99.95th=[ 735], | 99.99th=[ 735] bw ( KiB/s): min= 8192, max= 8192, per=30.98%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 750=33.33% cpu : usr=0.00%, sys=19.06%, ctx=385, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17174: Mon May 13 03:10:37 2024 write: IOPS=2, BW=9918KiB/s (10.2MB/s)(12.0MiB/1239msec) clat (msec): min=103, max=735, avg=410.39, stdev=316.61 lat (msec): min=103, max=736, avg=410.63, stdev=316.64 clat percentiles (msec): | 1.00th=[ 104], 5.00th=[ 104], 10.00th=[ 104], 20.00th=[ 104], | 30.00th=[ 104], 40.00th=[ 393], 50.00th=[ 393], 60.00th=[ 393], | 70.00th=[ 735], 80.00th=[ 735], 90.00th=[ 735], 95.00th=[ 735], | 99.00th=[ 735], 99.50th=[ 735], 99.90th=[ 735], 99.95th=[ 735], | 99.99th=[ 735] bw ( KiB/s): min= 8192, max= 8192, per=30.98%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=33.33%, 750=33.33% cpu : usr=0.00%, sys=20.52%, ctx=51, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17175: Mon May 13 03:10:37 2024 write: IOPS=4, BW=16.8MiB/s (17.6MB/s)(12.0MiB/714msec) clat (msec): min=104, max=413, avg=233.69, stdev=160.77 lat (msec): min=105, max=414, avg=233.92, stdev=160.77 clat percentiles (msec): | 1.00th=[ 106], 5.00th=[ 106], 10.00th=[ 106], 20.00th=[ 106], | 30.00th=[ 106], 40.00th=[ 182], 50.00th=[ 182], 60.00th=[ 182], | 70.00th=[ 414], 80.00th=[ 414], 90.00th=[ 414], 95.00th=[ 414], | 99.00th=[ 414], 99.50th=[ 414], 99.90th=[ 414], 99.95th=[ 414], | 99.99th=[ 414] bw ( KiB/s): min=16286, max=16286, per=61.58%, avg=16286.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.00%, sys=39.13%, ctx=38, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=13.4MiB/s (14.1MB/s), 3436KiB/s-10.1MiB/s (3519kB/s-10.6MB/s), io=16.0MiB (16.8MB), run=1187-1192msec WRITE: bw=25.8MiB/s (27.1MB/s), 6872KiB/s-16.8MiB/s (7037kB/s-17.6MB/s), io=32.0MiB (33.6MB), run=714-1239msec rand-rw: (groupid=0, jobs=1): err= 0: pid=17168: Mon May 13 03:10:37 2024 read: IOPS=8, BW=34.1MiB/s (35.7MB/s)(12.0MiB/352msec) clat (msec): min=77, max=173, avg=112.91, stdev=52.85 lat (msec): min=77, max=173, avg=112.91, stdev=52.85 clat percentiles (msec): | 1.00th=[ 78], 5.00th=[ 78], 10.00th=[ 78], 20.00th=[ 78], | 30.00th=[ 78], 40.00th=[ 88], 50.00th=[ 88], 60.00th=[ 88], | 70.00th=[ 174], 80.00th=[ 174], 90.00th=[ 174], 95.00th=[ 174], | 99.00th=[ 174], 99.50th=[ 174], 99.90th=[ 174], 99.95th=[ 174], | 99.99th=[ 174] lat (msec) : 100=66.67%, 250=33.33% cpu : usr=0.28%, sys=25.07%, ctx=64, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17169: Mon May 13 03:10:37 2024 read: IOPS=0, BW=2646KiB/s (2709kB/s)(4096KiB/1548msec) clat (nsec): min=271099k, max=271099k, avg=271098966.00, stdev= 0.00 lat (nsec): min=271100k, max=271100k, avg=271099896.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 271], 5.00th=[ 271], 10.00th=[ 271], 20.00th=[ 271], | 30.00th=[ 271], 40.00th=[ 271], 50.00th=[ 271], 60.00th=[ 271], | 70.00th=[ 271], 80.00th=[ 271], 90.00th=[ 271], 95.00th=[ 271], | 99.00th=[ 271], 99.50th=[ 271], 99.90th=[ 271], 99.95th=[ 271], | 99.99th=[ 271] bw ( KiB/s): min= 8175, max= 8175, per=77.25%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=1, BW=5292KiB/s (5419kB/s)(8192KiB/1548msec) clat (msec): min=382, max=888, avg=635.36, stdev=358.03 lat (msec): min=382, max=888, avg=635.63, stdev=357.97 clat percentiles (msec): | 1.00th=[ 384], 5.00th=[ 384], 10.00th=[ 384], 20.00th=[ 384], | 30.00th=[ 384], 40.00th=[ 384], 50.00th=[ 384], 60.00th=[ 885], | 70.00th=[ 885], 80.00th=[ 885], 90.00th=[ 885], 95.00th=[ 885], | 99.00th=[ 885], 99.50th=[ 885], 99.90th=[ 885], 99.95th=[ 885], | 99.99th=[ 885] bw ( KiB/s): min= 7670, max= 7670, per=37.64%, avg=7670.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 500=66.67%, 1000=33.33% cpu : usr=0.06%, sys=6.92%, ctx=40, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17170: Mon May 13 03:10:37 2024 write: IOPS=1, BW=7642KiB/s (7825kB/s)(12.0MiB/1608msec) clat (msec): min=86, max=884, avg=527.98, stdev=405.63 lat (msec): min=86, max=884, avg=528.23, stdev=405.68 clat percentiles (msec): | 1.00th=[ 87], 5.00th=[ 87], 10.00th=[ 87], 20.00th=[ 87], | 30.00th=[ 87], 40.00th=[ 617], 50.00th=[ 617], 60.00th=[ 617], | 70.00th=[ 885], 80.00th=[ 885], 90.00th=[ 885], 95.00th=[ 885], | 99.00th=[ 885], 99.50th=[ 885], 99.90th=[ 885], 99.95th=[ 885], | 99.99th=[ 885] bw ( KiB/s): min= 7366, max= 7393, per=36.21%, avg=7379.50, stdev=19.09, samples=2 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=2 lat (msec) : 100=33.33%, 750=33.33%, 1000=33.33% cpu : usr=0.06%, sys=13.13%, ctx=256, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17171: Mon May 13 03:10:37 2024 write: IOPS=4, BW=18.3MiB/s (19.2MB/s)(12.0MiB/655msec) clat (msec): min=59, max=433, avg=216.46, stdev=194.01 lat (msec): min=59, max=433, avg=216.76, stdev=193.91 clat percentiles (msec): | 1.00th=[ 60], 5.00th=[ 60], 10.00th=[ 60], 20.00th=[ 60], | 30.00th=[ 60], 40.00th=[ 157], 50.00th=[ 157], 60.00th=[ 157], | 70.00th=[ 435], 80.00th=[ 435], 90.00th=[ 435], 95.00th=[ 435], | 99.00th=[ 435], 99.50th=[ 435], 99.90th=[ 435], 99.95th=[ 435], | 99.99th=[ 435] bw ( KiB/s): min=16384, max=16384, per=80.40%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 250=33.33%, 500=33.33% cpu : usr=0.00%, sys=10.86%, ctx=33, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=10.3MiB/s (10.8MB/s), 2646KiB/s-34.1MiB/s (2709kB/s-35.7MB/s), io=16.0MiB (16.8MB), run=352-1548msec WRITE: bw=19.9MiB/s (20.9MB/s), 5292KiB/s-18.3MiB/s (5419kB/s-19.2MB/s), io=32.0MiB (33.6MB), run=655-1608msec