== sanity test 398b: DIO and buffer IO race ============== 23:41:39 (1708836099) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.19297 s, 42.2 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=9399: Sat Feb 24 23:41:55 2024 read: IOPS=107, BW=429KiB/s (440kB/s)(5992KiB/13955msec) clat (usec): min=42, max=167306, avg=5089.76, stdev=5150.91 lat (usec): min=43, max=167307, avg=5091.21, stdev=5150.81 clat percentiles (usec): | 1.00th=[ 76], 5.00th=[ 110], 10.00th=[ 2278], 20.00th=[ 2802], | 30.00th=[ 3261], 40.00th=[ 3752], 50.00th=[ 4359], 60.00th=[ 5211], | 70.00th=[ 6128], 80.00th=[ 7177], 90.00th=[ 8586], 95.00th=[ 10028], | 99.00th=[ 16057], 99.50th=[ 18220], 99.90th=[ 22152], 99.95th=[166724], | 99.99th=[166724] bw ( KiB/s): min= 288, max= 590, per=24.84%, avg=425.56, stdev=63.72, samples=27 iops : min= 72, max= 147, avg=106.26, stdev=15.87, samples=27 write: IOPS=112, BW=451KiB/s (462kB/s)(6296KiB/13955msec) clat (usec): min=445, max=30218, avg=3981.05, stdev=3715.47 lat (usec): min=446, max=30219, avg=3982.76, stdev=3716.00 clat percentiles (usec): | 1.00th=[ 515], 5.00th=[ 594], 10.00th=[ 660], 20.00th=[ 750], | 30.00th=[ 898], 40.00th=[ 1745], 50.00th=[ 3228], 60.00th=[ 4015], | 70.00th=[ 4948], 80.00th=[ 6718], 90.00th=[ 9241], 95.00th=[11338], | 99.00th=[15664], 99.50th=[17433], 99.90th=[23987], 99.95th=[30278], | 99.99th=[30278] bw ( KiB/s): min= 336, max= 600, per=24.79%, avg=444.81, stdev=64.94, samples=27 iops : min= 84, max= 150, avg=111.07, stdev=16.26, samples=27 lat (usec) : 50=0.03%, 100=2.02%, 250=1.37%, 500=0.42%, 750=9.83% lat (usec) : 1000=6.74% lat (msec) : 2=4.36%, 4=27.54%, 10=41.11%, 20=6.41%, 50=0.13% lat (msec) : 250=0.03% cpu : usr=0.24%, sys=22.73%, ctx=5509, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9401: Sat Feb 24 23:41:55 2024 read: IOPS=106, BW=428KiB/s (438kB/s)(5940KiB/13886msec) clat (usec): min=48, max=29208, avg=4998.70, stdev=3134.46 lat (usec): min=48, max=29208, avg=4999.96, stdev=3134.49 clat percentiles (usec): | 1.00th=[ 68], 5.00th=[ 113], 10.00th=[ 2311], 20.00th=[ 2769], | 30.00th=[ 3195], 40.00th=[ 3621], 50.00th=[ 4293], 60.00th=[ 5276], | 70.00th=[ 6063], 80.00th=[ 7046], 90.00th=[ 8717], 95.00th=[10290], | 99.00th=[16909], 99.50th=[19268], 99.90th=[23987], 99.95th=[29230], | 99.99th=[29230] bw ( KiB/s): min= 296, max= 622, per=24.88%, avg=426.15, stdev=67.00, samples=27 iops : min= 74, max= 155, avg=106.41, stdev=16.70, samples=27 write: IOPS=114, BW=457KiB/s (468kB/s)(6348KiB/13886msec) clat (usec): min=441, max=165075, avg=4029.75, stdev=5447.51 lat (usec): min=441, max=165076, avg=4030.80, stdev=5447.48 clat percentiles (usec): | 1.00th=[ 515], 5.00th=[ 603], 10.00th=[ 660], 20.00th=[ 766], | 30.00th=[ 906], 40.00th=[ 1532], 50.00th=[ 3195], 60.00th=[ 4015], | 70.00th=[ 5014], 80.00th=[ 6783], 90.00th=[ 8717], 95.00th=[ 11207], | 99.00th=[ 16188], 99.50th=[ 18482], 99.90th=[ 27132], 99.95th=[164627], | 99.99th=[164627] bw ( KiB/s): min= 352, max= 592, per=25.11%, avg=450.41, stdev=59.99, samples=27 iops : min= 88, max= 148, avg=112.48, stdev=14.99, samples=27 lat (usec) : 50=0.03%, 100=1.73%, 250=1.33%, 500=0.36%, 750=9.41% lat (usec) : 1000=7.13% lat (msec) : 2=5.14%, 4=28.16%, 10=40.69%, 20=5.70%, 50=0.29% lat (msec) : 250=0.03% cpu : usr=0.17%, sys=22.64%, ctx=5673, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9403: Sat Feb 24 23:41:55 2024 read: IOPS=107, BW=431KiB/s (441kB/s)(6028KiB/13984msec) clat (usec): min=49, max=25731, avg=4848.51, stdev=3207.53 lat (usec): min=50, max=25732, avg=4850.42, stdev=3208.20 clat percentiles (usec): | 1.00th=[ 67], 5.00th=[ 103], 10.00th=[ 2114], 20.00th=[ 2671], | 30.00th=[ 3130], 40.00th=[ 3589], 50.00th=[ 4228], 60.00th=[ 5014], | 70.00th=[ 5800], 80.00th=[ 6980], 90.00th=[ 8455], 95.00th=[10028], | 99.00th=[17171], 99.50th=[19530], 99.90th=[23200], 99.95th=[25822], | 99.99th=[25822] bw ( KiB/s): min= 296, max= 566, per=24.87%, avg=426.07, stdev=63.03, samples=27 iops : min= 74, max= 141, avg=106.41, stdev=15.65, samples=27 write: IOPS=111, BW=448KiB/s (458kB/s)(6260KiB/13984msec) clat (usec): min=451, max=168173, avg=4220.63, stdev=5578.81 lat (usec): min=451, max=168175, avg=4222.33, stdev=5578.49 clat percentiles (usec): | 1.00th=[ 502], 5.00th=[ 586], 10.00th=[ 660], 20.00th=[ 758], | 30.00th=[ 947], 40.00th=[ 2180], 50.00th=[ 3326], 60.00th=[ 4178], | 70.00th=[ 5407], 80.00th=[ 7046], 90.00th=[ 9372], 95.00th=[ 11600], | 99.00th=[ 16319], 99.50th=[ 18482], 99.90th=[ 24511], 99.95th=[168821], | 99.99th=[168821] bw ( KiB/s): min= 296, max= 544, per=24.51%, avg=439.78, stdev=66.12, samples=27 iops : min= 74, max= 136, avg=109.81, stdev=16.61, samples=27 lat (usec) : 50=0.03%, 100=2.25%, 250=1.86%, 500=0.68%, 750=9.41% lat (usec) : 1000=6.28% lat (msec) : 2=4.30%, 4=27.73%, 10=40.89%, 20=6.25%, 50=0.29% lat (msec) : 250=0.03% cpu : usr=0.26%, sys=22.53%, ctx=5585, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9405: Sat Feb 24 23:41:55 2024 read: IOPS=107, BW=432KiB/s (442kB/s)(6052KiB/14012msec) clat (usec): min=42, max=29237, avg=4987.57, stdev=3086.04 lat (usec): min=42, max=29238, avg=4988.46, stdev=3086.05 clat percentiles (usec): | 1.00th=[ 68], 5.00th=[ 106], 10.00th=[ 2278], 20.00th=[ 2737], | 30.00th=[ 3228], 40.00th=[ 3752], 50.00th=[ 4424], 60.00th=[ 5211], | 70.00th=[ 6063], 80.00th=[ 7046], 90.00th=[ 8455], 95.00th=[10159], | 99.00th=[15139], 99.50th=[16909], 99.90th=[26084], 99.95th=[29230], | 99.99th=[29230] bw ( KiB/s): min= 240, max= 632, per=25.19%, avg=431.54, stdev=75.30, samples=28 iops : min= 60, max= 158, avg=107.75, stdev=18.86, samples=28 write: IOPS=111, BW=445KiB/s (456kB/s)(6236KiB/14012msec) clat (usec): min=376, max=162356, avg=4102.83, stdev=5441.56 lat (usec): min=377, max=162359, avg=4103.91, stdev=5441.57 clat percentiles (usec): | 1.00th=[ 502], 5.00th=[ 586], 10.00th=[ 652], 20.00th=[ 750], | 30.00th=[ 881], 40.00th=[ 1450], 50.00th=[ 3195], 60.00th=[ 4080], | 70.00th=[ 5145], 80.00th=[ 7111], 90.00th=[ 9372], 95.00th=[ 11076], | 99.00th=[ 15270], 99.50th=[ 16057], 99.90th=[ 30540], 99.95th=[162530], | 99.99th=[162530] bw ( KiB/s): min= 296, max= 694, per=24.75%, avg=444.00, stdev=77.25, samples=28 iops : min= 74, max= 173, avg=110.89, stdev=19.25, samples=28 lat (usec) : 50=0.07%, 100=2.18%, 250=1.14%, 500=0.55%, 750=9.83% lat (usec) : 1000=7.03% lat (msec) : 2=4.20%, 4=26.63%, 10=41.76%, 20=6.38%, 50=0.20% lat (msec) : 250=0.03% cpu : usr=0.19%, sys=22.42%, ctx=5564, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1714KiB/s (1755kB/s), 428KiB/s-432KiB/s (438kB/s-442kB/s), io=23.4MiB (24.6MB), run=13886-14012msec WRITE: bw=1794KiB/s (1837kB/s), 445KiB/s-457KiB/s (456kB/s-468kB/s), io=24.6MiB (25.7MB), run=13886-14012msec rand-rw: (groupid=0, jobs=1): err= 0: pid=9400: Sat Feb 24 23:41:59 2024 read: IOPS=82, BW=329KiB/s (337kB/s)(5992KiB/18216msec) clat (usec): min=1534, max=167510, avg=4806.88, stdev=5757.39 lat (usec): min=1534, max=167513, avg=4807.82, stdev=5757.42 clat percentiles (usec): | 1.00th=[ 1811], 5.00th=[ 1958], 10.00th=[ 2073], 20.00th=[ 2245], | 30.00th=[ 2442], 40.00th=[ 2704], 50.00th=[ 3097], 60.00th=[ 3654], | 70.00th=[ 4490], 80.00th=[ 6325], 90.00th=[ 10028], 95.00th=[ 13698], | 99.00th=[ 19268], 99.50th=[ 24249], 99.90th=[ 27657], 99.95th=[166724], | 99.99th=[166724] bw ( KiB/s): min= 160, max= 728, per=24.66%, avg=325.00, stdev=174.44, samples=36 iops : min= 40, max= 182, avg=81.17, stdev=43.65, samples=36 write: IOPS=86, BW=346KiB/s (354kB/s)(6296KiB/18216msec) clat (usec): min=2278, max=36547, avg=6960.88, stdev=4477.47 lat (usec): min=2279, max=36548, avg=6962.00, stdev=4477.43 clat percentiles (usec): | 1.00th=[ 2671], 5.00th=[ 3032], 10.00th=[ 3261], 20.00th=[ 3589], | 30.00th=[ 3884], 40.00th=[ 4228], 50.00th=[ 4948], 60.00th=[ 6259], | 70.00th=[ 8225], 80.00th=[10552], 90.00th=[13435], 95.00th=[15795], | 99.00th=[21365], 99.50th=[23200], 99.90th=[34866], 99.95th=[36439], | 99.99th=[36439] bw ( KiB/s): min= 192, max= 720, per=24.71%, avg=341.00, stdev=174.25, samples=36 iops : min= 48, max= 180, avg=85.17, stdev=43.61, samples=36 lat (msec) : 2=3.06%, 4=46.32%, 10=34.54%, 20=14.88%, 50=1.17% lat (msec) : 250=0.03% cpu : usr=0.17%, sys=10.82%, ctx=3536, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9402: Sat Feb 24 23:41:59 2024 read: IOPS=81, BW=326KiB/s (334kB/s)(5940KiB/18202msec) clat (usec): min=1642, max=159104, avg=4956.80, stdev=5670.35 lat (usec): min=1643, max=159105, avg=4957.69, stdev=5670.36 clat percentiles (usec): | 1.00th=[ 1844], 5.00th=[ 1975], 10.00th=[ 2073], 20.00th=[ 2245], | 30.00th=[ 2409], 40.00th=[ 2704], 50.00th=[ 3130], 60.00th=[ 3752], | 70.00th=[ 4752], 80.00th=[ 6915], 90.00th=[ 10683], 95.00th=[ 14091], | 99.00th=[ 19006], 99.50th=[ 23725], 99.90th=[ 27657], 99.95th=[158335], | 99.99th=[158335] bw ( KiB/s): min= 112, max= 712, per=24.59%, avg=324.08, stdev=178.92, samples=36 iops : min= 28, max= 178, avg=80.92, stdev=44.79, samples=36 write: IOPS=87, BW=349KiB/s (357kB/s)(6348KiB/18202msec) clat (usec): min=2404, max=34864, avg=6794.39, stdev=4265.35 lat (usec): min=2405, max=34865, avg=6795.49, stdev=4265.35 clat percentiles (usec): | 1.00th=[ 2671], 5.00th=[ 3032], 10.00th=[ 3261], 20.00th=[ 3589], | 30.00th=[ 3851], 40.00th=[ 4228], 50.00th=[ 4948], 60.00th=[ 6194], | 70.00th=[ 7832], 80.00th=[10028], 90.00th=[12780], 95.00th=[15270], | 99.00th=[21890], 99.50th=[24773], 99.90th=[32637], 99.95th=[34866], | 99.99th=[34866] bw ( KiB/s): min= 184, max= 704, per=24.90%, avg=343.64, stdev=172.81, samples=36 iops : min= 46, max= 176, avg=85.81, stdev=43.26, samples=36 lat (msec) : 2=2.99%, 4=45.67%, 10=35.38%, 20=14.84%, 50=1.07% lat (msec) : 250=0.03% cpu : usr=0.27%, sys=10.70%, ctx=3530, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9404: Sat Feb 24 23:41:59 2024 read: IOPS=82, BW=331KiB/s (339kB/s)(6028KiB/18200msec) clat (usec): min=1635, max=26827, avg=4814.04, stdev=3975.10 lat (usec): min=1635, max=26829, avg=4815.09, stdev=3975.13 clat percentiles (usec): | 1.00th=[ 1827], 5.00th=[ 1975], 10.00th=[ 2089], 20.00th=[ 2245], | 30.00th=[ 2442], 40.00th=[ 2671], 50.00th=[ 3097], 60.00th=[ 3752], | 70.00th=[ 4752], 80.00th=[ 6915], 90.00th=[10552], 95.00th=[13698], | 99.00th=[19268], 99.50th=[23725], 99.90th=[25560], 99.95th=[26870], | 99.99th=[26870] bw ( KiB/s): min= 152, max= 712, per=24.86%, avg=327.67, stdev=172.58, samples=36 iops : min= 38, max= 178, avg=81.75, stdev=43.17, samples=36 write: IOPS=85, BW=344KiB/s (352kB/s)(6260KiB/18200msec) clat (msec): min=2, max=161, avg= 6.96, stdev= 5.90 lat (msec): min=2, max=161, avg= 6.96, stdev= 5.90 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 4], 10.00th=[ 4], 20.00th=[ 4], | 30.00th=[ 4], 40.00th=[ 5], 50.00th=[ 5], 60.00th=[ 7], | 70.00th=[ 9], 80.00th=[ 11], 90.00th=[ 14], 95.00th=[ 17], | 99.00th=[ 23], 99.50th=[ 24], 99.90th=[ 38], 99.95th=[ 163], | 99.99th=[ 163] bw ( KiB/s): min= 176, max= 710, per=24.58%, avg=339.22, stdev=180.65, samples=36 iops : min= 44, max= 177, avg=84.64, stdev=45.15, samples=36 lat (msec) : 2=2.90%, 4=46.65%, 10=34.24%, 20=15.01%, 50=1.17% lat (msec) : 250=0.03% cpu : usr=0.21%, sys=10.84%, ctx=3483, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9406: Sat Feb 24 23:41:59 2024 read: IOPS=83, BW=334KiB/s (342kB/s)(6052KiB/18129msec) clat (usec): min=1667, max=32835, avg=4837.94, stdev=3950.31 lat (usec): min=1668, max=32835, avg=4838.84, stdev=3950.31 clat percentiles (usec): | 1.00th=[ 1844], 5.00th=[ 2008], 10.00th=[ 2089], 20.00th=[ 2245], | 30.00th=[ 2442], 40.00th=[ 2671], 50.00th=[ 3097], 60.00th=[ 3720], | 70.00th=[ 4752], 80.00th=[ 6783], 90.00th=[10814], 95.00th=[13698], | 99.00th=[18744], 99.50th=[20055], 99.90th=[24511], 99.95th=[32900], | 99.99th=[32900] bw ( KiB/s): min= 143, max= 704, per=25.08%, avg=330.58, stdev=176.57, samples=36 iops : min= 35, max= 176, avg=82.47, stdev=44.19, samples=36 write: IOPS=85, BW=344KiB/s (352kB/s)(6236KiB/18129msec) clat (msec): min=2, max=161, avg= 6.90, stdev= 5.85 lat (msec): min=2, max=161, avg= 6.90, stdev= 5.85 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 4], 20.00th=[ 4], | 30.00th=[ 4], 40.00th=[ 5], 50.00th=[ 5], 60.00th=[ 7], | 70.00th=[ 9], 80.00th=[ 11], 90.00th=[ 14], 95.00th=[ 16], | 99.00th=[ 22], 99.50th=[ 24], 99.90th=[ 38], 99.95th=[ 161], | 99.99th=[ 161] bw ( KiB/s): min= 160, max= 728, per=24.76%, avg=341.69, stdev=179.82, samples=36 iops : min= 40, max= 182, avg=85.25, stdev=44.98, samples=36 lat (msec) : 2=2.47%, 4=46.55%, 10=34.70%, 20=15.30%, 50=0.94% lat (msec) : 250=0.03% cpu : usr=0.19%, sys=10.79%, ctx=3486, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1318KiB/s (1350kB/s), 326KiB/s-334KiB/s (334kB/s-342kB/s), io=23.4MiB (24.6MB), run=18129-18216msec WRITE: bw=1380KiB/s (1413kB/s), 344KiB/s-349KiB/s (352kB/s-357kB/s), io=24.6MiB (25.7MB), run=18129-18216msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=9440: Sat Feb 24 23:42:05 2024 read: IOPS=91, BW=1461KiB/s (1496kB/s)(5920KiB/4052msec) clat (usec): min=58, max=19348, avg=5849.34, stdev=3398.72 lat (usec): min=58, max=19349, avg=5851.23, stdev=3397.05 clat percentiles (usec): | 1.00th=[ 70], 5.00th=[ 96], 10.00th=[ 2540], 20.00th=[ 3523], | 30.00th=[ 4146], 40.00th=[ 4621], 50.00th=[ 5473], 60.00th=[ 6259], | 70.00th=[ 6980], 80.00th=[ 7963], 90.00th=[10028], 95.00th=[12518], | 99.00th=[16319], 99.50th=[17171], 99.90th=[19268], 99.95th=[19268], | 99.99th=[19268] bw ( KiB/s): min= 1248, max= 1792, per=24.55%, avg=1451.12, stdev=178.29, samples=8 iops : min= 78, max= 112, avg=90.50, stdev=11.14, samples=8 write: IOPS=98, BW=1572KiB/s (1609kB/s)(6368KiB/4052msec) clat (usec): min=629, max=22583, avg=4717.36, stdev=3848.44 lat (usec): min=630, max=22584, avg=4719.17, stdev=3848.10 clat percentiles (usec): | 1.00th=[ 725], 5.00th=[ 873], 10.00th=[ 963], 20.00th=[ 1156], | 30.00th=[ 1745], 40.00th=[ 2999], 50.00th=[ 3818], 60.00th=[ 4490], | 70.00th=[ 5932], 80.00th=[ 7635], 90.00th=[ 9765], 95.00th=[12387], | 99.00th=[17695], 99.50th=[19792], 99.90th=[22676], 99.95th=[22676], | 99.99th=[22676] bw ( KiB/s): min= 1120, max= 1948, per=25.26%, avg=1571.00, stdev=269.35, samples=8 iops : min= 70, max= 121, avg=98.00, stdev=16.69, samples=8 lat (usec) : 100=2.60%, 250=1.95%, 750=0.52%, 1000=5.99% lat (msec) : 2=10.03%, 4=20.05%, 10=48.83%, 20=9.90%, 50=0.13% cpu : usr=0.10%, sys=25.23%, ctx=1726, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9442: Sat Feb 24 23:42:05 2024 read: IOPS=89, BW=1430KiB/s (1464kB/s)(5744KiB/4017msec) clat (usec): min=55, max=19256, avg=5812.18, stdev=3446.19 lat (usec): min=56, max=19257, avg=5813.18, stdev=3446.45 clat percentiles (usec): | 1.00th=[ 69], 5.00th=[ 95], 10.00th=[ 2573], 20.00th=[ 3490], | 30.00th=[ 4080], 40.00th=[ 4621], 50.00th=[ 5342], 60.00th=[ 6259], | 70.00th=[ 6915], 80.00th=[ 7832], 90.00th=[ 9896], 95.00th=[12256], | 99.00th=[18744], 99.50th=[19268], 99.90th=[19268], 99.95th=[19268], | 99.99th=[19268] bw ( KiB/s): min= 1117, max= 1632, per=23.27%, avg=1375.57, stdev=219.97, samples=7 iops : min= 69, max= 102, avg=85.86, stdev=13.91, samples=7 write: IOPS=101, BW=1629KiB/s (1668kB/s)(6544KiB/4017msec) clat (usec): min=669, max=20141, avg=4686.97, stdev=3734.37 lat (usec): min=669, max=20142, avg=4697.74, stdev=3743.83 clat percentiles (usec): | 1.00th=[ 758], 5.00th=[ 865], 10.00th=[ 955], 20.00th=[ 1106], | 30.00th=[ 1663], 40.00th=[ 3032], 50.00th=[ 3949], 60.00th=[ 4752], | 70.00th=[ 5932], 80.00th=[ 7504], 90.00th=[10028], 95.00th=[12256], | 99.00th=[15664], 99.50th=[16319], 99.90th=[20055], 99.95th=[20055], | 99.99th=[20055] bw ( KiB/s): min= 1344, max= 1852, per=24.98%, avg=1553.71, stdev=209.58, samples=7 iops : min= 84, max= 115, avg=97.00, stdev=12.92, samples=7 lat (usec) : 100=2.73%, 250=1.56%, 750=0.52%, 1000=6.25% lat (msec) : 2=11.33%, 4=17.97%, 10=49.87%, 20=9.64%, 50=0.13% cpu : usr=0.27%, sys=25.24%, ctx=1750, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9444: Sat Feb 24 23:42:05 2024 read: IOPS=98, BW=1583KiB/s (1621kB/s)(6256KiB/3951msec) clat (usec): min=59, max=25801, avg=5585.10, stdev=3181.51 lat (usec): min=59, max=25802, avg=5585.93, stdev=3181.50 clat percentiles (usec): | 1.00th=[ 69], 5.00th=[ 92], 10.00th=[ 149], 20.00th=[ 3556], | 30.00th=[ 4080], 40.00th=[ 4621], 50.00th=[ 5342], 60.00th=[ 6194], | 70.00th=[ 6915], 80.00th=[ 7963], 90.00th=[ 9110], 95.00th=[10683], | 99.00th=[13566], 99.50th=[17171], 99.90th=[25822], 99.95th=[25822], | 99.99th=[25822] bw ( KiB/s): min= 1248, max= 1728, per=25.81%, avg=1525.86, stdev=168.16, samples=7 iops : min= 78, max= 108, avg=95.14, stdev=10.56, samples=7 write: IOPS=95, BW=1527KiB/s (1563kB/s)(6032KiB/3951msec) clat (usec): min=614, max=16568, avg=4662.26, stdev=3616.18 lat (usec): min=615, max=16569, avg=4663.48, stdev=3616.27 clat percentiles (usec): | 1.00th=[ 717], 5.00th=[ 898], 10.00th=[ 988], 20.00th=[ 1123], | 30.00th=[ 1729], 40.00th=[ 3195], 50.00th=[ 3785], 60.00th=[ 4555], | 70.00th=[ 5997], 80.00th=[ 8094], 90.00th=[ 9896], 95.00th=[11600], | 99.00th=[16057], 99.50th=[16581], 99.90th=[16581], 99.95th=[16581], | 99.99th=[16581] bw ( KiB/s): min= 1056, max= 1888, per=24.17%, avg=1503.00, stdev=252.44, samples=7 iops : min= 66, max= 118, avg=93.71, stdev=15.76, samples=7 lat (usec) : 100=3.26%, 250=2.34%, 750=0.65%, 1000=4.95% lat (msec) : 2=10.55%, 4=19.01%, 10=51.04%, 20=8.07%, 50=0.13% cpu : usr=0.13%, sys=25.34%, ctx=1685, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9445: Sat Feb 24 23:42:05 2024 read: IOPS=95, BW=1529KiB/s (1566kB/s)(6032KiB/3944msec) clat (usec): min=44, max=18271, avg=5804.25, stdev=3206.70 lat (usec): min=45, max=18272, avg=5806.36, stdev=3208.46 clat percentiles (usec): | 1.00th=[ 70], 5.00th=[ 108], 10.00th=[ 2737], 20.00th=[ 3621], | 30.00th=[ 4146], 40.00th=[ 4752], 50.00th=[ 5407], 60.00th=[ 6325], | 70.00th=[ 7177], 80.00th=[ 8094], 90.00th=[ 9634], 95.00th=[11600], | 99.00th=[15533], 99.50th=[17695], 99.90th=[18220], 99.95th=[18220], | 99.99th=[18220] bw ( KiB/s): min= 1213, max= 1728, per=24.97%, avg=1475.71, stdev=161.29, samples=7 iops : min= 75, max= 108, avg=92.00, stdev=10.31, samples=7 write: IOPS=99, BW=1586KiB/s (1624kB/s)(6256KiB/3944msec) clat (usec): min=678, max=17907, avg=4453.79, stdev=3602.95 lat (usec): min=679, max=17908, avg=4457.65, stdev=3605.84 clat percentiles (usec): | 1.00th=[ 775], 5.00th=[ 898], 10.00th=[ 971], 20.00th=[ 1106], | 30.00th=[ 1385], 40.00th=[ 2343], 50.00th=[ 3589], 60.00th=[ 4555], | 70.00th=[ 5997], 80.00th=[ 7504], 90.00th=[ 9634], 95.00th=[11338], | 99.00th=[15533], 99.50th=[16712], 99.90th=[17957], 99.95th=[17957], | 99.99th=[17957] bw ( KiB/s): min= 1312, max= 1792, per=24.68%, avg=1535.00, stdev=178.15, samples=7 iops : min= 82, max= 112, avg=95.71, stdev=11.18, samples=7 lat (usec) : 50=0.13%, 100=1.69%, 250=2.34%, 500=0.26%, 750=0.39% lat (usec) : 1000=6.64% lat (msec) : 2=12.63%, 4=16.67%, 10=50.26%, 20=8.98% cpu : usr=0.28%, sys=25.72%, ctx=1702, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=5911KiB/s (6053kB/s), 1430KiB/s-1583KiB/s (1464kB/s-1621kB/s), io=23.4MiB (24.5MB), run=3944-4052msec WRITE: bw=6219KiB/s (6368kB/s), 1527KiB/s-1629KiB/s (1563kB/s-1668kB/s), io=24.6MiB (25.8MB), run=3944-4052msec rand-rw: (groupid=0, jobs=1): err= 0: pid=9438: Sat Feb 24 23:42:05 2024 read: IOPS=75, BW=1204KiB/s (1233kB/s)(5920KiB/4917msec) clat (usec): min=2019, max=23200, avg=5181.90, stdev=3779.10 lat (usec): min=2020, max=23201, avg=5182.77, stdev=3779.11 clat percentiles (usec): | 1.00th=[ 2212], 5.00th=[ 2343], 10.00th=[ 2474], 20.00th=[ 2704], | 30.00th=[ 2933], 40.00th=[ 3228], 50.00th=[ 3654], 60.00th=[ 4359], | 70.00th=[ 5014], 80.00th=[ 6783], 90.00th=[10290], 95.00th=[13566], | 99.00th=[20579], 99.50th=[21890], 99.90th=[23200], 99.95th=[23200], | 99.99th=[23200] bw ( KiB/s): min= 704, max= 2016, per=22.55%, avg=1084.11, stdev=401.16, samples=9 iops : min= 44, max= 126, avg=67.67, stdev=25.09, samples=9 write: IOPS=80, BW=1295KiB/s (1326kB/s)(6368KiB/4917msec) clat (usec): min=2818, max=26803, avg=7516.35, stdev=4331.42 lat (usec): min=2820, max=26805, avg=7518.10, stdev=4331.39 clat percentiles (usec): | 1.00th=[ 2966], 5.00th=[ 3425], 10.00th=[ 3720], 20.00th=[ 4178], | 30.00th=[ 4490], 40.00th=[ 4883], 50.00th=[ 5669], 60.00th=[ 6849], | 70.00th=[ 8979], 80.00th=[11207], 90.00th=[14353], 95.00th=[15533], | 99.00th=[22152], 99.50th=[25822], 99.90th=[26870], 99.95th=[26870], | 99.99th=[26870] bw ( KiB/s): min= 864, max= 2208, per=23.75%, avg=1201.44, stdev=399.76, samples=9 iops : min= 54, max= 138, avg=75.00, stdev=25.00, samples=9 lat (msec) : 4=36.07%, 10=45.83%, 20=17.06%, 50=1.04% cpu : usr=0.16%, sys=12.18%, ctx=927, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9439: Sat Feb 24 23:42:05 2024 read: IOPS=72, BW=1153KiB/s (1181kB/s)(5744KiB/4982msec) clat (usec): min=1999, max=22908, avg=5496.56, stdev=4168.22 lat (usec): min=1999, max=22909, avg=5497.48, stdev=4168.21 clat percentiles (usec): | 1.00th=[ 2073], 5.00th=[ 2311], 10.00th=[ 2474], 20.00th=[ 2737], | 30.00th=[ 2999], 40.00th=[ 3326], 50.00th=[ 3916], 60.00th=[ 4490], | 70.00th=[ 5538], 80.00th=[ 7046], 90.00th=[12256], 95.00th=[15926], | 99.00th=[20317], 99.50th=[21365], 99.90th=[22938], 99.95th=[22938], | 99.99th=[22938] bw ( KiB/s): min= 800, max= 2176, per=22.04%, avg=1059.33, stdev=431.32, samples=9 iops : min= 50, max= 136, avg=66.11, stdev=26.99, samples=9 write: IOPS=82, BW=1314KiB/s (1345kB/s)(6544KiB/4982msec) clat (usec): min=2750, max=28779, avg=7335.28, stdev=4313.72 lat (usec): min=2753, max=28781, avg=7336.98, stdev=4313.69 clat percentiles (usec): | 1.00th=[ 2966], 5.00th=[ 3359], 10.00th=[ 3687], 20.00th=[ 4080], | 30.00th=[ 4490], 40.00th=[ 5014], 50.00th=[ 5669], 60.00th=[ 6587], | 70.00th=[ 8291], 80.00th=[10552], 90.00th=[13566], 95.00th=[16057], | 99.00th=[22676], 99.50th=[24249], 99.90th=[28705], 99.95th=[28705], | 99.99th=[28705] bw ( KiB/s): min= 768, max= 2176, per=23.26%, avg=1176.67, stdev=400.08, samples=9 iops : min= 48, max= 136, avg=73.44, stdev=25.08, samples=9 lat (msec) : 2=0.13%, 4=33.33%, 10=48.83%, 20=16.28%, 50=1.43% cpu : usr=0.18%, sys=11.76%, ctx=921, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9441: Sat Feb 24 23:42:05 2024 read: IOPS=79, BW=1272KiB/s (1302kB/s)(6256KiB/4920msec) clat (usec): min=2004, max=22009, avg=5045.67, stdev=3714.92 lat (usec): min=2005, max=22010, avg=5046.59, stdev=3714.90 clat percentiles (usec): | 1.00th=[ 2114], 5.00th=[ 2245], 10.00th=[ 2442], 20.00th=[ 2606], | 30.00th=[ 2769], 40.00th=[ 3130], 50.00th=[ 3556], 60.00th=[ 3982], | 70.00th=[ 4752], 80.00th=[ 6980], 90.00th=[10683], 95.00th=[13829], | 99.00th=[19006], 99.50th=[20841], 99.90th=[21890], 99.95th=[21890], | 99.99th=[21890] bw ( KiB/s): min= 736, max= 2112, per=23.67%, avg=1137.78, stdev=414.80, samples=9 iops : min= 46, max= 132, avg=71.11, stdev=25.93, samples=9 write: IOPS=76, BW=1226KiB/s (1255kB/s)(6032KiB/4920msec) clat (usec): min=2880, max=27536, avg=7793.38, stdev=4400.69 lat (usec): min=2881, max=27538, avg=7795.23, stdev=4400.67 clat percentiles (usec): | 1.00th=[ 3032], 5.00th=[ 3490], 10.00th=[ 3752], 20.00th=[ 4015], | 30.00th=[ 4424], 40.00th=[ 5211], 50.00th=[ 6325], 60.00th=[ 7898], | 70.00th=[ 9372], 80.00th=[11338], 90.00th=[14222], 95.00th=[16450], | 99.00th=[21890], 99.50th=[24249], 99.90th=[27657], 99.95th=[27657], | 99.99th=[27657] bw ( KiB/s): min= 832, max= 2272, per=22.35%, avg=1130.67, stdev=439.93, samples=9 iops : min= 52, max= 142, avg=70.67, stdev=27.50, samples=9 lat (msec) : 4=40.10%, 10=41.15%, 20=17.71%, 50=1.04% cpu : usr=0.14%, sys=11.89%, ctx=901, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9443: Sat Feb 24 23:42:05 2024 read: IOPS=75, BW=1216KiB/s (1245kB/s)(6032KiB/4962msec) clat (usec): min=1998, max=21837, avg=5416.98, stdev=3898.31 lat (usec): min=1999, max=21838, avg=5417.90, stdev=3898.30 clat percentiles (usec): | 1.00th=[ 2073], 5.00th=[ 2343], 10.00th=[ 2442], 20.00th=[ 2704], | 30.00th=[ 2999], 40.00th=[ 3326], 50.00th=[ 3851], 60.00th=[ 4359], | 70.00th=[ 5604], 80.00th=[ 7570], 90.00th=[11731], 95.00th=[13960], | 99.00th=[20055], 99.50th=[20841], 99.90th=[21890], 99.95th=[21890], | 99.99th=[21890] bw ( KiB/s): min= 832, max= 1888, per=22.63%, avg=1087.78, stdev=312.44, samples=9 iops : min= 52, max= 118, avg=67.89, stdev=19.59, samples=9 write: IOPS=78, BW=1261KiB/s (1291kB/s)(6256KiB/4962msec) clat (usec): min=2842, max=28609, avg=7446.03, stdev=4489.73 lat (usec): min=2843, max=28611, avg=7447.74, stdev=4489.78 clat percentiles (usec): | 1.00th=[ 2966], 5.00th=[ 3392], 10.00th=[ 3752], 20.00th=[ 4113], | 30.00th=[ 4424], 40.00th=[ 4883], 50.00th=[ 5669], 60.00th=[ 7046], | 70.00th=[ 8586], 80.00th=[10421], 90.00th=[13566], 95.00th=[16319], | 99.00th=[24773], 99.50th=[27657], 99.90th=[28705], 99.95th=[28705], | 99.99th=[28705] bw ( KiB/s): min= 894, max= 2208, per=22.84%, avg=1155.33, stdev=402.54, samples=9 iops : min= 55, max= 138, avg=72.11, stdev=25.23, samples=9 lat (msec) : 2=0.13%, 4=34.24%, 10=47.66%, 20=16.28%, 50=1.69% cpu : usr=0.20%, sys=11.91%, ctx=913, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=4808KiB/s (4923kB/s), 1153KiB/s-1272KiB/s (1181kB/s-1302kB/s), io=23.4MiB (24.5MB), run=4917-4982msec WRITE: bw=5058KiB/s (5180kB/s), 1226KiB/s-1314KiB/s (1255kB/s-1345kB/s), io=24.6MiB (25.8MB), run=4917-4982msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=9455: Sat Feb 24 23:42:07 2024 read: IOPS=15, BW=15.7MiB/s (16.5MB/s)(7168KiB/445msec) clat (usec): min=454, max=60024, avg=34958.78, stdev=24759.49 lat (usec): min=455, max=60025, avg=34959.84, stdev=24759.66 clat percentiles (usec): | 1.00th=[ 457], 5.00th=[ 457], 10.00th=[ 457], 20.00th=[ 469], | 30.00th=[39060], 40.00th=[39060], 50.00th=[40109], 60.00th=[49546], | 70.00th=[49546], 80.00th=[55313], 90.00th=[60031], 95.00th=[60031], | 99.00th=[60031], 99.50th=[60031], 99.90th=[60031], 99.95th=[60031], | 99.99th=[60031] write: IOPS=11, BW=11.2MiB/s (11.8MB/s)(5120KiB/445msec) clat (usec): min=22026, max=52187, avg=38231.70, stdev=11589.05 lat (usec): min=22075, max=52296, avg=38294.29, stdev=11601.12 clat percentiles (usec): | 1.00th=[22152], 5.00th=[22152], 10.00th=[22152], 20.00th=[22152], | 30.00th=[32900], 40.00th=[32900], 50.00th=[38536], 60.00th=[38536], | 70.00th=[45351], 80.00th=[45351], 90.00th=[52167], 95.00th=[52167], | 99.00th=[52167], 99.50th=[52167], 99.90th=[52167], 99.95th=[52167], | 99.99th=[52167] lat (usec) : 500=16.67% lat (msec) : 50=58.33%, 100=25.00% cpu : usr=0.00%, sys=54.28%, ctx=102, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9457: Sat Feb 24 23:42:07 2024 read: IOPS=2, BW=2260KiB/s (2315kB/s)(2048KiB/906msec) clat (usec): min=35278, max=51600, avg=43439.50, stdev=11541.76 lat (usec): min=35280, max=51602, avg=43441.66, stdev=11541.34 clat percentiles (usec): | 1.00th=[35390], 5.00th=[35390], 10.00th=[35390], 20.00th=[35390], | 30.00th=[35390], 40.00th=[35390], 50.00th=[35390], 60.00th=[51643], | 70.00th=[51643], 80.00th=[51643], 90.00th=[51643], 95.00th=[51643], | 99.00th=[51643], 99.50th=[51643], 99.90th=[51643], 99.95th=[51643], | 99.99th=[51643] bw ( KiB/s): min= 2019, max= 2019, per=8.93%, avg=2019.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=11, BW=11.0MiB/s (11.6MB/s)(10.0MiB/906msec) clat (msec): min=30, max=217, avg=80.85, stdev=59.80 lat (msec): min=31, max=217, avg=80.91, stdev=59.80 clat percentiles (msec): | 1.00th=[ 32], 5.00th=[ 32], 10.00th=[ 32], 20.00th=[ 34], | 30.00th=[ 35], 40.00th=[ 37], 50.00th=[ 59], 60.00th=[ 60], | 70.00th=[ 121], 80.00th=[ 121], 90.00th=[ 130], 95.00th=[ 218], | 99.00th=[ 218], 99.50th=[ 218], 99.90th=[ 218], 99.95th=[ 218], | 99.99th=[ 218] bw ( KiB/s): min= 8078, max= 8078, per=25.53%, avg=8078.00, stdev= 0.00, samples=1 iops : min= 7, max= 7, avg= 7.00, stdev= 0.00, samples=1 lat (msec) : 50=41.67%, 100=33.33%, 250=25.00% cpu : usr=0.00%, sys=33.81%, ctx=83, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9459: Sat Feb 24 23:42:07 2024 read: IOPS=7, BW=8084KiB/s (8278kB/s)(6144KiB/760msec) clat (usec): min=428, max=66228, avg=43900.48, stdev=25182.07 lat (usec): min=429, max=66230, avg=43901.75, stdev=25182.42 clat percentiles (usec): | 1.00th=[ 429], 5.00th=[ 429], 10.00th=[ 429], 20.00th=[28705], | 30.00th=[28705], 40.00th=[49546], 50.00th=[49546], 60.00th=[54264], | 70.00th=[64226], 80.00th=[64226], 90.00th=[66323], 95.00th=[66323], | 99.00th=[66323], 99.50th=[66323], 99.90th=[66323], 99.95th=[66323], | 99.99th=[66323] bw ( KiB/s): min= 4096, max= 4096, per=18.12%, avg=4096.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 write: IOPS=7, BW=8084KiB/s (8278kB/s)(6144KiB/760msec) clat (msec): min=31, max=121, avg=81.17, stdev=29.51 lat (msec): min=31, max=121, avg=81.62, stdev=29.56 clat percentiles (msec): | 1.00th=[ 32], 5.00th=[ 32], 10.00th=[ 32], 20.00th=[ 71], | 30.00th=[ 71], 40.00th=[ 83], 50.00th=[ 83], 60.00th=[ 90], | 70.00th=[ 92], 80.00th=[ 92], 90.00th=[ 123], 95.00th=[ 123], | 99.00th=[ 123], 99.50th=[ 123], 99.90th=[ 123], 99.95th=[ 123], | 99.99th=[ 123] bw ( KiB/s): min= 6144, max= 6144, per=19.41%, avg=6144.00, stdev= 0.00, samples=1 iops : min= 6, max= 6, avg= 6.00, stdev= 0.00, samples=1 lat (usec) : 500=8.33% lat (msec) : 50=25.00%, 100=58.33%, 250=8.33% cpu : usr=0.13%, sys=38.87%, ctx=82, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9461: Sat Feb 24 23:42:07 2024 read: IOPS=5, BW=5988KiB/s (6132kB/s)(5120KiB/855msec) clat (usec): min=381, max=73128, avg=15349.34, stdev=32307.78 lat (usec): min=381, max=73129, avg=15350.22, stdev=32307.67 clat percentiles (usec): | 1.00th=[ 383], 5.00th=[ 383], 10.00th=[ 383], 20.00th=[ 383], | 30.00th=[ 537], 40.00th=[ 537], 50.00th=[ 553], 60.00th=[ 553], | 70.00th=[ 2147], 80.00th=[ 2147], 90.00th=[72877], 95.00th=[72877], | 99.00th=[72877], 99.50th=[72877], 99.90th=[72877], 99.95th=[72877], | 99.99th=[72877] bw ( KiB/s): min= 2048, max= 2048, per=9.06%, avg=2048.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=8, BW=8384KiB/s (8585kB/s)(7168KiB/855msec) clat (msec): min=46, max=197, avg=109.61, stdev=46.76 lat (msec): min=46, max=197, avg=109.71, stdev=46.76 clat percentiles (msec): | 1.00th=[ 47], 5.00th=[ 47], 10.00th=[ 47], 20.00th=[ 86], | 30.00th=[ 90], 40.00th=[ 90], 50.00th=[ 103], 60.00th=[ 120], | 70.00th=[ 120], 80.00th=[ 127], 90.00th=[ 199], 95.00th=[ 199], | 99.00th=[ 199], 99.50th=[ 199], 99.90th=[ 199], 99.95th=[ 199], | 99.99th=[ 199] bw ( KiB/s): min= 8192, max= 8192, per=25.89%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 lat (usec) : 500=8.33%, 750=16.67% lat (msec) : 4=8.33%, 50=8.33%, 100=25.00%, 250=33.33% cpu : usr=0.12%, sys=24.24%, ctx=93, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=22.1MiB/s (23.1MB/s), 2260KiB/s-15.7MiB/s (2315kB/s-16.5MB/s), io=20.0MiB (20.0MB), run=445-906msec WRITE: bw=30.9MiB/s (32.4MB/s), 8084KiB/s-11.2MiB/s (8278kB/s-11.8MB/s), io=28.0MiB (29.4MB), run=445-906msec rand-rw: (groupid=0, jobs=1): err= 0: pid=9454: Sat Feb 24 23:42:07 2024 read: IOPS=11, BW=11.4MiB/s (11.0MB/s)(7168KiB/614msec) clat (usec): min=23372, max=80799, avg=45787.83, stdev=20286.85 lat (usec): min=23373, max=80800, avg=45789.02, stdev=20286.85 clat percentiles (usec): | 1.00th=[23462], 5.00th=[23462], 10.00th=[23462], 20.00th=[26608], | 30.00th=[36963], 40.00th=[36963], 50.00th=[39584], 60.00th=[53216], | 70.00th=[53216], 80.00th=[60031], 90.00th=[81265], 95.00th=[81265], | 99.00th=[81265], 99.50th=[81265], 99.90th=[81265], 99.95th=[81265], | 99.99th=[81265] bw ( KiB/s): min=10098, max=10098, per=47.34%, avg=10098.00, stdev= 0.00, samples=1 iops : min= 9, max= 9, avg= 9.00, stdev= 0.00, samples=1 write: IOPS=8, BW=8339KiB/s (8539kB/s)(5120KiB/614msec) clat (usec): min=35237, max=83299, avg=58060.21, stdev=18501.69 lat (usec): min=35321, max=83392, avg=58144.13, stdev=18505.75 clat percentiles (usec): | 1.00th=[35390], 5.00th=[35390], 10.00th=[35390], 20.00th=[35390], | 30.00th=[47449], 40.00th=[47449], 50.00th=[56361], 60.00th=[56361], | 70.00th=[67634], 80.00th=[67634], 90.00th=[83362], 95.00th=[83362], | 99.00th=[83362], 99.50th=[83362], 99.90th=[83362], 99.95th=[83362], | 99.99th=[83362] bw ( KiB/s): min= 8078, max= 8078, per=27.05%, avg=8078.00, stdev= 0.00, samples=1 iops : min= 7, max= 7, avg= 7.00, stdev= 0.00, samples=1 lat (msec) : 50=50.00%, 100=50.00% cpu : usr=0.16%, sys=26.43%, ctx=150, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9456: Sat Feb 24 23:42:07 2024 read: IOPS=2, BW=2309KiB/s (2364kB/s)(2048KiB/887msec) clat (usec): min=26335, max=42184, avg=34260.03, stdev=11207.17 lat (usec): min=26338, max=42186, avg=34262.36, stdev=11206.36 clat percentiles (usec): | 1.00th=[26346], 5.00th=[26346], 10.00th=[26346], 20.00th=[26346], | 30.00th=[26346], 40.00th=[26346], 50.00th=[26346], 60.00th=[42206], | 70.00th=[42206], 80.00th=[42206], 90.00th=[42206], 95.00th=[42206], | 99.00th=[42206], 99.50th=[42206], 99.90th=[42206], 99.95th=[42206], | 99.99th=[42206] bw ( KiB/s): min= 4096, max= 4096, per=19.20%, avg=4096.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 write: IOPS=11, BW=11.3MiB/s (11.8MB/s)(10.0MiB/887msec) clat (msec): min=28, max=188, avg=80.92, stdev=56.42 lat (msec): min=28, max=188, avg=80.99, stdev=56.41 clat percentiles (msec): | 1.00th=[ 29], 5.00th=[ 29], 10.00th=[ 29], 20.00th=[ 32], | 30.00th=[ 41], 40.00th=[ 41], 50.00th=[ 43], 60.00th=[ 59], | 70.00th=[ 120], 80.00th=[ 120], 90.00th=[ 144], 95.00th=[ 190], | 99.00th=[ 190], 99.50th=[ 190], 99.90th=[ 190], 99.95th=[ 190], | 99.99th=[ 190] bw ( KiB/s): min=10240, max=10240, per=34.29%, avg=10240.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 lat (msec) : 50=58.33%, 100=8.33%, 250=33.33% cpu : usr=0.11%, sys=16.93%, ctx=49, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9458: Sat Feb 24 23:42:07 2024 read: IOPS=6, BW=6649KiB/s (6809kB/s)(6144KiB/924msec) clat (msec): min=48, max=104, avg=70.87, stdev=19.12 lat (msec): min=48, max=104, avg=70.87, stdev=19.12 clat percentiles (msec): | 1.00th=[ 49], 5.00th=[ 49], 10.00th=[ 49], 20.00th=[ 60], | 30.00th=[ 60], 40.00th=[ 64], 50.00th=[ 64], 60.00th=[ 75], | 70.00th=[ 75], 80.00th=[ 75], 90.00th=[ 105], 95.00th=[ 105], | 99.00th=[ 105], 99.50th=[ 105], 99.90th=[ 105], 99.95th=[ 105], | 99.99th=[ 105] bw ( KiB/s): min= 4055, max= 4055, per=19.01%, avg=4055.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 write: IOPS=6, BW=6649KiB/s (6809kB/s)(6144KiB/924msec) clat (msec): min=28, max=120, avg=81.79, stdev=30.93 lat (msec): min=28, max=120, avg=81.85, stdev=30.94 clat percentiles (msec): | 1.00th=[ 29], 5.00th=[ 29], 10.00th=[ 29], 20.00th=[ 69], | 30.00th=[ 69], 40.00th=[ 89], 50.00th=[ 89], 60.00th=[ 90], | 70.00th=[ 96], 80.00th=[ 96], 90.00th=[ 121], 95.00th=[ 121], | 99.00th=[ 121], 99.50th=[ 121], 99.90th=[ 121], 99.95th=[ 121], | 99.99th=[ 121] bw ( KiB/s): min= 8110, max= 8110, per=27.15%, avg=8110.00, stdev= 0.00, samples=1 iops : min= 7, max= 7, avg= 7.00, stdev= 0.00, samples=1 lat (msec) : 50=16.67%, 100=66.67%, 250=16.67% cpu : usr=0.00%, sys=16.90%, ctx=65, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9460: Sat Feb 24 23:42:07 2024 read: IOPS=5, BW=5333KiB/s (5461kB/s)(5120KiB/960msec) clat (usec): min=20544, max=50467, avg=33494.41, stdev=14615.21 lat (usec): min=20545, max=50468, avg=33495.78, stdev=14614.96 clat percentiles (usec): | 1.00th=[20579], 5.00th=[20579], 10.00th=[20579], 20.00th=[20579], | 30.00th=[21103], 40.00th=[21103], 50.00th=[27395], 60.00th=[27395], | 70.00th=[47973], 80.00th=[47973], 90.00th=[50594], 95.00th=[50594], | 99.00th=[50594], 99.50th=[50594], 99.90th=[50594], 99.95th=[50594], | 99.99th=[50594] bw ( KiB/s): min= 6144, max= 6144, per=28.80%, avg=6144.00, stdev= 0.00, samples=1 iops : min= 6, max= 6, avg= 6.00, stdev= 0.00, samples=1 write: IOPS=7, BW=7467KiB/s (7646kB/s)(7168KiB/960msec) clat (msec): min=69, max=197, avg=111.83, stdev=44.33 lat (msec): min=69, max=197, avg=111.90, stdev=44.32 clat percentiles (msec): | 1.00th=[ 70], 5.00th=[ 70], 10.00th=[ 70], 20.00th=[ 72], | 30.00th=[ 87], 40.00th=[ 87], 50.00th=[ 108], 60.00th=[ 115], | 70.00th=[ 115], 80.00th=[ 136], 90.00th=[ 197], 95.00th=[ 197], | 99.00th=[ 197], 99.50th=[ 197], 99.90th=[ 197], 99.95th=[ 197], | 99.99th=[ 197] bw ( KiB/s): min= 8192, max= 8192, per=27.43%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 lat (msec) : 50=33.33%, 100=33.33%, 250=33.33% cpu : usr=0.00%, sys=15.64%, ctx=86, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=20.8MiB/s (21.8MB/s), 2309KiB/s-11.4MiB/s (2364kB/s-11.0MB/s), io=20.0MiB (20.0MB), run=614-960msec WRITE: bw=29.2MiB/s (30.6MB/s), 6649KiB/s-11.3MiB/s (6809kB/s-11.8MB/s), io=28.0MiB (29.4MB), run=614-960msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=9468: Sat Feb 24 23:42:09 2024 read: IOPS=5, BW=22.8MiB/s (23.9MB/s)(12.0MiB/527msec) clat (msec): min=43, max=259, avg=175.02, stdev=115.17 lat (msec): min=43, max=259, avg=175.02, stdev=115.17 clat percentiles (msec): | 1.00th=[ 44], 5.00th=[ 44], 10.00th=[ 44], 20.00th=[ 44], | 30.00th=[ 44], 40.00th=[ 222], 50.00th=[ 222], 60.00th=[ 222], | 70.00th=[ 259], 80.00th=[ 259], 90.00th=[ 259], 95.00th=[ 259], | 99.00th=[ 259], 99.50th=[ 259], 99.90th=[ 259], 99.95th=[ 259], | 99.99th=[ 259] bw ( KiB/s): min=16384, max=16384, per=85.90%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 50=33.33%, 250=33.33%, 500=33.33% cpu : usr=0.00%, sys=56.65%, ctx=143, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9469: Sat Feb 24 23:42:09 2024 read: IOPS=1, BW=4768KiB/s (4883kB/s)(4096KiB/859msec) clat (nsec): min=186291k, max=186291k, avg=186291287.00, stdev= 0.00 lat (nsec): min=186295k, max=186295k, avg=186294534.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 186], 5.00th=[ 186], 10.00th=[ 186], 20.00th=[ 186], | 30.00th=[ 186], 40.00th=[ 186], 50.00th=[ 186], 60.00th=[ 186], | 70.00th=[ 186], 80.00th=[ 186], 90.00th=[ 186], 95.00th=[ 186], | 99.00th=[ 186], 99.50th=[ 186], 99.90th=[ 186], 99.95th=[ 186], | 99.99th=[ 186] bw ( KiB/s): min= 8192, max= 8192, per=42.95%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=2, BW=9537KiB/s (9766kB/s)(8192KiB/859msec) clat (msec): min=248, max=421, avg=334.73, stdev=122.57 lat (msec): min=248, max=421, avg=334.96, stdev=122.67 clat percentiles (msec): | 1.00th=[ 249], 5.00th=[ 249], 10.00th=[ 249], 20.00th=[ 249], | 30.00th=[ 249], 40.00th=[ 249], 50.00th=[ 249], 60.00th=[ 422], | 70.00th=[ 422], 80.00th=[ 422], 90.00th=[ 422], 95.00th=[ 422], | 99.00th=[ 422], 99.50th=[ 422], 99.90th=[ 422], 99.95th=[ 422], | 99.99th=[ 422] lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.00%, sys=31.70%, ctx=105, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9470: Sat Feb 24 23:42:09 2024 write: IOPS=3, BW=15.9MiB/s (16.6MB/s)(12.0MiB/756msec) clat (msec): min=198, max=296, avg=248.87, stdev=49.55 lat (msec): min=198, max=297, avg=249.23, stdev=49.55 clat percentiles (msec): | 1.00th=[ 199], 5.00th=[ 199], 10.00th=[ 199], 20.00th=[ 199], | 30.00th=[ 199], 40.00th=[ 251], 50.00th=[ 251], 60.00th=[ 251], | 70.00th=[ 296], 80.00th=[ 296], 90.00th=[ 296], 95.00th=[ 296], | 99.00th=[ 296], 99.50th=[ 296], 99.90th=[ 296], 99.95th=[ 296], | 99.99th=[ 296] bw ( KiB/s): min=16384, max=16384, per=42.95%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=66.67% cpu : usr=0.00%, sys=38.15%, ctx=59, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9471: Sat Feb 24 23:42:09 2024 write: IOPS=4, BW=18.3MiB/s (19.2MB/s)(12.0MiB/656msec) clat (msec): min=122, max=377, avg=212.24, stdev=143.19 lat (msec): min=122, max=377, avg=212.92, stdev=142.91 clat percentiles (msec): | 1.00th=[ 123], 5.00th=[ 123], 10.00th=[ 123], 20.00th=[ 123], | 30.00th=[ 123], 40.00th=[ 138], 50.00th=[ 138], 60.00th=[ 138], | 70.00th=[ 376], 80.00th=[ 376], 90.00th=[ 376], 95.00th=[ 376], | 99.00th=[ 376], 99.50th=[ 376], 99.90th=[ 376], 99.95th=[ 376], | 99.99th=[ 376] bw ( KiB/s): min=16384, max=16384, per=42.95%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.00%, sys=46.72%, ctx=33, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=18.6MiB/s (19.5MB/s), 4768KiB/s-22.8MiB/s (4883kB/s-23.9MB/s), io=16.0MiB (16.8MB), run=527-859msec WRITE: bw=37.3MiB/s (39.1MB/s), 9537KiB/s-18.3MiB/s (9766kB/s-19.2MB/s), io=32.0MiB (33.6MB), run=656-859msec rand-rw: (groupid=0, jobs=1): err= 0: pid=9472: Sat Feb 24 23:42:09 2024 read: IOPS=3, BW=13.4MiB/s (14.0MB/s)(12.0MiB/897msec) clat (msec): min=177, max=370, avg=296.40, stdev=104.30 lat (msec): min=177, max=370, avg=296.41, stdev=104.30 clat percentiles (msec): | 1.00th=[ 178], 5.00th=[ 178], 10.00th=[ 178], 20.00th=[ 178], | 30.00th=[ 178], 40.00th=[ 342], 50.00th=[ 342], 60.00th=[ 342], | 70.00th=[ 372], 80.00th=[ 372], 90.00th=[ 372], 95.00th=[ 372], | 99.00th=[ 372], 99.50th=[ 372], 99.90th=[ 372], 99.95th=[ 372], | 99.99th=[ 372] bw ( KiB/s): min= 8192, max= 8192, per=49.60%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=66.67% cpu : usr=0.00%, sys=13.17%, ctx=59, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9473: Sat Feb 24 23:42:09 2024 read: IOPS=1, BW=4129KiB/s (4228kB/s)(4096KiB/992msec) clat (nsec): min=470722k, max=470722k, avg=470722108.00, stdev= 0.00 lat (nsec): min=470723k, max=470723k, avg=470723236.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 472], 5.00th=[ 472], 10.00th=[ 472], 20.00th=[ 472], | 30.00th=[ 472], 40.00th=[ 472], 50.00th=[ 472], 60.00th=[ 472], | 70.00th=[ 472], 80.00th=[ 472], 90.00th=[ 472], 95.00th=[ 472], | 99.00th=[ 472], 99.50th=[ 472], 99.90th=[ 472], 99.95th=[ 472], | 99.99th=[ 472] bw ( KiB/s): min= 8192, max= 8192, per=49.60%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=2, BW=8258KiB/s (8456kB/s)(8192KiB/992msec) clat (msec): min=145, max=363, avg=254.39, stdev=154.12 lat (msec): min=145, max=363, avg=254.69, stdev=154.17 clat percentiles (msec): | 1.00th=[ 146], 5.00th=[ 146], 10.00th=[ 146], 20.00th=[ 146], | 30.00th=[ 146], 40.00th=[ 146], 50.00th=[ 146], 60.00th=[ 363], | 70.00th=[ 363], 80.00th=[ 363], 90.00th=[ 363], 95.00th=[ 363], | 99.00th=[ 363], 99.50th=[ 363], 99.90th=[ 363], 99.95th=[ 363], | 99.99th=[ 363] lat (msec) : 250=33.33%, 500=66.67% cpu : usr=0.00%, sys=15.94%, ctx=68, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9474: Sat Feb 24 23:42:09 2024 write: IOPS=3, BW=12.8MiB/s (13.5MB/s)(12.0MiB/935msec) clat (msec): min=202, max=494, avg=308.70, stdev=161.71 lat (msec): min=203, max=495, avg=308.99, stdev=161.83 clat percentiles (msec): | 1.00th=[ 203], 5.00th=[ 203], 10.00th=[ 203], 20.00th=[ 203], | 30.00th=[ 203], 40.00th=[ 228], 50.00th=[ 228], 60.00th=[ 228], | 70.00th=[ 493], 80.00th=[ 493], 90.00th=[ 493], 95.00th=[ 493], | 99.00th=[ 493], 99.50th=[ 493], 99.90th=[ 493], 99.95th=[ 493], | 99.99th=[ 493] lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.00%, sys=13.28%, ctx=38, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9475: Sat Feb 24 23:42:09 2024 write: IOPS=3, BW=14.7MiB/s (15.4MB/s)(12.0MiB/819msec) clat (msec): min=162, max=392, avg=271.45, stdev=115.55 lat (msec): min=162, max=393, avg=271.70, stdev=115.62 clat percentiles (msec): | 1.00th=[ 163], 5.00th=[ 163], 10.00th=[ 163], 20.00th=[ 163], | 30.00th=[ 163], 40.00th=[ 259], 50.00th=[ 259], 60.00th=[ 259], | 70.00th=[ 393], 80.00th=[ 393], 90.00th=[ 393], 95.00th=[ 393], | 99.00th=[ 393], 99.50th=[ 393], 99.90th=[ 393], 99.95th=[ 393], | 99.99th=[ 393] bw ( KiB/s): min= 8192, max= 8192, per=24.80%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=66.67% cpu : usr=0.37%, sys=24.45%, ctx=81, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=16.1MiB/s (16.9MB/s), 4129KiB/s-13.4MiB/s (4228kB/s-14.0MB/s), io=16.0MiB (16.8MB), run=897-992msec WRITE: bw=32.3MiB/s (33.8MB/s), 8258KiB/s-14.7MiB/s (8456kB/s-15.4MB/s), io=32.0MiB (33.6MB), run=819-992msec