== sanity test 398b: DIO and buffer IO race ============== 07:47:30 (1712144850) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 0.973905 s, 51.7 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=31565: Wed Apr 3 07:47:45 2024 read: IOPS=111, BW=445KiB/s (455kB/s)(5992KiB/13471msec) clat (usec): min=44, max=167574, avg=3997.35, stdev=5495.99 lat (usec): min=45, max=167575, avg=3998.83, stdev=5496.26 clat percentiles (usec): | 1.00th=[ 55], 5.00th=[ 67], 10.00th=[ 77], 20.00th=[ 130], | 30.00th=[ 2343], 40.00th=[ 2737], 50.00th=[ 3195], 60.00th=[ 3720], | 70.00th=[ 4752], 80.00th=[ 6194], 90.00th=[ 8160], 95.00th=[ 9896], | 99.00th=[ 14615], 99.50th=[ 22152], 99.90th=[ 39060], 99.95th=[166724], | 99.99th=[166724] bw ( KiB/s): min= 280, max= 680, per=24.83%, avg=442.38, stdev=96.26, samples=26 iops : min= 70, max= 170, avg=110.54, stdev=24.10, samples=26 write: IOPS=116, BW=467KiB/s (479kB/s)(6296KiB/13471msec) clat (usec): min=443, max=48418, avg=4700.23, stdev=4677.88 lat (usec): min=443, max=48419, avg=4701.29, stdev=4677.88 clat percentiles (usec): | 1.00th=[ 474], 5.00th=[ 515], 10.00th=[ 562], 20.00th=[ 734], | 30.00th=[ 1057], 40.00th=[ 2442], 50.00th=[ 3654], 60.00th=[ 4686], | 70.00th=[ 5866], 80.00th=[ 7767], 90.00th=[10814], 95.00th=[13435], | 99.00th=[19792], 99.50th=[23725], 99.90th=[45876], 99.95th=[48497], | 99.99th=[48497] bw ( KiB/s): min= 352, max= 680, per=24.81%, avg=462.96, stdev=89.04, samples=26 iops : min= 88, max= 170, avg=115.69, stdev=22.29, samples=26 lat (usec) : 50=0.39%, 100=6.87%, 250=3.26%, 500=1.86%, 750=9.08% lat (usec) : 1000=4.33% lat (msec) : 2=5.21%, 4=27.31%, 10=33.01%, 20=7.91%, 50=0.75% lat (msec) : 250=0.03% cpu : usr=0.27%, sys=24.03%, ctx=5923, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31566: Wed Apr 3 07:47:45 2024 read: IOPS=110, BW=441KiB/s (452kB/s)(5940KiB/13464msec) clat (usec): min=42, max=64566, avg=3963.57, stdev=4003.88 lat (usec): min=43, max=64566, avg=3964.30, stdev=4003.87 clat percentiles (usec): | 1.00th=[ 47], 5.00th=[ 67], 10.00th=[ 73], 20.00th=[ 127], | 30.00th=[ 2311], 40.00th=[ 2802], 50.00th=[ 3294], 60.00th=[ 3949], | 70.00th=[ 4817], 80.00th=[ 6194], 90.00th=[ 8160], 95.00th=[ 9765], | 99.00th=[15926], 99.50th=[19530], 99.90th=[61080], 99.95th=[64750], | 99.99th=[64750] bw ( KiB/s): min= 272, max= 568, per=24.76%, avg=441.15, stdev=76.70, samples=26 iops : min= 68, max= 142, avg=110.23, stdev=19.21, samples=26 write: IOPS=117, BW=471KiB/s (483kB/s)(6348KiB/13464msec) clat (usec): min=389, max=164944, avg=4721.76, stdev=5931.59 lat (usec): min=389, max=164944, avg=4722.79, stdev=5931.85 clat percentiles (usec): | 1.00th=[ 478], 5.00th=[ 515], 10.00th=[ 562], 20.00th=[ 742], | 30.00th=[ 1004], 40.00th=[ 2343], 50.00th=[ 3621], 60.00th=[ 4752], | 70.00th=[ 5866], 80.00th=[ 7767], 90.00th=[ 11076], 95.00th=[ 13566], | 99.00th=[ 18744], 99.50th=[ 21103], 99.90th=[ 27395], 99.95th=[164627], | 99.99th=[164627] bw ( KiB/s): min= 320, max= 696, per=24.91%, avg=464.81, stdev=94.43, samples=26 iops : min= 80, max= 174, avg=116.15, stdev=23.59, samples=26 lat (usec) : 50=0.62%, 100=7.10%, 250=2.86%, 500=1.92%, 750=9.18% lat (usec) : 1000=4.75% lat (msec) : 2=5.05%, 4=25.78%, 10=34.28%, 20=7.94%, 50=0.42% lat (msec) : 100=0.07%, 250=0.03% cpu : usr=0.22%, sys=24.03%, ctx=5978, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31567: Wed Apr 3 07:47:45 2024 read: IOPS=111, BW=448KiB/s (459kB/s)(6028KiB/13457msec) clat (usec): min=44, max=65115, avg=3877.26, stdev=3686.15 lat (usec): min=44, max=65115, avg=3877.93, stdev=3686.15 clat percentiles (usec): | 1.00th=[ 49], 5.00th=[ 66], 10.00th=[ 74], 20.00th=[ 117], | 30.00th=[ 2409], 40.00th=[ 2802], 50.00th=[ 3294], 60.00th=[ 3851], | 70.00th=[ 4686], 80.00th=[ 6128], 90.00th=[ 8094], 95.00th=[ 9634], | 99.00th=[13566], 99.50th=[17957], 99.90th=[41157], 99.95th=[65274], | 99.99th=[65274] bw ( KiB/s): min= 288, max= 720, per=25.03%, avg=446.04, stdev=91.65, samples=26 iops : min= 72, max= 180, avg=111.42, stdev=22.92, samples=26 write: IOPS=116, BW=465KiB/s (476kB/s)(6260KiB/13457msec) clat (usec): min=453, max=167906, avg=4810.78, stdev=6274.97 lat (usec): min=454, max=167907, avg=4813.12, stdev=6274.45 clat percentiles (usec): | 1.00th=[ 478], 5.00th=[ 506], 10.00th=[ 562], 20.00th=[ 725], | 30.00th=[ 955], 40.00th=[ 2057], 50.00th=[ 3785], 60.00th=[ 4883], | 70.00th=[ 6194], 80.00th=[ 7767], 90.00th=[ 10421], 95.00th=[ 13304], | 99.00th=[ 20841], 99.50th=[ 23462], 99.90th=[ 50070], 99.95th=[168821], | 99.99th=[168821] bw ( KiB/s): min= 296, max= 672, per=24.69%, avg=460.77, stdev=101.35, samples=26 iops : min= 74, max= 168, avg=115.12, stdev=25.34, samples=26 lat (usec) : 50=0.55%, 100=7.42%, 250=2.93%, 500=2.18%, 750=8.89% lat (usec) : 1000=4.88% lat (msec) : 2=5.18%, 4=25.13%, 10=34.96%, 20=7.06%, 50=0.75% lat (msec) : 100=0.03%, 250=0.03% cpu : usr=0.18%, sys=23.79%, ctx=6037, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31568: Wed Apr 3 07:47:45 2024 read: IOPS=112, BW=452KiB/s (462kB/s)(6052KiB/13400msec) clat (usec): min=43, max=52252, avg=3710.26, stdev=3381.39 lat (usec): min=44, max=52253, avg=3711.08, stdev=3381.31 clat percentiles (usec): | 1.00th=[ 48], 5.00th=[ 65], 10.00th=[ 73], 20.00th=[ 111], | 30.00th=[ 2245], 40.00th=[ 2671], 50.00th=[ 3163], 60.00th=[ 3752], | 70.00th=[ 4621], 80.00th=[ 5932], 90.00th=[ 8029], 95.00th=[ 9634], | 99.00th=[13304], 99.50th=[16450], 99.90th=[20579], 99.95th=[52167], | 99.99th=[52167] bw ( KiB/s): min= 192, max= 632, per=25.09%, avg=447.04, stdev=98.52, samples=26 iops : min= 48, max= 158, avg=111.73, stdev=24.63, samples=26 write: IOPS=116, BW=465KiB/s (477kB/s)(6236KiB/13400msec) clat (usec): min=434, max=207522, avg=4939.36, stdev=7153.62 lat (usec): min=435, max=207523, avg=4940.79, stdev=7153.41 clat percentiles (usec): | 1.00th=[ 478], 5.00th=[ 510], 10.00th=[ 562], 20.00th=[ 725], | 30.00th=[ 979], 40.00th=[ 2311], 50.00th=[ 3851], 60.00th=[ 4883], | 70.00th=[ 6063], 80.00th=[ 7832], 90.00th=[ 10945], 95.00th=[ 13960], | 99.00th=[ 22414], 99.50th=[ 26346], 99.90th=[ 73925], 99.95th=[206570], | 99.99th=[206570] bw ( KiB/s): min= 208, max= 664, per=24.75%, avg=461.81, stdev=96.58, samples=26 iops : min= 52, max= 166, avg=115.42, stdev=24.15, samples=26 lat (usec) : 50=0.55%, 100=7.65%, 250=3.68%, 500=2.08%, 750=8.76% lat (usec) : 1000=4.75% lat (msec) : 2=4.92%, 4=25.39%, 10=33.63%, 20=7.85%, 50=0.65% lat (msec) : 100=0.07%, 250=0.03% cpu : usr=0.18%, sys=23.73%, ctx=5995, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1782KiB/s (1825kB/s), 441KiB/s-452KiB/s (452kB/s-462kB/s), io=23.4MiB (24.6MB), run=13400-13471msec WRITE: bw=1866KiB/s (1911kB/s), 465KiB/s-471KiB/s (476kB/s-483kB/s), io=24.6MiB (25.7MB), run=13400-13471msec rand-rw: (groupid=0, jobs=1): err= 0: pid=31569: Wed Apr 3 07:48:17 2024 read: IOPS=33, BW=134KiB/s (137kB/s)(5992KiB/44835msec) clat (usec): min=1582, max=170348, avg=4287.17, stdev=5403.58 lat (usec): min=1582, max=170351, avg=4288.13, stdev=5403.60 clat percentiles (usec): | 1.00th=[ 1778], 5.00th=[ 1942], 10.00th=[ 2057], 20.00th=[ 2245], | 30.00th=[ 2409], 40.00th=[ 2671], 50.00th=[ 3064], 60.00th=[ 3720], | 70.00th=[ 4293], 80.00th=[ 5342], 90.00th=[ 7504], 95.00th=[ 10159], | 99.00th=[ 17171], 99.50th=[ 20055], 99.90th=[ 47449], 99.95th=[170918], | 99.99th=[170918] bw ( KiB/s): min= 40, max= 264, per=25.07%, avg=133.38, stdev=51.63, samples=89 iops : min= 10, max= 66, avg=33.27, stdev=12.94, samples=89 write: IOPS=35, BW=140KiB/s (144kB/s)(6296KiB/44835msec) clat (msec): min=12, max=104, avg=24.35, stdev= 9.17 lat (msec): min=12, max=104, avg=24.35, stdev= 9.17 clat percentiles (msec): | 1.00th=[ 14], 5.00th=[ 15], 10.00th=[ 16], 20.00th=[ 17], | 30.00th=[ 19], 40.00th=[ 21], 50.00th=[ 23], 60.00th=[ 25], | 70.00th=[ 27], 80.00th=[ 30], 90.00th=[ 37], 95.00th=[ 44], | 99.00th=[ 55], 99.50th=[ 61], 99.90th=[ 75], 99.95th=[ 105], | 99.99th=[ 105] bw ( KiB/s): min= 48, max= 200, per=25.16%, avg=140.12, stdev=38.28, samples=89 iops : min= 12, max= 50, avg=34.96, stdev= 9.63, samples=89 lat (msec) : 2=3.52%, 4=28.29%, 10=14.42%, 20=21.09%, 50=31.64% lat (msec) : 100=0.98%, 250=0.07% cpu : usr=0.13%, sys=4.69%, ctx=3681, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31570: Wed Apr 3 07:48:17 2024 read: IOPS=32, BW=132KiB/s (135kB/s)(5940KiB/45079msec) clat (usec): min=1558, max=186309, avg=4345.23, stdev=5797.48 lat (usec): min=1558, max=186311, avg=4346.21, stdev=5797.51 clat percentiles (usec): | 1.00th=[ 1795], 5.00th=[ 1942], 10.00th=[ 2073], 20.00th=[ 2278], | 30.00th=[ 2442], 40.00th=[ 2704], 50.00th=[ 3097], 60.00th=[ 3785], | 70.00th=[ 4424], 80.00th=[ 5342], 90.00th=[ 7373], 95.00th=[ 10290], | 99.00th=[ 17695], 99.50th=[ 21103], 99.90th=[ 58459], 99.95th=[185598], | 99.99th=[185598] bw ( KiB/s): min= 16, max= 247, per=24.74%, avg=131.62, stdev=53.24, samples=90 iops : min= 4, max= 61, avg=32.81, stdev=13.31, samples=90 write: IOPS=35, BW=141KiB/s (144kB/s)(6348KiB/45079msec) clat (usec): min=10731, max=65687, avg=24286.30, stdev=8560.81 lat (usec): min=10732, max=65688, avg=24287.45, stdev=8560.74 clat percentiles (usec): | 1.00th=[12911], 5.00th=[14484], 10.00th=[15401], 20.00th=[17171], | 30.00th=[18744], 40.00th=[20841], 50.00th=[22676], 60.00th=[24511], | 70.00th=[26346], 80.00th=[28967], 90.00th=[36439], 95.00th=[42206], | 99.00th=[52167], 99.50th=[57934], 99.90th=[64750], 99.95th=[65799], | 99.99th=[65799] bw ( KiB/s): min= 56, max= 221, per=25.22%, avg=140.47, stdev=36.39, samples=90 iops : min= 14, max= 55, avg=35.02, stdev= 9.09, samples=90 lat (msec) : 2=3.45%, 4=27.05%, 10=15.14%, 20=20.93%, 50=32.71% lat (msec) : 100=0.68%, 250=0.03% cpu : usr=0.12%, sys=4.67%, ctx=3686, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31571: Wed Apr 3 07:48:17 2024 read: IOPS=33, BW=134KiB/s (138kB/s)(6028KiB/44847msec) clat (usec): min=1564, max=63196, avg=4419.77, stdev=4021.16 lat (usec): min=1565, max=63197, avg=4420.66, stdev=4021.15 clat percentiles (usec): | 1.00th=[ 1745], 5.00th=[ 1942], 10.00th=[ 2073], 20.00th=[ 2245], | 30.00th=[ 2507], 40.00th=[ 2769], 50.00th=[ 3261], 60.00th=[ 3916], | 70.00th=[ 4490], 80.00th=[ 5538], 90.00th=[ 7963], 95.00th=[10552], | 99.00th=[18220], 99.50th=[23462], 99.90th=[58459], 99.95th=[63177], | 99.99th=[63177] bw ( KiB/s): min= 40, max= 264, per=25.22%, avg=134.18, stdev=51.40, samples=89 iops : min= 10, max= 66, avg=33.46, stdev=12.88, samples=89 write: IOPS=34, BW=140KiB/s (143kB/s)(6260KiB/44847msec) clat (msec): min=12, max=204, avg=24.35, stdev= 9.80 lat (msec): min=12, max=204, avg=24.35, stdev= 9.80 clat percentiles (msec): | 1.00th=[ 14], 5.00th=[ 15], 10.00th=[ 16], 20.00th=[ 18], | 30.00th=[ 20], 40.00th=[ 21], 50.00th=[ 23], 60.00th=[ 25], | 70.00th=[ 27], 80.00th=[ 29], 90.00th=[ 36], 95.00th=[ 44], | 99.00th=[ 55], 99.50th=[ 61], 99.90th=[ 77], 99.95th=[ 205], | 99.99th=[ 205] bw ( KiB/s): min= 56, max= 192, per=24.98%, avg=139.12, stdev=36.98, samples=89 iops : min= 14, max= 48, avg=34.70, stdev= 9.30, samples=89 lat (msec) : 2=3.35%, 4=26.99%, 10=15.79%, 20=20.77%, 50=32.03% lat (msec) : 100=1.04%, 250=0.03% cpu : usr=0.09%, sys=4.75%, ctx=3699, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31572: Wed Apr 3 07:48:17 2024 read: IOPS=33, BW=135KiB/s (138kB/s)(6052KiB/44762msec) clat (usec): min=1532, max=63710, avg=4368.15, stdev=4111.56 lat (usec): min=1533, max=63711, avg=4369.04, stdev=4111.56 clat percentiles (usec): | 1.00th=[ 1762], 5.00th=[ 1926], 10.00th=[ 2024], 20.00th=[ 2212], | 30.00th=[ 2442], 40.00th=[ 2671], 50.00th=[ 3032], 60.00th=[ 3752], | 70.00th=[ 4424], 80.00th=[ 5407], 90.00th=[ 7898], 95.00th=[10552], | 99.00th=[21103], 99.50th=[24249], 99.90th=[57410], 99.95th=[63701], | 99.99th=[63701] bw ( KiB/s): min= 16, max= 280, per=25.38%, avg=135.01, stdev=51.98, samples=89 iops : min= 4, max= 70, avg=33.69, stdev=13.01, samples=89 write: IOPS=34, BW=139KiB/s (143kB/s)(6236KiB/44762msec) clat (msec): min=12, max=205, avg=24.42, stdev= 9.80 lat (msec): min=12, max=205, avg=24.42, stdev= 9.80 clat percentiles (msec): | 1.00th=[ 14], 5.00th=[ 15], 10.00th=[ 16], 20.00th=[ 17], | 30.00th=[ 19], 40.00th=[ 21], 50.00th=[ 23], 60.00th=[ 25], | 70.00th=[ 27], 80.00th=[ 30], 90.00th=[ 36], 95.00th=[ 42], | 99.00th=[ 54], 99.50th=[ 59], 99.90th=[ 72], 99.95th=[ 205], | 99.99th=[ 205] bw ( KiB/s): min= 56, max= 192, per=24.97%, avg=139.06, stdev=36.91, samples=89 iops : min= 14, max= 48, avg=34.70, stdev= 9.25, samples=89 lat (msec) : 2=4.33%, 4=27.28%, 10=14.68%, 20=20.96%, 50=31.77% lat (msec) : 100=0.94%, 250=0.03% cpu : usr=0.10%, sys=4.69%, ctx=3712, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=533KiB/s (545kB/s), 132KiB/s-135KiB/s (135kB/s-138kB/s), io=23.4MiB (24.6MB), run=44762-45079msec WRITE: bw=558KiB/s (571kB/s), 139KiB/s-141KiB/s (143kB/s-144kB/s), io=24.6MiB (25.7MB), run=44762-45079msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=31620: Wed Apr 3 07:48:23 2024 read: IOPS=77, BW=1233KiB/s (1263kB/s)(5920KiB/4801msec) clat (usec): min=48, max=45642, avg=9298.24, stdev=5752.78 lat (usec): min=48, max=45643, avg=9299.05, stdev=5752.79 clat percentiles (usec): | 1.00th=[ 62], 5.00th=[ 71], 10.00th=[ 77], 20.00th=[ 273], | 30.00th=[ 8848], 40.00th=[ 9634], 50.00th=[10159], 60.00th=[10945], | 70.00th=[11731], 80.00th=[13042], 90.00th=[14615], 95.00th=[16909], | 99.00th=[23725], 99.50th=[28705], 99.90th=[45876], 99.95th=[45876], | 99.99th=[45876] bw ( KiB/s): min= 864, max= 1472, per=24.20%, avg=1201.44, stdev=191.74, samples=9 iops : min= 54, max= 92, avg=75.00, stdev=11.90, samples=9 write: IOPS=82, BW=1326KiB/s (1358kB/s)(6368KiB/4801msec) clat (usec): min=647, max=66918, avg=3396.47, stdev=5349.43 lat (usec): min=648, max=66919, avg=3397.58, stdev=5349.43 clat percentiles (usec): | 1.00th=[ 668], 5.00th=[ 717], 10.00th=[ 766], 20.00th=[ 865], | 30.00th=[ 971], 40.00th=[ 1172], 50.00th=[ 1811], 60.00th=[ 2802], | 70.00th=[ 3359], 80.00th=[ 4752], 90.00th=[ 6718], 95.00th=[10683], | 99.00th=[24249], 99.50th=[45351], 99.90th=[66847], 99.95th=[66847], | 99.99th=[66847] bw ( KiB/s): min= 864, max= 1760, per=24.98%, avg=1304.44, stdev=315.27, samples=9 iops : min= 54, max= 110, avg=81.44, stdev=19.59, samples=9 lat (usec) : 50=0.13%, 100=6.64%, 250=2.73%, 500=0.39%, 750=4.69% lat (usec) : 1000=12.11% lat (msec) : 2=10.29%, 4=12.63%, 10=22.40%, 20=26.43%, 50=1.43% lat (msec) : 100=0.13% cpu : usr=0.19%, sys=21.94%, ctx=1968, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31621: Wed Apr 3 07:48:23 2024 read: IOPS=74, BW=1190KiB/s (1219kB/s)(5744KiB/4825msec) clat (usec): min=48, max=32808, avg=9277.87, stdev=5578.03 lat (usec): min=48, max=32808, avg=9279.38, stdev=5579.04 clat percentiles (usec): | 1.00th=[ 59], 5.00th=[ 69], 10.00th=[ 72], 20.00th=[ 172], | 30.00th=[ 8356], 40.00th=[ 9503], 50.00th=[10421], 60.00th=[11076], | 70.00th=[12256], 80.00th=[13173], 90.00th=[15270], 95.00th=[16188], | 99.00th=[21890], 99.50th=[31589], 99.90th=[32900], 99.95th=[32900], | 99.99th=[32900] bw ( KiB/s): min= 512, max= 1632, per=23.70%, avg=1176.22, stdev=326.97, samples=9 iops : min= 32, max= 102, avg=73.33, stdev=20.39, samples=9 write: IOPS=84, BW=1356KiB/s (1389kB/s)(6544KiB/4825msec) clat (usec): min=606, max=111819, avg=3634.82, stdev=7234.19 lat (usec): min=606, max=111821, avg=3635.93, stdev=7234.24 clat percentiles (usec): | 1.00th=[ 627], 5.00th=[ 701], 10.00th=[ 734], 20.00th=[ 816], | 30.00th=[ 971], 40.00th=[ 1139], 50.00th=[ 1844], 60.00th=[ 2933], | 70.00th=[ 3425], 80.00th=[ 5014], 90.00th=[ 6718], 95.00th=[ 9110], | 99.00th=[ 32375], 99.50th=[ 38011], 99.90th=[111674], 99.95th=[111674], | 99.99th=[111674] bw ( KiB/s): min= 832, max= 1696, per=25.18%, avg=1314.67, stdev=346.91, samples=9 iops : min= 52, max= 106, avg=82.00, stdev=21.52, samples=9 lat (usec) : 50=0.26%, 100=6.90%, 250=2.47%, 750=6.51%, 1000=11.33% lat (msec) : 2=9.24%, 4=12.37%, 10=23.70%, 20=25.52%, 50=1.43% lat (msec) : 100=0.13%, 250=0.13% cpu : usr=0.17%, sys=21.31%, ctx=1964, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31622: Wed Apr 3 07:48:23 2024 read: IOPS=82, BW=1313KiB/s (1344kB/s)(6256KiB/4765msec) clat (usec): min=47, max=49805, avg=8906.19, stdev=6025.14 lat (usec): min=47, max=49805, avg=8906.85, stdev=6025.17 clat percentiles (usec): | 1.00th=[ 50], 5.00th=[ 64], 10.00th=[ 71], 20.00th=[ 109], | 30.00th=[ 8029], 40.00th=[ 9241], 50.00th=[10159], 60.00th=[11076], | 70.00th=[12125], 80.00th=[13173], 90.00th=[14877], 95.00th=[16581], | 99.00th=[22152], 99.50th=[27132], 99.90th=[49546], 99.95th=[49546], | 99.99th=[49546] bw ( KiB/s): min= 672, max= 1760, per=25.99%, avg=1290.33, stdev=386.21, samples=9 iops : min= 42, max= 110, avg=80.56, stdev=24.11, samples=9 write: IOPS=79, BW=1266KiB/s (1296kB/s)(6032KiB/4765msec) clat (usec): min=631, max=102994, avg=3379.67, stdev=6360.70 lat (usec): min=631, max=102996, avg=3380.94, stdev=6360.73 clat percentiles (usec): | 1.00th=[ 652], 5.00th=[ 709], 10.00th=[ 750], 20.00th=[ 816], | 30.00th=[ 914], 40.00th=[ 1074], 50.00th=[ 1287], 60.00th=[ 2606], | 70.00th=[ 3228], 80.00th=[ 5342], 90.00th=[ 7308], 95.00th=[ 9765], | 99.00th=[ 22152], 99.50th=[ 34341], 99.90th=[103285], 99.95th=[103285], | 99.99th=[103285] bw ( KiB/s): min= 832, max= 1920, per=23.42%, avg=1222.78, stdev=311.74, samples=9 iops : min= 52, max= 120, avg=76.33, stdev=19.47, samples=9 lat (usec) : 50=0.52%, 100=9.24%, 250=2.73%, 750=5.34%, 1000=12.11% lat (msec) : 2=9.90%, 4=9.64%, 10=22.27%, 20=27.08%, 50=1.04% lat (msec) : 250=0.13% cpu : usr=0.23%, sys=20.86%, ctx=1982, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31623: Wed Apr 3 07:48:23 2024 read: IOPS=79, BW=1269KiB/s (1300kB/s)(6032KiB/4753msec) clat (usec): min=47, max=41127, avg=8980.37, stdev=5815.41 lat (usec): min=47, max=41128, avg=8980.99, stdev=5815.41 clat percentiles (usec): | 1.00th=[ 56], 5.00th=[ 69], 10.00th=[ 73], 20.00th=[ 119], | 30.00th=[ 8356], 40.00th=[ 9241], 50.00th=[ 9896], 60.00th=[10814], | 70.00th=[11731], 80.00th=[13042], 90.00th=[14484], 95.00th=[16712], | 99.00th=[25035], 99.50th=[26346], 99.90th=[41157], 99.95th=[41157], | 99.99th=[41157] bw ( KiB/s): min= 416, max= 1664, per=24.70%, avg=1226.33, stdev=340.14, samples=9 iops : min= 26, max= 104, avg=76.56, stdev=21.26, samples=9 write: IOPS=82, BW=1316KiB/s (1348kB/s)(6256KiB/4753msec) clat (usec): min=614, max=143057, avg=3479.21, stdev=8573.80 lat (usec): min=616, max=143058, avg=3480.34, stdev=8573.83 clat percentiles (usec): | 1.00th=[ 627], 5.00th=[ 685], 10.00th=[ 717], 20.00th=[ 799], | 30.00th=[ 873], 40.00th=[ 1020], 50.00th=[ 1188], 60.00th=[ 2245], | 70.00th=[ 3097], 80.00th=[ 4424], 90.00th=[ 6718], 95.00th=[ 8717], | 99.00th=[ 34341], 99.50th=[ 42206], 99.90th=[143655], 99.95th=[143655], | 99.99th=[143655] bw ( KiB/s): min= 512, max= 1952, per=24.71%, avg=1290.33, stdev=393.77, samples=9 iops : min= 32, max= 122, avg=80.56, stdev=24.67, samples=9 lat (usec) : 50=0.39%, 100=7.55%, 250=3.12%, 750=6.64%, 1000=12.89% lat (msec) : 2=11.20%, 4=9.24%, 10=22.79%, 20=24.09%, 50=1.95% lat (msec) : 250=0.13% cpu : usr=0.15%, sys=20.73%, ctx=1845, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=4964KiB/s (5083kB/s), 1190KiB/s-1313KiB/s (1219kB/s-1344kB/s), io=23.4MiB (24.5MB), run=4753-4825msec WRITE: bw=5223KiB/s (5348kB/s), 1266KiB/s-1356KiB/s (1296kB/s-1389kB/s), io=24.6MiB (25.8MB), run=4753-4825msec rand-rw: (groupid=0, jobs=1): err= 0: pid=31615: Wed Apr 3 07:48:29 2024 read: IOPS=32, BW=527KiB/s (539kB/s)(5920KiB/11240msec) clat (usec): min=1941, max=40441, avg=4573.14, stdev=4200.39 lat (usec): min=1941, max=40442, avg=4574.02, stdev=4200.41 clat percentiles (usec): | 1.00th=[ 2057], 5.00th=[ 2245], 10.00th=[ 2376], 20.00th=[ 2540], | 30.00th=[ 2704], 40.00th=[ 2933], 50.00th=[ 3261], 60.00th=[ 3621], | 70.00th=[ 3982], 80.00th=[ 4948], 90.00th=[ 8029], 95.00th=[12649], | 99.00th=[24773], 99.50th=[31065], 99.90th=[40633], 99.95th=[40633], | 99.99th=[40633] bw ( KiB/s): min= 256, max= 832, per=24.63%, avg=519.05, stdev=153.29, samples=22 iops : min= 16, max= 52, avg=32.32, stdev= 9.59, samples=22 write: IOPS=35, BW=567KiB/s (580kB/s)(6368KiB/11240msec) clat (usec): min=13038, max=56401, avg=23963.37, stdev=7481.35 lat (usec): min=13039, max=56402, avg=23965.10, stdev=7481.33 clat percentiles (usec): | 1.00th=[13435], 5.00th=[14484], 10.00th=[15664], 20.00th=[17433], | 30.00th=[19530], 40.00th=[21365], 50.00th=[22938], 60.00th=[24249], | 70.00th=[26084], 80.00th=[28967], 90.00th=[34341], 95.00th=[39060], | 99.00th=[47449], 99.50th=[54264], 99.90th=[56361], 99.95th=[56361], | 99.99th=[56361] bw ( KiB/s): min= 320, max= 768, per=25.38%, avg=562.68, stdev=131.18, samples=22 iops : min= 20, max= 48, avg=35.05, stdev= 8.20, samples=22 lat (msec) : 2=0.13%, 4=33.59%, 10=10.81%, 20=19.40%, 50=35.81% lat (msec) : 100=0.26% cpu : usr=0.10%, sys=6.05%, ctx=871, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31616: Wed Apr 3 07:48:29 2024 read: IOPS=31, BW=506KiB/s (518kB/s)(5744KiB/11363msec) clat (usec): min=2002, max=31035, avg=4544.71, stdev=3837.61 lat (usec): min=2002, max=31037, avg=4545.60, stdev=3837.62 clat percentiles (usec): | 1.00th=[ 2114], 5.00th=[ 2278], 10.00th=[ 2343], 20.00th=[ 2540], | 30.00th=[ 2737], 40.00th=[ 2900], 50.00th=[ 3163], 60.00th=[ 3523], | 70.00th=[ 4015], 80.00th=[ 4948], 90.00th=[ 8029], 95.00th=[13566], | 99.00th=[20055], 99.50th=[22938], 99.90th=[31065], 99.95th=[31065], | 99.99th=[31065] bw ( KiB/s): min= 160, max= 992, per=23.81%, avg=501.64, stdev=191.45, samples=22 iops : min= 10, max= 62, avg=31.27, stdev=11.91, samples=22 write: IOPS=35, BW=576KiB/s (590kB/s)(6544KiB/11363msec) clat (msec): min=12, max=111, avg=23.77, stdev= 8.90 lat (msec): min=12, max=111, avg=23.77, stdev= 8.90 clat percentiles (msec): | 1.00th=[ 14], 5.00th=[ 15], 10.00th=[ 16], 20.00th=[ 18], | 30.00th=[ 20], 40.00th=[ 21], 50.00th=[ 23], 60.00th=[ 24], | 70.00th=[ 27], 80.00th=[ 29], 90.00th=[ 34], 95.00th=[ 38], | 99.00th=[ 50], 99.50th=[ 64], 99.90th=[ 112], 99.95th=[ 112], | 99.99th=[ 112] bw ( KiB/s): min= 352, max= 736, per=25.51%, avg=565.64, stdev=111.33, samples=22 iops : min= 22, max= 46, avg=35.27, stdev= 6.91, samples=22 lat (msec) : 4=32.55%, 10=10.29%, 20=22.40%, 50=34.24%, 100=0.39% lat (msec) : 250=0.13% cpu : usr=0.11%, sys=6.02%, ctx=868, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31618: Wed Apr 3 07:48:29 2024 read: IOPS=35, BW=566KiB/s (579kB/s)(6256KiB/11056msec) clat (usec): min=2016, max=31122, avg=4670.44, stdev=4110.86 lat (usec): min=2016, max=31124, avg=4671.38, stdev=4110.84 clat percentiles (usec): | 1.00th=[ 2114], 5.00th=[ 2245], 10.00th=[ 2343], 20.00th=[ 2507], | 30.00th=[ 2704], 40.00th=[ 2900], 50.00th=[ 3195], 60.00th=[ 3458], | 70.00th=[ 3916], 80.00th=[ 5473], 90.00th=[ 9110], 95.00th=[13435], | 99.00th=[24773], 99.50th=[27395], 99.90th=[31065], 99.95th=[31065], | 99.99th=[31065] bw ( KiB/s): min= 256, max= 1152, per=26.98%, avg=568.55, stdev=226.40, samples=22 iops : min= 16, max= 72, avg=35.41, stdev=14.17, samples=22 write: IOPS=34, BW=546KiB/s (559kB/s)(6032KiB/11056msec) clat (msec): min=12, max=103, avg=24.46, stdev= 8.86 lat (msec): min=12, max=103, avg=24.46, stdev= 8.86 clat percentiles (msec): | 1.00th=[ 14], 5.00th=[ 15], 10.00th=[ 16], 20.00th=[ 18], | 30.00th=[ 20], 40.00th=[ 22], 50.00th=[ 24], 60.00th=[ 25], | 70.00th=[ 27], 80.00th=[ 30], 90.00th=[ 35], 95.00th=[ 40], | 99.00th=[ 55], 99.50th=[ 59], 99.90th=[ 104], 99.95th=[ 104], | 99.99th=[ 104] bw ( KiB/s): min= 352, max= 704, per=24.53%, avg=543.77, stdev=119.66, samples=22 iops : min= 22, max= 44, avg=33.86, stdev= 7.47, samples=22 lat (msec) : 4=36.46%, 10=9.64%, 20=19.79%, 50=33.46%, 100=0.52% lat (msec) : 250=0.13% cpu : usr=0.11%, sys=6.31%, ctx=891, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31619: Wed Apr 3 07:48:29 2024 read: IOPS=33, BW=543KiB/s (556kB/s)(6032KiB/11113msec) clat (usec): min=2090, max=25787, avg=4273.27, stdev=3293.19 lat (usec): min=2091, max=25787, avg=4274.19, stdev=3293.16 clat percentiles (usec): | 1.00th=[ 2180], 5.00th=[ 2311], 10.00th=[ 2409], 20.00th=[ 2573], | 30.00th=[ 2737], 40.00th=[ 2933], 50.00th=[ 3228], 60.00th=[ 3523], | 70.00th=[ 3851], 80.00th=[ 4686], 90.00th=[ 6915], 95.00th=[11076], | 99.00th=[20055], 99.50th=[25035], 99.90th=[25822], 99.95th=[25822], | 99.99th=[25822] bw ( KiB/s): min= 256, max= 864, per=25.61%, avg=539.64, stdev=169.70, samples=22 iops : min= 16, max= 54, avg=33.73, stdev=10.61, samples=22 write: IOPS=35, BW=563KiB/s (576kB/s)(6256KiB/11113msec) clat (msec): min=12, max=136, avg=24.28, stdev= 9.29 lat (msec): min=12, max=136, avg=24.28, stdev= 9.29 clat percentiles (msec): | 1.00th=[ 14], 5.00th=[ 15], 10.00th=[ 16], 20.00th=[ 18], | 30.00th=[ 20], 40.00th=[ 22], 50.00th=[ 23], 60.00th=[ 25], | 70.00th=[ 27], 80.00th=[ 30], 90.00th=[ 35], 95.00th=[ 39], | 99.00th=[ 48], 99.50th=[ 51], 99.90th=[ 138], 99.95th=[ 138], | 99.99th=[ 138] bw ( KiB/s): min= 352, max= 704, per=25.39%, avg=562.91, stdev=119.59, samples=22 iops : min= 22, max= 44, avg=35.18, stdev= 7.47, samples=22 lat (msec) : 4=35.55%, 10=10.81%, 20=20.05%, 50=33.33%, 100=0.13% lat (msec) : 250=0.13% cpu : usr=0.07%, sys=6.16%, ctx=866, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2108KiB/s (2158kB/s), 506KiB/s-566KiB/s (518kB/s-579kB/s), io=23.4MiB (24.5MB), run=11056-11363msec WRITE: bw=2218KiB/s (2271kB/s), 546KiB/s-576KiB/s (559kB/s-590kB/s), io=24.6MiB (25.8MB), run=11056-11363msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=31640: Wed Apr 3 07:48:34 2024 read: IOPS=2, BW=3013KiB/s (3085kB/s)(7168KiB/2379msec) clat (msec): min=29, max=823, avg=261.70, stdev=313.96 lat (msec): min=29, max=823, avg=261.70, stdev=313.96 clat percentiles (msec): | 1.00th=[ 30], 5.00th=[ 30], 10.00th=[ 30], 20.00th=[ 30], | 30.00th=[ 36], 40.00th=[ 36], 50.00th=[ 101], 60.00th=[ 251], | 70.00th=[ 251], 80.00th=[ 567], 90.00th=[ 827], 95.00th=[ 827], | 99.00th=[ 827], 99.50th=[ 827], 99.90th=[ 827], 99.95th=[ 827], | 99.99th=[ 827] bw ( KiB/s): min= 2048, max= 4096, per=65.23%, avg=3413.33, stdev=1182.41, samples=3 iops : min= 2, max= 4, avg= 3.33, stdev= 1.15, samples=3 write: IOPS=2, BW=2152KiB/s (2204kB/s)(5120KiB/2379msec) clat (msec): min=53, max=260, avg=107.11, stdev=86.92 lat (msec): min=54, max=260, avg=107.19, stdev=86.91 clat percentiles (msec): | 1.00th=[ 54], 5.00th=[ 54], 10.00th=[ 54], 20.00th=[ 54], | 30.00th=[ 58], 40.00th=[ 58], 50.00th=[ 75], 60.00th=[ 75], | 70.00th=[ 89], 80.00th=[ 89], 90.00th=[ 262], 95.00th=[ 262], | 99.00th=[ 262], 99.50th=[ 262], 99.90th=[ 262], 99.95th=[ 262], | 99.99th=[ 262] bw ( KiB/s): min= 2048, max= 6144, per=55.90%, avg=4096.00, stdev=2896.31, samples=2 iops : min= 2, max= 6, avg= 4.00, stdev= 2.83, samples=2 lat (msec) : 50=25.00%, 100=41.67%, 500=16.67%, 750=8.33%, 1000=8.33% cpu : usr=0.00%, sys=5.42%, ctx=222, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31641: Wed Apr 3 07:48:34 2024 read: IOPS=0, BW=589KiB/s (603kB/s)(2048KiB/3479msec) clat (msec): min=21, max=1209, avg=615.45, stdev=839.85 lat (msec): min=21, max=1209, avg=615.45, stdev=839.84 clat percentiles (msec): | 1.00th=[ 22], 5.00th=[ 22], 10.00th=[ 22], 20.00th=[ 22], | 30.00th=[ 22], 40.00th=[ 22], 50.00th=[ 22], 60.00th=[ 1217], | 70.00th=[ 1217], 80.00th=[ 1217], 90.00th=[ 1217], 95.00th=[ 1217], | 99.00th=[ 1217], 99.50th=[ 1217], 99.90th=[ 1217], 99.95th=[ 1217], | 99.99th=[ 1217] bw ( KiB/s): min= 1738, max= 2048, per=36.17%, avg=1893.00, stdev=219.20, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 write: IOPS=2, BW=2943KiB/s (3014kB/s)(10.0MiB/3479msec) clat (msec): min=40, max=1012, avg=223.66, stdev=334.53 lat (msec): min=40, max=1012, avg=223.73, stdev=334.51 clat percentiles (msec): | 1.00th=[ 41], 5.00th=[ 41], 10.00th=[ 41], 20.00th=[ 44], | 30.00th=[ 44], 40.00th=[ 47], 50.00th=[ 48], 60.00th=[ 61], | 70.00th=[ 182], 80.00th=[ 182], 90.00th=[ 651], 95.00th=[ 1011], | 99.00th=[ 1011], 99.50th=[ 1011], 99.90th=[ 1011], 99.95th=[ 1011], | 99.99th=[ 1011] bw ( KiB/s): min= 6954, max=10240, per=100.00%, avg=8597.00, stdev=2323.55, samples=2 iops : min= 6, max= 10, avg= 8.00, stdev= 2.83, samples=2 lat (msec) : 50=50.00%, 100=8.33%, 250=16.67%, 750=8.33% cpu : usr=0.03%, sys=3.82%, ctx=184, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31642: Wed Apr 3 07:48:34 2024 read: IOPS=1, BW=1570KiB/s (1608kB/s)(6144KiB/3913msec) clat (msec): min=24, max=500, avg=265.73, stdev=212.02 lat (msec): min=24, max=500, avg=265.74, stdev=212.02 clat percentiles (msec): | 1.00th=[ 25], 5.00th=[ 25], 10.00th=[ 25], 20.00th=[ 75], | 30.00th=[ 75], 40.00th=[ 130], 50.00th=[ 130], 60.00th=[ 414], | 70.00th=[ 451], 80.00th=[ 451], 90.00th=[ 502], 95.00th=[ 502], | 99.00th=[ 502], 99.50th=[ 502], 99.90th=[ 502], 99.95th=[ 502], | 99.99th=[ 502] bw ( KiB/s): min= 1759, max= 5278, per=56.05%, avg=2933.00, stdev=2030.83, samples=3 iops : min= 1, max= 5, avg= 2.33, stdev= 2.31, samples=3 write: IOPS=1, BW=1570KiB/s (1608kB/s)(6144KiB/3913msec) clat (msec): min=44, max=1862, avg=384.26, stdev=726.09 lat (msec): min=44, max=1862, avg=384.33, stdev=726.09 clat percentiles (msec): | 1.00th=[ 45], 5.00th=[ 45], 10.00th=[ 45], 20.00th=[ 45], | 30.00th=[ 45], 40.00th=[ 80], 50.00th=[ 80], 60.00th=[ 88], | 70.00th=[ 186], 80.00th=[ 186], 90.00th=[ 1871], 95.00th=[ 1871], | 99.00th=[ 1871], 99.50th=[ 1871], 99.90th=[ 1871], 99.95th=[ 1871], | 99.99th=[ 1871] bw ( KiB/s): min= 1759, max= 4096, per=37.99%, avg=2783.75, stdev=1204.88, samples=4 iops : min= 1, max= 4, avg= 2.25, stdev= 1.50, samples=4 lat (msec) : 50=25.00%, 100=25.00%, 250=16.67%, 500=16.67%, 750=8.33% cpu : usr=0.03%, sys=4.06%, ctx=255, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31643: Wed Apr 3 07:48:34 2024 read: IOPS=2, BW=2101KiB/s (2151kB/s)(5120KiB/2437msec) clat (msec): min=22, max=133, avg=78.67, stdev=52.64 lat (msec): min=22, max=133, avg=78.67, stdev=52.64 clat percentiles (msec): | 1.00th=[ 23], 5.00th=[ 23], 10.00th=[ 23], 20.00th=[ 23], | 30.00th=[ 33], 40.00th=[ 33], 50.00th=[ 74], 60.00th=[ 74], | 70.00th=[ 132], 80.00th=[ 132], 90.00th=[ 134], 95.00th=[ 134], | 99.00th=[ 134], 99.50th=[ 134], 99.90th=[ 134], 99.95th=[ 134], | 99.99th=[ 134] bw ( KiB/s): min= 2043, max= 2043, per=39.04%, avg=2043.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=2, BW=2941KiB/s (3012kB/s)(7168KiB/2437msec) clat (msec): min=43, max=1627, avg=290.12, stdev=589.72 lat (msec): min=43, max=1627, avg=290.18, stdev=589.73 clat percentiles (msec): | 1.00th=[ 44], 5.00th=[ 44], 10.00th=[ 44], 20.00th=[ 63], | 30.00th=[ 66], 40.00th=[ 66], 50.00th=[ 69], 60.00th=[ 82], | 70.00th=[ 82], 80.00th=[ 83], 90.00th=[ 1620], 95.00th=[ 1620], | 99.00th=[ 1620], 99.50th=[ 1620], 99.90th=[ 1620], 99.95th=[ 1620], | 99.99th=[ 1620] bw ( KiB/s): min= 2048, max= 6131, per=55.81%, avg=4089.50, stdev=2887.12, samples=2 iops : min= 2, max= 5, avg= 3.50, stdev= 2.12, samples=2 lat (msec) : 50=25.00%, 100=50.00%, 250=16.67% cpu : usr=0.00%, sys=5.30%, ctx=45, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=5234KiB/s (5359kB/s), 589KiB/s-3013KiB/s (603kB/s-3085kB/s), io=20.0MiB (20.0MB), run=2379-3913msec WRITE: bw=7327KiB/s (7503kB/s), 1570KiB/s-2943KiB/s (1608kB/s-3014kB/s), io=28.0MiB (29.4MB), run=2379-3913msec rand-rw: (groupid=0, jobs=1): err= 0: pid=31644: Wed Apr 3 07:48:35 2024 read: IOPS=2, BW=2453KiB/s (2512kB/s)(7168KiB/2922msec) clat (msec): min=43, max=868, avg=368.09, stdev=310.13 lat (msec): min=43, max=868, avg=368.10, stdev=310.13 clat percentiles (msec): | 1.00th=[ 44], 5.00th=[ 44], 10.00th=[ 44], 20.00th=[ 75], | 30.00th=[ 171], 40.00th=[ 171], 50.00th=[ 268], 60.00th=[ 558], | 70.00th=[ 558], 80.00th=[ 592], 90.00th=[ 869], 95.00th=[ 869], | 99.00th=[ 869], 99.50th=[ 869], 99.90th=[ 869], 99.95th=[ 869], | 99.99th=[ 869] bw ( KiB/s): min= 2048, max= 4096, per=52.47%, avg=2457.60, stdev=915.89, samples=5 iops : min= 2, max= 4, avg= 2.40, stdev= 0.89, samples=5 write: IOPS=1, BW=1752KiB/s (1794kB/s)(5120KiB/2922msec) clat (msec): min=15, max=144, avg=68.55, stdev=54.25 lat (msec): min=15, max=144, avg=68.62, stdev=54.26 clat percentiles (msec): | 1.00th=[ 16], 5.00th=[ 16], 10.00th=[ 16], 20.00th=[ 16], | 30.00th=[ 22], 40.00th=[ 22], 50.00th=[ 62], 60.00th=[ 62], | 70.00th=[ 100], 80.00th=[ 100], 90.00th=[ 146], 95.00th=[ 146], | 99.00th=[ 146], 99.50th=[ 146], 99.90th=[ 146], 99.95th=[ 146], | 99.99th=[ 146] bw ( KiB/s): min= 2048, max= 4096, per=52.05%, avg=3413.33, stdev=1182.41, samples=3 iops : min= 2, max= 4, avg= 3.33, stdev= 1.15, samples=3 lat (msec) : 20=8.33%, 50=16.67%, 100=25.00%, 250=16.67%, 500=8.33% lat (msec) : 750=16.67%, 1000=8.33% cpu : usr=0.00%, sys=39.44%, ctx=927, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31645: Wed Apr 3 07:48:35 2024 read: IOPS=0, BW=694KiB/s (710kB/s)(2048KiB/2953msec) clat (msec): min=37, max=1021, avg=529.43, stdev=695.49 lat (msec): min=37, max=1021, avg=529.43, stdev=695.49 clat percentiles (msec): | 1.00th=[ 38], 5.00th=[ 38], 10.00th=[ 38], 20.00th=[ 38], | 30.00th=[ 38], 40.00th=[ 38], 50.00th=[ 38], 60.00th=[ 1020], | 70.00th=[ 1020], 80.00th=[ 1020], 90.00th=[ 1020], 95.00th=[ 1020], | 99.00th=[ 1020], 99.50th=[ 1020], 99.90th=[ 1020], 99.95th=[ 1020], | 99.99th=[ 1020] bw ( KiB/s): min= 2048, max= 2048, per=43.72%, avg=2048.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 write: IOPS=3, BW=3468KiB/s (3551kB/s)(10.0MiB/2953msec) clat (msec): min=22, max=648, avg=188.45, stdev=242.06 lat (msec): min=22, max=649, avg=188.51, stdev=242.07 clat percentiles (msec): | 1.00th=[ 23], 5.00th=[ 23], 10.00th=[ 23], 20.00th=[ 39], | 30.00th=[ 39], 40.00th=[ 47], 50.00th=[ 51], 60.00th=[ 61], | 70.00th=[ 264], 80.00th=[ 264], 90.00th=[ 609], 95.00th=[ 651], | 99.00th=[ 651], 99.50th=[ 651], 99.90th=[ 651], 99.95th=[ 651], | 99.99th=[ 651] bw ( KiB/s): min= 2048, max=10240, per=93.69%, avg=6144.00, stdev=4096.00, samples=3 iops : min= 2, max= 10, avg= 6.00, stdev= 4.00, samples=3 lat (msec) : 50=41.67%, 100=16.67%, 250=8.33%, 500=8.33%, 750=16.67% cpu : usr=0.03%, sys=12.77%, ctx=158, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31646: Wed Apr 3 07:48:35 2024 read: IOPS=1, BW=1581KiB/s (1619kB/s)(6144KiB/3887msec) clat (msec): min=48, max=590, avg=298.82, stdev=244.51 lat (msec): min=48, max=590, avg=298.83, stdev=244.51 clat percentiles (msec): | 1.00th=[ 48], 5.00th=[ 48], 10.00th=[ 48], 20.00th=[ 58], | 30.00th=[ 58], 40.00th=[ 136], 50.00th=[ 136], 60.00th=[ 468], | 70.00th=[ 493], 80.00th=[ 493], 90.00th=[ 592], 95.00th=[ 592], | 99.00th=[ 592], 99.50th=[ 592], 99.90th=[ 592], 99.95th=[ 592], | 99.99th=[ 592] bw ( KiB/s): min= 2048, max= 6144, per=72.87%, avg=3413.33, stdev=2364.83, samples=3 iops : min= 2, max= 6, avg= 3.33, stdev= 2.31, samples=3 write: IOPS=1, BW=1581KiB/s (1619kB/s)(6144KiB/3887msec) clat (msec): min=23, max=1887, avg=347.35, stdev=754.70 lat (msec): min=23, max=1887, avg=347.43, stdev=754.69 clat percentiles (msec): | 1.00th=[ 24], 5.00th=[ 24], 10.00th=[ 24], 20.00th=[ 26], | 30.00th=[ 26], 40.00th=[ 34], 50.00th=[ 34], 60.00th=[ 42], | 70.00th=[ 73], 80.00th=[ 73], 90.00th=[ 1888], 95.00th=[ 1888], | 99.00th=[ 1888], 99.50th=[ 1888], 99.90th=[ 1888], 99.95th=[ 1888], | 99.99th=[ 1888] bw ( KiB/s): min= 2048, max= 4096, per=37.47%, avg=2457.60, stdev=915.89, samples=5 iops : min= 2, max= 4, avg= 2.40, stdev= 0.89, samples=5 lat (msec) : 50=41.67%, 100=16.67%, 250=8.33%, 500=16.67%, 750=8.33% cpu : usr=0.08%, sys=12.97%, ctx=848, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31647: Wed Apr 3 07:48:35 2024 read: IOPS=1, BW=1171KiB/s (1199kB/s)(5120KiB/4372msec) clat (msec): min=3, max=546, avg=312.11, stdev=256.54 lat (msec): min=3, max=546, avg=312.11, stdev=256.54 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 4], 10.00th=[ 4], 20.00th=[ 4], | 30.00th=[ 66], 40.00th=[ 66], 50.00th=[ 451], 60.00th=[ 451], | 70.00th=[ 493], 80.00th=[ 493], 90.00th=[ 550], 95.00th=[ 550], | 99.00th=[ 550], 99.50th=[ 550], 99.90th=[ 550], 99.95th=[ 550], | 99.99th=[ 550] bw ( KiB/s): min= 2043, max= 4096, per=58.26%, avg=2729.00, stdev=1183.86, samples=3 iops : min= 1, max= 4, avg= 2.33, stdev= 1.53, samples=3 write: IOPS=1, BW=1640KiB/s (1679kB/s)(7168KiB/4372msec) clat (msec): min=20, max=1929, avg=401.41, stdev=692.58 lat (msec): min=20, max=1929, avg=401.47, stdev=692.59 clat percentiles (msec): | 1.00th=[ 21], 5.00th=[ 21], 10.00th=[ 21], 20.00th=[ 28], | 30.00th=[ 42], 40.00th=[ 42], 50.00th=[ 90], 60.00th=[ 234], | 70.00th=[ 234], 80.00th=[ 468], 90.00th=[ 1921], 95.00th=[ 1921], | 99.00th=[ 1921], 99.50th=[ 1921], 99.90th=[ 1921], 99.95th=[ 1921], | 99.99th=[ 1921] bw ( KiB/s): min= 2043, max= 6131, per=54.58%, avg=3579.50, stdev=1956.46, samples=4 iops : min= 1, max= 5, avg= 3.00, stdev= 1.83, samples=4 lat (msec) : 4=8.33%, 50=25.00%, 100=16.67%, 250=8.33%, 500=25.00% lat (msec) : 750=8.33% cpu : usr=0.02%, sys=8.60%, ctx=834, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=4684KiB/s (4797kB/s), 694KiB/s-2453KiB/s (710kB/s-2512kB/s), io=20.0MiB (20.0MB), run=2922-4372msec WRITE: bw=6558KiB/s (6715kB/s), 1581KiB/s-3468KiB/s (1619kB/s-3551kB/s), io=28.0MiB (29.4MB), run=2922-4372msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=31657: Wed Apr 3 07:48:38 2024 read: IOPS=2, BW=9316KiB/s (9540kB/s)(12.0MiB/1319msec) clat (msec): min=188, max=862, avg=429.47, stdev=375.94 lat (msec): min=188, max=862, avg=429.48, stdev=375.94 clat percentiles (msec): | 1.00th=[ 188], 5.00th=[ 188], 10.00th=[ 188], 20.00th=[ 188], | 30.00th=[ 188], 40.00th=[ 239], 50.00th=[ 239], 60.00th=[ 239], | 70.00th=[ 860], 80.00th=[ 860], 90.00th=[ 860], 95.00th=[ 860], | 99.00th=[ 860], 99.50th=[ 860], 99.90th=[ 860], 99.95th=[ 860], | 99.99th=[ 860] bw ( KiB/s): min= 8192, max= 8192, per=79.40%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 1000=33.33% cpu : usr=0.00%, sys=75.11%, ctx=218, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31658: Wed Apr 3 07:48:38 2024 read: IOPS=0, BW=2579KiB/s (2641kB/s)(4096KiB/1588msec) clat (nsec): min=236307k, max=236307k, avg=236307262.00, stdev= 0.00 lat (nsec): min=236308k, max=236308k, avg=236307932.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 236], 5.00th=[ 236], 10.00th=[ 236], 20.00th=[ 236], | 30.00th=[ 236], 40.00th=[ 236], 50.00th=[ 236], 60.00th=[ 236], | 70.00th=[ 236], 80.00th=[ 236], 90.00th=[ 236], 95.00th=[ 236], | 99.00th=[ 236], 99.50th=[ 236], 99.90th=[ 236], 99.95th=[ 236], | 99.99th=[ 236] bw ( KiB/s): min= 8192, max= 8192, per=79.40%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=1, BW=5159KiB/s (5282kB/s)(8192KiB/1588msec) clat (msec): min=134, max=1214, avg=674.51, stdev=763.09 lat (msec): min=135, max=1214, avg=674.77, stdev=763.13 clat percentiles (msec): | 1.00th=[ 136], 5.00th=[ 136], 10.00th=[ 136], 20.00th=[ 136], | 30.00th=[ 136], 40.00th=[ 136], 50.00th=[ 136], 60.00th=[ 1217], | 70.00th=[ 1217], 80.00th=[ 1217], 90.00th=[ 1217], 95.00th=[ 1217], | 99.00th=[ 1217], 99.50th=[ 1217], 99.90th=[ 1217], 99.95th=[ 1217], | 99.99th=[ 1217] bw ( KiB/s): min= 8192, max= 8192, per=44.73%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67% cpu : usr=0.00%, sys=16.13%, ctx=70, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31659: Wed Apr 3 07:48:38 2024 write: IOPS=1, BW=6869KiB/s (7033kB/s)(12.0MiB/1789msec) clat (msec): min=106, max=1237, avg=585.70, stdev=585.19 lat (msec): min=106, max=1237, avg=586.03, stdev=585.03 clat percentiles (msec): | 1.00th=[ 107], 5.00th=[ 107], 10.00th=[ 107], 20.00th=[ 107], | 30.00th=[ 107], 40.00th=[ 414], 50.00th=[ 414], 60.00th=[ 414], | 70.00th=[ 1234], 80.00th=[ 1234], 90.00th=[ 1234], 95.00th=[ 1234], | 99.00th=[ 1234], 99.50th=[ 1234], 99.90th=[ 1234], 99.95th=[ 1234], | 99.99th=[ 1234] bw ( KiB/s): min= 8192, max= 8192, per=44.73%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=33.33% cpu : usr=0.00%, sys=14.09%, ctx=45, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31660: Wed Apr 3 07:48:38 2024 write: IOPS=4, BW=19.1MiB/s (20.0MB/s)(12.0MiB/628msec) clat (msec): min=99, max=411, avg=208.74, stdev=175.40 lat (msec): min=99, max=411, avg=208.94, stdev=175.46 clat percentiles (msec): | 1.00th=[ 100], 5.00th=[ 100], 10.00th=[ 100], 20.00th=[ 100], | 30.00th=[ 100], 40.00th=[ 116], 50.00th=[ 116], 60.00th=[ 116], | 70.00th=[ 414], 80.00th=[ 414], 90.00th=[ 414], 95.00th=[ 414], | 99.00th=[ 414], 99.50th=[ 414], 99.90th=[ 414], 99.95th=[ 414], | 99.99th=[ 414] bw ( KiB/s): min=16351, max=16351, per=89.27%, avg=16351.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 250=33.33%, 500=33.33% cpu : usr=0.00%, sys=43.22%, ctx=78, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=10.1MiB/s (10.6MB/s), 2579KiB/s-9316KiB/s (2641kB/s-9540kB/s), io=16.0MiB (16.8MB), run=1319-1588msec WRITE: bw=17.9MiB/s (18.8MB/s), 5159KiB/s-19.1MiB/s (5282kB/s-20.0MB/s), io=32.0MiB (33.6MB), run=628-1789msec rand-rw: (groupid=0, jobs=1): err= 0: pid=31661: Wed Apr 3 07:48:38 2024 read: IOPS=1, BW=6819KiB/s (6983kB/s)(12.0MiB/1802msec) clat (msec): min=105, max=1240, avg=600.12, stdev=581.35 lat (msec): min=105, max=1240, avg=600.12, stdev=581.35 clat percentiles (msec): | 1.00th=[ 106], 5.00th=[ 106], 10.00th=[ 106], 20.00th=[ 106], | 30.00th=[ 106], 40.00th=[ 456], 50.00th=[ 456], 60.00th=[ 456], | 70.00th=[ 1234], 80.00th=[ 1234], 90.00th=[ 1234], 95.00th=[ 1234], | 99.00th=[ 1234], 99.50th=[ 1234], 99.90th=[ 1234], 99.95th=[ 1234], | 99.99th=[ 1234] bw ( KiB/s): min= 8192, max= 8192, per=90.10%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=33.33% cpu : usr=0.00%, sys=5.89%, ctx=31, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31662: Wed Apr 3 07:48:38 2024 read: IOPS=0, BW=2345KiB/s (2401kB/s)(4096KiB/1747msec) clat (nsec): min=485669k, max=485669k, avg=485668833.00, stdev= 0.00 lat (nsec): min=485671k, max=485671k, avg=485670583.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 485], 5.00th=[ 485], 10.00th=[ 485], 20.00th=[ 485], | 30.00th=[ 485], 40.00th=[ 485], 50.00th=[ 485], 60.00th=[ 485], | 70.00th=[ 485], 80.00th=[ 485], 90.00th=[ 485], 95.00th=[ 485], | 99.00th=[ 485], 99.50th=[ 485], 99.90th=[ 485], 99.95th=[ 485], | 99.99th=[ 485] bw ( KiB/s): min= 8192, max= 8192, per=90.10%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=1, BW=4689KiB/s (4802kB/s)(8192KiB/1747msec) clat (msec): min=103, max=1149, avg=626.22, stdev=739.65 lat (msec): min=103, max=1149, avg=626.52, stdev=739.66 clat percentiles (msec): | 1.00th=[ 104], 5.00th=[ 104], 10.00th=[ 104], 20.00th=[ 104], | 30.00th=[ 104], 40.00th=[ 104], 50.00th=[ 104], 60.00th=[ 1150], | 70.00th=[ 1150], 80.00th=[ 1150], 90.00th=[ 1150], 95.00th=[ 1150], | 99.00th=[ 1150], 99.50th=[ 1150], 99.90th=[ 1150], 99.95th=[ 1150], | 99.99th=[ 1150] bw ( KiB/s): min= 8192, max= 8192, per=43.68%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=33.33% cpu : usr=0.06%, sys=6.93%, ctx=25, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31663: Wed Apr 3 07:48:38 2024 write: IOPS=1, BW=7637KiB/s (7820kB/s)(12.0MiB/1609msec) clat (msec): min=117, max=1184, avg=533.81, stdev=571.11 lat (msec): min=117, max=1185, avg=534.12, stdev=571.25 clat percentiles (msec): | 1.00th=[ 117], 5.00th=[ 117], 10.00th=[ 117], 20.00th=[ 117], | 30.00th=[ 117], 40.00th=[ 300], 50.00th=[ 300], 60.00th=[ 300], | 70.00th=[ 1183], 80.00th=[ 1183], 90.00th=[ 1183], 95.00th=[ 1183], | 99.00th=[ 1183], 99.50th=[ 1183], 99.90th=[ 1183], 99.95th=[ 1183], | 99.99th=[ 1183] bw ( KiB/s): min= 8175, max= 8192, per=43.63%, avg=8183.50, stdev=12.02, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 250=33.33%, 500=33.33% cpu : usr=0.06%, sys=8.20%, ctx=54, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31664: Wed Apr 3 07:48:38 2024 write: IOPS=1, BW=8187KiB/s (8383kB/s)(12.0MiB/1501msec) clat (msec): min=128, max=1121, avg=499.73, stdev=542.13 lat (msec): min=128, max=1121, avg=499.93, stdev=542.04 clat percentiles (msec): | 1.00th=[ 129], 5.00th=[ 129], 10.00th=[ 129], 20.00th=[ 129], | 30.00th=[ 129], 40.00th=[ 249], 50.00th=[ 249], 60.00th=[ 249], | 70.00th=[ 1116], 80.00th=[ 1116], 90.00th=[ 1116], 95.00th=[ 1116], | 99.00th=[ 1116], 99.50th=[ 1116], 99.90th=[ 1116], 99.95th=[ 1116], | 99.99th=[ 1116] bw ( KiB/s): min= 8192, max= 8192, per=43.68%, avg=8192.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 lat (msec) : 250=66.67% cpu : usr=0.00%, sys=11.53%, ctx=78, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=9092KiB/s (9310kB/s), 2345KiB/s-6819KiB/s (2401kB/s-6983kB/s), io=16.0MiB (16.8MB), run=1747-1802msec WRITE: bw=18.3MiB/s (19.2MB/s), 4689KiB/s-8187KiB/s (4802kB/s-8383kB/s), io=32.0MiB (33.6MB), run=1501-1747msec