== sanity test 398b: DIO and buffer IO race ============== 14:52:44 (1731354764) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.69558 s, 29.7 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=16066: Mon Nov 11 15:00:31 2024 read: IOPS=3, BW=12.9KiB/s (13.3kB/s)(5992KiB/462921msec) clat (usec): min=50, max=50017k, avg=71361.78, stdev=1825813.98 lat (usec): min=50, max=50017k, avg=71363.19, stdev=1825813.96 clat percentiles (usec): | 1.00th=[ 59], 5.00th=[ 1827], 10.00th=[ 2180], | 20.00th=[ 2573], 30.00th=[ 2966], 40.00th=[ 3458], | 50.00th=[ 4015], 60.00th=[ 4621], 70.00th=[ 5342], | 80.00th=[ 6063], 90.00th=[ 7570], 95.00th=[ 9110], | 99.00th=[ 14484], 99.50th=[ 20317], 99.90th=[17112761], | 99.95th=[17112761], 99.99th=[17112761] bw ( KiB/s): min= 8, max= 576, per=100.00%, avg=361.82, stdev=172.80, samples=33 iops : min= 2, max= 144, avg=90.36, stdev=43.21, samples=33 write: IOPS=3, BW=13.6KiB/s (13.9kB/s)(6296KiB/462921msec) clat (usec): min=419, max=50014k, avg=226124.44, stdev=3327845.25 lat (usec): min=420, max=50014k, avg=226125.96, stdev=3327845.19 clat percentiles (usec): | 1.00th=[ 498], 5.00th=[ 529], 10.00th=[ 562], | 20.00th=[ 660], 30.00th=[ 1139], 40.00th=[ 2278], | 50.00th=[ 2868], 60.00th=[ 3687], 70.00th=[ 4883], | 80.00th=[ 6325], 90.00th=[ 8455], 95.00th=[ 10159], | 99.00th=[ 19792], 99.50th=[ 29754], 99.90th=[17112761], | 99.95th=[17112761], 99.99th=[17112761] bw ( KiB/s): min= 7, max= 624, per=100.00%, avg=357.77, stdev=202.91, samples=35 iops : min= 1, max= 156, avg=89.34, stdev=50.77, samples=35 lat (usec) : 100=1.79%, 250=0.42%, 500=0.72%, 750=11.39%, 1000=2.47% lat (msec) : 2=5.47%, 4=34.41%, 10=38.77%, 20=3.84%, 50=0.39% lat (msec) : 250=0.03% cpu : usr=0.01%, sys=0.64%, ctx=6516, majf=0, minf=37 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=16067: Mon Nov 11 15:00:31 2024 read: IOPS=3, BW=12.8KiB/s (13.1kB/s)(5940KiB/463135msec) clat (usec): min=49, max=50027k, avg=71929.39, stdev=1834708.07 lat (usec): min=49, max=50027k, avg=71930.54, stdev=1834708.05 clat percentiles (usec): | 1.00th=[ 59], 5.00th=[ 1860], 10.00th=[ 2180], | 20.00th=[ 2638], 30.00th=[ 3064], 40.00th=[ 3490], | 50.00th=[ 4080], 60.00th=[ 4686], 70.00th=[ 5342], | 80.00th=[ 6194], 90.00th=[ 7504], 95.00th=[ 9241], | 99.00th=[ 14746], 99.50th=[ 15795], 99.90th=[17112761], | 99.95th=[17112761], 99.99th=[17112761] bw ( KiB/s): min= 8, max= 552, per=100.00%, avg=344.03, stdev=190.13, samples=34 iops : min= 2, max= 138, avg=85.88, stdev=47.54, samples=34 write: IOPS=3, BW=13.7KiB/s (14.0kB/s)(6348KiB/463135msec) clat (usec): min=419, max=50009k, avg=224478.19, stdev=3314609.40 lat (usec): min=419, max=50009k, avg=224478.95, stdev=3314609.39 clat percentiles (usec): | 1.00th=[ 498], 5.00th=[ 529], 10.00th=[ 562], | 20.00th=[ 668], 30.00th=[ 1123], 40.00th=[ 2311], | 50.00th=[ 2868], 60.00th=[ 3556], 70.00th=[ 4752], | 80.00th=[ 6325], 90.00th=[ 8717], 95.00th=[ 10945], | 99.00th=[ 19530], 99.50th=[ 185598], 99.90th=[17112761], | 99.95th=[17112761], 99.99th=[17112761] bw ( KiB/s): min= 8, max= 624, per=100.00%, avg=364.50, stdev=184.85, samples=34 iops : min= 2, max= 156, avg=91.00, stdev=46.21, samples=34 lat (usec) : 50=0.07%, 100=1.66%, 250=0.20%, 500=0.62%, 750=12.11% lat (usec) : 1000=2.47% lat (msec) : 2=5.08%, 4=34.02%, 10=38.54%, 20=4.65%, 50=0.26% lat (msec) : 250=0.03% cpu : usr=0.01%, sys=0.64%, ctx=6339, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=16068: Mon Nov 11 15:00:31 2024 read: IOPS=3, BW=13.0KiB/s (13.3kB/s)(6028KiB/463048msec) clat (usec): min=48, max=50029k, avg=70959.14, stdev=1821394.94 lat (usec): min=48, max=50029k, avg=70960.57, stdev=1821394.91 clat percentiles (usec): | 1.00th=[ 65], 5.00th=[ 1926], 10.00th=[ 2245], | 20.00th=[ 2671], 30.00th=[ 3130], 40.00th=[ 3621], | 50.00th=[ 4178], 60.00th=[ 4752], 70.00th=[ 5342], | 80.00th=[ 6063], 90.00th=[ 7504], 95.00th=[ 9241], | 99.00th=[ 13566], 99.50th=[ 17171], 99.90th=[17112761], | 99.95th=[17112761], 99.99th=[17112761] bw ( KiB/s): min= 7, max= 616, per=100.00%, avg=342.77, stdev=196.82, samples=35 iops : min= 1, max= 154, avg=85.63, stdev=49.26, samples=35 write: IOPS=3, BW=13.5KiB/s (13.8kB/s)(6260KiB/463048msec) clat (usec): min=402, max=50011k, avg=227478.59, stdev=3337517.76 lat (usec): min=403, max=50011k, avg=227479.69, stdev=3337517.73 clat percentiles (usec): | 1.00th=[ 490], 5.00th=[ 529], 10.00th=[ 562], | 20.00th=[ 644], 30.00th=[ 1037], 40.00th=[ 2212], | 50.00th=[ 2802], 60.00th=[ 3556], 70.00th=[ 4621], | 80.00th=[ 6063], 90.00th=[ 8356], 95.00th=[ 10552], | 99.00th=[ 20579], 99.50th=[ 166724], 99.90th=[17112761], | 99.95th=[17112761], 99.99th=[17112761] bw ( KiB/s): min= 8, max= 608, per=100.00%, avg=365.82, stdev=185.60, samples=34 iops : min= 2, max= 152, avg=91.41, stdev=46.43, samples=34 lat (usec) : 50=0.03%, 100=1.43%, 250=0.29%, 500=0.81%, 750=11.62% lat (usec) : 1000=2.44% lat (msec) : 2=5.01%, 4=34.28%, 10=39.10%, 20=4.33%, 50=0.33% lat (msec) : 250=0.03% cpu : usr=0.01%, sys=0.64%, ctx=6465, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=16069: Mon Nov 11 15:00:31 2024 read: IOPS=3, BW=13.1KiB/s (13.4kB/s)(6052KiB/463117msec) clat (usec): min=49, max=25305, avg=4528.80, stdev=2615.17 lat (usec): min=50, max=25306, avg=4529.46, stdev=2615.20 clat percentiles (usec): | 1.00th=[ 58], 5.00th=[ 1926], 10.00th=[ 2245], 20.00th=[ 2606], | 30.00th=[ 3064], 40.00th=[ 3556], 50.00th=[ 4080], 60.00th=[ 4621], | 70.00th=[ 5276], 80.00th=[ 6128], 90.00th=[ 7373], 95.00th=[ 8979], | 99.00th=[12780], 99.50th=[18220], 99.90th=[22414], 99.95th=[25297], | 99.99th=[25297] bw ( KiB/s): min= 8, max= 600, per=100.00%, avg=361.55, stdev=176.69, samples=33 iops : min= 2, max= 150, avg=90.30, stdev=44.13, samples=33 write: IOPS=3, BW=13.5KiB/s (13.8kB/s)(6236KiB/463117msec) clat (usec): min=418, max=50014k, avg=292617.20, stdev=3789188.91 lat (usec): min=418, max=50014k, avg=292617.97, stdev=3789188.90 clat percentiles (usec): | 1.00th=[ 494], 5.00th=[ 537], 10.00th=[ 570], | 20.00th=[ 685], 30.00th=[ 1074], 40.00th=[ 2212], | 50.00th=[ 2835], 60.00th=[ 3687], 70.00th=[ 4883], | 80.00th=[ 6390], 90.00th=[ 8586], 95.00th=[ 11207], | 99.00th=[ 22152], 99.50th=[17112761], 99.90th=[17112761], | 99.95th=[17112761], 99.99th=[17112761] bw ( KiB/s): min= 8, max= 720, per=100.00%, avg=351.83, stdev=197.40, samples=35 iops : min= 2, max= 180, avg=87.89, stdev=49.32, samples=35 lat (usec) : 50=0.07%, 100=1.63%, 250=0.26%, 500=0.68%, 750=10.90% lat (usec) : 1000=2.96% lat (msec) : 2=5.08%, 4=33.82%, 10=39.55%, 20=4.33%, 50=0.36% lat (msec) : 100=0.03%, 250=0.03% cpu : usr=0.01%, sys=0.64%, ctx=6478, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=51.8KiB/s (53.1kB/s), 12.8KiB/s-13.1KiB/s (13.1kB/s-13.4kB/s), io=23.4MiB (24.6MB), run=462921-463135msec WRITE: bw=54.3KiB/s (55.6kB/s), 13.5KiB/s-13.7KiB/s (13.8kB/s-14.0kB/s), io=24.6MiB (25.7MB), run=462921-463135msec rand-rw: (groupid=0, jobs=1): err= 0: pid=16062: Mon Nov 11 15:00:36 2024 read: IOPS=3, BW=12.8KiB/s (13.1kB/s)(5992KiB/468353msec) clat (usec): min=1661, max=50002k, avg=38332.15, stdev=1291793.07 lat (usec): min=1661, max=50002k, avg=38332.90, stdev=1291793.06 clat percentiles (usec): | 1.00th=[ 1795], 5.00th=[ 1958], 10.00th=[ 2114], | 20.00th=[ 2442], 30.00th=[ 2868], 40.00th=[ 3359], | 50.00th=[ 3818], 60.00th=[ 4228], 70.00th=[ 5145], | 80.00th=[ 6980], 90.00th=[ 9241], 95.00th=[ 11076], | 99.00th=[ 17695], 99.50th=[ 20579], 99.90th=[ 187696], | 99.95th=[17112761], 99.99th=[17112761] bw ( KiB/s): min= 7, max= 760, per=100.00%, avg=262.07, stdev=153.89, samples=45 iops : min= 1, max= 190, avg=65.40, stdev=38.51, samples=45 write: IOPS=3, BW=13.4KiB/s (13.8kB/s)(6296KiB/468353msec) clat (msec): min=2, max=50029, avg=261.02, stdev=3557.06 lat (msec): min=2, max=50029, avg=261.02, stdev=3557.06 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 4], 20.00th=[ 4], | 30.00th=[ 5], 40.00th=[ 6], 50.00th=[ 7], 60.00th=[ 7], | 70.00th=[ 8], 80.00th=[ 10], 90.00th=[ 12], 95.00th=[ 14], | 99.00th=[ 26], 99.50th=[17113], 99.90th=[17113], 99.95th=[17113], | 99.99th=[17113] bw ( KiB/s): min= 8, max= 704, per=100.00%, avg=274.16, stdev=146.01, samples=45 iops : min= 2, max= 176, avg=68.42, stdev=36.57, samples=45 lat (msec) : 2=3.09%, 4=35.25%, 10=49.67%, 20=10.97%, 50=0.68% lat (msec) : 250=0.03% cpu : usr=0.01%, sys=0.45%, ctx=3655, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=16063: Mon Nov 11 15:00:36 2024 read: IOPS=3, BW=12.7KiB/s (12.0kB/s)(5940KiB/468344msec) clat (usec): min=1606, max=50008k, avg=139565.62, stdev=2591529.63 lat (usec): min=1607, max=50008k, avg=139566.37, stdev=2591529.62 clat percentiles (usec): | 1.00th=[ 1762], 5.00th=[ 1942], 10.00th=[ 2114], | 20.00th=[ 2409], 30.00th=[ 2737], 40.00th=[ 3359], | 50.00th=[ 3818], 60.00th=[ 4146], 70.00th=[ 5080], | 80.00th=[ 6915], 90.00th=[ 9634], 95.00th=[ 11863], | 99.00th=[ 16188], 99.50th=[ 18220], 99.90th=[17112761], | 99.95th=[17112761], 99.99th=[17112761] bw ( KiB/s): min= 8, max= 672, per=100.00%, avg=260.20, stdev=147.29, samples=45 iops : min= 2, max= 168, avg=65.00, stdev=36.81, samples=45 write: IOPS=3, BW=13.6KiB/s (13.9kB/s)(6348KiB/468344msec) clat (msec): min=2, max=50027, avg=164.46, stdev=2803.54 lat (msec): min=2, max=50027, avg=164.46, stdev=2803.54 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 4], 20.00th=[ 4], | 30.00th=[ 5], 40.00th=[ 6], 50.00th=[ 7], 60.00th=[ 7], | 70.00th=[ 8], 80.00th=[ 10], 90.00th=[ 12], 95.00th=[ 14], | 99.00th=[ 23], 99.50th=[ 40], 99.90th=[17113], 99.95th=[17113], | 99.99th=[17113] bw ( KiB/s): min= 8, max= 776, per=100.00%, avg=276.02, stdev=153.99, samples=45 iops : min= 2, max= 194, avg=68.96, stdev=38.48, samples=45 lat (msec) : 2=3.03%, 4=35.38%, 10=48.70%, 20=12.01%, 50=0.55% lat (msec) : 250=0.03% cpu : usr=0.00%, sys=0.46%, ctx=3693, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=16064: Mon Nov 11 15:00:36 2024 read: IOPS=3, BW=12.9KiB/s (13.2kB/s)(6028KiB/468272msec) clat (usec): min=1531, max=50015k, avg=203939.50, stdev=3149562.77 lat (usec): min=1532, max=50015k, avg=203940.26, stdev=3149562.76 clat percentiles (usec): | 1.00th=[ 1745], 5.00th=[ 1942], 10.00th=[ 2073], | 20.00th=[ 2376], 30.00th=[ 2802], 40.00th=[ 3294], | 50.00th=[ 3752], 60.00th=[ 4178], 70.00th=[ 5145], | 80.00th=[ 6980], 90.00th=[ 9634], 95.00th=[ 12518], | 99.00th=[ 18482], 99.50th=[ 21627], 99.90th=[17112761], | 99.95th=[17112761], 99.99th=[17112761] bw ( KiB/s): min= 8, max= 840, per=100.00%, avg=270.68, stdev=156.82, samples=44 iops : min= 2, max= 210, avg=67.64, stdev=39.18, samples=44 write: IOPS=3, BW=13.4KiB/s (13.7kB/s)(6260KiB/468272msec) clat (msec): min=2, max=50017, avg=102.78, stdev=2188.01 lat (msec): min=2, max=50017, avg=102.78, stdev=2188.01 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 4], 10.00th=[ 4], 20.00th=[ 4], | 30.00th=[ 5], 40.00th=[ 6], 50.00th=[ 7], 60.00th=[ 7], | 70.00th=[ 8], 80.00th=[ 9], 90.00th=[ 12], 95.00th=[ 14], | 99.00th=[ 21], 99.50th=[ 31], 99.90th=[17113], 99.95th=[17113], | 99.99th=[17113] bw ( KiB/s): min= 16, max= 688, per=100.00%, avg=274.27, stdev=155.84, samples=45 iops : min= 4, max= 172, avg=68.53, stdev=38.95, samples=45 lat (msec) : 2=3.42%, 4=35.32%, 10=48.57%, 20=11.75%, 50=0.62% lat (msec) : 250=0.03% cpu : usr=0.01%, sys=0.47%, ctx=3668, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=16065: Mon Nov 11 15:00:36 2024 read: IOPS=3, BW=12.9KiB/s (13.2kB/s)(6052KiB/468281msec) clat (usec): min=1537, max=50012k, avg=136957.88, stdev=2568484.48 lat (usec): min=1537, max=50012k, avg=136958.71, stdev=2568484.47 clat percentiles (usec): | 1.00th=[ 1745], 5.00th=[ 1958], 10.00th=[ 2147], | 20.00th=[ 2442], 30.00th=[ 2802], 40.00th=[ 3425], | 50.00th=[ 3818], 60.00th=[ 4228], 70.00th=[ 5080], | 80.00th=[ 6915], 90.00th=[ 9110], 95.00th=[ 11076], | 99.00th=[ 15926], 99.50th=[ 18220], 99.90th=[17112761], | 99.95th=[17112761], 99.99th=[17112761] bw ( KiB/s): min= 8, max= 760, per=100.00%, avg=271.23, stdev=147.16, samples=44 iops : min= 2, max= 190, avg=67.77, stdev=36.78, samples=44 write: IOPS=3, BW=13.3KiB/s (13.6kB/s)(6236KiB/468281msec) clat (msec): min=2, max=50028, avg=167.40, stdev=2828.04 lat (msec): min=2, max=50028, avg=167.40, stdev=2828.04 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 4], 10.00th=[ 4], 20.00th=[ 4], | 30.00th=[ 5], 40.00th=[ 6], 50.00th=[ 7], 60.00th=[ 7], | 70.00th=[ 8], 80.00th=[ 10], 90.00th=[ 12], 95.00th=[ 14], | 99.00th=[ 21], 99.50th=[ 40], 99.90th=[17113], 99.95th=[17113], | 99.99th=[17113] bw ( KiB/s): min= 8, max= 720, per=100.00%, avg=273.73, stdev=147.77, samples=45 iops : min= 2, max= 180, avg=68.40, stdev=36.93, samples=45 lat (msec) : 2=3.03%, 4=35.25%, 10=50.20%, 20=10.68%, 50=0.52% lat (msec) : 250=0.03% cpu : usr=0.01%, sys=0.46%, ctx=3625, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=51.3KiB/s (52.5kB/s), 12.7KiB/s-12.9KiB/s (12.0kB/s-13.2kB/s), io=23.4MiB (24.6MB), run=468272-468353msec WRITE: bw=53.7KiB/s (54.0kB/s), 13.3KiB/s-13.6KiB/s (13.6kB/s-13.9kB/s), io=24.6MiB (25.7MB), run=468272-468353msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=16461: Mon Nov 11 15:02:23 2024 read: IOPS=3, BW=55.6KiB/s (56.9kB/s)(5920KiB/106505msec) clat (usec): min=58, max=86071, avg=11468.39, stdev=5814.50 lat (usec): min=59, max=86071, avg=11469.05, stdev=5814.57 clat percentiles (usec): | 1.00th=[ 72], 5.00th=[ 6783], 10.00th=[ 7570], 20.00th=[ 8586], | 30.00th=[ 9241], 40.00th=[ 9765], 50.00th=[10683], 60.00th=[11338], | 70.00th=[12387], 80.00th=[13829], 90.00th=[16450], 95.00th=[19268], | 99.00th=[26346], 99.50th=[35914], 99.90th=[86508], 99.95th=[86508], | 99.99th=[86508] bw ( KiB/s): min= 64, max= 1280, per=100.00%, avg=789.33, stdev=328.72, samples=15 iops : min= 4, max= 80, avg=49.33, stdev=20.54, samples=15 write: IOPS=3, BW=59.8KiB/s (61.2kB/s)(6368KiB/106505msec) clat (usec): min=668, max=50009k, avg=256767.07, stdev=3539366.87 lat (usec): min=668, max=50009k, avg=256768.13, stdev=3539366.86 clat percentiles (usec): | 1.00th=[ 758], 5.00th=[ 783], 10.00th=[ 857], | 20.00th=[ 1582], 30.00th=[ 2311], 40.00th=[ 2638], | 50.00th=[ 3359], 60.00th=[ 4555], 70.00th=[ 5866], | 80.00th=[ 8225], 90.00th=[ 13698], 95.00th=[ 17433], | 99.00th=[ 39584], 99.50th=[17112761], 99.90th=[17112761], | 99.95th=[17112761], 99.99th=[17112761] bw ( KiB/s): min= 192, max= 1280, per=100.00%, avg=846.93, stdev=351.35, samples=15 iops : min= 12, max= 80, avg=52.93, stdev=21.96, samples=15 lat (usec) : 100=0.78%, 250=0.65%, 750=0.39%, 1000=7.29% lat (msec) : 2=5.08%, 4=15.76%, 10=34.77%, 20=31.51%, 50=3.39% lat (msec) : 100=0.13% cpu : usr=0.01%, sys=1.22%, ctx=2958, majf=0, minf=37 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=16463: Mon Nov 11 15:02:23 2024 read: IOPS=3, BW=53.9KiB/s (55.2kB/s)(5744KiB/106534msec) clat (usec): min=58, max=51049, avg=11289.67, stdev=4935.15 lat (usec): min=59, max=51049, avg=11290.99, stdev=4935.64 clat percentiles (usec): | 1.00th=[ 61], 5.00th=[ 6587], 10.00th=[ 7373], 20.00th=[ 8356], | 30.00th=[ 8979], 40.00th=[ 9634], 50.00th=[10290], 60.00th=[11076], | 70.00th=[12125], 80.00th=[13829], 90.00th=[17171], 95.00th=[20317], | 99.00th=[27919], 99.50th=[32375], 99.90th=[51119], 99.95th=[51119], | 99.99th=[51119] bw ( KiB/s): min= 64, max= 1184, per=100.00%, avg=757.33, stdev=306.38, samples=15 iops : min= 4, max= 74, avg=47.33, stdev=19.15, samples=15 write: IOPS=3, BW=61.4KiB/s (62.9kB/s)(6544KiB/106534msec) clat (usec): min=734, max=50002k, avg=250529.84, stdev=3491444.36 lat (usec): min=735, max=50002k, avg=250531.19, stdev=3491444.34 clat percentiles (usec): | 1.00th=[ 758], 5.00th=[ 791], 10.00th=[ 906], | 20.00th=[ 1762], 30.00th=[ 2212], 40.00th=[ 2540], | 50.00th=[ 3490], 60.00th=[ 4621], 70.00th=[ 6390], | 80.00th=[ 10028], 90.00th=[ 14484], 95.00th=[ 17433], | 99.00th=[ 38011], 99.50th=[ 72877], 99.90th=[17112761], | 99.95th=[17112761], 99.99th=[17112761] bw ( KiB/s): min= 96, max= 1312, per=100.00%, avg=864.00, stdev=348.03, samples=15 iops : min= 6, max= 82, avg=54.00, stdev=21.75, samples=15 lat (usec) : 100=0.65%, 250=0.52%, 750=0.39%, 1000=6.25% lat (msec) : 2=5.86%, 4=16.15%, 10=34.77%, 20=30.99%, 50=3.78% lat (msec) : 100=0.39% cpu : usr=0.01%, sys=1.21%, ctx=2967, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=16464: Mon Nov 11 15:02:23 2024 read: IOPS=3, BW=58.7KiB/s (60.1kB/s)(6256KiB/106642msec) clat (usec): min=59, max=50023k, avg=139310.33, stdev=2529192.47 lat (usec): min=60, max=50023k, avg=139311.28, stdev=2529192.46 clat percentiles (usec): | 1.00th=[ 126], 5.00th=[ 6783], 10.00th=[ 7570], | 20.00th=[ 8291], 30.00th=[ 8979], 40.00th=[ 9765], | 50.00th=[ 10552], 60.00th=[ 11338], 70.00th=[ 12125], | 80.00th=[ 13566], 90.00th=[ 16909], 95.00th=[ 20317], | 99.00th=[ 28705], 99.50th=[ 70779], 99.90th=[17112761], | 99.95th=[17112761], 99.99th=[17112761] bw ( KiB/s): min= 95, max= 1344, per=100.00%, avg=810.40, stdev=334.26, samples=15 iops : min= 5, max= 84, avg=50.47, stdev=21.11, samples=15 write: IOPS=3, BW=56.6KiB/s (57.9kB/s)(6032KiB/106642msec) clat (usec): min=695, max=49999k, avg=138284.10, stdev=2574791.08 lat (usec): min=696, max=49999k, avg=138285.16, stdev=2574791.07 clat percentiles (usec): | 1.00th=[ 758], 5.00th=[ 799], 10.00th=[ 947], | 20.00th=[ 1876], 30.00th=[ 2376], 40.00th=[ 2835], | 50.00th=[ 3752], 60.00th=[ 4817], 70.00th=[ 5932], | 80.00th=[ 8291], 90.00th=[ 13435], 95.00th=[ 17695], | 99.00th=[ 32113], 99.50th=[ 69731], 99.90th=[17112761], | 99.95th=[17112761], 99.99th=[17112761] bw ( KiB/s): min= 95, max= 1312, per=100.00%, avg=761.33, stdev=342.18, samples=15 iops : min= 5, max= 82, avg=47.40, stdev=21.62, samples=15 lat (usec) : 100=0.39%, 250=0.52%, 750=0.13%, 1000=5.47% lat (msec) : 2=5.34%, 4=14.97%, 10=36.85%, 20=32.42%, 50=3.26% lat (msec) : 100=0.39% cpu : usr=0.00%, sys=1.25%, ctx=3081, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=16465: Mon Nov 11 15:02:23 2024 read: IOPS=3, BW=56.6KiB/s (57.0kB/s)(6032KiB/106563msec) clat (usec): min=59, max=49358, avg=11035.63, stdev=4791.07 lat (usec): min=60, max=49359, avg=11036.24, stdev=4791.14 clat percentiles (usec): | 1.00th=[ 63], 5.00th=[ 6194], 10.00th=[ 7111], 20.00th=[ 8160], | 30.00th=[ 8979], 40.00th=[ 9503], 50.00th=[10159], 60.00th=[11076], | 70.00th=[12125], 80.00th=[13698], 90.00th=[16712], 95.00th=[19006], | 99.00th=[26084], 99.50th=[30278], 99.90th=[49546], 99.95th=[49546], | 99.99th=[49546] bw ( KiB/s): min= 96, max= 1120, per=100.00%, avg=789.33, stdev=312.99, samples=15 iops : min= 6, max= 70, avg=49.33, stdev=19.56, samples=15 write: IOPS=3, BW=58.7KiB/s (60.1kB/s)(6256KiB/106563msec) clat (usec): min=618, max=50010k, avg=261700.08, stdev=3571635.56 lat (usec): min=619, max=50010k, avg=261700.99, stdev=3571635.57 clat percentiles (usec): | 1.00th=[ 742], 5.00th=[ 799], 10.00th=[ 914], | 20.00th=[ 1778], 30.00th=[ 2245], 40.00th=[ 2671], | 50.00th=[ 3458], 60.00th=[ 4752], 70.00th=[ 5932], | 80.00th=[ 8455], 90.00th=[ 14222], 95.00th=[ 19268], | 99.00th=[ 70779], 99.50th=[17112761], 99.90th=[17112761], | 99.95th=[17112761], 99.99th=[17112761] bw ( KiB/s): min= 96, max= 1344, per=100.00%, avg=825.60, stdev=348.91, samples=15 iops : min= 6, max= 84, avg=51.60, stdev=21.81, samples=15 lat (usec) : 100=1.04%, 250=0.65%, 750=0.78%, 1000=5.99% lat (msec) : 2=4.69%, 4=16.67%, 10=35.16%, 20=31.64%, 50=2.86% lat (msec) : 100=0.26% cpu : usr=0.00%, sys=1.18%, ctx=2976, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=225KiB/s (230kB/s), 53.9KiB/s-58.7KiB/s (55.2kB/s-60.1kB/s), io=23.4MiB (24.5MB), run=106505-106642msec WRITE: bw=236KiB/s (242kB/s), 56.6KiB/s-61.4KiB/s (57.9kB/s-62.9kB/s), io=24.6MiB (25.8MB), run=106505-106642msec rand-rw: (groupid=0, jobs=1): err= 0: pid=16458: Mon Nov 11 15:02:23 2024 read: IOPS=3, BW=55.6KiB/s (56.9kB/s)(5920KiB/106521msec) clat (usec): min=1705, max=49977k, avg=142087.82, stdev=2597803.23 lat (usec): min=1705, max=49977k, avg=142088.46, stdev=2597803.24 clat percentiles (usec): | 1.00th=[ 1745], 5.00th=[ 1909], 10.00th=[ 2057], | 20.00th=[ 2376], 30.00th=[ 2835], 40.00th=[ 3458], | 50.00th=[ 4752], 60.00th=[ 6521], 70.00th=[ 8455], | 80.00th=[ 10945], 90.00th=[ 13304], 95.00th=[ 18220], | 99.00th=[ 32375], 99.50th=[ 83362], 99.90th=[17112761], | 99.95th=[17112761], 99.99th=[17112761] bw ( KiB/s): min= 96, max= 1312, per=100.00%, avg=784.73, stdev=321.40, samples=15 iops : min= 6, max= 82, avg=48.93, stdev=20.07, samples=15 write: IOPS=3, BW=59.8KiB/s (61.2kB/s)(6368KiB/106521msec) clat (msec): min=2, max=50019, avg=135.37, stdev=2506.77 lat (msec): min=2, max=50019, avg=135.37, stdev=2506.77 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 4], 10.00th=[ 4], 20.00th=[ 5], | 30.00th=[ 6], 40.00th=[ 7], 50.00th=[ 8], 60.00th=[ 9], | 70.00th=[ 11], 80.00th=[ 14], 90.00th=[ 18], 95.00th=[ 27], | 99.00th=[ 41], 99.50th=[ 73], 99.90th=[17113], 99.95th=[17113], | 99.99th=[17113] bw ( KiB/s): min= 160, max= 1280, per=100.00%, avg=844.53, stdev=361.72, samples=15 iops : min= 10, max= 80, avg=52.67, stdev=22.65, samples=15 lat (msec) : 2=3.65%, 4=26.82%, 10=38.80%, 20=25.13%, 50=5.08% lat (msec) : 100=0.26% cpu : usr=0.01%, sys=0.52%, ctx=1242, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=16459: Mon Nov 11 15:02:23 2024 read: IOPS=3, BW=53.9KiB/s (55.2kB/s)(5744KiB/106573msec) clat (usec): min=1615, max=50011k, avg=146058.67, stdev=2639129.80 lat (usec): min=1615, max=50011k, avg=146059.34, stdev=2639129.79 clat percentiles (usec): | 1.00th=[ 1680], 5.00th=[ 1860], 10.00th=[ 1991], | 20.00th=[ 2311], 30.00th=[ 2704], 40.00th=[ 3425], | 50.00th=[ 4359], 60.00th=[ 6063], 70.00th=[ 7635], | 80.00th=[ 9503], 90.00th=[ 13566], 95.00th=[ 17957], | 99.00th=[ 38011], 99.50th=[ 96994], 99.90th=[17112761], | 99.95th=[17112761], 99.99th=[17112761] bw ( KiB/s): min= 159, max= 1181, per=100.00%, avg=752.80, stdev=308.17, samples=15 iops : min= 9, max= 73, avg=46.93, stdev=19.31, samples=15 write: IOPS=3, BW=61.4KiB/s (62.9kB/s)(6544KiB/106573msec) clat (msec): min=2, max=50022, avg=132.32, stdev=2472.98 lat (msec): min=2, max=50022, avg=132.32, stdev=2472.98 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 4], 10.00th=[ 4], 20.00th=[ 5], | 30.00th=[ 6], 40.00th=[ 7], 50.00th=[ 9], 60.00th=[ 10], | 70.00th=[ 12], 80.00th=[ 14], 90.00th=[ 18], 95.00th=[ 24], | 99.00th=[ 38], 99.50th=[ 82], 99.90th=[17113], 99.95th=[17113], | 99.99th=[17113] bw ( KiB/s): min= 95, max= 1344, per=100.00%, avg=857.40, stdev=328.08, samples=15 iops : min= 5, max= 84, avg=53.47, stdev=20.65, samples=15 lat (msec) : 2=4.69%, 4=24.09%, 10=43.62%, 20=22.01%, 50=4.95% lat (msec) : 100=0.39% cpu : usr=0.01%, sys=0.51%, ctx=1169, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=16460: Mon Nov 11 15:02:23 2024 read: IOPS=3, BW=58.6KiB/s (60.0kB/s)(6256KiB/106722msec) clat (usec): min=1722, max=50018k, avg=135283.25, stdev=2529165.10 lat (usec): min=1723, max=50018k, avg=135284.03, stdev=2529165.10 clat percentiles (usec): | 1.00th=[ 1762], 5.00th=[ 1991], 10.00th=[ 2147], | 20.00th=[ 2474], 30.00th=[ 2933], 40.00th=[ 3687], | 50.00th=[ 4948], 60.00th=[ 6980], 70.00th=[ 8029], | 80.00th=[ 10814], 90.00th=[ 14877], 95.00th=[ 21103], | 99.00th=[ 44827], 99.50th=[ 80217], 99.90th=[17112761], | 99.95th=[17112761], 99.99th=[17112761] bw ( KiB/s): min= 63, max= 1376, per=100.00%, avg=808.40, stdev=333.55, samples=15 iops : min= 3, max= 86, avg=50.40, stdev=21.07, samples=15 write: IOPS=3, BW=56.5KiB/s (57.9kB/s)(6032KiB/106722msec) clat (msec): min=2, max=50012, avg=142.62, stdev=2575.27 lat (msec): min=2, max=50012, avg=142.63, stdev=2575.27 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 4], 10.00th=[ 4], 20.00th=[ 6], | 30.00th=[ 7], 40.00th=[ 8], 50.00th=[ 9], 60.00th=[ 10], | 70.00th=[ 12], 80.00th=[ 14], 90.00th=[ 18], 95.00th=[ 23], | 99.00th=[ 39], 99.50th=[ 48], 99.90th=[17113], 99.95th=[17113], | 99.99th=[17113] bw ( KiB/s): min= 63, max= 1312, per=100.00%, avg=761.47, stdev=343.03, samples=15 iops : min= 3, max= 82, avg=47.47, stdev=21.68, samples=15 lat (msec) : 2=2.60%, 4=24.61%, 10=43.10%, 20=24.22%, 50=5.08% lat (msec) : 100=0.13% cpu : usr=0.01%, sys=0.55%, ctx=1338, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=16462: Mon Nov 11 15:02:23 2024 read: IOPS=3, BW=56.6KiB/s (57.0kB/s)(6032KiB/106573msec) clat (usec): min=1595, max=58891, avg=6834.84, stdev=6474.05 lat (usec): min=1596, max=58891, avg=6835.48, stdev=6474.05 clat percentiles (usec): | 1.00th=[ 1778], 5.00th=[ 1975], 10.00th=[ 2114], 20.00th=[ 2409], | 30.00th=[ 2835], 40.00th=[ 3359], 50.00th=[ 4424], 60.00th=[ 6063], | 70.00th=[ 8094], 80.00th=[10683], 90.00th=[13042], 95.00th=[18482], | 99.00th=[34341], 99.50th=[41681], 99.90th=[58983], 99.95th=[58983], | 99.99th=[58983] bw ( KiB/s): min= 63, max= 1120, per=100.00%, avg=778.40, stdev=292.88, samples=15 iops : min= 3, max= 70, avg=48.47, stdev=18.52, samples=15 write: IOPS=3, BW=58.7KiB/s (60.1kB/s)(6256KiB/106573msec) clat (msec): min=2, max=50031, avg=265.85, stdev=3572.19 lat (msec): min=2, max=50031, avg=265.85, stdev=3572.19 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 4], 20.00th=[ 5], | 30.00th=[ 6], 40.00th=[ 7], 50.00th=[ 9], 60.00th=[ 10], | 70.00th=[ 12], 80.00th=[ 15], 90.00th=[ 20], 95.00th=[ 24], | 99.00th=[ 66], 99.50th=[17113], 99.90th=[17113], 99.95th=[17113], | 99.99th=[17113] bw ( KiB/s): min= 159, max= 1216, per=100.00%, avg=812.40, stdev=321.46, samples=15 iops : min= 9, max= 76, avg=50.60, stdev=20.25, samples=15 lat (msec) : 2=2.86%, 4=26.82%, 10=41.41%, 20=22.40%, 50=5.73% lat (msec) : 100=0.52% cpu : usr=0.01%, sys=0.53%, ctx=1227, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=224KiB/s (230kB/s), 53.9KiB/s-58.6KiB/s (55.2kB/s-60.0kB/s), io=23.4MiB (24.5MB), run=106521-106722msec WRITE: bw=236KiB/s (242kB/s), 56.5KiB/s-61.4KiB/s (57.9kB/s-62.9kB/s), io=24.6MiB (25.8MB), run=106521-106722msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=16556: Mon Nov 11 15:02:30 2024 read: IOPS=1, BW=1639KiB/s (1678kB/s)(7168KiB/4374msec) clat (msec): min=42, max=1036, avg=562.36, stdev=427.85 lat (msec): min=42, max=1036, avg=562.37, stdev=427.85 clat percentiles (msec): | 1.00th=[ 43], 5.00th=[ 43], 10.00th=[ 43], 20.00th=[ 56], | 30.00th=[ 266], 40.00th=[ 266], 50.00th=[ 768], 60.00th=[ 810], | 70.00th=[ 810], 80.00th=[ 953], 90.00th=[ 1036], 95.00th=[ 1036], | 99.00th=[ 1036], 99.50th=[ 1036], 99.90th=[ 1036], 99.95th=[ 1036], | 99.99th=[ 1036] bw ( KiB/s): min= 1681, max= 2048, per=48.17%, avg=1901.20, stdev=201.01, samples=5 iops : min= 1, max= 2, avg= 1.60, stdev= 0.55, samples=5 write: IOPS=1, BW=1171KiB/s (1199kB/s)(5120KiB/4374msec) clat (msec): min=42, max=185, avg=84.58, stdev=61.11 lat (msec): min=43, max=185, avg=84.67, stdev=61.08 clat percentiles (msec): | 1.00th=[ 43], 5.00th=[ 43], 10.00th=[ 43], 20.00th=[ 43], | 30.00th=[ 46], 40.00th=[ 46], 50.00th=[ 50], 60.00th=[ 50], | 70.00th=[ 100], 80.00th=[ 100], 90.00th=[ 186], 95.00th=[ 186], | 99.00th=[ 186], 99.50th=[ 186], 99.90th=[ 186], 99.95th=[ 186], | 99.99th=[ 186] bw ( KiB/s): min= 2048, max= 5044, per=64.17%, avg=3546.00, stdev=2118.49, samples=2 iops : min= 2, max= 4, avg= 3.00, stdev= 1.41, samples=2 lat (msec) : 50=33.33%, 100=16.67%, 250=8.33%, 500=8.33%, 1000=25.00% cpu : usr=0.00%, sys=30.39%, ctx=1318, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=16557: Mon Nov 11 15:02:30 2024 read: IOPS=0, BW=468KiB/s (479kB/s)(2048KiB/4374msec) clat (usec): min=71154, max=73142, avg=72148.31, stdev=1405.57 lat (usec): min=71155, max=73142, avg=72149.27, stdev=1405.06 clat percentiles (usec): | 1.00th=[70779], 5.00th=[70779], 10.00th=[70779], 20.00th=[70779], | 30.00th=[70779], 40.00th=[70779], 50.00th=[70779], 60.00th=[72877], | 70.00th=[72877], 80.00th=[72877], 90.00th=[72877], 95.00th=[72877], | 99.00th=[72877], 99.50th=[72877], 99.90th=[72877], 99.95th=[72877], | 99.99th=[72877] bw ( KiB/s): min= 2048, max= 2048, per=51.89%, avg=2048.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 write: IOPS=2, BW=2341KiB/s (2397kB/s)(10.0MiB/4374msec) clat (msec): min=21, max=1783, avg=419.68, stdev=579.40 lat (msec): min=21, max=1783, avg=419.74, stdev=579.38 clat percentiles (msec): | 1.00th=[ 22], 5.00th=[ 22], 10.00th=[ 22], 20.00th=[ 45], | 30.00th=[ 46], 40.00th=[ 52], 50.00th=[ 75], 60.00th=[ 199], | 70.00th=[ 768], 80.00th=[ 768], 90.00th=[ 944], 95.00th=[ 1787], | 99.00th=[ 1787], 99.50th=[ 1787], 99.90th=[ 1787], 99.95th=[ 1787], | 99.99th=[ 1787] bw ( KiB/s): min= 1678, max= 6144, per=72.41%, avg=4001.25, stdev=1826.22, samples=4 iops : min= 1, max= 6, avg= 3.50, stdev= 2.08, samples=4 lat (msec) : 50=25.00%, 100=33.33%, 250=8.33%, 500=8.33%, 1000=16.67% cpu : usr=0.00%, sys=8.64%, ctx=107, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=16558: Mon Nov 11 15:02:30 2024 read: IOPS=1, BW=1184KiB/s (1213kB/s)(6144KiB/5188msec) clat (usec): min=1176, max=1053.9k, avg=368261.30, stdev=481292.16 lat (usec): min=1178, max=1053.9k, avg=368262.69, stdev=481292.01 clat percentiles (usec): | 1.00th=[ 1172], 5.00th=[ 1172], 10.00th=[ 1172], | 20.00th=[ 26084], 30.00th=[ 26084], 40.00th=[ 38536], | 50.00th=[ 38536], 60.00th=[ 181404], 70.00th=[ 910164], | 80.00th=[ 910164], 90.00th=[1052771], 95.00th=[1052771], | 99.00th=[1052771], 99.50th=[1052771], 99.90th=[1052771], | 99.95th=[1052771], 99.99th=[1052771] bw ( KiB/s): min= 4087, max= 5027, per=100.00%, avg=4557.00, stdev=664.68, samples=2 iops : min= 3, max= 4, avg= 3.50, stdev= 0.71, samples=2 write: IOPS=1, BW=1184KiB/s (1213kB/s)(6144KiB/5188msec) clat (msec): min=30, max=1774, avg=484.44, stdev=688.73 lat (msec): min=30, max=1774, avg=484.50, stdev=688.71 clat percentiles (msec): | 1.00th=[ 31], 5.00th=[ 31], 10.00th=[ 31], 20.00th=[ 43], | 30.00th=[ 43], 40.00th=[ 47], 50.00th=[ 47], 60.00th=[ 268], | 70.00th=[ 743], 80.00th=[ 743], 90.00th=[ 1770], 95.00th=[ 1770], | 99.00th=[ 1770], 99.50th=[ 1770], 99.90th=[ 1770], 99.95th=[ 1770], | 99.99th=[ 1770] bw ( KiB/s): min= 1675, max= 4087, per=41.72%, avg=2305.60, stdev=1012.92, samples=5 iops : min= 1, max= 3, avg= 1.60, stdev= 0.89, samples=5 lat (msec) : 2=8.33%, 50=41.67%, 250=8.33%, 500=8.33%, 750=8.33% lat (msec) : 1000=8.33% cpu : usr=0.02%, sys=10.18%, ctx=880, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=16559: Mon Nov 11 15:02:30 2024 read: IOPS=1, BW=1742KiB/s (1784kB/s)(5120KiB/2939msec) clat (usec): min=317, max=93435, avg=27222.28, stdev=39610.12 lat (usec): min=318, max=93435, avg=27223.17, stdev=39609.99 clat percentiles (usec): | 1.00th=[ 318], 5.00th=[ 318], 10.00th=[ 318], 20.00th=[ 318], | 30.00th=[ 334], 40.00th=[ 334], 50.00th=[ 7439], 60.00th=[ 7439], | 70.00th=[34341], 80.00th=[34341], 90.00th=[93848], 95.00th=[93848], | 99.00th=[93848], 99.50th=[93848], 99.90th=[93848], 99.95th=[93848], | 99.99th=[93848] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=2, BW=2439KiB/s (2497kB/s)(7168KiB/2939msec) clat (msec): min=25, max=1753, avg=391.51, stdev=649.38 lat (msec): min=25, max=1753, avg=391.58, stdev=649.39 clat percentiles (msec): | 1.00th=[ 26], 5.00th=[ 26], 10.00th=[ 26], 20.00th=[ 36], | 30.00th=[ 50], 40.00th=[ 50], 50.00th=[ 78], 60.00th=[ 86], | 70.00th=[ 86], 80.00th=[ 718], 90.00th=[ 1754], 95.00th=[ 1754], | 99.00th=[ 1754], 99.50th=[ 1754], 99.90th=[ 1754], 99.95th=[ 1754], | 99.99th=[ 1754] bw ( KiB/s): min= 2048, max= 6131, per=61.69%, avg=3409.00, stdev=2357.32, samples=3 iops : min= 2, max= 5, avg= 3.00, stdev= 1.73, samples=3 lat (usec) : 500=16.67% lat (msec) : 10=8.33%, 50=33.33%, 100=25.00%, 750=8.33% cpu : usr=0.00%, sys=10.62%, ctx=84, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=3948KiB/s (4042kB/s), 468KiB/s-1742KiB/s (479kB/s-1784kB/s), io=20.0MiB (20.0MB), run=2939-5188msec WRITE: bw=5527KiB/s (5659kB/s), 1171KiB/s-2439KiB/s (1199kB/s-2497kB/s), io=28.0MiB (29.4MB), run=2939-5188msec rand-rw: (groupid=0, jobs=1): err= 0: pid=16560: Mon Nov 11 15:02:30 2024 read: IOPS=1, BW=1610KiB/s (1649kB/s)(7168KiB/4452msec) clat (msec): min=18, max=1140, avg=544.21, stdev=495.21 lat (msec): min=18, max=1140, avg=544.21, stdev=495.21 clat percentiles (msec): | 1.00th=[ 19], 5.00th=[ 19], 10.00th=[ 19], 20.00th=[ 24], | 30.00th=[ 54], 40.00th=[ 54], 50.00th=[ 776], 60.00th=[ 785], | 70.00th=[ 785], 80.00th=[ 1011], 90.00th=[ 1133], 95.00th=[ 1133], | 99.00th=[ 1133], 99.50th=[ 1133], 99.90th=[ 1133], 99.95th=[ 1133], | 99.99th=[ 1133] bw ( KiB/s): min= 1678, max= 4096, per=62.88%, avg=2467.50, stdev=1099.59, samples=4 iops : min= 1, max= 4, avg= 2.25, stdev= 1.26, samples=4 write: IOPS=1, BW=1150KiB/s (1178kB/s)(5120KiB/4452msec) clat (msec): min=19, max=333, avg=119.07, stdev=133.02 lat (msec): min=19, max=333, avg=119.14, stdev=133.02 clat percentiles (msec): | 1.00th=[ 20], 5.00th=[ 20], 10.00th=[ 20], 20.00th=[ 20], | 30.00th=[ 23], 40.00th=[ 23], 50.00th=[ 59], 60.00th=[ 59], | 70.00th=[ 161], 80.00th=[ 161], 90.00th=[ 334], 95.00th=[ 334], | 99.00th=[ 334], 99.50th=[ 334], 99.90th=[ 334], 99.95th=[ 334], | 99.99th=[ 334] bw ( KiB/s): min= 2048, max= 2048, per=37.28%, avg=2048.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 20=16.67%, 50=16.67%, 100=16.67%, 250=8.33%, 500=8.33% lat (msec) : 1000=16.67% cpu : usr=0.04%, sys=3.30%, ctx=69, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=16561: Mon Nov 11 15:02:30 2024 read: IOPS=0, BW=463KiB/s (474kB/s)(2048KiB/4420msec) clat (usec): min=12852, max=24756, avg=18804.33, stdev=8417.67 lat (usec): min=12853, max=24757, avg=18805.28, stdev=8417.76 clat percentiles (usec): | 1.00th=[12911], 5.00th=[12911], 10.00th=[12911], 20.00th=[12911], | 30.00th=[12911], 40.00th=[12911], 50.00th=[12911], 60.00th=[24773], | 70.00th=[24773], 80.00th=[24773], 90.00th=[24773], 95.00th=[24773], | 99.00th=[24773], 99.50th=[24773], 99.90th=[24773], 99.95th=[24773], | 99.99th=[24773] bw ( KiB/s): min= 2043, max= 2048, per=52.13%, avg=2045.50, stdev= 3.54, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 write: IOPS=2, BW=2317KiB/s (2372kB/s)(10.0MiB/4420msec) clat (msec): min=40, max=1820, avg=430.41, stdev=587.28 lat (msec): min=40, max=1820, avg=430.48, stdev=587.27 clat percentiles (msec): | 1.00th=[ 41], 5.00th=[ 41], 10.00th=[ 41], 20.00th=[ 42], | 30.00th=[ 63], 40.00th=[ 64], 50.00th=[ 65], 60.00th=[ 209], | 70.00th=[ 844], 80.00th=[ 844], 90.00th=[ 902], 95.00th=[ 1821], | 99.00th=[ 1821], 99.50th=[ 1821], 99.90th=[ 1821], 99.95th=[ 1821], | 99.99th=[ 1821] bw ( KiB/s): min= 1675, max= 6131, per=69.40%, avg=3812.00, stdev=1847.73, samples=4 iops : min= 1, max= 5, avg= 3.25, stdev= 1.71, samples=4 lat (msec) : 20=8.33%, 50=25.00%, 100=25.00%, 250=8.33%, 500=8.33% lat (msec) : 1000=16.67% cpu : usr=0.00%, sys=3.91%, ctx=72, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=16562: Mon Nov 11 15:02:30 2024 read: IOPS=1, BW=1177KiB/s (1205kB/s)(6144KiB/5219msec) clat (msec): min=16, max=1011, avg=355.48, stdev=456.62 lat (msec): min=16, max=1011, avg=355.48, stdev=456.62 clat percentiles (msec): | 1.00th=[ 17], 5.00th=[ 17], 10.00th=[ 17], 20.00th=[ 40], | 30.00th=[ 40], 40.00th=[ 70], 50.00th=[ 70], 60.00th=[ 127], | 70.00th=[ 869], 80.00th=[ 869], 90.00th=[ 1011], 95.00th=[ 1011], | 99.00th=[ 1011], 99.50th=[ 1011], 99.90th=[ 1011], 99.95th=[ 1011], | 99.99th=[ 1011] bw ( KiB/s): min= 1673, max= 4096, per=77.47%, avg=3040.00, stdev=1241.08, samples=3 iops : min= 1, max= 4, avg= 2.67, stdev= 1.53, samples=3 write: IOPS=1, BW=1177KiB/s (1205kB/s)(6144KiB/5219msec) clat (msec): min=27, max=1733, avg=513.08, stdev=653.05 lat (msec): min=27, max=1733, avg=513.13, stdev=653.04 clat percentiles (msec): | 1.00th=[ 28], 5.00th=[ 28], 10.00th=[ 28], 20.00th=[ 73], | 30.00th=[ 73], 40.00th=[ 150], 50.00th=[ 150], 60.00th=[ 351], | 70.00th=[ 751], 80.00th=[ 751], 90.00th=[ 1737], 95.00th=[ 1737], | 99.00th=[ 1737], 99.50th=[ 1737], 99.90th=[ 1737], 99.95th=[ 1737], | 99.99th=[ 1737] bw ( KiB/s): min= 2043, max= 4096, per=52.49%, avg=2883.25, stdev=1014.65, samples=4 iops : min= 1, max= 4, avg= 2.50, stdev= 1.29, samples=4 lat (msec) : 20=8.33%, 50=16.67%, 100=16.67%, 250=16.67%, 500=8.33% lat (msec) : 750=8.33%, 1000=8.33% cpu : usr=0.02%, sys=2.30%, ctx=33, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=16563: Mon Nov 11 15:02:30 2024 read: IOPS=1, BW=1653KiB/s (1693kB/s)(5120KiB/3097msec) clat (usec): min=11087, max=54300, avg=26086.47, stdev=16713.10 lat (usec): min=11087, max=54300, avg=26087.38, stdev=16713.07 clat percentiles (usec): | 1.00th=[11076], 5.00th=[11076], 10.00th=[11076], 20.00th=[11076], | 30.00th=[16581], 40.00th=[16581], 50.00th=[23987], 60.00th=[23987], | 70.00th=[24249], 80.00th=[24249], 90.00th=[54264], 95.00th=[54264], | 99.00th=[54264], 99.50th=[54264], 99.90th=[54264], 99.95th=[54264], | 99.99th=[54264] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=2, BW=2314KiB/s (2370kB/s)(7168KiB/3097msec) clat (msec): min=26, max=1773, avg=414.33, stdev=663.97 lat (msec): min=26, max=1773, avg=414.40, stdev=663.96 clat percentiles (msec): | 1.00th=[ 27], 5.00th=[ 27], 10.00th=[ 27], 20.00th=[ 37], | 30.00th=[ 38], 40.00th=[ 38], 50.00th=[ 88], 60.00th=[ 115], | 70.00th=[ 115], 80.00th=[ 827], 90.00th=[ 1770], 95.00th=[ 1770], | 99.00th=[ 1770], 99.50th=[ 1770], 99.90th=[ 1770], 99.95th=[ 1770], | 99.99th=[ 1770] bw ( KiB/s): min= 2043, max= 4096, per=62.05%, avg=3408.67, stdev=1182.71, samples=3 iops : min= 1, max= 4, avg= 2.67, stdev= 1.53, samples=3 lat (msec) : 20=16.67%, 50=41.67%, 100=16.67%, 250=8.33%, 1000=8.33% cpu : usr=0.03%, sys=4.04%, ctx=182, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=3924KiB/s (4018kB/s), 463KiB/s-1653KiB/s (474kB/s-1693kB/s), io=20.0MiB (20.0MB), run=3097-5219msec WRITE: bw=5494KiB/s (5626kB/s), 1150KiB/s-2317KiB/s (1178kB/s-2372kB/s), io=28.0MiB (29.4MB), run=3097-5219msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=16572: Mon Nov 11 15:03:27 2024 read: IOPS=0, BW=220KiB/s (226kB/s)(12.0MiB/55781msec) clat (msec): min=197, max=53085, avg=18590.74, stdev=29895.29 lat (msec): min=197, max=53085, avg=18590.74, stdev=29895.29 clat percentiles (msec): | 1.00th=[ 199], 5.00th=[ 199], 10.00th=[ 199], 20.00th=[ 199], | 30.00th=[ 199], 40.00th=[ 2500], 50.00th=[ 2500], 60.00th=[ 2500], | 70.00th=[17113], 80.00th=[17113], 90.00th=[17113], 95.00th=[17113], | 99.00th=[17113], 99.50th=[17113], 99.90th=[17113], 99.95th=[17113], | 99.99th=[17113] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 lat (msec) : 250=33.33% cpu : usr=0.00%, sys=4.23%, ctx=1983, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=16573: Mon Nov 11 15:03:27 2024 read: IOPS=0, BW=72.9KiB/s (74.7kB/s)(4096KiB/56150msec) clat (nsec): min=182471k, max=182471k, avg=182470516.00, stdev= 0.00 lat (nsec): min=182473k, max=182473k, avg=182473191.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 184], 5.00th=[ 184], 10.00th=[ 184], 20.00th=[ 184], | 30.00th=[ 184], 40.00th=[ 184], 50.00th=[ 184], 60.00th=[ 184], | 70.00th=[ 184], 80.00th=[ 184], 90.00th=[ 184], 95.00th=[ 184], | 99.00th=[ 184], 99.50th=[ 184], 99.90th=[ 184], 99.95th=[ 184], | 99.99th=[ 184] bw ( KiB/s): min= 8094, max= 8094, per=100.00%, avg=8094.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=0, BW=146KiB/s (149kB/s)(8192KiB/56150msec) clat (msec): min=581, max=55371, avg=27976.53, stdev=38741.94 lat (msec): min=582, max=55371, avg=27976.77, stdev=38742.00 clat percentiles (msec): | 1.00th=[ 584], 5.00th=[ 584], 10.00th=[ 584], 20.00th=[ 584], | 30.00th=[ 584], 40.00th=[ 584], 50.00th=[ 584], 60.00th=[17113], | 70.00th=[17113], 80.00th=[17113], 90.00th=[17113], 95.00th=[17113], | 99.00th=[17113], 99.50th=[17113], 99.90th=[17113], 99.95th=[17113], | 99.99th=[17113] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 750=33.33% cpu : usr=0.00%, sys=0.71%, ctx=83, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=16574: Mon Nov 11 15:03:27 2024 write: IOPS=0, BW=231KiB/s (237kB/s)(12.0MiB/53200msec) clat (msec): min=282, max=50348, avg=17727.36, stdev=28273.38 lat (msec): min=282, max=50348, avg=17727.65, stdev=28273.40 clat percentiles (msec): | 1.00th=[ 284], 5.00th=[ 284], 10.00th=[ 284], 20.00th=[ 284], | 30.00th=[ 284], 40.00th=[ 2567], 50.00th=[ 2567], 60.00th=[ 2567], | 70.00th=[17113], 80.00th=[17113], 90.00th=[17113], 95.00th=[17113], | 99.00th=[17113], 99.50th=[17113], 99.90th=[17113], 99.95th=[17113], | 99.99th=[17113] bw ( KiB/s): min= 8143, max= 8159, per=100.00%, avg=8151.00, stdev=11.31, samples=2 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=2 lat (msec) : 500=33.33% cpu : usr=0.00%, sys=0.57%, ctx=78, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=16575: Mon Nov 11 15:03:27 2024 write: IOPS=0, BW=232KiB/s (237kB/s)(12.0MiB/53023msec) clat (msec): min=393, max=50102, avg=17669.27, stdev=28108.14 lat (msec): min=393, max=50103, avg=17669.55, stdev=28108.20 clat percentiles (msec): | 1.00th=[ 393], 5.00th=[ 393], 10.00th=[ 393], 20.00th=[ 393], | 30.00th=[ 393], 40.00th=[ 2500], 50.00th=[ 2500], 60.00th=[ 2500], | 70.00th=[17113], 80.00th=[17113], 90.00th=[17113], 95.00th=[17113], | 99.00th=[17113], 99.50th=[17113], 99.90th=[17113], 99.95th=[17113], | 99.99th=[17113] bw ( KiB/s): min= 8126, max= 8175, per=100.00%, avg=8150.50, stdev=34.65, samples=2 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=2 lat (msec) : 500=33.33% cpu : usr=0.00%, sys=0.55%, ctx=56, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=292KiB/s (299kB/s), 72.9KiB/s-220KiB/s (74.7kB/s-226kB/s), io=16.0MiB (16.8MB), run=55781-56150msec WRITE: bw=584KiB/s (598kB/s), 146KiB/s-232KiB/s (149kB/s-237kB/s), io=32.0MiB (33.6MB), run=53023-56150msec rand-rw: (groupid=0, jobs=1): err= 0: pid=16577: Mon Nov 11 15:03:27 2024 read: IOPS=0, BW=220KiB/s (226kB/s)(12.0MiB/55752msec) clat (msec): min=458, max=52599, avg=18570.84, stdev=29490.30 lat (msec): min=458, max=52599, avg=18570.84, stdev=29490.30 clat percentiles (msec): | 1.00th=[ 460], 5.00th=[ 460], 10.00th=[ 460], 20.00th=[ 460], | 30.00th=[ 460], 40.00th=[ 2668], 50.00th=[ 2668], 60.00th=[ 2668], | 70.00th=[17113], 80.00th=[17113], 90.00th=[17113], 95.00th=[17113], | 99.00th=[17113], 99.50th=[17113], 99.90th=[17113], 99.95th=[17113], | 99.99th=[17113] bw ( KiB/s): min= 7529, max= 8192, per=100.00%, avg=7860.50, stdev=468.81, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 500=33.33% cpu : usr=0.00%, sys=0.25%, ctx=52, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=16578: Mon Nov 11 15:03:27 2024 read: IOPS=0, BW=72.6KiB/s (74.3kB/s)(4096KiB/56419msec) clat (nsec): min=430942k, max=430942k, avg=430941721.00, stdev= 0.00 lat (nsec): min=430944k, max=430944k, avg=430943512.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 430], 5.00th=[ 430], 10.00th=[ 430], 20.00th=[ 430], | 30.00th=[ 430], 40.00th=[ 430], 50.00th=[ 430], 60.00th=[ 430], | 70.00th=[ 430], 80.00th=[ 430], 90.00th=[ 430], 95.00th=[ 430], | 99.00th=[ 430], 99.50th=[ 430], 99.90th=[ 430], 99.95th=[ 430], | 99.99th=[ 430] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=0, BW=145KiB/s (149kB/s)(8192KiB/56419msec) clat (msec): min=1269, max=54703, avg=27986.19, stdev=37783.34 lat (msec): min=1269, max=54703, avg=27986.56, stdev=37783.35 clat percentiles (msec): | 1.00th=[ 1267], 5.00th=[ 1267], 10.00th=[ 1267], 20.00th=[ 1267], | 30.00th=[ 1267], 40.00th=[ 1267], 50.00th=[ 1267], 60.00th=[17113], | 70.00th=[17113], 80.00th=[17113], 90.00th=[17113], 95.00th=[17113], | 99.00th=[17113], 99.50th=[17113], 99.90th=[17113], 99.95th=[17113], | 99.99th=[17113] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 500=33.33% cpu : usr=0.00%, sys=0.27%, ctx=80, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=16579: Mon Nov 11 15:03:27 2024 write: IOPS=0, BW=217KiB/s (222kB/s)(12.0MiB/56568msec) clat (msec): min=97, max=56040, avg=18849.92, stdev=32208.67 lat (msec): min=97, max=56041, avg=18850.18, stdev=32208.59 clat percentiles (msec): | 1.00th=[ 97], 5.00th=[ 97], 10.00th=[ 97], 20.00th=[ 97], | 30.00th=[ 97], 40.00th=[ 414], 50.00th=[ 414], 60.00th=[ 414], | 70.00th=[17113], 80.00th=[17113], 90.00th=[17113], 95.00th=[17113], | 99.00th=[17113], 99.50th=[17113], 99.90th=[17113], 99.95th=[17113], | 99.99th=[17113] bw ( KiB/s): min= 8175, max= 8175, per=100.00%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 500=33.33% cpu : usr=0.00%, sys=0.25%, ctx=62, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=16580: Mon Nov 11 15:03:27 2024 write: IOPS=0, BW=231KiB/s (236kB/s)(12.0MiB/53277msec) clat (msec): min=77, max=52569, avg=17742.39, stdev=30162.54 lat (msec): min=77, max=52570, avg=17742.74, stdev=30162.56 clat percentiles (msec): | 1.00th=[ 78], 5.00th=[ 78], 10.00th=[ 78], 20.00th=[ 78], | 30.00th=[ 78], 40.00th=[ 584], 50.00th=[ 584], 60.00th=[ 584], | 70.00th=[17113], 80.00th=[17113], 90.00th=[17113], 95.00th=[17113], | 99.00th=[17113], 99.50th=[17113], 99.90th=[17113], 99.95th=[17113], | 99.99th=[17113] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 750=33.33% cpu : usr=0.00%, sys=0.19%, ctx=89, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=290KiB/s (297kB/s), 72.6KiB/s-220KiB/s (74.3kB/s-226kB/s), io=16.0MiB (16.8MB), run=55752-56419msec WRITE: bw=579KiB/s (593kB/s), 145KiB/s-231KiB/s (149kB/s-236kB/s), io=32.0MiB (33.6MB), run=53277-56568msec