== sanity test 398b: DIO and buffer IO race ============== 23:40:44 (1708836044) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.22606 s, 41.1 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=6480: Sat Feb 24 23:41:00 2024 read: IOPS=114, BW=457KiB/s (468kB/s)(5992KiB/13116msec) clat (usec): min=46, max=143741, avg=4867.52, stdev=4657.95 lat (usec): min=46, max=143742, avg=4868.86, stdev=4658.02 clat percentiles (usec): | 1.00th=[ 64], 5.00th=[ 101], 10.00th=[ 2147], 20.00th=[ 2671], | 30.00th=[ 3032], 40.00th=[ 3589], 50.00th=[ 4293], 60.00th=[ 5014], | 70.00th=[ 5735], 80.00th=[ 6718], 90.00th=[ 8225], 95.00th=[ 10028], | 99.00th=[ 14877], 99.50th=[ 17171], 99.90th=[ 27919], 99.95th=[143655], | 99.99th=[143655] bw ( KiB/s): min= 328, max= 568, per=25.12%, avg=455.12, stdev=58.55, samples=26 iops : min= 82, max= 142, avg=113.65, stdev=14.63, samples=26 write: IOPS=120, BW=480KiB/s (492kB/s)(6296KiB/13116msec) clat (usec): min=436, max=24283, avg=3658.86, stdev=3369.01 lat (usec): min=437, max=24284, avg=3660.53, stdev=3368.95 clat percentiles (usec): | 1.00th=[ 474], 5.00th=[ 523], 10.00th=[ 570], 20.00th=[ 676], | 30.00th=[ 807], 40.00th=[ 1598], 50.00th=[ 2933], 60.00th=[ 3720], | 70.00th=[ 4621], 80.00th=[ 6128], 90.00th=[ 8455], 95.00th=[10552], | 99.00th=[13304], 99.50th=[14091], 99.90th=[21627], 99.95th=[24249], | 99.99th=[24249] bw ( KiB/s): min= 360, max= 696, per=25.11%, avg=476.35, stdev=69.56, samples=26 iops : min= 90, max= 174, avg=118.96, stdev=17.37, samples=26 lat (usec) : 50=0.07%, 100=2.25%, 250=0.81%, 500=1.69%, 750=11.49% lat (usec) : 1000=5.05% lat (msec) : 2=3.84%, 4=29.88%, 10=39.32%, 20=5.37%, 50=0.20% lat (msec) : 250=0.03% cpu : usr=0.25%, sys=22.15%, ctx=5597, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6481: Sat Feb 24 23:41:00 2024 read: IOPS=113, BW=452KiB/s (463kB/s)(5940KiB/13130msec) clat (usec): min=45, max=19491, avg=4596.19, stdev=2620.74 lat (usec): min=45, max=19491, avg=4597.07, stdev=2620.71 clat percentiles (usec): | 1.00th=[ 67], 5.00th=[ 1844], 10.00th=[ 2180], 20.00th=[ 2606], | 30.00th=[ 2999], 40.00th=[ 3490], 50.00th=[ 4113], 60.00th=[ 4752], | 70.00th=[ 5407], 80.00th=[ 6390], 90.00th=[ 7898], 95.00th=[ 9372], | 99.00th=[13829], 99.50th=[15926], 99.90th=[19006], 99.95th=[19530], | 99.99th=[19530] bw ( KiB/s): min= 312, max= 566, per=24.87%, avg=450.65, stdev=58.15, samples=26 iops : min= 78, max= 141, avg=112.62, stdev=14.50, samples=26 write: IOPS=120, BW=483KiB/s (495kB/s)(6348KiB/13130msec) clat (usec): min=411, max=144747, avg=3930.35, stdev=5053.92 lat (usec): min=412, max=144748, avg=3931.71, stdev=5053.92 clat percentiles (usec): | 1.00th=[ 469], 5.00th=[ 510], 10.00th=[ 562], 20.00th=[ 676], | 30.00th=[ 840], 40.00th=[ 1876], 50.00th=[ 2999], 60.00th=[ 3785], | 70.00th=[ 4883], 80.00th=[ 6652], 90.00th=[ 8848], 95.00th=[ 11207], | 99.00th=[ 15926], 99.50th=[ 17957], 99.90th=[ 25560], 99.95th=[145753], | 99.99th=[145753] bw ( KiB/s): min= 360, max= 654, per=25.30%, avg=479.85, stdev=74.15, samples=26 iops : min= 90, max= 163, avg=119.92, stdev=18.47, samples=26 lat (usec) : 50=0.03%, 100=1.40%, 250=0.78%, 500=1.95%, 750=11.00% lat (usec) : 1000=4.92% lat (msec) : 2=4.36%, 4=30.92%, 10=39.26%, 20=5.18%, 50=0.16% lat (msec) : 250=0.03% cpu : usr=0.21%, sys=22.14%, ctx=5432, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6482: Sat Feb 24 23:41:00 2024 read: IOPS=113, BW=455KiB/s (466kB/s)(6028KiB/13247msec) clat (usec): min=38, max=20660, avg=4646.95, stdev=2802.55 lat (usec): min=38, max=20661, avg=4647.75, stdev=2802.59 clat percentiles (usec): | 1.00th=[ 65], 5.00th=[ 106], 10.00th=[ 2114], 20.00th=[ 2573], | 30.00th=[ 2966], 40.00th=[ 3425], 50.00th=[ 4080], 60.00th=[ 4948], | 70.00th=[ 5735], 80.00th=[ 6652], 90.00th=[ 8029], 95.00th=[ 9372], | 99.00th=[14353], 99.50th=[17957], 99.90th=[20317], 99.95th=[20579], | 99.99th=[20579] bw ( KiB/s): min= 200, max= 600, per=24.77%, avg=448.85, stdev=89.12, samples=26 iops : min= 50, max= 150, avg=112.15, stdev=22.26, samples=26 write: IOPS=118, BW=473KiB/s (484kB/s)(6260KiB/13247msec) clat (usec): min=388, max=154234, avg=3948.96, stdev=5139.39 lat (usec): min=389, max=154236, avg=3951.46, stdev=5139.59 clat percentiles (usec): | 1.00th=[ 474], 5.00th=[ 515], 10.00th=[ 570], 20.00th=[ 701], | 30.00th=[ 865], 40.00th=[ 2212], 50.00th=[ 3032], 60.00th=[ 3851], | 70.00th=[ 5014], 80.00th=[ 6849], 90.00th=[ 8848], 95.00th=[ 10552], | 99.00th=[ 15008], 99.50th=[ 16450], 99.90th=[ 20055], 99.95th=[154141], | 99.99th=[154141] bw ( KiB/s): min= 280, max= 654, per=24.47%, avg=464.15, stdev=82.49, samples=26 iops : min= 70, max= 163, avg=116.00, stdev=20.57, samples=26 lat (usec) : 50=0.07%, 100=2.18%, 250=0.81%, 500=1.73%, 750=10.38% lat (usec) : 1000=4.85% lat (msec) : 2=4.07%, 4=31.15%, 10=39.58%, 20=5.08%, 50=0.07% lat (msec) : 250=0.03% cpu : usr=0.20%, sys=22.07%, ctx=5490, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6483: Sat Feb 24 23:41:00 2024 read: IOPS=114, BW=458KiB/s (469kB/s)(6052KiB/13218msec) clat (usec): min=50, max=21281, avg=4682.91, stdev=2864.19 lat (usec): min=50, max=21282, avg=4683.91, stdev=2864.16 clat percentiles (usec): | 1.00th=[ 63], 5.00th=[ 97], 10.00th=[ 2089], 20.00th=[ 2573], | 30.00th=[ 2999], 40.00th=[ 3490], 50.00th=[ 4178], 60.00th=[ 4948], | 70.00th=[ 5735], 80.00th=[ 6718], 90.00th=[ 8094], 95.00th=[ 9503], | 99.00th=[14353], 99.50th=[16581], 99.90th=[19792], 99.95th=[21365], | 99.99th=[21365] bw ( KiB/s): min= 288, max= 592, per=24.81%, avg=449.50, stdev=70.87, samples=26 iops : min= 72, max= 148, avg=112.35, stdev=17.72, samples=26 write: IOPS=117, BW=472KiB/s (483kB/s)(6236KiB/13218msec) clat (usec): min=401, max=154054, avg=3893.27, stdev=5238.79 lat (usec): min=402, max=154055, avg=3895.56, stdev=5238.89 clat percentiles (usec): | 1.00th=[ 453], 5.00th=[ 510], 10.00th=[ 562], 20.00th=[ 668], | 30.00th=[ 816], 40.00th=[ 1516], 50.00th=[ 3032], 60.00th=[ 3884], | 70.00th=[ 4948], 80.00th=[ 6456], 90.00th=[ 8717], 95.00th=[ 10421], | 99.00th=[ 16319], 99.50th=[ 18220], 99.90th=[ 36439], 99.95th=[154141], | 99.99th=[154141] bw ( KiB/s): min= 344, max= 640, per=24.43%, avg=463.35, stdev=73.39, samples=26 iops : min= 86, max= 160, avg=115.81, stdev=18.35, samples=26 lat (usec) : 100=2.57%, 250=0.81%, 500=2.38%, 750=10.71%, 1000=4.75% lat (msec) : 2=4.04%, 4=29.49%, 10=40.20%, 20=4.92%, 50=0.10% lat (msec) : 250=0.03% cpu : usr=0.22%, sys=21.93%, ctx=5566, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1813KiB/s (1856kB/s), 452KiB/s-458KiB/s (463kB/s-469kB/s), io=23.4MiB (24.6MB), run=13116-13247msec WRITE: bw=1898KiB/s (1943kB/s), 472KiB/s-483KiB/s (483kB/s-495kB/s), io=24.6MiB (25.7MB), run=13116-13247msec rand-rw: (groupid=0, jobs=1): err= 0: pid=6484: Sat Feb 24 23:41:04 2024 read: IOPS=88, BW=353KiB/s (361kB/s)(5992KiB/16984msec) clat (usec): min=1544, max=148346, avg=4464.24, stdev=5204.11 lat (usec): min=1545, max=148348, avg=4465.20, stdev=5204.11 clat percentiles (usec): | 1.00th=[ 1696], 5.00th=[ 1811], 10.00th=[ 1926], 20.00th=[ 2089], | 30.00th=[ 2245], 40.00th=[ 2474], 50.00th=[ 2835], 60.00th=[ 3359], | 70.00th=[ 4178], 80.00th=[ 5997], 90.00th=[ 9634], 95.00th=[ 12518], | 99.00th=[ 18220], 99.50th=[ 19268], 99.90th=[ 41681], 99.95th=[147850], | 99.99th=[147850] bw ( KiB/s): min= 160, max= 768, per=24.42%, avg=342.58, stdev=179.24, samples=33 iops : min= 40, max= 192, avg=85.48, stdev=44.86, samples=33 write: IOPS=92, BW=371KiB/s (380kB/s)(6296KiB/16984msec) clat (usec): min=2286, max=28785, avg=6508.35, stdev=4148.06 lat (usec): min=2287, max=28786, avg=6509.46, stdev=4148.10 clat percentiles (usec): | 1.00th=[ 2540], 5.00th=[ 2868], 10.00th=[ 3064], 20.00th=[ 3326], | 30.00th=[ 3556], 40.00th=[ 3949], 50.00th=[ 4686], 60.00th=[ 5997], | 70.00th=[ 7767], 80.00th=[ 9634], 90.00th=[12256], 95.00th=[14353], | 99.00th=[21365], 99.50th=[23725], 99.90th=[27132], 99.95th=[28705], | 99.99th=[28705] bw ( KiB/s): min= 208, max= 816, per=24.46%, avg=359.30, stdev=179.90, samples=33 iops : min= 52, max= 204, avg=89.67, stdev=45.02, samples=33 lat (msec) : 2=7.03%, 4=47.40%, 10=31.67%, 20=12.99%, 50=0.88% lat (msec) : 250=0.03% cpu : usr=0.29%, sys=10.53%, ctx=3427, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6485: Sat Feb 24 23:41:04 2024 read: IOPS=86, BW=348KiB/s (356kB/s)(5940KiB/17083msec) clat (usec): min=1563, max=148477, avg=4591.14, stdev=5310.30 lat (usec): min=1564, max=148478, avg=4592.02, stdev=5310.29 clat percentiles (usec): | 1.00th=[ 1713], 5.00th=[ 1827], 10.00th=[ 1909], 20.00th=[ 2073], | 30.00th=[ 2245], 40.00th=[ 2474], 50.00th=[ 2868], 60.00th=[ 3523], | 70.00th=[ 4424], 80.00th=[ 6259], 90.00th=[ 9896], 95.00th=[ 12911], | 99.00th=[ 19530], 99.50th=[ 21890], 99.90th=[ 25297], 99.95th=[147850], | 99.99th=[147850] bw ( KiB/s): min= 160, max= 768, per=24.68%, avg=346.29, stdev=195.06, samples=34 iops : min= 40, max= 192, avg=86.53, stdev=48.78, samples=34 write: IOPS=92, BW=372KiB/s (381kB/s)(6348KiB/17083msec) clat (usec): min=2295, max=32495, avg=6435.34, stdev=4067.43 lat (usec): min=2296, max=32497, avg=6436.38, stdev=4067.47 clat percentiles (usec): | 1.00th=[ 2474], 5.00th=[ 2769], 10.00th=[ 3032], 20.00th=[ 3326], | 30.00th=[ 3523], 40.00th=[ 3851], 50.00th=[ 4490], 60.00th=[ 5866], | 70.00th=[ 7832], 80.00th=[ 9765], 90.00th=[12387], 95.00th=[14746], | 99.00th=[18482], 99.50th=[21365], 99.90th=[25560], 99.95th=[32375], | 99.99th=[32375] bw ( KiB/s): min= 208, max= 768, per=25.13%, avg=369.12, stdev=191.47, samples=34 iops : min= 52, max= 192, avg=92.24, stdev=47.90, samples=34 lat (msec) : 2=7.55%, 4=46.42%, 10=31.64%, 20=13.61%, 50=0.75% lat (msec) : 250=0.03% cpu : usr=0.27%, sys=10.44%, ctx=3490, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6486: Sat Feb 24 23:41:04 2024 read: IOPS=88, BW=352KiB/s (361kB/s)(6028KiB/17108msec) clat (usec): min=1600, max=27573, avg=4579.95, stdev=3835.13 lat (usec): min=1601, max=27573, avg=4580.89, stdev=3835.20 clat percentiles (usec): | 1.00th=[ 1729], 5.00th=[ 1844], 10.00th=[ 1926], 20.00th=[ 2057], | 30.00th=[ 2212], 40.00th=[ 2442], 50.00th=[ 2835], 60.00th=[ 3490], | 70.00th=[ 4490], 80.00th=[ 6521], 90.00th=[10683], 95.00th=[12911], | 99.00th=[17695], 99.50th=[20579], 99.90th=[24249], 99.95th=[27657], | 99.99th=[27657] bw ( KiB/s): min= 168, max= 768, per=24.96%, avg=350.24, stdev=186.25, samples=34 iops : min= 42, max= 192, avg=87.47, stdev=46.63, samples=34 write: IOPS=91, BW=366KiB/s (375kB/s)(6260KiB/17108msec) clat (msec): min=2, max=152, avg= 6.49, stdev= 5.59 lat (msec): min=2, max=152, avg= 6.49, stdev= 5.59 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 4], 20.00th=[ 4], | 30.00th=[ 4], 40.00th=[ 4], 50.00th=[ 5], 60.00th=[ 6], | 70.00th=[ 8], 80.00th=[ 10], 90.00th=[ 13], 95.00th=[ 15], | 99.00th=[ 21], 99.50th=[ 25], 99.90th=[ 40], 99.95th=[ 153], | 99.99th=[ 153] bw ( KiB/s): min= 200, max= 784, per=24.69%, avg=362.71, stdev=197.34, samples=34 iops : min= 50, max= 196, avg=90.59, stdev=49.37, samples=34 lat (msec) : 2=7.49%, 4=46.68%, 10=30.66%, 20=14.29%, 50=0.85% lat (msec) : 250=0.03% cpu : usr=0.20%, sys=10.41%, ctx=3481, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6487: Sat Feb 24 23:41:04 2024 read: IOPS=88, BW=355KiB/s (363kB/s)(6052KiB/17052msec) clat (usec): min=1530, max=27581, avg=4562.07, stdev=3781.80 lat (usec): min=1530, max=27582, avg=4563.00, stdev=3781.79 clat percentiles (usec): | 1.00th=[ 1680], 5.00th=[ 1811], 10.00th=[ 1926], 20.00th=[ 2073], | 30.00th=[ 2212], 40.00th=[ 2507], 50.00th=[ 2900], 60.00th=[ 3523], | 70.00th=[ 4490], 80.00th=[ 6783], 90.00th=[10290], 95.00th=[12911], | 99.00th=[17957], 99.50th=[21103], 99.90th=[27395], 99.95th=[27657], | 99.99th=[27657] bw ( KiB/s): min= 191, max= 768, per=25.20%, avg=353.59, stdev=187.99, samples=34 iops : min= 47, max= 192, avg=88.24, stdev=47.03, samples=34 write: IOPS=91, BW=366KiB/s (374kB/s)(6236KiB/17052msec) clat (msec): min=2, max=152, avg= 6.48, stdev= 5.52 lat (msec): min=2, max=152, avg= 6.48, stdev= 5.52 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 4], 20.00th=[ 4], | 30.00th=[ 4], 40.00th=[ 4], 50.00th=[ 5], 60.00th=[ 6], | 70.00th=[ 8], 80.00th=[ 10], 90.00th=[ 13], 95.00th=[ 15], | 99.00th=[ 22], 99.50th=[ 26], 99.90th=[ 30], 99.95th=[ 153], | 99.99th=[ 153] bw ( KiB/s): min= 199, max= 752, per=24.79%, avg=364.18, stdev=192.73, samples=34 iops : min= 49, max= 188, avg=90.88, stdev=48.20, samples=34 lat (msec) : 2=7.16%, 4=46.81%, 10=31.77%, 20=13.35%, 50=0.88% lat (msec) : 250=0.03% cpu : usr=0.17%, sys=10.46%, ctx=3520, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1404KiB/s (1437kB/s), 348KiB/s-355KiB/s (356kB/s-363kB/s), io=23.4MiB (24.6MB), run=16984-17108msec WRITE: bw=1469KiB/s (1505kB/s), 366KiB/s-372KiB/s (374kB/s-381kB/s), io=24.6MiB (25.7MB), run=16984-17108msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=6516: Sat Feb 24 23:41:08 2024 read: IOPS=105, BW=1681KiB/s (1722kB/s)(5920KiB/3521msec) clat (usec): min=55, max=22305, avg=5066.81, stdev=3243.55 lat (usec): min=56, max=22306, avg=5067.59, stdev=3243.56 clat percentiles (usec): | 1.00th=[ 62], 5.00th=[ 85], 10.00th=[ 106], 20.00th=[ 2966], | 30.00th=[ 3556], 40.00th=[ 4080], 50.00th=[ 4817], 60.00th=[ 5604], | 70.00th=[ 6194], 80.00th=[ 7111], 90.00th=[ 8455], 95.00th=[10552], | 99.00th=[16450], 99.50th=[16581], 99.90th=[22414], 99.95th=[22414], | 99.99th=[22414] bw ( KiB/s): min= 1440, max= 1856, per=25.81%, avg=1681.71, stdev=129.41, samples=7 iops : min= 90, max= 116, avg=105.00, stdev= 8.14, samples=7 write: IOPS=113, BW=1809KiB/s (1852kB/s)(6368KiB/3521msec) clat (usec): min=612, max=16781, avg=4110.47, stdev=3424.92 lat (usec): min=613, max=16782, avg=4111.55, stdev=3424.94 clat percentiles (usec): | 1.00th=[ 660], 5.00th=[ 775], 10.00th=[ 857], 20.00th=[ 1020], | 30.00th=[ 1237], 40.00th=[ 1909], 50.00th=[ 3163], 60.00th=[ 4015], | 70.00th=[ 5669], 80.00th=[ 7111], 90.00th=[ 8848], 95.00th=[10683], | 99.00th=[15139], 99.50th=[16712], 99.90th=[16909], 99.95th=[16909], | 99.99th=[16909] bw ( KiB/s): min= 1309, max= 2304, per=26.46%, avg=1814.43, stdev=332.65, samples=7 iops : min= 81, max= 144, avg=113.29, stdev=21.00, samples=7 lat (usec) : 100=4.17%, 250=1.95%, 500=0.13%, 750=2.34%, 1000=7.42% lat (msec) : 2=11.72%, 4=21.61%, 10=43.88%, 20=6.64%, 50=0.13% cpu : usr=0.14%, sys=25.77%, ctx=1655, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6517: Sat Feb 24 23:41:08 2024 read: IOPS=101, BW=1625KiB/s (1664kB/s)(5744KiB/3535msec) clat (usec): min=59, max=21313, avg=4927.56, stdev=3092.26 lat (usec): min=60, max=21314, avg=4930.10, stdev=3092.26 clat percentiles (usec): | 1.00th=[ 68], 5.00th=[ 82], 10.00th=[ 112], 20.00th=[ 3064], | 30.00th=[ 3523], 40.00th=[ 4047], 50.00th=[ 4752], 60.00th=[ 5276], | 70.00th=[ 6259], 80.00th=[ 7373], 90.00th=[ 8356], 95.00th=[ 9372], | 99.00th=[17171], 99.50th=[20317], 99.90th=[21365], 99.95th=[21365], | 99.99th=[21365] bw ( KiB/s): min= 1440, max= 1920, per=24.90%, avg=1622.86, stdev=171.05, samples=7 iops : min= 90, max= 120, avg=101.43, stdev=10.69, samples=7 write: IOPS=115, BW=1851KiB/s (1896kB/s)(6544KiB/3535msec) clat (usec): min=618, max=19511, avg=4284.38, stdev=3550.84 lat (usec): min=619, max=19513, avg=4285.53, stdev=3550.82 clat percentiles (usec): | 1.00th=[ 660], 5.00th=[ 775], 10.00th=[ 857], 20.00th=[ 1074], | 30.00th=[ 1483], 40.00th=[ 2474], 50.00th=[ 3425], 60.00th=[ 4293], | 70.00th=[ 5407], 80.00th=[ 6783], 90.00th=[ 9765], 95.00th=[11469], | 99.00th=[15664], 99.50th=[17171], 99.90th=[19530], 99.95th=[19530], | 99.99th=[19530] bw ( KiB/s): min= 1664, max= 2112, per=26.93%, avg=1846.86, stdev=150.90, samples=7 iops : min= 104, max= 132, avg=115.43, stdev= 9.43, samples=7 lat (usec) : 100=3.12%, 250=3.12%, 500=0.26%, 750=2.34%, 1000=5.99% lat (msec) : 2=10.68%, 4=23.05%, 10=44.79%, 20=6.38%, 50=0.26% cpu : usr=0.23%, sys=26.06%, ctx=1596, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6519: Sat Feb 24 23:41:08 2024 read: IOPS=106, BW=1702KiB/s (1743kB/s)(6256KiB/3675msec) clat (usec): min=48, max=19622, avg=5174.52, stdev=3188.55 lat (usec): min=49, max=19623, avg=5175.28, stdev=3188.56 clat percentiles (usec): | 1.00th=[ 60], 5.00th=[ 89], 10.00th=[ 126], 20.00th=[ 3032], | 30.00th=[ 3621], 40.00th=[ 4293], 50.00th=[ 5080], 60.00th=[ 5604], | 70.00th=[ 6325], 80.00th=[ 7308], 90.00th=[ 8586], 95.00th=[ 9765], | 99.00th=[17957], 99.50th=[19530], 99.90th=[19530], 99.95th=[19530], | 99.99th=[19530] bw ( KiB/s): min= 1280, max= 2080, per=25.67%, avg=1673.14, stdev=277.57, samples=7 iops : min= 80, max= 130, avg=104.57, stdev=17.35, samples=7 write: IOPS=102, BW=1641KiB/s (1681kB/s)(6032KiB/3675msec) clat (usec): min=614, max=18722, avg=4351.65, stdev=3684.92 lat (usec): min=615, max=18723, avg=4352.77, stdev=3684.90 clat percentiles (usec): | 1.00th=[ 644], 5.00th=[ 758], 10.00th=[ 873], 20.00th=[ 1106], | 30.00th=[ 1385], 40.00th=[ 2409], 50.00th=[ 3294], 60.00th=[ 4293], | 70.00th=[ 5735], 80.00th=[ 7373], 90.00th=[ 9634], 95.00th=[11731], | 99.00th=[16188], 99.50th=[17433], 99.90th=[18744], 99.95th=[18744], | 99.99th=[18744] bw ( KiB/s): min= 1216, max= 1920, per=23.07%, avg=1581.71, stdev=237.22, samples=7 iops : min= 76, max= 120, avg=98.86, stdev=14.83, samples=7 lat (usec) : 50=0.26%, 100=3.12%, 250=2.34%, 750=2.21%, 1000=4.82% lat (msec) : 2=11.59%, 4=22.79%, 10=46.35%, 20=6.51% cpu : usr=0.16%, sys=25.37%, ctx=1658, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6521: Sat Feb 24 23:41:08 2024 read: IOPS=106, BW=1704KiB/s (1745kB/s)(6032KiB/3540msec) clat (usec): min=56, max=26019, avg=5153.69, stdev=3194.25 lat (usec): min=57, max=26020, avg=5154.66, stdev=3194.43 clat percentiles (usec): | 1.00th=[ 69], 5.00th=[ 87], 10.00th=[ 137], 20.00th=[ 3064], | 30.00th=[ 3752], 40.00th=[ 4178], 50.00th=[ 4752], 60.00th=[ 5473], | 70.00th=[ 6259], 80.00th=[ 7308], 90.00th=[ 8717], 95.00th=[10421], | 99.00th=[14615], 99.50th=[22414], 99.90th=[26084], 99.95th=[26084], | 99.99th=[26084] bw ( KiB/s): min= 1504, max= 1920, per=25.88%, avg=1686.86, stdev=164.95, samples=7 iops : min= 94, max= 120, avg=105.43, stdev=10.31, samples=7 write: IOPS=110, BW=1767KiB/s (1810kB/s)(6256KiB/3540msec) clat (usec): min=600, max=23833, avg=4049.76, stdev=3474.26 lat (usec): min=601, max=23834, avg=4051.39, stdev=3474.09 clat percentiles (usec): | 1.00th=[ 709], 5.00th=[ 758], 10.00th=[ 873], 20.00th=[ 1057], | 30.00th=[ 1221], 40.00th=[ 1991], 50.00th=[ 3195], 60.00th=[ 4146], | 70.00th=[ 5211], 80.00th=[ 6587], 90.00th=[ 8979], 95.00th=[10683], | 99.00th=[15139], 99.50th=[18482], 99.90th=[23725], 99.95th=[23725], | 99.99th=[23725] bw ( KiB/s): min= 1472, max= 2656, per=25.73%, avg=1764.57, stdev=409.14, samples=7 iops : min= 92, max= 166, avg=110.29, stdev=25.57, samples=7 lat (usec) : 100=3.65%, 250=1.56%, 500=0.13%, 750=1.95%, 1000=7.55% lat (msec) : 2=11.33%, 4=20.83%, 10=46.48%, 20=6.12%, 50=0.39% cpu : usr=0.40%, sys=25.83%, ctx=1619, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=6518KiB/s (6674kB/s), 1625KiB/s-1704KiB/s (1664kB/s-1745kB/s), io=23.4MiB (24.5MB), run=3521-3675msec WRITE: bw=6857KiB/s (7022kB/s), 1641KiB/s-1851KiB/s (1681kB/s-1896kB/s), io=24.6MiB (25.8MB), run=3521-3675msec rand-rw: (groupid=0, jobs=1): err= 0: pid=6515: Sat Feb 24 23:41:09 2024 read: IOPS=79, BW=1272KiB/s (1303kB/s)(5920KiB/4654msec) clat (usec): min=1868, max=24269, avg=5010.28, stdev=4005.82 lat (usec): min=1869, max=24270, avg=5011.09, stdev=4005.84 clat percentiles (usec): | 1.00th=[ 1926], 5.00th=[ 2073], 10.00th=[ 2245], 20.00th=[ 2409], | 30.00th=[ 2540], 40.00th=[ 2835], 50.00th=[ 3228], 60.00th=[ 3949], | 70.00th=[ 4883], 80.00th=[ 6915], 90.00th=[10945], 95.00th=[14222], | 99.00th=[20055], 99.50th=[22152], 99.90th=[24249], 99.95th=[24249], | 99.99th=[24249] bw ( KiB/s): min= 672, max= 2432, per=24.10%, avg=1240.33, stdev=597.74, samples=9 iops : min= 42, max= 152, avg=77.44, stdev=37.24, samples=9 write: IOPS=85, BW=1368KiB/s (1401kB/s)(6368KiB/4654msec) clat (usec): min=2416, max=29726, avg=7014.09, stdev=4377.40 lat (usec): min=2417, max=29727, avg=7015.57, stdev=4377.47 clat percentiles (usec): | 1.00th=[ 2737], 5.00th=[ 3130], 10.00th=[ 3425], 20.00th=[ 3720], | 30.00th=[ 4015], 40.00th=[ 4490], 50.00th=[ 5080], 60.00th=[ 6325], | 70.00th=[ 8586], 80.00th=[10159], 90.00th=[12780], 95.00th=[15795], | 99.00th=[22414], 99.50th=[27395], 99.90th=[29754], 99.95th=[29754], | 99.99th=[29754] bw ( KiB/s): min= 736, max= 2624, per=24.55%, avg=1329.22, stdev=628.93, samples=9 iops : min= 46, max= 164, avg=83.00, stdev=39.20, samples=9 lat (msec) : 2=0.52%, 4=44.01%, 10=38.28%, 20=16.02%, 50=1.17% cpu : usr=0.11%, sys=11.84%, ctx=905, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6518: Sat Feb 24 23:41:09 2024 read: IOPS=77, BW=1242KiB/s (1271kB/s)(5744KiB/4626msec) clat (usec): min=1901, max=25122, avg=4932.71, stdev=4000.83 lat (usec): min=1902, max=25123, avg=4933.50, stdev=4000.85 clat percentiles (usec): | 1.00th=[ 1975], 5.00th=[ 2114], 10.00th=[ 2180], 20.00th=[ 2311], | 30.00th=[ 2573], 40.00th=[ 2900], 50.00th=[ 3261], 60.00th=[ 3785], | 70.00th=[ 4883], 80.00th=[ 6652], 90.00th=[10683], 95.00th=[13304], | 99.00th=[22414], 99.50th=[24773], 99.90th=[25035], 99.95th=[25035], | 99.99th=[25035] bw ( KiB/s): min= 768, max= 2400, per=23.83%, avg=1226.33, stdev=580.56, samples=9 iops : min= 48, max= 150, avg=76.56, stdev=36.32, samples=9 write: IOPS=88, BW=1415KiB/s (1449kB/s)(6544KiB/4626msec) clat (usec): min=2703, max=30986, avg=6959.23, stdev=4290.89 lat (usec): min=2705, max=30987, avg=6960.82, stdev=4291.18 clat percentiles (usec): | 1.00th=[ 2868], 5.00th=[ 3097], 10.00th=[ 3326], 20.00th=[ 3621], | 30.00th=[ 3982], 40.00th=[ 4359], 50.00th=[ 5145], 60.00th=[ 6652], | 70.00th=[ 8225], 80.00th=[10552], 90.00th=[13042], 95.00th=[14746], | 99.00th=[21103], 99.50th=[23462], 99.90th=[31065], 99.95th=[31065], | 99.99th=[31065] bw ( KiB/s): min= 864, max= 2720, per=25.28%, avg=1368.56, stdev=618.69, samples=9 iops : min= 54, max= 170, avg=85.44, stdev=38.69, samples=9 lat (msec) : 2=0.52%, 4=45.05%, 10=37.11%, 20=16.02%, 50=1.30% cpu : usr=0.22%, sys=11.74%, ctx=904, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6520: Sat Feb 24 23:41:09 2024 read: IOPS=86, BW=1377KiB/s (1410kB/s)(6256KiB/4542msec) clat (usec): min=1889, max=19170, avg=4727.34, stdev=3514.87 lat (usec): min=1890, max=19171, avg=4728.15, stdev=3514.90 clat percentiles (usec): | 1.00th=[ 1942], 5.00th=[ 2114], 10.00th=[ 2180], 20.00th=[ 2376], | 30.00th=[ 2606], 40.00th=[ 2900], 50.00th=[ 3228], 60.00th=[ 3654], | 70.00th=[ 4686], 80.00th=[ 6456], 90.00th=[10290], 95.00th=[13173], | 99.00th=[16909], 99.50th=[17433], 99.90th=[19268], 99.95th=[19268], | 99.99th=[19268] bw ( KiB/s): min= 832, max= 3008, per=26.80%, avg=1379.33, stdev=692.88, samples=9 iops : min= 52, max= 188, avg=86.11, stdev=43.38, samples=9 write: IOPS=83, BW=1328KiB/s (1360kB/s)(6032KiB/4542msec) clat (usec): min=2478, max=31523, avg=7124.14, stdev=4824.99 lat (usec): min=2479, max=31525, avg=7125.55, stdev=4825.06 clat percentiles (usec): | 1.00th=[ 2704], 5.00th=[ 3064], 10.00th=[ 3425], 20.00th=[ 3687], | 30.00th=[ 4015], 40.00th=[ 4424], 50.00th=[ 5080], 60.00th=[ 6456], | 70.00th=[ 8029], 80.00th=[ 9765], 90.00th=[14353], 95.00th=[16909], | 99.00th=[29230], 99.50th=[31065], 99.90th=[31589], 99.95th=[31589], | 99.99th=[31589] bw ( KiB/s): min= 800, max= 2368, per=24.10%, avg=1304.67, stdev=564.66, samples=9 iops : min= 50, max= 148, avg=81.44, stdev=35.36, samples=9 lat (msec) : 2=1.04%, 4=46.09%, 10=38.28%, 20=13.54%, 50=1.04% cpu : usr=0.22%, sys=12.02%, ctx=887, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6522: Sat Feb 24 23:41:09 2024 read: IOPS=82, BW=1316KiB/s (1347kB/s)(6032KiB/4585msec) clat (usec): min=1783, max=20625, avg=5005.82, stdev=3726.37 lat (usec): min=1783, max=20626, avg=5006.63, stdev=3726.37 clat percentiles (usec): | 1.00th=[ 1926], 5.00th=[ 2114], 10.00th=[ 2245], 20.00th=[ 2442], | 30.00th=[ 2671], 40.00th=[ 2999], 50.00th=[ 3589], 60.00th=[ 4113], | 70.00th=[ 4883], 80.00th=[ 7046], 90.00th=[10290], 95.00th=[13566], | 99.00th=[19006], 99.50th=[20579], 99.90th=[20579], 99.95th=[20579], | 99.99th=[20579] bw ( KiB/s): min= 800, max= 2624, per=24.94%, avg=1283.56, stdev=584.04, samples=9 iops : min= 50, max= 164, avg=80.22, stdev=36.50, samples=9 write: IOPS=85, BW=1364KiB/s (1397kB/s)(6256KiB/4585msec) clat (usec): min=2676, max=31141, avg=6879.56, stdev=4380.94 lat (usec): min=2677, max=31142, avg=6880.98, stdev=4380.95 clat percentiles (usec): | 1.00th=[ 2737], 5.00th=[ 3261], 10.00th=[ 3458], 20.00th=[ 3785], | 30.00th=[ 4015], 40.00th=[ 4359], 50.00th=[ 4817], 60.00th=[ 5932], | 70.00th=[ 7308], 80.00th=[10159], 90.00th=[13304], 95.00th=[15139], | 99.00th=[22152], 99.50th=[28181], 99.90th=[31065], 99.95th=[31065], | 99.99th=[31065] bw ( KiB/s): min= 896, max= 2528, per=24.96%, avg=1351.11, stdev=628.25, samples=9 iops : min= 56, max= 158, avg=84.44, stdev=39.27, samples=9 lat (msec) : 2=0.91%, 4=42.45%, 10=40.62%, 20=14.84%, 50=1.17% cpu : usr=0.20%, sys=11.82%, ctx=902, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=5147KiB/s (5270kB/s), 1242KiB/s-1377KiB/s (1271kB/s-1410kB/s), io=23.4MiB (24.5MB), run=4542-4654msec WRITE: bw=5415KiB/s (5545kB/s), 1328KiB/s-1415KiB/s (1360kB/s-1449kB/s), io=24.6MiB (25.8MB), run=4542-4654msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=6531: Sat Feb 24 23:41:11 2024 read: IOPS=12, BW=12.3MiB/s (12.9MB/s)(7168KiB/570msec) clat (usec): min=448, max=65197, avg=37006.54, stdev=26734.88 lat (usec): min=449, max=65198, avg=37007.41, stdev=26734.94 clat percentiles (usec): | 1.00th=[ 449], 5.00th=[ 449], 10.00th=[ 449], 20.00th=[ 490], | 30.00th=[35390], 40.00th=[35390], 50.00th=[47973], 60.00th=[48497], | 70.00th=[48497], 80.00th=[61080], 90.00th=[65274], 95.00th=[65274], | 99.00th=[65274], 99.50th=[65274], 99.90th=[65274], 99.95th=[65274], | 99.99th=[65274] bw ( KiB/s): min=10240, max=10240, per=36.60%, avg=10240.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 write: IOPS=8, BW=8982KiB/s (9198kB/s)(5120KiB/570msec) clat (msec): min=40, max=109, avg=61.83, stdev=30.87 lat (msec): min=40, max=109, avg=61.89, stdev=30.88 clat percentiles (msec): | 1.00th=[ 41], 5.00th=[ 41], 10.00th=[ 41], 20.00th=[ 41], | 30.00th=[ 41], 40.00th=[ 41], 50.00th=[ 42], 60.00th=[ 42], | 70.00th=[ 78], 80.00th=[ 78], 90.00th=[ 110], 95.00th=[ 110], | 99.00th=[ 110], 99.50th=[ 110], 99.90th=[ 110], 99.95th=[ 110], | 99.99th=[ 110] bw ( KiB/s): min= 6144, max= 6144, per=15.69%, avg=6144.00, stdev= 0.00, samples=1 iops : min= 6, max= 6, avg= 6.00, stdev= 0.00, samples=1 lat (usec) : 500=16.67% lat (msec) : 50=50.00%, 100=25.00%, 250=8.33% cpu : usr=0.00%, sys=44.29%, ctx=133, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6532: Sat Feb 24 23:41:11 2024 read: IOPS=2, BW=2798KiB/s (2865kB/s)(2048KiB/732msec) clat (usec): min=41529, max=67829, avg=54679.60, stdev=18596.49 lat (usec): min=41530, max=67830, avg=54680.43, stdev=18596.62 clat percentiles (usec): | 1.00th=[41681], 5.00th=[41681], 10.00th=[41681], 20.00th=[41681], | 30.00th=[41681], 40.00th=[41681], 50.00th=[41681], 60.00th=[67634], | 70.00th=[67634], 80.00th=[67634], 90.00th=[67634], 95.00th=[67634], | 99.00th=[67634], 99.50th=[67634], 99.90th=[67634], 99.95th=[67634], | 99.99th=[67634] bw ( KiB/s): min= 4087, max= 4087, per=14.61%, avg=4087.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 write: IOPS=13, BW=13.7MiB/s (14.3MB/s)(10.0MiB/732msec) clat (msec): min=27, max=108, avg=62.08, stdev=29.15 lat (msec): min=27, max=108, avg=62.14, stdev=29.16 clat percentiles (msec): | 1.00th=[ 28], 5.00th=[ 28], 10.00th=[ 28], 20.00th=[ 36], | 30.00th=[ 38], 40.00th=[ 42], 50.00th=[ 51], 60.00th=[ 59], | 70.00th=[ 91], 80.00th=[ 91], 90.00th=[ 101], 95.00th=[ 109], | 99.00th=[ 109], 99.50th=[ 109], 99.90th=[ 109], 99.95th=[ 109], | 99.99th=[ 109] bw ( KiB/s): min=10219, max=10219, per=26.09%, avg=10219.00, stdev= 0.00, samples=1 iops : min= 9, max= 9, avg= 9.00, stdev= 0.00, samples=1 lat (msec) : 50=41.67%, 100=41.67%, 250=16.67% cpu : usr=0.00%, sys=37.89%, ctx=89, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6533: Sat Feb 24 23:41:11 2024 read: IOPS=10, BW=10.8MiB/s (11.3MB/s)(6144KiB/556msec) clat (usec): min=437, max=63044, avg=26307.30, stdev=29013.62 lat (usec): min=439, max=63045, avg=26308.50, stdev=29013.59 clat percentiles (usec): | 1.00th=[ 437], 5.00th=[ 437], 10.00th=[ 437], 20.00th=[ 449], | 30.00th=[ 449], 40.00th=[ 545], 50.00th=[ 545], 60.00th=[43254], | 70.00th=[50594], 80.00th=[50594], 90.00th=[63177], 95.00th=[63177], | 99.00th=[63177], 99.50th=[63177], 99.90th=[63177], 99.95th=[63177], | 99.99th=[63177] bw ( KiB/s): min=10240, max=10240, per=36.60%, avg=10240.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 write: IOPS=10, BW=10.8MiB/s (11.3MB/s)(6144KiB/556msec) clat (msec): min=31, max=101, avg=66.11, stdev=26.34 lat (msec): min=31, max=101, avg=66.17, stdev=26.35 clat percentiles (msec): | 1.00th=[ 32], 5.00th=[ 32], 10.00th=[ 32], 20.00th=[ 53], | 30.00th=[ 53], 40.00th=[ 54], 50.00th=[ 54], 60.00th=[ 65], | 70.00th=[ 93], 80.00th=[ 93], 90.00th=[ 102], 95.00th=[ 102], | 99.00th=[ 102], 99.50th=[ 102], 99.90th=[ 102], 99.95th=[ 102], | 99.99th=[ 102] bw ( KiB/s): min=12288, max=12288, per=31.37%, avg=12288.00, stdev= 0.00, samples=1 iops : min= 12, max= 12, avg=12.00, stdev= 0.00, samples=1 lat (usec) : 500=16.67%, 750=8.33% lat (msec) : 50=16.67%, 100=50.00%, 250=8.33% cpu : usr=0.18%, sys=36.04%, ctx=99, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6534: Sat Feb 24 23:41:11 2024 read: IOPS=8, BW=8576KiB/s (8782kB/s)(5120KiB/597msec) clat (usec): min=338, max=37964, avg=15235.43, stdev=20291.76 lat (usec): min=339, max=37966, avg=15236.72, stdev=20292.15 clat percentiles (usec): | 1.00th=[ 338], 5.00th=[ 338], 10.00th=[ 338], 20.00th=[ 338], | 30.00th=[ 453], 40.00th=[ 453], 50.00th=[ 465], 60.00th=[ 465], | 70.00th=[36963], 80.00th=[36963], 90.00th=[38011], 95.00th=[38011], | 99.00th=[38011], 99.50th=[38011], 99.90th=[38011], 99.95th=[38011], | 99.99th=[38011] bw ( KiB/s): min= 8175, max= 8175, per=29.22%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 7, max= 7, avg= 7.00, stdev= 0.00, samples=1 write: IOPS=11, BW=11.7MiB/s (12.3MB/s)(7168KiB/597msec) clat (msec): min=39, max=109, avg=74.14, stdev=24.33 lat (msec): min=39, max=109, avg=74.19, stdev=24.32 clat percentiles (msec): | 1.00th=[ 40], 5.00th=[ 40], 10.00th=[ 40], 20.00th=[ 55], | 30.00th=[ 61], 40.00th=[ 61], 50.00th=[ 73], 60.00th=[ 90], | 70.00th=[ 90], 80.00th=[ 92], 90.00th=[ 110], 95.00th=[ 110], | 99.00th=[ 110], 99.50th=[ 110], 99.90th=[ 110], 99.95th=[ 110], | 99.99th=[ 110] bw ( KiB/s): min=10219, max=10219, per=26.09%, avg=10219.00, stdev= 0.00, samples=1 iops : min= 9, max= 9, avg= 9.00, stdev= 0.00, samples=1 lat (usec) : 500=25.00% lat (msec) : 50=25.00%, 100=41.67%, 250=8.33% cpu : usr=0.00%, sys=37.58%, ctx=78, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=27.3MiB/s (28.6MB/s), 2798KiB/s-12.3MiB/s (2865kB/s-12.9MB/s), io=20.0MiB (20.0MB), run=556-732msec WRITE: bw=38.3MiB/s (40.1MB/s), 8982KiB/s-13.7MiB/s (9198kB/s-14.3MB/s), io=28.0MiB (29.4MB), run=556-732msec rand-rw: (groupid=0, jobs=1): err= 0: pid=6535: Sat Feb 24 23:41:11 2024 read: IOPS=8, BW=9005KiB/s (9221kB/s)(7168KiB/796msec) clat (msec): min=28, max=127, avg=72.09, stdev=38.23 lat (msec): min=28, max=127, avg=72.09, stdev=38.22 clat percentiles (msec): | 1.00th=[ 29], 5.00th=[ 29], 10.00th=[ 29], 20.00th=[ 41], | 30.00th=[ 41], 40.00th=[ 41], 50.00th=[ 66], 60.00th=[ 100], | 70.00th=[ 100], 80.00th=[ 105], 90.00th=[ 128], 95.00th=[ 128], | 99.00th=[ 128], 99.50th=[ 128], 99.90th=[ 128], 99.95th=[ 128], | 99.99th=[ 128] bw ( KiB/s): min= 8192, max= 8192, per=33.12%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=6, BW=6432KiB/s (6587kB/s)(5120KiB/796msec) clat (usec): min=26142, max=87381, avg=57972.20, stdev=28631.03 lat (usec): min=26223, max=87444, avg=58041.40, stdev=28627.89 clat percentiles (usec): | 1.00th=[26084], 5.00th=[26084], 10.00th=[26084], 20.00th=[26084], | 30.00th=[28181], 40.00th=[28181], 50.00th=[72877], 60.00th=[72877], | 70.00th=[74974], 80.00th=[74974], 90.00th=[87557], 95.00th=[87557], | 99.00th=[87557], 99.50th=[87557], 99.90th=[87557], 99.95th=[87557], | 99.99th=[87557] bw ( KiB/s): min= 2048, max= 2048, per=5.91%, avg=2048.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 50=41.67%, 100=41.67%, 250=16.67% cpu : usr=0.00%, sys=16.73%, ctx=76, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6536: Sat Feb 24 23:41:11 2024 read: IOPS=2, BW=2473KiB/s (2533kB/s)(2048KiB/828msec) clat (msec): min=64, max=107, avg=86.27, stdev=30.37 lat (msec): min=64, max=107, avg=86.27, stdev=30.37 clat percentiles (msec): | 1.00th=[ 65], 5.00th=[ 65], 10.00th=[ 65], 20.00th=[ 65], | 30.00th=[ 65], 40.00th=[ 65], 50.00th=[ 65], 60.00th=[ 108], | 70.00th=[ 108], 80.00th=[ 108], 90.00th=[ 108], 95.00th=[ 108], | 99.00th=[ 108], 99.50th=[ 108], 99.90th=[ 108], 99.95th=[ 108], | 99.99th=[ 108] bw ( KiB/s): min= 2048, max= 2048, per=8.28%, avg=2048.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=12, BW=12.1MiB/s (12.7MB/s)(10.0MiB/828msec) clat (usec): min=32237, max=92895, avg=65197.98, stdev=17564.65 lat (usec): min=32306, max=92922, avg=65252.87, stdev=17561.51 clat percentiles (usec): | 1.00th=[32113], 5.00th=[32113], 10.00th=[32113], 20.00th=[42206], | 30.00th=[57934], 40.00th=[63701], 50.00th=[66847], 60.00th=[68682], | 70.00th=[74974], 80.00th=[74974], 90.00th=[77071], 95.00th=[92799], | 99.00th=[92799], 99.50th=[92799], 99.90th=[92799], 99.95th=[92799], | 99.99th=[92799] bw ( KiB/s): min=10240, max=10240, per=29.57%, avg=10240.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 lat (msec) : 50=16.67%, 100=75.00%, 250=8.33% cpu : usr=0.24%, sys=19.83%, ctx=84, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6537: Sat Feb 24 23:41:11 2024 read: IOPS=7, BW=8010KiB/s (8203kB/s)(6144KiB/767msec) clat (usec): min=22319, max=55859, avg=44663.75, stdev=13399.39 lat (usec): min=22319, max=55861, avg=44664.86, stdev=13399.65 clat percentiles (usec): | 1.00th=[22414], 5.00th=[22414], 10.00th=[22414], 20.00th=[33817], | 30.00th=[33817], 40.00th=[51643], 50.00th=[51643], 60.00th=[51643], | 70.00th=[52691], 80.00th=[52691], 90.00th=[55837], 95.00th=[55837], | 99.00th=[55837], 99.50th=[55837], 99.90th=[55837], 99.95th=[55837], | 99.99th=[55837] bw ( KiB/s): min= 6144, max= 6144, per=24.84%, avg=6144.00, stdev= 0.00, samples=1 iops : min= 6, max= 6, avg= 6.00, stdev= 0.00, samples=1 write: IOPS=7, BW=8010KiB/s (8203kB/s)(6144KiB/767msec) clat (msec): min=39, max=126, avg=82.33, stdev=35.79 lat (msec): min=39, max=126, avg=82.40, stdev=35.79 clat percentiles (msec): | 1.00th=[ 40], 5.00th=[ 40], 10.00th=[ 40], 20.00th=[ 56], | 30.00th=[ 56], 40.00th=[ 62], 50.00th=[ 62], 60.00th=[ 91], | 70.00th=[ 121], 80.00th=[ 121], 90.00th=[ 127], 95.00th=[ 127], | 99.00th=[ 127], 99.50th=[ 127], 99.90th=[ 127], 99.95th=[ 127], | 99.99th=[ 127] bw ( KiB/s): min= 8192, max= 8192, per=23.66%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 lat (msec) : 50=25.00%, 100=58.33%, 250=16.67% cpu : usr=0.00%, sys=17.49%, ctx=56, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6538: Sat Feb 24 23:41:11 2024 read: IOPS=6, BW=6344KiB/s (6497kB/s)(5120KiB/807msec) clat (usec): min=26505, max=94278, avg=49361.07, stdev=28156.07 lat (usec): min=26507, max=94279, avg=49362.19, stdev=28155.85 clat percentiles (usec): | 1.00th=[26608], 5.00th=[26608], 10.00th=[26608], 20.00th=[26608], | 30.00th=[27132], 40.00th=[27132], 50.00th=[41157], 60.00th=[41157], | 70.00th=[57934], 80.00th=[57934], 90.00th=[93848], 95.00th=[93848], | 99.00th=[93848], 99.50th=[93848], 99.90th=[93848], 99.95th=[93848], | 99.99th=[93848] bw ( KiB/s): min= 4055, max= 4055, per=16.39%, avg=4055.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 write: IOPS=8, BW=8882KiB/s (9095kB/s)(7168KiB/807msec) clat (msec): min=25, max=155, avg=79.03, stdev=48.76 lat (msec): min=25, max=156, avg=79.09, stdev=48.76 clat percentiles (msec): | 1.00th=[ 26], 5.00th=[ 26], 10.00th=[ 26], 20.00th=[ 30], | 30.00th=[ 48], 40.00th=[ 48], 50.00th=[ 73], 60.00th=[ 103], | 70.00th=[ 103], 80.00th=[ 118], 90.00th=[ 157], 95.00th=[ 157], | 99.00th=[ 157], 99.50th=[ 157], 99.90th=[ 157], 99.95th=[ 157], | 99.99th=[ 157] bw ( KiB/s): min= 8110, max= 8110, per=23.42%, avg=8110.00, stdev= 0.00, samples=1 iops : min= 7, max= 7, avg= 7.00, stdev= 0.00, samples=1 lat (msec) : 50=50.00%, 100=25.00%, 250=25.00% cpu : usr=0.00%, sys=17.62%, ctx=76, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=24.2MiB/s (25.3MB/s), 2473KiB/s-9005KiB/s (2533kB/s-9221kB/s), io=20.0MiB (20.0MB), run=767-828msec WRITE: bw=33.8MiB/s (35.5MB/s), 6432KiB/s-12.1MiB/s (6587kB/s-12.7MB/s), io=28.0MiB (29.4MB), run=767-828msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=6544: Sat Feb 24 23:41:13 2024 read: IOPS=5, BW=23.5MiB/s (24.6MB/s)(12.0MiB/511msec) clat (msec): min=45, max=343, avg=169.70, stdev=155.19 lat (msec): min=45, max=343, avg=169.70, stdev=155.19 clat percentiles (msec): | 1.00th=[ 46], 5.00th=[ 46], 10.00th=[ 46], 20.00th=[ 46], | 30.00th=[ 46], 40.00th=[ 120], 50.00th=[ 120], 60.00th=[ 120], | 70.00th=[ 342], 80.00th=[ 342], 90.00th=[ 342], 95.00th=[ 342], | 99.00th=[ 342], 99.50th=[ 342], 99.90th=[ 342], 99.95th=[ 342], | 99.99th=[ 342] bw ( KiB/s): min=16351, max=16351, per=81.94%, avg=16351.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 lat (msec) : 50=33.33%, 250=33.33%, 500=33.33% cpu : usr=0.00%, sys=51.18%, ctx=232, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6545: Sat Feb 24 23:41:13 2024 read: IOPS=1, BW=4989KiB/s (5109kB/s)(4096KiB/821msec) clat (nsec): min=165621k, max=165621k, avg=165620802.00, stdev= 0.00 lat (nsec): min=165623k, max=165623k, avg=165623070.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 165], 5.00th=[ 165], 10.00th=[ 165], 20.00th=[ 165], | 30.00th=[ 165], 40.00th=[ 165], 50.00th=[ 165], 60.00th=[ 165], | 70.00th=[ 165], 80.00th=[ 165], 90.00th=[ 165], 95.00th=[ 165], | 99.00th=[ 165], 99.50th=[ 165], 99.90th=[ 165], 99.95th=[ 165], | 99.99th=[ 165] bw ( KiB/s): min= 8175, max= 8175, per=40.97%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=2, BW=9978KiB/s (10.2MB/s)(8192KiB/821msec) clat (msec): min=244, max=402, avg=323.19, stdev=111.79 lat (msec): min=244, max=402, avg=323.51, stdev=111.80 clat percentiles (msec): | 1.00th=[ 245], 5.00th=[ 245], 10.00th=[ 245], 20.00th=[ 245], | 30.00th=[ 245], 40.00th=[ 245], 50.00th=[ 245], 60.00th=[ 401], | 70.00th=[ 401], 80.00th=[ 401], 90.00th=[ 401], 95.00th=[ 401], | 99.00th=[ 401], 99.50th=[ 401], 99.90th=[ 401], 99.95th=[ 401], | 99.99th=[ 401] lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.00%, sys=31.95%, ctx=181, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6547: Sat Feb 24 23:41:13 2024 write: IOPS=4, BW=16.2MiB/s (17.0MB/s)(12.0MiB/740msec) clat (msec): min=184, max=285, avg=246.16, stdev=53.81 lat (msec): min=185, max=285, avg=246.40, stdev=53.81 clat percentiles (msec): | 1.00th=[ 186], 5.00th=[ 186], 10.00th=[ 186], 20.00th=[ 186], | 30.00th=[ 186], 40.00th=[ 271], 50.00th=[ 271], 60.00th=[ 271], | 70.00th=[ 284], 80.00th=[ 284], 90.00th=[ 284], 95.00th=[ 284], | 99.00th=[ 284], 99.50th=[ 284], 99.90th=[ 284], 99.95th=[ 284], | 99.99th=[ 284] bw ( KiB/s): min=16189, max=16189, per=40.56%, avg=16189.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=66.67% cpu : usr=0.00%, sys=31.80%, ctx=50, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6548: Sat Feb 24 23:41:13 2024 write: IOPS=4, BW=18.9MiB/s (19.8MB/s)(12.0MiB/634msec) clat (msec): min=78, max=389, avg=210.68, stdev=160.61 lat (msec): min=79, max=389, avg=210.99, stdev=160.60 clat percentiles (msec): | 1.00th=[ 80], 5.00th=[ 80], 10.00th=[ 80], 20.00th=[ 80], | 30.00th=[ 80], 40.00th=[ 165], 50.00th=[ 165], 60.00th=[ 165], | 70.00th=[ 388], 80.00th=[ 388], 90.00th=[ 388], 95.00th=[ 388], | 99.00th=[ 388], 99.50th=[ 388], 99.90th=[ 388], 99.95th=[ 388], | 99.99th=[ 388] bw ( KiB/s): min=16253, max=16253, per=40.72%, avg=16253.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 250=33.33%, 500=33.33% cpu : usr=0.00%, sys=40.60%, ctx=41, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=19.5MiB/s (20.4MB/s), 4989KiB/s-23.5MiB/s (5109kB/s-24.6MB/s), io=16.0MiB (16.8MB), run=511-821msec WRITE: bw=38.0MiB/s (40.9MB/s), 9978KiB/s-18.9MiB/s (10.2MB/s-19.8MB/s), io=32.0MiB (33.6MB), run=634-821msec rand-rw: (groupid=0, jobs=1): err= 0: pid=6549: Sat Feb 24 23:41:13 2024 read: IOPS=3, BW=14.6MiB/s (15.3MB/s)(12.0MiB/824msec) clat (msec): min=179, max=410, avg=274.18, stdev=120.93 lat (msec): min=179, max=410, avg=274.18, stdev=120.93 clat percentiles (msec): | 1.00th=[ 180], 5.00th=[ 180], 10.00th=[ 180], 20.00th=[ 180], | 30.00th=[ 180], 40.00th=[ 232], 50.00th=[ 232], 60.00th=[ 232], | 70.00th=[ 409], 80.00th=[ 409], 90.00th=[ 409], 95.00th=[ 409], | 99.00th=[ 409], 99.50th=[ 409], 99.90th=[ 409], 99.95th=[ 409], | 99.99th=[ 409] bw ( KiB/s): min= 8192, max= 8192, per=52.55%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.00%, sys=14.09%, ctx=38, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6550: Sat Feb 24 23:41:13 2024 read: IOPS=0, BW=3897KiB/s (3991kB/s)(4096KiB/1051msec) clat (nsec): min=434060k, max=434060k, avg=434059635.00, stdev= 0.00 lat (nsec): min=434061k, max=434061k, avg=434061065.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 435], 5.00th=[ 435], 10.00th=[ 435], 20.00th=[ 435], | 30.00th=[ 435], 40.00th=[ 435], 50.00th=[ 435], 60.00th=[ 435], | 70.00th=[ 435], 80.00th=[ 435], 90.00th=[ 435], 95.00th=[ 435], | 99.00th=[ 435], 99.50th=[ 435], 99.90th=[ 435], 99.95th=[ 435], | 99.99th=[ 435] bw ( KiB/s): min= 8192, max= 8192, per=52.55%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=1, BW=7794KiB/s (7982kB/s)(8192KiB/1051msec) clat (msec): min=133, max=481, avg=307.30, stdev=246.05 lat (msec): min=133, max=481, avg=307.57, stdev=246.04 clat percentiles (msec): | 1.00th=[ 134], 5.00th=[ 134], 10.00th=[ 134], 20.00th=[ 134], | 30.00th=[ 134], 40.00th=[ 134], 50.00th=[ 134], 60.00th=[ 481], | 70.00th=[ 481], 80.00th=[ 481], 90.00th=[ 481], 95.00th=[ 481], | 99.00th=[ 481], 99.50th=[ 481], 99.90th=[ 481], 99.95th=[ 481], | 99.99th=[ 481] bw ( KiB/s): min= 7816, max= 7816, per=25.07%, avg=7816.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=66.67% cpu : usr=0.10%, sys=12.19%, ctx=249, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6551: Sat Feb 24 23:41:13 2024 write: IOPS=3, BW=13.5MiB/s (14.1MB/s)(12.0MiB/890msec) clat (msec): min=206, max=342, avg=295.47, stdev=77.15 lat (msec): min=206, max=342, avg=295.75, stdev=77.15 clat percentiles (msec): | 1.00th=[ 207], 5.00th=[ 207], 10.00th=[ 207], 20.00th=[ 207], | 30.00th=[ 207], 40.00th=[ 338], 50.00th=[ 338], 60.00th=[ 338], | 70.00th=[ 342], 80.00th=[ 342], 90.00th=[ 342], 95.00th=[ 342], | 99.00th=[ 342], 99.50th=[ 342], 99.90th=[ 342], 99.95th=[ 342], | 99.99th=[ 342] bw ( KiB/s): min= 8175, max= 8175, per=26.22%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=66.67% cpu : usr=0.11%, sys=16.99%, ctx=37, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6552: Sat Feb 24 23:41:13 2024 write: IOPS=4, BW=16.9MiB/s (17.7MB/s)(12.0MiB/712msec) clat (msec): min=156, max=304, avg=233.20, stdev=74.12 lat (msec): min=156, max=304, avg=233.44, stdev=74.15 clat percentiles (msec): | 1.00th=[ 157], 5.00th=[ 157], 10.00th=[ 157], 20.00th=[ 157], | 30.00th=[ 157], 40.00th=[ 239], 50.00th=[ 239], 60.00th=[ 239], | 70.00th=[ 305], 80.00th=[ 305], 90.00th=[ 305], 95.00th=[ 305], | 99.00th=[ 305], 99.50th=[ 305], 99.90th=[ 305], 99.95th=[ 305], | 99.99th=[ 305] bw ( KiB/s): min= 8192, max= 8192, per=26.28%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.14%, sys=26.44%, ctx=57, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=15.2MiB/s (15.0MB/s), 3897KiB/s-14.6MiB/s (3991kB/s-15.3MB/s), io=16.0MiB (16.8MB), run=824-1051msec WRITE: bw=30.4MiB/s (31.9MB/s), 7794KiB/s-16.9MiB/s (7982kB/s-17.7MB/s), io=32.0MiB (33.6MB), run=712-1051msec