== sanity test 398b: DIO and buffer IO race ============== 13:24:07 (1722792247) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.21333 s, 41.5 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=5365: Sun Aug 4 13:24:22 2024 read: IOPS=122, BW=491KiB/s (503kB/s)(5992KiB/12204msec) clat (usec): min=52, max=208296, avg=4673.52, stdev=5845.85 lat (usec): min=52, max=208297, avg=4674.24, stdev=5845.84 clat percentiles (usec): | 1.00th=[ 62], 5.00th=[ 1795], 10.00th=[ 2180], 20.00th=[ 2638], | 30.00th=[ 3032], 40.00th=[ 3523], 50.00th=[ 4146], 60.00th=[ 4752], | 70.00th=[ 5407], 80.00th=[ 6194], 90.00th=[ 7439], 95.00th=[ 8979], | 99.00th=[ 14222], 99.50th=[ 16188], 99.90th=[ 20579], 99.95th=[208667], | 99.99th=[208667] bw ( KiB/s): min= 199, max= 672, per=25.18%, avg=485.42, stdev=81.75, samples=24 iops : min= 49, max= 168, avg=121.25, stdev=20.53, samples=24 write: IOPS=128, BW=516KiB/s (528kB/s)(6296KiB/12204msec) clat (usec): min=432, max=21630, avg=3251.76, stdev=2905.55 lat (usec): min=432, max=21631, avg=3253.13, stdev=2905.34 clat percentiles (usec): | 1.00th=[ 510], 5.00th=[ 537], 10.00th=[ 570], 20.00th=[ 627], | 30.00th=[ 807], 40.00th=[ 1827], 50.00th=[ 2606], 60.00th=[ 3261], | 70.00th=[ 4178], 80.00th=[ 5342], 90.00th=[ 7373], 95.00th=[ 8979], | 99.00th=[12256], 99.50th=[13698], 99.90th=[17695], 99.95th=[21627], | 99.99th=[21627] bw ( KiB/s): min= 231, max= 680, per=25.26%, avg=509.75, stdev=88.72, samples=24 iops : min= 57, max= 170, avg=127.33, stdev=22.27, samples=24 lat (usec) : 100=1.89%, 250=0.29%, 500=0.23%, 750=13.96%, 1000=3.52% lat (msec) : 2=4.07%, 4=34.11%, 10=38.48%, 20=3.35%, 50=0.07% lat (msec) : 250=0.03% cpu : usr=0.24%, sys=23.09%, ctx=5908, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5366: Sun Aug 4 13:24:22 2024 read: IOPS=121, BW=486KiB/s (498kB/s)(5940KiB/12217msec) clat (usec): min=54, max=161943, avg=4493.45, stdev=4702.42 lat (usec): min=54, max=161945, avg=4494.30, stdev=4702.59 clat percentiles (usec): | 1.00th=[ 61], 5.00th=[ 1827], 10.00th=[ 2245], 20.00th=[ 2606], | 30.00th=[ 2999], 40.00th=[ 3458], 50.00th=[ 3949], 60.00th=[ 4555], | 70.00th=[ 5211], 80.00th=[ 5932], 90.00th=[ 7111], 95.00th=[ 8356], | 99.00th=[ 13435], 99.50th=[ 14615], 99.90th=[ 16909], 99.95th=[162530], | 99.99th=[162530] bw ( KiB/s): min= 136, max= 680, per=25.07%, avg=483.42, stdev=95.65, samples=24 iops : min= 34, max= 170, avg=120.75, stdev=23.89, samples=24 write: IOPS=129, BW=520KiB/s (532kB/s)(6348KiB/12217msec) clat (usec): min=447, max=41309, avg=3437.32, stdev=3122.90 lat (usec): min=447, max=41310, avg=3438.31, stdev=3122.77 clat percentiles (usec): | 1.00th=[ 510], 5.00th=[ 545], 10.00th=[ 570], 20.00th=[ 635], | 30.00th=[ 865], 40.00th=[ 2057], 50.00th=[ 2737], 60.00th=[ 3326], | 70.00th=[ 4359], 80.00th=[ 5800], 90.00th=[ 7832], 95.00th=[ 9241], | 99.00th=[12518], 99.50th=[13829], 99.90th=[19792], 99.95th=[41157], | 99.99th=[41157] bw ( KiB/s): min= 216, max= 702, per=25.33%, avg=511.12, stdev=94.74, samples=24 iops : min= 54, max= 175, avg=127.67, stdev=23.70, samples=24 lat (usec) : 100=1.50%, 250=0.42%, 500=0.39%, 750=13.48%, 1000=3.09% lat (msec) : 2=4.43%, 4=35.87%, 10=37.70%, 20=3.06%, 50=0.03% lat (msec) : 250=0.03% cpu : usr=0.25%, sys=23.05%, ctx=5844, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5367: Sun Aug 4 13:24:22 2024 read: IOPS=122, BW=490KiB/s (502kB/s)(6028KiB/12295msec) clat (usec): min=51, max=20617, avg=4485.81, stdev=2448.78 lat (usec): min=52, max=20617, avg=4486.89, stdev=2448.85 clat percentiles (usec): | 1.00th=[ 61], 5.00th=[ 1909], 10.00th=[ 2212], 20.00th=[ 2638], | 30.00th=[ 2999], 40.00th=[ 3490], 50.00th=[ 4080], 60.00th=[ 4686], | 70.00th=[ 5342], 80.00th=[ 6128], 90.00th=[ 7308], 95.00th=[ 8586], | 99.00th=[14222], 99.50th=[15401], 99.90th=[18220], 99.95th=[20579], | 99.99th=[20579] bw ( KiB/s): min= 104, max= 664, per=25.16%, avg=485.00, stdev=102.34, samples=24 iops : min= 26, max= 166, avg=121.25, stdev=25.58, samples=24 write: IOPS=127, BW=509KiB/s (521kB/s)(6260KiB/12295msec) clat (usec): min=415, max=173613, avg=3481.35, stdev=5317.31 lat (usec): min=416, max=173614, avg=3482.84, stdev=5317.47 clat percentiles (usec): | 1.00th=[ 502], 5.00th=[ 545], 10.00th=[ 562], 20.00th=[ 644], | 30.00th=[ 824], 40.00th=[ 2073], 50.00th=[ 2704], 60.00th=[ 3359], | 70.00th=[ 4228], 80.00th=[ 5538], 90.00th=[ 7570], 95.00th=[ 9110], | 99.00th=[ 12649], 99.50th=[ 16188], 99.90th=[ 38536], 99.95th=[173016], | 99.99th=[173016] bw ( KiB/s): min= 120, max= 656, per=24.84%, avg=501.33, stdev=107.43, samples=24 iops : min= 30, max= 164, avg=125.33, stdev=26.86, samples=24 lat (usec) : 100=1.76%, 250=0.26%, 500=0.52%, 750=13.41%, 1000=2.80% lat (msec) : 2=4.17%, 4=35.38%, 10=38.74%, 20=2.80%, 50=0.13% lat (msec) : 250=0.03% cpu : usr=0.19%, sys=22.81%, ctx=5959, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5368: Sun Aug 4 13:24:22 2024 read: IOPS=121, BW=486KiB/s (498kB/s)(6052KiB/12453msec) clat (usec): min=52, max=22498, avg=4527.93, stdev=2444.39 lat (usec): min=53, max=22498, avg=4528.76, stdev=2444.16 clat percentiles (usec): | 1.00th=[ 63], 5.00th=[ 1926], 10.00th=[ 2212], 20.00th=[ 2606], | 30.00th=[ 3032], 40.00th=[ 3556], 50.00th=[ 4146], 60.00th=[ 4817], | 70.00th=[ 5473], 80.00th=[ 6194], 90.00th=[ 7242], 95.00th=[ 8586], | 99.00th=[12387], 99.50th=[14746], 99.90th=[21627], 99.95th=[22414], | 99.99th=[22414] bw ( KiB/s): min= 207, max= 600, per=24.74%, avg=477.00, stdev=79.14, samples=24 iops : min= 51, max= 150, avg=119.08, stdev=19.84, samples=24 write: IOPS=125, BW=501KiB/s (513kB/s)(6236KiB/12453msec) clat (usec): min=461, max=160927, avg=3531.00, stdev=5044.29 lat (usec): min=462, max=160929, avg=3531.92, stdev=5044.48 clat percentiles (usec): | 1.00th=[ 510], 5.00th=[ 545], 10.00th=[ 570], 20.00th=[ 652], | 30.00th=[ 906], 40.00th=[ 2114], 50.00th=[ 2769], 60.00th=[ 3392], | 70.00th=[ 4424], 80.00th=[ 5735], 90.00th=[ 7373], 95.00th=[ 9241], | 99.00th=[ 12649], 99.50th=[ 13960], 99.90th=[ 42730], 99.95th=[160433], | 99.99th=[160433] bw ( KiB/s): min= 215, max= 816, per=24.48%, avg=494.04, stdev=97.77, samples=24 iops : min= 53, max= 204, avg=123.33, stdev=24.57, samples=24 lat (usec) : 100=1.40%, 250=0.29%, 500=0.33%, 750=12.92%, 1000=2.70% lat (msec) : 2=4.65%, 4=34.70%, 10=39.91%, 20=2.93%, 50=0.13% lat (msec) : 250=0.03% cpu : usr=0.16%, sys=22.80%, ctx=5973, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1928KiB/s (1974kB/s), 486KiB/s-491KiB/s (498kB/s-503kB/s), io=23.4MiB (24.6MB), run=12204-12453msec WRITE: bw=2019KiB/s (2067kB/s), 501KiB/s-520KiB/s (513kB/s-532kB/s), io=24.6MiB (25.7MB), run=12204-12453msec rand-rw: (groupid=0, jobs=1): err= 0: pid=5361: Sun Aug 4 13:24:25 2024 read: IOPS=92, BW=372KiB/s (381kB/s)(5992KiB/16114msec) clat (usec): min=1572, max=209309, avg=4406.28, stdev=6108.58 lat (usec): min=1572, max=209311, avg=4407.09, stdev=6108.57 clat percentiles (usec): | 1.00th=[ 1795], 5.00th=[ 1942], 10.00th=[ 2073], 20.00th=[ 2278], | 30.00th=[ 2474], 40.00th=[ 2704], 50.00th=[ 2966], 60.00th=[ 3425], | 70.00th=[ 4228], 80.00th=[ 5932], 90.00th=[ 8979], 95.00th=[ 10945], | 99.00th=[ 16057], 99.50th=[ 19268], 99.90th=[ 23200], 99.95th=[208667], | 99.99th=[208667] bw ( KiB/s): min= 96, max= 704, per=25.06%, avg=370.41, stdev=142.67, samples=32 iops : min= 24, max= 176, avg=92.53, stdev=35.70, samples=32 write: IOPS=97, BW=391KiB/s (400kB/s)(6296KiB/16114msec) clat (usec): min=2185, max=26573, avg=5989.35, stdev=3408.78 lat (usec): min=2185, max=26574, avg=5990.21, stdev=3408.71 clat percentiles (usec): | 1.00th=[ 2573], 5.00th=[ 2933], 10.00th=[ 3097], 20.00th=[ 3490], | 30.00th=[ 3818], 40.00th=[ 4178], 50.00th=[ 4621], 60.00th=[ 5473], | 70.00th=[ 6849], 80.00th=[ 8356], 90.00th=[10552], 95.00th=[12518], | 99.00th=[18482], 99.50th=[21627], 99.90th=[25560], 99.95th=[26608], | 99.99th=[26608] bw ( KiB/s): min= 96, max= 704, per=25.16%, avg=389.16, stdev=147.73, samples=32 iops : min= 24, max= 176, avg=97.22, stdev=36.98, samples=32 lat (msec) : 2=3.42%, 4=47.88%, 10=39.29%, 20=8.92%, 50=0.46% lat (msec) : 250=0.03% cpu : usr=0.19%, sys=12.62%, ctx=3669, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5362: Sun Aug 4 13:24:25 2024 read: IOPS=92, BW=368KiB/s (377kB/s)(5940KiB/16138msec) clat (usec): min=1626, max=173597, avg=4376.82, stdev=5326.11 lat (usec): min=1627, max=173600, avg=4377.59, stdev=5326.18 clat percentiles (usec): | 1.00th=[ 1778], 5.00th=[ 1975], 10.00th=[ 2073], 20.00th=[ 2278], | 30.00th=[ 2474], 40.00th=[ 2737], 50.00th=[ 2999], 60.00th=[ 3392], | 70.00th=[ 4113], 80.00th=[ 5800], 90.00th=[ 8979], 95.00th=[ 10814], | 99.00th=[ 15533], 99.50th=[ 17171], 99.90th=[ 20317], 99.95th=[173016], | 99.99th=[173016] bw ( KiB/s): min= 72, max= 704, per=24.87%, avg=367.53, stdev=146.91, samples=32 iops : min= 18, max= 176, avg=91.78, stdev=36.69, samples=32 write: IOPS=98, BW=393KiB/s (403kB/s)(6348KiB/16138msec) clat (usec): min=2192, max=40753, avg=6019.41, stdev=3608.79 lat (usec): min=2193, max=40754, avg=6020.26, stdev=3608.73 clat percentiles (usec): | 1.00th=[ 2573], 5.00th=[ 2933], 10.00th=[ 3163], 20.00th=[ 3523], | 30.00th=[ 3818], 40.00th=[ 4178], 50.00th=[ 4621], 60.00th=[ 5407], | 70.00th=[ 6718], 80.00th=[ 8225], 90.00th=[10552], 95.00th=[12911], | 99.00th=[18482], 99.50th=[23987], 99.90th=[30540], 99.95th=[40633], | 99.99th=[40633] bw ( KiB/s): min= 112, max= 688, per=25.28%, avg=391.03, stdev=140.09, samples=32 iops : min= 28, max= 172, avg=97.66, stdev=34.98, samples=32 lat (msec) : 2=3.06%, 4=48.11%, 10=39.16%, 20=9.21%, 50=0.42% lat (msec) : 250=0.03% cpu : usr=0.22%, sys=12.78%, ctx=3645, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5363: Sun Aug 4 13:24:25 2024 read: IOPS=93, BW=374KiB/s (383kB/s)(6028KiB/16112msec) clat (usec): min=1559, max=23453, avg=4401.45, stdev=3197.39 lat (usec): min=1559, max=23454, avg=4402.14, stdev=3197.36 clat percentiles (usec): | 1.00th=[ 1795], 5.00th=[ 1942], 10.00th=[ 2073], 20.00th=[ 2278], | 30.00th=[ 2507], 40.00th=[ 2737], 50.00th=[ 3064], 60.00th=[ 3458], | 70.00th=[ 4359], 80.00th=[ 6259], 90.00th=[ 9241], 95.00th=[11338], | 99.00th=[16057], 99.50th=[17433], 99.90th=[21627], 99.95th=[23462], | 99.99th=[23462] bw ( KiB/s): min= 104, max= 694, per=25.24%, avg=373.06, stdev=143.17, samples=32 iops : min= 26, max= 173, avg=93.19, stdev=35.75, samples=32 write: IOPS=97, BW=389KiB/s (398kB/s)(6260KiB/16112msec) clat (msec): min=2, max=177, avg= 6.00, stdev= 5.52 lat (msec): min=2, max=177, avg= 6.00, stdev= 5.52 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 4], 20.00th=[ 4], | 30.00th=[ 4], 40.00th=[ 5], 50.00th=[ 5], 60.00th=[ 6], | 70.00th=[ 7], 80.00th=[ 8], 90.00th=[ 11], 95.00th=[ 13], | 99.00th=[ 20], 99.50th=[ 23], 99.90th=[ 31], 99.95th=[ 178], | 99.99th=[ 178] bw ( KiB/s): min= 120, max= 744, per=24.96%, avg=386.06, stdev=144.58, samples=32 iops : min= 30, max= 186, avg=96.44, stdev=36.11, samples=32 lat (msec) : 2=3.61%, 4=47.92%, 10=39.03%, 20=8.95%, 50=0.46% lat (msec) : 250=0.03% cpu : usr=0.20%, sys=12.92%, ctx=3619, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5364: Sun Aug 4 13:24:25 2024 read: IOPS=93, BW=373KiB/s (382kB/s)(6052KiB/16241msec) clat (usec): min=1652, max=22945, avg=4327.51, stdev=3153.82 lat (usec): min=1653, max=22946, avg=4328.27, stdev=3153.82 clat percentiles (usec): | 1.00th=[ 1811], 5.00th=[ 1958], 10.00th=[ 2089], 20.00th=[ 2311], | 30.00th=[ 2507], 40.00th=[ 2737], 50.00th=[ 2999], 60.00th=[ 3392], | 70.00th=[ 4178], 80.00th=[ 5932], 90.00th=[ 8979], 95.00th=[11207], | 99.00th=[16057], 99.50th=[17957], 99.90th=[20317], 99.95th=[22938], | 99.99th=[22938] bw ( KiB/s): min= 104, max= 728, per=25.05%, avg=370.22, stdev=139.98, samples=32 iops : min= 26, max= 182, avg=92.53, stdev=35.00, samples=32 write: IOPS=95, BW=384KiB/s (393kB/s)(6236KiB/16241msec) clat (msec): min=2, max=177, avg= 6.16, stdev= 5.63 lat (msec): min=2, max=177, avg= 6.16, stdev= 5.63 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 4], 20.00th=[ 4], | 30.00th=[ 4], 40.00th=[ 5], 50.00th=[ 5], 60.00th=[ 6], | 70.00th=[ 7], 80.00th=[ 9], 90.00th=[ 11], 95.00th=[ 13], | 99.00th=[ 20], 99.50th=[ 24], 99.90th=[ 37], 99.95th=[ 178], | 99.99th=[ 178] bw ( KiB/s): min= 120, max= 704, per=24.66%, avg=381.47, stdev=142.25, samples=32 iops : min= 30, max= 176, avg=95.34, stdev=35.58, samples=32 lat (msec) : 2=3.26%, 4=48.76%, 10=38.15%, 20=9.31%, 50=0.49% lat (msec) : 250=0.03% cpu : usr=0.23%, sys=12.59%, ctx=3687, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1478KiB/s (1514kB/s), 368KiB/s-374KiB/s (377kB/s-383kB/s), io=23.4MiB (24.6MB), run=16112-16241msec WRITE: bw=1548KiB/s (1585kB/s), 384KiB/s-393KiB/s (393kB/s-403kB/s), io=24.6MiB (25.7MB), run=16112-16241msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=5397: Sun Aug 4 13:24:33 2024 read: IOPS=55, BW=882KiB/s (903kB/s)(5920KiB/6711msec) clat (usec): min=61, max=88361, avg=12010.45, stdev=6208.04 lat (usec): min=61, max=88362, avg=12011.23, stdev=6208.15 clat percentiles (usec): | 1.00th=[ 78], 5.00th=[ 7439], 10.00th=[ 8225], 20.00th=[ 9241], | 30.00th=[ 9896], 40.00th=[10552], 50.00th=[10945], 60.00th=[11600], | 70.00th=[12649], 80.00th=[13566], 90.00th=[15926], 95.00th=[18482], | 99.00th=[33424], 99.50th=[56361], 99.90th=[88605], 99.95th=[88605], | 99.99th=[88605] bw ( KiB/s): min= 480, max= 1056, per=25.20%, avg=868.77, stdev=173.40, samples=13 iops : min= 30, max= 66, avg=54.23, stdev=10.91, samples=13 write: IOPS=59, BW=949KiB/s (972kB/s)(6368KiB/6711msec) clat (usec): min=636, max=53362, avg=5627.13, stdev=7099.33 lat (usec): min=636, max=53364, avg=5628.24, stdev=7099.43 clat percentiles (usec): | 1.00th=[ 799], 5.00th=[ 824], 10.00th=[ 889], 20.00th=[ 1270], | 30.00th=[ 2376], 40.00th=[ 2769], 50.00th=[ 3261], 60.00th=[ 3785], | 70.00th=[ 5342], 80.00th=[ 7308], 90.00th=[13173], 95.00th=[17433], | 99.00th=[44303], 99.50th=[50594], 99.90th=[53216], 99.95th=[53216], | 99.99th=[53216] bw ( KiB/s): min= 512, max= 1216, per=26.20%, avg=949.92, stdev=202.34, samples=13 iops : min= 32, max= 76, avg=59.31, stdev=12.59, samples=13 lat (usec) : 100=0.91%, 250=0.13%, 750=0.13%, 1000=6.51% lat (msec) : 2=6.25%, 4=19.14%, 10=26.04%, 20=36.85%, 50=3.52% lat (msec) : 100=0.52% cpu : usr=0.04%, sys=20.31%, ctx=2743, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5398: Sun Aug 4 13:24:33 2024 read: IOPS=52, BW=847KiB/s (868kB/s)(5744KiB/6779msec) clat (usec): min=65, max=62483, avg=12076.86, stdev=5025.60 lat (usec): min=65, max=62484, avg=12077.68, stdev=5025.66 clat percentiles (usec): | 1.00th=[ 69], 5.00th=[ 7701], 10.00th=[ 8291], 20.00th=[ 9372], | 30.00th=[10028], 40.00th=[10683], 50.00th=[11469], 60.00th=[12125], | 70.00th=[12780], 80.00th=[14091], 90.00th=[17171], 95.00th=[19268], | 99.00th=[27919], 99.50th=[40633], 99.90th=[62653], 99.95th=[62653], | 99.99th=[62653] bw ( KiB/s): min= 480, max= 1184, per=24.56%, avg=846.62, stdev=175.04, samples=13 iops : min= 30, max= 74, avg=52.85, stdev=10.92, samples=13 write: IOPS=60, BW=965KiB/s (989kB/s)(6544KiB/6779msec) clat (usec): min=761, max=124919, avg=5938.16, stdev=9041.67 lat (usec): min=762, max=124920, avg=5939.22, stdev=9041.71 clat percentiles (usec): | 1.00th=[ 791], 5.00th=[ 840], 10.00th=[ 914], 20.00th=[ 1434], | 30.00th=[ 2442], 40.00th=[ 2835], 50.00th=[ 3294], 60.00th=[ 4146], | 70.00th=[ 5473], 80.00th=[ 8029], 90.00th=[ 13698], 95.00th=[ 16581], | 99.00th=[ 33817], 99.50th=[ 51119], 99.90th=[125305], 99.95th=[125305], | 99.99th=[125305] bw ( KiB/s): min= 640, max= 1504, per=26.13%, avg=947.54, stdev=240.76, samples=13 iops : min= 40, max= 94, avg=59.15, stdev=15.07, samples=13 lat (usec) : 100=0.78%, 250=0.26%, 1000=7.16% lat (msec) : 2=5.21%, 4=19.01%, 10=26.30%, 20=37.37%, 50=3.39% lat (msec) : 100=0.39%, 250=0.13% cpu : usr=0.10%, sys=19.02%, ctx=2735, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5399: Sun Aug 4 13:24:33 2024 read: IOPS=57, BW=916KiB/s (938kB/s)(6256KiB/6830msec) clat (usec): min=61, max=43716, avg=11452.41, stdev=4546.74 lat (usec): min=61, max=43717, avg=11453.11, stdev=4546.77 clat percentiles (usec): | 1.00th=[ 67], 5.00th=[ 6915], 10.00th=[ 8029], 20.00th=[ 8979], | 30.00th=[ 9634], 40.00th=[10421], 50.00th=[11076], 60.00th=[11731], | 70.00th=[12649], 80.00th=[13173], 90.00th=[15008], 95.00th=[17957], | 99.00th=[30016], 99.50th=[34341], 99.90th=[43779], 99.95th=[43779], | 99.99th=[43779] bw ( KiB/s): min= 544, max= 1248, per=26.35%, avg=908.31, stdev=210.46, samples=13 iops : min= 34, max= 78, avg=56.77, stdev=13.15, samples=13 write: IOPS=55, BW=883KiB/s (904kB/s)(6032KiB/6830msec) clat (usec): min=618, max=126348, avg=6134.10, stdev=9685.73 lat (usec): min=619, max=126350, avg=6135.23, stdev=9685.77 clat percentiles (usec): | 1.00th=[ 783], 5.00th=[ 832], 10.00th=[ 898], 20.00th=[ 1582], | 30.00th=[ 2474], 40.00th=[ 2966], 50.00th=[ 3654], 60.00th=[ 4621], | 70.00th=[ 5800], 80.00th=[ 7767], 90.00th=[ 13566], 95.00th=[ 17171], | 99.00th=[ 49546], 99.50th=[ 65799], 99.90th=[126354], 99.95th=[126354], | 99.99th=[126354] bw ( KiB/s): min= 416, max= 1120, per=23.49%, avg=851.69, stdev=245.98, samples=13 iops : min= 26, max= 70, avg=53.23, stdev=15.37, samples=13 lat (usec) : 100=0.91%, 250=0.78%, 750=0.13%, 1000=6.51% lat (msec) : 2=4.82%, 4=15.36%, 10=30.60%, 20=37.89%, 50=2.60% lat (msec) : 100=0.26%, 250=0.13% cpu : usr=0.15%, sys=18.74%, ctx=2787, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5400: Sun Aug 4 13:24:33 2024 read: IOPS=54, BW=868KiB/s (889kB/s)(6032KiB/6948msec) clat (usec): min=62, max=42393, avg=11597.06, stdev=4558.72 lat (usec): min=62, max=42394, avg=11597.73, stdev=4558.76 clat percentiles (usec): | 1.00th=[ 65], 5.00th=[ 6915], 10.00th=[ 8029], 20.00th=[ 8979], | 30.00th=[ 9634], 40.00th=[10290], 50.00th=[11207], 60.00th=[11994], | 70.00th=[12649], 80.00th=[13960], 90.00th=[16188], 95.00th=[20055], | 99.00th=[26608], 99.50th=[32113], 99.90th=[42206], 99.95th=[42206], | 99.99th=[42206] bw ( KiB/s): min= 512, max= 1120, per=24.56%, avg=846.62, stdev=176.07, samples=13 iops : min= 32, max= 70, avg=52.85, stdev=11.01, samples=13 write: IOPS=56, BW=900KiB/s (922kB/s)(6256KiB/6948msec) clat (usec): min=764, max=126693, avg=6469.54, stdev=10278.84 lat (usec): min=765, max=126695, avg=6470.66, stdev=10278.87 clat percentiles (usec): | 1.00th=[ 799], 5.00th=[ 848], 10.00th=[ 922], 20.00th=[ 1926], | 30.00th=[ 2376], 40.00th=[ 2835], 50.00th=[ 3458], 60.00th=[ 4424], | 70.00th=[ 5407], 80.00th=[ 8291], 90.00th=[ 13960], 95.00th=[ 20055], | 99.00th=[ 47973], 99.50th=[ 81265], 99.90th=[126354], 99.95th=[126354], | 99.99th=[126354] bw ( KiB/s): min= 544, max= 1312, per=24.64%, avg=893.38, stdev=178.78, samples=13 iops : min= 34, max= 82, avg=55.77, stdev=11.14, samples=13 lat (usec) : 100=1.43%, 250=0.39%, 1000=6.51% lat (msec) : 2=4.30%, 4=18.23%, 10=28.12%, 20=36.20%, 50=4.43% lat (msec) : 100=0.26%, 250=0.13% cpu : usr=0.14%, sys=17.86%, ctx=2810, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=3447KiB/s (3530kB/s), 847KiB/s-916KiB/s (868kB/s-938kB/s), io=23.4MiB (24.5MB), run=6711-6948msec WRITE: bw=3627KiB/s (3714kB/s), 883KiB/s-965KiB/s (904kB/s-989kB/s), io=24.6MiB (25.8MB), run=6711-6948msec rand-rw: (groupid=0, jobs=1): err= 0: pid=5402: Sun Aug 4 13:24:33 2024 read: IOPS=55, BW=881KiB/s (902kB/s)(5920KiB/6719msec) clat (usec): min=1856, max=84543, avg=7048.55, stdev=8104.54 lat (usec): min=1856, max=84544, avg=7049.40, stdev=8104.65 clat percentiles (usec): | 1.00th=[ 1942], 5.00th=[ 2212], 10.00th=[ 2343], 20.00th=[ 2638], | 30.00th=[ 2999], 40.00th=[ 3392], 50.00th=[ 4146], 60.00th=[ 5669], | 70.00th=[ 7635], 80.00th=[ 9896], 90.00th=[13304], 95.00th=[17957], | 99.00th=[36963], 99.50th=[76022], 99.90th=[84411], 99.95th=[84411], | 99.99th=[84411] bw ( KiB/s): min= 480, max= 1120, per=25.11%, avg=868.92, stdev=209.78, samples=13 iops : min= 30, max= 70, avg=54.31, stdev=13.11, samples=13 write: IOPS=59, BW=948KiB/s (971kB/s)(6368KiB/6719msec) clat (usec): min=2730, max=58800, avg=10168.19, stdev=7642.16 lat (usec): min=2731, max=58801, avg=10169.47, stdev=7642.34 clat percentiles (usec): | 1.00th=[ 2933], 5.00th=[ 3392], 10.00th=[ 3851], 20.00th=[ 4621], | 30.00th=[ 5800], 40.00th=[ 7111], 50.00th=[ 7963], 60.00th=[ 9634], | 70.00th=[11338], 80.00th=[13566], 90.00th=[19268], 95.00th=[21890], | 99.00th=[50070], 99.50th=[52167], 99.90th=[58983], 99.95th=[58983], | 99.99th=[58983] bw ( KiB/s): min= 576, max= 1280, per=26.10%, avg=950.15, stdev=238.89, samples=13 iops : min= 36, max= 80, avg=59.38, stdev=14.93, samples=13 lat (msec) : 2=0.65%, 4=28.52%, 10=42.19%, 20=22.79%, 50=4.95% lat (msec) : 100=0.91% cpu : usr=0.10%, sys=9.11%, ctx=1137, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5403: Sun Aug 4 13:24:33 2024 read: IOPS=53, BW=849KiB/s (869kB/s)(5744KiB/6768msec) clat (usec): min=1769, max=75917, avg=7032.28, stdev=7046.19 lat (usec): min=1770, max=75918, avg=7033.04, stdev=7046.18 clat percentiles (usec): | 1.00th=[ 1893], 5.00th=[ 2212], 10.00th=[ 2343], 20.00th=[ 2737], | 30.00th=[ 3032], 40.00th=[ 3785], 50.00th=[ 4555], 60.00th=[ 5866], | 70.00th=[ 8029], 80.00th=[10683], 90.00th=[13829], 95.00th=[17695], | 99.00th=[29754], 99.50th=[57410], 99.90th=[76022], 99.95th=[76022], | 99.99th=[76022] bw ( KiB/s): min= 512, max= 1120, per=24.47%, avg=846.77, stdev=194.04, samples=13 iops : min= 32, max= 70, avg=52.92, stdev=12.13, samples=13 write: IOPS=60, BW=967KiB/s (990kB/s)(6544KiB/6768msec) clat (msec): min=2, max=139, avg=10.31, stdev= 9.90 lat (msec): min=2, max=139, avg=10.31, stdev= 9.90 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 4], 10.00th=[ 4], 20.00th=[ 5], | 30.00th=[ 6], 40.00th=[ 7], 50.00th=[ 8], 60.00th=[ 10], | 70.00th=[ 11], 80.00th=[ 15], 90.00th=[ 19], 95.00th=[ 24], | 99.00th=[ 41], 99.50th=[ 48], 99.90th=[ 140], 99.95th=[ 140], | 99.99th=[ 140] bw ( KiB/s): min= 640, max= 1504, per=26.03%, avg=947.69, stdev=232.43, samples=13 iops : min= 40, max= 94, avg=59.23, stdev=14.53, samples=13 lat (msec) : 2=0.91%, 4=27.08%, 10=42.71%, 20=22.92%, 50=5.86% lat (msec) : 100=0.39%, 250=0.13% cpu : usr=0.06%, sys=9.37%, ctx=1209, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5404: Sun Aug 4 13:24:33 2024 read: IOPS=57, BW=914KiB/s (935kB/s)(6256KiB/6848msec) clat (usec): min=1901, max=75620, avg=6919.52, stdev=6817.08 lat (usec): min=1901, max=75620, avg=6920.23, stdev=6817.06 clat percentiles (usec): | 1.00th=[ 1975], 5.00th=[ 2180], 10.00th=[ 2376], 20.00th=[ 2671], | 30.00th=[ 3032], 40.00th=[ 3720], 50.00th=[ 4359], 60.00th=[ 6390], | 70.00th=[ 8160], 80.00th=[10159], 90.00th=[12518], 95.00th=[16450], | 99.00th=[30802], 99.50th=[56886], 99.90th=[76022], 99.95th=[76022], | 99.99th=[76022] bw ( KiB/s): min= 512, max= 1280, per=26.32%, avg=910.62, stdev=232.56, samples=13 iops : min= 32, max= 80, avg=56.85, stdev=14.58, samples=13 write: IOPS=55, BW=881KiB/s (902kB/s)(6032KiB/6848msec) clat (msec): min=2, max=140, avg=10.81, stdev=10.03 lat (msec): min=2, max=140, avg=10.82, stdev=10.03 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 4], 10.00th=[ 4], 20.00th=[ 6], | 30.00th=[ 7], 40.00th=[ 8], 50.00th=[ 9], 60.00th=[ 10], | 70.00th=[ 12], 80.00th=[ 15], 90.00th=[ 19], 95.00th=[ 26], | 99.00th=[ 45], 99.50th=[ 59], 99.90th=[ 140], 99.95th=[ 140], | 99.99th=[ 140] bw ( KiB/s): min= 416, max= 1152, per=23.39%, avg=851.54, stdev=246.50, samples=13 iops : min= 26, max= 72, avg=53.15, stdev=15.48, samples=13 lat (msec) : 2=0.52%, 4=28.65%, 10=41.41%, 20=23.18%, 50=5.73% lat (msec) : 100=0.39%, 250=0.13% cpu : usr=0.10%, sys=9.23%, ctx=1225, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5405: Sun Aug 4 13:24:33 2024 read: IOPS=54, BW=872KiB/s (892kB/s)(6032KiB/6921msec) clat (usec): min=1803, max=53742, avg=7088.61, stdev=6136.99 lat (usec): min=1803, max=53743, avg=7089.28, stdev=6137.01 clat percentiles (usec): | 1.00th=[ 1975], 5.00th=[ 2245], 10.00th=[ 2343], 20.00th=[ 2671], | 30.00th=[ 2999], 40.00th=[ 3621], 50.00th=[ 4621], 60.00th=[ 6390], | 70.00th=[ 8848], 80.00th=[10945], 90.00th=[14615], 95.00th=[16909], | 99.00th=[32375], 99.50th=[36439], 99.90th=[53740], 99.95th=[53740], | 99.99th=[53740] bw ( KiB/s): min= 480, max= 1184, per=24.39%, avg=843.85, stdev=214.21, samples=13 iops : min= 30, max= 74, avg=52.54, stdev=13.45, samples=13 write: IOPS=56, BW=904KiB/s (926kB/s)(6256KiB/6921msec) clat (msec): min=2, max=147, avg=10.65, stdev=11.69 lat (msec): min=2, max=147, avg=10.66, stdev=11.69 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 4], 10.00th=[ 4], 20.00th=[ 5], | 30.00th=[ 6], 40.00th=[ 7], 50.00th=[ 9], 60.00th=[ 10], | 70.00th=[ 12], 80.00th=[ 14], 90.00th=[ 18], 95.00th=[ 25], | 99.00th=[ 53], 99.50th=[ 134], 99.90th=[ 148], 99.95th=[ 148], | 99.99th=[ 148] bw ( KiB/s): min= 544, max= 1248, per=24.46%, avg=890.54, stdev=174.57, samples=13 iops : min= 34, max= 78, avg=55.46, stdev=10.89, samples=13 lat (msec) : 2=0.78%, 4=28.26%, 10=38.15%, 20=27.73%, 50=4.43% lat (msec) : 100=0.39%, 250=0.26% cpu : usr=0.06%, sys=9.15%, ctx=1196, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=3461KiB/s (3544kB/s), 849KiB/s-914KiB/s (869kB/s-935kB/s), io=23.4MiB (24.5MB), run=6719-6921msec WRITE: bw=3641KiB/s (3728kB/s), 881KiB/s-967KiB/s (902kB/s-990kB/s), io=24.6MiB (25.8MB), run=6719-6921msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=5416: Sun Aug 4 13:24:38 2024 read: IOPS=1, BW=2032KiB/s (2081kB/s)(7168KiB/3527msec) clat (msec): min=20, max=1046, avg=322.95, stdev=450.24 lat (msec): min=20, max=1046, avg=322.95, stdev=450.24 clat percentiles (msec): | 1.00th=[ 21], 5.00th=[ 21], 10.00th=[ 21], 20.00th=[ 53], | 30.00th=[ 59], 40.00th=[ 59], 50.00th=[ 77], 60.00th=[ 96], | 70.00th=[ 96], 80.00th=[ 911], 90.00th=[ 1045], 95.00th=[ 1045], | 99.00th=[ 1045], 99.50th=[ 1045], 99.90th=[ 1045], 99.95th=[ 1045], | 99.99th=[ 1045] bw ( KiB/s): min= 4096, max= 6144, per=88.18%, avg=5120.00, stdev=1448.15, samples=2 iops : min= 4, max= 6, avg= 5.00, stdev= 1.41, samples=2 write: IOPS=1, BW=1452KiB/s (1486kB/s)(5120KiB/3527msec) clat (msec): min=29, max=910, avg=244.53, stdev=375.65 lat (msec): min=29, max=910, avg=244.59, stdev=375.66 clat percentiles (msec): | 1.00th=[ 30], 5.00th=[ 30], 10.00th=[ 30], 20.00th=[ 30], | 30.00th=[ 60], 40.00th=[ 60], 50.00th=[ 64], 60.00th=[ 64], | 70.00th=[ 161], 80.00th=[ 161], 90.00th=[ 911], 95.00th=[ 911], | 99.00th=[ 911], 99.50th=[ 911], 99.90th=[ 911], 99.95th=[ 911], | 99.99th=[ 911] bw ( KiB/s): min= 1828, max= 6144, per=49.03%, avg=3986.00, stdev=3051.87, samples=2 iops : min= 1, max= 6, avg= 3.50, stdev= 3.54, samples=2 lat (msec) : 50=16.67%, 100=50.00%, 250=8.33%, 1000=16.67% cpu : usr=0.03%, sys=3.66%, ctx=55, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5417: Sun Aug 4 13:24:38 2024 read: IOPS=0, BW=595KiB/s (610kB/s)(2048KiB/3440msec) clat (usec): min=24564, max=30867, avg=27715.87, stdev=4456.87 lat (usec): min=24593, max=30868, avg=27731.00, stdev=4436.50 clat percentiles (usec): | 1.00th=[24511], 5.00th=[24511], 10.00th=[24511], 20.00th=[24511], | 30.00th=[24511], 40.00th=[24511], 50.00th=[24511], 60.00th=[30802], | 70.00th=[30802], 80.00th=[30802], 90.00th=[30802], 95.00th=[30802], | 99.00th=[30802], 99.50th=[30802], 99.90th=[30802], 99.95th=[30802], | 99.99th=[30802] bw ( KiB/s): min= 2048, max= 2048, per=35.27%, avg=2048.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 write: IOPS=2, BW=2977KiB/s (3048kB/s)(10.0MiB/3440msec) clat (msec): min=33, max=1048, avg=336.50, stdev=432.76 lat (msec): min=33, max=1048, avg=336.58, stdev=432.76 clat percentiles (msec): | 1.00th=[ 34], 5.00th=[ 34], 10.00th=[ 34], 20.00th=[ 34], | 30.00th=[ 39], 40.00th=[ 44], 50.00th=[ 50], 60.00th=[ 121], | 70.00th=[ 827], 80.00th=[ 827], 90.00th=[ 986], 95.00th=[ 1045], | 99.00th=[ 1045], 99.50th=[ 1045], 99.90th=[ 1045], 99.95th=[ 1045], | 99.99th=[ 1045] bw ( KiB/s): min= 1780, max= 6144, per=43.25%, avg=3515.75, stdev=2035.21, samples=4 iops : min= 1, max= 6, avg= 3.00, stdev= 2.45, samples=4 lat (msec) : 50=58.33%, 250=16.67%, 1000=16.67% cpu : usr=0.00%, sys=4.25%, ctx=55, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5418: Sun Aug 4 13:24:38 2024 read: IOPS=2, BW=2559KiB/s (2620kB/s)(6144KiB/2401msec) clat (msec): min=20, max=1042, avg=351.82, stdev=450.77 lat (msec): min=20, max=1042, avg=351.82, stdev=450.77 clat percentiles (msec): | 1.00th=[ 22], 5.00th=[ 22], 10.00th=[ 22], 20.00th=[ 23], | 30.00th=[ 23], 40.00th=[ 57], 50.00th=[ 57], 60.00th=[ 169], | 70.00th=[ 802], 80.00th=[ 802], 90.00th=[ 1045], 95.00th=[ 1045], | 99.00th=[ 1045], 99.50th=[ 1045], 99.90th=[ 1045], 99.95th=[ 1045], | 99.99th=[ 1045] bw ( KiB/s): min= 2048, max= 6144, per=58.79%, avg=3413.33, stdev=2364.83, samples=3 iops : min= 2, max= 6, avg= 3.33, stdev= 2.31, samples=3 write: IOPS=2, BW=2559KiB/s (2620kB/s)(6144KiB/2401msec) clat (usec): min=24663, max=72339, avg=46050.32, stdev=21217.12 lat (usec): min=24722, max=72383, avg=46139.35, stdev=21195.03 clat percentiles (usec): | 1.00th=[24773], 5.00th=[24773], 10.00th=[24773], 20.00th=[26870], | 30.00th=[26870], 40.00th=[30016], 50.00th=[30016], 60.00th=[58459], | 70.00th=[64226], 80.00th=[64226], 90.00th=[71828], 95.00th=[71828], | 99.00th=[71828], 99.50th=[71828], 99.90th=[71828], 99.95th=[71828], | 99.99th=[71828] bw ( KiB/s): min= 2048, max= 6144, per=50.39%, avg=4096.00, stdev=2048.00, samples=3 iops : min= 2, max= 6, avg= 4.00, stdev= 2.00, samples=3 lat (msec) : 50=41.67%, 100=33.33%, 250=8.33%, 1000=8.33% cpu : usr=0.04%, sys=4.71%, ctx=70, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5420: Sun Aug 4 13:24:38 2024 read: IOPS=1, BW=1984KiB/s (2031kB/s)(5120KiB/2581msec) clat (usec): min=14663, max=36611, avg=24711.12, stdev=9504.99 lat (usec): min=14664, max=36612, avg=24712.36, stdev=9505.09 clat percentiles (usec): | 1.00th=[14615], 5.00th=[14615], 10.00th=[14615], 20.00th=[14615], | 30.00th=[15270], 40.00th=[15270], 50.00th=[27919], 60.00th=[27919], | 70.00th=[29230], 80.00th=[29230], 90.00th=[36439], 95.00th=[36439], | 99.00th=[36439], 99.50th=[36439], 99.90th=[36439], 99.95th=[36439], | 99.99th=[36439] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=2, BW=2777KiB/s (2844kB/s)(7168KiB/2581msec) clat (msec): min=25, max=1041, avg=348.03, stdev=474.80 lat (msec): min=26, max=1041, avg=348.10, stdev=474.79 clat percentiles (msec): | 1.00th=[ 26], 5.00th=[ 26], 10.00th=[ 26], 20.00th=[ 36], | 30.00th=[ 56], 40.00th=[ 56], 50.00th=[ 100], 60.00th=[ 140], | 70.00th=[ 140], 80.00th=[ 1036], 90.00th=[ 1045], 95.00th=[ 1045], | 99.00th=[ 1045], 99.50th=[ 1045], 99.90th=[ 1045], 99.95th=[ 1045], | 99.99th=[ 1045] bw ( KiB/s): min= 2048, max= 6131, per=50.33%, avg=4091.67, stdev=2041.50, samples=3 iops : min= 2, max= 5, avg= 3.67, stdev= 1.53, samples=3 lat (msec) : 20=16.67%, 50=41.67%, 100=16.67%, 250=8.33% cpu : usr=0.04%, sys=4.22%, ctx=53, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=5807KiB/s (5946kB/s), 595KiB/s-2559KiB/s (610kB/s-2620kB/s), io=20.0MiB (20.0MB), run=2401-3527msec WRITE: bw=8129KiB/s (8324kB/s), 1452KiB/s-2977KiB/s (1486kB/s-3048kB/s), io=28.0MiB (29.4MB), run=2401-3527msec rand-rw: (groupid=0, jobs=1): err= 0: pid=5419: Sun Aug 4 13:24:39 2024 read: IOPS=1, BW=1582KiB/s (1620kB/s)(7168KiB/4530msec) clat (msec): min=49, max=919, avg=606.85, stdev=311.29 lat (msec): min=49, max=919, avg=606.85, stdev=311.29 clat percentiles (msec): | 1.00th=[ 50], 5.00th=[ 50], 10.00th=[ 50], 20.00th=[ 456], | 30.00th=[ 464], 40.00th=[ 464], 50.00th=[ 642], 60.00th=[ 844], | 70.00th=[ 844], 80.00th=[ 877], 90.00th=[ 919], 95.00th=[ 919], | 99.00th=[ 919], 99.50th=[ 919], 99.90th=[ 919], 99.95th=[ 919], | 99.99th=[ 919] bw ( KiB/s): min= 1828, max= 3663, per=49.56%, avg=2240.20, stdev=800.95, samples=5 iops : min= 1, max= 3, avg= 1.60, stdev= 0.89, samples=5 write: IOPS=1, BW=1130KiB/s (1157kB/s)(5120KiB/4530msec) clat (usec): min=17389, max=60056, avg=33454.07, stdev=17133.58 lat (usec): min=17424, max=60177, avg=33538.73, stdev=17158.08 clat percentiles (usec): | 1.00th=[17433], 5.00th=[17433], 10.00th=[17433], 20.00th=[17433], | 30.00th=[23987], 40.00th=[23987], 50.00th=[25035], 60.00th=[25035], | 70.00th=[40633], 80.00th=[40633], 90.00th=[60031], 95.00th=[60031], | 99.00th=[60031], 99.50th=[60031], 99.90th=[60031], 99.95th=[60031], | 99.99th=[60031] bw ( KiB/s): min= 1828, max= 5495, per=48.21%, avg=3051.33, stdev=2116.28, samples=3 iops : min= 1, max= 5, avg= 2.33, stdev= 2.31, samples=3 lat (msec) : 20=8.33%, 50=33.33%, 100=8.33%, 500=16.67%, 750=8.33% lat (msec) : 1000=25.00% cpu : usr=0.02%, sys=22.15%, ctx=1588, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5421: Sun Aug 4 13:24:39 2024 read: IOPS=0, BW=587KiB/s (601kB/s)(2048KiB/3491msec) clat (msec): min=50, max=135, avg=92.98, stdev=59.90 lat (msec): min=50, max=135, avg=92.98, stdev=59.90 clat percentiles (msec): | 1.00th=[ 51], 5.00th=[ 51], 10.00th=[ 51], 20.00th=[ 51], | 30.00th=[ 51], 40.00th=[ 51], 50.00th=[ 51], 60.00th=[ 136], | 70.00th=[ 136], 80.00th=[ 136], 90.00th=[ 136], 95.00th=[ 136], | 99.00th=[ 136], 99.50th=[ 136], 99.90th=[ 136], 99.95th=[ 136], | 99.99th=[ 136] bw ( KiB/s): min= 1822, max= 2048, per=42.81%, avg=1935.00, stdev=159.81, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 write: IOPS=2, BW=2933KiB/s (3004kB/s)(10.0MiB/3491msec) clat (msec): min=24, max=1067, avg=320.74, stdev=442.41 lat (msec): min=24, max=1067, avg=320.82, stdev=442.40 clat percentiles (msec): | 1.00th=[ 25], 5.00th=[ 25], 10.00th=[ 25], 20.00th=[ 25], | 30.00th=[ 42], 40.00th=[ 42], 50.00th=[ 48], 60.00th=[ 80], | 70.00th=[ 810], 80.00th=[ 810], 90.00th=[ 986], 95.00th=[ 1070], | 99.00th=[ 1070], 99.50th=[ 1070], 99.90th=[ 1070], 99.95th=[ 1070], | 99.99th=[ 1070] bw ( KiB/s): min= 1822, max= 9110, per=67.19%, avg=4252.33, stdev=4206.86, samples=3 iops : min= 1, max= 8, avg= 3.33, stdev= 4.04, samples=3 lat (msec) : 50=41.67%, 100=25.00%, 250=8.33%, 1000=16.67% cpu : usr=0.00%, sys=9.89%, ctx=103, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5422: Sun Aug 4 13:24:39 2024 read: IOPS=4, BW=4099KiB/s (4197kB/s)(6144KiB/1499msec) clat (usec): min=494, max=61140, avg=21105.92, stdev=26488.77 lat (usec): min=494, max=61141, avg=21106.74, stdev=26488.84 clat percentiles (usec): | 1.00th=[ 494], 5.00th=[ 494], 10.00th=[ 494], 20.00th=[ 3163], | 30.00th=[ 3163], 40.00th=[ 6390], 50.00th=[ 6390], 60.00th=[ 7046], | 70.00th=[48497], 80.00th=[48497], 90.00th=[61080], 95.00th=[61080], | 99.00th=[61080], 99.50th=[61080], 99.90th=[61080], 99.95th=[61080], | 99.99th=[61080] write: IOPS=4, BW=4099KiB/s (4197kB/s)(6144KiB/1499msec) clat (msec): min=26, max=1098, avg=222.94, stdev=429.87 lat (msec): min=26, max=1098, avg=223.07, stdev=429.82 clat percentiles (msec): | 1.00th=[ 27], 5.00th=[ 27], 10.00th=[ 27], 20.00th=[ 27], | 30.00th=[ 27], 40.00th=[ 39], 50.00th=[ 39], 60.00th=[ 45], | 70.00th=[ 105], 80.00th=[ 105], 90.00th=[ 1099], 95.00th=[ 1099], | 99.00th=[ 1099], 99.50th=[ 1099], 99.90th=[ 1099], 99.95th=[ 1099], | 99.99th=[ 1099] bw ( KiB/s): min= 4096, max= 4096, per=64.72%, avg=4096.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (usec) : 500=8.33% lat (msec) : 4=8.33%, 10=16.67%, 50=41.67%, 100=8.33%, 250=8.33% cpu : usr=0.00%, sys=11.28%, ctx=91, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5423: Sun Aug 4 13:24:39 2024 read: IOPS=1, BW=1974KiB/s (2021kB/s)(5120KiB/2594msec) clat (usec): min=350, max=51046, avg=18038.54, stdev=24506.21 lat (usec): min=350, max=51048, avg=18039.78, stdev=24506.72 clat percentiles (usec): | 1.00th=[ 351], 5.00th=[ 351], 10.00th=[ 351], 20.00th=[ 351], | 30.00th=[ 506], 40.00th=[ 506], 50.00th=[ 570], 60.00th=[ 570], | 70.00th=[37487], 80.00th=[37487], 90.00th=[51119], 95.00th=[51119], | 99.00th=[51119], 99.50th=[51119], 99.90th=[51119], 99.95th=[51119], | 99.99th=[51119] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=2, BW=2763KiB/s (2830kB/s)(7168KiB/2594msec) clat (msec): min=31, max=1038, avg=348.25, stdev=470.52 lat (msec): min=31, max=1038, avg=348.31, stdev=470.51 clat percentiles (msec): | 1.00th=[ 32], 5.00th=[ 32], 10.00th=[ 32], 20.00th=[ 42], | 30.00th=[ 50], 40.00th=[ 50], 50.00th=[ 106], 60.00th=[ 140], | 70.00th=[ 140], 80.00th=[ 1028], 90.00th=[ 1036], 95.00th=[ 1036], | 99.00th=[ 1036], 99.50th=[ 1036], 99.90th=[ 1036], 99.95th=[ 1036], | 99.99th=[ 1036] bw ( KiB/s): min= 2048, max= 8192, per=64.72%, avg=4096.00, stdev=3547.24, samples=3 iops : min= 2, max= 8, avg= 4.00, stdev= 3.46, samples=3 lat (usec) : 500=8.33%, 750=16.67% lat (msec) : 50=33.33%, 100=8.33%, 250=16.67% cpu : usr=0.04%, sys=10.53%, ctx=111, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=4521KiB/s (4629kB/s), 587KiB/s-4099KiB/s (601kB/s-4197kB/s), io=20.0MiB (20.0MB), run=1499-4530msec WRITE: bw=6329KiB/s (6481kB/s), 1130KiB/s-4099KiB/s (1157kB/s-4197kB/s), io=28.0MiB (29.4MB), run=1499-4530msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=5433: Sun Aug 4 13:24:43 2024 read: IOPS=0, BW=3961KiB/s (4056kB/s)(12.0MiB/3102msec) clat (msec): min=247, max=1658, avg=1027.97, stdev=717.31 lat (msec): min=247, max=1658, avg=1027.98, stdev=717.31 clat percentiles (msec): | 1.00th=[ 247], 5.00th=[ 247], 10.00th=[ 247], 20.00th=[ 247], | 30.00th=[ 247], 40.00th=[ 1183], 50.00th=[ 1183], 60.00th=[ 1183], | 70.00th=[ 1653], 80.00th=[ 1653], 90.00th=[ 1653], 95.00th=[ 1653], | 99.00th=[ 1653], 99.50th=[ 1653], 99.90th=[ 1653], 99.95th=[ 1653], | 99.99th=[ 1653] bw ( KiB/s): min= 8175, max= 8192, per=100.00%, avg=8183.50, stdev=12.02, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 250=33.33% cpu : usr=0.03%, sys=41.81%, ctx=812, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5435: Sun Aug 4 13:24:43 2024 read: IOPS=0, BW=1979KiB/s (2026kB/s)(4096KiB/2070msec) clat (nsec): min=216080k, max=216080k, avg=216079514.00, stdev= 0.00 lat (nsec): min=216082k, max=216082k, avg=216081567.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 218], 5.00th=[ 218], 10.00th=[ 218], 20.00th=[ 218], | 30.00th=[ 218], 40.00th=[ 218], 50.00th=[ 218], 60.00th=[ 218], | 70.00th=[ 218], 80.00th=[ 218], 90.00th=[ 218], 95.00th=[ 218], | 99.00th=[ 218], 99.50th=[ 218], 99.90th=[ 218], 99.95th=[ 218], | 99.99th=[ 218] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=0, BW=3957KiB/s (4052kB/s)(8192KiB/2070msec) clat (msec): min=118, max=1685, avg=901.98, stdev=1108.05 lat (msec): min=118, max=1685, avg=902.31, stdev=1107.99 clat percentiles (msec): | 1.00th=[ 118], 5.00th=[ 118], 10.00th=[ 118], 20.00th=[ 118], | 30.00th=[ 118], 40.00th=[ 118], 50.00th=[ 118], 60.00th=[ 1687], | 70.00th=[ 1687], 80.00th=[ 1687], 90.00th=[ 1687], 95.00th=[ 1687], | 99.00th=[ 1687], 99.50th=[ 1687], 99.90th=[ 1687], 99.95th=[ 1687], | 99.99th=[ 1687] bw ( KiB/s): min= 8192, max= 8192, per=78.33%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67% cpu : usr=0.05%, sys=14.21%, ctx=47, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5437: Sun Aug 4 13:24:43 2024 write: IOPS=0, BW=3922KiB/s (4016kB/s)(12.0MiB/3133msec) clat (msec): min=366, max=1744, avg=1030.44, stdev=690.38 lat (msec): min=366, max=1744, avg=1030.69, stdev=690.47 clat percentiles (msec): | 1.00th=[ 368], 5.00th=[ 368], 10.00th=[ 368], 20.00th=[ 368], | 30.00th=[ 368], 40.00th=[ 978], 50.00th=[ 978], 60.00th=[ 978], | 70.00th=[ 1737], 80.00th=[ 1737], 90.00th=[ 1737], 95.00th=[ 1737], | 99.00th=[ 1737], 99.50th=[ 1737], 99.90th=[ 1737], 99.95th=[ 1737], | 99.99th=[ 1737] bw ( KiB/s): min= 5665, max= 8192, per=66.25%, avg=6928.50, stdev=1786.86, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 500=33.33%, 1000=33.33% cpu : usr=0.00%, sys=9.36%, ctx=44, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5439: Sun Aug 4 13:24:43 2024 write: IOPS=1, BW=5919KiB/s (6061kB/s)(12.0MiB/2076msec) clat (msec): min=116, max=1704, avg=679.24, stdev=889.52 lat (msec): min=116, max=1705, avg=679.56, stdev=889.48 clat percentiles (msec): | 1.00th=[ 117], 5.00th=[ 117], 10.00th=[ 117], 20.00th=[ 117], | 30.00th=[ 117], 40.00th=[ 218], 50.00th=[ 218], 60.00th=[ 218], | 70.00th=[ 1703], 80.00th=[ 1703], 90.00th=[ 1703], 95.00th=[ 1703], | 99.00th=[ 1703], 99.50th=[ 1703], 99.90th=[ 1703], 99.95th=[ 1703], | 99.99th=[ 1703] bw ( KiB/s): min= 8192, max= 8192, per=78.33%, avg=8192.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 lat (msec) : 250=66.67% cpu : usr=0.00%, sys=15.08%, ctx=77, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=5282KiB/s (5409kB/s), 1979KiB/s-3961KiB/s (2026kB/s-4056kB/s), io=16.0MiB (16.8MB), run=2070-3102msec WRITE: bw=10.2MiB/s (10.7MB/s), 3922KiB/s-5919KiB/s (4016kB/s-6061kB/s), io=32.0MiB (33.6MB), run=2070-3133msec rand-rw: (groupid=0, jobs=1): err= 0: pid=5432: Sun Aug 4 13:24:43 2024 read: IOPS=0, BW=3735KiB/s (3825kB/s)(12.0MiB/3290msec) clat (msec): min=527, max=1822, avg=1077.77, stdev=669.04 lat (msec): min=527, max=1822, avg=1077.77, stdev=669.04 clat percentiles (msec): | 1.00th=[ 527], 5.00th=[ 527], 10.00th=[ 527], 20.00th=[ 527], | 30.00th=[ 527], 40.00th=[ 885], 50.00th=[ 885], 60.00th=[ 885], | 70.00th=[ 1821], 80.00th=[ 1821], 90.00th=[ 1821], 95.00th=[ 1821], | 99.00th=[ 1821], 99.50th=[ 1821], 99.90th=[ 1821], 99.95th=[ 1821], | 99.99th=[ 1821] bw ( KiB/s): min= 5809, max= 5809, per=100.00%, avg=5809.00, stdev= 0.00, samples=2 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=2 lat (msec) : 750=33.33%, 1000=33.33% cpu : usr=0.06%, sys=4.56%, ctx=67, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5434: Sun Aug 4 13:24:43 2024 read: IOPS=0, BW=1686KiB/s (1726kB/s)(4096KiB/2430msec) clat (nsec): min=482290k, max=482290k, avg=482290245.00, stdev= 0.00 lat (nsec): min=482291k, max=482291k, avg=482291472.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 481], 5.00th=[ 481], 10.00th=[ 481], 20.00th=[ 481], | 30.00th=[ 481], 40.00th=[ 481], 50.00th=[ 481], 60.00th=[ 481], | 70.00th=[ 481], 80.00th=[ 481], 90.00th=[ 481], 95.00th=[ 481], | 99.00th=[ 481], 99.50th=[ 481], 99.90th=[ 481], 99.95th=[ 481], | 99.99th=[ 481] bw ( KiB/s): min= 5927, max= 5927, per=100.00%, avg=5927.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=0, BW=3371KiB/s (3452kB/s)(8192KiB/2430msec) clat (msec): min=45, max=1859, avg=952.52, stdev=1282.09 lat (msec): min=46, max=1859, avg=952.75, stdev=1282.26 clat percentiles (msec): | 1.00th=[ 46], 5.00th=[ 46], 10.00th=[ 46], 20.00th=[ 46], | 30.00th=[ 46], 40.00th=[ 46], 50.00th=[ 46], 60.00th=[ 1854], | 70.00th=[ 1854], 80.00th=[ 1854], 90.00th=[ 1854], 95.00th=[ 1854], | 99.00th=[ 1854], 99.50th=[ 1854], 99.90th=[ 1854], 99.95th=[ 1854], | 99.99th=[ 1854] bw ( KiB/s): min= 5927, max= 5927, per=43.96%, avg=5927.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 50=33.33%, 500=33.33% cpu : usr=0.00%, sys=3.75%, ctx=48, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5436: Sun Aug 4 13:24:43 2024 write: IOPS=5, BW=20.3MiB/s (21.3MB/s)(12.0MiB/591msec) clat (msec): min=135, max=271, avg=183.55, stdev=76.52 lat (msec): min=136, max=272, avg=183.82, stdev=76.51 clat percentiles (msec): | 1.00th=[ 136], 5.00th=[ 136], 10.00th=[ 136], 20.00th=[ 136], | 30.00th=[ 136], 40.00th=[ 144], 50.00th=[ 144], 60.00th=[ 144], | 70.00th=[ 271], 80.00th=[ 271], 90.00th=[ 271], 95.00th=[ 271], | 99.00th=[ 271], 99.50th=[ 271], 99.90th=[ 271], 99.95th=[ 271], | 99.99th=[ 271] bw ( KiB/s): min=16384, max=16384, per=100.00%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.00%, sys=26.27%, ctx=75, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5438: Sun Aug 4 13:24:43 2024 write: IOPS=4, BW=18.9MiB/s (19.8MB/s)(12.0MiB/634msec) clat (msec): min=75, max=404, avg=205.01, stdev=175.07 lat (msec): min=75, max=404, avg=205.34, stdev=175.01 clat percentiles (msec): | 1.00th=[ 75], 5.00th=[ 75], 10.00th=[ 75], 20.00th=[ 75], | 30.00th=[ 75], 40.00th=[ 136], 50.00th=[ 136], 60.00th=[ 136], | 70.00th=[ 405], 80.00th=[ 405], 90.00th=[ 405], 95.00th=[ 405], | 99.00th=[ 405], 99.50th=[ 405], 99.90th=[ 405], 99.95th=[ 405], | 99.99th=[ 405] bw ( KiB/s): min=16384, max=16384, per=100.00%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 250=33.33%, 500=33.33% cpu : usr=0.16%, sys=13.11%, ctx=51, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=4980KiB/s (5099kB/s), 1686KiB/s-3735KiB/s (1726kB/s-3825kB/s), io=16.0MiB (16.8MB), run=2430-3290msec WRITE: bw=13.2MiB/s (13.8MB/s), 3371KiB/s-20.3MiB/s (3452kB/s-21.3MB/s), io=32.0MiB (33.6MB), run=591-2430msec