== sanity test 398b: DIO and buffer IO race ============== 22:32:19 (1746153139) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.6376 s, 30.7 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=32526: Thu May 1 22:32:52 2025 read: IOPS=52, BW=208KiB/s (213kB/s)(5992KiB/28766msec) clat (usec): min=47, max=250004, avg=8233.39, stdev=9691.76 lat (usec): min=48, max=250005, avg=8236.33, stdev=9692.48 clat percentiles (usec): | 1.00th=[ 73], 5.00th=[ 91], 10.00th=[ 101], 20.00th=[ 135], | 30.00th=[ 4113], 40.00th=[ 5473], 50.00th=[ 6652], 60.00th=[ 8225], | 70.00th=[ 10421], 80.00th=[ 13698], 90.00th=[ 18220], 95.00th=[ 21365], | 99.00th=[ 30016], 99.50th=[ 36963], 99.90th=[ 69731], 99.95th=[250610], | 99.99th=[250610] bw ( KiB/s): min= 104, max= 408, per=24.87%, avg=206.95, stdev=65.66, samples=57 iops : min= 26, max= 102, avg=51.47, stdev=16.46, samples=57 write: IOPS=54, BW=219KiB/s (224kB/s)(6296KiB/28766msec) clat (usec): min=467, max=119654, avg=10352.87, stdev=11083.83 lat (usec): min=467, max=119655, avg=10354.10, stdev=11083.84 clat percentiles (usec): | 1.00th=[ 619], 5.00th=[ 717], 10.00th=[ 766], 20.00th=[ 971], | 30.00th=[ 1975], 40.00th=[ 4686], 50.00th=[ 8029], 60.00th=[ 10552], | 70.00th=[ 13042], 80.00th=[ 17433], 90.00th=[ 23462], 95.00th=[ 30278], | 99.00th=[ 47449], 99.50th=[ 60556], 99.90th=[ 96994], 99.95th=[120062], | 99.99th=[120062] bw ( KiB/s): min= 96, max= 424, per=24.85%, avg=216.47, stdev=65.77, samples=57 iops : min= 24, max= 106, avg=53.86, stdev=16.52, samples=57 lat (usec) : 50=0.10%, 100=4.46%, 250=6.80%, 500=0.81%, 750=4.52% lat (usec) : 1000=6.25% lat (msec) : 2=4.98%, 4=5.96%, 10=29.04%, 20=25.72%, 50=10.94% lat (msec) : 100=0.36%, 250=0.03%, 500=0.03% cpu : usr=0.15%, sys=14.46%, ctx=4938, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32527: Thu May 1 22:32:52 2025 read: IOPS=51, BW=207KiB/s (212kB/s)(5940KiB/28644msec) clat (usec): min=47, max=114552, avg=8325.10, stdev=8304.73 lat (usec): min=47, max=114553, avg=8326.72, stdev=8304.74 clat percentiles (usec): | 1.00th=[ 78], 5.00th=[ 94], 10.00th=[ 106], 20.00th=[ 161], | 30.00th=[ 4686], 40.00th=[ 5800], 50.00th=[ 6849], 60.00th=[ 8160], | 70.00th=[ 9896], 80.00th=[ 13173], 90.00th=[ 17695], 95.00th=[ 21890], | 99.00th=[ 33424], 99.50th=[ 40109], 99.90th=[108528], 99.95th=[114820], | 99.99th=[114820] bw ( KiB/s): min= 80, max= 368, per=24.92%, avg=207.32, stdev=65.03, samples=57 iops : min= 20, max= 92, avg=51.63, stdev=16.28, samples=57 write: IOPS=55, BW=222KiB/s (227kB/s)(6348KiB/28644msec) clat (usec): min=442, max=243625, avg=10175.00, stdev=11591.84 lat (usec): min=442, max=243626, avg=10176.64, stdev=11591.52 clat percentiles (usec): | 1.00th=[ 635], 5.00th=[ 717], 10.00th=[ 766], 20.00th=[ 979], | 30.00th=[ 1958], 40.00th=[ 4359], 50.00th=[ 7898], 60.00th=[ 10421], | 70.00th=[ 13173], 80.00th=[ 17433], 90.00th=[ 23462], 95.00th=[ 28967], | 99.00th=[ 44303], 99.50th=[ 52691], 99.90th=[ 71828], 99.95th=[244319], | 99.99th=[244319] bw ( KiB/s): min= 119, max= 360, per=25.35%, avg=220.82, stdev=60.84, samples=57 iops : min= 29, max= 90, avg=55.00, stdev=15.23, samples=57 lat (usec) : 50=0.03%, 100=3.68%, 250=6.41%, 500=0.62%, 750=4.13% lat (usec) : 1000=6.93% lat (msec) : 2=5.14%, 4=5.92%, 10=31.54%, 20=24.41%, 50=10.74% lat (msec) : 100=0.33%, 250=0.10% cpu : usr=0.24%, sys=14.71%, ctx=4943, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32529: Thu May 1 22:32:52 2025 read: IOPS=52, BW=209KiB/s (214kB/s)(6028KiB/28850msec) clat (usec): min=55, max=249450, avg=8574.23, stdev=10872.91 lat (usec): min=55, max=249451, avg=8576.60, stdev=10873.21 clat percentiles (usec): | 1.00th=[ 80], 5.00th=[ 91], 10.00th=[ 101], 20.00th=[ 129], | 30.00th=[ 4146], 40.00th=[ 5538], 50.00th=[ 6915], 60.00th=[ 8455], | 70.00th=[ 10421], 80.00th=[ 13829], 90.00th=[ 18220], 95.00th=[ 22152], | 99.00th=[ 34866], 99.50th=[ 41681], 99.90th=[143655], 99.95th=[248513], | 99.99th=[248513] bw ( KiB/s): min= 96, max= 376, per=24.72%, avg=205.63, stdev=62.80, samples=57 iops : min= 24, max= 94, avg=51.18, stdev=15.72, samples=57 write: IOPS=54, BW=217KiB/s (222kB/s)(6260KiB/28850msec) clat (usec): min=443, max=104486, avg=10090.56, stdev=10786.59 lat (usec): min=444, max=104487, avg=10091.85, stdev=10786.51 clat percentiles (usec): | 1.00th=[ 627], 5.00th=[ 709], 10.00th=[ 766], 20.00th=[ 963], | 30.00th=[ 1729], 40.00th=[ 4293], 50.00th=[ 7701], 60.00th=[ 10421], | 70.00th=[ 13042], 80.00th=[ 16909], 90.00th=[ 23725], 95.00th=[ 29754], | 99.00th=[ 47449], 99.50th=[ 54789], 99.90th=[101188], 99.95th=[104334], | 99.99th=[104334] bw ( KiB/s): min= 87, max= 335, per=24.47%, avg=213.16, stdev=63.44, samples=57 iops : min= 21, max= 83, avg=53.07, stdev=15.90, samples=57 lat (usec) : 100=4.65%, 250=6.90%, 500=0.55%, 750=4.69%, 1000=6.15% lat (msec) : 2=5.99%, 4=5.34%, 10=28.94%, 20=25.62%, 50=10.55% lat (msec) : 100=0.49%, 250=0.13% cpu : usr=0.16%, sys=14.30%, ctx=4845, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32531: Thu May 1 22:32:52 2025 read: IOPS=52, BW=212KiB/s (217kB/s)(6052KiB/28581msec) clat (usec): min=61, max=243117, avg=8323.09, stdev=10044.23 lat (usec): min=62, max=243119, avg=8324.89, stdev=10043.68 clat percentiles (usec): | 1.00th=[ 78], 5.00th=[ 92], 10.00th=[ 102], 20.00th=[ 133], | 30.00th=[ 4424], 40.00th=[ 5604], 50.00th=[ 6849], 60.00th=[ 8291], | 70.00th=[ 10290], 80.00th=[ 13173], 90.00th=[ 17695], 95.00th=[ 21627], | 99.00th=[ 31327], 99.50th=[ 35390], 99.90th=[128451], 99.95th=[242222], | 99.99th=[242222] bw ( KiB/s): min= 104, max= 400, per=25.34%, avg=210.84, stdev=63.36, samples=57 iops : min= 26, max= 100, avg=52.51, stdev=15.86, samples=57 write: IOPS=54, BW=218KiB/s (223kB/s)(6236KiB/28581msec) clat (usec): min=421, max=133278, avg=10165.27, stdev=11075.49 lat (usec): min=422, max=133279, avg=10167.87, stdev=11075.62 clat percentiles (usec): | 1.00th=[ 627], 5.00th=[ 725], 10.00th=[ 775], 20.00th=[ 1012], | 30.00th=[ 1893], 40.00th=[ 4424], 50.00th=[ 7635], 60.00th=[ 10028], | 70.00th=[ 12911], 80.00th=[ 16909], 90.00th=[ 24249], 95.00th=[ 30802], | 99.00th=[ 50594], 99.50th=[ 58983], 99.90th=[ 79168], 99.95th=[133694], | 99.99th=[133694] bw ( KiB/s): min= 95, max= 384, per=24.91%, avg=217.00, stdev=62.91, samples=57 iops : min= 23, max= 96, avg=54.05, stdev=15.72, samples=57 lat (usec) : 100=4.46%, 250=6.67%, 500=0.75%, 750=4.10%, 1000=6.25% lat (msec) : 2=5.73%, 4=5.79%, 10=30.37%, 20=25.10%, 50=10.09% lat (msec) : 100=0.59%, 250=0.10% cpu : usr=0.26%, sys=14.37%, ctx=4894, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=832KiB/s (852kB/s), 207KiB/s-212KiB/s (212kB/s-217kB/s), io=23.4MiB (24.6MB), run=28581-28850msec WRITE: bw=871KiB/s (892kB/s), 217KiB/s-222KiB/s (222kB/s-227kB/s), io=24.6MiB (25.7MB), run=28581-28850msec rand-rw: (groupid=0, jobs=1): err= 0: pid=32528: Thu May 1 22:33:57 2025 read: IOPS=16, BW=64.1KiB/s (65.7kB/s)(5992KiB/93423msec) clat (msec): min=2, max=247, avg= 9.37, stdev=10.52 lat (msec): min=2, max=247, avg= 9.38, stdev=10.52 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 4], 10.00th=[ 4], 20.00th=[ 5], | 30.00th=[ 6], 40.00th=[ 7], 50.00th=[ 8], 60.00th=[ 9], | 70.00th=[ 10], 80.00th=[ 12], 90.00th=[ 17], 95.00th=[ 23], | 99.00th=[ 47], 99.50th=[ 62], 99.90th=[ 130], 99.95th=[ 247], | 99.99th=[ 247] bw ( KiB/s): min= 7, max= 168, per=25.10%, avg=64.00, stdev=35.60, samples=185 iops : min= 1, max= 42, avg=15.77, stdev= 8.92, samples=185 write: IOPS=16, BW=67.4KiB/s (69.0kB/s)(6296KiB/93423msec) clat (msec): min=15, max=204, avg=50.35, stdev=22.64 lat (msec): min=15, max=204, avg=50.35, stdev=22.64 clat percentiles (msec): | 1.00th=[ 20], 5.00th=[ 26], 10.00th=[ 29], 20.00th=[ 33], | 30.00th=[ 37], 40.00th=[ 41], 50.00th=[ 45], 60.00th=[ 51], | 70.00th=[ 56], 80.00th=[ 65], 90.00th=[ 81], 95.00th=[ 94], | 99.00th=[ 132], 99.50th=[ 146], 99.90th=[ 174], 99.95th=[ 205], | 99.99th=[ 205] bw ( KiB/s): min= 8, max= 136, per=25.05%, avg=66.88, stdev=23.86, samples=186 iops : min= 2, max= 34, avg=16.49, stdev= 6.04, samples=186 lat (msec) : 4=7.85%, 10=27.57%, 20=10.58%, 50=33.11%, 100=18.85% lat (msec) : 250=2.05% cpu : usr=0.04%, sys=3.38%, ctx=3922, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32530: Thu May 1 22:33:57 2025 read: IOPS=15, BW=63.2KiB/s (64.7kB/s)(5940KiB/94036msec) clat (msec): min=2, max=246, avg= 8.98, stdev= 9.96 lat (msec): min=2, max=246, avg= 8.98, stdev= 9.96 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 4], 10.00th=[ 4], 20.00th=[ 5], | 30.00th=[ 5], 40.00th=[ 6], 50.00th=[ 8], 60.00th=[ 9], | 70.00th=[ 10], 80.00th=[ 12], 90.00th=[ 16], 95.00th=[ 22], | 99.00th=[ 44], 99.50th=[ 53], 99.90th=[ 131], 99.95th=[ 247], | 99.99th=[ 247] bw ( KiB/s): min= 8, max= 184, per=24.99%, avg=63.73, stdev=34.67, samples=184 iops : min= 2, max= 46, avg=15.76, stdev= 8.69, samples=184 write: IOPS=16, BW=67.5KiB/s (69.1kB/s)(6348KiB/94036msec) clat (msec): min=15, max=171, avg=50.76, stdev=23.34 lat (msec): min=15, max=171, avg=50.77, stdev=23.34 clat percentiles (msec): | 1.00th=[ 20], 5.00th=[ 25], 10.00th=[ 28], 20.00th=[ 33], | 30.00th=[ 37], 40.00th=[ 42], 50.00th=[ 46], 60.00th=[ 51], | 70.00th=[ 57], 80.00th=[ 64], 90.00th=[ 81], 95.00th=[ 103], | 99.00th=[ 133], 99.50th=[ 148], 99.90th=[ 159], 99.95th=[ 171], | 99.99th=[ 171] bw ( KiB/s): min= 16, max= 136, per=25.00%, avg=66.75, stdev=24.02, samples=187 iops : min= 4, max= 34, avg=16.51, stdev= 6.06, samples=187 lat (msec) : 4=8.69%, 10=27.38%, 20=10.09%, 50=32.52%, 100=18.52% lat (msec) : 250=2.80% cpu : usr=0.05%, sys=3.27%, ctx=3879, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32532: Thu May 1 22:33:57 2025 read: IOPS=16, BW=64.6KiB/s (66.2kB/s)(6028KiB/93243msec) clat (msec): min=2, max=153, avg= 9.28, stdev= 8.27 lat (msec): min=2, max=153, avg= 9.28, stdev= 8.27 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 4], 10.00th=[ 4], 20.00th=[ 5], | 30.00th=[ 6], 40.00th=[ 7], 50.00th=[ 8], 60.00th=[ 9], | 70.00th=[ 10], 80.00th=[ 12], 90.00th=[ 17], 95.00th=[ 24], | 99.00th=[ 38], 99.50th=[ 54], 99.90th=[ 96], 99.95th=[ 155], | 99.99th=[ 155] bw ( KiB/s): min= 7, max= 184, per=25.33%, avg=64.58, stdev=35.37, samples=185 iops : min= 1, max= 46, avg=15.92, stdev= 8.87, samples=185 write: IOPS=16, BW=67.1KiB/s (68.7kB/s)(6260KiB/93243msec) clat (msec): min=13, max=303, avg=50.55, stdev=23.40 lat (msec): min=13, max=303, avg=50.55, stdev=23.40 clat percentiles (msec): | 1.00th=[ 21], 5.00th=[ 25], 10.00th=[ 29], 20.00th=[ 33], | 30.00th=[ 37], 40.00th=[ 41], 50.00th=[ 46], 60.00th=[ 51], | 70.00th=[ 56], 80.00th=[ 65], 90.00th=[ 79], 95.00th=[ 100], | 99.00th=[ 130], 99.50th=[ 146], 99.90th=[ 176], 99.95th=[ 305], | 99.99th=[ 305] bw ( KiB/s): min= 15, max= 136, per=24.97%, avg=66.67, stdev=24.57, samples=186 iops : min= 3, max= 34, avg=16.44, stdev= 6.21, samples=186 lat (msec) : 4=7.81%, 10=27.31%, 20=10.94%, 50=32.58%, 100=18.91% lat (msec) : 250=2.41%, 500=0.03% cpu : usr=0.11%, sys=3.33%, ctx=3954, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32533: Thu May 1 22:33:57 2025 read: IOPS=16, BW=65.1KiB/s (66.7kB/s)(6052KiB/92918msec) clat (msec): min=2, max=154, avg= 9.23, stdev= 9.23 lat (msec): min=2, max=154, avg= 9.23, stdev= 9.23 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 4], 10.00th=[ 4], 20.00th=[ 5], | 30.00th=[ 5], 40.00th=[ 7], 50.00th=[ 7], 60.00th=[ 9], | 70.00th=[ 10], 80.00th=[ 12], 90.00th=[ 17], 95.00th=[ 24], | 99.00th=[ 46], 99.50th=[ 58], 99.90th=[ 114], 99.95th=[ 155], | 99.99th=[ 155] bw ( KiB/s): min= 7, max= 168, per=25.69%, avg=65.50, stdev=35.18, samples=183 iops : min= 1, max= 42, avg=16.19, stdev= 8.84, samples=183 write: IOPS=16, BW=67.1KiB/s (68.7kB/s)(6236KiB/92918msec) clat (msec): min=15, max=305, avg=50.55, stdev=23.56 lat (msec): min=15, max=305, avg=50.55, stdev=23.56 clat percentiles (msec): | 1.00th=[ 20], 5.00th=[ 26], 10.00th=[ 29], 20.00th=[ 33], | 30.00th=[ 37], 40.00th=[ 41], 50.00th=[ 45], 60.00th=[ 50], | 70.00th=[ 56], 80.00th=[ 64], 90.00th=[ 82], 95.00th=[ 96], | 99.00th=[ 132], 99.50th=[ 153], 99.90th=[ 178], 99.95th=[ 305], | 99.99th=[ 305] bw ( KiB/s): min= 15, max= 128, per=24.99%, avg=66.72, stdev=24.12, samples=185 iops : min= 3, max= 32, avg=16.49, stdev= 6.08, samples=185 lat (msec) : 4=8.07%, 10=28.39%, 20=10.06%, 50=33.01%, 100=18.29% lat (msec) : 250=2.15%, 500=0.03% cpu : usr=0.09%, sys=3.34%, ctx=3901, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=255KiB/s (261kB/s), 63.2KiB/s-65.1KiB/s (64.7kB/s-66.7kB/s), io=23.4MiB (24.6MB), run=92918-94036msec WRITE: bw=267KiB/s (274kB/s), 67.1KiB/s-67.5KiB/s (68.7kB/s-69.1kB/s), io=24.6MiB (25.7MB), run=92918-94036msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=32619: Thu May 1 22:34:03 2025 read: IOPS=81, BW=1304KiB/s (1335kB/s)(5920KiB/4541msec) clat (usec): min=58, max=41232, avg=5813.18, stdev=5907.81 lat (usec): min=58, max=41233, avg=5814.17, stdev=5907.85 clat percentiles (usec): | 1.00th=[ 75], 5.00th=[ 92], 10.00th=[ 99], 20.00th=[ 115], | 30.00th=[ 147], 40.00th=[ 4178], 50.00th=[ 5276], 60.00th=[ 6390], | 70.00th=[ 7570], 80.00th=[ 9634], 90.00th=[12387], 95.00th=[15008], | 99.00th=[32375], 99.50th=[33424], 99.90th=[41157], 99.95th=[41157], | 99.99th=[41157] bw ( KiB/s): min= 928, max= 1696, per=26.19%, avg=1300.22, stdev=252.59, samples=9 iops : min= 58, max= 106, avg=81.00, stdev=15.74, samples=9 write: IOPS=87, BW=1402KiB/s (1436kB/s)(6368KiB/4541msec) clat (usec): min=632, max=49222, avg=5925.50, stdev=7107.56 lat (usec): min=634, max=49223, avg=5926.98, stdev=7107.54 clat percentiles (usec): | 1.00th=[ 742], 5.00th=[ 881], 10.00th=[ 1037], 20.00th=[ 1188], | 30.00th=[ 1401], 40.00th=[ 1745], 50.00th=[ 2671], 60.00th=[ 4817], | 70.00th=[ 6652], 80.00th=[ 9503], 90.00th=[16057], 95.00th=[20317], | 99.00th=[34341], 99.50th=[37487], 99.90th=[49021], 99.95th=[49021], | 99.99th=[49021] bw ( KiB/s): min= 1120, max= 1852, per=26.87%, avg=1403.33, stdev=260.58, samples=9 iops : min= 70, max= 115, avg=87.44, stdev=16.20, samples=9 lat (usec) : 100=5.47%, 250=9.90%, 500=0.65%, 750=0.65%, 1000=4.04% lat (msec) : 2=17.97%, 4=9.51%, 10=33.72%, 20=14.19%, 50=3.91% cpu : usr=0.33%, sys=20.53%, ctx=1168, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32620: Thu May 1 22:34:03 2025 read: IOPS=78, BW=1259KiB/s (1289kB/s)(5744KiB/4563msec) clat (usec): min=49, max=27023, avg=5325.03, stdev=5524.57 lat (usec): min=49, max=27024, avg=5326.37, stdev=5524.12 clat percentiles (usec): | 1.00th=[ 53], 5.00th=[ 89], 10.00th=[ 98], 20.00th=[ 111], | 30.00th=[ 126], 40.00th=[ 355], 50.00th=[ 4948], 60.00th=[ 6194], | 70.00th=[ 7570], 80.00th=[10159], 90.00th=[13173], 95.00th=[16188], | 99.00th=[20841], 99.50th=[22938], 99.90th=[27132], 99.95th=[27132], | 99.99th=[27132] bw ( KiB/s): min= 768, max= 1632, per=25.26%, avg=1254.00, stdev=257.01, samples=9 iops : min= 48, max= 102, avg=78.11, stdev=16.01, samples=9 write: IOPS=89, BW=1434KiB/s (1469kB/s)(6544KiB/4563msec) clat (usec): min=578, max=46665, avg=6266.92, stdev=7029.43 lat (usec): min=579, max=46665, avg=6268.33, stdev=7029.44 clat percentiles (usec): | 1.00th=[ 791], 5.00th=[ 979], 10.00th=[ 1029], 20.00th=[ 1172], | 30.00th=[ 1352], 40.00th=[ 1827], 50.00th=[ 3097], 60.00th=[ 5145], | 70.00th=[ 8455], 80.00th=[10683], 90.00th=[15139], 95.00th=[20055], | 99.00th=[29754], 99.50th=[38011], 99.90th=[46924], 99.95th=[46924], | 99.99th=[46924] bw ( KiB/s): min= 1024, max= 1888, per=27.34%, avg=1428.11, stdev=298.89, samples=9 iops : min= 64, max= 118, avg=89.00, stdev=18.71, samples=9 lat (usec) : 50=0.13%, 100=5.73%, 250=12.24%, 500=1.04%, 750=0.39% lat (usec) : 1000=3.65% lat (msec) : 2=18.88%, 4=8.85%, 10=27.47%, 20=18.49%, 50=3.12% cpu : usr=0.22%, sys=19.09%, ctx=1142, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32621: Thu May 1 22:34:03 2025 read: IOPS=85, BW=1375KiB/s (1408kB/s)(6256KiB/4550msec) clat (usec): min=51, max=44778, avg=5231.65, stdev=5894.83 lat (usec): min=51, max=44779, avg=5232.61, stdev=5894.81 clat percentiles (usec): | 1.00th=[ 68], 5.00th=[ 88], 10.00th=[ 95], 20.00th=[ 106], | 30.00th=[ 121], 40.00th=[ 251], 50.00th=[ 4424], 60.00th=[ 5932], | 70.00th=[ 7635], 80.00th=[ 9896], 90.00th=[13566], 95.00th=[17171], | 99.00th=[19792], 99.50th=[26608], 99.90th=[44827], 99.95th=[44827], | 99.99th=[44827] bw ( KiB/s): min= 640, max= 1664, per=27.91%, avg=1385.56, stdev=307.19, samples=9 iops : min= 40, max= 104, avg=86.44, stdev=19.12, samples=9 write: IOPS=82, BW=1326KiB/s (1358kB/s)(6032KiB/4550msec) clat (usec): min=653, max=45508, avg=6220.21, stdev=6814.28 lat (usec): min=654, max=45509, avg=6221.58, stdev=6814.23 clat percentiles (usec): | 1.00th=[ 766], 5.00th=[ 963], 10.00th=[ 1057], 20.00th=[ 1205], | 30.00th=[ 1450], 40.00th=[ 2073], 50.00th=[ 3195], 60.00th=[ 5276], | 70.00th=[ 7504], 80.00th=[10159], 90.00th=[16450], 95.00th=[20317], | 99.00th=[30016], 99.50th=[38536], 99.90th=[45351], 99.95th=[45351], | 99.99th=[45351] bw ( KiB/s): min= 800, max= 1728, per=25.37%, avg=1325.33, stdev=296.60, samples=9 iops : min= 50, max= 108, avg=82.67, stdev=18.58, samples=9 lat (usec) : 100=7.29%, 250=13.02%, 500=1.69%, 750=0.65%, 1000=3.12% lat (msec) : 2=16.02%, 4=8.72%, 10=29.56%, 20=16.93%, 50=2.99% cpu : usr=0.44%, sys=19.08%, ctx=1102, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32624: Thu May 1 22:34:03 2025 read: IOPS=78, BW=1250KiB/s (1280kB/s)(6032KiB/4824msec) clat (usec): min=62, max=66443, avg=5074.46, stdev=6618.71 lat (usec): min=63, max=66444, avg=5075.49, stdev=6618.74 clat percentiles (usec): | 1.00th=[ 74], 5.00th=[ 89], 10.00th=[ 94], 20.00th=[ 110], | 30.00th=[ 122], 40.00th=[ 176], 50.00th=[ 4359], 60.00th=[ 5276], | 70.00th=[ 7439], 80.00th=[ 9372], 90.00th=[11863], 95.00th=[14353], | 99.00th=[22414], 99.50th=[57410], 99.90th=[66323], 99.95th=[66323], | 99.99th=[66323] bw ( KiB/s): min= 224, max= 1600, per=23.91%, avg=1187.22, stdev=458.62, samples=9 iops : min= 14, max= 100, avg=74.11, stdev=28.60, samples=9 write: IOPS=81, BW=1297KiB/s (1328kB/s)(6256KiB/4824msec) clat (usec): min=659, max=87810, avg=7026.55, stdev=9689.35 lat (usec): min=660, max=87812, avg=7028.00, stdev=9689.35 clat percentiles (usec): | 1.00th=[ 758], 5.00th=[ 947], 10.00th=[ 1057], 20.00th=[ 1188], | 30.00th=[ 1532], 40.00th=[ 2057], 50.00th=[ 3458], 60.00th=[ 5407], | 70.00th=[ 8094], 80.00th=[10814], 90.00th=[15533], 95.00th=[22938], | 99.00th=[53216], 99.50th=[67634], 99.90th=[87557], 99.95th=[87557], | 99.99th=[87557] bw ( KiB/s): min= 352, max= 1884, per=24.02%, avg=1254.67, stdev=444.94, samples=9 iops : min= 22, max= 117, avg=78.33, stdev=27.68, samples=9 lat (usec) : 100=6.51%, 250=13.80%, 500=1.04%, 750=0.39%, 1000=3.65% lat (msec) : 2=16.54%, 4=8.20%, 10=29.69%, 20=16.28%, 50=2.99% lat (msec) : 100=0.91% cpu : usr=0.39%, sys=18.99%, ctx=1076, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=4965KiB/s (5084kB/s), 1250KiB/s-1375KiB/s (1280kB/s-1408kB/s), io=23.4MiB (24.5MB), run=4541-4824msec WRITE: bw=5224KiB/s (5349kB/s), 1297KiB/s-1434KiB/s (1328kB/s-1469kB/s), io=24.6MiB (25.8MB), run=4541-4824msec rand-rw: (groupid=0, jobs=1): err= 0: pid=32622: Thu May 1 22:34:22 2025 read: IOPS=15, BW=248KiB/s (254kB/s)(5920KiB/23876msec) clat (usec): min=2663, max=63830, avg=7707.74, stdev=6319.21 lat (usec): min=2664, max=63830, avg=7709.23, stdev=6319.12 clat percentiles (usec): | 1.00th=[ 2933], 5.00th=[ 3458], 10.00th=[ 3687], 20.00th=[ 4228], | 30.00th=[ 4555], 40.00th=[ 5080], 50.00th=[ 5669], 60.00th=[ 6456], | 70.00th=[ 7635], 80.00th=[ 9110], 90.00th=[14091], 95.00th=[19006], | 99.00th=[32375], 99.50th=[50594], 99.90th=[63701], 99.95th=[63701], | 99.99th=[63701] bw ( KiB/s): min= 32, max= 512, per=24.50%, avg=243.55, stdev=115.05, samples=47 iops : min= 2, max= 32, avg=15.06, stdev= 7.16, samples=47 write: IOPS=16, BW=267KiB/s (273kB/s)(6368KiB/23876msec) clat (msec): min=16, max=130, avg=52.70, stdev=18.05 lat (msec): min=16, max=130, avg=52.70, stdev=18.05 clat percentiles (msec): | 1.00th=[ 24], 5.00th=[ 28], 10.00th=[ 32], 20.00th=[ 37], | 30.00th=[ 43], 40.00th=[ 46], 50.00th=[ 52], 60.00th=[ 55], | 70.00th=[ 60], 80.00th=[ 66], 90.00th=[ 78], 95.00th=[ 87], | 99.00th=[ 109], 99.50th=[ 110], 99.90th=[ 131], 99.95th=[ 131], | 99.99th=[ 131] bw ( KiB/s): min= 159, max= 416, per=25.24%, avg=263.98, stdev=63.16, samples=47 iops : min= 9, max= 26, avg=16.34, stdev= 3.99, samples=47 lat (msec) : 4=8.07%, 10=32.03%, 20=6.12%, 50=25.91%, 100=26.69% lat (msec) : 250=1.17% cpu : usr=0.03%, sys=3.40%, ctx=867, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32623: Thu May 1 22:34:22 2025 read: IOPS=14, BW=238KiB/s (244kB/s)(5744KiB/24088msec) clat (usec): min=2505, max=60402, avg=7777.76, stdev=7696.30 lat (usec): min=2506, max=60403, avg=7780.19, stdev=7696.46 clat percentiles (usec): | 1.00th=[ 2737], 5.00th=[ 3326], 10.00th=[ 3621], 20.00th=[ 4080], | 30.00th=[ 4555], 40.00th=[ 4948], 50.00th=[ 5473], 60.00th=[ 5997], | 70.00th=[ 6783], 80.00th=[ 8356], 90.00th=[13042], 95.00th=[22938], | 99.00th=[49546], 99.50th=[53740], 99.90th=[60556], 99.95th=[60556], | 99.99th=[60556] bw ( KiB/s): min= 64, max= 576, per=23.99%, avg=238.46, stdev=123.15, samples=48 iops : min= 4, max= 36, avg=14.73, stdev= 7.72, samples=48 write: IOPS=16, BW=272KiB/s (278kB/s)(6544KiB/24088msec) clat (msec): min=16, max=126, avg=51.69, stdev=18.99 lat (msec): min=16, max=126, avg=51.69, stdev=18.99 clat percentiles (msec): | 1.00th=[ 22], 5.00th=[ 27], 10.00th=[ 29], 20.00th=[ 35], | 30.00th=[ 40], 40.00th=[ 47], 50.00th=[ 50], 60.00th=[ 55], | 70.00th=[ 60], 80.00th=[ 66], 90.00th=[ 77], 95.00th=[ 87], | 99.00th=[ 112], 99.50th=[ 118], 99.90th=[ 127], 99.95th=[ 127], | 99.99th=[ 127] bw ( KiB/s): min= 95, max= 481, per=25.86%, avg=270.52, stdev=77.83, samples=48 iops : min= 5, max= 30, avg=16.75, stdev= 4.84, samples=48 lat (msec) : 4=8.85%, 10=30.73%, 20=4.82%, 50=28.78%, 100=25.65% lat (msec) : 250=1.17% cpu : usr=0.05%, sys=3.43%, ctx=898, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32625: Thu May 1 22:34:22 2025 read: IOPS=16, BW=272KiB/s (278kB/s)(6256KiB/23035msec) clat (usec): min=2926, max=63082, avg=7792.89, stdev=6928.31 lat (usec): min=2927, max=63083, avg=7795.00, stdev=6928.02 clat percentiles (usec): | 1.00th=[ 3163], 5.00th=[ 3589], 10.00th=[ 3818], 20.00th=[ 4293], | 30.00th=[ 4883], 40.00th=[ 5276], 50.00th=[ 5800], 60.00th=[ 6390], | 70.00th=[ 7308], 80.00th=[ 8291], 90.00th=[12780], 95.00th=[21365], | 99.00th=[41681], 99.50th=[55313], 99.90th=[63177], 99.95th=[63177], | 99.99th=[63177] bw ( KiB/s): min= 64, max= 672, per=27.34%, avg=271.72, stdev=143.22, samples=46 iops : min= 4, max= 42, avg=16.72, stdev= 8.95, samples=46 write: IOPS=16, BW=262KiB/s (268kB/s)(6032KiB/23035msec) clat (msec): min=19, max=164, avg=52.94, stdev=20.38 lat (msec): min=19, max=164, avg=52.94, stdev=20.38 clat percentiles (msec): | 1.00th=[ 24], 5.00th=[ 28], 10.00th=[ 31], 20.00th=[ 37], | 30.00th=[ 41], 40.00th=[ 45], 50.00th=[ 50], 60.00th=[ 55], | 70.00th=[ 62], 80.00th=[ 67], 90.00th=[ 81], 95.00th=[ 93], | 99.00th=[ 129], 99.50th=[ 134], 99.90th=[ 165], 99.95th=[ 165], | 99.99th=[ 165] bw ( KiB/s): min= 128, max= 384, per=24.98%, avg=261.28, stdev=67.58, samples=46 iops : min= 8, max= 24, avg=16.07, stdev= 4.26, samples=46 lat (msec) : 4=7.03%, 10=36.72%, 20=4.69%, 50=27.86%, 100=22.27% lat (msec) : 250=1.43% cpu : usr=0.07%, sys=3.39%, ctx=879, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32626: Thu May 1 22:34:22 2025 read: IOPS=15, BW=256KiB/s (262kB/s)(6032KiB/23590msec) clat (usec): min=2695, max=73275, avg=7439.64, stdev=6951.59 lat (usec): min=2696, max=73276, avg=7441.18, stdev=6951.52 clat percentiles (usec): | 1.00th=[ 2999], 5.00th=[ 3556], 10.00th=[ 3785], 20.00th=[ 4228], | 30.00th=[ 4686], 40.00th=[ 5145], 50.00th=[ 5735], 60.00th=[ 6325], | 70.00th=[ 7242], 80.00th=[ 8356], 90.00th=[10945], 95.00th=[15533], | 99.00th=[43779], 99.50th=[60556], 99.90th=[72877], 99.95th=[72877], | 99.99th=[72877] bw ( KiB/s): min= 64, max= 448, per=25.46%, avg=253.09, stdev=98.52, samples=47 iops : min= 4, max= 28, avg=15.64, stdev= 6.20, samples=47 write: IOPS=16, BW=265KiB/s (272kB/s)(6256KiB/23590msec) clat (msec): min=22, max=117, avg=52.88, stdev=18.50 lat (msec): min=22, max=117, avg=52.89, stdev=18.50 clat percentiles (msec): | 1.00th=[ 24], 5.00th=[ 30], 10.00th=[ 32], 20.00th=[ 37], | 30.00th=[ 42], 40.00th=[ 46], 50.00th=[ 51], 60.00th=[ 54], | 70.00th=[ 61], 80.00th=[ 67], 90.00th=[ 79], 95.00th=[ 89], | 99.00th=[ 113], 99.50th=[ 115], 99.90th=[ 117], 99.95th=[ 117], | 99.99th=[ 117] bw ( KiB/s): min= 96, max= 416, per=25.30%, avg=264.64, stdev=72.49, samples=47 iops : min= 6, max= 26, avg=16.36, stdev= 4.63, samples=47 lat (msec) : 4=6.38%, 10=36.59%, 20=4.43%, 50=26.95%, 100=24.61% lat (msec) : 250=1.04% cpu : usr=0.00%, sys=3.52%, ctx=883, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=994KiB/s (1018kB/s), 238KiB/s-272KiB/s (244kB/s-278kB/s), io=23.4MiB (24.5MB), run=23035-24088msec WRITE: bw=1046KiB/s (1071kB/s), 262KiB/s-272KiB/s (268kB/s-278kB/s), io=24.6MiB (25.8MB), run=23035-24088msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=32653: Thu May 1 22:34:24 2025 read: IOPS=8, BW=8615KiB/s (8822kB/s)(7168KiB/832msec) clat (usec): min=881, max=152065, avg=59203.42, stdev=46383.55 lat (usec): min=883, max=152067, avg=59205.21, stdev=46383.37 clat percentiles (usec): | 1.00th=[ 881], 5.00th=[ 881], 10.00th=[ 881], 20.00th=[ 39060], | 30.00th=[ 46924], 40.00th=[ 46924], 50.00th=[ 47449], 60.00th=[ 54789], | 70.00th=[ 54789], 80.00th=[ 72877], 90.00th=[152044], 95.00th=[152044], | 99.00th=[152044], 99.50th=[152044], 99.90th=[152044], 99.95th=[152044], | 99.99th=[152044] bw ( KiB/s): min=10118, max=10118, per=53.61%, avg=10118.00, stdev= 0.00, samples=1 iops : min= 9, max= 9, avg= 9.00, stdev= 0.00, samples=1 write: IOPS=6, BW=6154KiB/s (6302kB/s)(5120KiB/832msec) clat (msec): min=24, max=119, avg=68.69, stdev=35.50 lat (msec): min=25, max=119, avg=68.74, stdev=35.48 clat percentiles (msec): | 1.00th=[ 26], 5.00th=[ 26], 10.00th=[ 26], 20.00th=[ 26], | 30.00th=[ 51], 40.00th=[ 51], 50.00th=[ 66], 60.00th=[ 66], | 70.00th=[ 84], 80.00th=[ 84], 90.00th=[ 120], 95.00th=[ 120], | 99.00th=[ 120], 99.50th=[ 120], 99.90th=[ 120], 99.95th=[ 120], | 99.99th=[ 120] bw ( KiB/s): min= 2023, max= 2023, per=7.66%, avg=2023.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (usec) : 1000=8.33% lat (msec) : 50=41.67%, 100=33.33%, 250=16.67% cpu : usr=0.00%, sys=29.00%, ctx=451, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32654: Thu May 1 22:34:24 2025 read: IOPS=2, BW=3012KiB/s (3084kB/s)(2048KiB/680msec) clat (usec): min=36369, max=46131, avg=41250.53, stdev=6902.96 lat (usec): min=36370, max=46134, avg=41252.50, stdev=6903.60 clat percentiles (usec): | 1.00th=[36439], 5.00th=[36439], 10.00th=[36439], 20.00th=[36439], | 30.00th=[36439], 40.00th=[36439], 50.00th=[36439], 60.00th=[45876], | 70.00th=[45876], 80.00th=[45876], 90.00th=[45876], 95.00th=[45876], | 99.00th=[45876], 99.50th=[45876], 99.90th=[45876], 99.95th=[45876], | 99.99th=[45876] bw ( KiB/s): min= 4079, max= 4079, per=21.61%, avg=4079.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 write: IOPS=14, BW=14.7MiB/s (15.4MB/s)(10.0MiB/680msec) clat (usec): min=29209, max=93284, avg=58306.63, stdev=23115.28 lat (usec): min=29285, max=93352, avg=58382.22, stdev=23116.85 clat percentiles (usec): | 1.00th=[29230], 5.00th=[29230], 10.00th=[29230], 20.00th=[32637], | 30.00th=[41157], 40.00th=[43254], 50.00th=[52167], 60.00th=[57934], | 70.00th=[81265], 80.00th=[81265], 90.00th=[89654], 95.00th=[92799], | 99.00th=[92799], 99.50th=[92799], 99.90th=[92799], 99.95th=[92799], | 99.99th=[92799] bw ( KiB/s): min=12239, max=12239, per=46.32%, avg=12239.00, stdev= 0.00, samples=1 iops : min= 11, max= 11, avg=11.00, stdev= 0.00, samples=1 lat (msec) : 50=50.00%, 100=50.00% cpu : usr=0.00%, sys=35.94%, ctx=298, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32655: Thu May 1 22:34:24 2025 read: IOPS=6, BW=7078KiB/s (7248kB/s)(6144KiB/868msec) clat (usec): min=556, max=145216, avg=45929.74, stdev=57990.80 lat (usec): min=557, max=145217, avg=45931.35, stdev=57990.84 clat percentiles (usec): | 1.00th=[ 553], 5.00th=[ 553], 10.00th=[ 553], 20.00th=[ 578], | 30.00th=[ 578], 40.00th=[ 603], 50.00th=[ 603], 60.00th=[ 56886], | 70.00th=[ 71828], 80.00th=[ 71828], 90.00th=[145753], 95.00th=[145753], | 99.00th=[145753], 99.50th=[145753], 99.90th=[145753], 99.95th=[145753], | 99.99th=[145753] bw ( KiB/s): min= 2035, max= 2035, per=10.78%, avg=2035.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=6, BW=7078KiB/s (7248kB/s)(6144KiB/868msec) clat (msec): min=38, max=248, avg=92.16, stdev=80.40 lat (msec): min=38, max=248, avg=92.20, stdev=80.40 clat percentiles (msec): | 1.00th=[ 39], 5.00th=[ 39], 10.00th=[ 39], 20.00th=[ 40], | 30.00th=[ 40], 40.00th=[ 51], 50.00th=[ 51], 60.00th=[ 78], | 70.00th=[ 100], 80.00th=[ 100], 90.00th=[ 249], 95.00th=[ 249], | 99.00th=[ 249], 99.50th=[ 249], 99.90th=[ 249], 99.95th=[ 249], | 99.99th=[ 249] bw ( KiB/s): min= 6107, max= 6107, per=23.11%, avg=6107.00, stdev= 0.00, samples=1 iops : min= 5, max= 5, avg= 5.00, stdev= 0.00, samples=1 lat (usec) : 750=25.00% lat (msec) : 50=16.67%, 100=41.67%, 250=16.67% cpu : usr=0.00%, sys=23.41%, ctx=446, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32656: Thu May 1 22:34:24 2025 read: IOPS=4, BW=4719KiB/s (4832kB/s)(5120KiB/1085msec) clat (usec): min=450, max=54254, avg=37041.26, stdev=20984.81 lat (usec): min=451, max=54256, avg=37042.62, stdev=20984.66 clat percentiles (usec): | 1.00th=[ 449], 5.00th=[ 449], 10.00th=[ 449], 20.00th=[ 449], | 30.00th=[42730], 40.00th=[42730], 50.00th=[43779], 60.00th=[43779], | 70.00th=[44303], 80.00th=[44303], 90.00th=[54264], 95.00th=[54264], | 99.00th=[54264], 99.50th=[54264], 99.90th=[54264], 99.95th=[54264], | 99.99th=[54264] bw ( KiB/s): min= 8192, max= 8192, per=43.40%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=6, BW=6606KiB/s (6765kB/s)(7168KiB/1085msec) clat (msec): min=48, max=237, avg=111.53, stdev=64.57 lat (msec): min=48, max=237, avg=111.59, stdev=64.58 clat percentiles (msec): | 1.00th=[ 49], 5.00th=[ 49], 10.00th=[ 49], 20.00th=[ 69], | 30.00th=[ 85], 40.00th=[ 85], 50.00th=[ 93], 60.00th=[ 94], | 70.00th=[ 94], 80.00th=[ 155], 90.00th=[ 239], 95.00th=[ 239], | 99.00th=[ 239], 99.50th=[ 239], 99.90th=[ 239], 99.95th=[ 239], | 99.99th=[ 239] bw ( KiB/s): min= 4096, max= 8192, per=23.25%, avg=6144.00, stdev=2896.31, samples=2 iops : min= 4, max= 8, avg= 6.00, stdev= 2.83, samples=2 lat (usec) : 500=8.33% lat (msec) : 50=33.33%, 100=41.67%, 250=16.67% cpu : usr=0.09%, sys=21.49%, ctx=450, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=18.4MiB/s (19.3MB/s), 3012KiB/s-8615KiB/s (3084kB/s-8822kB/s), io=20.0MiB (20.0MB), run=680-1085msec WRITE: bw=25.8MiB/s (27.1MB/s), 6154KiB/s-14.7MiB/s (6302kB/s-15.4MB/s), io=28.0MiB (29.4MB), run=680-1085msec rand-rw: (groupid=0, jobs=1): err= 0: pid=32657: Thu May 1 22:34:24 2025 read: IOPS=6, BW=6906KiB/s (7071kB/s)(7168KiB/1038msec) clat (msec): min=10, max=160, avg=56.64, stdev=51.53 lat (msec): min=10, max=160, avg=56.64, stdev=51.53 clat percentiles (msec): | 1.00th=[ 11], 5.00th=[ 11], 10.00th=[ 11], 20.00th=[ 29], | 30.00th=[ 34], 40.00th=[ 34], 50.00th=[ 37], 60.00th=[ 40], | 70.00th=[ 40], 80.00th=[ 88], 90.00th=[ 161], 95.00th=[ 161], | 99.00th=[ 161], 99.50th=[ 161], 99.90th=[ 161], 99.95th=[ 161], | 99.99th=[ 161] bw ( KiB/s): min= 6131, max= 6131, per=34.82%, avg=6131.00, stdev= 0.00, samples=1 iops : min= 5, max= 5, avg= 5.00, stdev= 0.00, samples=1 write: IOPS=4, BW=4933KiB/s (5051kB/s)(5120KiB/1038msec) clat (msec): min=53, max=157, avg=108.62, stdev=47.80 lat (msec): min=53, max=157, avg=108.72, stdev=47.82 clat percentiles (msec): | 1.00th=[ 54], 5.00th=[ 54], 10.00th=[ 54], 20.00th=[ 54], | 30.00th=[ 68], 40.00th=[ 68], 50.00th=[ 110], 60.00th=[ 110], | 70.00th=[ 155], 80.00th=[ 155], 90.00th=[ 159], 95.00th=[ 159], | 99.00th=[ 159], 99.50th=[ 159], 99.90th=[ 159], 99.95th=[ 159], | 99.99th=[ 159] bw ( KiB/s): min= 2043, max= 2043, per=8.29%, avg=2043.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 20=8.33%, 50=33.33%, 100=25.00%, 250=33.33% cpu : usr=0.29%, sys=4.73%, ctx=37, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32658: Thu May 1 22:34:24 2025 read: IOPS=1, BW=1761KiB/s (1803kB/s)(2048KiB/1163msec) clat (usec): min=18441, max=32788, avg=25615.29, stdev=10145.12 lat (usec): min=18442, max=32791, avg=25616.85, stdev=10146.11 clat percentiles (usec): | 1.00th=[18482], 5.00th=[18482], 10.00th=[18482], 20.00th=[18482], | 30.00th=[18482], 40.00th=[18482], 50.00th=[18482], 60.00th=[32900], | 70.00th=[32900], 80.00th=[32900], 90.00th=[32900], 95.00th=[32900], | 99.00th=[32900], 99.50th=[32900], 99.90th=[32900], 99.95th=[32900], | 99.99th=[32900] bw ( KiB/s): min= 1796, max= 2043, per=10.90%, avg=1919.50, stdev=174.66, samples=2 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=2 write: IOPS=8, BW=8805KiB/s (9016kB/s)(10.0MiB/1163msec) clat (msec): min=40, max=212, avg=97.33, stdev=50.54 lat (msec): min=40, max=212, avg=97.40, stdev=50.54 clat percentiles (msec): | 1.00th=[ 41], 5.00th=[ 41], 10.00th=[ 41], 20.00th=[ 55], | 30.00th=[ 66], 40.00th=[ 70], 50.00th=[ 75], 60.00th=[ 84], | 70.00th=[ 130], 80.00th=[ 130], 90.00th=[ 130], 95.00th=[ 213], | 99.00th=[ 213], 99.50th=[ 213], 99.90th=[ 213], 99.95th=[ 213], | 99.99th=[ 213] bw ( KiB/s): min= 4087, max=10778, per=30.15%, avg=7432.50, stdev=4731.25, samples=2 iops : min= 3, max= 10, avg= 6.50, stdev= 4.95, samples=2 lat (msec) : 20=8.33%, 50=16.67%, 100=41.67%, 250=33.33% cpu : usr=0.00%, sys=5.68%, ctx=37, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32659: Thu May 1 22:34:24 2025 read: IOPS=5, BW=5366KiB/s (5495kB/s)(6144KiB/1145msec) clat (msec): min=10, max=127, avg=45.37, stdev=47.11 lat (msec): min=10, max=127, avg=45.37, stdev=47.11 clat percentiles (msec): | 1.00th=[ 11], 5.00th=[ 11], 10.00th=[ 11], 20.00th=[ 11], | 30.00th=[ 11], 40.00th=[ 14], 50.00th=[ 14], 60.00th=[ 36], | 70.00th=[ 74], 80.00th=[ 74], 90.00th=[ 128], 95.00th=[ 128], | 99.00th=[ 128], 99.50th=[ 128], 99.90th=[ 128], 99.95th=[ 128], | 99.99th=[ 128] bw ( KiB/s): min= 8192, max= 8192, per=46.52%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=5, BW=5366KiB/s (5495kB/s)(6144KiB/1145msec) clat (msec): min=36, max=294, avg=136.95, stdev=90.10 lat (msec): min=36, max=294, avg=137.03, stdev=90.10 clat percentiles (msec): | 1.00th=[ 37], 5.00th=[ 37], 10.00th=[ 37], 20.00th=[ 88], | 30.00th=[ 88], 40.00th=[ 91], 50.00th=[ 91], 60.00th=[ 140], | 70.00th=[ 174], 80.00th=[ 174], 90.00th=[ 296], 95.00th=[ 296], | 99.00th=[ 296], 99.50th=[ 296], 99.90th=[ 296], 99.95th=[ 296], | 99.99th=[ 296] bw ( KiB/s): min= 4087, max= 4096, per=16.60%, avg=4091.50, stdev= 6.36, samples=2 iops : min= 3, max= 4, avg= 3.50, stdev= 0.71, samples=2 lat (msec) : 20=25.00%, 50=16.67%, 100=25.00%, 250=25.00%, 500=8.33% cpu : usr=0.09%, sys=6.56%, ctx=117, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32660: Thu May 1 22:34:24 2025 read: IOPS=5, BW=5559KiB/s (5693kB/s)(5120KiB/921msec) clat (usec): min=11593, max=85270, avg=38219.89, stdev=27871.20 lat (usec): min=11595, max=85272, avg=38221.71, stdev=27871.19 clat percentiles (usec): | 1.00th=[11600], 5.00th=[11600], 10.00th=[11600], 20.00th=[11600], | 30.00th=[27919], 40.00th=[27919], 50.00th=[29492], 60.00th=[29492], | 70.00th=[36963], 80.00th=[36963], 90.00th=[85459], 95.00th=[85459], | 99.00th=[85459], 99.50th=[85459], 99.90th=[85459], 99.95th=[85459], | 99.99th=[85459] bw ( KiB/s): min= 8175, max= 8175, per=46.43%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 7, max= 7, avg= 7.00, stdev= 0.00, samples=1 write: IOPS=7, BW=7783KiB/s (7970kB/s)(7168KiB/921msec) clat (msec): min=67, max=210, avg=100.75, stdev=50.11 lat (msec): min=67, max=210, avg=100.83, stdev=50.10 clat percentiles (msec): | 1.00th=[ 68], 5.00th=[ 68], 10.00th=[ 68], 20.00th=[ 74], | 30.00th=[ 74], 40.00th=[ 74], 50.00th=[ 81], 60.00th=[ 93], | 70.00th=[ 93], 80.00th=[ 107], 90.00th=[ 211], 95.00th=[ 211], | 99.00th=[ 211], 99.50th=[ 211], 99.90th=[ 211], 99.95th=[ 211], | 99.99th=[ 211] bw ( KiB/s): min= 8175, max= 8175, per=33.16%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 7, max= 7, avg= 7.00, stdev= 0.00, samples=1 lat (msec) : 20=8.33%, 50=25.00%, 100=50.00%, 250=16.67% cpu : usr=0.00%, sys=6.63%, ctx=31, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=17.2MiB/s (18.0MB/s), 1761KiB/s-6906KiB/s (1803kB/s-7071kB/s), io=20.0MiB (20.0MB), run=921-1163msec WRITE: bw=24.1MiB/s (25.2MB/s), 4933KiB/s-8805KiB/s (5051kB/s-9016kB/s), io=28.0MiB (29.4MB), run=921-1163msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=32667: Thu May 1 22:34:26 2025 read: IOPS=4, BW=16.6MiB/s (17.4MB/s)(12.0MiB/724msec) clat (msec): min=182, max=294, avg=229.02, stdev=58.53 lat (msec): min=182, max=294, avg=229.02, stdev=58.53 clat percentiles (msec): | 1.00th=[ 182], 5.00th=[ 182], 10.00th=[ 182], 20.00th=[ 182], | 30.00th=[ 182], 40.00th=[ 211], 50.00th=[ 211], 60.00th=[ 211], | 70.00th=[ 296], 80.00th=[ 296], 90.00th=[ 296], 95.00th=[ 296], | 99.00th=[ 296], 99.50th=[ 296], 99.90th=[ 296], 99.95th=[ 296], | 99.99th=[ 296] bw ( KiB/s): min=16351, max=16351, per=91.52%, avg=16351.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.14%, sys=2.63%, ctx=20, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32668: Thu May 1 22:34:26 2025 read: IOPS=1, BW=4467KiB/s (4574kB/s)(4096KiB/917msec) clat (nsec): min=209687k, max=209687k, avg=209687144.00, stdev= 0.00 lat (nsec): min=209690k, max=209690k, avg=209689567.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 209], 5.00th=[ 209], 10.00th=[ 209], 20.00th=[ 209], | 30.00th=[ 209], 40.00th=[ 209], 50.00th=[ 209], 60.00th=[ 209], | 70.00th=[ 209], 80.00th=[ 209], 90.00th=[ 209], 95.00th=[ 209], | 99.00th=[ 209], 99.50th=[ 209], 99.90th=[ 209], 99.95th=[ 209], | 99.99th=[ 209] bw ( KiB/s): min= 8192, max= 8192, per=45.85%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=2, BW=8933KiB/s (9148kB/s)(8192KiB/917msec) clat (msec): min=103, max=549, avg=326.41, stdev=315.51 lat (msec): min=103, max=549, avg=326.55, stdev=315.52 clat percentiles (msec): | 1.00th=[ 104], 5.00th=[ 104], 10.00th=[ 104], 20.00th=[ 104], | 30.00th=[ 104], 40.00th=[ 104], 50.00th=[ 104], 60.00th=[ 550], | 70.00th=[ 550], 80.00th=[ 550], 90.00th=[ 550], 95.00th=[ 550], | 99.00th=[ 550], 99.50th=[ 550], 99.90th=[ 550], 99.95th=[ 550], | 99.99th=[ 550] lat (msec) : 250=66.67%, 750=33.33% cpu : usr=0.33%, sys=6.55%, ctx=144, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32670: Thu May 1 22:34:26 2025 write: IOPS=4, BW=18.5MiB/s (19.4MB/s)(12.0MiB/650msec) clat (msec): min=105, max=416, avg=211.89, stdev=176.97 lat (msec): min=107, max=416, avg=212.67, stdev=176.53 clat percentiles (msec): | 1.00th=[ 107], 5.00th=[ 107], 10.00th=[ 107], 20.00th=[ 107], | 30.00th=[ 107], 40.00th=[ 114], 50.00th=[ 114], 60.00th=[ 114], | 70.00th=[ 418], 80.00th=[ 418], 90.00th=[ 418], 95.00th=[ 418], | 99.00th=[ 418], 99.50th=[ 418], 99.90th=[ 418], 99.95th=[ 418], | 99.99th=[ 418] bw ( KiB/s): min=16384, max=16384, per=45.85%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.46%, sys=7.40%, ctx=70, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32672: Thu May 1 22:34:26 2025 write: IOPS=4, BW=18.2MiB/s (19.1MB/s)(12.0MiB/658msec) clat (msec): min=116, max=326, avg=208.90, stdev=107.19 lat (msec): min=117, max=326, avg=209.18, stdev=107.13 clat percentiles (msec): | 1.00th=[ 117], 5.00th=[ 117], 10.00th=[ 117], 20.00th=[ 117], | 30.00th=[ 117], 40.00th=[ 184], 50.00th=[ 184], 60.00th=[ 184], | 70.00th=[ 326], 80.00th=[ 326], 90.00th=[ 326], 95.00th=[ 326], | 99.00th=[ 326], 99.50th=[ 326], 99.90th=[ 326], 99.95th=[ 326], | 99.99th=[ 326] bw ( KiB/s): min=16384, max=16384, per=45.85%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.00%, sys=8.52%, ctx=75, majf=0, minf=28 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=17.4MiB/s (18.3MB/s), 4467KiB/s-16.6MiB/s (4574kB/s-17.4MB/s), io=16.0MiB (16.8MB), run=724-917msec WRITE: bw=34.9MiB/s (36.6MB/s), 8933KiB/s-18.5MiB/s (9148kB/s-19.4MB/s), io=32.0MiB (33.6MB), run=650-917msec rand-rw: (groupid=0, jobs=1): err= 0: pid=32669: Thu May 1 22:34:26 2025 read: IOPS=5, BW=23.0MiB/s (25.1MB/s)(12.0MiB/501msec) clat (msec): min=55, max=213, avg=159.72, stdev=90.66 lat (msec): min=55, max=213, avg=159.72, stdev=90.66 clat percentiles (msec): | 1.00th=[ 55], 5.00th=[ 55], 10.00th=[ 55], 20.00th=[ 55], | 30.00th=[ 55], 40.00th=[ 211], 50.00th=[ 211], 60.00th=[ 211], | 70.00th=[ 213], 80.00th=[ 213], 90.00th=[ 213], 95.00th=[ 213], | 99.00th=[ 213], 99.50th=[ 213], 99.90th=[ 213], 99.95th=[ 213], | 99.99th=[ 213] lat (msec) : 100=33.33%, 250=66.67% cpu : usr=0.20%, sys=32.60%, ctx=1204, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32671: Thu May 1 22:34:26 2025 read: IOPS=1, BW=5178KiB/s (5303kB/s)(4096KiB/791msec) clat (nsec): min=119366k, max=119366k, avg=119365599.00, stdev= 0.00 lat (nsec): min=119369k, max=119369k, avg=119368729.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 120], 5.00th=[ 120], 10.00th=[ 120], 20.00th=[ 120], | 30.00th=[ 120], 40.00th=[ 120], 50.00th=[ 120], 60.00th=[ 120], | 70.00th=[ 120], 80.00th=[ 120], 90.00th=[ 120], 95.00th=[ 120], | 99.00th=[ 120], 99.50th=[ 120], 99.90th=[ 120], 99.95th=[ 120], | 99.99th=[ 120] bw ( KiB/s): min= 8192, max= 8192, per=39.55%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=2, BW=10.1MiB/s (10.6MB/s)(8192KiB/791msec) clat (msec): min=286, max=288, avg=287.37, stdev= 1.86 lat (msec): min=287, max=288, avg=288.03, stdev= 1.36 clat percentiles (msec): | 1.00th=[ 288], 5.00th=[ 288], 10.00th=[ 288], 20.00th=[ 288], | 30.00th=[ 288], 40.00th=[ 288], 50.00th=[ 288], 60.00th=[ 288], | 70.00th=[ 288], 80.00th=[ 288], 90.00th=[ 288], 95.00th=[ 288], | 99.00th=[ 288], 99.50th=[ 288], 99.90th=[ 288], 99.95th=[ 288], | 99.99th=[ 288] lat (msec) : 250=33.33%, 500=66.67% cpu : usr=0.25%, sys=28.86%, ctx=288, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32673: Thu May 1 22:34:26 2025 write: IOPS=3, BW=12.8MiB/s (13.4MB/s)(12.0MiB/938msec) clat (msec): min=218, max=383, avg=281.27, stdev=89.28 lat (msec): min=218, max=383, avg=281.53, stdev=89.29 clat percentiles (msec): | 1.00th=[ 220], 5.00th=[ 220], 10.00th=[ 220], 20.00th=[ 220], | 30.00th=[ 220], 40.00th=[ 243], 50.00th=[ 243], 60.00th=[ 243], | 70.00th=[ 384], 80.00th=[ 384], 90.00th=[ 384], 95.00th=[ 384], | 99.00th=[ 384], 99.50th=[ 384], 99.90th=[ 384], 99.95th=[ 384], | 99.99th=[ 384] bw ( KiB/s): min= 8192, max= 8192, per=23.45%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.00%, sys=31.27%, ctx=64, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32674: Thu May 1 22:34:26 2025 write: IOPS=3, BW=15.9MiB/s (16.6MB/s)(12.0MiB/756msec) clat (msec): min=155, max=348, avg=231.83, stdev=102.31 lat (msec): min=155, max=348, avg=232.19, stdev=102.25 clat percentiles (msec): | 1.00th=[ 157], 5.00th=[ 157], 10.00th=[ 157], 20.00th=[ 157], | 30.00th=[ 157], 40.00th=[ 192], 50.00th=[ 192], 60.00th=[ 192], | 70.00th=[ 347], 80.00th=[ 347], 90.00th=[ 347], 95.00th=[ 347], | 99.00th=[ 347], 99.50th=[ 347], 99.90th=[ 347], 99.95th=[ 347], | 99.99th=[ 347] bw ( KiB/s): min= 8192, max= 8192, per=23.45%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.00%, sys=38.68%, ctx=166, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=20.2MiB/s (21.2MB/s), 5178KiB/s-23.0MiB/s (5303kB/s-25.1MB/s), io=16.0MiB (16.8MB), run=501-791msec WRITE: bw=34.1MiB/s (35.8MB/s), 10.1MiB/s-15.9MiB/s (10.6MB/s-16.6MB/s), io=32.0MiB (33.6MB), run=756-938msec