== sanity test 398b: DIO and buffer IO race ============== 23:37:18 (1746157038) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 2.19241 s, 23.0 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=22348: Thu May 1 23:38:00 2025 read: IOPS=40, BW=161KiB/s (165kB/s)(5992KiB/37258msec) clat (usec): min=78, max=74452, avg=13677.75, stdev=9080.52 lat (usec): min=79, max=74453, avg=13679.35, stdev=9080.63 clat percentiles (usec): | 1.00th=[ 97], 5.00th=[ 130], 10.00th=[ 4948], 20.00th=[ 7111], | 30.00th=[ 8717], 40.00th=[10159], 50.00th=[11994], 60.00th=[13829], | 70.00th=[16319], 80.00th=[19530], 90.00th=[24511], 95.00th=[29754], | 99.00th=[45351], 99.50th=[53216], 99.90th=[68682], 99.95th=[74974], | 99.99th=[74974] bw ( KiB/s): min= 32, max= 320, per=25.32%, avg=160.04, stdev=47.15, samples=74 iops : min= 8, max= 80, avg=39.72, stdev=11.78, samples=74 write: IOPS=42, BW=169KiB/s (173kB/s)(6296KiB/37258msec) clat (usec): min=614, max=321406, avg=10526.11, stdev=13798.48 lat (usec): min=616, max=321407, avg=10527.55, stdev=13798.48 clat percentiles (usec): | 1.00th=[ 693], 5.00th=[ 758], 10.00th=[ 816], 20.00th=[ 938], | 30.00th=[ 1614], 40.00th=[ 3163], 50.00th=[ 7111], 60.00th=[ 9765], | 70.00th=[ 13566], 80.00th=[ 18482], 90.00th=[ 26084], 95.00th=[ 31589], | 99.00th=[ 51643], 99.50th=[ 60031], 99.90th=[ 84411], 99.95th=[320865], | 99.99th=[320865] bw ( KiB/s): min= 8, max= 256, per=25.38%, avg=168.03, stdev=44.35, samples=74 iops : min= 2, max= 64, avg=41.72, stdev=11.10, samples=74 lat (usec) : 100=0.75%, 250=2.25%, 500=0.29%, 750=2.15%, 1000=9.08% lat (msec) : 2=6.18%, 4=4.82%, 10=25.16%, 20=31.12%, 50=17.35% lat (msec) : 100=0.81%, 500=0.03% cpu : usr=0.21%, sys=12.20%, ctx=5064, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22349: Thu May 1 23:38:00 2025 read: IOPS=39, BW=157KiB/s (161kB/s)(5940KiB/37771msec) clat (usec): min=66, max=323914, avg=14187.22, stdev=12346.44 lat (usec): min=67, max=323916, avg=14194.02, stdev=12347.00 clat percentiles (usec): | 1.00th=[ 96], 5.00th=[ 127], 10.00th=[ 4948], 20.00th=[ 7242], | 30.00th=[ 8717], 40.00th=[ 10290], 50.00th=[ 12256], 60.00th=[ 14353], | 70.00th=[ 16909], 80.00th=[ 19792], 90.00th=[ 25297], 95.00th=[ 30802], | 99.00th=[ 46924], 99.50th=[ 56361], 99.90th=[ 77071], 99.95th=[325059], | 99.99th=[325059] bw ( KiB/s): min= 31, max= 255, per=24.68%, avg=155.99, stdev=42.48, samples=75 iops : min= 7, max= 63, avg=38.63, stdev=10.66, samples=75 write: IOPS=42, BW=168KiB/s (172kB/s)(6348KiB/37771msec) clat (usec): min=464, max=77903, avg=10394.48, stdev=10951.58 lat (usec): min=465, max=77905, avg=10397.47, stdev=10950.51 clat percentiles (usec): | 1.00th=[ 660], 5.00th=[ 750], 10.00th=[ 807], 20.00th=[ 963], | 30.00th=[ 1631], 40.00th=[ 3326], 50.00th=[ 7308], 60.00th=[10552], | 70.00th=[13960], 80.00th=[18482], 90.00th=[25297], 95.00th=[32113], | 99.00th=[47449], 99.50th=[52167], 99.90th=[72877], 99.95th=[78119], | 99.99th=[78119] bw ( KiB/s): min= 63, max= 304, per=25.06%, avg=165.89, stdev=50.37, samples=75 iops : min= 15, max= 76, avg=41.11, stdev=12.61, samples=75 lat (usec) : 100=0.81%, 250=2.60%, 500=0.16%, 750=2.70%, 1000=8.07% lat (msec) : 2=6.48%, 4=4.33%, 10=23.21%, 20=33.53%, 50=17.22% lat (msec) : 100=0.85%, 500=0.03% cpu : usr=0.24%, sys=12.06%, ctx=5198, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22350: Thu May 1 23:38:00 2025 read: IOPS=39, BW=159KiB/s (163kB/s)(6028KiB/37935msec) clat (usec): min=75, max=334633, avg=13892.73, stdev=12167.14 lat (usec): min=76, max=334634, avg=13894.63, stdev=12166.72 clat percentiles (usec): | 1.00th=[ 93], 5.00th=[ 124], 10.00th=[ 4621], 20.00th=[ 7046], | 30.00th=[ 8586], 40.00th=[ 10290], 50.00th=[ 12125], 60.00th=[ 14222], | 70.00th=[ 16581], 80.00th=[ 20055], 90.00th=[ 25035], 95.00th=[ 30016], | 99.00th=[ 44303], 99.50th=[ 47973], 99.90th=[ 58459], 99.95th=[333448], | 99.99th=[333448] bw ( KiB/s): min= 24, max= 240, per=24.75%, avg=156.44, stdev=42.91, samples=75 iops : min= 6, max= 60, avg=38.84, stdev=10.70, samples=75 write: IOPS=41, BW=165KiB/s (169kB/s)(6260KiB/37935msec) clat (usec): min=451, max=94468, avg=10735.08, stdev=11440.38 lat (usec): min=452, max=94470, avg=10737.31, stdev=11440.34 clat percentiles (usec): | 1.00th=[ 676], 5.00th=[ 758], 10.00th=[ 816], 20.00th=[ 938], | 30.00th=[ 1663], 40.00th=[ 3752], 50.00th=[ 7504], 60.00th=[10552], | 70.00th=[14353], 80.00th=[19268], 90.00th=[26084], 95.00th=[32637], | 99.00th=[47449], 99.50th=[59507], 99.90th=[82314], 99.95th=[94897], | 99.99th=[94897] bw ( KiB/s): min= 32, max= 280, per=24.50%, avg=162.19, stdev=47.16, samples=75 iops : min= 8, max= 70, avg=40.28, stdev=11.82, samples=75 lat (usec) : 100=0.85%, 250=2.64%, 500=0.20%, 750=2.41%, 1000=9.31% lat (msec) : 2=5.34%, 4=4.00%, 10=23.96%, 20=31.97%, 50=18.68% lat (msec) : 100=0.62%, 500=0.03% cpu : usr=0.16%, sys=12.10%, ctx=5182, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22352: Thu May 1 23:38:00 2025 read: IOPS=39, BW=160KiB/s (164kB/s)(6052KiB/37835msec) clat (usec): min=69, max=327361, avg=14315.68, stdev=12366.98 lat (usec): min=70, max=327363, avg=14317.62, stdev=12366.51 clat percentiles (usec): | 1.00th=[ 96], 5.00th=[ 131], 10.00th=[ 5145], 20.00th=[ 7177], | 30.00th=[ 8848], 40.00th=[ 10421], 50.00th=[ 12387], 60.00th=[ 14353], | 70.00th=[ 17171], 80.00th=[ 20317], 90.00th=[ 25560], 95.00th=[ 30540], | 99.00th=[ 44827], 99.50th=[ 54264], 99.90th=[103285], 99.95th=[329253], | 99.99th=[329253] bw ( KiB/s): min= 32, max= 264, per=25.11%, avg=158.68, stdev=44.66, samples=75 iops : min= 8, max= 66, avg=39.40, stdev=11.18, samples=75 write: IOPS=41, BW=165KiB/s (169kB/s)(6236KiB/37835msec) clat (usec): min=465, max=67229, avg=10248.29, stdev=10807.50 lat (usec): min=466, max=67230, avg=10249.68, stdev=10807.54 clat percentiles (usec): | 1.00th=[ 693], 5.00th=[ 758], 10.00th=[ 799], 20.00th=[ 971], | 30.00th=[ 1565], 40.00th=[ 3032], 50.00th=[ 7177], 60.00th=[10290], | 70.00th=[13698], 80.00th=[18482], 90.00th=[25560], 95.00th=[32637], | 99.00th=[45351], 99.50th=[49021], 99.90th=[55837], 99.95th=[67634], | 99.99th=[67634] bw ( KiB/s): min= 32, max= 328, per=24.74%, avg=163.79, stdev=53.55, samples=75 iops : min= 8, max= 82, avg=40.68, stdev=13.40, samples=75 lat (usec) : 100=0.78%, 250=2.28%, 500=0.20%, 750=2.31%, 1000=8.40% lat (msec) : 2=6.87%, 4=4.20%, 10=23.37%, 20=32.68%, 50=18.29% lat (msec) : 100=0.55%, 250=0.03%, 500=0.03% cpu : usr=0.13%, sys=12.08%, ctx=5159, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=633KiB/s (648kB/s), 157KiB/s-161KiB/s (161kB/s-165kB/s), io=23.4MiB (24.6MB), run=37258-37935msec WRITE: bw=663KiB/s (679kB/s), 165KiB/s-169KiB/s (169kB/s-173kB/s), io=24.6MiB (25.7MB), run=37258-37935msec rand-rw: (groupid=0, jobs=1): err= 0: pid=22353: Thu May 1 23:38:14 2025 read: IOPS=29, BW=117KiB/s (120kB/s)(5992KiB/51025msec) clat (msec): min=2, max=300, avg=12.84, stdev=14.75 lat (msec): min=2, max=300, avg=12.84, stdev=14.75 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 4], 10.00th=[ 5], 20.00th=[ 5], | 30.00th=[ 6], 40.00th=[ 7], 50.00th=[ 8], 60.00th=[ 10], | 70.00th=[ 12], 80.00th=[ 18], 90.00th=[ 30], 95.00th=[ 41], | 99.00th=[ 68], 99.50th=[ 81], 99.90th=[ 102], 99.95th=[ 300], | 99.99th=[ 300] bw ( KiB/s): min= 15, max= 328, per=24.93%, avg=117.17, stdev=77.80, samples=102 iops : min= 3, max= 82, avg=29.05, stdev=19.50, samples=102 write: IOPS=30, BW=123KiB/s (126kB/s)(6296KiB/51025msec) clat (msec): min=4, max=126, avg=20.08, stdev=15.12 lat (msec): min=4, max=126, avg=20.08, stdev=15.12 clat percentiles (msec): | 1.00th=[ 6], 5.00th=[ 8], 10.00th=[ 8], 20.00th=[ 10], | 30.00th=[ 11], 40.00th=[ 12], 50.00th=[ 14], 60.00th=[ 18], | 70.00th=[ 24], 80.00th=[ 30], 90.00th=[ 41], 95.00th=[ 50], | 99.00th=[ 78], 99.50th=[ 92], 99.90th=[ 124], 99.95th=[ 127], | 99.99th=[ 127] bw ( KiB/s): min= 39, max= 336, per=25.23%, avg=124.11, stdev=78.28, samples=101 iops : min= 9, max= 84, avg=30.78, stdev=19.63, samples=101 lat (msec) : 4=4.07%, 10=39.29%, 20=30.73%, 50=22.17%, 100=3.52% lat (msec) : 250=0.20%, 500=0.03% cpu : usr=0.13%, sys=5.84%, ctx=3389, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22354: Thu May 1 23:38:14 2025 read: IOPS=29, BW=117KiB/s (120kB/s)(5940KiB/50890msec) clat (msec): min=2, max=346, avg=12.53, stdev=15.39 lat (msec): min=2, max=346, avg=12.53, stdev=15.39 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 4], 10.00th=[ 5], 20.00th=[ 5], | 30.00th=[ 6], 40.00th=[ 7], 50.00th=[ 8], 60.00th=[ 9], | 70.00th=[ 12], 80.00th=[ 17], 90.00th=[ 28], 95.00th=[ 40], | 99.00th=[ 67], 99.50th=[ 82], 99.90th=[ 104], 99.95th=[ 347], | 99.99th=[ 347] bw ( KiB/s): min= 8, max= 384, per=24.59%, avg=115.58, stdev=80.70, samples=101 iops : min= 2, max= 96, avg=28.61, stdev=20.24, samples=101 write: IOPS=31, BW=125KiB/s (128kB/s)(6348KiB/50890msec) clat (msec): min=4, max=124, avg=20.25, stdev=14.93 lat (msec): min=4, max=124, avg=20.25, stdev=14.93 clat percentiles (msec): | 1.00th=[ 6], 5.00th=[ 8], 10.00th=[ 8], 20.00th=[ 10], | 30.00th=[ 11], 40.00th=[ 12], 50.00th=[ 14], 60.00th=[ 18], | 70.00th=[ 24], 80.00th=[ 32], 90.00th=[ 44], 95.00th=[ 52], | 99.00th=[ 70], 99.50th=[ 74], 99.90th=[ 93], 99.95th=[ 125], | 99.99th=[ 125] bw ( KiB/s): min= 8, max= 368, per=24.93%, avg=122.63, stdev=79.60, samples=101 iops : min= 2, max= 92, avg=30.38, stdev=19.98, samples=101 lat (msec) : 4=4.46%, 10=41.34%, 20=28.12%, 50=21.84%, 100=4.13% lat (msec) : 250=0.07%, 500=0.03% cpu : usr=0.11%, sys=5.72%, ctx=3352, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22355: Thu May 1 23:38:14 2025 read: IOPS=29, BW=119KiB/s (122kB/s)(6028KiB/50630msec) clat (usec): min=2511, max=96964, avg=12639.74, stdev=12594.32 lat (usec): min=2511, max=96965, avg=12641.00, stdev=12594.33 clat percentiles (usec): | 1.00th=[ 3032], 5.00th=[ 3621], 10.00th=[ 4146], 20.00th=[ 5014], | 30.00th=[ 5669], 40.00th=[ 6587], 50.00th=[ 7832], 60.00th=[ 9503], | 70.00th=[12256], 80.00th=[17433], 90.00th=[28181], 95.00th=[38011], | 99.00th=[63177], 99.50th=[82314], 99.90th=[93848], 99.95th=[96994], | 99.99th=[96994] bw ( KiB/s): min= 31, max= 320, per=25.48%, avg=119.74, stdev=73.28, samples=100 iops : min= 7, max= 80, avg=29.75, stdev=18.31, samples=100 write: IOPS=30, BW=124KiB/s (127kB/s)(6260KiB/50630msec) clat (msec): min=4, max=277, avg=20.03, stdev=15.60 lat (msec): min=4, max=277, avg=20.03, stdev=15.60 clat percentiles (msec): | 1.00th=[ 6], 5.00th=[ 8], 10.00th=[ 9], 20.00th=[ 10], | 30.00th=[ 11], 40.00th=[ 13], 50.00th=[ 15], 60.00th=[ 18], | 70.00th=[ 23], 80.00th=[ 29], 90.00th=[ 40], 95.00th=[ 50], | 99.00th=[ 70], 99.50th=[ 78], 99.90th=[ 125], 99.95th=[ 279], | 99.99th=[ 279] bw ( KiB/s): min= 16, max= 359, per=24.98%, avg=122.90, stdev=78.40, samples=101 iops : min= 4, max= 89, avg=30.54, stdev=19.59, samples=101 lat (msec) : 4=3.97%, 10=39.49%, 20=30.05%, 50=22.95%, 100=3.45% lat (msec) : 250=0.07%, 500=0.03% cpu : usr=0.13%, sys=5.88%, ctx=3392, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22356: Thu May 1 23:38:14 2025 read: IOPS=29, BW=119KiB/s (122kB/s)(6052KiB/50932msec) clat (msec): min=2, max=116, avg=12.96, stdev=12.97 lat (msec): min=2, max=116, avg=12.97, stdev=12.97 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 4], 10.00th=[ 5], 20.00th=[ 5], | 30.00th=[ 6], 40.00th=[ 7], 50.00th=[ 8], 60.00th=[ 10], | 70.00th=[ 13], 80.00th=[ 19], 90.00th=[ 31], 95.00th=[ 40], | 99.00th=[ 63], 99.50th=[ 82], 99.90th=[ 102], 99.95th=[ 116], | 99.99th=[ 116] bw ( KiB/s): min= 15, max= 360, per=25.13%, avg=118.10, stdev=76.43, samples=100 iops : min= 3, max= 90, avg=29.25, stdev=19.12, samples=100 write: IOPS=30, BW=122KiB/s (125kB/s)(6236KiB/50932msec) clat (msec): min=3, max=279, avg=19.93, stdev=15.55 lat (msec): min=3, max=279, avg=19.94, stdev=15.55 clat percentiles (msec): | 1.00th=[ 6], 5.00th=[ 8], 10.00th=[ 8], 20.00th=[ 10], | 30.00th=[ 11], 40.00th=[ 12], 50.00th=[ 15], 60.00th=[ 18], | 70.00th=[ 24], 80.00th=[ 31], 90.00th=[ 40], 95.00th=[ 48], | 99.00th=[ 69], 99.50th=[ 84], 99.90th=[ 112], 99.95th=[ 279], | 99.99th=[ 279] bw ( KiB/s): min= 16, max= 309, per=24.50%, avg=120.56, stdev=75.50, samples=101 iops : min= 4, max= 77, avg=29.87, stdev=18.90, samples=101 lat (msec) : 4=4.36%, 10=38.25%, 20=30.44%, 50=23.54%, 100=3.29% lat (msec) : 250=0.10%, 500=0.03% cpu : usr=0.10%, sys=5.95%, ctx=3375, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=471KiB/s (482kB/s), 117KiB/s-119KiB/s (120kB/s-122kB/s), io=23.4MiB (24.6MB), run=50630-51025msec WRITE: bw=493KiB/s (505kB/s), 122KiB/s-125KiB/s (125kB/s-128kB/s), io=24.6MiB (25.7MB), run=50630-51025msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=22410: Thu May 1 23:38:22 2025 read: IOPS=51, BW=823KiB/s (843kB/s)(5920KiB/7195msec) clat (usec): min=68, max=57499, avg=10200.18, stdev=7884.17 lat (usec): min=69, max=57500, avg=10201.79, stdev=7885.54 clat percentiles (usec): | 1.00th=[ 91], 5.00th=[ 108], 10.00th=[ 121], 20.00th=[ 5145], | 30.00th=[ 6325], 40.00th=[ 7701], 50.00th=[ 8979], 60.00th=[10421], | 70.00th=[12518], 80.00th=[15008], 90.00th=[18482], 95.00th=[22414], | 99.00th=[40633], 99.50th=[50594], 99.90th=[57410], 99.95th=[57410], | 99.99th=[57410] bw ( KiB/s): min= 448, max= 1085, per=24.40%, avg=811.93, stdev=190.81, samples=14 iops : min= 28, max= 67, avg=50.43, stdev=11.71, samples=14 write: IOPS=55, BW=885KiB/s (906kB/s)(6368KiB/7195msec) clat (usec): min=893, max=46537, avg=8317.04, stdev=8125.87 lat (usec): min=894, max=46539, avg=8318.70, stdev=8125.80 clat percentiles (usec): | 1.00th=[ 971], 5.00th=[ 1090], 10.00th=[ 1156], 20.00th=[ 1336], | 30.00th=[ 2008], 40.00th=[ 3326], 50.00th=[ 5538], 60.00th=[ 8029], | 70.00th=[10683], 80.00th=[14484], 90.00th=[20579], 95.00th=[24511], | 99.00th=[32113], 99.50th=[43254], 99.90th=[46400], 99.95th=[46400], | 99.99th=[46400] bw ( KiB/s): min= 448, max= 1272, per=25.23%, avg=883.43, stdev=196.95, samples=14 iops : min= 28, max= 79, avg=54.93, stdev=12.36, samples=14 lat (usec) : 100=1.43%, 250=5.21%, 500=0.13%, 1000=0.91% lat (msec) : 2=14.84%, 4=7.16%, 10=33.33%, 20=27.47%, 50=9.24% lat (msec) : 100=0.26% cpu : usr=0.21%, sys=15.94%, ctx=1336, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22411: Thu May 1 23:38:22 2025 read: IOPS=53, BW=855KiB/s (876kB/s)(5744KiB/6716msec) clat (usec): min=73, max=52123, avg=9996.15, stdev=6983.49 lat (usec): min=74, max=52123, avg=9997.19, stdev=6983.43 clat percentiles (usec): | 1.00th=[ 92], 5.00th=[ 112], 10.00th=[ 131], 20.00th=[ 5211], | 30.00th=[ 6259], 40.00th=[ 7767], 50.00th=[ 8979], 60.00th=[10421], | 70.00th=[12387], 80.00th=[15270], 90.00th=[18482], 95.00th=[20841], | 99.00th=[31327], 99.50th=[52167], 99.90th=[52167], 99.95th=[52167], | 99.99th=[52167] bw ( KiB/s): min= 479, max= 1088, per=25.73%, avg=856.15, stdev=182.83, samples=13 iops : min= 29, max= 68, avg=53.31, stdev=11.51, samples=13 write: IOPS=60, BW=974KiB/s (998kB/s)(6544KiB/6716msec) clat (usec): min=854, max=44700, avg=7402.62, stdev=7487.82 lat (usec): min=856, max=44702, avg=7404.40, stdev=7488.25 clat percentiles (usec): | 1.00th=[ 971], 5.00th=[ 1090], 10.00th=[ 1172], 20.00th=[ 1319], | 30.00th=[ 1713], 40.00th=[ 2737], 50.00th=[ 4948], 60.00th=[ 6915], | 70.00th=[ 9241], 80.00th=[12125], 90.00th=[18220], 95.00th=[23200], | 99.00th=[31327], 99.50th=[36439], 99.90th=[44827], 99.95th=[44827], | 99.99th=[44827] bw ( KiB/s): min= 640, max= 1408, per=27.47%, avg=962.00, stdev=238.86, samples=13 iops : min= 40, max= 88, avg=59.92, stdev=15.06, samples=13 lat (usec) : 100=1.30%, 250=4.17%, 1000=0.91% lat (msec) : 2=17.19%, 4=7.16%, 10=33.72%, 20=28.12%, 50=7.16% lat (msec) : 100=0.26% cpu : usr=0.31%, sys=16.07%, ctx=1366, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22413: Thu May 1 23:38:22 2025 read: IOPS=55, BW=880KiB/s (901kB/s)(6256KiB/7108msec) clat (usec): min=87, max=42804, avg=9880.14, stdev=7110.68 lat (usec): min=88, max=42805, avg=9881.36, stdev=7110.52 clat percentiles (usec): | 1.00th=[ 93], 5.00th=[ 106], 10.00th=[ 133], 20.00th=[ 5080], | 30.00th=[ 6652], 40.00th=[ 7832], 50.00th=[ 8848], 60.00th=[10028], | 70.00th=[11600], 80.00th=[14353], 90.00th=[18220], 95.00th=[24249], | 99.00th=[34866], 99.50th=[40633], 99.90th=[42730], 99.95th=[42730], | 99.99th=[42730] bw ( KiB/s): min= 638, max= 1184, per=26.70%, avg=888.71, stdev=182.33, samples=14 iops : min= 39, max= 74, avg=55.36, stdev=11.56, samples=14 write: IOPS=53, BW=849KiB/s (869kB/s)(6032KiB/7108msec) clat (usec): min=903, max=49182, avg=8529.11, stdev=8089.02 lat (usec): min=904, max=49184, avg=8530.62, stdev=8089.08 clat percentiles (usec): | 1.00th=[ 1004], 5.00th=[ 1156], 10.00th=[ 1205], 20.00th=[ 1450], | 30.00th=[ 2147], 40.00th=[ 3982], 50.00th=[ 6456], 60.00th=[ 8029], | 70.00th=[10945], 80.00th=[14746], 90.00th=[20579], 95.00th=[24249], | 99.00th=[33817], 99.50th=[43779], 99.90th=[49021], 99.95th=[49021], | 99.99th=[49021] bw ( KiB/s): min= 544, max= 1312, per=24.00%, avg=840.57, stdev=208.91, samples=14 iops : min= 34, max= 82, avg=52.36, stdev=12.93, samples=14 lat (usec) : 100=1.56%, 250=5.08%, 500=0.52%, 1000=0.39% lat (msec) : 2=13.93%, 4=5.60%, 10=35.42%, 20=28.65%, 50=8.85% cpu : usr=0.17%, sys=16.24%, ctx=1421, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22415: Thu May 1 23:38:22 2025 read: IOPS=54, BW=872KiB/s (893kB/s)(6032KiB/6915msec) clat (usec): min=70, max=41806, avg=10260.19, stdev=7049.37 lat (usec): min=70, max=41808, avg=10262.89, stdev=7048.67 clat percentiles (usec): | 1.00th=[ 89], 5.00th=[ 113], 10.00th=[ 151], 20.00th=[ 5604], | 30.00th=[ 6718], 40.00th=[ 7701], 50.00th=[ 9241], 60.00th=[10552], | 70.00th=[12649], 80.00th=[15139], 90.00th=[20317], 95.00th=[23725], | 99.00th=[34341], 99.50th=[35390], 99.90th=[41681], 99.95th=[41681], | 99.99th=[41681] bw ( KiB/s): min= 576, max= 1216, per=25.95%, avg=863.69, stdev=156.39, samples=13 iops : min= 36, max= 76, avg=53.85, stdev= 9.86, samples=13 write: IOPS=56, BW=905KiB/s (926kB/s)(6256KiB/6915msec) clat (usec): min=847, max=41166, avg=7580.36, stdev=7611.32 lat (usec): min=849, max=41168, avg=7581.92, stdev=7611.36 clat percentiles (usec): | 1.00th=[ 906], 5.00th=[ 1106], 10.00th=[ 1172], 20.00th=[ 1418], | 30.00th=[ 1991], 40.00th=[ 3163], 50.00th=[ 5014], 60.00th=[ 6652], | 70.00th=[ 9765], 80.00th=[12911], 90.00th=[17433], 95.00th=[22938], | 99.00th=[36963], 99.50th=[40633], 99.90th=[41157], 99.95th=[41157], | 99.99th=[41157] bw ( KiB/s): min= 640, max= 1120, per=25.93%, avg=908.00, stdev=140.68, samples=13 iops : min= 40, max= 70, avg=56.62, stdev= 8.91, samples=13 lat (usec) : 100=1.43%, 250=4.30%, 500=0.65%, 750=0.13%, 1000=1.04% lat (msec) : 2=14.32%, 4=8.33%, 10=33.46%, 20=27.08%, 50=9.24% cpu : usr=0.19%, sys=16.76%, ctx=1400, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=3329KiB/s (3409kB/s), 823KiB/s-880KiB/s (843kB/s-901kB/s), io=23.4MiB (24.5MB), run=6716-7195msec WRITE: bw=3502KiB/s (3586kB/s), 849KiB/s-974KiB/s (869kB/s-998kB/s), io=24.6MiB (25.8MB), run=6716-7195msec rand-rw: (groupid=0, jobs=1): err= 0: pid=22412: Thu May 1 23:38:25 2025 read: IOPS=35, BW=572KiB/s (586kB/s)(5920KiB/10350msec) clat (usec): min=2790, max=78714, avg=10524.79, stdev=9719.32 lat (usec): min=2792, max=78716, avg=10525.95, stdev=9719.37 clat percentiles (usec): | 1.00th=[ 3064], 5.00th=[ 3720], 10.00th=[ 4047], 20.00th=[ 4817], | 30.00th=[ 5407], 40.00th=[ 6456], 50.00th=[ 7242], 60.00th=[ 8356], | 70.00th=[10028], 80.00th=[12649], 90.00th=[23462], 95.00th=[30278], | 99.00th=[53216], 99.50th=[63177], 99.90th=[79168], 99.95th=[79168], | 99.99th=[79168] bw ( KiB/s): min= 285, max= 1024, per=24.21%, avg=554.50, stdev=222.38, samples=20 iops : min= 17, max= 64, avg=34.25, stdev=14.13, samples=20 write: IOPS=38, BW=615KiB/s (630kB/s)(6368KiB/10350msec) clat (usec): min=5101, max=54430, avg=16063.49, stdev=8491.51 lat (usec): min=5102, max=54431, avg=16065.82, stdev=8491.60 clat percentiles (usec): | 1.00th=[ 5538], 5.00th=[ 6980], 10.00th=[ 8160], 20.00th=[ 9503], | 30.00th=[10814], 40.00th=[11994], 50.00th=[13042], 60.00th=[15401], | 70.00th=[17957], 80.00th=[22152], 90.00th=[27657], 95.00th=[32113], | 99.00th=[45876], 99.50th=[52167], 99.90th=[54264], 99.95th=[54264], | 99.99th=[54264] bw ( KiB/s): min= 191, max= 1056, per=24.74%, avg=595.90, stdev=241.32, samples=20 iops : min= 11, max= 66, avg=36.85, stdev=15.28, samples=20 lat (msec) : 4=4.43%, 10=41.54%, 20=35.42%, 50=17.84%, 100=0.78% cpu : usr=0.22%, sys=7.08%, ctx=862, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22414: Thu May 1 23:38:25 2025 read: IOPS=34, BW=550KiB/s (564kB/s)(5744KiB/10438msec) clat (usec): min=2600, max=64539, avg=9897.29, stdev=8343.22 lat (usec): min=2601, max=64540, avg=9898.45, stdev=8343.19 clat percentiles (usec): | 1.00th=[ 3326], 5.00th=[ 3654], 10.00th=[ 4228], 20.00th=[ 4817], | 30.00th=[ 5604], 40.00th=[ 6063], 50.00th=[ 6980], 60.00th=[ 7898], | 70.00th=[ 9503], 80.00th=[12387], 90.00th=[20841], 95.00th=[28967], | 99.00th=[45351], 99.50th=[47449], 99.90th=[64750], 99.95th=[64750], | 99.99th=[64750] bw ( KiB/s): min= 255, max= 1056, per=23.30%, avg=533.50, stdev=231.83, samples=20 iops : min= 15, max= 66, avg=32.90, stdev=14.53, samples=20 write: IOPS=39, BW=627KiB/s (642kB/s)(6544KiB/10438msec) clat (usec): min=4004, max=70986, avg=16770.77, stdev=10269.22 lat (usec): min=4006, max=70989, avg=16773.34, stdev=10268.66 clat percentiles (usec): | 1.00th=[ 5735], 5.00th=[ 7308], 10.00th=[ 8160], 20.00th=[ 9372], | 30.00th=[10552], 40.00th=[11600], 50.00th=[12649], 60.00th=[14615], | 70.00th=[18220], 80.00th=[23987], 90.00th=[31589], 95.00th=[39060], | 99.00th=[51643], 99.50th=[53740], 99.90th=[70779], 99.95th=[70779], | 99.99th=[70779] bw ( KiB/s): min= 255, max= 1216, per=24.87%, avg=599.15, stdev=235.74, samples=20 iops : min= 15, max= 76, avg=37.00, stdev=14.89, samples=20 lat (msec) : 4=3.39%, 10=43.75%, 20=33.85%, 50=18.23%, 100=0.78% cpu : usr=0.06%, sys=7.08%, ctx=867, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22416: Thu May 1 23:38:25 2025 read: IOPS=37, BW=603KiB/s (617kB/s)(6256KiB/10377msec) clat (usec): min=2760, max=78089, avg=10675.69, stdev=9511.51 lat (usec): min=2761, max=78502, avg=10678.43, stdev=9519.03 clat percentiles (usec): | 1.00th=[ 3097], 5.00th=[ 3818], 10.00th=[ 4293], 20.00th=[ 5080], | 30.00th=[ 5800], 40.00th=[ 6390], 50.00th=[ 7242], 60.00th=[ 8225], | 70.00th=[10028], 80.00th=[13042], 90.00th=[21365], 95.00th=[34341], | 99.00th=[46924], 99.50th=[51643], 99.90th=[78119], 99.95th=[78119], | 99.99th=[78119] bw ( KiB/s): min= 287, max= 1408, per=25.84%, avg=591.65, stdev=301.90, samples=20 iops : min= 17, max= 88, avg=36.75, stdev=19.02, samples=20 write: IOPS=36, BW=581KiB/s (595kB/s)(6032KiB/10377msec) clat (usec): min=4750, max=64513, avg=16261.73, stdev=9412.35 lat (usec): min=4752, max=64514, avg=16263.95, stdev=9412.48 clat percentiles (usec): | 1.00th=[ 6325], 5.00th=[ 7242], 10.00th=[ 7832], 20.00th=[ 9110], | 30.00th=[10421], 40.00th=[11600], 50.00th=[13042], 60.00th=[15533], | 70.00th=[17957], 80.00th=[21890], 90.00th=[28443], 95.00th=[35914], | 99.00th=[47449], 99.50th=[62653], 99.90th=[64750], 99.95th=[64750], | 99.99th=[64750] bw ( KiB/s): min= 224, max= 960, per=23.10%, avg=556.40, stdev=220.08, samples=20 iops : min= 14, max= 60, avg=34.55, stdev=13.83, samples=20 lat (msec) : 4=3.91%, 10=44.92%, 20=32.42%, 50=18.10%, 100=0.65% cpu : usr=0.11%, sys=7.06%, ctx=890, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22417: Thu May 1 23:38:25 2025 read: IOPS=36, BW=577KiB/s (591kB/s)(6032KiB/10458msec) clat (usec): min=2635, max=54598, avg=10425.43, stdev=8699.70 lat (usec): min=2636, max=54599, avg=10426.72, stdev=8699.76 clat percentiles (usec): | 1.00th=[ 3163], 5.00th=[ 3720], 10.00th=[ 4178], 20.00th=[ 4817], | 30.00th=[ 5604], 40.00th=[ 6194], 50.00th=[ 7046], 60.00th=[ 8160], | 70.00th=[10028], 80.00th=[13960], 90.00th=[22938], 95.00th=[28967], | 99.00th=[45351], 99.50th=[51643], 99.90th=[54789], 99.95th=[54789], | 99.99th=[54789] bw ( KiB/s): min= 286, max= 1088, per=23.93%, avg=547.95, stdev=217.68, samples=20 iops : min= 17, max= 68, avg=33.80, stdev=13.71, samples=20 write: IOPS=37, BW=598KiB/s (613kB/s)(6256KiB/10458msec) clat (usec): min=4865, max=70547, avg=16441.34, stdev=9388.46 lat (usec): min=4866, max=70549, avg=16450.69, stdev=9401.50 clat percentiles (usec): | 1.00th=[ 5800], 5.00th=[ 7308], 10.00th=[ 8029], 20.00th=[ 9634], | 30.00th=[10552], 40.00th=[11863], 50.00th=[13435], 60.00th=[14615], | 70.00th=[18220], 80.00th=[23462], 90.00th=[29754], 95.00th=[34866], | 99.00th=[45876], 99.50th=[63177], 99.90th=[70779], 99.95th=[70779], | 99.99th=[70779] bw ( KiB/s): min= 319, max= 1120, per=23.94%, avg=576.75, stdev=233.72, samples=20 iops : min= 19, max= 70, avg=35.60, stdev=14.74, samples=20 lat (msec) : 4=3.39%, 10=43.36%, 20=33.72%, 50=19.01%, 100=0.52% cpu : usr=0.00%, sys=7.02%, ctx=877, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2290KiB/s (2345kB/s), 550KiB/s-603KiB/s (564kB/s-617kB/s), io=23.4MiB (24.5MB), run=10350-10458msec WRITE: bw=2410KiB/s (2467kB/s), 581KiB/s-627KiB/s (595kB/s-642kB/s), io=24.6MiB (25.8MB), run=10350-10458msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=22436: Thu May 1 23:38:27 2025 read: IOPS=8, BW=9085KiB/s (9303kB/s)(7168KiB/789msec) clat (usec): min=604, max=86375, avg=52830.85, stdev=26606.11 lat (usec): min=606, max=86376, avg=52832.49, stdev=26605.96 clat percentiles (usec): | 1.00th=[ 603], 5.00th=[ 603], 10.00th=[ 603], 20.00th=[46924], | 30.00th=[46924], 40.00th=[46924], 50.00th=[58983], 60.00th=[64750], | 70.00th=[64750], 80.00th=[65274], 90.00th=[86508], 95.00th=[86508], | 99.00th=[86508], 99.50th=[86508], 99.90th=[86508], 99.95th=[86508], | 99.99th=[86508] bw ( KiB/s): min=10240, max=10240, per=59.35%, avg=10240.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 write: IOPS=6, BW=6489KiB/s (6645kB/s)(5120KiB/789msec) clat (msec): min=35, max=197, avg=80.51, stdev=67.44 lat (msec): min=35, max=197, avg=80.57, stdev=67.41 clat percentiles (msec): | 1.00th=[ 36], 5.00th=[ 36], 10.00th=[ 36], 20.00th=[ 36], | 30.00th=[ 41], 40.00th=[ 41], 50.00th=[ 51], 60.00th=[ 51], | 70.00th=[ 80], 80.00th=[ 80], 90.00th=[ 199], 95.00th=[ 199], | 99.00th=[ 199], 99.50th=[ 199], 99.90th=[ 199], 99.95th=[ 199], | 99.99th=[ 199] bw ( KiB/s): min= 6144, max= 6144, per=25.44%, avg=6144.00, stdev= 0.00, samples=1 iops : min= 6, max= 6, avg= 6.00, stdev= 0.00, samples=1 lat (usec) : 750=8.33% lat (msec) : 50=33.33%, 100=50.00%, 250=8.33% cpu : usr=0.13%, sys=27.03%, ctx=504, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22438: Thu May 1 23:38:27 2025 read: IOPS=1, BW=1787KiB/s (1830kB/s)(2048KiB/1146msec) clat (usec): min=1236, max=51933, avg=26585.17, stdev=35848.11 lat (usec): min=1237, max=51935, avg=26586.75, stdev=35848.93 clat percentiles (usec): | 1.00th=[ 1237], 5.00th=[ 1237], 10.00th=[ 1237], 20.00th=[ 1237], | 30.00th=[ 1237], 40.00th=[ 1237], 50.00th=[ 1237], 60.00th=[52167], | 70.00th=[52167], 80.00th=[52167], 90.00th=[52167], 95.00th=[52167], | 99.00th=[52167], 99.50th=[52167], 99.90th=[52167], 99.95th=[52167], | 99.99th=[52167] bw ( KiB/s): min= 1868, max= 2048, per=11.35%, avg=1958.00, stdev=127.28, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 write: IOPS=8, BW=8935KiB/s (9150kB/s)(10.0MiB/1146msec) clat (msec): min=33, max=312, avg=98.55, stdev=85.33 lat (msec): min=34, max=312, avg=98.65, stdev=85.33 clat percentiles (msec): | 1.00th=[ 34], 5.00th=[ 34], 10.00th=[ 34], 20.00th=[ 42], | 30.00th=[ 51], 40.00th=[ 59], 50.00th=[ 61], 60.00th=[ 71], | 70.00th=[ 117], 80.00th=[ 117], 90.00th=[ 169], 95.00th=[ 313], | 99.00th=[ 313], 99.50th=[ 313], 99.90th=[ 313], 99.95th=[ 313], | 99.99th=[ 313] bw ( KiB/s): min= 6144, max= 7474, per=28.19%, avg=6809.00, stdev=940.45, samples=2 iops : min= 6, max= 7, avg= 6.50, stdev= 0.71, samples=2 lat (msec) : 2=8.33%, 50=16.67%, 100=50.00%, 250=16.67%, 500=8.33% cpu : usr=0.17%, sys=22.27%, ctx=157, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22439: Thu May 1 23:38:27 2025 read: IOPS=6, BW=6508KiB/s (6665kB/s)(6144KiB/944msec) clat (msec): min=49, max=135, avg=92.60, stdev=37.83 lat (msec): min=49, max=135, avg=92.60, stdev=37.83 clat percentiles (msec): | 1.00th=[ 50], 5.00th=[ 50], 10.00th=[ 50], 20.00th=[ 57], | 30.00th=[ 57], 40.00th=[ 74], 50.00th=[ 74], 60.00th=[ 109], | 70.00th=[ 133], 80.00th=[ 133], 90.00th=[ 136], 95.00th=[ 136], | 99.00th=[ 136], 99.50th=[ 136], 99.90th=[ 136], 99.95th=[ 136], | 99.99th=[ 136] bw ( KiB/s): min= 4096, max= 4096, per=23.74%, avg=4096.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 write: IOPS=6, BW=6508KiB/s (6665kB/s)(6144KiB/944msec) clat (usec): min=45835, max=78047, avg=56262.70, stdev=11262.11 lat (usec): min=45911, max=78133, avg=56324.46, stdev=11267.73 clat percentiles (usec): | 1.00th=[45876], 5.00th=[45876], 10.00th=[45876], 20.00th=[51643], | 30.00th=[51643], 40.00th=[52167], 50.00th=[52167], 60.00th=[52691], | 70.00th=[56886], 80.00th=[56886], 90.00th=[78119], 95.00th=[78119], | 99.00th=[78119], 99.50th=[78119], 99.90th=[78119], 99.95th=[78119], | 99.99th=[78119] bw ( KiB/s): min= 8192, max= 8192, per=33.91%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 lat (msec) : 50=16.67%, 100=58.33%, 250=25.00% cpu : usr=0.00%, sys=25.34%, ctx=515, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22440: Thu May 1 23:38:27 2025 read: IOPS=4, BW=4313KiB/s (4417kB/s)(5120KiB/1187msec) clat (usec): min=486, max=93763, avg=44097.56, stdev=43482.86 lat (usec): min=488, max=93764, avg=44099.07, stdev=43482.70 clat percentiles (usec): | 1.00th=[ 486], 5.00th=[ 486], 10.00th=[ 486], 20.00th=[ 486], | 30.00th=[ 603], 40.00th=[ 603], 50.00th=[45351], 60.00th=[45351], | 70.00th=[80217], 80.00th=[80217], 90.00th=[93848], 95.00th=[93848], | 99.00th=[93848], 99.50th=[93848], 99.90th=[93848], 99.95th=[93848], | 99.99th=[93848] bw ( KiB/s): min= 2048, max= 2048, per=11.87%, avg=2048.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=5, BW=6039KiB/s (6184kB/s)(7168KiB/1187msec) clat (msec): min=33, max=345, avg=117.83, stdev=117.08 lat (msec): min=33, max=346, avg=117.90, stdev=117.09 clat percentiles (msec): | 1.00th=[ 34], 5.00th=[ 34], 10.00th=[ 34], 20.00th=[ 40], | 30.00th=[ 42], 40.00th=[ 42], 50.00th=[ 72], 60.00th=[ 87], | 70.00th=[ 87], 80.00th=[ 207], 90.00th=[ 347], 95.00th=[ 347], | 99.00th=[ 347], 99.50th=[ 347], 99.90th=[ 347], 99.95th=[ 347], | 99.99th=[ 347] bw ( KiB/s): min= 4087, max= 4096, per=16.94%, avg=4091.50, stdev= 6.36, samples=2 iops : min= 3, max= 4, avg= 3.50, stdev= 0.71, samples=2 lat (usec) : 500=8.33%, 750=8.33% lat (msec) : 50=33.33%, 100=33.33%, 250=8.33%, 500=8.33% cpu : usr=0.00%, sys=18.21%, ctx=399, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=16.8MiB/s (17.7MB/s), 1787KiB/s-9085KiB/s (1830kB/s-9303kB/s), io=20.0MiB (20.0MB), run=789-1187msec WRITE: bw=23.6MiB/s (24.7MB/s), 6039KiB/s-8935KiB/s (6184kB/s-9150kB/s), io=28.0MiB (29.4MB), run=789-1187msec rand-rw: (groupid=0, jobs=1): err= 0: pid=22433: Thu May 1 23:38:28 2025 read: IOPS=5, BW=5842KiB/s (5982kB/s)(7168KiB/1227msec) clat (msec): min=20, max=143, avg=70.04, stdev=51.91 lat (msec): min=20, max=143, avg=70.04, stdev=51.91 clat percentiles (msec): | 1.00th=[ 21], 5.00th=[ 21], 10.00th=[ 21], 20.00th=[ 22], | 30.00th=[ 44], 40.00th=[ 44], 50.00th=[ 60], 60.00th=[ 62], | 70.00th=[ 62], 80.00th=[ 140], 90.00th=[ 144], 95.00th=[ 144], | 99.00th=[ 144], 99.50th=[ 144], 99.90th=[ 144], 99.95th=[ 144], | 99.99th=[ 144] bw ( KiB/s): min= 4087, max= 6144, per=33.00%, avg=5115.50, stdev=1454.52, samples=2 iops : min= 3, max= 6, avg= 4.50, stdev= 2.12, samples=2 write: IOPS=4, BW=4173KiB/s (4273kB/s)(5120KiB/1227msec) clat (msec): min=72, max=189, avg=123.29, stdev=45.74 lat (msec): min=72, max=189, avg=123.38, stdev=45.75 clat percentiles (msec): | 1.00th=[ 73], 5.00th=[ 73], 10.00th=[ 73], 20.00th=[ 73], | 30.00th=[ 95], 40.00th=[ 95], 50.00th=[ 113], 60.00th=[ 113], | 70.00th=[ 146], 80.00th=[ 146], 90.00th=[ 190], 95.00th=[ 190], | 99.00th=[ 190], 99.50th=[ 190], 99.90th=[ 190], 99.95th=[ 190], | 99.99th=[ 190] bw ( KiB/s): min= 6131, max= 6131, per=28.25%, avg=6131.00, stdev= 0.00, samples=1 iops : min= 5, max= 5, avg= 5.00, stdev= 0.00, samples=1 lat (msec) : 50=25.00%, 100=33.33%, 250=41.67% cpu : usr=0.00%, sys=6.20%, ctx=56, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22434: Thu May 1 23:38:28 2025 read: IOPS=1, BW=1640KiB/s (1679kB/s)(2048KiB/1249msec) clat (msec): min=19, max=174, avg=97.13, stdev=109.76 lat (msec): min=19, max=174, avg=97.14, stdev=109.76 clat percentiles (msec): | 1.00th=[ 20], 5.00th=[ 20], 10.00th=[ 20], 20.00th=[ 20], | 30.00th=[ 20], 40.00th=[ 20], 50.00th=[ 20], 60.00th=[ 176], | 70.00th=[ 176], 80.00th=[ 176], 90.00th=[ 176], 95.00th=[ 176], | 99.00th=[ 176], 99.50th=[ 176], 99.90th=[ 176], 99.95th=[ 176], | 99.99th=[ 176] bw ( KiB/s): min= 2048, max= 2048, per=13.21%, avg=2048.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=8, BW=8199KiB/s (8395kB/s)(10.0MiB/1249msec) clat (msec): min=18, max=333, avg=90.72, stdev=96.64 lat (msec): min=18, max=333, avg=90.81, stdev=96.64 clat percentiles (msec): | 1.00th=[ 18], 5.00th=[ 18], 10.00th=[ 18], 20.00th=[ 21], | 30.00th=[ 22], 40.00th=[ 39], 50.00th=[ 58], 60.00th=[ 73], | 70.00th=[ 104], 80.00th=[ 104], 90.00th=[ 165], 95.00th=[ 334], | 99.00th=[ 334], 99.50th=[ 334], 99.90th=[ 334], 99.95th=[ 334], | 99.99th=[ 334] bw ( KiB/s): min= 4096, max= 4096, per=18.87%, avg=4096.00, stdev= 0.00, samples=2 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=2 lat (msec) : 20=16.67%, 50=25.00%, 100=25.00%, 250=25.00%, 500=8.33% cpu : usr=0.16%, sys=6.41%, ctx=49, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22435: Thu May 1 23:38:28 2025 read: IOPS=5, BW=5835KiB/s (5975kB/s)(6144KiB/1053msec) clat (msec): min=25, max=244, avg=92.80, stdev=80.80 lat (msec): min=25, max=244, avg=92.80, stdev=80.80 clat percentiles (msec): | 1.00th=[ 26], 5.00th=[ 26], 10.00th=[ 26], 20.00th=[ 44], | 30.00th=[ 44], 40.00th=[ 57], 50.00th=[ 57], 60.00th=[ 67], | 70.00th=[ 121], 80.00th=[ 121], 90.00th=[ 245], 95.00th=[ 245], | 99.00th=[ 245], 99.50th=[ 245], 99.90th=[ 245], 99.95th=[ 245], | 99.99th=[ 245] bw ( KiB/s): min= 4087, max= 6144, per=33.00%, avg=5115.50, stdev=1454.52, samples=2 iops : min= 3, max= 6, avg= 4.50, stdev= 2.12, samples=2 write: IOPS=5, BW=5835KiB/s (5975kB/s)(6144KiB/1053msec) clat (msec): min=23, max=139, avg=79.71, stdev=52.05 lat (msec): min=23, max=139, avg=79.81, stdev=52.04 clat percentiles (msec): | 1.00th=[ 24], 5.00th=[ 24], 10.00th=[ 24], 20.00th=[ 28], | 30.00th=[ 28], 40.00th=[ 56], 50.00th=[ 56], 60.00th=[ 97], | 70.00th=[ 136], 80.00th=[ 136], 90.00th=[ 140], 95.00th=[ 140], | 99.00th=[ 140], 99.50th=[ 140], 99.90th=[ 140], 99.95th=[ 140], | 99.99th=[ 140] bw ( KiB/s): min= 4096, max= 8175, per=28.27%, avg=6135.50, stdev=2884.29, samples=2 iops : min= 4, max= 7, avg= 5.50, stdev= 2.12, samples=2 lat (msec) : 50=33.33%, 100=33.33%, 250=33.33% cpu : usr=0.00%, sys=5.04%, ctx=44, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22437: Thu May 1 23:38:28 2025 read: IOPS=3, BW=3876KiB/s (3969kB/s)(5120KiB/1321msec) clat (msec): min=17, max=307, avg=102.40, stdev=122.45 lat (msec): min=17, max=307, avg=102.41, stdev=122.45 clat percentiles (msec): | 1.00th=[ 18], 5.00th=[ 18], 10.00th=[ 18], 20.00th=[ 18], | 30.00th=[ 19], 40.00th=[ 19], 50.00th=[ 46], 60.00th=[ 46], | 70.00th=[ 123], 80.00th=[ 123], 90.00th=[ 309], 95.00th=[ 309], | 99.00th=[ 309], 99.50th=[ 309], 99.90th=[ 309], 99.95th=[ 309], | 99.99th=[ 309] bw ( KiB/s): min= 2048, max= 2048, per=13.21%, avg=2048.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=5, BW=5426KiB/s (5556kB/s)(7168KiB/1321msec) clat (msec): min=19, max=184, avg=96.65, stdev=62.45 lat (msec): min=19, max=184, avg=96.72, stdev=62.45 clat percentiles (msec): | 1.00th=[ 20], 5.00th=[ 20], 10.00th=[ 20], 20.00th=[ 40], | 30.00th=[ 85], 40.00th=[ 85], 50.00th=[ 86], 60.00th=[ 89], | 70.00th=[ 89], 80.00th=[ 176], 90.00th=[ 186], 95.00th=[ 186], | 99.00th=[ 186], 99.50th=[ 186], 99.90th=[ 186], 99.95th=[ 186], | 99.99th=[ 186] bw ( KiB/s): min= 2048, max= 6131, per=18.84%, avg=4089.50, stdev=2887.12, samples=2 iops : min= 2, max= 5, avg= 3.50, stdev= 2.12, samples=2 lat (msec) : 20=25.00%, 50=16.67%, 100=25.00%, 250=25.00%, 500=8.33% cpu : usr=0.00%, sys=7.58%, ctx=62, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=15.1MiB/s (15.9MB/s), 1640KiB/s-5842KiB/s (1679kB/s-5982kB/s), io=20.0MiB (20.0MB), run=1053-1321msec WRITE: bw=21.2MiB/s (22.2MB/s), 4173KiB/s-8199KiB/s (4273kB/s-8395kB/s), io=28.0MiB (29.4MB), run=1053-1321msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=22447: Thu May 1 23:38:30 2025 read: IOPS=3, BW=15.4MiB/s (16.2MB/s)(12.0MiB/779msec) clat (msec): min=172, max=312, avg=233.14, stdev=71.85 lat (msec): min=172, max=312, avg=233.14, stdev=71.85 clat percentiles (msec): | 1.00th=[ 174], 5.00th=[ 174], 10.00th=[ 174], 20.00th=[ 174], | 30.00th=[ 174], 40.00th=[ 215], 50.00th=[ 215], 60.00th=[ 215], | 70.00th=[ 313], 80.00th=[ 313], 90.00th=[ 313], 95.00th=[ 313], | 99.00th=[ 313], 99.50th=[ 313], 99.90th=[ 313], 99.95th=[ 313], | 99.99th=[ 313] bw ( KiB/s): min=16384, max=16384, per=100.00%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.26%, sys=26.48%, ctx=1013, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22449: Thu May 1 23:38:30 2025 read: IOPS=0, BW=3734KiB/s (3823kB/s)(4096KiB/1097msec) clat (nsec): min=183959k, max=183959k, avg=183959491.00, stdev= 0.00 lat (nsec): min=183962k, max=183962k, avg=183961827.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 184], 5.00th=[ 184], 10.00th=[ 184], 20.00th=[ 184], | 30.00th=[ 184], 40.00th=[ 184], 50.00th=[ 184], 60.00th=[ 184], | 70.00th=[ 184], 80.00th=[ 184], 90.00th=[ 184], 95.00th=[ 184], | 99.00th=[ 184], 99.50th=[ 184], 99.90th=[ 184], 99.95th=[ 184], | 99.99th=[ 184] bw ( KiB/s): min= 8192, max= 8192, per=54.85%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=1, BW=7468KiB/s (7647kB/s)(8192KiB/1097msec) clat (msec): min=250, max=546, avg=398.72, stdev=208.97 lat (msec): min=251, max=546, avg=399.10, stdev=208.94 clat percentiles (msec): | 1.00th=[ 251], 5.00th=[ 251], 10.00th=[ 251], 20.00th=[ 251], | 30.00th=[ 251], 40.00th=[ 251], 50.00th=[ 251], 60.00th=[ 550], | 70.00th=[ 550], 80.00th=[ 550], 90.00th=[ 550], 95.00th=[ 550], | 99.00th=[ 550], 99.50th=[ 550], 99.90th=[ 550], 99.95th=[ 550], | 99.99th=[ 550] lat (msec) : 250=33.33%, 500=33.33%, 750=33.33% cpu : usr=0.00%, sys=24.09%, ctx=277, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22450: Thu May 1 23:38:30 2025 write: IOPS=2, BW=10.8MiB/s (11.3MB/s)(12.0MiB/1111msec) clat (msec): min=197, max=471, avg=364.66, stdev=146.46 lat (msec): min=198, max=473, avg=365.42, stdev=146.89 clat percentiles (msec): | 1.00th=[ 199], 5.00th=[ 199], 10.00th=[ 199], 20.00th=[ 199], | 30.00th=[ 199], 40.00th=[ 426], 50.00th=[ 426], 60.00th=[ 426], | 70.00th=[ 472], 80.00th=[ 472], 90.00th=[ 472], 95.00th=[ 472], | 99.00th=[ 472], 99.50th=[ 472], 99.90th=[ 472], 99.95th=[ 472], | 99.99th=[ 472] bw ( KiB/s): min= 8192, max= 8192, per=27.78%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=66.67% cpu : usr=0.09%, sys=30.45%, ctx=56, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22452: Thu May 1 23:38:30 2025 write: IOPS=3, BW=15.7MiB/s (16.5MB/s)(12.0MiB/764msec) clat (msec): min=120, max=386, avg=239.19, stdev=135.07 lat (msec): min=120, max=386, avg=239.46, stdev=135.07 clat percentiles (msec): | 1.00th=[ 121], 5.00th=[ 121], 10.00th=[ 121], 20.00th=[ 121], | 30.00th=[ 121], 40.00th=[ 211], 50.00th=[ 211], 60.00th=[ 211], | 70.00th=[ 388], 80.00th=[ 388], 90.00th=[ 388], 95.00th=[ 388], | 99.00th=[ 388], 99.50th=[ 388], 99.90th=[ 388], 99.95th=[ 388], | 99.99th=[ 388] bw ( KiB/s): min=16384, max=16384, per=55.55%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.00%, sys=31.98%, ctx=90, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=14.6MiB/s (15.3MB/s), 3734KiB/s-15.4MiB/s (3823kB/s-16.2MB/s), io=16.0MiB (16.8MB), run=779-1097msec WRITE: bw=28.8MiB/s (30.2MB/s), 7468KiB/s-15.7MiB/s (7647kB/s-16.5MB/s), io=32.0MiB (33.6MB), run=764-1111msec rand-rw: (groupid=0, jobs=1): err= 0: pid=22448: Thu May 1 23:38:30 2025 read: IOPS=2, BW=9416KiB/s (9642kB/s)(12.0MiB/1305msec) clat (msec): min=264, max=496, avg=397.89, stdev=119.77 lat (msec): min=264, max=496, avg=397.89, stdev=119.77 clat percentiles (msec): | 1.00th=[ 266], 5.00th=[ 266], 10.00th=[ 266], 20.00th=[ 266], | 30.00th=[ 266], 40.00th=[ 435], 50.00th=[ 435], 60.00th=[ 435], | 70.00th=[ 498], 80.00th=[ 498], 90.00th=[ 498], 95.00th=[ 498], | 99.00th=[ 498], 99.50th=[ 498], 99.90th=[ 498], 99.95th=[ 498], | 99.99th=[ 498] bw ( KiB/s): min=16286, max=16286, per=100.00%, avg=16286.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 lat (msec) : 500=100.00% cpu : usr=0.08%, sys=6.60%, ctx=47, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22451: Thu May 1 23:38:30 2025 read: IOPS=1, BW=4330KiB/s (4434kB/s)(4096KiB/946msec) clat (nsec): min=154138k, max=154138k, avg=154137751.00, stdev= 0.00 lat (nsec): min=154139k, max=154139k, avg=154139486.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 155], 5.00th=[ 155], 10.00th=[ 155], 20.00th=[ 155], | 30.00th=[ 155], 40.00th=[ 155], 50.00th=[ 155], 60.00th=[ 155], | 70.00th=[ 155], 80.00th=[ 155], 90.00th=[ 155], 95.00th=[ 155], | 99.00th=[ 155], 99.50th=[ 155], 99.90th=[ 155], 99.95th=[ 155], | 99.99th=[ 155] bw ( KiB/s): min= 8175, max= 8175, per=65.12%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=2, BW=8660KiB/s (8867kB/s)(8192KiB/946msec) clat (msec): min=228, max=541, avg=384.74, stdev=221.61 lat (msec): min=228, max=541, avg=385.09, stdev=221.58 clat percentiles (msec): | 1.00th=[ 228], 5.00th=[ 228], 10.00th=[ 228], 20.00th=[ 228], | 30.00th=[ 228], 40.00th=[ 228], 50.00th=[ 228], 60.00th=[ 542], | 70.00th=[ 542], 80.00th=[ 542], 90.00th=[ 542], 95.00th=[ 542], | 99.00th=[ 542], 99.50th=[ 542], 99.90th=[ 542], 99.95th=[ 542], | 99.99th=[ 542] lat (msec) : 250=66.67%, 750=33.33% cpu : usr=0.00%, sys=4.55%, ctx=23, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22453: Thu May 1 23:38:30 2025 write: IOPS=2, BW=9163KiB/s (9383kB/s)(12.0MiB/1341msec) clat (msec): min=280, max=465, avg=395.23, stdev=99.93 lat (msec): min=281, max=466, avg=395.92, stdev=100.31 clat percentiles (msec): | 1.00th=[ 279], 5.00th=[ 279], 10.00th=[ 279], 20.00th=[ 279], | 30.00th=[ 279], 40.00th=[ 439], 50.00th=[ 439], 60.00th=[ 439], | 70.00th=[ 464], 80.00th=[ 464], 90.00th=[ 464], 95.00th=[ 464], | 99.00th=[ 464], 99.50th=[ 464], 99.90th=[ 464], 99.95th=[ 464], | 99.99th=[ 464] bw ( KiB/s): min=16384, max=16384, per=67.05%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 500=100.00% cpu : usr=0.22%, sys=2.16%, ctx=27, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22454: Thu May 1 23:38:30 2025 write: IOPS=2, BW=11.6MiB/s (12.1MB/s)(12.0MiB/1036msec) clat (msec): min=195, max=442, avg=308.38, stdev=124.90 lat (msec): min=195, max=442, avg=308.97, stdev=124.78 clat percentiles (msec): | 1.00th=[ 194], 5.00th=[ 194], 10.00th=[ 194], 20.00th=[ 194], | 30.00th=[ 194], 40.00th=[ 288], 50.00th=[ 288], 60.00th=[ 288], | 70.00th=[ 443], 80.00th=[ 443], 90.00th=[ 443], 95.00th=[ 443], | 99.00th=[ 443], 99.50th=[ 443], 99.90th=[ 443], 99.95th=[ 443], | 99.99th=[ 443] bw ( KiB/s): min=16384, max=16384, per=67.05%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=66.67% cpu : usr=0.10%, sys=10.24%, ctx=54, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=12.3MiB/s (12.9MB/s), 4330KiB/s-9416KiB/s (4434kB/s-9642kB/s), io=16.0MiB (16.8MB), run=946-1305msec WRITE: bw=23.9MiB/s (25.0MB/s), 8660KiB/s-11.6MiB/s (8867kB/s-12.1MB/s), io=32.0MiB (33.6MB), run=946-1341msec