== sanity test 398b: DIO and buffer IO race ============== 20:31:42 (1746145902) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 2.66296 s, 18.9 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=22740: Thu May 1 20:32:29 2025 read: IOPS=35, BW=142KiB/s (145kB/s)(5992KiB/42322msec) clat (usec): min=63, max=328388, avg=16118.18, stdev=13356.93 lat (usec): min=64, max=328396, avg=16122.58, stdev=13384.54 clat percentiles (usec): | 1.00th=[ 98], 5.00th=[ 133], 10.00th=[ 5800], 20.00th=[ 8094], | 30.00th=[ 10028], 40.00th=[ 11994], 50.00th=[ 13960], 60.00th=[ 16319], | 70.00th=[ 19006], 80.00th=[ 23200], 90.00th=[ 28181], 95.00th=[ 33817], | 99.00th=[ 51119], 99.50th=[ 58983], 99.90th=[128451], 99.95th=[329253], | 99.99th=[329253] bw ( KiB/s): min= 40, max= 287, per=24.82%, avg=140.50, stdev=38.18, samples=84 iops : min= 10, max= 71, avg=34.79, stdev= 9.54, samples=84 write: IOPS=37, BW=149KiB/s (152kB/s)(6296KiB/42322msec) clat (usec): min=508, max=84968, avg=11443.69, stdev=12034.58 lat (usec): min=509, max=84969, avg=11449.31, stdev=12035.48 clat percentiles (usec): | 1.00th=[ 693], 5.00th=[ 742], 10.00th=[ 783], 20.00th=[ 988], | 30.00th=[ 2008], 40.00th=[ 4490], 50.00th=[ 8586], 60.00th=[11338], | 70.00th=[14615], 80.00th=[19530], 90.00th=[28181], 95.00th=[34866], | 99.00th=[51643], 99.50th=[60031], 99.90th=[82314], 99.95th=[85459], | 99.99th=[85459] bw ( KiB/s): min= 16, max= 264, per=24.80%, avg=147.07, stdev=43.79, samples=84 iops : min= 4, max= 66, avg=36.43, stdev=11.00, samples=84 lat (usec) : 100=0.62%, 250=2.15%, 500=0.10%, 750=2.96%, 1000=7.45% lat (msec) : 2=5.05%, 4=4.88%, 10=19.69%, 20=34.18%, 50=21.71% lat (msec) : 100=1.14%, 250=0.03%, 500=0.03% cpu : usr=0.11%, sys=10.94%, ctx=5132, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22741: Thu May 1 20:32:29 2025 read: IOPS=35, BW=141KiB/s (144kB/s)(5940KiB/42181msec) clat (usec): min=86, max=300630, avg=15474.81, stdev=12896.26 lat (usec): min=87, max=300633, avg=15477.80, stdev=12897.68 clat percentiles (usec): | 1.00th=[ 94], 5.00th=[ 122], 10.00th=[ 5604], 20.00th=[ 7701], | 30.00th=[ 9503], 40.00th=[ 11207], 50.00th=[ 13173], 60.00th=[ 15533], | 70.00th=[ 18744], 80.00th=[ 21890], 90.00th=[ 27132], 95.00th=[ 32900], | 99.00th=[ 56361], 99.50th=[ 69731], 99.90th=[ 88605], 99.95th=[299893], | 99.99th=[299893] bw ( KiB/s): min= 8, max= 224, per=24.72%, avg=139.93, stdev=41.40, samples=84 iops : min= 2, max= 56, avg=34.63, stdev=10.43, samples=84 write: IOPS=37, BW=150KiB/s (154kB/s)(6348KiB/42181msec) clat (usec): min=526, max=116558, avg=11970.44, stdev=13218.31 lat (usec): min=528, max=116559, avg=11973.28, stdev=13217.76 clat percentiles (usec): | 1.00th=[ 676], 5.00th=[ 742], 10.00th=[ 791], 20.00th=[ 971], | 30.00th=[ 1827], 40.00th=[ 3818], 50.00th=[ 8455], 60.00th=[ 11731], | 70.00th=[ 15401], 80.00th=[ 20841], 90.00th=[ 28967], 95.00th=[ 38011], | 99.00th=[ 55313], 99.50th=[ 67634], 99.90th=[107480], 99.95th=[116917], | 99.99th=[116917] bw ( KiB/s): min= 16, max= 328, per=25.17%, avg=149.27, stdev=47.28, samples=84 iops : min= 4, max= 82, avg=36.96, stdev=11.85, samples=84 lat (usec) : 100=1.30%, 250=1.86%, 500=0.07%, 750=3.26%, 1000=7.52% lat (msec) : 2=5.63%, 4=4.88%, 10=19.34%, 20=32.68%, 50=21.91% lat (msec) : 100=1.46%, 250=0.07%, 500=0.03% cpu : usr=0.10%, sys=10.80%, ctx=4981, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22742: Thu May 1 20:32:29 2025 read: IOPS=35, BW=142KiB/s (146kB/s)(6028KiB/42384msec) clat (usec): min=78, max=342905, avg=15888.01, stdev=14244.34 lat (usec): min=78, max=342905, avg=15889.05, stdev=14244.35 clat percentiles (usec): | 1.00th=[ 94], 5.00th=[ 123], 10.00th=[ 5080], 20.00th=[ 7832], | 30.00th=[ 9765], 40.00th=[ 11469], 50.00th=[ 13304], 60.00th=[ 15795], | 70.00th=[ 19006], 80.00th=[ 22938], 90.00th=[ 28181], 95.00th=[ 34341], | 99.00th=[ 56361], 99.50th=[ 65274], 99.90th=[162530], 99.95th=[341836], | 99.99th=[341836] bw ( KiB/s): min= 16, max= 263, per=24.84%, avg=140.62, stdev=39.03, samples=84 iops : min= 4, max= 65, avg=34.77, stdev= 9.71, samples=84 write: IOPS=36, BW=148KiB/s (151kB/s)(6260KiB/42384msec) clat (usec): min=519, max=94237, avg=11682.82, stdev=12650.15 lat (usec): min=519, max=94238, avg=11685.33, stdev=12649.59 clat percentiles (usec): | 1.00th=[ 668], 5.00th=[ 750], 10.00th=[ 799], 20.00th=[ 988], | 30.00th=[ 1926], 40.00th=[ 3621], 50.00th=[ 8160], 60.00th=[11207], | 70.00th=[15008], 80.00th=[21103], 90.00th=[28967], 95.00th=[34341], | 99.00th=[54264], 99.50th=[63177], 99.90th=[89654], 99.95th=[93848], | 99.99th=[93848] bw ( KiB/s): min= 32, max= 319, per=24.57%, avg=145.73, stdev=48.00, samples=84 iops : min= 8, max= 79, avg=36.06, stdev=12.00, samples=84 lat (usec) : 100=1.14%, 250=2.57%, 500=0.16%, 750=2.70%, 1000=7.65% lat (msec) : 2=5.44%, 4=5.44%, 10=18.65%, 20=31.84%, 50=22.88% lat (msec) : 100=1.46%, 250=0.03%, 500=0.03% cpu : usr=0.16%, sys=10.76%, ctx=4930, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22743: Thu May 1 20:32:29 2025 read: IOPS=35, BW=144KiB/s (147kB/s)(6052KiB/42092msec) clat (usec): min=73, max=101548, avg=15993.37, stdev=10283.03 lat (usec): min=74, max=101549, avg=15994.76, stdev=10283.14 clat percentiles (usec): | 1.00th=[ 99], 5.00th=[ 141], 10.00th=[ 6063], 20.00th=[ 8291], | 30.00th=[ 10290], 40.00th=[ 11994], 50.00th=[ 14091], 60.00th=[ 16450], | 70.00th=[ 19268], 80.00th=[ 22676], 90.00th=[ 28181], 95.00th=[ 34866], | 99.00th=[ 50070], 99.50th=[ 58983], 99.90th=[ 73925], 99.95th=[101188], | 99.99th=[101188] bw ( KiB/s): min= 70, max= 248, per=25.59%, avg=144.84, stdev=35.47, samples=83 iops : min= 17, max= 62, avg=35.89, stdev= 8.91, samples=83 write: IOPS=37, BW=148KiB/s (152kB/s)(6236KiB/42092msec) clat (usec): min=470, max=326391, avg=11345.67, stdev=14340.97 lat (usec): min=470, max=326394, avg=11348.11, stdev=14341.06 clat percentiles (usec): | 1.00th=[ 668], 5.00th=[ 734], 10.00th=[ 783], 20.00th=[ 938], | 30.00th=[ 1958], 40.00th=[ 3785], 50.00th=[ 7898], 60.00th=[ 10683], | 70.00th=[ 14353], 80.00th=[ 19530], 90.00th=[ 28443], 95.00th=[ 36439], | 99.00th=[ 49546], 99.50th=[ 59507], 99.90th=[ 87557], 99.95th=[325059], | 99.99th=[325059] bw ( KiB/s): min= 23, max= 263, per=24.86%, avg=147.40, stdev=45.57, samples=84 iops : min= 5, max= 65, avg=36.52, stdev=11.44, samples=84 lat (usec) : 100=0.55%, 250=2.15%, 500=0.13%, 750=3.52%, 1000=7.42% lat (msec) : 2=4.62%, 4=5.27%, 10=19.50%, 20=33.56%, 50=22.27% lat (msec) : 100=0.94%, 250=0.03%, 500=0.03% cpu : usr=0.16%, sys=10.94%, ctx=5104, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=567KiB/s (580kB/s), 141KiB/s-144KiB/s (144kB/s-147kB/s), io=23.4MiB (24.6MB), run=42092-42384msec WRITE: bw=593KiB/s (607kB/s), 148KiB/s-150KiB/s (151kB/s-154kB/s), io=24.6MiB (25.7MB), run=42092-42384msec rand-rw: (groupid=0, jobs=1): err= 0: pid=22744: Thu May 1 20:32:44 2025 read: IOPS=26, BW=104KiB/s (107kB/s)(5992KiB/57565msec) clat (msec): min=2, max=336, avg=14.61, stdev=16.31 lat (msec): min=2, max=336, avg=14.61, stdev=16.31 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 5], 10.00th=[ 5], 20.00th=[ 6], | 30.00th=[ 7], 40.00th=[ 8], 50.00th=[ 10], 60.00th=[ 11], | 70.00th=[ 15], 80.00th=[ 20], 90.00th=[ 34], 95.00th=[ 46], | 99.00th=[ 74], 99.50th=[ 81], 99.90th=[ 97], 99.95th=[ 338], | 99.99th=[ 338] bw ( KiB/s): min= 31, max= 303, per=24.91%, avg=103.62, stdev=65.35, samples=115 iops : min= 7, max= 75, avg=25.51, stdev=16.39, samples=115 write: IOPS=27, BW=109KiB/s (112kB/s)(6296KiB/57565msec) clat (msec): min=4, max=113, avg=22.59, stdev=15.82 lat (msec): min=4, max=113, avg=22.59, stdev=15.82 clat percentiles (msec): | 1.00th=[ 7], 5.00th=[ 9], 10.00th=[ 10], 20.00th=[ 11], | 30.00th=[ 13], 40.00th=[ 15], 50.00th=[ 17], 60.00th=[ 21], | 70.00th=[ 26], 80.00th=[ 33], 90.00th=[ 45], 95.00th=[ 54], | 99.00th=[ 80], 99.50th=[ 93], 99.90th=[ 113], 99.95th=[ 113], | 99.99th=[ 113] bw ( KiB/s): min= 8, max= 272, per=24.93%, avg=108.70, stdev=66.62, samples=115 iops : min= 2, max= 68, avg=26.78, stdev=16.74, samples=115 lat (msec) : 4=1.79%, 10=32.94%, 20=34.54%, 50=25.49%, 100=5.08% lat (msec) : 250=0.13%, 500=0.03% cpu : usr=0.05%, sys=5.32%, ctx=3480, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22745: Thu May 1 20:32:44 2025 read: IOPS=25, BW=103KiB/s (106kB/s)(5940KiB/57626msec) clat (msec): min=2, max=337, avg=14.43, stdev=15.85 lat (msec): min=2, max=337, avg=14.43, stdev=15.85 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 5], 10.00th=[ 5], 20.00th=[ 6], | 30.00th=[ 7], 40.00th=[ 8], 50.00th=[ 9], 60.00th=[ 11], | 70.00th=[ 14], 80.00th=[ 20], 90.00th=[ 33], 95.00th=[ 44], | 99.00th=[ 69], 99.50th=[ 79], 99.90th=[ 97], 99.95th=[ 338], | 99.99th=[ 338] bw ( KiB/s): min= 8, max= 320, per=24.68%, avg=102.66, stdev=67.19, samples=115 iops : min= 2, max= 80, avg=25.35, stdev=16.88, samples=115 write: IOPS=27, BW=110KiB/s (113kB/s)(6348KiB/57626msec) clat (msec): min=4, max=163, avg=22.73, stdev=16.19 lat (msec): min=4, max=163, avg=22.73, stdev=16.19 clat percentiles (msec): | 1.00th=[ 7], 5.00th=[ 9], 10.00th=[ 10], 20.00th=[ 11], | 30.00th=[ 13], 40.00th=[ 14], 50.00th=[ 17], 60.00th=[ 21], | 70.00th=[ 27], 80.00th=[ 34], 90.00th=[ 45], 95.00th=[ 57], | 99.00th=[ 77], 99.50th=[ 91], 99.90th=[ 110], 99.95th=[ 163], | 99.99th=[ 163] bw ( KiB/s): min= 24, max= 272, per=25.09%, avg=109.41, stdev=66.98, samples=115 iops : min= 6, max= 68, avg=27.03, stdev=16.85, samples=115 lat (msec) : 4=1.95%, 10=32.45%, 20=34.86%, 50=25.46%, 100=5.08% lat (msec) : 250=0.16%, 500=0.03% cpu : usr=0.12%, sys=5.14%, ctx=3423, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22746: Thu May 1 20:32:44 2025 read: IOPS=26, BW=105KiB/s (107kB/s)(6028KiB/57424msec) clat (usec): min=2900, max=97338, avg=14357.11, stdev=13776.52 lat (usec): min=2900, max=97339, avg=14358.36, stdev=13776.52 clat percentiles (usec): | 1.00th=[ 3523], 5.00th=[ 4146], 10.00th=[ 4621], 20.00th=[ 5538], | 30.00th=[ 6456], 40.00th=[ 7635], 50.00th=[ 8848], 60.00th=[11076], | 70.00th=[14222], 80.00th=[20317], 90.00th=[31327], 95.00th=[44827], | 99.00th=[69731], 99.50th=[76022], 99.90th=[96994], 99.95th=[96994], | 99.99th=[96994] bw ( KiB/s): min= 8, max= 320, per=25.04%, avg=104.15, stdev=66.84, samples=114 iops : min= 2, max= 80, avg=25.73, stdev=16.79, samples=114 write: IOPS=27, BW=109KiB/s (112kB/s)(6260KiB/57424msec) clat (msec): min=4, max=340, avg=22.78, stdev=18.06 lat (msec): min=4, max=340, avg=22.78, stdev=18.06 clat percentiles (msec): | 1.00th=[ 7], 5.00th=[ 9], 10.00th=[ 10], 20.00th=[ 11], | 30.00th=[ 13], 40.00th=[ 14], 50.00th=[ 17], 60.00th=[ 21], | 70.00th=[ 27], 80.00th=[ 34], 90.00th=[ 45], 95.00th=[ 54], | 99.00th=[ 81], 99.50th=[ 102], 99.90th=[ 148], 99.95th=[ 342], | 99.99th=[ 342] bw ( KiB/s): min= 24, max= 272, per=24.76%, avg=107.94, stdev=67.57, samples=114 iops : min= 6, max= 68, avg=26.68, stdev=17.01, samples=114 lat (msec) : 4=1.79%, 10=33.33%, 20=34.31%, 50=25.33%, 100=4.98% lat (msec) : 250=0.23%, 500=0.03% cpu : usr=0.17%, sys=5.17%, ctx=3411, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22747: Thu May 1 20:32:44 2025 read: IOPS=26, BW=105KiB/s (108kB/s)(6052KiB/57619msec) clat (usec): min=2867, max=96497, avg=14432.53, stdev=13654.98 lat (usec): min=2868, max=96498, avg=14433.67, stdev=13654.97 clat percentiles (usec): | 1.00th=[ 3425], 5.00th=[ 4178], 10.00th=[ 4752], 20.00th=[ 5669], | 30.00th=[ 6587], 40.00th=[ 7635], 50.00th=[ 9110], 60.00th=[11076], | 70.00th=[14222], 80.00th=[20055], 90.00th=[32375], 95.00th=[45351], | 99.00th=[65799], 99.50th=[78119], 99.90th=[94897], 99.95th=[96994], | 99.99th=[96994] bw ( KiB/s): min= 23, max= 319, per=25.37%, avg=105.54, stdev=68.05, samples=114 iops : min= 5, max= 79, avg=26.14, stdev=17.04, samples=114 write: IOPS=27, BW=108KiB/s (111kB/s)(6236KiB/57619msec) clat (msec): min=4, max=353, avg=22.87, stdev=18.36 lat (msec): min=4, max=353, avg=22.87, stdev=18.36 clat percentiles (msec): | 1.00th=[ 7], 5.00th=[ 9], 10.00th=[ 10], 20.00th=[ 11], | 30.00th=[ 13], 40.00th=[ 14], 50.00th=[ 17], 60.00th=[ 21], | 70.00th=[ 27], 80.00th=[ 34], 90.00th=[ 45], 95.00th=[ 55], | 99.00th=[ 82], 99.50th=[ 97], 99.90th=[ 165], 99.95th=[ 355], | 99.99th=[ 355] bw ( KiB/s): min= 31, max= 288, per=24.72%, avg=107.77, stdev=66.29, samples=115 iops : min= 7, max= 72, avg=26.69, stdev=16.60, samples=115 lat (msec) : 4=1.92%, 10=33.07%, 20=34.24%, 50=25.62%, 100=4.98% lat (msec) : 250=0.13%, 500=0.03% cpu : usr=0.12%, sys=5.30%, ctx=3495, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=417KiB/s (427kB/s), 103KiB/s-105KiB/s (106kB/s-108kB/s), io=23.4MiB (24.6MB), run=57424-57626msec WRITE: bw=436KiB/s (447kB/s), 108KiB/s-110KiB/s (111kB/s-113kB/s), io=24.6MiB (25.7MB), run=57424-57626msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=22819: Thu May 1 20:32:55 2025 read: IOPS=42, BW=676KiB/s (693kB/s)(5920KiB/8751msec) clat (usec): min=83, max=61515, avg=12929.44, stdev=9378.31 lat (usec): min=84, max=61516, avg=12930.58, stdev=9378.44 clat percentiles (usec): | 1.00th=[ 95], 5.00th=[ 111], 10.00th=[ 149], 20.00th=[ 6718], | 30.00th=[ 8455], 40.00th=[ 9765], 50.00th=[11076], 60.00th=[13304], | 70.00th=[15795], 80.00th=[19006], 90.00th=[23462], 95.00th=[29492], | 99.00th=[45876], 99.50th=[51643], 99.90th=[61604], 99.95th=[61604], | 99.99th=[61604] bw ( KiB/s): min= 384, max= 1120, per=24.12%, avg=660.29, stdev=205.23, samples=17 iops : min= 24, max= 70, avg=40.88, stdev=12.77, samples=17 write: IOPS=45, BW=728KiB/s (745kB/s)(6368KiB/8751msec) clat (usec): min=827, max=59332, avg=9779.87, stdev=9783.68 lat (usec): min=828, max=59333, avg=9784.95, stdev=9781.44 clat percentiles (usec): | 1.00th=[ 1004], 5.00th=[ 1123], 10.00th=[ 1205], 20.00th=[ 1614], | 30.00th=[ 2409], 40.00th=[ 4178], 50.00th=[ 7111], 60.00th=[ 9765], | 70.00th=[12387], 80.00th=[15926], 90.00th=[24511], 95.00th=[29754], | 99.00th=[42206], 99.50th=[58459], 99.90th=[59507], 99.95th=[59507], | 99.99th=[59507] bw ( KiB/s): min= 352, max= 1028, per=24.89%, avg=716.47, stdev=173.42, samples=17 iops : min= 22, max= 64, avg=44.41, stdev=10.86, samples=17 lat (usec) : 100=0.91%, 250=4.95%, 500=0.52%, 1000=0.52% lat (msec) : 2=13.54%, 4=6.90%, 10=24.48%, 20=31.90%, 50=15.76% lat (msec) : 100=0.52% cpu : usr=0.23%, sys=13.79%, ctx=1377, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22820: Thu May 1 20:32:55 2025 read: IOPS=42, BW=676KiB/s (692kB/s)(5744KiB/8500msec) clat (usec): min=55, max=56630, avg=12308.25, stdev=8736.65 lat (usec): min=55, max=56631, avg=12309.37, stdev=8736.72 clat percentiles (usec): | 1.00th=[ 89], 5.00th=[ 113], 10.00th=[ 141], 20.00th=[ 5932], | 30.00th=[ 7898], 40.00th=[ 9503], 50.00th=[11600], 60.00th=[13698], | 70.00th=[15401], 80.00th=[17433], 90.00th=[22414], 95.00th=[27132], | 99.00th=[46400], 99.50th=[50594], 99.90th=[56886], 99.95th=[56886], | 99.99th=[56886] bw ( KiB/s): min= 287, max= 960, per=24.02%, avg=657.44, stdev=161.34, samples=16 iops : min= 17, max= 60, avg=40.81, stdev=10.30, samples=16 write: IOPS=48, BW=770KiB/s (788kB/s)(6544KiB/8500msec) clat (usec): min=934, max=66624, avg=9645.82, stdev=9645.78 lat (usec): min=935, max=66625, avg=9648.49, stdev=9645.24 clat percentiles (usec): | 1.00th=[ 1045], 5.00th=[ 1156], 10.00th=[ 1221], 20.00th=[ 1696], | 30.00th=[ 2409], 40.00th=[ 3982], 50.00th=[ 6521], 60.00th=[ 9241], | 70.00th=[12125], 80.00th=[16712], 90.00th=[23725], 95.00th=[28443], | 99.00th=[39584], 99.50th=[48497], 99.90th=[66847], 99.95th=[66847], | 99.99th=[66847] bw ( KiB/s): min= 510, max= 1184, per=25.54%, avg=735.31, stdev=170.32, samples=16 iops : min= 31, max= 74, avg=45.69, stdev=10.89, samples=16 lat (usec) : 100=0.78%, 250=4.95%, 500=0.13%, 750=0.26%, 1000=0.39% lat (msec) : 2=12.89%, 4=8.33%, 10=26.56%, 20=31.38%, 50=13.80% lat (msec) : 100=0.52% cpu : usr=0.16%, sys=14.34%, ctx=1331, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22821: Thu May 1 20:32:55 2025 read: IOPS=44, BW=715KiB/s (733kB/s)(6256KiB/8744msec) clat (usec): min=78, max=48370, avg=11816.14, stdev=7870.10 lat (usec): min=78, max=48371, avg=11819.21, stdev=7870.24 clat percentiles (usec): | 1.00th=[ 95], 5.00th=[ 111], 10.00th=[ 139], 20.00th=[ 6063], | 30.00th=[ 7635], 40.00th=[ 9372], 50.00th=[10945], 60.00th=[12780], | 70.00th=[15270], 80.00th=[17695], 90.00th=[21365], 95.00th=[26084], | 99.00th=[35914], 99.50th=[40109], 99.90th=[48497], 99.95th=[48497], | 99.99th=[48497] bw ( KiB/s): min= 384, max= 1111, per=25.68%, avg=702.94, stdev=197.88, samples=17 iops : min= 24, max= 69, avg=43.71, stdev=12.48, samples=17 write: IOPS=43, BW=690KiB/s (706kB/s)(6032KiB/8744msec) clat (usec): min=866, max=53270, avg=10500.10, stdev=10392.67 lat (usec): min=868, max=53273, avg=10502.53, stdev=10392.36 clat percentiles (usec): | 1.00th=[ 955], 5.00th=[ 1106], 10.00th=[ 1188], 20.00th=[ 1500], | 30.00th=[ 2245], 40.00th=[ 4047], 50.00th=[ 7570], 60.00th=[ 9765], | 70.00th=[13698], 80.00th=[19006], 90.00th=[25560], 95.00th=[33424], | 99.00th=[41157], 99.50th=[46924], 99.90th=[53216], 99.95th=[53216], | 99.99th=[53216] bw ( KiB/s): min= 416, max= 992, per=23.07%, avg=664.12, stdev=175.81, samples=17 iops : min= 26, max= 62, avg=41.29, stdev=11.00, samples=17 lat (usec) : 100=1.43%, 250=5.08%, 500=0.13%, 1000=1.17% lat (msec) : 2=12.50%, 4=6.25%, 10=26.17%, 20=31.12%, 50=16.02% lat (msec) : 100=0.13% cpu : usr=0.26%, sys=13.71%, ctx=1348, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22822: Thu May 1 20:32:55 2025 read: IOPS=45, BW=728KiB/s (745kB/s)(6032KiB/8286msec) clat (usec): min=63, max=93781, avg=12354.61, stdev=9359.17 lat (usec): min=64, max=93783, avg=12355.70, stdev=9359.23 clat percentiles (usec): | 1.00th=[ 92], 5.00th=[ 113], 10.00th=[ 130], 20.00th=[ 5997], | 30.00th=[ 8029], 40.00th=[ 9765], 50.00th=[11994], 60.00th=[13829], | 70.00th=[15795], 80.00th=[17695], 90.00th=[21890], 95.00th=[25822], | 99.00th=[38536], 99.50th=[74974], 99.90th=[93848], 99.95th=[93848], | 99.99th=[93848] bw ( KiB/s): min= 383, max= 1149, per=25.75%, avg=704.81, stdev=173.90, samples=16 iops : min= 23, max= 71, avg=43.69, stdev=10.86, samples=16 write: IOPS=47, BW=755KiB/s (773kB/s)(6256KiB/8286msec) clat (usec): min=841, max=42767, avg=9197.53, stdev=8923.19 lat (usec): min=842, max=42768, avg=9200.70, stdev=8926.17 clat percentiles (usec): | 1.00th=[ 914], 5.00th=[ 1123], 10.00th=[ 1237], 20.00th=[ 1516], | 30.00th=[ 2311], 40.00th=[ 3294], 50.00th=[ 6259], 60.00th=[ 8979], | 70.00th=[12125], 80.00th=[15795], 90.00th=[22152], 95.00th=[28181], | 99.00th=[40633], 99.50th=[42730], 99.90th=[42730], 99.95th=[42730], | 99.99th=[42730] bw ( KiB/s): min= 320, max= 1181, per=25.59%, avg=736.81, stdev=217.55, samples=16 iops : min= 20, max= 73, avg=45.69, stdev=13.62, samples=16 lat (usec) : 100=0.65%, 250=5.60%, 500=0.52%, 750=0.39%, 1000=1.04% lat (msec) : 2=12.37%, 4=8.72%, 10=22.27%, 20=35.68%, 50=12.50% lat (msec) : 100=0.26% cpu : usr=0.13%, sys=14.57%, ctx=1332, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2737KiB/s (2803kB/s), 676KiB/s-728KiB/s (692kB/s-745kB/s), io=23.4MiB (24.5MB), run=8286-8751msec WRITE: bw=2880KiB/s (2949kB/s), 690KiB/s-770KiB/s (706kB/s-788kB/s), io=24.6MiB (25.8MB), run=8286-8751msec rand-rw: (groupid=0, jobs=1): err= 0: pid=22814: Thu May 1 20:32:59 2025 read: IOPS=29, BW=469KiB/s (480kB/s)(5920KiB/12626msec) clat (usec): min=3346, max=65446, avg=12248.79, stdev=9772.71 lat (usec): min=3347, max=65448, avg=12250.02, stdev=9772.70 clat percentiles (usec): | 1.00th=[ 3720], 5.00th=[ 4490], 10.00th=[ 5014], 20.00th=[ 6325], | 30.00th=[ 7177], 40.00th=[ 7832], 50.00th=[ 8717], 60.00th=[ 9896], | 70.00th=[11600], 80.00th=[15926], 90.00th=[25560], 95.00th=[33817], | 99.00th=[52167], 99.50th=[63177], 99.90th=[65274], 99.95th=[65274], | 99.99th=[65274] bw ( KiB/s): min= 159, max= 1056, per=25.13%, avg=466.60, stdev=212.28, samples=25 iops : min= 9, max= 66, avg=28.72, stdev=13.51, samples=25 write: IOPS=31, BW=504KiB/s (516kB/s)(6368KiB/12626msec) clat (usec): min=5209, max=81645, avg=20251.48, stdev=12946.42 lat (usec): min=5210, max=81648, avg=20254.05, stdev=12946.29 clat percentiles (usec): | 1.00th=[ 5997], 5.00th=[ 8586], 10.00th=[ 9634], 20.00th=[11076], | 30.00th=[12125], 40.00th=[13304], 50.00th=[15270], 60.00th=[18220], | 70.00th=[22152], 80.00th=[28967], 90.00th=[38536], 95.00th=[47449], | 99.00th=[69731], 99.50th=[72877], 99.90th=[81265], 99.95th=[81265], | 99.99th=[81265] bw ( KiB/s): min= 256, max= 960, per=25.84%, avg=504.92, stdev=228.02, samples=25 iops : min= 16, max= 60, avg=31.12, stdev=14.40, samples=25 lat (msec) : 4=0.91%, 10=34.64%, 20=39.32%, 50=22.66%, 100=2.47% cpu : usr=0.10%, sys=6.20%, ctx=883, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22815: Thu May 1 20:32:59 2025 read: IOPS=27, BW=446KiB/s (456kB/s)(5744KiB/12893msec) clat (usec): min=3407, max=69095, avg=12242.11, stdev=10297.67 lat (usec): min=3408, max=69096, avg=12243.44, stdev=10297.65 clat percentiles (usec): | 1.00th=[ 3687], 5.00th=[ 4424], 10.00th=[ 4948], 20.00th=[ 5735], | 30.00th=[ 6587], 40.00th=[ 7308], 50.00th=[ 8455], 60.00th=[ 9896], | 70.00th=[11469], 80.00th=[16057], 90.00th=[25560], 95.00th=[34866], | 99.00th=[53216], 99.50th=[63177], 99.90th=[68682], 99.95th=[68682], | 99.99th=[68682] bw ( KiB/s): min= 128, max= 1152, per=24.10%, avg=447.48, stdev=245.20, samples=25 iops : min= 8, max= 72, avg=27.64, stdev=15.45, samples=25 write: IOPS=31, BW=508KiB/s (520kB/s)(6544KiB/12893msec) clat (usec): min=5711, max=73815, avg=20595.66, stdev=11854.61 lat (usec): min=5713, max=73818, avg=20598.86, stdev=11855.31 clat percentiles (usec): | 1.00th=[ 6521], 5.00th=[ 8979], 10.00th=[ 9896], 20.00th=[11469], | 30.00th=[12780], 40.00th=[14484], 50.00th=[16450], 60.00th=[19792], | 70.00th=[22938], 80.00th=[28443], 90.00th=[38536], 95.00th=[46924], | 99.00th=[55837], 99.50th=[58459], 99.90th=[73925], 99.95th=[73925], | 99.99th=[73925] bw ( KiB/s): min= 223, max= 1024, per=25.99%, avg=507.76, stdev=220.08, samples=25 iops : min= 13, max= 64, avg=31.40, stdev=13.99, samples=25 lat (msec) : 4=0.78%, 10=33.85%, 20=37.50%, 50=25.26%, 100=2.60% cpu : usr=0.04%, sys=6.09%, ctx=891, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22816: Thu May 1 20:32:59 2025 read: IOPS=30, BW=487KiB/s (499kB/s)(6256KiB/12836msec) clat (usec): min=2954, max=54019, avg=12935.50, stdev=10539.05 lat (usec): min=2955, max=54020, avg=12936.74, stdev=10539.02 clat percentiles (usec): | 1.00th=[ 3392], 5.00th=[ 4424], 10.00th=[ 4948], 20.00th=[ 5866], | 30.00th=[ 6587], 40.00th=[ 7308], 50.00th=[ 8455], 60.00th=[10028], | 70.00th=[13566], 80.00th=[17957], 90.00th=[30540], 95.00th=[38536], | 99.00th=[48497], 99.50th=[50070], 99.90th=[54264], 99.95th=[54264], | 99.99th=[54264] bw ( KiB/s): min= 160, max= 1245, per=26.37%, avg=489.64, stdev=285.14, samples=25 iops : min= 10, max= 77, avg=30.28, stdev=17.75, samples=25 write: IOPS=29, BW=470KiB/s (481kB/s)(6032KiB/12836msec) clat (usec): min=5503, max=83261, avg=20389.07, stdev=13057.11 lat (usec): min=5506, max=83264, avg=20392.67, stdev=13059.03 clat percentiles (usec): | 1.00th=[ 6390], 5.00th=[ 8356], 10.00th=[ 9503], 20.00th=[11207], | 30.00th=[12518], 40.00th=[13960], 50.00th=[15664], 60.00th=[18482], | 70.00th=[22152], 80.00th=[27657], 90.00th=[38011], 95.00th=[47973], | 99.00th=[73925], 99.50th=[81265], 99.90th=[83362], 99.95th=[83362], | 99.99th=[83362] bw ( KiB/s): min= 224, max= 926, per=23.88%, avg=466.68, stdev=212.93, samples=25 iops : min= 14, max= 57, avg=28.84, stdev=13.23, samples=25 lat (msec) : 4=0.91%, 10=35.29%, 20=38.15%, 50=23.57%, 100=2.08% cpu : usr=0.00%, sys=6.16%, ctx=896, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22817: Thu May 1 20:32:59 2025 read: IOPS=29, BW=474KiB/s (486kB/s)(6032KiB/12717msec) clat (usec): min=2963, max=69191, avg=12877.33, stdev=10657.34 lat (usec): min=2965, max=69192, avg=12881.92, stdev=10658.14 clat percentiles (usec): | 1.00th=[ 3589], 5.00th=[ 4621], 10.00th=[ 4948], 20.00th=[ 5997], | 30.00th=[ 6849], 40.00th=[ 7701], 50.00th=[ 8586], 60.00th=[10290], | 70.00th=[12780], 80.00th=[17695], 90.00th=[26346], 95.00th=[39060], | 99.00th=[52167], 99.50th=[60556], 99.90th=[68682], 99.95th=[68682], | 99.99th=[68682] bw ( KiB/s): min= 224, max= 1085, per=25.47%, avg=472.92, stdev=212.83, samples=25 iops : min= 14, max= 67, avg=29.24, stdev=13.13, samples=25 write: IOPS=30, BW=492KiB/s (504kB/s)(6256KiB/12717msec) clat (usec): min=5964, max=78082, avg=19951.77, stdev=12117.32 lat (usec): min=5965, max=78084, avg=19957.65, stdev=12115.76 clat percentiles (usec): | 1.00th=[ 6652], 5.00th=[ 8848], 10.00th=[ 9634], 20.00th=[10945], | 30.00th=[12387], 40.00th=[13960], 50.00th=[15664], 60.00th=[18220], | 70.00th=[21890], 80.00th=[26870], 90.00th=[34866], 95.00th=[45351], | 99.00th=[69731], 99.50th=[70779], 99.90th=[78119], 99.95th=[78119], | 99.99th=[78119] bw ( KiB/s): min= 223, max= 960, per=25.32%, avg=494.72, stdev=221.05, samples=25 iops : min= 13, max= 60, avg=30.60, stdev=13.74, samples=25 lat (msec) : 4=0.78%, 10=34.64%, 20=38.28%, 50=23.57%, 100=2.73% cpu : usr=0.09%, sys=6.04%, ctx=891, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1858KiB/s (1902kB/s), 446KiB/s-487KiB/s (456kB/s-499kB/s), io=23.4MiB (24.5MB), run=12626-12893msec WRITE: bw=1955KiB/s (2001kB/s), 470KiB/s-508KiB/s (481kB/s-520kB/s), io=24.6MiB (25.8MB), run=12626-12893msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=22838: Thu May 1 20:33:03 2025 read: IOPS=2, BW=3007KiB/s (3079kB/s)(7168KiB/2384msec) clat (msec): min=4, max=1452, avg=270.28, stdev=522.49 lat (msec): min=4, max=1452, avg=270.29, stdev=522.49 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 5], 10.00th=[ 5], 20.00th=[ 62], | 30.00th=[ 70], 40.00th=[ 70], 50.00th=[ 83], 60.00th=[ 91], | 70.00th=[ 91], 80.00th=[ 131], 90.00th=[ 1452], 95.00th=[ 1452], | 99.00th=[ 1452], 99.50th=[ 1452], 99.90th=[ 1452], 99.95th=[ 1452], | 99.99th=[ 1452] bw ( KiB/s): min= 2043, max= 8000, per=72.85%, avg=5021.50, stdev=4212.24, samples=2 iops : min= 1, max= 7, avg= 4.00, stdev= 4.24, samples=2 write: IOPS=2, BW=2147KiB/s (2198kB/s)(5120KiB/2385msec) clat (msec): min=46, max=158, avg=84.38, stdev=47.05 lat (msec): min=46, max=158, avg=84.46, stdev=47.07 clat percentiles (msec): | 1.00th=[ 46], 5.00th=[ 46], 10.00th=[ 46], 20.00th=[ 46], | 30.00th=[ 47], 40.00th=[ 47], 50.00th=[ 69], 60.00th=[ 69], | 70.00th=[ 102], 80.00th=[ 102], 90.00th=[ 159], 95.00th=[ 159], | 99.00th=[ 159], 99.50th=[ 159], 99.90th=[ 159], 99.95th=[ 159], | 99.99th=[ 159] bw ( KiB/s): min= 2000, max= 2000, per=20.73%, avg=2000.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 10=8.33%, 50=16.67%, 100=41.67%, 250=25.00% cpu : usr=0.17%, sys=15.94%, ctx=613, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22843: Thu May 1 20:33:03 2025 read: IOPS=0, BW=748KiB/s (766kB/s)(2048KiB/2738msec) clat (msec): min=65, max=187, avg=126.66, stdev=86.48 lat (msec): min=65, max=187, avg=126.66, stdev=86.48 clat percentiles (msec): | 1.00th=[ 66], 5.00th=[ 66], 10.00th=[ 66], 20.00th=[ 66], | 30.00th=[ 66], 40.00th=[ 66], 50.00th=[ 66], 60.00th=[ 188], | 70.00th=[ 188], 80.00th=[ 188], 90.00th=[ 188], 95.00th=[ 188], | 99.00th=[ 188], 99.50th=[ 188], 99.90th=[ 188], 99.95th=[ 188], | 99.99th=[ 188] bw ( KiB/s): min= 2048, max= 2048, per=29.71%, avg=2048.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 write: IOPS=3, BW=3740KiB/s (3830kB/s)(10.0MiB/2738msec) clat (msec): min=34, max=1537, avg=241.32, stdev=458.15 lat (msec): min=35, max=1537, avg=241.41, stdev=458.14 clat percentiles (msec): | 1.00th=[ 35], 5.00th=[ 35], 10.00th=[ 35], 20.00th=[ 37], | 30.00th=[ 59], 40.00th=[ 77], 50.00th=[ 91], 60.00th=[ 101], | 70.00th=[ 159], 80.00th=[ 159], 90.00th=[ 188], 95.00th=[ 1536], | 99.00th=[ 1536], 99.50th=[ 1536], 99.90th=[ 1536], 99.95th=[ 1536], | 99.99th=[ 1536] bw ( KiB/s): min= 6023, max=10240, per=84.26%, avg=8131.50, stdev=2981.87, samples=2 iops : min= 5, max= 10, avg= 7.50, stdev= 3.54, samples=2 lat (msec) : 50=16.67%, 100=33.33%, 250=41.67% cpu : usr=0.00%, sys=9.46%, ctx=298, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22844: Thu May 1 20:33:03 2025 read: IOPS=2, BW=2429KiB/s (2488kB/s)(6144KiB/2529msec) clat (usec): min=709, max=233757, avg=72793.97, stdev=85588.21 lat (usec): min=712, max=233759, avg=72795.89, stdev=85588.16 clat percentiles (usec): | 1.00th=[ 709], 5.00th=[ 709], 10.00th=[ 709], 20.00th=[ 3130], | 30.00th=[ 3130], 40.00th=[ 49546], 50.00th=[ 49546], 60.00th=[ 66323], | 70.00th=[ 82314], 80.00th=[ 82314], 90.00th=[233833], 95.00th=[233833], | 99.00th=[233833], 99.50th=[233833], 99.90th=[233833], 99.95th=[233833], | 99.99th=[233833] bw ( KiB/s): min= 4096, max= 6144, per=74.28%, avg=5120.00, stdev=1448.15, samples=2 iops : min= 4, max= 6, avg= 5.00, stdev= 1.41, samples=2 write: IOPS=2, BW=2429KiB/s (2488kB/s)(6144KiB/2529msec) clat (msec): min=60, max=1604, avg=346.20, stdev=617.20 lat (msec): min=60, max=1604, avg=346.70, stdev=616.98 clat percentiles (msec): | 1.00th=[ 61], 5.00th=[ 61], 10.00th=[ 61], 20.00th=[ 70], | 30.00th=[ 70], 40.00th=[ 73], 50.00th=[ 73], 60.00th=[ 120], | 70.00th=[ 153], 80.00th=[ 153], 90.00th=[ 1603], 95.00th=[ 1603], | 99.00th=[ 1603], 99.50th=[ 1603], 99.90th=[ 1603], 99.95th=[ 1603], | 99.99th=[ 1603] bw ( KiB/s): min= 4096, max= 4096, per=42.45%, avg=4096.00, stdev= 0.00, samples=3 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=3 lat (usec) : 750=8.33% lat (msec) : 4=8.33%, 50=8.33%, 100=41.67%, 250=25.00% cpu : usr=0.04%, sys=9.18%, ctx=169, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22845: Thu May 1 20:33:03 2025 read: IOPS=1, BW=1723KiB/s (1765kB/s)(5120KiB/2971msec) clat (usec): min=594, max=77163, avg=32870.19, stdev=37788.84 lat (usec): min=596, max=77165, avg=32871.90, stdev=37788.77 clat percentiles (usec): | 1.00th=[ 594], 5.00th=[ 594], 10.00th=[ 594], 20.00th=[ 594], | 30.00th=[ 1549], 40.00th=[ 1549], 50.00th=[14746], 60.00th=[14746], | 70.00th=[70779], 80.00th=[70779], 90.00th=[77071], 95.00th=[77071], | 99.00th=[77071], 99.50th=[77071], 99.90th=[77071], 99.95th=[77071], | 99.99th=[77071] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=2, BW=2413KiB/s (2471kB/s)(7168KiB/2971msec) clat (msec): min=71, max=1593, avg=391.15, stdev=547.36 lat (msec): min=71, max=1593, avg=391.24, stdev=547.37 clat percentiles (msec): | 1.00th=[ 71], 5.00th=[ 71], 10.00th=[ 71], 20.00th=[ 113], | 30.00th=[ 123], 40.00th=[ 123], 50.00th=[ 157], 60.00th=[ 203], | 70.00th=[ 203], 80.00th=[ 481], 90.00th=[ 1586], 95.00th=[ 1586], | 99.00th=[ 1586], 99.50th=[ 1586], 99.90th=[ 1586], 99.95th=[ 1586], | 99.99th=[ 1586] bw ( KiB/s): min= 2015, max= 6144, per=35.21%, avg=3398.00, stdev=2378.13, samples=3 iops : min= 1, max= 6, avg= 2.67, stdev= 2.89, samples=3 lat (usec) : 750=8.33% lat (msec) : 2=8.33%, 20=8.33%, 100=25.00%, 250=33.33%, 500=8.33% cpu : usr=0.00%, sys=5.62%, ctx=288, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=6893KiB/s (7059kB/s), 748KiB/s-3007KiB/s (766kB/s-3079kB/s), io=20.0MiB (20.0MB), run=2384-2971msec WRITE: bw=9651KiB/s (9882kB/s), 2147KiB/s-3740KiB/s (2198kB/s-3830kB/s), io=28.0MiB (29.4MB), run=2385-2971msec rand-rw: (groupid=0, jobs=1): err= 0: pid=22839: Thu May 1 20:33:03 2025 read: IOPS=2, BW=2227KiB/s (2281kB/s)(7168KiB/3218msec) clat (msec): min=21, max=1697, avg=338.64, stdev=610.56 lat (msec): min=21, max=1697, avg=338.64, stdev=610.56 clat percentiles (msec): | 1.00th=[ 22], 5.00th=[ 22], 10.00th=[ 22], 20.00th=[ 36], | 30.00th=[ 45], 40.00th=[ 45], 50.00th=[ 65], 60.00th=[ 150], | 70.00th=[ 150], 80.00th=[ 355], 90.00th=[ 1703], 95.00th=[ 1703], | 99.00th=[ 1703], 99.50th=[ 1703], 99.90th=[ 1703], 99.95th=[ 1703], | 99.99th=[ 1703] bw ( KiB/s): min= 2043, max= 4096, per=53.56%, avg=3408.67, stdev=1182.71, samples=3 iops : min= 1, max= 4, avg= 2.67, stdev= 1.53, samples=3 write: IOPS=1, BW=1591KiB/s (1629kB/s)(5120KiB/3218msec) clat (msec): min=58, max=218, avg=135.82, stdev=64.02 lat (msec): min=58, max=218, avg=135.96, stdev=63.98 clat percentiles (msec): | 1.00th=[ 59], 5.00th=[ 59], 10.00th=[ 59], 20.00th=[ 59], | 30.00th=[ 87], 40.00th=[ 87], 50.00th=[ 155], 60.00th=[ 155], | 70.00th=[ 163], 80.00th=[ 163], 90.00th=[ 220], 95.00th=[ 220], | 99.00th=[ 220], 99.50th=[ 220], 99.90th=[ 220], 99.95th=[ 220], | 99.99th=[ 220] bw ( KiB/s): min= 2048, max= 4087, per=34.43%, avg=3067.50, stdev=1441.79, samples=2 iops : min= 2, max= 3, avg= 2.50, stdev= 0.71, samples=2 lat (msec) : 50=25.00%, 100=25.00%, 250=33.33%, 500=8.33% cpu : usr=0.12%, sys=2.67%, ctx=48, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22840: Thu May 1 20:33:03 2025 read: IOPS=0, BW=640KiB/s (655kB/s)(2048KiB/3201msec) clat (msec): min=66, max=163, avg=114.67, stdev=68.55 lat (msec): min=66, max=163, avg=114.67, stdev=68.55 clat percentiles (msec): | 1.00th=[ 67], 5.00th=[ 67], 10.00th=[ 67], 20.00th=[ 67], | 30.00th=[ 67], 40.00th=[ 67], 50.00th=[ 67], 60.00th=[ 163], | 70.00th=[ 163], 80.00th=[ 163], 90.00th=[ 163], 95.00th=[ 163], | 99.00th=[ 163], 99.50th=[ 163], 99.90th=[ 163], 99.95th=[ 163], | 99.99th=[ 163] bw ( KiB/s): min= 2039, max= 2048, per=32.11%, avg=2043.50, stdev= 6.36, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 write: IOPS=3, BW=3199KiB/s (3276kB/s)(10.0MiB/3201msec) clat (msec): min=25, max=1626, avg=282.32, stdev=486.23 lat (msec): min=25, max=1626, avg=282.46, stdev=486.23 clat percentiles (msec): | 1.00th=[ 26], 5.00th=[ 26], 10.00th=[ 26], 20.00th=[ 29], | 30.00th=[ 66], 40.00th=[ 92], 50.00th=[ 104], 60.00th=[ 126], | 70.00th=[ 184], 80.00th=[ 184], 90.00th=[ 430], 95.00th=[ 1620], | 99.00th=[ 1620], 99.50th=[ 1620], 99.90th=[ 1620], 99.95th=[ 1620], | 99.99th=[ 1620] bw ( KiB/s): min= 2043, max= 6119, per=45.50%, avg=4054.00, stdev=2038.54, samples=3 iops : min= 1, max= 5, avg= 3.00, stdev= 2.00, samples=3 lat (msec) : 50=16.67%, 100=25.00%, 250=41.67%, 500=8.33% cpu : usr=0.00%, sys=2.28%, ctx=51, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22841: Thu May 1 20:33:03 2025 read: IOPS=1, BW=2044KiB/s (2093kB/s)(6144KiB/3006msec) clat (msec): min=24, max=200, avg=66.84, stdev=67.48 lat (msec): min=24, max=200, avg=66.84, stdev=67.48 clat percentiles (msec): | 1.00th=[ 26], 5.00th=[ 26], 10.00th=[ 26], 20.00th=[ 32], | 30.00th=[ 32], 40.00th=[ 33], 50.00th=[ 33], 60.00th=[ 42], | 70.00th=[ 70], 80.00th=[ 70], 90.00th=[ 201], 95.00th=[ 201], | 99.00th=[ 201], 99.50th=[ 201], 99.90th=[ 201], 99.95th=[ 201], | 99.99th=[ 201] bw ( KiB/s): min= 4096, max= 6131, per=80.35%, avg=5113.50, stdev=1438.96, samples=2 iops : min= 4, max= 5, avg= 4.50, stdev= 0.71, samples=2 write: IOPS=1, BW=2044KiB/s (2093kB/s)(6144KiB/3006msec) clat (msec): min=88, max=1729, avg=413.48, stdev=648.91 lat (msec): min=88, max=1729, avg=413.57, stdev=648.92 clat percentiles (msec): | 1.00th=[ 89], 5.00th=[ 89], 10.00th=[ 89], 20.00th=[ 90], | 30.00th=[ 90], 40.00th=[ 136], 50.00th=[ 136], 60.00th=[ 144], | 70.00th=[ 296], 80.00th=[ 296], 90.00th=[ 1737], 95.00th=[ 1737], | 99.00th=[ 1737], 99.50th=[ 1737], 99.90th=[ 1737], 99.95th=[ 1737], | 99.99th=[ 1737] bw ( KiB/s): min= 2048, max= 6131, per=45.93%, avg=4091.67, stdev=2041.50, samples=3 iops : min= 2, max= 5, avg= 3.67, stdev= 1.53, samples=3 lat (msec) : 50=33.33%, 100=25.00%, 250=25.00%, 500=8.33% cpu : usr=0.00%, sys=2.60%, ctx=66, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22842: Thu May 1 20:33:03 2025 read: IOPS=1, BW=1638KiB/s (1677kB/s)(5120KiB/3126msec) clat (msec): min=29, max=171, avg=92.63, stdev=72.13 lat (msec): min=29, max=171, avg=92.64, stdev=72.13 clat percentiles (msec): | 1.00th=[ 30], 5.00th=[ 30], 10.00th=[ 30], 20.00th=[ 30], | 30.00th=[ 44], 40.00th=[ 44], 50.00th=[ 48], 60.00th=[ 48], | 70.00th=[ 171], 80.00th=[ 171], 90.00th=[ 171], 95.00th=[ 171], | 99.00th=[ 171], 99.50th=[ 171], 99.90th=[ 171], 99.95th=[ 171], | 99.99th=[ 171] bw ( KiB/s): min= 2043, max= 6144, per=64.32%, avg=4093.50, stdev=2899.84, samples=2 iops : min= 1, max= 6, avg= 3.50, stdev= 3.54, samples=2 write: IOPS=2, BW=2293KiB/s (2348kB/s)(7168KiB/3126msec) clat (msec): min=65, max=1566, avg=375.33, stdev=532.27 lat (msec): min=65, max=1566, avg=375.44, stdev=532.25 clat percentiles (msec): | 1.00th=[ 66], 5.00th=[ 66], 10.00th=[ 66], 20.00th=[ 110], | 30.00th=[ 180], 40.00th=[ 180], 50.00th=[ 182], 60.00th=[ 186], | 70.00th=[ 186], 80.00th=[ 338], 90.00th=[ 1569], 95.00th=[ 1569], | 99.00th=[ 1569], 99.50th=[ 1569], 99.90th=[ 1569], 99.95th=[ 1569], | 99.99th=[ 1569] bw ( KiB/s): min= 2048, max= 4096, per=34.46%, avg=3069.75, stdev=1179.82, samples=4 iops : min= 2, max= 4, avg= 2.75, stdev= 0.96, samples=4 lat (msec) : 50=25.00%, 100=8.33%, 250=50.00%, 500=8.33% cpu : usr=0.06%, sys=2.27%, ctx=103, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=6364KiB/s (6517kB/s), 640KiB/s-2227KiB/s (655kB/s-2281kB/s), io=20.0MiB (20.0MB), run=3006-3218msec WRITE: bw=8910KiB/s (9124kB/s), 1591KiB/s-3199KiB/s (1629kB/s-3276kB/s), io=28.0MiB (29.4MB), run=3006-3218msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=22856: Thu May 1 20:33:07 2025 read: IOPS=4, BW=19.7MiB/s (20.6MB/s)(12.0MiB/610msec) clat (msec): min=2, max=331, avg=195.31, stdev=172.02 lat (msec): min=2, max=331, avg=195.31, stdev=172.02 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 3], 20.00th=[ 3], | 30.00th=[ 3], 40.00th=[ 253], 50.00th=[ 253], 60.00th=[ 253], | 70.00th=[ 334], 80.00th=[ 334], 90.00th=[ 334], 95.00th=[ 334], | 99.00th=[ 334], 99.50th=[ 334], 99.90th=[ 334], 99.95th=[ 334], | 99.99th=[ 334] bw ( KiB/s): min= 8192, max= 8192, per=64.95%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 4=33.33%, 500=66.67% cpu : usr=0.33%, sys=22.82%, ctx=493, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22858: Thu May 1 20:33:07 2025 read: IOPS=0, BW=3153KiB/s (3229kB/s)(4096KiB/1299msec) clat (nsec): min=188856k, max=188856k, avg=188856152.00, stdev= 0.00 lat (nsec): min=188859k, max=188859k, avg=188859123.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 190], 5.00th=[ 190], 10.00th=[ 190], 20.00th=[ 190], | 30.00th=[ 190], 40.00th=[ 190], 50.00th=[ 190], 60.00th=[ 190], | 70.00th=[ 190], 80.00th=[ 190], 90.00th=[ 190], 95.00th=[ 190], | 99.00th=[ 190], 99.50th=[ 190], 99.90th=[ 190], 99.95th=[ 190], | 99.99th=[ 190] bw ( KiB/s): min= 8192, max= 8192, per=64.95%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=1, BW=6306KiB/s (6458kB/s)(8192KiB/1299msec) clat (msec): min=225, max=723, avg=474.48, stdev=351.78 lat (msec): min=226, max=728, avg=477.36, stdev=355.36 clat percentiles (msec): | 1.00th=[ 226], 5.00th=[ 226], 10.00th=[ 226], 20.00th=[ 226], | 30.00th=[ 226], 40.00th=[ 226], 50.00th=[ 226], 60.00th=[ 726], | 70.00th=[ 726], 80.00th=[ 726], 90.00th=[ 726], 95.00th=[ 726], | 99.00th=[ 726], 99.50th=[ 726], 99.90th=[ 726], 99.95th=[ 726], | 99.99th=[ 726] bw ( KiB/s): min= 8094, max= 8094, per=41.25%, avg=8094.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 750=33.33% cpu : usr=0.00%, sys=16.95%, ctx=282, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22860: Thu May 1 20:33:07 2025 write: IOPS=1, BW=7358KiB/s (7535kB/s)(12.0MiB/1670msec) clat (msec): min=175, max=793, avg=538.90, stdev=323.27 lat (msec): min=179, max=793, avg=540.63, stdev=321.00 clat percentiles (msec): | 1.00th=[ 176], 5.00th=[ 176], 10.00th=[ 176], 20.00th=[ 176], | 30.00th=[ 176], 40.00th=[ 651], 50.00th=[ 651], 60.00th=[ 651], | 70.00th=[ 793], 80.00th=[ 793], 90.00th=[ 793], 95.00th=[ 793], | 99.00th=[ 793], 99.50th=[ 793], 99.90th=[ 793], 99.95th=[ 793], | 99.99th=[ 793] bw ( KiB/s): min= 8192, max= 8192, per=41.75%, avg=8192.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 lat (msec) : 250=33.33%, 750=33.33%, 1000=33.33% cpu : usr=0.00%, sys=19.53%, ctx=83, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22861: Thu May 1 20:33:07 2025 write: IOPS=2, BW=8684KiB/s (8893kB/s)(12.0MiB/1415msec) clat (msec): min=237, max=748, avg=440.88, stdev=271.11 lat (msec): min=238, max=749, avg=441.26, stdev=271.07 clat percentiles (msec): | 1.00th=[ 239], 5.00th=[ 239], 10.00th=[ 239], 20.00th=[ 239], | 30.00th=[ 239], 40.00th=[ 338], 50.00th=[ 338], 60.00th=[ 338], | 70.00th=[ 751], 80.00th=[ 751], 90.00th=[ 751], 95.00th=[ 751], | 99.00th=[ 751], 99.50th=[ 751], 99.90th=[ 751], 99.95th=[ 751], | 99.99th=[ 751] bw ( KiB/s): min= 8192, max= 8192, per=41.75%, avg=8192.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 lat (msec) : 250=33.33%, 500=33.33%, 750=33.33% cpu : usr=0.00%, sys=18.95%, ctx=129, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=12.3MiB/s (12.9MB/s), 3153KiB/s-19.7MiB/s (3229kB/s-20.6MB/s), io=16.0MiB (16.8MB), run=610-1299msec WRITE: bw=19.2MiB/s (20.1MB/s), 6306KiB/s-8684KiB/s (6458kB/s-8893kB/s), io=32.0MiB (33.6MB), run=1299-1670msec rand-rw: (groupid=0, jobs=1): err= 0: pid=22854: Thu May 1 20:33:07 2025 read: IOPS=2, BW=10.1MiB/s (10.6MB/s)(12.0MiB/1187msec) clat (msec): min=158, max=689, avg=384.38, stdev=274.05 lat (msec): min=158, max=689, avg=384.38, stdev=274.05 clat percentiles (msec): | 1.00th=[ 159], 5.00th=[ 159], 10.00th=[ 159], 20.00th=[ 159], | 30.00th=[ 159], 40.00th=[ 305], 50.00th=[ 305], 60.00th=[ 305], | 70.00th=[ 693], 80.00th=[ 693], 90.00th=[ 693], 95.00th=[ 693], | 99.00th=[ 693], 99.50th=[ 693], 99.90th=[ 693], 99.95th=[ 693], | 99.99th=[ 693] bw ( KiB/s): min= 8143, max= 8143, per=90.31%, avg=8143.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=33.33%, 750=33.33% cpu : usr=0.00%, sys=5.31%, ctx=29, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22855: Thu May 1 20:33:07 2025 read: IOPS=0, BW=2254KiB/s (2308kB/s)(4096KiB/1817msec) clat (nsec): min=1030.7M, max=1030.7M, avg=1030689107.00, stdev= 0.00 lat (nsec): min=1030.7M, max=1030.7M, avg=1030696826.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 1028], 5.00th=[ 1028], 10.00th=[ 1028], 20.00th=[ 1028], | 30.00th=[ 1028], 40.00th=[ 1028], 50.00th=[ 1028], 60.00th=[ 1028], | 70.00th=[ 1028], 80.00th=[ 1028], 90.00th=[ 1028], 95.00th=[ 1028], | 99.00th=[ 1028], 99.50th=[ 1028], 99.90th=[ 1028], 99.95th=[ 1028], | 99.99th=[ 1028] bw ( KiB/s): min= 6410, max= 6410, per=71.09%, avg=6410.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=1, BW=4509KiB/s (4617kB/s)(8192KiB/1817msec) clat (msec): min=152, max=547, avg=350.29, stdev=279.22 lat (msec): min=153, max=548, avg=350.64, stdev=279.17 clat percentiles (msec): | 1.00th=[ 153], 5.00th=[ 153], 10.00th=[ 153], 20.00th=[ 153], | 30.00th=[ 153], 40.00th=[ 153], 50.00th=[ 153], 60.00th=[ 550], | 70.00th=[ 550], 80.00th=[ 550], 90.00th=[ 550], 95.00th=[ 550], | 99.00th=[ 550], 99.50th=[ 550], 99.90th=[ 550], 99.95th=[ 550], | 99.99th=[ 550] bw ( KiB/s): min= 6410, max= 6410, per=39.61%, avg=6410.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 750=33.33% cpu : usr=0.17%, sys=1.32%, ctx=13, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22857: Thu May 1 20:33:07 2025 write: IOPS=1, BW=6068KiB/s (6214kB/s)(12.0MiB/2025msec) clat (msec): min=277, max=946, avg=650.68, stdev=341.35 lat (msec): min=277, max=947, avg=651.14, stdev=341.20 clat percentiles (msec): | 1.00th=[ 279], 5.00th=[ 279], 10.00th=[ 279], 20.00th=[ 279], | 30.00th=[ 279], 40.00th=[ 726], 50.00th=[ 726], 60.00th=[ 726], | 70.00th=[ 944], 80.00th=[ 944], 90.00th=[ 944], 95.00th=[ 944], | 99.00th=[ 944], 99.50th=[ 944], 99.90th=[ 944], 99.95th=[ 944], | 99.99th=[ 944] bw ( KiB/s): min= 6370, max= 6370, per=39.37%, avg=6370.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 500=33.33%, 750=33.33%, 1000=33.33% cpu : usr=0.15%, sys=7.02%, ctx=53, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22859: Thu May 1 20:33:07 2025 write: IOPS=2, BW=9894KiB/s (10.1MB/s)(12.0MiB/1242msec) clat (msec): min=70, max=980, avg=407.26, stdev=498.70 lat (msec): min=71, max=980, avg=407.68, stdev=498.58 clat percentiles (msec): | 1.00th=[ 71], 5.00th=[ 71], 10.00th=[ 71], 20.00th=[ 71], | 30.00th=[ 71], 40.00th=[ 171], 50.00th=[ 171], 60.00th=[ 171], | 70.00th=[ 978], 80.00th=[ 978], 90.00th=[ 978], 95.00th=[ 978], | 99.00th=[ 978], 99.50th=[ 978], 99.90th=[ 978], 99.95th=[ 978], | 99.99th=[ 978] bw ( KiB/s): min=16351, max=16351, per=100.00%, avg=16351.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 250=33.33%, 1000=33.33% cpu : usr=0.08%, sys=1.61%, ctx=7, majf=0, minf=28 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=9017KiB/s (9233kB/s), 2254KiB/s-10.1MiB/s (2308kB/s-10.6MB/s), io=16.0MiB (16.8MB), run=1187-1817msec WRITE: bw=15.8MiB/s (16.6MB/s), 4509KiB/s-9894KiB/s (4617kB/s-10.1MB/s), io=32.0MiB (33.6MB), run=1242-2025msec