== sanity test 398b: DIO and buffer IO race ============== 02:32:04 (1739518324) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 2.06602 s, 24.4 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=17149: Fri Feb 14 02:32:51 2025 read: IOPS=35, BW=141KiB/s (145kB/s)(5992KiB/42372msec) clat (usec): min=60, max=103772, avg=15746.41, stdev=10420.71 lat (usec): min=60, max=103773, avg=15747.50, stdev=10420.72 clat percentiles (usec): | 1.00th=[ 89], 5.00th=[ 126], 10.00th=[ 6128], 20.00th=[ 8586], | 30.00th=[ 10290], 40.00th=[ 11994], 50.00th=[ 13960], 60.00th=[ 16057], | 70.00th=[ 18482], 80.00th=[ 22152], 90.00th=[ 27919], 95.00th=[ 34341], | 99.00th=[ 50594], 99.50th=[ 58983], 99.90th=[ 93848], 99.95th=[103285], | 99.99th=[103285] bw ( KiB/s): min= 24, max= 231, per=25.07%, avg=140.40, stdev=34.73, samples=84 iops : min= 6, max= 57, avg=34.64, stdev= 8.69, samples=84 write: IOPS=37, BW=149KiB/s (152kB/s)(6296KiB/42372msec) clat (usec): min=518, max=366933, avg=11838.35, stdev=15547.68 lat (usec): min=519, max=366934, avg=11840.62, stdev=15547.11 clat percentiles (usec): | 1.00th=[ 668], 5.00th=[ 734], 10.00th=[ 766], 20.00th=[ 865], | 30.00th=[ 1729], 40.00th=[ 3884], 50.00th=[ 8225], 60.00th=[ 11338], | 70.00th=[ 15139], 80.00th=[ 20841], 90.00th=[ 28443], 95.00th=[ 36963], | 99.00th=[ 54789], 99.50th=[ 68682], 99.90th=[113771], 99.95th=[367002], | 99.99th=[367002] bw ( KiB/s): min= 8, max= 272, per=25.12%, avg=147.19, stdev=45.11, samples=84 iops : min= 2, max= 68, avg=36.33, stdev=11.28, samples=84 lat (usec) : 100=1.11%, 250=2.02%, 500=0.07%, 750=3.48%, 1000=8.66% lat (msec) : 2=4.56%, 4=4.30%, 10=18.29%, 20=34.18%, 50=22.01% lat (msec) : 100=1.24%, 250=0.07%, 500=0.03% cpu : usr=0.12%, sys=10.61%, ctx=5020, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17151: Fri Feb 14 02:32:51 2025 read: IOPS=34, BW=139KiB/s (142kB/s)(5940KiB/42860msec) clat (usec): min=78, max=387646, avg=16022.96, stdev=14068.57 lat (usec): min=78, max=387649, avg=16024.08, stdev=14068.59 clat percentiles (usec): | 1.00th=[ 94], 5.00th=[ 129], 10.00th=[ 5866], 20.00th=[ 8291], | 30.00th=[ 10028], 40.00th=[ 11863], 50.00th=[ 13960], 60.00th=[ 16319], | 70.00th=[ 18744], 80.00th=[ 22414], 90.00th=[ 28705], 95.00th=[ 33424], | 99.00th=[ 49021], 99.50th=[ 60031], 99.90th=[ 87557], 99.95th=[387974], | 99.99th=[387974] bw ( KiB/s): min= 8, max= 263, per=24.39%, avg=136.58, stdev=38.57, samples=85 iops : min= 2, max= 65, avg=33.85, stdev= 9.65, samples=85 write: IOPS=37, BW=148KiB/s (152kB/s)(6348KiB/42860msec) clat (usec): min=599, max=100147, avg=11904.57, stdev=12899.91 lat (usec): min=600, max=100148, avg=11905.99, stdev=12900.18 clat percentiles (usec): | 1.00th=[ 676], 5.00th=[ 742], 10.00th=[ 783], 20.00th=[ 914], | 30.00th=[ 1844], 40.00th=[ 3720], 50.00th=[ 8291], 60.00th=[ 11469], | 70.00th=[ 15139], 80.00th=[ 21103], 90.00th=[ 30540], 95.00th=[ 37487], | 99.00th=[ 52691], 99.50th=[ 58459], 99.90th=[ 85459], 99.95th=[100140], | 99.99th=[100140] bw ( KiB/s): min= 63, max= 288, per=24.99%, avg=146.45, stdev=41.34, samples=84 iops : min= 15, max= 72, avg=36.32, stdev=10.32, samples=84 lat (usec) : 100=0.88%, 250=1.79%, 500=0.16%, 750=2.96%, 1000=8.56% lat (msec) : 2=4.95%, 4=5.14%, 10=18.00%, 20=33.72%, 50=22.62% lat (msec) : 100=1.14%, 250=0.03%, 500=0.03% cpu : usr=0.16%, sys=10.38%, ctx=4897, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17154: Fri Feb 14 02:32:51 2025 read: IOPS=35, BW=141KiB/s (145kB/s)(6028KiB/42680msec) clat (usec): min=69, max=101585, avg=16102.17, stdev=10437.28 lat (usec): min=70, max=101586, avg=16103.22, stdev=10437.30 clat percentiles (usec): | 1.00th=[ 92], 5.00th=[ 130], 10.00th=[ 6128], 20.00th=[ 8356], | 30.00th=[ 10421], 40.00th=[ 12387], 50.00th=[ 14615], 60.00th=[ 16909], | 70.00th=[ 19530], 80.00th=[ 22938], 90.00th=[ 27657], 95.00th=[ 32375], | 99.00th=[ 53740], 99.50th=[ 68682], 99.90th=[ 83362], 99.95th=[101188], | 99.99th=[101188] bw ( KiB/s): min= 56, max= 248, per=25.36%, avg=142.01, stdev=34.50, samples=84 iops : min= 14, max= 62, avg=35.21, stdev= 8.61, samples=84 write: IOPS=36, BW=147KiB/s (150kB/s)(6260KiB/42680msec) clat (usec): min=486, max=380899, avg=11639.51, stdev=15534.01 lat (usec): min=487, max=380901, avg=11643.56, stdev=15532.66 clat percentiles (usec): | 1.00th=[ 660], 5.00th=[ 734], 10.00th=[ 775], 20.00th=[ 881], | 30.00th=[ 1516], 40.00th=[ 3359], 50.00th=[ 8029], 60.00th=[ 11076], | 70.00th=[ 14746], 80.00th=[ 20841], 90.00th=[ 28705], 95.00th=[ 36963], | 99.00th=[ 55313], 99.50th=[ 59507], 99.90th=[ 92799], 99.95th=[379585], | 99.99th=[379585] bw ( KiB/s): min= 56, max= 271, per=25.14%, avg=147.32, stdev=44.86, samples=84 iops : min= 14, max= 67, avg=36.55, stdev=11.24, samples=84 lat (usec) : 100=0.91%, 250=1.82%, 500=0.20%, 750=3.81%, 1000=8.50% lat (msec) : 2=4.82%, 4=4.39%, 10=17.90%, 20=32.71%, 50=23.73% lat (msec) : 100=1.14%, 250=0.03%, 500=0.03% cpu : usr=0.17%, sys=10.42%, ctx=4821, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17156: Fri Feb 14 02:32:51 2025 read: IOPS=35, BW=141KiB/s (145kB/s)(6052KiB/42832msec) clat (usec): min=80, max=401699, avg=16392.11, stdev=14088.85 lat (usec): min=81, max=401700, avg=16393.63, stdev=14088.74 clat percentiles (usec): | 1.00th=[ 95], 5.00th=[ 799], 10.00th=[ 6259], 20.00th=[ 8225], | 30.00th=[ 10159], 40.00th=[ 12125], 50.00th=[ 14091], 60.00th=[ 16909], | 70.00th=[ 19792], 80.00th=[ 23200], 90.00th=[ 28705], 95.00th=[ 34341], | 99.00th=[ 50070], 99.50th=[ 54789], 99.90th=[ 83362], 99.95th=[400557], | 99.99th=[400557] bw ( KiB/s): min= 16, max= 208, per=24.98%, avg=139.89, stdev=31.60, samples=85 iops : min= 4, max= 52, avg=34.64, stdev= 7.93, samples=85 write: IOPS=36, BW=146KiB/s (149kB/s)(6236KiB/42832msec) clat (usec): min=506, max=98504, avg=11454.57, stdev=12552.68 lat (usec): min=507, max=98504, avg=11459.68, stdev=12556.28 clat percentiles (usec): | 1.00th=[ 652], 5.00th=[ 742], 10.00th=[ 775], 20.00th=[ 889], | 30.00th=[ 1745], 40.00th=[ 3884], 50.00th=[ 7963], 60.00th=[11207], | 70.00th=[14746], 80.00th=[19792], 90.00th=[28443], 95.00th=[36439], | 99.00th=[53740], 99.50th=[60031], 99.90th=[94897], 99.95th=[98042], | 99.99th=[98042] bw ( KiB/s): min= 32, max= 232, per=24.63%, avg=144.31, stdev=40.20, samples=85 iops : min= 8, max= 58, avg=35.74, stdev=10.07, samples=85 lat (usec) : 100=0.65%, 250=1.73%, 750=3.58%, 1000=7.91% lat (msec) : 2=4.43%, 4=4.92%, 10=19.40%, 20=33.17%, 50=23.01% lat (msec) : 100=1.17%, 500=0.03% cpu : usr=0.15%, sys=10.52%, ctx=5045, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=560KiB/s (574kB/s), 139KiB/s-141KiB/s (142kB/s-145kB/s), io=23.4MiB (24.6MB), run=42372-42860msec WRITE: bw=587KiB/s (601kB/s), 146KiB/s-149KiB/s (149kB/s-152kB/s), io=24.6MiB (25.7MB), run=42372-42860msec rand-rw: (groupid=0, jobs=1): err= 0: pid=17150: Fri Feb 14 02:33:09 2025 read: IOPS=24, BW=99.6KiB/s (102kB/s)(5992KiB/60180msec) clat (msec): min=2, max=412, avg=15.33, stdev=17.12 lat (msec): min=2, max=412, avg=15.33, stdev=17.12 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 5], 10.00th=[ 6], 20.00th=[ 7], | 30.00th=[ 8], 40.00th=[ 9], 50.00th=[ 10], 60.00th=[ 12], | 70.00th=[ 16], 80.00th=[ 22], 90.00th=[ 34], 95.00th=[ 46], | 99.00th=[ 69], 99.50th=[ 78], 99.90th=[ 118], 99.95th=[ 414], | 99.99th=[ 414] bw ( KiB/s): min= 24, max= 264, per=25.03%, avg=99.61, stdev=57.15, samples=119 iops : min= 6, max= 66, avg=24.59, stdev=14.29, samples=119 write: IOPS=26, BW=105KiB/s (107kB/s)(6296KiB/60180msec) clat (msec): min=4, max=143, avg=23.51, stdev=15.79 lat (msec): min=4, max=143, avg=23.52, stdev=15.79 clat percentiles (msec): | 1.00th=[ 8], 5.00th=[ 10], 10.00th=[ 11], 20.00th=[ 12], | 30.00th=[ 14], 40.00th=[ 16], 50.00th=[ 18], 60.00th=[ 21], | 70.00th=[ 27], 80.00th=[ 34], 90.00th=[ 46], 95.00th=[ 56], | 99.00th=[ 78], 99.50th=[ 92], 99.90th=[ 110], 99.95th=[ 144], | 99.99th=[ 144] bw ( KiB/s): min= 31, max= 256, per=25.20%, avg=104.85, stdev=58.18, samples=119 iops : min= 7, max= 64, avg=25.90, stdev=14.57, samples=119 lat (msec) : 4=0.91%, 10=28.71%, 20=38.05%, 50=26.79%, 100=5.34% lat (msec) : 250=0.16%, 500=0.03% cpu : usr=0.10%, sys=5.16%, ctx=3461, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17152: Fri Feb 14 02:33:09 2025 read: IOPS=24, BW=98.5KiB/s (101kB/s)(5940KiB/60315msec) clat (msec): min=2, max=394, avg=15.38, stdev=17.10 lat (msec): min=2, max=394, avg=15.38, stdev=17.10 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 5], 10.00th=[ 6], 20.00th=[ 7], | 30.00th=[ 8], 40.00th=[ 9], 50.00th=[ 10], 60.00th=[ 12], | 70.00th=[ 15], 80.00th=[ 22], 90.00th=[ 35], 95.00th=[ 46], | 99.00th=[ 70], 99.50th=[ 83], 99.90th=[ 122], 99.95th=[ 397], | 99.99th=[ 397] bw ( KiB/s): min= 8, max= 288, per=24.55%, avg=97.73, stdev=60.59, samples=120 iops : min= 2, max= 72, avg=24.17, stdev=15.20, samples=120 write: IOPS=26, BW=105KiB/s (108kB/s)(6348KiB/60315msec) clat (msec): min=5, max=122, avg=23.53, stdev=15.62 lat (msec): min=5, max=122, avg=23.53, stdev=15.62 clat percentiles (msec): | 1.00th=[ 8], 5.00th=[ 10], 10.00th=[ 11], 20.00th=[ 12], | 30.00th=[ 14], 40.00th=[ 16], 50.00th=[ 19], 60.00th=[ 22], | 70.00th=[ 27], 80.00th=[ 34], 90.00th=[ 44], 95.00th=[ 54], | 99.00th=[ 83], 99.50th=[ 100], 99.90th=[ 114], 99.95th=[ 123], | 99.99th=[ 123] bw ( KiB/s): min= 31, max= 240, per=25.21%, avg=104.86, stdev=57.05, samples=119 iops : min= 7, max= 60, avg=25.96, stdev=14.31, samples=119 lat (msec) : 4=0.85%, 10=28.97%, 20=37.50%, 50=27.38%, 100=5.01% lat (msec) : 250=0.26%, 500=0.03% cpu : usr=0.09%, sys=5.35%, ctx=3479, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17153: Fri Feb 14 02:33:09 2025 read: IOPS=24, BW=99.0KiB/s (102kB/s)(6028KiB/60310msec) clat (usec): min=2476, max=93054, avg=14265.50, stdev=12870.66 lat (usec): min=2478, max=93055, avg=14267.82, stdev=12870.01 clat percentiles (usec): | 1.00th=[ 3556], 5.00th=[ 4490], 10.00th=[ 5014], 20.00th=[ 6063], | 30.00th=[ 7046], 40.00th=[ 8029], 50.00th=[ 9372], 60.00th=[11469], | 70.00th=[13960], 80.00th=[19268], 90.00th=[31065], 95.00th=[41681], | 99.00th=[65799], 99.50th=[73925], 99.90th=[89654], 99.95th=[92799], | 99.99th=[92799] bw ( KiB/s): min= 15, max= 295, per=25.10%, avg=99.92, stdev=58.65, samples=119 iops : min= 3, max= 73, avg=24.66, stdev=14.74, samples=119 write: IOPS=25, BW=104KiB/s (106kB/s)(6260KiB/60310msec) clat (msec): min=5, max=419, avg=24.71, stdev=19.87 lat (msec): min=5, max=419, avg=24.71, stdev=19.87 clat percentiles (msec): | 1.00th=[ 8], 5.00th=[ 10], 10.00th=[ 11], 20.00th=[ 12], | 30.00th=[ 14], 40.00th=[ 16], 50.00th=[ 19], 60.00th=[ 22], | 70.00th=[ 28], 80.00th=[ 35], 90.00th=[ 48], 95.00th=[ 60], | 99.00th=[ 91], 99.50th=[ 104], 99.90th=[ 144], 99.95th=[ 422], | 99.99th=[ 422] bw ( KiB/s): min= 8, max= 247, per=24.86%, avg=103.43, stdev=58.28, samples=119 iops : min= 2, max= 61, avg=25.53, stdev=14.65, samples=119 lat (msec) : 4=0.94%, 10=29.33%, 20=37.60%, 50=26.43%, 100=5.34% lat (msec) : 250=0.33%, 500=0.03% cpu : usr=0.03%, sys=5.27%, ctx=3458, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17155: Fri Feb 14 02:33:09 2025 read: IOPS=25, BW=101KiB/s (103kB/s)(6052KiB/60151msec) clat (msec): min=3, max=122, avg=14.87, stdev=14.26 lat (msec): min=3, max=122, avg=14.87, stdev=14.26 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 5], 10.00th=[ 6], 20.00th=[ 7], | 30.00th=[ 7], 40.00th=[ 9], 50.00th=[ 10], 60.00th=[ 12], | 70.00th=[ 15], 80.00th=[ 20], 90.00th=[ 34], 95.00th=[ 46], | 99.00th=[ 70], 99.50th=[ 87], 99.90th=[ 105], 99.95th=[ 124], | 99.99th=[ 124] bw ( KiB/s): min= 8, max= 264, per=25.37%, avg=100.98, stdev=60.31, samples=119 iops : min= 2, max= 66, avg=24.96, stdev=15.12, samples=119 write: IOPS=25, BW=104KiB/s (106kB/s)(6236KiB/60151msec) clat (msec): min=5, max=375, avg=24.03, stdev=18.25 lat (msec): min=5, max=375, avg=24.03, stdev=18.25 clat percentiles (msec): | 1.00th=[ 7], 5.00th=[ 9], 10.00th=[ 11], 20.00th=[ 13], | 30.00th=[ 14], 40.00th=[ 16], 50.00th=[ 18], 60.00th=[ 22], | 70.00th=[ 27], 80.00th=[ 35], 90.00th=[ 45], 95.00th=[ 59], | 99.00th=[ 81], 99.50th=[ 93], 99.90th=[ 113], 99.95th=[ 376], | 99.99th=[ 376] bw ( KiB/s): min= 23, max= 255, per=24.97%, avg=103.87, stdev=57.41, samples=119 iops : min= 5, max= 63, avg=25.68, stdev=14.40, samples=119 lat (msec) : 4=1.24%, 10=29.59%, 20=37.73%, 50=25.75%, 100=5.40% lat (msec) : 250=0.26%, 500=0.03% cpu : usr=0.09%, sys=5.24%, ctx=3500, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=398KiB/s (408kB/s), 98.5KiB/s-101KiB/s (101kB/s-103kB/s), io=23.4MiB (24.6MB), run=60151-60315msec WRITE: bw=417KiB/s (427kB/s), 104KiB/s-105KiB/s (106kB/s-108kB/s), io=24.6MiB (25.7MB), run=60151-60315msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=17219: Fri Feb 14 02:33:19 2025 read: IOPS=43, BW=693KiB/s (709kB/s)(5920KiB/8546msec) clat (usec): min=72, max=61217, avg=12192.46, stdev=9938.26 lat (usec): min=72, max=61218, avg=12193.57, stdev=9938.25 clat percentiles (usec): | 1.00th=[ 79], 5.00th=[ 96], 10.00th=[ 111], 20.00th=[ 1336], | 30.00th=[ 7439], 40.00th=[ 9110], 50.00th=[10945], 60.00th=[13042], | 70.00th=[14877], 80.00th=[19006], 90.00th=[23987], 95.00th=[28967], | 99.00th=[44303], 99.50th=[59507], 99.90th=[61080], 99.95th=[61080], | 99.99th=[61080] bw ( KiB/s): min= 480, max= 1152, per=25.83%, avg=684.47, stdev=184.38, samples=17 iops : min= 30, max= 72, avg=42.53, stdev=11.52, samples=17 write: IOPS=46, BW=745KiB/s (763kB/s)(6368KiB/8546msec) clat (usec): min=847, max=63388, avg=10036.84, stdev=11004.34 lat (usec): min=848, max=63389, avg=10038.48, stdev=11004.34 clat percentiles (usec): | 1.00th=[ 971], 5.00th=[ 1090], 10.00th=[ 1205], 20.00th=[ 1434], | 30.00th=[ 2278], 40.00th=[ 3720], 50.00th=[ 6521], 60.00th=[ 8848], | 70.00th=[12125], 80.00th=[17171], 90.00th=[23987], 95.00th=[32637], | 99.00th=[54264], 99.50th=[55837], 99.90th=[63177], 99.95th=[63177], | 99.99th=[63177] bw ( KiB/s): min= 541, max= 1024, per=26.51%, avg=739.12, stdev=145.19, samples=17 iops : min= 33, max= 64, avg=45.94, stdev= 9.20, samples=17 lat (usec) : 100=3.12%, 250=5.99%, 500=0.13%, 750=0.13%, 1000=1.30% lat (msec) : 2=13.41%, 4=7.16%, 10=23.44%, 20=29.17%, 50=15.10% lat (msec) : 100=1.04% cpu : usr=0.13%, sys=13.61%, ctx=1246, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17221: Fri Feb 14 02:33:19 2025 read: IOPS=41, BW=667KiB/s (683kB/s)(5744KiB/8614msec) clat (usec): min=69, max=57641, avg=12325.62, stdev=9360.98 lat (usec): min=69, max=57641, avg=12326.70, stdev=9361.01 clat percentiles (usec): | 1.00th=[ 84], 5.00th=[ 99], 10.00th=[ 115], 20.00th=[ 4948], | 30.00th=[ 7373], 40.00th=[ 9110], 50.00th=[11207], 60.00th=[13698], | 70.00th=[16057], 80.00th=[19268], 90.00th=[23987], 95.00th=[26608], | 99.00th=[46924], 99.50th=[52167], 99.90th=[57410], 99.95th=[57410], | 99.99th=[57410] bw ( KiB/s): min= 384, max= 832, per=24.84%, avg=658.29, stdev=117.75, samples=17 iops : min= 24, max= 52, avg=40.94, stdev= 7.40, samples=17 write: IOPS=47, BW=760KiB/s (778kB/s)(6544KiB/8614msec) clat (usec): min=598, max=70141, avg=9983.19, stdev=10570.56 lat (usec): min=599, max=70143, avg=9984.93, stdev=10570.76 clat percentiles (usec): | 1.00th=[ 881], 5.00th=[ 1057], 10.00th=[ 1139], 20.00th=[ 1336], | 30.00th=[ 1909], 40.00th=[ 3326], 50.00th=[ 6718], 60.00th=[ 9765], | 70.00th=[13173], 80.00th=[17957], 90.00th=[24511], 95.00th=[29230], | 99.00th=[44827], 99.50th=[61080], 99.90th=[69731], 99.95th=[69731], | 99.99th=[69731] bw ( KiB/s): min= 542, max= 1149, per=26.45%, avg=737.29, stdev=161.16, samples=17 iops : min= 33, max= 71, avg=45.88, stdev=10.04, samples=17 lat (usec) : 100=2.47%, 250=4.82%, 500=0.39%, 750=0.65%, 1000=1.30% lat (msec) : 2=15.62%, 4=6.38%, 10=21.35%, 20=30.08%, 50=16.28% lat (msec) : 100=0.65% cpu : usr=0.16%, sys=13.65%, ctx=1258, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17223: Fri Feb 14 02:33:19 2025 read: IOPS=43, BW=692KiB/s (709kB/s)(6256KiB/9038msec) clat (usec): min=72, max=94357, avg=13062.92, stdev=10550.68 lat (usec): min=72, max=94358, avg=13069.89, stdev=10544.27 clat percentiles (usec): | 1.00th=[ 83], 5.00th=[ 105], 10.00th=[ 125], 20.00th=[ 5473], | 30.00th=[ 7898], 40.00th=[ 9896], 50.00th=[11731], 60.00th=[13304], | 70.00th=[16450], 80.00th=[20841], 90.00th=[25560], 95.00th=[29754], | 99.00th=[47973], 99.50th=[64226], 99.90th=[93848], 99.95th=[93848], | 99.99th=[93848] bw ( KiB/s): min= 448, max= 992, per=25.05%, avg=663.94, stdev=166.69, samples=17 iops : min= 28, max= 62, avg=41.29, stdev=10.34, samples=17 write: IOPS=41, BW=667KiB/s (683kB/s)(6032KiB/9038msec) clat (usec): min=902, max=70088, avg=10075.08, stdev=11213.51 lat (usec): min=904, max=70090, avg=10078.16, stdev=11215.24 clat percentiles (usec): | 1.00th=[ 930], 5.00th=[ 1074], 10.00th=[ 1188], 20.00th=[ 1369], | 30.00th=[ 1975], 40.00th=[ 2999], 50.00th=[ 6063], 60.00th=[ 8717], | 70.00th=[12911], 80.00th=[17171], 90.00th=[25035], 95.00th=[33162], | 99.00th=[49546], 99.50th=[67634], 99.90th=[69731], 99.95th=[69731], | 99.99th=[69731] bw ( KiB/s): min= 288, max= 958, per=22.73%, avg=633.82, stdev=168.57, samples=17 iops : min= 18, max= 59, avg=39.41, stdev=10.35, samples=17 lat (usec) : 100=1.82%, 250=6.25%, 500=0.52%, 750=0.13%, 1000=0.91% lat (msec) : 2=14.06%, 4=6.25%, 10=21.74%, 20=29.69%, 50=17.84% lat (msec) : 100=0.78% cpu : usr=0.20%, sys=13.22%, ctx=1276, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17224: Fri Feb 14 02:33:19 2025 read: IOPS=43, BW=691KiB/s (708kB/s)(6032KiB/8726msec) clat (usec): min=62, max=80060, avg=12948.64, stdev=10347.16 lat (usec): min=62, max=80061, avg=12949.72, stdev=10347.15 clat percentiles (usec): | 1.00th=[ 85], 5.00th=[ 102], 10.00th=[ 116], 20.00th=[ 6194], | 30.00th=[ 8225], 40.00th=[10028], 50.00th=[11469], 60.00th=[13304], | 70.00th=[15533], 80.00th=[18744], 90.00th=[25560], 95.00th=[30802], | 99.00th=[56361], 99.50th=[63701], 99.90th=[80217], 99.95th=[80217], | 99.99th=[80217] bw ( KiB/s): min= 384, max= 896, per=25.40%, avg=673.12, stdev=160.19, samples=17 iops : min= 24, max= 56, avg=41.76, stdev=10.00, samples=17 write: IOPS=44, BW=717KiB/s (734kB/s)(6256KiB/8726msec) clat (usec): min=737, max=60733, avg=9528.94, stdev=10600.63 lat (usec): min=738, max=60736, avg=9530.57, stdev=10600.71 clat percentiles (usec): | 1.00th=[ 889], 5.00th=[ 1074], 10.00th=[ 1172], 20.00th=[ 1352], | 30.00th=[ 1942], 40.00th=[ 2933], 50.00th=[ 5800], 60.00th=[ 8848], | 70.00th=[11731], 80.00th=[16188], 90.00th=[23725], 95.00th=[28967], | 99.00th=[54264], 99.50th=[60556], 99.90th=[60556], 99.95th=[60556], | 99.99th=[60556] bw ( KiB/s): min= 416, max= 1056, per=25.29%, avg=705.12, stdev=158.41, samples=17 iops : min= 26, max= 66, avg=43.76, stdev= 9.99, samples=17 lat (usec) : 100=2.21%, 250=5.47%, 500=0.13%, 750=0.52%, 1000=1.56% lat (msec) : 2=14.06%, 4=7.55%, 10=20.83%, 20=32.16%, 50=14.32% lat (msec) : 100=1.17% cpu : usr=0.26%, sys=13.32%, ctx=1213, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2650KiB/s (2714kB/s), 667KiB/s-693KiB/s (683kB/s-709kB/s), io=23.4MiB (24.5MB), run=8546-9038msec WRITE: bw=2788KiB/s (2855kB/s), 667KiB/s-760KiB/s (683kB/s-778kB/s), io=24.6MiB (25.8MB), run=8546-9038msec rand-rw: (groupid=0, jobs=1): err= 0: pid=17217: Fri Feb 14 02:33:24 2025 read: IOPS=26, BW=427KiB/s (437kB/s)(5920KiB/13866msec) clat (usec): min=3162, max=79318, avg=13766.45, stdev=12365.71 lat (usec): min=3163, max=79320, avg=13767.82, stdev=12366.04 clat percentiles (usec): | 1.00th=[ 3785], 5.00th=[ 4686], 10.00th=[ 5276], 20.00th=[ 6587], | 30.00th=[ 7439], 40.00th=[ 8356], 50.00th=[ 9372], 60.00th=[11076], | 70.00th=[12780], 80.00th=[16188], 90.00th=[29230], 95.00th=[40633], | 99.00th=[65799], 99.50th=[74974], 99.90th=[79168], 99.95th=[79168], | 99.99th=[79168] bw ( KiB/s): min= 95, max= 896, per=24.45%, avg=421.22, stdev=201.02, samples=27 iops : min= 5, max= 56, avg=26.07, stdev=12.67, samples=27 write: IOPS=28, BW=459KiB/s (470kB/s)(6368KiB/13866msec) clat (msec): min=5, max=107, avg=21.81, stdev=13.77 lat (msec): min=5, max=107, avg=21.81, stdev=13.77 clat percentiles (msec): | 1.00th=[ 7], 5.00th=[ 10], 10.00th=[ 11], 20.00th=[ 13], | 30.00th=[ 14], 40.00th=[ 16], 50.00th=[ 17], 60.00th=[ 20], | 70.00th=[ 24], 80.00th=[ 31], 90.00th=[ 40], 95.00th=[ 51], | 99.00th=[ 73], 99.50th=[ 96], 99.90th=[ 108], 99.95th=[ 108], | 99.99th=[ 108] bw ( KiB/s): min= 160, max= 864, per=25.25%, avg=457.78, stdev=213.67, samples=27 iops : min= 10, max= 54, avg=28.37, stdev=13.39, samples=27 lat (msec) : 4=0.78%, 10=29.04%, 20=42.71%, 50=23.31%, 100=4.04% lat (msec) : 250=0.13% cpu : usr=0.06%, sys=6.25%, ctx=919, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17218: Fri Feb 14 02:33:24 2025 read: IOPS=25, BW=413KiB/s (423kB/s)(5744KiB/13899msec) clat (usec): min=2786, max=66679, avg=13177.37, stdev=10948.94 lat (usec): min=2787, max=66681, avg=13178.48, stdev=10948.96 clat percentiles (usec): | 1.00th=[ 3425], 5.00th=[ 4686], 10.00th=[ 5211], 20.00th=[ 6325], | 30.00th=[ 7177], 40.00th=[ 8094], 50.00th=[ 9241], 60.00th=[10552], | 70.00th=[12518], 80.00th=[16581], 90.00th=[27657], 95.00th=[40633], | 99.00th=[52691], 99.50th=[63701], 99.90th=[66847], 99.95th=[66847], | 99.99th=[66847] bw ( KiB/s): min= 96, max= 928, per=23.89%, avg=411.70, stdev=229.50, samples=27 iops : min= 6, max= 58, avg=25.33, stdev=14.36, samples=27 write: IOPS=29, BW=471KiB/s (482kB/s)(6544KiB/13899msec) clat (msec): min=5, max=123, avg=22.26, stdev=14.67 lat (msec): min=5, max=123, avg=22.26, stdev=14.67 clat percentiles (msec): | 1.00th=[ 8], 5.00th=[ 10], 10.00th=[ 11], 20.00th=[ 13], | 30.00th=[ 14], 40.00th=[ 16], 50.00th=[ 17], 60.00th=[ 20], | 70.00th=[ 24], 80.00th=[ 30], 90.00th=[ 43], 95.00th=[ 52], | 99.00th=[ 78], 99.50th=[ 79], 99.90th=[ 124], 99.95th=[ 124], | 99.99th=[ 124] bw ( KiB/s): min= 224, max= 928, per=25.52%, avg=462.70, stdev=197.99, samples=27 iops : min= 14, max= 58, avg=28.52, stdev=12.45, samples=27 lat (msec) : 4=0.65%, 10=29.04%, 20=41.80%, 50=24.87%, 100=3.39% lat (msec) : 250=0.26% cpu : usr=0.11%, sys=6.02%, ctx=887, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17220: Fri Feb 14 02:33:24 2025 read: IOPS=28, BW=458KiB/s (469kB/s)(6256KiB/13663msec) clat (usec): min=3270, max=70329, avg=13470.05, stdev=11369.18 lat (usec): min=3271, max=70330, avg=13471.23, stdev=11369.21 clat percentiles (usec): | 1.00th=[ 3916], 5.00th=[ 4752], 10.00th=[ 5276], 20.00th=[ 6194], | 30.00th=[ 7439], 40.00th=[ 8225], 50.00th=[ 9241], 60.00th=[10683], | 70.00th=[12780], 80.00th=[16712], 90.00th=[30802], 95.00th=[39584], | 99.00th=[59507], 99.50th=[63177], 99.90th=[70779], 99.95th=[70779], | 99.99th=[70779] bw ( KiB/s): min= 192, max= 1152, per=26.73%, avg=460.56, stdev=241.09, samples=27 iops : min= 12, max= 72, avg=28.56, stdev=15.13, samples=27 write: IOPS=27, BW=441KiB/s (452kB/s)(6032KiB/13663msec) clat (msec): min=5, max=103, avg=22.07, stdev=14.15 lat (msec): min=5, max=103, avg=22.07, stdev=14.15 clat percentiles (msec): | 1.00th=[ 7], 5.00th=[ 10], 10.00th=[ 11], 20.00th=[ 13], | 30.00th=[ 14], 40.00th=[ 16], 50.00th=[ 18], 60.00th=[ 21], | 70.00th=[ 25], 80.00th=[ 30], 90.00th=[ 40], 95.00th=[ 50], | 99.00th=[ 82], 99.50th=[ 94], 99.90th=[ 104], 99.95th=[ 104], | 99.99th=[ 104] bw ( KiB/s): min= 96, max= 832, per=24.16%, avg=438.04, stdev=199.15, samples=27 iops : min= 6, max= 52, avg=27.15, stdev=12.45, samples=27 lat (msec) : 4=0.91%, 10=31.25%, 20=39.58%, 50=25.13%, 100=2.99% lat (msec) : 250=0.13% cpu : usr=0.10%, sys=6.24%, ctx=892, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17222: Fri Feb 14 02:33:24 2025 read: IOPS=28, BW=450KiB/s (461kB/s)(6032KiB/13403msec) clat (usec): min=3352, max=71086, avg=13192.36, stdev=10779.12 lat (usec): min=3353, max=71087, avg=13193.48, stdev=10779.14 clat percentiles (usec): | 1.00th=[ 3752], 5.00th=[ 4686], 10.00th=[ 5407], 20.00th=[ 6456], | 30.00th=[ 7046], 40.00th=[ 8029], 50.00th=[ 9503], 60.00th=[11076], | 70.00th=[13435], 80.00th=[17171], 90.00th=[26346], 95.00th=[37487], | 99.00th=[54264], 99.50th=[68682], 99.90th=[70779], 99.95th=[70779], | 99.99th=[70779] bw ( KiB/s): min= 223, max= 800, per=25.41%, avg=437.73, stdev=179.65, samples=26 iops : min= 13, max= 50, avg=27.15, stdev=11.31, samples=26 write: IOPS=29, BW=467KiB/s (478kB/s)(6256KiB/13403msec) clat (msec): min=6, max=116, avg=21.48, stdev=13.85 lat (msec): min=6, max=116, avg=21.48, stdev=13.85 clat percentiles (msec): | 1.00th=[ 7], 5.00th=[ 9], 10.00th=[ 11], 20.00th=[ 12], | 30.00th=[ 14], 40.00th=[ 16], 50.00th=[ 18], 60.00th=[ 20], | 70.00th=[ 23], 80.00th=[ 30], 90.00th=[ 40], 95.00th=[ 50], | 99.00th=[ 83], 99.50th=[ 90], 99.90th=[ 116], 99.95th=[ 116], | 99.99th=[ 116] bw ( KiB/s): min= 192, max= 960, per=25.16%, avg=456.19, stdev=207.36, samples=26 iops : min= 12, max= 60, avg=28.31, stdev=13.00, samples=26 lat (msec) : 4=0.78%, 10=29.56%, 20=42.84%, 50=23.70%, 100=2.99% lat (msec) : 250=0.13% cpu : usr=0.00%, sys=6.28%, ctx=889, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1723KiB/s (1765kB/s), 413KiB/s-458KiB/s (423kB/s-469kB/s), io=23.4MiB (24.5MB), run=13403-13899msec WRITE: bw=1813KiB/s (1857kB/s), 441KiB/s-471KiB/s (452kB/s-482kB/s), io=24.6MiB (25.8MB), run=13403-13899msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=17241: Fri Feb 14 02:33:28 2025 read: IOPS=4, BW=5006KiB/s (5126kB/s)(7168KiB/1432msec) clat (usec): min=490, max=777920, avg=165392.47, stdev=285331.95 lat (usec): min=491, max=777921, avg=165393.99, stdev=285331.62 clat percentiles (usec): | 1.00th=[ 490], 5.00th=[ 490], 10.00th=[ 490], 20.00th=[ 570], | 30.00th=[ 734], 40.00th=[ 734], 50.00th=[ 52691], 60.00th=[ 64750], | 70.00th=[ 64750], 80.00th=[261096], 90.00th=[775947], 95.00th=[775947], | 99.00th=[775947], 99.50th=[775947], 99.90th=[775947], 99.95th=[775947], | 99.99th=[775947] bw ( KiB/s): min= 4096, max= 4096, per=53.00%, avg=4096.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 write: IOPS=3, BW=3575KiB/s (3661kB/s)(5120KiB/1432msec) clat (usec): min=31055, max=76169, avg=51269.51, stdev=17976.62 lat (usec): min=31074, max=76251, avg=51333.33, stdev=17987.72 clat percentiles (usec): | 1.00th=[31065], 5.00th=[31065], 10.00th=[31065], 20.00th=[31065], | 30.00th=[42730], 40.00th=[42730], 50.00th=[43779], 60.00th=[43779], | 70.00th=[62653], 80.00th=[62653], 90.00th=[76022], 95.00th=[76022], | 99.00th=[76022], 99.50th=[76022], 99.90th=[76022], 99.95th=[76022], | 99.99th=[76022] lat (usec) : 500=8.33%, 750=16.67% lat (msec) : 50=25.00%, 100=33.33%, 500=8.33%, 1000=8.33% cpu : usr=0.00%, sys=17.47%, ctx=518, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17242: Fri Feb 14 02:33:28 2025 read: IOPS=0, BW=773KiB/s (791kB/s)(2048KiB/2650msec) clat (msec): min=65, max=1393, avg=729.73, stdev=939.28 lat (msec): min=65, max=1393, avg=729.73, stdev=939.28 clat percentiles (msec): | 1.00th=[ 66], 5.00th=[ 66], 10.00th=[ 66], 20.00th=[ 66], | 30.00th=[ 66], 40.00th=[ 66], 50.00th=[ 66], 60.00th=[ 1401], | 70.00th=[ 1401], 80.00th=[ 1401], 90.00th=[ 1401], 95.00th=[ 1401], | 99.00th=[ 1401], 99.50th=[ 1401], 99.90th=[ 1401], 99.95th=[ 1401], | 99.99th=[ 1401] bw ( KiB/s): min= 1738, max= 2039, per=24.44%, avg=1888.50, stdev=212.84, samples=2 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=2 write: IOPS=3, BW=3864KiB/s (3957kB/s)(10.0MiB/2650msec) clat (msec): min=28, max=314, avg=112.94, stdev=98.82 lat (msec): min=28, max=314, avg=113.14, stdev=99.04 clat percentiles (msec): | 1.00th=[ 29], 5.00th=[ 29], 10.00th=[ 29], 20.00th=[ 49], | 30.00th=[ 51], 40.00th=[ 52], 50.00th=[ 58], 60.00th=[ 70], | 70.00th=[ 146], 80.00th=[ 146], 90.00th=[ 262], 95.00th=[ 317], | 99.00th=[ 317], 99.50th=[ 317], 99.90th=[ 317], 99.95th=[ 317], | 99.99th=[ 317] bw ( KiB/s): min= 4079, max= 6131, per=47.61%, avg=5150.67, stdev=1029.04, samples=3 iops : min= 3, max= 5, avg= 4.33, stdev= 1.15, samples=3 lat (msec) : 50=25.00%, 100=33.33%, 250=16.67%, 500=16.67% cpu : usr=0.00%, sys=15.10%, ctx=499, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17243: Fri Feb 14 02:33:28 2025 read: IOPS=5, BW=5535KiB/s (5668kB/s)(6144KiB/1110msec) clat (usec): min=468, max=248325, avg=71814.99, stdev=91526.41 lat (usec): min=470, max=248326, avg=71816.38, stdev=91526.46 clat percentiles (usec): | 1.00th=[ 469], 5.00th=[ 469], 10.00th=[ 469], 20.00th=[ 1029], | 30.00th=[ 1029], 40.00th=[ 48497], 50.00th=[ 48497], 60.00th=[ 63177], | 70.00th=[ 69731], 80.00th=[ 69731], 90.00th=[248513], 95.00th=[248513], | 99.00th=[248513], 99.50th=[248513], 99.90th=[248513], 99.95th=[248513], | 99.99th=[248513] bw ( KiB/s): min=10240, max=10240, per=100.00%, avg=10240.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 write: IOPS=5, BW=5535KiB/s (5668kB/s)(6144KiB/1110msec) clat (msec): min=46, max=297, avg=96.41, stdev=98.68 lat (msec): min=46, max=297, avg=96.49, stdev=98.70 clat percentiles (msec): | 1.00th=[ 47], 5.00th=[ 47], 10.00th=[ 47], 20.00th=[ 53], | 30.00th=[ 53], 40.00th=[ 54], 50.00th=[ 54], 60.00th=[ 61], | 70.00th=[ 69], 80.00th=[ 69], 90.00th=[ 296], 95.00th=[ 296], | 99.00th=[ 296], 99.50th=[ 296], 99.90th=[ 296], 99.95th=[ 296], | 99.99th=[ 296] bw ( KiB/s): min= 6144, max= 6144, per=56.79%, avg=6144.00, stdev= 0.00, samples=2 iops : min= 6, max= 6, avg= 6.00, stdev= 0.00, samples=2 lat (usec) : 500=8.33% lat (msec) : 2=8.33%, 50=16.67%, 100=50.00%, 250=8.33%, 500=8.33% cpu : usr=0.00%, sys=17.67%, ctx=233, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17244: Fri Feb 14 02:33:28 2025 read: IOPS=4, BW=4580KiB/s (4690kB/s)(5120KiB/1118msec) clat (usec): min=469, max=251556, avg=71192.12, stdev=105209.33 lat (usec): min=470, max=251558, avg=71193.43, stdev=105209.58 clat percentiles (usec): | 1.00th=[ 469], 5.00th=[ 469], 10.00th=[ 469], 20.00th=[ 469], | 30.00th=[ 570], 40.00th=[ 570], 50.00th=[ 29492], 60.00th=[ 29492], | 70.00th=[ 73925], 80.00th=[ 73925], 90.00th=[250610], 95.00th=[250610], | 99.00th=[250610], 99.50th=[250610], 99.90th=[250610], 99.95th=[250610], | 99.99th=[250610] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=6, BW=6411KiB/s (6565kB/s)(7168KiB/1118msec) clat (msec): min=34, max=325, avg=103.86, stdev=100.16 lat (msec): min=34, max=325, avg=103.92, stdev=100.16 clat percentiles (msec): | 1.00th=[ 35], 5.00th=[ 35], 10.00th=[ 35], 20.00th=[ 50], | 30.00th=[ 57], 40.00th=[ 57], 50.00th=[ 72], 60.00th=[ 89], | 70.00th=[ 89], 80.00th=[ 102], 90.00th=[ 326], 95.00th=[ 326], | 99.00th=[ 326], 99.50th=[ 326], 99.90th=[ 326], 99.95th=[ 326], | 99.99th=[ 326] bw ( KiB/s): min= 6144, max= 8192, per=66.25%, avg=7168.00, stdev=1448.15, samples=2 iops : min= 6, max= 8, avg= 7.00, stdev= 1.41, samples=2 lat (usec) : 500=8.33%, 750=8.33% lat (msec) : 50=25.00%, 100=33.33%, 250=8.33%, 500=16.67% cpu : usr=0.09%, sys=18.53%, ctx=227, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=7728KiB/s (7914kB/s), 773KiB/s-5535KiB/s (791kB/s-5668kB/s), io=20.0MiB (20.0MB), run=1110-2650msec WRITE: bw=10.6MiB/s (11.1MB/s), 3575KiB/s-6411KiB/s (3661kB/s-6565kB/s), io=28.0MiB (29.4MB), run=1110-2650msec rand-rw: (groupid=0, jobs=1): err= 0: pid=17246: Fri Feb 14 02:33:28 2025 read: IOPS=2, BW=2470KiB/s (2529kB/s)(7168KiB/2902msec) clat (msec): min=57, max=917, avg=319.76, stdev=316.34 lat (msec): min=57, max=917, avg=319.76, stdev=316.34 clat percentiles (msec): | 1.00th=[ 57], 5.00th=[ 57], 10.00th=[ 57], 20.00th=[ 100], | 30.00th=[ 138], 40.00th=[ 138], 50.00th=[ 169], 60.00th=[ 275], | 70.00th=[ 275], 80.00th=[ 584], 90.00th=[ 919], 95.00th=[ 919], | 99.00th=[ 919], 99.50th=[ 919], 99.90th=[ 919], 99.95th=[ 919], | 99.99th=[ 919] bw ( KiB/s): min= 2048, max= 4087, per=39.08%, avg=2557.75, stdev=1019.50, samples=4 iops : min= 2, max= 3, avg= 2.25, stdev= 0.50, samples=4 write: IOPS=1, BW=1764KiB/s (1807kB/s)(5120KiB/2902msec) clat (msec): min=85, max=178, avg=112.01, stdev=37.39 lat (msec): min=85, max=178, avg=112.07, stdev=37.37 clat percentiles (msec): | 1.00th=[ 86], 5.00th=[ 86], 10.00th=[ 86], 20.00th=[ 86], | 30.00th=[ 99], 40.00th=[ 99], 50.00th=[ 99], 60.00th=[ 99], | 70.00th=[ 101], 80.00th=[ 101], 90.00th=[ 178], 95.00th=[ 178], | 99.00th=[ 178], 99.50th=[ 178], 99.90th=[ 178], 99.95th=[ 178], | 99.99th=[ 178] bw ( KiB/s): min= 2048, max= 6131, per=44.63%, avg=4089.50, stdev=2887.12, samples=2 iops : min= 2, max= 5, avg= 3.50, stdev= 2.12, samples=2 lat (msec) : 100=50.00%, 250=25.00%, 500=8.33%, 750=8.33%, 1000=8.33% cpu : usr=0.00%, sys=4.93%, ctx=63, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17247: Fri Feb 14 02:33:28 2025 read: IOPS=0, BW=691KiB/s (708kB/s)(2048KiB/2963msec) clat (msec): min=40, max=1286, avg=663.34, stdev=881.47 lat (msec): min=40, max=1286, avg=663.34, stdev=881.47 clat percentiles (msec): | 1.00th=[ 41], 5.00th=[ 41], 10.00th=[ 41], 20.00th=[ 41], | 30.00th=[ 41], 40.00th=[ 41], 50.00th=[ 41], 60.00th=[ 1284], | 70.00th=[ 1284], 80.00th=[ 1284], 90.00th=[ 1284], 95.00th=[ 1284], | 99.00th=[ 1284], 99.50th=[ 1284], 99.90th=[ 1284], 99.95th=[ 1284], | 99.99th=[ 1284] bw ( KiB/s): min= 2043, max= 2048, per=31.25%, avg=2045.50, stdev= 3.54, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 write: IOPS=3, BW=3456KiB/s (3539kB/s)(10.0MiB/2963msec) clat (msec): min=49, max=371, avg=152.79, stdev=86.30 lat (msec): min=49, max=371, avg=153.28, stdev=86.43 clat percentiles (msec): | 1.00th=[ 50], 5.00th=[ 50], 10.00th=[ 50], 20.00th=[ 111], | 30.00th=[ 113], 40.00th=[ 116], 50.00th=[ 126], 60.00th=[ 129], | 70.00th=[ 171], 80.00th=[ 171], 90.00th=[ 199], 95.00th=[ 372], | 99.00th=[ 372], 99.50th=[ 372], 99.90th=[ 372], 99.95th=[ 372], | 99.99th=[ 372] bw ( KiB/s): min= 2043, max= 8175, per=44.62%, avg=4088.67, stdev=3538.87, samples=3 iops : min= 1, max= 7, avg= 3.33, stdev= 3.21, samples=3 lat (msec) : 50=16.67%, 250=66.67%, 500=8.33% cpu : usr=0.00%, sys=7.22%, ctx=171, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17248: Fri Feb 14 02:33:28 2025 read: IOPS=2, BW=2084KiB/s (2134kB/s)(6144KiB/2948msec) clat (msec): min=52, max=225, avg=92.54, stdev=67.31 lat (msec): min=52, max=225, avg=92.54, stdev=67.31 clat percentiles (msec): | 1.00th=[ 53], 5.00th=[ 53], 10.00th=[ 53], 20.00th=[ 53], | 30.00th=[ 53], 40.00th=[ 54], 50.00th=[ 54], 60.00th=[ 79], | 70.00th=[ 94], 80.00th=[ 94], 90.00th=[ 226], 95.00th=[ 226], | 99.00th=[ 226], 99.50th=[ 226], 99.90th=[ 226], 99.95th=[ 226], | 99.99th=[ 226] bw ( KiB/s): min= 4096, max= 4096, per=62.58%, avg=4096.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 write: IOPS=2, BW=2084KiB/s (2134kB/s)(6144KiB/2948msec) clat (msec): min=48, max=1837, avg=390.69, stdev=709.70 lat (msec): min=48, max=1837, avg=390.77, stdev=709.69 clat percentiles (msec): | 1.00th=[ 49], 5.00th=[ 49], 10.00th=[ 49], 20.00th=[ 78], | 30.00th=[ 78], 40.00th=[ 100], 50.00th=[ 100], 60.00th=[ 127], | 70.00th=[ 157], 80.00th=[ 157], 90.00th=[ 1838], 95.00th=[ 1838], | 99.00th=[ 1838], 99.50th=[ 1838], 99.90th=[ 1838], 99.95th=[ 1838], | 99.99th=[ 1838] bw ( KiB/s): min= 2048, max= 4087, per=33.48%, avg=3067.50, stdev=1441.79, samples=2 iops : min= 2, max= 3, avg= 2.50, stdev= 0.71, samples=2 lat (msec) : 50=8.33%, 100=58.33%, 250=25.00% cpu : usr=0.00%, sys=3.77%, ctx=67, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17249: Fri Feb 14 02:33:28 2025 read: IOPS=1, BW=1636KiB/s (1676kB/s)(5120KiB/3129msec) clat (msec): min=30, max=173, avg=118.85, stdev=56.78 lat (msec): min=30, max=173, avg=118.86, stdev=56.78 clat percentiles (msec): | 1.00th=[ 31], 5.00th=[ 31], 10.00th=[ 31], 20.00th=[ 31], | 30.00th=[ 96], 40.00th=[ 96], 50.00th=[ 146], 60.00th=[ 146], | 70.00th=[ 148], 80.00th=[ 148], 90.00th=[ 174], 95.00th=[ 174], | 99.00th=[ 174], 99.50th=[ 174], 99.90th=[ 174], 99.95th=[ 174], | 99.99th=[ 174] bw ( KiB/s): min= 6131, max= 6131, per=93.67%, avg=6131.00, stdev= 0.00, samples=1 iops : min= 5, max= 5, avg= 5.00, stdev= 0.00, samples=1 write: IOPS=2, BW=2291KiB/s (2346kB/s)(7168KiB/3129msec) clat (msec): min=26, max=1938, avg=349.04, stdev=703.35 lat (msec): min=26, max=1939, avg=349.11, stdev=703.37 clat percentiles (msec): | 1.00th=[ 27], 5.00th=[ 27], 10.00th=[ 27], 20.00th=[ 32], | 30.00th=[ 36], 40.00th=[ 36], 50.00th=[ 103], 60.00th=[ 140], | 70.00th=[ 140], 80.00th=[ 169], 90.00th=[ 1938], 95.00th=[ 1938], | 99.00th=[ 1938], 99.50th=[ 1938], 99.90th=[ 1938], 99.95th=[ 1938], | 99.99th=[ 1938] bw ( KiB/s): min= 2048, max= 6131, per=44.63%, avg=4089.50, stdev=2887.12, samples=2 iops : min= 2, max= 5, avg= 3.50, stdev= 2.12, samples=2 lat (msec) : 50=33.33%, 100=8.33%, 250=50.00% cpu : usr=0.00%, sys=3.77%, ctx=71, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=6545KiB/s (6702kB/s), 691KiB/s-2470KiB/s (708kB/s-2529kB/s), io=20.0MiB (20.0MB), run=2902-3129msec WRITE: bw=9163KiB/s (9383kB/s), 1764KiB/s-3456KiB/s (1807kB/s-3539kB/s), io=28.0MiB (29.4MB), run=2902-3129msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=17261: Fri Feb 14 02:33:31 2025 read: IOPS=3, BW=15.0MiB/s (16.8MB/s)(12.0MiB/751msec) clat (msec): min=89, max=351, avg=238.35, stdev=134.43 lat (msec): min=89, max=351, avg=238.35, stdev=134.43 clat percentiles (msec): | 1.00th=[ 90], 5.00th=[ 90], 10.00th=[ 90], 20.00th=[ 90], | 30.00th=[ 90], 40.00th=[ 275], 50.00th=[ 275], 60.00th=[ 275], | 70.00th=[ 351], 80.00th=[ 351], 90.00th=[ 351], 95.00th=[ 351], | 99.00th=[ 351], 99.50th=[ 351], 99.90th=[ 351], 99.95th=[ 351], | 99.99th=[ 351] bw ( KiB/s): min= 8175, max= 8175, per=61.53%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 500=66.67% cpu : usr=0.00%, sys=16.27%, ctx=241, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17262: Fri Feb 14 02:33:31 2025 read: IOPS=0, BW=3322KiB/s (3402kB/s)(4096KiB/1233msec) clat (nsec): min=302022k, max=302022k, avg=302022430.00, stdev= 0.00 lat (nsec): min=302025k, max=302025k, avg=302024693.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 305], 5.00th=[ 305], 10.00th=[ 305], 20.00th=[ 305], | 30.00th=[ 305], 40.00th=[ 305], 50.00th=[ 305], 60.00th=[ 305], | 70.00th=[ 305], 80.00th=[ 305], 90.00th=[ 305], 95.00th=[ 305], | 99.00th=[ 305], 99.50th=[ 305], 99.90th=[ 305], 99.95th=[ 305], | 99.99th=[ 305] bw ( KiB/s): min= 8159, max= 8159, per=61.41%, avg=8159.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=1, BW=6644KiB/s (6803kB/s)(8192KiB/1233msec) clat (msec): min=409, max=498, avg=454.15, stdev=62.65 lat (msec): min=410, max=498, avg=454.36, stdev=62.54 clat percentiles (msec): | 1.00th=[ 409], 5.00th=[ 409], 10.00th=[ 409], 20.00th=[ 409], | 30.00th=[ 409], 40.00th=[ 409], 50.00th=[ 409], 60.00th=[ 498], | 70.00th=[ 498], 80.00th=[ 498], 90.00th=[ 498], 95.00th=[ 498], | 99.00th=[ 498], 99.50th=[ 498], 99.90th=[ 498], 99.95th=[ 498], | 99.99th=[ 498] bw ( KiB/s): min= 8126, max= 8126, per=37.84%, avg=8126.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 500=100.00% cpu : usr=0.00%, sys=20.78%, ctx=169, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17263: Fri Feb 14 02:33:31 2025 write: IOPS=1, BW=8138KiB/s (8333kB/s)(12.0MiB/1510msec) clat (msec): min=208, max=728, avg=487.42, stdev=262.42 lat (msec): min=208, max=728, avg=487.69, stdev=262.41 clat percentiles (msec): | 1.00th=[ 209], 5.00th=[ 209], 10.00th=[ 209], 20.00th=[ 209], | 30.00th=[ 209], 40.00th=[ 527], 50.00th=[ 527], 60.00th=[ 527], | 70.00th=[ 726], 80.00th=[ 726], 90.00th=[ 726], 95.00th=[ 726], | 99.00th=[ 726], 99.50th=[ 726], 99.90th=[ 726], 99.95th=[ 726], | 99.99th=[ 726] bw ( KiB/s): min=16318, max=16318, per=75.99%, avg=16318.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 750=66.67% cpu : usr=0.00%, sys=20.68%, ctx=71, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17264: Fri Feb 14 02:33:31 2025 write: IOPS=1, BW=8052KiB/s (8246kB/s)(12.0MiB/1526msec) clat (msec): min=258, max=759, avg=503.44, stdev=250.87 lat (msec): min=258, max=759, avg=503.85, stdev=250.83 clat percentiles (msec): | 1.00th=[ 259], 5.00th=[ 259], 10.00th=[ 259], 20.00th=[ 259], | 30.00th=[ 259], 40.00th=[ 493], 50.00th=[ 493], 60.00th=[ 493], | 70.00th=[ 760], 80.00th=[ 760], 90.00th=[ 760], 95.00th=[ 760], | 99.00th=[ 760], 99.50th=[ 760], 99.90th=[ 760], 99.95th=[ 760], | 99.99th=[ 760] bw ( KiB/s): min= 8192, max= 8192, per=38.15%, avg=8192.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 lat (msec) : 500=66.67%, 1000=33.33% cpu : usr=0.00%, sys=19.34%, ctx=85, majf=0, minf=28 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=12.0MiB/s (13.6MB/s), 3322KiB/s-15.0MiB/s (3402kB/s-16.8MB/s), io=16.0MiB (16.8MB), run=751-1233msec WRITE: bw=20.0MiB/s (21.0MB/s), 6644KiB/s-8138KiB/s (6803kB/s-8333kB/s), io=32.0MiB (33.6MB), run=1233-1526msec rand-rw: (groupid=0, jobs=1): err= 0: pid=17256: Fri Feb 14 02:33:32 2025 read: IOPS=4, BW=19.4MiB/s (20.3MB/s)(12.0MiB/620msec) clat (msec): min=162, max=236, avg=190.03, stdev=40.57 lat (msec): min=162, max=236, avg=190.04, stdev=40.57 clat percentiles (msec): | 1.00th=[ 163], 5.00th=[ 163], 10.00th=[ 163], 20.00th=[ 163], | 30.00th=[ 163], 40.00th=[ 171], 50.00th=[ 171], 60.00th=[ 171], | 70.00th=[ 236], 80.00th=[ 236], 90.00th=[ 236], 95.00th=[ 236], | 99.00th=[ 236], 99.50th=[ 236], 99.90th=[ 236], 99.95th=[ 236], | 99.99th=[ 236] bw ( KiB/s): min=16384, max=16384, per=100.00%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 250=100.00% cpu : usr=0.16%, sys=15.19%, ctx=63, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17257: Fri Feb 14 02:33:32 2025 read: IOPS=0, BW=1807KiB/s (1850kB/s)(4096KiB/2267msec) clat (nsec): min=467051k, max=467051k, avg=467051478.00, stdev= 0.00 lat (nsec): min=467054k, max=467054k, avg=467053597.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 468], 5.00th=[ 468], 10.00th=[ 468], 20.00th=[ 468], | 30.00th=[ 468], 40.00th=[ 468], 50.00th=[ 468], 60.00th=[ 468], | 70.00th=[ 468], 80.00th=[ 468], 90.00th=[ 468], 95.00th=[ 468], | 99.00th=[ 468], 99.50th=[ 468], 99.90th=[ 468], 99.95th=[ 468], | 99.99th=[ 468] bw ( KiB/s): min= 7816, max= 7816, per=100.00%, avg=7816.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=0, BW=3614KiB/s (3700kB/s)(8192KiB/2267msec) clat (msec): min=362, max=1356, avg=859.01, stdev=702.86 lat (msec): min=362, max=1356, avg=859.33, stdev=702.87 clat percentiles (msec): | 1.00th=[ 363], 5.00th=[ 363], 10.00th=[ 363], 20.00th=[ 363], | 30.00th=[ 363], 40.00th=[ 363], 50.00th=[ 363], 60.00th=[ 1351], | 70.00th=[ 1351], 80.00th=[ 1351], 90.00th=[ 1351], 95.00th=[ 1351], | 99.00th=[ 1351], 99.50th=[ 1351], 99.90th=[ 1351], 99.95th=[ 1351], | 99.99th=[ 1351] bw ( KiB/s): min= 7742, max= 7742, per=53.56%, avg=7742.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 500=66.67% cpu : usr=0.00%, sys=7.77%, ctx=67, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17258: Fri Feb 14 02:33:32 2025 write: IOPS=4, BW=17.5MiB/s (18.3MB/s)(12.0MiB/687msec) clat (msec): min=210, max=230, avg=223.10, stdev=10.93 lat (msec): min=210, max=231, avg=223.36, stdev=10.89 clat percentiles (msec): | 1.00th=[ 211], 5.00th=[ 211], 10.00th=[ 211], 20.00th=[ 211], | 30.00th=[ 211], 40.00th=[ 228], 50.00th=[ 228], 60.00th=[ 228], | 70.00th=[ 232], 80.00th=[ 232], 90.00th=[ 232], 95.00th=[ 232], | 99.00th=[ 232], 99.50th=[ 232], 99.90th=[ 232], 99.95th=[ 232], | 99.99th=[ 232] bw ( KiB/s): min=16318, max=16318, per=100.00%, avg=16318.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 lat (msec) : 250=100.00% cpu : usr=0.44%, sys=10.79%, ctx=30, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17259: Fri Feb 14 02:33:32 2025 write: IOPS=2, BW=9163KiB/s (9383kB/s)(12.0MiB/1341msec) clat (msec): min=100, max=1091, avg=437.06, stdev=566.91 lat (msec): min=101, max=1091, avg=437.31, stdev=566.83 clat percentiles (msec): | 1.00th=[ 102], 5.00th=[ 102], 10.00th=[ 102], 20.00th=[ 102], | 30.00th=[ 102], 40.00th=[ 120], 50.00th=[ 120], 60.00th=[ 120], | 70.00th=[ 1099], 80.00th=[ 1099], 90.00th=[ 1099], 95.00th=[ 1099], | 99.00th=[ 1099], 99.50th=[ 1099], 99.90th=[ 1099], 99.95th=[ 1099], | 99.99th=[ 1099] bw ( KiB/s): min=16351, max=16351, per=100.00%, avg=16351.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67% cpu : usr=0.00%, sys=8.43%, ctx=47, majf=0, minf=28 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=7227KiB/s (7401kB/s), 1807KiB/s-19.4MiB/s (1850kB/s-20.3MB/s), io=16.0MiB (16.8MB), run=620-2267msec WRITE: bw=14.1MiB/s (14.8MB/s), 3614KiB/s-17.5MiB/s (3700kB/s-18.3MB/s), io=32.0MiB (33.6MB), run=687-2267msec