== sanity test 398b: DIO and buffer IO race ============== 19:02:58 (1746140578) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 2.1399 s, 23.5 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=24293: Thu May 1 19:03:45 2025 read: IOPS=35, BW=142KiB/s (145kB/s)(5992KiB/42316msec) clat (usec): min=72, max=406047, avg=16266.57, stdev=14422.41 lat (usec): min=72, max=406049, avg=16267.69, stdev=14422.42 clat percentiles (usec): | 1.00th=[ 92], 5.00th=[ 129], 10.00th=[ 5932], 20.00th=[ 8356], | 30.00th=[ 10290], 40.00th=[ 11863], 50.00th=[ 13829], 60.00th=[ 16188], | 70.00th=[ 19268], 80.00th=[ 23200], 90.00th=[ 29492], 95.00th=[ 34341], | 99.00th=[ 55313], 99.50th=[ 59507], 99.90th=[ 64750], 99.95th=[404751], | 99.99th=[404751] bw ( KiB/s): min= 16, max= 262, per=24.79%, avg=140.56, stdev=41.30, samples=84 iops : min= 4, max= 65, avg=34.80, stdev=10.36, samples=84 write: IOPS=37, BW=149KiB/s (152kB/s)(6296KiB/42316msec) clat (usec): min=522, max=90816, avg=11279.72, stdev=12550.56 lat (usec): min=523, max=90818, avg=11282.31, stdev=12551.68 clat percentiles (usec): | 1.00th=[ 685], 5.00th=[ 750], 10.00th=[ 799], 20.00th=[ 914], | 30.00th=[ 1663], 40.00th=[ 3326], 50.00th=[ 7570], 60.00th=[11076], | 70.00th=[14746], 80.00th=[19530], 90.00th=[28443], 95.00th=[35914], | 99.00th=[54789], 99.50th=[67634], 99.90th=[79168], 99.95th=[90702], | 99.99th=[90702] bw ( KiB/s): min= 47, max= 288, per=25.09%, avg=149.01, stdev=44.93, samples=83 iops : min= 11, max= 72, avg=36.90, stdev=11.25, samples=83 lat (usec) : 100=1.01%, 250=2.05%, 500=0.13%, 750=2.64%, 1000=8.98% lat (msec) : 2=5.34%, 4=5.11%, 10=17.45%, 20=33.63%, 50=22.46% lat (msec) : 100=1.17%, 500=0.03% cpu : usr=0.15%, sys=10.76%, ctx=4896, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24294: Thu May 1 19:03:45 2025 read: IOPS=35, BW=140KiB/s (144kB/s)(5940KiB/42301msec) clat (usec): min=75, max=323303, avg=15691.46, stdev=13562.79 lat (usec): min=76, max=323306, avg=15695.03, stdev=13562.56 clat percentiles (usec): | 1.00th=[ 96], 5.00th=[ 123], 10.00th=[ 5538], 20.00th=[ 7898], | 30.00th=[ 9372], 40.00th=[ 11207], 50.00th=[ 13435], 60.00th=[ 15533], | 70.00th=[ 18482], 80.00th=[ 22152], 90.00th=[ 28443], 95.00th=[ 33817], | 99.00th=[ 58459], 99.50th=[ 62653], 99.90th=[ 93848], 99.95th=[325059], | 99.99th=[325059] bw ( KiB/s): min= 15, max= 240, per=24.85%, avg=140.90, stdev=39.12, samples=83 iops : min= 3, max= 60, avg=34.96, stdev= 9.78, samples=83 write: IOPS=37, BW=150KiB/s (154kB/s)(6348KiB/42301msec) clat (usec): min=578, max=76420, avg=11819.49, stdev=12811.92 lat (usec): min=579, max=76421, avg=11825.04, stdev=12813.58 clat percentiles (usec): | 1.00th=[ 685], 5.00th=[ 742], 10.00th=[ 791], 20.00th=[ 947], | 30.00th=[ 1778], 40.00th=[ 3720], 50.00th=[ 7898], 60.00th=[11207], | 70.00th=[15270], 80.00th=[21365], 90.00th=[30016], 95.00th=[36963], | 99.00th=[55837], 99.50th=[64750], 99.90th=[76022], 99.95th=[76022], | 99.99th=[76022] bw ( KiB/s): min= 80, max= 287, per=25.25%, avg=149.99, stdev=46.75, samples=83 iops : min= 20, max= 71, avg=37.23, stdev=11.66, samples=83 lat (usec) : 100=0.88%, 250=2.57%, 500=0.07%, 750=2.80%, 1000=8.33% lat (msec) : 2=5.34%, 4=4.88%, 10=20.74%, 20=30.40%, 50=22.36% lat (msec) : 100=1.60%, 500=0.03% cpu : usr=0.21%, sys=10.55%, ctx=4769, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24296: Thu May 1 19:03:45 2025 read: IOPS=35, BW=143KiB/s (147kB/s)(6028KiB/42007msec) clat (usec): min=73, max=402765, avg=15940.59, stdev=14262.25 lat (usec): min=73, max=402765, avg=15941.79, stdev=14262.28 clat percentiles (usec): | 1.00th=[ 91], 5.00th=[ 129], 10.00th=[ 5800], 20.00th=[ 8160], | 30.00th=[ 10028], 40.00th=[ 11731], 50.00th=[ 13960], 60.00th=[ 16319], | 70.00th=[ 19006], 80.00th=[ 22152], 90.00th=[ 27919], 95.00th=[ 34866], | 99.00th=[ 50594], 99.50th=[ 59507], 99.90th=[ 84411], 99.95th=[404751], | 99.99th=[404751] bw ( KiB/s): min= 8, max= 264, per=25.24%, avg=143.08, stdev=43.84, samples=84 iops : min= 2, max= 66, avg=35.45, stdev=10.99, samples=84 write: IOPS=37, BW=149KiB/s (153kB/s)(6260KiB/42007msec) clat (usec): min=483, max=78993, avg=11375.26, stdev=12173.42 lat (usec): min=483, max=78994, avg=11376.91, stdev=12173.20 clat percentiles (usec): | 1.00th=[ 701], 5.00th=[ 750], 10.00th=[ 791], 20.00th=[ 930], | 30.00th=[ 1762], 40.00th=[ 3556], 50.00th=[ 7898], 60.00th=[11338], | 70.00th=[14877], 80.00th=[20317], 90.00th=[28443], 95.00th=[35390], | 99.00th=[52691], 99.50th=[61604], 99.90th=[71828], 99.95th=[79168], | 99.99th=[79168] bw ( KiB/s): min= 24, max= 304, per=24.99%, avg=148.42, stdev=49.44, samples=84 iops : min= 6, max= 76, avg=36.79, stdev=12.42, samples=84 lat (usec) : 100=0.98%, 250=2.08%, 500=0.13%, 750=2.57%, 1000=8.85% lat (msec) : 2=5.14%, 4=4.95%, 10=18.55%, 20=33.66%, 50=21.91% lat (msec) : 100=1.14%, 500=0.03% cpu : usr=0.11%, sys=10.84%, ctx=4828, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24298: Thu May 1 19:03:45 2025 read: IOPS=35, BW=143KiB/s (147kB/s)(6052KiB/42235msec) clat (usec): min=74, max=80129, avg=16050.57, stdev=10953.31 lat (usec): min=75, max=80130, avg=16053.52, stdev=10954.10 clat percentiles (usec): | 1.00th=[ 94], 5.00th=[ 133], 10.00th=[ 6128], 20.00th=[ 8356], | 30.00th=[ 9896], 40.00th=[11469], 50.00th=[13435], 60.00th=[15795], | 70.00th=[19268], 80.00th=[23200], 90.00th=[29754], 95.00th=[36439], | 99.00th=[55837], 99.50th=[65274], 99.90th=[72877], 99.95th=[80217], | 99.99th=[80217] bw ( KiB/s): min= 55, max= 232, per=25.37%, avg=143.86, stdev=39.17, samples=83 iops : min= 13, max= 58, avg=35.59, stdev= 9.77, samples=83 write: IOPS=36, BW=148KiB/s (151kB/s)(6236KiB/42235msec) clat (usec): min=586, max=349714, avg=11359.24, stdev=14753.07 lat (usec): min=587, max=349715, avg=11360.75, stdev=14753.04 clat percentiles (usec): | 1.00th=[ 701], 5.00th=[ 758], 10.00th=[ 799], 20.00th=[ 971], | 30.00th=[ 1909], 40.00th=[ 3589], 50.00th=[ 7701], 60.00th=[ 11207], | 70.00th=[ 14615], 80.00th=[ 19268], 90.00th=[ 27395], 95.00th=[ 35390], | 99.00th=[ 52167], 99.50th=[ 61080], 99.90th=[ 83362], 99.95th=[350225], | 99.99th=[350225] bw ( KiB/s): min= 63, max= 287, per=25.00%, avg=148.48, stdev=48.71, samples=83 iops : min= 15, max= 71, avg=36.75, stdev=12.26, samples=83 lat (usec) : 100=0.94%, 250=2.18%, 750=2.41%, 1000=8.07% lat (msec) : 2=5.01%, 4=5.99%, 10=19.14%, 20=33.14%, 50=21.74% lat (msec) : 100=1.33%, 500=0.03% cpu : usr=0.17%, sys=10.77%, ctx=4894, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=567KiB/s (581kB/s), 140KiB/s-143KiB/s (144kB/s-147kB/s), io=23.4MiB (24.6MB), run=42007-42316msec WRITE: bw=594KiB/s (608kB/s), 148KiB/s-150KiB/s (151kB/s-154kB/s), io=24.6MiB (25.7MB), run=42007-42316msec rand-rw: (groupid=0, jobs=1): err= 0: pid=24292: Thu May 1 19:04:07 2025 read: IOPS=23, BW=93.9KiB/s (96.2kB/s)(5992KiB/63783msec) clat (msec): min=3, max=405, avg=15.71, stdev=16.46 lat (msec): min=3, max=405, avg=15.72, stdev=16.46 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 6], 10.00th=[ 7], 20.00th=[ 8], | 30.00th=[ 9], 40.00th=[ 10], 50.00th=[ 11], 60.00th=[ 13], | 70.00th=[ 16], 80.00th=[ 20], 90.00th=[ 32], 95.00th=[ 44], | 99.00th=[ 70], 99.50th=[ 84], 99.90th=[ 101], 99.95th=[ 405], | 99.99th=[ 405] bw ( KiB/s): min= 23, max= 232, per=24.95%, avg=93.82, stdev=46.62, samples=126 iops : min= 5, max= 58, avg=23.12, stdev=11.73, samples=126 write: IOPS=24, BW=98.7KiB/s (101kB/s)(6296KiB/63783msec) clat (msec): min=5, max=113, avg=25.45, stdev=14.91 lat (msec): min=5, max=113, avg=25.45, stdev=14.91 clat percentiles (msec): | 1.00th=[ 9], 5.00th=[ 11], 10.00th=[ 13], 20.00th=[ 15], | 30.00th=[ 17], 40.00th=[ 19], 50.00th=[ 21], 60.00th=[ 24], | 70.00th=[ 28], 80.00th=[ 35], 90.00th=[ 46], 95.00th=[ 55], | 99.00th=[ 81], 99.50th=[ 93], 99.90th=[ 111], 99.95th=[ 114], | 99.99th=[ 114] bw ( KiB/s): min= 23, max= 192, per=25.00%, avg=98.51, stdev=44.69, samples=126 iops : min= 5, max= 48, avg=24.29, stdev=11.25, samples=126 lat (msec) : 4=0.20%, 10=22.53%, 20=41.24%, 50=30.79%, 100=5.08% lat (msec) : 250=0.13%, 500=0.03% cpu : usr=0.07%, sys=5.07%, ctx=3463, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24295: Thu May 1 19:04:07 2025 read: IOPS=23, BW=93.1KiB/s (95.4kB/s)(5940KiB/63781msec) clat (msec): min=3, max=351, avg=15.53, stdev=15.78 lat (msec): min=3, max=351, avg=15.53, stdev=15.78 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 6], 10.00th=[ 6], 20.00th=[ 8], | 30.00th=[ 9], 40.00th=[ 10], 50.00th=[ 11], 60.00th=[ 13], | 70.00th=[ 16], 80.00th=[ 20], 90.00th=[ 32], 95.00th=[ 42], | 99.00th=[ 77], 99.50th=[ 86], 99.90th=[ 107], 99.95th=[ 351], | 99.99th=[ 351] bw ( KiB/s): min= 7, max= 232, per=24.57%, avg=92.38, stdev=48.41, samples=127 iops : min= 1, max= 58, avg=22.70, stdev=12.15, samples=127 write: IOPS=24, BW=99.5KiB/s (102kB/s)(6348KiB/63781msec) clat (msec): min=5, max=116, avg=25.54, stdev=15.50 lat (msec): min=5, max=116, avg=25.54, stdev=15.50 clat percentiles (msec): | 1.00th=[ 9], 5.00th=[ 11], 10.00th=[ 13], 20.00th=[ 15], | 30.00th=[ 17], 40.00th=[ 19], 50.00th=[ 21], 60.00th=[ 24], | 70.00th=[ 28], 80.00th=[ 34], 90.00th=[ 47], 95.00th=[ 58], | 99.00th=[ 82], 99.50th=[ 90], 99.90th=[ 110], 99.95th=[ 116], | 99.99th=[ 116] bw ( KiB/s): min= 23, max= 208, per=25.18%, avg=99.21, stdev=45.33, samples=126 iops : min= 5, max= 52, avg=24.40, stdev=11.39, samples=126 lat (msec) : 4=0.49%, 10=22.92%, 20=40.56%, 50=30.47%, 100=5.27% lat (msec) : 250=0.26%, 500=0.03% cpu : usr=0.08%, sys=4.92%, ctx=3446, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24297: Thu May 1 19:04:07 2025 read: IOPS=23, BW=94.8KiB/s (97.0kB/s)(6028KiB/63614msec) clat (usec): min=3247, max=95161, avg=15391.00, stdev=12850.42 lat (usec): min=3247, max=95162, avg=15393.12, stdev=12850.26 clat percentiles (usec): | 1.00th=[ 4359], 5.00th=[ 5407], 10.00th=[ 6128], 20.00th=[ 7308], | 30.00th=[ 8291], 40.00th=[ 9372], 50.00th=[10552], 60.00th=[12518], | 70.00th=[15139], 80.00th=[20055], 90.00th=[32375], 95.00th=[44303], | 99.00th=[65799], 99.50th=[74974], 99.90th=[92799], 99.95th=[94897], | 99.99th=[94897] bw ( KiB/s): min= 16, max= 239, per=25.23%, avg=94.87, stdev=47.35, samples=126 iops : min= 4, max= 59, avg=23.42, stdev=11.83, samples=126 write: IOPS=24, BW=98.4KiB/s (101kB/s)(6260KiB/63614msec) clat (msec): min=5, max=340, avg=25.69, stdev=17.38 lat (msec): min=5, max=340, avg=25.69, stdev=17.38 clat percentiles (msec): | 1.00th=[ 9], 5.00th=[ 11], 10.00th=[ 13], 20.00th=[ 15], | 30.00th=[ 17], 40.00th=[ 18], 50.00th=[ 21], 60.00th=[ 24], | 70.00th=[ 28], 80.00th=[ 36], 90.00th=[ 48], 95.00th=[ 56], | 99.00th=[ 88], 99.50th=[ 100], 99.90th=[ 115], 99.95th=[ 342], | 99.99th=[ 342] bw ( KiB/s): min= 32, max= 191, per=25.00%, avg=98.50, stdev=45.19, samples=126 iops : min= 8, max= 47, avg=24.33, stdev=11.34, samples=126 lat (msec) : 4=0.26%, 10=23.63%, 20=40.36%, 50=29.92%, 100=5.63% lat (msec) : 250=0.16%, 500=0.03% cpu : usr=0.11%, sys=4.91%, ctx=3406, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24299: Thu May 1 19:04:07 2025 read: IOPS=23, BW=95.0KiB/s (97.3kB/s)(6052KiB/63689msec) clat (msec): min=3, max=104, avg=15.38, stdev=12.88 lat (msec): min=3, max=104, avg=15.38, stdev=12.88 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 6], 10.00th=[ 7], 20.00th=[ 8], | 30.00th=[ 9], 40.00th=[ 10], 50.00th=[ 11], 60.00th=[ 13], | 70.00th=[ 15], 80.00th=[ 21], 90.00th=[ 34], 95.00th=[ 44], | 99.00th=[ 66], 99.50th=[ 78], 99.90th=[ 92], 99.95th=[ 105], | 99.99th=[ 105] bw ( KiB/s): min= 8, max= 224, per=25.32%, avg=95.19, stdev=46.77, samples=126 iops : min= 2, max= 56, avg=23.39, stdev=11.74, samples=126 write: IOPS=24, BW=97.9KiB/s (100kB/s)(6236KiB/63689msec) clat (msec): min=5, max=388, avg=25.80, stdev=17.98 lat (msec): min=5, max=388, avg=25.81, stdev=17.98 clat percentiles (msec): | 1.00th=[ 9], 5.00th=[ 12], 10.00th=[ 13], 20.00th=[ 15], | 30.00th=[ 17], 40.00th=[ 19], 50.00th=[ 21], 60.00th=[ 24], | 70.00th=[ 28], 80.00th=[ 35], 90.00th=[ 46], 95.00th=[ 56], | 99.00th=[ 89], 99.50th=[ 100], 99.90th=[ 121], 99.95th=[ 388], | 99.99th=[ 388] bw ( KiB/s): min= 39, max= 224, per=24.82%, avg=97.79, stdev=44.15, samples=126 iops : min= 9, max= 56, avg=24.04, stdev=11.12, samples=126 lat (msec) : 4=0.26%, 10=22.95%, 20=40.36%, 50=31.25%, 100=4.92% lat (msec) : 250=0.23%, 500=0.03% cpu : usr=0.09%, sys=4.92%, ctx=3429, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=376KiB/s (385kB/s), 93.1KiB/s-95.0KiB/s (95.4kB/s-97.3kB/s), io=23.4MiB (24.6MB), run=63614-63783msec WRITE: bw=394KiB/s (404kB/s), 97.9KiB/s-99.5KiB/s (100kB/s-102kB/s), io=24.6MiB (25.7MB), run=63614-63783msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=24362: Thu May 1 19:04:18 2025 read: IOPS=38, BW=621KiB/s (636kB/s)(5920KiB/9539msec) clat (usec): min=84, max=130578, avg=13796.83, stdev=11293.92 lat (usec): min=84, max=130578, avg=13798.45, stdev=11294.03 clat percentiles (usec): | 1.00th=[ 98], 5.00th=[ 112], 10.00th=[ 133], 20.00th=[ 6652], | 30.00th=[ 8717], 40.00th=[ 10683], 50.00th=[ 12256], 60.00th=[ 14353], | 70.00th=[ 16909], 80.00th=[ 20841], 90.00th=[ 25035], 95.00th=[ 31065], | 99.00th=[ 46400], 99.50th=[ 59507], 99.90th=[130548], 99.95th=[130548], | 99.99th=[130548] bw ( KiB/s): min= 320, max= 864, per=26.33%, avg=618.26, stdev=150.66, samples=19 iops : min= 20, max= 54, avg=38.37, stdev= 9.32, samples=19 write: IOPS=41, BW=668KiB/s (684kB/s)(6368KiB/9539msec) clat (usec): min=818, max=60859, avg=11050.78, stdev=12437.68 lat (usec): min=819, max=60861, avg=11061.32, stdev=12443.69 clat percentiles (usec): | 1.00th=[ 1020], 5.00th=[ 1106], 10.00th=[ 1205], 20.00th=[ 1352], | 30.00th=[ 1975], 40.00th=[ 2966], 50.00th=[ 6063], 60.00th=[ 9634], | 70.00th=[13960], 80.00th=[20055], 90.00th=[27919], 95.00th=[37487], | 99.00th=[58459], 99.50th=[59507], 99.90th=[61080], 99.95th=[61080], | 99.99th=[61080] bw ( KiB/s): min= 416, max= 1216, per=26.92%, avg=665.32, stdev=207.69, samples=19 iops : min= 26, max= 76, avg=41.32, stdev=12.97, samples=19 lat (usec) : 100=0.78%, 250=5.47%, 500=0.52%, 750=0.39%, 1000=0.52% lat (msec) : 2=15.36%, 4=7.68%, 10=18.10%, 20=30.47%, 50=19.27% lat (msec) : 100=1.30%, 250=0.13% cpu : usr=0.27%, sys=12.33%, ctx=1215, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24363: Thu May 1 19:04:18 2025 read: IOPS=35, BW=563KiB/s (577kB/s)(5744KiB/10197msec) clat (usec): min=77, max=70108, avg=14574.90, stdev=10830.78 lat (usec): min=78, max=70109, avg=14575.90, stdev=10830.80 clat percentiles (usec): | 1.00th=[ 91], 5.00th=[ 111], 10.00th=[ 127], 20.00th=[ 6521], | 30.00th=[ 8848], 40.00th=[10945], 50.00th=[13173], 60.00th=[15533], | 70.00th=[18220], 80.00th=[22152], 90.00th=[27919], 95.00th=[34341], | 99.00th=[49021], 99.50th=[55837], 99.90th=[69731], 99.95th=[69731], | 99.99th=[69731] bw ( KiB/s): min= 158, max= 832, per=23.61%, avg=554.45, stdev=182.42, samples=20 iops : min= 9, max= 52, avg=34.25, stdev=11.56, samples=20 write: IOPS=40, BW=642KiB/s (657kB/s)(6544KiB/10197msec) clat (usec): min=957, max=110716, avg=11795.73, stdev=14049.30 lat (usec): min=957, max=110717, avg=11798.88, stdev=14048.22 clat percentiles (usec): | 1.00th=[ 1012], 5.00th=[ 1074], 10.00th=[ 1139], 20.00th=[ 1319], | 30.00th=[ 2212], 40.00th=[ 4015], 50.00th=[ 6849], 60.00th=[ 10028], | 70.00th=[ 14091], 80.00th=[ 19268], 90.00th=[ 29230], 95.00th=[ 39060], | 99.00th=[ 65799], 99.50th=[ 81265], 99.90th=[110625], 99.95th=[110625], | 99.99th=[110625] bw ( KiB/s): min= 254, max= 1024, per=25.30%, avg=625.15, stdev=211.39, samples=20 iops : min= 15, max= 64, avg=38.70, stdev=13.28, samples=20 lat (usec) : 100=1.17%, 250=5.08%, 500=0.13%, 750=0.39%, 1000=0.78% lat (msec) : 2=14.71%, 4=6.12%, 10=19.79%, 20=29.82%, 50=20.31% lat (msec) : 100=1.56%, 250=0.13% cpu : usr=0.17%, sys=12.36%, ctx=1292, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24364: Thu May 1 19:04:18 2025 read: IOPS=38, BW=620KiB/s (635kB/s)(6256KiB/10094msec) clat (usec): min=53, max=67946, avg=14527.35, stdev=10911.66 lat (usec): min=53, max=67948, avg=14531.12, stdev=10912.55 clat percentiles (usec): | 1.00th=[ 82], 5.00th=[ 106], 10.00th=[ 119], 20.00th=[ 6783], | 30.00th=[ 8717], 40.00th=[10683], 50.00th=[12911], 60.00th=[15270], | 70.00th=[18220], 80.00th=[21890], 90.00th=[28967], 95.00th=[34866], | 99.00th=[50594], 99.50th=[51643], 99.90th=[67634], 99.95th=[67634], | 99.99th=[67634] bw ( KiB/s): min= 318, max= 1053, per=26.25%, avg=616.30, stdev=178.39, samples=20 iops : min= 19, max= 65, avg=38.10, stdev=11.06, samples=20 write: IOPS=37, BW=598KiB/s (612kB/s)(6032KiB/10094msec) clat (usec): min=799, max=68565, avg=11279.53, stdev=12384.59 lat (usec): min=800, max=68566, avg=11281.71, stdev=12384.08 clat percentiles (usec): | 1.00th=[ 979], 5.00th=[ 1123], 10.00th=[ 1205], 20.00th=[ 1565], | 30.00th=[ 2376], 40.00th=[ 3392], 50.00th=[ 6390], 60.00th=[10028], | 70.00th=[15401], 80.00th=[20055], 90.00th=[27919], 95.00th=[35914], | 99.00th=[54264], 99.50th=[56361], 99.90th=[68682], 99.95th=[68682], | 99.99th=[68682] bw ( KiB/s): min= 287, max= 990, per=23.72%, avg=586.00, stdev=176.10, samples=20 iops : min= 17, max= 61, avg=36.20, stdev=11.01, samples=20 lat (usec) : 100=1.43%, 250=6.38%, 500=0.13%, 1000=0.78% lat (msec) : 2=12.50%, 4=7.68%, 10=19.27%, 20=29.17%, 50=20.96% lat (msec) : 100=1.69% cpu : usr=0.06%, sys=12.33%, ctx=1304, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24366: Thu May 1 19:04:18 2025 read: IOPS=37, BW=605KiB/s (620kB/s)(6032KiB/9968msec) clat (usec): min=64, max=77944, avg=14136.25, stdev=11170.01 lat (usec): min=65, max=77946, avg=14140.02, stdev=11170.35 clat percentiles (usec): | 1.00th=[ 93], 5.00th=[ 103], 10.00th=[ 121], 20.00th=[ 5342], | 30.00th=[ 8356], 40.00th=[10421], 50.00th=[12649], 60.00th=[15270], | 70.00th=[17433], 80.00th=[21627], 90.00th=[28705], 95.00th=[33817], | 99.00th=[54789], 99.50th=[72877], 99.90th=[78119], 99.95th=[78119], | 99.99th=[78119] bw ( KiB/s): min= 222, max= 896, per=25.13%, avg=590.05, stdev=203.83, samples=19 iops : min= 13, max= 56, avg=36.53, stdev=13.01, samples=19 write: IOPS=39, BW=628KiB/s (643kB/s)(6256KiB/9968msec) clat (usec): min=1014, max=66247, avg=11390.20, stdev=11658.29 lat (usec): min=1016, max=66248, avg=11393.31, stdev=11657.00 clat percentiles (usec): | 1.00th=[ 1029], 5.00th=[ 1139], 10.00th=[ 1221], 20.00th=[ 1532], | 30.00th=[ 2278], 40.00th=[ 3752], 50.00th=[ 7177], 60.00th=[11207], | 70.00th=[15270], 80.00th=[20055], 90.00th=[28181], 95.00th=[34866], | 99.00th=[48497], 99.50th=[53740], 99.90th=[66323], 99.95th=[66323], | 99.99th=[66323] bw ( KiB/s): min= 222, max= 992, per=25.10%, avg=620.21, stdev=228.05, samples=19 iops : min= 13, max= 62, avg=38.42, stdev=14.38, samples=19 lat (usec) : 100=1.43%, 250=5.60%, 500=0.52%, 750=0.13% lat (msec) : 2=13.15%, 4=8.72%, 10=18.23%, 20=30.73%, 50=20.57% lat (msec) : 100=0.91% cpu : usr=0.09%, sys=12.08%, ctx=1291, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2349KiB/s (2405kB/s), 563KiB/s-621KiB/s (577kB/s-636kB/s), io=23.4MiB (24.5MB), run=9539-10197msec WRITE: bw=2471KiB/s (2531kB/s), 598KiB/s-668KiB/s (612kB/s-684kB/s), io=24.6MiB (25.8MB), run=9539-10197msec rand-rw: (groupid=0, jobs=1): err= 0: pid=24365: Thu May 1 19:04:24 2025 read: IOPS=23, BW=381KiB/s (390kB/s)(5920KiB/15528msec) clat (usec): min=3355, max=99469, avg=15065.51, stdev=11739.82 lat (usec): min=3356, max=99470, avg=15066.81, stdev=11739.78 clat percentiles (usec): | 1.00th=[ 4293], 5.00th=[ 5473], 10.00th=[ 6063], 20.00th=[ 7504], | 30.00th=[ 8848], 40.00th=[10159], 50.00th=[11207], 60.00th=[12780], | 70.00th=[15139], 80.00th=[19268], 90.00th=[27657], 95.00th=[41157], | 99.00th=[57934], 99.50th=[68682], 99.90th=[99091], 99.95th=[99091], | 99.99th=[99091] bw ( KiB/s): min= 128, max= 796, per=24.94%, avg=380.06, stdev=161.63, samples=31 iops : min= 8, max= 49, avg=23.35, stdev=10.12, samples=31 write: IOPS=25, BW=410KiB/s (420kB/s)(6368KiB/15528msec) clat (msec): min=7, max=147, avg=24.75, stdev=14.62 lat (msec): min=7, max=147, avg=24.75, stdev=14.62 clat percentiles (msec): | 1.00th=[ 9], 5.00th=[ 11], 10.00th=[ 13], 20.00th=[ 15], | 30.00th=[ 17], 40.00th=[ 19], 50.00th=[ 21], 60.00th=[ 23], | 70.00th=[ 27], 80.00th=[ 33], 90.00th=[ 45], 95.00th=[ 52], | 99.00th=[ 89], 99.50th=[ 97], 99.90th=[ 148], 99.95th=[ 148], | 99.99th=[ 148] bw ( KiB/s): min= 192, max= 768, per=25.49%, avg=408.90, stdev=159.82, samples=31 iops : min= 12, max= 48, avg=25.16, stdev= 9.99, samples=31 lat (msec) : 4=0.26%, 10=20.18%, 20=43.36%, 50=31.77%, 100=4.30% lat (msec) : 250=0.13% cpu : usr=0.11%, sys=5.51%, ctx=886, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24367: Thu May 1 19:04:24 2025 read: IOPS=22, BW=366KiB/s (374kB/s)(5744KiB/15709msec) clat (msec): min=3, max=122, avg=15.14, stdev=13.27 lat (msec): min=3, max=122, avg=15.15, stdev=13.27 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 6], 10.00th=[ 6], 20.00th=[ 8], | 30.00th=[ 9], 40.00th=[ 10], 50.00th=[ 11], 60.00th=[ 13], | 70.00th=[ 14], 80.00th=[ 18], 90.00th=[ 34], 95.00th=[ 42], | 99.00th=[ 65], 99.50th=[ 94], 99.90th=[ 123], 99.95th=[ 123], | 99.99th=[ 123] bw ( KiB/s): min= 95, max= 736, per=23.79%, avg=362.61, stdev=178.93, samples=31 iops : min= 5, max= 46, avg=22.29, stdev=11.26, samples=31 write: IOPS=26, BW=417KiB/s (427kB/s)(6544KiB/15709msec) clat (msec): min=6, max=109, avg=25.04, stdev=14.36 lat (msec): min=6, max=109, avg=25.04, stdev=14.36 clat percentiles (msec): | 1.00th=[ 8], 5.00th=[ 11], 10.00th=[ 13], 20.00th=[ 15], | 30.00th=[ 17], 40.00th=[ 19], 50.00th=[ 21], 60.00th=[ 24], | 70.00th=[ 28], 80.00th=[ 34], 90.00th=[ 43], 95.00th=[ 55], | 99.00th=[ 74], 99.50th=[ 85], 99.90th=[ 110], 99.95th=[ 110], | 99.99th=[ 110] bw ( KiB/s): min= 223, max= 768, per=25.69%, avg=412.13, stdev=156.32, samples=31 iops : min= 13, max= 48, avg=25.39, stdev= 9.85, samples=31 lat (msec) : 4=0.13%, 10=20.96%, 20=42.06%, 50=33.07%, 100=3.52% lat (msec) : 250=0.26% cpu : usr=0.09%, sys=5.39%, ctx=877, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24368: Thu May 1 19:04:24 2025 read: IOPS=25, BW=404KiB/s (414kB/s)(6256KiB/15483msec) clat (usec): min=3794, max=99705, avg=14544.57, stdev=12454.11 lat (usec): min=3794, max=99705, avg=14545.74, stdev=12454.08 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 6], 10.00th=[ 6], 20.00th=[ 7], | 30.00th=[ 9], 40.00th=[ 9], 50.00th=[ 11], 60.00th=[ 12], | 70.00th=[ 14], 80.00th=[ 20], 90.00th=[ 29], 95.00th=[ 42], | 99.00th=[ 66], 99.50th=[ 80], 99.90th=[ 101], 99.95th=[ 101], | 99.99th=[ 101] bw ( KiB/s): min= 96, max= 1113, per=26.48%, avg=403.57, stdev=218.65, samples=30 iops : min= 6, max= 69, avg=24.90, stdev=13.73, samples=30 write: IOPS=24, BW=390KiB/s (399kB/s)(6032KiB/15483msec) clat (msec): min=7, max=146, avg=25.51, stdev=14.70 lat (msec): min=7, max=146, avg=25.51, stdev=14.70 clat percentiles (msec): | 1.00th=[ 10], 5.00th=[ 11], 10.00th=[ 13], 20.00th=[ 15], | 30.00th=[ 17], 40.00th=[ 19], 50.00th=[ 22], 60.00th=[ 24], | 70.00th=[ 29], 80.00th=[ 36], 90.00th=[ 44], 95.00th=[ 52], | 99.00th=[ 72], 99.50th=[ 88], 99.90th=[ 146], 99.95th=[ 146], | 99.99th=[ 146] bw ( KiB/s): min= 126, max= 702, per=23.64%, avg=379.13, stdev=150.39, samples=30 iops : min= 7, max= 43, avg=23.37, stdev= 9.49, samples=30 lat (msec) : 4=0.39%, 10=25.78%, 20=37.11%, 50=32.68%, 100=3.91% lat (msec) : 250=0.13% cpu : usr=0.12%, sys=5.30%, ctx=878, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24369: Thu May 1 19:04:24 2025 read: IOPS=24, BW=389KiB/s (398kB/s)(6032KiB/15504msec) clat (usec): min=3281, max=76321, avg=16509.33, stdev=13454.22 lat (usec): min=3282, max=76322, avg=16511.94, stdev=13452.97 clat percentiles (usec): | 1.00th=[ 4047], 5.00th=[ 4948], 10.00th=[ 5997], 20.00th=[ 7767], | 30.00th=[ 8717], 40.00th=[ 9765], 50.00th=[11076], 60.00th=[13042], | 70.00th=[15926], 80.00th=[22938], 90.00th=[38011], 95.00th=[47449], | 99.00th=[67634], 99.50th=[70779], 99.90th=[76022], 99.95th=[76022], | 99.99th=[76022] bw ( KiB/s): min= 96, max= 670, per=25.41%, avg=387.29, stdev=159.96, samples=31 iops : min= 6, max= 41, avg=23.81, stdev=10.01, samples=31 write: IOPS=25, BW=404KiB/s (413kB/s)(6256KiB/15504msec) clat (usec): min=6515, max=84551, avg=23538.35, stdev=13261.60 lat (usec): min=6517, max=84552, avg=23540.63, stdev=13261.60 clat percentiles (usec): | 1.00th=[ 7504], 5.00th=[10421], 10.00th=[12125], 20.00th=[13566], | 30.00th=[15533], 40.00th=[17695], 50.00th=[19530], 60.00th=[21890], | 70.00th=[25297], 80.00th=[32900], 90.00th=[40633], 95.00th=[51119], | 99.00th=[72877], 99.50th=[83362], 99.90th=[84411], 99.95th=[84411], | 99.99th=[84411] bw ( KiB/s): min= 96, max= 768, per=25.11%, avg=402.74, stdev=182.72, samples=31 iops : min= 6, max= 48, avg=24.77, stdev=11.42, samples=31 lat (msec) : 4=0.39%, 10=23.05%, 20=39.97%, 50=32.03%, 100=4.56% cpu : usr=0.03%, sys=5.62%, ctx=887, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1525KiB/s (1561kB/s), 366KiB/s-404KiB/s (374kB/s-414kB/s), io=23.4MiB (24.5MB), run=15483-15709msec WRITE: bw=1604KiB/s (1643kB/s), 390KiB/s-417KiB/s (399kB/s-427kB/s), io=24.6MiB (25.8MB), run=15483-15709msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=24388: Thu May 1 19:04:28 2025 read: IOPS=4, BW=4604KiB/s (4714kB/s)(7168KiB/1557msec) clat (usec): min=841, max=457631, avg=130692.95, stdev=154851.11 lat (usec): min=842, max=457633, avg=130694.56, stdev=154851.14 clat percentiles (usec): | 1.00th=[ 840], 5.00th=[ 840], 10.00th=[ 840], 20.00th=[ 2966], | 30.00th=[ 89654], 40.00th=[ 89654], 50.00th=[102237], 60.00th=[105382], | 70.00th=[105382], 80.00th=[156238], 90.00th=[459277], 95.00th=[459277], | 99.00th=[459277], 99.50th=[459277], 99.90th=[459277], 99.95th=[459277], | 99.99th=[459277] bw ( KiB/s): min= 2048, max= 8159, per=46.38%, avg=4085.00, stdev=3528.19, samples=3 iops : min= 2, max= 7, avg= 3.67, stdev= 2.89, samples=3 write: IOPS=3, BW=3288KiB/s (3367kB/s)(5120KiB/1557msec) clat (msec): min=37, max=295, avg=114.29, stdev=104.80 lat (msec): min=37, max=295, avg=114.36, stdev=104.82 clat percentiles (msec): | 1.00th=[ 38], 5.00th=[ 38], 10.00th=[ 38], 20.00th=[ 38], | 30.00th=[ 57], 40.00th=[ 57], 50.00th=[ 71], 60.00th=[ 71], | 70.00th=[ 111], 80.00th=[ 111], 90.00th=[ 296], 95.00th=[ 296], | 99.00th=[ 296], 99.50th=[ 296], 99.90th=[ 296], 99.95th=[ 296], | 99.99th=[ 296] bw ( KiB/s): min= 4079, max= 4096, per=33.15%, avg=4087.50, stdev=12.02, samples=2 iops : min= 3, max= 4, avg= 3.50, stdev= 0.71, samples=2 lat (usec) : 1000=8.33% lat (msec) : 4=8.33%, 50=8.33%, 100=25.00%, 250=33.33%, 500=16.67% cpu : usr=0.00%, sys=15.17%, ctx=304, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24389: Thu May 1 19:04:28 2025 read: IOPS=1, BW=1170KiB/s (1198kB/s)(2048KiB/1751msec) clat (usec): min=2529, max=70723, avg=36626.34, stdev=48220.67 lat (usec): min=2530, max=70725, avg=36627.85, stdev=48221.13 clat percentiles (usec): | 1.00th=[ 2540], 5.00th=[ 2540], 10.00th=[ 2540], 20.00th=[ 2540], | 30.00th=[ 2540], 40.00th=[ 2540], 50.00th=[ 2540], 60.00th=[70779], | 70.00th=[70779], 80.00th=[70779], 90.00th=[70779], 95.00th=[70779], | 99.00th=[70779], 99.50th=[70779], 99.90th=[70779], 99.95th=[70779], | 99.99th=[70779] bw ( KiB/s): min= 2048, max= 2048, per=23.25%, avg=2048.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 write: IOPS=5, BW=5848KiB/s (5988kB/s)(10.0MiB/1751msec) clat (msec): min=30, max=612, avg=165.58, stdev=181.04 lat (msec): min=30, max=612, avg=165.66, stdev=181.04 clat percentiles (msec): | 1.00th=[ 31], 5.00th=[ 31], 10.00th=[ 31], 20.00th=[ 62], | 30.00th=[ 71], 40.00th=[ 71], 50.00th=[ 86], 60.00th=[ 106], | 70.00th=[ 144], 80.00th=[ 144], 90.00th=[ 355], 95.00th=[ 617], | 99.00th=[ 617], 99.50th=[ 617], 99.90th=[ 617], 99.95th=[ 617], | 99.99th=[ 617] bw ( KiB/s): min= 4079, max=10240, per=58.06%, avg=7159.50, stdev=4356.48, samples=2 iops : min= 3, max= 10, avg= 6.50, stdev= 4.95, samples=2 lat (msec) : 4=8.33%, 50=8.33%, 100=41.67%, 250=25.00%, 500=8.33% lat (msec) : 750=8.33% cpu : usr=0.00%, sys=14.40%, ctx=230, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24390: Thu May 1 19:04:28 2025 read: IOPS=2, BW=2678KiB/s (2743kB/s)(6144KiB/2294msec) clat (usec): min=530, max=288459, avg=108166.21, stdev=107672.07 lat (usec): min=531, max=288460, avg=108167.74, stdev=107672.00 clat percentiles (usec): | 1.00th=[ 529], 5.00th=[ 529], 10.00th=[ 529], 20.00th=[ 586], | 30.00th=[ 586], 40.00th=[ 92799], 50.00th=[ 92799], 60.00th=[113771], | 70.00th=[154141], 80.00th=[154141], 90.00th=[287310], 95.00th=[287310], | 99.00th=[287310], 99.50th=[287310], 99.90th=[287310], 99.95th=[287310], | 99.99th=[287310] bw ( KiB/s): min= 4071, max= 4087, per=46.31%, avg=4079.00, stdev=11.31, samples=2 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=2 write: IOPS=2, BW=2678KiB/s (2743kB/s)(6144KiB/2294msec) clat (msec): min=55, max=597, avg=265.98, stdev=202.77 lat (msec): min=55, max=597, avg=266.04, stdev=202.77 clat percentiles (msec): | 1.00th=[ 56], 5.00th=[ 56], 10.00th=[ 56], 20.00th=[ 133], | 30.00th=[ 133], 40.00th=[ 153], 50.00th=[ 153], 60.00th=[ 251], | 70.00th=[ 409], 80.00th=[ 409], 90.00th=[ 600], 95.00th=[ 600], | 99.00th=[ 600], 99.50th=[ 600], 99.90th=[ 600], 99.95th=[ 600], | 99.99th=[ 600] bw ( KiB/s): min= 2043, max= 4096, per=22.13%, avg=2729.00, stdev=1183.86, samples=3 iops : min= 1, max= 4, avg= 2.33, stdev= 1.53, samples=3 lat (usec) : 750=16.67% lat (msec) : 100=16.67%, 250=33.33%, 500=25.00%, 750=8.33% cpu : usr=0.00%, sys=9.29%, ctx=192, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24391: Thu May 1 19:04:28 2025 read: IOPS=2, BW=2202KiB/s (2255kB/s)(5120KiB/2325msec) clat (usec): min=453, max=104427, avg=41926.54, stdev=49441.84 lat (usec): min=454, max=104428, avg=41927.82, stdev=49441.28 clat percentiles (usec): | 1.00th=[ 453], 5.00th=[ 453], 10.00th=[ 453], 20.00th=[ 453], | 30.00th=[ 734], 40.00th=[ 734], 50.00th=[ 18482], 60.00th=[ 18482], | 70.00th=[ 85459], 80.00th=[ 85459], 90.00th=[104334], 95.00th=[104334], | 99.00th=[104334], 99.50th=[104334], 99.90th=[104334], 99.95th=[104334], | 99.99th=[104334] bw ( KiB/s): min= 8175, max= 8175, per=92.81%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 7, max= 7, avg= 7.00, stdev= 0.00, samples=1 write: IOPS=3, BW=3083KiB/s (3157kB/s)(7168KiB/2325msec) clat (msec): min=96, max=602, avg=280.06, stdev=208.11 lat (msec): min=96, max=602, avg=280.13, stdev=208.11 clat percentiles (msec): | 1.00th=[ 96], 5.00th=[ 96], 10.00th=[ 96], 20.00th=[ 123], | 30.00th=[ 165], 40.00th=[ 165], 50.00th=[ 171], 60.00th=[ 255], | 70.00th=[ 255], 80.00th=[ 550], 90.00th=[ 600], 95.00th=[ 600], | 99.00th=[ 600], 99.50th=[ 600], 99.90th=[ 600], 99.95th=[ 600], | 99.99th=[ 600] bw ( KiB/s): min= 2043, max= 4087, per=20.73%, avg=2556.50, stdev=1020.34, samples=4 iops : min= 1, max= 3, avg= 2.00, stdev= 0.82, samples=4 lat (usec) : 500=8.33%, 750=8.33% lat (msec) : 20=8.33%, 100=16.67%, 250=33.33%, 500=8.33%, 750=16.67% cpu : usr=0.00%, sys=9.98%, ctx=96, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=8809KiB/s (9020kB/s), 1170KiB/s-4604KiB/s (1198kB/s-4714kB/s), io=20.0MiB (20.0MB), run=1557-2325msec WRITE: bw=12.0MiB/s (12.6MB/s), 2678KiB/s-5848KiB/s (2743kB/s-5988kB/s), io=28.0MiB (29.4MB), run=1557-2325msec rand-rw: (groupid=0, jobs=1): err= 0: pid=24392: Thu May 1 19:04:28 2025 read: IOPS=2, BW=2786KiB/s (2853kB/s)(7168KiB/2573msec) clat (msec): min=39, max=523, avg=230.39, stdev=211.06 lat (msec): min=39, max=523, avg=230.40, stdev=211.06 clat percentiles (msec): | 1.00th=[ 40], 5.00th=[ 40], 10.00th=[ 40], 20.00th=[ 50], | 30.00th=[ 88], 40.00th=[ 88], 50.00th=[ 101], 60.00th=[ 330], | 70.00th=[ 330], 80.00th=[ 485], 90.00th=[ 523], 95.00th=[ 523], | 99.00th=[ 523], 99.50th=[ 523], 99.90th=[ 523], 99.95th=[ 523], | 99.99th=[ 523] bw ( KiB/s): min= 1875, max= 4096, per=38.34%, avg=3025.50, stdev=1237.83, samples=4 iops : min= 1, max= 4, avg= 2.50, stdev= 1.73, samples=4 write: IOPS=1, BW=1990KiB/s (2038kB/s)(5120KiB/2573msec) clat (msec): min=48, max=331, avg=155.52, stdev=120.51 lat (msec): min=48, max=331, avg=155.60, stdev=120.52 clat percentiles (msec): | 1.00th=[ 49], 5.00th=[ 49], 10.00th=[ 49], 20.00th=[ 49], | 30.00th=[ 50], 40.00th=[ 50], 50.00th=[ 130], 60.00th=[ 130], | 70.00th=[ 220], 80.00th=[ 220], 90.00th=[ 330], 95.00th=[ 330], | 99.00th=[ 330], 99.50th=[ 330], 99.90th=[ 330], 99.95th=[ 330], | 99.99th=[ 330] bw ( KiB/s): min= 2048, max= 7501, per=43.22%, avg=4774.50, stdev=3855.85, samples=2 iops : min= 2, max= 7, avg= 4.50, stdev= 3.54, samples=2 lat (msec) : 50=33.33%, 100=16.67%, 250=16.67%, 500=25.00%, 750=8.33% cpu : usr=0.00%, sys=5.37%, ctx=76, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24393: Thu May 1 19:04:28 2025 read: IOPS=0, BW=834KiB/s (854kB/s)(2048KiB/2457msec) clat (msec): min=79, max=235, avg=157.53, stdev=110.24 lat (msec): min=79, max=235, avg=157.54, stdev=110.24 clat percentiles (msec): | 1.00th=[ 80], 5.00th=[ 80], 10.00th=[ 80], 20.00th=[ 80], | 30.00th=[ 80], 40.00th=[ 80], 50.00th=[ 80], 60.00th=[ 236], | 70.00th=[ 236], 80.00th=[ 236], 90.00th=[ 236], 95.00th=[ 236], | 99.00th=[ 236], 99.50th=[ 236], 99.90th=[ 236], 99.95th=[ 236], | 99.99th=[ 236] bw ( KiB/s): min= 2031, max= 2043, per=25.81%, avg=2037.00, stdev= 8.49, samples=2 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=2 write: IOPS=4, BW=4168KiB/s (4268kB/s)(10.0MiB/2457msec) clat (msec): min=67, max=588, avg=201.41, stdev=149.36 lat (msec): min=68, max=588, avg=201.52, stdev=149.32 clat percentiles (msec): | 1.00th=[ 68], 5.00th=[ 68], 10.00th=[ 68], 20.00th=[ 69], | 30.00th=[ 125], 40.00th=[ 150], 50.00th=[ 171], 60.00th=[ 174], | 70.00th=[ 245], 80.00th=[ 245], 90.00th=[ 251], 95.00th=[ 592], | 99.00th=[ 592], 99.50th=[ 592], 99.90th=[ 592], 99.95th=[ 592], | 99.99th=[ 592] bw ( KiB/s): min= 4063, max= 4071, per=36.80%, avg=4065.67, stdev= 4.62, samples=3 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=3 lat (msec) : 100=25.00%, 250=58.33%, 500=8.33%, 750=8.33% cpu : usr=0.00%, sys=5.33%, ctx=53, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24394: Thu May 1 19:04:28 2025 read: IOPS=2, BW=2368KiB/s (2424kB/s)(6144KiB/2595msec) clat (msec): min=52, max=195, avg=123.91, stdev=59.69 lat (msec): min=52, max=195, avg=123.91, stdev=59.69 clat percentiles (msec): | 1.00th=[ 53], 5.00th=[ 53], 10.00th=[ 53], 20.00th=[ 72], | 30.00th=[ 72], 40.00th=[ 95], 50.00th=[ 95], 60.00th=[ 148], | 70.00th=[ 184], 80.00th=[ 184], 90.00th=[ 197], 95.00th=[ 197], | 99.00th=[ 197], 99.50th=[ 197], 99.90th=[ 197], 99.95th=[ 197], | 99.99th=[ 197] bw ( KiB/s): min= 2043, max= 4096, per=32.41%, avg=2557.50, stdev=1025.67, samples=4 iops : min= 1, max= 4, avg= 2.00, stdev= 1.41, samples=4 write: IOPS=2, BW=2368KiB/s (2424kB/s)(6144KiB/2595msec) clat (msec): min=40, max=572, avg=302.17, stdev=247.98 lat (msec): min=40, max=572, avg=302.27, stdev=248.00 clat percentiles (msec): | 1.00th=[ 41], 5.00th=[ 41], 10.00th=[ 41], 20.00th=[ 63], | 30.00th=[ 63], 40.00th=[ 134], 50.00th=[ 134], 60.00th=[ 502], | 70.00th=[ 502], 80.00th=[ 502], 90.00th=[ 575], 95.00th=[ 575], | 99.00th=[ 575], 99.50th=[ 575], 99.90th=[ 575], 99.95th=[ 575], | 99.99th=[ 575] bw ( KiB/s): min= 2043, max= 4096, per=23.16%, avg=2558.75, stdev=1024.84, samples=4 iops : min= 1, max= 4, avg= 2.25, stdev= 1.26, samples=4 lat (msec) : 50=8.33%, 100=33.33%, 250=33.33%, 750=25.00% cpu : usr=0.08%, sys=4.51%, ctx=59, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24395: Thu May 1 19:04:28 2025 read: IOPS=2, BW=2195KiB/s (2247kB/s)(5120KiB/2333msec) clat (msec): min=67, max=309, avg=149.56, stdev=95.63 lat (msec): min=67, max=309, avg=149.56, stdev=95.63 clat percentiles (msec): | 1.00th=[ 67], 5.00th=[ 67], 10.00th=[ 67], 20.00th=[ 67], | 30.00th=[ 88], 40.00th=[ 88], 50.00th=[ 131], 60.00th=[ 131], | 70.00th=[ 153], 80.00th=[ 153], 90.00th=[ 309], 95.00th=[ 309], | 99.00th=[ 309], 99.50th=[ 309], 99.90th=[ 309], 99.95th=[ 309], | 99.99th=[ 309] bw ( KiB/s): min= 4087, max= 4087, per=51.79%, avg=4087.00, stdev= 0.00, samples=2 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=2 write: IOPS=3, BW=3072KiB/s (3146kB/s)(7168KiB/2333msec) clat (msec): min=60, max=549, avg=213.22, stdev=166.57 lat (msec): min=60, max=549, avg=213.30, stdev=166.57 clat percentiles (msec): | 1.00th=[ 61], 5.00th=[ 61], 10.00th=[ 61], 20.00th=[ 62], | 30.00th=[ 138], 40.00th=[ 138], 50.00th=[ 203], 60.00th=[ 236], | 70.00th=[ 236], 80.00th=[ 245], 90.00th=[ 550], 95.00th=[ 550], | 99.00th=[ 550], 99.50th=[ 550], 99.90th=[ 550], 99.95th=[ 550], | 99.99th=[ 550] bw ( KiB/s): min= 2043, max= 6144, per=30.87%, avg=3410.00, stdev=2367.71, samples=3 iops : min= 1, max= 6, avg= 2.67, stdev= 2.89, samples=3 lat (msec) : 100=33.33%, 250=50.00%, 500=8.33%, 750=8.33% cpu : usr=0.04%, sys=6.13%, ctx=61, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=7892KiB/s (8082kB/s), 834KiB/s-2786KiB/s (854kB/s-2853kB/s), io=20.0MiB (20.0MB), run=2333-2595msec WRITE: bw=10.8MiB/s (11.3MB/s), 1990KiB/s-4168KiB/s (2038kB/s-4268kB/s), io=28.0MiB (29.4MB), run=2333-2595msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=24404: Thu May 1 19:04:32 2025 read: IOPS=2, BW=8203KiB/s (8400kB/s)(12.0MiB/1498msec) clat (msec): min=234, max=644, avg=464.89, stdev=209.44 lat (msec): min=234, max=644, avg=464.89, stdev=209.44 clat percentiles (msec): | 1.00th=[ 234], 5.00th=[ 234], 10.00th=[ 234], 20.00th=[ 234], | 30.00th=[ 234], 40.00th=[ 518], 50.00th=[ 518], 60.00th=[ 518], | 70.00th=[ 642], 80.00th=[ 642], 90.00th=[ 642], 95.00th=[ 642], | 99.00th=[ 642], 99.50th=[ 642], 99.90th=[ 642], 99.95th=[ 642], | 99.99th=[ 642] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 lat (msec) : 250=33.33%, 750=66.67% cpu : usr=0.00%, sys=12.02%, ctx=506, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24405: Thu May 1 19:04:32 2025 read: IOPS=0, BW=1820KiB/s (1864kB/s)(4096KiB/2250msec) clat (nsec): min=241088k, max=241088k, avg=241087875.00, stdev= 0.00 lat (nsec): min=241091k, max=241091k, avg=241090503.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 241], 5.00th=[ 241], 10.00th=[ 241], 20.00th=[ 241], | 30.00th=[ 241], 40.00th=[ 241], 50.00th=[ 241], 60.00th=[ 241], | 70.00th=[ 241], 80.00th=[ 241], 90.00th=[ 241], 95.00th=[ 241], | 99.00th=[ 241], 99.50th=[ 241], 99.90th=[ 241], 99.95th=[ 241], | 99.99th=[ 241] bw ( KiB/s): min= 8143, max= 8143, per=100.00%, avg=8143.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=0, BW=3641KiB/s (3728kB/s)(8192KiB/2250msec) clat (msec): min=684, max=1254, avg=969.29, stdev=402.77 lat (msec): min=684, max=1254, avg=969.57, stdev=402.77 clat percentiles (msec): | 1.00th=[ 684], 5.00th=[ 684], 10.00th=[ 684], 20.00th=[ 684], | 30.00th=[ 684], 40.00th=[ 684], 50.00th=[ 684], 60.00th=[ 1250], | 70.00th=[ 1250], 80.00th=[ 1250], 90.00th=[ 1250], 95.00th=[ 1250], | 99.00th=[ 1250], 99.50th=[ 1250], 99.90th=[ 1250], 99.95th=[ 1250], | 99.99th=[ 1250] bw ( KiB/s): min= 6681, max= 6681, per=45.88%, avg=6681.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 750=33.33% cpu : usr=0.00%, sys=9.43%, ctx=123, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24406: Thu May 1 19:04:32 2025 write: IOPS=1, BW=6689KiB/s (6850kB/s)(12.0MiB/1837msec) clat (msec): min=434, max=884, avg=593.77, stdev=251.94 lat (msec): min=435, max=884, avg=594.09, stdev=251.90 clat percentiles (msec): | 1.00th=[ 435], 5.00th=[ 435], 10.00th=[ 435], 20.00th=[ 435], | 30.00th=[ 435], 40.00th=[ 464], 50.00th=[ 464], 60.00th=[ 464], | 70.00th=[ 885], 80.00th=[ 885], 90.00th=[ 885], 95.00th=[ 885], | 99.00th=[ 885], 99.50th=[ 885], 99.90th=[ 885], 99.95th=[ 885], | 99.99th=[ 885] bw ( KiB/s): min= 7585, max= 8192, per=54.17%, avg=7888.50, stdev=429.21, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 500=66.67%, 1000=33.33% cpu : usr=0.00%, sys=17.21%, ctx=54, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24407: Thu May 1 19:04:32 2025 write: IOPS=1, BW=7953KiB/s (8144kB/s)(12.0MiB/1545msec) clat (msec): min=268, max=826, avg=493.03, stdev=294.89 lat (msec): min=268, max=827, avg=493.33, stdev=294.90 clat percentiles (msec): | 1.00th=[ 268], 5.00th=[ 268], 10.00th=[ 268], 20.00th=[ 268], | 30.00th=[ 268], 40.00th=[ 384], 50.00th=[ 384], 60.00th=[ 384], | 70.00th=[ 827], 80.00th=[ 827], 90.00th=[ 827], 95.00th=[ 827], | 99.00th=[ 827], 99.50th=[ 827], 99.90th=[ 827], 99.95th=[ 827], | 99.99th=[ 827] bw ( KiB/s): min= 7922, max= 8078, per=54.93%, avg=8000.00, stdev=110.31, samples=2 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=2 lat (msec) : 500=66.67%, 1000=33.33% cpu : usr=0.00%, sys=16.06%, ctx=112, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=7282KiB/s (7457kB/s), 1820KiB/s-8203KiB/s (1864kB/s-8400kB/s), io=16.0MiB (16.8MB), run=1498-2250msec WRITE: bw=14.2MiB/s (14.9MB/s), 3641KiB/s-7953KiB/s (3728kB/s-8144kB/s), io=32.0MiB (33.6MB), run=1545-2250msec rand-rw: (groupid=0, jobs=1): err= 0: pid=24408: Thu May 1 19:04:32 2025 read: IOPS=1, BW=5846KiB/s (5986kB/s)(12.0MiB/2102msec) clat (msec): min=461, max=907, avg=610.99, stdev=256.73 lat (msec): min=461, max=907, avg=611.00, stdev=256.73 clat percentiles (msec): | 1.00th=[ 464], 5.00th=[ 464], 10.00th=[ 464], 20.00th=[ 464], | 30.00th=[ 464], 40.00th=[ 464], 50.00th=[ 464], 60.00th=[ 464], | 70.00th=[ 911], 80.00th=[ 911], 90.00th=[ 911], 95.00th=[ 911], | 99.00th=[ 911], 99.50th=[ 911], 99.90th=[ 911], 99.95th=[ 911], | 99.99th=[ 911] bw ( KiB/s): min= 8094, max= 8159, per=100.00%, avg=8126.50, stdev=45.96, samples=2 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=2 lat (msec) : 500=66.67%, 1000=33.33% cpu : usr=0.05%, sys=3.62%, ctx=35, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24409: Thu May 1 19:04:32 2025 read: IOPS=0, BW=1605KiB/s (1644kB/s)(4096KiB/2552msec) clat (nsec): min=856598k, max=856598k, avg=856598308.00, stdev= 0.00 lat (nsec): min=856600k, max=856600k, avg=856600493.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 860], 5.00th=[ 860], 10.00th=[ 860], 20.00th=[ 860], | 30.00th=[ 860], 40.00th=[ 860], 50.00th=[ 860], 60.00th=[ 860], | 70.00th=[ 860], 80.00th=[ 860], 90.00th=[ 860], 95.00th=[ 860], | 99.00th=[ 860], 99.50th=[ 860], 99.90th=[ 860], 99.95th=[ 860], | 99.99th=[ 860] bw ( KiB/s): min= 8110, max= 8110, per=100.00%, avg=8110.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=0, BW=3210KiB/s (3287kB/s)(8192KiB/2552msec) clat (msec): min=322, max=1227, avg=775.18, stdev=639.65 lat (msec): min=323, max=1227, avg=775.49, stdev=639.68 clat percentiles (msec): | 1.00th=[ 321], 5.00th=[ 321], 10.00th=[ 321], 20.00th=[ 321], | 30.00th=[ 321], 40.00th=[ 321], 50.00th=[ 321], 60.00th=[ 1234], | 70.00th=[ 1234], 80.00th=[ 1234], 90.00th=[ 1234], 95.00th=[ 1234], | 99.00th=[ 1234], 99.50th=[ 1234], 99.90th=[ 1234], 99.95th=[ 1234], | 99.99th=[ 1234] lat (msec) : 500=33.33%, 1000=33.33% cpu : usr=0.04%, sys=6.78%, ctx=69, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24410: Thu May 1 19:04:32 2025 write: IOPS=1, BW=5660KiB/s (5796kB/s)(12.0MiB/2171msec) clat (msec): min=435, max=920, avg=610.69, stdev=268.95 lat (msec): min=435, max=920, avg=611.02, stdev=268.94 clat percentiles (msec): | 1.00th=[ 435], 5.00th=[ 435], 10.00th=[ 435], 20.00th=[ 435], | 30.00th=[ 435], 40.00th=[ 477], 50.00th=[ 477], 60.00th=[ 477], | 70.00th=[ 919], 80.00th=[ 919], 90.00th=[ 919], 95.00th=[ 919], | 99.00th=[ 919], 99.50th=[ 919], 99.90th=[ 919], 99.95th=[ 919], | 99.99th=[ 919] bw ( KiB/s): min= 7953, max= 8047, per=62.31%, avg=8000.00, stdev=66.47, samples=2 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=2 lat (msec) : 500=66.67%, 1000=33.33% cpu : usr=0.05%, sys=4.15%, ctx=46, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=24411: Thu May 1 19:04:32 2025 write: IOPS=1, BW=6065KiB/s (6211kB/s)(12.0MiB/2026msec) clat (msec): min=171, max=973, avg=641.60, stdev=418.17 lat (msec): min=172, max=973, avg=641.91, stdev=418.18 clat percentiles (msec): | 1.00th=[ 174], 5.00th=[ 174], 10.00th=[ 174], 20.00th=[ 174], | 30.00th=[ 174], 40.00th=[ 776], 50.00th=[ 776], 60.00th=[ 776], | 70.00th=[ 978], 80.00th=[ 978], 90.00th=[ 978], 95.00th=[ 978], | 99.00th=[ 978], 99.50th=[ 978], 99.90th=[ 978], 99.95th=[ 978], | 99.99th=[ 978] bw ( KiB/s): min= 8175, max= 8175, per=63.67%, avg=8175.00, stdev= 0.00, samples=2 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=2 lat (msec) : 250=33.33%, 1000=66.67% cpu : usr=0.05%, sys=7.95%, ctx=74, majf=0, minf=28 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=6420KiB/s (6574kB/s), 1605KiB/s-5846KiB/s (1644kB/s-5986kB/s), io=16.0MiB (16.8MB), run=2102-2552msec WRITE: bw=12.5MiB/s (13.1MB/s), 3210KiB/s-6065KiB/s (3287kB/s-6211kB/s), io=32.0MiB (33.6MB), run=2026-2552msec