== sanity test 398b: DIO and buffer IO race ============== 14:10:34 (1744135834) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.0361 s, 48.6 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=12221: Tue Apr 8 14:10:50 2025 read: IOPS=109, BW=439KiB/s (450kB/s)(5992KiB/13639msec) clat (usec): min=58, max=147045, avg=4118.74, stdev=4950.62 lat (usec): min=58, max=147047, avg=4120.05, stdev=4950.72 clat percentiles (usec): | 1.00th=[ 67], 5.00th=[ 77], 10.00th=[ 92], 20.00th=[ 1909], | 30.00th=[ 2507], 40.00th=[ 2900], 50.00th=[ 3326], 60.00th=[ 3884], | 70.00th=[ 4948], 80.00th=[ 6521], 90.00th=[ 8455], 95.00th=[ 9896], | 99.00th=[ 13173], 99.50th=[ 15795], 99.90th=[ 39060], 99.95th=[147850], | 99.99th=[147850] bw ( KiB/s): min= 224, max= 640, per=24.99%, avg=438.63, stdev=84.87, samples=27 iops : min= 56, max= 160, avg=109.56, stdev=21.20, samples=27 write: IOPS=115, BW=462KiB/s (473kB/s)(6296KiB/13639msec) clat (usec): min=502, max=54491, avg=4685.32, stdev=4653.90 lat (usec): min=502, max=54492, avg=4686.41, stdev=4654.02 clat percentiles (usec): | 1.00th=[ 523], 5.00th=[ 570], 10.00th=[ 619], 20.00th=[ 766], | 30.00th=[ 1123], 40.00th=[ 2343], 50.00th=[ 3752], 60.00th=[ 4686], | 70.00th=[ 5800], 80.00th=[ 7701], 90.00th=[10814], 95.00th=[13173], | 99.00th=[19006], 99.50th=[22152], 99.90th=[52691], 99.95th=[54264], | 99.99th=[54264] bw ( KiB/s): min= 256, max= 616, per=24.99%, avg=459.33, stdev=82.01, samples=27 iops : min= 64, max= 154, avg=114.74, stdev=20.52, samples=27 lat (usec) : 100=5.66%, 250=3.52%, 500=0.13%, 750=9.86%, 1000=4.17% lat (msec) : 2=6.28%, 4=27.51%, 10=34.11%, 20=8.24%, 50=0.42% lat (msec) : 100=0.07%, 250=0.03% cpu : usr=0.17%, sys=23.30%, ctx=6031, majf=0, minf=37 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=12222: Tue Apr 8 14:10:50 2025 read: IOPS=109, BW=439KiB/s (450kB/s)(5940KiB/13531msec) clat (usec): min=53, max=37008, avg=3954.47, stdev=3325.66 lat (usec): min=53, max=37009, avg=3955.52, stdev=3325.59 clat percentiles (usec): | 1.00th=[ 67], 5.00th=[ 75], 10.00th=[ 92], 20.00th=[ 1975], | 30.00th=[ 2474], 40.00th=[ 2835], 50.00th=[ 3228], 60.00th=[ 3818], | 70.00th=[ 4752], 80.00th=[ 6259], 90.00th=[ 8094], 95.00th=[ 9241], | 99.00th=[15270], 99.50th=[21365], 99.90th=[26608], 99.95th=[36963], | 99.99th=[36963] bw ( KiB/s): min= 272, max= 600, per=24.98%, avg=438.33, stdev=80.62, samples=27 iops : min= 68, max= 150, avg=109.48, stdev=20.19, samples=27 write: IOPS=117, BW=469KiB/s (480kB/s)(6348KiB/13531msec) clat (usec): min=449, max=159497, avg=4774.33, stdev=6052.08 lat (usec): min=450, max=159498, avg=4775.70, stdev=6051.94 clat percentiles (usec): | 1.00th=[ 519], 5.00th=[ 578], 10.00th=[ 619], 20.00th=[ 758], | 30.00th=[ 1020], 40.00th=[ 2073], 50.00th=[ 3785], 60.00th=[ 4621], | 70.00th=[ 6063], 80.00th=[ 7767], 90.00th=[ 10421], 95.00th=[ 13304], | 99.00th=[ 19530], 99.50th=[ 23987], 99.90th=[ 46924], 99.95th=[160433], | 99.99th=[160433] bw ( KiB/s): min= 326, max= 638, per=25.42%, avg=467.30, stdev=89.84, samples=27 iops : min= 81, max= 159, avg=116.74, stdev=22.45, samples=27 lat (usec) : 100=5.37%, 250=3.26%, 500=0.42%, 750=9.83%, 1000=5.34% lat (msec) : 2=5.99%, 4=27.05%, 10=34.77%, 20=7.19%, 50=0.75% lat (msec) : 250=0.03% cpu : usr=0.22%, sys=23.18%, ctx=6002, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=12223: Tue Apr 8 14:10:50 2025 read: IOPS=110, BW=441KiB/s (452kB/s)(6028KiB/13664msec) clat (usec): min=52, max=20638, avg=3908.86, stdev=3055.19 lat (usec): min=52, max=20639, avg=3909.85, stdev=3055.06 clat percentiles (usec): | 1.00th=[ 61], 5.00th=[ 74], 10.00th=[ 88], 20.00th=[ 1876], | 30.00th=[ 2442], 40.00th=[ 2835], 50.00th=[ 3294], 60.00th=[ 3884], | 70.00th=[ 4948], 80.00th=[ 6325], 90.00th=[ 8029], 95.00th=[ 9372], | 99.00th=[12780], 99.50th=[16188], 99.90th=[18482], 99.95th=[20579], | 99.99th=[20579] bw ( KiB/s): min= 192, max= 592, per=24.92%, avg=437.41, stdev=89.68, samples=27 iops : min= 48, max= 148, avg=109.26, stdev=22.41, samples=27 write: IOPS=114, BW=458KiB/s (469kB/s)(6260KiB/13664msec) clat (usec): min=412, max=235563, avg=4918.65, stdev=7301.67 lat (usec): min=413, max=235564, avg=4919.89, stdev=7301.66 clat percentiles (usec): | 1.00th=[ 510], 5.00th=[ 545], 10.00th=[ 603], 20.00th=[ 758], | 30.00th=[ 1156], 40.00th=[ 2606], 50.00th=[ 3884], 60.00th=[ 4948], | 70.00th=[ 6128], 80.00th=[ 7898], 90.00th=[ 10814], 95.00th=[ 13698], | 99.00th=[ 18220], 99.50th=[ 23725], 99.90th=[ 35914], 99.95th=[235930], | 99.99th=[235930] bw ( KiB/s): min= 272, max= 608, per=24.64%, avg=452.85, stdev=78.24, samples=27 iops : min= 68, max= 152, avg=113.11, stdev=19.52, samples=27 lat (usec) : 100=6.22%, 250=2.99%, 500=0.42%, 750=9.93%, 1000=3.94% lat (msec) : 2=5.21%, 4=27.47%, 10=35.71%, 20=7.68%, 50=0.39% lat (msec) : 250=0.03% cpu : usr=0.17%, sys=23.05%, ctx=6105, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=12224: Tue Apr 8 14:10:50 2025 read: IOPS=110, BW=443KiB/s (453kB/s)(6052KiB/13675msec) clat (usec): min=55, max=209202, avg=4011.01, stdev=6097.25 lat (usec): min=55, max=209203, avg=4011.74, stdev=6097.25 clat percentiles (usec): | 1.00th=[ 65], 5.00th=[ 76], 10.00th=[ 91], 20.00th=[ 832], | 30.00th=[ 2442], 40.00th=[ 2835], 50.00th=[ 3261], 60.00th=[ 3818], | 70.00th=[ 4817], 80.00th=[ 6194], 90.00th=[ 8094], 95.00th=[ 9503], | 99.00th=[ 12911], 99.50th=[ 15533], 99.90th=[ 21365], 99.95th=[208667], | 99.99th=[208667] bw ( KiB/s): min= 280, max= 592, per=24.97%, avg=438.15, stdev=76.99, samples=27 iops : min= 70, max= 148, avg=109.48, stdev=19.24, samples=27 write: IOPS=114, BW=456KiB/s (467kB/s)(6236KiB/13675msec) clat (usec): min=429, max=52778, avg=4830.37, stdev=4575.30 lat (usec): min=430, max=52778, avg=4831.45, stdev=4575.18 clat percentiles (usec): | 1.00th=[ 523], 5.00th=[ 570], 10.00th=[ 635], 20.00th=[ 791], | 30.00th=[ 1123], 40.00th=[ 2606], 50.00th=[ 3916], 60.00th=[ 5014], | 70.00th=[ 6259], 80.00th=[ 8029], 90.00th=[10552], 95.00th=[13566], | 99.00th=[19530], 99.50th=[22152], 99.90th=[33817], 99.95th=[52691], | 99.99th=[52691] bw ( KiB/s): min= 312, max= 576, per=24.58%, avg=451.78, stdev=73.02, samples=27 iops : min= 78, max= 144, avg=112.89, stdev=18.30, samples=27 lat (usec) : 100=5.99%, 250=3.68%, 500=0.39%, 750=8.66%, 1000=5.47% lat (msec) : 2=5.21%, 4=26.63%, 10=35.87%, 20=7.62%, 50=0.42% lat (msec) : 100=0.03%, 250=0.03% cpu : usr=0.16%, sys=23.12%, ctx=5985, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1756KiB/s (1798kB/s), 439KiB/s-443KiB/s (450kB/s-453kB/s), io=23.4MiB (24.6MB), run=13531-13675msec WRITE: bw=1838KiB/s (1883kB/s), 456KiB/s-469KiB/s (467kB/s-480kB/s), io=24.6MiB (25.7MB), run=13531-13675msec rand-rw: (groupid=0, jobs=1): err= 0: pid=12217: Tue Apr 8 14:11:08 2025 read: IOPS=46, BW=186KiB/s (190kB/s)(5992KiB/32228msec) clat (usec): min=1578, max=162148, avg=3866.89, stdev=5434.82 lat (usec): min=1579, max=162149, avg=3867.61, stdev=5434.84 clat percentiles (usec): | 1.00th=[ 1713], 5.00th=[ 1811], 10.00th=[ 1876], 20.00th=[ 1991], | 30.00th=[ 2089], 40.00th=[ 2212], 50.00th=[ 2343], 60.00th=[ 2573], | 70.00th=[ 3195], 80.00th=[ 4948], 90.00th=[ 7832], 95.00th=[ 10552], | 99.00th=[ 16188], 99.50th=[ 21627], 99.90th=[ 51643], 99.95th=[162530], | 99.99th=[162530] bw ( KiB/s): min= 40, max= 328, per=25.07%, avg=185.48, stdev=78.60, samples=64 iops : min= 10, max= 82, avg=46.27, stdev=19.69, samples=64 write: IOPS=48, BW=195KiB/s (200kB/s)(6296KiB/32228msec) clat (usec): min=7862, max=59023, avg=16736.07, stdev=6986.45 lat (usec): min=7863, max=59024, avg=16736.95, stdev=6986.49 clat percentiles (usec): | 1.00th=[ 8586], 5.00th=[ 9372], 10.00th=[10159], 20.00th=[11207], | 30.00th=[12387], 40.00th=[13435], 50.00th=[14746], 60.00th=[16188], | 70.00th=[17957], 80.00th=[21103], 90.00th=[27395], 95.00th=[31327], | 99.00th=[39060], 99.50th=[42206], 99.90th=[56886], 99.95th=[58983], | 99.99th=[58983] bw ( KiB/s): min= 64, max= 288, per=25.13%, avg=194.73, stdev=68.83, samples=64 iops : min= 16, max= 72, avg=48.58, stdev=17.25, samples=64 lat (msec) : 2=10.55%, 4=26.04%, 10=14.32%, 20=37.53%, 50=11.39% lat (msec) : 100=0.13%, 250=0.03% cpu : usr=0.10%, sys=6.32%, ctx=3627, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=12218: Tue Apr 8 14:11:08 2025 read: IOPS=45, BW=184KiB/s (188kB/s)(5940KiB/32356msec) clat (usec): min=1598, max=210069, avg=3785.89, stdev=6323.43 lat (usec): min=1599, max=210072, avg=3786.67, stdev=6323.49 clat percentiles (usec): | 1.00th=[ 1729], 5.00th=[ 1844], 10.00th=[ 1909], 20.00th=[ 1991], | 30.00th=[ 2089], 40.00th=[ 2180], 50.00th=[ 2311], 60.00th=[ 2540], | 70.00th=[ 3097], 80.00th=[ 4490], 90.00th=[ 7570], 95.00th=[ 9765], | 99.00th=[ 16188], 99.50th=[ 22938], 99.90th=[ 44827], 99.95th=[210764], | 99.99th=[210764] bw ( KiB/s): min= 40, max= 360, per=24.78%, avg=183.37, stdev=83.34, samples=64 iops : min= 10, max= 90, avg=45.75, stdev=20.89, samples=64 write: IOPS=49, BW=196KiB/s (201kB/s)(6348KiB/32356msec) clat (usec): min=7829, max=86900, avg=16787.14, stdev=7167.03 lat (usec): min=7830, max=86900, avg=16788.06, stdev=7167.07 clat percentiles (usec): | 1.00th=[ 8455], 5.00th=[ 9372], 10.00th=[10028], 20.00th=[11076], | 30.00th=[12256], 40.00th=[13566], 50.00th=[14877], 60.00th=[16319], | 70.00th=[17957], 80.00th=[21103], 90.00th=[27132], 95.00th=[31065], | 99.00th=[39584], 99.50th=[42730], 99.90th=[60031], 99.95th=[86508], | 99.99th=[86508] bw ( KiB/s): min= 72, max= 280, per=25.16%, avg=195.00, stdev=66.85, samples=64 iops : min= 18, max= 70, avg=48.66, stdev=16.76, samples=64 lat (msec) : 2=9.86%, 4=27.44%, 10=13.70%, 20=37.08%, 50=11.82% lat (msec) : 100=0.07%, 250=0.03% cpu : usr=0.09%, sys=6.31%, ctx=3574, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=12219: Tue Apr 8 14:11:08 2025 read: IOPS=46, BW=188KiB/s (192kB/s)(6028KiB/32136msec) clat (usec): min=1635, max=40921, avg=3894.53, stdev=3544.91 lat (usec): min=1636, max=40921, avg=3895.31, stdev=3544.92 clat percentiles (usec): | 1.00th=[ 1729], 5.00th=[ 1827], 10.00th=[ 1893], 20.00th=[ 2008], | 30.00th=[ 2114], 40.00th=[ 2180], 50.00th=[ 2343], 60.00th=[ 2671], | 70.00th=[ 3523], 80.00th=[ 5473], 90.00th=[ 8291], 95.00th=[10421], | 99.00th=[15795], 99.50th=[23200], 99.90th=[40633], 99.95th=[41157], | 99.99th=[41157] bw ( KiB/s): min= 56, max= 328, per=25.34%, avg=187.48, stdev=70.83, samples=64 iops : min= 14, max= 82, avg=46.77, stdev=17.77, samples=64 write: IOPS=48, BW=195KiB/s (199kB/s)(6260KiB/32136msec) clat (msec): min=7, max=234, avg=16.72, stdev= 8.79 lat (msec): min=7, max=234, avg=16.73, stdev= 8.79 clat percentiles (msec): | 1.00th=[ 9], 5.00th=[ 10], 10.00th=[ 11], 20.00th=[ 12], | 30.00th=[ 13], 40.00th=[ 14], 50.00th=[ 15], 60.00th=[ 17], | 70.00th=[ 18], 80.00th=[ 21], 90.00th=[ 27], 95.00th=[ 31], | 99.00th=[ 39], 99.50th=[ 41], 99.90th=[ 87], 99.95th=[ 234], | 99.99th=[ 234] bw ( KiB/s): min= 64, max= 288, per=25.06%, avg=194.23, stdev=68.64, samples=64 iops : min= 16, max= 72, avg=48.45, stdev=17.20, samples=64 lat (msec) : 2=9.93%, 4=25.91%, 10=15.79%, 20=36.72%, 50=11.59% lat (msec) : 100=0.03%, 250=0.03% cpu : usr=0.13%, sys=6.31%, ctx=3623, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=12220: Tue Apr 8 14:11:08 2025 read: IOPS=46, BW=187KiB/s (191kB/s)(6052KiB/32408msec) clat (usec): min=1621, max=37582, avg=3738.54, stdev=3256.55 lat (usec): min=1622, max=37583, avg=3739.99, stdev=3256.32 clat percentiles (usec): | 1.00th=[ 1729], 5.00th=[ 1844], 10.00th=[ 1893], 20.00th=[ 1991], | 30.00th=[ 2073], 40.00th=[ 2212], 50.00th=[ 2376], 60.00th=[ 2638], | 70.00th=[ 3294], 80.00th=[ 4817], 90.00th=[ 7767], 95.00th=[10159], | 99.00th=[15664], 99.50th=[19792], 99.90th=[34866], 99.95th=[37487], | 99.99th=[37487] bw ( KiB/s): min= 56, max= 367, per=25.07%, avg=185.48, stdev=75.91, samples=64 iops : min= 14, max= 91, avg=46.27, stdev=18.99, samples=64 write: IOPS=48, BW=192KiB/s (197kB/s)(6236KiB/32408msec) clat (msec): min=8, max=223, avg=17.10, stdev= 8.76 lat (msec): min=8, max=223, avg=17.10, stdev= 8.76 clat percentiles (msec): | 1.00th=[ 9], 5.00th=[ 10], 10.00th=[ 11], 20.00th=[ 12], | 30.00th=[ 13], 40.00th=[ 14], 50.00th=[ 16], 60.00th=[ 17], | 70.00th=[ 19], 80.00th=[ 22], 90.00th=[ 28], 95.00th=[ 32], | 99.00th=[ 39], 99.50th=[ 45], 99.90th=[ 61], 99.95th=[ 224], | 99.99th=[ 224] bw ( KiB/s): min= 72, max= 280, per=24.68%, avg=191.23, stdev=65.92, samples=64 iops : min= 18, max= 70, avg=47.70, stdev=16.53, samples=64 lat (msec) : 2=10.19%, 4=27.02%, 10=13.80%, 20=36.85%, 50=11.98% lat (msec) : 100=0.13%, 250=0.03% cpu : usr=0.12%, sys=6.38%, ctx=3649, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=741KiB/s (759kB/s), 184KiB/s-188KiB/s (188kB/s-192kB/s), io=23.4MiB (24.6MB), run=32136-32408msec WRITE: bw=776KiB/s (794kB/s), 192KiB/s-196KiB/s (197kB/s-201kB/s), io=24.6MiB (25.7MB), run=32136-32408msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=12262: Tue Apr 8 14:11:12 2025 read: IOPS=146, BW=2345KiB/s (2401kB/s)(5920KiB/2525msec) clat (usec): min=61, max=22794, avg=3579.11, stdev=2522.70 lat (usec): min=61, max=22794, avg=3579.77, stdev=2522.79 clat percentiles (usec): | 1.00th=[ 68], 5.00th=[ 74], 10.00th=[ 86], 20.00th=[ 2212], | 30.00th=[ 2573], 40.00th=[ 2835], 50.00th=[ 3261], 60.00th=[ 3785], | 70.00th=[ 4424], 80.00th=[ 5145], 90.00th=[ 6194], 95.00th=[ 7373], | 99.00th=[12256], 99.50th=[16909], 99.90th=[22676], 99.95th=[22676], | 99.99th=[22676] bw ( KiB/s): min= 2080, max= 2464, per=26.89%, avg=2335.00, stdev=150.11, samples=5 iops : min= 130, max= 154, avg=145.80, stdev= 9.39, samples=5 write: IOPS=157, BW=2522KiB/s (2583kB/s)(6368KiB/2525msec) clat (usec): min=637, max=26640, avg=2961.13, stdev=2851.79 lat (usec): min=638, max=26641, avg=2962.06, stdev=2851.80 clat percentiles (usec): | 1.00th=[ 709], 5.00th=[ 775], 10.00th=[ 832], 20.00th=[ 947], | 30.00th=[ 1074], 40.00th=[ 1369], 50.00th=[ 2040], 60.00th=[ 2802], | 70.00th=[ 3654], 80.00th=[ 4752], 90.00th=[ 6063], 95.00th=[ 7308], | 99.00th=[14877], 99.50th=[22938], 99.90th=[26608], 99.95th=[26608], | 99.99th=[26608] bw ( KiB/s): min= 2112, max= 3097, per=27.58%, avg=2520.20, stdev=398.04, samples=5 iops : min= 132, max= 193, avg=157.40, stdev=24.67, samples=5 lat (usec) : 100=5.47%, 250=1.82%, 500=0.13%, 750=1.56%, 1000=10.94% lat (msec) : 2=13.54%, 4=35.81%, 10=28.65%, 20=1.69%, 50=0.39% cpu : usr=0.20%, sys=30.35%, ctx=1723, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=12264: Tue Apr 8 14:11:12 2025 read: IOPS=130, BW=2083KiB/s (2133kB/s)(5744KiB/2758msec) clat (usec): min=61, max=19714, avg=3183.98, stdev=2636.52 lat (usec): min=61, max=19715, avg=3185.64, stdev=2636.87 clat percentiles (usec): | 1.00th=[ 70], 5.00th=[ 73], 10.00th=[ 83], 20.00th=[ 117], | 30.00th=[ 2180], 40.00th=[ 2573], 50.00th=[ 3064], 60.00th=[ 3523], | 70.00th=[ 4228], 80.00th=[ 5014], 90.00th=[ 6259], 95.00th=[ 7570], | 99.00th=[11338], 99.50th=[15270], 99.90th=[19792], 99.95th=[19792], | 99.99th=[19792] bw ( KiB/s): min= 960, max= 2528, per=22.76%, avg=1976.60, stdev=623.00, samples=5 iops : min= 60, max= 158, avg=123.40, stdev=38.92, samples=5 write: IOPS=148, BW=2373KiB/s (2430kB/s)(6544KiB/2758msec) clat (usec): min=677, max=37053, avg=3753.40, stdev=4812.56 lat (usec): min=677, max=37054, avg=3754.32, stdev=4812.57 clat percentiles (usec): | 1.00th=[ 701], 5.00th=[ 758], 10.00th=[ 807], 20.00th=[ 906], | 30.00th=[ 1057], 40.00th=[ 1270], 50.00th=[ 2040], 60.00th=[ 3032], | 70.00th=[ 3851], 80.00th=[ 5407], 90.00th=[ 8717], 95.00th=[11994], | 99.00th=[25560], 99.50th=[27395], 99.90th=[36963], 99.95th=[36963], | 99.99th=[36963] bw ( KiB/s): min= 1152, max= 2874, per=24.43%, avg=2232.40, stdev=696.92, samples=5 iops : min= 72, max= 179, avg=139.40, stdev=43.41, samples=5 lat (usec) : 100=7.55%, 250=4.56%, 500=0.13%, 750=2.60%, 1000=12.37% lat (msec) : 2=12.24%, 4=29.69%, 10=26.56%, 20=3.12%, 50=1.17% cpu : usr=0.40%, sys=26.37%, ctx=1636, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=12265: Tue Apr 8 14:11:12 2025 read: IOPS=152, BW=2432KiB/s (2491kB/s)(6256KiB/2572msec) clat (usec): min=60, max=25140, avg=3258.65, stdev=2940.73 lat (usec): min=60, max=25141, avg=3264.06, stdev=2938.60 clat percentiles (usec): | 1.00th=[ 64], 5.00th=[ 73], 10.00th=[ 82], 20.00th=[ 119], | 30.00th=[ 2311], 40.00th=[ 2737], 50.00th=[ 3064], 60.00th=[ 3490], | 70.00th=[ 3982], 80.00th=[ 4817], 90.00th=[ 5866], 95.00th=[ 7635], | 99.00th=[16319], 99.50th=[19792], 99.90th=[25035], 99.95th=[25035], | 99.99th=[25035] bw ( KiB/s): min= 1664, max= 2944, per=28.07%, avg=2437.40, stdev=483.42, samples=5 iops : min= 104, max= 184, avg=152.20, stdev=30.22, samples=5 write: IOPS=146, BW=2345KiB/s (2402kB/s)(6032KiB/2572msec) clat (usec): min=612, max=27762, avg=3202.99, stdev=3532.16 lat (usec): min=613, max=27764, avg=3204.44, stdev=3532.18 clat percentiles (usec): | 1.00th=[ 676], 5.00th=[ 783], 10.00th=[ 840], 20.00th=[ 914], | 30.00th=[ 1057], 40.00th=[ 1270], 50.00th=[ 1811], 60.00th=[ 2769], | 70.00th=[ 3556], 80.00th=[ 4948], 90.00th=[ 6980], 95.00th=[ 9241], | 99.00th=[19792], 99.50th=[22938], 99.90th=[27657], 99.95th=[27657], | 99.99th=[27657] bw ( KiB/s): min= 1984, max= 2682, per=25.27%, avg=2309.20, stdev=271.72, samples=5 iops : min= 124, max= 167, avg=144.20, stdev=16.77, samples=5 lat (usec) : 100=8.20%, 250=4.30%, 500=0.52%, 750=1.17%, 1000=11.72% lat (msec) : 2=12.89%, 4=33.46%, 10=24.61%, 20=2.60%, 50=0.52% cpu : usr=0.35%, sys=27.46%, ctx=1666, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=12266: Tue Apr 8 14:11:12 2025 read: IOPS=146, BW=2349KiB/s (2405kB/s)(6032KiB/2568msec) clat (usec): min=58, max=31293, avg=3350.01, stdev=3073.92 lat (usec): min=59, max=31294, avg=3350.65, stdev=3073.92 clat percentiles (usec): | 1.00th=[ 63], 5.00th=[ 72], 10.00th=[ 74], 20.00th=[ 103], | 30.00th=[ 2245], 40.00th=[ 2671], 50.00th=[ 2999], 60.00th=[ 3523], | 70.00th=[ 4293], 80.00th=[ 5145], 90.00th=[ 6259], 95.00th=[ 7504], | 99.00th=[14484], 99.50th=[21103], 99.90th=[31327], 99.95th=[31327], | 99.99th=[31327] bw ( KiB/s): min= 1792, max= 2560, per=26.45%, avg=2296.60, stdev=298.13, samples=5 iops : min= 112, max= 160, avg=143.40, stdev=18.54, samples=5 write: IOPS=152, BW=2436KiB/s (2495kB/s)(6256KiB/2568msec) clat (usec): min=659, max=28723, avg=3105.62, stdev=3265.28 lat (usec): min=660, max=28723, avg=3106.56, stdev=3265.32 clat percentiles (usec): | 1.00th=[ 717], 5.00th=[ 775], 10.00th=[ 824], 20.00th=[ 922], | 30.00th=[ 1020], 40.00th=[ 1237], 50.00th=[ 1745], 60.00th=[ 2704], | 70.00th=[ 3687], 80.00th=[ 5145], 90.00th=[ 6980], 95.00th=[ 8455], | 99.00th=[19006], 99.50th=[24249], 99.90th=[28705], 99.95th=[28705], | 99.99th=[28705] bw ( KiB/s): min= 1920, max= 3072, per=26.33%, avg=2405.40, stdev=421.26, samples=5 iops : min= 120, max= 192, avg=150.20, stdev=26.35, samples=5 lat (usec) : 100=9.37%, 250=2.34%, 500=0.26%, 750=1.56%, 1000=11.98% lat (msec) : 2=13.67%, 4=30.21%, 10=28.26%, 20=1.82%, 50=0.52% cpu : usr=0.23%, sys=27.58%, ctx=1736, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=8685KiB/s (8893kB/s), 2083KiB/s-2432KiB/s (2133kB/s-2491kB/s), io=23.4MiB (24.5MB), run=2525-2758msec WRITE: bw=9137KiB/s (9356kB/s), 2345KiB/s-2522KiB/s (2402kB/s-2583kB/s), io=24.6MiB (25.8MB), run=2525-2758msec rand-rw: (groupid=0, jobs=1): err= 0: pid=12259: Tue Apr 8 14:11:17 2025 read: IOPS=48, BW=783KiB/s (802kB/s)(5920KiB/7562msec) clat (usec): min=1945, max=31611, avg=3652.94, stdev=3191.58 lat (usec): min=1946, max=31612, avg=3653.63, stdev=3191.57 clat percentiles (usec): | 1.00th=[ 1975], 5.00th=[ 2057], 10.00th=[ 2114], 20.00th=[ 2180], | 30.00th=[ 2278], 40.00th=[ 2376], 50.00th=[ 2474], 60.00th=[ 2606], | 70.00th=[ 2999], 80.00th=[ 3949], 90.00th=[ 6390], 95.00th=[10159], | 99.00th=[16057], 99.50th=[26870], 99.90th=[31589], 99.95th=[31589], | 99.99th=[31589] bw ( KiB/s): min= 447, max= 1280, per=24.23%, avg=758.79, stdev=237.73, samples=14 iops : min= 27, max= 80, avg=47.36, stdev=14.95, samples=14 write: IOPS=52, BW=842KiB/s (862kB/s)(6368KiB/7562msec) clat (usec): min=8628, max=40884, avg=15443.83, stdev=4749.42 lat (usec): min=8629, max=40885, avg=15445.17, stdev=4749.45 clat percentiles (usec): | 1.00th=[ 8979], 5.00th=[ 9634], 10.00th=[10290], 20.00th=[11338], | 30.00th=[12518], 40.00th=[13566], 50.00th=[14746], 60.00th=[15926], | 70.00th=[16909], 80.00th=[18482], 90.00th=[21103], 95.00th=[25035], | 99.00th=[31065], 99.50th=[36963], 99.90th=[40633], 99.95th=[40633], | 99.99th=[40633] bw ( KiB/s): min= 447, max= 1120, per=25.04%, avg=825.07, stdev=217.96, samples=14 iops : min= 27, max= 70, avg=51.50, stdev=13.75, samples=14 lat (msec) : 2=1.04%, 4=37.50%, 10=10.94%, 20=43.88%, 50=6.64% cpu : usr=0.08%, sys=7.39%, ctx=913, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=12260: Tue Apr 8 14:11:17 2025 read: IOPS=46, BW=751KiB/s (769kB/s)(5744KiB/7647msec) clat (usec): min=1933, max=31552, avg=3473.41, stdev=3037.24 lat (usec): min=1933, max=31552, avg=3474.12, stdev=3037.28 clat percentiles (usec): | 1.00th=[ 1958], 5.00th=[ 2073], 10.00th=[ 2114], 20.00th=[ 2180], | 30.00th=[ 2245], 40.00th=[ 2343], 50.00th=[ 2442], 60.00th=[ 2606], | 70.00th=[ 2802], 80.00th=[ 3458], 90.00th=[ 6259], 95.00th=[ 9372], | 99.00th=[15401], 99.50th=[23725], 99.90th=[31589], 99.95th=[31589], | 99.99th=[31589] bw ( KiB/s): min= 320, max= 1088, per=23.81%, avg=745.67, stdev=234.09, samples=15 iops : min= 20, max= 68, avg=46.60, stdev=14.63, samples=15 write: IOPS=53, BW=856KiB/s (876kB/s)(6544KiB/7647msec) clat (usec): min=8212, max=39702, avg=15510.66, stdev=5096.74 lat (usec): min=8213, max=39704, avg=15511.97, stdev=5096.79 clat percentiles (usec): | 1.00th=[ 8848], 5.00th=[ 9372], 10.00th=[10028], 20.00th=[11469], | 30.00th=[12387], 40.00th=[13435], 50.00th=[14746], 60.00th=[15795], | 70.00th=[16712], 80.00th=[18482], 90.00th=[22414], 95.00th=[25822], | 99.00th=[33817], 99.50th=[34866], 99.90th=[39584], 99.95th=[39584], | 99.99th=[39584] bw ( KiB/s): min= 512, max= 1095, per=25.78%, avg=849.53, stdev=214.84, samples=15 iops : min= 32, max= 68, avg=53.07, stdev=13.39, samples=15 lat (msec) : 2=0.91%, 4=38.02%, 10=10.55%, 20=42.32%, 50=8.20% cpu : usr=0.14%, sys=7.44%, ctx=880, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=12261: Tue Apr 8 14:11:17 2025 read: IOPS=54, BW=866KiB/s (887kB/s)(6256KiB/7225msec) clat (usec): min=1874, max=26415, avg=3442.96, stdev=2997.09 lat (usec): min=1875, max=26416, avg=3443.65, stdev=2997.09 clat percentiles (usec): | 1.00th=[ 1958], 5.00th=[ 2024], 10.00th=[ 2089], 20.00th=[ 2180], | 30.00th=[ 2245], 40.00th=[ 2311], 50.00th=[ 2442], 60.00th=[ 2540], | 70.00th=[ 2769], 80.00th=[ 3261], 90.00th=[ 6325], 95.00th=[10028], | 99.00th=[16909], 99.50th=[24511], 99.90th=[26346], 99.95th=[26346], | 99.99th=[26346] bw ( KiB/s): min= 383, max= 1440, per=27.80%, avg=870.79, stdev=372.47, samples=14 iops : min= 23, max= 90, avg=54.36, stdev=23.38, samples=14 write: IOPS=52, BW=835KiB/s (855kB/s)(6032KiB/7225msec) clat (usec): min=8448, max=45554, avg=15400.45, stdev=5499.90 lat (usec): min=8449, max=45555, avg=15401.74, stdev=5500.01 clat percentiles (usec): | 1.00th=[ 8848], 5.00th=[ 9634], 10.00th=[10159], 20.00th=[10945], | 30.00th=[11863], 40.00th=[12780], 50.00th=[14091], 60.00th=[15664], | 70.00th=[16712], 80.00th=[17957], 90.00th=[23462], 95.00th=[26870], | 99.00th=[33817], 99.50th=[39060], 99.90th=[45351], 99.95th=[45351], | 99.99th=[45351] bw ( KiB/s): min= 512, max= 1088, per=25.04%, avg=825.00, stdev=224.06, samples=14 iops : min= 32, max= 68, avg=51.50, stdev=14.08, samples=14 lat (msec) : 2=1.69%, 4=41.41%, 10=8.98%, 20=39.97%, 50=7.94% cpu : usr=0.08%, sys=7.83%, ctx=926, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=12263: Tue Apr 8 14:11:17 2025 read: IOPS=50, BW=812KiB/s (832kB/s)(6032KiB/7428msec) clat (usec): min=1922, max=29287, avg=3305.73, stdev=2695.69 lat (usec): min=1922, max=29288, avg=3306.46, stdev=2695.67 clat percentiles (usec): | 1.00th=[ 1958], 5.00th=[ 2040], 10.00th=[ 2089], 20.00th=[ 2180], | 30.00th=[ 2245], 40.00th=[ 2311], 50.00th=[ 2409], 60.00th=[ 2540], | 70.00th=[ 2671], 80.00th=[ 3195], 90.00th=[ 5669], 95.00th=[ 9896], | 99.00th=[14746], 99.50th=[15270], 99.90th=[29230], 99.95th=[29230], | 99.99th=[29230] bw ( KiB/s): min= 415, max= 1280, per=25.24%, avg=790.64, stdev=272.21, samples=14 iops : min= 25, max= 80, avg=49.29, stdev=17.18, samples=14 write: IOPS=52, BW=842KiB/s (862kB/s)(6256KiB/7428msec) clat (usec): min=8185, max=40824, avg=15775.27, stdev=5464.74 lat (usec): min=8187, max=40826, avg=15776.81, stdev=5464.86 clat percentiles (usec): | 1.00th=[ 8717], 5.00th=[ 9503], 10.00th=[10159], 20.00th=[11469], | 30.00th=[12387], 40.00th=[13435], 50.00th=[14746], 60.00th=[15795], | 70.00th=[17171], 80.00th=[19006], 90.00th=[22414], 95.00th=[27395], | 99.00th=[34866], 99.50th=[39060], 99.90th=[40633], 99.95th=[40633], | 99.99th=[40633] bw ( KiB/s): min= 480, max= 1056, per=25.17%, avg=829.43, stdev=221.91, samples=14 iops : min= 30, max= 66, avg=51.71, stdev=14.05, samples=14 lat (msec) : 2=0.78%, 4=40.76%, 10=9.77%, 20=39.58%, 50=9.11% cpu : usr=0.13%, sys=7.47%, ctx=903, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=3132KiB/s (3207kB/s), 751KiB/s-866KiB/s (769kB/s-887kB/s), io=23.4MiB (24.5MB), run=7225-7647msec WRITE: bw=3295KiB/s (3374kB/s), 835KiB/s-856KiB/s (855kB/s-876kB/s), io=24.6MiB (25.8MB), run=7225-7647msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=12281: Tue Apr 8 14:11:19 2025 read: IOPS=13, BW=13.6MiB/s (14.3MB/s)(7168KiB/515msec) clat (usec): min=462, max=57451, avg=33473.28, stdev=17386.52 lat (usec): min=463, max=57451, avg=33474.39, stdev=17386.62 clat percentiles (usec): | 1.00th=[ 461], 5.00th=[ 461], 10.00th=[ 461], 20.00th=[28967], | 30.00th=[32375], 40.00th=[32375], 50.00th=[32375], 60.00th=[40633], | 70.00th=[40633], 80.00th=[42206], 90.00th=[57410], 95.00th=[57410], | 99.00th=[57410], 99.50th=[57410], 99.90th=[57410], 99.95th=[57410], | 99.99th=[57410] bw ( KiB/s): min=12288, max=12288, per=45.06%, avg=12288.00, stdev= 0.00, samples=1 iops : min= 12, max= 12, avg=12.00, stdev= 0.00, samples=1 write: IOPS=9, BW=9942KiB/s (10.2MB/s)(5120KiB/515msec) clat (usec): min=28677, max=94789, avg=54601.81, stdev=25135.35 lat (usec): min=28704, max=94837, avg=54650.81, stdev=25139.19 clat percentiles (usec): | 1.00th=[28705], 5.00th=[28705], 10.00th=[28705], 20.00th=[28705], | 30.00th=[42730], 40.00th=[42730], 50.00th=[46400], 60.00th=[46400], | 70.00th=[60556], 80.00th=[60556], 90.00th=[94897], 95.00th=[94897], | 99.00th=[94897], 99.50th=[94897], 99.90th=[94897], 99.95th=[94897], | 99.99th=[94897] bw ( KiB/s): min= 8192, max= 8192, per=21.46%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 lat (usec) : 500=8.33% lat (msec) : 50=66.67%, 100=25.00% cpu : usr=0.00%, sys=40.86%, ctx=729, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=12282: Tue Apr 8 14:11:19 2025 read: IOPS=2, BW=2727KiB/s (2792kB/s)(2048KiB/751msec) clat (usec): min=31073, max=46699, avg=38886.80, stdev=11049.03 lat (usec): min=31074, max=46703, avg=38888.93, stdev=11051.08 clat percentiles (usec): | 1.00th=[31065], 5.00th=[31065], 10.00th=[31065], 20.00th=[31065], | 30.00th=[31065], 40.00th=[31065], 50.00th=[31065], 60.00th=[46924], | 70.00th=[46924], 80.00th=[46924], 90.00th=[46924], 95.00th=[46924], | 99.00th=[46924], 99.50th=[46924], 99.90th=[46924], 99.95th=[46924], | 99.99th=[46924] bw ( KiB/s): min= 4096, max= 4096, per=15.02%, avg=4096.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 write: IOPS=13, BW=13.3MiB/s (13.0MB/s)(10.0MiB/751msec) clat (usec): min=22166, max=96427, avg=64389.51, stdev=24175.80 lat (usec): min=22229, max=96463, avg=64440.36, stdev=24173.69 clat percentiles (usec): | 1.00th=[22152], 5.00th=[22152], 10.00th=[22152], 20.00th=[32113], | 30.00th=[46400], 40.00th=[61080], 50.00th=[65799], 60.00th=[73925], | 70.00th=[80217], 80.00th=[80217], 90.00th=[87557], 95.00th=[95945], | 99.00th=[95945], 99.50th=[95945], 99.90th=[95945], 99.95th=[95945], | 99.99th=[95945] bw ( KiB/s): min=10240, max=10240, per=26.82%, avg=10240.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 lat (msec) : 50=41.67%, 100=58.33% cpu : usr=0.00%, sys=32.13%, ctx=311, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=12283: Tue Apr 8 14:11:19 2025 read: IOPS=11, BW=11.0MiB/s (11.6MB/s)(6144KiB/544msec) clat (usec): min=671, max=45802, avg=16056.38, stdev=18384.85 lat (usec): min=672, max=45803, avg=16057.32, stdev=18384.92 clat percentiles (usec): | 1.00th=[ 668], 5.00th=[ 668], 10.00th=[ 668], 20.00th=[ 865], | 30.00th=[ 865], 40.00th=[ 3458], 50.00th=[ 3458], 60.00th=[16057], | 70.00th=[29492], 80.00th=[29492], 90.00th=[45876], 95.00th=[45876], | 99.00th=[45876], 99.50th=[45876], 99.90th=[45876], 99.95th=[45876], | 99.99th=[45876] bw ( KiB/s): min=10240, max=10240, per=37.55%, avg=10240.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 write: IOPS=11, BW=11.0MiB/s (11.6MB/s)(6144KiB/544msec) clat (msec): min=40, max=101, avg=72.29, stdev=27.40 lat (msec): min=40, max=101, avg=72.34, stdev=27.41 clat percentiles (msec): | 1.00th=[ 41], 5.00th=[ 41], 10.00th=[ 41], 20.00th=[ 50], | 30.00th=[ 50], 40.00th=[ 54], 50.00th=[ 54], 60.00th=[ 92], | 70.00th=[ 97], 80.00th=[ 97], 90.00th=[ 102], 95.00th=[ 102], | 99.00th=[ 102], 99.50th=[ 102], 99.90th=[ 102], 99.95th=[ 102], | 99.99th=[ 102] bw ( KiB/s): min=10240, max=10240, per=26.82%, avg=10240.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 lat (usec) : 750=8.33%, 1000=8.33% lat (msec) : 4=8.33%, 20=8.33%, 50=33.33%, 100=25.00%, 250=8.33% cpu : usr=0.00%, sys=29.10%, ctx=381, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=12284: Tue Apr 8 14:11:19 2025 read: IOPS=8, BW=9078KiB/s (9296kB/s)(5120KiB/564msec) clat (usec): min=552, max=47043, avg=27181.17, stdev=23867.78 lat (usec): min=554, max=47045, avg=27182.49, stdev=23867.64 clat percentiles (usec): | 1.00th=[ 553], 5.00th=[ 553], 10.00th=[ 553], 20.00th=[ 553], | 30.00th=[ 1811], 40.00th=[ 1811], 50.00th=[40633], 60.00th=[40633], | 70.00th=[45876], 80.00th=[45876], 90.00th=[46924], 95.00th=[46924], | 99.00th=[46924], 99.50th=[46924], 99.90th=[46924], 99.95th=[46924], | 99.99th=[46924] bw ( KiB/s): min= 8192, max= 8192, per=30.04%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=12, BW=12.4MiB/s (13.0MB/s)(7168KiB/564msec) clat (usec): min=31864, max=72594, avg=56977.16, stdev=15405.51 lat (usec): min=32000, max=72670, avg=57843.66, stdev=15687.08 clat percentiles (usec): | 1.00th=[31851], 5.00th=[31851], 10.00th=[31851], 20.00th=[42730], | 30.00th=[51643], 40.00th=[51643], 50.00th=[62653], 60.00th=[65274], | 70.00th=[65274], 80.00th=[71828], 90.00th=[72877], 95.00th=[72877], | 99.00th=[72877], 99.50th=[72877], 99.90th=[72877], 99.95th=[72877], | 99.99th=[72877] bw ( KiB/s): min=10240, max=10240, per=26.82%, avg=10240.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 lat (usec) : 750=8.33% lat (msec) : 2=8.33%, 50=41.67%, 100=41.67% cpu : usr=0.00%, sys=30.73%, ctx=374, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=26.6MiB/s (27.9MB/s), 2727KiB/s-13.6MiB/s (2792kB/s-14.3MB/s), io=20.0MiB (20.0MB), run=515-751msec WRITE: bw=37.3MiB/s (39.1MB/s), 9942KiB/s-13.3MiB/s (10.2MB/s-13.0MB/s), io=28.0MiB (29.4MB), run=515-751msec rand-rw: (groupid=0, jobs=1): err= 0: pid=12286: Tue Apr 8 14:11:19 2025 read: IOPS=9, BW=9469KiB/s (9696kB/s)(7168KiB/757msec) clat (msec): min=12, max=182, avg=75.86, stdev=55.73 lat (msec): min=12, max=182, avg=75.86, stdev=55.73 clat percentiles (msec): | 1.00th=[ 13], 5.00th=[ 13], 10.00th=[ 13], 20.00th=[ 32], | 30.00th=[ 53], 40.00th=[ 53], 50.00th=[ 72], 60.00th=[ 78], | 70.00th=[ 78], 80.00th=[ 103], 90.00th=[ 184], 95.00th=[ 184], | 99.00th=[ 184], 99.50th=[ 184], 99.90th=[ 184], 99.95th=[ 184], | 99.99th=[ 184] bw ( KiB/s): min= 8192, max= 8192, per=30.88%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=6, BW=6764KiB/s (6926kB/s)(5120KiB/757msec) clat (usec): min=24187, max=55149, avg=39228.38, stdev=11362.08 lat (usec): min=24251, max=55215, avg=39290.10, stdev=11363.46 clat percentiles (usec): | 1.00th=[24249], 5.00th=[24249], 10.00th=[24249], 20.00th=[24249], | 30.00th=[35390], 40.00th=[35390], 50.00th=[37487], 60.00th=[37487], | 70.00th=[43779], 80.00th=[43779], 90.00th=[55313], 95.00th=[55313], | 99.00th=[55313], 99.50th=[55313], 99.90th=[55313], 99.95th=[55313], | 99.99th=[55313] bw ( KiB/s): min= 2048, max= 2048, per=5.51%, avg=2048.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 20=8.33%, 50=41.67%, 100=33.33%, 250=16.67% cpu : usr=0.00%, sys=12.30%, ctx=79, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=12287: Tue Apr 8 14:11:19 2025 read: IOPS=2, BW=2653KiB/s (2717kB/s)(2048KiB/772msec) clat (usec): min=59520, max=70016, avg=64768.58, stdev=7421.41 lat (usec): min=59521, max=70017, avg=64769.55, stdev=7421.30 clat percentiles (usec): | 1.00th=[59507], 5.00th=[59507], 10.00th=[59507], 20.00th=[59507], | 30.00th=[59507], 40.00th=[59507], 50.00th=[59507], 60.00th=[69731], | 70.00th=[69731], 80.00th=[69731], 90.00th=[69731], 95.00th=[69731], | 99.00th=[69731], 99.50th=[69731], 99.90th=[69731], 99.95th=[69731], | 99.99th=[69731] bw ( KiB/s): min= 2048, max= 2048, per=7.72%, avg=2048.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=12, BW=12.0MiB/s (13.6MB/s)(10.0MiB/772msec) clat (usec): min=20961, max=94274, avg=57749.99, stdev=23357.96 lat (usec): min=21024, max=94344, avg=57810.53, stdev=23360.40 clat percentiles (usec): | 1.00th=[20841], 5.00th=[20841], 10.00th=[20841], 20.00th=[36963], | 30.00th=[41157], 40.00th=[42206], 50.00th=[50594], 60.00th=[62129], | 70.00th=[80217], 80.00th=[80217], 90.00th=[82314], 95.00th=[93848], | 99.00th=[93848], 99.50th=[93848], 99.90th=[93848], 99.95th=[93848], | 99.99th=[93848] bw ( KiB/s): min=10240, max=10240, per=27.57%, avg=10240.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 lat (msec) : 50=33.33%, 100=66.67% cpu : usr=0.00%, sys=14.14%, ctx=79, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=12288: Tue Apr 8 14:11:19 2025 read: IOPS=9, BW=9615KiB/s (9846kB/s)(6144KiB/639msec) clat (usec): min=13384, max=73594, avg=34196.63, stdev=24160.64 lat (usec): min=13386, max=73595, avg=34197.66, stdev=24160.41 clat percentiles (usec): | 1.00th=[13435], 5.00th=[13435], 10.00th=[13435], 20.00th=[16319], | 30.00th=[16319], 40.00th=[18744], 50.00th=[18744], 60.00th=[30278], | 70.00th=[53216], 80.00th=[53216], 90.00th=[73925], 95.00th=[73925], | 99.00th=[73925], 99.50th=[73925], 99.90th=[73925], 99.95th=[73925], | 99.99th=[73925] bw ( KiB/s): min= 8192, max= 8192, per=30.88%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=9, BW=9615KiB/s (9846kB/s)(6144KiB/639msec) clat (msec): min=29, max=139, avg=60.71, stdev=41.91 lat (msec): min=29, max=139, avg=60.77, stdev=41.91 clat percentiles (msec): | 1.00th=[ 30], 5.00th=[ 30], 10.00th=[ 30], 20.00th=[ 37], | 30.00th=[ 37], 40.00th=[ 39], 50.00th=[ 39], 60.00th=[ 44], | 70.00th=[ 78], 80.00th=[ 78], 90.00th=[ 140], 95.00th=[ 140], | 99.00th=[ 140], 99.50th=[ 140], 99.90th=[ 140], 99.95th=[ 140], | 99.99th=[ 140] bw ( KiB/s): min= 8192, max= 8192, per=22.06%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 lat (msec) : 20=25.00%, 50=41.67%, 100=25.00%, 250=8.33% cpu : usr=0.16%, sys=14.42%, ctx=230, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=12289: Tue Apr 8 14:11:19 2025 read: IOPS=6, BW=7004KiB/s (7172kB/s)(5120KiB/731msec) clat (usec): min=16769, max=91501, avg=33936.49, stdev=32393.87 lat (usec): min=16770, max=91502, avg=33937.63, stdev=32393.73 clat percentiles (usec): | 1.00th=[16712], 5.00th=[16712], 10.00th=[16712], 20.00th=[16712], | 30.00th=[17433], 40.00th=[17433], 50.00th=[17957], 60.00th=[17957], | 70.00th=[25822], 80.00th=[25822], 90.00th=[91751], 95.00th=[91751], | 99.00th=[91751], 99.50th=[91751], 99.90th=[91751], 99.95th=[91751], | 99.99th=[91751] bw ( KiB/s): min= 4096, max= 4096, per=15.44%, avg=4096.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 write: IOPS=9, BW=9806KiB/s (10.0MB/s)(7168KiB/731msec) clat (msec): min=23, max=110, avg=71.14, stdev=33.78 lat (msec): min=23, max=110, avg=71.20, stdev=33.78 clat percentiles (msec): | 1.00th=[ 24], 5.00th=[ 24], 10.00th=[ 24], 20.00th=[ 28], | 30.00th=[ 68], 40.00th=[ 68], 50.00th=[ 84], 60.00th=[ 93], | 70.00th=[ 93], 80.00th=[ 94], 90.00th=[ 111], 95.00th=[ 111], | 99.00th=[ 111], 99.50th=[ 111], 99.90th=[ 111], 99.95th=[ 111], | 99.99th=[ 111] bw ( KiB/s): min= 8192, max= 8192, per=22.06%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 lat (msec) : 20=25.00%, 50=25.00%, 100=41.67%, 250=8.33% cpu : usr=0.00%, sys=14.79%, ctx=117, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=25.9MiB/s (27.2MB/s), 2653KiB/s-9615KiB/s (2717kB/s-9846kB/s), io=20.0MiB (20.0MB), run=639-772msec WRITE: bw=36.3MiB/s (38.0MB/s), 6764KiB/s-12.0MiB/s (6926kB/s-13.6MB/s), io=28.0MiB (29.4MB), run=639-772msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=12299: Tue Apr 8 14:11:20 2025 read: IOPS=6, BW=27.9MiB/s (29.3MB/s)(12.0MiB/430msec) clat (msec): min=127, max=155, avg=138.22, stdev=15.12 lat (msec): min=127, max=155, avg=138.22, stdev=15.12 clat percentiles (msec): | 1.00th=[ 128], 5.00th=[ 128], 10.00th=[ 128], 20.00th=[ 128], | 30.00th=[ 128], 40.00th=[ 132], 50.00th=[ 132], 60.00th=[ 132], | 70.00th=[ 157], 80.00th=[ 157], 90.00th=[ 157], 95.00th=[ 157], | 99.00th=[ 157], 99.50th=[ 157], 99.90th=[ 157], 99.95th=[ 157], | 99.99th=[ 157] lat (msec) : 250=100.00% cpu : usr=0.23%, sys=24.24%, ctx=648, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=12300: Tue Apr 8 14:11:20 2025 read: IOPS=1, BW=7571KiB/s (7753kB/s)(4096KiB/541msec) clat (nsec): min=131576k, max=131576k, avg=131575790.00, stdev= 0.00 lat (nsec): min=131577k, max=131577k, avg=131577242.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 132], 5.00th=[ 132], 10.00th=[ 132], 20.00th=[ 132], | 30.00th=[ 132], 40.00th=[ 132], 50.00th=[ 132], 60.00th=[ 132], | 70.00th=[ 132], 80.00th=[ 132], 90.00th=[ 132], 95.00th=[ 132], | 99.00th=[ 132], 99.50th=[ 132], 99.90th=[ 132], 99.95th=[ 132], | 99.99th=[ 132] bw ( KiB/s): min= 8192, max= 8192, per=27.05%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=3, BW=14.8MiB/s (15.5MB/s)(8192KiB/541msec) clat (msec): min=168, max=220, avg=194.83, stdev=36.88 lat (msec): min=169, max=221, avg=195.03, stdev=36.79 clat percentiles (msec): | 1.00th=[ 169], 5.00th=[ 169], 10.00th=[ 169], 20.00th=[ 169], | 30.00th=[ 169], 40.00th=[ 169], 50.00th=[ 169], 60.00th=[ 222], | 70.00th=[ 222], 80.00th=[ 222], 90.00th=[ 222], 95.00th=[ 222], | 99.00th=[ 222], 99.50th=[ 222], 99.90th=[ 222], 99.95th=[ 222], | 99.99th=[ 222] bw ( KiB/s): min= 8192, max= 8192, per=14.90%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=100.00% cpu : usr=0.00%, sys=29.26%, ctx=271, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=12301: Tue Apr 8 14:11:20 2025 write: IOPS=5, BW=20.1MiB/s (21.1MB/s)(12.0MiB/596msec) clat (msec): min=188, max=202, avg=193.66, stdev= 7.92 lat (msec): min=189, max=203, avg=194.00, stdev= 7.83 clat percentiles (msec): | 1.00th=[ 188], 5.00th=[ 188], 10.00th=[ 188], 20.00th=[ 188], | 30.00th=[ 188], 40.00th=[ 190], 50.00th=[ 190], 60.00th=[ 190], | 70.00th=[ 203], 80.00th=[ 203], 90.00th=[ 203], 95.00th=[ 203], | 99.00th=[ 203], 99.50th=[ 203], 99.90th=[ 203], 99.95th=[ 203], | 99.99th=[ 203] bw ( KiB/s): min=16384, max=16384, per=29.80%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 250=100.00% cpu : usr=0.00%, sys=36.81%, ctx=275, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=12302: Tue Apr 8 14:11:20 2025 write: IOPS=5, BW=20.4MiB/s (21.4MB/s)(12.0MiB/587msec) clat (msec): min=129, max=278, avg=189.76, stdev=78.69 lat (msec): min=129, max=279, avg=190.02, stdev=78.70 clat percentiles (msec): | 1.00th=[ 130], 5.00th=[ 130], 10.00th=[ 130], 20.00th=[ 130], | 30.00th=[ 130], 40.00th=[ 161], 50.00th=[ 161], 60.00th=[ 161], | 70.00th=[ 279], 80.00th=[ 279], 90.00th=[ 279], 95.00th=[ 279], | 99.00th=[ 279], 99.50th=[ 279], 99.90th=[ 279], 99.95th=[ 279], | 99.99th=[ 279] bw ( KiB/s): min=16318, max=16318, per=29.68%, avg=16318.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.00%, sys=40.96%, ctx=73, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=29.6MiB/s (31.0MB/s), 7571KiB/s-27.9MiB/s (7753kB/s-29.3MB/s), io=16.0MiB (16.8MB), run=430-541msec WRITE: bw=53.7MiB/s (56.3MB/s), 14.8MiB/s-20.4MiB/s (15.5MB/s-21.4MB/s), io=32.0MiB (33.6MB), run=541-596msec rand-rw: (groupid=0, jobs=1): err= 0: pid=12295: Tue Apr 8 14:11:20 2025 read: IOPS=9, BW=39.2MiB/s (41.1MB/s)(12.0MiB/306msec) clat (msec): min=58, max=173, avg=97.45, stdev=65.88 lat (msec): min=58, max=173, avg=97.45, stdev=65.88 clat percentiles (msec): | 1.00th=[ 59], 5.00th=[ 59], 10.00th=[ 59], 20.00th=[ 59], | 30.00th=[ 59], 40.00th=[ 61], 50.00th=[ 61], 60.00th=[ 61], | 70.00th=[ 174], 80.00th=[ 174], 90.00th=[ 174], 95.00th=[ 174], | 99.00th=[ 174], 99.50th=[ 174], 99.90th=[ 174], 99.95th=[ 174], | 99.99th=[ 174] lat (msec) : 100=66.67%, 250=33.33% cpu : usr=0.33%, sys=21.31%, ctx=75, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=12296: Tue Apr 8 14:11:20 2025 read: IOPS=1, BW=6321KiB/s (6473kB/s)(4096KiB/648msec) clat (nsec): min=254279k, max=254279k, avg=254278536.00, stdev= 0.00 lat (nsec): min=254281k, max=254281k, avg=254281046.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 255], 5.00th=[ 255], 10.00th=[ 255], 20.00th=[ 255], | 30.00th=[ 255], 40.00th=[ 255], 50.00th=[ 255], 60.00th=[ 255], | 70.00th=[ 255], 80.00th=[ 255], 90.00th=[ 255], 95.00th=[ 255], | 99.00th=[ 255], 99.50th=[ 255], 99.90th=[ 255], 99.95th=[ 255], | 99.99th=[ 255] bw ( KiB/s): min= 7846, max= 7846, per=31.03%, avg=7846.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=3, BW=12.3MiB/s (12.9MB/s)(8192KiB/648msec) clat (msec): min=170, max=208, avg=189.49, stdev=26.78 lat (msec): min=170, max=208, avg=189.81, stdev=26.73 clat percentiles (msec): | 1.00th=[ 171], 5.00th=[ 171], 10.00th=[ 171], 20.00th=[ 171], | 30.00th=[ 171], 40.00th=[ 171], 50.00th=[ 171], 60.00th=[ 209], | 70.00th=[ 209], 80.00th=[ 209], 90.00th=[ 209], 95.00th=[ 209], | 99.00th=[ 209], 99.50th=[ 209], 99.90th=[ 209], 99.95th=[ 209], | 99.99th=[ 209] bw ( KiB/s): min= 7846, max= 7846, per=18.51%, avg=7846.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.15%, sys=12.98%, ctx=27, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=12297: Tue Apr 8 14:11:20 2025 write: IOPS=3, BW=15.5MiB/s (16.3MB/s)(12.0MiB/773msec) clat (msec): min=74, max=365, avg=248.79, stdev=153.97 lat (msec): min=74, max=365, avg=249.09, stdev=154.00 clat percentiles (msec): | 1.00th=[ 74], 5.00th=[ 74], 10.00th=[ 74], 20.00th=[ 74], | 30.00th=[ 74], 40.00th=[ 309], 50.00th=[ 309], 60.00th=[ 309], | 70.00th=[ 368], 80.00th=[ 368], 90.00th=[ 368], 95.00th=[ 368], | 99.00th=[ 368], 99.50th=[ 368], 99.90th=[ 368], 99.95th=[ 368], | 99.99th=[ 368] bw ( KiB/s): min= 7772, max= 7772, per=18.33%, avg=7772.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 500=66.67% cpu : usr=0.13%, sys=11.01%, ctx=74, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=12298: Tue Apr 8 14:11:20 2025 write: IOPS=6, BW=25.9MiB/s (27.1MB/s)(12.0MiB/464msec) clat (msec): min=60, max=264, avg=150.15, stdev=104.12 lat (msec): min=60, max=264, avg=150.39, stdev=104.10 clat percentiles (msec): | 1.00th=[ 61], 5.00th=[ 61], 10.00th=[ 61], 20.00th=[ 61], | 30.00th=[ 61], 40.00th=[ 127], 50.00th=[ 127], 60.00th=[ 127], | 70.00th=[ 266], 80.00th=[ 266], 90.00th=[ 266], 95.00th=[ 266], | 99.00th=[ 266], 99.50th=[ 266], 99.90th=[ 266], 99.95th=[ 266], | 99.99th=[ 266] lat (msec) : 100=33.33%, 250=33.33%, 500=33.33% cpu : usr=0.00%, sys=13.17%, ctx=45, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=24.7MiB/s (25.9MB/s), 6321KiB/s-39.2MiB/s (6473kB/s-41.1MB/s), io=16.0MiB (16.8MB), run=306-648msec WRITE: bw=41.4MiB/s (43.4MB/s), 12.3MiB/s-25.9MiB/s (12.9MB/s-27.1MB/s), io=32.0MiB (33.6MB), run=464-773msec