== sanity test 398b: DIO and buffer IO race ============== 22:26:23 (1733801183) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.09753 s, 45.9 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=1867: Mon Dec 9 22:26:39 2024 read: IOPS=114, BW=456KiB/s (467kB/s)(5992KiB/13134msec) clat (usec): min=48, max=184868, avg=4134.38, stdev=5792.25 lat (usec): min=48, max=184868, avg=4135.69, stdev=5792.39 clat percentiles (usec): | 1.00th=[ 53], 5.00th=[ 62], 10.00th=[ 76], 20.00th=[ 2008], | 30.00th=[ 2540], 40.00th=[ 2966], 50.00th=[ 3326], 60.00th=[ 3949], | 70.00th=[ 4883], 80.00th=[ 6194], 90.00th=[ 8029], 95.00th=[ 9634], | 99.00th=[ 14615], 99.50th=[ 21627], 99.90th=[ 35914], 99.95th=[185598], | 99.99th=[185598] bw ( KiB/s): min= 216, max= 632, per=25.49%, avg=454.96, stdev=90.64, samples=26 iops : min= 54, max= 158, avg=113.69, stdev=22.64, samples=26 write: IOPS=119, BW=479KiB/s (491kB/s)(6296KiB/13134msec) clat (usec): min=446, max=30044, avg=4330.12, stdev=4199.69 lat (usec): min=446, max=30045, avg=4330.89, stdev=4199.67 clat percentiles (usec): | 1.00th=[ 498], 5.00th=[ 545], 10.00th=[ 586], 20.00th=[ 750], | 30.00th=[ 963], 40.00th=[ 1762], 50.00th=[ 3326], 60.00th=[ 4424], | 70.00th=[ 5604], 80.00th=[ 7373], 90.00th=[ 9896], 95.00th=[12518], | 99.00th=[17695], 99.50th=[20579], 99.90th=[30016], 99.95th=[30016], | 99.99th=[30016] bw ( KiB/s): min= 256, max= 616, per=25.44%, avg=475.54, stdev=80.57, samples=26 iops : min= 64, max= 154, avg=118.85, stdev=20.09, samples=26 lat (usec) : 50=0.26%, 100=6.51%, 250=2.21%, 500=0.75%, 750=9.64% lat (usec) : 1000=5.76% lat (msec) : 2=5.79%, 4=27.70%, 10=34.37%, 20=6.38%, 50=0.59% lat (msec) : 250=0.03% cpu : usr=0.18%, sys=24.29%, ctx=6254, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1869: Mon Dec 9 22:26:39 2024 read: IOPS=111, BW=445KiB/s (455kB/s)(5940KiB/13363msec) clat (usec): min=46, max=30997, avg=3981.45, stdev=3381.97 lat (usec): min=47, max=30998, avg=3982.08, stdev=3381.97 clat percentiles (usec): | 1.00th=[ 54], 5.00th=[ 62], 10.00th=[ 78], 20.00th=[ 1876], | 30.00th=[ 2474], 40.00th=[ 2900], 50.00th=[ 3294], 60.00th=[ 3916], | 70.00th=[ 4752], 80.00th=[ 6194], 90.00th=[ 8160], 95.00th=[ 9765], | 99.00th=[14091], 99.50th=[22414], 99.90th=[27919], 99.95th=[31065], | 99.99th=[31065] bw ( KiB/s): min= 223, max= 600, per=24.74%, avg=441.62, stdev=85.86, samples=26 iops : min= 55, max= 150, avg=110.31, stdev=21.54, samples=26 write: IOPS=118, BW=475KiB/s (486kB/s)(6348KiB/13363msec) clat (usec): min=485, max=215499, avg=4610.66, stdev=6926.69 lat (usec): min=486, max=215500, avg=4612.78, stdev=6927.42 clat percentiles (usec): | 1.00th=[ 506], 5.00th=[ 553], 10.00th=[ 594], 20.00th=[ 742], | 30.00th=[ 971], 40.00th=[ 1876], 50.00th=[ 3425], 60.00th=[ 4555], | 70.00th=[ 5604], 80.00th=[ 7373], 90.00th=[ 10159], 95.00th=[ 13173], | 99.00th=[ 19792], 99.50th=[ 24773], 99.90th=[ 34866], 99.95th=[214959], | 99.99th=[214959] bw ( KiB/s): min= 287, max= 686, per=24.95%, avg=466.27, stdev=87.86, samples=26 iops : min= 71, max= 171, avg=116.46, stdev=22.00, samples=26 lat (usec) : 50=0.26%, 100=7.00%, 250=1.76%, 500=0.46%, 750=10.61% lat (usec) : 1000=5.31% lat (msec) : 2=5.79%, 4=26.89%, 10=34.28%, 20=6.87%, 50=0.75% lat (msec) : 250=0.03% cpu : usr=0.16%, sys=23.56%, ctx=6134, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1870: Mon Dec 9 22:26:39 2024 read: IOPS=112, BW=448KiB/s (459kB/s)(6028KiB/13445msec) clat (usec): min=47, max=42286, avg=3936.91, stdev=3630.40 lat (usec): min=47, max=42287, avg=3938.43, stdev=3630.96 clat percentiles (usec): | 1.00th=[ 52], 5.00th=[ 60], 10.00th=[ 71], 20.00th=[ 127], | 30.00th=[ 2409], 40.00th=[ 2769], 50.00th=[ 3228], 60.00th=[ 3851], | 70.00th=[ 4817], 80.00th=[ 6128], 90.00th=[ 8029], 95.00th=[10028], | 99.00th=[14353], 99.50th=[21627], 99.90th=[40633], 99.95th=[42206], | 99.99th=[42206] bw ( KiB/s): min= 120, max= 632, per=24.59%, avg=438.96, stdev=105.95, samples=26 iops : min= 30, max= 158, avg=109.69, stdev=26.45, samples=26 write: IOPS=116, BW=466KiB/s (477kB/s)(6260KiB/13445msec) clat (usec): min=466, max=204344, avg=4714.28, stdev=6737.45 lat (usec): min=467, max=204347, avg=4716.16, stdev=6737.01 clat percentiles (usec): | 1.00th=[ 506], 5.00th=[ 553], 10.00th=[ 586], 20.00th=[ 725], | 30.00th=[ 996], 40.00th=[ 2212], 50.00th=[ 3687], 60.00th=[ 4686], | 70.00th=[ 5866], 80.00th=[ 7832], 90.00th=[ 10159], 95.00th=[ 12649], | 99.00th=[ 19530], 99.50th=[ 20579], 99.90th=[ 50594], 99.95th=[204473], | 99.99th=[204473] bw ( KiB/s): min= 176, max= 608, per=24.31%, avg=454.38, stdev=86.37, samples=26 iops : min= 44, max= 152, avg=113.54, stdev=21.62, samples=26 lat (usec) : 50=0.36%, 100=8.04%, 250=2.05%, 500=0.65%, 750=10.25% lat (usec) : 1000=4.62% lat (msec) : 2=5.08%, 4=26.33%, 10=34.57%, 20=7.42%, 50=0.55% lat (msec) : 100=0.03%, 250=0.03% cpu : usr=0.16%, sys=23.48%, ctx=6260, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1871: Mon Dec 9 22:26:39 2024 read: IOPS=114, BW=457KiB/s (467kB/s)(6052KiB/13257msec) clat (usec): min=48, max=35158, avg=3964.82, stdev=3272.50 lat (usec): min=48, max=35159, avg=3965.87, stdev=3272.87 clat percentiles (usec): | 1.00th=[ 57], 5.00th=[ 63], 10.00th=[ 80], 20.00th=[ 1991], | 30.00th=[ 2540], 40.00th=[ 2900], 50.00th=[ 3359], 60.00th=[ 3916], | 70.00th=[ 4948], 80.00th=[ 6194], 90.00th=[ 7832], 95.00th=[ 9372], | 99.00th=[13566], 99.50th=[15270], 99.90th=[32113], 99.95th=[35390], | 99.99th=[35390] bw ( KiB/s): min= 224, max= 592, per=25.21%, avg=450.00, stdev=86.98, samples=26 iops : min= 56, max= 148, avg=112.42, stdev=21.75, samples=26 write: IOPS=117, BW=470KiB/s (482kB/s)(6236KiB/13257msec) clat (usec): min=423, max=181306, avg=4575.88, stdev=6258.61 lat (usec): min=424, max=181307, avg=4577.21, stdev=6258.53 clat percentiles (usec): | 1.00th=[ 510], 5.00th=[ 545], 10.00th=[ 578], 20.00th=[ 717], | 30.00th=[ 979], 40.00th=[ 1975], 50.00th=[ 3556], 60.00th=[ 4490], | 70.00th=[ 5800], 80.00th=[ 7504], 90.00th=[ 10290], 95.00th=[ 12518], | 99.00th=[ 19006], 99.50th=[ 22152], 99.90th=[ 48497], 99.95th=[181404], | 99.99th=[181404] bw ( KiB/s): min= 272, max= 592, per=24.84%, avg=464.19, stdev=78.00, samples=26 iops : min= 68, max= 148, avg=115.96, stdev=19.53, samples=26 lat (usec) : 50=0.16%, 100=7.16%, 250=1.43%, 500=0.72%, 750=10.42% lat (usec) : 1000=4.62% lat (msec) : 2=5.76%, 4=27.64%, 10=34.80%, 20=6.67%, 50=0.59% lat (msec) : 250=0.03% cpu : usr=0.24%, sys=23.99%, ctx=6410, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1786KiB/s (1829kB/s), 445KiB/s-457KiB/s (455kB/s-467kB/s), io=23.4MiB (24.6MB), run=13134-13445msec WRITE: bw=1870KiB/s (1915kB/s), 466KiB/s-479KiB/s (477kB/s-491kB/s), io=24.6MiB (25.7MB), run=13134-13445msec rand-rw: (groupid=0, jobs=1): err= 0: pid=1864: Mon Dec 9 22:27:01 2024 read: IOPS=41, BW=167KiB/s (171kB/s)(5992KiB/35805msec) clat (usec): min=1455, max=220076, avg=4490.39, stdev=6527.19 lat (usec): min=1455, max=220077, avg=4491.16, stdev=6527.23 clat percentiles (usec): | 1.00th=[ 1647], 5.00th=[ 1811], 10.00th=[ 1926], 20.00th=[ 2114], | 30.00th=[ 2409], 40.00th=[ 3163], 50.00th=[ 3589], 60.00th=[ 3884], | 70.00th=[ 4359], 80.00th=[ 5669], 90.00th=[ 8029], 95.00th=[ 11076], | 99.00th=[ 15926], 99.50th=[ 20317], 99.90th=[ 51643], 99.95th=[219153], | 99.99th=[219153] bw ( KiB/s): min= 55, max= 280, per=24.90%, avg=166.85, stdev=65.34, samples=71 iops : min= 13, max= 70, avg=41.61, stdev=16.38, samples=71 write: IOPS=43, BW=176KiB/s (180kB/s)(6296KiB/35805msec) clat (usec): min=8426, max=51591, avg=18392.30, stdev=7013.27 lat (usec): min=8427, max=51592, avg=18393.13, stdev=7013.29 clat percentiles (usec): | 1.00th=[10159], 5.00th=[11076], 10.00th=[11600], 20.00th=[12649], | 30.00th=[13829], 40.00th=[15139], 50.00th=[16712], 60.00th=[18220], | 70.00th=[19530], 80.00th=[22676], 90.00th=[28181], 95.00th=[33817], | 99.00th=[41157], 99.50th=[44827], 99.90th=[51119], 99.95th=[51643], | 99.99th=[51643] bw ( KiB/s): min= 47, max= 248, per=24.96%, avg=174.96, stdev=52.49, samples=71 iops : min= 11, max= 62, avg=43.63, stdev=13.14, samples=71 lat (msec) : 2=7.19%, 4=23.54%, 10=15.27%, 20=39.62%, 50=14.26% lat (msec) : 100=0.10%, 250=0.03% cpu : usr=0.07%, sys=5.61%, ctx=4050, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1865: Mon Dec 9 22:27:01 2024 read: IOPS=41, BW=166KiB/s (170kB/s)(5940KiB/35813msec) clat (usec): min=1495, max=183711, avg=4462.61, stdev=5846.33 lat (usec): min=1496, max=183713, avg=4463.36, stdev=5846.37 clat percentiles (usec): | 1.00th=[ 1663], 5.00th=[ 1811], 10.00th=[ 1893], 20.00th=[ 2057], | 30.00th=[ 2376], 40.00th=[ 3163], 50.00th=[ 3556], 60.00th=[ 3916], | 70.00th=[ 4424], 80.00th=[ 5669], 90.00th=[ 7635], 95.00th=[ 10421], | 99.00th=[ 17957], 99.50th=[ 23725], 99.90th=[ 60031], 99.95th=[183501], | 99.99th=[183501] bw ( KiB/s): min= 15, max= 328, per=24.72%, avg=165.63, stdev=70.82, samples=71 iops : min= 3, max= 82, avg=41.32, stdev=17.73, samples=71 write: IOPS=44, BW=177KiB/s (182kB/s)(6348KiB/35813msec) clat (usec): min=8895, max=65684, avg=18310.26, stdev=7051.73 lat (usec): min=8895, max=65684, avg=18311.07, stdev=7051.77 clat percentiles (usec): | 1.00th=[10028], 5.00th=[10945], 10.00th=[11600], 20.00th=[12780], | 30.00th=[13829], 40.00th=[15270], 50.00th=[16712], 60.00th=[17957], | 70.00th=[19530], 80.00th=[21890], 90.00th=[27919], 95.00th=[33162], | 99.00th=[42730], 99.50th=[45876], 99.90th=[62129], 99.95th=[65799], | 99.99th=[65799] bw ( KiB/s): min= 55, max= 248, per=25.16%, avg=176.34, stdev=51.50, samples=71 iops : min= 13, max= 62, avg=44.00, stdev=12.90, samples=71 lat (msec) : 2=8.04%, 4=22.20%, 10=15.85%, 20=39.94%, 50=13.83% lat (msec) : 100=0.10%, 250=0.03% cpu : usr=0.11%, sys=5.50%, ctx=4028, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1866: Mon Dec 9 22:27:01 2024 read: IOPS=42, BW=170KiB/s (174kB/s)(6028KiB/35480msec) clat (usec): min=1506, max=34261, avg=4402.68, stdev=3195.50 lat (usec): min=1507, max=34262, avg=4403.43, stdev=3195.50 clat percentiles (usec): | 1.00th=[ 1647], 5.00th=[ 1827], 10.00th=[ 1942], 20.00th=[ 2180], | 30.00th=[ 2540], 40.00th=[ 3261], 50.00th=[ 3621], 60.00th=[ 3916], | 70.00th=[ 4686], 80.00th=[ 5800], 90.00th=[ 7701], 95.00th=[10552], | 99.00th=[15795], 99.50th=[20579], 99.90th=[31851], 99.95th=[34341], | 99.99th=[34341] bw ( KiB/s): min= 55, max= 304, per=25.26%, avg=169.27, stdev=61.69, samples=70 iops : min= 13, max= 76, avg=42.24, stdev=15.43, samples=70 write: IOPS=44, BW=176KiB/s (181kB/s)(6260KiB/35480msec) clat (msec): min=9, max=202, avg=18.35, stdev= 8.39 lat (msec): min=9, max=202, avg=18.35, stdev= 8.39 clat percentiles (msec): | 1.00th=[ 11], 5.00th=[ 11], 10.00th=[ 12], 20.00th=[ 13], | 30.00th=[ 14], 40.00th=[ 16], 50.00th=[ 17], 60.00th=[ 18], | 70.00th=[ 20], 80.00th=[ 22], 90.00th=[ 29], 95.00th=[ 34], | 99.00th=[ 41], 99.50th=[ 44], 99.90th=[ 56], 99.95th=[ 203], | 99.99th=[ 203] bw ( KiB/s): min= 63, max= 248, per=25.03%, avg=175.44, stdev=53.07, samples=70 iops : min= 15, max= 62, avg=43.79, stdev=13.28, samples=70 lat (msec) : 2=6.22%, 4=24.22%, 10=16.41%, 20=39.23%, 50=13.83% lat (msec) : 100=0.07%, 250=0.03% cpu : usr=0.10%, sys=5.67%, ctx=4064, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1868: Mon Dec 9 22:27:01 2024 read: IOPS=42, BW=170KiB/s (174kB/s)(6052KiB/35535msec) clat (usec): min=1495, max=60153, avg=4352.99, stdev=3526.31 lat (usec): min=1495, max=60154, avg=4353.75, stdev=3526.32 clat percentiles (usec): | 1.00th=[ 1663], 5.00th=[ 1811], 10.00th=[ 1909], 20.00th=[ 2114], | 30.00th=[ 2409], 40.00th=[ 3163], 50.00th=[ 3589], 60.00th=[ 3916], | 70.00th=[ 4424], 80.00th=[ 5735], 90.00th=[ 7767], 95.00th=[10290], | 99.00th=[15795], 99.50th=[19006], 99.90th=[43254], 99.95th=[60031], | 99.99th=[60031] bw ( KiB/s): min= 56, max= 328, per=25.42%, avg=170.28, stdev=65.52, samples=71 iops : min= 14, max= 82, avg=42.51, stdev=16.39, samples=71 write: IOPS=43, BW=175KiB/s (180kB/s)(6236KiB/35535msec) clat (msec): min=8, max=192, avg=18.49, stdev= 8.32 lat (msec): min=8, max=192, avg=18.49, stdev= 8.32 clat percentiles (msec): | 1.00th=[ 11], 5.00th=[ 12], 10.00th=[ 12], 20.00th=[ 13], | 30.00th=[ 14], 40.00th=[ 16], 50.00th=[ 17], 60.00th=[ 19], | 70.00th=[ 20], 80.00th=[ 23], 90.00th=[ 29], 95.00th=[ 33], | 99.00th=[ 44], 99.50th=[ 48], 99.90th=[ 66], 99.95th=[ 192], | 99.99th=[ 192] bw ( KiB/s): min= 64, max= 240, per=25.00%, avg=175.24, stdev=50.77, samples=71 iops : min= 16, max= 60, avg=43.75, stdev=12.67, samples=71 lat (msec) : 2=7.36%, 4=23.47%, 10=16.21%, 20=39.26%, 50=13.48% lat (msec) : 100=0.20%, 250=0.03% cpu : usr=0.10%, sys=5.54%, ctx=4052, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=670KiB/s (687kB/s), 166KiB/s-170KiB/s (170kB/s-174kB/s), io=23.4MiB (24.6MB), run=35480-35813msec WRITE: bw=702KiB/s (719kB/s), 175KiB/s-177KiB/s (180kB/s-182kB/s), io=24.6MiB (25.7MB), run=35480-35813msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=1914: Mon Dec 9 22:27:07 2024 read: IOPS=80, BW=1282KiB/s (1313kB/s)(5920KiB/4618msec) clat (usec): min=52, max=50470, avg=8941.89, stdev=5855.97 lat (usec): min=52, max=50471, avg=8942.56, stdev=5856.00 clat percentiles (usec): | 1.00th=[ 58], 5.00th=[ 65], 10.00th=[ 75], 20.00th=[ 106], | 30.00th=[ 8356], 40.00th=[ 9503], 50.00th=[10290], 60.00th=[10945], | 70.00th=[11731], 80.00th=[12649], 90.00th=[14091], 95.00th=[16057], | 99.00th=[22676], 99.50th=[29754], 99.90th=[50594], 99.95th=[50594], | 99.99th=[50594] bw ( KiB/s): min= 928, max= 1532, per=25.13%, avg=1293.44, stdev=185.86, samples=9 iops : min= 58, max= 95, avg=80.67, stdev=11.46, samples=9 write: IOPS=86, BW=1379KiB/s (1412kB/s)(6368KiB/4618msec) clat (usec): min=624, max=32844, avg=3247.45, stdev=3666.84 lat (usec): min=625, max=32845, avg=3248.53, stdev=3666.85 clat percentiles (usec): | 1.00th=[ 685], 5.00th=[ 758], 10.00th=[ 816], 20.00th=[ 922], | 30.00th=[ 1020], 40.00th=[ 1221], 50.00th=[ 1549], 60.00th=[ 2835], | 70.00th=[ 3720], 80.00th=[ 5211], 90.00th=[ 7046], 95.00th=[ 8848], | 99.00th=[17433], 99.50th=[25822], 99.90th=[32900], 99.95th=[32900], | 99.99th=[32900] bw ( KiB/s): min= 1021, max= 1888, per=25.78%, avg=1396.56, stdev=314.93, samples=9 iops : min= 63, max= 118, avg=87.11, stdev=19.76, samples=9 lat (usec) : 100=8.85%, 250=2.34%, 500=0.13%, 750=2.34%, 1000=12.50% lat (msec) : 2=12.50%, 4=10.81%, 10=22.27%, 20=27.34%, 50=0.78% lat (msec) : 100=0.13% cpu : usr=0.13%, sys=22.92%, ctx=1979, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1915: Mon Dec 9 22:27:07 2024 read: IOPS=77, BW=1238KiB/s (1268kB/s)(5744KiB/4640msec) clat (usec): min=52, max=22167, avg=8776.03, stdev=5495.14 lat (usec): min=52, max=22168, avg=8776.70, stdev=5495.13 clat percentiles (usec): | 1.00th=[ 57], 5.00th=[ 65], 10.00th=[ 76], 20.00th=[ 118], | 30.00th=[ 8160], 40.00th=[ 9241], 50.00th=[10290], 60.00th=[11076], | 70.00th=[11994], 80.00th=[13042], 90.00th=[14746], 95.00th=[16188], | 99.00th=[19006], 99.50th=[20841], 99.90th=[22152], 99.95th=[22152], | 99.99th=[22152] bw ( KiB/s): min= 704, max= 1600, per=23.82%, avg=1226.11, stdev=288.04, samples=9 iops : min= 44, max= 100, avg=76.44, stdev=18.07, samples=9 write: IOPS=88, BW=1410KiB/s (1444kB/s)(6544KiB/4640msec) clat (usec): min=587, max=30576, avg=3503.86, stdev=4074.68 lat (usec): min=588, max=30578, avg=3505.12, stdev=4074.58 clat percentiles (usec): | 1.00th=[ 660], 5.00th=[ 750], 10.00th=[ 799], 20.00th=[ 873], | 30.00th=[ 988], 40.00th=[ 1205], 50.00th=[ 2180], 60.00th=[ 3130], | 70.00th=[ 4015], 80.00th=[ 5080], 90.00th=[ 7373], 95.00th=[10683], | 99.00th=[22676], 99.50th=[23462], 99.90th=[30540], 99.95th=[30540], | 99.99th=[30540] bw ( KiB/s): min= 928, max= 1920, per=25.32%, avg=1371.67, stdev=258.95, samples=9 iops : min= 58, max= 120, avg=85.56, stdev=16.14, samples=9 lat (usec) : 100=8.20%, 250=2.99%, 750=2.99%, 1000=13.41% lat (msec) : 2=10.16%, 4=10.68%, 10=24.48%, 20=26.04%, 50=1.04% cpu : usr=0.11%, sys=22.81%, ctx=1965, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1916: Mon Dec 9 22:27:07 2024 read: IOPS=84, BW=1355KiB/s (1388kB/s)(6256KiB/4617msec) clat (usec): min=51, max=21455, avg=8807.87, stdev=5449.29 lat (usec): min=51, max=21456, avg=8808.56, stdev=5449.32 clat percentiles (usec): | 1.00th=[ 56], 5.00th=[ 65], 10.00th=[ 73], 20.00th=[ 108], | 30.00th=[ 8291], 40.00th=[ 9372], 50.00th=[10290], 60.00th=[10945], | 70.00th=[11994], 80.00th=[13042], 90.00th=[14746], 95.00th=[15664], | 99.00th=[19530], 99.50th=[19792], 99.90th=[21365], 99.95th=[21365], | 99.99th=[21365] bw ( KiB/s): min= 992, max= 1952, per=26.31%, avg=1354.33, stdev=302.37, samples=9 iops : min= 62, max= 122, avg=84.56, stdev=18.92, samples=9 write: IOPS=81, BW=1306KiB/s (1338kB/s)(6032KiB/4617msec) clat (usec): min=696, max=33579, avg=2954.81, stdev=3172.25 lat (usec): min=698, max=33581, avg=2955.91, stdev=3172.34 clat percentiles (usec): | 1.00th=[ 709], 5.00th=[ 766], 10.00th=[ 807], 20.00th=[ 873], | 30.00th=[ 947], 40.00th=[ 1123], 50.00th=[ 1418], 60.00th=[ 2671], | 70.00th=[ 3490], 80.00th=[ 4948], 90.00th=[ 7111], 95.00th=[ 8225], | 99.00th=[13042], 99.50th=[17957], 99.90th=[33817], 99.95th=[33817], | 99.99th=[33817] bw ( KiB/s): min= 990, max= 1952, per=23.69%, avg=1283.33, stdev=289.78, samples=9 iops : min= 61, max= 122, avg=80.11, stdev=18.22, samples=9 lat (usec) : 100=9.77%, 250=2.47%, 500=0.13%, 750=1.56%, 1000=15.10% lat (msec) : 2=10.29%, 4=10.16%, 10=22.40%, 20=27.86%, 50=0.26% cpu : usr=0.15%, sys=22.68%, ctx=2014, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1917: Mon Dec 9 22:27:07 2024 read: IOPS=81, BW=1297KiB/s (1328kB/s)(6032KiB/4652msec) clat (usec): min=51, max=37234, avg=8524.84, stdev=6209.42 lat (usec): min=51, max=37235, avg=8525.47, stdev=6209.44 clat percentiles (usec): | 1.00th=[ 54], 5.00th=[ 62], 10.00th=[ 65], 20.00th=[ 88], | 30.00th=[ 6783], 40.00th=[ 9241], 50.00th=[10290], 60.00th=[11076], | 70.00th=[11731], 80.00th=[12649], 90.00th=[14615], 95.00th=[16450], | 99.00th=[24511], 99.50th=[32637], 99.90th=[37487], 99.95th=[37487], | 99.99th=[37487] bw ( KiB/s): min= 768, max= 1568, per=24.72%, avg=1272.56, stdev=223.13, samples=9 iops : min= 48, max= 98, avg=79.44, stdev=13.90, samples=9 write: IOPS=84, BW=1345KiB/s (1377kB/s)(6256KiB/4652msec) clat (usec): min=647, max=26394, avg=3487.73, stdev=3401.20 lat (usec): min=648, max=26395, avg=3488.83, stdev=3401.23 clat percentiles (usec): | 1.00th=[ 685], 5.00th=[ 766], 10.00th=[ 816], 20.00th=[ 906], | 30.00th=[ 1074], 40.00th=[ 1254], 50.00th=[ 2409], 60.00th=[ 3392], | 70.00th=[ 4146], 80.00th=[ 5932], 90.00th=[ 7373], 95.00th=[10028], | 99.00th=[15795], 99.50th=[20317], 99.90th=[26346], 99.95th=[26346], | 99.99th=[26346] bw ( KiB/s): min= 800, max= 1728, per=24.61%, avg=1333.00, stdev=277.42, samples=9 iops : min= 50, max= 108, avg=83.22, stdev=17.42, samples=9 lat (usec) : 100=11.85%, 250=2.34%, 500=0.13%, 750=2.21%, 1000=11.20% lat (msec) : 2=11.33%, 4=9.77%, 10=23.05%, 20=26.69%, 50=1.43% cpu : usr=0.06%, sys=22.30%, ctx=1993, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=5149KiB/s (5272kB/s), 1238KiB/s-1355KiB/s (1268kB/s-1388kB/s), io=23.4MiB (24.5MB), run=4617-4652msec WRITE: bw=5417KiB/s (5547kB/s), 1306KiB/s-1410KiB/s (1338kB/s-1444kB/s), io=24.6MiB (25.8MB), run=4617-4652msec rand-rw: (groupid=0, jobs=1): err= 0: pid=1909: Mon Dec 9 22:27:11 2024 read: IOPS=41, BW=669KiB/s (685kB/s)(5920KiB/8849msec) clat (usec): min=1894, max=51487, avg=4321.17, stdev=4066.82 lat (usec): min=1895, max=51488, avg=4322.04, stdev=4066.81 clat percentiles (usec): | 1.00th=[ 1958], 5.00th=[ 2180], 10.00th=[ 2245], 20.00th=[ 2442], | 30.00th=[ 2573], 40.00th=[ 2737], 50.00th=[ 2933], 60.00th=[ 3163], | 70.00th=[ 3621], 80.00th=[ 5145], 90.00th=[ 8029], 95.00th=[11207], | 99.00th=[18482], 99.50th=[28181], 99.90th=[51643], 99.95th=[51643], | 99.99th=[51643] bw ( KiB/s): min= 288, max= 1216, per=24.65%, avg=656.82, stdev=211.14, samples=17 iops : min= 18, max= 76, avg=41.00, stdev=13.22, samples=17 write: IOPS=44, BW=720KiB/s (737kB/s)(6368KiB/8849msec) clat (usec): min=9626, max=43306, avg=18130.79, stdev=5629.67 lat (usec): min=9628, max=43308, avg=18132.68, stdev=5629.53 clat percentiles (usec): | 1.00th=[10290], 5.00th=[11207], 10.00th=[11994], 20.00th=[13173], | 30.00th=[14615], 40.00th=[15926], 50.00th=[17171], 60.00th=[18744], | 70.00th=[20055], 80.00th=[21627], 90.00th=[26084], 95.00th=[29492], | 99.00th=[34866], 99.50th=[38536], 99.90th=[43254], 99.95th=[43254], | 99.99th=[43254] bw ( KiB/s): min= 448, max= 960, per=25.37%, avg=711.41, stdev=168.80, samples=17 iops : min= 28, max= 60, avg=44.41, stdev=10.57, samples=17 lat (msec) : 2=0.52%, 4=34.77%, 10=10.03%, 20=38.80%, 50=15.76% lat (msec) : 100=0.13% cpu : usr=0.18%, sys=6.82%, ctx=894, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1910: Mon Dec 9 22:27:11 2024 read: IOPS=39, BW=639KiB/s (654kB/s)(5744KiB/8987msec) clat (usec): min=1880, max=40839, avg=4124.26, stdev=3881.66 lat (usec): min=1881, max=40839, avg=4125.06, stdev=3881.67 clat percentiles (usec): | 1.00th=[ 1991], 5.00th=[ 2114], 10.00th=[ 2180], 20.00th=[ 2311], | 30.00th=[ 2442], 40.00th=[ 2573], 50.00th=[ 2737], 60.00th=[ 2966], | 70.00th=[ 3392], 80.00th=[ 4621], 90.00th=[ 7701], 95.00th=[12518], | 99.00th=[19792], 99.50th=[27132], 99.90th=[40633], 99.95th=[40633], | 99.99th=[40633] bw ( KiB/s): min= 224, max= 1152, per=23.66%, avg=630.47, stdev=263.56, samples=17 iops : min= 14, max= 72, avg=39.35, stdev=16.47, samples=17 write: IOPS=45, BW=728KiB/s (746kB/s)(6544KiB/8987msec) clat (usec): min=9466, max=71002, avg=18320.32, stdev=6493.46 lat (usec): min=9467, max=71005, avg=18322.00, stdev=6493.46 clat percentiles (usec): | 1.00th=[10159], 5.00th=[11076], 10.00th=[11600], 20.00th=[12911], | 30.00th=[14222], 40.00th=[15795], 50.00th=[17171], 60.00th=[18744], | 70.00th=[20055], 80.00th=[22152], 90.00th=[27132], 95.00th=[29492], | 99.00th=[38536], 99.50th=[41157], 99.90th=[70779], 99.95th=[70779], | 99.99th=[70779] bw ( KiB/s): min= 480, max= 1024, per=25.37%, avg=711.41, stdev=174.44, samples=17 iops : min= 30, max= 64, avg=44.41, stdev=10.94, samples=17 lat (msec) : 2=0.91%, 4=35.03%, 10=8.07%, 20=39.32%, 50=16.54% lat (msec) : 100=0.13% cpu : usr=0.11%, sys=6.78%, ctx=889, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1911: Mon Dec 9 22:27:11 2024 read: IOPS=44, BW=714KiB/s (731kB/s)(6256KiB/8767msec) clat (usec): min=1906, max=31443, avg=4348.25, stdev=3774.75 lat (usec): min=1906, max=31444, avg=4349.06, stdev=3774.77 clat percentiles (usec): | 1.00th=[ 1926], 5.00th=[ 2114], 10.00th=[ 2212], 20.00th=[ 2376], | 30.00th=[ 2507], 40.00th=[ 2638], 50.00th=[ 2835], 60.00th=[ 3064], | 70.00th=[ 3720], 80.00th=[ 5407], 90.00th=[ 8717], 95.00th=[12649], | 99.00th=[19268], 99.50th=[24511], 99.90th=[31327], 99.95th=[31327], | 99.99th=[31327] bw ( KiB/s): min= 352, max= 1376, per=26.98%, avg=718.94, stdev=269.27, samples=17 iops : min= 22, max= 86, avg=44.88, stdev=16.87, samples=17 write: IOPS=43, BW=688KiB/s (705kB/s)(6032KiB/8767msec) clat (usec): min=9079, max=43874, avg=18672.25, stdev=5932.57 lat (usec): min=9080, max=43877, avg=18673.84, stdev=5932.57 clat percentiles (usec): | 1.00th=[10159], 5.00th=[11600], 10.00th=[12256], 20.00th=[13566], | 30.00th=[15008], 40.00th=[16319], 50.00th=[17433], 60.00th=[18744], | 70.00th=[20055], 80.00th=[22938], 90.00th=[26870], 95.00th=[30802], | 99.00th=[36439], 99.50th=[40109], 99.90th=[43779], 99.95th=[43779], | 99.99th=[43779] bw ( KiB/s): min= 448, max= 928, per=24.30%, avg=681.29, stdev=181.32, samples=17 iops : min= 28, max= 58, avg=42.53, stdev=11.35, samples=17 lat (msec) : 2=1.17%, 4=36.33%, 10=9.51%, 20=37.63%, 50=15.36% cpu : usr=0.17%, sys=7.13%, ctx=902, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1912: Mon Dec 9 22:27:11 2024 read: IOPS=42, BW=683KiB/s (699kB/s)(6032KiB/8836msec) clat (usec): min=1924, max=25299, avg=4219.25, stdev=3475.73 lat (usec): min=1925, max=25300, avg=4220.07, stdev=3475.69 clat percentiles (usec): | 1.00th=[ 2008], 5.00th=[ 2089], 10.00th=[ 2212], 20.00th=[ 2343], | 30.00th=[ 2507], 40.00th=[ 2704], 50.00th=[ 2900], 60.00th=[ 3130], | 70.00th=[ 3720], 80.00th=[ 5211], 90.00th=[ 7832], 95.00th=[11994], | 99.00th=[21890], 99.50th=[22938], 99.90th=[25297], 99.95th=[25297], | 99.99th=[25297] bw ( KiB/s): min= 416, max= 1184, per=25.28%, avg=673.76, stdev=208.82, samples=17 iops : min= 26, max= 74, avg=42.06, stdev=13.08, samples=17 write: IOPS=44, BW=708KiB/s (725kB/s)(6256KiB/8836msec) clat (usec): min=9470, max=74190, avg=18474.92, stdev=6303.24 lat (usec): min=9471, max=74193, avg=18476.75, stdev=6303.34 clat percentiles (usec): | 1.00th=[10290], 5.00th=[11207], 10.00th=[11994], 20.00th=[13304], | 30.00th=[15270], 40.00th=[16712], 50.00th=[17695], 60.00th=[18744], | 70.00th=[19792], 80.00th=[21627], 90.00th=[25822], 95.00th=[29492], | 99.00th=[36963], 99.50th=[49546], 99.90th=[73925], 99.95th=[73925], | 99.99th=[73925] bw ( KiB/s): min= 448, max= 960, per=25.10%, avg=703.88, stdev=169.40, samples=17 iops : min= 28, max= 60, avg=43.94, stdev=10.62, samples=17 lat (msec) : 2=0.65%, 4=34.77%, 10=10.55%, 20=38.54%, 50=15.36% lat (msec) : 100=0.13% cpu : usr=0.18%, sys=6.55%, ctx=892, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2665KiB/s (2729kB/s), 639KiB/s-714KiB/s (654kB/s-731kB/s), io=23.4MiB (24.5MB), run=8767-8987msec WRITE: bw=2804KiB/s (2871kB/s), 688KiB/s-728KiB/s (705kB/s-746kB/s), io=24.6MiB (25.8MB), run=8767-8987msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=1941: Mon Dec 9 22:27:17 2024 read: IOPS=1, BW=1630KiB/s (1669kB/s)(7168KiB/4398msec) clat (msec): min=92, max=936, avg=592.97, stdev=356.45 lat (msec): min=92, max=936, avg=592.97, stdev=356.45 clat percentiles (msec): | 1.00th=[ 93], 5.00th=[ 93], 10.00th=[ 93], 20.00th=[ 207], | 30.00th=[ 380], 40.00th=[ 380], 50.00th=[ 768], 60.00th=[ 869], | 70.00th=[ 869], 80.00th=[ 894], 90.00th=[ 936], 95.00th=[ 936], | 99.00th=[ 936], 99.50th=[ 936], 99.90th=[ 936], 99.95th=[ 936], | 99.99th=[ 936] bw ( KiB/s): min= 1700, max= 2048, per=41.46%, avg=1930.33, stdev=178.43, samples=6 iops : min= 1, max= 2, avg= 1.33, stdev= 0.52, samples=6 write: IOPS=1, BW=1164KiB/s (1192kB/s)(5120KiB/4398msec) clat (usec): min=18714, max=65399, avg=46770.05, stdev=20393.28 lat (usec): min=18770, max=65517, avg=46845.47, stdev=20415.95 clat percentiles (usec): | 1.00th=[18744], 5.00th=[18744], 10.00th=[18744], 20.00th=[18744], | 30.00th=[33162], 40.00th=[33162], 50.00th=[51643], 60.00th=[51643], | 70.00th=[64750], 80.00th=[64750], 90.00th=[65274], 95.00th=[65274], | 99.00th=[65274], 99.50th=[65274], 99.90th=[65274], 99.95th=[65274], | 99.99th=[65274] bw ( KiB/s): min= 1700, max= 5102, per=45.25%, avg=2950.00, stdev=1871.79, samples=3 iops : min= 1, max= 4, avg= 2.33, stdev= 1.53, samples=3 lat (msec) : 20=8.33%, 50=8.33%, 100=33.33%, 250=8.33%, 500=8.33% lat (msec) : 1000=33.33% cpu : usr=0.00%, sys=26.56%, ctx=1365, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1942: Mon Dec 9 22:27:17 2024 read: IOPS=0, BW=911KiB/s (933kB/s)(2048KiB/2247msec) clat (usec): min=553, max=46266, avg=23410.20, stdev=32323.89 lat (usec): min=555, max=46268, avg=23411.67, stdev=32323.86 clat percentiles (usec): | 1.00th=[ 553], 5.00th=[ 553], 10.00th=[ 553], 20.00th=[ 553], | 30.00th=[ 553], 40.00th=[ 553], 50.00th=[ 553], 60.00th=[46400], | 70.00th=[46400], 80.00th=[46400], 90.00th=[46400], 95.00th=[46400], | 99.00th=[46400], 99.50th=[46400], 99.90th=[46400], 99.95th=[46400], | 99.99th=[46400] bw ( KiB/s): min= 2048, max= 2048, per=43.99%, avg=2048.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 write: IOPS=4, BW=4557KiB/s (4667kB/s)(10.0MiB/2247msec) clat (msec): min=19, max=1107, avg=214.02, stdev=348.33 lat (msec): min=19, max=1107, avg=214.17, stdev=348.38 clat percentiles (msec): | 1.00th=[ 20], 5.00th=[ 20], 10.00th=[ 20], 20.00th=[ 26], | 30.00th=[ 36], 40.00th=[ 47], 50.00th=[ 47], 60.00th=[ 57], | 70.00th=[ 234], 80.00th=[ 234], 90.00th=[ 506], 95.00th=[ 1116], | 99.00th=[ 1116], 99.50th=[ 1116], 99.90th=[ 1116], 99.95th=[ 1116], | 99.99th=[ 1116] bw ( KiB/s): min= 2048, max=14336, per=100.00%, avg=8192.00, stdev=8688.93, samples=2 iops : min= 2, max= 14, avg= 8.00, stdev= 8.49, samples=2 lat (usec) : 750=8.33% lat (msec) : 20=8.33%, 50=41.67%, 100=16.67%, 250=8.33%, 750=8.33% cpu : usr=0.00%, sys=12.82%, ctx=145, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1943: Mon Dec 9 22:27:17 2024 read: IOPS=3, BW=3212KiB/s (3289kB/s)(6144KiB/1913msec) clat (usec): min=322, max=224475, avg=69138.56, stdev=82835.41 lat (usec): min=322, max=224477, avg=69139.39, stdev=82835.66 clat percentiles (usec): | 1.00th=[ 322], 5.00th=[ 322], 10.00th=[ 322], 20.00th=[ 392], | 30.00th=[ 392], 40.00th=[ 49546], 50.00th=[ 49546], 60.00th=[ 56886], | 70.00th=[ 83362], 80.00th=[ 83362], 90.00th=[225444], 95.00th=[225444], | 99.00th=[225444], 99.50th=[225444], 99.90th=[225444], 99.95th=[225444], | 99.99th=[225444] bw ( KiB/s): min= 4096, max= 4096, per=87.97%, avg=4096.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 write: IOPS=3, BW=3212KiB/s (3289kB/s)(6144KiB/1913msec) clat (msec): min=21, max=1141, avg=237.18, stdev=443.55 lat (msec): min=21, max=1141, avg=237.22, stdev=443.55 clat percentiles (msec): | 1.00th=[ 22], 5.00th=[ 22], 10.00th=[ 22], 20.00th=[ 51], | 30.00th=[ 51], 40.00th=[ 53], 50.00th=[ 53], 60.00th=[ 68], | 70.00th=[ 90], 80.00th=[ 90], 90.00th=[ 1150], 95.00th=[ 1150], | 99.00th=[ 1150], 99.50th=[ 1150], 99.90th=[ 1150], 99.95th=[ 1150], | 99.99th=[ 1150] bw ( KiB/s): min= 2048, max= 4096, per=47.12%, avg=3072.00, stdev=1448.15, samples=2 iops : min= 2, max= 4, avg= 3.00, stdev= 1.41, samples=2 lat (usec) : 500=16.67% lat (msec) : 50=16.67%, 100=50.00%, 250=8.33% cpu : usr=0.00%, sys=15.01%, ctx=197, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1944: Mon Dec 9 22:27:17 2024 read: IOPS=1, BW=1995KiB/s (2043kB/s)(5120KiB/2566msec) clat (usec): min=540, max=664997, avg=192711.53, stdev=268898.85 lat (usec): min=542, max=664997, avg=192712.50, stdev=268898.61 clat percentiles (usec): | 1.00th=[ 545], 5.00th=[ 545], 10.00th=[ 545], 20.00th=[ 545], | 30.00th=[ 54789], 40.00th=[ 54789], 50.00th=[113771], 60.00th=[113771], | 70.00th=[129500], 80.00th=[129500], 90.00th=[666895], 95.00th=[666895], | 99.00th=[666895], 99.50th=[666895], 99.90th=[666895], 99.95th=[666895], | 99.99th=[666895] bw ( KiB/s): min= 2048, max= 4096, per=58.65%, avg=2730.67, stdev=1182.41, samples=3 iops : min= 2, max= 4, avg= 2.67, stdev= 1.15, samples=3 write: IOPS=2, BW=2793KiB/s (2860kB/s)(7168KiB/2566msec) clat (msec): min=21, max=1120, avg=217.85, stdev=403.76 lat (msec): min=21, max=1120, avg=217.91, stdev=403.76 clat percentiles (msec): | 1.00th=[ 22], 5.00th=[ 22], 10.00th=[ 22], 20.00th=[ 22], | 30.00th=[ 34], 40.00th=[ 34], 50.00th=[ 40], 60.00th=[ 72], | 70.00th=[ 72], 80.00th=[ 218], 90.00th=[ 1116], 95.00th=[ 1116], | 99.00th=[ 1116], 99.50th=[ 1116], 99.90th=[ 1116], 99.95th=[ 1116], | 99.99th=[ 1116] bw ( KiB/s): min= 2048, max= 6144, per=73.30%, avg=4778.67, stdev=2364.83, samples=3 iops : min= 2, max= 6, avg= 4.67, stdev= 2.31, samples=3 lat (usec) : 750=8.33% lat (msec) : 50=33.33%, 100=16.67%, 250=25.00%, 750=8.33% cpu : usr=0.04%, sys=17.89%, ctx=423, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=4657KiB/s (4768kB/s), 911KiB/s-3212KiB/s (933kB/s-3289kB/s), io=20.0MiB (20.0MB), run=1913-4398msec WRITE: bw=6519KiB/s (6676kB/s), 1164KiB/s-4557KiB/s (1192kB/s-4667kB/s), io=28.0MiB (29.4MB), run=1913-4398msec rand-rw: (groupid=0, jobs=1): err= 0: pid=1937: Mon Dec 9 22:27:17 2024 read: IOPS=1, BW=1593KiB/s (1631kB/s)(7168KiB/4500msec) clat (msec): min=15, max=1049, avg=354.03, stdev=402.64 lat (msec): min=15, max=1049, avg=354.03, stdev=402.64 clat percentiles (msec): | 1.00th=[ 16], 5.00th=[ 16], 10.00th=[ 16], 20.00th=[ 18], | 30.00th=[ 82], 40.00th=[ 82], 50.00th=[ 207], 60.00th=[ 347], | 70.00th=[ 347], 80.00th=[ 760], 90.00th=[ 1053], 95.00th=[ 1053], | 99.00th=[ 1053], 99.50th=[ 1053], 99.90th=[ 1053], 99.95th=[ 1053], | 99.99th=[ 1053] bw ( KiB/s): min= 1689, max= 3373, per=49.25%, avg=2241.20, stdev=651.51, samples=5 iops : min= 1, max= 3, avg= 2.00, stdev= 0.71, samples=5 write: IOPS=1, BW=1138KiB/s (1165kB/s)(5120KiB/4500msec) clat (msec): min=58, max=954, avg=392.12, stdev=436.32 lat (msec): min=58, max=954, avg=392.18, stdev=436.30 clat percentiles (msec): | 1.00th=[ 59], 5.00th=[ 59], 10.00th=[ 59], 20.00th=[ 59], | 30.00th=[ 78], 40.00th=[ 78], 50.00th=[ 96], 60.00th=[ 96], | 70.00th=[ 776], 80.00th=[ 776], 90.00th=[ 953], 95.00th=[ 953], | 99.00th=[ 953], 99.50th=[ 953], 99.90th=[ 953], 99.95th=[ 953], | 99.99th=[ 953] bw ( KiB/s): min= 1686, max= 3379, per=35.32%, avg=2250.33, stdev=977.45, samples=3 iops : min= 1, max= 3, avg= 1.67, stdev= 1.15, samples=3 lat (msec) : 20=16.67%, 100=33.33%, 250=8.33%, 500=8.33%, 1000=25.00% cpu : usr=0.00%, sys=3.25%, ctx=57, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1938: Mon Dec 9 22:27:17 2024 read: IOPS=0, BW=861KiB/s (881kB/s)(2048KiB/2380msec) clat (msec): min=18, max=109, avg=63.96, stdev=64.17 lat (msec): min=18, max=109, avg=63.96, stdev=64.17 clat percentiles (msec): | 1.00th=[ 19], 5.00th=[ 19], 10.00th=[ 19], 20.00th=[ 19], | 30.00th=[ 19], 40.00th=[ 19], 50.00th=[ 19], 60.00th=[ 110], | 70.00th=[ 110], 80.00th=[ 110], 90.00th=[ 110], 95.00th=[ 110], | 99.00th=[ 110], 99.50th=[ 110], 99.90th=[ 110], 99.95th=[ 110], | 99.99th=[ 110] bw ( KiB/s): min= 2048, max= 2048, per=45.00%, avg=2048.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 write: IOPS=4, BW=4303KiB/s (4406kB/s)(10.0MiB/2380msec) clat (msec): min=25, max=1106, avg=219.17, stdev=346.21 lat (msec): min=25, max=1106, avg=219.24, stdev=346.21 clat percentiles (msec): | 1.00th=[ 26], 5.00th=[ 26], 10.00th=[ 26], 20.00th=[ 31], | 30.00th=[ 32], 40.00th=[ 33], 50.00th=[ 34], 60.00th=[ 36], | 70.00th=[ 288], 80.00th=[ 288], 90.00th=[ 477], 95.00th=[ 1099], | 99.00th=[ 1099], 99.50th=[ 1099], 99.90th=[ 1099], 99.95th=[ 1099], | 99.99th=[ 1099] bw ( KiB/s): min= 6144, max=10240, per=100.00%, avg=8192.00, stdev=2896.31, samples=2 iops : min= 6, max= 10, avg= 8.00, stdev= 2.83, samples=2 lat (msec) : 20=8.33%, 50=50.00%, 250=16.67%, 500=16.67% cpu : usr=0.21%, sys=4.62%, ctx=54, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1939: Mon Dec 9 22:27:17 2024 read: IOPS=2, BW=2916KiB/s (2986kB/s)(6144KiB/2107msec) clat (msec): min=25, max=249, avg=97.59, stdev=81.20 lat (msec): min=25, max=249, avg=97.59, stdev=81.20 clat percentiles (msec): | 1.00th=[ 26], 5.00th=[ 26], 10.00th=[ 26], 20.00th=[ 32], | 30.00th=[ 32], 40.00th=[ 88], 50.00th=[ 88], 60.00th=[ 88], | 70.00th=[ 103], 80.00th=[ 103], 90.00th=[ 251], 95.00th=[ 251], | 99.00th=[ 251], 99.50th=[ 251], 99.90th=[ 251], 99.95th=[ 251], | 99.99th=[ 251] bw ( KiB/s): min= 2048, max= 8192, per=100.00%, avg=5120.00, stdev=4344.46, samples=2 iops : min= 2, max= 8, avg= 5.00, stdev= 4.24, samples=2 write: IOPS=2, BW=2916KiB/s (2986kB/s)(6144KiB/2107msec) clat (msec): min=31, max=1133, avg=242.78, stdev=437.02 lat (msec): min=31, max=1133, avg=242.84, stdev=437.01 clat percentiles (msec): | 1.00th=[ 32], 5.00th=[ 32], 10.00th=[ 32], 20.00th=[ 47], | 30.00th=[ 47], 40.00th=[ 55], 50.00th=[ 55], 60.00th=[ 87], | 70.00th=[ 105], 80.00th=[ 105], 90.00th=[ 1133], 95.00th=[ 1133], | 99.00th=[ 1133], 99.50th=[ 1133], 99.90th=[ 1133], 99.95th=[ 1133], | 99.99th=[ 1133] bw ( KiB/s): min= 2048, max= 4096, per=53.58%, avg=3413.33, stdev=1182.41, samples=3 iops : min= 2, max= 4, avg= 3.33, stdev= 1.15, samples=3 lat (msec) : 50=33.33%, 100=33.33%, 250=25.00% cpu : usr=0.09%, sys=5.51%, ctx=76, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1940: Mon Dec 9 22:27:17 2024 read: IOPS=2, BW=2475KiB/s (2534kB/s)(5120KiB/2069msec) clat (msec): min=12, max=510, avg=117.99, stdev=219.67 lat (msec): min=12, max=510, avg=117.99, stdev=219.67 clat percentiles (msec): | 1.00th=[ 13], 5.00th=[ 13], 10.00th=[ 13], 20.00th=[ 13], | 30.00th=[ 14], 40.00th=[ 14], 50.00th=[ 23], 60.00th=[ 23], | 70.00th=[ 32], 80.00th=[ 32], 90.00th=[ 510], 95.00th=[ 510], | 99.00th=[ 510], 99.50th=[ 510], 99.90th=[ 510], 99.95th=[ 510], | 99.99th=[ 510] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=3, BW=3464KiB/s (3548kB/s)(7168KiB/2069msec) clat (msec): min=34, max=1114, avg=208.66, stdev=400.51 lat (msec): min=34, max=1114, avg=208.73, stdev=400.49 clat percentiles (msec): | 1.00th=[ 35], 5.00th=[ 35], 10.00th=[ 35], 20.00th=[ 37], | 30.00th=[ 40], 40.00th=[ 40], 50.00th=[ 45], 60.00th=[ 75], | 70.00th=[ 75], 80.00th=[ 116], 90.00th=[ 1116], 95.00th=[ 1116], | 99.00th=[ 1116], 99.50th=[ 1116], 99.90th=[ 1116], 99.95th=[ 1116], | 99.99th=[ 1116] bw ( KiB/s): min= 2048, max= 8192, per=75.01%, avg=4778.67, stdev=3128.37, samples=3 iops : min= 2, max= 8, avg= 4.67, stdev= 3.06, samples=3 lat (msec) : 20=16.67%, 50=50.00%, 100=8.33%, 250=8.33%, 750=8.33% cpu : usr=0.00%, sys=4.35%, ctx=52, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=4551KiB/s (4660kB/s), 861KiB/s-2916KiB/s (881kB/s-2986kB/s), io=20.0MiB (20.0MB), run=2069-4500msec WRITE: bw=6372KiB/s (6524kB/s), 1138KiB/s-4303KiB/s (1165kB/s-4406kB/s), io=28.0MiB (29.4MB), run=2069-4500msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=1953: Mon Dec 9 22:27:22 2024 read: IOPS=0, BW=2985KiB/s (3056kB/s)(12.0MiB/4117msec) clat (msec): min=49, max=2369, avg=1363.54, stdev=1190.14 lat (msec): min=49, max=2369, avg=1363.54, stdev=1190.14 clat percentiles (msec): | 1.00th=[ 51], 5.00th=[ 51], 10.00th=[ 51], 20.00th=[ 51], | 30.00th=[ 51], 40.00th=[ 1670], 50.00th=[ 1670], 60.00th=[ 1670], | 70.00th=[ 2366], 80.00th=[ 2366], 90.00th=[ 2366], 95.00th=[ 2366], | 99.00th=[ 2366], 99.50th=[ 2366], 99.90th=[ 2366], 99.95th=[ 2366], | 99.99th=[ 2366] bw ( KiB/s): min= 7301, max= 8159, per=100.00%, avg=7730.00, stdev=606.70, samples=2 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=2 lat (msec) : 50=33.33% cpu : usr=0.00%, sys=43.00%, ctx=992, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1954: Mon Dec 9 22:27:22 2024 read: IOPS=0, BW=931KiB/s (953kB/s)(4096KiB/4400msec) clat (nsec): min=59510k, max=59510k, avg=59509927.00, stdev= 0.00 lat (nsec): min=59512k, max=59512k, avg=59511989.00, stdev= 0.00 clat percentiles (usec): | 1.00th=[59507], 5.00th=[59507], 10.00th=[59507], 20.00th=[59507], | 30.00th=[59507], 40.00th=[59507], 50.00th=[59507], 60.00th=[59507], | 70.00th=[59507], 80.00th=[59507], 90.00th=[59507], 95.00th=[59507], | 99.00th=[59507], 99.50th=[59507], 99.90th=[59507], 99.95th=[59507], | 99.99th=[59507] bw ( KiB/s): min= 8062, max= 8062, per=100.00%, avg=8062.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=0, BW=1862KiB/s (1907kB/s)(8192KiB/4400msec) clat (msec): min=428, max=3868, avg=2148.76, stdev=2432.46 lat (msec): min=428, max=3869, avg=2149.00, stdev=2432.54 clat percentiles (msec): | 1.00th=[ 430], 5.00th=[ 430], 10.00th=[ 430], 20.00th=[ 430], | 30.00th=[ 430], 40.00th=[ 430], 50.00th=[ 430], 60.00th=[ 3876], | 70.00th=[ 3876], 80.00th=[ 3876], 90.00th=[ 3876], 95.00th=[ 3876], | 99.00th=[ 3876], 99.50th=[ 3876], 99.90th=[ 3876], 99.95th=[ 3876], | 99.99th=[ 3876] bw ( KiB/s): min= 6989, max= 6989, per=93.85%, avg=6989.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 500=33.33% cpu : usr=0.00%, sys=4.68%, ctx=236, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1955: Mon Dec 9 22:27:22 2024 write: IOPS=0, BW=2820KiB/s (2888kB/s)(12.0MiB/4357msec) clat (msec): min=297, max=2159, avg=1446.69, stdev=1005.17 lat (msec): min=297, max=2159, avg=1447.08, stdev=1005.09 clat percentiles (msec): | 1.00th=[ 296], 5.00th=[ 296], 10.00th=[ 296], 20.00th=[ 296], | 30.00th=[ 296], 40.00th=[ 1888], 50.00th=[ 1888], 60.00th=[ 1888], | 70.00th=[ 2165], 80.00th=[ 2165], 90.00th=[ 2165], 95.00th=[ 2165], | 99.00th=[ 2165], 99.50th=[ 2165], 99.90th=[ 2165], 99.95th=[ 2165], | 99.99th=[ 2165] bw ( KiB/s): min= 6977, max= 8047, per=100.00%, avg=7512.00, stdev=756.60, samples=2 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=2 lat (msec) : 500=33.33% cpu : usr=0.00%, sys=6.22%, ctx=194, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1956: Mon Dec 9 22:27:22 2024 write: IOPS=4, BW=16.8MiB/s (17.6MB/s)(12.0MiB/713msec) clat (msec): min=140, max=281, avg=224.56, stdev=74.38 lat (msec): min=140, max=281, avg=224.80, stdev=74.31 clat percentiles (msec): | 1.00th=[ 140], 5.00th=[ 140], 10.00th=[ 140], 20.00th=[ 140], | 30.00th=[ 140], 40.00th=[ 253], 50.00th=[ 253], 60.00th=[ 253], | 70.00th=[ 284], 80.00th=[ 284], 90.00th=[ 284], 95.00th=[ 284], | 99.00th=[ 284], 99.50th=[ 284], 99.90th=[ 284], 99.95th=[ 284], | 99.99th=[ 284] bw ( KiB/s): min=16221, max=16221, per=100.00%, avg=16221.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=66.67% cpu : usr=0.00%, sys=32.30%, ctx=73, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=3724KiB/s (3813kB/s), 931KiB/s-2985KiB/s (953kB/s-3056kB/s), io=16.0MiB (16.8MB), run=4117-4400msec WRITE: bw=7447KiB/s (7626kB/s), 1862KiB/s-16.8MiB/s (1907kB/s-17.6MB/s), io=32.0MiB (33.6MB), run=713-4400msec rand-rw: (groupid=0, jobs=1): err= 0: pid=1958: Mon Dec 9 22:27:23 2024 read: IOPS=0, BW=2833KiB/s (2901kB/s)(12.0MiB/4338msec) clat (msec): min=404, max=2185, avg=1435.49, stdev=923.03 lat (msec): min=404, max=2185, avg=1435.49, stdev=923.03 clat percentiles (msec): | 1.00th=[ 405], 5.00th=[ 405], 10.00th=[ 405], 20.00th=[ 405], | 30.00th=[ 405], 40.00th=[ 1720], 50.00th=[ 1720], 60.00th=[ 1720], | 70.00th=[ 2198], 80.00th=[ 2198], 90.00th=[ 2198], 95.00th=[ 2198], | 99.00th=[ 2198], 99.50th=[ 2198], 99.90th=[ 2198], 99.95th=[ 2198], | 99.99th=[ 2198] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 lat (msec) : 500=33.33% cpu : usr=0.00%, sys=1.94%, ctx=41, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1959: Mon Dec 9 22:27:23 2024 read: IOPS=0, BW=877KiB/s (898kB/s)(4096KiB/4672msec) clat (nsec): min=342160k, max=342160k, avg=342160432.00, stdev= 0.00 lat (nsec): min=342162k, max=342162k, avg=342161553.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 342], 5.00th=[ 342], 10.00th=[ 342], 20.00th=[ 342], | 30.00th=[ 342], 40.00th=[ 342], 50.00th=[ 342], 60.00th=[ 342], | 70.00th=[ 342], 80.00th=[ 342], 90.00th=[ 342], 95.00th=[ 342], | 99.00th=[ 342], 99.50th=[ 342], 99.90th=[ 342], 99.95th=[ 342], | 99.99th=[ 342] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=0, BW=1753KiB/s (1796kB/s)(8192KiB/4672msec) clat (msec): min=235, max=4048, avg=2141.93, stdev=2696.52 lat (msec): min=235, max=4048, avg=2142.21, stdev=2696.52 clat percentiles (msec): | 1.00th=[ 236], 5.00th=[ 236], 10.00th=[ 236], 20.00th=[ 236], | 30.00th=[ 236], 40.00th=[ 236], 50.00th=[ 236], 60.00th=[ 4044], | 70.00th=[ 4044], 80.00th=[ 4044], 90.00th=[ 4044], 95.00th=[ 4044], | 99.00th=[ 4044], 99.50th=[ 4044], 99.90th=[ 4044], 99.95th=[ 4044], | 99.99th=[ 4044] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=33.33% cpu : usr=0.04%, sys=1.71%, ctx=49, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1960: Mon Dec 9 22:27:23 2024 write: IOPS=0, BW=2631KiB/s (2694kB/s)(12.0MiB/4670msec) clat (msec): min=62, max=4029, avg=1540.92, stdev=2167.97 lat (msec): min=62, max=4029, avg=1541.28, stdev=2167.89 clat percentiles (msec): | 1.00th=[ 63], 5.00th=[ 63], 10.00th=[ 63], 20.00th=[ 63], | 30.00th=[ 63], 40.00th=[ 531], 50.00th=[ 531], 60.00th=[ 531], | 70.00th=[ 4044], 80.00th=[ 4044], 90.00th=[ 4044], 95.00th=[ 4044], | 99.00th=[ 4044], 99.50th=[ 4044], 99.90th=[ 4044], 99.95th=[ 4044], | 99.99th=[ 4044] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 750=33.33% cpu : usr=0.04%, sys=1.80%, ctx=38, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1961: Mon Dec 9 22:27:23 2024 write: IOPS=0, BW=2533KiB/s (2594kB/s)(12.0MiB/4851msec) clat (msec): min=157, max=4157, avg=1606.70, stdev=2215.51 lat (msec): min=157, max=4157, avg=1607.06, stdev=2215.42 clat percentiles (msec): | 1.00th=[ 157], 5.00th=[ 157], 10.00th=[ 157], 20.00th=[ 157], | 30.00th=[ 157], 40.00th=[ 506], 50.00th=[ 506], 60.00th=[ 506], | 70.00th=[ 4144], 80.00th=[ 4144], 90.00th=[ 4144], 95.00th=[ 4144], | 99.00th=[ 4144], 99.50th=[ 4144], 99.90th=[ 4144], 99.95th=[ 4144], | 99.99th=[ 4144] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 750=33.33% cpu : usr=0.04%, sys=2.91%, ctx=102, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=3507KiB/s (3591kB/s), 877KiB/s-2833KiB/s (898kB/s-2901kB/s), io=16.0MiB (16.8MB), run=4338-4672msec WRITE: bw=6755KiB/s (6917kB/s), 1753KiB/s-2631KiB/s (1796kB/s-2694kB/s), io=32.0MiB (33.6MB), run=4670-4851msec