== sanity test 398b: DIO and buffer IO race ============== 07:20:11 (1715340011) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.1956 s, 42.1 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=17336: Fri May 10 07:20:27 2024 read: IOPS=111, BW=445KiB/s (456kB/s)(5992KiB/13453msec) clat (usec): min=54, max=174634, avg=4985.74, stdev=5118.77 lat (usec): min=55, max=174635, avg=4986.58, stdev=5118.94 clat percentiles (usec): | 1.00th=[ 60], 5.00th=[ 130], 10.00th=[ 2343], 20.00th=[ 2835], | 30.00th=[ 3294], 40.00th=[ 3851], 50.00th=[ 4490], 60.00th=[ 5211], | 70.00th=[ 5997], 80.00th=[ 6718], 90.00th=[ 8029], 95.00th=[ 9241], | 99.00th=[ 13698], 99.50th=[ 14746], 99.90th=[ 27657], 99.95th=[175113], | 99.99th=[175113] bw ( KiB/s): min= 263, max= 584, per=24.58%, avg=438.31, stdev=74.81, samples=26 iops : min= 65, max= 146, avg=109.50, stdev=18.79, samples=26 write: IOPS=116, BW=468KiB/s (479kB/s)(6296KiB/13453msec) clat (usec): min=470, max=18250, avg=3751.72, stdev=3267.69 lat (usec): min=471, max=18251, avg=3752.97, stdev=3267.44 clat percentiles (usec): | 1.00th=[ 519], 5.00th=[ 545], 10.00th=[ 594], 20.00th=[ 725], | 30.00th=[ 947], 40.00th=[ 2245], 50.00th=[ 3097], 60.00th=[ 3752], | 70.00th=[ 4752], 80.00th=[ 6325], 90.00th=[ 8455], 95.00th=[10290], | 99.00th=[14091], 99.50th=[15270], 99.90th=[17433], 99.95th=[18220], | 99.99th=[18220] bw ( KiB/s): min= 319, max= 608, per=24.53%, avg=458.00, stdev=63.71, samples=26 iops : min= 79, max= 152, avg=114.42, stdev=15.98, samples=26 lat (usec) : 100=1.89%, 250=0.62%, 500=0.20%, 750=10.74%, 1000=5.21% lat (msec) : 2=4.04%, 4=30.11%, 10=42.71%, 20=4.39%, 50=0.07% lat (msec) : 250=0.03% cpu : usr=0.19%, sys=22.82%, ctx=5963, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17338: Fri May 10 07:20:27 2024 read: IOPS=110, BW=443KiB/s (454kB/s)(5940KiB/13407msec) clat (usec): min=47, max=24363, avg=4901.82, stdev=2657.04 lat (usec): min=47, max=24363, avg=4902.47, stdev=2657.04 clat percentiles (usec): | 1.00th=[ 60], 5.00th=[ 1958], 10.00th=[ 2409], 20.00th=[ 2835], | 30.00th=[ 3261], 40.00th=[ 3851], 50.00th=[ 4490], 60.00th=[ 5211], | 70.00th=[ 5932], 80.00th=[ 6783], 90.00th=[ 8029], 95.00th=[ 9372], | 99.00th=[13698], 99.50th=[15926], 99.90th=[21365], 99.95th=[24249], | 99.99th=[24249] bw ( KiB/s): min= 271, max= 600, per=24.74%, avg=441.12, stdev=71.43, samples=26 iops : min= 67, max= 150, avg=110.19, stdev=17.94, samples=26 write: IOPS=118, BW=473KiB/s (485kB/s)(6348KiB/13407msec) clat (usec): min=434, max=174174, avg=3811.48, stdev=5322.56 lat (usec): min=434, max=174175, avg=3812.94, stdev=5322.42 clat percentiles (usec): | 1.00th=[ 506], 5.00th=[ 553], 10.00th=[ 594], 20.00th=[ 758], | 30.00th=[ 1029], 40.00th=[ 2343], 50.00th=[ 3097], 60.00th=[ 3752], | 70.00th=[ 4752], 80.00th=[ 6194], 90.00th=[ 8225], 95.00th=[ 9765], | 99.00th=[ 13960], 99.50th=[ 15664], 99.90th=[ 20579], 99.95th=[175113], | 99.99th=[175113] bw ( KiB/s): min= 343, max= 584, per=24.91%, avg=465.12, stdev=63.11, samples=26 iops : min= 85, max= 146, avg=116.19, stdev=15.88, samples=26 lat (usec) : 50=0.07%, 100=1.50%, 250=0.46%, 500=0.39%, 750=9.93% lat (usec) : 1000=4.98% lat (msec) : 2=4.69%, 4=31.12%, 10=42.74%, 20=3.97%, 50=0.13% lat (msec) : 250=0.03% cpu : usr=0.23%, sys=22.97%, ctx=5879, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17340: Fri May 10 07:20:27 2024 read: IOPS=112, BW=449KiB/s (459kB/s)(6028KiB/13440msec) clat (usec): min=48, max=175442, avg=5021.80, stdev=5116.83 lat (usec): min=49, max=175443, avg=5022.43, stdev=5116.84 clat percentiles (usec): | 1.00th=[ 63], 5.00th=[ 2057], 10.00th=[ 2409], 20.00th=[ 2835], | 30.00th=[ 3294], 40.00th=[ 3851], 50.00th=[ 4490], 60.00th=[ 5080], | 70.00th=[ 5866], 80.00th=[ 6783], 90.00th=[ 8225], 95.00th=[ 9765], | 99.00th=[ 13042], 99.50th=[ 14877], 99.90th=[ 23462], 99.95th=[175113], | 99.99th=[175113] bw ( KiB/s): min= 264, max= 600, per=24.97%, avg=445.23, stdev=63.88, samples=26 iops : min= 66, max= 150, avg=111.31, stdev=15.97, samples=26 write: IOPS=116, BW=466KiB/s (477kB/s)(6260KiB/13440msec) clat (usec): min=468, max=22548, avg=3700.71, stdev=3219.90 lat (usec): min=468, max=22549, avg=3701.47, stdev=3219.90 clat percentiles (usec): | 1.00th=[ 510], 5.00th=[ 553], 10.00th=[ 594], 20.00th=[ 717], | 30.00th=[ 914], 40.00th=[ 1876], 50.00th=[ 3097], 60.00th=[ 3818], | 70.00th=[ 4817], 80.00th=[ 6259], 90.00th=[ 8356], 95.00th=[ 9896], | 99.00th=[13173], 99.50th=[14877], 99.90th=[17433], 99.95th=[22676], | 99.99th=[22676] bw ( KiB/s): min= 304, max= 632, per=24.59%, avg=459.08, stdev=83.97, samples=26 iops : min= 76, max= 158, avg=114.77, stdev=20.99, samples=26 lat (usec) : 50=0.03%, 100=1.63%, 250=0.46%, 500=0.29%, 750=10.68% lat (usec) : 1000=5.63% lat (msec) : 2=4.10%, 4=29.69%, 10=42.90%, 20=4.49%, 50=0.07% lat (msec) : 250=0.03% cpu : usr=0.14%, sys=23.03%, ctx=5933, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17342: Fri May 10 07:20:27 2024 read: IOPS=112, BW=450KiB/s (460kB/s)(6052KiB/13460msec) clat (usec): min=48, max=33713, avg=4837.90, stdev=2723.42 lat (usec): min=48, max=33714, avg=4838.83, stdev=2723.30 clat percentiles (usec): | 1.00th=[ 71], 5.00th=[ 2073], 10.00th=[ 2474], 20.00th=[ 2868], | 30.00th=[ 3261], 40.00th=[ 3720], 50.00th=[ 4293], 60.00th=[ 5014], | 70.00th=[ 5735], 80.00th=[ 6652], 90.00th=[ 8094], 95.00th=[ 9372], | 99.00th=[13435], 99.50th=[16057], 99.90th=[25297], 99.95th=[33817], | 99.99th=[33817] bw ( KiB/s): min= 119, max= 600, per=24.71%, avg=440.50, stdev=86.63, samples=26 iops : min= 29, max= 150, avg=110.04, stdev=21.77, samples=26 write: IOPS=115, BW=463KiB/s (474kB/s)(6236KiB/13460msec) clat (usec): min=457, max=191481, avg=3889.01, stdev=5835.71 lat (usec): min=457, max=191483, avg=3889.78, stdev=5835.72 clat percentiles (usec): | 1.00th=[ 510], 5.00th=[ 553], 10.00th=[ 586], 20.00th=[ 725], | 30.00th=[ 906], 40.00th=[ 2114], 50.00th=[ 3032], 60.00th=[ 3785], | 70.00th=[ 4883], 80.00th=[ 6325], 90.00th=[ 8586], 95.00th=[ 10683], | 99.00th=[ 15270], 99.50th=[ 16581], 99.90th=[ 22152], 99.95th=[191890], | 99.99th=[191890] bw ( KiB/s): min= 159, max= 696, per=24.39%, avg=455.27, stdev=92.60, samples=26 iops : min= 39, max= 174, avg=113.73, stdev=23.23, samples=26 lat (usec) : 50=0.03%, 100=1.63%, 250=0.36%, 500=0.42%, 750=10.29% lat (usec) : 1000=5.70% lat (msec) : 2=3.94%, 4=31.84%, 10=40.69%, 20=4.95%, 50=0.13% lat (msec) : 250=0.03% cpu : usr=0.15%, sys=22.73%, ctx=5949, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1784KiB/s (1827kB/s), 443KiB/s-450KiB/s (454kB/s-460kB/s), io=23.4MiB (24.6MB), run=13407-13460msec WRITE: bw=1868KiB/s (1913kB/s), 463KiB/s-473KiB/s (474kB/s-485kB/s), io=24.6MiB (25.7MB), run=13407-13460msec rand-rw: (groupid=0, jobs=1): err= 0: pid=17335: Fri May 10 07:20:31 2024 read: IOPS=88, BW=355KiB/s (364kB/s)(5992KiB/16857msec) clat (usec): min=1611, max=177588, avg=4493.11, stdev=5632.24 lat (usec): min=1611, max=177592, avg=4493.78, stdev=5632.31 clat percentiles (usec): | 1.00th=[ 1745], 5.00th=[ 1876], 10.00th=[ 1958], 20.00th=[ 2114], | 30.00th=[ 2311], 40.00th=[ 2606], 50.00th=[ 2900], 60.00th=[ 3425], | 70.00th=[ 4359], 80.00th=[ 5932], 90.00th=[ 9372], 95.00th=[ 11994], | 99.00th=[ 17171], 99.50th=[ 20317], 99.90th=[ 23200], 99.95th=[177210], | 99.99th=[177210] bw ( KiB/s): min= 152, max= 752, per=24.62%, avg=348.36, stdev=175.75, samples=33 iops : min= 38, max= 188, avg=86.94, stdev=43.95, samples=33 write: IOPS=93, BW=373KiB/s (382kB/s)(6296KiB/16857msec) clat (usec): min=2236, max=28027, avg=6386.48, stdev=4096.75 lat (usec): min=2237, max=28028, avg=6387.31, stdev=4096.76 clat percentiles (usec): | 1.00th=[ 2474], 5.00th=[ 2835], 10.00th=[ 3032], 20.00th=[ 3294], | 30.00th=[ 3523], 40.00th=[ 3916], 50.00th=[ 4621], 60.00th=[ 5866], | 70.00th=[ 7504], 80.00th=[ 9634], 90.00th=[11994], 95.00th=[14091], | 99.00th=[21365], 99.50th=[24511], 99.90th=[27657], 99.95th=[27919], | 99.99th=[27919] bw ( KiB/s): min= 160, max= 840, per=24.57%, avg=363.88, stdev=177.48, samples=33 iops : min= 40, max= 210, avg=90.82, stdev=44.38, samples=33 lat (msec) : 2=6.25%, 4=47.20%, 10=32.98%, 20=12.66%, 50=0.88% lat (msec) : 250=0.03% cpu : usr=0.19%, sys=11.61%, ctx=3580, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17337: Fri May 10 07:20:31 2024 read: IOPS=87, BW=350KiB/s (359kB/s)(5940KiB/16951msec) clat (usec): min=1592, max=187395, avg=4682.26, stdev=6010.48 lat (usec): min=1593, max=187396, avg=4682.91, stdev=6010.50 clat percentiles (usec): | 1.00th=[ 1713], 5.00th=[ 1827], 10.00th=[ 1926], 20.00th=[ 2089], | 30.00th=[ 2245], 40.00th=[ 2540], 50.00th=[ 2966], 60.00th=[ 3556], | 70.00th=[ 4555], 80.00th=[ 6718], 90.00th=[ 10290], 95.00th=[ 12387], | 99.00th=[ 18482], 99.50th=[ 20579], 99.90th=[ 27919], 99.95th=[187696], | 99.99th=[187696] bw ( KiB/s): min= 104, max= 758, per=24.19%, avg=342.30, stdev=189.63, samples=33 iops : min= 26, max= 189, avg=85.42, stdev=47.41, samples=33 write: IOPS=93, BW=374KiB/s (383kB/s)(6348KiB/16951msec) clat (usec): min=2162, max=35684, avg=6252.42, stdev=3996.52 lat (usec): min=2163, max=35684, avg=6253.25, stdev=3996.65 clat percentiles (usec): | 1.00th=[ 2474], 5.00th=[ 2802], 10.00th=[ 2966], 20.00th=[ 3261], | 30.00th=[ 3523], 40.00th=[ 3884], 50.00th=[ 4555], 60.00th=[ 5604], | 70.00th=[ 7439], 80.00th=[ 9241], 90.00th=[11863], 95.00th=[13698], | 99.00th=[20055], 99.50th=[23725], 99.90th=[27132], 99.95th=[35914], | 99.99th=[35914] bw ( KiB/s): min= 160, max= 752, per=24.39%, avg=361.21, stdev=174.46, samples=33 iops : min= 40, max= 188, avg=90.15, stdev=43.63, samples=33 lat (msec) : 2=7.06%, 4=46.68%, 10=32.81%, 20=12.66%, 50=0.75% lat (msec) : 250=0.03% cpu : usr=0.18%, sys=11.45%, ctx=3601, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17339: Fri May 10 07:20:31 2024 read: IOPS=89, BW=358KiB/s (367kB/s)(6028KiB/16839msec) clat (usec): min=1588, max=29570, avg=4497.99, stdev=3532.60 lat (usec): min=1589, max=29570, avg=4498.72, stdev=3532.61 clat percentiles (usec): | 1.00th=[ 1713], 5.00th=[ 1876], 10.00th=[ 1958], 20.00th=[ 2114], | 30.00th=[ 2278], 40.00th=[ 2638], 50.00th=[ 2999], 60.00th=[ 3523], | 70.00th=[ 4621], 80.00th=[ 6390], 90.00th=[ 9765], 95.00th=[12387], | 99.00th=[17171], 99.50th=[19006], 99.90th=[22938], 99.95th=[29492], | 99.99th=[29492] bw ( KiB/s): min= 112, max= 832, per=24.81%, avg=351.03, stdev=177.71, samples=33 iops : min= 28, max= 208, avg=87.61, stdev=44.44, samples=33 write: IOPS=92, BW=372KiB/s (381kB/s)(6260KiB/16839msec) clat (msec): min=2, max=191, avg= 6.38, stdev= 6.11 lat (msec): min=2, max=191, avg= 6.38, stdev= 6.11 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 4], 20.00th=[ 4], | 30.00th=[ 4], 40.00th=[ 4], 50.00th=[ 5], 60.00th=[ 6], | 70.00th=[ 8], 80.00th=[ 10], 90.00th=[ 12], 95.00th=[ 15], | 99.00th=[ 20], 99.50th=[ 24], 99.90th=[ 29], 99.95th=[ 192], | 99.99th=[ 192] bw ( KiB/s): min= 152, max= 808, per=24.49%, avg=362.67, stdev=179.12, samples=33 iops : min= 38, max= 202, avg=90.52, stdev=44.79, samples=33 lat (msec) : 2=6.12%, 4=47.04%, 10=34.21%, 20=12.01%, 50=0.59% lat (msec) : 250=0.03% cpu : usr=0.16%, sys=11.63%, ctx=3575, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17341: Fri May 10 07:20:31 2024 read: IOPS=89, BW=357KiB/s (365kB/s)(6052KiB/16969msec) clat (usec): min=1588, max=24343, avg=4559.20, stdev=3615.94 lat (usec): min=1589, max=24344, avg=4559.94, stdev=3615.91 clat percentiles (usec): | 1.00th=[ 1729], 5.00th=[ 1860], 10.00th=[ 1942], 20.00th=[ 2114], | 30.00th=[ 2245], 40.00th=[ 2540], 50.00th=[ 2933], 60.00th=[ 3589], | 70.00th=[ 4555], 80.00th=[ 6783], 90.00th=[10290], 95.00th=[12387], | 99.00th=[17171], 99.50th=[19006], 99.90th=[22938], 99.95th=[24249], | 99.99th=[24249] bw ( KiB/s): min= 120, max= 736, per=24.40%, avg=345.21, stdev=176.42, samples=33 iops : min= 30, max= 184, avg=86.15, stdev=44.12, samples=33 write: IOPS=91, BW=367KiB/s (376kB/s)(6236KiB/16969msec) clat (msec): min=2, max=190, avg= 6.41, stdev= 6.16 lat (msec): min=2, max=190, avg= 6.41, stdev= 6.16 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 3], 20.00th=[ 4], | 30.00th=[ 4], 40.00th=[ 4], 50.00th=[ 5], 60.00th=[ 6], | 70.00th=[ 8], 80.00th=[ 10], 90.00th=[ 12], 95.00th=[ 15], | 99.00th=[ 21], 99.50th=[ 26], 99.90th=[ 38], 99.95th=[ 190], | 99.99th=[ 190] bw ( KiB/s): min= 160, max= 760, per=24.01%, avg=355.64, stdev=179.98, samples=33 iops : min= 40, max= 190, avg=88.76, stdev=45.01, samples=33 lat (msec) : 2=6.87%, 4=46.74%, 10=32.42%, 20=13.22%, 50=0.72% lat (msec) : 250=0.03% cpu : usr=0.14%, sys=11.73%, ctx=3617, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1415KiB/s (1449kB/s), 350KiB/s-358KiB/s (359kB/s-367kB/s), io=23.4MiB (24.6MB), run=16839-16969msec WRITE: bw=1482KiB/s (1517kB/s), 367KiB/s-374KiB/s (376kB/s-383kB/s), io=24.6MiB (25.7MB), run=16839-16969msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=17372: Fri May 10 07:20:39 2024 read: IOPS=52, BW=837KiB/s (857kB/s)(5920KiB/7075msec) clat (usec): min=60, max=133242, avg=12454.67, stdev=7455.46 lat (usec): min=61, max=133243, avg=12455.32, stdev=7455.46 clat percentiles (usec): | 1.00th=[ 104], 5.00th=[ 8094], 10.00th=[ 8586], 20.00th=[ 9503], | 30.00th=[ 10290], 40.00th=[ 11207], 50.00th=[ 11600], 60.00th=[ 12387], | 70.00th=[ 13173], 80.00th=[ 14091], 90.00th=[ 15664], 95.00th=[ 18744], | 99.00th=[ 31065], 99.50th=[ 35390], 99.90th=[133694], 99.95th=[133694], | 99.99th=[133694] bw ( KiB/s): min= 480, max= 1120, per=25.07%, avg=831.64, stdev=167.33, samples=14 iops : min= 30, max= 70, avg=51.86, stdev=10.46, samples=14 write: IOPS=56, BW=900KiB/s (922kB/s)(6368KiB/7075msec) clat (usec): min=798, max=90320, avg=6168.74, stdev=8245.96 lat (usec): min=799, max=90321, avg=6171.26, stdev=8245.44 clat percentiles (usec): | 1.00th=[ 816], 5.00th=[ 873], 10.00th=[ 1057], 20.00th=[ 2278], | 30.00th=[ 2638], 40.00th=[ 2999], 50.00th=[ 3621], 60.00th=[ 4817], | 70.00th=[ 6128], 80.00th=[ 8586], 90.00th=[13304], 95.00th=[16712], | 99.00th=[50594], 99.50th=[61604], 99.90th=[90702], 99.95th=[90702], | 99.99th=[90702] bw ( KiB/s): min= 608, max= 1216, per=25.79%, avg=900.21, stdev=201.84, samples=14 iops : min= 38, max= 76, avg=56.14, stdev=12.61, samples=14 lat (usec) : 100=0.39%, 250=0.39%, 1000=4.56% lat (msec) : 2=4.56%, 4=18.88%, 10=28.26%, 20=39.45%, 50=2.86% lat (msec) : 100=0.52%, 250=0.13% cpu : usr=0.13%, sys=21.23%, ctx=2876, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17373: Fri May 10 07:20:39 2024 read: IOPS=51, BW=825KiB/s (845kB/s)(5744KiB/6959msec) clat (usec): min=61, max=49402, avg=11627.08, stdev=4047.14 lat (usec): min=62, max=49403, avg=11629.52, stdev=4048.47 clat percentiles (usec): | 1.00th=[ 77], 5.00th=[ 7635], 10.00th=[ 8291], 20.00th=[ 9241], | 30.00th=[10028], 40.00th=[10683], 50.00th=[11076], 60.00th=[11731], | 70.00th=[12780], 80.00th=[13698], 90.00th=[15533], 95.00th=[17171], | 99.00th=[23462], 99.50th=[35390], 99.90th=[49546], 99.95th=[49546], | 99.99th=[49546] bw ( KiB/s): min= 576, max= 1024, per=24.63%, avg=817.08, stdev=137.54, samples=13 iops : min= 36, max= 64, avg=51.00, stdev= 8.66, samples=13 write: IOPS=58, BW=940KiB/s (963kB/s)(6544KiB/6959msec) clat (usec): min=691, max=193481, avg=6787.83, stdev=12951.59 lat (usec): min=692, max=193482, avg=6788.93, stdev=12951.58 clat percentiles (usec): | 1.00th=[ 791], 5.00th=[ 857], 10.00th=[ 979], 20.00th=[ 1663], | 30.00th=[ 2507], 40.00th=[ 2835], 50.00th=[ 3523], 60.00th=[ 4490], | 70.00th=[ 5800], 80.00th=[ 8225], 90.00th=[ 13960], 95.00th=[ 19006], | 99.00th=[ 53216], 99.50th=[ 71828], 99.90th=[193987], 99.95th=[193987], | 99.99th=[193987] bw ( KiB/s): min= 544, max= 1312, per=26.16%, avg=913.08, stdev=218.22, samples=13 iops : min= 34, max= 82, avg=57.00, stdev=13.71, samples=13 lat (usec) : 100=0.78%, 750=0.26%, 1000=5.34% lat (msec) : 2=5.99%, 4=18.75%, 10=26.95%, 20=38.28%, 50=2.86% lat (msec) : 100=0.65%, 250=0.13% cpu : usr=0.06%, sys=17.46%, ctx=2696, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17374: Fri May 10 07:20:39 2024 read: IOPS=54, BW=867KiB/s (887kB/s)(6256KiB/7219msec) clat (usec): min=61, max=137568, avg=11838.79, stdev=7495.30 lat (usec): min=62, max=137569, avg=11839.45, stdev=7495.29 clat percentiles (usec): | 1.00th=[ 86], 5.00th=[ 7570], 10.00th=[ 8291], 20.00th=[ 9241], | 30.00th=[ 9896], 40.00th=[ 10421], 50.00th=[ 11076], 60.00th=[ 11731], | 70.00th=[ 12518], 80.00th=[ 13304], 90.00th=[ 14877], 95.00th=[ 17171], | 99.00th=[ 25560], 99.50th=[ 50070], 99.90th=[137364], 99.95th=[137364], | 99.99th=[137364] bw ( KiB/s): min= 606, max= 1184, per=25.84%, avg=857.00, stdev=147.27, samples=14 iops : min= 37, max= 74, avg=53.50, stdev= 9.32, samples=14 write: IOPS=52, BW=836KiB/s (856kB/s)(6032KiB/7219msec) clat (usec): min=680, max=88204, avg=6849.80, stdev=9995.25 lat (usec): min=681, max=88205, avg=6852.39, stdev=9994.53 clat percentiles (usec): | 1.00th=[ 758], 5.00th=[ 816], 10.00th=[ 914], 20.00th=[ 1450], | 30.00th=[ 2573], 40.00th=[ 2966], 50.00th=[ 3589], 60.00th=[ 4555], | 70.00th=[ 6259], 80.00th=[ 8586], 90.00th=[15533], 95.00th=[23200], | 99.00th=[47973], 99.50th=[86508], 99.90th=[88605], 99.95th=[88605], | 99.99th=[88605] bw ( KiB/s): min= 384, max= 1216, per=23.12%, avg=806.71, stdev=213.15, samples=14 iops : min= 24, max= 76, avg=50.36, stdev=13.36, samples=14 lat (usec) : 100=0.78%, 750=0.39%, 1000=5.08% lat (msec) : 2=5.34%, 4=15.89%, 10=30.34%, 20=37.50%, 50=4.04% lat (msec) : 100=0.52%, 250=0.13% cpu : usr=0.07%, sys=18.14%, ctx=2835, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17375: Fri May 10 07:20:39 2024 read: IOPS=52, BW=842KiB/s (863kB/s)(6032KiB/7160msec) clat (usec): min=61, max=59600, avg=12201.63, stdev=5090.84 lat (usec): min=62, max=59601, avg=12202.32, stdev=5090.83 clat percentiles (usec): | 1.00th=[ 2933], 5.00th=[ 7963], 10.00th=[ 8586], 20.00th=[ 9372], | 30.00th=[10159], 40.00th=[10814], 50.00th=[11469], 60.00th=[12125], | 70.00th=[12911], 80.00th=[14091], 90.00th=[16057], 95.00th=[18220], | 99.00th=[30802], 99.50th=[56886], 99.90th=[59507], 99.95th=[59507], | 99.99th=[59507] bw ( KiB/s): min= 544, max= 960, per=24.94%, avg=827.29, stdev=120.45, samples=14 iops : min= 34, max= 60, avg=51.64, stdev= 7.60, samples=14 write: IOPS=54, BW=874KiB/s (895kB/s)(6256KiB/7160msec) clat (usec): min=777, max=188284, avg=6526.96, stdev=11742.35 lat (usec): min=777, max=188285, avg=6528.04, stdev=11742.33 clat percentiles (usec): | 1.00th=[ 791], 5.00th=[ 865], 10.00th=[ 1004], 20.00th=[ 2278], | 30.00th=[ 2671], 40.00th=[ 3097], 50.00th=[ 3621], 60.00th=[ 4490], | 70.00th=[ 6063], 80.00th=[ 7898], 90.00th=[ 14484], 95.00th=[ 18220], | 99.00th=[ 47973], 99.50th=[ 62653], 99.90th=[187696], 99.95th=[187696], | 99.99th=[187696] bw ( KiB/s): min= 448, max= 1184, per=24.95%, avg=870.71, stdev=177.56, samples=14 iops : min= 28, max= 74, avg=54.36, stdev=11.09, samples=14 lat (usec) : 100=0.26%, 250=0.13%, 1000=5.08% lat (msec) : 2=4.30%, 4=18.49%, 10=29.17%, 20=39.19%, 50=2.86% lat (msec) : 100=0.39%, 250=0.13% cpu : usr=0.13%, sys=20.31%, ctx=2888, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=3318KiB/s (3398kB/s), 825KiB/s-867KiB/s (845kB/s-887kB/s), io=23.4MiB (24.5MB), run=6959-7219msec WRITE: bw=3491KiB/s (3575kB/s), 836KiB/s-940KiB/s (856kB/s-963kB/s), io=24.6MiB (25.8MB), run=6959-7219msec rand-rw: (groupid=0, jobs=1): err= 0: pid=17377: Fri May 10 07:20:39 2024 read: IOPS=52, BW=841KiB/s (861kB/s)(5920KiB/7039msec) clat (usec): min=1722, max=61147, avg=7068.35, stdev=6309.53 lat (usec): min=1723, max=61147, avg=7069.10, stdev=6309.51 clat percentiles (usec): | 1.00th=[ 2057], 5.00th=[ 2343], 10.00th=[ 2474], 20.00th=[ 2802], | 30.00th=[ 3163], 40.00th=[ 3752], 50.00th=[ 4817], 60.00th=[ 6456], | 70.00th=[ 8356], 80.00th=[10683], 90.00th=[13173], 95.00th=[15926], | 99.00th=[30016], 99.50th=[46400], 99.90th=[61080], 99.95th=[61080], | 99.99th=[61080] bw ( KiB/s): min= 448, max= 1216, per=25.31%, avg=841.14, stdev=190.94, samples=14 iops : min= 28, max= 76, avg=52.57, stdev=11.93, samples=14 write: IOPS=56, BW=905KiB/s (926kB/s)(6368KiB/7039msec) clat (msec): min=2, max=176, avg=11.09, stdev=12.59 lat (msec): min=2, max=176, avg=11.10, stdev=12.59 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 4], 10.00th=[ 4], 20.00th=[ 5], | 30.00th=[ 7], 40.00th=[ 8], 50.00th=[ 9], 60.00th=[ 11], | 70.00th=[ 12], 80.00th=[ 14], 90.00th=[ 18], 95.00th=[ 22], | 99.00th=[ 82], 99.50th=[ 95], 99.90th=[ 178], 99.95th=[ 178], | 99.99th=[ 178] bw ( KiB/s): min= 608, max= 1216, per=25.82%, avg=902.86, stdev=198.92, samples=14 iops : min= 38, max= 76, avg=56.43, stdev=12.43, samples=14 lat (msec) : 2=0.26%, 4=25.52%, 10=40.89%, 20=28.12%, 50=4.43% lat (msec) : 100=0.65%, 250=0.13% cpu : usr=0.09%, sys=9.32%, ctx=1239, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17378: Fri May 10 07:20:39 2024 read: IOPS=51, BW=827KiB/s (847kB/s)(5744KiB/6946msec) clat (msec): min=2, max=141, avg= 7.08, stdev= 9.39 lat (msec): min=2, max=141, avg= 7.08, stdev= 9.39 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 3], 20.00th=[ 3], | 30.00th=[ 4], 40.00th=[ 4], 50.00th=[ 5], 60.00th=[ 6], | 70.00th=[ 8], 80.00th=[ 11], 90.00th=[ 13], 95.00th=[ 17], | 99.00th=[ 46], 99.50th=[ 55], 99.90th=[ 142], 99.95th=[ 142], | 99.99th=[ 142] bw ( KiB/s): min= 512, max= 1152, per=24.59%, avg=817.23, stdev=167.10, samples=13 iops : min= 32, max= 72, avg=51.08, stdev=10.44, samples=13 write: IOPS=58, BW=942KiB/s (965kB/s)(6544KiB/6946msec) clat (usec): min=2875, max=95835, avg=10750.73, stdev=9696.06 lat (usec): min=2876, max=95836, avg=10751.96, stdev=9696.03 clat percentiles (usec): | 1.00th=[ 3032], 5.00th=[ 3654], 10.00th=[ 4015], 20.00th=[ 5276], | 30.00th=[ 6325], 40.00th=[ 7177], 50.00th=[ 8160], 60.00th=[ 9372], | 70.00th=[10945], 80.00th=[14091], 90.00th=[18744], 95.00th=[23725], | 99.00th=[56361], 99.50th=[62653], 99.90th=[95945], 99.95th=[95945], | 99.99th=[95945] bw ( KiB/s): min= 608, max= 1248, per=26.11%, avg=913.23, stdev=189.14, samples=13 iops : min= 38, max= 78, avg=57.08, stdev=11.82, samples=13 lat (msec) : 4=26.04%, 10=45.31%, 20=22.92%, 50=4.43%, 100=1.17% lat (msec) : 250=0.13% cpu : usr=0.03%, sys=9.34%, ctx=1187, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17379: Fri May 10 07:20:39 2024 read: IOPS=54, BW=868KiB/s (889kB/s)(6256KiB/7205msec) clat (usec): min=1892, max=58805, avg=7180.20, stdev=6045.72 lat (usec): min=1893, max=58805, avg=7181.06, stdev=6045.84 clat percentiles (usec): | 1.00th=[ 2024], 5.00th=[ 2278], 10.00th=[ 2507], 20.00th=[ 2933], | 30.00th=[ 3359], 40.00th=[ 3982], 50.00th=[ 5080], 60.00th=[ 7046], | 70.00th=[ 8717], 80.00th=[10552], 90.00th=[13435], 95.00th=[17433], | 99.00th=[26608], 99.50th=[50070], 99.90th=[58983], 99.95th=[58983], | 99.99th=[58983] bw ( KiB/s): min= 672, max= 1216, per=25.99%, avg=864.00, stdev=152.69, samples=14 iops : min= 42, max= 76, avg=54.00, stdev= 9.54, samples=14 write: IOPS=52, BW=837KiB/s (857kB/s)(6032KiB/7205msec) clat (msec): min=2, max=195, avg=11.64, stdev=13.13 lat (msec): min=2, max=195, avg=11.64, stdev=13.13 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 4], 10.00th=[ 4], 20.00th=[ 6], | 30.00th=[ 7], 40.00th=[ 8], 50.00th=[ 9], 60.00th=[ 11], | 70.00th=[ 13], 80.00th=[ 16], 90.00th=[ 21], 95.00th=[ 26], | 99.00th=[ 54], 99.50th=[ 89], 99.90th=[ 197], 99.95th=[ 197], | 99.99th=[ 197] bw ( KiB/s): min= 352, max= 1152, per=23.07%, avg=806.86, stdev=206.69, samples=14 iops : min= 22, max= 72, avg=50.43, stdev=12.92, samples=14 lat (msec) : 2=0.26%, 4=25.65%, 10=42.45%, 20=24.87%, 50=5.86% lat (msec) : 100=0.78%, 250=0.13% cpu : usr=0.07%, sys=9.29%, ctx=1265, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17380: Fri May 10 07:20:39 2024 read: IOPS=52, BW=847KiB/s (868kB/s)(6032KiB/7120msec) clat (usec): min=1888, max=62032, avg=7579.46, stdev=7784.83 lat (usec): min=1888, max=62032, avg=7580.12, stdev=7784.84 clat percentiles (usec): | 1.00th=[ 1958], 5.00th=[ 2278], 10.00th=[ 2442], 20.00th=[ 2802], | 30.00th=[ 3195], 40.00th=[ 3785], 50.00th=[ 4752], 60.00th=[ 6521], | 70.00th=[ 8848], 80.00th=[10945], 90.00th=[14877], 95.00th=[19006], | 99.00th=[55313], 99.50th=[60556], 99.90th=[62129], 99.95th=[62129], | 99.99th=[62129] bw ( KiB/s): min= 512, max= 1024, per=24.81%, avg=824.71, stdev=143.19, samples=14 iops : min= 32, max= 64, avg=51.36, stdev= 8.95, samples=14 write: IOPS=54, BW=879KiB/s (900kB/s)(6256KiB/7120msec) clat (msec): min=2, max=201, avg=10.88, stdev=11.98 lat (msec): min=2, max=201, avg=10.88, stdev=11.98 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 4], 10.00th=[ 5], 20.00th=[ 6], | 30.00th=[ 7], 40.00th=[ 8], 50.00th=[ 10], 60.00th=[ 11], | 70.00th=[ 12], 80.00th=[ 14], 90.00th=[ 19], 95.00th=[ 22], | 99.00th=[ 48], 99.50th=[ 77], 99.90th=[ 201], 99.95th=[ 201], | 99.99th=[ 201] bw ( KiB/s): min= 448, max= 1117, per=24.89%, avg=870.36, stdev=160.33, samples=14 iops : min= 28, max= 69, avg=54.21, stdev= 9.87, samples=14 lat (msec) : 2=0.52%, 4=24.22%, 10=43.23%, 20=26.56%, 50=4.43% lat (msec) : 100=0.91%, 250=0.13% cpu : usr=0.10%, sys=9.49%, ctx=1261, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=3324KiB/s (3404kB/s), 827KiB/s-868KiB/s (847kB/s-889kB/s), io=23.4MiB (24.5MB), run=6946-7205msec WRITE: bw=3498KiB/s (3582kB/s), 837KiB/s-942KiB/s (857kB/s-965kB/s), io=24.6MiB (25.8MB), run=6946-7205msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=17393: Fri May 10 07:20:42 2024 read: IOPS=3, BW=4054KiB/s (4152kB/s)(7168KiB/1768msec) clat (usec): min=349, max=715062, avg=226994.91, stdev=335409.65 lat (usec): min=350, max=715064, avg=226995.97, stdev=335410.00 clat percentiles (usec): | 1.00th=[ 351], 5.00th=[ 351], 10.00th=[ 351], 20.00th=[ 416], | 30.00th=[ 429], 40.00th=[ 429], 50.00th=[ 41157], 60.00th=[119014], | 70.00th=[119014], 80.00th=[717226], 90.00th=[717226], 95.00th=[717226], | 99.00th=[717226], 99.50th=[717226], 99.90th=[717226], 99.95th=[717226], | 99.99th=[717226] bw ( KiB/s): min= 2048, max= 2048, per=22.98%, avg=2048.00, stdev= 0.00, samples=3 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=3 write: IOPS=2, BW=2896KiB/s (2965kB/s)(5120KiB/1768msec) clat (usec): min=17465, max=55167, avg=34920.45, stdev=17399.97 lat (usec): min=17483, max=55342, avg=34988.39, stdev=17436.56 clat percentiles (usec): | 1.00th=[17433], 5.00th=[17433], 10.00th=[17433], 20.00th=[17433], | 30.00th=[18220], 40.00th=[18220], 50.00th=[33817], 60.00th=[33817], | 70.00th=[49546], 80.00th=[49546], 90.00th=[55313], 95.00th=[55313], | 99.00th=[55313], 99.50th=[55313], 99.90th=[55313], 99.95th=[55313], | 99.99th=[55313] lat (usec) : 500=25.00% lat (msec) : 20=16.67%, 50=25.00%, 100=8.33%, 250=8.33%, 750=16.67% cpu : usr=0.06%, sys=38.37%, ctx=593, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17395: Fri May 10 07:20:42 2024 read: IOPS=0, BW=891KiB/s (913kB/s)(2048KiB/2298msec) clat (msec): min=39, max=270, avg=154.94, stdev=163.31 lat (msec): min=39, max=270, avg=154.94, stdev=163.31 clat percentiles (msec): | 1.00th=[ 40], 5.00th=[ 40], 10.00th=[ 40], 20.00th=[ 40], | 30.00th=[ 40], 40.00th=[ 40], 50.00th=[ 40], 60.00th=[ 271], | 70.00th=[ 271], 80.00th=[ 271], 90.00th=[ 271], 95.00th=[ 271], | 99.00th=[ 271], 99.50th=[ 271], 99.90th=[ 271], 99.95th=[ 271], | 99.99th=[ 271] bw ( KiB/s): min= 2000, max= 2048, per=22.71%, avg=2024.00, stdev=33.94, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 write: IOPS=4, BW=4456KiB/s (4563kB/s)(10.0MiB/2298msec) clat (msec): min=28, max=841, avg=197.95, stdev=305.75 lat (msec): min=28, max=841, avg=198.02, stdev=305.74 clat percentiles (msec): | 1.00th=[ 29], 5.00th=[ 29], 10.00th=[ 29], 20.00th=[ 33], | 30.00th=[ 36], 40.00th=[ 41], 50.00th=[ 48], 60.00th=[ 50], | 70.00th=[ 148], 80.00th=[ 148], 90.00th=[ 701], 95.00th=[ 844], | 99.00th=[ 844], 99.50th=[ 844], 99.90th=[ 844], 99.95th=[ 844], | 99.99th=[ 844] bw ( KiB/s): min= 2000, max= 8000, per=43.13%, avg=5381.33, stdev=3071.85, samples=3 iops : min= 1, max= 7, avg= 4.67, stdev= 3.21, samples=3 lat (msec) : 50=58.33%, 100=8.33%, 250=8.33%, 500=8.33%, 750=8.33% lat (msec) : 1000=8.33% cpu : usr=0.04%, sys=14.93%, ctx=189, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17397: Fri May 10 07:20:42 2024 read: IOPS=4, BW=4208KiB/s (4309kB/s)(6144KiB/1460msec) clat (msec): min=4, max=136, avg=60.02, stdev=49.00 lat (msec): min=4, max=136, avg=60.02, stdev=49.00 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 5], 10.00th=[ 5], 20.00th=[ 10], | 30.00th=[ 10], 40.00th=[ 61], 50.00th=[ 61], 60.00th=[ 68], | 70.00th=[ 83], 80.00th=[ 83], 90.00th=[ 136], 95.00th=[ 136], | 99.00th=[ 136], 99.50th=[ 136], 99.90th=[ 136], 99.95th=[ 136], | 99.99th=[ 136] write: IOPS=4, BW=4208KiB/s (4309kB/s)(6144KiB/1460msec) clat (msec): min=34, max=823, avg=183.13, stdev=314.37 lat (msec): min=34, max=823, avg=183.18, stdev=314.38 clat percentiles (msec): | 1.00th=[ 35], 5.00th=[ 35], 10.00th=[ 35], 20.00th=[ 43], | 30.00th=[ 43], 40.00th=[ 44], 50.00th=[ 44], 60.00th=[ 58], | 70.00th=[ 99], 80.00th=[ 99], 90.00th=[ 827], 95.00th=[ 827], | 99.00th=[ 827], 99.50th=[ 827], 99.90th=[ 827], 99.95th=[ 827], | 99.99th=[ 827] bw ( KiB/s): min= 2043, max= 4096, per=24.60%, avg=3069.50, stdev=1451.69, samples=2 iops : min= 1, max= 4, avg= 2.50, stdev= 2.12, samples=2 lat (msec) : 10=16.67%, 50=25.00%, 100=41.67%, 250=8.33%, 1000=8.33% cpu : usr=0.00%, sys=23.10%, ctx=151, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17399: Fri May 10 07:20:42 2024 read: IOPS=2, BW=2465KiB/s (2524kB/s)(5120KiB/2077msec) clat (usec): min=434, max=577413, avg=161592.68, stdev=236153.10 lat (usec): min=435, max=577414, avg=161593.66, stdev=236152.95 clat percentiles (usec): | 1.00th=[ 437], 5.00th=[ 437], 10.00th=[ 437], 20.00th=[ 437], | 30.00th=[ 35390], 40.00th=[ 35390], 50.00th=[ 95945], 60.00th=[ 95945], | 70.00th=[ 98042], 80.00th=[ 98042], 90.00th=[574620], 95.00th=[574620], | 99.00th=[574620], 99.50th=[574620], 99.90th=[574620], 99.95th=[574620], | 99.99th=[574620] bw ( KiB/s): min= 8192, max= 8192, per=91.92%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=3, BW=3451KiB/s (3534kB/s)(7168KiB/2077msec) clat (msec): min=20, max=808, avg=180.47, stdev=282.04 lat (msec): min=20, max=809, avg=180.52, stdev=282.05 clat percentiles (msec): | 1.00th=[ 21], 5.00th=[ 21], 10.00th=[ 21], 20.00th=[ 21], | 30.00th=[ 72], 40.00th=[ 72], 50.00th=[ 80], 60.00th=[ 87], | 70.00th=[ 87], 80.00th=[ 178], 90.00th=[ 810], 95.00th=[ 810], | 99.00th=[ 810], 99.50th=[ 810], 99.90th=[ 810], 99.95th=[ 810], | 99.99th=[ 810] bw ( KiB/s): min= 2043, max= 8192, per=38.22%, avg=4768.67, stdev=3133.31, samples=3 iops : min= 1, max= 8, avg= 4.00, stdev= 3.61, samples=3 lat (usec) : 500=8.33% lat (msec) : 50=25.00%, 100=41.67%, 250=8.33%, 750=8.33%, 1000=8.33% cpu : usr=0.00%, sys=21.05%, ctx=447, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=8912KiB/s (9126kB/s), 891KiB/s-4208KiB/s (913kB/s-4309kB/s), io=20.0MiB (20.0MB), run=1460-2298msec WRITE: bw=12.2MiB/s (12.8MB/s), 2896KiB/s-4456KiB/s (2965kB/s-4563kB/s), io=28.0MiB (29.4MB), run=1460-2298msec rand-rw: (groupid=0, jobs=1): err= 0: pid=17392: Fri May 10 07:20:42 2024 read: IOPS=2, BW=2923KiB/s (2993kB/s)(7168KiB/2452msec) clat (msec): min=18, max=856, avg=308.08, stdev=359.70 lat (msec): min=18, max=856, avg=308.08, stdev=359.70 clat percentiles (msec): | 1.00th=[ 19], 5.00th=[ 19], 10.00th=[ 19], 20.00th=[ 20], | 30.00th=[ 32], 40.00th=[ 32], 50.00th=[ 48], 60.00th=[ 542], | 70.00th=[ 542], 80.00th=[ 642], 90.00th=[ 860], 95.00th=[ 860], | 99.00th=[ 860], 99.50th=[ 860], 99.90th=[ 860], 99.95th=[ 860], | 99.99th=[ 860] bw ( KiB/s): min= 2043, max= 2048, per=24.50%, avg=2046.33, stdev= 2.89, samples=3 iops : min= 1, max= 2, avg= 1.67, stdev= 0.58, samples=3 write: IOPS=2, BW=2088KiB/s (2138kB/s)(5120KiB/2452msec) clat (msec): min=26, max=124, avg=56.82, stdev=39.49 lat (msec): min=26, max=124, avg=56.88, stdev=39.51 clat percentiles (msec): | 1.00th=[ 27], 5.00th=[ 27], 10.00th=[ 27], 20.00th=[ 27], | 30.00th=[ 32], 40.00th=[ 32], 50.00th=[ 49], 60.00th=[ 49], | 70.00th=[ 54], 80.00th=[ 54], 90.00th=[ 125], 95.00th=[ 125], | 99.00th=[ 125], 99.50th=[ 125], 99.90th=[ 125], 99.95th=[ 125], | 99.99th=[ 125] bw ( KiB/s): min= 2048, max= 2048, per=17.51%, avg=2048.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 20=16.67%, 50=41.67%, 100=8.33%, 250=8.33%, 750=16.67% lat (msec) : 1000=8.33% cpu : usr=0.00%, sys=5.88%, ctx=55, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17394: Fri May 10 07:20:42 2024 read: IOPS=0, BW=895KiB/s (917kB/s)(2048KiB/2287msec) clat (usec): min=15847, max=63501, avg=39674.30, stdev=33696.67 lat (usec): min=15848, max=63502, avg=39675.33, stdev=33696.84 clat percentiles (usec): | 1.00th=[15795], 5.00th=[15795], 10.00th=[15795], 20.00th=[15795], | 30.00th=[15795], 40.00th=[15795], 50.00th=[15795], 60.00th=[63701], | 70.00th=[63701], 80.00th=[63701], 90.00th=[63701], 95.00th=[63701], | 99.00th=[63701], 99.50th=[63701], 99.90th=[63701], 99.95th=[63701], | 99.99th=[63701] bw ( KiB/s): min= 2048, max= 2048, per=24.52%, avg=2048.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 write: IOPS=4, BW=4477KiB/s (4585kB/s)(10.0MiB/2287msec) clat (msec): min=18, max=1056, avg=219.04, stdev=386.52 lat (msec): min=19, max=1056, avg=219.11, stdev=386.52 clat percentiles (msec): | 1.00th=[ 20], 5.00th=[ 20], 10.00th=[ 20], 20.00th=[ 22], | 30.00th=[ 26], 40.00th=[ 26], 50.00th=[ 34], 60.00th=[ 39], | 70.00th=[ 91], 80.00th=[ 91], 90.00th=[ 835], 95.00th=[ 1053], | 99.00th=[ 1053], 99.50th=[ 1053], 99.90th=[ 1053], 99.95th=[ 1053], | 99.99th=[ 1053] bw ( KiB/s): min= 6144, max=10240, per=70.06%, avg=8192.00, stdev=2896.31, samples=2 iops : min= 6, max= 10, avg= 8.00, stdev= 2.83, samples=2 lat (msec) : 20=16.67%, 50=50.00%, 100=16.67%, 1000=8.33% cpu : usr=0.00%, sys=4.20%, ctx=32, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17396: Fri May 10 07:20:42 2024 read: IOPS=2, BW=2770KiB/s (2837kB/s)(6144KiB/2218msec) clat (msec): min=20, max=896, avg=192.45, stdev=345.84 lat (msec): min=20, max=896, avg=192.45, stdev=345.84 clat percentiles (msec): | 1.00th=[ 21], 5.00th=[ 21], 10.00th=[ 21], 20.00th=[ 22], | 30.00th=[ 22], 40.00th=[ 67], 50.00th=[ 67], 60.00th=[ 67], | 70.00th=[ 84], 80.00th=[ 84], 90.00th=[ 894], 95.00th=[ 894], | 99.00th=[ 894], 99.50th=[ 894], 99.90th=[ 894], 99.95th=[ 894], | 99.99th=[ 894] bw ( KiB/s): min= 2043, max= 8192, per=61.27%, avg=5117.50, stdev=4348.00, samples=2 iops : min= 1, max= 8, avg= 4.50, stdev= 4.95, samples=2 write: IOPS=2, BW=2770KiB/s (2837kB/s)(6144KiB/2218msec) clat (msec): min=25, max=816, avg=174.48, stdev=315.21 lat (msec): min=25, max=816, avg=174.54, stdev=315.21 clat percentiles (msec): | 1.00th=[ 26], 5.00th=[ 26], 10.00th=[ 26], 20.00th=[ 31], | 30.00th=[ 31], 40.00th=[ 46], 50.00th=[ 46], 60.00th=[ 58], | 70.00th=[ 72], 80.00th=[ 72], 90.00th=[ 818], 95.00th=[ 818], | 99.00th=[ 818], 99.50th=[ 818], 99.90th=[ 818], 99.95th=[ 818], | 99.99th=[ 818] bw ( KiB/s): min= 2043, max= 6144, per=35.02%, avg=4094.33, stdev=2050.50, samples=3 iops : min= 1, max= 6, avg= 3.67, stdev= 2.52, samples=3 lat (msec) : 50=41.67%, 100=41.67%, 1000=16.67% cpu : usr=0.00%, sys=4.92%, ctx=66, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17398: Fri May 10 07:20:42 2024 read: IOPS=2, BW=2286KiB/s (2341kB/s)(5120KiB/2240msec) clat (msec): min=21, max=911, avg=219.82, stdev=387.95 lat (msec): min=21, max=911, avg=219.82, stdev=387.95 clat percentiles (msec): | 1.00th=[ 22], 5.00th=[ 22], 10.00th=[ 22], 20.00th=[ 22], | 30.00th=[ 30], 40.00th=[ 30], 50.00th=[ 42], 60.00th=[ 42], | 70.00th=[ 94], 80.00th=[ 94], 90.00th=[ 911], 95.00th=[ 911], | 99.00th=[ 911], 99.50th=[ 911], 99.90th=[ 911], 99.95th=[ 911], | 99.99th=[ 911] bw ( KiB/s): min= 8192, max= 8192, per=98.08%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=3, BW=3200KiB/s (3277kB/s)(7168KiB/2240msec) clat (msec): min=17, max=842, avg=162.64, stdev=301.14 lat (msec): min=17, max=842, avg=162.69, stdev=301.15 clat percentiles (msec): | 1.00th=[ 18], 5.00th=[ 18], 10.00th=[ 18], 20.00th=[ 20], | 30.00th=[ 23], 40.00th=[ 23], 50.00th=[ 71], 60.00th=[ 77], | 70.00th=[ 77], 80.00th=[ 91], 90.00th=[ 844], 95.00th=[ 844], | 99.00th=[ 844], 99.50th=[ 844], 99.90th=[ 844], 99.95th=[ 844], | 99.99th=[ 844] bw ( KiB/s): min= 2048, max= 8192, per=40.87%, avg=4778.67, stdev=3128.37, samples=3 iops : min= 2, max= 8, avg= 4.67, stdev= 3.06, samples=3 lat (msec) : 20=16.67%, 50=33.33%, 100=33.33%, 1000=16.67% cpu : usr=0.00%, sys=4.56%, ctx=80, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=8352KiB/s (8553kB/s), 895KiB/s-2923KiB/s (917kB/s-2993kB/s), io=20.0MiB (20.0MB), run=2218-2452msec WRITE: bw=11.4MiB/s (11.0MB/s), 2088KiB/s-4477KiB/s (2138kB/s-4585kB/s), io=28.0MiB (29.4MB), run=2218-2452msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=17411: Fri May 10 07:20:46 2024 read: IOPS=1, BW=5469KiB/s (5600kB/s)(12.0MiB/2247msec) clat (msec): min=158, max=1410, avg=748.51, stdev=628.96 lat (msec): min=158, max=1410, avg=748.51, stdev=628.96 clat percentiles (msec): | 1.00th=[ 159], 5.00th=[ 159], 10.00th=[ 159], 20.00th=[ 159], | 30.00th=[ 159], 40.00th=[ 676], 50.00th=[ 676], 60.00th=[ 676], | 70.00th=[ 1418], 80.00th=[ 1418], 90.00th=[ 1418], 95.00th=[ 1418], | 99.00th=[ 1418], 99.50th=[ 1418], 99.90th=[ 1418], 99.95th=[ 1418], | 99.99th=[ 1418] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 lat (msec) : 250=33.33%, 750=33.33% cpu : usr=0.00%, sys=49.51%, ctx=887, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17412: Fri May 10 07:20:46 2024 read: IOPS=0, BW=1709KiB/s (1750kB/s)(4096KiB/2397msec) clat (nsec): min=148730k, max=148730k, avg=148730044.00, stdev= 0.00 lat (nsec): min=148732k, max=148732k, avg=148731686.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 148], 5.00th=[ 148], 10.00th=[ 148], 20.00th=[ 148], | 30.00th=[ 148], 40.00th=[ 148], 50.00th=[ 148], 60.00th=[ 148], | 70.00th=[ 148], 80.00th=[ 148], 90.00th=[ 148], 95.00th=[ 148], | 99.00th=[ 148], 99.50th=[ 148], 99.90th=[ 148], 99.95th=[ 148], | 99.99th=[ 148] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=0, BW=3418KiB/s (3500kB/s)(8192KiB/2397msec) clat (msec): min=112, max=2131, avg=1122.14, stdev=1427.80 lat (msec): min=112, max=2131, avg=1122.39, stdev=1427.79 clat percentiles (msec): | 1.00th=[ 113], 5.00th=[ 113], 10.00th=[ 113], 20.00th=[ 113], | 30.00th=[ 113], 40.00th=[ 113], 50.00th=[ 113], 60.00th=[ 2140], | 70.00th=[ 2140], 80.00th=[ 2140], 90.00th=[ 2140], 95.00th=[ 2140], | 99.00th=[ 2140], 99.50th=[ 2140], 99.90th=[ 2140], 99.95th=[ 2140], | 99.99th=[ 2140] lat (msec) : 250=66.67% cpu : usr=0.00%, sys=11.06%, ctx=136, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17413: Fri May 10 07:20:46 2024 write: IOPS=1, BW=4925KiB/s (5043kB/s)(12.0MiB/2495msec) clat (msec): min=348, max=1751, avg=831.10, stdev=797.45 lat (msec): min=348, max=1751, avg=831.42, stdev=797.53 clat percentiles (msec): | 1.00th=[ 351], 5.00th=[ 351], 10.00th=[ 351], 20.00th=[ 351], | 30.00th=[ 351], 40.00th=[ 393], 50.00th=[ 393], 60.00th=[ 393], | 70.00th=[ 1754], 80.00th=[ 1754], 90.00th=[ 1754], 95.00th=[ 1754], | 99.00th=[ 1754], 99.50th=[ 1754], 99.90th=[ 1754], 99.95th=[ 1754], | 99.99th=[ 1754] bw ( KiB/s): min= 8192, max= 8192, per=62.38%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 500=66.67% cpu : usr=0.08%, sys=10.43%, ctx=19, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17414: Fri May 10 07:20:46 2024 write: IOPS=1, BW=5729KiB/s (5866kB/s)(12.0MiB/2145msec) clat (msec): min=172, max=1750, avg=714.12, stdev=897.54 lat (msec): min=172, max=1750, avg=714.36, stdev=897.56 clat percentiles (msec): | 1.00th=[ 174], 5.00th=[ 174], 10.00th=[ 174], 20.00th=[ 174], | 30.00th=[ 174], 40.00th=[ 220], 50.00th=[ 220], 60.00th=[ 220], | 70.00th=[ 1754], 80.00th=[ 1754], 90.00th=[ 1754], 95.00th=[ 1754], | 99.00th=[ 1754], 99.50th=[ 1754], 99.90th=[ 1754], 99.95th=[ 1754], | 99.99th=[ 1754] bw ( KiB/s): min= 8175, max= 8192, per=62.31%, avg=8183.50, stdev=12.02, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 250=66.67% cpu : usr=0.00%, sys=13.57%, ctx=37, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=6835KiB/s (6999kB/s), 1709KiB/s-5469KiB/s (1750kB/s-5600kB/s), io=16.0MiB (16.8MB), run=2247-2397msec WRITE: bw=12.8MiB/s (13.4MB/s), 3418KiB/s-5729KiB/s (3500kB/s-5866kB/s), io=32.0MiB (33.6MB), run=2145-2495msec rand-rw: (groupid=0, jobs=1): err= 0: pid=17407: Fri May 10 07:20:46 2024 read: IOPS=1, BW=4828KiB/s (4944kB/s)(12.0MiB/2545msec) clat (msec): min=294, max=1829, avg=847.33, stdev=852.84 lat (msec): min=294, max=1829, avg=847.33, stdev=852.84 clat percentiles (msec): | 1.00th=[ 296], 5.00th=[ 296], 10.00th=[ 296], 20.00th=[ 296], | 30.00th=[ 296], 40.00th=[ 418], 50.00th=[ 418], 60.00th=[ 418], | 70.00th=[ 1838], 80.00th=[ 1838], 90.00th=[ 1838], 95.00th=[ 1838], | 99.00th=[ 1838], 99.50th=[ 1838], 99.90th=[ 1838], 99.95th=[ 1838], | 99.99th=[ 1838] bw ( KiB/s): min= 5483, max= 5483, per=86.48%, avg=5483.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 500=66.67% cpu : usr=0.00%, sys=3.66%, ctx=48, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17408: Fri May 10 07:20:46 2024 read: IOPS=0, BW=1585KiB/s (1623kB/s)(4096KiB/2584msec) clat (nsec): min=201134k, max=201134k, avg=201134011.00, stdev= 0.00 lat (nsec): min=201135k, max=201135k, avg=201135101.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 201], 5.00th=[ 201], 10.00th=[ 201], 20.00th=[ 201], | 30.00th=[ 201], 40.00th=[ 201], 50.00th=[ 201], 60.00th=[ 201], | 70.00th=[ 201], 80.00th=[ 201], 90.00th=[ 201], 95.00th=[ 201], | 99.00th=[ 201], 99.50th=[ 201], 99.90th=[ 201], 99.95th=[ 201], | 99.99th=[ 201] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=0, BW=3170KiB/s (3246kB/s)(8192KiB/2584msec) clat (msec): min=160, max=2202, avg=1181.54, stdev=1443.98 lat (msec): min=160, max=2202, avg=1181.80, stdev=1443.96 clat percentiles (msec): | 1.00th=[ 161], 5.00th=[ 161], 10.00th=[ 161], 20.00th=[ 161], | 30.00th=[ 161], 40.00th=[ 161], 50.00th=[ 161], 60.00th=[ 2198], | 70.00th=[ 2198], 80.00th=[ 2198], 90.00th=[ 2198], 95.00th=[ 2198], | 99.00th=[ 2198], 99.50th=[ 2198], 99.90th=[ 2198], 99.95th=[ 2198], | 99.99th=[ 2198] bw ( KiB/s): min= 7393, max= 7393, per=58.30%, avg=7393.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67% cpu : usr=0.00%, sys=5.42%, ctx=62, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17409: Fri May 10 07:20:46 2024 write: IOPS=6, BW=25.5MiB/s (26.7MB/s)(12.0MiB/471msec) clat (msec): min=62, max=320, avg=154.51, stdev=144.24 lat (msec): min=62, max=322, avg=155.18, stdev=144.96 clat percentiles (msec): | 1.00th=[ 63], 5.00th=[ 63], 10.00th=[ 63], 20.00th=[ 63], | 30.00th=[ 63], 40.00th=[ 81], 50.00th=[ 81], 60.00th=[ 81], | 70.00th=[ 321], 80.00th=[ 321], 90.00th=[ 321], 95.00th=[ 321], | 99.00th=[ 321], 99.50th=[ 321], 99.90th=[ 321], 99.95th=[ 321], | 99.99th=[ 321] lat (msec) : 100=66.67%, 500=33.33% cpu : usr=0.21%, sys=27.23%, ctx=68, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=17410: Fri May 10 07:20:46 2024 write: IOPS=1, BW=6865KiB/s (7030kB/s)(12.0MiB/1790msec) clat (msec): min=128, max=1325, avg=595.98, stdev=640.34 lat (msec): min=128, max=1326, avg=596.21, stdev=640.36 clat percentiles (msec): | 1.00th=[ 129], 5.00th=[ 129], 10.00th=[ 129], 20.00th=[ 129], | 30.00th=[ 129], 40.00th=[ 334], 50.00th=[ 334], 60.00th=[ 334], | 70.00th=[ 1334], 80.00th=[ 1334], 90.00th=[ 1334], 95.00th=[ 1334], | 99.00th=[ 1334], 99.50th=[ 1334], 99.90th=[ 1334], 99.95th=[ 1334], | 99.99th=[ 1334] bw ( KiB/s): min=16351, max=16351, per=100.00%, avg=16351.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=33.33% cpu : usr=0.06%, sys=4.53%, ctx=34, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=6341KiB/s (6493kB/s), 1585KiB/s-4828KiB/s (1623kB/s-4944kB/s), io=16.0MiB (16.8MB), run=2545-2584msec WRITE: bw=12.4MiB/s (12.0MB/s), 3170KiB/s-25.5MiB/s (3246kB/s-26.7MB/s), io=32.0MiB (33.6MB), run=471-2584msec