== sanity test 398b: DIO and buffer IO race ============== 06:41:57 (1731757317) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.55197 s, 32.4 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=28258: Sat Nov 16 06:42:13 2024 read: IOPS=126, BW=505KiB/s (517kB/s)(5992KiB/11865msec) clat (usec): min=47, max=172167, avg=3810.39, stdev=5438.98 lat (usec): min=48, max=172168, avg=3811.11, stdev=5439.11 clat percentiles (usec): | 1.00th=[ 53], 5.00th=[ 62], 10.00th=[ 74], 20.00th=[ 1942], | 30.00th=[ 2343], 40.00th=[ 2671], 50.00th=[ 3130], 60.00th=[ 3785], | 70.00th=[ 4621], 80.00th=[ 5669], 90.00th=[ 7111], 95.00th=[ 8225], | 99.00th=[ 12649], 99.50th=[ 20579], 99.90th=[ 52691], 99.95th=[173016], | 99.99th=[173016] bw ( KiB/s): min= 264, max= 696, per=24.90%, avg=500.96, stdev=87.18, samples=23 iops : min= 66, max= 174, avg=125.13, stdev=21.80, samples=23 write: IOPS=132, BW=531KiB/s (543kB/s)(6296KiB/11865msec) clat (usec): min=462, max=35972, avg=3834.54, stdev=3577.55 lat (usec): min=463, max=35973, avg=3835.52, stdev=3577.54 clat percentiles (usec): | 1.00th=[ 498], 5.00th=[ 529], 10.00th=[ 553], 20.00th=[ 627], | 30.00th=[ 848], 40.00th=[ 2008], 50.00th=[ 3097], 60.00th=[ 4080], | 70.00th=[ 5080], 80.00th=[ 6390], 90.00th=[ 8455], 95.00th=[10421], | 99.00th=[14877], 99.50th=[17957], 99.90th=[29492], 99.95th=[35914], | 99.99th=[35914] bw ( KiB/s): min= 320, max= 776, per=25.01%, avg=526.65, stdev=112.46, samples=23 iops : min= 80, max= 194, avg=131.57, stdev=28.11, samples=23 lat (usec) : 50=0.13%, 100=7.36%, 250=1.11%, 500=0.62%, 750=13.70% lat (usec) : 1000=2.73% lat (msec) : 2=4.85%, 4=30.53%, 10=34.96%, 20=3.55%, 50=0.39% lat (msec) : 100=0.03%, 250=0.03% cpu : usr=0.17%, sys=24.54%, ctx=6256, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28259: Sat Nov 16 06:42:13 2024 read: IOPS=125, BW=502KiB/s (514kB/s)(5940KiB/11829msec) clat (usec): min=47, max=31041, avg=3632.89, stdev=2895.81 lat (usec): min=48, max=31042, avg=3633.77, stdev=2895.75 clat percentiles (usec): | 1.00th=[ 54], 5.00th=[ 62], 10.00th=[ 76], 20.00th=[ 2024], | 30.00th=[ 2343], 40.00th=[ 2671], 50.00th=[ 3032], 60.00th=[ 3556], | 70.00th=[ 4359], 80.00th=[ 5473], 90.00th=[ 7046], 95.00th=[ 8455], | 99.00th=[12911], 99.50th=[16712], 99.90th=[27919], 99.95th=[31065], | 99.99th=[31065] bw ( KiB/s): min= 328, max= 680, per=24.90%, avg=501.00, stdev=93.66, samples=23 iops : min= 82, max= 170, avg=125.13, stdev=23.46, samples=23 write: IOPS=134, BW=537KiB/s (550kB/s)(6348KiB/11829msec) clat (usec): min=458, max=168846, avg=3974.43, stdev=5812.84 lat (usec): min=458, max=168847, avg=3975.42, stdev=5812.87 clat percentiles (usec): | 1.00th=[ 502], 5.00th=[ 537], 10.00th=[ 562], 20.00th=[ 644], | 30.00th=[ 947], 40.00th=[ 2073], 50.00th=[ 3097], 60.00th=[ 3982], | 70.00th=[ 4883], 80.00th=[ 6128], 90.00th=[ 8291], 95.00th=[ 10421], | 99.00th=[ 17695], 99.50th=[ 24773], 99.90th=[ 70779], 99.95th=[168821], | 99.99th=[168821] bw ( KiB/s): min= 368, max= 808, per=25.16%, avg=529.83, stdev=92.48, samples=23 iops : min= 92, max= 202, avg=132.35, stdev=23.14, samples=23 lat (usec) : 50=0.13%, 100=6.54%, 250=0.91%, 500=0.55%, 750=12.37% lat (usec) : 1000=3.12% lat (msec) : 2=5.92%, 4=33.33%, 10=33.04%, 20=3.55%, 50=0.46% lat (msec) : 100=0.03%, 250=0.03% cpu : usr=0.17%, sys=24.63%, ctx=6292, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28260: Sat Nov 16 06:42:13 2024 read: IOPS=126, BW=505KiB/s (517kB/s)(6028KiB/11933msec) clat (usec): min=47, max=64515, avg=3662.79, stdev=3318.87 lat (usec): min=47, max=64516, avg=3664.01, stdev=3319.09 clat percentiles (usec): | 1.00th=[ 52], 5.00th=[ 61], 10.00th=[ 72], 20.00th=[ 1844], | 30.00th=[ 2311], 40.00th=[ 2671], 50.00th=[ 3064], 60.00th=[ 3654], | 70.00th=[ 4555], 80.00th=[ 5669], 90.00th=[ 7177], 95.00th=[ 8455], | 99.00th=[12125], 99.50th=[16188], 99.90th=[32637], 99.95th=[64750], | 99.99th=[64750] bw ( KiB/s): min= 256, max= 720, per=24.77%, avg=498.30, stdev=111.52, samples=23 iops : min= 64, max= 180, avg=124.52, stdev=27.88, samples=23 write: IOPS=131, BW=525KiB/s (537kB/s)(6260KiB/11933msec) clat (usec): min=484, max=169076, avg=4015.22, stdev=5480.97 lat (usec): min=485, max=169077, avg=4016.81, stdev=5480.61 clat percentiles (usec): | 1.00th=[ 502], 5.00th=[ 529], 10.00th=[ 553], 20.00th=[ 652], | 30.00th=[ 906], 40.00th=[ 2245], 50.00th=[ 3228], 60.00th=[ 4178], | 70.00th=[ 5080], 80.00th=[ 6325], 90.00th=[ 8586], 95.00th=[ 10814], | 99.00th=[ 16319], 99.50th=[ 18482], 99.90th=[ 24249], 99.95th=[168821], | 99.99th=[168821] bw ( KiB/s): min= 320, max= 680, per=24.49%, avg=515.70, stdev=95.63, samples=23 iops : min= 80, max= 170, avg=128.87, stdev=23.93, samples=23 lat (usec) : 50=0.29%, 100=7.62%, 250=1.07%, 500=0.52%, 750=12.40% lat (usec) : 1000=3.68% lat (msec) : 2=5.34%, 4=30.01%, 10=34.41%, 20=4.33%, 50=0.26% lat (msec) : 100=0.03%, 250=0.03% cpu : usr=0.23%, sys=24.40%, ctx=6385, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28261: Sat Nov 16 06:42:13 2024 read: IOPS=127, BW=509KiB/s (522kB/s)(6052KiB/11881msec) clat (usec): min=46, max=50971, avg=3581.42, stdev=3061.99 lat (usec): min=46, max=50971, avg=3582.19, stdev=3061.87 clat percentiles (usec): | 1.00th=[ 51], 5.00th=[ 61], 10.00th=[ 73], 20.00th=[ 1926], | 30.00th=[ 2343], 40.00th=[ 2704], 50.00th=[ 3032], 60.00th=[ 3523], | 70.00th=[ 4293], 80.00th=[ 5342], 90.00th=[ 7177], 95.00th=[ 8586], | 99.00th=[11863], 99.50th=[14615], 99.90th=[26608], 99.95th=[51119], | 99.99th=[51119] bw ( KiB/s): min= 280, max= 672, per=25.03%, avg=503.52, stdev=94.93, samples=23 iops : min= 70, max= 168, avg=125.83, stdev=23.71, samples=23 write: IOPS=131, BW=525KiB/s (537kB/s)(6236KiB/11881msec) clat (usec): min=478, max=169113, avg=4066.79, stdev=5579.83 lat (usec): min=479, max=169114, avg=4068.58, stdev=5580.51 clat percentiles (usec): | 1.00th=[ 506], 5.00th=[ 537], 10.00th=[ 570], 20.00th=[ 660], | 30.00th=[ 1020], 40.00th=[ 2245], 50.00th=[ 3261], 60.00th=[ 4080], | 70.00th=[ 5080], 80.00th=[ 6456], 90.00th=[ 8717], 95.00th=[ 10814], | 99.00th=[ 16057], 99.50th=[ 22152], 99.90th=[ 35390], 99.95th=[168821], | 99.99th=[168821] bw ( KiB/s): min= 352, max= 696, per=24.70%, avg=520.22, stdev=100.28, samples=23 iops : min= 88, max= 174, avg=130.00, stdev=25.06, samples=23 lat (usec) : 50=0.42%, 100=7.71%, 250=0.94%, 500=0.36%, 750=11.69% lat (usec) : 1000=3.22% lat (msec) : 2=5.99%, 4=32.36%, 10=32.68%, 20=4.20%, 50=0.36% lat (msec) : 100=0.03%, 250=0.03% cpu : usr=0.19%, sys=24.12%, ctx=6345, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2012KiB/s (2061kB/s), 502KiB/s-509KiB/s (514kB/s-522kB/s), io=23.4MiB (24.6MB), run=11829-11933msec WRITE: bw=2107KiB/s (2157kB/s), 525KiB/s-537KiB/s (537kB/s-550kB/s), io=24.6MiB (25.7MB), run=11829-11933msec rand-rw: (groupid=0, jobs=1): err= 0: pid=28263: Sat Nov 16 06:42:34 2024 read: IOPS=45, BW=181KiB/s (185kB/s)(5992KiB/33151msec) clat (usec): min=1585, max=173111, avg=3501.67, stdev=5219.20 lat (usec): min=1586, max=173113, avg=3502.40, stdev=5219.22 clat percentiles (usec): | 1.00th=[ 1680], 5.00th=[ 1811], 10.00th=[ 1876], 20.00th=[ 1958], | 30.00th=[ 2040], 40.00th=[ 2147], 50.00th=[ 2278], 60.00th=[ 2507], | 70.00th=[ 2900], 80.00th=[ 4113], 90.00th=[ 6783], 95.00th=[ 8979], | 99.00th=[ 13698], 99.50th=[ 18482], 99.90th=[ 41681], 99.95th=[173016], | 99.99th=[173016] bw ( KiB/s): min= 56, max= 328, per=25.24%, avg=180.73, stdev=64.36, samples=66 iops : min= 14, max= 82, avg=45.09, stdev=16.08, samples=66 write: IOPS=47, BW=190KiB/s (194kB/s)(6296KiB/33151msec) clat (usec): min=8877, max=72078, avg=17669.29, stdev=6112.72 lat (usec): min=8878, max=72079, avg=17670.13, stdev=6112.74 clat percentiles (usec): | 1.00th=[ 9634], 5.00th=[10552], 10.00th=[11338], 20.00th=[12387], | 30.00th=[13829], 40.00th=[15401], 50.00th=[16712], 60.00th=[17957], | 70.00th=[19268], 80.00th=[21103], 90.00th=[26346], 95.00th=[29492], | 99.00th=[36439], 99.50th=[38011], 99.90th=[55837], 99.95th=[71828], | 99.99th=[71828] bw ( KiB/s): min= 64, max= 272, per=25.33%, avg=189.70, stdev=51.01, samples=66 iops : min= 16, max= 68, avg=47.33, stdev=12.77, samples=66 lat (msec) : 2=11.98%, 4=26.99%, 10=9.37%, 20=38.61%, 50=12.96% lat (msec) : 100=0.07%, 250=0.03% cpu : usr=0.11%, sys=5.67%, ctx=3513, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28264: Sat Nov 16 06:42:34 2024 read: IOPS=44, BW=177KiB/s (181kB/s)(5940KiB/33531msec) clat (usec): min=1534, max=173072, avg=3508.26, stdev=5256.75 lat (usec): min=1535, max=173074, avg=3509.04, stdev=5256.78 clat percentiles (usec): | 1.00th=[ 1713], 5.00th=[ 1811], 10.00th=[ 1876], 20.00th=[ 1958], | 30.00th=[ 2057], 40.00th=[ 2147], 50.00th=[ 2311], 60.00th=[ 2573], | 70.00th=[ 2999], 80.00th=[ 3916], 90.00th=[ 6652], 95.00th=[ 8717], | 99.00th=[ 16319], 99.50th=[ 20579], 99.90th=[ 38536], 99.95th=[173016], | 99.99th=[173016] bw ( KiB/s): min= 16, max= 360, per=24.80%, avg=177.58, stdev=72.62, samples=66 iops : min= 4, max= 90, avg=44.30, stdev=18.17, samples=66 write: IOPS=47, BW=189KiB/s (194kB/s)(6348KiB/33531msec) clat (usec): min=8697, max=72227, avg=17786.29, stdev=6271.95 lat (usec): min=8697, max=72228, avg=17787.12, stdev=6271.97 clat percentiles (usec): | 1.00th=[ 9372], 5.00th=[10552], 10.00th=[11338], 20.00th=[12387], | 30.00th=[13829], 40.00th=[15401], 50.00th=[16909], 60.00th=[18220], | 70.00th=[19530], 80.00th=[20841], 90.00th=[26608], 95.00th=[30540], | 99.00th=[36963], 99.50th=[40633], 99.90th=[50594], 99.95th=[71828], | 99.99th=[71828] bw ( KiB/s): min= 72, max= 272, per=25.23%, avg=188.97, stdev=51.14, samples=66 iops : min= 18, max= 68, avg=47.15, stdev=12.79, samples=66 lat (msec) : 2=11.52%, 4=27.38%, 10=9.34%, 20=38.12%, 50=13.54% lat (msec) : 100=0.07%, 250=0.03% cpu : usr=0.12%, sys=5.72%, ctx=3527, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28265: Sat Nov 16 06:42:34 2024 read: IOPS=45, BW=183KiB/s (187kB/s)(6028KiB/32944msec) clat (usec): min=1601, max=41202, avg=3553.94, stdev=3135.63 lat (usec): min=1601, max=41203, avg=3554.63, stdev=3135.66 clat percentiles (usec): | 1.00th=[ 1729], 5.00th=[ 1827], 10.00th=[ 1876], 20.00th=[ 1975], | 30.00th=[ 2057], 40.00th=[ 2147], 50.00th=[ 2311], 60.00th=[ 2573], | 70.00th=[ 2999], 80.00th=[ 4490], 90.00th=[ 7308], 95.00th=[ 9110], | 99.00th=[16712], 99.50th=[19792], 99.90th=[32113], 99.95th=[41157], | 99.99th=[41157] bw ( KiB/s): min= 48, max= 344, per=25.55%, avg=182.91, stdev=60.92, samples=65 iops : min= 12, max= 86, avg=45.65, stdev=15.22, samples=65 write: IOPS=47, BW=190KiB/s (195kB/s)(6260KiB/32944msec) clat (msec): min=8, max=162, avg=17.55, stdev= 7.02 lat (msec): min=8, max=162, avg=17.55, stdev= 7.02 clat percentiles (msec): | 1.00th=[ 10], 5.00th=[ 11], 10.00th=[ 12], 20.00th=[ 13], | 30.00th=[ 14], 40.00th=[ 16], 50.00th=[ 17], 60.00th=[ 18], | 70.00th=[ 20], 80.00th=[ 21], 90.00th=[ 26], 95.00th=[ 30], | 99.00th=[ 37], 99.50th=[ 45], 99.90th=[ 55], 99.95th=[ 163], | 99.99th=[ 163] bw ( KiB/s): min= 56, max= 256, per=25.27%, avg=189.31, stdev=52.66, samples=65 iops : min= 14, max= 64, avg=47.25, stdev=13.18, samples=65 lat (msec) : 2=11.82%, 4=26.60%, 10=10.51%, 20=38.48%, 50=12.50% lat (msec) : 100=0.07%, 250=0.03% cpu : usr=0.11%, sys=5.61%, ctx=3492, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28266: Sat Nov 16 06:42:34 2024 read: IOPS=45, BW=184KiB/s (188kB/s)(6052KiB/32951msec) clat (usec): min=1615, max=29729, avg=3417.52, stdev=2862.15 lat (usec): min=1616, max=29730, avg=3418.25, stdev=2862.18 clat percentiles (usec): | 1.00th=[ 1696], 5.00th=[ 1811], 10.00th=[ 1876], 20.00th=[ 1958], | 30.00th=[ 2040], 40.00th=[ 2147], 50.00th=[ 2278], 60.00th=[ 2507], | 70.00th=[ 2900], 80.00th=[ 4015], 90.00th=[ 6652], 95.00th=[ 9110], | 99.00th=[15008], 99.50th=[19006], 99.90th=[29754], 99.95th=[29754], | 99.99th=[29754] bw ( KiB/s): min= 32, max= 336, per=25.52%, avg=182.71, stdev=72.13, samples=65 iops : min= 8, max= 84, avg=45.63, stdev=18.05, samples=65 write: IOPS=47, BW=189KiB/s (194kB/s)(6236KiB/32951msec) clat (msec): min=8, max=190, avg=17.76, stdev= 7.44 lat (msec): min=8, max=190, avg=17.76, stdev= 7.44 clat percentiles (msec): | 1.00th=[ 10], 5.00th=[ 11], 10.00th=[ 12], 20.00th=[ 13], | 30.00th=[ 14], 40.00th=[ 16], 50.00th=[ 17], 60.00th=[ 19], | 70.00th=[ 20], 80.00th=[ 22], 90.00th=[ 27], 95.00th=[ 31], | 99.00th=[ 36], 99.50th=[ 40], 99.90th=[ 56], 99.95th=[ 190], | 99.99th=[ 190] bw ( KiB/s): min= 80, max= 240, per=25.18%, avg=188.62, stdev=49.59, samples=65 iops : min= 20, max= 60, avg=47.11, stdev=12.41, samples=65 lat (msec) : 2=12.86%, 4=26.50%, 10=9.31%, 20=38.77%, 50=12.50% lat (msec) : 100=0.03%, 250=0.03% cpu : usr=0.11%, sys=5.59%, ctx=3511, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=716KiB/s (733kB/s), 177KiB/s-184KiB/s (181kB/s-188kB/s), io=23.4MiB (24.6MB), run=32944-33531msec WRITE: bw=750KiB/s (768kB/s), 189KiB/s-190KiB/s (194kB/s-195kB/s), io=24.6MiB (25.7MB), run=32944-33531msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=28302: Sat Nov 16 06:42:40 2024 read: IOPS=84, BW=1350KiB/s (1383kB/s)(5920KiB/4384msec) clat (usec): min=50, max=36094, avg=8779.79, stdev=5679.19 lat (usec): min=51, max=36095, avg=8780.50, stdev=5679.34 clat percentiles (usec): | 1.00th=[ 54], 5.00th=[ 62], 10.00th=[ 66], 20.00th=[ 139], | 30.00th=[ 7832], 40.00th=[ 8848], 50.00th=[ 9634], 60.00th=[10421], | 70.00th=[11076], 80.00th=[12256], 90.00th=[13698], 95.00th=[15533], | 99.00th=[28181], 99.50th=[35390], 99.90th=[35914], 99.95th=[35914], | 99.99th=[35914] bw ( KiB/s): min= 736, max= 1536, per=23.92%, avg=1283.12, stdev=251.66, samples=8 iops : min= 46, max= 96, avg=80.00, stdev=15.63, samples=8 write: IOPS=90, BW=1453KiB/s (1487kB/s)(6368KiB/4384msec) clat (usec): min=637, max=28232, avg=2814.79, stdev=3131.22 lat (usec): min=638, max=28232, avg=2820.21, stdev=3129.88 clat percentiles (usec): | 1.00th=[ 652], 5.00th=[ 693], 10.00th=[ 750], 20.00th=[ 807], | 30.00th=[ 857], 40.00th=[ 971], 50.00th=[ 1434], 60.00th=[ 2507], | 70.00th=[ 3032], 80.00th=[ 4490], 90.00th=[ 6325], 95.00th=[ 8455], | 99.00th=[14222], 99.50th=[20055], 99.90th=[28181], 99.95th=[28181], | 99.99th=[28181] bw ( KiB/s): min= 896, max= 1888, per=24.79%, avg=1399.12, stdev=348.25, samples=8 iops : min= 56, max= 118, avg=87.25, stdev=21.77, samples=8 lat (usec) : 100=9.24%, 250=0.52%, 500=0.13%, 750=5.73%, 1000=15.89% lat (msec) : 2=5.60%, 4=13.41%, 10=26.04%, 20=22.01%, 50=1.43% cpu : usr=0.07%, sys=22.52%, ctx=2020, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28303: Sat Nov 16 06:42:40 2024 read: IOPS=82, BW=1321KiB/s (1353kB/s)(5744KiB/4348msec) clat (usec): min=47, max=22436, avg=8754.12, stdev=4619.58 lat (usec): min=48, max=22437, avg=8754.69, stdev=4619.60 clat percentiles (usec): | 1.00th=[ 55], 5.00th=[ 63], 10.00th=[ 75], 20.00th=[ 7046], | 30.00th=[ 7963], 40.00th=[ 8848], 50.00th=[ 9634], 60.00th=[10290], | 70.00th=[10945], 80.00th=[12125], 90.00th=[13566], 95.00th=[15401], | 99.00th=[18220], 99.50th=[18744], 99.90th=[22414], 99.95th=[22414], | 99.99th=[22414] bw ( KiB/s): min= 832, max= 1632, per=23.93%, avg=1283.37, stdev=231.98, samples=8 iops : min= 52, max= 102, avg=80.12, stdev=14.54, samples=8 write: IOPS=94, BW=1505KiB/s (1541kB/s)(6544KiB/4348msec) clat (usec): min=638, max=48161, avg=2783.75, stdev=3838.48 lat (usec): min=639, max=48162, avg=2784.62, stdev=3838.48 clat percentiles (usec): | 1.00th=[ 652], 5.00th=[ 685], 10.00th=[ 734], 20.00th=[ 791], | 30.00th=[ 840], 40.00th=[ 930], 50.00th=[ 1319], 60.00th=[ 2474], | 70.00th=[ 3064], 80.00th=[ 4178], 90.00th=[ 5800], 95.00th=[ 7439], | 99.00th=[12387], 99.50th=[31851], 99.90th=[47973], 99.95th=[47973], | 99.99th=[47973] bw ( KiB/s): min= 1056, max= 1984, per=25.71%, avg=1451.00, stdev=342.87, samples=8 iops : min= 66, max= 124, avg=90.62, stdev=21.35, samples=8 lat (usec) : 50=0.13%, 100=6.90%, 250=0.78%, 500=0.26%, 750=6.77% lat (usec) : 1000=16.41% lat (msec) : 2=5.08%, 4=13.54%, 10=28.12%, 20=21.48%, 50=0.52% cpu : usr=0.16%, sys=22.41%, ctx=2177, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28304: Sat Nov 16 06:42:40 2024 read: IOPS=87, BW=1401KiB/s (1435kB/s)(6256KiB/4465msec) clat (usec): min=50, max=36265, avg=8233.48, stdev=5390.42 lat (usec): min=51, max=36266, avg=8234.49, stdev=5390.59 clat percentiles (usec): | 1.00th=[ 53], 5.00th=[ 62], 10.00th=[ 64], 20.00th=[ 80], | 30.00th=[ 7439], 40.00th=[ 8356], 50.00th=[ 9241], 60.00th=[10028], | 70.00th=[10814], 80.00th=[11731], 90.00th=[13435], 95.00th=[15664], | 99.00th=[21627], 99.50th=[31851], 99.90th=[36439], 99.95th=[36439], | 99.99th=[36439] bw ( KiB/s): min= 576, max= 1664, per=24.00%, avg=1287.25, stdev=355.21, samples=8 iops : min= 36, max= 104, avg=80.38, stdev=22.18, samples=8 write: IOPS=84, BW=1351KiB/s (1383kB/s)(6032KiB/4465msec) clat (usec): min=617, max=70808, avg=3141.91, stdev=4900.17 lat (usec): min=618, max=70809, avg=3142.81, stdev=4900.19 clat percentiles (usec): | 1.00th=[ 644], 5.00th=[ 693], 10.00th=[ 750], 20.00th=[ 799], | 30.00th=[ 848], 40.00th=[ 988], 50.00th=[ 2147], 60.00th=[ 2638], | 70.00th=[ 3163], 80.00th=[ 4490], 90.00th=[ 6194], 95.00th=[ 8717], | 99.00th=[19530], 99.50th=[31851], 99.90th=[70779], 99.95th=[70779], | 99.99th=[70779] bw ( KiB/s): min= 704, max= 1728, per=22.60%, avg=1275.50, stdev=345.94, samples=8 iops : min= 44, max= 108, avg=79.62, stdev=21.75, samples=8 lat (usec) : 100=10.68%, 250=0.78%, 750=5.08%, 1000=15.36% lat (msec) : 2=3.52%, 4=14.06%, 10=28.26%, 20=21.35%, 50=0.78% lat (msec) : 100=0.13% cpu : usr=0.13%, sys=21.55%, ctx=2181, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28306: Sat Nov 16 06:42:40 2024 read: IOPS=89, BW=1437KiB/s (1472kB/s)(6032KiB/4197msec) clat (usec): min=50, max=33501, avg=8016.12, stdev=5347.54 lat (usec): min=51, max=33502, avg=8016.68, stdev=5347.55 clat percentiles (usec): | 1.00th=[ 53], 5.00th=[ 62], 10.00th=[ 63], 20.00th=[ 81], | 30.00th=[ 7177], 40.00th=[ 8291], 50.00th=[ 9372], 60.00th=[10028], | 70.00th=[10814], 80.00th=[11863], 90.00th=[13435], 95.00th=[15533], | 99.00th=[19006], 99.50th=[21890], 99.90th=[33424], 99.95th=[33424], | 99.99th=[33424] bw ( KiB/s): min= 896, max= 1632, per=26.37%, avg=1414.75, stdev=242.65, samples=8 iops : min= 56, max= 102, avg=88.12, stdev=15.13, samples=8 write: IOPS=93, BW=1491KiB/s (1526kB/s)(6256KiB/4197msec) clat (usec): min=622, max=53476, avg=2851.41, stdev=4053.52 lat (usec): min=623, max=53478, avg=2852.31, stdev=4053.55 clat percentiles (usec): | 1.00th=[ 660], 5.00th=[ 709], 10.00th=[ 750], 20.00th=[ 791], | 30.00th=[ 840], 40.00th=[ 930], 50.00th=[ 1254], 60.00th=[ 2474], | 70.00th=[ 2966], 80.00th=[ 4359], 90.00th=[ 5932], 95.00th=[ 7504], | 99.00th=[18744], 99.50th=[29230], 99.90th=[53216], 99.95th=[53216], | 99.99th=[53216] bw ( KiB/s): min= 992, max= 1920, per=26.13%, avg=1474.75, stdev=281.00, samples=8 iops : min= 62, max= 120, avg=91.87, stdev=17.64, samples=8 lat (usec) : 100=10.94%, 250=1.17%, 500=0.13%, 750=5.08%, 1000=18.10% lat (msec) : 2=4.69%, 4=11.98%, 10=25.91%, 20=21.35%, 50=0.52% lat (msec) : 100=0.13% cpu : usr=0.10%, sys=21.90%, ctx=1995, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=5364KiB/s (5493kB/s), 1321KiB/s-1437KiB/s (1353kB/s-1472kB/s), io=23.4MiB (24.5MB), run=4197-4465msec WRITE: bw=5644KiB/s (5779kB/s), 1351KiB/s-1505KiB/s (1383kB/s-1541kB/s), io=24.6MiB (25.8MB), run=4197-4465msec rand-rw: (groupid=0, jobs=1): err= 0: pid=28305: Sat Nov 16 06:42:44 2024 read: IOPS=43, BW=695KiB/s (711kB/s)(5920KiB/8521msec) clat (usec): min=1762, max=19682, avg=3826.77, stdev=2889.89 lat (usec): min=1763, max=19683, avg=3827.50, stdev=2889.85 clat percentiles (usec): | 1.00th=[ 1893], 5.00th=[ 1975], 10.00th=[ 2057], 20.00th=[ 2147], | 30.00th=[ 2245], 40.00th=[ 2376], 50.00th=[ 2540], 60.00th=[ 2868], | 70.00th=[ 3490], 80.00th=[ 4883], 90.00th=[ 7635], 95.00th=[10421], | 99.00th=[15926], 99.50th=[16581], 99.90th=[19792], 99.95th=[19792], | 99.99th=[19792] bw ( KiB/s): min= 416, max= 1152, per=25.48%, avg=692.47, stdev=210.83, samples=17 iops : min= 26, max= 72, avg=43.18, stdev=13.23, samples=17 write: IOPS=46, BW=747KiB/s (765kB/s)(6368KiB/8521msec) clat (usec): min=8852, max=44092, avg=17577.63, stdev=5566.58 lat (usec): min=8852, max=44094, avg=17578.80, stdev=5566.59 clat percentiles (usec): | 1.00th=[ 9503], 5.00th=[10814], 10.00th=[11600], 20.00th=[12649], | 30.00th=[14091], 40.00th=[15664], 50.00th=[16712], 60.00th=[18220], | 70.00th=[19268], 80.00th=[21103], 90.00th=[24773], 95.00th=[28705], | 99.00th=[35914], 99.50th=[36439], 99.90th=[44303], 99.95th=[44303], | 99.99th=[44303] bw ( KiB/s): min= 384, max= 960, per=26.06%, avg=745.18, stdev=186.33, samples=17 iops : min= 24, max= 60, avg=46.47, stdev=11.72, samples=17 lat (msec) : 2=3.26%, 4=32.55%, 10=10.81%, 20=40.10%, 50=13.28% cpu : usr=0.04%, sys=6.09%, ctx=880, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28307: Sat Nov 16 06:42:44 2024 read: IOPS=40, BW=652KiB/s (668kB/s)(5744KiB/8810msec) clat (usec): min=1799, max=24385, avg=3782.12, stdev=3193.76 lat (usec): min=1799, max=24385, avg=3782.77, stdev=3193.78 clat percentiles (usec): | 1.00th=[ 1909], 5.00th=[ 1991], 10.00th=[ 2040], 20.00th=[ 2114], | 30.00th=[ 2212], 40.00th=[ 2278], 50.00th=[ 2442], 60.00th=[ 2769], | 70.00th=[ 3294], 80.00th=[ 4555], 90.00th=[ 7898], 95.00th=[10683], | 99.00th=[17171], 99.50th=[23200], 99.90th=[24511], 99.95th=[24511], | 99.99th=[24511] bw ( KiB/s): min= 288, max= 1088, per=24.01%, avg=652.59, stdev=245.03, samples=17 iops : min= 18, max= 68, avg=40.76, stdev=15.30, samples=17 write: IOPS=46, BW=743KiB/s (761kB/s)(6544KiB/8810msec) clat (usec): min=9251, max=38412, avg=17930.38, stdev=5344.47 lat (usec): min=9252, max=38413, avg=17931.51, stdev=5344.51 clat percentiles (usec): | 1.00th=[10159], 5.00th=[11207], 10.00th=[11994], 20.00th=[13435], | 30.00th=[15008], 40.00th=[15926], 50.00th=[16909], 60.00th=[18220], | 70.00th=[19268], 80.00th=[21627], 90.00th=[25297], 95.00th=[28443], | 99.00th=[34341], 99.50th=[35914], 99.90th=[38536], 99.95th=[38536], | 99.99th=[38536] bw ( KiB/s): min= 480, max= 961, per=25.80%, avg=737.94, stdev=159.88, samples=17 iops : min= 30, max= 60, avg=46.12, stdev= 9.99, samples=17 lat (msec) : 2=2.99%, 4=32.68%, 10=8.72%, 20=41.28%, 50=14.32% cpu : usr=0.11%, sys=5.71%, ctx=881, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28308: Sat Nov 16 06:42:44 2024 read: IOPS=46, BW=748KiB/s (765kB/s)(6256KiB/8369msec) clat (usec): min=1808, max=23096, avg=3776.10, stdev=3044.90 lat (usec): min=1808, max=23097, avg=3776.92, stdev=3044.83 clat percentiles (usec): | 1.00th=[ 1860], 5.00th=[ 1958], 10.00th=[ 2057], 20.00th=[ 2114], | 30.00th=[ 2212], 40.00th=[ 2311], 50.00th=[ 2442], 60.00th=[ 2704], | 70.00th=[ 3097], 80.00th=[ 4490], 90.00th=[ 8586], 95.00th=[11076], | 99.00th=[15401], 99.50th=[15926], 99.90th=[23200], 99.95th=[23200], | 99.99th=[23200] bw ( KiB/s): min= 384, max= 1536, per=27.80%, avg=755.56, stdev=318.66, samples=16 iops : min= 24, max= 96, avg=47.06, stdev=19.83, samples=16 write: IOPS=45, BW=721KiB/s (738kB/s)(6032KiB/8369msec) clat (usec): min=8789, max=69602, avg=18226.63, stdev=6385.61 lat (usec): min=8790, max=69603, avg=18227.70, stdev=6385.63 clat percentiles (usec): | 1.00th=[ 9634], 5.00th=[10683], 10.00th=[11469], 20.00th=[12911], | 30.00th=[14222], 40.00th=[16188], 50.00th=[17695], 60.00th=[18482], | 70.00th=[20055], 80.00th=[21890], 90.00th=[26084], 95.00th=[29492], | 99.00th=[39060], 99.50th=[45351], 99.90th=[69731], 99.95th=[69731], | 99.99th=[69731] bw ( KiB/s): min= 512, max= 960, per=24.74%, avg=707.62, stdev=163.75, samples=16 iops : min= 32, max= 60, avg=44.06, stdev=10.16, samples=16 lat (msec) : 2=3.52%, 4=35.68%, 10=8.72%, 20=37.63%, 50=14.32% lat (msec) : 100=0.13% cpu : usr=0.11%, sys=6.01%, ctx=897, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28309: Sat Nov 16 06:42:44 2024 read: IOPS=43, BW=703KiB/s (720kB/s)(6032KiB/8578msec) clat (usec): min=1803, max=20095, avg=3787.32, stdev=3024.86 lat (usec): min=1804, max=20095, avg=3787.96, stdev=3024.85 clat percentiles (usec): | 1.00th=[ 1876], 5.00th=[ 1942], 10.00th=[ 2008], 20.00th=[ 2114], | 30.00th=[ 2212], 40.00th=[ 2311], 50.00th=[ 2507], 60.00th=[ 2769], | 70.00th=[ 3228], 80.00th=[ 4752], 90.00th=[ 7832], 95.00th=[11076], | 99.00th=[16188], 99.50th=[19268], 99.90th=[20055], 99.95th=[20055], | 99.99th=[20055] bw ( KiB/s): min= 384, max= 992, per=25.62%, avg=696.47, stdev=158.20, samples=17 iops : min= 24, max= 62, avg=43.53, stdev= 9.89, samples=17 write: IOPS=45, BW=729KiB/s (747kB/s)(6256KiB/8578msec) clat (usec): min=8945, max=51595, avg=18106.80, stdev=5908.93 lat (usec): min=8946, max=51596, avg=18107.95, stdev=5908.94 clat percentiles (usec): | 1.00th=[ 9372], 5.00th=[11207], 10.00th=[11863], 20.00th=[13173], | 30.00th=[14746], 40.00th=[16188], 50.00th=[17171], 60.00th=[18220], | 70.00th=[19530], 80.00th=[21627], 90.00th=[24773], 95.00th=[29492], | 99.00th=[38011], 99.50th=[46924], 99.90th=[51643], 99.95th=[51643], | 99.99th=[51643] bw ( KiB/s): min= 416, max= 896, per=25.41%, avg=726.59, stdev=159.51, samples=17 iops : min= 26, max= 56, avg=45.41, stdev= 9.97, samples=17 lat (msec) : 2=4.82%, 4=32.55%, 10=9.11%, 20=39.32%, 50=14.06% lat (msec) : 100=0.13% cpu : usr=0.13%, sys=5.74%, ctx=892, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2719KiB/s (2784kB/s), 652KiB/s-748KiB/s (668kB/s-765kB/s), io=23.4MiB (24.5MB), run=8369-8810msec WRITE: bw=2860KiB/s (2929kB/s), 721KiB/s-747KiB/s (738kB/s-765kB/s), io=24.6MiB (25.8MB), run=8369-8810msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=28327: Sat Nov 16 06:42:49 2024 read: IOPS=2, BW=2752KiB/s (2818kB/s)(7168KiB/2605msec) clat (msec): min=12, max=986, avg=161.98, stdev=363.99 lat (msec): min=12, max=986, avg=161.98, stdev=363.99 clat percentiles (msec): | 1.00th=[ 13], 5.00th=[ 13], 10.00th=[ 13], 20.00th=[ 18], | 30.00th=[ 18], 40.00th=[ 18], 50.00th=[ 22], 60.00th=[ 22], | 70.00th=[ 22], 80.00th=[ 57], 90.00th=[ 986], 95.00th=[ 986], | 99.00th=[ 986], 99.50th=[ 986], 99.90th=[ 986], 99.95th=[ 986], | 99.99th=[ 986] bw ( KiB/s): min=10240, max=10240, per=100.00%, avg=10240.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 write: IOPS=1, BW=1965KiB/s (2013kB/s)(5120KiB/2605msec) clat (msec): min=42, max=1154, avg=285.80, stdev=486.21 lat (msec): min=42, max=1154, avg=285.85, stdev=486.22 clat percentiles (msec): | 1.00th=[ 43], 5.00th=[ 43], 10.00th=[ 43], 20.00th=[ 43], | 30.00th=[ 59], 40.00th=[ 59], 50.00th=[ 66], 60.00th=[ 66], | 70.00th=[ 108], 80.00th=[ 108], 90.00th=[ 1150], 95.00th=[ 1150], | 99.00th=[ 1150], 99.50th=[ 1150], 99.90th=[ 1150], 99.95th=[ 1150], | 99.99th=[ 1150] bw ( KiB/s): min= 2035, max= 4096, per=34.02%, avg=2722.00, stdev=1189.92, samples=3 iops : min= 1, max= 4, avg= 2.00, stdev= 1.73, samples=3 lat (msec) : 20=25.00%, 50=25.00%, 100=25.00%, 250=8.33%, 1000=8.33% cpu : usr=0.04%, sys=3.73%, ctx=45, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28328: Sat Nov 16 06:42:49 2024 read: IOPS=1, BW=1504KiB/s (1540kB/s)(2048KiB/1362msec) clat (msec): min=22, max=763, avg=392.95, stdev=524.17 lat (msec): min=22, max=763, avg=392.96, stdev=524.17 clat percentiles (msec): | 1.00th=[ 23], 5.00th=[ 23], 10.00th=[ 23], 20.00th=[ 23], | 30.00th=[ 23], 40.00th=[ 23], 50.00th=[ 23], 60.00th=[ 768], | 70.00th=[ 768], 80.00th=[ 768], 90.00th=[ 768], 95.00th=[ 768], | 99.00th=[ 768], 99.50th=[ 768], 99.90th=[ 768], 99.95th=[ 768], | 99.99th=[ 768] bw ( KiB/s): min= 2048, max= 2048, per=35.84%, avg=2048.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=7, BW=7518KiB/s (7699kB/s)(10.0MiB/1362msec) clat (usec): min=34747, max=91557, avg=52492.59, stdev=18821.39 lat (usec): min=34833, max=91628, avg=52559.60, stdev=18824.96 clat percentiles (usec): | 1.00th=[34866], 5.00th=[34866], 10.00th=[34866], 20.00th=[36963], | 30.00th=[40633], 40.00th=[40633], 50.00th=[47449], 60.00th=[48497], | 70.00th=[55313], 80.00th=[55313], 90.00th=[80217], 95.00th=[91751], | 99.00th=[91751], 99.50th=[91751], 99.90th=[91751], 99.95th=[91751], | 99.99th=[91751] bw ( KiB/s): min=10240, max=10240, per=100.00%, avg=10240.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 lat (msec) : 50=66.67%, 100=25.00%, 1000=8.33% cpu : usr=0.07%, sys=7.94%, ctx=52, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28329: Sat Nov 16 06:42:49 2024 read: IOPS=5, BW=5442KiB/s (5573kB/s)(6144KiB/1129msec) clat (msec): min=14, max=693, avg=135.56, stdev=273.40 lat (msec): min=14, max=693, avg=135.57, stdev=273.40 clat percentiles (msec): | 1.00th=[ 15], 5.00th=[ 15], 10.00th=[ 15], 20.00th=[ 17], | 30.00th=[ 17], 40.00th=[ 17], 50.00th=[ 17], 60.00th=[ 27], | 70.00th=[ 47], 80.00th=[ 47], 90.00th=[ 693], 95.00th=[ 693], | 99.00th=[ 693], 99.50th=[ 693], 99.90th=[ 693], 99.95th=[ 693], | 99.99th=[ 693] bw ( KiB/s): min=10240, max=10240, per=100.00%, avg=10240.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 write: IOPS=5, BW=5442KiB/s (5573kB/s)(6144KiB/1129msec) clat (usec): min=35781, max=76115, avg=51409.80, stdev=14679.44 lat (usec): min=35817, max=76143, avg=51448.52, stdev=14682.93 clat percentiles (usec): | 1.00th=[35914], 5.00th=[35914], 10.00th=[35914], 20.00th=[41681], | 30.00th=[41681], 40.00th=[42206], 50.00th=[42206], 60.00th=[55837], | 70.00th=[56361], 80.00th=[56361], 90.00th=[76022], 95.00th=[76022], | 99.00th=[76022], 99.50th=[76022], 99.90th=[76022], 99.95th=[76022], | 99.99th=[76022] bw ( KiB/s): min=12288, max=12288, per=100.00%, avg=12288.00, stdev= 0.00, samples=1 iops : min= 12, max= 12, avg=12.00, stdev= 0.00, samples=1 lat (msec) : 20=25.00%, 50=41.67%, 100=25.00%, 750=8.33% cpu : usr=0.00%, sys=6.56%, ctx=35, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28330: Sat Nov 16 06:42:49 2024 read: IOPS=1, BW=1429KiB/s (1463kB/s)(5120KiB/3584msec) clat (msec): min=26, max=2223, avg=600.95, stdev=950.57 lat (msec): min=26, max=2223, avg=600.95, stdev=950.57 clat percentiles (msec): | 1.00th=[ 27], 5.00th=[ 27], 10.00th=[ 27], 20.00th=[ 27], | 30.00th=[ 34], 40.00th=[ 34], 50.00th=[ 34], 60.00th=[ 34], | 70.00th=[ 684], 80.00th=[ 684], 90.00th=[ 2232], 95.00th=[ 2232], | 99.00th=[ 2232], 99.50th=[ 2232], 99.90th=[ 2232], 99.95th=[ 2232], | 99.99th=[ 2232] bw ( KiB/s): min= 2048, max= 6144, per=71.68%, avg=4096.00, stdev=2896.31, samples=2 iops : min= 2, max= 6, avg= 4.00, stdev= 2.83, samples=2 write: IOPS=1, BW=2000KiB/s (2048kB/s)(7168KiB/3584msec) clat (msec): min=42, max=153, avg=80.37, stdev=40.39 lat (msec): min=42, max=153, avg=80.44, stdev=40.38 clat percentiles (msec): | 1.00th=[ 43], 5.00th=[ 43], 10.00th=[ 43], 20.00th=[ 51], | 30.00th=[ 51], 40.00th=[ 51], 50.00th=[ 64], 60.00th=[ 95], | 70.00th=[ 95], 80.00th=[ 108], 90.00th=[ 155], 95.00th=[ 155], | 99.00th=[ 155], 99.50th=[ 155], 99.90th=[ 155], 99.95th=[ 155], | 99.99th=[ 155] bw ( KiB/s): min= 6144, max= 8192, per=89.60%, avg=7168.00, stdev=1448.15, samples=2 iops : min= 6, max= 8, avg= 7.00, stdev= 1.41, samples=2 lat (msec) : 50=33.33%, 100=33.33%, 250=16.67%, 750=8.33% cpu : usr=0.00%, sys=2.82%, ctx=69, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=5714KiB/s (5851kB/s), 1429KiB/s-5442KiB/s (1463kB/s-5573kB/s), io=20.0MiB (20.0MB), run=1129-3584msec WRITE: bw=8000KiB/s (8192kB/s), 1965KiB/s-7518KiB/s (2013kB/s-7699kB/s), io=28.0MiB (29.4MB), run=1129-3584msec rand-rw: (groupid=0, jobs=1): err= 0: pid=28332: Sat Nov 16 06:42:49 2024 read: IOPS=1, BW=1903KiB/s (1949kB/s)(7168KiB/3767msec) clat (usec): min=716, max=1082.9k, avg=435049.39, stdev=489256.42 lat (usec): min=717, max=1082.9k, avg=435050.59, stdev=489256.26 clat percentiles (usec): | 1.00th=[ 717], 5.00th=[ 717], 10.00th=[ 717], | 20.00th=[ 25822], 30.00th=[ 60031], 40.00th=[ 60031], | 50.00th=[ 102237], 60.00th=[ 876610], 70.00th=[ 876610], | 80.00th=[ 893387], 90.00th=[1082131], 95.00th=[1082131], | 99.00th=[1082131], 99.50th=[1082131], 99.90th=[1082131], | 99.95th=[1082131], 99.99th=[1082131] bw ( KiB/s): min= 2048, max= 4096, per=62.79%, avg=3413.33, stdev=1182.41, samples=3 iops : min= 2, max= 4, avg= 3.33, stdev= 1.15, samples=3 write: IOPS=1, BW=1359KiB/s (1392kB/s)(5120KiB/3767msec) clat (msec): min=37, max=464, avg=135.12, stdev=184.69 lat (msec): min=37, max=464, avg=135.18, stdev=184.71 clat percentiles (msec): | 1.00th=[ 38], 5.00th=[ 38], 10.00th=[ 38], 20.00th=[ 38], | 30.00th=[ 41], 40.00th=[ 41], 50.00th=[ 53], 60.00th=[ 53], | 70.00th=[ 81], 80.00th=[ 81], 90.00th=[ 464], 95.00th=[ 464], | 99.00th=[ 464], 99.50th=[ 464], 99.90th=[ 464], 99.95th=[ 464], | 99.99th=[ 464] bw ( KiB/s): min= 2048, max= 6144, per=53.82%, avg=4096.00, stdev=2896.31, samples=2 iops : min= 2, max= 6, avg= 4.00, stdev= 2.83, samples=2 lat (usec) : 750=8.33% lat (msec) : 50=25.00%, 100=25.00%, 250=8.33%, 500=8.33%, 1000=16.67% cpu : usr=0.00%, sys=16.30%, ctx=952, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28333: Sat Nov 16 06:42:49 2024 read: IOPS=0, BW=544KiB/s (557kB/s)(2048KiB/3765msec) clat (msec): min=58, max=1105, avg=581.93, stdev=739.97 lat (msec): min=58, max=1105, avg=581.93, stdev=739.97 clat percentiles (msec): | 1.00th=[ 59], 5.00th=[ 59], 10.00th=[ 59], 20.00th=[ 59], | 30.00th=[ 59], 40.00th=[ 59], 50.00th=[ 59], 60.00th=[ 1099], | 70.00th=[ 1099], 80.00th=[ 1099], 90.00th=[ 1099], 95.00th=[ 1099], | 99.00th=[ 1099], 99.50th=[ 1099], 99.90th=[ 1099], 99.95th=[ 1099], | 99.99th=[ 1099] bw ( KiB/s): min= 2048, max= 2048, per=37.67%, avg=2048.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 write: IOPS=2, BW=2720KiB/s (2785kB/s)(10.0MiB/3765msec) clat (msec): min=28, max=1213, avg=254.68, stdev=426.94 lat (msec): min=28, max=1213, avg=254.73, stdev=426.94 clat percentiles (msec): | 1.00th=[ 29], 5.00th=[ 29], 10.00th=[ 29], 20.00th=[ 31], | 30.00th=[ 46], 40.00th=[ 59], 50.00th=[ 60], 60.00th=[ 73], | 70.00th=[ 75], 80.00th=[ 75], 90.00th=[ 885], 95.00th=[ 1217], | 99.00th=[ 1217], 99.50th=[ 1217], 99.90th=[ 1217], 99.95th=[ 1217], | 99.99th=[ 1217] bw ( KiB/s): min= 4096, max= 6144, per=62.79%, avg=4778.67, stdev=1182.41, samples=3 iops : min= 4, max= 6, avg= 4.67, stdev= 1.15, samples=3 lat (msec) : 50=25.00%, 100=50.00%, 1000=8.33% cpu : usr=0.00%, sys=13.07%, ctx=387, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28334: Sat Nov 16 06:42:49 2024 read: IOPS=1, BW=1743KiB/s (1785kB/s)(6144KiB/3524msec) clat (msec): min=7, max=1221, avg=546.85, stdev=581.82 lat (msec): min=7, max=1221, avg=546.85, stdev=581.82 clat percentiles (msec): | 1.00th=[ 8], 5.00th=[ 8], 10.00th=[ 8], 20.00th=[ 21], | 30.00th=[ 21], 40.00th=[ 45], 50.00th=[ 45], 60.00th=[ 894], | 70.00th=[ 1083], 80.00th=[ 1083], 90.00th=[ 1217], 95.00th=[ 1217], | 99.00th=[ 1217], 99.50th=[ 1217], 99.90th=[ 1217], 99.95th=[ 1217], | 99.99th=[ 1217] bw ( KiB/s): min= 2048, max= 4096, per=62.79%, avg=3413.33, stdev=1182.41, samples=3 iops : min= 2, max= 4, avg= 3.33, stdev= 1.15, samples=3 write: IOPS=1, BW=1743KiB/s (1785kB/s)(6144KiB/3524msec) clat (usec): min=16275, max=67557, avg=34531.89, stdev=17856.76 lat (usec): min=16352, max=67593, avg=34584.94, stdev=17853.20 clat percentiles (usec): | 1.00th=[16319], 5.00th=[16319], 10.00th=[16319], 20.00th=[25297], | 30.00th=[25297], 40.00th=[27395], 50.00th=[27395], 60.00th=[31327], | 70.00th=[39584], 80.00th=[39584], 90.00th=[67634], 95.00th=[67634], | 99.00th=[67634], 99.50th=[67634], 99.90th=[67634], 99.95th=[67634], | 99.99th=[67634] bw ( KiB/s): min= 4096, max= 8192, per=80.73%, avg=6144.00, stdev=2896.31, samples=2 iops : min= 4, max= 8, avg= 6.00, stdev= 2.83, samples=2 lat (msec) : 10=8.33%, 20=8.33%, 50=50.00%, 100=8.33%, 1000=8.33% cpu : usr=0.00%, sys=16.52%, ctx=946, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28335: Sat Nov 16 06:42:49 2024 read: IOPS=1, BW=1453KiB/s (1488kB/s)(5120KiB/3524msec) clat (usec): min=366, max=1068.0k, avg=603327.85, stdev=518522.94 lat (usec): min=367, max=1068.0k, avg=603328.79, stdev=518522.68 clat percentiles (usec): | 1.00th=[ 367], 5.00th=[ 367], 10.00th=[ 367], | 20.00th=[ 367], 30.00th=[ 86508], 40.00th=[ 86508], | 50.00th=[ 851444], 60.00th=[ 851444], 70.00th=[1010828], | 80.00th=[1010828], 90.00th=[1069548], 95.00th=[1069548], | 99.00th=[1069548], 99.50th=[1069548], 99.90th=[1069548], | 99.95th=[1069548], 99.99th=[1069548] bw ( KiB/s): min= 2043, max= 4096, per=50.20%, avg=2729.00, stdev=1183.86, samples=3 iops : min= 1, max= 4, avg= 2.33, stdev= 1.53, samples=3 write: IOPS=1, BW=2034KiB/s (2083kB/s)(7168KiB/3524msec) clat (msec): min=30, max=137, avg=69.68, stdev=33.47 lat (msec): min=30, max=137, avg=69.75, stdev=33.46 clat percentiles (msec): | 1.00th=[ 31], 5.00th=[ 31], 10.00th=[ 31], 20.00th=[ 53], | 30.00th=[ 57], 40.00th=[ 57], 50.00th=[ 64], 60.00th=[ 71], | 70.00th=[ 71], 80.00th=[ 77], 90.00th=[ 138], 95.00th=[ 138], | 99.00th=[ 138], 99.50th=[ 138], 99.90th=[ 138], 99.95th=[ 138], | 99.99th=[ 138] bw ( KiB/s): min= 6131, max= 8175, per=93.98%, avg=7153.00, stdev=1445.33, samples=2 iops : min= 5, max= 7, avg= 6.00, stdev= 1.41, samples=2 lat (usec) : 500=8.33% lat (msec) : 50=8.33%, 100=50.00%, 250=8.33%, 1000=8.33% cpu : usr=0.00%, sys=16.54%, ctx=865, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=5437KiB/s (5567kB/s), 544KiB/s-1903KiB/s (557kB/s-1949kB/s), io=20.0MiB (20.0MB), run=3524-3767msec WRITE: bw=7611KiB/s (7794kB/s), 1359KiB/s-2720KiB/s (1392kB/s-2785kB/s), io=28.0MiB (29.4MB), run=3524-3767msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=28344: Sat Nov 16 06:42:52 2024 read: IOPS=1, BW=5174KiB/s (5298kB/s)(12.0MiB/2375msec) clat (msec): min=131, max=2045, avg=789.69, stdev=1088.11 lat (msec): min=131, max=2045, avg=789.69, stdev=1088.11 clat percentiles (msec): | 1.00th=[ 131], 5.00th=[ 131], 10.00th=[ 131], 20.00th=[ 131], | 30.00th=[ 131], 40.00th=[ 192], 50.00th=[ 192], 60.00th=[ 192], | 70.00th=[ 2039], 80.00th=[ 2039], 90.00th=[ 2039], 95.00th=[ 2039], | 99.00th=[ 2039], 99.50th=[ 2039], 99.90th=[ 2039], 99.95th=[ 2039], | 99.99th=[ 2039] bw ( KiB/s): min=11889, max=11889, per=100.00%, avg=11889.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67% cpu : usr=0.13%, sys=21.44%, ctx=766, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28345: Sat Nov 16 06:42:52 2024 read: IOPS=1, BW=6543KiB/s (6700kB/s)(4096KiB/626msec) clat (nsec): min=142085k, max=142085k, avg=142084834.00, stdev= 0.00 lat (nsec): min=142087k, max=142087k, avg=142087056.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 142], 5.00th=[ 142], 10.00th=[ 142], 20.00th=[ 142], | 30.00th=[ 142], 40.00th=[ 142], 50.00th=[ 142], 60.00th=[ 142], | 70.00th=[ 142], 80.00th=[ 142], 90.00th=[ 142], 95.00th=[ 142], | 99.00th=[ 142], 99.50th=[ 142], 99.90th=[ 142], 99.95th=[ 142], | 99.99th=[ 142] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=3, BW=12.8MiB/s (13.4MB/s)(8192KiB/626msec) clat (msec): min=148, max=288, avg=218.45, stdev=99.33 lat (msec): min=148, max=288, avg=218.70, stdev=99.33 clat percentiles (msec): | 1.00th=[ 148], 5.00th=[ 148], 10.00th=[ 148], 20.00th=[ 148], | 30.00th=[ 148], 40.00th=[ 148], 50.00th=[ 148], 60.00th=[ 288], | 70.00th=[ 288], 80.00th=[ 288], 90.00th=[ 288], 95.00th=[ 288], | 99.00th=[ 288], 99.50th=[ 288], 99.90th=[ 288], 99.95th=[ 288], | 99.99th=[ 288] bw ( KiB/s): min= 8192, max= 8192, per=62.40%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.00%, sys=34.40%, ctx=106, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28347: Sat Nov 16 06:42:52 2024 write: IOPS=1, BW=4923KiB/s (5041kB/s)(12.0MiB/2496msec) clat (msec): min=206, max=1942, avg=819.06, stdev=974.32 lat (msec): min=207, max=1942, avg=819.35, stdev=974.34 clat percentiles (msec): | 1.00th=[ 207], 5.00th=[ 207], 10.00th=[ 207], 20.00th=[ 207], | 30.00th=[ 207], 40.00th=[ 309], 50.00th=[ 309], 60.00th=[ 309], | 70.00th=[ 1938], 80.00th=[ 1938], 90.00th=[ 1938], 95.00th=[ 1938], | 99.00th=[ 1938], 99.50th=[ 1938], 99.90th=[ 1938], 99.95th=[ 1938], | 99.99th=[ 1938] bw ( KiB/s): min= 5497, max= 8192, per=52.14%, avg=6844.50, stdev=1905.65, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 250=33.33%, 500=33.33% cpu : usr=0.00%, sys=11.90%, ctx=91, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28349: Sat Nov 16 06:42:52 2024 write: IOPS=6, BW=24.9MiB/s (26.1MB/s)(12.0MiB/482msec) clat (msec): min=109, max=230, avg=153.33, stdev=66.79 lat (msec): min=109, max=230, avg=153.59, stdev=66.71 clat percentiles (msec): | 1.00th=[ 110], 5.00th=[ 110], 10.00th=[ 110], 20.00th=[ 110], | 30.00th=[ 110], 40.00th=[ 121], 50.00th=[ 121], 60.00th=[ 121], | 70.00th=[ 230], 80.00th=[ 230], 90.00th=[ 230], 95.00th=[ 230], | 99.00th=[ 230], 99.50th=[ 230], 99.90th=[ 230], 99.95th=[ 230], | 99.99th=[ 230] lat (msec) : 250=100.00% cpu : usr=0.00%, sys=48.44%, ctx=110, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=6899KiB/s (7064kB/s), 5174KiB/s-6543KiB/s (5298kB/s-6700kB/s), io=16.0MiB (16.8MB), run=626-2375msec WRITE: bw=12.8MiB/s (13.4MB/s), 4923KiB/s-24.9MiB/s (5041kB/s-26.1MB/s), io=32.0MiB (33.6MB), run=482-2496msec rand-rw: (groupid=0, jobs=1): err= 0: pid=28346: Sat Nov 16 06:42:52 2024 read: IOPS=1, BW=4894KiB/s (5011kB/s)(12.0MiB/2511msec) clat (msec): min=257, max=1907, avg=832.25, stdev=931.60 lat (msec): min=257, max=1907, avg=832.25, stdev=931.60 clat percentiles (msec): | 1.00th=[ 257], 5.00th=[ 257], 10.00th=[ 257], 20.00th=[ 257], | 30.00th=[ 257], 40.00th=[ 334], 50.00th=[ 334], 60.00th=[ 334], | 70.00th=[ 1905], 80.00th=[ 1905], 90.00th=[ 1905], 95.00th=[ 1905], | 99.00th=[ 1905], 99.50th=[ 1905], 99.90th=[ 1905], 99.95th=[ 1905], | 99.99th=[ 1905] bw ( KiB/s): min= 6032, max= 8192, per=100.00%, avg=7112.00, stdev=1527.35, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 500=66.67% cpu : usr=0.00%, sys=2.67%, ctx=34, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28348: Sat Nov 16 06:42:52 2024 read: IOPS=1, BW=5769KiB/s (5907kB/s)(4096KiB/710msec) clat (nsec): min=223007k, max=223007k, avg=223006768.00, stdev= 0.00 lat (nsec): min=223008k, max=223008k, avg=223008346.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 224], 5.00th=[ 224], 10.00th=[ 224], 20.00th=[ 224], | 30.00th=[ 224], 40.00th=[ 224], 50.00th=[ 224], 60.00th=[ 224], | 70.00th=[ 224], 80.00th=[ 224], 90.00th=[ 224], 95.00th=[ 224], | 99.00th=[ 224], 99.50th=[ 224], 99.90th=[ 224], 99.95th=[ 224], | 99.99th=[ 224] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=2, BW=11.3MiB/s (11.8MB/s)(8192KiB/710msec) clat (msec): min=113, max=358, avg=236.26, stdev=173.16 lat (msec): min=114, max=358, avg=236.50, stdev=173.21 clat percentiles (msec): | 1.00th=[ 114], 5.00th=[ 114], 10.00th=[ 114], 20.00th=[ 114], | 30.00th=[ 114], 40.00th=[ 114], 50.00th=[ 114], 60.00th=[ 359], | 70.00th=[ 359], 80.00th=[ 359], 90.00th=[ 359], 95.00th=[ 359], | 99.00th=[ 359], 99.50th=[ 359], 99.90th=[ 359], 99.95th=[ 359], | 99.99th=[ 359] bw ( KiB/s): min= 8192, max= 8192, per=64.33%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.00%, sys=10.30%, ctx=248, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28350: Sat Nov 16 06:42:52 2024 write: IOPS=1, BW=4776KiB/s (4890kB/s)(12.0MiB/2573msec) clat (msec): min=308, max=1920, avg=852.08, stdev=925.02 lat (msec): min=309, max=1920, avg=852.40, stdev=925.06 clat percentiles (msec): | 1.00th=[ 309], 5.00th=[ 309], 10.00th=[ 309], 20.00th=[ 309], | 30.00th=[ 309], 40.00th=[ 330], 50.00th=[ 330], 60.00th=[ 330], | 70.00th=[ 1921], 80.00th=[ 1921], 90.00th=[ 1921], 95.00th=[ 1921], | 99.00th=[ 1921], 99.50th=[ 1921], 99.90th=[ 1921], 99.95th=[ 1921], | 99.99th=[ 1921] bw ( KiB/s): min= 6041, max= 8192, per=55.88%, avg=7116.50, stdev=1520.99, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 500=66.67% cpu : usr=0.00%, sys=2.68%, ctx=31, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28351: Sat Nov 16 06:42:52 2024 write: IOPS=5, BW=22.9MiB/s (24.1MB/s)(12.0MiB/523msec) clat (msec): min=90, max=213, avg=152.57, stdev=61.66 lat (msec): min=90, max=213, avg=152.91, stdev=61.74 clat percentiles (msec): | 1.00th=[ 90], 5.00th=[ 90], 10.00th=[ 90], 20.00th=[ 90], | 30.00th=[ 90], 40.00th=[ 155], 50.00th=[ 155], 60.00th=[ 155], | 70.00th=[ 213], 80.00th=[ 213], 90.00th=[ 213], 95.00th=[ 213], | 99.00th=[ 213], 99.50th=[ 213], 99.90th=[ 213], 99.95th=[ 213], | 99.99th=[ 213] bw ( KiB/s): min=16384, max=16384, per=100.00%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 250=66.67% cpu : usr=0.00%, sys=21.07%, ctx=239, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=6525KiB/s (6681kB/s), 4894KiB/s-5769KiB/s (5011kB/s-5907kB/s), io=16.0MiB (16.8MB), run=710-2511msec WRITE: bw=12.4MiB/s (13.0MB/s), 4776KiB/s-22.9MiB/s (4890kB/s-24.1MB/s), io=32.0MiB (33.6MB), run=523-2573msec