== sanity test 398b: DIO and buffer IO race ============== 06:54:41 (1731758081) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.18232 s, 42.6 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=12408: Sat Nov 16 06:54:56 2024 read: IOPS=122, BW=491KiB/s (503kB/s)(5992KiB/12204msec) clat (usec): min=49, max=149762, avg=4608.25, stdev=4565.72 lat (usec): min=49, max=149765, avg=4609.19, stdev=4565.98 clat percentiles (usec): | 1.00th=[ 62], 5.00th=[ 1811], 10.00th=[ 2245], 20.00th=[ 2606], | 30.00th=[ 3032], 40.00th=[ 3425], 50.00th=[ 4047], 60.00th=[ 4686], | 70.00th=[ 5407], 80.00th=[ 6194], 90.00th=[ 7439], 95.00th=[ 8717], | 99.00th=[ 14222], 99.50th=[ 16319], 99.90th=[ 26346], 99.95th=[149947], | 99.99th=[149947] bw ( KiB/s): min= 272, max= 600, per=25.43%, avg=486.25, stdev=76.89, samples=24 iops : min= 68, max= 150, avg=121.54, stdev=19.20, samples=24 write: IOPS=128, BW=516KiB/s (528kB/s)(6296KiB/12204msec) clat (usec): min=436, max=20878, avg=3337.40, stdev=2908.28 lat (usec): min=436, max=20879, avg=3339.22, stdev=2908.55 clat percentiles (usec): | 1.00th=[ 506], 5.00th=[ 537], 10.00th=[ 562], 20.00th=[ 652], | 30.00th=[ 930], 40.00th=[ 2040], 50.00th=[ 2737], 60.00th=[ 3359], | 70.00th=[ 4178], 80.00th=[ 5538], 90.00th=[ 7439], 95.00th=[ 9241], | 99.00th=[12125], 99.50th=[13829], 99.90th=[18220], 99.95th=[20841], | 99.99th=[20841] bw ( KiB/s): min= 336, max= 688, per=25.47%, avg=509.96, stdev=70.10, samples=24 iops : min= 84, max= 172, avg=127.46, stdev=17.55, samples=24 lat (usec) : 50=0.03%, 100=1.76%, 250=0.36%, 500=0.42%, 750=12.47% lat (usec) : 1000=3.61% lat (msec) : 2=4.88%, 4=35.09%, 10=38.18%, 20=3.03%, 50=0.13% lat (msec) : 250=0.03% cpu : usr=0.16%, sys=23.22%, ctx=6014, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=12409: Sat Nov 16 06:54:56 2024 read: IOPS=122, BW=492KiB/s (504kB/s)(5940KiB/12080msec) clat (usec): min=57, max=32469, avg=4472.15, stdev=2478.80 lat (usec): min=57, max=32469, avg=4473.31, stdev=2478.33 clat percentiles (usec): | 1.00th=[ 60], 5.00th=[ 1876], 10.00th=[ 2212], 20.00th=[ 2606], | 30.00th=[ 3032], 40.00th=[ 3458], 50.00th=[ 4047], 60.00th=[ 4686], | 70.00th=[ 5342], 80.00th=[ 6128], 90.00th=[ 7177], 95.00th=[ 8717], | 99.00th=[13042], 99.50th=[15008], 99.90th=[19268], 99.95th=[32375], | 99.99th=[32375] bw ( KiB/s): min= 296, max= 608, per=25.61%, avg=489.67, stdev=71.16, samples=24 iops : min= 74, max= 152, avg=122.42, stdev=17.79, samples=24 write: IOPS=131, BW=525KiB/s (538kB/s)(6348KiB/12080msec) clat (usec): min=425, max=155322, avg=3403.13, stdev=4895.76 lat (usec): min=425, max=155323, avg=3404.39, stdev=4895.75 clat percentiles (usec): | 1.00th=[ 506], 5.00th=[ 537], 10.00th=[ 562], 20.00th=[ 652], | 30.00th=[ 881], 40.00th=[ 1729], 50.00th=[ 2573], 60.00th=[ 3261], | 70.00th=[ 4146], 80.00th=[ 5538], 90.00th=[ 7635], 95.00th=[ 9110], | 99.00th=[ 13173], 99.50th=[ 15270], 99.90th=[ 35914], 99.95th=[156238], | 99.99th=[156238] bw ( KiB/s): min= 336, max= 648, per=26.04%, avg=521.33, stdev=74.40, samples=24 iops : min= 84, max= 162, avg=130.33, stdev=18.60, samples=24 lat (usec) : 100=1.63%, 250=0.26%, 500=0.26%, 750=12.92%, 1000=4.46% lat (msec) : 2=5.08%, 4=34.57%, 10=37.70%, 20=3.03%, 50=0.07% lat (msec) : 250=0.03% cpu : usr=0.17%, sys=23.27%, ctx=6006, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=12410: Sat Nov 16 06:54:56 2024 read: IOPS=120, BW=480KiB/s (492kB/s)(6028KiB/12553msec) clat (usec): min=51, max=31403, avg=4515.11, stdev=2691.58 lat (usec): min=51, max=31404, avg=4516.98, stdev=2691.46 clat percentiles (usec): | 1.00th=[ 60], 5.00th=[ 194], 10.00th=[ 2180], 20.00th=[ 2638], | 30.00th=[ 3032], 40.00th=[ 3425], 50.00th=[ 4080], 60.00th=[ 4686], | 70.00th=[ 5342], 80.00th=[ 6194], 90.00th=[ 7635], 95.00th=[ 8848], | 99.00th=[13304], 99.50th=[15795], 99.90th=[27657], 99.95th=[31327], | 99.99th=[31327] bw ( KiB/s): min= 232, max= 560, per=24.60%, avg=470.37, stdev=72.95, samples=24 iops : min= 58, max= 140, avg=117.50, stdev=18.20, samples=24 write: IOPS=124, BW=499KiB/s (511kB/s)(6260KiB/12553msec) clat (usec): min=435, max=146170, avg=3633.68, stdev=4733.98 lat (usec): min=435, max=146172, avg=3634.45, stdev=4734.00 clat percentiles (usec): | 1.00th=[ 498], 5.00th=[ 537], 10.00th=[ 562], 20.00th=[ 676], | 30.00th=[ 930], 40.00th=[ 2180], 50.00th=[ 2966], 60.00th=[ 3621], | 70.00th=[ 4555], 80.00th=[ 5997], 90.00th=[ 7898], 95.00th=[ 9634], | 99.00th=[ 12649], 99.50th=[ 13960], 99.90th=[ 26870], 99.95th=[145753], | 99.99th=[145753] bw ( KiB/s): min= 312, max= 616, per=24.30%, avg=486.42, stdev=70.13, samples=24 iops : min= 78, max= 154, avg=121.50, stdev=17.53, samples=24 lat (usec) : 100=2.08%, 250=0.42%, 500=0.68%, 750=11.59%, 1000=3.84% lat (msec) : 2=4.52%, 4=33.82%, 10=39.55%, 20=3.29%, 50=0.16% lat (msec) : 250=0.03% cpu : usr=0.20%, sys=22.54%, ctx=6112, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=12411: Sat Nov 16 06:54:56 2024 read: IOPS=122, BW=490KiB/s (502kB/s)(6052KiB/12339msec) clat (usec): min=52, max=29478, avg=4492.03, stdev=2611.09 lat (usec): min=52, max=29479, avg=4492.71, stdev=2611.09 clat percentiles (usec): | 1.00th=[ 61], 5.00th=[ 1680], 10.00th=[ 2147], 20.00th=[ 2606], | 30.00th=[ 2999], 40.00th=[ 3458], 50.00th=[ 3982], 60.00th=[ 4686], | 70.00th=[ 5342], 80.00th=[ 6194], 90.00th=[ 7439], 95.00th=[ 8848], | 99.00th=[13566], 99.50th=[16188], 99.90th=[20579], 99.95th=[29492], | 99.99th=[29492] bw ( KiB/s): min= 248, max= 598, per=25.34%, avg=484.50, stdev=74.73, samples=24 iops : min= 62, max= 149, avg=121.04, stdev=18.68, samples=24 write: IOPS=126, BW=505KiB/s (518kB/s)(6236KiB/12339msec) clat (usec): min=410, max=141831, avg=3513.77, stdev=4653.87 lat (usec): min=411, max=141833, avg=3514.77, stdev=4654.17 clat percentiles (usec): | 1.00th=[ 494], 5.00th=[ 529], 10.00th=[ 562], 20.00th=[ 676], | 30.00th=[ 938], 40.00th=[ 2073], 50.00th=[ 2835], 60.00th=[ 3490], | 70.00th=[ 4293], 80.00th=[ 5800], 90.00th=[ 7570], 95.00th=[ 9110], | 99.00th=[ 12911], 99.50th=[ 14746], 99.90th=[ 34866], 99.95th=[141558], | 99.99th=[141558] bw ( KiB/s): min= 312, max= 664, per=25.00%, avg=500.50, stdev=80.67, samples=24 iops : min= 78, max= 166, avg=125.04, stdev=20.24, samples=24 lat (usec) : 100=2.02%, 250=0.39%, 500=0.75%, 750=11.43%, 1000=3.87% lat (msec) : 2=5.05%, 4=35.19%, 10=38.28%, 20=2.83%, 50=0.16% lat (msec) : 250=0.03% cpu : usr=0.28%, sys=22.90%, ctx=6016, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1913KiB/s (1959kB/s), 480KiB/s-492KiB/s (492kB/s-504kB/s), io=23.4MiB (24.6MB), run=12080-12553msec WRITE: bw=2003KiB/s (2051kB/s), 499KiB/s-525KiB/s (511kB/s-538kB/s), io=24.6MiB (25.7MB), run=12080-12553msec rand-rw: (groupid=0, jobs=1): err= 0: pid=12403: Sat Nov 16 06:54:59 2024 read: IOPS=95, BW=380KiB/s (389kB/s)(5992KiB/15763msec) clat (usec): min=1588, max=157819, avg=4310.77, stdev=5167.27 lat (usec): min=1588, max=157820, avg=4311.51, stdev=5167.26 clat percentiles (usec): | 1.00th=[ 1696], 5.00th=[ 1811], 10.00th=[ 1909], 20.00th=[ 2089], | 30.00th=[ 2245], 40.00th=[ 2507], 50.00th=[ 2769], 60.00th=[ 3228], | 70.00th=[ 4146], 80.00th=[ 5932], 90.00th=[ 8979], 95.00th=[ 11207], | 99.00th=[ 16319], 99.50th=[ 19268], 99.90th=[ 26346], 99.95th=[158335], | 99.99th=[158335] bw ( KiB/s): min= 175, max= 800, per=24.64%, avg=374.29, stdev=173.29, samples=31 iops : min= 43, max= 200, avg=93.45, stdev=43.40, samples=31 write: IOPS=99, BW=399KiB/s (409kB/s)(6296KiB/15763msec) clat (usec): min=2297, max=31339, avg=5874.12, stdev=3549.98 lat (usec): min=2298, max=31340, avg=5874.88, stdev=3549.99 clat percentiles (usec): | 1.00th=[ 2507], 5.00th=[ 2737], 10.00th=[ 2933], 20.00th=[ 3195], | 30.00th=[ 3458], 40.00th=[ 3818], 50.00th=[ 4293], 60.00th=[ 5276], | 70.00th=[ 6849], 80.00th=[ 8586], 90.00th=[10683], 95.00th=[12911], | 99.00th=[18220], 99.50th=[21103], 99.90th=[26870], 99.95th=[31327], | 99.99th=[31327] bw ( KiB/s): min= 191, max= 816, per=24.76%, avg=393.65, stdev=176.75, samples=31 iops : min= 47, max= 204, avg=98.29, stdev=44.26, samples=31 lat (msec) : 2=7.62%, 4=48.73%, 10=33.30%, 20=9.90%, 50=0.42% lat (msec) : 250=0.03% cpu : usr=0.18%, sys=11.50%, ctx=3574, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=12404: Sat Nov 16 06:54:59 2024 read: IOPS=93, BW=376KiB/s (385kB/s)(5940KiB/15806msec) clat (usec): min=1566, max=164128, avg=4250.00, stdev=5259.28 lat (usec): min=1566, max=164129, avg=4250.65, stdev=5259.30 clat percentiles (usec): | 1.00th=[ 1696], 5.00th=[ 1827], 10.00th=[ 1893], 20.00th=[ 2024], | 30.00th=[ 2180], 40.00th=[ 2442], 50.00th=[ 2769], 60.00th=[ 3228], | 70.00th=[ 4146], 80.00th=[ 5735], 90.00th=[ 8979], 95.00th=[ 10945], | 99.00th=[ 16319], 99.50th=[ 17171], 99.90th=[ 26084], 99.95th=[164627], | 99.99th=[164627] bw ( KiB/s): min= 136, max= 848, per=24.44%, avg=371.29, stdev=182.48, samples=31 iops : min= 34, max= 212, avg=92.81, stdev=45.60, samples=31 write: IOPS=100, BW=402KiB/s (411kB/s)(6348KiB/15806msec) clat (usec): min=2298, max=38777, avg=5945.77, stdev=3581.73 lat (usec): min=2298, max=38778, avg=5946.66, stdev=3581.68 clat percentiles (usec): | 1.00th=[ 2540], 5.00th=[ 2802], 10.00th=[ 2966], 20.00th=[ 3195], | 30.00th=[ 3490], 40.00th=[ 3851], 50.00th=[ 4490], 60.00th=[ 5473], | 70.00th=[ 6980], 80.00th=[ 8717], 90.00th=[10814], 95.00th=[12518], | 99.00th=[17957], 99.50th=[20841], 99.90th=[25560], 99.95th=[38536], | 99.99th=[38536] bw ( KiB/s): min= 208, max= 814, per=24.70%, avg=392.71, stdev=171.51, samples=31 iops : min= 52, max= 203, avg=98.16, stdev=42.84, samples=31 lat (msec) : 2=8.56%, 4=46.91%, 10=34.41%, 20=9.73%, 50=0.36% lat (msec) : 250=0.03% cpu : usr=0.17%, sys=11.22%, ctx=3713, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=12405: Sat Nov 16 06:54:59 2024 read: IOPS=95, BW=383KiB/s (392kB/s)(6028KiB/15739msec) clat (usec): min=1503, max=26369, avg=4238.94, stdev=3299.87 lat (usec): min=1503, max=26370, avg=4239.61, stdev=3299.87 clat percentiles (usec): | 1.00th=[ 1680], 5.00th=[ 1827], 10.00th=[ 1909], 20.00th=[ 2073], | 30.00th=[ 2245], 40.00th=[ 2474], 50.00th=[ 2802], 60.00th=[ 3294], | 70.00th=[ 4228], 80.00th=[ 5932], 90.00th=[ 9110], 95.00th=[11600], | 99.00th=[15795], 99.50th=[18482], 99.90th=[23462], 99.95th=[26346], | 99.99th=[26346] bw ( KiB/s): min= 120, max= 800, per=24.94%, avg=378.81, stdev=176.49, samples=31 iops : min= 30, max= 200, avg=94.68, stdev=44.12, samples=31 write: IOPS=99, BW=398KiB/s (407kB/s)(6260KiB/15739msec) clat (msec): min=2, max=167, avg= 5.94, stdev= 5.41 lat (msec): min=2, max=167, avg= 5.94, stdev= 5.41 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 3], 20.00th=[ 4], | 30.00th=[ 4], 40.00th=[ 4], 50.00th=[ 5], 60.00th=[ 6], | 70.00th=[ 7], 80.00th=[ 9], 90.00th=[ 11], 95.00th=[ 13], | 99.00th=[ 20], 99.50th=[ 23], 99.90th=[ 32], 99.95th=[ 167], | 99.99th=[ 167] bw ( KiB/s): min= 176, max= 760, per=24.67%, avg=392.23, stdev=179.74, samples=31 iops : min= 44, max= 190, avg=98.03, stdev=44.95, samples=31 lat (msec) : 2=7.65%, 4=48.47%, 10=33.82%, 20=9.44%, 50=0.59% lat (msec) : 250=0.03% cpu : usr=0.25%, sys=11.32%, ctx=3608, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=12406: Sat Nov 16 06:54:59 2024 read: IOPS=96, BW=386KiB/s (395kB/s)(6052KiB/15677msec) clat (usec): min=1517, max=28952, avg=4158.21, stdev=3242.99 lat (usec): min=1518, max=28952, avg=4158.93, stdev=3242.94 clat percentiles (usec): | 1.00th=[ 1696], 5.00th=[ 1827], 10.00th=[ 1909], 20.00th=[ 2040], | 30.00th=[ 2180], 40.00th=[ 2376], 50.00th=[ 2769], 60.00th=[ 3261], | 70.00th=[ 4146], 80.00th=[ 5866], 90.00th=[ 8979], 95.00th=[11076], | 99.00th=[15664], 99.50th=[17957], 99.90th=[26608], 99.95th=[28967], | 99.99th=[28967] bw ( KiB/s): min= 144, max= 800, per=25.16%, avg=382.13, stdev=172.95, samples=31 iops : min= 36, max= 200, avg=95.48, stdev=43.26, samples=31 write: IOPS=99, BW=398KiB/s (407kB/s)(6236KiB/15677msec) clat (msec): min=2, max=167, avg= 5.98, stdev= 5.43 lat (msec): min=2, max=167, avg= 5.98, stdev= 5.43 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 3], 20.00th=[ 4], | 30.00th=[ 4], 40.00th=[ 4], 50.00th=[ 5], 60.00th=[ 6], | 70.00th=[ 7], 80.00th=[ 9], 90.00th=[ 11], 95.00th=[ 13], | 99.00th=[ 20], 99.50th=[ 22], 99.90th=[ 35], 99.95th=[ 167], | 99.99th=[ 167] bw ( KiB/s): min= 160, max= 784, per=24.78%, avg=394.00, stdev=177.98, samples=31 iops : min= 40, max= 196, avg=98.45, stdev=44.51, samples=31 lat (msec) : 2=8.79%, 4=47.07%, 10=34.77%, 20=8.79%, 50=0.55% lat (msec) : 250=0.03% cpu : usr=0.16%, sys=11.34%, ctx=3722, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1519KiB/s (1556kB/s), 376KiB/s-386KiB/s (385kB/s-395kB/s), io=23.4MiB (24.6MB), run=15677-15806msec WRITE: bw=1591KiB/s (1629kB/s), 398KiB/s-402KiB/s (407kB/s-411kB/s), io=24.6MiB (25.7MB), run=15677-15806msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=12439: Sat Nov 16 06:55:06 2024 read: IOPS=63, BW=1019KiB/s (1044kB/s)(5920KiB/5808msec) clat (usec): min=1869, max=32690, avg=6331.21, stdev=4837.62 lat (usec): min=1869, max=32690, avg=6331.90, stdev=4837.64 clat percentiles (usec): | 1.00th=[ 1991], 5.00th=[ 2114], 10.00th=[ 2278], 20.00th=[ 2606], | 30.00th=[ 2900], 40.00th=[ 3326], 50.00th=[ 4146], 60.00th=[ 5669], | 70.00th=[ 7898], 80.00th=[10290], 90.00th=[12780], 95.00th=[14877], | 99.00th=[23725], 99.50th=[31589], 99.90th=[32637], 99.95th=[32637], | 99.99th=[32637] bw ( KiB/s): min= 830, max= 1216, per=24.23%, avg=998.91, stdev=127.16, samples=11 iops : min= 51, max= 76, avg=62.27, stdev= 7.99, samples=11 write: IOPS=68, BW=1096KiB/s (1123kB/s)(6368KiB/5808msec) clat (usec): min=2680, max=35022, avg=8678.55, stdev=5090.83 lat (usec): min=2681, max=35024, avg=8679.69, stdev=5090.80 clat percentiles (usec): | 1.00th=[ 2933], 5.00th=[ 3326], 10.00th=[ 3654], 20.00th=[ 4424], | 30.00th=[ 5211], 40.00th=[ 6259], 50.00th=[ 7439], 60.00th=[ 8717], | 70.00th=[10290], 80.00th=[11994], 90.00th=[15664], 95.00th=[18220], | 99.00th=[27657], 99.50th=[30540], 99.90th=[34866], 99.95th=[34866], | 99.99th=[34866] bw ( KiB/s): min= 672, max= 1440, per=25.24%, avg=1094.91, stdev=233.35, samples=11 iops : min= 42, max= 90, avg=68.27, stdev=14.69, samples=11 lat (msec) : 2=0.52%, 4=31.38%, 10=40.76%, 20=24.61%, 50=2.73% cpu : usr=0.09%, sys=9.39%, ctx=1083, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=12440: Sat Nov 16 06:55:06 2024 read: IOPS=63, BW=1018KiB/s (1043kB/s)(5744KiB/5642msec) clat (usec): min=1840, max=26260, avg=5924.68, stdev=4232.11 lat (usec): min=1840, max=26261, avg=5925.35, stdev=4232.11 clat percentiles (usec): | 1.00th=[ 1991], 5.00th=[ 2180], 10.00th=[ 2311], 20.00th=[ 2704], | 30.00th=[ 2999], 40.00th=[ 3359], 50.00th=[ 3916], 60.00th=[ 5276], | 70.00th=[ 7439], 80.00th=[ 9372], 90.00th=[11731], 95.00th=[14615], | 99.00th=[19530], 99.50th=[22152], 99.90th=[26346], 99.95th=[26346], | 99.99th=[26346] bw ( KiB/s): min= 819, max= 1280, per=24.67%, avg=1017.00, stdev=176.58, samples=11 iops : min= 51, max= 80, avg=63.55, stdev=11.06, samples=11 write: IOPS=72, BW=1160KiB/s (1188kB/s)(6544KiB/5642msec) clat (usec): min=2457, max=56769, avg=8476.02, stdev=5753.61 lat (usec): min=2458, max=56770, avg=8477.19, stdev=5753.59 clat percentiles (usec): | 1.00th=[ 2802], 5.00th=[ 3228], 10.00th=[ 3556], 20.00th=[ 4047], | 30.00th=[ 4686], 40.00th=[ 5735], 50.00th=[ 6587], 60.00th=[ 7701], | 70.00th=[ 9896], 80.00th=[12387], 90.00th=[16581], 95.00th=[19530], | 99.00th=[26870], 99.50th=[27132], 99.90th=[56886], 99.95th=[56886], | 99.99th=[56886] bw ( KiB/s): min= 960, max= 1408, per=26.75%, avg=1160.36, stdev=132.24, samples=11 iops : min= 60, max= 88, avg=72.45, stdev= 8.31, samples=11 lat (msec) : 2=0.52%, 4=33.33%, 10=42.97%, 20=20.31%, 50=2.73% lat (msec) : 100=0.13% cpu : usr=0.12%, sys=8.63%, ctx=958, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=12441: Sat Nov 16 06:55:06 2024 read: IOPS=74, BW=1200KiB/s (1229kB/s)(6256KiB/5214msec) clat (usec): min=1808, max=28515, avg=5738.98, stdev=4400.55 lat (usec): min=1808, max=28516, avg=5739.65, stdev=4400.56 clat percentiles (usec): | 1.00th=[ 1942], 5.00th=[ 2147], 10.00th=[ 2311], 20.00th=[ 2606], | 30.00th=[ 2933], 40.00th=[ 3425], 50.00th=[ 4080], 60.00th=[ 5014], | 70.00th=[ 6456], 80.00th=[ 8094], 90.00th=[11076], 95.00th=[14484], | 99.00th=[25035], 99.50th=[26870], 99.90th=[28443], 99.95th=[28443], | 99.99th=[28443] bw ( KiB/s): min= 896, max= 1504, per=29.63%, avg=1221.80, stdev=193.51, samples=10 iops : min= 56, max= 94, avg=76.20, stdev=12.06, samples=10 write: IOPS=72, BW=1157KiB/s (1185kB/s)(6032KiB/5214msec) clat (usec): min=2721, max=47836, avg=7777.72, stdev=5140.07 lat (usec): min=2722, max=47837, avg=7778.85, stdev=5140.06 clat percentiles (usec): | 1.00th=[ 3064], 5.00th=[ 3326], 10.00th=[ 3523], 20.00th=[ 3916], | 30.00th=[ 4359], 40.00th=[ 4948], 50.00th=[ 5932], 60.00th=[ 7111], | 70.00th=[ 8717], 80.00th=[11469], 90.00th=[15270], 95.00th=[18220], | 99.00th=[22938], 99.50th=[25297], 99.90th=[47973], 99.95th=[47973], | 99.99th=[47973] bw ( KiB/s): min= 960, max= 1341, per=26.76%, avg=1161.00, stdev=135.07, samples=10 iops : min= 60, max= 83, avg=72.40, stdev= 8.25, samples=10 lat (msec) : 2=0.78%, 4=34.51%, 10=45.83%, 20=16.67%, 50=2.21% cpu : usr=0.08%, sys=9.27%, ctx=895, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=12442: Sat Nov 16 06:55:06 2024 read: IOPS=71, BW=1143KiB/s (1170kB/s)(6032KiB/5279msec) clat (usec): min=1865, max=36651, avg=5446.20, stdev=4119.36 lat (usec): min=1865, max=36651, avg=5446.83, stdev=4119.36 clat percentiles (usec): | 1.00th=[ 1893], 5.00th=[ 2114], 10.00th=[ 2278], 20.00th=[ 2507], | 30.00th=[ 2835], 40.00th=[ 3130], 50.00th=[ 3589], 60.00th=[ 4490], | 70.00th=[ 6194], 80.00th=[ 8586], 90.00th=[10814], 95.00th=[13042], | 99.00th=[18482], 99.50th=[23725], 99.90th=[36439], 99.95th=[36439], | 99.99th=[36439] bw ( KiB/s): min= 928, max= 1469, per=27.69%, avg=1141.80, stdev=199.07, samples=10 iops : min= 58, max= 91, avg=71.20, stdev=12.31, samples=10 write: IOPS=74, BW=1185KiB/s (1214kB/s)(6256KiB/5279msec) clat (usec): min=2494, max=54684, avg=8163.42, stdev=5796.82 lat (usec): min=2494, max=54686, avg=8164.54, stdev=5796.85 clat percentiles (usec): | 1.00th=[ 2704], 5.00th=[ 3097], 10.00th=[ 3359], 20.00th=[ 3884], | 30.00th=[ 4359], 40.00th=[ 5276], 50.00th=[ 6259], 60.00th=[ 7570], | 70.00th=[ 9634], 80.00th=[11469], 90.00th=[15401], 95.00th=[19268], | 99.00th=[26870], 99.50th=[35390], 99.90th=[54789], 99.95th=[54789], | 99.99th=[54789] bw ( KiB/s): min= 960, max= 1344, per=27.50%, avg=1193.00, stdev=121.23, samples=10 iops : min= 60, max= 84, avg=74.40, stdev= 7.72, samples=10 lat (msec) : 2=1.56%, 4=37.50%, 10=40.10%, 20=18.49%, 50=2.21% lat (msec) : 100=0.13% cpu : usr=0.15%, sys=9.08%, ctx=936, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=4124KiB/s (4223kB/s), 1018KiB/s-1200KiB/s (1043kB/s-1229kB/s), io=23.4MiB (24.5MB), run=5214-5808msec WRITE: bw=4339KiB/s (4443kB/s), 1096KiB/s-1185KiB/s (1123kB/s-1214kB/s), io=24.6MiB (25.8MB), run=5214-5808msec rand-rw: (groupid=0, jobs=1): err= 0: pid=12443: Sat Nov 16 06:55:06 2024 read: IOPS=63, BW=1022KiB/s (1047kB/s)(5920KiB/5792msec) clat (usec): min=59, max=34180, avg=11188.90, stdev=4049.29 lat (usec): min=59, max=34181, avg=11189.65, stdev=4049.31 clat percentiles (usec): | 1.00th=[ 62], 5.00th=[ 6325], 10.00th=[ 7570], 20.00th=[ 8979], | 30.00th=[ 9765], 40.00th=[10421], 50.00th=[10945], 60.00th=[11600], | 70.00th=[12387], 80.00th=[13435], 90.00th=[15664], 95.00th=[17171], | 99.00th=[24773], 99.50th=[25297], 99.90th=[34341], 99.95th=[34341], | 99.99th=[34341] bw ( KiB/s): min= 832, max= 1152, per=25.09%, avg=1009.18, stdev=101.33, samples=11 iops : min= 52, max= 72, avg=63.00, stdev= 6.28, samples=11 write: IOPS=68, BW=1099KiB/s (1126kB/s)(6368KiB/5792msec) clat (usec): min=697, max=24985, avg=4114.64, stdev=3906.89 lat (usec): min=698, max=24986, avg=4115.73, stdev=3906.91 clat percentiles (usec): | 1.00th=[ 717], 5.00th=[ 783], 10.00th=[ 832], 20.00th=[ 1057], | 30.00th=[ 2180], 40.00th=[ 2507], 50.00th=[ 2868], 60.00th=[ 3392], | 70.00th=[ 4359], 80.00th=[ 5866], 90.00th=[ 9503], 95.00th=[12911], | 99.00th=[18482], 99.50th=[23200], 99.90th=[25035], 99.95th=[25035], | 99.99th=[25035] bw ( KiB/s): min= 800, max= 1376, per=25.64%, avg=1084.82, stdev=183.57, samples=11 iops : min= 50, max= 86, avg=67.73, stdev=11.49, samples=11 lat (usec) : 100=1.56%, 250=0.52%, 750=1.04%, 1000=8.46% lat (msec) : 2=4.43%, 4=21.09%, 10=25.91%, 20=35.42%, 50=1.56% cpu : usr=0.07%, sys=20.17%, ctx=2672, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=12444: Sat Nov 16 06:55:06 2024 read: IOPS=62, BW=999KiB/s (1023kB/s)(5744KiB/5748msec) clat (usec): min=52, max=22892, avg=11082.84, stdev=3991.90 lat (usec): min=53, max=22892, avg=11083.86, stdev=3992.33 clat percentiles (usec): | 1.00th=[ 62], 5.00th=[ 126], 10.00th=[ 7767], 20.00th=[ 8979], | 30.00th=[ 9765], 40.00th=[10421], 50.00th=[11076], 60.00th=[11863], | 70.00th=[12518], 80.00th=[13566], 90.00th=[15270], 95.00th=[18482], | 99.00th=[21890], 99.50th=[22676], 99.90th=[22938], 99.95th=[22938], | 99.99th=[22938] bw ( KiB/s): min= 704, max= 1248, per=24.44%, avg=983.00, stdev=157.81, samples=11 iops : min= 44, max= 78, avg=61.36, stdev= 9.78, samples=11 write: IOPS=71, BW=1138KiB/s (1166kB/s)(6544KiB/5748msec) clat (usec): min=678, max=57968, avg=4219.44, stdev=4905.80 lat (usec): min=679, max=57969, avg=4220.63, stdev=4905.77 clat percentiles (usec): | 1.00th=[ 709], 5.00th=[ 783], 10.00th=[ 816], 20.00th=[ 1037], | 30.00th=[ 2024], 40.00th=[ 2540], 50.00th=[ 2900], 60.00th=[ 3261], | 70.00th=[ 4080], 80.00th=[ 5735], 90.00th=[ 9634], 95.00th=[13173], | 99.00th=[18220], 99.50th=[21890], 99.90th=[57934], 99.95th=[57934], | 99.99th=[57934] bw ( KiB/s): min= 896, max= 1536, per=26.26%, avg=1111.00, stdev=219.28, samples=11 iops : min= 56, max= 96, avg=69.36, stdev=13.67, samples=11 lat (usec) : 100=1.82%, 250=0.52%, 750=1.04%, 1000=9.24% lat (msec) : 2=5.60%, 4=21.48%, 10=24.61%, 20=34.24%, 50=1.30% lat (msec) : 100=0.13% cpu : usr=0.07%, sys=19.71%, ctx=2529, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=12445: Sat Nov 16 06:55:06 2024 read: IOPS=65, BW=1051KiB/s (1076kB/s)(6256KiB/5955msec) clat (usec): min=60, max=21033, avg=11083.63, stdev=3403.96 lat (usec): min=61, max=21034, avg=11084.44, stdev=3403.97 clat percentiles (usec): | 1.00th=[ 65], 5.00th=[ 7242], 10.00th=[ 7701], 20.00th=[ 8717], | 30.00th=[ 9634], 40.00th=[10421], 50.00th=[11076], 60.00th=[11863], | 70.00th=[12649], 80.00th=[13435], 90.00th=[15270], 95.00th=[16188], | 99.00th=[18482], 99.50th=[20317], 99.90th=[21103], 99.95th=[21103], | 99.99th=[21103] bw ( KiB/s): min= 768, max= 1376, per=24.95%, avg=1003.36, stdev=156.79, samples=11 iops : min= 48, max= 86, avg=62.64, stdev= 9.76, samples=11 write: IOPS=63, BW=1013KiB/s (1037kB/s)(6032KiB/5955msec) clat (usec): min=679, max=37018, avg=4163.20, stdev=4473.83 lat (usec): min=680, max=37019, avg=4164.43, stdev=4473.81 clat percentiles (usec): | 1.00th=[ 717], 5.00th=[ 791], 10.00th=[ 816], 20.00th=[ 898], | 30.00th=[ 1254], 40.00th=[ 2343], 50.00th=[ 2769], 60.00th=[ 3359], | 70.00th=[ 4424], 80.00th=[ 5932], 90.00th=[ 9503], 95.00th=[14484], | 99.00th=[19006], 99.50th=[22414], 99.90th=[36963], 99.95th=[36963], | 99.99th=[36963] bw ( KiB/s): min= 640, max= 1216, per=22.96%, avg=971.36, stdev=205.27, samples=11 iops : min= 40, max= 76, avg=60.64, stdev=12.76, samples=11 lat (usec) : 100=1.69%, 750=1.17%, 1000=11.33% lat (msec) : 2=4.56%, 4=16.28%, 10=27.60%, 20=36.85%, 50=0.52% cpu : usr=0.13%, sys=19.72%, ctx=2663, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=12446: Sat Nov 16 06:55:06 2024 read: IOPS=65, BW=1053KiB/s (1079kB/s)(6032KiB/5726msec) clat (usec): min=51, max=23019, avg=10911.59, stdev=3902.92 lat (usec): min=51, max=23020, avg=10912.25, stdev=3902.92 clat percentiles (usec): | 1.00th=[ 63], 5.00th=[ 129], 10.00th=[ 7767], 20.00th=[ 8979], | 30.00th=[ 9765], 40.00th=[10421], 50.00th=[10945], 60.00th=[11731], | 70.00th=[12387], 80.00th=[13304], 90.00th=[14877], 95.00th=[17171], | 99.00th=[21627], 99.50th=[22938], 99.90th=[22938], 99.95th=[22938], | 99.99th=[22938] bw ( KiB/s): min= 800, max= 1216, per=25.53%, avg=1026.64, stdev=150.03, samples=11 iops : min= 50, max= 76, avg=64.09, stdev= 9.28, samples=11 write: IOPS=68, BW=1093KiB/s (1119kB/s)(6256KiB/5726msec) clat (usec): min=640, max=38896, avg=3998.28, stdev=3945.66 lat (usec): min=641, max=38897, avg=4004.32, stdev=3951.11 clat percentiles (usec): | 1.00th=[ 758], 5.00th=[ 799], 10.00th=[ 824], 20.00th=[ 1020], | 30.00th=[ 2147], 40.00th=[ 2507], 50.00th=[ 2900], 60.00th=[ 3458], | 70.00th=[ 4146], 80.00th=[ 5407], 90.00th=[ 8586], 95.00th=[12256], | 99.00th=[20579], 99.50th=[23200], 99.90th=[39060], 99.95th=[39060], | 99.99th=[39060] bw ( KiB/s): min= 832, max= 1344, per=25.36%, avg=1073.18, stdev=174.49, samples=11 iops : min= 52, max= 84, avg=67.00, stdev=10.80, samples=11 lat (usec) : 100=2.34%, 250=0.26%, 750=0.65%, 1000=9.11% lat (msec) : 2=4.69%, 4=20.83%, 10=26.17%, 20=34.51%, 50=1.43% cpu : usr=0.17%, sys=19.70%, ctx=2638, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=4022KiB/s (4119kB/s), 999KiB/s-1053KiB/s (1023kB/s-1079kB/s), io=23.4MiB (24.5MB), run=5726-5955msec WRITE: bw=4232KiB/s (4333kB/s), 1013KiB/s-1138KiB/s (1037kB/s-1166kB/s), io=24.6MiB (25.8MB), run=5726-5955msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=12457: Sat Nov 16 06:55:08 2024 read: IOPS=3, BW=4022KiB/s (4119kB/s)(7168KiB/1782msec) clat (usec): min=599, max=508706, avg=223842.58, stdev=232220.63 lat (usec): min=600, max=508707, avg=223844.13, stdev=232220.57 clat percentiles (usec): | 1.00th=[ 603], 5.00th=[ 603], 10.00th=[ 603], 20.00th=[ 635], | 30.00th=[ 73925], 40.00th=[ 73925], 50.00th=[ 86508], 60.00th=[429917], | 70.00th=[429917], 80.00th=[467665], 90.00th=[509608], 95.00th=[509608], | 99.00th=[509608], 99.50th=[509608], 99.90th=[509608], 99.95th=[509608], | 99.99th=[509608] bw ( KiB/s): min= 2043, max= 7447, per=33.47%, avg=3846.00, stdev=3118.56, samples=3 iops : min= 1, max= 7, avg= 3.33, stdev= 3.21, samples=3 write: IOPS=2, BW=2873KiB/s (2942kB/s)(5120KiB/1782msec) clat (usec): min=25199, max=63848, avg=40256.46, stdev=17230.24 lat (usec): min=25237, max=63882, avg=40314.10, stdev=17236.63 clat percentiles (usec): | 1.00th=[25297], 5.00th=[25297], 10.00th=[25297], 20.00th=[25297], | 30.00th=[27919], 40.00th=[27919], 50.00th=[31065], 60.00th=[31065], | 70.00th=[53216], 80.00th=[53216], 90.00th=[63701], 95.00th=[63701], | 99.00th=[63701], 99.50th=[63701], 99.90th=[63701], 99.95th=[63701], | 99.99th=[63701] bw ( KiB/s): min= 9309, max= 9309, per=57.86%, avg=9309.00, stdev= 0.00, samples=1 iops : min= 9, max= 9, avg= 9.00, stdev= 0.00, samples=1 lat (usec) : 750=16.67% lat (msec) : 50=25.00%, 100=33.33%, 500=16.67%, 750=8.33% cpu : usr=0.00%, sys=30.04%, ctx=872, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=12459: Sat Nov 16 06:55:08 2024 read: IOPS=1, BW=1781KiB/s (1824kB/s)(2048KiB/1150msec) clat (usec): min=13067, max=41836, avg=27452.16, stdev=20342.46 lat (usec): min=13069, max=41837, avg=27453.65, stdev=20342.43 clat percentiles (usec): | 1.00th=[13042], 5.00th=[13042], 10.00th=[13042], 20.00th=[13042], | 30.00th=[13042], 40.00th=[13042], 50.00th=[13042], 60.00th=[41681], | 70.00th=[41681], 80.00th=[41681], 90.00th=[41681], 95.00th=[41681], | 99.00th=[41681], 99.50th=[41681], 99.90th=[41681], 99.95th=[41681], | 99.99th=[41681] bw ( KiB/s): min= 2048, max= 2048, per=17.82%, avg=2048.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 write: IOPS=8, BW=8904KiB/s (9118kB/s)(10.0MiB/1150msec) clat (msec): min=26, max=499, avg=103.56, stdev=150.74 lat (msec): min=26, max=499, avg=103.73, stdev=150.84 clat percentiles (msec): | 1.00th=[ 27], 5.00th=[ 27], 10.00th=[ 27], 20.00th=[ 35], | 30.00th=[ 35], 40.00th=[ 36], 50.00th=[ 36], 60.00th=[ 42], | 70.00th=[ 58], 80.00th=[ 58], 90.00th=[ 222], 95.00th=[ 502], | 99.00th=[ 502], 99.50th=[ 502], 99.90th=[ 502], 99.95th=[ 502], | 99.99th=[ 502] bw ( KiB/s): min= 4096, max=12288, per=50.92%, avg=8192.00, stdev=5792.62, samples=2 iops : min= 4, max= 12, avg= 8.00, stdev= 5.66, samples=2 lat (msec) : 20=8.33%, 50=66.67%, 100=8.33%, 250=8.33%, 500=8.33% cpu : usr=0.09%, sys=21.50%, ctx=110, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=12462: Sat Nov 16 06:55:08 2024 read: IOPS=3, BW=3675KiB/s (3763kB/s)(6144KiB/1672msec) clat (usec): min=472, max=505626, avg=127095.62, stdev=190820.90 lat (usec): min=473, max=505627, avg=127097.08, stdev=190820.73 clat percentiles (usec): | 1.00th=[ 474], 5.00th=[ 474], 10.00th=[ 474], 20.00th=[ 611], | 30.00th=[ 611], 40.00th=[ 56886], 50.00th=[ 56886], 60.00th=[ 87557], | 70.00th=[111674], 80.00th=[111674], 90.00th=[505414], 95.00th=[505414], | 99.00th=[505414], 99.50th=[505414], 99.90th=[505414], 99.95th=[505414], | 99.99th=[505414] bw ( KiB/s): min= 4096, max= 5626, per=42.30%, avg=4861.00, stdev=1081.87, samples=2 iops : min= 4, max= 5, avg= 4.50, stdev= 0.71, samples=2 write: IOPS=3, BW=3675KiB/s (3763kB/s)(6144KiB/1672msec) clat (msec): min=18, max=635, avg=141.68, stdev=244.23 lat (msec): min=18, max=635, avg=141.79, stdev=244.22 clat percentiles (msec): | 1.00th=[ 20], 5.00th=[ 20], 10.00th=[ 20], 20.00th=[ 25], | 30.00th=[ 25], 40.00th=[ 28], 50.00th=[ 28], 60.00th=[ 34], | 70.00th=[ 110], 80.00th=[ 110], 90.00th=[ 634], 95.00th=[ 634], | 99.00th=[ 634], 99.50th=[ 634], 99.90th=[ 634], 99.95th=[ 634], | 99.99th=[ 634] bw ( KiB/s): min= 3750, max= 4096, per=24.74%, avg=3980.67, stdev=199.76, samples=3 iops : min= 3, max= 4, avg= 3.67, stdev= 0.58, samples=3 lat (usec) : 500=8.33%, 750=8.33% lat (msec) : 20=8.33%, 50=25.00%, 100=16.67%, 250=16.67%, 750=16.67% cpu : usr=0.06%, sys=22.92%, ctx=448, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=12464: Sat Nov 16 06:55:08 2024 read: IOPS=4, BW=5000KiB/s (5120kB/s)(5120KiB/1024msec) clat (usec): min=486, max=109233, avg=40880.38, stdev=44177.13 lat (usec): min=487, max=109233, avg=40881.47, stdev=44176.85 clat percentiles (usec): | 1.00th=[ 486], 5.00th=[ 486], 10.00th=[ 486], 20.00th=[ 486], | 30.00th=[ 2900], 40.00th=[ 2900], 50.00th=[ 43254], 60.00th=[ 43254], | 70.00th=[ 48497], 80.00th=[ 48497], 90.00th=[109577], 95.00th=[109577], | 99.00th=[109577], 99.50th=[109577], 99.90th=[109577], 99.95th=[109577], | 99.99th=[109577] bw ( KiB/s): min= 8192, max= 8192, per=71.28%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=6, BW=7000KiB/s (7168kB/s)(7168KiB/1024msec) clat (msec): min=27, max=562, avg=110.81, stdev=199.55 lat (msec): min=28, max=563, avg=110.88, stdev=199.55 clat percentiles (msec): | 1.00th=[ 28], 5.00th=[ 28], 10.00th=[ 28], 20.00th=[ 31], | 30.00th=[ 31], 40.00th=[ 31], 50.00th=[ 32], 60.00th=[ 42], | 70.00th=[ 42], 80.00th=[ 51], 90.00th=[ 567], 95.00th=[ 567], | 99.00th=[ 567], 99.50th=[ 567], 99.90th=[ 567], 99.95th=[ 567], | 99.99th=[ 567] bw ( KiB/s): min= 2048, max=10240, per=38.19%, avg=6144.00, stdev=5792.62, samples=2 iops : min= 2, max= 10, avg= 6.00, stdev= 5.66, samples=2 lat (usec) : 500=8.33% lat (msec) : 4=8.33%, 50=58.33%, 100=8.33%, 250=8.33%, 750=8.33% cpu : usr=0.00%, sys=25.90%, ctx=157, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=11.2MiB/s (11.8MB/s), 1781KiB/s-5000KiB/s (1824kB/s-5120kB/s), io=20.0MiB (20.0MB), run=1024-1782msec WRITE: bw=15.7MiB/s (16.5MB/s), 2873KiB/s-8904KiB/s (2942kB/s-9118kB/s), io=28.0MiB (29.4MB), run=1024-1782msec rand-rw: (groupid=0, jobs=1): err= 0: pid=12458: Sat Nov 16 06:55:08 2024 read: IOPS=3, BW=3965KiB/s (4060kB/s)(7168KiB/1808msec) clat (msec): min=22, max=611, avg=211.09, stdev=228.67 lat (msec): min=22, max=611, avg=211.09, stdev=228.67 clat percentiles (msec): | 1.00th=[ 23], 5.00th=[ 23], 10.00th=[ 23], 20.00th=[ 24], | 30.00th=[ 57], 40.00th=[ 57], 50.00th=[ 70], 60.00th=[ 309], | 70.00th=[ 309], 80.00th=[ 384], 90.00th=[ 609], 95.00th=[ 609], | 99.00th=[ 609], 99.50th=[ 609], 99.90th=[ 609], 99.95th=[ 609], | 99.99th=[ 609] bw ( KiB/s): min= 2048, max= 7314, per=33.58%, avg=3803.33, stdev=3040.33, samples=3 iops : min= 2, max= 7, avg= 3.67, stdev= 2.89, samples=3 write: IOPS=2, BW=2832KiB/s (2900kB/s)(5120KiB/1808msec) clat (msec): min=18, max=108, avg=54.69, stdev=33.31 lat (msec): min=18, max=108, avg=54.76, stdev=33.30 clat percentiles (msec): | 1.00th=[ 19], 5.00th=[ 19], 10.00th=[ 19], 20.00th=[ 19], | 30.00th=[ 41], 40.00th=[ 41], 50.00th=[ 53], 60.00th=[ 53], | 70.00th=[ 54], 80.00th=[ 54], 90.00th=[ 109], 95.00th=[ 109], | 99.00th=[ 109], 99.50th=[ 109], 99.90th=[ 109], 99.95th=[ 109], | 99.99th=[ 109] bw ( KiB/s): min= 9142, max= 9142, per=57.65%, avg=9142.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 lat (msec) : 20=8.33%, 50=25.00%, 100=33.33%, 250=8.33%, 500=16.67% lat (msec) : 750=8.33% cpu : usr=0.00%, sys=5.37%, ctx=141, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=12460: Sat Nov 16 06:55:08 2024 read: IOPS=1, BW=1690KiB/s (1730kB/s)(2048KiB/1212msec) clat (usec): min=18909, max=86154, avg=52532.14, stdev=47549.41 lat (usec): min=18910, max=86155, avg=52533.29, stdev=47549.14 clat percentiles (usec): | 1.00th=[19006], 5.00th=[19006], 10.00th=[19006], 20.00th=[19006], | 30.00th=[19006], 40.00th=[19006], 50.00th=[19006], 60.00th=[86508], | 70.00th=[86508], 80.00th=[86508], 90.00th=[86508], 95.00th=[86508], | 99.00th=[86508], 99.50th=[86508], 99.90th=[86508], 99.95th=[86508], | 99.99th=[86508] bw ( KiB/s): min= 2048, max= 2048, per=18.08%, avg=2048.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 write: IOPS=8, BW=8449KiB/s (8652kB/s)(10.0MiB/1212msec) clat (msec): min=22, max=484, avg=108.19, stdev=143.10 lat (msec): min=22, max=484, avg=108.25, stdev=143.11 clat percentiles (msec): | 1.00th=[ 23], 5.00th=[ 23], 10.00th=[ 23], 20.00th=[ 24], | 30.00th=[ 41], 40.00th=[ 47], 50.00th=[ 48], 60.00th=[ 54], | 70.00th=[ 78], 80.00th=[ 78], 90.00th=[ 211], 95.00th=[ 485], | 99.00th=[ 485], 99.50th=[ 485], 99.90th=[ 485], 99.95th=[ 485], | 99.99th=[ 485] bw ( KiB/s): min= 4096, max=12288, per=51.66%, avg=8192.00, stdev=5792.62, samples=2 iops : min= 4, max= 12, avg= 8.00, stdev= 5.66, samples=2 lat (msec) : 20=8.33%, 50=41.67%, 100=33.33%, 250=8.33%, 500=8.33% cpu : usr=0.00%, sys=11.48%, ctx=108, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=12461: Sat Nov 16 06:55:08 2024 read: IOPS=3, BW=3623KiB/s (3710kB/s)(6144KiB/1696msec) clat (msec): min=18, max=676, avg=145.50, stdev=261.44 lat (msec): min=18, max=676, avg=145.50, stdev=261.44 clat percentiles (msec): | 1.00th=[ 19], 5.00th=[ 19], 10.00th=[ 19], 20.00th=[ 22], | 30.00th=[ 22], 40.00th=[ 27], 50.00th=[ 27], 60.00th=[ 41], | 70.00th=[ 89], 80.00th=[ 89], 90.00th=[ 676], 95.00th=[ 676], | 99.00th=[ 676], 99.50th=[ 676], 99.90th=[ 676], 99.95th=[ 676], | 99.99th=[ 676] bw ( KiB/s): min=10240, max=10240, per=90.40%, avg=10240.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 write: IOPS=3, BW=3623KiB/s (3710kB/s)(6144KiB/1696msec) clat (msec): min=18, max=612, avg=128.26, stdev=237.93 lat (msec): min=18, max=612, avg=128.31, stdev=237.93 clat percentiles (msec): | 1.00th=[ 19], 5.00th=[ 19], 10.00th=[ 19], 20.00th=[ 22], | 30.00th=[ 22], 40.00th=[ 24], 50.00th=[ 24], 60.00th=[ 29], | 70.00th=[ 65], 80.00th=[ 65], 90.00th=[ 617], 95.00th=[ 617], | 99.00th=[ 617], 99.50th=[ 617], 99.90th=[ 617], 99.95th=[ 617], | 99.99th=[ 617] bw ( KiB/s): min= 1835, max= 6144, per=25.38%, avg=4025.00, stdev=2155.38, samples=3 iops : min= 1, max= 6, avg= 3.67, stdev= 2.52, samples=3 lat (msec) : 20=16.67%, 50=50.00%, 100=16.67%, 750=16.67% cpu : usr=0.06%, sys=5.78%, ctx=60, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=12463: Sat Nov 16 06:55:08 2024 read: IOPS=4, BW=4328KiB/s (4432kB/s)(5120KiB/1183msec) clat (msec): min=22, max=196, avg=78.96, stdev=69.68 lat (msec): min=22, max=196, avg=78.96, stdev=69.68 clat percentiles (msec): | 1.00th=[ 23], 5.00th=[ 23], 10.00th=[ 23], 20.00th=[ 23], | 30.00th=[ 37], 40.00th=[ 37], 50.00th=[ 57], 60.00th=[ 57], | 70.00th=[ 83], 80.00th=[ 83], 90.00th=[ 197], 95.00th=[ 197], | 99.00th=[ 197], 99.50th=[ 197], 99.90th=[ 197], 99.95th=[ 197], | 99.99th=[ 197] bw ( KiB/s): min= 8192, max= 8192, per=72.32%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=5, BW=6059KiB/s (6205kB/s)(7168KiB/1183msec) clat (msec): min=15, max=500, avg=108.56, stdev=175.38 lat (msec): min=15, max=500, avg=108.63, stdev=175.38 clat percentiles (msec): | 1.00th=[ 16], 5.00th=[ 16], 10.00th=[ 16], 20.00th=[ 26], | 30.00th=[ 27], 40.00th=[ 27], 50.00th=[ 43], 60.00th=[ 45], | 70.00th=[ 45], 80.00th=[ 106], 90.00th=[ 502], 95.00th=[ 502], | 99.00th=[ 502], 99.50th=[ 502], 99.90th=[ 502], 99.95th=[ 502], | 99.99th=[ 502] bw ( KiB/s): min= 6144, max= 8192, per=45.20%, avg=7168.00, stdev=1448.15, samples=2 iops : min= 6, max= 8, avg= 7.00, stdev= 1.41, samples=2 lat (msec) : 20=8.33%, 50=50.00%, 100=16.67%, 250=16.67%, 750=8.33% cpu : usr=0.08%, sys=7.61%, ctx=81, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=11.1MiB/s (11.6MB/s), 1690KiB/s-4328KiB/s (1730kB/s-4432kB/s), io=20.0MiB (20.0MB), run=1183-1808msec WRITE: bw=15.5MiB/s (16.2MB/s), 2832KiB/s-8449KiB/s (2900kB/s-8652kB/s), io=28.0MiB (29.4MB), run=1183-1808msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=12471: Sat Nov 16 06:55:11 2024 read: IOPS=1, BW=8058KiB/s (8251kB/s)(12.0MiB/1525msec) clat (msec): min=159, max=1157, avg=503.66, stdev=566.08 lat (msec): min=159, max=1157, avg=503.66, stdev=566.08 clat percentiles (msec): | 1.00th=[ 161], 5.00th=[ 161], 10.00th=[ 161], 20.00th=[ 161], | 30.00th=[ 161], 40.00th=[ 194], 50.00th=[ 194], 60.00th=[ 194], | 70.00th=[ 1150], 80.00th=[ 1150], 90.00th=[ 1150], 95.00th=[ 1150], | 99.00th=[ 1150], 99.50th=[ 1150], 99.90th=[ 1150], 99.95th=[ 1150], | 99.99th=[ 1150] bw ( KiB/s): min= 8175, max= 8175, per=90.96%, avg=8175.00, stdev= 0.00, samples=2 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=2 lat (msec) : 250=66.67% cpu : usr=0.00%, sys=57.55%, ctx=486, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=12472: Sat Nov 16 06:55:11 2024 read: IOPS=0, BW=2247KiB/s (2301kB/s)(4096KiB/1823msec) clat (nsec): min=187790k, max=187790k, avg=187790272.00, stdev= 0.00 lat (nsec): min=187793k, max=187793k, avg=187792750.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 188], 5.00th=[ 188], 10.00th=[ 188], 20.00th=[ 188], | 30.00th=[ 188], 40.00th=[ 188], 50.00th=[ 188], 60.00th=[ 188], | 70.00th=[ 188], 80.00th=[ 188], 90.00th=[ 188], 95.00th=[ 188], | 99.00th=[ 188], 99.50th=[ 188], 99.90th=[ 188], 99.95th=[ 188], | 99.99th=[ 188] bw ( KiB/s): min= 8175, max= 8175, per=90.96%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=1, BW=4494KiB/s (4602kB/s)(8192KiB/1823msec) clat (msec): min=156, max=1449, avg=803.34, stdev=914.16 lat (msec): min=157, max=1450, avg=803.54, stdev=914.25 clat percentiles (msec): | 1.00th=[ 157], 5.00th=[ 157], 10.00th=[ 157], 20.00th=[ 157], | 30.00th=[ 157], 40.00th=[ 157], 50.00th=[ 157], 60.00th=[ 1452], | 70.00th=[ 1452], 80.00th=[ 1452], 90.00th=[ 1452], 95.00th=[ 1452], | 99.00th=[ 1452], 99.50th=[ 1452], 99.90th=[ 1452], 99.95th=[ 1452], | 99.99th=[ 1452] bw ( KiB/s): min= 8175, max= 8175, per=45.48%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67% cpu : usr=0.00%, sys=14.27%, ctx=293, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=12473: Sat Nov 16 06:55:11 2024 write: IOPS=1, BW=6793KiB/s (6956kB/s)(12.0MiB/1809msec) clat (msec): min=93, max=1404, avg=598.00, stdev=705.66 lat (msec): min=93, max=1404, avg=598.30, stdev=705.54 clat percentiles (msec): | 1.00th=[ 93], 5.00th=[ 93], 10.00th=[ 93], 20.00th=[ 93], | 30.00th=[ 93], 40.00th=[ 296], 50.00th=[ 296], 60.00th=[ 296], | 70.00th=[ 1401], 80.00th=[ 1401], 90.00th=[ 1401], 95.00th=[ 1401], | 99.00th=[ 1401], 99.50th=[ 1401], 99.90th=[ 1401], 99.95th=[ 1401], | 99.99th=[ 1401] bw ( KiB/s): min= 8175, max= 8175, per=45.48%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 500=33.33% cpu : usr=0.00%, sys=14.77%, ctx=18, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=12476: Sat Nov 16 06:55:11 2024 write: IOPS=4, BW=18.5MiB/s (19.4MB/s)(12.0MiB/649msec) clat (msec): min=111, max=383, avg=205.16, stdev=154.24 lat (msec): min=112, max=383, avg=205.47, stdev=154.20 clat percentiles (msec): | 1.00th=[ 112], 5.00th=[ 112], 10.00th=[ 112], 20.00th=[ 112], | 30.00th=[ 112], 40.00th=[ 122], 50.00th=[ 122], 60.00th=[ 122], | 70.00th=[ 384], 80.00th=[ 384], 90.00th=[ 384], 95.00th=[ 384], | 99.00th=[ 384], 99.50th=[ 384], 99.90th=[ 384], 99.95th=[ 384], | 99.99th=[ 384] bw ( KiB/s): min=16351, max=16351, per=90.97%, avg=16351.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.00%, sys=42.13%, ctx=90, majf=0, minf=28 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=8987KiB/s (9203kB/s), 2247KiB/s-8058KiB/s (2301kB/s-8251kB/s), io=16.0MiB (16.8MB), run=1525-1823msec WRITE: bw=17.6MiB/s (18.4MB/s), 4494KiB/s-18.5MiB/s (4602kB/s-19.4MB/s), io=32.0MiB (33.6MB), run=649-1823msec rand-rw: (groupid=0, jobs=1): err= 0: pid=12474: Sat Nov 16 06:55:11 2024 read: IOPS=1, BW=6234KiB/s (6384kB/s)(12.0MiB/1971msec) clat (msec): min=67, max=1418, avg=644.28, stdev=696.70 lat (msec): min=67, max=1418, avg=644.28, stdev=696.70 clat percentiles (msec): | 1.00th=[ 68], 5.00th=[ 68], 10.00th=[ 68], 20.00th=[ 68], | 30.00th=[ 68], 40.00th=[ 447], 50.00th=[ 447], 60.00th=[ 447], | 70.00th=[ 1418], 80.00th=[ 1418], 90.00th=[ 1418], 95.00th=[ 1418], | 99.00th=[ 1418], 99.50th=[ 1418], 99.90th=[ 1418], 99.95th=[ 1418], | 99.99th=[ 1418] bw ( KiB/s): min= 8192, max= 8192, per=98.56%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 500=33.33% cpu : usr=0.00%, sys=3.30%, ctx=47, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=12475: Sat Nov 16 06:55:11 2024 read: IOPS=0, BW=2146KiB/s (2197kB/s)(4096KiB/1909msec) clat (nsec): min=405964k, max=405964k, avg=405963544.00, stdev= 0.00 lat (nsec): min=405966k, max=405966k, avg=405966137.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 405], 5.00th=[ 405], 10.00th=[ 405], 20.00th=[ 405], | 30.00th=[ 405], 40.00th=[ 405], 50.00th=[ 405], 60.00th=[ 405], | 70.00th=[ 405], 80.00th=[ 405], 90.00th=[ 405], 95.00th=[ 405], | 99.00th=[ 405], 99.50th=[ 405], 99.90th=[ 405], 99.95th=[ 405], | 99.99th=[ 405] bw ( KiB/s): min= 8192, max= 8192, per=98.56%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=1, BW=4291KiB/s (4394kB/s)(8192KiB/1909msec) clat (msec): min=38, max=1437, avg=737.66, stdev=989.03 lat (msec): min=38, max=1437, avg=737.92, stdev=989.03 clat percentiles (msec): | 1.00th=[ 39], 5.00th=[ 39], 10.00th=[ 39], 20.00th=[ 39], | 30.00th=[ 39], 40.00th=[ 39], 50.00th=[ 39], 60.00th=[ 1435], | 70.00th=[ 1435], 80.00th=[ 1435], 90.00th=[ 1435], 95.00th=[ 1435], | 99.00th=[ 1435], 99.50th=[ 1435], 99.90th=[ 1435], 99.95th=[ 1435], | 99.99th=[ 1435] bw ( KiB/s): min= 8192, max= 8192, per=47.73%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 50=33.33%, 500=33.33% cpu : usr=0.05%, sys=4.61%, ctx=57, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=12477: Sat Nov 16 06:55:11 2024 write: IOPS=1, BW=7407KiB/s (7585kB/s)(12.0MiB/1659msec) clat (msec): min=55, max=1378, avg=550.15, stdev=721.52 lat (msec): min=55, max=1378, avg=550.45, stdev=721.60 clat percentiles (msec): | 1.00th=[ 56], 5.00th=[ 56], 10.00th=[ 56], 20.00th=[ 56], | 30.00th=[ 56], 40.00th=[ 218], 50.00th=[ 218], 60.00th=[ 218], | 70.00th=[ 1385], 80.00th=[ 1385], 90.00th=[ 1385], 95.00th=[ 1385], | 99.00th=[ 1385], 99.50th=[ 1385], 99.90th=[ 1385], 99.95th=[ 1385], | 99.99th=[ 1385] bw ( KiB/s): min= 8175, max= 8175, per=47.63%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 250=33.33% cpu : usr=0.18%, sys=8.69%, ctx=213, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=12478: Sat Nov 16 06:55:11 2024 write: IOPS=1, BW=7548KiB/s (7729kB/s)(12.0MiB/1628msec) clat (msec): min=31, max=1380, avg=531.33, stdev=739.30 lat (msec): min=31, max=1380, avg=531.57, stdev=739.22 clat percentiles (msec): | 1.00th=[ 32], 5.00th=[ 32], 10.00th=[ 32], 20.00th=[ 32], | 30.00th=[ 32], 40.00th=[ 182], 50.00th=[ 182], 60.00th=[ 182], | 70.00th=[ 1385], 80.00th=[ 1385], 90.00th=[ 1385], 95.00th=[ 1385], | 99.00th=[ 1385], 99.50th=[ 1385], 99.90th=[ 1385], 99.95th=[ 1385], | 99.99th=[ 1385] bw ( KiB/s): min= 8192, max= 8192, per=47.73%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 50=33.33%, 250=33.33% cpu : usr=0.00%, sys=5.35%, ctx=62, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=8313KiB/s (8512kB/s), 2146KiB/s-6234KiB/s (2197kB/s-6384kB/s), io=16.0MiB (16.8MB), run=1909-1971msec WRITE: bw=16.8MiB/s (17.6MB/s), 4291KiB/s-7548KiB/s (4394kB/s-7729kB/s), io=32.0MiB (33.6MB), run=1628-1909msec