== sanity test 398b: DIO and buffer IO race ============== 03:18:55 (1726125535) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.12894 s, 44.6 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=7271: Thu Sep 12 03:19:07 2024 read: IOPS=154, BW=617KiB/s (631kB/s)(5992KiB/9719msec) clat (usec): min=38, max=33434, avg=3563.55, stdev=2603.22 lat (usec): min=38, max=33434, avg=3564.68, stdev=2604.49 clat percentiles (usec): | 1.00th=[ 44], 5.00th=[ 48], 10.00th=[ 86], 20.00th=[ 1876], | 30.00th=[ 2212], 40.00th=[ 2573], 50.00th=[ 3130], 60.00th=[ 3687], | 70.00th=[ 4490], 80.00th=[ 5211], 90.00th=[ 6390], 95.00th=[ 7439], | 99.00th=[11469], 99.50th=[14222], 99.90th=[29230], 99.95th=[33424], | 99.99th=[33424] bw ( KiB/s): min= 264, max= 976, per=24.76%, avg=605.16, stdev=130.80, samples=19 iops : min= 66, max= 244, avg=151.21, stdev=32.74, samples=19 write: IOPS=161, BW=648KiB/s (663kB/s)(6296KiB/9719msec) clat (usec): min=321, max=131104, avg=2735.17, stdev=4230.23 lat (usec): min=322, max=131105, avg=2735.82, stdev=4230.35 clat percentiles (usec): | 1.00th=[ 383], 5.00th=[ 408], 10.00th=[ 429], 20.00th=[ 469], | 30.00th=[ 529], 40.00th=[ 750], 50.00th=[ 1811], 60.00th=[ 2606], | 70.00th=[ 3458], 80.00th=[ 4686], 90.00th=[ 6456], 95.00th=[ 8160], | 99.00th=[ 11207], 99.50th=[ 13829], 99.90th=[ 23987], 99.95th=[131597], | 99.99th=[131597] bw ( KiB/s): min= 304, max= 968, per=24.70%, avg=632.11, stdev=126.30, samples=19 iops : min= 76, max= 242, avg=157.95, stdev=31.58, samples=19 lat (usec) : 50=2.54%, 100=2.44%, 250=0.23%, 500=13.51%, 750=7.10% lat (usec) : 1000=2.05% lat (msec) : 2=10.45%, 4=31.48%, 10=28.26%, 20=1.79%, 50=0.13% lat (msec) : 250=0.03% cpu : usr=0.12%, sys=21.09%, ctx=6334, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=7272: Thu Sep 12 03:19:07 2024 read: IOPS=152, BW=609KiB/s (624kB/s)(5940KiB/9748msec) clat (usec): min=37, max=32074, avg=3387.18, stdev=2453.41 lat (usec): min=37, max=32074, avg=3387.66, stdev=2453.44 clat percentiles (usec): | 1.00th=[ 44], 5.00th=[ 51], 10.00th=[ 69], 20.00th=[ 1876], | 30.00th=[ 2212], 40.00th=[ 2507], 50.00th=[ 2933], 60.00th=[ 3490], | 70.00th=[ 4178], 80.00th=[ 4948], 90.00th=[ 6128], 95.00th=[ 7111], | 99.00th=[10552], 99.50th=[14222], 99.90th=[30016], 99.95th=[32113], | 99.99th=[32113] bw ( KiB/s): min= 256, max= 912, per=24.61%, avg=601.42, stdev=130.19, samples=19 iops : min= 64, max= 228, avg=150.26, stdev=32.60, samples=19 write: IOPS=162, BW=651KiB/s (667kB/s)(6348KiB/9748msec) clat (usec): min=321, max=167809, avg=2925.22, stdev=5055.39 lat (usec): min=322, max=167810, avg=2926.69, stdev=5055.97 clat percentiles (usec): | 1.00th=[ 383], 5.00th=[ 412], 10.00th=[ 437], 20.00th=[ 482], | 30.00th=[ 562], 40.00th=[ 848], 50.00th=[ 2057], 60.00th=[ 2835], | 70.00th=[ 3687], 80.00th=[ 5014], 90.00th=[ 6652], 95.00th=[ 8094], | 99.00th=[ 12256], 99.50th=[ 15139], 99.90th=[ 31065], 99.95th=[168821], | 99.99th=[168821] bw ( KiB/s): min= 328, max= 840, per=24.73%, avg=632.95, stdev=111.39, samples=19 iops : min= 82, max= 210, avg=158.16, stdev=27.84, samples=19 lat (usec) : 50=2.31%, 100=3.06%, 250=0.16%, 500=11.85%, 750=8.11% lat (usec) : 1000=1.79% lat (msec) : 2=9.60%, 4=33.33%, 10=28.16%, 20=1.43%, 50=0.16% lat (msec) : 250=0.03% cpu : usr=0.17%, sys=20.66%, ctx=6292, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=7273: Thu Sep 12 03:19:07 2024 read: IOPS=153, BW=614KiB/s (628kB/s)(6028KiB/9823msec) clat (usec): min=36, max=34806, avg=3528.75, stdev=2489.87 lat (usec): min=36, max=34807, avg=3529.99, stdev=2490.26 clat percentiles (usec): | 1.00th=[ 44], 5.00th=[ 52], 10.00th=[ 553], 20.00th=[ 1893], | 30.00th=[ 2180], 40.00th=[ 2573], 50.00th=[ 3032], 60.00th=[ 3621], | 70.00th=[ 4359], 80.00th=[ 5211], 90.00th=[ 6390], 95.00th=[ 7439], | 99.00th=[11731], 99.50th=[13435], 99.90th=[23725], 99.95th=[34866], | 99.99th=[34866] bw ( KiB/s): min= 240, max= 904, per=24.78%, avg=605.58, stdev=127.90, samples=19 iops : min= 60, max= 226, avg=151.32, stdev=31.97, samples=19 write: IOPS=159, BW=637KiB/s (653kB/s)(6260KiB/9823msec) clat (usec): min=329, max=184033, avg=2830.55, stdev=5317.13 lat (usec): min=329, max=184034, avg=2831.20, stdev=5317.10 clat percentiles (usec): | 1.00th=[ 379], 5.00th=[ 408], 10.00th=[ 424], 20.00th=[ 474], | 30.00th=[ 545], 40.00th=[ 832], 50.00th=[ 2008], 60.00th=[ 2769], | 70.00th=[ 3556], 80.00th=[ 4752], 90.00th=[ 6521], 95.00th=[ 7898], | 99.00th=[ 11731], 99.50th=[ 13829], 99.90th=[ 21103], 99.95th=[183501], | 99.99th=[183501] bw ( KiB/s): min= 312, max= 896, per=24.47%, avg=626.21, stdev=120.46, samples=19 iops : min= 78, max= 224, avg=156.47, stdev=30.16, samples=19 lat (usec) : 50=2.31%, 100=2.28%, 250=0.26%, 500=12.37%, 750=7.23% lat (usec) : 1000=1.92% lat (msec) : 2=10.64%, 4=32.19%, 10=29.10%, 20=1.56%, 50=0.10% lat (msec) : 250=0.03% cpu : usr=0.15%, sys=20.54%, ctx=6375, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=7275: Thu Sep 12 03:19:07 2024 read: IOPS=154, BW=619KiB/s (634kB/s)(6052KiB/9774msec) clat (usec): min=36, max=168392, avg=3570.53, stdev=4951.39 lat (usec): min=37, max=168393, avg=3571.65, stdev=4951.24 clat percentiles (usec): | 1.00th=[ 43], 5.00th=[ 51], 10.00th=[ 143], 20.00th=[ 1844], | 30.00th=[ 2180], 40.00th=[ 2474], 50.00th=[ 2900], 60.00th=[ 3523], | 70.00th=[ 4293], 80.00th=[ 5014], 90.00th=[ 6194], 95.00th=[ 7570], | 99.00th=[ 13435], 99.50th=[ 16319], 99.90th=[ 23987], 99.95th=[168821], | 99.99th=[168821] bw ( KiB/s): min= 272, max= 856, per=24.69%, avg=603.47, stdev=115.16, samples=19 iops : min= 68, max= 214, avg=150.79, stdev=28.81, samples=19 write: IOPS=159, BW=638KiB/s (653kB/s)(6236KiB/9774msec) clat (usec): min=320, max=18182, avg=2756.17, stdev=2725.37 lat (usec): min=320, max=18182, avg=2756.78, stdev=2725.32 clat percentiles (usec): | 1.00th=[ 375], 5.00th=[ 404], 10.00th=[ 429], 20.00th=[ 469], | 30.00th=[ 545], 40.00th=[ 816], 50.00th=[ 2008], 60.00th=[ 2802], | 70.00th=[ 3654], 80.00th=[ 4817], 90.00th=[ 6587], 95.00th=[ 8094], | 99.00th=[11338], 99.50th=[13435], 99.90th=[17433], 99.95th=[18220], | 99.99th=[18220] bw ( KiB/s): min= 344, max= 1064, per=24.41%, avg=624.53, stdev=141.13, samples=19 iops : min= 86, max= 266, avg=156.05, stdev=35.33, samples=19 lat (usec) : 50=2.38%, 100=2.41%, 250=0.26%, 500=12.99%, 750=6.64% lat (usec) : 1000=2.12% lat (msec) : 2=10.97%, 4=31.67%, 10=28.65%, 20=1.73%, 50=0.16% lat (msec) : 250=0.03% cpu : usr=0.10%, sys=20.95%, ctx=6332, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2444KiB/s (2503kB/s), 609KiB/s-619KiB/s (624kB/s-634kB/s), io=23.4MiB (24.6MB), run=9719-9823msec WRITE: bw=2559KiB/s (2621kB/s), 637KiB/s-651KiB/s (653kB/s-667kB/s), io=24.6MiB (25.7MB), run=9719-9823msec rand-rw: (groupid=0, jobs=1): err= 0: pid=7274: Thu Sep 12 03:19:39 2024 read: IOPS=95, BW=381KiB/s (390kB/s)(5992KiB/15724msec) clat (usec): min=1654, max=194290, avg=4119.90, stdev=5534.41 lat (usec): min=1654, max=194291, avg=4120.47, stdev=5534.42 clat percentiles (usec): | 1.00th=[ 1795], 5.00th=[ 2024], 10.00th=[ 2311], 20.00th=[ 2573], | 30.00th=[ 2737], 40.00th=[ 2868], 50.00th=[ 2999], 60.00th=[ 3195], | 70.00th=[ 3818], 80.00th=[ 5014], 90.00th=[ 7177], 95.00th=[ 9372], | 99.00th=[ 14091], 99.50th=[ 16319], 99.90th=[ 31065], 99.95th=[193987], | 99.99th=[193987] bw ( KiB/s): min= 152, max= 528, per=66.29%, avg=380.52, stdev=94.57, samples=31 iops : min= 38, max= 132, avg=95.06, stdev=23.64, samples=31 write: IOPS=100, BW=400KiB/s (410kB/s)(6296KiB/15724msec) clat (usec): min=2252, max=74749, avg=6017.45, stdev=3252.02 lat (usec): min=2253, max=74749, avg=6018.11, stdev=3252.01 clat percentiles (usec): | 1.00th=[ 2737], 5.00th=[ 3261], 10.00th=[ 3752], 20.00th=[ 4228], | 30.00th=[ 4555], 40.00th=[ 5014], 50.00th=[ 5407], 60.00th=[ 5669], | 70.00th=[ 6063], 80.00th=[ 7111], 90.00th=[ 9110], 95.00th=[10945], | 99.00th=[15795], 99.50th=[21890], 99.90th=[38011], 99.95th=[74974], | 99.99th=[74974] bw ( KiB/s): min= 160, max= 568, per=66.24%, avg=398.10, stdev=97.78, samples=31 iops : min= 40, max= 142, avg=99.45, stdev=24.48, samples=31 lat (msec) : 2=2.21%, 4=40.17%, 10=51.82%, 20=5.34%, 50=0.39% lat (msec) : 100=0.03%, 250=0.03% cpu : usr=0.15%, sys=9.20%, ctx=3387, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=7276: Thu Sep 12 03:19:39 2024 read: IOPS=83, BW=335KiB/s (343kB/s)(5940KiB/17751msec) clat (usec): min=1600, max=169425, avg=3845.39, stdev=4854.73 lat (usec): min=1601, max=169427, avg=3845.96, stdev=4854.74 clat percentiles (usec): | 1.00th=[ 1844], 5.00th=[ 2147], 10.00th=[ 2409], 20.00th=[ 2606], | 30.00th=[ 2671], 40.00th=[ 2802], 50.00th=[ 2900], 60.00th=[ 3032], | 70.00th=[ 3326], 80.00th=[ 4424], 90.00th=[ 6390], 95.00th=[ 8717], | 99.00th=[ 12780], 99.50th=[ 14484], 99.90th=[ 25822], 99.95th=[168821], | 99.99th=[168821] bw ( KiB/s): min= 112, max= 632, per=65.50%, avg=375.97, stdev=129.49, samples=31 iops : min= 28, max= 158, avg=93.97, stdev=32.42, samples=31 write: IOPS=89, BW=358KiB/s (366kB/s)(6348KiB/17751msec) clat (msec): min=2, max=2300, avg= 7.54, stdev=58.42 lat (msec): min=2, max=2300, avg= 7.54, stdev=58.42 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 4], 10.00th=[ 4], 20.00th=[ 4], | 30.00th=[ 5], 40.00th=[ 5], 50.00th=[ 6], 60.00th=[ 6], | 70.00th=[ 6], 80.00th=[ 7], 90.00th=[ 9], 95.00th=[ 11], | 99.00th=[ 16], 99.50th=[ 27], 99.90th=[ 305], 99.95th=[ 2299], | 99.99th=[ 2299] bw ( KiB/s): min= 95, max= 624, per=66.25%, avg=398.16, stdev=126.80, samples=31 iops : min= 23, max= 156, avg=99.52, stdev=31.76, samples=31 lat (msec) : 2=1.56%, 4=46.09%, 10=47.40%, 20=4.49%, 50=0.33% lat (msec) : 250=0.07%, 500=0.03% cpu : usr=0.12%, sys=8.16%, ctx=3320, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=7277: Thu Sep 12 03:19:39 2024 read: IOPS=37, BW=151KiB/s (155kB/s)(6028KiB/39952msec) clat (usec): min=1591, max=2840.7k, avg=9544.66, stdev=73956.16 lat (usec): min=1592, max=2840.7k, avg=9545.73, stdev=73956.18 clat percentiles (usec): | 1.00th=[ 1893], 5.00th=[ 2311], 10.00th=[ 2638], | 20.00th=[ 2999], 30.00th=[ 3654], 40.00th=[ 3884], | 50.00th=[ 3982], 60.00th=[ 4080], 70.00th=[ 5080], | 80.00th=[ 8979], 90.00th=[ 15008], 95.00th=[ 23462], | 99.00th=[ 67634], 99.50th=[ 93848], 99.90th=[ 143655], | 99.95th=[2835350], 99.99th=[2835350] bw ( KiB/s): min= 8, max= 512, per=38.30%, avg=219.81, stdev=121.96, samples=54 iops : min= 2, max= 128, avg=54.81, stdev=30.58, samples=54 write: IOPS=39, BW=157KiB/s (160kB/s)(6260KiB/39952msec) clat (msec): min=2, max=6436, avg=16.28, stdev=189.01 lat (msec): min=2, max=6436, avg=16.29, stdev=189.01 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 5], 10.00th=[ 5], 20.00th=[ 6], | 30.00th=[ 6], 40.00th=[ 6], 50.00th=[ 6], 60.00th=[ 8], | 70.00th=[ 9], 80.00th=[ 11], 90.00th=[ 16], 95.00th=[ 21], | 99.00th=[ 80], 99.50th=[ 117], 99.90th=[ 3742], 99.95th=[ 6409], | 99.99th=[ 6409] bw ( KiB/s): min= 23, max= 512, per=37.86%, avg=227.52, stdev=112.03, samples=54 iops : min= 5, max= 128, avg=56.74, stdev=28.10, samples=54 lat (msec) : 2=0.85%, 4=27.54%, 10=52.02%, 20=14.00%, 50=3.91% lat (msec) : 100=1.24%, 250=0.33%, 1000=0.03% cpu : usr=0.16%, sys=25.72%, ctx=4313, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=7278: Thu Sep 12 03:19:39 2024 read: IOPS=36, BW=145KiB/s (148kB/s)(6052KiB/41803msec) clat (usec): min=1570, max=8311.4k, avg=17075.33, stdev=236835.41 lat (usec): min=1570, max=8311.4k, avg=17076.29, stdev=236835.39 clat percentiles (usec): | 1.00th=[ 1844], 5.00th=[ 2212], 10.00th=[ 2638], | 20.00th=[ 3589], 30.00th=[ 3851], 40.00th=[ 3982], | 50.00th=[ 4080], 60.00th=[ 4752], 70.00th=[ 8291], | 80.00th=[ 9503], 90.00th=[ 14353], 95.00th=[ 23987], | 99.00th=[ 78119], 99.50th=[ 112722], 99.90th=[3674211], | 99.95th=[8287945], 99.99th=[8287945] bw ( KiB/s): min= 7, max= 407, per=35.28%, avg=202.53, stdev=102.75, samples=59 iops : min= 1, max= 101, avg=50.51, stdev=25.64, samples=59 write: IOPS=37, BW=149KiB/s (153kB/s)(6236KiB/41803msec) clat (msec): min=2, max=154, avg=10.17, stdev=11.73 lat (msec): min=2, max=154, avg=10.17, stdev=11.73 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 4], 10.00th=[ 5], 20.00th=[ 6], | 30.00th=[ 6], 40.00th=[ 6], 50.00th=[ 8], 60.00th=[ 9], | 70.00th=[ 10], 80.00th=[ 12], 90.00th=[ 16], 95.00th=[ 26], | 99.00th=[ 73], 99.50th=[ 96], 99.90th=[ 120], 99.95th=[ 155], | 99.99th=[ 155] bw ( KiB/s): min= 7, max= 424, per=34.83%, avg=209.31, stdev=107.87, samples=59 iops : min= 1, max= 106, avg=52.20, stdev=26.96, samples=59 lat (msec) : 2=1.33%, 4=24.54%, 10=51.01%, 20=16.63%, 50=4.65% lat (msec) : 100=1.33%, 250=0.39% cpu : usr=0.12%, sys=27.74%, ctx=4281, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=574KiB/s (588kB/s), 145KiB/s-381KiB/s (148kB/s-390kB/s), io=23.4MiB (24.6MB), run=15724-41803msec WRITE: bw=601KiB/s (616kB/s), 149KiB/s-400KiB/s (153kB/s-410kB/s), io=24.6MiB (25.7MB), run=15724-41803msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=7339: Thu Sep 12 03:19:49 2024 read: IOPS=44, BW=715KiB/s (732kB/s)(5920KiB/8281msec) clat (usec): min=46, max=30853, avg=11462.72, stdev=5353.53 lat (usec): min=46, max=30853, avg=11463.60, stdev=5353.61 clat percentiles (usec): | 1.00th=[ 56], 5.00th=[ 92], 10.00th=[ 5997], 20.00th=[ 8094], | 30.00th=[ 9503], 40.00th=[10290], 50.00th=[11338], 60.00th=[12387], | 70.00th=[13698], 80.00th=[15008], 90.00th=[17695], 95.00th=[20055], | 99.00th=[25822], 99.50th=[29492], 99.90th=[30802], 99.95th=[30802], | 99.99th=[30802] bw ( KiB/s): min= 588, max= 1403, per=31.17%, avg=901.55, stdev=232.98, samples=11 iops : min= 36, max= 87, avg=56.00, stdev=14.44, samples=11 write: IOPS=48, BW=769KiB/s (787kB/s)(6368KiB/8281msec) clat (usec): min=559, max=2713.0k, avg=10093.09, stdev=135863.48 lat (usec): min=560, max=2713.0k, avg=10094.55, stdev=135863.46 clat percentiles (usec): | 1.00th=[ 627], 5.00th=[ 766], 10.00th=[ 816], | 20.00th=[ 1045], 30.00th=[ 1270], 40.00th=[ 1663], | 50.00th=[ 2040], 60.00th=[ 2769], 70.00th=[ 3982], | 80.00th=[ 4948], 90.00th=[ 7439], 95.00th=[ 9634], | 99.00th=[ 16450], 99.50th=[ 21890], 99.90th=[2701132], | 99.95th=[2701132], 99.99th=[2701132] bw ( KiB/s): min= 640, max= 1556, per=32.06%, avg=975.45, stdev=270.44, samples=11 iops : min= 40, max= 97, avg=60.64, stdev=16.91, samples=11 lat (usec) : 50=0.26%, 100=2.47%, 250=1.30%, 750=2.21%, 1000=7.16% lat (msec) : 2=16.15%, 4=10.94%, 10=26.69%, 20=30.21%, 50=2.47% cpu : usr=0.07%, sys=14.20%, ctx=2617, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=7340: Thu Sep 12 03:19:49 2024 read: IOPS=45, BW=735KiB/s (752kB/s)(5744KiB/7819msec) clat (usec): min=41, max=29208, avg=11081.51, stdev=4921.69 lat (usec): min=42, max=29210, avg=11082.23, stdev=4921.77 clat percentiles (usec): | 1.00th=[ 49], 5.00th=[ 72], 10.00th=[ 6456], 20.00th=[ 8094], | 30.00th=[ 8848], 40.00th=[ 9896], 50.00th=[10814], 60.00th=[11863], | 70.00th=[12911], 80.00th=[14746], 90.00th=[16909], 95.00th=[19530], | 99.00th=[24249], 99.50th=[25035], 99.90th=[29230], 99.95th=[29230], | 99.99th=[29230] bw ( KiB/s): min= 256, max= 1584, per=30.13%, avg=871.45, stdev=352.85, samples=11 iops : min= 16, max= 99, avg=54.27, stdev=22.08, samples=11 write: IOPS=52, BW=837KiB/s (857kB/s)(6544KiB/7819msec) clat (usec): min=554, max=2200.6k, avg=9233.06, stdev=109077.69 lat (usec): min=555, max=2200.6k, avg=9234.24, stdev=109077.67 clat percentiles (usec): | 1.00th=[ 586], 5.00th=[ 652], 10.00th=[ 709], | 20.00th=[ 848], 30.00th=[ 1045], 40.00th=[ 1385], | 50.00th=[ 2114], 60.00th=[ 2900], 70.00th=[ 3752], | 80.00th=[ 5276], 90.00th=[ 7898], 95.00th=[ 10290], | 99.00th=[ 22938], 99.50th=[ 35390], 99.90th=[2197816], | 99.95th=[2197816], 99.99th=[2197816] bw ( KiB/s): min= 448, max= 1600, per=32.42%, avg=986.64, stdev=360.88, samples=11 iops : min= 28, max= 100, avg=61.36, stdev=22.66, samples=11 lat (usec) : 50=0.52%, 100=2.34%, 250=0.52%, 750=6.51%, 1000=8.07% lat (msec) : 2=10.94%, 4=12.11%, 10=27.99%, 20=27.99%, 50=2.73% lat (msec) : 250=0.13% cpu : usr=0.04%, sys=13.66%, ctx=2730, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=7341: Thu Sep 12 03:19:49 2024 read: IOPS=86, BW=1391KiB/s (1424kB/s)(6256KiB/4499msec) clat (usec): min=39, max=35213, avg=8631.49, stdev=3982.48 lat (usec): min=39, max=35214, avg=8632.18, stdev=3982.49 clat percentiles (usec): | 1.00th=[ 46], 5.00th=[ 79], 10.00th=[ 6063], 20.00th=[ 6783], | 30.00th=[ 7177], 40.00th=[ 7635], 50.00th=[ 8225], 60.00th=[ 8717], | 70.00th=[ 9241], 80.00th=[10421], 90.00th=[12649], 95.00th=[15008], | 99.00th=[22676], 99.50th=[32375], 99.90th=[35390], 99.95th=[35390], | 99.99th=[35390] bw ( KiB/s): min= 800, max= 1760, per=47.15%, avg=1363.62, stdev=332.09, samples=8 iops : min= 50, max= 110, avg=85.12, stdev=20.88, samples=8 write: IOPS=83, BW=1341KiB/s (1373kB/s)(6032KiB/4499msec) clat (usec): min=511, max=21899, avg=2842.20, stdev=2623.01 lat (usec): min=512, max=21900, avg=2844.49, stdev=2622.09 clat percentiles (usec): | 1.00th=[ 578], 5.00th=[ 619], 10.00th=[ 717], 20.00th=[ 857], | 30.00th=[ 1483], 40.00th=[ 1876], 50.00th=[ 2114], 60.00th=[ 2442], | 70.00th=[ 3228], 80.00th=[ 4228], 90.00th=[ 5342], 95.00th=[ 7701], | 99.00th=[14091], 99.50th=[19268], 99.90th=[21890], 99.95th=[21890], | 99.99th=[21890] bw ( KiB/s): min= 896, max= 1856, per=43.89%, avg=1335.62, stdev=338.94, samples=8 iops : min= 56, max= 116, avg=83.38, stdev=21.29, samples=8 lat (usec) : 50=1.04%, 100=1.82%, 250=0.26%, 750=6.51%, 1000=5.47% lat (msec) : 2=10.29%, 4=15.76%, 10=46.09%, 20=11.59%, 50=1.17% cpu : usr=0.13%, sys=19.36%, ctx=3226, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=7342: Thu Sep 12 03:19:49 2024 read: IOPS=82, BW=1319KiB/s (1351kB/s)(6032KiB/4572msec) clat (usec): min=45, max=37262, avg=8851.37, stdev=3715.56 lat (usec): min=46, max=37263, avg=8851.97, stdev=3715.54 clat percentiles (usec): | 1.00th=[ 58], 5.00th=[ 5800], 10.00th=[ 6259], 20.00th=[ 6718], | 30.00th=[ 7242], 40.00th=[ 7701], 50.00th=[ 8160], 60.00th=[ 8586], | 70.00th=[ 9241], 80.00th=[10683], 90.00th=[12518], 95.00th=[15664], | 99.00th=[23725], 99.50th=[23987], 99.90th=[37487], 99.95th=[37487], | 99.99th=[37487] bw ( KiB/s): min= 704, max= 1728, per=45.35%, avg=1311.44, stdev=327.99, samples=9 iops : min= 44, max= 108, avg=81.78, stdev=20.57, samples=9 write: IOPS=85, BW=1368KiB/s (1401kB/s)(6256KiB/4572msec) clat (usec): min=550, max=41430, avg=2966.23, stdev=3542.01 lat (usec): min=551, max=41431, avg=2967.51, stdev=3542.47 clat percentiles (usec): | 1.00th=[ 578], 5.00th=[ 635], 10.00th=[ 676], 20.00th=[ 816], | 30.00th=[ 1156], 40.00th=[ 1811], 50.00th=[ 2114], 60.00th=[ 2409], | 70.00th=[ 3064], 80.00th=[ 4080], 90.00th=[ 5342], 95.00th=[ 9503], | 99.00th=[15926], 99.50th=[24249], 99.90th=[41681], 99.95th=[41681], | 99.99th=[41681] bw ( KiB/s): min= 800, max= 1916, per=44.84%, avg=1364.56, stdev=369.68, samples=9 iops : min= 50, max= 119, avg=85.11, stdev=23.06, samples=9 lat (usec) : 50=0.39%, 100=1.04%, 750=8.07%, 1000=6.25% lat (msec) : 2=9.64%, 4=16.54%, 10=43.88%, 20=12.76%, 50=1.43% cpu : usr=0.02%, sys=19.10%, ctx=3018, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2892KiB/s (2962kB/s), 715KiB/s-1391KiB/s (732kB/s-1424kB/s), io=23.4MiB (24.5MB), run=4499-8281msec WRITE: bw=3043KiB/s (3116kB/s), 769KiB/s-1368KiB/s (787kB/s-1401kB/s), io=24.6MiB (25.8MB), run=4499-8281msec rand-rw: (groupid=0, jobs=1): err= 0: pid=7334: Thu Sep 12 03:19:51 2024 read: IOPS=97, BW=1556KiB/s (1594kB/s)(5920KiB/3804msec) clat (usec): min=1478, max=19724, avg=3910.01, stdev=2979.65 lat (usec): min=1478, max=19724, avg=3910.60, stdev=2979.69 clat percentiles (usec): | 1.00th=[ 1532], 5.00th=[ 1680], 10.00th=[ 1745], 20.00th=[ 1893], | 30.00th=[ 2073], 40.00th=[ 2278], 50.00th=[ 2606], 60.00th=[ 3228], | 70.00th=[ 4228], 80.00th=[ 5538], 90.00th=[ 7963], 95.00th=[10028], | 99.00th=[15795], 99.50th=[18744], 99.90th=[19792], 99.95th=[19792], | 99.99th=[19792] bw ( KiB/s): min= 896, max= 1952, per=69.37%, avg=1494.86, stdev=390.05, samples=7 iops : min= 56, max= 122, avg=93.43, stdev=24.38, samples=7 write: IOPS=104, BW=1674KiB/s (1714kB/s)(6368KiB/3804msec) clat (usec): min=2114, max=52147, avg=5889.48, stdev=4604.29 lat (usec): min=2114, max=52148, avg=5890.52, stdev=4604.50 clat percentiles (usec): | 1.00th=[ 2212], 5.00th=[ 2507], 10.00th=[ 2802], 20.00th=[ 3097], | 30.00th=[ 3458], 40.00th=[ 3884], 50.00th=[ 4752], 60.00th=[ 5342], | 70.00th=[ 6456], 80.00th=[ 7635], 90.00th=[10552], 95.00th=[12256], | 99.00th=[28443], 99.50th=[33817], 99.90th=[52167], 99.95th=[52167], | 99.99th=[52167] bw ( KiB/s): min= 1024, max= 2240, per=72.19%, avg=1636.57, stdev=476.94, samples=7 iops : min= 64, max= 140, avg=102.29, stdev=29.81, samples=7 lat (msec) : 2=12.63%, 4=41.54%, 10=37.76%, 20=7.03%, 50=0.91% lat (msec) : 100=0.13% cpu : usr=0.24%, sys=11.52%, ctx=939, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=7335: Thu Sep 12 03:19:51 2024 read: IOPS=34, BW=552KiB/s (565kB/s)(5744KiB/10407msec) clat (usec): min=1671, max=199760, avg=9897.21, stdev=16821.63 lat (usec): min=1672, max=199763, avg=9898.03, stdev=16821.72 clat percentiles (usec): | 1.00th=[ 1713], 5.00th=[ 1958], 10.00th=[ 2278], 20.00th=[ 2802], | 30.00th=[ 3654], 40.00th=[ 4359], 50.00th=[ 5211], 60.00th=[ 7111], | 70.00th=[ 9110], 80.00th=[ 11469], 90.00th=[ 19530], 95.00th=[ 30540], | 99.00th=[ 90702], 99.50th=[147850], 99.90th=[200279], 99.95th=[200279], | 99.99th=[200279] bw ( KiB/s): min= 160, max= 1312, per=30.73%, avg=662.29, stdev=291.08, samples=17 iops : min= 10, max= 82, avg=41.35, stdev=18.17, samples=17 write: IOPS=39, BW=629KiB/s (644kB/s)(6544KiB/10407msec) clat (msec): min=2, max=2058, avg=16.68, stdev=102.60 lat (msec): min=2, max=2058, avg=16.68, stdev=102.60 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 4], 10.00th=[ 4], 20.00th=[ 6], | 30.00th=[ 6], 40.00th=[ 7], 50.00th=[ 8], 60.00th=[ 10], | 70.00th=[ 11], 80.00th=[ 15], 90.00th=[ 20], 95.00th=[ 31], | 99.00th=[ 104], 99.50th=[ 171], 99.90th=[ 2056], 99.95th=[ 2056], | 99.99th=[ 2056] bw ( KiB/s): min= 160, max= 1568, per=33.20%, avg=752.59, stdev=337.23, samples=17 iops : min= 10, max= 98, avg=47.00, stdev=21.07, samples=17 lat (msec) : 2=2.86%, 4=19.27%, 10=47.01%, 20=21.22%, 50=7.16% lat (msec) : 100=1.56%, 250=0.78% cpu : usr=0.30%, sys=30.62%, ctx=1214, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=7336: Thu Sep 12 03:19:51 2024 read: IOPS=84, BW=1351KiB/s (1383kB/s)(6256KiB/4632msec) clat (usec): min=1515, max=34597, avg=4733.82, stdev=4278.80 lat (usec): min=1515, max=34597, avg=4734.41, stdev=4278.88 clat percentiles (usec): | 1.00th=[ 1565], 5.00th=[ 1696], 10.00th=[ 1795], 20.00th=[ 1991], | 30.00th=[ 2212], 40.00th=[ 2507], 50.00th=[ 3064], 60.00th=[ 3982], | 70.00th=[ 5211], 80.00th=[ 7111], 90.00th=[ 8848], 95.00th=[12256], | 99.00th=[27657], 99.50th=[33162], 99.90th=[34341], 99.95th=[34341], | 99.99th=[34341] bw ( KiB/s): min= 704, max= 1632, per=62.36%, avg=1343.78, stdev=353.00, samples=9 iops : min= 44, max= 102, avg=83.89, stdev=22.18, samples=9 write: IOPS=81, BW=1302KiB/s (1333kB/s)(6032KiB/4632msec) clat (usec): min=2286, max=50170, avg=7278.16, stdev=4983.79 lat (usec): min=2286, max=50171, avg=7279.12, stdev=4983.89 clat percentiles (usec): | 1.00th=[ 2343], 5.00th=[ 2769], 10.00th=[ 3130], 20.00th=[ 3949], | 30.00th=[ 4948], 40.00th=[ 5669], 50.00th=[ 6456], 60.00th=[ 6980], | 70.00th=[ 7832], 80.00th=[ 9110], 90.00th=[11731], 95.00th=[15008], | 99.00th=[26346], 99.50th=[42730], 99.90th=[50070], 99.95th=[50070], | 99.99th=[50070] bw ( KiB/s): min= 768, max= 1888, per=55.51%, avg=1258.33, stdev=391.13, samples=9 iops : min= 48, max= 118, avg=78.56, stdev=24.50, samples=9 lat (msec) : 2=10.29%, 4=30.47%, 10=48.18%, 20=8.85%, 50=2.08% lat (msec) : 100=0.13% cpu : usr=0.22%, sys=13.13%, ctx=1172, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=7337: Thu Sep 12 03:19:51 2024 read: IOPS=33, BW=543KiB/s (556kB/s)(6032KiB/11112msec) clat (usec): min=1557, max=320173, avg=10227.98, stdev=23125.14 lat (usec): min=1558, max=320174, avg=10228.94, stdev=23125.16 clat percentiles (usec): | 1.00th=[ 1647], 5.00th=[ 2147], 10.00th=[ 2999], 20.00th=[ 3785], | 30.00th=[ 4228], 40.00th=[ 4490], 50.00th=[ 6194], 60.00th=[ 8160], | 70.00th=[ 9110], 80.00th=[ 10552], 90.00th=[ 16319], 95.00th=[ 21890], | 99.00th=[ 91751], 99.50th=[270533], 99.90th=[320865], 99.95th=[320865], | 99.99th=[320865] bw ( KiB/s): min= 160, max= 1344, per=32.37%, avg=697.65, stdev=308.17, samples=17 iops : min= 10, max= 84, avg=43.53, stdev=19.23, samples=17 write: IOPS=35, BW=563KiB/s (577kB/s)(6256KiB/11112msec) clat (msec): min=2, max=2886, avg=18.42, stdev=145.81 lat (msec): min=2, max=2886, avg=18.43, stdev=145.81 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 5], 10.00th=[ 5], 20.00th=[ 6], | 30.00th=[ 6], 40.00th=[ 7], 50.00th=[ 8], 60.00th=[ 11], | 70.00th=[ 11], 80.00th=[ 14], 90.00th=[ 21], 95.00th=[ 33], | 99.00th=[ 64], 99.50th=[ 96], 99.90th=[ 2903], 99.95th=[ 2903], | 99.99th=[ 2903] bw ( KiB/s): min= 256, max= 1344, per=32.11%, avg=727.82, stdev=285.46, samples=17 iops : min= 16, max= 84, avg=45.41, stdev=17.83, samples=17 lat (msec) : 2=1.69%, 4=12.24%, 10=55.08%, 20=23.31%, 50=5.86% lat (msec) : 100=1.30%, 250=0.13%, 500=0.26% cpu : usr=0.04%, sys=30.55%, ctx=1316, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2156KiB/s (2207kB/s), 543KiB/s-1556KiB/s (556kB/s-1594kB/s), io=23.4MiB (24.5MB), run=3804-11112msec WRITE: bw=2268KiB/s (2322kB/s), 563KiB/s-1674KiB/s (577kB/s-1714kB/s), io=24.6MiB (25.8MB), run=3804-11112msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=7357: Thu Sep 12 03:19:55 2024 read: IOPS=2, BW=2491KiB/s (2551kB/s)(7168KiB/2877msec) clat (usec): min=253, max=882852, avg=249843.01, stdev=323895.19 lat (usec): min=253, max=882853, avg=249843.81, stdev=323895.12 clat percentiles (usec): | 1.00th=[ 253], 5.00th=[ 253], 10.00th=[ 253], 20.00th=[ 416], | 30.00th=[ 64226], 40.00th=[ 64226], 50.00th=[ 67634], 60.00th=[299893], | 70.00th=[299893], 80.00th=[434111], 90.00th=[884999], 95.00th=[884999], | 99.00th=[884999], 99.50th=[884999], 99.90th=[884999], 99.95th=[884999], | 99.99th=[884999] bw ( KiB/s): min= 2048, max= 6144, per=57.56%, avg=4096.00, stdev=2048.00, samples=3 iops : min= 2, max= 6, avg= 4.00, stdev= 2.00, samples=3 write: IOPS=1, BW=1780KiB/s (1822kB/s)(5120KiB/2877msec) clat (msec): min=18, max=856, avg=204.16, stdev=364.83 lat (msec): min=18, max=856, avg=204.20, stdev=364.82 clat percentiles (msec): | 1.00th=[ 18], 5.00th=[ 18], 10.00th=[ 18], 20.00th=[ 18], | 30.00th=[ 40], 40.00th=[ 40], 50.00th=[ 45], 60.00th=[ 45], | 70.00th=[ 64], 80.00th=[ 64], 90.00th=[ 860], 95.00th=[ 860], | 99.00th=[ 860], 99.50th=[ 860], 99.90th=[ 860], 99.95th=[ 860], | 99.99th=[ 860] bw ( KiB/s): min= 2043, max= 6144, per=34.25%, avg=3411.67, stdev=2366.27, samples=3 iops : min= 1, max= 6, avg= 3.00, stdev= 2.65, samples=3 lat (usec) : 500=16.67% lat (msec) : 20=8.33%, 50=16.67%, 100=25.00%, 500=16.67%, 1000=16.67% cpu : usr=0.00%, sys=15.23%, ctx=808, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=7358: Thu Sep 12 03:19:55 2024 read: IOPS=3, BW=3080KiB/s (3154kB/s)(2048KiB/665msec) clat (usec): min=406, max=101718, avg=51062.68, stdev=71638.24 lat (usec): min=407, max=101720, avg=51064.01, stdev=71639.17 clat percentiles (usec): | 1.00th=[ 408], 5.00th=[ 408], 10.00th=[ 408], 20.00th=[ 408], | 30.00th=[ 408], 40.00th=[ 408], 50.00th=[ 408], 60.00th=[102237], | 70.00th=[102237], 80.00th=[102237], 90.00th=[102237], 95.00th=[102237], | 99.00th=[102237], 99.50th=[102237], 99.90th=[102237], 99.95th=[102237], | 99.99th=[102237] bw ( KiB/s): min= 4096, max= 4096, per=57.56%, avg=4096.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 write: IOPS=15, BW=15.0MiB/s (15.8MB/s)(10.0MiB/665msec) clat (msec): min=16, max=199, avg=50.29, stdev=55.38 lat (msec): min=16, max=200, avg=50.34, stdev=55.38 clat percentiles (msec): | 1.00th=[ 17], 5.00th=[ 17], 10.00th=[ 17], 20.00th=[ 17], | 30.00th=[ 18], 40.00th=[ 27], 50.00th=[ 27], 60.00th=[ 35], | 70.00th=[ 59], 80.00th=[ 59], 90.00th=[ 68], 95.00th=[ 201], | 99.00th=[ 201], 99.50th=[ 201], 99.90th=[ 201], 99.95th=[ 201], | 99.99th=[ 201] bw ( KiB/s): min=16384, max=16384, per=100.00%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 16, max= 16, avg=16.00, stdev= 0.00, samples=1 lat (usec) : 500=8.33% lat (msec) : 20=25.00%, 50=33.33%, 100=16.67%, 250=16.67% cpu : usr=0.00%, sys=44.43%, ctx=152, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=7359: Thu Sep 12 03:19:55 2024 read: IOPS=2, BW=2136KiB/s (2188kB/s)(6144KiB/2876msec) clat (usec): min=343, max=922050, avg=420643.78, stdev=448510.22 lat (usec): min=343, max=922051, avg=420644.39, stdev=448510.26 clat percentiles (usec): | 1.00th=[ 343], 5.00th=[ 343], 10.00th=[ 343], 20.00th=[ 8029], | 30.00th=[ 8029], 40.00th=[ 50070], 50.00th=[ 50070], 60.00th=[666895], | 70.00th=[876610], 80.00th=[876610], 90.00th=[918553], 95.00th=[918553], | 99.00th=[918553], 99.50th=[918553], 99.90th=[918553], 99.95th=[918553], | 99.99th=[918553] bw ( KiB/s): min= 2048, max= 6144, per=47.97%, avg=3413.33, stdev=2364.83, samples=3 iops : min= 2, max= 6, avg= 3.33, stdev= 2.31, samples=3 write: IOPS=2, BW=2136KiB/s (2188kB/s)(6144KiB/2876msec) clat (usec): min=45796, max=84497, avg=56044.30, stdev=14618.55 lat (usec): min=45813, max=84570, avg=56103.57, stdev=14632.02 clat percentiles (usec): | 1.00th=[45876], 5.00th=[45876], 10.00th=[45876], 20.00th=[47449], | 30.00th=[47449], 40.00th=[49546], 50.00th=[49546], 60.00th=[50070], | 70.00th=[58459], 80.00th=[58459], 90.00th=[84411], 95.00th=[84411], | 99.00th=[84411], 99.50th=[84411], 99.90th=[84411], 99.95th=[84411], | 99.99th=[84411] bw ( KiB/s): min= 2048, max= 8192, per=41.12%, avg=4096.00, stdev=3547.24, samples=3 iops : min= 2, max= 8, avg= 4.00, stdev= 3.46, samples=3 lat (usec) : 500=8.33% lat (msec) : 10=8.33%, 50=25.00%, 100=33.33%, 750=8.33%, 1000=16.67% cpu : usr=0.00%, sys=16.73%, ctx=839, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=7360: Thu Sep 12 03:19:55 2024 read: IOPS=1, BW=1779KiB/s (1822kB/s)(5120KiB/2878msec) clat (msec): min=5, max=834, avg=450.91, stdev=397.83 lat (msec): min=5, max=834, avg=450.91, stdev=397.83 clat percentiles (msec): | 1.00th=[ 6], 5.00th=[ 6], 10.00th=[ 6], 20.00th=[ 6], | 30.00th=[ 36], 40.00th=[ 36], 50.00th=[ 676], 60.00th=[ 676], | 70.00th=[ 709], 80.00th=[ 709], 90.00th=[ 835], 95.00th=[ 835], | 99.00th=[ 835], 99.50th=[ 835], 99.90th=[ 835], 99.95th=[ 835], | 99.99th=[ 835] bw ( KiB/s): min= 2043, max= 4087, per=38.31%, avg=2726.00, stdev=1178.66, samples=3 iops : min= 1, max= 3, avg= 2.00, stdev= 1.00, samples=3 write: IOPS=2, BW=2491KiB/s (2550kB/s)(7168KiB/2878msec) clat (msec): min=31, max=245, avg=79.60, stdev=78.52 lat (msec): min=31, max=245, avg=79.66, stdev=78.52 clat percentiles (msec): | 1.00th=[ 32], 5.00th=[ 32], 10.00th=[ 32], 20.00th=[ 32], | 30.00th=[ 41], 40.00th=[ 41], 50.00th=[ 42], 60.00th=[ 53], | 70.00th=[ 53], 80.00th=[ 115], 90.00th=[ 245], 95.00th=[ 245], | 99.00th=[ 245], 99.50th=[ 245], 99.90th=[ 245], 99.95th=[ 245], | 99.99th=[ 245] bw ( KiB/s): min= 6144, max= 8192, per=71.95%, avg=7168.00, stdev=1448.15, samples=2 iops : min= 6, max= 8, avg= 7.00, stdev= 1.41, samples=2 lat (msec) : 10=8.33%, 50=41.67%, 100=8.33%, 250=16.67%, 750=16.67% lat (msec) : 1000=8.33% cpu : usr=0.00%, sys=16.27%, ctx=778, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=7116KiB/s (7287kB/s), 1779KiB/s-3080KiB/s (1822kB/s-3154kB/s), io=20.0MiB (20.0MB), run=665-2878msec WRITE: bw=9962KiB/s (10.2MB/s), 1780KiB/s-15.0MiB/s (1822kB/s-15.8MB/s), io=28.0MiB (29.4MB), run=665-2878msec rand-rw: (groupid=0, jobs=1): err= 0: pid=7361: Thu Sep 12 03:19:55 2024 read: IOPS=2, BW=2500KiB/s (2560kB/s)(7168KiB/2867msec) clat (msec): min=22, max=671, avg=208.42, stdev=268.56 lat (msec): min=22, max=671, avg=208.42, stdev=268.56 clat percentiles (msec): | 1.00th=[ 23], 5.00th=[ 23], 10.00th=[ 23], 20.00th=[ 24], | 30.00th=[ 49], 40.00th=[ 49], 50.00th=[ 77], 60.00th=[ 101], | 70.00th=[ 101], 80.00th=[ 518], 90.00th=[ 676], 95.00th=[ 676], | 99.00th=[ 676], 99.50th=[ 676], 99.90th=[ 676], 99.95th=[ 676], | 99.99th=[ 676] bw ( KiB/s): min= 1831, max= 8192, per=56.90%, avg=4023.67, stdev=3611.51, samples=3 iops : min= 1, max= 8, avg= 3.67, stdev= 3.79, samples=3 write: IOPS=1, BW=1786KiB/s (1829kB/s)(5120KiB/2867msec) clat (msec): min=28, max=1017, avg=253.77, stdev=428.38 lat (msec): min=29, max=1017, avg=253.85, stdev=428.37 clat percentiles (msec): | 1.00th=[ 29], 5.00th=[ 29], 10.00th=[ 29], 20.00th=[ 29], | 30.00th=[ 34], 40.00th=[ 34], 50.00th=[ 70], 60.00th=[ 70], | 70.00th=[ 121], 80.00th=[ 121], 90.00th=[ 1020], 95.00th=[ 1020], | 99.00th=[ 1020], 99.50th=[ 1020], 99.90th=[ 1020], 99.95th=[ 1020], | 99.99th=[ 1020] bw ( KiB/s): min= 2048, max= 3663, per=31.56%, avg=3124.67, stdev=932.42, samples=3 iops : min= 2, max= 3, avg= 2.67, stdev= 0.58, samples=3 lat (msec) : 50=41.67%, 100=16.67%, 250=16.67%, 750=16.67% cpu : usr=0.07%, sys=6.28%, ctx=72, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=7362: Thu Sep 12 03:19:55 2024 read: IOPS=1, BW=1144KiB/s (1172kB/s)(2048KiB/1790msec) clat (msec): min=60, max=305, avg=182.63, stdev=173.10 lat (msec): min=60, max=305, avg=182.63, stdev=173.10 clat percentiles (msec): | 1.00th=[ 61], 5.00th=[ 61], 10.00th=[ 61], 20.00th=[ 61], | 30.00th=[ 61], 40.00th=[ 61], 50.00th=[ 61], 60.00th=[ 305], | 70.00th=[ 305], 80.00th=[ 305], 90.00th=[ 305], 95.00th=[ 305], | 99.00th=[ 305], 99.50th=[ 305], 99.90th=[ 305], 99.95th=[ 305], | 99.99th=[ 305] bw ( KiB/s): min= 2048, max= 2048, per=28.96%, avg=2048.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 write: IOPS=5, BW=5721KiB/s (5858kB/s)(10.0MiB/1790msec) clat (msec): min=20, max=855, avg=134.25, stdev=254.31 lat (msec): min=20, max=855, avg=134.31, stdev=254.30 clat percentiles (msec): | 1.00th=[ 21], 5.00th=[ 21], 10.00th=[ 21], 20.00th=[ 29], | 30.00th=[ 47], 40.00th=[ 50], 50.00th=[ 50], 60.00th=[ 58], | 70.00th=[ 71], 80.00th=[ 71], 90.00th=[ 102], 95.00th=[ 852], | 99.00th=[ 852], 99.50th=[ 852], 99.90th=[ 852], 99.95th=[ 852], | 99.99th=[ 852] bw ( KiB/s): min= 8192, max=10240, per=93.09%, avg=9216.00, stdev=1448.15, samples=2 iops : min= 8, max= 10, avg= 9.00, stdev= 1.41, samples=2 lat (msec) : 50=41.67%, 100=33.33%, 250=8.33%, 500=8.33%, 1000=8.33% cpu : usr=0.11%, sys=12.07%, ctx=77, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=7363: Thu Sep 12 03:19:55 2024 read: IOPS=2, BW=2147KiB/s (2198kB/s)(6144KiB/2862msec) clat (msec): min=25, max=893, avg=288.06, stdev=380.57 lat (msec): min=25, max=893, avg=288.06, stdev=380.57 clat percentiles (msec): | 1.00th=[ 26], 5.00th=[ 26], 10.00th=[ 26], 20.00th=[ 44], | 30.00th=[ 44], 40.00th=[ 51], 50.00th=[ 51], 60.00th=[ 73], | 70.00th=[ 642], 80.00th=[ 642], 90.00th=[ 894], 95.00th=[ 894], | 99.00th=[ 894], 99.50th=[ 894], 99.90th=[ 894], 99.95th=[ 894], | 99.99th=[ 894] bw ( KiB/s): min= 2043, max= 6144, per=48.25%, avg=3411.67, stdev=2366.27, samples=3 iops : min= 1, max= 6, avg= 3.00, stdev= 2.65, samples=3 write: IOPS=2, BW=2147KiB/s (2198kB/s)(6144KiB/2862msec) clat (msec): min=20, max=550, avg=174.60, stdev=221.82 lat (msec): min=20, max=550, avg=174.64, stdev=221.81 clat percentiles (msec): | 1.00th=[ 21], 5.00th=[ 21], 10.00th=[ 21], 20.00th=[ 27], | 30.00th=[ 27], 40.00th=[ 39], 50.00th=[ 39], 60.00th=[ 67], | 70.00th=[ 347], 80.00th=[ 347], 90.00th=[ 550], 95.00th=[ 550], | 99.00th=[ 550], 99.50th=[ 550], 99.90th=[ 550], 99.95th=[ 550], | 99.99th=[ 550] bw ( KiB/s): min= 4096, max= 4096, per=41.37%, avg=4096.00, stdev= 0.00, samples=3 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=3 lat (msec) : 50=41.67%, 100=25.00%, 500=8.33%, 750=16.67%, 1000=8.33% cpu : usr=0.07%, sys=5.77%, ctx=90, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=7364: Thu Sep 12 03:19:55 2024 read: IOPS=1, BW=1768KiB/s (1810kB/s)(5120KiB/2896msec) clat (msec): min=36, max=843, avg=354.72, stdev=373.06 lat (msec): min=36, max=843, avg=354.72, stdev=373.06 clat percentiles (msec): | 1.00th=[ 37], 5.00th=[ 37], 10.00th=[ 37], 20.00th=[ 37], | 30.00th=[ 106], 40.00th=[ 106], 50.00th=[ 120], 60.00th=[ 120], | 70.00th=[ 667], 80.00th=[ 667], 90.00th=[ 844], 95.00th=[ 844], | 99.00th=[ 844], 99.50th=[ 844], 99.90th=[ 844], 99.95th=[ 844], | 99.99th=[ 844] bw ( KiB/s): min= 4087, max= 4096, per=57.86%, avg=4091.50, stdev= 6.36, samples=2 iops : min= 3, max= 4, avg= 3.50, stdev= 0.71, samples=2 write: IOPS=2, BW=2475KiB/s (2535kB/s)(7168KiB/2896msec) clat (msec): min=26, max=530, avg=155.40, stdev=182.70 lat (msec): min=26, max=530, avg=155.46, stdev=182.69 clat percentiles (msec): | 1.00th=[ 27], 5.00th=[ 27], 10.00th=[ 27], 20.00th=[ 30], | 30.00th=[ 54], 40.00th=[ 54], 50.00th=[ 91], 60.00th=[ 102], | 70.00th=[ 102], 80.00th=[ 255], 90.00th=[ 531], 95.00th=[ 531], | 99.00th=[ 531], 99.50th=[ 531], 99.90th=[ 531], 99.95th=[ 531], | 99.99th=[ 531] bw ( KiB/s): min= 2043, max= 6144, per=48.25%, avg=4777.00, stdev=2367.71, samples=3 iops : min= 1, max= 6, avg= 4.33, stdev= 2.89, samples=3 lat (msec) : 50=25.00%, 100=16.67%, 250=25.00%, 500=8.33%, 750=16.67% lat (msec) : 1000=8.33% cpu : usr=0.03%, sys=5.87%, ctx=75, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=7072KiB/s (7242kB/s), 1144KiB/s-2500KiB/s (1172kB/s-2560kB/s), io=20.0MiB (20.0MB), run=1790-2896msec WRITE: bw=9901KiB/s (10.1MB/s), 1786KiB/s-5721KiB/s (1829kB/s-5858kB/s), io=28.0MiB (29.4MB), run=1790-2896msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=7372: Thu Sep 12 03:19:59 2024 read: IOPS=1, BW=6059KiB/s (6205kB/s)(12.0MiB/2028msec) clat (msec): min=158, max=1634, avg=662.98, stdev=841.92 lat (msec): min=158, max=1634, avg=662.98, stdev=841.92 clat percentiles (msec): | 1.00th=[ 159], 5.00th=[ 159], 10.00th=[ 159], 20.00th=[ 159], | 30.00th=[ 159], 40.00th=[ 197], 50.00th=[ 197], 60.00th=[ 197], | 70.00th=[ 1636], 80.00th=[ 1636], 90.00th=[ 1636], 95.00th=[ 1636], | 99.00th=[ 1636], 99.50th=[ 1636], 99.90th=[ 1636], 99.95th=[ 1636], | 99.99th=[ 1636] bw ( KiB/s): min= 8015, max= 8192, per=100.00%, avg=8103.50, stdev=125.16, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 250=66.67% cpu : usr=0.00%, sys=50.57%, ctx=511, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=7373: Thu Sep 12 03:19:59 2024 read: IOPS=0, BW=1716KiB/s (1757kB/s)(4096KiB/2387msec) clat (nsec): min=189348k, max=189348k, avg=189348094.00, stdev= 0.00 lat (nsec): min=189350k, max=189350k, avg=189350350.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 190], 5.00th=[ 190], 10.00th=[ 190], 20.00th=[ 190], | 30.00th=[ 190], 40.00th=[ 190], 50.00th=[ 190], 60.00th=[ 190], | 70.00th=[ 190], 80.00th=[ 190], 90.00th=[ 190], 95.00th=[ 190], | 99.00th=[ 190], 99.50th=[ 190], 99.90th=[ 190], 99.95th=[ 190], | 99.99th=[ 190] bw ( KiB/s): min= 8175, max= 8175, per=100.00%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=0, BW=3432KiB/s (3514kB/s)(8192KiB/2387msec) clat (msec): min=123, max=2044, avg=1083.81, stdev=1357.96 lat (msec): min=123, max=2044, avg=1084.05, stdev=1358.00 clat percentiles (msec): | 1.00th=[ 124], 5.00th=[ 124], 10.00th=[ 124], 20.00th=[ 124], | 30.00th=[ 124], 40.00th=[ 124], 50.00th=[ 124], 60.00th=[ 2039], | 70.00th=[ 2039], 80.00th=[ 2039], 90.00th=[ 2039], 95.00th=[ 2039], | 99.00th=[ 2039], 99.50th=[ 2039], 99.90th=[ 2039], 99.95th=[ 2039], | 99.99th=[ 2039] bw ( KiB/s): min= 8175, max= 8175, per=59.55%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67% cpu : usr=0.00%, sys=13.50%, ctx=100, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=7374: Thu Sep 12 03:19:59 2024 write: IOPS=1, BW=5213KiB/s (5339kB/s)(12.0MiB/2357msec) clat (msec): min=121, max=1964, avg=778.05, stdev=1029.30 lat (msec): min=121, max=1964, avg=778.34, stdev=1029.24 clat percentiles (msec): | 1.00th=[ 122], 5.00th=[ 122], 10.00th=[ 122], 20.00th=[ 122], | 30.00th=[ 122], 40.00th=[ 249], 50.00th=[ 249], 60.00th=[ 249], | 70.00th=[ 1972], 80.00th=[ 1972], 90.00th=[ 1972], 95.00th=[ 1972], | 99.00th=[ 1972], 99.50th=[ 1972], 99.90th=[ 1972], 99.95th=[ 1972], | 99.99th=[ 1972] bw ( KiB/s): min= 8192, max= 8192, per=59.68%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67% cpu : usr=0.00%, sys=12.56%, ctx=32, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=7375: Thu Sep 12 03:19:59 2024 write: IOPS=4, BW=18.3MiB/s (19.2MB/s)(12.0MiB/655msec) clat (msec): min=111, max=394, avg=211.91, stdev=158.23 lat (msec): min=112, max=394, avg=212.23, stdev=158.25 clat percentiles (msec): | 1.00th=[ 112], 5.00th=[ 112], 10.00th=[ 112], 20.00th=[ 112], | 30.00th=[ 112], 40.00th=[ 130], 50.00th=[ 130], 60.00th=[ 130], | 70.00th=[ 397], 80.00th=[ 397], 90.00th=[ 397], 95.00th=[ 397], | 99.00th=[ 397], 99.50th=[ 397], 99.90th=[ 397], 99.95th=[ 397], | 99.99th=[ 397] bw ( KiB/s): min=16384, max=16384, per=100.00%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.00%, sys=39.30%, ctx=55, majf=0, minf=28 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=6864KiB/s (7029kB/s), 1716KiB/s-6059KiB/s (1757kB/s-6205kB/s), io=16.0MiB (16.8MB), run=2028-2387msec WRITE: bw=13.4MiB/s (14.1MB/s), 3432KiB/s-18.3MiB/s (3514kB/s-19.2MB/s), io=32.0MiB (33.6MB), run=655-2387msec rand-rw: (groupid=0, jobs=1): err= 0: pid=7376: Thu Sep 12 03:19:59 2024 read: IOPS=1, BW=4855KiB/s (4972kB/s)(12.0MiB/2531msec) clat (msec): min=65, max=2044, avg=831.22, stdev=1062.80 lat (msec): min=65, max=2044, avg=831.22, stdev=1062.80 clat percentiles (msec): | 1.00th=[ 66], 5.00th=[ 66], 10.00th=[ 66], 20.00th=[ 66], | 30.00th=[ 66], 40.00th=[ 384], 50.00th=[ 384], 60.00th=[ 384], | 70.00th=[ 2039], 80.00th=[ 2039], 90.00th=[ 2039], 95.00th=[ 2039], | 99.00th=[ 2039], 99.50th=[ 2039], 99.90th=[ 2039], 99.95th=[ 2039], | 99.99th=[ 2039] bw ( KiB/s): min= 7742, max= 8175, per=100.00%, avg=7958.50, stdev=306.18, samples=2 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=2 lat (msec) : 100=33.33%, 500=33.33% cpu : usr=0.12%, sys=4.58%, ctx=58, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=7377: Thu Sep 12 03:19:59 2024 read: IOPS=0, BW=1591KiB/s (1629kB/s)(4096KiB/2574msec) clat (nsec): min=467653k, max=467653k, avg=467653226.00, stdev= 0.00 lat (nsec): min=467654k, max=467654k, avg=467654385.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 468], 5.00th=[ 468], 10.00th=[ 468], 20.00th=[ 468], | 30.00th=[ 468], 40.00th=[ 468], 50.00th=[ 468], 60.00th=[ 468], | 70.00th=[ 468], 80.00th=[ 468], 90.00th=[ 468], 95.00th=[ 468], | 99.00th=[ 468], 99.50th=[ 468], 99.90th=[ 468], 99.95th=[ 468], | 99.99th=[ 468] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=0, BW=3183KiB/s (3259kB/s)(8192KiB/2574msec) clat (msec): min=120, max=1939, avg=1029.69, stdev=1286.13 lat (msec): min=120, max=1939, avg=1029.94, stdev=1286.34 clat percentiles (msec): | 1.00th=[ 121], 5.00th=[ 121], 10.00th=[ 121], 20.00th=[ 121], | 30.00th=[ 121], 40.00th=[ 121], 50.00th=[ 121], 60.00th=[ 1938], | 70.00th=[ 1938], 80.00th=[ 1938], 90.00th=[ 1938], 95.00th=[ 1938], | 99.00th=[ 1938], 99.50th=[ 1938], 99.90th=[ 1938], 99.95th=[ 1938], | 99.99th=[ 1938] bw ( KiB/s): min= 8192, max= 8192, per=64.35%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=33.33% cpu : usr=0.16%, sys=4.16%, ctx=52, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=7378: Thu Sep 12 03:19:59 2024 write: IOPS=1, BW=4774KiB/s (4888kB/s)(12.0MiB/2574msec) clat (msec): min=58, max=2048, avg=845.20, stdev=1058.68 lat (msec): min=58, max=2049, avg=845.42, stdev=1058.76 clat percentiles (msec): | 1.00th=[ 59], 5.00th=[ 59], 10.00th=[ 59], 20.00th=[ 59], | 30.00th=[ 59], 40.00th=[ 430], 50.00th=[ 430], 60.00th=[ 430], | 70.00th=[ 2056], 80.00th=[ 2056], 90.00th=[ 2056], 95.00th=[ 2056], | 99.00th=[ 2056], 99.50th=[ 2056], 99.90th=[ 2056], 99.95th=[ 2056], | 99.99th=[ 2056] bw ( KiB/s): min= 7742, max= 8175, per=62.52%, avg=7958.50, stdev=306.18, samples=2 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=2 lat (msec) : 100=33.33%, 500=33.33% cpu : usr=0.12%, sys=2.99%, ctx=33, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=7379: Thu Sep 12 03:19:59 2024 write: IOPS=1, BW=5483KiB/s (5615kB/s)(12.0MiB/2241msec) clat (msec): min=147, max=1851, avg=738.79, stdev=964.35 lat (msec): min=147, max=1851, avg=738.99, stdev=964.26 clat percentiles (msec): | 1.00th=[ 148], 5.00th=[ 148], 10.00th=[ 148], 20.00th=[ 148], | 30.00th=[ 148], 40.00th=[ 218], 50.00th=[ 218], 60.00th=[ 218], | 70.00th=[ 1854], 80.00th=[ 1854], 90.00th=[ 1854], 95.00th=[ 1854], | 99.00th=[ 1854], 99.50th=[ 1854], 99.90th=[ 1854], 99.95th=[ 1854], | 99.99th=[ 1854] bw ( KiB/s): min= 8192, max= 8192, per=64.35%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67% cpu : usr=0.09%, sys=9.06%, ctx=190, majf=0, minf=28 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=6365KiB/s (6518kB/s), 1591KiB/s-4855KiB/s (1629kB/s-4972kB/s), io=16.0MiB (16.8MB), run=2531-2574msec WRITE: bw=12.4MiB/s (13.0MB/s), 3183KiB/s-5483KiB/s (3259kB/s-5615kB/s), io=32.0MiB (33.6MB), run=2241-2574msec