== sanity test 398b: DIO and buffer IO race ============== 23:41:16 (1708836076) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.14271 s, 44.0 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=8321: Sat Feb 24 23:41:32 2024 read: IOPS=110, BW=443KiB/s (454kB/s)(5992KiB/13523msec) clat (usec): min=47, max=155172, avg=4935.31, stdev=4831.32 lat (usec): min=48, max=155173, avg=4937.90, stdev=4831.13 clat percentiles (usec): | 1.00th=[ 61], 5.00th=[ 103], 10.00th=[ 2212], 20.00th=[ 2704], | 30.00th=[ 3163], 40.00th=[ 3687], 50.00th=[ 4293], 60.00th=[ 5080], | 70.00th=[ 5997], 80.00th=[ 6849], 90.00th=[ 8160], 95.00th=[ 9634], | 99.00th=[ 14615], 99.50th=[ 17695], 99.90th=[ 23200], 99.95th=[154141], | 99.99th=[154141] bw ( KiB/s): min= 264, max= 616, per=25.12%, avg=442.89, stdev=71.79, samples=27 iops : min= 66, max= 154, avg=110.67, stdev=17.94, samples=27 write: IOPS=116, BW=466KiB/s (477kB/s)(6296KiB/13523msec) clat (usec): min=385, max=25734, avg=3851.90, stdev=3594.39 lat (usec): min=386, max=25735, avg=3852.93, stdev=3594.40 clat percentiles (usec): | 1.00th=[ 474], 5.00th=[ 523], 10.00th=[ 578], 20.00th=[ 693], | 30.00th=[ 865], 40.00th=[ 1680], 50.00th=[ 3064], 60.00th=[ 3884], | 70.00th=[ 5014], 80.00th=[ 6521], 90.00th=[ 8979], 95.00th=[10814], | 99.00th=[14746], 99.50th=[16712], 99.90th=[23462], 99.95th=[25822], | 99.99th=[25822] bw ( KiB/s): min= 304, max= 648, per=25.19%, avg=465.07, stdev=79.07, samples=27 iops : min= 76, max= 162, avg=116.22, stdev=19.78, samples=27 lat (usec) : 50=0.10%, 100=2.25%, 250=0.75%, 500=1.56%, 750=10.68% lat (usec) : 1000=5.05% lat (msec) : 2=4.65%, 4=28.65%, 10=40.49%, 20=5.57%, 50=0.23% lat (msec) : 250=0.03% cpu : usr=0.24%, sys=22.16%, ctx=5684, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8322: Sat Feb 24 23:41:32 2024 read: IOPS=109, BW=439KiB/s (450kB/s)(5940KiB/13519msec) clat (usec): min=44, max=21316, avg=4741.12, stdev=2771.50 lat (usec): min=44, max=21317, avg=4742.02, stdev=2771.55 clat percentiles (usec): | 1.00th=[ 68], 5.00th=[ 116], 10.00th=[ 2180], 20.00th=[ 2638], | 30.00th=[ 3097], 40.00th=[ 3589], 50.00th=[ 4146], 60.00th=[ 4948], | 70.00th=[ 5866], 80.00th=[ 6849], 90.00th=[ 8225], 95.00th=[ 9372], | 99.00th=[13960], 99.50th=[15795], 99.90th=[19792], 99.95th=[21365], | 99.99th=[21365] bw ( KiB/s): min= 280, max= 576, per=24.90%, avg=439.04, stdev=63.00, samples=27 iops : min= 70, max= 144, avg=109.70, stdev=15.75, samples=27 write: IOPS=117, BW=470KiB/s (481kB/s)(6348KiB/13519msec) clat (usec): min=429, max=154380, avg=4035.46, stdev=5140.78 lat (usec): min=430, max=154381, avg=4038.50, stdev=5141.84 clat percentiles (usec): | 1.00th=[ 465], 5.00th=[ 529], 10.00th=[ 578], 20.00th=[ 717], | 30.00th=[ 922], 40.00th=[ 2212], 50.00th=[ 3195], 60.00th=[ 4047], | 70.00th=[ 5211], 80.00th=[ 6783], 90.00th=[ 8848], 95.00th=[ 10814], | 99.00th=[ 15270], 99.50th=[ 16450], 99.90th=[ 21890], 99.95th=[154141], | 99.99th=[154141] bw ( KiB/s): min= 352, max= 712, per=25.42%, avg=469.26, stdev=73.23, samples=27 iops : min= 88, max= 178, avg=117.26, stdev=18.37, samples=27 lat (usec) : 50=0.03%, 100=1.82%, 250=1.01%, 500=1.50%, 750=9.77% lat (usec) : 1000=5.31% lat (msec) : 2=4.30%, 4=30.37%, 10=40.49%, 20=5.31%, 50=0.07% lat (msec) : 250=0.03% cpu : usr=0.27%, sys=22.47%, ctx=5730, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8323: Sat Feb 24 23:41:32 2024 read: IOPS=110, BW=443KiB/s (453kB/s)(6028KiB/13617msec) clat (usec): min=42, max=28138, avg=4809.71, stdev=3019.63 lat (usec): min=42, max=28139, avg=4813.24, stdev=3021.53 clat percentiles (usec): | 1.00th=[ 64], 5.00th=[ 100], 10.00th=[ 2024], 20.00th=[ 2638], | 30.00th=[ 3097], 40.00th=[ 3621], 50.00th=[ 4293], 60.00th=[ 5080], | 70.00th=[ 5866], 80.00th=[ 6980], 90.00th=[ 8356], 95.00th=[ 9634], | 99.00th=[15533], 99.50th=[17695], 99.90th=[21103], 99.95th=[28181], | 99.99th=[28181] bw ( KiB/s): min= 224, max= 600, per=24.97%, avg=440.22, stdev=74.28, samples=27 iops : min= 56, max= 150, avg=110.00, stdev=18.61, samples=27 write: IOPS=114, BW=460KiB/s (471kB/s)(6260KiB/13617msec) clat (usec): min=409, max=171890, avg=4020.12, stdev=5494.63 lat (usec): min=409, max=171891, avg=4022.00, stdev=5494.59 clat percentiles (usec): | 1.00th=[ 453], 5.00th=[ 519], 10.00th=[ 562], 20.00th=[ 701], | 30.00th=[ 881], 40.00th=[ 2147], 50.00th=[ 3195], 60.00th=[ 3982], | 70.00th=[ 5276], 80.00th=[ 6849], 90.00th=[ 8848], 95.00th=[ 10421], | 99.00th=[ 14877], 99.50th=[ 16712], 99.90th=[ 23462], 99.95th=[170918], | 99.99th=[170918] bw ( KiB/s): min= 312, max= 616, per=24.67%, avg=455.33, stdev=71.04, samples=27 iops : min= 78, max= 154, avg=113.78, stdev=17.79, samples=27 lat (usec) : 50=0.16%, 100=2.31%, 250=1.27%, 500=1.95%, 750=9.99% lat (usec) : 1000=4.88% lat (msec) : 2=4.23%, 4=28.19%, 10=41.67%, 20=5.14%, 50=0.16% lat (msec) : 250=0.03% cpu : usr=0.24%, sys=22.41%, ctx=5609, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8324: Sat Feb 24 23:41:32 2024 read: IOPS=112, BW=450KiB/s (461kB/s)(6052KiB/13452msec) clat (usec): min=46, max=164588, avg=4827.99, stdev=4972.26 lat (usec): min=46, max=164589, avg=4829.36, stdev=4972.11 clat percentiles (usec): | 1.00th=[ 68], 5.00th=[ 126], 10.00th=[ 2212], 20.00th=[ 2671], | 30.00th=[ 3064], 40.00th=[ 3490], 50.00th=[ 4178], 60.00th=[ 4883], | 70.00th=[ 5800], 80.00th=[ 6718], 90.00th=[ 8225], 95.00th=[ 9634], | 99.00th=[ 13960], 99.50th=[ 16909], 99.90th=[ 27132], 99.95th=[164627], | 99.99th=[164627] bw ( KiB/s): min= 304, max= 550, per=25.26%, avg=445.27, stdev=56.89, samples=26 iops : min= 76, max= 137, avg=111.19, stdev=14.19, samples=26 write: IOPS=115, BW=464KiB/s (475kB/s)(6236KiB/13452msec) clat (usec): min=408, max=31772, avg=3902.30, stdev=3661.24 lat (usec): min=409, max=31774, avg=3903.71, stdev=3661.91 clat percentiles (usec): | 1.00th=[ 461], 5.00th=[ 510], 10.00th=[ 570], 20.00th=[ 693], | 30.00th=[ 840], 40.00th=[ 1975], 50.00th=[ 3163], 60.00th=[ 3949], | 70.00th=[ 5014], 80.00th=[ 6652], 90.00th=[ 8979], 95.00th=[10814], | 99.00th=[15401], 99.50th=[18482], 99.90th=[22938], 99.95th=[31851], | 99.99th=[31851] bw ( KiB/s): min= 352, max= 704, per=24.94%, avg=460.38, stdev=77.58, samples=26 iops : min= 88, max= 176, avg=114.96, stdev=19.42, samples=26 lat (usec) : 50=0.03%, 100=1.56%, 250=1.07%, 500=2.02%, 750=9.93% lat (usec) : 1000=5.21% lat (msec) : 2=3.84%, 4=30.99%, 10=39.78%, 20=5.24%, 50=0.29% lat (msec) : 250=0.03% cpu : usr=0.24%, sys=22.48%, ctx=5611, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1763KiB/s (1806kB/s), 439KiB/s-450KiB/s (450kB/s-461kB/s), io=23.4MiB (24.6MB), run=13452-13617msec WRITE: bw=1846KiB/s (1891kB/s), 460KiB/s-470KiB/s (471kB/s-481kB/s), io=24.6MiB (25.7MB), run=13452-13617msec rand-rw: (groupid=0, jobs=1): err= 0: pid=8326: Sat Feb 24 23:41:36 2024 read: IOPS=87, BW=351KiB/s (359kB/s)(5992KiB/17068msec) clat (usec): min=1610, max=156447, avg=4561.31, stdev=5335.65 lat (usec): min=1611, max=156448, avg=4562.23, stdev=5335.67 clat percentiles (usec): | 1.00th=[ 1696], 5.00th=[ 1860], 10.00th=[ 1958], 20.00th=[ 2114], | 30.00th=[ 2311], 40.00th=[ 2606], 50.00th=[ 2999], 60.00th=[ 3523], | 70.00th=[ 4424], 80.00th=[ 5997], 90.00th=[ 9503], 95.00th=[ 12256], | 99.00th=[ 20055], 99.50th=[ 22152], 99.90th=[ 26346], 99.95th=[156238], | 99.99th=[156238] bw ( KiB/s): min= 183, max= 752, per=24.98%, avg=349.74, stdev=183.06, samples=34 iops : min= 45, max= 188, avg=87.32, stdev=45.82, samples=34 write: IOPS=92, BW=369KiB/s (378kB/s)(6296KiB/17068msec) clat (usec): min=2170, max=30306, avg=6483.90, stdev=4212.07 lat (usec): min=2171, max=30308, avg=6485.09, stdev=4212.27 clat percentiles (usec): | 1.00th=[ 2442], 5.00th=[ 2802], 10.00th=[ 3032], 20.00th=[ 3294], | 30.00th=[ 3556], 40.00th=[ 3884], 50.00th=[ 4752], 60.00th=[ 6063], | 70.00th=[ 7767], 80.00th=[ 9634], 90.00th=[12125], 95.00th=[14615], | 99.00th=[22152], 99.50th=[26084], 99.90th=[27657], 99.95th=[30278], | 99.99th=[30278] bw ( KiB/s): min= 200, max= 840, per=25.06%, avg=367.38, stdev=190.66, samples=34 iops : min= 50, max= 210, avg=91.74, stdev=47.72, samples=34 lat (msec) : 2=6.51%, 4=46.94%, 10=33.01%, 20=12.34%, 50=1.17% lat (msec) : 250=0.03% cpu : usr=0.22%, sys=10.85%, ctx=3523, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8327: Sat Feb 24 23:41:36 2024 read: IOPS=86, BW=348KiB/s (356kB/s)(5940KiB/17088msec) clat (usec): min=1574, max=168206, avg=4626.59, stdev=5647.18 lat (usec): min=1575, max=168207, avg=4627.52, stdev=5647.18 clat percentiles (usec): | 1.00th=[ 1713], 5.00th=[ 1844], 10.00th=[ 1926], 20.00th=[ 2089], | 30.00th=[ 2278], 40.00th=[ 2507], 50.00th=[ 2900], 60.00th=[ 3490], | 70.00th=[ 4359], 80.00th=[ 6390], 90.00th=[ 10159], 95.00th=[ 12649], | 99.00th=[ 19006], 99.50th=[ 20317], 99.90th=[ 27132], 99.95th=[168821], | 99.99th=[168821] bw ( KiB/s): min= 152, max= 744, per=24.73%, avg=346.21, stdev=194.10, samples=34 iops : min= 38, max= 186, avg=86.44, stdev=48.58, samples=34 write: IOPS=92, BW=371KiB/s (380kB/s)(6348KiB/17088msec) clat (usec): min=2138, max=43310, avg=6419.52, stdev=4354.13 lat (usec): min=2138, max=43311, avg=6420.72, stdev=4354.14 clat percentiles (usec): | 1.00th=[ 2442], 5.00th=[ 2802], 10.00th=[ 2999], 20.00th=[ 3228], | 30.00th=[ 3523], 40.00th=[ 3916], 50.00th=[ 4621], 60.00th=[ 5735], | 70.00th=[ 7504], 80.00th=[ 9503], 90.00th=[12256], 95.00th=[14615], | 99.00th=[22152], 99.50th=[26870], 99.90th=[36439], 99.95th=[43254], | 99.99th=[43254] bw ( KiB/s): min= 207, max= 784, per=25.16%, avg=368.79, stdev=185.90, samples=34 iops : min= 51, max= 196, avg=92.09, stdev=46.54, samples=34 lat (msec) : 2=7.32%, 4=46.61%, 10=32.39%, 20=12.60%, 50=1.04% lat (msec) : 250=0.03% cpu : usr=0.27%, sys=10.70%, ctx=3484, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8328: Sat Feb 24 23:41:36 2024 read: IOPS=88, BW=353KiB/s (361kB/s)(6028KiB/17079msec) clat (usec): min=1563, max=32969, avg=4563.69, stdev=3742.53 lat (usec): min=1564, max=32970, avg=4564.55, stdev=3742.56 clat percentiles (usec): | 1.00th=[ 1696], 5.00th=[ 1844], 10.00th=[ 1926], 20.00th=[ 2114], | 30.00th=[ 2278], 40.00th=[ 2606], 50.00th=[ 2966], 60.00th=[ 3589], | 70.00th=[ 4359], 80.00th=[ 6587], 90.00th=[ 9765], 95.00th=[12911], | 99.00th=[18220], 99.50th=[21103], 99.90th=[26346], 99.95th=[32900], | 99.99th=[32900] bw ( KiB/s): min= 168, max= 792, per=25.11%, avg=351.47, stdev=181.10, samples=34 iops : min= 42, max= 198, avg=87.82, stdev=45.31, samples=34 write: IOPS=91, BW=367KiB/s (375kB/s)(6260KiB/17079msec) clat (msec): min=2, max=169, avg= 6.50, stdev= 5.86 lat (msec): min=2, max=169, avg= 6.50, stdev= 5.86 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 4], 20.00th=[ 4], | 30.00th=[ 4], 40.00th=[ 4], 50.00th=[ 5], 60.00th=[ 6], | 70.00th=[ 8], 80.00th=[ 10], 90.00th=[ 13], 95.00th=[ 15], | 99.00th=[ 22], 99.50th=[ 25], 99.90th=[ 29], 99.95th=[ 169], | 99.99th=[ 169] bw ( KiB/s): min= 200, max= 784, per=24.87%, avg=364.65, stdev=188.59, samples=34 iops : min= 50, max= 196, avg=91.12, stdev=47.17, samples=34 lat (msec) : 2=6.80%, 4=46.91%, 10=32.65%, 20=12.57%, 50=1.04% lat (msec) : 250=0.03% cpu : usr=0.29%, sys=10.83%, ctx=3535, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8329: Sat Feb 24 23:41:36 2024 read: IOPS=88, BW=353KiB/s (361kB/s)(6052KiB/17146msec) clat (usec): min=1565, max=25123, avg=4411.37, stdev=3630.16 lat (usec): min=1565, max=25124, avg=4412.34, stdev=3630.17 clat percentiles (usec): | 1.00th=[ 1713], 5.00th=[ 1844], 10.00th=[ 1942], 20.00th=[ 2073], | 30.00th=[ 2245], 40.00th=[ 2507], 50.00th=[ 2966], 60.00th=[ 3458], | 70.00th=[ 4178], 80.00th=[ 6063], 90.00th=[ 9634], 95.00th=[12125], | 99.00th=[18744], 99.50th=[21627], 99.90th=[24511], 99.95th=[25035], | 99.99th=[25035] bw ( KiB/s): min= 152, max= 768, per=24.95%, avg=349.29, stdev=184.83, samples=34 iops : min= 38, max= 192, avg=87.26, stdev=46.20, samples=34 write: IOPS=90, BW=364KiB/s (372kB/s)(6236KiB/17146msec) clat (msec): min=2, max=169, avg= 6.70, stdev= 6.02 lat (msec): min=2, max=169, avg= 6.70, stdev= 6.02 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 4], 20.00th=[ 4], | 30.00th=[ 4], 40.00th=[ 4], 50.00th=[ 5], 60.00th=[ 6], | 70.00th=[ 8], 80.00th=[ 10], 90.00th=[ 13], 95.00th=[ 16], | 99.00th=[ 23], 99.50th=[ 25], 99.90th=[ 37], 99.95th=[ 169], | 99.99th=[ 169] bw ( KiB/s): min= 184, max= 806, per=24.56%, avg=360.12, stdev=191.37, samples=34 iops : min= 46, max= 201, avg=89.97, stdev=47.83, samples=34 lat (msec) : 2=6.87%, 4=48.27%, 10=30.50%, 20=13.12%, 50=1.20% lat (msec) : 250=0.03% cpu : usr=0.24%, sys=10.70%, ctx=3501, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1400KiB/s (1434kB/s), 348KiB/s-353KiB/s (356kB/s-361kB/s), io=23.4MiB (24.6MB), run=17068-17146msec WRITE: bw=1466KiB/s (1501kB/s), 364KiB/s-371KiB/s (372kB/s-380kB/s), io=24.6MiB (25.7MB), run=17068-17146msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=8362: Sat Feb 24 23:41:41 2024 read: IOPS=95, BW=1527KiB/s (1564kB/s)(5920KiB/3877msec) clat (usec): min=68, max=22498, avg=5710.37, stdev=3571.86 lat (usec): min=69, max=22499, avg=5711.27, stdev=3571.89 clat percentiles (usec): | 1.00th=[ 71], 5.00th=[ 97], 10.00th=[ 127], 20.00th=[ 3392], | 30.00th=[ 3982], 40.00th=[ 4621], 50.00th=[ 5342], 60.00th=[ 5997], | 70.00th=[ 6915], 80.00th=[ 7898], 90.00th=[ 9896], 95.00th=[11338], | 99.00th=[17433], 99.50th=[20841], 99.90th=[22414], 99.95th=[22414], | 99.99th=[22414] bw ( KiB/s): min= 1184, max= 1856, per=24.15%, avg=1462.86, stdev=232.02, samples=7 iops : min= 74, max= 116, avg=91.43, stdev=14.50, samples=7 write: IOPS=102, BW=1643KiB/s (1682kB/s)(6368KiB/3877msec) clat (usec): min=645, max=25549, avg=4407.67, stdev=3934.98 lat (usec): min=646, max=25551, avg=4408.92, stdev=3935.00 clat percentiles (usec): | 1.00th=[ 701], 5.00th=[ 807], 10.00th=[ 889], 20.00th=[ 1090], | 30.00th=[ 1401], 40.00th=[ 2089], 50.00th=[ 3163], 60.00th=[ 4359], | 70.00th=[ 5669], 80.00th=[ 7177], 90.00th=[ 9765], 95.00th=[12125], | 99.00th=[17957], 99.50th=[19530], 99.90th=[25560], 99.95th=[25560], | 99.99th=[25560] bw ( KiB/s): min= 1248, max= 1856, per=25.03%, avg=1595.43, stdev=244.70, samples=7 iops : min= 78, max= 116, avg=99.71, stdev=15.29, samples=7 lat (usec) : 100=2.86%, 250=2.47%, 750=1.04%, 1000=6.38% lat (msec) : 2=13.28%, 4=18.36%, 10=46.74%, 20=8.46%, 50=0.39% cpu : usr=0.15%, sys=25.83%, ctx=1729, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8363: Sat Feb 24 23:41:41 2024 read: IOPS=92, BW=1482KiB/s (1518kB/s)(5744KiB/3875msec) clat (usec): min=51, max=23467, avg=5606.37, stdev=3620.55 lat (usec): min=51, max=23468, avg=5609.01, stdev=3619.23 clat percentiles (usec): | 1.00th=[ 65], 5.00th=[ 84], 10.00th=[ 117], 20.00th=[ 3294], | 30.00th=[ 3785], 40.00th=[ 4359], 50.00th=[ 5145], 60.00th=[ 6063], | 70.00th=[ 6980], 80.00th=[ 8029], 90.00th=[10028], 95.00th=[11600], | 99.00th=[17171], 99.50th=[19006], 99.90th=[23462], 99.95th=[23462], | 99.99th=[23462] bw ( KiB/s): min= 1152, max= 1728, per=23.76%, avg=1439.43, stdev=211.87, samples=7 iops : min= 72, max= 108, avg=89.86, stdev=13.17, samples=7 write: IOPS=105, BW=1689KiB/s (1729kB/s)(6544KiB/3875msec) clat (usec): min=646, max=17979, avg=4524.57, stdev=3585.41 lat (usec): min=647, max=17980, avg=4526.87, stdev=3587.18 clat percentiles (usec): | 1.00th=[ 717], 5.00th=[ 816], 10.00th=[ 938], 20.00th=[ 1139], | 30.00th=[ 1729], 40.00th=[ 2868], 50.00th=[ 3523], 60.00th=[ 4490], | 70.00th=[ 5800], 80.00th=[ 7308], 90.00th=[10290], 95.00th=[11600], | 99.00th=[14222], 99.50th=[15008], 99.90th=[17957], 99.95th=[17957], | 99.99th=[17957] bw ( KiB/s): min= 1408, max= 2144, per=25.53%, avg=1627.00, stdev=278.35, samples=7 iops : min= 88, max= 134, avg=101.57, stdev=17.49, samples=7 lat (usec) : 100=3.26%, 250=2.34%, 500=0.26%, 750=1.30%, 1000=5.34% lat (msec) : 2=11.20%, 4=22.14%, 10=43.49%, 20=10.55%, 50=0.13% cpu : usr=0.23%, sys=25.79%, ctx=1778, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8364: Sat Feb 24 23:41:41 2024 read: IOPS=98, BW=1582KiB/s (1620kB/s)(6256KiB/3954msec) clat (usec): min=70, max=19567, avg=5524.69, stdev=3290.62 lat (usec): min=71, max=19567, avg=5526.28, stdev=3290.86 clat percentiles (usec): | 1.00th=[ 74], 5.00th=[ 102], 10.00th=[ 130], 20.00th=[ 3261], | 30.00th=[ 3916], 40.00th=[ 4490], 50.00th=[ 5145], 60.00th=[ 5997], | 70.00th=[ 7046], 80.00th=[ 8225], 90.00th=[ 9372], 95.00th=[10552], | 99.00th=[16909], 99.50th=[18220], 99.90th=[19530], 99.95th=[19530], | 99.99th=[19530] bw ( KiB/s): min= 1024, max= 1984, per=24.75%, avg=1499.00, stdev=396.71, samples=7 iops : min= 64, max= 124, avg=93.57, stdev=24.87, samples=7 write: IOPS=95, BW=1526KiB/s (1562kB/s)(6032KiB/3954msec) clat (usec): min=689, max=24317, avg=4727.90, stdev=4061.71 lat (usec): min=690, max=24319, avg=4731.60, stdev=4062.21 clat percentiles (usec): | 1.00th=[ 701], 5.00th=[ 783], 10.00th=[ 881], 20.00th=[ 1123], | 30.00th=[ 1369], 40.00th=[ 2606], 50.00th=[ 3752], 60.00th=[ 4817], | 70.00th=[ 6259], 80.00th=[ 7701], 90.00th=[10421], 95.00th=[12780], | 99.00th=[17433], 99.50th=[20317], 99.90th=[24249], 99.95th=[24249], | 99.99th=[24249] bw ( KiB/s): min= 1216, max= 2048, per=23.38%, avg=1489.71, stdev=276.24, samples=7 iops : min= 76, max= 128, avg=93.00, stdev=17.23, samples=7 lat (usec) : 100=2.34%, 250=3.39%, 500=0.13%, 750=1.43%, 1000=6.51% lat (msec) : 2=10.29%, 4=18.88%, 10=48.31%, 20=8.46%, 50=0.26% cpu : usr=0.25%, sys=25.17%, ctx=1699, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8365: Sat Feb 24 23:41:41 2024 read: IOPS=96, BW=1551KiB/s (1588kB/s)(6032KiB/3890msec) clat (usec): min=57, max=30699, avg=5632.01, stdev=3733.87 lat (usec): min=58, max=30699, avg=5634.86, stdev=3738.39 clat percentiles (usec): | 1.00th=[ 65], 5.00th=[ 92], 10.00th=[ 137], 20.00th=[ 3261], | 30.00th=[ 3884], 40.00th=[ 4359], 50.00th=[ 5080], 60.00th=[ 5932], | 70.00th=[ 6849], 80.00th=[ 8029], 90.00th=[ 9896], 95.00th=[11731], | 99.00th=[19530], 99.50th=[23462], 99.90th=[30802], 99.95th=[30802], | 99.99th=[30802] bw ( KiB/s): min= 1152, max= 1824, per=24.66%, avg=1493.86, stdev=282.51, samples=7 iops : min= 72, max= 114, avg=93.14, stdev=17.71, samples=7 write: IOPS=100, BW=1608KiB/s (1647kB/s)(6256KiB/3890msec) clat (usec): min=610, max=19391, avg=4492.05, stdev=3661.26 lat (usec): min=612, max=19392, avg=4493.29, stdev=3661.25 clat percentiles (usec): | 1.00th=[ 644], 5.00th=[ 807], 10.00th=[ 898], 20.00th=[ 1106], | 30.00th=[ 1418], 40.00th=[ 2606], 50.00th=[ 3752], 60.00th=[ 4752], | 70.00th=[ 5735], 80.00th=[ 7308], 90.00th=[ 9896], 95.00th=[11600], | 99.00th=[14877], 99.50th=[19006], 99.90th=[19268], 99.95th=[19268], | 99.99th=[19268] bw ( KiB/s): min= 1088, max= 1792, per=24.59%, avg=1566.86, stdev=228.63, samples=7 iops : min= 68, max= 112, avg=97.71, stdev=14.17, samples=7 lat (usec) : 100=2.99%, 250=2.60%, 500=0.13%, 750=1.43%, 1000=6.51% lat (msec) : 2=11.07%, 4=17.45%, 10=48.83%, 20=8.59%, 50=0.39% cpu : usr=0.15%, sys=25.97%, ctx=1720, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=6058KiB/s (6203kB/s), 1482KiB/s-1582KiB/s (1518kB/s-1620kB/s), io=23.4MiB (24.5MB), run=3875-3954msec WRITE: bw=6373KiB/s (6526kB/s), 1526KiB/s-1689KiB/s (1562kB/s-1729kB/s), io=24.6MiB (25.8MB), run=3875-3954msec rand-rw: (groupid=0, jobs=1): err= 0: pid=8356: Sat Feb 24 23:41:42 2024 read: IOPS=75, BW=1203KiB/s (1232kB/s)(5920KiB/4921msec) clat (usec): min=1888, max=22124, avg=5180.07, stdev=3989.25 lat (usec): min=1888, max=22125, avg=5181.47, stdev=3989.02 clat percentiles (usec): | 1.00th=[ 2024], 5.00th=[ 2278], 10.00th=[ 2376], 20.00th=[ 2573], | 30.00th=[ 2802], 40.00th=[ 3163], 50.00th=[ 3490], 60.00th=[ 4178], | 70.00th=[ 5080], 80.00th=[ 6783], 90.00th=[11207], 95.00th=[14484], | 99.00th=[19792], 99.50th=[21627], 99.90th=[22152], 99.95th=[22152], | 99.99th=[22152] bw ( KiB/s): min= 704, max= 2048, per=22.26%, avg=1077.33, stdev=416.31, samples=9 iops : min= 44, max= 128, avg=67.33, stdev=26.02, samples=9 write: IOPS=80, BW=1294KiB/s (1325kB/s)(6368KiB/4921msec) clat (usec): min=2547, max=28969, avg=7524.97, stdev=4384.83 lat (usec): min=2548, max=28970, avg=7526.71, stdev=4384.87 clat percentiles (usec): | 1.00th=[ 2802], 5.00th=[ 3294], 10.00th=[ 3556], 20.00th=[ 4015], | 30.00th=[ 4424], 40.00th=[ 5080], 50.00th=[ 5932], 60.00th=[ 7308], | 70.00th=[ 8848], 80.00th=[10552], 90.00th=[14091], 95.00th=[16319], | 99.00th=[22938], 99.50th=[24773], 99.90th=[28967], 99.95th=[28967], | 99.99th=[28967] bw ( KiB/s): min= 928, max= 2080, per=23.32%, avg=1187.56, stdev=357.21, samples=9 iops : min= 58, max= 130, avg=74.22, stdev=22.33, samples=9 lat (msec) : 2=0.39%, 4=37.24%, 10=45.31%, 20=15.62%, 50=1.43% cpu : usr=0.20%, sys=11.71%, ctx=865, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8357: Sat Feb 24 23:41:42 2024 read: IOPS=72, BW=1161KiB/s (1189kB/s)(5744KiB/4948msec) clat (usec): min=1909, max=21464, avg=5289.14, stdev=3958.78 lat (usec): min=1910, max=21465, avg=5290.02, stdev=3958.82 clat percentiles (usec): | 1.00th=[ 2073], 5.00th=[ 2278], 10.00th=[ 2409], 20.00th=[ 2704], | 30.00th=[ 2933], 40.00th=[ 3261], 50.00th=[ 3752], 60.00th=[ 4293], | 70.00th=[ 4883], 80.00th=[ 6849], 90.00th=[11863], 95.00th=[14353], | 99.00th=[19792], 99.50th=[21365], 99.90th=[21365], 99.95th=[21365], | 99.99th=[21365] bw ( KiB/s): min= 640, max= 1920, per=21.89%, avg=1059.56, stdev=360.07, samples=9 iops : min= 40, max= 120, avg=66.22, stdev=22.50, samples=9 write: IOPS=82, BW=1323KiB/s (1354kB/s)(6544KiB/4948msec) clat (usec): min=2812, max=28922, avg=7434.52, stdev=4613.89 lat (usec): min=2813, max=28924, avg=7436.17, stdev=4614.00 clat percentiles (usec): | 1.00th=[ 3064], 5.00th=[ 3359], 10.00th=[ 3523], 20.00th=[ 3916], | 30.00th=[ 4228], 40.00th=[ 4883], 50.00th=[ 5800], 60.00th=[ 6718], | 70.00th=[ 8160], 80.00th=[10421], 90.00th=[14222], 95.00th=[17433], | 99.00th=[23200], 99.50th=[24511], 99.90th=[28967], 99.95th=[28967], | 99.99th=[28967] bw ( KiB/s): min= 832, max= 2176, per=23.39%, avg=1191.11, stdev=404.70, samples=9 iops : min= 52, max= 136, avg=74.44, stdev=25.29, samples=9 lat (msec) : 2=0.13%, 4=36.72%, 10=45.31%, 20=16.28%, 50=1.56% cpu : usr=0.14%, sys=11.68%, ctx=859, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8358: Sat Feb 24 23:41:42 2024 read: IOPS=79, BW=1271KiB/s (1302kB/s)(6256KiB/4922msec) clat (usec): min=2024, max=23309, avg=5282.51, stdev=3767.93 lat (usec): min=2025, max=23309, avg=5283.39, stdev=3767.94 clat percentiles (usec): | 1.00th=[ 2057], 5.00th=[ 2278], 10.00th=[ 2376], 20.00th=[ 2638], | 30.00th=[ 3032], 40.00th=[ 3425], 50.00th=[ 3752], 60.00th=[ 4293], | 70.00th=[ 5342], 80.00th=[ 7177], 90.00th=[11207], 95.00th=[13960], | 99.00th=[17957], 99.50th=[21103], 99.90th=[23200], 99.95th=[23200], | 99.99th=[23200] bw ( KiB/s): min= 864, max= 1984, per=23.51%, avg=1137.78, stdev=349.85, samples=9 iops : min= 54, max= 124, avg=71.11, stdev=21.87, samples=9 write: IOPS=76, BW=1226KiB/s (1255kB/s)(6032KiB/4922msec) clat (usec): min=2855, max=33148, avg=7553.09, stdev=4485.30 lat (usec): min=2856, max=33151, avg=7555.00, stdev=4485.17 clat percentiles (usec): | 1.00th=[ 2966], 5.00th=[ 3392], 10.00th=[ 3687], 20.00th=[ 4047], | 30.00th=[ 4490], 40.00th=[ 5014], 50.00th=[ 5800], 60.00th=[ 7439], | 70.00th=[ 8848], 80.00th=[10552], 90.00th=[13435], 95.00th=[16712], | 99.00th=[24511], 99.50th=[29492], 99.90th=[33162], 99.95th=[33162], | 99.99th=[33162] bw ( KiB/s): min= 736, max= 2176, per=22.20%, avg=1130.67, stdev=421.20, samples=9 iops : min= 46, max= 136, avg=70.67, stdev=26.32, samples=9 lat (msec) : 4=36.59%, 10=45.70%, 20=16.67%, 50=1.04% cpu : usr=0.18%, sys=11.70%, ctx=872, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8359: Sat Feb 24 23:41:42 2024 read: IOPS=77, BW=1234KiB/s (1263kB/s)(6032KiB/4890msec) clat (usec): min=1890, max=26067, avg=5295.41, stdev=4020.11 lat (usec): min=1891, max=26068, avg=5296.59, stdev=4019.94 clat percentiles (usec): | 1.00th=[ 2147], 5.00th=[ 2311], 10.00th=[ 2442], 20.00th=[ 2671], | 30.00th=[ 2868], 40.00th=[ 3195], 50.00th=[ 3687], 60.00th=[ 4293], | 70.00th=[ 5080], 80.00th=[ 7242], 90.00th=[11076], 95.00th=[14615], | 99.00th=[20317], 99.50th=[22938], 99.90th=[26084], 99.95th=[26084], | 99.99th=[26084] bw ( KiB/s): min= 768, max= 1952, per=23.29%, avg=1127.11, stdev=340.83, samples=9 iops : min= 48, max= 122, avg=70.44, stdev=21.30, samples=9 write: IOPS=79, BW=1279KiB/s (1310kB/s)(6256KiB/4890msec) clat (usec): min=2613, max=32509, avg=7375.89, stdev=4302.50 lat (usec): min=2613, max=32510, avg=7377.77, stdev=4302.62 clat percentiles (usec): | 1.00th=[ 2900], 5.00th=[ 3392], 10.00th=[ 3720], 20.00th=[ 4047], | 30.00th=[ 4424], 40.00th=[ 5014], 50.00th=[ 5932], 60.00th=[ 6980], | 70.00th=[ 8356], 80.00th=[10421], 90.00th=[12780], 95.00th=[16188], | 99.00th=[24249], 99.50th=[25560], 99.90th=[32637], 99.95th=[32637], | 99.99th=[32637] bw ( KiB/s): min= 768, max= 2176, per=23.18%, avg=1180.44, stdev=411.50, samples=9 iops : min= 48, max= 136, avg=73.78, stdev=25.72, samples=9 lat (msec) : 2=0.13%, 4=36.33%, 10=47.01%, 20=14.97%, 50=1.56% cpu : usr=0.31%, sys=11.95%, ctx=896, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=4841KiB/s (4957kB/s), 1161KiB/s-1271KiB/s (1189kB/s-1302kB/s), io=23.4MiB (24.5MB), run=4890-4948msec WRITE: bw=5093KiB/s (5215kB/s), 1226KiB/s-1323KiB/s (1255kB/s-1354kB/s), io=24.6MiB (25.8MB), run=4890-4948msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=8374: Sat Feb 24 23:41:43 2024 read: IOPS=13, BW=13.5MiB/s (14.1MB/s)(7168KiB/520msec) clat (usec): min=38529, max=55918, avg=43655.90, stdev=6574.84 lat (usec): min=38531, max=55919, avg=43657.13, stdev=6574.53 clat percentiles (usec): | 1.00th=[38536], 5.00th=[38536], 10.00th=[38536], 20.00th=[39060], | 30.00th=[39060], 40.00th=[39060], 50.00th=[40109], 60.00th=[44827], | 70.00th=[44827], 80.00th=[48497], 90.00th=[55837], 95.00th=[55837], | 99.00th=[55837], 99.50th=[55837], 99.90th=[55837], 99.95th=[55837], | 99.99th=[55837] bw ( KiB/s): min=12288, max=12288, per=48.72%, avg=12288.00, stdev= 0.00, samples=1 iops : min= 12, max= 12, avg=12.00, stdev= 0.00, samples=1 write: IOPS=9, BW=9846KiB/s (10.1MB/s)(5120KiB/520msec) clat (usec): min=22949, max=66594, avg=42544.91, stdev=17302.45 lat (usec): min=22986, max=66640, avg=42627.09, stdev=17283.26 clat percentiles (usec): | 1.00th=[22938], 5.00th=[22938], 10.00th=[22938], 20.00th=[22938], | 30.00th=[30540], 40.00th=[30540], 50.00th=[40633], 60.00th=[40633], | 70.00th=[52167], 80.00th=[52167], 90.00th=[66847], 95.00th=[66847], | 99.00th=[66847], 99.50th=[66847], 99.90th=[66847], 99.95th=[66847], | 99.99th=[66847] bw ( KiB/s): min=10240, max=10240, per=29.00%, avg=10240.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 lat (msec) : 50=75.00%, 100=25.00% cpu : usr=0.00%, sys=50.67%, ctx=136, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8375: Sat Feb 24 23:41:43 2024 read: IOPS=2, BW=2742KiB/s (2807kB/s)(2048KiB/747msec) clat (usec): min=57997, max=58817, avg=58407.43, stdev=580.29 lat (usec): min=57997, max=58819, avg=58408.53, stdev=580.81 clat percentiles (usec): | 1.00th=[57934], 5.00th=[57934], 10.00th=[57934], 20.00th=[57934], | 30.00th=[57934], 40.00th=[57934], 50.00th=[57934], 60.00th=[58983], | 70.00th=[58983], 80.00th=[58983], 90.00th=[58983], 95.00th=[58983], | 99.00th=[58983], 99.50th=[58983], 99.90th=[58983], 99.95th=[58983], | 99.99th=[58983] bw ( KiB/s): min= 4096, max= 4096, per=16.24%, avg=4096.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 write: IOPS=13, BW=13.4MiB/s (14.0MB/s)(10.0MiB/747msec) clat (msec): min=30, max=140, avg=62.79, stdev=34.11 lat (msec): min=30, max=140, avg=62.86, stdev=34.11 clat percentiles (msec): | 1.00th=[ 31], 5.00th=[ 31], 10.00th=[ 31], 20.00th=[ 36], | 30.00th=[ 37], 40.00th=[ 37], 50.00th=[ 52], 60.00th=[ 57], | 70.00th=[ 78], 80.00th=[ 78], 90.00th=[ 92], 95.00th=[ 140], | 99.00th=[ 140], 99.50th=[ 140], 99.90th=[ 140], 99.95th=[ 140], | 99.99th=[ 140] bw ( KiB/s): min=10240, max=10240, per=29.00%, avg=10240.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 lat (msec) : 50=33.33%, 100=58.33%, 250=8.33% cpu : usr=0.00%, sys=37.67%, ctx=93, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8376: Sat Feb 24 23:41:43 2024 read: IOPS=11, BW=11.2MiB/s (11.8MB/s)(6144KiB/535msec) clat (usec): min=440, max=74951, avg=20641.08, stdev=30829.10 lat (usec): min=441, max=74952, avg=20642.37, stdev=30829.32 clat percentiles (usec): | 1.00th=[ 441], 5.00th=[ 441], 10.00th=[ 441], 20.00th=[ 461], | 30.00th=[ 461], 40.00th=[ 469], 50.00th=[ 469], 60.00th=[ 7046], | 70.00th=[40633], 80.00th=[40633], 90.00th=[74974], 95.00th=[74974], | 99.00th=[74974], 99.50th=[74974], 99.90th=[74974], 99.95th=[74974], | 99.99th=[74974] bw ( KiB/s): min=10240, max=10240, per=40.60%, avg=10240.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 write: IOPS=11, BW=11.2MiB/s (11.8MB/s)(6144KiB/535msec) clat (msec): min=24, max=109, avg=68.29, stdev=32.26 lat (msec): min=24, max=109, avg=68.33, stdev=32.26 clat percentiles (msec): | 1.00th=[ 25], 5.00th=[ 25], 10.00th=[ 25], 20.00th=[ 47], | 30.00th=[ 47], 40.00th=[ 60], 50.00th=[ 60], 60.00th=[ 70], | 70.00th=[ 101], 80.00th=[ 101], 90.00th=[ 110], 95.00th=[ 110], | 99.00th=[ 110], 99.50th=[ 110], 99.90th=[ 110], 99.95th=[ 110], | 99.99th=[ 110] bw ( KiB/s): min=12288, max=12288, per=34.80%, avg=12288.00, stdev= 0.00, samples=1 iops : min= 12, max= 12, avg=12.00, stdev= 0.00, samples=1 lat (usec) : 500=25.00% lat (msec) : 10=8.33%, 50=25.00%, 100=33.33%, 250=8.33% cpu : usr=0.00%, sys=31.65%, ctx=70, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8377: Sat Feb 24 23:41:43 2024 read: IOPS=6, BW=6305KiB/s (6457kB/s)(5120KiB/812msec) clat (usec): min=602, max=83929, avg=49705.21, stdev=33459.62 lat (usec): min=604, max=83930, avg=49706.37, stdev=33459.23 clat percentiles (usec): | 1.00th=[ 603], 5.00th=[ 603], 10.00th=[ 603], 20.00th=[ 603], | 30.00th=[33817], 40.00th=[33817], 50.00th=[56361], 60.00th=[56361], | 70.00th=[73925], 80.00th=[73925], 90.00th=[84411], 95.00th=[84411], | 99.00th=[84411], 99.50th=[84411], 99.90th=[84411], 99.95th=[84411], | 99.99th=[84411] bw ( KiB/s): min= 4096, max= 4096, per=16.24%, avg=4096.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 write: IOPS=8, BW=8828KiB/s (9039kB/s)(7168KiB/812msec) clat (msec): min=29, max=113, avg=80.36, stdev=29.93 lat (msec): min=29, max=113, avg=80.41, stdev=29.94 clat percentiles (msec): | 1.00th=[ 30], 5.00th=[ 30], 10.00th=[ 30], 20.00th=[ 51], | 30.00th=[ 79], 40.00th=[ 79], 50.00th=[ 96], 60.00th=[ 96], | 70.00th=[ 96], 80.00th=[ 100], 90.00th=[ 113], 95.00th=[ 113], | 99.00th=[ 113], 99.50th=[ 113], 99.90th=[ 113], 99.95th=[ 113], | 99.99th=[ 113] bw ( KiB/s): min= 8192, max= 8192, per=23.20%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 lat (usec) : 750=8.33% lat (msec) : 50=16.67%, 100=66.67%, 250=8.33% cpu : usr=0.00%, sys=34.03%, ctx=95, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=24.6MiB/s (25.8MB/s), 2742KiB/s-13.5MiB/s (2807kB/s-14.1MB/s), io=20.0MiB (20.0MB), run=520-812msec WRITE: bw=34.5MiB/s (36.2MB/s), 8828KiB/s-13.4MiB/s (9039kB/s-14.0MB/s), io=28.0MiB (29.4MB), run=520-812msec rand-rw: (groupid=0, jobs=1): err= 0: pid=8378: Sat Feb 24 23:41:43 2024 read: IOPS=10, BW=10.4MiB/s (10.0MB/s)(7168KiB/670msec) clat (usec): min=26414, max=62338, avg=45148.13, stdev=10626.20 lat (usec): min=26415, max=62339, avg=45149.44, stdev=10626.08 clat percentiles (usec): | 1.00th=[26346], 5.00th=[26346], 10.00th=[26346], 20.00th=[42206], | 30.00th=[43779], 40.00th=[43779], 50.00th=[45351], 60.00th=[46924], | 70.00th=[46924], 80.00th=[49021], 90.00th=[62129], 95.00th=[62129], | 99.00th=[62129], 99.50th=[62129], 99.90th=[62129], 99.95th=[62129], | 99.99th=[62129] bw ( KiB/s): min=10199, max=10199, per=41.58%, avg=10199.00, stdev= 0.00, samples=1 iops : min= 9, max= 9, avg= 9.00, stdev= 0.00, samples=1 write: IOPS=7, BW=7642KiB/s (7825kB/s)(5120KiB/670msec) clat (usec): min=57316, max=88319, avg=70411.24, stdev=11338.68 lat (usec): min=57349, max=88393, avg=70477.66, stdev=11348.88 clat percentiles (usec): | 1.00th=[57410], 5.00th=[57410], 10.00th=[57410], 20.00th=[57410], | 30.00th=[66323], 40.00th=[66323], 50.00th=[68682], 60.00th=[68682], | 70.00th=[71828], 80.00th=[71828], 90.00th=[88605], 95.00th=[88605], | 99.00th=[88605], 99.50th=[88605], 99.90th=[88605], 99.95th=[88605], | 99.99th=[88605] bw ( KiB/s): min= 8159, max= 8159, per=23.76%, avg=8159.00, stdev= 0.00, samples=1 iops : min= 7, max= 7, avg= 7.00, stdev= 0.00, samples=1 lat (msec) : 50=50.00%, 100=50.00% cpu : usr=0.30%, sys=22.27%, ctx=74, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8379: Sat Feb 24 23:41:43 2024 read: IOPS=2, BW=2491KiB/s (2551kB/s)(2048KiB/822msec) clat (msec): min=51, max=111, avg=81.19, stdev=42.63 lat (msec): min=51, max=111, avg=81.20, stdev=42.63 clat percentiles (msec): | 1.00th=[ 52], 5.00th=[ 52], 10.00th=[ 52], 20.00th=[ 52], | 30.00th=[ 52], 40.00th=[ 52], 50.00th=[ 52], 60.00th=[ 112], | 70.00th=[ 112], 80.00th=[ 112], 90.00th=[ 112], 95.00th=[ 112], | 99.00th=[ 112], 99.50th=[ 112], 99.90th=[ 112], 99.95th=[ 112], | 99.99th=[ 112] bw ( KiB/s): min= 2003, max= 2003, per=8.17%, avg=2003.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=12, BW=12.2MiB/s (12.8MB/s)(10.0MiB/822msec) clat (msec): min=24, max=117, avg=65.03, stdev=31.41 lat (msec): min=24, max=117, avg=65.09, stdev=31.41 clat percentiles (msec): | 1.00th=[ 25], 5.00th=[ 25], 10.00th=[ 25], 20.00th=[ 31], | 30.00th=[ 36], 40.00th=[ 51], 50.00th=[ 59], 60.00th=[ 62], | 70.00th=[ 95], 80.00th=[ 95], 90.00th=[ 99], 95.00th=[ 117], | 99.00th=[ 117], 99.50th=[ 117], 99.90th=[ 117], 99.95th=[ 117], | 99.99th=[ 117] bw ( KiB/s): min=10019, max=10019, per=29.18%, avg=10019.00, stdev= 0.00, samples=1 iops : min= 9, max= 9, avg= 9.00, stdev= 0.00, samples=1 lat (msec) : 50=25.00%, 100=58.33%, 250=16.67% cpu : usr=0.00%, sys=22.05%, ctx=87, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8380: Sat Feb 24 23:41:43 2024 read: IOPS=7, BW=7604KiB/s (7786kB/s)(6144KiB/808msec) clat (msec): min=27, max=110, avg=63.20, stdev=30.42 lat (msec): min=27, max=110, avg=63.21, stdev=30.42 clat percentiles (msec): | 1.00th=[ 28], 5.00th=[ 28], 10.00th=[ 28], 20.00th=[ 42], | 30.00th=[ 42], 40.00th=[ 51], 50.00th=[ 51], 60.00th=[ 63], | 70.00th=[ 87], 80.00th=[ 87], 90.00th=[ 111], 95.00th=[ 111], | 99.00th=[ 111], 99.50th=[ 111], 99.90th=[ 111], 99.95th=[ 111], | 99.99th=[ 111] bw ( KiB/s): min= 4039, max= 4039, per=16.47%, avg=4039.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 write: IOPS=7, BW=7604KiB/s (7786kB/s)(6144KiB/808msec) clat (msec): min=32, max=126, avg=70.55, stdev=41.42 lat (msec): min=32, max=126, avg=70.61, stdev=41.41 clat percentiles (msec): | 1.00th=[ 33], 5.00th=[ 33], 10.00th=[ 33], 20.00th=[ 36], | 30.00th=[ 36], 40.00th=[ 45], 50.00th=[ 45], 60.00th=[ 67], | 70.00th=[ 117], 80.00th=[ 117], 90.00th=[ 127], 95.00th=[ 127], | 99.00th=[ 127], 99.50th=[ 127], 99.90th=[ 127], 99.95th=[ 127], | 99.99th=[ 127] bw ( KiB/s): min= 8078, max= 8078, per=23.53%, avg=8078.00, stdev= 0.00, samples=1 iops : min= 7, max= 7, avg= 7.00, stdev= 0.00, samples=1 lat (msec) : 50=41.67%, 100=33.33%, 250=25.00% cpu : usr=0.00%, sys=16.60%, ctx=84, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8381: Sat Feb 24 23:41:43 2024 read: IOPS=5, BW=6132KiB/s (6279kB/s)(5120KiB/835msec) clat (usec): min=21168, max=70412, avg=42248.31, stdev=25092.06 lat (usec): min=21170, max=70413, avg=42249.65, stdev=25091.66 clat percentiles (usec): | 1.00th=[21103], 5.00th=[21103], 10.00th=[21103], 20.00th=[21103], | 30.00th=[22414], 40.00th=[22414], 50.00th=[28443], 60.00th=[28443], | 70.00th=[68682], 80.00th=[68682], 90.00th=[70779], 95.00th=[70779], | 99.00th=[70779], 99.50th=[70779], 99.90th=[70779], 99.95th=[70779], | 99.99th=[70779] write: IOPS=8, BW=8584KiB/s (8790kB/s)(7168KiB/835msec) clat (msec): min=29, max=143, avg=88.85, stdev=40.72 lat (msec): min=29, max=143, avg=88.92, stdev=40.71 clat percentiles (msec): | 1.00th=[ 30], 5.00th=[ 30], 10.00th=[ 30], 20.00th=[ 44], | 30.00th=[ 82], 40.00th=[ 82], 50.00th=[ 97], 60.00th=[ 110], | 70.00th=[ 110], 80.00th=[ 117], 90.00th=[ 144], 95.00th=[ 144], | 99.00th=[ 144], 99.50th=[ 144], 99.90th=[ 144], 99.95th=[ 144], | 99.99th=[ 144] bw ( KiB/s): min= 8143, max= 8143, per=23.71%, avg=8143.00, stdev= 0.00, samples=1 iops : min= 7, max= 7, avg= 7.00, stdev= 0.00, samples=1 lat (msec) : 50=41.67%, 100=33.33%, 250=25.00% cpu : usr=0.00%, sys=18.35%, ctx=77, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=23.0MiB/s (25.1MB/s), 2491KiB/s-10.4MiB/s (2551kB/s-10.0MB/s), io=20.0MiB (20.0MB), run=670-835msec WRITE: bw=33.5MiB/s (35.2MB/s), 7604KiB/s-12.2MiB/s (7786kB/s-12.8MB/s), io=28.0MiB (29.4MB), run=670-835msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=8392: Sat Feb 24 23:41:45 2024 read: IOPS=7, BW=31.2MiB/s (32.7MB/s)(12.0MiB/385msec) clat (msec): min=60, max=184, avg=127.30, stdev=62.33 lat (msec): min=60, max=184, avg=127.30, stdev=62.33 clat percentiles (msec): | 1.00th=[ 61], 5.00th=[ 61], 10.00th=[ 61], 20.00th=[ 61], | 30.00th=[ 61], 40.00th=[ 138], 50.00th=[ 138], 60.00th=[ 138], | 70.00th=[ 184], 80.00th=[ 184], 90.00th=[ 184], 95.00th=[ 184], | 99.00th=[ 184], 99.50th=[ 184], 99.90th=[ 184], 99.95th=[ 184], | 99.99th=[ 184] lat (msec) : 100=33.33%, 250=66.67% cpu : usr=0.00%, sys=44.01%, ctx=153, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8393: Sat Feb 24 23:41:45 2024 read: IOPS=1, BW=5928KiB/s (6070kB/s)(4096KiB/691msec) clat (nsec): min=158741k, max=158741k, avg=158741322.00, stdev= 0.00 lat (nsec): min=158742k, max=158742k, avg=158742322.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 159], 5.00th=[ 159], 10.00th=[ 159], 20.00th=[ 159], | 30.00th=[ 159], 40.00th=[ 159], 50.00th=[ 159], 60.00th=[ 159], | 70.00th=[ 159], 80.00th=[ 159], 90.00th=[ 159], 95.00th=[ 159], | 99.00th=[ 159], 99.50th=[ 159], 99.90th=[ 159], 99.95th=[ 159], | 99.99th=[ 159] bw ( KiB/s): min= 8192, max= 8192, per=34.55%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=2, BW=11.6MiB/s (12.1MB/s)(8192KiB/691msec) clat (msec): min=261, max=268, avg=265.16, stdev= 4.69 lat (msec): min=262, max=268, avg=265.42, stdev= 4.71 clat percentiles (msec): | 1.00th=[ 262], 5.00th=[ 262], 10.00th=[ 262], 20.00th=[ 262], | 30.00th=[ 262], 40.00th=[ 262], 50.00th=[ 262], 60.00th=[ 271], | 70.00th=[ 271], 80.00th=[ 271], 90.00th=[ 271], 95.00th=[ 271], | 99.00th=[ 271], 99.50th=[ 271], 99.90th=[ 271], 99.95th=[ 271], | 99.99th=[ 271] bw ( KiB/s): min= 8192, max= 8192, per=17.28%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=66.67% cpu : usr=0.00%, sys=35.94%, ctx=232, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8394: Sat Feb 24 23:41:45 2024 write: IOPS=4, BW=18.1MiB/s (18.9MB/s)(12.0MiB/664msec) clat (msec): min=90, max=338, avg=220.08, stdev=124.14 lat (msec): min=91, max=338, avg=220.33, stdev=124.15 clat percentiles (msec): | 1.00th=[ 91], 5.00th=[ 91], 10.00th=[ 91], 20.00th=[ 91], | 30.00th=[ 91], 40.00th=[ 232], 50.00th=[ 232], 60.00th=[ 232], | 70.00th=[ 338], 80.00th=[ 338], 90.00th=[ 338], 95.00th=[ 338], | 99.00th=[ 338], 99.50th=[ 338], 99.90th=[ 338], 99.95th=[ 338], | 99.99th=[ 338] bw ( KiB/s): min= 8192, max= 8192, per=17.28%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 250=33.33%, 500=33.33% cpu : usr=0.00%, sys=42.08%, ctx=19, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8395: Sat Feb 24 23:41:45 2024 write: IOPS=5, BW=20.5MiB/s (21.5MB/s)(12.0MiB/585msec) clat (msec): min=93, max=271, avg=194.09, stdev=91.47 lat (msec): min=93, max=271, avg=194.39, stdev=91.50 clat percentiles (msec): | 1.00th=[ 93], 5.00th=[ 93], 10.00th=[ 93], 20.00th=[ 93], | 30.00th=[ 93], 40.00th=[ 218], 50.00th=[ 218], 60.00th=[ 218], | 70.00th=[ 271], 80.00th=[ 271], 90.00th=[ 271], 95.00th=[ 271], | 99.00th=[ 271], 99.50th=[ 271], 99.90th=[ 271], 99.95th=[ 271], | 99.99th=[ 271] bw ( KiB/s): min=16384, max=16384, per=34.55%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 250=33.33%, 500=33.33% cpu : usr=0.17%, sys=45.72%, ctx=41, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=23.2MiB/s (24.3MB/s), 5928KiB/s-31.2MiB/s (6070kB/s-32.7MB/s), io=16.0MiB (16.8MB), run=385-691msec WRITE: bw=46.3MiB/s (48.6MB/s), 11.6MiB/s-20.5MiB/s (12.1MB/s-21.5MB/s), io=32.0MiB (33.6MB), run=585-691msec rand-rw: (groupid=0, jobs=1): err= 0: pid=8387: Sat Feb 24 23:41:45 2024 read: IOPS=5, BW=20.1MiB/s (21.0MB/s)(12.0MiB/598msec) clat (msec): min=140, max=230, avg=197.33, stdev=49.27 lat (msec): min=140, max=230, avg=197.33, stdev=49.28 clat percentiles (msec): | 1.00th=[ 142], 5.00th=[ 142], 10.00th=[ 142], 20.00th=[ 142], | 30.00th=[ 142], 40.00th=[ 222], 50.00th=[ 222], 60.00th=[ 222], | 70.00th=[ 232], 80.00th=[ 232], 90.00th=[ 232], 95.00th=[ 232], | 99.00th=[ 232], 99.50th=[ 232], 99.90th=[ 232], 99.95th=[ 232], | 99.99th=[ 232] bw ( KiB/s): min=16384, max=16384, per=59.80%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 250=100.00% cpu : usr=0.00%, sys=18.26%, ctx=37, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8388: Sat Feb 24 23:41:45 2024 read: IOPS=2, BW=8480KiB/s (8684kB/s)(4096KiB/483msec) clat (nsec): min=72504k, max=72504k, avg=72504464.00, stdev= 0.00 lat (nsec): min=72506k, max=72506k, avg=72505937.00, stdev= 0.00 clat percentiles (usec): | 1.00th=[72877], 5.00th=[72877], 10.00th=[72877], 20.00th=[72877], | 30.00th=[72877], 40.00th=[72877], 50.00th=[72877], 60.00th=[72877], | 70.00th=[72877], 80.00th=[72877], 90.00th=[72877], 95.00th=[72877], | 99.00th=[72877], 99.50th=[72877], 99.90th=[72877], 99.95th=[72877], | 99.99th=[72877] write: IOPS=4, BW=16.6MiB/s (17.4MB/s)(8192KiB/483msec) clat (msec): min=117, max=286, avg=201.93, stdev=119.44 lat (msec): min=117, max=286, avg=202.24, stdev=119.41 clat percentiles (msec): | 1.00th=[ 118], 5.00th=[ 118], 10.00th=[ 118], 20.00th=[ 118], | 30.00th=[ 118], 40.00th=[ 118], 50.00th=[ 118], 60.00th=[ 288], | 70.00th=[ 288], 80.00th=[ 288], 90.00th=[ 288], 95.00th=[ 288], | 99.00th=[ 288], 99.50th=[ 288], 99.90th=[ 288], 99.95th=[ 288], | 99.99th=[ 288] lat (msec) : 100=33.33%, 250=33.33%, 500=33.33% cpu : usr=0.00%, sys=21.99%, ctx=40, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8389: Sat Feb 24 23:41:45 2024 write: IOPS=3, BW=12.2MiB/s (12.8MB/s)(12.0MiB/981msec) clat (msec): min=87, max=500, avg=324.78, stdev=213.18 lat (msec): min=87, max=500, avg=325.05, stdev=213.18 clat percentiles (msec): | 1.00th=[ 88], 5.00th=[ 88], 10.00th=[ 88], 20.00th=[ 88], | 30.00th=[ 88], 40.00th=[ 388], 50.00th=[ 388], 60.00th=[ 388], | 70.00th=[ 502], 80.00th=[ 502], 90.00th=[ 502], 95.00th=[ 502], | 99.00th=[ 502], 99.50th=[ 502], 99.90th=[ 502], 99.95th=[ 502], | 99.99th=[ 502] lat (msec) : 100=33.33%, 500=33.33%, 750=33.33% cpu : usr=0.00%, sys=18.88%, ctx=74, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=8390: Sat Feb 24 23:41:45 2024 write: IOPS=5, BW=20.0MiB/s (21.0MB/s)(12.0MiB/572msec) clat (msec): min=79, max=382, avg=188.93, stdev=167.74 lat (msec): min=79, max=382, avg=189.16, stdev=167.77 clat percentiles (msec): | 1.00th=[ 80], 5.00th=[ 80], 10.00th=[ 80], 20.00th=[ 80], | 30.00th=[ 80], 40.00th=[ 106], 50.00th=[ 106], 60.00th=[ 106], | 70.00th=[ 384], 80.00th=[ 384], 90.00th=[ 384], 95.00th=[ 384], | 99.00th=[ 384], 99.50th=[ 384], 99.90th=[ 384], 99.95th=[ 384], | 99.99th=[ 384] bw ( KiB/s): min=16351, max=16351, per=48.95%, avg=16351.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 250=33.33%, 500=33.33% cpu : usr=0.00%, sys=17.86%, ctx=72, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=26.8MiB/s (28.1MB/s), 8480KiB/s-20.1MiB/s (8684kB/s-21.0MB/s), io=16.0MiB (16.8MB), run=483-598msec WRITE: bw=32.6MiB/s (34.2MB/s), 12.2MiB/s-20.0MiB/s (12.8MB/s-21.0MB/s), io=32.0MiB (33.6MB), run=483-981msec