== sanity test 398b: DIO and buffer IO race ============== 19:11:08 (1743462668) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.37394 s, 36.6 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=29685: Mon Mar 31 19:11:25 2025 read: IOPS=104, BW=419KiB/s (429kB/s)(5992KiB/14296msec) clat (usec): min=63, max=119818, avg=5143.67, stdev=3980.98 lat (usec): min=63, max=119820, avg=5144.49, stdev=3981.02 clat percentiles (usec): | 1.00th=[ 81], 5.00th=[ 424], 10.00th=[ 2540], 20.00th=[ 3032], | 30.00th=[ 3425], 40.00th=[ 3982], 50.00th=[ 4555], 60.00th=[ 5342], | 70.00th=[ 6194], 80.00th=[ 6980], 90.00th=[ 8717], 95.00th=[ 9896], | 99.00th=[ 13173], 99.50th=[ 15270], 99.90th=[ 19530], 99.95th=[120062], | 99.99th=[120062] bw ( KiB/s): min= 231, max= 504, per=24.61%, avg=413.21, stdev=58.44, samples=28 iops : min= 57, max= 126, avg=103.14, stdev=14.63, samples=28 write: IOPS=110, BW=440KiB/s (451kB/s)(6296KiB/14296msec) clat (usec): min=491, max=20660, avg=4145.66, stdev=3504.61 lat (usec): min=492, max=20660, avg=4146.93, stdev=3505.06 clat percentiles (usec): | 1.00th=[ 553], 5.00th=[ 635], 10.00th=[ 701], 20.00th=[ 848], | 30.00th=[ 1172], 40.00th=[ 2704], 50.00th=[ 3458], 60.00th=[ 4293], | 70.00th=[ 5276], 80.00th=[ 6849], 90.00th=[ 8979], 95.00th=[10814], | 99.00th=[15664], 99.50th=[17957], 99.90th=[20317], 99.95th=[20579], | 99.99th=[20579] bw ( KiB/s): min= 287, max= 544, per=24.74%, avg=434.86, stdev=63.74, samples=28 iops : min= 71, max= 136, avg=108.57, stdev=15.96, samples=28 lat (usec) : 100=0.91%, 250=1.46%, 500=0.10%, 750=7.26%, 1000=6.64% lat (msec) : 2=4.59%, 4=28.16%, 10=45.15%, 20=5.57%, 50=0.13% lat (msec) : 250=0.03% cpu : usr=0.22%, sys=22.88%, ctx=5819, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29686: Mon Mar 31 19:11:25 2025 read: IOPS=104, BW=418KiB/s (428kB/s)(5940KiB/14197msec) clat (usec): min=53, max=113236, avg=5258.90, stdev=4123.58 lat (usec): min=54, max=113237, avg=5260.57, stdev=4123.80 clat percentiles (usec): | 1.00th=[ 76], 5.00th=[ 2147], 10.00th=[ 2540], 20.00th=[ 3032], | 30.00th=[ 3458], 40.00th=[ 3949], 50.00th=[ 4555], 60.00th=[ 5342], | 70.00th=[ 6194], 80.00th=[ 7046], 90.00th=[ 8455], 95.00th=[ 10028], | 99.00th=[ 16057], 99.50th=[ 19006], 99.90th=[ 34341], 99.95th=[112722], | 99.99th=[112722] bw ( KiB/s): min= 280, max= 584, per=24.68%, avg=414.43, stdev=55.07, samples=28 iops : min= 70, max= 146, avg=103.50, stdev=13.73, samples=28 write: IOPS=111, BW=447KiB/s (458kB/s)(6348KiB/14197msec) clat (usec): min=450, max=29959, avg=3986.98, stdev=3494.97 lat (usec): min=451, max=29960, avg=3988.19, stdev=3495.11 clat percentiles (usec): | 1.00th=[ 562], 5.00th=[ 635], 10.00th=[ 693], 20.00th=[ 840], | 30.00th=[ 1090], 40.00th=[ 2278], 50.00th=[ 3261], 60.00th=[ 4015], | 70.00th=[ 4948], 80.00th=[ 6783], 90.00th=[ 8979], 95.00th=[10683], | 99.00th=[14746], 99.50th=[15926], 99.90th=[19792], 99.95th=[30016], | 99.99th=[30016] bw ( KiB/s): min= 336, max= 536, per=24.89%, avg=437.57, stdev=64.12, samples=28 iops : min= 84, max= 134, avg=109.29, stdev=16.05, samples=28 lat (usec) : 100=1.11%, 250=0.88%, 500=0.07%, 750=7.29%, 1000=7.29% lat (msec) : 2=5.57%, 4=28.52%, 10=43.16%, 20=5.89%, 50=0.20% lat (msec) : 250=0.03% cpu : usr=0.26%, sys=22.75%, ctx=5910, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29687: Mon Mar 31 19:11:25 2025 read: IOPS=106, BW=427KiB/s (438kB/s)(6028KiB/14107msec) clat (usec): min=63, max=19341, avg=5143.01, stdev=2719.32 lat (usec): min=63, max=19342, avg=5145.15, stdev=2719.36 clat percentiles (usec): | 1.00th=[ 83], 5.00th=[ 2212], 10.00th=[ 2606], 20.00th=[ 3097], | 30.00th=[ 3523], 40.00th=[ 4015], 50.00th=[ 4686], 60.00th=[ 5407], | 70.00th=[ 6063], 80.00th=[ 6980], 90.00th=[ 8455], 95.00th=[ 9765], | 99.00th=[15139], 99.50th=[16188], 99.90th=[17957], 99.95th=[19268], | 99.99th=[19268] bw ( KiB/s): min= 344, max= 528, per=25.41%, avg=426.57, stdev=54.22, samples=28 iops : min= 86, max= 132, avg=106.46, stdev=13.46, samples=28 write: IOPS=110, BW=444KiB/s (454kB/s)(6260KiB/14107msec) clat (usec): min=501, max=140991, avg=4038.09, stdev=4865.83 lat (usec): min=501, max=140993, avg=4038.97, stdev=4865.86 clat percentiles (usec): | 1.00th=[ 553], 5.00th=[ 627], 10.00th=[ 685], 20.00th=[ 807], | 30.00th=[ 1074], 40.00th=[ 2180], 50.00th=[ 3261], 60.00th=[ 4047], | 70.00th=[ 5080], 80.00th=[ 6652], 90.00th=[ 8848], 95.00th=[ 10683], | 99.00th=[ 14615], 99.50th=[ 17433], 99.90th=[ 21365], 99.95th=[141558], | 99.99th=[141558] bw ( KiB/s): min= 288, max= 560, per=25.11%, avg=441.46, stdev=65.85, samples=28 iops : min= 72, max= 140, avg=110.18, stdev=16.48, samples=28 lat (usec) : 100=0.94%, 250=1.04%, 500=0.07%, 750=7.94%, 1000=6.80% lat (msec) : 2=5.37%, 4=27.73%, 10=44.47%, 20=5.53%, 50=0.07% lat (msec) : 250=0.03% cpu : usr=0.19%, sys=22.98%, ctx=5845, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29688: Mon Mar 31 19:11:25 2025 read: IOPS=106, BW=427KiB/s (437kB/s)(6052KiB/14173msec) clat (usec): min=63, max=31410, avg=5198.61, stdev=2907.38 lat (usec): min=64, max=31411, avg=5200.57, stdev=2907.21 clat percentiles (usec): | 1.00th=[ 91], 5.00th=[ 2180], 10.00th=[ 2573], 20.00th=[ 3064], | 30.00th=[ 3458], 40.00th=[ 4015], 50.00th=[ 4621], 60.00th=[ 5342], | 70.00th=[ 6194], 80.00th=[ 7111], 90.00th=[ 8586], 95.00th=[10159], | 99.00th=[15139], 99.50th=[16450], 99.90th=[26608], 99.95th=[31327], | 99.99th=[31327] bw ( KiB/s): min= 264, max= 520, per=25.26%, avg=424.14, stdev=58.14, samples=28 iops : min= 66, max= 130, avg=105.93, stdev=14.55, samples=28 write: IOPS=109, BW=440KiB/s (451kB/s)(6236KiB/14173msec) clat (usec): min=480, max=121133, avg=4009.30, stdev=4500.85 lat (usec): min=481, max=121136, avg=4010.23, stdev=4500.88 clat percentiles (usec): | 1.00th=[ 562], 5.00th=[ 635], 10.00th=[ 709], 20.00th=[ 832], | 30.00th=[ 1045], 40.00th=[ 2442], 50.00th=[ 3228], 60.00th=[ 3949], | 70.00th=[ 5014], 80.00th=[ 6521], 90.00th=[ 8979], 95.00th=[ 10552], | 99.00th=[ 14353], 99.50th=[ 15926], 99.90th=[ 23200], 99.95th=[121111], | 99.99th=[121111] bw ( KiB/s): min= 344, max= 584, per=24.86%, avg=437.00, stdev=57.01, samples=28 iops : min= 86, max= 146, avg=109.14, stdev=14.20, samples=28 lat (usec) : 100=0.62%, 250=1.30%, 500=0.07%, 750=6.77%, 1000=7.65% lat (msec) : 2=5.18%, 4=28.84%, 10=43.72%, 20=5.66%, 50=0.16% lat (msec) : 250=0.03% cpu : usr=0.16%, sys=22.97%, ctx=5851, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1680KiB/s (1720kB/s), 418KiB/s-427KiB/s (428kB/s-438kB/s), io=23.4MiB (24.6MB), run=14107-14296msec WRITE: bw=1759KiB/s (1801kB/s), 440KiB/s-447KiB/s (451kB/s-458kB/s), io=24.6MiB (25.7MB), run=14107-14296msec rand-rw: (groupid=0, jobs=1): err= 0: pid=29680: Mon Mar 31 19:11:29 2025 read: IOPS=80, BW=323KiB/s (331kB/s)(5992KiB/18541msec) clat (usec): min=1621, max=192497, avg=5165.31, stdev=6221.59 lat (usec): min=1621, max=192498, avg=5166.10, stdev=6221.61 clat percentiles (usec): | 1.00th=[ 1909], 5.00th=[ 2040], 10.00th=[ 2147], 20.00th=[ 2343], | 30.00th=[ 2638], 40.00th=[ 2999], 50.00th=[ 3490], 60.00th=[ 4228], | 70.00th=[ 5145], 80.00th=[ 7177], 90.00th=[ 10683], 95.00th=[ 13435], | 99.00th=[ 19530], 99.50th=[ 22676], 99.90th=[ 30016], 99.95th=[191890], | 99.99th=[191890] bw ( KiB/s): min= 160, max= 744, per=25.01%, avg=322.84, stdev=151.23, samples=37 iops : min= 40, max= 186, avg=80.59, stdev=37.79, samples=37 write: IOPS=84, BW=340KiB/s (348kB/s)(6296KiB/18541msec) clat (usec): min=2572, max=25673, avg=6821.29, stdev=3949.65 lat (usec): min=2573, max=25674, avg=6822.34, stdev=3949.62 clat percentiles (usec): | 1.00th=[ 2802], 5.00th=[ 3130], 10.00th=[ 3359], 20.00th=[ 3654], | 30.00th=[ 4047], 40.00th=[ 4555], 50.00th=[ 5276], 60.00th=[ 6325], | 70.00th=[ 8160], 80.00th=[ 9896], 90.00th=[12387], 95.00th=[14746], | 99.00th=[20579], 99.50th=[22414], 99.90th=[24773], 99.95th=[25560], | 99.99th=[25560] bw ( KiB/s): min= 176, max= 712, per=25.08%, avg=339.05, stdev=154.47, samples=37 iops : min= 44, max= 178, avg=84.65, stdev=38.61, samples=37 lat (msec) : 2=1.56%, 4=41.24%, 10=41.73%, 20=14.45%, 50=0.98% lat (msec) : 250=0.03% cpu : usr=0.25%, sys=11.68%, ctx=3669, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29681: Mon Mar 31 19:11:29 2025 read: IOPS=80, BW=320KiB/s (328kB/s)(5940KiB/18552msec) clat (usec): min=1651, max=180281, avg=5159.44, stdev=5993.53 lat (usec): min=1651, max=180282, avg=5160.32, stdev=5993.57 clat percentiles (usec): | 1.00th=[ 1893], 5.00th=[ 2057], 10.00th=[ 2147], 20.00th=[ 2343], | 30.00th=[ 2573], 40.00th=[ 2900], 50.00th=[ 3458], 60.00th=[ 4080], | 70.00th=[ 5145], 80.00th=[ 7177], 90.00th=[ 11076], 95.00th=[ 13435], | 99.00th=[ 19530], 99.50th=[ 21365], 99.90th=[ 26084], 99.95th=[179307], | 99.99th=[179307] bw ( KiB/s): min= 112, max= 768, per=24.79%, avg=320.00, stdev=159.49, samples=37 iops : min= 28, max= 192, avg=79.89, stdev=39.87, samples=37 write: IOPS=85, BW=342KiB/s (350kB/s)(6348KiB/18552msec) clat (usec): min=2533, max=29498, avg=6820.64, stdev=4034.63 lat (usec): min=2534, max=29499, avg=6821.61, stdev=4034.67 clat percentiles (usec): | 1.00th=[ 2769], 5.00th=[ 3097], 10.00th=[ 3326], 20.00th=[ 3654], | 30.00th=[ 4047], 40.00th=[ 4555], 50.00th=[ 5342], 60.00th=[ 6325], | 70.00th=[ 7767], 80.00th=[ 9896], 90.00th=[12387], 95.00th=[15008], | 99.00th=[20055], 99.50th=[23725], 99.90th=[28443], 99.95th=[29492], | 99.99th=[29492] bw ( KiB/s): min= 192, max= 726, per=25.22%, avg=341.00, stdev=151.20, samples=37 iops : min= 48, max= 181, avg=85.14, stdev=37.80, samples=37 lat (msec) : 2=1.76%, 4=41.54%, 10=40.66%, 20=15.10%, 50=0.91% lat (msec) : 250=0.03% cpu : usr=0.25%, sys=11.55%, ctx=3580, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29682: Mon Mar 31 19:11:29 2025 read: IOPS=81, BW=324KiB/s (332kB/s)(6028KiB/18591msec) clat (usec): min=1719, max=29200, avg=4988.98, stdev=3724.78 lat (usec): min=1719, max=29201, avg=4989.76, stdev=3724.80 clat percentiles (usec): | 1.00th=[ 1876], 5.00th=[ 2024], 10.00th=[ 2147], 20.00th=[ 2311], | 30.00th=[ 2638], 40.00th=[ 3032], 50.00th=[ 3523], 60.00th=[ 4228], | 70.00th=[ 5014], 80.00th=[ 7111], 90.00th=[10421], 95.00th=[12780], | 99.00th=[18744], 99.50th=[21890], 99.90th=[25560], 99.95th=[29230], | 99.99th=[29230] bw ( KiB/s): min= 136, max= 760, per=25.03%, avg=323.16, stdev=147.75, samples=37 iops : min= 34, max= 190, avg=80.73, stdev=36.94, samples=37 write: IOPS=84, BW=337KiB/s (345kB/s)(6260KiB/18591msec) clat (msec): min=2, max=183, avg= 7.03, stdev= 6.18 lat (msec): min=2, max=183, avg= 7.03, stdev= 6.18 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 4], 10.00th=[ 4], 20.00th=[ 4], | 30.00th=[ 5], 40.00th=[ 5], 50.00th=[ 6], 60.00th=[ 7], | 70.00th=[ 8], 80.00th=[ 10], 90.00th=[ 13], 95.00th=[ 16], | 99.00th=[ 23], 99.50th=[ 25], 99.90th=[ 35], 99.95th=[ 184], | 99.99th=[ 184] bw ( KiB/s): min= 176, max= 728, per=24.81%, avg=335.49, stdev=155.95, samples=37 iops : min= 44, max= 182, avg=83.81, stdev=38.99, samples=37 lat (msec) : 2=1.99%, 4=40.40%, 10=42.12%, 20=14.32%, 50=1.14% lat (msec) : 250=0.03% cpu : usr=0.18%, sys=11.72%, ctx=3639, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29683: Mon Mar 31 19:11:29 2025 read: IOPS=81, BW=326KiB/s (333kB/s)(6052KiB/18588msec) clat (usec): min=1720, max=25865, avg=4952.15, stdev=3860.07 lat (usec): min=1720, max=25866, avg=4953.03, stdev=3860.08 clat percentiles (usec): | 1.00th=[ 1926], 5.00th=[ 2040], 10.00th=[ 2147], 20.00th=[ 2376], | 30.00th=[ 2573], 40.00th=[ 2933], 50.00th=[ 3425], 60.00th=[ 4047], | 70.00th=[ 4948], 80.00th=[ 6783], 90.00th=[10421], 95.00th=[13042], | 99.00th=[20055], 99.50th=[22152], 99.90th=[25035], 99.95th=[25822], | 99.99th=[25822] bw ( KiB/s): min= 152, max= 726, per=25.11%, avg=324.16, stdev=152.51, samples=37 iops : min= 38, max= 181, avg=80.95, stdev=38.12, samples=37 write: IOPS=83, BW=335KiB/s (344kB/s)(6236KiB/18588msec) clat (msec): min=2, max=183, avg= 7.07, stdev= 6.15 lat (msec): min=2, max=183, avg= 7.08, stdev= 6.15 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 4], 10.00th=[ 4], 20.00th=[ 4], | 30.00th=[ 5], 40.00th=[ 5], 50.00th=[ 6], 60.00th=[ 7], | 70.00th=[ 9], 80.00th=[ 10], 90.00th=[ 13], 95.00th=[ 15], | 99.00th=[ 24], 99.50th=[ 27], 99.90th=[ 35], 99.95th=[ 184], | 99.99th=[ 184] bw ( KiB/s): min= 184, max= 704, per=24.73%, avg=334.32, stdev=152.58, samples=37 iops : min= 46, max= 176, avg=83.49, stdev=38.12, samples=37 lat (msec) : 2=1.50%, 4=42.35%, 10=40.92%, 20=13.96%, 50=1.24% lat (msec) : 250=0.03% cpu : usr=0.15%, sys=11.76%, ctx=3651, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1292KiB/s (1323kB/s), 320KiB/s-326KiB/s (328kB/s-333kB/s), io=23.4MiB (24.6MB), run=18541-18591msec WRITE: bw=1352KiB/s (1385kB/s), 335KiB/s-342KiB/s (344kB/s-350kB/s), io=24.6MiB (25.7MB), run=18541-18591msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=29717: Mon Mar 31 19:11:34 2025 read: IOPS=96, BW=1548KiB/s (1585kB/s)(5920KiB/3824msec) clat (usec): min=71, max=23055, avg=5481.66, stdev=3437.52 lat (usec): min=72, max=23056, avg=5482.57, stdev=3437.50 clat percentiles (usec): | 1.00th=[ 79], 5.00th=[ 114], 10.00th=[ 147], 20.00th=[ 3195], | 30.00th=[ 3818], 40.00th=[ 4359], 50.00th=[ 5014], 60.00th=[ 5735], | 70.00th=[ 6456], 80.00th=[ 7701], 90.00th=[ 9241], 95.00th=[10683], | 99.00th=[17957], 99.50th=[21627], 99.90th=[22938], 99.95th=[22938], | 99.99th=[22938] bw ( KiB/s): min= 1309, max= 1696, per=25.09%, avg=1503.57, stdev=130.07, samples=7 iops : min= 81, max= 106, avg=93.86, stdev= 8.34, samples=7 write: IOPS=104, BW=1665KiB/s (1705kB/s)(6368KiB/3824msec) clat (usec): min=665, max=26259, avg=4398.11, stdev=3764.96 lat (usec): min=666, max=26261, avg=4399.76, stdev=3765.01 clat percentiles (usec): | 1.00th=[ 807], 5.00th=[ 914], 10.00th=[ 1004], 20.00th=[ 1205], | 30.00th=[ 1483], 40.00th=[ 2147], 50.00th=[ 3359], 60.00th=[ 4293], | 70.00th=[ 5407], 80.00th=[ 7111], 90.00th=[10028], 95.00th=[11863], | 99.00th=[16450], 99.50th=[18220], 99.90th=[26346], 99.95th=[26346], | 99.99th=[26346] bw ( KiB/s): min= 1184, max= 2336, per=26.16%, avg=1649.71, stdev=356.84, samples=7 iops : min= 74, max= 146, avg=103.00, stdev=22.32, samples=7 lat (usec) : 100=1.56%, 250=3.39%, 500=0.13%, 750=0.13%, 1000=4.82% lat (msec) : 2=14.84%, 4=20.70%, 10=45.83%, 20=8.07%, 50=0.52% cpu : usr=0.29%, sys=25.48%, ctx=1583, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29718: Mon Mar 31 19:11:34 2025 read: IOPS=90, BW=1445KiB/s (1480kB/s)(5744KiB/3975msec) clat (usec): min=70, max=19209, avg=5606.98, stdev=3411.64 lat (usec): min=70, max=19210, avg=5607.81, stdev=3411.67 clat percentiles (usec): | 1.00th=[ 76], 5.00th=[ 114], 10.00th=[ 161], 20.00th=[ 3228], | 30.00th=[ 3982], 40.00th=[ 4490], 50.00th=[ 5211], 60.00th=[ 5932], | 70.00th=[ 6849], 80.00th=[ 8160], 90.00th=[10028], 95.00th=[11600], | 99.00th=[16909], 99.50th=[19006], 99.90th=[19268], 99.95th=[19268], | 99.99th=[19268] bw ( KiB/s): min= 1184, max= 1632, per=23.72%, avg=1421.29, stdev=197.66, samples=7 iops : min= 74, max= 102, avg=88.71, stdev=12.50, samples=7 write: IOPS=102, BW=1646KiB/s (1686kB/s)(6544KiB/3975msec) clat (usec): min=653, max=19113, avg=4757.61, stdev=3789.19 lat (usec): min=654, max=19114, avg=4759.90, stdev=3789.83 clat percentiles (usec): | 1.00th=[ 840], 5.00th=[ 922], 10.00th=[ 1057], 20.00th=[ 1254], | 30.00th=[ 1631], 40.00th=[ 2868], 50.00th=[ 3884], 60.00th=[ 4817], | 70.00th=[ 5866], 80.00th=[ 7963], 90.00th=[10421], 95.00th=[12125], | 99.00th=[15270], 99.50th=[17171], 99.90th=[19006], 99.95th=[19006], | 99.99th=[19006] bw ( KiB/s): min= 1376, max= 1792, per=25.44%, avg=1604.00, stdev=178.00, samples=7 iops : min= 86, max= 112, avg=100.14, stdev=11.02, samples=7 lat (usec) : 100=1.30%, 250=3.65%, 500=0.26%, 750=0.39%, 1000=3.78% lat (msec) : 2=14.06%, 4=18.36%, 10=47.14%, 20=11.07% cpu : usr=0.20%, sys=24.79%, ctx=1727, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29720: Mon Mar 31 19:11:34 2025 read: IOPS=97, BW=1566KiB/s (1603kB/s)(6256KiB/3996msec) clat (usec): min=70, max=20201, avg=5507.82, stdev=3374.50 lat (usec): min=71, max=20202, avg=5517.65, stdev=3381.01 clat percentiles (usec): | 1.00th=[ 78], 5.00th=[ 105], 10.00th=[ 159], 20.00th=[ 3359], | 30.00th=[ 4047], 40.00th=[ 4686], 50.00th=[ 5211], 60.00th=[ 5932], | 70.00th=[ 6783], 80.00th=[ 7701], 90.00th=[ 9110], 95.00th=[10552], | 99.00th=[18744], 99.50th=[20055], 99.90th=[20317], 99.95th=[20317], | 99.99th=[20317] bw ( KiB/s): min= 960, max= 1728, per=25.09%, avg=1503.43, stdev=267.34, samples=7 iops : min= 60, max= 108, avg=93.86, stdev=16.64, samples=7 write: IOPS=94, BW=1510KiB/s (1546kB/s)(6032KiB/3996msec) clat (usec): min=676, max=25885, avg=4780.73, stdev=4046.12 lat (usec): min=677, max=25886, avg=4781.98, stdev=4046.11 clat percentiles (usec): | 1.00th=[ 865], 5.00th=[ 963], 10.00th=[ 1057], 20.00th=[ 1270], | 30.00th=[ 1565], 40.00th=[ 2737], 50.00th=[ 3785], 60.00th=[ 4752], | 70.00th=[ 5997], 80.00th=[ 7504], 90.00th=[10552], 95.00th=[12649], | 99.00th=[15795], 99.50th=[24511], 99.90th=[25822], 99.95th=[25822], | 99.99th=[25822] bw ( KiB/s): min= 1117, max= 1824, per=23.63%, avg=1489.86, stdev=261.90, samples=7 iops : min= 69, max= 114, avg=93.00, stdev=16.56, samples=7 lat (usec) : 100=1.56%, 250=4.17%, 500=0.39%, 750=0.13%, 1000=4.04% lat (msec) : 2=13.54%, 4=17.06%, 10=50.52%, 20=7.94%, 50=0.65% cpu : usr=0.28%, sys=25.03%, ctx=1693, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29722: Mon Mar 31 19:11:34 2025 read: IOPS=97, BW=1567KiB/s (1605kB/s)(6032KiB/3849msec) clat (usec): min=69, max=20909, avg=5302.69, stdev=3321.86 lat (usec): min=69, max=20910, avg=5306.24, stdev=3326.73 clat percentiles (usec): | 1.00th=[ 72], 5.00th=[ 94], 10.00th=[ 139], 20.00th=[ 3032], | 30.00th=[ 3654], 40.00th=[ 4490], 50.00th=[ 5014], 60.00th=[ 5735], | 70.00th=[ 6456], 80.00th=[ 7767], 90.00th=[ 9241], 95.00th=[10945], | 99.00th=[17433], 99.50th=[18220], 99.90th=[20841], 99.95th=[20841], | 99.99th=[20841] bw ( KiB/s): min= 1376, max= 1724, per=25.38%, avg=1521.14, stdev=134.67, samples=7 iops : min= 86, max= 107, avg=94.86, stdev= 8.25, samples=7 write: IOPS=101, BW=1625KiB/s (1664kB/s)(6256KiB/3849msec) clat (usec): min=715, max=22763, avg=4613.48, stdev=3861.11 lat (usec): min=716, max=22764, avg=4615.31, stdev=3860.67 clat percentiles (usec): | 1.00th=[ 807], 5.00th=[ 930], 10.00th=[ 988], 20.00th=[ 1188], | 30.00th=[ 1516], 40.00th=[ 2343], 50.00th=[ 3687], 60.00th=[ 4555], | 70.00th=[ 5735], 80.00th=[ 7570], 90.00th=[10552], 95.00th=[12256], | 99.00th=[15795], 99.50th=[19530], 99.90th=[22676], 99.95th=[22676], | 99.99th=[22676] bw ( KiB/s): min= 1344, max= 1856, per=25.50%, avg=1608.00, stdev=208.55, samples=7 iops : min= 84, max= 116, avg=100.29, stdev=13.00, samples=7 lat (usec) : 100=2.60%, 250=3.78%, 500=0.13%, 750=0.13%, 1000=5.47% lat (msec) : 2=13.54%, 4=18.49%, 10=46.74%, 20=8.85%, 50=0.26% cpu : usr=0.23%, sys=25.23%, ctx=1636, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=5994KiB/s (6138kB/s), 1445KiB/s-1567KiB/s (1480kB/s-1605kB/s), io=23.4MiB (24.5MB), run=3824-3996msec WRITE: bw=6306KiB/s (6458kB/s), 1510KiB/s-1665KiB/s (1546kB/s-1705kB/s), io=24.6MiB (25.8MB), run=3824-3996msec rand-rw: (groupid=0, jobs=1): err= 0: pid=29719: Mon Mar 31 19:11:35 2025 read: IOPS=71, BW=1147KiB/s (1175kB/s)(5920KiB/5160msec) clat (usec): min=2050, max=29890, avg=5471.24, stdev=4246.07 lat (usec): min=2051, max=29891, avg=5472.26, stdev=4245.99 clat percentiles (usec): | 1.00th=[ 2311], 5.00th=[ 2540], 10.00th=[ 2671], 20.00th=[ 2900], | 30.00th=[ 3064], 40.00th=[ 3294], 50.00th=[ 3654], 60.00th=[ 4228], | 70.00th=[ 4948], 80.00th=[ 6849], 90.00th=[12518], 95.00th=[15795], | 99.00th=[19530], 99.50th=[24773], 99.90th=[30016], 99.95th=[30016], | 99.99th=[30016] bw ( KiB/s): min= 576, max= 2048, per=24.06%, avg=1116.60, stdev=523.38, samples=10 iops : min= 36, max= 128, avg=69.70, stdev=32.74, samples=10 write: IOPS=77, BW=1234KiB/s (1264kB/s)(6368KiB/5160msec) clat (usec): min=3071, max=38287, avg=7752.04, stdev=4971.50 lat (usec): min=3074, max=38289, avg=7754.15, stdev=4971.28 clat percentiles (usec): | 1.00th=[ 3359], 5.00th=[ 3687], 10.00th=[ 3851], 20.00th=[ 4293], | 30.00th=[ 4555], 40.00th=[ 4948], 50.00th=[ 5604], 60.00th=[ 6849], | 70.00th=[ 8356], 80.00th=[10945], 90.00th=[14615], 95.00th=[17695], | 99.00th=[26870], 99.50th=[33817], 99.90th=[38536], 99.95th=[38536], | 99.99th=[38536] bw ( KiB/s): min= 704, max= 2208, per=24.83%, avg=1212.60, stdev=525.60, samples=10 iops : min= 44, max= 138, avg=75.70, stdev=32.90, samples=10 lat (msec) : 4=34.11%, 10=46.48%, 20=17.58%, 50=1.82% cpu : usr=0.23%, sys=12.68%, ctx=963, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29721: Mon Mar 31 19:11:35 2025 read: IOPS=69, BW=1118KiB/s (1145kB/s)(5744KiB/5137msec) clat (usec): min=2093, max=30730, avg=5509.44, stdev=4230.02 lat (usec): min=2094, max=30730, avg=5510.41, stdev=4229.97 clat percentiles (usec): | 1.00th=[ 2311], 5.00th=[ 2474], 10.00th=[ 2606], 20.00th=[ 2835], | 30.00th=[ 2999], 40.00th=[ 3195], 50.00th=[ 3654], 60.00th=[ 4228], | 70.00th=[ 5211], 80.00th=[ 7504], 90.00th=[12125], 95.00th=[14877], | 99.00th=[20579], 99.50th=[21890], 99.90th=[30802], 99.95th=[30802], | 99.99th=[30802] bw ( KiB/s): min= 704, max= 2304, per=23.79%, avg=1104.00, stdev=525.22, samples=10 iops : min= 44, max= 144, avg=69.00, stdev=32.83, samples=10 write: IOPS=79, BW=1274KiB/s (1304kB/s)(6544KiB/5137msec) clat (usec): min=3090, max=30630, avg=7561.64, stdev=4481.08 lat (usec): min=3092, max=30635, avg=7563.38, stdev=4481.03 clat percentiles (usec): | 1.00th=[ 3261], 5.00th=[ 3621], 10.00th=[ 3851], 20.00th=[ 4178], | 30.00th=[ 4490], 40.00th=[ 5014], 50.00th=[ 5669], 60.00th=[ 6980], | 70.00th=[ 8717], 80.00th=[10683], 90.00th=[14222], 95.00th=[16188], | 99.00th=[21890], 99.50th=[26346], 99.90th=[30540], 99.95th=[30540], | 99.99th=[30540] bw ( KiB/s): min= 928, max= 2560, per=25.56%, avg=1248.00, stdev=546.61, samples=10 iops : min= 58, max= 160, avg=78.00, stdev=34.16, samples=10 lat (msec) : 4=33.33%, 10=47.92%, 20=16.93%, 50=1.82% cpu : usr=0.23%, sys=12.56%, ctx=960, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29723: Mon Mar 31 19:11:35 2025 read: IOPS=77, BW=1238KiB/s (1268kB/s)(6256KiB/5052msec) clat (usec): min=2173, max=31171, avg=5494.46, stdev=4065.79 lat (usec): min=2173, max=31172, avg=5495.33, stdev=4065.76 clat percentiles (usec): | 1.00th=[ 2278], 5.00th=[ 2507], 10.00th=[ 2638], 20.00th=[ 2835], | 30.00th=[ 3064], 40.00th=[ 3359], 50.00th=[ 3785], 60.00th=[ 4293], | 70.00th=[ 5407], 80.00th=[ 7439], 90.00th=[11863], 95.00th=[15008], | 99.00th=[18482], 99.50th=[20055], 99.90th=[31065], 99.95th=[31065], | 99.99th=[31065] bw ( KiB/s): min= 766, max= 2528, per=26.73%, avg=1240.50, stdev=575.11, samples=10 iops : min= 47, max= 158, avg=77.20, stdev=36.03, samples=10 write: IOPS=74, BW=1194KiB/s (1223kB/s)(6032KiB/5052msec) clat (usec): min=2822, max=38442, avg=7668.47, stdev=4470.26 lat (usec): min=2823, max=38444, avg=7670.27, stdev=4470.18 clat percentiles (usec): | 1.00th=[ 3228], 5.00th=[ 3556], 10.00th=[ 3916], 20.00th=[ 4228], | 30.00th=[ 4555], 40.00th=[ 4883], 50.00th=[ 5866], 60.00th=[ 7242], | 70.00th=[ 9110], 80.00th=[10945], 90.00th=[14353], 95.00th=[16450], | 99.00th=[21365], 99.50th=[23200], 99.90th=[38536], 99.95th=[38536], | 99.99th=[38536] bw ( KiB/s): min= 800, max= 2107, per=24.03%, avg=1173.20, stdev=473.74, samples=10 iops : min= 50, max= 131, avg=73.00, stdev=29.55, samples=10 lat (msec) : 4=34.64%, 10=46.61%, 20=17.97%, 50=0.78% cpu : usr=0.26%, sys=12.41%, ctx=944, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29724: Mon Mar 31 19:11:35 2025 read: IOPS=74, BW=1197KiB/s (1226kB/s)(6032KiB/5039msec) clat (usec): min=2177, max=24649, avg=5499.74, stdev=4198.17 lat (usec): min=2178, max=24650, avg=5500.77, stdev=4198.19 clat percentiles (usec): | 1.00th=[ 2311], 5.00th=[ 2474], 10.00th=[ 2606], 20.00th=[ 2835], | 30.00th=[ 3064], 40.00th=[ 3261], 50.00th=[ 3720], 60.00th=[ 4228], | 70.00th=[ 5145], 80.00th=[ 6915], 90.00th=[12387], 95.00th=[15270], | 99.00th=[20841], 99.50th=[22938], 99.90th=[24773], 99.95th=[24773], | 99.99th=[24773] bw ( KiB/s): min= 894, max= 2272, per=25.51%, avg=1183.80, stdev=491.72, samples=10 iops : min= 55, max= 142, avg=73.90, stdev=30.79, samples=10 write: IOPS=77, BW=1242KiB/s (1271kB/s)(6256KiB/5039msec) clat (usec): min=2999, max=27405, avg=7422.09, stdev=4439.54 lat (usec): min=3000, max=27406, avg=7423.85, stdev=4439.45 clat percentiles (usec): | 1.00th=[ 3261], 5.00th=[ 3621], 10.00th=[ 3916], 20.00th=[ 4228], | 30.00th=[ 4555], 40.00th=[ 4883], 50.00th=[ 5473], 60.00th=[ 6390], | 70.00th=[ 8225], 80.00th=[10159], 90.00th=[13960], 95.00th=[16909], | 99.00th=[22676], 99.50th=[27132], 99.90th=[27395], 99.95th=[27395], | 99.99th=[27395] bw ( KiB/s): min= 736, max= 2176, per=25.36%, avg=1238.10, stdev=499.95, samples=10 iops : min= 46, max= 136, avg=77.30, stdev=31.27, samples=10 lat (msec) : 4=34.11%, 10=48.31%, 20=16.02%, 50=1.56% cpu : usr=0.30%, sys=12.58%, ctx=984, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=4642KiB/s (4753kB/s), 1118KiB/s-1238KiB/s (1145kB/s-1268kB/s), io=23.4MiB (24.5MB), run=5039-5160msec WRITE: bw=4884KiB/s (5001kB/s), 1194KiB/s-1274KiB/s (1223kB/s-1304kB/s), io=24.6MiB (25.8MB), run=5039-5160msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=29735: Mon Mar 31 19:11:37 2025 read: IOPS=9, BW=9752KiB/s (9986kB/s)(7168KiB/735msec) clat (usec): min=588, max=63740, avg=44114.00, stdev=20368.20 lat (usec): min=589, max=63741, avg=44115.56, stdev=20368.17 clat percentiles (usec): | 1.00th=[ 586], 5.00th=[ 586], 10.00th=[ 586], 20.00th=[43254], | 30.00th=[45876], 40.00th=[45876], 50.00th=[48497], 60.00th=[51643], | 70.00th=[51643], 80.00th=[55313], 90.00th=[63701], 95.00th=[63701], | 99.00th=[63701], 99.50th=[63701], 99.90th=[63701], 99.95th=[63701], | 99.99th=[63701] bw ( KiB/s): min=10240, max=10240, per=43.45%, avg=10240.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 write: IOPS=6, BW=6966KiB/s (7133kB/s)(5120KiB/735msec) clat (msec): min=35, max=113, avg=78.36, stdev=29.43 lat (msec): min=35, max=113, avg=78.41, stdev=29.44 clat percentiles (msec): | 1.00th=[ 36], 5.00th=[ 36], 10.00th=[ 36], 20.00th=[ 36], | 30.00th=[ 67], 40.00th=[ 67], 50.00th=[ 86], 60.00th=[ 86], | 70.00th=[ 92], 80.00th=[ 92], 90.00th=[ 114], 95.00th=[ 114], | 99.00th=[ 114], 99.50th=[ 114], 99.90th=[ 114], 99.95th=[ 114], | 99.99th=[ 114] bw ( KiB/s): min= 4096, max= 4096, per=12.41%, avg=4096.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (usec) : 750=8.33% lat (msec) : 50=33.33%, 100=50.00%, 250=8.33% cpu : usr=0.14%, sys=34.83%, ctx=500, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29737: Mon Mar 31 19:11:37 2025 read: IOPS=2, BW=2357KiB/s (2413kB/s)(2048KiB/869msec) clat (msec): min=37, max=106, avg=72.47, stdev=48.75 lat (msec): min=37, max=106, avg=72.47, stdev=48.75 clat percentiles (msec): | 1.00th=[ 39], 5.00th=[ 39], 10.00th=[ 39], 20.00th=[ 39], | 30.00th=[ 39], 40.00th=[ 39], 50.00th=[ 39], 60.00th=[ 107], | 70.00th=[ 107], 80.00th=[ 107], 90.00th=[ 107], 95.00th=[ 107], | 99.00th=[ 107], 99.50th=[ 107], 99.90th=[ 107], 99.95th=[ 107], | 99.99th=[ 107] bw ( KiB/s): min= 2043, max= 2043, per=8.67%, avg=2043.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=11, BW=11.5MiB/s (12.1MB/s)(10.0MiB/869msec) clat (msec): min=22, max=147, avg=68.03, stdev=44.58 lat (msec): min=22, max=147, avg=68.17, stdev=44.57 clat percentiles (msec): | 1.00th=[ 22], 5.00th=[ 22], 10.00th=[ 22], 20.00th=[ 23], | 30.00th=[ 28], 40.00th=[ 53], 50.00th=[ 56], 60.00th=[ 59], | 70.00th=[ 96], 80.00th=[ 96], 90.00th=[ 136], 95.00th=[ 148], | 99.00th=[ 148], 99.50th=[ 148], 99.90th=[ 148], 99.95th=[ 148], | 99.99th=[ 148] bw ( KiB/s): min= 8175, max= 8175, per=24.78%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 7, max= 7, avg= 7.00, stdev= 0.00, samples=1 lat (msec) : 50=33.33%, 100=41.67%, 250=25.00% cpu : usr=0.12%, sys=40.67%, ctx=399, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29739: Mon Mar 31 19:11:37 2025 read: IOPS=11, BW=11.1MiB/s (11.6MB/s)(6144KiB/541msec) clat (usec): min=575, max=53629, avg=19884.71, stdev=22545.52 lat (usec): min=578, max=53630, avg=19886.15, stdev=22545.53 clat percentiles (usec): | 1.00th=[ 578], 5.00th=[ 578], 10.00th=[ 578], 20.00th=[ 652], | 30.00th=[ 652], 40.00th=[ 742], 50.00th=[ 742], 60.00th=[30016], | 70.00th=[33817], 80.00th=[33817], 90.00th=[53740], 95.00th=[53740], | 99.00th=[53740], 99.50th=[53740], 99.90th=[53740], 99.95th=[53740], | 99.99th=[53740] bw ( KiB/s): min=10240, max=10240, per=43.45%, avg=10240.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 write: IOPS=11, BW=11.1MiB/s (11.6MB/s)(6144KiB/541msec) clat (msec): min=32, max=152, avg=64.32, stdev=44.44 lat (msec): min=32, max=152, avg=64.37, stdev=44.44 clat percentiles (msec): | 1.00th=[ 33], 5.00th=[ 33], 10.00th=[ 33], 20.00th=[ 46], | 30.00th=[ 46], 40.00th=[ 46], 50.00th=[ 46], 60.00th=[ 46], | 70.00th=[ 65], 80.00th=[ 65], 90.00th=[ 153], 95.00th=[ 153], | 99.00th=[ 153], 99.50th=[ 153], 99.90th=[ 153], 99.95th=[ 153], | 99.99th=[ 153] bw ( KiB/s): min=12288, max=12288, per=37.24%, avg=12288.00, stdev= 0.00, samples=1 iops : min= 12, max= 12, avg=12.00, stdev= 0.00, samples=1 lat (usec) : 750=25.00% lat (msec) : 50=50.00%, 100=16.67%, 250=8.33% cpu : usr=0.00%, sys=36.48%, ctx=428, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29740: Mon Mar 31 19:11:37 2025 read: IOPS=9, BW=9326KiB/s (9550kB/s)(5120KiB/549msec) clat (usec): min=753, max=52164, avg=30044.32, stdev=19871.16 lat (usec): min=755, max=52166, avg=30046.31, stdev=19871.02 clat percentiles (usec): | 1.00th=[ 750], 5.00th=[ 750], 10.00th=[ 750], 20.00th=[ 750], | 30.00th=[22152], 40.00th=[22152], 50.00th=[32375], 60.00th=[32375], | 70.00th=[42730], 80.00th=[42730], 90.00th=[52167], 95.00th=[52167], | 99.00th=[52167], 99.50th=[52167], 99.90th=[52167], 99.95th=[52167], | 99.99th=[52167] bw ( KiB/s): min= 8175, max= 8175, per=34.69%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 7, max= 7, avg= 7.00, stdev= 0.00, samples=1 write: IOPS=12, BW=12.8MiB/s (13.4MB/s)(7168KiB/549msec) clat (msec): min=33, max=103, avg=55.71, stdev=28.45 lat (msec): min=34, max=103, avg=55.80, stdev=28.46 clat percentiles (msec): | 1.00th=[ 34], 5.00th=[ 34], 10.00th=[ 34], 20.00th=[ 35], | 30.00th=[ 39], 40.00th=[ 39], 50.00th=[ 39], 60.00th=[ 54], | 70.00th=[ 54], 80.00th=[ 89], 90.00th=[ 104], 95.00th=[ 104], | 99.00th=[ 104], 99.50th=[ 104], 99.90th=[ 104], 99.95th=[ 104], | 99.99th=[ 104] bw ( KiB/s): min=12263, max=12263, per=37.17%, avg=12263.00, stdev= 0.00, samples=1 iops : min= 11, max= 11, avg=11.00, stdev= 0.00, samples=1 lat (usec) : 1000=8.33% lat (msec) : 50=58.33%, 100=25.00%, 250=8.33% cpu : usr=0.00%, sys=43.25%, ctx=434, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=23.0MiB/s (24.1MB/s), 2357KiB/s-11.1MiB/s (2413kB/s-11.6MB/s), io=20.0MiB (20.0MB), run=541-869msec WRITE: bw=32.2MiB/s (33.8MB/s), 6966KiB/s-12.8MiB/s (7133kB/s-13.4MB/s), io=28.0MiB (29.4MB), run=541-869msec rand-rw: (groupid=0, jobs=1): err= 0: pid=29734: Mon Mar 31 19:11:37 2025 read: IOPS=7, BW=7766KiB/s (7952kB/s)(7168KiB/923msec) clat (msec): min=31, max=159, avg=88.12, stdev=46.07 lat (msec): min=31, max=159, avg=88.12, stdev=46.07 clat percentiles (msec): | 1.00th=[ 32], 5.00th=[ 32], 10.00th=[ 32], 20.00th=[ 58], | 30.00th=[ 62], 40.00th=[ 62], 50.00th=[ 70], 60.00th=[ 102], | 70.00th=[ 102], 80.00th=[ 138], 90.00th=[ 159], 95.00th=[ 159], | 99.00th=[ 159], 99.50th=[ 159], 99.90th=[ 159], 99.95th=[ 159], | 99.99th=[ 159] bw ( KiB/s): min= 5965, max= 5965, per=27.50%, avg=5965.00, stdev= 0.00, samples=1 iops : min= 5, max= 5, avg= 5.00, stdev= 0.00, samples=1 write: IOPS=5, BW=5547KiB/s (5680kB/s)(5120KiB/923msec) clat (msec): min=28, max=119, avg=54.59, stdev=38.38 lat (msec): min=28, max=119, avg=54.67, stdev=38.38 clat percentiles (msec): | 1.00th=[ 29], 5.00th=[ 29], 10.00th=[ 29], 20.00th=[ 29], | 30.00th=[ 32], 40.00th=[ 32], 50.00th=[ 33], 60.00th=[ 33], | 70.00th=[ 62], 80.00th=[ 62], 90.00th=[ 120], 95.00th=[ 120], | 99.00th=[ 120], 99.50th=[ 120], 99.90th=[ 120], 99.95th=[ 120], | 99.99th=[ 120] bw ( KiB/s): min= 1988, max= 1988, per=6.55%, avg=1988.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 50=33.33%, 100=33.33%, 250=33.33% cpu : usr=0.11%, sys=13.76%, ctx=54, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29736: Mon Mar 31 19:11:37 2025 read: IOPS=2, BW=2169KiB/s (2222kB/s)(2048KiB/944msec) clat (usec): min=39479, max=82242, avg=60861.06, stdev=30238.03 lat (usec): min=39481, max=82247, avg=60864.36, stdev=30240.11 clat percentiles (usec): | 1.00th=[39584], 5.00th=[39584], 10.00th=[39584], 20.00th=[39584], | 30.00th=[39584], 40.00th=[39584], 50.00th=[39584], 60.00th=[82314], | 70.00th=[82314], 80.00th=[82314], 90.00th=[82314], 95.00th=[82314], | 99.00th=[82314], 99.50th=[82314], 99.90th=[82314], 99.95th=[82314], | 99.99th=[82314] bw ( KiB/s): min= 2007, max= 2007, per=9.25%, avg=2007.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=10, BW=10.6MiB/s (11.1MB/s)(10.0MiB/944msec) clat (msec): min=20, max=183, avg=74.10, stdev=45.10 lat (msec): min=20, max=183, avg=74.19, stdev=45.11 clat percentiles (msec): | 1.00th=[ 21], 5.00th=[ 21], 10.00th=[ 21], 20.00th=[ 44], | 30.00th=[ 48], 40.00th=[ 52], 50.00th=[ 68], 60.00th=[ 69], | 70.00th=[ 75], 80.00th=[ 75], 90.00th=[ 111], 95.00th=[ 184], | 99.00th=[ 184], 99.50th=[ 184], 99.90th=[ 184], 99.95th=[ 184], | 99.99th=[ 184] bw ( KiB/s): min= 6023, max= 6023, per=19.83%, avg=6023.00, stdev= 0.00, samples=1 iops : min= 5, max= 5, avg= 5.00, stdev= 0.00, samples=1 lat (msec) : 50=33.33%, 100=50.00%, 250=16.67% cpu : usr=0.21%, sys=15.69%, ctx=175, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29738: Mon Mar 31 19:11:37 2025 read: IOPS=6, BW=6990KiB/s (7158kB/s)(6144KiB/879msec) clat (usec): min=18141, max=73893, avg=38692.69, stdev=23890.86 lat (usec): min=18144, max=73896, avg=38694.96, stdev=23891.11 clat percentiles (usec): | 1.00th=[18220], 5.00th=[18220], 10.00th=[18220], 20.00th=[18482], | 30.00th=[18482], 40.00th=[19268], 50.00th=[19268], 60.00th=[43779], | 70.00th=[58459], 80.00th=[58459], 90.00th=[73925], 95.00th=[73925], | 99.00th=[73925], 99.50th=[73925], 99.90th=[73925], 99.95th=[73925], | 99.99th=[73925] bw ( KiB/s): min= 4007, max= 4007, per=18.47%, avg=4007.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 write: IOPS=6, BW=6990KiB/s (7158kB/s)(6144KiB/879msec) clat (msec): min=42, max=168, avg=94.92, stdev=56.85 lat (msec): min=42, max=168, avg=94.99, stdev=56.84 clat percentiles (msec): | 1.00th=[ 43], 5.00th=[ 43], 10.00th=[ 43], 20.00th=[ 58], | 30.00th=[ 58], 40.00th=[ 66], 50.00th=[ 66], 60.00th=[ 70], | 70.00th=[ 167], 80.00th=[ 167], 90.00th=[ 169], 95.00th=[ 169], | 99.00th=[ 169], 99.50th=[ 169], 99.90th=[ 169], 99.95th=[ 169], | 99.99th=[ 169] bw ( KiB/s): min= 6011, max= 6011, per=19.79%, avg=6011.00, stdev= 0.00, samples=1 iops : min= 5, max= 5, avg= 5.00, stdev= 0.00, samples=1 lat (msec) : 20=25.00%, 50=16.67%, 100=41.67%, 250=16.67% cpu : usr=0.00%, sys=13.33%, ctx=93, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29741: Mon Mar 31 19:11:37 2025 read: IOPS=5, BW=5785KiB/s (5924kB/s)(5120KiB/885msec) clat (usec): min=30368, max=84340, avg=56573.96, stdev=24397.88 lat (usec): min=30370, max=84343, avg=56576.24, stdev=24397.83 clat percentiles (usec): | 1.00th=[30278], 5.00th=[30278], 10.00th=[30278], 20.00th=[30278], | 30.00th=[31589], 40.00th=[31589], 50.00th=[64750], 60.00th=[64750], | 70.00th=[71828], 80.00th=[71828], 90.00th=[84411], 95.00th=[84411], | 99.00th=[84411], 99.50th=[84411], 99.90th=[84411], 99.95th=[84411], | 99.99th=[84411] write: IOPS=7, BW=8099KiB/s (8294kB/s)(7168KiB/885msec) clat (msec): min=20, max=137, avg=76.01, stdev=49.99 lat (msec): min=20, max=137, avg=76.08, stdev=50.00 clat percentiles (msec): | 1.00th=[ 21], 5.00th=[ 21], 10.00th=[ 21], 20.00th=[ 33], | 30.00th=[ 45], 40.00th=[ 45], 50.00th=[ 51], 60.00th=[ 115], | 70.00th=[ 115], 80.00th=[ 132], 90.00th=[ 138], 95.00th=[ 138], | 99.00th=[ 138], 99.50th=[ 138], 99.90th=[ 138], 99.95th=[ 138], | 99.99th=[ 138] bw ( KiB/s): min= 8192, max= 8192, per=26.97%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 lat (msec) : 50=41.67%, 100=33.33%, 250=25.00% cpu : usr=0.23%, sys=11.99%, ctx=79, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=21.2MiB/s (22.2MB/s), 2169KiB/s-7766KiB/s (2222kB/s-7952kB/s), io=20.0MiB (20.0MB), run=879-944msec WRITE: bw=29.7MiB/s (31.1MB/s), 5547KiB/s-10.6MiB/s (5680kB/s-11.1MB/s), io=28.0MiB (29.4MB), run=879-944msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=29752: Mon Mar 31 19:11:39 2025 read: IOPS=6, BW=27.6MiB/s (28.9MB/s)(12.0MiB/435msec) clat (msec): min=67, max=217, avg=139.90, stdev=75.43 lat (msec): min=67, max=217, avg=139.90, stdev=75.43 clat percentiles (msec): | 1.00th=[ 67], 5.00th=[ 67], 10.00th=[ 67], 20.00th=[ 67], | 30.00th=[ 67], 40.00th=[ 136], 50.00th=[ 136], 60.00th=[ 136], | 70.00th=[ 218], 80.00th=[ 218], 90.00th=[ 218], 95.00th=[ 218], | 99.00th=[ 218], 99.50th=[ 218], 99.90th=[ 218], 99.95th=[ 218], | 99.99th=[ 218] lat (msec) : 100=33.33%, 250=66.67% cpu : usr=0.00%, sys=39.40%, ctx=1761, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29753: Mon Mar 31 19:11:39 2025 read: IOPS=1, BW=5611KiB/s (5746kB/s)(4096KiB/730msec) clat (nsec): min=96457k, max=96457k, avg=96456910.00, stdev= 0.00 lat (nsec): min=96461k, max=96461k, avg=96460983.00, stdev= 0.00 clat percentiles (usec): | 1.00th=[95945], 5.00th=[95945], 10.00th=[95945], 20.00th=[95945], | 30.00th=[95945], 40.00th=[95945], 50.00th=[95945], 60.00th=[95945], | 70.00th=[95945], 80.00th=[95945], 90.00th=[95945], 95.00th=[95945], | 99.00th=[95945], 99.50th=[95945], 99.90th=[95945], 99.95th=[95945], | 99.99th=[95945] bw ( KiB/s): min= 8175, max= 8175, per=36.43%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=2, BW=10.0MiB/s (11.5MB/s)(8192KiB/730msec) clat (msec): min=274, max=331, avg=303.24, stdev=39.95 lat (msec): min=275, max=331, avg=303.54, stdev=39.92 clat percentiles (msec): | 1.00th=[ 275], 5.00th=[ 275], 10.00th=[ 275], 20.00th=[ 275], | 30.00th=[ 275], 40.00th=[ 275], 50.00th=[ 275], 60.00th=[ 334], | 70.00th=[ 334], 80.00th=[ 334], 90.00th=[ 334], 95.00th=[ 334], | 99.00th=[ 334], 99.50th=[ 334], 99.90th=[ 334], 99.95th=[ 334], | 99.99th=[ 334] bw ( KiB/s): min= 8175, max= 8175, per=23.13%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 500=66.67% cpu : usr=0.00%, sys=39.51%, ctx=749, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29754: Mon Mar 31 19:11:39 2025 write: IOPS=3, BW=12.9MiB/s (13.6MB/s)(12.0MiB/927msec) clat (msec): min=117, max=390, avg=290.66, stdev=150.65 lat (msec): min=117, max=391, avg=291.10, stdev=150.76 clat percentiles (msec): | 1.00th=[ 117], 5.00th=[ 117], 10.00th=[ 117], 20.00th=[ 117], | 30.00th=[ 117], 40.00th=[ 363], 50.00th=[ 363], 60.00th=[ 363], | 70.00th=[ 393], 80.00th=[ 393], 90.00th=[ 393], 95.00th=[ 393], | 99.00th=[ 393], 99.50th=[ 393], 99.90th=[ 393], 99.95th=[ 393], | 99.99th=[ 393] bw ( KiB/s): min= 8175, max= 8175, per=23.13%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=66.67% cpu : usr=0.22%, sys=36.07%, ctx=36, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29755: Mon Mar 31 19:11:39 2025 write: IOPS=3, BW=15.0MiB/s (15.7MB/s)(12.0MiB/800msec) clat (msec): min=201, max=325, avg=249.70, stdev=66.66 lat (msec): min=202, max=326, avg=250.06, stdev=66.68 clat percentiles (msec): | 1.00th=[ 203], 5.00th=[ 203], 10.00th=[ 203], 20.00th=[ 203], | 30.00th=[ 203], 40.00th=[ 222], 50.00th=[ 222], 60.00th=[ 222], | 70.00th=[ 326], 80.00th=[ 326], 90.00th=[ 326], 95.00th=[ 326], | 99.00th=[ 326], 99.50th=[ 326], 99.90th=[ 326], 99.95th=[ 326], | 99.99th=[ 326] bw ( KiB/s): min=16384, max=16384, per=46.35%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.13%, sys=45.81%, ctx=80, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=21.9MiB/s (22.0MB/s), 5611KiB/s-27.6MiB/s (5746kB/s-28.9MB/s), io=16.0MiB (16.8MB), run=435-730msec WRITE: bw=34.5MiB/s (36.2MB/s), 10.0MiB/s-15.0MiB/s (11.5MB/s-15.7MB/s), io=32.0MiB (33.6MB), run=730-927msec rand-rw: (groupid=0, jobs=1): err= 0: pid=29747: Mon Mar 31 19:11:40 2025 read: IOPS=4, BW=17.7MiB/s (18.5MB/s)(12.0MiB/679msec) clat (msec): min=82, max=312, avg=206.23, stdev=115.84 lat (msec): min=82, max=312, avg=206.23, stdev=115.84 clat percentiles (msec): | 1.00th=[ 83], 5.00th=[ 83], 10.00th=[ 83], 20.00th=[ 83], | 30.00th=[ 83], 40.00th=[ 224], 50.00th=[ 224], 60.00th=[ 224], | 70.00th=[ 313], 80.00th=[ 313], 90.00th=[ 313], 95.00th=[ 313], | 99.00th=[ 313], 99.50th=[ 313], 99.90th=[ 313], 99.95th=[ 313], | 99.99th=[ 313] bw ( KiB/s): min= 8192, max= 8192, per=50.15%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 250=33.33%, 500=33.33% cpu : usr=0.00%, sys=14.60%, ctx=46, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29748: Mon Mar 31 19:11:40 2025 read: IOPS=0, BW=4084KiB/s (4182kB/s)(4096KiB/1003msec) clat (nsec): min=165003k, max=165003k, avg=165002699.00, stdev= 0.00 lat (nsec): min=165005k, max=165005k, avg=165004689.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 165], 5.00th=[ 165], 10.00th=[ 165], 20.00th=[ 165], | 30.00th=[ 165], 40.00th=[ 165], 50.00th=[ 165], 60.00th=[ 165], | 70.00th=[ 165], 80.00th=[ 165], 90.00th=[ 165], 95.00th=[ 165], | 99.00th=[ 165], 99.50th=[ 165], 99.90th=[ 165], 99.95th=[ 165], | 99.99th=[ 165] bw ( KiB/s): min= 8192, max= 8192, per=50.15%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=1, BW=8167KiB/s (8364kB/s)(8192KiB/1003msec) clat (msec): min=358, max=457, avg=407.68, stdev=69.77 lat (msec): min=358, max=457, avg=408.02, stdev=69.68 clat percentiles (msec): | 1.00th=[ 359], 5.00th=[ 359], 10.00th=[ 359], 20.00th=[ 359], | 30.00th=[ 359], 40.00th=[ 359], 50.00th=[ 359], 60.00th=[ 456], | 70.00th=[ 456], 80.00th=[ 456], 90.00th=[ 456], 95.00th=[ 456], | 99.00th=[ 456], 99.50th=[ 456], 99.90th=[ 456], 99.95th=[ 456], | 99.99th=[ 456] lat (msec) : 250=33.33%, 500=66.67% cpu : usr=0.10%, sys=12.97%, ctx=157, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29749: Mon Mar 31 19:11:40 2025 write: IOPS=2, BW=11.1MiB/s (11.6MB/s)(12.0MiB/1081msec) clat (msec): min=120, max=620, avg=354.33, stdev=251.59 lat (msec): min=120, max=621, avg=354.65, stdev=251.72 clat percentiles (msec): | 1.00th=[ 122], 5.00th=[ 122], 10.00th=[ 122], 20.00th=[ 122], | 30.00th=[ 122], 40.00th=[ 321], 50.00th=[ 321], 60.00th=[ 321], | 70.00th=[ 617], 80.00th=[ 617], 90.00th=[ 617], 95.00th=[ 617], | 99.00th=[ 617], 99.50th=[ 617], 99.90th=[ 617], 99.95th=[ 617], | 99.99th=[ 617] bw ( KiB/s): min=15723, max=15723, per=51.87%, avg=15723.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=33.33%, 750=33.33% cpu : usr=0.19%, sys=15.91%, ctx=297, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=29751: Mon Mar 31 19:11:40 2025 write: IOPS=3, BW=15.8MiB/s (16.6MB/s)(12.0MiB/760msec) clat (msec): min=101, max=508, avg=250.16, stdev=224.85 lat (msec): min=102, max=509, avg=250.53, stdev=224.80 clat percentiles (msec): | 1.00th=[ 103], 5.00th=[ 103], 10.00th=[ 103], 20.00th=[ 103], | 30.00th=[ 103], 40.00th=[ 140], 50.00th=[ 140], 60.00th=[ 140], | 70.00th=[ 510], 80.00th=[ 510], 90.00th=[ 510], 95.00th=[ 510], | 99.00th=[ 510], 99.50th=[ 510], 99.90th=[ 510], 99.95th=[ 510], | 99.99th=[ 510] bw ( KiB/s): min=16351, max=16351, per=53.94%, avg=16351.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 750=33.33% cpu : usr=0.00%, sys=10.94%, ctx=56, majf=0, minf=28 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=15.0MiB/s (16.7MB/s), 4084KiB/s-17.7MiB/s (4182kB/s-18.5MB/s), io=16.0MiB (16.8MB), run=679-1003msec WRITE: bw=29.6MiB/s (31.0MB/s), 8167KiB/s-15.8MiB/s (8364kB/s-16.6MB/s), io=32.0MiB (33.6MB), run=760-1081msec