== sanity test 398b: DIO and buffer IO race ============== 18:46:35 (1741905995) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.16545 s, 43.2 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=22941: Thu Mar 13 18:46:49 2025 read: IOPS=122, BW=489KiB/s (500kB/s)(5992KiB/12264msec) clat (usec): min=54, max=168936, avg=4532.27, stdev=5003.50 lat (usec): min=54, max=168936, avg=4533.61, stdev=5003.50 clat percentiles (usec): | 1.00th=[ 67], 5.00th=[ 1696], 10.00th=[ 2073], 20.00th=[ 2442], | 30.00th=[ 2868], 40.00th=[ 3359], 50.00th=[ 3916], 60.00th=[ 4555], | 70.00th=[ 5276], 80.00th=[ 6063], 90.00th=[ 7439], 95.00th=[ 8848], | 99.00th=[ 14746], 99.50th=[ 16909], 99.90th=[ 24249], 99.95th=[168821], | 99.99th=[168821] bw ( KiB/s): min= 256, max= 672, per=24.53%, avg=480.00, stdev=87.32, samples=24 iops : min= 64, max= 168, avg=120.00, stdev=21.83, samples=24 write: IOPS=128, BW=513KiB/s (526kB/s)(6296KiB/12264msec) clat (usec): min=415, max=38154, avg=3432.72, stdev=3110.51 lat (usec): min=415, max=38155, avg=3433.84, stdev=3110.23 clat percentiles (usec): | 1.00th=[ 502], 5.00th=[ 529], 10.00th=[ 553], 20.00th=[ 627], | 30.00th=[ 848], 40.00th=[ 2057], 50.00th=[ 2802], 60.00th=[ 3458], | 70.00th=[ 4424], 80.00th=[ 5735], 90.00th=[ 7767], 95.00th=[ 9241], | 99.00th=[13304], 99.50th=[14091], 99.90th=[21103], 99.95th=[38011], | 99.99th=[38011] bw ( KiB/s): min= 304, max= 656, per=24.52%, avg=502.33, stdev=77.56, samples=24 iops : min= 76, max= 164, avg=125.58, stdev=19.39, samples=24 lat (usec) : 100=1.86%, 250=0.36%, 500=0.52%, 750=13.54%, 1000=2.80% lat (msec) : 2=5.40%, 4=34.37%, 10=37.63%, 20=3.39%, 50=0.10% lat (msec) : 250=0.03% cpu : usr=0.17%, sys=21.88%, ctx=5631, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22943: Thu Mar 13 18:46:49 2025 read: IOPS=124, BW=497KiB/s (509kB/s)(5940KiB/11955msec) clat (usec): min=55, max=204929, avg=4518.17, stdev=5777.88 lat (usec): min=56, max=204930, avg=4518.79, stdev=5777.90 clat percentiles (usec): | 1.00th=[ 66], 5.00th=[ 149], 10.00th=[ 2114], 20.00th=[ 2540], | 30.00th=[ 2966], 40.00th=[ 3392], 50.00th=[ 3949], 60.00th=[ 4555], | 70.00th=[ 5211], 80.00th=[ 6063], 90.00th=[ 7308], 95.00th=[ 8586], | 99.00th=[ 13960], 99.50th=[ 15533], 99.90th=[ 21103], 99.95th=[204473], | 99.99th=[204473] bw ( KiB/s): min= 216, max= 624, per=24.98%, avg=488.91, stdev=86.68, samples=23 iops : min= 54, max= 156, avg=122.13, stdev=21.75, samples=23 write: IOPS=132, BW=531KiB/s (544kB/s)(6348KiB/11955msec) clat (usec): min=422, max=22840, avg=3254.69, stdev=3015.93 lat (usec): min=422, max=22841, avg=3257.56, stdev=3017.54 clat percentiles (usec): | 1.00th=[ 498], 5.00th=[ 523], 10.00th=[ 545], 20.00th=[ 611], | 30.00th=[ 775], 40.00th=[ 1647], 50.00th=[ 2671], 60.00th=[ 3261], | 70.00th=[ 4113], 80.00th=[ 5407], 90.00th=[ 7242], 95.00th=[ 9110], | 99.00th=[13042], 99.50th=[15270], 99.90th=[22414], 99.95th=[22938], | 99.99th=[22938] bw ( KiB/s): min= 264, max= 648, per=25.25%, avg=517.30, stdev=95.72, samples=23 iops : min= 66, max= 162, avg=129.26, stdev=23.89, samples=23 lat (usec) : 100=2.12%, 250=0.33%, 500=0.81%, 750=14.36%, 1000=3.32% lat (msec) : 2=4.49%, 4=34.80%, 10=36.43%, 20=3.16%, 50=0.16% lat (msec) : 250=0.03% cpu : usr=0.20%, sys=21.93%, ctx=5518, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22945: Thu Mar 13 18:46:49 2025 read: IOPS=126, BW=505KiB/s (517kB/s)(6028KiB/11934msec) clat (usec): min=53, max=20633, avg=4282.56, stdev=2463.59 lat (usec): min=53, max=20634, avg=4283.15, stdev=2463.60 clat percentiles (usec): | 1.00th=[ 68], 5.00th=[ 1729], 10.00th=[ 2114], 20.00th=[ 2474], | 30.00th=[ 2835], 40.00th=[ 3228], 50.00th=[ 3785], 60.00th=[ 4424], | 70.00th=[ 5080], 80.00th=[ 5866], 90.00th=[ 7242], 95.00th=[ 8455], | 99.00th=[13042], 99.50th=[14615], 99.90th=[20055], 99.95th=[20579], | 99.99th=[20579] bw ( KiB/s): min= 240, max= 664, per=25.54%, avg=499.83, stdev=82.96, samples=23 iops : min= 60, max= 166, avg=124.96, stdev=20.74, samples=23 write: IOPS=131, BW=525KiB/s (537kB/s)(6260KiB/11934msec) clat (usec): min=477, max=217350, avg=3460.40, stdev=6171.66 lat (usec): min=478, max=217352, avg=3461.43, stdev=6171.62 clat percentiles (usec): | 1.00th=[ 498], 5.00th=[ 529], 10.00th=[ 553], 20.00th=[ 611], | 30.00th=[ 783], 40.00th=[ 1795], 50.00th=[ 2606], 60.00th=[ 3326], | 70.00th=[ 4359], 80.00th=[ 5669], 90.00th=[ 7701], 95.00th=[ 9110], | 99.00th=[ 12780], 99.50th=[ 14091], 99.90th=[ 21627], 99.95th=[217056], | 99.99th=[217056] bw ( KiB/s): min= 320, max= 704, per=25.17%, avg=515.83, stdev=88.19, samples=23 iops : min= 80, max= 176, avg=128.96, stdev=22.05, samples=23 lat (usec) : 100=1.73%, 250=0.46%, 500=0.55%, 750=14.19%, 1000=2.80% lat (msec) : 2=5.27%, 4=35.35%, 10=36.69%, 20=2.83%, 50=0.10% lat (msec) : 250=0.03% cpu : usr=0.15%, sys=22.45%, ctx=5543, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22947: Thu Mar 13 18:46:49 2025 read: IOPS=126, BW=506KiB/s (519kB/s)(6052KiB/11950msec) clat (usec): min=57, max=20037, avg=4283.21, stdev=2360.62 lat (usec): min=58, max=20038, avg=4284.15, stdev=2360.65 clat percentiles (usec): | 1.00th=[ 68], 5.00th=[ 1811], 10.00th=[ 2180], 20.00th=[ 2540], | 30.00th=[ 2933], 40.00th=[ 3294], 50.00th=[ 3818], 60.00th=[ 4424], | 70.00th=[ 5080], 80.00th=[ 5800], 90.00th=[ 6915], 95.00th=[ 8586], | 99.00th=[13173], 99.50th=[14746], 99.90th=[17433], 99.95th=[20055], | 99.99th=[20055] bw ( KiB/s): min= 247, max= 600, per=25.53%, avg=499.70, stdev=79.15, samples=23 iops : min= 61, max= 150, avg=124.87, stdev=19.88, samples=23 write: IOPS=130, BW=522KiB/s (534kB/s)(6236KiB/11950msec) clat (usec): min=416, max=213185, avg=3466.03, stdev=6103.81 lat (usec): min=417, max=213186, avg=3466.86, stdev=6103.81 clat percentiles (usec): | 1.00th=[ 490], 5.00th=[ 529], 10.00th=[ 553], 20.00th=[ 619], | 30.00th=[ 807], 40.00th=[ 1680], 50.00th=[ 2671], 60.00th=[ 3359], | 70.00th=[ 4228], 80.00th=[ 5669], 90.00th=[ 7635], 95.00th=[ 9110], | 99.00th=[ 11994], 99.50th=[ 13960], 99.90th=[ 25035], 99.95th=[212861], | 99.99th=[212861] bw ( KiB/s): min= 311, max= 792, per=25.24%, avg=517.09, stdev=92.32, samples=23 iops : min= 77, max= 198, avg=129.22, stdev=23.14, samples=23 lat (usec) : 100=1.76%, 250=0.36%, 500=0.68%, 750=13.44%, 1000=3.55% lat (msec) : 2=4.72%, 4=35.61%, 10=36.98%, 20=2.80%, 50=0.07% lat (msec) : 250=0.03% cpu : usr=0.21%, sys=22.07%, ctx=5551, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1958KiB/s (2005kB/s), 489KiB/s-506KiB/s (500kB/s-519kB/s), io=23.4MiB (24.6MB), run=11934-12264msec WRITE: bw=2050KiB/s (2099kB/s), 513KiB/s-531KiB/s (526kB/s-544kB/s), io=24.6MiB (25.7MB), run=11934-12264msec rand-rw: (groupid=0, jobs=1): err= 0: pid=22940: Thu Mar 13 18:46:53 2025 read: IOPS=96, BW=386KiB/s (395kB/s)(5992KiB/15521msec) clat (usec): min=1511, max=170689, avg=4210.41, stdev=5371.21 lat (usec): min=1512, max=170691, avg=4211.02, stdev=5371.22 clat percentiles (usec): | 1.00th=[ 1614], 5.00th=[ 1745], 10.00th=[ 1827], 20.00th=[ 1958], | 30.00th=[ 2114], 40.00th=[ 2376], 50.00th=[ 2802], 60.00th=[ 3326], | 70.00th=[ 4113], 80.00th=[ 5800], 90.00th=[ 8717], 95.00th=[ 11207], | 99.00th=[ 15926], 99.50th=[ 18220], 99.90th=[ 24773], 99.95th=[170918], | 99.99th=[170918] bw ( KiB/s): min= 176, max= 824, per=25.04%, avg=385.65, stdev=186.42, samples=31 iops : min= 44, max= 206, avg=96.29, stdev=46.68, samples=31 write: IOPS=101, BW=406KiB/s (415kB/s)(6296KiB/15521msec) clat (usec): min=2042, max=41238, avg=5814.19, stdev=3681.25 lat (usec): min=2043, max=41238, avg=5814.89, stdev=3681.25 clat percentiles (usec): | 1.00th=[ 2376], 5.00th=[ 2638], 10.00th=[ 2868], 20.00th=[ 3130], | 30.00th=[ 3359], 40.00th=[ 3752], 50.00th=[ 4293], 60.00th=[ 5342], | 70.00th=[ 6587], 80.00th=[ 8291], 90.00th=[10814], 95.00th=[12649], | 99.00th=[18482], 99.50th=[21890], 99.90th=[28181], 99.95th=[41157], | 99.99th=[41157] bw ( KiB/s): min= 184, max= 888, per=25.12%, avg=405.00, stdev=192.03, samples=31 iops : min= 46, max= 222, avg=101.13, stdev=48.07, samples=31 lat (msec) : 2=11.88%, 4=44.76%, 10=33.24%, 20=9.64%, 50=0.46% lat (msec) : 250=0.03% cpu : usr=0.12%, sys=11.11%, ctx=3618, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22942: Thu Mar 13 18:46:53 2025 read: IOPS=95, BW=382KiB/s (391kB/s)(5940KiB/15567msec) clat (usec): min=1376, max=211051, avg=4168.39, stdev=6214.95 lat (usec): min=1376, max=211053, avg=4169.06, stdev=6215.06 clat percentiles (usec): | 1.00th=[ 1614], 5.00th=[ 1745], 10.00th=[ 1827], 20.00th=[ 1975], | 30.00th=[ 2114], 40.00th=[ 2376], 50.00th=[ 2704], 60.00th=[ 3195], | 70.00th=[ 4015], 80.00th=[ 5473], 90.00th=[ 8848], 95.00th=[ 11076], | 99.00th=[ 14746], 99.50th=[ 16581], 99.90th=[ 28443], 99.95th=[210764], | 99.99th=[210764] bw ( KiB/s): min= 120, max= 800, per=24.73%, avg=380.77, stdev=196.39, samples=31 iops : min= 30, max= 200, avg=95.10, stdev=49.13, samples=31 write: IOPS=101, BW=408KiB/s (418kB/s)(6348KiB/15567msec) clat (usec): min=2164, max=30944, avg=5870.28, stdev=3664.89 lat (usec): min=2164, max=30944, avg=5870.98, stdev=3664.90 clat percentiles (usec): | 1.00th=[ 2409], 5.00th=[ 2671], 10.00th=[ 2835], 20.00th=[ 3097], | 30.00th=[ 3392], 40.00th=[ 3785], 50.00th=[ 4359], 60.00th=[ 5342], | 70.00th=[ 6718], 80.00th=[ 8455], 90.00th=[10945], 95.00th=[13042], | 99.00th=[18744], 99.50th=[21365], 99.90th=[28181], 99.95th=[31065], | 99.99th=[31065] bw ( KiB/s): min= 200, max= 808, per=25.21%, avg=406.32, stdev=183.50, samples=31 iops : min= 50, max= 202, avg=101.48, stdev=45.94, samples=31 lat (msec) : 2=10.42%, 4=46.03%, 10=33.11%, 20=9.99%, 50=0.42% lat (msec) : 250=0.03% cpu : usr=0.19%, sys=10.93%, ctx=3564, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22944: Thu Mar 13 18:46:53 2025 read: IOPS=96, BW=388KiB/s (397kB/s)(6028KiB/15545msec) clat (usec): min=1540, max=23899, avg=4127.61, stdev=3201.44 lat (usec): min=1541, max=23900, avg=4128.22, stdev=3201.44 clat percentiles (usec): | 1.00th=[ 1631], 5.00th=[ 1745], 10.00th=[ 1827], 20.00th=[ 1958], | 30.00th=[ 2114], 40.00th=[ 2376], 50.00th=[ 2802], 60.00th=[ 3359], | 70.00th=[ 4146], 80.00th=[ 5932], 90.00th=[ 8979], 95.00th=[10814], | 99.00th=[16450], 99.50th=[19006], 99.90th=[22152], 99.95th=[23987], | 99.99th=[23987] bw ( KiB/s): min= 144, max= 848, per=25.15%, avg=387.29, stdev=185.47, samples=31 iops : min= 36, max= 212, avg=96.77, stdev=46.41, samples=31 write: IOPS=100, BW=403KiB/s (412kB/s)(6260KiB/15545msec) clat (msec): min=2, max=214, avg= 5.92, stdev= 6.45 lat (msec): min=2, max=214, avg= 5.92, stdev= 6.45 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 3], 20.00th=[ 4], | 30.00th=[ 4], 40.00th=[ 4], 50.00th=[ 5], 60.00th=[ 6], | 70.00th=[ 7], 80.00th=[ 9], 90.00th=[ 11], 95.00th=[ 13], | 99.00th=[ 19], 99.50th=[ 27], 99.90th=[ 31], 99.95th=[ 215], | 99.99th=[ 215] bw ( KiB/s): min= 176, max= 792, per=24.94%, avg=402.00, stdev=192.70, samples=31 iops : min= 44, max= 198, avg=100.45, stdev=48.20, samples=31 lat (msec) : 2=11.69%, 4=44.86%, 10=33.46%, 20=9.51%, 50=0.46% lat (msec) : 250=0.03% cpu : usr=0.15%, sys=11.15%, ctx=3613, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22946: Thu Mar 13 18:46:53 2025 read: IOPS=97, BW=388KiB/s (398kB/s)(6052KiB/15590msec) clat (usec): min=1487, max=22010, avg=4118.58, stdev=3227.58 lat (usec): min=1487, max=22011, avg=4119.23, stdev=3227.58 clat percentiles (usec): | 1.00th=[ 1614], 5.00th=[ 1729], 10.00th=[ 1827], 20.00th=[ 1958], | 30.00th=[ 2147], 40.00th=[ 2409], 50.00th=[ 2802], 60.00th=[ 3294], | 70.00th=[ 4080], 80.00th=[ 5800], 90.00th=[ 8979], 95.00th=[10945], | 99.00th=[15926], 99.50th=[18744], 99.90th=[21627], 99.95th=[21890], | 99.99th=[21890] bw ( KiB/s): min= 152, max= 824, per=25.08%, avg=386.26, stdev=187.70, samples=31 iops : min= 38, max= 206, avg=96.52, stdev=46.95, samples=31 write: IOPS=100, BW=400KiB/s (410kB/s)(6236KiB/15590msec) clat (msec): min=2, max=213, avg= 5.96, stdev= 6.40 lat (msec): min=2, max=213, avg= 5.96, stdev= 6.40 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 3], 20.00th=[ 4], | 30.00th=[ 4], 40.00th=[ 4], 50.00th=[ 5], 60.00th=[ 6], | 70.00th=[ 7], 80.00th=[ 9], 90.00th=[ 11], 95.00th=[ 13], | 99.00th=[ 20], 99.50th=[ 22], 99.90th=[ 31], 99.95th=[ 213], | 99.99th=[ 213] bw ( KiB/s): min= 176, max= 816, per=24.73%, avg=398.65, stdev=191.21, samples=31 iops : min= 44, max= 204, avg=99.61, stdev=47.84, samples=31 lat (msec) : 2=11.00%, 4=45.70%, 10=33.33%, 20=9.44%, 50=0.49% lat (msec) : 250=0.03% cpu : usr=0.18%, sys=11.12%, ctx=3572, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1540KiB/s (1577kB/s), 382KiB/s-388KiB/s (391kB/s-398kB/s), io=23.4MiB (24.6MB), run=15521-15590msec WRITE: bw=1613KiB/s (1651kB/s), 400KiB/s-408KiB/s (410kB/s-418kB/s), io=24.6MiB (25.7MB), run=15521-15590msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=22971: Thu Mar 13 18:46:57 2025 read: IOPS=115, BW=1840KiB/s (1884kB/s)(5920KiB/3217msec) clat (usec): min=65, max=25742, avg=4499.09, stdev=2968.04 lat (usec): min=66, max=25742, avg=4499.66, stdev=2968.05 clat percentiles (usec): | 1.00th=[ 67], 5.00th=[ 79], 10.00th=[ 92], 20.00th=[ 2606], | 30.00th=[ 2999], 40.00th=[ 3589], 50.00th=[ 4228], 60.00th=[ 4948], | 70.00th=[ 5604], 80.00th=[ 6259], 90.00th=[ 7701], 95.00th=[ 9503], | 99.00th=[14484], 99.50th=[15795], 99.90th=[25822], 99.95th=[25822], | 99.99th=[25822] bw ( KiB/s): min= 1472, max= 2272, per=23.92%, avg=1780.67, stdev=299.94, samples=6 iops : min= 92, max= 142, avg=111.17, stdev=18.87, samples=6 write: IOPS=123, BW=1979KiB/s (2027kB/s)(6368KiB/3217msec) clat (usec): min=658, max=15633, avg=3769.92, stdev=2996.47 lat (usec): min=659, max=15634, avg=3770.79, stdev=2996.53 clat percentiles (usec): | 1.00th=[ 693], 5.00th=[ 758], 10.00th=[ 816], 20.00th=[ 922], | 30.00th=[ 1385], 40.00th=[ 2311], 50.00th=[ 3064], 60.00th=[ 3818], | 70.00th=[ 4752], 80.00th=[ 6325], 90.00th=[ 8225], 95.00th=[10159], | 99.00th=[12387], 99.50th=[13435], 99.90th=[15664], 99.95th=[15664], | 99.99th=[15664] bw ( KiB/s): min= 1600, max= 2272, per=24.91%, avg=1951.17, stdev=269.03, samples=6 iops : min= 100, max= 142, avg=121.83, stdev=16.79, samples=6 lat (usec) : 100=5.08%, 250=0.91%, 750=2.34%, 1000=9.64% lat (msec) : 2=7.03%, 4=30.08%, 10=40.49%, 20=4.30%, 50=0.13% cpu : usr=0.12%, sys=24.28%, ctx=1561, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22973: Thu Mar 13 18:46:57 2025 read: IOPS=113, BW=1822KiB/s (1865kB/s)(5744KiB/3153msec) clat (usec): min=56, max=18932, avg=4380.80, stdev=2688.20 lat (usec): min=57, max=18933, avg=4381.84, stdev=2688.29 clat percentiles (usec): | 1.00th=[ 67], 5.00th=[ 72], 10.00th=[ 113], 20.00th=[ 2573], | 30.00th=[ 3097], 40.00th=[ 3556], 50.00th=[ 4113], 60.00th=[ 4686], | 70.00th=[ 5342], 80.00th=[ 6128], 90.00th=[ 7767], 95.00th=[ 8979], | 99.00th=[12911], 99.50th=[14746], 99.90th=[19006], 99.95th=[19006], | 99.99th=[19006] bw ( KiB/s): min= 1405, max= 2304, per=24.19%, avg=1800.83, stdev=352.02, samples=6 iops : min= 87, max= 144, avg=112.17, stdev=22.27, samples=6 write: IOPS=129, BW=2075KiB/s (2125kB/s)(6544KiB/3153msec) clat (usec): min=639, max=18133, avg=3772.62, stdev=3280.11 lat (usec): min=640, max=18134, avg=3774.77, stdev=3280.20 clat percentiles (usec): | 1.00th=[ 709], 5.00th=[ 775], 10.00th=[ 824], 20.00th=[ 906], | 30.00th=[ 1188], 40.00th=[ 2180], 50.00th=[ 2933], 60.00th=[ 3654], | 70.00th=[ 4686], 80.00th=[ 6128], 90.00th=[ 8029], 95.00th=[10814], | 99.00th=[15139], 99.50th=[15401], 99.90th=[18220], 99.95th=[18220], | 99.99th=[18220] bw ( KiB/s): min= 1728, max= 2459, per=25.98%, avg=2035.00, stdev=249.24, samples=6 iops : min= 108, max= 153, avg=126.83, stdev=15.38, samples=6 lat (usec) : 100=4.30%, 250=1.04%, 750=1.69%, 1000=11.46% lat (msec) : 2=7.42%, 4=31.25%, 10=38.80%, 20=4.04% cpu : usr=0.29%, sys=24.40%, ctx=1550, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22975: Thu Mar 13 18:46:57 2025 read: IOPS=123, BW=1982KiB/s (2030kB/s)(6256KiB/3156msec) clat (usec): min=64, max=17019, avg=4531.26, stdev=2709.23 lat (usec): min=64, max=17019, avg=4531.88, stdev=2709.27 clat percentiles (usec): | 1.00th=[ 68], 5.00th=[ 82], 10.00th=[ 2147], 20.00th=[ 2671], | 30.00th=[ 3097], 40.00th=[ 3523], 50.00th=[ 4146], 60.00th=[ 4686], | 70.00th=[ 5538], 80.00th=[ 6259], 90.00th=[ 8094], 95.00th=[ 9634], | 99.00th=[13960], 99.50th=[14615], 99.90th=[16909], 99.95th=[16909], | 99.99th=[16909] bw ( KiB/s): min= 1472, max= 2331, per=26.04%, avg=1939.00, stdev=340.00, samples=6 iops : min= 92, max= 145, avg=120.83, stdev=21.16, samples=6 write: IOPS=119, BW=1911KiB/s (1957kB/s)(6032KiB/3156msec) clat (usec): min=657, max=16346, avg=3551.37, stdev=3165.86 lat (usec): min=658, max=16346, avg=3553.69, stdev=3166.73 clat percentiles (usec): | 1.00th=[ 701], 5.00th=[ 775], 10.00th=[ 816], 20.00th=[ 898], | 30.00th=[ 1057], 40.00th=[ 1680], 50.00th=[ 2638], 60.00th=[ 3425], | 70.00th=[ 4424], 80.00th=[ 5800], 90.00th=[ 7963], 95.00th=[10552], | 99.00th=[14615], 99.50th=[14746], 99.90th=[16319], 99.95th=[16319], | 99.99th=[16319] bw ( KiB/s): min= 1692, max= 2208, per=23.46%, avg=1838.00, stdev=189.93, samples=6 iops : min= 105, max= 138, avg=114.50, stdev=12.14, samples=6 lat (usec) : 100=3.78%, 250=0.65%, 750=1.69%, 1000=11.85% lat (msec) : 2=7.81%, 4=30.73%, 10=38.54%, 20=4.95% cpu : usr=0.35%, sys=24.85%, ctx=1591, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22977: Thu Mar 13 18:46:57 2025 read: IOPS=125, BW=2003KiB/s (2051kB/s)(6032KiB/3012msec) clat (usec): min=66, max=17161, avg=4598.81, stdev=2882.59 lat (usec): min=66, max=17162, avg=4599.38, stdev=2882.59 clat percentiles (usec): | 1.00th=[ 68], 5.00th=[ 81], 10.00th=[ 225], 20.00th=[ 2704], | 30.00th=[ 3130], 40.00th=[ 3752], 50.00th=[ 4228], 60.00th=[ 4883], | 70.00th=[ 5538], 80.00th=[ 6456], 90.00th=[ 7767], 95.00th=[ 9503], | 99.00th=[16057], 99.50th=[16909], 99.90th=[17171], 99.95th=[17171], | 99.99th=[17171] bw ( KiB/s): min= 1724, max= 2208, per=26.57%, avg=1978.00, stdev=253.11, samples=6 iops : min= 107, max= 138, avg=123.50, stdev=15.97, samples=6 write: IOPS=129, BW=2077KiB/s (2127kB/s)(6256KiB/3012msec) clat (usec): min=619, max=17643, avg=3232.14, stdev=2894.94 lat (usec): min=620, max=17644, avg=3232.99, stdev=2894.96 clat percentiles (usec): | 1.00th=[ 693], 5.00th=[ 766], 10.00th=[ 799], 20.00th=[ 857], | 30.00th=[ 947], 40.00th=[ 1188], 50.00th=[ 2409], 60.00th=[ 3228], | 70.00th=[ 3982], 80.00th=[ 5473], 90.00th=[ 7504], 95.00th=[ 9241], | 99.00th=[11469], 99.50th=[14222], 99.90th=[17695], 99.95th=[17695], | 99.99th=[17695] bw ( KiB/s): min= 1824, max= 2496, per=26.48%, avg=2073.83, stdev=256.62, samples=6 iops : min= 114, max= 156, avg=129.50, stdev=15.97, samples=6 lat (usec) : 100=4.17%, 250=0.78%, 750=2.60%, 1000=14.58% lat (msec) : 2=7.42%, 4=28.12%, 10=38.41%, 20=3.91% cpu : usr=0.13%, sys=25.31%, ctx=1432, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=7445KiB/s (7624kB/s), 1822KiB/s-2003KiB/s (1865kB/s-2051kB/s), io=23.4MiB (24.5MB), run=3012-3217msec WRITE: bw=7833KiB/s (8021kB/s), 1911KiB/s-2077KiB/s (1957kB/s-2127kB/s), io=24.6MiB (25.8MB), run=3012-3217msec rand-rw: (groupid=0, jobs=1): err= 0: pid=22972: Thu Mar 13 18:46:58 2025 read: IOPS=90, BW=1442KiB/s (1477kB/s)(5920KiB/4105msec) clat (usec): min=1721, max=18105, avg=4383.57, stdev=3170.91 lat (usec): min=1722, max=18106, avg=4384.16, stdev=3170.92 clat percentiles (usec): | 1.00th=[ 1795], 5.00th=[ 1926], 10.00th=[ 1991], 20.00th=[ 2180], | 30.00th=[ 2343], 40.00th=[ 2573], 50.00th=[ 2999], 60.00th=[ 3589], | 70.00th=[ 4883], 80.00th=[ 5932], 90.00th=[ 9241], 95.00th=[11731], | 99.00th=[15664], 99.50th=[16188], 99.90th=[18220], 99.95th=[18220], | 99.99th=[18220] bw ( KiB/s): min= 928, max= 2560, per=24.10%, avg=1402.62, stdev=589.93, samples=8 iops : min= 58, max= 160, avg=87.37, stdev=36.85, samples=8 write: IOPS=96, BW=1551KiB/s (1589kB/s)(6368KiB/4105msec) clat (usec): min=2543, max=28574, avg=6182.10, stdev=3991.26 lat (usec): min=2544, max=28575, avg=6183.05, stdev=3991.26 clat percentiles (usec): | 1.00th=[ 2671], 5.00th=[ 2966], 10.00th=[ 3130], 20.00th=[ 3359], | 30.00th=[ 3589], 40.00th=[ 3851], 50.00th=[ 4424], 60.00th=[ 5604], | 70.00th=[ 6915], 80.00th=[ 8717], 90.00th=[11994], 95.00th=[13960], | 99.00th=[23200], 99.50th=[27919], 99.90th=[28705], 99.95th=[28705], | 99.99th=[28705] bw ( KiB/s): min= 960, max= 2944, per=25.00%, avg=1530.62, stdev=674.87, samples=8 iops : min= 60, max= 184, avg=95.38, stdev=42.22, samples=8 lat (msec) : 2=5.08%, 4=48.18%, 10=34.51%, 20=11.46%, 50=0.78% cpu : usr=0.27%, sys=11.70%, ctx=927, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22974: Thu Mar 13 18:46:58 2025 read: IOPS=87, BW=1400KiB/s (1434kB/s)(5744KiB/4102msec) clat (usec): min=1619, max=24850, avg=4365.37, stdev=3534.25 lat (usec): min=1620, max=24851, avg=4365.96, stdev=3534.25 clat percentiles (usec): | 1.00th=[ 1795], 5.00th=[ 1991], 10.00th=[ 2073], 20.00th=[ 2180], | 30.00th=[ 2311], 40.00th=[ 2540], 50.00th=[ 2868], 60.00th=[ 3294], | 70.00th=[ 4146], 80.00th=[ 5997], 90.00th=[ 8848], 95.00th=[12518], | 99.00th=[18220], 99.50th=[22152], 99.90th=[24773], 99.95th=[24773], | 99.99th=[24773] bw ( KiB/s): min= 800, max= 2464, per=23.72%, avg=1380.00, stdev=576.11, samples=8 iops : min= 50, max= 154, avg=86.25, stdev=36.01, samples=8 write: IOPS=99, BW=1595KiB/s (1634kB/s)(6544KiB/4102msec) clat (usec): min=2455, max=22154, avg=6057.22, stdev=3460.92 lat (usec): min=2456, max=22155, avg=6058.45, stdev=3460.65 clat percentiles (usec): | 1.00th=[ 2573], 5.00th=[ 2900], 10.00th=[ 3097], 20.00th=[ 3326], | 30.00th=[ 3556], 40.00th=[ 3982], 50.00th=[ 4555], 60.00th=[ 5538], | 70.00th=[ 6915], 80.00th=[ 9241], 90.00th=[11469], 95.00th=[13173], | 99.00th=[15795], 99.50th=[17433], 99.90th=[22152], 99.95th=[22152], | 99.99th=[22152] bw ( KiB/s): min= 1024, max= 2976, per=25.42%, avg=1556.00, stdev=654.00, samples=8 iops : min= 64, max= 186, avg=97.25, stdev=40.88, samples=8 lat (msec) : 2=2.34%, 4=50.78%, 10=34.51%, 20=11.85%, 50=0.52% cpu : usr=0.12%, sys=11.78%, ctx=902, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22976: Thu Mar 13 18:46:58 2025 read: IOPS=96, BW=1548KiB/s (1585kB/s)(6256KiB/4041msec) clat (usec): min=1739, max=18760, avg=4346.08, stdev=3086.12 lat (usec): min=1739, max=18760, avg=4346.66, stdev=3086.12 clat percentiles (usec): | 1.00th=[ 1827], 5.00th=[ 1975], 10.00th=[ 2040], 20.00th=[ 2180], | 30.00th=[ 2376], 40.00th=[ 2606], 50.00th=[ 2999], 60.00th=[ 3687], | 70.00th=[ 4621], 80.00th=[ 6128], 90.00th=[ 8979], 95.00th=[11207], | 99.00th=[16057], 99.50th=[17695], 99.90th=[18744], 99.95th=[18744], | 99.99th=[18744] bw ( KiB/s): min= 800, max= 3136, per=26.58%, avg=1546.62, stdev=741.25, samples=8 iops : min= 50, max= 196, avg=96.38, stdev=46.40, samples=8 write: IOPS=93, BW=1493KiB/s (1529kB/s)(6032KiB/4041msec) clat (usec): min=2462, max=24408, avg=6027.47, stdev=3839.64 lat (usec): min=2463, max=24409, avg=6028.39, stdev=3839.62 clat percentiles (usec): | 1.00th=[ 2671], 5.00th=[ 2900], 10.00th=[ 3097], 20.00th=[ 3326], | 30.00th=[ 3589], 40.00th=[ 3884], 50.00th=[ 4228], 60.00th=[ 5145], | 70.00th=[ 6783], 80.00th=[ 8455], 90.00th=[11731], 95.00th=[14746], | 99.00th=[20317], 99.50th=[20841], 99.90th=[24511], 99.95th=[24511], | 99.99th=[24511] bw ( KiB/s): min= 928, max= 2560, per=23.96%, avg=1466.75, stdev=584.04, samples=8 iops : min= 58, max= 160, avg=91.38, stdev=36.54, samples=8 lat (msec) : 2=3.39%, 4=51.04%, 10=34.90%, 20=10.16%, 50=0.52% cpu : usr=0.17%, sys=11.63%, ctx=926, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22978: Thu Mar 13 18:46:58 2025 read: IOPS=91, BW=1466KiB/s (1501kB/s)(6032KiB/4116msec) clat (usec): min=1635, max=18709, avg=4267.51, stdev=3275.88 lat (usec): min=1636, max=18710, avg=4268.10, stdev=3275.87 clat percentiles (usec): | 1.00th=[ 1745], 5.00th=[ 1876], 10.00th=[ 1958], 20.00th=[ 2114], | 30.00th=[ 2278], 40.00th=[ 2507], 50.00th=[ 2802], 60.00th=[ 3261], | 70.00th=[ 4228], 80.00th=[ 5800], 90.00th=[10290], 95.00th=[11338], | 99.00th=[15270], 99.50th=[18220], 99.90th=[18744], 99.95th=[18744], | 99.99th=[18744] bw ( KiB/s): min= 894, max= 2976, per=24.54%, avg=1427.75, stdev=686.11, samples=8 iops : min= 55, max= 186, avg=89.12, stdev=42.98, samples=8 write: IOPS=94, BW=1520KiB/s (1556kB/s)(6256KiB/4116msec) clat (usec): min=2503, max=26825, avg=6222.00, stdev=4174.95 lat (usec): min=2504, max=26826, avg=6222.93, stdev=4174.95 clat percentiles (usec): | 1.00th=[ 2540], 5.00th=[ 2835], 10.00th=[ 3130], 20.00th=[ 3359], | 30.00th=[ 3589], 40.00th=[ 3916], 50.00th=[ 4359], 60.00th=[ 5145], | 70.00th=[ 6587], 80.00th=[ 8979], 90.00th=[11994], 95.00th=[15008], | 99.00th=[22152], 99.50th=[25560], 99.90th=[26870], 99.95th=[26870], | 99.99th=[26870] bw ( KiB/s): min= 896, max= 2784, per=24.36%, avg=1491.62, stdev=680.36, samples=8 iops : min= 56, max= 174, avg=93.13, stdev=42.59, samples=8 lat (msec) : 2=5.60%, 4=49.61%, 10=31.12%, 20=12.89%, 50=0.78% cpu : usr=0.22%, sys=11.40%, ctx=902, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=5819KiB/s (5959kB/s), 1400KiB/s-1548KiB/s (1434kB/s-1585kB/s), io=23.4MiB (24.5MB), run=4041-4116msec WRITE: bw=6122KiB/s (6269kB/s), 1493KiB/s-1595KiB/s (1529kB/s-1634kB/s), io=24.6MiB (25.8MB), run=4041-4116msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=22986: Thu Mar 13 18:46:59 2025 read: IOPS=12, BW=12.7MiB/s (13.3MB/s)(7168KiB/551msec) clat (usec): min=387, max=61311, avg=29457.39, stdev=17890.32 lat (usec): min=387, max=61312, avg=29458.48, stdev=17890.50 clat percentiles (usec): | 1.00th=[ 388], 5.00th=[ 388], 10.00th=[ 388], 20.00th=[23725], | 30.00th=[27395], 40.00th=[27395], 50.00th=[28705], 60.00th=[31065], | 70.00th=[31065], 80.00th=[33817], 90.00th=[61080], 95.00th=[61080], | 99.00th=[61080], 99.50th=[61080], 99.90th=[61080], 99.95th=[61080], | 99.99th=[61080] write: IOPS=9, BW=9292KiB/s (9515kB/s)(5120KiB/551msec) clat (usec): min=40612, max=89982, avg=60623.98, stdev=18244.48 lat (usec): min=40637, max=90041, avg=60663.92, stdev=18255.92 clat percentiles (usec): | 1.00th=[40633], 5.00th=[40633], 10.00th=[40633], 20.00th=[40633], | 30.00th=[53216], 40.00th=[53216], 50.00th=[57410], 60.00th=[57410], | 70.00th=[62129], 80.00th=[62129], 90.00th=[89654], 95.00th=[89654], | 99.00th=[89654], 99.50th=[89654], 99.90th=[89654], 99.95th=[89654], | 99.99th=[89654] lat (usec) : 500=8.33% lat (msec) : 50=50.00%, 100=41.67% cpu : usr=0.00%, sys=37.64%, ctx=910, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22987: Thu Mar 13 18:46:59 2025 read: IOPS=3, BW=3309KiB/s (3388kB/s)(2048KiB/619msec) clat (usec): min=22736, max=23783, avg=23259.93, stdev=740.28 lat (usec): min=22737, max=23784, avg=23260.84, stdev=739.92 clat percentiles (usec): | 1.00th=[22676], 5.00th=[22676], 10.00th=[22676], 20.00th=[22676], | 30.00th=[22676], 40.00th=[22676], 50.00th=[22676], 60.00th=[23725], | 70.00th=[23725], 80.00th=[23725], 90.00th=[23725], 95.00th=[23725], | 99.00th=[23725], 99.50th=[23725], 99.90th=[23725], 99.95th=[23725], | 99.99th=[23725] bw ( KiB/s): min= 4096, max= 4096, per=12.38%, avg=4096.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 write: IOPS=16, BW=16.2MiB/s (16.9MB/s)(10.0MiB/619msec) clat (usec): min=26032, max=84860, avg=49023.06, stdev=19895.84 lat (usec): min=26090, max=84931, avg=49392.04, stdev=20145.57 clat percentiles (usec): | 1.00th=[26084], 5.00th=[26084], 10.00th=[26084], 20.00th=[26608], | 30.00th=[34866], 40.00th=[35914], 50.00th=[42206], 60.00th=[43779], | 70.00th=[65274], 80.00th=[65274], 90.00th=[68682], 95.00th=[84411], | 99.00th=[84411], 99.50th=[84411], 99.90th=[84411], 99.95th=[84411], | 99.99th=[84411] bw ( KiB/s): min=12288, max=12288, per=26.53%, avg=12288.00, stdev= 0.00, samples=1 iops : min= 12, max= 12, avg=12.00, stdev= 0.00, samples=1 lat (msec) : 50=66.67%, 100=33.33% cpu : usr=0.00%, sys=39.48%, ctx=125, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22988: Thu Mar 13 18:46:59 2025 read: IOPS=13, BW=13.6MiB/s (14.2MB/s)(6144KiB/442msec) clat (usec): min=400, max=36480, avg=23771.43, stdev=12628.10 lat (usec): min=401, max=36481, avg=23772.29, stdev=12628.18 clat percentiles (usec): | 1.00th=[ 400], 5.00th=[ 400], 10.00th=[ 400], 20.00th=[23200], | 30.00th=[23200], 40.00th=[23725], 50.00th=[23725], 60.00th=[25822], | 70.00th=[32900], 80.00th=[32900], 90.00th=[36439], 95.00th=[36439], | 99.00th=[36439], 99.50th=[36439], 99.90th=[36439], 99.95th=[36439], | 99.99th=[36439] write: IOPS=13, BW=13.6MiB/s (14.2MB/s)(6144KiB/442msec) clat (usec): min=26764, max=91332, avg=48658.82, stdev=28324.10 lat (usec): min=26847, max=91379, avg=48699.22, stdev=28318.98 clat percentiles (usec): | 1.00th=[26870], 5.00th=[26870], 10.00th=[26870], 20.00th=[28181], | 30.00th=[28181], 40.00th=[30278], 50.00th=[30278], 60.00th=[38011], | 70.00th=[77071], 80.00th=[77071], 90.00th=[91751], 95.00th=[91751], | 99.00th=[91751], 99.50th=[91751], 99.90th=[91751], 99.95th=[91751], | 99.99th=[91751] lat (usec) : 500=8.33% lat (msec) : 50=75.00%, 100=16.67% cpu : usr=0.00%, sys=37.87%, ctx=309, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22989: Thu Mar 13 18:46:59 2025 read: IOPS=9, BW=9624KiB/s (9855kB/s)(5120KiB/532msec) clat (usec): min=347, max=24659, avg=5248.98, stdev=10851.04 lat (usec): min=348, max=24660, avg=5249.61, stdev=10851.21 clat percentiles (usec): | 1.00th=[ 347], 5.00th=[ 347], 10.00th=[ 347], 20.00th=[ 347], | 30.00th=[ 371], 40.00th=[ 371], 50.00th=[ 383], 60.00th=[ 383], | 70.00th=[ 486], 80.00th=[ 486], 90.00th=[24773], 95.00th=[24773], | 99.00th=[24773], 99.50th=[24773], 99.90th=[24773], 99.95th=[24773], | 99.99th=[24773] bw ( KiB/s): min= 8175, max= 8175, per=24.71%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 7, max= 7, avg= 7.00, stdev= 0.00, samples=1 write: IOPS=13, BW=13.2MiB/s (13.8MB/s)(7168KiB/532msec) clat (usec): min=33328, max=96555, avg=60224.93, stdev=19725.60 lat (usec): min=33425, max=96573, avg=60285.87, stdev=19703.09 clat percentiles (usec): | 1.00th=[33424], 5.00th=[33424], 10.00th=[33424], 20.00th=[53740], | 30.00th=[54264], 40.00th=[54264], 50.00th=[54789], 60.00th=[55837], | 70.00th=[55837], 80.00th=[72877], 90.00th=[96994], 95.00th=[96994], | 99.00th=[96994], 99.50th=[96994], 99.90th=[96994], 99.95th=[96994], | 99.99th=[96994] bw ( KiB/s): min=12263, max=12263, per=26.48%, avg=12263.00, stdev= 0.00, samples=1 iops : min= 11, max= 11, avg=11.00, stdev= 0.00, samples=1 lat (usec) : 500=33.33% lat (msec) : 50=16.67%, 100=50.00% cpu : usr=0.00%, sys=28.81%, ctx=258, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=32.3MiB/s (33.9MB/s), 3309KiB/s-13.6MiB/s (3388kB/s-14.2MB/s), io=20.0MiB (20.0MB), run=442-619msec WRITE: bw=45.2MiB/s (47.4MB/s), 9292KiB/s-16.2MiB/s (9515kB/s-16.9MB/s), io=28.0MiB (29.4MB), run=442-619msec rand-rw: (groupid=0, jobs=1): err= 0: pid=22991: Thu Mar 13 18:46:59 2025 read: IOPS=11, BW=11.2MiB/s (11.7MB/s)(7168KiB/625msec) clat (usec): min=12838, max=98482, avg=32046.82, stdev=30815.18 lat (usec): min=12839, max=98483, avg=32047.70, stdev=30815.15 clat percentiles (usec): | 1.00th=[12780], 5.00th=[12780], 10.00th=[12780], 20.00th=[13960], | 30.00th=[14484], 40.00th=[14484], 50.00th=[18220], 60.00th=[26870], | 70.00th=[26870], 80.00th=[39584], 90.00th=[98042], 95.00th=[98042], | 99.00th=[98042], 99.50th=[98042], 99.90th=[98042], 99.95th=[98042], | 99.99th=[98042] bw ( KiB/s): min=10240, max=10240, per=33.25%, avg=10240.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 write: IOPS=8, BW=8192KiB/s (8389kB/s)(5120KiB/625msec) clat (msec): min=17, max=172, avg=62.79, stdev=62.23 lat (msec): min=17, max=172, avg=62.83, stdev=62.22 clat percentiles (msec): | 1.00th=[ 18], 5.00th=[ 18], 10.00th=[ 18], 20.00th=[ 18], | 30.00th=[ 33], 40.00th=[ 33], 50.00th=[ 44], 60.00th=[ 44], | 70.00th=[ 48], 80.00th=[ 48], 90.00th=[ 174], 95.00th=[ 174], | 99.00th=[ 174], 99.50th=[ 174], 99.90th=[ 174], 99.95th=[ 174], | 99.99th=[ 174] bw ( KiB/s): min= 6144, max= 6144, per=14.25%, avg=6144.00, stdev= 0.00, samples=1 iops : min= 6, max= 6, avg= 6.00, stdev= 0.00, samples=1 lat (msec) : 20=41.67%, 50=41.67%, 100=8.33%, 250=8.33% cpu : usr=0.16%, sys=16.19%, ctx=73, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22992: Thu Mar 13 18:46:59 2025 read: IOPS=3, BW=3080KiB/s (3154kB/s)(2048KiB/665msec) clat (usec): min=65702, max=87557, avg=76630.07, stdev=15453.30 lat (usec): min=65703, max=87558, avg=76631.04, stdev=15453.65 clat percentiles (usec): | 1.00th=[65799], 5.00th=[65799], 10.00th=[65799], 20.00th=[65799], | 30.00th=[65799], 40.00th=[65799], 50.00th=[65799], 60.00th=[87557], | 70.00th=[87557], 80.00th=[87557], 90.00th=[87557], 95.00th=[87557], | 99.00th=[87557], 99.50th=[87557], 99.90th=[87557], 99.95th=[87557], | 99.99th=[87557] bw ( KiB/s): min= 2039, max= 2039, per=6.62%, avg=2039.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=15, BW=15.0MiB/s (15.8MB/s)(10.0MiB/665msec) clat (msec): min=15, max=139, avg=46.16, stdev=38.14 lat (msec): min=15, max=139, avg=46.21, stdev=38.14 clat percentiles (msec): | 1.00th=[ 16], 5.00th=[ 16], 10.00th=[ 16], 20.00th=[ 16], | 30.00th=[ 18], 40.00th=[ 22], 50.00th=[ 28], 60.00th=[ 42], | 70.00th=[ 59], 80.00th=[ 59], 90.00th=[ 69], 95.00th=[ 140], | 99.00th=[ 140], 99.50th=[ 140], 99.90th=[ 140], 99.95th=[ 140], | 99.99th=[ 140] bw ( KiB/s): min=10199, max=10199, per=23.66%, avg=10199.00, stdev= 0.00, samples=1 iops : min= 9, max= 9, avg= 9.00, stdev= 0.00, samples=1 lat (msec) : 20=25.00%, 50=25.00%, 100=41.67%, 250=8.33% cpu : usr=0.00%, sys=15.51%, ctx=55, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22993: Thu Mar 13 18:46:59 2025 read: IOPS=9, BW=9676KiB/s (9908kB/s)(6144KiB/635msec) clat (usec): min=12147, max=94592, avg=34403.22, stdev=33239.39 lat (usec): min=12148, max=94593, avg=34404.01, stdev=33239.35 clat percentiles (usec): | 1.00th=[12125], 5.00th=[12125], 10.00th=[12125], 20.00th=[14746], | 30.00th=[14746], 40.00th=[15795], 50.00th=[15795], 60.00th=[16188], | 70.00th=[52691], 80.00th=[52691], 90.00th=[94897], 95.00th=[94897], | 99.00th=[94897], 99.50th=[94897], 99.90th=[94897], 99.95th=[94897], | 99.99th=[94897] bw ( KiB/s): min= 4096, max= 4096, per=13.30%, avg=4096.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 write: IOPS=9, BW=9676KiB/s (9908kB/s)(6144KiB/635msec) clat (usec): min=16862, max=97449, avg=63281.12, stdev=33807.26 lat (usec): min=16882, max=97467, avg=63330.77, stdev=33823.53 clat percentiles (usec): | 1.00th=[16909], 5.00th=[16909], 10.00th=[16909], 20.00th=[28443], | 30.00th=[28443], 40.00th=[68682], 50.00th=[68682], 60.00th=[71828], | 70.00th=[95945], 80.00th=[95945], 90.00th=[96994], 95.00th=[96994], | 99.00th=[96994], 99.50th=[96994], 99.90th=[96994], 99.95th=[96994], | 99.99th=[96994] bw ( KiB/s): min= 8192, max= 8192, per=19.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 lat (msec) : 20=41.67%, 50=8.33%, 100=50.00% cpu : usr=0.00%, sys=13.25%, ctx=220, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22994: Thu Mar 13 18:46:59 2025 read: IOPS=8, BW=8339KiB/s (8539kB/s)(5120KiB/614msec) clat (usec): min=13909, max=39812, avg=26402.38, stdev=12093.75 lat (usec): min=13910, max=39812, avg=26403.34, stdev=12093.62 clat percentiles (usec): | 1.00th=[13960], 5.00th=[13960], 10.00th=[13960], 20.00th=[13960], | 30.00th=[15795], 40.00th=[15795], 50.00th=[24511], 60.00th=[24511], | 70.00th=[38011], 80.00th=[38011], 90.00th=[39584], 95.00th=[39584], | 99.00th=[39584], 99.50th=[39584], 99.90th=[39584], 99.95th=[39584], | 99.99th=[39584] bw ( KiB/s): min= 8192, max= 8192, per=26.60%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=11, BW=11.4MiB/s (11.0MB/s)(7168KiB/614msec) clat (msec): min=22, max=126, avg=57.96, stdev=39.38 lat (msec): min=22, max=126, avg=58.02, stdev=39.39 clat percentiles (msec): | 1.00th=[ 23], 5.00th=[ 23], 10.00th=[ 23], 20.00th=[ 27], | 30.00th=[ 28], 40.00th=[ 28], 50.00th=[ 47], 60.00th=[ 63], | 70.00th=[ 63], 80.00th=[ 95], 90.00th=[ 127], 95.00th=[ 127], | 99.00th=[ 127], 99.50th=[ 127], 99.90th=[ 127], 99.95th=[ 127], | 99.99th=[ 127] bw ( KiB/s): min= 8192, max= 8192, per=19.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 lat (msec) : 20=16.67%, 50=58.33%, 100=16.67%, 250=8.33% cpu : usr=0.00%, sys=13.87%, ctx=111, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=30.1MiB/s (31.5MB/s), 3080KiB/s-11.2MiB/s (3154kB/s-11.7MB/s), io=20.0MiB (20.0MB), run=614-665msec WRITE: bw=42.1MiB/s (44.1MB/s), 8192KiB/s-15.0MiB/s (8389kB/s-15.8MB/s), io=28.0MiB (29.4MB), run=614-665msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=22999: Thu Mar 13 18:47:01 2025 read: IOPS=13, BW=54.5MiB/s (57.2MB/s)(12.0MiB/220msec) clat (usec): min=1418, max=106448, avg=71083.03, stdev=60333.59 lat (usec): min=1419, max=106448, avg=71084.04, stdev=60333.34 clat percentiles (usec): | 1.00th=[ 1418], 5.00th=[ 1418], 10.00th=[ 1418], 20.00th=[ 1418], | 30.00th=[ 1418], 40.00th=[105382], 50.00th=[105382], 60.00th=[105382], | 70.00th=[106431], 80.00th=[106431], 90.00th=[106431], 95.00th=[106431], | 99.00th=[106431], 99.50th=[106431], 99.90th=[106431], 99.95th=[106431], | 99.99th=[106431] lat (msec) : 2=33.33%, 250=66.67% cpu : usr=0.00%, sys=37.44%, ctx=956, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23000: Thu Mar 13 18:47:01 2025 read: IOPS=1, BW=6024KiB/s (6168kB/s)(4096KiB/680msec) clat (nsec): min=118988k, max=118988k, avg=118987965.00, stdev= 0.00 lat (nsec): min=118990k, max=118990k, avg=118989688.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 120], 5.00th=[ 120], 10.00th=[ 120], 20.00th=[ 120], | 30.00th=[ 120], 40.00th=[ 120], 50.00th=[ 120], 60.00th=[ 120], | 70.00th=[ 120], 80.00th=[ 120], 90.00th=[ 120], 95.00th=[ 120], | 99.00th=[ 120], 99.50th=[ 120], 99.90th=[ 120], 99.95th=[ 120], | 99.99th=[ 120] bw ( KiB/s): min= 8192, max= 8192, per=34.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=2, BW=11.8MiB/s (12.3MB/s)(8192KiB/680msec) clat (msec): min=239, max=315, avg=277.22, stdev=54.01 lat (msec): min=239, max=315, avg=277.44, stdev=53.89 clat percentiles (msec): | 1.00th=[ 239], 5.00th=[ 239], 10.00th=[ 239], 20.00th=[ 239], | 30.00th=[ 239], 40.00th=[ 239], 50.00th=[ 239], 60.00th=[ 317], | 70.00th=[ 317], 80.00th=[ 317], 90.00th=[ 317], 95.00th=[ 317], | 99.00th=[ 317], 99.50th=[ 317], 99.90th=[ 317], 99.95th=[ 317], | 99.99th=[ 317] bw ( KiB/s): min= 8192, max= 8192, per=17.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.15%, sys=32.55%, ctx=258, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23001: Thu Mar 13 18:47:01 2025 write: IOPS=5, BW=21.5MiB/s (22.5MB/s)(12.0MiB/559msec) clat (msec): min=88, max=295, avg=182.79, stdev=105.01 lat (msec): min=88, max=296, avg=183.06, stdev=105.01 clat percentiles (msec): | 1.00th=[ 89], 5.00th=[ 89], 10.00th=[ 89], 20.00th=[ 89], | 30.00th=[ 89], 40.00th=[ 165], 50.00th=[ 165], 60.00th=[ 165], | 70.00th=[ 296], 80.00th=[ 296], 90.00th=[ 296], 95.00th=[ 296], | 99.00th=[ 296], 99.50th=[ 296], 99.90th=[ 296], 99.95th=[ 296], | 99.99th=[ 296] bw ( KiB/s): min=16384, max=16384, per=34.00%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 250=33.33%, 500=33.33% cpu : usr=0.00%, sys=40.25%, ctx=53, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23002: Thu Mar 13 18:47:01 2025 write: IOPS=5, BW=22.3MiB/s (23.4MB/s)(12.0MiB/538msec) clat (msec): min=111, max=259, avg=175.87, stdev=76.11 lat (msec): min=111, max=259, avg=176.12, stdev=76.10 clat percentiles (msec): | 1.00th=[ 111], 5.00th=[ 111], 10.00th=[ 111], 20.00th=[ 111], | 30.00th=[ 111], 40.00th=[ 157], 50.00th=[ 157], 60.00th=[ 157], | 70.00th=[ 259], 80.00th=[ 259], 90.00th=[ 259], 95.00th=[ 259], | 99.00th=[ 259], 99.50th=[ 259], 99.90th=[ 259], 99.95th=[ 259], | 99.99th=[ 259] bw ( KiB/s): min=16351, max=16351, per=33.93%, avg=16351.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.00%, sys=44.13%, ctx=164, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=23.5MiB/s (24.7MB/s), 6024KiB/s-54.5MiB/s (6168kB/s-57.2MB/s), io=16.0MiB (16.8MB), run=220-680msec WRITE: bw=47.1MiB/s (49.3MB/s), 11.8MiB/s-22.3MiB/s (12.3MB/s-23.4MB/s), io=32.0MiB (33.6MB), run=538-680msec rand-rw: (groupid=0, jobs=1): err= 0: pid=23004: Thu Mar 13 18:47:01 2025 read: IOPS=4, BW=17.1MiB/s (17.9MB/s)(12.0MiB/703msec) clat (msec): min=41, max=326, avg=221.33, stdev=156.15 lat (msec): min=41, max=326, avg=221.33, stdev=156.15 clat percentiles (msec): | 1.00th=[ 42], 5.00th=[ 42], 10.00th=[ 42], 20.00th=[ 42], | 30.00th=[ 42], 40.00th=[ 296], 50.00th=[ 296], 60.00th=[ 296], | 70.00th=[ 326], 80.00th=[ 326], 90.00th=[ 326], 95.00th=[ 326], | 99.00th=[ 326], 99.50th=[ 326], 99.90th=[ 326], 99.95th=[ 326], | 99.99th=[ 326] bw ( KiB/s): min= 8192, max= 8192, per=35.65%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 50=33.33%, 500=66.67% cpu : usr=0.00%, sys=11.40%, ctx=54, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23005: Thu Mar 13 18:47:01 2025 read: IOPS=1, BW=5745KiB/s (5883kB/s)(4096KiB/713msec) clat (nsec): min=289166k, max=289166k, avg=289166129.00, stdev= 0.00 lat (nsec): min=289167k, max=289167k, avg=289167176.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 288], 5.00th=[ 288], 10.00th=[ 288], 20.00th=[ 288], | 30.00th=[ 288], 40.00th=[ 288], 50.00th=[ 288], 60.00th=[ 288], | 70.00th=[ 288], 80.00th=[ 288], 90.00th=[ 288], 95.00th=[ 288], | 99.00th=[ 288], 99.50th=[ 288], 99.90th=[ 288], 99.95th=[ 288], | 99.99th=[ 288] bw ( KiB/s): min= 8192, max= 8192, per=35.65%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=2, BW=11.2MiB/s (11.8MB/s)(8192KiB/713msec) clat (msec): min=111, max=286, avg=198.97, stdev=124.07 lat (msec): min=111, max=286, avg=199.11, stdev=124.04 clat percentiles (msec): | 1.00th=[ 112], 5.00th=[ 112], 10.00th=[ 112], 20.00th=[ 112], | 30.00th=[ 112], 40.00th=[ 112], 50.00th=[ 112], 60.00th=[ 288], | 70.00th=[ 288], 80.00th=[ 288], 90.00th=[ 288], 95.00th=[ 288], | 99.00th=[ 288], 99.50th=[ 288], 99.90th=[ 288], 99.95th=[ 288], | 99.99th=[ 288] lat (msec) : 250=33.33%, 500=66.67% cpu : usr=0.00%, sys=11.80%, ctx=48, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23006: Thu Mar 13 18:47:01 2025 write: IOPS=4, BW=17.5MiB/s (18.3MB/s)(12.0MiB/687msec) clat (msec): min=171, max=247, avg=222.03, stdev=43.40 lat (msec): min=172, max=247, avg=222.32, stdev=43.44 clat percentiles (msec): | 1.00th=[ 171], 5.00th=[ 171], 10.00th=[ 171], 20.00th=[ 171], | 30.00th=[ 171], 40.00th=[ 247], 50.00th=[ 247], 60.00th=[ 247], | 70.00th=[ 247], 80.00th=[ 247], 90.00th=[ 247], 95.00th=[ 247], | 99.00th=[ 247], 99.50th=[ 247], 99.90th=[ 247], 99.95th=[ 247], | 99.99th=[ 247] bw ( KiB/s): min= 8175, max= 8175, per=17.79%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 250=100.00% cpu : usr=0.00%, sys=14.58%, ctx=44, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23007: Thu Mar 13 18:47:01 2025 write: IOPS=4, BW=18.9MiB/s (19.8MB/s)(12.0MiB/635msec) clat (msec): min=171, max=218, avg=201.83, stdev=25.97 lat (msec): min=172, max=219, avg=202.05, stdev=25.93 clat percentiles (msec): | 1.00th=[ 171], 5.00th=[ 171], 10.00th=[ 171], 20.00th=[ 171], | 30.00th=[ 171], 40.00th=[ 215], 50.00th=[ 215], 60.00th=[ 215], | 70.00th=[ 220], 80.00th=[ 220], 90.00th=[ 220], 95.00th=[ 220], | 99.00th=[ 220], 99.50th=[ 220], 99.90th=[ 220], 99.95th=[ 220], | 99.99th=[ 220] bw ( KiB/s): min=16351, max=16351, per=35.58%, avg=16351.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 lat (msec) : 250=100.00% cpu : usr=0.00%, sys=23.66%, ctx=91, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=22.4MiB/s (23.5MB/s), 5745KiB/s-17.1MiB/s (5883kB/s-17.9MB/s), io=16.0MiB (16.8MB), run=703-713msec WRITE: bw=44.9MiB/s (47.1MB/s), 11.2MiB/s-18.9MiB/s (11.8MB/s-19.8MB/s), io=32.0MiB (33.6MB), run=635-713msec