== sanity test 398b: DIO and buffer IO race ============== 23:22:10 (1708834930) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.10151 s, 45.7 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=25398: Sat Feb 24 23:22:26 2024 read: IOPS=117, BW=470KiB/s (482kB/s)(5992KiB/12739msec) clat (usec): min=40, max=135946, avg=3439.43, stdev=5180.65 lat (usec): min=41, max=135946, avg=3440.23, stdev=5180.61 clat percentiles (usec): | 1.00th=[ 47], 5.00th=[ 60], 10.00th=[ 66], 20.00th=[ 76], | 30.00th=[ 139], 40.00th=[ 2089], 50.00th=[ 2507], 60.00th=[ 3032], | 70.00th=[ 3916], 80.00th=[ 5538], 90.00th=[ 8225], 95.00th=[ 10290], | 99.00th=[ 16057], 99.50th=[ 21365], 99.90th=[ 52167], 99.95th=[135267], | 99.99th=[135267] bw ( KiB/s): min= 320, max= 704, per=24.58%, avg=463.00, stdev=94.59, samples=25 iops : min= 80, max= 176, avg=115.72, stdev=23.66, samples=25 write: IOPS=123, BW=494KiB/s (506kB/s)(6296KiB/12739msec) clat (usec): min=377, max=36586, avg=4763.26, stdev=4566.85 lat (usec): min=378, max=36587, avg=4764.22, stdev=4566.79 clat percentiles (usec): | 1.00th=[ 433], 5.00th=[ 469], 10.00th=[ 502], 20.00th=[ 594], | 30.00th=[ 1106], 40.00th=[ 2573], 50.00th=[ 3589], 60.00th=[ 4817], | 70.00th=[ 6128], 80.00th=[ 8094], 90.00th=[11076], 95.00th=[14353], | 99.00th=[18744], 99.50th=[20841], 99.90th=[27132], 99.95th=[36439], | 99.99th=[36439] bw ( KiB/s): min= 360, max= 704, per=24.73%, avg=487.96, stdev=83.49, samples=25 iops : min= 90, max= 176, avg=121.96, stdev=20.90, samples=25 lat (usec) : 50=1.07%, 100=12.21%, 250=1.82%, 500=5.18%, 750=8.07% lat (usec) : 1000=1.86% lat (msec) : 2=6.80%, 4=24.80%, 10=29.00%, 20=8.46%, 50=0.65% lat (msec) : 100=0.03%, 250=0.03% cpu : usr=0.14%, sys=21.14%, ctx=5376, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25399: Sat Feb 24 23:22:26 2024 read: IOPS=116, BW=468KiB/s (479kB/s)(5940KiB/12705msec) clat (usec): min=41, max=50517, avg=3264.36, stdev=3441.24 lat (usec): min=41, max=50517, avg=3265.18, stdev=3441.26 clat percentiles (usec): | 1.00th=[ 47], 5.00th=[ 61], 10.00th=[ 67], 20.00th=[ 82], | 30.00th=[ 1663], 40.00th=[ 2147], 50.00th=[ 2540], 60.00th=[ 2999], | 70.00th=[ 3851], 80.00th=[ 5407], 90.00th=[ 7898], 95.00th=[ 9634], | 99.00th=[13829], 99.50th=[15795], 99.90th=[22676], 99.95th=[50594], | 99.99th=[50594] bw ( KiB/s): min= 303, max= 680, per=24.85%, avg=468.08, stdev=84.48, samples=25 iops : min= 75, max= 170, avg=116.96, stdev=21.21, samples=25 write: IOPS=124, BW=500KiB/s (512kB/s)(6348KiB/12705msec) clat (usec): min=402, max=134289, avg=4896.00, stdev=5717.12 lat (usec): min=402, max=134290, avg=4896.79, stdev=5717.14 clat percentiles (usec): | 1.00th=[ 433], 5.00th=[ 469], 10.00th=[ 498], 20.00th=[ 603], | 30.00th=[ 865], 40.00th=[ 2376], 50.00th=[ 3851], 60.00th=[ 4817], | 70.00th=[ 6194], 80.00th=[ 8160], 90.00th=[ 11469], 95.00th=[ 14353], | 99.00th=[ 21627], 99.50th=[ 23725], 99.90th=[ 28443], 99.95th=[135267], | 99.99th=[135267] bw ( KiB/s): min= 352, max= 656, per=25.10%, avg=495.28, stdev=78.46, samples=25 iops : min= 88, max= 164, avg=123.76, stdev=19.67, samples=25 lat (usec) : 50=0.91%, 100=10.61%, 250=2.15%, 500=5.60%, 750=9.05% lat (usec) : 1000=2.15% lat (msec) : 2=6.32%, 4=24.19%, 10=30.14%, 20=8.07%, 50=0.75% lat (msec) : 100=0.03%, 250=0.03% cpu : usr=0.20%, sys=21.40%, ctx=5393, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25400: Sat Feb 24 23:22:26 2024 read: IOPS=118, BW=475KiB/s (487kB/s)(6028KiB/12678msec) clat (usec): min=40, max=37983, avg=3299.27, stdev=3419.98 lat (usec): min=40, max=37983, avg=3300.68, stdev=3419.88 clat percentiles (usec): | 1.00th=[ 47], 5.00th=[ 60], 10.00th=[ 66], 20.00th=[ 83], | 30.00th=[ 1762], 40.00th=[ 2180], 50.00th=[ 2606], 60.00th=[ 3097], | 70.00th=[ 3949], 80.00th=[ 5276], 90.00th=[ 7767], 95.00th=[ 9896], | 99.00th=[13698], 99.50th=[14877], 99.90th=[35390], 99.95th=[38011], | 99.99th=[38011] bw ( KiB/s): min= 288, max= 616, per=25.10%, avg=472.92, stdev=91.74, samples=25 iops : min= 72, max= 154, avg=118.20, stdev=22.95, samples=25 write: IOPS=123, BW=494KiB/s (506kB/s)(6260KiB/12678msec) clat (usec): min=401, max=152829, avg=4867.46, stdev=6366.19 lat (usec): min=401, max=152831, avg=4868.27, stdev=6366.23 clat percentiles (usec): | 1.00th=[ 433], 5.00th=[ 474], 10.00th=[ 502], 20.00th=[ 603], | 30.00th=[ 1045], 40.00th=[ 2474], 50.00th=[ 3687], 60.00th=[ 4686], | 70.00th=[ 5997], 80.00th=[ 7701], 90.00th=[ 11207], 95.00th=[ 13435], | 99.00th=[ 20317], 99.50th=[ 26608], 99.90th=[ 69731], 99.95th=[152044], | 99.99th=[152044] bw ( KiB/s): min= 336, max= 640, per=24.88%, avg=490.84, stdev=79.73, samples=25 iops : min= 84, max= 160, avg=122.68, stdev=19.96, samples=25 lat (usec) : 50=1.37%, 100=10.38%, 250=1.99%, 500=5.31%, 750=7.71% lat (usec) : 1000=2.38% lat (msec) : 2=7.32%, 4=25.13%, 10=29.52%, 20=8.20%, 50=0.55% lat (msec) : 100=0.10%, 250=0.03% cpu : usr=0.17%, sys=20.97%, ctx=5412, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25401: Sat Feb 24 23:22:26 2024 read: IOPS=119, BW=477KiB/s (488kB/s)(6052KiB/12694msec) clat (usec): min=38, max=24491, avg=3099.19, stdev=3208.44 lat (usec): min=38, max=24492, avg=3100.10, stdev=3208.55 clat percentiles (usec): | 1.00th=[ 45], 5.00th=[ 57], 10.00th=[ 65], 20.00th=[ 74], | 30.00th=[ 131], 40.00th=[ 2057], 50.00th=[ 2442], 60.00th=[ 2966], | 70.00th=[ 3720], 80.00th=[ 5145], 90.00th=[ 7439], 95.00th=[ 9634], | 99.00th=[13566], 99.50th=[15926], 99.90th=[21627], 99.95th=[24511], | 99.99th=[24511] bw ( KiB/s): min= 240, max= 608, per=25.19%, avg=474.52, stdev=89.54, samples=25 iops : min= 60, max= 152, avg=118.60, stdev=22.40, samples=25 write: IOPS=122, BW=491KiB/s (503kB/s)(6236KiB/12694msec) clat (usec): min=415, max=176404, avg=5074.73, stdev=6658.08 lat (usec): min=416, max=176405, avg=5076.57, stdev=6657.86 clat percentiles (usec): | 1.00th=[ 437], 5.00th=[ 469], 10.00th=[ 502], 20.00th=[ 611], | 30.00th=[ 922], 40.00th=[ 2474], 50.00th=[ 3621], 60.00th=[ 5014], | 70.00th=[ 6456], 80.00th=[ 8291], 90.00th=[ 11600], 95.00th=[ 15008], | 99.00th=[ 21103], 99.50th=[ 25822], 99.90th=[ 47449], 99.95th=[177210], | 99.99th=[177210] bw ( KiB/s): min= 296, max= 672, per=24.80%, avg=489.24, stdev=89.52, samples=25 iops : min= 74, max= 168, avg=122.28, stdev=22.40, samples=25 lat (usec) : 50=1.73%, 100=11.78%, 250=1.95%, 500=5.14%, 750=8.14% lat (usec) : 1000=2.38% lat (msec) : 2=6.64%, 4=24.54%, 10=28.45%, 20=8.50%, 50=0.72% lat (msec) : 250=0.03% cpu : usr=0.19%, sys=21.09%, ctx=5373, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1885KiB/s (1930kB/s), 468KiB/s-477KiB/s (479kB/s-488kB/s), io=23.4MiB (24.6MB), run=12678-12739msec WRITE: bw=1973KiB/s (2021kB/s), 491KiB/s-500KiB/s (503kB/s-512kB/s), io=24.6MiB (25.7MB), run=12678-12739msec rand-rw: (groupid=0, jobs=1): err= 0: pid=25394: Sat Feb 24 23:22:54 2024 read: IOPS=36, BW=148KiB/s (151kB/s)(5992KiB/40587msec) clat (usec): min=1418, max=138332, avg=3833.06, stdev=5102.34 lat (usec): min=1419, max=138333, avg=3833.88, stdev=5102.32 clat percentiles (usec): | 1.00th=[ 1565], 5.00th=[ 1713], 10.00th=[ 1811], 20.00th=[ 1926], | 30.00th=[ 2040], 40.00th=[ 2212], 50.00th=[ 2442], 60.00th=[ 3195], | 70.00th=[ 3621], 80.00th=[ 4228], 90.00th=[ 6915], 95.00th=[ 10683], | 99.00th=[ 18482], 99.50th=[ 25297], 99.90th=[ 64226], 99.95th=[137364], | 99.99th=[137364] bw ( KiB/s): min= 40, max= 256, per=25.05%, avg=147.80, stdev=61.88, samples=80 iops : min= 10, max= 64, avg=36.88, stdev=15.48, samples=80 write: IOPS=38, BW=155KiB/s (159kB/s)(6296KiB/40587msec) clat (usec): min=10542, max=74558, avg=22075.06, stdev=8919.75 lat (usec): min=10543, max=74559, avg=22076.08, stdev=8919.95 clat percentiles (usec): | 1.00th=[12256], 5.00th=[13304], 10.00th=[13829], 20.00th=[15008], | 30.00th=[16450], 40.00th=[18482], 50.00th=[20317], 60.00th=[21890], | 70.00th=[23200], 80.00th=[25297], 90.00th=[35390], 95.00th=[42206], | 99.00th=[53740], 99.50th=[58459], 99.90th=[71828], 99.95th=[74974], | 99.99th=[74974] bw ( KiB/s): min= 56, max= 216, per=25.04%, avg=154.50, stdev=48.66, samples=80 iops : min= 14, max= 54, avg=38.55, stdev=12.17, samples=80 lat (msec) : 2=12.76%, 4=24.97%, 10=8.24%, 20=27.05%, 50=26.04% lat (msec) : 100=0.91%, 250=0.03% cpu : usr=0.10%, sys=4.00%, ctx=3585, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25395: Sat Feb 24 23:22:54 2024 read: IOPS=36, BW=146KiB/s (149kB/s)(5940KiB/40688msec) clat (usec): min=1405, max=125280, avg=3876.14, stdev=4553.13 lat (usec): min=1406, max=125282, avg=3876.90, stdev=4553.13 clat percentiles (usec): | 1.00th=[ 1631], 5.00th=[ 1745], 10.00th=[ 1844], 20.00th=[ 1942], | 30.00th=[ 2073], 40.00th=[ 2245], 50.00th=[ 2606], 60.00th=[ 3326], | 70.00th=[ 3752], 80.00th=[ 4359], 90.00th=[ 7635], 95.00th=[ 11076], | 99.00th=[ 15270], 99.50th=[ 17695], 99.90th=[ 64226], 99.95th=[125305], | 99.99th=[125305] bw ( KiB/s): min= 16, max= 304, per=24.73%, avg=145.91, stdev=65.99, samples=81 iops : min= 4, max= 76, avg=36.40, stdev=16.50, samples=81 write: IOPS=39, BW=156KiB/s (160kB/s)(6348KiB/40688msec) clat (usec): min=11241, max=74682, avg=21948.36, stdev=8708.35 lat (usec): min=11242, max=74683, avg=21949.28, stdev=8708.36 clat percentiles (usec): | 1.00th=[12256], 5.00th=[13042], 10.00th=[13698], 20.00th=[14877], | 30.00th=[16188], 40.00th=[18482], 50.00th=[20317], 60.00th=[21627], | 70.00th=[23200], 80.00th=[25560], 90.00th=[35390], 95.00th=[40633], | 99.00th=[50594], 99.50th=[53216], 99.90th=[71828], 99.95th=[74974], | 99.99th=[74974] bw ( KiB/s): min= 64, max= 208, per=25.23%, avg=155.64, stdev=46.60, samples=81 iops : min= 16, max= 52, avg=38.83, stdev=11.64, samples=81 lat (msec) : 2=11.52%, 4=25.16%, 10=8.66%, 20=27.83%, 50=26.14% lat (msec) : 100=0.65%, 250=0.03% cpu : usr=0.11%, sys=4.01%, ctx=3634, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25396: Sat Feb 24 23:22:54 2024 read: IOPS=37, BW=149KiB/s (152kB/s)(6028KiB/40560msec) clat (usec): min=1425, max=45125, avg=4033.07, stdev=3698.69 lat (usec): min=1425, max=45126, avg=4033.83, stdev=3698.74 clat percentiles (usec): | 1.00th=[ 1614], 5.00th=[ 1778], 10.00th=[ 1860], 20.00th=[ 1975], | 30.00th=[ 2114], 40.00th=[ 2343], 50.00th=[ 2835], 60.00th=[ 3458], | 70.00th=[ 3785], 80.00th=[ 4621], 90.00th=[ 8225], 95.00th=[11207], | 99.00th=[18482], 99.50th=[22676], 99.90th=[37487], 99.95th=[45351], | 99.99th=[45351] bw ( KiB/s): min= 48, max= 328, per=25.19%, avg=148.60, stdev=60.66, samples=80 iops : min= 12, max= 82, avg=37.07, stdev=15.15, samples=80 write: IOPS=38, BW=154KiB/s (158kB/s)(6260KiB/40560msec) clat (msec): min=11, max=148, avg=21.97, stdev= 9.19 lat (msec): min=11, max=148, avg=21.97, stdev= 9.19 clat percentiles (msec): | 1.00th=[ 13], 5.00th=[ 14], 10.00th=[ 14], 20.00th=[ 16], | 30.00th=[ 17], 40.00th=[ 19], 50.00th=[ 21], 60.00th=[ 22], | 70.00th=[ 24], 80.00th=[ 26], 90.00th=[ 35], 95.00th=[ 41], | 99.00th=[ 53], 99.50th=[ 56], 99.90th=[ 72], 99.95th=[ 148], | 99.99th=[ 148] bw ( KiB/s): min= 64, max= 216, per=24.91%, avg=153.70, stdev=47.98, samples=80 iops : min= 16, max= 54, avg=38.35, stdev=11.99, samples=80 lat (msec) : 2=11.07%, 4=25.03%, 10=9.70%, 20=27.64%, 50=25.75% lat (msec) : 100=0.78%, 250=0.03% cpu : usr=0.09%, sys=4.08%, ctx=3676, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25397: Sat Feb 24 23:22:54 2024 read: IOPS=37, BW=150KiB/s (154kB/s)(6052KiB/40300msec) clat (usec): min=1490, max=33779, avg=3697.97, stdev=3005.83 lat (usec): min=1491, max=33780, avg=3698.76, stdev=3005.84 clat percentiles (usec): | 1.00th=[ 1614], 5.00th=[ 1778], 10.00th=[ 1827], 20.00th=[ 1942], | 30.00th=[ 2057], 40.00th=[ 2245], 50.00th=[ 2573], 60.00th=[ 3294], | 70.00th=[ 3687], 80.00th=[ 4293], 90.00th=[ 7046], 95.00th=[10290], | 99.00th=[15664], 99.50th=[18220], 99.90th=[26870], 99.95th=[33817], | 99.99th=[33817] bw ( KiB/s): min= 48, max= 288, per=25.32%, avg=149.40, stdev=61.98, samples=80 iops : min= 12, max= 72, avg=37.27, stdev=15.48, samples=80 write: IOPS=38, BW=155KiB/s (158kB/s)(6236KiB/40300msec) clat (msec): min=11, max=174, avg=22.20, stdev= 9.74 lat (msec): min=11, max=174, avg=22.20, stdev= 9.74 clat percentiles (msec): | 1.00th=[ 13], 5.00th=[ 14], 10.00th=[ 14], 20.00th=[ 16], | 30.00th=[ 17], 40.00th=[ 19], 50.00th=[ 21], 60.00th=[ 22], | 70.00th=[ 24], 80.00th=[ 26], 90.00th=[ 36], 95.00th=[ 42], | 99.00th=[ 54], 99.50th=[ 58], 99.90th=[ 69], 99.95th=[ 176], | 99.99th=[ 176] bw ( KiB/s): min= 56, max= 208, per=24.99%, avg=154.20, stdev=47.81, samples=80 iops : min= 14, max= 52, avg=38.47, stdev=11.95, samples=80 lat (msec) : 2=12.47%, 4=25.00%, 10=9.15%, 20=27.34%, 50=25.03% lat (msec) : 100=0.98%, 250=0.03% cpu : usr=0.06%, sys=4.16%, ctx=3673, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=590KiB/s (604kB/s), 146KiB/s-150KiB/s (149kB/s-154kB/s), io=23.4MiB (24.6MB), run=40300-40688msec WRITE: bw=618KiB/s (633kB/s), 154KiB/s-156KiB/s (158kB/s-160kB/s), io=24.6MiB (25.7MB), run=40300-40688msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=25444: Sat Feb 24 23:22:57 2024 read: IOPS=152, BW=2445KiB/s (2504kB/s)(5920KiB/2421msec) clat (usec): min=45, max=12464, avg=2785.52, stdev=2624.89 lat (usec): min=46, max=12465, avg=2786.15, stdev=2624.92 clat percentiles (usec): | 1.00th=[ 49], 5.00th=[ 60], 10.00th=[ 67], 20.00th=[ 73], | 30.00th=[ 95], 40.00th=[ 2212], 50.00th=[ 2737], 60.00th=[ 3130], | 70.00th=[ 3687], 80.00th=[ 4686], 90.00th=[ 6194], 95.00th=[ 7635], | 99.00th=[11469], 99.50th=[11731], 99.90th=[12518], 99.95th=[12518], | 99.99th=[12518] bw ( KiB/s): min= 2400, max= 2464, per=24.74%, avg=2448.00, stdev=32.00, samples=4 iops : min= 150, max= 154, avg=153.00, stdev= 2.00, samples=4 write: IOPS=164, BW=2630KiB/s (2693kB/s)(6368KiB/2421msec) clat (usec): min=556, max=31856, avg=3469.22, stdev=3469.55 lat (usec): min=557, max=31857, avg=3471.58, stdev=3468.56 clat percentiles (usec): | 1.00th=[ 603], 5.00th=[ 668], 10.00th=[ 717], 20.00th=[ 807], | 30.00th=[ 947], 40.00th=[ 1205], 50.00th=[ 2008], 60.00th=[ 3359], | 70.00th=[ 4555], 80.00th=[ 5866], 90.00th=[ 8225], 95.00th=[ 9896], | 99.00th=[13829], 99.50th=[17695], 99.90th=[31851], 99.95th=[31851], | 99.99th=[31851] bw ( KiB/s): min= 2368, max= 3360, per=26.06%, avg=2712.00, stdev=468.39, samples=4 iops : min= 148, max= 210, avg=169.50, stdev=29.27, samples=4 lat (usec) : 50=0.91%, 100=13.54%, 250=2.47%, 500=0.39%, 750=7.94% lat (usec) : 1000=8.85% lat (msec) : 2=9.51%, 4=25.91%, 10=27.34%, 20=2.99%, 50=0.13% cpu : usr=0.21%, sys=27.98%, ctx=1568, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25445: Sat Feb 24 23:22:57 2024 read: IOPS=148, BW=2376KiB/s (2433kB/s)(5744KiB/2418msec) clat (usec): min=43, max=10683, avg=2722.49, stdev=2565.27 lat (usec): min=43, max=10684, avg=2723.19, stdev=2565.21 clat percentiles (usec): | 1.00th=[ 47], 5.00th=[ 58], 10.00th=[ 64], 20.00th=[ 70], | 30.00th=[ 81], 40.00th=[ 2114], 50.00th=[ 2540], 60.00th=[ 3294], | 70.00th=[ 3884], 80.00th=[ 5080], 90.00th=[ 6259], 95.00th=[ 7635], | 99.00th=[ 9110], 99.50th=[ 9765], 99.90th=[10683], 99.95th=[10683], | 99.99th=[10683] bw ( KiB/s): min= 2112, max= 2720, per=24.50%, avg=2424.00, stdev=260.46, samples=4 iops : min= 132, max= 170, avg=151.50, stdev=16.28, samples=4 write: IOPS=169, BW=2706KiB/s (2771kB/s)(6544KiB/2418msec) clat (usec): min=573, max=34062, avg=3503.97, stdev=3976.63 lat (usec): min=574, max=34063, avg=3505.43, stdev=3976.28 clat percentiles (usec): | 1.00th=[ 611], 5.00th=[ 644], 10.00th=[ 701], 20.00th=[ 775], | 30.00th=[ 881], 40.00th=[ 1090], 50.00th=[ 1778], 60.00th=[ 3032], | 70.00th=[ 4293], 80.00th=[ 5800], 90.00th=[ 8225], 95.00th=[10421], | 99.00th=[16712], 99.50th=[21890], 99.90th=[33817], 99.95th=[33817], | 99.99th=[33817] bw ( KiB/s): min= 2560, max= 3200, per=26.44%, avg=2752.00, stdev=300.19, samples=4 iops : min= 160, max= 200, avg=172.00, stdev=18.76, samples=4 lat (usec) : 50=1.17%, 100=14.19%, 250=2.34%, 500=0.26%, 750=9.11% lat (usec) : 1000=10.03% lat (msec) : 2=8.59%, 4=23.57%, 10=27.47%, 20=2.73%, 50=0.52% cpu : usr=0.21%, sys=27.68%, ctx=1492, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25446: Sat Feb 24 23:22:57 2024 read: IOPS=162, BW=2606KiB/s (2668kB/s)(6256KiB/2401msec) clat (usec): min=46, max=26097, avg=2888.01, stdev=3535.24 lat (usec): min=47, max=26098, avg=2889.16, stdev=3535.15 clat percentiles (usec): | 1.00th=[ 49], 5.00th=[ 58], 10.00th=[ 65], 20.00th=[ 70], | 30.00th=[ 85], 40.00th=[ 2024], 50.00th=[ 2507], 60.00th=[ 2999], | 70.00th=[ 3752], 80.00th=[ 4883], 90.00th=[ 6456], 95.00th=[ 7701], | 99.00th=[21365], 99.50th=[25822], 99.90th=[26084], 99.95th=[26084], | 99.99th=[26084] bw ( KiB/s): min= 2144, max= 2816, per=25.47%, avg=2520.00, stdev=332.94, samples=4 iops : min= 134, max= 176, avg=157.50, stdev=20.81, samples=4 write: IOPS=157, BW=2512KiB/s (2573kB/s)(6032KiB/2401msec) clat (usec): min=592, max=20999, avg=3354.25, stdev=3304.30 lat (usec): min=593, max=21000, avg=3355.12, stdev=3304.31 clat percentiles (usec): | 1.00th=[ 594], 5.00th=[ 635], 10.00th=[ 685], 20.00th=[ 766], | 30.00th=[ 898], 40.00th=[ 1090], 50.00th=[ 1893], 60.00th=[ 3130], | 70.00th=[ 4080], 80.00th=[ 5932], 90.00th=[ 8455], 95.00th=[10159], | 99.00th=[14353], 99.50th=[15926], 99.90th=[21103], 99.95th=[21103], | 99.99th=[21103] bw ( KiB/s): min= 2112, max= 2848, per=23.98%, avg=2496.00, stdev=303.58, samples=4 iops : min= 132, max= 178, avg=156.00, stdev=18.97, samples=4 lat (usec) : 50=1.30%, 100=16.02%, 250=2.34%, 500=0.39%, 750=8.59% lat (usec) : 1000=8.20% lat (msec) : 2=8.59%, 4=25.00%, 10=25.78%, 20=3.12%, 50=0.65% cpu : usr=0.04%, sys=27.62%, ctx=1544, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25447: Sat Feb 24 23:22:57 2024 read: IOPS=161, BW=2578KiB/s (2640kB/s)(6032KiB/2340msec) clat (usec): min=46, max=25552, avg=2956.52, stdev=2989.87 lat (usec): min=46, max=25553, avg=2962.95, stdev=2998.81 clat percentiles (usec): | 1.00th=[ 48], 5.00th=[ 54], 10.00th=[ 65], 20.00th=[ 70], | 30.00th=[ 91], 40.00th=[ 2311], 50.00th=[ 2737], 60.00th=[ 3326], | 70.00th=[ 4228], 80.00th=[ 4948], 90.00th=[ 6587], 95.00th=[ 7898], | 99.00th=[13173], 99.50th=[16319], 99.90th=[25560], 99.95th=[25560], | 99.99th=[25560] bw ( KiB/s): min= 2240, max= 2912, per=25.63%, avg=2536.00, stdev=279.43, samples=4 iops : min= 140, max= 182, avg=158.50, stdev=17.46, samples=4 write: IOPS=167, BW=2674KiB/s (2738kB/s)(6256KiB/2340msec) clat (usec): min=576, max=16004, avg=3107.00, stdev=3144.30 lat (usec): min=577, max=16005, avg=3107.98, stdev=3144.40 clat percentiles (usec): | 1.00th=[ 594], 5.00th=[ 644], 10.00th=[ 676], 20.00th=[ 758], | 30.00th=[ 857], 40.00th=[ 996], 50.00th=[ 1336], 60.00th=[ 2638], | 70.00th=[ 3818], 80.00th=[ 5735], 90.00th=[ 7767], 95.00th=[ 9896], | 99.00th=[13042], 99.50th=[13829], 99.90th=[16057], 99.95th=[16057], | 99.99th=[16057] bw ( KiB/s): min= 2400, max= 2944, per=25.44%, avg=2648.00, stdev=237.14, samples=4 iops : min= 150, max= 184, avg=165.50, stdev=14.82, samples=4 lat (usec) : 50=1.30%, 100=14.06%, 250=1.95%, 500=0.52%, 750=9.77% lat (usec) : 1000=11.07% lat (msec) : 2=7.94%, 4=23.05%, 10=27.08%, 20=3.12%, 50=0.13% cpu : usr=0.00%, sys=28.90%, ctx=1487, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=9893KiB/s (10.1MB/s), 2376KiB/s-2606KiB/s (2433kB/s-2668kB/s), io=23.4MiB (24.5MB), run=2340-2421msec WRITE: bw=10.2MiB/s (10.7MB/s), 2512KiB/s-2706KiB/s (2573kB/s-2771kB/s), io=24.6MiB (25.8MB), run=2340-2421msec rand-rw: (groupid=0, jobs=1): err= 0: pid=25449: Sat Feb 24 23:23:04 2024 read: IOPS=38, BW=610KiB/s (625kB/s)(5920KiB/9700msec) clat (usec): min=1826, max=40118, avg=3519.75, stdev=3373.36 lat (usec): min=1826, max=40119, avg=3520.49, stdev=3373.37 clat percentiles (usec): | 1.00th=[ 1876], 5.00th=[ 1975], 10.00th=[ 2040], 20.00th=[ 2147], | 30.00th=[ 2212], 40.00th=[ 2311], 50.00th=[ 2409], 60.00th=[ 2573], | 70.00th=[ 2868], 80.00th=[ 3458], 90.00th=[ 5997], 95.00th=[10683], | 99.00th=[15795], 99.50th=[22938], 99.90th=[40109], 99.95th=[40109], | 99.99th=[40109] bw ( KiB/s): min= 320, max= 1088, per=24.92%, avg=601.21, stdev=217.50, samples=19 iops : min= 20, max= 68, avg=37.53, stdev=13.64, samples=19 write: IOPS=41, BW=656KiB/s (672kB/s)(6368KiB/9700msec) clat (usec): min=11934, max=55816, avg=21078.59, stdev=6922.67 lat (usec): min=11935, max=55817, avg=21079.94, stdev=6922.81 clat percentiles (usec): | 1.00th=[12256], 5.00th=[13173], 10.00th=[13698], 20.00th=[14877], | 30.00th=[16581], 40.00th=[18482], 50.00th=[20055], 60.00th=[21890], | 70.00th=[23200], 80.00th=[24773], 90.00th=[28705], 95.00th=[34866], | 99.00th=[46400], 99.50th=[55837], 99.90th=[55837], 99.95th=[55837], | 99.99th=[55837] bw ( KiB/s): min= 383, max= 800, per=25.67%, avg=651.74, stdev=153.64, samples=19 iops : min= 23, max= 50, avg=40.68, stdev= 9.70, samples=19 lat (msec) : 2=2.99%, 4=37.11%, 10=5.60%, 20=27.60%, 50=26.43% lat (msec) : 100=0.26% cpu : usr=0.14%, sys=4.94%, ctx=858, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25450: Sat Feb 24 23:23:04 2024 read: IOPS=36, BW=579KiB/s (593kB/s)(5744KiB/9923msec) clat (usec): min=1806, max=22026, avg=3345.21, stdev=2681.65 lat (usec): min=1806, max=22027, avg=3345.92, stdev=2681.67 clat percentiles (usec): | 1.00th=[ 1876], 5.00th=[ 1991], 10.00th=[ 2057], 20.00th=[ 2147], | 30.00th=[ 2212], 40.00th=[ 2311], 50.00th=[ 2409], 60.00th=[ 2573], | 70.00th=[ 2835], 80.00th=[ 3228], 90.00th=[ 5735], 95.00th=[ 9765], | 99.00th=[15926], 99.50th=[18744], 99.90th=[22152], 99.95th=[22152], | 99.99th=[22152] bw ( KiB/s): min= 256, max= 1120, per=24.06%, avg=580.58, stdev=242.79, samples=19 iops : min= 16, max= 70, avg=36.16, stdev=15.10, samples=19 write: IOPS=41, BW=659KiB/s (675kB/s)(6544KiB/9923msec) clat (usec): min=11362, max=49464, avg=21305.24, stdev=6652.69 lat (usec): min=11364, max=49465, avg=21306.83, stdev=6652.43 clat percentiles (usec): | 1.00th=[12649], 5.00th=[13698], 10.00th=[14091], 20.00th=[15139], | 30.00th=[16909], 40.00th=[19268], 50.00th=[20841], 60.00th=[22152], | 70.00th=[23200], 80.00th=[24773], 90.00th=[29230], 95.00th=[36439], | 99.00th=[42730], 99.50th=[44827], 99.90th=[49546], 99.95th=[49546], | 99.99th=[49546] bw ( KiB/s): min= 384, max= 832, per=25.59%, avg=649.63, stdev=142.62, samples=19 iops : min= 24, max= 52, avg=40.47, stdev= 8.90, samples=19 lat (msec) : 2=2.34%, 4=37.37%, 10=4.95%, 20=25.39%, 50=29.95% cpu : usr=0.04%, sys=4.82%, ctx=878, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25451: Sat Feb 24 23:23:04 2024 read: IOPS=41, BW=664KiB/s (680kB/s)(6256KiB/9417msec) clat (usec): min=1828, max=20710, avg=3340.51, stdev=2602.80 lat (usec): min=1829, max=20711, avg=3341.21, stdev=2602.81 clat percentiles (usec): | 1.00th=[ 1909], 5.00th=[ 2008], 10.00th=[ 2057], 20.00th=[ 2147], | 30.00th=[ 2245], 40.00th=[ 2311], 50.00th=[ 2409], 60.00th=[ 2540], | 70.00th=[ 2835], 80.00th=[ 3097], 90.00th=[ 6325], 95.00th=[ 8848], | 99.00th=[15664], 99.50th=[20055], 99.90th=[20841], 99.95th=[20841], | 99.99th=[20841] bw ( KiB/s): min= 320, max= 1088, per=27.90%, avg=673.33, stdev=258.91, samples=18 iops : min= 20, max= 68, avg=41.89, stdev=16.15, samples=18 write: IOPS=40, BW=641KiB/s (656kB/s)(6032KiB/9417msec) clat (usec): min=11630, max=48478, avg=21489.90, stdev=6557.72 lat (usec): min=11631, max=48480, avg=21491.10, stdev=6557.78 clat percentiles (usec): | 1.00th=[12518], 5.00th=[13698], 10.00th=[14484], 20.00th=[15795], | 30.00th=[17433], 40.00th=[19530], 50.00th=[20841], 60.00th=[22152], | 70.00th=[23200], 80.00th=[25035], 90.00th=[29492], 95.00th=[35390], | 99.00th=[43254], 99.50th=[45351], 99.90th=[48497], 99.95th=[48497], | 99.99th=[48497] bw ( KiB/s): min= 415, max= 768, per=24.98%, avg=634.28, stdev=129.40, samples=18 iops : min= 25, max= 48, avg=39.44, stdev= 8.10, samples=18 lat (msec) : 2=2.47%, 4=40.49%, 10=6.25%, 20=22.92%, 50=27.86% cpu : usr=0.10%, sys=5.06%, ctx=853, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25452: Sat Feb 24 23:23:04 2024 read: IOPS=38, BW=616KiB/s (630kB/s)(6032KiB/9797msec) clat (usec): min=1814, max=22127, avg=3532.35, stdev=2996.02 lat (usec): min=1814, max=22127, avg=3533.07, stdev=2996.02 clat percentiles (usec): | 1.00th=[ 1876], 5.00th=[ 2008], 10.00th=[ 2057], 20.00th=[ 2114], | 30.00th=[ 2212], 40.00th=[ 2311], 50.00th=[ 2442], 60.00th=[ 2606], | 70.00th=[ 2769], 80.00th=[ 3326], 90.00th=[ 6915], 95.00th=[10945], | 99.00th=[16712], 99.50th=[20579], 99.90th=[22152], 99.95th=[22152], | 99.99th=[22152] bw ( KiB/s): min= 224, max= 864, per=25.11%, avg=605.95, stdev=203.73, samples=19 iops : min= 14, max= 54, avg=37.68, stdev=12.68, samples=19 write: IOPS=39, BW=639KiB/s (654kB/s)(6256KiB/9797msec) clat (usec): min=12082, max=60742, avg=21632.02, stdev=6504.45 lat (usec): min=12083, max=60744, avg=21633.28, stdev=6504.47 clat percentiles (usec): | 1.00th=[12256], 5.00th=[13698], 10.00th=[14091], 20.00th=[15926], | 30.00th=[17957], 40.00th=[19530], 50.00th=[21103], 60.00th=[22152], | 70.00th=[23200], 80.00th=[25560], 90.00th=[29492], 95.00th=[33424], | 99.00th=[43254], 99.50th=[49546], 99.90th=[60556], 99.95th=[60556], | 99.99th=[60556] bw ( KiB/s): min= 384, max= 800, per=24.93%, avg=632.89, stdev=143.14, samples=19 iops : min= 24, max= 50, avg=39.37, stdev= 8.91, samples=19 lat (msec) : 2=2.08%, 4=38.80%, 10=5.34%, 20=24.22%, 50=29.43% lat (msec) : 100=0.13% cpu : usr=0.05%, sys=4.77%, ctx=869, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2414KiB/s (2472kB/s), 579KiB/s-664KiB/s (593kB/s-680kB/s), io=23.4MiB (24.5MB), run=9417-9923msec WRITE: bw=2540KiB/s (2601kB/s), 639KiB/s-659KiB/s (654kB/s-675kB/s), io=24.6MiB (25.8MB), run=9417-9923msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=25471: Sat Feb 24 23:23:06 2024 read: IOPS=16, BW=16.4MiB/s (17.1MB/s)(7168KiB/428msec) clat (usec): min=440, max=49943, avg=26040.30, stdev=18751.59 lat (usec): min=441, max=49944, avg=26041.21, stdev=18751.45 clat percentiles (usec): | 1.00th=[ 441], 5.00th=[ 441], 10.00th=[ 441], 20.00th=[ 478], | 30.00th=[28705], 40.00th=[28705], 50.00th=[30802], 60.00th=[34866], | 70.00th=[34866], 80.00th=[36963], 90.00th=[50070], 95.00th=[50070], | 99.00th=[50070], 99.50th=[50070], 99.90th=[50070], 99.95th=[50070], | 99.99th=[50070] write: IOPS=11, BW=11.7MiB/s (12.2MB/s)(5120KiB/428msec) clat (usec): min=26378, max=70593, avg=48079.10, stdev=17211.18 lat (usec): min=26388, max=70618, avg=48115.68, stdev=17220.76 clat percentiles (usec): | 1.00th=[26346], 5.00th=[26346], 10.00th=[26346], 20.00th=[26346], | 30.00th=[37487], 40.00th=[37487], 50.00th=[47973], 60.00th=[47973], | 70.00th=[57934], 80.00th=[57934], 90.00th=[70779], 95.00th=[70779], | 99.00th=[70779], 99.50th=[70779], 99.90th=[70779], 99.95th=[70779], | 99.99th=[70779] lat (usec) : 500=16.67% lat (msec) : 50=66.67%, 100=16.67% cpu : usr=0.00%, sys=44.73%, ctx=108, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25472: Sat Feb 24 23:23:06 2024 read: IOPS=2, BW=2817KiB/s (2885kB/s)(2048KiB/727msec) clat (usec): min=41035, max=50975, avg=46005.26, stdev=7028.58 lat (usec): min=41036, max=50976, avg=46006.09, stdev=7028.53 clat percentiles (usec): | 1.00th=[41157], 5.00th=[41157], 10.00th=[41157], 20.00th=[41157], | 30.00th=[41157], 40.00th=[41157], 50.00th=[41157], 60.00th=[51119], | 70.00th=[51119], 80.00th=[51119], 90.00th=[51119], 95.00th=[51119], | 99.00th=[51119], 99.50th=[51119], 99.90th=[51119], 99.95th=[51119], | 99.99th=[51119] bw ( KiB/s): min= 4087, max= 4087, per=14.51%, avg=4087.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 write: IOPS=13, BW=13.8MiB/s (14.4MB/s)(10.0MiB/727msec) clat (msec): min=29, max=142, avg=62.56, stdev=32.88 lat (msec): min=29, max=142, avg=62.62, stdev=32.88 clat percentiles (msec): | 1.00th=[ 30], 5.00th=[ 30], 10.00th=[ 30], 20.00th=[ 40], | 30.00th=[ 43], 40.00th=[ 45], 50.00th=[ 51], 60.00th=[ 57], | 70.00th=[ 75], 80.00th=[ 75], 90.00th=[ 88], 95.00th=[ 142], | 99.00th=[ 142], 99.50th=[ 142], 99.90th=[ 142], 99.95th=[ 142], | 99.99th=[ 142] bw ( KiB/s): min=10219, max=10219, per=25.91%, avg=10219.00, stdev= 0.00, samples=1 iops : min= 9, max= 9, avg= 9.00, stdev= 0.00, samples=1 lat (msec) : 50=41.67%, 100=50.00%, 250=8.33% cpu : usr=0.00%, sys=31.13%, ctx=98, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25473: Sat Feb 24 23:23:06 2024 read: IOPS=11, BW=11.5MiB/s (12.1MB/s)(6144KiB/521msec) clat (usec): min=397, max=56310, avg=30718.52, stdev=24083.41 lat (usec): min=397, max=56310, avg=30719.45, stdev=24083.52 clat percentiles (usec): | 1.00th=[ 396], 5.00th=[ 396], 10.00th=[ 396], 20.00th=[ 453], | 30.00th=[ 453], 40.00th=[41157], 50.00th=[41157], 60.00th=[42730], | 70.00th=[43254], 80.00th=[43254], 90.00th=[56361], 95.00th=[56361], | 99.00th=[56361], 99.50th=[56361], 99.90th=[56361], 99.95th=[56361], | 99.99th=[56361] bw ( KiB/s): min=10240, max=10240, per=36.35%, avg=10240.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 write: IOPS=11, BW=11.5MiB/s (12.1MB/s)(6144KiB/521msec) clat (msec): min=14, max=133, avg=54.34, stdev=44.42 lat (msec): min=14, max=133, avg=54.39, stdev=44.44 clat percentiles (msec): | 1.00th=[ 15], 5.00th=[ 15], 10.00th=[ 15], 20.00th=[ 25], | 30.00th=[ 25], 40.00th=[ 29], 50.00th=[ 29], 60.00th=[ 51], | 70.00th=[ 75], 80.00th=[ 75], 90.00th=[ 134], 95.00th=[ 134], | 99.00th=[ 134], 99.50th=[ 134], 99.90th=[ 134], 99.95th=[ 134], | 99.99th=[ 134] bw ( KiB/s): min=12288, max=12288, per=31.16%, avg=12288.00, stdev= 0.00, samples=1 iops : min= 12, max= 12, avg=12.00, stdev= 0.00, samples=1 lat (usec) : 500=16.67% lat (msec) : 20=8.33%, 50=41.67%, 100=25.00%, 250=8.33% cpu : usr=0.00%, sys=37.88%, ctx=67, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25474: Sat Feb 24 23:23:06 2024 read: IOPS=8, BW=9046KiB/s (9263kB/s)(5120KiB/566msec) clat (usec): min=312, max=8023, avg=2786.52, stdev=3454.53 lat (usec): min=313, max=8023, avg=2787.40, stdev=3454.27 clat percentiles (usec): | 1.00th=[ 314], 5.00th=[ 314], 10.00th=[ 314], 20.00th=[ 314], | 30.00th=[ 441], 40.00th=[ 441], 50.00th=[ 506], 60.00th=[ 506], | 70.00th=[ 4621], 80.00th=[ 4621], 90.00th=[ 8029], 95.00th=[ 8029], | 99.00th=[ 8029], 99.50th=[ 8029], 99.90th=[ 8029], 99.95th=[ 8029], | 99.99th=[ 8029] bw ( KiB/s): min= 8175, max= 8175, per=29.02%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 7, max= 7, avg= 7.00, stdev= 0.00, samples=1 write: IOPS=12, BW=12.4MiB/s (12.0MB/s)(7168KiB/566msec) clat (msec): min=35, max=147, avg=78.41, stdev=45.58 lat (msec): min=35, max=147, avg=78.76, stdev=45.34 clat percentiles (msec): | 1.00th=[ 36], 5.00th=[ 36], 10.00th=[ 36], 20.00th=[ 44], | 30.00th=[ 45], 40.00th=[ 45], 50.00th=[ 50], 60.00th=[ 106], | 70.00th=[ 106], 80.00th=[ 123], 90.00th=[ 148], 95.00th=[ 148], | 99.00th=[ 148], 99.50th=[ 148], 99.90th=[ 148], 99.95th=[ 148], | 99.99th=[ 148] bw ( KiB/s): min=10219, max=10219, per=25.91%, avg=10219.00, stdev= 0.00, samples=1 iops : min= 9, max= 9, avg= 9.00, stdev= 0.00, samples=1 lat (usec) : 500=16.67%, 750=8.33% lat (msec) : 10=16.67%, 50=33.33%, 250=25.00% cpu : usr=0.18%, sys=20.18%, ctx=71, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=27.5MiB/s (28.8MB/s), 2817KiB/s-16.4MiB/s (2885kB/s-17.1MB/s), io=20.0MiB (20.0MB), run=428-727msec WRITE: bw=38.5MiB/s (40.4MB/s), 11.5MiB/s-13.8MiB/s (12.1MB/s-14.4MB/s), io=28.0MiB (29.4MB), run=428-727msec rand-rw: (groupid=0, jobs=1): err= 0: pid=25466: Sat Feb 24 23:23:06 2024 read: IOPS=12, BW=12.5MiB/s (13.1MB/s)(7168KiB/561msec) clat (usec): min=25402, max=57931, avg=38949.07, stdev=13625.46 lat (usec): min=25403, max=57932, avg=38950.02, stdev=13625.68 clat percentiles (usec): | 1.00th=[25297], 5.00th=[25297], 10.00th=[25297], 20.00th=[27395], | 30.00th=[28181], 40.00th=[28181], 50.00th=[36963], 60.00th=[40109], | 70.00th=[40109], 80.00th=[56886], 90.00th=[57934], 95.00th=[57934], | 99.00th=[57934], 99.50th=[57934], 99.90th=[57934], 99.95th=[57934], | 99.99th=[57934] bw ( KiB/s): min=12288, max=12288, per=46.02%, avg=12288.00, stdev= 0.00, samples=1 iops : min= 12, max= 12, avg=12.00, stdev= 0.00, samples=1 write: IOPS=8, BW=9127KiB/s (9346kB/s)(5120KiB/561msec) clat (usec): min=43971, max=81304, avg=57495.41, stdev=14018.62 lat (usec): min=44029, max=81365, avg=57553.10, stdev=14020.13 clat percentiles (usec): | 1.00th=[43779], 5.00th=[43779], 10.00th=[43779], 20.00th=[43779], | 30.00th=[53216], 40.00th=[53216], 50.00th=[54264], 60.00th=[54264], | 70.00th=[54789], 80.00th=[54789], 90.00th=[81265], 95.00th=[81265], | 99.00th=[81265], 99.50th=[81265], 99.90th=[81265], 99.95th=[81265], | 99.99th=[81265] bw ( KiB/s): min= 8192, max= 8192, per=21.91%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 lat (msec) : 50=50.00%, 100=50.00% cpu : usr=0.18%, sys=17.14%, ctx=33, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25467: Sat Feb 24 23:23:06 2024 read: IOPS=2, BW=2821KiB/s (2889kB/s)(2048KiB/726msec) clat (usec): min=21216, max=43756, avg=32486.61, stdev=15938.59 lat (usec): min=21217, max=43757, avg=32487.51, stdev=15938.57 clat percentiles (usec): | 1.00th=[21103], 5.00th=[21103], 10.00th=[21103], 20.00th=[21103], | 30.00th=[21103], 40.00th=[21103], 50.00th=[21103], 60.00th=[43779], | 70.00th=[43779], 80.00th=[43779], 90.00th=[43779], 95.00th=[43779], | 99.00th=[43779], 99.50th=[43779], 99.90th=[43779], 99.95th=[43779], | 99.99th=[43779] bw ( KiB/s): min= 4096, max= 4096, per=15.34%, avg=4096.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 write: IOPS=13, BW=13.8MiB/s (14.4MB/s)(10.0MiB/726msec) clat (msec): min=39, max=148, avg=65.91, stdev=34.00 lat (msec): min=39, max=148, avg=65.98, stdev=34.00 clat percentiles (msec): | 1.00th=[ 40], 5.00th=[ 40], 10.00th=[ 40], 20.00th=[ 41], | 30.00th=[ 48], 40.00th=[ 48], 50.00th=[ 52], 60.00th=[ 56], | 70.00th=[ 66], 80.00th=[ 66], 90.00th=[ 102], 95.00th=[ 148], | 99.00th=[ 148], 99.50th=[ 148], 99.90th=[ 148], 99.95th=[ 148], | 99.99th=[ 148] bw ( KiB/s): min=10240, max=10240, per=27.39%, avg=10240.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 lat (msec) : 50=50.00%, 100=33.33%, 250=16.67% cpu : usr=0.14%, sys=16.41%, ctx=53, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25469: Sat Feb 24 23:23:06 2024 read: IOPS=8, BW=8225KiB/s (8422kB/s)(6144KiB/747msec) clat (msec): min=19, max=102, avg=49.16, stdev=30.22 lat (msec): min=19, max=102, avg=49.16, stdev=30.22 clat percentiles (msec): | 1.00th=[ 20], 5.00th=[ 20], 10.00th=[ 20], 20.00th=[ 30], | 30.00th=[ 30], 40.00th=[ 36], 50.00th=[ 36], 60.00th=[ 45], | 70.00th=[ 63], 80.00th=[ 63], 90.00th=[ 104], 95.00th=[ 104], | 99.00th=[ 104], 99.50th=[ 104], 99.90th=[ 104], 99.95th=[ 104], | 99.99th=[ 104] bw ( KiB/s): min= 4096, max= 4096, per=15.34%, avg=4096.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 write: IOPS=8, BW=8225KiB/s (8422kB/s)(6144KiB/747msec) clat (msec): min=44, max=144, avg=75.13, stdev=35.81 lat (msec): min=44, max=144, avg=75.18, stdev=35.83 clat percentiles (msec): | 1.00th=[ 45], 5.00th=[ 45], 10.00th=[ 45], 20.00th=[ 51], | 30.00th=[ 51], 40.00th=[ 69], 50.00th=[ 69], 60.00th=[ 69], | 70.00th=[ 73], 80.00th=[ 73], 90.00th=[ 144], 95.00th=[ 144], | 99.00th=[ 144], 99.50th=[ 144], 99.90th=[ 144], 99.95th=[ 144], | 99.99th=[ 144] bw ( KiB/s): min= 8192, max= 8192, per=21.91%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 lat (msec) : 20=8.33%, 50=33.33%, 100=41.67%, 250=16.67% cpu : usr=0.00%, sys=14.48%, ctx=46, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25470: Sat Feb 24 23:23:06 2024 read: IOPS=6, BW=6675KiB/s (6836kB/s)(5120KiB/767msec) clat (msec): min=26, max=100, avg=48.57, stdev=29.70 lat (msec): min=26, max=100, avg=48.57, stdev=29.70 clat percentiles (msec): | 1.00th=[ 27], 5.00th=[ 27], 10.00th=[ 27], 20.00th=[ 27], | 30.00th=[ 34], 40.00th=[ 34], 50.00th=[ 38], 60.00th=[ 38], | 70.00th=[ 45], 80.00th=[ 45], 90.00th=[ 101], 95.00th=[ 101], | 99.00th=[ 101], 99.50th=[ 101], 99.90th=[ 101], 99.95th=[ 101], | 99.99th=[ 101] bw ( KiB/s): min= 4096, max= 4096, per=15.34%, avg=4096.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 write: IOPS=9, BW=9346KiB/s (9570kB/s)(7168KiB/767msec) clat (msec): min=41, max=124, avg=74.68, stdev=34.38 lat (msec): min=41, max=124, avg=74.74, stdev=34.39 clat percentiles (msec): | 1.00th=[ 42], 5.00th=[ 42], 10.00th=[ 42], 20.00th=[ 44], | 30.00th=[ 52], 40.00th=[ 52], 50.00th=[ 56], 60.00th=[ 103], | 70.00th=[ 103], 80.00th=[ 105], 90.00th=[ 125], 95.00th=[ 125], | 99.00th=[ 125], 99.50th=[ 125], 99.90th=[ 125], 99.95th=[ 125], | 99.99th=[ 125] bw ( KiB/s): min= 8192, max= 8192, per=21.91%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 lat (msec) : 50=50.00%, 100=16.67%, 250=33.33% cpu : usr=0.39%, sys=15.27%, ctx=41, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=26.1MiB/s (27.3MB/s), 2821KiB/s-12.5MiB/s (2889kB/s-13.1MB/s), io=20.0MiB (20.0MB), run=561-767msec WRITE: bw=36.5MiB/s (38.3MB/s), 8225KiB/s-13.8MiB/s (8422kB/s-14.4MB/s), io=28.0MiB (29.4MB), run=561-767msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=25480: Sat Feb 24 23:23:08 2024 read: IOPS=5, BW=20.3MiB/s (21.3MB/s)(12.0MiB/592msec) clat (msec): min=168, max=224, avg=195.70, stdev=28.00 lat (msec): min=168, max=224, avg=195.70, stdev=28.01 clat percentiles (msec): | 1.00th=[ 169], 5.00th=[ 169], 10.00th=[ 169], 20.00th=[ 169], | 30.00th=[ 169], 40.00th=[ 194], 50.00th=[ 194], 60.00th=[ 194], | 70.00th=[ 224], 80.00th=[ 224], 90.00th=[ 224], 95.00th=[ 224], | 99.00th=[ 224], 99.50th=[ 224], 99.90th=[ 224], 99.95th=[ 224], | 99.99th=[ 224] bw ( KiB/s): min=16384, max=16384, per=63.60%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 250=100.00% cpu : usr=0.00%, sys=41.79%, ctx=228, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25482: Sat Feb 24 23:23:08 2024 read: IOPS=1, BW=6440KiB/s (6595kB/s)(4096KiB/636msec) clat (nsec): min=167329k, max=167329k, avg=167328613.00, stdev= 0.00 lat (nsec): min=167330k, max=167330k, avg=167329523.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 167], 5.00th=[ 167], 10.00th=[ 167], 20.00th=[ 167], | 30.00th=[ 167], 40.00th=[ 167], 50.00th=[ 167], 60.00th=[ 167], | 70.00th=[ 167], 80.00th=[ 167], 90.00th=[ 167], 95.00th=[ 167], | 99.00th=[ 167], 99.50th=[ 167], 99.90th=[ 167], 99.95th=[ 167], | 99.99th=[ 167] bw ( KiB/s): min= 8192, max= 8192, per=31.80%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=3, BW=12.6MiB/s (13.2MB/s)(8192KiB/636msec) clat (msec): min=112, max=347, avg=229.67, stdev=166.03 lat (msec): min=112, max=347, avg=229.92, stdev=166.08 clat percentiles (msec): | 1.00th=[ 113], 5.00th=[ 113], 10.00th=[ 113], 20.00th=[ 113], | 30.00th=[ 113], 40.00th=[ 113], 50.00th=[ 113], 60.00th=[ 347], | 70.00th=[ 347], 80.00th=[ 347], 90.00th=[ 347], 95.00th=[ 347], | 99.00th=[ 347], 99.50th=[ 347], 99.90th=[ 347], 99.95th=[ 347], | 99.99th=[ 347] bw ( KiB/s): min= 8192, max= 8192, per=16.10%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.00%, sys=26.93%, ctx=98, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25484: Sat Feb 24 23:23:08 2024 write: IOPS=4, BW=18.6MiB/s (19.5MB/s)(12.0MiB/644msec) clat (msec): min=187, max=222, avg=204.31, stdev=17.59 lat (msec): min=187, max=222, avg=204.68, stdev=17.46 clat percentiles (msec): | 1.00th=[ 188], 5.00th=[ 188], 10.00th=[ 188], 20.00th=[ 188], | 30.00th=[ 188], 40.00th=[ 203], 50.00th=[ 203], 60.00th=[ 203], | 70.00th=[ 224], 80.00th=[ 224], 90.00th=[ 224], 95.00th=[ 224], | 99.00th=[ 224], 99.50th=[ 224], 99.90th=[ 224], 99.95th=[ 224], | 99.99th=[ 224] bw ( KiB/s): min=16384, max=16384, per=32.20%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 250=100.00% cpu : usr=0.00%, sys=29.86%, ctx=62, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25486: Sat Feb 24 23:23:08 2024 write: IOPS=6, BW=24.6MiB/s (25.8MB/s)(12.0MiB/488msec) clat (msec): min=86, max=198, avg=160.25, stdev=63.60 lat (msec): min=87, max=198, avg=160.50, stdev=63.58 clat percentiles (msec): | 1.00th=[ 87], 5.00th=[ 87], 10.00th=[ 87], 20.00th=[ 87], | 30.00th=[ 87], 40.00th=[ 197], 50.00th=[ 197], 60.00th=[ 197], | 70.00th=[ 199], 80.00th=[ 199], 90.00th=[ 199], 95.00th=[ 199], | 99.00th=[ 199], 99.50th=[ 199], 99.90th=[ 199], 99.95th=[ 199], | 99.99th=[ 199] lat (msec) : 100=33.33%, 250=66.67% cpu : usr=0.20%, sys=40.57%, ctx=45, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=25.2MiB/s (26.4MB/s), 6440KiB/s-20.3MiB/s (6595kB/s-21.3MB/s), io=16.0MiB (16.8MB), run=592-636msec WRITE: bw=49.7MiB/s (52.1MB/s), 12.6MiB/s-24.6MiB/s (13.2MB/s-25.8MB/s), io=32.0MiB (33.6MB), run=488-644msec rand-rw: (groupid=0, jobs=1): err= 0: pid=25481: Sat Feb 24 23:23:08 2024 read: IOPS=3, BW=13.8MiB/s (14.5MB/s)(12.0MiB/869msec) clat (msec): min=166, max=402, avg=289.21, stdev=118.68 lat (msec): min=166, max=402, avg=289.21, stdev=118.68 clat percentiles (msec): | 1.00th=[ 167], 5.00th=[ 167], 10.00th=[ 167], 20.00th=[ 167], | 30.00th=[ 167], 40.00th=[ 300], 50.00th=[ 300], 60.00th=[ 300], | 70.00th=[ 405], 80.00th=[ 405], 90.00th=[ 405], 95.00th=[ 405], | 99.00th=[ 405], 99.50th=[ 405], 99.90th=[ 405], 99.95th=[ 405], | 99.99th=[ 405] bw ( KiB/s): min=11130, max=11130, per=59.04%, avg=11130.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=66.67% cpu : usr=0.00%, sys=9.33%, ctx=27, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25483: Sat Feb 24 23:23:08 2024 read: IOPS=1, BW=5133KiB/s (5256kB/s)(4096KiB/798msec) clat (nsec): min=325488k, max=325488k, avg=325488283.00, stdev= 0.00 lat (nsec): min=325489k, max=325489k, avg=325489181.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 326], 5.00th=[ 326], 10.00th=[ 326], 20.00th=[ 326], | 30.00th=[ 326], 40.00th=[ 326], 50.00th=[ 326], 60.00th=[ 326], | 70.00th=[ 326], 80.00th=[ 326], 90.00th=[ 326], 95.00th=[ 326], | 99.00th=[ 326], 99.50th=[ 326], 99.90th=[ 326], 99.95th=[ 326], | 99.99th=[ 326] bw ( KiB/s): min= 5626, max= 5626, per=29.84%, avg=5626.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=2, BW=10.0MiB/s (10.5MB/s)(8192KiB/798msec) clat (msec): min=151, max=314, avg=233.22, stdev=115.20 lat (msec): min=151, max=314, avg=233.39, stdev=115.25 clat percentiles (msec): | 1.00th=[ 153], 5.00th=[ 153], 10.00th=[ 153], 20.00th=[ 153], | 30.00th=[ 153], 40.00th=[ 153], 50.00th=[ 153], 60.00th=[ 317], | 70.00th=[ 317], 80.00th=[ 317], 90.00th=[ 317], 95.00th=[ 317], | 99.00th=[ 317], 99.50th=[ 317], 99.90th=[ 317], 99.95th=[ 317], | 99.99th=[ 317] bw ( KiB/s): min= 5626, max= 5626, per=14.63%, avg=5626.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=66.67% cpu : usr=0.00%, sys=14.05%, ctx=92, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25485: Sat Feb 24 23:23:08 2024 write: IOPS=3, BW=14.1MiB/s (14.8MB/s)(12.0MiB/852msec) clat (msec): min=262, max=300, avg=276.99, stdev=20.42 lat (msec): min=262, max=300, avg=277.37, stdev=20.29 clat percentiles (msec): | 1.00th=[ 262], 5.00th=[ 262], 10.00th=[ 262], 20.00th=[ 262], | 30.00th=[ 262], 40.00th=[ 271], 50.00th=[ 271], 60.00th=[ 271], | 70.00th=[ 300], 80.00th=[ 300], 90.00th=[ 300], 95.00th=[ 300], | 99.00th=[ 300], 99.50th=[ 300], 99.90th=[ 300], 99.95th=[ 300], | 99.99th=[ 300] bw ( KiB/s): min= 8192, max= 8192, per=21.30%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 500=100.00% cpu : usr=0.00%, sys=14.34%, ctx=118, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25487: Sat Feb 24 23:23:08 2024 write: IOPS=7, BW=30.0MiB/s (31.5MB/s)(12.0MiB/400msec) clat (msec): min=67, max=229, avg=132.42, stdev=85.93 lat (msec): min=67, max=230, avg=132.58, stdev=86.03 clat percentiles (msec): | 1.00th=[ 67], 5.00th=[ 67], 10.00th=[ 67], 20.00th=[ 67], | 30.00th=[ 67], 40.00th=[ 101], 50.00th=[ 101], 60.00th=[ 101], | 70.00th=[ 230], 80.00th=[ 230], 90.00th=[ 230], 95.00th=[ 230], | 99.00th=[ 230], 99.50th=[ 230], 99.90th=[ 230], 99.95th=[ 230], | 99.99th=[ 230] lat (msec) : 100=33.33%, 250=66.67% cpu : usr=0.00%, sys=21.55%, ctx=38, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=18.4MiB/s (19.3MB/s), 5133KiB/s-13.8MiB/s (5256kB/s-14.5MB/s), io=16.0MiB (16.8MB), run=798-869msec WRITE: bw=37.6MiB/s (39.4MB/s), 10.0MiB/s-30.0MiB/s (10.5MB/s-31.5MB/s), io=32.0MiB (33.6MB), run=400-852msec