== sanity test 398b: DIO and buffer IO race ============== 19:22:29 (1712272949) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.39073 s, 36.2 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=15625: Thu Apr 4 19:22:46 2024 read: IOPS=106, BW=425KiB/s (435kB/s)(5992KiB/14110msec) clat (usec): min=46, max=205959, avg=5385.82, stdev=6040.31 lat (usec): min=47, max=205960, avg=5387.35, stdev=6040.67 clat percentiles (usec): | 1.00th=[ 59], 5.00th=[ 101], 10.00th=[ 2376], 20.00th=[ 2999], | 30.00th=[ 3458], 40.00th=[ 4047], 50.00th=[ 4752], 60.00th=[ 5538], | 70.00th=[ 6325], 80.00th=[ 7308], 90.00th=[ 8848], 95.00th=[ 10683], | 99.00th=[ 15795], 99.50th=[ 18744], 99.90th=[ 31065], 99.95th=[206570], | 99.99th=[206570] bw ( KiB/s): min= 216, max= 598, per=25.26%, avg=422.93, stdev=75.88, samples=28 iops : min= 54, max= 149, avg=105.64, stdev=18.94, samples=28 write: IOPS=111, BW=446KiB/s (457kB/s)(6296KiB/14110msec) clat (usec): min=394, max=24090, avg=3791.99, stdev=3526.26 lat (usec): min=394, max=24091, avg=3792.97, stdev=3526.27 clat percentiles (usec): | 1.00th=[ 441], 5.00th=[ 537], 10.00th=[ 603], 20.00th=[ 717], | 30.00th=[ 865], 40.00th=[ 1565], 50.00th=[ 3097], 60.00th=[ 3851], | 70.00th=[ 4752], 80.00th=[ 6456], 90.00th=[ 8717], 95.00th=[10814], | 99.00th=[15664], 99.50th=[17171], 99.90th=[20055], 99.95th=[23987], | 99.99th=[23987] bw ( KiB/s): min= 256, max= 624, per=25.27%, avg=442.96, stdev=80.56, samples=28 iops : min= 64, max= 156, avg=110.64, stdev=20.11, samples=28 lat (usec) : 50=0.16%, 100=2.21%, 250=0.33%, 500=1.76%, 750=9.73% lat (usec) : 1000=6.05% lat (msec) : 2=4.33%, 4=26.11%, 10=42.74%, 20=6.35%, 50=0.20% lat (msec) : 250=0.03% cpu : usr=0.35%, sys=22.36%, ctx=5768, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=15626: Thu Apr 4 19:22:46 2024 read: IOPS=103, BW=414KiB/s (424kB/s)(5940KiB/14339msec) clat (usec): min=38, max=22838, avg=5251.14, stdev=3000.57 lat (usec): min=39, max=22839, avg=5252.82, stdev=3000.96 clat percentiles (usec): | 1.00th=[ 58], 5.00th=[ 1565], 10.00th=[ 2442], 20.00th=[ 2966], | 30.00th=[ 3490], 40.00th=[ 4015], 50.00th=[ 4686], 60.00th=[ 5473], | 70.00th=[ 6325], 80.00th=[ 7373], 90.00th=[ 8979], 95.00th=[10552], | 99.00th=[15664], 99.50th=[18482], 99.90th=[20579], 99.95th=[22938], | 99.99th=[22938] bw ( KiB/s): min= 200, max= 558, per=24.38%, avg=408.11, stdev=79.38, samples=28 iops : min= 50, max= 139, avg=101.93, stdev=19.80, samples=28 write: IOPS=110, BW=443KiB/s (453kB/s)(6348KiB/14339msec) clat (usec): min=362, max=202940, avg=4075.21, stdev=6169.71 lat (usec): min=362, max=202941, avg=4076.21, stdev=6169.74 clat percentiles (usec): | 1.00th=[ 453], 5.00th=[ 545], 10.00th=[ 619], 20.00th=[ 734], | 30.00th=[ 881], 40.00th=[ 2245], 50.00th=[ 3228], 60.00th=[ 4015], | 70.00th=[ 5145], 80.00th=[ 6521], 90.00th=[ 8848], 95.00th=[ 10945], | 99.00th=[ 15795], 99.50th=[ 17957], 99.90th=[ 26084], 99.95th=[202376], | 99.99th=[202376] bw ( KiB/s): min= 256, max= 576, per=24.50%, avg=429.57, stdev=75.40, samples=28 iops : min= 64, max= 144, avg=107.29, stdev=18.83, samples=28 lat (usec) : 50=0.20%, 100=1.56%, 250=0.62%, 500=1.27%, 750=10.09% lat (usec) : 1000=5.86% lat (msec) : 2=3.55%, 4=27.12%, 10=43.42%, 20=6.02%, 50=0.26% lat (msec) : 250=0.03% cpu : usr=0.34%, sys=22.35%, ctx=5799, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=15628: Thu Apr 4 19:22:46 2024 read: IOPS=105, BW=421KiB/s (431kB/s)(6028KiB/14312msec) clat (usec): min=39, max=27265, avg=5210.00, stdev=3014.93 lat (usec): min=39, max=27266, avg=5211.04, stdev=3014.81 clat percentiles (usec): | 1.00th=[ 63], 5.00th=[ 2024], 10.00th=[ 2507], 20.00th=[ 2999], | 30.00th=[ 3425], 40.00th=[ 3982], 50.00th=[ 4686], 60.00th=[ 5342], | 70.00th=[ 6194], 80.00th=[ 7177], 90.00th=[ 8717], 95.00th=[10159], | 99.00th=[15008], 99.50th=[19530], 99.90th=[26608], 99.95th=[27395], | 99.99th=[27395] bw ( KiB/s): min= 104, max= 590, per=24.98%, avg=418.11, stdev=95.68, samples=28 iops : min= 26, max= 147, avg=104.43, stdev=23.90, samples=28 write: IOPS=109, BW=437KiB/s (448kB/s)(6260KiB/14312msec) clat (usec): min=390, max=214140, avg=4080.73, stdev=6420.40 lat (usec): min=390, max=214141, avg=4082.29, stdev=6420.17 clat percentiles (usec): | 1.00th=[ 449], 5.00th=[ 537], 10.00th=[ 619], 20.00th=[ 725], | 30.00th=[ 889], 40.00th=[ 1909], 50.00th=[ 3326], 60.00th=[ 4080], | 70.00th=[ 5211], 80.00th=[ 6718], 90.00th=[ 8848], 95.00th=[ 10814], | 99.00th=[ 15795], 99.50th=[ 18482], 99.90th=[ 33817], 99.95th=[214959], | 99.99th=[214959] bw ( KiB/s): min= 128, max= 590, per=24.65%, avg=432.11, stdev=94.32, samples=28 iops : min= 32, max= 147, avg=107.93, stdev=23.59, samples=28 lat (usec) : 50=0.20%, 100=1.66%, 250=0.26%, 500=1.33%, 750=10.32% lat (usec) : 1000=5.27% lat (msec) : 2=3.94%, 4=26.73%, 10=44.21%, 20=5.66%, 50=0.39% lat (msec) : 250=0.03% cpu : usr=0.26%, sys=22.28%, ctx=5813, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=15629: Thu Apr 4 19:22:46 2024 read: IOPS=105, BW=422KiB/s (433kB/s)(6052KiB/14326msec) clat (usec): min=46, max=207278, avg=5329.64, stdev=6136.54 lat (usec): min=46, max=207279, avg=5331.25, stdev=6136.66 clat percentiles (usec): | 1.00th=[ 68], 5.00th=[ 143], 10.00th=[ 2311], 20.00th=[ 2933], | 30.00th=[ 3359], 40.00th=[ 3884], 50.00th=[ 4490], 60.00th=[ 5276], | 70.00th=[ 6128], 80.00th=[ 7308], 90.00th=[ 9110], 95.00th=[ 10814], | 99.00th=[ 16909], 99.50th=[ 20055], 99.90th=[ 39584], 99.95th=[206570], | 99.99th=[206570] bw ( KiB/s): min= 232, max= 574, per=24.88%, avg=416.57, stdev=81.86, samples=28 iops : min= 58, max= 143, avg=104.00, stdev=20.40, samples=28 write: IOPS=108, BW=435KiB/s (446kB/s)(6236KiB/14326msec) clat (usec): min=353, max=26803, avg=3968.15, stdev=3641.36 lat (usec): min=354, max=26804, avg=3970.86, stdev=3640.59 clat percentiles (usec): | 1.00th=[ 457], 5.00th=[ 553], 10.00th=[ 644], 20.00th=[ 734], | 30.00th=[ 889], 40.00th=[ 2114], 50.00th=[ 3261], 60.00th=[ 4015], | 70.00th=[ 5080], 80.00th=[ 6718], 90.00th=[ 8848], 95.00th=[10945], | 99.00th=[15533], 99.50th=[19006], 99.90th=[24511], 99.95th=[26870], | 99.99th=[26870] bw ( KiB/s): min= 280, max= 646, per=24.48%, avg=429.14, stdev=88.27, samples=28 iops : min= 70, max= 161, avg=107.14, stdev=22.03, samples=28 lat (usec) : 50=0.13%, 100=1.95%, 250=0.55%, 500=1.30%, 750=9.86% lat (usec) : 1000=5.76% lat (msec) : 2=3.71%, 4=27.90%, 10=42.06%, 20=6.35%, 50=0.39% lat (msec) : 250=0.03% cpu : usr=0.30%, sys=22.02%, ctx=5870, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1675KiB/s (1715kB/s), 414KiB/s-425KiB/s (424kB/s-435kB/s), io=23.4MiB (24.6MB), run=14110-14339msec WRITE: bw=1753KiB/s (1795kB/s), 435KiB/s-446KiB/s (446kB/s-457kB/s), io=24.6MiB (25.7MB), run=14110-14339msec rand-rw: (groupid=0, jobs=1): err= 0: pid=15630: Thu Apr 4 19:22:51 2024 read: IOPS=80, BW=322KiB/s (330kB/s)(5992KiB/18604msec) clat (usec): min=1448, max=207559, avg=4803.02, stdev=6364.08 lat (usec): min=1449, max=207561, avg=4803.91, stdev=6364.12 clat percentiles (usec): | 1.00th=[ 1598], 5.00th=[ 1762], 10.00th=[ 1860], 20.00th=[ 2073], | 30.00th=[ 2343], 40.00th=[ 2835], 50.00th=[ 3425], 60.00th=[ 4015], | 70.00th=[ 4817], 80.00th=[ 6456], 90.00th=[ 9765], 95.00th=[ 12780], | 99.00th=[ 17695], 99.50th=[ 20841], 99.90th=[ 26346], 99.95th=[206570], | 99.99th=[206570] bw ( KiB/s): min= 128, max= 744, per=24.83%, avg=320.27, stdev=169.12, samples=37 iops : min= 32, max= 186, avg=79.97, stdev=42.29, samples=37 write: IOPS=84, BW=338KiB/s (347kB/s)(6296KiB/18604msec) clat (usec): min=2099, max=40970, avg=7226.78, stdev=4794.30 lat (usec): min=2100, max=40972, avg=7228.08, stdev=4794.54 clat percentiles (usec): | 1.00th=[ 2540], 5.00th=[ 2900], 10.00th=[ 3130], 20.00th=[ 3490], | 30.00th=[ 3982], 40.00th=[ 4817], 50.00th=[ 5669], 60.00th=[ 6587], | 70.00th=[ 8291], 80.00th=[10552], 90.00th=[13304], 95.00th=[16319], | 99.00th=[25297], 99.50th=[27919], 99.90th=[38011], 99.95th=[41157], | 99.99th=[41157] bw ( KiB/s): min= 136, max= 726, per=24.89%, avg=336.27, stdev=164.38, samples=37 iops : min= 34, max= 181, avg=83.97, stdev=41.12, samples=37 lat (msec) : 2=8.14%, 4=36.46%, 10=39.19%, 20=14.65%, 50=1.53% lat (msec) : 250=0.03% cpu : usr=0.25%, sys=10.41%, ctx=3587, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=15631: Thu Apr 4 19:22:51 2024 read: IOPS=79, BW=320KiB/s (327kB/s)(5940KiB/18579msec) clat (usec): min=1356, max=208209, avg=4968.29, stdev=6582.74 lat (usec): min=1357, max=208210, avg=4969.22, stdev=6582.70 clat percentiles (usec): | 1.00th=[ 1549], 5.00th=[ 1745], 10.00th=[ 1860], 20.00th=[ 2057], | 30.00th=[ 2343], 40.00th=[ 2835], 50.00th=[ 3392], 60.00th=[ 4047], | 70.00th=[ 5014], 80.00th=[ 6783], 90.00th=[ 10421], 95.00th=[ 13042], | 99.00th=[ 20579], 99.50th=[ 22676], 99.90th=[ 32113], 99.95th=[208667], | 99.99th=[208667] bw ( KiB/s): min= 112, max= 752, per=24.47%, avg=315.72, stdev=172.58, samples=36 iops : min= 28, max= 188, avg=78.89, stdev=43.16, samples=36 write: IOPS=85, BW=342KiB/s (350kB/s)(6348KiB/18579msec) clat (usec): min=2266, max=40757, avg=7039.42, stdev=4630.55 lat (usec): min=2267, max=40758, avg=7040.42, stdev=4630.63 clat percentiles (usec): | 1.00th=[ 2442], 5.00th=[ 2900], 10.00th=[ 3130], 20.00th=[ 3458], | 30.00th=[ 3949], 40.00th=[ 4686], 50.00th=[ 5473], 60.00th=[ 6390], | 70.00th=[ 8029], 80.00th=[10159], 90.00th=[13042], 95.00th=[15926], | 99.00th=[24249], 99.50th=[26870], 99.90th=[37487], 99.95th=[40633], | 99.99th=[40633] bw ( KiB/s): min= 176, max= 752, per=24.60%, avg=332.39, stdev=159.69, samples=36 iops : min= 44, max= 188, avg=83.06, stdev=39.95, samples=36 lat (msec) : 2=8.50%, 4=36.30%, 10=39.13%, 20=14.45%, 50=1.60% lat (msec) : 250=0.03% cpu : usr=0.28%, sys=10.32%, ctx=3683, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=15632: Thu Apr 4 19:22:51 2024 read: IOPS=81, BW=325KiB/s (333kB/s)(6028KiB/18558msec) clat (usec): min=1436, max=27392, avg=4898.92, stdev=3717.11 lat (usec): min=1437, max=27392, avg=4899.76, stdev=3717.13 clat percentiles (usec): | 1.00th=[ 1598], 5.00th=[ 1762], 10.00th=[ 1876], 20.00th=[ 2147], | 30.00th=[ 2540], 40.00th=[ 3097], 50.00th=[ 3589], 60.00th=[ 4228], | 70.00th=[ 5145], 80.00th=[ 6915], 90.00th=[10159], 95.00th=[13042], | 99.00th=[17433], 99.50th=[21627], 99.90th=[26346], 99.95th=[27395], | 99.99th=[27395] bw ( KiB/s): min= 112, max= 744, per=25.13%, avg=324.19, stdev=158.31, samples=37 iops : min= 28, max= 186, avg=80.97, stdev=39.59, samples=37 write: IOPS=84, BW=337KiB/s (345kB/s)(6260KiB/18558msec) clat (msec): min=2, max=209, avg= 7.12, stdev= 6.85 lat (msec): min=2, max=209, avg= 7.12, stdev= 6.85 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 4], 20.00th=[ 4], | 30.00th=[ 4], 40.00th=[ 5], 50.00th=[ 6], 60.00th=[ 7], | 70.00th=[ 9], 80.00th=[ 11], 90.00th=[ 14], 95.00th=[ 17], | 99.00th=[ 24], 99.50th=[ 27], 99.90th=[ 33], 99.95th=[ 211], | 99.99th=[ 211] bw ( KiB/s): min= 136, max= 824, per=24.88%, avg=336.08, stdev=174.56, samples=37 iops : min= 34, max= 206, avg=83.95, stdev=43.64, samples=37 lat (msec) : 2=7.32%, 4=36.59%, 10=40.10%, 20=14.58%, 50=1.37% lat (msec) : 250=0.03% cpu : usr=0.18%, sys=10.49%, ctx=3743, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=15633: Thu Apr 4 19:22:51 2024 read: IOPS=81, BW=328KiB/s (336kB/s)(6052KiB/18457msec) clat (usec): min=1494, max=27514, avg=4853.47, stdev=3835.93 lat (usec): min=1495, max=27516, avg=4854.34, stdev=3835.95 clat percentiles (usec): | 1.00th=[ 1598], 5.00th=[ 1745], 10.00th=[ 1860], 20.00th=[ 2114], | 30.00th=[ 2442], 40.00th=[ 2966], 50.00th=[ 3490], 60.00th=[ 4113], | 70.00th=[ 5014], 80.00th=[ 6783], 90.00th=[10290], 95.00th=[13173], | 99.00th=[19268], 99.50th=[22676], 99.90th=[26084], 99.95th=[27395], | 99.99th=[27395] bw ( KiB/s): min= 144, max= 768, per=25.02%, avg=322.72, stdev=162.29, samples=36 iops : min= 36, max= 192, avg=80.58, stdev=40.60, samples=36 write: IOPS=84, BW=338KiB/s (346kB/s)(6236KiB/18457msec) clat (msec): min=2, max=207, avg= 7.11, stdev= 6.69 lat (msec): min=2, max=207, avg= 7.11, stdev= 6.69 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 4], 20.00th=[ 4], | 30.00th=[ 4], 40.00th=[ 5], 50.00th=[ 6], 60.00th=[ 7], | 70.00th=[ 9], 80.00th=[ 11], 90.00th=[ 13], 95.00th=[ 16], | 99.00th=[ 23], 99.50th=[ 27], 99.90th=[ 39], 99.95th=[ 207], | 99.99th=[ 207] bw ( KiB/s): min= 160, max= 750, per=24.69%, avg=333.61, stdev=166.84, samples=36 iops : min= 40, max= 187, avg=83.31, stdev=41.73, samples=36 lat (msec) : 2=7.85%, 4=36.36%, 10=39.94%, 20=14.62%, 50=1.20% lat (msec) : 250=0.03% cpu : usr=0.28%, sys=10.43%, ctx=3668, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1291KiB/s (1322kB/s), 320KiB/s-328KiB/s (327kB/s-336kB/s), io=23.4MiB (24.6MB), run=18457-18604msec WRITE: bw=1351KiB/s (1384kB/s), 337KiB/s-342KiB/s (345kB/s-350kB/s), io=24.6MiB (25.7MB), run=18457-18604msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=15661: Thu Apr 4 19:22:58 2024 read: IOPS=58, BW=933KiB/s (956kB/s)(5920KiB/6344msec) clat (usec): min=48, max=36954, avg=11252.78, stdev=4137.25 lat (usec): min=48, max=36954, avg=11253.49, stdev=4137.28 clat percentiles (usec): | 1.00th=[ 74], 5.00th=[ 7111], 10.00th=[ 8029], 20.00th=[ 8848], | 30.00th=[ 9634], 40.00th=[10290], 50.00th=[10814], 60.00th=[11600], | 70.00th=[12256], 80.00th=[13042], 90.00th=[15008], 95.00th=[16712], | 99.00th=[27132], 99.50th=[29230], 99.90th=[36963], 99.95th=[36963], | 99.99th=[36963] bw ( KiB/s): min= 734, max= 1120, per=25.06%, avg=930.25, stdev=112.16, samples=12 iops : min= 45, max= 70, avg=58.00, stdev= 7.10, samples=12 write: IOPS=62, BW=1004KiB/s (1028kB/s)(6368KiB/6344msec) clat (usec): min=617, max=51999, avg=5459.12, stdev=6270.00 lat (usec): min=617, max=52011, avg=5460.44, stdev=6270.18 clat percentiles (usec): | 1.00th=[ 660], 5.00th=[ 807], 10.00th=[ 930], 20.00th=[ 1221], | 30.00th=[ 2376], 40.00th=[ 2900], 50.00th=[ 3458], 60.00th=[ 4490], | 70.00th=[ 5866], 80.00th=[ 8225], 90.00th=[11863], 95.00th=[15795], | 99.00th=[35914], 99.50th=[51643], 99.90th=[52167], 99.95th=[52167], | 99.99th=[52167] bw ( KiB/s): min= 800, max= 1344, per=26.01%, avg=1015.58, stdev=161.68, samples=12 iops : min= 50, max= 84, avg=63.33, stdev=10.18, samples=12 lat (usec) : 50=0.13%, 100=1.17%, 250=0.26%, 750=1.56%, 1000=4.82% lat (msec) : 2=5.99%, 4=16.93%, 10=30.86%, 20=35.42%, 50=2.60% lat (msec) : 100=0.26% cpu : usr=0.22%, sys=17.86%, ctx=2729, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=15662: Thu Apr 4 19:22:58 2024 read: IOPS=57, BW=916KiB/s (938kB/s)(5744KiB/6271msec) clat (usec): min=60, max=36066, avg=11396.07, stdev=3690.55 lat (usec): min=61, max=36066, avg=11396.79, stdev=3690.59 clat percentiles (usec): | 1.00th=[ 88], 5.00th=[ 7373], 10.00th=[ 8029], 20.00th=[ 8979], | 30.00th=[ 9634], 40.00th=[10421], 50.00th=[11207], 60.00th=[11863], | 70.00th=[12387], 80.00th=[13304], 90.00th=[15401], 95.00th=[17433], | 99.00th=[23200], 99.50th=[23987], 99.90th=[35914], 99.95th=[35914], | 99.99th=[35914] bw ( KiB/s): min= 736, max= 1088, per=24.63%, avg=914.17, stdev=137.43, samples=12 iops : min= 46, max= 68, avg=57.00, stdev= 8.41, samples=12 write: IOPS=65, BW=1044KiB/s (1069kB/s)(6544KiB/6271msec) clat (usec): min=622, max=49595, avg=5307.29, stdev=5396.25 lat (usec): min=622, max=49596, avg=5308.66, stdev=5396.38 clat percentiles (usec): | 1.00th=[ 685], 5.00th=[ 816], 10.00th=[ 963], 20.00th=[ 1860], | 30.00th=[ 2474], 40.00th=[ 2966], 50.00th=[ 3589], 60.00th=[ 4555], | 70.00th=[ 5669], 80.00th=[ 7570], 90.00th=[11863], 95.00th=[14877], | 99.00th=[24249], 99.50th=[30540], 99.90th=[49546], 99.95th=[49546], | 99.99th=[49546] bw ( KiB/s): min= 736, max= 1312, per=26.21%, avg=1023.67, stdev=171.82, samples=12 iops : min= 46, max= 82, avg=63.83, stdev=10.88, samples=12 lat (usec) : 100=0.52%, 250=0.13%, 500=0.13%, 750=1.69%, 1000=5.34% lat (msec) : 2=4.43%, 4=18.23%, 10=30.60%, 20=36.46%, 50=2.47% cpu : usr=0.11%, sys=18.05%, ctx=2705, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=15663: Thu Apr 4 19:22:58 2024 read: IOPS=60, BW=975KiB/s (998kB/s)(6256KiB/6417msec) clat (usec): min=68, max=50071, avg=11194.86, stdev=4057.82 lat (usec): min=69, max=50073, avg=11195.84, stdev=4057.88 clat percentiles (usec): | 1.00th=[ 85], 5.00th=[ 7373], 10.00th=[ 8225], 20.00th=[ 8979], | 30.00th=[ 9634], 40.00th=[10290], 50.00th=[10814], 60.00th=[11469], | 70.00th=[12125], 80.00th=[13173], 90.00th=[14615], 95.00th=[16909], | 99.00th=[24773], 99.50th=[34866], 99.90th=[50070], 99.95th=[50070], | 99.99th=[50070] bw ( KiB/s): min= 736, max= 1184, per=26.21%, avg=973.08, stdev=171.97, samples=12 iops : min= 46, max= 74, avg=60.75, stdev=10.66, samples=12 write: IOPS=58, BW=940KiB/s (963kB/s)(6032KiB/6417msec) clat (usec): min=564, max=45400, avg=5382.47, stdev=6245.03 lat (usec): min=565, max=45401, avg=5385.53, stdev=6243.92 clat percentiles (usec): | 1.00th=[ 668], 5.00th=[ 799], 10.00th=[ 922], 20.00th=[ 1532], | 30.00th=[ 2409], 40.00th=[ 2769], 50.00th=[ 3294], 60.00th=[ 4178], | 70.00th=[ 5211], 80.00th=[ 7570], 90.00th=[13042], 95.00th=[16319], | 99.00th=[39584], 99.50th=[42206], 99.90th=[45351], 99.95th=[45351], | 99.99th=[45351] bw ( KiB/s): min= 608, max= 1120, per=23.56%, avg=919.83, stdev=169.63, samples=12 iops : min= 38, max= 70, avg=57.42, stdev=10.71, samples=12 lat (usec) : 100=0.91%, 250=0.26%, 500=0.13%, 750=1.30%, 1000=5.08% lat (msec) : 2=4.82%, 4=17.71%, 10=31.12%, 20=36.33%, 50=2.21% lat (msec) : 100=0.13% cpu : usr=0.22%, sys=17.92%, ctx=2786, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=15664: Thu Apr 4 19:22:58 2024 read: IOPS=58, BW=935KiB/s (957kB/s)(6032KiB/6452msec) clat (usec): min=53, max=33998, avg=11334.38, stdev=3843.12 lat (usec): min=54, max=33999, avg=11335.16, stdev=3843.14 clat percentiles (usec): | 1.00th=[ 78], 5.00th=[ 7570], 10.00th=[ 8225], 20.00th=[ 8979], | 30.00th=[ 9634], 40.00th=[10159], 50.00th=[10683], 60.00th=[11469], | 70.00th=[12518], 80.00th=[13829], 90.00th=[15139], 95.00th=[17171], | 99.00th=[27919], 99.50th=[32637], 99.90th=[33817], 99.95th=[33817], | 99.99th=[33817] bw ( KiB/s): min= 768, max= 1152, per=24.78%, avg=920.00, stdev=113.45, samples=12 iops : min= 48, max= 72, avg=57.50, stdev= 7.09, samples=12 write: IOPS=60, BW=970KiB/s (993kB/s)(6256KiB/6452msec) clat (usec): min=589, max=47062, avg=5553.17, stdev=6063.01 lat (usec): min=589, max=47064, avg=5554.62, stdev=6062.95 clat percentiles (usec): | 1.00th=[ 619], 5.00th=[ 742], 10.00th=[ 906], 20.00th=[ 1156], | 30.00th=[ 2343], 40.00th=[ 2802], 50.00th=[ 3326], 60.00th=[ 4359], | 70.00th=[ 5932], 80.00th=[ 8455], 90.00th=[13566], 95.00th=[16057], | 99.00th=[33424], 99.50th=[42730], 99.90th=[46924], 99.95th=[46924], | 99.99th=[46924] bw ( KiB/s): min= 608, max= 1184, per=24.79%, avg=968.00, stdev=166.07, samples=12 iops : min= 38, max= 74, avg=60.50, stdev=10.38, samples=12 lat (usec) : 100=0.78%, 250=0.13%, 750=2.86%, 1000=4.30% lat (msec) : 2=5.21%, 4=17.58%, 10=30.73%, 20=35.94%, 50=2.47% cpu : usr=0.16%, sys=17.86%, ctx=2700, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=3712KiB/s (3801kB/s), 916KiB/s-975KiB/s (938kB/s-998kB/s), io=23.4MiB (24.5MB), run=6271-6452msec WRITE: bw=3906KiB/s (4000kB/s), 940KiB/s-1044KiB/s (963kB/s-1069kB/s), io=24.6MiB (25.8MB), run=6271-6452msec rand-rw: (groupid=0, jobs=1): err= 0: pid=15665: Thu Apr 4 19:22:58 2024 read: IOPS=58, BW=931KiB/s (953kB/s)(5920KiB/6358msec) clat (usec): min=1817, max=36075, avg=6319.10, stdev=5043.09 lat (usec): min=1818, max=36076, avg=6320.01, stdev=5043.20 clat percentiles (usec): | 1.00th=[ 1975], 5.00th=[ 2212], 10.00th=[ 2442], 20.00th=[ 2769], | 30.00th=[ 3032], 40.00th=[ 3556], 50.00th=[ 4228], 60.00th=[ 5473], | 70.00th=[ 7242], 80.00th=[ 9765], 90.00th=[12387], 95.00th=[14746], | 99.00th=[27132], 99.50th=[35390], 99.90th=[35914], 99.95th=[35914], | 99.99th=[35914] bw ( KiB/s): min= 736, max= 1149, per=25.06%, avg=930.25, stdev=127.32, samples=12 iops : min= 46, max= 71, avg=58.00, stdev= 7.92, samples=12 write: IOPS=62, BW=1002KiB/s (1026kB/s)(6368KiB/6358msec) clat (usec): min=2783, max=40135, avg=10074.98, stdev=5962.87 lat (usec): min=2784, max=40137, avg=10076.49, stdev=5962.82 clat percentiles (usec): | 1.00th=[ 2966], 5.00th=[ 3687], 10.00th=[ 4113], 20.00th=[ 4948], | 30.00th=[ 6325], 40.00th=[ 7111], 50.00th=[ 8717], 60.00th=[10290], | 70.00th=[11863], 80.00th=[14091], 90.00th=[17695], 95.00th=[20579], | 99.00th=[33424], 99.50th=[39584], 99.90th=[40109], 99.95th=[40109], | 99.99th=[40109] bw ( KiB/s): min= 800, max= 1344, per=25.80%, avg=1007.58, stdev=150.26, samples=12 iops : min= 50, max= 84, avg=62.83, stdev= 9.39, samples=12 lat (msec) : 2=0.52%, 4=26.17%, 10=42.84%, 20=26.17%, 50=4.30% cpu : usr=0.17%, sys=10.08%, ctx=1175, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=15666: Thu Apr 4 19:22:58 2024 read: IOPS=57, BW=912KiB/s (934kB/s)(5744KiB/6298msec) clat (usec): min=1946, max=29470, avg=6205.92, stdev=4851.96 lat (usec): min=1946, max=29471, avg=6206.85, stdev=4852.02 clat percentiles (usec): | 1.00th=[ 2073], 5.00th=[ 2245], 10.00th=[ 2343], 20.00th=[ 2573], | 30.00th=[ 2933], 40.00th=[ 3392], 50.00th=[ 3982], 60.00th=[ 4948], | 70.00th=[ 7570], 80.00th=[10028], 90.00th=[12518], 95.00th=[15533], | 99.00th=[22938], 99.50th=[26084], 99.90th=[29492], 99.95th=[29492], | 99.99th=[29492] bw ( KiB/s): min= 672, max= 1088, per=24.50%, avg=909.33, stdev=132.61, samples=12 iops : min= 42, max= 68, avg=56.83, stdev= 8.29, samples=12 write: IOPS=64, BW=1039KiB/s (1064kB/s)(6544KiB/6298msec) clat (usec): min=2887, max=59248, avg=9924.46, stdev=6422.84 lat (usec): min=2889, max=59251, avg=9926.00, stdev=6422.88 clat percentiles (usec): | 1.00th=[ 3294], 5.00th=[ 3785], 10.00th=[ 4293], 20.00th=[ 5407], | 30.00th=[ 6194], 40.00th=[ 7242], 50.00th=[ 8094], 60.00th=[ 9372], | 70.00th=[11207], 80.00th=[13173], 90.00th=[17957], 95.00th=[21365], | 99.00th=[31065], 99.50th=[40633], 99.90th=[59507], 99.95th=[59507], | 99.99th=[59507] bw ( KiB/s): min= 768, max= 1312, per=26.15%, avg=1021.33, stdev=190.28, samples=12 iops : min= 48, max= 82, avg=63.83, stdev=11.89, samples=12 lat (msec) : 2=0.26%, 4=26.43%, 10=44.79%, 20=24.48%, 50=3.78% lat (msec) : 100=0.26% cpu : usr=0.17%, sys=9.80%, ctx=1121, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=15667: Thu Apr 4 19:22:58 2024 read: IOPS=60, BW=970KiB/s (993kB/s)(6256KiB/6451msec) clat (usec): min=1852, max=56801, avg=6372.17, stdev=5046.91 lat (usec): min=1852, max=56801, avg=6372.92, stdev=5046.90 clat percentiles (usec): | 1.00th=[ 1942], 5.00th=[ 2245], 10.00th=[ 2474], 20.00th=[ 2737], | 30.00th=[ 3064], 40.00th=[ 3654], 50.00th=[ 4490], 60.00th=[ 6128], | 70.00th=[ 7963], 80.00th=[ 9634], 90.00th=[11863], 95.00th=[13829], | 99.00th=[23725], 99.50th=[33817], 99.90th=[56886], 99.95th=[56886], | 99.99th=[56886] bw ( KiB/s): min= 704, max= 1216, per=26.15%, avg=970.67, stdev=190.70, samples=12 iops : min= 44, max= 76, avg=60.67, stdev=11.92, samples=12 write: IOPS=58, BW=935KiB/s (957kB/s)(6032KiB/6451msec) clat (usec): min=2556, max=56919, avg=10473.73, stdev=6489.93 lat (usec): min=2557, max=56921, avg=10475.44, stdev=6489.92 clat percentiles (usec): | 1.00th=[ 3458], 5.00th=[ 3916], 10.00th=[ 4293], 20.00th=[ 5604], | 30.00th=[ 6587], 40.00th=[ 7767], 50.00th=[ 9372], 60.00th=[10421], | 70.00th=[11731], 80.00th=[13829], 90.00th=[16909], 95.00th=[21103], | 99.00th=[38536], 99.50th=[43254], 99.90th=[56886], 99.95th=[56886], | 99.99th=[56886] bw ( KiB/s): min= 672, max= 1088, per=23.55%, avg=920.00, stdev=148.30, samples=12 iops : min= 42, max= 68, avg=57.50, stdev= 9.27, samples=12 lat (msec) : 2=0.52%, 4=25.91%, 10=42.45%, 20=27.21%, 50=3.65% lat (msec) : 100=0.26% cpu : usr=0.12%, sys=10.23%, ctx=1243, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=15668: Thu Apr 4 19:22:58 2024 read: IOPS=58, BW=936KiB/s (958kB/s)(6032KiB/6445msec) clat (usec): min=1957, max=35584, avg=6034.68, stdev=4405.17 lat (usec): min=1957, max=35584, avg=6035.52, stdev=4405.31 clat percentiles (usec): | 1.00th=[ 1991], 5.00th=[ 2245], 10.00th=[ 2474], 20.00th=[ 2737], | 30.00th=[ 2999], 40.00th=[ 3556], 50.00th=[ 4047], 60.00th=[ 5407], | 70.00th=[ 7046], 80.00th=[ 9372], 90.00th=[11600], 95.00th=[14353], | 99.00th=[20055], 99.50th=[26346], 99.90th=[35390], 99.95th=[35390], | 99.99th=[35390] bw ( KiB/s): min= 768, max= 1088, per=24.63%, avg=914.33, stdev=99.72, samples=12 iops : min= 48, max= 68, avg=57.00, stdev= 6.22, samples=12 write: IOPS=60, BW=971KiB/s (994kB/s)(6256KiB/6445msec) clat (usec): min=2707, max=39674, avg=10638.07, stdev=6059.78 lat (usec): min=2715, max=39676, avg=10639.69, stdev=6059.63 clat percentiles (usec): | 1.00th=[ 3261], 5.00th=[ 3851], 10.00th=[ 4555], 20.00th=[ 5866], | 30.00th=[ 7111], 40.00th=[ 8455], 50.00th=[ 9503], 60.00th=[10683], | 70.00th=[12125], 80.00th=[14222], 90.00th=[17433], 95.00th=[20579], | 99.00th=[34341], 99.50th=[36439], 99.90th=[39584], 99.95th=[39584], | 99.99th=[39584] bw ( KiB/s): min= 638, max= 1184, per=24.70%, avg=964.92, stdev=155.10, samples=12 iops : min= 39, max= 74, avg=60.17, stdev= 9.82, samples=12 lat (msec) : 2=0.65%, 4=26.82%, 10=41.28%, 20=27.47%, 50=3.78% cpu : usr=0.14%, sys=9.99%, ctx=1219, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=3713KiB/s (3802kB/s), 912KiB/s-970KiB/s (934kB/s-993kB/s), io=23.4MiB (24.5MB), run=6298-6451msec WRITE: bw=3906KiB/s (4000kB/s), 935KiB/s-1039KiB/s (957kB/s-1064kB/s), io=24.6MiB (25.8MB), run=6298-6451msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=15679: Thu Apr 4 19:23:02 2024 read: IOPS=2, BW=2536KiB/s (2596kB/s)(7168KiB/2827msec) clat (usec): min=478, max=526254, avg=327317.44, stdev=212948.64 lat (usec): min=479, max=526256, avg=327318.87, stdev=212948.86 clat percentiles (usec): | 1.00th=[ 478], 5.00th=[ 478], 10.00th=[ 478], 20.00th=[ 42730], | 30.00th=[396362], 40.00th=[396362], 50.00th=[429917], 60.00th=[442500], | 70.00th=[442500], 80.00th=[455082], 90.00th=[526386], 95.00th=[526386], | 99.00th=[526386], 99.50th=[526386], 99.90th=[526386], 99.95th=[526386], | 99.99th=[526386] bw ( KiB/s): min= 1771, max= 3543, per=30.87%, avg=2236.20, stdev=743.54, samples=5 iops : min= 1, max= 3, avg= 1.80, stdev= 0.84, samples=5 write: IOPS=1, BW=1811KiB/s (1855kB/s)(5120KiB/2827msec) clat (msec): min=24, max=348, avg=105.08, stdev=137.86 lat (msec): min=24, max=348, avg=105.14, stdev=137.86 clat percentiles (msec): | 1.00th=[ 25], 5.00th=[ 25], 10.00th=[ 25], 20.00th=[ 25], | 30.00th=[ 28], 40.00th=[ 28], 50.00th=[ 45], 60.00th=[ 45], | 70.00th=[ 81], 80.00th=[ 81], 90.00th=[ 351], 95.00th=[ 351], | 99.00th=[ 351], 99.50th=[ 351], 99.90th=[ 351], 99.95th=[ 351], | 99.99th=[ 351] bw ( KiB/s): min= 1771, max= 5314, per=29.11%, avg=2952.00, stdev=2045.55, samples=3 iops : min= 1, max= 5, avg= 2.33, stdev= 2.31, samples=3 lat (usec) : 500=8.33% lat (msec) : 50=33.33%, 100=8.33%, 500=41.67%, 750=8.33% cpu : usr=0.04%, sys=18.51%, ctx=1423, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=15680: Thu Apr 4 19:23:02 2024 read: IOPS=0, BW=999KiB/s (1023kB/s)(2048KiB/2051msec) clat (msec): min=38, max=448, avg=243.23, stdev=289.65 lat (msec): min=38, max=448, avg=243.23, stdev=289.65 clat percentiles (msec): | 1.00th=[ 39], 5.00th=[ 39], 10.00th=[ 39], 20.00th=[ 39], | 30.00th=[ 39], 40.00th=[ 39], 50.00th=[ 39], 60.00th=[ 447], | 70.00th=[ 447], 80.00th=[ 447], 90.00th=[ 447], 95.00th=[ 447], | 99.00th=[ 447], 99.50th=[ 447], 99.90th=[ 447], 99.95th=[ 447], | 99.99th=[ 447] bw ( KiB/s): min= 2043, max= 2043, per=28.20%, avg=2043.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=4, BW=4993KiB/s (5113kB/s)(10.0MiB/2051msec) clat (msec): min=23, max=531, avg=156.29, stdev=196.32 lat (msec): min=23, max=531, avg=156.36, stdev=196.31 clat percentiles (msec): | 1.00th=[ 24], 5.00th=[ 24], 10.00th=[ 24], 20.00th=[ 44], | 30.00th=[ 58], 40.00th=[ 62], 50.00th=[ 67], 60.00th=[ 70], | 70.00th=[ 115], 80.00th=[ 115], 90.00th=[ 523], 95.00th=[ 531], | 99.00th=[ 531], 99.50th=[ 531], 99.90th=[ 531], 99.95th=[ 531], | 99.99th=[ 531] bw ( KiB/s): min= 3506, max= 6144, per=47.57%, avg=4825.00, stdev=1865.35, samples=2 iops : min= 3, max= 6, avg= 4.50, stdev= 2.12, samples=2 lat (msec) : 50=25.00%, 100=41.67%, 250=8.33%, 500=8.33%, 750=16.67% cpu : usr=0.05%, sys=16.78%, ctx=392, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=15682: Thu Apr 4 19:23:02 2024 read: IOPS=4, BW=4804KiB/s (4919kB/s)(6144KiB/1279msec) clat (usec): min=425, max=460905, avg=86531.52, stdev=184054.19 lat (usec): min=426, max=460906, avg=86532.84, stdev=184053.85 clat percentiles (usec): | 1.00th=[ 424], 5.00th=[ 424], 10.00th=[ 424], 20.00th=[ 453], | 30.00th=[ 453], 40.00th=[ 660], 50.00th=[ 660], 60.00th=[ 16909], | 70.00th=[ 39584], 80.00th=[ 39584], 90.00th=[459277], 95.00th=[459277], | 99.00th=[459277], 99.50th=[459277], 99.90th=[459277], 99.95th=[459277], | 99.99th=[459277] bw ( KiB/s): min= 4096, max= 4096, per=56.54%, avg=4096.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 write: IOPS=4, BW=4804KiB/s (4919kB/s)(6144KiB/1279msec) clat (msec): min=23, max=578, avg=126.26, stdev=221.84 lat (msec): min=23, max=578, avg=126.30, stdev=221.85 clat percentiles (msec): | 1.00th=[ 24], 5.00th=[ 24], 10.00th=[ 24], 20.00th=[ 30], | 30.00th=[ 30], 40.00th=[ 31], 50.00th=[ 31], 60.00th=[ 46], | 70.00th=[ 51], 80.00th=[ 51], 90.00th=[ 575], 95.00th=[ 575], | 99.00th=[ 575], 99.50th=[ 575], 99.90th=[ 575], 99.95th=[ 575], | 99.99th=[ 575] bw ( KiB/s): min= 4096, max= 4096, per=40.39%, avg=4096.00, stdev= 0.00, samples=2 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=2 lat (usec) : 500=16.67%, 750=8.33% lat (msec) : 20=8.33%, 50=50.00%, 500=8.33%, 750=8.33% cpu : usr=0.00%, sys=18.39%, ctx=348, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=15684: Thu Apr 4 19:23:02 2024 read: IOPS=3, BW=4022KiB/s (4119kB/s)(5120KiB/1273msec) clat (usec): min=468, max=482149, avg=108392.47, stdev=209320.70 lat (usec): min=470, max=482152, avg=108394.21, stdev=209320.98 clat percentiles (usec): | 1.00th=[ 469], 5.00th=[ 469], 10.00th=[ 469], 20.00th=[ 469], | 30.00th=[ 5604], 40.00th=[ 5604], 50.00th=[ 21365], 60.00th=[ 21365], | 70.00th=[ 32375], 80.00th=[ 32375], 90.00th=[480248], 95.00th=[480248], | 99.00th=[480248], 99.50th=[480248], 99.90th=[480248], 99.95th=[480248], | 99.99th=[480248] bw ( KiB/s): min= 2048, max= 2048, per=28.27%, avg=2048.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=5, BW=5631KiB/s (5766kB/s)(7168KiB/1273msec) clat (msec): min=22, max=516, avg=104.01, stdev=182.45 lat (msec): min=22, max=516, avg=104.08, stdev=182.45 clat percentiles (msec): | 1.00th=[ 23], 5.00th=[ 23], 10.00th=[ 23], 20.00th=[ 24], | 30.00th=[ 25], 40.00th=[ 25], 50.00th=[ 30], 60.00th=[ 49], | 70.00th=[ 49], 80.00th=[ 63], 90.00th=[ 518], 95.00th=[ 518], | 99.00th=[ 518], 99.50th=[ 518], 99.90th=[ 518], 99.95th=[ 518], | 99.99th=[ 518] bw ( KiB/s): min= 2043, max= 6144, per=40.36%, avg=4093.50, stdev=2899.84, samples=2 iops : min= 1, max= 6, avg= 3.50, stdev= 3.54, samples=2 lat (usec) : 500=8.33% lat (msec) : 10=8.33%, 50=58.33%, 100=8.33%, 500=8.33%, 750=8.33% cpu : usr=0.00%, sys=22.01%, ctx=367, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=7244KiB/s (7418kB/s), 999KiB/s-4804KiB/s (1023kB/s-4919kB/s), io=20.0MiB (20.0MB), run=1273-2827msec WRITE: bw=9.90MiB/s (10.4MB/s), 1811KiB/s-5631KiB/s (1855kB/s-5766kB/s), io=28.0MiB (29.4MB), run=1273-2827msec rand-rw: (groupid=0, jobs=1): err= 0: pid=15681: Thu Apr 4 19:23:02 2024 read: IOPS=2, BW=2487KiB/s (2547kB/s)(7168KiB/2882msec) clat (msec): min=25, max=565, avg=275.13, stdev=220.65 lat (msec): min=25, max=565, avg=275.13, stdev=220.65 clat percentiles (msec): | 1.00th=[ 26], 5.00th=[ 26], 10.00th=[ 26], 20.00th=[ 43], | 30.00th=[ 83], 40.00th=[ 83], 50.00th=[ 347], 60.00th=[ 414], | 70.00th=[ 414], 80.00th=[ 451], 90.00th=[ 567], 95.00th=[ 567], | 99.00th=[ 567], 99.50th=[ 567], 99.90th=[ 567], 99.95th=[ 567], | 99.99th=[ 567] bw ( KiB/s): min= 1765, max= 4096, per=33.79%, avg=2401.00, stdev=955.43, samples=5 iops : min= 1, max= 4, avg= 2.20, stdev= 1.10, samples=5 write: IOPS=1, BW=1777KiB/s (1819kB/s)(5120KiB/2882msec) clat (msec): min=30, max=425, avg=190.69, stdev=192.34 lat (msec): min=31, max=425, avg=190.77, stdev=192.31 clat percentiles (msec): | 1.00th=[ 32], 5.00th=[ 32], 10.00th=[ 32], 20.00th=[ 32], | 30.00th=[ 53], 40.00th=[ 53], 50.00th=[ 70], 60.00th=[ 70], | 70.00th=[ 376], 80.00th=[ 376], 90.00th=[ 426], 95.00th=[ 426], | 99.00th=[ 426], 99.50th=[ 426], 99.90th=[ 426], 99.95th=[ 426], | 99.99th=[ 426] bw ( KiB/s): min= 1765, max= 4096, per=26.50%, avg=2636.33, stdev=1272.00, samples=3 iops : min= 1, max= 4, avg= 2.33, stdev= 1.53, samples=3 lat (msec) : 50=25.00%, 100=25.00%, 500=41.67%, 750=8.33% cpu : usr=0.07%, sys=5.48%, ctx=62, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=15683: Thu Apr 4 19:23:02 2024 read: IOPS=0, BW=979KiB/s (1002kB/s)(2048KiB/2092msec) clat (msec): min=71, max=410, avg=241.30, stdev=239.57 lat (msec): min=71, max=410, avg=241.30, stdev=239.57 clat percentiles (msec): | 1.00th=[ 72], 5.00th=[ 72], 10.00th=[ 72], 20.00th=[ 72], | 30.00th=[ 72], 40.00th=[ 72], 50.00th=[ 72], 60.00th=[ 409], | 70.00th=[ 409], 80.00th=[ 409], 90.00th=[ 409], 95.00th=[ 409], | 99.00th=[ 409], 99.50th=[ 409], 99.90th=[ 409], 99.95th=[ 409], | 99.99th=[ 409] bw ( KiB/s): min= 2039, max= 2048, per=28.76%, avg=2043.50, stdev= 6.36, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 write: IOPS=4, BW=4895KiB/s (5012kB/s)(10.0MiB/2092msec) clat (msec): min=37, max=578, avg=160.49, stdev=209.81 lat (msec): min=37, max=578, avg=160.57, stdev=209.80 clat percentiles (msec): | 1.00th=[ 37], 5.00th=[ 37], 10.00th=[ 37], 20.00th=[ 45], | 30.00th=[ 54], 40.00th=[ 61], 50.00th=[ 64], 60.00th=[ 66], | 70.00th=[ 96], 80.00th=[ 96], 90.00th=[ 535], 95.00th=[ 575], | 99.00th=[ 575], 99.50th=[ 575], 99.90th=[ 575], 99.95th=[ 575], | 99.99th=[ 575] bw ( KiB/s): min= 4096, max= 6144, per=54.82%, avg=5453.00, stdev=1175.26, samples=3 iops : min= 4, max= 6, avg= 5.00, stdev= 1.00, samples=3 lat (msec) : 50=16.67%, 100=58.33%, 500=8.33%, 750=16.67% cpu : usr=0.10%, sys=8.66%, ctx=190, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=15685: Thu Apr 4 19:23:02 2024 read: IOPS=3, BW=3146KiB/s (3221kB/s)(6144KiB/1953msec) clat (msec): min=22, max=526, avg=113.33, stdev=202.69 lat (msec): min=22, max=526, avg=113.33, stdev=202.69 clat percentiles (msec): | 1.00th=[ 23], 5.00th=[ 23], 10.00th=[ 23], 20.00th=[ 25], | 30.00th=[ 25], 40.00th=[ 25], 50.00th=[ 25], 60.00th=[ 27], | 70.00th=[ 56], 80.00th=[ 56], 90.00th=[ 527], 95.00th=[ 527], | 99.00th=[ 527], 99.50th=[ 527], 99.90th=[ 527], 99.95th=[ 527], | 99.99th=[ 527] bw ( KiB/s): min= 4096, max= 4096, per=57.64%, avg=4096.00, stdev= 0.00, samples=2 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=2 write: IOPS=3, BW=3146KiB/s (3221kB/s)(6144KiB/1953msec) clat (msec): min=26, max=588, avg=209.77, stdev=233.19 lat (msec): min=26, max=588, avg=209.85, stdev=233.19 clat percentiles (msec): | 1.00th=[ 27], 5.00th=[ 27], 10.00th=[ 27], 20.00th=[ 46], | 30.00th=[ 46], 40.00th=[ 79], 50.00th=[ 79], 60.00th=[ 109], | 70.00th=[ 409], 80.00th=[ 409], 90.00th=[ 592], 95.00th=[ 592], | 99.00th=[ 592], 99.50th=[ 592], 99.90th=[ 592], 99.95th=[ 592], | 99.99th=[ 592] bw ( KiB/s): min= 4096, max= 4096, per=41.17%, avg=4096.00, stdev= 0.00, samples=2 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=2 lat (msec) : 50=50.00%, 100=16.67%, 250=8.33%, 500=8.33%, 750=16.67% cpu : usr=0.00%, sys=7.94%, ctx=113, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=15686: Thu Apr 4 19:23:02 2024 read: IOPS=2, BW=2475KiB/s (2534kB/s)(5120KiB/2069msec) clat (msec): min=25, max=426, avg=113.47, stdev=175.10 lat (msec): min=25, max=426, avg=113.47, stdev=175.10 clat percentiles (msec): | 1.00th=[ 26], 5.00th=[ 26], 10.00th=[ 26], 20.00th=[ 26], | 30.00th=[ 35], 40.00th=[ 35], 50.00th=[ 41], 60.00th=[ 41], | 70.00th=[ 42], 80.00th=[ 42], 90.00th=[ 426], 95.00th=[ 426], | 99.00th=[ 426], 99.50th=[ 426], 99.90th=[ 426], 99.95th=[ 426], | 99.99th=[ 426] bw ( KiB/s): min= 4087, max= 4096, per=57.58%, avg=4091.50, stdev= 6.36, samples=2 iops : min= 3, max= 4, avg= 3.50, stdev= 0.71, samples=2 write: IOPS=3, BW=3464KiB/s (3548kB/s)(7168KiB/2069msec) clat (msec): min=30, max=600, avg=214.18, stdev=256.87 lat (msec): min=30, max=600, avg=214.25, stdev=256.87 clat percentiles (msec): | 1.00th=[ 31], 5.00th=[ 31], 10.00th=[ 31], 20.00th=[ 32], | 30.00th=[ 70], 40.00th=[ 70], 50.00th=[ 72], 60.00th=[ 123], | 70.00th=[ 123], 80.00th=[ 575], 90.00th=[ 600], 95.00th=[ 600], | 99.00th=[ 600], 99.50th=[ 600], 99.90th=[ 600], 99.95th=[ 600], | 99.99th=[ 600] bw ( KiB/s): min= 2048, max= 4096, per=30.86%, avg=3069.75, stdev=1179.82, samples=4 iops : min= 2, max= 4, avg= 2.75, stdev= 0.96, samples=4 lat (msec) : 50=50.00%, 100=16.67%, 250=8.33%, 500=8.33%, 750=16.67% cpu : usr=0.05%, sys=7.83%, ctx=66, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=7106KiB/s (7277kB/s), 979KiB/s-3146KiB/s (1002kB/s-3221kB/s), io=20.0MiB (20.0MB), run=1953-2882msec WRITE: bw=9949KiB/s (10.2MB/s), 1777KiB/s-4895KiB/s (1819kB/s-5012kB/s), io=28.0MiB (29.4MB), run=1953-2882msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=15694: Thu Apr 4 19:23:05 2024 read: IOPS=1, BW=7119KiB/s (7290kB/s)(12.0MiB/1726msec) clat (msec): min=184, max=1335, avg=574.22, stdev=659.49 lat (msec): min=184, max=1335, avg=574.23, stdev=659.49 clat percentiles (msec): | 1.00th=[ 186], 5.00th=[ 186], 10.00th=[ 186], 20.00th=[ 186], | 30.00th=[ 186], 40.00th=[ 203], 50.00th=[ 203], 60.00th=[ 203], | 70.00th=[ 1334], 80.00th=[ 1334], 90.00th=[ 1334], 95.00th=[ 1334], | 99.00th=[ 1334], 99.50th=[ 1334], 99.90th=[ 1334], 99.95th=[ 1334], | 99.99th=[ 1334] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67% cpu : usr=0.00%, sys=61.68%, ctx=539, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=15695: Thu Apr 4 19:23:05 2024 read: IOPS=0, BW=1712KiB/s (1753kB/s)(4096KiB/2393msec) clat (nsec): min=161286k, max=161286k, avg=161285502.00, stdev= 0.00 lat (nsec): min=161287k, max=161287k, avg=161286622.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 161], 5.00th=[ 161], 10.00th=[ 161], 20.00th=[ 161], | 30.00th=[ 161], 40.00th=[ 161], 50.00th=[ 161], 60.00th=[ 161], | 70.00th=[ 161], 80.00th=[ 161], 90.00th=[ 161], 95.00th=[ 161], | 99.00th=[ 161], 99.50th=[ 161], 99.90th=[ 161], 99.95th=[ 161], | 99.99th=[ 161] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=0, BW=3423KiB/s (3505kB/s)(8192KiB/2393msec) clat (msec): min=223, max=2006, avg=1114.65, stdev=1260.57 lat (msec): min=223, max=2006, avg=1115.00, stdev=1260.53 clat percentiles (msec): | 1.00th=[ 224], 5.00th=[ 224], 10.00th=[ 224], 20.00th=[ 224], | 30.00th=[ 224], 40.00th=[ 224], 50.00th=[ 224], 60.00th=[ 2005], | 70.00th=[ 2005], 80.00th=[ 2005], 90.00th=[ 2005], 95.00th=[ 2005], | 99.00th=[ 2005], 99.50th=[ 2005], 99.90th=[ 2005], 99.95th=[ 2005], | 99.99th=[ 2005] bw ( KiB/s): min= 8192, max= 8192, per=59.83%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67% cpu : usr=0.00%, sys=10.62%, ctx=115, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=15696: Thu Apr 4 19:23:05 2024 write: IOPS=1, BW=6314KiB/s (6466kB/s)(12.0MiB/1946msec) clat (msec): min=133, max=1551, avg=647.32, stdev=785.41 lat (msec): min=133, max=1551, avg=647.57, stdev=785.35 clat percentiles (msec): | 1.00th=[ 134], 5.00th=[ 134], 10.00th=[ 134], 20.00th=[ 134], | 30.00th=[ 134], 40.00th=[ 257], 50.00th=[ 257], 60.00th=[ 257], | 70.00th=[ 1552], 80.00th=[ 1552], 90.00th=[ 1552], 95.00th=[ 1552], | 99.00th=[ 1552], 99.50th=[ 1552], 99.90th=[ 1552], 99.95th=[ 1552], | 99.99th=[ 1552] bw ( KiB/s): min= 8192, max= 8192, per=59.83%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=33.33% cpu : usr=0.15%, sys=15.31%, ctx=35, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=15697: Thu Apr 4 19:23:05 2024 write: IOPS=1, BW=6344KiB/s (6496kB/s)(12.0MiB/1937msec) clat (msec): min=99, max=1569, avg=639.42, stdev=809.28 lat (msec): min=99, max=1570, avg=639.76, stdev=809.22 clat percentiles (msec): | 1.00th=[ 100], 5.00th=[ 100], 10.00th=[ 100], 20.00th=[ 100], | 30.00th=[ 100], 40.00th=[ 249], 50.00th=[ 249], 60.00th=[ 249], | 70.00th=[ 1569], 80.00th=[ 1569], 90.00th=[ 1569], 95.00th=[ 1569], | 99.00th=[ 1569], 99.50th=[ 1569], 99.90th=[ 1569], 99.95th=[ 1569], | 99.99th=[ 1569] bw ( KiB/s): min= 8192, max= 8192, per=59.83%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 250=33.33% cpu : usr=0.00%, sys=14.10%, ctx=43, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=6847KiB/s (7011kB/s), 1712KiB/s-7119KiB/s (1753kB/s-7290kB/s), io=16.0MiB (16.8MB), run=1726-2393msec WRITE: bw=13.4MiB/s (14.0MB/s), 3423KiB/s-6344KiB/s (3505kB/s-6496kB/s), io=32.0MiB (33.6MB), run=1937-2393msec rand-rw: (groupid=0, jobs=1): err= 0: pid=15698: Thu Apr 4 19:23:05 2024 read: IOPS=1, BW=5191KiB/s (5316kB/s)(12.0MiB/2367msec) clat (msec): min=115, max=1880, avg=788.52, stdev=953.80 lat (msec): min=115, max=1880, avg=788.52, stdev=953.80 clat percentiles (msec): | 1.00th=[ 116], 5.00th=[ 116], 10.00th=[ 116], 20.00th=[ 116], | 30.00th=[ 116], 40.00th=[ 372], 50.00th=[ 372], 60.00th=[ 372], | 70.00th=[ 1888], 80.00th=[ 1888], 90.00th=[ 1888], 95.00th=[ 1888], | 99.00th=[ 1888], 99.50th=[ 1888], 99.90th=[ 1888], 99.95th=[ 1888], | 99.99th=[ 1888] bw ( KiB/s): min= 7801, max= 7801, per=100.00%, avg=7801.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=33.33% cpu : usr=0.00%, sys=4.99%, ctx=30, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=15699: Thu Apr 4 19:23:05 2024 read: IOPS=0, BW=1646KiB/s (1686kB/s)(4096KiB/2488msec) clat (nsec): min=393572k, max=393572k, avg=393571827.00, stdev= 0.00 lat (nsec): min=393574k, max=393574k, avg=393574387.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 393], 5.00th=[ 393], 10.00th=[ 393], 20.00th=[ 393], | 30.00th=[ 393], 40.00th=[ 393], 50.00th=[ 393], 60.00th=[ 393], | 70.00th=[ 393], 80.00th=[ 393], 90.00th=[ 393], 95.00th=[ 393], | 99.00th=[ 393], 99.50th=[ 393], 99.90th=[ 393], 99.95th=[ 393], | 99.99th=[ 393] bw ( KiB/s): min= 7907, max= 7907, per=100.00%, avg=7907.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=0, BW=3293KiB/s (3372kB/s)(8192KiB/2488msec) clat (msec): min=119, max=1958, avg=1039.01, stdev=1299.67 lat (msec): min=120, max=1958, avg=1039.36, stdev=1299.65 clat percentiles (msec): | 1.00th=[ 121], 5.00th=[ 121], 10.00th=[ 121], 20.00th=[ 121], | 30.00th=[ 121], 40.00th=[ 121], 50.00th=[ 121], 60.00th=[ 1955], | 70.00th=[ 1955], 80.00th=[ 1955], 90.00th=[ 1955], 95.00th=[ 1955], | 99.00th=[ 1955], 99.50th=[ 1955], 99.90th=[ 1955], 99.95th=[ 1955], | 99.99th=[ 1955] bw ( KiB/s): min= 7907, max= 7907, per=60.04%, avg=7907.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=33.33% cpu : usr=0.04%, sys=6.80%, ctx=31, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=15700: Thu Apr 4 19:23:05 2024 write: IOPS=1, BW=5357KiB/s (5485kB/s)(12.0MiB/2294msec) clat (msec): min=99, max=1863, avg=761.51, stdev=960.50 lat (msec): min=100, max=1863, avg=761.83, stdev=960.46 clat percentiles (msec): | 1.00th=[ 101], 5.00th=[ 101], 10.00th=[ 101], 20.00th=[ 101], | 30.00th=[ 101], 40.00th=[ 321], 50.00th=[ 321], 60.00th=[ 321], | 70.00th=[ 1871], 80.00th=[ 1871], 90.00th=[ 1871], 95.00th=[ 1871], | 99.00th=[ 1871], 99.50th=[ 1871], 99.90th=[ 1871], 99.95th=[ 1871], | 99.99th=[ 1871] bw ( KiB/s): min= 8192, max= 8192, per=62.20%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 500=33.33% cpu : usr=0.00%, sys=9.16%, ctx=71, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=15701: Thu Apr 4 19:23:05 2024 write: IOPS=6, BW=25.8MiB/s (27.0MB/s)(12.0MiB/466msec) clat (msec): min=103, max=217, avg=150.67, stdev=59.19 lat (msec): min=104, max=217, avg=151.00, stdev=59.24 clat percentiles (msec): | 1.00th=[ 105], 5.00th=[ 105], 10.00th=[ 105], 20.00th=[ 105], | 30.00th=[ 105], 40.00th=[ 131], 50.00th=[ 131], 60.00th=[ 131], | 70.00th=[ 218], 80.00th=[ 218], 90.00th=[ 218], 95.00th=[ 218], | 99.00th=[ 218], 99.50th=[ 218], 99.90th=[ 218], 99.95th=[ 218], | 99.99th=[ 218] lat (msec) : 250=100.00% cpu : usr=0.22%, sys=23.01%, ctx=37, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=6585KiB/s (6743kB/s), 1646KiB/s-5191KiB/s (1686kB/s-5316kB/s), io=16.0MiB (16.8MB), run=2367-2488msec WRITE: bw=12.9MiB/s (13.5MB/s), 3293KiB/s-25.8MiB/s (3372kB/s-27.0MB/s), io=32.0MiB (33.6MB), run=466-2488msec