== sanity test 398b: DIO and buffer IO race ============== 16:26:48 (1709069208) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 2.09182 s, 24.1 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=1235: Tue Feb 27 16:27:30 2024 read: IOPS=40, BW=161KiB/s (165kB/s)(5992KiB/37246msec) clat (usec): min=55, max=848511, avg=14014.49, stdev=24407.12 lat (usec): min=56, max=848512, avg=14016.11, stdev=24407.46 clat percentiles (usec): | 1.00th=[ 80], 5.00th=[ 102], 10.00th=[ 4178], 20.00th=[ 6456], | 30.00th=[ 8291], 40.00th=[ 10159], 50.00th=[ 11600], 60.00th=[ 13566], | 70.00th=[ 15926], 80.00th=[ 19006], 90.00th=[ 25035], 95.00th=[ 30540], | 99.00th=[ 43779], 99.50th=[ 53740], 99.90th=[283116], 99.95th=[851444], | 99.99th=[851444] bw ( KiB/s): min= 55, max= 264, per=25.08%, avg=161.51, stdev=50.09, samples=73 iops : min= 13, max= 66, avg=40.05, stdev=12.60, samples=73 write: IOPS=42, BW=169KiB/s (173kB/s)(6296KiB/37246msec) clat (usec): min=478, max=79087, avg=10244.23, stdev=11718.20 lat (usec): min=478, max=79089, avg=10250.40, stdev=11716.22 clat percentiles (usec): | 1.00th=[ 562], 5.00th=[ 635], 10.00th=[ 676], 20.00th=[ 783], | 30.00th=[ 1336], 40.00th=[ 3785], 50.00th=[ 7046], 60.00th=[ 9765], | 70.00th=[12780], 80.00th=[17171], 90.00th=[25822], 95.00th=[33817], | 99.00th=[56886], 99.50th=[64750], 99.90th=[78119], 99.95th=[79168], | 99.99th=[79168] bw ( KiB/s): min= 48, max= 320, per=25.14%, avg=169.47, stdev=52.69, samples=73 iops : min= 12, max= 80, avg=42.05, stdev=13.17, samples=73 lat (usec) : 100=2.15%, 250=1.63%, 500=0.16%, 750=8.89%, 1000=4.69% lat (msec) : 2=4.13%, 4=3.81%, 10=24.97%, 20=32.94%, 50=15.56% lat (msec) : 100=1.01%, 500=0.03%, 1000=0.03% cpu : usr=0.26%, sys=10.78%, ctx=4870, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1237: Tue Feb 27 16:27:30 2024 read: IOPS=41, BW=164KiB/s (168kB/s)(5940KiB/36149msec) clat (usec): min=65, max=75652, avg=13624.85, stdev=9585.77 lat (usec): min=66, max=75653, avg=13628.36, stdev=9585.79 clat percentiles (usec): | 1.00th=[ 79], 5.00th=[ 106], 10.00th=[ 4555], 20.00th=[ 6587], | 30.00th=[ 8160], 40.00th=[ 9765], 50.00th=[11207], 60.00th=[13829], | 70.00th=[16450], 80.00th=[20579], 90.00th=[25560], 95.00th=[31065], | 99.00th=[44827], 99.50th=[51643], 99.90th=[72877], 99.95th=[76022], | 99.99th=[76022] bw ( KiB/s): min= 55, max= 287, per=25.42%, avg=163.71, stdev=49.44, samples=72 iops : min= 13, max= 71, avg=40.54, stdev=12.45, samples=72 write: IOPS=43, BW=176KiB/s (180kB/s)(6348KiB/36149msec) clat (usec): min=489, max=277601, avg=9962.05, stdev=13071.10 lat (usec): min=490, max=277604, avg=9964.02, stdev=13070.72 clat percentiles (usec): | 1.00th=[ 545], 5.00th=[ 627], 10.00th=[ 685], 20.00th=[ 791], | 30.00th=[ 1319], 40.00th=[ 3032], 50.00th=[ 6587], 60.00th=[ 9372], | 70.00th=[ 12387], 80.00th=[ 16909], 90.00th=[ 23987], 95.00th=[ 32637], | 99.00th=[ 50594], 99.50th=[ 55837], 99.90th=[120062], 99.95th=[278922], | 99.99th=[278922] bw ( KiB/s): min= 79, max= 376, per=25.89%, avg=174.47, stdev=54.18, samples=72 iops : min= 19, max= 94, avg=43.24, stdev=13.60, samples=72 lat (usec) : 100=2.15%, 250=1.17%, 500=0.07%, 750=8.92%, 1000=4.65% lat (msec) : 2=4.72%, 4=4.39%, 10=26.50%, 20=30.01%, 50=16.57% lat (msec) : 100=0.78%, 250=0.03%, 500=0.03% cpu : usr=0.13%, sys=10.92%, ctx=4807, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1239: Tue Feb 27 16:27:30 2024 read: IOPS=40, BW=162KiB/s (166kB/s)(6028KiB/37197msec) clat (usec): min=57, max=85793, avg=13285.52, stdev=9722.87 lat (usec): min=58, max=85795, avg=13287.36, stdev=9722.64 clat percentiles (usec): | 1.00th=[ 77], 5.00th=[ 101], 10.00th=[ 4228], 20.00th=[ 6456], | 30.00th=[ 8094], 40.00th=[ 9765], 50.00th=[11469], 60.00th=[13435], | 70.00th=[15795], 80.00th=[19006], 90.00th=[24511], 95.00th=[30540], | 99.00th=[48497], 99.50th=[62129], 99.90th=[78119], 99.95th=[85459], | 99.99th=[85459] bw ( KiB/s): min= 72, max= 271, per=25.02%, avg=161.11, stdev=45.31, samples=74 iops : min= 18, max= 67, avg=39.92, stdev=11.37, samples=74 write: IOPS=42, BW=168KiB/s (172kB/s)(6260KiB/37197msec) clat (usec): min=506, max=219925, avg=10885.69, stdev=13966.45 lat (usec): min=508, max=219926, avg=10887.20, stdev=13966.56 clat percentiles (usec): | 1.00th=[ 570], 5.00th=[ 635], 10.00th=[ 685], 20.00th=[ 775], | 30.00th=[ 1205], 40.00th=[ 2638], 50.00th=[ 6390], 60.00th=[ 9634], | 70.00th=[ 13304], 80.00th=[ 19006], 90.00th=[ 28181], 95.00th=[ 36439], | 99.00th=[ 56886], 99.50th=[ 66847], 99.90th=[120062], 99.95th=[219153], | 99.99th=[219153] bw ( KiB/s): min= 71, max= 312, per=24.78%, avg=167.00, stdev=52.43, samples=74 iops : min= 17, max= 78, avg=41.39, stdev=13.17, samples=74 lat (usec) : 100=2.41%, 250=1.50%, 500=0.03%, 750=9.60%, 1000=4.33% lat (msec) : 2=5.14%, 4=4.10%, 10=24.06%, 20=30.79%, 50=16.73% lat (msec) : 100=1.24%, 250=0.07% cpu : usr=0.14%, sys=10.76%, ctx=4746, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1242: Tue Feb 27 16:27:30 2024 read: IOPS=41, BW=167KiB/s (171kB/s)(6052KiB/36163msec) clat (usec): min=50, max=113499, avg=13515.52, stdev=10250.13 lat (usec): min=51, max=113500, avg=13517.06, stdev=10249.73 clat percentiles (usec): | 1.00th=[ 83], 5.00th=[ 106], 10.00th=[ 3982], 20.00th=[ 6325], | 30.00th=[ 7963], 40.00th=[ 9634], 50.00th=[ 11469], 60.00th=[ 13566], | 70.00th=[ 15926], 80.00th=[ 19530], 90.00th=[ 25560], 95.00th=[ 30540], | 99.00th=[ 45876], 99.50th=[ 63701], 99.90th=[ 91751], 99.95th=[113771], | 99.99th=[113771] bw ( KiB/s): min= 56, max= 335, per=25.87%, avg=166.60, stdev=51.53, samples=72 iops : min= 14, max= 83, avg=41.22, stdev=12.90, samples=72 write: IOPS=43, BW=172KiB/s (177kB/s)(6236KiB/36163msec) clat (usec): min=488, max=233406, avg=9986.41, stdev=12801.00 lat (usec): min=489, max=233408, avg=9988.35, stdev=12801.09 clat percentiles (usec): | 1.00th=[ 562], 5.00th=[ 635], 10.00th=[ 676], 20.00th=[ 783], | 30.00th=[ 1172], 40.00th=[ 2474], 50.00th=[ 5932], 60.00th=[ 8848], | 70.00th=[ 12387], 80.00th=[ 17171], 90.00th=[ 26608], 95.00th=[ 33817], | 99.00th=[ 50070], 99.50th=[ 56886], 99.90th=[ 76022], 99.95th=[233833], | 99.99th=[233833] bw ( KiB/s): min= 56, max= 303, per=25.44%, avg=171.49, stdev=51.98, samples=72 iops : min= 14, max= 75, avg=42.44, stdev=12.97, samples=72 lat (usec) : 100=1.89%, 250=1.92%, 500=0.23%, 750=8.98%, 1000=5.31% lat (msec) : 2=4.49%, 4=4.56%, 10=25.26%, 20=29.92%, 50=16.50% lat (msec) : 100=0.88%, 250=0.07% cpu : usr=0.19%, sys=10.75%, ctx=4857, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=645KiB/s (660kB/s), 161KiB/s-167KiB/s (165kB/s-171kB/s), io=23.4MiB (24.6MB), run=36149-37246msec WRITE: bw=675KiB/s (691kB/s), 168KiB/s-176KiB/s (172kB/s-180kB/s), io=24.6MiB (25.7MB), run=36149-37246msec rand-rw: (groupid=0, jobs=1): err= 0: pid=1236: Tue Feb 27 16:27:48 2024 read: IOPS=27, BW=108KiB/s (111kB/s)(5992KiB/55391msec) clat (msec): min=2, max=283, avg=13.49, stdev=14.05 lat (msec): min=2, max=283, avg=13.50, stdev=14.05 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 4], 10.00th=[ 5], 20.00th=[ 6], | 30.00th=[ 7], 40.00th=[ 8], 50.00th=[ 10], 60.00th=[ 11], | 70.00th=[ 14], 80.00th=[ 18], 90.00th=[ 29], 95.00th=[ 39], | 99.00th=[ 69], 99.50th=[ 77], 99.90th=[ 102], 99.95th=[ 284], | 99.99th=[ 284] bw ( KiB/s): min= 23, max= 288, per=24.83%, avg=107.27, stdev=58.04, samples=110 iops : min= 5, max= 72, avg=26.45, stdev=14.55, samples=110 write: IOPS=28, BW=114KiB/s (116kB/s)(6296KiB/55391msec) clat (msec): min=4, max=135, avg=22.27, stdev=14.73 lat (msec): min=4, max=135, avg=22.27, stdev=14.74 clat percentiles (msec): | 1.00th=[ 6], 5.00th=[ 9], 10.00th=[ 10], 20.00th=[ 12], | 30.00th=[ 14], 40.00th=[ 16], 50.00th=[ 18], 60.00th=[ 22], | 70.00th=[ 26], 80.00th=[ 31], 90.00th=[ 40], 95.00th=[ 52], | 99.00th=[ 73], 99.50th=[ 94], 99.90th=[ 130], 99.95th=[ 136], | 99.99th=[ 136] bw ( KiB/s): min= 23, max= 320, per=24.90%, avg=112.53, stdev=57.93, samples=110 iops : min= 5, max= 80, avg=27.76, stdev=14.54, samples=110 lat (msec) : 4=2.86%, 10=30.70%, 20=36.17%, 50=26.43%, 100=3.58% lat (msec) : 250=0.23%, 500=0.03% cpu : usr=0.12%, sys=5.14%, ctx=3428, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1238: Tue Feb 27 16:27:48 2024 read: IOPS=26, BW=107KiB/s (110kB/s)(5940KiB/55408msec) clat (msec): min=2, max=225, avg=13.30, stdev=12.90 lat (msec): min=2, max=225, avg=13.30, stdev=12.90 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 4], 10.00th=[ 5], 20.00th=[ 6], | 30.00th=[ 7], 40.00th=[ 8], 50.00th=[ 9], 60.00th=[ 12], | 70.00th=[ 14], 80.00th=[ 18], 90.00th=[ 28], 95.00th=[ 37], | 99.00th=[ 64], 99.50th=[ 75], 99.90th=[ 88], 99.95th=[ 226], | 99.99th=[ 226] bw ( KiB/s): min= 15, max= 351, per=24.63%, avg=106.41, stdev=61.76, samples=110 iops : min= 3, max= 87, avg=26.28, stdev=15.47, samples=110 write: IOPS=28, BW=115KiB/s (117kB/s)(6348KiB/55408msec) clat (msec): min=4, max=148, avg=22.40, stdev=15.34 lat (msec): min=4, max=148, avg=22.40, stdev=15.34 clat percentiles (msec): | 1.00th=[ 6], 5.00th=[ 8], 10.00th=[ 10], 20.00th=[ 12], | 30.00th=[ 14], 40.00th=[ 16], 50.00th=[ 18], 60.00th=[ 21], | 70.00th=[ 25], 80.00th=[ 31], 90.00th=[ 42], 95.00th=[ 53], | 99.00th=[ 80], 99.50th=[ 97], 99.90th=[ 136], 99.95th=[ 148], | 99.99th=[ 148] bw ( KiB/s): min= 32, max= 328, per=25.07%, avg=113.32, stdev=58.44, samples=110 iops : min= 8, max= 82, avg=28.01, stdev=14.63, samples=110 lat (msec) : 4=3.22%, 10=30.27%, 20=36.13%, 50=26.17%, 100=4.00% lat (msec) : 250=0.20% cpu : usr=0.11%, sys=5.01%, ctx=3440, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1240: Tue Feb 27 16:27:48 2024 read: IOPS=27, BW=109KiB/s (111kB/s)(6028KiB/55512msec) clat (usec): min=2216, max=95163, avg=13398.39, stdev=12104.42 lat (usec): min=2216, max=95163, avg=13399.77, stdev=12104.36 clat percentiles (usec): | 1.00th=[ 3163], 5.00th=[ 3884], 10.00th=[ 4555], 20.00th=[ 5604], | 30.00th=[ 6652], 40.00th=[ 7898], 50.00th=[ 9241], 60.00th=[11207], | 70.00th=[13566], 80.00th=[17957], 90.00th=[28181], 95.00th=[39060], | 99.00th=[63177], 99.50th=[72877], 99.90th=[92799], 99.95th=[94897], | 99.99th=[94897] bw ( KiB/s): min= 24, max= 256, per=25.27%, avg=109.17, stdev=54.63, samples=110 iops : min= 6, max= 64, avg=26.98, stdev=13.70, samples=110 write: IOPS=28, BW=113KiB/s (115kB/s)(6260KiB/55512msec) clat (msec): min=3, max=632, avg=22.48, stdev=22.01 lat (msec): min=3, max=632, avg=22.48, stdev=22.01 clat percentiles (msec): | 1.00th=[ 6], 5.00th=[ 9], 10.00th=[ 10], 20.00th=[ 12], | 30.00th=[ 13], 40.00th=[ 15], 50.00th=[ 18], 60.00th=[ 21], | 70.00th=[ 25], 80.00th=[ 31], 90.00th=[ 42], 95.00th=[ 52], | 99.00th=[ 78], 99.50th=[ 91], 99.90th=[ 220], 99.95th=[ 634], | 99.99th=[ 634] bw ( KiB/s): min= 32, max= 344, per=25.07%, avg=113.33, stdev=60.84, samples=110 iops : min= 8, max= 86, avg=28.02, stdev=15.24, samples=110 lat (msec) : 4=2.83%, 10=31.05%, 20=36.78%, 50=25.29%, 100=3.81% lat (msec) : 250=0.20%, 750=0.03% cpu : usr=0.07%, sys=5.03%, ctx=3433, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1241: Tue Feb 27 16:27:48 2024 read: IOPS=27, BW=109KiB/s (112kB/s)(6052KiB/55566msec) clat (usec): min=2511, max=99863, avg=13847.81, stdev=12728.36 lat (usec): min=2512, max=99865, avg=13849.14, stdev=12728.38 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 4], 10.00th=[ 5], 20.00th=[ 6], | 30.00th=[ 7], 40.00th=[ 8], 50.00th=[ 10], 60.00th=[ 12], | 70.00th=[ 14], 80.00th=[ 20], 90.00th=[ 30], 95.00th=[ 41], | 99.00th=[ 64], 99.50th=[ 70], 99.90th=[ 97], 99.95th=[ 101], | 99.99th=[ 101] bw ( KiB/s): min= 31, max= 320, per=25.11%, avg=108.49, stdev=60.03, samples=111 iops : min= 7, max= 80, avg=26.82, stdev=15.09, samples=111 write: IOPS=28, BW=112KiB/s (115kB/s)(6236KiB/55566msec) clat (msec): min=3, max=233, avg=22.13, stdev=15.45 lat (msec): min=3, max=233, avg=22.13, stdev=15.45 clat percentiles (msec): | 1.00th=[ 6], 5.00th=[ 8], 10.00th=[ 10], 20.00th=[ 12], | 30.00th=[ 13], 40.00th=[ 15], 50.00th=[ 18], 60.00th=[ 21], | 70.00th=[ 25], 80.00th=[ 32], 90.00th=[ 42], 95.00th=[ 52], | 99.00th=[ 75], 99.50th=[ 83], 99.90th=[ 132], 99.95th=[ 234], | 99.99th=[ 234] bw ( KiB/s): min= 31, max= 336, per=24.69%, avg=111.59, stdev=58.79, samples=111 iops : min= 7, max= 84, avg=27.59, stdev=14.77, samples=111 lat (msec) : 4=3.03%, 10=31.15%, 20=34.90%, 50=26.92%, 100=3.91% lat (msec) : 250=0.10% cpu : usr=0.08%, sys=5.13%, ctx=3426, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=432KiB/s (443kB/s), 107KiB/s-109KiB/s (110kB/s-112kB/s), io=23.4MiB (24.6MB), run=55391-55566msec WRITE: bw=452KiB/s (463kB/s), 112KiB/s-115KiB/s (115kB/s-117kB/s), io=24.6MiB (25.7MB), run=55391-55566msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=1309: Tue Feb 27 16:28:00 2024 read: IOPS=36, BW=592KiB/s (606kB/s)(5920KiB/10006msec) clat (usec): min=59, max=66014, avg=14353.52, stdev=11321.69 lat (usec): min=60, max=66015, avg=14354.72, stdev=11321.74 clat percentiles (usec): | 1.00th=[ 76], 5.00th=[ 95], 10.00th=[ 106], 20.00th=[ 6325], | 30.00th=[ 8717], 40.00th=[10552], 50.00th=[12387], 60.00th=[14746], | 70.00th=[17171], 80.00th=[20841], 90.00th=[27657], 95.00th=[34866], | 99.00th=[57934], 99.50th=[63701], 99.90th=[65799], 99.95th=[65799], | 99.99th=[65799] bw ( KiB/s): min= 224, max= 960, per=25.64%, avg=588.60, stdev=179.57, samples=20 iops : min= 14, max= 60, avg=36.35, stdev=11.22, samples=20 write: IOPS=39, BW=636KiB/s (652kB/s)(6368KiB/10006msec) clat (usec): min=755, max=63061, avg=11736.67, stdev=12862.42 lat (usec): min=756, max=63061, avg=11740.22, stdev=12861.23 clat percentiles (usec): | 1.00th=[ 799], 5.00th=[ 922], 10.00th=[ 1029], 20.00th=[ 1156], | 30.00th=[ 1762], 40.00th=[ 3851], 50.00th=[ 6783], 60.00th=[10290], | 70.00th=[14484], 80.00th=[21890], 90.00th=[32113], 95.00th=[38011], | 99.00th=[52691], 99.50th=[63177], 99.90th=[63177], 99.95th=[63177], | 99.99th=[63177] bw ( KiB/s): min= 286, max= 1024, per=26.22%, avg=633.40, stdev=201.71, samples=20 iops : min= 17, max= 64, avg=39.15, stdev=12.70, samples=20 lat (usec) : 100=3.52%, 250=2.86%, 500=0.39%, 750=0.26%, 1000=4.04% lat (msec) : 2=12.24%, 4=5.08%, 10=20.44%, 20=27.99%, 50=21.22% lat (msec) : 100=1.95% cpu : usr=0.15%, sys=12.06%, ctx=1302, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1310: Tue Feb 27 16:28:00 2024 read: IOPS=34, BW=559KiB/s (573kB/s)(5744KiB/10272msec) clat (usec): min=61, max=75105, avg=15796.38, stdev=11635.52 lat (usec): min=61, max=75106, avg=15797.59, stdev=11635.49 clat percentiles (usec): | 1.00th=[ 87], 5.00th=[ 102], 10.00th=[ 130], 20.00th=[ 7504], | 30.00th=[ 9503], 40.00th=[11994], 50.00th=[14353], 60.00th=[16581], | 70.00th=[18744], 80.00th=[22676], 90.00th=[29754], 95.00th=[39060], | 99.00th=[60031], 99.50th=[66323], 99.90th=[74974], 99.95th=[74974], | 99.99th=[74974] bw ( KiB/s): min= 288, max= 830, per=23.97%, avg=550.30, stdev=148.54, samples=20 iops : min= 18, max= 51, avg=33.85, stdev= 9.29, samples=20 write: IOPS=39, BW=637KiB/s (652kB/s)(6544KiB/10272msec) clat (usec): min=684, max=66733, avg=11205.24, stdev=12772.85 lat (usec): min=685, max=66735, avg=11207.10, stdev=12772.78 clat percentiles (usec): | 1.00th=[ 775], 5.00th=[ 922], 10.00th=[ 1004], 20.00th=[ 1139], | 30.00th=[ 2008], 40.00th=[ 3425], 50.00th=[ 6325], 60.00th=[ 9896], | 70.00th=[13566], 80.00th=[19268], 90.00th=[31065], 95.00th=[38011], | 99.00th=[56361], 99.50th=[65274], 99.90th=[66847], 99.95th=[66847], | 99.99th=[66847] bw ( KiB/s): min= 415, max= 894, per=25.82%, avg=623.75, stdev=128.26, samples=20 iops : min= 25, max= 55, avg=38.45, stdev= 7.90, samples=20 lat (usec) : 100=2.08%, 250=2.86%, 500=0.26%, 750=0.39%, 1000=4.69% lat (msec) : 2=10.94%, 4=6.51%, 10=19.40%, 20=30.47%, 50=20.70% lat (msec) : 100=1.69% cpu : usr=0.05%, sys=11.81%, ctx=1340, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1311: Tue Feb 27 16:28:00 2024 read: IOPS=38, BW=609KiB/s (623kB/s)(6256KiB/10275msec) clat (usec): min=57, max=79709, avg=15418.53, stdev=11356.82 lat (usec): min=58, max=79710, avg=15419.69, stdev=11356.87 clat percentiles (usec): | 1.00th=[ 84], 5.00th=[ 101], 10.00th=[ 469], 20.00th=[ 7177], | 30.00th=[ 9503], 40.00th=[10814], 50.00th=[13304], 60.00th=[16057], | 70.00th=[18744], 80.00th=[22938], 90.00th=[28181], 95.00th=[36963], | 99.00th=[56886], 99.50th=[58983], 99.90th=[80217], 99.95th=[80217], | 99.99th=[80217] bw ( KiB/s): min= 384, max= 832, per=26.20%, avg=601.55, stdev=148.96, samples=20 iops : min= 24, max= 52, avg=37.20, stdev= 9.36, samples=20 write: IOPS=36, BW=587KiB/s (601kB/s)(6032KiB/10275msec) clat (usec): min=790, max=73089, avg=11191.16, stdev=12177.47 lat (usec): min=792, max=73091, avg=11202.99, stdev=12175.72 clat percentiles (usec): | 1.00th=[ 840], 5.00th=[ 947], 10.00th=[ 1020], 20.00th=[ 1172], | 30.00th=[ 2089], 40.00th=[ 4080], 50.00th=[ 7570], 60.00th=[10683], | 70.00th=[14746], 80.00th=[18482], 90.00th=[28181], 95.00th=[36439], | 99.00th=[52691], 99.50th=[68682], 99.90th=[72877], 99.95th=[72877], | 99.99th=[72877] bw ( KiB/s): min= 287, max= 1088, per=23.31%, avg=563.20, stdev=196.54, samples=20 iops : min= 17, max= 68, avg=34.80, stdev=12.35, samples=20 lat (usec) : 100=2.47%, 250=2.47%, 500=0.26%, 750=0.13%, 1000=3.91% lat (msec) : 2=10.68%, 4=5.34%, 10=20.44%, 20=32.81%, 50=20.05% lat (msec) : 100=1.43% cpu : usr=0.16%, sys=11.78%, ctx=1274, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1312: Tue Feb 27 16:28:00 2024 read: IOPS=36, BW=578KiB/s (592kB/s)(6032KiB/10429msec) clat (usec): min=62, max=88471, avg=15660.14, stdev=12237.35 lat (usec): min=62, max=88473, avg=15661.30, stdev=12237.40 clat percentiles (usec): | 1.00th=[ 72], 5.00th=[ 94], 10.00th=[ 106], 20.00th=[ 6718], | 30.00th=[ 9634], 40.00th=[11863], 50.00th=[13829], 60.00th=[16450], | 70.00th=[19006], 80.00th=[23987], 90.00th=[29230], 95.00th=[37487], | 99.00th=[55313], 99.50th=[70779], 99.90th=[88605], 99.95th=[88605], | 99.99th=[88605] bw ( KiB/s): min= 318, max= 766, per=23.81%, avg=546.65, stdev=112.58, samples=20 iops : min= 19, max= 47, avg=33.65, stdev= 6.95, samples=20 write: IOPS=37, BW=600KiB/s (614kB/s)(6256KiB/10429msec) clat (usec): min=834, max=74248, avg=11520.90, stdev=13333.25 lat (usec): min=836, max=74249, avg=11530.10, stdev=13337.36 clat percentiles (usec): | 1.00th=[ 857], 5.00th=[ 938], 10.00th=[ 1037], 20.00th=[ 1270], | 30.00th=[ 1926], 40.00th=[ 3523], 50.00th=[ 6849], 60.00th=[ 9634], | 70.00th=[13566], 80.00th=[20579], 90.00th=[30016], 95.00th=[37487], | 99.00th=[61080], 99.50th=[73925], 99.90th=[73925], 99.95th=[73925], | 99.99th=[73925] bw ( KiB/s): min= 349, max= 954, per=23.94%, avg=578.40, stdev=177.49, samples=20 iops : min= 21, max= 59, avg=35.65, stdev=11.01, samples=20 lat (usec) : 100=3.78%, 250=3.12%, 750=0.26%, 1000=4.43% lat (msec) : 2=11.07%, 4=6.12%, 10=17.32%, 20=29.69%, 50=22.14% lat (msec) : 100=2.08% cpu : usr=0.22%, sys=11.52%, ctx=1259, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2297KiB/s (2352kB/s), 559KiB/s-609KiB/s (573kB/s-623kB/s), io=23.4MiB (24.5MB), run=10006-10429msec WRITE: bw=2416KiB/s (2474kB/s), 587KiB/s-637KiB/s (601kB/s-652kB/s), io=24.6MiB (25.8MB), run=10006-10429msec rand-rw: (groupid=0, jobs=1): err= 0: pid=1314: Tue Feb 27 16:28:05 2024 read: IOPS=23, BW=382KiB/s (391kB/s)(5920KiB/15498msec) clat (msec): min=3, max=101, avg=15.15, stdev=13.50 lat (msec): min=3, max=101, avg=15.15, stdev=13.50 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 6], 10.00th=[ 6], 20.00th=[ 7], | 30.00th=[ 9], 40.00th=[ 10], 50.00th=[ 11], 60.00th=[ 13], | 70.00th=[ 16], 80.00th=[ 21], 90.00th=[ 29], 95.00th=[ 43], | 99.00th=[ 82], 99.50th=[ 94], 99.90th=[ 102], 99.95th=[ 102], | 99.99th=[ 102] bw ( KiB/s): min= 127, max= 766, per=24.17%, avg=373.00, stdev=172.73, samples=30 iops : min= 7, max= 47, avg=22.80, stdev=10.84, samples=30 write: IOPS=25, BW=411KiB/s (421kB/s)(6368KiB/15498msec) clat (msec): min=6, max=116, avg=24.82, stdev=17.19 lat (msec): min=6, max=116, avg=24.83, stdev=17.18 clat percentiles (msec): | 1.00th=[ 8], 5.00th=[ 11], 10.00th=[ 12], 20.00th=[ 14], | 30.00th=[ 15], 40.00th=[ 17], 50.00th=[ 19], 60.00th=[ 22], | 70.00th=[ 27], 80.00th=[ 35], 90.00th=[ 47], 95.00th=[ 62], | 99.00th=[ 94], 99.50th=[ 115], 99.90th=[ 116], 99.95th=[ 116], | 99.99th=[ 116] bw ( KiB/s): min= 159, max= 827, per=25.14%, avg=408.10, stdev=202.54, samples=30 iops : min= 9, max= 51, avg=25.00, stdev=12.67, samples=30 lat (msec) : 4=0.52%, 10=24.74%, 20=40.76%, 50=28.65%, 100=4.82% lat (msec) : 250=0.52% cpu : usr=0.12%, sys=5.32%, ctx=859, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1315: Tue Feb 27 16:28:05 2024 read: IOPS=23, BW=370KiB/s (379kB/s)(5744KiB/15519msec) clat (usec): min=3520, max=95712, avg=14582.03, stdev=13730.41 lat (usec): min=3521, max=95713, avg=14583.47, stdev=13730.32 clat percentiles (usec): | 1.00th=[ 3949], 5.00th=[ 4686], 10.00th=[ 5342], 20.00th=[ 6390], | 30.00th=[ 7439], 40.00th=[ 8291], 50.00th=[ 9634], 60.00th=[11076], | 70.00th=[14746], 80.00th=[19006], 90.00th=[29230], 95.00th=[42730], | 99.00th=[74974], 99.50th=[89654], 99.90th=[95945], 99.95th=[95945], | 99.99th=[95945] bw ( KiB/s): min= 96, max= 796, per=23.89%, avg=368.61, stdev=203.15, samples=31 iops : min= 6, max= 49, avg=22.65, stdev=12.62, samples=31 write: IOPS=26, BW=422KiB/s (432kB/s)(6544KiB/15519msec) clat (msec): min=5, max=112, avg=25.10, stdev=16.31 lat (msec): min=5, max=112, avg=25.10, stdev=16.31 clat percentiles (msec): | 1.00th=[ 8], 5.00th=[ 11], 10.00th=[ 13], 20.00th=[ 14], | 30.00th=[ 16], 40.00th=[ 18], 50.00th=[ 21], 60.00th=[ 23], | 70.00th=[ 27], 80.00th=[ 33], 90.00th=[ 48], 95.00th=[ 61], | 99.00th=[ 86], 99.50th=[ 88], 99.90th=[ 112], 99.95th=[ 112], | 99.99th=[ 112] bw ( KiB/s): min= 159, max= 734, per=25.83%, avg=419.23, stdev=169.56, samples=31 iops : min= 9, max= 45, avg=25.81, stdev=10.58, samples=31 lat (msec) : 4=0.52%, 10=26.30%, 20=37.50%, 50=29.04%, 100=6.51% lat (msec) : 250=0.13% cpu : usr=0.22%, sys=5.04%, ctx=871, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1316: Tue Feb 27 16:28:05 2024 read: IOPS=25, BW=403KiB/s (413kB/s)(6256KiB/15523msec) clat (msec): min=3, max=108, avg=15.05, stdev=13.57 lat (msec): min=3, max=108, avg=15.05, stdev=13.57 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 6], 10.00th=[ 6], 20.00th=[ 7], | 30.00th=[ 8], 40.00th=[ 9], 50.00th=[ 10], 60.00th=[ 12], | 70.00th=[ 15], 80.00th=[ 21], 90.00th=[ 31], 95.00th=[ 45], | 99.00th=[ 70], 99.50th=[ 101], 99.90th=[ 109], 99.95th=[ 109], | 99.99th=[ 109] bw ( KiB/s): min= 127, max= 960, per=26.05%, avg=401.93, stdev=225.65, samples=30 iops : min= 7, max= 60, avg=24.67, stdev=14.16, samples=30 write: IOPS=24, BW=389KiB/s (398kB/s)(6032KiB/15523msec) clat (usec): min=6856, max=97967, avg=25530.34, stdev=15395.45 lat (usec): min=6859, max=97969, avg=25532.63, stdev=15395.34 clat percentiles (usec): | 1.00th=[ 8356], 5.00th=[10814], 10.00th=[11731], 20.00th=[13829], | 30.00th=[15664], 40.00th=[17433], 50.00th=[19530], 60.00th=[23200], | 70.00th=[28443], 80.00th=[38011], 90.00th=[49021], 95.00th=[57410], | 99.00th=[73925], 99.50th=[87557], 99.90th=[98042], 99.95th=[98042], | 99.99th=[98042] bw ( KiB/s): min= 159, max= 704, per=23.26%, avg=377.47, stdev=164.96, samples=30 iops : min= 9, max= 44, avg=23.13, stdev=10.39, samples=30 lat (msec) : 4=0.26%, 10=27.08%, 20=38.67%, 50=27.99%, 100=5.73% lat (msec) : 250=0.26% cpu : usr=0.16%, sys=5.37%, ctx=849, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1317: Tue Feb 27 16:28:05 2024 read: IOPS=24, BW=389KiB/s (399kB/s)(6032KiB/15500msec) clat (usec): min=3599, max=94307, avg=16024.77, stdev=14496.05 lat (usec): min=3600, max=94308, avg=16026.11, stdev=14496.03 clat percentiles (usec): | 1.00th=[ 4228], 5.00th=[ 4948], 10.00th=[ 5866], 20.00th=[ 6718], | 30.00th=[ 8029], 40.00th=[ 9372], 50.00th=[10814], 60.00th=[12649], | 70.00th=[16057], 80.00th=[21627], 90.00th=[32637], 95.00th=[47973], | 99.00th=[76022], 99.50th=[90702], 99.90th=[93848], 99.95th=[93848], | 99.99th=[93848] bw ( KiB/s): min= 128, max= 864, per=24.76%, avg=382.07, stdev=172.23, samples=30 iops : min= 8, max= 54, avg=23.47, stdev=10.71, samples=30 write: IOPS=25, BW=404KiB/s (413kB/s)(6256KiB/15500msec) clat (usec): min=5835, max=91901, avg=24133.56, stdev=15037.45 lat (usec): min=5837, max=91903, avg=24136.22, stdev=15037.31 clat percentiles (usec): | 1.00th=[ 7832], 5.00th=[ 9765], 10.00th=[11338], 20.00th=[13304], | 30.00th=[15270], 40.00th=[17171], 50.00th=[19268], 60.00th=[21627], | 70.00th=[25822], 80.00th=[32900], 90.00th=[45351], 95.00th=[55313], | 99.00th=[80217], 99.50th=[87557], 99.90th=[91751], 99.95th=[91751], | 99.99th=[91751] bw ( KiB/s): min= 127, max= 798, per=24.65%, avg=400.10, stdev=202.42, samples=30 iops : min= 7, max= 49, avg=24.60, stdev=12.58, samples=30 lat (msec) : 4=0.39%, 10=24.35%, 20=41.15%, 50=27.99%, 100=6.12% cpu : usr=0.02%, sys=5.28%, ctx=866, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1543KiB/s (1580kB/s), 370KiB/s-403KiB/s (379kB/s-413kB/s), io=23.4MiB (24.5MB), run=15498-15523msec WRITE: bw=1623KiB/s (1662kB/s), 389KiB/s-422KiB/s (398kB/s-432kB/s), io=24.6MiB (25.8MB), run=15498-15523msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=1334: Tue Feb 27 16:28:08 2024 read: IOPS=5, BW=5973KiB/s (6117kB/s)(7168KiB/1200msec) clat (usec): min=476, max=184009, avg=97823.32, stdev=54874.65 lat (usec): min=478, max=184011, avg=97825.06, stdev=54874.69 clat percentiles (usec): | 1.00th=[ 478], 5.00th=[ 478], 10.00th=[ 478], 20.00th=[ 73925], | 30.00th=[ 94897], 40.00th=[ 94897], 50.00th=[105382], 60.00th=[107480], | 70.00th=[107480], 80.00th=[117965], 90.00th=[183501], 95.00th=[183501], | 99.00th=[183501], 99.50th=[183501], 99.90th=[183501], 99.95th=[183501], | 99.99th=[183501] bw ( KiB/s): min= 4096, max= 6119, per=37.73%, avg=5107.50, stdev=1430.48, samples=2 iops : min= 4, max= 5, avg= 4.50, stdev= 0.71, samples=2 write: IOPS=4, BW=4267KiB/s (4369kB/s)(5120KiB/1200msec) clat (msec): min=80, max=121, avg=102.35, stdev=15.00 lat (msec): min=80, max=121, avg=102.73, stdev=15.15 clat percentiles (msec): | 1.00th=[ 81], 5.00th=[ 81], 10.00th=[ 81], 20.00th=[ 81], | 30.00th=[ 97], 40.00th=[ 97], 50.00th=[ 106], 60.00th=[ 106], | 70.00th=[ 108], 80.00th=[ 108], 90.00th=[ 122], 95.00th=[ 122], | 99.00th=[ 122], 99.50th=[ 122], 99.90th=[ 122], 99.95th=[ 122], | 99.99th=[ 122] bw ( KiB/s): min= 8192, max= 8192, per=43.23%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 lat (usec) : 500=8.33% lat (msec) : 100=33.33%, 250=58.33% cpu : usr=0.00%, sys=25.77%, ctx=90, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1335: Tue Feb 27 16:28:08 2024 read: IOPS=1, BW=1354KiB/s (1386kB/s)(2048KiB/1513msec) clat (msec): min=107, max=184, avg=145.87, stdev=54.47 lat (msec): min=107, max=184, avg=145.87, stdev=54.48 clat percentiles (msec): | 1.00th=[ 108], 5.00th=[ 108], 10.00th=[ 108], 20.00th=[ 108], | 30.00th=[ 108], 40.00th=[ 108], 50.00th=[ 108], 60.00th=[ 184], | 70.00th=[ 184], 80.00th=[ 184], 90.00th=[ 184], 95.00th=[ 184], | 99.00th=[ 184], 99.50th=[ 184], 99.90th=[ 184], 99.95th=[ 184], | 99.99th=[ 184] bw ( KiB/s): min= 2048, max= 2048, per=15.13%, avg=2048.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=6, BW=6768KiB/s (6930kB/s)(10.0MiB/1513msec) clat (msec): min=54, max=265, avg=119.60, stdev=71.72 lat (msec): min=54, max=265, avg=119.68, stdev=71.72 clat percentiles (msec): | 1.00th=[ 55], 5.00th=[ 55], 10.00th=[ 55], 20.00th=[ 59], | 30.00th=[ 61], 40.00th=[ 61], 50.00th=[ 104], 60.00th=[ 114], | 70.00th=[ 144], 80.00th=[ 144], 90.00th=[ 215], 95.00th=[ 266], | 99.00th=[ 266], 99.50th=[ 266], 99.90th=[ 266], 99.95th=[ 266], | 99.99th=[ 266] bw ( KiB/s): min= 2048, max= 8192, per=27.02%, avg=5120.00, stdev=4344.46, samples=2 iops : min= 2, max= 8, avg= 5.00, stdev= 4.24, samples=2 lat (msec) : 100=33.33%, 250=58.33%, 500=8.33% cpu : usr=0.00%, sys=19.64%, ctx=97, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1336: Tue Feb 27 16:28:08 2024 read: IOPS=4, BW=4766KiB/s (4881kB/s)(6144KiB/1289msec) clat (usec): min=460, max=126012, avg=58546.74, stdev=59642.61 lat (usec): min=461, max=126014, avg=58548.30, stdev=59642.46 clat percentiles (usec): | 1.00th=[ 461], 5.00th=[ 461], 10.00th=[ 461], 20.00th=[ 3916], | 30.00th=[ 3916], 40.00th=[ 9896], 50.00th=[ 9896], 60.00th=[ 99091], | 70.00th=[112722], 80.00th=[112722], 90.00th=[126354], 95.00th=[126354], | 99.00th=[126354], 99.50th=[126354], 99.90th=[126354], 99.95th=[126354], | 99.99th=[126354] bw ( KiB/s): min= 4055, max= 4055, per=29.96%, avg=4055.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 write: IOPS=4, BW=4766KiB/s (4881kB/s)(6144KiB/1289msec) clat (msec): min=40, max=411, avg=155.52, stdev=146.79 lat (msec): min=40, max=412, avg=155.65, stdev=146.90 clat percentiles (msec): | 1.00th=[ 41], 5.00th=[ 41], 10.00th=[ 41], 20.00th=[ 47], | 30.00th=[ 47], 40.00th=[ 54], 50.00th=[ 54], 60.00th=[ 146], | 70.00th=[ 236], 80.00th=[ 236], 90.00th=[ 414], 95.00th=[ 414], | 99.00th=[ 414], 99.50th=[ 414], 99.90th=[ 414], 99.95th=[ 414], | 99.99th=[ 414] bw ( KiB/s): min= 4039, max= 4055, per=21.36%, avg=4047.00, stdev=11.31, samples=2 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=2 lat (usec) : 500=8.33% lat (msec) : 4=8.33%, 10=8.33%, 50=16.67%, 100=16.67%, 250=33.33% lat (msec) : 500=8.33% cpu : usr=0.00%, sys=15.61%, ctx=106, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1337: Tue Feb 27 16:28:08 2024 read: IOPS=3, BW=3762KiB/s (3852kB/s)(5120KiB/1361msec) clat (usec): min=1460, max=102422, avg=48894.10, stdev=36526.44 lat (usec): min=1462, max=102423, avg=48895.64, stdev=36526.15 clat percentiles (usec): | 1.00th=[ 1467], 5.00th=[ 1467], 10.00th=[ 1467], 20.00th=[ 1467], | 30.00th=[ 39060], 40.00th=[ 39060], 50.00th=[ 43254], 60.00th=[ 43254], | 70.00th=[ 58459], 80.00th=[ 58459], 90.00th=[102237], 95.00th=[102237], | 99.00th=[102237], 99.50th=[102237], 99.90th=[102237], 99.95th=[102237], | 99.99th=[102237] bw ( KiB/s): min= 8192, max= 8192, per=60.52%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=5, BW=5267KiB/s (5393kB/s)(7168KiB/1361msec) clat (msec): min=72, max=350, avg=156.94, stdev=94.37 lat (msec): min=72, max=350, avg=157.00, stdev=94.37 clat percentiles (msec): | 1.00th=[ 72], 5.00th=[ 72], 10.00th=[ 72], 20.00th=[ 92], | 30.00th=[ 109], 40.00th=[ 109], 50.00th=[ 122], 60.00th=[ 163], | 70.00th=[ 163], 80.00th=[ 190], 90.00th=[ 351], 95.00th=[ 351], | 99.00th=[ 351], 99.50th=[ 351], 99.90th=[ 351], 99.95th=[ 351], | 99.99th=[ 351] bw ( KiB/s): min= 4096, max= 4096, per=21.61%, avg=4096.00, stdev= 0.00, samples=2 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=2 lat (msec) : 2=8.33%, 50=16.67%, 100=25.00%, 250=41.67%, 500=8.33% cpu : usr=0.15%, sys=13.75%, ctx=71, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=13.2MiB/s (13.9MB/s), 1354KiB/s-5973KiB/s (1386kB/s-6117kB/s), io=20.0MiB (20.0MB), run=1200-1513msec WRITE: bw=18.5MiB/s (19.4MB/s), 4267KiB/s-6768KiB/s (4369kB/s-6930kB/s), io=28.0MiB (29.4MB), run=1200-1513msec rand-rw: (groupid=0, jobs=1): err= 0: pid=1339: Tue Feb 27 16:28:09 2024 read: IOPS=3, BW=3207KiB/s (3284kB/s)(7168KiB/2235msec) clat (msec): min=40, max=521, avg=201.97, stdev=171.83 lat (msec): min=40, max=521, avg=201.97, stdev=171.83 clat percentiles (msec): | 1.00th=[ 42], 5.00th=[ 42], 10.00th=[ 42], 20.00th=[ 66], | 30.00th=[ 91], 40.00th=[ 91], 50.00th=[ 134], 60.00th=[ 255], | 70.00th=[ 255], 80.00th=[ 305], 90.00th=[ 523], 95.00th=[ 523], | 99.00th=[ 523], 99.50th=[ 523], 99.90th=[ 523], 99.95th=[ 523], | 99.99th=[ 523] bw ( KiB/s): min= 2048, max= 4096, per=28.53%, avg=2560.00, stdev=1024.00, samples=4 iops : min= 2, max= 4, avg= 2.50, stdev= 1.00, samples=4 write: IOPS=2, BW=2291KiB/s (2346kB/s)(5120KiB/2235msec) clat (msec): min=79, max=265, avg=161.15, stdev=84.58 lat (msec): min=79, max=265, avg=161.24, stdev=84.57 clat percentiles (msec): | 1.00th=[ 80], 5.00th=[ 80], 10.00th=[ 80], 20.00th=[ 80], | 30.00th=[ 99], 40.00th=[ 99], 50.00th=[ 127], 60.00th=[ 127], | 70.00th=[ 236], 80.00th=[ 236], 90.00th=[ 266], 95.00th=[ 266], | 99.00th=[ 266], 99.50th=[ 266], 99.90th=[ 266], 99.95th=[ 266], | 99.99th=[ 266] bw ( KiB/s): min= 2048, max= 4096, per=24.45%, avg=3072.00, stdev=1448.15, samples=2 iops : min= 2, max= 4, avg= 3.00, stdev= 1.41, samples=2 lat (msec) : 50=8.33%, 100=33.33%, 250=25.00%, 500=25.00%, 750=8.33% cpu : usr=0.00%, sys=6.09%, ctx=63, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1340: Tue Feb 27 16:28:09 2024 read: IOPS=0, BW=954KiB/s (977kB/s)(2048KiB/2146msec) clat (msec): min=124, max=326, avg=225.87, stdev=142.92 lat (msec): min=124, max=326, avg=225.87, stdev=142.92 clat percentiles (msec): | 1.00th=[ 126], 5.00th=[ 126], 10.00th=[ 126], 20.00th=[ 126], | 30.00th=[ 126], 40.00th=[ 126], 50.00th=[ 126], 60.00th=[ 326], | 70.00th=[ 326], 80.00th=[ 326], 90.00th=[ 326], 95.00th=[ 326], | 99.00th=[ 326], 99.50th=[ 326], 99.90th=[ 326], 99.95th=[ 326], | 99.99th=[ 326] bw ( KiB/s): min= 2027, max= 2048, per=22.70%, avg=2037.50, stdev=14.85, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 write: IOPS=4, BW=4772KiB/s (4886kB/s)(10.0MiB/2146msec) clat (msec): min=67, max=281, avg=168.96, stdev=79.79 lat (msec): min=67, max=281, avg=169.09, stdev=79.84 clat percentiles (msec): | 1.00th=[ 68], 5.00th=[ 68], 10.00th=[ 68], 20.00th=[ 81], | 30.00th=[ 99], 40.00th=[ 107], 50.00th=[ 133], 60.00th=[ 205], | 70.00th=[ 241], 80.00th=[ 241], 90.00th=[ 247], 95.00th=[ 284], | 99.00th=[ 284], 99.50th=[ 284], 99.90th=[ 284], 99.95th=[ 284], | 99.99th=[ 284] bw ( KiB/s): min= 4079, max= 6144, per=43.39%, avg=5451.33, stdev=1188.49, samples=3 iops : min= 3, max= 6, avg= 4.67, stdev= 1.53, samples=3 lat (msec) : 100=25.00%, 250=58.33%, 500=16.67% cpu : usr=0.05%, sys=8.48%, ctx=67, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1342: Tue Feb 27 16:28:09 2024 read: IOPS=2, BW=2692KiB/s (2757kB/s)(6144KiB/2282msec) clat (msec): min=59, max=281, avg=129.73, stdev=78.71 lat (msec): min=59, max=281, avg=129.73, stdev=78.71 clat percentiles (msec): | 1.00th=[ 59], 5.00th=[ 59], 10.00th=[ 59], 20.00th=[ 84], | 30.00th=[ 84], 40.00th=[ 111], 50.00th=[ 111], 60.00th=[ 113], | 70.00th=[ 131], 80.00th=[ 131], 90.00th=[ 284], 95.00th=[ 284], | 99.00th=[ 284], 99.50th=[ 284], 99.90th=[ 284], 99.95th=[ 284], | 99.99th=[ 284] bw ( KiB/s): min= 4096, max= 4096, per=45.64%, avg=4096.00, stdev= 0.00, samples=2 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=2 write: IOPS=2, BW=2692KiB/s (2757kB/s)(6144KiB/2282msec) clat (msec): min=89, max=348, avg=249.06, stdev=99.49 lat (msec): min=89, max=348, avg=249.13, stdev=99.50 clat percentiles (msec): | 1.00th=[ 90], 5.00th=[ 90], 10.00th=[ 90], 20.00th=[ 203], | 30.00th=[ 203], 40.00th=[ 211], 50.00th=[ 211], 60.00th=[ 321], | 70.00th=[ 321], 80.00th=[ 321], 90.00th=[ 351], 95.00th=[ 351], | 99.00th=[ 351], 99.50th=[ 351], 99.90th=[ 351], 99.95th=[ 351], | 99.99th=[ 351] bw ( KiB/s): min= 2043, max= 4087, per=20.35%, avg=2556.50, stdev=1020.34, samples=4 iops : min= 1, max= 3, avg= 2.00, stdev= 0.82, samples=4 lat (msec) : 100=25.00%, 250=41.67%, 500=33.33% cpu : usr=0.00%, sys=5.66%, ctx=53, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1343: Tue Feb 27 16:28:09 2024 read: IOPS=2, BW=2316KiB/s (2371kB/s)(5120KiB/2211msec) clat (msec): min=37, max=214, avg=121.11, stdev=69.14 lat (msec): min=38, max=214, avg=121.12, stdev=69.14 clat percentiles (msec): | 1.00th=[ 39], 5.00th=[ 39], 10.00th=[ 39], 20.00th=[ 39], | 30.00th=[ 78], 40.00th=[ 78], 50.00th=[ 115], 60.00th=[ 115], | 70.00th=[ 161], 80.00th=[ 161], 90.00th=[ 215], 95.00th=[ 215], | 99.00th=[ 215], 99.50th=[ 215], 99.90th=[ 215], 99.95th=[ 215], | 99.99th=[ 215] bw ( KiB/s): min= 2048, max= 6119, per=45.50%, avg=4083.50, stdev=2878.63, samples=2 iops : min= 2, max= 5, avg= 3.50, stdev= 2.12, samples=2 write: IOPS=3, BW=3242KiB/s (3320kB/s)(7168KiB/2211msec) clat (msec): min=99, max=523, avg=227.76, stdev=155.19 lat (msec): min=99, max=523, avg=228.00, stdev=155.09 clat percentiles (msec): | 1.00th=[ 101], 5.00th=[ 101], 10.00th=[ 101], 20.00th=[ 109], | 30.00th=[ 148], 40.00th=[ 148], 50.00th=[ 163], 60.00th=[ 203], | 70.00th=[ 203], 80.00th=[ 351], 90.00th=[ 523], 95.00th=[ 523], | 99.00th=[ 523], 99.50th=[ 523], 99.90th=[ 523], 99.95th=[ 523], | 99.99th=[ 523] bw ( KiB/s): min= 2039, max= 4096, per=27.14%, avg=3410.33, stdev=1187.61, samples=3 iops : min= 1, max= 4, avg= 3.00, stdev= 1.73, samples=3 lat (msec) : 50=8.33%, 100=16.67%, 250=58.33%, 500=8.33%, 750=8.33% cpu : usr=0.00%, sys=5.52%, ctx=60, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=8975KiB/s (9190kB/s), 954KiB/s-3207KiB/s (977kB/s-3284kB/s), io=20.0MiB (20.0MB), run=2146-2282msec WRITE: bw=12.3MiB/s (12.9MB/s), 2291KiB/s-4772KiB/s (2346kB/s-4886kB/s), io=28.0MiB (29.4MB), run=2146-2282msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=1349: Tue Feb 27 16:28:12 2024 read: IOPS=2, BW=10.6MiB/s (11.1MB/s)(12.0MiB/1132msec) clat (msec): min=201, max=466, avg=377.05, stdev=151.91 lat (msec): min=201, max=466, avg=377.05, stdev=151.91 clat percentiles (msec): | 1.00th=[ 203], 5.00th=[ 203], 10.00th=[ 203], 20.00th=[ 203], | 30.00th=[ 203], 40.00th=[ 464], 50.00th=[ 464], 60.00th=[ 464], | 70.00th=[ 468], 80.00th=[ 468], 90.00th=[ 468], 95.00th=[ 468], | 99.00th=[ 468], 99.50th=[ 468], 99.90th=[ 468], 99.95th=[ 468], | 99.99th=[ 468] bw ( KiB/s): min= 8192, max= 8192, per=88.41%, avg=8192.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 lat (msec) : 250=33.33%, 500=66.67% cpu : usr=0.00%, sys=23.78%, ctx=150, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1350: Tue Feb 27 16:28:12 2024 read: IOPS=0, BW=2317KiB/s (2372kB/s)(4096KiB/1768msec) clat (nsec): min=406393k, max=406393k, avg=406393013.00, stdev= 0.00 lat (nsec): min=406397k, max=406397k, avg=406396592.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 405], 5.00th=[ 405], 10.00th=[ 405], 20.00th=[ 405], | 30.00th=[ 405], 40.00th=[ 405], 50.00th=[ 405], 60.00th=[ 405], | 70.00th=[ 405], 80.00th=[ 405], 90.00th=[ 405], 95.00th=[ 405], | 99.00th=[ 405], 99.50th=[ 405], 99.90th=[ 405], 99.95th=[ 405], | 99.99th=[ 405] bw ( KiB/s): min= 8192, max= 8192, per=88.41%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=1, BW=4633KiB/s (4745kB/s)(8192KiB/1768msec) clat (msec): min=610, max=728, avg=669.36, stdev=83.59 lat (msec): min=611, max=728, avg=670.13, stdev=82.77 clat percentiles (msec): | 1.00th=[ 609], 5.00th=[ 609], 10.00th=[ 609], 20.00th=[ 609], | 30.00th=[ 609], 40.00th=[ 609], 50.00th=[ 609], 60.00th=[ 726], | 70.00th=[ 726], 80.00th=[ 726], 90.00th=[ 726], 95.00th=[ 726], | 99.00th=[ 726], 99.50th=[ 726], 99.90th=[ 726], 99.95th=[ 726], | 99.99th=[ 726] bw ( KiB/s): min= 5587, max= 5587, per=30.15%, avg=5587.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 500=33.33%, 750=66.67% cpu : usr=0.00%, sys=13.41%, ctx=187, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1351: Tue Feb 27 16:28:12 2024 write: IOPS=2, BW=9022KiB/s (9239kB/s)(12.0MiB/1362msec) clat (msec): min=246, max=557, avg=453.15, stdev=179.26 lat (msec): min=246, max=557, avg=453.48, stdev=179.26 clat percentiles (msec): | 1.00th=[ 247], 5.00th=[ 247], 10.00th=[ 247], 20.00th=[ 247], | 30.00th=[ 247], 40.00th=[ 558], 50.00th=[ 558], 60.00th=[ 558], | 70.00th=[ 558], 80.00th=[ 558], 90.00th=[ 558], 95.00th=[ 558], | 99.00th=[ 558], 99.50th=[ 558], 99.90th=[ 558], 99.95th=[ 558], | 99.99th=[ 558] bw ( KiB/s): min= 8175, max= 8175, per=44.11%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 750=66.67% cpu : usr=0.00%, sys=19.54%, ctx=59, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1352: Tue Feb 27 16:28:12 2024 write: IOPS=2, BW=9288KiB/s (9511kB/s)(12.0MiB/1323msec) clat (msec): min=204, max=691, avg=432.21, stdev=244.82 lat (msec): min=204, max=691, avg=432.63, stdev=244.81 clat percentiles (msec): | 1.00th=[ 205], 5.00th=[ 205], 10.00th=[ 205], 20.00th=[ 205], | 30.00th=[ 205], 40.00th=[ 401], 50.00th=[ 401], 60.00th=[ 401], | 70.00th=[ 693], 80.00th=[ 693], 90.00th=[ 693], 95.00th=[ 693], | 99.00th=[ 693], 99.50th=[ 693], 99.90th=[ 693], 99.95th=[ 693], | 99.99th=[ 693] bw ( KiB/s): min= 8175, max= 8192, per=44.16%, avg=8183.50, stdev=12.02, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 250=33.33%, 500=33.33%, 750=33.33% cpu : usr=0.00%, sys=17.31%, ctx=56, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=9267KiB/s (9489kB/s), 2317KiB/s-10.6MiB/s (2372kB/s-11.1MB/s), io=16.0MiB (16.8MB), run=1132-1768msec WRITE: bw=18.1MiB/s (18.0MB/s), 4633KiB/s-9288KiB/s (4745kB/s-9511kB/s), io=32.0MiB (33.6MB), run=1323-1768msec rand-rw: (groupid=0, jobs=1): err= 0: pid=1354: Tue Feb 27 16:28:12 2024 read: IOPS=1, BW=7511KiB/s (7691kB/s)(12.0MiB/1636msec) clat (msec): min=86, max=1003, avg=545.03, stdev=458.28 lat (msec): min=86, max=1003, avg=545.03, stdev=458.28 clat percentiles (msec): | 1.00th=[ 87], 5.00th=[ 87], 10.00th=[ 87], 20.00th=[ 87], | 30.00th=[ 87], 40.00th=[ 542], 50.00th=[ 542], 60.00th=[ 542], | 70.00th=[ 1003], 80.00th=[ 1003], 90.00th=[ 1003], 95.00th=[ 1003], | 99.00th=[ 1003], 99.50th=[ 1003], 99.90th=[ 1003], 99.95th=[ 1003], | 99.99th=[ 1003] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 750=33.33% cpu : usr=0.12%, sys=6.97%, ctx=49, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1355: Tue Feb 27 16:28:12 2024 read: IOPS=0, BW=2029KiB/s (2077kB/s)(4096KiB/2019msec) clat (nsec): min=1082.7M, max=1082.7M, avg=1082671586.00, stdev= 0.00 lat (nsec): min=1082.7M, max=1082.7M, avg=1082673474.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 1083], 5.00th=[ 1083], 10.00th=[ 1083], 20.00th=[ 1083], | 30.00th=[ 1083], 40.00th=[ 1083], 50.00th=[ 1083], 60.00th=[ 1083], | 70.00th=[ 1083], 80.00th=[ 1083], 90.00th=[ 1083], 95.00th=[ 1083], | 99.00th=[ 1083], 99.50th=[ 1083], 99.90th=[ 1083], 99.95th=[ 1083], | 99.99th=[ 1083] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=0, BW=4057KiB/s (4155kB/s)(8192KiB/2019msec) clat (msec): min=354, max=580, avg=467.28, stdev=159.99 lat (msec): min=354, max=580, avg=467.67, stdev=160.01 clat percentiles (msec): | 1.00th=[ 355], 5.00th=[ 355], 10.00th=[ 355], 20.00th=[ 355], | 30.00th=[ 355], 40.00th=[ 355], 50.00th=[ 355], 60.00th=[ 584], | 70.00th=[ 584], 80.00th=[ 584], 90.00th=[ 584], 95.00th=[ 584], | 99.00th=[ 584], 99.50th=[ 584], 99.90th=[ 584], 99.95th=[ 584], | 99.99th=[ 584] bw ( KiB/s): min= 8192, max= 8192, per=52.65%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 500=33.33%, 750=33.33% cpu : usr=0.00%, sys=7.33%, ctx=53, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1356: Tue Feb 27 16:28:12 2024 write: IOPS=1, BW=7709KiB/s (7894kB/s)(12.0MiB/1594msec) clat (msec): min=138, max=777, avg=528.13, stdev=341.71 lat (msec): min=139, max=778, avg=528.46, stdev=341.75 clat percentiles (msec): | 1.00th=[ 140], 5.00th=[ 140], 10.00th=[ 140], 20.00th=[ 140], | 30.00th=[ 140], 40.00th=[ 667], 50.00th=[ 667], 60.00th=[ 667], | 70.00th=[ 776], 80.00th=[ 776], 90.00th=[ 776], 95.00th=[ 776], | 99.00th=[ 776], 99.50th=[ 776], 99.90th=[ 776], 99.95th=[ 776], | 99.99th=[ 776] bw ( KiB/s): min= 8094, max= 8175, per=52.28%, avg=8134.50, stdev=57.28, samples=2 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=2 lat (msec) : 250=33.33%, 750=33.33%, 1000=33.33% cpu : usr=0.50%, sys=8.60%, ctx=50, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1357: Tue Feb 27 16:28:12 2024 write: IOPS=1, BW=5835KiB/s (5975kB/s)(12.0MiB/2106msec) clat (msec): min=175, max=1150, avg=698.94, stdev=491.71 lat (msec): min=175, max=1150, avg=699.29, stdev=491.73 clat percentiles (msec): | 1.00th=[ 176], 5.00th=[ 176], 10.00th=[ 176], 20.00th=[ 176], | 30.00th=[ 176], 40.00th=[ 768], 50.00th=[ 768], 60.00th=[ 768], | 70.00th=[ 1150], 80.00th=[ 1150], 90.00th=[ 1150], 95.00th=[ 1150], | 99.00th=[ 1150], 99.50th=[ 1150], 99.90th=[ 1150], 99.95th=[ 1150], | 99.99th=[ 1150] bw ( KiB/s): min= 8110, max= 8192, per=52.39%, avg=8151.00, stdev=57.98, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 250=33.33%, 1000=33.33% cpu : usr=0.10%, sys=9.36%, ctx=157, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=8115KiB/s (8310kB/s), 2029KiB/s-7511KiB/s (2077kB/s-7691kB/s), io=16.0MiB (16.8MB), run=1636-2019msec WRITE: bw=15.2MiB/s (15.9MB/s), 4057KiB/s-7709KiB/s (4155kB/s-7894kB/s), io=32.0MiB (33.6MB), run=1594-2106msec