== sanity test 398b: DIO and buffer IO race ============== 07:58:29 (1744891109) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.0359 s, 48.6 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=9611: Thu Apr 17 07:58:43 2025 read: IOPS=122, BW=492KiB/s (504kB/s)(5992KiB/12186msec) clat (usec): min=50, max=169091, avg=3666.87, stdev=5175.03 lat (usec): min=50, max=169092, avg=3667.50, stdev=5175.07 clat percentiles (usec): | 1.00th=[ 64], 5.00th=[ 70], 10.00th=[ 84], 20.00th=[ 1942], | 30.00th=[ 2278], 40.00th=[ 2606], 50.00th=[ 2966], 60.00th=[ 3490], | 70.00th=[ 4424], 80.00th=[ 5407], 90.00th=[ 6783], 95.00th=[ 8291], | 99.00th=[ 12780], 99.50th=[ 16057], 99.90th=[ 49021], 99.95th=[168821], | 99.99th=[168821] bw ( KiB/s): min= 200, max= 849, per=24.70%, avg=486.67, stdev=113.06, samples=24 iops : min= 50, max= 212, avg=121.62, stdev=28.25, samples=24 write: IOPS=129, BW=517KiB/s (529kB/s)(6296KiB/12186msec) clat (usec): min=477, max=42156, avg=4156.22, stdev=3914.08 lat (usec): min=477, max=42156, avg=4156.94, stdev=3914.08 clat percentiles (usec): | 1.00th=[ 498], 5.00th=[ 537], 10.00th=[ 570], 20.00th=[ 660], | 30.00th=[ 979], 40.00th=[ 2245], 50.00th=[ 3490], 60.00th=[ 4359], | 70.00th=[ 5342], 80.00th=[ 6915], 90.00th=[ 9241], 95.00th=[11207], | 99.00th=[16450], 99.50th=[21890], 99.90th=[32113], 99.95th=[42206], | 99.99th=[42206] bw ( KiB/s): min= 224, max= 862, per=24.76%, avg=510.83, stdev=118.08, samples=24 iops : min= 56, max= 215, avg=127.67, stdev=29.45, samples=24 lat (usec) : 100=6.18%, 250=1.79%, 500=0.75%, 750=11.78%, 1000=3.29% lat (msec) : 2=6.58%, 4=30.27%, 10=34.21%, 20=4.75%, 50=0.36% lat (msec) : 250=0.03% cpu : usr=0.17%, sys=22.66%, ctx=5774, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9612: Thu Apr 17 07:58:43 2025 read: IOPS=124, BW=499KiB/s (511kB/s)(5940KiB/11905msec) clat (usec): min=53, max=27220, avg=3567.85, stdev=2773.51 lat (usec): min=53, max=27221, avg=3568.63, stdev=2773.42 clat percentiles (usec): | 1.00th=[ 62], 5.00th=[ 70], 10.00th=[ 83], 20.00th=[ 1942], | 30.00th=[ 2311], 40.00th=[ 2606], 50.00th=[ 3032], 60.00th=[ 3458], | 70.00th=[ 4293], 80.00th=[ 5407], 90.00th=[ 6849], 95.00th=[ 8291], | 99.00th=[12518], 99.50th=[14877], 99.90th=[24249], 99.95th=[27132], | 99.99th=[27132] bw ( KiB/s): min= 192, max= 694, per=25.25%, avg=497.48, stdev=115.59, samples=23 iops : min= 48, max= 173, avg=124.22, stdev=28.91, samples=23 write: IOPS=133, BW=533KiB/s (546kB/s)(6348KiB/11905msec) clat (usec): min=432, max=168261, avg=4070.28, stdev=5522.60 lat (usec): min=432, max=168261, avg=4071.37, stdev=5522.37 clat percentiles (usec): | 1.00th=[ 502], 5.00th=[ 537], 10.00th=[ 562], 20.00th=[ 635], | 30.00th=[ 881], 40.00th=[ 1827], 50.00th=[ 3294], 60.00th=[ 4178], | 70.00th=[ 5342], 80.00th=[ 6718], 90.00th=[ 8717], 95.00th=[ 10814], | 99.00th=[ 15008], 99.50th=[ 18482], 99.90th=[ 36439], 99.95th=[168821], | 99.99th=[168821] bw ( KiB/s): min= 256, max= 744, per=25.41%, avg=524.26, stdev=98.89, samples=23 iops : min= 64, max= 186, avg=130.91, stdev=24.80, samples=23 lat (usec) : 100=6.05%, 250=1.37%, 500=0.68%, 750=13.02%, 1000=3.19% lat (msec) : 2=6.84%, 4=31.18%, 10=33.07%, 20=4.23%, 50=0.33% lat (msec) : 250=0.03% cpu : usr=0.16%, sys=23.66%, ctx=5533, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9613: Thu Apr 17 07:58:43 2025 read: IOPS=126, BW=506KiB/s (518kB/s)(6028KiB/11923msec) clat (usec): min=51, max=166696, avg=3643.47, stdev=5065.34 lat (usec): min=51, max=166697, avg=3644.28, stdev=5065.37 clat percentiles (usec): | 1.00th=[ 60], 5.00th=[ 71], 10.00th=[ 84], 20.00th=[ 1893], | 30.00th=[ 2278], 40.00th=[ 2606], 50.00th=[ 2966], 60.00th=[ 3490], | 70.00th=[ 4359], 80.00th=[ 5473], 90.00th=[ 6980], 95.00th=[ 8356], | 99.00th=[ 10683], 99.50th=[ 13435], 99.90th=[ 41157], 99.95th=[166724], | 99.99th=[166724] bw ( KiB/s): min= 208, max= 712, per=25.50%, avg=502.43, stdev=107.72, samples=23 iops : min= 52, max= 178, avg=125.52, stdev=26.96, samples=23 write: IOPS=131, BW=525KiB/s (538kB/s)(6260KiB/11923msec) clat (usec): min=458, max=54494, avg=4014.08, stdev=4025.75 lat (usec): min=458, max=54495, avg=4015.03, stdev=4025.66 clat percentiles (usec): | 1.00th=[ 498], 5.00th=[ 537], 10.00th=[ 570], 20.00th=[ 668], | 30.00th=[ 930], 40.00th=[ 1975], 50.00th=[ 3261], 60.00th=[ 4146], | 70.00th=[ 5080], 80.00th=[ 6390], 90.00th=[ 8848], 95.00th=[11076], | 99.00th=[16712], 99.50th=[19268], 99.90th=[42206], 99.95th=[54264], | 99.99th=[54264] bw ( KiB/s): min= 304, max= 712, per=25.15%, avg=518.78, stdev=86.21, samples=23 iops : min= 76, max= 178, avg=129.61, stdev=21.56, samples=23 lat (usec) : 100=6.18%, 250=2.02%, 500=0.62%, 750=11.82%, 1000=3.78% lat (msec) : 2=7.03%, 4=30.40%, 10=33.76%, 20=4.04%, 50=0.29% lat (msec) : 100=0.03%, 250=0.03% cpu : usr=0.17%, sys=23.33%, ctx=5688, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9614: Thu Apr 17 07:58:43 2025 read: IOPS=127, BW=510KiB/s (522kB/s)(6052KiB/11861msec) clat (usec): min=53, max=163021, avg=3624.56, stdev=5069.35 lat (usec): min=53, max=163022, avg=3625.42, stdev=5069.55 clat percentiles (usec): | 1.00th=[ 61], 5.00th=[ 70], 10.00th=[ 82], 20.00th=[ 1811], | 30.00th=[ 2311], 40.00th=[ 2606], 50.00th=[ 2999], 60.00th=[ 3589], | 70.00th=[ 4424], 80.00th=[ 5342], 90.00th=[ 6849], 95.00th=[ 8160], | 99.00th=[ 11469], 99.50th=[ 16057], 99.90th=[ 49021], 99.95th=[162530], | 99.99th=[162530] bw ( KiB/s): min= 224, max= 680, per=25.60%, avg=504.39, stdev=114.15, samples=23 iops : min= 56, max= 170, avg=125.96, stdev=28.63, samples=23 write: IOPS=131, BW=526KiB/s (538kB/s)(6236KiB/11861msec) clat (usec): min=417, max=27145, avg=3996.29, stdev=3722.08 lat (usec): min=417, max=27145, avg=3997.02, stdev=3722.07 clat percentiles (usec): | 1.00th=[ 490], 5.00th=[ 523], 10.00th=[ 545], 20.00th=[ 635], | 30.00th=[ 865], 40.00th=[ 2180], 50.00th=[ 3359], 60.00th=[ 4080], | 70.00th=[ 5145], 80.00th=[ 6587], 90.00th=[ 8848], 95.00th=[11076], | 99.00th=[16450], 99.50th=[20055], 99.90th=[24249], 99.95th=[27132], | 99.99th=[27132] bw ( KiB/s): min= 264, max= 734, per=25.26%, avg=521.13, stdev=96.39, samples=23 iops : min= 66, max= 183, avg=130.13, stdev=24.12, samples=23 lat (usec) : 100=7.06%, 250=2.05%, 500=1.01%, 750=12.47%, 1000=3.16% lat (msec) : 2=5.79%, 4=30.60%, 10=33.33%, 20=4.13%, 50=0.36% lat (msec) : 250=0.03% cpu : usr=0.16%, sys=23.37%, ctx=5664, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1970KiB/s (2018kB/s), 492KiB/s-510KiB/s (504kB/s-522kB/s), io=23.4MiB (24.6MB), run=11861-12186msec WRITE: bw=2063KiB/s (2113kB/s), 517KiB/s-533KiB/s (529kB/s-546kB/s), io=24.6MiB (25.7MB), run=11861-12186msec rand-rw: (groupid=0, jobs=1): err= 0: pid=9615: Thu Apr 17 07:59:05 2025 read: IOPS=44, BW=176KiB/s (181kB/s)(5992KiB/33966msec) clat (usec): min=1573, max=171508, avg=3813.27, stdev=5614.50 lat (usec): min=1573, max=171511, avg=3813.97, stdev=5614.55 clat percentiles (usec): | 1.00th=[ 1713], 5.00th=[ 1844], 10.00th=[ 1909], 20.00th=[ 2024], | 30.00th=[ 2147], 40.00th=[ 2278], 50.00th=[ 2442], 60.00th=[ 2868], | 70.00th=[ 3654], 80.00th=[ 4490], 90.00th=[ 6849], 95.00th=[ 9372], | 99.00th=[ 15664], 99.50th=[ 19792], 99.90th=[ 57934], 99.95th=[170918], | 99.99th=[170918] bw ( KiB/s): min= 56, max= 304, per=25.17%, avg=176.67, stdev=65.69, samples=67 iops : min= 14, max= 76, avg=44.13, stdev=16.44, samples=67 write: IOPS=46, BW=185KiB/s (190kB/s)(6296KiB/33966msec) clat (usec): min=8665, max=60865, avg=17863.32, stdev=6050.77 lat (usec): min=8666, max=60865, avg=17864.14, stdev=6050.76 clat percentiles (usec): | 1.00th=[ 9503], 5.00th=[10945], 10.00th=[11600], 20.00th=[12780], | 30.00th=[13960], 40.00th=[15401], 50.00th=[17171], 60.00th=[18482], | 70.00th=[19530], 80.00th=[21365], 90.00th=[25035], 95.00th=[30016], | 99.00th=[36963], 99.50th=[39584], 99.90th=[59507], 99.95th=[61080], | 99.99th=[61080] bw ( KiB/s): min= 48, max= 256, per=25.14%, avg=184.79, stdev=49.62, samples=67 iops : min= 12, max= 64, avg=46.16, stdev=12.40, samples=67 lat (msec) : 2=8.92%, 4=27.86%, 10=11.23%, 20=37.89%, 50=13.90% lat (msec) : 100=0.16%, 250=0.03% cpu : usr=0.08%, sys=5.55%, ctx=3675, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9616: Thu Apr 17 07:59:05 2025 read: IOPS=43, BW=174KiB/s (178kB/s)(5940KiB/34100msec) clat (usec): min=1585, max=169680, avg=3800.53, stdev=5413.66 lat (usec): min=1585, max=169681, avg=3801.20, stdev=5413.67 clat percentiles (usec): | 1.00th=[ 1762], 5.00th=[ 1876], 10.00th=[ 1926], 20.00th=[ 2024], | 30.00th=[ 2114], 40.00th=[ 2245], 50.00th=[ 2442], 60.00th=[ 2769], | 70.00th=[ 3621], 80.00th=[ 4555], 90.00th=[ 6718], 95.00th=[ 9372], | 99.00th=[ 16450], 99.50th=[ 22152], 99.90th=[ 38536], 99.95th=[168821], | 99.99th=[168821] bw ( KiB/s): min= 16, max= 328, per=24.83%, avg=174.34, stdev=65.80, samples=68 iops : min= 4, max= 82, avg=43.57, stdev=16.46, samples=68 write: IOPS=46, BW=186KiB/s (191kB/s)(6348KiB/34100msec) clat (usec): min=8227, max=59806, avg=17844.77, stdev=6016.20 lat (usec): min=8228, max=59807, avg=17845.60, stdev=6016.22 clat percentiles (usec): | 1.00th=[ 9765], 5.00th=[10945], 10.00th=[11731], 20.00th=[12911], | 30.00th=[13960], 40.00th=[15401], 50.00th=[16909], 60.00th=[18482], | 70.00th=[19792], 80.00th=[21365], 90.00th=[25560], 95.00th=[29754], | 99.00th=[37487], 99.50th=[42206], 99.90th=[54264], 99.95th=[60031], | 99.99th=[60031] bw ( KiB/s): min= 56, max= 264, per=25.30%, avg=185.99, stdev=49.39, samples=68 iops : min= 14, max= 66, avg=46.49, stdev=12.37, samples=68 lat (msec) : 2=8.56%, 4=28.22%, 10=10.51%, 20=37.89%, 50=14.65% lat (msec) : 100=0.13%, 250=0.03% cpu : usr=0.12%, sys=5.45%, ctx=3629, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9617: Thu Apr 17 07:59:05 2025 read: IOPS=44, BW=176KiB/s (181kB/s)(6028KiB/34191msec) clat (usec): min=1617, max=20874, avg=3702.89, stdev=2579.21 lat (usec): min=1618, max=20875, avg=3703.57, stdev=2579.21 clat percentiles (usec): | 1.00th=[ 1745], 5.00th=[ 1860], 10.00th=[ 1942], 20.00th=[ 2040], | 30.00th=[ 2147], 40.00th=[ 2278], 50.00th=[ 2540], 60.00th=[ 3097], | 70.00th=[ 3851], 80.00th=[ 4883], 90.00th=[ 7308], 95.00th=[ 8848], | 99.00th=[13829], 99.50th=[16188], 99.90th=[20317], 99.95th=[20841], | 99.99th=[20841] bw ( KiB/s): min= 48, max= 312, per=25.12%, avg=176.34, stdev=57.57, samples=68 iops : min= 12, max= 78, avg=44.07, stdev=14.40, samples=68 write: IOPS=45, BW=183KiB/s (187kB/s)(6260KiB/34191msec) clat (msec): min=8, max=199, avg=18.19, stdev= 7.70 lat (msec): min=8, max=199, avg=18.19, stdev= 7.70 clat percentiles (msec): | 1.00th=[ 10], 5.00th=[ 12], 10.00th=[ 12], 20.00th=[ 13], | 30.00th=[ 15], 40.00th=[ 16], 50.00th=[ 18], 60.00th=[ 19], | 70.00th=[ 20], 80.00th=[ 22], 90.00th=[ 27], 95.00th=[ 31], | 99.00th=[ 40], 99.50th=[ 45], 99.90th=[ 60], 99.95th=[ 201], | 99.99th=[ 201] bw ( KiB/s): min= 48, max= 256, per=24.86%, avg=182.69, stdev=48.91, samples=68 iops : min= 12, max= 64, avg=45.66, stdev=12.24, samples=68 lat (msec) : 2=7.88%, 4=27.93%, 10=12.43%, 20=37.17%, 50=14.45% lat (msec) : 100=0.10%, 250=0.03% cpu : usr=0.12%, sys=5.56%, ctx=3664, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9618: Thu Apr 17 07:59:05 2025 read: IOPS=44, BW=177KiB/s (182kB/s)(6052KiB/34124msec) clat (usec): min=1465, max=57165, avg=3678.79, stdev=3294.53 lat (usec): min=1466, max=57165, avg=3679.44, stdev=3294.53 clat percentiles (usec): | 1.00th=[ 1745], 5.00th=[ 1860], 10.00th=[ 1926], 20.00th=[ 2024], | 30.00th=[ 2114], 40.00th=[ 2212], 50.00th=[ 2442], 60.00th=[ 2966], | 70.00th=[ 3687], 80.00th=[ 4555], 90.00th=[ 6718], 95.00th=[ 9241], | 99.00th=[15270], 99.50th=[17957], 99.90th=[41681], 99.95th=[57410], | 99.99th=[57410] bw ( KiB/s): min= 32, max= 312, per=25.26%, avg=177.32, stdev=60.57, samples=68 iops : min= 8, max= 78, avg=44.26, stdev=15.14, samples=68 write: IOPS=45, BW=183KiB/s (187kB/s)(6236KiB/34124msec) clat (msec): min=8, max=220, avg=18.23, stdev= 7.74 lat (msec): min=8, max=220, avg=18.23, stdev= 7.74 clat percentiles (msec): | 1.00th=[ 10], 5.00th=[ 12], 10.00th=[ 12], 20.00th=[ 14], | 30.00th=[ 15], 40.00th=[ 16], 50.00th=[ 18], 60.00th=[ 19], | 70.00th=[ 20], 80.00th=[ 22], 90.00th=[ 26], 95.00th=[ 31], | 99.00th=[ 37], 99.50th=[ 40], 99.90th=[ 49], 99.95th=[ 222], | 99.99th=[ 222] bw ( KiB/s): min= 48, max= 256, per=24.81%, avg=182.38, stdev=47.00, samples=68 iops : min= 12, max= 64, avg=45.53, stdev=11.75, samples=68 lat (msec) : 2=8.66%, 4=28.45%, 10=10.74%, 20=37.37%, 50=14.71% lat (msec) : 100=0.03%, 250=0.03% cpu : usr=0.08%, sys=5.54%, ctx=3697, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=702KiB/s (719kB/s), 174KiB/s-177KiB/s (178kB/s-182kB/s), io=23.4MiB (24.6MB), run=33966-34191msec WRITE: bw=735KiB/s (753kB/s), 183KiB/s-186KiB/s (187kB/s-191kB/s), io=24.6MiB (25.7MB), run=33966-34191msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=9656: Thu Apr 17 07:59:09 2025 read: IOPS=147, BW=2365KiB/s (2422kB/s)(5920KiB/2503msec) clat (usec): min=57, max=17530, avg=3391.46, stdev=2378.10 lat (usec): min=58, max=17531, avg=3392.06, stdev=2378.14 clat percentiles (usec): | 1.00th=[ 62], 5.00th=[ 71], 10.00th=[ 82], 20.00th=[ 947], | 30.00th=[ 2474], 40.00th=[ 2868], 50.00th=[ 3261], 60.00th=[ 3589], | 70.00th=[ 4293], 80.00th=[ 5080], 90.00th=[ 6259], 95.00th=[ 7373], | 99.00th=[ 9896], 99.50th=[13435], 99.90th=[17433], 99.95th=[17433], | 99.99th=[17433] bw ( KiB/s): min= 2048, max= 2784, per=27.91%, avg=2361.60, stdev=285.68, samples=5 iops : min= 128, max= 174, avg=147.60, stdev=17.85, samples=5 write: IOPS=159, BW=2544KiB/s (2605kB/s)(6368KiB/2503msec) clat (usec): min=651, max=31042, avg=3077.50, stdev=3460.64 lat (usec): min=652, max=31043, avg=3078.39, stdev=3460.66 clat percentiles (usec): | 1.00th=[ 668], 5.00th=[ 766], 10.00th=[ 816], 20.00th=[ 881], | 30.00th=[ 979], 40.00th=[ 1123], 50.00th=[ 1434], 60.00th=[ 2376], | 70.00th=[ 3425], 80.00th=[ 4948], 90.00th=[ 7373], 95.00th=[ 8979], | 99.00th=[16319], 99.50th=[24249], 99.90th=[31065], 99.95th=[31065], | 99.99th=[31065] bw ( KiB/s): min= 2176, max= 3392, per=28.55%, avg=2540.80, stdev=490.34, samples=5 iops : min= 136, max= 212, avg=158.80, stdev=30.65, samples=5 lat (usec) : 100=7.03%, 250=2.08%, 500=0.39%, 750=1.95%, 1000=14.58% lat (msec) : 2=13.15%, 4=31.51%, 10=26.69%, 20=2.34%, 50=0.26% cpu : usr=0.12%, sys=29.82%, ctx=1544, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9658: Thu Apr 17 07:59:09 2025 read: IOPS=126, BW=2029KiB/s (2078kB/s)(5744KiB/2831msec) clat (usec): min=57, max=44719, avg=3226.68, stdev=3592.91 lat (usec): min=57, max=44720, avg=3227.30, stdev=3592.92 clat percentiles (usec): | 1.00th=[ 61], 5.00th=[ 70], 10.00th=[ 72], 20.00th=[ 85], | 30.00th=[ 167], 40.00th=[ 2474], 50.00th=[ 2900], 60.00th=[ 3523], | 70.00th=[ 4490], 80.00th=[ 5276], 90.00th=[ 6456], 95.00th=[ 7767], | 99.00th=[13566], 99.50th=[21365], 99.90th=[44827], 99.95th=[44827], | 99.99th=[44827] bw ( KiB/s): min= 288, max= 2656, per=22.30%, avg=1886.20, stdev=930.99, samples=5 iops : min= 18, max= 166, avg=117.60, stdev=58.11, samples=5 write: IOPS=144, BW=2312KiB/s (2367kB/s)(6544KiB/2831msec) clat (usec): min=645, max=32223, avg=3778.01, stdev=5048.24 lat (usec): min=646, max=32224, avg=3778.88, stdev=5048.29 clat percentiles (usec): | 1.00th=[ 676], 5.00th=[ 725], 10.00th=[ 750], 20.00th=[ 824], | 30.00th=[ 922], 40.00th=[ 1045], 50.00th=[ 1434], 60.00th=[ 2933], | 70.00th=[ 3916], 80.00th=[ 5735], 90.00th=[ 8717], 95.00th=[11994], | 99.00th=[26608], 99.50th=[28181], 99.90th=[32113], 99.95th=[32113], | 99.99th=[32113] bw ( KiB/s): min= 512, max= 3097, per=23.05%, avg=2052.00, stdev=955.30, samples=5 iops : min= 32, max= 193, avg=128.00, stdev=59.56, samples=5 lat (usec) : 100=12.24%, 250=2.34%, 750=5.34%, 1000=14.19% lat (msec) : 2=10.16%, 4=23.96%, 10=27.60%, 20=2.21%, 50=1.95% cpu : usr=0.07%, sys=25.23%, ctx=1375, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9660: Thu Apr 17 07:59:09 2025 read: IOPS=153, BW=2461KiB/s (2520kB/s)(6256KiB/2542msec) clat (usec): min=59, max=33270, avg=3529.84, stdev=3587.99 lat (usec): min=60, max=33271, avg=3531.08, stdev=3587.77 clat percentiles (usec): | 1.00th=[ 62], 5.00th=[ 71], 10.00th=[ 78], 20.00th=[ 123], | 30.00th=[ 2311], 40.00th=[ 2704], 50.00th=[ 3032], 60.00th=[ 3621], | 70.00th=[ 4359], 80.00th=[ 5342], 90.00th=[ 6587], 95.00th=[ 7767], | 99.00th=[17695], 99.50th=[32900], 99.90th=[33162], 99.95th=[33162], | 99.99th=[33162] bw ( KiB/s): min= 1568, max= 3136, per=29.34%, avg=2482.00, stdev=569.82, samples=5 iops : min= 98, max= 196, avg=155.00, stdev=35.59, samples=5 write: IOPS=148, BW=2373KiB/s (2430kB/s)(6032KiB/2542msec) clat (usec): min=662, max=29053, avg=2885.69, stdev=3116.81 lat (usec): min=663, max=29055, avg=2886.61, stdev=3116.83 clat percentiles (usec): | 1.00th=[ 676], 5.00th=[ 742], 10.00th=[ 775], 20.00th=[ 840], | 30.00th=[ 947], 40.00th=[ 1037], 50.00th=[ 1287], 60.00th=[ 2376], | 70.00th=[ 3294], 80.00th=[ 5014], 90.00th=[ 6915], 95.00th=[ 8356], | 99.00th=[13304], 99.50th=[23462], 99.90th=[28967], 99.95th=[28967], | 99.99th=[28967] bw ( KiB/s): min= 1920, max= 2816, per=26.45%, avg=2354.20, stdev=336.51, samples=5 iops : min= 120, max= 176, avg=147.00, stdev=21.10, samples=5 lat (usec) : 100=9.11%, 250=2.73%, 500=0.13%, 750=2.86%, 1000=15.23% lat (msec) : 2=11.46%, 4=28.65%, 10=27.47%, 20=1.69%, 50=0.65% cpu : usr=0.24%, sys=27.74%, ctx=1555, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9661: Thu Apr 17 07:59:09 2025 read: IOPS=148, BW=2375KiB/s (2432kB/s)(6032KiB/2540msec) clat (usec): min=61, max=45882, avg=3167.99, stdev=3215.79 lat (usec): min=61, max=45883, avg=3169.05, stdev=3215.64 clat percentiles (usec): | 1.00th=[ 63], 5.00th=[ 71], 10.00th=[ 83], 20.00th=[ 109], | 30.00th=[ 2278], 40.00th=[ 2671], 50.00th=[ 2966], 60.00th=[ 3425], | 70.00th=[ 4113], 80.00th=[ 4948], 90.00th=[ 5866], 95.00th=[ 7177], | 99.00th=[ 9634], 99.50th=[16909], 99.90th=[45876], 99.95th=[45876], | 99.99th=[45876] bw ( KiB/s): min= 1440, max= 2752, per=28.13%, avg=2379.60, stdev=537.16, samples=5 iops : min= 90, max= 172, avg=148.60, stdev=33.52, samples=5 write: IOPS=153, BW=2463KiB/s (2522kB/s)(6256KiB/2540msec) clat (usec): min=642, max=23894, avg=3111.27, stdev=3574.29 lat (usec): min=643, max=23895, avg=3112.13, stdev=3574.32 clat percentiles (usec): | 1.00th=[ 685], 5.00th=[ 750], 10.00th=[ 791], 20.00th=[ 840], | 30.00th=[ 938], 40.00th=[ 1090], 50.00th=[ 1450], 60.00th=[ 2311], | 70.00th=[ 3458], 80.00th=[ 5014], 90.00th=[ 7308], 95.00th=[10028], | 99.00th=[21103], 99.50th=[22152], 99.90th=[23987], 99.95th=[23987], | 99.99th=[23987] bw ( KiB/s): min= 1664, max= 3072, per=27.96%, avg=2488.60, stdev=531.96, samples=5 iops : min= 104, max= 192, avg=155.40, stdev=33.27, samples=5 lat (usec) : 100=8.59%, 250=3.91%, 500=0.13%, 750=2.99%, 1000=14.71% lat (msec) : 2=12.37%, 4=28.91%, 10=25.39%, 20=2.34%, 50=0.65% cpu : usr=0.08%, sys=27.14%, ctx=1536, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=8461KiB/s (8664kB/s), 2029KiB/s-2461KiB/s (2078kB/s-2520kB/s), io=23.4MiB (24.5MB), run=2503-2831msec WRITE: bw=8901KiB/s (9115kB/s), 2312KiB/s-2544KiB/s (2367kB/s-2605kB/s), io=24.6MiB (25.8MB), run=2503-2831msec rand-rw: (groupid=0, jobs=1): err= 0: pid=9654: Thu Apr 17 07:59:15 2025 read: IOPS=43, BW=697KiB/s (713kB/s)(5920KiB/8497msec) clat (usec): min=1852, max=37661, avg=3891.78, stdev=4036.52 lat (usec): min=1853, max=37662, avg=3892.44, stdev=4036.55 clat percentiles (usec): | 1.00th=[ 1909], 5.00th=[ 2040], 10.00th=[ 2114], 20.00th=[ 2180], | 30.00th=[ 2245], 40.00th=[ 2311], 50.00th=[ 2442], 60.00th=[ 2606], | 70.00th=[ 2933], 80.00th=[ 4015], 90.00th=[ 7635], 95.00th=[12387], | 99.00th=[21103], 99.50th=[33817], 99.90th=[37487], 99.95th=[37487], | 99.99th=[37487] bw ( KiB/s): min= 416, max= 1120, per=24.35%, avg=677.75, stdev=220.04, samples=16 iops : min= 26, max= 70, avg=42.25, stdev=13.81, samples=16 write: IOPS=46, BW=749KiB/s (767kB/s)(6368KiB/8497msec) clat (usec): min=8881, max=42621, avg=17603.26, stdev=5369.25 lat (usec): min=8882, max=42624, avg=17604.45, stdev=5369.37 clat percentiles (usec): | 1.00th=[10290], 5.00th=[11207], 10.00th=[11731], 20.00th=[13042], | 30.00th=[14091], 40.00th=[15664], 50.00th=[17171], 60.00th=[18220], | 70.00th=[19006], 80.00th=[20317], 90.00th=[23987], 95.00th=[28443], | 99.00th=[36963], 99.50th=[40633], 99.90th=[42730], 99.95th=[42730], | 99.99th=[42730] bw ( KiB/s): min= 384, max= 958, per=25.06%, avg=733.75, stdev=200.40, samples=16 iops : min= 24, max= 59, avg=45.75, stdev=12.48, samples=16 lat (msec) : 2=1.56%, 4=36.85%, 10=6.51%, 20=43.10%, 50=11.98% cpu : usr=0.08%, sys=6.21%, ctx=884, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9655: Thu Apr 17 07:59:15 2025 read: IOPS=41, BW=668KiB/s (684kB/s)(5744KiB/8604msec) clat (usec): min=1895, max=33486, avg=3902.27, stdev=3832.90 lat (usec): min=1895, max=33487, avg=3902.90, stdev=3832.91 clat percentiles (usec): | 1.00th=[ 1926], 5.00th=[ 2040], 10.00th=[ 2114], 20.00th=[ 2180], | 30.00th=[ 2278], 40.00th=[ 2343], 50.00th=[ 2474], 60.00th=[ 2671], | 70.00th=[ 3064], 80.00th=[ 4015], 90.00th=[ 7504], 95.00th=[12649], | 99.00th=[21627], 99.50th=[26346], 99.90th=[33424], 99.95th=[33424], | 99.99th=[33424] bw ( KiB/s): min= 288, max= 1152, per=23.87%, avg=664.35, stdev=231.75, samples=17 iops : min= 18, max= 72, avg=41.47, stdev=14.49, samples=17 write: IOPS=47, BW=761KiB/s (779kB/s)(6544KiB/8604msec) clat (usec): min=10009, max=61178, avg=17457.49, stdev=5923.98 lat (usec): min=10010, max=61180, avg=17458.68, stdev=5924.02 clat percentiles (usec): | 1.00th=[10290], 5.00th=[11076], 10.00th=[11469], 20.00th=[12518], | 30.00th=[13829], 40.00th=[15139], 50.00th=[16712], 60.00th=[18220], | 70.00th=[19268], 80.00th=[20317], 90.00th=[24249], 95.00th=[27657], | 99.00th=[39584], 99.50th=[41681], 99.90th=[61080], 99.95th=[61080], | 99.99th=[61080] bw ( KiB/s): min= 448, max= 992, per=25.90%, avg=758.47, stdev=191.98, samples=17 iops : min= 28, max= 62, avg=47.35, stdev=12.03, samples=17 lat (msec) : 2=1.17%, 4=36.20%, 10=6.12%, 20=43.75%, 50=12.50% lat (msec) : 100=0.26% cpu : usr=0.05%, sys=6.28%, ctx=898, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9657: Thu Apr 17 07:59:15 2025 read: IOPS=48, BW=778KiB/s (796kB/s)(6256KiB/8045msec) clat (usec): min=1882, max=37059, avg=3590.29, stdev=3378.73 lat (usec): min=1883, max=37059, avg=3591.11, stdev=3378.67 clat percentiles (usec): | 1.00th=[ 1958], 5.00th=[ 2040], 10.00th=[ 2114], 20.00th=[ 2212], | 30.00th=[ 2311], 40.00th=[ 2376], 50.00th=[ 2507], 60.00th=[ 2638], | 70.00th=[ 2868], 80.00th=[ 3359], 90.00th=[ 5800], 95.00th=[11207], | 99.00th=[19792], 99.50th=[21103], 99.90th=[36963], 99.95th=[36963], | 99.99th=[36963] bw ( KiB/s): min= 352, max= 1440, per=28.09%, avg=781.75, stdev=329.52, samples=16 iops : min= 22, max= 90, avg=48.75, stdev=20.63, samples=16 write: IOPS=46, BW=750KiB/s (768kB/s)(6032KiB/8045msec) clat (usec): min=9505, max=56207, avg=17570.93, stdev=5909.17 lat (usec): min=9506, max=56208, avg=17572.05, stdev=5909.26 clat percentiles (usec): | 1.00th=[10421], 5.00th=[11207], 10.00th=[11731], 20.00th=[12649], | 30.00th=[13698], 40.00th=[15401], 50.00th=[16581], 60.00th=[17695], | 70.00th=[19006], 80.00th=[20841], 90.00th=[24511], 95.00th=[31065], | 99.00th=[36439], 99.50th=[39584], 99.90th=[56361], 99.95th=[56361], | 99.99th=[56361] bw ( KiB/s): min= 448, max= 928, per=25.54%, avg=747.75, stdev=175.17, samples=16 iops : min= 28, max= 58, avg=46.62, stdev=10.93, samples=16 lat (msec) : 2=1.56%, 4=40.62%, 10=5.86%, 20=39.84%, 50=11.98% lat (msec) : 100=0.13% cpu : usr=0.11%, sys=6.76%, ctx=921, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9659: Thu Apr 17 07:59:15 2025 read: IOPS=44, BW=715KiB/s (732kB/s)(6032KiB/8433msec) clat (usec): min=1852, max=21016, avg=3481.69, stdev=2822.06 lat (usec): min=1852, max=21016, avg=3482.48, stdev=2822.29 clat percentiles (usec): | 1.00th=[ 1909], 5.00th=[ 2040], 10.00th=[ 2114], 20.00th=[ 2180], | 30.00th=[ 2245], 40.00th=[ 2311], 50.00th=[ 2409], 60.00th=[ 2540], | 70.00th=[ 2868], 80.00th=[ 3490], 90.00th=[ 6587], 95.00th=[10290], | 99.00th=[15533], 99.50th=[19268], 99.90th=[21103], 99.95th=[21103], | 99.99th=[21103] bw ( KiB/s): min= 288, max= 1248, per=24.93%, avg=693.87, stdev=247.00, samples=16 iops : min= 18, max= 78, avg=43.31, stdev=15.45, samples=16 write: IOPS=46, BW=742KiB/s (760kB/s)(6256KiB/8433msec) clat (usec): min=10114, max=64233, avg=18098.99, stdev=6284.91 lat (usec): min=10115, max=64237, avg=18100.20, stdev=6285.03 clat percentiles (usec): | 1.00th=[10421], 5.00th=[11076], 10.00th=[11731], 20.00th=[12649], | 30.00th=[14091], 40.00th=[16057], 50.00th=[17433], 60.00th=[18744], | 70.00th=[19792], 80.00th=[21627], 90.00th=[25297], 95.00th=[28705], | 99.00th=[41157], 99.50th=[51119], 99.90th=[64226], 99.95th=[64226], | 99.99th=[64226] bw ( KiB/s): min= 416, max= 928, per=24.93%, avg=729.88, stdev=183.02, samples=16 iops : min= 26, max= 58, avg=45.56, stdev=11.46, samples=16 lat (msec) : 2=1.69%, 4=38.80%, 10=6.12%, 20=38.93%, 50=14.19% lat (msec) : 100=0.26% cpu : usr=0.13%, sys=6.34%, ctx=907, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2784KiB/s (2851kB/s), 668KiB/s-778KiB/s (684kB/s-796kB/s), io=23.4MiB (24.5MB), run=8045-8604msec WRITE: bw=2929KiB/s (2999kB/s), 742KiB/s-761KiB/s (760kB/s-779kB/s), io=24.6MiB (25.8MB), run=8045-8604msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=9677: Thu Apr 17 07:59:17 2025 read: IOPS=5, BW=5314KiB/s (5441kB/s)(7168KiB/1349msec) clat (usec): min=533, max=44861, avg=32046.07, stdev=14608.91 lat (usec): min=534, max=44862, avg=32047.05, stdev=14608.88 clat percentiles (usec): | 1.00th=[ 537], 5.00th=[ 537], 10.00th=[ 537], 20.00th=[31851], | 30.00th=[32375], 40.00th=[32375], 50.00th=[36439], 60.00th=[38011], | 70.00th=[38011], 80.00th=[40109], 90.00th=[44827], 95.00th=[44827], | 99.00th=[44827], 99.50th=[44827], 99.90th=[44827], 99.95th=[44827], | 99.99th=[44827] bw ( KiB/s): min=12288, max=12288, per=92.29%, avg=12288.00, stdev= 0.00, samples=1 iops : min= 12, max= 12, avg=12.00, stdev= 0.00, samples=1 write: IOPS=3, BW=3795KiB/s (3886kB/s)(5120KiB/1349msec) clat (msec): min=25, max=907, avg=220.54, stdev=384.44 lat (msec): min=25, max=907, avg=220.57, stdev=384.43 clat percentiles (msec): | 1.00th=[ 26], 5.00th=[ 26], 10.00th=[ 26], 20.00th=[ 26], | 30.00th=[ 43], 40.00th=[ 43], 50.00th=[ 62], 60.00th=[ 62], | 70.00th=[ 67], 80.00th=[ 67], 90.00th=[ 911], 95.00th=[ 911], | 99.00th=[ 911], 99.50th=[ 911], 99.90th=[ 911], 99.95th=[ 911], | 99.99th=[ 911] bw ( KiB/s): min= 8192, max= 8192, per=43.94%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 lat (usec) : 750=8.33% lat (msec) : 50=66.67%, 100=16.67%, 1000=8.33% cpu : usr=0.00%, sys=12.91%, ctx=763, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9678: Thu Apr 17 07:59:17 2025 read: IOPS=1, BW=1332KiB/s (1364kB/s)(2048KiB/1538msec) clat (usec): min=44437, max=49177, avg=46807.21, stdev=3351.54 lat (usec): min=44438, max=49177, avg=46808.16, stdev=3351.18 clat percentiles (usec): | 1.00th=[44303], 5.00th=[44303], 10.00th=[44303], 20.00th=[44303], | 30.00th=[44303], 40.00th=[44303], 50.00th=[44303], 60.00th=[49021], | 70.00th=[49021], 80.00th=[49021], 90.00th=[49021], 95.00th=[49021], | 99.00th=[49021], 99.50th=[49021], 99.90th=[49021], 99.95th=[49021], | 99.99th=[49021] bw ( KiB/s): min= 4096, max= 4096, per=30.76%, avg=4096.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 write: IOPS=6, BW=6658KiB/s (6818kB/s)(10.0MiB/1538msec) clat (msec): min=31, max=843, avg=138.33, stdev=248.81 lat (msec): min=31, max=843, avg=138.38, stdev=248.80 clat percentiles (msec): | 1.00th=[ 32], 5.00th=[ 32], 10.00th=[ 32], 20.00th=[ 40], | 30.00th=[ 43], 40.00th=[ 49], 50.00th=[ 55], 60.00th=[ 64], | 70.00th=[ 85], 80.00th=[ 85], 90.00th=[ 101], 95.00th=[ 844], | 99.00th=[ 844], 99.50th=[ 844], 99.90th=[ 844], 99.95th=[ 844], | 99.99th=[ 844] bw ( KiB/s): min= 4055, max=10240, per=32.88%, avg=6130.33, stdev=3559.13, samples=3 iops : min= 3, max= 10, avg= 5.67, stdev= 3.79, samples=3 lat (msec) : 50=50.00%, 100=33.33%, 250=8.33%, 1000=8.33% cpu : usr=0.00%, sys=12.69%, ctx=348, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9679: Thu Apr 17 07:59:17 2025 read: IOPS=4, BW=4538KiB/s (4647kB/s)(6144KiB/1354msec) clat (usec): min=26431, max=37632, avg=32115.73, stdev=4461.94 lat (usec): min=26432, max=37632, avg=32116.43, stdev=4461.95 clat percentiles (usec): | 1.00th=[26346], 5.00th=[26346], 10.00th=[26346], 20.00th=[27395], | 30.00th=[27395], 40.00th=[32113], 50.00th=[32113], 60.00th=[33817], | 70.00th=[35390], 80.00th=[35390], 90.00th=[37487], 95.00th=[37487], | 99.00th=[37487], 99.50th=[37487], 99.90th=[37487], 99.95th=[37487], | 99.99th=[37487] bw ( KiB/s): min= 2048, max= 8192, per=38.45%, avg=5120.00, stdev=4344.46, samples=2 iops : min= 2, max= 8, avg= 5.00, stdev= 4.24, samples=2 write: IOPS=4, BW=4538KiB/s (4647kB/s)(6144KiB/1354msec) clat (msec): min=27, max=793, avg=182.82, stdev=301.94 lat (msec): min=27, max=793, avg=182.86, stdev=301.93 clat percentiles (msec): | 1.00th=[ 28], 5.00th=[ 28], 10.00th=[ 28], 20.00th=[ 29], | 30.00th=[ 29], 40.00th=[ 35], 50.00th=[ 35], 60.00th=[ 88], | 70.00th=[ 125], 80.00th=[ 125], 90.00th=[ 793], 95.00th=[ 793], | 99.00th=[ 793], 99.50th=[ 793], 99.90th=[ 793], 99.95th=[ 793], | 99.99th=[ 793] bw ( KiB/s): min=10240, max=10240, per=54.93%, avg=10240.00, stdev= 0.00, samples=1 iops : min= 10, max= 10, avg=10.00, stdev= 0.00, samples=1 lat (msec) : 50=75.00%, 100=8.33%, 250=8.33%, 1000=8.33% cpu : usr=0.00%, sys=16.70%, ctx=1191, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9680: Thu Apr 17 07:59:17 2025 read: IOPS=3, BW=3380KiB/s (3461kB/s)(5120KiB/1515msec) clat (usec): min=468, max=801466, avg=179633.07, stdev=348420.76 lat (usec): min=469, max=801467, avg=179634.62, stdev=348420.28 clat percentiles (usec): | 1.00th=[ 469], 5.00th=[ 469], 10.00th=[ 469], 20.00th=[ 469], | 30.00th=[ 553], 40.00th=[ 553], 50.00th=[ 46924], 60.00th=[ 46924], | 70.00th=[ 49021], 80.00th=[ 49021], 90.00th=[801113], 95.00th=[801113], | 99.00th=[801113], 99.50th=[801113], 99.90th=[801113], 99.95th=[801113], | 99.99th=[801113] bw ( KiB/s): min= 8192, max= 8192, per=61.52%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=4, BW=4731KiB/s (4845kB/s)(7168KiB/1515msec) clat (msec): min=40, max=107, avg=77.15, stdev=26.37 lat (msec): min=40, max=107, avg=77.20, stdev=26.38 clat percentiles (msec): | 1.00th=[ 41], 5.00th=[ 41], 10.00th=[ 41], 20.00th=[ 44], | 30.00th=[ 78], 40.00th=[ 78], 50.00th=[ 84], 60.00th=[ 84], | 70.00th=[ 84], 80.00th=[ 105], 90.00th=[ 108], 95.00th=[ 108], | 99.00th=[ 108], 99.50th=[ 108], 99.90th=[ 108], 99.95th=[ 108], | 99.99th=[ 108] bw ( KiB/s): min= 4096, max= 8192, per=32.96%, avg=6144.00, stdev=2896.31, samples=2 iops : min= 4, max= 8, avg= 6.00, stdev= 2.83, samples=2 lat (usec) : 500=8.33%, 750=8.33% lat (msec) : 50=33.33%, 100=25.00%, 250=16.67%, 1000=8.33% cpu : usr=0.00%, sys=18.89%, ctx=596, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=13.0MiB/s (13.6MB/s), 1332KiB/s-5314KiB/s (1364kB/s-5441kB/s), io=20.0MiB (20.0MB), run=1349-1538msec WRITE: bw=18.2MiB/s (19.1MB/s), 3795KiB/s-6658KiB/s (3886kB/s-6818kB/s), io=28.0MiB (29.4MB), run=1349-1538msec rand-rw: (groupid=0, jobs=1): err= 0: pid=9673: Thu Apr 17 07:59:17 2025 read: IOPS=5, BW=5385KiB/s (5515kB/s)(7168KiB/1331msec) clat (usec): min=13158, max=47607, avg=24768.45, stdev=11611.92 lat (usec): min=13159, max=47608, avg=24769.47, stdev=11611.94 clat percentiles (usec): | 1.00th=[13173], 5.00th=[13173], 10.00th=[13173], 20.00th=[15795], | 30.00th=[16909], 40.00th=[16909], 50.00th=[26084], 60.00th=[26346], | 70.00th=[26346], 80.00th=[27395], 90.00th=[47449], 95.00th=[47449], | 99.00th=[47449], 99.50th=[47449], 99.90th=[47449], 99.95th=[47449], | 99.99th=[47449] bw ( KiB/s): min=12288, max=12288, per=92.46%, avg=12288.00, stdev= 0.00, samples=1 iops : min= 12, max= 12, avg=12.00, stdev= 0.00, samples=1 write: IOPS=3, BW=3847KiB/s (3939kB/s)(5120KiB/1331msec) clat (msec): min=43, max=908, avg=226.95, stdev=381.18 lat (msec): min=43, max=908, avg=227.00, stdev=381.16 clat percentiles (msec): | 1.00th=[ 44], 5.00th=[ 44], 10.00th=[ 44], 20.00th=[ 44], | 30.00th=[ 55], 40.00th=[ 55], 50.00th=[ 57], 60.00th=[ 57], | 70.00th=[ 72], 80.00th=[ 72], 90.00th=[ 911], 95.00th=[ 911], | 99.00th=[ 911], 99.50th=[ 911], 99.90th=[ 911], 99.95th=[ 911], | 99.99th=[ 911] bw ( KiB/s): min= 8192, max= 8192, per=44.03%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 lat (msec) : 20=25.00%, 50=41.67%, 100=25.00%, 1000=8.33% cpu : usr=0.00%, sys=5.79%, ctx=64, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9674: Thu Apr 17 07:59:17 2025 read: IOPS=1, BW=1337KiB/s (1369kB/s)(2048KiB/1532msec) clat (usec): min=18150, max=32521, avg=25335.97, stdev=10161.25 lat (usec): min=18152, max=32521, avg=25337.42, stdev=10160.37 clat percentiles (usec): | 1.00th=[18220], 5.00th=[18220], 10.00th=[18220], 20.00th=[18220], | 30.00th=[18220], 40.00th=[18220], 50.00th=[18220], 60.00th=[32637], | 70.00th=[32637], 80.00th=[32637], 90.00th=[32637], 95.00th=[32637], | 99.00th=[32637], 99.50th=[32637], 99.90th=[32637], 99.95th=[32637], | 99.99th=[32637] bw ( KiB/s): min= 4096, max= 4096, per=30.82%, avg=4096.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 write: IOPS=6, BW=6684KiB/s (6844kB/s)(10.0MiB/1532msec) clat (msec): min=40, max=836, avg=147.21, stdev=243.96 lat (msec): min=40, max=837, avg=147.30, stdev=244.00 clat percentiles (msec): | 1.00th=[ 41], 5.00th=[ 41], 10.00th=[ 41], 20.00th=[ 51], | 30.00th=[ 53], 40.00th=[ 53], 50.00th=[ 57], 60.00th=[ 62], | 70.00th=[ 114], 80.00th=[ 114], 90.00th=[ 123], 95.00th=[ 835], | 99.00th=[ 835], 99.50th=[ 835], 99.90th=[ 835], 99.95th=[ 835], | 99.99th=[ 835] bw ( KiB/s): min= 4096, max=10240, per=33.02%, avg=6144.00, stdev=3547.24, samples=3 iops : min= 4, max= 10, avg= 6.00, stdev= 3.46, samples=3 lat (msec) : 20=8.33%, 50=16.67%, 100=50.00%, 250=16.67%, 1000=8.33% cpu : usr=0.07%, sys=6.92%, ctx=36, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9675: Thu Apr 17 07:59:17 2025 read: IOPS=4, BW=4309KiB/s (4412kB/s)(6144KiB/1426msec) clat (usec): min=16148, max=70933, avg=33314.46, stdev=19658.58 lat (usec): min=16150, max=70934, avg=33315.47, stdev=19658.46 clat percentiles (usec): | 1.00th=[16188], 5.00th=[16188], 10.00th=[16188], 20.00th=[23200], | 30.00th=[23200], 40.00th=[23725], 50.00th=[23725], 60.00th=[29230], | 70.00th=[36439], 80.00th=[36439], 90.00th=[70779], 95.00th=[70779], | 99.00th=[70779], 99.50th=[70779], 99.90th=[70779], 99.95th=[70779], | 99.99th=[70779] bw ( KiB/s): min= 2048, max= 8175, per=38.46%, avg=5111.50, stdev=4332.44, samples=2 iops : min= 2, max= 7, avg= 4.50, stdev= 3.54, samples=2 write: IOPS=4, BW=4309KiB/s (4412kB/s)(6144KiB/1426msec) clat (msec): min=35, max=898, avg=197.41, stdev=344.05 lat (msec): min=35, max=898, avg=197.47, stdev=344.04 clat percentiles (msec): | 1.00th=[ 36], 5.00th=[ 36], 10.00th=[ 36], 20.00th=[ 47], | 30.00th=[ 47], 40.00th=[ 50], 50.00th=[ 50], 60.00th=[ 52], | 70.00th=[ 104], 80.00th=[ 104], 90.00th=[ 902], 95.00th=[ 902], | 99.00th=[ 902], 99.50th=[ 902], 99.90th=[ 902], 99.95th=[ 902], | 99.99th=[ 902] bw ( KiB/s): min=10219, max=10219, per=54.92%, avg=10219.00, stdev= 0.00, samples=1 iops : min= 9, max= 9, avg= 9.00, stdev= 0.00, samples=1 lat (msec) : 20=8.33%, 50=58.33%, 100=16.67%, 250=8.33%, 1000=8.33% cpu : usr=0.00%, sys=6.88%, ctx=46, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9676: Thu Apr 17 07:59:17 2025 read: IOPS=3, BW=3323KiB/s (3402kB/s)(5120KiB/1541msec) clat (msec): min=21, max=827, avg=195.13, stdev=353.89 lat (msec): min=21, max=827, avg=195.13, stdev=353.89 clat percentiles (msec): | 1.00th=[ 22], 5.00th=[ 22], 10.00th=[ 22], 20.00th=[ 22], | 30.00th=[ 35], 40.00th=[ 35], 50.00th=[ 40], 60.00th=[ 40], | 70.00th=[ 53], 80.00th=[ 53], 90.00th=[ 827], 95.00th=[ 827], | 99.00th=[ 827], 99.50th=[ 827], 99.90th=[ 827], 99.95th=[ 827], | 99.99th=[ 827] bw ( KiB/s): min= 4096, max= 4096, per=30.82%, avg=4096.00, stdev= 0.00, samples=2 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=2 write: IOPS=4, BW=4652KiB/s (4763kB/s)(7168KiB/1541msec) clat (msec): min=41, max=131, avg=76.45, stdev=29.37 lat (msec): min=41, max=131, avg=76.54, stdev=29.37 clat percentiles (msec): | 1.00th=[ 42], 5.00th=[ 42], 10.00th=[ 42], 20.00th=[ 58], | 30.00th=[ 61], 40.00th=[ 61], 50.00th=[ 71], 60.00th=[ 87], | 70.00th=[ 87], 80.00th=[ 88], 90.00th=[ 132], 95.00th=[ 132], | 99.00th=[ 132], 99.50th=[ 132], 99.90th=[ 132], 99.95th=[ 132], | 99.99th=[ 132] bw ( KiB/s): min= 6144, max= 8192, per=38.53%, avg=7168.00, stdev=1448.15, samples=2 iops : min= 6, max= 8, avg= 7.00, stdev= 1.41, samples=2 lat (msec) : 50=33.33%, 100=50.00%, 250=8.33%, 1000=8.33% cpu : usr=0.00%, sys=6.95%, ctx=56, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=12.0MiB/s (13.6MB/s), 1337KiB/s-5385KiB/s (1369kB/s-5515kB/s), io=20.0MiB (20.0MB), run=1331-1541msec WRITE: bw=18.2MiB/s (19.1MB/s), 3847KiB/s-6684KiB/s (3939kB/s-6844kB/s), io=28.0MiB (29.4MB), run=1331-1541msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=9687: Thu Apr 17 07:59:19 2025 read: IOPS=7, BW=31.5MiB/s (33.0MB/s)(12.0MiB/381msec) clat (msec): min=99, max=149, avg=117.21, stdev=27.85 lat (msec): min=99, max=149, avg=117.21, stdev=27.85 clat percentiles (msec): | 1.00th=[ 101], 5.00th=[ 101], 10.00th=[ 101], 20.00th=[ 101], | 30.00th=[ 101], 40.00th=[ 103], 50.00th=[ 103], 60.00th=[ 103], | 70.00th=[ 150], 80.00th=[ 150], 90.00th=[ 150], 95.00th=[ 150], | 99.00th=[ 150], 99.50th=[ 150], 99.90th=[ 150], 99.95th=[ 150], | 99.99th=[ 150] lat (msec) : 100=33.33%, 250=66.67% cpu : usr=0.00%, sys=22.11%, ctx=859, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9689: Thu Apr 17 07:59:19 2025 read: IOPS=1, BW=6113KiB/s (6260kB/s)(4096KiB/670msec) clat (nsec): min=148444k, max=148444k, avg=148443519.00, stdev= 0.00 lat (nsec): min=148445k, max=148445k, avg=148444560.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 148], 5.00th=[ 148], 10.00th=[ 148], 20.00th=[ 148], | 30.00th=[ 148], 40.00th=[ 148], 50.00th=[ 148], 60.00th=[ 148], | 70.00th=[ 148], 80.00th=[ 148], 90.00th=[ 148], 95.00th=[ 148], | 99.00th=[ 148], 99.50th=[ 148], 99.90th=[ 148], 99.95th=[ 148], | 99.99th=[ 148] bw ( KiB/s): min= 8175, max= 8175, per=33.43%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=2, BW=11.9MiB/s (12.5MB/s)(8192KiB/670msec) clat (msec): min=174, max=329, avg=251.73, stdev=109.30 lat (msec): min=174, max=329, avg=251.98, stdev=109.31 clat percentiles (msec): | 1.00th=[ 176], 5.00th=[ 176], 10.00th=[ 176], 20.00th=[ 176], | 30.00th=[ 176], 40.00th=[ 176], 50.00th=[ 176], 60.00th=[ 330], | 70.00th=[ 330], 80.00th=[ 330], 90.00th=[ 330], 95.00th=[ 330], | 99.00th=[ 330], 99.50th=[ 330], 99.90th=[ 330], 99.95th=[ 330], | 99.99th=[ 330] bw ( KiB/s): min= 8175, max= 8175, per=16.72%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.00%, sys=30.04%, ctx=475, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9691: Thu Apr 17 07:59:19 2025 write: IOPS=4, BW=18.5MiB/s (19.4MB/s)(12.0MiB/648msec) clat (msec): min=117, max=272, avg=196.79, stdev=77.68 lat (msec): min=117, max=273, avg=197.08, stdev=77.67 clat percentiles (msec): | 1.00th=[ 118], 5.00th=[ 118], 10.00th=[ 118], 20.00th=[ 118], | 30.00th=[ 118], 40.00th=[ 201], 50.00th=[ 201], 60.00th=[ 201], | 70.00th=[ 275], 80.00th=[ 275], 90.00th=[ 275], 95.00th=[ 275], | 99.00th=[ 275], 99.50th=[ 275], 99.90th=[ 275], 99.95th=[ 275], | 99.99th=[ 275] bw ( KiB/s): min= 8192, max= 8192, per=16.75%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.00%, sys=35.86%, ctx=136, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9693: Thu Apr 17 07:59:19 2025 write: IOPS=6, BW=24.9MiB/s (26.2MB/s)(12.0MiB/481msec) clat (msec): min=102, max=259, avg=156.82, stdev=88.98 lat (msec): min=102, max=259, avg=157.05, stdev=89.02 clat percentiles (msec): | 1.00th=[ 103], 5.00th=[ 103], 10.00th=[ 103], 20.00th=[ 103], | 30.00th=[ 103], 40.00th=[ 109], 50.00th=[ 109], 60.00th=[ 109], | 70.00th=[ 259], 80.00th=[ 259], 90.00th=[ 259], 95.00th=[ 259], | 99.00th=[ 259], 99.50th=[ 259], 99.90th=[ 259], 99.95th=[ 259], | 99.99th=[ 259] lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.00%, sys=48.54%, ctx=99, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=23.9MiB/s (25.0MB/s), 6113KiB/s-31.5MiB/s (6260kB/s-33.0MB/s), io=16.0MiB (16.8MB), run=381-670msec WRITE: bw=47.8MiB/s (50.1MB/s), 11.9MiB/s-24.9MiB/s (12.5MB/s-26.2MB/s), io=32.0MiB (33.6MB), run=481-670msec rand-rw: (groupid=0, jobs=1): err= 0: pid=9688: Thu Apr 17 07:59:19 2025 read: IOPS=3, BW=15.9MiB/s (16.6MB/s)(12.0MiB/757msec) clat (msec): min=41, max=393, avg=236.75, stdev=179.09 lat (msec): min=41, max=393, avg=236.76, stdev=179.09 clat percentiles (msec): | 1.00th=[ 42], 5.00th=[ 42], 10.00th=[ 42], 20.00th=[ 42], | 30.00th=[ 42], 40.00th=[ 275], 50.00th=[ 275], 60.00th=[ 275], | 70.00th=[ 393], 80.00th=[ 393], 90.00th=[ 393], 95.00th=[ 393], | 99.00th=[ 393], 99.50th=[ 393], 99.90th=[ 393], 99.95th=[ 393], | 99.99th=[ 393] bw ( KiB/s): min= 8192, max= 8192, per=38.80%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 50=33.33%, 500=66.67% cpu : usr=0.00%, sys=8.33%, ctx=47, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9690: Thu Apr 17 07:59:19 2025 read: IOPS=1, BW=5278KiB/s (5405kB/s)(4096KiB/776msec) clat (nsec): min=269389k, max=269389k, avg=269388802.00, stdev= 0.00 lat (nsec): min=269390k, max=269390k, avg=269389800.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 271], 5.00th=[ 271], 10.00th=[ 271], 20.00th=[ 271], | 30.00th=[ 271], 40.00th=[ 271], 50.00th=[ 271], 60.00th=[ 271], | 70.00th=[ 271], 80.00th=[ 271], 90.00th=[ 271], 95.00th=[ 271], | 99.00th=[ 271], 99.50th=[ 271], 99.90th=[ 271], 99.95th=[ 271], | 99.99th=[ 271] bw ( KiB/s): min= 8192, max= 8192, per=38.80%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=2, BW=10.3MiB/s (10.8MB/s)(8192KiB/776msec) clat (msec): min=125, max=282, avg=204.00, stdev=111.31 lat (msec): min=125, max=283, avg=204.26, stdev=111.41 clat percentiles (msec): | 1.00th=[ 126], 5.00th=[ 126], 10.00th=[ 126], 20.00th=[ 126], | 30.00th=[ 126], 40.00th=[ 126], 50.00th=[ 126], 60.00th=[ 284], | 70.00th=[ 284], 80.00th=[ 284], 90.00th=[ 284], 95.00th=[ 284], | 99.00th=[ 284], 99.50th=[ 284], 99.90th=[ 284], 99.95th=[ 284], | 99.99th=[ 284] bw ( KiB/s): min= 8192, max= 8192, per=19.40%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=66.67% cpu : usr=0.00%, sys=11.35%, ctx=88, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9692: Thu Apr 17 07:59:19 2025 write: IOPS=4, BW=17.0MiB/s (17.9MB/s)(12.0MiB/704msec) clat (msec): min=148, max=286, avg=231.62, stdev=72.80 lat (msec): min=149, max=286, avg=231.90, stdev=72.90 clat percentiles (msec): | 1.00th=[ 150], 5.00th=[ 150], 10.00th=[ 150], 20.00th=[ 150], | 30.00th=[ 150], 40.00th=[ 259], 50.00th=[ 259], 60.00th=[ 259], | 70.00th=[ 288], 80.00th=[ 288], 90.00th=[ 288], 95.00th=[ 288], | 99.00th=[ 288], 99.50th=[ 288], 99.90th=[ 288], 99.95th=[ 288], | 99.99th=[ 288] bw ( KiB/s): min= 8192, max= 8192, per=19.40%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=66.67% cpu : usr=0.28%, sys=15.79%, ctx=116, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=9694: Thu Apr 17 07:59:19 2025 write: IOPS=5, BW=21.5MiB/s (22.5MB/s)(12.0MiB/559msec) clat (msec): min=108, max=205, avg=155.95, stdev=48.76 lat (msec): min=108, max=205, avg=156.19, stdev=48.83 clat percentiles (msec): | 1.00th=[ 109], 5.00th=[ 109], 10.00th=[ 109], 20.00th=[ 109], | 30.00th=[ 109], 40.00th=[ 155], 50.00th=[ 155], 60.00th=[ 155], | 70.00th=[ 207], 80.00th=[ 207], 90.00th=[ 207], 95.00th=[ 207], | 99.00th=[ 207], 99.50th=[ 207], 99.90th=[ 207], 99.95th=[ 207], | 99.99th=[ 207] bw ( KiB/s): min=16384, max=16384, per=38.80%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 250=100.00% cpu : usr=0.00%, sys=19.89%, ctx=165, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=20.6MiB/s (21.6MB/s), 5278KiB/s-15.9MiB/s (5405kB/s-16.6MB/s), io=16.0MiB (16.8MB), run=757-776msec WRITE: bw=41.2MiB/s (43.2MB/s), 10.3MiB/s-21.5MiB/s (10.8MB/s-22.5MB/s), io=32.0MiB (33.6MB), run=559-776msec