== sanity test 398b: DIO and buffer IO race ============== 02:39:34 (1735371574) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.08517 s, 46.4 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=5626: Sat Dec 28 02:39:48 2024 read: IOPS=126, BW=507KiB/s (519kB/s)(5992KiB/11816msec) clat (usec): min=47, max=165261, avg=3793.61, stdev=5272.96 lat (usec): min=47, max=165262, avg=3794.38, stdev=5273.07 clat percentiles (usec): | 1.00th=[ 51], 5.00th=[ 59], 10.00th=[ 65], 20.00th=[ 225], | 30.00th=[ 2245], 40.00th=[ 2606], 50.00th=[ 3032], 60.00th=[ 3752], | 70.00th=[ 4686], 80.00th=[ 5866], 90.00th=[ 7308], 95.00th=[ 8586], | 99.00th=[ 15270], 99.50th=[ 18744], 99.90th=[ 36439], 99.95th=[164627], | 99.99th=[164627] bw ( KiB/s): min= 280, max= 728, per=24.60%, avg=499.96, stdev=99.67, samples=23 iops : min= 70, max= 182, avg=124.91, stdev=24.93, samples=23 write: IOPS=133, BW=533KiB/s (546kB/s)(6296KiB/11816msec) clat (usec): min=394, max=42394, avg=3820.18, stdev=3628.52 lat (usec): min=394, max=42394, avg=3821.60, stdev=3628.07 clat percentiles (usec): | 1.00th=[ 478], 5.00th=[ 510], 10.00th=[ 537], 20.00th=[ 611], | 30.00th=[ 848], 40.00th=[ 1975], 50.00th=[ 3032], 60.00th=[ 3851], | 70.00th=[ 5080], 80.00th=[ 6456], 90.00th=[ 8455], 95.00th=[11076], | 99.00th=[14877], 99.50th=[16712], 99.90th=[24249], 99.95th=[42206], | 99.99th=[42206] bw ( KiB/s): min= 360, max= 750, per=24.60%, avg=523.22, stdev=103.58, samples=23 iops : min= 90, max= 187, avg=130.74, stdev=25.84, samples=23 lat (usec) : 50=0.29%, 100=8.63%, 250=0.85%, 500=1.60%, 750=12.30% lat (usec) : 1000=3.09% lat (msec) : 2=5.53%, 4=29.88%, 10=33.01%, 20=4.49%, 50=0.29% lat (msec) : 250=0.03% cpu : usr=0.14%, sys=24.49%, ctx=6287, majf=0, minf=36 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5628: Sat Dec 28 02:39:48 2024 read: IOPS=125, BW=503KiB/s (515kB/s)(5940KiB/11815msec) clat (usec): min=47, max=31991, avg=3618.69, stdev=3267.04 lat (usec): min=47, max=31991, avg=3619.71, stdev=3267.27 clat percentiles (usec): | 1.00th=[ 50], 5.00th=[ 59], 10.00th=[ 62], 20.00th=[ 120], | 30.00th=[ 2212], 40.00th=[ 2606], 50.00th=[ 3032], 60.00th=[ 3556], | 70.00th=[ 4359], 80.00th=[ 5866], 90.00th=[ 7439], 95.00th=[ 8848], | 99.00th=[15139], 99.50th=[20841], 99.90th=[31065], 99.95th=[32113], | 99.99th=[32113] bw ( KiB/s): min= 304, max= 696, per=24.49%, avg=497.57, stdev=111.18, samples=23 iops : min= 76, max= 174, avg=124.30, stdev=27.82, samples=23 write: IOPS=134, BW=537KiB/s (550kB/s)(6348KiB/11815msec) clat (usec): min=424, max=164269, avg=3984.98, stdev=5590.94 lat (usec): min=425, max=164270, avg=3986.08, stdev=5591.06 clat percentiles (usec): | 1.00th=[ 486], 5.00th=[ 510], 10.00th=[ 529], 20.00th=[ 586], | 30.00th=[ 783], 40.00th=[ 1680], 50.00th=[ 3064], 60.00th=[ 3916], | 70.00th=[ 4948], 80.00th=[ 6456], 90.00th=[ 8848], 95.00th=[ 11731], | 99.00th=[ 16581], 99.50th=[ 20579], 99.90th=[ 42206], 99.95th=[164627], | 99.99th=[164627] bw ( KiB/s): min= 352, max= 832, per=24.65%, avg=524.26, stdev=93.12, samples=23 iops : min= 88, max= 208, avg=131.00, stdev=23.26, samples=23 lat (usec) : 50=0.59%, 100=8.82%, 250=0.75%, 500=1.86%, 750=13.67% lat (usec) : 1000=2.05% lat (msec) : 2=6.12%, 4=29.82%, 10=31.22%, 20=4.56%, 50=0.52% lat (msec) : 250=0.03% cpu : usr=0.15%, sys=24.01%, ctx=6286, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5630: Sat Dec 28 02:39:48 2024 read: IOPS=128, BW=512KiB/s (524kB/s)(6028KiB/11771msec) clat (usec): min=48, max=32140, avg=3592.22, stdev=2966.03 lat (usec): min=49, max=32141, avg=3592.88, stdev=2966.08 clat percentiles (usec): | 1.00th=[ 50], 5.00th=[ 59], 10.00th=[ 69], 20.00th=[ 865], | 30.00th=[ 2278], 40.00th=[ 2671], 50.00th=[ 3064], 60.00th=[ 3621], | 70.00th=[ 4490], 80.00th=[ 5669], 90.00th=[ 7111], 95.00th=[ 8586], | 99.00th=[11469], 99.50th=[16581], 99.90th=[25822], 99.95th=[32113], | 99.99th=[32113] bw ( KiB/s): min= 320, max= 752, per=25.12%, avg=510.35, stdev=111.48, samples=23 iops : min= 80, max= 188, avg=127.52, stdev=27.84, samples=23 write: IOPS=132, BW=532KiB/s (545kB/s)(6260KiB/11771msec) clat (usec): min=414, max=164407, avg=3985.97, stdev=5511.35 lat (usec): min=415, max=164408, avg=3986.95, stdev=5511.34 clat percentiles (usec): | 1.00th=[ 490], 5.00th=[ 519], 10.00th=[ 537], 20.00th=[ 644], | 30.00th=[ 848], 40.00th=[ 1713], 50.00th=[ 3097], 60.00th=[ 4080], | 70.00th=[ 5014], 80.00th=[ 6456], 90.00th=[ 8848], 95.00th=[ 10945], | 99.00th=[ 16319], 99.50th=[ 19792], 99.90th=[ 37487], 99.95th=[164627], | 99.99th=[164627] bw ( KiB/s): min= 336, max= 702, per=24.86%, avg=528.78, stdev=94.02, samples=23 iops : min= 84, max= 175, avg=132.13, stdev=23.47, samples=23 lat (usec) : 50=0.52%, 100=8.27%, 250=0.72%, 500=1.14%, 750=12.24% lat (usec) : 1000=3.87% lat (msec) : 2=6.18%, 4=28.68%, 10=33.63%, 20=4.39%, 50=0.33% lat (msec) : 250=0.03% cpu : usr=0.17%, sys=24.50%, ctx=6325, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5632: Sat Dec 28 02:39:48 2024 read: IOPS=129, BW=519KiB/s (532kB/s)(6052KiB/11650msec) clat (usec): min=47, max=163351, avg=3565.17, stdev=5079.41 lat (usec): min=48, max=163352, avg=3565.81, stdev=5079.43 clat percentiles (usec): | 1.00th=[ 54], 5.00th=[ 60], 10.00th=[ 69], 20.00th=[ 1663], | 30.00th=[ 2180], 40.00th=[ 2540], 50.00th=[ 2933], 60.00th=[ 3458], | 70.00th=[ 4228], 80.00th=[ 5276], 90.00th=[ 6980], 95.00th=[ 8455], | 99.00th=[ 11731], 99.50th=[ 14484], 99.90th=[ 40633], 99.95th=[162530], | 99.99th=[162530] bw ( KiB/s): min= 360, max= 792, per=25.45%, avg=517.17, stdev=89.66, samples=23 iops : min= 90, max= 198, avg=129.26, stdev=22.44, samples=23 write: IOPS=133, BW=535KiB/s (548kB/s)(6236KiB/11650msec) clat (usec): min=406, max=37287, avg=3935.65, stdev=4023.12 lat (usec): min=406, max=37288, avg=3936.86, stdev=4023.05 clat percentiles (usec): | 1.00th=[ 490], 5.00th=[ 515], 10.00th=[ 537], 20.00th=[ 594], | 30.00th=[ 775], 40.00th=[ 1500], 50.00th=[ 2999], 60.00th=[ 4015], | 70.00th=[ 5014], 80.00th=[ 6521], 90.00th=[ 9241], 95.00th=[10945], | 99.00th=[18482], 99.50th=[22152], 99.90th=[33817], 99.95th=[37487], | 99.99th=[37487] bw ( KiB/s): min= 376, max= 776, per=25.12%, avg=534.22, stdev=98.29, samples=23 iops : min= 94, max= 194, avg=133.52, stdev=24.60, samples=23 lat (usec) : 50=0.29%, 100=8.33%, 250=0.98%, 500=1.30%, 750=13.61% lat (usec) : 1000=2.67% lat (msec) : 2=6.38%, 4=29.72%, 10=32.06%, 20=4.07%, 50=0.55% lat (msec) : 250=0.03% cpu : usr=0.15%, sys=24.34%, ctx=6325, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2032KiB/s (2081kB/s), 503KiB/s-519KiB/s (515kB/s-532kB/s), io=23.4MiB (24.6MB), run=11650-11816msec WRITE: bw=2128KiB/s (2179kB/s), 532KiB/s-537KiB/s (545kB/s-550kB/s), io=24.6MiB (25.7MB), run=11650-11816msec rand-rw: (groupid=0, jobs=1): err= 0: pid=5625: Sat Dec 28 02:40:10 2024 read: IOPS=46, BW=184KiB/s (189kB/s)(5992KiB/32497msec) clat (usec): min=1604, max=167468, avg=3313.71, stdev=5245.19 lat (usec): min=1604, max=167470, avg=3314.47, stdev=5245.26 clat percentiles (usec): | 1.00th=[ 1680], 5.00th=[ 1762], 10.00th=[ 1811], 20.00th=[ 1876], | 30.00th=[ 1942], 40.00th=[ 2008], 50.00th=[ 2089], 60.00th=[ 2212], | 70.00th=[ 2507], 80.00th=[ 3392], 90.00th=[ 6456], 95.00th=[ 9372], | 99.00th=[ 14877], 99.50th=[ 19530], 99.90th=[ 44303], 99.95th=[166724], | 99.99th=[166724] bw ( KiB/s): min= 48, max= 368, per=25.38%, avg=184.53, stdev=75.87, samples=64 iops : min= 12, max= 92, avg=46.06, stdev=18.99, samples=64 write: IOPS=48, BW=194KiB/s (198kB/s)(6296KiB/32497msec) clat (usec): min=8717, max=52983, avg=17417.30, stdev=6484.74 lat (usec): min=8718, max=52984, avg=17418.06, stdev=6484.76 clat percentiles (usec): | 1.00th=[10028], 5.00th=[10552], 10.00th=[11076], 20.00th=[11863], | 30.00th=[13173], 40.00th=[15139], 50.00th=[16319], 60.00th=[17433], | 70.00th=[18482], 80.00th=[20317], 90.00th=[26608], 95.00th=[31851], | 99.00th=[39060], 99.50th=[42730], 99.90th=[50594], 99.95th=[53216], | 99.99th=[53216] bw ( KiB/s): min= 48, max= 264, per=25.37%, avg=193.03, stdev=58.24, samples=64 iops : min= 12, max= 66, avg=48.19, stdev=14.57, samples=64 lat (msec) : 2=19.04%, 4=21.42%, 10=6.77%, 20=41.73%, 50=10.94% lat (msec) : 100=0.07%, 250=0.03% cpu : usr=0.09%, sys=5.25%, ctx=3411, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5627: Sat Dec 28 02:40:10 2024 read: IOPS=44, BW=180KiB/s (184kB/s)(5940KiB/33024msec) clat (usec): min=1592, max=167109, avg=3230.78, stdev=5052.59 lat (usec): min=1592, max=167110, avg=3231.41, stdev=5052.61 clat percentiles (usec): | 1.00th=[ 1663], 5.00th=[ 1762], 10.00th=[ 1827], 20.00th=[ 1876], | 30.00th=[ 1942], 40.00th=[ 2008], 50.00th=[ 2089], 60.00th=[ 2212], | 70.00th=[ 2507], 80.00th=[ 3228], 90.00th=[ 6259], 95.00th=[ 8979], | 99.00th=[ 14091], 99.50th=[ 19006], 99.90th=[ 30016], 99.95th=[166724], | 99.99th=[166724] bw ( KiB/s): min= 16, max= 344, per=24.87%, avg=180.78, stdev=77.81, samples=65 iops : min= 4, max= 86, avg=45.14, stdev=19.49, samples=65 write: IOPS=48, BW=192KiB/s (197kB/s)(6348KiB/33024msec) clat (usec): min=9140, max=58081, avg=17710.79, stdev=6733.77 lat (usec): min=9141, max=58082, avg=17711.55, stdev=6733.82 clat percentiles (usec): | 1.00th=[ 9896], 5.00th=[10552], 10.00th=[11076], 20.00th=[11994], | 30.00th=[13698], 40.00th=[15270], 50.00th=[16450], 60.00th=[17695], | 70.00th=[18482], 80.00th=[20841], 90.00th=[27657], 95.00th=[32113], | 99.00th=[40633], 99.50th=[45876], 99.90th=[50594], 99.95th=[57934], | 99.99th=[57934] bw ( KiB/s): min= 72, max= 254, per=25.27%, avg=192.28, stdev=57.10, samples=65 iops : min= 18, max= 63, avg=48.02, stdev=14.27, samples=65 lat (msec) : 2=18.78%, 4=21.45%, 10=7.03%, 20=40.95%, 50=11.69% lat (msec) : 100=0.07%, 250=0.03% cpu : usr=0.13%, sys=5.03%, ctx=3410, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5629: Sat Dec 28 02:40:10 2024 read: IOPS=46, BW=185KiB/s (190kB/s)(6028KiB/32505msec) clat (usec): min=1539, max=37101, avg=3251.75, stdev=2905.06 lat (usec): min=1539, max=37102, avg=3252.45, stdev=2905.14 clat percentiles (usec): | 1.00th=[ 1663], 5.00th=[ 1745], 10.00th=[ 1811], 20.00th=[ 1893], | 30.00th=[ 1942], 40.00th=[ 2008], 50.00th=[ 2089], 60.00th=[ 2245], | 70.00th=[ 2540], 80.00th=[ 3589], 90.00th=[ 6980], 95.00th=[ 9110], | 99.00th=[14746], 99.50th=[18482], 99.90th=[32375], 99.95th=[36963], | 99.99th=[36963] bw ( KiB/s): min= 48, max= 320, per=25.50%, avg=185.41, stdev=67.07, samples=64 iops : min= 12, max= 80, avg=46.28, stdev=16.78, samples=64 write: IOPS=48, BW=193KiB/s (197kB/s)(6260KiB/32505msec) clat (msec): min=8, max=197, avg=17.56, stdev= 7.96 lat (msec): min=8, max=197, avg=17.56, stdev= 7.96 clat percentiles (msec): | 1.00th=[ 11], 5.00th=[ 11], 10.00th=[ 12], 20.00th=[ 12], | 30.00th=[ 14], 40.00th=[ 16], 50.00th=[ 17], 60.00th=[ 18], | 70.00th=[ 19], 80.00th=[ 21], 90.00th=[ 27], 95.00th=[ 31], | 99.00th=[ 41], 99.50th=[ 46], 99.90th=[ 61], 99.95th=[ 199], | 99.99th=[ 199] bw ( KiB/s): min= 56, max= 272, per=25.23%, avg=192.03, stdev=59.70, samples=64 iops : min= 14, max= 68, avg=47.94, stdev=14.94, samples=64 lat (msec) : 2=18.55%, 4=21.52%, 10=7.68%, 20=41.05%, 50=11.07% lat (msec) : 100=0.10%, 250=0.03% cpu : usr=0.09%, sys=5.14%, ctx=3425, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5631: Sat Dec 28 02:40:10 2024 read: IOPS=46, BW=188KiB/s (192kB/s)(6052KiB/32238msec) clat (usec): min=1588, max=26576, avg=3147.29, stdev=2702.65 lat (usec): min=1589, max=26576, avg=3147.92, stdev=2702.66 clat percentiles (usec): | 1.00th=[ 1696], 5.00th=[ 1778], 10.00th=[ 1827], 20.00th=[ 1893], | 30.00th=[ 1942], 40.00th=[ 1991], 50.00th=[ 2073], 60.00th=[ 2212], | 70.00th=[ 2474], 80.00th=[ 3261], 90.00th=[ 6259], 95.00th=[ 9110], | 99.00th=[14484], 99.50th=[16909], 99.90th=[25035], 99.95th=[26608], | 99.99th=[26608] bw ( KiB/s): min= 56, max= 328, per=25.72%, avg=187.02, stdev=69.80, samples=64 iops : min= 14, max= 82, avg=46.67, stdev=17.43, samples=64 write: IOPS=48, BW=193KiB/s (198kB/s)(6236KiB/32238msec) clat (msec): min=9, max=181, avg=17.55, stdev= 7.67 lat (msec): min=9, max=181, avg=17.55, stdev= 7.67 clat percentiles (msec): | 1.00th=[ 11], 5.00th=[ 11], 10.00th=[ 12], 20.00th=[ 12], | 30.00th=[ 14], 40.00th=[ 16], 50.00th=[ 17], 60.00th=[ 18], | 70.00th=[ 19], 80.00th=[ 21], 90.00th=[ 27], 95.00th=[ 32], | 99.00th=[ 40], 99.50th=[ 42], 99.90th=[ 60], 99.95th=[ 182], | 99.99th=[ 182] bw ( KiB/s): min= 80, max= 264, per=25.38%, avg=193.14, stdev=57.26, samples=64 iops : min= 20, max= 66, avg=48.20, stdev=14.33, samples=64 lat (msec) : 2=19.95%, 4=20.67%, 10=7.45%, 20=40.79%, 50=11.04% lat (msec) : 100=0.07%, 250=0.03% cpu : usr=0.09%, sys=5.19%, ctx=3393, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=727KiB/s (745kB/s), 180KiB/s-188KiB/s (184kB/s-192kB/s), io=23.4MiB (24.6MB), run=32238-33024msec WRITE: bw=761KiB/s (780kB/s), 192KiB/s-194KiB/s (197kB/s-198kB/s), io=24.6MiB (25.7MB), run=32238-33024msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=5676: Sat Dec 28 02:40:15 2024 read: IOPS=82, BW=1325KiB/s (1357kB/s)(5920KiB/4467msec) clat (usec): min=52, max=166841, avg=8652.82, stdev=9599.87 lat (usec): min=53, max=166842, avg=8653.38, stdev=9599.90 clat percentiles (usec): | 1.00th=[ 55], 5.00th=[ 63], 10.00th=[ 73], 20.00th=[ 147], | 30.00th=[ 7963], 40.00th=[ 8717], 50.00th=[ 9372], 60.00th=[ 10028], | 70.00th=[ 10683], 80.00th=[ 11600], 90.00th=[ 12911], 95.00th=[ 14222], | 99.00th=[ 21890], 99.50th=[ 30278], 99.90th=[166724], 99.95th=[166724], | 99.99th=[166724] bw ( KiB/s): min= 704, max= 1632, per=24.15%, avg=1272.00, stdev=275.67, samples=8 iops : min= 44, max= 102, avg=79.50, stdev=17.23, samples=8 write: IOPS=89, BW=1426KiB/s (1460kB/s)(6368KiB/4467msec) clat (usec): min=629, max=30957, avg=3151.60, stdev=3850.70 lat (usec): min=630, max=30958, avg=3152.49, stdev=3850.69 clat percentiles (usec): | 1.00th=[ 660], 5.00th=[ 701], 10.00th=[ 750], 20.00th=[ 791], | 30.00th=[ 832], 40.00th=[ 922], 50.00th=[ 1516], 60.00th=[ 2540], | 70.00th=[ 3294], 80.00th=[ 5080], 90.00th=[ 6849], 95.00th=[10683], | 99.00th=[21365], 99.50th=[24511], 99.90th=[31065], 99.95th=[31065], | 99.99th=[31065] bw ( KiB/s): min= 800, max= 1920, per=25.05%, avg=1388.00, stdev=355.49, samples=8 iops : min= 50, max= 120, avg=86.75, stdev=22.22, samples=8 lat (usec) : 100=8.98%, 250=0.78%, 500=0.13%, 750=5.86%, 1000=16.54% lat (msec) : 2=5.34%, 4=10.94%, 10=29.43%, 20=20.83%, 50=1.04% lat (msec) : 250=0.13% cpu : usr=0.25%, sys=23.82%, ctx=2028, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5678: Sat Dec 28 02:40:15 2024 read: IOPS=82, BW=1318KiB/s (1350kB/s)(5744KiB/4357msec) clat (usec): min=52, max=26739, avg=8056.46, stdev=4975.22 lat (usec): min=52, max=26739, avg=8059.03, stdev=4972.26 clat percentiles (usec): | 1.00th=[ 55], 5.00th=[ 61], 10.00th=[ 63], 20.00th=[ 91], | 30.00th=[ 7767], 40.00th=[ 8586], 50.00th=[ 9110], 60.00th=[ 9765], | 70.00th=[10683], 80.00th=[11469], 90.00th=[12780], 95.00th=[14484], | 99.00th=[21365], 99.50th=[22676], 99.90th=[26870], 99.95th=[26870], | 99.99th=[26870] bw ( KiB/s): min= 320, max= 1600, per=24.38%, avg=1284.00, stdev=415.27, samples=8 iops : min= 20, max= 100, avg=80.25, stdev=25.95, samples=8 write: IOPS=93, BW=1502KiB/s (1538kB/s)(6544KiB/4357msec) clat (usec): min=626, max=233616, avg=3465.41, stdev=11967.48 lat (usec): min=627, max=233618, avg=3468.36, stdev=11967.48 clat percentiles (usec): | 1.00th=[ 668], 5.00th=[ 717], 10.00th=[ 750], 20.00th=[ 791], | 30.00th=[ 832], 40.00th=[ 898], 50.00th=[ 1401], 60.00th=[ 2507], | 70.00th=[ 3097], 80.00th=[ 4686], 90.00th=[ 6063], 95.00th=[ 8455], | 99.00th=[ 21627], 99.50th=[ 29230], 99.90th=[233833], 99.95th=[233833], | 99.99th=[233833] bw ( KiB/s): min= 608, max= 2016, per=26.20%, avg=1452.00, stdev=417.91, samples=8 iops : min= 38, max= 126, avg=90.75, stdev=26.12, samples=8 lat (usec) : 100=10.03%, 250=0.52%, 750=5.60%, 1000=18.23% lat (msec) : 2=3.78%, 4=13.93%, 10=29.17%, 20=17.32%, 50=1.30% lat (msec) : 250=0.13% cpu : usr=0.11%, sys=20.80%, ctx=1964, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5680: Sat Dec 28 02:40:15 2024 read: IOPS=85, BW=1376KiB/s (1409kB/s)(6256KiB/4547msec) clat (usec): min=49, max=28365, avg=8181.13, stdev=4795.25 lat (usec): min=50, max=28366, avg=8181.88, stdev=4794.94 clat percentiles (usec): | 1.00th=[ 53], 5.00th=[ 63], 10.00th=[ 73], 20.00th=[ 167], | 30.00th=[ 7570], 40.00th=[ 8455], 50.00th=[ 9503], 60.00th=[10028], | 70.00th=[10683], 80.00th=[11600], 90.00th=[13173], 95.00th=[14353], | 99.00th=[17695], 99.50th=[18482], 99.90th=[28443], 99.95th=[28443], | 99.99th=[28443] bw ( KiB/s): min= 544, max= 1600, per=24.68%, avg=1300.00, stdev=339.92, samples=8 iops : min= 34, max= 100, avg=81.25, stdev=21.25, samples=8 write: IOPS=82, BW=1327KiB/s (1358kB/s)(6032KiB/4547msec) clat (usec): min=614, max=171008, avg=3442.28, stdev=9471.50 lat (usec): min=615, max=171010, avg=3443.23, stdev=9471.56 clat percentiles (usec): | 1.00th=[ 660], 5.00th=[ 701], 10.00th=[ 750], 20.00th=[ 799], | 30.00th=[ 832], 40.00th=[ 922], 50.00th=[ 1434], 60.00th=[ 2573], | 70.00th=[ 2933], 80.00th=[ 4490], 90.00th=[ 6259], 95.00th=[ 11731], | 99.00th=[ 19792], 99.50th=[ 33424], 99.90th=[170918], 99.95th=[170918], | 99.99th=[170918] bw ( KiB/s): min= 704, max= 1760, per=23.24%, avg=1288.00, stdev=343.27, samples=8 iops : min= 44, max= 110, avg=80.50, stdev=21.45, samples=8 lat (usec) : 50=0.13%, 100=8.59%, 250=2.08%, 750=5.08%, 1000=16.67% lat (msec) : 2=3.91%, 4=12.37%, 10=27.86%, 20=22.79%, 50=0.39% lat (msec) : 250=0.13% cpu : usr=0.09%, sys=20.69%, ctx=2138, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5682: Sat Dec 28 02:40:15 2024 read: IOPS=87, BW=1398KiB/s (1432kB/s)(6032KiB/4314msec) clat (usec): min=48, max=24431, avg=7675.65, stdev=5076.78 lat (usec): min=48, max=24431, avg=7676.23, stdev=5076.73 clat percentiles (usec): | 1.00th=[ 52], 5.00th=[ 60], 10.00th=[ 62], 20.00th=[ 75], | 30.00th=[ 7111], 40.00th=[ 8225], 50.00th=[ 8979], 60.00th=[10028], | 70.00th=[10683], 80.00th=[11469], 90.00th=[12911], 95.00th=[14222], | 99.00th=[16581], 99.50th=[21890], 99.90th=[24511], 99.95th=[24511], | 99.99th=[24511] bw ( KiB/s): min= 224, max= 1760, per=25.58%, avg=1347.50, stdev=477.96, samples=8 iops : min= 14, max= 110, avg=84.12, stdev=29.84, samples=8 write: IOPS=90, BW=1450KiB/s (1485kB/s)(6256KiB/4314msec) clat (usec): min=649, max=166385, avg=3452.19, stdev=9406.58 lat (usec): min=650, max=166387, avg=3453.09, stdev=9406.67 clat percentiles (usec): | 1.00th=[ 676], 5.00th=[ 709], 10.00th=[ 758], 20.00th=[ 799], | 30.00th=[ 840], 40.00th=[ 914], 50.00th=[ 1123], 60.00th=[ 2376], | 70.00th=[ 2999], 80.00th=[ 4293], 90.00th=[ 6390], 95.00th=[ 9372], | 99.00th=[ 31327], 99.50th=[ 43779], 99.90th=[166724], 99.95th=[166724], | 99.99th=[166724] bw ( KiB/s): min= 320, max= 1920, per=25.62%, avg=1419.62, stdev=476.72, samples=8 iops : min= 20, max= 120, avg=88.62, stdev=29.78, samples=8 lat (usec) : 50=0.26%, 100=12.11%, 250=0.52%, 500=0.13%, 750=4.95% lat (usec) : 1000=18.88% lat (msec) : 2=4.82%, 4=10.94%, 10=25.65%, 20=20.70%, 50=0.91% lat (msec) : 250=0.13% cpu : usr=0.19%, sys=20.98%, ctx=1934, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=5268KiB/s (5394kB/s), 1318KiB/s-1398KiB/s (1350kB/s-1432kB/s), io=23.4MiB (24.5MB), run=4314-4547msec WRITE: bw=5542KiB/s (5675kB/s), 1327KiB/s-1502KiB/s (1358kB/s-1538kB/s), io=24.6MiB (25.8MB), run=4314-4547msec rand-rw: (groupid=0, jobs=1): err= 0: pid=5675: Sat Dec 28 02:40:19 2024 read: IOPS=42, BW=683KiB/s (700kB/s)(5920KiB/8665msec) clat (usec): min=1774, max=165152, avg=4100.11, stdev=8928.83 lat (usec): min=1774, max=165153, avg=4100.77, stdev=8928.85 clat percentiles (usec): | 1.00th=[ 1811], 5.00th=[ 1926], 10.00th=[ 1991], 20.00th=[ 2073], | 30.00th=[ 2147], 40.00th=[ 2245], 50.00th=[ 2442], 60.00th=[ 2638], | 70.00th=[ 3097], 80.00th=[ 4424], 90.00th=[ 6915], 95.00th=[ 10683], | 99.00th=[ 18482], 99.50th=[ 21890], 99.90th=[164627], 99.95th=[164627], | 99.99th=[164627] bw ( KiB/s): min= 384, max= 1216, per=24.90%, avg=673.88, stdev=222.13, samples=17 iops : min= 24, max= 76, avg=42.12, stdev=13.88, samples=17 write: IOPS=45, BW=735KiB/s (753kB/s)(6368KiB/8665msec) clat (usec): min=9706, max=44918, avg=17925.40, stdev=5686.79 lat (usec): min=9707, max=44919, avg=17926.53, stdev=5686.82 clat percentiles (usec): | 1.00th=[10028], 5.00th=[10945], 10.00th=[11338], 20.00th=[12649], | 30.00th=[14746], 40.00th=[15926], 50.00th=[17171], 60.00th=[18220], | 70.00th=[19268], 80.00th=[21890], 90.00th=[25822], 95.00th=[28967], | 99.00th=[36439], 99.50th=[44827], 99.90th=[44827], 99.95th=[44827], | 99.99th=[44827] bw ( KiB/s): min= 320, max= 992, per=25.65%, avg=730.35, stdev=204.97, samples=17 iops : min= 20, max= 62, avg=45.65, stdev=12.81, samples=17 lat (msec) : 2=5.60%, 4=31.64%, 10=8.20%, 20=40.23%, 50=14.19% lat (msec) : 250=0.13% cpu : usr=0.05%, sys=5.64%, ctx=868, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5677: Sat Dec 28 02:40:19 2024 read: IOPS=40, BW=649KiB/s (665kB/s)(5744KiB/8850msec) clat (usec): min=1715, max=33479, avg=3544.71, stdev=3228.68 lat (usec): min=1716, max=33480, avg=3545.31, stdev=3228.68 clat percentiles (usec): | 1.00th=[ 1778], 5.00th=[ 1909], 10.00th=[ 1958], 20.00th=[ 2057], | 30.00th=[ 2114], 40.00th=[ 2212], 50.00th=[ 2311], 60.00th=[ 2540], | 70.00th=[ 2900], 80.00th=[ 4015], 90.00th=[ 6980], 95.00th=[10552], | 99.00th=[17695], 99.50th=[20841], 99.90th=[33424], 99.95th=[33424], | 99.99th=[33424] bw ( KiB/s): min= 64, max= 1056, per=23.99%, avg=649.29, stdev=270.58, samples=17 iops : min= 4, max= 66, avg=40.53, stdev=16.93, samples=17 write: IOPS=46, BW=739KiB/s (757kB/s)(6544KiB/8850msec) clat (msec): min=9, max=235, avg=18.43, stdev=12.01 lat (msec): min=9, max=235, avg=18.43, stdev=12.01 clat percentiles (msec): | 1.00th=[ 11], 5.00th=[ 11], 10.00th=[ 12], 20.00th=[ 14], | 30.00th=[ 15], 40.00th=[ 17], 50.00th=[ 18], 60.00th=[ 19], | 70.00th=[ 20], 80.00th=[ 23], 90.00th=[ 26], 95.00th=[ 28], | 99.00th=[ 33], 99.50th=[ 38], 99.90th=[ 236], 99.95th=[ 236], | 99.99th=[ 236] bw ( KiB/s): min= 288, max= 1056, per=25.52%, avg=726.47, stdev=199.86, samples=17 iops : min= 18, max= 66, avg=45.35, stdev=12.53, samples=17 lat (msec) : 2=6.51%, 4=30.73%, 10=7.42%, 20=40.23%, 50=14.97% lat (msec) : 250=0.13% cpu : usr=0.10%, sys=5.40%, ctx=862, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5679: Sat Dec 28 02:40:19 2024 read: IOPS=46, BW=743KiB/s (761kB/s)(6256KiB/8419msec) clat (usec): min=1774, max=33720, avg=3629.69, stdev=3513.63 lat (usec): min=1775, max=33721, avg=3630.35, stdev=3513.62 clat percentiles (usec): | 1.00th=[ 1844], 5.00th=[ 1893], 10.00th=[ 1958], 20.00th=[ 2040], | 30.00th=[ 2147], 40.00th=[ 2278], 50.00th=[ 2409], 60.00th=[ 2606], | 70.00th=[ 2900], 80.00th=[ 4293], 90.00th=[ 6587], 95.00th=[ 9372], | 99.00th=[21890], 99.50th=[27657], 99.90th=[33817], 99.95th=[33817], | 99.99th=[33817] bw ( KiB/s): min= 288, max= 1536, per=27.71%, avg=749.81, stdev=330.36, samples=16 iops : min= 18, max= 96, avg=46.75, stdev=20.72, samples=16 write: IOPS=44, BW=716KiB/s (734kB/s)(6032KiB/8419msec) clat (msec): min=9, max=190, avg=18.42, stdev=10.52 lat (msec): min=9, max=190, avg=18.42, stdev=10.52 clat percentiles (msec): | 1.00th=[ 11], 5.00th=[ 11], 10.00th=[ 12], 20.00th=[ 13], | 30.00th=[ 15], 40.00th=[ 17], 50.00th=[ 18], 60.00th=[ 19], | 70.00th=[ 20], 80.00th=[ 22], 90.00th=[ 26], 95.00th=[ 30], | 99.00th=[ 38], 99.50th=[ 43], 99.90th=[ 190], 99.95th=[ 190], | 99.99th=[ 190] bw ( KiB/s): min= 320, max= 992, per=24.58%, avg=699.75, stdev=196.89, samples=16 iops : min= 20, max= 62, avg=43.62, stdev=12.32, samples=16 lat (msec) : 2=7.42%, 4=32.55%, 10=8.72%, 20=36.59%, 50=14.58% lat (msec) : 250=0.13% cpu : usr=0.14%, sys=5.76%, ctx=872, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5681: Sat Dec 28 02:40:19 2024 read: IOPS=43, BW=698KiB/s (714kB/s)(6032KiB/8647msec) clat (usec): min=1774, max=19094, avg=3612.53, stdev=2924.47 lat (usec): min=1775, max=19094, avg=3613.16, stdev=2924.45 clat percentiles (usec): | 1.00th=[ 1844], 5.00th=[ 1958], 10.00th=[ 1991], 20.00th=[ 2057], | 30.00th=[ 2147], 40.00th=[ 2212], 50.00th=[ 2376], 60.00th=[ 2638], | 70.00th=[ 3163], 80.00th=[ 4490], 90.00th=[ 6849], 95.00th=[11338], | 99.00th=[16909], 99.50th=[17433], 99.90th=[19006], 99.95th=[19006], | 99.99th=[19006] bw ( KiB/s): min= 224, max= 1088, per=25.67%, avg=694.59, stdev=238.60, samples=17 iops : min= 14, max= 68, avg=43.41, stdev=14.91, samples=17 write: IOPS=45, BW=723KiB/s (741kB/s)(6256KiB/8647msec) clat (msec): min=9, max=170, avg=18.47, stdev= 9.50 lat (msec): min=9, max=170, avg=18.47, stdev= 9.50 clat percentiles (msec): | 1.00th=[ 11], 5.00th=[ 11], 10.00th=[ 12], 20.00th=[ 14], | 30.00th=[ 15], 40.00th=[ 17], 50.00th=[ 18], 60.00th=[ 19], | 70.00th=[ 20], 80.00th=[ 23], 90.00th=[ 26], 95.00th=[ 29], | 99.00th=[ 41], 99.50th=[ 44], 99.90th=[ 171], 99.95th=[ 171], | 99.99th=[ 171] bw ( KiB/s): min= 320, max= 1024, per=25.26%, avg=719.06, stdev=193.37, samples=17 iops : min= 20, max= 64, avg=44.94, stdev=12.09, samples=17 lat (msec) : 2=5.47%, 4=32.94%, 10=7.94%, 20=38.54%, 50=14.97% lat (msec) : 250=0.13% cpu : usr=0.08%, sys=5.47%, ctx=873, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2706KiB/s (2771kB/s), 649KiB/s-743KiB/s (665kB/s-761kB/s), io=23.4MiB (24.5MB), run=8419-8850msec WRITE: bw=2847KiB/s (2916kB/s), 716KiB/s-739KiB/s (734kB/s-757kB/s), io=24.6MiB (25.8MB), run=8419-8850msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=5695: Sat Dec 28 02:40:24 2024 read: IOPS=2, BW=2400KiB/s (2457kB/s)(7168KiB/2987msec) clat (msec): min=15, max=951, avg=203.84, stdev=336.81 lat (msec): min=15, max=951, avg=203.84, stdev=336.81 clat percentiles (msec): | 1.00th=[ 16], 5.00th=[ 16], 10.00th=[ 16], 20.00th=[ 23], | 30.00th=[ 39], 40.00th=[ 39], 50.00th=[ 58], 60.00th=[ 138], | 70.00th=[ 138], 80.00th=[ 205], 90.00th=[ 953], 95.00th=[ 953], | 99.00th=[ 953], 99.50th=[ 953], 99.90th=[ 953], 99.95th=[ 953], | 99.99th=[ 953] bw ( KiB/s): min= 4096, max= 6144, per=96.11%, avg=5120.00, stdev=1448.15, samples=2 iops : min= 4, max= 6, avg= 5.00, stdev= 1.41, samples=2 write: IOPS=1, BW=1714KiB/s (1755kB/s)(5120KiB/2987msec) clat (msec): min=41, max=1000, avg=309.85, stdev=400.86 lat (msec): min=41, max=1000, avg=309.91, stdev=400.86 clat percentiles (msec): | 1.00th=[ 42], 5.00th=[ 42], 10.00th=[ 42], 20.00th=[ 42], | 30.00th=[ 60], 40.00th=[ 60], 50.00th=[ 136], 60.00th=[ 136], | 70.00th=[ 313], 80.00th=[ 313], 90.00th=[ 1003], 95.00th=[ 1003], | 99.00th=[ 1003], 99.50th=[ 1003], 99.90th=[ 1003], 99.95th=[ 1003], | 99.99th=[ 1003] bw ( KiB/s): min= 1809, max= 2048, per=25.86%, avg=1928.50, stdev=169.00, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 20=8.33%, 50=25.00%, 100=16.67%, 250=25.00%, 500=8.33% lat (msec) : 1000=8.33% cpu : usr=0.07%, sys=3.52%, ctx=44, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5697: Sat Dec 28 02:40:24 2024 read: IOPS=0, BW=533KiB/s (546kB/s)(2048KiB/3844msec) clat (usec): min=21852, max=71442, avg=46647.33, stdev=35065.48 lat (usec): min=21853, max=71443, avg=46648.47, stdev=35065.97 clat percentiles (usec): | 1.00th=[21890], 5.00th=[21890], 10.00th=[21890], 20.00th=[21890], | 30.00th=[21890], 40.00th=[21890], 50.00th=[21890], 60.00th=[71828], | 70.00th=[71828], 80.00th=[71828], 90.00th=[71828], 95.00th=[71828], | 99.00th=[71828], 99.50th=[71828], 99.90th=[71828], 99.95th=[71828], | 99.99th=[71828] bw ( KiB/s): min= 1796, max= 2048, per=36.08%, avg=1922.00, stdev=178.19, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 write: IOPS=2, BW=2664KiB/s (2728kB/s)(10.0MiB/3844msec) clat (msec): min=29, max=1246, avg=374.10, stdev=478.07 lat (msec): min=29, max=1246, avg=374.19, stdev=478.06 clat percentiles (msec): | 1.00th=[ 30], 5.00th=[ 30], 10.00th=[ 30], 20.00th=[ 30], | 30.00th=[ 31], 40.00th=[ 59], 50.00th=[ 66], 60.00th=[ 69], | 70.00th=[ 927], 80.00th=[ 927], 90.00th=[ 961], 95.00th=[ 1250], | 99.00th=[ 1250], 99.50th=[ 1250], 99.90th=[ 1250], 99.95th=[ 1250], | 99.99th=[ 1250] bw ( KiB/s): min= 1796, max= 8982, per=66.48%, avg=4958.00, stdev=3669.73, samples=3 iops : min= 1, max= 8, avg= 4.33, stdev= 3.51, samples=3 lat (msec) : 50=33.33%, 100=33.33%, 500=8.33%, 1000=16.67% cpu : usr=0.08%, sys=3.43%, ctx=63, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5699: Sat Dec 28 02:40:24 2024 read: IOPS=1, BW=1610KiB/s (1648kB/s)(6144KiB/3817msec) clat (msec): min=14, max=1303, avg=389.30, stdev=575.22 lat (msec): min=14, max=1303, avg=389.31, stdev=575.22 clat percentiles (msec): | 1.00th=[ 15], 5.00th=[ 15], 10.00th=[ 15], 20.00th=[ 17], | 30.00th=[ 17], 40.00th=[ 22], 50.00th=[ 22], 60.00th=[ 53], | 70.00th=[ 927], 80.00th=[ 927], 90.00th=[ 1301], 95.00th=[ 1301], | 99.00th=[ 1301], 99.50th=[ 1301], 99.90th=[ 1301], 99.95th=[ 1301], | 99.99th=[ 1301] bw ( KiB/s): min= 1753, max= 4096, per=49.41%, avg=2632.33, stdev=1276.13, samples=3 iops : min= 1, max= 4, avg= 2.33, stdev= 1.53, samples=3 write: IOPS=1, BW=1610KiB/s (1648kB/s)(6144KiB/3817msec) clat (msec): min=30, max=917, avg=239.11, stdev=337.70 lat (msec): min=30, max=917, avg=239.18, stdev=337.70 clat percentiles (msec): | 1.00th=[ 31], 5.00th=[ 31], 10.00th=[ 31], 20.00th=[ 66], | 30.00th=[ 66], 40.00th=[ 70], 50.00th=[ 70], 60.00th=[ 161], | 70.00th=[ 190], 80.00th=[ 190], 90.00th=[ 919], 95.00th=[ 919], | 99.00th=[ 919], 99.50th=[ 919], 99.90th=[ 919], 99.95th=[ 919], | 99.99th=[ 919] bw ( KiB/s): min= 2048, max= 6144, per=54.92%, avg=4096.00, stdev=2896.31, samples=2 iops : min= 2, max= 6, avg= 4.00, stdev= 2.83, samples=2 lat (msec) : 20=16.67%, 50=16.67%, 100=25.00%, 250=16.67%, 1000=16.67% cpu : usr=0.00%, sys=3.01%, ctx=51, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5701: Sat Dec 28 02:40:24 2024 read: IOPS=2, BW=2726KiB/s (2792kB/s)(5120KiB/1878msec) clat (usec): min=10755, max=56954, avg=20740.97, stdev=20282.91 lat (usec): min=10756, max=56955, avg=20741.67, stdev=20282.89 clat percentiles (usec): | 1.00th=[10814], 5.00th=[10814], 10.00th=[10814], 20.00th=[10814], | 30.00th=[10814], 40.00th=[10814], 50.00th=[11338], 60.00th=[11338], | 70.00th=[13829], 80.00th=[13829], 90.00th=[56886], 95.00th=[56886], | 99.00th=[56886], 99.50th=[56886], 99.90th=[56886], 99.95th=[56886], | 99.99th=[56886] bw ( KiB/s): min= 2043, max= 6144, per=76.84%, avg=4093.50, stdev=2899.84, samples=2 iops : min= 1, max= 6, avg= 3.50, stdev= 3.54, samples=2 write: IOPS=3, BW=3817KiB/s (3908kB/s)(7168KiB/1878msec) clat (msec): min=33, max=1113, avg=251.25, stdev=388.22 lat (msec): min=33, max=1113, avg=251.31, stdev=388.20 clat percentiles (msec): | 1.00th=[ 34], 5.00th=[ 34], 10.00th=[ 34], 20.00th=[ 40], | 30.00th=[ 71], 40.00th=[ 71], 50.00th=[ 92], 60.00th=[ 148], | 70.00th=[ 148], 80.00th=[ 262], 90.00th=[ 1116], 95.00th=[ 1116], | 99.00th=[ 1116], 99.50th=[ 1116], 99.90th=[ 1116], 99.95th=[ 1116], | 99.99th=[ 1116] bw ( KiB/s): min= 8175, max= 8175, per=100.00%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 7, max= 7, avg= 7.00, stdev= 0.00, samples=1 lat (msec) : 20=33.33%, 50=16.67%, 100=25.00%, 250=8.33%, 500=8.33% cpu : usr=0.05%, sys=4.85%, ctx=36, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=5328KiB/s (5456kB/s), 533KiB/s-2726KiB/s (546kB/s-2792kB/s), io=20.0MiB (20.0MB), run=1878-3844msec WRITE: bw=7459KiB/s (7638kB/s), 1610KiB/s-3817KiB/s (1648kB/s-3908kB/s), io=28.0MiB (29.4MB), run=1878-3844msec rand-rw: (groupid=0, jobs=1): err= 0: pid=5696: Sat Dec 28 02:40:25 2024 read: IOPS=2, BW=2482KiB/s (2542kB/s)(7168KiB/2888msec) clat (usec): min=448, max=964295, avg=326632.13, stdev=425580.80 lat (usec): min=449, max=964296, avg=326633.26, stdev=425580.49 clat percentiles (usec): | 1.00th=[ 449], 5.00th=[ 449], 10.00th=[ 449], 20.00th=[ 5145], | 30.00th=[ 43779], 40.00th=[ 43779], 50.00th=[152044], 60.00th=[206570], | 70.00th=[206570], 80.00th=[918553], 90.00th=[960496], 95.00th=[960496], | 99.00th=[960496], 99.50th=[960496], 99.90th=[960496], 99.95th=[960496], | 99.99th=[960496] bw ( KiB/s): min= 4096, max= 6144, per=100.00%, avg=5120.00, stdev=1448.15, samples=2 iops : min= 4, max= 6, avg= 5.00, stdev= 1.41, samples=2 write: IOPS=1, BW=1773KiB/s (1815kB/s)(5120KiB/2888msec) clat (msec): min=21, max=269, avg=115.56, stdev=115.48 lat (msec): min=21, max=269, avg=115.61, stdev=115.48 clat percentiles (msec): | 1.00th=[ 22], 5.00th=[ 22], 10.00th=[ 22], 20.00th=[ 22], | 30.00th=[ 26], 40.00th=[ 26], 50.00th=[ 53], 60.00th=[ 53], | 70.00th=[ 209], 80.00th=[ 209], 90.00th=[ 271], 95.00th=[ 271], | 99.00th=[ 271], 99.50th=[ 271], 99.90th=[ 271], 99.95th=[ 271], | 99.99th=[ 271] bw ( KiB/s): min= 2048, max= 6144, per=65.31%, avg=4096.00, stdev=2896.31, samples=2 iops : min= 2, max= 6, avg= 4.00, stdev= 2.83, samples=2 lat (usec) : 500=8.33% lat (msec) : 10=8.33%, 50=25.00%, 100=8.33%, 250=25.00%, 500=8.33% lat (msec) : 1000=16.67% cpu : usr=0.07%, sys=22.45%, ctx=842, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5698: Sat Dec 28 02:40:25 2024 read: IOPS=0, BW=541KiB/s (554kB/s)(2048KiB/3785msec) clat (msec): min=65, max=1080, avg=572.90, stdev=718.23 lat (msec): min=65, max=1080, avg=572.91, stdev=718.23 clat percentiles (msec): | 1.00th=[ 66], 5.00th=[ 66], 10.00th=[ 66], 20.00th=[ 66], | 30.00th=[ 66], 40.00th=[ 66], 50.00th=[ 66], 60.00th=[ 1083], | 70.00th=[ 1083], 80.00th=[ 1083], 90.00th=[ 1083], 95.00th=[ 1083], | 99.00th=[ 1083], 99.50th=[ 1083], 99.90th=[ 1083], 99.95th=[ 1083], | 99.99th=[ 1083] bw ( KiB/s): min= 2048, max= 2048, per=45.71%, avg=2048.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 write: IOPS=2, BW=2705KiB/s (2770kB/s)(10.0MiB/3785msec) clat (msec): min=21, max=1032, avg=262.93, stdev=391.03 lat (msec): min=21, max=1032, avg=262.99, stdev=391.05 clat percentiles (msec): | 1.00th=[ 22], 5.00th=[ 22], 10.00th=[ 22], 20.00th=[ 23], | 30.00th=[ 26], 40.00th=[ 27], 50.00th=[ 66], 60.00th=[ 69], | 70.00th=[ 296], 80.00th=[ 296], 90.00th=[ 944], 95.00th=[ 1036], | 99.00th=[ 1036], 99.50th=[ 1036], 99.90th=[ 1036], 99.95th=[ 1036], | 99.99th=[ 1036] bw ( KiB/s): min= 2048, max= 8192, per=65.31%, avg=4096.00, stdev=2896.31, samples=4 iops : min= 2, max= 8, avg= 4.00, stdev= 2.83, samples=4 lat (msec) : 50=33.33%, 100=25.00%, 250=8.33%, 500=8.33%, 1000=8.33% cpu : usr=0.00%, sys=12.13%, ctx=390, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5700: Sat Dec 28 02:40:25 2024 read: IOPS=1, BW=1685KiB/s (1726kB/s)(6144KiB/3646msec) clat (usec): min=341, max=887595, avg=172939.59, stdev=351816.91 lat (usec): min=341, max=887596, avg=172940.45, stdev=351817.01 clat percentiles (usec): | 1.00th=[ 343], 5.00th=[ 343], 10.00th=[ 343], 20.00th=[ 545], | 30.00th=[ 545], 40.00th=[ 5342], 50.00th=[ 5342], 60.00th=[ 63701], | 70.00th=[ 80217], 80.00th=[ 80217], 90.00th=[884999], 95.00th=[884999], | 99.00th=[884999], 99.50th=[884999], 99.90th=[884999], 99.95th=[884999], | 99.99th=[884999] bw ( KiB/s): min= 2043, max= 6144, per=76.15%, avg=3411.67, stdev=2366.27, samples=3 iops : min= 1, max= 6, avg= 3.00, stdev= 2.65, samples=3 write: IOPS=1, BW=1685KiB/s (1726kB/s)(6144KiB/3646msec) clat (msec): min=20, max=1145, avg=430.66, stdev=499.88 lat (msec): min=20, max=1145, avg=430.70, stdev=499.87 clat percentiles (msec): | 1.00th=[ 21], 5.00th=[ 21], 10.00th=[ 21], 20.00th=[ 69], | 30.00th=[ 69], 40.00th=[ 104], 50.00th=[ 104], 60.00th=[ 264], | 70.00th=[ 986], 80.00th=[ 986], 90.00th=[ 1150], 95.00th=[ 1150], | 99.00th=[ 1150], 99.50th=[ 1150], 99.90th=[ 1150], 99.95th=[ 1150], | 99.99th=[ 1150] bw ( KiB/s): min= 2048, max= 6131, per=65.24%, avg=4091.67, stdev=2041.50, samples=3 iops : min= 2, max= 5, avg= 3.67, stdev= 1.53, samples=3 lat (usec) : 500=8.33%, 750=8.33% lat (msec) : 10=8.33%, 50=8.33%, 100=25.00%, 250=8.33%, 500=8.33% lat (msec) : 1000=16.67% cpu : usr=0.00%, sys=10.12%, ctx=413, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5702: Sat Dec 28 02:40:25 2024 read: IOPS=1, BW=1120KiB/s (1147kB/s)(5120KiB/4571msec) clat (usec): min=326, max=1852.8k, avg=534496.27, stdev=805686.20 lat (usec): min=327, max=1852.8k, avg=534497.33, stdev=805685.77 clat percentiles (usec): | 1.00th=[ 326], 5.00th=[ 326], 10.00th=[ 326], | 20.00th=[ 326], 30.00th=[ 824], 40.00th=[ 824], | 50.00th=[ 50594], 60.00th=[ 50594], 70.00th=[ 767558], | 80.00th=[ 767558], 90.00th=[1853883], 95.00th=[1853883], | 99.00th=[1853883], 99.50th=[1853883], 99.90th=[1853883], | 99.95th=[1853883], 99.99th=[1853883] bw ( KiB/s): min= 4096, max= 4096, per=91.43%, avg=4096.00, stdev= 0.00, samples=2 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=2 write: IOPS=1, BW=1568KiB/s (1606kB/s)(7168KiB/4571msec) clat (msec): min=32, max=1263, avg=266.51, stdev=448.39 lat (msec): min=32, max=1263, avg=266.57, stdev=448.39 clat percentiles (msec): | 1.00th=[ 33], 5.00th=[ 33], 10.00th=[ 33], 20.00th=[ 39], | 30.00th=[ 50], 40.00th=[ 50], 50.00th=[ 80], 60.00th=[ 117], | 70.00th=[ 117], 80.00th=[ 284], 90.00th=[ 1267], 95.00th=[ 1267], | 99.00th=[ 1267], 99.50th=[ 1267], 99.90th=[ 1267], 99.95th=[ 1267], | 99.99th=[ 1267] bw ( KiB/s): min= 2048, max= 6144, per=76.19%, avg=4778.67, stdev=2364.83, samples=3 iops : min= 2, max= 6, avg= 4.67, stdev= 2.31, samples=3 lat (usec) : 500=8.33%, 1000=8.33% lat (msec) : 50=25.00%, 100=16.67%, 250=8.33%, 500=8.33%, 1000=8.33% cpu : usr=0.02%, sys=9.12%, ctx=697, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=4480KiB/s (4588kB/s), 541KiB/s-2482KiB/s (554kB/s-2542kB/s), io=20.0MiB (20.0MB), run=2888-4571msec WRITE: bw=6273KiB/s (6423kB/s), 1568KiB/s-2705KiB/s (1606kB/s-2770kB/s), io=28.0MiB (29.4MB), run=2888-4571msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=5711: Sat Dec 28 02:40:28 2024 read: IOPS=12, BW=51.5MiB/s (54.0MB/s)(12.0MiB/233msec) clat (msec): min=26, max=144, avg=71.84, stdev=63.36 lat (msec): min=26, max=144, avg=71.84, stdev=63.36 clat percentiles (msec): | 1.00th=[ 27], 5.00th=[ 27], 10.00th=[ 27], 20.00th=[ 27], | 30.00th=[ 27], 40.00th=[ 45], 50.00th=[ 45], 60.00th=[ 45], | 70.00th=[ 144], 80.00th=[ 144], 90.00th=[ 144], 95.00th=[ 144], | 99.00th=[ 144], 99.50th=[ 144], 99.90th=[ 144], 99.95th=[ 144], | 99.99th=[ 144] lat (msec) : 50=66.67%, 250=33.33% cpu : usr=0.00%, sys=25.86%, ctx=70, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5713: Sat Dec 28 02:40:28 2024 read: IOPS=0, BW=1903KiB/s (1949kB/s)(4096KiB/2152msec) clat (nsec): min=319434k, max=319434k, avg=319433820.00, stdev= 0.00 lat (nsec): min=319436k, max=319436k, avg=319435875.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 321], 5.00th=[ 321], 10.00th=[ 321], 20.00th=[ 321], | 30.00th=[ 321], 40.00th=[ 321], 50.00th=[ 321], 60.00th=[ 321], | 70.00th=[ 321], 80.00th=[ 321], 90.00th=[ 321], 95.00th=[ 321], | 99.00th=[ 321], 99.50th=[ 321], 99.90th=[ 321], 99.95th=[ 321], | 99.99th=[ 321] bw ( KiB/s): min= 6014, max= 6014, per=79.00%, avg=6014.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=0, BW=3807KiB/s (3898kB/s)(8192KiB/2152msec) clat (msec): min=70, max=1749, avg=909.66, stdev=1187.01 lat (msec): min=70, max=1749, avg=909.83, stdev=1187.15 clat percentiles (msec): | 1.00th=[ 71], 5.00th=[ 71], 10.00th=[ 71], 20.00th=[ 71], | 30.00th=[ 71], 40.00th=[ 71], 50.00th=[ 71], 60.00th=[ 1754], | 70.00th=[ 1754], 80.00th=[ 1754], 90.00th=[ 1754], 95.00th=[ 1754], | 99.00th=[ 1754], 99.50th=[ 1754], 99.90th=[ 1754], 99.95th=[ 1754], | 99.99th=[ 1754] bw ( KiB/s): min= 6014, max= 6014, per=39.55%, avg=6014.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 500=33.33% cpu : usr=0.00%, sys=3.95%, ctx=103, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5716: Sat Dec 28 02:40:28 2024 write: IOPS=1, BW=5702KiB/s (5839kB/s)(12.0MiB/2155msec) clat (msec): min=220, max=1666, avg=714.25, stdev=824.79 lat (msec): min=221, max=1666, avg=714.63, stdev=824.80 clat percentiles (msec): | 1.00th=[ 222], 5.00th=[ 222], 10.00th=[ 222], 20.00th=[ 222], | 30.00th=[ 222], 40.00th=[ 255], 50.00th=[ 255], 60.00th=[ 255], | 70.00th=[ 1670], 80.00th=[ 1670], 90.00th=[ 1670], 95.00th=[ 1670], | 99.00th=[ 1670], 99.50th=[ 1670], 99.90th=[ 1670], 99.95th=[ 1670], | 99.99th=[ 1670] bw ( KiB/s): min= 5587, max= 5587, per=36.74%, avg=5587.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=33.33% cpu : usr=0.00%, sys=6.82%, ctx=54, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5717: Sat Dec 28 02:40:28 2024 write: IOPS=6, BW=25.6MiB/s (26.9MB/s)(12.0MiB/468msec) clat (msec): min=67, max=271, avg=140.99, stdev=113.21 lat (msec): min=68, max=271, avg=141.25, stdev=113.24 clat percentiles (msec): | 1.00th=[ 68], 5.00th=[ 68], 10.00th=[ 68], 20.00th=[ 68], | 30.00th=[ 68], 40.00th=[ 84], 50.00th=[ 84], 60.00th=[ 84], | 70.00th=[ 271], 80.00th=[ 271], 90.00th=[ 271], 95.00th=[ 271], | 99.00th=[ 271], 99.50th=[ 271], 99.90th=[ 271], 99.95th=[ 271], | 99.99th=[ 271] lat (msec) : 100=66.67%, 500=33.33% cpu : usr=0.43%, sys=14.78%, ctx=65, majf=0, minf=28 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=7613KiB/s (7796kB/s), 1903KiB/s-51.5MiB/s (1949kB/s-54.0MB/s), io=16.0MiB (16.8MB), run=233-2152msec WRITE: bw=14.8MiB/s (15.6MB/s), 3807KiB/s-25.6MiB/s (3898kB/s-26.9MB/s), io=32.0MiB (33.6MB), run=468-2155msec rand-rw: (groupid=0, jobs=1): err= 0: pid=5712: Sat Dec 28 02:40:28 2024 read: IOPS=1, BW=6032KiB/s (6177kB/s)(12.0MiB/2037msec) clat (msec): min=143, max=1391, avg=671.90, stdev=645.70 lat (msec): min=143, max=1391, avg=671.90, stdev=645.70 clat percentiles (msec): | 1.00th=[ 144], 5.00th=[ 144], 10.00th=[ 144], 20.00th=[ 144], | 30.00th=[ 144], 40.00th=[ 481], 50.00th=[ 481], 60.00th=[ 481], | 70.00th=[ 1385], 80.00th=[ 1385], 90.00th=[ 1385], 95.00th=[ 1385], | 99.00th=[ 1385], 99.50th=[ 1385], 99.90th=[ 1385], 99.95th=[ 1385], | 99.99th=[ 1385] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=33.33%, 500=33.33% cpu : usr=0.00%, sys=50.29%, ctx=594, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5714: Sat Dec 28 02:40:28 2024 read: IOPS=0, BW=1995KiB/s (2043kB/s)(4096KiB/2053msec) clat (nsec): min=178523k, max=178523k, avg=178523241.00, stdev= 0.00 lat (nsec): min=178524k, max=178524k, avg=178524176.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 180], 5.00th=[ 180], 10.00th=[ 180], 20.00th=[ 180], | 30.00th=[ 180], 40.00th=[ 180], 50.00th=[ 180], 60.00th=[ 180], | 70.00th=[ 180], 80.00th=[ 180], 90.00th=[ 180], 95.00th=[ 180], | 99.00th=[ 180], 99.50th=[ 180], 99.90th=[ 180], 99.95th=[ 180], | 99.99th=[ 180] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=0, BW=3990KiB/s (4086kB/s)(8192KiB/2053msec) clat (msec): min=165, max=1702, avg=934.00, stdev=1086.36 lat (msec): min=166, max=1702, avg=934.20, stdev=1086.29 clat percentiles (msec): | 1.00th=[ 167], 5.00th=[ 167], 10.00th=[ 167], 20.00th=[ 167], | 30.00th=[ 167], 40.00th=[ 167], 50.00th=[ 167], 60.00th=[ 1703], | 70.00th=[ 1703], 80.00th=[ 1703], 90.00th=[ 1703], 95.00th=[ 1703], | 99.00th=[ 1703], 99.50th=[ 1703], 99.90th=[ 1703], 99.95th=[ 1703], | 99.99th=[ 1703] bw ( KiB/s): min= 8192, max= 8192, per=55.70%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67% cpu : usr=0.00%, sys=11.84%, ctx=168, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5715: Sat Dec 28 02:40:28 2024 write: IOPS=1, BW=5515KiB/s (5648kB/s)(12.0MiB/2228msec) clat (msec): min=258, max=1657, avg=739.91, stdev=794.79 lat (msec): min=259, max=1657, avg=740.14, stdev=794.81 clat percentiles (msec): | 1.00th=[ 259], 5.00th=[ 259], 10.00th=[ 259], 20.00th=[ 259], | 30.00th=[ 259], 40.00th=[ 305], 50.00th=[ 305], 60.00th=[ 305], | 70.00th=[ 1653], 80.00th=[ 1653], 90.00th=[ 1653], 95.00th=[ 1653], | 99.00th=[ 1653], 99.50th=[ 1653], 99.90th=[ 1653], 99.95th=[ 1653], | 99.99th=[ 1653] bw ( KiB/s): min= 5859, max= 8192, per=47.77%, avg=7025.50, stdev=1649.68, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 500=66.67% cpu : usr=0.00%, sys=9.88%, ctx=59, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5718: Sat Dec 28 02:40:28 2024 write: IOPS=1, BW=7006KiB/s (7174kB/s)(12.0MiB/1754msec) clat (msec): min=116, max=1480, avg=581.53, stdev=778.88 lat (msec): min=117, max=1480, avg=581.90, stdev=778.79 clat percentiles (msec): | 1.00th=[ 117], 5.00th=[ 117], 10.00th=[ 117], 20.00th=[ 117], | 30.00th=[ 117], 40.00th=[ 148], 50.00th=[ 148], 60.00th=[ 148], | 70.00th=[ 1485], 80.00th=[ 1485], 90.00th=[ 1485], 95.00th=[ 1485], | 99.00th=[ 1485], 99.50th=[ 1485], 99.90th=[ 1485], 99.95th=[ 1485], | 99.99th=[ 1485] bw ( KiB/s): min= 8192, max= 8192, per=55.70%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67% cpu : usr=0.00%, sys=15.80%, ctx=84, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=7981KiB/s (8172kB/s), 1995KiB/s-6032KiB/s (2043kB/s-6177kB/s), io=16.0MiB (16.8MB), run=2037-2053msec WRITE: bw=14.4MiB/s (15.1MB/s), 3990KiB/s-7006KiB/s (4086kB/s-7174kB/s), io=32.0MiB (33.6MB), run=1754-2228msec