== sanity test 398c: run fio to test AIO ================= 18:34:54 (1746138894) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB) copied, 0.137442 s, 305 MB/s osc.lustre-OST0000-osc-ffff88012ea8b800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=6706: Thu May 1 18:36:28 2025 write: IOPS=27, BW=112KiB/s (114kB/s)(10.0MiB/91760msec) slat (usec): min=11, max=3893, avg=128.03, stdev=248.77 clat (msec): min=48, max=1050, avg=572.78, stdev=144.53 lat (msec): min=48, max=1050, avg=572.91, stdev=144.54 clat percentiles (msec): | 1.00th=[ 224], 5.00th=[ 300], 10.00th=[ 397], 20.00th=[ 464], | 30.00th=[ 510], 40.00th=[ 550], 50.00th=[ 575], 60.00th=[ 600], | 70.00th=[ 642], 80.00th=[ 684], 90.00th=[ 743], 95.00th=[ 818], | 99.00th=[ 936], 99.50th=[ 969], 99.90th=[ 1053], 99.95th=[ 1053], | 99.99th=[ 1053] bw ( KiB/s): min= 24, max= 256, per=24.88%, avg=110.98, stdev=33.11, samples=183 iops : min= 6, max= 64, avg=27.54, stdev= 8.32, samples=183 lat (msec) : 50=0.04%, 100=0.04%, 250=2.46%, 500=24.57%, 750=63.28% lat (msec) : 1000=9.38% cpu : usr=0.11%, sys=0.54%, ctx=1517, majf=0, minf=32 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=6707: Thu May 1 18:36:28 2025 write: IOPS=27, BW=112KiB/s (114kB/s)(10.0MiB/91803msec) slat (usec): min=11, max=8759, avg=129.90, stdev=303.06 clat (msec): min=165, max=1065, avg=573.02, stdev=140.48 lat (msec): min=165, max=1065, avg=573.15, stdev=140.49 clat percentiles (msec): | 1.00th=[ 245], 5.00th=[ 313], 10.00th=[ 405], 20.00th=[ 472], | 30.00th=[ 510], 40.00th=[ 542], 50.00th=[ 567], 60.00th=[ 600], | 70.00th=[ 642], 80.00th=[ 676], 90.00th=[ 751], 95.00th=[ 810], | 99.00th=[ 919], 99.50th=[ 953], 99.90th=[ 1028], 99.95th=[ 1053], | 99.99th=[ 1062] bw ( KiB/s): min= 31, max= 255, per=24.84%, avg=110.80, stdev=34.84, samples=183 iops : min= 7, max= 63, avg=27.45, stdev= 8.72, samples=183 lat (msec) : 250=1.68%, 500=24.96%, 750=63.63%, 1000=9.61% cpu : usr=0.11%, sys=0.50%, ctx=1458, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=6708: Thu May 1 18:36:28 2025 write: IOPS=27, BW=112KiB/s (114kB/s)(10.0MiB/91802msec) slat (usec): min=11, max=3954, avg=127.12, stdev=252.25 clat (msec): min=165, max=1050, avg=572.98, stdev=142.37 lat (msec): min=165, max=1050, avg=573.11, stdev=142.39 clat percentiles (msec): | 1.00th=[ 234], 5.00th=[ 330], 10.00th=[ 409], 20.00th=[ 464], | 30.00th=[ 510], 40.00th=[ 542], 50.00th=[ 575], 60.00th=[ 600], | 70.00th=[ 634], 80.00th=[ 676], 90.00th=[ 751], 95.00th=[ 810], | 99.00th=[ 927], 99.50th=[ 961], 99.90th=[ 1053], 99.95th=[ 1053], | 99.99th=[ 1053] bw ( KiB/s): min= 32, max= 256, per=24.88%, avg=110.96, stdev=34.31, samples=183 iops : min= 8, max= 64, avg=27.52, stdev= 8.60, samples=183 lat (msec) : 250=2.34%, 500=24.77%, 750=63.05%, 1000=9.57% cpu : usr=0.07%, sys=0.57%, ctx=1518, majf=0, minf=28 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=6709: Thu May 1 18:36:28 2025 write: IOPS=27, BW=112KiB/s (114kB/s)(10.0MiB/91759msec) slat (usec): min=11, max=4333, avg=125.61, stdev=272.70 clat (msec): min=54, max=1031, avg=572.86, stdev=146.10 lat (msec): min=54, max=1031, avg=572.98, stdev=146.11 clat percentiles (msec): | 1.00th=[ 226], 5.00th=[ 309], 10.00th=[ 401], 20.00th=[ 472], | 30.00th=[ 510], 40.00th=[ 542], 50.00th=[ 575], 60.00th=[ 600], | 70.00th=[ 642], 80.00th=[ 684], 90.00th=[ 760], 95.00th=[ 818], | 99.00th=[ 936], 99.50th=[ 961], 99.90th=[ 1028], 99.95th=[ 1028], | 99.99th=[ 1028] bw ( KiB/s): min= 39, max= 256, per=24.89%, avg=110.99, stdev=34.63, samples=183 iops : min= 9, max= 64, avg=27.55, stdev= 8.68, samples=183 lat (msec) : 100=0.08%, 250=2.15%, 500=25.20%, 750=61.99%, 1000=10.43% cpu : usr=0.07%, sys=0.54%, ctx=1420, majf=0, minf=29 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=446KiB/s (457kB/s), 112KiB/s-112KiB/s (114kB/s-114kB/s), io=40.0MiB (41.9MB), run=91759-91803msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=6792: Thu May 1 18:37:22 2025 read: IOPS=23, BW=94.9KiB/s (97.2kB/s)(5048KiB/53185msec) slat (usec): min=24, max=19754, avg=4844.81, stdev=2097.38 clat (msec): min=4, max=514, avg=220.55, stdev=53.42 lat (msec): min=7, max=523, avg=225.40, stdev=53.96 clat percentiles (msec): | 1.00th=[ 34], 5.00th=[ 146], 10.00th=[ 161], 20.00th=[ 180], | 30.00th=[ 194], 40.00th=[ 207], 50.00th=[ 220], 60.00th=[ 234], | 70.00th=[ 247], 80.00th=[ 264], 90.00th=[ 288], 95.00th=[ 300], | 99.00th=[ 338], 99.50th=[ 363], 99.90th=[ 510], 99.95th=[ 514], | 99.99th=[ 514] bw ( KiB/s): min= 31, max= 184, per=25.24%, avg=94.63, stdev=32.03, samples=106 iops : min= 7, max= 46, avg=23.48, stdev= 8.06, samples=106 write: IOPS=24, BW=97.6KiB/s (99.0kB/s)(5192KiB/53185msec) slat (usec): min=16, max=2929, avg=146.65, stdev=264.08 clat (msec): min=69, max=880, avg=435.79, stdev=116.09 lat (msec): min=69, max=880, avg=435.94, stdev=116.10 clat percentiles (msec): | 1.00th=[ 188], 5.00th=[ 257], 10.00th=[ 296], 20.00th=[ 334], | 30.00th=[ 376], 40.00th=[ 405], 50.00th=[ 430], 60.00th=[ 456], | 70.00th=[ 493], 80.00th=[ 531], 90.00th=[ 584], 95.00th=[ 634], | 99.00th=[ 768], 99.50th=[ 810], 99.90th=[ 860], 99.95th=[ 877], | 99.99th=[ 877] bw ( KiB/s): min= 40, max= 192, per=24.86%, avg=96.97, stdev=26.54, samples=106 iops : min= 10, max= 48, avg=24.07, stdev= 6.63, samples=106 lat (msec) : 10=0.27%, 20=0.12%, 50=0.23%, 100=0.23%, 250=37.62% lat (msec) : 500=47.89%, 750=13.12%, 1000=0.51% cpu : usr=0.00%, sys=1.86%, ctx=2711, majf=0, minf=32 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6793: Thu May 1 18:37:22 2025 read: IOPS=23, BW=93.2KiB/s (95.4kB/s)(4976KiB/53411msec) slat (usec): min=1483, max=24127, avg=4969.26, stdev=2144.84 clat (msec): min=15, max=352, avg=220.35, stdev=47.37 lat (msec): min=19, max=365, avg=225.32, stdev=47.71 clat percentiles (msec): | 1.00th=[ 99], 5.00th=[ 150], 10.00th=[ 165], 20.00th=[ 184], | 30.00th=[ 194], 40.00th=[ 209], 50.00th=[ 220], 60.00th=[ 232], | 70.00th=[ 245], 80.00th=[ 259], 90.00th=[ 284], 95.00th=[ 296], | 99.00th=[ 326], 99.50th=[ 334], 99.90th=[ 351], 99.95th=[ 351], | 99.99th=[ 351] bw ( KiB/s): min= 23, max= 168, per=24.63%, avg=92.35, stdev=32.09, samples=106 iops : min= 5, max= 42, avg=22.90, stdev= 8.00, samples=106 write: IOPS=24, BW=98.6KiB/s (101kB/s)(5264KiB/53411msec) slat (usec): min=13, max=5741, avg=147.23, stdev=326.69 clat (msec): min=148, max=820, avg=435.45, stdev=114.39 lat (msec): min=148, max=821, avg=435.60, stdev=114.39 clat percentiles (msec): | 1.00th=[ 194], 5.00th=[ 259], 10.00th=[ 292], 20.00th=[ 338], | 30.00th=[ 372], 40.00th=[ 405], 50.00th=[ 430], 60.00th=[ 460], | 70.00th=[ 493], 80.00th=[ 527], 90.00th=[ 575], 95.00th=[ 634], | 99.00th=[ 751], 99.50th=[ 793], 99.90th=[ 818], 99.95th=[ 818], | 99.99th=[ 818] bw ( KiB/s): min= 40, max= 192, per=25.07%, avg=97.78, stdev=28.66, samples=106 iops : min= 10, max= 48, avg=24.25, stdev= 7.24, samples=106 lat (msec) : 20=0.04%, 50=0.39%, 100=0.08%, 250=37.30%, 500=47.93% lat (msec) : 750=13.75%, 1000=0.51% cpu : usr=0.19%, sys=1.67%, ctx=2739, majf=0, minf=32 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6794: Thu May 1 18:37:22 2025 read: IOPS=23, BW=93.3KiB/s (95.5kB/s)(4984KiB/53444msec) slat (usec): min=1490, max=24734, avg=4867.52, stdev=2119.07 clat (msec): min=5, max=417, avg=221.76, stdev=48.09 lat (msec): min=9, max=421, avg=226.63, stdev=48.43 clat percentiles (msec): | 1.00th=[ 97], 5.00th=[ 150], 10.00th=[ 165], 20.00th=[ 182], | 30.00th=[ 197], 40.00th=[ 209], 50.00th=[ 222], 60.00th=[ 236], | 70.00th=[ 247], 80.00th=[ 262], 90.00th=[ 284], 95.00th=[ 300], | 99.00th=[ 326], 99.50th=[ 334], 99.90th=[ 414], 99.95th=[ 418], | 99.99th=[ 418] bw ( KiB/s): min= 16, max= 192, per=24.85%, avg=93.20, stdev=31.42, samples=106 iops : min= 4, max= 48, avg=23.07, stdev= 7.89, samples=106 write: IOPS=24, BW=98.3KiB/s (101kB/s)(5256KiB/53444msec) slat (usec): min=14, max=4654, avg=137.24, stdev=260.73 clat (msec): min=173, max=836, avg=434.63, stdev=110.36 lat (msec): min=173, max=836, avg=434.77, stdev=110.37 clat percentiles (msec): | 1.00th=[ 207], 5.00th=[ 262], 10.00th=[ 305], 20.00th=[ 338], | 30.00th=[ 376], 40.00th=[ 405], 50.00th=[ 430], 60.00th=[ 456], | 70.00th=[ 493], 80.00th=[ 518], 90.00th=[ 575], 95.00th=[ 634], | 99.00th=[ 709], 99.50th=[ 768], 99.90th=[ 835], 99.95th=[ 835], | 99.99th=[ 835] bw ( KiB/s): min= 47, max= 208, per=24.94%, avg=97.27, stdev=27.63, samples=106 iops : min= 11, max= 52, avg=24.08, stdev= 6.92, samples=106 lat (msec) : 10=0.04%, 20=0.04%, 50=0.27%, 100=0.20%, 250=36.91% lat (msec) : 500=49.02%, 750=13.12%, 1000=0.39% cpu : usr=0.22%, sys=1.61%, ctx=2744, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=6795: Thu May 1 18:37:22 2025 read: IOPS=23, BW=94.7KiB/s (96.0kB/s)(5060KiB/53443msec) slat (usec): min=1561, max=19456, avg=4934.44, stdev=1998.64 clat (msec): min=5, max=421, avg=221.36, stdev=47.80 lat (msec): min=9, max=427, avg=226.30, stdev=48.14 clat percentiles (msec): | 1.00th=[ 103], 5.00th=[ 150], 10.00th=[ 163], 20.00th=[ 182], | 30.00th=[ 199], 40.00th=[ 211], 50.00th=[ 224], 60.00th=[ 234], | 70.00th=[ 245], 80.00th=[ 257], 90.00th=[ 279], 95.00th=[ 296], | 99.00th=[ 334], 99.50th=[ 347], 99.90th=[ 418], 99.95th=[ 422], | 99.99th=[ 422] bw ( KiB/s): min= 32, max= 168, per=25.23%, avg=94.61, stdev=29.95, samples=106 iops : min= 8, max= 42, avg=23.46, stdev= 7.55, samples=106 write: IOPS=24, BW=96.9KiB/s (99.3kB/s)(5180KiB/53443msec) slat (usec): min=14, max=9137, avg=138.24, stdev=342.78 clat (msec): min=165, max=836, avg=438.22, stdev=110.23 lat (msec): min=165, max=836, avg=438.36, stdev=110.23 clat percentiles (msec): | 1.00th=[ 201], 5.00th=[ 264], 10.00th=[ 305], 20.00th=[ 347], | 30.00th=[ 380], 40.00th=[ 405], 50.00th=[ 435], 60.00th=[ 460], | 70.00th=[ 489], 80.00th=[ 531], 90.00th=[ 584], 95.00th=[ 634], | 99.00th=[ 718], 99.50th=[ 776], 99.90th=[ 818], 99.95th=[ 835], | 99.99th=[ 835] bw ( KiB/s): min= 55, max= 192, per=24.67%, avg=96.20, stdev=24.73, samples=106 iops : min= 13, max= 48, avg=23.86, stdev= 6.19, samples=106 lat (msec) : 10=0.08%, 50=0.23%, 100=0.12%, 250=38.09%, 500=47.77% lat (msec) : 750=13.36%, 1000=0.35% cpu : usr=0.10%, sys=1.75%, ctx=2763, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=375KiB/s (385kB/s), 93.2KiB/s-94.9KiB/s (95.4kB/s-97.2kB/s), io=19.6MiB (20.5MB), run=53185-53444msec WRITE: bw=391KiB/s (400kB/s), 96.9KiB/s-98.6KiB/s (99.3kB/s-101kB/s), io=20.4MiB (21.4MB), run=53185-53444msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.7 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=6842: Thu May 1 18:37:24 2025 read: IOPS=4, BW=170MiB/s (178MB/s)(40.0MiB/235msec) slat (nsec): min=5119.7k, max=5119.7k, avg=5119651.00, stdev= 0.00 clat (nsec): min=214415k, max=214415k, avg=214414509.00, stdev= 0.00 lat (nsec): min=219557k, max=219557k, avg=219556535.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 215], 5.00th=[ 215], 10.00th=[ 215], 20.00th=[ 215], | 30.00th=[ 215], 40.00th=[ 215], 50.00th=[ 215], 60.00th=[ 215], | 70.00th=[ 215], 80.00th=[ 215], 90.00th=[ 215], 95.00th=[ 215], | 99.00th=[ 215], 99.50th=[ 215], 99.90th=[ 215], 99.95th=[ 215], | 99.99th=[ 215] lat (msec) : 250=100.00% cpu : usr=0.43%, sys=1.28%, ctx=44, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=170MiB/s (178MB/s), 170MiB/s-170MiB/s (178MB/s-178MB/s), io=40.0MiB (41.9MB), run=235-235msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout