== sanity test 398c: run fio to test AIO ================= 14:31:01 (1776191461) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.675287 s, 62.1 MB/s osc.lustre-OST0000-osc-ffff92e3882da800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=125043: Tue Apr 14 14:34:01 2026 write: IOPS=15, BW=60.2KiB/s (61.6kB/s)(10.0MiB/170110msec); 0 zone resets slat (usec): min=25, max=16926, avg=338.23, stdev=1085.81 clat (msec): min=364, max=2264, avg=1061.98, stdev=308.17 lat (msec): min=364, max=2264, avg=1062.33, stdev=308.17 clat percentiles (msec): | 1.00th=[ 472], 5.00th=[ 617], 10.00th=[ 726], 20.00th=[ 818], | 30.00th=[ 894], 40.00th=[ 953], 50.00th=[ 1011], 60.00th=[ 1083], | 70.00th=[ 1167], 80.00th=[ 1301], 90.00th=[ 1452], 95.00th=[ 1636], | 99.00th=[ 2022], 99.50th=[ 2089], 99.90th=[ 2198], 99.95th=[ 2198], | 99.99th=[ 2265] bw ( KiB/s): min= 7, max= 128, per=25.23%, avg=60.55, stdev=26.89, samples=332 iops : min= 1, max= 32, avg=14.61, stdev= 6.74, samples=332 lat (msec) : 500=1.91%, 750=9.92%, 1000=35.74%, 2000=51.13%, >=2000=1.29% cpu : usr=0.07%, sys=0.27%, ctx=1608, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=125044: Tue Apr 14 14:34:01 2026 write: IOPS=15, BW=60.2KiB/s (61.6kB/s)(10.0MiB/170111msec); 0 zone resets slat (usec): min=26, max=14890, avg=355.62, stdev=1160.37 clat (msec): min=359, max=2233, avg=1061.96, stdev=304.76 lat (msec): min=360, max=2233, avg=1062.32, stdev=304.76 clat percentiles (msec): | 1.00th=[ 485], 5.00th=[ 625], 10.00th=[ 709], 20.00th=[ 835], | 30.00th=[ 902], 40.00th=[ 961], 50.00th=[ 1011], 60.00th=[ 1083], | 70.00th=[ 1150], 80.00th=[ 1301], 90.00th=[ 1452], 95.00th=[ 1620], | 99.00th=[ 2039], 99.50th=[ 2089], 99.90th=[ 2198], 99.95th=[ 2232], | 99.99th=[ 2232] bw ( KiB/s): min= 7, max= 136, per=24.78%, avg=59.46, stdev=25.54, samples=338 iops : min= 1, max= 34, avg=14.34, stdev= 6.43, samples=338 lat (msec) : 500=1.41%, 750=11.09%, 1000=35.20%, 2000=51.05%, >=2000=1.25% cpu : usr=0.06%, sys=0.28%, ctx=1731, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=125045: Tue Apr 14 14:34:01 2026 write: IOPS=15, BW=60.3KiB/s (61.8kB/s)(10.0MiB/169704msec); 0 zone resets slat (usec): min=25, max=21766, avg=340.58, stdev=1166.10 clat (msec): min=124, max=2293, avg=1059.77, stdev=317.08 lat (msec): min=125, max=2294, avg=1060.12, stdev=317.03 clat percentiles (msec): | 1.00th=[ 401], 5.00th=[ 600], 10.00th=[ 709], 20.00th=[ 827], | 30.00th=[ 894], 40.00th=[ 953], 50.00th=[ 1011], 60.00th=[ 1083], | 70.00th=[ 1167], 80.00th=[ 1284], 90.00th=[ 1469], 95.00th=[ 1636], | 99.00th=[ 2056], 99.50th=[ 2089], 99.90th=[ 2265], 99.95th=[ 2299], | 99.99th=[ 2299] bw ( KiB/s): min= 7, max= 150, per=25.00%, avg=60.01, stdev=24.20, samples=335 iops : min= 1, max= 37, avg=14.47, stdev= 6.05, samples=335 lat (msec) : 250=0.51%, 500=1.56%, 750=10.04%, 1000=35.90%, 2000=50.59% lat (msec) : >=2000=1.41% cpu : usr=0.06%, sys=0.30%, ctx=1919, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=125046: Tue Apr 14 14:34:01 2026 write: IOPS=15, BW=60.2KiB/s (61.6kB/s)(10.0MiB/170094msec); 0 zone resets slat (usec): min=26, max=22445, avg=350.23, stdev=1197.71 clat (msec): min=382, max=2264, avg=1062.05, stdev=304.30 lat (msec): min=384, max=2264, avg=1062.40, stdev=304.32 clat percentiles (msec): | 1.00th=[ 481], 5.00th=[ 625], 10.00th=[ 709], 20.00th=[ 827], | 30.00th=[ 902], 40.00th=[ 961], 50.00th=[ 1020], 60.00th=[ 1083], | 70.00th=[ 1167], 80.00th=[ 1284], 90.00th=[ 1452], 95.00th=[ 1653], | 99.00th=[ 2022], 99.50th=[ 2089], 99.90th=[ 2198], 99.95th=[ 2198], | 99.99th=[ 2265] bw ( KiB/s): min= 7, max= 127, per=24.87%, avg=59.69, stdev=25.03, samples=336 iops : min= 1, max= 31, avg=14.38, stdev= 6.25, samples=336 lat (msec) : 500=1.41%, 750=11.17%, 1000=34.22%, 2000=51.95%, >=2000=1.25% cpu : usr=0.05%, sys=0.29%, ctx=1807, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=241KiB/s (247kB/s), 60.2KiB/s-60.3KiB/s (61.6kB/s-61.8kB/s), io=40.0MiB (41.9MB), run=169704-170111msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=125080: Tue Apr 14 14:36:02 2026 read: IOPS=11, BW=44.9KiB/s (46.0kB/s)(5048KiB/112361msec) slat (usec): min=39, max=70020, avg=13718.51, stdev=7848.57 clat (msec): min=10, max=1482, avg=448.96, stdev=143.97 lat (msec): min=10, max=1500, avg=462.69, stdev=145.63 clat percentiles (msec): | 1.00th=[ 94], 5.00th=[ 288], 10.00th=[ 313], 20.00th=[ 351], | 30.00th=[ 376], 40.00th=[ 397], 50.00th=[ 426], 60.00th=[ 456], | 70.00th=[ 485], 80.00th=[ 542], 90.00th=[ 617], 95.00th=[ 701], | 99.00th=[ 919], 99.50th=[ 978], 99.90th=[ 1452], 99.95th=[ 1485], | 99.99th=[ 1485] bw ( KiB/s): min= 7, max= 119, per=25.55%, avg=45.22, stdev=21.94, samples=218 iops : min= 1, max= 29, avg=10.70, stdev= 5.50, samples=218 write: IOPS=11, BW=46.2KiB/s (47.3kB/s)(5192KiB/112361msec); 0 zone resets slat (usec): min=29, max=12592, avg=368.46, stdev=924.79 clat (msec): min=75, max=2205, avg=933.53, stdev=264.24 lat (msec): min=77, max=2205, avg=933.91, stdev=264.29 clat percentiles (msec): | 1.00th=[ 388], 5.00th=[ 514], 10.00th=[ 600], 20.00th=[ 718], | 30.00th=[ 793], 40.00th=[ 852], 50.00th=[ 919], 60.00th=[ 986], | 70.00th=[ 1070], 80.00th=[ 1167], 90.00th=[ 1267], 95.00th=[ 1368], | 99.00th=[ 1519], 99.50th=[ 1720], 99.90th=[ 2198], 99.95th=[ 2198], | 99.99th=[ 2198] bw ( KiB/s): min= 7, max= 119, per=24.73%, avg=45.75, stdev=19.59, samples=221 iops : min= 1, max= 29, avg=10.84, stdev= 4.91, samples=221 lat (msec) : 20=0.12%, 50=0.04%, 100=0.51%, 250=0.62%, 500=37.03% lat (msec) : 750=21.48%, 1000=20.51%, 2000=19.53%, >=2000=0.16% cpu : usr=0.13%, sys=0.69%, ctx=3176, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=125081: Tue Apr 14 14:36:02 2026 read: IOPS=11, BW=44.1KiB/s (45.1kB/s)(4976KiB/112876msec) slat (usec): min=2614, max=58847, avg=13917.71, stdev=8068.99 clat (msec): min=26, max=980, avg=450.60, stdev=126.01 lat (msec): min=32, max=1024, avg=464.53, stdev=126.98 clat percentiles (msec): | 1.00th=[ 247], 5.00th=[ 284], 10.00th=[ 317], 20.00th=[ 351], | 30.00th=[ 380], 40.00th=[ 401], 50.00th=[ 430], 60.00th=[ 460], | 70.00th=[ 489], 80.00th=[ 531], 90.00th=[ 634], 95.00th=[ 701], | 99.00th=[ 810], 99.50th=[ 911], 99.90th=[ 969], 99.95th=[ 978], | 99.99th=[ 978] bw ( KiB/s): min= 7, max= 127, per=24.55%, avg=43.46, stdev=21.57, samples=223 iops : min= 1, max= 31, avg=10.26, stdev= 5.41, samples=223 write: IOPS=11, BW=46.6KiB/s (47.8kB/s)(5264KiB/112876msec); 0 zone resets slat (usec): min=29, max=21805, avg=516.87, stdev=1610.13 clat (msec): min=271, max=1718, avg=931.88, stdev=256.77 lat (msec): min=271, max=1722, avg=932.40, stdev=256.79 clat percentiles (msec): | 1.00th=[ 372], 5.00th=[ 523], 10.00th=[ 600], 20.00th=[ 709], | 30.00th=[ 793], 40.00th=[ 852], 50.00th=[ 919], 60.00th=[ 978], | 70.00th=[ 1083], 80.00th=[ 1167], 90.00th=[ 1267], 95.00th=[ 1351], | 99.00th=[ 1519], 99.50th=[ 1586], 99.90th=[ 1636], 99.95th=[ 1720], | 99.99th=[ 1720] bw ( KiB/s): min= 7, max= 111, per=24.92%, avg=46.11, stdev=21.17, samples=223 iops : min= 1, max= 27, avg=10.93, stdev= 5.32, samples=223 lat (msec) : 50=0.04%, 100=0.20%, 250=0.35%, 500=36.88%, 750=22.70% lat (msec) : 1000=20.35%, 2000=19.49% cpu : usr=0.14%, sys=0.82%, ctx=3470, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=125082: Tue Apr 14 14:36:02 2026 read: IOPS=11, BW=44.2KiB/s (45.3kB/s)(4984KiB/112711msec) slat (usec): min=1468, max=58477, avg=13895.43, stdev=7962.71 clat (msec): min=14, max=1023, avg=449.23, stdev=129.58 lat (msec): min=26, max=1064, avg=463.13, stdev=130.66 clat percentiles (msec): | 1.00th=[ 247], 5.00th=[ 292], 10.00th=[ 309], 20.00th=[ 347], | 30.00th=[ 372], 40.00th=[ 401], 50.00th=[ 426], 60.00th=[ 456], | 70.00th=[ 489], 80.00th=[ 542], 90.00th=[ 609], 95.00th=[ 693], | 99.00th=[ 885], 99.50th=[ 969], 99.90th=[ 1020], 99.95th=[ 1020], | 99.99th=[ 1020] bw ( KiB/s): min= 7, max= 120, per=25.03%, avg=44.30, stdev=22.15, samples=220 iops : min= 1, max= 30, avg=10.48, stdev= 5.54, samples=220 write: IOPS=11, BW=46.6KiB/s (47.8kB/s)(5256KiB/112711msec); 0 zone resets slat (usec): min=29, max=33911, avg=509.73, stdev=1776.87 clat (msec): min=316, max=1629, avg=931.54, stdev=254.89 lat (msec): min=316, max=1629, avg=932.05, stdev=254.95 clat percentiles (msec): | 1.00th=[ 359], 5.00th=[ 518], 10.00th=[ 617], 20.00th=[ 718], | 30.00th=[ 785], 40.00th=[ 852], 50.00th=[ 919], 60.00th=[ 978], | 70.00th=[ 1083], 80.00th=[ 1167], 90.00th=[ 1267], 95.00th=[ 1334], | 99.00th=[ 1502], 99.50th=[ 1552], 99.90th=[ 1620], 99.95th=[ 1636], | 99.99th=[ 1636] bw ( KiB/s): min= 7, max= 119, per=24.79%, avg=45.87, stdev=19.31, samples=223 iops : min= 1, max= 29, avg=10.87, stdev= 4.88, samples=223 lat (msec) : 20=0.04%, 250=0.51%, 500=37.07%, 750=22.34%, 1000=20.16% lat (msec) : 2000=19.88% cpu : usr=0.12%, sys=0.81%, ctx=3377, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=125083: Tue Apr 14 14:36:02 2026 read: IOPS=11, BW=45.0KiB/s (46.1kB/s)(5060KiB/112361msec) slat (usec): min=2623, max=54260, avg=14113.64, stdev=7901.58 clat (msec): min=33, max=1007, avg=446.62, stdev=134.55 lat (msec): min=44, max=1039, avg=460.74, stdev=135.56 clat percentiles (msec): | 1.00th=[ 220], 5.00th=[ 288], 10.00th=[ 313], 20.00th=[ 347], | 30.00th=[ 376], 40.00th=[ 401], 50.00th=[ 426], 60.00th=[ 456], | 70.00th=[ 485], 80.00th=[ 531], 90.00th=[ 609], 95.00th=[ 718], | 99.00th=[ 927], 99.50th=[ 986], 99.90th=[ 1003], 99.95th=[ 1011], | 99.99th=[ 1011] bw ( KiB/s): min= 7, max= 112, per=25.54%, avg=45.20, stdev=19.87, samples=219 iops : min= 1, max= 28, avg=10.70, stdev= 4.98, samples=219 write: IOPS=11, BW=46.1KiB/s (47.2kB/s)(5180KiB/112361msec); 0 zone resets slat (usec): min=31, max=20780, avg=418.99, stdev=1255.55 clat (msec): min=235, max=1842, avg=936.26, stdev=259.50 lat (msec): min=235, max=1843, avg=936.68, stdev=259.51 clat percentiles (msec): | 1.00th=[ 393], 5.00th=[ 523], 10.00th=[ 617], 20.00th=[ 709], | 30.00th=[ 785], 40.00th=[ 860], 50.00th=[ 911], 60.00th=[ 986], | 70.00th=[ 1083], 80.00th=[ 1167], 90.00th=[ 1267], 95.00th=[ 1368], | 99.00th=[ 1603], 99.50th=[ 1703], 99.90th=[ 1787], 99.95th=[ 1838], | 99.99th=[ 1838] bw ( KiB/s): min= 7, max= 95, per=24.43%, avg=45.19, stdev=18.73, samples=223 iops : min= 1, max= 23, avg=10.70, stdev= 4.72, samples=223 lat (msec) : 50=0.12%, 100=0.27%, 250=0.74%, 500=37.38%, 750=21.41% lat (msec) : 1000=20.82%, 2000=19.26% cpu : usr=0.12%, sys=0.81%, ctx=3409, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=178KiB/s (182kB/s), 44.1KiB/s-45.0KiB/s (45.1kB/s-46.1kB/s), io=19.6MiB (20.5MB), run=112361-112876msec WRITE: bw=185KiB/s (190kB/s), 46.1KiB/s-46.6KiB/s (47.2kB/s-47.8kB/s), io=20.4MiB (21.4MB), run=112361-112876msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=125112: Tue Apr 14 14:36:12 2026 read: IOPS=1, BW=75.9MiB/s (79.6MB/s)(40.0MiB/527msec) slat (nsec): min=57807k, max=57807k, avg=57806516.00, stdev= 0.00 clat (nsec): min=454973k, max=454973k, avg=454972518.00, stdev= 0.00 lat (nsec): min=512784k, max=512784k, avg=512783919.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 456], 5.00th=[ 456], 10.00th=[ 456], 20.00th=[ 456], | 30.00th=[ 456], 40.00th=[ 456], 50.00th=[ 456], 60.00th=[ 456], | 70.00th=[ 456], 80.00th=[ 456], 90.00th=[ 456], 95.00th=[ 456], | 99.00th=[ 456], 99.50th=[ 456], 99.90th=[ 456], 99.95th=[ 456], | 99.99th=[ 456] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=2.47%, ctx=5, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=75.9MiB/s (79.6MB/s), 75.9MiB/s-75.9MiB/s (79.6MB/s-79.6MB/s), io=40.0MiB (41.9MB), run=527-527msec debug=super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck