== sanity test 398c: run fio to test AIO ================= 01:07:37 (1788844057) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.464743 s, 90.3 MB/s osc.lustre-OST0000-osc-ffff9c33037db000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=145841: Tue Sep 8 01:10:31 2026 write: IOPS=15, BW=62.9KiB/s (64.4kB/s)(10.0MiB/162850msec); 0 zone resets slat (usec): min=28, max=14379, avg=204.03, stdev=735.91 clat (msec): min=74, max=2114, avg=1017.10, stdev=270.48 lat (msec): min=75, max=2115, avg=1017.30, stdev=270.45 clat percentiles (msec): | 1.00th=[ 397], 5.00th=[ 575], 10.00th=[ 684], 20.00th=[ 827], | 30.00th=[ 902], 40.00th=[ 961], 50.00th=[ 1011], 60.00th=[ 1070], | 70.00th=[ 1133], 80.00th=[ 1200], 90.00th=[ 1334], 95.00th=[ 1485], | 99.00th=[ 1787], 99.50th=[ 1854], 99.90th=[ 2106], 99.95th=[ 2106], | 99.99th=[ 2123] bw ( KiB/s): min= 7, max= 153, per=25.15%, avg=62.86, stdev=26.76, samples=320 iops : min= 1, max= 38, avg=15.19, stdev= 6.70, samples=320 lat (msec) : 100=0.08%, 250=0.39%, 500=2.66%, 750=11.48%, 1000=32.70% lat (msec) : 2000=52.46%, >=2000=0.23% cpu : usr=0.04%, sys=0.22%, ctx=1854, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145842: Tue Sep 8 01:10:31 2026 write: IOPS=15, BW=62.7KiB/s (64.2kB/s)(10.0MiB/163265msec); 0 zone resets slat (usec): min=22, max=10236, avg=197.67, stdev=633.07 clat (msec): min=396, max=2178, avg=1019.05, stdev=260.61 lat (msec): min=396, max=2178, avg=1019.25, stdev=260.61 clat percentiles (msec): | 1.00th=[ 460], 5.00th=[ 609], 10.00th=[ 693], 20.00th=[ 818], | 30.00th=[ 894], 40.00th=[ 961], 50.00th=[ 1011], 60.00th=[ 1070], | 70.00th=[ 1133], 80.00th=[ 1217], 90.00th=[ 1334], 95.00th=[ 1469], | 99.00th=[ 1770], 99.50th=[ 1921], 99.90th=[ 2106], 99.95th=[ 2106], | 99.99th=[ 2165] bw ( KiB/s): min= 7, max= 255, per=25.01%, avg=62.52, stdev=29.04, samples=322 iops : min= 1, max= 63, avg=15.11, stdev= 7.25, samples=322 lat (msec) : 500=3.20%, 750=10.59%, 1000=33.55%, 2000=52.38%, >=2000=0.27% cpu : usr=0.04%, sys=0.20%, ctx=1739, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145843: Tue Sep 8 01:10:31 2026 write: IOPS=15, BW=62.8KiB/s (64.3kB/s)(10.0MiB/163091msec); 0 zone resets slat (usec): min=26, max=17501, avg=217.12, stdev=820.80 clat (msec): min=271, max=2152, avg=1017.94, stdev=262.78 lat (msec): min=271, max=2153, avg=1018.16, stdev=262.79 clat percentiles (msec): | 1.00th=[ 460], 5.00th=[ 600], 10.00th=[ 693], 20.00th=[ 810], | 30.00th=[ 894], 40.00th=[ 953], 50.00th=[ 1011], 60.00th=[ 1070], | 70.00th=[ 1133], 80.00th=[ 1200], 90.00th=[ 1334], 95.00th=[ 1469], | 99.00th=[ 1804], 99.50th=[ 1888], 99.90th=[ 2072], 99.95th=[ 2123], | 99.99th=[ 2165] bw ( KiB/s): min= 7, max= 247, per=25.00%, avg=62.50, stdev=30.09, samples=322 iops : min= 1, max= 61, avg=15.11, stdev= 7.52, samples=322 lat (msec) : 500=3.24%, 750=10.86%, 1000=34.22%, 2000=51.37%, >=2000=0.31% cpu : usr=0.05%, sys=0.21%, ctx=1756, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145844: Tue Sep 8 01:10:31 2026 write: IOPS=15, BW=62.8KiB/s (64.3kB/s)(10.0MiB/162984msec); 0 zone resets slat (usec): min=27, max=14951, avg=215.11, stdev=857.54 clat (msec): min=241, max=2178, avg=1017.60, stdev=265.91 lat (msec): min=241, max=2178, avg=1017.81, stdev=265.92 clat percentiles (msec): | 1.00th=[ 443], 5.00th=[ 592], 10.00th=[ 693], 20.00th=[ 810], | 30.00th=[ 894], 40.00th=[ 961], 50.00th=[ 1011], 60.00th=[ 1070], | 70.00th=[ 1116], 80.00th=[ 1217], 90.00th=[ 1334], 95.00th=[ 1469], | 99.00th=[ 1804], 99.50th=[ 1921], 99.90th=[ 2106], 99.95th=[ 2165], | 99.99th=[ 2165] bw ( KiB/s): min= 8, max= 128, per=24.91%, avg=62.28, stdev=26.81, samples=323 iops : min= 2, max= 32, avg=15.05, stdev= 6.70, samples=323 lat (msec) : 250=0.62%, 500=2.58%, 750=11.09%, 1000=33.12%, 2000=52.27% lat (msec) : >=2000=0.31% cpu : usr=0.05%, sys=0.19%, ctx=1805, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=251KiB/s (257kB/s), 62.7KiB/s-62.9KiB/s (64.2kB/s-64.4kB/s), io=40.0MiB (41.9MB), run=162850-163265msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=145878: Tue Sep 8 01:12:27 2026 read: IOPS=11, BW=47.2KiB/s (48.4kB/s)(5048KiB/106867msec) slat (usec): min=2607, max=57987, avg=14725.35, stdev=8273.81 clat (msec): min=39, max=949, avg=427.40, stdev=114.42 lat (msec): min=54, max=964, avg=442.13, stdev=115.99 clat percentiles (msec): | 1.00th=[ 213], 5.00th=[ 271], 10.00th=[ 300], 20.00th=[ 338], | 30.00th=[ 368], 40.00th=[ 388], 50.00th=[ 409], 60.00th=[ 435], | 70.00th=[ 468], 80.00th=[ 506], 90.00th=[ 567], 95.00th=[ 642], | 99.00th=[ 802], 99.50th=[ 852], 99.90th=[ 911], 99.95th=[ 953], | 99.99th=[ 953] bw ( KiB/s): min= 7, max= 110, per=25.04%, avg=46.83, stdev=20.73, samples=211 iops : min= 1, max= 27, avg=11.24, stdev= 5.19, samples=211 write: IOPS=12, BW=48.6KiB/s (49.7kB/s)(5192KiB/106867msec); 0 zone resets slat (usec): min=24, max=15023, avg=247.30, stdev=838.32 clat (msec): min=274, max=1973, avg=884.28, stdev=272.66 lat (msec): min=274, max=1973, avg=884.53, stdev=272.74 clat percentiles (msec): | 1.00th=[ 384], 5.00th=[ 498], 10.00th=[ 567], 20.00th=[ 667], | 30.00th=[ 735], 40.00th=[ 785], 50.00th=[ 844], 60.00th=[ 911], | 70.00th=[ 978], 80.00th=[ 1083], 90.00th=[ 1250], 95.00th=[ 1385], | 99.00th=[ 1737], 99.50th=[ 1787], 99.90th=[ 1905], 99.95th=[ 1972], | 99.99th=[ 1972] bw ( KiB/s): min= 7, max= 110, per=24.66%, avg=48.09, stdev=21.24, samples=212 iops : min= 1, max= 27, avg=11.55, stdev= 5.35, samples=212 lat (msec) : 50=0.04%, 100=0.08%, 250=1.13%, 500=40.16%, 750=24.57% lat (msec) : 1000=19.73%, 2000=14.30% cpu : usr=0.09%, sys=0.53%, ctx=4448, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145879: Tue Sep 8 01:12:27 2026 read: IOPS=11, BW=46.6KiB/s (47.7kB/s)(4976KiB/106782msec) slat (usec): min=2908, max=67698, avg=14331.28, stdev=7868.09 clat (msec): min=8, max=934, avg=424.90, stdev=116.89 lat (msec): min=16, max=968, avg=439.24, stdev=118.46 clat percentiles (msec): | 1.00th=[ 194], 5.00th=[ 271], 10.00th=[ 296], 20.00th=[ 334], | 30.00th=[ 363], 40.00th=[ 388], 50.00th=[ 409], 60.00th=[ 443], | 70.00th=[ 472], 80.00th=[ 506], 90.00th=[ 558], 95.00th=[ 625], | 99.00th=[ 802], 99.50th=[ 844], 99.90th=[ 927], 99.95th=[ 936], | 99.99th=[ 936] bw ( KiB/s): min= 7, max= 104, per=24.55%, avg=45.90, stdev=21.80, samples=212 iops : min= 1, max= 26, avg=11.00, stdev= 5.47, samples=212 write: IOPS=12, BW=49.3KiB/s (50.5kB/s)(5264KiB/106782msec); 0 zone resets slat (usec): min=31, max=25332, avg=292.51, stdev=1271.81 clat (msec): min=248, max=1955, avg=880.39, stdev=280.42 lat (msec): min=248, max=1955, avg=880.69, stdev=280.52 clat percentiles (msec): | 1.00th=[ 326], 5.00th=[ 489], 10.00th=[ 550], 20.00th=[ 659], | 30.00th=[ 718], 40.00th=[ 785], 50.00th=[ 844], 60.00th=[ 911], | 70.00th=[ 978], 80.00th=[ 1083], 90.00th=[ 1267], 95.00th=[ 1435], | 99.00th=[ 1703], 99.50th=[ 1821], 99.90th=[ 1938], 99.95th=[ 1955], | 99.99th=[ 1955] bw ( KiB/s): min= 7, max= 118, per=25.12%, avg=48.98, stdev=22.26, samples=211 iops : min= 1, max= 29, avg=11.78, stdev= 5.59, samples=211 lat (msec) : 10=0.04%, 50=0.16%, 100=0.12%, 250=1.21%, 500=39.69% lat (msec) : 750=23.79%, 1000=21.21%, 2000=13.79% cpu : usr=0.09%, sys=0.51%, ctx=4043, majf=0, minf=16 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145880: Tue Sep 8 01:12:27 2026 read: IOPS=11, BW=46.7KiB/s (47.8kB/s)(4984KiB/106789msec) slat (usec): min=3049, max=59483, avg=14406.70, stdev=8141.31 clat (msec): min=24, max=936, avg=419.02, stdev=117.15 lat (msec): min=37, max=952, avg=433.43, stdev=118.81 clat percentiles (msec): | 1.00th=[ 165], 5.00th=[ 266], 10.00th=[ 292], 20.00th=[ 330], | 30.00th=[ 355], 40.00th=[ 380], 50.00th=[ 405], 60.00th=[ 435], | 70.00th=[ 464], 80.00th=[ 502], 90.00th=[ 558], 95.00th=[ 634], | 99.00th=[ 810], 99.50th=[ 835], 99.90th=[ 936], 99.95th=[ 936], | 99.99th=[ 936] bw ( KiB/s): min= 7, max= 136, per=24.92%, avg=46.60, stdev=23.27, samples=210 iops : min= 1, max= 34, avg=11.18, stdev= 5.85, samples=210 write: IOPS=12, BW=49.2KiB/s (50.4kB/s)(5256KiB/106789msec); 0 zone resets slat (usec): min=30, max=15418, avg=242.22, stdev=914.13 clat (msec): min=281, max=1867, avg=886.65, stdev=272.01 lat (msec): min=281, max=1867, avg=886.89, stdev=272.05 clat percentiles (msec): | 1.00th=[ 401], 5.00th=[ 506], 10.00th=[ 575], 20.00th=[ 667], | 30.00th=[ 735], 40.00th=[ 785], 50.00th=[ 852], 60.00th=[ 919], | 70.00th=[ 986], 80.00th=[ 1099], 90.00th=[ 1267], 95.00th=[ 1401], | 99.00th=[ 1703], 99.50th=[ 1754], 99.90th=[ 1838], 99.95th=[ 1871], | 99.99th=[ 1871] bw ( KiB/s): min= 7, max= 112, per=24.89%, avg=48.53, stdev=20.46, samples=212 iops : min= 1, max= 28, avg=11.66, stdev= 5.14, samples=212 lat (msec) : 50=0.12%, 100=0.27%, 250=1.13%, 500=39.61%, 750=23.87% lat (msec) : 1000=20.51%, 2000=14.49% cpu : usr=0.09%, sys=0.58%, ctx=4545, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145881: Tue Sep 8 01:12:27 2026 read: IOPS=11, BW=47.7KiB/s (48.8kB/s)(5060KiB/106087msec) slat (usec): min=53, max=56408, avg=14245.03, stdev=8364.82 clat (msec): min=20, max=1098, avg=421.39, stdev=124.07 lat (msec): min=21, max=1118, avg=435.64, stdev=125.65 clat percentiles (msec): | 1.00th=[ 66], 5.00th=[ 257], 10.00th=[ 292], 20.00th=[ 334], | 30.00th=[ 363], 40.00th=[ 388], 50.00th=[ 409], 60.00th=[ 435], | 70.00th=[ 468], 80.00th=[ 498], 90.00th=[ 558], 95.00th=[ 634], | 99.00th=[ 835], 99.50th=[ 885], 99.90th=[ 1083], 99.95th=[ 1099], | 99.99th=[ 1099] bw ( KiB/s): min= 7, max= 120, per=25.56%, avg=47.80, stdev=20.96, samples=208 iops : min= 1, max= 30, avg=11.48, stdev= 5.27, samples=208 write: IOPS=12, BW=48.8KiB/s (49.0kB/s)(5180KiB/106087msec); 0 zone resets slat (usec): min=27, max=12011, avg=255.15, stdev=860.94 clat (msec): min=116, max=1938, avg=883.70, stdev=279.64 lat (msec): min=116, max=1944, avg=883.96, stdev=279.68 clat percentiles (msec): | 1.00th=[ 334], 5.00th=[ 510], 10.00th=[ 558], 20.00th=[ 659], | 30.00th=[ 726], 40.00th=[ 785], 50.00th=[ 844], 60.00th=[ 911], | 70.00th=[ 978], 80.00th=[ 1083], 90.00th=[ 1267], 95.00th=[ 1418], | 99.00th=[ 1770], 99.50th=[ 1838], 99.90th=[ 1905], 99.95th=[ 1938], | 99.99th=[ 1938] bw ( KiB/s): min= 7, max= 120, per=24.85%, avg=48.45, stdev=21.70, samples=209 iops : min= 1, max= 30, avg=11.65, stdev= 5.46, samples=209 lat (msec) : 50=0.31%, 100=0.39%, 250=1.76%, 500=39.10%, 750=23.83% lat (msec) : 1000=20.55%, 2000=14.06% cpu : usr=0.10%, sys=0.60%, ctx=4418, majf=0, minf=16 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=188KiB/s (192kB/s), 46.6KiB/s-47.7KiB/s (47.7kB/s-48.8kB/s), io=19.6MiB (20.5MB), run=106087-106867msec WRITE: bw=195KiB/s (200kB/s), 48.6KiB/s-49.3KiB/s (49.7kB/s-50.5kB/s), io=20.4MiB (21.4MB), run=106087-106867msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=145900: Tue Sep 8 01:12:37 2026 read: IOPS=1, BW=69.8MiB/s (73.2MB/s)(40.0MiB/573msec) slat (nsec): min=43205k, max=43205k, avg=43204846.00, stdev= 0.00 clat (nsec): min=504253k, max=504253k, avg=504253314.00, stdev= 0.00 lat (nsec): min=547462k, max=547462k, avg=547462052.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 506], 5.00th=[ 506], 10.00th=[ 506], 20.00th=[ 506], | 30.00th=[ 506], 40.00th=[ 506], 50.00th=[ 506], 60.00th=[ 506], | 70.00th=[ 506], 80.00th=[ 506], 90.00th=[ 506], 95.00th=[ 506], | 99.00th=[ 506], 99.50th=[ 506], 99.90th=[ 506], 99.95th=[ 506], | 99.99th=[ 506] lat (msec) : 750=100.00% cpu : usr=0.17%, sys=2.27%, ctx=42, majf=0, minf=7 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=69.8MiB/s (73.2MB/s), 69.8MiB/s-69.8MiB/s (73.2MB/s-73.2MB/s), io=40.0MiB (41.9MB), run=573-573msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout