== sanity test 398c: run fio to test AIO ================= 12:42:29 (1786466549) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.405386 s, 103 MB/s osc.lustre-OST0000-osc-ffff918c16309800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=155423: Tue Aug 11 12:44:45 2026 write: IOPS=20, BW=80.3KiB/s (82.3kB/s)(10.0MiB/127472msec); 0 zone resets slat (usec): min=38, max=24406, avg=391.14, stdev=958.40 clat (msec): min=113, max=1668, avg=795.67, stdev=197.42 lat (msec): min=114, max=1668, avg=796.06, stdev=197.36 clat percentiles (msec): | 1.00th=[ 239], 5.00th=[ 489], 10.00th=[ 592], 20.00th=[ 651], | 30.00th=[ 701], 40.00th=[ 743], 50.00th=[ 793], 60.00th=[ 835], | 70.00th=[ 885], 80.00th=[ 944], 90.00th=[ 1036], 95.00th=[ 1116], | 99.00th=[ 1284], 99.50th=[ 1334], 99.90th=[ 1552], 99.95th=[ 1620], | 99.99th=[ 1670] bw ( KiB/s): min= 15, max= 262, per=24.82%, avg=79.68, stdev=28.80, samples=253 iops : min= 3, max= 65, avg=19.45, stdev= 7.24, samples=253 lat (msec) : 250=1.05%, 500=4.30%, 750=35.27%, 1000=46.68%, 2000=12.70% cpu : usr=0.08%, sys=0.42%, ctx=2630, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=155424: Tue Aug 11 12:44:45 2026 write: IOPS=20, BW=80.4KiB/s (82.3kB/s)(10.0MiB/127431msec); 0 zone resets slat (usec): min=36, max=12316, avg=374.42, stdev=857.02 clat (msec): min=206, max=1694, avg=795.36, stdev=198.77 lat (msec): min=206, max=1695, avg=795.74, stdev=198.80 clat percentiles (msec): | 1.00th=[ 262], 5.00th=[ 460], 10.00th=[ 584], 20.00th=[ 659], | 30.00th=[ 701], 40.00th=[ 743], 50.00th=[ 793], 60.00th=[ 835], | 70.00th=[ 885], 80.00th=[ 936], 90.00th=[ 1028], 95.00th=[ 1116], | 99.00th=[ 1318], 99.50th=[ 1636], 99.90th=[ 1687], 99.95th=[ 1687], | 99.99th=[ 1687] bw ( KiB/s): min= 8, max= 256, per=25.33%, avg=81.31, stdev=33.84, samples=248 iops : min= 2, max= 64, avg=19.85, stdev= 8.48, samples=248 lat (msec) : 250=0.62%, 500=5.23%, 750=35.43%, 1000=46.60%, 2000=12.11% cpu : usr=0.06%, sys=0.45%, ctx=2445, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=155425: Tue Aug 11 12:44:45 2026 write: IOPS=20, BW=80.5KiB/s (82.4kB/s)(10.0MiB/127277msec); 0 zone resets slat (usec): min=37, max=12171, avg=365.11, stdev=857.89 clat (msec): min=153, max=1646, avg=794.77, stdev=198.80 lat (msec): min=153, max=1646, avg=795.13, stdev=198.81 clat percentiles (msec): | 1.00th=[ 215], 5.00th=[ 472], 10.00th=[ 584], 20.00th=[ 651], | 30.00th=[ 709], 40.00th=[ 751], 50.00th=[ 793], 60.00th=[ 835], | 70.00th=[ 885], 80.00th=[ 944], 90.00th=[ 1020], 95.00th=[ 1116], | 99.00th=[ 1318], 99.50th=[ 1368], 99.90th=[ 1536], 99.95th=[ 1536], | 99.99th=[ 1653] bw ( KiB/s): min= 7, max= 256, per=24.80%, avg=79.61, stdev=31.97, samples=253 iops : min= 1, max= 64, avg=19.43, stdev= 8.05, samples=253 lat (msec) : 250=1.48%, 500=4.10%, 750=33.83%, 1000=48.71%, 2000=11.88% cpu : usr=0.06%, sys=0.40%, ctx=2378, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=155426: Tue Aug 11 12:44:45 2026 write: IOPS=20, BW=80.3KiB/s (82.3kB/s)(10.0MiB/127470msec); 0 zone resets slat (usec): min=37, max=9139, avg=325.33, stdev=689.01 clat (msec): min=195, max=1667, avg=795.52, stdev=190.80 lat (msec): min=196, max=1668, avg=795.85, stdev=190.79 clat percentiles (msec): | 1.00th=[ 255], 5.00th=[ 498], 10.00th=[ 592], 20.00th=[ 651], | 30.00th=[ 701], 40.00th=[ 743], 50.00th=[ 793], 60.00th=[ 844], | 70.00th=[ 885], 80.00th=[ 936], 90.00th=[ 1020], 95.00th=[ 1099], | 99.00th=[ 1267], 99.50th=[ 1318], 99.90th=[ 1636], 99.95th=[ 1636], | 99.99th=[ 1670] bw ( KiB/s): min= 7, max= 256, per=24.92%, avg=80.00, stdev=34.26, samples=252 iops : min= 1, max= 64, avg=19.52, stdev= 8.58, samples=252 lat (msec) : 250=0.70%, 500=4.30%, 750=35.94%, 1000=47.27%, 2000=11.80% cpu : usr=0.06%, sys=0.39%, ctx=2269, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=321KiB/s (329kB/s), 80.3KiB/s-80.5KiB/s (82.3kB/s-82.4kB/s), io=40.0MiB (41.9MB), run=127277-127472msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=155454: Tue Aug 11 12:46:20 2026 read: IOPS=14, BW=58.1KiB/s (59.5kB/s)(5048KiB/86832msec) slat (usec): min=2292, max=55564, avg=12119.91, stdev=6371.64 clat (msec): min=26, max=730, avg=335.47, stdev=94.67 lat (msec): min=37, max=750, avg=347.60, stdev=96.29 clat percentiles (msec): | 1.00th=[ 155], 5.00th=[ 205], 10.00th=[ 234], 20.00th=[ 262], | 30.00th=[ 284], 40.00th=[ 305], 50.00th=[ 326], 60.00th=[ 347], | 70.00th=[ 372], 80.00th=[ 405], 90.00th=[ 451], 95.00th=[ 510], | 99.00th=[ 617], 99.50th=[ 667], 99.90th=[ 709], 99.95th=[ 735], | 99.99th=[ 735] bw ( KiB/s): min= 8, max= 127, per=24.99%, avg=57.48, stdev=24.20, samples=172 iops : min= 2, max= 31, avg=13.90, stdev= 6.02, samples=172 write: IOPS=14, BW=59.8KiB/s (61.2kB/s)(5192KiB/86832msec); 0 zone resets slat (usec): min=56, max=10124, avg=394.99, stdev=777.13 clat (msec): min=155, max=1511, avg=729.20, stdev=210.36 lat (msec): min=156, max=1512, avg=729.60, stdev=210.36 clat percentiles (msec): | 1.00th=[ 313], 5.00th=[ 372], 10.00th=[ 456], 20.00th=[ 550], | 30.00th=[ 625], 40.00th=[ 684], 50.00th=[ 726], 60.00th=[ 785], | 70.00th=[ 835], 80.00th=[ 885], 90.00th=[ 978], 95.00th=[ 1070], | 99.00th=[ 1318], 99.50th=[ 1418], 99.90th=[ 1502], 99.95th=[ 1519], | 99.99th=[ 1519] bw ( KiB/s): min= 15, max= 136, per=24.58%, avg=58.99, stdev=22.56, samples=172 iops : min= 3, max= 34, avg=14.27, stdev= 5.63, samples=172 lat (msec) : 50=0.20%, 100=0.20%, 250=7.50%, 500=46.25%, 750=22.66% lat (msec) : 1000=18.95%, 2000=4.26% cpu : usr=0.12%, sys=0.92%, ctx=5005, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=155455: Tue Aug 11 12:46:20 2026 read: IOPS=14, BW=57.2KiB/s (58.5kB/s)(4976KiB/87041msec) slat (usec): min=2891, max=46660, avg=11898.12, stdev=6000.71 clat (msec): min=8, max=703, avg=336.55, stdev=97.12 lat (msec): min=12, max=725, avg=348.45, stdev=98.36 clat percentiles (msec): | 1.00th=[ 92], 5.00th=[ 207], 10.00th=[ 230], 20.00th=[ 262], | 30.00th=[ 284], 40.00th=[ 305], 50.00th=[ 326], 60.00th=[ 347], | 70.00th=[ 372], 80.00th=[ 405], 90.00th=[ 460], 95.00th=[ 506], | 99.00th=[ 634], 99.50th=[ 667], 99.90th=[ 701], 99.95th=[ 701], | 99.99th=[ 701] bw ( KiB/s): min= 8, max= 120, per=24.53%, avg=56.41, stdev=24.02, samples=173 iops : min= 2, max= 30, avg=13.62, stdev= 6.02, samples=173 write: IOPS=15, BW=60.5KiB/s (61.9kB/s)(5264KiB/87041msec); 0 zone resets slat (usec): min=44, max=11545, avg=411.26, stdev=787.34 clat (msec): min=161, max=1523, avg=725.89, stdev=213.53 lat (msec): min=162, max=1523, avg=726.31, stdev=213.55 clat percentiles (msec): | 1.00th=[ 262], 5.00th=[ 372], 10.00th=[ 439], 20.00th=[ 542], | 30.00th=[ 625], 40.00th=[ 684], 50.00th=[ 735], 60.00th=[ 785], | 70.00th=[ 827], 80.00th=[ 885], 90.00th=[ 995], 95.00th=[ 1070], | 99.00th=[ 1284], 99.50th=[ 1418], 99.90th=[ 1469], 99.95th=[ 1519], | 99.99th=[ 1519] bw ( KiB/s): min= 15, max= 128, per=24.93%, avg=59.82, stdev=22.46, samples=173 iops : min= 3, max= 32, avg=14.48, stdev= 5.61, samples=173 lat (msec) : 10=0.04%, 20=0.12%, 50=0.20%, 100=0.20%, 250=7.62% lat (msec) : 500=46.02%, 750=21.48%, 1000=19.49%, 2000=4.84% cpu : usr=0.15%, sys=0.99%, ctx=5192, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=155456: Tue Aug 11 12:46:20 2026 read: IOPS=14, BW=57.3KiB/s (58.6kB/s)(4984KiB/87041msec) slat (usec): min=58, max=49266, avg=12391.42, stdev=6578.72 clat (msec): min=7, max=729, avg=337.74, stdev=97.87 lat (msec): min=16, max=742, avg=350.14, stdev=99.30 clat percentiles (msec): | 1.00th=[ 91], 5.00th=[ 209], 10.00th=[ 234], 20.00th=[ 264], | 30.00th=[ 288], 40.00th=[ 305], 50.00th=[ 326], 60.00th=[ 347], | 70.00th=[ 372], 80.00th=[ 401], 90.00th=[ 456], 95.00th=[ 523], | 99.00th=[ 642], 99.50th=[ 684], 99.90th=[ 709], 99.95th=[ 726], | 99.99th=[ 726] bw ( KiB/s): min= 7, max= 136, per=24.63%, avg=56.66, stdev=25.46, samples=173 iops : min= 1, max= 34, avg=13.69, stdev= 6.36, samples=173 write: IOPS=15, BW=60.4KiB/s (61.8kB/s)(5256KiB/87041msec); 0 zone resets slat (usec): min=48, max=18995, avg=441.76, stdev=1167.12 clat (msec): min=154, max=1526, avg=726.69, stdev=216.28 lat (msec): min=154, max=1527, avg=727.13, stdev=216.28 clat percentiles (msec): | 1.00th=[ 275], 5.00th=[ 372], 10.00th=[ 430], 20.00th=[ 542], | 30.00th=[ 625], 40.00th=[ 684], 50.00th=[ 735], 60.00th=[ 785], | 70.00th=[ 835], 80.00th=[ 894], 90.00th=[ 986], 95.00th=[ 1070], | 99.00th=[ 1284], 99.50th=[ 1469], 99.90th=[ 1519], 99.95th=[ 1536], | 99.99th=[ 1536] bw ( KiB/s): min= 15, max= 127, per=24.92%, avg=59.81, stdev=23.30, samples=172 iops : min= 3, max= 31, avg=14.48, stdev= 5.80, samples=172 lat (msec) : 10=0.04%, 20=0.04%, 50=0.12%, 100=0.31%, 250=7.81% lat (msec) : 500=45.43%, 750=22.58%, 1000=19.30%, 2000=4.38% cpu : usr=0.11%, sys=0.94%, ctx=5046, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=155457: Tue Aug 11 12:46:20 2026 read: IOPS=14, BW=58.4KiB/s (59.8kB/s)(5060KiB/86633msec) slat (usec): min=2216, max=49318, avg=11943.61, stdev=6345.85 clat (msec): min=15, max=684, avg=336.21, stdev=96.08 lat (msec): min=22, max=696, avg=348.16, stdev=97.27 clat percentiles (msec): | 1.00th=[ 80], 5.00th=[ 207], 10.00th=[ 234], 20.00th=[ 264], | 30.00th=[ 288], 40.00th=[ 305], 50.00th=[ 321], 60.00th=[ 347], | 70.00th=[ 372], 80.00th=[ 401], 90.00th=[ 460], 95.00th=[ 527], | 99.00th=[ 617], 99.50th=[ 642], 99.90th=[ 684], 99.95th=[ 684], | 99.99th=[ 684] bw ( KiB/s): min= 8, max= 135, per=25.32%, avg=58.24, stdev=24.71, samples=171 iops : min= 2, max= 33, avg=14.09, stdev= 6.12, samples=171 write: IOPS=14, BW=59.8KiB/s (61.2kB/s)(5180KiB/86633msec); 0 zone resets slat (usec): min=40, max=12817, avg=389.35, stdev=767.00 clat (msec): min=142, max=1535, avg=727.59, stdev=213.24 lat (msec): min=142, max=1536, avg=727.98, stdev=213.26 clat percentiles (msec): | 1.00th=[ 313], 5.00th=[ 372], 10.00th=[ 435], 20.00th=[ 542], | 30.00th=[ 625], 40.00th=[ 684], 50.00th=[ 735], 60.00th=[ 776], | 70.00th=[ 835], 80.00th=[ 885], 90.00th=[ 1003], 95.00th=[ 1099], | 99.00th=[ 1250], 99.50th=[ 1318], 99.90th=[ 1502], 99.95th=[ 1536], | 99.99th=[ 1536] bw ( KiB/s): min= 7, max= 143, per=24.57%, avg=58.98, stdev=24.44, samples=172 iops : min= 1, max= 35, avg=14.27, stdev= 6.11, samples=172 lat (msec) : 20=0.04%, 50=0.20%, 100=0.31%, 250=6.91%, 500=46.60% lat (msec) : 750=23.01%, 1000=18.01%, 2000=4.92% cpu : usr=0.12%, sys=0.96%, ctx=5117, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=231KiB/s (236kB/s), 57.2KiB/s-58.4KiB/s (58.5kB/s-59.8kB/s), io=19.6MiB (20.5MB), run=86633-87041msec WRITE: bw=240KiB/s (246kB/s), 59.8KiB/s-60.5KiB/s (61.2kB/s-61.9kB/s), io=20.4MiB (21.4MB), run=86633-87041msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=155473: Tue Aug 11 12:46:27 2026 read: IOPS=2, BW=110MiB/s (115MB/s)(40.0MiB/365msec) slat (nsec): min=21099k, max=21099k, avg=21099311.00, stdev= 0.00 clat (nsec): min=331236k, max=331236k, avg=331235687.00, stdev= 0.00 lat (nsec): min=352339k, max=352339k, avg=352339179.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 330], 5.00th=[ 330], 10.00th=[ 330], 20.00th=[ 330], | 30.00th=[ 330], 40.00th=[ 330], 50.00th=[ 330], 60.00th=[ 330], | 70.00th=[ 330], 80.00th=[ 330], 90.00th=[ 330], 95.00th=[ 330], | 99.00th=[ 330], 99.50th=[ 330], 99.90th=[ 330], 99.95th=[ 330], | 99.99th=[ 330] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=4.12%, ctx=31, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=110MiB/s (115MB/s), 110MiB/s-110MiB/s (115MB/s-115MB/s), io=40.0MiB (41.9MB), run=365-365msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout