== sanity test 398c: run fio to test AIO ================= 01:45:55 (1786945555) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.171736 s, 244 MB/s osc.lustre-OST0000-osc-ffffa01456f31800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=186297: Mon Aug 17 01:46:10 2026 write: IOPS=233, BW=936KiB/s (958kB/s)(10.0MiB/10945msec); 0 zone resets slat (usec): min=41, max=28348, avg=234.77, stdev=596.45 clat (msec): min=11, max=155, avg=67.80, stdev=14.64 lat (msec): min=11, max=155, avg=68.04, stdev=14.62 clat percentiles (msec): | 1.00th=[ 40], 5.00th=[ 47], 10.00th=[ 51], 20.00th=[ 55], | 30.00th=[ 59], 40.00th=[ 64], 50.00th=[ 68], 60.00th=[ 71], | 70.00th=[ 75], 80.00th=[ 81], 90.00th=[ 88], 95.00th=[ 92], | 99.00th=[ 102], 99.50th=[ 114], 99.90th=[ 157], 99.95th=[ 157], | 99.99th=[ 157] bw ( KiB/s): min= 798, max= 1200, per=25.09%, avg=938.95, stdev=119.46, samples=21 iops : min= 199, max= 300, avg=234.52, stdev=29.94, samples=21 lat (msec) : 20=0.23%, 50=8.95%, 100=89.73%, 250=1.09% cpu : usr=0.33%, sys=3.88%, ctx=2732, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=186298: Mon Aug 17 01:46:10 2026 write: IOPS=235, BW=941KiB/s (964kB/s)(10.0MiB/10881msec); 0 zone resets slat (usec): min=46, max=2982, avg=229.07, stdev=207.13 clat (msec): min=7, max=152, avg=67.69, stdev=14.49 lat (msec): min=7, max=152, avg=67.92, stdev=14.50 clat percentiles (msec): | 1.00th=[ 40], 5.00th=[ 48], 10.00th=[ 51], 20.00th=[ 55], | 30.00th=[ 59], 40.00th=[ 64], 50.00th=[ 68], 60.00th=[ 71], | 70.00th=[ 74], 80.00th=[ 81], 90.00th=[ 87], 95.00th=[ 92], | 99.00th=[ 106], 99.50th=[ 112], 99.90th=[ 146], 99.95th=[ 153], | 99.99th=[ 153] bw ( KiB/s): min= 776, max= 1200, per=25.19%, avg=942.52, stdev=131.05, samples=21 iops : min= 194, max= 300, avg=235.43, stdev=32.84, samples=21 lat (msec) : 10=0.04%, 20=0.04%, 50=9.88%, 100=88.52%, 250=1.52% cpu : usr=0.41%, sys=3.68%, ctx=2697, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=186299: Mon Aug 17 01:46:10 2026 write: IOPS=233, BW=936KiB/s (958kB/s)(10.0MiB/10941msec); 0 zone resets slat (usec): min=42, max=3249, avg=227.17, stdev=191.97 clat (msec): min=13, max=151, avg=67.71, stdev=14.51 lat (msec): min=14, max=152, avg=67.94, stdev=14.52 clat percentiles (msec): | 1.00th=[ 37], 5.00th=[ 47], 10.00th=[ 51], 20.00th=[ 55], | 30.00th=[ 59], 40.00th=[ 64], 50.00th=[ 68], 60.00th=[ 71], | 70.00th=[ 75], 80.00th=[ 81], 90.00th=[ 87], 95.00th=[ 92], | 99.00th=[ 106], 99.50th=[ 111], 99.90th=[ 136], 99.95th=[ 153], | 99.99th=[ 153] bw ( KiB/s): min= 782, max= 1176, per=25.02%, avg=936.33, stdev=123.68, samples=21 iops : min= 195, max= 294, avg=233.86, stdev=30.97, samples=21 lat (msec) : 20=0.27%, 50=8.67%, 100=89.38%, 250=1.68% cpu : usr=0.31%, sys=3.75%, ctx=2709, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=186300: Mon Aug 17 01:46:10 2026 write: IOPS=233, BW=936KiB/s (958kB/s)(10.0MiB/10945msec); 0 zone resets slat (usec): min=41, max=2457, avg=228.83, stdev=202.51 clat (msec): min=15, max=132, avg=67.79, stdev=14.03 lat (msec): min=15, max=132, avg=68.02, stdev=14.03 clat percentiles (msec): | 1.00th=[ 43], 5.00th=[ 48], 10.00th=[ 51], 20.00th=[ 55], | 30.00th=[ 59], 40.00th=[ 64], 50.00th=[ 67], 60.00th=[ 71], | 70.00th=[ 75], 80.00th=[ 80], 90.00th=[ 87], 95.00th=[ 92], | 99.00th=[ 104], 99.50th=[ 111], 99.90th=[ 117], 99.95th=[ 133], | 99.99th=[ 133] bw ( KiB/s): min= 776, max= 1192, per=25.03%, avg=936.81, stdev=117.23, samples=21 iops : min= 194, max= 298, avg=234.00, stdev=29.39, samples=21 lat (msec) : 20=0.04%, 50=8.98%, 100=89.69%, 250=1.29% cpu : usr=0.37%, sys=3.86%, ctx=2747, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=3742KiB/s (3832kB/s), 936KiB/s-941KiB/s (958kB/s-964kB/s), io=40.0MiB (41.9MB), run=10881-10945msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=186312: Mon Aug 17 01:46:23 2026 read: IOPS=126, BW=508KiB/s (520kB/s)(5048KiB/9938msec) slat (usec): min=1395, max=12185, avg=4029.54, stdev=1355.15 clat (usec): min=10683, max=97018, avg=58876.43, stdev=10880.76 lat (msec): min=12, max=104, avg=62.91, stdev=11.09 clat percentiles (usec): | 1.00th=[35390], 5.00th=[41681], 10.00th=[45351], 20.00th=[50070], | 30.00th=[53216], 40.00th=[55837], 50.00th=[58459], 60.00th=[61080], | 70.00th=[64226], 80.00th=[68682], 90.00th=[73925], 95.00th=[77071], | 99.00th=[83362], 99.50th=[85459], 99.90th=[89654], 99.95th=[96994], | 99.99th=[96994] bw ( KiB/s): min= 440, max= 624, per=25.18%, avg=508.47, stdev=41.60, samples=19 iops : min= 110, max= 156, avg=127.05, stdev=10.43, samples=19 write: IOPS=130, BW=522KiB/s (535kB/s)(5192KiB/9938msec); 0 zone resets slat (usec): min=49, max=1615, avg=206.43, stdev=145.93 clat (usec): min=20886, max=93326, avg=60428.38, stdev=9770.48 lat (usec): min=21067, max=93474, avg=60636.04, stdev=9767.48 clat percentiles (usec): | 1.00th=[40109], 5.00th=[44827], 10.00th=[48497], 20.00th=[52167], | 30.00th=[55313], 40.00th=[57934], 50.00th=[60031], 60.00th=[62129], | 70.00th=[64750], 80.00th=[68682], 90.00th=[72877], 95.00th=[78119], | 99.00th=[85459], 99.50th=[87557], 99.90th=[91751], 99.95th=[93848], | 99.99th=[93848] bw ( KiB/s): min= 464, max= 648, per=24.87%, avg=522.68, stdev=62.38, samples=19 iops : min= 116, max= 162, avg=130.58, stdev=15.62, samples=19 lat (msec) : 20=0.12%, 50=16.91%, 100=82.97% cpu : usr=0.44%, sys=4.48%, ctx=4443, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=186313: Mon Aug 17 01:46:23 2026 read: IOPS=125, BW=501KiB/s (513kB/s)(4976KiB/9935msec) slat (usec): min=1560, max=10733, avg=4059.55, stdev=1331.05 clat (msec): min=23, max=125, avg=58.71, stdev=10.47 lat (msec): min=29, max=130, avg=62.77, stdev=10.70 clat percentiles (msec): | 1.00th=[ 36], 5.00th=[ 43], 10.00th=[ 47], 20.00th=[ 51], | 30.00th=[ 54], 40.00th=[ 56], 50.00th=[ 58], 60.00th=[ 61], | 70.00th=[ 64], 80.00th=[ 68], 90.00th=[ 73], 95.00th=[ 77], | 99.00th=[ 87], 99.50th=[ 89], 99.90th=[ 110], 99.95th=[ 126], | 99.99th=[ 126] bw ( KiB/s): min= 408, max= 596, per=24.90%, avg=502.74, stdev=54.34, samples=19 iops : min= 102, max= 149, avg=125.63, stdev=13.60, samples=19 write: IOPS=132, BW=530KiB/s (543kB/s)(5264KiB/9935msec); 0 zone resets slat (usec): min=40, max=1463, avg=212.24, stdev=163.07 clat (msec): min=34, max=112, avg=60.61, stdev= 9.71 lat (msec): min=34, max=113, avg=60.82, stdev= 9.71 clat percentiles (msec): | 1.00th=[ 41], 5.00th=[ 47], 10.00th=[ 50], 20.00th=[ 53], | 30.00th=[ 56], 40.00th=[ 58], 50.00th=[ 61], 60.00th=[ 63], | 70.00th=[ 65], 80.00th=[ 69], 90.00th=[ 73], 95.00th=[ 79], | 99.00th=[ 87], 99.50th=[ 90], 99.90th=[ 94], 99.95th=[ 113], | 99.99th=[ 113] bw ( KiB/s): min= 437, max= 776, per=25.28%, avg=531.37, stdev=76.71, samples=19 iops : min= 109, max= 194, avg=132.79, stdev=19.22, samples=19 lat (msec) : 50=15.20%, 100=84.69%, 250=0.12% cpu : usr=0.48%, sys=4.54%, ctx=4404, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=186314: Mon Aug 17 01:46:23 2026 read: IOPS=125, BW=502KiB/s (514kB/s)(4984KiB/9934msec) slat (usec): min=1339, max=10893, avg=4074.17, stdev=1400.32 clat (usec): min=10111, max=93549, avg=58239.61, stdev=10345.19 lat (usec): min=14854, max=98181, avg=62317.26, stdev=10629.48 clat percentiles (usec): | 1.00th=[36439], 5.00th=[42730], 10.00th=[45351], 20.00th=[50070], | 30.00th=[52691], 40.00th=[54789], 50.00th=[56886], 60.00th=[60556], | 70.00th=[63701], 80.00th=[66847], 90.00th=[70779], 95.00th=[76022], | 99.00th=[86508], 99.50th=[88605], 99.90th=[90702], 99.95th=[93848], | 99.99th=[93848] bw ( KiB/s): min= 376, max= 672, per=25.06%, avg=506.05, stdev=68.61, samples=19 iops : min= 94, max= 168, avg=126.47, stdev=17.20, samples=19 write: IOPS=132, BW=529KiB/s (542kB/s)(5256KiB/9934msec); 0 zone resets slat (usec): min=57, max=2476, avg=223.41, stdev=193.62 clat (msec): min=34, max=103, avg=60.85, stdev= 9.51 lat (msec): min=34, max=103, avg=61.08, stdev= 9.51 clat percentiles (msec): | 1.00th=[ 42], 5.00th=[ 46], 10.00th=[ 50], 20.00th=[ 54], | 30.00th=[ 56], 40.00th=[ 58], 50.00th=[ 61], 60.00th=[ 63], | 70.00th=[ 66], 80.00th=[ 69], 90.00th=[ 74], 95.00th=[ 79], | 99.00th=[ 84], 99.50th=[ 90], 99.90th=[ 104], 99.95th=[ 104], | 99.99th=[ 104] bw ( KiB/s): min= 432, max= 640, per=25.13%, avg=528.16, stdev=52.05, samples=19 iops : min= 108, max= 160, avg=132.00, stdev=12.98, samples=19 lat (msec) : 20=0.04%, 50=16.02%, 100=83.87%, 250=0.08% cpu : usr=0.52%, sys=4.52%, ctx=4493, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=186315: Mon Aug 17 01:46:23 2026 read: IOPS=127, BW=512KiB/s (524kB/s)(5060KiB/9885msec) slat (usec): min=83, max=13117, avg=4018.02, stdev=1421.05 clat (msec): min=6, max=115, avg=58.34, stdev=11.39 lat (msec): min=6, max=117, avg=62.37, stdev=11.65 clat percentiles (msec): | 1.00th=[ 34], 5.00th=[ 42], 10.00th=[ 45], 20.00th=[ 50], | 30.00th=[ 53], 40.00th=[ 56], 50.00th=[ 58], 60.00th=[ 61], | 70.00th=[ 64], 80.00th=[ 68], 90.00th=[ 73], 95.00th=[ 79], | 99.00th=[ 86], 99.50th=[ 92], 99.90th=[ 103], 99.95th=[ 115], | 99.99th=[ 115] bw ( KiB/s): min= 424, max= 656, per=25.47%, avg=514.16, stdev=63.68, samples=19 iops : min= 106, max= 164, avg=128.47, stdev=15.94, samples=19 write: IOPS=131, BW=524KiB/s (537kB/s)(5180KiB/9885msec); 0 zone resets slat (usec): min=49, max=4592, avg=217.60, stdev=231.64 clat (msec): min=20, max=140, avg=60.77, stdev=10.44 lat (msec): min=20, max=140, avg=60.99, stdev=10.45 clat percentiles (msec): | 1.00th=[ 40], 5.00th=[ 45], 10.00th=[ 48], 20.00th=[ 53], | 30.00th=[ 56], 40.00th=[ 58], 50.00th=[ 61], 60.00th=[ 63], | 70.00th=[ 66], 80.00th=[ 69], 90.00th=[ 74], 95.00th=[ 79], | 99.00th=[ 87], 99.50th=[ 93], 99.90th=[ 118], 99.95th=[ 140], | 99.99th=[ 140] bw ( KiB/s): min= 432, max= 629, per=24.94%, avg=524.21, stdev=59.82, samples=19 iops : min= 108, max= 157, avg=131.00, stdev=14.94, samples=19 lat (msec) : 10=0.12%, 20=0.12%, 50=17.46%, 100=82.11%, 250=0.20% cpu : usr=0.49%, sys=4.51%, ctx=4513, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2019KiB/s (2068kB/s), 501KiB/s-512KiB/s (513kB/s-524kB/s), io=19.6MiB (20.5MB), run=9885-9938msec WRITE: bw=2102KiB/s (2153kB/s), 522KiB/s-530KiB/s (535kB/s-543kB/s), io=20.4MiB (21.4MB), run=9885-9938msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=186320: Mon Aug 17 01:46:27 2026 read: IOPS=7, BW=308MiB/s (323MB/s)(40.0MiB/130msec) slat (nsec): min=14682k, max=14682k, avg=14682429.00, stdev= 0.00 clat (nsec): min=109517k, max=109517k, avg=109516735.00, stdev= 0.00 lat (nsec): min=124205k, max=124205k, avg=124205334.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 110], 5.00th=[ 110], 10.00th=[ 110], 20.00th=[ 110], | 30.00th=[ 110], 40.00th=[ 110], 50.00th=[ 110], 60.00th=[ 110], | 70.00th=[ 110], 80.00th=[ 110], 90.00th=[ 110], 95.00th=[ 110], | 99.00th=[ 110], 99.50th=[ 110], 99.90th=[ 110], 99.95th=[ 110], | 99.99th=[ 110] lat (msec) : 250=100.00% cpu : usr=0.78%, sys=8.53%, ctx=10, majf=0, minf=9 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=308MiB/s (323MB/s), 308MiB/s-308MiB/s (323MB/s-323MB/s), io=40.0MiB (41.9MB), run=130-130msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout