== sanity test 398c: run fio to test AIO ================= 03:24:48 (1788765888) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.175909 s, 238 MB/s osc.lustre-OST0000-osc-ffff958004c25800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=188916: Mon Sep 7 03:25:05 2026 write: IOPS=200, BW=801KiB/s (821kB/s)(10.0MiB/12779msec); 0 zone resets slat (usec): min=22, max=5497, avg=130.13, stdev=239.58 clat (msec): min=16, max=119, avg=79.14, stdev=12.66 lat (msec): min=17, max=119, avg=79.27, stdev=12.68 clat percentiles (msec): | 1.00th=[ 50], 5.00th=[ 62], 10.00th=[ 65], 20.00th=[ 68], | 30.00th=[ 73], 40.00th=[ 77], 50.00th=[ 80], 60.00th=[ 83], | 70.00th=[ 86], 80.00th=[ 89], 90.00th=[ 96], 95.00th=[ 102], | 99.00th=[ 111], 99.50th=[ 112], 99.90th=[ 117], 99.95th=[ 120], | 99.99th=[ 120] bw ( KiB/s): min= 664, max= 952, per=24.83%, avg=795.76, stdev=68.01, samples=25 iops : min= 166, max= 238, avg=198.80, stdev=17.02, samples=25 lat (msec) : 20=0.20%, 50=0.86%, 100=92.89%, 250=6.05% cpu : usr=0.27%, sys=1.70%, ctx=2174, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=188917: Mon Sep 7 03:25:05 2026 write: IOPS=201, BW=805KiB/s (824kB/s)(10.0MiB/12723msec); 0 zone resets slat (usec): min=24, max=4818, avg=154.40, stdev=249.37 clat (msec): min=8, max=130, avg=79.24, stdev=12.47 lat (msec): min=9, max=130, avg=79.40, stdev=12.46 clat percentiles (msec): | 1.00th=[ 48], 5.00th=[ 61], 10.00th=[ 63], 20.00th=[ 70], | 30.00th=[ 73], 40.00th=[ 77], 50.00th=[ 79], 60.00th=[ 82], | 70.00th=[ 86], 80.00th=[ 90], 90.00th=[ 96], 95.00th=[ 101], | 99.00th=[ 110], 99.50th=[ 116], 99.90th=[ 131], 99.95th=[ 131], | 99.99th=[ 131] bw ( KiB/s): min= 664, max= 1022, per=25.00%, avg=801.24, stdev=78.74, samples=25 iops : min= 166, max= 255, avg=200.16, stdev=19.63, samples=25 lat (msec) : 10=0.04%, 50=1.21%, 100=94.02%, 250=4.73% cpu : usr=0.39%, sys=1.97%, ctx=2973, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=188918: Mon Sep 7 03:25:05 2026 write: IOPS=200, BW=803KiB/s (822kB/s)(10.0MiB/12752msec); 0 zone resets slat (usec): min=30, max=3787, avg=155.13, stdev=212.38 clat (msec): min=7, max=119, avg=79.03, stdev=12.53 lat (msec): min=8, max=119, avg=79.18, stdev=12.53 clat percentiles (msec): | 1.00th=[ 48], 5.00th=[ 61], 10.00th=[ 64], 20.00th=[ 69], | 30.00th=[ 73], 40.00th=[ 77], 50.00th=[ 79], 60.00th=[ 82], | 70.00th=[ 86], 80.00th=[ 90], 90.00th=[ 95], 95.00th=[ 102], | 99.00th=[ 109], 99.50th=[ 112], 99.90th=[ 116], 99.95th=[ 117], | 99.99th=[ 120] bw ( KiB/s): min= 664, max= 928, per=24.93%, avg=798.92, stdev=65.50, samples=25 iops : min= 166, max= 232, avg=199.60, stdev=16.39, samples=25 lat (msec) : 10=0.04%, 20=0.12%, 50=0.90%, 100=93.32%, 250=5.62% cpu : usr=0.27%, sys=2.23%, ctx=2995, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=188919: Mon Sep 7 03:25:05 2026 write: IOPS=200, BW=803KiB/s (822kB/s)(10.0MiB/12753msec); 0 zone resets slat (usec): min=26, max=27247, avg=163.62, stdev=586.99 clat (msec): min=25, max=119, avg=79.06, stdev=12.73 lat (msec): min=25, max=119, avg=79.22, stdev=12.69 clat percentiles (msec): | 1.00th=[ 51], 5.00th=[ 59], 10.00th=[ 64], 20.00th=[ 69], | 30.00th=[ 73], 40.00th=[ 77], 50.00th=[ 79], 60.00th=[ 82], | 70.00th=[ 85], 80.00th=[ 90], 90.00th=[ 96], 95.00th=[ 101], | 99.00th=[ 109], 99.50th=[ 111], 99.90th=[ 112], 99.95th=[ 120], | 99.99th=[ 120] bw ( KiB/s): min= 680, max= 933, per=24.95%, avg=799.60, stdev=63.48, samples=25 iops : min= 170, max= 233, avg=199.76, stdev=15.83, samples=25 lat (msec) : 50=0.94%, 100=92.93%, 250=6.13% cpu : usr=0.31%, sys=2.22%, ctx=3000, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=3205KiB/s (3282kB/s), 801KiB/s-805KiB/s (821kB/s-824kB/s), io=40.0MiB (41.9MB), run=12723-12779msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=188930: Mon Sep 7 03:25:18 2026 read: IOPS=135, BW=543KiB/s (556kB/s)(5048KiB/9298msec) slat (usec): min=1436, max=10139, avg=3684.61, stdev=1194.04 clat (usec): min=10740, max=90259, avg=54850.55, stdev=9803.66 lat (usec): min=13115, max=95860, avg=58538.73, stdev=10111.43 clat percentiles (usec): | 1.00th=[34341], 5.00th=[39060], 10.00th=[42206], 20.00th=[46400], | 30.00th=[49546], 40.00th=[52691], 50.00th=[55313], 60.00th=[57410], | 70.00th=[59507], 80.00th=[62129], 90.00th=[66323], 95.00th=[71828], | 99.00th=[80217], 99.50th=[81265], 99.90th=[88605], 99.95th=[90702], | 99.99th=[90702] bw ( KiB/s): min= 400, max= 712, per=25.07%, avg=541.06, stdev=83.87, samples=18 iops : min= 100, max= 178, avg=135.11, stdev=21.10, samples=18 write: IOPS=139, BW=558KiB/s (572kB/s)(5192KiB/9298msec); 0 zone resets slat (usec): min=23, max=1665, avg=134.65, stdev=120.97 clat (usec): min=24677, max=85695, avg=56588.84, stdev=8882.60 lat (usec): min=24820, max=85846, avg=56724.95, stdev=8882.29 clat percentiles (usec): | 1.00th=[36963], 5.00th=[42206], 10.00th=[45876], 20.00th=[49546], | 30.00th=[52167], 40.00th=[53740], 50.00th=[55837], 60.00th=[58459], | 70.00th=[61080], 80.00th=[63701], 90.00th=[67634], 95.00th=[72877], | 99.00th=[79168], 99.50th=[80217], 99.90th=[83362], 99.95th=[85459], | 99.99th=[85459] bw ( KiB/s): min= 447, max= 670, per=24.87%, avg=558.61, stdev=62.87, samples=18 iops : min= 111, max= 167, avg=139.50, stdev=15.79, samples=18 lat (msec) : 20=0.04%, 50=26.91%, 100=73.05% cpu : usr=0.48%, sys=3.80%, ctx=4757, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=188931: Mon Sep 7 03:25:18 2026 read: IOPS=135, BW=540KiB/s (553kB/s)(4976KiB/9210msec) slat (usec): min=27, max=10643, avg=3688.84, stdev=1235.75 clat (msec): min=6, max=107, avg=54.81, stdev=10.79 lat (msec): min=9, max=112, avg=58.50, stdev=11.05 clat percentiles (msec): | 1.00th=[ 33], 5.00th=[ 40], 10.00th=[ 43], 20.00th=[ 46], | 30.00th=[ 50], 40.00th=[ 52], 50.00th=[ 55], 60.00th=[ 57], | 70.00th=[ 60], 80.00th=[ 64], 90.00th=[ 69], 95.00th=[ 73], | 99.00th=[ 85], 99.50th=[ 91], 99.90th=[ 106], 99.95th=[ 108], | 99.99th=[ 108] bw ( KiB/s): min= 432, max= 712, per=25.00%, avg=539.50, stdev=73.59, samples=18 iops : min= 108, max= 178, avg=134.72, stdev=18.50, samples=18 write: IOPS=142, BW=572KiB/s (585kB/s)(5264KiB/9210msec); 0 zone resets slat (usec): min=27, max=2788, avg=138.08, stdev=154.72 clat (usec): min=20043, max=89443, avg=56350.95, stdev=9444.53 lat (usec): min=20112, max=89512, avg=56490.29, stdev=9453.11 clat percentiles (usec): | 1.00th=[35914], 5.00th=[41157], 10.00th=[44827], 20.00th=[49021], | 30.00th=[51643], 40.00th=[53740], 50.00th=[55837], 60.00th=[57934], | 70.00th=[61080], 80.00th=[64226], 90.00th=[68682], 95.00th=[72877], | 99.00th=[79168], 99.50th=[81265], 99.90th=[89654], 99.95th=[89654], | 99.99th=[89654] bw ( KiB/s): min= 456, max= 728, per=25.54%, avg=573.61, stdev=73.18, samples=18 iops : min= 114, max= 182, avg=143.28, stdev=18.32, samples=18 lat (msec) : 10=0.04%, 20=0.08%, 50=28.98%, 100=70.78%, 250=0.12% cpu : usr=0.33%, sys=3.94%, ctx=4743, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=188932: Mon Sep 7 03:25:18 2026 read: IOPS=134, BW=536KiB/s (549kB/s)(4984KiB/9293msec) slat (usec): min=40, max=9254, avg=3706.01, stdev=1211.68 clat (usec): min=7787, max=98869, avg=54912.35, stdev=10433.12 lat (msec): min=9, max=103, avg=58.62, stdev=10.76 clat percentiles (usec): | 1.00th=[33817], 5.00th=[39584], 10.00th=[42206], 20.00th=[45876], | 30.00th=[49021], 40.00th=[51643], 50.00th=[54789], 60.00th=[57410], | 70.00th=[60031], 80.00th=[62653], 90.00th=[67634], 95.00th=[72877], | 99.00th=[81265], 99.50th=[86508], 99.90th=[98042], 99.95th=[99091], | 99.99th=[99091] bw ( KiB/s): min= 423, max= 696, per=24.93%, avg=538.00, stdev=79.76, samples=18 iops : min= 105, max= 174, avg=134.33, stdev=20.11, samples=18 write: IOPS=141, BW=566KiB/s (579kB/s)(5256KiB/9293msec); 0 zone resets slat (usec): min=30, max=2281, avg=133.62, stdev=145.62 clat (usec): min=28521, max=85802, avg=56767.68, stdev=8873.71 lat (usec): min=28646, max=86001, avg=56902.71, stdev=8877.78 clat percentiles (usec): | 1.00th=[35914], 5.00th=[41681], 10.00th=[45876], 20.00th=[49546], | 30.00th=[52167], 40.00th=[54264], 50.00th=[56361], 60.00th=[58459], | 70.00th=[61604], 80.00th=[64750], 90.00th=[67634], 95.00th=[71828], | 99.00th=[79168], 99.50th=[81265], 99.90th=[85459], 99.95th=[85459], | 99.99th=[85459] bw ( KiB/s): min= 408, max= 768, per=25.07%, avg=563.00, stdev=89.73, samples=18 iops : min= 102, max= 192, avg=140.61, stdev=22.48, samples=18 lat (msec) : 10=0.12%, 50=26.95%, 100=72.93% cpu : usr=0.48%, sys=3.80%, ctx=4743, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=188933: Mon Sep 7 03:25:18 2026 read: IOPS=136, BW=546KiB/s (559kB/s)(5060KiB/9268msec) slat (usec): min=1251, max=10266, avg=3665.39, stdev=1227.00 clat (usec): min=9077, max=95763, avg=54674.70, stdev=10104.22 lat (usec): min=14312, max=99690, avg=58345.61, stdev=10409.78 clat percentiles (usec): | 1.00th=[35390], 5.00th=[39060], 10.00th=[42206], 20.00th=[45876], | 30.00th=[49021], 40.00th=[51643], 50.00th=[54264], 60.00th=[56886], | 70.00th=[59507], 80.00th=[62653], 90.00th=[67634], 95.00th=[71828], | 99.00th=[81265], 99.50th=[85459], 99.90th=[89654], 99.95th=[95945], | 99.99th=[95945] bw ( KiB/s): min= 440, max= 704, per=25.35%, avg=547.11, stdev=77.59, samples=18 iops : min= 110, max= 176, avg=136.61, stdev=19.52, samples=18 write: IOPS=139, BW=559KiB/s (572kB/s)(5180KiB/9268msec); 0 zone resets slat (usec): min=37, max=2026, avg=130.64, stdev=118.39 clat (usec): min=34904, max=89501, avg=56760.12, stdev=9109.09 lat (usec): min=35020, max=89561, avg=56892.30, stdev=9114.28 clat percentiles (usec): | 1.00th=[37487], 5.00th=[42206], 10.00th=[45351], 20.00th=[49021], | 30.00th=[52167], 40.00th=[54264], 50.00th=[56361], 60.00th=[58459], | 70.00th=[60556], 80.00th=[63701], 90.00th=[67634], 95.00th=[71828], | 99.00th=[83362], 99.50th=[85459], 99.90th=[89654], 99.95th=[89654], | 99.99th=[89654] bw ( KiB/s): min= 486, max= 696, per=24.85%, avg=558.06, stdev=62.67, samples=18 iops : min= 121, max= 174, avg=139.39, stdev=15.75, samples=18 lat (msec) : 10=0.04%, 50=27.66%, 100=72.30% cpu : usr=0.62%, sys=3.72%, ctx=4756, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2158KiB/s (2210kB/s), 536KiB/s-546KiB/s (549kB/s-559kB/s), io=19.6MiB (20.5MB), run=9210-9298msec WRITE: bw=2247KiB/s (2301kB/s), 558KiB/s-572KiB/s (572kB/s-585kB/s), io=20.4MiB (21.4MB), run=9210-9298msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=188938: Mon Sep 7 03:25:22 2026 read: IOPS=6, BW=274MiB/s (287MB/s)(40.0MiB/146msec) slat (nsec): min=12767k, max=12767k, avg=12766927.00, stdev= 0.00 clat (nsec): min=125966k, max=125966k, avg=125965515.00, stdev= 0.00 lat (nsec): min=138736k, max=138736k, avg=138736272.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 127], 5.00th=[ 127], 10.00th=[ 127], 20.00th=[ 127], | 30.00th=[ 127], 40.00th=[ 127], 50.00th=[ 127], 60.00th=[ 127], | 70.00th=[ 127], 80.00th=[ 127], 90.00th=[ 127], 95.00th=[ 127], | 99.00th=[ 127], 99.50th=[ 127], 99.90th=[ 127], 99.95th=[ 127], | 99.99th=[ 127] lat (msec) : 250=100.00% cpu : usr=0.00%, sys=8.28%, ctx=11, majf=0, minf=9 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=274MiB/s (287MB/s), 274MiB/s-274MiB/s (287MB/s-287MB/s), io=40.0MiB (41.9MB), run=146-146msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout