== sanity test 398c: run fio to test AIO ================= 05:57:28 (1787824648) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.600276 s, 69.9 MB/s osc.lustre-OST0000-osc-ffff9521466b4000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=179787: Thu Aug 27 05:58:06 2026 write: IOPS=97, BW=389KiB/s (398kB/s)(10.0MiB/26316msec); 0 zone resets slat (usec): min=22, max=15760, avg=175.15, stdev=719.25 clat (msec): min=28, max=359, avg=163.52, stdev=42.79 lat (msec): min=28, max=359, avg=163.69, stdev=42.80 clat percentiles (msec): | 1.00th=[ 71], 5.00th=[ 107], 10.00th=[ 118], 20.00th=[ 130], | 30.00th=[ 140], 40.00th=[ 148], 50.00th=[ 159], 60.00th=[ 169], | 70.00th=[ 184], 80.00th=[ 199], 90.00th=[ 220], 95.00th=[ 241], | 99.00th=[ 284], 99.50th=[ 284], 99.90th=[ 296], 99.95th=[ 359], | 99.99th=[ 359] bw ( KiB/s): min= 239, max= 632, per=25.02%, avg=387.29, stdev=84.41, samples=52 iops : min= 59, max= 158, avg=96.42, stdev=21.17, samples=52 lat (msec) : 50=0.70%, 100=3.28%, 250=91.76%, 500=4.26% cpu : usr=0.15%, sys=0.93%, ctx=2197, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=179788: Thu Aug 27 05:58:06 2026 write: IOPS=96, BW=387KiB/s (397kB/s)(10.0MiB/26443msec); 0 zone resets slat (usec): min=25, max=15560, avg=190.78, stdev=644.48 clat (msec): min=28, max=291, avg=164.29, stdev=40.92 lat (msec): min=28, max=291, avg=164.49, stdev=40.93 clat percentiles (msec): | 1.00th=[ 71], 5.00th=[ 107], 10.00th=[ 118], 20.00th=[ 132], | 30.00th=[ 142], 40.00th=[ 153], 50.00th=[ 161], 60.00th=[ 171], | 70.00th=[ 184], 80.00th=[ 197], 90.00th=[ 218], 95.00th=[ 241], | 99.00th=[ 275], 99.50th=[ 279], 99.90th=[ 292], 99.95th=[ 292], | 99.99th=[ 292] bw ( KiB/s): min= 254, max= 600, per=24.95%, avg=386.21, stdev=76.85, samples=52 iops : min= 63, max= 150, avg=96.15, stdev=19.23, samples=52 lat (msec) : 50=0.16%, 100=4.14%, 250=92.42%, 500=3.28% cpu : usr=0.23%, sys=0.98%, ctx=2546, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=179789: Thu Aug 27 05:58:06 2026 write: IOPS=97, BW=389KiB/s (399kB/s)(10.0MiB/26298msec); 0 zone resets slat (usec): min=23, max=29414, avg=176.70, stdev=845.01 clat (msec): min=31, max=305, avg=164.00, stdev=42.26 lat (msec): min=31, max=305, avg=164.17, stdev=42.29 clat percentiles (msec): | 1.00th=[ 69], 5.00th=[ 105], 10.00th=[ 116], 20.00th=[ 130], | 30.00th=[ 140], 40.00th=[ 148], 50.00th=[ 161], 60.00th=[ 174], | 70.00th=[ 184], 80.00th=[ 199], 90.00th=[ 218], 95.00th=[ 241], | 99.00th=[ 288], 99.50th=[ 288], 99.90th=[ 305], 99.95th=[ 305], | 99.99th=[ 305] bw ( KiB/s): min= 253, max= 677, per=25.00%, avg=387.04, stdev=91.62, samples=52 iops : min= 63, max= 169, avg=96.33, stdev=22.93, samples=52 lat (msec) : 50=0.08%, 100=4.10%, 250=92.42%, 500=3.40% cpu : usr=0.21%, sys=0.92%, ctx=1893, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=179790: Thu Aug 27 05:58:06 2026 write: IOPS=96, BW=387KiB/s (396kB/s)(10.0MiB/26446msec); 0 zone resets slat (usec): min=22, max=15741, avg=221.57, stdev=843.38 clat (msec): min=17, max=308, avg=164.40, stdev=41.87 lat (msec): min=24, max=308, avg=164.63, stdev=41.86 clat percentiles (msec): | 1.00th=[ 67], 5.00th=[ 104], 10.00th=[ 116], 20.00th=[ 131], | 30.00th=[ 142], 40.00th=[ 153], 50.00th=[ 163], 60.00th=[ 171], | 70.00th=[ 184], 80.00th=[ 199], 90.00th=[ 220], 95.00th=[ 236], | 99.00th=[ 275], 99.50th=[ 275], 99.90th=[ 309], 99.95th=[ 309], | 99.99th=[ 309] bw ( KiB/s): min= 262, max= 632, per=24.92%, avg=385.69, stdev=75.08, samples=52 iops : min= 65, max= 158, avg=96.02, stdev=18.81, samples=52 lat (msec) : 20=0.04%, 50=0.27%, 100=4.18%, 250=91.99%, 500=3.52% cpu : usr=0.20%, sys=1.12%, ctx=2605, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1549KiB/s (1586kB/s), 387KiB/s-389KiB/s (396kB/s-399kB/s), io=40.0MiB (41.9MB), run=26298-26446msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=179804: Thu Aug 27 05:58:43 2026 read: IOPS=44, BW=178KiB/s (183kB/s)(5048KiB/28308msec) slat (usec): min=34, max=47086, avg=12735.93, stdev=7127.52 clat (msec): min=16, max=412, avg=166.87, stdev=42.69 lat (msec): min=39, max=425, avg=179.61, stdev=44.57 clat percentiles (msec): | 1.00th=[ 86], 5.00th=[ 105], 10.00th=[ 114], 20.00th=[ 131], | 30.00th=[ 144], 40.00th=[ 155], 50.00th=[ 163], 60.00th=[ 174], | 70.00th=[ 186], 80.00th=[ 203], 90.00th=[ 224], 95.00th=[ 239], | 99.00th=[ 279], 99.50th=[ 296], 99.90th=[ 330], 99.95th=[ 414], | 99.99th=[ 414] bw ( KiB/s): min= 87, max= 279, per=24.97%, avg=176.82, stdev=43.30, samples=56 iops : min= 21, max= 69, avg=43.71, stdev=10.89, samples=56 write: IOPS=45, BW=183KiB/s (188kB/s)(5192KiB/28308msec); 0 zone resets slat (usec): min=26, max=11569, avg=229.03, stdev=705.18 clat (msec): min=15, max=400, avg=171.98, stdev=43.92 lat (msec): min=15, max=401, avg=172.21, stdev=43.97 clat percentiles (msec): | 1.00th=[ 93], 5.00th=[ 110], 10.00th=[ 123], 20.00th=[ 138], | 30.00th=[ 148], 40.00th=[ 159], 50.00th=[ 167], 60.00th=[ 178], | 70.00th=[ 188], 80.00th=[ 203], 90.00th=[ 230], 95.00th=[ 255], | 99.00th=[ 305], 99.50th=[ 317], 99.90th=[ 359], 99.95th=[ 401], | 99.99th=[ 401] bw ( KiB/s): min= 95, max= 303, per=24.71%, avg=182.36, stdev=42.90, samples=56 iops : min= 23, max= 75, avg=45.09, stdev=10.68, samples=56 lat (msec) : 20=0.08%, 50=0.20%, 100=2.85%, 250=92.23%, 500=4.65% cpu : usr=0.28%, sys=2.14%, ctx=4320, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=179805: Thu Aug 27 05:58:43 2026 read: IOPS=43, BW=176KiB/s (180kB/s)(4976KiB/28284msec) slat (usec): min=35, max=71339, avg=12564.99, stdev=7320.85 clat (msec): min=67, max=331, avg=166.73, stdev=42.32 lat (msec): min=72, max=357, avg=179.31, stdev=44.22 clat percentiles (msec): | 1.00th=[ 92], 5.00th=[ 106], 10.00th=[ 114], 20.00th=[ 131], | 30.00th=[ 142], 40.00th=[ 153], 50.00th=[ 163], 60.00th=[ 174], | 70.00th=[ 186], 80.00th=[ 201], 90.00th=[ 222], 95.00th=[ 239], | 99.00th=[ 296], 99.50th=[ 313], 99.90th=[ 330], 99.95th=[ 334], | 99.99th=[ 334] bw ( KiB/s): min= 45, max= 295, per=24.73%, avg=175.11, stdev=46.93, samples=56 iops : min= 11, max= 73, avg=43.27, stdev=11.71, samples=56 write: IOPS=46, BW=186KiB/s (191kB/s)(5264KiB/28284msec); 0 zone resets slat (usec): min=29, max=17262, avg=262.75, stdev=898.77 clat (msec): min=81, max=322, avg=171.80, stdev=42.19 lat (msec): min=81, max=322, avg=172.07, stdev=42.21 clat percentiles (msec): | 1.00th=[ 95], 5.00th=[ 109], 10.00th=[ 118], 20.00th=[ 136], | 30.00th=[ 148], 40.00th=[ 159], 50.00th=[ 167], 60.00th=[ 180], | 70.00th=[ 192], 80.00th=[ 207], 90.00th=[ 228], 95.00th=[ 251], | 99.00th=[ 284], 99.50th=[ 292], 99.90th=[ 300], 99.95th=[ 321], | 99.99th=[ 321] bw ( KiB/s): min= 106, max= 295, per=25.16%, avg=185.70, stdev=45.80, samples=56 iops : min= 26, max= 73, avg=45.93, stdev=11.46, samples=56 lat (msec) : 100=2.30%, 250=93.12%, 500=4.57% cpu : usr=0.34%, sys=2.11%, ctx=4449, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=179806: Thu Aug 27 05:58:43 2026 read: IOPS=44, BW=177KiB/s (182kB/s)(4984KiB/28119msec) slat (usec): min=33, max=57738, avg=12695.32, stdev=6971.68 clat (msec): min=28, max=332, avg=168.37, stdev=43.08 lat (msec): min=31, max=359, avg=181.08, stdev=44.75 clat percentiles (msec): | 1.00th=[ 84], 5.00th=[ 105], 10.00th=[ 116], 20.00th=[ 131], | 30.00th=[ 144], 40.00th=[ 155], 50.00th=[ 167], 60.00th=[ 178], | 70.00th=[ 190], 80.00th=[ 205], 90.00th=[ 226], 95.00th=[ 241], | 99.00th=[ 279], 99.50th=[ 288], 99.90th=[ 326], 99.95th=[ 334], | 99.99th=[ 334] bw ( KiB/s): min= 95, max= 295, per=24.96%, avg=176.71, stdev=40.49, samples=55 iops : min= 23, max= 73, avg=43.67, stdev=10.12, samples=55 write: IOPS=46, BW=187KiB/s (191kB/s)(5256KiB/28119msec); 0 zone resets slat (usec): min=24, max=8966, avg=208.40, stdev=530.32 clat (msec): min=18, max=327, avg=169.97, stdev=41.53 lat (msec): min=18, max=327, avg=170.18, stdev=41.53 clat percentiles (msec): | 1.00th=[ 92], 5.00th=[ 106], 10.00th=[ 120], 20.00th=[ 136], | 30.00th=[ 146], 40.00th=[ 157], 50.00th=[ 167], 60.00th=[ 178], | 70.00th=[ 190], 80.00th=[ 205], 90.00th=[ 226], 95.00th=[ 243], | 99.00th=[ 279], 99.50th=[ 296], 99.90th=[ 326], 99.95th=[ 330], | 99.99th=[ 330] bw ( KiB/s): min= 79, max= 295, per=25.08%, avg=185.13, stdev=48.84, samples=55 iops : min= 19, max= 73, avg=45.80, stdev=12.22, samples=55 lat (msec) : 20=0.04%, 50=0.20%, 100=3.24%, 250=92.93%, 500=3.59% cpu : usr=0.30%, sys=2.09%, ctx=4107, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=179807: Thu Aug 27 05:58:43 2026 read: IOPS=44, BW=179KiB/s (183kB/s)(5060KiB/28288msec) slat (usec): min=49, max=56127, avg=12722.31, stdev=7355.59 clat (msec): min=32, max=372, avg=166.77, stdev=44.63 lat (msec): min=50, max=381, avg=179.50, stdev=46.37 clat percentiles (msec): | 1.00th=[ 87], 5.00th=[ 103], 10.00th=[ 115], 20.00th=[ 129], | 30.00th=[ 142], 40.00th=[ 155], 50.00th=[ 163], 60.00th=[ 174], | 70.00th=[ 186], 80.00th=[ 201], 90.00th=[ 222], 95.00th=[ 245], | 99.00th=[ 305], 99.50th=[ 330], 99.90th=[ 372], 99.95th=[ 372], | 99.99th=[ 372] bw ( KiB/s): min= 32, max= 311, per=25.16%, avg=178.12, stdev=46.07, samples=56 iops : min= 8, max= 77, avg=44.04, stdev=11.49, samples=56 write: IOPS=45, BW=183KiB/s (188kB/s)(5180KiB/28288msec); 0 zone resets slat (usec): min=22, max=22951, avg=266.16, stdev=953.69 clat (msec): min=88, max=323, avg=172.03, stdev=41.52 lat (msec): min=89, max=323, avg=172.30, stdev=41.53 clat percentiles (msec): | 1.00th=[ 99], 5.00th=[ 111], 10.00th=[ 121], 20.00th=[ 138], | 30.00th=[ 148], 40.00th=[ 159], 50.00th=[ 169], 60.00th=[ 180], | 70.00th=[ 190], 80.00th=[ 205], 90.00th=[ 228], 95.00th=[ 251], | 99.00th=[ 292], 99.50th=[ 313], 99.90th=[ 326], 99.95th=[ 326], | 99.99th=[ 326] bw ( KiB/s): min= 79, max= 268, per=24.73%, avg=182.52, stdev=40.90, samples=56 iops : min= 19, max= 67, avg=45.14, stdev=10.25, samples=56 lat (msec) : 50=0.08%, 100=2.77%, 250=92.42%, 500=4.73% cpu : usr=0.27%, sys=2.10%, ctx=4211, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=709KiB/s (726kB/s), 176KiB/s-179KiB/s (180kB/s-183kB/s), io=19.6MiB (20.5MB), run=28119-28308msec WRITE: bw=738KiB/s (756kB/s), 183KiB/s-187KiB/s (188kB/s-191kB/s), io=20.4MiB (21.4MB), run=28119-28308msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=179816: Thu Aug 27 05:58:53 2026 read: IOPS=2, BW=82.6MiB/s (86.7MB/s)(40.0MiB/484msec) slat (nsec): min=13208k, max=13208k, avg=13207868.00, stdev= 0.00 clat (nsec): min=454438k, max=454438k, avg=454438113.00, stdev= 0.00 lat (nsec): min=467651k, max=467651k, avg=467650819.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 456], 5.00th=[ 456], 10.00th=[ 456], 20.00th=[ 456], | 30.00th=[ 456], 40.00th=[ 456], 50.00th=[ 456], 60.00th=[ 456], | 70.00th=[ 456], 80.00th=[ 456], 90.00th=[ 456], 95.00th=[ 456], | 99.00th=[ 456], 99.50th=[ 456], 99.90th=[ 456], 99.95th=[ 456], | 99.99th=[ 456] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=2.07%, ctx=4, majf=0, minf=9 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=82.6MiB/s (86.7MB/s), 82.6MiB/s-82.6MiB/s (86.7MB/s-86.7MB/s), io=40.0MiB (41.9MB), run=484-484msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout