== sanity test 398c: run fio to test AIO ================= 00:46:50 (1788842810) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.48804 s, 85.9 MB/s osc.lustre-OST0000-osc-ffff9de7e0398800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=146318: Tue Sep 8 00:49:40 2026 write: IOPS=15, BW=63.2KiB/s (64.7kB/s)(10.0MiB/161961msec); 0 zone resets slat (usec): min=27, max=24462, avg=258.47, stdev=1078.87 clat (msec): min=312, max=2506, avg=1010.97, stdev=296.01 lat (msec): min=313, max=2506, avg=1011.24, stdev=296.04 clat percentiles (msec): | 1.00th=[ 409], 5.00th=[ 558], 10.00th=[ 676], 20.00th=[ 793], | 30.00th=[ 852], 40.00th=[ 919], 50.00th=[ 969], 60.00th=[ 1036], | 70.00th=[ 1116], 80.00th=[ 1234], 90.00th=[ 1401], 95.00th=[ 1552], | 99.00th=[ 1871], 99.50th=[ 1921], 99.90th=[ 2165], 99.95th=[ 2500], | 99.99th=[ 2500] bw ( KiB/s): min= 7, max= 152, per=24.99%, avg=62.97, stdev=28.31, samples=320 iops : min= 1, max= 38, avg=15.30, stdev= 7.09, samples=320 lat (msec) : 500=3.71%, 750=11.95%, 1000=38.36%, 2000=45.62%, >=2000=0.35% cpu : usr=0.05%, sys=0.24%, ctx=2316, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=146319: Tue Sep 8 00:49:40 2026 write: IOPS=15, BW=63.2KiB/s (64.7kB/s)(10.0MiB/162007msec); 0 zone resets slat (usec): min=23, max=29426, avg=258.86, stdev=1290.07 clat (msec): min=156, max=2481, avg=1011.52, stdev=298.57 lat (msec): min=156, max=2481, avg=1011.79, stdev=298.64 clat percentiles (msec): | 1.00th=[ 393], 5.00th=[ 558], 10.00th=[ 676], 20.00th=[ 785], | 30.00th=[ 852], 40.00th=[ 902], 50.00th=[ 961], 60.00th=[ 1045], | 70.00th=[ 1133], 80.00th=[ 1250], 90.00th=[ 1401], 95.00th=[ 1552], | 99.00th=[ 1838], 99.50th=[ 1921], 99.90th=[ 2140], 99.95th=[ 2232], | 99.99th=[ 2467] bw ( KiB/s): min= 7, max= 165, per=24.90%, avg=62.76, stdev=29.61, samples=321 iops : min= 1, max= 41, avg=15.26, stdev= 7.42, samples=321 lat (msec) : 250=0.16%, 500=3.59%, 750=11.33%, 1000=40.78%, 2000=43.91% lat (msec) : >=2000=0.23% cpu : usr=0.05%, sys=0.22%, ctx=1968, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=146320: Tue Sep 8 00:49:40 2026 write: IOPS=15, BW=63.4KiB/s (64.9kB/s)(10.0MiB/161490msec); 0 zone resets slat (usec): min=28, max=13324, avg=258.09, stdev=934.81 clat (msec): min=106, max=2482, avg=1008.60, stdev=302.78 lat (msec): min=111, max=2482, avg=1008.86, stdev=302.78 clat percentiles (msec): | 1.00th=[ 334], 5.00th=[ 542], 10.00th=[ 676], 20.00th=[ 785], | 30.00th=[ 852], 40.00th=[ 902], 50.00th=[ 961], 60.00th=[ 1045], | 70.00th=[ 1133], 80.00th=[ 1234], 90.00th=[ 1385], 95.00th=[ 1569], | 99.00th=[ 1854], 99.50th=[ 1905], 99.90th=[ 2400], 99.95th=[ 2400], | 99.99th=[ 2467] bw ( KiB/s): min= 7, max= 255, per=25.14%, avg=63.36, stdev=32.15, samples=318 iops : min= 1, max= 63, avg=15.40, stdev= 8.04, samples=318 lat (msec) : 250=0.62%, 500=2.30%, 750=13.20%, 1000=38.91%, 2000=44.73% lat (msec) : >=2000=0.23% cpu : usr=0.05%, sys=0.24%, ctx=2079, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=146321: Tue Sep 8 00:49:40 2026 write: IOPS=15, BW=63.2KiB/s (64.7kB/s)(10.0MiB/162004msec); 0 zone resets slat (usec): min=29, max=30398, avg=258.53, stdev=1237.35 clat (msec): min=306, max=2360, avg=1011.33, stdev=299.46 lat (msec): min=306, max=2360, avg=1011.59, stdev=299.54 clat percentiles (msec): | 1.00th=[ 401], 5.00th=[ 550], 10.00th=[ 684], 20.00th=[ 785], | 30.00th=[ 852], 40.00th=[ 911], 50.00th=[ 969], 60.00th=[ 1045], | 70.00th=[ 1133], 80.00th=[ 1250], 90.00th=[ 1401], 95.00th=[ 1536], | 99.00th=[ 1871], 99.50th=[ 1938], 99.90th=[ 2366], 99.95th=[ 2366], | 99.99th=[ 2366] bw ( KiB/s): min= 7, max= 176, per=25.23%, avg=63.57, stdev=28.05, samples=317 iops : min= 1, max= 44, avg=15.45, stdev= 7.00, samples=317 lat (msec) : 500=3.67%, 750=12.46%, 1000=38.95%, 2000=44.49%, >=2000=0.43% cpu : usr=0.05%, sys=0.24%, ctx=2160, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=253KiB/s (259kB/s), 63.2KiB/s-63.4KiB/s (64.7kB/s-64.9kB/s), io=40.0MiB (41.9MB), run=161490-162007msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=146354: Tue Sep 8 00:51:46 2026 read: IOPS=10, BW=43.9KiB/s (44.0kB/s)(5048KiB/114892msec) slat (usec): min=2670, max=64165, avg=14415.24, stdev=8603.58 clat (msec): min=7, max=1250, avg=443.38, stdev=144.68 lat (msec): min=14, max=1280, avg=457.80, stdev=146.51 clat percentiles (msec): | 1.00th=[ 218], 5.00th=[ 271], 10.00th=[ 296], 20.00th=[ 338], | 30.00th=[ 363], 40.00th=[ 393], 50.00th=[ 414], 60.00th=[ 443], | 70.00th=[ 485], 80.00th=[ 531], 90.00th=[ 625], 95.00th=[ 718], | 99.00th=[ 995], 99.50th=[ 1036], 99.90th=[ 1234], 99.95th=[ 1250], | 99.99th=[ 1250] bw ( KiB/s): min= 7, max= 119, per=25.61%, avg=44.57, stdev=21.25, samples=222 iops : min= 1, max= 29, avg=10.68, stdev= 5.32, samples=222 write: IOPS=11, BW=45.2KiB/s (46.3kB/s)(5192KiB/114892msec); 0 zone resets slat (usec): min=29, max=6820, avg=214.40, stdev=492.13 clat (msec): min=230, max=2687, avg=968.10, stdev=306.62 lat (msec): min=230, max=2687, avg=968.32, stdev=306.65 clat percentiles (msec): | 1.00th=[ 464], 5.00th=[ 575], 10.00th=[ 625], 20.00th=[ 709], | 30.00th=[ 785], 40.00th=[ 869], 50.00th=[ 936], 60.00th=[ 1003], | 70.00th=[ 1083], 80.00th=[ 1183], 90.00th=[ 1351], 95.00th=[ 1519], | 99.00th=[ 1921], 99.50th=[ 2198], 99.90th=[ 2635], 99.95th=[ 2702], | 99.99th=[ 2702] bw ( KiB/s): min= 7, max= 104, per=24.87%, avg=45.01, stdev=19.33, samples=226 iops : min= 1, max= 26, avg=10.79, stdev= 4.87, samples=226 lat (msec) : 10=0.04%, 50=0.04%, 100=0.08%, 250=1.48%, 500=35.55% lat (msec) : 750=23.01%, 1000=19.06%, 2000=20.35%, >=2000=0.39% cpu : usr=0.10%, sys=0.55%, ctx=4502, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=146355: Tue Sep 8 00:51:46 2026 read: IOPS=10, BW=43.3KiB/s (44.3kB/s)(4976KiB/115021msec) slat (usec): min=2718, max=73084, avg=14432.98, stdev=9232.70 clat (msec): min=24, max=1174, avg=435.10, stdev=143.72 lat (msec): min=33, max=1184, avg=449.54, stdev=146.14 clat percentiles (msec): | 1.00th=[ 190], 5.00th=[ 264], 10.00th=[ 296], 20.00th=[ 326], | 30.00th=[ 355], 40.00th=[ 384], 50.00th=[ 405], 60.00th=[ 439], | 70.00th=[ 472], 80.00th=[ 527], 90.00th=[ 625], 95.00th=[ 709], | 99.00th=[ 902], 99.50th=[ 995], 99.90th=[ 1099], 99.95th=[ 1167], | 99.99th=[ 1167] bw ( KiB/s): min= 7, max= 127, per=24.75%, avg=43.07, stdev=22.98, samples=226 iops : min= 1, max= 31, avg=10.31, stdev= 5.79, samples=226 write: IOPS=11, BW=45.8KiB/s (46.9kB/s)(5264KiB/115021msec); 0 zone resets slat (usec): min=27, max=12660, avg=232.82, stdev=754.56 clat (msec): min=206, max=2792, avg=970.79, stdev=311.83 lat (msec): min=206, max=2793, avg=971.02, stdev=311.88 clat percentiles (msec): | 1.00th=[ 443], 5.00th=[ 558], 10.00th=[ 617], 20.00th=[ 709], | 30.00th=[ 793], 40.00th=[ 877], 50.00th=[ 936], 60.00th=[ 1011], | 70.00th=[ 1083], 80.00th=[ 1200], 90.00th=[ 1351], 95.00th=[ 1536], | 99.00th=[ 1972], 99.50th=[ 2366], 99.90th=[ 2702], 99.95th=[ 2802], | 99.99th=[ 2802] bw ( KiB/s): min= 7, max= 104, per=25.25%, avg=45.71, stdev=20.73, samples=226 iops : min= 1, max= 26, avg=10.96, stdev= 5.20, samples=226 lat (msec) : 50=0.16%, 100=0.16%, 250=1.48%, 500=36.02%, 750=21.02% lat (msec) : 1000=20.20%, 2000=20.47%, >=2000=0.51% cpu : usr=0.08%, sys=0.58%, ctx=4278, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=146356: Tue Sep 8 00:51:46 2026 read: IOPS=10, BW=43.4KiB/s (44.5kB/s)(4984KiB/114815msec) slat (usec): min=53, max=58111, avg=14075.06, stdev=8300.28 clat (msec): min=11, max=1274, avg=441.22, stdev=146.53 lat (msec): min=19, max=1292, avg=455.30, stdev=148.24 clat percentiles (msec): | 1.00th=[ 115], 5.00th=[ 264], 10.00th=[ 292], 20.00th=[ 330], | 30.00th=[ 363], 40.00th=[ 393], 50.00th=[ 418], 60.00th=[ 447], | 70.00th=[ 485], 80.00th=[ 542], 90.00th=[ 634], 95.00th=[ 718], | 99.00th=[ 936], 99.50th=[ 995], 99.90th=[ 1183], 99.95th=[ 1267], | 99.99th=[ 1267] bw ( KiB/s): min= 7, max= 128, per=25.13%, avg=43.73, stdev=22.00, samples=224 iops : min= 1, max= 32, avg=10.47, stdev= 5.52, samples=224 write: IOPS=11, BW=45.8KiB/s (46.9kB/s)(5256KiB/114815msec); 0 zone resets slat (usec): min=29, max=14182, avg=240.29, stdev=786.53 clat (msec): min=198, max=2450, avg=965.35, stdev=307.55 lat (msec): min=198, max=2450, avg=965.59, stdev=307.52 clat percentiles (msec): | 1.00th=[ 418], 5.00th=[ 558], 10.00th=[ 617], 20.00th=[ 709], | 30.00th=[ 785], 40.00th=[ 860], 50.00th=[ 919], 60.00th=[ 1003], | 70.00th=[ 1083], 80.00th=[ 1200], 90.00th=[ 1368], 95.00th=[ 1485], | 99.00th=[ 1989], 99.50th=[ 2123], 99.90th=[ 2433], 99.95th=[ 2467], | 99.99th=[ 2467] bw ( KiB/s): min= 7, max= 112, per=24.98%, avg=45.22, stdev=21.61, samples=227 iops : min= 1, max= 28, avg=10.85, stdev= 5.45, samples=227 lat (msec) : 20=0.12%, 50=0.16%, 100=0.20%, 250=1.37%, 500=35.20% lat (msec) : 750=22.66%, 1000=19.65%, 2000=20.23%, >=2000=0.43% cpu : usr=0.08%, sys=0.56%, ctx=4125, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=146357: Tue Sep 8 00:51:46 2026 read: IOPS=11, BW=44.1KiB/s (45.2kB/s)(5060KiB/114755msec) slat (usec): min=2452, max=62806, avg=14381.94, stdev=8785.91 clat (msec): min=21, max=1120, avg=445.49, stdev=143.46 lat (msec): min=33, max=1133, avg=459.87, stdev=145.38 clat percentiles (msec): | 1.00th=[ 226], 5.00th=[ 271], 10.00th=[ 300], 20.00th=[ 330], | 30.00th=[ 363], 40.00th=[ 388], 50.00th=[ 418], 60.00th=[ 451], | 70.00th=[ 489], 80.00th=[ 542], 90.00th=[ 634], 95.00th=[ 726], | 99.00th=[ 927], 99.50th=[ 995], 99.90th=[ 1070], 99.95th=[ 1116], | 99.99th=[ 1116] bw ( KiB/s): min= 7, max= 96, per=25.11%, avg=43.70, stdev=20.43, samples=227 iops : min= 1, max= 24, avg=10.46, stdev= 5.12, samples=227 write: IOPS=11, BW=45.1KiB/s (46.2kB/s)(5180KiB/114755msec); 0 zone resets slat (usec): min=33, max=14170, avg=257.05, stdev=923.43 clat (msec): min=262, max=2723, avg=965.72, stdev=305.63 lat (msec): min=262, max=2723, avg=965.98, stdev=305.72 clat percentiles (msec): | 1.00th=[ 426], 5.00th=[ 558], 10.00th=[ 617], 20.00th=[ 718], | 30.00th=[ 802], 40.00th=[ 869], 50.00th=[ 927], 60.00th=[ 1003], | 70.00th=[ 1083], 80.00th=[ 1183], 90.00th=[ 1334], 95.00th=[ 1469], | 99.00th=[ 1938], 99.50th=[ 2366], 99.90th=[ 2702], 99.95th=[ 2735], | 99.99th=[ 2735] bw ( KiB/s): min= 7, max= 96, per=24.69%, avg=44.70, stdev=20.53, samples=227 iops : min= 1, max= 24, avg=10.72, stdev= 5.15, samples=227 lat (msec) : 50=0.12%, 250=1.02%, 500=35.98%, 750=22.23%, 1000=20.20% lat (msec) : 2000=20.00%, >=2000=0.47% cpu : usr=0.10%, sys=0.53%, ctx=4225, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=174KiB/s (179kB/s), 43.3KiB/s-44.1KiB/s (44.3kB/s-45.2kB/s), io=19.6MiB (20.5MB), run=114755-115021msec WRITE: bw=182KiB/s (186kB/s), 45.1KiB/s-45.8KiB/s (46.2kB/s-46.9kB/s), io=20.4MiB (21.4MB), run=114755-115021msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=146377: Tue Sep 8 00:51:52 2026 read: IOPS=2, BW=100MiB/s (105MB/s)(40.0MiB/400msec) slat (nsec): min=18706k, max=18706k, avg=18706114.00, stdev= 0.00 clat (nsec): min=366626k, max=366626k, avg=366625823.00, stdev= 0.00 lat (nsec): min=385337k, max=385337k, avg=385337177.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 368], 5.00th=[ 368], 10.00th=[ 368], 20.00th=[ 368], | 30.00th=[ 368], 40.00th=[ 368], 50.00th=[ 368], 60.00th=[ 368], | 70.00th=[ 368], 80.00th=[ 368], 90.00th=[ 368], 95.00th=[ 368], | 99.00th=[ 368], 99.50th=[ 368], 99.90th=[ 368], 99.95th=[ 368], | 99.99th=[ 368] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=3.01%, ctx=43, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=100MiB/s (105MB/s), 100MiB/s-100MiB/s (105MB/s-105MB/s), io=40.0MiB (41.9MB), run=400-400msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout