== sanity test 398c: run fio to test AIO ================= 08:57:45 (1786107465) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.754595 s, 55.6 MB/s osc.lustre-OST0000-osc-ffff8bfc0472a800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=145836: Fri Aug 7 09:00:37 2026 write: IOPS=15, BW=63.8KiB/s (65.4kB/s)(10.0MiB/160385msec); 0 zone resets slat (usec): min=39, max=22065, avg=322.32, stdev=935.47 clat (msec): min=157, max=2378, avg=1000.65, stdev=266.67 lat (msec): min=158, max=2378, avg=1000.97, stdev=266.67 clat percentiles (msec): | 1.00th=[ 292], 5.00th=[ 634], 10.00th=[ 709], 20.00th=[ 802], | 30.00th=[ 869], 40.00th=[ 927], 50.00th=[ 978], 60.00th=[ 1036], | 70.00th=[ 1099], 80.00th=[ 1183], 90.00th=[ 1334], 95.00th=[ 1469], | 99.00th=[ 1770], 99.50th=[ 2165], 99.90th=[ 2366], 99.95th=[ 2366], | 99.99th=[ 2366] bw ( KiB/s): min= 7, max= 159, per=24.89%, avg=63.48, stdev=27.13, samples=317 iops : min= 1, max= 39, avg=15.36, stdev= 6.79, samples=317 lat (msec) : 250=0.47%, 500=1.60%, 750=11.56%, 1000=39.65%, 2000=46.21% lat (msec) : >=2000=0.51% cpu : usr=0.05%, sys=0.33%, ctx=2152, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145837: Fri Aug 7 09:00:37 2026 write: IOPS=15, BW=63.9KiB/s (65.4kB/s)(10.0MiB/160251msec); 0 zone resets slat (usec): min=37, max=19451, avg=377.02, stdev=1177.96 clat (msec): min=148, max=2379, avg=1000.64, stdev=264.33 lat (msec): min=154, max=2379, avg=1001.02, stdev=264.29 clat percentiles (msec): | 1.00th=[ 326], 5.00th=[ 642], 10.00th=[ 709], 20.00th=[ 802], | 30.00th=[ 869], 40.00th=[ 919], 50.00th=[ 969], 60.00th=[ 1036], | 70.00th=[ 1116], 80.00th=[ 1200], 90.00th=[ 1351], 95.00th=[ 1469], | 99.00th=[ 1703], 99.50th=[ 1838], 99.90th=[ 2333], 99.95th=[ 2366], | 99.99th=[ 2366] bw ( KiB/s): min= 7, max= 167, per=24.81%, avg=63.27, stdev=27.59, samples=318 iops : min= 1, max= 41, avg=15.31, stdev= 6.91, samples=318 lat (msec) : 250=0.66%, 500=1.64%, 750=10.66%, 1000=42.66%, 2000=44.14% lat (msec) : >=2000=0.23% cpu : usr=0.06%, sys=0.36%, ctx=2363, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145838: Fri Aug 7 09:00:37 2026 write: IOPS=15, BW=63.9KiB/s (65.4kB/s)(10.0MiB/160320msec); 0 zone resets slat (usec): min=38, max=22859, avg=326.42, stdev=1067.26 clat (msec): min=159, max=2269, avg=1000.37, stdev=258.89 lat (msec): min=160, max=2269, avg=1000.70, stdev=258.86 clat percentiles (msec): | 1.00th=[ 321], 5.00th=[ 642], 10.00th=[ 709], 20.00th=[ 810], | 30.00th=[ 869], 40.00th=[ 927], 50.00th=[ 978], 60.00th=[ 1036], | 70.00th=[ 1116], 80.00th=[ 1200], 90.00th=[ 1318], 95.00th=[ 1435], | 99.00th=[ 1770], 99.50th=[ 1905], 99.90th=[ 2232], 99.95th=[ 2232], | 99.99th=[ 2265] bw ( KiB/s): min= 7, max= 128, per=25.14%, avg=64.10, stdev=29.34, samples=314 iops : min= 1, max= 32, avg=15.51, stdev= 7.34, samples=314 lat (msec) : 250=0.35%, 500=2.07%, 750=11.05%, 1000=39.77%, 2000=46.29% lat (msec) : >=2000=0.47% cpu : usr=0.05%, sys=0.35%, ctx=1993, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145839: Fri Aug 7 09:00:37 2026 write: IOPS=15, BW=63.8KiB/s (65.3kB/s)(10.0MiB/160470msec); 0 zone resets slat (usec): min=37, max=10995, avg=298.36, stdev=726.19 clat (msec): min=159, max=2375, avg=1001.11, stdev=268.80 lat (msec): min=160, max=2375, avg=1001.41, stdev=268.81 clat percentiles (msec): | 1.00th=[ 326], 5.00th=[ 634], 10.00th=[ 701], 20.00th=[ 793], | 30.00th=[ 869], 40.00th=[ 936], 50.00th=[ 986], 60.00th=[ 1036], | 70.00th=[ 1099], 80.00th=[ 1200], 90.00th=[ 1334], 95.00th=[ 1435], | 99.00th=[ 1787], 99.50th=[ 2165], 99.90th=[ 2333], 99.95th=[ 2333], | 99.99th=[ 2366] bw ( KiB/s): min= 7, max= 175, per=24.88%, avg=63.44, stdev=27.58, samples=317 iops : min= 1, max= 43, avg=15.34, stdev= 6.91, samples=317 lat (msec) : 250=0.86%, 500=0.98%, 750=13.52%, 1000=38.52%, 2000=45.59% lat (msec) : >=2000=0.55% cpu : usr=0.05%, sys=0.32%, ctx=2062, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=255KiB/s (261kB/s), 63.8KiB/s-63.9KiB/s (65.3kB/s-65.4kB/s), io=40.0MiB (41.9MB), run=160251-160470msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=145872: Fri Aug 7 09:02:39 2026 read: IOPS=11, BW=45.3KiB/s (46.4kB/s)(5048KiB/111492msec) slat (usec): min=83, max=57410, avg=16048.63, stdev=8753.09 clat (msec): min=17, max=1518, avg=426.78, stdev=139.66 lat (msec): min=36, max=1540, avg=442.83, stdev=141.46 clat percentiles (msec): | 1.00th=[ 136], 5.00th=[ 259], 10.00th=[ 288], 20.00th=[ 321], | 30.00th=[ 351], 40.00th=[ 380], 50.00th=[ 405], 60.00th=[ 430], | 70.00th=[ 472], 80.00th=[ 527], 90.00th=[ 600], 95.00th=[ 676], | 99.00th=[ 827], 99.50th=[ 885], 99.90th=[ 1452], 99.95th=[ 1519], | 99.99th=[ 1519] bw ( KiB/s): min= 7, max= 118, per=25.36%, avg=45.13, stdev=21.60, samples=219 iops : min= 1, max= 29, avg=10.75, stdev= 5.45, samples=219 write: IOPS=11, BW=46.6KiB/s (47.7kB/s)(5192KiB/111492msec); 0 zone resets slat (usec): min=40, max=17501, avg=496.84, stdev=1378.40 clat (msec): min=235, max=2298, avg=942.03, stdev=268.16 lat (msec): min=235, max=2298, avg=942.53, stdev=268.14 clat percentiles (msec): | 1.00th=[ 447], 5.00th=[ 584], 10.00th=[ 642], 20.00th=[ 726], | 30.00th=[ 793], 40.00th=[ 852], 50.00th=[ 911], 60.00th=[ 978], | 70.00th=[ 1053], 80.00th=[ 1150], 90.00th=[ 1284], 95.00th=[ 1401], | 99.00th=[ 1838], 99.50th=[ 2005], 99.90th=[ 2165], 99.95th=[ 2299], | 99.99th=[ 2299] bw ( KiB/s): min= 7, max= 104, per=24.87%, avg=46.26, stdev=18.54, samples=219 iops : min= 1, max= 26, avg=11.04, stdev= 4.65, samples=219 lat (msec) : 20=0.04%, 50=0.20%, 100=0.20%, 250=1.68%, 500=36.45% lat (msec) : 750=22.23%, 1000=20.31%, 2000=18.63%, >=2000=0.27% cpu : usr=0.11%, sys=0.85%, ctx=4633, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145873: Fri Aug 7 09:02:39 2026 read: IOPS=11, BW=44.3KiB/s (45.4kB/s)(4976KiB/112312msec) slat (usec): min=2885, max=99966, avg=16203.28, stdev=8950.77 clat (msec): min=24, max=1004, avg=429.30, stdev=130.75 lat (msec): min=33, max=1052, avg=445.51, stdev=133.21 clat percentiles (msec): | 1.00th=[ 131], 5.00th=[ 257], 10.00th=[ 288], 20.00th=[ 330], | 30.00th=[ 359], 40.00th=[ 384], 50.00th=[ 409], 60.00th=[ 439], | 70.00th=[ 481], 80.00th=[ 531], 90.00th=[ 609], 95.00th=[ 667], | 99.00th=[ 793], 99.50th=[ 844], 99.90th=[ 961], 99.95th=[ 1003], | 99.99th=[ 1003] bw ( KiB/s): min= 7, max= 120, per=24.68%, avg=43.92, stdev=20.73, samples=221 iops : min= 1, max= 30, avg=10.45, stdev= 5.20, samples=221 write: IOPS=11, BW=46.9KiB/s (47.0kB/s)(5264KiB/112312msec); 0 zone resets slat (usec): min=43, max=30550, avg=520.34, stdev=1741.21 clat (msec): min=283, max=2293, avg=940.78, stdev=248.49 lat (msec): min=283, max=2293, avg=941.30, stdev=248.56 clat percentiles (msec): | 1.00th=[ 468], 5.00th=[ 575], 10.00th=[ 642], 20.00th=[ 726], | 30.00th=[ 793], 40.00th=[ 860], 50.00th=[ 919], 60.00th=[ 995], | 70.00th=[ 1053], 80.00th=[ 1133], 90.00th=[ 1250], 95.00th=[ 1368], | 99.00th=[ 1603], 99.50th=[ 1737], 99.90th=[ 2232], 99.95th=[ 2299], | 99.99th=[ 2299] bw ( KiB/s): min= 8, max= 110, per=25.10%, avg=46.68, stdev=18.97, samples=221 iops : min= 2, max= 27, avg=11.14, stdev= 4.76, samples=221 lat (msec) : 50=0.16%, 100=0.20%, 250=1.76%, 500=35.16%, 750=22.42% lat (msec) : 1000=20.59%, 2000=19.53%, >=2000=0.20% cpu : usr=0.09%, sys=0.83%, ctx=4571, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145874: Fri Aug 7 09:02:39 2026 read: IOPS=11, BW=44.4KiB/s (45.5kB/s)(4984KiB/112266msec) slat (usec): min=3185, max=68702, avg=16511.82, stdev=8821.84 clat (msec): min=24, max=1005, avg=429.00, stdev=127.89 lat (msec): min=37, max=1018, avg=445.52, stdev=129.62 clat percentiles (msec): | 1.00th=[ 180], 5.00th=[ 264], 10.00th=[ 292], 20.00th=[ 334], | 30.00th=[ 363], 40.00th=[ 384], 50.00th=[ 409], 60.00th=[ 435], | 70.00th=[ 472], 80.00th=[ 518], 90.00th=[ 600], 95.00th=[ 684], | 99.00th=[ 818], 99.50th=[ 860], 99.90th=[ 961], 99.95th=[ 1003], | 99.99th=[ 1003] bw ( KiB/s): min= 7, max= 134, per=24.83%, avg=44.19, stdev=20.60, samples=221 iops : min= 1, max= 33, avg=10.52, stdev= 5.18, samples=221 write: IOPS=11, BW=46.8KiB/s (47.9kB/s)(5256KiB/112266msec); 0 zone resets slat (usec): min=59, max=23049, avg=431.46, stdev=1063.88 clat (msec): min=361, max=2294, avg=941.04, stdev=258.64 lat (msec): min=361, max=2294, avg=941.48, stdev=258.62 clat percentiles (msec): | 1.00th=[ 468], 5.00th=[ 575], 10.00th=[ 642], 20.00th=[ 726], | 30.00th=[ 785], 40.00th=[ 852], 50.00th=[ 911], 60.00th=[ 978], | 70.00th=[ 1053], 80.00th=[ 1133], 90.00th=[ 1284], 95.00th=[ 1385], | 99.00th=[ 1687], 99.50th=[ 2022], 99.90th=[ 2198], 99.95th=[ 2299], | 99.99th=[ 2299] bw ( KiB/s): min= 7, max= 104, per=24.95%, avg=46.40, stdev=18.93, samples=221 iops : min= 1, max= 26, avg=11.08, stdev= 4.75, samples=221 lat (msec) : 50=0.08%, 100=0.31%, 250=1.33%, 500=36.21%, 750=21.99% lat (msec) : 1000=21.05%, 2000=18.71%, >=2000=0.31% cpu : usr=0.12%, sys=0.82%, ctx=4600, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145875: Fri Aug 7 09:02:39 2026 read: IOPS=11, BW=45.5KiB/s (46.6kB/s)(5060KiB/111091msec) slat (usec): min=3111, max=64375, avg=16518.79, stdev=9026.32 clat (msec): min=17, max=1331, avg=428.96, stdev=134.67 lat (msec): min=22, max=1350, avg=445.50, stdev=136.84 clat percentiles (msec): | 1.00th=[ 174], 5.00th=[ 257], 10.00th=[ 288], 20.00th=[ 330], | 30.00th=[ 355], 40.00th=[ 380], 50.00th=[ 405], 60.00th=[ 439], | 70.00th=[ 481], 80.00th=[ 523], 90.00th=[ 600], 95.00th=[ 667], | 99.00th=[ 844], 99.50th=[ 902], 99.90th=[ 1284], 99.95th=[ 1334], | 99.99th=[ 1334] bw ( KiB/s): min= 7, max= 135, per=25.78%, avg=45.88, stdev=21.03, samples=216 iops : min= 1, max= 33, avg=10.93, stdev= 5.26, samples=216 write: IOPS=11, BW=46.6KiB/s (47.7kB/s)(5180KiB/111091msec); 0 zone resets slat (usec): min=44, max=13344, avg=490.13, stdev=1217.06 clat (msec): min=355, max=2288, avg=933.74, stdev=261.10 lat (msec): min=355, max=2289, avg=934.23, stdev=261.08 clat percentiles (msec): | 1.00th=[ 464], 5.00th=[ 558], 10.00th=[ 625], 20.00th=[ 718], | 30.00th=[ 785], 40.00th=[ 844], 50.00th=[ 894], 60.00th=[ 961], | 70.00th=[ 1036], 80.00th=[ 1133], 90.00th=[ 1267], 95.00th=[ 1418], | 99.00th=[ 1687], 99.50th=[ 1787], 99.90th=[ 2165], 99.95th=[ 2299], | 99.99th=[ 2299] bw ( KiB/s): min= 7, max= 95, per=24.94%, avg=46.39, stdev=19.46, samples=218 iops : min= 1, max= 23, avg=11.07, stdev= 4.87, samples=218 lat (msec) : 20=0.04%, 50=0.04%, 100=0.12%, 250=2.03%, 500=35.78% lat (msec) : 750=22.50%, 1000=21.80%, 2000=17.58%, >=2000=0.12% cpu : usr=0.12%, sys=0.84%, ctx=4478, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=179KiB/s (183kB/s), 44.3KiB/s-45.5KiB/s (45.4kB/s-46.6kB/s), io=19.6MiB (20.5MB), run=111091-112312msec WRITE: bw=186KiB/s (190kB/s), 46.6KiB/s-46.9KiB/s (47.7kB/s-47.0kB/s), io=20.4MiB (21.4MB), run=111091-112312msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=145908: Fri Aug 7 09:02:53 2026 read: IOPS=1, BW=68.6MiB/s (71.9MB/s)(40.0MiB/583msec) slat (nsec): min=45518k, max=45518k, avg=45518190.00, stdev= 0.00 clat (nsec): min=494346k, max=494346k, avg=494346250.00, stdev= 0.00 lat (nsec): min=539869k, max=539869k, avg=539868864.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 493], 5.00th=[ 493], 10.00th=[ 493], 20.00th=[ 493], | 30.00th=[ 493], 40.00th=[ 493], 50.00th=[ 493], 60.00th=[ 493], | 70.00th=[ 493], 80.00th=[ 493], 90.00th=[ 493], 95.00th=[ 493], | 99.00th=[ 493], 99.50th=[ 493], 99.90th=[ 493], 99.95th=[ 493], | 99.99th=[ 493] lat (msec) : 500=100.00% cpu : usr=0.17%, sys=2.41%, ctx=13, majf=0, minf=9 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=68.6MiB/s (71.9MB/s), 68.6MiB/s-68.6MiB/s (71.9MB/s-71.9MB/s), io=40.0MiB (41.9MB), run=583-583msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout