== sanity test 398c: run fio to test AIO ================= 18:43:28 (1768261408) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.568903 s, 73.7 MB/s osc.lustre-OST0000-osc-ffff99feb1cfe000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=147453: Mon Jan 12 18:46:45 2026 write: IOPS=13, BW=55.4KiB/s (56.7kB/s)(10.0MiB/184841msec); 0 zone resets slat (usec): min=25, max=21245, avg=314.90, stdev=1170.50 clat (msec): min=216, max=3040, avg=1153.86, stdev=414.54 lat (msec): min=216, max=3040, avg=1154.18, stdev=414.60 clat percentiles (msec): | 1.00th=[ 372], 5.00th=[ 584], 10.00th=[ 667], 20.00th=[ 844], | 30.00th=[ 944], 40.00th=[ 1028], 50.00th=[ 1099], 60.00th=[ 1183], | 70.00th=[ 1284], 80.00th=[ 1435], 90.00th=[ 1653], 95.00th=[ 1921], | 99.00th=[ 2467], 99.50th=[ 2802], 99.90th=[ 2869], 99.95th=[ 3037], | 99.99th=[ 3037] bw ( KiB/s): min= 7, max= 128, per=25.81%, avg=57.04, stdev=29.97, samples=352 iops : min= 1, max= 32, avg=13.74, stdev= 7.52, samples=352 lat (msec) : 250=0.62%, 500=1.64%, 750=11.21%, 1000=23.40%, 2000=58.87% lat (msec) : >=2000=4.26% cpu : usr=0.05%, sys=0.25%, ctx=1834, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=147454: Mon Jan 12 18:46:45 2026 write: IOPS=13, BW=55.4KiB/s (56.8kB/s)(10.0MiB/184707msec); 0 zone resets slat (usec): min=24, max=41015, avg=304.75, stdev=1538.74 clat (msec): min=269, max=3059, avg=1152.90, stdev=407.18 lat (msec): min=269, max=3059, avg=1153.20, stdev=407.26 clat percentiles (msec): | 1.00th=[ 414], 5.00th=[ 584], 10.00th=[ 667], 20.00th=[ 835], | 30.00th=[ 944], 40.00th=[ 1036], 50.00th=[ 1099], 60.00th=[ 1183], | 70.00th=[ 1284], 80.00th=[ 1435], 90.00th=[ 1653], 95.00th=[ 1938], | 99.00th=[ 2400], 99.50th=[ 2668], 99.90th=[ 3004], 99.95th=[ 3071], | 99.99th=[ 3071] bw ( KiB/s): min= 7, max= 252, per=25.74%, avg=56.88, stdev=31.54, samples=353 iops : min= 1, max= 63, avg=13.70, stdev= 7.92, samples=353 lat (msec) : 500=2.89%, 750=10.66%, 1000=21.80%, 2000=60.31%, >=2000=4.34% cpu : usr=0.05%, sys=0.26%, ctx=1746, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=147455: Mon Jan 12 18:46:45 2026 write: IOPS=13, BW=55.4KiB/s (56.7kB/s)(10.0MiB/184783msec); 0 zone resets slat (usec): min=28, max=26836, avg=272.78, stdev=1095.85 clat (msec): min=301, max=3118, avg=1153.30, stdev=404.41 lat (msec): min=301, max=3118, avg=1153.57, stdev=404.46 clat percentiles (msec): | 1.00th=[ 460], 5.00th=[ 558], 10.00th=[ 667], 20.00th=[ 844], | 30.00th=[ 953], 40.00th=[ 1028], 50.00th=[ 1116], 60.00th=[ 1183], | 70.00th=[ 1267], 80.00th=[ 1418], 90.00th=[ 1636], 95.00th=[ 1972], | 99.00th=[ 2366], 99.50th=[ 2601], 99.90th=[ 3104], 99.95th=[ 3104], | 99.99th=[ 3104] bw ( KiB/s): min= 7, max= 142, per=25.85%, avg=57.12, stdev=32.36, samples=352 iops : min= 1, max= 35, avg=13.75, stdev= 8.13, samples=352 lat (msec) : 500=2.73%, 750=10.82%, 1000=22.70%, 2000=59.18%, >=2000=4.57% cpu : usr=0.05%, sys=0.25%, ctx=1816, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=147456: Mon Jan 12 18:46:45 2026 write: IOPS=13, BW=55.5KiB/s (56.8kB/s)(10.0MiB/184453msec); 0 zone resets slat (usec): min=23, max=49267, avg=307.39, stdev=1531.61 clat (msec): min=87, max=3115, avg=1151.95, stdev=411.43 lat (msec): min=87, max=3115, avg=1152.26, stdev=411.50 clat percentiles (msec): | 1.00th=[ 384], 5.00th=[ 550], 10.00th=[ 676], 20.00th=[ 844], | 30.00th=[ 944], 40.00th=[ 1028], 50.00th=[ 1099], 60.00th=[ 1183], | 70.00th=[ 1301], 80.00th=[ 1418], 90.00th=[ 1670], 95.00th=[ 1905], | 99.00th=[ 2366], 99.50th=[ 2635], 99.90th=[ 3037], 99.95th=[ 3037], | 99.99th=[ 3104] bw ( KiB/s): min= 7, max= 150, per=25.67%, avg=56.72, stdev=28.79, samples=354 iops : min= 1, max= 37, avg=13.66, stdev= 7.22, samples=354 lat (msec) : 100=0.04%, 250=0.35%, 500=3.16%, 750=10.16%, 1000=22.77% lat (msec) : 2000=59.57%, >=2000=3.95% cpu : usr=0.05%, sys=0.27%, ctx=2103, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=222KiB/s (227kB/s), 55.4KiB/s-55.5KiB/s (56.7kB/s-56.8kB/s), io=40.0MiB (41.9MB), run=184453-184841msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=147501: Mon Jan 12 18:49:02 2026 read: IOPS=9, BW=39.9KiB/s (40.8kB/s)(5048KiB/126614msec) slat (usec): min=3753, max=88232, avg=18205.38, stdev=10793.74 clat (msec): min=27, max=1241, avg=485.46, stdev=157.60 lat (msec): min=40, max=1275, avg=503.68, stdev=159.73 clat percentiles (msec): | 1.00th=[ 97], 5.00th=[ 288], 10.00th=[ 326], 20.00th=[ 363], | 30.00th=[ 401], 40.00th=[ 435], 50.00th=[ 472], 60.00th=[ 506], | 70.00th=[ 542], 80.00th=[ 584], 90.00th=[ 684], 95.00th=[ 743], | 99.00th=[ 1028], 99.50th=[ 1133], 99.90th=[ 1217], 99.95th=[ 1234], | 99.99th=[ 1234] bw ( KiB/s): min= 5, max= 118, per=25.53%, avg=40.09, stdev=21.27, samples=246 iops : min= 1, max= 29, avg= 9.50, stdev= 5.33, samples=246 write: IOPS=10, BW=41.0KiB/s (41.0kB/s)(5192KiB/126614msec); 0 zone resets slat (usec): min=30, max=46980, avg=443.67, stdev=1882.18 clat (msec): min=311, max=1980, avg=1067.40, stdev=288.38 lat (msec): min=311, max=1980, avg=1067.85, stdev=288.35 clat percentiles (msec): | 1.00th=[ 472], 5.00th=[ 642], 10.00th=[ 726], 20.00th=[ 818], | 30.00th=[ 902], 40.00th=[ 978], 50.00th=[ 1045], 60.00th=[ 1099], | 70.00th=[ 1200], 80.00th=[ 1318], 90.00th=[ 1469], 95.00th=[ 1586], | 99.00th=[ 1804], 99.50th=[ 1871], 99.90th=[ 1989], 99.95th=[ 1989], | 99.99th=[ 1989] bw ( KiB/s): min= 7, max= 110, per=25.07%, avg=41.11, stdev=20.24, samples=246 iops : min= 1, max= 27, avg= 9.74, stdev= 5.09, samples=246 lat (msec) : 50=0.16%, 100=0.39%, 250=0.82%, 500=28.75%, 750=23.40% lat (msec) : 1000=17.23%, 2000=29.26% cpu : usr=0.14%, sys=0.79%, ctx=4898, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=147502: Mon Jan 12 18:49:02 2026 read: IOPS=9, BW=39.1KiB/s (40.1kB/s)(4976KiB/127110msec) slat (usec): min=3308, max=93715, avg=18150.14, stdev=10497.32 clat (msec): min=26, max=1295, avg=490.56, stdev=158.67 lat (msec): min=49, max=1318, avg=508.72, stdev=160.76 clat percentiles (msec): | 1.00th=[ 201], 5.00th=[ 292], 10.00th=[ 330], 20.00th=[ 372], | 30.00th=[ 401], 40.00th=[ 439], 50.00th=[ 472], 60.00th=[ 502], | 70.00th=[ 535], 80.00th=[ 592], 90.00th=[ 684], 95.00th=[ 776], | 99.00th=[ 1028], 99.50th=[ 1183], 99.90th=[ 1250], 99.95th=[ 1301], | 99.99th=[ 1301] bw ( KiB/s): min= 5, max= 134, per=24.99%, avg=39.23, stdev=20.53, samples=247 iops : min= 1, max= 33, avg= 9.28, stdev= 5.14, samples=247 write: IOPS=10, BW=41.4KiB/s (42.4kB/s)(5264KiB/127110msec); 0 zone resets slat (usec): min=32, max=15412, avg=305.77, stdev=926.17 clat (msec): min=302, max=2038, avg=1061.69, stdev=300.60 lat (msec): min=302, max=2040, avg=1062.00, stdev=300.59 clat percentiles (msec): | 1.00th=[ 464], 5.00th=[ 600], 10.00th=[ 693], 20.00th=[ 810], | 30.00th=[ 885], 40.00th=[ 969], 50.00th=[ 1036], 60.00th=[ 1116], | 70.00th=[ 1200], 80.00th=[ 1318], 90.00th=[ 1469], 95.00th=[ 1620], | 99.00th=[ 1821], 99.50th=[ 1871], 99.90th=[ 2039], 99.95th=[ 2039], | 99.99th=[ 2039] bw ( KiB/s): min= 7, max= 95, per=25.10%, avg=41.16, stdev=19.78, samples=250 iops : min= 1, max= 23, avg= 9.76, stdev= 4.98, samples=250 lat (msec) : 50=0.04%, 100=0.12%, 250=1.02%, 500=28.52%, 750=23.98% lat (msec) : 1000=16.80%, 2000=29.41%, >=2000=0.12% cpu : usr=0.11%, sys=0.79%, ctx=4681, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=147503: Mon Jan 12 18:49:02 2026 read: IOPS=9, BW=39.2KiB/s (40.2kB/s)(4984KiB/127045msec) slat (usec): min=3374, max=68307, avg=17978.83, stdev=10036.03 clat (msec): min=11, max=1340, avg=490.51, stdev=154.07 lat (msec): min=25, max=1394, avg=508.50, stdev=155.89 clat percentiles (msec): | 1.00th=[ 142], 5.00th=[ 300], 10.00th=[ 338], 20.00th=[ 376], | 30.00th=[ 409], 40.00th=[ 439], 50.00th=[ 468], 60.00th=[ 498], | 70.00th=[ 550], 80.00th=[ 600], 90.00th=[ 684], 95.00th=[ 760], | 99.00th=[ 1011], 99.50th=[ 1133], 99.90th=[ 1250], 99.95th=[ 1334], | 99.99th=[ 1334] bw ( KiB/s): min= 7, max= 118, per=24.92%, avg=39.12, stdev=20.67, samples=249 iops : min= 1, max= 29, avg= 9.25, stdev= 5.20, samples=249 write: IOPS=10, BW=41.4KiB/s (42.4kB/s)(5256KiB/127045msec); 0 zone resets slat (usec): min=36, max=17958, avg=440.41, stdev=1388.91 clat (msec): min=293, max=1988, avg=1061.79, stdev=299.77 lat (msec): min=293, max=1989, avg=1062.23, stdev=299.77 clat percentiles (msec): | 1.00th=[ 401], 5.00th=[ 617], 10.00th=[ 709], 20.00th=[ 810], | 30.00th=[ 885], 40.00th=[ 961], 50.00th=[ 1028], 60.00th=[ 1116], | 70.00th=[ 1200], 80.00th=[ 1318], 90.00th=[ 1485], 95.00th=[ 1586], | 99.00th=[ 1838], 99.50th=[ 1888], 99.90th=[ 1972], 99.95th=[ 1989], | 99.99th=[ 1989] bw ( KiB/s): min= 7, max= 104, per=24.73%, avg=40.55, stdev=19.56, samples=252 iops : min= 1, max= 26, avg= 9.61, stdev= 4.92, samples=252 lat (msec) : 20=0.04%, 50=0.08%, 100=0.27%, 250=0.78%, 500=29.30% lat (msec) : 750=21.99%, 1000=19.30%, 2000=28.24% cpu : usr=0.13%, sys=0.86%, ctx=4968, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=147504: Mon Jan 12 18:49:02 2026 read: IOPS=10, BW=40.0KiB/s (40.0kB/s)(5060KiB/126498msec) slat (usec): min=42, max=107985, avg=17850.32, stdev=10687.80 clat (msec): min=22, max=1489, avg=492.50, stdev=162.85 lat (msec): min=35, max=1514, avg=510.36, stdev=165.20 clat percentiles (msec): | 1.00th=[ 159], 5.00th=[ 296], 10.00th=[ 330], 20.00th=[ 372], | 30.00th=[ 401], 40.00th=[ 439], 50.00th=[ 468], 60.00th=[ 502], | 70.00th=[ 542], 80.00th=[ 600], 90.00th=[ 684], 95.00th=[ 776], | 99.00th=[ 1070], 99.50th=[ 1183], 99.90th=[ 1301], 99.95th=[ 1485], | 99.99th=[ 1485] bw ( KiB/s): min= 7, max= 103, per=25.66%, avg=40.29, stdev=19.55, samples=245 iops : min= 1, max= 25, avg= 9.54, stdev= 4.88, samples=245 write: IOPS=10, BW=40.9KiB/s (41.9kB/s)(5180KiB/126498msec); 0 zone resets slat (usec): min=34, max=23784, avg=439.66, stdev=1504.95 clat (msec): min=132, max=2005, avg=1062.69, stdev=301.00 lat (msec): min=135, max=2005, avg=1063.13, stdev=300.98 clat percentiles (msec): | 1.00th=[ 334], 5.00th=[ 600], 10.00th=[ 709], 20.00th=[ 827], | 30.00th=[ 894], 40.00th=[ 961], 50.00th=[ 1036], 60.00th=[ 1116], | 70.00th=[ 1200], 80.00th=[ 1301], 90.00th=[ 1469], 95.00th=[ 1586], | 99.00th=[ 1871], 99.50th=[ 1938], 99.90th=[ 1989], 99.95th=[ 2005], | 99.99th=[ 2005] bw ( KiB/s): min= 7, max= 110, per=24.94%, avg=40.91, stdev=19.38, samples=246 iops : min= 1, max= 27, avg= 9.69, stdev= 4.89, samples=246 lat (msec) : 50=0.16%, 100=0.23%, 250=0.94%, 500=29.49%, 750=21.99% lat (msec) : 1000=18.87%, 2000=28.28%, >=2000=0.04% cpu : usr=0.12%, sys=0.79%, ctx=4902, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=158KiB/s (162kB/s), 39.1KiB/s-40.0KiB/s (40.1kB/s-40.0kB/s), io=19.6MiB (20.5MB), run=126498-127110msec WRITE: bw=164KiB/s (168kB/s), 40.9KiB/s-41.4KiB/s (41.9kB/s-42.4kB/s), io=20.4MiB (21.4MB), run=126498-127110msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=147528: Mon Jan 12 18:49:15 2026 read: IOPS=1, BW=74.2MiB/s (77.8MB/s)(40.0MiB/539msec) slat (nsec): min=36293k, max=36293k, avg=36293470.00, stdev= 0.00 clat (nsec): min=459927k, max=459927k, avg=459927234.00, stdev= 0.00 lat (nsec): min=496225k, max=496225k, avg=496224991.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 460], 5.00th=[ 460], 10.00th=[ 460], 20.00th=[ 460], | 30.00th=[ 460], 40.00th=[ 460], 50.00th=[ 460], 60.00th=[ 460], | 70.00th=[ 460], 80.00th=[ 460], 90.00th=[ 460], 95.00th=[ 460], | 99.00th=[ 460], 99.50th=[ 460], 99.90th=[ 460], 99.95th=[ 460], | 99.99th=[ 460] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=2.60%, ctx=5, majf=0, minf=7 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=74.2MiB/s (77.8MB/s), 74.2MiB/s-74.2MiB/s (77.8MB/s-77.8MB/s), io=40.0MiB (41.9MB), run=539-539msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout