== sanity test 398c: run fio to test AIO ================= 22:32:40 (1780453960) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.642811 s, 65.2 MB/s osc.lustre-OST0000-osc-ffff8e3a04cf7800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=146734: Tue Jun 2 22:35:38 2026 write: IOPS=15, BW=61.7KiB/s (63.2kB/s)(10.0MiB/165848msec); 0 zone resets slat (usec): min=38, max=17708, avg=346.25, stdev=1214.96 clat (msec): min=225, max=1821, avg=1035.03, stdev=250.57 lat (msec): min=226, max=1827, avg=1035.37, stdev=250.61 clat percentiles (msec): | 1.00th=[ 338], 5.00th=[ 676], 10.00th=[ 760], 20.00th=[ 844], | 30.00th=[ 902], 40.00th=[ 969], 50.00th=[ 1028], 60.00th=[ 1083], | 70.00th=[ 1133], 80.00th=[ 1217], 90.00th=[ 1368], 95.00th=[ 1469], | 99.00th=[ 1653], 99.50th=[ 1703], 99.90th=[ 1770], 99.95th=[ 1821], | 99.99th=[ 1821] bw ( KiB/s): min= 7, max= 256, per=25.45%, avg=62.61, stdev=27.64, samples=322 iops : min= 1, max= 64, avg=15.22, stdev= 6.96, samples=322 lat (msec) : 250=0.62%, 500=1.25%, 750=7.46%, 1000=35.27%, 2000=55.39% cpu : usr=0.06%, sys=0.28%, ctx=2212, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=146735: Tue Jun 2 22:35:38 2026 write: IOPS=15, BW=61.7KiB/s (63.2kB/s)(10.0MiB/165850msec); 0 zone resets slat (usec): min=50, max=18323, avg=327.07, stdev=1208.81 clat (msec): min=179, max=1823, avg=1035.90, stdev=250.13 lat (msec): min=179, max=1823, avg=1036.23, stdev=250.12 clat percentiles (msec): | 1.00th=[ 330], 5.00th=[ 684], 10.00th=[ 760], 20.00th=[ 835], | 30.00th=[ 911], 40.00th=[ 978], 50.00th=[ 1036], 60.00th=[ 1083], | 70.00th=[ 1150], 80.00th=[ 1217], 90.00th=[ 1351], 95.00th=[ 1469], | 99.00th=[ 1703], 99.50th=[ 1754], 99.90th=[ 1804], 99.95th=[ 1821], | 99.99th=[ 1821] bw ( KiB/s): min= 7, max= 192, per=25.21%, avg=62.02, stdev=26.57, samples=325 iops : min= 1, max= 48, avg=15.08, stdev= 6.67, samples=325 lat (msec) : 250=0.16%, 500=2.34%, 750=6.64%, 1000=35.51%, 2000=55.35% cpu : usr=0.04%, sys=0.28%, ctx=2150, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=146736: Tue Jun 2 22:35:38 2026 write: IOPS=15, BW=61.7KiB/s (63.2kB/s)(10.0MiB/165848msec); 0 zone resets slat (usec): min=40, max=21202, avg=288.24, stdev=946.53 clat (msec): min=220, max=1817, avg=1035.26, stdev=246.72 lat (msec): min=220, max=1817, avg=1035.55, stdev=246.74 clat percentiles (msec): | 1.00th=[ 342], 5.00th=[ 667], 10.00th=[ 751], 20.00th=[ 835], | 30.00th=[ 902], 40.00th=[ 978], 50.00th=[ 1028], 60.00th=[ 1083], | 70.00th=[ 1150], 80.00th=[ 1217], 90.00th=[ 1368], 95.00th=[ 1452], | 99.00th=[ 1670], 99.50th=[ 1720], 99.90th=[ 1787], 99.95th=[ 1821], | 99.99th=[ 1821] bw ( KiB/s): min= 7, max= 256, per=25.38%, avg=62.42, stdev=29.78, samples=323 iops : min= 1, max= 64, avg=15.17, stdev= 7.51, samples=323 lat (msec) : 250=0.62%, 500=1.45%, 750=7.70%, 1000=35.35%, 2000=54.88% cpu : usr=0.05%, sys=0.25%, ctx=1938, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=146737: Tue Jun 2 22:35:38 2026 write: IOPS=15, BW=61.7KiB/s (63.2kB/s)(10.0MiB/165846msec); 0 zone resets slat (usec): min=53, max=19716, avg=287.13, stdev=862.97 clat (msec): min=221, max=1879, avg=1035.05, stdev=248.98 lat (msec): min=221, max=1879, avg=1035.34, stdev=248.98 clat percentiles (msec): | 1.00th=[ 334], 5.00th=[ 684], 10.00th=[ 768], 20.00th=[ 844], | 30.00th=[ 911], 40.00th=[ 969], 50.00th=[ 1028], 60.00th=[ 1083], | 70.00th=[ 1150], 80.00th=[ 1234], 90.00th=[ 1368], 95.00th=[ 1469], | 99.00th=[ 1670], 99.50th=[ 1737], 99.90th=[ 1821], 99.95th=[ 1888], | 99.99th=[ 1888] bw ( KiB/s): min= 7, max= 256, per=25.06%, avg=61.65, stdev=24.97, samples=327 iops : min= 1, max= 64, avg=14.98, stdev= 6.30, samples=327 lat (msec) : 250=0.23%, 500=1.76%, 750=6.60%, 1000=36.52%, 2000=54.88% cpu : usr=0.04%, sys=0.26%, ctx=2220, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=247KiB/s (253kB/s), 61.7KiB/s-61.7KiB/s (63.2kB/s-63.2kB/s), io=40.0MiB (41.9MB), run=165846-165850msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=146771: Tue Jun 2 22:37:43 2026 read: IOPS=10, BW=43.7KiB/s (44.8kB/s)(5048KiB/115409msec) slat (usec): min=2898, max=68602, avg=14518.94, stdev=8158.93 clat (msec): min=24, max=1366, avg=446.76, stdev=130.34 lat (msec): min=34, max=1388, avg=461.29, stdev=131.51 clat percentiles (msec): | 1.00th=[ 207], 5.00th=[ 279], 10.00th=[ 317], 20.00th=[ 351], | 30.00th=[ 376], 40.00th=[ 405], 50.00th=[ 430], 60.00th=[ 460], | 70.00th=[ 489], 80.00th=[ 535], 90.00th=[ 600], 95.00th=[ 659], | 99.00th=[ 802], 99.50th=[ 1133], 99.90th=[ 1334], 99.95th=[ 1368], | 99.99th=[ 1368] bw ( KiB/s): min= 7, max= 136, per=25.09%, avg=43.41, stdev=21.48, samples=227 iops : min= 1, max= 34, avg=10.34, stdev= 5.39, samples=227 write: IOPS=11, BW=44.0KiB/s (46.1kB/s)(5192KiB/115409msec); 0 zone resets slat (usec): min=55, max=18804, avg=339.67, stdev=1007.13 clat (msec): min=281, max=1809, avg=970.50, stdev=256.37 lat (msec): min=281, max=1809, avg=970.84, stdev=256.36 clat percentiles (msec): | 1.00th=[ 414], 5.00th=[ 558], 10.00th=[ 659], 20.00th=[ 751], | 30.00th=[ 835], 40.00th=[ 902], 50.00th=[ 961], 60.00th=[ 1020], | 70.00th=[ 1083], 80.00th=[ 1150], 90.00th=[ 1318], 95.00th=[ 1452], | 99.00th=[ 1603], 99.50th=[ 1670], 99.90th=[ 1787], 99.95th=[ 1804], | 99.99th=[ 1804] bw ( KiB/s): min= 7, max= 119, per=25.01%, avg=45.02, stdev=20.19, samples=226 iops : min= 1, max= 29, avg=10.74, stdev= 5.06, samples=226 lat (msec) : 50=0.08%, 100=0.23%, 250=0.78%, 500=36.29%, 750=20.90% lat (msec) : 1000=19.30%, 2000=22.42% cpu : usr=0.09%, sys=0.60%, ctx=4709, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=146772: Tue Jun 2 22:37:43 2026 read: IOPS=10, BW=43.1KiB/s (44.1kB/s)(4976KiB/115484msec) slat (usec): min=2933, max=78563, avg=14548.94, stdev=8128.27 clat (msec): min=18, max=850, avg=447.66, stdev=124.16 lat (msec): min=37, max=868, avg=462.22, stdev=125.30 clat percentiles (msec): | 1.00th=[ 111], 5.00th=[ 275], 10.00th=[ 309], 20.00th=[ 347], | 30.00th=[ 380], 40.00th=[ 409], 50.00th=[ 443], 60.00th=[ 468], | 70.00th=[ 502], 80.00th=[ 542], 90.00th=[ 609], 95.00th=[ 676], | 99.00th=[ 793], 99.50th=[ 810], 99.90th=[ 844], 99.95th=[ 852], | 99.99th=[ 852] bw ( KiB/s): min= 7, max= 136, per=24.68%, avg=42.69, stdev=21.27, samples=228 iops : min= 1, max= 34, avg=10.16, stdev= 5.33, samples=228 write: IOPS=11, BW=45.6KiB/s (46.7kB/s)(5264KiB/115484msec); 0 zone resets slat (usec): min=53, max=10960, avg=318.07, stdev=738.29 clat (msec): min=266, max=1811, avg=964.66, stdev=265.24 lat (msec): min=267, max=1812, avg=964.98, stdev=265.25 clat percentiles (msec): | 1.00th=[ 342], 5.00th=[ 558], 10.00th=[ 642], 20.00th=[ 751], | 30.00th=[ 827], 40.00th=[ 885], 50.00th=[ 953], 60.00th=[ 1020], | 70.00th=[ 1083], 80.00th=[ 1167], 90.00th=[ 1318], 95.00th=[ 1452], | 99.00th=[ 1586], 99.50th=[ 1636], 99.90th=[ 1770], 99.95th=[ 1804], | 99.99th=[ 1804] bw ( KiB/s): min= 7, max= 103, per=24.97%, avg=44.95, stdev=19.86, samples=229 iops : min= 1, max= 25, avg=10.72, stdev= 5.00, samples=229 lat (msec) : 20=0.04%, 50=0.20%, 100=0.23%, 250=1.05%, 500=34.06% lat (msec) : 750=22.46%, 1000=19.80%, 2000=22.15% cpu : usr=0.08%, sys=0.61%, ctx=4376, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=146773: Tue Jun 2 22:37:43 2026 read: IOPS=10, BW=43.3KiB/s (44.3kB/s)(4984KiB/115146msec) slat (usec): min=85, max=80277, avg=14482.42, stdev=8303.37 clat (msec): min=18, max=1353, avg=443.79, stdev=121.91 lat (msec): min=18, max=1367, avg=458.28, stdev=123.13 clat percentiles (msec): | 1.00th=[ 197], 5.00th=[ 279], 10.00th=[ 313], 20.00th=[ 347], | 30.00th=[ 380], 40.00th=[ 405], 50.00th=[ 435], 60.00th=[ 460], | 70.00th=[ 493], 80.00th=[ 527], 90.00th=[ 600], 95.00th=[ 651], | 99.00th=[ 776], 99.50th=[ 802], 99.90th=[ 1351], 99.95th=[ 1351], | 99.99th=[ 1351] bw ( KiB/s): min= 7, max= 119, per=25.46%, avg=44.05, stdev=21.47, samples=222 iops : min= 1, max= 29, avg=10.50, stdev= 5.37, samples=222 write: IOPS=11, BW=45.6KiB/s (46.7kB/s)(5256KiB/115146msec); 0 zone resets slat (usec): min=51, max=31935, avg=386.24, stdev=1208.05 clat (msec): min=134, max=1837, avg=966.20, stdev=270.16 lat (msec): min=134, max=1837, avg=966.59, stdev=270.18 clat percentiles (msec): | 1.00th=[ 313], 5.00th=[ 550], 10.00th=[ 642], 20.00th=[ 760], | 30.00th=[ 827], 40.00th=[ 894], 50.00th=[ 953], 60.00th=[ 1011], | 70.00th=[ 1083], 80.00th=[ 1167], 90.00th=[ 1318], 95.00th=[ 1469], | 99.00th=[ 1636], 99.50th=[ 1703], 99.90th=[ 1838], 99.95th=[ 1838], | 99.99th=[ 1838] bw ( KiB/s): min= 7, max= 111, per=24.96%, avg=44.93, stdev=20.60, samples=228 iops : min= 1, max= 27, avg=10.72, stdev= 5.19, samples=228 lat (msec) : 20=0.08%, 50=0.12%, 100=0.08%, 250=1.37%, 500=34.80% lat (msec) : 750=21.41%, 1000=20.51%, 2000=21.64% cpu : usr=0.09%, sys=0.61%, ctx=4826, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=146774: Tue Jun 2 22:37:43 2026 read: IOPS=10, BW=43.8KiB/s (44.9kB/s)(5060KiB/115407msec) slat (usec): min=2623, max=68719, avg=14417.47, stdev=7846.83 clat (msec): min=25, max=1354, avg=445.63, stdev=126.68 lat (msec): min=34, max=1389, avg=460.05, stdev=127.82 clat percentiles (msec): | 1.00th=[ 178], 5.00th=[ 284], 10.00th=[ 317], 20.00th=[ 351], | 30.00th=[ 376], 40.00th=[ 405], 50.00th=[ 435], 60.00th=[ 460], | 70.00th=[ 493], 80.00th=[ 531], 90.00th=[ 592], 95.00th=[ 651], | 99.00th=[ 802], 99.50th=[ 1083], 99.90th=[ 1267], 99.95th=[ 1351], | 99.99th=[ 1351] bw ( KiB/s): min= 7, max= 111, per=25.52%, avg=44.15, stdev=20.27, samples=224 iops : min= 1, max= 27, avg=10.52, stdev= 5.09, samples=224 write: IOPS=11, BW=44.9KiB/s (45.0kB/s)(5180KiB/115407msec); 0 zone resets slat (usec): min=53, max=15928, avg=336.61, stdev=851.34 clat (msec): min=267, max=1795, avg=973.21, stdev=267.65 lat (msec): min=268, max=1795, avg=973.55, stdev=267.66 clat percentiles (msec): | 1.00th=[ 330], 5.00th=[ 542], 10.00th=[ 642], 20.00th=[ 751], | 30.00th=[ 827], 40.00th=[ 902], 50.00th=[ 969], 60.00th=[ 1028], | 70.00th=[ 1099], 80.00th=[ 1200], 90.00th=[ 1318], 95.00th=[ 1469], | 99.00th=[ 1620], 99.50th=[ 1636], 99.90th=[ 1770], 99.95th=[ 1804], | 99.99th=[ 1804] bw ( KiB/s): min= 7, max= 103, per=24.92%, avg=44.85, stdev=19.53, samples=226 iops : min= 1, max= 25, avg=10.70, stdev= 4.90, samples=226 lat (msec) : 50=0.12%, 100=0.20%, 250=0.90%, 500=35.94%, 750=21.41% lat (msec) : 1000=18.55%, 2000=22.89% cpu : usr=0.08%, sys=0.62%, ctx=4506, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=174KiB/s (178kB/s), 43.1KiB/s-43.8KiB/s (44.1kB/s-44.9kB/s), io=19.6MiB (20.5MB), run=115146-115484msec WRITE: bw=181KiB/s (185kB/s), 44.9KiB/s-45.6KiB/s (45.0kB/s-46.7kB/s), io=20.4MiB (21.4MB), run=115146-115484msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=146795: Tue Jun 2 22:37:54 2026 read: IOPS=2, BW=92.2MiB/s (96.6MB/s)(40.0MiB/434msec) slat (nsec): min=36750k, max=36750k, avg=36750378.00, stdev= 0.00 clat (nsec): min=377610k, max=377610k, avg=377610391.00, stdev= 0.00 lat (nsec): min=414366k, max=414366k, avg=414366277.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 380], 5.00th=[ 380], 10.00th=[ 380], 20.00th=[ 380], | 30.00th=[ 380], 40.00th=[ 380], 50.00th=[ 380], 60.00th=[ 380], | 70.00th=[ 380], 80.00th=[ 380], 90.00th=[ 380], 95.00th=[ 380], | 99.00th=[ 380], 99.50th=[ 380], 99.90th=[ 380], 99.95th=[ 380], | 99.99th=[ 380] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=3.00%, ctx=7, majf=0, minf=9 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=92.2MiB/s (96.6MB/s), 92.2MiB/s-92.2MiB/s (96.6MB/s-96.6MB/s), io=40.0MiB (41.9MB), run=434-434msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout