== sanity test 398c: run fio to test AIO ================= 05:49:26 (1779702566) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.449979 s, 93.2 MB/s osc.lustre-OST0000-osc-ffff8ca0451b0000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=145774: Mon May 25 05:52:33 2026 write: IOPS=14, BW=58.0KiB/s (59.4kB/s)(10.0MiB/176535msec); 0 zone resets slat (usec): min=44, max=23773, avg=421.92, stdev=1325.84 clat (msec): min=106, max=2217, avg=1101.72, stdev=290.04 lat (msec): min=107, max=2218, avg=1102.15, stdev=290.00 clat percentiles (msec): | 1.00th=[ 485], 5.00th=[ 659], 10.00th=[ 743], 20.00th=[ 869], | 30.00th=[ 953], 40.00th=[ 1020], 50.00th=[ 1083], 60.00th=[ 1150], | 70.00th=[ 1234], 80.00th=[ 1318], 90.00th=[ 1485], 95.00th=[ 1620], | 99.00th=[ 1888], 99.50th=[ 1972], 99.90th=[ 2140], 99.95th=[ 2165], | 99.99th=[ 2232] bw ( KiB/s): min= 7, max= 136, per=26.05%, avg=60.18, stdev=29.64, samples=334 iops : min= 1, max= 34, avg=14.48, stdev= 7.41, samples=334 lat (msec) : 250=0.08%, 500=1.95%, 750=8.09%, 1000=26.80%, 2000=62.66% lat (msec) : >=2000=0.43% cpu : usr=0.04%, sys=0.32%, ctx=2204, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145775: Mon May 25 05:52:33 2026 write: IOPS=14, BW=58.1KiB/s (59.5kB/s)(10.0MiB/176322msec); 0 zone resets slat (usec): min=39, max=23699, avg=471.61, stdev=1394.54 clat (msec): min=153, max=2281, avg=1101.04, stdev=293.33 lat (msec): min=153, max=2281, avg=1101.53, stdev=293.32 clat percentiles (msec): | 1.00th=[ 468], 5.00th=[ 667], 10.00th=[ 726], 20.00th=[ 869], | 30.00th=[ 953], 40.00th=[ 1011], 50.00th=[ 1099], 60.00th=[ 1167], | 70.00th=[ 1234], 80.00th=[ 1301], 90.00th=[ 1485], 95.00th=[ 1620], | 99.00th=[ 1888], 99.50th=[ 2056], 99.90th=[ 2232], 99.95th=[ 2265], | 99.99th=[ 2265] bw ( KiB/s): min= 7, max= 208, per=25.42%, avg=58.73, stdev=29.55, samples=342 iops : min= 1, max= 52, avg=14.12, stdev= 7.40, samples=342 lat (msec) : 250=0.08%, 500=1.41%, 750=10.08%, 1000=26.21%, 2000=61.52% lat (msec) : >=2000=0.70% cpu : usr=0.05%, sys=0.32%, ctx=2341, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145776: Mon May 25 05:52:33 2026 write: IOPS=14, BW=57.0KiB/s (59.4kB/s)(10.0MiB/176594msec); 0 zone resets slat (usec): min=38, max=24773, avg=446.92, stdev=1294.20 clat (msec): min=453, max=2181, avg=1102.02, stdev=285.65 lat (msec): min=454, max=2188, avg=1102.47, stdev=285.64 clat percentiles (msec): | 1.00th=[ 531], 5.00th=[ 651], 10.00th=[ 743], 20.00th=[ 852], | 30.00th=[ 944], 40.00th=[ 1028], 50.00th=[ 1099], 60.00th=[ 1167], | 70.00th=[ 1234], 80.00th=[ 1318], 90.00th=[ 1469], 95.00th=[ 1603], | 99.00th=[ 1888], 99.50th=[ 2005], 99.90th=[ 2165], 99.95th=[ 2165], | 99.99th=[ 2165] bw ( KiB/s): min= 7, max= 128, per=25.66%, avg=59.27, stdev=30.71, samples=339 iops : min= 1, max= 32, avg=14.26, stdev= 7.66, samples=339 lat (msec) : 500=0.23%, 750=10.66%, 1000=25.74%, 2000=62.85%, >=2000=0.51% cpu : usr=0.04%, sys=0.31%, ctx=2031, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145777: Mon May 25 05:52:33 2026 write: IOPS=14, BW=57.0KiB/s (59.4kB/s)(10.0MiB/176595msec); 0 zone resets slat (usec): min=38, max=29006, avg=439.84, stdev=1359.13 clat (msec): min=387, max=2141, avg=1101.83, stdev=283.98 lat (msec): min=387, max=2141, avg=1102.28, stdev=284.01 clat percentiles (msec): | 1.00th=[ 502], 5.00th=[ 667], 10.00th=[ 743], 20.00th=[ 869], | 30.00th=[ 944], 40.00th=[ 1020], 50.00th=[ 1099], 60.00th=[ 1150], | 70.00th=[ 1234], 80.00th=[ 1318], 90.00th=[ 1502], 95.00th=[ 1620], | 99.00th=[ 1854], 99.50th=[ 2005], 99.90th=[ 2140], 99.95th=[ 2140], | 99.99th=[ 2140] bw ( KiB/s): min= 7, max= 128, per=25.81%, avg=59.62, stdev=28.18, samples=337 iops : min= 1, max= 32, avg=14.35, stdev= 7.06, samples=337 lat (msec) : 500=0.98%, 750=10.16%, 1000=26.68%, 2000=61.68%, >=2000=0.51% cpu : usr=0.06%, sys=0.31%, ctx=2014, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=232KiB/s (238kB/s), 57.0KiB/s-58.1KiB/s (59.4kB/s-59.5kB/s), io=40.0MiB (41.9MB), run=176322-176595msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=145814: Mon May 25 05:54:41 2026 read: IOPS=10, BW=41.9KiB/s (42.9kB/s)(5048KiB/120595msec) slat (usec): min=2828, max=57071, avg=16556.38, stdev=8565.68 clat (msec): min=17, max=1042, avg=471.58, stdev=119.72 lat (msec): min=37, max=1054, avg=488.15, stdev=120.62 clat percentiles (msec): | 1.00th=[ 236], 5.00th=[ 309], 10.00th=[ 338], 20.00th=[ 380], | 30.00th=[ 409], 40.00th=[ 435], 50.00th=[ 460], 60.00th=[ 485], | 70.00th=[ 518], 80.00th=[ 550], 90.00th=[ 625], 95.00th=[ 684], | 99.00th=[ 852], 99.50th=[ 885], 99.90th=[ 1003], 99.95th=[ 1045], | 99.99th=[ 1045] bw ( KiB/s): min= 7, max= 103, per=25.23%, avg=41.88, stdev=19.41, samples=236 iops : min= 1, max= 25, avg= 9.94, stdev= 4.89, samples=236 write: IOPS=10, BW=43.1KiB/s (44.1kB/s)(5192KiB/120595msec); 0 zone resets slat (usec): min=55, max=14978, avg=461.44, stdev=1103.81 clat (msec): min=415, max=2092, avg=1008.13, stdev=258.94 lat (msec): min=415, max=2093, avg=1008.60, stdev=258.94 clat percentiles (msec): | 1.00th=[ 498], 5.00th=[ 600], 10.00th=[ 684], 20.00th=[ 785], | 30.00th=[ 852], 40.00th=[ 927], 50.00th=[ 995], 60.00th=[ 1053], | 70.00th=[ 1133], 80.00th=[ 1234], 90.00th=[ 1351], 95.00th=[ 1469], | 99.00th=[ 1620], 99.50th=[ 1653], 99.90th=[ 1871], 99.95th=[ 2089], | 99.99th=[ 2089] bw ( KiB/s): min= 7, max= 103, per=24.71%, avg=42.76, stdev=18.78, samples=237 iops : min= 1, max= 25, avg=10.16, stdev= 4.69, samples=237 lat (msec) : 20=0.04%, 50=0.04%, 100=0.20%, 250=0.35%, 500=31.99% lat (msec) : 750=23.83%, 1000=18.91%, 2000=24.61%, >=2000=0.04% cpu : usr=0.08%, sys=0.79%, ctx=5000, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145815: Mon May 25 05:54:41 2026 read: IOPS=10, BW=41.2KiB/s (42.2kB/s)(4976KiB/120672msec) slat (usec): min=159, max=75774, avg=16331.14, stdev=8537.53 clat (msec): min=33, max=1122, avg=475.87, stdev=122.17 lat (msec): min=44, max=1133, avg=492.21, stdev=123.21 clat percentiles (msec): | 1.00th=[ 253], 5.00th=[ 317], 10.00th=[ 347], 20.00th=[ 376], | 30.00th=[ 409], 40.00th=[ 435], 50.00th=[ 460], 60.00th=[ 489], | 70.00th=[ 518], 80.00th=[ 558], 90.00th=[ 642], 95.00th=[ 709], | 99.00th=[ 852], 99.50th=[ 894], 99.90th=[ 944], 99.95th=[ 1116], | 99.99th=[ 1116] bw ( KiB/s): min= 7, max= 103, per=25.04%, avg=41.57, stdev=19.33, samples=234 iops : min= 1, max= 25, avg= 9.87, stdev= 4.85, samples=234 write: IOPS=10, BW=43.6KiB/s (44.7kB/s)(5264KiB/120672msec); 0 zone resets slat (usec): min=52, max=22768, avg=450.72, stdev=1386.30 clat (msec): min=166, max=2022, avg=1000.13, stdev=262.14 lat (msec): min=166, max=2022, avg=1000.58, stdev=262.18 clat percentiles (msec): | 1.00th=[ 451], 5.00th=[ 600], 10.00th=[ 684], 20.00th=[ 785], | 30.00th=[ 860], 40.00th=[ 919], 50.00th=[ 978], 60.00th=[ 1045], | 70.00th=[ 1133], 80.00th=[ 1234], 90.00th=[ 1351], 95.00th=[ 1469], | 99.00th=[ 1620], 99.50th=[ 1670], 99.90th=[ 1905], 99.95th=[ 2022], | 99.99th=[ 2022] bw ( KiB/s): min= 7, max= 96, per=25.19%, avg=43.58, stdev=20.01, samples=236 iops : min= 1, max= 24, avg=10.37, stdev= 4.96, samples=236 lat (msec) : 50=0.12%, 100=0.08%, 250=0.39%, 500=31.60%, 750=23.36% lat (msec) : 1000=20.62%, 2000=23.79%, >=2000=0.04% cpu : usr=0.09%, sys=0.74%, ctx=4769, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145816: Mon May 25 05:54:41 2026 read: IOPS=10, BW=41.3KiB/s (42.3kB/s)(4984KiB/120592msec) slat (usec): min=3733, max=56635, avg=16467.31, stdev=8721.36 clat (msec): min=34, max=1053, avg=468.80, stdev=123.08 lat (msec): min=51, max=1066, avg=485.28, stdev=123.55 clat percentiles (msec): | 1.00th=[ 220], 5.00th=[ 309], 10.00th=[ 338], 20.00th=[ 376], | 30.00th=[ 405], 40.00th=[ 430], 50.00th=[ 456], 60.00th=[ 485], | 70.00th=[ 514], 80.00th=[ 558], 90.00th=[ 625], 95.00th=[ 684], | 99.00th=[ 835], 99.50th=[ 894], 99.90th=[ 961], 99.95th=[ 1053], | 99.99th=[ 1053] bw ( KiB/s): min= 7, max= 111, per=25.08%, avg=41.63, stdev=19.36, samples=235 iops : min= 1, max= 27, avg= 9.89, stdev= 4.81, samples=235 write: IOPS=10, BW=43.6KiB/s (44.6kB/s)(5256KiB/120592msec); 0 zone resets slat (usec): min=51, max=22986, avg=438.84, stdev=1261.94 clat (msec): min=368, max=2127, avg=1004.78, stdev=258.85 lat (msec): min=369, max=2127, avg=1005.22, stdev=258.76 clat percentiles (msec): | 1.00th=[ 518], 5.00th=[ 609], 10.00th=[ 693], 20.00th=[ 776], | 30.00th=[ 860], 40.00th=[ 919], 50.00th=[ 986], 60.00th=[ 1053], | 70.00th=[ 1116], 80.00th=[ 1234], 90.00th=[ 1351], 95.00th=[ 1469], | 99.00th=[ 1603], 99.50th=[ 1804], 99.90th=[ 2056], 99.95th=[ 2123], | 99.99th=[ 2123] bw ( KiB/s): min= 7, max= 111, per=25.40%, avg=43.95, stdev=18.61, samples=233 iops : min= 1, max= 27, avg=10.46, stdev= 4.66, samples=233 lat (msec) : 50=0.08%, 100=0.31%, 250=0.35%, 500=31.91%, 750=23.79% lat (msec) : 1000=18.98%, 2000=24.45%, >=2000=0.12% cpu : usr=0.10%, sys=0.70%, ctx=4470, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145817: Mon May 25 05:54:41 2026 read: IOPS=10, BW=42.1KiB/s (43.1kB/s)(5060KiB/120086msec) slat (usec): min=3356, max=63776, avg=16885.64, stdev=8896.41 clat (msec): min=28, max=994, avg=474.28, stdev=120.42 lat (msec): min=44, max=1012, avg=491.18, stdev=121.32 clat percentiles (msec): | 1.00th=[ 224], 5.00th=[ 309], 10.00th=[ 338], 20.00th=[ 380], | 30.00th=[ 414], 40.00th=[ 439], 50.00th=[ 464], 60.00th=[ 489], | 70.00th=[ 523], 80.00th=[ 558], 90.00th=[ 625], 95.00th=[ 684], | 99.00th=[ 827], 99.50th=[ 894], 99.90th=[ 986], 99.95th=[ 995], | 99.99th=[ 995] bw ( KiB/s): min= 7, max= 111, per=25.53%, avg=42.38, stdev=19.83, samples=234 iops : min= 1, max= 27, avg=10.06, stdev= 4.98, samples=234 write: IOPS=10, BW=43.1KiB/s (44.2kB/s)(5180KiB/120086msec); 0 zone resets slat (usec): min=56, max=22907, avg=532.41, stdev=1462.38 clat (msec): min=405, max=2050, avg=1000.07, stdev=256.59 lat (msec): min=406, max=2051, avg=1000.61, stdev=256.60 clat percentiles (msec): | 1.00th=[ 493], 5.00th=[ 592], 10.00th=[ 676], 20.00th=[ 785], | 30.00th=[ 852], 40.00th=[ 911], 50.00th=[ 986], 60.00th=[ 1053], | 70.00th=[ 1133], 80.00th=[ 1234], 90.00th=[ 1334], 95.00th=[ 1435], | 99.00th=[ 1636], 99.50th=[ 1670], 99.90th=[ 2022], 99.95th=[ 2056], | 99.99th=[ 2056] bw ( KiB/s): min= 7, max= 103, per=24.85%, avg=42.98, stdev=18.33, samples=235 iops : min= 1, max= 25, avg=10.22, stdev= 4.60, samples=235 lat (msec) : 50=0.08%, 100=0.27%, 250=0.27%, 500=31.80%, 750=24.41% lat (msec) : 1000=18.75%, 2000=24.34%, >=2000=0.08% cpu : usr=0.10%, sys=0.83%, ctx=4878, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=166KiB/s (170kB/s), 41.2KiB/s-42.1KiB/s (42.2kB/s-43.1kB/s), io=19.6MiB (20.5MB), run=120086-120672msec WRITE: bw=173KiB/s (177kB/s), 43.1KiB/s-43.6KiB/s (44.1kB/s-44.7kB/s), io=20.4MiB (21.4MB), run=120086-120672msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=145839: Mon May 25 05:54:53 2026 read: IOPS=1, BW=67.0MiB/s (70.3MB/s)(40.0MiB/597msec) slat (nsec): min=65666k, max=65666k, avg=65665984.00, stdev= 0.00 clat (nsec): min=500263k, max=500263k, avg=500262683.00, stdev= 0.00 lat (nsec): min=565935k, max=565935k, avg=565934570.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 502], 5.00th=[ 502], 10.00th=[ 502], 20.00th=[ 502], | 30.00th=[ 502], 40.00th=[ 502], 50.00th=[ 502], 60.00th=[ 502], | 70.00th=[ 502], 80.00th=[ 502], 90.00th=[ 502], 95.00th=[ 502], | 99.00th=[ 502], 99.50th=[ 502], 99.90th=[ 502], 99.95th=[ 502], | 99.99th=[ 502] lat (msec) : 750=100.00% cpu : usr=0.00%, sys=2.35%, ctx=43, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=67.0MiB/s (70.3MB/s), 67.0MiB/s-67.0MiB/s (70.3MB/s-70.3MB/s), io=40.0MiB (41.9MB), run=597-597msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout