== sanity test 398c: run fio to test AIO ================= 04:30:31 (1779697831) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.556402 s, 75.4 MB/s osc.lustre-OST0000-osc-ffff9f3b3104a800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=146790: Mon May 25 04:33:39 2026 write: IOPS=14, BW=57.8KiB/s (59.2kB/s)(10.0MiB/177132msec); 0 zone resets slat (usec): min=48, max=25280, avg=290.95, stdev=1150.97 clat (msec): min=135, max=2061, avg=1105.99, stdev=289.11 lat (msec): min=136, max=2062, avg=1106.28, stdev=289.12 clat percentiles (msec): | 1.00th=[ 451], 5.00th=[ 651], 10.00th=[ 760], 20.00th=[ 877], | 30.00th=[ 953], 40.00th=[ 1020], 50.00th=[ 1099], 60.00th=[ 1183], | 70.00th=[ 1234], 80.00th=[ 1334], 90.00th=[ 1485], 95.00th=[ 1586], | 99.00th=[ 1821], 99.50th=[ 1905], 99.90th=[ 2056], 99.95th=[ 2056], | 99.99th=[ 2056] bw ( KiB/s): min= 7, max= 255, per=26.07%, avg=60.23, stdev=28.45, samples=334 iops : min= 1, max= 63, avg=14.57, stdev= 7.14, samples=334 lat (msec) : 250=0.62%, 500=1.25%, 750=7.30%, 1000=28.59%, 2000=61.99% lat (msec) : >=2000=0.23% cpu : usr=0.03%, sys=0.23%, ctx=1592, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=146791: Mon May 25 04:33:39 2026 write: IOPS=14, BW=57.8KiB/s (59.2kB/s)(10.0MiB/177087msec); 0 zone resets slat (usec): min=47, max=31179, avg=291.39, stdev=982.43 clat (msec): min=124, max=2097, avg=1106.24, stdev=286.59 lat (msec): min=124, max=2097, avg=1106.53, stdev=286.61 clat percentiles (msec): | 1.00th=[ 430], 5.00th=[ 659], 10.00th=[ 768], 20.00th=[ 877], | 30.00th=[ 944], 40.00th=[ 1020], 50.00th=[ 1099], 60.00th=[ 1183], | 70.00th=[ 1250], 80.00th=[ 1334], 90.00th=[ 1485], 95.00th=[ 1586], | 99.00th=[ 1804], 99.50th=[ 1955], 99.90th=[ 2005], 99.95th=[ 2022], | 99.99th=[ 2106] bw ( KiB/s): min= 7, max= 128, per=25.31%, avg=58.46, stdev=27.63, samples=344 iops : min= 1, max= 32, avg=14.13, stdev= 6.94, samples=344 lat (msec) : 250=0.62%, 500=0.66%, 750=6.84%, 1000=30.35%, 2000=61.37% lat (msec) : >=2000=0.16% cpu : usr=0.04%, sys=0.23%, ctx=1800, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=146792: Mon May 25 04:33:39 2026 write: IOPS=14, BW=57.9KiB/s (59.2kB/s)(10.0MiB/176982msec); 0 zone resets slat (usec): min=40, max=13727, avg=267.06, stdev=813.00 clat (msec): min=202, max=2061, avg=1104.88, stdev=281.91 lat (msec): min=202, max=2061, avg=1105.15, stdev=281.90 clat percentiles (msec): | 1.00th=[ 439], 5.00th=[ 659], 10.00th=[ 760], 20.00th=[ 885], | 30.00th=[ 953], 40.00th=[ 1028], 50.00th=[ 1099], 60.00th=[ 1167], | 70.00th=[ 1234], 80.00th=[ 1318], 90.00th=[ 1469], 95.00th=[ 1586], | 99.00th=[ 1821], 99.50th=[ 1888], 99.90th=[ 2022], 99.95th=[ 2022], | 99.99th=[ 2056] bw ( KiB/s): min= 7, max= 128, per=24.94%, avg=57.62, stdev=25.84, samples=349 iops : min= 1, max= 32, avg=13.91, stdev= 6.50, samples=349 lat (msec) : 250=0.27%, 500=1.60%, 750=7.66%, 1000=27.77%, 2000=62.58% lat (msec) : >=2000=0.12% cpu : usr=0.03%, sys=0.25%, ctx=1927, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=146793: Mon May 25 04:33:39 2026 write: IOPS=14, BW=57.8KiB/s (59.2kB/s)(10.0MiB/177129msec); 0 zone resets slat (usec): min=39, max=11419, avg=261.90, stdev=766.39 clat (msec): min=264, max=2096, avg=1105.89, stdev=280.98 lat (msec): min=265, max=2096, avg=1106.15, stdev=280.99 clat percentiles (msec): | 1.00th=[ 464], 5.00th=[ 667], 10.00th=[ 760], 20.00th=[ 885], | 30.00th=[ 944], 40.00th=[ 1011], 50.00th=[ 1099], 60.00th=[ 1183], | 70.00th=[ 1250], 80.00th=[ 1334], 90.00th=[ 1469], 95.00th=[ 1586], | 99.00th=[ 1838], 99.50th=[ 1921], 99.90th=[ 2089], 99.95th=[ 2089], | 99.99th=[ 2089] bw ( KiB/s): min= 7, max= 128, per=25.09%, avg=57.96, stdev=27.66, samples=347 iops : min= 1, max= 32, avg=14.01, stdev= 6.95, samples=347 lat (msec) : 500=1.72%, 750=7.62%, 1000=28.95%, 2000=61.52%, >=2000=0.20% cpu : usr=0.04%, sys=0.25%, ctx=1733, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=231KiB/s (237kB/s), 57.8KiB/s-57.9KiB/s (59.2kB/s-59.2kB/s), io=40.0MiB (41.9MB), run=176982-177132msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=146828: Mon May 25 04:35:56 2026 read: IOPS=10, BW=40.0KiB/s (40.0kB/s)(5048KiB/126184msec) slat (usec): min=56, max=60557, avg=15662.87, stdev=9267.26 clat (msec): min=13, max=1042, avg=486.05, stdev=149.09 lat (msec): min=17, max=1079, avg=501.72, stdev=151.15 clat percentiles (msec): | 1.00th=[ 97], 5.00th=[ 284], 10.00th=[ 321], 20.00th=[ 368], | 30.00th=[ 405], 40.00th=[ 435], 50.00th=[ 468], 60.00th=[ 502], | 70.00th=[ 558], 80.00th=[ 609], 90.00th=[ 684], 95.00th=[ 751], | 99.00th=[ 877], 99.50th=[ 927], 99.90th=[ 1036], 99.95th=[ 1045], | 99.99th=[ 1045] bw ( KiB/s): min= 7, max= 119, per=25.31%, avg=39.73, stdev=20.53, samples=248 iops : min= 1, max= 29, avg= 9.43, stdev= 5.19, samples=248 write: IOPS=10, BW=41.1KiB/s (42.1kB/s)(5192KiB/126184msec); 0 zone resets slat (usec): min=49, max=35906, avg=340.26, stdev=1316.49 clat (msec): min=245, max=1925, avg=1066.21, stdev=295.49 lat (msec): min=245, max=1925, avg=1066.56, stdev=295.51 clat percentiles (msec): | 1.00th=[ 489], 5.00th=[ 617], 10.00th=[ 693], 20.00th=[ 810], | 30.00th=[ 902], 40.00th=[ 969], 50.00th=[ 1036], 60.00th=[ 1116], | 70.00th=[ 1217], 80.00th=[ 1334], 90.00th=[ 1485], 95.00th=[ 1586], | 99.00th=[ 1787], 99.50th=[ 1871], 99.90th=[ 1921], 99.95th=[ 1921], | 99.99th=[ 1921] bw ( KiB/s): min= 7, max= 103, per=24.86%, avg=40.77, stdev=19.08, samples=248 iops : min= 1, max= 25, avg= 9.69, stdev= 4.82, samples=248 lat (msec) : 20=0.04%, 50=0.12%, 100=0.59%, 250=0.59%, 500=28.75% lat (msec) : 750=23.83%, 1000=17.81%, 2000=28.28% cpu : usr=0.10%, sys=0.56%, ctx=4315, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=146829: Mon May 25 04:35:56 2026 read: IOPS=9, BW=39.1KiB/s (40.0kB/s)(4976KiB/127245msec) slat (usec): min=3441, max=71725, avg=16079.76, stdev=9108.39 clat (msec): min=14, max=1005, avg=493.53, stdev=141.64 lat (msec): min=38, max=1019, avg=509.62, stdev=143.47 clat percentiles (msec): | 1.00th=[ 218], 5.00th=[ 305], 10.00th=[ 338], 20.00th=[ 376], | 30.00th=[ 409], 40.00th=[ 439], 50.00th=[ 477], 60.00th=[ 514], | 70.00th=[ 558], 80.00th=[ 609], 90.00th=[ 693], 95.00th=[ 751], | 99.00th=[ 852], 99.50th=[ 894], 99.90th=[ 969], 99.95th=[ 1003], | 99.99th=[ 1003] bw ( KiB/s): min= 7, max= 96, per=24.90%, avg=39.09, stdev=18.48, samples=248 iops : min= 1, max= 24, avg= 9.27, stdev= 4.67, samples=248 write: IOPS=10, BW=41.4KiB/s (42.4kB/s)(5264KiB/127245msec); 0 zone resets slat (usec): min=52, max=22257, avg=324.77, stdev=992.50 clat (msec): min=196, max=1961, avg=1062.13, stdev=301.14 lat (msec): min=196, max=1961, avg=1062.45, stdev=301.15 clat percentiles (msec): | 1.00th=[ 498], 5.00th=[ 600], 10.00th=[ 684], 20.00th=[ 810], | 30.00th=[ 877], 40.00th=[ 953], 50.00th=[ 1036], 60.00th=[ 1116], | 70.00th=[ 1217], 80.00th=[ 1318], 90.00th=[ 1485], 95.00th=[ 1620], | 99.00th=[ 1770], 99.50th=[ 1888], 99.90th=[ 1921], 99.95th=[ 1955], | 99.99th=[ 1955] bw ( KiB/s): min= 7, max= 88, per=25.16%, avg=41.26, stdev=20.22, samples=249 iops : min= 1, max= 22, avg= 9.82, stdev= 5.11, samples=249 lat (msec) : 20=0.04%, 50=0.12%, 100=0.20%, 250=0.66%, 500=27.70% lat (msec) : 750=25.20%, 1000=18.36%, 2000=27.73% cpu : usr=0.07%, sys=0.58%, ctx=4332, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=146830: Mon May 25 04:35:56 2026 read: IOPS=9, BW=39.2KiB/s (40.1kB/s)(4984KiB/127153msec) slat (usec): min=2546, max=70784, avg=16495.94, stdev=9550.29 clat (msec): min=20, max=1054, avg=488.71, stdev=143.25 lat (msec): min=31, max=1074, avg=505.21, stdev=145.17 clat percentiles (msec): | 1.00th=[ 234], 5.00th=[ 296], 10.00th=[ 330], 20.00th=[ 372], | 30.00th=[ 405], 40.00th=[ 430], 50.00th=[ 464], 60.00th=[ 498], | 70.00th=[ 550], 80.00th=[ 609], 90.00th=[ 684], 95.00th=[ 760], | 99.00th=[ 877], 99.50th=[ 902], 99.90th=[ 1020], 99.95th=[ 1053], | 99.99th=[ 1053] bw ( KiB/s): min= 7, max= 111, per=25.71%, avg=40.37, stdev=20.38, samples=242 iops : min= 1, max= 27, avg= 9.60, stdev= 5.16, samples=242 write: IOPS=10, BW=41.3KiB/s (42.3kB/s)(5256KiB/127153msec); 0 zone resets slat (usec): min=42, max=8508, avg=293.58, stdev=654.54 clat (msec): min=354, max=1986, avg=1066.02, stdev=296.79 lat (msec): min=354, max=1986, avg=1066.32, stdev=296.78 clat percentiles (msec): | 1.00th=[ 506], 5.00th=[ 634], 10.00th=[ 701], 20.00th=[ 793], | 30.00th=[ 885], 40.00th=[ 961], 50.00th=[ 1028], 60.00th=[ 1099], | 70.00th=[ 1234], 80.00th=[ 1334], 90.00th=[ 1485], 95.00th=[ 1603], | 99.00th=[ 1754], 99.50th=[ 1854], 99.90th=[ 1938], 99.95th=[ 1989], | 99.99th=[ 1989] bw ( KiB/s): min= 7, max= 103, per=25.17%, avg=41.29, stdev=19.35, samples=248 iops : min= 1, max= 25, avg= 9.83, stdev= 4.89, samples=248 lat (msec) : 50=0.16%, 100=0.04%, 250=0.47%, 500=29.14%, 750=23.95% lat (msec) : 1000=18.87%, 2000=27.38% cpu : usr=0.09%, sys=0.59%, ctx=4335, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=146831: Mon May 25 04:35:56 2026 read: IOPS=9, BW=39.8KiB/s (40.8kB/s)(5060KiB/127041msec) slat (usec): min=2838, max=61186, avg=15984.46, stdev=9330.26 clat (msec): min=20, max=1036, avg=490.55, stdev=144.63 lat (msec): min=42, max=1080, avg=506.54, stdev=146.49 clat percentiles (msec): | 1.00th=[ 199], 5.00th=[ 296], 10.00th=[ 330], 20.00th=[ 376], | 30.00th=[ 409], 40.00th=[ 439], 50.00th=[ 472], 60.00th=[ 502], | 70.00th=[ 558], 80.00th=[ 609], 90.00th=[ 693], 95.00th=[ 743], | 99.00th=[ 894], 99.50th=[ 927], 99.90th=[ 1036], 99.95th=[ 1036], | 99.99th=[ 1036] bw ( KiB/s): min= 7, max= 96, per=25.56%, avg=40.13, stdev=20.40, samples=247 iops : min= 1, max= 24, avg= 9.53, stdev= 5.14, samples=247 write: IOPS=10, BW=40.8KiB/s (41.8kB/s)(5180KiB/127041msec); 0 zone resets slat (usec): min=41, max=29036, avg=401.82, stdev=1538.93 clat (msec): min=400, max=1986, avg=1071.23, stdev=292.53 lat (msec): min=400, max=1986, avg=1071.64, stdev=292.58 clat percentiles (msec): | 1.00th=[ 502], 5.00th=[ 634], 10.00th=[ 701], 20.00th=[ 818], | 30.00th=[ 885], 40.00th=[ 969], 50.00th=[ 1045], 60.00th=[ 1133], | 70.00th=[ 1217], 80.00th=[ 1334], 90.00th=[ 1485], 95.00th=[ 1603], | 99.00th=[ 1754], 99.50th=[ 1854], 99.90th=[ 1921], 99.95th=[ 1989], | 99.99th=[ 1989] bw ( KiB/s): min= 7, max= 95, per=24.83%, avg=40.73, stdev=19.21, samples=248 iops : min= 1, max= 23, avg= 9.69, stdev= 4.83, samples=248 lat (msec) : 50=0.12%, 100=0.27%, 250=0.39%, 500=28.71%, 750=24.49% lat (msec) : 1000=17.81%, 2000=28.20% cpu : usr=0.08%, sys=0.61%, ctx=4501, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=158KiB/s (161kB/s), 39.1KiB/s-40.0KiB/s (40.0kB/s-40.0kB/s), io=19.6MiB (20.5MB), run=126184-127245msec WRITE: bw=164KiB/s (168kB/s), 40.8KiB/s-41.4KiB/s (41.8kB/s-42.4kB/s), io=20.4MiB (21.4MB), run=126184-127245msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=146854: Mon May 25 04:36:08 2026 read: IOPS=1, BW=63.9MiB/s (67.0MB/s)(40.0MiB/626msec) slat (nsec): min=36385k, max=36385k, avg=36385168.00, stdev= 0.00 clat (nsec): min=569371k, max=569371k, avg=569371215.00, stdev= 0.00 lat (nsec): min=605761k, max=605761k, avg=605760776.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 567], 5.00th=[ 567], 10.00th=[ 567], 20.00th=[ 567], | 30.00th=[ 567], 40.00th=[ 567], 50.00th=[ 567], 60.00th=[ 567], | 70.00th=[ 567], 80.00th=[ 567], 90.00th=[ 567], 95.00th=[ 567], | 99.00th=[ 567], 99.50th=[ 567], 99.90th=[ 567], 99.95th=[ 567], | 99.99th=[ 567] lat (msec) : 750=100.00% cpu : usr=0.00%, sys=1.92%, ctx=6, majf=0, minf=9 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=63.9MiB/s (67.0MB/s), 63.9MiB/s-63.9MiB/s (67.0MB/s-67.0MB/s), io=40.0MiB (41.9MB), run=626-626msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout