== sanity test 398c: run fio to test AIO ================= 22:13:12 (1788833592) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.722102 s, 58.1 MB/s osc.lustre-OST0000-osc-ffff8ecbb167c800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=147953: Mon Sep 7 22:15:52 2026 write: IOPS=17, BW=69.2KiB/s (70.8kB/s)(10.0MiB/148077msec); 0 zone resets slat (usec): min=22, max=13979, avg=171.78, stdev=626.75 clat (msec): min=232, max=1924, avg=924.45, stdev=284.72 lat (msec): min=232, max=1924, avg=924.63, stdev=284.75 clat percentiles (msec): | 1.00th=[ 326], 5.00th=[ 535], 10.00th=[ 617], 20.00th=[ 709], | 30.00th=[ 768], 40.00th=[ 827], 50.00th=[ 885], 60.00th=[ 953], | 70.00th=[ 1020], 80.00th=[ 1116], 90.00th=[ 1301], 95.00th=[ 1502], | 99.00th=[ 1770], 99.50th=[ 1854], 99.90th=[ 1921], 99.95th=[ 1921], | 99.99th=[ 1921] bw ( KiB/s): min= 7, max= 254, per=25.63%, avg=70.73, stdev=32.31, samples=285 iops : min= 1, max= 63, avg=17.20, stdev= 8.10, samples=285 lat (msec) : 250=0.62%, 500=3.09%, 750=24.06%, 1000=39.10%, 2000=33.12% cpu : usr=0.04%, sys=0.21%, ctx=1749, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=147954: Mon Sep 7 22:15:52 2026 write: IOPS=17, BW=69.4KiB/s (71.0kB/s)(10.0MiB/147641msec); 0 zone resets slat (usec): min=22, max=18708, avg=170.90, stdev=644.30 clat (msec): min=118, max=1925, avg=922.11, stdev=288.76 lat (msec): min=118, max=1925, avg=922.28, stdev=288.77 clat percentiles (msec): | 1.00th=[ 334], 5.00th=[ 531], 10.00th=[ 600], 20.00th=[ 693], | 30.00th=[ 751], 40.00th=[ 818], 50.00th=[ 877], 60.00th=[ 953], | 70.00th=[ 1036], 80.00th=[ 1133], 90.00th=[ 1334], 95.00th=[ 1502], | 99.00th=[ 1737], 99.50th=[ 1804], 99.90th=[ 1871], 99.95th=[ 1888], | 99.99th=[ 1921] bw ( KiB/s): min= 7, max= 144, per=25.18%, avg=69.50, stdev=33.61, samples=290 iops : min= 1, max= 36, avg=16.90, stdev= 8.44, samples=290 lat (msec) : 250=0.39%, 500=3.59%, 750=25.35%, 1000=36.56%, 2000=34.10% cpu : usr=0.05%, sys=0.23%, ctx=1907, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=147955: Mon Sep 7 22:15:52 2026 write: IOPS=17, BW=69.2KiB/s (70.9kB/s)(10.0MiB/147988msec); 0 zone resets slat (usec): min=21, max=27751, avg=179.76, stdev=833.43 clat (msec): min=201, max=1923, avg=923.76, stdev=287.49 lat (msec): min=201, max=1923, avg=923.95, stdev=287.48 clat percentiles (msec): | 1.00th=[ 334], 5.00th=[ 527], 10.00th=[ 609], 20.00th=[ 701], | 30.00th=[ 768], 40.00th=[ 827], 50.00th=[ 885], 60.00th=[ 944], | 70.00th=[ 1028], 80.00th=[ 1116], 90.00th=[ 1301], 95.00th=[ 1502], | 99.00th=[ 1770], 99.50th=[ 1871], 99.90th=[ 1921], 99.95th=[ 1921], | 99.99th=[ 1921] bw ( KiB/s): min= 7, max= 152, per=25.09%, avg=69.26, stdev=29.32, samples=291 iops : min= 1, max= 38, avg=16.84, stdev= 7.36, samples=291 lat (msec) : 250=0.62%, 500=3.24%, 750=22.66%, 1000=40.70%, 2000=32.77% cpu : usr=0.05%, sys=0.21%, ctx=1802, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=147956: Mon Sep 7 22:15:52 2026 write: IOPS=17, BW=69.1KiB/s (70.8kB/s)(10.0MiB/148124msec); 0 zone resets slat (usec): min=22, max=20411, avg=167.58, stdev=800.45 clat (msec): min=91, max=2011, avg=924.57, stdev=290.13 lat (msec): min=92, max=2011, avg=924.73, stdev=290.14 clat percentiles (msec): | 1.00th=[ 330], 5.00th=[ 527], 10.00th=[ 617], 20.00th=[ 693], | 30.00th=[ 768], 40.00th=[ 827], 50.00th=[ 885], 60.00th=[ 961], | 70.00th=[ 1028], 80.00th=[ 1116], 90.00th=[ 1301], 95.00th=[ 1519], | 99.00th=[ 1770], 99.50th=[ 1871], 99.90th=[ 2005], 99.95th=[ 2005], | 99.99th=[ 2005] bw ( KiB/s): min= 15, max= 200, per=25.24%, avg=69.67, stdev=32.40, samples=289 iops : min= 3, max= 50, avg=16.94, stdev= 8.11, samples=289 lat (msec) : 100=0.04%, 250=0.51%, 500=3.40%, 750=23.83%, 1000=39.61% lat (msec) : 2000=32.54%, >=2000=0.08% cpu : usr=0.04%, sys=0.22%, ctx=1743, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=277KiB/s (283kB/s), 69.1KiB/s-69.4KiB/s (70.8kB/s-71.0kB/s), io=40.0MiB (41.9MB), run=147641-148124msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=147986: Mon Sep 7 22:17:33 2026 read: IOPS=13, BW=54.8KiB/s (56.1kB/s)(5048KiB/92138msec) slat (usec): min=2907, max=61148, avg=12751.61, stdev=7224.42 clat (msec): min=12, max=888, avg=360.43, stdev=89.35 lat (msec): min=23, max=896, avg=373.19, stdev=90.30 clat percentiles (msec): | 1.00th=[ 153], 5.00th=[ 232], 10.00th=[ 257], 20.00th=[ 292], | 30.00th=[ 313], 40.00th=[ 338], 50.00th=[ 359], 60.00th=[ 380], | 70.00th=[ 401], 80.00th=[ 430], 90.00th=[ 468], 95.00th=[ 493], | 99.00th=[ 617], 99.50th=[ 651], 99.90th=[ 818], 99.95th=[ 885], | 99.99th=[ 885] bw ( KiB/s): min= 15, max= 119, per=25.14%, avg=54.55, stdev=22.50, samples=182 iops : min= 3, max= 29, avg=13.17, stdev= 5.64, samples=182 write: IOPS=14, BW=56.3KiB/s (57.7kB/s)(5192KiB/92138msec); 0 zone resets slat (usec): min=27, max=11385, avg=212.53, stdev=739.95 clat (msec): min=299, max=1461, avg=770.22, stdev=193.79 lat (msec): min=299, max=1461, avg=770.43, stdev=193.78 clat percentiles (msec): | 1.00th=[ 338], 5.00th=[ 451], 10.00th=[ 514], 20.00th=[ 617], | 30.00th=[ 676], 40.00th=[ 718], 50.00th=[ 768], 60.00th=[ 818], | 70.00th=[ 869], 80.00th=[ 927], 90.00th=[ 1020], 95.00th=[ 1083], | 99.00th=[ 1267], 99.50th=[ 1284], 99.90th=[ 1435], 99.95th=[ 1469], | 99.99th=[ 1469] bw ( KiB/s): min= 7, max= 111, per=24.61%, avg=55.61, stdev=19.26, samples=183 iops : min= 1, max= 27, avg=13.44, stdev= 4.83, samples=183 lat (msec) : 20=0.04%, 50=0.08%, 100=0.23%, 250=3.91%, 500=47.58% lat (msec) : 750=20.47%, 1000=21.48%, 2000=6.21% cpu : usr=0.13%, sys=0.65%, ctx=4484, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=147987: Mon Sep 7 22:17:33 2026 read: IOPS=13, BW=54.0KiB/s (55.3kB/s)(4976KiB/92146msec) slat (usec): min=29, max=46866, avg=12857.09, stdev=6926.88 clat (msec): min=8, max=1291, avg=362.38, stdev=105.57 lat (msec): min=12, max=1299, avg=375.24, stdev=106.46 clat percentiles (msec): | 1.00th=[ 82], 5.00th=[ 226], 10.00th=[ 251], 20.00th=[ 288], | 30.00th=[ 317], 40.00th=[ 338], 50.00th=[ 359], 60.00th=[ 380], | 70.00th=[ 401], 80.00th=[ 430], 90.00th=[ 468], 95.00th=[ 498], | 99.00th=[ 600], 99.50th=[ 1083], 99.90th=[ 1234], 99.95th=[ 1301], | 99.99th=[ 1301] bw ( KiB/s): min= 7, max= 127, per=24.73%, avg=53.66, stdev=23.75, samples=182 iops : min= 1, max= 31, avg=12.95, stdev= 5.98, samples=182 write: IOPS=14, BW=57.1KiB/s (58.5kB/s)(5264KiB/92146msec); 0 zone resets slat (usec): min=25, max=26912, avg=201.23, stdev=902.90 clat (msec): min=168, max=1504, avg=764.40, stdev=198.57 lat (msec): min=168, max=1504, avg=764.60, stdev=198.54 clat percentiles (msec): | 1.00th=[ 255], 5.00th=[ 451], 10.00th=[ 523], 20.00th=[ 600], | 30.00th=[ 667], 40.00th=[ 718], 50.00th=[ 768], 60.00th=[ 810], | 70.00th=[ 860], 80.00th=[ 927], 90.00th=[ 1011], 95.00th=[ 1083], | 99.00th=[ 1267], 99.50th=[ 1301], 99.90th=[ 1502], 99.95th=[ 1502], | 99.99th=[ 1502] bw ( KiB/s): min= 15, max= 119, per=25.11%, avg=56.75, stdev=19.38, samples=182 iops : min= 3, max= 29, avg=13.73, stdev= 4.83, samples=182 lat (msec) : 10=0.04%, 50=0.16%, 100=0.35%, 250=4.57%, 500=45.51% lat (msec) : 750=21.95%, 1000=21.52%, 2000=5.90% cpu : usr=0.12%, sys=0.59%, ctx=4007, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=147988: Mon Sep 7 22:17:33 2026 read: IOPS=13, BW=54.0KiB/s (55.3kB/s)(4984KiB/92235msec) slat (usec): min=2531, max=50734, avg=12917.26, stdev=7075.92 clat (msec): min=15, max=669, avg=362.57, stdev=85.11 lat (msec): min=19, max=691, avg=375.50, stdev=86.12 clat percentiles (msec): | 1.00th=[ 182], 5.00th=[ 230], 10.00th=[ 255], 20.00th=[ 296], | 30.00th=[ 321], 40.00th=[ 342], 50.00th=[ 359], 60.00th=[ 380], | 70.00th=[ 405], 80.00th=[ 430], 90.00th=[ 468], 95.00th=[ 498], | 99.00th=[ 600], 99.50th=[ 617], 99.90th=[ 659], 99.95th=[ 667], | 99.99th=[ 667] bw ( KiB/s): min= 7, max= 143, per=24.78%, avg=53.78, stdev=23.43, samples=183 iops : min= 1, max= 35, avg=12.97, stdev= 5.90, samples=183 write: IOPS=14, BW=56.0KiB/s (58.4kB/s)(5256KiB/92235msec); 0 zone resets slat (usec): min=27, max=15097, avg=214.50, stdev=785.38 clat (msec): min=89, max=1352, avg=765.05, stdev=200.03 lat (msec): min=89, max=1352, avg=765.27, stdev=200.01 clat percentiles (msec): | 1.00th=[ 326], 5.00th=[ 443], 10.00th=[ 506], 20.00th=[ 592], | 30.00th=[ 659], 40.00th=[ 718], 50.00th=[ 768], 60.00th=[ 818], | 70.00th=[ 860], 80.00th=[ 927], 90.00th=[ 1020], 95.00th=[ 1116], | 99.00th=[ 1234], 99.50th=[ 1284], 99.90th=[ 1334], 99.95th=[ 1351], | 99.99th=[ 1351] bw ( KiB/s): min= 8, max= 127, per=24.93%, avg=56.35, stdev=20.13, samples=182 iops : min= 2, max= 31, avg=13.62, stdev= 5.02, samples=182 lat (msec) : 20=0.04%, 50=0.08%, 100=0.16%, 250=4.14%, 500=46.88% lat (msec) : 750=21.13%, 1000=20.86%, 2000=6.72% cpu : usr=0.11%, sys=0.62%, ctx=4179, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=147989: Mon Sep 7 22:17:33 2026 read: IOPS=13, BW=54.9KiB/s (56.2kB/s)(5060KiB/92137msec) slat (usec): min=2550, max=60410, avg=12939.66, stdev=6805.74 clat (msec): min=29, max=829, avg=363.12, stdev=88.86 lat (msec): min=37, max=837, avg=376.07, stdev=89.79 clat percentiles (msec): | 1.00th=[ 153], 5.00th=[ 230], 10.00th=[ 259], 20.00th=[ 292], | 30.00th=[ 317], 40.00th=[ 342], 50.00th=[ 359], 60.00th=[ 380], | 70.00th=[ 405], 80.00th=[ 430], 90.00th=[ 472], 95.00th=[ 510], | 99.00th=[ 600], 99.50th=[ 634], 99.90th=[ 659], 99.95th=[ 827], | 99.99th=[ 827] bw ( KiB/s): min= 7, max= 111, per=25.36%, avg=55.04, stdev=21.10, samples=181 iops : min= 1, max= 27, avg=13.29, stdev= 5.29, samples=181 write: IOPS=14, BW=56.2KiB/s (57.6kB/s)(5180KiB/92137msec); 0 zone resets slat (usec): min=25, max=10594, avg=226.57, stdev=774.54 clat (msec): min=262, max=1346, avg=768.85, stdev=198.52 lat (msec): min=263, max=1346, avg=769.08, stdev=198.51 clat percentiles (msec): | 1.00th=[ 300], 5.00th=[ 451], 10.00th=[ 514], 20.00th=[ 600], | 30.00th=[ 667], 40.00th=[ 718], 50.00th=[ 776], 60.00th=[ 818], | 70.00th=[ 860], 80.00th=[ 936], 90.00th=[ 1020], 95.00th=[ 1099], | 99.00th=[ 1250], 99.50th=[ 1318], 99.90th=[ 1334], 99.95th=[ 1351], | 99.99th=[ 1351] bw ( KiB/s): min= 7, max= 143, per=24.52%, avg=55.41, stdev=21.57, samples=183 iops : min= 1, max= 35, avg=13.39, stdev= 5.37, samples=183 lat (msec) : 50=0.16%, 100=0.20%, 250=3.67%, 500=46.91%, 750=21.25% lat (msec) : 1000=21.56%, 2000=6.25% cpu : usr=0.09%, sys=0.66%, ctx=4210, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=218KiB/s (223kB/s), 54.0KiB/s-54.9KiB/s (55.3kB/s-56.2kB/s), io=19.6MiB (20.5MB), run=92137-92235msec WRITE: bw=227KiB/s (232kB/s), 56.2KiB/s-57.1KiB/s (57.6kB/s-58.5kB/s), io=20.4MiB (21.4MB), run=92137-92235msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=148007: Mon Sep 7 22:17:43 2026 read: IOPS=1, BW=73.3MiB/s (76.8MB/s)(40.0MiB/546msec) slat (nsec): min=23639k, max=23639k, avg=23638954.00, stdev= 0.00 clat (nsec): min=504289k, max=504289k, avg=504288831.00, stdev= 0.00 lat (nsec): min=527934k, max=527934k, avg=527933960.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 506], 5.00th=[ 506], 10.00th=[ 506], 20.00th=[ 506], | 30.00th=[ 506], 40.00th=[ 506], 50.00th=[ 506], 60.00th=[ 506], | 70.00th=[ 506], 80.00th=[ 506], 90.00th=[ 506], 95.00th=[ 506], | 99.00th=[ 506], 99.50th=[ 506], 99.90th=[ 506], 99.95th=[ 506], | 99.99th=[ 506] lat (msec) : 750=100.00% cpu : usr=0.00%, sys=1.83%, ctx=5, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=73.3MiB/s (76.8MB/s), 73.3MiB/s-73.3MiB/s (76.8MB/s-76.8MB/s), io=40.0MiB (41.9MB), run=546-546msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout