== sanity test 398c: run fio to test AIO ================= 02:06:52 (1787465212) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.603556 s, 69.5 MB/s osc.lustre-OST0000-osc-ffff959685c2f000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=146173: Sun Aug 23 02:11:04 2026 write: IOPS=10, BW=42.9KiB/s (43.9kB/s)(10.0MiB/238609msec); 0 zone resets slat (usec): min=29, max=31680, avg=293.94, stdev=1405.68 clat (msec): min=328, max=2970, avg=1489.81, stdev=441.93 lat (msec): min=329, max=2970, avg=1490.11, stdev=441.90 clat percentiles (msec): | 1.00th=[ 451], 5.00th=[ 785], 10.00th=[ 953], 20.00th=[ 1133], | 30.00th=[ 1250], 40.00th=[ 1351], 50.00th=[ 1469], 60.00th=[ 1586], | 70.00th=[ 1703], 80.00th=[ 1871], 90.00th=[ 2072], 95.00th=[ 2265], | 99.00th=[ 2601], 99.50th=[ 2668], 99.90th=[ 2903], 99.95th=[ 2970], | 99.99th=[ 2970] bw ( KiB/s): min= 7, max= 247, per=25.56%, avg=43.71, stdev=25.66, samples=458 iops : min= 1, max= 61, avg=10.39, stdev= 6.45, samples=458 lat (msec) : 500=1.33%, 750=2.58%, 1000=8.83%, 2000=74.69%, >=2000=12.58% cpu : usr=0.04%, sys=0.17%, ctx=1827, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=146174: Sun Aug 23 02:11:04 2026 write: IOPS=10, BW=42.9KiB/s (43.9kB/s)(10.0MiB/238613msec); 0 zone resets slat (usec): min=29, max=25260, avg=268.49, stdev=1158.68 clat (msec): min=159, max=2680, avg=1490.56, stdev=441.56 lat (msec): min=160, max=2680, avg=1490.83, stdev=441.55 clat percentiles (msec): | 1.00th=[ 405], 5.00th=[ 818], 10.00th=[ 961], 20.00th=[ 1116], | 30.00th=[ 1250], 40.00th=[ 1368], 50.00th=[ 1485], 60.00th=[ 1586], | 70.00th=[ 1720], 80.00th=[ 1888], 90.00th=[ 2106], 95.00th=[ 2232], | 99.00th=[ 2500], 99.50th=[ 2601], 99.90th=[ 2668], 99.95th=[ 2668], | 99.99th=[ 2668] bw ( KiB/s): min= 7, max= 135, per=26.97%, avg=46.12, stdev=26.20, samples=434 iops : min= 1, max= 33, avg=10.99, stdev= 6.55, samples=434 lat (msec) : 250=0.08%, 500=2.11%, 750=2.11%, 1000=8.91%, 2000=72.66% lat (msec) : >=2000=14.14% cpu : usr=0.04%, sys=0.16%, ctx=1841, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=146175: Sun Aug 23 02:11:04 2026 write: IOPS=10, BW=42.9KiB/s (43.0kB/s)(10.0MiB/238539msec); 0 zone resets slat (usec): min=30, max=29841, avg=283.43, stdev=1261.69 clat (msec): min=395, max=2965, avg=1489.31, stdev=441.92 lat (msec): min=395, max=2965, avg=1489.59, stdev=441.96 clat percentiles (msec): | 1.00th=[ 439], 5.00th=[ 810], 10.00th=[ 953], 20.00th=[ 1133], | 30.00th=[ 1250], 40.00th=[ 1351], 50.00th=[ 1469], 60.00th=[ 1569], | 70.00th=[ 1720], 80.00th=[ 1871], 90.00th=[ 2056], 95.00th=[ 2265], | 99.00th=[ 2534], 99.50th=[ 2668], 99.90th=[ 2937], 99.95th=[ 2970], | 99.99th=[ 2970] bw ( KiB/s): min= 7, max= 255, per=26.02%, avg=44.50, stdev=24.73, samples=450 iops : min= 1, max= 63, avg=10.58, stdev= 6.20, samples=450 lat (msec) : 500=1.25%, 750=2.19%, 1000=8.48%, 2000=75.27%, >=2000=12.81% cpu : usr=0.04%, sys=0.18%, ctx=1734, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=146176: Sun Aug 23 02:11:04 2026 write: IOPS=10, BW=42.9KiB/s (43.9kB/s)(10.0MiB/238706msec); 0 zone resets slat (usec): min=28, max=27633, avg=230.75, stdev=1074.24 clat (msec): min=402, max=2972, avg=1490.37, stdev=441.49 lat (msec): min=402, max=2972, avg=1490.60, stdev=441.51 clat percentiles (msec): | 1.00th=[ 456], 5.00th=[ 793], 10.00th=[ 969], 20.00th=[ 1116], | 30.00th=[ 1250], 40.00th=[ 1351], 50.00th=[ 1452], 60.00th=[ 1586], | 70.00th=[ 1703], 80.00th=[ 1871], 90.00th=[ 2072], 95.00th=[ 2265], | 99.00th=[ 2601], 99.50th=[ 2635], 99.90th=[ 2903], 99.95th=[ 2970], | 99.99th=[ 2970] bw ( KiB/s): min= 7, max= 254, per=26.44%, avg=45.21, stdev=25.96, samples=443 iops : min= 1, max= 63, avg=10.77, stdev= 6.50, samples=443 lat (msec) : 500=1.25%, 750=2.77%, 1000=7.58%, 2000=75.31%, >=2000=13.09% cpu : usr=0.04%, sys=0.15%, ctx=1488, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=172KiB/s (176kB/s), 42.9KiB/s-42.9KiB/s (43.9kB/s-43.0kB/s), io=40.0MiB (41.9MB), run=238539-238706msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=146227: Sun Aug 23 02:13:43 2026 read: IOPS=8, BW=34.3KiB/s (35.1kB/s)(5048KiB/147262msec) slat (usec): min=38, max=81900, avg=18833.82, stdev=10930.69 clat (msec): min=17, max=1511, avg=584.47, stdev=216.62 lat (msec): min=30, max=1531, avg=603.31, stdev=219.83 clat percentiles (msec): | 1.00th=[ 140], 5.00th=[ 313], 10.00th=[ 359], 20.00th=[ 409], | 30.00th=[ 456], 40.00th=[ 502], 50.00th=[ 535], 60.00th=[ 592], | 70.00th=[ 676], 80.00th=[ 768], 90.00th=[ 877], 95.00th=[ 978], | 99.00th=[ 1167], 99.50th=[ 1267], 99.90th=[ 1502], 99.95th=[ 1519], | 99.99th=[ 1519] bw ( KiB/s): min= 7, max= 95, per=25.76%, avg=34.77, stdev=18.67, samples=283 iops : min= 1, max= 23, avg= 8.16, stdev= 4.72, samples=283 write: IOPS=8, BW=35.3KiB/s (36.1kB/s)(5192KiB/147262msec); 0 zone resets slat (usec): min=33, max=42867, avg=333.79, stdev=1689.98 clat (msec): min=143, max=3018, avg=1227.21, stdev=435.02 lat (msec): min=143, max=3018, avg=1227.55, stdev=435.09 clat percentiles (msec): | 1.00th=[ 422], 5.00th=[ 642], 10.00th=[ 735], 20.00th=[ 869], | 30.00th=[ 953], 40.00th=[ 1083], 50.00th=[ 1167], 60.00th=[ 1284], | 70.00th=[ 1401], 80.00th=[ 1552], 90.00th=[ 1787], 95.00th=[ 1989], | 99.00th=[ 2668], 99.50th=[ 2769], 99.90th=[ 3004], 99.95th=[ 3004], | 99.99th=[ 3004] bw ( KiB/s): min= 7, max= 104, per=25.32%, avg=35.45, stdev=19.47, samples=285 iops : min= 1, max= 26, avg= 8.33, stdev= 4.93, samples=285 lat (msec) : 20=0.04%, 50=0.23%, 100=0.12%, 250=0.86%, 500=19.02% lat (msec) : 750=23.36%, 1000=20.98%, 2000=32.89%, >=2000=2.50% cpu : usr=0.11%, sys=0.58%, ctx=4349, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=146228: Sun Aug 23 02:13:43 2026 read: IOPS=8, BW=33.6KiB/s (34.4kB/s)(4976KiB/148260msec) slat (usec): min=3156, max=79236, avg=18680.49, stdev=11212.42 clat (msec): min=11, max=1371, avg=571.91, stdev=207.91 lat (msec): min=23, max=1387, avg=590.60, stdev=210.52 clat percentiles (msec): | 1.00th=[ 171], 5.00th=[ 313], 10.00th=[ 351], 20.00th=[ 401], | 30.00th=[ 443], 40.00th=[ 489], 50.00th=[ 542], 60.00th=[ 584], | 70.00th=[ 642], 80.00th=[ 735], 90.00th=[ 860], 95.00th=[ 969], | 99.00th=[ 1167], 99.50th=[ 1318], 99.90th=[ 1368], 99.95th=[ 1368], | 99.99th=[ 1368] bw ( KiB/s): min= 7, max= 120, per=25.49%, avg=34.41, stdev=20.06, samples=281 iops : min= 1, max= 30, avg= 8.09, stdev= 5.09, samples=281 write: IOPS=8, BW=35.5KiB/s (36.4kB/s)(5264KiB/148260msec); 0 zone resets slat (usec): min=30, max=42756, avg=334.09, stdev=1925.39 clat (msec): min=268, max=3051, avg=1241.07, stdev=433.13 lat (msec): min=268, max=3052, avg=1241.43, stdev=433.25 clat percentiles (msec): | 1.00th=[ 510], 5.00th=[ 642], 10.00th=[ 760], 20.00th=[ 885], | 30.00th=[ 978], 40.00th=[ 1083], 50.00th=[ 1183], 60.00th=[ 1301], | 70.00th=[ 1401], 80.00th=[ 1536], 90.00th=[ 1804], 95.00th=[ 2056], | 99.00th=[ 2668], 99.50th=[ 2903], 99.90th=[ 3037], 99.95th=[ 3037], | 99.99th=[ 3037] bw ( KiB/s): min= 7, max= 94, per=25.11%, avg=35.16, stdev=18.71, samples=292 iops : min= 1, max= 23, avg= 8.28, stdev= 4.70, samples=292 lat (msec) : 20=0.04%, 50=0.12%, 100=0.04%, 250=0.90%, 500=19.92% lat (msec) : 750=23.79%, 1000=18.79%, 2000=33.40%, >=2000=3.01% cpu : usr=0.11%, sys=0.58%, ctx=4329, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=146229: Sun Aug 23 02:13:43 2026 read: IOPS=8, BW=33.6KiB/s (34.4kB/s)(4984KiB/148270msec) slat (usec): min=3069, max=86649, avg=18305.08, stdev=11123.46 clat (msec): min=16, max=1378, avg=577.71, stdev=207.66 lat (msec): min=25, max=1397, avg=596.02, stdev=211.24 clat percentiles (msec): | 1.00th=[ 213], 5.00th=[ 321], 10.00th=[ 359], 20.00th=[ 414], | 30.00th=[ 451], 40.00th=[ 493], 50.00th=[ 531], 60.00th=[ 575], | 70.00th=[ 651], 80.00th=[ 743], 90.00th=[ 869], 95.00th=[ 969], | 99.00th=[ 1183], 99.50th=[ 1234], 99.90th=[ 1351], 99.95th=[ 1385], | 99.99th=[ 1385] bw ( KiB/s): min= 7, max= 111, per=25.92%, avg=34.99, stdev=18.47, samples=278 iops : min= 1, max= 27, avg= 8.23, stdev= 4.68, samples=278 write: IOPS=8, BW=35.4KiB/s (36.3kB/s)(5256KiB/148270msec); 0 zone resets slat (usec): min=31, max=20094, avg=315.08, stdev=1188.35 clat (msec): min=312, max=3006, avg=1236.16, stdev=432.69 lat (msec): min=312, max=3007, avg=1236.49, stdev=432.76 clat percentiles (msec): | 1.00th=[ 506], 5.00th=[ 642], 10.00th=[ 735], 20.00th=[ 869], | 30.00th=[ 969], 40.00th=[ 1083], 50.00th=[ 1183], 60.00th=[ 1301], | 70.00th=[ 1418], 80.00th=[ 1552], 90.00th=[ 1804], 95.00th=[ 2022], | 99.00th=[ 2635], 99.50th=[ 2769], 99.90th=[ 2970], 99.95th=[ 3004], | 99.99th=[ 3004] bw ( KiB/s): min= 7, max= 112, per=25.17%, avg=35.24, stdev=18.75, samples=290 iops : min= 1, max= 28, avg= 8.29, stdev= 4.74, samples=290 lat (msec) : 20=0.04%, 50=0.23%, 100=0.04%, 250=0.43%, 500=20.00% lat (msec) : 750=24.10%, 1000=18.98%, 2000=33.55%, >=2000=2.62% cpu : usr=0.10%, sys=0.56%, ctx=4381, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=146230: Sun Aug 23 02:13:43 2026 read: IOPS=8, BW=34.3KiB/s (35.1kB/s)(5060KiB/147643msec) slat (usec): min=3103, max=99218, avg=18494.62, stdev=11364.57 clat (msec): min=14, max=1398, avg=574.57, stdev=207.59 lat (msec): min=21, max=1421, avg=593.07, stdev=210.27 clat percentiles (msec): | 1.00th=[ 140], 5.00th=[ 313], 10.00th=[ 351], 20.00th=[ 409], | 30.00th=[ 451], 40.00th=[ 493], 50.00th=[ 531], 60.00th=[ 584], | 70.00th=[ 651], 80.00th=[ 760], 90.00th=[ 860], 95.00th=[ 961], | 99.00th=[ 1150], 99.50th=[ 1200], 99.90th=[ 1301], 99.95th=[ 1401], | 99.99th=[ 1401] bw ( KiB/s): min= 7, max= 119, per=26.24%, avg=35.42, stdev=19.01, samples=279 iops : min= 1, max= 29, avg= 8.35, stdev= 4.78, samples=279 write: IOPS=8, BW=35.1KiB/s (35.9kB/s)(5180KiB/147643msec); 0 zone resets slat (usec): min=31, max=31382, avg=361.77, stdev=1493.97 clat (msec): min=276, max=3054, avg=1241.08, stdev=428.55 lat (msec): min=277, max=3054, avg=1241.45, stdev=428.62 clat percentiles (msec): | 1.00th=[ 567], 5.00th=[ 659], 10.00th=[ 760], 20.00th=[ 894], | 30.00th=[ 969], 40.00th=[ 1099], 50.00th=[ 1183], 60.00th=[ 1301], | 70.00th=[ 1401], 80.00th=[ 1536], 90.00th=[ 1787], 95.00th=[ 2022], | 99.00th=[ 2735], 99.50th=[ 2869], 99.90th=[ 3004], 99.95th=[ 3071], | 99.99th=[ 3071] bw ( KiB/s): min= 7, max= 88, per=25.40%, avg=35.55, stdev=19.48, samples=283 iops : min= 1, max= 22, avg= 8.38, stdev= 4.92, samples=283 lat (msec) : 20=0.16%, 50=0.08%, 100=0.04%, 250=0.94%, 500=19.96% lat (msec) : 750=23.01%, 1000=20.00%, 2000=32.89%, >=2000=2.93% cpu : usr=0.09%, sys=0.55%, ctx=4487, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=135KiB/s (139kB/s), 33.6KiB/s-34.3KiB/s (34.4kB/s-35.1kB/s), io=19.6MiB (20.5MB), run=147262-148270msec WRITE: bw=141KiB/s (144kB/s), 35.1KiB/s-35.5KiB/s (35.9kB/s-36.4kB/s), io=20.4MiB (21.4MB), run=147262-148270msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=146258: Sun Aug 23 02:13:57 2026 read: IOPS=1, BW=56.3MiB/s (59.1MB/s)(40.0MiB/710msec) slat (nsec): min=53563k, max=53563k, avg=53563379.00, stdev= 0.00 clat (nsec): min=634876k, max=634876k, avg=634876377.00, stdev= 0.00 lat (nsec): min=688445k, max=688445k, avg=688444674.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 634], 5.00th=[ 634], 10.00th=[ 634], 20.00th=[ 634], | 30.00th=[ 634], 40.00th=[ 634], 50.00th=[ 634], 60.00th=[ 634], | 70.00th=[ 634], 80.00th=[ 634], 90.00th=[ 634], 95.00th=[ 634], | 99.00th=[ 634], 99.50th=[ 634], 99.90th=[ 634], 99.95th=[ 634], | 99.99th=[ 634] lat (msec) : 750=100.00% cpu : usr=0.00%, sys=1.27%, ctx=36, majf=0, minf=7 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=56.3MiB/s (59.1MB/s), 56.3MiB/s-56.3MiB/s (59.1MB/s-59.1MB/s), io=40.0MiB (41.9MB), run=710-710msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout