== sanity test 398c: run fio to test AIO ================= 05:55:10 (1781258110) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.510241 s, 82.2 MB/s osc.lustre-OST0000-osc-ffff9b1dc8517000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=125002: Fri Jun 12 05:57:53 2026 write: IOPS=16, BW=66.8KiB/s (68.4kB/s)(10.0MiB/153198msec); 0 zone resets slat (usec): min=26, max=31600, avg=313.25, stdev=1259.52 clat (msec): min=374, max=2004, avg=956.61, stdev=241.39 lat (msec): min=375, max=2005, avg=956.92, stdev=241.38 clat percentiles (msec): | 1.00th=[ 405], 5.00th=[ 584], 10.00th=[ 709], 20.00th=[ 776], | 30.00th=[ 827], 40.00th=[ 877], 50.00th=[ 927], 60.00th=[ 986], | 70.00th=[ 1053], 80.00th=[ 1150], 90.00th=[ 1267], 95.00th=[ 1385], | 99.00th=[ 1620], 99.50th=[ 1703], 99.90th=[ 1905], 99.95th=[ 2005], | 99.99th=[ 2005] bw ( KiB/s): min= 7, max= 215, per=24.77%, avg=66.15, stdev=27.46, samples=304 iops : min= 1, max= 53, avg=16.06, stdev= 6.88, samples=304 lat (msec) : 500=3.20%, 750=12.70%, 1000=46.41%, 2000=37.62%, >=2000=0.08% cpu : usr=0.05%, sys=0.30%, ctx=1979, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=125003: Fri Jun 12 05:57:53 2026 write: IOPS=16, BW=66.9KiB/s (68.5kB/s)(10.0MiB/153118msec); 0 zone resets slat (usec): min=25, max=20837, avg=304.94, stdev=1142.77 clat (msec): min=388, max=2007, avg=956.16, stdev=238.61 lat (msec): min=388, max=2007, avg=956.46, stdev=238.61 clat percentiles (msec): | 1.00th=[ 397], 5.00th=[ 584], 10.00th=[ 701], 20.00th=[ 776], | 30.00th=[ 827], 40.00th=[ 877], 50.00th=[ 927], 60.00th=[ 986], | 70.00th=[ 1053], 80.00th=[ 1150], 90.00th=[ 1267], 95.00th=[ 1351], | 99.00th=[ 1636], 99.50th=[ 1737], 99.90th=[ 1955], 99.95th=[ 2005], | 99.99th=[ 2005] bw ( KiB/s): min= 7, max= 255, per=25.28%, avg=67.50, stdev=28.54, samples=298 iops : min= 1, max= 63, avg=16.39, stdev= 7.15, samples=298 lat (msec) : 500=2.70%, 750=13.28%, 1000=45.74%, 2000=38.20%, >=2000=0.08% cpu : usr=0.06%, sys=0.28%, ctx=1772, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=125004: Fri Jun 12 05:57:53 2026 write: IOPS=16, BW=66.0KiB/s (68.6kB/s)(10.0MiB/152867msec); 0 zone resets slat (usec): min=26, max=17028, avg=311.56, stdev=1087.97 clat (msec): min=297, max=1945, avg=954.69, stdev=241.41 lat (msec): min=297, max=1945, avg=955.00, stdev=241.44 clat percentiles (msec): | 1.00th=[ 384], 5.00th=[ 600], 10.00th=[ 693], 20.00th=[ 776], | 30.00th=[ 827], 40.00th=[ 877], 50.00th=[ 927], 60.00th=[ 1003], | 70.00th=[ 1062], 80.00th=[ 1150], 90.00th=[ 1250], 95.00th=[ 1368], | 99.00th=[ 1636], 99.50th=[ 1720], 99.90th=[ 1854], 99.95th=[ 1938], | 99.99th=[ 1938] bw ( KiB/s): min= 15, max= 256, per=25.37%, avg=67.73, stdev=27.74, samples=297 iops : min= 3, max= 64, avg=16.45, stdev= 6.96, samples=297 lat (msec) : 500=2.93%, 750=13.71%, 1000=43.55%, 2000=39.80% cpu : usr=0.05%, sys=0.29%, ctx=1747, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=125005: Fri Jun 12 05:57:53 2026 write: IOPS=16, BW=66.0KiB/s (68.6kB/s)(10.0MiB/152868msec); 0 zone resets slat (usec): min=22, max=18423, avg=317.23, stdev=1041.24 clat (msec): min=86, max=2005, avg=954.71, stdev=249.10 lat (msec): min=86, max=2005, avg=955.03, stdev=249.09 clat percentiles (msec): | 1.00th=[ 338], 5.00th=[ 584], 10.00th=[ 693], 20.00th=[ 776], | 30.00th=[ 827], 40.00th=[ 877], 50.00th=[ 936], 60.00th=[ 986], | 70.00th=[ 1062], 80.00th=[ 1150], 90.00th=[ 1267], 95.00th=[ 1385], | 99.00th=[ 1620], 99.50th=[ 1720], 99.90th=[ 1871], 99.95th=[ 2005], | 99.99th=[ 2005] bw ( KiB/s): min= 7, max= 184, per=24.81%, avg=66.24, stdev=30.30, samples=303 iops : min= 1, max= 46, avg=16.08, stdev= 7.61, samples=303 lat (msec) : 100=0.16%, 250=0.39%, 500=3.01%, 750=13.20%, 1000=44.77% lat (msec) : 2000=38.40%, >=2000=0.08% cpu : usr=0.06%, sys=0.27%, ctx=1812, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=267KiB/s (274kB/s), 66.8KiB/s-66.0KiB/s (68.4kB/s-68.6kB/s), io=40.0MiB (41.9MB), run=152867-153198msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=125037: Fri Jun 12 05:59:47 2026 read: IOPS=11, BW=47.8KiB/s (48.0kB/s)(5048KiB/105598msec) slat (usec): min=1597, max=51182, avg=11972.02, stdev=6919.26 clat (msec): min=13, max=1244, avg=423.85, stdev=120.16 lat (msec): min=20, max=1287, avg=435.83, stdev=121.33 clat percentiles (msec): | 1.00th=[ 222], 5.00th=[ 268], 10.00th=[ 292], 20.00th=[ 326], | 30.00th=[ 351], 40.00th=[ 376], 50.00th=[ 405], 60.00th=[ 443], | 70.00th=[ 481], 80.00th=[ 518], 90.00th=[ 584], 95.00th=[ 625], | 99.00th=[ 743], 99.50th=[ 818], 99.90th=[ 1133], 99.95th=[ 1250], | 99.99th=[ 1250] bw ( KiB/s): min= 7, max= 118, per=25.14%, avg=47.76, stdev=21.87, samples=207 iops : min= 1, max= 29, avg=11.44, stdev= 5.47, samples=207 write: IOPS=12, BW=49.2KiB/s (50.3kB/s)(5192KiB/105598msec); 0 zone resets slat (usec): min=24, max=27001, avg=461.97, stdev=1747.71 clat (msec): min=307, max=1958, avg=875.62, stdev=285.02 lat (msec): min=308, max=1958, avg=876.08, stdev=284.97 clat percentiles (msec): | 1.00th=[ 397], 5.00th=[ 502], 10.00th=[ 550], 20.00th=[ 617], | 30.00th=[ 701], 40.00th=[ 768], 50.00th=[ 835], 60.00th=[ 911], | 70.00th=[ 978], 80.00th=[ 1116], 90.00th=[ 1267], 95.00th=[ 1435], | 99.00th=[ 1703], 99.50th=[ 1737], 99.90th=[ 1854], 99.95th=[ 1955], | 99.99th=[ 1955] bw ( KiB/s): min= 7, max= 112, per=24.83%, avg=48.91, stdev=20.69, samples=208 iops : min= 1, max= 28, avg=11.73, stdev= 5.23, samples=208 lat (msec) : 20=0.04%, 50=0.04%, 100=0.04%, 250=1.33%, 500=38.09% lat (msec) : 750=27.89%, 1000=18.52%, 2000=14.06% cpu : usr=0.12%, sys=0.74%, ctx=3312, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=125038: Fri Jun 12 05:59:47 2026 read: IOPS=11, BW=47.1KiB/s (48.3kB/s)(4976KiB/105599msec) slat (usec): min=2354, max=51656, avg=12024.40, stdev=7206.35 clat (msec): min=15, max=1241, avg=424.96, stdev=136.67 lat (msec): min=28, max=1278, avg=437.00, stdev=138.02 clat percentiles (msec): | 1.00th=[ 192], 5.00th=[ 255], 10.00th=[ 279], 20.00th=[ 317], | 30.00th=[ 351], 40.00th=[ 376], 50.00th=[ 405], 60.00th=[ 439], | 70.00th=[ 481], 80.00th=[ 518], 90.00th=[ 592], 95.00th=[ 651], | 99.00th=[ 927], 99.50th=[ 1028], 99.90th=[ 1116], 99.95th=[ 1234], | 99.99th=[ 1234] bw ( KiB/s): min= 7, max= 111, per=24.63%, avg=46.80, stdev=22.95, samples=208 iops : min= 1, max= 27, avg=11.20, stdev= 5.76, samples=208 write: IOPS=12, BW=49.8KiB/s (51.0kB/s)(5264KiB/105599msec); 0 zone resets slat (usec): min=25, max=16580, avg=369.86, stdev=1167.01 clat (msec): min=264, max=1896, avg=868.74, stdev=277.23 lat (msec): min=265, max=1897, avg=869.11, stdev=277.22 clat percentiles (msec): | 1.00th=[ 359], 5.00th=[ 502], 10.00th=[ 550], 20.00th=[ 617], | 30.00th=[ 693], 40.00th=[ 760], 50.00th=[ 827], 60.00th=[ 919], | 70.00th=[ 986], 80.00th=[ 1099], 90.00th=[ 1250], 95.00th=[ 1418], | 99.00th=[ 1636], 99.50th=[ 1703], 99.90th=[ 1754], 99.95th=[ 1905], | 99.99th=[ 1905] bw ( KiB/s): min= 7, max= 120, per=25.20%, avg=49.65, stdev=22.40, samples=208 iops : min= 1, max= 30, avg=11.91, stdev= 5.65, samples=208 lat (msec) : 20=0.04%, 50=0.08%, 100=0.20%, 250=1.80%, 500=37.23% lat (msec) : 750=28.09%, 1000=17.93%, 2000=14.65% cpu : usr=0.11%, sys=0.73%, ctx=3293, majf=0, minf=16 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=125039: Fri Jun 12 05:59:47 2026 read: IOPS=11, BW=47.2KiB/s (48.4kB/s)(4984KiB/105487msec) slat (usec): min=32, max=78885, avg=11739.39, stdev=7131.09 clat (msec): min=9, max=1245, avg=421.70, stdev=129.78 lat (msec): min=9, max=1279, avg=433.45, stdev=131.37 clat percentiles (msec): | 1.00th=[ 144], 5.00th=[ 249], 10.00th=[ 284], 20.00th=[ 321], | 30.00th=[ 351], 40.00th=[ 376], 50.00th=[ 409], 60.00th=[ 443], | 70.00th=[ 477], 80.00th=[ 510], 90.00th=[ 584], 95.00th=[ 642], | 99.00th=[ 818], 99.50th=[ 894], 99.90th=[ 1083], 99.95th=[ 1250], | 99.99th=[ 1250] bw ( KiB/s): min= 7, max= 135, per=24.65%, avg=46.84, stdev=25.27, samples=209 iops : min= 1, max= 33, avg=11.21, stdev= 6.36, samples=209 write: IOPS=12, BW=49.8KiB/s (51.0kB/s)(5256KiB/105487msec); 0 zone resets slat (usec): min=27, max=18672, avg=404.56, stdev=1393.37 clat (msec): min=121, max=1954, avg=872.31, stdev=283.53 lat (msec): min=121, max=1954, avg=872.72, stdev=283.50 clat percentiles (msec): | 1.00th=[ 330], 5.00th=[ 510], 10.00th=[ 558], 20.00th=[ 634], | 30.00th=[ 701], 40.00th=[ 768], 50.00th=[ 827], 60.00th=[ 919], | 70.00th=[ 986], 80.00th=[ 1099], 90.00th=[ 1267], 95.00th=[ 1418], | 99.00th=[ 1670], 99.50th=[ 1754], 99.90th=[ 1854], 99.95th=[ 1955], | 99.99th=[ 1955] bw ( KiB/s): min= 7, max= 102, per=24.92%, avg=49.10, stdev=21.27, samples=209 iops : min= 1, max= 25, avg=11.78, stdev= 5.36, samples=209 lat (msec) : 10=0.08%, 50=0.12%, 100=0.20%, 250=2.46%, 500=37.19% lat (msec) : 750=26.99%, 1000=18.05%, 2000=14.92% cpu : usr=0.13%, sys=0.71%, ctx=3360, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=125040: Fri Jun 12 05:59:47 2026 read: IOPS=11, BW=47.9KiB/s (49.1kB/s)(5060KiB/105536msec) slat (usec): min=2258, max=74357, avg=12055.12, stdev=7458.76 clat (msec): min=24, max=1242, avg=423.75, stdev=120.00 lat (msec): min=30, max=1249, avg=435.82, stdev=121.43 clat percentiles (msec): | 1.00th=[ 197], 5.00th=[ 264], 10.00th=[ 288], 20.00th=[ 326], | 30.00th=[ 355], 40.00th=[ 380], 50.00th=[ 405], 60.00th=[ 435], | 70.00th=[ 477], 80.00th=[ 514], 90.00th=[ 584], 95.00th=[ 634], | 99.00th=[ 802], 99.50th=[ 844], 99.90th=[ 1053], 99.95th=[ 1250], | 99.99th=[ 1250] bw ( KiB/s): min= 7, max= 134, per=25.07%, avg=47.62, stdev=24.25, samples=208 iops : min= 1, max= 33, avg=11.40, stdev= 6.08, samples=208 write: IOPS=12, BW=49.1KiB/s (50.3kB/s)(5180KiB/105536msec); 0 zone resets slat (usec): min=28, max=18399, avg=396.04, stdev=1330.72 clat (msec): min=251, max=1841, avg=876.49, stdev=287.21 lat (msec): min=251, max=1841, avg=876.89, stdev=287.25 clat percentiles (msec): | 1.00th=[ 326], 5.00th=[ 498], 10.00th=[ 550], 20.00th=[ 634], | 30.00th=[ 701], 40.00th=[ 760], 50.00th=[ 827], 60.00th=[ 902], | 70.00th=[ 986], 80.00th=[ 1133], 90.00th=[ 1267], 95.00th=[ 1435], | 99.00th=[ 1653], 99.50th=[ 1720], 99.90th=[ 1770], 99.95th=[ 1838], | 99.99th=[ 1838] bw ( KiB/s): min= 7, max= 103, per=24.54%, avg=48.35, stdev=19.52, samples=210 iops : min= 1, max= 25, avg=11.59, stdev= 4.92, samples=210 lat (msec) : 50=0.04%, 250=1.48%, 500=38.55%, 750=28.01%, 1000=17.23% lat (msec) : 2000=14.69% cpu : usr=0.11%, sys=0.66%, ctx=3250, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=190KiB/s (195kB/s), 47.1KiB/s-47.9KiB/s (48.3kB/s-49.1kB/s), io=19.6MiB (20.5MB), run=105487-105599msec WRITE: bw=198KiB/s (203kB/s), 49.1KiB/s-49.8KiB/s (50.3kB/s-51.0kB/s), io=20.4MiB (21.4MB), run=105487-105599msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=125060: Fri Jun 12 05:59:57 2026 read: IOPS=1, BW=74.1MiB/s (77.7MB/s)(40.0MiB/540msec) slat (nsec): min=61011k, max=61011k, avg=61010743.00, stdev= 0.00 clat (nsec): min=453001k, max=453001k, avg=453001080.00, stdev= 0.00 lat (nsec): min=514016k, max=514016k, avg=514016233.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 456], 5.00th=[ 456], 10.00th=[ 456], 20.00th=[ 456], | 30.00th=[ 456], 40.00th=[ 456], 50.00th=[ 456], 60.00th=[ 456], | 70.00th=[ 456], 80.00th=[ 456], 90.00th=[ 456], 95.00th=[ 456], | 99.00th=[ 456], 99.50th=[ 456], 99.90th=[ 456], 99.95th=[ 456], | 99.99th=[ 456] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=2.78%, ctx=4, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=74.1MiB/s (77.7MB/s), 74.1MiB/s-74.1MiB/s (77.7MB/s-77.7MB/s), io=40.0MiB (41.9MB), run=540-540msec debug=super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck