== sanity test 398c: run fio to test AIO ================= 03:48:48 (1787471328) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.47383 s, 88.5 MB/s osc.lustre-OST0000-osc-ffff8d4843e60000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=144635: Sun Aug 23 03:51:55 2026 write: IOPS=14, BW=58.2KiB/s (59.6kB/s)(10.0MiB/176076msec); 0 zone resets slat (usec): min=51, max=38867, avg=460.43, stdev=1592.03 clat (msec): min=153, max=2022, avg=1099.51, stdev=284.09 lat (msec): min=153, max=2022, avg=1099.98, stdev=284.06 clat percentiles (msec): | 1.00th=[ 279], 5.00th=[ 684], 10.00th=[ 776], 20.00th=[ 885], | 30.00th=[ 961], 40.00th=[ 1020], 50.00th=[ 1070], 60.00th=[ 1150], | 70.00th=[ 1234], 80.00th=[ 1334], 90.00th=[ 1485], 95.00th=[ 1586], | 99.00th=[ 1821], 99.50th=[ 1854], 99.90th=[ 1905], 99.95th=[ 1921], | 99.99th=[ 2022] bw ( KiB/s): min= 7, max= 215, per=25.28%, avg=58.65, stdev=30.57, samples=342 iops : min= 1, max= 53, avg=14.13, stdev= 7.66, samples=342 lat (msec) : 250=0.70%, 500=1.17%, 750=6.99%, 1000=28.59%, 2000=62.50% lat (msec) : >=2000=0.04% cpu : usr=0.06%, sys=0.31%, ctx=1993, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=144636: Sun Aug 23 03:51:55 2026 write: IOPS=14, BW=58.1KiB/s (59.5kB/s)(10.0MiB/176238msec); 0 zone resets slat (usec): min=49, max=24372, avg=374.27, stdev=1052.35 clat (msec): min=104, max=1957, avg=1100.03, stdev=279.60 lat (msec): min=104, max=1958, avg=1100.41, stdev=279.58 clat percentiles (msec): | 1.00th=[ 300], 5.00th=[ 667], 10.00th=[ 751], 20.00th=[ 877], | 30.00th=[ 961], 40.00th=[ 1028], 50.00th=[ 1099], 60.00th=[ 1167], | 70.00th=[ 1250], 80.00th=[ 1334], 90.00th=[ 1452], 95.00th=[ 1586], | 99.00th=[ 1737], 99.50th=[ 1821], 99.90th=[ 1888], 99.95th=[ 1888], | 99.99th=[ 1955] bw ( KiB/s): min= 7, max= 154, per=25.23%, avg=58.53, stdev=33.62, samples=343 iops : min= 1, max= 38, avg=14.10, stdev= 8.39, samples=343 lat (msec) : 250=0.55%, 500=0.94%, 750=8.32%, 1000=25.59%, 2000=64.61% cpu : usr=0.06%, sys=0.30%, ctx=2089, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=144637: Sun Aug 23 03:51:55 2026 write: IOPS=14, BW=58.0KiB/s (59.4kB/s)(10.0MiB/176457msec); 0 zone resets slat (usec): min=52, max=25652, avg=388.13, stdev=1153.78 clat (msec): min=216, max=1913, avg=1101.26, stdev=267.89 lat (msec): min=218, max=1913, avg=1101.65, stdev=267.89 clat percentiles (msec): | 1.00th=[ 309], 5.00th=[ 667], 10.00th=[ 776], 20.00th=[ 902], | 30.00th=[ 961], 40.00th=[ 1020], 50.00th=[ 1083], 60.00th=[ 1150], | 70.00th=[ 1234], 80.00th=[ 1334], 90.00th=[ 1469], 95.00th=[ 1536], | 99.00th=[ 1720], 99.50th=[ 1787], 99.90th=[ 1838], 99.95th=[ 1838], | 99.99th=[ 1921] bw ( KiB/s): min= 7, max= 128, per=25.69%, avg=59.61, stdev=30.85, samples=337 iops : min= 1, max= 32, avg=14.37, stdev= 7.73, samples=337 lat (msec) : 250=0.62%, 500=0.62%, 750=7.19%, 1000=28.40%, 2000=63.16% cpu : usr=0.04%, sys=0.29%, ctx=1709, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=144638: Sun Aug 23 03:51:55 2026 write: IOPS=14, BW=58.0KiB/s (59.4kB/s)(10.0MiB/176458msec); 0 zone resets slat (usec): min=51, max=33436, avg=376.44, stdev=1171.91 clat (msec): min=199, max=2020, avg=1101.19, stdev=272.31 lat (msec): min=200, max=2020, avg=1101.57, stdev=272.29 clat percentiles (msec): | 1.00th=[ 313], 5.00th=[ 667], 10.00th=[ 785], 20.00th=[ 885], | 30.00th=[ 961], 40.00th=[ 1020], 50.00th=[ 1083], 60.00th=[ 1167], | 70.00th=[ 1234], 80.00th=[ 1334], 90.00th=[ 1469], 95.00th=[ 1552], | 99.00th=[ 1720], 99.50th=[ 1821], 99.90th=[ 1888], 99.95th=[ 1955], | 99.99th=[ 2022] bw ( KiB/s): min= 7, max= 128, per=24.73%, avg=57.36, stdev=25.97, samples=350 iops : min= 1, max= 32, avg=13.81, stdev= 6.48, samples=350 lat (msec) : 250=0.62%, 500=0.62%, 750=7.46%, 1000=28.40%, 2000=62.85% lat (msec) : >=2000=0.04% cpu : usr=0.07%, sys=0.30%, ctx=2022, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=232KiB/s (238kB/s), 58.0KiB/s-58.2KiB/s (59.4kB/s-59.6kB/s), io=40.0MiB (41.9MB), run=176076-176458msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=144676: Sun Aug 23 03:54:15 2026 read: IOPS=9, BW=38.7KiB/s (39.6kB/s)(5048KiB/130431msec) slat (usec): min=3255, max=90734, avg=16773.51, stdev=9314.65 clat (msec): min=30, max=1050, avg=506.98, stdev=140.65 lat (msec): min=42, max=1067, avg=523.76, stdev=142.85 clat percentiles (msec): | 1.00th=[ 203], 5.00th=[ 309], 10.00th=[ 347], 20.00th=[ 388], | 30.00th=[ 430], 40.00th=[ 464], 50.00th=[ 493], 60.00th=[ 527], | 70.00th=[ 567], 80.00th=[ 609], 90.00th=[ 684], 95.00th=[ 776], | 99.00th=[ 911], 99.50th=[ 953], 99.90th=[ 1045], 99.95th=[ 1053], | 99.99th=[ 1053] bw ( KiB/s): min= 7, max= 95, per=24.95%, avg=38.18, stdev=18.67, samples=258 iops : min= 1, max= 23, avg= 8.97, stdev= 4.66, samples=258 write: IOPS=9, BW=39.8KiB/s (40.8kB/s)(5192KiB/130431msec); 0 zone resets slat (usec): min=54, max=18448, avg=594.85, stdev=1501.20 clat (msec): min=427, max=2110, avg=1095.76, stdev=297.60 lat (msec): min=428, max=2110, avg=1096.36, stdev=297.56 clat percentiles (msec): | 1.00th=[ 514], 5.00th=[ 659], 10.00th=[ 735], 20.00th=[ 827], | 30.00th=[ 911], 40.00th=[ 995], 50.00th=[ 1099], 60.00th=[ 1150], | 70.00th=[ 1234], 80.00th=[ 1334], 90.00th=[ 1485], 95.00th=[ 1620], | 99.00th=[ 1905], 99.50th=[ 1989], 99.90th=[ 2106], 99.95th=[ 2106], | 99.99th=[ 2106] bw ( KiB/s): min= 7, max= 95, per=25.02%, avg=39.78, stdev=17.93, samples=254 iops : min= 1, max= 23, avg= 9.37, stdev= 4.51, samples=254 lat (msec) : 50=0.08%, 100=0.04%, 250=0.59%, 500=25.31%, 750=26.48% lat (msec) : 1000=17.15%, 2000=30.16%, >=2000=0.20% cpu : usr=0.10%, sys=0.71%, ctx=4619, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=144677: Sun Aug 23 03:54:15 2026 read: IOPS=9, BW=38.1KiB/s (38.0kB/s)(4976KiB/130769msec) slat (usec): min=103, max=91510, avg=16914.17, stdev=9894.04 clat (msec): min=10, max=1103, avg=502.73, stdev=148.36 lat (msec): min=17, max=1130, avg=519.66, stdev=151.18 clat percentiles (msec): | 1.00th=[ 106], 5.00th=[ 313], 10.00th=[ 342], 20.00th=[ 384], | 30.00th=[ 426], 40.00th=[ 451], 50.00th=[ 485], 60.00th=[ 523], | 70.00th=[ 567], 80.00th=[ 617], 90.00th=[ 693], 95.00th=[ 768], | 99.00th=[ 944], 99.50th=[ 986], 99.90th=[ 1036], 99.95th=[ 1099], | 99.99th=[ 1099] bw ( KiB/s): min= 7, max= 96, per=24.74%, avg=37.85, stdev=19.35, samples=256 iops : min= 1, max= 24, avg= 8.88, stdev= 4.83, samples=256 write: IOPS=10, BW=40.3KiB/s (41.2kB/s)(5264KiB/130769msec); 0 zone resets slat (usec): min=56, max=11984, avg=551.17, stdev=1206.32 clat (msec): min=138, max=2055, avg=1097.32, stdev=306.81 lat (msec): min=142, max=2056, avg=1097.88, stdev=306.76 clat percentiles (msec): | 1.00th=[ 456], 5.00th=[ 642], 10.00th=[ 726], 20.00th=[ 835], | 30.00th=[ 911], 40.00th=[ 1003], 50.00th=[ 1099], 60.00th=[ 1150], | 70.00th=[ 1234], 80.00th=[ 1334], 90.00th=[ 1519], 95.00th=[ 1653], | 99.00th=[ 1905], 99.50th=[ 1955], 99.90th=[ 2022], 99.95th=[ 2056], | 99.99th=[ 2056] bw ( KiB/s): min= 7, max= 95, per=25.10%, avg=39.91, stdev=18.68, samples=257 iops : min= 1, max= 23, avg= 9.40, stdev= 4.65, samples=257 lat (msec) : 20=0.16%, 50=0.12%, 100=0.16%, 250=0.43%, 500=26.56% lat (msec) : 750=24.80%, 1000=16.45%, 2000=31.25%, >=2000=0.08% cpu : usr=0.09%, sys=0.74%, ctx=4498, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=144678: Sun Aug 23 03:54:15 2026 read: IOPS=9, BW=38.1KiB/s (39.0kB/s)(4984KiB/130762msec) slat (usec): min=3454, max=69617, avg=17200.67, stdev=9413.43 clat (msec): min=10, max=1067, avg=504.29, stdev=135.79 lat (msec): min=16, max=1094, avg=521.50, stdev=138.01 clat percentiles (msec): | 1.00th=[ 234], 5.00th=[ 313], 10.00th=[ 347], 20.00th=[ 393], | 30.00th=[ 430], 40.00th=[ 460], 50.00th=[ 489], 60.00th=[ 527], | 70.00th=[ 567], 80.00th=[ 609], 90.00th=[ 684], 95.00th=[ 735], | 99.00th=[ 911], 99.50th=[ 961], 99.90th=[ 1020], 99.95th=[ 1070], | 99.99th=[ 1070] bw ( KiB/s): min= 7, max= 88, per=24.84%, avg=38.00, stdev=19.07, samples=256 iops : min= 1, max= 22, avg= 8.93, stdev= 4.79, samples=256 write: IOPS=10, BW=40.2KiB/s (41.2kB/s)(5256KiB/130762msec); 0 zone resets slat (usec): min=56, max=16051, avg=541.48, stdev=1318.50 clat (msec): min=421, max=2118, avg=1095.02, stdev=299.54 lat (msec): min=421, max=2118, avg=1095.57, stdev=299.53 clat percentiles (msec): | 1.00th=[ 542], 5.00th=[ 667], 10.00th=[ 718], 20.00th=[ 827], | 30.00th=[ 911], 40.00th=[ 1003], 50.00th=[ 1083], 60.00th=[ 1150], | 70.00th=[ 1217], 80.00th=[ 1318], 90.00th=[ 1502], 95.00th=[ 1653], | 99.00th=[ 1888], 99.50th=[ 1955], 99.90th=[ 2056], 99.95th=[ 2123], | 99.99th=[ 2123] bw ( KiB/s): min= 7, max= 88, per=25.01%, avg=39.77, stdev=18.55, samples=257 iops : min= 1, max= 22, avg= 9.37, stdev= 4.65, samples=257 lat (msec) : 20=0.04%, 50=0.04%, 100=0.08%, 250=0.47%, 500=25.27% lat (msec) : 750=27.07%, 1000=16.02%, 2000=30.90%, >=2000=0.12% cpu : usr=0.10%, sys=0.74%, ctx=4692, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=144679: Sun Aug 23 03:54:15 2026 read: IOPS=9, BW=38.8KiB/s (39.7kB/s)(5060KiB/130580msec) slat (usec): min=3414, max=77837, avg=17367.22, stdev=9494.94 clat (msec): min=33, max=1118, avg=513.82, stdev=145.00 lat (msec): min=38, max=1129, avg=531.20, stdev=147.33 clat percentiles (msec): | 1.00th=[ 205], 5.00th=[ 321], 10.00th=[ 351], 20.00th=[ 393], | 30.00th=[ 435], 40.00th=[ 464], 50.00th=[ 502], 60.00th=[ 535], | 70.00th=[ 575], 80.00th=[ 617], 90.00th=[ 701], 95.00th=[ 785], | 99.00th=[ 936], 99.50th=[ 995], 99.90th=[ 1053], 99.95th=[ 1116], | 99.99th=[ 1116] bw ( KiB/s): min= 7, max= 95, per=25.15%, avg=38.48, stdev=18.40, samples=256 iops : min= 1, max= 23, avg= 9.04, stdev= 4.61, samples=256 write: IOPS=9, BW=39.7KiB/s (40.6kB/s)(5180KiB/130580msec); 0 zone resets slat (usec): min=56, max=19966, avg=523.69, stdev=1350.92 clat (msec): min=457, max=2078, avg=1091.75, stdev=292.58 lat (msec): min=458, max=2078, avg=1092.28, stdev=292.57 clat percentiles (msec): | 1.00th=[ 531], 5.00th=[ 667], 10.00th=[ 726], 20.00th=[ 810], | 30.00th=[ 902], 40.00th=[ 995], 50.00th=[ 1099], 60.00th=[ 1150], | 70.00th=[ 1234], 80.00th=[ 1334], 90.00th=[ 1485], 95.00th=[ 1603], | 99.00th=[ 1871], 99.50th=[ 1972], 99.90th=[ 2022], 99.95th=[ 2072], | 99.99th=[ 2072] bw ( KiB/s): min= 7, max= 95, per=24.72%, avg=39.31, stdev=18.24, samples=257 iops : min= 1, max= 23, avg= 9.25, stdev= 4.58, samples=257 lat (msec) : 50=0.12%, 100=0.16%, 250=0.27%, 500=24.26%, 750=27.15% lat (msec) : 1000=17.81%, 2000=30.08%, >=2000=0.16% cpu : usr=0.11%, sys=0.71%, ctx=4744, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=153KiB/s (157kB/s), 38.1KiB/s-38.8KiB/s (38.0kB/s-39.7kB/s), io=19.6MiB (20.5MB), run=130431-130769msec WRITE: bw=160KiB/s (164kB/s), 39.7KiB/s-40.3KiB/s (40.6kB/s-41.2kB/s), io=20.4MiB (21.4MB), run=130431-130769msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=144702: Sun Aug 23 03:54:25 2026 read: IOPS=1, BW=62.7MiB/s (65.7MB/s)(40.0MiB/638msec) slat (nsec): min=34699k, max=34699k, avg=34699264.00, stdev= 0.00 clat (nsec): min=549433k, max=549433k, avg=549433398.00, stdev= 0.00 lat (nsec): min=584162k, max=584162k, avg=584162137.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 550], 5.00th=[ 550], 10.00th=[ 550], 20.00th=[ 550], | 30.00th=[ 550], 40.00th=[ 550], 50.00th=[ 550], 60.00th=[ 550], | 70.00th=[ 550], 80.00th=[ 550], 90.00th=[ 550], 95.00th=[ 550], | 99.00th=[ 550], 99.50th=[ 550], 99.90th=[ 550], 99.95th=[ 550], | 99.99th=[ 550] lat (msec) : 750=100.00% cpu : usr=0.00%, sys=1.73%, ctx=5, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=62.7MiB/s (65.7MB/s), 62.7MiB/s-62.7MiB/s (65.7MB/s-65.7MB/s), io=40.0MiB (41.9MB), run=638-638msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout