== sanity test 398c: run fio to test AIO ================= 02:39:18 (1788849558) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.279442 s, 150 MB/s osc.lustre-OST0000-osc-ffff9fb689a44000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=146153: Tue Sep 8 02:41:23 2026 write: IOPS=21, BW=86.8KiB/s (88.9kB/s)(10.0MiB/117943msec); 0 zone resets slat (usec): min=21, max=15777, avg=200.28, stdev=789.86 clat (msec): min=107, max=1374, avg=736.61, stdev=193.54 lat (msec): min=107, max=1375, avg=736.81, stdev=193.57 clat percentiles (msec): | 1.00th=[ 218], 5.00th=[ 439], 10.00th=[ 518], 20.00th=[ 600], | 30.00th=[ 642], 40.00th=[ 693], 50.00th=[ 726], 60.00th=[ 768], | 70.00th=[ 818], 80.00th=[ 877], 90.00th=[ 978], 95.00th=[ 1083], | 99.00th=[ 1234], 99.50th=[ 1284], 99.90th=[ 1334], 99.95th=[ 1368], | 99.99th=[ 1368] bw ( KiB/s): min= 7, max= 255, per=25.14%, avg=87.00, stdev=37.23, samples=232 iops : min= 1, max= 63, avg=21.33, stdev= 9.31, samples=232 lat (msec) : 250=1.88%, 500=7.19%, 750=45.31%, 1000=36.88%, 2000=8.75% cpu : usr=0.05%, sys=0.30%, ctx=1543, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=146154: Tue Sep 8 02:41:23 2026 write: IOPS=21, BW=86.6KiB/s (88.7kB/s)(10.0MiB/118198msec); 0 zone resets slat (usec): min=22, max=12540, avg=200.34, stdev=678.49 clat (msec): min=185, max=1351, avg=737.84, stdev=187.47 lat (msec): min=185, max=1351, avg=738.04, stdev=187.46 clat percentiles (msec): | 1.00th=[ 271], 5.00th=[ 439], 10.00th=[ 514], 20.00th=[ 600], | 30.00th=[ 642], 40.00th=[ 693], 50.00th=[ 735], 60.00th=[ 776], | 70.00th=[ 818], 80.00th=[ 877], 90.00th=[ 978], 95.00th=[ 1083], | 99.00th=[ 1234], 99.50th=[ 1284], 99.90th=[ 1318], 99.95th=[ 1351], | 99.99th=[ 1351] bw ( KiB/s): min= 15, max= 226, per=24.82%, avg=85.89, stdev=30.62, samples=235 iops : min= 3, max= 56, avg=21.06, stdev= 7.71, samples=235 lat (msec) : 250=0.78%, 500=8.09%, 750=45.55%, 1000=37.50%, 2000=8.09% cpu : usr=0.06%, sys=0.29%, ctx=1652, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=146155: Tue Sep 8 02:41:23 2026 write: IOPS=21, BW=86.7KiB/s (88.8kB/s)(10.0MiB/118134msec); 0 zone resets slat (usec): min=22, max=10531, avg=208.25, stdev=707.83 clat (msec): min=61, max=1375, avg=737.56, stdev=191.11 lat (msec): min=61, max=1375, avg=737.77, stdev=191.11 clat percentiles (msec): | 1.00th=[ 205], 5.00th=[ 443], 10.00th=[ 523], 20.00th=[ 600], | 30.00th=[ 651], 40.00th=[ 684], 50.00th=[ 726], 60.00th=[ 776], | 70.00th=[ 827], 80.00th=[ 885], 90.00th=[ 969], 95.00th=[ 1070], | 99.00th=[ 1217], 99.50th=[ 1267], 99.90th=[ 1301], 99.95th=[ 1368], | 99.99th=[ 1368] bw ( KiB/s): min= 15, max= 239, per=24.93%, avg=86.26, stdev=32.57, samples=234 iops : min= 3, max= 59, avg=21.15, stdev= 8.12, samples=234 lat (msec) : 100=0.16%, 250=1.29%, 500=7.54%, 750=45.86%, 1000=37.03% lat (msec) : 2000=8.12% cpu : usr=0.08%, sys=0.29%, ctx=2143, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=146156: Tue Sep 8 02:41:23 2026 write: IOPS=21, BW=86.7KiB/s (88.7kB/s)(10.0MiB/118166msec); 0 zone resets slat (usec): min=22, max=11737, avg=179.36, stdev=597.34 clat (msec): min=177, max=1326, avg=737.60, stdev=187.39 lat (msec): min=177, max=1326, avg=737.78, stdev=187.41 clat percentiles (msec): | 1.00th=[ 222], 5.00th=[ 447], 10.00th=[ 514], 20.00th=[ 600], | 30.00th=[ 651], 40.00th=[ 684], 50.00th=[ 735], 60.00th=[ 768], | 70.00th=[ 818], 80.00th=[ 877], 90.00th=[ 978], 95.00th=[ 1083], | 99.00th=[ 1217], 99.50th=[ 1250], 99.90th=[ 1301], 99.95th=[ 1301], | 99.99th=[ 1334] bw ( KiB/s): min= 7, max= 251, per=25.04%, avg=86.64, stdev=32.90, samples=233 iops : min= 1, max= 62, avg=21.24, stdev= 8.25, samples=233 lat (msec) : 250=1.25%, 500=7.50%, 750=45.47%, 1000=37.11%, 2000=8.67% cpu : usr=0.05%, sys=0.26%, ctx=1822, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=347KiB/s (355kB/s), 86.6KiB/s-86.8KiB/s (88.7kB/s-88.9kB/s), io=40.0MiB (41.9MB), run=117943-118198msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=146183: Tue Sep 8 02:42:57 2026 read: IOPS=14, BW=59.0KiB/s (60.5kB/s)(5048KiB/85510msec) slat (usec): min=2722, max=52652, avg=12011.27, stdev=6556.70 clat (msec): min=22, max=887, avg=338.64, stdev=81.48 lat (msec): min=36, max=892, avg=350.66, stdev=82.32 clat percentiles (msec): | 1.00th=[ 184], 5.00th=[ 228], 10.00th=[ 251], 20.00th=[ 279], | 30.00th=[ 296], 40.00th=[ 313], 50.00th=[ 330], 60.00th=[ 347], | 70.00th=[ 368], 80.00th=[ 401], 90.00th=[ 443], 95.00th=[ 481], | 99.00th=[ 558], 99.50th=[ 634], 99.90th=[ 827], 99.95th=[ 885], | 99.99th=[ 885] bw ( KiB/s): min= 7, max= 128, per=25.11%, avg=58.75, stdev=24.83, samples=169 iops : min= 1, max= 32, avg=14.20, stdev= 6.23, samples=169 write: IOPS=15, BW=60.7KiB/s (62.2kB/s)(5192KiB/85510msec); 0 zone resets slat (usec): min=27, max=13272, avg=255.94, stdev=747.06 clat (msec): min=223, max=1425, avg=710.30, stdev=223.44 lat (msec): min=224, max=1425, avg=710.56, stdev=223.47 clat percentiles (msec): | 1.00th=[ 279], 5.00th=[ 372], 10.00th=[ 418], 20.00th=[ 527], | 30.00th=[ 592], 40.00th=[ 651], 50.00th=[ 693], 60.00th=[ 751], | 70.00th=[ 802], 80.00th=[ 877], 90.00th=[ 986], 95.00th=[ 1150], | 99.00th=[ 1334], 99.50th=[ 1385], 99.90th=[ 1418], 99.95th=[ 1418], | 99.99th=[ 1418] bw ( KiB/s): min= 7, max= 143, per=24.69%, avg=60.25, stdev=24.25, samples=169 iops : min= 1, max= 35, avg=14.57, stdev= 6.11, samples=169 lat (msec) : 50=0.16%, 100=0.04%, 250=4.92%, 500=51.17%, 750=23.55% lat (msec) : 1000=15.31%, 2000=4.84% cpu : usr=0.12%, sys=0.73%, ctx=4342, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=146184: Tue Sep 8 02:42:57 2026 read: IOPS=14, BW=58.1KiB/s (59.5kB/s)(4976KiB/85573msec) slat (usec): min=2825, max=42900, avg=12234.28, stdev=6349.41 clat (msec): min=13, max=696, avg=338.43, stdev=83.73 lat (msec): min=23, max=706, avg=350.68, stdev=84.80 clat percentiles (msec): | 1.00th=[ 138], 5.00th=[ 222], 10.00th=[ 245], 20.00th=[ 275], | 30.00th=[ 296], 40.00th=[ 317], 50.00th=[ 334], 60.00th=[ 351], | 70.00th=[ 368], 80.00th=[ 397], 90.00th=[ 447], 95.00th=[ 485], | 99.00th=[ 575], 99.50th=[ 600], 99.90th=[ 651], 99.95th=[ 701], | 99.99th=[ 701] bw ( KiB/s): min= 15, max= 141, per=24.57%, avg=57.49, stdev=25.25, samples=170 iops : min= 3, max= 35, avg=13.88, stdev= 6.34, samples=170 write: IOPS=15, BW=61.5KiB/s (62.0kB/s)(5264KiB/85573msec); 0 zone resets slat (usec): min=27, max=12338, avg=283.54, stdev=866.22 clat (msec): min=168, max=1403, avg=706.94, stdev=217.34 lat (msec): min=169, max=1403, avg=707.23, stdev=217.34 clat percentiles (msec): | 1.00th=[ 279], 5.00th=[ 368], 10.00th=[ 435], 20.00th=[ 523], | 30.00th=[ 600], 40.00th=[ 651], 50.00th=[ 701], 60.00th=[ 743], | 70.00th=[ 802], 80.00th=[ 860], 90.00th=[ 978], 95.00th=[ 1133], | 99.00th=[ 1318], 99.50th=[ 1368], 99.90th=[ 1401], 99.95th=[ 1401], | 99.99th=[ 1401] bw ( KiB/s): min= 7, max= 135, per=24.89%, avg=60.74, stdev=21.58, samples=170 iops : min= 1, max= 33, avg=14.69, stdev= 5.48, samples=170 lat (msec) : 20=0.04%, 50=0.12%, 100=0.27%, 250=5.16%, 500=50.31% lat (msec) : 750=23.87%, 1000=15.51%, 2000=4.73% cpu : usr=0.11%, sys=0.74%, ctx=4515, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=146185: Tue Sep 8 02:42:57 2026 read: IOPS=14, BW=58.2KiB/s (59.6kB/s)(4984KiB/85566msec) slat (usec): min=2467, max=49322, avg=12363.11, stdev=6617.55 clat (msec): min=20, max=690, avg=341.32, stdev=82.10 lat (msec): min=25, max=708, avg=353.70, stdev=83.27 clat percentiles (msec): | 1.00th=[ 174], 5.00th=[ 226], 10.00th=[ 253], 20.00th=[ 275], | 30.00th=[ 296], 40.00th=[ 317], 50.00th=[ 334], 60.00th=[ 355], | 70.00th=[ 372], 80.00th=[ 405], 90.00th=[ 447], 95.00th=[ 485], | 99.00th=[ 558], 99.50th=[ 592], 99.90th=[ 659], 99.95th=[ 693], | 99.99th=[ 693] bw ( KiB/s): min= 7, max= 143, per=24.69%, avg=57.77, stdev=24.31, samples=170 iops : min= 1, max= 35, avg=13.95, stdev= 6.10, samples=170 write: IOPS=15, BW=61.4KiB/s (62.9kB/s)(5256KiB/85566msec); 0 zone resets slat (usec): min=25, max=8341, avg=234.75, stdev=603.40 clat (msec): min=117, max=1430, avg=704.21, stdev=221.65 lat (msec): min=117, max=1430, avg=704.45, stdev=221.67 clat percentiles (msec): | 1.00th=[ 288], 5.00th=[ 368], 10.00th=[ 422], 20.00th=[ 514], | 30.00th=[ 584], 40.00th=[ 642], 50.00th=[ 701], 60.00th=[ 743], | 70.00th=[ 802], 80.00th=[ 869], 90.00th=[ 978], 95.00th=[ 1116], | 99.00th=[ 1368], 99.50th=[ 1385], 99.90th=[ 1401], 99.95th=[ 1435], | 99.99th=[ 1435] bw ( KiB/s): min= 15, max= 127, per=24.80%, avg=60.51, stdev=23.51, samples=170 iops : min= 3, max= 31, avg=14.64, stdev= 5.90, samples=170 lat (msec) : 50=0.23%, 100=0.08%, 250=4.45%, 500=51.48%, 750=24.49% lat (msec) : 1000=14.57%, 2000=4.69% cpu : usr=0.15%, sys=0.76%, ctx=4315, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=146186: Tue Sep 8 02:42:57 2026 read: IOPS=14, BW=59.3KiB/s (60.8kB/s)(5060KiB/85275msec) slat (usec): min=54, max=51833, avg=12202.05, stdev=6558.73 clat (msec): min=16, max=822, avg=343.61, stdev=85.61 lat (msec): min=31, max=826, avg=355.83, stdev=86.59 clat percentiles (msec): | 1.00th=[ 169], 5.00th=[ 222], 10.00th=[ 251], 20.00th=[ 279], | 30.00th=[ 300], 40.00th=[ 317], 50.00th=[ 338], 60.00th=[ 355], | 70.00th=[ 376], 80.00th=[ 405], 90.00th=[ 447], 95.00th=[ 493], | 99.00th=[ 600], 99.50th=[ 634], 99.90th=[ 726], 99.95th=[ 827], | 99.99th=[ 827] bw ( KiB/s): min= 7, max= 128, per=25.34%, avg=59.30, stdev=24.33, samples=168 iops : min= 1, max= 32, avg=14.33, stdev= 6.13, samples=168 write: IOPS=15, BW=60.7KiB/s (62.2kB/s)(5180KiB/85275msec); 0 zone resets slat (usec): min=27, max=26223, avg=264.42, stdev=1002.22 clat (msec): min=99, max=1528, avg=704.86, stdev=227.25 lat (msec): min=99, max=1528, avg=705.13, stdev=227.31 clat percentiles (msec): | 1.00th=[ 241], 5.00th=[ 355], 10.00th=[ 422], 20.00th=[ 514], | 30.00th=[ 592], 40.00th=[ 642], 50.00th=[ 701], 60.00th=[ 743], | 70.00th=[ 793], 80.00th=[ 860], 90.00th=[ 986], 95.00th=[ 1150], | 99.00th=[ 1351], 99.50th=[ 1452], 99.90th=[ 1536], 99.95th=[ 1536], | 99.99th=[ 1536] bw ( KiB/s): min= 7, max= 167, per=24.56%, avg=59.92, stdev=24.95, samples=169 iops : min= 1, max= 41, avg=14.50, stdev= 6.28, samples=169 lat (msec) : 20=0.08%, 50=0.16%, 100=0.20%, 250=5.08%, 500=51.17% lat (msec) : 750=23.71%, 1000=14.73%, 2000=4.88% cpu : usr=0.11%, sys=0.74%, ctx=4118, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=235KiB/s (240kB/s), 58.1KiB/s-59.3KiB/s (59.5kB/s-60.8kB/s), io=19.6MiB (20.5MB), run=85275-85573msec WRITE: bw=244KiB/s (250kB/s), 60.7KiB/s-61.5KiB/s (62.2kB/s-62.0kB/s), io=20.4MiB (21.4MB), run=85275-85573msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=146202: Tue Sep 8 02:43:06 2026 read: IOPS=2, BW=104MiB/s (110MB/s)(40.0MiB/383msec) slat (nsec): min=19891k, max=19891k, avg=19890806.00, stdev= 0.00 clat (nsec): min=352665k, max=352665k, avg=352664927.00, stdev= 0.00 lat (nsec): min=372561k, max=372561k, avg=372560814.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 355], 5.00th=[ 355], 10.00th=[ 355], 20.00th=[ 355], | 30.00th=[ 355], 40.00th=[ 355], 50.00th=[ 355], 60.00th=[ 355], | 70.00th=[ 355], 80.00th=[ 355], 90.00th=[ 355], 95.00th=[ 355], | 99.00th=[ 355], 99.50th=[ 355], 99.90th=[ 355], 99.95th=[ 355], | 99.99th=[ 355] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=3.40%, ctx=42, majf=0, minf=7 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=104MiB/s (110MB/s), 104MiB/s-104MiB/s (110MB/s-110MB/s), io=40.0MiB (41.9MB), run=383-383msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout