== sanity test 398c: run fio to test AIO ================= 17:20:10 (1788729610) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.227781 s, 184 MB/s osc.lustre-OST0000-osc-ffff8c3d20106800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=146771: Sun Sep 6 17:21:44 2026 write: IOPS=28, BW=115KiB/s (117kB/s)(10.0MiB/89363msec); 0 zone resets slat (usec): min=23, max=7126, avg=171.22, stdev=410.41 clat (msec): min=172, max=870, avg=558.16, stdev=111.56 lat (msec): min=172, max=870, avg=558.33, stdev=111.58 clat percentiles (msec): | 1.00th=[ 245], 5.00th=[ 376], 10.00th=[ 422], 20.00th=[ 472], | 30.00th=[ 510], 40.00th=[ 542], 50.00th=[ 567], 60.00th=[ 592], | 70.00th=[ 609], 80.00th=[ 642], 90.00th=[ 701], 95.00th=[ 743], | 99.00th=[ 802], 99.50th=[ 835], 99.90th=[ 860], 99.95th=[ 869], | 99.99th=[ 869] bw ( KiB/s): min= 48, max= 256, per=24.87%, avg=113.65, stdev=28.54, samples=178 iops : min= 12, max= 64, avg=28.07, stdev= 7.21, samples=178 lat (msec) : 250=2.03%, 500=25.27%, 750=69.02%, 1000=3.67% cpu : usr=0.06%, sys=0.31%, ctx=2154, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=146772: Sun Sep 6 17:21:44 2026 write: IOPS=28, BW=114KiB/s (117kB/s)(10.0MiB/89439msec); 0 zone resets slat (usec): min=23, max=10736, avg=184.43, stdev=448.35 clat (msec): min=244, max=922, avg=558.11, stdev=106.98 lat (msec): min=244, max=922, avg=558.30, stdev=106.97 clat percentiles (msec): | 1.00th=[ 275], 5.00th=[ 376], 10.00th=[ 430], 20.00th=[ 477], | 30.00th=[ 506], 40.00th=[ 542], 50.00th=[ 558], 60.00th=[ 584], | 70.00th=[ 609], 80.00th=[ 642], 90.00th=[ 693], 95.00th=[ 735], | 99.00th=[ 810], 99.50th=[ 835], 99.90th=[ 869], 99.95th=[ 885], | 99.99th=[ 919] bw ( KiB/s): min= 15, max= 256, per=24.90%, avg=113.78, stdev=30.33, samples=178 iops : min= 3, max= 64, avg=28.10, stdev= 7.63, samples=178 lat (msec) : 250=0.70%, 500=27.81%, 750=67.77%, 1000=3.71% cpu : usr=0.06%, sys=0.33%, ctx=2393, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=146773: Sun Sep 6 17:21:44 2026 write: IOPS=28, BW=115KiB/s (117kB/s)(10.0MiB/89415msec); 0 zone resets slat (usec): min=21, max=6399, avg=163.06, stdev=368.33 clat (msec): min=235, max=921, avg=558.01, stdev=110.07 lat (msec): min=235, max=922, avg=558.17, stdev=110.09 clat percentiles (msec): | 1.00th=[ 259], 5.00th=[ 380], 10.00th=[ 426], 20.00th=[ 472], | 30.00th=[ 510], 40.00th=[ 542], 50.00th=[ 558], 60.00th=[ 584], | 70.00th=[ 609], 80.00th=[ 642], 90.00th=[ 701], 95.00th=[ 743], | 99.00th=[ 802], 99.50th=[ 827], 99.90th=[ 844], 99.95th=[ 844], | 99.99th=[ 919] bw ( KiB/s): min= 32, max= 256, per=24.87%, avg=113.65, stdev=29.88, samples=178 iops : min= 8, max= 64, avg=28.07, stdev= 7.51, samples=178 lat (msec) : 250=0.08%, 500=26.41%, 750=69.02%, 1000=4.49% cpu : usr=0.06%, sys=0.29%, ctx=2185, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=146774: Sun Sep 6 17:21:44 2026 write: IOPS=28, BW=115KiB/s (117kB/s)(10.0MiB/89307msec); 0 zone resets slat (usec): min=23, max=8240, avg=176.39, stdev=398.48 clat (msec): min=134, max=886, avg=557.43, stdev=111.20 lat (msec): min=134, max=886, avg=557.61, stdev=111.20 clat percentiles (msec): | 1.00th=[ 245], 5.00th=[ 376], 10.00th=[ 430], 20.00th=[ 472], | 30.00th=[ 506], 40.00th=[ 542], 50.00th=[ 567], 60.00th=[ 584], | 70.00th=[ 617], 80.00th=[ 642], 90.00th=[ 693], 95.00th=[ 735], | 99.00th=[ 802], 99.50th=[ 835], 99.90th=[ 852], 99.95th=[ 852], | 99.99th=[ 885] bw ( KiB/s): min= 40, max= 256, per=24.93%, avg=113.93, stdev=28.35, samples=177 iops : min= 10, max= 64, avg=28.14, stdev= 7.11, samples=177 lat (msec) : 250=1.48%, 500=26.17%, 750=68.75%, 1000=3.59% cpu : usr=0.05%, sys=0.33%, ctx=2444, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=458KiB/s (469kB/s), 114KiB/s-115KiB/s (117kB/s-117kB/s), io=40.0MiB (41.9MB), run=89307-89439msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=146797: Sun Sep 6 17:22:42 2026 read: IOPS=23, BW=92.9KiB/s (95.1kB/s)(5048KiB/54358msec) slat (usec): min=1956, max=31263, avg=6482.58, stdev=2844.86 clat (msec): min=39, max=546, avg=215.06, stdev=46.25 lat (msec): min=45, max=550, avg=221.55, stdev=46.65 clat percentiles (msec): | 1.00th=[ 111], 5.00th=[ 148], 10.00th=[ 163], 20.00th=[ 180], | 30.00th=[ 194], 40.00th=[ 203], 50.00th=[ 215], 60.00th=[ 226], | 70.00th=[ 236], 80.00th=[ 249], 90.00th=[ 271], 95.00th=[ 288], | 99.00th=[ 351], 99.50th=[ 380], 99.90th=[ 418], 99.95th=[ 550], | 99.99th=[ 550] bw ( KiB/s): min= 31, max= 191, per=25.05%, avg=92.42, stdev=33.09, samples=108 iops : min= 7, max= 47, avg=22.81, stdev= 8.29, samples=108 write: IOPS=23, BW=95.5KiB/s (97.8kB/s)(5192KiB/54358msec); 0 zone resets slat (usec): min=28, max=4192, avg=190.36, stdev=314.34 clat (msec): min=158, max=776, avg=453.07, stdev=111.27 lat (msec): min=159, max=776, avg=453.26, stdev=111.27 clat percentiles (msec): | 1.00th=[ 211], 5.00th=[ 264], 10.00th=[ 300], 20.00th=[ 347], | 30.00th=[ 397], 40.00th=[ 430], 50.00th=[ 456], 60.00th=[ 489], | 70.00th=[ 518], 80.00th=[ 550], 90.00th=[ 592], 95.00th=[ 625], | 99.00th=[ 693], 99.50th=[ 709], 99.90th=[ 735], 99.95th=[ 776], | 99.99th=[ 776] bw ( KiB/s): min= 40, max= 184, per=24.73%, avg=94.96, stdev=26.50, samples=108 iops : min= 10, max= 46, avg=23.44, stdev= 6.65, samples=108 lat (msec) : 50=0.20%, 100=0.20%, 250=41.91%, 500=39.92%, 750=17.73% lat (msec) : 1000=0.04% cpu : usr=0.13%, sys=0.84%, ctx=5121, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=146798: Sun Sep 6 17:22:42 2026 read: IOPS=22, BW=91.5KiB/s (93.7kB/s)(4976KiB/54358msec) slat (usec): min=51, max=21256, avg=6336.33, stdev=2729.96 clat (msec): min=8, max=417, avg=215.07, stdev=46.05 lat (msec): min=8, max=422, avg=221.42, stdev=46.55 clat percentiles (msec): | 1.00th=[ 99], 5.00th=[ 146], 10.00th=[ 163], 20.00th=[ 178], | 30.00th=[ 194], 40.00th=[ 205], 50.00th=[ 215], 60.00th=[ 226], | 70.00th=[ 236], 80.00th=[ 249], 90.00th=[ 271], 95.00th=[ 292], | 99.00th=[ 326], 99.50th=[ 351], 99.90th=[ 414], 99.95th=[ 418], | 99.99th=[ 418] bw ( KiB/s): min= 24, max= 167, per=24.64%, avg=90.92, stdev=31.35, samples=108 iops : min= 6, max= 41, avg=22.44, stdev= 7.78, samples=108 write: IOPS=24, BW=96.8KiB/s (99.2kB/s)(5264KiB/54358msec); 0 zone resets slat (usec): min=33, max=4673, avg=172.94, stdev=283.04 clat (msec): min=53, max=766, avg=451.06, stdev=113.71 lat (msec): min=53, max=766, avg=451.23, stdev=113.73 clat percentiles (msec): | 1.00th=[ 184], 5.00th=[ 264], 10.00th=[ 296], 20.00th=[ 351], | 30.00th=[ 393], 40.00th=[ 426], 50.00th=[ 456], 60.00th=[ 485], | 70.00th=[ 514], 80.00th=[ 550], 90.00th=[ 600], 95.00th=[ 634], | 99.00th=[ 684], 99.50th=[ 701], 99.90th=[ 751], 99.95th=[ 768], | 99.99th=[ 768] bw ( KiB/s): min= 55, max= 176, per=25.05%, avg=96.18, stdev=23.94, samples=108 iops : min= 13, max= 44, avg=23.75, stdev= 6.11, samples=108 lat (msec) : 10=0.08%, 20=0.04%, 50=0.23%, 100=0.27%, 250=40.78% lat (msec) : 500=40.35%, 750=18.16%, 1000=0.08% cpu : usr=0.11%, sys=0.85%, ctx=4498, majf=0, minf=16 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=146799: Sun Sep 6 17:22:42 2026 read: IOPS=22, BW=91.7KiB/s (93.9kB/s)(4984KiB/54361msec) slat (usec): min=1872, max=22687, avg=6408.56, stdev=2714.42 clat (msec): min=9, max=387, avg=213.92, stdev=46.49 lat (msec): min=14, max=403, avg=220.33, stdev=47.02 clat percentiles (msec): | 1.00th=[ 100], 5.00th=[ 138], 10.00th=[ 163], 20.00th=[ 178], | 30.00th=[ 192], 40.00th=[ 203], 50.00th=[ 215], 60.00th=[ 226], | 70.00th=[ 239], 80.00th=[ 249], 90.00th=[ 271], 95.00th=[ 288], | 99.00th=[ 326], 99.50th=[ 334], 99.90th=[ 359], 99.95th=[ 388], | 99.99th=[ 388] bw ( KiB/s): min= 24, max= 184, per=24.82%, avg=91.57, stdev=31.76, samples=108 iops : min= 6, max= 46, avg=22.60, stdev= 7.95, samples=108 write: IOPS=24, BW=96.7KiB/s (99.0kB/s)(5256KiB/54361msec); 0 zone resets slat (usec): min=32, max=8047, avg=197.46, stdev=399.04 clat (msec): min=154, max=723, avg=451.88, stdev=109.42 lat (msec): min=154, max=723, avg=452.08, stdev=109.42 clat percentiles (msec): | 1.00th=[ 203], 5.00th=[ 266], 10.00th=[ 300], 20.00th=[ 355], | 30.00th=[ 401], 40.00th=[ 426], 50.00th=[ 451], 60.00th=[ 485], | 70.00th=[ 518], 80.00th=[ 550], 90.00th=[ 592], 95.00th=[ 625], | 99.00th=[ 676], 99.50th=[ 693], 99.90th=[ 709], 99.95th=[ 726], | 99.99th=[ 726] bw ( KiB/s): min= 48, max= 160, per=24.92%, avg=95.69, stdev=23.77, samples=108 iops : min= 12, max= 40, avg=23.63, stdev= 6.01, samples=108 lat (msec) : 10=0.04%, 20=0.08%, 50=0.31%, 100=0.08%, 250=40.59% lat (msec) : 500=40.86%, 750=18.05% cpu : usr=0.09%, sys=0.89%, ctx=4994, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=146800: Sun Sep 6 17:22:42 2026 read: IOPS=23, BW=93.1KiB/s (95.3kB/s)(5060KiB/54361msec) slat (usec): min=1689, max=21240, avg=6385.65, stdev=2740.34 clat (msec): min=8, max=409, avg=216.76, stdev=46.30 lat (msec): min=13, max=417, avg=223.15, stdev=46.87 clat percentiles (msec): | 1.00th=[ 107], 5.00th=[ 148], 10.00th=[ 167], 20.00th=[ 180], | 30.00th=[ 194], 40.00th=[ 205], 50.00th=[ 218], 60.00th=[ 228], | 70.00th=[ 239], 80.00th=[ 251], 90.00th=[ 275], 95.00th=[ 292], | 99.00th=[ 334], 99.50th=[ 359], 99.90th=[ 405], 99.95th=[ 409], | 99.99th=[ 409] bw ( KiB/s): min= 23, max= 168, per=25.14%, avg=92.75, stdev=30.64, samples=108 iops : min= 5, max= 42, avg=22.89, stdev= 7.65, samples=108 write: IOPS=23, BW=95.3KiB/s (97.6kB/s)(5180KiB/54361msec); 0 zone resets slat (usec): min=29, max=6075, avg=193.54, stdev=356.61 clat (msec): min=151, max=770, avg=452.58, stdev=113.82 lat (msec): min=151, max=770, avg=452.78, stdev=113.82 clat percentiles (msec): | 1.00th=[ 184], 5.00th=[ 257], 10.00th=[ 300], 20.00th=[ 355], | 30.00th=[ 397], 40.00th=[ 422], 50.00th=[ 456], 60.00th=[ 485], | 70.00th=[ 514], 80.00th=[ 558], 90.00th=[ 600], 95.00th=[ 642], | 99.00th=[ 701], 99.50th=[ 718], 99.90th=[ 768], 99.95th=[ 768], | 99.99th=[ 768] bw ( KiB/s): min= 47, max= 176, per=24.61%, avg=94.49, stdev=23.14, samples=108 iops : min= 11, max= 44, avg=23.32, stdev= 5.84, samples=108 lat (msec) : 10=0.04%, 20=0.08%, 50=0.23%, 100=0.08%, 250=40.94% lat (msec) : 500=41.29%, 750=17.23%, 1000=0.12% cpu : usr=0.12%, sys=0.83%, ctx=4785, majf=0, minf=16 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=369KiB/s (378kB/s), 91.5KiB/s-93.1KiB/s (93.7kB/s-95.3kB/s), io=19.6MiB (20.5MB), run=54358-54361msec WRITE: bw=384KiB/s (394kB/s), 95.3KiB/s-96.8KiB/s (97.6kB/s-99.2kB/s), io=20.4MiB (21.4MB), run=54358-54361msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=146812: Sun Sep 6 17:22:47 2026 read: IOPS=4, BW=192MiB/s (202MB/s)(40.0MiB/208msec) slat (nsec): min=14062k, max=14062k, avg=14062108.00, stdev= 0.00 clat (nsec): min=178622k, max=178622k, avg=178622284.00, stdev= 0.00 lat (nsec): min=192689k, max=192689k, avg=192689038.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 180], 5.00th=[ 180], 10.00th=[ 180], 20.00th=[ 180], | 30.00th=[ 180], 40.00th=[ 180], 50.00th=[ 180], 60.00th=[ 180], | 70.00th=[ 180], 80.00th=[ 180], 90.00th=[ 180], 95.00th=[ 180], | 99.00th=[ 180], 99.50th=[ 180], 99.90th=[ 180], 99.95th=[ 180], | 99.99th=[ 180] lat (msec) : 250=100.00% cpu : usr=0.00%, sys=5.31%, ctx=3, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=192MiB/s (202MB/s), 192MiB/s-192MiB/s (202MB/s-202MB/s), io=40.0MiB (41.9MB), run=208-208msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout