== sanity test 398c: run fio to test AIO ================= 05:48:32 (1784281712) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.220411 s, 190 MB/s osc.lustre-OST0000-osc-ffff919269d47000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=188749: Fri Jul 17 05:48:51 2026 write: IOPS=176, BW=707KiB/s (724kB/s)(10.0MiB/14486msec); 0 zone resets slat (usec): min=45, max=5326, avg=223.02, stdev=367.31 clat (msec): min=9, max=159, avg=89.73, stdev=17.19 lat (msec): min=9, max=160, avg=89.95, stdev=17.19 clat percentiles (msec): | 1.00th=[ 47], 5.00th=[ 61], 10.00th=[ 69], 20.00th=[ 77], | 30.00th=[ 82], 40.00th=[ 86], 50.00th=[ 90], 60.00th=[ 94], | 70.00th=[ 100], 80.00th=[ 104], 90.00th=[ 111], 95.00th=[ 117], | 99.00th=[ 131], 99.50th=[ 136], 99.90th=[ 142], 99.95th=[ 159], | 99.99th=[ 161] bw ( KiB/s): min= 550, max= 944, per=24.95%, avg=704.89, stdev=96.30, samples=28 iops : min= 137, max= 236, avg=176.07, stdev=24.02, samples=28 lat (msec) : 10=0.04%, 20=0.08%, 50=1.17%, 100=71.21%, 250=27.50% cpu : usr=0.18%, sys=2.17%, ctx=2818, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=188750: Fri Jul 17 05:48:51 2026 write: IOPS=177, BW=711KiB/s (728kB/s)(10.0MiB/14402msec); 0 zone resets slat (usec): min=34, max=6522, avg=230.41, stdev=368.46 clat (msec): min=9, max=176, avg=89.59, stdev=18.06 lat (msec): min=9, max=176, avg=89.82, stdev=18.06 clat percentiles (msec): | 1.00th=[ 44], 5.00th=[ 62], 10.00th=[ 67], 20.00th=[ 75], | 30.00th=[ 82], 40.00th=[ 86], 50.00th=[ 90], 60.00th=[ 94], | 70.00th=[ 100], 80.00th=[ 104], 90.00th=[ 112], 95.00th=[ 120], | 99.00th=[ 130], 99.50th=[ 148], 99.90th=[ 169], 99.95th=[ 178], | 99.99th=[ 178] bw ( KiB/s): min= 560, max= 1051, per=25.10%, avg=709.00, stdev=101.01, samples=28 iops : min= 140, max= 262, avg=177.07, stdev=25.13, samples=28 lat (msec) : 10=0.04%, 20=0.04%, 50=1.99%, 100=70.20%, 250=27.73% cpu : usr=0.25%, sys=2.33%, ctx=2818, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=188751: Fri Jul 17 05:48:51 2026 write: IOPS=176, BW=706KiB/s (723kB/s)(10.0MiB/14495msec); 0 zone resets slat (usec): min=37, max=5482, avg=225.40, stdev=324.63 clat (msec): min=22, max=181, avg=89.81, stdev=17.16 lat (msec): min=23, max=182, avg=90.04, stdev=17.16 clat percentiles (msec): | 1.00th=[ 50], 5.00th=[ 62], 10.00th=[ 69], 20.00th=[ 77], | 30.00th=[ 82], 40.00th=[ 86], 50.00th=[ 90], 60.00th=[ 94], | 70.00th=[ 99], 80.00th=[ 104], 90.00th=[ 111], 95.00th=[ 117], | 99.00th=[ 130], 99.50th=[ 136], 99.90th=[ 182], 99.95th=[ 182], | 99.99th=[ 182] bw ( KiB/s): min= 568, max= 938, per=24.98%, avg=705.82, stdev=90.20, samples=28 iops : min= 142, max= 234, avg=176.29, stdev=22.44, samples=28 lat (msec) : 50=1.09%, 100=72.81%, 250=26.09% cpu : usr=0.26%, sys=2.26%, ctx=2823, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=188752: Fri Jul 17 05:48:51 2026 write: IOPS=176, BW=706KiB/s (723kB/s)(10.0MiB/14495msec); 0 zone resets slat (usec): min=47, max=5753, avg=229.04, stdev=336.83 clat (msec): min=32, max=169, avg=89.85, stdev=17.47 lat (msec): min=32, max=169, avg=90.08, stdev=17.46 clat percentiles (msec): | 1.00th=[ 47], 5.00th=[ 62], 10.00th=[ 69], 20.00th=[ 77], | 30.00th=[ 82], 40.00th=[ 86], 50.00th=[ 90], 60.00th=[ 94], | 70.00th=[ 100], 80.00th=[ 104], 90.00th=[ 112], 95.00th=[ 117], | 99.00th=[ 129], 99.50th=[ 138], 99.90th=[ 169], 99.95th=[ 169], | 99.99th=[ 169] bw ( KiB/s): min= 526, max= 949, per=24.93%, avg=704.21, stdev=90.24, samples=28 iops : min= 131, max= 237, avg=175.89, stdev=22.51, samples=28 lat (msec) : 50=1.84%, 100=70.35%, 250=27.81% cpu : usr=0.26%, sys=2.23%, ctx=2967, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=2826KiB/s (2894kB/s), 706KiB/s-711KiB/s (723kB/s-728kB/s), io=40.0MiB (41.9MB), run=14402-14495msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=188764: Fri Jul 17 05:49:09 2026 read: IOPS=91, BW=367KiB/s (376kB/s)(5048KiB/13737msec) slat (usec): min=1732, max=17628, avg=5552.84, stdev=2153.74 clat (msec): min=12, max=154, avg=81.03, stdev=15.17 lat (msec): min=16, max=160, avg=86.59, stdev=15.46 clat percentiles (msec): | 1.00th=[ 51], 5.00th=[ 58], 10.00th=[ 63], 20.00th=[ 69], | 30.00th=[ 73], 40.00th=[ 78], 50.00th=[ 81], 60.00th=[ 85], | 70.00th=[ 89], 80.00th=[ 93], 90.00th=[ 102], 95.00th=[ 108], | 99.00th=[ 116], 99.50th=[ 126], 99.90th=[ 129], 99.95th=[ 155], | 99.99th=[ 155] bw ( KiB/s): min= 288, max= 448, per=25.14%, avg=367.11, stdev=40.46, samples=27 iops : min= 72, max= 112, avg=91.59, stdev=10.09, samples=27 write: IOPS=94, BW=378KiB/s (387kB/s)(5192KiB/13737msec); 0 zone resets slat (usec): min=40, max=4030, avg=235.23, stdev=326.67 clat (msec): min=28, max=132, avg=83.91, stdev=13.54 lat (msec): min=28, max=132, avg=84.15, stdev=13.55 clat percentiles (msec): | 1.00th=[ 56], 5.00th=[ 63], 10.00th=[ 67], 20.00th=[ 73], | 30.00th=[ 78], 40.00th=[ 81], 50.00th=[ 84], 60.00th=[ 87], | 70.00th=[ 90], 80.00th=[ 94], 90.00th=[ 102], 95.00th=[ 109], | 99.00th=[ 118], 99.50th=[ 126], 99.90th=[ 130], 99.95th=[ 133], | 99.99th=[ 133] bw ( KiB/s): min= 303, max= 472, per=24.88%, avg=378.15, stdev=46.94, samples=27 iops : min= 75, max= 118, avg=94.33, stdev=11.79, samples=27 lat (msec) : 20=0.08%, 50=0.55%, 100=88.40%, 250=10.98% cpu : usr=0.29%, sys=3.19%, ctx=4762, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=188765: Fri Jul 17 05:49:09 2026 read: IOPS=91, BW=366KiB/s (374kB/s)(4976KiB/13611msec) slat (usec): min=53, max=15194, avg=5511.40, stdev=2212.05 clat (msec): min=12, max=168, avg=80.81, stdev=15.22 lat (msec): min=13, max=175, avg=86.33, stdev=15.57 clat percentiles (msec): | 1.00th=[ 48], 5.00th=[ 58], 10.00th=[ 64], 20.00th=[ 70], | 30.00th=[ 74], 40.00th=[ 78], 50.00th=[ 81], 60.00th=[ 85], | 70.00th=[ 88], 80.00th=[ 92], 90.00th=[ 100], 95.00th=[ 107], | 99.00th=[ 121], 99.50th=[ 129], 99.90th=[ 161], 99.95th=[ 169], | 99.99th=[ 169] bw ( KiB/s): min= 295, max= 432, per=24.96%, avg=364.46, stdev=40.63, samples=26 iops : min= 73, max= 108, avg=90.92, stdev=10.23, samples=26 write: IOPS=96, BW=387KiB/s (396kB/s)(5264KiB/13611msec); 0 zone resets slat (usec): min=54, max=5330, avg=217.00, stdev=289.27 clat (msec): min=38, max=168, avg=83.43, stdev=14.12 lat (msec): min=38, max=168, avg=83.65, stdev=14.13 clat percentiles (msec): | 1.00th=[ 53], 5.00th=[ 62], 10.00th=[ 67], 20.00th=[ 72], | 30.00th=[ 77], 40.00th=[ 80], 50.00th=[ 84], 60.00th=[ 86], | 70.00th=[ 90], 80.00th=[ 94], 90.00th=[ 102], 95.00th=[ 108], | 99.00th=[ 122], 99.50th=[ 126], 99.90th=[ 150], 99.95th=[ 169], | 99.99th=[ 169] bw ( KiB/s): min= 320, max= 488, per=25.53%, avg=388.08, stdev=42.18, samples=26 iops : min= 80, max= 122, avg=96.81, stdev=10.55, samples=26 lat (msec) : 20=0.20%, 50=1.05%, 100=88.44%, 250=10.31% cpu : usr=0.37%, sys=3.06%, ctx=4665, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=188766: Fri Jul 17 05:49:09 2026 read: IOPS=90, BW=363KiB/s (372kB/s)(4984KiB/13737msec) slat (usec): min=1820, max=19481, avg=5533.22, stdev=2217.05 clat (msec): min=27, max=140, avg=80.99, stdev=14.87 lat (msec): min=33, max=148, avg=86.53, stdev=15.18 clat percentiles (msec): | 1.00th=[ 49], 5.00th=[ 58], 10.00th=[ 63], 20.00th=[ 69], | 30.00th=[ 73], 40.00th=[ 77], 50.00th=[ 81], 60.00th=[ 85], | 70.00th=[ 89], 80.00th=[ 93], 90.00th=[ 101], 95.00th=[ 106], | 99.00th=[ 118], 99.50th=[ 125], 99.90th=[ 138], 99.95th=[ 140], | 99.99th=[ 140] bw ( KiB/s): min= 271, max= 472, per=24.92%, avg=363.89, stdev=51.66, samples=27 iops : min= 67, max= 118, avg=90.78, stdev=12.98, samples=27 write: IOPS=95, BW=383KiB/s (392kB/s)(5256KiB/13737msec); 0 zone resets slat (usec): min=50, max=3830, avg=204.15, stdev=259.08 clat (msec): min=42, max=129, avg=84.11, stdev=13.16 lat (msec): min=42, max=129, avg=84.32, stdev=13.16 clat percentiles (msec): | 1.00th=[ 56], 5.00th=[ 63], 10.00th=[ 68], 20.00th=[ 73], | 30.00th=[ 78], 40.00th=[ 82], 50.00th=[ 85], 60.00th=[ 87], | 70.00th=[ 91], 80.00th=[ 95], 90.00th=[ 102], 95.00th=[ 107], | 99.00th=[ 118], 99.50th=[ 120], 99.90th=[ 127], 99.95th=[ 130], | 99.99th=[ 130] bw ( KiB/s): min= 271, max= 448, per=25.10%, avg=381.44, stdev=42.60, samples=27 iops : min= 67, max= 112, avg=95.15, stdev=10.60, samples=27 lat (msec) : 50=0.82%, 100=87.89%, 250=11.29% cpu : usr=0.35%, sys=3.04%, ctx=4588, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=188767: Fri Jul 17 05:49:09 2026 read: IOPS=92, BW=369KiB/s (378kB/s)(5060KiB/13720msec) slat (usec): min=1689, max=16966, avg=5489.88, stdev=2125.21 clat (msec): min=29, max=144, avg=80.97, stdev=14.27 lat (msec): min=34, max=148, avg=86.46, stdev=14.60 clat percentiles (msec): | 1.00th=[ 49], 5.00th=[ 58], 10.00th=[ 64], 20.00th=[ 69], | 30.00th=[ 73], 40.00th=[ 78], 50.00th=[ 82], 60.00th=[ 85], | 70.00th=[ 89], 80.00th=[ 93], 90.00th=[ 100], 95.00th=[ 104], | 99.00th=[ 116], 99.50th=[ 118], 99.90th=[ 140], 99.95th=[ 144], | 99.99th=[ 144] bw ( KiB/s): min= 287, max= 480, per=25.24%, avg=368.56, stdev=43.21, samples=27 iops : min= 71, max= 120, avg=91.93, stdev=10.81, samples=27 write: IOPS=94, BW=378KiB/s (387kB/s)(5180KiB/13720msec); 0 zone resets slat (usec): min=39, max=6276, avg=231.19, stdev=371.53 clat (msec): min=34, max=130, avg=83.87, stdev=13.26 lat (msec): min=34, max=130, avg=84.11, stdev=13.24 clat percentiles (msec): | 1.00th=[ 54], 5.00th=[ 64], 10.00th=[ 68], 20.00th=[ 73], | 30.00th=[ 78], 40.00th=[ 81], 50.00th=[ 84], 60.00th=[ 87], | 70.00th=[ 90], 80.00th=[ 94], 90.00th=[ 102], 95.00th=[ 108], | 99.00th=[ 116], 99.50th=[ 123], 99.90th=[ 127], 99.95th=[ 131], | 99.99th=[ 131] bw ( KiB/s): min= 296, max= 488, per=24.70%, avg=375.44, stdev=48.96, samples=27 iops : min= 74, max= 122, avg=93.67, stdev=12.25, samples=27 lat (msec) : 50=0.90%, 100=88.98%, 250=10.12% cpu : usr=0.39%, sys=2.94%, ctx=4629, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1461KiB/s (1496kB/s), 363KiB/s-369KiB/s (372kB/s-378kB/s), io=19.6MiB (20.5MB), run=13611-13737msec WRITE: bw=1521KiB/s (1557kB/s), 378KiB/s-387KiB/s (387kB/s-396kB/s), io=20.4MiB (21.4MB), run=13611-13737msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=188772: Fri Jul 17 05:49:13 2026 read: IOPS=5, BW=222MiB/s (233MB/s)(40.0MiB/180msec) slat (nsec): min=15219k, max=15219k, avg=15218768.00, stdev= 0.00 clat (nsec): min=153609k, max=153609k, avg=153609005.00, stdev= 0.00 lat (nsec): min=168833k, max=168833k, avg=168833144.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 155], 5.00th=[ 155], 10.00th=[ 155], 20.00th=[ 155], | 30.00th=[ 155], 40.00th=[ 155], 50.00th=[ 155], 60.00th=[ 155], | 70.00th=[ 155], 80.00th=[ 155], 90.00th=[ 155], 95.00th=[ 155], | 99.00th=[ 155], 99.50th=[ 155], 99.90th=[ 155], 99.95th=[ 155], | 99.99th=[ 155] lat (msec) : 250=100.00% cpu : usr=0.00%, sys=6.15%, ctx=12, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=222MiB/s (233MB/s), 222MiB/s-222MiB/s (233MB/s-233MB/s), io=40.0MiB (41.9MB), run=180-180msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout