== sanity test 398c: run fio to test AIO ================= 10:46:35 (1789137995) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.431112 s, 97.3 MB/s osc.lustre-OST0000-osc-ffff8e71890b7000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=182212: Fri Sep 11 10:47:07 2026 write: IOPS=109, BW=438KiB/s (448kB/s)(10.0MiB/23391msec); 0 zone resets slat (usec): min=25, max=19326, avg=213.68, stdev=643.00 clat (msec): min=35, max=359, avg=145.42, stdev=32.53 lat (msec): min=35, max=359, avg=145.64, stdev=32.52 clat percentiles (msec): | 1.00th=[ 90], 5.00th=[ 107], 10.00th=[ 113], 20.00th=[ 121], | 30.00th=[ 127], 40.00th=[ 133], 50.00th=[ 142], 60.00th=[ 150], | 70.00th=[ 159], 80.00th=[ 169], 90.00th=[ 188], 95.00th=[ 201], | 99.00th=[ 241], 99.50th=[ 255], 99.90th=[ 359], 99.95th=[ 359], | 99.99th=[ 359] bw ( KiB/s): min= 278, max= 600, per=25.13%, avg=439.07, stdev=70.34, samples=46 iops : min= 69, max= 150, avg=109.48, stdev=17.66, samples=46 lat (msec) : 50=0.62%, 100=2.27%, 250=96.45%, 500=0.66% cpu : usr=0.21%, sys=1.02%, ctx=2546, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=182213: Fri Sep 11 10:47:07 2026 write: IOPS=109, BW=437KiB/s (447kB/s)(10.0MiB/23434msec); 0 zone resets slat (usec): min=29, max=15018, avg=215.75, stdev=673.76 clat (msec): min=24, max=373, avg=145.75, stdev=32.60 lat (msec): min=24, max=374, avg=145.97, stdev=32.59 clat percentiles (msec): | 1.00th=[ 77], 5.00th=[ 107], 10.00th=[ 112], 20.00th=[ 120], | 30.00th=[ 127], 40.00th=[ 134], 50.00th=[ 144], 60.00th=[ 150], | 70.00th=[ 159], 80.00th=[ 169], 90.00th=[ 186], 95.00th=[ 199], | 99.00th=[ 245], 99.50th=[ 271], 99.90th=[ 376], 99.95th=[ 376], | 99.99th=[ 376] bw ( KiB/s): min= 295, max= 608, per=25.11%, avg=438.72, stdev=70.82, samples=46 iops : min= 73, max= 152, avg=109.39, stdev=17.77, samples=46 lat (msec) : 50=0.08%, 100=2.89%, 250=96.29%, 500=0.74% cpu : usr=0.20%, sys=1.09%, ctx=2812, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=182214: Fri Sep 11 10:47:07 2026 write: IOPS=109, BW=437KiB/s (447kB/s)(10.0MiB/23434msec); 0 zone resets slat (usec): min=29, max=15482, avg=224.14, stdev=659.18 clat (msec): min=22, max=279, avg=145.64, stdev=31.13 lat (msec): min=22, max=279, avg=145.87, stdev=31.12 clat percentiles (msec): | 1.00th=[ 91], 5.00th=[ 106], 10.00th=[ 111], 20.00th=[ 121], | 30.00th=[ 128], 40.00th=[ 133], 50.00th=[ 140], 60.00th=[ 150], | 70.00th=[ 161], 80.00th=[ 169], 90.00th=[ 188], 95.00th=[ 203], | 99.00th=[ 234], 99.50th=[ 271], 99.90th=[ 275], 99.95th=[ 279], | 99.99th=[ 279] bw ( KiB/s): min= 287, max= 604, per=25.01%, avg=436.85, stdev=69.78, samples=46 iops : min= 71, max= 151, avg=108.89, stdev=17.53, samples=46 lat (msec) : 50=0.20%, 100=2.89%, 250=96.05%, 500=0.86% cpu : usr=0.22%, sys=1.14%, ctx=2842, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=182215: Fri Sep 11 10:47:07 2026 write: IOPS=109, BW=438KiB/s (449kB/s)(10.0MiB/23357msec); 0 zone resets slat (usec): min=29, max=17754, avg=255.37, stdev=739.28 clat (msec): min=31, max=291, avg=145.50, stdev=31.40 lat (msec): min=31, max=291, avg=145.75, stdev=31.39 clat percentiles (msec): | 1.00th=[ 84], 5.00th=[ 106], 10.00th=[ 112], 20.00th=[ 121], | 30.00th=[ 127], 40.00th=[ 134], 50.00th=[ 140], 60.00th=[ 150], | 70.00th=[ 159], 80.00th=[ 171], 90.00th=[ 186], 95.00th=[ 201], | 99.00th=[ 241], 99.50th=[ 262], 99.90th=[ 292], 99.95th=[ 292], | 99.99th=[ 292] bw ( KiB/s): min= 270, max= 616, per=25.14%, avg=439.11, stdev=71.38, samples=46 iops : min= 67, max= 154, avg=109.48, stdev=17.93, samples=46 lat (msec) : 50=0.12%, 100=2.73%, 250=96.52%, 500=0.62% cpu : usr=0.23%, sys=1.42%, ctx=3046, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1748KiB/s (1790kB/s), 437KiB/s-438KiB/s (447kB/s-449kB/s), io=40.0MiB (41.9MB), run=23357-23434msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=182229: Fri Sep 11 10:47:40 2026 read: IOPS=48, BW=193KiB/s (198kB/s)(5048KiB/26119msec) slat (usec): min=37, max=45600, avg=11104.53, stdev=5624.78 clat (msec): min=23, max=353, avg=154.89, stdev=36.45 lat (msec): min=33, max=359, avg=166.00, stdev=38.09 clat percentiles (msec): | 1.00th=[ 84], 5.00th=[ 103], 10.00th=[ 112], 20.00th=[ 126], | 30.00th=[ 136], 40.00th=[ 144], 50.00th=[ 150], 60.00th=[ 161], | 70.00th=[ 171], 80.00th=[ 182], 90.00th=[ 201], 95.00th=[ 222], | 99.00th=[ 255], 99.50th=[ 275], 99.90th=[ 342], 99.95th=[ 355], | 99.99th=[ 355] bw ( KiB/s): min= 111, max= 295, per=25.13%, avg=193.02, stdev=39.07, samples=51 iops : min= 27, max= 73, avg=47.71, stdev= 9.80, samples=51 write: IOPS=49, BW=199KiB/s (204kB/s)(5192KiB/26119msec); 0 zone resets slat (usec): min=33, max=9667, avg=257.98, stdev=628.67 clat (msec): min=44, max=370, avg=159.87, stdev=37.01 lat (msec): min=44, max=370, avg=160.13, stdev=37.03 clat percentiles (msec): | 1.00th=[ 88], 5.00th=[ 107], 10.00th=[ 118], 20.00th=[ 131], | 30.00th=[ 140], 40.00th=[ 148], 50.00th=[ 157], 60.00th=[ 165], | 70.00th=[ 176], 80.00th=[ 190], 90.00th=[ 207], 95.00th=[ 224], | 99.00th=[ 262], 99.50th=[ 292], 99.90th=[ 355], 99.95th=[ 372], | 99.99th=[ 372] bw ( KiB/s): min= 120, max= 311, per=24.92%, avg=199.10, stdev=39.60, samples=51 iops : min= 30, max= 77, avg=49.24, stdev= 9.88, samples=51 lat (msec) : 50=0.20%, 100=3.67%, 250=94.77%, 500=1.37% cpu : usr=0.28%, sys=2.13%, ctx=4674, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=182230: Fri Sep 11 10:47:40 2026 read: IOPS=47, BW=190KiB/s (195kB/s)(4976KiB/26126msec) slat (usec): min=3059, max=58152, avg=10972.00, stdev=5325.73 clat (msec): min=65, max=333, avg=155.67, stdev=36.05 lat (msec): min=73, max=350, avg=166.65, stdev=37.46 clat percentiles (msec): | 1.00th=[ 90], 5.00th=[ 105], 10.00th=[ 113], 20.00th=[ 125], | 30.00th=[ 136], 40.00th=[ 144], 50.00th=[ 153], 60.00th=[ 161], | 70.00th=[ 169], 80.00th=[ 184], 90.00th=[ 203], 95.00th=[ 228], | 99.00th=[ 255], 99.50th=[ 266], 99.90th=[ 292], 99.95th=[ 334], | 99.99th=[ 334] bw ( KiB/s): min= 103, max= 279, per=24.68%, avg=189.51, stdev=41.77, samples=51 iops : min= 25, max= 69, avg=46.84, stdev=10.48, samples=51 write: IOPS=50, BW=201KiB/s (206kB/s)(5264KiB/26126msec); 0 zone resets slat (usec): min=33, max=10149, avg=272.99, stdev=701.80 clat (msec): min=41, max=315, avg=158.84, stdev=34.54 lat (msec): min=41, max=315, avg=159.12, stdev=34.57 clat percentiles (msec): | 1.00th=[ 92], 5.00th=[ 111], 10.00th=[ 120], 20.00th=[ 129], | 30.00th=[ 138], 40.00th=[ 148], 50.00th=[ 157], 60.00th=[ 165], | 70.00th=[ 174], 80.00th=[ 186], 90.00th=[ 207], 95.00th=[ 224], | 99.00th=[ 249], 99.50th=[ 253], 99.90th=[ 271], 99.95th=[ 317], | 99.99th=[ 317] bw ( KiB/s): min= 119, max= 294, per=25.27%, avg=201.94, stdev=42.20, samples=51 iops : min= 29, max= 73, avg=49.96, stdev=10.58, samples=51 lat (msec) : 50=0.04%, 100=2.62%, 250=96.37%, 500=0.98% cpu : usr=0.30%, sys=1.91%, ctx=4734, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=182231: Fri Sep 11 10:47:40 2026 read: IOPS=47, BW=191KiB/s (195kB/s)(4984KiB/26117msec) slat (usec): min=3044, max=40954, avg=11141.04, stdev=5607.99 clat (msec): min=44, max=310, avg=153.72, stdev=34.73 lat (msec): min=49, max=323, avg=164.86, stdev=36.26 clat percentiles (msec): | 1.00th=[ 87], 5.00th=[ 104], 10.00th=[ 113], 20.00th=[ 126], | 30.00th=[ 136], 40.00th=[ 142], 50.00th=[ 150], 60.00th=[ 159], | 70.00th=[ 169], 80.00th=[ 184], 90.00th=[ 201], 95.00th=[ 218], | 99.00th=[ 249], 99.50th=[ 255], 99.90th=[ 292], 99.95th=[ 313], | 99.99th=[ 313] bw ( KiB/s): min= 111, max= 312, per=24.92%, avg=191.35, stdev=41.82, samples=51 iops : min= 27, max= 78, avg=47.29, stdev=10.48, samples=51 write: IOPS=50, BW=201KiB/s (206kB/s)(5256KiB/26117msec); 0 zone resets slat (usec): min=31, max=9956, avg=283.75, stdev=710.22 clat (msec): min=38, max=372, avg=160.43, stdev=38.11 lat (msec): min=38, max=372, avg=160.71, stdev=38.12 clat percentiles (msec): | 1.00th=[ 85], 5.00th=[ 109], 10.00th=[ 118], 20.00th=[ 130], | 30.00th=[ 140], 40.00th=[ 148], 50.00th=[ 157], 60.00th=[ 167], | 70.00th=[ 178], 80.00th=[ 190], 90.00th=[ 207], 95.00th=[ 228], | 99.00th=[ 259], 99.50th=[ 300], 99.90th=[ 372], 99.95th=[ 372], | 99.99th=[ 372] bw ( KiB/s): min= 103, max= 310, per=25.12%, avg=200.71, stdev=44.46, samples=51 iops : min= 25, max= 77, avg=49.65, stdev=11.11, samples=51 lat (msec) : 50=0.12%, 100=3.20%, 250=95.39%, 500=1.29% cpu : usr=0.32%, sys=2.01%, ctx=4700, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=182232: Fri Sep 11 10:47:40 2026 read: IOPS=48, BW=194KiB/s (198kB/s)(5060KiB/26117msec) slat (usec): min=2588, max=43452, avg=11084.65, stdev=5708.29 clat (msec): min=69, max=286, avg=154.45, stdev=36.96 lat (msec): min=83, max=304, avg=165.54, stdev=38.41 clat percentiles (msec): | 1.00th=[ 89], 5.00th=[ 104], 10.00th=[ 111], 20.00th=[ 124], | 30.00th=[ 133], 40.00th=[ 142], 50.00th=[ 150], 60.00th=[ 159], | 70.00th=[ 169], 80.00th=[ 184], 90.00th=[ 207], 95.00th=[ 228], | 99.00th=[ 257], 99.50th=[ 271], 99.90th=[ 284], 99.95th=[ 288], | 99.99th=[ 288] bw ( KiB/s): min= 119, max= 304, per=25.27%, avg=194.06, stdev=37.68, samples=51 iops : min= 29, max= 76, avg=48.00, stdev= 9.43, samples=51 write: IOPS=49, BW=198KiB/s (203kB/s)(5180KiB/26117msec); 0 zone resets slat (usec): min=37, max=19739, avg=285.17, stdev=821.65 clat (msec): min=71, max=391, avg=159.28, stdev=35.21 lat (msec): min=71, max=391, avg=159.57, stdev=35.24 clat percentiles (msec): | 1.00th=[ 88], 5.00th=[ 109], 10.00th=[ 121], 20.00th=[ 130], | 30.00th=[ 140], 40.00th=[ 146], 50.00th=[ 155], 60.00th=[ 165], | 70.00th=[ 176], 80.00th=[ 188], 90.00th=[ 205], 95.00th=[ 222], | 99.00th=[ 255], 99.50th=[ 257], 99.90th=[ 342], 99.95th=[ 393], | 99.99th=[ 393] bw ( KiB/s): min= 103, max= 288, per=24.81%, avg=198.24, stdev=42.11, samples=51 iops : min= 25, max= 72, avg=49.02, stdev=10.56, samples=51 lat (msec) : 100=3.05%, 250=95.62%, 500=1.33% cpu : usr=0.26%, sys=2.11%, ctx=4852, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=768KiB/s (787kB/s), 190KiB/s-194KiB/s (195kB/s-198kB/s), io=19.6MiB (20.5MB), run=26117-26126msec WRITE: bw=800KiB/s (819kB/s), 198KiB/s-201KiB/s (203kB/s-206kB/s), io=20.4MiB (21.4MB), run=26117-26126msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=182240: Fri Sep 11 10:47:47 2026 read: IOPS=2, BW=118MiB/s (124MB/s)(40.0MiB/339msec) slat (nsec): min=30031k, max=30031k, avg=30031364.00, stdev= 0.00 clat (nsec): min=290365k, max=290365k, avg=290364568.00, stdev= 0.00 lat (nsec): min=320400k, max=320400k, avg=320400280.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 292], 5.00th=[ 292], 10.00th=[ 292], 20.00th=[ 292], | 30.00th=[ 292], 40.00th=[ 292], 50.00th=[ 292], 60.00th=[ 292], | 70.00th=[ 292], 80.00th=[ 292], 90.00th=[ 292], 95.00th=[ 292], | 99.00th=[ 292], 99.50th=[ 292], 99.90th=[ 292], 99.95th=[ 292], | 99.99th=[ 292] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=3.85%, ctx=10, majf=0, minf=7 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=118MiB/s (124MB/s), 118MiB/s-118MiB/s (124MB/s-124MB/s), io=40.0MiB (41.9MB), run=339-339msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout