== sanity test 398c: run fio to test AIO ================= 11:23:26 (1788535406) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.736092 s, 57.0 MB/s osc.lustre-OST0000-osc-ffff9cb54fb64800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=180351: Fri Sep 4 11:24:13 2026 write: IOPS=68, BW=272KiB/s (279kB/s)(10.0MiB/37611msec); 0 zone resets slat (usec): min=22, max=22096, avg=248.01, stdev=1059.24 clat (msec): min=92, max=520, avg=234.21, stdev=68.00 lat (msec): min=92, max=520, avg=234.46, stdev=68.00 clat percentiles (msec): | 1.00th=[ 109], 5.00th=[ 133], 10.00th=[ 148], 20.00th=[ 176], | 30.00th=[ 194], 40.00th=[ 215], 50.00th=[ 228], 60.00th=[ 247], | 70.00th=[ 264], 80.00th=[ 284], 90.00th=[ 321], 95.00th=[ 363], | 99.00th=[ 422], 99.50th=[ 435], 99.90th=[ 439], 99.95th=[ 518], | 99.99th=[ 523] bw ( KiB/s): min= 127, max= 512, per=24.92%, avg=270.86, stdev=69.60, samples=74 iops : min= 31, max= 128, avg=67.24, stdev=17.49, samples=74 lat (msec) : 100=0.62%, 250=60.94%, 500=38.36%, 750=0.08% cpu : usr=0.10%, sys=0.69%, ctx=1746, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=180352: Fri Sep 4 11:24:13 2026 write: IOPS=67, BW=272KiB/s (278kB/s)(10.0MiB/37658msec); 0 zone resets slat (usec): min=27, max=38189, avg=268.19, stdev=1238.65 clat (msec): min=22, max=499, avg=234.18, stdev=69.34 lat (msec): min=22, max=499, avg=234.45, stdev=69.35 clat percentiles (msec): | 1.00th=[ 102], 5.00th=[ 134], 10.00th=[ 148], 20.00th=[ 178], | 30.00th=[ 197], 40.00th=[ 211], 50.00th=[ 228], 60.00th=[ 247], | 70.00th=[ 266], 80.00th=[ 288], 90.00th=[ 326], 95.00th=[ 359], | 99.00th=[ 430], 99.50th=[ 439], 99.90th=[ 493], 99.95th=[ 502], | 99.99th=[ 502] bw ( KiB/s): min= 127, max= 480, per=24.85%, avg=270.16, stdev=70.61, samples=74 iops : min= 31, max= 120, avg=67.05, stdev=17.75, samples=74 lat (msec) : 50=0.12%, 100=0.74%, 250=61.37%, 500=37.77% cpu : usr=0.17%, sys=0.82%, ctx=2483, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=180353: Fri Sep 4 11:24:13 2026 write: IOPS=68, BW=273KiB/s (280kB/s)(10.0MiB/37491msec); 0 zone resets slat (usec): min=22, max=26959, avg=267.09, stdev=1089.33 clat (msec): min=25, max=504, avg=233.87, stdev=70.00 lat (msec): min=27, max=505, avg=234.13, stdev=70.03 clat percentiles (msec): | 1.00th=[ 101], 5.00th=[ 130], 10.00th=[ 148], 20.00th=[ 174], | 30.00th=[ 194], 40.00th=[ 211], 50.00th=[ 226], 60.00th=[ 247], | 70.00th=[ 264], 80.00th=[ 288], 90.00th=[ 334], 95.00th=[ 363], | 99.00th=[ 430], 99.50th=[ 439], 99.90th=[ 506], 99.95th=[ 506], | 99.99th=[ 506] bw ( KiB/s): min= 135, max= 496, per=25.07%, avg=272.47, stdev=68.76, samples=74 iops : min= 33, max= 124, avg=67.64, stdev=17.31, samples=74 lat (msec) : 50=0.12%, 100=0.82%, 250=61.95%, 500=36.88%, 750=0.23% cpu : usr=0.19%, sys=0.81%, ctx=2655, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=180354: Fri Sep 4 11:24:13 2026 write: IOPS=68, BW=273KiB/s (279kB/s)(10.0MiB/37522msec); 0 zone resets slat (usec): min=26, max=16290, avg=208.62, stdev=817.09 clat (msec): min=17, max=509, avg=233.76, stdev=68.84 lat (msec): min=18, max=509, avg=233.98, stdev=68.86 clat percentiles (msec): | 1.00th=[ 99], 5.00th=[ 133], 10.00th=[ 153], 20.00th=[ 174], | 30.00th=[ 197], 40.00th=[ 215], 50.00th=[ 226], 60.00th=[ 243], | 70.00th=[ 271], 80.00th=[ 288], 90.00th=[ 321], 95.00th=[ 355], | 99.00th=[ 422], 99.50th=[ 477], 99.90th=[ 510], 99.95th=[ 510], | 99.99th=[ 510] bw ( KiB/s): min= 123, max= 456, per=25.08%, avg=272.62, stdev=77.31, samples=74 iops : min= 30, max= 114, avg=67.68, stdev=19.46, samples=74 lat (msec) : 20=0.04%, 50=0.08%, 100=1.05%, 250=62.97%, 500=35.55% lat (msec) : 750=0.31% cpu : usr=0.10%, sys=0.69%, ctx=1641, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1088KiB/s (1114kB/s), 272KiB/s-273KiB/s (278kB/s-280kB/s), io=40.0MiB (41.9MB), run=37491-37658msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=180370: Fri Sep 4 11:25:01 2026 read: IOPS=34, BW=139KiB/s (142kB/s)(5048KiB/36402msec) slat (usec): min=3622, max=86257, avg=16297.35, stdev=9549.65 clat (msec): min=26, max=409, avg=213.30, stdev=51.31 lat (msec): min=54, max=427, avg=229.60, stdev=53.24 clat percentiles (msec): | 1.00th=[ 113], 5.00th=[ 140], 10.00th=[ 153], 20.00th=[ 171], | 30.00th=[ 186], 40.00th=[ 197], 50.00th=[ 207], 60.00th=[ 220], | 70.00th=[ 234], 80.00th=[ 253], 90.00th=[ 284], 95.00th=[ 305], | 99.00th=[ 355], 99.50th=[ 372], 99.90th=[ 409], 99.95th=[ 409], | 99.99th=[ 409] bw ( KiB/s): min= 38, max= 239, per=25.06%, avg=137.85, stdev=34.87, samples=72 iops : min= 9, max= 59, avg=33.92, stdev= 8.74, samples=72 write: IOPS=35, BW=143KiB/s (146kB/s)(5192KiB/36402msec); 0 zone resets slat (usec): min=29, max=18871, avg=316.93, stdev=1020.13 clat (msec): min=62, max=484, avg=221.70, stdev=51.49 lat (msec): min=62, max=484, avg=222.02, stdev=51.47 clat percentiles (msec): | 1.00th=[ 124], 5.00th=[ 150], 10.00th=[ 163], 20.00th=[ 180], | 30.00th=[ 192], 40.00th=[ 203], 50.00th=[ 215], 60.00th=[ 228], | 70.00th=[ 243], 80.00th=[ 264], 90.00th=[ 292], 95.00th=[ 317], | 99.00th=[ 355], 99.50th=[ 384], 99.90th=[ 477], 99.95th=[ 485], | 99.99th=[ 485] bw ( KiB/s): min= 7, max= 239, per=24.83%, avg=142.29, stdev=38.74, samples=72 iops : min= 1, max= 59, avg=35.03, stdev= 9.73, samples=72 lat (msec) : 50=0.08%, 100=0.16%, 250=75.98%, 500=23.79% cpu : usr=0.27%, sys=1.55%, ctx=4582, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=180371: Fri Sep 4 11:25:01 2026 read: IOPS=34, BW=137KiB/s (140kB/s)(4976KiB/36406msec) slat (usec): min=3087, max=66627, avg=16376.05, stdev=8986.52 clat (msec): min=71, max=415, avg=214.26, stdev=51.78 lat (msec): min=102, max=440, avg=230.65, stdev=53.69 clat percentiles (msec): | 1.00th=[ 116], 5.00th=[ 144], 10.00th=[ 157], 20.00th=[ 171], | 30.00th=[ 186], 40.00th=[ 194], 50.00th=[ 207], 60.00th=[ 220], | 70.00th=[ 236], 80.00th=[ 255], 90.00th=[ 288], 95.00th=[ 317], | 99.00th=[ 351], 99.50th=[ 372], 99.90th=[ 414], 99.95th=[ 418], | 99.99th=[ 418] bw ( KiB/s): min= 22, max= 223, per=24.72%, avg=135.97, stdev=35.99, samples=72 iops : min= 5, max= 55, avg=33.44, stdev= 9.08, samples=72 write: IOPS=36, BW=145KiB/s (148kB/s)(5264KiB/36406msec); 0 zone resets slat (usec): min=33, max=11014, avg=257.62, stdev=712.30 clat (msec): min=21, max=406, avg=221.36, stdev=49.68 lat (msec): min=21, max=406, avg=221.62, stdev=49.65 clat percentiles (msec): | 1.00th=[ 130], 5.00th=[ 153], 10.00th=[ 165], 20.00th=[ 180], | 30.00th=[ 192], 40.00th=[ 205], 50.00th=[ 215], 60.00th=[ 228], | 70.00th=[ 243], 80.00th=[ 262], 90.00th=[ 292], 95.00th=[ 317], | 99.00th=[ 355], 99.50th=[ 376], 99.90th=[ 401], 99.95th=[ 409], | 99.99th=[ 409] bw ( KiB/s): min= 64, max= 239, per=25.19%, avg=144.33, stdev=37.71, samples=72 iops : min= 16, max= 59, avg=35.56, stdev= 9.41, samples=72 lat (msec) : 50=0.04%, 100=0.16%, 250=75.90%, 500=23.91% cpu : usr=0.25%, sys=1.67%, ctx=4365, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=180372: Fri Sep 4 11:25:01 2026 read: IOPS=34, BW=137KiB/s (140kB/s)(4984KiB/36412msec) slat (usec): min=54, max=77241, avg=16465.40, stdev=9712.08 clat (msec): min=12, max=463, avg=214.86, stdev=54.38 lat (msec): min=13, max=486, avg=231.33, stdev=56.37 clat percentiles (msec): | 1.00th=[ 105], 5.00th=[ 140], 10.00th=[ 155], 20.00th=[ 171], | 30.00th=[ 184], 40.00th=[ 197], 50.00th=[ 209], 60.00th=[ 222], | 70.00th=[ 236], 80.00th=[ 255], 90.00th=[ 288], 95.00th=[ 313], | 99.00th=[ 363], 99.50th=[ 393], 99.90th=[ 464], 99.95th=[ 464], | 99.99th=[ 464] bw ( KiB/s): min= 79, max= 239, per=24.85%, avg=136.67, stdev=35.69, samples=72 iops : min= 19, max= 59, avg=33.62, stdev= 8.90, samples=72 write: IOPS=36, BW=144KiB/s (148kB/s)(5256KiB/36412msec); 0 zone resets slat (usec): min=29, max=11870, avg=262.15, stdev=825.98 clat (msec): min=41, max=391, avg=222.95, stdev=49.42 lat (msec): min=47, max=391, avg=223.22, stdev=49.33 clat percentiles (msec): | 1.00th=[ 125], 5.00th=[ 150], 10.00th=[ 167], 20.00th=[ 182], | 30.00th=[ 194], 40.00th=[ 207], 50.00th=[ 218], 60.00th=[ 230], | 70.00th=[ 243], 80.00th=[ 262], 90.00th=[ 292], 95.00th=[ 313], | 99.00th=[ 355], 99.50th=[ 363], 99.90th=[ 380], 99.95th=[ 393], | 99.99th=[ 393] bw ( KiB/s): min= 71, max= 216, per=25.03%, avg=143.40, stdev=33.53, samples=72 iops : min= 17, max= 54, avg=35.31, stdev= 8.46, samples=72 lat (msec) : 20=0.08%, 50=0.04%, 100=0.43%, 250=74.96%, 500=24.49% cpu : usr=0.20%, sys=1.79%, ctx=4347, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=180373: Fri Sep 4 11:25:01 2026 read: IOPS=34, BW=139KiB/s (142kB/s)(5060KiB/36426msec) slat (usec): min=2980, max=67520, avg=16288.26, stdev=9074.17 clat (msec): min=40, max=494, avg=212.24, stdev=52.54 lat (msec): min=61, max=535, avg=228.54, stdev=54.84 clat percentiles (msec): | 1.00th=[ 112], 5.00th=[ 136], 10.00th=[ 153], 20.00th=[ 171], | 30.00th=[ 184], 40.00th=[ 194], 50.00th=[ 205], 60.00th=[ 218], | 70.00th=[ 232], 80.00th=[ 251], 90.00th=[ 279], 95.00th=[ 309], | 99.00th=[ 363], 99.50th=[ 384], 99.90th=[ 464], 99.95th=[ 493], | 99.99th=[ 493] bw ( KiB/s): min= 7, max= 222, per=25.11%, avg=138.11, stdev=36.81, samples=72 iops : min= 1, max= 55, avg=33.97, stdev= 9.25, samples=72 write: IOPS=35, BW=142KiB/s (146kB/s)(5180KiB/36426msec); 0 zone resets slat (usec): min=33, max=18659, avg=278.84, stdev=1008.82 clat (msec): min=81, max=445, avg=222.60, stdev=52.23 lat (msec): min=82, max=445, avg=222.88, stdev=52.20 clat percentiles (msec): | 1.00th=[ 126], 5.00th=[ 150], 10.00th=[ 161], 20.00th=[ 178], | 30.00th=[ 192], 40.00th=[ 205], 50.00th=[ 218], 60.00th=[ 228], | 70.00th=[ 243], 80.00th=[ 264], 90.00th=[ 296], 95.00th=[ 321], | 99.00th=[ 368], 99.50th=[ 376], 99.90th=[ 439], 99.95th=[ 447], | 99.99th=[ 447] bw ( KiB/s): min= 30, max= 230, per=24.62%, avg=141.07, stdev=37.00, samples=72 iops : min= 7, max= 57, avg=34.72, stdev= 9.29, samples=72 lat (msec) : 50=0.04%, 100=0.31%, 250=76.80%, 500=22.85% cpu : usr=0.29%, sys=1.64%, ctx=4327, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=551KiB/s (564kB/s), 137KiB/s-139KiB/s (140kB/s-142kB/s), io=19.6MiB (20.5MB), run=36402-36426msec WRITE: bw=574KiB/s (587kB/s), 142KiB/s-145KiB/s (146kB/s-148kB/s), io=20.4MiB (21.4MB), run=36402-36426msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=180383: Fri Sep 4 11:25:14 2026 read: IOPS=1, BW=64.1MiB/s (67.2MB/s)(40.0MiB/624msec) slat (nsec): min=51274k, max=51274k, avg=51274170.00, stdev= 0.00 clat (nsec): min=547630k, max=547630k, avg=547630088.00, stdev= 0.00 lat (nsec): min=598910k, max=598910k, avg=598910321.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 550], 5.00th=[ 550], 10.00th=[ 550], 20.00th=[ 550], | 30.00th=[ 550], 40.00th=[ 550], 50.00th=[ 550], 60.00th=[ 550], | 70.00th=[ 550], 80.00th=[ 550], 90.00th=[ 550], 95.00th=[ 550], | 99.00th=[ 550], 99.50th=[ 550], 99.90th=[ 550], 99.95th=[ 550], | 99.99th=[ 550] lat (msec) : 750=100.00% cpu : usr=0.00%, sys=2.57%, ctx=4, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=64.1MiB/s (67.2MB/s), 64.1MiB/s-64.1MiB/s (67.2MB/s-67.2MB/s), io=40.0MiB (41.9MB), run=624-624msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout