== sanity test 398c: run fio to test AIO ================= 13:19:03 (1763489943) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.162534 s, 258 MB/s osc.lustre-OST0000-osc-ffff9aae45010000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=147341: Tue Nov 18 13:19:51 2025 write: IOPS=57, BW=229KiB/s (235kB/s)(10.0MiB/44651msec); 0 zone resets slat (usec): min=23, max=2012, avg=130.92, stdev=101.42 clat (msec): min=88, max=557, avg=278.61, stdev=61.08 lat (msec): min=88, max=557, avg=278.74, stdev=61.08 clat percentiles (msec): | 1.00th=[ 95], 5.00th=[ 188], 10.00th=[ 205], 20.00th=[ 230], | 30.00th=[ 249], 40.00th=[ 264], 50.00th=[ 275], 60.00th=[ 292], | 70.00th=[ 305], 80.00th=[ 330], 90.00th=[ 359], 95.00th=[ 380], | 99.00th=[ 422], 99.50th=[ 439], 99.90th=[ 535], 99.95th=[ 558], | 99.99th=[ 558] bw ( KiB/s): min= 144, max= 413, per=24.98%, avg=229.10, stdev=43.23, samples=88 iops : min= 36, max= 103, avg=57.17, stdev=10.89, samples=88 lat (msec) : 100=1.25%, 250=30.47%, 500=68.09%, 750=0.20% cpu : usr=0.10%, sys=0.63%, ctx=2967, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=147342: Tue Nov 18 13:19:51 2025 write: IOPS=57, BW=230KiB/s (235kB/s)(10.0MiB/44607msec); 0 zone resets slat (usec): min=22, max=1195, avg=124.61, stdev=84.98 clat (msec): min=49, max=652, avg=278.48, stdev=63.62 lat (msec): min=49, max=652, avg=278.61, stdev=63.63 clat percentiles (msec): | 1.00th=[ 101], 5.00th=[ 184], 10.00th=[ 205], 20.00th=[ 230], | 30.00th=[ 247], 40.00th=[ 262], 50.00th=[ 275], 60.00th=[ 292], | 70.00th=[ 309], 80.00th=[ 330], 90.00th=[ 363], 95.00th=[ 380], | 99.00th=[ 418], 99.50th=[ 430], 99.90th=[ 567], 99.95th=[ 634], | 99.99th=[ 651] bw ( KiB/s): min= 136, max= 516, per=25.01%, avg=229.36, stdev=50.90, samples=88 iops : min= 34, max= 129, avg=57.24, stdev=12.80, samples=88 lat (msec) : 50=0.62%, 250=31.99%, 500=67.15%, 750=0.23% cpu : usr=0.13%, sys=0.59%, ctx=2836, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=147343: Tue Nov 18 13:19:51 2025 write: IOPS=57, BW=229KiB/s (235kB/s)(10.0MiB/44651msec); 0 zone resets slat (usec): min=26, max=1892, avg=131.75, stdev=101.85 clat (msec): min=93, max=465, avg=278.66, stdev=59.97 lat (msec): min=93, max=466, avg=278.79, stdev=59.97 clat percentiles (msec): | 1.00th=[ 114], 5.00th=[ 190], 10.00th=[ 207], 20.00th=[ 230], | 30.00th=[ 247], 40.00th=[ 262], 50.00th=[ 275], 60.00th=[ 292], | 70.00th=[ 309], 80.00th=[ 330], 90.00th=[ 359], 95.00th=[ 384], | 99.00th=[ 414], 99.50th=[ 426], 99.90th=[ 451], 99.95th=[ 451], | 99.99th=[ 468] bw ( KiB/s): min= 135, max= 414, per=24.94%, avg=228.66, stdev=44.36, samples=88 iops : min= 33, max= 103, avg=57.06, stdev=11.17, samples=88 lat (msec) : 100=0.94%, 250=32.30%, 500=66.76% cpu : usr=0.11%, sys=0.62%, ctx=3040, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=147344: Tue Nov 18 13:19:51 2025 write: IOPS=57, BW=229KiB/s (235kB/s)(10.0MiB/44651msec); 0 zone resets slat (usec): min=33, max=2401, avg=135.31, stdev=114.24 clat (msec): min=43, max=642, avg=278.64, stdev=62.41 lat (msec): min=43, max=643, avg=278.78, stdev=62.41 clat percentiles (msec): | 1.00th=[ 99], 5.00th=[ 188], 10.00th=[ 205], 20.00th=[ 230], | 30.00th=[ 247], 40.00th=[ 264], 50.00th=[ 275], 60.00th=[ 292], | 70.00th=[ 309], 80.00th=[ 330], 90.00th=[ 359], 95.00th=[ 380], | 99.00th=[ 422], 99.50th=[ 447], 99.90th=[ 542], 99.95th=[ 609], | 99.99th=[ 642] bw ( KiB/s): min= 144, max= 413, per=25.00%, avg=229.28, stdev=44.32, samples=88 iops : min= 36, max= 103, avg=57.22, stdev=11.15, samples=88 lat (msec) : 50=0.04%, 100=1.13%, 250=32.23%, 500=66.41%, 750=0.20% cpu : usr=0.10%, sys=0.66%, ctx=2987, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=917KiB/s (939kB/s), 229KiB/s-230KiB/s (235kB/s-235kB/s), io=40.0MiB (41.9MB), run=44607-44651msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=147360: Tue Nov 18 13:20:23 2025 read: IOPS=44, BW=180KiB/s (184kB/s)(5048KiB/28057msec) slat (usec): min=33, max=10536, avg=3224.71, stdev=1110.60 clat (msec): min=3, max=277, avg=112.48, stdev=26.71 lat (msec): min=5, max=282, avg=115.71, stdev=27.01 clat percentiles (msec): | 1.00th=[ 50], 5.00th=[ 78], 10.00th=[ 85], 20.00th=[ 92], | 30.00th=[ 99], 40.00th=[ 105], 50.00th=[ 110], 60.00th=[ 116], | 70.00th=[ 124], 80.00th=[ 132], 90.00th=[ 148], 95.00th=[ 157], | 99.00th=[ 186], 99.50th=[ 203], 99.90th=[ 275], 99.95th=[ 279], | 99.99th=[ 279] bw ( KiB/s): min= 79, max= 272, per=25.27%, avg=180.45, stdev=46.81, samples=55 iops : min= 19, max= 68, avg=45.04, stdev=11.67, samples=55 write: IOPS=46, BW=185KiB/s (189kB/s)(5192KiB/28057msec); 0 zone resets slat (usec): min=34, max=2777, avg=129.81, stdev=108.43 clat (msec): min=35, max=418, avg=232.94, stdev=62.24 lat (msec): min=35, max=418, avg=233.07, stdev=62.24 clat percentiles (msec): | 1.00th=[ 111], 5.00th=[ 144], 10.00th=[ 159], 20.00th=[ 178], | 30.00th=[ 197], 40.00th=[ 213], 50.00th=[ 230], 60.00th=[ 245], | 70.00th=[ 262], 80.00th=[ 292], 90.00th=[ 317], 95.00th=[ 342], | 99.00th=[ 376], 99.50th=[ 384], 99.90th=[ 414], 99.95th=[ 418], | 99.99th=[ 418] bw ( KiB/s): min= 128, max= 280, per=24.92%, avg=185.15, stdev=34.98, samples=55 iops : min= 32, max= 70, avg=46.22, stdev= 8.75, samples=55 lat (msec) : 4=0.08%, 10=0.04%, 20=0.04%, 50=0.70%, 100=16.29% lat (msec) : 250=64.34%, 500=18.52% cpu : usr=0.20%, sys=1.43%, ctx=4918, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=147361: Tue Nov 18 13:20:23 2025 read: IOPS=44, BW=177KiB/s (181kB/s)(4976KiB/28093msec) slat (usec): min=1291, max=10555, avg=3238.92, stdev=1145.48 clat (msec): min=4, max=210, avg=112.16, stdev=24.80 lat (msec): min=7, max=221, avg=115.40, stdev=25.09 clat percentiles (msec): | 1.00th=[ 67], 5.00th=[ 78], 10.00th=[ 85], 20.00th=[ 93], | 30.00th=[ 99], 40.00th=[ 105], 50.00th=[ 110], 60.00th=[ 116], | 70.00th=[ 124], 80.00th=[ 132], 90.00th=[ 146], 95.00th=[ 155], | 99.00th=[ 180], 99.50th=[ 192], 99.90th=[ 205], 99.95th=[ 211], | 99.99th=[ 211] bw ( KiB/s): min= 72, max= 296, per=24.71%, avg=176.45, stdev=50.11, samples=55 iops : min= 18, max= 74, avg=44.04, stdev=12.52, samples=55 write: IOPS=46, BW=187KiB/s (192kB/s)(5264KiB/28093msec); 0 zone resets slat (usec): min=32, max=2048, avg=130.97, stdev=98.98 clat (msec): min=50, max=398, avg=231.94, stdev=61.13 lat (msec): min=50, max=398, avg=232.08, stdev=61.14 clat percentiles (msec): | 1.00th=[ 111], 5.00th=[ 146], 10.00th=[ 159], 20.00th=[ 178], | 30.00th=[ 197], 40.00th=[ 211], 50.00th=[ 228], 60.00th=[ 243], | 70.00th=[ 264], 80.00th=[ 288], 90.00th=[ 321], 95.00th=[ 342], | 99.00th=[ 368], 99.50th=[ 384], 99.90th=[ 397], 99.95th=[ 401], | 99.99th=[ 401] bw ( KiB/s): min= 112, max= 280, per=25.08%, avg=186.35, stdev=37.20, samples=55 iops : min= 28, max= 70, avg=46.51, stdev= 9.29, samples=55 lat (msec) : 10=0.08%, 20=0.12%, 50=0.04%, 100=16.68%, 250=64.96% lat (msec) : 500=18.12% cpu : usr=0.22%, sys=1.41%, ctx=4905, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=147362: Tue Nov 18 13:20:23 2025 read: IOPS=44, BW=178KiB/s (182kB/s)(4984KiB/28077msec) slat (usec): min=1298, max=10405, avg=3262.76, stdev=1176.58 clat (msec): min=4, max=208, avg=112.36, stdev=25.15 lat (msec): min=7, max=218, avg=115.63, stdev=25.42 clat percentiles (msec): | 1.00th=[ 59], 5.00th=[ 79], 10.00th=[ 85], 20.00th=[ 93], | 30.00th=[ 99], 40.00th=[ 106], 50.00th=[ 111], 60.00th=[ 117], | 70.00th=[ 124], 80.00th=[ 132], 90.00th=[ 146], 95.00th=[ 157], | 99.00th=[ 176], 99.50th=[ 186], 99.90th=[ 203], 99.95th=[ 209], | 99.99th=[ 209] bw ( KiB/s): min= 88, max= 312, per=24.98%, avg=178.33, stdev=48.56, samples=55 iops : min= 22, max= 78, avg=44.51, stdev=12.13, samples=55 write: IOPS=46, BW=187KiB/s (192kB/s)(5256KiB/28077msec); 0 zone resets slat (usec): min=31, max=1513, avg=132.39, stdev=94.41 clat (msec): min=66, max=408, avg=231.53, stdev=60.13 lat (msec): min=66, max=408, avg=231.66, stdev=60.14 clat percentiles (msec): | 1.00th=[ 116], 5.00th=[ 142], 10.00th=[ 159], 20.00th=[ 176], | 30.00th=[ 194], 40.00th=[ 213], 50.00th=[ 228], 60.00th=[ 243], | 70.00th=[ 259], 80.00th=[ 288], 90.00th=[ 317], 95.00th=[ 338], | 99.00th=[ 372], 99.50th=[ 384], 99.90th=[ 409], 99.95th=[ 409], | 99.99th=[ 409] bw ( KiB/s): min= 112, max= 256, per=24.99%, avg=185.67, stdev=32.88, samples=55 iops : min= 28, max= 64, avg=46.35, stdev= 8.20, samples=55 lat (msec) : 10=0.08%, 20=0.12%, 50=0.23%, 100=16.02%, 250=65.51% lat (msec) : 500=18.05% cpu : usr=0.21%, sys=1.40%, ctx=5033, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=147363: Tue Nov 18 13:20:23 2025 read: IOPS=45, BW=180KiB/s (185kB/s)(5060KiB/28077msec) slat (usec): min=1203, max=8678, avg=3258.58, stdev=1129.28 clat (msec): min=5, max=243, avg=113.26, stdev=26.20 lat (msec): min=8, max=247, avg=116.52, stdev=26.47 clat percentiles (msec): | 1.00th=[ 66], 5.00th=[ 79], 10.00th=[ 85], 20.00th=[ 92], | 30.00th=[ 99], 40.00th=[ 106], 50.00th=[ 111], 60.00th=[ 117], | 70.00th=[ 124], 80.00th=[ 133], 90.00th=[ 148], 95.00th=[ 157], | 99.00th=[ 192], 99.50th=[ 209], 99.90th=[ 241], 99.95th=[ 245], | 99.99th=[ 245] bw ( KiB/s): min= 96, max= 304, per=25.36%, avg=181.04, stdev=42.34, samples=55 iops : min= 24, max= 76, avg=45.18, stdev=10.58, samples=55 write: IOPS=46, BW=184KiB/s (189kB/s)(5180KiB/28077msec); 0 zone resets slat (usec): min=36, max=1162, avg=126.34, stdev=76.69 clat (msec): min=90, max=407, avg=232.25, stdev=58.92 lat (msec): min=91, max=407, avg=232.38, stdev=58.92 clat percentiles (msec): | 1.00th=[ 125], 5.00th=[ 144], 10.00th=[ 159], 20.00th=[ 180], | 30.00th=[ 197], 40.00th=[ 213], 50.00th=[ 228], 60.00th=[ 243], | 70.00th=[ 262], 80.00th=[ 284], 90.00th=[ 317], 95.00th=[ 338], | 99.00th=[ 372], 99.50th=[ 380], 99.90th=[ 388], 99.95th=[ 409], | 99.99th=[ 409] bw ( KiB/s): min= 80, max= 248, per=24.75%, avg=183.89, stdev=32.40, samples=55 iops : min= 20, max= 62, avg=45.89, stdev= 8.08, samples=55 lat (msec) : 10=0.04%, 20=0.12%, 50=0.16%, 100=15.86%, 250=66.05% lat (msec) : 500=17.77% cpu : usr=0.18%, sys=1.45%, ctx=4950, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=714KiB/s (731kB/s), 177KiB/s-180KiB/s (181kB/s-185kB/s), io=19.6MiB (20.5MB), run=28057-28093msec WRITE: bw=744KiB/s (762kB/s), 184KiB/s-187KiB/s (189kB/s-192kB/s), io=20.4MiB (21.4MB), run=28057-28093msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=147371: Tue Nov 18 13:20:27 2025 read: IOPS=10, BW=417MiB/s (437MB/s)(40.0MiB/96msec) slat (nsec): min=10734k, max=10734k, avg=10733914.00, stdev= 0.00 clat (nsec): min=79478k, max=79478k, avg=79477681.00, stdev= 0.00 lat (nsec): min=90216k, max=90216k, avg=90215885.00, stdev= 0.00 clat percentiles (usec): | 1.00th=[79168], 5.00th=[79168], 10.00th=[79168], 20.00th=[79168], | 30.00th=[79168], 40.00th=[79168], 50.00th=[79168], 60.00th=[79168], | 70.00th=[79168], 80.00th=[79168], 90.00th=[79168], 95.00th=[79168], | 99.00th=[79168], 99.50th=[79168], 99.90th=[79168], 99.95th=[79168], | 99.99th=[79168] lat (msec) : 100=100.00% cpu : usr=0.00%, sys=9.47%, ctx=42, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=417MiB/s (437MB/s), 417MiB/s-417MiB/s (437MB/s-437MB/s), io=40.0MiB (41.9MB), run=96-96msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout