== sanity test 398c: run fio to test AIO ================= 07:04:34 (1787742274) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.303564 s, 138 MB/s osc.lustre-OST0000-osc-ffff925ed15d4800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=180009: Wed Aug 26 07:05:21 2026 write: IOPS=74, BW=299KiB/s (307kB/s)(10.0MiB/34209msec); 0 zone resets slat (usec): min=22, max=15348, avg=254.67, stdev=851.40 clat (msec): min=20, max=417, avg=212.22, stdev=60.49 lat (msec): min=20, max=417, avg=212.47, stdev=60.54 clat percentiles (msec): | 1.00th=[ 105], 5.00th=[ 125], 10.00th=[ 142], 20.00th=[ 157], | 30.00th=[ 176], 40.00th=[ 192], 50.00th=[ 207], 60.00th=[ 222], | 70.00th=[ 239], 80.00th=[ 262], 90.00th=[ 296], 95.00th=[ 326], | 99.00th=[ 368], 99.50th=[ 405], 99.90th=[ 418], 99.95th=[ 418], | 99.99th=[ 418] bw ( KiB/s): min= 95, max= 508, per=25.20%, avg=301.19, stdev=83.57, samples=67 iops : min= 23, max= 127, avg=74.94, stdev=20.95, samples=67 lat (msec) : 50=0.04%, 100=0.59%, 250=74.96%, 500=24.41% cpu : usr=0.20%, sys=1.02%, ctx=2010, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=180010: Wed Aug 26 07:05:21 2026 write: IOPS=75, BW=303KiB/s (310kB/s)(10.0MiB/33807msec); 0 zone resets slat (usec): min=22, max=38957, avg=259.67, stdev=1257.95 clat (msec): min=27, max=635, avg=210.65, stdev=64.70 lat (msec): min=28, max=635, avg=210.91, stdev=64.76 clat percentiles (msec): | 1.00th=[ 97], 5.00th=[ 124], 10.00th=[ 138], 20.00th=[ 153], | 30.00th=[ 176], 40.00th=[ 192], 50.00th=[ 207], 60.00th=[ 222], | 70.00th=[ 236], 80.00th=[ 257], 90.00th=[ 292], 95.00th=[ 309], | 99.00th=[ 368], 99.50th=[ 498], 99.90th=[ 634], 99.95th=[ 634], | 99.99th=[ 634] bw ( KiB/s): min= 167, max= 508, per=25.44%, avg=304.06, stdev=72.61, samples=66 iops : min= 41, max= 127, avg=75.65, stdev=18.15, samples=66 lat (msec) : 50=0.16%, 100=1.21%, 250=75.43%, 500=22.77%, 750=0.43% cpu : usr=0.20%, sys=0.99%, ctx=1935, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=180011: Wed Aug 26 07:05:21 2026 write: IOPS=75, BW=301KiB/s (308kB/s)(10.0MiB/34024msec); 0 zone resets slat (usec): min=23, max=30502, avg=285.26, stdev=1311.00 clat (msec): min=40, max=642, avg=211.68, stdev=64.70 lat (msec): min=40, max=642, avg=211.97, stdev=64.72 clat percentiles (msec): | 1.00th=[ 100], 5.00th=[ 126], 10.00th=[ 136], 20.00th=[ 153], | 30.00th=[ 174], 40.00th=[ 192], 50.00th=[ 209], 60.00th=[ 222], | 70.00th=[ 236], 80.00th=[ 264], 90.00th=[ 292], 95.00th=[ 326], | 99.00th=[ 376], 99.50th=[ 430], 99.90th=[ 642], 99.95th=[ 642], | 99.99th=[ 642] bw ( KiB/s): min= 88, max= 516, per=25.15%, avg=300.51, stdev=81.64, samples=67 iops : min= 22, max= 129, avg=74.75, stdev=20.44, samples=67 lat (msec) : 50=0.12%, 100=0.98%, 250=74.77%, 500=23.87%, 750=0.27% cpu : usr=0.21%, sys=1.08%, ctx=2301, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=180012: Wed Aug 26 07:05:21 2026 write: IOPS=74, BW=299KiB/s (306kB/s)(10.0MiB/34271msec); 0 zone resets slat (usec): min=26, max=19606, avg=280.18, stdev=1162.83 clat (msec): min=20, max=423, avg=212.13, stdev=59.22 lat (msec): min=20, max=423, avg=212.41, stdev=59.28 clat percentiles (msec): | 1.00th=[ 97], 5.00th=[ 131], 10.00th=[ 142], 20.00th=[ 157], | 30.00th=[ 176], 40.00th=[ 192], 50.00th=[ 211], 60.00th=[ 226], | 70.00th=[ 241], 80.00th=[ 259], 90.00th=[ 288], 95.00th=[ 317], | 99.00th=[ 372], 99.50th=[ 376], 99.90th=[ 422], 99.95th=[ 426], | 99.99th=[ 426] bw ( KiB/s): min= 95, max= 508, per=24.96%, avg=298.22, stdev=84.64, samples=67 iops : min= 23, max= 127, avg=74.21, stdev=21.20, samples=67 lat (msec) : 50=0.04%, 100=1.05%, 250=74.34%, 500=24.57% cpu : usr=0.20%, sys=0.94%, ctx=1972, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1195KiB/s (1224kB/s), 299KiB/s-303KiB/s (306kB/s-310kB/s), io=40.0MiB (41.9MB), run=33807-34271msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=180028: Wed Aug 26 07:06:17 2026 read: IOPS=27, BW=112KiB/s (114kB/s)(5048KiB/45194msec) slat (usec): min=64, max=298696, avg=21303.73, stdev=15912.89 clat (msec): min=29, max=622, avg=267.16, stdev=88.66 lat (msec): min=50, max=653, avg=288.47, stdev=92.84 clat percentiles (msec): | 1.00th=[ 107], 5.00th=[ 150], 10.00th=[ 169], 20.00th=[ 197], | 30.00th=[ 215], 40.00th=[ 234], 50.00th=[ 253], 60.00th=[ 271], | 70.00th=[ 300], 80.00th=[ 334], 90.00th=[ 393], 95.00th=[ 443], | 99.00th=[ 523], 99.50th=[ 550], 99.90th=[ 609], 99.95th=[ 625], | 99.99th=[ 625] bw ( KiB/s): min= 30, max= 190, per=24.89%, avg=110.53, stdev=33.30, samples=89 iops : min= 7, max= 47, avg=27.06, stdev= 8.34, samples=89 write: IOPS=28, BW=115KiB/s (118kB/s)(5192KiB/45194msec); 0 zone resets slat (usec): min=24, max=48165, avg=418.21, stdev=2333.29 clat (msec): min=28, max=626, avg=272.67, stdev=84.62 lat (msec): min=28, max=637, avg=273.09, stdev=84.63 clat percentiles (msec): | 1.00th=[ 125], 5.00th=[ 163], 10.00th=[ 184], 20.00th=[ 207], | 30.00th=[ 226], 40.00th=[ 243], 50.00th=[ 257], 60.00th=[ 275], | 70.00th=[ 296], 80.00th=[ 326], 90.00th=[ 388], 95.00th=[ 439], | 99.00th=[ 567], 99.50th=[ 592], 99.90th=[ 609], 99.95th=[ 625], | 99.99th=[ 625] bw ( KiB/s): min= 39, max= 207, per=24.58%, avg=113.57, stdev=39.06, samples=89 iops : min= 9, max= 51, avg=27.84, stdev= 9.78, samples=89 lat (msec) : 50=0.08%, 100=0.35%, 250=46.48%, 500=50.98%, 750=2.11% cpu : usr=0.25%, sys=1.65%, ctx=3642, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=180029: Wed Aug 26 07:06:17 2026 read: IOPS=27, BW=111KiB/s (113kB/s)(4976KiB/45008msec) slat (msec): min=3, max=117, avg=20.19, stdev=13.05 clat (msec): min=13, max=578, avg=265.98, stdev=85.88 lat (msec): min=27, max=588, avg=286.18, stdev=89.06 clat percentiles (msec): | 1.00th=[ 134], 5.00th=[ 159], 10.00th=[ 174], 20.00th=[ 197], | 30.00th=[ 213], 40.00th=[ 230], 50.00th=[ 247], 60.00th=[ 271], | 70.00th=[ 296], 80.00th=[ 334], 90.00th=[ 388], 95.00th=[ 439], | 99.00th=[ 535], 99.50th=[ 550], 99.90th=[ 575], 99.95th=[ 575], | 99.99th=[ 575] bw ( KiB/s): min= 31, max= 199, per=24.80%, avg=110.09, stdev=36.01, samples=88 iops : min= 7, max= 49, avg=26.95, stdev= 9.00, samples=88 write: IOPS=29, BW=117KiB/s (120kB/s)(5264KiB/45008msec); 0 zone resets slat (usec): min=29, max=66458, avg=468.38, stdev=2415.20 clat (msec): min=109, max=581, avg=273.13, stdev=84.32 lat (msec): min=109, max=582, avg=273.60, stdev=84.38 clat percentiles (msec): | 1.00th=[ 138], 5.00th=[ 167], 10.00th=[ 186], 20.00th=[ 205], | 30.00th=[ 222], 40.00th=[ 236], 50.00th=[ 255], 60.00th=[ 275], | 70.00th=[ 300], 80.00th=[ 338], 90.00th=[ 393], 95.00th=[ 439], | 99.00th=[ 531], 99.50th=[ 550], 99.90th=[ 567], 99.95th=[ 584], | 99.99th=[ 584] bw ( KiB/s): min= 31, max= 239, per=25.36%, avg=117.17, stdev=37.79, samples=88 iops : min= 7, max= 59, avg=28.74, stdev= 9.44, samples=88 lat (msec) : 20=0.04%, 100=0.04%, 250=49.18%, 500=48.83%, 750=1.91% cpu : usr=0.26%, sys=1.63%, ctx=3889, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=180030: Wed Aug 26 07:06:17 2026 read: IOPS=27, BW=111KiB/s (113kB/s)(4984KiB/45006msec) slat (msec): min=2, max=192, avg=20.58, stdev=13.28 clat (msec): min=31, max=625, avg=265.49, stdev=86.08 lat (msec): min=47, max=655, avg=286.08, stdev=89.48 clat percentiles (msec): | 1.00th=[ 126], 5.00th=[ 155], 10.00th=[ 174], 20.00th=[ 197], | 30.00th=[ 215], 40.00th=[ 230], 50.00th=[ 249], 60.00th=[ 271], | 70.00th=[ 296], 80.00th=[ 334], 90.00th=[ 380], 95.00th=[ 439], | 99.00th=[ 550], 99.50th=[ 558], 99.90th=[ 609], 99.95th=[ 625], | 99.99th=[ 625] bw ( KiB/s): min= 37, max= 215, per=24.97%, avg=110.85, stdev=34.33, samples=88 iops : min= 9, max= 53, avg=27.16, stdev= 8.56, samples=88 write: IOPS=29, BW=117KiB/s (120kB/s)(5256KiB/45006msec); 0 zone resets slat (usec): min=25, max=24106, avg=336.04, stdev=1229.76 clat (msec): min=64, max=609, avg=273.60, stdev=85.97 lat (msec): min=64, max=609, avg=273.94, stdev=85.95 clat percentiles (msec): | 1.00th=[ 134], 5.00th=[ 169], 10.00th=[ 184], 20.00th=[ 205], | 30.00th=[ 224], 40.00th=[ 241], 50.00th=[ 257], 60.00th=[ 271], | 70.00th=[ 300], 80.00th=[ 334], 90.00th=[ 397], 95.00th=[ 447], | 99.00th=[ 558], 99.50th=[ 575], 99.90th=[ 609], 99.95th=[ 609], | 99.99th=[ 609] bw ( KiB/s): min= 32, max= 198, per=25.21%, avg=116.47, stdev=36.15, samples=88 iops : min= 8, max= 49, avg=28.58, stdev= 9.04, samples=88 lat (msec) : 50=0.08%, 100=0.04%, 250=48.83%, 500=49.02%, 750=2.03% cpu : usr=0.28%, sys=1.60%, ctx=3908, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=180031: Wed Aug 26 07:06:17 2026 read: IOPS=28, BW=113KiB/s (115kB/s)(5060KiB/44951msec) slat (usec): min=46, max=308378, avg=20452.27, stdev=14999.11 clat (msec): min=24, max=624, avg=267.42, stdev=87.63 lat (msec): min=24, max=647, avg=287.88, stdev=91.41 clat percentiles (msec): | 1.00th=[ 117], 5.00th=[ 157], 10.00th=[ 174], 20.00th=[ 197], | 30.00th=[ 220], 40.00th=[ 234], 50.00th=[ 251], 60.00th=[ 268], | 70.00th=[ 292], 80.00th=[ 330], 90.00th=[ 388], 95.00th=[ 447], | 99.00th=[ 542], 99.50th=[ 558], 99.90th=[ 584], 99.95th=[ 625], | 99.99th=[ 625] bw ( KiB/s): min= 47, max= 207, per=25.31%, avg=112.36, stdev=31.99, samples=88 iops : min= 11, max= 51, avg=27.55, stdev= 8.00, samples=88 write: IOPS=28, BW=115KiB/s (118kB/s)(5180KiB/44951msec); 0 zone resets slat (usec): min=26, max=30873, avg=345.88, stdev=1363.25 clat (msec): min=86, max=635, avg=273.32, stdev=84.87 lat (msec): min=86, max=635, avg=273.67, stdev=84.93 clat percentiles (msec): | 1.00th=[ 133], 5.00th=[ 165], 10.00th=[ 186], 20.00th=[ 207], | 30.00th=[ 226], 40.00th=[ 241], 50.00th=[ 255], 60.00th=[ 275], | 70.00th=[ 300], 80.00th=[ 334], 90.00th=[ 397], 95.00th=[ 443], | 99.00th=[ 531], 99.50th=[ 550], 99.90th=[ 634], 99.95th=[ 634], | 99.99th=[ 634] bw ( KiB/s): min= 39, max= 184, per=24.82%, avg=114.66, stdev=36.39, samples=88 iops : min= 9, max= 46, avg=28.11, stdev= 9.10, samples=88 lat (msec) : 50=0.12%, 100=0.16%, 250=47.81%, 500=49.92%, 750=1.99% cpu : usr=0.29%, sys=1.53%, ctx=3721, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=444KiB/s (455kB/s), 111KiB/s-113KiB/s (113kB/s-115kB/s), io=19.6MiB (20.5MB), run=44951-45194msec WRITE: bw=462KiB/s (473kB/s), 115KiB/s-117KiB/s (118kB/s-120kB/s), io=20.4MiB (21.4MB), run=44951-45194msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=180041: Wed Aug 26 07:06:27 2026 read: IOPS=1, BW=72.7MiB/s (76.3MB/s)(40.0MiB/550msec) slat (nsec): min=35381k, max=35381k, avg=35381082.00, stdev= 0.00 clat (nsec): min=489305k, max=489305k, avg=489304701.00, stdev= 0.00 lat (nsec): min=524690k, max=524690k, avg=524689803.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 489], 5.00th=[ 489], 10.00th=[ 489], 20.00th=[ 489], | 30.00th=[ 489], 40.00th=[ 489], 50.00th=[ 489], 60.00th=[ 489], | 70.00th=[ 489], 80.00th=[ 489], 90.00th=[ 489], 95.00th=[ 489], | 99.00th=[ 489], 99.50th=[ 489], 99.90th=[ 489], 99.95th=[ 489], | 99.99th=[ 489] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=2.55%, ctx=11, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=72.7MiB/s (76.3MB/s), 72.7MiB/s-72.7MiB/s (76.3MB/s-76.3MB/s), io=40.0MiB (41.9MB), run=550-550msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout