== sanity test 398c: run fio to test AIO ================= 04:29:32 (1789028972) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.420143 s, 99.8 MB/s osc.lustre-OST0000-osc-ffff97f183063800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=179850: Thu Sep 10 04:30:22 2026 write: IOPS=63, BW=254KiB/s (260kB/s)(10.0MiB/40343msec); 0 zone resets slat (usec): min=30, max=13596, avg=213.53, stdev=694.40 clat (msec): min=31, max=567, avg=250.60, stdev=81.21 lat (msec): min=31, max=567, avg=250.81, stdev=81.18 clat percentiles (msec): | 1.00th=[ 110], 5.00th=[ 140], 10.00th=[ 157], 20.00th=[ 182], | 30.00th=[ 201], 40.00th=[ 215], 50.00th=[ 234], 60.00th=[ 264], | 70.00th=[ 292], 80.00th=[ 326], 90.00th=[ 355], 95.00th=[ 397], | 99.00th=[ 477], 99.50th=[ 481], 99.90th=[ 567], 99.95th=[ 567], | 99.99th=[ 567] bw ( KiB/s): min= 126, max= 502, per=24.88%, avg=252.55, stdev=83.78, samples=80 iops : min= 31, max= 125, avg=62.66, stdev=20.92, samples=80 lat (msec) : 50=0.08%, 100=0.39%, 250=55.94%, 500=43.12%, 750=0.47% cpu : usr=0.11%, sys=0.70%, ctx=1722, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=179851: Thu Sep 10 04:30:22 2026 write: IOPS=63, BW=255KiB/s (261kB/s)(10.0MiB/40159msec); 0 zone resets slat (usec): min=30, max=21131, avg=285.00, stdev=1111.27 clat (msec): min=33, max=566, avg=250.40, stdev=81.50 lat (msec): min=33, max=567, avg=250.69, stdev=81.52 clat percentiles (msec): | 1.00th=[ 100], 5.00th=[ 138], 10.00th=[ 157], 20.00th=[ 180], | 30.00th=[ 201], 40.00th=[ 215], 50.00th=[ 236], 60.00th=[ 266], | 70.00th=[ 292], 80.00th=[ 330], 90.00th=[ 359], 95.00th=[ 393], | 99.00th=[ 456], 99.50th=[ 550], 99.90th=[ 567], 99.95th=[ 567], | 99.99th=[ 567] bw ( KiB/s): min= 126, max= 502, per=24.91%, avg=252.81, stdev=87.41, samples=79 iops : min= 31, max= 125, avg=62.71, stdev=21.88, samples=79 lat (msec) : 50=0.04%, 100=1.02%, 250=53.28%, 500=45.04%, 750=0.62% cpu : usr=0.11%, sys=0.77%, ctx=1689, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=179852: Thu Sep 10 04:30:22 2026 write: IOPS=63, BW=254KiB/s (260kB/s)(10.0MiB/40312msec); 0 zone resets slat (usec): min=30, max=13994, avg=260.54, stdev=937.50 clat (msec): min=26, max=572, avg=250.47, stdev=81.77 lat (msec): min=26, max=572, avg=250.73, stdev=81.81 clat percentiles (msec): | 1.00th=[ 110], 5.00th=[ 140], 10.00th=[ 157], 20.00th=[ 182], | 30.00th=[ 199], 40.00th=[ 213], 50.00th=[ 234], 60.00th=[ 264], | 70.00th=[ 292], 80.00th=[ 321], 90.00th=[ 359], 95.00th=[ 397], | 99.00th=[ 468], 99.50th=[ 550], 99.90th=[ 575], 99.95th=[ 575], | 99.99th=[ 575] bw ( KiB/s): min= 126, max= 502, per=24.89%, avg=252.62, stdev=86.40, samples=80 iops : min= 31, max= 125, avg=62.67, stdev=21.56, samples=80 lat (msec) : 50=0.16%, 100=0.20%, 250=55.59%, 500=43.48%, 750=0.59% cpu : usr=0.15%, sys=0.79%, ctx=1972, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=179853: Thu Sep 10 04:30:22 2026 write: IOPS=63, BW=254KiB/s (260kB/s)(10.0MiB/40298msec); 0 zone resets slat (usec): min=29, max=25670, avg=294.75, stdev=1052.46 clat (msec): min=31, max=560, avg=250.46, stdev=80.28 lat (msec): min=32, max=560, avg=250.75, stdev=80.30 clat percentiles (msec): | 1.00th=[ 110], 5.00th=[ 140], 10.00th=[ 159], 20.00th=[ 184], | 30.00th=[ 199], 40.00th=[ 215], 50.00th=[ 239], 60.00th=[ 266], | 70.00th=[ 292], 80.00th=[ 326], 90.00th=[ 359], 95.00th=[ 388], | 99.00th=[ 481], 99.50th=[ 527], 99.90th=[ 542], 99.95th=[ 550], | 99.99th=[ 558] bw ( KiB/s): min= 126, max= 502, per=24.89%, avg=252.59, stdev=82.65, samples=80 iops : min= 31, max= 125, avg=62.65, stdev=20.63, samples=80 lat (msec) : 50=0.08%, 100=0.70%, 250=54.92%, 500=43.75%, 750=0.55% cpu : usr=0.19%, sys=0.84%, ctx=2515, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1015KiB/s (1040kB/s), 254KiB/s-255KiB/s (260kB/s-261kB/s), io=40.0MiB (41.9MB), run=40159-40343msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=179869: Thu Sep 10 04:31:09 2026 read: IOPS=32, BW=128KiB/s (132kB/s)(5048KiB/39307msec) slat (usec): min=39, max=102891, avg=17632.09, stdev=10591.11 clat (msec): min=21, max=521, avg=233.26, stdev=64.13 lat (msec): min=21, max=534, avg=250.90, stdev=66.92 clat percentiles (msec): | 1.00th=[ 92], 5.00th=[ 142], 10.00th=[ 161], 20.00th=[ 182], | 30.00th=[ 201], 40.00th=[ 213], 50.00th=[ 230], 60.00th=[ 245], | 70.00th=[ 262], 80.00th=[ 284], 90.00th=[ 309], 95.00th=[ 347], | 99.00th=[ 422], 99.50th=[ 460], 99.90th=[ 514], 99.95th=[ 523], | 99.99th=[ 523] bw ( KiB/s): min= 31, max= 232, per=25.16%, avg=127.79, stdev=36.12, samples=77 iops : min= 7, max= 58, avg=31.43, stdev= 9.11, samples=77 write: IOPS=33, BW=132KiB/s (135kB/s)(5192KiB/39307msec); 0 zone resets slat (usec): min=32, max=32454, avg=341.38, stdev=1351.37 clat (msec): min=74, max=508, avg=239.41, stdev=59.19 lat (msec): min=75, max=508, avg=239.75, stdev=59.17 clat percentiles (msec): | 1.00th=[ 116], 5.00th=[ 153], 10.00th=[ 171], 20.00th=[ 190], | 30.00th=[ 207], 40.00th=[ 222], 50.00th=[ 236], 60.00th=[ 249], | 70.00th=[ 266], 80.00th=[ 284], 90.00th=[ 309], 95.00th=[ 347], | 99.00th=[ 414], 99.50th=[ 435], 99.90th=[ 477], 99.95th=[ 510], | 99.99th=[ 510] bw ( KiB/s): min= 63, max= 240, per=24.92%, avg=131.82, stdev=33.19, samples=77 iops : min= 15, max= 60, avg=32.43, stdev= 8.38, samples=77 lat (msec) : 50=0.20%, 100=0.47%, 250=61.45%, 500=37.73%, 750=0.16% cpu : usr=0.27%, sys=1.74%, ctx=4477, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=179870: Thu Sep 10 04:31:09 2026 read: IOPS=31, BW=126KiB/s (129kB/s)(4976KiB/39446msec) slat (usec): min=3120, max=78432, avg=17683.53, stdev=9973.67 clat (msec): min=42, max=473, avg=231.78, stdev=61.27 lat (msec): min=48, max=499, avg=249.48, stdev=63.11 clat percentiles (msec): | 1.00th=[ 102], 5.00th=[ 140], 10.00th=[ 161], 20.00th=[ 184], | 30.00th=[ 201], 40.00th=[ 213], 50.00th=[ 228], 60.00th=[ 243], | 70.00th=[ 259], 80.00th=[ 284], 90.00th=[ 313], 95.00th=[ 338], | 99.00th=[ 393], 99.50th=[ 435], 99.90th=[ 472], 99.95th=[ 472], | 99.99th=[ 472] bw ( KiB/s): min= 21, max= 190, per=24.61%, avg=125.00, stdev=33.11, samples=78 iops : min= 5, max= 47, avg=30.71, stdev= 8.30, samples=78 write: IOPS=33, BW=133KiB/s (137kB/s)(5264KiB/39446msec); 0 zone resets slat (usec): min=35, max=12778, avg=288.81, stdev=857.89 clat (msec): min=44, max=462, avg=240.19, stdev=59.03 lat (msec): min=44, max=463, avg=240.48, stdev=59.04 clat percentiles (msec): | 1.00th=[ 117], 5.00th=[ 150], 10.00th=[ 171], 20.00th=[ 192], | 30.00th=[ 207], 40.00th=[ 222], 50.00th=[ 236], 60.00th=[ 253], | 70.00th=[ 266], 80.00th=[ 288], 90.00th=[ 313], 95.00th=[ 347], | 99.00th=[ 405], 99.50th=[ 418], 99.90th=[ 430], 99.95th=[ 464], | 99.99th=[ 464] bw ( KiB/s): min= 56, max= 280, per=25.08%, avg=132.67, stdev=39.85, samples=78 iops : min= 14, max= 70, avg=32.65, stdev=10.04, samples=78 lat (msec) : 50=0.12%, 100=0.55%, 250=60.62%, 500=38.71% cpu : usr=0.33%, sys=1.88%, ctx=4177, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=179871: Thu Sep 10 04:31:09 2026 read: IOPS=31, BW=127KiB/s (130kB/s)(4984KiB/39386msec) slat (usec): min=3064, max=76896, avg=17663.66, stdev=10032.39 clat (msec): min=47, max=436, avg=230.72, stdev=57.93 lat (msec): min=54, max=463, avg=248.40, stdev=60.46 clat percentiles (msec): | 1.00th=[ 114], 5.00th=[ 142], 10.00th=[ 159], 20.00th=[ 184], | 30.00th=[ 197], 40.00th=[ 211], 50.00th=[ 228], 60.00th=[ 243], | 70.00th=[ 259], 80.00th=[ 279], 90.00th=[ 309], 95.00th=[ 330], | 99.00th=[ 380], 99.50th=[ 409], 99.90th=[ 422], 99.95th=[ 439], | 99.99th=[ 439] bw ( KiB/s): min= 35, max= 296, per=24.83%, avg=126.14, stdev=38.85, samples=78 iops : min= 8, max= 74, avg=31.01, stdev= 9.79, samples=78 write: IOPS=33, BW=133KiB/s (137kB/s)(5256KiB/39386msec); 0 zone resets slat (usec): min=33, max=18981, avg=356.89, stdev=1341.41 clat (msec): min=100, max=452, avg=240.39, stdev=54.65 lat (msec): min=101, max=452, avg=240.75, stdev=54.68 clat percentiles (msec): | 1.00th=[ 127], 5.00th=[ 161], 10.00th=[ 176], 20.00th=[ 194], | 30.00th=[ 209], 40.00th=[ 226], 50.00th=[ 236], 60.00th=[ 251], | 70.00th=[ 264], 80.00th=[ 284], 90.00th=[ 309], 95.00th=[ 338], | 99.00th=[ 393], 99.50th=[ 422], 99.90th=[ 451], 99.95th=[ 451], | 99.99th=[ 451] bw ( KiB/s): min= 42, max= 214, per=25.02%, avg=132.35, stdev=33.05, samples=78 iops : min= 10, max= 53, avg=32.56, stdev= 8.32, samples=78 lat (msec) : 50=0.04%, 100=0.16%, 250=61.84%, 500=37.97% cpu : usr=0.31%, sys=1.93%, ctx=4266, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=179872: Thu Sep 10 04:31:09 2026 read: IOPS=32, BW=128KiB/s (132kB/s)(5060KiB/39396msec) slat (usec): min=3182, max=73741, avg=17961.70, stdev=10460.29 clat (msec): min=34, max=443, avg=230.18, stdev=56.95 lat (msec): min=43, max=463, avg=248.15, stdev=59.65 clat percentiles (msec): | 1.00th=[ 125], 5.00th=[ 142], 10.00th=[ 161], 20.00th=[ 184], | 30.00th=[ 197], 40.00th=[ 213], 50.00th=[ 226], 60.00th=[ 239], | 70.00th=[ 253], 80.00th=[ 275], 90.00th=[ 309], 95.00th=[ 330], | 99.00th=[ 393], 99.50th=[ 409], 99.90th=[ 443], 99.95th=[ 443], | 99.99th=[ 443] bw ( KiB/s): min= 49, max= 256, per=25.12%, avg=127.62, stdev=36.73, samples=78 iops : min= 12, max= 64, avg=31.38, stdev= 9.27, samples=78 write: IOPS=32, BW=131KiB/s (135kB/s)(5180KiB/39396msec); 0 zone resets slat (usec): min=32, max=24105, avg=314.57, stdev=1219.69 clat (msec): min=105, max=454, avg=241.19, stdev=54.37 lat (msec): min=107, max=455, avg=241.51, stdev=54.41 clat percentiles (msec): | 1.00th=[ 136], 5.00th=[ 161], 10.00th=[ 176], 20.00th=[ 197], | 30.00th=[ 209], 40.00th=[ 224], 50.00th=[ 239], 60.00th=[ 253], | 70.00th=[ 268], 80.00th=[ 288], 90.00th=[ 313], 95.00th=[ 334], | 99.00th=[ 388], 99.50th=[ 405], 99.90th=[ 426], 99.95th=[ 456], | 99.99th=[ 456] bw ( KiB/s): min= 49, max= 192, per=24.73%, avg=130.83, stdev=31.23, samples=78 iops : min= 12, max= 48, avg=32.19, stdev= 7.84, samples=78 lat (msec) : 50=0.04%, 100=0.12%, 250=63.12%, 500=36.72% cpu : usr=0.32%, sys=1.93%, ctx=4326, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=509KiB/s (521kB/s), 126KiB/s-128KiB/s (129kB/s-132kB/s), io=19.6MiB (20.5MB), run=39307-39446msec WRITE: bw=530KiB/s (542kB/s), 131KiB/s-133KiB/s (135kB/s-137kB/s), io=20.4MiB (21.4MB), run=39307-39446msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=179883: Thu Sep 10 04:31:19 2026 read: IOPS=2, BW=81.0MiB/s (85.9MB/s)(40.0MiB/488msec) slat (nsec): min=45023k, max=45023k, avg=45022906.00, stdev= 0.00 clat (nsec): min=402138k, max=402138k, avg=402137759.00, stdev= 0.00 lat (nsec): min=447166k, max=447166k, avg=447165918.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 401], 5.00th=[ 401], 10.00th=[ 401], 20.00th=[ 401], | 30.00th=[ 401], 40.00th=[ 401], 50.00th=[ 401], 60.00th=[ 401], | 70.00th=[ 401], 80.00th=[ 401], 90.00th=[ 401], 95.00th=[ 401], | 99.00th=[ 401], 99.50th=[ 401], 99.90th=[ 401], 99.95th=[ 401], | 99.99th=[ 401] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=1.64%, ctx=4, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=81.0MiB/s (85.9MB/s), 81.0MiB/s-81.0MiB/s (85.9MB/s-85.9MB/s), io=40.0MiB (41.9MB), run=488-488msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout