== sanity test 398c: run fio to test AIO ================= 14:45:12 (1767728712) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.18426 s, 228 MB/s osc.lustre-OST0000-osc-ffff8f18882a6800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=150570: Tue Jan 6 14:46:04 2026 write: IOPS=53, BW=214KiB/s (219kB/s)(10.0MiB/47874msec); 0 zone resets slat (usec): min=20, max=4050, avg=142.97, stdev=195.73 clat (msec): min=107, max=556, avg=298.91, stdev=78.30 lat (msec): min=107, max=556, avg=299.06, stdev=78.31 clat percentiles (msec): | 1.00th=[ 115], 5.00th=[ 192], 10.00th=[ 209], 20.00th=[ 234], | 30.00th=[ 251], 40.00th=[ 268], 50.00th=[ 288], 60.00th=[ 309], | 70.00th=[ 334], 80.00th=[ 363], 90.00th=[ 409], 95.00th=[ 447], | 99.00th=[ 502], 99.50th=[ 523], 99.90th=[ 542], 99.95th=[ 558], | 99.99th=[ 558] bw ( KiB/s): min= 127, max= 336, per=24.98%, avg=213.32, stdev=54.74, samples=95 iops : min= 31, max= 84, avg=53.15, stdev=13.76, samples=95 lat (msec) : 250=28.91%, 500=69.96%, 750=1.13% cpu : usr=0.10%, sys=0.59%, ctx=2930, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=150571: Tue Jan 6 14:46:04 2026 write: IOPS=53, BW=214KiB/s (219kB/s)(10.0MiB/47926msec); 0 zone resets slat (usec): min=23, max=4512, avg=144.30, stdev=215.87 clat (msec): min=86, max=617, avg=299.14, stdev=78.65 lat (msec): min=87, max=617, avg=299.28, stdev=78.65 clat percentiles (msec): | 1.00th=[ 163], 5.00th=[ 192], 10.00th=[ 207], 20.00th=[ 232], | 30.00th=[ 251], 40.00th=[ 271], 50.00th=[ 292], 60.00th=[ 313], | 70.00th=[ 334], 80.00th=[ 363], 90.00th=[ 409], 95.00th=[ 443], | 99.00th=[ 506], 99.50th=[ 550], 99.90th=[ 609], 99.95th=[ 617], | 99.99th=[ 617] bw ( KiB/s): min= 103, max= 328, per=24.94%, avg=212.97, stdev=53.93, samples=95 iops : min= 25, max= 82, avg=53.06, stdev=13.55, samples=95 lat (msec) : 100=0.62%, 250=28.75%, 500=69.53%, 750=1.09% cpu : usr=0.09%, sys=0.57%, ctx=2978, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=150572: Tue Jan 6 14:46:04 2026 write: IOPS=53, BW=214KiB/s (219kB/s)(10.0MiB/47926msec); 0 zone resets slat (usec): min=23, max=3097, avg=132.44, stdev=158.09 clat (msec): min=90, max=605, avg=299.05, stdev=77.58 lat (msec): min=90, max=605, avg=299.18, stdev=77.59 clat percentiles (msec): | 1.00th=[ 161], 5.00th=[ 192], 10.00th=[ 211], 20.00th=[ 234], | 30.00th=[ 249], 40.00th=[ 266], 50.00th=[ 288], 60.00th=[ 313], | 70.00th=[ 334], 80.00th=[ 368], 90.00th=[ 405], 95.00th=[ 435], | 99.00th=[ 498], 99.50th=[ 514], 99.90th=[ 600], 99.95th=[ 609], | 99.99th=[ 609] bw ( KiB/s): min= 63, max= 383, per=24.95%, avg=213.04, stdev=59.07, samples=95 iops : min= 15, max= 95, avg=53.07, stdev=14.83, samples=95 lat (msec) : 100=0.66%, 250=29.73%, 500=68.75%, 750=0.86% cpu : usr=0.09%, sys=0.54%, ctx=2875, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=150573: Tue Jan 6 14:46:04 2026 write: IOPS=53, BW=214KiB/s (219kB/s)(10.0MiB/47935msec); 0 zone resets slat (usec): min=23, max=4476, avg=143.27, stdev=197.20 clat (msec): min=49, max=545, avg=299.08, stdev=77.85 lat (msec): min=49, max=545, avg=299.22, stdev=77.86 clat percentiles (msec): | 1.00th=[ 150], 5.00th=[ 194], 10.00th=[ 211], 20.00th=[ 234], | 30.00th=[ 253], 40.00th=[ 268], 50.00th=[ 288], 60.00th=[ 313], | 70.00th=[ 334], 80.00th=[ 363], 90.00th=[ 414], 95.00th=[ 447], | 99.00th=[ 489], 99.50th=[ 506], 99.90th=[ 523], 99.95th=[ 542], | 99.99th=[ 550] bw ( KiB/s): min= 127, max= 336, per=24.87%, avg=212.37, stdev=52.26, samples=95 iops : min= 31, max= 84, avg=52.91, stdev=13.13, samples=95 lat (msec) : 50=0.04%, 100=0.08%, 250=29.30%, 500=70.04%, 750=0.55% cpu : usr=0.11%, sys=0.58%, ctx=3129, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=854KiB/s (875kB/s), 214KiB/s-214KiB/s (219kB/s-219kB/s), io=40.0MiB (41.9MB), run=47874-47935msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=150591: Tue Jan 6 14:46:35 2026 read: IOPS=46, BW=186KiB/s (191kB/s)(5048KiB/27110msec) slat (usec): min=1281, max=10705, avg=3504.06, stdev=1420.63 clat (msec): min=5, max=257, avg=99.24, stdev=22.93 lat (msec): min=8, max=259, avg=102.75, stdev=23.09 clat percentiles (msec): | 1.00th=[ 43], 5.00th=[ 68], 10.00th=[ 74], 20.00th=[ 83], | 30.00th=[ 88], 40.00th=[ 93], 50.00th=[ 97], 60.00th=[ 103], | 70.00th=[ 109], 80.00th=[ 116], 90.00th=[ 128], 95.00th=[ 138], | 99.00th=[ 159], 99.50th=[ 169], 99.90th=[ 239], 99.95th=[ 257], | 99.99th=[ 257] bw ( KiB/s): min= 96, max= 272, per=25.18%, avg=186.34, stdev=42.62, samples=53 iops : min= 24, max= 68, avg=46.42, stdev=10.64, samples=53 write: IOPS=47, BW=192KiB/s (196kB/s)(5192KiB/27110msec); 0 zone resets slat (usec): min=35, max=4837, avg=137.62, stdev=209.89 clat (msec): min=62, max=396, avg=233.37, stdev=56.68 lat (msec): min=62, max=397, avg=233.51, stdev=56.69 clat percentiles (msec): | 1.00th=[ 104], 5.00th=[ 140], 10.00th=[ 161], 20.00th=[ 186], | 30.00th=[ 203], 40.00th=[ 218], 50.00th=[ 232], 60.00th=[ 249], | 70.00th=[ 264], 80.00th=[ 284], 90.00th=[ 305], 95.00th=[ 330], | 99.00th=[ 380], 99.50th=[ 388], 99.90th=[ 393], 99.95th=[ 397], | 99.99th=[ 397] bw ( KiB/s): min= 120, max= 264, per=24.80%, avg=190.94, stdev=32.13, samples=53 iops : min= 30, max= 66, avg=47.58, stdev= 8.04, samples=53 lat (msec) : 10=0.04%, 20=0.16%, 50=0.51%, 100=26.88%, 250=52.81% lat (msec) : 500=19.61% cpu : usr=0.17%, sys=1.51%, ctx=4904, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=150592: Tue Jan 6 14:46:35 2026 read: IOPS=45, BW=184KiB/s (188kB/s)(4976KiB/27110msec) slat (usec): min=36, max=11288, avg=3554.25, stdev=1460.39 clat (msec): min=3, max=182, avg=100.58, stdev=21.78 lat (msec): min=6, max=185, avg=104.14, stdev=22.06 clat percentiles (msec): | 1.00th=[ 54], 5.00th=[ 67], 10.00th=[ 77], 20.00th=[ 84], | 30.00th=[ 90], 40.00th=[ 94], 50.00th=[ 100], 60.00th=[ 105], | 70.00th=[ 110], 80.00th=[ 120], 90.00th=[ 131], 95.00th=[ 138], | 99.00th=[ 153], 99.50th=[ 159], 99.90th=[ 174], 99.95th=[ 184], | 99.99th=[ 184] bw ( KiB/s): min= 96, max= 296, per=24.68%, avg=182.64, stdev=45.93, samples=53 iops : min= 24, max= 74, avg=45.51, stdev=11.51, samples=53 write: IOPS=48, BW=194KiB/s (199kB/s)(5264KiB/27110msec); 0 zone resets slat (usec): min=21, max=1894, avg=128.65, stdev=107.56 clat (msec): min=36, max=403, avg=230.80, stdev=58.11 lat (msec): min=36, max=403, avg=230.93, stdev=58.11 clat percentiles (msec): | 1.00th=[ 80], 5.00th=[ 134], 10.00th=[ 159], 20.00th=[ 186], | 30.00th=[ 203], 40.00th=[ 218], 50.00th=[ 232], 60.00th=[ 245], | 70.00th=[ 259], 80.00th=[ 279], 90.00th=[ 300], 95.00th=[ 326], | 99.00th=[ 384], 99.50th=[ 393], 99.90th=[ 405], 99.95th=[ 405], | 99.99th=[ 405] bw ( KiB/s): min= 136, max= 288, per=25.03%, avg=192.70, stdev=36.07, samples=53 iops : min= 34, max= 72, avg=48.02, stdev= 8.96, samples=53 lat (msec) : 4=0.04%, 10=0.04%, 20=0.12%, 50=0.35%, 100=25.35% lat (msec) : 250=55.59%, 500=18.52% cpu : usr=0.21%, sys=1.42%, ctx=4847, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=150593: Tue Jan 6 14:46:35 2026 read: IOPS=45, BW=184KiB/s (188kB/s)(4984KiB/27110msec) slat (usec): min=1281, max=11279, avg=3587.02, stdev=1411.52 clat (msec): min=4, max=237, avg=99.57, stdev=23.61 lat (msec): min=8, max=241, avg=103.16, stdev=23.80 clat percentiles (msec): | 1.00th=[ 24], 5.00th=[ 67], 10.00th=[ 74], 20.00th=[ 83], | 30.00th=[ 89], 40.00th=[ 94], 50.00th=[ 99], 60.00th=[ 104], | 70.00th=[ 109], 80.00th=[ 117], 90.00th=[ 130], 95.00th=[ 138], | 99.00th=[ 155], 99.50th=[ 169], 99.90th=[ 236], 99.95th=[ 239], | 99.99th=[ 239] bw ( KiB/s): min= 79, max= 312, per=24.91%, avg=184.32, stdev=45.43, samples=53 iops : min= 19, max= 78, avg=45.92, stdev=11.32, samples=53 write: IOPS=48, BW=194KiB/s (199kB/s)(5256KiB/27110msec); 0 zone resets slat (usec): min=32, max=1929, avg=135.54, stdev=132.08 clat (msec): min=47, max=406, avg=231.56, stdev=56.73 lat (msec): min=47, max=406, avg=231.70, stdev=56.73 clat percentiles (msec): | 1.00th=[ 106], 5.00th=[ 140], 10.00th=[ 159], 20.00th=[ 186], | 30.00th=[ 201], 40.00th=[ 215], 50.00th=[ 232], 60.00th=[ 247], | 70.00th=[ 262], 80.00th=[ 279], 90.00th=[ 305], 95.00th=[ 326], | 99.00th=[ 384], 99.50th=[ 388], 99.90th=[ 405], 99.95th=[ 405], | 99.99th=[ 405] bw ( KiB/s): min= 128, max= 280, per=25.00%, avg=192.49, stdev=35.68, samples=53 iops : min= 32, max= 70, avg=47.96, stdev= 8.90, samples=53 lat (msec) : 10=0.16%, 20=0.27%, 50=0.47%, 100=25.55%, 250=54.41% lat (msec) : 500=19.14% cpu : usr=0.23%, sys=1.44%, ctx=4941, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=150594: Tue Jan 6 14:46:35 2026 read: IOPS=46, BW=187KiB/s (191kB/s)(5060KiB/27110msec) slat (usec): min=1261, max=9732, avg=3500.15, stdev=1420.42 clat (msec): min=4, max=184, avg=99.37, stdev=22.01 lat (msec): min=8, max=188, avg=102.87, stdev=22.26 clat percentiles (msec): | 1.00th=[ 46], 5.00th=[ 68], 10.00th=[ 74], 20.00th=[ 82], | 30.00th=[ 89], 40.00th=[ 94], 50.00th=[ 100], 60.00th=[ 104], | 70.00th=[ 109], 80.00th=[ 116], 90.00th=[ 130], 95.00th=[ 138], | 99.00th=[ 153], 99.50th=[ 159], 99.90th=[ 182], 99.95th=[ 186], | 99.99th=[ 186] bw ( KiB/s): min= 64, max= 296, per=25.24%, avg=186.77, stdev=43.83, samples=53 iops : min= 16, max= 74, avg=46.53, stdev=10.96, samples=53 write: IOPS=47, BW=191KiB/s (196kB/s)(5180KiB/27110msec); 0 zone resets slat (usec): min=28, max=2870, avg=131.02, stdev=130.65 clat (msec): min=48, max=396, avg=233.63, stdev=55.53 lat (msec): min=48, max=396, avg=233.76, stdev=55.53 clat percentiles (msec): | 1.00th=[ 102], 5.00th=[ 144], 10.00th=[ 165], 20.00th=[ 188], | 30.00th=[ 203], 40.00th=[ 222], 50.00th=[ 234], 60.00th=[ 247], | 70.00th=[ 259], 80.00th=[ 275], 90.00th=[ 305], 95.00th=[ 330], | 99.00th=[ 372], 99.50th=[ 380], 99.90th=[ 393], 99.95th=[ 397], | 99.99th=[ 397] bw ( KiB/s): min= 111, max= 264, per=24.64%, avg=189.74, stdev=34.00, samples=53 iops : min= 27, max= 66, avg=47.28, stdev= 8.51, samples=53 lat (msec) : 10=0.08%, 20=0.08%, 50=0.47%, 100=25.90%, 250=54.73% lat (msec) : 500=18.75% cpu : usr=0.18%, sys=1.45%, ctx=5010, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=740KiB/s (758kB/s), 184KiB/s-187KiB/s (188kB/s-191kB/s), io=19.6MiB (20.5MB), run=27110-27110msec WRITE: bw=771KiB/s (789kB/s), 191KiB/s-194KiB/s (196kB/s-199kB/s), io=20.4MiB (21.4MB), run=27110-27110msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=150602: Tue Jan 6 14:46:38 2026 read: IOPS=6, BW=274MiB/s (287MB/s)(40.0MiB/146msec) slat (nsec): min=12227k, max=12227k, avg=12226920.00, stdev= 0.00 clat (nsec): min=122956k, max=122956k, avg=122955764.00, stdev= 0.00 lat (nsec): min=135186k, max=135186k, avg=135186488.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 124], 5.00th=[ 124], 10.00th=[ 124], 20.00th=[ 124], | 30.00th=[ 124], 40.00th=[ 124], 50.00th=[ 124], 60.00th=[ 124], | 70.00th=[ 124], 80.00th=[ 124], 90.00th=[ 124], 95.00th=[ 124], | 99.00th=[ 124], 99.50th=[ 124], 99.90th=[ 124], 99.95th=[ 124], | 99.99th=[ 124] lat (msec) : 250=100.00% cpu : usr=0.00%, sys=7.59%, ctx=43, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=274MiB/s (287MB/s), 274MiB/s-274MiB/s (287MB/s-287MB/s), io=40.0MiB (41.9MB), run=146-146msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout