== sanity test 398c: run fio to test AIO ================= 05:10:55 (1787130655) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.571295 s, 73.4 MB/s osc.lustre-OST0000-osc-ffff8da06e7a1000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=180506: Wed Aug 19 05:11:35 2026 write: IOPS=85, BW=341KiB/s (349kB/s)(10.0MiB/30052msec); 0 zone resets slat (usec): min=52, max=22863, avg=372.45, stdev=1083.30 clat (msec): min=18, max=325, avg=186.31, stdev=39.36 lat (msec): min=18, max=325, avg=186.68, stdev=39.33 clat percentiles (msec): | 1.00th=[ 110], 5.00th=[ 130], 10.00th=[ 140], 20.00th=[ 155], | 30.00th=[ 165], 40.00th=[ 174], 50.00th=[ 184], 60.00th=[ 192], | 70.00th=[ 203], 80.00th=[ 218], 90.00th=[ 243], 95.00th=[ 257], | 99.00th=[ 292], 99.50th=[ 313], 99.90th=[ 326], 99.95th=[ 326], | 99.99th=[ 326] bw ( KiB/s): min= 222, max= 471, per=24.96%, avg=339.97, stdev=56.45, samples=59 iops : min= 55, max= 117, avg=84.61, stdev=14.16, samples=59 lat (msec) : 20=0.04%, 50=0.08%, 100=0.31%, 250=92.70%, 500=6.88% cpu : usr=0.17%, sys=1.35%, ctx=2905, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=180507: Wed Aug 19 05:11:35 2026 write: IOPS=85, BW=341KiB/s (349kB/s)(10.0MiB/30054msec); 0 zone resets slat (usec): min=48, max=17070, avg=355.91, stdev=995.59 clat (msec): min=51, max=325, avg=186.51, stdev=38.88 lat (msec): min=52, max=325, avg=186.87, stdev=38.90 clat percentiles (msec): | 1.00th=[ 102], 5.00th=[ 131], 10.00th=[ 144], 20.00th=[ 155], | 30.00th=[ 165], 40.00th=[ 174], 50.00th=[ 182], 60.00th=[ 190], | 70.00th=[ 203], 80.00th=[ 218], 90.00th=[ 241], 95.00th=[ 259], | 99.00th=[ 288], 99.50th=[ 292], 99.90th=[ 317], 99.95th=[ 317], | 99.99th=[ 326] bw ( KiB/s): min= 207, max= 459, per=24.89%, avg=339.07, stdev=55.77, samples=59 iops : min= 51, max= 114, avg=84.36, stdev=14.01, samples=59 lat (msec) : 100=0.98%, 250=91.80%, 500=7.23% cpu : usr=0.19%, sys=1.34%, ctx=2693, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=180508: Wed Aug 19 05:11:35 2026 write: IOPS=85, BW=343KiB/s (351kB/s)(10.0MiB/29865msec); 0 zone resets slat (usec): min=48, max=19879, avg=402.94, stdev=1119.27 clat (msec): min=16, max=404, avg=185.98, stdev=40.63 lat (msec): min=23, max=404, avg=186.39, stdev=40.60 clat percentiles (msec): | 1.00th=[ 93], 5.00th=[ 128], 10.00th=[ 138], 20.00th=[ 153], | 30.00th=[ 165], 40.00th=[ 176], 50.00th=[ 182], 60.00th=[ 192], | 70.00th=[ 203], 80.00th=[ 215], 90.00th=[ 241], 95.00th=[ 262], | 99.00th=[ 292], 99.50th=[ 296], 99.90th=[ 405], 99.95th=[ 405], | 99.99th=[ 405] bw ( KiB/s): min= 223, max= 514, per=25.13%, avg=342.31, stdev=60.98, samples=59 iops : min= 55, max= 128, avg=85.17, stdev=15.30, samples=59 lat (msec) : 20=0.04%, 50=0.12%, 100=0.86%, 250=91.21%, 500=7.77% cpu : usr=0.18%, sys=1.29%, ctx=2826, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=180509: Wed Aug 19 05:11:35 2026 write: IOPS=85, BW=341KiB/s (349kB/s)(10.0MiB/30018msec); 0 zone resets slat (usec): min=53, max=13928, avg=349.88, stdev=876.50 clat (msec): min=25, max=325, avg=186.11, stdev=38.44 lat (msec): min=25, max=326, avg=186.46, stdev=38.48 clat percentiles (msec): | 1.00th=[ 114], 5.00th=[ 131], 10.00th=[ 142], 20.00th=[ 155], | 30.00th=[ 165], 40.00th=[ 176], 50.00th=[ 184], 60.00th=[ 190], | 70.00th=[ 203], 80.00th=[ 215], 90.00th=[ 241], 95.00th=[ 257], | 99.00th=[ 284], 99.50th=[ 292], 99.90th=[ 317], 99.95th=[ 317], | 99.99th=[ 326] bw ( KiB/s): min= 252, max= 484, per=24.94%, avg=339.73, stdev=57.61, samples=59 iops : min= 63, max= 121, avg=84.54, stdev=14.47, samples=59 lat (msec) : 50=0.12%, 100=0.62%, 250=92.73%, 500=6.52% cpu : usr=0.20%, sys=1.31%, ctx=2813, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1363KiB/s (1396kB/s), 341KiB/s-343KiB/s (349kB/s-351kB/s), io=40.0MiB (41.9MB), run=29865-30054msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=180524: Wed Aug 19 05:12:19 2026 read: IOPS=36, BW=148KiB/s (151kB/s)(5048KiB/34136msec) slat (usec): min=58, max=85058, avg=14879.49, stdev=7802.23 clat (msec): min=38, max=372, avg=202.44, stdev=45.01 lat (msec): min=38, max=380, avg=217.33, stdev=46.76 clat percentiles (msec): | 1.00th=[ 89], 5.00th=[ 138], 10.00th=[ 150], 20.00th=[ 167], | 30.00th=[ 178], 40.00th=[ 188], 50.00th=[ 199], 60.00th=[ 209], | 70.00th=[ 222], 80.00th=[ 239], 90.00th=[ 262], 95.00th=[ 284], | 99.00th=[ 321], 99.50th=[ 334], 99.90th=[ 347], 99.95th=[ 372], | 99.99th=[ 372] bw ( KiB/s): min= 78, max= 216, per=25.30%, avg=147.24, stdev=31.01, samples=67 iops : min= 19, max= 54, avg=36.37, stdev= 7.86, samples=67 write: IOPS=38, BW=152KiB/s (156kB/s)(5192KiB/34136msec); 0 zone resets slat (usec): min=54, max=10105, avg=346.27, stdev=824.29 clat (msec): min=71, max=367, avg=208.59, stdev=43.71 lat (msec): min=72, max=367, avg=208.94, stdev=43.71 clat percentiles (msec): | 1.00th=[ 109], 5.00th=[ 148], 10.00th=[ 159], 20.00th=[ 171], | 30.00th=[ 184], 40.00th=[ 194], 50.00th=[ 205], 60.00th=[ 215], | 70.00th=[ 230], 80.00th=[ 245], 90.00th=[ 266], 95.00th=[ 288], | 99.00th=[ 326], 99.50th=[ 342], 99.90th=[ 359], 99.95th=[ 368], | 99.99th=[ 368] bw ( KiB/s): min= 87, max= 224, per=25.05%, avg=151.82, stdev=33.15, samples=67 iops : min= 21, max= 56, avg=37.54, stdev= 8.35, samples=67 lat (msec) : 50=0.16%, 100=0.59%, 250=83.71%, 500=15.55% cpu : usr=0.22%, sys=1.80%, ctx=4719, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=180525: Wed Aug 19 05:12:19 2026 read: IOPS=36, BW=145KiB/s (148kB/s)(4976KiB/34413msec) slat (usec): min=3922, max=65097, avg=15315.13, stdev=7818.91 clat (msec): min=93, max=356, avg=203.77, stdev=43.47 lat (msec): min=106, max=368, avg=219.10, stdev=45.00 clat percentiles (msec): | 1.00th=[ 122], 5.00th=[ 142], 10.00th=[ 155], 20.00th=[ 167], | 30.00th=[ 178], 40.00th=[ 188], 50.00th=[ 199], 60.00th=[ 211], | 70.00th=[ 224], 80.00th=[ 241], 90.00th=[ 266], 95.00th=[ 284], | 99.00th=[ 321], 99.50th=[ 330], 99.90th=[ 347], 99.95th=[ 355], | 99.99th=[ 355] bw ( KiB/s): min= 79, max= 224, per=24.72%, avg=143.88, stdev=31.44, samples=68 iops : min= 19, max= 56, avg=35.54, stdev= 7.96, samples=68 write: IOPS=38, BW=153KiB/s (157kB/s)(5264KiB/34413msec); 0 zone resets slat (usec): min=55, max=10833, avg=321.11, stdev=753.75 clat (msec): min=34, max=357, avg=209.19, stdev=41.34 lat (msec): min=35, max=358, avg=209.51, stdev=41.38 clat percentiles (msec): | 1.00th=[ 129], 5.00th=[ 148], 10.00th=[ 159], 20.00th=[ 174], | 30.00th=[ 188], 40.00th=[ 199], 50.00th=[ 207], 60.00th=[ 218], | 70.00th=[ 230], 80.00th=[ 243], 90.00th=[ 264], 95.00th=[ 284], | 99.00th=[ 321], 99.50th=[ 334], 99.90th=[ 351], 99.95th=[ 359], | 99.99th=[ 359] bw ( KiB/s): min= 55, max= 216, per=25.21%, avg=152.79, stdev=30.87, samples=68 iops : min= 13, max= 54, avg=37.76, stdev= 7.79, samples=68 lat (msec) : 50=0.04%, 100=0.08%, 250=84.61%, 500=15.27% cpu : usr=0.24%, sys=1.90%, ctx=4438, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=180526: Wed Aug 19 05:12:19 2026 read: IOPS=36, BW=145KiB/s (148kB/s)(4984KiB/34434msec) slat (usec): min=3841, max=63865, avg=15152.50, stdev=7864.75 clat (msec): min=25, max=366, avg=203.34, stdev=44.65 lat (msec): min=37, max=391, avg=218.50, stdev=46.08 clat percentiles (msec): | 1.00th=[ 120], 5.00th=[ 142], 10.00th=[ 155], 20.00th=[ 169], | 30.00th=[ 178], 40.00th=[ 188], 50.00th=[ 199], 60.00th=[ 209], | 70.00th=[ 222], 80.00th=[ 239], 90.00th=[ 264], 95.00th=[ 288], | 99.00th=[ 334], 99.50th=[ 351], 99.90th=[ 368], 99.95th=[ 368], | 99.99th=[ 368] bw ( KiB/s): min= 67, max= 231, per=24.70%, avg=143.75, stdev=36.94, samples=68 iops : min= 16, max= 57, avg=35.50, stdev= 9.28, samples=68 write: IOPS=38, BW=153KiB/s (156kB/s)(5256KiB/34434msec); 0 zone resets slat (usec): min=54, max=19749, avg=355.84, stdev=961.31 clat (msec): min=34, max=373, avg=209.08, stdev=44.55 lat (msec): min=34, max=373, avg=209.44, stdev=44.51 clat percentiles (msec): | 1.00th=[ 120], 5.00th=[ 148], 10.00th=[ 161], 20.00th=[ 176], | 30.00th=[ 184], 40.00th=[ 194], 50.00th=[ 203], 60.00th=[ 215], | 70.00th=[ 232], 80.00th=[ 247], 90.00th=[ 268], 95.00th=[ 284], | 99.00th=[ 334], 99.50th=[ 347], 99.90th=[ 372], 99.95th=[ 372], | 99.99th=[ 372] bw ( KiB/s): min= 83, max= 216, per=24.91%, avg=150.94, stdev=35.14, samples=68 iops : min= 20, max= 54, avg=37.31, stdev= 8.91, samples=68 lat (msec) : 50=0.20%, 100=0.47%, 250=83.05%, 500=16.29% cpu : usr=0.24%, sys=1.94%, ctx=4453, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=180527: Wed Aug 19 05:12:19 2026 read: IOPS=36, BW=147KiB/s (150kB/s)(5060KiB/34433msec) slat (usec): min=3356, max=54221, avg=14992.87, stdev=7525.02 clat (msec): min=105, max=349, avg=203.27, stdev=41.52 lat (msec): min=117, max=375, avg=218.27, stdev=43.06 clat percentiles (msec): | 1.00th=[ 124], 5.00th=[ 142], 10.00th=[ 155], 20.00th=[ 167], | 30.00th=[ 178], 40.00th=[ 190], 50.00th=[ 199], 60.00th=[ 211], | 70.00th=[ 224], 80.00th=[ 239], 90.00th=[ 259], 95.00th=[ 275], | 99.00th=[ 321], 99.50th=[ 334], 99.90th=[ 347], 99.95th=[ 351], | 99.99th=[ 351] bw ( KiB/s): min= 79, max= 215, per=25.14%, avg=146.34, stdev=29.74, samples=68 iops : min= 19, max= 53, avg=36.15, stdev= 7.46, samples=68 write: IOPS=37, BW=150KiB/s (154kB/s)(5180KiB/34433msec); 0 zone resets slat (usec): min=56, max=12220, avg=348.43, stdev=843.02 clat (msec): min=111, max=359, avg=209.55, stdev=40.91 lat (msec): min=112, max=359, avg=209.90, stdev=40.91 clat percentiles (msec): | 1.00th=[ 136], 5.00th=[ 146], 10.00th=[ 161], 20.00th=[ 176], | 30.00th=[ 186], 40.00th=[ 194], 50.00th=[ 205], 60.00th=[ 218], | 70.00th=[ 230], 80.00th=[ 245], 90.00th=[ 264], 95.00th=[ 284], | 99.00th=[ 321], 99.50th=[ 330], 99.90th=[ 355], 99.95th=[ 359], | 99.99th=[ 359] bw ( KiB/s): min= 78, max= 239, per=24.67%, avg=149.47, stdev=33.15, samples=68 iops : min= 19, max= 59, avg=36.94, stdev= 8.38, samples=68 lat (msec) : 250=85.20%, 500=14.80% cpu : usr=0.25%, sys=1.86%, ctx=4587, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=583KiB/s (597kB/s), 145KiB/s-148KiB/s (148kB/s-151kB/s), io=19.6MiB (20.5MB), run=34136-34434msec WRITE: bw=607KiB/s (621kB/s), 150KiB/s-153KiB/s (154kB/s-157kB/s), io=20.4MiB (21.4MB), run=34136-34434msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=180536: Wed Aug 19 05:12:29 2026 read: IOPS=1, BW=66.9MiB/s (70.1MB/s)(40.0MiB/598msec) slat (nsec): min=29086k, max=29086k, avg=29086130.00, stdev= 0.00 clat (nsec): min=549152k, max=549152k, avg=549152273.00, stdev= 0.00 lat (nsec): min=578244k, max=578244k, avg=578243866.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 550], 5.00th=[ 550], 10.00th=[ 550], 20.00th=[ 550], | 30.00th=[ 550], 40.00th=[ 550], 50.00th=[ 550], 60.00th=[ 550], | 70.00th=[ 550], 80.00th=[ 550], 90.00th=[ 550], 95.00th=[ 550], | 99.00th=[ 550], 99.50th=[ 550], 99.90th=[ 550], 99.95th=[ 550], | 99.99th=[ 550] lat (msec) : 750=100.00% cpu : usr=0.00%, sys=2.01%, ctx=4, majf=0, minf=9 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=66.9MiB/s (70.1MB/s), 66.9MiB/s-66.9MiB/s (70.1MB/s-70.1MB/s), io=40.0MiB (41.9MB), run=598-598msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout