== sanity test 398c: run fio to test AIO ================= 00:38:22 (1789187902) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.254985 s, 164 MB/s osc.lustre-OST0000-osc-ffff919008dae800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=179283: Sat Sep 12 00:38:44 2026 write: IOPS=150, BW=600KiB/s (615kB/s)(10.0MiB/17054msec); 0 zone resets slat (usec): min=22, max=6753, avg=136.25, stdev=347.42 clat (msec): min=43, max=204, avg=105.80, stdev=21.47 lat (msec): min=43, max=204, avg=105.94, stdev=21.47 clat percentiles (msec): | 1.00th=[ 52], 5.00th=[ 72], 10.00th=[ 81], 20.00th=[ 90], | 30.00th=[ 95], 40.00th=[ 101], 50.00th=[ 104], 60.00th=[ 110], | 70.00th=[ 115], 80.00th=[ 124], 90.00th=[ 134], 95.00th=[ 144], | 99.00th=[ 161], 99.50th=[ 165], 99.90th=[ 205], 99.95th=[ 205], | 99.99th=[ 205] bw ( KiB/s): min= 432, max= 816, per=25.19%, avg=603.67, stdev=94.37, samples=33 iops : min= 108, max= 204, avg=150.70, stdev=23.61, samples=33 lat (msec) : 50=0.55%, 100=38.98%, 250=60.47% cpu : usr=0.15%, sys=1.34%, ctx=2710, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=179284: Sat Sep 12 00:38:44 2026 write: IOPS=149, BW=599KiB/s (614kB/s)(10.0MiB/17085msec); 0 zone resets slat (usec): min=22, max=11219, avg=153.88, stdev=493.02 clat (msec): min=45, max=195, avg=105.98, stdev=21.38 lat (msec): min=47, max=195, avg=106.14, stdev=21.37 clat percentiles (msec): | 1.00th=[ 53], 5.00th=[ 73], 10.00th=[ 82], 20.00th=[ 89], | 30.00th=[ 94], 40.00th=[ 100], 50.00th=[ 106], 60.00th=[ 111], | 70.00th=[ 116], 80.00th=[ 124], 90.00th=[ 133], 95.00th=[ 144], | 99.00th=[ 161], 99.50th=[ 163], 99.90th=[ 174], 99.95th=[ 174], | 99.99th=[ 197] bw ( KiB/s): min= 510, max= 768, per=25.08%, avg=600.91, stdev=78.40, samples=33 iops : min= 127, max= 192, avg=149.97, stdev=19.59, samples=33 lat (msec) : 50=0.62%, 100=40.16%, 250=59.22% cpu : usr=0.27%, sys=1.42%, ctx=2697, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=179285: Sat Sep 12 00:38:44 2026 write: IOPS=149, BW=599KiB/s (614kB/s)(10.0MiB/17091msec); 0 zone resets slat (usec): min=22, max=5766, avg=135.51, stdev=356.55 clat (msec): min=21, max=175, avg=105.84, stdev=21.31 lat (msec): min=22, max=175, avg=105.98, stdev=21.32 clat percentiles (msec): | 1.00th=[ 58], 5.00th=[ 73], 10.00th=[ 82], 20.00th=[ 88], | 30.00th=[ 94], 40.00th=[ 101], 50.00th=[ 106], 60.00th=[ 111], | 70.00th=[ 117], 80.00th=[ 124], 90.00th=[ 134], 95.00th=[ 142], | 99.00th=[ 163], 99.50th=[ 165], 99.90th=[ 169], 99.95th=[ 176], | 99.99th=[ 176] bw ( KiB/s): min= 392, max= 776, per=25.11%, avg=601.73, stdev=93.62, samples=33 iops : min= 98, max= 194, avg=150.21, stdev=23.43, samples=33 lat (msec) : 50=0.12%, 100=40.35%, 250=59.53% cpu : usr=0.17%, sys=1.23%, ctx=2494, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=179286: Sat Sep 12 00:38:44 2026 write: IOPS=150, BW=603KiB/s (618kB/s)(10.0MiB/16977msec); 0 zone resets slat (usec): min=22, max=7298, avg=162.49, stdev=419.99 clat (msec): min=10, max=242, avg=105.78, stdev=22.82 lat (msec): min=10, max=242, avg=105.94, stdev=22.81 clat percentiles (msec): | 1.00th=[ 51], 5.00th=[ 71], 10.00th=[ 82], 20.00th=[ 88], | 30.00th=[ 94], 40.00th=[ 100], 50.00th=[ 105], 60.00th=[ 110], | 70.00th=[ 116], 80.00th=[ 125], 90.00th=[ 133], 95.00th=[ 142], | 99.00th=[ 163], 99.50th=[ 171], 99.90th=[ 243], 99.95th=[ 243], | 99.99th=[ 243] bw ( KiB/s): min= 456, max= 800, per=25.27%, avg=605.48, stdev=82.84, samples=33 iops : min= 114, max= 200, avg=151.15, stdev=20.70, samples=33 lat (msec) : 20=0.16%, 50=0.66%, 100=39.84%, 250=59.34% cpu : usr=0.29%, sys=1.46%, ctx=2947, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=2397KiB/s (2454kB/s), 599KiB/s-603KiB/s (614kB/s-618kB/s), io=40.0MiB (41.9MB), run=16977-17091msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=179298: Sat Sep 12 00:39:09 2026 read: IOPS=66, BW=267KiB/s (273kB/s)(5048KiB/18919msec) slat (usec): min=1711, max=35368, avg=7871.07, stdev=4282.04 clat (msec): min=45, max=227, avg=111.26, stdev=26.13 lat (msec): min=52, max=248, avg=119.14, stdev=27.27 clat percentiles (msec): | 1.00th=[ 60], 5.00th=[ 77], 10.00th=[ 81], 20.00th=[ 90], | 30.00th=[ 97], 40.00th=[ 103], 50.00th=[ 109], 60.00th=[ 114], | 70.00th=[ 123], 80.00th=[ 131], 90.00th=[ 146], 95.00th=[ 161], | 99.00th=[ 188], 99.50th=[ 197], 99.90th=[ 224], 99.95th=[ 228], | 99.99th=[ 228] bw ( KiB/s): min= 152, max= 344, per=25.13%, avg=266.35, stdev=49.72, samples=37 iops : min= 38, max= 86, avg=66.24, stdev=12.40, samples=37 write: IOPS=68, BW=274KiB/s (281kB/s)(5192KiB/18919msec); 0 zone resets slat (usec): min=32, max=4265, avg=163.01, stdev=347.13 clat (msec): min=54, max=220, avg=115.23, stdev=25.53 lat (msec): min=55, max=220, avg=115.39, stdev=25.53 clat percentiles (msec): | 1.00th=[ 67], 5.00th=[ 82], 10.00th=[ 87], 20.00th=[ 94], | 30.00th=[ 100], 40.00th=[ 107], 50.00th=[ 112], 60.00th=[ 118], | 70.00th=[ 126], 80.00th=[ 136], 90.00th=[ 148], 95.00th=[ 163], | 99.00th=[ 192], 99.50th=[ 201], 99.90th=[ 215], 99.95th=[ 220], | 99.99th=[ 220] bw ( KiB/s): min= 191, max= 415, per=24.87%, avg=274.57, stdev=47.67, samples=37 iops : min= 47, max= 103, avg=68.30, stdev=11.92, samples=37 lat (msec) : 50=0.08%, 100=33.28%, 250=66.64% cpu : usr=0.31%, sys=2.22%, ctx=4813, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=179299: Sat Sep 12 00:39:09 2026 read: IOPS=65, BW=263KiB/s (270kB/s)(4976KiB/18894msec) slat (usec): min=1571, max=32679, avg=7761.94, stdev=4190.21 clat (msec): min=55, max=228, avg=112.11, stdev=25.94 lat (msec): min=62, max=244, avg=119.88, stdev=27.14 clat percentiles (msec): | 1.00th=[ 65], 5.00th=[ 75], 10.00th=[ 83], 20.00th=[ 91], | 30.00th=[ 97], 40.00th=[ 104], 50.00th=[ 110], 60.00th=[ 115], | 70.00th=[ 123], 80.00th=[ 132], 90.00th=[ 146], 95.00th=[ 159], | 99.00th=[ 197], 99.50th=[ 209], 99.90th=[ 215], 99.95th=[ 230], | 99.99th=[ 230] bw ( KiB/s): min= 144, max= 341, per=24.71%, avg=261.97, stdev=41.80, samples=37 iops : min= 36, max= 85, avg=65.14, stdev=10.41, samples=37 write: IOPS=69, BW=279KiB/s (285kB/s)(5264KiB/18894msec); 0 zone resets slat (usec): min=29, max=8248, avg=159.75, stdev=417.76 clat (msec): min=46, max=219, avg=114.70, stdev=25.82 lat (msec): min=46, max=219, avg=114.86, stdev=25.83 clat percentiles (msec): | 1.00th=[ 68], 5.00th=[ 80], 10.00th=[ 86], 20.00th=[ 94], | 30.00th=[ 101], 40.00th=[ 106], 50.00th=[ 112], 60.00th=[ 118], | 70.00th=[ 125], 80.00th=[ 133], 90.00th=[ 148], 95.00th=[ 163], | 99.00th=[ 190], 99.50th=[ 197], 99.90th=[ 218], 99.95th=[ 220], | 99.99th=[ 220] bw ( KiB/s): min= 152, max= 423, per=25.28%, avg=279.05, stdev=56.64, samples=37 iops : min= 38, max= 105, avg=69.41, stdev=14.14, samples=37 lat (msec) : 50=0.08%, 100=32.50%, 250=67.42% cpu : usr=0.26%, sys=2.24%, ctx=4704, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=179300: Sat Sep 12 00:39:09 2026 read: IOPS=66, BW=266KiB/s (272kB/s)(4984KiB/18754msec) slat (usec): min=45, max=28517, avg=7895.89, stdev=4122.55 clat (msec): min=11, max=254, avg=111.04, stdev=28.34 lat (msec): min=22, max=268, avg=118.94, stdev=29.28 clat percentiles (msec): | 1.00th=[ 57], 5.00th=[ 73], 10.00th=[ 80], 20.00th=[ 89], | 30.00th=[ 96], 40.00th=[ 103], 50.00th=[ 109], 60.00th=[ 115], | 70.00th=[ 124], 80.00th=[ 131], 90.00th=[ 144], 95.00th=[ 159], | 99.00th=[ 215], 99.50th=[ 226], 99.90th=[ 245], 99.95th=[ 255], | 99.99th=[ 255] bw ( KiB/s): min= 183, max= 384, per=25.07%, avg=265.78, stdev=48.03, samples=37 iops : min= 45, max= 96, avg=66.11, stdev=12.07, samples=37 write: IOPS=70, BW=280KiB/s (287kB/s)(5256KiB/18754msec); 0 zone resets slat (usec): min=31, max=4601, avg=152.20, stdev=342.25 clat (msec): min=16, max=254, avg=114.94, stdev=27.00 lat (msec): min=17, max=254, avg=115.10, stdev=27.01 clat percentiles (msec): | 1.00th=[ 68], 5.00th=[ 83], 10.00th=[ 88], 20.00th=[ 94], | 30.00th=[ 100], 40.00th=[ 105], 50.00th=[ 111], 60.00th=[ 117], | 70.00th=[ 124], 80.00th=[ 134], 90.00th=[ 150], 95.00th=[ 163], | 99.00th=[ 203], 99.50th=[ 224], 99.90th=[ 255], 99.95th=[ 255], | 99.99th=[ 255] bw ( KiB/s): min= 144, max= 444, per=25.28%, avg=279.08, stdev=61.00, samples=37 iops : min= 36, max= 111, avg=69.43, stdev=15.19, samples=37 lat (msec) : 20=0.08%, 50=0.20%, 100=33.67%, 250=65.90%, 500=0.16% cpu : usr=0.27%, sys=2.17%, ctx=4700, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=179301: Sat Sep 12 00:39:09 2026 read: IOPS=66, BW=267KiB/s (274kB/s)(5060KiB/18921msec) slat (usec): min=1650, max=114080, avg=7647.09, stdev=4967.95 clat (msec): min=11, max=242, avg=111.48, stdev=27.89 lat (msec): min=25, max=249, avg=119.13, stdev=28.83 clat percentiles (msec): | 1.00th=[ 58], 5.00th=[ 75], 10.00th=[ 81], 20.00th=[ 90], | 30.00th=[ 96], 40.00th=[ 102], 50.00th=[ 108], 60.00th=[ 115], | 70.00th=[ 123], 80.00th=[ 132], 90.00th=[ 148], 95.00th=[ 161], | 99.00th=[ 203], 99.50th=[ 220], 99.90th=[ 236], 99.95th=[ 243], | 99.99th=[ 243] bw ( KiB/s): min= 184, max= 408, per=25.15%, avg=266.59, stdev=48.69, samples=37 iops : min= 46, max= 102, avg=66.30, stdev=12.17, samples=37 write: IOPS=68, BW=274KiB/s (280kB/s)(5180KiB/18921msec); 0 zone resets slat (usec): min=23, max=7897, avg=161.67, stdev=435.98 clat (msec): min=51, max=226, avg=115.42, stdev=25.79 lat (msec): min=52, max=226, avg=115.58, stdev=25.78 clat percentiles (msec): | 1.00th=[ 63], 5.00th=[ 80], 10.00th=[ 88], 20.00th=[ 95], | 30.00th=[ 102], 40.00th=[ 108], 50.00th=[ 113], 60.00th=[ 118], | 70.00th=[ 126], 80.00th=[ 134], 90.00th=[ 146], 95.00th=[ 163], | 99.00th=[ 199], 99.50th=[ 205], 99.90th=[ 222], 99.95th=[ 226], | 99.99th=[ 226] bw ( KiB/s): min= 175, max= 423, per=24.68%, avg=272.51, stdev=51.42, samples=37 iops : min= 43, max= 105, avg=67.78, stdev=12.79, samples=37 lat (msec) : 20=0.04%, 50=0.16%, 100=32.11%, 250=67.70% cpu : usr=0.27%, sys=2.20%, ctx=4438, majf=0, minf=16 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1061KiB/s (1086kB/s), 263KiB/s-267KiB/s (270kB/s-274kB/s), io=19.6MiB (20.5MB), run=18754-18921msec WRITE: bw=1104KiB/s (1131kB/s), 274KiB/s-280KiB/s (280kB/s-287kB/s), io=20.4MiB (21.4MB), run=18754-18921msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=179308: Sat Sep 12 00:39:15 2026 read: IOPS=3, BW=153MiB/s (161MB/s)(40.0MiB/261msec) slat (nsec): min=26852k, max=26852k, avg=26851946.00, stdev= 0.00 clat (nsec): min=212831k, max=212831k, avg=212830693.00, stdev= 0.00 lat (nsec): min=239687k, max=239687k, avg=239686887.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 213], 5.00th=[ 213], 10.00th=[ 213], 20.00th=[ 213], | 30.00th=[ 213], 40.00th=[ 213], 50.00th=[ 213], 60.00th=[ 213], | 70.00th=[ 213], 80.00th=[ 213], 90.00th=[ 213], 95.00th=[ 213], | 99.00th=[ 213], 99.50th=[ 213], 99.90th=[ 213], 99.95th=[ 213], | 99.99th=[ 213] lat (msec) : 250=100.00% cpu : usr=0.00%, sys=3.08%, ctx=5, majf=0, minf=9 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=153MiB/s (161MB/s), 153MiB/s-153MiB/s (161MB/s-161MB/s), io=40.0MiB (41.9MB), run=261-261msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout