== sanity test 398c: run fio to test AIO ================= 12:57:46 (1789318666) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.127959 s, 328 MB/s osc.lustre-OST0000-osc-ffff90d678614800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=155850: Sun Sep 13 12:58:16 2026 write: IOPS=96, BW=387KiB/s (396kB/s)(10.0MiB/26454msec); 0 zone resets slat (usec): min=21, max=1242, avg=126.00, stdev=75.21 clat (msec): min=39, max=285, avg=165.01, stdev=34.96 lat (msec): min=39, max=285, avg=165.14, stdev=34.97 clat percentiles (msec): | 1.00th=[ 101], 5.00th=[ 112], 10.00th=[ 122], 20.00th=[ 134], | 30.00th=[ 144], 40.00th=[ 155], 50.00th=[ 165], 60.00th=[ 176], | 70.00th=[ 184], 80.00th=[ 194], 90.00th=[ 211], 95.00th=[ 226], | 99.00th=[ 245], 99.50th=[ 251], 99.90th=[ 288], 99.95th=[ 288], | 99.99th=[ 288] bw ( KiB/s): min= 264, max= 609, per=25.01%, avg=386.42, stdev=65.09, samples=52 iops : min= 66, max= 152, avg=96.56, stdev=16.30, samples=52 lat (msec) : 50=0.62%, 100=0.35%, 250=98.40%, 500=0.62% cpu : usr=0.19%, sys=1.08%, ctx=2592, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=155851: Sun Sep 13 12:58:16 2026 write: IOPS=96, BW=387KiB/s (396kB/s)(10.0MiB/26454msec); 0 zone resets slat (usec): min=20, max=1397, avg=129.01, stdev=77.91 clat (msec): min=32, max=285, avg=165.09, stdev=34.83 lat (msec): min=32, max=285, avg=165.22, stdev=34.84 clat percentiles (msec): | 1.00th=[ 99], 5.00th=[ 112], 10.00th=[ 122], 20.00th=[ 136], | 30.00th=[ 144], 40.00th=[ 155], 50.00th=[ 165], 60.00th=[ 176], | 70.00th=[ 184], 80.00th=[ 194], 90.00th=[ 211], 95.00th=[ 224], | 99.00th=[ 243], 99.50th=[ 249], 99.90th=[ 264], 99.95th=[ 264], | 99.99th=[ 288] bw ( KiB/s): min= 279, max= 609, per=24.99%, avg=386.12, stdev=63.98, samples=52 iops : min= 69, max= 152, avg=96.48, stdev=16.02, samples=52 lat (msec) : 50=0.62%, 100=0.55%, 250=98.40%, 500=0.43% cpu : usr=0.19%, sys=1.10%, ctx=2697, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=155852: Sun Sep 13 12:58:16 2026 write: IOPS=96, BW=387KiB/s (396kB/s)(10.0MiB/26487msec); 0 zone resets slat (usec): min=22, max=640, avg=124.66, stdev=68.55 clat (msec): min=58, max=259, avg=165.23, stdev=34.43 lat (msec): min=59, max=259, avg=165.36, stdev=34.44 clat percentiles (msec): | 1.00th=[ 94], 5.00th=[ 112], 10.00th=[ 124], 20.00th=[ 136], | 30.00th=[ 144], 40.00th=[ 157], 50.00th=[ 165], 60.00th=[ 176], | 70.00th=[ 184], 80.00th=[ 194], 90.00th=[ 209], 95.00th=[ 226], | 99.00th=[ 243], 99.50th=[ 249], 99.90th=[ 257], 99.95th=[ 257], | 99.99th=[ 259] bw ( KiB/s): min= 264, max= 512, per=24.91%, avg=384.92, stdev=59.92, samples=52 iops : min= 66, max= 128, avg=96.17, stdev=14.99, samples=52 lat (msec) : 100=2.23%, 250=97.50%, 500=0.27% cpu : usr=0.15%, sys=1.12%, ctx=2635, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=155853: Sun Sep 13 12:58:16 2026 write: IOPS=96, BW=386KiB/s (396kB/s)(10.0MiB/26503msec); 0 zone resets slat (usec): min=22, max=1561, avg=128.33, stdev=78.08 clat (msec): min=23, max=252, avg=165.31, stdev=34.03 lat (msec): min=23, max=253, avg=165.44, stdev=34.03 clat percentiles (msec): | 1.00th=[ 94], 5.00th=[ 114], 10.00th=[ 123], 20.00th=[ 136], | 30.00th=[ 146], 40.00th=[ 157], 50.00th=[ 167], 60.00th=[ 176], | 70.00th=[ 184], 80.00th=[ 194], 90.00th=[ 209], 95.00th=[ 226], | 99.00th=[ 243], 99.50th=[ 247], 99.90th=[ 253], 99.95th=[ 253], | 99.99th=[ 253] bw ( KiB/s): min= 264, max= 546, per=24.86%, avg=384.13, stdev=62.56, samples=52 iops : min= 66, max= 136, avg=95.98, stdev=15.65, samples=52 lat (msec) : 50=0.20%, 100=1.84%, 250=97.73%, 500=0.23% cpu : usr=0.21%, sys=1.05%, ctx=2858, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1545KiB/s (1583kB/s), 386KiB/s-387KiB/s (396kB/s-396kB/s), io=40.0MiB (41.9MB), run=26454-26503msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=155866: Sun Sep 13 12:58:38 2026 read: IOPS=66, BW=268KiB/s (274kB/s)(5048KiB/18861msec) slat (usec): min=958, max=6502, avg=2292.33, stdev=756.75 clat (msec): min=3, max=142, avg=72.40, stdev=13.57 lat (msec): min=5, max=144, avg=74.69, stdev=13.68 clat percentiles (msec): | 1.00th=[ 40], 5.00th=[ 53], 10.00th=[ 57], 20.00th=[ 63], | 30.00th=[ 66], 40.00th=[ 69], 50.00th=[ 73], 60.00th=[ 77], | 70.00th=[ 81], 80.00th=[ 84], 90.00th=[ 89], 95.00th=[ 94], | 99.00th=[ 104], 99.50th=[ 107], 99.90th=[ 133], 99.95th=[ 142], | 99.99th=[ 142] bw ( KiB/s): min= 136, max= 360, per=25.14%, avg=266.73, stdev=59.60, samples=37 iops : min= 34, max= 90, avg=66.68, stdev=14.90, samples=37 write: IOPS=68, BW=275KiB/s (282kB/s)(5192KiB/18861msec); 0 zone resets slat (usec): min=32, max=1451, avg=117.11, stdev=80.58 clat (msec): min=63, max=270, avg=159.19, stdev=35.45 lat (msec): min=63, max=270, avg=159.31, stdev=35.45 clat percentiles (msec): | 1.00th=[ 86], 5.00th=[ 105], 10.00th=[ 115], 20.00th=[ 129], | 30.00th=[ 140], 40.00th=[ 148], 50.00th=[ 159], 60.00th=[ 167], | 70.00th=[ 176], 80.00th=[ 188], 90.00th=[ 203], 95.00th=[ 224], | 99.00th=[ 257], 99.50th=[ 262], 99.90th=[ 271], 99.95th=[ 271], | 99.99th=[ 271] bw ( KiB/s): min= 200, max= 344, per=24.80%, avg=274.03, stdev=35.85, samples=37 iops : min= 50, max= 86, avg=68.49, stdev= 8.99, samples=37 lat (msec) : 4=0.08%, 10=0.04%, 20=0.16%, 50=1.52%, 100=48.40% lat (msec) : 250=49.10%, 500=0.70% cpu : usr=0.28%, sys=1.72%, ctx=5181, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=155867: Sun Sep 13 12:58:38 2026 read: IOPS=65, BW=263KiB/s (270kB/s)(4976KiB/18900msec) slat (usec): min=940, max=6794, avg=2276.86, stdev=761.50 clat (msec): min=3, max=113, avg=73.08, stdev=13.20 lat (msec): min=5, max=117, avg=75.36, stdev=13.27 clat percentiles (msec): | 1.00th=[ 42], 5.00th=[ 53], 10.00th=[ 58], 20.00th=[ 63], | 30.00th=[ 67], 40.00th=[ 70], 50.00th=[ 73], 60.00th=[ 78], | 70.00th=[ 81], 80.00th=[ 85], 90.00th=[ 90], 95.00th=[ 94], | 99.00th=[ 102], 99.50th=[ 105], 99.90th=[ 110], 99.95th=[ 114], | 99.99th=[ 114] bw ( KiB/s): min= 96, max= 368, per=24.67%, avg=261.76, stdev=64.95, samples=37 iops : min= 24, max= 92, avg=65.43, stdev=16.24, samples=37 write: IOPS=69, BW=279KiB/s (285kB/s)(5264KiB/18900msec); 0 zone resets slat (usec): min=21, max=768, avg=113.04, stdev=70.77 clat (msec): min=48, max=270, avg=158.03, stdev=36.32 lat (msec): min=49, max=270, avg=158.15, stdev=36.32 clat percentiles (msec): | 1.00th=[ 70], 5.00th=[ 104], 10.00th=[ 115], 20.00th=[ 130], | 30.00th=[ 140], 40.00th=[ 148], 50.00th=[ 157], 60.00th=[ 165], | 70.00th=[ 176], 80.00th=[ 186], 90.00th=[ 203], 95.00th=[ 224], | 99.00th=[ 251], 99.50th=[ 262], 99.90th=[ 266], 99.95th=[ 271], | 99.99th=[ 271] bw ( KiB/s): min= 200, max= 384, per=25.01%, avg=276.41, stdev=39.78, samples=37 iops : min= 50, max= 96, avg=69.08, stdev= 9.95, samples=37 lat (msec) : 4=0.08%, 10=0.04%, 20=0.16%, 50=1.41%, 100=48.59% lat (msec) : 250=49.18%, 500=0.55% cpu : usr=0.20%, sys=1.79%, ctx=5007, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=155868: Sun Sep 13 12:58:38 2026 read: IOPS=65, BW=264KiB/s (270kB/s)(4984KiB/18893msec) slat (usec): min=955, max=5640, avg=2294.85, stdev=761.98 clat (msec): min=2, max=116, avg=73.29, stdev=13.32 lat (msec): min=5, max=118, avg=75.58, stdev=13.38 clat percentiles (msec): | 1.00th=[ 43], 5.00th=[ 53], 10.00th=[ 58], 20.00th=[ 63], | 30.00th=[ 67], 40.00th=[ 70], 50.00th=[ 74], 60.00th=[ 78], | 70.00th=[ 81], 80.00th=[ 84], 90.00th=[ 90], 95.00th=[ 95], | 99.00th=[ 104], 99.50th=[ 106], 99.90th=[ 115], 99.95th=[ 117], | 99.99th=[ 117] bw ( KiB/s): min= 176, max= 360, per=24.85%, avg=263.62, stdev=51.37, samples=37 iops : min= 44, max= 90, avg=65.89, stdev=12.85, samples=37 write: IOPS=69, BW=278KiB/s (285kB/s)(5256KiB/18893msec); 0 zone resets slat (usec): min=28, max=547, avg=113.31, stdev=64.26 clat (msec): min=52, max=277, avg=157.70, stdev=35.14 lat (msec): min=52, max=277, avg=157.81, stdev=35.15 clat percentiles (msec): | 1.00th=[ 80], 5.00th=[ 104], 10.00th=[ 115], 20.00th=[ 129], | 30.00th=[ 140], 40.00th=[ 148], 50.00th=[ 157], 60.00th=[ 165], | 70.00th=[ 174], 80.00th=[ 186], 90.00th=[ 203], 95.00th=[ 222], | 99.00th=[ 251], 99.50th=[ 255], 99.90th=[ 266], 99.95th=[ 279], | 99.99th=[ 279] bw ( KiB/s): min= 176, max= 392, per=24.95%, avg=275.68, stdev=45.75, samples=37 iops : min= 44, max= 98, avg=68.92, stdev=11.44, samples=37 lat (msec) : 4=0.04%, 10=0.04%, 20=0.16%, 50=1.37%, 100=48.05% lat (msec) : 250=49.92%, 500=0.43% cpu : usr=0.28%, sys=1.72%, ctx=5205, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=155869: Sun Sep 13 12:58:38 2026 read: IOPS=67, BW=270KiB/s (277kB/s)(5060KiB/18739msec) slat (usec): min=32, max=7206, avg=2257.42, stdev=781.55 clat (msec): min=3, max=137, avg=72.38, stdev=14.06 lat (msec): min=4, max=140, avg=74.64, stdev=14.18 clat percentiles (msec): | 1.00th=[ 18], 5.00th=[ 52], 10.00th=[ 56], 20.00th=[ 63], | 30.00th=[ 67], 40.00th=[ 70], 50.00th=[ 73], 60.00th=[ 77], | 70.00th=[ 81], 80.00th=[ 84], 90.00th=[ 89], 95.00th=[ 93], | 99.00th=[ 102], 99.50th=[ 105], 99.90th=[ 125], 99.95th=[ 138], | 99.99th=[ 138] bw ( KiB/s): min= 168, max= 392, per=25.51%, avg=270.70, stdev=51.24, samples=37 iops : min= 42, max= 98, avg=67.68, stdev=12.81, samples=37 write: IOPS=69, BW=276KiB/s (283kB/s)(5180KiB/18739msec); 0 zone resets slat (usec): min=25, max=897, avg=115.52, stdev=71.97 clat (msec): min=25, max=268, avg=158.30, stdev=35.82 lat (msec): min=25, max=268, avg=158.41, stdev=35.81 clat percentiles (msec): | 1.00th=[ 75], 5.00th=[ 105], 10.00th=[ 115], 20.00th=[ 130], | 30.00th=[ 140], 40.00th=[ 148], 50.00th=[ 157], 60.00th=[ 165], | 70.00th=[ 176], 80.00th=[ 186], 90.00th=[ 203], 95.00th=[ 222], | 99.00th=[ 253], 99.50th=[ 262], 99.90th=[ 268], 99.95th=[ 268], | 99.99th=[ 268] bw ( KiB/s): min= 168, max= 360, per=24.94%, avg=275.62, stdev=39.47, samples=37 iops : min= 42, max= 90, avg=68.89, stdev= 9.87, samples=37 lat (msec) : 4=0.04%, 10=0.20%, 20=0.31%, 50=1.64%, 100=48.44% lat (msec) : 250=48.75%, 500=0.62% cpu : usr=0.25%, sys=1.77%, ctx=5252, majf=1, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1062KiB/s (1087kB/s), 263KiB/s-270KiB/s (270kB/s-277kB/s), io=19.6MiB (20.5MB), run=18739-18900msec WRITE: bw=1105KiB/s (1132kB/s), 275KiB/s-279KiB/s (282kB/s-285kB/s), io=20.4MiB (21.4MB), run=18739-18900msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=155875: Sun Sep 13 12:58:41 2026 read: IOPS=7, BW=294MiB/s (308MB/s)(40.0MiB/136msec) slat (nsec): min=14050k, max=14050k, avg=14050411.00, stdev= 0.00 clat (nsec): min=116407k, max=116407k, avg=116407267.00, stdev= 0.00 lat (nsec): min=130463k, max=130463k, avg=130462616.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 117], 5.00th=[ 117], 10.00th=[ 117], 20.00th=[ 117], | 30.00th=[ 117], 40.00th=[ 117], 50.00th=[ 117], 60.00th=[ 117], | 70.00th=[ 117], 80.00th=[ 117], 90.00th=[ 117], 95.00th=[ 117], | 99.00th=[ 117], 99.50th=[ 117], 99.90th=[ 117], 99.95th=[ 117], | 99.99th=[ 117] lat (msec) : 250=100.00% cpu : usr=0.00%, sys=9.63%, ctx=42, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=294MiB/s (308MB/s), 294MiB/s-294MiB/s (308MB/s-308MB/s), io=40.0MiB (41.9MB), run=136-136msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout