== sanity test 398c: run fio to test AIO ================= 03:53:09 (1749541989) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB) copied, 0.0955948 s, 439 MB/s osc.lustre-OST0000-osc-ffff8800b7c1f800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=18170: Tue Jun 10 03:53:37 2025 write: IOPS=96, BW=384KiB/s (393kB/s)(10.0MiB/26664msec) slat (usec): min=12, max=437, avg=75.98, stdev=51.01 clat (msec): min=61, max=238, avg=166.37, stdev=28.87 lat (msec): min=61, max=238, avg=166.45, stdev=28.87 clat percentiles (msec): | 1.00th=[ 102], 5.00th=[ 120], 10.00th=[ 129], 20.00th=[ 140], | 30.00th=[ 153], 40.00th=[ 161], 50.00th=[ 169], 60.00th=[ 176], | 70.00th=[ 184], 80.00th=[ 194], 90.00th=[ 205], 95.00th=[ 211], | 99.00th=[ 224], 99.50th=[ 228], 99.90th=[ 234], 99.95th=[ 236], | 99.99th=[ 239] bw ( KiB/s): min= 272, max= 512, per=24.93%, avg=382.72, stdev=51.73, samples=53 iops : min= 68, max= 128, avg=95.62, stdev=12.93, samples=53 lat (msec) : 100=0.86%, 250=99.14% cpu : usr=0.18%, sys=0.77%, ctx=2083, majf=0, minf=29 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=18171: Tue Jun 10 03:53:37 2025 write: IOPS=95, BW=384KiB/s (393kB/s)(10.0MiB/26673msec) slat (usec): min=13, max=425, avg=79.74, stdev=47.57 clat (msec): min=26, max=244, avg=166.32, stdev=29.00 lat (msec): min=26, max=244, avg=166.40, stdev=29.00 clat percentiles (msec): | 1.00th=[ 102], 5.00th=[ 118], 10.00th=[ 128], 20.00th=[ 142], | 30.00th=[ 153], 40.00th=[ 159], 50.00th=[ 167], 60.00th=[ 176], | 70.00th=[ 184], 80.00th=[ 192], 90.00th=[ 205], 95.00th=[ 213], | 99.00th=[ 226], 99.50th=[ 228], 99.90th=[ 234], 99.95th=[ 234], | 99.99th=[ 245] bw ( KiB/s): min= 304, max= 504, per=24.92%, avg=382.49, stdev=45.93, samples=53 iops : min= 76, max= 126, avg=95.62, stdev=11.48, samples=53 lat (msec) : 50=0.12%, 100=0.78%, 250=99.10% cpu : usr=0.22%, sys=0.79%, ctx=2461, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=18172: Tue Jun 10 03:53:37 2025 write: IOPS=95, BW=384KiB/s (393kB/s)(10.0MiB/26681msec) slat (usec): min=13, max=411, avg=81.43, stdev=48.71 clat (msec): min=35, max=244, avg=166.36, stdev=28.76 lat (msec): min=36, max=244, avg=166.45, stdev=28.76 clat percentiles (msec): | 1.00th=[ 105], 5.00th=[ 121], 10.00th=[ 128], 20.00th=[ 140], | 30.00th=[ 153], 40.00th=[ 159], 50.00th=[ 167], 60.00th=[ 176], | 70.00th=[ 182], 80.00th=[ 192], 90.00th=[ 205], 95.00th=[ 213], | 99.00th=[ 228], 99.50th=[ 230], 99.90th=[ 234], 99.95th=[ 239], | 99.99th=[ 245] bw ( KiB/s): min= 304, max= 496, per=24.92%, avg=382.49, stdev=47.49, samples=53 iops : min= 76, max= 124, avg=95.62, stdev=11.87, samples=53 lat (msec) : 50=0.04%, 100=0.47%, 250=99.49% cpu : usr=0.14%, sys=0.90%, ctx=2550, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=18173: Tue Jun 10 03:53:37 2025 write: IOPS=96, BW=385KiB/s (394kB/s)(10.0MiB/26629msec) slat (usec): min=14, max=516, avg=80.06, stdev=51.84 clat (msec): min=30, max=233, avg=166.23, stdev=30.21 lat (msec): min=30, max=233, avg=166.31, stdev=30.21 clat percentiles (msec): | 1.00th=[ 102], 5.00th=[ 120], 10.00th=[ 127], 20.00th=[ 142], | 30.00th=[ 153], 40.00th=[ 161], 50.00th=[ 167], 60.00th=[ 176], | 70.00th=[ 184], 80.00th=[ 194], 90.00th=[ 205], 95.00th=[ 213], | 99.00th=[ 226], 99.50th=[ 230], 99.90th=[ 234], 99.95th=[ 234], | 99.99th=[ 234] bw ( KiB/s): min= 296, max= 520, per=24.98%, avg=383.51, stdev=50.66, samples=53 iops : min= 74, max= 130, avg=95.85, stdev=12.67, samples=53 lat (msec) : 50=0.39%, 100=0.55%, 250=99.06% cpu : usr=0.15%, sys=0.84%, ctx=2451, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1535KiB/s (1572kB/s), 384KiB/s-385KiB/s (393kB/s-394kB/s), io=40.0MiB (41.9MB), run=26629-26681msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=18205: Tue Jun 10 03:53:54 2025 read: IOPS=80, BW=322KiB/s (330kB/s)(5048KiB/15677msec) slat (usec): min=18, max=4231, avg=1285.18, stdev=332.71 clat (usec): min=1635, max=144443, avg=62454.87, stdev=12755.21 lat (msec): min=2, max=146, avg=63.74, stdev=12.81 clat percentiles (msec): | 1.00th=[ 16], 5.00th=[ 46], 10.00th=[ 50], 20.00th=[ 54], | 30.00th=[ 57], 40.00th=[ 60], 50.00th=[ 63], 60.00th=[ 66], | 70.00th=[ 69], 80.00th=[ 72], 90.00th=[ 77], 95.00th=[ 83], | 99.00th=[ 91], 99.50th=[ 95], 99.90th=[ 133], 99.95th=[ 144], | 99.99th=[ 144] bw ( KiB/s): min= 192, max= 424, per=25.35%, avg=322.74, stdev=52.77, samples=31 iops : min= 48, max= 106, avg=80.61, stdev=13.20, samples=31 write: IOPS=82, BW=331KiB/s (339kB/s)(5192KiB/15677msec) slat (usec): min=13, max=253, avg=65.61, stdev=36.22 clat (msec): min=13, max=225, avg=131.05, stdev=30.03 lat (msec): min=13, max=225, avg=131.12, stdev=30.03 clat percentiles (msec): | 1.00th=[ 68], 5.00th=[ 86], 10.00th=[ 93], 20.00th=[ 106], | 30.00th=[ 114], 40.00th=[ 124], 50.00th=[ 131], 60.00th=[ 138], | 70.00th=[ 146], 80.00th=[ 155], 90.00th=[ 171], 95.00th=[ 182], | 99.00th=[ 205], 99.50th=[ 213], 99.90th=[ 224], 99.95th=[ 226], | 99.99th=[ 226] bw ( KiB/s): min= 280, max= 384, per=24.98%, avg=331.00, stdev=27.26, samples=31 iops : min= 70, max= 96, avg=82.68, stdev= 6.85, samples=31 lat (msec) : 2=0.12%, 4=0.16%, 10=0.08%, 20=0.31%, 50=5.62% lat (msec) : 100=50.62%, 250=43.09% cpu : usr=0.27%, sys=1.76%, ctx=3073, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=18206: Tue Jun 10 03:53:54 2025 read: IOPS=78, BW=316KiB/s (323kB/s)(4976KiB/15764msec) slat (usec): min=771, max=3538, avg=1293.59, stdev=304.20 clat (msec): min=3, max=103, avg=63.06, stdev=11.53 lat (msec): min=5, max=104, avg=64.35, stdev=11.52 clat percentiles (msec): | 1.00th=[ 34], 5.00th=[ 47], 10.00th=[ 51], 20.00th=[ 55], | 30.00th=[ 57], 40.00th=[ 61], 50.00th=[ 64], 60.00th=[ 67], | 70.00th=[ 69], 80.00th=[ 73], 90.00th=[ 78], 95.00th=[ 83], | 99.00th=[ 89], 99.50th=[ 94], 99.90th=[ 95], 99.95th=[ 105], | 99.99th=[ 105] bw ( KiB/s): min= 224, max= 448, per=24.68%, avg=314.23, stdev=58.65, samples=31 iops : min= 56, max= 112, avg=78.48, stdev=14.66, samples=31 write: IOPS=83, BW=334KiB/s (342kB/s)(5264KiB/15764msec) slat (usec): min=13, max=301, avg=62.62, stdev=33.50 clat (msec): min=15, max=223, avg=130.54, stdev=29.42 lat (msec): min=16, max=224, avg=130.60, stdev=29.42 clat percentiles (msec): | 1.00th=[ 70], 5.00th=[ 83], 10.00th=[ 92], 20.00th=[ 107], | 30.00th=[ 115], 40.00th=[ 124], 50.00th=[ 131], 60.00th=[ 138], | 70.00th=[ 146], 80.00th=[ 155], 90.00th=[ 169], 95.00th=[ 180], | 99.00th=[ 203], 99.50th=[ 209], 99.90th=[ 220], 99.95th=[ 224], | 99.99th=[ 224] bw ( KiB/s): min= 280, max= 416, per=24.98%, avg=331.00, stdev=37.97, samples=31 iops : min= 70, max= 104, avg=82.68, stdev= 9.51, samples=31 lat (msec) : 4=0.04%, 10=0.12%, 20=0.23%, 50=4.30%, 100=51.76% lat (msec) : 250=43.55% cpu : usr=0.23%, sys=1.76%, ctx=2955, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=18207: Tue Jun 10 03:53:54 2025 read: IOPS=79, BW=316KiB/s (324kB/s)(4984KiB/15763msec) slat (usec): min=718, max=3286, avg=1295.47, stdev=302.83 clat (usec): min=1149, max=95609, avg=62212.59, stdev=12304.14 lat (usec): min=2444, max=96870, avg=63510.09, stdev=12313.27 clat percentiles (usec): | 1.00th=[12256], 5.00th=[45351], 10.00th=[48497], 20.00th=[53216], | 30.00th=[56361], 40.00th=[58983], 50.00th=[62129], 60.00th=[65274], | 70.00th=[68682], 80.00th=[71828], 90.00th=[76022], 95.00th=[81265], | 99.00th=[90702], 99.50th=[93848], 99.90th=[94897], 99.95th=[95945], | 99.99th=[95945] bw ( KiB/s): min= 208, max= 424, per=24.79%, avg=315.55, stdev=55.83, samples=31 iops : min= 52, max= 106, avg=78.84, stdev=13.95, samples=31 write: IOPS=83, BW=333KiB/s (341kB/s)(5256KiB/15763msec) slat (usec): min=13, max=290, avg=66.01, stdev=36.20 clat (msec): min=48, max=218, avg=131.37, stdev=29.92 lat (msec): min=49, max=218, avg=131.44, stdev=29.92 clat percentiles (msec): | 1.00th=[ 69], 5.00th=[ 83], 10.00th=[ 93], 20.00th=[ 107], | 30.00th=[ 116], 40.00th=[ 124], 50.00th=[ 132], 60.00th=[ 140], | 70.00th=[ 146], 80.00th=[ 157], 90.00th=[ 174], 95.00th=[ 182], | 99.00th=[ 203], 99.50th=[ 211], 99.90th=[ 215], 99.95th=[ 220], | 99.99th=[ 220] bw ( KiB/s): min= 256, max= 416, per=24.87%, avg=329.48, stdev=35.54, samples=31 iops : min= 64, max= 104, avg=82.32, stdev= 8.92, samples=31 lat (msec) : 2=0.08%, 4=0.12%, 10=0.23%, 20=0.20%, 50=5.90% lat (msec) : 100=50.27%, 250=43.20% cpu : usr=0.23%, sys=1.78%, ctx=3155, majf=0, minf=32 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=18208: Tue Jun 10 03:53:54 2025 read: IOPS=80, BW=323KiB/s (331kB/s)(5060KiB/15676msec) slat (usec): min=768, max=3402, avg=1280.58, stdev=288.96 clat (msec): min=7, max=144, avg=62.65, stdev=12.40 lat (msec): min=8, max=146, avg=63.93, stdev=12.41 clat percentiles (msec): | 1.00th=[ 16], 5.00th=[ 46], 10.00th=[ 50], 20.00th=[ 54], | 30.00th=[ 57], 40.00th=[ 60], 50.00th=[ 63], 60.00th=[ 66], | 70.00th=[ 69], 80.00th=[ 72], 90.00th=[ 78], 95.00th=[ 82], | 99.00th=[ 93], 99.50th=[ 94], 99.90th=[ 144], 99.95th=[ 144], | 99.99th=[ 144] bw ( KiB/s): min= 160, max= 440, per=25.47%, avg=324.29, stdev=59.03, samples=31 iops : min= 40, max= 110, avg=81.00, stdev=14.75, samples=31 write: IOPS=82, BW=330KiB/s (338kB/s)(5180KiB/15676msec) slat (usec): min=14, max=244, avg=64.04, stdev=34.38 clat (msec): min=50, max=222, avg=130.92, stdev=29.94 lat (msec): min=50, max=222, avg=130.98, stdev=29.94 clat percentiles (msec): | 1.00th=[ 72], 5.00th=[ 85], 10.00th=[ 92], 20.00th=[ 105], | 30.00th=[ 115], 40.00th=[ 124], 50.00th=[ 131], 60.00th=[ 138], | 70.00th=[ 146], 80.00th=[ 157], 90.00th=[ 171], 95.00th=[ 182], | 99.00th=[ 209], 99.50th=[ 220], 99.90th=[ 222], 99.95th=[ 222], | 99.99th=[ 222] bw ( KiB/s): min= 256, max= 408, per=24.90%, avg=329.97, stdev=36.45, samples=31 iops : min= 64, max= 102, avg=82.42, stdev= 9.14, samples=31 lat (msec) : 10=0.16%, 20=0.39%, 50=5.16%, 100=52.07%, 250=42.23% cpu : usr=0.30%, sys=1.71%, ctx=3217, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1273KiB/s (1304kB/s), 316KiB/s-323KiB/s (323kB/s-331kB/s), io=19.6MiB (20.5MB), run=15676-15764msec WRITE: bw=1325KiB/s (1357kB/s), 330KiB/s-334KiB/s (338kB/s-342kB/s), io=20.4MiB (21.4MB), run=15676-15764msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.7 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=18224: Tue Jun 10 03:53:55 2025 read: IOPS=12, BW=500MiB/s (524MB/s)(40.0MiB/80msec) slat (nsec): min=6645.0k, max=6645.0k, avg=6645972.00, stdev= 0.00 clat (nsec): min=68994k, max=68994k, avg=68993785.00, stdev= 0.00 lat (nsec): min=75659k, max=75659k, avg=75659195.00, stdev= 0.00 clat percentiles (usec): | 1.00th=[68682], 5.00th=[68682], 10.00th=[68682], 20.00th=[68682], | 30.00th=[68682], 40.00th=[68682], 50.00th=[68682], 60.00th=[68682], | 70.00th=[68682], 80.00th=[68682], 90.00th=[68682], 95.00th=[68682], | 99.00th=[68682], 99.50th=[68682], 99.90th=[68682], 99.95th=[68682], | 99.99th=[68682] lat (msec) : 100=100.00% cpu : usr=0.00%, sys=7.59%, ctx=16, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=500MiB/s (524MB/s), 500MiB/s-500MiB/s (524MB/s-524MB/s), io=40.0MiB (41.9MB), run=80-80msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout