== sanity test 398c: run fio to test AIO ================= 15:30:14 (1744054214) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB) copied, 0.105636 s, 397 MB/s osc.lustre-OST0000-osc-ffff88012c498000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=10983: Mon Apr 7 15:30:52 2025 write: IOPS=70, BW=280KiB/s (287kB/s)(10.0MiB/36553msec) slat (usec): min=14, max=368, avg=89.31, stdev=44.23 clat (msec): min=95, max=462, avg=227.99, stdev=45.42 lat (msec): min=95, max=462, avg=228.08, stdev=45.43 clat percentiles (msec): | 1.00th=[ 116], 5.00th=[ 146], 10.00th=[ 169], 20.00th=[ 190], | 30.00th=[ 209], 40.00th=[ 222], 50.00th=[ 230], 60.00th=[ 241], | 70.00th=[ 251], 80.00th=[ 264], 90.00th=[ 284], 95.00th=[ 300], | 99.00th=[ 334], 99.50th=[ 342], 99.90th=[ 430], 99.95th=[ 447], | 99.99th=[ 464] bw ( KiB/s): min= 168, max= 448, per=24.91%, avg=278.74, stdev=45.79, samples=73 iops : min= 42, max= 112, avg=69.56, stdev=11.45, samples=73 lat (msec) : 100=0.04%, 250=69.80%, 500=30.16% cpu : usr=0.13%, sys=0.77%, ctx=2052, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=10984: Mon Apr 7 15:30:52 2025 write: IOPS=70, BW=280KiB/s (287kB/s)(10.0MiB/36518msec) slat (usec): min=13, max=631, avg=90.72, stdev=48.66 clat (msec): min=14, max=548, avg=227.89, stdev=47.38 lat (msec): min=14, max=548, avg=227.98, stdev=47.39 clat percentiles (msec): | 1.00th=[ 116], 5.00th=[ 148], 10.00th=[ 169], 20.00th=[ 192], | 30.00th=[ 207], 40.00th=[ 220], 50.00th=[ 230], 60.00th=[ 241], | 70.00th=[ 249], 80.00th=[ 264], 90.00th=[ 284], 95.00th=[ 300], | 99.00th=[ 334], 99.50th=[ 363], 99.90th=[ 531], 99.95th=[ 531], | 99.99th=[ 550] bw ( KiB/s): min= 104, max= 464, per=24.91%, avg=278.78, stdev=52.91, samples=73 iops : min= 26, max= 116, avg=69.60, stdev=13.25, samples=73 lat (msec) : 20=0.04%, 50=0.12%, 100=0.39%, 250=70.82%, 500=28.48% lat (msec) : 750=0.16% cpu : usr=0.17%, sys=0.73%, ctx=2049, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=10985: Mon Apr 7 15:30:52 2025 write: IOPS=70, BW=280KiB/s (287kB/s)(10.0MiB/36536msec) slat (usec): min=13, max=552, avg=91.20, stdev=47.90 clat (msec): min=85, max=428, avg=227.89, stdev=44.50 lat (msec): min=85, max=428, avg=227.99, stdev=44.50 clat percentiles (msec): | 1.00th=[ 115], 5.00th=[ 148], 10.00th=[ 169], 20.00th=[ 192], | 30.00th=[ 209], 40.00th=[ 220], 50.00th=[ 230], 60.00th=[ 241], | 70.00th=[ 249], 80.00th=[ 264], 90.00th=[ 284], 95.00th=[ 300], | 99.00th=[ 330], 99.50th=[ 342], 99.90th=[ 380], 99.95th=[ 380], | 99.99th=[ 430] bw ( KiB/s): min= 208, max= 432, per=24.91%, avg=278.77, stdev=44.29, samples=73 iops : min= 52, max= 108, avg=69.59, stdev=11.09, samples=73 lat (msec) : 100=0.04%, 250=70.55%, 500=29.41% cpu : usr=0.10%, sys=0.80%, ctx=2157, majf=0, minf=28 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=10986: Mon Apr 7 15:30:52 2025 write: IOPS=69, BW=280KiB/s (287kB/s)(10.0MiB/36582msec) slat (usec): min=14, max=544, avg=89.57, stdev=47.12 clat (msec): min=45, max=357, avg=228.16, stdev=43.41 lat (msec): min=45, max=357, avg=228.25, stdev=43.42 clat percentiles (msec): | 1.00th=[ 122], 5.00th=[ 144], 10.00th=[ 171], 20.00th=[ 194], | 30.00th=[ 211], 40.00th=[ 222], 50.00th=[ 230], 60.00th=[ 241], | 70.00th=[ 249], 80.00th=[ 262], 90.00th=[ 284], 95.00th=[ 296], | 99.00th=[ 330], 99.50th=[ 338], 99.90th=[ 351], 99.95th=[ 359], | 99.99th=[ 359] bw ( KiB/s): min= 192, max= 384, per=24.88%, avg=278.36, stdev=42.58, samples=73 iops : min= 48, max= 96, avg=69.51, stdev=10.59, samples=73 lat (msec) : 50=0.08%, 100=0.12%, 250=70.94%, 500=28.87% cpu : usr=0.13%, sys=0.76%, ctx=1981, majf=0, minf=29 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1120KiB/s (1147kB/s), 280KiB/s-280KiB/s (287kB/s-287kB/s), io=40.0MiB (41.9MB), run=36518-36582msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=11025: Mon Apr 7 15:31:13 2025 read: IOPS=63, BW=254KiB/s (261kB/s)(5048KiB/19836msec) slat (usec): min=711, max=4255, avg=1416.66, stdev=447.01 clat (msec): min=2, max=139, avg=79.70, stdev=19.30 lat (msec): min=3, max=141, avg=81.12, stdev=19.53 clat percentiles (msec): | 1.00th=[ 43], 5.00th=[ 54], 10.00th=[ 58], 20.00th=[ 65], | 30.00th=[ 70], 40.00th=[ 73], 50.00th=[ 78], 60.00th=[ 82], | 70.00th=[ 88], 80.00th=[ 96], 90.00th=[ 107], 95.00th=[ 116], | 99.00th=[ 131], 99.50th=[ 133], 99.90th=[ 140], 99.95th=[ 140], | 99.99th=[ 140] bw ( KiB/s): min= 104, max= 400, per=24.89%, avg=251.62, stdev=67.90, samples=39 iops : min= 26, max= 100, avg=62.85, stdev=16.96, samples=39 write: IOPS=65, BW=262KiB/s (268kB/s)(5192KiB/19836msec) slat (usec): min=14, max=1186, avg=67.93, stdev=56.15 clat (msec): min=58, max=364, avg=165.09, stdev=47.50 lat (msec): min=58, max=364, avg=165.15, stdev=47.52 clat percentiles (msec): | 1.00th=[ 84], 5.00th=[ 101], 10.00th=[ 112], 20.00th=[ 126], | 30.00th=[ 136], 40.00th=[ 148], 50.00th=[ 157], 60.00th=[ 171], | 70.00th=[ 184], 80.00th=[ 201], 90.00th=[ 230], 95.00th=[ 255], | 99.00th=[ 313], 99.50th=[ 326], 99.90th=[ 342], 99.95th=[ 363], | 99.99th=[ 363] bw ( KiB/s): min= 176, max= 352, per=24.62%, avg=259.21, stdev=46.88, samples=39 iops : min= 44, max= 88, avg=64.74, stdev=11.77, samples=39 lat (msec) : 4=0.04%, 20=0.20%, 50=1.41%, 100=42.34%, 250=52.89% lat (msec) : 500=3.12% cpu : usr=0.21%, sys=1.48%, ctx=3421, majf=0, minf=34 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=11026: Mon Apr 7 15:31:13 2025 read: IOPS=62, BW=251KiB/s (257kB/s)(4976KiB/19807msec) slat (usec): min=32, max=4329, avg=1432.25, stdev=461.13 clat (usec): min=1815, max=142843, avg=79826.67, stdev=19711.03 lat (msec): min=2, max=144, avg=81.26, stdev=19.94 clat percentiles (msec): | 1.00th=[ 43], 5.00th=[ 54], 10.00th=[ 58], 20.00th=[ 65], | 30.00th=[ 70], 40.00th=[ 73], 50.00th=[ 78], 60.00th=[ 82], | 70.00th=[ 88], 80.00th=[ 96], 90.00th=[ 107], 95.00th=[ 118], | 99.00th=[ 133], 99.50th=[ 140], 99.90th=[ 144], 99.95th=[ 144], | 99.99th=[ 144] bw ( KiB/s): min= 96, max= 448, per=24.71%, avg=249.82, stdev=72.08, samples=39 iops : min= 24, max= 112, avg=62.44, stdev=18.00, samples=39 write: IOPS=66, BW=266KiB/s (272kB/s)(5264KiB/19807msec) slat (usec): min=14, max=319, avg=68.47, stdev=40.01 clat (msec): min=32, max=346, avg=163.76, stdev=46.88 lat (msec): min=32, max=346, avg=163.83, stdev=46.89 clat percentiles (msec): | 1.00th=[ 79], 5.00th=[ 100], 10.00th=[ 111], 20.00th=[ 126], | 30.00th=[ 136], 40.00th=[ 146], 50.00th=[ 159], 60.00th=[ 169], | 70.00th=[ 182], 80.00th=[ 201], 90.00th=[ 226], 95.00th=[ 253], | 99.00th=[ 300], 99.50th=[ 317], 99.90th=[ 342], 99.95th=[ 347], | 99.99th=[ 347] bw ( KiB/s): min= 168, max= 360, per=25.01%, avg=263.36, stdev=47.49, samples=39 iops : min= 42, max= 90, avg=65.82, stdev=11.88, samples=39 lat (msec) : 2=0.04%, 10=0.08%, 20=0.12%, 50=1.41%, 100=42.11% lat (msec) : 250=53.40%, 500=2.85% cpu : usr=0.15%, sys=1.57%, ctx=3262, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=11027: Mon Apr 7 15:31:13 2025 read: IOPS=62, BW=251KiB/s (257kB/s)(4984KiB/19836msec) slat (usec): min=684, max=3974, avg=1451.11, stdev=465.83 clat (msec): min=4, max=154, avg=80.18, stdev=20.49 lat (msec): min=5, max=156, avg=81.63, stdev=20.72 clat percentiles (msec): | 1.00th=[ 39], 5.00th=[ 55], 10.00th=[ 57], 20.00th=[ 64], | 30.00th=[ 69], 40.00th=[ 73], 50.00th=[ 78], 60.00th=[ 83], | 70.00th=[ 88], 80.00th=[ 97], 90.00th=[ 111], 95.00th=[ 118], | 99.00th=[ 134], 99.50th=[ 140], 99.90th=[ 142], 99.95th=[ 155], | 99.99th=[ 155] bw ( KiB/s): min= 104, max= 384, per=24.75%, avg=250.18, stdev=60.17, samples=39 iops : min= 26, max= 96, avg=62.49, stdev=15.04, samples=39 write: IOPS=66, BW=265KiB/s (271kB/s)(5256KiB/19836msec) slat (usec): min=12, max=326, avg=66.51, stdev=38.65 clat (msec): min=54, max=343, avg=163.71, stdev=46.00 lat (msec): min=54, max=343, avg=163.78, stdev=46.01 clat percentiles (msec): | 1.00th=[ 82], 5.00th=[ 101], 10.00th=[ 112], 20.00th=[ 125], | 30.00th=[ 136], 40.00th=[ 148], 50.00th=[ 159], 60.00th=[ 169], | 70.00th=[ 180], 80.00th=[ 199], 90.00th=[ 226], 95.00th=[ 255], | 99.00th=[ 292], 99.50th=[ 317], 99.90th=[ 338], 99.95th=[ 342], | 99.99th=[ 342] bw ( KiB/s): min= 160, max= 376, per=24.79%, avg=261.05, stdev=54.07, samples=39 iops : min= 40, max= 94, avg=65.21, stdev=13.48, samples=39 lat (msec) : 10=0.04%, 20=0.23%, 50=1.41%, 100=40.94%, 250=54.53% lat (msec) : 500=2.85% cpu : usr=0.17%, sys=1.54%, ctx=3464, majf=0, minf=32 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=11028: Mon Apr 7 15:31:13 2025 read: IOPS=64, BW=257KiB/s (263kB/s)(5060KiB/19725msec) slat (usec): min=732, max=4478, avg=1427.67, stdev=463.10 clat (msec): min=2, max=154, avg=79.87, stdev=20.79 lat (msec): min=3, max=156, avg=81.30, stdev=21.04 clat percentiles (msec): | 1.00th=[ 16], 5.00th=[ 54], 10.00th=[ 58], 20.00th=[ 64], | 30.00th=[ 69], 40.00th=[ 73], 50.00th=[ 78], 60.00th=[ 83], | 70.00th=[ 89], 80.00th=[ 97], 90.00th=[ 109], 95.00th=[ 120], | 99.00th=[ 131], 99.50th=[ 136], 99.90th=[ 142], 99.95th=[ 155], | 99.99th=[ 155] bw ( KiB/s): min= 120, max= 456, per=25.32%, avg=255.95, stdev=65.82, samples=39 iops : min= 30, max= 114, avg=63.95, stdev=16.46, samples=39 write: IOPS=65, BW=263KiB/s (269kB/s)(5180KiB/19725msec) slat (usec): min=14, max=301, avg=66.60, stdev=35.96 clat (msec): min=62, max=358, avg=163.68, stdev=45.94 lat (msec): min=62, max=358, avg=163.75, stdev=45.96 clat percentiles (msec): | 1.00th=[ 81], 5.00th=[ 101], 10.00th=[ 111], 20.00th=[ 126], | 30.00th=[ 136], 40.00th=[ 146], 50.00th=[ 157], 60.00th=[ 169], | 70.00th=[ 182], 80.00th=[ 201], 90.00th=[ 224], 95.00th=[ 253], | 99.00th=[ 292], 99.50th=[ 321], 99.90th=[ 359], 99.95th=[ 359], | 99.99th=[ 359] bw ( KiB/s): min= 152, max= 360, per=24.75%, avg=260.67, stdev=53.98, samples=39 iops : min= 38, max= 90, avg=65.13, stdev=13.46, samples=39 lat (msec) : 4=0.08%, 20=0.47%, 50=1.37%, 100=41.52%, 250=53.95% lat (msec) : 500=2.62% cpu : usr=0.22%, sys=1.53%, ctx=3183, majf=0, minf=32 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1012KiB/s (1036kB/s), 251KiB/s-257KiB/s (257kB/s-263kB/s), io=19.6MiB (20.5MB), run=19725-19836msec WRITE: bw=1053KiB/s (1079kB/s), 262KiB/s-266KiB/s (268kB/s-272kB/s), io=20.4MiB (21.4MB), run=19725-19836msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.7 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=11048: Mon Apr 7 15:31:14 2025 read: IOPS=11, BW=471MiB/s (493MB/s)(40.0MiB/85msec) slat (nsec): min=6679.2k, max=6679.2k, avg=6679241.00, stdev= 0.00 clat (nsec): min=72662k, max=72662k, avg=72662210.00, stdev= 0.00 lat (nsec): min=79364k, max=79364k, avg=79364065.00, stdev= 0.00 clat percentiles (usec): | 1.00th=[72877], 5.00th=[72877], 10.00th=[72877], 20.00th=[72877], | 30.00th=[72877], 40.00th=[72877], 50.00th=[72877], 60.00th=[72877], | 70.00th=[72877], 80.00th=[72877], 90.00th=[72877], 95.00th=[72877], | 99.00th=[72877], 99.50th=[72877], 99.90th=[72877], 99.95th=[72877], | 99.99th=[72877] lat (msec) : 100=100.00% cpu : usr=0.00%, sys=7.14%, ctx=17, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=471MiB/s (493MB/s), 471MiB/s-471MiB/s (493MB/s-493MB/s), io=40.0MiB (41.9MB), run=85-85msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout