== sanity test 398c: run fio to test AIO ================= 17:49:56 (1713217796) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB) copied, 0.136083 s, 308 MB/s osc.lustre-OST0000-osc-ffff8800b583e800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=16165: Mon Apr 15 17:50:27 2024 write: IOPS=86, BW=345KiB/s (353kB/s)(10.0MiB/29685msec) slat (usec): min=13, max=430, avg=54.86, stdev=36.89 clat (msec): min=31, max=292, avg=185.11, stdev=31.53 lat (msec): min=31, max=292, avg=185.17, stdev=31.53 clat percentiles (msec): | 1.00th=[ 117], 5.00th=[ 138], 10.00th=[ 148], 20.00th=[ 157], | 30.00th=[ 167], 40.00th=[ 176], 50.00th=[ 186], 60.00th=[ 192], | 70.00th=[ 201], 80.00th=[ 211], 90.00th=[ 228], 95.00th=[ 241], | 99.00th=[ 259], 99.50th=[ 264], 99.90th=[ 288], 99.95th=[ 292], | 99.99th=[ 292] bw ( KiB/s): min= 256, max= 512, per=24.95%, avg=343.80, stdev=45.48, samples=59 iops : min= 64, max= 128, avg=85.90, stdev=11.37, samples=59 lat (msec) : 50=0.04%, 100=0.27%, 250=97.46%, 500=2.23% cpu : usr=0.10%, sys=0.62%, ctx=1959, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=16166: Mon Apr 15 17:50:27 2024 write: IOPS=86, BW=345KiB/s (354kB/s)(10.0MiB/29650msec) slat (usec): min=13, max=258, avg=52.70, stdev=31.72 clat (msec): min=54, max=348, avg=184.89, stdev=32.31 lat (msec): min=54, max=348, avg=184.94, stdev=32.31 clat percentiles (msec): | 1.00th=[ 117], 5.00th=[ 138], 10.00th=[ 146], 20.00th=[ 157], | 30.00th=[ 167], 40.00th=[ 176], 50.00th=[ 184], 60.00th=[ 192], | 70.00th=[ 201], 80.00th=[ 211], 90.00th=[ 226], 95.00th=[ 239], | 99.00th=[ 264], 99.50th=[ 271], 99.90th=[ 326], 99.95th=[ 326], | 99.99th=[ 351] bw ( KiB/s): min= 264, max= 512, per=25.02%, avg=344.75, stdev=45.26, samples=59 iops : min= 66, max= 128, avg=86.14, stdev=11.31, samples=59 lat (msec) : 100=0.39%, 250=97.19%, 500=2.42% cpu : usr=0.11%, sys=0.62%, ctx=1749, majf=0, minf=28 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=16167: Mon Apr 15 17:50:27 2024 write: IOPS=86, BW=345KiB/s (353kB/s)(10.0MiB/29674msec) slat (usec): min=13, max=287, avg=51.23, stdev=32.15 clat (msec): min=31, max=301, avg=185.04, stdev=32.36 lat (msec): min=31, max=301, avg=185.10, stdev=32.36 clat percentiles (msec): | 1.00th=[ 112], 5.00th=[ 138], 10.00th=[ 144], 20.00th=[ 157], | 30.00th=[ 167], 40.00th=[ 178], 50.00th=[ 186], 60.00th=[ 192], | 70.00th=[ 201], 80.00th=[ 211], 90.00th=[ 228], 95.00th=[ 241], | 99.00th=[ 262], 99.50th=[ 271], 99.90th=[ 288], 99.95th=[ 300], | 99.99th=[ 300] bw ( KiB/s): min= 264, max= 496, per=24.96%, avg=343.97, stdev=45.26, samples=59 iops : min= 66, max= 124, avg=85.97, stdev=11.32, samples=59 lat (msec) : 50=0.04%, 100=0.35%, 250=97.23%, 500=2.38% cpu : usr=0.13%, sys=0.57%, ctx=1814, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=16168: Mon Apr 15 17:50:27 2024 write: IOPS=86, BW=345KiB/s (353kB/s)(10.0MiB/29704msec) slat (usec): min=12, max=578, avg=52.44, stdev=39.12 clat (msec): min=53, max=292, avg=185.26, stdev=31.89 lat (msec): min=53, max=292, avg=185.31, stdev=31.89 clat percentiles (msec): | 1.00th=[ 112], 5.00th=[ 140], 10.00th=[ 148], 20.00th=[ 157], | 30.00th=[ 167], 40.00th=[ 176], 50.00th=[ 184], 60.00th=[ 194], | 70.00th=[ 203], 80.00th=[ 213], 90.00th=[ 228], 95.00th=[ 241], | 99.00th=[ 259], 99.50th=[ 271], 99.90th=[ 279], 99.95th=[ 288], | 99.99th=[ 292] bw ( KiB/s): min= 256, max= 520, per=24.86%, avg=342.61, stdev=47.57, samples=59 iops : min= 64, max= 130, avg=85.63, stdev=11.88, samples=59 lat (msec) : 100=0.43%, 250=97.27%, 500=2.30% cpu : usr=0.10%, sys=0.58%, ctx=1916, majf=0, minf=29 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1379KiB/s (1412kB/s), 345KiB/s-345KiB/s (353kB/s-354kB/s), io=40.0MiB (41.9MB), run=29650-29704msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=16202: Mon Apr 15 17:50:46 2024 read: IOPS=71, BW=286KiB/s (292kB/s)(5048KiB/17679msec) slat (usec): min=28, max=4348, avg=1234.71, stdev=334.19 clat (usec): min=1674, max=165168, avg=73349.88, stdev=14705.39 lat (msec): min=3, max=166, avg=74.59, stdev=14.73 clat percentiles (msec): | 1.00th=[ 39], 5.00th=[ 53], 10.00th=[ 59], 20.00th=[ 63], | 30.00th=[ 67], 40.00th=[ 70], 50.00th=[ 73], 60.00th=[ 77], | 70.00th=[ 80], 80.00th=[ 84], 90.00th=[ 91], 95.00th=[ 95], | 99.00th=[ 114], 99.50th=[ 136], 99.90th=[ 165], 99.95th=[ 165], | 99.99th=[ 165] bw ( KiB/s): min= 184, max= 384, per=25.12%, avg=285.14, stdev=51.63, samples=35 iops : min= 46, max= 96, avg=71.20, stdev=12.89, samples=35 write: IOPS=73, BW=294KiB/s (301kB/s)(5192KiB/17679msec) slat (usec): min=13, max=1361, avg=46.25, stdev=42.77 clat (msec): min=68, max=255, avg=145.24, stdev=33.85 lat (msec): min=68, max=255, avg=145.29, stdev=33.86 clat percentiles (msec): | 1.00th=[ 79], 5.00th=[ 93], 10.00th=[ 104], 20.00th=[ 116], | 30.00th=[ 125], 40.00th=[ 136], 50.00th=[ 144], 60.00th=[ 153], | 70.00th=[ 163], 80.00th=[ 174], 90.00th=[ 190], 95.00th=[ 205], | 99.00th=[ 234], 99.50th=[ 245], 99.90th=[ 255], 99.95th=[ 255], | 99.99th=[ 255] bw ( KiB/s): min= 215, max= 368, per=24.82%, avg=293.14, stdev=35.14, samples=35 iops : min= 53, max= 92, avg=73.20, stdev= 8.79, samples=35 lat (msec) : 2=0.04%, 10=0.08%, 20=0.27%, 50=1.33%, 100=50.51% lat (msec) : 250=47.62%, 500=0.16% cpu : usr=0.16%, sys=1.41%, ctx=2736, majf=0, minf=33 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=16203: Mon Apr 15 17:50:46 2024 read: IOPS=70, BW=281KiB/s (288kB/s)(4976KiB/17679msec) slat (usec): min=32, max=3173, avg=1231.93, stdev=293.43 clat (usec): min=1662, max=164224, avg=73267.09, stdev=13614.82 lat (msec): min=3, max=165, avg=74.50, stdev=13.63 clat percentiles (msec): | 1.00th=[ 43], 5.00th=[ 55], 10.00th=[ 59], 20.00th=[ 63], | 30.00th=[ 67], 40.00th=[ 70], 50.00th=[ 73], 60.00th=[ 77], | 70.00th=[ 80], 80.00th=[ 84], 90.00th=[ 90], 95.00th=[ 94], | 99.00th=[ 112], 99.50th=[ 120], 99.90th=[ 163], 99.95th=[ 165], | 99.99th=[ 165] bw ( KiB/s): min= 167, max= 384, per=24.72%, avg=280.57, stdev=53.94, samples=35 iops : min= 41, max= 96, avg=70.06, stdev=13.50, samples=35 write: IOPS=74, BW=298KiB/s (305kB/s)(5264KiB/17679msec) slat (usec): min=12, max=1059, avg=45.62, stdev=34.85 clat (msec): min=26, max=278, avg=144.36, stdev=33.54 lat (msec): min=26, max=278, avg=144.41, stdev=33.54 clat percentiles (msec): | 1.00th=[ 80], 5.00th=[ 93], 10.00th=[ 104], 20.00th=[ 115], | 30.00th=[ 125], 40.00th=[ 136], 50.00th=[ 144], 60.00th=[ 150], | 70.00th=[ 161], 80.00th=[ 171], 90.00th=[ 190], 95.00th=[ 205], | 99.00th=[ 228], 99.50th=[ 245], 99.90th=[ 279], 99.95th=[ 279], | 99.99th=[ 279] bw ( KiB/s): min= 239, max= 400, per=25.21%, avg=297.71, stdev=38.90, samples=35 iops : min= 59, max= 100, avg=74.34, stdev= 9.72, samples=35 lat (msec) : 2=0.04%, 20=0.16%, 50=1.09%, 100=50.20%, 250=48.36% lat (msec) : 500=0.16% cpu : usr=0.23%, sys=1.36%, ctx=2770, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=16204: Mon Apr 15 17:50:46 2024 read: IOPS=70, BW=282KiB/s (289kB/s)(4984KiB/17669msec) slat (usec): min=30, max=2963, avg=1241.13, stdev=300.53 clat (msec): min=3, max=199, avg=73.37, stdev=13.62 lat (msec): min=5, max=200, avg=74.62, stdev=13.63 clat percentiles (msec): | 1.00th=[ 43], 5.00th=[ 53], 10.00th=[ 58], 20.00th=[ 64], | 30.00th=[ 67], 40.00th=[ 70], 50.00th=[ 73], 60.00th=[ 77], | 70.00th=[ 80], 80.00th=[ 84], 90.00th=[ 91], 95.00th=[ 94], | 99.00th=[ 106], 99.50th=[ 110], 99.90th=[ 155], 99.95th=[ 199], | 99.99th=[ 199] bw ( KiB/s): min= 167, max= 407, per=24.96%, avg=283.31, stdev=63.91, samples=35 iops : min= 41, max= 101, avg=70.74, stdev=15.97, samples=35 write: IOPS=74, BW=297KiB/s (305kB/s)(5256KiB/17669msec) slat (usec): min=13, max=173, avg=44.41, stdev=20.18 clat (msec): min=26, max=263, avg=144.22, stdev=34.12 lat (msec): min=26, max=263, avg=144.27, stdev=34.12 clat percentiles (msec): | 1.00th=[ 81], 5.00th=[ 93], 10.00th=[ 103], 20.00th=[ 113], | 30.00th=[ 125], 40.00th=[ 134], 50.00th=[ 142], 60.00th=[ 153], | 70.00th=[ 161], 80.00th=[ 171], 90.00th=[ 188], 95.00th=[ 205], | 99.00th=[ 232], 99.50th=[ 245], 99.90th=[ 255], 99.95th=[ 264], | 99.99th=[ 264] bw ( KiB/s): min= 208, max= 408, per=25.15%, avg=297.03, stdev=39.34, samples=35 iops : min= 52, max= 102, avg=74.17, stdev= 9.83, samples=35 lat (msec) : 4=0.04%, 10=0.04%, 20=0.16%, 50=1.25%, 100=51.09% lat (msec) : 250=47.23%, 500=0.20% cpu : usr=0.20%, sys=1.37%, ctx=2833, majf=0, minf=29 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=16205: Mon Apr 15 17:50:46 2024 read: IOPS=71, BW=287KiB/s (294kB/s)(5060KiB/17613msec) slat (usec): min=31, max=4091, avg=1224.74, stdev=308.23 clat (msec): min=2, max=172, avg=73.49, stdev=14.04 lat (msec): min=3, max=173, avg=74.72, stdev=14.07 clat percentiles (msec): | 1.00th=[ 48], 5.00th=[ 54], 10.00th=[ 59], 20.00th=[ 63], | 30.00th=[ 67], 40.00th=[ 70], 50.00th=[ 73], 60.00th=[ 77], | 70.00th=[ 80], 80.00th=[ 84], 90.00th=[ 90], 95.00th=[ 94], | 99.00th=[ 114], 99.50th=[ 142], 99.90th=[ 171], 99.95th=[ 174], | 99.99th=[ 174] bw ( KiB/s): min= 192, max= 376, per=25.33%, avg=287.49, stdev=51.96, samples=35 iops : min= 48, max= 94, avg=71.83, stdev=12.94, samples=35 write: IOPS=73, BW=294KiB/s (301kB/s)(5180KiB/17613msec) slat (usec): min=13, max=590, avg=46.54, stdev=27.91 clat (msec): min=68, max=278, avg=144.46, stdev=33.88 lat (msec): min=68, max=278, avg=144.51, stdev=33.88 clat percentiles (msec): | 1.00th=[ 84], 5.00th=[ 93], 10.00th=[ 102], 20.00th=[ 114], | 30.00th=[ 126], 40.00th=[ 134], 50.00th=[ 142], 60.00th=[ 153], | 70.00th=[ 161], 80.00th=[ 171], 90.00th=[ 190], 95.00th=[ 203], | 99.00th=[ 234], 99.50th=[ 251], 99.90th=[ 264], 99.95th=[ 279], | 99.99th=[ 279] bw ( KiB/s): min= 216, max= 360, per=24.87%, avg=293.66, stdev=36.24, samples=35 iops : min= 54, max= 90, avg=73.37, stdev= 9.07, samples=35 lat (msec) : 4=0.04%, 20=0.12%, 50=1.17%, 100=51.68%, 250=46.72% lat (msec) : 500=0.27% cpu : usr=0.16%, sys=1.41%, ctx=2829, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1135KiB/s (1162kB/s), 281KiB/s-287KiB/s (288kB/s-294kB/s), io=19.6MiB (20.5MB), run=17613-17679msec WRITE: bw=1182KiB/s (1210kB/s), 294KiB/s-298KiB/s (301kB/s-305kB/s), io=20.4MiB (21.4MB), run=17613-17679msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.7 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=16222: Mon Apr 15 17:50:47 2024 read: IOPS=5, BW=230MiB/s (241MB/s)(40.0MiB/174msec) slat (nsec): min=6075.1k, max=6075.1k, avg=6075068.00, stdev= 0.00 clat (nsec): min=167149k, max=167149k, avg=167149026.00, stdev= 0.00 lat (nsec): min=173233k, max=173233k, avg=173232981.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 167], 5.00th=[ 167], 10.00th=[ 167], 20.00th=[ 167], | 30.00th=[ 167], 40.00th=[ 167], 50.00th=[ 167], 60.00th=[ 167], | 70.00th=[ 167], 80.00th=[ 167], 90.00th=[ 167], 95.00th=[ 167], | 99.00th=[ 167], 99.50th=[ 167], 99.90th=[ 167], 99.95th=[ 167], | 99.99th=[ 167] lat (msec) : 250=100.00% cpu : usr=0.00%, sys=3.47%, ctx=16, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=230MiB/s (241MB/s), 230MiB/s-230MiB/s (241MB/s-241MB/s), io=40.0MiB (41.9MB), run=174-174msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout