== sanity test 398c: run fio to test AIO ================= 20:40:19 (1746146419) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB) copied, 0.132546 s, 316 MB/s osc.lustre-OST0000-osc-ffff88009df5a000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=2055: Thu May 1 20:41:23 2025 write: IOPS=40, BW=163KiB/s (167kB/s)(10.0MiB/62676msec) slat (usec): min=11, max=3514, avg=107.81, stdev=202.46 clat (msec): min=180, max=757, avg=391.19, stdev=107.68 lat (msec): min=180, max=757, avg=391.30, stdev=107.69 clat percentiles (msec): | 1.00th=[ 199], 5.00th=[ 224], 10.00th=[ 251], 20.00th=[ 296], | 30.00th=[ 326], 40.00th=[ 355], 50.00th=[ 384], 60.00th=[ 418], | 70.00th=[ 447], 80.00th=[ 489], 90.00th=[ 542], 95.00th=[ 575], | 99.00th=[ 634], 99.50th=[ 693], 99.90th=[ 743], 99.95th=[ 743], | 99.99th=[ 760] bw ( KiB/s): min= 79, max= 280, per=24.92%, avg=162.76, stdev=45.57, samples=125 iops : min= 19, max= 70, avg=40.60, stdev=11.40, samples=125 lat (msec) : 250=9.84%, 500=72.27%, 750=17.85%, 1000=0.04% cpu : usr=0.15%, sys=0.60%, ctx=1672, majf=0, minf=32 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=2056: Thu May 1 20:41:23 2025 write: IOPS=40, BW=163KiB/s (167kB/s)(10.0MiB/62721msec) slat (usec): min=10, max=3768, avg=106.91, stdev=196.24 clat (msec): min=177, max=725, avg=391.46, stdev=107.23 lat (msec): min=177, max=725, avg=391.57, stdev=107.24 clat percentiles (msec): | 1.00th=[ 194], 5.00th=[ 226], 10.00th=[ 251], 20.00th=[ 296], | 30.00th=[ 330], 40.00th=[ 355], 50.00th=[ 384], 60.00th=[ 414], | 70.00th=[ 447], 80.00th=[ 493], 90.00th=[ 535], 95.00th=[ 567], | 99.00th=[ 651], 99.50th=[ 693], 99.90th=[ 709], 99.95th=[ 726], | 99.99th=[ 726] bw ( KiB/s): min= 56, max= 304, per=24.92%, avg=162.76, stdev=48.03, samples=125 iops : min= 14, max= 76, avg=40.60, stdev=12.04, samples=125 lat (msec) : 250=10.08%, 500=71.33%, 750=18.59% cpu : usr=0.33%, sys=0.41%, ctx=1615, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=2057: Thu May 1 20:41:23 2025 write: IOPS=40, BW=164KiB/s (168kB/s)(10.0MiB/62526msec) slat (usec): min=10, max=2330, avg=100.38, stdev=154.27 clat (msec): min=35, max=763, avg=390.31, stdev=110.34 lat (msec): min=35, max=763, avg=390.41, stdev=110.35 clat percentiles (msec): | 1.00th=[ 194], 5.00th=[ 222], 10.00th=[ 251], 20.00th=[ 292], | 30.00th=[ 326], 40.00th=[ 355], 50.00th=[ 388], 60.00th=[ 418], | 70.00th=[ 451], 80.00th=[ 485], 90.00th=[ 535], 95.00th=[ 584], | 99.00th=[ 667], 99.50th=[ 709], 99.90th=[ 735], 99.95th=[ 760], | 99.99th=[ 760] bw ( KiB/s): min= 71, max= 272, per=24.91%, avg=162.64, stdev=46.11, samples=125 iops : min= 17, max= 68, avg=40.48, stdev=11.59, samples=125 lat (msec) : 50=0.04%, 100=0.23%, 250=9.49%, 500=73.12%, 750=17.03% lat (msec) : 1000=0.08% cpu : usr=0.00%, sys=0.75%, ctx=1604, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=2058: Thu May 1 20:41:23 2025 write: IOPS=40, BW=163KiB/s (167kB/s)(10.0MiB/62722msec) slat (usec): min=11, max=3223, avg=104.59, stdev=187.63 clat (msec): min=180, max=763, avg=391.45, stdev=106.03 lat (msec): min=182, max=763, avg=391.55, stdev=106.04 clat percentiles (msec): | 1.00th=[ 199], 5.00th=[ 228], 10.00th=[ 255], 20.00th=[ 296], | 30.00th=[ 330], 40.00th=[ 355], 50.00th=[ 384], 60.00th=[ 414], | 70.00th=[ 447], 80.00th=[ 485], 90.00th=[ 531], 95.00th=[ 567], | 99.00th=[ 659], 99.50th=[ 701], 99.90th=[ 760], 99.95th=[ 760], | 99.99th=[ 768] bw ( KiB/s): min= 80, max= 264, per=24.91%, avg=162.63, stdev=44.97, samples=125 iops : min= 20, max= 66, avg=40.47, stdev=11.27, samples=125 lat (msec) : 250=9.22%, 500=73.12%, 750=17.54%, 1000=0.12% cpu : usr=0.04%, sys=0.67%, ctx=1730, majf=0, minf=27 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=653KiB/s (669kB/s), 163KiB/s-164KiB/s (167kB/s-168kB/s), io=40.0MiB (41.9MB), run=62526-62722msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=2118: Thu May 1 20:42:04 2025 read: IOPS=31, BW=127KiB/s (130kB/s)(5048KiB/39899msec) slat (usec): min=925, max=12570, avg=3275.56, stdev=1578.37 clat (msec): min=4, max=401, avg=165.14, stdev=47.15 lat (msec): min=6, max=405, avg=168.42, stdev=47.81 clat percentiles (msec): | 1.00th=[ 70], 5.00th=[ 100], 10.00th=[ 111], 20.00th=[ 126], | 30.00th=[ 140], 40.00th=[ 150], 50.00th=[ 161], 60.00th=[ 171], | 70.00th=[ 186], 80.00th=[ 203], 90.00th=[ 228], 95.00th=[ 243], | 99.00th=[ 292], 99.50th=[ 317], 99.90th=[ 359], 99.95th=[ 401], | 99.99th=[ 401] bw ( KiB/s): min= 40, max= 272, per=25.16%, avg=126.29, stdev=42.80, samples=79 iops : min= 10, max= 68, avg=31.43, stdev=10.74, samples=79 write: IOPS=32, BW=130KiB/s (133kB/s)(5192KiB/39899msec) slat (usec): min=12, max=2263, avg=103.51, stdev=146.24 clat (msec): min=97, max=653, avg=326.96, stdev=89.12 lat (msec): min=97, max=653, avg=327.06, stdev=89.12 clat percentiles (msec): | 1.00th=[ 144], 5.00th=[ 188], 10.00th=[ 215], 20.00th=[ 247], | 30.00th=[ 271], 40.00th=[ 300], 50.00th=[ 326], 60.00th=[ 351], | 70.00th=[ 376], 80.00th=[ 405], 90.00th=[ 451], 95.00th=[ 481], | 99.00th=[ 527], 99.50th=[ 542], 99.90th=[ 642], 99.95th=[ 651], | 99.99th=[ 651] bw ( KiB/s): min= 55, max= 232, per=24.85%, avg=129.73, stdev=33.31, samples=79 iops : min= 13, max= 58, avg=32.29, stdev= 8.44, samples=79 lat (msec) : 10=0.08%, 20=0.04%, 50=0.23%, 100=2.34%, 250=55.43% lat (msec) : 500=40.39%, 750=1.48% cpu : usr=0.32%, sys=1.60%, ctx=2757, majf=0, minf=34 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2119: Thu May 1 20:42:04 2025 read: IOPS=31, BW=124KiB/s (127kB/s)(4976KiB/39968msec) slat (usec): min=930, max=13519, avg=3262.68, stdev=1570.95 clat (msec): min=3, max=340, avg=168.51, stdev=46.64 lat (msec): min=5, max=344, avg=171.77, stdev=47.26 clat percentiles (msec): | 1.00th=[ 74], 5.00th=[ 103], 10.00th=[ 115], 20.00th=[ 129], | 30.00th=[ 142], 40.00th=[ 155], 50.00th=[ 165], 60.00th=[ 176], | 70.00th=[ 188], 80.00th=[ 209], 90.00th=[ 230], 95.00th=[ 249], | 99.00th=[ 292], 99.50th=[ 313], 99.90th=[ 334], 99.95th=[ 342], | 99.99th=[ 342] bw ( KiB/s): min= 48, max= 256, per=24.65%, avg=123.75, stdev=40.91, samples=79 iops : min= 12, max= 64, avg=30.71, stdev=10.30, samples=79 write: IOPS=32, BW=132KiB/s (135kB/s)(5264KiB/39968msec) slat (usec): min=13, max=2274, avg=115.78, stdev=184.85 clat (msec): min=99, max=579, avg=322.57, stdev=90.64 lat (msec): min=99, max=579, avg=322.69, stdev=90.65 clat percentiles (msec): | 1.00th=[ 144], 5.00th=[ 186], 10.00th=[ 207], 20.00th=[ 239], | 30.00th=[ 264], 40.00th=[ 292], 50.00th=[ 321], 60.00th=[ 347], | 70.00th=[ 376], 80.00th=[ 405], 90.00th=[ 443], 95.00th=[ 477], | 99.00th=[ 542], 99.50th=[ 542], 99.90th=[ 575], 99.95th=[ 584], | 99.99th=[ 584] bw ( KiB/s): min= 64, max= 224, per=25.08%, avg=130.94, stdev=35.96, samples=79 iops : min= 16, max= 56, avg=32.51, stdev= 9.02, samples=79 lat (msec) : 4=0.04%, 10=0.04%, 20=0.04%, 50=0.23%, 100=1.56% lat (msec) : 250=57.07%, 500=40.00%, 750=1.02% cpu : usr=0.18%, sys=1.76%, ctx=2668, majf=0, minf=32 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2120: Thu May 1 20:42:04 2025 read: IOPS=31, BW=125KiB/s (128kB/s)(4984KiB/39894msec) slat (usec): min=1126, max=16725, avg=3289.34, stdev=1650.97 clat (msec): min=3, max=355, avg=164.75, stdev=46.67 lat (msec): min=6, max=357, avg=168.05, stdev=47.36 clat percentiles (msec): | 1.00th=[ 27], 5.00th=[ 100], 10.00th=[ 113], 20.00th=[ 128], | 30.00th=[ 138], 40.00th=[ 150], 50.00th=[ 163], 60.00th=[ 174], | 70.00th=[ 186], 80.00th=[ 203], 90.00th=[ 228], 95.00th=[ 241], | 99.00th=[ 288], 99.50th=[ 317], 99.90th=[ 351], 99.95th=[ 355], | 99.99th=[ 355] bw ( KiB/s): min= 63, max= 256, per=24.95%, avg=125.27, stdev=41.38, samples=79 iops : min= 15, max= 64, avg=31.16, stdev=10.38, samples=79 write: IOPS=32, BW=132KiB/s (135kB/s)(5256KiB/39894msec) slat (usec): min=11, max=2732, avg=109.58, stdev=174.98 clat (msec): min=53, max=654, avg=325.49, stdev=90.36 lat (msec): min=54, max=654, avg=325.61, stdev=90.36 clat percentiles (msec): | 1.00th=[ 144], 5.00th=[ 186], 10.00th=[ 211], 20.00th=[ 241], | 30.00th=[ 268], 40.00th=[ 296], 50.00th=[ 326], 60.00th=[ 355], | 70.00th=[ 372], 80.00th=[ 401], 90.00th=[ 443], 95.00th=[ 481], | 99.00th=[ 527], 99.50th=[ 542], 99.90th=[ 651], 99.95th=[ 659], | 99.99th=[ 659] bw ( KiB/s): min= 56, max= 248, per=25.08%, avg=130.94, stdev=36.48, samples=79 iops : min= 14, max= 62, avg=32.58, stdev= 9.21, samples=79 lat (msec) : 4=0.04%, 10=0.08%, 20=0.04%, 50=0.43%, 100=2.03% lat (msec) : 250=56.29%, 500=39.69%, 750=1.41% cpu : usr=0.21%, sys=1.71%, ctx=2778, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2121: Thu May 1 20:42:04 2025 read: IOPS=31, BW=128KiB/s (131kB/s)(5060KiB/39637msec) slat (usec): min=23, max=13603, avg=3280.29, stdev=1563.50 clat (msec): min=3, max=555, avg=165.45, stdev=49.76 lat (msec): min=5, max=558, avg=168.73, stdev=50.34 clat percentiles (msec): | 1.00th=[ 36], 5.00th=[ 100], 10.00th=[ 110], 20.00th=[ 126], | 30.00th=[ 138], 40.00th=[ 150], 50.00th=[ 163], 60.00th=[ 174], | 70.00th=[ 186], 80.00th=[ 203], 90.00th=[ 228], 95.00th=[ 245], | 99.00th=[ 288], 99.50th=[ 326], 99.90th=[ 447], 99.95th=[ 558], | 99.99th=[ 558] bw ( KiB/s): min= 23, max= 232, per=25.41%, avg=127.58, stdev=40.52, samples=79 iops : min= 5, max= 58, avg=31.81, stdev=10.12, samples=79 write: IOPS=32, BW=131KiB/s (134kB/s)(5180KiB/39637msec) slat (usec): min=13, max=3986, avg=120.12, stdev=218.37 clat (msec): min=51, max=697, avg=324.30, stdev=95.20 lat (msec): min=51, max=697, avg=324.42, stdev=95.21 clat percentiles (msec): | 1.00th=[ 122], 5.00th=[ 176], 10.00th=[ 205], 20.00th=[ 239], | 30.00th=[ 268], 40.00th=[ 296], 50.00th=[ 321], 60.00th=[ 351], | 70.00th=[ 376], 80.00th=[ 409], 90.00th=[ 447], 95.00th=[ 485], | 99.00th=[ 542], 99.50th=[ 651], 99.90th=[ 701], 99.95th=[ 701], | 99.99th=[ 701] bw ( KiB/s): min= 64, max= 248, per=24.89%, avg=129.91, stdev=36.62, samples=79 iops : min= 16, max= 62, avg=32.39, stdev= 9.13, samples=79 lat (msec) : 4=0.04%, 10=0.16%, 20=0.16%, 50=0.23%, 100=2.23% lat (msec) : 250=56.56%, 500=38.91%, 750=1.72% cpu : usr=0.33%, sys=1.64%, ctx=2717, majf=0, minf=29 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=502KiB/s (514kB/s), 124KiB/s-128KiB/s (127kB/s-131kB/s), io=19.6MiB (20.5MB), run=39637-39968msec WRITE: bw=523KiB/s (535kB/s), 130KiB/s-132KiB/s (133kB/s-135kB/s), io=20.4MiB (21.4MB), run=39637-39968msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.7 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=2157: Thu May 1 20:42:06 2025 read: IOPS=4, BW=172MiB/s (181MB/s)(40.0MiB/232msec) slat (nsec): min=11897k, max=11897k, avg=11897041.00, stdev= 0.00 clat (nsec): min=211639k, max=211639k, avg=211639424.00, stdev= 0.00 lat (nsec): min=223561k, max=223561k, avg=223561314.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 211], 5.00th=[ 211], 10.00th=[ 211], 20.00th=[ 211], | 30.00th=[ 211], 40.00th=[ 211], 50.00th=[ 211], 60.00th=[ 211], | 70.00th=[ 211], 80.00th=[ 211], 90.00th=[ 211], 95.00th=[ 211], | 99.00th=[ 211], 99.50th=[ 211], 99.90th=[ 211], 99.95th=[ 211], | 99.99th=[ 211] lat (msec) : 250=100.00% cpu : usr=0.00%, sys=0.87%, ctx=4, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=172MiB/s (181MB/s), 172MiB/s-172MiB/s (181MB/s-181MB/s), io=40.0MiB (41.9MB), run=232-232msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout