== sanity test 398c: run fio to test AIO ================= 07:23:34 (1744543414)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB) copied, 0.134073 s, 313 MB/s
osc.lustre-OST0000-osc-ffff88013693b800.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=8245: Sun Apr 13 07:24:15 2025
  write: IOPS=66, BW=267KiB/s (273kB/s)(10.0MiB/38350msec)
    slat (usec): min=13, max=464, avg=71.79, stdev=43.65
    clat (msec): min=44, max=419, avg=239.42, stdev=65.85
     lat (msec): min=44, max=419, avg=239.50, stdev=65.86
    clat percentiles (msec):
     |  1.00th=[  115],  5.00th=[  136], 10.00th=[  146], 20.00th=[  174],
     | 30.00th=[  197], 40.00th=[  222], 50.00th=[  245], 60.00th=[  268],
     | 70.00th=[  288], 80.00th=[  300], 90.00th=[  321], 95.00th=[  334],
     | 99.00th=[  363], 99.50th=[  376], 99.90th=[  388], 99.95th=[  405],
     | 99.99th=[  418]
   bw (  KiB/s): min=  159, max=  464, per=24.83%, avg=264.71, stdev=72.73, samples=76
   iops        : min=   39, max=  116, avg=66.08, stdev=18.19, samples=76
  lat (msec)   : 50=0.12%, 100=0.39%, 250=51.41%, 500=48.09%
  cpu          : usr=0.13%, sys=0.59%, ctx=1871, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=8246: Sun Apr 13 07:24:15 2025
  write: IOPS=66, BW=267KiB/s (273kB/s)(10.0MiB/38388msec)
    slat (usec): min=13, max=483, avg=72.23, stdev=42.78
    clat (msec): min=55, max=419, avg=239.39, stdev=65.58
     lat (msec): min=55, max=419, avg=239.46, stdev=65.59
    clat percentiles (msec):
     |  1.00th=[  120],  5.00th=[  136], 10.00th=[  146], 20.00th=[  171],
     | 30.00th=[  197], 40.00th=[  222], 50.00th=[  243], 60.00th=[  266],
     | 70.00th=[  284], 80.00th=[  305], 90.00th=[  326], 95.00th=[  338],
     | 99.00th=[  363], 99.50th=[  376], 99.90th=[  388], 99.95th=[  401],
     | 99.99th=[  418]
   bw (  KiB/s): min=  176, max=  510, per=24.80%, avg=264.36, stdev=73.76, samples=76
   iops        : min=   44, max=  127, avg=65.97, stdev=18.46, samples=76
  lat (msec)   : 100=0.16%, 250=52.54%, 500=47.30%
  cpu          : usr=0.11%, sys=0.62%, ctx=1806, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=8247: Sun Apr 13 07:24:15 2025
  write: IOPS=66, BW=267KiB/s (273kB/s)(10.0MiB/38388msec)
    slat (usec): min=14, max=462, avg=72.40, stdev=42.17
    clat (msec): min=91, max=402, avg=239.55, stdev=65.70
     lat (msec): min=92, max=402, avg=239.62, stdev=65.71
    clat percentiles (msec):
     |  1.00th=[  120],  5.00th=[  134], 10.00th=[  146], 20.00th=[  171],
     | 30.00th=[  197], 40.00th=[  220], 50.00th=[  245], 60.00th=[  268],
     | 70.00th=[  288], 80.00th=[  300], 90.00th=[  326], 95.00th=[  338],
     | 99.00th=[  368], 99.50th=[  376], 99.90th=[  397], 99.95th=[  401],
     | 99.99th=[  405]
   bw (  KiB/s): min=  152, max=  512, per=24.76%, avg=263.96, stdev=74.45, samples=76
   iops        : min=   38, max=  128, avg=65.88, stdev=18.63, samples=76
  lat (msec)   : 100=0.04%, 250=51.91%, 500=48.05%
  cpu          : usr=0.14%, sys=0.57%, ctx=1859, majf=0, minf=28
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=8248: Sun Apr 13 07:24:15 2025
  write: IOPS=66, BW=267KiB/s (273kB/s)(10.0MiB/38404msec)
    slat (usec): min=13, max=438, avg=75.07, stdev=45.93
    clat (msec): min=72, max=422, avg=239.51, stdev=65.87
     lat (msec): min=72, max=423, avg=239.59, stdev=65.88
    clat percentiles (msec):
     |  1.00th=[  116],  5.00th=[  136], 10.00th=[  148], 20.00th=[  174],
     | 30.00th=[  197], 40.00th=[  222], 50.00th=[  243], 60.00th=[  266],
     | 70.00th=[  288], 80.00th=[  300], 90.00th=[  326], 95.00th=[  338],
     | 99.00th=[  363], 99.50th=[  384], 99.90th=[  405], 99.95th=[  422],
     | 99.99th=[  422]
   bw (  KiB/s): min=  160, max=  512, per=24.72%, avg=263.53, stdev=73.42, samples=76
   iops        : min=   40, max=  128, avg=65.84, stdev=18.38, samples=76
  lat (msec)   : 100=0.35%, 250=52.03%, 500=47.62%
  cpu          : usr=0.12%, sys=0.61%, ctx=2058, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=1067KiB/s (1092kB/s), 267KiB/s-267KiB/s (273kB/s-273kB/s), io=40.0MiB (41.9MB), run=38350-38404msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=8288: Sun Apr 13 07:24:34 2025
   read: IOPS=66, BW=265KiB/s (271kB/s)(5048KiB/19063msec)
    slat (usec): min=690, max=3976, avg=1410.14, stdev=480.45
    clat (msec): min=2, max=163, avg=77.77, stdev=23.09
     lat (msec): min=3, max=164, avg=79.18, stdev=23.39
    clat percentiles (msec):
     |  1.00th=[   31],  5.00th=[   50], 10.00th=[   55], 20.00th=[   61],
     | 30.00th=[   64], 40.00th=[   68], 50.00th=[   73], 60.00th=[   80],
     | 70.00th=[   87], 80.00th=[   96], 90.00th=[  111], 95.00th=[  125],
     | 99.00th=[  142], 99.50th=[  146], 99.90th=[  163], 99.95th=[  163],
     | 99.99th=[  163]
   bw (  KiB/s): min=  144, max=  432, per=25.19%, avg=265.00, stdev=75.41, samples=38
   iops        : min=   36, max=  108, avg=66.21, stdev=18.88, samples=38
  write: IOPS=68, BW=272KiB/s (279kB/s)(5192KiB/19063msec)
    slat (usec): min=13, max=390, avg=67.08, stdev=40.25
    clat (msec): min=63, max=349, avg=157.46, stdev=50.28
     lat (msec): min=63, max=349, avg=157.53, stdev=50.29
    clat percentiles (msec):
     |  1.00th=[   77],  5.00th=[   94], 10.00th=[  105], 20.00th=[  120],
     | 30.00th=[  128], 40.00th=[  138], 50.00th=[  146], 60.00th=[  157],
     | 70.00th=[  171], 80.00th=[  192], 90.00th=[  228], 95.00th=[  266],
     | 99.00th=[  317], 99.50th=[  334], 99.90th=[  347], 99.95th=[  351],
     | 99.99th=[  351]
   bw (  KiB/s): min=  136, max=  352, per=24.72%, avg=270.68, stdev=63.74, samples=38
   iops        : min=   34, max=   88, avg=67.63, stdev=15.97, samples=38
  lat (msec)   : 4=0.04%, 10=0.16%, 20=0.16%, 50=2.34%, 100=42.23%
  lat (msec)   : 250=51.72%, 500=3.36%
  cpu          : usr=0.25%, sys=1.47%, ctx=3531, majf=0, minf=33
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=8289: Sun Apr 13 07:24:34 2025
   read: IOPS=65, BW=261KiB/s (267kB/s)(4976KiB/19063msec)
    slat (usec): min=33, max=4158, avg=1415.64, stdev=496.55
    clat (usec): min=1058, max=164259, avg=79016.79, stdev=24241.20
     lat (msec): min=2, max=166, avg=80.43, stdev=24.57
    clat percentiles (msec):
     |  1.00th=[   41],  5.00th=[   50], 10.00th=[   55], 20.00th=[   61],
     | 30.00th=[   65], 40.00th=[   68], 50.00th=[   73], 60.00th=[   80],
     | 70.00th=[   89], 80.00th=[   99], 90.00th=[  114], 95.00th=[  127],
     | 99.00th=[  148], 99.50th=[  157], 99.90th=[  165], 99.95th=[  165],
     | 99.99th=[  165]
   bw (  KiB/s): min=  104, max=  440, per=24.81%, avg=261.00, stdev=77.25, samples=38
   iops        : min=   26, max=  110, avg=65.21, stdev=19.32, samples=38
  write: IOPS=69, BW=276KiB/s (283kB/s)(5264KiB/19063msec)
    slat (usec): min=12, max=315, avg=64.29, stdev=36.44
    clat (msec): min=29, max=348, avg=155.50, stdev=49.96
     lat (msec): min=29, max=348, avg=155.57, stdev=49.98
    clat percentiles (msec):
     |  1.00th=[   71],  5.00th=[   92], 10.00th=[  105], 20.00th=[  116],
     | 30.00th=[  126], 40.00th=[  136], 50.00th=[  146], 60.00th=[  157],
     | 70.00th=[  174], 80.00th=[  192], 90.00th=[  226], 95.00th=[  257],
     | 99.00th=[  313], 99.50th=[  326], 99.90th=[  347], 99.95th=[  351],
     | 99.99th=[  351]
   bw (  KiB/s): min=  136, max=  400, per=25.09%, avg=274.68, stdev=72.57, samples=38
   iops        : min=   34, max=  100, avg=68.63, stdev=18.20, samples=38
  lat (msec)   : 2=0.04%, 4=0.08%, 10=0.12%, 20=0.08%, 50=2.42%
  lat (msec)   : 100=40.82%, 250=53.40%, 500=3.05%
  cpu          : usr=0.24%, sys=1.49%, ctx=3383, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=8290: Sun Apr 13 07:24:34 2025
   read: IOPS=65, BW=261KiB/s (268kB/s)(4984KiB/19063msec)
    slat (usec): min=658, max=4164, avg=1406.25, stdev=481.81
    clat (msec): min=2, max=175, avg=77.07, stdev=22.81
     lat (msec): min=3, max=177, avg=78.48, stdev=23.10
    clat percentiles (msec):
     |  1.00th=[   37],  5.00th=[   50], 10.00th=[   55], 20.00th=[   59],
     | 30.00th=[   64], 40.00th=[   68], 50.00th=[   72], 60.00th=[   79],
     | 70.00th=[   87], 80.00th=[   95], 90.00th=[  109], 95.00th=[  117],
     | 99.00th=[  144], 99.50th=[  157], 99.90th=[  174], 99.95th=[  176],
     | 99.99th=[  176]
   bw (  KiB/s): min=  120, max=  392, per=24.89%, avg=261.84, stdev=77.54, samples=38
   iops        : min=   30, max=   98, avg=65.42, stdev=19.42, samples=38
  write: IOPS=68, BW=276KiB/s (282kB/s)(5256KiB/19063msec)
    slat (usec): min=15, max=1164, avg=67.37, stdev=49.24
    clat (msec): min=58, max=351, avg=157.27, stdev=50.27
     lat (msec): min=58, max=352, avg=157.34, stdev=50.28
    clat percentiles (msec):
     |  1.00th=[   77],  5.00th=[   95], 10.00th=[  106], 20.00th=[  117],
     | 30.00th=[  127], 40.00th=[  138], 50.00th=[  146], 60.00th=[  157],
     | 70.00th=[  174], 80.00th=[  194], 90.00th=[  228], 95.00th=[  259],
     | 99.00th=[  313], 99.50th=[  330], 99.90th=[  351], 99.95th=[  351],
     | 99.99th=[  351]
   bw (  KiB/s): min=  152, max=  376, per=25.01%, avg=273.84, stdev=69.39, samples=38
   iops        : min=   38, max=   94, avg=68.42, stdev=17.37, samples=38
  lat (msec)   : 4=0.04%, 10=0.04%, 20=0.31%, 50=2.19%, 100=41.29%
  lat (msec)   : 250=52.81%, 500=3.32%
  cpu          : usr=0.19%, sys=1.55%, ctx=3443, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=8291: Sun Apr 13 07:24:34 2025
   read: IOPS=66, BW=266KiB/s (272kB/s)(5060KiB/19046msec)
    slat (usec): min=605, max=4616, avg=1417.97, stdev=479.16
    clat (usec): min=1590, max=167248, avg=77516.35, stdev=23956.10
     lat (msec): min=3, max=169, avg=78.94, stdev=24.24
    clat percentiles (msec):
     |  1.00th=[   14],  5.00th=[   50], 10.00th=[   55], 20.00th=[   60],
     | 30.00th=[   64], 40.00th=[   68], 50.00th=[   73], 60.00th=[   78],
     | 70.00th=[   86], 80.00th=[   95], 90.00th=[  111], 95.00th=[  126],
     | 99.00th=[  148], 99.50th=[  157], 99.90th=[  165], 99.95th=[  167],
     | 99.99th=[  167]
   bw (  KiB/s): min=  112, max=  384, per=25.23%, avg=265.42, stdev=73.74, samples=38
   iops        : min=   28, max=   96, avg=66.32, stdev=18.47, samples=38
  write: IOPS=67, BW=272KiB/s (279kB/s)(5180KiB/19046msec)
    slat (usec): min=13, max=363, avg=66.38, stdev=37.11
    clat (msec): min=67, max=342, avg=157.64, stdev=50.57
     lat (msec): min=68, max=342, avg=157.71, stdev=50.58
    clat percentiles (msec):
     |  1.00th=[   77],  5.00th=[   94], 10.00th=[  105], 20.00th=[  118],
     | 30.00th=[  127], 40.00th=[  136], 50.00th=[  146], 60.00th=[  157],
     | 70.00th=[  174], 80.00th=[  197], 90.00th=[  230], 95.00th=[  262],
     | 99.00th=[  317], 99.50th=[  334], 99.90th=[  342], 99.95th=[  342],
     | 99.99th=[  342]
   bw (  KiB/s): min=  144, max=  368, per=24.68%, avg=270.26, stdev=67.27, samples=38
   iops        : min=   36, max=   92, avg=67.53, stdev=16.86, samples=38
  lat (msec)   : 2=0.04%, 4=0.04%, 10=0.16%, 20=0.43%, 50=1.99%
  lat (msec)   : 100=42.58%, 250=51.29%, 500=3.48%
  cpu          : usr=0.22%, sys=1.51%, ctx=3520, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=1053KiB/s (1078kB/s), 261KiB/s-266KiB/s (267kB/s-272kB/s), io=19.6MiB (20.5MB), run=19046-19063msec
  WRITE: bw=1096KiB/s (1122kB/s), 272KiB/s-276KiB/s (279kB/s-283kB/s), io=20.4MiB (21.4MB), run=19046-19063msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.7
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=8310: Sun Apr 13 07:24:36 2025
   read: IOPS=8, BW=351MiB/s (368MB/s)(40.0MiB/114msec)
    slat (nsec): min=5516.5k, max=5516.5k, avg=5516517.00, stdev= 0.00
    clat (nsec): min=101832k, max=101832k, avg=101832015.00, stdev= 0.00
     lat (nsec): min=107357k, max=107357k, avg=107357169.00, stdev= 0.00
    clat percentiles (msec):
     |  1.00th=[  103],  5.00th=[  103], 10.00th=[  103], 20.00th=[  103],
     | 30.00th=[  103], 40.00th=[  103], 50.00th=[  103], 60.00th=[  103],
     | 70.00th=[  103], 80.00th=[  103], 90.00th=[  103], 95.00th=[  103],
     | 99.00th=[  103], 99.50th=[  103], 99.90th=[  103], 99.95th=[  103],
     | 99.99th=[  103]
  lat (msec)   : 250=100.00%
  cpu          : usr=0.00%, sys=5.31%, ctx=5, majf=0, minf=29
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=351MiB/s (368MB/s), 351MiB/s-351MiB/s (368MB/s-368MB/s), io=40.0MiB (41.9MB), run=114-114msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout