== sanity test 398c: run fio to test AIO ================= 15:37:13 (1734122233)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB) copied, 0.092897 s, 452 MB/s
osc.lustre-OST0000-osc-ffff880132ef4800.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=32325: Fri Dec 13 15:37:48 2024
  write: IOPS=76, BW=306KiB/s (314kB/s)(10.0MiB/33447msec)
    slat (usec): min=12, max=698, avg=84.54, stdev=54.59
    clat (msec): min=62, max=365, avg=208.67, stdev=48.03
     lat (msec): min=62, max=365, avg=208.75, stdev=48.04
    clat percentiles (msec):
     |  1.00th=[  108],  5.00th=[  134], 10.00th=[  148], 20.00th=[  171],
     | 30.00th=[  184], 40.00th=[  194], 50.00th=[  205], 60.00th=[  215],
     | 70.00th=[  232], 80.00th=[  249], 90.00th=[  275], 95.00th=[  292],
     | 99.00th=[  330], 99.50th=[  334], 99.90th=[  347], 99.95th=[  355],
     | 99.99th=[  368]
   bw (  KiB/s): min=  215, max=  448, per=24.86%, avg=304.24, stdev=55.16, samples=66
   iops        : min=   53, max=  112, avg=75.97, stdev=13.80, samples=66
  lat (msec)   : 100=0.78%, 250=79.80%, 500=19.41%
  cpu          : usr=0.14%, sys=0.74%, ctx=2399, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=32326: Fri Dec 13 15:37:48 2024
  write: IOPS=76, BW=306KiB/s (314kB/s)(10.0MiB/33446msec)
    slat (usec): min=13, max=442, avg=86.43, stdev=52.36
    clat (msec): min=73, max=359, avg=208.73, stdev=47.85
     lat (msec): min=73, max=360, avg=208.82, stdev=47.87
    clat percentiles (msec):
     |  1.00th=[  108],  5.00th=[  134], 10.00th=[  150], 20.00th=[  169],
     | 30.00th=[  182], 40.00th=[  194], 50.00th=[  205], 60.00th=[  218],
     | 70.00th=[  234], 80.00th=[  251], 90.00th=[  271], 95.00th=[  288],
     | 99.00th=[  330], 99.50th=[  334], 99.90th=[  355], 99.95th=[  359],
     | 99.99th=[  359]
   bw (  KiB/s): min=  215, max=  456, per=24.87%, avg=304.44, stdev=55.84, samples=66
   iops        : min=   53, max=  114, avg=76.08, stdev=13.97, samples=66
  lat (msec)   : 100=0.74%, 250=79.22%, 500=20.04%
  cpu          : usr=0.16%, sys=0.75%, ctx=2508, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=32327: Fri Dec 13 15:37:48 2024
  write: IOPS=76, BW=306KiB/s (313kB/s)(10.0MiB/33456msec)
    slat (usec): min=14, max=386, avg=81.03, stdev=49.42
    clat (msec): min=21, max=368, avg=208.61, stdev=47.77
     lat (msec): min=21, max=368, avg=208.70, stdev=47.79
    clat percentiles (msec):
     |  1.00th=[  109],  5.00th=[  134], 10.00th=[  150], 20.00th=[  171],
     | 30.00th=[  184], 40.00th=[  194], 50.00th=[  205], 60.00th=[  218],
     | 70.00th=[  232], 80.00th=[  249], 90.00th=[  271], 95.00th=[  292],
     | 99.00th=[  330], 99.50th=[  338], 99.90th=[  368], 99.95th=[  368],
     | 99.99th=[  368]
   bw (  KiB/s): min=  192, max=  464, per=24.82%, avg=303.83, stdev=55.27, samples=66
   iops        : min=   48, max=  116, avg=75.92, stdev=13.81, samples=66
  lat (msec)   : 50=0.08%, 100=0.47%, 250=80.27%, 500=19.18%
  cpu          : usr=0.15%, sys=0.72%, ctx=2246, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=32328: Fri Dec 13 15:37:48 2024
  write: IOPS=76, BW=306KiB/s (314kB/s)(10.0MiB/33437msec)
    slat (usec): min=13, max=842, avg=84.16, stdev=55.23
    clat (msec): min=36, max=357, avg=208.71, stdev=48.79
     lat (msec): min=37, max=357, avg=208.79, stdev=48.80
    clat percentiles (msec):
     |  1.00th=[  100],  5.00th=[  131], 10.00th=[  150], 20.00th=[  169],
     | 30.00th=[  184], 40.00th=[  194], 50.00th=[  207], 60.00th=[  220],
     | 70.00th=[  234], 80.00th=[  247], 90.00th=[  275], 95.00th=[  292],
     | 99.00th=[  330], 99.50th=[  334], 99.90th=[  355], 99.95th=[  359],
     | 99.99th=[  359]
   bw (  KiB/s): min=  208, max=  496, per=24.93%, avg=305.15, stdev=62.90, samples=66
   iops        : min=   52, max=  124, avg=76.24, stdev=15.73, samples=66
  lat (msec)   : 50=0.35%, 100=0.70%, 250=80.31%, 500=18.63%
  cpu          : usr=0.18%, sys=0.71%, ctx=2401, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=1224KiB/s (1254kB/s), 306KiB/s-306KiB/s (313kB/s-314kB/s), io=40.0MiB (41.9MB), run=33437-33456msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=32364: Fri Dec 13 15:38:08 2024
   read: IOPS=67, BW=268KiB/s (274kB/s)(5048KiB/18834msec)
    slat (usec): min=644, max=4773, avg=1397.18, stdev=380.54
    clat (msec): min=16, max=131, avg=77.36, stdev=15.21
     lat (msec): min=18, max=133, avg=78.76, stdev=15.34
    clat percentiles (msec):
     |  1.00th=[   48],  5.00th=[   55], 10.00th=[   59], 20.00th=[   64],
     | 30.00th=[   69], 40.00th=[   73], 50.00th=[   78], 60.00th=[   81],
     | 70.00th=[   86], 80.00th=[   90], 90.00th=[   99], 95.00th=[  105],
     | 99.00th=[  117], 99.50th=[  120], 99.90th=[  126], 99.95th=[  132],
     | 99.99th=[  132]
   bw (  KiB/s): min=  168, max=  368, per=25.13%, avg=267.16, stdev=50.12, samples=37
   iops        : min=   42, max=   92, avg=66.73, stdev=12.50, samples=37
  write: IOPS=68, BW=276KiB/s (282kB/s)(5192KiB/18834msec)
    slat (usec): min=16, max=1814, avg=71.66, stdev=93.31
    clat (msec): min=64, max=273, avg=154.90, stdev=38.11
     lat (msec): min=64, max=273, avg=154.98, stdev=38.11
    clat percentiles (msec):
     |  1.00th=[   84],  5.00th=[   99], 10.00th=[  108], 20.00th=[  123],
     | 30.00th=[  133], 40.00th=[  142], 50.00th=[  153], 60.00th=[  163],
     | 70.00th=[  174], 80.00th=[  188], 90.00th=[  207], 95.00th=[  228],
     | 99.00th=[  255], 99.50th=[  262], 99.90th=[  271], 99.95th=[  275],
     | 99.99th=[  275]
   bw (  KiB/s): min=  160, max=  368, per=24.88%, avg=275.16, stdev=54.05, samples=37
   iops        : min=   40, max=   92, avg=68.73, stdev=13.54, samples=37
  lat (msec)   : 20=0.04%, 50=0.86%, 100=47.38%, 250=51.02%, 500=0.70%
  cpu          : usr=0.19%, sys=1.59%, ctx=3386, majf=0, minf=33
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=32365: Fri Dec 13 15:38:08 2024
   read: IOPS=66, BW=265KiB/s (271kB/s)(4976KiB/18798msec)
    slat (usec): min=62, max=4558, avg=1376.81, stdev=389.71
    clat (usec): min=1770, max=136664, avg=76127.52, stdev=16351.26
     lat (msec): min=2, max=138, avg=77.51, stdev=16.47
    clat percentiles (msec):
     |  1.00th=[   21],  5.00th=[   54], 10.00th=[   58], 20.00th=[   64],
     | 30.00th=[   68], 40.00th=[   72], 50.00th=[   77], 60.00th=[   80],
     | 70.00th=[   84], 80.00th=[   88], 90.00th=[   97], 95.00th=[  103],
     | 99.00th=[  118], 99.50th=[  121], 99.90th=[  136], 99.95th=[  138],
     | 99.99th=[  138]
   bw (  KiB/s): min=  160, max=  400, per=24.79%, avg=263.49, stdev=60.76, samples=37
   iops        : min=   40, max=  100, avg=65.81, stdev=15.17, samples=37
  write: IOPS=70, BW=280KiB/s (287kB/s)(5264KiB/18798msec)
    slat (usec): min=15, max=367, avg=65.37, stdev=37.52
    clat (msec): min=28, max=272, avg=155.05, stdev=39.70
     lat (msec): min=28, max=272, avg=155.12, stdev=39.71
    clat percentiles (msec):
     |  1.00th=[   75],  5.00th=[   92], 10.00th=[  106], 20.00th=[  124],
     | 30.00th=[  134], 40.00th=[  142], 50.00th=[  153], 60.00th=[  163],
     | 70.00th=[  176], 80.00th=[  188], 90.00th=[  207], 95.00th=[  226],
     | 99.00th=[  249], 99.50th=[  257], 99.90th=[  271], 99.95th=[  271],
     | 99.99th=[  271]
   bw (  KiB/s): min=  192, max=  384, per=25.15%, avg=278.19, stdev=37.31, samples=37
   iops        : min=   48, max=   96, avg=69.49, stdev= 9.35, samples=37
  lat (msec)   : 2=0.08%, 4=0.08%, 20=0.31%, 50=1.05%, 100=47.77%
  lat (msec)   : 250=50.23%, 500=0.47%
  cpu          : usr=0.23%, sys=1.53%, ctx=3254, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=32366: Fri Dec 13 15:38:08 2024
   read: IOPS=66, BW=264KiB/s (270kB/s)(4984KiB/18875msec)
    slat (usec): min=635, max=3871, avg=1403.39, stdev=370.50
    clat (usec): min=1246, max=132715, avg=76775.29, stdev=15404.19
     lat (msec): min=2, max=134, avg=78.18, stdev=15.52
    clat percentiles (msec):
     |  1.00th=[   47],  5.00th=[   55], 10.00th=[   59], 20.00th=[   64],
     | 30.00th=[   69], 40.00th=[   73], 50.00th=[   78], 60.00th=[   81],
     | 70.00th=[   85], 80.00th=[   89], 90.00th=[   96], 95.00th=[  103],
     | 99.00th=[  115], 99.50th=[  118], 99.90th=[  131], 99.95th=[  133],
     | 99.99th=[  133]
   bw (  KiB/s): min=  136, max=  384, per=24.81%, avg=263.70, stdev=62.82, samples=37
   iops        : min=   34, max=   96, avg=65.86, stdev=15.69, samples=37
  write: IOPS=69, BW=278KiB/s (285kB/s)(5256KiB/18875msec)
    slat (usec): min=15, max=536, avg=63.93, stdev=37.88
    clat (msec): min=30, max=284, avg=155.27, stdev=39.17
     lat (msec): min=30, max=284, avg=155.33, stdev=39.17
    clat percentiles (msec):
     |  1.00th=[   79],  5.00th=[   94], 10.00th=[  105], 20.00th=[  122],
     | 30.00th=[  136], 40.00th=[  144], 50.00th=[  155], 60.00th=[  163],
     | 70.00th=[  174], 80.00th=[  188], 90.00th=[  207], 95.00th=[  228],
     | 99.00th=[  259], 99.50th=[  262], 99.90th=[  271], 99.95th=[  284],
     | 99.99th=[  284]
   bw (  KiB/s): min=  208, max=  344, per=24.86%, avg=274.95, stdev=40.98, samples=37
   iops        : min=   52, max=   86, avg=68.68, stdev=10.28, samples=37
  lat (msec)   : 2=0.16%, 50=0.94%, 100=48.28%, 250=49.88%, 500=0.74%
  cpu          : usr=0.22%, sys=1.58%, ctx=3103, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=32367: Fri Dec 13 15:38:08 2024
   read: IOPS=67, BW=268KiB/s (275kB/s)(5060KiB/18855msec)
    slat (usec): min=712, max=4313, avg=1379.17, stdev=368.16
    clat (usec): min=1099, max=157668, avg=77466.80, stdev=16279.37
     lat (msec): min=2, max=159, avg=78.85, stdev=16.40
    clat percentiles (msec):
     |  1.00th=[   47],  5.00th=[   54], 10.00th=[   59], 20.00th=[   65],
     | 30.00th=[   69], 40.00th=[   73], 50.00th=[   77], 60.00th=[   81],
     | 70.00th=[   86], 80.00th=[   90], 90.00th=[   99], 95.00th=[  107],
     | 99.00th=[  118], 99.50th=[  123], 99.90th=[  157], 99.95th=[  159],
     | 99.99th=[  159]
   bw (  KiB/s): min=  184, max=  416, per=25.30%, avg=268.89, stdev=58.08, samples=37
   iops        : min=   46, max=  104, avg=67.16, stdev=14.53, samples=37
  write: IOPS=68, BW=275KiB/s (281kB/s)(5180KiB/18855msec)
    slat (usec): min=14, max=352, avg=65.18, stdev=36.68
    clat (msec): min=64, max=284, avg=155.36, stdev=39.39
     lat (msec): min=64, max=284, avg=155.43, stdev=39.39
    clat percentiles (msec):
     |  1.00th=[   83],  5.00th=[   94], 10.00th=[  105], 20.00th=[  122],
     | 30.00th=[  133], 40.00th=[  142], 50.00th=[  153], 60.00th=[  163],
     | 70.00th=[  176], 80.00th=[  188], 90.00th=[  207], 95.00th=[  228],
     | 99.00th=[  249], 99.50th=[  262], 99.90th=[  279], 99.95th=[  284],
     | 99.99th=[  284]
   bw (  KiB/s): min=  160, max=  368, per=24.72%, avg=273.43, stdev=45.36, samples=37
   iops        : min=   40, max=   92, avg=68.30, stdev=11.34, samples=37
  lat (msec)   : 2=0.08%, 20=0.20%, 50=0.90%, 100=48.09%, 250=50.27%
  lat (msec)   : 500=0.47%
  cpu          : usr=0.25%, sys=1.52%, ctx=3338, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=1063KiB/s (1089kB/s), 264KiB/s-268KiB/s (270kB/s-275kB/s), io=19.6MiB (20.5MB), run=18798-18875msec
  WRITE: bw=1107KiB/s (1133kB/s), 275KiB/s-280KiB/s (281kB/s-287kB/s), io=20.4MiB (21.4MB), run=18798-18875msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.7
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=32386: Fri Dec 13 15:38:09 2024
   read: IOPS=13, BW=526MiB/s (552MB/s)(40.0MiB/76msec)
    slat (nsec): min=6257.3k, max=6257.3k, avg=6257261.00, stdev= 0.00
    clat (nsec): min=65629k, max=65629k, avg=65628754.00, stdev= 0.00
     lat (nsec): min=71899k, max=71899k, avg=71899002.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[65799],  5.00th=[65799], 10.00th=[65799], 20.00th=[65799],
     | 30.00th=[65799], 40.00th=[65799], 50.00th=[65799], 60.00th=[65799],
     | 70.00th=[65799], 80.00th=[65799], 90.00th=[65799], 95.00th=[65799],
     | 99.00th=[65799], 99.50th=[65799], 99.90th=[65799], 99.95th=[65799],
     | 99.99th=[65799]
  lat (msec)   : 100=100.00%
  cpu          : usr=0.00%, sys=8.00%, ctx=14, majf=0, minf=29
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=526MiB/s (552MB/s), 526MiB/s-526MiB/s (552MB/s-552MB/s), io=40.0MiB (41.9MB), run=76-76msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout