== sanity test 398c: run fio to test AIO ================= 21:52:26 (1749606746)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB) copied, 0.103642 s, 405 MB/s
osc.lustre-OST0000-osc-ffff8800a6865000.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=19489: Tue Jun 10 21:52:53 2025
  write: IOPS=98, BW=395KiB/s (404kB/s)(10.0MiB/25943msec)
    slat (usec): min=13, max=425, avg=71.24, stdev=45.56
    clat (msec): min=67, max=274, avg=161.85, stdev=28.44
     lat (msec): min=67, max=274, avg=161.92, stdev=28.44
    clat percentiles (msec):
     |  1.00th=[  101],  5.00th=[  120], 10.00th=[  128], 20.00th=[  138],
     | 30.00th=[  146], 40.00th=[  153], 50.00th=[  161], 60.00th=[  167],
     | 70.00th=[  176], 80.00th=[  186], 90.00th=[  201], 95.00th=[  213],
     | 99.00th=[  236], 99.50th=[  243], 99.90th=[  262], 99.95th=[  271],
     | 99.99th=[  275]
   bw (  KiB/s): min=  296, max=  496, per=24.92%, avg=393.24, stdev=42.13, samples=51
   iops        : min=   74, max=  124, avg=98.29, stdev=10.53, samples=51
  lat (msec)   : 100=0.98%, 250=98.75%, 500=0.27%
  cpu          : usr=0.09%, sys=0.82%, ctx=2294, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=19490: Tue Jun 10 21:52:53 2025
  write: IOPS=98, BW=395KiB/s (405kB/s)(10.0MiB/25910msec)
    slat (usec): min=12, max=429, avg=73.29, stdev=43.97
    clat (msec): min=41, max=264, avg=161.45, stdev=29.07
     lat (msec): min=41, max=264, avg=161.52, stdev=29.07
    clat percentiles (msec):
     |  1.00th=[  104],  5.00th=[  117], 10.00th=[  126], 20.00th=[  138],
     | 30.00th=[  146], 40.00th=[  153], 50.00th=[  159], 60.00th=[  167],
     | 70.00th=[  176], 80.00th=[  186], 90.00th=[  201], 95.00th=[  213],
     | 99.00th=[  236], 99.50th=[  241], 99.90th=[  259], 99.95th=[  262],
     | 99.99th=[  266]
   bw (  KiB/s): min=  312, max=  512, per=25.02%, avg=394.75, stdev=40.51, samples=51
   iops        : min=   78, max=  128, avg=98.63, stdev=10.12, samples=51
  lat (msec)   : 50=0.04%, 100=0.66%, 250=99.10%, 500=0.20%
  cpu          : usr=0.20%, sys=0.80%, ctx=2373, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=19491: Tue Jun 10 21:52:53 2025
  write: IOPS=98, BW=395KiB/s (405kB/s)(10.0MiB/25903msec)
    slat (usec): min=13, max=348, avg=70.04, stdev=43.12
    clat (msec): min=41, max=288, avg=161.66, stdev=29.91
     lat (msec): min=41, max=288, avg=161.73, stdev=29.91
    clat percentiles (msec):
     |  1.00th=[   99],  5.00th=[  117], 10.00th=[  126], 20.00th=[  138],
     | 30.00th=[  146], 40.00th=[  153], 50.00th=[  161], 60.00th=[  167],
     | 70.00th=[  176], 80.00th=[  186], 90.00th=[  203], 95.00th=[  213],
     | 99.00th=[  234], 99.50th=[  243], 99.90th=[  259], 99.95th=[  262],
     | 99.99th=[  288]
   bw (  KiB/s): min=  304, max=  520, per=25.07%, avg=395.63, stdev=43.09, samples=51
   iops        : min=   76, max=  130, avg=98.80, stdev=10.80, samples=51
  lat (msec)   : 50=0.31%, 100=0.78%, 250=98.55%, 500=0.35%
  cpu          : usr=0.15%, sys=0.81%, ctx=2141, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=19492: Tue Jun 10 21:52:53 2025
  write: IOPS=98, BW=395KiB/s (404kB/s)(10.0MiB/25927msec)
    slat (usec): min=12, max=283, avg=70.08, stdev=42.20
    clat (msec): min=55, max=274, avg=161.77, stdev=28.98
     lat (msec): min=55, max=274, avg=161.84, stdev=28.97
    clat percentiles (msec):
     |  1.00th=[  102],  5.00th=[  117], 10.00th=[  126], 20.00th=[  138],
     | 30.00th=[  146], 40.00th=[  155], 50.00th=[  161], 60.00th=[  167],
     | 70.00th=[  176], 80.00th=[  186], 90.00th=[  201], 95.00th=[  213],
     | 99.00th=[  234], 99.50th=[  241], 99.90th=[  266], 99.95th=[  271],
     | 99.99th=[  275]
   bw (  KiB/s): min=  296, max=  496, per=24.98%, avg=394.14, stdev=39.64, samples=51
   iops        : min=   74, max=  124, avg=98.49, stdev= 9.93, samples=51
  lat (msec)   : 100=0.47%, 250=99.26%, 500=0.27%
  cpu          : usr=0.10%, sys=0.84%, ctx=2346, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=1579KiB/s (1617kB/s), 395KiB/s-395KiB/s (404kB/s-405kB/s), io=40.0MiB (41.9MB), run=25903-25943msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=19523: Tue Jun 10 21:53:11 2025
   read: IOPS=76, BW=305KiB/s (312kB/s)(5048KiB/16561msec)
    slat (usec): min=682, max=3601, avg=1219.12, stdev=299.89
    clat (usec): min=1348, max=151711, avg=67336.75, stdev=12461.85
     lat (msec): min=2, max=152, avg=68.56, stdev=12.45
    clat percentiles (msec):
     |  1.00th=[   40],  5.00th=[   50], 10.00th=[   54], 20.00th=[   58],
     | 30.00th=[   62], 40.00th=[   65], 50.00th=[   67], 60.00th=[   70],
     | 70.00th=[   73], 80.00th=[   78], 90.00th=[   83], 95.00th=[   87],
     | 99.00th=[   95], 99.50th=[   97], 99.90th=[  150], 99.95th=[  153],
     | 99.99th=[  153]
   bw (  KiB/s): min=  184, max=  384, per=25.12%, avg=303.70, stdev=48.87, samples=33
   iops        : min=   46, max=   96, avg=75.88, stdev=12.20, samples=33
  write: IOPS=78, BW=314KiB/s (321kB/s)(5192KiB/16561msec)
    slat (usec): min=14, max=362, avg=59.04, stdev=33.46
    clat (msec): min=59, max=231, avg=137.10, stdev=29.82
     lat (msec): min=59, max=231, avg=137.17, stdev=29.82
    clat percentiles (msec):
     |  1.00th=[   72],  5.00th=[   90], 10.00th=[  101], 20.00th=[  112],
     | 30.00th=[  120], 40.00th=[  129], 50.00th=[  136], 60.00th=[  144],
     | 70.00th=[  153], 80.00th=[  163], 90.00th=[  178], 95.00th=[  188],
     | 99.00th=[  207], 99.50th=[  211], 99.90th=[  218], 99.95th=[  232],
     | 99.99th=[  232]
   bw (  KiB/s): min=  224, max=  416, per=24.87%, avg=313.15, stdev=35.17, samples=33
   iops        : min=   56, max=  104, avg=78.24, stdev= 8.80, samples=33
  lat (msec)   : 2=0.08%, 20=0.20%, 50=2.54%, 100=51.17%, 250=46.02%
  cpu          : usr=0.28%, sys=1.55%, ctx=3191, majf=0, minf=33
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=19524: Tue Jun 10 21:53:11 2025
   read: IOPS=75, BW=300KiB/s (308kB/s)(4976KiB/16561msec)
    slat (usec): min=708, max=3013, avg=1208.79, stdev=276.26
    clat (msec): min=9, max=116, avg=66.45, stdev=11.91
     lat (msec): min=10, max=117, avg=67.67, stdev=11.92
    clat percentiles (msec):
     |  1.00th=[   40],  5.00th=[   48], 10.00th=[   53], 20.00th=[   57],
     | 30.00th=[   61], 40.00th=[   64], 50.00th=[   67], 60.00th=[   69],
     | 70.00th=[   73], 80.00th=[   77], 90.00th=[   82], 95.00th=[   87],
     | 99.00th=[   95], 99.50th=[   97], 99.90th=[  101], 99.95th=[  117],
     | 99.99th=[  117]
   bw (  KiB/s): min=  208, max=  400, per=24.66%, avg=298.12, stdev=52.56, samples=33
   iops        : min=   52, max=  100, avg=74.48, stdev=13.10, samples=33
  write: IOPS=79, BW=318KiB/s (325kB/s)(5264KiB/16561msec)
    slat (usec): min=13, max=693, avg=61.02, stdev=37.73
    clat (msec): min=55, max=232, avg=137.11, stdev=29.55
     lat (msec): min=55, max=232, avg=137.17, stdev=29.55
    clat percentiles (msec):
     |  1.00th=[   72],  5.00th=[   89], 10.00th=[  101], 20.00th=[  114],
     | 30.00th=[  122], 40.00th=[  129], 50.00th=[  136], 60.00th=[  144],
     | 70.00th=[  153], 80.00th=[  163], 90.00th=[  178], 95.00th=[  188],
     | 99.00th=[  205], 99.50th=[  215], 99.90th=[  222], 99.95th=[  234],
     | 99.99th=[  234]
   bw (  KiB/s): min=  255, max=  432, per=25.22%, avg=317.52, stdev=35.24, samples=33
   iops        : min=   63, max=  108, avg=79.33, stdev= 8.86, samples=33
  lat (msec)   : 10=0.04%, 20=0.12%, 50=3.09%, 100=50.47%, 250=46.29%
  cpu          : usr=0.24%, sys=1.56%, ctx=3303, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=19525: Tue Jun 10 21:53:11 2025
   read: IOPS=75, BW=300KiB/s (308kB/s)(4984KiB/16586msec)
    slat (usec): min=724, max=3025, avg=1199.14, stdev=274.31
    clat (usec): min=2237, max=96741, avg=66658.18, stdev=11732.78
     lat (usec): min=3635, max=97667, avg=67859.16, stdev=11737.36
    clat percentiles (usec):
     |  1.00th=[37487],  5.00th=[49021], 10.00th=[52691], 20.00th=[57410],
     | 30.00th=[61080], 40.00th=[64226], 50.00th=[66847], 60.00th=[69731],
     | 70.00th=[72877], 80.00th=[76022], 90.00th=[81265], 95.00th=[85459],
     | 99.00th=[91751], 99.50th=[93848], 99.90th=[96994], 99.95th=[96994],
     | 99.99th=[96994]
   bw (  KiB/s): min=  200, max=  400, per=24.72%, avg=298.85, stdev=55.88, samples=33
   iops        : min=   50, max=  100, avg=74.67, stdev=13.94, samples=33
  write: IOPS=79, BW=317KiB/s (324kB/s)(5256KiB/16586msec)
    slat (usec): min=14, max=960, avg=60.84, stdev=40.55
    clat (msec): min=22, max=229, avg=137.25, stdev=30.49
     lat (msec): min=22, max=229, avg=137.31, stdev=30.49
    clat percentiles (msec):
     |  1.00th=[   61],  5.00th=[   90], 10.00th=[  101], 20.00th=[  113],
     | 30.00th=[  123], 40.00th=[  129], 50.00th=[  136], 60.00th=[  142],
     | 70.00th=[  153], 80.00th=[  165], 90.00th=[  178], 95.00th=[  188],
     | 99.00th=[  205], 99.50th=[  211], 99.90th=[  215], 99.95th=[  230],
     | 99.99th=[  230]
   bw (  KiB/s): min=  256, max=  424, per=24.89%, avg=313.39, stdev=32.28, samples=33
   iops        : min=   64, max=  106, avg=78.30, stdev= 8.08, samples=33
  lat (msec)   : 4=0.04%, 20=0.23%, 50=2.93%, 100=50.51%, 250=46.29%
  cpu          : usr=0.25%, sys=1.57%, ctx=3202, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=19526: Tue Jun 10 21:53:11 2025
   read: IOPS=76, BW=307KiB/s (314kB/s)(5060KiB/16498msec)
    slat (usec): min=15, max=3325, avg=1206.89, stdev=312.37
    clat (usec): min=1241, max=141513, avg=66826.78, stdev=13370.64
     lat (msec): min=2, max=142, avg=68.04, stdev=13.40
    clat percentiles (msec):
     |  1.00th=[   21],  5.00th=[   47], 10.00th=[   53], 20.00th=[   58],
     | 30.00th=[   62], 40.00th=[   64], 50.00th=[   68], 60.00th=[   70],
     | 70.00th=[   73], 80.00th=[   78], 90.00th=[   83], 95.00th=[   86],
     | 99.00th=[   95], 99.50th=[   97], 99.90th=[  142], 99.95th=[  142],
     | 99.99th=[  142]
   bw (  KiB/s): min=  168, max=  376, per=25.22%, avg=304.94, stdev=42.25, samples=32
   iops        : min=   42, max=   94, avg=76.19, stdev=10.58, samples=32
  write: IOPS=78, BW=314KiB/s (322kB/s)(5180KiB/16498msec)
    slat (usec): min=13, max=522, avg=62.05, stdev=38.64
    clat (msec): min=16, max=230, avg=137.17, stdev=29.12
     lat (msec): min=17, max=230, avg=137.23, stdev=29.12
    clat percentiles (msec):
     |  1.00th=[   72],  5.00th=[   93], 10.00th=[  102], 20.00th=[  113],
     | 30.00th=[  123], 40.00th=[  129], 50.00th=[  136], 60.00th=[  144],
     | 70.00th=[  153], 80.00th=[  161], 90.00th=[  176], 95.00th=[  186],
     | 99.00th=[  207], 99.50th=[  213], 99.90th=[  222], 99.95th=[  230],
     | 99.99th=[  230]
   bw (  KiB/s): min=  248, max=  384, per=24.68%, avg=310.69, stdev=34.10, samples=32
   iops        : min=   62, max=   96, avg=77.63, stdev= 8.50, samples=32
  lat (msec)   : 2=0.08%, 4=0.12%, 10=0.08%, 20=0.23%, 50=3.12%
  lat (msec)   : 100=50.31%, 250=46.05%
  cpu          : usr=0.25%, sys=1.55%, ctx=3348, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=1210KiB/s (1239kB/s), 300KiB/s-307KiB/s (308kB/s-314kB/s), io=19.6MiB (20.5MB), run=16498-16586msec
  WRITE: bw=1260KiB/s (1290kB/s), 314KiB/s-318KiB/s (321kB/s-325kB/s), io=20.4MiB (21.4MB), run=16498-16586msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.7
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=19543: Tue Jun 10 21:53:12 2025
   read: IOPS=9, BW=367MiB/s (385MB/s)(40.0MiB/109msec)
    slat (nsec): min=6716.2k, max=6716.2k, avg=6716161.00, stdev= 0.00
    clat (nsec): min=96664k, max=96664k, avg=96664388.00, stdev= 0.00
     lat (nsec): min=103394k, max=103394k, avg=103393548.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[96994],  5.00th=[96994], 10.00th=[96994], 20.00th=[96994],
     | 30.00th=[96994], 40.00th=[96994], 50.00th=[96994], 60.00th=[96994],
     | 70.00th=[96994], 80.00th=[96994], 90.00th=[96994], 95.00th=[96994],
     | 99.00th=[96994], 99.50th=[96994], 99.90th=[96994], 99.95th=[96994],
     | 99.99th=[96994]
  lat (msec)   : 100=100.00%
  cpu          : usr=0.00%, sys=6.48%, ctx=7, majf=0, minf=29
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=367MiB/s (385MB/s), 367MiB/s-367MiB/s (385MB/s-385MB/s), io=40.0MiB (41.9MB), run=109-109msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout