== sanity test 398c: run fio to test AIO ================= 17:46:20 (1751319980)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB) copied, 0.092889 s, 452 MB/s
osc.lustre-OST0000-osc-ffff8800ad0d8800.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=4001: Mon Jun 30 17:46:44 2025
  write: IOPS=113, BW=454KiB/s (465kB/s)(10.0MiB/22570msec)
    slat (usec): min=11, max=237, avg=54.11, stdev=30.63
    clat (msec): min=59, max=236, avg=140.92, stdev=23.99
     lat (msec): min=59, max=236, avg=140.98, stdev=23.99
    clat percentiles (msec):
     |  1.00th=[   86],  5.00th=[  103], 10.00th=[  110], 20.00th=[  122],
     | 30.00th=[  130], 40.00th=[  136], 50.00th=[  142], 60.00th=[  146],
     | 70.00th=[  153], 80.00th=[  161], 90.00th=[  171], 95.00th=[  182],
     | 99.00th=[  199], 99.50th=[  207], 99.90th=[  230], 99.95th=[  230],
     | 99.99th=[  236]
   bw (  KiB/s): min=  320, max=  624, per=24.94%, avg=452.44, stdev=54.67, samples=45
   iops        : min=   80, max=  156, avg=113.11, stdev=13.67, samples=45
  lat (msec)   : 100=4.18%, 250=95.82%
  cpu          : usr=0.16%, sys=0.78%, ctx=1943, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=4002: Mon Jun 30 17:46:44 2025
  write: IOPS=113, BW=454KiB/s (464kB/s)(10.0MiB/22579msec)
    slat (usec): min=11, max=300, avg=54.84, stdev=32.79
    clat (msec): min=69, max=212, avg=140.99, stdev=23.67
     lat (msec): min=69, max=212, avg=141.04, stdev=23.67
    clat percentiles (msec):
     |  1.00th=[   89],  5.00th=[  104], 10.00th=[  111], 20.00th=[  121],
     | 30.00th=[  130], 40.00th=[  136], 50.00th=[  140], 60.00th=[  148],
     | 70.00th=[  153], 80.00th=[  161], 90.00th=[  171], 95.00th=[  182],
     | 99.00th=[  197], 99.50th=[  203], 99.90th=[  211], 99.95th=[  211],
     | 99.99th=[  213]
   bw (  KiB/s): min=  312, max=  592, per=24.94%, avg=452.44, stdev=56.51, samples=45
   iops        : min=   78, max=  148, avg=113.11, stdev=14.13, samples=45
  lat (msec)   : 100=3.87%, 250=96.13%
  cpu          : usr=0.10%, sys=0.79%, ctx=2082, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=4003: Mon Jun 30 17:46:44 2025
  write: IOPS=113, BW=454KiB/s (464kB/s)(10.0MiB/22578msec)
    slat (usec): min=11, max=221, avg=52.99, stdev=30.49
    clat (msec): min=55, max=220, avg=140.98, stdev=23.71
     lat (msec): min=55, max=221, avg=141.03, stdev=23.71
    clat percentiles (msec):
     |  1.00th=[   90],  5.00th=[  104], 10.00th=[  110], 20.00th=[  121],
     | 30.00th=[  130], 40.00th=[  136], 50.00th=[  140], 60.00th=[  146],
     | 70.00th=[  153], 80.00th=[  161], 90.00th=[  171], 95.00th=[  180],
     | 99.00th=[  201], 99.50th=[  207], 99.90th=[  211], 99.95th=[  211],
     | 99.99th=[  222]
   bw (  KiB/s): min=  344, max=  592, per=24.93%, avg=452.31, stdev=53.87, samples=45
   iops        : min=   86, max=  148, avg=113.00, stdev=13.52, samples=45
  lat (msec)   : 100=3.71%, 250=96.29%
  cpu          : usr=0.14%, sys=0.76%, ctx=1940, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=4004: Mon Jun 30 17:46:44 2025
  write: IOPS=113, BW=454KiB/s (465kB/s)(10.0MiB/22538msec)
    slat (usec): min=12, max=30209, avg=68.83, stdev=600.43
    clat (msec): min=47, max=234, avg=140.71, stdev=24.59
     lat (msec): min=48, max=234, avg=140.78, stdev=24.58
    clat percentiles (msec):
     |  1.00th=[   81],  5.00th=[  103], 10.00th=[  110], 20.00th=[  121],
     | 30.00th=[  129], 40.00th=[  136], 50.00th=[  142], 60.00th=[  146],
     | 70.00th=[  155], 80.00th=[  161], 90.00th=[  171], 95.00th=[  182],
     | 99.00th=[  197], 99.50th=[  205], 99.90th=[  220], 99.95th=[  226],
     | 99.99th=[  234]
   bw (  KiB/s): min=  352, max=  600, per=24.94%, avg=452.42, stdev=51.71, samples=45
   iops        : min=   88, max=  150, avg=113.09, stdev=12.94, samples=45
  lat (msec)   : 50=0.04%, 100=4.38%, 250=95.59%
  cpu          : usr=0.18%, sys=0.90%, ctx=2155, majf=0, minf=27
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=1814KiB/s (1858kB/s), 454KiB/s-454KiB/s (464kB/s-465kB/s), io=40.0MiB (41.9MB), run=22538-22579msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=4032: Mon Jun 30 17:46:58 2025
   read: IOPS=94, BW=380KiB/s (389kB/s)(5048KiB/13296msec)
    slat (usec): min=407, max=2624, avg=767.61, stdev=203.35
    clat (usec): min=1726, max=121376, avg=58413.29, stdev=10928.49
     lat (msec): min=2, max=122, avg=59.18, stdev=10.94
    clat percentiles (msec):
     |  1.00th=[   35],  5.00th=[   42], 10.00th=[   46], 20.00th=[   50],
     | 30.00th=[   53], 40.00th=[   57], 50.00th=[   59], 60.00th=[   62],
     | 70.00th=[   65], 80.00th=[   68], 90.00th=[   71], 95.00th=[   75],
     | 99.00th=[   86], 99.50th=[   87], 99.90th=[  105], 99.95th=[  122],
     | 99.99th=[  122]
   bw (  KiB/s): min=  272, max=  504, per=25.27%, avg=380.31, stdev=61.52, samples=26
   iops        : min=   68, max=  126, avg=95.08, stdev=15.38, samples=26
  write: IOPS=97, BW=390KiB/s (400kB/s)(5192KiB/13296msec)
    slat (usec): min=11, max=299, avg=48.01, stdev=25.69
    clat (msec): min=45, max=197, avg=106.11, stdev=24.30
     lat (msec): min=45, max=197, avg=106.16, stdev=24.30
    clat percentiles (msec):
     |  1.00th=[   55],  5.00th=[   70], 10.00th=[   77], 20.00th=[   86],
     | 30.00th=[   93], 40.00th=[  100], 50.00th=[  105], 60.00th=[  112],
     | 70.00th=[  118], 80.00th=[  126], 90.00th=[  138], 95.00th=[  148],
     | 99.00th=[  169], 99.50th=[  176], 99.90th=[  190], 99.95th=[  199],
     | 99.99th=[  199]
   bw (  KiB/s): min=  296, max=  480, per=24.86%, avg=389.54, stdev=40.97, samples=26
   iops        : min=   74, max=  120, avg=97.38, stdev=10.24, samples=26
  lat (msec)   : 2=0.04%, 10=0.16%, 20=0.08%, 50=10.82%, 100=59.38%
  lat (msec)   : 250=29.53%
  cpu          : usr=0.31%, sys=1.62%, ctx=2850, majf=0, minf=33
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=4033: Mon Jun 30 17:46:58 2025
   read: IOPS=93, BW=374KiB/s (383kB/s)(4976KiB/13304msec)
    slat (usec): min=17, max=2551, avg=751.81, stdev=169.50
    clat (usec): min=1116, max=103175, avg=58192.30, stdev=11010.12
     lat (usec): min=1133, max=103814, avg=58945.99, stdev=11021.47
    clat percentiles (msec):
     |  1.00th=[   32],  5.00th=[   43], 10.00th=[   46], 20.00th=[   51],
     | 30.00th=[   54], 40.00th=[   56], 50.00th=[   58], 60.00th=[   61],
     | 70.00th=[   64], 80.00th=[   67], 90.00th=[   71], 95.00th=[   75],
     | 99.00th=[   86], 99.50th=[   89], 99.90th=[   93], 99.95th=[  104],
     | 99.99th=[  104]
   bw (  KiB/s): min=  224, max=  528, per=24.89%, avg=374.65, stdev=65.81, samples=26
   iops        : min=   56, max=  132, avg=93.58, stdev=16.46, samples=26
  write: IOPS=98, BW=396KiB/s (405kB/s)(5264KiB/13304msec)
    slat (usec): min=11, max=197, avg=47.74, stdev=25.78
    clat (msec): min=11, max=205, avg=105.87, stdev=24.55
     lat (msec): min=11, max=205, avg=105.92, stdev=24.55
    clat percentiles (msec):
     |  1.00th=[   51],  5.00th=[   69], 10.00th=[   78], 20.00th=[   87],
     | 30.00th=[   92], 40.00th=[  100], 50.00th=[  105], 60.00th=[  111],
     | 70.00th=[  117], 80.00th=[  126], 90.00th=[  138], 95.00th=[  150],
     | 99.00th=[  169], 99.50th=[  176], 99.90th=[  205], 99.95th=[  207],
     | 99.99th=[  207]
   bw (  KiB/s): min=  312, max=  488, per=25.19%, avg=394.65, stdev=39.09, samples=26
   iops        : min=   78, max=  122, avg=98.58, stdev= 9.80, samples=26
  lat (msec)   : 2=0.08%, 10=0.12%, 20=0.23%, 50=9.65%, 100=60.00%
  lat (msec)   : 250=29.92%
  cpu          : usr=0.26%, sys=1.65%, ctx=2922, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=4034: Mon Jun 30 17:46:58 2025
   read: IOPS=93, BW=374KiB/s (383kB/s)(4984KiB/13328msec)
    slat (usec): min=439, max=2470, avg=766.38, stdev=174.83
    clat (usec): min=812, max=116291, avg=58149.42, stdev=11402.62
     lat (usec): min=1753, max=117027, avg=58917.69, stdev=11407.71
    clat percentiles (msec):
     |  1.00th=[   13],  5.00th=[   42], 10.00th=[   46], 20.00th=[   50],
     | 30.00th=[   54], 40.00th=[   56], 50.00th=[   58], 60.00th=[   61],
     | 70.00th=[   64], 80.00th=[   67], 90.00th=[   72], 95.00th=[   75],
     | 99.00th=[   87], 99.50th=[   92], 99.90th=[  116], 99.95th=[  116],
     | 99.99th=[  116]
   bw (  KiB/s): min=  272, max=  488, per=24.96%, avg=375.58, stdev=62.38, samples=26
   iops        : min=   68, max=  122, avg=93.81, stdev=15.61, samples=26
  write: IOPS=98, BW=394KiB/s (404kB/s)(5256KiB/13328msec)
    slat (usec): min=12, max=720, avg=48.91, stdev=32.51
    clat (msec): min=42, max=174, avg=106.21, stdev=23.90
     lat (msec): min=42, max=174, avg=106.26, stdev=23.90
    clat percentiles (msec):
     |  1.00th=[   59],  5.00th=[   69], 10.00th=[   77], 20.00th=[   87],
     | 30.00th=[   93], 40.00th=[  100], 50.00th=[  105], 60.00th=[  110],
     | 70.00th=[  118], 80.00th=[  127], 90.00th=[  140], 95.00th=[  150],
     | 99.00th=[  163], 99.50th=[  169], 99.90th=[  176], 99.95th=[  176],
     | 99.99th=[  176]
   bw (  KiB/s): min=  320, max=  480, per=25.03%, avg=392.19, stdev=42.18, samples=26
   iops        : min=   80, max=  120, avg=97.96, stdev=10.60, samples=26
  lat (usec)   : 1000=0.04%
  lat (msec)   : 2=0.04%, 10=0.04%, 20=0.39%, 50=10.23%, 100=58.95%
  lat (msec)   : 250=30.31%
  cpu          : usr=0.23%, sys=1.70%, ctx=2922, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=4035: Mon Jun 30 17:46:58 2025
   read: IOPS=95, BW=381KiB/s (390kB/s)(5060KiB/13271msec)
    slat (usec): min=469, max=3514, avg=761.72, stdev=182.18
    clat (usec): min=942, max=104512, avg=58436.51, stdev=11136.01
     lat (usec): min=1579, max=105407, avg=59199.97, stdev=11149.08
    clat percentiles (msec):
     |  1.00th=[   34],  5.00th=[   43], 10.00th=[   46], 20.00th=[   51],
     | 30.00th=[   53], 40.00th=[   56], 50.00th=[   58], 60.00th=[   61],
     | 70.00th=[   64], 80.00th=[   68], 90.00th=[   72], 95.00th=[   77],
     | 99.00th=[   87], 99.50th=[   91], 99.90th=[  104], 99.95th=[  105],
     | 99.99th=[  105]
   bw (  KiB/s): min=  264, max=  488, per=25.47%, avg=383.27, stdev=55.66, samples=26
   iops        : min=   66, max=  122, avg=95.73, stdev=13.96, samples=26
  write: IOPS=97, BW=390KiB/s (400kB/s)(5180KiB/13271msec)
    slat (usec): min=12, max=775, avg=49.02, stdev=32.63
    clat (msec): min=43, max=206, avg=105.90, stdev=24.59
     lat (msec): min=43, max=207, avg=105.95, stdev=24.59
    clat percentiles (msec):
     |  1.00th=[   59],  5.00th=[   71], 10.00th=[   78], 20.00th=[   86],
     | 30.00th=[   92], 40.00th=[   99], 50.00th=[  103], 60.00th=[  110],
     | 70.00th=[  116], 80.00th=[  127], 90.00th=[  140], 95.00th=[  150],
     | 99.00th=[  176], 99.50th=[  186], 99.90th=[  207], 99.95th=[  207],
     | 99.99th=[  207]
   bw (  KiB/s): min=  344, max=  432, per=24.83%, avg=389.12, stdev=28.49, samples=26
   iops        : min=   86, max=  108, avg=97.19, stdev= 7.17, samples=26
  lat (usec)   : 1000=0.04%
  lat (msec)   : 2=0.04%, 20=0.27%, 50=9.80%, 100=61.76%, 250=28.09%
  cpu          : usr=0.24%, sys=1.66%, ctx=2972, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=1506KiB/s (1542kB/s), 374KiB/s-381KiB/s (383kB/s-390kB/s), io=19.6MiB (20.5MB), run=13271-13328msec
  WRITE: bw=1568KiB/s (1605kB/s), 390KiB/s-396KiB/s (400kB/s-405kB/s), io=20.4MiB (21.4MB), run=13271-13328msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.7
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=4049: Mon Jun 30 17:47:00 2025
   read: IOPS=12, BW=500MiB/s (524MB/s)(40.0MiB/80msec)
    slat (nsec): min=5610.1k, max=5610.1k, avg=5610085.00, stdev= 0.00
    clat (nsec): min=70433k, max=70433k, avg=70433144.00, stdev= 0.00
     lat (nsec): min=76053k, max=76053k, avg=76052903.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[70779],  5.00th=[70779], 10.00th=[70779], 20.00th=[70779],
     | 30.00th=[70779], 40.00th=[70779], 50.00th=[70779], 60.00th=[70779],
     | 70.00th=[70779], 80.00th=[70779], 90.00th=[70779], 95.00th=[70779],
     | 99.00th=[70779], 99.50th=[70779], 99.90th=[70779], 99.95th=[70779],
     | 99.99th=[70779]
  lat (msec)   : 100=100.00%
  cpu          : usr=0.00%, sys=6.33%, ctx=7, majf=0, minf=29
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=500MiB/s (524MB/s), 500MiB/s-500MiB/s (524MB/s-524MB/s), io=40.0MiB (41.9MB), run=80-80msec
debug=super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck