== sanity test 398c: run fio to test AIO ================= 13:45:02 (1750182302)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB) copied, 0.219876 s, 191 MB/s
osc.lustre-OST0000-osc-ffff8800a7a15000.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=10491: Tue Jun 17 13:45:15 2025
  write: IOPS=224, BW=899KiB/s (921kB/s)(10.0MiB/11390msec)
    slat (usec): min=11, max=5272, avg=106.41, stdev=278.96
    clat (msec): min=20, max=121, avg=70.74, stdev=14.58
     lat (msec): min=20, max=121, avg=70.85, stdev=14.59
    clat percentiles (msec):
     |  1.00th=[   41],  5.00th=[   50], 10.00th=[   55], 20.00th=[   59],
     | 30.00th=[   63], 40.00th=[   67], 50.00th=[   70], 60.00th=[   73],
     | 70.00th=[   78], 80.00th=[   82], 90.00th=[   89], 95.00th=[   99],
     | 99.00th=[  113], 99.50th=[  115], 99.90th=[  123], 99.95th=[  123],
     | 99.99th=[  123]
   bw (  KiB/s): min=  760, max= 1157, per=25.08%, avg=901.45, stdev=121.83, samples=22
   iops        : min=  190, max=  289, avg=225.23, stdev=30.51, samples=22
  lat (msec)   : 50=5.55%, 100=90.00%, 250=4.45%
  cpu          : usr=0.65%, sys=3.12%, ctx=1152, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=10492: Tue Jun 17 13:45:15 2025
  write: IOPS=224, BW=900KiB/s (922kB/s)(10.0MiB/11379msec)
    slat (usec): min=11, max=3313, avg=87.36, stdev=208.61
    clat (msec): min=5, max=133, avg=70.63, stdev=15.13
     lat (msec): min=5, max=133, avg=70.72, stdev=15.13
    clat percentiles (msec):
     |  1.00th=[   41],  5.00th=[   48], 10.00th=[   54], 20.00th=[   59],
     | 30.00th=[   63], 40.00th=[   66], 50.00th=[   70], 60.00th=[   74],
     | 70.00th=[   78], 80.00th=[   81], 90.00th=[   90], 95.00th=[   99],
     | 99.00th=[  127], 99.50th=[  132], 99.90th=[  134], 99.95th=[  134],
     | 99.99th=[  134]
   bw (  KiB/s): min=  664, max= 1165, per=25.14%, avg=903.45, stdev=129.15, samples=22
   iops        : min=  166, max=  291, avg=225.77, stdev=32.27, samples=22
  lat (msec)   : 10=0.04%, 50=5.82%, 100=90.23%, 250=3.91%
  cpu          : usr=0.65%, sys=2.65%, ctx=920, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=10493: Tue Jun 17 13:45:15 2025
  write: IOPS=225, BW=903KiB/s (925kB/s)(10.0MiB/11338msec)
    slat (usec): min=11, max=3234, avg=105.95, stdev=246.20
    clat (msec): min=17, max=114, avg=70.52, stdev=14.84
     lat (msec): min=18, max=114, avg=70.63, stdev=14.85
    clat percentiles (msec):
     |  1.00th=[   42],  5.00th=[   49], 10.00th=[   54], 20.00th=[   59],
     | 30.00th=[   63], 40.00th=[   66], 50.00th=[   69], 60.00th=[   73],
     | 70.00th=[   78], 80.00th=[   82], 90.00th=[   92], 95.00th=[   99],
     | 99.00th=[  111], 99.50th=[  115], 99.90th=[  115], 99.95th=[  115],
     | 99.99th=[  115]
   bw (  KiB/s): min=  696, max= 1160, per=25.22%, avg=906.55, stdev=127.67, samples=22
   iops        : min=  174, max=  290, avg=226.59, stdev=31.94, samples=22
  lat (msec)   : 20=0.04%, 50=6.05%, 100=89.18%, 250=4.73%
  cpu          : usr=0.00%, sys=3.63%, ctx=1117, majf=0, minf=28
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=10494: Tue Jun 17 13:45:15 2025
  write: IOPS=224, BW=899KiB/s (920kB/s)(10.0MiB/11395msec)
    slat (usec): min=11, max=4018, avg=106.33, stdev=237.17
    clat (msec): min=8, max=128, avg=70.60, stdev=14.86
     lat (msec): min=8, max=129, avg=70.71, stdev=14.86
    clat percentiles (msec):
     |  1.00th=[   41],  5.00th=[   48], 10.00th=[   55], 20.00th=[   59],
     | 30.00th=[   63], 40.00th=[   66], 50.00th=[   70], 60.00th=[   73],
     | 70.00th=[   78], 80.00th=[   82], 90.00th=[   89], 95.00th=[   99],
     | 99.00th=[  115], 99.50th=[  122], 99.90th=[  127], 99.95th=[  129],
     | 99.99th=[  130]
   bw (  KiB/s): min=  664, max= 1157, per=25.04%, avg=899.91, stdev=133.36, samples=22
   iops        : min=  166, max=  289, avg=224.95, stdev=33.31, samples=22
  lat (msec)   : 10=0.04%, 20=0.08%, 50=5.98%, 100=89.18%, 250=4.73%
  cpu          : usr=1.25%, sys=2.83%, ctx=1356, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=3595KiB/s (3681kB/s), 899KiB/s-903KiB/s (920kB/s-925kB/s), io=40.0MiB (41.9MB), run=11338-11395msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=10514: Tue Jun 17 13:45:27 2025
   read: IOPS=131, BW=527KiB/s (539kB/s)(5048KiB/9582msec)
    slat (usec): min=25, max=11775, avg=3776.98, stdev=1575.33
    clat (msec): min=8, max=115, avg=56.35, stdev=13.34
     lat (msec): min=12, max=126, avg=60.13, stdev=13.77
    clat percentiles (msec):
     |  1.00th=[   31],  5.00th=[   36], 10.00th=[   41], 20.00th=[   46],
     | 30.00th=[   50], 40.00th=[   53], 50.00th=[   56], 60.00th=[   59],
     | 70.00th=[   63], 80.00th=[   67], 90.00th=[   73], 95.00th=[   81],
     | 99.00th=[   93], 99.50th=[  100], 99.90th=[  112], 99.95th=[  116],
     | 99.99th=[  116]
   bw (  KiB/s): min=  392, max=  648, per=25.07%, avg=524.89, stdev=81.03, samples=19
   iops        : min=   98, max=  162, avg=131.16, stdev=20.30, samples=19
  write: IOPS=135, BW=542KiB/s (555kB/s)(5192KiB/9582msec)
    slat (usec): min=12, max=4156, avg=96.22, stdev=213.28
    clat (msec): min=11, max=126, avg=58.54, stdev=12.80
     lat (msec): min=11, max=126, avg=58.64, stdev=12.81
    clat percentiles (msec):
     |  1.00th=[   35],  5.00th=[   42], 10.00th=[   45], 20.00th=[   49],
     | 30.00th=[   52], 40.00th=[   55], 50.00th=[   58], 60.00th=[   61],
     | 70.00th=[   64], 80.00th=[   68], 90.00th=[   75], 95.00th=[   81],
     | 99.00th=[  103], 99.50th=[  108], 99.90th=[  124], 99.95th=[  127],
     | 99.99th=[  127]
   bw (  KiB/s): min=  368, max=  672, per=24.83%, avg=541.26, stdev=81.93, samples=19
   iops        : min=   92, max=  168, avg=135.26, stdev=20.48, samples=19
  lat (msec)   : 10=0.04%, 20=0.12%, 50=29.53%, 100=69.53%, 250=0.78%
  cpu          : usr=1.41%, sys=5.66%, ctx=2295, majf=0, minf=33
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=10515: Tue Jun 17 13:45:27 2025
   read: IOPS=130, BW=520KiB/s (533kB/s)(4976KiB/9564msec)
    slat (usec): min=29, max=12499, avg=3794.03, stdev=1558.26
    clat (msec): min=8, max=116, avg=55.71, stdev=12.76
     lat (msec): min=18, max=125, avg=59.51, stdev=13.22
    clat percentiles (msec):
     |  1.00th=[   32],  5.00th=[   37], 10.00th=[   41], 20.00th=[   45],
     | 30.00th=[   49], 40.00th=[   53], 50.00th=[   55], 60.00th=[   58],
     | 70.00th=[   62], 80.00th=[   65], 90.00th=[   72], 95.00th=[   79],
     | 99.00th=[   90], 99.50th=[  104], 99.90th=[  112], 99.95th=[  116],
     | 99.99th=[  116]
   bw (  KiB/s): min=  351, max=  664, per=24.74%, avg=518.00, stdev=88.35, samples=19
   iops        : min=   87, max=  166, avg=129.37, stdev=22.17, samples=19
  write: IOPS=137, BW=550KiB/s (564kB/s)(5264KiB/9564msec)
    slat (usec): min=16, max=1980, avg=97.31, stdev=167.78
    clat (msec): min=9, max=125, avg=58.92, stdev=13.17
     lat (msec): min=9, max=125, avg=59.02, stdev=13.18
    clat percentiles (msec):
     |  1.00th=[   35],  5.00th=[   40], 10.00th=[   44], 20.00th=[   48],
     | 30.00th=[   52], 40.00th=[   55], 50.00th=[   58], 60.00th=[   61],
     | 70.00th=[   65], 80.00th=[   69], 90.00th=[   77], 95.00th=[   82],
     | 99.00th=[   99], 99.50th=[  112], 99.90th=[  125], 99.95th=[  126],
     | 99.99th=[  126]
   bw (  KiB/s): min=  400, max=  680, per=25.28%, avg=551.16, stdev=68.95, samples=19
   iops        : min=  100, max=  170, avg=137.68, stdev=17.20, samples=19
  lat (msec)   : 10=0.08%, 50=29.96%, 100=69.10%, 250=0.86%
  cpu          : usr=0.88%, sys=6.20%, ctx=2279, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=10516: Tue Jun 17 13:45:27 2025
   read: IOPS=130, BW=521KiB/s (534kB/s)(4984KiB/9562msec)
    slat (usec): min=22, max=13562, avg=3840.50, stdev=1632.14
    clat (msec): min=4, max=124, avg=56.05, stdev=13.00
     lat (msec): min=8, max=130, avg=59.89, stdev=13.60
    clat percentiles (msec):
     |  1.00th=[   32],  5.00th=[   38], 10.00th=[   41], 20.00th=[   45],
     | 30.00th=[   49], 40.00th=[   53], 50.00th=[   55], 60.00th=[   58],
     | 70.00th=[   62], 80.00th=[   66], 90.00th=[   72], 95.00th=[   79],
     | 99.00th=[   91], 99.50th=[  108], 99.90th=[  115], 99.95th=[  125],
     | 99.99th=[  125]
   bw (  KiB/s): min=  368, max=  656, per=24.91%, avg=521.63, stdev=76.52, samples=19
   iops        : min=   92, max=  164, avg=130.37, stdev=19.16, samples=19
  write: IOPS=137, BW=550KiB/s (563kB/s)(5256KiB/9562msec)
    slat (usec): min=14, max=3314, avg=86.99, stdev=160.56
    clat (msec): min=30, max=122, avg=58.53, stdev=13.16
     lat (msec): min=30, max=123, avg=58.62, stdev=13.16
    clat percentiles (msec):
     |  1.00th=[   36],  5.00th=[   40], 10.00th=[   43], 20.00th=[   48],
     | 30.00th=[   52], 40.00th=[   55], 50.00th=[   58], 60.00th=[   61],
     | 70.00th=[   65], 80.00th=[   69], 90.00th=[   74], 95.00th=[   81],
     | 99.00th=[  104], 99.50th=[  110], 99.90th=[  124], 99.95th=[  124],
     | 99.99th=[  124]
   bw (  KiB/s): min=  400, max=  728, per=25.13%, avg=547.74, stdev=95.65, samples=19
   iops        : min=  100, max=  182, avg=136.89, stdev=23.94, samples=19
  lat (msec)   : 10=0.04%, 50=30.74%, 100=68.20%, 250=1.02%
  cpu          : usr=1.60%, sys=5.42%, ctx=2266, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=10517: Tue Jun 17 13:45:27 2025
   read: IOPS=132, BW=528KiB/s (541kB/s)(5060KiB/9581msec)
    slat (usec): min=17, max=14240, avg=3766.40, stdev=1566.52
    clat (msec): min=10, max=117, avg=56.66, stdev=13.60
     lat (msec): min=13, max=127, avg=60.43, stdev=14.06
    clat percentiles (msec):
     |  1.00th=[   31],  5.00th=[   37], 10.00th=[   41], 20.00th=[   46],
     | 30.00th=[   50], 40.00th=[   53], 50.00th=[   56], 60.00th=[   59],
     | 70.00th=[   64], 80.00th=[   68], 90.00th=[   74], 95.00th=[   81],
     | 99.00th=[   91], 99.50th=[  108], 99.90th=[  114], 99.95th=[  117],
     | 99.99th=[  117]
   bw (  KiB/s): min=  376, max=  656, per=25.12%, avg=526.00, stdev=76.88, samples=19
   iops        : min=   94, max=  164, avg=131.42, stdev=19.22, samples=19
  write: IOPS=135, BW=541KiB/s (554kB/s)(5180KiB/9581msec)
    slat (usec): min=12, max=2387, avg=83.24, stdev=155.35
    clat (msec): min=22, max=123, avg=59.00, stdev=12.80
     lat (msec): min=23, max=123, avg=59.09, stdev=12.81
    clat percentiles (msec):
     |  1.00th=[   35],  5.00th=[   41], 10.00th=[   44], 20.00th=[   48],
     | 30.00th=[   53], 40.00th=[   56], 50.00th=[   58], 60.00th=[   61],
     | 70.00th=[   65], 80.00th=[   69], 90.00th=[   75], 95.00th=[   81],
     | 99.00th=[  101], 99.50th=[  108], 99.90th=[  123], 99.95th=[  124],
     | 99.99th=[  124]
   bw (  KiB/s): min=  352, max=  688, per=24.69%, avg=538.26, stdev=87.88, samples=19
   iops        : min=   88, max=  172, avg=134.47, stdev=22.00, samples=19
  lat (msec)   : 20=0.08%, 50=28.52%, 100=70.51%, 250=0.90%
  cpu          : usr=2.80%, sys=4.19%, ctx=2251, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=2094KiB/s (2145kB/s), 520KiB/s-528KiB/s (533kB/s-541kB/s), io=19.6MiB (20.5MB), run=9562-9582msec
  WRITE: bw=2180KiB/s (2233kB/s), 541KiB/s-550KiB/s (554kB/s-564kB/s), io=20.4MiB (21.4MB), run=9562-9582msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.7
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=10528: Tue Jun 17 13:45:28 2025
   read: IOPS=2, BW=110MiB/s (115MB/s)(40.0MiB/365msec)
    slat (nsec): min=16057k, max=16057k, avg=16057264.00, stdev= 0.00
    clat (nsec): min=332992k, max=332992k, avg=332992321.00, stdev= 0.00
     lat (nsec): min=349076k, max=349076k, avg=349076427.00, stdev= 0.00
    clat percentiles (msec):
     |  1.00th=[  334],  5.00th=[  334], 10.00th=[  334], 20.00th=[  334],
     | 30.00th=[  334], 40.00th=[  334], 50.00th=[  334], 60.00th=[  334],
     | 70.00th=[  334], 80.00th=[  334], 90.00th=[  334], 95.00th=[  334],
     | 99.00th=[  334], 99.50th=[  334], 99.90th=[  334], 99.95th=[  334],
     | 99.99th=[  334]
  lat (msec)   : 500=100.00%
  cpu          : usr=0.55%, sys=1.92%, ctx=10, majf=0, minf=29
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=110MiB/s (115MB/s), 110MiB/s-110MiB/s (115MB/s-115MB/s), io=40.0MiB (41.9MB), run=365-365msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout