== sanity test 398c: run fio to test AIO ================= 03:49:58 (1771404598)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB, 40 MiB) copied, 0.143927 s, 291 MB/s
osc.lustre-OST0000-osc-ffff8a2e60280800.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=183105: Wed Feb 18 03:50:07 2026
  write: IOPS=429, BW=1718KiB/s (1759kB/s)(10.0MiB/5962msec); 0 zone resets
    slat (usec): min=38, max=3133, avg=180.01, stdev=116.72
    clat (usec): min=18497, max=58112, avg=36715.10, stdev=6239.62
     lat (usec): min=18934, max=58298, avg=36896.25, stdev=6251.85
    clat percentiles (usec):
     |  1.00th=[21627],  5.00th=[28443], 10.00th=[30278], 20.00th=[31851],
     | 30.00th=[33162], 40.00th=[34341], 50.00th=[35914], 60.00th=[36963],
     | 70.00th=[39060], 80.00th=[41681], 90.00th=[45351], 95.00th=[47973],
     | 99.00th=[54264], 99.50th=[55837], 99.90th=[57410], 99.95th=[57934],
     | 99.99th=[57934]
   bw (  KiB/s): min= 1520, max= 1939, per=24.83%, avg=1705.73, stdev=139.88, samples=11
   iops        : min=  380, max=  484, avg=426.36, stdev=34.85, samples=11
  lat (msec)   : 20=0.55%, 50=96.45%, 100=3.01%
  cpu          : usr=0.49%, sys=5.94%, ctx=2481, majf=0, minf=9
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=183106: Wed Feb 18 03:50:07 2026
  write: IOPS=429, BW=1718KiB/s (1759kB/s)(10.0MiB/5960msec); 0 zone resets
    slat (usec): min=46, max=1148, avg=175.51, stdev=89.63
    clat (usec): min=4736, max=59110, avg=36720.14, stdev=6479.54
     lat (usec): min=4891, max=59520, avg=36896.96, stdev=6481.59
    clat percentiles (usec):
     |  1.00th=[21890],  5.00th=[28443], 10.00th=[30278], 20.00th=[31851],
     | 30.00th=[32900], 40.00th=[34341], 50.00th=[35914], 60.00th=[36963],
     | 70.00th=[39060], 80.00th=[42206], 90.00th=[45351], 95.00th=[48497],
     | 99.00th=[55313], 99.50th=[57410], 99.90th=[58983], 99.95th=[58983],
     | 99.99th=[58983]
   bw (  KiB/s): min= 1512, max= 1952, per=24.90%, avg=1710.55, stdev=145.19, samples=11
   iops        : min=  378, max=  488, avg=427.64, stdev=36.30, samples=11
  lat (msec)   : 10=0.08%, 20=0.51%, 50=96.48%, 100=2.93%
  cpu          : usr=0.64%, sys=6.16%, ctx=2488, majf=0, minf=12
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=183107: Wed Feb 18 03:50:07 2026
  write: IOPS=430, BW=1720KiB/s (1762kB/s)(10.0MiB/5952msec); 0 zone resets
    slat (usec): min=49, max=26243, avg=191.89, stdev=523.49
    clat (usec): min=15102, max=62595, avg=36706.39, stdev=6538.39
     lat (usec): min=15399, max=62837, avg=36899.49, stdev=6532.41
    clat percentiles (usec):
     |  1.00th=[20841],  5.00th=[28181], 10.00th=[30016], 20.00th=[32113],
     | 30.00th=[32900], 40.00th=[34341], 50.00th=[35914], 60.00th=[37487],
     | 70.00th=[39060], 80.00th=[42206], 90.00th=[45351], 95.00th=[47973],
     | 99.00th=[55313], 99.50th=[57934], 99.90th=[59507], 99.95th=[59507],
     | 99.99th=[62653]
   bw (  KiB/s): min= 1496, max= 1971, per=24.97%, avg=1715.18, stdev=151.80, samples=11
   iops        : min=  374, max=  492, avg=428.73, stdev=37.82, samples=11
  lat (msec)   : 20=0.66%, 50=95.62%, 100=3.71%
  cpu          : usr=0.71%, sys=6.34%, ctx=2452, majf=0, minf=11
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=183108: Wed Feb 18 03:50:07 2026
  write: IOPS=432, BW=1732KiB/s (1773kB/s)(10.0MiB/5913msec); 0 zone resets
    slat (usec): min=48, max=1318, avg=190.44, stdev=104.43
    clat (usec): min=6150, max=66706, avg=36704.38, stdev=6509.59
     lat (usec): min=6256, max=66782, avg=36896.78, stdev=6525.20
    clat percentiles (usec):
     |  1.00th=[21365],  5.00th=[28181], 10.00th=[30278], 20.00th=[31851],
     | 30.00th=[33162], 40.00th=[34341], 50.00th=[35914], 60.00th=[37487],
     | 70.00th=[39060], 80.00th=[41681], 90.00th=[45351], 95.00th=[48497],
     | 99.00th=[54789], 99.50th=[56361], 99.90th=[65799], 99.95th=[66847],
     | 99.99th=[66847]
   bw (  KiB/s): min= 1512, max= 2066, per=25.09%, avg=1723.82, stdev=167.66, samples=11
   iops        : min=  378, max=  516, avg=430.91, stdev=41.81, samples=11
  lat (msec)   : 10=0.04%, 20=0.66%, 50=96.17%, 100=3.12%
  cpu          : usr=0.59%, sys=6.11%, ctx=2549, majf=0, minf=9
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=6870KiB/s (7035kB/s), 1718KiB/s-1732KiB/s (1759kB/s-1773kB/s), io=40.0MiB (41.9MB), run=5913-5962msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=183118: Wed Feb 18 03:50:17 2026
  read: IOPS=187, BW=751KiB/s (769kB/s)(5048KiB/6719msec)
    slat (usec): min=1128, max=9545, avg=2894.18, stdev=1006.08
    clat (usec): min=23100, max=80887, avg=40050.01, stdev=7549.91
     lat (usec): min=24707, max=86936, avg=42946.78, stdev=7792.40
    clat percentiles (usec):
     |  1.00th=[26870],  5.00th=[29754], 10.00th=[31327], 20.00th=[33817],
     | 30.00th=[35914], 40.00th=[37487], 50.00th=[39060], 60.00th=[41157],
     | 70.00th=[43254], 80.00th=[45351], 90.00th=[49021], 95.00th=[53216],
     | 99.00th=[66323], 99.50th=[69731], 99.90th=[79168], 99.95th=[81265],
     | 99.99th=[81265]
   bw (  KiB/s): min=  661, max=  848, per=25.23%, avg=753.46, stdev=58.97, samples=13
   iops        : min=  165, max=  212, avg=188.31, stdev=14.77, samples=13
  write: IOPS=193, BW=773KiB/s (791kB/s)(5192KiB/6719msec); 0 zone resets
    slat (usec): min=52, max=4934, avg=172.45, stdev=167.15
    clat (usec): min=10519, max=78539, avg=40147.21, stdev=6823.33
     lat (usec): min=13188, max=78707, avg=40320.88, stdev=6807.23
    clat percentiles (usec):
     |  1.00th=[28443],  5.00th=[31327], 10.00th=[32900], 20.00th=[34866],
     | 30.00th=[36439], 40.00th=[37487], 50.00th=[39060], 60.00th=[40633],
     | 70.00th=[42206], 80.00th=[44827], 90.00th=[49021], 95.00th=[53216],
     | 99.00th=[61604], 99.50th=[66323], 99.90th=[74974], 99.95th=[78119],
     | 99.99th=[78119]
   bw (  KiB/s): min=  600, max=  864, per=24.98%, avg=776.77, stdev=77.07, samples=13
   iops        : min=  150, max=  216, avg=194.15, stdev=19.24, samples=13
  lat (msec)   : 20=0.12%, 50=91.33%, 100=8.55%
  cpu          : usr=0.54%, sys=5.72%, ctx=4483, majf=0, minf=12
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=183119: Wed Feb 18 03:50:17 2026
  read: IOPS=185, BW=741KiB/s (759kB/s)(4976KiB/6714msec)
    slat (usec): min=1188, max=9886, avg=2882.50, stdev=1019.38
    clat (usec): min=23475, max=78614, avg=40155.62, stdev=7248.05
     lat (usec): min=25786, max=83247, avg=43040.43, stdev=7451.59
    clat percentiles (usec):
     |  1.00th=[27657],  5.00th=[30016], 10.00th=[31851], 20.00th=[34341],
     | 30.00th=[36439], 40.00th=[37487], 50.00th=[39584], 60.00th=[41157],
     | 70.00th=[42730], 80.00th=[45351], 90.00th=[49546], 95.00th=[53216],
     | 99.00th=[62653], 99.50th=[67634], 99.90th=[76022], 99.95th=[78119],
     | 99.99th=[78119]
   bw (  KiB/s): min=  615, max=  904, per=24.85%, avg=741.92, stdev=82.01, samples=13
   iops        : min=  153, max=  226, avg=185.38, stdev=20.59, samples=13
  write: IOPS=196, BW=784KiB/s (803kB/s)(5264KiB/6714msec); 0 zone resets
    slat (usec): min=45, max=3807, avg=174.12, stdev=188.22
    clat (usec): min=24953, max=75228, avg=40072.02, stdev=6812.24
     lat (usec): min=25106, max=75442, avg=40247.26, stdev=6808.83
    clat percentiles (usec):
     |  1.00th=[28181],  5.00th=[31065], 10.00th=[32637], 20.00th=[34866],
     | 30.00th=[35914], 40.00th=[37487], 50.00th=[39060], 60.00th=[40633],
     | 70.00th=[42206], 80.00th=[44827], 90.00th=[48497], 95.00th=[52167],
     | 99.00th=[62129], 99.50th=[68682], 99.90th=[74974], 99.95th=[74974],
     | 99.99th=[74974]
   bw (  KiB/s): min=  664, max=  896, per=25.39%, avg=789.31, stdev=78.57, samples=13
   iops        : min=  166, max=  224, avg=197.23, stdev=19.67, samples=13
  lat (msec)   : 50=91.76%, 100=8.24%
  cpu          : usr=0.49%, sys=5.69%, ctx=4390, majf=0, minf=12
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=183120: Wed Feb 18 03:50:17 2026
  read: IOPS=186, BW=746KiB/s (764kB/s)(4984KiB/6683msec)
    slat (usec): min=1062, max=9000, avg=2882.23, stdev=952.91
    clat (usec): min=8784, max=73954, avg=39535.84, stdev=7213.99
     lat (usec): min=10841, max=76534, avg=42420.82, stdev=7357.76
    clat percentiles (usec):
     |  1.00th=[26870],  5.00th=[29230], 10.00th=[31065], 20.00th=[33817],
     | 30.00th=[35390], 40.00th=[36963], 50.00th=[38536], 60.00th=[40633],
     | 70.00th=[42730], 80.00th=[44827], 90.00th=[49021], 95.00th=[52167],
     | 99.00th=[61080], 99.50th=[65274], 99.90th=[72877], 99.95th=[73925],
     | 99.99th=[73925]
   bw (  KiB/s): min=  624, max=  840, per=25.14%, avg=750.69, stdev=75.96, samples=13
   iops        : min=  156, max=  210, avg=187.62, stdev=18.98, samples=13
  write: IOPS=196, BW=786KiB/s (805kB/s)(5256KiB/6683msec); 0 zone resets
    slat (usec): min=42, max=1623, avg=170.68, stdev=104.76
    clat (usec): min=12473, max=78029, avg=40347.87, stdev=7373.19
     lat (usec): min=12607, max=78142, avg=40519.74, stdev=7374.50
    clat percentiles (usec):
     |  1.00th=[28443],  5.00th=[31065], 10.00th=[32900], 20.00th=[34341],
     | 30.00th=[35914], 40.00th=[37487], 50.00th=[39060], 60.00th=[40633],
     | 70.00th=[42730], 80.00th=[45351], 90.00th=[49546], 95.00th=[55313],
     | 99.00th=[63701], 99.50th=[68682], 99.90th=[78119], 99.95th=[78119],
     | 99.99th=[78119]
   bw (  KiB/s): min=  584, max=  896, per=25.33%, avg=787.46, stdev=88.80, samples=13
   iops        : min=  146, max=  224, avg=196.77, stdev=22.25, samples=13
  lat (msec)   : 10=0.04%, 20=0.04%, 50=91.33%, 100=8.59%
  cpu          : usr=0.61%, sys=5.66%, ctx=4476, majf=0, minf=13
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=183121: Wed Feb 18 03:50:17 2026
  read: IOPS=190, BW=763KiB/s (781kB/s)(5060KiB/6636msec)
    slat (usec): min=89, max=11867, avg=2863.36, stdev=1060.43
    clat (usec): min=7575, max=67464, avg=39700.38, stdev=7168.20
     lat (usec): min=9574, max=71487, avg=42566.05, stdev=7399.05
    clat percentiles (usec):
     |  1.00th=[26608],  5.00th=[29754], 10.00th=[31327], 20.00th=[33817],
     | 30.00th=[35914], 40.00th=[37487], 50.00th=[39060], 60.00th=[40633],
     | 70.00th=[42730], 80.00th=[44827], 90.00th=[49021], 95.00th=[52691],
     | 99.00th=[59507], 99.50th=[63177], 99.90th=[65274], 99.95th=[67634],
     | 99.99th=[67634]
   bw (  KiB/s): min=  680, max=  880, per=25.45%, avg=759.92, stdev=66.11, samples=13
   iops        : min=  170, max=  220, avg=189.92, stdev=16.49, samples=13
  write: IOPS=195, BW=781KiB/s (799kB/s)(5180KiB/6636msec); 0 zone resets
    slat (usec): min=49, max=775, avg=166.93, stdev=88.55
    clat (usec): min=10406, max=68785, avg=40073.49, stdev=6673.69
     lat (usec): min=10552, max=68898, avg=40241.89, stdev=6672.41
    clat percentiles (usec):
     |  1.00th=[28443],  5.00th=[31327], 10.00th=[32900], 20.00th=[34866],
     | 30.00th=[36439], 40.00th=[37487], 50.00th=[39060], 60.00th=[40633],
     | 70.00th=[42730], 80.00th=[45351], 90.00th=[49021], 95.00th=[52691],
     | 99.00th=[62653], 99.50th=[64226], 99.90th=[66847], 99.95th=[68682],
     | 99.99th=[68682]
   bw (  KiB/s): min=  616, max=  960, per=25.03%, avg=778.15, stdev=79.02, samples=13
   iops        : min=  154, max=  240, avg=194.46, stdev=19.74, samples=13
  lat (msec)   : 10=0.08%, 20=0.23%, 50=91.48%, 100=8.20%
  cpu          : usr=0.41%, sys=5.86%, ctx=4527, majf=0, minf=12
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=2987KiB/s (3058kB/s), 741KiB/s-763KiB/s (759kB/s-781kB/s), io=19.6MiB (20.5MB), run=6636-6719msec
  WRITE: bw=3109KiB/s (3184kB/s), 773KiB/s-786KiB/s (791kB/s-805kB/s), io=20.4MiB (21.4MB), run=6636-6719msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.19
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=183126: Wed Feb 18 03:50:21 2026
  read: IOPS=8, BW=345MiB/s (362MB/s)(40.0MiB/116msec)
    slat (nsec): min=11420k, max=11420k, avg=11419634.00, stdev= 0.00
    clat (nsec): min=98522k, max=98522k, avg=98521895.00, stdev= 0.00
     lat (nsec): min=109946k, max=109946k, avg=109946003.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[98042],  5.00th=[98042], 10.00th=[98042], 20.00th=[98042],
     | 30.00th=[98042], 40.00th=[98042], 50.00th=[98042], 60.00th=[98042],
     | 70.00th=[98042], 80.00th=[98042], 90.00th=[98042], 95.00th=[98042],
     | 99.00th=[98042], 99.50th=[98042], 99.90th=[98042], 99.95th=[98042],
     | 99.99th=[98042]
  lat (msec)   : 100=100.00%
  cpu          : usr=0.00%, sys=8.70%, ctx=11, majf=0, minf=8
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=345MiB/s (362MB/s), 345MiB/s-345MiB/s (362MB/s-362MB/s), io=40.0MiB (41.9MB), run=116-116msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout