== sanity test 398c: run fio to test AIO ================= 10:46:18 (1766418378)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB, 40 MiB) copied, 0.160614 s, 261 MB/s
osc.lustre-OST0000-osc-ffff8ec808173000.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=184494: Mon Dec 22 10:46:31 2025
  write: IOPS=293, BW=1173KiB/s (1202kB/s)(10.0MiB/8727msec); 0 zone resets
    slat (usec): min=21, max=804, avg=102.27, stdev=73.48
    clat (usec): min=33074, max=95254, avg=53974.53, stdev=9262.14
     lat (usec): min=33237, max=95427, avg=54078.01, stdev=9264.94
    clat percentiles (usec):
     |  1.00th=[35390],  5.00th=[41681], 10.00th=[43779], 20.00th=[46924],
     | 30.00th=[48497], 40.00th=[50070], 50.00th=[52167], 60.00th=[55837],
     | 70.00th=[58459], 80.00th=[61080], 90.00th=[65274], 95.00th=[68682],
     | 99.00th=[85459], 99.50th=[93848], 99.90th=[93848], 99.95th=[93848],
     | 99.99th=[94897]
   bw (  KiB/s): min= 1016, max= 1424, per=24.91%, avg=1168.88, stdev=106.86, samples=17
   iops        : min=  254, max=  356, avg=292.18, stdev=26.78, samples=17
  lat (msec)   : 50=38.67%, 100=61.33%
  cpu          : usr=0.38%, sys=2.66%, ctx=2258, majf=0, minf=9
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=184495: Mon Dec 22 10:46:31 2025
  write: IOPS=293, BW=1174KiB/s (1202kB/s)(10.0MiB/8722msec); 0 zone resets
    slat (usec): min=21, max=1087, avg=96.59, stdev=73.24
    clat (usec): min=30018, max=95743, avg=53983.60, stdev=9222.21
     lat (usec): min=30246, max=96001, avg=54081.39, stdev=9226.58
    clat percentiles (usec):
     |  1.00th=[36439],  5.00th=[41681], 10.00th=[43779], 20.00th=[46400],
     | 30.00th=[48497], 40.00th=[50594], 50.00th=[52167], 60.00th=[55837],
     | 70.00th=[58459], 80.00th=[61080], 90.00th=[65799], 95.00th=[69731],
     | 99.00th=[84411], 99.50th=[87557], 99.90th=[94897], 99.95th=[95945],
     | 99.99th=[95945]
   bw (  KiB/s): min= 1019, max= 1408, per=24.92%, avg=1169.71, stdev=107.16, samples=17
   iops        : min=  254, max=  352, avg=292.35, stdev=26.86, samples=17
  lat (msec)   : 50=38.12%, 100=61.88%
  cpu          : usr=0.47%, sys=2.34%, ctx=2497, majf=0, minf=9
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=184496: Mon Dec 22 10:46:31 2025
  write: IOPS=295, BW=1183KiB/s (1212kB/s)(10.0MiB/8655msec); 0 zone resets
    slat (usec): min=22, max=1239, avg=118.88, stdev=78.43
    clat (usec): min=9766, max=95657, avg=53887.35, stdev=9550.63
     lat (usec): min=9817, max=95807, avg=54007.50, stdev=9552.57
    clat percentiles (usec):
     |  1.00th=[33424],  5.00th=[41157], 10.00th=[43779], 20.00th=[46400],
     | 30.00th=[48497], 40.00th=[50070], 50.00th=[52167], 60.00th=[55313],
     | 70.00th=[58459], 80.00th=[61080], 90.00th=[65799], 95.00th=[69731],
     | 99.00th=[83362], 99.50th=[90702], 99.90th=[95945], 99.95th=[95945],
     | 99.99th=[95945]
   bw (  KiB/s): min= 1024, max= 1432, per=25.13%, avg=1179.53, stdev=115.51, samples=17
   iops        : min=  256, max=  358, avg=294.82, stdev=28.92, samples=17
  lat (msec)   : 10=0.04%, 20=0.16%, 50=38.28%, 100=61.52%
  cpu          : usr=0.39%, sys=3.11%, ctx=2748, majf=0, minf=10
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=184497: Mon Dec 22 10:46:31 2025
  write: IOPS=293, BW=1175KiB/s (1203kB/s)(10.0MiB/8717msec); 0 zone resets
    slat (usec): min=22, max=1505, avg=112.59, stdev=79.49
    clat (usec): min=15710, max=96251, avg=53953.40, stdev=9554.60
     lat (usec): min=15893, max=96307, avg=54067.21, stdev=9554.38
    clat percentiles (usec):
     |  1.00th=[34866],  5.00th=[41681], 10.00th=[43779], 20.00th=[46400],
     | 30.00th=[48497], 40.00th=[50070], 50.00th=[51643], 60.00th=[54789],
     | 70.00th=[57410], 80.00th=[61080], 90.00th=[66323], 95.00th=[70779],
     | 99.00th=[86508], 99.50th=[93848], 99.90th=[95945], 99.95th=[95945],
     | 99.99th=[95945]
   bw (  KiB/s): min= 1008, max= 1424, per=24.99%, avg=1173.00, stdev=120.09, samples=17
   iops        : min=  252, max=  356, avg=293.18, stdev=30.08, samples=17
  lat (msec)   : 20=0.04%, 50=37.30%, 100=62.66%
  cpu          : usr=0.52%, sys=2.97%, ctx=2811, majf=0, minf=9
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=4693KiB/s (4806kB/s), 1173KiB/s-1183KiB/s (1202kB/s-1212kB/s), io=40.0MiB (41.9MB), run=8655-8727msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=184509: Mon Dec 22 10:46:43 2025
  read: IOPS=142, BW=569KiB/s (582kB/s)(5048KiB/8877msec)
    slat (usec): min=23, max=46501, avg=3902.28, stdev=1751.54
    clat (msec): min=7, max=105, avg=52.49, stdev= 9.30
     lat (msec): min=7, max=111, avg=56.39, stdev= 9.56
    clat percentiles (msec):
     |  1.00th=[   31],  5.00th=[   41], 10.00th=[   44], 20.00th=[   46],
     | 30.00th=[   48], 40.00th=[   51], 50.00th=[   52], 60.00th=[   54],
     | 70.00th=[   56], 80.00th=[   60], 90.00th=[   64], 95.00th=[   67],
     | 99.00th=[   81], 99.50th=[   84], 99.90th=[  102], 99.95th=[  106],
     | 99.99th=[  106]
   bw (  KiB/s): min=  432, max=  632, per=25.19%, avg=567.59, stdev=51.92, samples=17
   iops        : min=  108, max=  158, avg=141.88, stdev=12.97, samples=17
  write: IOPS=146, BW=585KiB/s (599kB/s)(5192KiB/8877msec); 0 zone resets
    slat (usec): min=21, max=1879, avg=117.10, stdev=87.32
    clat (msec): min=23, max=100, avg=54.28, stdev= 8.03
     lat (msec): min=23, max=101, avg=54.40, stdev= 8.04
    clat percentiles (msec):
     |  1.00th=[   37],  5.00th=[   43], 10.00th=[   46], 20.00th=[   48],
     | 30.00th=[   51], 40.00th=[   53], 50.00th=[   54], 60.00th=[   56],
     | 70.00th=[   58], 80.00th=[   60], 90.00th=[   64], 95.00th=[   68],
     | 99.00th=[   81], 99.50th=[   86], 99.90th=[   97], 99.95th=[  102],
     | 99.99th=[  102]
   bw (  KiB/s): min=  496, max=  640, per=24.95%, avg=585.41, stdev=38.08, samples=17
   iops        : min=  124, max=  160, avg=146.35, stdev= 9.52, samples=17
  lat (msec)   : 10=0.12%, 20=0.27%, 50=34.84%, 100=64.65%, 250=0.12%
  cpu          : usr=0.50%, sys=4.16%, ctx=4747, majf=0, minf=14
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=184510: Mon Dec 22 10:46:43 2025
  read: IOPS=139, BW=559KiB/s (573kB/s)(4976KiB/8895msec)
    slat (usec): min=1561, max=12500, avg=3867.03, stdev=1160.00
    clat (msec): min=27, max=101, avg=52.75, stdev= 7.85
     lat (msec): min=33, max=105, avg=56.62, stdev= 7.98
    clat percentiles (msec):
     |  1.00th=[   36],  5.00th=[   41], 10.00th=[   44], 20.00th=[   47],
     | 30.00th=[   49], 40.00th=[   51], 50.00th=[   53], 60.00th=[   55],
     | 70.00th=[   57], 80.00th=[   59], 90.00th=[   63], 95.00th=[   67],
     | 99.00th=[   73], 99.50th=[   77], 99.90th=[   91], 99.95th=[  102],
     | 99.99th=[  102]
   bw (  KiB/s): min=  448, max=  632, per=24.75%, avg=557.53, stdev=49.31, samples=17
   iops        : min=  112, max=  158, avg=139.35, stdev=12.33, samples=17
  write: IOPS=147, BW=592KiB/s (606kB/s)(5264KiB/8895msec); 0 zone resets
    slat (usec): min=36, max=1079, avg=113.77, stdev=74.24
    clat (usec): min=21488, max=93531, avg=53968.49, stdev=6982.46
     lat (usec): min=22179, max=93612, avg=54083.39, stdev=6981.81
    clat percentiles (usec):
     |  1.00th=[40109],  5.00th=[43254], 10.00th=[45876], 20.00th=[48497],
     | 30.00th=[50594], 40.00th=[52167], 50.00th=[53216], 60.00th=[55313],
     | 70.00th=[57410], 80.00th=[59507], 90.00th=[62653], 95.00th=[65274],
     | 99.00th=[73925], 99.50th=[78119], 99.90th=[85459], 99.95th=[93848],
     | 99.99th=[93848]
   bw (  KiB/s): min=  496, max=  688, per=25.29%, avg=593.41, stdev=49.01, samples=17
   iops        : min=  124, max=  172, avg=148.24, stdev=12.35, samples=17
  lat (msec)   : 50=33.05%, 100=66.91%, 250=0.04%
  cpu          : usr=0.60%, sys=3.95%, ctx=4717, majf=0, minf=13
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=184511: Mon Dec 22 10:46:43 2025
  read: IOPS=140, BW=560KiB/s (574kB/s)(4984KiB/8895msec)
    slat (usec): min=1472, max=8415, avg=3872.74, stdev=1103.30
    clat (usec): min=9947, max=86210, avg=52822.86, stdev=8000.28
     lat (usec): min=15051, max=89145, avg=56698.87, stdev=8078.89
    clat percentiles (usec):
     |  1.00th=[35914],  5.00th=[40633], 10.00th=[42730], 20.00th=[46400],
     | 30.00th=[48497], 40.00th=[50594], 50.00th=[52691], 60.00th=[54264],
     | 70.00th=[56361], 80.00th=[58983], 90.00th=[62653], 95.00th=[65799],
     | 99.00th=[73925], 99.50th=[79168], 99.90th=[83362], 99.95th=[86508],
     | 99.99th=[86508]
   bw (  KiB/s): min=  464, max=  640, per=24.99%, avg=563.00, stdev=62.32, samples=17
   iops        : min=  116, max=  160, avg=140.71, stdev=15.64, samples=17
  write: IOPS=147, BW=591KiB/s (605kB/s)(5256KiB/8895msec); 0 zone resets
    slat (usec): min=35, max=1470, avg=119.55, stdev=93.48
    clat (usec): min=34205, max=90553, avg=53736.28, stdev=7425.09
     lat (usec): min=34402, max=90647, avg=53857.25, stdev=7423.38
    clat percentiles (usec):
     |  1.00th=[39584],  5.00th=[42730], 10.00th=[44827], 20.00th=[47449],
     | 30.00th=[49546], 40.00th=[51643], 50.00th=[53216], 60.00th=[55313],
     | 70.00th=[56886], 80.00th=[59507], 90.00th=[62653], 95.00th=[65799],
     | 99.00th=[76022], 99.50th=[79168], 99.90th=[89654], 99.95th=[90702],
     | 99.99th=[90702]
   bw (  KiB/s): min=  456, max=  720, per=25.07%, avg=588.12, stdev=65.26, samples=17
   iops        : min=  114, max=  180, avg=147.00, stdev=16.32, samples=17
  lat (msec)   : 10=0.04%, 20=0.04%, 50=34.65%, 100=65.27%
  cpu          : usr=0.54%, sys=4.01%, ctx=4754, majf=0, minf=14
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=184512: Mon Dec 22 10:46:43 2025
  read: IOPS=142, BW=568KiB/s (582kB/s)(5060KiB/8904msec)
    slat (usec): min=1444, max=19084, avg=3832.64, stdev=1227.27
    clat (usec): min=10453, max=79624, avg=52546.02, stdev=7719.51
     lat (usec): min=14185, max=82642, avg=56382.29, stdev=7886.80
    clat percentiles (usec):
     |  1.00th=[34341],  5.00th=[40633], 10.00th=[43254], 20.00th=[46400],
     | 30.00th=[48497], 40.00th=[50070], 50.00th=[52167], 60.00th=[54264],
     | 70.00th=[56361], 80.00th=[58983], 90.00th=[62129], 95.00th=[65274],
     | 99.00th=[71828], 99.50th=[73925], 99.90th=[77071], 99.95th=[79168],
     | 99.99th=[79168]
   bw (  KiB/s): min=  496, max=  648, per=25.10%, avg=565.41, stdev=45.19, samples=17
   iops        : min=  124, max=  162, avg=141.35, stdev=11.30, samples=17
  write: IOPS=145, BW=582KiB/s (596kB/s)(5180KiB/8904msec); 0 zone resets
    slat (usec): min=36, max=783, avg=114.56, stdev=73.45
    clat (usec): min=23391, max=88919, avg=54022.10, stdev=7254.10
     lat (usec): min=23447, max=88976, avg=54137.90, stdev=7252.07
    clat percentiles (usec):
     |  1.00th=[39060],  5.00th=[42730], 10.00th=[45351], 20.00th=[47973],
     | 30.00th=[50070], 40.00th=[52167], 50.00th=[53740], 60.00th=[55313],
     | 70.00th=[57410], 80.00th=[60031], 90.00th=[63177], 95.00th=[65799],
     | 99.00th=[73925], 99.50th=[77071], 99.90th=[82314], 99.95th=[88605],
     | 99.99th=[88605]
   bw (  KiB/s): min=  512, max=  720, per=24.65%, avg=578.29, stdev=58.60, samples=17
   iops        : min=  128, max=  180, avg=144.53, stdev=14.68, samples=17
  lat (msec)   : 20=0.04%, 50=34.41%, 100=65.55%
  cpu          : usr=0.53%, sys=4.09%, ctx=4785, majf=0, minf=13
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=2254KiB/s (2308kB/s), 559KiB/s-569KiB/s (573kB/s-582kB/s), io=19.6MiB (20.5MB), run=8877-8904msec
  WRITE: bw=2346KiB/s (2403kB/s), 582KiB/s-592KiB/s (596kB/s-606kB/s), io=20.4MiB (21.4MB), run=8877-8904msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.19
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=184516: Mon Dec 22 10:46:47 2025
  read: IOPS=6, BW=250MiB/s (262MB/s)(40.0MiB/160msec)
    slat (nsec): min=12086k, max=12086k, avg=12086110.00, stdev= 0.00
    clat (nsec): min=138279k, max=138279k, avg=138279429.00, stdev= 0.00
     lat (nsec): min=150369k, max=150369k, avg=150369461.00, stdev= 0.00
    clat percentiles (msec):
     |  1.00th=[  138],  5.00th=[  138], 10.00th=[  138], 20.00th=[  138],
     | 30.00th=[  138], 40.00th=[  138], 50.00th=[  138], 60.00th=[  138],
     | 70.00th=[  138], 80.00th=[  138], 90.00th=[  138], 95.00th=[  138],
     | 99.00th=[  138], 99.50th=[  138], 99.90th=[  138], 99.95th=[  138],
     | 99.99th=[  138]
  lat (msec)   : 250=100.00%
  cpu          : usr=0.00%, sys=7.55%, ctx=3, majf=0, minf=7
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=250MiB/s (262MB/s), 250MiB/s-250MiB/s (262MB/s-262MB/s), io=40.0MiB (41.9MB), run=160-160msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout