== sanity test 398c: run fio to test AIO ================= 00:33:28 (1782534808)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB, 40 MiB) copied, 0.149678 s, 280 MB/s
osc.lustre-OST0000-osc-ffff9444121b3000.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=183051: Sat Jun 27 00:33:36 2026
  write: IOPS=501, BW=2006KiB/s (2054kB/s)(10.0MiB/5105msec); 0 zone resets
    slat (usec): min=38, max=24240, avg=185.33, stdev=483.96
    clat (usec): min=14323, max=58522, avg=31526.95, stdev=4968.01
     lat (usec): min=14479, max=58646, avg=31713.43, stdev=4965.93
    clat percentiles (usec):
     |  1.00th=[18744],  5.00th=[24249], 10.00th=[26346], 20.00th=[27919],
     | 30.00th=[29230], 40.00th=[30278], 50.00th=[31327], 60.00th=[32375],
     | 70.00th=[33424], 80.00th=[34866], 90.00th=[36963], 95.00th=[39060],
     | 99.00th=[47973], 99.50th=[51643], 99.90th=[57934], 99.95th=[58459],
     | 99.99th=[58459]
   bw (  KiB/s): min= 1848, max= 2176, per=24.92%, avg=1995.56, stdev=131.95, samples=9
   iops        : min=  462, max=  544, avg=498.89, stdev=32.99, samples=9
  lat (msec)   : 20=1.56%, 50=97.93%, 100=0.51%
  cpu          : usr=0.69%, sys=7.05%, ctx=2609, majf=0, minf=8
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=183052: Sat Jun 27 00:33:36 2026
  write: IOPS=500, BW=2002KiB/s (2050kB/s)(10.0MiB/5115msec); 0 zone resets
    slat (usec): min=35, max=1311, avg=172.21, stdev=93.01
    clat (usec): min=10675, max=58508, avg=31500.59, stdev=4935.43
     lat (usec): min=10818, max=58776, avg=31674.00, stdev=4944.35
    clat percentiles (usec):
     |  1.00th=[18744],  5.00th=[23987], 10.00th=[26084], 20.00th=[27657],
     | 30.00th=[29230], 40.00th=[30278], 50.00th=[31327], 60.00th=[32375],
     | 70.00th=[33424], 80.00th=[34866], 90.00th=[36963], 95.00th=[39060],
     | 99.00th=[46924], 99.50th=[50070], 99.90th=[57934], 99.95th=[58459],
     | 99.99th=[58459]
   bw (  KiB/s): min= 1824, max= 2184, per=24.77%, avg=1983.22, stdev=130.89, samples=9
   iops        : min=  456, max=  546, avg=495.78, stdev=32.70, samples=9
  lat (msec)   : 20=1.02%, 50=98.48%, 100=0.51%
  cpu          : usr=0.66%, sys=6.30%, ctx=2671, majf=0, minf=10
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=183053: Sat Jun 27 00:33:36 2026
  write: IOPS=503, BW=2013KiB/s (2061kB/s)(10.0MiB/5087msec); 0 zone resets
    slat (usec): min=35, max=822, avg=168.90, stdev=89.38
    clat (usec): min=5131, max=58456, avg=31572.50, stdev=5065.70
     lat (usec): min=5267, max=58642, avg=31742.53, stdev=5079.33
    clat percentiles (usec):
     |  1.00th=[17171],  5.00th=[23987], 10.00th=[26346], 20.00th=[27919],
     | 30.00th=[29230], 40.00th=[30278], 50.00th=[31327], 60.00th=[32375],
     | 70.00th=[33817], 80.00th=[34866], 90.00th=[36963], 95.00th=[39060],
     | 99.00th=[47973], 99.50th=[50594], 99.90th=[58459], 99.95th=[58459],
     | 99.99th=[58459]
   bw (  KiB/s): min= 1832, max= 2259, per=25.03%, avg=2003.89, stdev=150.47, samples=9
   iops        : min=  458, max=  564, avg=500.89, stdev=37.46, samples=9
  lat (msec)   : 10=0.04%, 20=1.64%, 50=97.81%, 100=0.51%
  cpu          : usr=0.65%, sys=6.31%, ctx=2548, majf=0, minf=10
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=183054: Sat Jun 27 00:33:36 2026
  write: IOPS=500, BW=2004KiB/s (2052kB/s)(10.0MiB/5110msec); 0 zone resets
    slat (usec): min=42, max=1679, avg=170.48, stdev=93.77
    clat (usec): min=4407, max=58551, avg=31488.35, stdev=5126.33
     lat (usec): min=4493, max=58702, avg=31660.01, stdev=5134.04
    clat percentiles (usec):
     |  1.00th=[18744],  5.00th=[24249], 10.00th=[26084], 20.00th=[27919],
     | 30.00th=[29230], 40.00th=[30278], 50.00th=[31327], 60.00th=[32375],
     | 70.00th=[33424], 80.00th=[34866], 90.00th=[36963], 95.00th=[39060],
     | 99.00th=[46924], 99.50th=[52691], 99.90th=[58459], 99.95th=[58459],
     | 99.99th=[58459]
   bw (  KiB/s): min= 1824, max= 2176, per=24.84%, avg=1988.89, stdev=138.65, samples=9
   iops        : min=  456, max=  544, avg=497.22, stdev=34.66, samples=9
  lat (msec)   : 10=0.20%, 20=1.29%, 50=98.01%, 100=0.51%
  cpu          : usr=0.59%, sys=6.75%, ctx=2627, majf=0, minf=9
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=8008KiB/s (8200kB/s), 2002KiB/s-2013KiB/s (2050kB/s-2061kB/s), io=40.0MiB (41.9MB), run=5087-5115msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=183064: Sat Jun 27 00:33:45 2026
  read: IOPS=208, BW=834KiB/s (854kB/s)(5048KiB/6052msec)
    slat (usec): min=54, max=7625, avg=2517.88, stdev=931.91
    clat (usec): min=7744, max=76745, avg=36240.50, stdev=7626.86
     lat (usec): min=8999, max=80441, avg=38760.08, stdev=7924.03
    clat percentiles (usec):
     |  1.00th=[23200],  5.00th=[26084], 10.00th=[28443], 20.00th=[30540],
     | 30.00th=[32113], 40.00th=[33817], 50.00th=[34866], 60.00th=[36439],
     | 70.00th=[38536], 80.00th=[41681], 90.00th=[46400], 95.00th=[50070],
     | 99.00th=[57934], 99.50th=[71828], 99.90th=[76022], 99.95th=[77071],
     | 99.99th=[77071]
   bw (  KiB/s): min=  688, max= 1008, per=24.85%, avg=821.18, stdev=94.69, samples=11
   iops        : min=  172, max=  252, avg=205.27, stdev=23.65, samples=11
  write: IOPS=214, BW=858KiB/s (878kB/s)(5192KiB/6052msec); 0 zone resets
    slat (usec): min=40, max=2721, avg=162.78, stdev=112.79
    clat (usec): min=22735, max=73082, avg=36477.25, stdev=6398.78
     lat (usec): min=22869, max=73216, avg=36641.06, stdev=6397.95
    clat percentiles (usec):
     |  1.00th=[25822],  5.00th=[28443], 10.00th=[29754], 20.00th=[31327],
     | 30.00th=[32637], 40.00th=[33817], 50.00th=[35390], 60.00th=[36439],
     | 70.00th=[38536], 80.00th=[41157], 90.00th=[45351], 95.00th=[48497],
     | 99.00th=[55313], 99.50th=[62129], 99.90th=[71828], 99.95th=[72877],
     | 99.99th=[72877]
   bw (  KiB/s): min=  584, max=  992, per=24.87%, avg=855.64, stdev=119.04, samples=11
   iops        : min=  146, max=  248, avg=213.91, stdev=29.76, samples=11
  lat (msec)   : 10=0.12%, 20=0.12%, 50=95.98%, 100=3.79%
  cpu          : usr=0.30%, sys=5.98%, ctx=4475, majf=0, minf=13
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=183065: Sat Jun 27 00:33:45 2026
  read: IOPS=204, BW=820KiB/s (839kB/s)(4976KiB/6070msec)
    slat (usec): min=88, max=7503, avg=2581.70, stdev=922.58
    clat (usec): min=12944, max=76335, avg=36555.95, stdev=7371.87
     lat (usec): min=14965, max=79528, avg=39139.51, stdev=7707.68
    clat percentiles (usec):
     |  1.00th=[22414],  5.00th=[26870], 10.00th=[28705], 20.00th=[30802],
     | 30.00th=[32637], 40.00th=[33817], 50.00th=[35390], 60.00th=[36963],
     | 70.00th=[39060], 80.00th=[41681], 90.00th=[46400], 95.00th=[50070],
     | 99.00th=[56886], 99.50th=[64750], 99.90th=[73925], 99.95th=[76022],
     | 99.99th=[76022]
   bw (  KiB/s): min=  688, max=  912, per=24.36%, avg=805.00, stdev=76.79, samples=11
   iops        : min=  172, max=  228, avg=201.18, stdev=19.23, samples=11
  write: IOPS=216, BW=867KiB/s (888kB/s)(5264KiB/6070msec); 0 zone resets
    slat (usec): min=55, max=877, avg=163.09, stdev=91.75
    clat (usec): min=13992, max=67774, avg=36352.11, stdev=6455.33
     lat (usec): min=14172, max=67932, avg=36516.23, stdev=6454.60
    clat percentiles (usec):
     |  1.00th=[25822],  5.00th=[28443], 10.00th=[29754], 20.00th=[31327],
     | 30.00th=[32637], 40.00th=[33817], 50.00th=[34866], 60.00th=[36439],
     | 70.00th=[38536], 80.00th=[41157], 90.00th=[45351], 95.00th=[49021],
     | 99.00th=[57934], 99.50th=[61604], 99.90th=[65274], 99.95th=[67634],
     | 99.99th=[67634]
   bw (  KiB/s): min=  688, max= 1000, per=25.04%, avg=861.64, stdev=106.65, samples=11
   iops        : min=  172, max=  250, avg=215.36, stdev=26.62, samples=11
  lat (msec)   : 20=0.31%, 50=95.31%, 100=4.38%
  cpu          : usr=0.38%, sys=5.93%, ctx=4412, majf=0, minf=12
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=183066: Sat Jun 27 00:33:45 2026
  read: IOPS=206, BW=825KiB/s (845kB/s)(4984KiB/6041msec)
    slat (usec): min=37, max=7849, avg=2525.67, stdev=928.70
    clat (usec): min=14068, max=67460, avg=35991.52, stdev=6845.17
     lat (usec): min=14141, max=70640, avg=38519.15, stdev=7216.62
    clat percentiles (usec):
     |  1.00th=[22676],  5.00th=[26608], 10.00th=[28705], 20.00th=[30278],
     | 30.00th=[32375], 40.00th=[33817], 50.00th=[35390], 60.00th=[36439],
     | 70.00th=[38536], 80.00th=[40633], 90.00th=[45351], 95.00th=[49546],
     | 99.00th=[54789], 99.50th=[57410], 99.90th=[62129], 99.95th=[67634],
     | 99.99th=[67634]
   bw (  KiB/s): min=  616, max=  992, per=24.87%, avg=822.09, stdev=122.94, samples=11
   iops        : min=  154, max=  248, avg=205.45, stdev=30.67, samples=11
  write: IOPS=217, BW=870KiB/s (891kB/s)(5256KiB/6041msec); 0 zone resets
    slat (usec): min=35, max=931, avg=163.54, stdev=94.45
    clat (usec): min=11179, max=78822, avg=36739.16, stdev=7003.63
     lat (usec): min=11375, max=78939, avg=36903.70, stdev=7005.77
    clat percentiles (usec):
     |  1.00th=[25822],  5.00th=[28443], 10.00th=[30016], 20.00th=[31589],
     | 30.00th=[32900], 40.00th=[33817], 50.00th=[35390], 60.00th=[36963],
     | 70.00th=[39060], 80.00th=[41681], 90.00th=[45876], 95.00th=[49021],
     | 99.00th=[60556], 99.50th=[64750], 99.90th=[74974], 99.95th=[79168],
     | 99.99th=[79168]
   bw (  KiB/s): min=  680, max=  960, per=25.04%, avg=861.73, stdev=75.38, samples=11
   iops        : min=  170, max=  240, avg=215.36, stdev=18.84, samples=11
  lat (msec)   : 20=0.66%, 50=95.16%, 100=4.18%
  cpu          : usr=0.58%, sys=5.70%, ctx=4426, majf=0, minf=11
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=183067: Sat Jun 27 00:33:45 2026
  read: IOPS=208, BW=833KiB/s (853kB/s)(5060KiB/6071msec)
    slat (usec): min=996, max=7865, avg=2525.29, stdev=904.09
    clat (usec): min=3500, max=75275, avg=36056.54, stdev=7038.16
     lat (usec): min=5386, max=78492, avg=38583.85, stdev=7331.87
    clat percentiles (usec):
     |  1.00th=[22414],  5.00th=[26870], 10.00th=[28181], 20.00th=[30802],
     | 30.00th=[32375], 40.00th=[33817], 50.00th=[34866], 60.00th=[36963],
     | 70.00th=[38536], 80.00th=[41157], 90.00th=[45876], 95.00th=[49546],
     | 99.00th=[55837], 99.50th=[58983], 99.90th=[66323], 99.95th=[74974],
     | 99.99th=[74974]
   bw (  KiB/s): min=  664, max= 1008, per=24.91%, avg=823.18, stdev=104.56, samples=11
   iops        : min=  166, max=  252, avg=205.73, stdev=26.13, samples=11
  write: IOPS=213, BW=853KiB/s (874kB/s)(5180KiB/6071msec); 0 zone resets
    slat (usec): min=52, max=1595, avg=158.55, stdev=90.74
    clat (usec): min=7966, max=72339, avg=36669.75, stdev=6784.21
     lat (usec): min=8153, max=72494, avg=36829.38, stdev=6787.33
    clat percentiles (usec):
     |  1.00th=[26084],  5.00th=[28443], 10.00th=[29754], 20.00th=[31589],
     | 30.00th=[32900], 40.00th=[33817], 50.00th=[35390], 60.00th=[36963],
     | 70.00th=[39060], 80.00th=[41157], 90.00th=[45876], 95.00th=[49021],
     | 99.00th=[59507], 99.50th=[64750], 99.90th=[68682], 99.95th=[71828],
     | 99.99th=[71828]
   bw (  KiB/s): min=  672, max=  960, per=24.47%, avg=842.09, stdev=87.94, samples=11
   iops        : min=  168, max=  240, avg=210.45, stdev=21.99, samples=11
  lat (msec)   : 4=0.04%, 10=0.08%, 20=0.35%, 50=94.96%, 100=4.57%
  cpu          : usr=0.66%, sys=5.63%, ctx=4566, majf=0, minf=11
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=3306KiB/s (3385kB/s), 820KiB/s-834KiB/s (839kB/s-854kB/s), io=19.6MiB (20.5MB), run=6041-6071msec
  WRITE: bw=3441KiB/s (3524kB/s), 853KiB/s-870KiB/s (874kB/s-891kB/s), io=20.4MiB (21.4MB), run=6041-6071msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.19
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=183071: Sat Jun 27 00:33:48 2026
  read: IOPS=9, BW=385MiB/s (403MB/s)(40.0MiB/104msec)
    slat (nsec): min=12838k, max=12838k, avg=12837609.00, stdev= 0.00
    clat (nsec): min=85686k, max=85686k, avg=85686462.00, stdev= 0.00
     lat (nsec): min=98589k, max=98589k, avg=98588608.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[85459],  5.00th=[85459], 10.00th=[85459], 20.00th=[85459],
     | 30.00th=[85459], 40.00th=[85459], 50.00th=[85459], 60.00th=[85459],
     | 70.00th=[85459], 80.00th=[85459], 90.00th=[85459], 95.00th=[85459],
     | 99.00th=[85459], 99.50th=[85459], 99.90th=[85459], 99.95th=[85459],
     | 99.99th=[85459]
  lat (msec)   : 100=100.00%
  cpu          : usr=0.00%, sys=10.58%, ctx=11, majf=0, minf=9
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=385MiB/s (403MB/s), 385MiB/s-385MiB/s (403MB/s-403MB/s), io=40.0MiB (41.9MB), run=104-104msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout