== sanity test 398c: run fio to test AIO ================= 10:07:20 (1770649640)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB, 40 MiB) copied, 0.147898 s, 284 MB/s
osc.lustre-OST0000-osc-ffff933091f14800.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=196509: Mon Feb  9 10:07:29 2026
  write: IOPS=465, BW=1862KiB/s (1907kB/s)(10.0MiB/5500msec); 0 zone resets
    slat (usec): min=28, max=27000, avg=116.80, stdev=534.43
    clat (usec): min=10527, max=59913, avg=33934.43, stdev=4004.57
     lat (usec): min=10777, max=59980, avg=34052.42, stdev=3978.00
    clat percentiles (usec):
     |  1.00th=[24511],  5.00th=[28181], 10.00th=[30016], 20.00th=[31327],
     | 30.00th=[32375], 40.00th=[33162], 50.00th=[33817], 60.00th=[34866],
     | 70.00th=[35390], 80.00th=[36439], 90.00th=[38011], 95.00th=[39584],
     | 99.00th=[44827], 99.50th=[50070], 99.90th=[56886], 99.95th=[56886],
     | 99.99th=[60031]
   bw (  KiB/s): min= 1648, max= 2064, per=25.17%, avg=1872.30, stdev=103.96, samples=10
   iops        : min=  412, max=  516, avg=468.00, stdev=25.96, samples=10
  lat (msec)   : 20=0.78%, 50=98.71%, 100=0.51%
  cpu          : usr=0.35%, sys=4.51%, ctx=2813, majf=0, minf=8
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=196510: Mon Feb  9 10:07:29 2026
  write: IOPS=464, BW=1860KiB/s (1904kB/s)(10.0MiB/5506msec); 0 zone resets
    slat (usec): min=25, max=894, avg=100.66, stdev=57.54
    clat (usec): min=15079, max=52560, avg=33912.26, stdev=3887.41
     lat (usec): min=15159, max=52696, avg=34014.14, stdev=3887.36
    clat percentiles (usec):
     |  1.00th=[24773],  5.00th=[27919], 10.00th=[29754], 20.00th=[31327],
     | 30.00th=[32375], 40.00th=[33162], 50.00th=[33817], 60.00th=[34341],
     | 70.00th=[35390], 80.00th=[36439], 90.00th=[38011], 95.00th=[39584],
     | 99.00th=[44827], 99.50th=[50070], 99.90th=[52691], 99.95th=[52691],
     | 99.99th=[52691]
   bw (  KiB/s): min= 1640, max= 2088, per=25.14%, avg=1869.80, stdev=110.88, samples=10
   iops        : min=  410, max=  522, avg=467.40, stdev=27.71, samples=10
  lat (msec)   : 20=0.62%, 50=98.98%, 100=0.39%
  cpu          : usr=0.36%, sys=3.85%, ctx=2601, majf=0, minf=8
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=196511: Mon Feb  9 10:07:29 2026
  write: IOPS=464, BW=1860KiB/s (1904kB/s)(10.0MiB/5506msec); 0 zone resets
    slat (usec): min=26, max=502, avg=100.82, stdev=53.82
    clat (usec): min=7018, max=50746, avg=33882.19, stdev=3749.41
     lat (usec): min=7084, max=50841, avg=33984.23, stdev=3749.88
    clat percentiles (usec):
     |  1.00th=[24511],  5.00th=[28443], 10.00th=[29754], 20.00th=[31327],
     | 30.00th=[32375], 40.00th=[33162], 50.00th=[33817], 60.00th=[34341],
     | 70.00th=[35390], 80.00th=[36439], 90.00th=[38011], 95.00th=[39584],
     | 99.00th=[44303], 99.50th=[49546], 99.90th=[50594], 99.95th=[50594],
     | 99.99th=[50594]
   bw (  KiB/s): min= 1648, max= 2088, per=25.08%, avg=1865.40, stdev=108.18, samples=10
   iops        : min=  412, max=  522, avg=466.30, stdev=27.06, samples=10
  lat (msec)   : 10=0.04%, 20=0.43%, 50=99.26%, 100=0.27%
  cpu          : usr=0.35%, sys=3.94%, ctx=2715, majf=0, minf=9
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=196512: Mon Feb  9 10:07:29 2026
  write: IOPS=467, BW=1870KiB/s (1915kB/s)(10.0MiB/5476msec); 0 zone resets
    slat (usec): min=28, max=480, avg=100.71, stdev=53.64
    clat (usec): min=6024, max=61818, avg=34047.75, stdev=4258.53
     lat (usec): min=6076, max=61950, avg=34149.65, stdev=4258.62
    clat percentiles (usec):
     |  1.00th=[24511],  5.00th=[28181], 10.00th=[29754], 20.00th=[31327],
     | 30.00th=[32375], 40.00th=[33162], 50.00th=[33817], 60.00th=[34866],
     | 70.00th=[35390], 80.00th=[36439], 90.00th=[38011], 95.00th=[40109],
     | 99.00th=[47449], 99.50th=[55837], 99.90th=[61604], 99.95th=[61604],
     | 99.99th=[61604]
   bw (  KiB/s): min= 1656, max= 2096, per=25.30%, avg=1882.00, stdev=113.13, samples=10
   iops        : min=  414, max=  524, avg=470.50, stdev=28.28, samples=10
  lat (msec)   : 10=0.04%, 20=0.66%, 50=98.63%, 100=0.66%
  cpu          : usr=0.53%, sys=3.69%, ctx=2753, majf=0, minf=8
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=7439KiB/s (7618kB/s), 1860KiB/s-1870KiB/s (1904kB/s-1915kB/s), io=40.0MiB (41.9MB), run=5476-5506msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=196523: Mon Feb  9 10:07:38 2026
  read: IOPS=200, BW=803KiB/s (823kB/s)(5048KiB/6284msec)
    slat (usec): min=1080, max=11823, avg=2911.45, stdev=1052.40
    clat (usec): min=22652, max=62227, avg=37382.52, stdev=6635.38
     lat (usec): min=24936, max=64703, avg=40295.39, stdev=6996.67
    clat percentiles (usec):
     |  1.00th=[25297],  5.00th=[28181], 10.00th=[30016], 20.00th=[31851],
     | 30.00th=[33424], 40.00th=[34866], 50.00th=[36439], 60.00th=[38011],
     | 70.00th=[40109], 80.00th=[42730], 90.00th=[46924], 95.00th=[50070],
     | 99.00th=[55837], 99.50th=[56361], 99.90th=[60031], 99.95th=[62129],
     | 99.99th=[62129]
   bw (  KiB/s): min=  540, max=  992, per=25.30%, avg=805.67, stdev=135.15, samples=12
   iops        : min=  135, max=  248, avg=201.42, stdev=33.79, samples=12
  write: IOPS=206, BW=826KiB/s (846kB/s)(5192KiB/6284msec); 0 zone resets
    slat (usec): min=36, max=612, avg=94.86, stdev=51.78
    clat (usec): min=9559, max=60689, avg=37004.69, stdev=5755.63
     lat (usec): min=9610, max=60927, avg=37100.63, stdev=5751.99
    clat percentiles (usec):
     |  1.00th=[26870],  5.00th=[29230], 10.00th=[30540], 20.00th=[32113],
     | 30.00th=[33424], 40.00th=[34866], 50.00th=[35914], 60.00th=[37487],
     | 70.00th=[39060], 80.00th=[41681], 90.00th=[45351], 95.00th=[47973],
     | 99.00th=[53216], 99.50th=[56361], 99.90th=[56886], 99.95th=[60556],
     | 99.99th=[60556]
   bw (  KiB/s): min=  546, max= 1048, per=25.02%, avg=829.50, stdev=126.70, samples=12
   iops        : min=  136, max=  262, avg=207.33, stdev=31.78, samples=12
  lat (msec)   : 10=0.04%, 50=96.02%, 100=3.95%
  cpu          : usr=0.35%, sys=4.55%, ctx=4951, majf=0, minf=10
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=196524: Mon Feb  9 10:07:38 2026
  read: IOPS=199, BW=799KiB/s (818kB/s)(4976KiB/6229msec)
    slat (usec): min=23, max=14487, avg=2928.15, stdev=1139.66
    clat (usec): min=23429, max=73080, avg=37552.92, stdev=6768.88
     lat (usec): min=25604, max=73104, avg=40482.64, stdev=7087.19
    clat percentiles (usec):
     |  1.00th=[25297],  5.00th=[28443], 10.00th=[30016], 20.00th=[32113],
     | 30.00th=[33817], 40.00th=[34866], 50.00th=[36439], 60.00th=[38011],
     | 70.00th=[40109], 80.00th=[42730], 90.00th=[46400], 95.00th=[50070],
     | 99.00th=[57410], 99.50th=[58983], 99.90th=[66847], 99.95th=[72877],
     | 99.99th=[72877]
   bw (  KiB/s): min=  585, max=  960, per=25.15%, avg=800.75, stdev=130.92, samples=12
   iops        : min=  146, max=  240, avg=200.17, stdev=32.77, samples=12
  write: IOPS=211, BW=845KiB/s (865kB/s)(5264KiB/6229msec); 0 zone resets
    slat (usec): min=24, max=1003, avg=96.45, stdev=61.01
    clat (usec): min=18405, max=61162, avg=37181.49, stdev=6076.80
     lat (usec): min=18499, max=61263, avg=37279.04, stdev=6074.45
    clat percentiles (usec):
     |  1.00th=[26084],  5.00th=[29230], 10.00th=[30540], 20.00th=[32375],
     | 30.00th=[33424], 40.00th=[34866], 50.00th=[35914], 60.00th=[37487],
     | 70.00th=[39584], 80.00th=[42206], 90.00th=[45876], 95.00th=[47973],
     | 99.00th=[54264], 99.50th=[56886], 99.90th=[60031], 99.95th=[61080],
     | 99.99th=[61080]
   bw (  KiB/s): min=  655, max= 1056, per=25.64%, avg=849.92, stdev=110.75, samples=12
   iops        : min=  163, max=  264, avg=212.42, stdev=27.81, samples=12
  lat (msec)   : 20=0.08%, 50=95.66%, 100=4.26%
  cpu          : usr=0.58%, sys=4.30%, ctx=4883, majf=0, minf=13
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=196525: Mon Feb  9 10:07:38 2026
  read: IOPS=197, BW=791KiB/s (810kB/s)(4984KiB/6300msec)
    slat (usec): min=1120, max=14815, avg=3011.59, stdev=1147.97
    clat (usec): min=6125, max=65152, avg=37566.83, stdev=6709.23
     lat (usec): min=8102, max=71193, avg=40579.87, stdev=7121.21
    clat percentiles (usec):
     |  1.00th=[25035],  5.00th=[28443], 10.00th=[30016], 20.00th=[32113],
     | 30.00th=[33817], 40.00th=[34866], 50.00th=[36439], 60.00th=[38011],
     | 70.00th=[40109], 80.00th=[43254], 90.00th=[47449], 95.00th=[50594],
     | 99.00th=[54789], 99.50th=[56361], 99.90th=[61604], 99.95th=[65274],
     | 99.99th=[65274]
   bw (  KiB/s): min=  527, max=  968, per=25.00%, avg=795.92, stdev=115.71, samples=12
   iops        : min=  131, max=  242, avg=198.92, stdev=29.09, samples=12
  write: IOPS=208, BW=834KiB/s (854kB/s)(5256KiB/6300msec); 0 zone resets
    slat (usec): min=35, max=1184, avg=99.29, stdev=62.06
    clat (usec): min=18169, max=75484, avg=37447.49, stdev=6236.15
     lat (usec): min=18300, max=75629, avg=37547.86, stdev=6234.10
    clat percentiles (usec):
     |  1.00th=[26608],  5.00th=[29492], 10.00th=[31065], 20.00th=[32637],
     | 30.00th=[33817], 40.00th=[34866], 50.00th=[35914], 60.00th=[37487],
     | 70.00th=[39584], 80.00th=[42730], 90.00th=[45351], 95.00th=[49546],
     | 99.00th=[56361], 99.50th=[57934], 99.90th=[63177], 99.95th=[74974],
     | 99.99th=[74974]
   bw (  KiB/s): min=  610, max= 1032, per=25.04%, avg=830.17, stdev=147.09, samples=12
   iops        : min=  152, max=  258, avg=207.50, stdev=36.84, samples=12
  lat (msec)   : 10=0.04%, 20=0.04%, 50=94.69%, 100=5.23%
  cpu          : usr=0.38%, sys=4.51%, ctx=4934, majf=0, minf=11
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=196526: Mon Feb  9 10:07:38 2026
  read: IOPS=200, BW=803KiB/s (822kB/s)(5060KiB/6301msec)
    slat (usec): min=1013, max=14316, avg=2941.03, stdev=1139.72
    clat (usec): min=7533, max=61844, avg=37156.50, stdev=6501.30
     lat (usec): min=10111, max=65599, avg=40098.90, stdev=6901.72
    clat percentiles (usec):
     |  1.00th=[25822],  5.00th=[28443], 10.00th=[29754], 20.00th=[31851],
     | 30.00th=[33424], 40.00th=[34866], 50.00th=[35914], 60.00th=[37487],
     | 70.00th=[39584], 80.00th=[42206], 90.00th=[46400], 95.00th=[50070],
     | 99.00th=[54264], 99.50th=[56886], 99.90th=[61604], 99.95th=[61604],
     | 99.99th=[61604]
   bw (  KiB/s): min=  527, max=  992, per=25.29%, avg=805.25, stdev=132.23, samples=12
   iops        : min=  131, max=  248, avg=201.25, stdev=33.20, samples=12
  write: IOPS=205, BW=822KiB/s (842kB/s)(5180KiB/6301msec); 0 zone resets
    slat (usec): min=38, max=646, avg=98.54, stdev=56.32
    clat (usec): min=23849, max=61222, avg=37449.61, stdev=5989.05
     lat (usec): min=23890, max=61263, avg=37549.35, stdev=5988.54
    clat percentiles (usec):
     |  1.00th=[26608],  5.00th=[30016], 10.00th=[30802], 20.00th=[32375],
     | 30.00th=[33817], 40.00th=[34866], 50.00th=[36439], 60.00th=[38011],
     | 70.00th=[40109], 80.00th=[42206], 90.00th=[45876], 95.00th=[47973],
     | 99.00th=[55313], 99.50th=[56361], 99.90th=[59507], 99.95th=[61080],
     | 99.99th=[61080]
   bw (  KiB/s): min=  559, max=  944, per=24.77%, avg=821.25, stdev=119.75, samples=12
   iops        : min=  139, max=  236, avg=205.25, stdev=30.09, samples=12
  lat (msec)   : 10=0.04%, 20=0.04%, 50=95.74%, 100=4.18%
  cpu          : usr=0.35%, sys=4.54%, ctx=4903, majf=0, minf=11
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=3185KiB/s (3261kB/s), 791KiB/s-803KiB/s (810kB/s-823kB/s), io=19.6MiB (20.5MB), run=6229-6301msec
  WRITE: bw=3316KiB/s (3395kB/s), 822KiB/s-845KiB/s (842kB/s-865kB/s), io=20.4MiB (21.4MB), run=6229-6301msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.19
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=196530: Mon Feb  9 10:07:42 2026
  read: IOPS=10, BW=400MiB/s (419MB/s)(40.0MiB/100msec)
    slat (nsec): min=8833.2k, max=8833.2k, avg=8833217.00, stdev= 0.00
    clat (nsec): min=85735k, max=85735k, avg=85734543.00, stdev= 0.00
     lat (nsec): min=94571k, max=94571k, avg=94570581.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[85459],  5.00th=[85459], 10.00th=[85459], 20.00th=[85459],
     | 30.00th=[85459], 40.00th=[85459], 50.00th=[85459], 60.00th=[85459],
     | 70.00th=[85459], 80.00th=[85459], 90.00th=[85459], 95.00th=[85459],
     | 99.00th=[85459], 99.50th=[85459], 99.90th=[85459], 99.95th=[85459],
     | 99.99th=[85459]
  lat (msec)   : 100=100.00%
  cpu          : usr=0.00%, sys=9.09%, ctx=8, majf=0, minf=8
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=400MiB/s (419MB/s), 400MiB/s-400MiB/s (419MB/s-419MB/s), io=40.0MiB (41.9MB), run=100-100msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout