== sanity test 398c: run fio to test AIO ================= 10:10:37 (1783519837)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB, 40 MiB) copied, 0.142678 s, 294 MB/s
osc.lustre-OST0000-osc-ffff930c1fe8c800.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=188566: Wed Jul  8 10:10:46 2026
  write: IOPS=502, BW=2010KiB/s (2058kB/s)(10.0MiB/5095msec); 0 zone resets
    slat (usec): min=42, max=1196, avg=185.51, stdev=96.47
    clat (usec): min=12279, max=53541, avg=31605.71, stdev=4363.22
     lat (usec): min=12412, max=53743, avg=31792.43, stdev=4369.81
    clat percentiles (usec):
     |  1.00th=[16319],  5.00th=[24773], 10.00th=[26870], 20.00th=[28967],
     | 30.00th=[30016], 40.00th=[30802], 50.00th=[31589], 60.00th=[32375],
     | 70.00th=[33424], 80.00th=[34866], 90.00th=[36963], 95.00th=[38536],
     | 99.00th=[42730], 99.50th=[42730], 99.90th=[50070], 99.95th=[53216],
     | 99.99th=[53740]
   bw (  KiB/s): min= 1792, max= 2267, per=25.03%, avg=1999.44, stdev=158.24, samples=9
   iops        : min=  448, max=  566, avg=499.78, stdev=39.40, samples=9
  lat (msec)   : 20=1.80%, 50=98.05%, 100=0.16%
  cpu          : usr=0.39%, sys=7.22%, ctx=2639, majf=0, minf=9
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=188567: Wed Jul  8 10:10:46 2026
  write: IOPS=500, BW=2000KiB/s (2048kB/s)(10.0MiB/5119msec); 0 zone resets
    slat (usec): min=44, max=26699, avg=185.16, stdev=533.79
    clat (usec): min=6432, max=44015, avg=31608.47, stdev=4160.65
     lat (usec): min=6488, max=44183, avg=31794.86, stdev=4149.47
    clat percentiles (usec):
     |  1.00th=[18220],  5.00th=[25297], 10.00th=[26870], 20.00th=[28967],
     | 30.00th=[30016], 40.00th=[30802], 50.00th=[31589], 60.00th=[32375],
     | 70.00th=[33162], 80.00th=[34341], 90.00th=[36963], 95.00th=[38011],
     | 99.00th=[42206], 99.50th=[42730], 99.90th=[42730], 99.95th=[43779],
     | 99.99th=[43779]
   bw (  KiB/s): min= 1792, max= 2176, per=24.86%, avg=1986.22, stdev=141.35, samples=9
   iops        : min=  448, max=  544, avg=496.56, stdev=35.34, samples=9
  lat (msec)   : 10=0.04%, 20=1.64%, 50=98.32%
  cpu          : usr=0.53%, sys=7.03%, ctx=2537, majf=0, minf=12
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=188568: Wed Jul  8 10:10:46 2026
  write: IOPS=499, BW=1998KiB/s (2046kB/s)(10.0MiB/5124msec); 0 zone resets
    slat (usec): min=55, max=729, avg=184.87, stdev=98.47
    clat (usec): min=3161, max=43354, avg=31522.82, stdev=4126.54
     lat (usec): min=3274, max=43529, avg=31708.83, stdev=4129.86
    clat percentiles (usec):
     |  1.00th=[17957],  5.00th=[24511], 10.00th=[26870], 20.00th=[28967],
     | 30.00th=[30016], 40.00th=[30802], 50.00th=[31589], 60.00th=[32375],
     | 70.00th=[33424], 80.00th=[34341], 90.00th=[36439], 95.00th=[38011],
     | 99.00th=[42206], 99.50th=[42730], 99.90th=[42730], 99.95th=[43254],
     | 99.99th=[43254]
   bw (  KiB/s): min= 1792, max= 2192, per=24.83%, avg=1983.44, stdev=127.36, samples=9
   iops        : min=  448, max=  548, avg=495.78, stdev=31.76, samples=9
  lat (msec)   : 4=0.04%, 20=1.33%, 50=98.63%
  cpu          : usr=0.61%, sys=7.30%, ctx=2738, majf=0, minf=13
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=188569: Wed Jul  8 10:10:46 2026
  write: IOPS=499, BW=1997KiB/s (2045kB/s)(10.0MiB/5127msec); 0 zone resets
    slat (usec): min=41, max=717, avg=200.43, stdev=97.87
    clat (usec): min=14797, max=43088, avg=31494.40, stdev=4053.06
     lat (usec): min=14976, max=43473, avg=31696.08, stdev=4064.63
    clat percentiles (usec):
     |  1.00th=[19006],  5.00th=[24773], 10.00th=[26870], 20.00th=[28705],
     | 30.00th=[29754], 40.00th=[30802], 50.00th=[31589], 60.00th=[32375],
     | 70.00th=[33162], 80.00th=[34341], 90.00th=[36439], 95.00th=[38011],
     | 99.00th=[41681], 99.50th=[42730], 99.90th=[42730], 99.95th=[43254],
     | 99.99th=[43254]
   bw (  KiB/s): min= 1792, max= 2200, per=24.77%, avg=1979.00, stdev=131.94, samples=9
   iops        : min=  448, max=  550, avg=494.67, stdev=32.92, samples=9
  lat (msec)   : 20=1.60%, 50=98.40%
  cpu          : usr=0.49%, sys=7.67%, ctx=2613, majf=0, minf=11
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=7989KiB/s (8181kB/s), 1997KiB/s-2010KiB/s (2045kB/s-2058kB/s), io=40.0MiB (41.9MB), run=5095-5127msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=188579: Wed Jul  8 10:10:56 2026
  read: IOPS=185, BW=741KiB/s (759kB/s)(5048KiB/6811msec)
    slat (usec): min=1148, max=16714, avg=2943.47, stdev=1060.49
    clat (usec): min=12213, max=71448, avg=40429.41, stdev=7827.03
     lat (usec): min=14431, max=74108, avg=43374.48, stdev=8193.74
    clat percentiles (usec):
     |  1.00th=[25560],  5.00th=[29754], 10.00th=[31589], 20.00th=[34341],
     | 30.00th=[35914], 40.00th=[37487], 50.00th=[39584], 60.00th=[41157],
     | 70.00th=[43779], 80.00th=[46400], 90.00th=[51119], 95.00th=[54264],
     | 99.00th=[63177], 99.50th=[66847], 99.90th=[69731], 99.95th=[71828],
     | 99.99th=[71828]
   bw (  KiB/s): min=  592, max=  880, per=24.97%, avg=735.69, stdev=104.44, samples=13
   iops        : min=  148, max=  220, avg=183.92, stdev=26.11, samples=13
  write: IOPS=190, BW=762KiB/s (781kB/s)(5192KiB/6811msec); 0 zone resets
    slat (usec): min=56, max=1787, avg=182.65, stdev=105.21
    clat (usec): min=12048, max=73785, avg=40898.73, stdev=7490.64
     lat (usec): min=12204, max=73929, avg=41082.49, stdev=7481.35
    clat percentiles (usec):
     |  1.00th=[28443],  5.00th=[31327], 10.00th=[32637], 20.00th=[34341],
     | 30.00th=[36439], 40.00th=[37487], 50.00th=[39060], 60.00th=[41157],
     | 70.00th=[44303], 80.00th=[47973], 90.00th=[51119], 95.00th=[54789],
     | 99.00th=[61080], 99.50th=[64750], 99.90th=[73925], 99.95th=[73925],
     | 99.99th=[73925]
   bw (  KiB/s): min=  592, max=  944, per=24.61%, avg=754.92, stdev=114.36, samples=13
   iops        : min=  148, max=  236, avg=188.69, stdev=28.60, samples=13
  lat (msec)   : 20=0.23%, 50=86.37%, 100=13.40%
  cpu          : usr=0.51%, sys=5.99%, ctx=4783, majf=0, minf=12
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=188580: Wed Jul  8 10:10:56 2026
  read: IOPS=182, BW=731KiB/s (748kB/s)(4976KiB/6809msec)
    slat (usec): min=1139, max=8898, avg=2969.42, stdev=1003.16
    clat (usec): min=5214, max=72962, avg=40635.44, stdev=7741.78
     lat (usec): min=8336, max=76983, avg=43606.31, stdev=8087.44
    clat percentiles (usec):
     |  1.00th=[26870],  5.00th=[29754], 10.00th=[31851], 20.00th=[34341],
     | 30.00th=[35914], 40.00th=[38011], 50.00th=[39584], 60.00th=[41157],
     | 70.00th=[43779], 80.00th=[46924], 90.00th=[51119], 95.00th=[54789],
     | 99.00th=[62653], 99.50th=[64750], 99.90th=[68682], 99.95th=[72877],
     | 99.99th=[72877]
   bw (  KiB/s): min=  504, max=  832, per=24.62%, avg=725.38, stdev=106.14, samples=13
   iops        : min=  126, max=  208, avg=181.31, stdev=26.53, samples=13
  write: IOPS=193, BW=773KiB/s (792kB/s)(5264KiB/6809msec); 0 zone resets
    slat (usec): min=42, max=957, avg=179.15, stdev=90.92
    clat (usec): min=17863, max=73724, avg=40914.34, stdev=7472.57
     lat (usec): min=17953, max=73886, avg=41094.68, stdev=7469.31
    clat percentiles (usec):
     |  1.00th=[28443],  5.00th=[31065], 10.00th=[32900], 20.00th=[34866],
     | 30.00th=[36439], 40.00th=[38011], 50.00th=[39584], 60.00th=[41157],
     | 70.00th=[43779], 80.00th=[47449], 90.00th=[51119], 95.00th=[54264],
     | 99.00th=[63177], 99.50th=[65274], 99.90th=[68682], 99.95th=[73925],
     | 99.99th=[73925]
   bw (  KiB/s): min=  616, max=  920, per=25.13%, avg=770.62, stdev=110.12, samples=13
   iops        : min=  154, max=  230, avg=192.62, stdev=27.50, samples=13
  lat (msec)   : 10=0.04%, 20=0.08%, 50=87.19%, 100=12.70%
  cpu          : usr=0.29%, sys=6.17%, ctx=4744, majf=0, minf=11
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=188581: Wed Jul  8 10:10:56 2026
  read: IOPS=183, BW=734KiB/s (752kB/s)(4984KiB/6786msec)
    slat (usec): min=43, max=9090, avg=2948.69, stdev=991.66
    clat (usec): min=19822, max=73707, avg=40499.48, stdev=7420.86
     lat (usec): min=24620, max=79795, avg=43449.77, stdev=7737.42
    clat percentiles (usec):
     |  1.00th=[27132],  5.00th=[30016], 10.00th=[32113], 20.00th=[34341],
     | 30.00th=[36439], 40.00th=[38011], 50.00th=[39584], 60.00th=[41157],
     | 70.00th=[43779], 80.00th=[46400], 90.00th=[50070], 95.00th=[53216],
     | 99.00th=[63177], 99.50th=[66847], 99.90th=[70779], 99.95th=[73925],
     | 99.99th=[73925]
   bw (  KiB/s): min=  520, max=  904, per=24.90%, avg=733.69, stdev=117.71, samples=13
   iops        : min=  130, max=  226, avg=183.38, stdev=29.40, samples=13
  write: IOPS=193, BW=775KiB/s (793kB/s)(5256KiB/6786msec); 0 zone resets
    slat (usec): min=47, max=1020, avg=177.05, stdev=92.09
    clat (usec): min=12814, max=74061, avg=41108.20, stdev=7662.05
     lat (usec): min=12993, max=74166, avg=41286.35, stdev=7657.52
    clat percentiles (usec):
     |  1.00th=[28705],  5.00th=[31327], 10.00th=[33162], 20.00th=[34866],
     | 30.00th=[35914], 40.00th=[38011], 50.00th=[39060], 60.00th=[41157],
     | 70.00th=[44303], 80.00th=[47449], 90.00th=[52167], 95.00th=[55313],
     | 99.00th=[63701], 99.50th=[66323], 99.90th=[70779], 99.95th=[73925],
     | 99.99th=[73925]
   bw (  KiB/s): min=  608, max=  936, per=25.05%, avg=768.15, stdev=96.20, samples=13
   iops        : min=  152, max=  234, avg=192.00, stdev=24.03, samples=13
  lat (msec)   : 20=0.16%, 50=87.42%, 100=12.42%
  cpu          : usr=0.29%, sys=6.13%, ctx=4695, majf=0, minf=11
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=188582: Wed Jul  8 10:10:56 2026
  read: IOPS=185, BW=744KiB/s (762kB/s)(5060KiB/6803msec)
    slat (usec): min=1118, max=12210, avg=2949.05, stdev=1014.44
    clat (usec): min=22555, max=73409, avg=40626.06, stdev=7704.14
     lat (usec): min=25066, max=76573, avg=43576.94, stdev=8069.40
    clat percentiles (usec):
     |  1.00th=[27132],  5.00th=[30016], 10.00th=[31589], 20.00th=[34341],
     | 30.00th=[36439], 40.00th=[38011], 50.00th=[39584], 60.00th=[40633],
     | 70.00th=[43254], 80.00th=[47449], 90.00th=[51119], 95.00th=[54264],
     | 99.00th=[62129], 99.50th=[65799], 99.90th=[70779], 99.95th=[73925],
     | 99.99th=[73925]
   bw (  KiB/s): min=  592, max=  848, per=25.08%, avg=738.77, stdev=91.35, samples=13
   iops        : min=  148, max=  212, avg=184.69, stdev=22.84, samples=13
  write: IOPS=190, BW=761KiB/s (780kB/s)(5180KiB/6803msec); 0 zone resets
    slat (usec): min=64, max=724, avg=177.91, stdev=93.54
    clat (usec): min=10016, max=77487, avg=40877.37, stdev=7530.00
     lat (usec): min=10332, max=77656, avg=41056.34, stdev=7527.20
    clat percentiles (usec):
     |  1.00th=[28443],  5.00th=[31327], 10.00th=[32900], 20.00th=[34341],
     | 30.00th=[35914], 40.00th=[37487], 50.00th=[39584], 60.00th=[41157],
     | 70.00th=[44303], 80.00th=[47449], 90.00th=[51119], 95.00th=[54264],
     | 99.00th=[62129], 99.50th=[68682], 99.90th=[73925], 99.95th=[77071],
     | 99.99th=[77071]
   bw (  KiB/s): min=  520, max=  872, per=24.60%, avg=754.46, stdev=117.59, samples=13
   iops        : min=  130, max=  218, avg=188.62, stdev=29.40, samples=13
  lat (msec)   : 20=0.04%, 50=86.80%, 100=13.16%
  cpu          : usr=0.43%, sys=6.06%, ctx=4732, majf=0, minf=10
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=2946KiB/s (3017kB/s), 731KiB/s-744KiB/s (748kB/s-762kB/s), io=19.6MiB (20.5MB), run=6786-6811msec
  WRITE: bw=3067KiB/s (3141kB/s), 761KiB/s-775KiB/s (780kB/s-793kB/s), io=20.4MiB (21.4MB), run=6786-6811msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.19
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=188587: Wed Jul  8 10:10:59 2026
  read: IOPS=10, BW=417MiB/s (437MB/s)(40.0MiB/96msec)
    slat (nsec): min=11701k, max=11701k, avg=11700623.00, stdev= 0.00
    clat (nsec): min=78472k, max=78472k, avg=78471859.00, stdev= 0.00
     lat (nsec): min=90300k, max=90300k, avg=90300059.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[78119],  5.00th=[78119], 10.00th=[78119], 20.00th=[78119],
     | 30.00th=[78119], 40.00th=[78119], 50.00th=[78119], 60.00th=[78119],
     | 70.00th=[78119], 80.00th=[78119], 90.00th=[78119], 95.00th=[78119],
     | 99.00th=[78119], 99.50th=[78119], 99.90th=[78119], 99.95th=[78119],
     | 99.99th=[78119]
  lat (msec)   : 100=100.00%
  cpu          : usr=0.00%, sys=11.58%, ctx=7, majf=0, minf=9
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=417MiB/s (437MB/s), 417MiB/s-417MiB/s (437MB/s-437MB/s), io=40.0MiB (41.9MB), run=96-96msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout