== sanity test 398c: run fio to test AIO ================= 06:43:49 (1783075429)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB, 40 MiB) copied, 0.121397 s, 346 MB/s
osc.lustre-OST0000-osc-ffff8b90747a1800.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=182294: Fri Jul  3 06:43:57 2026
  write: IOPS=522, BW=2092KiB/s (2142kB/s)(10.0MiB/4896msec); 0 zone resets
    slat (usec): min=34, max=971, avg=168.19, stdev=87.89
    clat (usec): min=8702, max=61445, avg=30378.83, stdev=3992.65
     lat (usec): min=8930, max=61594, avg=30547.97, stdev=3994.89
    clat percentiles (usec):
     |  1.00th=[19006],  5.00th=[24773], 10.00th=[26346], 20.00th=[27395],
     | 30.00th=[28443], 40.00th=[29492], 50.00th=[30016], 60.00th=[31065],
     | 70.00th=[32375], 80.00th=[33817], 90.00th=[35390], 95.00th=[36963],
     | 99.00th=[40109], 99.50th=[41681], 99.90th=[52167], 99.95th=[52167],
     | 99.99th=[61604]
   bw (  KiB/s): min= 1824, max= 2323, per=25.18%, avg=2096.33, stdev=165.53, samples=9
   iops        : min=  456, max=  580, avg=524.00, stdev=41.26, samples=9
  lat (msec)   : 10=0.04%, 20=1.48%, 50=98.36%, 100=0.12%
  cpu          : usr=0.45%, sys=6.88%, ctx=2749, majf=0, minf=7
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=182295: Fri Jul  3 06:43:57 2026
  write: IOPS=522, BW=2090KiB/s (2140kB/s)(10.0MiB/4900msec); 0 zone resets
    slat (usec): min=36, max=947, avg=175.50, stdev=90.39
    clat (usec): min=12856, max=64578, avg=30326.51, stdev=4090.48
     lat (usec): min=13057, max=64635, avg=30502.93, stdev=4099.09
    clat percentiles (usec):
     |  1.00th=[19006],  5.00th=[24773], 10.00th=[26084], 20.00th=[27395],
     | 30.00th=[28443], 40.00th=[29230], 50.00th=[30016], 60.00th=[30802],
     | 70.00th=[32113], 80.00th=[33424], 90.00th=[34866], 95.00th=[36439],
     | 99.00th=[39060], 99.50th=[41157], 99.90th=[62129], 99.95th=[64750],
     | 99.99th=[64750]
   bw (  KiB/s): min= 1856, max= 2316, per=25.16%, avg=2094.67, stdev=165.19, samples=9
   iops        : min=  464, max=  579, avg=523.67, stdev=41.30, samples=9
  lat (msec)   : 20=1.52%, 50=98.16%, 100=0.31%
  cpu          : usr=0.49%, sys=7.12%, ctx=2814, majf=0, minf=8
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=182296: Fri Jul  3 06:43:57 2026
  write: IOPS=520, BW=2082KiB/s (2132kB/s)(10.0MiB/4919msec); 0 zone resets
    slat (usec): min=36, max=3441, avg=166.25, stdev=111.19
    clat (usec): min=7834, max=47960, avg=30241.72, stdev=3869.76
     lat (usec): min=7979, max=48085, avg=30409.00, stdev=3880.39
    clat percentiles (usec):
     |  1.00th=[19792],  5.00th=[24773], 10.00th=[26084], 20.00th=[27395],
     | 30.00th=[28443], 40.00th=[29492], 50.00th=[30016], 60.00th=[30802],
     | 70.00th=[32375], 80.00th=[33424], 90.00th=[34866], 95.00th=[36963],
     | 99.00th=[38536], 99.50th=[40633], 99.90th=[42206], 99.95th=[47973],
     | 99.99th=[47973]
   bw (  KiB/s): min= 1832, max= 2224, per=25.00%, avg=2081.22, stdev=146.63, samples=9
   iops        : min=  458, max=  556, avg=520.22, stdev=36.58, samples=9
  lat (msec)   : 10=0.08%, 20=1.13%, 50=98.79%
  cpu          : usr=0.51%, sys=7.14%, ctx=2875, majf=0, minf=9
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=182297: Fri Jul  3 06:43:57 2026
  write: IOPS=520, BW=2082KiB/s (2132kB/s)(10.0MiB/4919msec); 0 zone resets
    slat (usec): min=41, max=26992, avg=184.59, stdev=538.07
    clat (usec): min=6378, max=57835, avg=30265.05, stdev=3955.25
     lat (usec): min=6481, max=57910, avg=30450.75, stdev=3947.92
    clat percentiles (usec):
     |  1.00th=[19530],  5.00th=[24773], 10.00th=[26084], 20.00th=[27395],
     | 30.00th=[28443], 40.00th=[29492], 50.00th=[30016], 60.00th=[30802],
     | 70.00th=[32113], 80.00th=[33424], 90.00th=[34866], 95.00th=[36439],
     | 99.00th=[38536], 99.50th=[40109], 99.90th=[48497], 99.95th=[52167],
     | 99.99th=[57934]
   bw (  KiB/s): min= 1840, max= 2252, per=25.04%, avg=2084.89, stdev=153.75, samples=9
   iops        : min=  460, max=  563, avg=521.22, stdev=38.44, samples=9
  lat (msec)   : 10=0.12%, 20=1.64%, 50=98.16%, 100=0.08%
  cpu          : usr=0.51%, sys=7.56%, ctx=2866, majf=0, minf=8
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=8327KiB/s (8527kB/s), 2082KiB/s-2092KiB/s (2132kB/s-2142kB/s), io=40.0MiB (41.9MB), run=4896-4919msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=182307: Fri Jul  3 06:44:06 2026
  read: IOPS=213, BW=854KiB/s (875kB/s)(5048KiB/5908msec)
    slat (usec): min=41, max=8603, avg=2445.73, stdev=806.97
    clat (usec): min=6872, max=56738, avg=35143.98, stdev=5390.42
     lat (usec): min=9052, max=60495, avg=37591.53, stdev=5504.00
    clat percentiles (usec):
     |  1.00th=[24511],  5.00th=[27132], 10.00th=[28705], 20.00th=[30540],
     | 30.00th=[32113], 40.00th=[33424], 50.00th=[34866], 60.00th=[35914],
     | 70.00th=[37487], 80.00th=[39584], 90.00th=[42206], 95.00th=[44827],
     | 99.00th=[49546], 99.50th=[50594], 99.90th=[55313], 99.95th=[56886],
     | 99.99th=[56886]
   bw (  KiB/s): min=  720, max=  944, per=25.15%, avg=853.27, stdev=67.51, samples=11
   iops        : min=  180, max=  236, avg=213.27, stdev=16.86, samples=11
  write: IOPS=219, BW=879KiB/s (900kB/s)(5192KiB/5908msec); 0 zone resets
    slat (usec): min=44, max=2690, avg=158.10, stdev=104.91
    clat (usec): min=24880, max=57312, avg=35766.81, stdev=4667.98
     lat (usec): min=24946, max=57625, avg=35925.97, stdev=4676.11
    clat percentiles (usec):
     |  1.00th=[26870],  5.00th=[28967], 10.00th=[30540], 20.00th=[31851],
     | 30.00th=[32900], 40.00th=[34341], 50.00th=[35390], 60.00th=[36439],
     | 70.00th=[38011], 80.00th=[39060], 90.00th=[41681], 95.00th=[43779],
     | 99.00th=[50594], 99.50th=[52691], 99.90th=[54789], 99.95th=[57410],
     | 99.99th=[57410]
   bw (  KiB/s): min=  736, max= 1048, per=24.95%, avg=881.27, stdev=92.55, samples=11
   iops        : min=  184, max=  262, avg=220.27, stdev=23.18, samples=11
  lat (msec)   : 10=0.08%, 50=98.83%, 100=1.09%
  cpu          : usr=0.49%, sys=6.21%, ctx=4643, majf=0, minf=14
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=182308: Fri Jul  3 06:44:06 2026
  read: IOPS=211, BW=844KiB/s (865kB/s)(4976KiB/5894msec)
    slat (usec): min=40, max=9033, avg=2482.92, stdev=783.12
    clat (usec): min=4379, max=55251, avg=34999.61, stdev=5364.30
     lat (usec): min=6145, max=57719, avg=37484.32, stdev=5539.01
    clat percentiles (usec):
     |  1.00th=[23987],  5.00th=[27132], 10.00th=[28705], 20.00th=[30540],
     | 30.00th=[32113], 40.00th=[33424], 50.00th=[34866], 60.00th=[35914],
     | 70.00th=[37487], 80.00th=[39060], 90.00th=[41681], 95.00th=[43779],
     | 99.00th=[50070], 99.50th=[51643], 99.90th=[54264], 99.95th=[55313],
     | 99.99th=[55313]
   bw (  KiB/s): min=  723, max=  936, per=24.80%, avg=841.55, stdev=76.40, samples=11
   iops        : min=  180, max=  234, avg=210.27, stdev=19.24, samples=11
  write: IOPS=223, BW=893KiB/s (915kB/s)(5264KiB/5894msec); 0 zone resets
    slat (usec): min=41, max=790, avg=156.10, stdev=78.60
    clat (usec): min=13100, max=54849, avg=35747.37, stdev=4703.33
     lat (usec): min=13288, max=54961, avg=35904.51, stdev=4708.87
    clat percentiles (usec):
     |  1.00th=[26870],  5.00th=[28967], 10.00th=[30278], 20.00th=[31851],
     | 30.00th=[33162], 40.00th=[34341], 50.00th=[35390], 60.00th=[36439],
     | 70.00th=[38011], 80.00th=[39060], 90.00th=[41681], 95.00th=[44303],
     | 99.00th=[48497], 99.50th=[50070], 99.90th=[51643], 99.95th=[54789],
     | 99.99th=[54789]
   bw (  KiB/s): min=  824, max= 1128, per=25.38%, avg=896.36, stdev=86.30, samples=11
   iops        : min=  206, max=  282, avg=224.00, stdev=21.64, samples=11
  lat (msec)   : 10=0.08%, 20=0.23%, 50=98.75%, 100=0.94%
  cpu          : usr=0.37%, sys=6.35%, ctx=4598, majf=0, minf=12
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=182309: Fri Jul  3 06:44:06 2026
  read: IOPS=211, BW=845KiB/s (865kB/s)(4984KiB/5898msec)
    slat (usec): min=40, max=8908, avg=2465.85, stdev=833.43
    clat (usec): min=2628, max=58199, avg=35218.44, stdev=5419.57
     lat (usec): min=4482, max=60896, avg=37686.11, stdev=5536.97
    clat percentiles (usec):
     |  1.00th=[24249],  5.00th=[27132], 10.00th=[28443], 20.00th=[31065],
     | 30.00th=[32637], 40.00th=[33817], 50.00th=[34866], 60.00th=[36439],
     | 70.00th=[37487], 80.00th=[39060], 90.00th=[41681], 95.00th=[44303],
     | 99.00th=[51119], 99.50th=[53216], 99.90th=[57934], 99.95th=[58459],
     | 99.99th=[58459]
   bw (  KiB/s): min=  710, max=  976, per=25.03%, avg=849.36, stdev=83.81, samples=11
   iops        : min=  177, max=  244, avg=212.27, stdev=21.06, samples=11
  write: IOPS=222, BW=891KiB/s (913kB/s)(5256KiB/5898msec); 0 zone resets
    slat (usec): min=41, max=855, avg=158.05, stdev=76.37
    clat (usec): min=12940, max=52672, avg=35772.00, stdev=4816.63
     lat (usec): min=13221, max=52769, avg=35931.20, stdev=4823.69
    clat percentiles (usec):
     |  1.00th=[26084],  5.00th=[29230], 10.00th=[30278], 20.00th=[31851],
     | 30.00th=[32900], 40.00th=[34341], 50.00th=[35390], 60.00th=[36439],
     | 70.00th=[38011], 80.00th=[39584], 90.00th=[42206], 95.00th=[44827],
     | 99.00th=[49021], 99.50th=[49546], 99.90th=[51119], 99.95th=[52691],
     | 99.99th=[52691]
   bw (  KiB/s): min=  760, max= 1048, per=25.10%, avg=886.45, stdev=99.67, samples=11
   iops        : min=  190, max=  262, avg=221.55, stdev=24.89, samples=11
  lat (msec)   : 4=0.04%, 10=0.04%, 20=0.08%, 50=99.06%, 100=0.78%
  cpu          : usr=0.42%, sys=6.34%, ctx=4543, majf=0, minf=14
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=182310: Fri Jul  3 06:44:06 2026
  read: IOPS=213, BW=856KiB/s (876kB/s)(5060KiB/5914msec)
    slat (usec): min=951, max=8735, avg=2457.02, stdev=783.43
    clat (usec): min=2867, max=57730, avg=35301.31, stdev=5795.34
     lat (usec): min=6603, max=59777, avg=37760.48, stdev=5928.56
    clat percentiles (usec):
     |  1.00th=[20579],  5.00th=[27132], 10.00th=[28705], 20.00th=[31065],
     | 30.00th=[32375], 40.00th=[33817], 50.00th=[34866], 60.00th=[36439],
     | 70.00th=[38011], 80.00th=[39584], 90.00th=[42206], 95.00th=[44827],
     | 99.00th=[50594], 99.50th=[52691], 99.90th=[56361], 99.95th=[57934],
     | 99.99th=[57934]
   bw (  KiB/s): min=  710, max=  968, per=25.00%, avg=848.36, stdev=68.00, samples=11
   iops        : min=  177, max=  242, avg=212.00, stdev=17.14, samples=11
  write: IOPS=218, BW=876KiB/s (897kB/s)(5180KiB/5914msec); 0 zone resets
    slat (usec): min=55, max=3214, avg=164.42, stdev=114.99
    clat (usec): min=6608, max=57191, avg=35471.94, stdev=4747.57
     lat (usec): min=6763, max=57314, avg=35637.37, stdev=4754.80
    clat percentiles (usec):
     |  1.00th=[26084],  5.00th=[29230], 10.00th=[30278], 20.00th=[31851],
     | 30.00th=[32900], 40.00th=[33817], 50.00th=[35390], 60.00th=[36439],
     | 70.00th=[37487], 80.00th=[39060], 90.00th=[41157], 95.00th=[43254],
     | 99.00th=[48497], 99.50th=[49546], 99.90th=[54264], 99.95th=[57410],
     | 99.99th=[57410]
   bw (  KiB/s): min=  729, max=  952, per=24.73%, avg=873.36, stdev=65.69, samples=11
   iops        : min=  182, max=  238, avg=218.27, stdev=16.47, samples=11
  lat (msec)   : 4=0.04%, 10=0.35%, 20=0.23%, 50=98.63%, 100=0.74%
  cpu          : usr=0.44%, sys=6.31%, ctx=4703, majf=0, minf=13
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=3393KiB/s (3475kB/s), 844KiB/s-856KiB/s (865kB/s-876kB/s), io=19.6MiB (20.5MB), run=5894-5914msec
  WRITE: bw=3533KiB/s (3617kB/s), 876KiB/s-893KiB/s (897kB/s-915kB/s), io=20.4MiB (21.4MB), run=5894-5914msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.19
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=182315: Fri Jul  3 06:44:09 2026
  read: IOPS=9, BW=388MiB/s (407MB/s)(40.0MiB/103msec)
    slat (nsec): min=7354.8k, max=7354.8k, avg=7354811.00, stdev= 0.00
    clat (nsec): min=90602k, max=90602k, avg=90601883.00, stdev= 0.00
     lat (nsec): min=97960k, max=97960k, avg=97960233.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[90702],  5.00th=[90702], 10.00th=[90702], 20.00th=[90702],
     | 30.00th=[90702], 40.00th=[90702], 50.00th=[90702], 60.00th=[90702],
     | 70.00th=[90702], 80.00th=[90702], 90.00th=[90702], 95.00th=[90702],
     | 99.00th=[90702], 99.50th=[90702], 99.90th=[90702], 99.95th=[90702],
     | 99.99th=[90702]
  lat (msec)   : 100=100.00%
  cpu          : usr=0.00%, sys=7.84%, ctx=3, majf=0, minf=9
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=388MiB/s (407MB/s), 388MiB/s-388MiB/s (407MB/s-407MB/s), io=40.0MiB (41.9MB), run=103-103msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout