== sanity test 398c: run fio to test AIO ================= 11:35:48 (1786116948)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB, 40 MiB) copied, 0.125036 s, 335 MB/s
osc.lustre-OST0000-osc-ffff995084dfc800.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=186401: Fri Aug  7 11:35:57 2026
  write: IOPS=473, BW=1895KiB/s (1940kB/s)(10.0MiB/5405msec); 0 zone resets
    slat (usec): min=43, max=766, avg=203.40, stdev=104.01
    clat (usec): min=9284, max=62721, avg=33511.76, stdev=4089.14
     lat (usec): min=9460, max=62821, avg=33716.34, stdev=4097.37
    clat percentiles (usec):
     |  1.00th=[23200],  5.00th=[27657], 10.00th=[29492], 20.00th=[30802],
     | 30.00th=[31589], 40.00th=[32375], 50.00th=[33162], 60.00th=[34341],
     | 70.00th=[35390], 80.00th=[35914], 90.00th=[38011], 95.00th=[40109],
     | 99.00th=[43779], 99.50th=[45351], 99.90th=[56886], 99.95th=[56886],
     | 99.99th=[62653]
   bw (  KiB/s): min= 1720, max= 2048, per=25.11%, avg=1888.40, stdev=107.90, samples=10
   iops        : min=  430, max=  512, avg=472.10, stdev=26.98, samples=10
  lat (msec)   : 10=0.08%, 20=0.08%, 50=99.38%, 100=0.47%
  cpu          : usr=0.56%, sys=7.12%, ctx=2656, majf=0, minf=9
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=186402: Fri Aug  7 11:35:57 2026
  write: IOPS=470, BW=1881KiB/s (1926kB/s)(10.0MiB/5445msec); 0 zone resets
    slat (usec): min=53, max=1597, avg=197.45, stdev=104.71
    clat (usec): min=4057, max=48275, avg=33483.79, stdev=3654.54
     lat (usec): min=4312, max=48357, avg=33682.28, stdev=3658.07
    clat percentiles (usec):
     |  1.00th=[24249],  5.00th=[28443], 10.00th=[29492], 20.00th=[30802],
     | 30.00th=[31589], 40.00th=[32375], 50.00th=[33162], 60.00th=[34341],
     | 70.00th=[35390], 80.00th=[35914], 90.00th=[38011], 95.00th=[39584],
     | 99.00th=[43254], 99.50th=[44303], 99.90th=[45351], 99.95th=[45351],
     | 99.99th=[48497]
   bw (  KiB/s): min= 1728, max= 2040, per=24.89%, avg=1872.40, stdev=101.10, samples=10
   iops        : min=  432, max=  510, avg=468.10, stdev=25.27, samples=10
  lat (msec)   : 10=0.04%, 50=99.96%
  cpu          : usr=0.44%, sys=7.37%, ctx=2718, majf=0, minf=11
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=186403: Fri Aug  7 11:35:57 2026
  write: IOPS=470, BW=1883KiB/s (1929kB/s)(10.0MiB/5437msec); 0 zone resets
    slat (usec): min=52, max=25341, avg=194.85, stdev=507.18
    clat (usec): min=4885, max=53057, avg=33468.34, stdev=3893.27
     lat (usec): min=4989, max=53255, avg=33664.30, stdev=3870.81
    clat percentiles (usec):
     |  1.00th=[23200],  5.00th=[27657], 10.00th=[29492], 20.00th=[30802],
     | 30.00th=[31589], 40.00th=[32637], 50.00th=[33162], 60.00th=[34341],
     | 70.00th=[34866], 80.00th=[35914], 90.00th=[38011], 95.00th=[40109],
     | 99.00th=[43779], 99.50th=[44303], 99.90th=[52691], 99.95th=[52691],
     | 99.99th=[53216]
   bw (  KiB/s): min= 1696, max= 2016, per=24.96%, avg=1877.50, stdev=107.77, samples=10
   iops        : min=  424, max=  504, avg=469.30, stdev=26.92, samples=10
  lat (msec)   : 10=0.08%, 20=0.08%, 50=99.65%, 100=0.20%
  cpu          : usr=0.53%, sys=6.88%, ctx=2811, majf=0, minf=11
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=186404: Fri Aug  7 11:35:57 2026
  write: IOPS=470, BW=1881KiB/s (1926kB/s)(10.0MiB/5443msec); 0 zone resets
    slat (usec): min=42, max=742, avg=194.58, stdev=99.67
    clat (usec): min=20384, max=51877, avg=33443.53, stdev=3601.97
     lat (usec): min=20884, max=51975, avg=33639.28, stdev=3614.44
    clat percentiles (usec):
     |  1.00th=[24249],  5.00th=[27919], 10.00th=[29492], 20.00th=[30802],
     | 30.00th=[31589], 40.00th=[32375], 50.00th=[33162], 60.00th=[34341],
     | 70.00th=[35390], 80.00th=[35914], 90.00th=[38011], 95.00th=[39584],
     | 99.00th=[43254], 99.50th=[43779], 99.90th=[45876], 99.95th=[48497],
     | 99.99th=[51643]
   bw (  KiB/s): min= 1712, max= 2032, per=24.93%, avg=1875.00, stdev=109.18, samples=10
   iops        : min=  428, max=  508, avg=468.70, stdev=27.31, samples=10
  lat (msec)   : 50=99.96%, 100=0.04%
  cpu          : usr=0.59%, sys=6.76%, ctx=2769, majf=0, minf=10
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=7522KiB/s (7703kB/s), 1881KiB/s-1895KiB/s (1926kB/s-1940kB/s), io=40.0MiB (41.9MB), run=5405-5445msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=186414: Fri Aug  7 11:36:06 2026
  read: IOPS=208, BW=833KiB/s (853kB/s)(5048KiB/6058msec)
    slat (usec): min=1114, max=6210, avg=2537.89, stdev=675.20
    clat (usec): min=19544, max=52412, avg=35881.33, stdev=4459.87
     lat (usec): min=22583, max=54483, avg=38420.90, stdev=4514.81
    clat percentiles (usec):
     |  1.00th=[26084],  5.00th=[28705], 10.00th=[30278], 20.00th=[32375],
     | 30.00th=[33424], 40.00th=[34866], 50.00th=[35914], 60.00th=[36963],
     | 70.00th=[38011], 80.00th=[39584], 90.00th=[41157], 95.00th=[43254],
     | 99.00th=[47449], 99.50th=[49546], 99.90th=[51119], 99.95th=[52167],
     | 99.99th=[52167]
   bw (  KiB/s): min=  760, max=  880, per=25.00%, avg=827.91, stdev=38.72, samples=11
   iops        : min=  190, max=  220, avg=206.91, stdev= 9.73, samples=11
  write: IOPS=214, BW=857KiB/s (878kB/s)(5192KiB/6058msec); 0 zone resets
    slat (usec): min=55, max=900, avg=170.82, stdev=87.08
    clat (usec): min=20740, max=47754, avg=36404.72, stdev=3469.89
     lat (usec): min=20853, max=47872, avg=36576.69, stdev=3467.16
    clat percentiles (usec):
     |  1.00th=[28443],  5.00th=[30802], 10.00th=[32113], 20.00th=[33817],
     | 30.00th=[34341], 40.00th=[35390], 50.00th=[36439], 60.00th=[37487],
     | 70.00th=[38011], 80.00th=[39060], 90.00th=[40633], 95.00th=[42206],
     | 99.00th=[44827], 99.50th=[45351], 99.90th=[47449], 99.95th=[47973],
     | 99.99th=[47973]
   bw (  KiB/s): min=  759, max= 1024, per=25.08%, avg=864.64, stdev=79.45, samples=11
   iops        : min=  189, max=  256, avg=216.09, stdev=19.96, samples=11
  lat (msec)   : 20=0.04%, 50=99.88%, 100=0.08%
  cpu          : usr=0.51%, sys=6.36%, ctx=4760, majf=0, minf=15
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=186415: Fri Aug  7 11:36:06 2026
  read: IOPS=205, BW=822KiB/s (842kB/s)(4976KiB/6051msec)
    slat (usec): min=1162, max=5533, avg=2555.04, stdev=673.06
    clat (usec): min=19793, max=55563, avg=36040.20, stdev=4505.13
     lat (usec): min=22990, max=58797, avg=38596.76, stdev=4500.81
    clat percentiles (usec):
     |  1.00th=[26346],  5.00th=[28967], 10.00th=[30278], 20.00th=[32113],
     | 30.00th=[33817], 40.00th=[34866], 50.00th=[35914], 60.00th=[36963],
     | 70.00th=[38536], 80.00th=[39584], 90.00th=[41681], 95.00th=[43254],
     | 99.00th=[47449], 99.50th=[49021], 99.90th=[53740], 99.95th=[55313],
     | 99.99th=[55313]
   bw (  KiB/s): min=  740, max=  888, per=24.65%, avg=816.36, stdev=45.83, samples=11
   iops        : min=  185, max=  222, avg=204.09, stdev=11.46, samples=11
  write: IOPS=217, BW=870KiB/s (891kB/s)(5264KiB/6051msec); 0 zone resets
    slat (usec): min=58, max=1874, avg=168.22, stdev=92.97
    clat (usec): min=25047, max=49808, avg=36421.09, stdev=3519.14
     lat (usec): min=25256, max=49920, avg=36590.40, stdev=3523.94
    clat percentiles (usec):
     |  1.00th=[28443],  5.00th=[30802], 10.00th=[31851], 20.00th=[33424],
     | 30.00th=[34341], 40.00th=[35390], 50.00th=[36439], 60.00th=[36963],
     | 70.00th=[38011], 80.00th=[39584], 90.00th=[41157], 95.00th=[42206],
     | 99.00th=[45351], 99.50th=[45876], 99.90th=[46924], 99.95th=[50070],
     | 99.99th=[50070]
   bw (  KiB/s): min=  800, max=  976, per=25.47%, avg=878.09, stdev=52.64, samples=11
   iops        : min=  200, max=  244, avg=219.45, stdev=13.18, samples=11
  lat (msec)   : 20=0.04%, 50=99.80%, 100=0.16%
  cpu          : usr=0.46%, sys=6.33%, ctx=4860, majf=0, minf=14
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=186416: Fri Aug  7 11:36:06 2026
  read: IOPS=205, BW=823KiB/s (843kB/s)(4984KiB/6054msec)
    slat (usec): min=1118, max=5244, avg=2574.09, stdev=685.45
    clat (usec): min=16312, max=60407, avg=36107.45, stdev=4413.26
     lat (usec): min=18853, max=63133, avg=38682.87, stdev=4417.98
    clat percentiles (usec):
     |  1.00th=[26870],  5.00th=[29492], 10.00th=[30540], 20.00th=[32375],
     | 30.00th=[33817], 40.00th=[34866], 50.00th=[35914], 60.00th=[36963],
     | 70.00th=[38536], 80.00th=[39584], 90.00th=[41681], 95.00th=[43254],
     | 99.00th=[47449], 99.50th=[48497], 99.90th=[58459], 99.95th=[60556],
     | 99.99th=[60556]
   bw (  KiB/s): min=  720, max=  920, per=24.99%, avg=827.82, stdev=72.16, samples=11
   iops        : min=  180, max=  230, avg=206.91, stdev=18.07, samples=11
  write: IOPS=217, BW=868KiB/s (889kB/s)(5256KiB/6054msec); 0 zone resets
    slat (usec): min=56, max=632, avg=172.03, stdev=81.69
    clat (usec): min=24841, max=58131, avg=36390.21, stdev=3529.30
     lat (usec): min=25023, max=58209, avg=36563.38, stdev=3532.31
    clat percentiles (usec):
     |  1.00th=[28967],  5.00th=[30540], 10.00th=[31851], 20.00th=[33424],
     | 30.00th=[34341], 40.00th=[35390], 50.00th=[36439], 60.00th=[36963],
     | 70.00th=[38011], 80.00th=[39060], 90.00th=[40633], 95.00th=[42206],
     | 99.00th=[45351], 99.50th=[45876], 99.90th=[54789], 99.95th=[57934],
     | 99.99th=[57934]
   bw (  KiB/s): min=  792, max=  976, per=25.13%, avg=866.36, stdev=65.80, samples=11
   iops        : min=  198, max=  244, avg=216.55, stdev=16.49, samples=11
  lat (msec)   : 20=0.04%, 50=99.77%, 100=0.20%
  cpu          : usr=0.43%, sys=6.43%, ctx=4751, majf=0, minf=14
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=186417: Fri Aug  7 11:36:06 2026
  read: IOPS=209, BW=838KiB/s (858kB/s)(5060KiB/6037msec)
    slat (usec): min=98, max=6697, avg=2544.51, stdev=729.84
    clat (usec): min=5181, max=49777, avg=36093.82, stdev=4870.08
     lat (usec): min=5320, max=52175, avg=38639.66, stdev=4901.43
    clat percentiles (usec):
     |  1.00th=[25822],  5.00th=[28705], 10.00th=[30278], 20.00th=[32113],
     | 30.00th=[33817], 40.00th=[34866], 50.00th=[36439], 60.00th=[37487],
     | 70.00th=[38536], 80.00th=[40109], 90.00th=[42206], 95.00th=[43779],
     | 99.00th=[46400], 99.50th=[48497], 99.90th=[49021], 99.95th=[49546],
     | 99.99th=[49546]
   bw (  KiB/s): min=  792, max=  896, per=25.32%, avg=838.55, stdev=32.56, samples=11
   iops        : min=  198, max=  224, avg=209.64, stdev= 8.14, samples=11
  write: IOPS=214, BW=858KiB/s (879kB/s)(5180KiB/6037msec); 0 zone resets
    slat (usec): min=58, max=2747, avg=170.51, stdev=109.00
    clat (usec): min=18090, max=56387, avg=36519.62, stdev=3873.55
     lat (usec): min=18292, max=56537, avg=36691.16, stdev=3870.72
    clat percentiles (usec):
     |  1.00th=[28181],  5.00th=[30540], 10.00th=[31851], 20.00th=[33424],
     | 30.00th=[34341], 40.00th=[35390], 50.00th=[36439], 60.00th=[37487],
     | 70.00th=[38536], 80.00th=[40109], 90.00th=[41157], 95.00th=[42730],
     | 99.00th=[45876], 99.50th=[47973], 99.90th=[55837], 99.95th=[56361],
     | 99.99th=[56361]
   bw (  KiB/s): min=  800, max=  960, per=24.94%, avg=859.82, stdev=48.24, samples=11
   iops        : min=  200, max=  240, avg=214.91, stdev=12.08, samples=11
  lat (msec)   : 10=0.12%, 20=0.16%, 50=99.57%, 100=0.16%
  cpu          : usr=0.40%, sys=6.38%, ctx=4722, majf=0, minf=15
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=3313KiB/s (3392kB/s), 822KiB/s-838KiB/s (842kB/s-858kB/s), io=19.6MiB (20.5MB), run=6037-6058msec
  WRITE: bw=3449KiB/s (3531kB/s), 857KiB/s-870KiB/s (878kB/s-891kB/s), io=20.4MiB (21.4MB), run=6037-6058msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.19
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=186422: Fri Aug  7 11:36:10 2026
  read: IOPS=9, BW=381MiB/s (399MB/s)(40.0MiB/105msec)
    slat (nsec): min=8893.5k, max=8893.5k, avg=8893489.00, stdev= 0.00
    clat (nsec): min=91385k, max=91385k, avg=91385316.00, stdev= 0.00
     lat (nsec): min=100282k, max=100282k, avg=100281825.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[91751],  5.00th=[91751], 10.00th=[91751], 20.00th=[91751],
     | 30.00th=[91751], 40.00th=[91751], 50.00th=[91751], 60.00th=[91751],
     | 70.00th=[91751], 80.00th=[91751], 90.00th=[91751], 95.00th=[91751],
     | 99.00th=[91751], 99.50th=[91751], 99.90th=[91751], 99.95th=[91751],
     | 99.99th=[91751]
  lat (msec)   : 100=100.00%
  cpu          : usr=0.00%, sys=7.69%, ctx=8, majf=0, minf=9
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=381MiB/s (399MB/s), 381MiB/s-381MiB/s (399MB/s-399MB/s), io=40.0MiB (41.9MB), run=105-105msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout