== sanity test 398c: run fio to test AIO ================= 09:25:56 (1788355556)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB, 40 MiB) copied, 0.130291 s, 322 MB/s
osc.lustre-OST0000-osc-ffff9106c7b45800.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=192517: Wed Sep  2 09:26:04 2026
  write: IOPS=518, BW=2073KiB/s (2123kB/s)(10.0MiB/4940msec); 0 zone resets
    slat (usec): min=41, max=804, avg=180.21, stdev=101.38
    clat (usec): min=3927, max=48438, avg=30317.49, stdev=2773.53
     lat (usec): min=4205, max=48801, avg=30498.88, stdev=2775.05
    clat percentiles (usec):
     |  1.00th=[22938],  5.00th=[25822], 10.00th=[27395], 20.00th=[28443],
     | 30.00th=[28967], 40.00th=[29754], 50.00th=[30540], 60.00th=[31065],
     | 70.00th=[31589], 80.00th=[32375], 90.00th=[33424], 95.00th=[34341],
     | 99.00th=[38011], 99.50th=[39060], 99.90th=[45876], 99.95th=[45876],
     | 99.99th=[48497]
   bw (  KiB/s): min= 1976, max= 2208, per=24.99%, avg=2071.78, stdev=73.11, samples=9
   iops        : min=  494, max=  552, avg=517.89, stdev=18.31, samples=9
  lat (msec)   : 4=0.04%, 10=0.04%, 20=0.04%, 50=99.88%
  cpu          : usr=0.49%, sys=6.68%, ctx=3114, majf=0, minf=8
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=192518: Wed Sep  2 09:26:04 2026
  write: IOPS=519, BW=2078KiB/s (2128kB/s)(10.0MiB/4927msec); 0 zone resets
    slat (usec): min=51, max=25987, avg=196.56, stdev=518.32
    clat (usec): min=5092, max=52976, avg=30290.19, stdev=2918.94
     lat (usec): min=5330, max=53581, avg=30487.87, stdev=2930.05
    clat percentiles (usec):
     |  1.00th=[22676],  5.00th=[26084], 10.00th=[27132], 20.00th=[28181],
     | 30.00th=[28967], 40.00th=[29754], 50.00th=[30278], 60.00th=[30802],
     | 70.00th=[31589], 80.00th=[32113], 90.00th=[33424], 95.00th=[34341],
     | 99.00th=[38011], 99.50th=[39584], 99.90th=[50070], 99.95th=[50594],
     | 99.99th=[53216]
   bw (  KiB/s): min= 2000, max= 2232, per=25.04%, avg=2075.78, stdev=68.15, samples=9
   iops        : min=  500, max=  558, avg=518.89, stdev=17.03, samples=9
  lat (msec)   : 10=0.04%, 20=0.31%, 50=99.49%, 100=0.16%
  cpu          : usr=0.41%, sys=7.75%, ctx=3207, majf=0, minf=9
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=192519: Wed Sep  2 09:26:04 2026
  write: IOPS=518, BW=2074KiB/s (2123kB/s)(10.0MiB/4938msec); 0 zone resets
    slat (usec): min=49, max=780, avg=185.96, stdev=94.76
    clat (usec): min=19777, max=55188, avg=30342.22, stdev=2662.11
     lat (usec): min=20028, max=55420, avg=30529.32, stdev=2670.05
    clat percentiles (usec):
     |  1.00th=[23725],  5.00th=[26084], 10.00th=[27395], 20.00th=[28443],
     | 30.00th=[28967], 40.00th=[29754], 50.00th=[30540], 60.00th=[31065],
     | 70.00th=[31589], 80.00th=[32113], 90.00th=[33424], 95.00th=[34341],
     | 99.00th=[38011], 99.50th=[39584], 99.90th=[49546], 99.95th=[54789],
     | 99.99th=[55313]
   bw (  KiB/s): min= 1992, max= 2232, per=25.00%, avg=2072.67, stdev=72.61, samples=9
   iops        : min=  498, max=  558, avg=518.11, stdev=18.18, samples=9
  lat (msec)   : 20=0.04%, 50=99.88%, 100=0.08%
  cpu          : usr=0.55%, sys=7.51%, ctx=3044, majf=0, minf=10
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=192520: Wed Sep  2 09:26:04 2026
  write: IOPS=522, BW=2090KiB/s (2140kB/s)(10.0MiB/4899msec); 0 zone resets
    slat (usec): min=51, max=718, avg=186.48, stdev=98.42
    clat (usec): min=4521, max=52910, avg=30378.08, stdev=3142.49
     lat (usec): min=4719, max=53065, avg=30565.78, stdev=3143.21
    clat percentiles (usec):
     |  1.00th=[22414],  5.00th=[25822], 10.00th=[27132], 20.00th=[28181],
     | 30.00th=[29230], 40.00th=[29754], 50.00th=[30540], 60.00th=[31065],
     | 70.00th=[31589], 80.00th=[32375], 90.00th=[33424], 95.00th=[34866],
     | 99.00th=[39060], 99.50th=[45876], 99.90th=[51643], 99.95th=[51643],
     | 99.99th=[52691]
   bw (  KiB/s): min= 1968, max= 2240, per=25.17%, avg=2086.44, stdev=85.94, samples=9
   iops        : min=  492, max=  560, avg=521.56, stdev=21.42, samples=9
  lat (msec)   : 10=0.08%, 20=0.35%, 50=99.26%, 100=0.31%
  cpu          : usr=0.45%, sys=7.16%, ctx=3079, majf=0, minf=8
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=8291KiB/s (8490kB/s), 2073KiB/s-2090KiB/s (2123kB/s-2140kB/s), io=40.0MiB (41.9MB), run=4899-4940msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=192530: Wed Sep  2 09:26:13 2026
  read: IOPS=210, BW=840KiB/s (860kB/s)(5048KiB/6008msec)
    slat (usec): min=38, max=5852, avg=2511.92, stdev=763.37
    clat (usec): min=20905, max=61009, avg=35555.46, stdev=5448.04
     lat (usec): min=22318, max=64493, avg=38068.93, stdev=5656.82
    clat percentiles (usec):
     |  1.00th=[24249],  5.00th=[27395], 10.00th=[28967], 20.00th=[31065],
     | 30.00th=[32637], 40.00th=[33817], 50.00th=[35390], 60.00th=[36439],
     | 70.00th=[38011], 80.00th=[39584], 90.00th=[42206], 95.00th=[45351],
     | 99.00th=[51119], 99.50th=[52167], 99.90th=[57934], 99.95th=[61080],
     | 99.99th=[61080]
   bw (  KiB/s): min=  672, max= 1056, per=24.80%, avg=827.91, stdev=96.61, samples=11
   iops        : min=  168, max=  264, avg=206.91, stdev=24.17, samples=11
  write: IOPS=216, BW=864KiB/s (885kB/s)(5192KiB/6008msec); 0 zone resets
    slat (usec): min=40, max=3262, avg=168.26, stdev=119.90
    clat (usec): min=14298, max=60341, avg=36429.95, stdev=5048.28
     lat (usec): min=14372, max=60409, avg=36599.31, stdev=5057.57
    clat percentiles (usec):
     |  1.00th=[26608],  5.00th=[28967], 10.00th=[30540], 20.00th=[32375],
     | 30.00th=[33817], 40.00th=[34866], 50.00th=[35914], 60.00th=[37487],
     | 70.00th=[38536], 80.00th=[39584], 90.00th=[42206], 95.00th=[44827],
     | 99.00th=[51643], 99.50th=[54264], 99.90th=[59507], 99.95th=[60556],
     | 99.99th=[60556]
   bw (  KiB/s): min=  759, max= 1024, per=24.86%, avg=863.91, stdev=85.99, samples=11
   iops        : min=  189, max=  256, avg=215.91, stdev=21.59, samples=11
  lat (msec)   : 20=0.04%, 50=98.36%, 100=1.60%
  cpu          : usr=0.60%, sys=6.21%, ctx=4560, majf=0, minf=11
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=192531: Wed Sep  2 09:26:13 2026
  read: IOPS=206, BW=828KiB/s (848kB/s)(4976KiB/6010msec)
    slat (usec): min=89, max=5569, avg=2565.80, stdev=744.03
    clat (usec): min=14459, max=56940, avg=35637.40, stdev=5715.00
     lat (usec): min=17098, max=60753, avg=38204.76, stdev=5889.77
    clat percentiles (usec):
     |  1.00th=[24249],  5.00th=[27132], 10.00th=[28967], 20.00th=[30802],
     | 30.00th=[32375], 40.00th=[33817], 50.00th=[35390], 60.00th=[36963],
     | 70.00th=[38536], 80.00th=[40109], 90.00th=[42730], 95.00th=[45876],
     | 99.00th=[52167], 99.50th=[53740], 99.90th=[56361], 99.95th=[56886],
     | 99.99th=[56886]
   bw (  KiB/s): min=  656, max=  952, per=24.35%, avg=812.73, stdev=82.81, samples=11
   iops        : min=  164, max=  238, avg=203.18, stdev=20.70, samples=11
  write: IOPS=218, BW=876KiB/s (897kB/s)(5264KiB/6010msec); 0 zone resets
    slat (usec): min=56, max=2806, avg=168.35, stdev=109.04
    clat (usec): min=12380, max=55644, avg=36382.87, stdev=5158.46
     lat (usec): min=12489, max=55803, avg=36552.24, stdev=5167.94
    clat percentiles (usec):
     |  1.00th=[24773],  5.00th=[28967], 10.00th=[30540], 20.00th=[32637],
     | 30.00th=[33817], 40.00th=[34866], 50.00th=[35914], 60.00th=[37487],
     | 70.00th=[38536], 80.00th=[40109], 90.00th=[42730], 95.00th=[45351],
     | 99.00th=[50594], 99.50th=[52691], 99.90th=[55313], 99.95th=[55837],
     | 99.99th=[55837]
   bw (  KiB/s): min=  744, max= 1072, per=25.23%, avg=876.82, stdev=90.65, samples=11
   iops        : min=  186, max=  268, avg=219.18, stdev=22.67, samples=11
  lat (msec)   : 20=0.27%, 50=98.16%, 100=1.56%
  cpu          : usr=0.47%, sys=6.37%, ctx=4583, majf=0, minf=14
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=192532: Wed Sep  2 09:26:13 2026
  read: IOPS=208, BW=833KiB/s (853kB/s)(4984KiB/5982msec)
    slat (usec): min=40, max=8875, avg=2576.09, stdev=833.70
    clat (usec): min=2908, max=57112, avg=35604.34, stdev=6108.35
     lat (usec): min=5054, max=62163, avg=38181.96, stdev=6353.95
    clat percentiles (usec):
     |  1.00th=[19530],  5.00th=[27132], 10.00th=[28967], 20.00th=[31065],
     | 30.00th=[32900], 40.00th=[34341], 50.00th=[35390], 60.00th=[36963],
     | 70.00th=[38011], 80.00th=[39584], 90.00th=[42206], 95.00th=[45351],
     | 99.00th=[53216], 99.50th=[54789], 99.90th=[56886], 99.95th=[56886],
     | 99.99th=[56886]
   bw (  KiB/s): min=  672, max= 1032, per=24.91%, avg=831.55, stdev=99.63, samples=11
   iops        : min=  168, max=  258, avg=207.82, stdev=24.86, samples=11
  write: IOPS=219, BW=879KiB/s (900kB/s)(5256KiB/5982msec); 0 zone resets
    slat (usec): min=39, max=729, avg=170.21, stdev=83.27
    clat (usec): min=12417, max=60110, avg=36338.99, stdev=5404.28
     lat (usec): min=12549, max=60198, avg=36510.34, stdev=5409.61
    clat percentiles (usec):
     |  1.00th=[22938],  5.00th=[28705], 10.00th=[30278], 20.00th=[32637],
     | 30.00th=[33817], 40.00th=[34866], 50.00th=[35914], 60.00th=[36963],
     | 70.00th=[38536], 80.00th=[40109], 90.00th=[42730], 95.00th=[45876],
     | 99.00th=[51643], 99.50th=[52167], 99.90th=[60031], 99.95th=[60031],
     | 99.99th=[60031]
   bw (  KiB/s): min=  736, max= 1032, per=24.98%, avg=868.18, stdev=84.57, samples=11
   iops        : min=  184, max=  258, avg=217.00, stdev=21.15, samples=11
  lat (msec)   : 4=0.04%, 10=0.16%, 20=0.70%, 50=97.15%, 100=1.95%
  cpu          : usr=0.55%, sys=6.29%, ctx=4501, majf=0, minf=13
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=192533: Wed Sep  2 09:26:13 2026
  read: IOPS=210, BW=842KiB/s (862kB/s)(5060KiB/6011msec)
    slat (usec): min=969, max=8906, avg=2573.48, stdev=799.27
    clat (usec): min=6206, max=62781, avg=35515.58, stdev=5840.29
     lat (usec): min=12433, max=66200, avg=38090.56, stdev=6045.19
    clat percentiles (usec):
     |  1.00th=[22938],  5.00th=[26870], 10.00th=[28967], 20.00th=[31065],
     | 30.00th=[32375], 40.00th=[33817], 50.00th=[35390], 60.00th=[36439],
     | 70.00th=[38011], 80.00th=[39584], 90.00th=[42206], 95.00th=[45351],
     | 99.00th=[54789], 99.50th=[56361], 99.90th=[61604], 99.95th=[62653],
     | 99.99th=[62653]
   bw (  KiB/s): min=  736, max= 1032, per=24.97%, avg=833.64, stdev=81.12, samples=11
   iops        : min=  184, max=  258, avg=208.36, stdev=20.33, samples=11
  write: IOPS=215, BW=862KiB/s (882kB/s)(5180KiB/6011msec); 0 zone resets
    slat (usec): min=51, max=861, avg=170.40, stdev=84.95
    clat (usec): min=12710, max=57592, avg=36278.42, stdev=5005.02
     lat (usec): min=12881, max=57717, avg=36449.98, stdev=5014.23
    clat percentiles (usec):
     |  1.00th=[25822],  5.00th=[28967], 10.00th=[30540], 20.00th=[32113],
     | 30.00th=[33817], 40.00th=[34866], 50.00th=[35914], 60.00th=[36963],
     | 70.00th=[38011], 80.00th=[39584], 90.00th=[42206], 95.00th=[44827],
     | 99.00th=[51643], 99.50th=[52691], 99.90th=[56361], 99.95th=[57410],
     | 99.99th=[57410]
   bw (  KiB/s): min=  720, max=  992, per=24.58%, avg=854.09, stdev=77.49, samples=11
   iops        : min=  180, max=  248, avg=213.45, stdev=19.45, samples=11
  lat (msec)   : 10=0.04%, 20=0.27%, 50=97.97%, 100=1.72%
  cpu          : usr=0.55%, sys=6.39%, ctx=4691, majf=0, minf=12
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=3339KiB/s (3419kB/s), 828KiB/s-842KiB/s (848kB/s-862kB/s), io=19.6MiB (20.5MB), run=5982-6011msec
  WRITE: bw=3476KiB/s (3559kB/s), 862KiB/s-879KiB/s (882kB/s-900kB/s), io=20.4MiB (21.4MB), run=5982-6011msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.19
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=192537: Wed Sep  2 09:26:16 2026
  read: IOPS=10, BW=426MiB/s (446MB/s)(40.0MiB/94msec)
    slat (nsec): min=7742.5k, max=7742.5k, avg=7742479.00, stdev= 0.00
    clat (nsec): min=81259k, max=81259k, avg=81259079.00, stdev= 0.00
     lat (nsec): min=89005k, max=89005k, avg=89004810.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[81265],  5.00th=[81265], 10.00th=[81265], 20.00th=[81265],
     | 30.00th=[81265], 40.00th=[81265], 50.00th=[81265], 60.00th=[81265],
     | 70.00th=[81265], 80.00th=[81265], 90.00th=[81265], 95.00th=[81265],
     | 99.00th=[81265], 99.50th=[81265], 99.90th=[81265], 99.95th=[81265],
     | 99.99th=[81265]
  lat (msec)   : 100=100.00%
  cpu          : usr=0.00%, sys=8.60%, ctx=4, majf=0, minf=9
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=426MiB/s (446MB/s), 426MiB/s-426MiB/s (446MB/s-446MB/s), io=40.0MiB (41.9MB), run=94-94msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout