== sanity test 398c: run fio to test AIO ================= 07:07:38 (1773659258)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB, 40 MiB) copied, 0.14523 s, 289 MB/s
osc.lustre-OST0000-osc-ffffa03384f93800.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=182880: Mon Mar 16 07:07:50 2026
  write: IOPS=321, BW=1286KiB/s (1317kB/s)(10.0MiB/7962msec); 0 zone resets
    slat (usec): min=38, max=3160, avg=183.86, stdev=150.47
    clat (usec): min=24812, max=79832, avg=49185.68, stdev=9566.30
     lat (usec): min=25045, max=80280, avg=49370.74, stdev=9579.82
    clat percentiles (usec):
     |  1.00th=[31851],  5.00th=[35914], 10.00th=[38011], 20.00th=[40633],
     | 30.00th=[42206], 40.00th=[44827], 50.00th=[49021], 60.00th=[51643],
     | 70.00th=[54264], 80.00th=[56886], 90.00th=[61080], 95.00th=[66847],
     | 99.00th=[74974], 99.50th=[77071], 99.90th=[79168], 99.95th=[79168],
     | 99.99th=[80217]
   bw (  KiB/s): min= 1024, max= 1520, per=24.91%, avg=1281.53, stdev=169.73, samples=15
   iops        : min=  256, max=  380, avg=320.33, stdev=42.39, samples=15
  lat (msec)   : 50=54.65%, 100=45.35%
  cpu          : usr=0.40%, sys=4.37%, ctx=2580, majf=0, minf=9
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=182881: Mon Mar 16 07:07:50 2026
  write: IOPS=324, BW=1299KiB/s (1330kB/s)(10.0MiB/7886msec); 0 zone resets
    slat (usec): min=39, max=2823, avg=199.01, stdev=141.06
    clat (usec): min=12498, max=99143, avg=49019.15, stdev=9951.69
     lat (usec): min=12667, max=99503, avg=49219.70, stdev=9966.54
    clat percentiles (usec):
     |  1.00th=[31851],  5.00th=[35914], 10.00th=[38011], 20.00th=[40109],
     | 30.00th=[42206], 40.00th=[44827], 50.00th=[47973], 60.00th=[51643],
     | 70.00th=[53740], 80.00th=[56886], 90.00th=[61604], 95.00th=[66847],
     | 99.00th=[76022], 99.50th=[80217], 99.90th=[87557], 99.95th=[93848],
     | 99.99th=[99091]
   bw (  KiB/s): min= 1056, max= 1565, per=25.24%, avg=1298.47, stdev=180.49, samples=15
   iops        : min=  264, max=  391, avg=324.60, stdev=45.10, samples=15
  lat (msec)   : 20=0.16%, 50=55.08%, 100=44.77%
  cpu          : usr=0.52%, sys=4.93%, ctx=2594, majf=0, minf=9
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=182882: Mon Mar 16 07:07:50 2026
  write: IOPS=323, BW=1292KiB/s (1323kB/s)(10.0MiB/7923msec); 0 zone resets
    slat (usec): min=37, max=3094, avg=172.65, stdev=138.44
    clat (usec): min=5417, max=79402, avg=49017.84, stdev=9891.55
     lat (usec): min=7626, max=79774, avg=49191.63, stdev=9899.73
    clat percentiles (usec):
     |  1.00th=[26870],  5.00th=[35390], 10.00th=[37487], 20.00th=[40633],
     | 30.00th=[42730], 40.00th=[44827], 50.00th=[47973], 60.00th=[51119],
     | 70.00th=[54264], 80.00th=[56361], 90.00th=[61604], 95.00th=[66847],
     | 99.00th=[74974], 99.50th=[77071], 99.90th=[79168], 99.95th=[79168],
     | 99.99th=[79168]
   bw (  KiB/s): min= 1040, max= 1528, per=25.03%, avg=1287.53, stdev=163.87, samples=15
   iops        : min=  260, max=  382, avg=321.87, stdev=40.95, samples=15
  lat (msec)   : 10=0.04%, 20=0.31%, 50=55.74%, 100=43.91%
  cpu          : usr=0.45%, sys=4.46%, ctx=2643, majf=0, minf=10
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=182883: Mon Mar 16 07:07:50 2026
  write: IOPS=323, BW=1294KiB/s (1325kB/s)(10.0MiB/7915msec); 0 zone resets
    slat (usec): min=43, max=20896, avg=227.10, stdev=458.75
    clat (usec): min=5442, max=99645, avg=48938.42, stdev=10257.45
     lat (usec): min=5780, max=99879, avg=49166.82, stdev=10245.09
    clat percentiles (msec):
     |  1.00th=[   27],  5.00th=[   36], 10.00th=[   38], 20.00th=[   41],
     | 30.00th=[   43], 40.00th=[   45], 50.00th=[   48], 60.00th=[   52],
     | 70.00th=[   54], 80.00th=[   57], 90.00th=[   62], 95.00th=[   67],
     | 99.00th=[   75], 99.50th=[   80], 99.90th=[  100], 99.95th=[  100],
     | 99.99th=[  101]
   bw (  KiB/s): min= 1048, max= 1544, per=25.17%, avg=1295.00, stdev=172.13, samples=15
   iops        : min=  262, max=  386, avg=323.73, stdev=43.01, samples=15
  lat (msec)   : 10=0.20%, 20=0.23%, 50=55.12%, 100=44.45%
  cpu          : usr=0.49%, sys=5.65%, ctx=2648, majf=0, minf=10
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=5144KiB/s (5268kB/s), 1286KiB/s-1299KiB/s (1317kB/s-1330kB/s), io=40.0MiB (41.9MB), run=7886-7962msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=182893: Mon Mar 16 07:08:01 2026
  read: IOPS=158, BW=634KiB/s (649kB/s)(5048KiB/7960msec)
    slat (usec): min=1277, max=11077, avg=3225.66, stdev=1143.69
    clat (usec): min=5837, max=77340, avg=47578.42, stdev=8635.20
     lat (usec): min=8335, max=81158, avg=50807.12, stdev=8869.76
    clat percentiles (usec):
     |  1.00th=[29754],  5.00th=[35390], 10.00th=[37487], 20.00th=[40633],
     | 30.00th=[42730], 40.00th=[44827], 50.00th=[46400], 60.00th=[49021],
     | 70.00th=[51119], 80.00th=[54789], 90.00th=[60031], 95.00th=[63177],
     | 99.00th=[70779], 99.50th=[71828], 99.90th=[74974], 99.95th=[77071],
     | 99.99th=[77071]
   bw (  KiB/s): min=  509, max=  728, per=25.37%, avg=639.60, stdev=60.32, samples=15
   iops        : min=  127, max=  182, avg=159.80, stdev=15.16, samples=15
  write: IOPS=163, BW=652KiB/s (668kB/s)(5192KiB/7960msec); 0 zone resets
    slat (usec): min=57, max=2082, avg=189.30, stdev=126.41
    clat (usec): min=13658, max=85260, avg=47905.58, stdev=7440.59
     lat (usec): min=14287, max=85363, avg=48096.10, stdev=7438.79
    clat percentiles (usec):
     |  1.00th=[34341],  5.00th=[36963], 10.00th=[39060], 20.00th=[41681],
     | 30.00th=[43779], 40.00th=[45876], 50.00th=[47449], 60.00th=[49021],
     | 70.00th=[51119], 80.00th=[53740], 90.00th=[57934], 95.00th=[61080],
     | 99.00th=[68682], 99.50th=[71828], 99.90th=[77071], 99.95th=[85459],
     | 99.99th=[85459]
   bw (  KiB/s): min=  536, max=  792, per=24.95%, avg=654.67, stdev=74.40, samples=15
   iops        : min=  134, max=  198, avg=163.53, stdev=18.64, samples=15
  lat (msec)   : 10=0.08%, 20=0.04%, 50=64.65%, 100=35.23%
  cpu          : usr=0.50%, sys=5.33%, ctx=4649, majf=0, minf=14
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=182894: Mon Mar 16 07:08:01 2026
  read: IOPS=156, BW=627KiB/s (642kB/s)(4976KiB/7940msec)
    slat (usec): min=1276, max=11492, avg=3230.79, stdev=1124.69
    clat (usec): min=10914, max=82875, avg=47552.27, stdev=8773.81
     lat (usec): min=17010, max=87183, avg=50787.54, stdev=8985.52
    clat percentiles (usec):
     |  1.00th=[30016],  5.00th=[34866], 10.00th=[37487], 20.00th=[40109],
     | 30.00th=[42206], 40.00th=[44303], 50.00th=[46400], 60.00th=[49021],
     | 70.00th=[51643], 80.00th=[54789], 90.00th=[58983], 95.00th=[63177],
     | 99.00th=[70779], 99.50th=[72877], 99.90th=[78119], 99.95th=[83362],
     | 99.99th=[83362]
   bw (  KiB/s): min=  568, max=  704, per=24.92%, avg=628.20, stdev=43.18, samples=15
   iops        : min=  142, max=  176, avg=156.93, stdev=10.87, samples=15
  write: IOPS=165, BW=663KiB/s (679kB/s)(5264KiB/7940msec); 0 zone resets
    slat (usec): min=38, max=3413, avg=182.52, stdev=157.02
    clat (usec): min=23951, max=82750, avg=47984.25, stdev=8170.23
     lat (usec): min=24079, max=82864, avg=48169.12, stdev=8177.82
    clat percentiles (usec):
     |  1.00th=[32900],  5.00th=[36963], 10.00th=[38536], 20.00th=[41157],
     | 30.00th=[43254], 40.00th=[44827], 50.00th=[46924], 60.00th=[49021],
     | 70.00th=[51643], 80.00th=[54264], 90.00th=[58983], 95.00th=[62653],
     | 99.00th=[69731], 99.50th=[73925], 99.90th=[78119], 99.95th=[82314],
     | 99.99th=[82314]
   bw (  KiB/s): min=  557, max=  848, per=25.48%, avg=668.60, stdev=82.53, samples=15
   iops        : min=  139, max=  212, avg=167.07, stdev=20.67, samples=15
  lat (msec)   : 20=0.04%, 50=63.63%, 100=36.33%
  cpu          : usr=0.44%, sys=5.35%, ctx=4620, majf=0, minf=13
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=182895: Mon Mar 16 07:08:01 2026
  read: IOPS=157, BW=630KiB/s (646kB/s)(4984KiB/7906msec)
    slat (usec): min=43, max=11310, avg=3166.12, stdev=1169.04
    clat (usec): min=5873, max=84775, avg=47124.82, stdev=9308.80
     lat (usec): min=5951, max=88615, avg=50293.92, stdev=9603.87
    clat percentiles (usec):
     |  1.00th=[28443],  5.00th=[33817], 10.00th=[36439], 20.00th=[39584],
     | 30.00th=[41681], 40.00th=[43779], 50.00th=[45876], 60.00th=[48497],
     | 70.00th=[51643], 80.00th=[55313], 90.00th=[59507], 95.00th=[63177],
     | 99.00th=[70779], 99.50th=[71828], 99.90th=[79168], 99.95th=[84411],
     | 99.99th=[84411]
   bw (  KiB/s): min=  488, max=  808, per=25.29%, avg=637.60, stdev=85.04, samples=15
   iops        : min=  122, max=  202, avg=159.27, stdev=21.35, samples=15
  write: IOPS=166, BW=665KiB/s (681kB/s)(5256KiB/7906msec); 0 zone resets
    slat (usec): min=40, max=2163, avg=192.62, stdev=153.65
    clat (usec): min=8871, max=99092, avg=48211.96, stdev=8577.87
     lat (usec): min=9191, max=99206, avg=48405.81, stdev=8582.38
    clat percentiles (usec):
     |  1.00th=[32375],  5.00th=[36439], 10.00th=[38536], 20.00th=[41157],
     | 30.00th=[43254], 40.00th=[45351], 50.00th=[47449], 60.00th=[49546],
     | 70.00th=[51643], 80.00th=[54789], 90.00th=[59507], 95.00th=[63177],
     | 99.00th=[71828], 99.50th=[79168], 99.90th=[91751], 99.95th=[99091],
     | 99.99th=[99091]
   bw (  KiB/s): min=  576, max=  822, per=25.36%, avg=665.53, stdev=63.59, samples=15
   iops        : min=  144, max=  205, avg=166.27, stdev=15.85, samples=15
  lat (msec)   : 10=0.16%, 20=0.12%, 50=64.26%, 100=35.47%
  cpu          : usr=0.44%, sys=5.36%, ctx=4556, majf=0, minf=15
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=182896: Mon Mar 16 07:08:01 2026
  read: IOPS=158, BW=636KiB/s (651kB/s)(5060KiB/7960msec)
    slat (usec): min=1173, max=11585, avg=3199.23, stdev=1171.57
    clat (usec): min=5742, max=73044, avg=47574.01, stdev=8582.68
     lat (usec): min=10959, max=78821, avg=50776.25, stdev=8784.41
    clat percentiles (usec):
     |  1.00th=[31327],  5.00th=[34866], 10.00th=[37487], 20.00th=[40633],
     | 30.00th=[42206], 40.00th=[44303], 50.00th=[46400], 60.00th=[49021],
     | 70.00th=[52167], 80.00th=[55313], 90.00th=[59507], 95.00th=[62653],
     | 99.00th=[69731], 99.50th=[71828], 99.90th=[72877], 99.95th=[72877],
     | 99.99th=[72877]
   bw (  KiB/s): min=  560, max=  744, per=25.39%, avg=640.07, stdev=55.56, samples=15
   iops        : min=  140, max=  186, avg=159.93, stdev=13.91, samples=15
  write: IOPS=162, BW=651KiB/s (666kB/s)(5180KiB/7960msec); 0 zone resets
    slat (usec): min=58, max=2764, avg=178.28, stdev=123.38
    clat (usec): min=21591, max=86659, avg=48049.46, stdev=7755.49
     lat (usec): min=22034, max=86803, avg=48228.89, stdev=7757.61
    clat percentiles (usec):
     |  1.00th=[33817],  5.00th=[36963], 10.00th=[39060], 20.00th=[41157],
     | 30.00th=[43254], 40.00th=[45351], 50.00th=[47449], 60.00th=[49021],
     | 70.00th=[51643], 80.00th=[54264], 90.00th=[57934], 95.00th=[62129],
     | 99.00th=[68682], 99.50th=[69731], 99.90th=[86508], 99.95th=[86508],
     | 99.99th=[86508]
   bw (  KiB/s): min=  541, max=  768, per=24.90%, avg=653.27, stdev=71.24, samples=15
   iops        : min=  135, max=  192, avg=163.20, stdev=17.88, samples=15
  lat (msec)   : 10=0.04%, 50=63.95%, 100=36.02%
  cpu          : usr=0.49%, sys=5.28%, ctx=4539, majf=0, minf=13
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=2521KiB/s (2582kB/s), 627KiB/s-636KiB/s (642kB/s-651kB/s), io=19.6MiB (20.5MB), run=7906-7960msec
  WRITE: bw=2625KiB/s (2688kB/s), 651KiB/s-665KiB/s (666kB/s-681kB/s), io=20.4MiB (21.4MB), run=7906-7960msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.19
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=182901: Mon Mar 16 07:08:05 2026
  read: IOPS=5, BW=240MiB/s (251MB/s)(40.0MiB/167msec)
    slat (nsec): min=12783k, max=12783k, avg=12782815.00, stdev= 0.00
    clat (nsec): min=146806k, max=146806k, avg=146805798.00, stdev= 0.00
     lat (nsec): min=159593k, max=159593k, avg=159592957.00, stdev= 0.00
    clat percentiles (msec):
     |  1.00th=[  148],  5.00th=[  148], 10.00th=[  148], 20.00th=[  148],
     | 30.00th=[  148], 40.00th=[  148], 50.00th=[  148], 60.00th=[  148],
     | 70.00th=[  148], 80.00th=[  148], 90.00th=[  148], 95.00th=[  148],
     | 99.00th=[  148], 99.50th=[  148], 99.90th=[  148], 99.95th=[  148],
     | 99.99th=[  148]
  lat (msec)   : 250=100.00%
  cpu          : usr=0.00%, sys=6.63%, ctx=12, majf=0, minf=8
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=240MiB/s (251MB/s), 240MiB/s-240MiB/s (251MB/s-251MB/s), io=40.0MiB (41.9MB), run=167-167msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout