== sanity test 398c: run fio to test AIO ================= 18:14:22 (1756678462)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB, 40 MiB) copied, 0.130734 s, 321 MB/s
osc.lustre-OST0000-osc-ffff9e18e0b69000.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=194827: Sun Aug 31 18:14:34 2025
  write: IOPS=300, BW=1202KiB/s (1231kB/s)(10.0MiB/8521msec); 0 zone resets
    slat (usec): min=31, max=755, avg=162.57, stdev=88.38
    clat (usec): min=5861, max=93821, avg=52482.39, stdev=11134.55
     lat (usec): min=5955, max=93970, avg=52646.55, stdev=11166.06
    clat percentiles (usec):
     |  1.00th=[26084],  5.00th=[31851], 10.00th=[37487], 20.00th=[42730],
     | 30.00th=[46924], 40.00th=[51119], 50.00th=[53740], 60.00th=[56361],
     | 70.00th=[59507], 80.00th=[62129], 90.00th=[65274], 95.00th=[67634],
     | 99.00th=[74974], 99.50th=[82314], 99.90th=[87557], 99.95th=[89654],
     | 99.99th=[93848]
   bw (  KiB/s): min= 1008, max= 1576, per=25.04%, avg=1203.31, stdev=163.88, samples=16
   iops        : min=  252, max=  394, avg=300.81, stdev=40.97, samples=16
  lat (msec)   : 10=0.12%, 50=37.15%, 100=62.73%
  cpu          : usr=0.41%, sys=5.01%, ctx=1369, majf=0, minf=9
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=194828: Sun Aug 31 18:14:34 2025
  write: IOPS=301, BW=1205KiB/s (1234kB/s)(10.0MiB/8500msec); 0 zone resets
    slat (usec): min=32, max=28250, avg=175.12, stdev=562.60
    clat (msec): min=8, max=104, avg=52.41, stdev=11.31
     lat (msec): min=8, max=104, avg=52.59, stdev=11.31
    clat percentiles (msec):
     |  1.00th=[   26],  5.00th=[   31], 10.00th=[   37], 20.00th=[   43],
     | 30.00th=[   47], 40.00th=[   51], 50.00th=[   54], 60.00th=[   57],
     | 70.00th=[   60], 80.00th=[   63], 90.00th=[   66], 95.00th=[   68],
     | 99.00th=[   77], 99.50th=[   84], 99.90th=[   97], 99.95th=[   97],
     | 99.99th=[  105]
   bw (  KiB/s): min= 1016, max= 1600, per=25.15%, avg=1208.63, stdev=168.92, samples=16
   iops        : min=  254, max=  400, avg=302.12, stdev=42.22, samples=16
  lat (msec)   : 10=0.04%, 20=0.04%, 50=38.91%, 100=60.98%, 250=0.04%
  cpu          : usr=0.45%, sys=5.05%, ctx=1479, majf=0, minf=10
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=194829: Sun Aug 31 18:14:34 2025
  write: IOPS=303, BW=1214KiB/s (1243kB/s)(10.0MiB/8438msec); 0 zone resets
    slat (usec): min=27, max=791, avg=168.49, stdev=90.25
    clat (msec): min=16, max=122, avg=52.47, stdev=11.74
     lat (msec): min=16, max=122, avg=52.64, stdev=11.77
    clat percentiles (msec):
     |  1.00th=[   26],  5.00th=[   31], 10.00th=[   37], 20.00th=[   43],
     | 30.00th=[   47], 40.00th=[   51], 50.00th=[   54], 60.00th=[   57],
     | 70.00th=[   60], 80.00th=[   63], 90.00th=[   66], 95.00th=[   69],
     | 99.00th=[   78], 99.50th=[   86], 99.90th=[  111], 99.95th=[  123],
     | 99.99th=[  123]
   bw (  KiB/s): min= 1016, max= 1592, per=25.32%, avg=1216.69, stdev=178.53, samples=16
   iops        : min=  254, max=  398, avg=304.12, stdev=44.58, samples=16
  lat (msec)   : 20=0.08%, 50=39.14%, 100=60.51%, 250=0.27%
  cpu          : usr=0.50%, sys=4.87%, ctx=1547, majf=0, minf=10
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=194830: Sun Aug 31 18:14:34 2025
  write: IOPS=300, BW=1202KiB/s (1230kB/s)(10.0MiB/8522msec); 0 zone resets
    slat (usec): min=24, max=887, avg=161.56, stdev=85.13
    clat (usec): min=22056, max=83490, avg=52428.62, stdev=10930.41
     lat (usec): min=22165, max=83563, avg=52592.16, stdev=10960.09
    clat percentiles (usec):
     |  1.00th=[26608],  5.00th=[31589], 10.00th=[36439], 20.00th=[43254],
     | 30.00th=[46924], 40.00th=[50594], 50.00th=[53740], 60.00th=[56886],
     | 70.00th=[59507], 80.00th=[62129], 90.00th=[65274], 95.00th=[67634],
     | 99.00th=[72877], 99.50th=[78119], 99.90th=[83362], 99.95th=[83362],
     | 99.99th=[83362]
   bw (  KiB/s): min= 1016, max= 1568, per=25.01%, avg=1201.88, stdev=168.48, samples=16
   iops        : min=  254, max=  392, avg=300.44, stdev=42.12, samples=16
  lat (msec)   : 50=39.34%, 100=60.66%
  cpu          : usr=0.46%, sys=4.68%, ctx=1421, majf=0, minf=9
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=4806KiB/s (4922kB/s), 1202KiB/s-1214KiB/s (1230kB/s-1243kB/s), io=40.0MiB (41.9MB), run=8438-8522msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=194841: Sun Aug 31 18:14:43 2025
  read: IOPS=208, BW=834KiB/s (854kB/s)(5048KiB/6053msec)
    slat (usec): min=1139, max=13128, avg=2835.68, stdev=861.19
    clat (usec): min=8624, max=55435, avg=35479.59, stdev=5244.86
     lat (usec): min=10154, max=58283, avg=38317.55, stdev=5337.05
    clat percentiles (usec):
     |  1.00th=[23987],  5.00th=[27132], 10.00th=[28967], 20.00th=[30802],
     | 30.00th=[32637], 40.00th=[34341], 50.00th=[35390], 60.00th=[36963],
     | 70.00th=[38011], 80.00th=[39584], 90.00th=[41681], 95.00th=[43779],
     | 99.00th=[48497], 99.50th=[49546], 99.90th=[53740], 99.95th=[55313],
     | 99.99th=[55313]
   bw (  KiB/s): min=  717, max=  936, per=25.00%, avg=828.82, stdev=70.42, samples=11
   iops        : min=  179, max=  234, avg=207.18, stdev=17.65, samples=11
  write: IOPS=214, BW=858KiB/s (878kB/s)(5192KiB/6053msec); 0 zone resets
    slat (usec): min=30, max=3343, avg=132.58, stdev=114.46
    clat (usec): min=8701, max=48364, avg=36134.84, stdev=4256.19
     lat (usec): min=8810, max=48534, avg=36269.00, stdev=4266.96
    clat percentiles (usec):
     |  1.00th=[27132],  5.00th=[29230], 10.00th=[30802], 20.00th=[32637],
     | 30.00th=[33817], 40.00th=[34866], 50.00th=[35914], 60.00th=[36963],
     | 70.00th=[38536], 80.00th=[39584], 90.00th=[41681], 95.00th=[43254],
     | 99.00th=[46400], 99.50th=[47449], 99.90th=[47973], 99.95th=[48497],
     | 99.99th=[48497]
   bw (  KiB/s): min=  724, max=  952, per=24.98%, avg=862.18, stdev=65.52, samples=11
   iops        : min=  181, max=  238, avg=215.55, stdev=16.38, samples=11
  lat (msec)   : 10=0.08%, 20=0.16%, 50=99.53%, 100=0.23%
  cpu          : usr=0.68%, sys=7.50%, ctx=2496, majf=0, minf=11
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=194842: Sun Aug 31 18:14:43 2025
  read: IOPS=207, BW=829KiB/s (849kB/s)(4976KiB/6004msec)
    slat (usec): min=38, max=9900, avg=2844.68, stdev=847.64
    clat (usec): min=11999, max=59555, avg=35600.62, stdev=5361.98
     lat (usec): min=12150, max=62578, avg=38447.58, stdev=5442.25
    clat percentiles (usec):
     |  1.00th=[23725],  5.00th=[27132], 10.00th=[28967], 20.00th=[31065],
     | 30.00th=[32900], 40.00th=[34341], 50.00th=[35390], 60.00th=[36963],
     | 70.00th=[38011], 80.00th=[40109], 90.00th=[42206], 95.00th=[44827],
     | 99.00th=[49021], 99.50th=[52691], 99.90th=[56886], 99.95th=[59507],
     | 99.99th=[59507]
   bw (  KiB/s): min=  743, max=  944, per=24.94%, avg=826.82, stdev=58.96, samples=11
   iops        : min=  185, max=  236, avg=206.64, stdev=14.85, samples=11
  write: IOPS=219, BW=877KiB/s (898kB/s)(5264KiB/6004msec); 0 zone resets
    slat (usec): min=32, max=1862, avg=127.31, stdev=79.49
    clat (usec): min=6044, max=71179, avg=36307.75, stdev=4819.78
     lat (usec): min=6130, max=71348, avg=36436.65, stdev=4823.15
    clat percentiles (usec):
     |  1.00th=[26346],  5.00th=[29492], 10.00th=[30802], 20.00th=[32637],
     | 30.00th=[33817], 40.00th=[34866], 50.00th=[36439], 60.00th=[36963],
     | 70.00th=[38536], 80.00th=[39584], 90.00th=[41681], 95.00th=[43254],
     | 99.00th=[49021], 99.50th=[57934], 99.90th=[68682], 99.95th=[70779],
     | 99.99th=[70779]
   bw (  KiB/s): min=  784, max=  968, per=25.74%, avg=888.36, stdev=55.27, samples=11
   iops        : min=  196, max=  242, avg=222.09, stdev=13.82, samples=11
  lat (msec)   : 10=0.04%, 20=0.08%, 50=99.10%, 100=0.78%
  cpu          : usr=0.75%, sys=7.40%, ctx=2452, majf=0, minf=12
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=194843: Sun Aug 31 18:14:43 2025
  read: IOPS=206, BW=824KiB/s (844kB/s)(4984KiB/6047msec)
    slat (usec): min=1180, max=9893, avg=2871.55, stdev=829.00
    clat (usec): min=3731, max=62279, avg=35697.79, stdev=5386.09
     lat (usec): min=5926, max=64139, avg=38571.57, stdev=5447.17
    clat percentiles (usec):
     |  1.00th=[23725],  5.00th=[27395], 10.00th=[29230], 20.00th=[31065],
     | 30.00th=[32900], 40.00th=[34341], 50.00th=[35390], 60.00th=[36963],
     | 70.00th=[38536], 80.00th=[40109], 90.00th=[42206], 95.00th=[44303],
     | 99.00th=[49546], 99.50th=[52167], 99.90th=[55313], 99.95th=[62129],
     | 99.99th=[62129]
   bw (  KiB/s): min=  724, max=  920, per=25.11%, avg=832.36, stdev=56.81, samples=11
   iops        : min=  181, max=  230, avg=208.09, stdev=14.20, samples=11
  write: IOPS=217, BW=869KiB/s (890kB/s)(5256KiB/6047msec); 0 zone resets
    slat (usec): min=30, max=669, avg=126.50, stdev=65.10
    clat (usec): min=23262, max=61961, avg=36268.94, stdev=4492.50
     lat (usec): min=23368, max=62052, avg=36396.69, stdev=4498.11
    clat percentiles (usec):
     |  1.00th=[26870],  5.00th=[29492], 10.00th=[31327], 20.00th=[32637],
     | 30.00th=[33817], 40.00th=[34866], 50.00th=[35914], 60.00th=[36963],
     | 70.00th=[38011], 80.00th=[39584], 90.00th=[41681], 95.00th=[43779],
     | 99.00th=[49021], 99.50th=[52691], 99.90th=[61604], 99.95th=[62129],
     | 99.99th=[62129]
   bw (  KiB/s): min=  760, max= 1056, per=25.13%, avg=867.27, stdev=107.76, samples=11
   iops        : min=  190, max=  264, avg=216.82, stdev=26.94, samples=11
  lat (msec)   : 4=0.04%, 10=0.04%, 20=0.08%, 50=98.98%, 100=0.86%
  cpu          : usr=0.56%, sys=7.53%, ctx=2474, majf=0, minf=13
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=194844: Sun Aug 31 18:14:43 2025
  read: IOPS=209, BW=837KiB/s (858kB/s)(5060KiB/6042msec)
    slat (usec): min=1184, max=5793, avg=2849.94, stdev=762.16
    clat (usec): min=12942, max=54806, avg=35756.92, stdev=5143.88
     lat (usec): min=15606, max=58665, avg=38609.03, stdev=5226.87
    clat percentiles (usec):
     |  1.00th=[24249],  5.00th=[27395], 10.00th=[29492], 20.00th=[31327],
     | 30.00th=[32900], 40.00th=[34341], 50.00th=[35914], 60.00th=[36963],
     | 70.00th=[38011], 80.00th=[40109], 90.00th=[42206], 95.00th=[44827],
     | 99.00th=[49021], 99.50th=[49546], 99.90th=[54789], 99.95th=[54789],
     | 99.99th=[54789]
   bw (  KiB/s): min=  717, max=  920, per=25.38%, avg=841.18, stdev=59.39, samples=11
   iops        : min=  179, max=  230, avg=210.27, stdev=14.90, samples=11
  write: IOPS=214, BW=857KiB/s (878kB/s)(5180KiB/6042msec); 0 zone resets
    slat (usec): min=24, max=1896, avg=127.53, stdev=83.69
    clat (usec): min=23424, max=64052, avg=35960.98, stdev=4385.93
     lat (usec): min=23520, max=64205, avg=36089.98, stdev=4394.19
    clat percentiles (usec):
     |  1.00th=[26870],  5.00th=[29230], 10.00th=[30802], 20.00th=[32113],
     | 30.00th=[33817], 40.00th=[34866], 50.00th=[35914], 60.00th=[36963],
     | 70.00th=[38011], 80.00th=[39584], 90.00th=[41681], 95.00th=[43254],
     | 99.00th=[47973], 99.50th=[49021], 99.90th=[57934], 99.95th=[64226],
     | 99.99th=[64226]
   bw (  KiB/s): min=  769, max=  968, per=24.95%, avg=861.18, stdev=59.95, samples=11
   iops        : min=  192, max=  242, avg=215.27, stdev=15.03, samples=11
  lat (msec)   : 20=0.08%, 50=99.57%, 100=0.35%
  cpu          : usr=0.58%, sys=7.53%, ctx=2442, majf=0, minf=12
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=3315KiB/s (3395kB/s), 824KiB/s-837KiB/s (844kB/s-858kB/s), io=19.6MiB (20.5MB), run=6004-6053msec
  WRITE: bw=3452KiB/s (3534kB/s), 857KiB/s-877KiB/s (878kB/s-898kB/s), io=20.4MiB (21.4MB), run=6004-6053msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.19
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=194848: Sun Aug 31 18:14:46 2025
  read: IOPS=6, BW=272MiB/s (285MB/s)(40.0MiB/147msec)
    slat (nsec): min=9940.9k, max=9940.9k, avg=9940902.00, stdev= 0.00
    clat (nsec): min=128184k, max=128184k, avg=128184062.00, stdev= 0.00
     lat (nsec): min=138127k, max=138127k, avg=138127443.00, stdev= 0.00
    clat percentiles (msec):
     |  1.00th=[  129],  5.00th=[  129], 10.00th=[  129], 20.00th=[  129],
     | 30.00th=[  129], 40.00th=[  129], 50.00th=[  129], 60.00th=[  129],
     | 70.00th=[  129], 80.00th=[  129], 90.00th=[  129], 95.00th=[  129],
     | 99.00th=[  129], 99.50th=[  129], 99.90th=[  129], 99.95th=[  129],
     | 99.99th=[  129]
  lat (msec)   : 250=100.00%
  cpu          : usr=0.00%, sys=7.53%, ctx=3, majf=0, minf=8
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=272MiB/s (285MB/s), 272MiB/s-272MiB/s (285MB/s-285MB/s), io=40.0MiB (41.9MB), run=147-147msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout