== sanity test 398c: run fio to test AIO ================= 09:53:42 (1742910822)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB) copied, 0.125563 s, 334 MB/s
osc.lustre-OST0000-osc-ffff88012a5e9000.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=29593: Tue Mar 25 09:53:49 2025
  write: IOPS=428, BW=1714KiB/s (1755kB/s)(10.0MiB/5976msec)
    slat (usec): min=13, max=728, avg=66.89, stdev=44.22
    clat (usec): min=5183, max=59987, avg=37009.80, stdev=6839.67
     lat (usec): min=5296, max=60028, avg=37078.61, stdev=6850.77
    clat percentiles (usec):
     |  1.00th=[20841],  5.00th=[23725], 10.00th=[25822], 20.00th=[31065],
     | 30.00th=[34341], 40.00th=[36439], 50.00th=[38011], 60.00th=[39584],
     | 70.00th=[41157], 80.00th=[42730], 90.00th=[45351], 95.00th=[46400],
     | 99.00th=[47973], 99.50th=[50070], 99.90th=[54789], 99.95th=[54789],
     | 99.99th=[60031]
   bw (  KiB/s): min= 1552, max= 2176, per=25.05%, avg=1716.73, stdev=186.67, samples=11
   iops        : min=  388, max=  544, avg=429.18, stdev=46.67, samples=11
  lat (msec)   : 10=0.12%, 20=0.20%, 50=99.41%, 100=0.27%
  cpu          : usr=0.33%, sys=3.55%, ctx=1042, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=29594: Tue Mar 25 09:53:49 2025
  write: IOPS=431, BW=1727KiB/s (1768kB/s)(10.0MiB/5930msec)
    slat (usec): min=13, max=355, avg=78.06, stdev=43.62
    clat (usec): min=1921, max=74354, avg=36790.04, stdev=7374.33
     lat (usec): min=2123, max=74404, avg=36870.03, stdev=7385.76
    clat percentiles (usec):
     |  1.00th=[19006],  5.00th=[23462], 10.00th=[25560], 20.00th=[31065],
     | 30.00th=[34341], 40.00th=[36439], 50.00th=[38011], 60.00th=[39584],
     | 70.00th=[41157], 80.00th=[42730], 90.00th=[44827], 95.00th=[46400],
     | 99.00th=[47973], 99.50th=[52167], 99.90th=[69731], 99.95th=[73925],
     | 99.99th=[73925]
   bw (  KiB/s): min= 1544, max= 2192, per=25.28%, avg=1732.73, stdev=209.42, samples=11
   iops        : min=  386, max=  548, avg=433.18, stdev=52.35, samples=11
  lat (msec)   : 2=0.04%, 4=0.08%, 10=0.35%, 20=0.86%, 50=98.09%
  lat (msec)   : 100=0.59%
  cpu          : usr=0.84%, sys=3.79%, ctx=1537, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=29595: Tue Mar 25 09:53:49 2025
  write: IOPS=428, BW=1715KiB/s (1756kB/s)(10.0MiB/5972msec)
    slat (usec): min=12, max=555, avg=73.64, stdev=47.11
    clat (usec): min=16651, max=77018, avg=37009.50, stdev=6919.56
     lat (usec): min=16723, max=77067, avg=37085.05, stdev=6931.73
    clat percentiles (usec):
     |  1.00th=[20055],  5.00th=[23987], 10.00th=[26346], 20.00th=[31327],
     | 30.00th=[34866], 40.00th=[36439], 50.00th=[38011], 60.00th=[39584],
     | 70.00th=[41157], 80.00th=[42730], 90.00th=[44827], 95.00th=[46400],
     | 99.00th=[47973], 99.50th=[48497], 99.90th=[69731], 99.95th=[77071],
     | 99.99th=[77071]
   bw (  KiB/s): min= 1536, max= 2176, per=25.12%, avg=1721.82, stdev=188.68, samples=11
   iops        : min=  384, max=  544, avg=430.45, stdev=47.17, samples=11
  lat (msec)   : 20=0.78%, 50=98.87%, 100=0.35%
  cpu          : usr=0.52%, sys=3.80%, ctx=1376, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=29596: Tue Mar 25 09:53:49 2025
  write: IOPS=428, BW=1715KiB/s (1756kB/s)(10.0MiB/5972msec)
    slat (usec): min=13, max=939, avg=73.45, stdev=47.80
    clat (usec): min=18390, max=69415, avg=37044.71, stdev=6834.05
     lat (usec): min=18448, max=69501, avg=37120.10, stdev=6845.41
    clat percentiles (usec):
     |  1.00th=[20055],  5.00th=[23987], 10.00th=[26346], 20.00th=[31327],
     | 30.00th=[34866], 40.00th=[36439], 50.00th=[38011], 60.00th=[39584],
     | 70.00th=[41157], 80.00th=[42730], 90.00th=[44827], 95.00th=[46400],
     | 99.00th=[47973], 99.50th=[48497], 99.90th=[64226], 99.95th=[64226],
     | 99.99th=[69731]
   bw (  KiB/s): min= 1536, max= 2184, per=25.09%, avg=1719.64, stdev=194.16, samples=11
   iops        : min=  384, max=  546, avg=429.91, stdev=48.54, samples=11
  lat (msec)   : 20=0.94%, 50=98.67%, 100=0.39%
  cpu          : usr=0.59%, sys=3.87%, ctx=1392, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=6854KiB/s (7019kB/s), 1714KiB/s-1727KiB/s (1755kB/s-1768kB/s), io=40.0MiB (41.9MB), run=5930-5976msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=29610: Tue Mar 25 09:53:53 2025
   read: IOPS=422, BW=1691KiB/s (1732kB/s)(5048KiB/2985msec)
    slat (usec): min=663, max=4378, avg=1232.13, stdev=340.13
    clat (usec): min=3485, max=41572, avg=17532.41, stdev=4221.08
     lat (usec): min=5046, max=43769, avg=18766.44, stdev=4409.72
    clat percentiles (usec):
     |  1.00th=[11600],  5.00th=[12911], 10.00th=[13435], 20.00th=[14353],
     | 30.00th=[15139], 40.00th=[15926], 50.00th=[16712], 60.00th=[17433],
     | 70.00th=[18482], 80.00th=[19792], 90.00th=[22414], 95.00th=[25035],
     | 99.00th=[36439], 99.50th=[38011], 99.90th=[41157], 99.95th=[41681],
     | 99.99th=[41681]
   bw (  KiB/s): min= 1240, max= 1984, per=25.03%, avg=1681.60, stdev=293.25, samples=5
   iops        : min=  310, max=  496, avg=420.40, stdev=73.31, samples=5
  write: IOPS=434, BW=1739KiB/s (1781kB/s)(5192KiB/2985msec)
    slat (usec): min=13, max=1388, avg=52.28, stdev=52.65
    clat (usec): min=5942, max=41840, avg=18082.49, stdev=4330.77
     lat (usec): min=5976, max=41934, avg=18136.09, stdev=4338.07
    clat percentiles (usec):
     |  1.00th=[12387],  5.00th=[13829], 10.00th=[14353], 20.00th=[15139],
     | 30.00th=[15795], 40.00th=[16450], 50.00th=[16909], 60.00th=[17695],
     | 70.00th=[18482], 80.00th=[20317], 90.00th=[23200], 95.00th=[25560],
     | 99.00th=[37487], 99.50th=[39060], 99.90th=[40109], 99.95th=[41681],
     | 99.99th=[41681]
   bw (  KiB/s): min= 1280, max= 2104, per=25.32%, avg=1771.20, stdev=326.37, samples=5
   iops        : min=  320, max=  526, avg=442.80, stdev=81.59, samples=5
  lat (msec)   : 4=0.04%, 10=0.16%, 20=79.96%, 50=19.84%
  cpu          : usr=0.64%, sys=8.28%, ctx=2412, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=29611: Tue Mar 25 09:53:53 2025
   read: IOPS=416, BW=1666KiB/s (1706kB/s)(4976KiB/2986msec)
    slat (usec): min=665, max=3146, avg=1227.84, stdev=334.60
    clat (usec): min=10184, max=43468, avg=17493.69, stdev=4411.65
     lat (usec): min=11137, max=46256, avg=18723.58, stdev=4612.32
    clat percentiles (usec):
     |  1.00th=[11731],  5.00th=[12518], 10.00th=[13304], 20.00th=[14353],
     | 30.00th=[15139], 40.00th=[15795], 50.00th=[16581], 60.00th=[17695],
     | 70.00th=[18482], 80.00th=[19530], 90.00th=[22414], 95.00th=[25035],
     | 99.00th=[35914], 99.50th=[41157], 99.90th=[42730], 99.95th=[43254],
     | 99.99th=[43254]
   bw (  KiB/s): min= 1112, max= 1928, per=24.79%, avg=1665.60, stdev=335.40, samples=5
   iops        : min=  278, max=  482, avg=416.40, stdev=83.85, samples=5
  write: IOPS=440, BW=1763KiB/s (1805kB/s)(5264KiB/2986msec)
    slat (usec): min=14, max=402, avg=49.61, stdev=32.15
    clat (usec): min=3166, max=42744, avg=18067.10, stdev=4373.89
     lat (usec): min=3247, max=42783, avg=18117.89, stdev=4380.60
    clat percentiles (usec):
     |  1.00th=[12649],  5.00th=[13829], 10.00th=[14484], 20.00th=[15270],
     | 30.00th=[15926], 40.00th=[16319], 50.00th=[16909], 60.00th=[17695],
     | 70.00th=[18482], 80.00th=[20055], 90.00th=[22938], 95.00th=[26084],
     | 99.00th=[37487], 99.50th=[39060], 99.90th=[40109], 99.95th=[42730],
     | 99.99th=[42730]
   bw (  KiB/s): min= 1360, max= 2184, per=25.60%, avg=1790.40, stdev=317.16, samples=5
   iops        : min=  340, max=  546, avg=447.60, stdev=79.29, samples=5
  lat (msec)   : 4=0.04%, 10=0.04%, 20=81.33%, 50=18.59%
  cpu          : usr=1.17%, sys=7.60%, ctx=2344, majf=0, minf=35
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=29612: Tue Mar 25 09:53:53 2025
   read: IOPS=417, BW=1669KiB/s (1709kB/s)(4984KiB/2987msec)
    slat (usec): min=657, max=2976, avg=1228.68, stdev=335.19
    clat (usec): min=6711, max=43480, avg=17461.44, stdev=4319.20
     lat (usec): min=7977, max=46392, avg=18692.34, stdev=4508.31
    clat percentiles (usec):
     |  1.00th=[11469],  5.00th=[12649], 10.00th=[13304], 20.00th=[14353],
     | 30.00th=[15139], 40.00th=[15795], 50.00th=[16581], 60.00th=[17433],
     | 70.00th=[18220], 80.00th=[19792], 90.00th=[22152], 95.00th=[24773],
     | 99.00th=[35914], 99.50th=[39060], 99.90th=[41681], 99.95th=[43254],
     | 99.99th=[43254]
   bw (  KiB/s): min= 1264, max= 2080, per=24.96%, avg=1676.80, stdev=330.75, samples=5
   iops        : min=  316, max=  520, avg=419.20, stdev=82.69, samples=5
  write: IOPS=439, BW=1760KiB/s (1802kB/s)(5256KiB/2987msec)
    slat (usec): min=12, max=802, avg=51.94, stdev=39.28
    clat (usec): min=10556, max=42393, avg=18176.46, stdev=4246.96
     lat (usec): min=10593, max=42434, avg=18229.79, stdev=4251.93
    clat percentiles (usec):
     |  1.00th=[12518],  5.00th=[13698], 10.00th=[14353], 20.00th=[15139],
     | 30.00th=[15795], 40.00th=[16581], 50.00th=[17171], 60.00th=[17695],
     | 70.00th=[18744], 80.00th=[20055], 90.00th=[23462], 95.00th=[25822],
     | 99.00th=[36439], 99.50th=[38011], 99.90th=[41681], 99.95th=[42206],
     | 99.99th=[42206]
   bw (  KiB/s): min= 1240, max= 2008, per=25.39%, avg=1776.00, stdev=312.67, samples=5
   iops        : min=  310, max=  502, avg=444.00, stdev=78.17, samples=5
  lat (msec)   : 10=0.08%, 20=80.04%, 50=19.88%
  cpu          : usr=1.07%, sys=7.87%, ctx=2359, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=29613: Tue Mar 25 09:53:53 2025
   read: IOPS=428, BW=1714KiB/s (1755kB/s)(5060KiB/2952msec)
    slat (usec): min=34, max=3213, avg=1232.70, stdev=347.44
    clat (usec): min=2758, max=42743, avg=17410.92, stdev=4346.28
     lat (usec): min=2922, max=45746, avg=18645.79, stdev=4535.81
    clat percentiles (usec):
     |  1.00th=[11076],  5.00th=[12518], 10.00th=[13435], 20.00th=[14353],
     | 30.00th=[15139], 40.00th=[15926], 50.00th=[16712], 60.00th=[17433],
     | 70.00th=[18220], 80.00th=[19530], 90.00th=[22414], 95.00th=[25035],
     | 99.00th=[35390], 99.50th=[38011], 99.90th=[42730], 99.95th=[42730],
     | 99.99th=[42730]
   bw (  KiB/s): min= 1368, max= 1992, per=25.67%, avg=1724.80, stdev=250.41, samples=5
   iops        : min=  342, max=  498, avg=431.20, stdev=62.60, samples=5
  write: IOPS=438, BW=1755KiB/s (1797kB/s)(5180KiB/2952msec)
    slat (usec): min=13, max=472, avg=52.15, stdev=37.44
    clat (usec): min=11351, max=44143, avg=18100.37, stdev=4192.70
     lat (usec): min=11401, max=44231, avg=18153.73, stdev=4202.21
    clat percentiles (usec):
     |  1.00th=[12911],  5.00th=[13829], 10.00th=[14615], 20.00th=[15139],
     | 30.00th=[15926], 40.00th=[16450], 50.00th=[16909], 60.00th=[17695],
     | 70.00th=[18482], 80.00th=[20055], 90.00th=[23200], 95.00th=[26346],
     | 99.00th=[35390], 99.50th=[39060], 99.90th=[42206], 99.95th=[44303],
     | 99.99th=[44303]
   bw (  KiB/s): min= 1360, max= 2080, per=25.46%, avg=1780.80, stdev=291.68, samples=5
   iops        : min=  340, max=  520, avg=445.20, stdev=72.92, samples=5
  lat (msec)   : 4=0.20%, 10=0.08%, 20=80.35%, 50=19.38%
  cpu          : usr=1.25%, sys=7.76%, ctx=2424, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=6718KiB/s (6880kB/s), 1666KiB/s-1714KiB/s (1706kB/s-1755kB/s), io=19.6MiB (20.5MB), run=2952-2987msec
  WRITE: bw=6994KiB/s (7162kB/s), 1739KiB/s-1763KiB/s (1781kB/s-1805kB/s), io=20.4MiB (21.4MB), run=2952-2987msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.7
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=29619: Tue Mar 25 09:53:55 2025
   read: IOPS=6, BW=261MiB/s (274MB/s)(40.0MiB/153msec)
    slat (nsec): min=6324.6k, max=6324.6k, avg=6324577.00, stdev= 0.00
    clat (nsec): min=140801k, max=140801k, avg=140801470.00, stdev= 0.00
     lat (nsec): min=147137k, max=147137k, avg=147136879.00, stdev= 0.00
    clat percentiles (msec):
     |  1.00th=[  142],  5.00th=[  142], 10.00th=[  142], 20.00th=[  142],
     | 30.00th=[  142], 40.00th=[  142], 50.00th=[  142], 60.00th=[  142],
     | 70.00th=[  142], 80.00th=[  142], 90.00th=[  142], 95.00th=[  142],
     | 99.00th=[  142], 99.50th=[  142], 99.90th=[  142], 99.95th=[  142],
     | 99.99th=[  142]
  lat (msec)   : 250=100.00%
  cpu          : usr=0.00%, sys=4.61%, ctx=7, majf=0, minf=29
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=261MiB/s (274MB/s), 261MiB/s-261MiB/s (274MB/s-274MB/s), io=40.0MiB (41.9MB), run=153-153msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout