== sanity test 398c: run fio to test AIO ================= 20:56:02 (1777424162)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB, 40 MiB) copied, 0.143896 s, 291 MB/s
osc.lustre-OST0000-osc-ffff9a9060767000.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=190662: Tue Apr 28 20:56:12 2026
  write: IOPS=393, BW=1573KiB/s (1611kB/s)(10.0MiB/6509msec); 0 zone resets
    slat (usec): min=46, max=4131, avg=189.98, stdev=122.53
    clat (usec): min=7195, max=67237, avg=40436.20, stdev=7697.16
     lat (usec): min=7381, max=67380, avg=40627.40, stdev=7698.58
    clat percentiles (usec):
     |  1.00th=[26608],  5.00th=[30016], 10.00th=[31589], 20.00th=[33817],
     | 30.00th=[35390], 40.00th=[37487], 50.00th=[39584], 60.00th=[41681],
     | 70.00th=[43254], 80.00th=[46924], 90.00th=[51119], 95.00th=[54789],
     | 99.00th=[60556], 99.50th=[62653], 99.90th=[64226], 99.95th=[65274],
     | 99.99th=[67634]
   bw (  KiB/s): min= 1408, max= 1776, per=24.89%, avg=1555.83, stdev=116.06, samples=12
   iops        : min=  352, max=  444, avg=388.92, stdev=29.01, samples=12
  lat (msec)   : 10=0.04%, 20=0.08%, 50=87.30%, 100=12.58%
  cpu          : usr=0.49%, sys=5.79%, ctx=2750, majf=0, minf=10
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=190663: Tue Apr 28 20:56:12 2026
  write: IOPS=390, BW=1563KiB/s (1601kB/s)(10.0MiB/6551msec); 0 zone resets
    slat (usec): min=50, max=23210, avg=202.35, stdev=479.28
    clat (usec): min=6996, max=69535, avg=40441.51, stdev=7837.32
     lat (usec): min=7622, max=69728, avg=40645.10, stdev=7833.83
    clat percentiles (usec):
     |  1.00th=[26084],  5.00th=[29754], 10.00th=[31327], 20.00th=[33817],
     | 30.00th=[35390], 40.00th=[36963], 50.00th=[39584], 60.00th=[41681],
     | 70.00th=[43779], 80.00th=[46400], 90.00th=[51643], 95.00th=[55313],
     | 99.00th=[61604], 99.50th=[63177], 99.90th=[67634], 99.95th=[67634],
     | 99.99th=[69731]
   bw (  KiB/s): min= 1416, max= 1768, per=24.71%, avg=1544.58, stdev=110.04, samples=12
   iops        : min=  354, max=  442, avg=386.08, stdev=27.52, samples=12
  lat (msec)   : 10=0.08%, 20=0.08%, 50=86.68%, 100=13.16%
  cpu          : usr=0.49%, sys=5.80%, ctx=2806, majf=0, minf=13
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=190664: Tue Apr 28 20:56:12 2026
  write: IOPS=391, BW=1564KiB/s (1602kB/s)(10.0MiB/6547msec); 0 zone resets
    slat (usec): min=44, max=1350, avg=189.86, stdev=104.16
    clat (usec): min=7054, max=69377, avg=40411.36, stdev=7901.55
     lat (usec): min=7225, max=69737, avg=40602.35, stdev=7907.86
    clat percentiles (usec):
     |  1.00th=[26608],  5.00th=[29754], 10.00th=[31327], 20.00th=[33424],
     | 30.00th=[35390], 40.00th=[37487], 50.00th=[39584], 60.00th=[41681],
     | 70.00th=[43779], 80.00th=[46924], 90.00th=[51119], 95.00th=[55313],
     | 99.00th=[60556], 99.50th=[62129], 99.90th=[66847], 99.95th=[67634],
     | 99.99th=[69731]
   bw (  KiB/s): min= 1424, max= 1736, per=24.69%, avg=1543.50, stdev=105.77, samples=12
   iops        : min=  356, max=  434, avg=385.83, stdev=26.47, samples=12
  lat (msec)   : 10=0.27%, 50=86.80%, 100=12.93%
  cpu          : usr=0.43%, sys=5.90%, ctx=2858, majf=0, minf=9
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=190665: Tue Apr 28 20:56:12 2026
  write: IOPS=392, BW=1571KiB/s (1609kB/s)(10.0MiB/6517msec); 0 zone resets
    slat (usec): min=40, max=1158, avg=169.50, stdev=100.88
    clat (usec): min=4086, max=64617, avg=40433.31, stdev=7869.00
     lat (usec): min=4250, max=65258, avg=40604.08, stdev=7876.61
    clat percentiles (usec):
     |  1.00th=[26608],  5.00th=[29754], 10.00th=[31065], 20.00th=[33817],
     | 30.00th=[35390], 40.00th=[37487], 50.00th=[39584], 60.00th=[42206],
     | 70.00th=[43779], 80.00th=[46924], 90.00th=[51119], 95.00th=[55313],
     | 99.00th=[60556], 99.50th=[63701], 99.90th=[64750], 99.95th=[64750],
     | 99.99th=[64750]
   bw (  KiB/s): min= 1408, max= 1784, per=24.80%, avg=1550.33, stdev=117.09, samples=12
   iops        : min=  352, max=  446, avg=387.58, stdev=29.27, samples=12
  lat (msec)   : 10=0.16%, 20=0.12%, 50=87.50%, 100=12.23%
  cpu          : usr=0.41%, sys=5.00%, ctx=2813, majf=0, minf=8
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=6252KiB/s (6403kB/s), 1563KiB/s-1573KiB/s (1601kB/s-1611kB/s), io=40.0MiB (41.9MB), run=6509-6551msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=190676: Tue Apr 28 20:56:21 2026
  read: IOPS=209, BW=840KiB/s (860kB/s)(5048KiB/6012msec)
    slat (usec): min=1021, max=7178, avg=2593.40, stdev=837.58
    clat (usec): min=8619, max=56392, avg=35894.89, stdev=5622.95
     lat (usec): min=11545, max=60205, avg=38490.12, stdev=5688.97
    clat percentiles (usec):
     |  1.00th=[24249],  5.00th=[27395], 10.00th=[29230], 20.00th=[31327],
     | 30.00th=[32900], 40.00th=[34341], 50.00th=[35390], 60.00th=[36963],
     | 70.00th=[38536], 80.00th=[40109], 90.00th=[43254], 95.00th=[45876],
     | 99.00th=[50594], 99.50th=[52167], 99.90th=[54264], 99.95th=[56361],
     | 99.99th=[56361]
   bw (  KiB/s): min=  784, max=  896, per=24.79%, avg=827.36, stdev=35.19, samples=11
   iops        : min=  196, max=  224, avg=206.82, stdev= 8.82, samples=11
  write: IOPS=215, BW=864KiB/s (884kB/s)(5192KiB/6012msec); 0 zone resets
    slat (usec): min=53, max=1007, avg=157.99, stdev=84.88
    clat (usec): min=8276, max=57749, avg=35946.92, stdev=4789.61
     lat (usec): min=8605, max=57873, avg=36105.96, stdev=4789.52
    clat percentiles (usec):
     |  1.00th=[27395],  5.00th=[29492], 10.00th=[30540], 20.00th=[32113],
     | 30.00th=[33162], 40.00th=[34341], 50.00th=[35390], 60.00th=[36439],
     | 70.00th=[38011], 80.00th=[39584], 90.00th=[41681], 95.00th=[44303],
     | 99.00th=[49546], 99.50th=[50594], 99.90th=[55837], 99.95th=[57934],
     | 99.99th=[57934]
   bw (  KiB/s): min=  728, max= 1048, per=24.83%, avg=862.91, stdev=104.06, samples=11
   iops        : min=  182, max=  262, avg=215.73, stdev=26.02, samples=11
  lat (msec)   : 10=0.12%, 20=0.23%, 50=98.59%, 100=1.05%
  cpu          : usr=0.55%, sys=5.87%, ctx=4412, majf=0, minf=11
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=190677: Tue Apr 28 20:56:21 2026
  read: IOPS=208, BW=835KiB/s (855kB/s)(4976KiB/5960msec)
    slat (usec): min=42, max=10106, avg=2644.31, stdev=946.46
    clat (usec): min=3048, max=60032, avg=35543.54, stdev=6087.14
     lat (usec): min=4827, max=62629, avg=38189.65, stdev=6245.16
    clat percentiles (usec):
     |  1.00th=[23462],  5.00th=[27132], 10.00th=[28705], 20.00th=[30802],
     | 30.00th=[32375], 40.00th=[33817], 50.00th=[35390], 60.00th=[36439],
     | 70.00th=[38011], 80.00th=[40109], 90.00th=[42730], 95.00th=[46400],
     | 99.00th=[51643], 99.50th=[52691], 99.90th=[59507], 99.95th=[60031],
     | 99.99th=[60031]
   bw (  KiB/s): min=  740, max=  936, per=24.77%, avg=826.55, stdev=64.65, samples=11
   iops        : min=  185, max=  234, avg=206.64, stdev=16.16, samples=11
  write: IOPS=220, BW=883KiB/s (904kB/s)(5264KiB/5960msec); 0 zone resets
    slat (usec): min=47, max=3932, avg=162.30, stdev=133.75
    clat (usec): min=12850, max=59842, avg=36071.27, stdev=4813.36
     lat (usec): min=12946, max=60021, avg=36234.81, stdev=4813.21
    clat percentiles (usec):
     |  1.00th=[26870],  5.00th=[29492], 10.00th=[30802], 20.00th=[32113],
     | 30.00th=[33424], 40.00th=[34341], 50.00th=[35390], 60.00th=[36963],
     | 70.00th=[38011], 80.00th=[39584], 90.00th=[42206], 95.00th=[44827],
     | 99.00th=[49021], 99.50th=[51119], 99.90th=[58459], 99.95th=[60031],
     | 99.99th=[60031]
   bw (  KiB/s): min=  768, max= 1032, per=25.54%, avg=887.55, stdev=70.47, samples=11
   iops        : min=  192, max=  258, avg=221.82, stdev=17.61, samples=11
  lat (msec)   : 4=0.04%, 10=0.20%, 20=0.16%, 50=98.52%, 100=1.09%
  cpu          : usr=0.44%, sys=5.97%, ctx=4460, majf=0, minf=12
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=190678: Tue Apr 28 20:56:21 2026
  read: IOPS=207, BW=830KiB/s (850kB/s)(4984KiB/6005msec)
    slat (usec): min=1028, max=7981, avg=2627.38, stdev=840.67
    clat (usec): min=21555, max=59927, avg=35749.21, stdev=5553.11
     lat (usec): min=23720, max=62903, avg=38378.36, stdev=5564.60
    clat percentiles (usec):
     |  1.00th=[24773],  5.00th=[27395], 10.00th=[28967], 20.00th=[31065],
     | 30.00th=[32637], 40.00th=[33817], 50.00th=[35390], 60.00th=[36439],
     | 70.00th=[38011], 80.00th=[40633], 90.00th=[43254], 95.00th=[45876],
     | 99.00th=[50594], 99.50th=[52691], 99.90th=[54264], 99.95th=[60031],
     | 99.99th=[60031]
   bw (  KiB/s): min=  704, max=  984, per=24.94%, avg=832.18, stdev=93.93, samples=11
   iops        : min=  176, max=  246, avg=208.00, stdev=23.55, samples=11
  write: IOPS=218, BW=875KiB/s (896kB/s)(5256KiB/6005msec); 0 zone resets
    slat (usec): min=55, max=882, avg=161.41, stdev=86.08
    clat (usec): min=21992, max=59199, avg=35917.16, stdev=4852.81
     lat (usec): min=22140, max=59291, avg=36079.73, stdev=4849.88
    clat percentiles (usec):
     |  1.00th=[26870],  5.00th=[28705], 10.00th=[30278], 20.00th=[31851],
     | 30.00th=[33162], 40.00th=[34341], 50.00th=[35390], 60.00th=[36439],
     | 70.00th=[38011], 80.00th=[39584], 90.00th=[42730], 95.00th=[44827],
     | 99.00th=[49021], 99.50th=[52691], 99.90th=[58459], 99.95th=[58983],
     | 99.99th=[58983]
   bw (  KiB/s): min=  752, max= 1008, per=24.95%, avg=866.91, stdev=86.03, samples=11
   iops        : min=  188, max=  252, avg=216.73, stdev=21.51, samples=11
  lat (msec)   : 50=98.87%, 100=1.13%
  cpu          : usr=0.45%, sys=5.96%, ctx=4372, majf=0, minf=11
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=190679: Tue Apr 28 20:56:21 2026
  read: IOPS=211, BW=844KiB/s (864kB/s)(5060KiB/5995msec)
    slat (usec): min=1051, max=7831, avg=2638.60, stdev=886.16
    clat (usec): min=22733, max=57641, avg=35506.21, stdev=5385.67
     lat (usec): min=25232, max=61483, avg=38146.66, stdev=5492.69
    clat percentiles (usec):
     |  1.00th=[25035],  5.00th=[27657], 10.00th=[28967], 20.00th=[30802],
     | 30.00th=[32375], 40.00th=[33817], 50.00th=[34866], 60.00th=[36439],
     | 70.00th=[38011], 80.00th=[39584], 90.00th=[42206], 95.00th=[44827],
     | 99.00th=[51119], 99.50th=[52167], 99.90th=[57410], 99.95th=[57410],
     | 99.99th=[57410]
   bw (  KiB/s): min=  752, max=  928, per=25.22%, avg=841.73, stdev=57.96, samples=11
   iops        : min=  188, max=  232, avg=210.36, stdev=14.58, samples=11
  write: IOPS=216, BW=864KiB/s (885kB/s)(5180KiB/5995msec); 0 zone resets
    slat (usec): min=45, max=1521, avg=159.01, stdev=87.17
    clat (usec): min=21681, max=65402, avg=36173.95, stdev=4720.49
     lat (usec): min=21993, max=65547, avg=36334.16, stdev=4720.77
    clat percentiles (usec):
     |  1.00th=[27657],  5.00th=[29754], 10.00th=[30802], 20.00th=[32375],
     | 30.00th=[33424], 40.00th=[34341], 50.00th=[35390], 60.00th=[36963],
     | 70.00th=[38011], 80.00th=[40109], 90.00th=[42730], 95.00th=[44827],
     | 99.00th=[48497], 99.50th=[51643], 99.90th=[59507], 99.95th=[65274],
     | 99.99th=[65274]
   bw (  KiB/s): min=  791, max=  968, per=24.76%, avg=860.27, stdev=48.46, samples=11
   iops        : min=  197, max=  242, avg=215.00, stdev=12.22, samples=11
  lat (msec)   : 50=98.91%, 100=1.09%
  cpu          : usr=0.57%, sys=5.79%, ctx=4452, majf=0, minf=11
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=3338KiB/s (3418kB/s), 830KiB/s-844KiB/s (850kB/s-864kB/s), io=19.6MiB (20.5MB), run=5960-6012msec
  WRITE: bw=3475KiB/s (3558kB/s), 864KiB/s-883KiB/s (884kB/s-904kB/s), io=20.4MiB (21.4MB), run=5960-6012msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.19
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=190684: Tue Apr 28 20:56:24 2026
  read: IOPS=9, BW=385MiB/s (403MB/s)(40.0MiB/104msec)
    slat (nsec): min=8895.7k, max=8895.7k, avg=8895724.00, stdev= 0.00
    clat (nsec): min=90251k, max=90251k, avg=90251361.00, stdev= 0.00
     lat (nsec): min=99151k, max=99151k, avg=99150604.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[90702],  5.00th=[90702], 10.00th=[90702], 20.00th=[90702],
     | 30.00th=[90702], 40.00th=[90702], 50.00th=[90702], 60.00th=[90702],
     | 70.00th=[90702], 80.00th=[90702], 90.00th=[90702], 95.00th=[90702],
     | 99.00th=[90702], 99.50th=[90702], 99.90th=[90702], 99.95th=[90702],
     | 99.99th=[90702]
  lat (msec)   : 100=100.00%
  cpu          : usr=0.00%, sys=8.74%, ctx=10, majf=0, minf=9
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=385MiB/s (403MB/s), 385MiB/s-385MiB/s (403MB/s-403MB/s), io=40.0MiB (41.9MB), run=104-104msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout