== sanity test 398c: run fio to test AIO ================= 11:34:01 (1775835241)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB, 40 MiB) copied, 0.131159 s, 320 MB/s
osc.lustre-OST0000-osc-ffff8db785277000.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=183689: Fri Apr 10 11:34:11 2026
  write: IOPS=381, BW=1525KiB/s (1562kB/s)(10.0MiB/6714msec); 0 zone resets
    slat (usec): min=38, max=26340, avg=203.78, stdev=538.73
    clat (usec): min=6010, max=86862, avg=41428.43, stdev=9972.86
     lat (usec): min=6409, max=87122, avg=41633.43, stdev=9954.03
    clat percentiles (usec):
     |  1.00th=[24773],  5.00th=[28705], 10.00th=[30278], 20.00th=[33817],
     | 30.00th=[35914], 40.00th=[38011], 50.00th=[39584], 60.00th=[41681],
     | 70.00th=[43779], 80.00th=[48497], 90.00th=[54789], 95.00th=[61604],
     | 99.00th=[73925], 99.50th=[78119], 99.90th=[80217], 99.95th=[81265],
     | 99.99th=[86508]
   bw (  KiB/s): min= 1165, max= 1865, per=24.98%, avg=1519.23, stdev=180.41, samples=13
   iops        : min=  291, max=  466, avg=379.77, stdev=45.10, samples=13
  lat (msec)   : 10=0.16%, 20=0.16%, 50=82.66%, 100=17.03%
  cpu          : usr=0.43%, sys=6.03%, ctx=2593, majf=0, minf=10
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=183690: Fri Apr 10 11:34:11 2026
  write: IOPS=380, BW=1521KiB/s (1557kB/s)(10.0MiB/6733msec); 0 zone resets
    slat (usec): min=37, max=2804, avg=186.54, stdev=133.79
    clat (usec): min=19665, max=87011, avg=41525.81, stdev=9752.85
     lat (usec): min=20113, max=87196, avg=41713.47, stdev=9768.48
    clat percentiles (usec):
     |  1.00th=[26870],  5.00th=[29230], 10.00th=[30016], 20.00th=[33817],
     | 30.00th=[36439], 40.00th=[38011], 50.00th=[40109], 60.00th=[41681],
     | 70.00th=[44303], 80.00th=[47973], 90.00th=[54789], 95.00th=[61080],
     | 99.00th=[72877], 99.50th=[78119], 99.90th=[86508], 99.95th=[86508],
     | 99.99th=[86508]
   bw (  KiB/s): min= 1157, max= 1840, per=24.88%, avg=1513.00, stdev=176.35, samples=13
   iops        : min=  289, max=  460, avg=378.23, stdev=44.13, samples=13
  lat (msec)   : 20=0.04%, 50=83.83%, 100=16.13%
  cpu          : usr=0.46%, sys=5.61%, ctx=2668, majf=0, minf=10
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=183691: Fri Apr 10 11:34:11 2026
  write: IOPS=380, BW=1521KiB/s (1557kB/s)(10.0MiB/6734msec); 0 zone resets
    slat (usec): min=35, max=3462, avg=169.71, stdev=134.27
    clat (usec): min=3605, max=87219, avg=41526.37, stdev=9908.88
     lat (usec): min=4031, max=87303, avg=41697.25, stdev=9924.07
    clat percentiles (usec):
     |  1.00th=[26608],  5.00th=[28705], 10.00th=[30802], 20.00th=[33817],
     | 30.00th=[35914], 40.00th=[38011], 50.00th=[39584], 60.00th=[41681],
     | 70.00th=[44303], 80.00th=[47973], 90.00th=[55837], 95.00th=[61604],
     | 99.00th=[71828], 99.50th=[76022], 99.90th=[87557], 99.95th=[87557],
     | 99.99th=[87557]
   bw (  KiB/s): min= 1165, max= 1750, per=24.84%, avg=1511.00, stdev=169.74, samples=13
   iops        : min=  291, max=  437, avg=377.69, stdev=42.42, samples=13
  lat (msec)   : 4=0.04%, 10=0.08%, 20=0.04%, 50=82.46%, 100=17.38%
  cpu          : usr=0.43%, sys=4.95%, ctx=2448, majf=0, minf=12
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=183692: Fri Apr 10 11:34:11 2026
  write: IOPS=382, BW=1530KiB/s (1567kB/s)(10.0MiB/6693msec); 0 zone resets
    slat (usec): min=39, max=3837, avg=200.58, stdev=148.08
    clat (usec): min=3830, max=82451, avg=41542.89, stdev=10068.52
     lat (usec): min=4058, max=82647, avg=41744.68, stdev=10065.83
    clat percentiles (usec):
     |  1.00th=[26084],  5.00th=[28705], 10.00th=[30540], 20.00th=[33817],
     | 30.00th=[35914], 40.00th=[38011], 50.00th=[39584], 60.00th=[41681],
     | 70.00th=[44303], 80.00th=[49021], 90.00th=[55313], 95.00th=[61604],
     | 99.00th=[73925], 99.50th=[78119], 99.90th=[81265], 99.95th=[81265],
     | 99.99th=[82314]
   bw (  KiB/s): min= 1181, max= 1929, per=25.08%, avg=1525.38, stdev=192.41, samples=13
   iops        : min=  295, max=  482, avg=381.31, stdev=48.10, samples=13
  lat (msec)   : 4=0.04%, 10=0.08%, 20=0.12%, 50=82.15%, 100=17.62%
  cpu          : usr=0.57%, sys=5.68%, ctx=2685, majf=0, minf=10
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=6083KiB/s (6229kB/s), 1521KiB/s-1530KiB/s (1557kB/s-1567kB/s), io=40.0MiB (41.9MB), run=6693-6734msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=183702: Fri Apr 10 11:34:20 2026
  read: IOPS=198, BW=796KiB/s (815kB/s)(5048KiB/6342msec)
    slat (usec): min=54, max=7116, avg=2574.49, stdev=876.75
    clat (usec): min=5531, max=74497, avg=37784.05, stdev=7344.72
     lat (usec): min=5758, max=77359, avg=40360.55, stdev=7604.38
    clat percentiles (usec):
     |  1.00th=[22414],  5.00th=[28443], 10.00th=[30540], 20.00th=[32900],
     | 30.00th=[34341], 40.00th=[35390], 50.00th=[36963], 60.00th=[38536],
     | 70.00th=[40109], 80.00th=[42206], 90.00th=[45876], 95.00th=[50594],
     | 99.00th=[62129], 99.50th=[63701], 99.90th=[72877], 99.95th=[74974],
     | 99.99th=[74974]
   bw (  KiB/s): min=  672, max=  912, per=25.44%, avg=798.92, stdev=75.45, samples=12
   iops        : min=  168, max=  228, avg=199.67, stdev=18.87, samples=12
  write: IOPS=204, BW=819KiB/s (838kB/s)(5192KiB/6342msec); 0 zone resets
    slat (usec): min=43, max=813, avg=167.95, stdev=80.37
    clat (usec): min=23140, max=72799, avg=38612.08, stdev=6320.31
     lat (usec): min=23286, max=72936, avg=38781.09, stdev=6321.15
    clat percentiles (usec):
     |  1.00th=[27395],  5.00th=[30540], 10.00th=[31851], 20.00th=[33817],
     | 30.00th=[35390], 40.00th=[36439], 50.00th=[37487], 60.00th=[39060],
     | 70.00th=[40109], 80.00th=[42730], 90.00th=[46400], 95.00th=[50070],
     | 99.00th=[61604], 99.50th=[64226], 99.90th=[69731], 99.95th=[72877],
     | 99.99th=[72877]
   bw (  KiB/s): min=  720, max=  936, per=25.06%, avg=819.00, stdev=70.14, samples=12
   iops        : min=  180, max=  234, avg=204.75, stdev=17.54, samples=12
  lat (msec)   : 10=0.39%, 20=0.04%, 50=94.38%, 100=5.20%
  cpu          : usr=0.47%, sys=6.29%, ctx=4764, majf=0, minf=15
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=183703: Fri Apr 10 11:34:20 2026
  read: IOPS=194, BW=779KiB/s (797kB/s)(4976KiB/6391msec)
    slat (usec): min=1093, max=9201, avg=2628.90, stdev=911.93
    clat (usec): min=5202, max=64171, avg=38060.87, stdev=6686.26
     lat (usec): min=10702, max=70593, avg=40692.05, stdev=6887.34
    clat percentiles (usec):
     |  1.00th=[26346],  5.00th=[29230], 10.00th=[30802], 20.00th=[32900],
     | 30.00th=[34341], 40.00th=[35914], 50.00th=[36963], 60.00th=[38536],
     | 70.00th=[40109], 80.00th=[42206], 90.00th=[46400], 95.00th=[50594],
     | 99.00th=[60031], 99.50th=[61080], 99.90th=[64226], 99.95th=[64226],
     | 99.99th=[64226]
   bw (  KiB/s): min=  645, max=  872, per=24.83%, avg=779.75, stdev=73.38, samples=12
   iops        : min=  161, max=  218, avg=194.92, stdev=18.39, samples=12
  write: IOPS=205, BW=824KiB/s (843kB/s)(5264KiB/6391msec); 0 zone resets
    slat (usec): min=39, max=968, avg=165.70, stdev=86.11
    clat (usec): min=14483, max=69714, avg=38588.39, stdev=6090.22
     lat (usec): min=14762, max=69895, avg=38755.11, stdev=6088.81
    clat percentiles (usec):
     |  1.00th=[27919],  5.00th=[30802], 10.00th=[31851], 20.00th=[33817],
     | 30.00th=[34866], 40.00th=[36439], 50.00th=[37487], 60.00th=[39060],
     | 70.00th=[40633], 80.00th=[42730], 90.00th=[46400], 95.00th=[50070],
     | 99.00th=[57934], 99.50th=[60031], 99.90th=[63701], 99.95th=[69731],
     | 99.99th=[69731]
   bw (  KiB/s): min=  688, max= 1032, per=25.33%, avg=827.67, stdev=91.87, samples=12
   iops        : min=  172, max=  258, avg=206.92, stdev=22.97, samples=12
  lat (msec)   : 10=0.04%, 20=0.08%, 50=94.65%, 100=5.23%
  cpu          : usr=0.49%, sys=6.06%, ctx=4697, majf=0, minf=13
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=183704: Fri Apr 10 11:34:20 2026
  read: IOPS=194, BW=780KiB/s (799kB/s)(4984KiB/6391msec)
    slat (usec): min=1126, max=7242, avg=2582.90, stdev=871.04
    clat (usec): min=21992, max=67031, avg=38116.35, stdev=6402.77
     lat (usec): min=23925, max=69224, avg=40701.47, stdev=6563.84
    clat percentiles (usec):
     |  1.00th=[26084],  5.00th=[29230], 10.00th=[30802], 20.00th=[32900],
     | 30.00th=[34341], 40.00th=[35914], 50.00th=[37487], 60.00th=[39060],
     | 70.00th=[40633], 80.00th=[42730], 90.00th=[45351], 95.00th=[49546],
     | 99.00th=[58983], 99.50th=[61604], 99.90th=[64750], 99.95th=[66847],
     | 99.99th=[66847]
   bw (  KiB/s): min=  648, max=  952, per=25.02%, avg=785.75, stdev=83.85, samples=12
   iops        : min=  162, max=  238, avg=196.42, stdev=20.99, samples=12
  write: IOPS=205, BW=822KiB/s (842kB/s)(5256KiB/6391msec); 0 zone resets
    slat (usec): min=48, max=978, avg=168.75, stdev=87.51
    clat (usec): min=18937, max=64352, avg=38625.90, stdev=6321.94
     lat (usec): min=19106, max=64433, avg=38796.14, stdev=6321.57
    clat percentiles (usec):
     |  1.00th=[28181],  5.00th=[30278], 10.00th=[31589], 20.00th=[33424],
     | 30.00th=[34866], 40.00th=[36439], 50.00th=[37487], 60.00th=[39060],
     | 70.00th=[40633], 80.00th=[43254], 90.00th=[46400], 95.00th=[50594],
     | 99.00th=[58983], 99.50th=[61080], 99.90th=[63177], 99.95th=[64226],
     | 99.99th=[64226]
   bw (  KiB/s): min=  722, max=  944, per=25.04%, avg=818.17, stdev=65.54, samples=12
   iops        : min=  180, max=  236, avg=204.50, stdev=16.45, samples=12
  lat (msec)   : 20=0.04%, 50=94.84%, 100=5.12%
  cpu          : usr=0.50%, sys=6.06%, ctx=4794, majf=0, minf=15
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=183705: Fri Apr 10 11:34:20 2026
  read: IOPS=198, BW=792KiB/s (811kB/s)(5060KiB/6387msec)
    slat (usec): min=1096, max=7988, avg=2588.13, stdev=827.05
    clat (usec): min=23607, max=65046, avg=38083.62, stdev=6632.13
     lat (usec): min=26137, max=69696, avg=40674.28, stdev=6782.33
    clat percentiles (usec):
     |  1.00th=[26346],  5.00th=[28967], 10.00th=[30540], 20.00th=[32900],
     | 30.00th=[34341], 40.00th=[35914], 50.00th=[37487], 60.00th=[38536],
     | 70.00th=[40109], 80.00th=[42206], 90.00th=[46400], 95.00th=[50594],
     | 99.00th=[58459], 99.50th=[61604], 99.90th=[64750], 99.95th=[65274],
     | 99.99th=[65274]
   bw (  KiB/s): min=  680, max=  904, per=25.27%, avg=793.33, stdev=76.34, samples=12
   iops        : min=  170, max=  226, avg=198.33, stdev=19.09, samples=12
  write: IOPS=202, BW=811KiB/s (830kB/s)(5180KiB/6387msec); 0 zone resets
    slat (usec): min=55, max=1667, avg=173.18, stdev=114.32
    clat (usec): min=25261, max=69868, avg=38627.18, stdev=6258.45
     lat (usec): min=25407, max=70017, avg=38801.49, stdev=6257.31
    clat percentiles (usec):
     |  1.00th=[28181],  5.00th=[30540], 10.00th=[31851], 20.00th=[33817],
     | 30.00th=[34866], 40.00th=[36439], 50.00th=[37487], 60.00th=[39060],
     | 70.00th=[40633], 80.00th=[42730], 90.00th=[46400], 95.00th=[51119],
     | 99.00th=[59507], 99.50th=[61604], 99.90th=[64226], 99.95th=[69731],
     | 99.99th=[69731]
   bw (  KiB/s): min=  720, max=  936, per=24.89%, avg=813.25, stdev=65.93, samples=12
   iops        : min=  180, max=  234, avg=203.25, stdev=16.56, samples=12
  lat (msec)   : 50=94.38%, 100=5.62%
  cpu          : usr=0.60%, sys=6.15%, ctx=4755, majf=0, minf=12
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=3140KiB/s (3215kB/s), 779KiB/s-796KiB/s (797kB/s-815kB/s), io=19.6MiB (20.5MB), run=6342-6391msec
  WRITE: bw=3269KiB/s (3347kB/s), 811KiB/s-824KiB/s (830kB/s-843kB/s), io=20.4MiB (21.4MB), run=6342-6391msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.19
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=183710: Fri Apr 10 11:34:24 2026
  read: IOPS=7, BW=294MiB/s (308MB/s)(40.0MiB/136msec)
    slat (nsec): min=12463k, max=12463k, avg=12462952.00, stdev= 0.00
    clat (nsec): min=117822k, max=117822k, avg=117821667.00, stdev= 0.00
     lat (nsec): min=130289k, max=130289k, avg=130289149.00, stdev= 0.00
    clat percentiles (msec):
     |  1.00th=[  118],  5.00th=[  118], 10.00th=[  118], 20.00th=[  118],
     | 30.00th=[  118], 40.00th=[  118], 50.00th=[  118], 60.00th=[  118],
     | 70.00th=[  118], 80.00th=[  118], 90.00th=[  118], 95.00th=[  118],
     | 99.00th=[  118], 99.50th=[  118], 99.90th=[  118], 99.95th=[  118],
     | 99.99th=[  118]
  lat (msec)   : 250=100.00%
  cpu          : usr=0.00%, sys=8.15%, ctx=11, majf=0, minf=8
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=294MiB/s (308MB/s), 294MiB/s-294MiB/s (308MB/s-308MB/s), io=40.0MiB (41.9MB), run=136-136msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout