== sanity test 398c: run fio to test AIO ================= 21:49:11 (1778291351)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB, 40 MiB) copied, 0.138137 s, 304 MB/s
osc.lustre-OST0000-osc-ffff9a4a42d3e000.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=183715: Fri May  8 21:49:23 2026
  write: IOPS=332, BW=1328KiB/s (1360kB/s)(10.0MiB/7710msec); 0 zone resets
    slat (usec): min=36, max=4257, avg=179.87, stdev=144.80
    clat (usec): min=20391, max=76273, avg=47545.60, stdev=9602.64
     lat (usec): min=20488, max=76380, avg=47726.64, stdev=9599.25
    clat percentiles (usec):
     |  1.00th=[31065],  5.00th=[35390], 10.00th=[36963], 20.00th=[39060],
     | 30.00th=[41157], 40.00th=[43254], 50.00th=[45351], 60.00th=[47973],
     | 70.00th=[51119], 80.00th=[56361], 90.00th=[61604], 95.00th=[66847],
     | 99.00th=[72877], 99.50th=[76022], 99.90th=[76022], 99.95th=[76022],
     | 99.99th=[76022]
   bw (  KiB/s): min= 1028, max= 1529, per=24.98%, avg=1327.07, stdev=155.94, samples=15
   iops        : min=  257, max=  382, avg=331.73, stdev=38.95, samples=15
  lat (msec)   : 50=67.15%, 100=32.85%
  cpu          : usr=0.53%, sys=4.48%, ctx=2291, majf=0, minf=9
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=183716: Fri May  8 21:49:23 2026
  write: IOPS=332, BW=1329KiB/s (1361kB/s)(10.0MiB/7705msec); 0 zone resets
    slat (usec): min=38, max=3019, avg=180.75, stdev=142.21
    clat (usec): min=20958, max=81300, avg=47566.65, stdev=9397.28
     lat (usec): min=21323, max=81465, avg=47749.04, stdev=9397.42
    clat percentiles (usec):
     |  1.00th=[31327],  5.00th=[35914], 10.00th=[36963], 20.00th=[39584],
     | 30.00th=[41157], 40.00th=[43779], 50.00th=[45351], 60.00th=[47973],
     | 70.00th=[51643], 80.00th=[55837], 90.00th=[61604], 95.00th=[65274],
     | 99.00th=[73925], 99.50th=[76022], 99.90th=[76022], 99.95th=[80217],
     | 99.99th=[81265]
   bw (  KiB/s): min= 1041, max= 1529, per=25.03%, avg=1329.53, stdev=138.41, samples=15
   iops        : min=  260, max=  382, avg=332.33, stdev=34.60, samples=15
  lat (msec)   : 50=65.74%, 100=34.26%
  cpu          : usr=0.27%, sys=4.85%, ctx=2671, majf=0, minf=9
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=183717: Fri May  8 21:49:23 2026
  write: IOPS=332, BW=1331KiB/s (1363kB/s)(10.0MiB/7694msec); 0 zone resets
    slat (usec): min=36, max=29651, avg=201.24, stdev=602.70
    clat (usec): min=8860, max=81629, avg=47517.96, stdev=9506.47
     lat (usec): min=8958, max=81823, avg=47720.73, stdev=9494.22
    clat percentiles (usec):
     |  1.00th=[32113],  5.00th=[35390], 10.00th=[36963], 20.00th=[39584],
     | 30.00th=[41157], 40.00th=[43254], 50.00th=[45351], 60.00th=[48497],
     | 70.00th=[51643], 80.00th=[55837], 90.00th=[61080], 95.00th=[64750],
     | 99.00th=[72877], 99.50th=[74974], 99.90th=[81265], 99.95th=[81265],
     | 99.99th=[81265]
   bw (  KiB/s): min= 1080, max= 1529, per=25.10%, avg=1333.20, stdev=144.23, samples=15
   iops        : min=  270, max=  382, avg=333.27, stdev=36.02, samples=15
  lat (msec)   : 10=0.04%, 20=0.04%, 50=66.13%, 100=33.79%
  cpu          : usr=0.51%, sys=4.97%, ctx=2570, majf=0, minf=10
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=183718: Fri May  8 21:49:23 2026
  write: IOPS=333, BW=1335KiB/s (1367kB/s)(10.0MiB/7673msec); 0 zone resets
    slat (usec): min=45, max=2856, avg=180.35, stdev=139.32
    clat (usec): min=9794, max=82289, avg=47704.81, stdev=9666.35
     lat (usec): min=9919, max=82436, avg=47886.42, stdev=9665.17
    clat percentiles (usec):
     |  1.00th=[32113],  5.00th=[34866], 10.00th=[36963], 20.00th=[39584],
     | 30.00th=[41681], 40.00th=[43254], 50.00th=[45351], 60.00th=[48497],
     | 70.00th=[52167], 80.00th=[55837], 90.00th=[61080], 95.00th=[67634],
     | 99.00th=[72877], 99.50th=[74974], 99.90th=[81265], 99.95th=[82314],
     | 99.99th=[82314]
   bw (  KiB/s): min= 1072, max= 1537, per=25.17%, avg=1337.27, stdev=130.81, samples=15
   iops        : min=  268, max=  384, avg=334.27, stdev=32.69, samples=15
  lat (msec)   : 10=0.04%, 50=64.61%, 100=35.35%
  cpu          : usr=0.35%, sys=4.74%, ctx=2768, majf=0, minf=10
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=5313KiB/s (5440kB/s), 1328KiB/s-1335KiB/s (1360kB/s-1367kB/s), io=40.0MiB (41.9MB), run=7673-7710msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=183728: Fri May  8 21:49:33 2026
  read: IOPS=173, BW=695KiB/s (711kB/s)(5048KiB/7268msec)
    slat (usec): min=1082, max=10383, avg=2962.58, stdev=1205.95
    clat (usec): min=12459, max=89260, avg=42969.20, stdev=8859.62
     lat (usec): min=14619, max=91435, avg=45933.64, stdev=9205.63
    clat percentiles (usec):
     |  1.00th=[26608],  5.00th=[30540], 10.00th=[33162], 20.00th=[36439],
     | 30.00th=[38011], 40.00th=[40109], 50.00th=[41681], 60.00th=[43779],
     | 70.00th=[46400], 80.00th=[49021], 90.00th=[54264], 95.00th=[58983],
     | 99.00th=[71828], 99.50th=[73925], 99.90th=[86508], 99.95th=[89654],
     | 99.99th=[89654]
   bw (  KiB/s): min=  552, max=  840, per=25.30%, avg=697.14, stdev=88.57, samples=14
   iops        : min=  138, max=  210, avg=174.21, stdev=22.19, samples=14
  write: IOPS=178, BW=714KiB/s (732kB/s)(5192KiB/7268msec); 0 zone resets
    slat (usec): min=57, max=1937, avg=160.47, stdev=113.42
    clat (usec): min=25678, max=94713, avg=44180.21, stdev=8525.28
     lat (usec): min=25822, max=94811, avg=44342.89, stdev=8520.76
    clat percentiles (usec):
     |  1.00th=[29754],  5.00th=[32375], 10.00th=[34341], 20.00th=[36963],
     | 30.00th=[39060], 40.00th=[41157], 50.00th=[43254], 60.00th=[45351],
     | 70.00th=[47973], 80.00th=[50594], 90.00th=[55313], 95.00th=[60031],
     | 99.00th=[67634], 99.50th=[70779], 99.90th=[89654], 99.95th=[94897],
     | 99.99th=[94897]
   bw (  KiB/s): min=  616, max=  824, per=25.11%, avg=720.14, stdev=61.92, samples=14
   iops        : min=  154, max=  206, avg=179.93, stdev=15.52, samples=14
  lat (msec)   : 20=0.04%, 50=80.82%, 100=19.14%
  cpu          : usr=0.34%, sys=4.90%, ctx=4753, majf=0, minf=13
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=183729: Fri May  8 21:49:33 2026
  read: IOPS=170, BW=683KiB/s (700kB/s)(4976KiB/7282msec)
    slat (usec): min=1148, max=10713, avg=2943.73, stdev=1239.55
    clat (usec): min=18251, max=78459, avg=43422.65, stdev=8940.79
     lat (usec): min=22760, max=85529, avg=46368.35, stdev=9305.79
    clat percentiles (usec):
     |  1.00th=[27395],  5.00th=[30278], 10.00th=[33162], 20.00th=[35914],
     | 30.00th=[38011], 40.00th=[40109], 50.00th=[42206], 60.00th=[44303],
     | 70.00th=[46924], 80.00th=[50070], 90.00th=[54789], 95.00th=[60556],
     | 99.00th=[68682], 99.50th=[72877], 99.90th=[78119], 99.95th=[78119],
     | 99.99th=[78119]
   bw (  KiB/s): min=  576, max=  784, per=24.88%, avg=685.50, stdev=62.51, samples=14
   iops        : min=  144, max=  196, avg=171.29, stdev=15.68, samples=14
  write: IOPS=180, BW=723KiB/s (740kB/s)(5264KiB/7282msec); 0 zone resets
    slat (usec): min=40, max=3730, avg=166.31, stdev=153.11
    clat (usec): min=14413, max=74671, avg=43985.86, stdev=8312.70
     lat (usec): min=14549, max=74826, avg=44153.22, stdev=8316.33
    clat percentiles (usec):
     |  1.00th=[30016],  5.00th=[32900], 10.00th=[34341], 20.00th=[36963],
     | 30.00th=[38536], 40.00th=[40633], 50.00th=[42730], 60.00th=[44827],
     | 70.00th=[47449], 80.00th=[50594], 90.00th=[54789], 95.00th=[59507],
     | 99.00th=[68682], 99.50th=[71828], 99.90th=[73925], 99.95th=[74974],
     | 99.99th=[74974]
   bw (  KiB/s): min=  544, max=  864, per=25.29%, avg=725.43, stdev=95.43, samples=14
   iops        : min=  136, max=  216, avg=181.29, stdev=23.89, samples=14
  lat (msec)   : 20=0.08%, 50=79.02%, 100=20.90%
  cpu          : usr=0.49%, sys=4.86%, ctx=4655, majf=0, minf=13
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=183730: Fri May  8 21:49:33 2026
  read: IOPS=172, BW=689KiB/s (705kB/s)(4984KiB/7235msec)
    slat (usec): min=53, max=9206, avg=2908.76, stdev=1193.25
    clat (usec): min=5813, max=99043, avg=43033.69, stdev=9597.36
     lat (msec): min=5, max=102, avg=45.94, stdev= 9.92
    clat percentiles (usec):
     |  1.00th=[24249],  5.00th=[30540], 10.00th=[32900], 20.00th=[35914],
     | 30.00th=[38011], 40.00th=[40109], 50.00th=[42206], 60.00th=[44303],
     | 70.00th=[46400], 80.00th=[49546], 90.00th=[54264], 95.00th=[60556],
     | 99.00th=[70779], 99.50th=[78119], 99.90th=[96994], 99.95th=[99091],
     | 99.99th=[99091]
   bw (  KiB/s): min=  528, max=  840, per=25.20%, avg=694.21, stdev=82.62, samples=14
   iops        : min=  132, max=  210, avg=173.43, stdev=20.65, samples=14
  write: IOPS=181, BW=726KiB/s (744kB/s)(5256KiB/7235msec); 0 zone resets
    slat (usec): min=36, max=3650, avg=173.45, stdev=189.39
    clat (msec): min=22, max=107, avg=44.18, stdev= 9.27
     lat (msec): min=22, max=107, avg=44.36, stdev= 9.28
    clat percentiles (msec):
     |  1.00th=[   30],  5.00th=[   33], 10.00th=[   35], 20.00th=[   37],
     | 30.00th=[   39], 40.00th=[   41], 50.00th=[   43], 60.00th=[   45],
     | 70.00th=[   48], 80.00th=[   51], 90.00th=[   56], 95.00th=[   61],
     | 99.00th=[   69], 99.50th=[   88], 99.90th=[  108], 99.95th=[  108],
     | 99.99th=[  108]
   bw (  KiB/s): min=  606, max=  880, per=25.38%, avg=727.86, stdev=87.64, samples=14
   iops        : min=  151, max=  220, avg=181.86, stdev=22.00, samples=14
  lat (msec)   : 10=0.16%, 20=0.23%, 50=79.80%, 100=19.65%, 250=0.16%
  cpu          : usr=0.44%, sys=4.81%, ctx=4651, majf=0, minf=14
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=183731: Fri May  8 21:49:33 2026
  read: IOPS=173, BW=695KiB/s (711kB/s)(5060KiB/7284msec)
    slat (usec): min=1218, max=10488, avg=2946.69, stdev=1232.49
    clat (usec): min=3015, max=79133, avg=43380.05, stdev=9103.45
     lat (usec): min=7626, max=86197, avg=46328.32, stdev=9424.09
    clat percentiles (usec):
     |  1.00th=[25297],  5.00th=[31065], 10.00th=[33424], 20.00th=[35914],
     | 30.00th=[38011], 40.00th=[40109], 50.00th=[42206], 60.00th=[44303],
     | 70.00th=[46924], 80.00th=[50594], 90.00th=[55313], 95.00th=[58983],
     | 99.00th=[70779], 99.50th=[73925], 99.90th=[79168], 99.95th=[79168],
     | 99.99th=[79168]
   bw (  KiB/s): min=  582, max=  808, per=25.23%, avg=695.00, stdev=74.90, samples=14
   iops        : min=  145, max=  202, avg=173.64, stdev=18.83, samples=14
  write: IOPS=177, BW=711KiB/s (728kB/s)(5180KiB/7284msec); 0 zone resets
    slat (usec): min=52, max=1071, avg=163.79, stdev=105.00
    clat (usec): min=7633, max=75550, avg=44011.66, stdev=8366.00
     lat (usec): min=7795, max=75680, avg=44177.30, stdev=8369.84
    clat percentiles (usec):
     |  1.00th=[29230],  5.00th=[32375], 10.00th=[33817], 20.00th=[36963],
     | 30.00th=[39060], 40.00th=[41157], 50.00th=[43254], 60.00th=[45351],
     | 70.00th=[47973], 80.00th=[50594], 90.00th=[55313], 95.00th=[59507],
     | 99.00th=[65274], 99.50th=[67634], 99.90th=[70779], 99.95th=[76022],
     | 99.99th=[76022]
   bw (  KiB/s): min=  544, max=  848, per=24.72%, avg=708.86, stdev=84.71, samples=14
   iops        : min=  136, max=  212, avg=177.14, stdev=21.17, samples=14
  lat (msec)   : 4=0.04%, 10=0.08%, 20=0.27%, 50=78.16%, 100=21.45%
  cpu          : usr=0.52%, sys=4.75%, ctx=4717, majf=0, minf=13
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=2755KiB/s (2821kB/s), 683KiB/s-695KiB/s (700kB/s-711kB/s), io=19.6MiB (20.5MB), run=7235-7284msec
  WRITE: bw=2868KiB/s (2937kB/s), 711KiB/s-726KiB/s (728kB/s-744kB/s), io=20.4MiB (21.4MB), run=7235-7284msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.19
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=183736: Fri May  8 21:49:37 2026
  read: IOPS=9, BW=360MiB/s (378MB/s)(40.0MiB/111msec)
    slat (nsec): min=9495.6k, max=9495.6k, avg=9495606.00, stdev= 0.00
    clat (nsec): min=96574k, max=96574k, avg=96573628.00, stdev= 0.00
     lat (nsec): min=106073k, max=106073k, avg=106073317.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[96994],  5.00th=[96994], 10.00th=[96994], 20.00th=[96994],
     | 30.00th=[96994], 40.00th=[96994], 50.00th=[96994], 60.00th=[96994],
     | 70.00th=[96994], 80.00th=[96994], 90.00th=[96994], 95.00th=[96994],
     | 99.00th=[96994], 99.50th=[96994], 99.90th=[96994], 99.95th=[96994],
     | 99.99th=[96994]
  lat (msec)   : 100=100.00%
  cpu          : usr=0.00%, sys=8.18%, ctx=7, majf=0, minf=8
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=360MiB/s (378MB/s), 360MiB/s-360MiB/s (378MB/s-378MB/s), io=40.0MiB (41.9MB), run=111-111msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout