== sanity test 398c: run fio to test AIO ================= 03:11:29 (1769760689)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB, 40 MiB) copied, 0.172152 s, 244 MB/s
osc.lustre-OST0000-osc-ffffa0ad87088800.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=182536: Fri Jan 30 03:11:41 2026
  write: IOPS=317, BW=1269KiB/s (1299kB/s)(10.0MiB/8072msec); 0 zone resets
    slat (usec): min=40, max=2390, avg=205.24, stdev=151.51
    clat (usec): min=9111, max=98676, avg=50172.47, stdev=9274.70
     lat (usec): min=9188, max=98940, avg=50379.00, stdev=9272.47
    clat percentiles (usec):
     |  1.00th=[31589],  5.00th=[35914], 10.00th=[38536], 20.00th=[42730],
     | 30.00th=[44827], 40.00th=[47449], 50.00th=[50070], 60.00th=[52167],
     | 70.00th=[54264], 80.00th=[57410], 90.00th=[61080], 95.00th=[64750],
     | 99.00th=[76022], 99.50th=[78119], 99.90th=[90702], 99.95th=[90702],
     | 99.99th=[99091]
   bw (  KiB/s): min= 1093, max= 1536, per=25.07%, avg=1261.13, stdev=109.33, samples=15
   iops        : min=  273, max=  384, avg=315.27, stdev=27.36, samples=15
  lat (msec)   : 10=0.04%, 20=0.31%, 50=49.65%, 100=50.00%
  cpu          : usr=0.57%, sys=4.58%, ctx=2800, majf=0, minf=12
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=182537: Fri Jan 30 03:11:41 2026
  write: IOPS=314, BW=1258KiB/s (1288kB/s)(10.0MiB/8141msec); 0 zone resets
    slat (usec): min=34, max=4006, avg=174.13, stdev=158.98
    clat (usec): min=28591, max=78206, avg=50274.85, stdev=8762.74
     lat (usec): min=28764, max=78442, avg=50450.16, stdev=8762.91
    clat percentiles (usec):
     |  1.00th=[32375],  5.00th=[36439], 10.00th=[39060], 20.00th=[42730],
     | 30.00th=[45351], 40.00th=[47973], 50.00th=[50070], 60.00th=[52167],
     | 70.00th=[53740], 80.00th=[57410], 90.00th=[61604], 95.00th=[65799],
     | 99.00th=[72877], 99.50th=[76022], 99.90th=[78119], 99.95th=[78119],
     | 99.99th=[78119]
   bw (  KiB/s): min=  926, max= 1528, per=25.01%, avg=1258.38, stdev=137.14, samples=16
   iops        : min=  231, max=  382, avg=314.56, stdev=34.37, samples=16
  lat (msec)   : 50=49.65%, 100=50.35%
  cpu          : usr=0.45%, sys=3.78%, ctx=2508, majf=0, minf=12
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=182538: Fri Jan 30 03:11:41 2026
  write: IOPS=315, BW=1263KiB/s (1293kB/s)(10.0MiB/8110msec); 0 zone resets
    slat (usec): min=35, max=19952, avg=198.86, stdev=415.78
    clat (usec): min=4170, max=81820, avg=50135.32, stdev=8962.94
     lat (usec): min=4627, max=81983, avg=50335.45, stdev=8941.56
    clat percentiles (usec):
     |  1.00th=[32375],  5.00th=[36963], 10.00th=[39060], 20.00th=[42730],
     | 30.00th=[44827], 40.00th=[47449], 50.00th=[50070], 60.00th=[52691],
     | 70.00th=[54264], 80.00th=[57410], 90.00th=[61604], 95.00th=[65799],
     | 99.00th=[74974], 99.50th=[77071], 99.90th=[81265], 99.95th=[81265],
     | 99.99th=[82314]
   bw (  KiB/s): min=  990, max= 1520, per=24.93%, avg=1254.27, stdev=127.16, samples=15
   iops        : min=  247, max=  380, avg=313.53, stdev=31.86, samples=15
  lat (msec)   : 10=0.12%, 20=0.08%, 50=50.04%, 100=49.77%
  cpu          : usr=0.52%, sys=4.43%, ctx=2657, majf=0, minf=11
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=182539: Fri Jan 30 03:11:41 2026
  write: IOPS=314, BW=1260KiB/s (1290kB/s)(10.0MiB/8127msec); 0 zone resets
    slat (usec): min=39, max=2082, avg=191.62, stdev=127.46
    clat (usec): min=8199, max=79002, avg=50198.78, stdev=8799.20
     lat (usec): min=8417, max=79187, avg=50391.69, stdev=8798.93
    clat percentiles (usec):
     |  1.00th=[32113],  5.00th=[36963], 10.00th=[39060], 20.00th=[42730],
     | 30.00th=[45351], 40.00th=[46924], 50.00th=[50070], 60.00th=[52167],
     | 70.00th=[54264], 80.00th=[57410], 90.00th=[61604], 95.00th=[64750],
     | 99.00th=[72877], 99.50th=[77071], 99.90th=[79168], 99.95th=[79168],
     | 99.99th=[79168]
   bw (  KiB/s): min=  927, max= 1536, per=24.89%, avg=1252.20, stdev=137.87, samples=15
   iops        : min=  231, max=  384, avg=313.00, stdev=34.59, samples=15
  lat (msec)   : 10=0.04%, 20=0.04%, 50=50.20%, 100=49.73%
  cpu          : usr=0.43%, sys=4.79%, ctx=2627, majf=0, minf=10
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=5031KiB/s (5152kB/s), 1258KiB/s-1269KiB/s (1288kB/s-1299kB/s), io=40.0MiB (41.9MB), run=8072-8141msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=182550: Fri Jan 30 03:11:52 2026
  read: IOPS=164, BW=657KiB/s (673kB/s)(5048KiB/7683msec)
    slat (usec): min=1254, max=10039, avg=3227.51, stdev=1130.95
    clat (usec): min=24312, max=88920, avg=45568.34, stdev=8652.25
     lat (usec): min=27836, max=92018, avg=48799.09, stdev=8885.67
    clat percentiles (usec):
     |  1.00th=[30016],  5.00th=[33817], 10.00th=[35914], 20.00th=[38536],
     | 30.00th=[40633], 40.00th=[42206], 50.00th=[44303], 60.00th=[46924],
     | 70.00th=[49546], 80.00th=[52167], 90.00th=[56886], 95.00th=[61604],
     | 99.00th=[70779], 99.50th=[73925], 99.90th=[84411], 99.95th=[88605],
     | 99.99th=[88605]
   bw (  KiB/s): min=  560, max=  744, per=25.12%, avg=655.87, stdev=57.95, samples=15
   iops        : min=  140, max=  186, avg=163.93, stdev=14.49, samples=15
  write: IOPS=168, BW=676KiB/s (692kB/s)(5192KiB/7683msec); 0 zone resets
    slat (usec): min=39, max=2414, avg=183.65, stdev=141.76
    clat (usec): min=15725, max=79112, avg=46421.39, stdev=8177.87
     lat (usec): min=15845, max=79316, avg=46606.22, stdev=8180.55
    clat percentiles (usec):
     |  1.00th=[31851],  5.00th=[35390], 10.00th=[37487], 20.00th=[40109],
     | 30.00th=[42206], 40.00th=[43779], 50.00th=[45351], 60.00th=[46400],
     | 70.00th=[49021], 80.00th=[52167], 90.00th=[57410], 95.00th=[62653],
     | 99.00th=[71828], 99.50th=[74974], 99.90th=[78119], 99.95th=[79168],
     | 99.99th=[79168]
   bw (  KiB/s): min=  544, max=  776, per=24.88%, avg=676.33, stdev=78.01, samples=15
   iops        : min=  136, max=  194, avg=169.07, stdev=19.51, samples=15
  lat (msec)   : 20=0.04%, 50=73.28%, 100=26.68%
  cpu          : usr=0.43%, sys=5.39%, ctx=4544, majf=0, minf=14
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=182551: Fri Jan 30 03:11:52 2026
  read: IOPS=161, BW=648KiB/s (663kB/s)(4976KiB/7684msec)
    slat (usec): min=1164, max=14169, avg=3276.82, stdev=1212.54
    clat (usec): min=7983, max=80707, avg=45421.39, stdev=8980.78
     lat (usec): min=13669, max=86840, avg=48701.12, stdev=9258.64
    clat percentiles (usec):
     |  1.00th=[29492],  5.00th=[33424], 10.00th=[35390], 20.00th=[38011],
     | 30.00th=[40109], 40.00th=[42206], 50.00th=[44303], 60.00th=[46400],
     | 70.00th=[49021], 80.00th=[51643], 90.00th=[57410], 95.00th=[62653],
     | 99.00th=[72877], 99.50th=[76022], 99.90th=[80217], 99.95th=[80217],
     | 99.99th=[80217]
   bw (  KiB/s): min=  576, max=  752, per=24.74%, avg=645.87, stdev=58.51, samples=15
   iops        : min=  144, max=  188, avg=161.47, stdev=14.63, samples=15
  write: IOPS=171, BW=685KiB/s (702kB/s)(5264KiB/7684msec); 0 zone resets
    slat (usec): min=52, max=3204, avg=176.33, stdev=131.95
    clat (usec): min=23089, max=80893, avg=46614.59, stdev=8196.34
     lat (usec): min=23262, max=81182, avg=46792.12, stdev=8205.32
    clat percentiles (usec):
     |  1.00th=[32375],  5.00th=[35914], 10.00th=[37487], 20.00th=[40109],
     | 30.00th=[41681], 40.00th=[43254], 50.00th=[45351], 60.00th=[47449],
     | 70.00th=[49546], 80.00th=[52691], 90.00th=[57410], 95.00th=[62653],
     | 99.00th=[71828], 99.50th=[74974], 99.90th=[78119], 99.95th=[81265],
     | 99.99th=[81265]
   bw (  KiB/s): min=  504, max=  848, per=25.26%, avg=686.47, stdev=99.43, samples=15
   iops        : min=  126, max=  212, avg=171.60, stdev=24.86, samples=15
  lat (msec)   : 10=0.04%, 20=0.04%, 50=72.93%, 100=26.99%
  cpu          : usr=0.53%, sys=5.15%, ctx=4531, majf=0, minf=14
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=182552: Fri Jan 30 03:11:52 2026
  read: IOPS=162, BW=652KiB/s (667kB/s)(4984KiB/7646msec)
    slat (usec): min=45, max=8996, avg=3208.67, stdev=1196.26
    clat (usec): min=25602, max=95744, avg=45477.03, stdev=8889.06
     lat (msec): min=28, max=100, avg=48.69, stdev= 9.10
    clat percentiles (usec):
     |  1.00th=[30016],  5.00th=[32900], 10.00th=[34866], 20.00th=[38011],
     | 30.00th=[40109], 40.00th=[42206], 50.00th=[44303], 60.00th=[46400],
     | 70.00th=[49546], 80.00th=[52167], 90.00th=[56886], 95.00th=[62129],
     | 99.00th=[70779], 99.50th=[76022], 99.90th=[84411], 99.95th=[95945],
     | 99.99th=[95945]
   bw (  KiB/s): min=  472, max=  760, per=24.96%, avg=651.60, stdev=76.97, samples=15
   iops        : min=  118, max=  190, avg=162.87, stdev=19.25, samples=15
  write: IOPS=171, BW=687KiB/s (704kB/s)(5256KiB/7646msec); 0 zone resets
    slat (usec): min=37, max=3382, avg=186.52, stdev=159.41
    clat (usec): min=13656, max=95170, avg=46595.97, stdev=8861.24
     lat (usec): min=13788, max=95290, avg=46784.16, stdev=8866.76
    clat percentiles (usec):
     |  1.00th=[31327],  5.00th=[34866], 10.00th=[36963], 20.00th=[39584],
     | 30.00th=[41681], 40.00th=[43254], 50.00th=[44827], 60.00th=[47449],
     | 70.00th=[49546], 80.00th=[52691], 90.00th=[58459], 95.00th=[62653],
     | 99.00th=[73925], 99.50th=[79168], 99.90th=[94897], 99.95th=[94897],
     | 99.99th=[94897]
   bw (  KiB/s): min=  560, max=  856, per=25.17%, avg=684.07, stdev=83.76, samples=15
   iops        : min=  140, max=  214, avg=171.00, stdev=20.93, samples=15
  lat (msec)   : 20=0.16%, 50=71.95%, 100=27.89%
  cpu          : usr=0.59%, sys=5.11%, ctx=4503, majf=0, minf=13
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=182553: Fri Jan 30 03:11:52 2026
  read: IOPS=164, BW=660KiB/s (676kB/s)(5060KiB/7669msec)
    slat (usec): min=1319, max=11530, avg=3243.28, stdev=1144.60
    clat (usec): min=12193, max=89488, avg=45404.80, stdev=9022.34
     lat (usec): min=18096, max=92322, avg=48651.69, stdev=9248.64
    clat percentiles (usec):
     |  1.00th=[29492],  5.00th=[32900], 10.00th=[35390], 20.00th=[38011],
     | 30.00th=[40109], 40.00th=[42206], 50.00th=[44303], 60.00th=[46400],
     | 70.00th=[49021], 80.00th=[52167], 90.00th=[57410], 95.00th=[62653],
     | 99.00th=[72877], 99.50th=[76022], 99.90th=[80217], 99.95th=[89654],
     | 99.99th=[89654]
   bw (  KiB/s): min=  520, max=  792, per=25.29%, avg=660.33, stdev=82.11, samples=15
   iops        : min=  130, max=  198, avg=165.07, stdev=20.53, samples=15
  write: IOPS=168, BW=675KiB/s (692kB/s)(5180KiB/7669msec); 0 zone resets
    slat (usec): min=36, max=2094, avg=191.23, stdev=147.05
    clat (msec): min=29, max=100, avg=46.54, stdev= 8.26
     lat (msec): min=29, max=100, avg=46.73, stdev= 8.27
    clat percentiles (msec):
     |  1.00th=[   33],  5.00th=[   36], 10.00th=[   38], 20.00th=[   40],
     | 30.00th=[   42], 40.00th=[   44], 50.00th=[   46], 60.00th=[   47],
     | 70.00th=[   50], 80.00th=[   53], 90.00th=[   58], 95.00th=[   63],
     | 99.00th=[   71], 99.50th=[   74], 99.90th=[  101], 99.95th=[  101],
     | 99.99th=[  101]
   bw (  KiB/s): min=  536, max=  816, per=24.87%, avg=676.00, stdev=71.57, samples=15
   iops        : min=  134, max=  204, avg=169.00, stdev=17.89, samples=15
  lat (msec)   : 20=0.04%, 50=72.85%, 100=27.07%, 250=0.04%
  cpu          : usr=0.53%, sys=5.31%, ctx=4549, majf=0, minf=13
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=2612KiB/s (2674kB/s), 648KiB/s-660KiB/s (663kB/s-676kB/s), io=19.6MiB (20.5MB), run=7646-7684msec
  WRITE: bw=2719KiB/s (2784kB/s), 675KiB/s-687KiB/s (692kB/s-704kB/s), io=20.4MiB (21.4MB), run=7646-7684msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.19
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=182558: Fri Jan 30 03:11:57 2026
  read: IOPS=4, BW=182MiB/s (191MB/s)(40.0MiB/220msec)
    slat (nsec): min=17274k, max=17274k, avg=17273890.00, stdev= 0.00
    clat (nsec): min=172204k, max=172204k, avg=172204116.00, stdev= 0.00
     lat (nsec): min=189482k, max=189482k, avg=189482018.00, stdev= 0.00
    clat percentiles (msec):
     |  1.00th=[  174],  5.00th=[  174], 10.00th=[  174], 20.00th=[  174],
     | 30.00th=[  174], 40.00th=[  174], 50.00th=[  174], 60.00th=[  174],
     | 70.00th=[  174], 80.00th=[  174], 90.00th=[  174], 95.00th=[  174],
     | 99.00th=[  174], 99.50th=[  174], 99.90th=[  174], 99.95th=[  174],
     | 99.99th=[  174]
  lat (msec)   : 250=100.00%
  cpu          : usr=0.00%, sys=5.48%, ctx=12, majf=0, minf=8
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=182MiB/s (191MB/s), 182MiB/s-182MiB/s (191MB/s-191MB/s), io=40.0MiB (41.9MB), run=220-220msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout