== sanity test 398c: run fio to test AIO ================= 12:14:45 (1790957685)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB, 40 MiB) copied, 0.159265 s, 263 MB/s
osc.lustre-OST0000-osc-ffff9d7904a52800.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=182392: Fri Oct  2 12:14:57 2026
  write: IOPS=322, BW=1288KiB/s (1319kB/s)(10.0MiB/7949msec); 0 zone resets
    slat (usec): min=22, max=1992, avg=120.92, stdev=97.34
    clat (usec): min=9750, max=88004, avg=49420.44, stdev=11818.23
     lat (usec): min=9908, max=88150, avg=49542.54, stdev=11822.38
    clat percentiles (usec):
     |  1.00th=[26346],  5.00th=[33817], 10.00th=[36439], 20.00th=[39060],
     | 30.00th=[41681], 40.00th=[44303], 50.00th=[47449], 60.00th=[50594],
     | 70.00th=[54789], 80.00th=[59507], 90.00th=[66323], 95.00th=[70779],
     | 99.00th=[80217], 99.50th=[81265], 99.90th=[87557], 99.95th=[87557],
     | 99.99th=[87557]
   bw (  KiB/s): min=  880, max= 1647, per=24.81%, avg=1273.00, stdev=219.78, samples=15
   iops        : min=  220, max=  411, avg=318.20, stdev=54.85, samples=15
  lat (msec)   : 10=0.04%, 20=0.16%, 50=58.24%, 100=41.56%
  cpu          : usr=0.44%, sys=3.17%, ctx=2695, majf=0, minf=8
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=182393: Fri Oct  2 12:14:57 2026
  write: IOPS=322, BW=1288KiB/s (1319kB/s)(10.0MiB/7949msec); 0 zone resets
    slat (usec): min=29, max=1224, avg=119.40, stdev=94.53
    clat (usec): min=10674, max=87655, avg=49499.49, stdev=11706.73
     lat (usec): min=10804, max=87785, avg=49620.08, stdev=11712.37
    clat percentiles (usec):
     |  1.00th=[26608],  5.00th=[33817], 10.00th=[36963], 20.00th=[39584],
     | 30.00th=[41681], 40.00th=[44303], 50.00th=[47449], 60.00th=[50594],
     | 70.00th=[55313], 80.00th=[59507], 90.00th=[66323], 95.00th=[70779],
     | 99.00th=[80217], 99.50th=[82314], 99.90th=[84411], 99.95th=[84411],
     | 99.99th=[87557]
   bw (  KiB/s): min=  896, max= 1653, per=24.77%, avg=1270.73, stdev=222.75, samples=15
   iops        : min=  224, max=  413, avg=317.67, stdev=55.66, samples=15
  lat (msec)   : 20=0.20%, 50=57.50%, 100=42.30%
  cpu          : usr=0.65%, sys=2.94%, ctx=2709, majf=0, minf=9
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=182394: Fri Oct  2 12:14:57 2026
  write: IOPS=320, BW=1283KiB/s (1314kB/s)(10.0MiB/7981msec); 0 zone resets
    slat (usec): min=26, max=1631, avg=112.42, stdev=93.05
    clat (usec): min=7061, max=87851, avg=49426.32, stdev=11564.15
     lat (usec): min=7412, max=87984, avg=49539.91, stdev=11572.78
    clat percentiles (usec):
     |  1.00th=[29492],  5.00th=[34341], 10.00th=[37487], 20.00th=[39584],
     | 30.00th=[41681], 40.00th=[44303], 50.00th=[47449], 60.00th=[51119],
     | 70.00th=[54789], 80.00th=[58983], 90.00th=[65799], 95.00th=[71828],
     | 99.00th=[79168], 99.50th=[81265], 99.90th=[85459], 99.95th=[86508],
     | 99.99th=[87557]
   bw (  KiB/s): min=  888, max= 1624, per=24.56%, avg=1259.87, stdev=202.80, samples=15
   iops        : min=  222, max=  406, avg=314.93, stdev=50.66, samples=15
  lat (msec)   : 10=0.08%, 20=0.27%, 50=56.45%, 100=43.20%
  cpu          : usr=0.38%, sys=3.22%, ctx=2667, majf=0, minf=9
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=182395: Fri Oct  2 12:14:57 2026
  write: IOPS=320, BW=1283KiB/s (1314kB/s)(10.0MiB/7983msec); 0 zone resets
    slat (usec): min=28, max=25682, avg=124.01, stdev=515.54
    clat (usec): min=8399, max=88492, avg=49468.13, stdev=11614.31
     lat (usec): min=23046, max=88592, avg=49593.26, stdev=11595.41
    clat percentiles (usec):
     |  1.00th=[26608],  5.00th=[33817], 10.00th=[36963], 20.00th=[39060],
     | 30.00th=[41681], 40.00th=[44827], 50.00th=[47449], 60.00th=[50594],
     | 70.00th=[54789], 80.00th=[59507], 90.00th=[66847], 95.00th=[70779],
     | 99.00th=[79168], 99.50th=[81265], 99.90th=[83362], 99.95th=[87557],
     | 99.99th=[88605]
   bw (  KiB/s): min=  896, max= 1552, per=24.62%, avg=1263.07, stdev=209.61, samples=15
   iops        : min=  224, max=  388, avg=315.73, stdev=52.36, samples=15
  lat (msec)   : 10=0.04%, 50=58.55%, 100=41.41%
  cpu          : usr=0.49%, sys=3.12%, ctx=2686, majf=0, minf=8
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=5131KiB/s (5254kB/s), 1283KiB/s-1288KiB/s (1314kB/s-1319kB/s), io=40.0MiB (41.9MB), run=7949-7983msec
7% of I/O are 1-page (PAGE_SIZE=4096), max 50%
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=182406: Fri Oct  2 12:15:08 2026
  read: IOPS=162, BW=650KiB/s (665kB/s)(5048KiB/7771msec)
    slat (usec): min=31, max=12322, avg=3154.66, stdev=1120.42
    clat (usec): min=10132, max=92895, avg=46389.16, stdev=10318.29
     lat (usec): min=10212, max=95385, avg=49547.09, stdev=10747.91
    clat percentiles (usec):
     |  1.00th=[24511],  5.00th=[32900], 10.00th=[35390], 20.00th=[38536],
     | 30.00th=[40633], 40.00th=[42730], 50.00th=[44827], 60.00th=[47449],
     | 70.00th=[50070], 80.00th=[54264], 90.00th=[60556], 95.00th=[65274],
     | 99.00th=[72877], 99.50th=[79168], 99.90th=[90702], 99.95th=[92799],
     | 99.99th=[92799]
   bw (  KiB/s): min=  464, max=  760, per=25.45%, avg=649.00, stdev=96.23, samples=15
   iops        : min=  116, max=  190, avg=162.20, stdev=24.10, samples=15
  write: IOPS=167, BW=668KiB/s (684kB/s)(5192KiB/7771msec); 0 zone resets
    slat (usec): min=29, max=1816, avg=121.01, stdev=102.04
    clat (usec): min=19742, max=90459, avg=47335.48, stdev=9518.79
     lat (usec): min=19888, max=90526, avg=47457.65, stdev=9522.97
    clat percentiles (usec):
     |  1.00th=[24773],  5.00th=[35390], 10.00th=[36963], 20.00th=[39584],
     | 30.00th=[41157], 40.00th=[43254], 50.00th=[45876], 60.00th=[47973],
     | 70.00th=[51119], 80.00th=[55313], 90.00th=[61080], 95.00th=[64750],
     | 99.00th=[70779], 99.50th=[74974], 99.90th=[87557], 99.95th=[90702],
     | 99.99th=[90702]
   bw (  KiB/s): min=  534, max=  784, per=25.25%, avg=670.47, stdev=83.14, samples=15
   iops        : min=  133, max=  196, avg=167.53, stdev=20.86, samples=15
  lat (msec)   : 20=0.47%, 50=67.07%, 100=32.46%
  cpu          : usr=0.54%, sys=4.21%, ctx=4661, majf=0, minf=12
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=182407: Fri Oct  2 12:15:08 2026
  read: IOPS=158, BW=632KiB/s (648kB/s)(4976KiB/7868msec)
    slat (usec): min=34, max=69316, avg=3237.07, stdev=3461.40
    clat (msec): min=3, max=270, avg=46.27, stdev=16.66
     lat (msec): min=5, max=277, avg=49.51, stdev=17.98
    clat percentiles (msec):
     |  1.00th=[   13],  5.00th=[   33], 10.00th=[   35], 20.00th=[   39],
     | 30.00th=[   41], 40.00th=[   43], 50.00th=[   45], 60.00th=[   47],
     | 70.00th=[   50], 80.00th=[   54], 90.00th=[   60], 95.00th=[   64],
     | 99.00th=[   77], 99.50th=[  115], 99.90th=[  271], 99.95th=[  271],
     | 99.99th=[  271]
   bw (  KiB/s): min=  320, max=  776, per=24.45%, avg=623.47, stdev=135.06, samples=15
   iops        : min=   80, max=  194, avg=155.80, stdev=33.80, samples=15
  write: IOPS=167, BW=669KiB/s (685kB/s)(5264KiB/7868msec); 0 zone resets
    slat (usec): min=29, max=1765, avg=116.15, stdev=106.07
    clat (msec): min=7, max=279, avg=48.35, stdev=21.31
     lat (msec): min=8, max=279, avg=48.47, stdev=21.31
    clat percentiles (msec):
     |  1.00th=[   26],  5.00th=[   35], 10.00th=[   37], 20.00th=[   40],
     | 30.00th=[   42], 40.00th=[   43], 50.00th=[   45], 60.00th=[   48],
     | 70.00th=[   51], 80.00th=[   55], 90.00th=[   62], 95.00th=[   67],
     | 99.00th=[   75], 99.50th=[  271], 99.90th=[  279], 99.95th=[  279],
     | 99.99th=[  279]
   bw (  KiB/s): min=  472, max=  864, per=24.99%, avg=663.47, stdev=116.77, samples=15
   iops        : min=  118, max=  216, avg=165.80, stdev=29.25, samples=15
  lat (msec)   : 4=0.04%, 10=0.43%, 20=0.59%, 50=68.91%, 100=29.38%
  lat (msec)   : 250=0.20%, 500=0.47%
  cpu          : usr=0.71%, sys=4.00%, ctx=4618, majf=0, minf=13
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=182408: Fri Oct  2 12:15:08 2026
  read: IOPS=159, BW=639KiB/s (654kB/s)(4984KiB/7802msec)
    slat (usec): min=1173, max=12735, avg=3137.26, stdev=1113.57
    clat (usec): min=6184, max=95690, avg=46590.90, stdev=9993.00
     lat (usec): min=9514, max=98657, avg=49731.13, stdev=10379.93
    clat percentiles (usec):
     |  1.00th=[29754],  5.00th=[33424], 10.00th=[35390], 20.00th=[38536],
     | 30.00th=[40633], 40.00th=[42730], 50.00th=[44827], 60.00th=[47449],
     | 70.00th=[49546], 80.00th=[54789], 90.00th=[60556], 95.00th=[65799],
     | 99.00th=[74974], 99.50th=[80217], 99.90th=[91751], 99.95th=[95945],
     | 99.99th=[95945]
   bw (  KiB/s): min=  516, max=  824, per=25.22%, avg=643.13, stdev=89.33, samples=15
   iops        : min=  129, max=  206, avg=160.73, stdev=22.36, samples=15
  write: IOPS=168, BW=674KiB/s (690kB/s)(5256KiB/7802msec); 0 zone resets
    slat (usec): min=34, max=2016, avg=115.47, stdev=97.28
    clat (usec): min=11562, max=89487, avg=47213.66, stdev=9038.70
     lat (usec): min=11717, max=89590, avg=47330.50, stdev=9043.21
    clat percentiles (usec):
     |  1.00th=[31851],  5.00th=[35390], 10.00th=[37487], 20.00th=[39584],
     | 30.00th=[41681], 40.00th=[43254], 50.00th=[45351], 60.00th=[47449],
     | 70.00th=[51119], 80.00th=[55313], 90.00th=[59507], 95.00th=[63701],
     | 99.00th=[71828], 99.50th=[74974], 99.90th=[89654], 99.95th=[89654],
     | 99.99th=[89654]
   bw (  KiB/s): min=  456, max=  856, per=25.39%, avg=674.13, stdev=127.22, samples=15
   iops        : min=  114, max=  214, avg=168.47, stdev=31.86, samples=15
  lat (msec)   : 10=0.04%, 20=0.04%, 50=69.22%, 100=30.70%
  cpu          : usr=0.56%, sys=4.22%, ctx=4772, majf=0, minf=13
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=182409: Fri Oct  2 12:15:08 2026
  read: IOPS=161, BW=646KiB/s (662kB/s)(5060KiB/7829msec)
    slat (usec): min=1149, max=12018, avg=3140.65, stdev=1070.72
    clat (usec): min=22740, max=80385, avg=46386.44, stdev=9510.57
     lat (usec): min=25182, max=84234, avg=49529.87, stdev=9882.57
    clat percentiles (usec):
     |  1.00th=[29492],  5.00th=[33424], 10.00th=[35914], 20.00th=[38536],
     | 30.00th=[40633], 40.00th=[42206], 50.00th=[44827], 60.00th=[46924],
     | 70.00th=[50070], 80.00th=[54264], 90.00th=[59507], 95.00th=[64750],
     | 99.00th=[72877], 99.50th=[74974], 99.90th=[79168], 99.95th=[80217],
     | 99.99th=[80217]
   bw (  KiB/s): min=  439, max=  808, per=25.44%, avg=648.67, stdev=107.66, samples=15
   iops        : min=  109, max=  202, avg=162.07, stdev=27.05, samples=15
  write: IOPS=165, BW=662KiB/s (678kB/s)(5180KiB/7829msec); 0 zone resets
    slat (usec): min=32, max=4494, avg=121.76, stdev=172.70
    clat (usec): min=28808, max=80831, avg=47654.59, stdev=9185.48
     lat (usec): min=28884, max=80909, avg=47777.65, stdev=9196.69
    clat percentiles (usec):
     |  1.00th=[32375],  5.00th=[35914], 10.00th=[37487], 20.00th=[39584],
     | 30.00th=[41681], 40.00th=[43779], 50.00th=[45876], 60.00th=[47973],
     | 70.00th=[51119], 80.00th=[55313], 90.00th=[61080], 95.00th=[65274],
     | 99.00th=[71828], 99.50th=[74974], 99.90th=[81265], 99.95th=[81265],
     | 99.99th=[81265]
   bw (  KiB/s): min=  520, max=  816, per=24.88%, avg=660.67, stdev=91.38, samples=15
   iops        : min=  130, max=  204, avg=165.07, stdev=22.92, samples=15
  lat (msec)   : 50=67.97%, 100=32.03%
  cpu          : usr=0.66%, sys=4.04%, ctx=4782, majf=0, minf=11
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=2551KiB/s (2612kB/s), 632KiB/s-650KiB/s (648kB/s-665kB/s), io=19.6MiB (20.5MB), run=7771-7868msec
  WRITE: bw=2655KiB/s (2719kB/s), 662KiB/s-674KiB/s (678kB/s-690kB/s), io=20.4MiB (21.4MB), run=7771-7868msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.19
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=182413: Fri Oct  2 12:15:12 2026
  read: IOPS=9, BW=377MiB/s (396MB/s)(40.0MiB/106msec)
    slat (nsec): min=11444k, max=11444k, avg=11444034.00, stdev= 0.00
    clat (nsec): min=89283k, max=89283k, avg=89283358.00, stdev= 0.00
     lat (nsec): min=100734k, max=100734k, avg=100733625.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[89654],  5.00th=[89654], 10.00th=[89654], 20.00th=[89654],
     | 30.00th=[89654], 40.00th=[89654], 50.00th=[89654], 60.00th=[89654],
     | 70.00th=[89654], 80.00th=[89654], 90.00th=[89654], 95.00th=[89654],
     | 99.00th=[89654], 99.50th=[89654], 99.90th=[89654], 99.95th=[89654],
     | 99.99th=[89654]
  lat (msec)   : 100=100.00%
  cpu          : usr=0.00%, sys=9.52%, ctx=11, majf=0, minf=8
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=377MiB/s (396MB/s), 377MiB/s-377MiB/s (396MB/s-396MB/s), io=40.0MiB (41.9MB), run=106-106msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout