== sanity test 398c: run fio to test AIO ================= 23:41:23 (1763354483)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB, 40 MiB) copied, 0.154745 s, 271 MB/s
osc.lustre-OST0000-osc-ffffa02a1d73f000.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=184764: Sun Nov 16 23:41:35 2025
  write: IOPS=299, BW=1200KiB/s (1229kB/s)(10.0MiB/8535msec); 0 zone resets
    slat (usec): min=31, max=1477, avg=117.10, stdev=88.70
    clat (usec): min=8006, max=99462, avg=53146.85, stdev=8903.59
     lat (usec): min=8057, max=99543, avg=53265.37, stdev=8903.30
    clat percentiles (usec):
     |  1.00th=[34866],  5.00th=[40109], 10.00th=[42730], 20.00th=[45876],
     | 30.00th=[48497], 40.00th=[50070], 50.00th=[52691], 60.00th=[54264],
     | 70.00th=[56886], 80.00th=[60031], 90.00th=[64226], 95.00th=[68682],
     | 99.00th=[78119], 99.50th=[80217], 99.90th=[99091], 99.95th=[99091],
     | 99.99th=[99091]
   bw (  KiB/s): min= 1024, max= 1376, per=25.17%, avg=1200.87, stdev=111.36, samples=16
   iops        : min=  256, max=  344, avg=300.12, stdev=27.77, samples=16
  lat (msec)   : 10=0.04%, 20=0.08%, 50=39.02%, 100=60.86%
  cpu          : usr=0.68%, sys=2.67%, ctx=2837, majf=0, minf=8
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=184765: Sun Nov 16 23:41:35 2025
  write: IOPS=298, BW=1195KiB/s (1224kB/s)(10.0MiB/8569msec); 0 zone resets
    slat (usec): min=34, max=27249, avg=129.06, stdev=544.11
    clat (msec): min=7, max=100, avg=53.07, stdev= 8.89
     lat (msec): min=7, max=100, avg=53.20, stdev= 8.87
    clat percentiles (msec):
     |  1.00th=[   35],  5.00th=[   40], 10.00th=[   43], 20.00th=[   46],
     | 30.00th=[   49], 40.00th=[   51], 50.00th=[   53], 60.00th=[   55],
     | 70.00th=[   58], 80.00th=[   61], 90.00th=[   64], 95.00th=[   69],
     | 99.00th=[   77], 99.50th=[   80], 99.90th=[   88], 99.95th=[   95],
     | 99.99th=[  101]
   bw (  KiB/s): min=  960, max= 1360, per=25.02%, avg=1193.81, stdev=112.34, samples=16
   iops        : min=  240, max=  340, avg=298.38, stdev=28.05, samples=16
  lat (msec)   : 10=0.08%, 20=0.08%, 50=39.22%, 100=60.59%, 250=0.04%
  cpu          : usr=0.41%, sys=3.19%, ctx=2779, majf=0, minf=10
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=184766: Sun Nov 16 23:41:35 2025
  write: IOPS=298, BW=1193KiB/s (1221kB/s)(10.0MiB/8585msec); 0 zone resets
    slat (usec): min=22, max=4780, avg=108.41, stdev=141.56
    clat (usec): min=9409, max=81145, avg=53129.02, stdev=8483.47
     lat (usec): min=9534, max=81285, avg=53238.55, stdev=8488.63
    clat percentiles (usec):
     |  1.00th=[36963],  5.00th=[39584], 10.00th=[42206], 20.00th=[45876],
     | 30.00th=[48497], 40.00th=[50594], 50.00th=[52691], 60.00th=[54789],
     | 70.00th=[56886], 80.00th=[59507], 90.00th=[64226], 95.00th=[67634],
     | 99.00th=[76022], 99.50th=[79168], 99.90th=[80217], 99.95th=[81265],
     | 99.99th=[81265]
   bw (  KiB/s): min= 1016, max= 1360, per=24.97%, avg=1191.50, stdev=106.57, samples=16
   iops        : min=  254, max=  340, avg=297.81, stdev=26.63, samples=16
  lat (msec)   : 10=0.04%, 50=36.29%, 100=63.67%
  cpu          : usr=0.36%, sys=2.61%, ctx=2704, majf=0, minf=9
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=184767: Sun Nov 16 23:41:35 2025
  write: IOPS=298, BW=1193KiB/s (1221kB/s)(10.0MiB/8585msec); 0 zone resets
    slat (usec): min=21, max=1238, avg=108.81, stdev=85.47
    clat (usec): min=6571, max=85067, avg=53160.14, stdev=8677.50
     lat (usec): min=6776, max=85497, avg=53270.96, stdev=8681.30
    clat percentiles (usec):
     |  1.00th=[35914],  5.00th=[39584], 10.00th=[43254], 20.00th=[45876],
     | 30.00th=[48497], 40.00th=[50594], 50.00th=[52691], 60.00th=[54789],
     | 70.00th=[56886], 80.00th=[59507], 90.00th=[64226], 95.00th=[68682],
     | 99.00th=[77071], 99.50th=[80217], 99.90th=[84411], 99.95th=[85459],
     | 99.99th=[85459]
   bw (  KiB/s): min= 1000, max= 1368, per=24.98%, avg=1191.56, stdev=104.51, samples=16
   iops        : min=  250, max=  342, avg=297.81, stdev=26.12, samples=16
  lat (msec)   : 10=0.04%, 50=38.24%, 100=61.72%
  cpu          : usr=0.36%, sys=2.97%, ctx=2476, majf=0, minf=8
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=4771KiB/s (4886kB/s), 1193KiB/s-1200KiB/s (1221kB/s-1229kB/s), io=40.0MiB (41.9MB), run=8535-8585msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=184779: Sun Nov 16 23:41:46 2025
  read: IOPS=157, BW=629KiB/s (644kB/s)(5048KiB/8026msec)
    slat (usec): min=1383, max=9925, avg=3390.62, stdev=1086.16
    clat (usec): min=9157, max=84304, avg=47584.18, stdev=8819.15
     lat (usec): min=11896, max=89074, avg=50978.35, stdev=9033.18
    clat percentiles (usec):
     |  1.00th=[29754],  5.00th=[34866], 10.00th=[37487], 20.00th=[40109],
     | 30.00th=[42730], 40.00th=[44827], 50.00th=[46924], 60.00th=[49021],
     | 70.00th=[51643], 80.00th=[54264], 90.00th=[58459], 95.00th=[62129],
     | 99.00th=[77071], 99.50th=[81265], 99.90th=[84411], 99.95th=[84411],
     | 99.99th=[84411]
   bw (  KiB/s): min=  520, max=  736, per=25.02%, avg=625.60, stdev=56.86, samples=15
   iops        : min=  130, max=  184, avg=156.33, stdev=14.22, samples=15
  write: IOPS=161, BW=647KiB/s (662kB/s)(5192KiB/8026msec); 0 zone resets
    slat (usec): min=35, max=1055, avg=113.21, stdev=86.24
    clat (usec): min=12091, max=80853, avg=48556.85, stdev=8069.75
     lat (usec): min=12159, max=81489, avg=48671.18, stdev=8067.91
    clat percentiles (usec):
     |  1.00th=[33817],  5.00th=[36963], 10.00th=[39060], 20.00th=[41681],
     | 30.00th=[44303], 40.00th=[46400], 50.00th=[47973], 60.00th=[49546],
     | 70.00th=[51643], 80.00th=[54264], 90.00th=[58459], 95.00th=[63177],
     | 99.00th=[72877], 99.50th=[76022], 99.90th=[81265], 99.95th=[81265],
     | 99.99th=[81265]
   bw (  KiB/s): min=  504, max=  792, per=24.69%, avg=642.80, stdev=78.43, samples=15
   iops        : min=  126, max=  198, avg=160.60, stdev=19.62, samples=15
  lat (msec)   : 10=0.04%, 20=0.04%, 50=63.05%, 100=36.88%
  cpu          : usr=0.50%, sys=4.05%, ctx=4705, majf=0, minf=13
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=184780: Sun Nov 16 23:41:46 2025
  read: IOPS=156, BW=626KiB/s (641kB/s)(4976KiB/7950msec)
    slat (usec): min=35, max=9841, avg=3376.19, stdev=1162.50
    clat (usec): min=4580, max=87664, avg=47515.99, stdev=9206.07
     lat (usec): min=5659, max=91462, avg=50895.37, stdev=9495.72
    clat percentiles (usec):
     |  1.00th=[27657],  5.00th=[35390], 10.00th=[37487], 20.00th=[40633],
     | 30.00th=[42730], 40.00th=[44827], 50.00th=[46924], 60.00th=[49021],
     | 70.00th=[51119], 80.00th=[54264], 90.00th=[58983], 95.00th=[62653],
     | 99.00th=[78119], 99.50th=[80217], 99.90th=[84411], 99.95th=[87557],
     | 99.99th=[87557]
   bw (  KiB/s): min=  496, max=  704, per=24.95%, avg=623.80, stdev=53.66, samples=15
   iops        : min=  124, max=  176, avg=155.87, stdev=13.41, samples=15
  write: IOPS=165, BW=662KiB/s (678kB/s)(5264KiB/7950msec); 0 zone resets
    slat (usec): min=30, max=1948, avg=114.15, stdev=117.74
    clat (usec): min=22359, max=83854, avg=48243.88, stdev=7545.71
     lat (usec): min=22436, max=83918, avg=48359.26, stdev=7547.78
    clat percentiles (usec):
     |  1.00th=[33817],  5.00th=[37487], 10.00th=[39584], 20.00th=[42206],
     | 30.00th=[43779], 40.00th=[45876], 50.00th=[46924], 60.00th=[49021],
     | 70.00th=[51119], 80.00th=[53740], 90.00th=[58459], 95.00th=[61080],
     | 99.00th=[71828], 99.50th=[76022], 99.90th=[83362], 99.95th=[83362],
     | 99.99th=[83362]
   bw (  KiB/s): min=  552, max=  792, per=25.43%, avg=662.00, stdev=75.86, samples=15
   iops        : min=  138, max=  198, avg=165.40, stdev=18.96, samples=15
  lat (msec)   : 10=0.27%, 50=64.02%, 100=35.70%
  cpu          : usr=0.53%, sys=4.09%, ctx=4640, majf=0, minf=13
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=184781: Sun Nov 16 23:41:46 2025
  read: IOPS=155, BW=623KiB/s (638kB/s)(4984KiB/7995msec)
    slat (usec): min=1243, max=11513, avg=3374.59, stdev=1109.74
    clat (usec): min=21551, max=83856, avg=47124.82, stdev=8169.75
     lat (usec): min=29431, max=88466, avg=50503.07, stdev=8408.24
    clat percentiles (usec):
     |  1.00th=[30802],  5.00th=[34866], 10.00th=[37487], 20.00th=[40109],
     | 30.00th=[42206], 40.00th=[44827], 50.00th=[46400], 60.00th=[48497],
     | 70.00th=[51119], 80.00th=[53740], 90.00th=[57934], 95.00th=[61080],
     | 99.00th=[68682], 99.50th=[71828], 99.90th=[82314], 99.95th=[83362],
     | 99.99th=[83362]
   bw (  KiB/s): min=  526, max=  720, per=25.02%, avg=625.47, stdev=68.73, samples=15
   iops        : min=  131, max=  180, avg=156.27, stdev=17.28, samples=15
  write: IOPS=164, BW=657KiB/s (673kB/s)(5256KiB/7995msec); 0 zone resets
    slat (usec): min=35, max=2404, avg=116.91, stdev=128.67
    clat (usec): min=18292, max=81292, avg=48642.40, stdev=8002.13
     lat (usec): min=18376, max=81462, avg=48760.78, stdev=8004.88
    clat percentiles (usec):
     |  1.00th=[34341],  5.00th=[37487], 10.00th=[39584], 20.00th=[42206],
     | 30.00th=[43779], 40.00th=[45876], 50.00th=[47449], 60.00th=[49546],
     | 70.00th=[51643], 80.00th=[54789], 90.00th=[58459], 95.00th=[62653],
     | 99.00th=[73925], 99.50th=[77071], 99.90th=[81265], 99.95th=[81265],
     | 99.99th=[81265]
   bw (  KiB/s): min=  520, max=  750, per=25.06%, avg=652.40, stdev=69.50, samples=15
   iops        : min=  130, max=  187, avg=162.93, stdev=17.25, samples=15
  lat (msec)   : 20=0.04%, 50=64.26%, 100=35.70%
  cpu          : usr=0.49%, sys=4.20%, ctx=4638, majf=0, minf=12
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=184782: Sun Nov 16 23:41:46 2025
  read: IOPS=157, BW=631KiB/s (646kB/s)(5060KiB/8022msec)
    slat (usec): min=1272, max=10282, avg=3376.77, stdev=1132.13
    clat (usec): min=20669, max=88698, avg=47498.95, stdev=8307.57
     lat (usec): min=23301, max=91184, avg=50879.06, stdev=8514.93
    clat percentiles (usec):
     |  1.00th=[31589],  5.00th=[34866], 10.00th=[37487], 20.00th=[40633],
     | 30.00th=[42730], 40.00th=[45351], 50.00th=[46924], 60.00th=[49021],
     | 70.00th=[51119], 80.00th=[53740], 90.00th=[57934], 95.00th=[62129],
     | 99.00th=[72877], 99.50th=[78119], 99.90th=[84411], 99.95th=[88605],
     | 99.99th=[88605]
   bw (  KiB/s): min=  480, max=  720, per=25.05%, avg=626.33, stdev=66.71, samples=15
   iops        : min=  120, max=  180, avg=156.47, stdev=16.69, samples=15
  write: IOPS=161, BW=646KiB/s (661kB/s)(5180KiB/8022msec); 0 zone resets
    slat (usec): min=30, max=4715, avg=120.27, stdev=187.41
    clat (usec): min=29326, max=79442, avg=48517.00, stdev=7918.14
     lat (usec): min=29427, max=79506, avg=48638.56, stdev=7923.68
    clat percentiles (usec):
     |  1.00th=[34341],  5.00th=[37487], 10.00th=[39584], 20.00th=[42206],
     | 30.00th=[43779], 40.00th=[45876], 50.00th=[47449], 60.00th=[49546],
     | 70.00th=[51643], 80.00th=[54264], 90.00th=[59507], 95.00th=[62653],
     | 99.00th=[73925], 99.50th=[76022], 99.90th=[79168], 99.95th=[79168],
     | 99.99th=[79168]
   bw (  KiB/s): min=  464, max=  768, per=24.80%, avg=645.67, stdev=73.16, samples=15
   iops        : min=  116, max=  192, avg=161.33, stdev=18.29, samples=15
  lat (msec)   : 50=63.71%, 100=36.29%
  cpu          : usr=0.57%, sys=3.99%, ctx=4683, majf=0, minf=12
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=2500KiB/s (2560kB/s), 623KiB/s-631KiB/s (638kB/s-646kB/s), io=19.6MiB (20.5MB), run=7950-8026msec
  WRITE: bw=2603KiB/s (2666kB/s), 646KiB/s-662KiB/s (661kB/s-678kB/s), io=20.4MiB (21.4MB), run=7950-8026msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.19
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=184786: Sun Nov 16 23:41:50 2025
  read: IOPS=7, BW=294MiB/s (308MB/s)(40.0MiB/136msec)
    slat (nsec): min=12419k, max=12419k, avg=12418695.00, stdev= 0.00
    clat (nsec): min=115841k, max=115841k, avg=115840527.00, stdev= 0.00
     lat (nsec): min=128264k, max=128264k, avg=128263822.00, stdev= 0.00
    clat percentiles (msec):
     |  1.00th=[  116],  5.00th=[  116], 10.00th=[  116], 20.00th=[  116],
     | 30.00th=[  116], 40.00th=[  116], 50.00th=[  116], 60.00th=[  116],
     | 70.00th=[  116], 80.00th=[  116], 90.00th=[  116], 95.00th=[  116],
     | 99.00th=[  116], 99.50th=[  116], 99.90th=[  116], 99.95th=[  116],
     | 99.99th=[  116]
  lat (msec)   : 250=100.00%
  cpu          : usr=0.00%, sys=7.41%, ctx=11, majf=0, minf=9
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=294MiB/s (308MB/s), 294MiB/s-294MiB/s (308MB/s-308MB/s), io=40.0MiB (41.9MB), run=136-136msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout