== sanity test 398c: run fio to test AIO ================= 02:38:55 (1786343935)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB, 40 MiB) copied, 0.154706 s, 271 MB/s
osc.lustre-OST0000-osc-ffff8ebc11ec8800.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=180469: Mon Aug 10 02:39:08 2026
  write: IOPS=277, BW=1109KiB/s (1135kB/s)(10.0MiB/9237msec); 0 zone resets
    slat (usec): min=36, max=4714, avg=194.61, stdev=241.09
    clat (msec): min=15, max=156, avg=57.46, stdev=20.31
     lat (msec): min=15, max=157, avg=57.65, stdev=20.32
    clat percentiles (msec):
     |  1.00th=[   23],  5.00th=[   31], 10.00th=[   35], 20.00th=[   39],
     | 30.00th=[   45], 40.00th=[   49], 50.00th=[   54], 60.00th=[   61],
     | 70.00th=[   68], 80.00th=[   75], 90.00th=[   85], 95.00th=[   94],
     | 99.00th=[  110], 99.50th=[  115], 99.90th=[  157], 99.95th=[  157],
     | 99.99th=[  157]
   bw (  KiB/s): min=  744, max= 1580, per=25.17%, avg=1109.89, stdev=242.01, samples=18
   iops        : min=  186, max=  395, avg=277.39, stdev=60.51, samples=18
  lat (msec)   : 20=0.12%, 50=43.40%, 100=53.79%, 250=2.70%
  cpu          : usr=0.37%, sys=3.67%, ctx=2658, majf=0, minf=9
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=180470: Mon Aug 10 02:39:08 2026
  write: IOPS=275, BW=1102KiB/s (1129kB/s)(10.0MiB/9290msec); 0 zone resets
    slat (usec): min=36, max=4738, avg=177.93, stdev=182.93
    clat (msec): min=4, max=129, avg=57.54, stdev=19.72
     lat (msec): min=4, max=129, avg=57.72, stdev=19.73
    clat percentiles (msec):
     |  1.00th=[   27],  5.00th=[   32], 10.00th=[   35], 20.00th=[   40],
     | 30.00th=[   44], 40.00th=[   50], 50.00th=[   54], 60.00th=[   62],
     | 70.00th=[   69], 80.00th=[   77], 90.00th=[   85], 95.00th=[   93],
     | 99.00th=[  108], 99.50th=[  120], 99.90th=[  130], 99.95th=[  130],
     | 99.99th=[  130]
   bw (  KiB/s): min=  760, max= 1592, per=25.02%, avg=1103.17, stdev=224.16, samples=18
   iops        : min=  190, max=  398, avg=275.72, stdev=56.07, samples=18
  lat (msec)   : 10=0.12%, 20=0.04%, 50=43.36%, 100=54.38%, 250=2.11%
  cpu          : usr=0.44%, sys=3.35%, ctx=2711, majf=0, minf=11
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=180471: Mon Aug 10 02:39:08 2026
  write: IOPS=275, BW=1103KiB/s (1129kB/s)(10.0MiB/9285msec); 0 zone resets
    slat (usec): min=42, max=25280, avg=186.23, stdev=520.14
    clat (msec): min=11, max=145, avg=57.54, stdev=19.55
     lat (msec): min=11, max=145, avg=57.73, stdev=19.55
    clat percentiles (msec):
     |  1.00th=[   23],  5.00th=[   31], 10.00th=[   35], 20.00th=[   39],
     | 30.00th=[   45], 40.00th=[   50], 50.00th=[   55], 60.00th=[   63],
     | 70.00th=[   69], 80.00th=[   75], 90.00th=[   84], 95.00th=[   93],
     | 99.00th=[  107], 99.50th=[  114], 99.90th=[  127], 99.95th=[  132],
     | 99.99th=[  146]
   bw (  KiB/s): min=  728, max= 1572, per=25.06%, avg=1104.94, stdev=240.19, samples=18
   iops        : min=  182, max=  393, avg=276.17, stdev=60.08, samples=18
  lat (msec)   : 20=0.12%, 50=42.34%, 100=55.39%, 250=2.15%
  cpu          : usr=0.37%, sys=3.68%, ctx=2609, majf=0, minf=13
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=180472: Mon Aug 10 02:39:08 2026
  write: IOPS=276, BW=1104KiB/s (1131kB/s)(10.0MiB/9274msec); 0 zone resets
    slat (usec): min=37, max=3985, avg=204.30, stdev=230.05
    clat (msec): min=5, max=141, avg=57.42, stdev=19.55
     lat (msec): min=5, max=141, avg=57.62, stdev=19.56
    clat percentiles (msec):
     |  1.00th=[   24],  5.00th=[   31], 10.00th=[   35], 20.00th=[   39],
     | 30.00th=[   45], 40.00th=[   50], 50.00th=[   55], 60.00th=[   63],
     | 70.00th=[   69], 80.00th=[   75], 90.00th=[   84], 95.00th=[   92],
     | 99.00th=[  108], 99.50th=[  115], 99.90th=[  127], 99.95th=[  142],
     | 99.99th=[  142]
   bw (  KiB/s): min=  720, max= 1608, per=25.08%, avg=1105.83, stdev=240.15, samples=18
   iops        : min=  180, max=  402, avg=276.39, stdev=60.07, samples=18
  lat (msec)   : 10=0.08%, 20=0.16%, 50=42.42%, 100=55.31%, 250=2.03%
  cpu          : usr=0.42%, sys=3.96%, ctx=2850, majf=0, minf=12
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=4409KiB/s (4515kB/s), 1102KiB/s-1109KiB/s (1129kB/s-1135kB/s), io=40.0MiB (41.9MB), run=9237-9290msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=180483: Mon Aug 10 02:39:20 2026
  read: IOPS=143, BW=573KiB/s (586kB/s)(5048KiB/8817msec)
    slat (usec): min=1142, max=18052, avg=3607.25, stdev=1936.75
    clat (msec): min=14, max=106, avg=52.05, stdev=15.75
     lat (msec): min=24, max=111, avg=55.66, stdev=16.51
    clat percentiles (msec):
     |  1.00th=[   30],  5.00th=[   34], 10.00th=[   36], 20.00th=[   40],
     | 30.00th=[   43], 40.00th=[   45], 50.00th=[   48], 60.00th=[   52],
     | 70.00th=[   57], 80.00th=[   65], 90.00th=[   77], 95.00th=[   86],
     | 99.00th=[   96], 99.50th=[   97], 99.90th=[  102], 99.95th=[  107],
     | 99.99th=[  107]
   bw (  KiB/s): min=  375, max=  784, per=25.02%, avg=569.53, stdev=127.82, samples=17
   iops        : min=   93, max=  196, avg=142.12, stdev=32.14, samples=17
  write: IOPS=147, BW=589KiB/s (603kB/s)(5192KiB/8817msec); 0 zone resets
    slat (usec): min=55, max=4578, avg=180.88, stdev=211.77
    clat (msec): min=20, max=103, avg=53.39, stdev=15.41
     lat (msec): min=20, max=103, avg=53.58, stdev=15.42
    clat percentiles (msec):
     |  1.00th=[   31],  5.00th=[   35], 10.00th=[   37], 20.00th=[   41],
     | 30.00th=[   44], 40.00th=[   47], 50.00th=[   50], 60.00th=[   54],
     | 70.00th=[   59], 80.00th=[   68], 90.00th=[   79], 95.00th=[   85],
     | 99.00th=[   93], 99.50th=[   96], 99.90th=[  101], 99.95th=[  104],
     | 99.99th=[  104]
   bw (  KiB/s): min=  350, max=  832, per=24.78%, avg=587.12, stdev=121.26, samples=17
   iops        : min=   87, max=  208, avg=146.53, stdev=30.47, samples=17
  lat (msec)   : 20=0.04%, 50=53.83%, 100=45.98%, 250=0.16%
  cpu          : usr=0.35%, sys=4.39%, ctx=4635, majf=0, minf=12
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=180484: Mon Aug 10 02:39:20 2026
  read: IOPS=141, BW=565KiB/s (578kB/s)(4976KiB/8812msec)
    slat (usec): min=1082, max=20816, avg=3661.21, stdev=1957.31
    clat (msec): min=21, max=104, avg=51.96, stdev=15.37
     lat (msec): min=25, max=109, avg=55.62, stdev=16.18
    clat percentiles (msec):
     |  1.00th=[   30],  5.00th=[   33], 10.00th=[   36], 20.00th=[   39],
     | 30.00th=[   43], 40.00th=[   45], 50.00th=[   48], 60.00th=[   52],
     | 70.00th=[   57], 80.00th=[   66], 90.00th=[   77], 95.00th=[   83],
     | 99.00th=[   95], 99.50th=[   99], 99.90th=[  103], 99.95th=[  105],
     | 99.99th=[  105]
   bw (  KiB/s): min=  350, max=  800, per=24.77%, avg=563.76, stdev=137.90, samples=17
   iops        : min=   87, max=  200, avg=140.65, stdev=34.70, samples=17
  write: IOPS=149, BW=597KiB/s (612kB/s)(5264KiB/8812msec); 0 zone resets
    slat (usec): min=39, max=4219, avg=173.97, stdev=164.96
    clat (msec): min=6, max=111, avg=53.53, stdev=15.65
     lat (msec): min=7, max=111, avg=53.71, stdev=15.65
    clat percentiles (msec):
     |  1.00th=[   30],  5.00th=[   35], 10.00th=[   37], 20.00th=[   41],
     | 30.00th=[   44], 40.00th=[   47], 50.00th=[   50], 60.00th=[   54],
     | 70.00th=[   60], 80.00th=[   68], 90.00th=[   77], 95.00th=[   85],
     | 99.00th=[   96], 99.50th=[  103], 99.90th=[  111], 99.95th=[  112],
     | 99.99th=[  112]
   bw (  KiB/s): min=  438, max=  768, per=25.19%, avg=596.76, stdev=110.83, samples=17
   iops        : min=  109, max=  192, avg=148.94, stdev=27.84, samples=17
  lat (msec)   : 10=0.04%, 50=53.16%, 100=46.25%, 250=0.55%
  cpu          : usr=0.44%, sys=4.28%, ctx=4549, majf=0, minf=13
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=180485: Mon Aug 10 02:39:20 2026
  read: IOPS=141, BW=566KiB/s (579kB/s)(4984KiB/8812msec)
    slat (usec): min=1160, max=17230, avg=3724.45, stdev=1936.54
    clat (msec): min=4, max=114, avg=52.82, stdev=16.40
     lat (msec): min=7, max=126, avg=56.54, stdev=17.18
    clat percentiles (msec):
     |  1.00th=[   30],  5.00th=[   33], 10.00th=[   36], 20.00th=[   40],
     | 30.00th=[   43], 40.00th=[   45], 50.00th=[   49], 60.00th=[   53],
     | 70.00th=[   58], 80.00th=[   66], 90.00th=[   79], 95.00th=[   87],
     | 99.00th=[   99], 99.50th=[  102], 99.90th=[  113], 99.95th=[  114],
     | 99.99th=[  114]
   bw (  KiB/s): min=  398, max=  752, per=24.87%, avg=566.00, stdev=109.59, samples=17
   iops        : min=   99, max=  188, avg=141.24, stdev=27.59, samples=17
  write: IOPS=149, BW=596KiB/s (611kB/s)(5256KiB/8812msec); 0 zone resets
    slat (usec): min=57, max=4656, avg=184.81, stdev=226.33
    clat (msec): min=25, max=106, avg=52.75, stdev=14.96
     lat (msec): min=26, max=106, avg=52.93, stdev=14.97
    clat percentiles (msec):
     |  1.00th=[   30],  5.00th=[   35], 10.00th=[   38], 20.00th=[   41],
     | 30.00th=[   44], 40.00th=[   46], 50.00th=[   49], 60.00th=[   53],
     | 70.00th=[   59], 80.00th=[   67], 90.00th=[   75], 95.00th=[   83],
     | 99.00th=[   94], 99.50th=[   97], 99.90th=[  105], 99.95th=[  107],
     | 99.99th=[  107]
   bw (  KiB/s): min=  358, max=  880, per=24.99%, avg=592.12, stdev=142.94, samples=17
   iops        : min=   89, max=  220, avg=147.76, stdev=35.87, samples=17
  lat (msec)   : 10=0.04%, 20=0.04%, 50=53.91%, 100=45.55%, 250=0.47%
  cpu          : usr=0.40%, sys=4.53%, ctx=4530, majf=0, minf=13
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=180486: Mon Aug 10 02:39:20 2026
  read: IOPS=144, BW=578KiB/s (592kB/s)(5060KiB/8753msec)
    slat (usec): min=66, max=16276, avg=3630.88, stdev=1877.22
    clat (msec): min=7, max=110, avg=52.05, stdev=15.47
     lat (msec): min=9, max=119, avg=55.68, stdev=16.19
    clat percentiles (msec):
     |  1.00th=[   28],  5.00th=[   33], 10.00th=[   35], 20.00th=[   40],
     | 30.00th=[   43], 40.00th=[   46], 50.00th=[   50], 60.00th=[   53],
     | 70.00th=[   57], 80.00th=[   65], 90.00th=[   75], 95.00th=[   83],
     | 99.00th=[   95], 99.50th=[   99], 99.90th=[  107], 99.95th=[  111],
     | 99.99th=[  111]
   bw (  KiB/s): min=  390, max=  736, per=25.36%, avg=577.24, stdev=115.35, samples=17
   iops        : min=   97, max=  184, avg=144.00, stdev=29.06, samples=17
  write: IOPS=147, BW=592KiB/s (606kB/s)(5180KiB/8753msec); 0 zone resets
    slat (usec): min=43, max=3319, avg=187.75, stdev=203.77
    clat (msec): min=15, max=101, avg=53.40, stdev=15.33
     lat (msec): min=15, max=101, avg=53.59, stdev=15.34
    clat percentiles (msec):
     |  1.00th=[   29],  5.00th=[   35], 10.00th=[   37], 20.00th=[   41],
     | 30.00th=[   44], 40.00th=[   46], 50.00th=[   51], 60.00th=[   54],
     | 70.00th=[   59], 80.00th=[   68], 90.00th=[   78], 95.00th=[   84],
     | 99.00th=[   95], 99.50th=[   97], 99.90th=[  102], 99.95th=[  102],
     | 99.99th=[  102]
   bw (  KiB/s): min=  366, max=  808, per=24.91%, avg=590.06, stdev=111.64, samples=17
   iops        : min=   91, max=  202, avg=147.29, stdev=28.03, samples=17
  lat (msec)   : 10=0.08%, 20=0.16%, 50=52.15%, 100=47.30%, 250=0.31%
  cpu          : usr=0.39%, sys=4.34%, ctx=4520, majf=0, minf=13
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=2276KiB/s (2331kB/s), 565KiB/s-578KiB/s (578kB/s-592kB/s), io=19.6MiB (20.5MB), run=8753-8817msec
  WRITE: bw=2370KiB/s (2426kB/s), 589KiB/s-597KiB/s (603kB/s-612kB/s), io=20.4MiB (21.4MB), run=8753-8817msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.19
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=180491: Mon Aug 10 02:39:24 2026
  read: IOPS=5, BW=238MiB/s (250MB/s)(40.0MiB/168msec)
    slat (nsec): min=13116k, max=13116k, avg=13116403.00, stdev= 0.00
    clat (nsec): min=142181k, max=142181k, avg=142181444.00, stdev= 0.00
     lat (nsec): min=155302k, max=155302k, avg=155301944.00, stdev= 0.00
    clat percentiles (msec):
     |  1.00th=[  142],  5.00th=[  142], 10.00th=[  142], 20.00th=[  142],
     | 30.00th=[  142], 40.00th=[  142], 50.00th=[  142], 60.00th=[  142],
     | 70.00th=[  142], 80.00th=[  142], 90.00th=[  142], 95.00th=[  142],
     | 99.00th=[  142], 99.50th=[  142], 99.90th=[  142], 99.95th=[  142],
     | 99.99th=[  142]
  lat (msec)   : 250=100.00%
  cpu          : usr=0.00%, sys=6.59%, ctx=11, majf=0, minf=8
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=238MiB/s (250MB/s), 238MiB/s-238MiB/s (250MB/s-250MB/s), io=40.0MiB (41.9MB), run=168-168msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout