== sanity test 398c: run fio to test AIO ================= 18:51:33 (1777503093)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB, 40 MiB) copied, 0.216292 s, 194 MB/s
osc.lustre-OST0000-osc-ffff9ce8a5acf800.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=180325: Wed Apr 29 18:51:52 2026
  write: IOPS=185, BW=741KiB/s (759kB/s)(10.0MiB/13814msec); 0 zone resets
    slat (usec): min=40, max=7227, avg=241.57, stdev=278.80
    clat (msec): min=15, max=214, avg=85.78, stdev=20.86
     lat (msec): min=15, max=214, avg=86.02, stdev=20.87
    clat percentiles (msec):
     |  1.00th=[   39],  5.00th=[   55], 10.00th=[   64], 20.00th=[   70],
     | 30.00th=[   75], 40.00th=[   80], 50.00th=[   84], 60.00th=[   89],
     | 70.00th=[   94], 80.00th=[  102], 90.00th=[  111], 95.00th=[  121],
     | 99.00th=[  153], 99.50th=[  174], 99.90th=[  203], 99.95th=[  215],
     | 99.99th=[  215]
   bw (  KiB/s): min=  464, max=  983, per=24.96%, avg=740.15, stdev=113.91, samples=27
   iops        : min=  116, max=  245, avg=184.78, stdev=28.34, samples=27
  lat (msec)   : 20=0.04%, 50=2.81%, 100=76.17%, 250=20.98%
  cpu          : usr=0.32%, sys=3.17%, ctx=2495, majf=0, minf=9
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=180326: Wed Apr 29 18:51:52 2026
  write: IOPS=185, BW=743KiB/s (761kB/s)(10.0MiB/13786msec); 0 zone resets
    slat (usec): min=41, max=3914, avg=240.11, stdev=278.86
    clat (msec): min=6, max=216, avg=85.59, stdev=20.97
     lat (msec): min=7, max=216, avg=85.84, stdev=20.99
    clat percentiles (msec):
     |  1.00th=[   37],  5.00th=[   55], 10.00th=[   64], 20.00th=[   71],
     | 30.00th=[   75], 40.00th=[   80], 50.00th=[   84], 60.00th=[   88],
     | 70.00th=[   94], 80.00th=[  101], 90.00th=[  110], 95.00th=[  121],
     | 99.00th=[  153], 99.50th=[  174], 99.90th=[  215], 99.95th=[  215],
     | 99.99th=[  218]
   bw (  KiB/s): min=  512, max= 1004, per=24.98%, avg=740.59, stdev=113.59, samples=27
   iops        : min=  128, max=  251, avg=184.93, stdev=28.33, samples=27
  lat (msec)   : 10=0.04%, 20=0.08%, 50=2.85%, 100=76.29%, 250=20.74%
  cpu          : usr=0.24%, sys=3.17%, ctx=2510, majf=0, minf=10
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=180327: Wed Apr 29 18:51:52 2026
  write: IOPS=185, BW=741KiB/s (759kB/s)(10.0MiB/13813msec); 0 zone resets
    slat (usec): min=38, max=3298, avg=213.58, stdev=211.64
    clat (msec): min=5, max=215, avg=85.74, stdev=21.55
     lat (msec): min=6, max=216, avg=85.96, stdev=21.56
    clat percentiles (msec):
     |  1.00th=[   40],  5.00th=[   57], 10.00th=[   64], 20.00th=[   71],
     | 30.00th=[   75], 40.00th=[   79], 50.00th=[   84], 60.00th=[   88],
     | 70.00th=[   95], 80.00th=[  102], 90.00th=[  112], 95.00th=[  122],
     | 99.00th=[  144], 99.50th=[  199], 99.90th=[  215], 99.95th=[  215],
     | 99.99th=[  215]
   bw (  KiB/s): min=  472, max=  966, per=24.92%, avg=738.74, stdev=118.92, samples=27
   iops        : min=  118, max=  241, avg=184.44, stdev=29.71, samples=27
  lat (msec)   : 10=0.04%, 20=0.12%, 50=2.66%, 100=75.70%, 250=21.48%
  cpu          : usr=0.35%, sys=2.71%, ctx=2535, majf=0, minf=11
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=180328: Wed Apr 29 18:51:52 2026
  write: IOPS=186, BW=748KiB/s (766kB/s)(10.0MiB/13692msec); 0 zone resets
    slat (usec): min=40, max=4429, avg=256.14, stdev=292.80
    clat (msec): min=10, max=230, avg=85.18, stdev=22.75
     lat (msec): min=10, max=230, avg=85.44, stdev=22.77
    clat percentiles (msec):
     |  1.00th=[   34],  5.00th=[   54], 10.00th=[   63], 20.00th=[   70],
     | 30.00th=[   75], 40.00th=[   79], 50.00th=[   83], 60.00th=[   88],
     | 70.00th=[   94], 80.00th=[  101], 90.00th=[  111], 95.00th=[  121],
     | 99.00th=[  155], 99.50th=[  203], 99.90th=[  215], 99.95th=[  215],
     | 99.99th=[  232]
   bw (  KiB/s): min=  480, max= 1134, per=25.06%, avg=743.07, stdev=128.83, samples=27
   iops        : min=  120, max=  283, avg=185.52, stdev=32.10, samples=27
  lat (msec)   : 20=0.27%, 50=3.44%, 100=76.02%, 250=20.27%
  cpu          : usr=0.37%, sys=3.28%, ctx=2795, majf=0, minf=10
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=2965KiB/s (3036kB/s), 741KiB/s-748KiB/s (759kB/s-766kB/s), io=40.0MiB (41.9MB), run=13692-13814msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=180339: Wed Apr 29 18:52:09 2026
  read: IOPS=102, BW=412KiB/s (422kB/s)(5048KiB/12253msec)
    slat (usec): min=96, max=17384, avg=4909.74, stdev=2196.89
    clat (msec): min=4, max=139, avg=73.04, stdev=16.20
     lat (msec): min=5, max=146, avg=77.95, stdev=16.75
    clat percentiles (msec):
     |  1.00th=[   40],  5.00th=[   51], 10.00th=[   54], 20.00th=[   59],
     | 30.00th=[   64], 40.00th=[   68], 50.00th=[   72], 60.00th=[   77],
     | 70.00th=[   82], 80.00th=[   88], 90.00th=[   95], 95.00th=[  101],
     | 99.00th=[  110], 99.50th=[  116], 99.90th=[  126], 99.95th=[  140],
     | 99.99th=[  140]
   bw (  KiB/s): min=  302, max=  544, per=25.34%, avg=411.08, stdev=54.43, samples=24
   iops        : min=   75, max=  136, avg=102.54, stdev=13.72, samples=24
  write: IOPS=105, BW=424KiB/s (434kB/s)(5192KiB/12253msec); 0 zone resets
    slat (usec): min=61, max=11619, avg=240.53, stdev=396.79
    clat (msec): min=23, max=132, avg=74.82, stdev=14.45
     lat (msec): min=24, max=132, avg=75.06, stdev=14.45
    clat percentiles (msec):
     |  1.00th=[   46],  5.00th=[   53], 10.00th=[   57], 20.00th=[   63],
     | 30.00th=[   67], 40.00th=[   71], 50.00th=[   74], 60.00th=[   79],
     | 70.00th=[   83], 80.00th=[   88], 90.00th=[   94], 95.00th=[  100],
     | 99.00th=[  109], 99.50th=[  112], 99.90th=[  125], 99.95th=[  133],
     | 99.99th=[  133]
   bw (  KiB/s): min=  248, max=  592, per=25.09%, avg=423.83, stdev=69.52, samples=24
   iops        : min=   62, max=  148, avg=105.75, stdev=17.43, samples=24
  lat (msec)   : 10=0.12%, 20=0.08%, 50=4.06%, 100=91.25%, 250=4.49%
  cpu          : usr=0.27%, sys=4.02%, ctx=4620, majf=0, minf=14
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=180340: Wed Apr 29 18:52:09 2026
  read: IOPS=100, BW=403KiB/s (412kB/s)(4976KiB/12360msec)
    slat (usec): min=1482, max=20094, avg=4891.46, stdev=2131.82
    clat (msec): min=8, max=133, avg=73.20, stdev=15.49
     lat (msec): min=15, max=138, avg=78.10, stdev=16.01
    clat percentiles (msec):
     |  1.00th=[   44],  5.00th=[   51], 10.00th=[   54], 20.00th=[   60],
     | 30.00th=[   65], 40.00th=[   68], 50.00th=[   72], 60.00th=[   77],
     | 70.00th=[   82], 80.00th=[   87], 90.00th=[   94], 95.00th=[  100],
     | 99.00th=[  112], 99.50th=[  116], 99.90th=[  128], 99.95th=[  133],
     | 99.99th=[  133]
   bw (  KiB/s): min=  272, max=  528, per=24.85%, avg=403.00, stdev=70.80, samples=24
   iops        : min=   68, max=  132, avg=100.54, stdev=17.73, samples=24
  write: IOPS=106, BW=426KiB/s (436kB/s)(5264KiB/12360msec); 0 zone resets
    slat (usec): min=51, max=5119, avg=238.94, stdev=303.83
    clat (msec): min=31, max=132, avg=75.29, stdev=14.54
     lat (msec): min=31, max=133, avg=75.53, stdev=14.56
    clat percentiles (msec):
     |  1.00th=[   47],  5.00th=[   53], 10.00th=[   57], 20.00th=[   63],
     | 30.00th=[   67], 40.00th=[   71], 50.00th=[   75], 60.00th=[   79],
     | 70.00th=[   83], 80.00th=[   88], 90.00th=[   95], 95.00th=[  102],
     | 99.00th=[  110], 99.50th=[  114], 99.90th=[  121], 99.95th=[  133],
     | 99.99th=[  133]
   bw (  KiB/s): min=  303, max=  584, per=25.23%, avg=426.12, stdev=70.61, samples=24
   iops        : min=   75, max=  146, avg=106.29, stdev=17.81, samples=24
  lat (msec)   : 10=0.04%, 50=3.48%, 100=91.13%, 250=5.35%
  cpu          : usr=0.39%, sys=3.85%, ctx=4623, majf=0, minf=13
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=180341: Wed Apr 29 18:52:09 2026
  read: IOPS=100, BW=403KiB/s (413kB/s)(4984KiB/12367msec)
    slat (usec): min=1221, max=17291, avg=4887.17, stdev=2192.17
    clat (msec): min=5, max=123, avg=72.98, stdev=15.61
     lat (msec): min=9, max=128, avg=77.87, stdev=16.11
    clat percentiles (msec):
     |  1.00th=[   41],  5.00th=[   51], 10.00th=[   54], 20.00th=[   60],
     | 30.00th=[   64], 40.00th=[   69], 50.00th=[   72], 60.00th=[   77],
     | 70.00th=[   82], 80.00th=[   87], 90.00th=[   93], 95.00th=[  102],
     | 99.00th=[  112], 99.50th=[  113], 99.90th=[  118], 99.95th=[  124],
     | 99.99th=[  124]
   bw (  KiB/s): min=  263, max=  544, per=24.84%, avg=402.83, stdev=64.64, samples=24
   iops        : min=   65, max=  136, avg=100.46, stdev=16.26, samples=24
  write: IOPS=106, BW=425KiB/s (435kB/s)(5256KiB/12367msec); 0 zone resets
    slat (usec): min=40, max=3140, avg=238.76, stdev=244.51
    clat (msec): min=24, max=118, avg=75.41, stdev=14.51
     lat (msec): min=24, max=118, avg=75.65, stdev=14.52
    clat percentiles (msec):
     |  1.00th=[   47],  5.00th=[   54], 10.00th=[   58], 20.00th=[   63],
     | 30.00th=[   67], 40.00th=[   71], 50.00th=[   74], 60.00th=[   79],
     | 70.00th=[   83], 80.00th=[   89], 90.00th=[   96], 95.00th=[  102],
     | 99.00th=[  108], 99.50th=[  112], 99.90th=[  117], 99.95th=[  118],
     | 99.99th=[  118]
   bw (  KiB/s): min=  296, max=  520, per=24.95%, avg=421.46, stdev=69.94, samples=24
   iops        : min=   74, max=  130, avg=105.12, stdev=17.61, samples=24
  lat (msec)   : 10=0.04%, 20=0.04%, 50=3.36%, 100=90.78%, 250=5.78%
  cpu          : usr=0.36%, sys=4.00%, ctx=4569, majf=0, minf=15
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=180342: Wed Apr 29 18:52:09 2026
  read: IOPS=102, BW=409KiB/s (419kB/s)(5060KiB/12363msec)
    slat (usec): min=1590, max=14752, avg=4878.18, stdev=2045.41
    clat (msec): min=29, max=119, avg=72.81, stdev=14.94
     lat (msec): min=35, max=125, avg=77.69, stdev=15.46
    clat percentiles (msec):
     |  1.00th=[   43],  5.00th=[   51], 10.00th=[   55], 20.00th=[   59],
     | 30.00th=[   65], 40.00th=[   68], 50.00th=[   72], 60.00th=[   77],
     | 70.00th=[   81], 80.00th=[   86], 90.00th=[   93], 95.00th=[  100],
     | 99.00th=[  110], 99.50th=[  113], 99.90th=[  118], 99.95th=[  120],
     | 99.99th=[  120]
   bw (  KiB/s): min=  276, max=  528, per=25.30%, avg=410.33, stdev=60.09, samples=24
   iops        : min=   69, max=  132, avg=102.38, stdev=15.03, samples=24
  write: IOPS=104, BW=419KiB/s (429kB/s)(5180KiB/12363msec); 0 zone resets
    slat (usec): min=53, max=3220, avg=220.37, stdev=211.33
    clat (msec): min=40, max=139, avg=75.82, stdev=14.35
     lat (msec): min=40, max=139, avg=76.05, stdev=14.35
    clat percentiles (msec):
     |  1.00th=[   49],  5.00th=[   54], 10.00th=[   58], 20.00th=[   63],
     | 30.00th=[   67], 40.00th=[   72], 50.00th=[   77], 60.00th=[   80],
     | 70.00th=[   84], 80.00th=[   89], 90.00th=[   95], 95.00th=[  101],
     | 99.00th=[  108], 99.50th=[  116], 99.90th=[  131], 99.95th=[  140],
     | 99.99th=[  140]
   bw (  KiB/s): min=  344, max=  544, per=24.72%, avg=417.50, stdev=56.09, samples=24
   iops        : min=   86, max=  136, avg=104.17, stdev=14.14, samples=24
  lat (msec)   : 50=3.55%, 100=91.29%, 250=5.16%
  cpu          : usr=0.41%, sys=3.92%, ctx=4586, majf=0, minf=14
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=1623KiB/s (1662kB/s), 403KiB/s-412KiB/s (412kB/s-422kB/s), io=19.6MiB (20.5MB), run=12253-12367msec
  WRITE: bw=1689KiB/s (1730kB/s), 419KiB/s-426KiB/s (429kB/s-436kB/s), io=20.4MiB (21.4MB), run=12253-12367msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.19
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=180348: Wed Apr 29 18:52:14 2026
  read: IOPS=5, BW=229MiB/s (240MB/s)(40.0MiB/175msec)
    slat (nsec): min=15133k, max=15133k, avg=15132756.00, stdev= 0.00
    clat (nsec): min=151092k, max=151092k, avg=151092485.00, stdev= 0.00
     lat (nsec): min=166229k, max=166229k, avg=166228754.00, stdev= 0.00
    clat percentiles (msec):
     |  1.00th=[  153],  5.00th=[  153], 10.00th=[  153], 20.00th=[  153],
     | 30.00th=[  153], 40.00th=[  153], 50.00th=[  153], 60.00th=[  153],
     | 70.00th=[  153], 80.00th=[  153], 90.00th=[  153], 95.00th=[  153],
     | 99.00th=[  153], 99.50th=[  153], 99.90th=[  153], 99.95th=[  153],
     | 99.99th=[  153]
  lat (msec)   : 250=100.00%
  cpu          : usr=0.00%, sys=6.32%, ctx=11, majf=0, minf=7
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=229MiB/s (240MB/s), 229MiB/s-229MiB/s (240MB/s-240MB/s), io=40.0MiB (41.9MB), run=175-175msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout