== sanity test 398c: run fio to test AIO ================= 03:38:51 (1776929931)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB, 40 MiB) copied, 0.386082 s, 109 MB/s
osc.lustre-OST0000-osc-ffff8b434abf7000.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=180648: Thu Apr 23 03:39:18 2026
  write: IOPS=136, BW=544KiB/s (557kB/s)(10.0MiB/18810msec); 0 zone resets
    slat (usec): min=83, max=4745, avg=498.07, stdev=390.92
    clat (msec): min=29, max=245, avg=116.54, stdev=25.48
     lat (msec): min=29, max=246, avg=117.05, stdev=25.53
    clat percentiles (msec):
     |  1.00th=[   72],  5.00th=[   81], 10.00th=[   89], 20.00th=[   96],
     | 30.00th=[  102], 40.00th=[  108], 50.00th=[  115], 60.00th=[  121],
     | 70.00th=[  127], 80.00th=[  134], 90.00th=[  146], 95.00th=[  165],
     | 99.00th=[  199], 99.50th=[  203], 99.90th=[  243], 99.95th=[  245],
     | 99.99th=[  245]
   bw (  KiB/s): min=  413, max=  784, per=24.97%, avg=543.51, stdev=75.94, samples=37
   iops        : min=  103, max=  196, avg=135.62, stdev=19.03, samples=37
  lat (msec)   : 50=0.62%, 100=26.60%, 250=72.77%
  cpu          : usr=0.51%, sys=4.78%, ctx=2987, majf=0, minf=8
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=180649: Thu Apr 23 03:39:18 2026
  write: IOPS=137, BW=548KiB/s (561kB/s)(10.0MiB/18679msec); 0 zone resets
    slat (usec): min=84, max=4616, avg=507.60, stdev=396.39
    clat (msec): min=24, max=246, avg=116.08, stdev=26.49
     lat (msec): min=24, max=246, avg=116.59, stdev=26.56
    clat percentiles (msec):
     |  1.00th=[   44],  5.00th=[   81], 10.00th=[   88], 20.00th=[   96],
     | 30.00th=[  102], 40.00th=[  108], 50.00th=[  115], 60.00th=[  121],
     | 70.00th=[  127], 80.00th=[  136], 90.00th=[  148], 95.00th=[  165],
     | 99.00th=[  197], 99.50th=[  207], 99.90th=[  243], 99.95th=[  245],
     | 99.99th=[  247]
   bw (  KiB/s): min=  405, max=  776, per=25.10%, avg=546.32, stdev=75.66, samples=37
   iops        : min=  101, max=  194, avg=136.30, stdev=18.96, samples=37
  lat (msec)   : 50=1.09%, 100=25.98%, 250=72.93%
  cpu          : usr=0.42%, sys=4.93%, ctx=3241, majf=0, minf=9
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=180650: Thu Apr 23 03:39:18 2026
  write: IOPS=136, BW=546KiB/s (559kB/s)(10.0MiB/18743msec); 0 zone resets
    slat (usec): min=76, max=6639, avg=498.82, stdev=421.67
    clat (msec): min=31, max=245, avg=116.33, stdev=26.16
     lat (msec): min=32, max=245, avg=116.83, stdev=26.23
    clat percentiles (msec):
     |  1.00th=[   51],  5.00th=[   81], 10.00th=[   89], 20.00th=[   96],
     | 30.00th=[  102], 40.00th=[  108], 50.00th=[  115], 60.00th=[  121],
     | 70.00th=[  127], 80.00th=[  136], 90.00th=[  148], 95.00th=[  165],
     | 99.00th=[  199], 99.50th=[  205], 99.90th=[  245], 99.95th=[  245],
     | 99.99th=[  245]
   bw (  KiB/s): min=  413, max=  768, per=25.06%, avg=545.49, stdev=73.82, samples=37
   iops        : min=  103, max=  192, avg=136.11, stdev=18.49, samples=37
  lat (msec)   : 50=0.98%, 100=26.64%, 250=72.38%
  cpu          : usr=0.45%, sys=4.85%, ctx=3035, majf=0, minf=9
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=180651: Thu Apr 23 03:39:18 2026
  write: IOPS=136, BW=544KiB/s (557kB/s)(10.0MiB/18809msec); 0 zone resets
    slat (usec): min=82, max=4250, avg=484.13, stdev=382.58
    clat (msec): min=44, max=245, avg=116.67, stdev=25.45
     lat (msec): min=45, max=246, avg=117.16, stdev=25.53
    clat percentiles (msec):
     |  1.00th=[   72],  5.00th=[   82], 10.00th=[   90], 20.00th=[   96],
     | 30.00th=[  102], 40.00th=[  109], 50.00th=[  115], 60.00th=[  121],
     | 70.00th=[  127], 80.00th=[  134], 90.00th=[  148], 95.00th=[  163],
     | 99.00th=[  199], 99.50th=[  205], 99.90th=[  247], 99.95th=[  247],
     | 99.99th=[  247]
   bw (  KiB/s): min=  421, max=  776, per=24.95%, avg=543.22, stdev=77.10, samples=37
   iops        : min=  105, max=  194, avg=135.54, stdev=19.33, samples=37
  lat (msec)   : 50=0.62%, 100=26.45%, 250=72.93%
  cpu          : usr=0.43%, sys=4.61%, ctx=3036, majf=0, minf=9
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=2178KiB/s (2230kB/s), 544KiB/s-548KiB/s (557kB/s-561kB/s), io=40.0MiB (41.9MB), run=18679-18810msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=180664: Thu Apr 23 03:39:37 2026
  read: IOPS=88, BW=355KiB/s (363kB/s)(5048KiB/14221msec)
    slat (usec): min=2097, max=15668, avg=5644.41, stdev=1804.33
    clat (msec): min=30, max=138, avg=84.55, stdev=14.33
     lat (msec): min=36, max=145, avg=90.20, stdev=14.78
    clat percentiles (msec):
     |  1.00th=[   53],  5.00th=[   64], 10.00th=[   68], 20.00th=[   73],
     | 30.00th=[   78], 40.00th=[   81], 50.00th=[   84], 60.00th=[   88],
     | 70.00th=[   91], 80.00th=[   96], 90.00th=[  104], 95.00th=[  111],
     | 99.00th=[  123], 99.50th=[  133], 99.90th=[  138], 99.95th=[  138],
     | 99.99th=[  138]
   bw (  KiB/s): min=  236, max=  432, per=25.08%, avg=353.82, stdev=48.14, samples=28
   iops        : min=   59, max=  108, avg=88.21, stdev=12.03, samples=28
  write: IOPS=91, BW=365KiB/s (374kB/s)(5192KiB/14221msec); 0 zone resets
    slat (usec): min=92, max=5536, avg=353.98, stdev=280.69
    clat (msec): min=29, max=129, avg=85.96, stdev=12.64
     lat (msec): min=30, max=130, avg=86.32, stdev=12.66
    clat percentiles (msec):
     |  1.00th=[   62],  5.00th=[   68], 10.00th=[   71], 20.00th=[   75],
     | 30.00th=[   79], 40.00th=[   82], 50.00th=[   85], 60.00th=[   89],
     | 70.00th=[   92], 80.00th=[   96], 90.00th=[  104], 95.00th=[  109],
     | 99.00th=[  118], 99.50th=[  122], 99.90th=[  129], 99.95th=[  130],
     | 99.99th=[  130]
   bw (  KiB/s): min=  304, max=  472, per=24.80%, avg=364.25, stdev=42.62, samples=28
   iops        : min=   76, max=  118, avg=90.82, stdev=10.66, samples=28
  lat (msec)   : 50=0.31%, 100=85.90%, 250=13.79%
  cpu          : usr=0.43%, sys=5.52%, ctx=4899, majf=0, minf=11
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=180665: Thu Apr 23 03:39:37 2026
  read: IOPS=87, BW=351KiB/s (359kB/s)(4976KiB/14187msec)
    slat (usec): min=2035, max=21999, avg=5616.48, stdev=1884.78
    clat (msec): min=18, max=140, avg=84.59, stdev=14.21
     lat (msec): min=24, max=146, avg=90.21, stdev=14.62
    clat percentiles (msec):
     |  1.00th=[   55],  5.00th=[   65], 10.00th=[   68], 20.00th=[   73],
     | 30.00th=[   78], 40.00th=[   81], 50.00th=[   84], 60.00th=[   88],
     | 70.00th=[   92], 80.00th=[   96], 90.00th=[  104], 95.00th=[  109],
     | 99.00th=[  123], 99.50th=[  125], 99.90th=[  136], 99.95th=[  142],
     | 99.99th=[  142]
   bw (  KiB/s): min=  263, max=  439, per=24.79%, avg=349.86, stdev=44.49, samples=28
   iops        : min=   65, max=  109, avg=87.21, stdev=11.10, samples=28
  write: IOPS=92, BW=371KiB/s (380kB/s)(5264KiB/14187msec); 0 zone resets
    slat (usec): min=10, max=5979, avg=350.31, stdev=291.24
    clat (msec): min=41, max=144, avg=86.02, stdev=13.75
     lat (msec): min=42, max=144, avg=86.38, stdev=13.79
    clat percentiles (msec):
     |  1.00th=[   62],  5.00th=[   67], 10.00th=[   70], 20.00th=[   74],
     | 30.00th=[   79], 40.00th=[   82], 50.00th=[   85], 60.00th=[   88],
     | 70.00th=[   92], 80.00th=[   97], 90.00th=[  105], 95.00th=[  111],
     | 99.00th=[  125], 99.50th=[  129], 99.90th=[  138], 99.95th=[  144],
     | 99.99th=[  144]
   bw (  KiB/s): min=  256, max=  455, per=25.32%, avg=371.93, stdev=48.56, samples=28
   iops        : min=   64, max=  113, avg=92.71, stdev=12.22, samples=28
  lat (msec)   : 20=0.04%, 50=0.16%, 100=85.47%, 250=14.34%
  cpu          : usr=0.47%, sys=5.59%, ctx=4866, majf=0, minf=11
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=180666: Thu Apr 23 03:39:37 2026
  read: IOPS=87, BW=351KiB/s (359kB/s)(4984KiB/14216msec)
    slat (usec): min=2025, max=20667, avg=5660.22, stdev=1848.84
    clat (msec): min=47, max=131, avg=84.22, stdev=13.49
     lat (msec): min=50, max=138, avg=89.89, stdev=13.94
    clat percentiles (msec):
     |  1.00th=[   57],  5.00th=[   64], 10.00th=[   68], 20.00th=[   73],
     | 30.00th=[   78], 40.00th=[   81], 50.00th=[   84], 60.00th=[   87],
     | 70.00th=[   91], 80.00th=[   95], 90.00th=[  103], 95.00th=[  108],
     | 99.00th=[  122], 99.50th=[  126], 99.90th=[  132], 99.95th=[  132],
     | 99.99th=[  132]
   bw (  KiB/s): min=  250, max=  440, per=24.89%, avg=351.18, stdev=53.49, samples=28
   iops        : min=   62, max=  110, avg=87.54, stdev=13.41, samples=28
  write: IOPS=92, BW=370KiB/s (379kB/s)(5256KiB/14216msec); 0 zone resets
    slat (usec): min=93, max=3032, avg=358.96, stdev=261.14
    clat (msec): min=44, max=138, avg=86.35, stdev=13.27
     lat (msec): min=44, max=138, avg=86.71, stdev=13.30
    clat percentiles (msec):
     |  1.00th=[   62],  5.00th=[   68], 10.00th=[   71], 20.00th=[   75],
     | 30.00th=[   79], 40.00th=[   83], 50.00th=[   85], 60.00th=[   89],
     | 70.00th=[   93], 80.00th=[   99], 90.00th=[  105], 95.00th=[  110],
     | 99.00th=[  121], 99.50th=[  127], 99.90th=[  136], 99.95th=[  138],
     | 99.99th=[  138]
   bw (  KiB/s): min=  288, max=  472, per=25.07%, avg=368.29, stdev=46.34, samples=28
   iops        : min=   72, max=  118, avg=91.82, stdev=11.61, samples=28
  lat (msec)   : 50=0.12%, 100=84.80%, 250=15.08%
  cpu          : usr=0.45%, sys=5.56%, ctx=4892, majf=0, minf=11
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=180667: Thu Apr 23 03:39:37 2026
  read: IOPS=89, BW=359KiB/s (367kB/s)(5060KiB/14111msec)
    slat (usec): min=141, max=15013, avg=5578.95, stdev=1876.00
    clat (msec): min=7, max=135, avg=83.12, stdev=15.09
     lat (msec): min=8, max=140, avg=88.70, stdev=15.66
    clat percentiles (msec):
     |  1.00th=[   51],  5.00th=[   63], 10.00th=[   67], 20.00th=[   72],
     | 30.00th=[   77], 40.00th=[   80], 50.00th=[   83], 60.00th=[   86],
     | 70.00th=[   90], 80.00th=[   95], 90.00th=[  103], 95.00th=[  109],
     | 99.00th=[  123], 99.50th=[  126], 99.90th=[  133], 99.95th=[  136],
     | 99.99th=[  136]
   bw (  KiB/s): min=  279, max=  496, per=25.46%, avg=359.19, stdev=51.06, samples=27
   iops        : min=   69, max=  124, avg=89.52, stdev=12.82, samples=27
  write: IOPS=91, BW=367KiB/s (376kB/s)(5180KiB/14111msec); 0 zone resets
    slat (usec): min=92, max=1900, avg=352.92, stdev=229.27
    clat (msec): min=33, max=159, avg=87.02, stdev=13.65
     lat (msec): min=34, max=159, avg=87.38, stdev=13.67
    clat percentiles (msec):
     |  1.00th=[   60],  5.00th=[   68], 10.00th=[   71], 20.00th=[   75],
     | 30.00th=[   80], 40.00th=[   83], 50.00th=[   86], 60.00th=[   90],
     | 70.00th=[   93], 80.00th=[   99], 90.00th=[  106], 95.00th=[  111],
     | 99.00th=[  121], 99.50th=[  127], 99.90th=[  142], 99.95th=[  161],
     | 99.99th=[  161]
   bw (  KiB/s): min=  288, max=  456, per=24.96%, avg=366.59, stdev=38.21, samples=27
   iops        : min=   72, max=  114, avg=91.37, stdev= 9.57, samples=27
  lat (msec)   : 10=0.12%, 20=0.16%, 50=0.35%, 100=84.57%, 250=14.80%
  cpu          : usr=0.52%, sys=5.48%, ctx=4823, majf=0, minf=13
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=1411KiB/s (1445kB/s), 351KiB/s-359KiB/s (359kB/s-367kB/s), io=19.6MiB (20.5MB), run=14111-14221msec
  WRITE: bw=1469KiB/s (1504kB/s), 365KiB/s-371KiB/s (374kB/s-380kB/s), io=20.4MiB (21.4MB), run=14111-14221msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.19
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=180673: Thu Apr 23 03:39:47 2026
  read: IOPS=3, BW=141MiB/s (148MB/s)(40.0MiB/284msec)
    slat (nsec): min=28139k, max=28139k, avg=28138854.00, stdev= 0.00
    clat (nsec): min=243937k, max=243937k, avg=243937228.00, stdev= 0.00
     lat (nsec): min=272433k, max=272433k, avg=272432589.00, stdev= 0.00
    clat percentiles (msec):
     |  1.00th=[  245],  5.00th=[  245], 10.00th=[  245], 20.00th=[  245],
     | 30.00th=[  245], 40.00th=[  245], 50.00th=[  245], 60.00th=[  245],
     | 70.00th=[  245], 80.00th=[  245], 90.00th=[  245], 95.00th=[  245],
     | 99.00th=[  245], 99.50th=[  245], 99.90th=[  245], 99.95th=[  245],
     | 99.99th=[  245]
  lat (msec)   : 250=100.00%
  cpu          : usr=0.35%, sys=8.48%, ctx=10, majf=0, minf=7
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=141MiB/s (148MB/s), 141MiB/s-141MiB/s (148MB/s-148MB/s), io=40.0MiB (41.9MB), run=284-284msec