== sanity test 398c: run fio to test AIO ================= 03:52:06 (1732438326)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB) copied, 0.0882153 s, 475 MB/s
osc.lustre-OST0000-osc-ffff8800b5cfc000.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=30305: Sun Nov 24 03:52:41 2024
  write: IOPS=77, BW=310KiB/s (318kB/s)(10.0MiB/33012msec)
    slat (usec): min=14, max=536, avg=90.45, stdev=60.94
    clat (msec): min=36, max=398, avg=205.91, stdev=65.79
     lat (msec): min=36, max=398, avg=206.00, stdev=65.81
    clat percentiles (msec):
     |  1.00th=[   92],  5.00th=[  124], 10.00th=[  136], 20.00th=[  150],
     | 30.00th=[  163], 40.00th=[  174], 50.00th=[  186], 60.00th=[  207],
     | 70.00th=[  239], 80.00th=[  271], 90.00th=[  309], 95.00th=[  330],
     | 99.00th=[  368], 99.50th=[  380], 99.90th=[  388], 99.95th=[  393],
     | 99.99th=[  397]
   bw (  KiB/s): min=  152, max=  504, per=24.87%, avg=308.35, stdev=89.68, samples=66
   iops        : min=   38, max=  126, avg=76.98, stdev=22.39, samples=66
  lat (msec)   : 50=0.20%, 100=1.09%, 250=73.01%, 500=25.70%
  cpu          : usr=0.12%, sys=0.80%, ctx=2578, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=30306: Sun Nov 24 03:52:41 2024
  write: IOPS=77, BW=310KiB/s (318kB/s)(10.0MiB/32996msec)
    slat (usec): min=14, max=558, avg=89.63, stdev=55.63
    clat (msec): min=48, max=398, avg=205.94, stdev=65.70
     lat (msec): min=48, max=398, avg=206.04, stdev=65.72
    clat percentiles (msec):
     |  1.00th=[   92],  5.00th=[  126], 10.00th=[  136], 20.00th=[  150],
     | 30.00th=[  163], 40.00th=[  174], 50.00th=[  188], 60.00th=[  209],
     | 70.00th=[  239], 80.00th=[  271], 90.00th=[  309], 95.00th=[  330],
     | 99.00th=[  359], 99.50th=[  372], 99.90th=[  384], 99.95th=[  384],
     | 99.99th=[  397]
   bw (  KiB/s): min=  176, max=  512, per=24.87%, avg=308.37, stdev=89.55, samples=65
   iops        : min=   44, max=  128, avg=76.95, stdev=22.36, samples=65
  lat (msec)   : 50=0.04%, 100=1.33%, 250=73.01%, 500=25.62%
  cpu          : usr=0.21%, sys=0.73%, ctx=2642, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=30307: Sun Nov 24 03:52:41 2024
  write: IOPS=77, BW=310KiB/s (318kB/s)(10.0MiB/32996msec)
    slat (usec): min=13, max=1709, avg=88.92, stdev=66.68
    clat (msec): min=46, max=398, avg=205.87, stdev=65.78
     lat (msec): min=46, max=398, avg=205.96, stdev=65.80
    clat percentiles (msec):
     |  1.00th=[   91],  5.00th=[  126], 10.00th=[  134], 20.00th=[  150],
     | 30.00th=[  163], 40.00th=[  174], 50.00th=[  188], 60.00th=[  211],
     | 70.00th=[  236], 80.00th=[  271], 90.00th=[  305], 95.00th=[  330],
     | 99.00th=[  359], 99.50th=[  376], 99.90th=[  393], 99.95th=[  397],
     | 99.99th=[  397]
   bw (  KiB/s): min=  176, max=  512, per=24.87%, avg=308.37, stdev=88.96, samples=65
   iops        : min=   44, max=  128, avg=76.97, stdev=22.22, samples=65
  lat (msec)   : 50=0.66%, 100=0.66%, 250=73.32%, 500=25.35%
  cpu          : usr=0.15%, sys=0.81%, ctx=2470, majf=0, minf=28
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=30308: Sun Nov 24 03:52:41 2024
  write: IOPS=77, BW=310KiB/s (318kB/s)(10.0MiB/32985msec)
    slat (usec): min=14, max=3386, avg=96.90, stdev=94.43
    clat (msec): min=30, max=390, avg=205.88, stdev=66.58
     lat (msec): min=30, max=390, avg=205.98, stdev=66.60
    clat percentiles (msec):
     |  1.00th=[   85],  5.00th=[  124], 10.00th=[  136], 20.00th=[  150],
     | 30.00th=[  163], 40.00th=[  174], 50.00th=[  190], 60.00th=[  211],
     | 70.00th=[  239], 80.00th=[  271], 90.00th=[  305], 95.00th=[  334],
     | 99.00th=[  359], 99.50th=[  368], 99.90th=[  388], 99.95th=[  388],
     | 99.99th=[  388]
   bw (  KiB/s): min=  168, max=  568, per=24.90%, avg=308.74, stdev=91.19, samples=65
   iops        : min=   42, max=  142, avg=77.05, stdev=22.77, samples=65
  lat (msec)   : 50=0.51%, 100=1.33%, 250=72.42%, 500=25.74%
  cpu          : usr=0.13%, sys=0.83%, ctx=2898, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=1241KiB/s (1271kB/s), 310KiB/s-310KiB/s (318kB/s-318kB/s), io=40.0MiB (41.9MB), run=32985-33012msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=30344: Sun Nov 24 03:53:00 2024
   read: IOPS=70, BW=283KiB/s (289kB/s)(5048KiB/17866msec)
    slat (usec): min=688, max=4380, avg=1401.87, stdev=421.58
    clat (usec): min=1329, max=158135, avg=73760.18, stdev=17437.93
     lat (msec): min=3, max=160, avg=75.16, stdev=17.61
    clat percentiles (msec):
     |  1.00th=[   45],  5.00th=[   52], 10.00th=[   56], 20.00th=[   61],
     | 30.00th=[   65], 40.00th=[   68], 50.00th=[   72], 60.00th=[   77],
     | 70.00th=[   81], 80.00th=[   86], 90.00th=[   96], 95.00th=[  106],
     | 99.00th=[  128], 99.50th=[  131], 99.90th=[  157], 99.95th=[  159],
     | 99.99th=[  159]
   bw (  KiB/s): min=  168, max=  392, per=25.27%, avg=282.80, stdev=55.11, samples=35
   iops        : min=   42, max=   98, avg=70.57, stdev=13.80, samples=35
  write: IOPS=72, BW=291KiB/s (298kB/s)(5192KiB/17866msec)
    slat (usec): min=15, max=371, avg=73.24, stdev=42.17
    clat (msec): min=52, max=282, avg=146.58, stdev=38.88
     lat (msec): min=52, max=282, avg=146.66, stdev=38.89
    clat percentiles (msec):
     |  1.00th=[   75],  5.00th=[   91], 10.00th=[  104], 20.00th=[  114],
     | 30.00th=[  124], 40.00th=[  132], 50.00th=[  142], 60.00th=[  153],
     | 70.00th=[  163], 80.00th=[  178], 90.00th=[  199], 95.00th=[  224],
     | 99.00th=[  262], 99.50th=[  268], 99.90th=[  279], 99.95th=[  284],
     | 99.99th=[  284]
   bw (  KiB/s): min=  200, max=  384, per=24.98%, avg=291.03, stdev=54.21, samples=35
   iops        : min=   50, max=   96, avg=72.63, stdev=13.51, samples=35
  lat (msec)   : 2=0.04%, 4=0.08%, 10=0.04%, 20=0.08%, 50=1.56%
  lat (msec)   : 100=47.77%, 250=49.57%, 500=0.86%
  cpu          : usr=0.26%, sys=1.95%, ctx=3108, majf=0, minf=33
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=30345: Sun Nov 24 03:53:00 2024
   read: IOPS=69, BW=278KiB/s (284kB/s)(4976KiB/17922msec)
    slat (usec): min=648, max=4056, avg=1406.94, stdev=404.31
    clat (msec): min=3, max=130, avg=73.76, stdev=17.05
     lat (msec): min=4, max=132, avg=75.17, stdev=17.23
    clat percentiles (msec):
     |  1.00th=[   42],  5.00th=[   52], 10.00th=[   55], 20.00th=[   60],
     | 30.00th=[   65], 40.00th=[   68], 50.00th=[   72], 60.00th=[   77],
     | 70.00th=[   81], 80.00th=[   87], 90.00th=[   99], 95.00th=[  107],
     | 99.00th=[  120], 99.50th=[  130], 99.90th=[  131], 99.95th=[  131],
     | 99.99th=[  131]
   bw (  KiB/s): min=  176, max=  392, per=24.87%, avg=278.26, stdev=62.08, samples=35
   iops        : min=   44, max=   98, avg=69.46, stdev=15.48, samples=35
  write: IOPS=73, BW=294KiB/s (301kB/s)(5264KiB/17922msec)
    slat (usec): min=15, max=301, avg=70.52, stdev=39.04
    clat (msec): min=54, max=293, avg=146.46, stdev=39.01
     lat (msec): min=54, max=293, avg=146.53, stdev=39.02
    clat percentiles (msec):
     |  1.00th=[   74],  5.00th=[   93], 10.00th=[  104], 20.00th=[  115],
     | 30.00th=[  122], 40.00th=[  132], 50.00th=[  142], 60.00th=[  150],
     | 70.00th=[  163], 80.00th=[  178], 90.00th=[  201], 95.00th=[  224],
     | 99.00th=[  249], 99.50th=[  264], 99.90th=[  288], 99.95th=[  296],
     | 99.99th=[  296]
   bw (  KiB/s): min=  175, max=  400, per=25.18%, avg=293.34, stdev=58.42, samples=35
   iops        : min=   43, max=  100, avg=73.23, stdev=14.60, samples=35
  lat (msec)   : 4=0.04%, 20=0.04%, 50=2.03%, 100=46.33%, 250=51.13%
  lat (msec)   : 500=0.43%
  cpu          : usr=0.33%, sys=1.90%, ctx=3002, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=30346: Sun Nov 24 03:53:00 2024
   read: IOPS=69, BW=278KiB/s (285kB/s)(4984KiB/17909msec)
    slat (usec): min=680, max=4172, avg=1391.48, stdev=413.67
    clat (msec): min=10, max=153, avg=73.60, stdev=17.04
     lat (msec): min=11, max=155, avg=75.00, stdev=17.23
    clat percentiles (msec):
     |  1.00th=[   46],  5.00th=[   52], 10.00th=[   55], 20.00th=[   61],
     | 30.00th=[   65], 40.00th=[   68], 50.00th=[   71], 60.00th=[   75],
     | 70.00th=[   80], 80.00th=[   87], 90.00th=[   99], 95.00th=[  107],
     | 99.00th=[  124], 99.50th=[  127], 99.90th=[  138], 99.95th=[  155],
     | 99.99th=[  155]
   bw (  KiB/s): min=  160, max=  407, per=25.03%, avg=280.06, stdev=72.91, samples=35
   iops        : min=   40, max=  101, avg=69.89, stdev=18.22, samples=35
  write: IOPS=73, BW=293KiB/s (301kB/s)(5256KiB/17909msec)
    slat (usec): min=15, max=1793, avg=72.80, stdev=65.06
    clat (msec): min=52, max=295, avg=146.36, stdev=38.12
     lat (msec): min=52, max=295, avg=146.44, stdev=38.13
    clat percentiles (msec):
     |  1.00th=[   73],  5.00th=[   93], 10.00th=[  104], 20.00th=[  115],
     | 30.00th=[  125], 40.00th=[  132], 50.00th=[  142], 60.00th=[  153],
     | 70.00th=[  161], 80.00th=[  178], 90.00th=[  197], 95.00th=[  222],
     | 99.00th=[  255], 99.50th=[  262], 99.90th=[  296], 99.95th=[  296],
     | 99.99th=[  296]
   bw (  KiB/s): min=  184, max=  392, per=25.12%, avg=292.63, stdev=58.40, samples=35
   iops        : min=   46, max=   98, avg=73.03, stdev=14.58, samples=35
  lat (msec)   : 20=0.16%, 50=1.76%, 100=46.37%, 250=51.17%, 500=0.55%
  cpu          : usr=0.22%, sys=1.97%, ctx=3130, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=30347: Sun Nov 24 03:53:00 2024
   read: IOPS=71, BW=285KiB/s (292kB/s)(5060KiB/17735msec)
    slat (usec): min=59, max=4190, avg=1383.51, stdev=428.08
    clat (usec): min=1941, max=270962, avg=73147.04, stdev=19675.53
     lat (msec): min=2, max=272, avg=74.53, stdev=19.84
    clat percentiles (msec):
     |  1.00th=[   17],  5.00th=[   50], 10.00th=[   55], 20.00th=[   60],
     | 30.00th=[   64], 40.00th=[   67], 50.00th=[   71], 60.00th=[   75],
     | 70.00th=[   80], 80.00th=[   86], 90.00th=[   99], 95.00th=[  106],
     | 99.00th=[  127], 99.50th=[  131], 99.90th=[  222], 99.95th=[  271],
     | 99.99th=[  271]
   bw (  KiB/s): min=  136, max=  416, per=25.67%, avg=287.23, stdev=64.46, samples=35
   iops        : min=   34, max=  104, avg=71.74, stdev=16.10, samples=35
  write: IOPS=73, BW=292KiB/s (299kB/s)(5180KiB/17735msec)
    slat (usec): min=14, max=421, avg=72.48, stdev=44.98
    clat (msec): min=28, max=354, avg=145.99, stdev=41.22
     lat (msec): min=28, max=354, avg=146.07, stdev=41.23
    clat percentiles (msec):
     |  1.00th=[   67],  5.00th=[   90], 10.00th=[  102], 20.00th=[  113],
     | 30.00th=[  123], 40.00th=[  131], 50.00th=[  142], 60.00th=[  153],
     | 70.00th=[  163], 80.00th=[  178], 90.00th=[  197], 95.00th=[  226],
     | 99.00th=[  262], 99.50th=[  292], 99.90th=[  342], 99.95th=[  355],
     | 99.99th=[  355]
   bw (  KiB/s): min=  184, max=  432, per=25.09%, avg=292.26, stdev=56.18, samples=35
   iops        : min=   46, max=  108, avg=73.00, stdev=14.03, samples=35
  lat (msec)   : 2=0.04%, 4=0.23%, 10=0.12%, 20=0.16%, 50=2.30%
  lat (msec)   : 100=47.27%, 250=48.98%, 500=0.90%
  cpu          : usr=0.23%, sys=1.99%, ctx=3092, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=1120KiB/s (1147kB/s), 278KiB/s-285KiB/s (284kB/s-292kB/s), io=19.6MiB (20.5MB), run=17735-17922msec
  WRITE: bw=1166KiB/s (1194kB/s), 291KiB/s-294KiB/s (298kB/s-301kB/s), io=20.4MiB (21.4MB), run=17735-17922msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.7
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=30365: Sun Nov 24 03:53:01 2024
   read: IOPS=6, BW=267MiB/s (280MB/s)(40.0MiB/150msec)
    slat (nsec): min=5093.5k, max=5093.5k, avg=5093519.00, stdev= 0.00
    clat (nsec): min=137922k, max=137922k, avg=137922127.00, stdev= 0.00
     lat (nsec): min=143025k, max=143025k, avg=143024681.00, stdev= 0.00
    clat percentiles (msec):
     |  1.00th=[  138],  5.00th=[  138], 10.00th=[  138], 20.00th=[  138],
     | 30.00th=[  138], 40.00th=[  138], 50.00th=[  138], 60.00th=[  138],
     | 70.00th=[  138], 80.00th=[  138], 90.00th=[  138], 95.00th=[  138],
     | 99.00th=[  138], 99.50th=[  138], 99.90th=[  138], 99.95th=[  138],
     | 99.99th=[  138]
  lat (msec)   : 250=100.00%
  cpu          : usr=0.00%, sys=4.03%, ctx=4, majf=0, minf=29
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=267MiB/s (280MB/s), 267MiB/s-267MiB/s (280MB/s-280MB/s), io=40.0MiB (41.9MB), run=150-150msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout