== sanity test 398c: run fio to test AIO ================= 23:18:39 (1745464719)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB) copied, 0.137115 s, 306 MB/s
osc.lustre-OST0000-osc-b9682d57-4f6f-4022-ae5c-63debee576e2.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=8862: Wed Apr 23 23:19:08 2025
  write: IOPS=93, BW=375KiB/s (383kB/s)(10.0MiB/27343msec)
    slat (usec): min=13, max=2127, avg=64.67, stdev=56.22
    clat (msec): min=61, max=244, avg=170.49, stdev=28.75
     lat (msec): min=61, max=244, avg=170.56, stdev=28.75
    clat percentiles (msec):
     |  1.00th=[   97],  5.00th=[  123], 10.00th=[  132], 20.00th=[  144],
     | 30.00th=[  157], 40.00th=[  167], 50.00th=[  174], 60.00th=[  180],
     | 70.00th=[  186], 80.00th=[  194], 90.00th=[  207], 95.00th=[  218],
     | 99.00th=[  232], 99.50th=[  234], 99.90th=[  245], 99.95th=[  245],
     | 99.99th=[  245]
   bw (  KiB/s): min=  288, max=  456, per=24.81%, avg=371.61, stdev=39.66, samples=54
   iops        : min=   72, max=  114, avg=92.83, stdev= 9.91, samples=54
  lat (msec)   : 100=1.05%, 250=98.95%
  cpu          : usr=0.10%, sys=0.83%, ctx=1775, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=8863: Wed Apr 23 23:19:08 2025
  write: IOPS=93, BW=375KiB/s (384kB/s)(10.0MiB/27324msec)
    slat (usec): min=12, max=934, avg=62.72, stdev=41.76
    clat (msec): min=70, max=243, avg=170.48, stdev=28.92
     lat (msec): min=70, max=243, avg=170.54, stdev=28.92
    clat percentiles (msec):
     |  1.00th=[   99],  5.00th=[  123], 10.00th=[  133], 20.00th=[  144],
     | 30.00th=[  155], 40.00th=[  167], 50.00th=[  174], 60.00th=[  180],
     | 70.00th=[  188], 80.00th=[  197], 90.00th=[  205], 95.00th=[  215],
     | 99.00th=[  230], 99.50th=[  236], 99.90th=[  243], 99.95th=[  243],
     | 99.99th=[  245]
   bw (  KiB/s): min=  288, max=  512, per=24.91%, avg=373.11, stdev=44.46, samples=54
   iops        : min=   72, max=  128, avg=93.22, stdev=11.13, samples=54
  lat (msec)   : 100=1.48%, 250=98.52%
  cpu          : usr=0.13%, sys=0.74%, ctx=1811, majf=0, minf=28
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=8864: Wed Apr 23 23:19:08 2025
  write: IOPS=93, BW=375KiB/s (384kB/s)(10.0MiB/27325msec)
    slat (usec): min=12, max=1072, avg=63.36, stdev=42.64
    clat (msec): min=76, max=241, avg=170.52, stdev=28.46
     lat (msec): min=76, max=241, avg=170.58, stdev=28.46
    clat percentiles (msec):
     |  1.00th=[   84],  5.00th=[  123], 10.00th=[  133], 20.00th=[  146],
     | 30.00th=[  157], 40.00th=[  167], 50.00th=[  174], 60.00th=[  180],
     | 70.00th=[  186], 80.00th=[  194], 90.00th=[  207], 95.00th=[  215],
     | 99.00th=[  230], 99.50th=[  232], 99.90th=[  241], 99.95th=[  241],
     | 99.99th=[  243]
   bw (  KiB/s): min=  288, max=  512, per=24.86%, avg=372.43, stdev=43.76, samples=54
   iops        : min=   72, max=  128, avg=93.09, stdev=10.96, samples=54
  lat (msec)   : 100=1.68%, 250=98.32%
  cpu          : usr=0.12%, sys=0.79%, ctx=1718, majf=0, minf=33
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=8865: Wed Apr 23 23:19:08 2025
  write: IOPS=93, BW=375KiB/s (384kB/s)(10.0MiB/27302msec)
    slat (usec): min=13, max=1178, avg=65.68, stdev=51.37
    clat (msec): min=32, max=242, avg=170.42, stdev=29.71
     lat (msec): min=32, max=242, avg=170.49, stdev=29.71
    clat percentiles (msec):
     |  1.00th=[   94],  5.00th=[  123], 10.00th=[  132], 20.00th=[  144],
     | 30.00th=[  157], 40.00th=[  167], 50.00th=[  174], 60.00th=[  180],
     | 70.00th=[  188], 80.00th=[  197], 90.00th=[  207], 95.00th=[  215],
     | 99.00th=[  230], 99.50th=[  236], 99.90th=[  243], 99.95th=[  243],
     | 99.99th=[  243]
   bw (  KiB/s): min=  304, max=  528, per=24.98%, avg=374.20, stdev=43.91, samples=54
   iops        : min=   76, max=  132, avg=93.54, stdev=10.99, samples=54
  lat (msec)   : 50=0.35%, 100=1.17%, 250=98.48%
  cpu          : usr=0.13%, sys=0.83%, ctx=1834, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=1498KiB/s (1534kB/s), 375KiB/s-375KiB/s (383kB/s-384kB/s), io=40.0MiB (41.9MB), run=27302-27343msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=8897: Wed Apr 23 23:19:29 2025
   read: IOPS=66, BW=265KiB/s (272kB/s)(5048KiB/19024msec)
    slat (usec): min=45, max=5585, avg=1456.59, stdev=489.24
    clat (usec): min=1076, max=172530, avg=77757.17, stdev=21350.89
     lat (msec): min=2, max=173, avg=79.22, stdev=21.60
    clat percentiles (msec):
     |  1.00th=[   15],  5.00th=[   52], 10.00th=[   56], 20.00th=[   61],
     | 30.00th=[   66], 40.00th=[   70], 50.00th=[   75], 60.00th=[   81],
     | 70.00th=[   88], 80.00th=[   95], 90.00th=[  106], 95.00th=[  114],
     | 99.00th=[  134], 99.50th=[  142], 99.90th=[  171], 99.95th=[  174],
     | 99.99th=[  174]
   bw (  KiB/s): min=  112, max=  424, per=25.24%, avg=264.21, stdev=66.75, samples=38
   iops        : min=   28, max=  106, avg=65.89, stdev=16.71, samples=38
  write: IOPS=68, BW=273KiB/s (279kB/s)(5192KiB/19024msec)
    slat (usec): min=13, max=1827, avg=75.22, stdev=78.73
    clat (msec): min=22, max=317, avg=157.20, stdev=50.06
     lat (msec): min=22, max=317, avg=157.28, stdev=50.07
    clat percentiles (msec):
     |  1.00th=[   69],  5.00th=[   91], 10.00th=[  103], 20.00th=[  117],
     | 30.00th=[  128], 40.00th=[  136], 50.00th=[  144], 60.00th=[  159],
     | 70.00th=[  178], 80.00th=[  203], 90.00th=[  232], 95.00th=[  255],
     | 99.00th=[  288], 99.50th=[  296], 99.90th=[  300], 99.95th=[  317],
     | 99.99th=[  317]
   bw (  KiB/s): min=  152, max=  392, per=24.88%, avg=271.16, stdev=69.96, samples=38
   iops        : min=   38, max=   98, avg=67.63, stdev=17.52, samples=38
  lat (msec)   : 2=0.08%, 4=0.04%, 10=0.23%, 20=0.23%, 50=1.64%
  lat (msec)   : 100=44.22%, 250=50.59%, 500=2.97%
  cpu          : usr=0.19%, sys=1.90%, ctx=3391, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=8898: Wed Apr 23 23:19:29 2025
   read: IOPS=64, BW=260KiB/s (266kB/s)(4976KiB/19141msec)
    slat (usec): min=663, max=4103, avg=1446.21, stdev=457.47
    clat (usec): min=1881, max=149927, avg=78197.93, stdev=20835.44
     lat (msec): min=3, max=151, avg=79.65, stdev=21.06
    clat percentiles (msec):
     |  1.00th=[   14],  5.00th=[   52], 10.00th=[   56], 20.00th=[   62],
     | 30.00th=[   67], 40.00th=[   71], 50.00th=[   77], 60.00th=[   82],
     | 70.00th=[   89], 80.00th=[   95], 90.00th=[  106], 95.00th=[  115],
     | 99.00th=[  128], 99.50th=[  130], 99.90th=[  146], 99.95th=[  150],
     | 99.99th=[  150]
   bw (  KiB/s): min=   88, max=  384, per=24.70%, avg=258.66, stdev=67.29, samples=38
   iops        : min=   22, max=   96, avg=64.61, stdev=16.84, samples=38
  write: IOPS=68, BW=275KiB/s (282kB/s)(5264KiB/19141msec)
    slat (usec): min=13, max=546, avg=72.85, stdev=44.09
    clat (msec): min=56, max=309, avg=156.93, stdev=49.54
     lat (msec): min=56, max=310, avg=157.01, stdev=49.55
    clat percentiles (msec):
     |  1.00th=[   73],  5.00th=[   91], 10.00th=[  103], 20.00th=[  118],
     | 30.00th=[  128], 40.00th=[  136], 50.00th=[  146], 60.00th=[  157],
     | 70.00th=[  174], 80.00th=[  197], 90.00th=[  230], 95.00th=[  255],
     | 99.00th=[  296], 99.50th=[  305], 99.90th=[  309], 99.95th=[  309],
     | 99.99th=[  309]
   bw (  KiB/s): min=  168, max=  440, per=25.14%, avg=274.03, stdev=66.80, samples=38
   iops        : min=   42, max=  110, avg=68.45, stdev=16.73, samples=38
  lat (msec)   : 2=0.04%, 10=0.12%, 20=0.55%, 50=1.09%, 100=43.55%
  lat (msec)   : 250=51.64%, 500=3.01%
  cpu          : usr=0.27%, sys=1.78%, ctx=3381, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=8899: Wed Apr 23 23:19:29 2025
   read: IOPS=65, BW=260KiB/s (266kB/s)(4984KiB/19160msec)
    slat (usec): min=628, max=4138, avg=1434.27, stdev=466.05
    clat (msec): min=2, max=141, avg=78.50, stdev=20.35
     lat (msec): min=4, max=143, avg=79.94, stdev=20.60
    clat percentiles (msec):
     |  1.00th=[   34],  5.00th=[   53], 10.00th=[   57], 20.00th=[   63],
     | 30.00th=[   66], 40.00th=[   71], 50.00th=[   77], 60.00th=[   82],
     | 70.00th=[   89], 80.00th=[   97], 90.00th=[  107], 95.00th=[  114],
     | 99.00th=[  126], 99.50th=[  129], 99.90th=[  140], 99.95th=[  142],
     | 99.99th=[  142]
   bw (  KiB/s): min=  128, max=  448, per=24.81%, avg=259.71, stdev=77.39, samples=38
   iops        : min=   32, max=  112, avg=64.87, stdev=19.34, samples=38
  write: IOPS=68, BW=274KiB/s (281kB/s)(5256KiB/19160msec)
    slat (usec): min=14, max=2101, avg=75.70, stdev=71.19
    clat (msec): min=50, max=303, avg=157.06, stdev=49.02
     lat (msec): min=50, max=304, avg=157.14, stdev=49.03
    clat percentiles (msec):
     |  1.00th=[   74],  5.00th=[   93], 10.00th=[  105], 20.00th=[  117],
     | 30.00th=[  128], 40.00th=[  136], 50.00th=[  146], 60.00th=[  157],
     | 70.00th=[  174], 80.00th=[  199], 90.00th=[  232], 95.00th=[  257],
     | 99.00th=[  284], 99.50th=[  292], 99.90th=[  305], 99.95th=[  305],
     | 99.99th=[  305]
   bw (  KiB/s): min=  167, max=  408, per=24.93%, avg=271.71, stdev=65.02, samples=38
   iops        : min=   41, max=  102, avg=67.87, stdev=16.28, samples=38
  lat (msec)   : 4=0.04%, 10=0.04%, 20=0.35%, 50=1.29%, 100=42.97%
  lat (msec)   : 250=52.42%, 500=2.89%
  cpu          : usr=0.16%, sys=1.91%, ctx=3174, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=8900: Wed Apr 23 23:19:29 2025
   read: IOPS=66, BW=265KiB/s (271kB/s)(5060KiB/19111msec)
    slat (usec): min=656, max=4434, avg=1451.07, stdev=460.70
    clat (msec): min=11, max=162, avg=78.73, stdev=20.13
     lat (msec): min=13, max=164, avg=80.19, stdev=20.37
    clat percentiles (msec):
     |  1.00th=[   37],  5.00th=[   53], 10.00th=[   57], 20.00th=[   63],
     | 30.00th=[   67], 40.00th=[   71], 50.00th=[   75], 60.00th=[   82],
     | 70.00th=[   89], 80.00th=[   96], 90.00th=[  106], 95.00th=[  115],
     | 99.00th=[  132], 99.50th=[  138], 99.90th=[  150], 99.95th=[  163],
     | 99.99th=[  163]
   bw (  KiB/s): min=  103, max=  376, per=25.27%, avg=264.58, stdev=65.78, samples=38
   iops        : min=   25, max=   94, avg=66.11, stdev=16.52, samples=38
  write: IOPS=67, BW=271KiB/s (278kB/s)(5180KiB/19111msec)
    slat (usec): min=13, max=480, avg=72.59, stdev=42.70
    clat (msec): min=43, max=322, avg=157.17, stdev=49.54
     lat (msec): min=43, max=323, avg=157.24, stdev=49.55
    clat percentiles (msec):
     |  1.00th=[   75],  5.00th=[   91], 10.00th=[  104], 20.00th=[  120],
     | 30.00th=[  128], 40.00th=[  138], 50.00th=[  144], 60.00th=[  157],
     | 70.00th=[  171], 80.00th=[  197], 90.00th=[  232], 95.00th=[  255],
     | 99.00th=[  300], 99.50th=[  305], 99.90th=[  317], 99.95th=[  321],
     | 99.99th=[  321]
   bw (  KiB/s): min=  144, max=  392, per=24.70%, avg=269.21, stdev=69.08, samples=38
   iops        : min=   36, max=   98, avg=67.26, stdev=17.29, samples=38
  lat (msec)   : 20=0.23%, 50=1.37%, 100=43.79%, 250=51.64%, 500=2.97%
  cpu          : usr=0.20%, sys=1.87%, ctx=3221, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=1047KiB/s (1073kB/s), 260KiB/s-265KiB/s (266kB/s-272kB/s), io=19.6MiB (20.5MB), run=19024-19160msec
  WRITE: bw=1090KiB/s (1117kB/s), 271KiB/s-275KiB/s (278kB/s-282kB/s), io=20.4MiB (21.4MB), run=19024-19160msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.7
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=8919: Wed Apr 23 23:19:30 2025
   read: IOPS=5, BW=230MiB/s (241MB/s)(40.0MiB/174msec)
    slat (nsec): min=6715.5k, max=6715.5k, avg=6715536.00, stdev= 0.00
    clat (nsec): min=160077k, max=160077k, avg=160077148.00, stdev= 0.00
     lat (nsec): min=166803k, max=166803k, avg=166803279.00, stdev= 0.00
    clat percentiles (msec):
     |  1.00th=[  161],  5.00th=[  161], 10.00th=[  161], 20.00th=[  161],
     | 30.00th=[  161], 40.00th=[  161], 50.00th=[  161], 60.00th=[  161],
     | 70.00th=[  161], 80.00th=[  161], 90.00th=[  161], 95.00th=[  161],
     | 99.00th=[  161], 99.50th=[  161], 99.90th=[  161], 99.95th=[  161],
     | 99.99th=[  161]
  lat (msec)   : 250=100.00%
  cpu          : usr=0.00%, sys=3.47%, ctx=13, majf=0, minf=29
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=230MiB/s (241MB/s), 230MiB/s-230MiB/s (241MB/s-241MB/s), io=40.0MiB (41.9MB), run=174-174msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout