== sanity test 398c: run fio to test AIO ================= 03:39:39 (1753429179)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB, 40 MiB) copied, 0.132822 s, 316 MB/s
osc.lustre-OST0000-osc-ffff94841848d800.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=195310: Fri Jul 25 03:39:50 2025
  write: IOPS=333, BW=1333KiB/s (1365kB/s)(10.0MiB/7681msec); 0 zone resets
    slat (usec): min=23, max=1089, avg=102.20, stdev=82.31
    clat (usec): min=5548, max=93230, avg=47432.54, stdev=12372.99
     lat (usec): min=5635, max=93386, avg=47535.87, stdev=12371.34
    clat percentiles (usec):
     |  1.00th=[25822],  5.00th=[28967], 10.00th=[31327], 20.00th=[35914],
     | 30.00th=[40633], 40.00th=[43254], 50.00th=[45876], 60.00th=[50594],
     | 70.00th=[53740], 80.00th=[56886], 90.00th=[62653], 95.00th=[69731],
     | 99.00th=[80217], 99.50th=[85459], 99.90th=[89654], 99.95th=[89654],
     | 99.99th=[92799]
   bw (  KiB/s): min=  934, max= 2160, per=25.02%, avg=1334.27, stdev=324.26, samples=15
   iops        : min=  233, max=  540, avg=333.47, stdev=81.17, samples=15
  lat (msec)   : 10=0.04%, 20=0.08%, 50=58.05%, 100=41.84%
  cpu          : usr=0.59%, sys=3.02%, ctx=1203, majf=0, minf=10
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=195311: Fri Jul 25 03:39:50 2025
  write: IOPS=333, BW=1333KiB/s (1365kB/s)(10.0MiB/7681msec); 0 zone resets
    slat (usec): min=23, max=1582, avg=108.04, stdev=92.54
    clat (usec): min=12930, max=92904, avg=47464.63, stdev=12418.01
     lat (usec): min=13173, max=92951, avg=47573.82, stdev=12422.44
    clat percentiles (usec):
     |  1.00th=[25822],  5.00th=[28705], 10.00th=[31589], 20.00th=[35390],
     | 30.00th=[40633], 40.00th=[43254], 50.00th=[46400], 60.00th=[50594],
     | 70.00th=[53740], 80.00th=[57410], 90.00th=[63701], 95.00th=[69731],
     | 99.00th=[80217], 99.50th=[85459], 99.90th=[90702], 99.95th=[92799],
     | 99.99th=[92799]
   bw (  KiB/s): min=  958, max= 2160, per=25.05%, avg=1335.73, stdev=318.68, samples=15
   iops        : min=  239, max=  540, avg=333.87, stdev=79.74, samples=15
  lat (msec)   : 20=0.08%, 50=58.48%, 100=41.45%
  cpu          : usr=0.35%, sys=3.70%, ctx=1230, majf=0, minf=10
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=195312: Fri Jul 25 03:39:50 2025
  write: IOPS=336, BW=1347KiB/s (1380kB/s)(10.0MiB/7601msec); 0 zone resets
    slat (usec): min=23, max=1641, avg=115.38, stdev=96.67
    clat (usec): min=8079, max=99316, avg=47299.77, stdev=12746.07
     lat (usec): min=8160, max=99403, avg=47416.33, stdev=12753.02
    clat percentiles (usec):
     |  1.00th=[25035],  5.00th=[28967], 10.00th=[31065], 20.00th=[35390],
     | 30.00th=[40633], 40.00th=[43254], 50.00th=[45876], 60.00th=[50070],
     | 70.00th=[53216], 80.00th=[56886], 90.00th=[64226], 95.00th=[69731],
     | 99.00th=[81265], 99.50th=[88605], 99.90th=[93848], 99.95th=[99091],
     | 99.99th=[99091]
   bw (  KiB/s): min=  928, max= 2184, per=25.49%, avg=1359.07, stdev=327.62, samples=14
   iops        : min=  232, max=  546, avg=339.71, stdev=81.95, samples=14
  lat (msec)   : 10=0.08%, 20=0.20%, 50=59.18%, 100=40.55%
  cpu          : usr=0.49%, sys=3.59%, ctx=1467, majf=0, minf=12
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=195313: Fri Jul 25 03:39:50 2025
  write: IOPS=333, BW=1335KiB/s (1367kB/s)(10.0MiB/7669msec); 0 zone resets
    slat (usec): min=28, max=28859, avg=120.08, stdev=577.24
    clat (usec): min=11658, max=90503, avg=47429.40, stdev=12674.22
     lat (usec): min=11774, max=90555, avg=47550.60, stdev=12657.58
    clat percentiles (usec):
     |  1.00th=[25560],  5.00th=[28443], 10.00th=[31327], 20.00th=[35390],
     | 30.00th=[40633], 40.00th=[43779], 50.00th=[46400], 60.00th=[50594],
     | 70.00th=[53740], 80.00th=[56361], 90.00th=[64750], 95.00th=[70779],
     | 99.00th=[81265], 99.50th=[85459], 99.90th=[89654], 99.95th=[90702],
     | 99.99th=[90702]
   bw (  KiB/s): min=  928, max= 2152, per=25.11%, avg=1338.73, stdev=316.28, samples=15
   iops        : min=  232, max=  538, avg=334.60, stdev=79.14, samples=15
  lat (msec)   : 20=0.08%, 50=58.95%, 100=40.98%
  cpu          : usr=0.43%, sys=3.65%, ctx=1343, majf=0, minf=10
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=5333KiB/s (5461kB/s), 1333KiB/s-1347KiB/s (1365kB/s-1380kB/s), io=40.0MiB (41.9MB), run=7601-7681msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=195324: Fri Jul 25 03:40:02 2025
  read: IOPS=152, BW=609KiB/s (623kB/s)(5048KiB/8292msec)
    slat (usec): min=1221, max=10732, avg=3558.99, stdev=1183.34
    clat (usec): min=23522, max=82289, avg=49289.83, stdev=9153.21
     lat (usec): min=29744, max=86693, avg=52851.80, stdev=9357.78
    clat percentiles (usec):
     |  1.00th=[32375],  5.00th=[35914], 10.00th=[38011], 20.00th=[41157],
     | 30.00th=[43779], 40.00th=[45876], 50.00th=[48497], 60.00th=[50594],
     | 70.00th=[53740], 80.00th=[56361], 90.00th=[61080], 95.00th=[66323],
     | 99.00th=[73925], 99.50th=[76022], 99.90th=[80217], 99.95th=[82314],
     | 99.99th=[82314]
   bw (  KiB/s): min=  426, max=  720, per=25.14%, avg=608.38, stdev=75.09, samples=16
   iops        : min=  106, max=  180, avg=152.00, stdev=18.86, samples=16
  write: IOPS=156, BW=626KiB/s (641kB/s)(5192KiB/8292msec); 0 zone resets
    slat (usec): min=31, max=1019, avg=99.79, stdev=72.42
    clat (usec): min=21206, max=76907, avg=49784.76, stdev=8720.48
     lat (usec): min=21389, max=76965, avg=49885.61, stdev=8716.72
    clat percentiles (usec):
     |  1.00th=[32637],  5.00th=[36439], 10.00th=[39060], 20.00th=[42206],
     | 30.00th=[44303], 40.00th=[46400], 50.00th=[49021], 60.00th=[51643],
     | 70.00th=[54264], 80.00th=[57410], 90.00th=[61604], 95.00th=[65274],
     | 99.00th=[71828], 99.50th=[72877], 99.90th=[77071], 99.95th=[77071],
     | 99.99th=[77071]
   bw (  KiB/s): min=  439, max=  792, per=24.98%, avg=629.19, stdev=104.77, samples=16
   iops        : min=  109, max=  198, avg=157.19, stdev=26.27, samples=16
  lat (msec)   : 50=54.73%, 100=45.27%
  cpu          : usr=0.54%, sys=4.32%, ctx=2246, majf=0, minf=13
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=195325: Fri Jul 25 03:40:02 2025
  read: IOPS=150, BW=600KiB/s (615kB/s)(4976KiB/8290msec)
    slat (usec): min=1196, max=9483, avg=3522.64, stdev=1157.52
    clat (msec): min=25, max=100, avg=48.85, stdev= 9.27
     lat (msec): min=29, max=102, avg=52.38, stdev= 9.53
    clat percentiles (msec):
     |  1.00th=[   33],  5.00th=[   36], 10.00th=[   38], 20.00th=[   42],
     | 30.00th=[   44], 40.00th=[   46], 50.00th=[   48], 60.00th=[   51],
     | 70.00th=[   53], 80.00th=[   57], 90.00th=[   61], 95.00th=[   65],
     | 99.00th=[   75], 99.50th=[   84], 99.90th=[   91], 99.95th=[  101],
     | 99.99th=[  101]
   bw (  KiB/s): min=  413, max=  720, per=24.92%, avg=603.06, stdev=95.58, samples=16
   iops        : min=  103, max=  180, avg=150.69, stdev=23.92, samples=16
  write: IOPS=158, BW=635KiB/s (650kB/s)(5264KiB/8290msec); 0 zone resets
    slat (usec): min=25, max=1340, avg=99.04, stdev=78.98
    clat (usec): min=30429, max=95060, avg=50449.88, stdev=8935.45
     lat (usec): min=30494, max=95140, avg=50550.19, stdev=8938.86
    clat percentiles (usec):
     |  1.00th=[34341],  5.00th=[37487], 10.00th=[39584], 20.00th=[42206],
     | 30.00th=[44827], 40.00th=[47449], 50.00th=[49546], 60.00th=[52167],
     | 70.00th=[54789], 80.00th=[57934], 90.00th=[62653], 95.00th=[65799],
     | 99.00th=[74974], 99.50th=[77071], 99.90th=[88605], 99.95th=[94897],
     | 99.99th=[94897]
   bw (  KiB/s): min=  471, max=  848, per=25.41%, avg=640.19, stdev=89.62, samples=16
   iops        : min=  117, max=  212, avg=159.94, stdev=22.51, samples=16
  lat (msec)   : 50=55.78%, 100=44.18%, 250=0.04%
  cpu          : usr=0.46%, sys=4.36%, ctx=2219, majf=0, minf=13
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=195326: Fri Jul 25 03:40:02 2025
  read: IOPS=150, BW=601KiB/s (616kB/s)(4984KiB/8290msec)
    slat (usec): min=1267, max=8787, avg=3540.74, stdev=1197.89
    clat (usec): min=3839, max=88288, avg=49038.53, stdev=9163.62
     lat (usec): min=7315, max=92559, avg=52582.15, stdev=9426.35
    clat percentiles (usec):
     |  1.00th=[31589],  5.00th=[34866], 10.00th=[37487], 20.00th=[41157],
     | 30.00th=[44303], 40.00th=[46400], 50.00th=[48497], 60.00th=[50594],
     | 70.00th=[53216], 80.00th=[56361], 90.00th=[61080], 95.00th=[64750],
     | 99.00th=[72877], 99.50th=[78119], 99.90th=[85459], 99.95th=[88605],
     | 99.99th=[88605]
   bw (  KiB/s): min=  426, max=  768, per=25.02%, avg=605.38, stdev=90.76, samples=16
   iops        : min=  106, max=  192, avg=151.25, stdev=22.71, samples=16
  write: IOPS=158, BW=634KiB/s (649kB/s)(5256KiB/8290msec); 0 zone resets
    slat (usec): min=30, max=3608, avg=101.32, stdev=118.84
    clat (usec): min=30492, max=89937, avg=50273.36, stdev=8631.15
     lat (usec): min=30562, max=90041, avg=50375.71, stdev=8630.05
    clat percentiles (usec):
     |  1.00th=[33817],  5.00th=[37487], 10.00th=[39584], 20.00th=[42730],
     | 30.00th=[44827], 40.00th=[47449], 50.00th=[49546], 60.00th=[52167],
     | 70.00th=[54789], 80.00th=[57410], 90.00th=[61080], 95.00th=[65274],
     | 99.00th=[73925], 99.50th=[77071], 99.90th=[89654], 99.95th=[89654],
     | 99.99th=[89654]
   bw (  KiB/s): min=  432, max=  792, per=25.17%, avg=634.06, stdev=93.81, samples=16
   iops        : min=  108, max=  198, avg=158.44, stdev=23.54, samples=16
  lat (msec)   : 4=0.04%, 50=54.38%, 100=45.59%
  cpu          : usr=0.49%, sys=4.26%, ctx=2207, majf=0, minf=12
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=195327: Fri Jul 25 03:40:02 2025
  read: IOPS=154, BW=616KiB/s (631kB/s)(5060KiB/8209msec)
    slat (usec): min=37, max=10419, avg=3533.03, stdev=1265.44
    clat (usec): min=7496, max=90874, avg=48847.30, stdev=9912.78
     lat (usec): min=7547, max=94577, avg=52383.33, stdev=10271.32
    clat percentiles (usec):
     |  1.00th=[30278],  5.00th=[35390], 10.00th=[38011], 20.00th=[41157],
     | 30.00th=[43254], 40.00th=[45351], 50.00th=[47973], 60.00th=[50594],
     | 70.00th=[53216], 80.00th=[56361], 90.00th=[61080], 95.00th=[65799],
     | 99.00th=[74974], 99.50th=[80217], 99.90th=[87557], 99.95th=[90702],
     | 99.99th=[90702]
   bw (  KiB/s): min=  504, max=  744, per=25.59%, avg=619.19, stdev=69.91, samples=16
   iops        : min=  126, max=  186, avg=154.69, stdev=17.53, samples=16
  write: IOPS=157, BW=631KiB/s (646kB/s)(5180KiB/8209msec); 0 zone resets
    slat (usec): min=25, max=4465, avg=110.73, stdev=170.86
    clat (msec): min=9, max=103, avg=49.94, stdev= 9.19
     lat (msec): min=9, max=104, avg=50.05, stdev= 9.20
    clat percentiles (msec):
     |  1.00th=[   32],  5.00th=[   38], 10.00th=[   40], 20.00th=[   43],
     | 30.00th=[   45], 40.00th=[   47], 50.00th=[   50], 60.00th=[   52],
     | 70.00th=[   55], 80.00th=[   58], 90.00th=[   62], 95.00th=[   65],
     | 99.00th=[   74], 99.50th=[   78], 99.90th=[  104], 99.95th=[  104],
     | 99.99th=[  104]
   bw (  KiB/s): min=  512, max=  832, per=25.06%, avg=631.19, stdev=84.73, samples=16
   iops        : min=  128, max=  208, avg=157.69, stdev=21.24, samples=16
  lat (msec)   : 10=0.31%, 20=0.12%, 50=54.84%, 100=44.65%, 250=0.08%
  cpu          : usr=0.58%, sys=4.28%, ctx=2266, majf=0, minf=12
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=2420KiB/s (2478kB/s), 600KiB/s-616KiB/s (615kB/s-631kB/s), io=19.6MiB (20.5MB), run=8209-8292msec
  WRITE: bw=2520KiB/s (2580kB/s), 626KiB/s-635KiB/s (641kB/s-650kB/s), io=20.4MiB (21.4MB), run=8209-8292msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.19
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=195332: Fri Jul 25 03:40:06 2025
  read: IOPS=6, BW=250MiB/s (262MB/s)(40.0MiB/160msec)
    slat (nsec): min=12396k, max=12396k, avg=12395750.00, stdev= 0.00
    clat (nsec): min=138174k, max=138174k, avg=138173624.00, stdev= 0.00
     lat (nsec): min=150575k, max=150575k, avg=150575355.00, stdev= 0.00
    clat percentiles (msec):
     |  1.00th=[  138],  5.00th=[  138], 10.00th=[  138], 20.00th=[  138],
     | 30.00th=[  138], 40.00th=[  138], 50.00th=[  138], 60.00th=[  138],
     | 70.00th=[  138], 80.00th=[  138], 90.00th=[  138], 95.00th=[  138],
     | 99.00th=[  138], 99.50th=[  138], 99.90th=[  138], 99.95th=[  138],
     | 99.99th=[  138]
  lat (msec)   : 250=100.00%
  cpu          : usr=0.00%, sys=8.18%, ctx=2, majf=0, minf=7
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=250MiB/s (262MB/s), 250MiB/s-250MiB/s (262MB/s-262MB/s), io=40.0MiB (41.9MB), run=160-160msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout