== sanity test 398c: run fio to test AIO ================= 17:29:35 (1713216575)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB) copied, 0.124154 s, 338 MB/s
osc.lustre-OST0000-osc-ffff880136a48800.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=30056: Mon Apr 15 17:30:05 2024
  write: IOPS=91, BW=365KiB/s (374kB/s)(10.0MiB/28062msec)
    slat (usec): min=13, max=376, avg=55.65, stdev=34.49
    clat (msec): min=63, max=284, avg=175.10, stdev=32.33
     lat (msec): min=63, max=284, avg=175.16, stdev=32.33
    clat percentiles (msec):
     |  1.00th=[   92],  5.00th=[  122], 10.00th=[  136], 20.00th=[  150],
     | 30.00th=[  161], 40.00th=[  167], 50.00th=[  176], 60.00th=[  182],
     | 70.00th=[  192], 80.00th=[  201], 90.00th=[  215], 95.00th=[  230],
     | 99.00th=[  255], 99.50th=[  257], 99.90th=[  271], 99.95th=[  271],
     | 99.99th=[  284]
   bw (  KiB/s): min=  296, max=  542, per=24.91%, avg=363.41, stdev=47.48, samples=56
   iops        : min=   74, max=  135, avg=90.75, stdev=11.86, samples=56
  lat (msec)   : 100=1.33%, 250=97.03%, 500=1.64%
  cpu          : usr=0.14%, sys=0.60%, ctx=2429, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=30057: Mon Apr 15 17:30:05 2024
  write: IOPS=91, BW=365KiB/s (374kB/s)(10.0MiB/28027msec)
    slat (usec): min=13, max=379, avg=54.22, stdev=32.62
    clat (msec): min=47, max=282, avg=174.88, stdev=32.93
     lat (msec): min=47, max=282, avg=174.93, stdev=32.94
    clat percentiles (msec):
     |  1.00th=[   90],  5.00th=[  121], 10.00th=[  134], 20.00th=[  150],
     | 30.00th=[  161], 40.00th=[  167], 50.00th=[  176], 60.00th=[  182],
     | 70.00th=[  190], 80.00th=[  201], 90.00th=[  215], 95.00th=[  230],
     | 99.00th=[  251], 99.50th=[  257], 99.90th=[  284], 99.95th=[  284],
     | 99.99th=[  284]
   bw (  KiB/s): min=  264, max=  568, per=24.92%, avg=363.57, stdev=50.36, samples=56
   iops        : min=   66, max=  142, avg=90.89, stdev=12.59, samples=56
  lat (msec)   : 50=0.04%, 100=1.60%, 250=97.11%, 500=1.25%
  cpu          : usr=0.15%, sys=0.58%, ctx=2382, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=30058: Mon Apr 15 17:30:05 2024
  write: IOPS=91, BW=365KiB/s (374kB/s)(10.0MiB/28062msec)
    slat (usec): min=13, max=312, avg=56.97, stdev=35.08
    clat (msec): min=47, max=292, avg=175.09, stdev=33.47
     lat (msec): min=47, max=292, avg=175.14, stdev=33.47
    clat percentiles (msec):
     |  1.00th=[   92],  5.00th=[  121], 10.00th=[  136], 20.00th=[  150],
     | 30.00th=[  159], 40.00th=[  167], 50.00th=[  176], 60.00th=[  182],
     | 70.00th=[  190], 80.00th=[  201], 90.00th=[  218], 95.00th=[  230],
     | 99.00th=[  257], 99.50th=[  271], 99.90th=[  279], 99.95th=[  292],
     | 99.99th=[  292]
   bw (  KiB/s): min=  296, max=  560, per=24.92%, avg=363.52, stdev=50.52, samples=56
   iops        : min=   74, max=  140, avg=90.84, stdev=12.64, samples=56
  lat (msec)   : 50=0.04%, 100=1.72%, 250=96.33%, 500=1.91%
  cpu          : usr=0.12%, sys=0.61%, ctx=2617, majf=0, minf=28
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=30059: Mon Apr 15 17:30:05 2024
  write: IOPS=91, BW=366KiB/s (374kB/s)(10.0MiB/28003msec)
    slat (usec): min=13, max=488, avg=57.23, stdev=36.33
    clat (msec): min=17, max=284, avg=174.83, stdev=33.51
     lat (msec): min=18, max=284, avg=174.89, stdev=33.51
    clat percentiles (msec):
     |  1.00th=[   80],  5.00th=[  121], 10.00th=[  136], 20.00th=[  150],
     | 30.00th=[  161], 40.00th=[  167], 50.00th=[  176], 60.00th=[  182],
     | 70.00th=[  190], 80.00th=[  201], 90.00th=[  215], 95.00th=[  230],
     | 99.00th=[  253], 99.50th=[  262], 99.90th=[  275], 99.95th=[  275],
     | 99.99th=[  284]
   bw (  KiB/s): min=  256, max=  600, per=24.91%, avg=363.50, stdev=54.03, samples=56
   iops        : min=   64, max=  150, avg=90.82, stdev=13.53, samples=56
  lat (msec)   : 20=0.08%, 50=0.47%, 100=1.33%, 250=96.99%, 500=1.13%
  cpu          : usr=0.11%, sys=0.61%, ctx=2493, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=1460KiB/s (1495kB/s), 365KiB/s-366KiB/s (374kB/s-374kB/s), io=40.0MiB (41.9MB), run=28003-28062msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=30091: Mon Apr 15 17:30:23 2024
   read: IOPS=72, BW=291KiB/s (297kB/s)(5048KiB/17376msec)
    slat (usec): min=23, max=4247, avg=1241.08, stdev=319.31
    clat (msec): min=3, max=202, avg=71.80, stdev=14.51
     lat (msec): min=4, max=203, avg=73.04, stdev=14.52
    clat percentiles (msec):
     |  1.00th=[   40],  5.00th=[   53], 10.00th=[   58], 20.00th=[   63],
     | 30.00th=[   66], 40.00th=[   69], 50.00th=[   72], 60.00th=[   75],
     | 70.00th=[   78], 80.00th=[   82], 90.00th=[   87], 95.00th=[   92],
     | 99.00th=[  107], 99.50th=[  140], 99.90th=[  203], 99.95th=[  203],
     | 99.99th=[  203]
   bw (  KiB/s): min=  192, max=  360, per=25.20%, avg=290.76, stdev=42.07, samples=34
   iops        : min=   48, max=   90, avg=72.65, stdev=10.53, samples=34
  write: IOPS=74, BW=299KiB/s (306kB/s)(5192KiB/17376msec)
    slat (usec): min=15, max=269, avg=56.15, stdev=31.52
    clat (msec): min=37, max=264, avg=142.80, stdev=30.46
     lat (msec): min=37, max=264, avg=142.86, stdev=30.46
    clat percentiles (msec):
     |  1.00th=[   81],  5.00th=[   94], 10.00th=[  106], 20.00th=[  118],
     | 30.00th=[  128], 40.00th=[  134], 50.00th=[  142], 60.00th=[  148],
     | 70.00th=[  157], 80.00th=[  169], 90.00th=[  184], 95.00th=[  199],
     | 99.00th=[  220], 99.50th=[  222], 99.90th=[  249], 99.95th=[  266],
     | 99.99th=[  266]
   bw (  KiB/s): min=  224, max=  400, per=24.91%, avg=299.47, stdev=37.41, samples=34
   iops        : min=   56, max=  100, avg=74.82, stdev= 9.38, samples=34
  lat (msec)   : 4=0.08%, 20=0.39%, 50=1.21%, 100=50.47%, 250=47.81%
  lat (msec)   : 500=0.04%
  cpu          : usr=0.19%, sys=1.47%, ctx=3512, majf=0, minf=33
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=30092: Mon Apr 15 17:30:23 2024
   read: IOPS=71, BW=286KiB/s (293kB/s)(4976KiB/17376msec)
    slat (usec): min=680, max=3722, avg=1240.65, stdev=284.54
    clat (usec): min=1601, max=150796, avg=71209.91, stdev=13488.57
     lat (msec): min=2, max=151, avg=72.45, stdev=13.50
    clat percentiles (msec):
     |  1.00th=[   15],  5.00th=[   54], 10.00th=[   58], 20.00th=[   63],
     | 30.00th=[   65], 40.00th=[   68], 50.00th=[   72], 60.00th=[   75],
     | 70.00th=[   78], 80.00th=[   81], 90.00th=[   86], 95.00th=[   90],
     | 99.00th=[  103], 99.50th=[  106], 99.90th=[  150], 99.95th=[  150],
     | 99.99th=[  150]
   bw (  KiB/s): min=  192, max=  384, per=24.82%, avg=286.47, stdev=45.74, samples=34
   iops        : min=   48, max=   96, avg=71.53, stdev=11.37, samples=34
  write: IOPS=75, BW=303KiB/s (310kB/s)(5264KiB/17376msec)
    slat (usec): min=13, max=408, avg=52.67, stdev=30.15
    clat (msec): min=69, max=239, avg=142.39, stdev=29.51
     lat (msec): min=69, max=239, avg=142.45, stdev=29.51
    clat percentiles (msec):
     |  1.00th=[   81],  5.00th=[   94], 10.00th=[  106], 20.00th=[  117],
     | 30.00th=[  126], 40.00th=[  134], 50.00th=[  142], 60.00th=[  148],
     | 70.00th=[  157], 80.00th=[  167], 90.00th=[  182], 95.00th=[  194],
     | 99.00th=[  218], 99.50th=[  222], 99.90th=[  232], 99.95th=[  241],
     | 99.99th=[  241]
   bw (  KiB/s): min=  256, max=  368, per=25.16%, avg=302.47, stdev=26.98, samples=34
   iops        : min=   64, max=   92, avg=75.53, stdev= 6.76, samples=34
  lat (msec)   : 2=0.08%, 4=0.08%, 20=0.43%, 50=0.86%, 100=50.31%
  lat (msec)   : 250=48.24%
  cpu          : usr=0.28%, sys=1.40%, ctx=2990, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=30093: Mon Apr 15 17:30:23 2024
   read: IOPS=71, BW=287KiB/s (294kB/s)(4984KiB/17355msec)
    slat (usec): min=16, max=4111, avg=1238.84, stdev=314.72
    clat (usec): min=1366, max=182301, avg=71058.18, stdev=13723.94
     lat (msec): min=2, max=183, avg=72.30, stdev=13.77
    clat percentiles (msec):
     |  1.00th=[   17],  5.00th=[   52], 10.00th=[   57], 20.00th=[   63],
     | 30.00th=[   66], 40.00th=[   68], 50.00th=[   71], 60.00th=[   74],
     | 70.00th=[   78], 80.00th=[   81], 90.00th=[   85], 95.00th=[   90],
     | 99.00th=[  103], 99.50th=[  116], 99.90th=[  182], 99.95th=[  182],
     | 99.99th=[  182]
   bw (  KiB/s): min=  160, max=  384, per=25.04%, avg=288.91, stdev=56.86, samples=34
   iops        : min=   40, max=   96, avg=72.21, stdev=14.18, samples=34
  write: IOPS=75, BW=303KiB/s (310kB/s)(5256KiB/17355msec)
    slat (usec): min=13, max=411, avg=57.17, stdev=38.11
    clat (msec): min=33, max=243, avg=142.54, stdev=31.04
     lat (msec): min=33, max=243, avg=142.59, stdev=31.04
    clat percentiles (msec):
     |  1.00th=[   79],  5.00th=[   93], 10.00th=[  106], 20.00th=[  117],
     | 30.00th=[  127], 40.00th=[  134], 50.00th=[  142], 60.00th=[  150],
     | 70.00th=[  159], 80.00th=[  169], 90.00th=[  184], 95.00th=[  199],
     | 99.00th=[  220], 99.50th=[  222], 99.90th=[  241], 99.95th=[  245],
     | 99.99th=[  245]
   bw (  KiB/s): min=  224, max=  384, per=25.13%, avg=302.09, stdev=39.56, samples=34
   iops        : min=   56, max=   96, avg=75.50, stdev= 9.91, samples=34
  lat (msec)   : 2=0.08%, 4=0.08%, 10=0.04%, 20=0.31%, 50=1.48%
  lat (msec)   : 100=50.04%, 250=47.97%
  cpu          : usr=0.16%, sys=1.49%, ctx=3717, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=30094: Mon Apr 15 17:30:23 2024
   read: IOPS=73, BW=292KiB/s (299kB/s)(5060KiB/17311msec)
    slat (usec): min=694, max=3713, avg=1231.84, stdev=285.00
    clat (usec): min=1715, max=185268, avg=71371.32, stdev=14189.68
     lat (msec): min=3, max=186, avg=72.61, stdev=14.21
    clat percentiles (msec):
     |  1.00th=[   16],  5.00th=[   52], 10.00th=[   57], 20.00th=[   62],
     | 30.00th=[   66], 40.00th=[   68], 50.00th=[   72], 60.00th=[   75],
     | 70.00th=[   78], 80.00th=[   82], 90.00th=[   87], 95.00th=[   91],
     | 99.00th=[  104], 99.50th=[  107], 99.90th=[  184], 99.95th=[  186],
     | 99.99th=[  186]
   bw (  KiB/s): min=  192, max=  384, per=25.50%, avg=294.32, stdev=49.69, samples=34
   iops        : min=   48, max=   96, avg=73.56, stdev=12.39, samples=34
  write: IOPS=74, BW=299KiB/s (306kB/s)(5180KiB/17311msec)
    slat (usec): min=12, max=388, avg=55.44, stdev=34.59
    clat (msec): min=68, max=306, avg=142.52, stdev=31.89
     lat (msec): min=68, max=306, avg=142.57, stdev=31.89
    clat percentiles (msec):
     |  1.00th=[   81],  5.00th=[   94], 10.00th=[  106], 20.00th=[  117],
     | 30.00th=[  126], 40.00th=[  132], 50.00th=[  140], 60.00th=[  148],
     | 70.00th=[  157], 80.00th=[  169], 90.00th=[  184], 95.00th=[  199],
     | 99.00th=[  228], 99.50th=[  271], 99.90th=[  296], 99.95th=[  309],
     | 99.99th=[  309]
   bw (  KiB/s): min=  208, max=  376, per=24.92%, avg=299.50, stdev=40.78, samples=34
   iops        : min=   52, max=   94, avg=74.85, stdev=10.22, samples=34
  lat (msec)   : 2=0.04%, 4=0.08%, 10=0.04%, 20=0.35%, 50=1.45%
  lat (msec)   : 100=50.47%, 250=47.27%, 500=0.31%
  cpu          : usr=0.18%, sys=1.50%, ctx=3613, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=1155KiB/s (1183kB/s), 286KiB/s-292KiB/s (293kB/s-299kB/s), io=19.6MiB (20.5MB), run=17311-17376msec
  WRITE: bw=1202KiB/s (1231kB/s), 299KiB/s-303KiB/s (306kB/s-310kB/s), io=20.4MiB (21.4MB), run=17311-17376msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.7
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=30111: Mon Apr 15 17:30:25 2024
   read: IOPS=8, BW=333MiB/s (350MB/s)(40.0MiB/120msec)
    slat (nsec): min=5574.0k, max=5574.0k, avg=5574039.00, stdev= 0.00
    clat (nsec): min=107906k, max=107906k, avg=107905860.00, stdev= 0.00
     lat (nsec): min=113488k, max=113488k, avg=113488322.00, stdev= 0.00
    clat percentiles (msec):
     |  1.00th=[  108],  5.00th=[  108], 10.00th=[  108], 20.00th=[  108],
     | 30.00th=[  108], 40.00th=[  108], 50.00th=[  108], 60.00th=[  108],
     | 70.00th=[  108], 80.00th=[  108], 90.00th=[  108], 95.00th=[  108],
     | 99.00th=[  108], 99.50th=[  108], 99.90th=[  108], 99.95th=[  108],
     | 99.99th=[  108]
  lat (msec)   : 250=100.00%
  cpu          : usr=0.00%, sys=5.04%, ctx=4, majf=0, minf=29
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=333MiB/s (350MB/s), 333MiB/s-333MiB/s (350MB/s-350MB/s), io=40.0MiB (41.9MB), run=120-120msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout