== sanity test 398c: run fio to test AIO ================= 15:52:41 (1733172761)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB) copied, 0.10595 s, 396 MB/s
osc.lustre-OST0000-osc-ffff8800b631a000.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=13388: Mon Dec  2 15:52:46 2024
  write: IOPS=644, BW=2576KiB/s (2638kB/s)(10.0MiB/3975msec)
    slat (usec): min=12, max=678, avg=55.97, stdev=42.03
    clat (usec): min=8474, max=74852, avg=24707.33, stdev=4870.53
     lat (usec): min=8528, max=74910, avg=24764.68, stdev=4874.71
    clat percentiles (usec):
     |  1.00th=[16188],  5.00th=[17957], 10.00th=[20055], 20.00th=[21365],
     | 30.00th=[22414], 40.00th=[23462], 50.00th=[24249], 60.00th=[25035],
     | 70.00th=[25822], 80.00th=[27395], 90.00th=[29754], 95.00th=[32900],
     | 99.00th=[38536], 99.50th=[41157], 99.90th=[68682], 99.95th=[68682],
     | 99.99th=[74974]
   bw (  KiB/s): min= 2192, max= 3088, per=25.46%, avg=2623.14, stdev=295.68, samples=7
   iops        : min=  548, max=  772, avg=655.71, stdev=73.90, samples=7
  lat (msec)   : 10=0.04%, 20=8.95%, 50=90.66%, 100=0.35%
  cpu          : usr=0.55%, sys=4.48%, ctx=1394, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=13389: Mon Dec  2 15:52:46 2024
  write: IOPS=644, BW=2576KiB/s (2638kB/s)(10.0MiB/3975msec)
    slat (usec): min=11, max=595, avg=47.15, stdev=35.23
    clat (usec): min=3967, max=52449, avg=24673.28, stdev=4399.29
     lat (usec): min=4027, max=52790, avg=24721.64, stdev=4403.42
    clat percentiles (usec):
     |  1.00th=[15270],  5.00th=[18220], 10.00th=[20317], 20.00th=[21890],
     | 30.00th=[22676], 40.00th=[23200], 50.00th=[23987], 60.00th=[24773],
     | 70.00th=[26084], 80.00th=[28181], 90.00th=[30016], 95.00th=[32637],
     | 99.00th=[35914], 99.50th=[38536], 99.90th=[49021], 99.95th=[49021],
     | 99.99th=[52691]
   bw (  KiB/s): min= 2208, max= 3065, per=25.30%, avg=2607.00, stdev=274.56, samples=7
   iops        : min=  552, max=  766, avg=651.71, stdev=68.57, samples=7
  lat (msec)   : 4=0.04%, 10=0.12%, 20=7.42%, 50=92.38%, 100=0.04%
  cpu          : usr=1.26%, sys=2.99%, ctx=839, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=13390: Mon Dec  2 15:52:46 2024
  write: IOPS=650, BW=2600KiB/s (2663kB/s)(10.0MiB/3938msec)
    slat (usec): min=12, max=517, avg=59.10, stdev=41.28
    clat (usec): min=2974, max=69774, avg=24477.71, stdev=4807.21
     lat (usec): min=3014, max=69808, avg=24538.25, stdev=4808.41
    clat percentiles (usec):
     |  1.00th=[13829],  5.00th=[17695], 10.00th=[20055], 20.00th=[21365],
     | 30.00th=[22414], 40.00th=[23200], 50.00th=[23987], 60.00th=[24773],
     | 70.00th=[25822], 80.00th=[27395], 90.00th=[29492], 95.00th=[32375],
     | 99.00th=[38011], 99.50th=[44827], 99.90th=[55837], 99.95th=[55837],
     | 99.99th=[69731]
   bw (  KiB/s): min= 2192, max= 3208, per=25.57%, avg=2634.57, stdev=326.57, samples=7
   iops        : min=  548, max=  802, avg=658.57, stdev=81.63, samples=7
  lat (msec)   : 4=0.12%, 10=0.27%, 20=9.84%, 50=89.30%, 100=0.47%
  cpu          : usr=0.99%, sys=4.39%, ctx=1607, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=13391: Mon Dec  2 15:52:46 2024
  write: IOPS=644, BW=2578KiB/s (2640kB/s)(10.0MiB/3972msec)
    slat (usec): min=13, max=731, avg=56.27, stdev=44.36
    clat (usec): min=3990, max=71932, avg=24664.89, stdev=4740.43
     lat (usec): min=4057, max=71993, avg=24722.49, stdev=4745.25
    clat percentiles (usec):
     |  1.00th=[15008],  5.00th=[17957], 10.00th=[20055], 20.00th=[21365],
     | 30.00th=[22414], 40.00th=[23462], 50.00th=[24249], 60.00th=[25035],
     | 70.00th=[26084], 80.00th=[27657], 90.00th=[29754], 95.00th=[32900],
     | 99.00th=[39584], 99.50th=[41681], 99.90th=[58459], 99.95th=[68682],
     | 99.99th=[71828]
   bw (  KiB/s): min= 2192, max= 3096, per=25.40%, avg=2617.14, stdev=292.83, samples=7
   iops        : min=  548, max=  774, avg=654.29, stdev=73.21, samples=7
  lat (msec)   : 4=0.04%, 20=9.84%, 50=89.80%, 100=0.31%
  cpu          : usr=0.76%, sys=4.13%, ctx=1552, majf=0, minf=30
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=10.1MiB/s (10.6MB/s), 2576KiB/s-2600KiB/s (2638kB/s-2663kB/s), io=40.0MiB (41.9MB), run=3938-3975msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=13404: Mon Dec  2 15:52:51 2024
   read: IOPS=350, BW=1403KiB/s (1437kB/s)(5048KiB/3598msec)
    slat (usec): min=822, max=4497, avg=1423.96, stdev=352.71
    clat (usec): min=4745, max=34339, avg=21240.47, stdev=3531.75
     lat (usec): min=6245, max=36102, avg=22667.47, stdev=3606.91
    clat percentiles (usec):
     |  1.00th=[14353],  5.00th=[16057], 10.00th=[16909], 20.00th=[18220],
     | 30.00th=[19268], 40.00th=[20055], 50.00th=[20841], 60.00th=[21890],
     | 70.00th=[22938], 80.00th=[24249], 90.00th=[25822], 95.00th=[27132],
     | 99.00th=[31327], 99.50th=[32375], 99.90th=[33162], 99.95th=[34341],
     | 99.99th=[34341]
   bw (  KiB/s): min= 1240, max= 1568, per=25.04%, avg=1396.57, stdev=119.27, samples=7
   iops        : min=  310, max=  392, avg=349.14, stdev=29.82, samples=7
  write: IOPS=360, BW=1443KiB/s (1478kB/s)(5192KiB/3598msec)
    slat (usec): min=13, max=1265, avg=57.19, stdev=49.96
    clat (usec): min=8399, max=35750, avg=21933.41, stdev=3263.24
     lat (usec): min=8498, max=35809, avg=21991.90, stdev=3266.48
    clat percentiles (usec):
     |  1.00th=[15795],  5.00th=[16909], 10.00th=[17957], 20.00th=[19268],
     | 30.00th=[20055], 40.00th=[20841], 50.00th=[21627], 60.00th=[22414],
     | 70.00th=[23462], 80.00th=[24511], 90.00th=[26084], 95.00th=[27395],
     | 99.00th=[31065], 99.50th=[31851], 99.90th=[33817], 99.95th=[35914],
     | 99.99th=[35914]
   bw (  KiB/s): min= 1232, max= 1560, per=24.62%, avg=1429.71, stdev=124.17, samples=7
   iops        : min=  308, max=  390, avg=357.43, stdev=31.04, samples=7
  lat (msec)   : 10=0.16%, 20=33.75%, 50=66.09%
  cpu          : usr=1.39%, sys=7.20%, ctx=2381, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=13405: Mon Dec  2 15:52:51 2024
   read: IOPS=346, BW=1385KiB/s (1418kB/s)(4976KiB/3593msec)
    slat (usec): min=807, max=3289, avg=1414.61, stdev=331.11
    clat (usec): min=4591, max=37051, avg=21206.31, stdev=3567.71
     lat (usec): min=6532, max=38545, avg=22623.11, stdev=3637.08
    clat percentiles (usec):
     |  1.00th=[14353],  5.00th=[15926], 10.00th=[16909], 20.00th=[18220],
     | 30.00th=[19268], 40.00th=[20055], 50.00th=[20841], 60.00th=[21890],
     | 70.00th=[22938], 80.00th=[23987], 90.00th=[25822], 95.00th=[27395],
     | 99.00th=[31065], 99.50th=[32375], 99.90th=[35390], 99.95th=[36963],
     | 99.99th=[36963]
   bw (  KiB/s): min= 1120, max= 1480, per=24.73%, avg=1379.43, stdev=128.07, samples=7
   iops        : min=  280, max=  370, avg=344.86, stdev=32.02, samples=7
  write: IOPS=366, BW=1465KiB/s (1500kB/s)(5264KiB/3593msec)
    slat (usec): min=16, max=817, avg=54.12, stdev=42.01
    clat (usec): min=6601, max=37004, avg=21922.90, stdev=3361.20
     lat (usec): min=6635, max=37042, avg=21978.39, stdev=3360.54
    clat percentiles (usec):
     |  1.00th=[15533],  5.00th=[16909], 10.00th=[17957], 20.00th=[19268],
     | 30.00th=[20055], 40.00th=[20841], 50.00th=[21627], 60.00th=[22414],
     | 70.00th=[23462], 80.00th=[24773], 90.00th=[26346], 95.00th=[27657],
     | 99.00th=[30278], 99.50th=[31589], 99.90th=[35914], 99.95th=[36963],
     | 99.99th=[36963]
   bw (  KiB/s): min= 1384, max= 1736, per=25.18%, avg=1461.71, stdev=124.43, samples=7
   iops        : min=  346, max=  434, avg=365.43, stdev=31.11, samples=7
  lat (msec)   : 10=0.12%, 20=34.73%, 50=65.16%
  cpu          : usr=1.14%, sys=6.96%, ctx=2315, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=13406: Mon Dec  2 15:52:51 2024
   read: IOPS=346, BW=1386KiB/s (1419kB/s)(4984KiB/3597msec)
    slat (usec): min=761, max=3263, avg=1400.82, stdev=330.68
    clat (usec): min=4168, max=36131, avg=21208.59, stdev=3715.81
     lat (usec): min=5887, max=38647, avg=22611.89, stdev=3785.55
    clat percentiles (usec):
     |  1.00th=[13960],  5.00th=[15926], 10.00th=[16909], 20.00th=[18220],
     | 30.00th=[19006], 40.00th=[20055], 50.00th=[20841], 60.00th=[21890],
     | 70.00th=[22938], 80.00th=[24249], 90.00th=[26346], 95.00th=[27919],
     | 99.00th=[30802], 99.50th=[31327], 99.90th=[35390], 99.95th=[35914],
     | 99.99th=[35914]
   bw (  KiB/s): min= 1232, max= 1576, per=24.82%, avg=1384.00, stdev=122.55, samples=7
   iops        : min=  308, max=  394, avg=346.00, stdev=30.64, samples=7
  write: IOPS=365, BW=1461KiB/s (1496kB/s)(5256KiB/3597msec)
    slat (usec): min=12, max=665, avg=53.62, stdev=35.33
    clat (usec): min=3918, max=33240, avg=21874.98, stdev=3270.49
     lat (usec): min=3961, max=33287, avg=21929.87, stdev=3271.85
    clat percentiles (usec):
     |  1.00th=[15926],  5.00th=[16909], 10.00th=[17957], 20.00th=[19006],
     | 30.00th=[19792], 40.00th=[20841], 50.00th=[21627], 60.00th=[22414],
     | 70.00th=[23462], 80.00th=[24773], 90.00th=[26084], 95.00th=[27395],
     | 99.00th=[31065], 99.50th=[32375], 99.90th=[32900], 99.95th=[33162],
     | 99.99th=[33162]
   bw (  KiB/s): min= 1176, max= 1600, per=24.92%, avg=1446.86, stdev=145.87, samples=7
   iops        : min=  294, max=  400, avg=361.71, stdev=36.47, samples=7
  lat (msec)   : 4=0.04%, 10=0.12%, 20=35.82%, 50=64.02%
  cpu          : usr=0.50%, sys=7.76%, ctx=2461, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=13407: Mon Dec  2 15:52:51 2024
   read: IOPS=353, BW=1415KiB/s (1449kB/s)(5060KiB/3576msec)
    slat (usec): min=12, max=2974, avg=1419.05, stdev=353.77
    clat (usec): min=2176, max=36729, avg=21243.75, stdev=3872.55
     lat (usec): min=2353, max=38660, avg=22664.90, stdev=3957.33
    clat percentiles (usec):
     |  1.00th=[13566],  5.00th=[15533], 10.00th=[16450], 20.00th=[18220],
     | 30.00th=[19268], 40.00th=[20055], 50.00th=[21103], 60.00th=[21890],
     | 70.00th=[23462], 80.00th=[24249], 90.00th=[26084], 95.00th=[27657],
     | 99.00th=[31327], 99.50th=[31851], 99.90th=[34341], 99.95th=[36963],
     | 99.99th=[36963]
   bw (  KiB/s): min= 1304, max= 1576, per=25.31%, avg=1411.43, stdev=96.43, samples=7
   iops        : min=  326, max=  394, avg=352.86, stdev=24.11, samples=7
  write: IOPS=362, BW=1449KiB/s (1483kB/s)(5180KiB/3576msec)
    slat (usec): min=15, max=1135, avg=55.87, stdev=51.21
    clat (usec): min=11346, max=37098, avg=21881.97, stdev=3235.40
     lat (usec): min=11386, max=37147, avg=21939.09, stdev=3235.19
    clat percentiles (usec):
     |  1.00th=[15533],  5.00th=[16909], 10.00th=[17957], 20.00th=[19268],
     | 30.00th=[20055], 40.00th=[20841], 50.00th=[21627], 60.00th=[22414],
     | 70.00th=[23462], 80.00th=[24773], 90.00th=[26084], 95.00th=[27395],
     | 99.00th=[31065], 99.50th=[32113], 99.90th=[33162], 99.95th=[36963],
     | 99.99th=[36963]
   bw (  KiB/s): min= 1296, max= 1552, per=24.82%, avg=1441.14, stdev=87.81, samples=7
   iops        : min=  324, max=  388, avg=360.29, stdev=21.95, samples=7
  lat (msec)   : 4=0.20%, 10=0.12%, 20=33.91%, 50=65.78%
  cpu          : usr=1.01%, sys=7.33%, ctx=2507, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=5578KiB/s (5711kB/s), 1385KiB/s-1415KiB/s (1418kB/s-1449kB/s), io=19.6MiB (20.5MB), run=3576-3598msec
  WRITE: bw=5807KiB/s (5946kB/s), 1443KiB/s-1465KiB/s (1478kB/s-1500kB/s), io=20.4MiB (21.4MB), run=3576-3598msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.7
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=13414: Mon Dec  2 15:52:52 2024
   read: IOPS=9, BW=377MiB/s (396MB/s)(40.0MiB/106msec)
    slat (nsec): min=6259.0k, max=6259.0k, avg=6259990.00, stdev= 0.00
    clat (nsec): min=95195k, max=95195k, avg=95194619.00, stdev= 0.00
     lat (nsec): min=101481k, max=101481k, avg=101480769.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[94897],  5.00th=[94897], 10.00th=[94897], 20.00th=[94897],
     | 30.00th=[94897], 40.00th=[94897], 50.00th=[94897], 60.00th=[94897],
     | 70.00th=[94897], 80.00th=[94897], 90.00th=[94897], 95.00th=[94897],
     | 99.00th=[94897], 99.50th=[94897], 99.90th=[94897], 99.95th=[94897],
     | 99.99th=[94897]
  lat (msec)   : 100=100.00%
  cpu          : usr=0.00%, sys=6.67%, ctx=4, majf=0, minf=29
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=377MiB/s (396MB/s), 377MiB/s-377MiB/s (396MB/s-396MB/s), io=40.0MiB (41.9MB), run=106-106msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout