== sanity test 398c: run fio to test AIO ================= 03:32:59 (1746084779)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB) copied, 0.0698888 s, 600 MB/s
osc.lustre-OST0000-osc-ffff8800aaea3000.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=24440: Thu May  1 03:33:05 2025
  write: IOPS=666, BW=2665KiB/s (2729kB/s)(10.0MiB/3842msec)
    slat (usec): min=13, max=371, avg=51.62, stdev=29.79
    clat (usec): min=6335, max=42530, avg=23841.02, stdev=3411.72
     lat (usec): min=6421, max=42612, avg=23894.00, stdev=3415.34
    clat percentiles (usec):
     |  1.00th=[13698],  5.00th=[18744], 10.00th=[20317], 20.00th=[21627],
     | 30.00th=[22676], 40.00th=[23200], 50.00th=[23725], 60.00th=[24249],
     | 70.00th=[24773], 80.00th=[25822], 90.00th=[27657], 95.00th=[28967],
     | 99.00th=[33817], 99.50th=[34866], 99.90th=[40109], 99.95th=[42206],
     | 99.99th=[42730]
   bw (  KiB/s): min= 2432, max= 2840, per=24.96%, avg=2660.86, stdev=162.66, samples=7
   iops        : min=  608, max=  710, avg=665.14, stdev=40.63, samples=7
  lat (msec)   : 10=0.04%, 20=8.32%, 50=91.64%
  cpu          : usr=0.65%, sys=4.04%, ctx=1370, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=24441: Thu May  1 03:33:05 2025
  write: IOPS=666, BW=2666KiB/s (2730kB/s)(10.0MiB/3841msec)
    slat (usec): min=12, max=735, avg=55.83, stdev=34.83
    clat (usec): min=3102, max=35531, avg=23785.88, stdev=3330.36
     lat (usec): min=3182, max=35566, avg=23843.00, stdev=3336.50
    clat percentiles (usec):
     |  1.00th=[12387],  5.00th=[19530], 10.00th=[20317], 20.00th=[21627],
     | 30.00th=[22414], 40.00th=[23200], 50.00th=[23725], 60.00th=[24249],
     | 70.00th=[24773], 80.00th=[25822], 90.00th=[27395], 95.00th=[28967],
     | 99.00th=[33162], 99.50th=[33817], 99.90th=[35390], 99.95th=[35390],
     | 99.99th=[35390]
   bw (  KiB/s): min= 2432, max= 2824, per=24.84%, avg=2648.57, stdev=172.10, samples=7
   iops        : min=  608, max=  706, avg=662.00, stdev=43.05, samples=7
  lat (msec)   : 4=0.08%, 10=0.66%, 20=6.33%, 50=92.93%
  cpu          : usr=0.70%, sys=4.09%, ctx=1562, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=24442: Thu May  1 03:33:05 2025
  write: IOPS=666, BW=2666KiB/s (2730kB/s)(10.0MiB/3841msec)
    slat (usec): min=13, max=477, avg=53.31, stdev=29.25
    clat (usec): min=2318, max=38788, avg=23812.73, stdev=3472.01
     lat (usec): min=2387, max=38869, avg=23867.30, stdev=3476.93
    clat percentiles (usec):
     |  1.00th=[13435],  5.00th=[18744], 10.00th=[20317], 20.00th=[21890],
     | 30.00th=[22676], 40.00th=[23200], 50.00th=[23725], 60.00th=[24249],
     | 70.00th=[24773], 80.00th=[26084], 90.00th=[27657], 95.00th=[28967],
     | 99.00th=[33162], 99.50th=[34866], 99.90th=[35914], 99.95th=[38536],
     | 99.99th=[38536]
   bw (  KiB/s): min= 2432, max= 2864, per=24.94%, avg=2658.86, stdev=182.71, samples=7
   iops        : min=  608, max=  716, avg=664.57, stdev=45.68, samples=7
  lat (msec)   : 4=0.08%, 10=0.66%, 20=8.59%, 50=90.66%
  cpu          : usr=0.83%, sys=3.80%, ctx=1441, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=24443: Thu May  1 03:33:05 2025
  write: IOPS=670, BW=2682KiB/s (2746kB/s)(10.0MiB/3818msec)
    slat (usec): min=13, max=434, avg=55.83, stdev=33.05
    clat (usec): min=3507, max=45883, avg=23696.97, stdev=3749.33
     lat (usec): min=3558, max=45927, avg=23754.13, stdev=3750.34
    clat percentiles (usec):
     |  1.00th=[ 8455],  5.00th=[19006], 10.00th=[20317], 20.00th=[21627],
     | 30.00th=[22414], 40.00th=[23200], 50.00th=[23725], 60.00th=[24249],
     | 70.00th=[24773], 80.00th=[25822], 90.00th=[27395], 95.00th=[28967],
     | 99.00th=[33817], 99.50th=[35390], 99.90th=[41157], 99.95th=[43779],
     | 99.99th=[45876]
   bw (  KiB/s): min= 2432, max= 2968, per=25.14%, avg=2680.29, stdev=201.74, samples=7
   iops        : min=  608, max=  742, avg=670.00, stdev=50.41, samples=7
  lat (msec)   : 4=0.08%, 10=1.09%, 20=7.54%, 50=91.29%
  cpu          : usr=0.84%, sys=3.96%, ctx=1662, majf=0, minf=28
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=10.4MiB/s (10.9MB/s), 2665KiB/s-2682KiB/s (2729kB/s-2746kB/s), io=40.0MiB (41.9MB), run=3818-3842msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.7
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=24456: Thu May  1 03:33:10 2025
   read: IOPS=307, BW=1228KiB/s (1258kB/s)(5048KiB/4110msec)
    slat (usec): min=826, max=3879, avg=1574.58, stdev=434.34
    clat (usec): min=11533, max=42017, avg=24384.54, stdev=4296.64
     lat (usec): min=14035, max=44092, avg=25961.66, stdev=4418.68
    clat percentiles (usec):
     |  1.00th=[15795],  5.00th=[18482], 10.00th=[19268], 20.00th=[20841],
     | 30.00th=[21890], 40.00th=[22938], 50.00th=[23725], 60.00th=[25035],
     | 70.00th=[26346], 80.00th=[28181], 90.00th=[30016], 95.00th=[32375],
     | 99.00th=[35914], 99.50th=[36439], 99.90th=[38011], 99.95th=[42206],
     | 99.99th=[42206]
   bw (  KiB/s): min= 1128, max= 1440, per=25.06%, avg=1223.25, stdev=96.46, samples=8
   iops        : min=  282, max=  360, avg=305.75, stdev=24.15, samples=8
  write: IOPS=315, BW=1263KiB/s (1294kB/s)(5192KiB/4110msec)
    slat (usec): min=14, max=640, avg=55.19, stdev=41.10
    clat (usec): min=14491, max=43244, avg=25043.05, stdev=3882.54
     lat (usec): min=14540, max=43278, avg=25099.83, stdev=3883.10
    clat percentiles (usec):
     |  1.00th=[17171],  5.00th=[19530], 10.00th=[20579], 20.00th=[21890],
     | 30.00th=[22676], 40.00th=[23725], 50.00th=[24511], 60.00th=[25560],
     | 70.00th=[26870], 80.00th=[28181], 90.00th=[30016], 95.00th=[32375],
     | 99.00th=[35914], 99.50th=[36439], 99.90th=[38011], 99.95th=[43254],
     | 99.99th=[43254]
   bw (  KiB/s): min= 1048, max= 1512, per=24.74%, avg=1257.25, stdev=136.17, samples=8
   iops        : min=  262, max=  378, avg=314.25, stdev=34.08, samples=8
  lat (msec)   : 20=10.39%, 50=89.61%
  cpu          : usr=1.07%, sys=6.40%, ctx=2344, majf=0, minf=33
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=24457: Thu May  1 03:33:10 2025
   read: IOPS=305, BW=1221KiB/s (1250kB/s)(4976KiB/4076msec)
    slat (usec): min=35, max=4308, avg=1561.87, stdev=433.21
    clat (usec): min=1832, max=43568, avg=24234.10, stdev=4602.39
     lat (usec): min=1881, max=46635, avg=25798.60, stdev=4748.23
    clat percentiles (usec):
     |  1.00th=[16188],  5.00th=[18220], 10.00th=[19268], 20.00th=[20841],
     | 30.00th=[21627], 40.00th=[22676], 50.00th=[23725], 60.00th=[24773],
     | 70.00th=[26084], 80.00th=[27657], 90.00th=[30016], 95.00th=[32375],
     | 99.00th=[38011], 99.50th=[40109], 99.90th=[42206], 99.95th=[43779],
     | 99.99th=[43779]
   bw (  KiB/s): min= 1128, max= 1408, per=24.97%, avg=1219.00, stdev=98.44, samples=8
   iops        : min=  282, max=  352, avg=304.75, stdev=24.61, samples=8
  write: IOPS=322, BW=1291KiB/s (1322kB/s)(5264KiB/4076msec)
    slat (usec): min=13, max=558, avg=55.28, stdev=35.99
    clat (usec): min=5419, max=48951, avg=25015.50, stdev=4327.91
     lat (usec): min=5479, max=48995, avg=25072.20, stdev=4328.86
    clat percentiles (usec):
     |  1.00th=[14353],  5.00th=[19268], 10.00th=[20055], 20.00th=[21627],
     | 30.00th=[22676], 40.00th=[23725], 50.00th=[24773], 60.00th=[25822],
     | 70.00th=[26870], 80.00th=[28181], 90.00th=[30278], 95.00th=[32375],
     | 99.00th=[35914], 99.50th=[39584], 99.90th=[47449], 99.95th=[49021],
     | 99.99th=[49021]
   bw (  KiB/s): min= 1104, max= 1472, per=25.35%, avg=1288.00, stdev=114.50, samples=8
   iops        : min=  276, max=  368, avg=322.00, stdev=28.63, samples=8
  lat (msec)   : 2=0.08%, 4=0.04%, 10=0.27%, 20=11.52%, 50=88.09%
  cpu          : usr=0.96%, sys=6.77%, ctx=2279, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=24458: Thu May  1 03:33:10 2025
   read: IOPS=303, BW=1212KiB/s (1241kB/s)(4984KiB/4111msec)
    slat (usec): min=810, max=4330, avg=1567.36, stdev=423.47
    clat (usec): min=3111, max=40848, avg=24445.11, stdev=4504.95
     lat (usec): min=5150, max=43761, avg=26014.79, stdev=4611.90
    clat percentiles (usec):
     |  1.00th=[16188],  5.00th=[18220], 10.00th=[19268], 20.00th=[20841],
     | 30.00th=[21890], 40.00th=[22938], 50.00th=[23725], 60.00th=[25035],
     | 70.00th=[26346], 80.00th=[28181], 90.00th=[30278], 95.00th=[32375],
     | 99.00th=[37487], 99.50th=[38536], 99.90th=[40633], 99.95th=[40633],
     | 99.99th=[40633]
   bw (  KiB/s): min= 1040, max= 1384, per=24.87%, avg=1214.00, stdev=111.65, samples=8
   iops        : min=  260, max=  346, avg=303.50, stdev=27.91, samples=8
  write: IOPS=319, BW=1279KiB/s (1309kB/s)(5256KiB/4111msec)
    slat (usec): min=17, max=1914, avg=59.45, stdev=76.62
    clat (usec): min=14411, max=43011, avg=25004.30, stdev=3942.96
     lat (usec): min=14479, max=43098, avg=25065.05, stdev=3943.87
    clat percentiles (usec):
     |  1.00th=[17171],  5.00th=[19268], 10.00th=[20317], 20.00th=[21627],
     | 30.00th=[22938], 40.00th=[23987], 50.00th=[24511], 60.00th=[25560],
     | 70.00th=[26608], 80.00th=[27919], 90.00th=[30016], 95.00th=[31851],
     | 99.00th=[35914], 99.50th=[36963], 99.90th=[43254], 99.95th=[43254],
     | 99.99th=[43254]
   bw (  KiB/s): min= 1152, max= 1360, per=24.98%, avg=1269.00, stdev=85.20, samples=8
   iops        : min=  288, max=  340, avg=317.25, stdev=21.30, samples=8
  lat (msec)   : 4=0.04%, 20=11.33%, 50=88.63%
  cpu          : usr=1.12%, sys=6.57%, ctx=2367, majf=0, minf=36
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=24459: Thu May  1 03:33:10 2025
   read: IOPS=308, BW=1233KiB/s (1263kB/s)(5060KiB/4103msec)
    slat (usec): min=709, max=3945, avg=1547.26, stdev=422.29
    clat (usec): min=13103, max=42799, avg=24248.13, stdev=4377.06
     lat (usec): min=15747, max=45369, avg=25797.88, stdev=4501.78
    clat percentiles (usec):
     |  1.00th=[15926],  5.00th=[18220], 10.00th=[19006], 20.00th=[20579],
     | 30.00th=[21890], 40.00th=[22938], 50.00th=[23725], 60.00th=[24773],
     | 70.00th=[25822], 80.00th=[27395], 90.00th=[30016], 95.00th=[32375],
     | 99.00th=[36963], 99.50th=[38011], 99.90th=[42730], 99.95th=[42730],
     | 99.99th=[42730]
   bw (  KiB/s): min= 1144, max= 1392, per=25.22%, avg=1231.00, stdev=85.36, samples=8
   iops        : min=  286, max=  348, avg=307.75, stdev=21.34, samples=8
  write: IOPS=315, BW=1262KiB/s (1293kB/s)(5180KiB/4103msec)
    slat (usec): min=14, max=1920, avg=57.94, stdev=70.27
    clat (usec): min=3266, max=42982, avg=25065.20, stdev=4063.01
     lat (usec): min=3323, max=43011, avg=25124.48, stdev=4064.32
    clat percentiles (usec):
     |  1.00th=[17171],  5.00th=[19268], 10.00th=[20317], 20.00th=[21627],
     | 30.00th=[22676], 40.00th=[23725], 50.00th=[24773], 60.00th=[25822],
     | 70.00th=[26870], 80.00th=[28181], 90.00th=[30540], 95.00th=[32113],
     | 99.00th=[35914], 99.50th=[38011], 99.90th=[40633], 99.95th=[42730],
     | 99.99th=[42730]
   bw (  KiB/s): min= 1184, max= 1416, per=24.70%, avg=1255.00, stdev=80.28, samples=8
   iops        : min=  296, max=  354, avg=313.75, stdev=20.07, samples=8
  lat (msec)   : 4=0.04%, 20=11.95%, 50=88.01%
  cpu          : usr=1.49%, sys=6.19%, ctx=2367, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=4882KiB/s (4999kB/s), 1212KiB/s-1233KiB/s (1241kB/s-1263kB/s), io=19.6MiB (20.5MB), run=4076-4111msec
  WRITE: bw=5082KiB/s (5204kB/s), 1262KiB/s-1291KiB/s (1293kB/s-1322kB/s), io=20.4MiB (21.4MB), run=4076-4111msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.7
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=24466: Thu May  1 03:33:11 2025
   read: IOPS=9, BW=388MiB/s (407MB/s)(40.0MiB/103msec)
    slat (nsec): min=6680.8k, max=6680.8k, avg=6680844.00, stdev= 0.00
    clat (nsec): min=91669k, max=91669k, avg=91668717.00, stdev= 0.00
     lat (nsec): min=98372k, max=98372k, avg=98372394.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[91751],  5.00th=[91751], 10.00th=[91751], 20.00th=[91751],
     | 30.00th=[91751], 40.00th=[91751], 50.00th=[91751], 60.00th=[91751],
     | 70.00th=[91751], 80.00th=[91751], 90.00th=[91751], 95.00th=[91751],
     | 99.00th=[91751], 99.50th=[91751], 99.90th=[91751], 99.95th=[91751],
     | 99.99th=[91751]
  lat (msec)   : 100=100.00%
  cpu          : usr=0.00%, sys=6.86%, ctx=8, majf=0, minf=29
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=388MiB/s (407MB/s), 388MiB/s-388MiB/s (407MB/s-407MB/s), io=40.0MiB (41.9MB), run=103-103msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout