== sanity test 398c: run fio to test AIO ================= 09:24:58 (1788269098)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB, 40 MiB) copied, 0.152779 s, 275 MB/s
osc.lustre-OST0000-osc-ffff9a135093c800.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=179982: Tue Sep  1 09:25:09 2026
  write: IOPS=354, BW=1419KiB/s (1453kB/s)(10.0MiB/7218msec); 0 zone resets
    slat (usec): min=26, max=2526, avg=106.12, stdev=100.65
    clat (usec): min=9193, max=82453, avg=44915.55, stdev=9138.49
     lat (usec): min=9327, max=82550, avg=45022.76, stdev=9143.28
    clat percentiles (usec):
     |  1.00th=[24773],  5.00th=[33424], 10.00th=[35914], 20.00th=[38536],
     | 30.00th=[40109], 40.00th=[41681], 50.00th=[43779], 60.00th=[45876],
     | 70.00th=[47449], 80.00th=[51119], 90.00th=[55837], 95.00th=[61080],
     | 99.00th=[74974], 99.50th=[80217], 99.90th=[81265], 99.95th=[81265],
     | 99.99th=[82314]
   bw (  KiB/s): min= 1024, max= 1600, per=25.16%, avg=1417.00, stdev=160.56, samples=14
   iops        : min=  256, max=  400, avg=354.21, stdev=40.16, samples=14
  lat (msec)   : 10=0.08%, 20=0.43%, 50=76.84%, 100=22.66%
  cpu          : usr=0.46%, sys=2.91%, ctx=2725, majf=0, minf=9
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=179983: Tue Sep  1 09:25:09 2026
  write: IOPS=353, BW=1412KiB/s (1446kB/s)(10.0MiB/7250msec); 0 zone resets
    slat (usec): min=24, max=27933, avg=118.66, stdev=557.14
    clat (usec): min=8409, max=82983, avg=45002.59, stdev=8821.02
     lat (usec): min=8496, max=83047, avg=45122.51, stdev=8810.50
    clat percentiles (usec):
     |  1.00th=[25822],  5.00th=[33817], 10.00th=[36439], 20.00th=[38536],
     | 30.00th=[40109], 40.00th=[42206], 50.00th=[43779], 60.00th=[45351],
     | 70.00th=[47973], 80.00th=[51119], 90.00th=[55313], 95.00th=[61080],
     | 99.00th=[72877], 99.50th=[80217], 99.90th=[81265], 99.95th=[82314],
     | 99.99th=[83362]
   bw (  KiB/s): min= 1024, max= 1568, per=25.02%, avg=1409.64, stdev=151.08, samples=14
   iops        : min=  256, max=  392, avg=352.36, stdev=37.78, samples=14
  lat (msec)   : 10=0.04%, 20=0.12%, 50=77.11%, 100=22.73%
  cpu          : usr=0.57%, sys=3.24%, ctx=2815, majf=0, minf=9
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=179984: Tue Sep  1 09:25:09 2026
  write: IOPS=352, BW=1409KiB/s (1443kB/s)(10.0MiB/7269msec); 0 zone resets
    slat (usec): min=22, max=2189, avg=102.80, stdev=94.71
    clat (usec): min=23574, max=83387, avg=44970.63, stdev=9077.33
     lat (usec): min=23623, max=83428, avg=45074.54, stdev=9084.22
    clat percentiles (usec):
     |  1.00th=[26346],  5.00th=[33424], 10.00th=[35914], 20.00th=[38011],
     | 30.00th=[40109], 40.00th=[42206], 50.00th=[43779], 60.00th=[45351],
     | 70.00th=[47973], 80.00th=[50594], 90.00th=[56361], 95.00th=[61604],
     | 99.00th=[74974], 99.50th=[81265], 99.90th=[82314], 99.95th=[83362],
     | 99.99th=[83362]
   bw (  KiB/s): min=  984, max= 1560, per=24.92%, avg=1403.86, stdev=158.88, samples=14
   iops        : min=  246, max=  390, avg=350.93, stdev=39.74, samples=14
  lat (msec)   : 50=78.20%, 100=21.80%
  cpu          : usr=0.58%, sys=3.10%, ctx=2620, majf=0, minf=9
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=179985: Tue Sep  1 09:25:09 2026
  write: IOPS=352, BW=1408KiB/s (1442kB/s)(10.0MiB/7271msec); 0 zone resets
    slat (usec): min=19, max=4197, avg=99.90, stdev=121.60
    clat (usec): min=19843, max=81110, avg=44962.91, stdev=9158.94
     lat (usec): min=20016, max=81273, avg=45063.81, stdev=9164.22
    clat percentiles (usec):
     |  1.00th=[24249],  5.00th=[33817], 10.00th=[36439], 20.00th=[38011],
     | 30.00th=[40109], 40.00th=[42206], 50.00th=[43779], 60.00th=[45351],
     | 70.00th=[47973], 80.00th=[50594], 90.00th=[55837], 95.00th=[61080],
     | 99.00th=[77071], 99.50th=[81265], 99.90th=[81265], 99.95th=[81265],
     | 99.99th=[81265]
   bw (  KiB/s): min= 1008, max= 1552, per=24.89%, avg=1402.14, stdev=156.45, samples=14
   iops        : min=  252, max=  388, avg=350.50, stdev=39.14, samples=14
  lat (msec)   : 20=0.04%, 50=77.66%, 100=22.30%
  cpu          : usr=0.37%, sys=2.87%, ctx=2602, majf=0, minf=8
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=5633KiB/s (5769kB/s), 1408KiB/s-1419KiB/s (1442kB/s-1453kB/s), io=40.0MiB (41.9MB), run=7218-7271msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=179996: Tue Sep  1 09:25:20 2026
  read: IOPS=176, BW=707KiB/s (724kB/s)(5048KiB/7141msec)
    slat (usec): min=35, max=10548, avg=2867.31, stdev=1188.10
    clat (usec): min=6423, max=71182, avg=42449.26, stdev=8635.26
     lat (usec): min=6579, max=75527, avg=45319.11, stdev=9006.16
    clat percentiles (usec):
     |  1.00th=[26346],  5.00th=[31327], 10.00th=[33424], 20.00th=[35914],
     | 30.00th=[38011], 40.00th=[40109], 50.00th=[41157], 60.00th=[43254],
     | 70.00th=[44827], 80.00th=[47973], 90.00th=[54789], 95.00th=[59507],
     | 99.00th=[66323], 99.50th=[68682], 99.90th=[70779], 99.95th=[70779],
     | 99.99th=[70779]
   bw (  KiB/s): min=  574, max=  856, per=25.12%, avg=700.71, stdev=81.53, samples=14
   iops        : min=  143, max=  214, avg=174.93, stdev=20.38, samples=14
  write: IOPS=181, BW=727KiB/s (745kB/s)(5192KiB/7141msec); 0 zone resets
    slat (usec): min=25, max=2814, avg=102.42, stdev=132.26
    clat (usec): min=21906, max=74762, avg=43717.13, stdev=8183.41
     lat (usec): min=22020, max=74873, avg=43820.77, stdev=8188.23
    clat percentiles (usec):
     |  1.00th=[28705],  5.00th=[33424], 10.00th=[34866], 20.00th=[36963],
     | 30.00th=[38536], 40.00th=[40109], 50.00th=[41681], 60.00th=[44303],
     | 70.00th=[46924], 80.00th=[50594], 90.00th=[54789], 95.00th=[59507],
     | 99.00th=[67634], 99.50th=[69731], 99.90th=[70779], 99.95th=[74974],
     | 99.99th=[74974]
   bw (  KiB/s): min=  624, max=  840, per=24.93%, avg=724.07, stdev=58.15, samples=14
   iops        : min=  156, max=  210, avg=180.79, stdev=14.61, samples=14
  lat (msec)   : 10=0.20%, 20=0.23%, 50=81.05%, 100=18.52%
  cpu          : usr=0.49%, sys=3.87%, ctx=4562, majf=0, minf=14
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=179997: Tue Sep  1 09:25:20 2026
  read: IOPS=173, BW=694KiB/s (711kB/s)(4976KiB/7169msec)
    slat (usec): min=1074, max=9622, avg=2910.03, stdev=1111.13
    clat (usec): min=17123, max=71804, avg=42708.63, stdev=8099.69
     lat (usec): min=21243, max=76151, avg=45620.92, stdev=8339.44
    clat percentiles (usec):
     |  1.00th=[28967],  5.00th=[31851], 10.00th=[33817], 20.00th=[35914],
     | 30.00th=[38011], 40.00th=[39584], 50.00th=[41681], 60.00th=[43779],
     | 70.00th=[45876], 80.00th=[48497], 90.00th=[53740], 95.00th=[59507],
     | 99.00th=[65799], 99.50th=[67634], 99.90th=[70779], 99.95th=[71828],
     | 99.99th=[71828]
   bw (  KiB/s): min=  582, max=  840, per=24.81%, avg=692.21, stdev=81.76, samples=14
   iops        : min=  145, max=  210, avg=172.86, stdev=20.52, samples=14
  write: IOPS=183, BW=734KiB/s (752kB/s)(5264KiB/7169msec); 0 zone resets
    slat (usec): min=34, max=1937, avg=99.59, stdev=89.01
    clat (usec): min=9484, max=74097, avg=43472.13, stdev=7998.26
     lat (usec): min=9561, max=74194, avg=43572.97, stdev=8002.62
    clat percentiles (usec):
     |  1.00th=[30802],  5.00th=[32900], 10.00th=[34866], 20.00th=[37487],
     | 30.00th=[38536], 40.00th=[40109], 50.00th=[41681], 60.00th=[43779],
     | 70.00th=[45876], 80.00th=[50070], 90.00th=[54789], 95.00th=[59507],
     | 99.00th=[66847], 99.50th=[67634], 99.90th=[71828], 99.95th=[73925],
     | 99.99th=[73925]
   bw (  KiB/s): min=  616, max=  832, per=25.31%, avg=735.00, stdev=68.57, samples=14
   iops        : min=  154, max=  208, avg=183.50, stdev=17.20, samples=14
  lat (msec)   : 10=0.04%, 20=0.04%, 50=82.03%, 100=17.89%
  cpu          : usr=0.40%, sys=3.96%, ctx=4603, majf=0, minf=12
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=179998: Tue Sep  1 09:25:20 2026
  read: IOPS=173, BW=693KiB/s (710kB/s)(4984KiB/7192msec)
    slat (usec): min=1141, max=11679, avg=2929.11, stdev=1146.00
    clat (usec): min=4597, max=78050, avg=42693.55, stdev=8491.69
     lat (usec): min=10463, max=83653, avg=45625.36, stdev=8748.34
    clat percentiles (usec):
     |  1.00th=[27395],  5.00th=[31065], 10.00th=[33424], 20.00th=[35914],
     | 30.00th=[38011], 40.00th=[39584], 50.00th=[41681], 60.00th=[43254],
     | 70.00th=[45351], 80.00th=[48497], 90.00th=[53740], 95.00th=[58983],
     | 99.00th=[67634], 99.50th=[70779], 99.90th=[74974], 99.95th=[78119],
     | 99.99th=[78119]
   bw (  KiB/s): min=  630, max=  784, per=24.79%, avg=691.64, stdev=41.39, samples=14
   iops        : min=  157, max=  196, avg=172.71, stdev=10.35, samples=14
  write: IOPS=182, BW=731KiB/s (748kB/s)(5256KiB/7192msec); 0 zone resets
    slat (usec): min=33, max=2699, avg=98.52, stdev=94.72
    clat (usec): min=18122, max=79978, avg=43542.63, stdev=8118.13
     lat (usec): min=18250, max=80030, avg=43642.18, stdev=8119.84
    clat percentiles (usec):
     |  1.00th=[29492],  5.00th=[33162], 10.00th=[34866], 20.00th=[36963],
     | 30.00th=[39060], 40.00th=[40109], 50.00th=[42206], 60.00th=[43779],
     | 70.00th=[46400], 80.00th=[49546], 90.00th=[53740], 95.00th=[57934],
     | 99.00th=[70779], 99.50th=[73925], 99.90th=[80217], 99.95th=[80217],
     | 99.99th=[80217]
   bw (  KiB/s): min=  598, max=  944, per=24.98%, avg=725.36, stdev=102.28, samples=14
   iops        : min=  149, max=  236, avg=181.14, stdev=25.68, samples=14
  lat (msec)   : 10=0.04%, 20=0.12%, 50=81.76%, 100=18.09%
  cpu          : usr=0.43%, sys=3.95%, ctx=4642, majf=0, minf=11
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=179999: Tue Sep  1 09:25:20 2026
  read: IOPS=175, BW=704KiB/s (720kB/s)(5060KiB/7192msec)
    slat (usec): min=1105, max=9470, avg=2902.99, stdev=1129.21
    clat (usec): min=5800, max=77625, avg=42769.55, stdev=8568.96
     lat (usec): min=7300, max=79633, avg=45675.00, stdev=8819.26
    clat percentiles (usec):
     |  1.00th=[26346],  5.00th=[31327], 10.00th=[33424], 20.00th=[35914],
     | 30.00th=[38011], 40.00th=[39584], 50.00th=[41681], 60.00th=[43779],
     | 70.00th=[45351], 80.00th=[48497], 90.00th=[54264], 95.00th=[60031],
     | 99.00th=[66323], 99.50th=[68682], 99.90th=[74974], 99.95th=[78119],
     | 99.99th=[78119]
   bw (  KiB/s): min=  566, max=  784, per=25.17%, avg=702.29, stdev=61.03, samples=14
   iops        : min=  141, max=  196, avg=175.36, stdev=15.32, samples=14
  write: IOPS=180, BW=720KiB/s (738kB/s)(5180KiB/7192msec); 0 zone resets
    slat (usec): min=35, max=707, avg=97.40, stdev=58.38
    clat (usec): min=21614, max=77084, avg=43677.11, stdev=7823.83
     lat (usec): min=21671, max=77134, avg=43775.82, stdev=7824.49
    clat percentiles (usec):
     |  1.00th=[30016],  5.00th=[33424], 10.00th=[35390], 20.00th=[37487],
     | 30.00th=[39060], 40.00th=[40633], 50.00th=[42206], 60.00th=[44303],
     | 70.00th=[46400], 80.00th=[49546], 90.00th=[54264], 95.00th=[58983],
     | 99.00th=[69731], 99.50th=[71828], 99.90th=[74974], 99.95th=[77071],
     | 99.99th=[77071]
   bw (  KiB/s): min=  624, max=  864, per=24.70%, avg=717.14, stdev=72.75, samples=14
   iops        : min=  156, max=  216, avg=179.07, stdev=18.26, samples=14
  lat (msec)   : 10=0.12%, 20=0.08%, 50=82.42%, 100=17.38%
  cpu          : usr=0.51%, sys=3.89%, ctx=4657, majf=0, minf=11
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=2790KiB/s (2857kB/s), 693KiB/s-707KiB/s (710kB/s-724kB/s), io=19.6MiB (20.5MB), run=7141-7192msec
  WRITE: bw=2905KiB/s (2975kB/s), 720KiB/s-734KiB/s (738kB/s-752kB/s), io=20.4MiB (21.4MB), run=7141-7192msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.19
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=180004: Tue Sep  1 09:25:23 2026
  read: IOPS=9, BW=385MiB/s (403MB/s)(40.0MiB/104msec)
    slat (nsec): min=10842k, max=10842k, avg=10842382.00, stdev= 0.00
    clat (nsec): min=88198k, max=88198k, avg=88198334.00, stdev= 0.00
     lat (nsec): min=99045k, max=99045k, avg=99045211.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[88605],  5.00th=[88605], 10.00th=[88605], 20.00th=[88605],
     | 30.00th=[88605], 40.00th=[88605], 50.00th=[88605], 60.00th=[88605],
     | 70.00th=[88605], 80.00th=[88605], 90.00th=[88605], 95.00th=[88605],
     | 99.00th=[88605], 99.50th=[88605], 99.90th=[88605], 99.95th=[88605],
     | 99.99th=[88605]
  lat (msec)   : 100=100.00%
  cpu          : usr=0.00%, sys=9.71%, ctx=11, majf=0, minf=10
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=385MiB/s (403MB/s), 385MiB/s-385MiB/s (403MB/s-403MB/s), io=40.0MiB (41.9MB), run=104-104msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout