== sanity test 398c: run fio to test AIO ================= 13:36:41 (1774892201)
/usr/bin/fio
debug=0
40+0 records in
40+0 records out
41943040 bytes (42 MB, 40 MiB) copied, 0.118255 s, 355 MB/s
osc.lustre-OST0000-osc-ffff8eca09401800.rpc_stats=clear
writing 40M to OST0 by fio with 4 jobs...
rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-write: (groupid=0, jobs=1): err= 0: pid=183497: Mon Mar 30 13:36:48 2026
  write: IOPS=536, BW=2146KiB/s (2197kB/s)(10.0MiB/4772msec); 0 zone resets
    slat (usec): min=36, max=756, avg=169.13, stdev=88.79
    clat (usec): min=14980, max=49394, avg=29263.02, stdev=3564.30
     lat (usec): min=15330, max=49674, avg=29433.21, stdev=3575.37
    clat percentiles (usec):
     |  1.00th=[21365],  5.00th=[24511], 10.00th=[25822], 20.00th=[27132],
     | 30.00th=[27657], 40.00th=[28181], 50.00th=[28967], 60.00th=[29492],
     | 70.00th=[30278], 80.00th=[31065], 90.00th=[32637], 95.00th=[35914],
     | 99.00th=[41157], 99.50th=[45351], 99.90th=[49546], 99.95th=[49546],
     | 99.99th=[49546]
   bw (  KiB/s): min= 1888, max= 2296, per=24.98%, avg=2143.67, stdev=144.52, samples=9
   iops        : min=  472, max=  574, avg=535.89, stdev=36.15, samples=9
  lat (msec)   : 20=0.35%, 50=99.65%
  cpu          : usr=0.59%, sys=6.73%, ctx=2596, majf=0, minf=8
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=183498: Mon Mar 30 13:36:48 2026
  write: IOPS=536, BW=2146KiB/s (2197kB/s)(10.0MiB/4772msec); 0 zone resets
    slat (usec): min=35, max=791, avg=165.41, stdev=86.07
    clat (usec): min=3108, max=49389, avg=29291.86, stdev=3605.18
     lat (usec): min=3432, max=49677, avg=29458.26, stdev=3613.43
    clat percentiles (usec):
     |  1.00th=[21627],  5.00th=[24773], 10.00th=[25822], 20.00th=[26870],
     | 30.00th=[27657], 40.00th=[28443], 50.00th=[28967], 60.00th=[29492],
     | 70.00th=[30278], 80.00th=[30802], 90.00th=[32637], 95.00th=[36439],
     | 99.00th=[41157], 99.50th=[47973], 99.90th=[49021], 99.95th=[49546],
     | 99.99th=[49546]
   bw (  KiB/s): min= 1920, max= 2312, per=24.97%, avg=2143.56, stdev=139.61, samples=9
   iops        : min=  480, max=  578, avg=535.89, stdev=34.90, samples=9
  lat (msec)   : 4=0.04%, 20=0.78%, 50=99.18%
  cpu          : usr=0.71%, sys=7.00%, ctx=2602, majf=0, minf=9
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=183499: Mon Mar 30 13:36:48 2026
  write: IOPS=538, BW=2154KiB/s (2205kB/s)(10.0MiB/4755msec); 0 zone resets
    slat (usec): min=44, max=20928, avg=180.58, stdev=420.30
    clat (usec): min=5481, max=49521, avg=29199.25, stdev=3823.56
     lat (usec): min=5754, max=49821, avg=29380.86, stdev=3830.72
    clat percentiles (usec):
     |  1.00th=[20055],  5.00th=[24511], 10.00th=[25560], 20.00th=[26870],
     | 30.00th=[27657], 40.00th=[28443], 50.00th=[28967], 60.00th=[29492],
     | 70.00th=[30278], 80.00th=[31065], 90.00th=[32375], 95.00th=[36439],
     | 99.00th=[41681], 99.50th=[46400], 99.90th=[49546], 99.95th=[49546],
     | 99.99th=[49546]
   bw (  KiB/s): min= 1864, max= 2328, per=25.09%, avg=2153.56, stdev=152.82, samples=9
   iops        : min=  466, max=  582, avg=538.33, stdev=38.20, samples=9
  lat (msec)   : 10=0.12%, 20=0.90%, 50=98.98%
  cpu          : usr=0.57%, sys=7.38%, ctx=2644, majf=0, minf=9
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-write: (groupid=0, jobs=1): err= 0: pid=183500: Mon Mar 30 13:36:48 2026
  write: IOPS=538, BW=2155KiB/s (2207kB/s)(10.0MiB/4751msec); 0 zone resets
    slat (usec): min=38, max=601, avg=157.68, stdev=82.77
    clat (usec): min=17219, max=57792, avg=29471.21, stdev=4111.35
     lat (usec): min=17565, max=57870, avg=29629.93, stdev=4118.58
    clat percentiles (usec):
     |  1.00th=[21103],  5.00th=[24249], 10.00th=[25822], 20.00th=[26870],
     | 30.00th=[27657], 40.00th=[28443], 50.00th=[28967], 60.00th=[29492],
     | 70.00th=[30278], 80.00th=[31065], 90.00th=[32900], 95.00th=[37487],
     | 99.00th=[46400], 99.50th=[49021], 99.90th=[54789], 99.95th=[57934],
     | 99.99th=[57934]
   bw (  KiB/s): min= 1912, max= 2360, per=25.14%, avg=2157.67, stdev=155.64, samples=9
   iops        : min=  478, max=  590, avg=539.33, stdev=38.88, samples=9
  lat (msec)   : 20=0.70%, 50=98.83%, 100=0.47%
  cpu          : usr=0.69%, sys=6.27%, ctx=2566, majf=0, minf=9
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  WRITE: bw=8583KiB/s (8789kB/s), 2146KiB/s-2155KiB/s (2197kB/s-2207kB/s), io=40.0MiB (41.9MB), run=4751-4772msec
mix rw 40M to OST0 by fio with 4 jobs...
rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.19
Starting 4 processes

rand-rw: (groupid=0, jobs=1): err= 0: pid=183510: Mon Mar 30 13:36:57 2026
  read: IOPS=212, BW=851KiB/s (872kB/s)(5048KiB/5931msec)
    slat (usec): min=1007, max=10035, avg=2611.26, stdev=821.54
    clat (usec): min=4908, max=53588, avg=35129.66, stdev=5637.47
     lat (usec): min=7728, max=58456, avg=37742.55, stdev=5763.21
    clat percentiles (usec):
     |  1.00th=[22676],  5.00th=[27132], 10.00th=[28443], 20.00th=[30540],
     | 30.00th=[32113], 40.00th=[33424], 50.00th=[34866], 60.00th=[36439],
     | 70.00th=[38011], 80.00th=[39584], 90.00th=[42206], 95.00th=[44827],
     | 99.00th=[49546], 99.50th=[51643], 99.90th=[52691], 99.95th=[53740],
     | 99.99th=[53740]
   bw (  KiB/s): min=  699, max=  912, per=24.90%, avg=842.45, stdev=67.25, samples=11
   iops        : min=  174, max=  228, avg=210.55, stdev=16.97, samples=11
  write: IOPS=218, BW=875KiB/s (896kB/s)(5192KiB/5931msec); 0 zone resets
    slat (usec): min=54, max=825, avg=160.69, stdev=82.36
    clat (usec): min=4732, max=54272, avg=35459.47, stdev=4694.57
     lat (usec): min=4894, max=54405, avg=35621.24, stdev=4694.63
    clat percentiles (usec):
     |  1.00th=[26608],  5.00th=[29230], 10.00th=[30540], 20.00th=[31851],
     | 30.00th=[32900], 40.00th=[33817], 50.00th=[34866], 60.00th=[35914],
     | 70.00th=[37487], 80.00th=[39060], 90.00th=[41681], 95.00th=[43779],
     | 99.00th=[49546], 99.50th=[51119], 99.90th=[54264], 99.95th=[54264],
     | 99.99th=[54264]
   bw (  KiB/s): min=  712, max=  984, per=24.76%, avg=872.00, stdev=90.51, samples=11
   iops        : min=  178, max=  246, avg=218.00, stdev=22.63, samples=11
  lat (msec)   : 10=0.20%, 20=0.23%, 50=98.87%, 100=0.70%
  cpu          : usr=0.51%, sys=6.26%, ctx=4733, majf=0, minf=11
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=183511: Mon Mar 30 13:36:57 2026
  read: IOPS=211, BW=848KiB/s (868kB/s)(4976KiB/5870msec)
    slat (usec): min=33, max=10447, avg=2573.98, stdev=824.49
    clat (usec): min=6527, max=60712, avg=35083.78, stdev=5408.67
     lat (usec): min=9605, max=63686, avg=37659.57, stdev=5539.46
    clat percentiles (usec):
     |  1.00th=[24511],  5.00th=[27395], 10.00th=[28705], 20.00th=[30802],
     | 30.00th=[32375], 40.00th=[33817], 50.00th=[34866], 60.00th=[35914],
     | 70.00th=[37487], 80.00th=[39060], 90.00th=[41681], 95.00th=[44303],
     | 99.00th=[49546], 99.50th=[53216], 99.90th=[57934], 99.95th=[60556],
     | 99.99th=[60556]
   bw (  KiB/s): min=  712, max=  912, per=24.98%, avg=845.09, stdev=73.61, samples=11
   iops        : min=  178, max=  228, avg=211.27, stdev=18.40, samples=11
  write: IOPS=224, BW=897KiB/s (918kB/s)(5264KiB/5870msec); 0 zone resets
    slat (usec): min=33, max=1022, avg=156.14, stdev=82.40
    clat (usec): min=9287, max=53904, avg=35446.71, stdev=4970.83
     lat (usec): min=9474, max=54011, avg=35603.87, stdev=4976.07
    clat percentiles (usec):
     |  1.00th=[25560],  5.00th=[28705], 10.00th=[30278], 20.00th=[31851],
     | 30.00th=[32900], 40.00th=[33817], 50.00th=[34866], 60.00th=[35914],
     | 70.00th=[37487], 80.00th=[39060], 90.00th=[42206], 95.00th=[44303],
     | 99.00th=[49021], 99.50th=[50070], 99.90th=[51119], 99.95th=[53740],
     | 99.99th=[53740]
   bw (  KiB/s): min=  816, max= 1016, per=25.47%, avg=897.18, stdev=57.78, samples=11
   iops        : min=  204, max=  254, avg=224.27, stdev=14.46, samples=11
  lat (msec)   : 10=0.12%, 20=0.66%, 50=98.55%, 100=0.66%
  cpu          : usr=0.31%, sys=6.46%, ctx=4632, majf=0, minf=11
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=183512: Mon Mar 30 13:36:57 2026
  read: IOPS=210, BW=844KiB/s (864kB/s)(4984KiB/5908msec)
    slat (usec): min=57, max=9447, avg=2573.14, stdev=824.60
    clat (usec): min=5106, max=58432, avg=35231.21, stdev=5439.34
     lat (usec): min=7184, max=62078, avg=37806.09, stdev=5560.64
    clat percentiles (usec):
     |  1.00th=[24511],  5.00th=[27395], 10.00th=[28705], 20.00th=[30540],
     | 30.00th=[32375], 40.00th=[33817], 50.00th=[34866], 60.00th=[36439],
     | 70.00th=[38011], 80.00th=[39584], 90.00th=[42206], 95.00th=[44303],
     | 99.00th=[49546], 99.50th=[51119], 99.90th=[56886], 99.95th=[58459],
     | 99.99th=[58459]
   bw (  KiB/s): min=  731, max=  944, per=25.01%, avg=846.09, stdev=76.95, samples=11
   iops        : min=  182, max=  236, avg=211.45, stdev=19.35, samples=11
  write: IOPS=222, BW=890KiB/s (911kB/s)(5256KiB/5908msec); 0 zone resets
    slat (usec): min=43, max=701, avg=160.41, stdev=77.01
    clat (usec): min=24009, max=54648, avg=35595.19, stdev=4735.08
     lat (usec): min=24136, max=54837, avg=35756.85, stdev=4732.30
    clat percentiles (usec):
     |  1.00th=[26870],  5.00th=[29230], 10.00th=[30278], 20.00th=[31589],
     | 30.00th=[32900], 40.00th=[33817], 50.00th=[34866], 60.00th=[35914],
     | 70.00th=[37487], 80.00th=[39060], 90.00th=[41681], 95.00th=[44827],
     | 99.00th=[49546], 99.50th=[51643], 99.90th=[54264], 99.95th=[54789],
     | 99.99th=[54789]
   bw (  KiB/s): min=  768, max= 1088, per=25.09%, avg=883.55, stdev=106.19, samples=11
   iops        : min=  192, max=  272, avg=220.82, stdev=26.62, samples=11
  lat (msec)   : 10=0.08%, 20=0.04%, 50=99.02%, 100=0.86%
  cpu          : usr=0.58%, sys=6.25%, ctx=4638, majf=0, minf=11
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16
rand-rw: (groupid=0, jobs=1): err= 0: pid=183513: Mon Mar 30 13:36:57 2026
  read: IOPS=213, BW=854KiB/s (875kB/s)(5060KiB/5923msec)
    slat (usec): min=112, max=9762, avg=2561.62, stdev=809.64
    clat (usec): min=9332, max=58576, avg=35296.29, stdev=5583.08
     lat (usec): min=12050, max=61038, avg=37859.99, stdev=5717.57
    clat percentiles (usec):
     |  1.00th=[24249],  5.00th=[27132], 10.00th=[28705], 20.00th=[30540],
     | 30.00th=[32113], 40.00th=[33424], 50.00th=[34866], 60.00th=[36439],
     | 70.00th=[38011], 80.00th=[39584], 90.00th=[42206], 95.00th=[44303],
     | 99.00th=[51643], 99.50th=[53740], 99.90th=[56361], 99.95th=[58459],
     | 99.99th=[58459]
   bw (  KiB/s): min=  694, max=  896, per=25.17%, avg=851.45, stdev=57.94, samples=11
   iops        : min=  173, max=  224, avg=212.82, stdev=14.62, samples=11
  write: IOPS=218, BW=875KiB/s (896kB/s)(5180KiB/5923msec); 0 zone resets
    slat (usec): min=42, max=3340, avg=161.44, stdev=116.82
    clat (usec): min=8678, max=53539, avg=35434.95, stdev=4592.47
     lat (usec): min=8903, max=53682, avg=35597.42, stdev=4600.79
    clat percentiles (usec):
     |  1.00th=[26870],  5.00th=[29230], 10.00th=[30278], 20.00th=[31589],
     | 30.00th=[32637], 40.00th=[33817], 50.00th=[34866], 60.00th=[35914],
     | 70.00th=[37487], 80.00th=[39060], 90.00th=[41681], 95.00th=[43779],
     | 99.00th=[49021], 99.50th=[50070], 99.90th=[53216], 99.95th=[53740],
     | 99.99th=[53740]
   bw (  KiB/s): min=  720, max=  968, per=24.84%, avg=874.91, stdev=71.94, samples=11
   iops        : min=  180, max=  242, avg=218.73, stdev=17.98, samples=11
  lat (msec)   : 10=0.08%, 50=98.98%, 100=0.94%
  cpu          : usr=0.46%, sys=6.28%, ctx=4753, majf=0, minf=11
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=3384KiB/s (3465kB/s), 844KiB/s-854KiB/s (864kB/s-875kB/s), io=19.6MiB (20.5MB), run=5870-5931msec
  WRITE: bw=3523KiB/s (3607kB/s), 875KiB/s-897KiB/s (896kB/s-918kB/s), io=20.4MiB (21.4MB), run=5870-5931msec
AIO with large block size 40M
rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16
fio-3.19
Starting 1 process

rand-rw: (groupid=0, jobs=1): err= 0: pid=183517: Mon Mar 30 13:37:01 2026
  read: IOPS=9, BW=388MiB/s (407MB/s)(40.0MiB/103msec)
    slat (nsec): min=9647.7k, max=9647.7k, avg=9647686.00, stdev= 0.00
    clat (nsec): min=88027k, max=88027k, avg=88026600.00, stdev= 0.00
     lat (nsec): min=97679k, max=97679k, avg=97678745.00, stdev= 0.00
    clat percentiles (usec):
     |  1.00th=[87557],  5.00th=[87557], 10.00th=[87557], 20.00th=[87557],
     | 30.00th=[87557], 40.00th=[87557], 50.00th=[87557], 60.00th=[87557],
     | 70.00th=[87557], 80.00th=[87557], 90.00th=[87557], 95.00th=[87557],
     | 99.00th=[87557], 99.50th=[87557], 99.90th=[87557], 99.95th=[87557],
     | 99.99th=[87557]
  lat (msec)   : 100=100.00%
  cpu          : usr=0.00%, sys=7.84%, ctx=9, majf=0, minf=7
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=388MiB/s (407MB/s), 388MiB/s-388MiB/s (407MB/s-407MB/s), io=40.0MiB (41.9MB), run=103-103msec
debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout