== sanity test 398c: run fio to test AIO ================= 21:44:38 (1787103878) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.161944 s, 259 MB/s osc.lustre-OST0000-osc-ffff988c08602000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=190321: Tue Aug 18 21:44:52 2026 write: IOPS=243, BW=975KiB/s (999kB/s)(10.0MiB/10498msec); 0 zone resets slat (usec): min=35, max=2191, avg=191.52, stdev=190.94 clat (msec): min=8, max=127, avg=65.34, stdev=12.39 lat (msec): min=8, max=127, avg=65.53, stdev=12.39 clat percentiles (msec): | 1.00th=[ 43], 5.00th=[ 48], 10.00th=[ 51], 20.00th=[ 55], | 30.00th=[ 58], 40.00th=[ 62], 50.00th=[ 65], 60.00th=[ 68], | 70.00th=[ 71], 80.00th=[ 75], 90.00th=[ 82], 95.00th=[ 87], | 99.00th=[ 99], 99.50th=[ 101], 99.90th=[ 128], 99.95th=[ 128], | 99.99th=[ 128] bw ( KiB/s): min= 782, max= 1152, per=24.90%, avg=968.85, stdev=102.99, samples=20 iops : min= 195, max= 288, avg=242.10, stdev=25.78, samples=20 lat (msec) : 10=0.04%, 20=0.08%, 50=8.24%, 100=91.17%, 250=0.47% cpu : usr=0.41%, sys=2.96%, ctx=2788, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=190322: Tue Aug 18 21:44:52 2026 write: IOPS=243, BW=973KiB/s (996kB/s)(10.0MiB/10526msec); 0 zone resets slat (usec): min=37, max=3859, avg=217.81, stdev=232.91 clat (msec): min=17, max=124, avg=65.09, stdev=12.33 lat (msec): min=17, max=124, avg=65.31, stdev=12.33 clat percentiles (msec): | 1.00th=[ 42], 5.00th=[ 47], 10.00th=[ 51], 20.00th=[ 55], | 30.00th=[ 58], 40.00th=[ 62], 50.00th=[ 65], 60.00th=[ 68], | 70.00th=[ 71], 80.00th=[ 75], 90.00th=[ 82], 95.00th=[ 87], | 99.00th=[ 95], 99.50th=[ 110], 99.90th=[ 123], 99.95th=[ 123], | 99.99th=[ 125] bw ( KiB/s): min= 830, max= 1144, per=24.70%, avg=961.05, stdev=96.90, samples=20 iops : min= 207, max= 286, avg=240.15, stdev=24.29, samples=20 lat (msec) : 20=0.16%, 50=9.88%, 100=89.22%, 250=0.74% cpu : usr=0.42%, sys=3.52%, ctx=2768, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=190323: Tue Aug 18 21:44:52 2026 write: IOPS=243, BW=975KiB/s (998kB/s)(10.0MiB/10506msec); 0 zone resets slat (usec): min=50, max=1967, avg=200.59, stdev=184.92 clat (msec): min=36, max=125, avg=65.27, stdev=12.34 lat (msec): min=36, max=126, avg=65.47, stdev=12.35 clat percentiles (msec): | 1.00th=[ 41], 5.00th=[ 48], 10.00th=[ 51], 20.00th=[ 55], | 30.00th=[ 59], 40.00th=[ 62], 50.00th=[ 65], 60.00th=[ 68], | 70.00th=[ 71], 80.00th=[ 75], 90.00th=[ 81], 95.00th=[ 87], | 99.00th=[ 97], 99.50th=[ 114], 99.90th=[ 125], 99.95th=[ 126], | 99.99th=[ 126] bw ( KiB/s): min= 798, max= 1144, per=24.85%, avg=966.75, stdev=106.54, samples=20 iops : min= 199, max= 286, avg=241.60, stdev=26.72, samples=20 lat (msec) : 50=9.45%, 100=89.69%, 250=0.86% cpu : usr=0.25%, sys=3.29%, ctx=2711, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=190324: Tue Aug 18 21:44:52 2026 write: IOPS=243, BW=973KiB/s (996kB/s)(10.0MiB/10526msec); 0 zone resets slat (usec): min=40, max=31427, avg=230.75, stdev=656.13 clat (msec): min=5, max=123, avg=65.29, stdev=12.36 lat (msec): min=5, max=124, avg=65.52, stdev=12.34 clat percentiles (msec): | 1.00th=[ 41], 5.00th=[ 48], 10.00th=[ 51], 20.00th=[ 55], | 30.00th=[ 59], 40.00th=[ 63], 50.00th=[ 65], 60.00th=[ 68], | 70.00th=[ 71], 80.00th=[ 75], 90.00th=[ 81], 95.00th=[ 87], | 99.00th=[ 99], 99.50th=[ 109], 99.90th=[ 124], 99.95th=[ 124], | 99.99th=[ 124] bw ( KiB/s): min= 814, max= 1152, per=24.76%, avg=963.30, stdev=105.53, samples=20 iops : min= 203, max= 288, avg=240.75, stdev=26.48, samples=20 lat (msec) : 10=0.04%, 20=0.04%, 50=9.18%, 100=90.16%, 250=0.59% cpu : usr=0.36%, sys=3.67%, ctx=2746, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=3891KiB/s (3985kB/s), 973KiB/s-975KiB/s (996kB/s-999kB/s), io=40.0MiB (41.9MB), run=10498-10526msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=190335: Tue Aug 18 21:45:04 2026 read: IOPS=152, BW=608KiB/s (623kB/s)(5048KiB/8298msec) slat (usec): min=1097, max=12729, avg=3419.49, stdev=1220.64 clat (usec): min=25202, max=85944, avg=49340.77, stdev=10362.79 lat (usec): min=27585, max=91109, avg=52763.67, stdev=10771.81 clat percentiles (usec): | 1.00th=[29754], 5.00th=[33817], 10.00th=[36439], 20.00th=[40633], | 30.00th=[43254], 40.00th=[45876], 50.00th=[47973], 60.00th=[51119], | 70.00th=[53740], 80.00th=[57934], 90.00th=[63177], 95.00th=[68682], | 99.00th=[76022], 99.50th=[79168], 99.90th=[83362], 99.95th=[85459], | 99.99th=[85459] bw ( KiB/s): min= 432, max= 808, per=25.11%, avg=607.25, stdev=80.09, samples=16 iops : min= 108, max= 202, avg=151.69, stdev=20.01, samples=16 write: IOPS=156, BW=626KiB/s (641kB/s)(5192KiB/8298msec); 0 zone resets slat (usec): min=46, max=2250, avg=190.51, stdev=152.10 clat (usec): min=12180, max=89950, avg=50096.53, stdev=9778.21 lat (usec): min=12358, max=90114, avg=50288.26, stdev=9783.07 clat percentiles (usec): | 1.00th=[31851], 5.00th=[35390], 10.00th=[38536], 20.00th=[42206], | 30.00th=[44303], 40.00th=[46924], 50.00th=[49021], 60.00th=[51643], | 70.00th=[54264], 80.00th=[58459], 90.00th=[63177], 95.00th=[68682], | 99.00th=[74974], 99.50th=[79168], 99.90th=[88605], 99.95th=[89654], | 99.99th=[89654] bw ( KiB/s): min= 448, max= 832, per=24.88%, avg=626.31, stdev=106.24, samples=16 iops : min= 112, max= 208, avg=156.44, stdev=26.62, samples=16 lat (msec) : 20=0.04%, 50=55.70%, 100=44.26% cpu : usr=0.57%, sys=4.75%, ctx=4408, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=190336: Tue Aug 18 21:45:04 2026 read: IOPS=150, BW=602KiB/s (616kB/s)(4976KiB/8266msec) slat (usec): min=1199, max=9776, avg=3418.03, stdev=1228.09 clat (usec): min=11029, max=91332, avg=49029.57, stdev=10363.10 lat (usec): min=15641, max=98897, avg=52451.47, stdev=10745.40 clat percentiles (usec): | 1.00th=[28967], 5.00th=[33817], 10.00th=[36439], 20.00th=[40109], | 30.00th=[42730], 40.00th=[45351], 50.00th=[47973], 60.00th=[51119], | 70.00th=[53216], 80.00th=[56886], 90.00th=[63177], 95.00th=[67634], | 99.00th=[76022], 99.50th=[79168], 99.90th=[88605], 99.95th=[91751], | 99.99th=[91751] bw ( KiB/s): min= 472, max= 744, per=24.81%, avg=599.87, stdev=80.70, samples=16 iops : min= 118, max= 186, avg=149.88, stdev=20.16, samples=16 write: IOPS=159, BW=637KiB/s (652kB/s)(5264KiB/8266msec); 0 zone resets slat (usec): min=43, max=1876, avg=182.60, stdev=133.10 clat (usec): min=12112, max=87084, avg=50209.82, stdev=9846.77 lat (usec): min=12216, max=87247, avg=50393.55, stdev=9857.67 clat percentiles (usec): | 1.00th=[30278], 5.00th=[35390], 10.00th=[38011], 20.00th=[42730], | 30.00th=[44827], 40.00th=[47449], 50.00th=[49546], 60.00th=[51643], | 70.00th=[54789], 80.00th=[57934], 90.00th=[63177], 95.00th=[67634], | 99.00th=[77071], 99.50th=[81265], 99.90th=[85459], 99.95th=[87557], | 99.99th=[87557] bw ( KiB/s): min= 440, max= 880, per=25.34%, avg=637.69, stdev=98.81, samples=16 iops : min= 110, max= 220, avg=159.31, stdev=24.75, samples=16 lat (msec) : 20=0.12%, 50=54.22%, 100=45.66% cpu : usr=0.48%, sys=4.86%, ctx=4413, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=190337: Tue Aug 18 21:45:04 2026 read: IOPS=150, BW=601KiB/s (615kB/s)(4984KiB/8295msec) slat (usec): min=1127, max=9710, avg=3430.32, stdev=1233.34 clat (usec): min=8119, max=94083, avg=49217.54, stdev=10744.27 lat (usec): min=9627, max=98070, avg=52650.80, stdev=11126.11 clat percentiles (usec): | 1.00th=[28181], 5.00th=[33817], 10.00th=[36963], 20.00th=[40109], | 30.00th=[42730], 40.00th=[45351], 50.00th=[47973], 60.00th=[50594], | 70.00th=[53740], 80.00th=[57934], 90.00th=[63701], 95.00th=[68682], | 99.00th=[79168], 99.50th=[81265], 99.90th=[90702], 99.95th=[93848], | 99.99th=[93848] bw ( KiB/s): min= 432, max= 872, per=24.91%, avg=602.44, stdev=105.90, samples=16 iops : min= 108, max= 218, avg=150.50, stdev=26.52, samples=16 write: IOPS=158, BW=634KiB/s (649kB/s)(5256KiB/8295msec); 0 zone resets slat (usec): min=55, max=1122, avg=190.02, stdev=118.60 clat (usec): min=24286, max=93487, avg=50319.44, stdev=9894.26 lat (usec): min=24402, max=93604, avg=50510.72, stdev=9899.58 clat percentiles (usec): | 1.00th=[30016], 5.00th=[35914], 10.00th=[39060], 20.00th=[41681], | 30.00th=[44827], 40.00th=[46924], 50.00th=[49021], 60.00th=[51643], | 70.00th=[54789], 80.00th=[58983], 90.00th=[64750], 95.00th=[67634], | 99.00th=[73925], 99.50th=[79168], 99.90th=[92799], 99.95th=[93848], | 99.99th=[93848] bw ( KiB/s): min= 496, max= 760, per=25.08%, avg=631.25, stdev=82.24, samples=16 iops : min= 124, max= 190, avg=157.69, stdev=20.54, samples=16 lat (msec) : 10=0.04%, 50=55.43%, 100=44.53% cpu : usr=0.46%, sys=4.85%, ctx=4450, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=190338: Tue Aug 18 21:45:04 2026 read: IOPS=154, BW=617KiB/s (631kB/s)(5060KiB/8207msec) slat (usec): min=81, max=9664, avg=3410.78, stdev=1244.91 clat (usec): min=5445, max=86044, avg=48855.01, stdev=10949.51 lat (usec): min=5630, max=92129, avg=52268.89, stdev=11391.94 clat percentiles (usec): | 1.00th=[26346], 5.00th=[32637], 10.00th=[35914], 20.00th=[40109], | 30.00th=[42730], 40.00th=[45351], 50.00th=[47449], 60.00th=[50594], | 70.00th=[53740], 80.00th=[57410], 90.00th=[64226], 95.00th=[67634], | 99.00th=[76022], 99.50th=[80217], 99.90th=[84411], 99.95th=[86508], | 99.99th=[86508] bw ( KiB/s): min= 448, max= 824, per=25.50%, avg=616.56, stdev=82.05, samples=16 iops : min= 112, max= 206, avg=154.00, stdev=20.52, samples=16 write: IOPS=157, BW=631KiB/s (646kB/s)(5180KiB/8207msec); 0 zone resets slat (usec): min=55, max=1379, avg=188.08, stdev=131.82 clat (msec): min=19, max=106, avg=49.98, stdev=10.07 lat (msec): min=19, max=106, avg=50.17, stdev=10.07 clat percentiles (msec): | 1.00th=[ 31], 5.00th=[ 36], 10.00th=[ 39], 20.00th=[ 42], | 30.00th=[ 45], 40.00th=[ 47], 50.00th=[ 50], 60.00th=[ 52], | 70.00th=[ 55], 80.00th=[ 59], 90.00th=[ 63], 95.00th=[ 67], | 99.00th=[ 75], 99.50th=[ 81], 99.90th=[ 107], 99.95th=[ 107], | 99.99th=[ 107] bw ( KiB/s): min= 440, max= 832, per=24.96%, avg=628.12, stdev=111.92, samples=16 iops : min= 110, max= 208, avg=156.94, stdev=27.99, samples=16 lat (msec) : 10=0.12%, 20=0.20%, 50=55.16%, 100=44.45%, 250=0.08% cpu : usr=0.50%, sys=4.89%, ctx=4431, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2418KiB/s (2476kB/s), 601KiB/s-617KiB/s (615kB/s-631kB/s), io=19.6MiB (20.5MB), run=8207-8298msec WRITE: bw=2518KiB/s (2578kB/s), 626KiB/s-637KiB/s (641kB/s-652kB/s), io=20.4MiB (21.4MB), run=8207-8298msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=190342: Tue Aug 18 21:45:08 2026 read: IOPS=7, BW=310MiB/s (325MB/s)(40.0MiB/129msec) slat (nsec): min=11601k, max=11601k, avg=11600622.00, stdev= 0.00 clat (nsec): min=110127k, max=110127k, avg=110126831.00, stdev= 0.00 lat (nsec): min=121731k, max=121731k, avg=121730949.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 111], 5.00th=[ 111], 10.00th=[ 111], 20.00th=[ 111], | 30.00th=[ 111], 40.00th=[ 111], 50.00th=[ 111], 60.00th=[ 111], | 70.00th=[ 111], 80.00th=[ 111], 90.00th=[ 111], 95.00th=[ 111], | 99.00th=[ 111], 99.50th=[ 111], 99.90th=[ 111], 99.95th=[ 111], | 99.99th=[ 111] lat (msec) : 250=100.00% cpu : usr=0.00%, sys=8.59%, ctx=3, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=310MiB/s (325MB/s), 310MiB/s-310MiB/s (325MB/s-325MB/s), io=40.0MiB (41.9MB), run=129-129msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout