== sanity test 398c: run fio to test AIO ================= 08:37:25 (1790685445) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.895967 s, 46.8 MB/s osc.lustre-OST0000-osc-ffff8bb7cdc94000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=182083: Tue Sep 29 08:38:09 2026 write: IOPS=81, BW=325KiB/s (333kB/s)(10.0MiB/31503msec); 0 zone resets slat (usec): min=22, max=17510, avg=264.70, stdev=910.98 clat (msec): min=22, max=344, avg=195.95, stdev=47.61 lat (msec): min=25, max=345, avg=196.22, stdev=47.63 clat percentiles (msec): | 1.00th=[ 104], 5.00th=[ 129], 10.00th=[ 140], 20.00th=[ 153], | 30.00th=[ 169], 40.00th=[ 178], 50.00th=[ 190], 60.00th=[ 203], | 70.00th=[ 218], 80.00th=[ 239], 90.00th=[ 264], 95.00th=[ 284], | 99.00th=[ 313], 99.50th=[ 317], 99.90th=[ 330], 99.95th=[ 347], | 99.99th=[ 347] bw ( KiB/s): min= 214, max= 495, per=24.99%, avg=324.66, stdev=61.74, samples=62 iops : min= 53, max= 123, avg=80.82, stdev=15.49, samples=62 lat (msec) : 50=0.08%, 100=0.62%, 250=85.20%, 500=14.10% cpu : usr=0.16%, sys=0.97%, ctx=2467, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=182084: Tue Sep 29 08:38:09 2026 write: IOPS=81, BW=327KiB/s (334kB/s)(10.0MiB/31353msec); 0 zone resets slat (usec): min=25, max=16935, avg=239.73, stdev=734.46 clat (msec): min=15, max=396, avg=195.37, stdev=49.05 lat (msec): min=15, max=396, avg=195.62, stdev=49.06 clat percentiles (msec): | 1.00th=[ 97], 5.00th=[ 128], 10.00th=[ 138], 20.00th=[ 155], | 30.00th=[ 165], 40.00th=[ 178], 50.00th=[ 192], 60.00th=[ 205], | 70.00th=[ 220], 80.00th=[ 239], 90.00th=[ 262], 95.00th=[ 279], | 99.00th=[ 317], 99.50th=[ 347], 99.90th=[ 397], 99.95th=[ 397], | 99.99th=[ 397] bw ( KiB/s): min= 159, max= 488, per=25.14%, avg=326.61, stdev=67.78, samples=62 iops : min= 39, max= 122, avg=81.34, stdev=16.96, samples=62 lat (msec) : 20=0.08%, 50=0.12%, 100=1.05%, 250=84.65%, 500=14.10% cpu : usr=0.18%, sys=0.95%, ctx=2671, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=182085: Tue Sep 29 08:38:09 2026 write: IOPS=81, BW=325KiB/s (333kB/s)(10.0MiB/31486msec); 0 zone resets slat (usec): min=26, max=14168, avg=250.76, stdev=832.58 clat (msec): min=33, max=341, avg=195.72, stdev=47.68 lat (msec): min=33, max=341, avg=195.97, stdev=47.66 clat percentiles (msec): | 1.00th=[ 104], 5.00th=[ 128], 10.00th=[ 136], 20.00th=[ 153], | 30.00th=[ 167], 40.00th=[ 178], 50.00th=[ 194], 60.00th=[ 205], | 70.00th=[ 220], 80.00th=[ 241], 90.00th=[ 264], 95.00th=[ 279], | 99.00th=[ 313], 99.50th=[ 317], 99.90th=[ 334], 99.95th=[ 342], | 99.99th=[ 342] bw ( KiB/s): min= 172, max= 486, per=25.00%, avg=324.74, stdev=73.98, samples=62 iops : min= 43, max= 121, avg=80.79, stdev=18.48, samples=62 lat (msec) : 50=0.16%, 100=0.74%, 250=85.04%, 500=14.06% cpu : usr=0.17%, sys=0.91%, ctx=2160, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=182086: Tue Sep 29 08:38:09 2026 write: IOPS=81, BW=325KiB/s (333kB/s)(10.0MiB/31526msec); 0 zone resets slat (usec): min=23, max=22347, avg=199.93, stdev=798.16 clat (msec): min=16, max=331, avg=196.15, stdev=47.28 lat (msec): min=16, max=331, avg=196.35, stdev=47.28 clat percentiles (msec): | 1.00th=[ 100], 5.00th=[ 129], 10.00th=[ 142], 20.00th=[ 155], | 30.00th=[ 167], 40.00th=[ 178], 50.00th=[ 190], 60.00th=[ 203], | 70.00th=[ 222], 80.00th=[ 241], 90.00th=[ 264], 95.00th=[ 279], | 99.00th=[ 309], 99.50th=[ 313], 99.90th=[ 317], 99.95th=[ 317], | 99.99th=[ 330] bw ( KiB/s): min= 183, max= 512, per=24.92%, avg=323.68, stdev=62.30, samples=62 iops : min= 45, max= 128, avg=80.56, stdev=15.62, samples=62 lat (msec) : 20=0.04%, 100=0.98%, 250=83.98%, 500=15.00% cpu : usr=0.16%, sys=0.86%, ctx=1776, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1299KiB/s (1330kB/s), 325KiB/s-327KiB/s (333kB/s-334kB/s), io=40.0MiB (41.9MB), run=31353-31526msec 0% of I/O are 1-page (PAGE_SIZE=4096), max 50% mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=182102: Tue Sep 29 08:39:04 2026 read: IOPS=28, BW=116KiB/s (119kB/s)(5048KiB/43541msec) slat (usec): min=207, max=103099, avg=19404.80, stdev=11834.54 clat (msec): min=17, max=519, avg=254.40, stdev=70.32 lat (msec): min=28, max=553, avg=273.81, stdev=73.31 clat percentiles (msec): | 1.00th=[ 106], 5.00th=[ 157], 10.00th=[ 176], 20.00th=[ 197], | 30.00th=[ 215], 40.00th=[ 232], 50.00th=[ 247], 60.00th=[ 264], | 70.00th=[ 284], 80.00th=[ 313], 90.00th=[ 347], 95.00th=[ 376], | 99.00th=[ 468], 99.50th=[ 477], 99.90th=[ 498], 99.95th=[ 518], | 99.99th=[ 518] bw ( KiB/s): min= 47, max= 190, per=24.86%, avg=114.36, stdev=29.07, samples=85 iops : min= 11, max= 47, avg=28.06, stdev= 7.29, samples=85 write: IOPS=29, BW=119KiB/s (122kB/s)(5192KiB/43541msec); 0 zone resets slat (usec): min=25, max=15363, avg=287.82, stdev=1007.20 clat (msec): min=40, max=511, avg=266.51, stdev=69.22 lat (msec): min=40, max=511, avg=266.80, stdev=69.22 clat percentiles (msec): | 1.00th=[ 133], 5.00th=[ 169], 10.00th=[ 186], 20.00th=[ 209], | 30.00th=[ 224], 40.00th=[ 241], 50.00th=[ 259], 60.00th=[ 275], | 70.00th=[ 296], 80.00th=[ 321], 90.00th=[ 363], 95.00th=[ 393], | 99.00th=[ 456], 99.50th=[ 477], 99.90th=[ 510], 99.95th=[ 510], | 99.99th=[ 510] bw ( KiB/s): min= 48, max= 214, per=24.69%, avg=118.26, stdev=34.79, samples=85 iops : min= 12, max= 53, avg=29.04, stdev= 8.72, samples=85 lat (msec) : 20=0.04%, 50=0.20%, 100=0.43%, 250=47.46%, 500=51.76% lat (msec) : 750=0.12% cpu : usr=0.20%, sys=1.25%, ctx=3701, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=182103: Tue Sep 29 08:39:04 2026 read: IOPS=28, BW=115KiB/s (117kB/s)(4976KiB/43399msec) slat (usec): min=3584, max=74412, avg=19476.45, stdev=11408.55 clat (msec): min=96, max=493, avg=256.78, stdev=66.28 lat (msec): min=108, max=536, avg=276.27, stdev=68.93 clat percentiles (msec): | 1.00th=[ 127], 5.00th=[ 157], 10.00th=[ 178], 20.00th=[ 203], | 30.00th=[ 220], 40.00th=[ 234], 50.00th=[ 249], 60.00th=[ 266], | 70.00th=[ 284], 80.00th=[ 309], 90.00th=[ 347], 95.00th=[ 380], | 99.00th=[ 439], 99.50th=[ 451], 99.90th=[ 468], 99.95th=[ 493], | 99.99th=[ 493] bw ( KiB/s): min= 46, max= 184, per=24.81%, avg=114.12, stdev=30.38, samples=85 iops : min= 11, max= 46, avg=27.99, stdev= 7.64, samples=85 write: IOPS=30, BW=121KiB/s (124kB/s)(5264KiB/43399msec); 0 zone resets slat (usec): min=30, max=29201, avg=231.83, stdev=1082.28 clat (msec): min=49, max=480, avg=263.89, stdev=68.61 lat (msec): min=49, max=480, avg=264.13, stdev=68.57 clat percentiles (msec): | 1.00th=[ 136], 5.00th=[ 163], 10.00th=[ 186], 20.00th=[ 209], | 30.00th=[ 226], 40.00th=[ 241], 50.00th=[ 255], 60.00th=[ 271], | 70.00th=[ 292], 80.00th=[ 321], 90.00th=[ 359], 95.00th=[ 397], | 99.00th=[ 451], 99.50th=[ 464], 99.90th=[ 481], 99.95th=[ 481], | 99.99th=[ 481] bw ( KiB/s): min= 55, max= 229, per=25.22%, avg=120.81, stdev=35.41, samples=85 iops : min= 13, max= 57, avg=29.67, stdev= 8.87, samples=85 lat (msec) : 50=0.04%, 100=0.08%, 250=48.40%, 500=51.48% cpu : usr=0.22%, sys=1.29%, ctx=3959, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=182104: Tue Sep 29 08:39:04 2026 read: IOPS=28, BW=115KiB/s (117kB/s)(4984KiB/43451msec) slat (usec): min=2764, max=91366, avg=19448.71, stdev=11881.30 clat (msec): min=74, max=510, avg=255.39, stdev=67.43 lat (msec): min=87, max=531, avg=274.85, stdev=69.92 clat percentiles (msec): | 1.00th=[ 121], 5.00th=[ 157], 10.00th=[ 174], 20.00th=[ 201], | 30.00th=[ 220], 40.00th=[ 236], 50.00th=[ 251], 60.00th=[ 262], | 70.00th=[ 279], 80.00th=[ 313], 90.00th=[ 347], 95.00th=[ 380], | 99.00th=[ 430], 99.50th=[ 439], 99.90th=[ 481], 99.95th=[ 510], | 99.99th=[ 510] bw ( KiB/s): min= 40, max= 214, per=24.80%, avg=114.08, stdev=34.29, samples=85 iops : min= 10, max= 53, avg=27.99, stdev= 8.57, samples=85 write: IOPS=30, BW=121KiB/s (124kB/s)(5256KiB/43451msec); 0 zone resets slat (usec): min=30, max=10883, avg=244.84, stdev=796.03 clat (msec): min=85, max=497, avg=264.85, stdev=65.82 lat (msec): min=85, max=497, avg=265.10, stdev=65.80 clat percentiles (msec): | 1.00th=[ 150], 5.00th=[ 169], 10.00th=[ 186], 20.00th=[ 207], | 30.00th=[ 226], 40.00th=[ 243], 50.00th=[ 259], 60.00th=[ 275], | 70.00th=[ 292], 80.00th=[ 317], 90.00th=[ 351], 95.00th=[ 393], | 99.00th=[ 447], 99.50th=[ 456], 99.90th=[ 472], 99.95th=[ 498], | 99.99th=[ 498] bw ( KiB/s): min= 47, max= 205, per=25.06%, avg=120.06, stdev=34.46, samples=85 iops : min= 11, max= 51, avg=29.49, stdev= 8.69, samples=85 lat (msec) : 100=0.23%, 250=46.68%, 500=53.05%, 750=0.04% cpu : usr=0.20%, sys=1.46%, ctx=3733, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=182105: Tue Sep 29 08:39:04 2026 read: IOPS=29, BW=117KiB/s (120kB/s)(5060KiB/43333msec) slat (usec): min=53, max=154141, avg=19183.75, stdev=12258.90 clat (msec): min=24, max=488, avg=256.27, stdev=69.86 lat (msec): min=33, max=506, avg=275.47, stdev=72.34 clat percentiles (msec): | 1.00th=[ 106], 5.00th=[ 157], 10.00th=[ 178], 20.00th=[ 201], | 30.00th=[ 220], 40.00th=[ 234], 50.00th=[ 247], 60.00th=[ 264], | 70.00th=[ 284], 80.00th=[ 313], 90.00th=[ 351], 95.00th=[ 384], | 99.00th=[ 447], 99.50th=[ 451], 99.90th=[ 489], 99.95th=[ 489], | 99.99th=[ 489] bw ( KiB/s): min= 39, max= 176, per=25.29%, avg=116.35, stdev=29.39, samples=85 iops : min= 9, max= 44, avg=28.54, stdev= 7.36, samples=85 write: IOPS=29, BW=120KiB/s (122kB/s)(5180KiB/43333msec); 0 zone resets slat (usec): min=32, max=16693, avg=279.04, stdev=1146.35 clat (msec): min=70, max=473, avg=265.35, stdev=68.40 lat (msec): min=70, max=473, avg=265.64, stdev=68.37 clat percentiles (msec): | 1.00th=[ 138], 5.00th=[ 165], 10.00th=[ 182], 20.00th=[ 209], | 30.00th=[ 228], 40.00th=[ 241], 50.00th=[ 255], 60.00th=[ 271], | 70.00th=[ 296], 80.00th=[ 321], 90.00th=[ 359], 95.00th=[ 397], | 99.00th=[ 451], 99.50th=[ 464], 99.90th=[ 472], 99.95th=[ 472], | 99.99th=[ 472] bw ( KiB/s): min= 40, max= 230, per=24.72%, avg=118.41, stdev=34.86, samples=85 iops : min= 10, max= 57, avg=29.07, stdev= 8.73, samples=85 lat (msec) : 50=0.16%, 100=0.23%, 250=48.91%, 500=50.70% cpu : usr=0.21%, sys=1.32%, ctx=3947, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=461KiB/s (472kB/s), 115KiB/s-117KiB/s (117kB/s-120kB/s), io=19.6MiB (20.5MB), run=43333-43541msec WRITE: bw=480KiB/s (491kB/s), 119KiB/s-121KiB/s (122kB/s-124kB/s), io=20.4MiB (21.4MB), run=43333-43541msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=182115: Tue Sep 29 08:39:16 2026 read: IOPS=1, BW=69.9MiB/s (73.3MB/s)(40.0MiB/572msec) slat (nsec): min=30496k, max=30496k, avg=30496190.00, stdev= 0.00 clat (nsec): min=510476k, max=510476k, avg=510476025.00, stdev= 0.00 lat (nsec): min=547812k, max=547812k, avg=547812263.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 510], 5.00th=[ 510], 10.00th=[ 510], 20.00th=[ 510], | 30.00th=[ 510], 40.00th=[ 510], 50.00th=[ 510], 60.00th=[ 510], | 70.00th=[ 510], 80.00th=[ 510], 90.00th=[ 510], 95.00th=[ 510], | 99.00th=[ 510], 99.50th=[ 510], 99.90th=[ 510], 99.95th=[ 510], | 99.99th=[ 510] lat (msec) : 750=100.00% cpu : usr=0.00%, sys=2.10%, ctx=11, majf=0, minf=7 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=69.9MiB/s (73.3MB/s), 69.9MiB/s-69.9MiB/s (73.3MB/s-73.3MB/s), io=40.0MiB (41.9MB), run=572-572msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout