== sanity test 398c: run fio to test AIO ================= 06:32:16 (1786012336) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.576349 s, 72.8 MB/s osc.lustre-OST0000-osc-ffffa0d938ea1000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=178982: Thu Aug 6 06:33:11 2026 write: IOPS=58, BW=232KiB/s (238kB/s)(10.0MiB/44109msec); 0 zone resets slat (usec): min=46, max=20262, avg=233.45, stdev=800.99 clat (msec): min=73, max=868, avg=274.31, stdev=89.05 lat (msec): min=73, max=868, avg=274.55, stdev=89.10 clat percentiles (msec): | 1.00th=[ 105], 5.00th=[ 155], 10.00th=[ 176], 20.00th=[ 207], | 30.00th=[ 230], 40.00th=[ 251], 50.00th=[ 266], 60.00th=[ 284], | 70.00th=[ 309], 80.00th=[ 330], 90.00th=[ 376], 95.00th=[ 422], | 99.00th=[ 567], 99.50th=[ 634], 99.90th=[ 869], 99.95th=[ 869], | 99.99th=[ 869] bw ( KiB/s): min= 72, max= 507, per=24.85%, avg=230.61, stdev=73.19, samples=87 iops : min= 18, max= 126, avg=57.21, stdev=18.33, samples=87 lat (msec) : 100=0.66%, 250=39.02%, 500=58.01%, 750=2.11%, 1000=0.20% cpu : usr=0.10%, sys=0.63%, ctx=1681, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=178983: Thu Aug 6 06:33:11 2026 write: IOPS=58, BW=232KiB/s (238kB/s)(10.0MiB/44126msec); 0 zone resets slat (usec): min=48, max=28188, avg=293.20, stdev=1177.34 clat (msec): min=55, max=870, avg=274.50, stdev=91.22 lat (msec): min=56, max=870, avg=274.79, stdev=91.29 clat percentiles (msec): | 1.00th=[ 112], 5.00th=[ 148], 10.00th=[ 178], 20.00th=[ 207], | 30.00th=[ 226], 40.00th=[ 249], 50.00th=[ 271], 60.00th=[ 288], | 70.00th=[ 300], 80.00th=[ 334], 90.00th=[ 376], 95.00th=[ 430], | 99.00th=[ 567], 99.50th=[ 743], 99.90th=[ 768], 99.95th=[ 869], | 99.99th=[ 869] bw ( KiB/s): min= 31, max= 507, per=24.85%, avg=230.56, stdev=74.14, samples=87 iops : min= 7, max= 126, avg=57.18, stdev=18.53, samples=87 lat (msec) : 100=0.70%, 250=39.49%, 500=57.11%, 750=2.27%, 1000=0.43% cpu : usr=0.10%, sys=0.75%, ctx=2581, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=178984: Thu Aug 6 06:33:11 2026 write: IOPS=58, BW=234KiB/s (239kB/s)(10.0MiB/43853msec); 0 zone resets slat (usec): min=47, max=33091, avg=324.87, stdev=1293.12 clat (msec): min=15, max=780, avg=273.52, stdev=92.84 lat (msec): min=15, max=780, avg=273.84, stdev=92.87 clat percentiles (msec): | 1.00th=[ 94], 5.00th=[ 148], 10.00th=[ 174], 20.00th=[ 203], | 30.00th=[ 226], 40.00th=[ 251], 50.00th=[ 268], 60.00th=[ 284], | 70.00th=[ 300], 80.00th=[ 326], 90.00th=[ 380], 95.00th=[ 435], | 99.00th=[ 600], 99.50th=[ 693], 99.90th=[ 776], 99.95th=[ 785], | 99.99th=[ 785] bw ( KiB/s): min= 87, max= 444, per=25.18%, avg=233.71, stdev=72.38, samples=86 iops : min= 21, max= 111, avg=57.99, stdev=18.16, samples=86 lat (msec) : 20=0.04%, 50=0.23%, 100=1.29%, 250=38.44%, 500=57.58% lat (msec) : 750=2.27%, 1000=0.16% cpu : usr=0.11%, sys=0.78%, ctx=2646, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=178985: Thu Aug 6 06:33:11 2026 write: IOPS=58, BW=232KiB/s (238kB/s)(10.0MiB/44050msec); 0 zone resets slat (usec): min=47, max=22748, avg=299.49, stdev=1188.43 clat (msec): min=43, max=719, avg=274.36, stdev=88.58 lat (msec): min=43, max=719, avg=274.66, stdev=88.62 clat percentiles (msec): | 1.00th=[ 108], 5.00th=[ 148], 10.00th=[ 176], 20.00th=[ 209], | 30.00th=[ 228], 40.00th=[ 247], 50.00th=[ 268], 60.00th=[ 284], | 70.00th=[ 305], 80.00th=[ 334], 90.00th=[ 376], 95.00th=[ 426], | 99.00th=[ 575], 99.50th=[ 625], 99.90th=[ 718], 99.95th=[ 718], | 99.99th=[ 718] bw ( KiB/s): min= 104, max= 500, per=24.94%, avg=231.41, stdev=72.64, samples=87 iops : min= 26, max= 125, avg=57.39, stdev=18.19, samples=87 lat (msec) : 50=0.04%, 100=0.08%, 250=41.76%, 500=56.02%, 750=2.11% cpu : usr=0.09%, sys=0.79%, ctx=2270, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=928KiB/s (951kB/s), 232KiB/s-234KiB/s (238kB/s-239kB/s), io=40.0MiB (41.9MB), run=43853-44126msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=179004: Thu Aug 6 06:34:23 2026 read: IOPS=24, BW=99.8KiB/s (102kB/s)(5048KiB/50574msec) slat (msec): min=2, max=119, avg=21.34, stdev=14.81 clat (msec): min=20, max=625, avg=298.22, stdev=90.33 lat (msec): min=49, max=664, avg=319.57, stdev=94.12 clat percentiles (msec): | 1.00th=[ 136], 5.00th=[ 163], 10.00th=[ 190], 20.00th=[ 224], | 30.00th=[ 245], 40.00th=[ 268], 50.00th=[ 292], 60.00th=[ 309], | 70.00th=[ 334], 80.00th=[ 368], 90.00th=[ 422], 95.00th=[ 472], | 99.00th=[ 542], 99.50th=[ 550], 99.90th=[ 600], 99.95th=[ 625], | 99.99th=[ 625] bw ( KiB/s): min= 31, max= 167, per=25.23%, avg=99.93, stdev=29.54, samples=99 iops : min= 7, max= 41, avg=24.44, stdev= 7.38, samples=99 write: IOPS=25, BW=103KiB/s (105kB/s)(5192KiB/50574msec); 0 zone resets slat (usec): min=53, max=39717, avg=328.04, stdev=1589.49 clat (msec): min=100, max=631, avg=305.89, stdev=87.76 lat (msec): min=101, max=631, avg=306.22, stdev=87.84 clat percentiles (msec): | 1.00th=[ 148], 5.00th=[ 178], 10.00th=[ 199], 20.00th=[ 232], | 30.00th=[ 257], 40.00th=[ 275], 50.00th=[ 292], 60.00th=[ 313], | 70.00th=[ 342], 80.00th=[ 380], 90.00th=[ 430], 95.00th=[ 468], | 99.00th=[ 550], 99.50th=[ 584], 99.90th=[ 625], 99.95th=[ 634], | 99.99th=[ 634] bw ( KiB/s): min= 23, max= 198, per=24.94%, avg=103.00, stdev=35.74, samples=99 iops : min= 5, max= 49, avg=25.23, stdev= 8.93, samples=99 lat (msec) : 50=0.04%, 100=0.04%, 250=29.02%, 500=68.20%, 750=2.70% cpu : usr=0.13%, sys=1.26%, ctx=4080, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=179005: Thu Aug 6 06:34:23 2026 read: IOPS=24, BW=98.6KiB/s (101kB/s)(4976KiB/50484msec) slat (msec): min=3, max=120, avg=22.27, stdev=15.57 clat (msec): min=93, max=642, avg=295.95, stdev=90.03 lat (msec): min=107, max=715, avg=318.22, stdev=95.02 clat percentiles (msec): | 1.00th=[ 136], 5.00th=[ 169], 10.00th=[ 192], 20.00th=[ 220], | 30.00th=[ 245], 40.00th=[ 266], 50.00th=[ 284], 60.00th=[ 309], | 70.00th=[ 330], 80.00th=[ 359], 90.00th=[ 418], 95.00th=[ 477], | 99.00th=[ 567], 99.50th=[ 584], 99.90th=[ 625], 99.95th=[ 642], | 99.99th=[ 642] bw ( KiB/s): min= 7, max= 175, per=24.69%, avg=97.79, stdev=33.60, samples=100 iops : min= 1, max= 43, avg=23.91, stdev= 8.37, samples=100 write: IOPS=26, BW=104KiB/s (107kB/s)(5264KiB/50484msec); 0 zone resets slat (usec): min=51, max=24453, avg=288.01, stdev=1184.27 clat (msec): min=38, max=723, avg=309.32, stdev=89.70 lat (msec): min=39, max=723, avg=309.61, stdev=89.76 clat percentiles (msec): | 1.00th=[ 150], 5.00th=[ 182], 10.00th=[ 205], 20.00th=[ 234], | 30.00th=[ 259], 40.00th=[ 279], 50.00th=[ 300], 60.00th=[ 317], | 70.00th=[ 347], 80.00th=[ 380], 90.00th=[ 426], 95.00th=[ 481], | 99.00th=[ 550], 99.50th=[ 584], 99.90th=[ 709], 99.95th=[ 726], | 99.99th=[ 726] bw ( KiB/s): min= 7, max= 199, per=25.16%, avg=103.90, stdev=34.94, samples=100 iops : min= 1, max= 49, avg=25.45, stdev= 8.76, samples=100 lat (msec) : 50=0.08%, 100=0.08%, 250=28.87%, 500=67.42%, 750=3.55% cpu : usr=0.22%, sys=1.39%, ctx=4284, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=179006: Thu Aug 6 06:34:23 2026 read: IOPS=24, BW=98.6KiB/s (101kB/s)(4984KiB/50564msec) slat (usec): min=65, max=121548, avg=21968.51, stdev=15101.04 clat (msec): min=120, max=658, avg=299.47, stdev=92.48 lat (msec): min=127, max=691, avg=321.44, stdev=96.69 clat percentiles (msec): | 1.00th=[ 140], 5.00th=[ 169], 10.00th=[ 192], 20.00th=[ 222], | 30.00th=[ 245], 40.00th=[ 268], 50.00th=[ 288], 60.00th=[ 309], | 70.00th=[ 334], 80.00th=[ 372], 90.00th=[ 418], 95.00th=[ 477], | 99.00th=[ 575], 99.50th=[ 600], 99.90th=[ 642], 99.95th=[ 659], | 99.99th=[ 659] bw ( KiB/s): min= 31, max= 159, per=24.74%, avg=97.96, stdev=31.55, samples=100 iops : min= 7, max= 39, avg=23.97, stdev= 7.87, samples=100 write: IOPS=25, BW=104KiB/s (106kB/s)(5256KiB/50564msec); 0 zone resets slat (usec): min=51, max=26721, avg=334.03, stdev=1440.39 clat (msec): min=60, max=707, avg=309.98, stdev=92.31 lat (msec): min=60, max=707, avg=310.32, stdev=92.34 clat percentiles (msec): | 1.00th=[ 148], 5.00th=[ 174], 10.00th=[ 201], 20.00th=[ 236], | 30.00th=[ 257], 40.00th=[ 275], 50.00th=[ 296], 60.00th=[ 317], | 70.00th=[ 347], 80.00th=[ 384], 90.00th=[ 435], 95.00th=[ 485], | 99.00th=[ 550], 99.50th=[ 609], 99.90th=[ 701], 99.95th=[ 709], | 99.99th=[ 709] bw ( KiB/s): min= 47, max= 199, per=24.93%, avg=102.98, stdev=35.19, samples=100 iops : min= 11, max= 49, avg=25.23, stdev= 8.80, samples=100 lat (msec) : 100=0.04%, 250=29.10%, 500=67.07%, 750=3.79% cpu : usr=0.17%, sys=1.40%, ctx=4248, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=179007: Thu Aug 6 06:34:23 2026 read: IOPS=25, BW=100KiB/s (103kB/s)(5060KiB/50489msec) slat (msec): min=2, max=116, avg=21.62, stdev=15.15 clat (msec): min=37, max=587, avg=295.91, stdev=92.09 lat (msec): min=44, max=669, avg=317.53, stdev=96.91 clat percentiles (msec): | 1.00th=[ 136], 5.00th=[ 161], 10.00th=[ 186], 20.00th=[ 220], | 30.00th=[ 241], 40.00th=[ 262], 50.00th=[ 284], 60.00th=[ 309], | 70.00th=[ 330], 80.00th=[ 372], 90.00th=[ 430], 95.00th=[ 477], | 99.00th=[ 542], 99.50th=[ 550], 99.90th=[ 584], 99.95th=[ 592], | 99.99th=[ 592] bw ( KiB/s): min= 39, max= 199, per=25.41%, avg=100.62, stdev=33.01, samples=99 iops : min= 9, max= 49, avg=24.63, stdev= 8.27, samples=99 write: IOPS=25, BW=103KiB/s (105kB/s)(5180KiB/50489msec); 0 zone resets slat (usec): min=51, max=29598, avg=394.53, stdev=1605.60 clat (msec): min=130, max=721, avg=308.50, stdev=86.41 lat (msec): min=131, max=721, avg=308.90, stdev=86.48 clat percentiles (msec): | 1.00th=[ 150], 5.00th=[ 180], 10.00th=[ 199], 20.00th=[ 236], | 30.00th=[ 259], 40.00th=[ 279], 50.00th=[ 300], 60.00th=[ 321], | 70.00th=[ 347], 80.00th=[ 380], 90.00th=[ 426], 95.00th=[ 460], | 99.00th=[ 542], 99.50th=[ 575], 99.90th=[ 625], 99.95th=[ 718], | 99.99th=[ 718] bw ( KiB/s): min= 40, max= 191, per=24.95%, avg=103.06, stdev=33.59, samples=99 iops : min= 10, max= 47, avg=25.25, stdev= 8.42, samples=99 lat (msec) : 50=0.04%, 250=29.61%, 500=67.42%, 750=2.93% cpu : usr=0.16%, sys=1.45%, ctx=4033, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=397KiB/s (406kB/s), 98.6KiB/s-100KiB/s (101kB/s-103kB/s), io=19.6MiB (20.5MB), run=50484-50574msec WRITE: bw=413KiB/s (423kB/s), 103KiB/s-104KiB/s (105kB/s-107kB/s), io=20.4MiB (21.4MB), run=50484-50574msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=179019: Thu Aug 6 06:34:35 2026 read: IOPS=2, BW=83.5MiB/s (87.6MB/s)(40.0MiB/479msec) slat (nsec): min=30290k, max=30290k, avg=30289834.00, stdev= 0.00 clat (nsec): min=421378k, max=421378k, avg=421377725.00, stdev= 0.00 lat (nsec): min=451671k, max=451671k, avg=451671465.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 422], 5.00th=[ 422], 10.00th=[ 422], 20.00th=[ 422], | 30.00th=[ 422], 40.00th=[ 422], 50.00th=[ 422], 60.00th=[ 422], | 70.00th=[ 422], 80.00th=[ 422], 90.00th=[ 422], 95.00th=[ 422], | 99.00th=[ 422], 99.50th=[ 422], 99.90th=[ 422], 99.95th=[ 422], | 99.99th=[ 422] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=1.88%, ctx=4, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=83.5MiB/s (87.6MB/s), 83.5MiB/s-83.5MiB/s (87.6MB/s-87.6MB/s), io=40.0MiB (41.9MB), run=479-479msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout