== sanity test 398c: run fio to test AIO ================= 11:43:18 (1787845398) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.437058 s, 96.0 MB/s osc.lustre-OST0000-osc-ffff90a74bbc6000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=145169: Thu Aug 27 11:46:00 2026 write: IOPS=16, BW=67.1KiB/s (68.7kB/s)(10.0MiB/152534msec); 0 zone resets slat (usec): min=21, max=20874, avg=226.25, stdev=1027.29 clat (msec): min=298, max=1891, avg=952.04, stdev=277.20 lat (msec): min=298, max=1891, avg=952.27, stdev=277.21 clat percentiles (msec): | 1.00th=[ 388], 5.00th=[ 518], 10.00th=[ 609], 20.00th=[ 718], | 30.00th=[ 802], 40.00th=[ 869], 50.00th=[ 936], 60.00th=[ 1011], | 70.00th=[ 1083], 80.00th=[ 1167], 90.00th=[ 1318], 95.00th=[ 1469], | 99.00th=[ 1653], 99.50th=[ 1720], 99.90th=[ 1770], 99.95th=[ 1888], | 99.99th=[ 1888] bw ( KiB/s): min= 7, max= 167, per=25.23%, avg=67.62, stdev=31.64, samples=298 iops : min= 1, max= 41, avg=16.46, stdev= 7.94, samples=298 lat (msec) : 500=4.49%, 750=17.85%, 1000=36.33%, 2000=41.33% cpu : usr=0.04%, sys=0.23%, ctx=1658, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145170: Thu Aug 27 11:46:00 2026 write: IOPS=16, BW=67.1KiB/s (68.7kB/s)(10.0MiB/152716msec); 0 zone resets slat (usec): min=22, max=21309, avg=193.57, stdev=822.92 clat (msec): min=333, max=1816, avg=953.34, stdev=276.10 lat (msec): min=333, max=1816, avg=953.53, stdev=276.11 clat percentiles (msec): | 1.00th=[ 376], 5.00th=[ 510], 10.00th=[ 617], 20.00th=[ 709], | 30.00th=[ 810], 40.00th=[ 877], 50.00th=[ 936], 60.00th=[ 1011], | 70.00th=[ 1083], 80.00th=[ 1167], 90.00th=[ 1318], 95.00th=[ 1469], | 99.00th=[ 1670], 99.50th=[ 1720], 99.90th=[ 1787], 99.95th=[ 1821], | 99.99th=[ 1821] bw ( KiB/s): min= 7, max= 175, per=25.23%, avg=67.62, stdev=32.18, samples=298 iops : min= 1, max= 43, avg=16.46, stdev= 8.08, samples=298 lat (msec) : 500=4.53%, 750=19.65%, 1000=35.04%, 2000=40.78% cpu : usr=0.05%, sys=0.20%, ctx=1666, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145171: Thu Aug 27 11:46:00 2026 write: IOPS=16, BW=67.1KiB/s (68.7kB/s)(10.0MiB/152643msec); 0 zone resets slat (usec): min=22, max=12569, avg=200.17, stdev=761.84 clat (msec): min=130, max=1890, avg=953.38, stdev=283.82 lat (msec): min=137, max=1890, avg=953.58, stdev=283.80 clat percentiles (msec): | 1.00th=[ 338], 5.00th=[ 485], 10.00th=[ 609], 20.00th=[ 709], | 30.00th=[ 793], 40.00th=[ 877], 50.00th=[ 944], 60.00th=[ 1011], | 70.00th=[ 1083], 80.00th=[ 1183], 90.00th=[ 1318], 95.00th=[ 1469], | 99.00th=[ 1720], 99.50th=[ 1737], 99.90th=[ 1888], 99.95th=[ 1888], | 99.99th=[ 1888] bw ( KiB/s): min= 15, max= 223, per=25.23%, avg=67.62, stdev=30.79, samples=298 iops : min= 3, max= 55, avg=16.45, stdev= 7.75, samples=298 lat (msec) : 250=0.08%, 500=5.27%, 750=18.63%, 1000=34.53%, 2000=41.48% cpu : usr=0.05%, sys=0.19%, ctx=1736, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145172: Thu Aug 27 11:46:00 2026 write: IOPS=16, BW=67.1KiB/s (68.7kB/s)(10.0MiB/152643msec); 0 zone resets slat (usec): min=22, max=20049, avg=212.62, stdev=902.77 clat (msec): min=263, max=1890, avg=952.88, stdev=276.85 lat (msec): min=270, max=1890, avg=953.10, stdev=276.84 clat percentiles (msec): | 1.00th=[ 380], 5.00th=[ 498], 10.00th=[ 609], 20.00th=[ 718], | 30.00th=[ 810], 40.00th=[ 877], 50.00th=[ 936], 60.00th=[ 1003], | 70.00th=[ 1083], 80.00th=[ 1167], 90.00th=[ 1318], 95.00th=[ 1435], | 99.00th=[ 1687], 99.50th=[ 1737], 99.90th=[ 1770], 99.95th=[ 1787], | 99.99th=[ 1888] bw ( KiB/s): min= 8, max= 160, per=24.99%, avg=66.96, stdev=27.47, samples=301 iops : min= 2, max= 40, avg=16.29, stdev= 6.91, samples=301 lat (msec) : 500=5.04%, 750=17.50%, 1000=36.84%, 2000=40.62% cpu : usr=0.04%, sys=0.23%, ctx=1983, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=268KiB/s (275kB/s), 67.1KiB/s-67.1KiB/s (68.7kB/s-68.7kB/s), io=40.0MiB (41.9MB), run=152534-152716msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=145206: Thu Aug 27 11:47:55 2026 read: IOPS=12, BW=48.1KiB/s (49.3kB/s)(5048KiB/104879msec) slat (usec): min=2721, max=52679, avg=13880.57, stdev=7894.91 clat (msec): min=49, max=984, avg=410.78, stdev=116.80 lat (msec): min=57, max=997, avg=424.67, stdev=118.85 clat percentiles (msec): | 1.00th=[ 207], 5.00th=[ 266], 10.00th=[ 296], 20.00th=[ 326], | 30.00th=[ 351], 40.00th=[ 368], 50.00th=[ 388], 60.00th=[ 409], | 70.00th=[ 439], 80.00th=[ 485], 90.00th=[ 558], 95.00th=[ 617], | 99.00th=[ 852], 99.50th=[ 902], 99.90th=[ 936], 99.95th=[ 986], | 99.99th=[ 986] bw ( KiB/s): min= 7, max= 111, per=24.93%, avg=47.62, stdev=21.42, samples=208 iops : min= 1, max= 27, avg=11.42, stdev= 5.38, samples=208 write: IOPS=12, BW=49.5KiB/s (50.7kB/s)(5192KiB/104879msec); 0 zone resets slat (usec): min=28, max=32956, avg=267.99, stdev=1330.23 clat (msec): min=201, max=1911, avg=876.89, stdev=250.58 lat (msec): min=201, max=1911, avg=877.16, stdev=250.64 clat percentiles (msec): | 1.00th=[ 456], 5.00th=[ 523], 10.00th=[ 592], 20.00th=[ 667], | 30.00th=[ 751], 40.00th=[ 802], 50.00th=[ 844], 60.00th=[ 894], | 70.00th=[ 953], 80.00th=[ 1036], 90.00th=[ 1217], 95.00th=[ 1368], | 99.00th=[ 1653], 99.50th=[ 1703], 99.90th=[ 1888], 99.95th=[ 1905], | 99.99th=[ 1905] bw ( KiB/s): min= 7, max= 103, per=25.19%, avg=49.88, stdev=20.36, samples=204 iops : min= 1, max= 25, avg=11.99, stdev= 5.09, samples=204 lat (msec) : 50=0.04%, 100=0.12%, 250=1.25%, 500=40.62%, 750=21.95% lat (msec) : 1000=23.40%, 2000=12.62% cpu : usr=0.10%, sys=0.61%, ctx=4565, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145207: Thu Aug 27 11:47:55 2026 read: IOPS=11, BW=47.4KiB/s (48.6kB/s)(4976KiB/104874msec) slat (usec): min=37, max=60158, avg=14159.32, stdev=8082.76 clat (msec): min=17, max=1027, avg=403.91, stdev=120.57 lat (msec): min=32, max=1049, avg=418.07, stdev=122.68 clat percentiles (msec): | 1.00th=[ 92], 5.00th=[ 268], 10.00th=[ 288], 20.00th=[ 317], | 30.00th=[ 342], 40.00th=[ 368], 50.00th=[ 384], 60.00th=[ 405], | 70.00th=[ 430], 80.00th=[ 472], 90.00th=[ 558], 95.00th=[ 634], | 99.00th=[ 844], 99.50th=[ 919], 99.90th=[ 986], 99.95th=[ 1028], | 99.99th=[ 1028] bw ( KiB/s): min= 7, max= 119, per=24.65%, avg=47.09, stdev=22.79, samples=207 iops : min= 1, max= 29, avg=11.29, stdev= 5.70, samples=207 write: IOPS=12, BW=50.2KiB/s (51.4kB/s)(5264KiB/104874msec); 0 zone resets slat (usec): min=29, max=13606, avg=259.11, stdev=1008.93 clat (msec): min=240, max=2033, avg=878.83, stdev=254.76 lat (msec): min=240, max=2033, avg=879.09, stdev=254.81 clat percentiles (msec): | 1.00th=[ 300], 5.00th=[ 535], 10.00th=[ 592], 20.00th=[ 676], | 30.00th=[ 751], 40.00th=[ 802], 50.00th=[ 852], 60.00th=[ 894], | 70.00th=[ 961], 80.00th=[ 1045], 90.00th=[ 1217], 95.00th=[ 1368], | 99.00th=[ 1670], 99.50th=[ 1737], 99.90th=[ 2022], 99.95th=[ 2039], | 99.99th=[ 2039] bw ( KiB/s): min= 7, max= 111, per=25.15%, avg=49.80, stdev=20.20, samples=207 iops : min= 1, max= 27, avg=11.97, stdev= 5.09, samples=207 lat (msec) : 20=0.04%, 50=0.12%, 100=0.35%, 250=1.09%, 500=40.86% lat (msec) : 750=20.82%, 1000=23.98%, 2000=12.66%, >=2000=0.08% cpu : usr=0.10%, sys=0.65%, ctx=4549, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145208: Thu Aug 27 11:47:55 2026 read: IOPS=11, BW=47.5KiB/s (48.6kB/s)(4984KiB/105005msec) slat (usec): min=3044, max=59746, avg=14144.46, stdev=7773.31 clat (msec): min=7, max=952, avg=403.74, stdev=114.52 lat (msec): min=16, max=982, avg=417.89, stdev=116.34 clat percentiles (msec): | 1.00th=[ 190], 5.00th=[ 268], 10.00th=[ 292], 20.00th=[ 321], | 30.00th=[ 338], 40.00th=[ 363], 50.00th=[ 384], 60.00th=[ 409], | 70.00th=[ 430], 80.00th=[ 477], 90.00th=[ 550], 95.00th=[ 617], | 99.00th=[ 802], 99.50th=[ 844], 99.90th=[ 885], 99.95th=[ 953], | 99.99th=[ 953] bw ( KiB/s): min= 7, max= 119, per=24.96%, avg=47.67, stdev=21.54, samples=205 iops : min= 1, max= 29, avg=11.43, stdev= 5.44, samples=205 write: IOPS=12, BW=50.1KiB/s (51.3kB/s)(5256KiB/105005msec); 0 zone resets slat (usec): min=32, max=16830, avg=222.66, stdev=807.10 clat (msec): min=132, max=2059, avg=879.18, stdev=255.75 lat (msec): min=133, max=2059, avg=879.41, stdev=255.82 clat percentiles (msec): | 1.00th=[ 430], 5.00th=[ 531], 10.00th=[ 600], 20.00th=[ 676], | 30.00th=[ 743], 40.00th=[ 793], 50.00th=[ 844], 60.00th=[ 894], | 70.00th=[ 961], 80.00th=[ 1053], 90.00th=[ 1217], 95.00th=[ 1401], | 99.00th=[ 1687], 99.50th=[ 1821], 99.90th=[ 1938], 99.95th=[ 2056], | 99.99th=[ 2056] bw ( KiB/s): min= 7, max= 111, per=24.90%, avg=49.30, stdev=18.93, samples=207 iops : min= 1, max= 27, avg=11.84, stdev= 4.75, samples=207 lat (msec) : 10=0.04%, 20=0.08%, 50=0.12%, 100=0.16%, 250=1.21% lat (msec) : 500=41.09%, 750=21.41%, 1000=23.20%, 2000=12.66%, >=2000=0.04% cpu : usr=0.08%, sys=0.62%, ctx=4674, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145209: Thu Aug 27 11:47:55 2026 read: IOPS=12, BW=48.2KiB/s (49.4kB/s)(5060KiB/104938msec) slat (usec): min=2561, max=75778, avg=14384.66, stdev=8260.99 clat (msec): min=9, max=1041, avg=399.69, stdev=108.08 lat (msec): min=20, max=1067, avg=414.08, stdev=109.96 clat percentiles (msec): | 1.00th=[ 190], 5.00th=[ 268], 10.00th=[ 292], 20.00th=[ 321], | 30.00th=[ 347], 40.00th=[ 363], 50.00th=[ 384], 60.00th=[ 405], | 70.00th=[ 430], 80.00th=[ 472], 90.00th=[ 535], 95.00th=[ 592], | 99.00th=[ 760], 99.50th=[ 860], 99.90th=[ 1020], 99.95th=[ 1045], | 99.99th=[ 1045] bw ( KiB/s): min= 6, max= 135, per=25.16%, avg=48.05, stdev=22.49, samples=207 iops : min= 1, max= 33, avg=11.53, stdev= 5.64, samples=207 write: IOPS=12, BW=49.4KiB/s (50.5kB/s)(5180KiB/104938msec); 0 zone resets slat (usec): min=28, max=12125, avg=222.43, stdev=669.26 clat (msec): min=341, max=1927, avg=889.07, stdev=255.94 lat (msec): min=341, max=1927, avg=889.29, stdev=255.98 clat percentiles (msec): | 1.00th=[ 460], 5.00th=[ 542], 10.00th=[ 609], 20.00th=[ 684], | 30.00th=[ 751], 40.00th=[ 810], 50.00th=[ 852], 60.00th=[ 902], | 70.00th=[ 961], 80.00th=[ 1053], 90.00th=[ 1217], 95.00th=[ 1401], | 99.00th=[ 1737], 99.50th=[ 1804], 99.90th=[ 1888], 99.95th=[ 1921], | 99.99th=[ 1921] bw ( KiB/s): min= 7, max= 103, per=24.71%, avg=48.92, stdev=19.92, samples=207 iops : min= 1, max= 25, avg=11.75, stdev= 5.01, samples=207 lat (msec) : 10=0.04%, 50=0.12%, 100=0.16%, 250=1.09%, 500=42.07% lat (msec) : 750=20.23%, 1000=23.32%, 2000=12.97% cpu : usr=0.13%, sys=0.60%, ctx=4767, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=191KiB/s (196kB/s), 47.4KiB/s-48.2KiB/s (48.6kB/s-49.4kB/s), io=19.6MiB (20.5MB), run=104874-105005msec WRITE: bw=199KiB/s (204kB/s), 49.4KiB/s-50.2KiB/s (50.5kB/s-51.4kB/s), io=20.4MiB (21.4MB), run=104874-105005msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=145229: Thu Aug 27 11:48:07 2026 read: IOPS=1, BW=65.9MiB/s (69.1MB/s)(40.0MiB/607msec) slat (nsec): min=61314k, max=61314k, avg=61314368.00, stdev= 0.00 clat (nsec): min=518611k, max=518611k, avg=518611149.00, stdev= 0.00 lat (nsec): min=579956k, max=579956k, avg=579955892.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 518], 5.00th=[ 518], 10.00th=[ 518], 20.00th=[ 518], | 30.00th=[ 518], 40.00th=[ 518], 50.00th=[ 518], 60.00th=[ 518], | 70.00th=[ 518], 80.00th=[ 518], 90.00th=[ 518], 95.00th=[ 518], | 99.00th=[ 518], 99.50th=[ 518], 99.90th=[ 518], 99.95th=[ 518], | 99.99th=[ 518] lat (msec) : 750=100.00% cpu : usr=0.33%, sys=2.81%, ctx=44, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=65.9MiB/s (69.1MB/s), 65.9MiB/s-65.9MiB/s (69.1MB/s-69.1MB/s), io=40.0MiB (41.9MB), run=607-607msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout