== sanity test 398c: run fio to test AIO ================= 21:52:02 (1787795522) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.504115 s, 83.2 MB/s osc.lustre-OST0000-osc-ffff8eac51887000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=147871: Wed Aug 26 21:55:13 2026 write: IOPS=14, BW=57.4KiB/s (58.7kB/s)(10.0MiB/178510msec); 0 zone resets slat (usec): min=25, max=28953, avg=284.99, stdev=1117.70 clat (msec): min=77, max=3051, avg=1114.83, stdev=356.52 lat (msec): min=77, max=3051, avg=1115.12, stdev=356.50 clat percentiles (msec): | 1.00th=[ 439], 5.00th=[ 642], 10.00th=[ 768], 20.00th=[ 852], | 30.00th=[ 919], 40.00th=[ 978], 50.00th=[ 1045], 60.00th=[ 1133], | 70.00th=[ 1234], 80.00th=[ 1351], 90.00th=[ 1569], 95.00th=[ 1821], | 99.00th=[ 2265], 99.50th=[ 2500], 99.90th=[ 3004], 99.95th=[ 3037], | 99.99th=[ 3037] bw ( KiB/s): min= 7, max= 144, per=25.12%, avg=57.28, stdev=27.22, samples=351 iops : min= 1, max= 36, avg=13.83, stdev= 6.86, samples=351 lat (msec) : 100=0.08%, 250=0.08%, 500=1.99%, 750=6.84%, 1000=34.41% lat (msec) : 2000=54.34%, >=2000=2.27% cpu : usr=0.04%, sys=0.24%, ctx=1804, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=147872: Wed Aug 26 21:55:13 2026 write: IOPS=14, BW=57.2KiB/s (58.5kB/s)(10.0MiB/179141msec); 0 zone resets slat (usec): min=24, max=21394, avg=273.33, stdev=1045.14 clat (msec): min=425, max=3016, avg=1117.74, stdev=351.95 lat (msec): min=425, max=3017, avg=1118.02, stdev=351.99 clat percentiles (msec): | 1.00th=[ 451], 5.00th=[ 634], 10.00th=[ 776], 20.00th=[ 860], | 30.00th=[ 927], 40.00th=[ 969], 50.00th=[ 1045], 60.00th=[ 1133], | 70.00th=[ 1234], 80.00th=[ 1368], 90.00th=[ 1552], 95.00th=[ 1804], | 99.00th=[ 2232], 99.50th=[ 2635], 99.90th=[ 2836], 99.95th=[ 3004], | 99.99th=[ 3004] bw ( KiB/s): min= 7, max= 128, per=25.57%, avg=58.31, stdev=31.55, samples=345 iops : min= 1, max= 32, avg=14.08, stdev= 7.92, samples=345 lat (msec) : 500=1.45%, 750=7.34%, 1000=34.26%, 2000=54.84%, >=2000=2.11% cpu : usr=0.04%, sys=0.22%, ctx=1468, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=147873: Wed Aug 26 21:55:13 2026 write: IOPS=14, BW=57.2KiB/s (58.6kB/s)(10.0MiB/179090msec); 0 zone resets slat (usec): min=28, max=32961, avg=278.54, stdev=1310.62 clat (msec): min=405, max=3001, avg=1117.81, stdev=350.95 lat (msec): min=405, max=3001, avg=1118.09, stdev=350.97 clat percentiles (msec): | 1.00th=[ 468], 5.00th=[ 684], 10.00th=[ 785], 20.00th=[ 869], | 30.00th=[ 919], 40.00th=[ 986], 50.00th=[ 1036], 60.00th=[ 1116], | 70.00th=[ 1217], 80.00th=[ 1368], 90.00th=[ 1569], 95.00th=[ 1787], | 99.00th=[ 2299], 99.50th=[ 2500], 99.90th=[ 3004], 99.95th=[ 3004], | 99.99th=[ 3004] bw ( KiB/s): min= 7, max= 256, per=26.03%, avg=59.34, stdev=32.66, samples=339 iops : min= 1, max= 64, avg=14.35, stdev= 8.18, samples=339 lat (msec) : 500=2.50%, 750=4.88%, 1000=35.59%, 2000=55.04%, >=2000=1.99% cpu : usr=0.05%, sys=0.26%, ctx=1785, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=147874: Wed Aug 26 21:55:13 2026 write: IOPS=14, BW=57.2KiB/s (58.6kB/s)(10.0MiB/178928msec); 0 zone resets slat (usec): min=31, max=21681, avg=282.76, stdev=1083.93 clat (msec): min=206, max=2916, avg=1116.49, stdev=351.74 lat (msec): min=206, max=2916, avg=1116.78, stdev=351.80 clat percentiles (msec): | 1.00th=[ 456], 5.00th=[ 642], 10.00th=[ 760], 20.00th=[ 852], | 30.00th=[ 911], 40.00th=[ 969], 50.00th=[ 1036], 60.00th=[ 1133], | 70.00th=[ 1234], 80.00th=[ 1368], 90.00th=[ 1569], 95.00th=[ 1804], | 99.00th=[ 2165], 99.50th=[ 2400], 99.90th=[ 2903], 99.95th=[ 2903], | 99.99th=[ 2903] bw ( KiB/s): min= 7, max= 128, per=25.88%, avg=59.00, stdev=30.17, samples=341 iops : min= 1, max= 32, avg=14.25, stdev= 7.58, samples=341 lat (msec) : 250=0.04%, 500=1.33%, 750=7.70%, 1000=35.08%, 2000=53.67% lat (msec) : >=2000=2.19% cpu : usr=0.06%, sys=0.24%, ctx=1795, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=229KiB/s (234kB/s), 57.2KiB/s-57.4KiB/s (58.5kB/s-58.7kB/s), io=40.0MiB (41.9MB), run=178510-179141msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=147911: Wed Aug 26 21:57:23 2026 read: IOPS=10, BW=42.4KiB/s (43.4kB/s)(5048KiB/119066msec) slat (usec): min=531, max=75642, avg=16176.00, stdev=8914.14 clat (msec): min=27, max=1396, avg=466.26, stdev=152.83 lat (msec): min=33, max=1412, avg=482.44, stdev=154.17 clat percentiles (msec): | 1.00th=[ 150], 5.00th=[ 279], 10.00th=[ 309], 20.00th=[ 351], | 30.00th=[ 380], 40.00th=[ 414], 50.00th=[ 443], 60.00th=[ 481], | 70.00th=[ 518], 80.00th=[ 575], 90.00th=[ 642], 95.00th=[ 726], | 99.00th=[ 978], 99.50th=[ 1099], 99.90th=[ 1318], 99.95th=[ 1401], | 99.99th=[ 1401] bw ( KiB/s): min= 7, max= 134, per=25.19%, avg=42.31, stdev=19.46, samples=233 iops : min= 1, max= 33, avg=10.03, stdev= 4.94, samples=233 write: IOPS=10, BW=43.6KiB/s (44.7kB/s)(5192KiB/119066msec); 0 zone resets slat (usec): min=29, max=29727, avg=337.97, stdev=1206.04 clat (msec): min=172, max=1880, avg=996.48, stdev=272.10 lat (msec): min=173, max=1885, avg=996.82, stdev=272.04 clat percentiles (msec): | 1.00th=[ 439], 5.00th=[ 592], 10.00th=[ 684], 20.00th=[ 785], | 30.00th=[ 844], 40.00th=[ 902], 50.00th=[ 961], 60.00th=[ 1036], | 70.00th=[ 1116], 80.00th=[ 1217], 90.00th=[ 1351], 95.00th=[ 1485], | 99.00th=[ 1770], 99.50th=[ 1821], 99.90th=[ 1871], 99.95th=[ 1888], | 99.99th=[ 1888] bw ( KiB/s): min= 7, max= 119, per=24.84%, avg=43.48, stdev=20.02, samples=233 iops : min= 1, max= 29, avg=10.33, stdev= 5.05, samples=233 lat (msec) : 50=0.08%, 100=0.23%, 250=1.37%, 500=31.88%, 750=21.91% lat (msec) : 1000=21.37%, 2000=23.16% cpu : usr=0.12%, sys=0.69%, ctx=4123, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=147912: Wed Aug 26 21:57:23 2026 read: IOPS=10, BW=41.8KiB/s (42.8kB/s)(4976KiB/119066msec) slat (usec): min=3305, max=60256, avg=16125.47, stdev=8596.39 clat (msec): min=41, max=1121, avg=468.03, stdev=142.24 lat (msec): min=71, max=1145, avg=484.16, stdev=143.74 clat percentiles (msec): | 1.00th=[ 211], 5.00th=[ 284], 10.00th=[ 313], 20.00th=[ 351], | 30.00th=[ 380], 40.00th=[ 418], 50.00th=[ 451], 60.00th=[ 485], | 70.00th=[ 523], 80.00th=[ 575], 90.00th=[ 642], 95.00th=[ 718], | 99.00th=[ 953], 99.50th=[ 986], 99.90th=[ 1099], 99.95th=[ 1116], | 99.99th=[ 1116] bw ( KiB/s): min= 7, max= 120, per=24.82%, avg=41.70, stdev=19.81, samples=233 iops : min= 1, max= 30, avg= 9.88, stdev= 5.02, samples=233 write: IOPS=11, BW=44.2KiB/s (45.3kB/s)(5264KiB/119066msec); 0 zone resets slat (usec): min=31, max=19183, avg=339.50, stdev=1055.69 clat (msec): min=263, max=1958, avg=987.00, stdev=268.60 lat (msec): min=263, max=1958, avg=987.34, stdev=268.63 clat percentiles (msec): | 1.00th=[ 447], 5.00th=[ 584], 10.00th=[ 676], 20.00th=[ 768], | 30.00th=[ 844], 40.00th=[ 894], 50.00th=[ 944], 60.00th=[ 1011], | 70.00th=[ 1099], 80.00th=[ 1217], 90.00th=[ 1351], 95.00th=[ 1485], | 99.00th=[ 1687], 99.50th=[ 1770], 99.90th=[ 1938], 99.95th=[ 1955], | 99.99th=[ 1955] bw ( KiB/s): min= 7, max= 118, per=24.89%, avg=43.55, stdev=20.26, samples=236 iops : min= 1, max= 29, avg=10.35, stdev= 5.09, samples=236 lat (msec) : 50=0.04%, 100=0.20%, 250=0.74%, 500=31.48%, 750=23.55% lat (msec) : 1000=22.27%, 2000=21.72% cpu : usr=0.11%, sys=0.66%, ctx=4172, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=147913: Wed Aug 26 21:57:23 2026 read: IOPS=10, BW=41.8KiB/s (42.8kB/s)(4984KiB/119263msec) slat (usec): min=3216, max=81827, avg=15989.29, stdev=8738.62 clat (msec): min=44, max=1106, avg=460.63, stdev=140.28 lat (msec): min=56, max=1124, avg=476.63, stdev=141.56 clat percentiles (msec): | 1.00th=[ 209], 5.00th=[ 279], 10.00th=[ 313], 20.00th=[ 351], | 30.00th=[ 380], 40.00th=[ 409], 50.00th=[ 439], 60.00th=[ 468], | 70.00th=[ 510], 80.00th=[ 558], 90.00th=[ 634], 95.00th=[ 709], | 99.00th=[ 969], 99.50th=[ 1020], 99.90th=[ 1099], 99.95th=[ 1099], | 99.99th=[ 1099] bw ( KiB/s): min= 7, max= 135, per=24.82%, avg=41.69, stdev=22.22, samples=234 iops : min= 1, max= 33, avg= 9.88, stdev= 5.59, samples=234 write: IOPS=11, BW=44.1KiB/s (45.1kB/s)(5256KiB/119263msec); 0 zone resets slat (usec): min=32, max=12844, avg=333.26, stdev=993.90 clat (msec): min=180, max=1970, avg=997.63, stdev=272.57 lat (msec): min=180, max=1970, avg=997.96, stdev=272.60 clat percentiles (msec): | 1.00th=[ 435], 5.00th=[ 584], 10.00th=[ 676], 20.00th=[ 776], | 30.00th=[ 852], 40.00th=[ 902], 50.00th=[ 953], 60.00th=[ 1028], | 70.00th=[ 1133], 80.00th=[ 1234], 90.00th=[ 1351], 95.00th=[ 1485], | 99.00th=[ 1687], 99.50th=[ 1804], 99.90th=[ 1854], 99.95th=[ 1972], | 99.99th=[ 1972] bw ( KiB/s): min= 7, max= 96, per=25.10%, avg=43.92, stdev=18.39, samples=233 iops : min= 1, max= 24, avg=10.44, stdev= 4.63, samples=233 lat (msec) : 50=0.04%, 100=0.12%, 250=1.09%, 500=32.85%, 750=21.48% lat (msec) : 1000=21.91%, 2000=22.50% cpu : usr=0.11%, sys=0.67%, ctx=4331, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=147914: Wed Aug 26 21:57:23 2026 read: IOPS=10, BW=42.5KiB/s (43.5kB/s)(5060KiB/119135msec) slat (usec): min=3062, max=72403, avg=16376.59, stdev=9298.83 clat (msec): min=15, max=1082, avg=462.42, stdev=133.16 lat (msec): min=25, max=1106, avg=478.81, stdev=134.93 clat percentiles (msec): | 1.00th=[ 224], 5.00th=[ 292], 10.00th=[ 321], 20.00th=[ 355], | 30.00th=[ 380], 40.00th=[ 418], 50.00th=[ 451], 60.00th=[ 477], | 70.00th=[ 510], 80.00th=[ 558], 90.00th=[ 634], 95.00th=[ 693], | 99.00th=[ 885], 99.50th=[ 995], 99.90th=[ 1028], 99.95th=[ 1083], | 99.99th=[ 1083] bw ( KiB/s): min= 7, max= 111, per=25.10%, avg=42.17, stdev=19.99, samples=235 iops : min= 1, max= 27, avg=10.00, stdev= 5.04, samples=235 write: IOPS=10, BW=43.5KiB/s (44.5kB/s)(5180KiB/119135msec); 0 zone resets slat (usec): min=35, max=18476, avg=431.32, stdev=1363.95 clat (msec): min=278, max=1969, avg=1000.96, stdev=269.32 lat (msec): min=278, max=1969, avg=1001.39, stdev=269.33 clat percentiles (msec): | 1.00th=[ 447], 5.00th=[ 600], 10.00th=[ 676], 20.00th=[ 785], | 30.00th=[ 852], 40.00th=[ 902], 50.00th=[ 961], 60.00th=[ 1036], | 70.00th=[ 1116], 80.00th=[ 1234], 90.00th=[ 1351], 95.00th=[ 1485], | 99.00th=[ 1720], 99.50th=[ 1821], 99.90th=[ 1888], 99.95th=[ 1972], | 99.99th=[ 1972] bw ( KiB/s): min= 7, max= 111, per=24.74%, avg=43.30, stdev=19.54, samples=233 iops : min= 1, max= 27, avg=10.28, stdev= 4.95, samples=233 lat (msec) : 20=0.04%, 50=0.12%, 250=0.55%, 500=33.40%, 750=21.91% lat (msec) : 1000=21.52%, 2000=22.46% cpu : usr=0.12%, sys=0.66%, ctx=4221, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=168KiB/s (172kB/s), 41.8KiB/s-42.5KiB/s (42.8kB/s-43.5kB/s), io=19.6MiB (20.5MB), run=119066-119263msec WRITE: bw=175KiB/s (179kB/s), 43.5KiB/s-44.2KiB/s (44.5kB/s-45.3kB/s), io=20.4MiB (21.4MB), run=119066-119263msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=147934: Wed Aug 26 21:57:30 2026 read: IOPS=1, BW=70.5MiB/s (73.0MB/s)(40.0MiB/567msec) slat (nsec): min=14827k, max=14827k, avg=14826593.00, stdev= 0.00 clat (nsec): min=522328k, max=522328k, avg=522328481.00, stdev= 0.00 lat (nsec): min=537160k, max=537160k, avg=537159954.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 523], 5.00th=[ 523], 10.00th=[ 523], 20.00th=[ 523], | 30.00th=[ 523], 40.00th=[ 523], 50.00th=[ 523], 60.00th=[ 523], | 70.00th=[ 523], 80.00th=[ 523], 90.00th=[ 523], 95.00th=[ 523], | 99.00th=[ 523], 99.50th=[ 523], 99.90th=[ 523], 99.95th=[ 523], | 99.99th=[ 523] lat (msec) : 750=100.00% cpu : usr=0.00%, sys=1.94%, ctx=41, majf=0, minf=7 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=70.5MiB/s (73.0MB/s), 70.5MiB/s-70.5MiB/s (73.0MB/s-73.0MB/s), io=40.0MiB (41.9MB), run=567-567msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout