== sanity test 398c: run fio to test AIO ================= 05:52:20 (1787910740) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.518555 s, 80.9 MB/s osc.lustre-OST0000-osc-ffff9cb02b701000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=145002: Fri Aug 28 05:56:03 2026 write: IOPS=12, BW=48.5KiB/s (49.7kB/s)(10.0MiB/211102msec); 0 zone resets slat (usec): min=29, max=18072, avg=301.10, stdev=1153.54 clat (msec): min=281, max=3006, avg=1317.86, stdev=439.11 lat (msec): min=282, max=3006, avg=1318.16, stdev=439.09 clat percentiles (msec): | 1.00th=[ 372], 5.00th=[ 760], 10.00th=[ 869], 20.00th=[ 986], | 30.00th=[ 1083], 40.00th=[ 1150], 50.00th=[ 1250], 60.00th=[ 1334], | 70.00th=[ 1435], 80.00th=[ 1603], 90.00th=[ 1905], 95.00th=[ 2232], | 99.00th=[ 2668], 99.50th=[ 2802], 99.90th=[ 3004], 99.95th=[ 3004], | 99.99th=[ 3004] bw ( KiB/s): min= 7, max= 128, per=26.26%, avg=50.68, stdev=28.95, samples=395 iops : min= 1, max= 32, avg=12.12, stdev= 7.28, samples=395 lat (msec) : 500=1.33%, 750=3.01%, 1000=17.54%, 2000=70.27%, >=2000=7.85% cpu : usr=0.05%, sys=0.21%, ctx=1760, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145003: Fri Aug 28 05:56:03 2026 write: IOPS=12, BW=48.4KiB/s (49.6kB/s)(10.0MiB/211439msec); 0 zone resets slat (usec): min=29, max=22969, avg=331.54, stdev=1231.11 clat (msec): min=318, max=3008, avg=1319.79, stdev=431.83 lat (msec): min=318, max=3008, avg=1320.13, stdev=431.82 clat percentiles (msec): | 1.00th=[ 372], 5.00th=[ 776], 10.00th=[ 860], 20.00th=[ 995], | 30.00th=[ 1083], 40.00th=[ 1167], 50.00th=[ 1250], 60.00th=[ 1334], | 70.00th=[ 1435], 80.00th=[ 1603], 90.00th=[ 1905], 95.00th=[ 2232], | 99.00th=[ 2668], 99.50th=[ 2735], 99.90th=[ 2869], 99.95th=[ 2869], | 99.99th=[ 3004] bw ( KiB/s): min= 7, max= 128, per=25.35%, avg=48.92, stdev=27.98, samples=409 iops : min= 1, max= 32, avg=11.68, stdev= 7.02, samples=409 lat (msec) : 500=1.13%, 750=3.05%, 1000=16.84%, 2000=71.09%, >=2000=7.89% cpu : usr=0.05%, sys=0.23%, ctx=1980, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145004: Fri Aug 28 05:56:03 2026 write: IOPS=12, BW=48.6KiB/s (49.8kB/s)(10.0MiB/210724msec); 0 zone resets slat (usec): min=28, max=25905, avg=319.84, stdev=1263.37 clat (msec): min=128, max=2933, avg=1316.20, stdev=439.43 lat (msec): min=128, max=2934, avg=1316.52, stdev=439.45 clat percentiles (msec): | 1.00th=[ 347], 5.00th=[ 760], 10.00th=[ 860], 20.00th=[ 986], | 30.00th=[ 1083], 40.00th=[ 1150], 50.00th=[ 1250], 60.00th=[ 1334], | 70.00th=[ 1435], 80.00th=[ 1603], 90.00th=[ 1905], 95.00th=[ 2232], | 99.00th=[ 2635], 99.50th=[ 2836], 99.90th=[ 2903], 99.95th=[ 2937], | 99.99th=[ 2937] bw ( KiB/s): min= 7, max= 172, per=25.23%, avg=48.70, stdev=25.57, samples=410 iops : min= 1, max= 43, avg=11.63, stdev= 6.41, samples=410 lat (msec) : 250=0.35%, 500=1.25%, 750=2.77%, 1000=16.68%, 2000=70.55% lat (msec) : >=2000=8.40% cpu : usr=0.04%, sys=0.21%, ctx=1783, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145005: Fri Aug 28 05:56:03 2026 write: IOPS=12, BW=48.4KiB/s (49.6kB/s)(10.0MiB/211394msec); 0 zone resets slat (usec): min=29, max=25918, avg=328.39, stdev=1236.57 clat (msec): min=314, max=2918, avg=1319.44, stdev=427.38 lat (msec): min=314, max=2918, avg=1319.77, stdev=427.36 clat percentiles (msec): | 1.00th=[ 592], 5.00th=[ 768], 10.00th=[ 869], 20.00th=[ 995], | 30.00th=[ 1083], 40.00th=[ 1167], 50.00th=[ 1250], 60.00th=[ 1318], | 70.00th=[ 1435], 80.00th=[ 1603], 90.00th=[ 1854], 95.00th=[ 2265], | 99.00th=[ 2668], 99.50th=[ 2769], 99.90th=[ 2903], 99.95th=[ 2903], | 99.99th=[ 2903] bw ( KiB/s): min= 7, max= 128, per=25.72%, avg=49.65, stdev=29.55, samples=403 iops : min= 1, max= 32, avg=11.85, stdev= 7.44, samples=403 lat (msec) : 500=0.66%, 750=3.44%, 1000=16.09%, 2000=72.34%, >=2000=7.46% cpu : usr=0.05%, sys=0.22%, ctx=1875, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=194KiB/s (198kB/s), 48.4KiB/s-48.6KiB/s (49.6kB/s-49.8kB/s), io=40.0MiB (41.9MB), run=210724-211439msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=145046: Fri Aug 28 05:58:18 2026 read: IOPS=10, BW=40.1KiB/s (41.1kB/s)(5048KiB/125913msec) slat (usec): min=3148, max=71338, avg=16585.64, stdev=9129.29 clat (msec): min=35, max=940, avg=487.92, stdev=133.66 lat (msec): min=46, max=984, avg=504.51, stdev=134.89 clat percentiles (msec): | 1.00th=[ 201], 5.00th=[ 305], 10.00th=[ 334], 20.00th=[ 380], | 30.00th=[ 414], 40.00th=[ 447], 50.00th=[ 472], 60.00th=[ 506], | 70.00th=[ 542], 80.00th=[ 592], 90.00th=[ 684], 95.00th=[ 735], | 99.00th=[ 835], 99.50th=[ 869], 99.90th=[ 936], 99.95th=[ 944], | 99.99th=[ 944] bw ( KiB/s): min= 7, max= 112, per=25.28%, avg=40.20, stdev=19.17, samples=245 iops : min= 1, max= 28, avg= 9.50, stdev= 4.84, samples=245 write: IOPS=10, BW=41.2KiB/s (42.2kB/s)(5192KiB/125913msec); 0 zone resets slat (usec): min=34, max=20958, avg=431.21, stdev=1514.16 clat (msec): min=296, max=2523, avg=1058.42, stdev=271.36 lat (msec): min=296, max=2523, avg=1058.85, stdev=271.40 clat percentiles (msec): | 1.00th=[ 498], 5.00th=[ 642], 10.00th=[ 735], 20.00th=[ 835], | 30.00th=[ 911], 40.00th=[ 978], 50.00th=[ 1036], 60.00th=[ 1116], | 70.00th=[ 1183], 80.00th=[ 1284], 90.00th=[ 1401], 95.00th=[ 1502], | 99.00th=[ 1804], 99.50th=[ 1955], 99.90th=[ 2500], 99.95th=[ 2534], | 99.99th=[ 2534] bw ( KiB/s): min= 7, max= 100, per=24.72%, avg=40.78, stdev=19.02, samples=248 iops : min= 1, max= 25, avg= 9.64, stdev= 4.80, samples=248 lat (msec) : 50=0.08%, 100=0.12%, 250=0.62%, 500=28.67%, 750=23.55% lat (msec) : 1000=17.85%, 2000=28.95%, >=2000=0.16% cpu : usr=0.11%, sys=0.62%, ctx=4176, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145047: Fri Aug 28 05:58:18 2026 read: IOPS=9, BW=39.5KiB/s (40.4kB/s)(4976KiB/126081msec) slat (usec): min=3164, max=73586, avg=16576.21, stdev=9216.59 clat (msec): min=32, max=949, avg=480.87, stdev=129.49 lat (msec): min=46, max=987, avg=497.46, stdev=131.01 clat percentiles (msec): | 1.00th=[ 224], 5.00th=[ 296], 10.00th=[ 330], 20.00th=[ 376], | 30.00th=[ 409], 40.00th=[ 443], 50.00th=[ 468], 60.00th=[ 502], | 70.00th=[ 535], 80.00th=[ 584], 90.00th=[ 642], 95.00th=[ 726], | 99.00th=[ 852], 99.50th=[ 877], 99.90th=[ 936], 99.95th=[ 953], | 99.99th=[ 953] bw ( KiB/s): min= 7, max= 103, per=24.74%, avg=39.34, stdev=19.63, samples=246 iops : min= 1, max= 25, avg= 9.27, stdev= 4.92, samples=246 write: IOPS=10, BW=41.8KiB/s (42.8kB/s)(5264KiB/126081msec); 0 zone resets slat (usec): min=29, max=25832, avg=364.90, stdev=1233.10 clat (msec): min=262, max=2536, avg=1059.83, stdev=288.63 lat (msec): min=262, max=2536, avg=1060.20, stdev=288.68 clat percentiles (msec): | 1.00th=[ 443], 5.00th=[ 617], 10.00th=[ 709], 20.00th=[ 827], | 30.00th=[ 902], 40.00th=[ 978], 50.00th=[ 1045], 60.00th=[ 1116], | 70.00th=[ 1183], 80.00th=[ 1301], 90.00th=[ 1418], 95.00th=[ 1519], | 99.00th=[ 1854], 99.50th=[ 2089], 99.90th=[ 2500], 99.95th=[ 2534], | 99.99th=[ 2534] bw ( KiB/s): min= 7, max= 124, per=25.21%, avg=41.59, stdev=20.78, samples=247 iops : min= 1, max= 31, avg= 9.85, stdev= 5.25, samples=247 lat (msec) : 50=0.04%, 100=0.12%, 250=0.78%, 500=29.26%, 750=23.20% lat (msec) : 1000=17.19%, 2000=29.14%, >=2000=0.27% cpu : usr=0.11%, sys=0.63%, ctx=4319, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145048: Fri Aug 28 05:58:18 2026 read: IOPS=9, BW=39.7KiB/s (40.6kB/s)(4984KiB/125601msec) slat (usec): min=55, max=64383, avg=16652.85, stdev=8916.41 clat (msec): min=14, max=968, avg=481.21, stdev=134.85 lat (msec): min=14, max=980, avg=497.87, stdev=136.37 clat percentiles (msec): | 1.00th=[ 127], 5.00th=[ 292], 10.00th=[ 326], 20.00th=[ 376], | 30.00th=[ 409], 40.00th=[ 443], 50.00th=[ 468], 60.00th=[ 502], | 70.00th=[ 542], 80.00th=[ 584], 90.00th=[ 651], 95.00th=[ 726], | 99.00th=[ 835], 99.50th=[ 902], 99.90th=[ 961], 99.95th=[ 969], | 99.99th=[ 969] bw ( KiB/s): min= 7, max= 151, per=25.26%, avg=40.16, stdev=19.80, samples=243 iops : min= 1, max= 37, avg= 9.49, stdev= 4.99, samples=243 write: IOPS=10, BW=41.8KiB/s (42.9kB/s)(5256KiB/125601msec); 0 zone resets slat (usec): min=35, max=15836, avg=335.89, stdev=965.87 clat (msec): min=159, max=2548, avg=1056.02, stdev=287.87 lat (msec): min=163, max=2548, avg=1056.36, stdev=287.85 clat percentiles (msec): | 1.00th=[ 401], 5.00th=[ 609], 10.00th=[ 726], 20.00th=[ 818], | 30.00th=[ 885], 40.00th=[ 961], 50.00th=[ 1045], 60.00th=[ 1116], | 70.00th=[ 1200], 80.00th=[ 1301], 90.00th=[ 1418], 95.00th=[ 1502], | 99.00th=[ 1737], 99.50th=[ 2056], 99.90th=[ 2534], 99.95th=[ 2534], | 99.99th=[ 2534] bw ( KiB/s): min= 7, max= 103, per=24.91%, avg=41.10, stdev=19.17, samples=248 iops : min= 1, max= 25, avg= 9.72, stdev= 4.84, samples=248 lat (msec) : 20=0.12%, 50=0.12%, 100=0.20%, 250=0.74%, 500=28.36% lat (msec) : 750=23.40%, 1000=18.52%, 2000=28.28%, >=2000=0.27% cpu : usr=0.10%, sys=0.61%, ctx=3937, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145049: Fri Aug 28 05:58:18 2026 read: IOPS=10, BW=40.3KiB/s (41.2kB/s)(5060KiB/125612msec) slat (usec): min=3040, max=74596, avg=16430.48, stdev=8889.51 clat (msec): min=30, max=947, avg=481.94, stdev=131.34 lat (msec): min=36, max=966, avg=498.39, stdev=132.74 clat percentiles (msec): | 1.00th=[ 142], 5.00th=[ 292], 10.00th=[ 334], 20.00th=[ 376], | 30.00th=[ 409], 40.00th=[ 447], 50.00th=[ 472], 60.00th=[ 502], | 70.00th=[ 542], 80.00th=[ 584], 90.00th=[ 651], 95.00th=[ 718], | 99.00th=[ 844], 99.50th=[ 919], 99.90th=[ 936], 99.95th=[ 944], | 99.99th=[ 944] bw ( KiB/s): min= 7, max= 135, per=25.87%, avg=41.14, stdev=20.12, samples=240 iops : min= 1, max= 33, avg= 9.73, stdev= 5.09, samples=240 write: IOPS=10, BW=41.2KiB/s (42.2kB/s)(5180KiB/125612msec); 0 zone resets slat (usec): min=34, max=39411, avg=410.56, stdev=1655.75 clat (msec): min=369, max=2487, avg=1061.49, stdev=280.67 lat (msec): min=369, max=2487, avg=1061.90, stdev=280.67 clat percentiles (msec): | 1.00th=[ 435], 5.00th=[ 625], 10.00th=[ 735], 20.00th=[ 835], | 30.00th=[ 911], 40.00th=[ 978], 50.00th=[ 1053], 60.00th=[ 1116], | 70.00th=[ 1200], 80.00th=[ 1301], 90.00th=[ 1401], 95.00th=[ 1502], | 99.00th=[ 1888], 99.50th=[ 2056], 99.90th=[ 2500], 99.95th=[ 2500], | 99.99th=[ 2500] bw ( KiB/s): min= 7, max= 119, per=24.67%, avg=40.71, stdev=19.44, samples=248 iops : min= 1, max= 29, avg= 9.62, stdev= 4.87, samples=248 lat (msec) : 50=0.04%, 100=0.35%, 250=0.70%, 500=29.10%, 750=23.28% lat (msec) : 1000=17.66%, 2000=28.59%, >=2000=0.27% cpu : usr=0.10%, sys=0.67%, ctx=4574, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=159KiB/s (163kB/s), 39.5KiB/s-40.3KiB/s (40.4kB/s-41.2kB/s), io=19.6MiB (20.5MB), run=125601-126081msec WRITE: bw=166KiB/s (170kB/s), 41.2KiB/s-41.8KiB/s (42.2kB/s-42.9kB/s), io=20.4MiB (21.4MB), run=125601-126081msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=145071: Fri Aug 28 05:58:29 2026 read: IOPS=1, BW=63.6MiB/s (66.7MB/s)(40.0MiB/629msec) slat (nsec): min=60395k, max=60395k, avg=60394608.00, stdev= 0.00 clat (nsec): min=527277k, max=527277k, avg=527277111.00, stdev= 0.00 lat (nsec): min=587676k, max=587676k, avg=587676144.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 527], 5.00th=[ 527], 10.00th=[ 527], 20.00th=[ 527], | 30.00th=[ 527], 40.00th=[ 527], 50.00th=[ 527], 60.00th=[ 527], | 70.00th=[ 527], 80.00th=[ 527], 90.00th=[ 527], 95.00th=[ 527], | 99.00th=[ 527], 99.50th=[ 527], 99.90th=[ 527], 99.95th=[ 527], | 99.99th=[ 527] lat (msec) : 750=100.00% cpu : usr=0.00%, sys=2.23%, ctx=37, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=63.6MiB/s (66.7MB/s), 63.6MiB/s-63.6MiB/s (66.7MB/s-66.7MB/s), io=40.0MiB (41.9MB), run=629-629msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout