== sanity test 398c: run fio to test AIO ================= 01:14:02 (1790745242) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.4279 s, 98.0 MB/s osc.lustre-OST0000-osc-ffff90c4cfa27800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=145452: Wed Sep 30 01:16:53 2026 write: IOPS=15, BW=63.8KiB/s (65.4kB/s)(10.0MiB/160447msec); 0 zone resets slat (usec): min=30, max=20920, avg=318.65, stdev=1136.42 clat (msec): min=261, max=2127, avg=1001.45, stdev=258.23 lat (msec): min=261, max=2127, avg=1001.77, stdev=258.25 clat percentiles (msec): | 1.00th=[ 376], 5.00th=[ 567], 10.00th=[ 676], 20.00th=[ 802], | 30.00th=[ 877], 40.00th=[ 961], 50.00th=[ 1011], 60.00th=[ 1070], | 70.00th=[ 1116], 80.00th=[ 1200], 90.00th=[ 1318], 95.00th=[ 1401], | 99.00th=[ 1720], 99.50th=[ 1821], 99.90th=[ 2089], 99.95th=[ 2123], | 99.99th=[ 2123] bw ( KiB/s): min= 7, max= 135, per=24.80%, avg=63.24, stdev=29.29, samples=318 iops : min= 1, max= 33, avg=15.31, stdev= 7.33, samples=318 lat (msec) : 500=3.40%, 750=11.84%, 1000=32.54%, 2000=51.95%, >=2000=0.27% cpu : usr=0.06%, sys=0.27%, ctx=1846, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145453: Wed Sep 30 01:16:53 2026 write: IOPS=15, BW=63.9KiB/s (65.4kB/s)(10.0MiB/160284msec); 0 zone resets slat (usec): min=25, max=16104, avg=314.41, stdev=1100.71 clat (msec): min=144, max=2128, avg=1000.68, stdev=265.04 lat (msec): min=144, max=2128, avg=1001.00, stdev=265.02 clat percentiles (msec): | 1.00th=[ 321], 5.00th=[ 567], 10.00th=[ 676], 20.00th=[ 785], | 30.00th=[ 877], 40.00th=[ 953], 50.00th=[ 1003], 60.00th=[ 1070], | 70.00th=[ 1133], 80.00th=[ 1200], 90.00th=[ 1318], 95.00th=[ 1401], | 99.00th=[ 1737], 99.50th=[ 1871], 99.90th=[ 2056], 99.95th=[ 2072], | 99.99th=[ 2123] bw ( KiB/s): min= 8, max= 183, per=24.96%, avg=63.64, stdev=25.90, samples=316 iops : min= 2, max= 45, avg=15.41, stdev= 6.49, samples=316 lat (msec) : 250=0.39%, 500=2.85%, 750=12.27%, 1000=33.20%, 2000=51.05% lat (msec) : >=2000=0.23% cpu : usr=0.05%, sys=0.25%, ctx=1802, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145454: Wed Sep 30 01:16:53 2026 write: IOPS=15, BW=63.0KiB/s (65.5kB/s)(10.0MiB/160040msec); 0 zone resets slat (usec): min=28, max=20921, avg=318.48, stdev=1156.15 clat (msec): min=157, max=2145, avg=999.51, stdev=270.63 lat (msec): min=157, max=2145, avg=999.84, stdev=270.65 clat percentiles (msec): | 1.00th=[ 300], 5.00th=[ 531], 10.00th=[ 667], 20.00th=[ 793], | 30.00th=[ 877], 40.00th=[ 953], 50.00th=[ 1003], 60.00th=[ 1070], | 70.00th=[ 1133], 80.00th=[ 1217], 90.00th=[ 1334], 95.00th=[ 1401], | 99.00th=[ 1670], 99.50th=[ 1871], 99.90th=[ 2123], 99.95th=[ 2140], | 99.99th=[ 2140] bw ( KiB/s): min= 7, max= 219, per=24.78%, avg=63.19, stdev=27.02, samples=318 iops : min= 1, max= 54, avg=15.30, stdev= 6.77, samples=318 lat (msec) : 250=0.70%, 500=3.48%, 750=11.80%, 1000=32.93%, 2000=50.82% lat (msec) : >=2000=0.27% cpu : usr=0.06%, sys=0.26%, ctx=1770, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145455: Wed Sep 30 01:16:53 2026 write: IOPS=15, BW=63.8KiB/s (65.3kB/s)(10.0MiB/160457msec); 0 zone resets slat (usec): min=27, max=19593, avg=332.91, stdev=1197.86 clat (msec): min=214, max=2128, avg=1001.80, stdev=258.64 lat (msec): min=215, max=2128, avg=1002.13, stdev=258.67 clat percentiles (msec): | 1.00th=[ 321], 5.00th=[ 567], 10.00th=[ 676], 20.00th=[ 802], | 30.00th=[ 885], 40.00th=[ 953], 50.00th=[ 1003], 60.00th=[ 1070], | 70.00th=[ 1133], 80.00th=[ 1200], 90.00th=[ 1318], 95.00th=[ 1401], | 99.00th=[ 1670], 99.50th=[ 1804], 99.90th=[ 2089], 99.95th=[ 2123], | 99.99th=[ 2123] bw ( KiB/s): min= 7, max= 159, per=24.81%, avg=63.25, stdev=28.51, samples=318 iops : min= 1, max= 39, avg=15.31, stdev= 7.14, samples=318 lat (msec) : 250=0.08%, 500=3.63%, 750=11.33%, 1000=34.02%, 2000=50.78% lat (msec) : >=2000=0.16% cpu : usr=0.06%, sys=0.29%, ctx=1955, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=255KiB/s (261kB/s), 63.8KiB/s-63.0KiB/s (65.3kB/s-65.5kB/s), io=40.0MiB (41.9MB), run=160040-160457msec 8% of I/O are 1-page (PAGE_SIZE=4096), max 50% mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=145488: Wed Sep 30 01:18:54 2026 read: IOPS=11, BW=44.7KiB/s (45.8kB/s)(5048KiB/112933msec) slat (usec): min=3195, max=75114, avg=15166.97, stdev=8039.64 clat (msec): min=28, max=1087, avg=449.21, stdev=124.41 lat (msec): min=51, max=1114, avg=464.38, stdev=125.56 clat percentiles (msec): | 1.00th=[ 130], 5.00th=[ 275], 10.00th=[ 313], 20.00th=[ 355], | 30.00th=[ 384], 40.00th=[ 414], 50.00th=[ 439], 60.00th=[ 472], | 70.00th=[ 502], 80.00th=[ 542], 90.00th=[ 600], 95.00th=[ 667], | 99.00th=[ 793], 99.50th=[ 911], 99.90th=[ 986], 99.95th=[ 1083], | 99.99th=[ 1083] bw ( KiB/s): min= 7, max= 126, per=25.14%, avg=44.50, stdev=21.74, samples=222 iops : min= 1, max= 31, avg=10.63, stdev= 5.43, samples=222 write: IOPS=11, BW=45.0KiB/s (47.1kB/s)(5192KiB/112933msec); 0 zone resets slat (usec): min=32, max=19110, avg=298.20, stdev=950.53 clat (msec): min=317, max=2297, avg=937.15, stdev=312.28 lat (msec): min=317, max=2298, avg=937.45, stdev=312.28 clat percentiles (msec): | 1.00th=[ 376], 5.00th=[ 489], 10.00th=[ 558], 20.00th=[ 676], | 30.00th=[ 776], 40.00th=[ 852], 50.00th=[ 919], 60.00th=[ 995], | 70.00th=[ 1062], 80.00th=[ 1150], 90.00th=[ 1284], 95.00th=[ 1452], | 99.00th=[ 2005], 99.50th=[ 2123], 99.90th=[ 2165], 99.95th=[ 2299], | 99.99th=[ 2299] bw ( KiB/s): min= 7, max= 119, per=25.23%, avg=46.42, stdev=21.09, samples=219 iops : min= 1, max= 29, avg=11.10, stdev= 5.28, samples=219 lat (msec) : 50=0.16%, 100=0.16%, 250=1.41%, 500=35.27%, 750=25.43% lat (msec) : 1000=17.85%, 2000=19.14%, >=2000=0.59% cpu : usr=0.11%, sys=0.61%, ctx=3973, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145489: Wed Sep 30 01:18:54 2026 read: IOPS=11, BW=44.2KiB/s (45.3kB/s)(4976KiB/112536msec) slat (usec): min=51, max=56087, avg=15080.81, stdev=8235.78 clat (msec): min=8, max=1101, avg=445.61, stdev=122.17 lat (msec): min=11, max=1118, avg=460.70, stdev=123.50 clat percentiles (msec): | 1.00th=[ 184], 5.00th=[ 271], 10.00th=[ 309], 20.00th=[ 347], | 30.00th=[ 380], 40.00th=[ 409], 50.00th=[ 439], 60.00th=[ 468], | 70.00th=[ 498], 80.00th=[ 535], 90.00th=[ 600], 95.00th=[ 651], | 99.00th=[ 776], 99.50th=[ 835], 99.90th=[ 1036], 99.95th=[ 1099], | 99.99th=[ 1099] bw ( KiB/s): min= 7, max= 110, per=24.88%, avg=44.05, stdev=20.81, samples=221 iops : min= 1, max= 27, avg=10.51, stdev= 5.20, samples=221 write: IOPS=11, BW=46.8KiB/s (47.9kB/s)(5264KiB/112536msec); 0 zone resets slat (usec): min=31, max=8319, avg=280.70, stdev=694.02 clat (msec): min=161, max=2272, avg=931.71, stdev=318.37 lat (msec): min=161, max=2272, avg=932.00, stdev=318.34 clat percentiles (msec): | 1.00th=[ 321], 5.00th=[ 472], 10.00th=[ 550], 20.00th=[ 667], | 30.00th=[ 776], 40.00th=[ 844], 50.00th=[ 919], 60.00th=[ 986], | 70.00th=[ 1053], 80.00th=[ 1133], 90.00th=[ 1284], 95.00th=[ 1519], | 99.00th=[ 1989], 99.50th=[ 2056], 99.90th=[ 2106], 99.95th=[ 2265], | 99.99th=[ 2265] bw ( KiB/s): min= 7, max= 111, per=25.18%, avg=46.33, stdev=23.52, samples=222 iops : min= 1, max= 27, avg=11.07, stdev= 5.90, samples=222 lat (msec) : 10=0.04%, 20=0.08%, 50=0.08%, 100=0.16%, 250=1.56% lat (msec) : 500=35.82%, 750=24.02%, 1000=18.67%, 2000=19.18%, >=2000=0.39% cpu : usr=0.11%, sys=0.65%, ctx=4262, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145490: Wed Sep 30 01:18:54 2026 read: IOPS=11, BW=44.1KiB/s (45.1kB/s)(4984KiB/113104msec) slat (usec): min=3322, max=60636, avg=15113.20, stdev=8341.39 clat (msec): min=17, max=1040, avg=442.25, stdev=117.24 lat (msec): min=29, max=1058, avg=457.37, stdev=118.58 clat percentiles (msec): | 1.00th=[ 180], 5.00th=[ 271], 10.00th=[ 313], 20.00th=[ 355], | 30.00th=[ 384], 40.00th=[ 409], 50.00th=[ 430], 60.00th=[ 460], | 70.00th=[ 493], 80.00th=[ 527], 90.00th=[ 592], 95.00th=[ 651], | 99.00th=[ 768], 99.50th=[ 802], 99.90th=[ 1036], 99.95th=[ 1036], | 99.99th=[ 1036] bw ( KiB/s): min= 7, max= 143, per=25.33%, avg=44.83, stdev=22.06, samples=217 iops : min= 1, max= 35, avg=10.70, stdev= 5.54, samples=217 write: IOPS=11, BW=46.5KiB/s (47.6kB/s)(5256KiB/113104msec); 0 zone resets slat (usec): min=31, max=16452, avg=293.19, stdev=869.93 clat (msec): min=227, max=2178, avg=940.81, stdev=321.46 lat (msec): min=228, max=2178, avg=941.11, stdev=321.46 clat percentiles (msec): | 1.00th=[ 351], 5.00th=[ 481], 10.00th=[ 558], 20.00th=[ 676], | 30.00th=[ 776], 40.00th=[ 852], 50.00th=[ 927], 60.00th=[ 995], | 70.00th=[ 1062], 80.00th=[ 1150], 90.00th=[ 1301], 95.00th=[ 1536], | 99.00th=[ 2039], 99.50th=[ 2106], 99.90th=[ 2165], 99.95th=[ 2165], | 99.99th=[ 2165] bw ( KiB/s): min= 7, max= 143, per=25.03%, avg=46.05, stdev=20.15, samples=222 iops : min= 1, max= 35, avg=11.01, stdev= 5.06, samples=222 lat (msec) : 20=0.08%, 50=0.08%, 100=0.16%, 250=1.41%, 500=36.17% lat (msec) : 750=24.34%, 1000=17.58%, 2000=19.57%, >=2000=0.62% cpu : usr=0.12%, sys=0.64%, ctx=4184, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145491: Wed Sep 30 01:18:54 2026 read: IOPS=11, BW=44.9KiB/s (45.0kB/s)(5060KiB/112720msec) slat (usec): min=3050, max=61123, avg=15257.79, stdev=8389.49 clat (msec): min=15, max=1179, avg=446.01, stdev=122.78 lat (msec): min=23, max=1190, avg=461.28, stdev=124.34 clat percentiles (msec): | 1.00th=[ 171], 5.00th=[ 284], 10.00th=[ 309], 20.00th=[ 351], | 30.00th=[ 384], 40.00th=[ 414], 50.00th=[ 435], 60.00th=[ 464], | 70.00th=[ 498], 80.00th=[ 531], 90.00th=[ 609], 95.00th=[ 651], | 99.00th=[ 818], 99.50th=[ 885], 99.90th=[ 1036], 99.95th=[ 1183], | 99.99th=[ 1183] bw ( KiB/s): min= 7, max= 103, per=25.88%, avg=45.81, stdev=21.52, samples=216 iops : min= 1, max= 25, avg=10.94, stdev= 5.39, samples=216 write: IOPS=11, BW=45.0KiB/s (47.1kB/s)(5180KiB/112720msec); 0 zone resets slat (usec): min=33, max=9050, avg=267.43, stdev=621.22 clat (msec): min=224, max=2319, avg=939.43, stdev=312.91 lat (msec): min=227, max=2319, avg=939.70, stdev=312.91 clat percentiles (msec): | 1.00th=[ 388], 5.00th=[ 498], 10.00th=[ 575], 20.00th=[ 693], | 30.00th=[ 785], 40.00th=[ 835], 50.00th=[ 919], 60.00th=[ 986], | 70.00th=[ 1045], 80.00th=[ 1133], 90.00th=[ 1284], 95.00th=[ 1502], | 99.00th=[ 2056], 99.50th=[ 2106], 99.90th=[ 2265], 99.95th=[ 2333], | 99.99th=[ 2333] bw ( KiB/s): min= 7, max= 111, per=24.60%, avg=45.27, stdev=20.26, samples=224 iops : min= 1, max= 27, avg=10.81, stdev= 5.09, samples=224 lat (msec) : 20=0.12%, 50=0.04%, 100=0.31%, 250=1.09%, 500=36.13% lat (msec) : 750=23.98%, 1000=19.18%, 2000=18.48%, >=2000=0.66% cpu : usr=0.10%, sys=0.63%, ctx=4380, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=177KiB/s (182kB/s), 44.1KiB/s-44.9KiB/s (45.1kB/s-45.0kB/s), io=19.6MiB (20.5MB), run=112536-113104msec WRITE: bw=185KiB/s (189kB/s), 45.0KiB/s-46.8KiB/s (47.1kB/s-47.9kB/s), io=20.4MiB (21.4MB), run=112536-113104msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=145512: Wed Sep 30 01:19:06 2026 read: IOPS=1, BW=76.8MiB/s (80.5MB/s)(40.0MiB/521msec) slat (nsec): min=39192k, max=39192k, avg=39192009.00, stdev= 0.00 clat (nsec): min=453093k, max=453093k, avg=453092704.00, stdev= 0.00 lat (nsec): min=492290k, max=492290k, avg=492290499.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 456], 5.00th=[ 456], 10.00th=[ 456], 20.00th=[ 456], | 30.00th=[ 456], 40.00th=[ 456], 50.00th=[ 456], 60.00th=[ 456], | 70.00th=[ 456], 80.00th=[ 456], 90.00th=[ 456], 95.00th=[ 456], | 99.00th=[ 456], 99.50th=[ 456], 99.90th=[ 456], 99.95th=[ 456], | 99.99th=[ 456] lat (msec) : 500=100.00% cpu : usr=0.19%, sys=2.50%, ctx=4, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=76.8MiB/s (80.5MB/s), 76.8MiB/s-76.8MiB/s (80.5MB/s-80.5MB/s), io=40.0MiB (41.9MB), run=521-521msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout