== sanity test 398c: run fio to test AIO ================= 14:01:52 (1780336912) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.513588 s, 81.7 MB/s osc.lustre-OST0000-osc-ffff8b8ba0cb0800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=147263: Mon Jun 1 14:05:04 2026 write: IOPS=14, BW=56.6KiB/s (57.9kB/s)(10.0MiB/181050msec); 0 zone resets slat (usec): min=48, max=23058, avg=360.49, stdev=1177.23 clat (msec): min=119, max=2192, avg=1130.46, stdev=285.17 lat (msec): min=119, max=2192, avg=1130.83, stdev=285.18 clat percentiles (msec): | 1.00th=[ 288], 5.00th=[ 718], 10.00th=[ 818], 20.00th=[ 927], | 30.00th=[ 995], 40.00th=[ 1053], 50.00th=[ 1116], 60.00th=[ 1183], | 70.00th=[ 1267], 80.00th=[ 1351], 90.00th=[ 1485], 95.00th=[ 1603], | 99.00th=[ 1905], 99.50th=[ 2022], 99.90th=[ 2198], 99.95th=[ 2198], | 99.99th=[ 2198] bw ( KiB/s): min= 7, max= 182, per=24.97%, avg=56.44, stdev=27.43, samples=355 iops : min= 1, max= 45, avg=13.64, stdev= 6.90, samples=355 lat (msec) : 250=0.90%, 500=1.33%, 750=3.52%, 1000=24.96%, 2000=68.75% lat (msec) : >=2000=0.55% cpu : usr=0.03%, sys=0.29%, ctx=2036, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=147264: Mon Jun 1 14:05:04 2026 write: IOPS=14, BW=56.5KiB/s (57.9kB/s)(10.0MiB/181193msec); 0 zone resets slat (usec): min=45, max=21640, avg=334.48, stdev=1131.15 clat (msec): min=253, max=2183, avg=1130.86, stdev=275.50 lat (msec): min=253, max=2184, avg=1131.20, stdev=275.51 clat percentiles (msec): | 1.00th=[ 279], 5.00th=[ 726], 10.00th=[ 827], 20.00th=[ 927], | 30.00th=[ 995], 40.00th=[ 1045], 50.00th=[ 1116], 60.00th=[ 1183], | 70.00th=[ 1267], 80.00th=[ 1334], 90.00th=[ 1452], 95.00th=[ 1603], | 99.00th=[ 1821], 99.50th=[ 2022], 99.90th=[ 2072], 99.95th=[ 2198], | 99.99th=[ 2198] bw ( KiB/s): min= 7, max= 140, per=24.72%, avg=55.87, stdev=23.90, samples=359 iops : min= 1, max= 35, avg=13.50, stdev= 5.99, samples=359 lat (msec) : 500=2.34%, 750=3.32%, 1000=25.66%, 2000=68.12%, >=2000=0.55% cpu : usr=0.03%, sys=0.26%, ctx=1924, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=147265: Mon Jun 1 14:05:04 2026 write: IOPS=14, BW=56.5KiB/s (57.9kB/s)(10.0MiB/181198msec); 0 zone resets slat (usec): min=42, max=26546, avg=325.80, stdev=1222.17 clat (msec): min=212, max=2181, avg=1131.02, stdev=273.84 lat (msec): min=214, max=2181, avg=1131.34, stdev=273.88 clat percentiles (msec): | 1.00th=[ 338], 5.00th=[ 743], 10.00th=[ 844], 20.00th=[ 927], | 30.00th=[ 1003], 40.00th=[ 1045], 50.00th=[ 1099], 60.00th=[ 1183], | 70.00th=[ 1267], 80.00th=[ 1334], 90.00th=[ 1469], 95.00th=[ 1586], | 99.00th=[ 1888], 99.50th=[ 1972], 99.90th=[ 2072], 99.95th=[ 2198], | 99.99th=[ 2198] bw ( KiB/s): min= 7, max= 128, per=24.79%, avg=56.02, stdev=27.86, samples=358 iops : min= 1, max= 32, avg=13.54, stdev= 6.99, samples=358 lat (msec) : 250=0.16%, 500=1.84%, 750=3.20%, 1000=24.77%, 2000=69.65% lat (msec) : >=2000=0.39% cpu : usr=0.04%, sys=0.25%, ctx=1835, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=147266: Mon Jun 1 14:05:04 2026 write: IOPS=14, BW=56.5KiB/s (57.9kB/s)(10.0MiB/181147msec); 0 zone resets slat (usec): min=49, max=28400, avg=325.69, stdev=1108.50 clat (msec): min=217, max=2191, avg=1130.51, stdev=276.57 lat (msec): min=217, max=2192, avg=1130.84, stdev=276.59 clat percentiles (msec): | 1.00th=[ 279], 5.00th=[ 718], 10.00th=[ 818], 20.00th=[ 927], | 30.00th=[ 1003], 40.00th=[ 1053], 50.00th=[ 1116], 60.00th=[ 1183], | 70.00th=[ 1250], 80.00th=[ 1351], 90.00th=[ 1469], 95.00th=[ 1586], | 99.00th=[ 1888], 99.50th=[ 1955], 99.90th=[ 2072], 99.95th=[ 2140], | 99.99th=[ 2198] bw ( KiB/s): min= 7, max= 145, per=24.79%, avg=56.02, stdev=24.60, samples=358 iops : min= 1, max= 36, avg=13.54, stdev= 6.16, samples=358 lat (msec) : 250=0.74%, 500=1.13%, 750=4.41%, 1000=24.06%, 2000=69.30% lat (msec) : >=2000=0.35% cpu : usr=0.05%, sys=0.25%, ctx=1884, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=226KiB/s (231kB/s), 56.5KiB/s-56.6KiB/s (57.9kB/s-57.9kB/s), io=40.0MiB (41.9MB), run=181050-181198msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=147303: Mon Jun 1 14:07:11 2026 read: IOPS=10, BW=42.7KiB/s (43.7kB/s)(5048KiB/118292msec) slat (usec): min=3601, max=72020, avg=15960.41, stdev=8082.84 clat (msec): min=35, max=1037, avg=465.73, stdev=111.30 lat (msec): min=44, max=1058, avg=481.71, stdev=112.30 clat percentiles (msec): | 1.00th=[ 218], 5.00th=[ 309], 10.00th=[ 338], 20.00th=[ 384], | 30.00th=[ 409], 40.00th=[ 430], 50.00th=[ 460], 60.00th=[ 489], | 70.00th=[ 514], 80.00th=[ 542], 90.00th=[ 609], 95.00th=[ 651], | 99.00th=[ 776], 99.50th=[ 835], 99.90th=[ 1011], 99.95th=[ 1036], | 99.99th=[ 1036] bw ( KiB/s): min= 7, max= 127, per=25.20%, avg=42.33, stdev=19.24, samples=233 iops : min= 1, max= 31, avg=10.03, stdev= 4.85, samples=233 write: IOPS=10, BW=43.9KiB/s (44.9kB/s)(5192KiB/118292msec); 0 zone resets slat (usec): min=46, max=18170, avg=439.62, stdev=1117.72 clat (msec): min=325, max=1889, avg=986.20, stdev=278.32 lat (msec): min=325, max=1889, avg=986.65, stdev=278.26 clat percentiles (msec): | 1.00th=[ 426], 5.00th=[ 558], 10.00th=[ 625], 20.00th=[ 735], | 30.00th=[ 810], 40.00th=[ 894], 50.00th=[ 995], 60.00th=[ 1062], | 70.00th=[ 1133], 80.00th=[ 1217], 90.00th=[ 1351], 95.00th=[ 1469], | 99.00th=[ 1653], 99.50th=[ 1754], 99.90th=[ 1838], 99.95th=[ 1888], | 99.99th=[ 1888] bw ( KiB/s): min= 6, max= 112, per=24.74%, avg=43.29, stdev=18.42, samples=234 iops : min= 1, max= 28, avg=10.27, stdev= 4.65, samples=234 lat (msec) : 50=0.04%, 100=0.23%, 250=0.51%, 500=32.50%, 750=27.30% lat (msec) : 1000=14.49%, 2000=24.92% cpu : usr=0.09%, sys=0.73%, ctx=4437, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=147304: Mon Jun 1 14:07:11 2026 read: IOPS=10, BW=41.9KiB/s (42.9kB/s)(4976KiB/118775msec) slat (usec): min=85, max=78807, avg=16049.32, stdev=8568.41 clat (msec): min=11, max=928, avg=461.20, stdev=111.84 lat (msec): min=11, max=970, avg=477.26, stdev=112.86 clat percentiles (msec): | 1.00th=[ 213], 5.00th=[ 300], 10.00th=[ 334], 20.00th=[ 380], | 30.00th=[ 401], 40.00th=[ 426], 50.00th=[ 451], 60.00th=[ 481], | 70.00th=[ 506], 80.00th=[ 550], 90.00th=[ 600], 95.00th=[ 659], | 99.00th=[ 776], 99.50th=[ 877], 99.90th=[ 911], 99.95th=[ 927], | 99.99th=[ 927] bw ( KiB/s): min= 7, max= 96, per=24.85%, avg=41.75, stdev=19.48, samples=232 iops : min= 1, max= 24, avg= 9.89, stdev= 4.91, samples=232 write: IOPS=11, BW=44.3KiB/s (45.4kB/s)(5264KiB/118775msec); 0 zone resets slat (usec): min=56, max=12128, avg=460.93, stdev=1033.02 clat (msec): min=112, max=1897, avg=991.59, stdev=281.71 lat (msec): min=112, max=1898, avg=992.05, stdev=281.68 clat percentiles (msec): | 1.00th=[ 405], 5.00th=[ 535], 10.00th=[ 625], 20.00th=[ 743], | 30.00th=[ 835], 40.00th=[ 927], 50.00th=[ 986], 60.00th=[ 1070], | 70.00th=[ 1150], 80.00th=[ 1234], 90.00th=[ 1351], 95.00th=[ 1452], | 99.00th=[ 1670], 99.50th=[ 1787], 99.90th=[ 1871], 99.95th=[ 1905], | 99.99th=[ 1905] bw ( KiB/s): min= 7, max= 119, per=25.03%, avg=43.80, stdev=20.34, samples=235 iops : min= 1, max= 29, avg=10.41, stdev= 5.15, samples=235 lat (msec) : 20=0.08%, 50=0.04%, 100=0.20%, 250=0.78%, 500=33.52% lat (msec) : 750=23.83%, 1000=16.68%, 2000=24.88% cpu : usr=0.10%, sys=0.71%, ctx=4628, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=147305: Mon Jun 1 14:07:11 2026 read: IOPS=10, BW=42.1KiB/s (43.1kB/s)(4984KiB/118498msec) slat (usec): min=3603, max=56923, avg=16016.71, stdev=8270.13 clat (msec): min=15, max=912, avg=460.78, stdev=113.08 lat (msec): min=26, max=929, avg=476.82, stdev=114.05 clat percentiles (msec): | 1.00th=[ 236], 5.00th=[ 292], 10.00th=[ 334], 20.00th=[ 368], | 30.00th=[ 401], 40.00th=[ 430], 50.00th=[ 451], 60.00th=[ 481], | 70.00th=[ 506], 80.00th=[ 550], 90.00th=[ 600], 95.00th=[ 659], | 99.00th=[ 768], 99.50th=[ 869], 99.90th=[ 911], 99.95th=[ 911], | 99.99th=[ 911] bw ( KiB/s): min= 7, max= 135, per=24.97%, avg=41.95, stdev=20.44, samples=233 iops : min= 1, max= 33, avg= 9.94, stdev= 5.09, samples=233 write: IOPS=11, BW=44.4KiB/s (45.4kB/s)(5256KiB/118498msec); 0 zone resets slat (usec): min=55, max=11034, avg=440.46, stdev=1020.25 clat (msec): min=314, max=1858, avg=988.08, stdev=273.13 lat (msec): min=314, max=1858, avg=988.53, stdev=273.13 clat percentiles (msec): | 1.00th=[ 447], 5.00th=[ 531], 10.00th=[ 617], 20.00th=[ 743], | 30.00th=[ 835], 40.00th=[ 911], 50.00th=[ 986], 60.00th=[ 1062], | 70.00th=[ 1133], 80.00th=[ 1217], 90.00th=[ 1351], 95.00th=[ 1452], | 99.00th=[ 1586], 99.50th=[ 1636], 99.90th=[ 1720], 99.95th=[ 1854], | 99.99th=[ 1854] bw ( KiB/s): min= 7, max= 112, per=24.94%, avg=43.64, stdev=20.34, samples=234 iops : min= 1, max= 28, avg=10.36, stdev= 5.18, samples=234 lat (msec) : 20=0.04%, 50=0.08%, 100=0.08%, 250=0.47%, 500=33.48% lat (msec) : 750=24.69%, 1000=16.13%, 2000=25.04% cpu : usr=0.09%, sys=0.71%, ctx=4556, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=147306: Mon Jun 1 14:07:11 2026 read: IOPS=10, BW=42.8KiB/s (43.8kB/s)(5060KiB/118352msec) slat (usec): min=3663, max=66911, avg=15889.93, stdev=8288.05 clat (msec): min=28, max=940, avg=459.80, stdev=106.92 lat (msec): min=35, max=949, avg=475.69, stdev=108.17 clat percentiles (msec): | 1.00th=[ 199], 5.00th=[ 305], 10.00th=[ 338], 20.00th=[ 376], | 30.00th=[ 405], 40.00th=[ 430], 50.00th=[ 456], 60.00th=[ 481], | 70.00th=[ 506], 80.00th=[ 542], 90.00th=[ 592], 95.00th=[ 642], | 99.00th=[ 751], 99.50th=[ 793], 99.90th=[ 919], 99.95th=[ 944], | 99.99th=[ 944] bw ( KiB/s): min= 7, max= 128, per=25.65%, avg=43.09, stdev=18.94, samples=230 iops : min= 1, max= 32, avg=10.23, stdev= 4.76, samples=230 write: IOPS=10, BW=43.8KiB/s (44.8kB/s)(5180KiB/118352msec); 0 zone resets slat (usec): min=58, max=20277, avg=525.60, stdev=1519.73 clat (msec): min=230, max=1802, avg=994.68, stdev=282.45 lat (msec): min=230, max=1802, avg=995.21, stdev=282.54 clat percentiles (msec): | 1.00th=[ 418], 5.00th=[ 535], 10.00th=[ 609], 20.00th=[ 751], | 30.00th=[ 835], 40.00th=[ 911], 50.00th=[ 995], 60.00th=[ 1070], | 70.00th=[ 1150], 80.00th=[ 1234], 90.00th=[ 1385], 95.00th=[ 1469], | 99.00th=[ 1636], 99.50th=[ 1687], 99.90th=[ 1770], 99.95th=[ 1804], | 99.99th=[ 1804] bw ( KiB/s): min= 7, max= 111, per=24.49%, avg=42.86, stdev=18.88, samples=235 iops : min= 1, max= 27, avg=10.17, stdev= 4.79, samples=235 lat (msec) : 50=0.08%, 100=0.12%, 250=0.78%, 500=34.18%, 750=23.87% lat (msec) : 1000=16.09%, 2000=24.88% cpu : usr=0.10%, sys=0.74%, ctx=4659, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=169KiB/s (173kB/s), 41.9KiB/s-42.8KiB/s (42.9kB/s-43.8kB/s), io=19.6MiB (20.5MB), run=118292-118775msec WRITE: bw=176KiB/s (180kB/s), 43.8KiB/s-44.4KiB/s (44.8kB/s-45.4kB/s), io=20.4MiB (21.4MB), run=118292-118775msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=147329: Mon Jun 1 14:07:20 2026 read: IOPS=1, BW=63.8MiB/s (66.9MB/s)(40.0MiB/627msec) slat (nsec): min=56185k, max=56185k, avg=56185316.00, stdev= 0.00 clat (nsec): min=531770k, max=531770k, avg=531769675.00, stdev= 0.00 lat (nsec): min=587959k, max=587959k, avg=587959369.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 531], 5.00th=[ 531], 10.00th=[ 531], 20.00th=[ 531], | 30.00th=[ 531], 40.00th=[ 531], 50.00th=[ 531], 60.00th=[ 531], | 70.00th=[ 531], 80.00th=[ 531], 90.00th=[ 531], 95.00th=[ 531], | 99.00th=[ 531], 99.50th=[ 531], 99.90th=[ 531], 99.95th=[ 531], | 99.99th=[ 531] lat (msec) : 750=100.00% cpu : usr=0.00%, sys=1.76%, ctx=7, majf=0, minf=9 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=63.8MiB/s (66.9MB/s), 63.8MiB/s-63.8MiB/s (66.9MB/s-66.9MB/s), io=40.0MiB (41.9MB), run=627-627msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout