== sanity test 398c: run fio to test AIO ================= 05:46:50 (1769078810) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.520944 s, 80.5 MB/s osc.lustre-OST0000-osc-ffff8c0382c99800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=147508: Thu Jan 22 05:50:11 2026 write: IOPS=13, BW=53.0KiB/s (55.3kB/s)(10.0MiB/189747msec); 0 zone resets slat (usec): min=22, max=21183, avg=217.27, stdev=1117.74 clat (msec): min=162, max=2827, avg=1185.10, stdev=387.37 lat (msec): min=162, max=2827, avg=1185.32, stdev=387.38 clat percentiles (msec): | 1.00th=[ 236], 5.00th=[ 600], 10.00th=[ 760], 20.00th=[ 885], | 30.00th=[ 961], 40.00th=[ 1070], 50.00th=[ 1167], 60.00th=[ 1250], | 70.00th=[ 1334], 80.00th=[ 1485], 90.00th=[ 1703], 95.00th=[ 1871], | 99.00th=[ 2165], 99.50th=[ 2366], 99.90th=[ 2702], 99.95th=[ 2702], | 99.99th=[ 2836] bw ( KiB/s): min= 7, max= 255, per=25.73%, avg=55.32, stdev=34.73, samples=363 iops : min= 1, max= 63, avg=13.35, stdev= 8.71, samples=363 lat (msec) : 250=1.88%, 500=1.41%, 750=6.52%, 1000=23.63%, 2000=64.10% lat (msec) : >=2000=2.46% cpu : usr=0.04%, sys=0.20%, ctx=1609, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=147509: Thu Jan 22 05:50:11 2026 write: IOPS=13, BW=53.8KiB/s (55.1kB/s)(10.0MiB/190188msec); 0 zone resets slat (usec): min=21, max=18211, avg=207.35, stdev=927.55 clat (msec): min=88, max=2759, avg=1187.19, stdev=381.97 lat (msec): min=88, max=2759, avg=1187.40, stdev=381.94 clat percentiles (msec): | 1.00th=[ 259], 5.00th=[ 584], 10.00th=[ 760], 20.00th=[ 885], | 30.00th=[ 978], 40.00th=[ 1083], 50.00th=[ 1167], 60.00th=[ 1250], | 70.00th=[ 1334], 80.00th=[ 1469], 90.00th=[ 1703], 95.00th=[ 1871], | 99.00th=[ 2232], 99.50th=[ 2467], 99.90th=[ 2567], 99.95th=[ 2567], | 99.99th=[ 2769] bw ( KiB/s): min= 7, max= 191, per=25.19%, avg=54.15, stdev=29.07, samples=371 iops : min= 1, max= 47, avg=13.05, stdev= 7.31, samples=371 lat (msec) : 100=0.04%, 250=0.90%, 500=2.50%, 750=5.98%, 1000=23.01% lat (msec) : 2000=65.66%, >=2000=1.91% cpu : usr=0.04%, sys=0.19%, ctx=1763, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=147510: Thu Jan 22 05:50:11 2026 write: IOPS=13, BW=53.8KiB/s (55.1kB/s)(10.0MiB/190189msec); 0 zone resets slat (usec): min=23, max=16122, avg=200.32, stdev=846.07 clat (msec): min=178, max=2564, avg=1187.37, stdev=374.42 lat (msec): min=178, max=2564, avg=1187.57, stdev=374.41 clat percentiles (msec): | 1.00th=[ 275], 5.00th=[ 609], 10.00th=[ 751], 20.00th=[ 885], | 30.00th=[ 978], 40.00th=[ 1070], 50.00th=[ 1167], 60.00th=[ 1250], | 70.00th=[ 1334], 80.00th=[ 1469], 90.00th=[ 1703], 95.00th=[ 1871], | 99.00th=[ 2140], 99.50th=[ 2265], 99.90th=[ 2500], 99.95th=[ 2500], | 99.99th=[ 2567] bw ( KiB/s): min= 7, max= 255, per=25.68%, avg=55.21, stdev=30.78, samples=364 iops : min= 1, max= 63, avg=13.32, stdev= 7.72, samples=364 lat (msec) : 250=0.62%, 500=2.07%, 750=6.80%, 1000=24.06%, 2000=64.26% lat (msec) : >=2000=2.19% cpu : usr=0.03%, sys=0.18%, ctx=1652, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=147511: Thu Jan 22 05:50:11 2026 write: IOPS=13, BW=53.8KiB/s (55.1kB/s)(10.0MiB/190182msec); 0 zone resets slat (usec): min=21, max=32075, avg=192.17, stdev=975.02 clat (msec): min=186, max=2551, avg=1187.27, stdev=370.86 lat (msec): min=186, max=2551, avg=1187.46, stdev=370.86 clat percentiles (msec): | 1.00th=[ 279], 5.00th=[ 609], 10.00th=[ 760], 20.00th=[ 902], | 30.00th=[ 978], 40.00th=[ 1083], 50.00th=[ 1167], 60.00th=[ 1250], | 70.00th=[ 1334], 80.00th=[ 1469], 90.00th=[ 1687], 95.00th=[ 1838], | 99.00th=[ 2198], 99.50th=[ 2366], 99.90th=[ 2467], 99.95th=[ 2567], | 99.99th=[ 2567] bw ( KiB/s): min= 7, max= 255, per=25.54%, avg=54.91, stdev=30.07, samples=366 iops : min= 1, max= 63, avg=13.24, stdev= 7.52, samples=366 lat (msec) : 250=0.62%, 500=1.84%, 750=6.99%, 1000=23.01%, 2000=65.43% lat (msec) : >=2000=2.11% cpu : usr=0.04%, sys=0.19%, ctx=1705, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=215KiB/s (221kB/s), 53.8KiB/s-53.0KiB/s (55.1kB/s-55.3kB/s), io=40.0MiB (41.9MB), run=189747-190189msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=147549: Thu Jan 22 05:52:20 2026 read: IOPS=10, BW=41.6KiB/s (42.6kB/s)(5048KiB/121286msec) slat (usec): min=2781, max=68624, avg=15814.79, stdev=8857.30 clat (msec): min=35, max=1145, avg=469.76, stdev=123.03 lat (msec): min=45, max=1180, avg=485.59, stdev=124.44 clat percentiles (msec): | 1.00th=[ 213], 5.00th=[ 300], 10.00th=[ 334], 20.00th=[ 376], | 30.00th=[ 405], 40.00th=[ 435], 50.00th=[ 464], 60.00th=[ 485], | 70.00th=[ 514], 80.00th=[ 550], 90.00th=[ 625], 95.00th=[ 693], | 99.00th=[ 818], 99.50th=[ 869], 99.90th=[ 1116], 99.95th=[ 1150], | 99.99th=[ 1150] bw ( KiB/s): min= 7, max= 120, per=25.03%, avg=41.30, stdev=19.59, samples=238 iops : min= 1, max= 30, avg= 9.81, stdev= 4.91, samples=238 write: IOPS=10, BW=42.8KiB/s (43.8kB/s)(5192KiB/121286msec); 0 zone resets slat (usec): min=30, max=17871, avg=263.84, stdev=1049.90 clat (msec): min=356, max=1971, avg=1020.47, stdev=274.56 lat (msec): min=357, max=1971, avg=1020.73, stdev=274.48 clat percentiles (msec): | 1.00th=[ 477], 5.00th=[ 600], 10.00th=[ 684], 20.00th=[ 802], | 30.00th=[ 877], 40.00th=[ 936], 50.00th=[ 986], 60.00th=[ 1053], | 70.00th=[ 1133], 80.00th=[ 1234], 90.00th=[ 1385], 95.00th=[ 1502], | 99.00th=[ 1838], 99.50th=[ 1854], 99.90th=[ 1905], 99.95th=[ 1972], | 99.99th=[ 1972] bw ( KiB/s): min= 7, max= 104, per=24.72%, avg=42.52, stdev=19.06, samples=238 iops : min= 1, max= 26, avg=10.11, stdev= 4.79, samples=238 lat (msec) : 50=0.08%, 100=0.16%, 250=0.62%, 500=32.58%, 750=21.91% lat (msec) : 1000=20.20%, 2000=24.45% cpu : usr=0.10%, sys=0.61%, ctx=4409, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=147550: Thu Jan 22 05:52:20 2026 read: IOPS=10, BW=41.1KiB/s (42.1kB/s)(4976KiB/121106msec) slat (usec): min=2989, max=77811, avg=16456.14, stdev=9416.49 clat (msec): min=19, max=1084, avg=475.89, stdev=129.63 lat (msec): min=24, max=1101, avg=492.35, stdev=131.31 clat percentiles (msec): | 1.00th=[ 222], 5.00th=[ 300], 10.00th=[ 334], 20.00th=[ 372], | 30.00th=[ 405], 40.00th=[ 439], 50.00th=[ 464], 60.00th=[ 489], | 70.00th=[ 523], 80.00th=[ 575], 90.00th=[ 651], 95.00th=[ 709], | 99.00th=[ 844], 99.50th=[ 894], 99.90th=[ 1070], 99.95th=[ 1083], | 99.99th=[ 1083] bw ( KiB/s): min= 7, max= 111, per=24.81%, avg=40.94, stdev=19.27, samples=237 iops : min= 1, max= 27, avg= 9.72, stdev= 4.82, samples=237 write: IOPS=10, BW=43.5KiB/s (44.5kB/s)(5264KiB/121106msec); 0 zone resets slat (usec): min=29, max=16633, avg=276.12, stdev=1015.38 clat (msec): min=293, max=1972, avg=1004.82, stdev=264.68 lat (msec): min=293, max=1972, avg=1005.11, stdev=264.67 clat percentiles (msec): | 1.00th=[ 451], 5.00th=[ 600], 10.00th=[ 651], 20.00th=[ 793], | 30.00th=[ 869], 40.00th=[ 936], 50.00th=[ 995], 60.00th=[ 1062], | 70.00th=[ 1116], 80.00th=[ 1200], 90.00th=[ 1334], 95.00th=[ 1469], | 99.00th=[ 1787], 99.50th=[ 1871], 99.90th=[ 1905], 99.95th=[ 1972], | 99.99th=[ 1972] bw ( KiB/s): min= 7, max= 111, per=24.91%, avg=42.85, stdev=19.09, samples=240 iops : min= 1, max= 27, avg=10.20, stdev= 4.80, samples=240 lat (msec) : 20=0.04%, 50=0.04%, 100=0.16%, 250=0.59%, 500=31.02% lat (msec) : 750=23.48%, 1000=19.22%, 2000=25.47% cpu : usr=0.12%, sys=0.62%, ctx=4989, majf=0, minf=16 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=147551: Thu Jan 22 05:52:20 2026 read: IOPS=10, BW=41.3KiB/s (42.3kB/s)(4984KiB/120744msec) slat (usec): min=39, max=70232, avg=16035.46, stdev=9034.18 clat (msec): min=14, max=1110, avg=468.31, stdev=130.60 lat (msec): min=15, max=1150, avg=484.35, stdev=132.35 clat percentiles (msec): | 1.00th=[ 194], 5.00th=[ 288], 10.00th=[ 317], 20.00th=[ 372], | 30.00th=[ 401], 40.00th=[ 430], 50.00th=[ 456], 60.00th=[ 485], | 70.00th=[ 514], 80.00th=[ 558], 90.00th=[ 634], 95.00th=[ 701], | 99.00th=[ 835], 99.50th=[ 894], 99.90th=[ 1099], 99.95th=[ 1116], | 99.99th=[ 1116] bw ( KiB/s): min= 7, max= 103, per=24.77%, avg=40.87, stdev=19.66, samples=239 iops : min= 1, max= 25, avg= 9.71, stdev= 4.92, samples=239 write: IOPS=10, BW=43.5KiB/s (44.6kB/s)(5256KiB/120744msec); 0 zone resets slat (usec): min=29, max=14308, avg=223.58, stdev=687.72 clat (msec): min=117, max=2003, avg=1009.84, stdev=284.13 lat (msec): min=117, max=2003, avg=1010.06, stdev=284.15 clat percentiles (msec): | 1.00th=[ 363], 5.00th=[ 584], 10.00th=[ 642], 20.00th=[ 793], | 30.00th=[ 877], 40.00th=[ 936], 50.00th=[ 986], 60.00th=[ 1053], | 70.00th=[ 1133], 80.00th=[ 1217], 90.00th=[ 1351], 95.00th=[ 1502], | 99.00th=[ 1888], 99.50th=[ 1921], 99.90th=[ 2005], 99.95th=[ 2005], | 99.99th=[ 2005] bw ( KiB/s): min= 7, max= 135, per=25.14%, avg=43.25, stdev=22.12, samples=236 iops : min= 1, max= 33, avg=10.30, stdev= 5.56, samples=236 lat (msec) : 20=0.04%, 50=0.08%, 100=0.31%, 250=0.90%, 500=31.64% lat (msec) : 750=22.38%, 1000=19.65%, 2000=24.96%, >=2000=0.04% cpu : usr=0.11%, sys=0.61%, ctx=4904, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=147552: Thu Jan 22 05:52:20 2026 read: IOPS=10, BW=41.7KiB/s (42.7kB/s)(5060KiB/121229msec) slat (usec): min=2964, max=70744, avg=15893.52, stdev=9031.17 clat (msec): min=11, max=1109, avg=474.14, stdev=131.67 lat (msec): min=18, max=1150, avg=490.04, stdev=132.79 clat percentiles (msec): | 1.00th=[ 222], 5.00th=[ 296], 10.00th=[ 330], 20.00th=[ 372], | 30.00th=[ 405], 40.00th=[ 435], 50.00th=[ 460], 60.00th=[ 485], | 70.00th=[ 518], 80.00th=[ 575], 90.00th=[ 642], 95.00th=[ 709], | 99.00th=[ 860], 99.50th=[ 919], 99.90th=[ 969], 99.95th=[ 1116], | 99.99th=[ 1116] bw ( KiB/s): min= 7, max= 95, per=25.36%, avg=41.84, stdev=19.62, samples=236 iops : min= 1, max= 23, avg= 9.95, stdev= 4.92, samples=236 write: IOPS=10, BW=42.7KiB/s (43.8kB/s)(5180KiB/121229msec); 0 zone resets slat (usec): min=29, max=21494, avg=283.62, stdev=1268.13 clat (msec): min=389, max=1999, avg=1016.21, stdev=263.47 lat (msec): min=389, max=1999, avg=1016.50, stdev=263.48 clat percentiles (msec): | 1.00th=[ 485], 5.00th=[ 609], 10.00th=[ 676], 20.00th=[ 810], | 30.00th=[ 877], 40.00th=[ 944], 50.00th=[ 1003], 60.00th=[ 1053], | 70.00th=[ 1133], 80.00th=[ 1217], 90.00th=[ 1351], 95.00th=[ 1469], | 99.00th=[ 1821], 99.50th=[ 1854], 99.90th=[ 1938], 99.95th=[ 2005], | 99.99th=[ 2005] bw ( KiB/s): min= 7, max= 103, per=24.94%, avg=42.89, stdev=18.82, samples=236 iops : min= 1, max= 25, avg=10.22, stdev= 4.70, samples=236 lat (msec) : 20=0.08%, 50=0.04%, 100=0.31%, 250=0.43%, 500=31.76% lat (msec) : 750=22.70%, 1000=18.95%, 2000=25.74% cpu : usr=0.10%, sys=0.55%, ctx=4721, majf=0, minf=16 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=165KiB/s (169kB/s), 41.1KiB/s-41.7KiB/s (42.1kB/s-42.7kB/s), io=19.6MiB (20.5MB), run=120744-121286msec WRITE: bw=172KiB/s (176kB/s), 42.7KiB/s-43.5KiB/s (43.8kB/s-44.6kB/s), io=20.4MiB (21.4MB), run=120744-121286msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=147573: Thu Jan 22 05:52:30 2026 read: IOPS=1, BW=67.5MiB/s (70.7MB/s)(40.0MiB/593msec) slat (nsec): min=31357k, max=31357k, avg=31356581.00, stdev= 0.00 clat (nsec): min=542059k, max=542059k, avg=542059138.00, stdev= 0.00 lat (nsec): min=573420k, max=573420k, avg=573420107.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 542], 5.00th=[ 542], 10.00th=[ 542], 20.00th=[ 542], | 30.00th=[ 542], 40.00th=[ 542], 50.00th=[ 542], 60.00th=[ 542], | 70.00th=[ 542], 80.00th=[ 542], 90.00th=[ 542], 95.00th=[ 542], | 99.00th=[ 542], 99.50th=[ 542], 99.90th=[ 542], 99.95th=[ 542], | 99.99th=[ 542] lat (msec) : 750=100.00% cpu : usr=0.00%, sys=2.03%, ctx=28, majf=0, minf=7 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=67.5MiB/s (70.7MB/s), 67.5MiB/s-67.5MiB/s (70.7MB/s-70.7MB/s), io=40.0MiB (41.9MB), run=593-593msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout