== sanity test 398c: run fio to test AIO ================= 06:57:50 (1789383470) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.648848 s, 64.6 MB/s osc.lustre-OST0000-osc-ffff90df10366000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=144443: Mon Sep 14 07:01:05 2026 write: IOPS=13, BW=55.5KiB/s (56.9kB/s)(10.0MiB/184425msec); 0 zone resets slat (usec): min=28, max=20152, avg=184.84, stdev=835.62 clat (msec): min=165, max=4413, avg=1151.51, stdev=473.79 lat (msec): min=165, max=4413, avg=1151.69, stdev=473.77 clat percentiles (msec): | 1.00th=[ 464], 5.00th=[ 634], 10.00th=[ 709], 20.00th=[ 818], | 30.00th=[ 902], 40.00th=[ 969], 50.00th=[ 1045], 60.00th=[ 1133], | 70.00th=[ 1234], 80.00th=[ 1435], 90.00th=[ 1737], 95.00th=[ 2072], | 99.00th=[ 2635], 99.50th=[ 3842], 99.90th=[ 4245], 99.95th=[ 4279], | 99.99th=[ 4396] bw ( KiB/s): min= 7, max= 128, per=26.04%, avg=57.80, stdev=30.07, samples=348 iops : min= 1, max= 32, avg=13.99, stdev= 7.57, samples=348 lat (msec) : 250=0.62%, 500=0.90%, 750=12.11%, 1000=31.33%, 2000=49.61% lat (msec) : >=2000=5.43% cpu : usr=0.04%, sys=0.16%, ctx=1484, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=144444: Mon Sep 14 07:01:05 2026 write: IOPS=13, BW=55.6KiB/s (56.0kB/s)(10.0MiB/184059msec); 0 zone resets slat (usec): min=22, max=38384, avg=207.08, stdev=1204.46 clat (msec): min=166, max=4509, avg=1149.69, stdev=474.78 lat (msec): min=166, max=4509, avg=1149.90, stdev=474.85 clat percentiles (msec): | 1.00th=[ 426], 5.00th=[ 617], 10.00th=[ 718], 20.00th=[ 827], | 30.00th=[ 902], 40.00th=[ 969], 50.00th=[ 1036], 60.00th=[ 1133], | 70.00th=[ 1217], 80.00th=[ 1418], 90.00th=[ 1737], 95.00th=[ 2005], | 99.00th=[ 2635], 99.50th=[ 3876], 99.90th=[ 4396], 99.95th=[ 4530], | 99.99th=[ 4530] bw ( KiB/s): min= 7, max= 128, per=26.34%, avg=58.47, stdev=32.08, samples=344 iops : min= 1, max= 32, avg=14.15, stdev= 8.04, samples=344 lat (msec) : 250=0.62%, 500=0.66%, 750=10.27%, 1000=33.40%, 2000=49.92% lat (msec) : >=2000=5.12% cpu : usr=0.04%, sys=0.18%, ctx=1613, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=144445: Mon Sep 14 07:01:05 2026 write: IOPS=13, BW=55.5KiB/s (56.9kB/s)(10.0MiB/184349msec); 0 zone resets slat (usec): min=27, max=27673, avg=194.54, stdev=902.08 clat (msec): min=190, max=4584, avg=1151.06, stdev=479.61 lat (msec): min=193, max=4584, avg=1151.25, stdev=479.61 clat percentiles (msec): | 1.00th=[ 510], 5.00th=[ 651], 10.00th=[ 718], 20.00th=[ 818], | 30.00th=[ 894], 40.00th=[ 961], 50.00th=[ 1036], 60.00th=[ 1116], | 70.00th=[ 1234], 80.00th=[ 1418], 90.00th=[ 1754], 95.00th=[ 2072], | 99.00th=[ 2735], 99.50th=[ 3876], 99.90th=[ 4463], 99.95th=[ 4597], | 99.99th=[ 4597] bw ( KiB/s): min= 7, max= 128, per=26.27%, avg=58.32, stdev=30.52, samples=345 iops : min= 1, max= 32, avg=14.13, stdev= 7.67, samples=345 lat (msec) : 250=0.04%, 500=0.66%, 750=11.91%, 1000=32.50%, 2000=49.18% lat (msec) : >=2000=5.70% cpu : usr=0.05%, sys=0.17%, ctx=1748, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=144446: Mon Sep 14 07:01:05 2026 write: IOPS=13, BW=55.5KiB/s (56.9kB/s)(10.0MiB/184349msec); 0 zone resets slat (usec): min=21, max=21114, avg=160.71, stdev=661.90 clat (msec): min=138, max=4275, avg=1150.89, stdev=472.63 lat (msec): min=138, max=4275, avg=1151.05, stdev=472.66 clat percentiles (msec): | 1.00th=[ 489], 5.00th=[ 642], 10.00th=[ 709], 20.00th=[ 827], | 30.00th=[ 902], 40.00th=[ 969], 50.00th=[ 1045], 60.00th=[ 1116], | 70.00th=[ 1234], 80.00th=[ 1418], 90.00th=[ 1737], 95.00th=[ 2072], | 99.00th=[ 2668], 99.50th=[ 3876], 99.90th=[ 4144], 99.95th=[ 4279], | 99.99th=[ 4279] bw ( KiB/s): min= 7, max= 128, per=25.23%, avg=56.01, stdev=28.74, samples=359 iops : min= 1, max= 32, avg=13.54, stdev= 7.23, samples=359 lat (msec) : 250=0.04%, 500=1.41%, 750=11.68%, 1000=31.72%, 2000=49.41% lat (msec) : >=2000=5.74% cpu : usr=0.04%, sys=0.16%, ctx=1672, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=222KiB/s (227kB/s), 55.5KiB/s-55.6KiB/s (56.9kB/s-56.0kB/s), io=40.0MiB (41.9MB), run=184059-184425msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=144493: Mon Sep 14 07:03:04 2026 read: IOPS=11, BW=45.0KiB/s (47.1kB/s)(5048KiB/109850msec) slat (usec): min=3009, max=62642, avg=14291.62, stdev=7667.26 clat (msec): min=17, max=1101, avg=430.36, stdev=117.78 lat (msec): min=29, max=1109, avg=444.66, stdev=118.47 clat percentiles (msec): | 1.00th=[ 184], 5.00th=[ 279], 10.00th=[ 305], 20.00th=[ 342], | 30.00th=[ 368], 40.00th=[ 393], 50.00th=[ 414], 60.00th=[ 447], | 70.00th=[ 477], 80.00th=[ 518], 90.00th=[ 567], 95.00th=[ 625], | 99.00th=[ 802], 99.50th=[ 911], 99.90th=[ 1099], 99.95th=[ 1099], | 99.99th=[ 1099] bw ( KiB/s): min= 7, max= 119, per=25.40%, avg=46.23, stdev=20.27, samples=214 iops : min= 1, max= 29, avg=11.09, stdev= 5.11, samples=214 write: IOPS=11, BW=47.3KiB/s (48.4kB/s)(5192KiB/109850msec); 0 zone resets slat (usec): min=32, max=30126, avg=226.42, stdev=1096.04 clat (msec): min=202, max=1972, avg=918.92, stdev=254.35 lat (msec): min=202, max=1972, avg=919.15, stdev=254.37 clat percentiles (msec): | 1.00th=[ 447], 5.00th=[ 575], 10.00th=[ 642], 20.00th=[ 718], | 30.00th=[ 785], 40.00th=[ 827], 50.00th=[ 869], 60.00th=[ 936], | 70.00th=[ 1003], 80.00th=[ 1099], 90.00th=[ 1267], 95.00th=[ 1401], | 99.00th=[ 1687], 99.50th=[ 1770], 99.90th=[ 1938], 99.95th=[ 1972], | 99.99th=[ 1972] bw ( KiB/s): min= 7, max= 95, per=24.78%, avg=46.83, stdev=19.13, samples=217 iops : min= 1, max= 23, avg=11.24, stdev= 4.81, samples=217 lat (msec) : 20=0.04%, 50=0.04%, 100=0.12%, 250=1.25%, 500=37.11% lat (msec) : 750=22.66%, 1000=23.12%, 2000=15.66% cpu : usr=0.10%, sys=0.56%, ctx=4131, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=144494: Mon Sep 14 07:03:04 2026 read: IOPS=11, BW=45.2KiB/s (46.3kB/s)(4976KiB/110079msec) slat (usec): min=54, max=54091, avg=14204.35, stdev=7682.66 clat (msec): min=10, max=988, avg=431.41, stdev=116.00 lat (msec): min=11, max=994, avg=445.62, stdev=117.02 clat percentiles (msec): | 1.00th=[ 215], 5.00th=[ 275], 10.00th=[ 305], 20.00th=[ 338], | 30.00th=[ 363], 40.00th=[ 393], 50.00th=[ 418], 60.00th=[ 447], | 70.00th=[ 481], 80.00th=[ 518], 90.00th=[ 575], 95.00th=[ 634], | 99.00th=[ 785], 99.50th=[ 802], 99.90th=[ 944], 99.95th=[ 986], | 99.99th=[ 986] bw ( KiB/s): min= 7, max= 120, per=24.66%, avg=44.88, stdev=21.14, samples=217 iops : min= 1, max= 30, avg=10.75, stdev= 5.34, samples=217 write: IOPS=11, BW=47.8KiB/s (48.0kB/s)(5264KiB/110079msec); 0 zone resets slat (usec): min=32, max=13765, avg=232.68, stdev=711.23 clat (msec): min=136, max=1973, avg=916.07, stdev=254.99 lat (msec): min=137, max=1973, avg=916.30, stdev=254.99 clat percentiles (msec): | 1.00th=[ 347], 5.00th=[ 575], 10.00th=[ 642], 20.00th=[ 718], | 30.00th=[ 776], 40.00th=[ 835], 50.00th=[ 877], 60.00th=[ 936], | 70.00th=[ 1003], 80.00th=[ 1116], 90.00th=[ 1267], 95.00th=[ 1368], | 99.00th=[ 1603], 99.50th=[ 1754], 99.90th=[ 1955], 99.95th=[ 1972], | 99.99th=[ 1972] bw ( KiB/s): min= 7, max= 120, per=24.98%, avg=47.22, stdev=20.96, samples=218 iops : min= 1, max= 30, avg=11.33, stdev= 5.23, samples=218 lat (msec) : 20=0.08%, 50=0.08%, 100=0.16%, 250=1.41%, 500=36.09% lat (msec) : 750=22.93%, 1000=23.79%, 2000=15.47% cpu : usr=0.07%, sys=0.54%, ctx=4625, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=144495: Mon Sep 14 07:03:04 2026 read: IOPS=11, BW=45.3KiB/s (46.4kB/s)(4984KiB/110046msec) slat (usec): min=2653, max=55350, avg=14180.44, stdev=7740.08 clat (msec): min=16, max=966, avg=422.19, stdev=114.23 lat (msec): min=23, max=979, avg=436.38, stdev=115.20 clat percentiles (msec): | 1.00th=[ 146], 5.00th=[ 271], 10.00th=[ 305], 20.00th=[ 334], | 30.00th=[ 359], 40.00th=[ 384], 50.00th=[ 409], 60.00th=[ 439], | 70.00th=[ 468], 80.00th=[ 510], 90.00th=[ 558], 95.00th=[ 625], | 99.00th=[ 751], 99.50th=[ 835], 99.90th=[ 944], 99.95th=[ 969], | 99.99th=[ 969] bw ( KiB/s): min= 7, max= 112, per=25.17%, avg=45.80, stdev=20.82, samples=214 iops : min= 1, max= 28, avg=10.97, stdev= 5.23, samples=214 write: IOPS=11, BW=47.8KiB/s (48.9kB/s)(5256KiB/110046msec); 0 zone resets slat (usec): min=31, max=9240, avg=194.93, stdev=541.86 clat (msec): min=257, max=1907, avg=923.75, stdev=248.26 lat (msec): min=257, max=1907, avg=923.94, stdev=248.25 clat percentiles (msec): | 1.00th=[ 451], 5.00th=[ 617], 10.00th=[ 659], 20.00th=[ 726], | 30.00th=[ 785], 40.00th=[ 827], 50.00th=[ 885], 60.00th=[ 936], | 70.00th=[ 986], 80.00th=[ 1099], 90.00th=[ 1267], 95.00th=[ 1385], | 99.00th=[ 1687], 99.50th=[ 1888], 99.90th=[ 1905], 99.95th=[ 1905], | 99.99th=[ 1905] bw ( KiB/s): min= 7, max= 112, per=25.01%, avg=47.26, stdev=19.35, samples=217 iops : min= 1, max= 28, avg=11.35, stdev= 4.86, samples=217 lat (msec) : 20=0.04%, 50=0.12%, 100=0.16%, 250=1.13%, 500=37.50% lat (msec) : 750=20.98%, 1000=25.12%, 2000=14.96% cpu : usr=0.11%, sys=0.52%, ctx=4314, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=144496: Mon Sep 14 07:03:04 2026 read: IOPS=11, BW=45.0KiB/s (47.1kB/s)(5060KiB/110051msec) slat (msec): min=3, max=425, avg=14.40, stdev=14.16 clat (msec): min=18, max=974, avg=427.23, stdev=125.95 lat (msec): min=32, max=990, avg=441.64, stdev=126.65 clat percentiles (msec): | 1.00th=[ 104], 5.00th=[ 264], 10.00th=[ 292], 20.00th=[ 330], | 30.00th=[ 363], 40.00th=[ 384], 50.00th=[ 409], 60.00th=[ 447], | 70.00th=[ 477], 80.00th=[ 518], 90.00th=[ 584], 95.00th=[ 667], | 99.00th=[ 802], 99.50th=[ 852], 99.90th=[ 953], 99.95th=[ 978], | 99.99th=[ 978] bw ( KiB/s): min= 7, max= 126, per=25.40%, avg=46.23, stdev=20.33, samples=215 iops : min= 1, max= 31, avg=11.08, stdev= 5.13, samples=215 write: IOPS=11, BW=47.1KiB/s (48.2kB/s)(5180KiB/110051msec); 0 zone resets slat (usec): min=27, max=20329, avg=227.77, stdev=945.78 clat (msec): min=238, max=1898, avg=925.65, stdev=242.78 lat (msec): min=239, max=1898, avg=925.88, stdev=242.82 clat percentiles (msec): | 1.00th=[ 451], 5.00th=[ 584], 10.00th=[ 667], 20.00th=[ 735], | 30.00th=[ 793], 40.00th=[ 835], 50.00th=[ 885], 60.00th=[ 944], | 70.00th=[ 1020], 80.00th=[ 1116], 90.00th=[ 1267], 95.00th=[ 1401], | 99.00th=[ 1586], 99.50th=[ 1670], 99.90th=[ 1838], 99.95th=[ 1905], | 99.99th=[ 1905] bw ( KiB/s): min= 7, max= 103, per=24.55%, avg=46.39, stdev=19.05, samples=218 iops : min= 1, max= 25, avg=11.12, stdev= 4.77, samples=218 lat (msec) : 20=0.04%, 50=0.12%, 100=0.27%, 250=1.29%, 500=36.68% lat (msec) : 750=21.37%, 1000=24.18%, 2000=16.05% cpu : usr=0.09%, sys=0.51%, ctx=4373, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=182KiB/s (187kB/s), 45.2KiB/s-45.0KiB/s (46.3kB/s-47.1kB/s), io=19.6MiB (20.5MB), run=109850-110079msec WRITE: bw=190KiB/s (194kB/s), 47.1KiB/s-47.8KiB/s (48.2kB/s-48.0kB/s), io=20.4MiB (21.4MB), run=109850-110079msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=144516: Mon Sep 14 07:03:14 2026 read: IOPS=2, BW=80.8MiB/s (84.7MB/s)(40.0MiB/495msec) slat (nsec): min=30269k, max=30269k, avg=30268730.00, stdev= 0.00 clat (nsec): min=439409k, max=439409k, avg=439408766.00, stdev= 0.00 lat (nsec): min=469682k, max=469682k, avg=469682124.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 439], 5.00th=[ 439], 10.00th=[ 439], 20.00th=[ 439], | 30.00th=[ 439], 40.00th=[ 439], 50.00th=[ 439], 60.00th=[ 439], | 70.00th=[ 439], 80.00th=[ 439], 90.00th=[ 439], 95.00th=[ 439], | 99.00th=[ 439], 99.50th=[ 439], 99.90th=[ 439], 99.95th=[ 439], | 99.99th=[ 439] lat (msec) : 500=100.00% cpu : usr=0.20%, sys=2.43%, ctx=4, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=80.8MiB/s (84.7MB/s), 80.8MiB/s-80.8MiB/s (84.7MB/s-84.7MB/s), io=40.0MiB (41.9MB), run=495-495msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout