== sanity test 398c: run fio to test AIO ================= 05:07:16 (1780996036) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.760768 s, 55.1 MB/s osc.lustre-OST0000-osc-ffff8cb9b1f4f000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=145569: Tue Jun 9 05:10:38 2026 write: IOPS=13, BW=54.2KiB/s (55.5kB/s)(10.0MiB/189075msec); 0 zone resets slat (usec): min=39, max=18418, avg=403.12, stdev=1222.48 clat (msec): min=301, max=2393, avg=1180.29, stdev=340.16 lat (msec): min=301, max=2393, avg=1180.70, stdev=340.16 clat percentiles (msec): | 1.00th=[ 338], 5.00th=[ 651], 10.00th=[ 760], 20.00th=[ 911], | 30.00th=[ 1003], 40.00th=[ 1083], 50.00th=[ 1167], 60.00th=[ 1250], | 70.00th=[ 1334], 80.00th=[ 1435], 90.00th=[ 1586], 95.00th=[ 1770], | 99.00th=[ 2106], 99.50th=[ 2165], 99.90th=[ 2366], 99.95th=[ 2366], | 99.99th=[ 2400] bw ( KiB/s): min= 7, max= 247, per=24.99%, avg=53.98, stdev=27.44, samples=372 iops : min= 1, max= 61, avg=12.95, stdev= 6.88, samples=372 lat (msec) : 500=2.89%, 750=6.41%, 1000=20.08%, 2000=68.75%, >=2000=1.88% cpu : usr=0.05%, sys=0.29%, ctx=2186, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145570: Tue Jun 9 05:10:38 2026 write: IOPS=13, BW=54.2KiB/s (55.5kB/s)(10.0MiB/189002msec); 0 zone resets slat (usec): min=41, max=24456, avg=388.22, stdev=1237.60 clat (msec): min=129, max=2513, avg=1179.82, stdev=343.71 lat (msec): min=129, max=2513, avg=1180.21, stdev=343.71 clat percentiles (msec): | 1.00th=[ 338], 5.00th=[ 634], 10.00th=[ 785], 20.00th=[ 919], | 30.00th=[ 995], 40.00th=[ 1070], 50.00th=[ 1150], 60.00th=[ 1250], | 70.00th=[ 1351], 80.00th=[ 1435], 90.00th=[ 1586], 95.00th=[ 1804], | 99.00th=[ 2089], 99.50th=[ 2232], 99.90th=[ 2500], 99.95th=[ 2500], | 99.99th=[ 2500] bw ( KiB/s): min= 7, max= 255, per=25.33%, avg=54.71, stdev=30.76, samples=367 iops : min= 1, max= 63, avg=13.13, stdev= 7.70, samples=367 lat (msec) : 250=0.08%, 500=2.85%, 750=5.94%, 1000=21.60%, 2000=67.81% lat (msec) : >=2000=1.72% cpu : usr=0.05%, sys=0.31%, ctx=2123, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145571: Tue Jun 9 05:10:38 2026 write: IOPS=13, BW=54.3KiB/s (55.6kB/s)(10.0MiB/188585msec); 0 zone resets slat (usec): min=51, max=21834, avg=377.40, stdev=1208.62 clat (msec): min=151, max=2392, avg=1177.76, stdev=345.16 lat (msec): min=152, max=2392, avg=1178.14, stdev=345.19 clat percentiles (msec): | 1.00th=[ 342], 5.00th=[ 642], 10.00th=[ 751], 20.00th=[ 911], | 30.00th=[ 1011], 40.00th=[ 1083], 50.00th=[ 1167], 60.00th=[ 1250], | 70.00th=[ 1334], 80.00th=[ 1452], 90.00th=[ 1586], 95.00th=[ 1787], | 99.00th=[ 2123], 99.50th=[ 2165], 99.90th=[ 2366], 99.95th=[ 2366], | 99.99th=[ 2400] bw ( KiB/s): min= 7, max= 207, per=25.25%, avg=54.55, stdev=25.88, samples=368 iops : min= 1, max= 51, avg=13.09, stdev= 6.47, samples=368 lat (msec) : 250=0.31%, 500=2.34%, 750=7.42%, 1000=18.83%, 2000=69.18% lat (msec) : >=2000=1.91% cpu : usr=0.04%, sys=0.29%, ctx=2095, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145572: Tue Jun 9 05:10:38 2026 write: IOPS=13, BW=54.2KiB/s (55.5kB/s)(10.0MiB/189032msec); 0 zone resets slat (usec): min=42, max=19141, avg=366.99, stdev=1115.24 clat (msec): min=304, max=2547, avg=1179.83, stdev=345.65 lat (msec): min=304, max=2548, avg=1180.20, stdev=345.67 clat percentiles (msec): | 1.00th=[ 351], 5.00th=[ 642], 10.00th=[ 760], 20.00th=[ 911], | 30.00th=[ 1003], 40.00th=[ 1083], 50.00th=[ 1150], 60.00th=[ 1250], | 70.00th=[ 1334], 80.00th=[ 1452], 90.00th=[ 1603], 95.00th=[ 1804], | 99.00th=[ 2123], 99.50th=[ 2165], 99.90th=[ 2534], 99.95th=[ 2534], | 99.99th=[ 2534] bw ( KiB/s): min= 7, max= 254, per=25.05%, avg=54.11, stdev=27.91, samples=371 iops : min= 1, max= 63, avg=12.98, stdev= 7.00, samples=371 lat (msec) : 500=2.93%, 750=6.45%, 1000=20.12%, 2000=68.79%, >=2000=1.72% cpu : usr=0.04%, sys=0.29%, ctx=2187, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=217KiB/s (222kB/s), 54.2KiB/s-54.3KiB/s (55.5kB/s-55.6kB/s), io=40.0MiB (41.9MB), run=188585-189075msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=145608: Tue Jun 9 05:13:02 2026 read: IOPS=9, BW=37.9KiB/s (38.8kB/s)(5048KiB/133259msec) slat (usec): min=3422, max=80161, avg=17660.79, stdev=9391.78 clat (msec): min=29, max=1673, avg=519.92, stdev=154.11 lat (msec): min=55, max=1688, avg=537.59, stdev=155.04 clat percentiles (msec): | 1.00th=[ 228], 5.00th=[ 313], 10.00th=[ 355], 20.00th=[ 405], | 30.00th=[ 443], 40.00th=[ 472], 50.00th=[ 506], 60.00th=[ 531], | 70.00th=[ 567], 80.00th=[ 617], 90.00th=[ 709], 95.00th=[ 793], | 99.00th=[ 961], 99.50th=[ 995], 99.90th=[ 1552], 99.95th=[ 1670], | 99.99th=[ 1670] bw ( KiB/s): min= 7, max= 103, per=25.25%, avg=37.88, stdev=19.00, samples=260 iops : min= 1, max= 25, avg= 8.93, stdev= 4.82, samples=260 write: IOPS=9, BW=38.0KiB/s (39.9kB/s)(5192KiB/133259msec); 0 zone resets slat (usec): min=59, max=27143, avg=517.21, stdev=1634.41 clat (msec): min=342, max=2626, avg=1115.35, stdev=335.85 lat (msec): min=342, max=2626, avg=1115.87, stdev=335.86 clat percentiles (msec): | 1.00th=[ 481], 5.00th=[ 642], 10.00th=[ 726], 20.00th=[ 844], | 30.00th=[ 927], 40.00th=[ 1003], 50.00th=[ 1070], 60.00th=[ 1150], | 70.00th=[ 1250], 80.00th=[ 1351], 90.00th=[ 1536], 95.00th=[ 1720], | 99.00th=[ 2232], 99.50th=[ 2299], 99.90th=[ 2601], 99.95th=[ 2635], | 99.99th=[ 2635] bw ( KiB/s): min= 5, max= 110, per=24.81%, avg=38.71, stdev=20.54, samples=261 iops : min= 1, max= 27, avg= 9.14, stdev= 5.13, samples=261 lat (msec) : 50=0.04%, 100=0.08%, 250=0.66%, 500=23.79%, 750=26.72% lat (msec) : 1000=18.09%, 2000=29.49%, >=2000=1.13% cpu : usr=0.08%, sys=0.72%, ctx=4955, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145609: Tue Jun 9 05:13:02 2026 read: IOPS=9, BW=37.3KiB/s (38.2kB/s)(4976KiB/133354msec) slat (usec): min=149, max=81058, avg=17884.19, stdev=9617.29 clat (msec): min=40, max=1354, avg=520.13, stdev=155.90 lat (msec): min=49, max=1363, avg=538.03, stdev=157.96 clat percentiles (msec): | 1.00th=[ 247], 5.00th=[ 305], 10.00th=[ 355], 20.00th=[ 397], | 30.00th=[ 439], 40.00th=[ 468], 50.00th=[ 498], 60.00th=[ 531], | 70.00th=[ 575], 80.00th=[ 625], 90.00th=[ 726], 95.00th=[ 818], | 99.00th=[ 1003], 99.50th=[ 1028], 99.90th=[ 1133], 99.95th=[ 1351], | 99.99th=[ 1351] bw ( KiB/s): min= 7, max= 110, per=25.18%, avg=37.76, stdev=18.26, samples=257 iops : min= 1, max= 27, avg= 8.90, stdev= 4.58, samples=257 write: IOPS=9, BW=39.5KiB/s (40.4kB/s)(5264KiB/133354msec); 0 zone resets slat (usec): min=60, max=20057, avg=460.25, stdev=1313.49 clat (msec): min=184, max=2425, avg=1110.88, stdev=328.49 lat (msec): min=185, max=2425, avg=1111.35, stdev=328.44 clat percentiles (msec): | 1.00th=[ 435], 5.00th=[ 659], 10.00th=[ 751], 20.00th=[ 852], | 30.00th=[ 927], 40.00th=[ 995], 50.00th=[ 1070], 60.00th=[ 1150], | 70.00th=[ 1234], 80.00th=[ 1351], 90.00th=[ 1519], 95.00th=[ 1703], | 99.00th=[ 2089], 99.50th=[ 2198], 99.90th=[ 2366], 99.95th=[ 2433], | 99.99th=[ 2433] bw ( KiB/s): min= 7, max= 103, per=25.01%, avg=39.02, stdev=19.26, samples=262 iops : min= 1, max= 25, avg= 9.22, stdev= 4.84, samples=262 lat (msec) : 50=0.08%, 100=0.08%, 250=0.59%, 500=24.77%, 750=24.06% lat (msec) : 1000=19.53%, 2000=29.96%, >=2000=0.94% cpu : usr=0.10%, sys=0.69%, ctx=4940, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145610: Tue Jun 9 05:13:02 2026 read: IOPS=9, BW=37.4KiB/s (38.3kB/s)(4984KiB/133349msec) slat (usec): min=3064, max=64980, avg=17515.01, stdev=9483.66 clat (msec): min=29, max=1117, avg=515.91, stdev=146.53 lat (msec): min=59, max=1152, avg=533.44, stdev=148.02 clat percentiles (msec): | 1.00th=[ 228], 5.00th=[ 313], 10.00th=[ 355], 20.00th=[ 401], | 30.00th=[ 439], 40.00th=[ 464], 50.00th=[ 502], 60.00th=[ 531], | 70.00th=[ 575], 80.00th=[ 625], 90.00th=[ 701], 95.00th=[ 785], | 99.00th=[ 953], 99.50th=[ 986], 99.90th=[ 1116], 99.95th=[ 1116], | 99.99th=[ 1116] bw ( KiB/s): min= 7, max= 88, per=25.18%, avg=37.78, stdev=18.77, samples=258 iops : min= 1, max= 22, avg= 8.91, stdev= 4.78, samples=258 write: IOPS=9, BW=39.4KiB/s (40.4kB/s)(5256KiB/133349msec); 0 zone resets slat (usec): min=54, max=26350, avg=475.41, stdev=1301.23 clat (msec): min=200, max=2682, avg=1114.10, stdev=329.03 lat (msec): min=209, max=2682, avg=1114.58, stdev=328.91 clat percentiles (msec): | 1.00th=[ 447], 5.00th=[ 667], 10.00th=[ 743], 20.00th=[ 860], | 30.00th=[ 936], 40.00th=[ 1003], 50.00th=[ 1070], 60.00th=[ 1150], | 70.00th=[ 1234], 80.00th=[ 1351], 90.00th=[ 1519], 95.00th=[ 1720], | 99.00th=[ 2198], 99.50th=[ 2366], 99.90th=[ 2601], 99.95th=[ 2668], | 99.99th=[ 2668] bw ( KiB/s): min= 7, max= 95, per=25.06%, avg=39.09, stdev=18.69, samples=261 iops : min= 1, max= 23, avg= 9.24, stdev= 4.68, samples=261 lat (msec) : 50=0.08%, 100=0.16%, 250=0.59%, 500=24.30%, 750=25.62% lat (msec) : 1000=17.93%, 2000=30.12%, >=2000=1.21% cpu : usr=0.09%, sys=0.69%, ctx=4857, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145611: Tue Jun 9 05:13:02 2026 read: IOPS=9, BW=38.0KiB/s (38.9kB/s)(5060KiB/133096msec) slat (usec): min=3664, max=56765, avg=17334.20, stdev=9156.59 clat (msec): min=19, max=1699, avg=511.90, stdev=155.01 lat (msec): min=29, max=1715, avg=529.24, stdev=156.02 clat percentiles (msec): | 1.00th=[ 241], 5.00th=[ 313], 10.00th=[ 347], 20.00th=[ 393], | 30.00th=[ 435], 40.00th=[ 464], 50.00th=[ 498], 60.00th=[ 527], | 70.00th=[ 567], 80.00th=[ 600], 90.00th=[ 693], 95.00th=[ 785], | 99.00th=[ 953], 99.50th=[ 1116], 99.90th=[ 1620], 99.95th=[ 1703], | 99.99th=[ 1703] bw ( KiB/s): min= 7, max= 95, per=25.84%, avg=38.75, stdev=18.98, samples=255 iops : min= 1, max= 23, avg= 9.15, stdev= 4.77, samples=255 write: IOPS=9, BW=38.9KiB/s (39.9kB/s)(5180KiB/133096msec); 0 zone resets slat (usec): min=54, max=15399, avg=433.92, stdev=1063.68 clat (msec): min=218, max=2451, avg=1123.53, stdev=332.01 lat (msec): min=223, max=2451, avg=1123.97, stdev=331.96 clat percentiles (msec): | 1.00th=[ 464], 5.00th=[ 676], 10.00th=[ 768], 20.00th=[ 869], | 30.00th=[ 936], 40.00th=[ 1003], 50.00th=[ 1083], 60.00th=[ 1150], | 70.00th=[ 1250], 80.00th=[ 1351], 90.00th=[ 1552], 95.00th=[ 1737], | 99.00th=[ 2198], 99.50th=[ 2265], 99.90th=[ 2433], 99.95th=[ 2467], | 99.99th=[ 2467] bw ( KiB/s): min= 7, max= 95, per=24.72%, avg=38.56, stdev=18.88, samples=261 iops : min= 1, max= 23, avg= 9.10, stdev= 4.72, samples=261 lat (msec) : 20=0.04%, 50=0.08%, 100=0.12%, 250=0.47%, 500=25.39% lat (msec) : 750=24.49%, 1000=18.67%, 2000=29.49%, >=2000=1.25% cpu : usr=0.09%, sys=0.62%, ctx=4722, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=150KiB/s (154kB/s), 37.3KiB/s-38.0KiB/s (38.2kB/s-38.9kB/s), io=19.6MiB (20.5MB), run=133096-133354msec WRITE: bw=157KiB/s (160kB/s), 38.9KiB/s-39.5KiB/s (39.9kB/s-40.4kB/s), io=20.4MiB (21.4MB), run=133096-133354msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=145636: Tue Jun 9 05:13:14 2026 read: IOPS=1, BW=62.5MiB/s (65.5MB/s)(40.0MiB/640msec) slat (nsec): min=54276k, max=54276k, avg=54276280.00, stdev= 0.00 clat (nsec): min=538096k, max=538096k, avg=538096188.00, stdev= 0.00 lat (nsec): min=592378k, max=592378k, avg=592378243.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 542], 5.00th=[ 542], 10.00th=[ 542], 20.00th=[ 542], | 30.00th=[ 542], 40.00th=[ 542], 50.00th=[ 542], 60.00th=[ 542], | 70.00th=[ 542], 80.00th=[ 542], 90.00th=[ 542], 95.00th=[ 542], | 99.00th=[ 542], 99.50th=[ 542], 99.90th=[ 542], 99.95th=[ 542], | 99.99th=[ 542] lat (msec) : 750=100.00% cpu : usr=0.31%, sys=3.60%, ctx=31, majf=0, minf=9 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=62.5MiB/s (65.5MB/s), 62.5MiB/s-62.5MiB/s (65.5MB/s-65.5MB/s), io=40.0MiB (41.9MB), run=640-640msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout