== sanity test 398c: run fio to test AIO ================= 03:47:16 (1787471236) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.580257 s, 72.3 MB/s osc.lustre-OST0000-osc-ffff982d12257000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=144779: Sun Aug 23 03:50:28 2026 write: IOPS=14, BW=56.9KiB/s (58.2kB/s)(10.0MiB/180061msec); 0 zone resets slat (usec): min=50, max=24478, avg=353.37, stdev=1204.84 clat (msec): min=272, max=2592, avg=1123.75, stdev=318.45 lat (msec): min=272, max=2592, avg=1124.11, stdev=318.49 clat percentiles (msec): | 1.00th=[ 355], 5.00th=[ 634], 10.00th=[ 751], 20.00th=[ 869], | 30.00th=[ 953], 40.00th=[ 1028], 50.00th=[ 1099], 60.00th=[ 1183], | 70.00th=[ 1267], 80.00th=[ 1368], 90.00th=[ 1536], 95.00th=[ 1670], | 99.00th=[ 1972], 99.50th=[ 2198], 99.90th=[ 2467], 99.95th=[ 2500], | 99.99th=[ 2601] bw ( KiB/s): min= 7, max= 136, per=24.89%, avg=56.51, stdev=28.47, samples=356 iops : min= 1, max= 34, avg=13.64, stdev= 7.16, samples=356 lat (msec) : 500=1.99%, 750=7.97%, 1000=25.98%, 2000=63.44%, >=2000=0.62% cpu : usr=0.05%, sys=0.30%, ctx=2247, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=144780: Sun Aug 23 03:50:28 2026 write: IOPS=14, BW=56.9KiB/s (58.2kB/s)(10.0MiB/180116msec); 0 zone resets slat (usec): min=51, max=18758, avg=316.72, stdev=1025.77 clat (msec): min=332, max=2593, avg=1124.32, stdev=318.86 lat (msec): min=333, max=2593, avg=1124.64, stdev=318.88 clat percentiles (msec): | 1.00th=[ 351], 5.00th=[ 634], 10.00th=[ 751], 20.00th=[ 869], | 30.00th=[ 953], 40.00th=[ 1028], 50.00th=[ 1099], 60.00th=[ 1183], | 70.00th=[ 1267], 80.00th=[ 1368], 90.00th=[ 1519], 95.00th=[ 1687], | 99.00th=[ 1989], 99.50th=[ 2232], 99.90th=[ 2433], 99.95th=[ 2500], | 99.99th=[ 2601] bw ( KiB/s): min= 7, max= 143, per=24.96%, avg=56.67, stdev=28.26, samples=355 iops : min= 1, max= 35, avg=13.68, stdev= 7.10, samples=355 lat (msec) : 500=2.03%, 750=7.19%, 1000=26.84%, 2000=62.97%, >=2000=0.98% cpu : usr=0.04%, sys=0.29%, ctx=1902, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=144781: Sun Aug 23 03:50:28 2026 write: IOPS=14, BW=56.0KiB/s (58.3kB/s)(10.0MiB/179788msec); 0 zone resets slat (usec): min=45, max=32108, avg=365.33, stdev=1404.85 clat (msec): min=138, max=2372, avg=1122.87, stdev=323.67 lat (msec): min=138, max=2372, avg=1123.24, stdev=323.69 clat percentiles (msec): | 1.00th=[ 401], 5.00th=[ 625], 10.00th=[ 751], 20.00th=[ 869], | 30.00th=[ 953], 40.00th=[ 1028], 50.00th=[ 1099], 60.00th=[ 1167], | 70.00th=[ 1267], 80.00th=[ 1385], 90.00th=[ 1552], 95.00th=[ 1670], | 99.00th=[ 1921], 99.50th=[ 2265], 99.90th=[ 2366], 99.95th=[ 2366], | 99.99th=[ 2366] bw ( KiB/s): min= 7, max= 150, per=25.10%, avg=56.98, stdev=27.50, samples=353 iops : min= 1, max= 37, avg=13.76, stdev= 6.92, samples=353 lat (msec) : 250=0.31%, 500=1.84%, 750=7.70%, 1000=26.45%, 2000=62.93% lat (msec) : >=2000=0.78% cpu : usr=0.04%, sys=0.30%, ctx=2133, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=144782: Sun Aug 23 03:50:28 2026 write: IOPS=14, BW=56.9KiB/s (58.2kB/s)(10.0MiB/180112msec); 0 zone resets slat (usec): min=49, max=27128, avg=353.34, stdev=1167.31 clat (msec): min=302, max=2520, avg=1124.09, stdev=318.38 lat (msec): min=303, max=2520, avg=1124.45, stdev=318.39 clat percentiles (msec): | 1.00th=[ 355], 5.00th=[ 642], 10.00th=[ 751], 20.00th=[ 860], | 30.00th=[ 953], 40.00th=[ 1036], 50.00th=[ 1099], 60.00th=[ 1167], | 70.00th=[ 1267], 80.00th=[ 1385], 90.00th=[ 1536], 95.00th=[ 1670], | 99.00th=[ 1921], 99.50th=[ 2140], 99.90th=[ 2366], 99.95th=[ 2534], | 99.99th=[ 2534] bw ( KiB/s): min= 7, max= 128, per=25.40%, avg=57.65, stdev=26.24, samples=349 iops : min= 1, max= 32, avg=13.93, stdev= 6.59, samples=349 lat (msec) : 500=1.99%, 750=8.09%, 1000=25.47%, 2000=63.67%, >=2000=0.78% cpu : usr=0.05%, sys=0.31%, ctx=2260, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=227KiB/s (233kB/s), 56.9KiB/s-56.0KiB/s (58.2kB/s-58.3kB/s), io=40.0MiB (41.9MB), run=179788-180116msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=144817: Sun Aug 23 03:52:47 2026 read: IOPS=9, BW=39.7KiB/s (40.7kB/s)(5048KiB/127106msec) slat (usec): min=70, max=72899, avg=16248.67, stdev=9154.77 clat (msec): min=19, max=1349, avg=492.92, stdev=143.37 lat (msec): min=27, max=1359, avg=509.17, stdev=144.80 clat percentiles (msec): | 1.00th=[ 114], 5.00th=[ 309], 10.00th=[ 338], 20.00th=[ 384], | 30.00th=[ 414], 40.00th=[ 443], 50.00th=[ 481], 60.00th=[ 518], | 70.00th=[ 558], 80.00th=[ 600], 90.00th=[ 667], 95.00th=[ 743], | 99.00th=[ 869], 99.50th=[ 969], 99.90th=[ 1334], 99.95th=[ 1351], | 99.99th=[ 1351] bw ( KiB/s): min= 7, max= 105, per=25.67%, avg=40.31, stdev=19.70, samples=245 iops : min= 1, max= 26, avg= 9.55, stdev= 4.92, samples=245 write: IOPS=10, BW=40.8KiB/s (41.8kB/s)(5192KiB/127106msec); 0 zone resets slat (usec): min=57, max=27072, avg=403.23, stdev=1356.30 clat (msec): min=141, max=2235, avg=1070.21, stdev=320.92 lat (msec): min=141, max=2235, avg=1070.61, stdev=320.91 clat percentiles (msec): | 1.00th=[ 284], 5.00th=[ 575], 10.00th=[ 693], 20.00th=[ 802], | 30.00th=[ 911], 40.00th=[ 978], 50.00th=[ 1053], 60.00th=[ 1133], | 70.00th=[ 1217], 80.00th=[ 1318], 90.00th=[ 1469], 95.00th=[ 1653], | 99.00th=[ 1888], 99.50th=[ 1972], 99.90th=[ 2232], 99.95th=[ 2232], | 99.99th=[ 2232] bw ( KiB/s): min= 7, max= 97, per=24.72%, avg=40.30, stdev=18.89, samples=251 iops : min= 1, max= 24, avg= 9.55, stdev= 4.76, samples=251 lat (msec) : 20=0.04%, 50=0.35%, 100=0.08%, 250=0.82%, 500=28.05% lat (msec) : 750=25.31%, 1000=16.41%, 2000=28.71%, >=2000=0.23% cpu : usr=0.09%, sys=0.66%, ctx=4692, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=144818: Sun Aug 23 03:52:47 2026 read: IOPS=9, BW=39.1KiB/s (40.0kB/s)(4976KiB/127307msec) slat (usec): min=3680, max=63488, avg=16135.03, stdev=8695.84 clat (msec): min=13, max=1215, avg=489.25, stdev=142.26 lat (msec): min=31, max=1222, avg=505.39, stdev=143.45 clat percentiles (msec): | 1.00th=[ 215], 5.00th=[ 305], 10.00th=[ 338], 20.00th=[ 380], | 30.00th=[ 414], 40.00th=[ 443], 50.00th=[ 472], 60.00th=[ 506], | 70.00th=[ 542], 80.00th=[ 592], 90.00th=[ 667], 95.00th=[ 735], | 99.00th=[ 877], 99.50th=[ 1133], 99.90th=[ 1183], 99.95th=[ 1217], | 99.99th=[ 1217] bw ( KiB/s): min= 7, max= 95, per=24.46%, avg=38.41, stdev=19.08, samples=253 iops : min= 1, max= 23, avg= 9.08, stdev= 4.76, samples=253 write: IOPS=10, BW=41.3KiB/s (42.3kB/s)(5264KiB/127307msec); 0 zone resets slat (usec): min=55, max=21753, avg=420.96, stdev=1229.64 clat (msec): min=281, max=2276, avg=1067.55, stdev=311.54 lat (msec): min=282, max=2279, avg=1067.98, stdev=311.53 clat percentiles (msec): | 1.00th=[ 401], 5.00th=[ 592], 10.00th=[ 693], 20.00th=[ 818], | 30.00th=[ 911], 40.00th=[ 978], 50.00th=[ 1045], 60.00th=[ 1116], | 70.00th=[ 1200], 80.00th=[ 1318], 90.00th=[ 1469], 95.00th=[ 1586], | 99.00th=[ 1955], 99.50th=[ 2089], 99.90th=[ 2265], 99.95th=[ 2265], | 99.99th=[ 2265] bw ( KiB/s): min= 7, max= 103, per=25.31%, avg=41.25, stdev=19.62, samples=249 iops : min= 1, max= 25, avg= 9.79, stdev= 4.96, samples=249 lat (msec) : 20=0.04%, 50=0.08%, 100=0.20%, 250=0.43%, 500=29.06% lat (msec) : 750=24.06%, 1000=16.91%, 2000=28.75%, >=2000=0.47% cpu : usr=0.11%, sys=0.66%, ctx=4810, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=144819: Sun Aug 23 03:52:47 2026 read: IOPS=9, BW=39.1KiB/s (40.0kB/s)(4984KiB/127463msec) slat (usec): min=3258, max=71826, avg=16876.38, stdev=9262.65 clat (msec): min=76, max=1117, avg=490.85, stdev=128.35 lat (msec): min=100, max=1174, avg=507.73, stdev=129.66 clat percentiles (msec): | 1.00th=[ 259], 5.00th=[ 313], 10.00th=[ 338], 20.00th=[ 388], | 30.00th=[ 422], 40.00th=[ 447], 50.00th=[ 477], 60.00th=[ 510], | 70.00th=[ 542], 80.00th=[ 592], 90.00th=[ 659], 95.00th=[ 718], | 99.00th=[ 852], 99.50th=[ 860], 99.90th=[ 1116], 99.95th=[ 1116], | 99.99th=[ 1116] bw ( KiB/s): min= 7, max= 111, per=24.85%, avg=39.02, stdev=19.83, samples=250 iops : min= 1, max= 27, avg= 9.23, stdev= 4.98, samples=250 write: IOPS=10, BW=41.2KiB/s (42.2kB/s)(5256KiB/127463msec); 0 zone resets slat (usec): min=52, max=15821, avg=414.51, stdev=1161.42 clat (msec): min=208, max=2238, avg=1067.35, stdev=309.72 lat (msec): min=208, max=2238, avg=1067.76, stdev=309.71 clat percentiles (msec): | 1.00th=[ 351], 5.00th=[ 592], 10.00th=[ 709], 20.00th=[ 810], | 30.00th=[ 911], 40.00th=[ 978], 50.00th=[ 1045], 60.00th=[ 1116], | 70.00th=[ 1200], 80.00th=[ 1301], 90.00th=[ 1452], 95.00th=[ 1636], | 99.00th=[ 1888], 99.50th=[ 1989], 99.90th=[ 2140], 99.95th=[ 2232], | 99.99th=[ 2232] bw ( KiB/s): min= 7, max= 111, per=24.84%, avg=40.49, stdev=18.88, samples=252 iops : min= 1, max= 27, avg= 9.60, stdev= 4.73, samples=252 lat (msec) : 100=0.04%, 250=0.39%, 500=29.18%, 750=24.22%, 1000=16.88% lat (msec) : 2000=29.06%, >=2000=0.23% cpu : usr=0.10%, sys=0.67%, ctx=4705, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=144820: Sun Aug 23 03:52:47 2026 read: IOPS=9, BW=39.8KiB/s (40.7kB/s)(5060KiB/127240msec) slat (usec): min=2925, max=73483, avg=16538.52, stdev=9229.06 clat (msec): min=24, max=1322, avg=500.81, stdev=140.68 lat (msec): min=33, max=1332, avg=517.36, stdev=142.40 clat percentiles (msec): | 1.00th=[ 209], 5.00th=[ 317], 10.00th=[ 347], 20.00th=[ 388], | 30.00th=[ 426], 40.00th=[ 451], 50.00th=[ 485], 60.00th=[ 518], | 70.00th=[ 550], 80.00th=[ 600], 90.00th=[ 693], 95.00th=[ 751], | 99.00th=[ 869], 99.50th=[ 995], 99.90th=[ 1183], 99.95th=[ 1318], | 99.99th=[ 1318] bw ( KiB/s): min= 7, max= 119, per=25.25%, avg=39.64, stdev=18.33, samples=250 iops : min= 1, max= 29, avg= 9.38, stdev= 4.56, samples=250 write: IOPS=10, BW=40.7KiB/s (41.7kB/s)(5180KiB/127240msec); 0 zone resets slat (usec): min=54, max=16321, avg=419.27, stdev=1302.85 clat (msec): min=280, max=2311, avg=1062.65, stdev=309.47 lat (msec): min=281, max=2311, avg=1063.07, stdev=309.45 clat percentiles (msec): | 1.00th=[ 418], 5.00th=[ 625], 10.00th=[ 693], 20.00th=[ 810], | 30.00th=[ 902], 40.00th=[ 961], 50.00th=[ 1028], 60.00th=[ 1099], | 70.00th=[ 1200], 80.00th=[ 1301], 90.00th=[ 1469], 95.00th=[ 1620], | 99.00th=[ 1921], 99.50th=[ 2072], 99.90th=[ 2232], 99.95th=[ 2299], | 99.99th=[ 2299] bw ( KiB/s): min= 7, max= 88, per=24.75%, avg=40.34, stdev=17.11, samples=250 iops : min= 1, max= 22, avg= 9.56, stdev= 4.31, samples=250 lat (msec) : 50=0.16%, 100=0.08%, 250=0.39%, 500=26.91%, 750=26.88% lat (msec) : 1000=18.01%, 2000=27.27%, >=2000=0.31% cpu : usr=0.11%, sys=0.66%, ctx=4748, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=157KiB/s (161kB/s), 39.1KiB/s-39.8KiB/s (40.0kB/s-40.7kB/s), io=19.6MiB (20.5MB), run=127106-127463msec WRITE: bw=164KiB/s (168kB/s), 40.7KiB/s-41.3KiB/s (41.7kB/s-42.3kB/s), io=20.4MiB (21.4MB), run=127106-127463msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=144843: Sun Aug 23 03:53:00 2026 read: IOPS=1, BW=60.2MiB/s (63.2MB/s)(40.0MiB/664msec) slat (nsec): min=67844k, max=67844k, avg=67843834.00, stdev= 0.00 clat (nsec): min=561093k, max=561093k, avg=561093463.00, stdev= 0.00 lat (nsec): min=628942k, max=628942k, avg=628942098.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 558], 5.00th=[ 558], 10.00th=[ 558], 20.00th=[ 558], | 30.00th=[ 558], 40.00th=[ 558], 50.00th=[ 558], 60.00th=[ 558], | 70.00th=[ 558], 80.00th=[ 558], 90.00th=[ 558], 95.00th=[ 558], | 99.00th=[ 558], 99.50th=[ 558], 99.90th=[ 558], 99.95th=[ 558], | 99.99th=[ 558] lat (msec) : 750=100.00% cpu : usr=0.00%, sys=2.26%, ctx=21, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=60.2MiB/s (63.2MB/s), 60.2MiB/s-60.2MiB/s (63.2MB/s-63.2MB/s), io=40.0MiB (41.9MB), run=664-664msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout