== sanity test 398c: run fio to test AIO ================= 02:15:00 (1788588900) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.880329 s, 47.6 MB/s osc.lustre-OST0000-osc-ffff8aae83350800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=179941: Sat Sep 5 02:15:44 2026 write: IOPS=88, BW=352KiB/s (361kB/s)(10.0MiB/29067msec); 0 zone resets slat (usec): min=25, max=8906, avg=125.75, stdev=411.66 clat (msec): min=29, max=372, avg=180.45, stdev=46.82 lat (msec): min=29, max=372, avg=180.58, stdev=46.82 clat percentiles (msec): | 1.00th=[ 87], 5.00th=[ 112], 10.00th=[ 127], 20.00th=[ 144], | 30.00th=[ 159], 40.00th=[ 165], 50.00th=[ 176], 60.00th=[ 184], | 70.00th=[ 197], 80.00th=[ 218], 90.00th=[ 245], 95.00th=[ 266], | 99.00th=[ 305], 99.50th=[ 372], 99.90th=[ 372], 99.95th=[ 372], | 99.99th=[ 372] bw ( KiB/s): min= 230, max= 515, per=25.06%, avg=352.61, stdev=74.73, samples=57 iops : min= 57, max= 128, avg=87.79, stdev=18.69, samples=57 lat (msec) : 50=0.12%, 100=2.19%, 250=88.87%, 500=8.83% cpu : usr=0.17%, sys=0.83%, ctx=1881, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=179942: Sat Sep 5 02:15:44 2026 write: IOPS=87, BW=352KiB/s (360kB/s)(10.0MiB/29111msec); 0 zone resets slat (usec): min=21, max=11791, avg=136.83, stdev=491.53 clat (msec): min=20, max=380, avg=180.62, stdev=46.47 lat (msec): min=20, max=380, avg=180.76, stdev=46.46 clat percentiles (msec): | 1.00th=[ 94], 5.00th=[ 110], 10.00th=[ 128], 20.00th=[ 144], | 30.00th=[ 159], 40.00th=[ 167], 50.00th=[ 176], 60.00th=[ 186], | 70.00th=[ 199], 80.00th=[ 213], 90.00th=[ 245], 95.00th=[ 262], | 99.00th=[ 309], 99.50th=[ 359], 99.90th=[ 376], 99.95th=[ 376], | 99.99th=[ 380] bw ( KiB/s): min= 168, max= 500, per=24.88%, avg=350.09, stdev=80.54, samples=57 iops : min= 42, max= 125, avg=87.18, stdev=20.16, samples=57 lat (msec) : 50=0.04%, 100=2.66%, 250=89.06%, 500=8.24% cpu : usr=0.15%, sys=0.72%, ctx=1600, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=179943: Sat Sep 5 02:15:44 2026 write: IOPS=88, BW=354KiB/s (363kB/s)(10.0MiB/28886msec); 0 zone resets slat (usec): min=25, max=20194, avg=171.92, stdev=782.63 clat (msec): min=18, max=388, avg=180.18, stdev=46.67 lat (msec): min=19, max=388, avg=180.35, stdev=46.67 clat percentiles (msec): | 1.00th=[ 85], 5.00th=[ 112], 10.00th=[ 127], 20.00th=[ 140], | 30.00th=[ 159], 40.00th=[ 167], 50.00th=[ 178], 60.00th=[ 186], | 70.00th=[ 197], 80.00th=[ 213], 90.00th=[ 245], 95.00th=[ 262], | 99.00th=[ 326], 99.50th=[ 334], 99.90th=[ 376], 99.95th=[ 376], | 99.99th=[ 388] bw ( KiB/s): min= 175, max= 546, per=25.16%, avg=353.96, stdev=77.68, samples=57 iops : min= 43, max= 136, avg=88.14, stdev=19.48, samples=57 lat (msec) : 20=0.04%, 50=0.12%, 100=2.58%, 250=88.87%, 500=8.40% cpu : usr=0.13%, sys=0.78%, ctx=2404, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=179944: Sat Sep 5 02:15:44 2026 write: IOPS=88, BW=353KiB/s (361kB/s)(10.0MiB/29019msec); 0 zone resets slat (usec): min=20, max=28538, avg=167.91, stdev=785.08 clat (msec): min=15, max=382, avg=180.14, stdev=48.40 lat (msec): min=15, max=382, avg=180.31, stdev=48.39 clat percentiles (msec): | 1.00th=[ 84], 5.00th=[ 110], 10.00th=[ 126], 20.00th=[ 144], | 30.00th=[ 155], 40.00th=[ 165], 50.00th=[ 176], 60.00th=[ 188], | 70.00th=[ 197], 80.00th=[ 211], 90.00th=[ 243], 95.00th=[ 271], | 99.00th=[ 334], 99.50th=[ 351], 99.90th=[ 384], 99.95th=[ 384], | 99.99th=[ 384] bw ( KiB/s): min= 192, max= 585, per=25.05%, avg=352.40, stdev=80.05, samples=57 iops : min= 48, max= 146, avg=87.75, stdev=20.04, samples=57 lat (msec) : 20=0.04%, 50=0.12%, 100=2.89%, 250=87.62%, 500=9.34% cpu : usr=0.16%, sys=0.82%, ctx=2193, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1407KiB/s (1441kB/s), 352KiB/s-354KiB/s (360kB/s-363kB/s), io=40.0MiB (41.9MB), run=28886-29111msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=179959: Sat Sep 5 02:16:25 2026 read: IOPS=41, BW=166KiB/s (170kB/s)(5048KiB/30330msec) slat (usec): min=2358, max=52452, avg=13561.67, stdev=7595.34 clat (msec): min=47, max=414, avg=179.67, stdev=43.30 lat (msec): min=54, max=427, avg=193.24, stdev=44.79 clat percentiles (msec): | 1.00th=[ 97], 5.00th=[ 117], 10.00th=[ 129], 20.00th=[ 144], | 30.00th=[ 155], 40.00th=[ 167], 50.00th=[ 178], 60.00th=[ 190], | 70.00th=[ 201], 80.00th=[ 213], 90.00th=[ 232], 95.00th=[ 249], | 99.00th=[ 296], 99.50th=[ 351], 99.90th=[ 401], 99.95th=[ 414], | 99.99th=[ 414] bw ( KiB/s): min= 81, max= 239, per=25.03%, avg=165.45, stdev=36.06, samples=60 iops : min= 20, max= 59, avg=40.98, stdev= 9.05, samples=60 write: IOPS=42, BW=171KiB/s (175kB/s)(5192KiB/30330msec); 0 zone resets slat (usec): min=23, max=12084, avg=223.35, stdev=832.78 clat (msec): min=47, max=389, avg=182.43, stdev=42.59 lat (msec): min=47, max=389, avg=182.66, stdev=42.53 clat percentiles (msec): | 1.00th=[ 102], 5.00th=[ 118], 10.00th=[ 130], 20.00th=[ 146], | 30.00th=[ 159], 40.00th=[ 171], 50.00th=[ 182], 60.00th=[ 192], | 70.00th=[ 203], 80.00th=[ 215], 90.00th=[ 232], 95.00th=[ 249], | 99.00th=[ 313], 99.50th=[ 342], 99.90th=[ 376], 99.95th=[ 388], | 99.99th=[ 388] bw ( KiB/s): min= 81, max= 260, per=24.79%, avg=170.57, stdev=40.16, samples=60 iops : min= 20, max= 65, avg=42.28, stdev=10.11, samples=60 lat (msec) : 50=0.12%, 100=1.05%, 250=94.02%, 500=4.80% cpu : usr=0.29%, sys=1.76%, ctx=4273, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=179960: Sat Sep 5 02:16:25 2026 read: IOPS=41, BW=164KiB/s (168kB/s)(4976KiB/30314msec) slat (usec): min=2430, max=57422, avg=13425.17, stdev=7523.76 clat (msec): min=33, max=396, avg=178.42, stdev=44.10 lat (msec): min=54, max=410, avg=191.85, stdev=45.52 clat percentiles (msec): | 1.00th=[ 93], 5.00th=[ 115], 10.00th=[ 126], 20.00th=[ 142], | 30.00th=[ 153], 40.00th=[ 165], 50.00th=[ 176], 60.00th=[ 188], | 70.00th=[ 199], 80.00th=[ 213], 90.00th=[ 234], 95.00th=[ 255], | 99.00th=[ 305], 99.50th=[ 317], 99.90th=[ 351], 99.95th=[ 397], | 99.99th=[ 397] bw ( KiB/s): min= 87, max= 232, per=24.70%, avg=163.28, stdev=37.87, samples=60 iops : min= 21, max= 58, avg=40.43, stdev= 9.57, samples=60 write: IOPS=43, BW=174KiB/s (178kB/s)(5264KiB/30314msec); 0 zone resets slat (usec): min=24, max=13267, avg=187.85, stdev=626.33 clat (msec): min=74, max=360, avg=184.64, stdev=41.19 lat (msec): min=74, max=362, avg=184.83, stdev=41.22 clat percentiles (msec): | 1.00th=[ 102], 5.00th=[ 123], 10.00th=[ 134], 20.00th=[ 150], | 30.00th=[ 163], 40.00th=[ 174], 50.00th=[ 182], 60.00th=[ 192], | 70.00th=[ 203], 80.00th=[ 218], 90.00th=[ 239], 95.00th=[ 255], | 99.00th=[ 296], 99.50th=[ 317], 99.90th=[ 355], 99.95th=[ 359], | 99.99th=[ 359] bw ( KiB/s): min= 94, max= 316, per=25.16%, avg=173.07, stdev=39.26, samples=60 iops : min= 23, max= 79, avg=42.90, stdev= 9.83, samples=60 lat (msec) : 50=0.08%, 100=1.45%, 250=92.66%, 500=5.82% cpu : usr=0.24%, sys=1.84%, ctx=4235, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=179961: Sat Sep 5 02:16:25 2026 read: IOPS=41, BW=166KiB/s (170kB/s)(4984KiB/30094msec) slat (usec): min=2495, max=64352, avg=13223.23, stdev=7605.75 clat (msec): min=82, max=417, avg=178.43, stdev=42.74 lat (msec): min=87, max=431, avg=191.66, stdev=44.34 clat percentiles (msec): | 1.00th=[ 96], 5.00th=[ 113], 10.00th=[ 128], 20.00th=[ 144], | 30.00th=[ 155], 40.00th=[ 165], 50.00th=[ 176], 60.00th=[ 186], | 70.00th=[ 197], 80.00th=[ 209], 90.00th=[ 234], 95.00th=[ 251], | 99.00th=[ 313], 99.50th=[ 326], 99.90th=[ 372], 99.95th=[ 418], | 99.99th=[ 418] bw ( KiB/s): min= 74, max= 238, per=25.17%, avg=166.36, stdev=38.71, samples=59 iops : min= 18, max= 59, avg=41.22, stdev= 9.71, samples=59 write: IOPS=43, BW=175KiB/s (179kB/s)(5256KiB/30094msec); 0 zone resets slat (usec): min=23, max=11459, avg=212.55, stdev=721.25 clat (msec): min=85, max=444, avg=180.87, stdev=43.38 lat (msec): min=89, max=444, avg=181.09, stdev=43.40 clat percentiles (msec): | 1.00th=[ 103], 5.00th=[ 120], 10.00th=[ 132], 20.00th=[ 144], | 30.00th=[ 157], 40.00th=[ 167], 50.00th=[ 178], 60.00th=[ 188], | 70.00th=[ 199], 80.00th=[ 213], 90.00th=[ 234], 95.00th=[ 253], | 99.00th=[ 300], 99.50th=[ 351], 99.90th=[ 443], 99.95th=[ 443], | 99.99th=[ 443] bw ( KiB/s): min= 74, max= 245, per=25.42%, avg=174.88, stdev=45.62, samples=59 iops : min= 18, max= 61, avg=43.37, stdev=11.51, samples=59 lat (msec) : 100=1.33%, 250=93.40%, 500=5.27% cpu : usr=0.22%, sys=1.81%, ctx=4088, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=179962: Sat Sep 5 02:16:25 2026 read: IOPS=42, BW=169KiB/s (173kB/s)(5060KiB/30026msec) slat (usec): min=50, max=55725, avg=13438.60, stdev=7593.18 clat (msec): min=37, max=389, avg=176.77, stdev=43.24 lat (msec): min=37, max=397, avg=190.21, stdev=44.69 clat percentiles (msec): | 1.00th=[ 93], 5.00th=[ 112], 10.00th=[ 125], 20.00th=[ 142], | 30.00th=[ 153], 40.00th=[ 165], 50.00th=[ 176], 60.00th=[ 186], | 70.00th=[ 197], 80.00th=[ 211], 90.00th=[ 232], 95.00th=[ 249], | 99.00th=[ 296], 99.50th=[ 313], 99.90th=[ 376], 99.95th=[ 388], | 99.99th=[ 388] bw ( KiB/s): min= 78, max= 272, per=25.54%, avg=168.85, stdev=35.22, samples=59 iops : min= 19, max= 68, avg=41.86, stdev= 8.88, samples=59 write: IOPS=43, BW=173KiB/s (177kB/s)(5180KiB/30026msec); 0 zone resets slat (usec): min=25, max=32852, avg=262.64, stdev=1272.10 clat (msec): min=29, max=448, avg=184.37, stdev=42.46 lat (msec): min=29, max=448, avg=184.63, stdev=42.47 clat percentiles (msec): | 1.00th=[ 96], 5.00th=[ 120], 10.00th=[ 133], 20.00th=[ 150], | 30.00th=[ 163], 40.00th=[ 174], 50.00th=[ 184], 60.00th=[ 194], | 70.00th=[ 205], 80.00th=[ 218], 90.00th=[ 234], 95.00th=[ 253], | 99.00th=[ 300], 99.50th=[ 313], 99.90th=[ 447], 99.95th=[ 447], | 99.99th=[ 447] bw ( KiB/s): min= 93, max= 248, per=25.03%, avg=172.22, stdev=35.89, samples=59 iops : min= 23, max= 62, avg=42.69, stdev= 9.02, samples=59 lat (msec) : 50=0.16%, 100=1.45%, 250=93.05%, 500=5.35% cpu : usr=0.26%, sys=1.58%, ctx=4074, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=662KiB/s (678kB/s), 164KiB/s-169KiB/s (168kB/s-173kB/s), io=19.6MiB (20.5MB), run=30026-30330msec WRITE: bw=689KiB/s (705kB/s), 171KiB/s-175KiB/s (175kB/s-179kB/s), io=20.4MiB (21.4MB), run=30026-30330msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=179970: Sat Sep 5 02:16:34 2026 read: IOPS=1, BW=79.4MiB/s (83.2MB/s)(40.0MiB/504msec) slat (nsec): min=24138k, max=24138k, avg=24137585.00, stdev= 0.00 clat (nsec): min=467093k, max=467093k, avg=467093300.00, stdev= 0.00 lat (nsec): min=491236k, max=491236k, avg=491235975.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 468], 5.00th=[ 468], 10.00th=[ 468], 20.00th=[ 468], | 30.00th=[ 468], 40.00th=[ 468], 50.00th=[ 468], 60.00th=[ 468], | 70.00th=[ 468], 80.00th=[ 468], 90.00th=[ 468], 95.00th=[ 468], | 99.00th=[ 468], 99.50th=[ 468], 99.90th=[ 468], 99.95th=[ 468], | 99.99th=[ 468] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=1.99%, ctx=12, majf=0, minf=7 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=79.4MiB/s (83.2MB/s), 79.4MiB/s-79.4MiB/s (83.2MB/s-83.2MB/s), io=40.0MiB (41.9MB), run=504-504msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout