== sanity test 398c: run fio to test AIO ================= 08:55:57 (1789304157) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.671074 s, 62.5 MB/s osc.lustre-OST0000-osc-ffff93e4c61e9800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=180023: Sun Sep 13 08:56:38 2026 write: IOPS=82, BW=329KiB/s (337kB/s)(10.0MiB/31115msec); 0 zone resets slat (usec): min=23, max=9592, avg=191.03, stdev=682.57 clat (msec): min=31, max=438, avg=193.17, stdev=57.01 lat (msec): min=31, max=438, avg=193.37, stdev=56.99 clat percentiles (msec): | 1.00th=[ 106], 5.00th=[ 120], 10.00th=[ 133], 20.00th=[ 146], | 30.00th=[ 157], 40.00th=[ 171], 50.00th=[ 184], 60.00th=[ 199], | 70.00th=[ 215], 80.00th=[ 241], 90.00th=[ 266], 95.00th=[ 305], | 99.00th=[ 355], 99.50th=[ 414], 99.90th=[ 439], 99.95th=[ 439], | 99.99th=[ 439] bw ( KiB/s): min= 150, max= 512, per=24.83%, avg=326.74, stdev=80.21, samples=61 iops : min= 37, max= 128, avg=81.31, stdev=20.10, samples=61 lat (msec) : 50=0.08%, 100=0.55%, 250=85.12%, 500=14.26% cpu : usr=0.16%, sys=0.75%, ctx=2144, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=180024: Sun Sep 13 08:56:38 2026 write: IOPS=82, BW=329KiB/s (337kB/s)(10.0MiB/31114msec); 0 zone resets slat (usec): min=20, max=16675, avg=151.98, stdev=596.79 clat (msec): min=50, max=427, avg=193.36, stdev=56.17 lat (msec): min=50, max=427, avg=193.52, stdev=56.16 clat percentiles (msec): | 1.00th=[ 103], 5.00th=[ 120], 10.00th=[ 133], 20.00th=[ 148], | 30.00th=[ 157], 40.00th=[ 167], 50.00th=[ 188], 60.00th=[ 203], | 70.00th=[ 218], 80.00th=[ 239], 90.00th=[ 264], 95.00th=[ 296], | 99.00th=[ 368], 99.50th=[ 426], 99.90th=[ 426], 99.95th=[ 426], | 99.99th=[ 426] bw ( KiB/s): min= 128, max= 512, per=24.80%, avg=326.33, stdev=87.96, samples=61 iops : min= 32, max= 128, avg=81.21, stdev=22.00, samples=61 lat (msec) : 100=0.78%, 250=85.00%, 500=14.22% cpu : usr=0.12%, sys=0.75%, ctx=1416, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=180025: Sun Sep 13 08:56:38 2026 write: IOPS=82, BW=330KiB/s (337kB/s)(10.0MiB/31076msec); 0 zone resets slat (usec): min=23, max=26253, avg=214.19, stdev=955.24 clat (msec): min=20, max=463, avg=193.22, stdev=56.02 lat (msec): min=21, max=463, avg=193.44, stdev=56.02 clat percentiles (msec): | 1.00th=[ 100], 5.00th=[ 120], 10.00th=[ 133], 20.00th=[ 146], | 30.00th=[ 159], 40.00th=[ 171], 50.00th=[ 186], 60.00th=[ 199], | 70.00th=[ 213], 80.00th=[ 241], 90.00th=[ 262], 95.00th=[ 296], | 99.00th=[ 372], 99.50th=[ 376], 99.90th=[ 426], 99.95th=[ 443], | 99.99th=[ 464] bw ( KiB/s): min= 150, max= 504, per=24.90%, avg=327.67, stdev=78.90, samples=61 iops : min= 37, max= 126, avg=81.52, stdev=19.79, samples=61 lat (msec) : 50=0.08%, 100=1.09%, 250=84.84%, 500=13.98% cpu : usr=0.18%, sys=1.01%, ctx=2579, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=180026: Sun Sep 13 08:56:38 2026 write: IOPS=82, BW=330KiB/s (338kB/s)(10.0MiB/31003msec); 0 zone resets slat (usec): min=22, max=30971, avg=274.01, stdev=1180.23 clat (msec): min=27, max=461, avg=193.15, stdev=57.16 lat (msec): min=29, max=461, avg=193.43, stdev=57.09 clat percentiles (msec): | 1.00th=[ 92], 5.00th=[ 117], 10.00th=[ 132], 20.00th=[ 148], | 30.00th=[ 159], 40.00th=[ 171], 50.00th=[ 184], 60.00th=[ 199], | 70.00th=[ 215], 80.00th=[ 236], 90.00th=[ 266], 95.00th=[ 296], | 99.00th=[ 384], 99.50th=[ 397], 99.90th=[ 460], 99.95th=[ 460], | 99.99th=[ 460] bw ( KiB/s): min= 126, max= 503, per=24.95%, avg=328.33, stdev=78.15, samples=61 iops : min= 31, max= 125, avg=81.70, stdev=19.54, samples=61 lat (msec) : 50=0.27%, 100=1.45%, 250=84.10%, 500=14.18% cpu : usr=0.26%, sys=1.08%, ctx=2933, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1316KiB/s (1348kB/s), 329KiB/s-330KiB/s (337kB/s-338kB/s), io=40.0MiB (41.9MB), run=31003-31115msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=180042: Sun Sep 13 08:57:22 2026 read: IOPS=37, BW=148KiB/s (152kB/s)(5048KiB/34088msec) slat (usec): min=31, max=72724, avg=15105.46, stdev=8343.88 clat (msec): min=13, max=541, avg=201.80, stdev=57.25 lat (msec): min=16, max=562, avg=216.91, stdev=59.55 clat percentiles (msec): | 1.00th=[ 91], 5.00th=[ 118], 10.00th=[ 134], 20.00th=[ 157], | 30.00th=[ 174], 40.00th=[ 186], 50.00th=[ 201], 60.00th=[ 213], | 70.00th=[ 226], 80.00th=[ 243], 90.00th=[ 275], 95.00th=[ 300], | 99.00th=[ 347], 99.50th=[ 418], 99.90th=[ 514], 99.95th=[ 542], | 99.99th=[ 542] bw ( KiB/s): min= 63, max= 255, per=25.54%, avg=148.15, stdev=41.38, samples=67 iops : min= 15, max= 63, avg=36.54, stdev=10.34, samples=67 write: IOPS=38, BW=152KiB/s (156kB/s)(5192KiB/34088msec); 0 zone resets slat (usec): min=26, max=23043, avg=333.00, stdev=1275.17 clat (msec): min=74, max=484, avg=207.99, stdev=53.77 lat (msec): min=74, max=484, avg=208.32, stdev=53.73 clat percentiles (msec): | 1.00th=[ 102], 5.00th=[ 123], 10.00th=[ 146], 20.00th=[ 163], | 30.00th=[ 178], 40.00th=[ 190], 50.00th=[ 203], 60.00th=[ 218], | 70.00th=[ 232], 80.00th=[ 253], 90.00th=[ 279], 95.00th=[ 300], | 99.00th=[ 355], 99.50th=[ 368], 99.90th=[ 472], 99.95th=[ 485], | 99.99th=[ 485] bw ( KiB/s): min= 71, max= 310, per=25.31%, avg=152.63, stdev=41.12, samples=67 iops : min= 17, max= 77, avg=37.67, stdev=10.31, samples=67 lat (msec) : 20=0.12%, 50=0.12%, 100=1.17%, 250=79.77%, 500=18.75% lat (msec) : 750=0.08% cpu : usr=0.33%, sys=2.13%, ctx=4624, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=180043: Sun Sep 13 08:57:22 2026 read: IOPS=35, BW=144KiB/s (147kB/s)(4976KiB/34590msec) slat (usec): min=2452, max=65902, avg=15208.23, stdev=8570.28 clat (msec): min=82, max=454, avg=202.13, stdev=53.84 lat (msec): min=87, max=462, avg=217.34, stdev=56.42 clat percentiles (msec): | 1.00th=[ 103], 5.00th=[ 125], 10.00th=[ 138], 20.00th=[ 159], | 30.00th=[ 174], 40.00th=[ 184], 50.00th=[ 197], 60.00th=[ 211], | 70.00th=[ 226], 80.00th=[ 243], 90.00th=[ 275], 95.00th=[ 305], | 99.00th=[ 351], 99.50th=[ 363], 99.90th=[ 388], 99.95th=[ 456], | 99.99th=[ 456] bw ( KiB/s): min= 15, max= 262, per=24.69%, avg=143.18, stdev=42.50, samples=68 iops : min= 3, max= 65, avg=35.29, stdev=10.67, samples=68 write: IOPS=38, BW=152KiB/s (156kB/s)(5264KiB/34590msec); 0 zone resets slat (usec): min=25, max=20293, avg=297.11, stdev=1064.77 clat (msec): min=72, max=394, avg=211.31, stdev=54.63 lat (msec): min=72, max=394, avg=211.61, stdev=54.59 clat percentiles (msec): | 1.00th=[ 107], 5.00th=[ 127], 10.00th=[ 146], 20.00th=[ 167], | 30.00th=[ 180], 40.00th=[ 194], 50.00th=[ 207], 60.00th=[ 220], | 70.00th=[ 239], 80.00th=[ 257], 90.00th=[ 284], 95.00th=[ 309], | 99.00th=[ 363], 99.50th=[ 380], 99.90th=[ 393], 99.95th=[ 393], | 99.99th=[ 393] bw ( KiB/s): min= 71, max= 286, per=25.65%, avg=154.64, stdev=39.27, samples=67 iops : min= 17, max= 71, avg=38.16, stdev= 9.81, samples=67 lat (msec) : 100=0.62%, 250=79.57%, 500=19.80% cpu : usr=0.26%, sys=1.95%, ctx=4428, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=180044: Sun Sep 13 08:57:22 2026 read: IOPS=36, BW=144KiB/s (148kB/s)(4984KiB/34558msec) slat (usec): min=3378, max=71509, avg=15293.64, stdev=8628.38 clat (msec): min=23, max=393, avg=202.61, stdev=53.71 lat (msec): min=36, max=409, avg=217.91, stdev=55.65 clat percentiles (msec): | 1.00th=[ 95], 5.00th=[ 118], 10.00th=[ 134], 20.00th=[ 157], | 30.00th=[ 174], 40.00th=[ 186], 50.00th=[ 201], 60.00th=[ 215], | 70.00th=[ 230], 80.00th=[ 247], 90.00th=[ 271], 95.00th=[ 296], | 99.00th=[ 342], 99.50th=[ 359], 99.90th=[ 388], 99.95th=[ 393], | 99.99th=[ 393] bw ( KiB/s): min= 45, max= 262, per=24.84%, avg=144.04, stdev=40.67, samples=68 iops : min= 11, max= 65, avg=35.53, stdev=10.17, samples=68 write: IOPS=38, BW=152KiB/s (156kB/s)(5256KiB/34558msec); 0 zone resets slat (usec): min=25, max=24523, avg=327.84, stdev=1245.04 clat (msec): min=86, max=392, avg=210.54, stdev=53.13 lat (msec): min=86, max=392, avg=210.88, stdev=53.18 clat percentiles (msec): | 1.00th=[ 106], 5.00th=[ 128], 10.00th=[ 146], 20.00th=[ 165], | 30.00th=[ 180], 40.00th=[ 194], 50.00th=[ 207], 60.00th=[ 222], | 70.00th=[ 236], 80.00th=[ 257], 90.00th=[ 284], 95.00th=[ 305], | 99.00th=[ 338], 99.50th=[ 347], 99.90th=[ 393], 99.95th=[ 393], | 99.99th=[ 393] bw ( KiB/s): min= 30, max= 286, per=25.13%, avg=151.56, stdev=44.41, samples=68 iops : min= 7, max= 71, avg=37.38, stdev=11.13, samples=68 lat (msec) : 50=0.08%, 100=0.86%, 250=78.55%, 500=20.51% cpu : usr=0.28%, sys=2.05%, ctx=4485, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=180045: Sun Sep 13 08:57:22 2026 read: IOPS=36, BW=147KiB/s (151kB/s)(5060KiB/34317msec) slat (usec): min=3379, max=63335, avg=15290.22, stdev=8494.30 clat (msec): min=21, max=437, avg=201.01, stdev=53.00 lat (msec): min=29, max=449, avg=216.31, stdev=55.39 clat percentiles (msec): | 1.00th=[ 97], 5.00th=[ 121], 10.00th=[ 134], 20.00th=[ 155], | 30.00th=[ 169], 40.00th=[ 186], 50.00th=[ 201], 60.00th=[ 213], | 70.00th=[ 228], 80.00th=[ 245], 90.00th=[ 271], 95.00th=[ 292], | 99.00th=[ 338], 99.50th=[ 351], 99.90th=[ 422], 99.95th=[ 439], | 99.99th=[ 439] bw ( KiB/s): min= 38, max= 294, per=25.31%, avg=146.82, stdev=41.42, samples=68 iops : min= 9, max= 73, avg=36.21, stdev=10.36, samples=68 write: IOPS=37, BW=151KiB/s (155kB/s)(5180KiB/34317msec); 0 zone resets slat (usec): min=29, max=14164, avg=268.35, stdev=842.87 clat (msec): min=85, max=491, avg=208.76, stdev=52.64 lat (msec): min=85, max=491, avg=209.03, stdev=52.62 clat percentiles (msec): | 1.00th=[ 104], 5.00th=[ 124], 10.00th=[ 142], 20.00th=[ 165], | 30.00th=[ 182], 40.00th=[ 194], 50.00th=[ 207], 60.00th=[ 220], | 70.00th=[ 234], 80.00th=[ 251], 90.00th=[ 275], 95.00th=[ 300], | 99.00th=[ 351], 99.50th=[ 359], 99.90th=[ 401], 99.95th=[ 493], | 99.99th=[ 493] bw ( KiB/s): min= 30, max= 263, per=24.95%, avg=150.44, stdev=44.72, samples=68 iops : min= 7, max= 65, avg=37.12, stdev=11.21, samples=68 lat (msec) : 50=0.08%, 100=0.94%, 250=80.12%, 500=18.87% cpu : usr=0.31%, sys=1.93%, ctx=4500, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=580KiB/s (594kB/s), 144KiB/s-148KiB/s (147kB/s-152kB/s), io=19.6MiB (20.5MB), run=34088-34590msec WRITE: bw=604KiB/s (618kB/s), 151KiB/s-152KiB/s (155kB/s-156kB/s), io=20.4MiB (21.4MB), run=34088-34590msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=180055: Sun Sep 13 08:57:31 2026 read: IOPS=1, BW=46.0MiB/s (48.3MB/s)(40.0MiB/869msec) slat (nsec): min=42470k, max=42470k, avg=42470124.00, stdev= 0.00 clat (nsec): min=776364k, max=776364k, avg=776363510.00, stdev= 0.00 lat (nsec): min=818838k, max=818838k, avg=818838350.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 776], 5.00th=[ 776], 10.00th=[ 776], 20.00th=[ 776], | 30.00th=[ 776], 40.00th=[ 776], 50.00th=[ 776], 60.00th=[ 776], | 70.00th=[ 776], 80.00th=[ 776], 90.00th=[ 776], 95.00th=[ 776], | 99.00th=[ 776], 99.50th=[ 776], 99.90th=[ 776], 99.95th=[ 776], | 99.99th=[ 776] lat (msec) : 1000=100.00% cpu : usr=0.00%, sys=1.50%, ctx=9, majf=0, minf=7 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=46.0MiB/s (48.3MB/s), 46.0MiB/s-46.0MiB/s (48.3MB/s-48.3MB/s), io=40.0MiB (41.9MB), run=869-869msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout