== sanity test 398c: run fio to test AIO ================= 16:42:57 (1787344977) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.549667 s, 76.3 MB/s osc.lustre-OST0000-osc-ffff9e3a52fe7000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=179174: Fri Aug 21 16:43:34 2026 write: IOPS=98, BW=392KiB/s (402kB/s)(10.0MiB/26110msec); 0 zone resets slat (usec): min=20, max=12496, avg=191.87, stdev=687.24 clat (msec): min=30, max=341, avg=162.84, stdev=36.71 lat (msec): min=30, max=341, avg=163.03, stdev=36.71 clat percentiles (msec): | 1.00th=[ 103], 5.00th=[ 115], 10.00th=[ 126], 20.00th=[ 134], | 30.00th=[ 142], 40.00th=[ 150], 50.00th=[ 157], 60.00th=[ 165], | 70.00th=[ 174], 80.00th=[ 188], 90.00th=[ 211], 95.00th=[ 230], | 99.00th=[ 275], 99.50th=[ 305], 99.90th=[ 342], 99.95th=[ 342], | 99.99th=[ 342] bw ( KiB/s): min= 216, max= 512, per=25.00%, avg=391.24, stdev=66.60, samples=51 iops : min= 54, max= 128, avg=97.45, stdev=16.65, samples=51 lat (msec) : 50=0.12%, 100=0.39%, 250=96.45%, 500=3.05% cpu : usr=0.19%, sys=0.89%, ctx=2253, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=179176: Fri Aug 21 16:43:34 2026 write: IOPS=97, BW=392KiB/s (401kB/s)(10.0MiB/26141msec); 0 zone resets slat (usec): min=22, max=19296, avg=214.29, stdev=748.38 clat (msec): min=19, max=301, avg=163.04, stdev=37.26 lat (msec): min=20, max=301, avg=163.26, stdev=37.24 clat percentiles (msec): | 1.00th=[ 96], 5.00th=[ 114], 10.00th=[ 126], 20.00th=[ 136], | 30.00th=[ 144], 40.00th=[ 150], 50.00th=[ 157], 60.00th=[ 165], | 70.00th=[ 171], 80.00th=[ 186], 90.00th=[ 211], 95.00th=[ 243], | 99.00th=[ 288], 99.50th=[ 292], 99.90th=[ 292], 99.95th=[ 296], | 99.99th=[ 300] bw ( KiB/s): min= 255, max= 512, per=24.91%, avg=389.90, stdev=64.28, samples=51 iops : min= 63, max= 128, avg=97.10, stdev=16.05, samples=51 lat (msec) : 20=0.04%, 50=0.12%, 100=1.21%, 250=94.69%, 500=3.95% cpu : usr=0.22%, sys=0.95%, ctx=2862, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=179177: Fri Aug 21 16:43:34 2026 write: IOPS=98, BW=393KiB/s (402kB/s)(10.0MiB/26065msec); 0 zone resets slat (usec): min=22, max=12286, avg=172.60, stdev=575.68 clat (msec): min=48, max=323, avg=162.66, stdev=37.11 lat (msec): min=48, max=323, avg=162.84, stdev=37.13 clat percentiles (msec): | 1.00th=[ 96], 5.00th=[ 110], 10.00th=[ 125], 20.00th=[ 136], | 30.00th=[ 144], 40.00th=[ 150], 50.00th=[ 157], 60.00th=[ 167], | 70.00th=[ 176], 80.00th=[ 186], 90.00th=[ 209], 95.00th=[ 230], | 99.00th=[ 284], 99.50th=[ 313], 99.90th=[ 326], 99.95th=[ 326], | 99.99th=[ 326] bw ( KiB/s): min= 200, max= 512, per=25.03%, avg=391.71, stdev=64.29, samples=51 iops : min= 50, max= 128, avg=97.59, stdev=16.07, samples=51 lat (msec) : 50=0.04%, 100=1.56%, 250=94.84%, 500=3.55% cpu : usr=0.18%, sys=0.92%, ctx=2387, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=179178: Fri Aug 21 16:43:34 2026 write: IOPS=97, BW=391KiB/s (401kB/s)(10.0MiB/26165msec); 0 zone resets slat (usec): min=20, max=14432, avg=168.59, stdev=580.50 clat (msec): min=57, max=299, avg=162.91, stdev=36.19 lat (msec): min=58, max=299, avg=163.08, stdev=36.19 clat percentiles (msec): | 1.00th=[ 100], 5.00th=[ 115], 10.00th=[ 126], 20.00th=[ 136], | 30.00th=[ 144], 40.00th=[ 150], 50.00th=[ 157], 60.00th=[ 165], | 70.00th=[ 176], 80.00th=[ 184], 90.00th=[ 209], 95.00th=[ 241], | 99.00th=[ 279], 99.50th=[ 292], 99.90th=[ 300], 99.95th=[ 300], | 99.99th=[ 300] bw ( KiB/s): min= 200, max= 512, per=24.87%, avg=389.14, stdev=66.86, samples=51 iops : min= 50, max= 128, avg=96.94, stdev=16.73, samples=51 lat (msec) : 100=1.37%, 250=94.65%, 500=3.98% cpu : usr=0.18%, sys=1.06%, ctx=2586, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1565KiB/s (1603kB/s), 391KiB/s-393KiB/s (401kB/s-402kB/s), io=40.0MiB (41.9MB), run=26065-26165msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=179191: Fri Aug 21 16:44:16 2026 read: IOPS=38, BW=153KiB/s (157kB/s)(5048KiB/32886msec) slat (usec): min=2744, max=51788, avg=14390.58, stdev=7742.65 clat (msec): min=63, max=421, avg=193.69, stdev=50.87 lat (msec): min=86, max=437, avg=208.09, stdev=52.91 clat percentiles (msec): | 1.00th=[ 97], 5.00th=[ 120], 10.00th=[ 130], 20.00th=[ 150], | 30.00th=[ 167], 40.00th=[ 178], 50.00th=[ 190], 60.00th=[ 203], | 70.00th=[ 218], 80.00th=[ 236], 90.00th=[ 259], 95.00th=[ 284], | 99.00th=[ 334], 99.50th=[ 359], 99.90th=[ 409], 99.95th=[ 422], | 99.99th=[ 422] bw ( KiB/s): min= 62, max= 295, per=25.04%, avg=152.77, stdev=39.47, samples=65 iops : min= 15, max= 73, avg=37.77, stdev= 9.83, samples=65 write: IOPS=39, BW=158KiB/s (162kB/s)(5192KiB/32886msec); 0 zone resets slat (usec): min=26, max=16487, avg=213.01, stdev=781.39 clat (msec): min=90, max=375, avg=199.29, stdev=47.32 lat (msec): min=90, max=375, avg=199.50, stdev=47.36 clat percentiles (msec): | 1.00th=[ 101], 5.00th=[ 127], 10.00th=[ 142], 20.00th=[ 161], | 30.00th=[ 174], 40.00th=[ 186], 50.00th=[ 197], 60.00th=[ 207], | 70.00th=[ 222], 80.00th=[ 234], 90.00th=[ 259], 95.00th=[ 284], | 99.00th=[ 347], 99.50th=[ 363], 99.90th=[ 376], 99.95th=[ 376], | 99.99th=[ 376] bw ( KiB/s): min= 41, max= 231, per=24.83%, avg=157.68, stdev=38.58, samples=65 iops : min= 10, max= 57, avg=39.03, stdev= 9.59, samples=65 lat (msec) : 100=1.13%, 250=86.29%, 500=12.58% cpu : usr=0.23%, sys=1.53%, ctx=3916, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=179192: Fri Aug 21 16:44:16 2026 read: IOPS=37, BW=151KiB/s (155kB/s)(4976KiB/32851msec) slat (usec): min=2972, max=62297, avg=14218.71, stdev=8092.63 clat (msec): min=79, max=393, avg=192.25, stdev=53.26 lat (msec): min=86, max=413, avg=206.48, stdev=55.35 clat percentiles (msec): | 1.00th=[ 96], 5.00th=[ 113], 10.00th=[ 129], 20.00th=[ 146], | 30.00th=[ 161], 40.00th=[ 174], 50.00th=[ 186], 60.00th=[ 201], | 70.00th=[ 218], 80.00th=[ 234], 90.00th=[ 259], 95.00th=[ 288], | 99.00th=[ 355], 99.50th=[ 363], 99.90th=[ 388], 99.95th=[ 393], | 99.99th=[ 393] bw ( KiB/s): min= 7, max= 271, per=24.70%, avg=150.69, stdev=44.23, samples=65 iops : min= 1, max= 67, avg=37.25, stdev=11.06, samples=65 write: IOPS=40, BW=160KiB/s (164kB/s)(5264KiB/32851msec); 0 zone resets slat (usec): min=23, max=16190, avg=214.14, stdev=679.71 clat (msec): min=90, max=438, avg=200.38, stdev=49.59 lat (msec): min=90, max=438, avg=200.60, stdev=49.57 clat percentiles (msec): | 1.00th=[ 110], 5.00th=[ 128], 10.00th=[ 140], 20.00th=[ 157], | 30.00th=[ 171], 40.00th=[ 184], 50.00th=[ 197], 60.00th=[ 209], | 70.00th=[ 224], 80.00th=[ 239], 90.00th=[ 264], 95.00th=[ 284], | 99.00th=[ 351], 99.50th=[ 384], 99.90th=[ 422], 99.95th=[ 439], | 99.99th=[ 439] bw ( KiB/s): min= 42, max= 240, per=25.20%, avg=160.02, stdev=40.32, samples=65 iops : min= 10, max= 60, avg=39.58, stdev=10.05, samples=65 lat (msec) : 100=1.02%, 250=84.88%, 500=14.10% cpu : usr=0.31%, sys=1.78%, ctx=4624, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=179193: Fri Aug 21 16:44:16 2026 read: IOPS=37, BW=152KiB/s (155kB/s)(4984KiB/32855msec) slat (usec): min=2971, max=53681, avg=14544.20, stdev=8048.04 clat (msec): min=79, max=405, avg=194.38, stdev=52.24 lat (msec): min=86, max=422, avg=208.93, stdev=54.88 clat percentiles (msec): | 1.00th=[ 95], 5.00th=[ 115], 10.00th=[ 130], 20.00th=[ 150], | 30.00th=[ 163], 40.00th=[ 178], 50.00th=[ 190], 60.00th=[ 205], | 70.00th=[ 220], 80.00th=[ 239], 90.00th=[ 262], 95.00th=[ 288], | 99.00th=[ 330], 99.50th=[ 347], 99.90th=[ 368], 99.95th=[ 405], | 99.99th=[ 405] bw ( KiB/s): min= 49, max= 278, per=24.84%, avg=151.51, stdev=38.21, samples=65 iops : min= 12, max= 69, avg=37.48, stdev= 9.52, samples=65 write: IOPS=39, BW=160KiB/s (164kB/s)(5256KiB/32855msec); 0 zone resets slat (usec): min=22, max=19021, avg=223.24, stdev=844.53 clat (msec): min=92, max=391, avg=198.08, stdev=49.96 lat (msec): min=92, max=410, avg=198.31, stdev=50.01 clat percentiles (msec): | 1.00th=[ 105], 5.00th=[ 125], 10.00th=[ 136], 20.00th=[ 155], | 30.00th=[ 169], 40.00th=[ 182], 50.00th=[ 199], 60.00th=[ 211], | 70.00th=[ 222], 80.00th=[ 236], 90.00th=[ 266], 95.00th=[ 284], | 99.00th=[ 334], 99.50th=[ 368], 99.90th=[ 388], 99.95th=[ 393], | 99.99th=[ 393] bw ( KiB/s): min= 42, max= 279, per=25.04%, avg=158.98, stdev=48.92, samples=65 iops : min= 10, max= 69, avg=39.32, stdev=12.20, samples=65 lat (msec) : 100=1.02%, 250=84.38%, 500=14.61% cpu : usr=0.26%, sys=1.79%, ctx=4529, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=179194: Fri Aug 21 16:44:16 2026 read: IOPS=38, BW=155KiB/s (158kB/s)(5060KiB/32701msec) slat (usec): min=52, max=73782, avg=14716.40, stdev=8521.69 clat (msec): min=15, max=412, avg=192.21, stdev=53.43 lat (msec): min=15, max=427, avg=206.94, stdev=56.04 clat percentiles (msec): | 1.00th=[ 92], 5.00th=[ 113], 10.00th=[ 129], 20.00th=[ 148], | 30.00th=[ 163], 40.00th=[ 174], 50.00th=[ 188], 60.00th=[ 201], | 70.00th=[ 215], 80.00th=[ 236], 90.00th=[ 262], 95.00th=[ 288], | 99.00th=[ 347], 99.50th=[ 359], 99.90th=[ 405], 99.95th=[ 414], | 99.99th=[ 414] bw ( KiB/s): min= 80, max= 303, per=25.47%, avg=155.37, stdev=41.30, samples=64 iops : min= 20, max= 75, avg=38.44, stdev=10.32, samples=64 write: IOPS=39, BW=158KiB/s (162kB/s)(5180KiB/32701msec); 0 zone resets slat (usec): min=27, max=13507, avg=213.92, stdev=766.73 clat (msec): min=63, max=470, avg=201.01, stdev=52.55 lat (msec): min=64, max=470, avg=201.23, stdev=52.53 clat percentiles (msec): | 1.00th=[ 110], 5.00th=[ 127], 10.00th=[ 140], 20.00th=[ 159], | 30.00th=[ 171], 40.00th=[ 184], 50.00th=[ 199], 60.00th=[ 209], | 70.00th=[ 222], 80.00th=[ 239], 90.00th=[ 266], 95.00th=[ 292], | 99.00th=[ 380], 99.50th=[ 388], 99.90th=[ 472], 99.95th=[ 472], | 99.99th=[ 472] bw ( KiB/s): min= 79, max= 262, per=24.92%, avg=158.25, stdev=39.09, samples=64 iops : min= 19, max= 65, avg=39.16, stdev= 9.77, samples=64 lat (msec) : 20=0.04%, 50=0.08%, 100=1.09%, 250=84.18%, 500=14.61% cpu : usr=0.28%, sys=1.85%, ctx=4434, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=610KiB/s (625kB/s), 151KiB/s-155KiB/s (155kB/s-158kB/s), io=19.6MiB (20.5MB), run=32701-32886msec WRITE: bw=635KiB/s (651kB/s), 158KiB/s-160KiB/s (162kB/s-164kB/s), io=20.4MiB (21.4MB), run=32701-32886msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=179204: Fri Aug 21 16:44:27 2026 read: IOPS=1, BW=65.0MiB/s (68.2MB/s)(40.0MiB/615msec) slat (nsec): min=32705k, max=32705k, avg=32704763.00, stdev= 0.00 clat (nsec): min=564051k, max=564051k, avg=564051250.00, stdev= 0.00 lat (nsec): min=596760k, max=596760k, avg=596760256.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 567], 5.00th=[ 567], 10.00th=[ 567], 20.00th=[ 567], | 30.00th=[ 567], 40.00th=[ 567], 50.00th=[ 567], 60.00th=[ 567], | 70.00th=[ 567], 80.00th=[ 567], 90.00th=[ 567], 95.00th=[ 567], | 99.00th=[ 567], 99.50th=[ 567], 99.90th=[ 567], 99.95th=[ 567], | 99.99th=[ 567] lat (msec) : 750=100.00% cpu : usr=0.00%, sys=0.81%, ctx=11, majf=0, minf=7 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=65.0MiB/s (68.2MB/s), 65.0MiB/s-65.0MiB/s (68.2MB/s-68.2MB/s), io=40.0MiB (41.9MB), run=615-615msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout