== sanity test 398c: run fio to test AIO ================= 16:45:02 (1763156702) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.211602 s, 198 MB/s osc.lustre-OST0000-osc-ffff8984d3d4b000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=149354: Fri Nov 14 16:46:50 2025 write: IOPS=24, BW=98.8KiB/s (101kB/s)(10.0MiB/103681msec); 0 zone resets slat (usec): min=23, max=5045, avg=145.12, stdev=304.90 clat (msec): min=192, max=1083, avg=647.39, stdev=158.95 lat (msec): min=192, max=1084, avg=647.54, stdev=158.96 clat percentiles (msec): | 1.00th=[ 288], 5.00th=[ 380], 10.00th=[ 447], 20.00th=[ 506], | 30.00th=[ 558], 40.00th=[ 609], 50.00th=[ 642], 60.00th=[ 693], | 70.00th=[ 726], 80.00th=[ 776], 90.00th=[ 860], 95.00th=[ 919], | 99.00th=[ 1003], 99.50th=[ 1036], 99.90th=[ 1062], 99.95th=[ 1083], | 99.99th=[ 1083] bw ( KiB/s): min= 31, max= 255, per=24.89%, avg=98.05, stdev=31.51, samples=206 iops : min= 7, max= 63, avg=24.08, stdev= 7.88, samples=206 lat (msec) : 250=0.66%, 500=17.70%, 750=56.05%, 1000=24.45%, 2000=1.13% cpu : usr=0.06%, sys=0.33%, ctx=1824, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=149355: Fri Nov 14 16:46:50 2025 write: IOPS=24, BW=98.8KiB/s (101kB/s)(10.0MiB/103645msec); 0 zone resets slat (usec): min=26, max=5220, avg=145.57, stdev=265.46 clat (msec): min=101, max=1168, avg=647.17, stdev=164.49 lat (msec): min=101, max=1168, avg=647.32, stdev=164.50 clat percentiles (msec): | 1.00th=[ 201], 5.00th=[ 388], 10.00th=[ 435], 20.00th=[ 510], | 30.00th=[ 558], 40.00th=[ 609], 50.00th=[ 651], 60.00th=[ 693], | 70.00th=[ 735], 80.00th=[ 785], 90.00th=[ 860], 95.00th=[ 911], | 99.00th=[ 1020], 99.50th=[ 1062], 99.90th=[ 1167], 99.95th=[ 1167], | 99.99th=[ 1167] bw ( KiB/s): min= 8, max= 247, per=24.87%, avg=97.99, stdev=35.20, samples=206 iops : min= 2, max= 61, avg=24.06, stdev= 8.80, samples=206 lat (msec) : 250=1.21%, 500=16.25%, 750=56.29%, 1000=24.84%, 2000=1.41% cpu : usr=0.07%, sys=0.33%, ctx=1986, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=149356: Fri Nov 14 16:46:50 2025 write: IOPS=24, BW=98.8KiB/s (101kB/s)(10.0MiB/103681msec); 0 zone resets slat (usec): min=25, max=12364, avg=145.08, stdev=361.49 clat (msec): min=139, max=1277, avg=647.56, stdev=165.29 lat (msec): min=140, max=1277, avg=647.70, stdev=165.30 clat percentiles (msec): | 1.00th=[ 161], 5.00th=[ 388], 10.00th=[ 443], 20.00th=[ 514], | 30.00th=[ 558], 40.00th=[ 609], 50.00th=[ 642], 60.00th=[ 693], | 70.00th=[ 726], 80.00th=[ 785], 90.00th=[ 869], 95.00th=[ 919], | 99.00th=[ 1011], 99.50th=[ 1053], 99.90th=[ 1200], 99.95th=[ 1200], | 99.99th=[ 1284] bw ( KiB/s): min= 8, max= 255, per=25.00%, avg=98.48, stdev=35.85, samples=205 iops : min= 2, max= 63, avg=24.18, stdev= 8.99, samples=205 lat (msec) : 250=1.25%, 500=15.86%, 750=56.68%, 1000=24.96%, 2000=1.25% cpu : usr=0.05%, sys=0.32%, ctx=2126, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=149357: Fri Nov 14 16:46:50 2025 write: IOPS=24, BW=98.7KiB/s (101kB/s)(10.0MiB/103754msec); 0 zone resets slat (usec): min=24, max=4157, avg=155.85, stdev=287.50 clat (msec): min=94, max=1168, avg=647.86, stdev=161.13 lat (msec): min=94, max=1168, avg=648.02, stdev=161.13 clat percentiles (msec): | 1.00th=[ 268], 5.00th=[ 380], 10.00th=[ 447], 20.00th=[ 510], | 30.00th=[ 567], 40.00th=[ 609], 50.00th=[ 651], 60.00th=[ 693], | 70.00th=[ 735], 80.00th=[ 776], 90.00th=[ 869], 95.00th=[ 911], | 99.00th=[ 1003], 99.50th=[ 1045], 99.90th=[ 1116], 99.95th=[ 1167], | 99.99th=[ 1167] bw ( KiB/s): min= 15, max= 232, per=24.84%, avg=97.85, stdev=34.03, samples=206 iops : min= 3, max= 58, avg=24.03, stdev= 8.49, samples=206 lat (msec) : 100=0.04%, 250=0.94%, 500=17.11%, 750=56.41%, 1000=24.45% lat (msec) : 2000=1.05% cpu : usr=0.05%, sys=0.34%, ctx=2412, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=395KiB/s (404kB/s), 98.7KiB/s-98.8KiB/s (101kB/s-101kB/s), io=40.0MiB (41.9MB), run=103645-103754msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=149381: Fri Nov 14 16:47:50 2025 read: IOPS=22, BW=90.8KiB/s (92.0kB/s)(5048KiB/55590msec) slat (usec): min=2023, max=21653, avg=6147.63, stdev=2565.20 clat (msec): min=8, max=393, avg=221.50, stdev=56.78 lat (msec): min=14, max=400, avg=227.65, stdev=57.50 clat percentiles (msec): | 1.00th=[ 110], 5.00th=[ 146], 10.00th=[ 159], 20.00th=[ 178], | 30.00th=[ 190], 40.00th=[ 205], 50.00th=[ 215], 60.00th=[ 228], | 70.00th=[ 243], 80.00th=[ 262], 90.00th=[ 300], 95.00th=[ 330], | 99.00th=[ 376], 99.50th=[ 380], 99.90th=[ 393], 99.95th=[ 393], | 99.99th=[ 393] bw ( KiB/s): min= 31, max= 199, per=25.13%, avg=90.48, stdev=32.44, samples=110 iops : min= 7, max= 49, avg=22.18, stdev= 8.11, samples=110 write: IOPS=23, BW=93.4KiB/s (95.6kB/s)(5192KiB/55590msec); 0 zone resets slat (usec): min=29, max=2809, avg=165.44, stdev=226.59 clat (msec): min=191, max=920, avg=462.32, stdev=124.67 lat (msec): min=191, max=920, avg=462.48, stdev=124.69 clat percentiles (msec): | 1.00th=[ 215], 5.00th=[ 268], 10.00th=[ 300], 20.00th=[ 355], | 30.00th=[ 393], 40.00th=[ 426], 50.00th=[ 456], 60.00th=[ 489], | 70.00th=[ 523], 80.00th=[ 567], 90.00th=[ 634], 95.00th=[ 659], | 99.00th=[ 827], 99.50th=[ 835], 99.90th=[ 894], 99.95th=[ 919], | 99.99th=[ 919] bw ( KiB/s): min= 39, max= 167, per=24.70%, avg=92.64, stdev=24.96, samples=110 iops : min= 9, max= 41, avg=22.72, stdev= 6.31, samples=110 lat (msec) : 10=0.04%, 20=0.08%, 50=0.35%, 250=37.70%, 500=43.48% lat (msec) : 750=17.54%, 1000=0.82% cpu : usr=0.10%, sys=0.94%, ctx=5010, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=149382: Fri Nov 14 16:47:50 2025 read: IOPS=22, BW=89.5KiB/s (91.7kB/s)(4976KiB/55590msec) slat (usec): min=36, max=16360, avg=6157.61, stdev=2585.88 clat (msec): min=18, max=569, avg=220.13, stdev=57.14 lat (msec): min=18, max=573, avg=226.30, stdev=57.92 clat percentiles (msec): | 1.00th=[ 117], 5.00th=[ 144], 10.00th=[ 157], 20.00th=[ 174], | 30.00th=[ 190], 40.00th=[ 203], 50.00th=[ 215], 60.00th=[ 226], | 70.00th=[ 243], 80.00th=[ 262], 90.00th=[ 292], 95.00th=[ 326], | 99.00th=[ 380], 99.50th=[ 422], 99.90th=[ 456], 99.95th=[ 567], | 99.99th=[ 567] bw ( KiB/s): min= 32, max= 176, per=24.74%, avg=89.06, stdev=31.95, samples=110 iops : min= 8, max= 44, avg=21.83, stdev= 8.01, samples=110 write: IOPS=23, BW=94.7KiB/s (96.0kB/s)(5264KiB/55590msec); 0 zone resets slat (usec): min=30, max=3994, avg=155.25, stdev=209.05 clat (msec): min=87, max=912, avg=461.30, stdev=124.95 lat (msec): min=87, max=912, avg=461.46, stdev=124.96 clat percentiles (msec): | 1.00th=[ 197], 5.00th=[ 275], 10.00th=[ 309], 20.00th=[ 355], | 30.00th=[ 393], 40.00th=[ 422], 50.00th=[ 456], 60.00th=[ 485], | 70.00th=[ 527], 80.00th=[ 567], 90.00th=[ 625], 95.00th=[ 667], | 99.00th=[ 776], 99.50th=[ 835], 99.90th=[ 911], 99.95th=[ 911], | 99.99th=[ 911] bw ( KiB/s): min= 31, max= 168, per=25.13%, avg=94.24, stdev=29.26, samples=110 iops : min= 7, max= 42, avg=23.13, stdev= 7.40, samples=110 lat (msec) : 20=0.08%, 50=0.20%, 100=0.12%, 250=37.19%, 500=43.71% lat (msec) : 750=17.97%, 1000=0.74% cpu : usr=0.11%, sys=0.92%, ctx=4931, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=149383: Fri Nov 14 16:47:50 2025 read: IOPS=22, BW=89.5KiB/s (91.7kB/s)(4984KiB/55660msec) slat (usec): min=1831, max=24924, avg=6233.18, stdev=2585.45 clat (msec): min=4, max=416, avg=221.17, stdev=57.18 lat (msec): min=11, max=423, avg=227.41, stdev=57.93 clat percentiles (msec): | 1.00th=[ 54], 5.00th=[ 146], 10.00th=[ 157], 20.00th=[ 178], | 30.00th=[ 190], 40.00th=[ 205], 50.00th=[ 215], 60.00th=[ 228], | 70.00th=[ 245], 80.00th=[ 262], 90.00th=[ 300], 95.00th=[ 334], | 99.00th=[ 368], 99.50th=[ 384], 99.90th=[ 384], 99.95th=[ 418], | 99.99th=[ 418] bw ( KiB/s): min= 24, max= 199, per=24.81%, avg=89.33, stdev=30.73, samples=110 iops : min= 6, max= 49, avg=21.89, stdev= 7.71, samples=110 write: IOPS=23, BW=94.4KiB/s (96.7kB/s)(5256KiB/55660msec); 0 zone resets slat (usec): min=35, max=4385, avg=172.28, stdev=280.36 clat (msec): min=151, max=915, avg=460.88, stdev=126.25 lat (msec): min=152, max=915, avg=461.05, stdev=126.26 clat percentiles (msec): | 1.00th=[ 213], 5.00th=[ 264], 10.00th=[ 305], 20.00th=[ 351], | 30.00th=[ 388], 40.00th=[ 418], 50.00th=[ 451], 60.00th=[ 489], | 70.00th=[ 523], 80.00th=[ 567], 90.00th=[ 625], 95.00th=[ 667], | 99.00th=[ 827], 99.50th=[ 877], 99.90th=[ 894], 99.95th=[ 919], | 99.99th=[ 919] bw ( KiB/s): min= 48, max= 183, per=24.86%, avg=93.21, stdev=24.93, samples=110 iops : min= 12, max= 45, avg=22.86, stdev= 6.31, samples=110 lat (msec) : 10=0.04%, 20=0.08%, 50=0.35%, 100=0.12%, 250=36.80% lat (msec) : 500=43.79%, 750=17.97%, 1000=0.86% cpu : usr=0.12%, sys=0.92%, ctx=5021, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=149384: Fri Nov 14 16:47:50 2025 read: IOPS=22, BW=90.9KiB/s (93.1kB/s)(5060KiB/55660msec) slat (usec): min=2057, max=23146, avg=6229.36, stdev=2568.73 clat (msec): min=8, max=386, avg=221.27, stdev=55.19 lat (msec): min=14, max=392, avg=227.51, stdev=55.81 clat percentiles (msec): | 1.00th=[ 110], 5.00th=[ 146], 10.00th=[ 159], 20.00th=[ 178], | 30.00th=[ 192], 40.00th=[ 205], 50.00th=[ 215], 60.00th=[ 228], | 70.00th=[ 241], 80.00th=[ 264], 90.00th=[ 300], 95.00th=[ 326], | 99.00th=[ 359], 99.50th=[ 368], 99.90th=[ 380], 99.95th=[ 388], | 99.99th=[ 388] bw ( KiB/s): min= 23, max= 175, per=25.18%, avg=90.65, stdev=30.03, samples=110 iops : min= 5, max= 43, avg=22.23, stdev= 7.59, samples=110 write: IOPS=23, BW=93.1KiB/s (95.3kB/s)(5180KiB/55660msec); 0 zone resets slat (usec): min=35, max=2887, avg=159.46, stdev=211.85 clat (msec): min=158, max=879, avg=464.00, stdev=123.78 lat (msec): min=158, max=879, avg=464.16, stdev=123.78 clat percentiles (msec): | 1.00th=[ 209], 5.00th=[ 271], 10.00th=[ 313], 20.00th=[ 359], | 30.00th=[ 393], 40.00th=[ 426], 50.00th=[ 456], 60.00th=[ 489], | 70.00th=[ 527], 80.00th=[ 567], 90.00th=[ 625], 95.00th=[ 667], | 99.00th=[ 835], 99.50th=[ 860], 99.90th=[ 877], 99.95th=[ 877], | 99.99th=[ 877] bw ( KiB/s): min= 40, max= 160, per=24.59%, avg=92.23, stdev=24.94, samples=110 iops : min= 10, max= 40, avg=22.62, stdev= 6.28, samples=110 lat (msec) : 10=0.08%, 50=0.20%, 100=0.12%, 250=37.70%, 500=43.24% lat (msec) : 750=17.89%, 1000=0.78% cpu : usr=0.11%, sys=0.92%, ctx=5068, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=361KiB/s (369kB/s), 89.5KiB/s-90.9KiB/s (91.7kB/s-93.1kB/s), io=19.6MiB (20.5MB), run=55590-55660msec WRITE: bw=375KiB/s (384kB/s), 93.1KiB/s-94.7KiB/s (95.3kB/s-96.0kB/s), io=20.4MiB (21.4MB), run=55590-55660msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=149396: Fri Nov 14 16:47:54 2025 read: IOPS=5, BW=221MiB/s (232MB/s)(40.0MiB/181msec) slat (nsec): min=14244k, max=14244k, avg=14244078.00, stdev= 0.00 clat (nsec): min=155072k, max=155072k, avg=155072197.00, stdev= 0.00 lat (nsec): min=169321k, max=169321k, avg=169320727.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 155], 5.00th=[ 155], 10.00th=[ 155], 20.00th=[ 155], | 30.00th=[ 155], 40.00th=[ 155], 50.00th=[ 155], 60.00th=[ 155], | 70.00th=[ 155], 80.00th=[ 155], 90.00th=[ 155], 95.00th=[ 155], | 99.00th=[ 155], 99.50th=[ 155], 99.90th=[ 155], 99.95th=[ 155], | 99.99th=[ 155] lat (msec) : 250=100.00% cpu : usr=0.00%, sys=6.11%, ctx=38, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=221MiB/s (232MB/s), 221MiB/s-221MiB/s (232MB/s-232MB/s), io=40.0MiB (41.9MB), run=181-181msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout