== sanity test 398c: run fio to test AIO ================= 22:18:04 (1782613084) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.526408 s, 79.7 MB/s osc.lustre-OST0000-osc-ffff91092cedd800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=145117: Sat Jun 27 22:21:16 2026 write: IOPS=13, BW=55.0KiB/s (57.3kB/s)(10.0MiB/182907msec); 0 zone resets slat (usec): min=35, max=22723, avg=364.10, stdev=1259.28 clat (msec): min=195, max=3114, avg=1141.44, stdev=387.44 lat (msec): min=195, max=3114, avg=1141.81, stdev=387.47 clat percentiles (msec): | 1.00th=[ 372], 5.00th=[ 567], 10.00th=[ 743], 20.00th=[ 860], | 30.00th=[ 944], 40.00th=[ 1011], 50.00th=[ 1083], 60.00th=[ 1167], | 70.00th=[ 1284], 80.00th=[ 1418], 90.00th=[ 1670], 95.00th=[ 1821], | 99.00th=[ 2400], 99.50th=[ 2467], 99.90th=[ 3004], 99.95th=[ 3104], | 99.99th=[ 3104] bw ( KiB/s): min= 7, max= 253, per=25.61%, avg=57.11, stdev=32.69, samples=352 iops : min= 1, max= 63, avg=13.77, stdev= 8.20, samples=352 lat (msec) : 250=0.04%, 500=4.49%, 750=5.90%, 1000=28.24%, 2000=58.48% lat (msec) : >=2000=2.85% cpu : usr=0.04%, sys=0.28%, ctx=1839, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145118: Sat Jun 27 22:21:16 2026 write: IOPS=14, BW=56.1KiB/s (57.4kB/s)(10.0MiB/182651msec); 0 zone resets slat (usec): min=38, max=23153, avg=339.34, stdev=1068.97 clat (msec): min=161, max=2982, avg=1140.39, stdev=389.33 lat (msec): min=161, max=2982, avg=1140.73, stdev=389.33 clat percentiles (msec): | 1.00th=[ 372], 5.00th=[ 550], 10.00th=[ 726], 20.00th=[ 844], | 30.00th=[ 944], 40.00th=[ 1020], 50.00th=[ 1083], 60.00th=[ 1183], | 70.00th=[ 1284], 80.00th=[ 1418], 90.00th=[ 1653], 95.00th=[ 1854], | 99.00th=[ 2333], 99.50th=[ 2500], 99.90th=[ 2903], 99.95th=[ 2970], | 99.99th=[ 2970] bw ( KiB/s): min= 7, max= 174, per=25.24%, avg=56.29, stdev=27.66, samples=357 iops : min= 1, max= 43, avg=13.56, stdev= 6.94, samples=357 lat (msec) : 250=0.16%, 500=4.26%, 750=6.99%, 1000=25.51%, 2000=60.16% lat (msec) : >=2000=2.93% cpu : usr=0.04%, sys=0.31%, ctx=2147, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145119: Sat Jun 27 22:21:16 2026 write: IOPS=14, BW=56.0KiB/s (57.4kB/s)(10.0MiB/182723msec); 0 zone resets slat (usec): min=36, max=25000, avg=333.11, stdev=1141.33 clat (msec): min=195, max=3051, avg=1140.49, stdev=392.07 lat (msec): min=195, max=3051, avg=1140.83, stdev=392.08 clat percentiles (msec): | 1.00th=[ 334], 5.00th=[ 558], 10.00th=[ 726], 20.00th=[ 844], | 30.00th=[ 936], 40.00th=[ 1011], 50.00th=[ 1083], 60.00th=[ 1183], | 70.00th=[ 1284], 80.00th=[ 1418], 90.00th=[ 1653], 95.00th=[ 1838], | 99.00th=[ 2366], 99.50th=[ 2567], 99.90th=[ 2937], 99.95th=[ 3037], | 99.99th=[ 3037] bw ( KiB/s): min= 7, max= 175, per=26.05%, avg=58.10, stdev=31.01, samples=346 iops : min= 1, max= 43, avg=14.01, stdev= 7.78, samples=346 lat (msec) : 250=0.08%, 500=4.10%, 750=7.34%, 1000=26.52%, 2000=59.14% lat (msec) : >=2000=2.81% cpu : usr=0.04%, sys=0.28%, ctx=1905, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145120: Sat Jun 27 22:21:16 2026 write: IOPS=14, BW=56.0KiB/s (57.4kB/s)(10.0MiB/182722msec); 0 zone resets slat (usec): min=41, max=40087, avg=380.29, stdev=1326.57 clat (msec): min=246, max=2954, avg=1140.22, stdev=385.59 lat (msec): min=246, max=2954, avg=1140.60, stdev=385.57 clat percentiles (msec): | 1.00th=[ 334], 5.00th=[ 575], 10.00th=[ 743], 20.00th=[ 844], | 30.00th=[ 944], 40.00th=[ 1011], 50.00th=[ 1083], 60.00th=[ 1183], | 70.00th=[ 1284], 80.00th=[ 1418], 90.00th=[ 1636], 95.00th=[ 1838], | 99.00th=[ 2299], 99.50th=[ 2467], 99.90th=[ 2869], 99.95th=[ 2869], | 99.99th=[ 2970] bw ( KiB/s): min= 7, max= 240, per=25.17%, avg=56.13, stdev=31.86, samples=358 iops : min= 1, max= 60, avg=13.51, stdev= 7.96, samples=358 lat (msec) : 250=0.62%, 500=3.36%, 750=6.48%, 1000=27.85%, 2000=58.63% lat (msec) : >=2000=3.05% cpu : usr=0.05%, sys=0.28%, ctx=2123, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=224KiB/s (229kB/s), 55.0KiB/s-56.1KiB/s (57.3kB/s-57.4kB/s), io=40.0MiB (41.9MB), run=182651-182907msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=145156: Sat Jun 27 22:23:32 2026 read: IOPS=10, BW=40.1KiB/s (41.1kB/s)(5048KiB/125777msec) slat (usec): min=58, max=68946, avg=17380.89, stdev=9876.77 clat (msec): min=25, max=1849, avg=501.50, stdev=177.92 lat (msec): min=25, max=1860, avg=518.89, stdev=179.71 clat percentiles (msec): | 1.00th=[ 120], 5.00th=[ 296], 10.00th=[ 330], 20.00th=[ 372], | 30.00th=[ 409], 40.00th=[ 435], 50.00th=[ 464], 60.00th=[ 502], | 70.00th=[ 558], 80.00th=[ 617], 90.00th=[ 726], 95.00th=[ 818], | 99.00th=[ 1099], 99.50th=[ 1234], 99.90th=[ 1536], 99.95th=[ 1854], | 99.99th=[ 1854] bw ( KiB/s): min= 7, max= 119, per=25.18%, avg=40.04, stdev=19.67, samples=246 iops : min= 1, max= 29, avg= 9.42, stdev= 4.93, samples=246 write: IOPS=10, BW=41.3KiB/s (42.3kB/s)(5192KiB/125777msec); 0 zone resets slat (usec): min=40, max=17511, avg=498.74, stdev=1279.24 clat (msec): min=232, max=3119, avg=1043.90, stdev=337.36 lat (msec): min=232, max=3119, avg=1044.40, stdev=337.34 clat percentiles (msec): | 1.00th=[ 368], 5.00th=[ 592], 10.00th=[ 667], 20.00th=[ 776], | 30.00th=[ 869], 40.00th=[ 936], 50.00th=[ 1003], 60.00th=[ 1070], | 70.00th=[ 1150], 80.00th=[ 1267], 90.00th=[ 1469], 95.00th=[ 1603], | 99.00th=[ 2232], 99.50th=[ 2433], 99.90th=[ 2937], 99.95th=[ 3104], | 99.99th=[ 3104] bw ( KiB/s): min= 7, max= 104, per=24.80%, avg=40.93, stdev=20.04, samples=247 iops : min= 1, max= 26, avg= 9.65, stdev= 5.04, samples=247 lat (msec) : 50=0.20%, 100=0.23%, 250=0.86%, 500=29.14%, 750=23.24% lat (msec) : 1000=20.08%, 2000=25.51%, >=2000=0.74% cpu : usr=0.14%, sys=0.75%, ctx=4761, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145157: Sat Jun 27 22:23:32 2026 read: IOPS=9, BW=39.5KiB/s (40.5kB/s)(4976KiB/125896msec) slat (usec): min=3097, max=66961, avg=17407.79, stdev=9159.58 clat (msec): min=34, max=1338, avg=500.20, stdev=158.95 lat (msec): min=47, max=1367, avg=517.63, stdev=160.56 clat percentiles (msec): | 1.00th=[ 215], 5.00th=[ 305], 10.00th=[ 334], 20.00th=[ 376], | 30.00th=[ 405], 40.00th=[ 435], 50.00th=[ 468], 60.00th=[ 502], | 70.00th=[ 558], 80.00th=[ 634], 90.00th=[ 726], 95.00th=[ 793], | 99.00th=[ 944], 99.50th=[ 1028], 99.90th=[ 1217], 99.95th=[ 1334], | 99.99th=[ 1334] bw ( KiB/s): min= 7, max= 95, per=25.11%, avg=39.92, stdev=17.53, samples=243 iops : min= 1, max= 23, avg= 9.39, stdev= 4.41, samples=243 write: IOPS=10, BW=41.8KiB/s (42.8kB/s)(5264KiB/125896msec); 0 zone resets slat (usec): min=45, max=24468, avg=522.24, stdev=1578.55 clat (msec): min=286, max=2812, avg=1037.65, stdev=340.15 lat (msec): min=286, max=2812, avg=1038.18, stdev=340.18 clat percentiles (msec): | 1.00th=[ 351], 5.00th=[ 584], 10.00th=[ 676], 20.00th=[ 785], | 30.00th=[ 860], 40.00th=[ 927], 50.00th=[ 986], 60.00th=[ 1070], | 70.00th=[ 1150], 80.00th=[ 1267], 90.00th=[ 1469], 95.00th=[ 1569], | 99.00th=[ 2265], 99.50th=[ 2635], 99.90th=[ 2802], 99.95th=[ 2802], | 99.99th=[ 2802] bw ( KiB/s): min= 7, max= 111, per=25.07%, avg=41.37, stdev=21.28, samples=248 iops : min= 1, max= 27, avg= 9.76, stdev= 5.37, samples=248 lat (msec) : 50=0.08%, 100=0.20%, 250=0.35%, 500=29.49%, 750=23.55% lat (msec) : 1000=21.29%, 2000=24.22%, >=2000=0.82% cpu : usr=0.11%, sys=0.79%, ctx=4727, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145158: Sat Jun 27 22:23:32 2026 read: IOPS=9, BW=39.5KiB/s (40.5kB/s)(4984KiB/126027msec) slat (usec): min=3034, max=90897, avg=17034.00, stdev=9685.15 clat (msec): min=23, max=1352, avg=490.54, stdev=156.08 lat (msec): min=35, max=1369, avg=507.59, stdev=157.59 clat percentiles (msec): | 1.00th=[ 115], 5.00th=[ 292], 10.00th=[ 334], 20.00th=[ 376], | 30.00th=[ 409], 40.00th=[ 435], 50.00th=[ 464], 60.00th=[ 493], | 70.00th=[ 535], 80.00th=[ 609], 90.00th=[ 701], 95.00th=[ 776], | 99.00th=[ 944], 99.50th=[ 1036], 99.90th=[ 1150], 99.95th=[ 1351], | 99.99th=[ 1351] bw ( KiB/s): min= 7, max= 104, per=25.21%, avg=40.08, stdev=20.35, samples=242 iops : min= 1, max= 26, avg= 9.44, stdev= 5.10, samples=242 write: IOPS=10, BW=41.7KiB/s (42.7kB/s)(5256KiB/126027msec); 0 zone resets slat (usec): min=40, max=24349, avg=513.09, stdev=1500.46 clat (msec): min=81, max=2881, avg=1049.01, stdev=342.82 lat (msec): min=81, max=2881, avg=1049.54, stdev=342.82 clat percentiles (msec): | 1.00th=[ 305], 5.00th=[ 592], 10.00th=[ 684], 20.00th=[ 793], | 30.00th=[ 869], 40.00th=[ 936], 50.00th=[ 1011], 60.00th=[ 1083], | 70.00th=[ 1167], 80.00th=[ 1284], 90.00th=[ 1485], 95.00th=[ 1636], | 99.00th=[ 2198], 99.50th=[ 2400], 99.90th=[ 2735], 99.95th=[ 2869], | 99.99th=[ 2869] bw ( KiB/s): min= 7, max= 96, per=24.78%, avg=40.90, stdev=18.90, samples=248 iops : min= 1, max= 24, avg= 9.64, stdev= 4.76, samples=248 lat (msec) : 50=0.12%, 100=0.35%, 250=1.05%, 500=29.80%, 750=22.42% lat (msec) : 1000=19.65%, 2000=25.62%, >=2000=0.98% cpu : usr=0.12%, sys=0.78%, ctx=4845, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145159: Sat Jun 27 22:23:32 2026 read: IOPS=10, BW=40.2KiB/s (41.2kB/s)(5060KiB/125911msec) slat (usec): min=3183, max=76137, avg=17438.84, stdev=9729.87 clat (msec): min=57, max=1353, avg=493.54, stdev=156.20 lat (msec): min=68, max=1369, avg=510.98, stdev=157.67 clat percentiles (msec): | 1.00th=[ 247], 5.00th=[ 296], 10.00th=[ 330], 20.00th=[ 372], | 30.00th=[ 405], 40.00th=[ 430], 50.00th=[ 460], 60.00th=[ 493], | 70.00th=[ 542], 80.00th=[ 609], 90.00th=[ 693], 95.00th=[ 793], | 99.00th=[ 995], 99.50th=[ 1028], 99.90th=[ 1217], 99.95th=[ 1351], | 99.99th=[ 1351] bw ( KiB/s): min= 7, max= 96, per=25.49%, avg=40.53, stdev=19.28, samples=244 iops : min= 1, max= 24, avg= 9.55, stdev= 4.86, samples=244 write: IOPS=10, BW=41.1KiB/s (42.1kB/s)(5180KiB/125911msec); 0 zone resets slat (usec): min=50, max=19544, avg=510.33, stdev=1312.44 clat (msec): min=237, max=2786, avg=1052.83, stdev=340.72 lat (msec): min=237, max=2786, avg=1053.34, stdev=340.70 clat percentiles (msec): | 1.00th=[ 397], 5.00th=[ 600], 10.00th=[ 676], 20.00th=[ 785], | 30.00th=[ 852], 40.00th=[ 936], 50.00th=[ 1011], 60.00th=[ 1083], | 70.00th=[ 1167], 80.00th=[ 1301], 90.00th=[ 1485], 95.00th=[ 1620], | 99.00th=[ 2299], 99.50th=[ 2400], 99.90th=[ 2567], 99.95th=[ 2802], | 99.99th=[ 2802] bw ( KiB/s): min= 7, max= 111, per=24.61%, avg=40.61, stdev=18.92, samples=248 iops : min= 1, max= 27, avg= 9.56, stdev= 4.78, samples=248 lat (msec) : 100=0.12%, 250=0.47%, 500=31.02%, 750=22.58%, 1000=19.49% lat (msec) : 2000=25.39%, >=2000=0.94% cpu : usr=0.10%, sys=0.77%, ctx=4753, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=159KiB/s (163kB/s), 39.5KiB/s-40.2KiB/s (40.5kB/s-41.2kB/s), io=19.6MiB (20.5MB), run=125777-126027msec WRITE: bw=166KiB/s (170kB/s), 41.1KiB/s-41.8KiB/s (42.1kB/s-42.8kB/s), io=20.4MiB (21.4MB), run=125777-126027msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=145183: Sat Jun 27 22:23:44 2026 read: IOPS=1, BW=70.5MiB/s (73.0MB/s)(40.0MiB/567msec) slat (nsec): min=41734k, max=41734k, avg=41734395.00, stdev= 0.00 clat (nsec): min=493231k, max=493231k, avg=493230762.00, stdev= 0.00 lat (nsec): min=534971k, max=534971k, avg=534971222.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 493], 5.00th=[ 493], 10.00th=[ 493], 20.00th=[ 493], | 30.00th=[ 493], 40.00th=[ 493], 50.00th=[ 493], 60.00th=[ 493], | 70.00th=[ 493], 80.00th=[ 493], 90.00th=[ 493], 95.00th=[ 493], | 99.00th=[ 493], 99.50th=[ 493], 99.90th=[ 493], 99.95th=[ 493], | 99.99th=[ 493] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=2.83%, ctx=45, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=70.5MiB/s (73.0MB/s), 70.5MiB/s-70.5MiB/s (73.0MB/s-73.0MB/s), io=40.0MiB (41.9MB), run=567-567msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout