== sanity test 398c: run fio to test AIO ================= 08:03:34 (1790683414) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.273792 s, 153 MB/s osc.lustre-OST0000-osc-ffff9365d9ba9800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=146175: Tue Sep 29 08:06:34 2026 write: IOPS=15, BW=61.1KiB/s (62.5kB/s)(10.0MiB/167692msec); 0 zone resets slat (usec): min=23, max=23580, avg=295.79, stdev=1233.38 clat (msec): min=92, max=2440, avg=1047.26, stdev=381.04 lat (msec): min=93, max=2440, avg=1047.55, stdev=381.08 clat percentiles (msec): | 1.00th=[ 347], 5.00th=[ 502], 10.00th=[ 609], 20.00th=[ 751], | 30.00th=[ 818], 40.00th=[ 894], 50.00th=[ 978], 60.00th=[ 1083], | 70.00th=[ 1200], 80.00th=[ 1385], 90.00th=[ 1586], 95.00th=[ 1720], | 99.00th=[ 2056], 99.50th=[ 2198], 99.90th=[ 2400], 99.95th=[ 2433], | 99.99th=[ 2433] bw ( KiB/s): min= 7, max= 176, per=25.52%, avg=62.01, stdev=33.14, samples=324 iops : min= 1, max= 44, avg=14.95, stdev= 8.33, samples=324 lat (msec) : 100=0.04%, 250=0.59%, 500=4.30%, 750=15.16%, 1000=32.15% lat (msec) : 2000=46.41%, >=2000=1.37% cpu : usr=0.05%, sys=0.26%, ctx=1794, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=146176: Tue Sep 29 08:06:34 2026 write: IOPS=15, BW=60.0KiB/s (62.4kB/s)(10.0MiB/167953msec); 0 zone resets slat (usec): min=23, max=28323, avg=279.83, stdev=1165.66 clat (msec): min=234, max=2508, avg=1048.57, stdev=375.84 lat (msec): min=234, max=2508, avg=1048.85, stdev=375.90 clat percentiles (msec): | 1.00th=[ 401], 5.00th=[ 523], 10.00th=[ 625], 20.00th=[ 726], | 30.00th=[ 818], 40.00th=[ 894], 50.00th=[ 995], 60.00th=[ 1099], | 70.00th=[ 1200], 80.00th=[ 1351], 90.00th=[ 1586], 95.00th=[ 1737], | 99.00th=[ 2089], 99.50th=[ 2232], 99.90th=[ 2299], 99.95th=[ 2400], | 99.99th=[ 2500] bw ( KiB/s): min= 7, max= 168, per=25.37%, avg=61.65, stdev=29.90, samples=326 iops : min= 1, max= 42, avg=14.87, stdev= 7.53, samples=326 lat (msec) : 250=0.04%, 500=4.73%, 750=17.30%, 1000=28.40%, 2000=48.05% lat (msec) : >=2000=1.48% cpu : usr=0.06%, sys=0.27%, ctx=2040, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=146177: Tue Sep 29 08:06:34 2026 write: IOPS=15, BW=61.0KiB/s (62.5kB/s)(10.0MiB/167735msec); 0 zone resets slat (usec): min=24, max=31773, avg=311.31, stdev=1500.33 clat (msec): min=323, max=2506, avg=1047.03, stdev=377.51 lat (msec): min=323, max=2507, avg=1047.34, stdev=377.59 clat percentiles (msec): | 1.00th=[ 393], 5.00th=[ 510], 10.00th=[ 617], 20.00th=[ 743], | 30.00th=[ 818], 40.00th=[ 894], 50.00th=[ 986], 60.00th=[ 1083], | 70.00th=[ 1200], 80.00th=[ 1351], 90.00th=[ 1569], 95.00th=[ 1720], | 99.00th=[ 2089], 99.50th=[ 2232], 99.90th=[ 2400], 99.95th=[ 2400], | 99.99th=[ 2500] bw ( KiB/s): min= 7, max= 256, per=24.97%, avg=60.69, stdev=30.40, samples=331 iops : min= 1, max= 64, avg=14.62, stdev= 7.66, samples=331 lat (msec) : 500=4.73%, 750=16.29%, 1000=30.35%, 2000=46.76%, >=2000=1.88% cpu : usr=0.06%, sys=0.27%, ctx=1941, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=146178: Tue Sep 29 08:06:34 2026 write: IOPS=15, BW=60.0KiB/s (62.4kB/s)(10.0MiB/167954msec); 0 zone resets slat (usec): min=24, max=43146, avg=286.63, stdev=1426.55 clat (msec): min=337, max=2595, avg=1048.35, stdev=376.62 lat (msec): min=337, max=2595, avg=1048.64, stdev=376.67 clat percentiles (msec): | 1.00th=[ 355], 5.00th=[ 523], 10.00th=[ 617], 20.00th=[ 743], | 30.00th=[ 827], 40.00th=[ 902], 50.00th=[ 986], 60.00th=[ 1083], | 70.00th=[ 1200], 80.00th=[ 1368], 90.00th=[ 1552], 95.00th=[ 1754], | 99.00th=[ 2056], 99.50th=[ 2265], 99.90th=[ 2500], 99.95th=[ 2567], | 99.99th=[ 2601] bw ( KiB/s): min= 7, max= 224, per=25.68%, avg=62.41, stdev=32.61, samples=322 iops : min= 1, max= 56, avg=15.06, stdev= 8.20, samples=322 lat (msec) : 500=4.80%, 750=15.98%, 1000=31.25%, 2000=46.17%, >=2000=1.80% cpu : usr=0.05%, sys=0.24%, ctx=1683, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=244KiB/s (250kB/s), 60.0KiB/s-61.1KiB/s (62.4kB/s-62.5kB/s), io=40.0MiB (41.9MB), run=167692-167954msec 8% of I/O are 1-page (PAGE_SIZE=4096), max 50% mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=146212: Tue Sep 29 08:08:27 2026 read: IOPS=12, BW=48.3KiB/s (49.5kB/s)(5048KiB/104530msec) slat (usec): min=2563, max=63485, avg=14666.83, stdev=7890.03 clat (msec): min=14, max=778, avg=400.12, stdev=108.60 lat (msec): min=24, max=794, avg=414.79, stdev=110.04 clat percentiles (msec): | 1.00th=[ 80], 5.00th=[ 259], 10.00th=[ 279], 20.00th=[ 317], | 30.00th=[ 347], 40.00th=[ 368], 50.00th=[ 388], 60.00th=[ 414], | 70.00th=[ 439], 80.00th=[ 485], 90.00th=[ 542], 95.00th=[ 609], | 99.00th=[ 701], 99.50th=[ 735], 99.90th=[ 776], 99.95th=[ 776], | 99.99th=[ 776] bw ( KiB/s): min= 7, max= 103, per=24.88%, avg=47.52, stdev=21.99, samples=207 iops : min= 1, max= 25, avg=11.33, stdev= 5.52, samples=207 write: IOPS=12, BW=49.7KiB/s (50.9kB/s)(5192KiB/104530msec); 0 zone resets slat (usec): min=29, max=26137, avg=388.09, stdev=1365.49 clat (msec): min=276, max=1647, avg=882.62, stdev=231.59 lat (msec): min=276, max=1647, avg=883.01, stdev=231.58 clat percentiles (msec): | 1.00th=[ 430], 5.00th=[ 514], 10.00th=[ 592], 20.00th=[ 684], | 30.00th=[ 760], 40.00th=[ 818], 50.00th=[ 869], 60.00th=[ 927], | 70.00th=[ 986], 80.00th=[ 1062], 90.00th=[ 1183], 95.00th=[ 1284], | 99.00th=[ 1502], 99.50th=[ 1536], 99.90th=[ 1620], 99.95th=[ 1653], | 99.99th=[ 1653] bw ( KiB/s): min= 7, max= 103, per=24.57%, avg=48.89, stdev=21.20, samples=207 iops : min= 1, max= 25, avg=11.66, stdev= 5.35, samples=207 lat (msec) : 20=0.12%, 50=0.12%, 100=0.31%, 250=1.48%, 500=41.13% lat (msec) : 750=20.43%, 1000=22.07%, 2000=14.34% cpu : usr=0.11%, sys=0.75%, ctx=4459, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=146213: Tue Sep 29 08:08:27 2026 read: IOPS=11, BW=47.7KiB/s (48.8kB/s)(4976KiB/104325msec) slat (usec): min=3266, max=69850, avg=14544.72, stdev=8004.68 clat (msec): min=15, max=832, avg=399.73, stdev=108.75 lat (msec): min=22, max=837, avg=414.28, stdev=109.76 clat percentiles (msec): | 1.00th=[ 171], 5.00th=[ 251], 10.00th=[ 279], 20.00th=[ 317], | 30.00th=[ 338], 40.00th=[ 368], 50.00th=[ 388], 60.00th=[ 409], | 70.00th=[ 439], 80.00th=[ 485], 90.00th=[ 542], 95.00th=[ 592], | 99.00th=[ 684], 99.50th=[ 768], 99.90th=[ 793], 99.95th=[ 835], | 99.99th=[ 835] bw ( KiB/s): min= 7, max= 127, per=24.82%, avg=47.40, stdev=23.32, samples=205 iops : min= 1, max= 31, avg=11.29, stdev= 5.85, samples=205 write: IOPS=12, BW=50.5KiB/s (51.7kB/s)(5264KiB/104325msec); 0 zone resets slat (usec): min=31, max=21704, avg=384.70, stdev=1278.50 clat (msec): min=297, max=1636, avg=874.38, stdev=229.83 lat (msec): min=297, max=1636, avg=874.77, stdev=229.85 clat percentiles (msec): | 1.00th=[ 447], 5.00th=[ 514], 10.00th=[ 584], 20.00th=[ 667], | 30.00th=[ 735], 40.00th=[ 810], 50.00th=[ 860], 60.00th=[ 919], | 70.00th=[ 978], 80.00th=[ 1045], 90.00th=[ 1167], 95.00th=[ 1301], | 99.00th=[ 1485], 99.50th=[ 1536], 99.90th=[ 1636], 99.95th=[ 1636], | 99.99th=[ 1636] bw ( KiB/s): min= 7, max= 112, per=25.04%, avg=49.83, stdev=21.54, samples=207 iops : min= 1, max= 28, avg=11.90, stdev= 5.44, samples=207 lat (msec) : 20=0.04%, 50=0.23%, 100=0.20%, 250=1.91%, 500=39.65% lat (msec) : 750=22.50%, 1000=21.52%, 2000=13.95% cpu : usr=0.14%, sys=0.75%, ctx=4557, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=146214: Tue Sep 29 08:08:27 2026 read: IOPS=11, BW=47.6KiB/s (48.7kB/s)(4984KiB/104713msec) slat (usec): min=2863, max=66845, avg=14564.30, stdev=7682.23 clat (msec): min=22, max=795, avg=402.77, stdev=108.29 lat (msec): min=31, max=805, avg=417.35, stdev=109.64 clat percentiles (msec): | 1.00th=[ 192], 5.00th=[ 255], 10.00th=[ 279], 20.00th=[ 317], | 30.00th=[ 347], 40.00th=[ 368], 50.00th=[ 388], 60.00th=[ 414], | 70.00th=[ 439], 80.00th=[ 489], 90.00th=[ 558], 95.00th=[ 600], | 99.00th=[ 693], 99.50th=[ 726], 99.90th=[ 776], 99.95th=[ 793], | 99.99th=[ 793] bw ( KiB/s): min= 7, max= 119, per=24.88%, avg=47.52, stdev=24.41, samples=206 iops : min= 1, max= 29, avg=11.33, stdev= 6.12, samples=206 write: IOPS=12, BW=50.2KiB/s (51.4kB/s)(5256KiB/104713msec); 0 zone resets slat (usec): min=35, max=19310, avg=334.49, stdev=1032.37 clat (msec): min=152, max=1619, avg=877.17, stdev=240.28 lat (msec): min=154, max=1619, avg=877.50, stdev=240.25 clat percentiles (msec): | 1.00th=[ 326], 5.00th=[ 506], 10.00th=[ 575], 20.00th=[ 676], | 30.00th=[ 751], 40.00th=[ 818], 50.00th=[ 869], 60.00th=[ 927], | 70.00th=[ 986], 80.00th=[ 1053], 90.00th=[ 1200], 95.00th=[ 1318], | 99.00th=[ 1502], 99.50th=[ 1536], 99.90th=[ 1620], 99.95th=[ 1620], | 99.99th=[ 1620] bw ( KiB/s): min= 7, max= 112, per=24.87%, avg=49.50, stdev=19.81, samples=207 iops : min= 1, max= 28, avg=11.82, stdev= 5.02, samples=207 lat (msec) : 50=0.12%, 100=0.16%, 250=2.15%, 500=39.84%, 750=21.45% lat (msec) : 1000=22.03%, 2000=14.26% cpu : usr=0.14%, sys=0.79%, ctx=4338, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=146215: Tue Sep 29 08:08:27 2026 read: IOPS=12, BW=48.7KiB/s (49.9kB/s)(5060KiB/103913msec) slat (usec): min=64, max=65698, avg=14979.09, stdev=8173.36 clat (msec): min=9, max=784, avg=396.21, stdev=112.93 lat (msec): min=9, max=792, avg=411.20, stdev=114.60 clat percentiles (msec): | 1.00th=[ 69], 5.00th=[ 243], 10.00th=[ 271], 20.00th=[ 313], | 30.00th=[ 338], 40.00th=[ 359], 50.00th=[ 384], 60.00th=[ 409], | 70.00th=[ 439], 80.00th=[ 485], 90.00th=[ 550], 95.00th=[ 609], | 99.00th=[ 693], 99.50th=[ 726], 99.90th=[ 760], 99.95th=[ 785], | 99.99th=[ 785] bw ( KiB/s): min= 7, max= 111, per=25.47%, avg=48.65, stdev=21.10, samples=204 iops : min= 1, max= 27, avg=11.60, stdev= 5.30, samples=204 write: IOPS=12, BW=49.8KiB/s (51.0kB/s)(5180KiB/103913msec); 0 zone resets slat (usec): min=32, max=18477, avg=390.62, stdev=1199.14 clat (msec): min=74, max=1750, avg=881.15, stdev=240.94 lat (msec): min=74, max=1750, avg=881.55, stdev=240.90 clat percentiles (msec): | 1.00th=[ 414], 5.00th=[ 518], 10.00th=[ 592], 20.00th=[ 676], | 30.00th=[ 743], 40.00th=[ 810], 50.00th=[ 860], 60.00th=[ 944], | 70.00th=[ 1003], 80.00th=[ 1062], 90.00th=[ 1183], 95.00th=[ 1318], | 99.00th=[ 1519], 99.50th=[ 1620], 99.90th=[ 1754], 99.95th=[ 1754], | 99.99th=[ 1754] bw ( KiB/s): min= 7, max= 104, per=24.77%, avg=49.30, stdev=18.94, samples=205 iops : min= 1, max= 26, avg=11.77, stdev= 4.77, samples=205 lat (msec) : 10=0.04%, 20=0.08%, 50=0.27%, 100=0.31%, 250=2.70% lat (msec) : 500=39.26%, 750=22.38%, 1000=19.41%, 2000=15.55% cpu : usr=0.15%, sys=0.78%, ctx=4754, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=192KiB/s (196kB/s), 47.6KiB/s-48.7KiB/s (48.7kB/s-49.9kB/s), io=19.6MiB (20.5MB), run=103913-104713msec WRITE: bw=200KiB/s (204kB/s), 49.7KiB/s-50.5KiB/s (50.9kB/s-51.7kB/s), io=20.4MiB (21.4MB), run=103913-104713msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=146234: Tue Sep 29 08:08:36 2026 read: IOPS=1, BW=76.5MiB/s (80.2MB/s)(40.0MiB/523msec) slat (nsec): min=32757k, max=32757k, avg=32756511.00, stdev= 0.00 clat (nsec): min=473329k, max=473329k, avg=473328720.00, stdev= 0.00 lat (nsec): min=506092k, max=506092k, avg=506092014.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 472], 5.00th=[ 472], 10.00th=[ 472], 20.00th=[ 472], | 30.00th=[ 472], 40.00th=[ 472], 50.00th=[ 472], 60.00th=[ 472], | 70.00th=[ 472], 80.00th=[ 472], 90.00th=[ 472], 95.00th=[ 472], | 99.00th=[ 472], 99.50th=[ 472], 99.90th=[ 472], 99.95th=[ 472], | 99.99th=[ 472] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=2.68%, ctx=7, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=76.5MiB/s (80.2MB/s), 76.5MiB/s-76.5MiB/s (80.2MB/s-80.2MB/s), io=40.0MiB (41.9MB), run=523-523msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout