== sanity test 398c: run fio to test AIO ================= 23:27:29 (1763353649) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.176451 s, 238 MB/s osc.lustre-OST0000-osc-ffff914f50fb9000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=184305: Sun Nov 16 23:27:43 2025 write: IOPS=257, BW=1029KiB/s (1054kB/s)(10.0MiB/9950msec); 0 zone resets slat (usec): min=24, max=1642, avg=121.92, stdev=109.02 clat (msec): min=20, max=122, avg=61.97, stdev=12.63 lat (msec): min=20, max=122, avg=62.10, stdev=12.63 clat percentiles (msec): | 1.00th=[ 40], 5.00th=[ 46], 10.00th=[ 49], 20.00th=[ 52], | 30.00th=[ 55], 40.00th=[ 57], 50.00th=[ 61], 60.00th=[ 63], | 70.00th=[ 67], 80.00th=[ 72], 90.00th=[ 80], 95.00th=[ 86], | 99.00th=[ 104], 99.50th=[ 106], 99.90th=[ 117], 99.95th=[ 124], | 99.99th=[ 124] bw ( KiB/s): min= 792, max= 1165, per=25.21%, avg=1034.16, stdev=96.68, samples=19 iops : min= 198, max= 291, avg=258.42, stdev=24.20, samples=19 lat (msec) : 50=13.52%, 100=85.23%, 250=1.25% cpu : usr=0.43%, sys=2.41%, ctx=2746, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=184306: Sun Nov 16 23:27:43 2025 write: IOPS=256, BW=1027KiB/s (1052kB/s)(10.0MiB/9969msec); 0 zone resets slat (usec): min=32, max=1380, avg=123.71, stdev=105.11 clat (msec): min=12, max=114, avg=61.64, stdev=12.30 lat (msec): min=12, max=115, avg=61.77, stdev=12.30 clat percentiles (msec): | 1.00th=[ 39], 5.00th=[ 45], 10.00th=[ 49], 20.00th=[ 52], | 30.00th=[ 56], 40.00th=[ 57], 50.00th=[ 60], 60.00th=[ 63], | 70.00th=[ 66], 80.00th=[ 72], 90.00th=[ 80], 95.00th=[ 84], | 99.00th=[ 99], 99.50th=[ 103], 99.90th=[ 115], 99.95th=[ 115], | 99.99th=[ 115] bw ( KiB/s): min= 768, max= 1165, per=25.16%, avg=1032.16, stdev=96.79, samples=19 iops : min= 192, max= 291, avg=257.89, stdev=24.18, samples=19 lat (msec) : 20=0.12%, 50=13.91%, 100=85.35%, 250=0.62% cpu : usr=0.41%, sys=2.73%, ctx=2784, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=184307: Sun Nov 16 23:27:43 2025 write: IOPS=256, BW=1026KiB/s (1051kB/s)(10.0MiB/9980msec); 0 zone resets slat (usec): min=29, max=1808, avg=114.20, stdev=104.94 clat (msec): min=11, max=108, avg=61.69, stdev=12.00 lat (msec): min=11, max=108, avg=61.81, stdev=12.00 clat percentiles (msec): | 1.00th=[ 39], 5.00th=[ 45], 10.00th=[ 49], 20.00th=[ 53], | 30.00th=[ 56], 40.00th=[ 58], 50.00th=[ 60], 60.00th=[ 63], | 70.00th=[ 66], 80.00th=[ 72], 90.00th=[ 80], 95.00th=[ 84], | 99.00th=[ 95], 99.50th=[ 99], 99.90th=[ 107], 99.95th=[ 107], | 99.99th=[ 109] bw ( KiB/s): min= 768, max= 1229, per=25.08%, avg=1028.74, stdev=106.47, samples=19 iops : min= 192, max= 307, avg=257.05, stdev=26.62, samples=19 lat (msec) : 20=0.16%, 50=13.28%, 100=86.29%, 250=0.27% cpu : usr=0.41%, sys=2.24%, ctx=2685, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=184308: Sun Nov 16 23:27:43 2025 write: IOPS=256, BW=1026KiB/s (1050kB/s)(10.0MiB/9985msec); 0 zone resets slat (usec): min=31, max=1699, avg=118.30, stdev=105.65 clat (msec): min=24, max=106, avg=61.70, stdev=11.98 lat (msec): min=24, max=106, avg=61.82, stdev=11.98 clat percentiles (msec): | 1.00th=[ 41], 5.00th=[ 46], 10.00th=[ 49], 20.00th=[ 53], | 30.00th=[ 55], 40.00th=[ 57], 50.00th=[ 61], 60.00th=[ 63], | 70.00th=[ 66], 80.00th=[ 71], 90.00th=[ 80], 95.00th=[ 84], | 99.00th=[ 97], 99.50th=[ 103], 99.90th=[ 104], 99.95th=[ 107], | 99.99th=[ 107] bw ( KiB/s): min= 808, max= 1205, per=25.02%, avg=1026.21, stdev=92.17, samples=19 iops : min= 202, max= 301, avg=256.42, stdev=23.06, samples=19 lat (msec) : 50=14.18%, 100=85.31%, 250=0.51% cpu : usr=0.45%, sys=2.43%, ctx=2788, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=4102KiB/s (4201kB/s), 1026KiB/s-1029KiB/s (1050kB/s-1054kB/s), io=40.0MiB (41.9MB), run=9950-9985msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=184319: Sun Nov 16 23:27:56 2025 read: IOPS=134, BW=538KiB/s (551kB/s)(5048KiB/9379msec) slat (usec): min=1474, max=16592, avg=3892.77, stdev=1407.67 clat (usec): min=29110, max=92280, avg=55697.22, stdev=11845.46 lat (usec): min=33444, max=98284, avg=59594.88, stdev=12243.36 clat percentiles (usec): | 1.00th=[34341], 5.00th=[38536], 10.00th=[40633], 20.00th=[44827], | 30.00th=[47973], 40.00th=[51643], 50.00th=[54789], 60.00th=[58459], | 70.00th=[61604], 80.00th=[65274], 90.00th=[70779], 95.00th=[77071], | 99.00th=[85459], 99.50th=[88605], 99.90th=[91751], 99.95th=[92799], | 99.99th=[92799] bw ( KiB/s): min= 432, max= 680, per=25.20%, avg=539.06, stdev=65.58, samples=18 iops : min= 108, max= 170, avg=134.56, stdev=16.45, samples=18 write: IOPS=138, BW=554KiB/s (567kB/s)(5192KiB/9379msec); 0 zone resets slat (usec): min=35, max=2974, avg=129.97, stdev=175.73 clat (usec): min=8442, max=95237, avg=56777.87, stdev=10989.58 lat (usec): min=8534, max=95327, avg=56909.06, stdev=11005.81 clat percentiles (usec): | 1.00th=[36439], 5.00th=[40633], 10.00th=[43254], 20.00th=[46924], | 30.00th=[50070], 40.00th=[52691], 50.00th=[55837], 60.00th=[58983], | 70.00th=[62653], 80.00th=[66847], 90.00th=[71828], 95.00th=[77071], | 99.00th=[83362], 99.50th=[85459], 99.90th=[93848], 99.95th=[94897], | 99.99th=[94897] bw ( KiB/s): min= 400, max= 768, per=25.07%, avg=558.39, stdev=98.05, samples=18 iops : min= 100, max= 192, avg=139.39, stdev=24.60, samples=18 lat (msec) : 10=0.04%, 50=33.16%, 100=66.80% cpu : usr=0.44%, sys=3.68%, ctx=4791, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=184320: Sun Nov 16 23:27:56 2025 read: IOPS=133, BW=532KiB/s (545kB/s)(4976KiB/9349msec) slat (usec): min=33, max=15597, avg=3886.70, stdev=1479.71 clat (msec): min=10, max=125, avg=55.58, stdev=12.83 lat (msec): min=10, max=127, avg=59.47, stdev=13.26 clat percentiles (msec): | 1.00th=[ 33], 5.00th=[ 37], 10.00th=[ 41], 20.00th=[ 45], | 30.00th=[ 48], 40.00th=[ 52], 50.00th=[ 55], 60.00th=[ 58], | 70.00th=[ 62], 80.00th=[ 66], 90.00th=[ 73], 95.00th=[ 79], | 99.00th=[ 89], 99.50th=[ 96], 99.90th=[ 118], 99.95th=[ 126], | 99.99th=[ 126] bw ( KiB/s): min= 423, max= 712, per=24.94%, avg=533.56, stdev=74.67, samples=18 iops : min= 105, max= 178, avg=133.17, stdev=18.83, samples=18 write: IOPS=140, BW=563KiB/s (577kB/s)(5264KiB/9349msec); 0 zone resets slat (usec): min=33, max=1910, avg=116.39, stdev=108.64 clat (msec): min=22, max=125, avg=57.15, stdev=11.76 lat (msec): min=22, max=125, avg=57.26, stdev=11.76 clat percentiles (msec): | 1.00th=[ 35], 5.00th=[ 41], 10.00th=[ 43], 20.00th=[ 47], | 30.00th=[ 51], 40.00th=[ 54], 50.00th=[ 56], 60.00th=[ 60], | 70.00th=[ 63], 80.00th=[ 67], 90.00th=[ 73], 95.00th=[ 80], | 99.00th=[ 86], 99.50th=[ 91], 99.90th=[ 101], 99.95th=[ 126], | 99.99th=[ 126] bw ( KiB/s): min= 416, max= 824, per=25.49%, avg=567.61, stdev=102.24, samples=18 iops : min= 104, max= 206, avg=141.67, stdev=25.59, samples=18 lat (msec) : 20=0.20%, 50=33.36%, 100=66.33%, 250=0.12% cpu : usr=0.49%, sys=3.54%, ctx=4723, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=184321: Sun Nov 16 23:27:56 2025 read: IOPS=132, BW=532KiB/s (545kB/s)(4984KiB/9372msec) slat (usec): min=1352, max=17080, avg=3929.41, stdev=1484.37 clat (usec): min=11929, max=98186, avg=54854.98, stdev=11988.58 lat (msec): min=16, max=103, avg=58.79, stdev=12.45 clat percentiles (usec): | 1.00th=[33817], 5.00th=[38011], 10.00th=[40633], 20.00th=[44303], | 30.00th=[47449], 40.00th=[50070], 50.00th=[53740], 60.00th=[57410], | 70.00th=[61080], 80.00th=[64750], 90.00th=[70779], 95.00th=[76022], | 99.00th=[86508], 99.50th=[91751], 99.90th=[94897], 99.95th=[98042], | 99.99th=[98042] bw ( KiB/s): min= 408, max= 752, per=25.07%, avg=536.28, stdev=94.65, samples=18 iops : min= 102, max= 188, avg=133.83, stdev=23.73, samples=18 write: IOPS=140, BW=561KiB/s (574kB/s)(5256KiB/9372msec); 0 zone resets slat (usec): min=32, max=3537, avg=121.94, stdev=141.01 clat (msec): min=31, max=102, avg=57.65, stdev=11.96 lat (msec): min=31, max=102, avg=57.78, stdev=11.97 clat percentiles (msec): | 1.00th=[ 37], 5.00th=[ 41], 10.00th=[ 44], 20.00th=[ 48], | 30.00th=[ 51], 40.00th=[ 54], 50.00th=[ 57], 60.00th=[ 61], | 70.00th=[ 64], 80.00th=[ 68], 90.00th=[ 73], 95.00th=[ 79], | 99.00th=[ 92], 99.50th=[ 100], 99.90th=[ 103], 99.95th=[ 103], | 99.99th=[ 103] bw ( KiB/s): min= 447, max= 672, per=25.25%, avg=562.28, stdev=70.17, samples=18 iops : min= 111, max= 168, avg=140.33, stdev=17.66, samples=18 lat (msec) : 20=0.04%, 50=34.22%, 100=65.51%, 250=0.23% cpu : usr=0.48%, sys=3.62%, ctx=4756, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=184322: Sun Nov 16 23:27:56 2025 read: IOPS=135, BW=540KiB/s (553kB/s)(5060KiB/9364msec) slat (usec): min=1504, max=15302, avg=3882.35, stdev=1349.72 clat (msec): min=4, max=105, avg=55.38, stdev=12.16 lat (msec): min=19, max=108, avg=59.27, stdev=12.53 clat percentiles (msec): | 1.00th=[ 34], 5.00th=[ 39], 10.00th=[ 42], 20.00th=[ 45], | 30.00th=[ 48], 40.00th=[ 52], 50.00th=[ 54], 60.00th=[ 57], | 70.00th=[ 62], 80.00th=[ 66], 90.00th=[ 73], 95.00th=[ 77], | 99.00th=[ 88], 99.50th=[ 91], 99.90th=[ 103], 99.95th=[ 106], | 99.99th=[ 106] bw ( KiB/s): min= 439, max= 720, per=25.40%, avg=543.33, stdev=84.66, samples=18 iops : min= 109, max= 180, avg=135.61, stdev=21.26, samples=18 write: IOPS=138, BW=553KiB/s (566kB/s)(5180KiB/9364msec); 0 zone resets slat (usec): min=28, max=4322, avg=130.12, stdev=194.89 clat (usec): min=32919, max=99400, avg=56949.64, stdev=11510.41 lat (usec): min=32986, max=99457, avg=57081.02, stdev=11526.57 clat percentiles (usec): | 1.00th=[35914], 5.00th=[40109], 10.00th=[42730], 20.00th=[46400], | 30.00th=[49546], 40.00th=[53216], 50.00th=[56361], 60.00th=[59507], | 70.00th=[62653], 80.00th=[66847], 90.00th=[71828], 95.00th=[77071], | 99.00th=[87557], 99.50th=[90702], 99.90th=[94897], 99.95th=[99091], | 99.99th=[99091] bw ( KiB/s): min= 432, max= 752, per=24.91%, avg=554.72, stdev=88.90, samples=18 iops : min= 108, max= 188, avg=138.44, stdev=22.31, samples=18 lat (msec) : 10=0.04%, 50=33.67%, 100=66.21%, 250=0.08% cpu : usr=0.48%, sys=3.68%, ctx=4734, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2140KiB/s (2191kB/s), 532KiB/s-540KiB/s (545kB/s-553kB/s), io=19.6MiB (20.5MB), run=9349-9379msec WRITE: bw=2228KiB/s (2281kB/s), 553KiB/s-563KiB/s (566kB/s-577kB/s), io=20.4MiB (21.4MB), run=9349-9379msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=184327: Sun Nov 16 23:28:00 2025 read: IOPS=8, BW=345MiB/s (362MB/s)(40.0MiB/116msec) slat (nsec): min=12664k, max=12664k, avg=12664490.00, stdev= 0.00 clat (nsec): min=96772k, max=96772k, avg=96771978.00, stdev= 0.00 lat (nsec): min=109441k, max=109441k, avg=109441070.00, stdev= 0.00 clat percentiles (usec): | 1.00th=[96994], 5.00th=[96994], 10.00th=[96994], 20.00th=[96994], | 30.00th=[96994], 40.00th=[96994], 50.00th=[96994], 60.00th=[96994], | 70.00th=[96994], 80.00th=[96994], 90.00th=[96994], 95.00th=[96994], | 99.00th=[96994], 99.50th=[96994], 99.90th=[96994], 99.95th=[96994], | 99.99th=[96994] lat (msec) : 100=100.00% cpu : usr=0.00%, sys=9.57%, ctx=12, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=345MiB/s (362MB/s), 345MiB/s-345MiB/s (362MB/s-362MB/s), io=40.0MiB (41.9MB), run=116-116msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout