== sanity test 398c: run fio to test AIO ================= 19:05:00 (1746140700) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB) copied, 0.176752 s, 237 MB/s osc.lustre-OST0000-osc-ffff8800a7c03000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=22190: Thu May 1 19:05:13 2025 write: IOPS=245, BW=983KiB/s (1006kB/s)(10.0MiB/10420msec) slat (usec): min=10, max=2003, avg=95.04, stdev=161.53 clat (msec): min=6, max=117, avg=64.84, stdev=13.99 lat (msec): min=6, max=117, avg=64.93, stdev=14.00 clat percentiles (msec): | 1.00th=[ 41], 5.00th=[ 46], 10.00th=[ 51], 20.00th=[ 54], | 30.00th=[ 57], 40.00th=[ 60], 50.00th=[ 63], 60.00th=[ 66], | 70.00th=[ 70], 80.00th=[ 77], 90.00th=[ 85], 95.00th=[ 92], | 99.00th=[ 104], 99.50th=[ 114], 99.90th=[ 117], 99.95th=[ 117], | 99.99th=[ 117] bw ( KiB/s): min= 662, max= 1184, per=24.79%, avg=972.75, stdev=137.74, samples=20 iops : min= 165, max= 296, avg=243.10, stdev=34.51, samples=20 lat (msec) : 10=0.08%, 20=0.12%, 50=9.73%, 100=88.20%, 250=1.88% cpu : usr=0.73%, sys=2.99%, ctx=1447, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=22191: Thu May 1 19:05:13 2025 write: IOPS=245, BW=981KiB/s (1005kB/s)(10.0MiB/10438msec) slat (usec): min=11, max=3305, avg=96.42, stdev=173.03 clat (msec): min=8, max=117, avg=64.84, stdev=13.80 lat (msec): min=8, max=117, avg=64.94, stdev=13.81 clat percentiles (msec): | 1.00th=[ 41], 5.00th=[ 48], 10.00th=[ 51], 20.00th=[ 55], | 30.00th=[ 57], 40.00th=[ 60], 50.00th=[ 63], 60.00th=[ 66], | 70.00th=[ 69], 80.00th=[ 77], 90.00th=[ 85], 95.00th=[ 92], | 99.00th=[ 104], 99.50th=[ 114], 99.90th=[ 117], 99.95th=[ 118], | 99.99th=[ 118] bw ( KiB/s): min= 672, max= 1208, per=24.73%, avg=970.40, stdev=143.39, samples=20 iops : min= 168, max= 302, avg=242.55, stdev=35.83, samples=20 lat (msec) : 10=0.08%, 50=9.45%, 100=88.52%, 250=1.95% cpu : usr=0.00%, sys=3.75%, ctx=1355, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=22192: Thu May 1 19:05:13 2025 write: IOPS=245, BW=981KiB/s (1005kB/s)(10.0MiB/10438msec) slat (usec): min=11, max=3367, avg=93.39, stdev=179.43 clat (msec): min=12, max=115, avg=64.90, stdev=14.01 lat (msec): min=13, max=116, avg=64.99, stdev=14.01 clat percentiles (msec): | 1.00th=[ 40], 5.00th=[ 46], 10.00th=[ 51], 20.00th=[ 54], | 30.00th=[ 57], 40.00th=[ 60], 50.00th=[ 63], 60.00th=[ 66], | 70.00th=[ 69], 80.00th=[ 77], 90.00th=[ 85], 95.00th=[ 94], | 99.00th=[ 104], 99.50th=[ 113], 99.90th=[ 114], 99.95th=[ 115], | 99.99th=[ 115] bw ( KiB/s): min= 720, max= 1200, per=24.69%, avg=968.65, stdev=139.12, samples=20 iops : min= 180, max= 300, avg=242.10, stdev=34.79, samples=20 lat (msec) : 20=0.08%, 50=8.98%, 100=88.67%, 250=2.27% cpu : usr=2.13%, sys=1.42%, ctx=1236, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=22193: Thu May 1 19:05:13 2025 write: IOPS=245, BW=984KiB/s (1007kB/s)(10.0MiB/10408msec) slat (usec): min=10, max=2829, avg=86.84, stdev=174.97 clat (msec): min=5, max=122, avg=64.79, stdev=14.18 lat (msec): min=5, max=122, avg=64.88, stdev=14.19 clat percentiles (msec): | 1.00th=[ 37], 5.00th=[ 46], 10.00th=[ 51], 20.00th=[ 55], | 30.00th=[ 58], 40.00th=[ 61], 50.00th=[ 63], 60.00th=[ 66], | 70.00th=[ 70], 80.00th=[ 75], 90.00th=[ 85], 95.00th=[ 91], | 99.00th=[ 104], 99.50th=[ 114], 99.90th=[ 120], 99.95th=[ 123], | 99.99th=[ 123] bw ( KiB/s): min= 688, max= 1216, per=24.86%, avg=975.65, stdev=129.35, samples=20 iops : min= 172, max= 304, avg=243.85, stdev=32.35, samples=20 lat (msec) : 10=0.04%, 20=0.08%, 50=9.34%, 100=88.75%, 250=1.80% cpu : usr=0.00%, sys=3.33%, ctx=1097, majf=0, minf=28 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=3924KiB/s (4018kB/s), 981KiB/s-984KiB/s (1005kB/s-1007kB/s), io=40.0MiB (41.9MB), run=10408-10438msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=22211: Thu May 1 19:05:23 2025 read: IOPS=138, BW=555KiB/s (569kB/s)(5048KiB/9091msec) slat (usec): min=21, max=11894, avg=3757.96, stdev=1471.46 clat (usec): min=4866, max=94260, avg=53887.93, stdev=11735.69 lat (usec): min=4891, max=98136, avg=57651.49, stdev=12174.27 clat percentiles (usec): | 1.00th=[28967], 5.00th=[37487], 10.00th=[40633], 20.00th=[44303], | 30.00th=[47973], 40.00th=[50594], 50.00th=[52691], 60.00th=[55313], | 70.00th=[58983], 80.00th=[62653], 90.00th=[69731], 95.00th=[74974], | 99.00th=[85459], 99.50th=[87557], 99.90th=[92799], 99.95th=[93848], | 99.99th=[93848] bw ( KiB/s): min= 448, max= 694, per=25.43%, avg=553.89, stdev=65.95, samples=18 iops : min= 112, max= 173, avg=138.39, stdev=16.40, samples=18 write: IOPS=142, BW=571KiB/s (585kB/s)(5192KiB/9091msec) slat (usec): min=11, max=1329, avg=86.01, stdev=138.14 clat (usec): min=17950, max=94469, avg=55725.83, stdev=10881.10 lat (usec): min=17975, max=94497, avg=55813.35, stdev=10884.56 clat percentiles (usec): | 1.00th=[30278], 5.00th=[40633], 10.00th=[43254], 20.00th=[46924], | 30.00th=[49546], 40.00th=[52167], 50.00th=[54789], 60.00th=[57410], | 70.00th=[61080], 80.00th=[64750], 90.00th=[69731], 95.00th=[74974], | 99.00th=[83362], 99.50th=[89654], 99.90th=[93848], 99.95th=[94897], | 99.99th=[94897] bw ( KiB/s): min= 480, max= 736, per=25.16%, avg=570.33, stdev=77.54, samples=18 iops : min= 120, max= 184, avg=142.50, stdev=19.33, samples=18 lat (msec) : 10=0.27%, 20=0.27%, 50=34.41%, 100=65.04% cpu : usr=2.77%, sys=4.19%, ctx=2445, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22212: Thu May 1 19:05:23 2025 read: IOPS=135, BW=540KiB/s (553kB/s)(4976KiB/9208msec) slat (usec): min=1173, max=14696, avg=3728.11, stdev=1451.96 clat (msec): min=9, max=101, avg=54.18, stdev=12.01 lat (msec): min=12, max=106, avg=57.91, stdev=12.40 clat percentiles (msec): | 1.00th=[ 31], 5.00th=[ 38], 10.00th=[ 41], 20.00th=[ 45], | 30.00th=[ 48], 40.00th=[ 51], 50.00th=[ 54], 60.00th=[ 56], | 70.00th=[ 59], 80.00th=[ 64], 90.00th=[ 69], 95.00th=[ 75], | 99.00th=[ 94], 99.50th=[ 96], 99.90th=[ 100], 99.95th=[ 103], | 99.99th=[ 103] bw ( KiB/s): min= 440, max= 678, per=24.74%, avg=538.83, stdev=75.67, samples=18 iops : min= 110, max= 169, avg=134.61, stdev=18.91, samples=18 write: IOPS=142, BW=572KiB/s (585kB/s)(5264KiB/9208msec) slat (usec): min=13, max=2540, avg=107.83, stdev=196.25 clat (msec): min=14, max=106, avg=56.32, stdev=10.38 lat (msec): min=14, max=106, avg=56.43, stdev=10.38 clat percentiles (msec): | 1.00th=[ 38], 5.00th=[ 41], 10.00th=[ 44], 20.00th=[ 48], | 30.00th=[ 51], 40.00th=[ 54], 50.00th=[ 56], 60.00th=[ 59], | 70.00th=[ 62], 80.00th=[ 66], 90.00th=[ 70], 95.00th=[ 74], | 99.00th=[ 83], 99.50th=[ 87], 99.90th=[ 102], 99.95th=[ 107], | 99.99th=[ 107] bw ( KiB/s): min= 416, max= 710, per=25.20%, avg=571.22, stdev=78.49, samples=18 iops : min= 104, max= 177, avg=142.72, stdev=19.53, samples=18 lat (msec) : 10=0.04%, 20=0.08%, 50=33.83%, 100=65.94%, 250=0.12% cpu : usr=0.85%, sys=6.09%, ctx=2432, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22213: Thu May 1 19:05:23 2025 read: IOPS=135, BW=541KiB/s (554kB/s)(4984KiB/9213msec) slat (usec): min=1180, max=11899, avg=3762.56, stdev=1469.33 clat (msec): min=23, max=112, avg=54.42, stdev=12.29 lat (msec): min=25, max=114, avg=58.18, stdev=12.67 clat percentiles (msec): | 1.00th=[ 32], 5.00th=[ 37], 10.00th=[ 40], 20.00th=[ 44], | 30.00th=[ 48], 40.00th=[ 51], 50.00th=[ 54], 60.00th=[ 57], | 70.00th=[ 60], 80.00th=[ 64], 90.00th=[ 71], 95.00th=[ 77], | 99.00th=[ 88], 99.50th=[ 95], 99.90th=[ 111], 99.95th=[ 113], | 99.99th=[ 113] bw ( KiB/s): min= 424, max= 704, per=24.90%, avg=542.39, stdev=79.47, samples=18 iops : min= 106, max= 176, avg=135.50, stdev=19.88, samples=18 write: IOPS=142, BW=570KiB/s (584kB/s)(5256KiB/9213msec) slat (usec): min=12, max=4304, avg=97.40, stdev=193.65 clat (usec): min=30983, max=99215, avg=56193.54, stdev=10658.55 lat (usec): min=31048, max=99229, avg=56293.04, stdev=10659.09 clat percentiles (usec): | 1.00th=[32900], 5.00th=[41157], 10.00th=[43254], 20.00th=[47449], | 30.00th=[50070], 40.00th=[52691], 50.00th=[55313], 60.00th=[57934], | 70.00th=[61080], 80.00th=[64750], 90.00th=[70779], 95.00th=[76022], | 99.00th=[82314], 99.50th=[85459], 99.90th=[95945], 99.95th=[99091], | 99.99th=[99091] bw ( KiB/s): min= 440, max= 800, per=25.04%, avg=567.67, stdev=89.80, samples=18 iops : min= 110, max= 200, avg=141.83, stdev=22.40, samples=18 lat (msec) : 50=33.67%, 100=66.17%, 250=0.16% cpu : usr=0.98%, sys=6.00%, ctx=2457, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22214: Thu May 1 19:05:23 2025 read: IOPS=137, BW=549KiB/s (562kB/s)(5060KiB/9212msec) slat (usec): min=1330, max=10769, avg=3772.65, stdev=1437.55 clat (usec): min=9548, max=95166, avg=53799.68, stdev=11410.11 lat (msec): min=14, max=102, avg=57.58, stdev=11.84 clat percentiles (usec): | 1.00th=[32637], 5.00th=[36963], 10.00th=[40109], 20.00th=[44303], | 30.00th=[47449], 40.00th=[50594], 50.00th=[52691], 60.00th=[55313], | 70.00th=[58459], 80.00th=[62653], 90.00th=[68682], 95.00th=[73925], | 99.00th=[86508], 99.50th=[89654], 99.90th=[93848], 99.95th=[94897], | 99.99th=[94897] bw ( KiB/s): min= 400, max= 664, per=25.16%, avg=547.89, stdev=81.55, samples=18 iops : min= 100, max= 166, avg=136.94, stdev=20.39, samples=18 write: IOPS=140, BW=562KiB/s (576kB/s)(5180KiB/9212msec) slat (usec): min=14, max=2633, avg=92.24, stdev=159.53 clat (usec): min=14431, max=91176, avg=56498.32, stdev=10917.93 lat (usec): min=14486, max=91220, avg=56591.90, stdev=10922.97 clat percentiles (usec): | 1.00th=[34341], 5.00th=[40109], 10.00th=[43254], 20.00th=[46924], | 30.00th=[50070], 40.00th=[52691], 50.00th=[55313], 60.00th=[58983], | 70.00th=[62129], 80.00th=[65799], 90.00th=[70779], 95.00th=[74974], | 99.00th=[86508], 99.50th=[88605], 99.90th=[89654], 99.95th=[90702], | 99.99th=[90702] bw ( KiB/s): min= 416, max= 752, per=24.72%, avg=560.33, stdev=89.69, samples=18 iops : min= 104, max= 188, avg=140.06, stdev=22.43, samples=18 lat (msec) : 10=0.04%, 20=0.04%, 50=34.34%, 100=65.59% cpu : usr=0.76%, sys=6.17%, ctx=2541, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2178KiB/s (2231kB/s), 540KiB/s-555KiB/s (553kB/s-569kB/s), io=19.6MiB (20.5MB), run=9091-9213msec WRITE: bw=2268KiB/s (2322kB/s), 562KiB/s-572KiB/s (576kB/s-585kB/s), io=20.4MiB (21.4MB), run=9091-9213msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.7 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=22225: Thu May 1 19:05:25 2025 read: IOPS=4, BW=183MiB/s (192MB/s)(40.0MiB/219msec) slat (nsec): min=2240.1k, max=2240.1k, avg=2240116.00, stdev= 0.00 clat (nsec): min=211350k, max=211350k, avg=211350413.00, stdev= 0.00 lat (nsec): min=213609k, max=213609k, avg=213609239.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 211], 5.00th=[ 211], 10.00th=[ 211], 20.00th=[ 211], | 30.00th=[ 211], 40.00th=[ 211], 50.00th=[ 211], 60.00th=[ 211], | 70.00th=[ 211], 80.00th=[ 211], 90.00th=[ 211], 95.00th=[ 211], | 99.00th=[ 211], 99.50th=[ 211], 99.90th=[ 211], 99.95th=[ 211], | 99.99th=[ 211] lat (msec) : 250=100.00% cpu : usr=0.00%, sys=1.37%, ctx=3, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=183MiB/s (192MB/s), 183MiB/s-183MiB/s (192MB/s-192MB/s), io=40.0MiB (41.9MB), run=219-219msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout