-----============= acceptance-small: racer ============----- Fri May 10 08:07:56 EDT 2024 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 excepting tests: racer: /home/green/git/lustre-release/lustre/tests/racer/racer.sh with 2 MDTs === racer: start setup 08:08:00 (1715342880) === Starting client oleg243-client.virtnet: -o user_xattr,flock oleg243-server@tcp:/lustre /mnt/lustre2 Started clients oleg243-client.virtnet: 192.168.202.143@tcp:/lustre on /mnt/lustre2 type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project) oleg243-client.virtnet: executing check_config_client /mnt/lustre oleg243-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg243-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8800a941f000.idle_timeout=debug osc.lustre-OST0000-osc-ffff8800b6e79800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8800a941f000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8800b6e79800.idle_timeout=debug disable quota as required oleg243-server: oleg243-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === racer: finish setup 08:08:07 (1715342887) === RACERDIRS=/mnt/lustre /mnt/lustre2 keep default fallocate mode: 0 debug_raw_pointers=Y debug_raw_pointers=Y == racer test 1: racer on clients: oleg243-client.virtnet DURATION=300 ========================================================== 08:08:09 (1715342889) racers pids: 9655 9656 9657 9658 9660 9662 ./file_create.sh: line 60: ((: 13/0: division by 0 (error token is "0") ./file_create.sh: line 60: ((: 1/0: division by 0 (error token is "0") ./file_create.sh: line 60: ((: 4/0: division by 0 (error token is "0") ./file_create.sh: line 1: kill: (25834) - No such process ./file_create.sh: line 1: kill: (25951) - No such process ./dir_create.sh: line 1: kill: (26222) - No such process kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec] kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec] ./file_link.sh: line 1: kill: (26972) - No such process ./file_link.sh: line 1: kill: (26713) - No such process ./file_symlink.sh: line 1: kill: (27706) - No such process ./file_list.sh: line 1: kill: (23142) - No such process ./file_list.sh: line 1: kill: (23635) - No such process ./file_list.sh: line 1: kill: (25509) - No such process ./file_list.sh: line 1: kill: (25390) - No such process ./file_list.sh: line 1: kill: (25395) - No such process ./file_list.sh: line 1: kill: (22654) - No such process ./file_list.sh: line 1: kill: (21462) - No such process ./file_list.sh: line 1: kill: (23086) - No such process ./file_concat.sh: line 1: kill: (28275) - No such process ./file_concat.sh: line 1: kill: (28338) - No such process ./file_exec.sh: line 1: kill: (9855) - No such process ./file_exec.sh: line 1: kill: (9968) - No such process ./file_exec.sh: line 1: kill: (10307) - No such process ./file_exec.sh: line 1: kill: (10395) - No such process ./file_exec.sh: line 1: kill: (10291) - No such process ./file_exec.sh: line 1: kill: (10493) - No such process ./file_exec.sh: line 1: kill: (10640) - No such process ./file_exec.sh: line 1: kill: (10797) - No such process ./file_chown.sh: line 1: kill: (28655) - No such process ./file_chmod.sh: line 1: kill: (28642) - No such process ./file_chmod.sh: line 1: kill: (28460) - No such process kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec] ./file_chmod.sh: line 1: kill: (28387) - No such process ./file_chmod.sh: line 1: kill: (28661) - No such process ./file_mknod.sh: line 1: kill: (29030) - No such process ./file_mknod.sh: line 1: kill: (29091) - No such process ./file_mknod.sh: line 1: kill: (28984) - No such process ./file_mknod.sh: line 1: kill: (29090) - No such process ./file_mknod.sh: line 1: kill: (28951) - No such process ./file_mknod.sh: line 1: kill: (29076) - No such process ./file_truncate.sh: line 1: kill: (29202) - No such process ./file_truncate.sh: line 1: kill: (29185) - No such process ./file_truncate.sh: line 1: kill: (29261) - No such process ./file_truncate.sh: line 1: kill: (29395) - No such process ./file_truncate.sh: line 1: kill: (29403) - No such process ./file_truncate.sh: line 1: kill: (29346) - No such process ./file_truncate.sh: line 1: kill: (29282) - No such process ./file_truncate.sh: line 1: kill: (29411) - No such process ./file_truncate.sh: line 1: kill: (29230) - No such process ./file_delxattr.sh: line 1: kill: (29515) - No such process ./file_delxattr.sh: line 1: kill: (29534) - No such process ./file_delxattr.sh: line 1: kill: (29537) - No such process ./file_getxattr.sh: line 1: kill: (29489) - No such process ./file_getxattr.sh: line 1: kill: (29401) - No such process ./file_getxattr.sh: line 1: kill: (29517) - No such process ./file_getxattr.sh: line 1: kill: (29680) - No such process ./file_getxattr.sh: line 1: kill: (29703) - No such process ./file_getxattr.sh: line 1: kill: (29661) - No such process ./file_setxattr.sh: line 1: kill: (29848) - No such process ./file_setxattr.sh: line 1: kill: (29835) - No such process ./file_setxattr.sh: line 1: kill: (29783) - No such process ./file_setxattr.sh: line 1: kill: (29868) - No such process ./file_setxattr.sh: line 1: kill: (29857) - No such process ./file_setxattr.sh: line 1: kill: (29812) - No such process ./file_setxattr.sh: line 1: kill: (29874) - No such process ./file_setxattr.sh: line 1: kill: (29858) - No such process ./file_setxattr.sh: line 1: kill: (29876) - No such process ./file_setxattr.sh: line 1: kill: (29888) - No such process ./dir_remote.sh: line 1: kill: (29970) - No such process ./dir_remote.sh: line 1: kill: (29957) - No such process ./dir_remote.sh: line 1: kill: (29998) - No such process ./dir_remote.sh: line 1: kill: (29951) - No such process ./dir_remote.sh: line 1: kill: (30006) - No such process ./dir_remote.sh: line 1: kill: (29977) - No such process ./dir_remote.sh: line 1: kill: (29994) - No such process ./dir_migrate.sh: line 1: kill: (29996) - No such process ./file_fallocate.sh: line 1: kill: (30176) - No such process ./file_fallocate.sh: line 1: kill: (30186) - No such process ./file_fallocate.sh: line 1: kill: (30173) - No such process ./file_fallocate.sh: line 1: kill: (30174) - No such process ./file_fallocate.sh: line 1: kill: (30179) - No such process ./file_fallocate.sh: line 1: kill: (30190) - No such process ./file_fallocate.sh: line 1: kill: (30189) - No such process ./file_fallocate.sh: line 1: kill: (30188) - No such process ./file_fallocate.sh: line 1: kill: (30187) - No such process ./file_fallocate.sh: line 1: kill: (30212) - No such process ./file_fallocate.sh: line 1: kill: (30180) - No such process file_create.sh: no process found file_create.sh: no process found file_create.sh: no process found file_create.sh: no process found file_create.sh: no process found dir_create.sh: no process found dir_create.sh: no process found file_create.sh: no process found dir_create.sh: no process found dir_create.sh: no process found file_rm.sh: no process found dir_create.sh: no process found file_rm.sh: no process found file_rm.sh: no process found file_rename.sh: no process found dir_create.sh: no process found file_rm.sh: no process found file_rm.sh: no process found file_link.sh: no process found file_rename.sh: no process found file_rename.sh: no process found file_rm.sh: no process found file_rename.sh: no process found file_rename.sh: no process found file_symlink.sh: no process found file_rename.sh: no process found file_link.sh: no process found file_link.sh: no process found file_link.sh: no process found file_link.sh: no process found file_link.sh: no process found file_symlink.sh: no process found file_list.sh: no process found file_symlink.sh: no process found file_symlink.sh: no process found file_symlink.sh: no process found file_symlink.sh: no process found file_concat.sh: no process found file_list.sh: no process found file_list.sh: no process found file_list.sh: no process found file_exec.sh: no process found file_list.sh: no process found file_list.sh: no process found file_concat.sh: no process found file_concat.sh: no process found file_concat.sh: no process found file_concat.sh: no process found file_concat.sh: no process found file_chown.sh: no process found file_exec.sh: no process found file_exec.sh: no process found file_exec.sh: no process found file_exec.sh: no process found file_chmod.sh: no process found file_chown.sh: no process found file_chown.sh: no process found file_mknod.sh: no process found file_chown.sh: no process found file_exec.sh: no process found file_chown.sh: no process found file_chmod.sh: no process found file_chmod.sh: no process found file_chmod.sh: no process found file_chown.sh: no process found file_chmod.sh: no process found file_mknod.sh: no process found file_truncate.sh: no process found file_chmod.sh: no process found file_mknod.sh: no process found file_mknod.sh: no process found file_mknod.sh: no process found file_mknod.sh: no process found file_delxattr.sh: no process found file_truncate.sh: no process found file_truncate.sh: no process found file_truncate.sh: no process found file_truncate.sh: no process found file_delxattr.sh: no process found file_getxattr.sh: no process found file_truncate.sh: no process found file_delxattr.sh: no process found file_getxattr.sh: no process found file_delxattr.sh: no process found file_delxattr.sh: no process found file_delxattr.sh: no process found file_setxattr.sh: no process found file_getxattr.sh: no process found file_getxattr.sh: no process found file_setxattr.sh: no process found file_getxattr.sh: no process found file_getxattr.sh: no process found file_setxattr.sh: no process found dir_remote.sh: no process found dir_remote.sh: no process found file_setxattr.sh: no process found dir_remote.sh: no process found dir_remote.sh: no process found file_setxattr.sh: no process found dir_migrate.sh: no process found dir_migrate.sh: no process found dir_migrate.sh: no process found dir_migrate.sh: no process found file_setxattr.sh: no process found file_fallocate.sh: no process found dir_remote.sh: no process found file_fallocate.sh: no process found file_fallocate.sh: no process found file_fallocate.sh: no process found dir_migrate.sh: no process found dir_remote.sh: no process found file_fallocate.sh: no process found dir_migrate.sh: no process found file_fallocate.sh: no process found Running /home/green/git/lustre-release/lustre/tests/racer/racer.sh for 300 seconds. CTRL-C to exit layout: raid0 raid0 pfl pfl pfl flr flr flr sel sel sel layout: raid0 raid0 pfl pfl pfl flr flr flr sel sel sel layout: raid0 raid0 pfl pfl pfl flr flr flr sel sel sel racer cleanup sleeping 5 sec ... there should be NO racer processes: USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND Filesystem 1K-blocks Used Available Use% Mounted on 192.168.202.143@tcp:/lustre 7666232 89800 7124240 2% /mnt/lustre2 We survived /home/green/git/lustre-release/lustre/tests/racer/racer.sh for 300 seconds. Running /home/green/git/lustre-release/lustre/tests/racer/racer.sh for 300 seconds. CTRL-C to exit layout: raid0 raid0 pfl pfl pfl flr flr flr sel sel sel layout: raid0 raid0 pfl pfl pfl flr flr flr sel sel sel layout: raid0 raid0 pfl pfl pfl flr flr flr sel sel sel racer cleanup sleeping 5 sec ... there should be NO racer processes: USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND Filesystem 1K-blocks Used Available Use% Mounted on 192.168.202.143@tcp:/lustre 7666232 89800 7124240 2% /mnt/lustre2 We survived /home/green/git/lustre-release/lustre/tests/racer/racer.sh for 300 seconds. Running /home/green/git/lustre-release/lustre/tests/racer/racer.sh for 300 seconds. CTRL-C to exit layout: raid0 raid0 pfl pfl pfl flr flr flr sel sel sel layout: raid0 raid0 pfl pfl pfl flr flr flr sel sel sel layout: raid0 raid0 pfl pfl pfl flr flr flr sel sel sel racer cleanup ./file_exec.sh: line 16: 13256 Aborted $DIR/$file 0.$((RANDOM % 5 + 1)) 2> /dev/null sleeping 5 sec ... there should be NO racer processes: USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND Filesystem 1K-blocks Used Available Use% Mounted on 192.168.202.143@tcp:/lustre 7666232 89800 7124240 2% /mnt/lustre We survived /home/green/git/lustre-release/lustre/tests/racer/racer.sh for 300 seconds. Running /home/green/git/lustre-release/lustre/tests/racer/racer.sh for 300 seconds. CTRL-C to exit layout: raid0 raid0 pfl pfl pfl flr flr flr sel sel sel layout: raid0 raid0 pfl pfl pfl flr flr flr sel sel sel layout: raid0 raid0 pfl pfl pfl flr flr flr sel sel sel racer cleanup sleeping 5 sec ... there should be NO racer processes: USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND Filesystem 1K-blocks Used Available Use% Mounted on 192.168.202.143@tcp:/lustre 7666232 89800 7124240 2% /mnt/lustre We survived /home/green/git/lustre-release/lustre/tests/racer/racer.sh for 300 seconds. pid=9655 rc=0 Running /home/green/git/lustre-release/lustre/tests/racer/racer.sh for 300 seconds. CTRL-C to exit layout: raid0 raid0 pfl pfl pfl flr flr flr sel sel sel layout: raid0 raid0 pfl pfl pfl flr flr flr sel sel sel layout: raid0 raid0 pfl pfl pfl flr flr flr sel sel sel racer cleanup ./file_exec.sh: line 16: 15010 Illegal instruction $DIR/$file 0.$((RANDOM % 5 + 1)) 2> /dev/null sleeping 5 sec ... there should be NO racer processes: USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND Filesystem 1K-blocks Used Available Use% Mounted on 192.168.202.143@tcp:/lustre 7666232 89800 7124240 2% /mnt/lustre We survived /home/green/git/lustre-release/lustre/tests/racer/racer.sh for 300 seconds. pid=9656 rc=0 pid=9657 rc=0 Running /home/green/git/lustre-release/lustre/tests/racer/racer.sh for 300 seconds. CTRL-C to exit layout: raid0 raid0 pfl pfl pfl flr flr flr sel sel sel layout: raid0 raid0 pfl pfl pfl flr flr flr sel sel sel layout: raid0 raid0 pfl pfl pfl flr flr flr sel sel sel racer cleanup sleeping 5 sec ... there should be NO racer processes: USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND Filesystem 1K-blocks Used Available Use% Mounted on 192.168.202.143@tcp:/lustre 7666232 89800 7124240 2% /mnt/lustre2 We survived /home/green/git/lustre-release/lustre/tests/racer/racer.sh for 300 seconds. pid=9658 rc=0 pid=9660 rc=0 pid=9662 rc=0 PASS 1 (312s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == racer test 2: racer rename: oleg243-client.virtnet DURATION=300 ========================================================== 08:13:22 (1715343202) starting on oleg243-client.virtnet:/mnt/lustre/racer0 with: dir_create dir_remote file_rename file_rename file_create file_rm starting on oleg243-client.virtnet:/mnt/lustre/racer1 with: dir_create dir_remote file_rename file_rename file_create file_rm starting on oleg243-client.virtnet:/mnt/lustre2/racer0 with: dir_create dir_remote file_rename file_rename file_create file_rm starting on oleg243-client.virtnet:/mnt/lustre2/racer1 with: dir_create dir_remote file_rename file_rename file_create file_rm racers pids: 31138 31139 31140 31141 kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec] kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec] ./dir_remote.sh: line 1: kill: (16928) - No such process ./dir_remote.sh: line 1: kill: (16901) - No such process kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec] ./file_rename.sh: line 1: kill: (16959) - No such process kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec] kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec] ./file_rm.sh: line 1: kill: (16706) - No such process ./file_rm.sh: line 1: kill: (16999) - No such process kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec] dir_create.sh: no process found dir_remote.sh: no process found file_rename.sh: no process found dir_create.sh: no process found file_rename.sh: no process found dir_remote.sh: no process found file_create.sh: no process found file_rename.sh: no process found file_rm.sh: no process found file_rename.sh: no process found file_create.sh: no process found file_rm.sh: no process found dir_create.sh: no process found dir_create.sh: no process found Running /home/green/git/lustre-release/lustre/tests/racer/racer.sh for 300 seconds. CTRL-C to exit layout: raid0 raid0 pfl pfl pfl layout: raid0 raid0 pfl pfl pfl layout: raid0 raid0 pfl pfl pfl racer cleanup sleeping 5 sec ... there should be NO racer processes: USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND Filesystem 1K-blocks Used Available Use% Mounted on 192.168.202.143@tcp:/lustre 7666232 71284 7141664 1% /mnt/lustre We survived /home/green/git/lustre-release/lustre/tests/racer/racer.sh for 300 seconds. dir_remote.sh: no process found dir_remote.sh: no process found file_rename.sh: no process found file_rename.sh: no process found file_rename.sh: no process found Running /home/green/git/lustre-release/lustre/tests/racer/racer.sh for 300 seconds. CTRL-C to exit layout: raid0 raid0 pfl pfl pfl layout: raid0 raid0 pfl pfl pfl layout: raid0 raid0 pfl pfl pfl racer cleanup sleeping 5 sec ... there should be NO racer processes: USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND Filesystem 1K-blocks Used Available Use% Mounted on 192.168.202.143@tcp:/lustre 7666232 71284 7141664 1% /mnt/lustre2 We survived /home/green/git/lustre-release/lustre/tests/racer/racer.sh for 300 seconds. file_rename.sh: no process found file_create.sh: no process found file_create.sh: no process found file_rm.sh: no process found file_rm.sh: no process found Running /home/green/git/lustre-release/lustre/tests/racer/racer.sh for 300 seconds. CTRL-C to exit layout: raid0 raid0 pfl pfl pfl layout: raid0 raid0 pfl pfl pfl layout: raid0 raid0 pfl pfl pfl racer cleanup sleeping 5 sec ... there should be NO racer processes: USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND Filesystem 1K-blocks Used Available Use% Mounted on 192.168.202.143@tcp:/lustre 7666232 71284 7141664 1% /mnt/lustre2 We survived /home/green/git/lustre-release/lustre/tests/racer/racer.sh for 300 seconds. Running /home/green/git/lustre-release/lustre/tests/racer/racer.sh for 300 seconds. CTRL-C to exit layout: raid0 raid0 pfl pfl pfl layout: raid0 raid0 pfl pfl pfl layout: raid0 raid0 pfl pfl pfl racer cleanup sleeping 5 sec ... there should be NO racer processes: USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND Filesystem 1K-blocks Used Available Use% Mounted on 192.168.202.143@tcp:/lustre 7666232 71284 7141664 1% /mnt/lustre We survived /home/green/git/lustre-release/lustre/tests/racer/racer.sh for 300 seconds. pid=31138 rc=0 pid=31139 rc=0 pid=31140 rc=0 pid=31141 rc=0 PASS 2 (308s) debug_raw_pointers=0 debug_raw_pointers=0 == racer test complete, duration 635 sec ================= 08:18:31 (1715343511) === racer: start cleanup 08:18:31 (1715343511) === Stopping clients: oleg243-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg243-client.virtnet /mnt/lustre2 opts: === racer: finish cleanup 08:18:46 (1715343526) ===