-----============= acceptance-small: conf-sanity ============----- Mon Aug 31 00:52:30 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg352-server: /home/green/git/lustre-release/lustre/tests/except/conf-sanity.ex - see CLIENT_VERSION > v2_15_63-53-g1a6ef725c2 (34682651 > 34553653) for LU-16938, go 81 excepting tests: 32newtarball 108a 112a 112b 113 117 119 121 122a 123aa 123ab 123ac 123ad 123ae 123af 123ag 123ah 123ahi 123F 123G 123H 126 129 132 133 135 136 137 150 152 153a 153b 153c 155 802a skipping tests SLOW=no: 45 69 106 111 114 Stopping clients: oleg352-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg352-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg352-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg352-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg352-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg352-server oleg352-server: oleg352-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg352-server' oleg352-server: oleg352-server.virtnet: executing load_modules_local oleg352-server: Loading modules from /home/green/git/lustre-release/lustre oleg352-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg352-server: detected 4 online CPUs by sysfs oleg352-server: MODOPTS_LIBCFS= oleg352-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg352-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg352-server: oleg352-server.virtnet: executing set_default_debug -1 all pdsh@oleg352-client: oleg352-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg352-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg352-server: oleg352-server.virtnet: executing set_default_debug -1 all pdsh@oleg352-client: oleg352-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg352-server: oleg352-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg352-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg352-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg352-server stop mds service on oleg352-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg352-server == conf-sanity test 57k: unknown target_nid_policy stops the module load ========================================================== 00:55:33 (1788152133) umount lustre on /mnt/lustre..... stop ost1 service on oleg352-server stop mds service on oleg352-server unloading modules via unload_modules_local on: 'oleg352-server' oleg352-server: oleg352-server.virtnet: executing unload_modules_local oleg352-server: modules unloaded. LNET unconfigure error 22: Invalid argument unloading modules via unload_modules_local on: 'oleg352-server' oleg352-server: oleg352-server.virtnet: executing unload_modules_local oleg352-server: oleg352-server: LNET unconfigure error 22: Invalid argument modules unloaded. oleg352-server: modprobe: FATAL: Module ptlrpc not found in directory /lib/modules/4.18.0rh8.10-debug pdsh@oleg352-client: oleg352-server: ssh exited with exit code 1 pdsh@oleg352-client: oleg352-server: ssh exited with exit code 1 pdsh@oleg352-client: oleg352-client: ssh exited with exit code 2 pdsh@oleg352-client: oleg352-server: ssh exited with exit code 2 pdsh@oleg352-client: oleg352-server: ssh exited with exit code 2 pdsh@oleg352-client: oleg352-client: ssh exited with exit code 2 LNET unconfigure error 22: Invalid argument unloading modules via unload_modules_local on: 'oleg352-server' oleg352-server: oleg352-server.virtnet: executing unload_modules_local oleg352-server: oleg352-server: LNET unconfigure error 22: Invalid argument modules unloaded. PASS 57k (41s) == conf-sanity test 57l: nodemap nidrange visibility filters a client class at runtime ========================================================== 00:56:14 (1788152174) umount lustre on /mnt/lustre..... stop ost1 service on oleg352-server stop mds service on oleg352-server LNET unconfigure error 22: Invalid argument unloading modules via unload_modules_local on: 'oleg352-server' oleg352-server: oleg352-server.virtnet: executing unload_modules_local oleg352-server: oleg352-server: LNET unconfigure error 22: Invalid argument modules unloaded. setmodopts: MODOPTS_PTLRPC=lbug_on_grant_miscount=1 setmodopts: MODOPTS_LNET= external_disc_visibility=nodemap_nidrange SERVICENODE DEBUG: setup base_policy=none servicenode_policy=disabled SERVICENODE DEBUG: before load MODOPTS_PTLRPC='lbug_on_grant_miscount=1' SERVICENODE DEBUG: before load MODOPTS_LNET=' external_disc_visibility=nodemap_nidrange' /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ../lnet/lnet/lnet options: 'external_disc_visibility=nodemap_nidrange' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg352-server' oleg352-server: oleg352-server.virtnet: executing load_modules_local oleg352-server: Loading modules from /home/green/git/lustre-release/lustre oleg352-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg352-server: detected 4 online CPUs by sysfs oleg352-server: MODOPTS_LIBCFS= oleg352-server: Force libcfs to create 2 CPU partitions oleg352-server: ../lnet/lnet/lnet options: 'external_disc_visibility=nodemap_nidrange' oleg352-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg352-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg352-server: mdt/mdt options: 'mdt_enable_flr_ec=1' SERVICENODE DEBUG: modules loaded oleg352-server: oleg352-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre oleg352-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg352-client: oleg352-server: ssh exited with exit code 19 oleg352-server: oleg352-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre oleg352-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg352-client: oleg352-server: ssh exited with exit code 19 oleg352-server: oleg352-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre oleg352-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg352-client: oleg352-server: ssh exited with exit code 19 tunefs.lustre failed, reformatting instead Stopping clients: oleg352-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg352-client.virtnet /mnt/lustre2 (opts:-f) oleg352-server: oleg352-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg352-server' oleg352-server: oleg352-server.virtnet: executing load_modules_local oleg352-server: Loading modules from /home/green/git/lustre-release/lustre oleg352-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg352-server: detected 4 online CPUs by sysfs oleg352-server: MODOPTS_LIBCFS= oleg352-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg352-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg352-server: oleg352-server.virtnet: executing set_default_debug -1 all pdsh@oleg352-client: oleg352-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg352-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg352-server: oleg352-server.virtnet: executing set_default_debug -1 all pdsh@oleg352-client: oleg352-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg352-server: oleg352-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg352-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg352-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg352-server stop mds service on oleg352-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg352-server SERVICENODE DEBUG: writeconf_or_reformat complete oleg352-server: oleg352-server.virtnet: executing lnet_if_list SERVICENODE DEBUG: oss_if='ens2' client_if='ens2' oleg352-server: sh: /usr/sbin/ksocklnd-config: No such file or directory SERVICENODE DEBUG: net=tcp nid='192.168.203.152@tcp' SERVICENODE DEBUG: client net=tcp nid='192.168.203.52@tcp' SERVICENODE DEBUG: net=tcp1 nid='192.168.203.152@tcp1' SERVICENODE DEBUG: client net=tcp1 nid='192.168.203.52@tcp1' SERVICENODE DEBUG: before temporary target format SERVICENODE DEBUG: HOSTNAME=oleg352-client.virtnet OSS=oleg352-server SERVICENODE DEBUG: MODOPTS_PTLRPC='lbug_on_grant_miscount=1' SERVICENODE DEBUG: MODOPTS_LNET=' external_disc_visibility=nodemap_nidrange' SERVICENODE DEBUG: nets='tcp,tcp1' SERVICENODE DEBUG: nids='192.168.203.152@tcp,192.168.203.152@tcp1' SERVICENODE DEBUG: client LNet configuration net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: tcp local NI(s): - nid: 192.168.203.52@tcp status: up interfaces: 0: ens2 - net type: tcp1 local NI(s): - nid: 192.168.203.52@tcp1 status: up interfaces: 0: ens2 SERVICENODE DEBUG: OSS LNet configuration net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: tcp local NI(s): - nid: 192.168.203.152@tcp status: up interfaces: 0: ens2 - net type: tcp1 local NI(s): - nid: 192.168.203.152@tcp1 status: up interfaces: 0: ens2 peer: - primary nid: 192.168.203.152@tcp Multi-Rail: true peer ni: - nid: 192.168.203.152@tcp state: NA - nid: 192.168.203.152@tcp1 state: NA SERVICENODE DEBUG: temporary fsname='sv9982' SERVICENODE DEBUG: temporary OST1 device='lustre-ost1_2/ost1_2' Permanent disk data: Device: lustre-ost1_2/ost1_2 Target: sv9982:OST0000 Index: 0 Lustre FS: sv9982 Mount type: zfs Flags: 0x1062 (OST first_time update no_primnode ) Persistent mount opts: Parameters: mgsnode=192.168.203.152@tcp sys.timeout=20 failover.node=192.168.203.152@tcp autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1_2 /tmp/lustre-ost1_2 mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1_2/ost1_2 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1_2/ost1_2 properties lustre:mgsnode=192.168.203.152@tcp lustre:sys.timeout=20 lustre:failover.node=192.168.203.152@tcp lustre:autodegrade=on lustre:version=1 lustre:flags=4194 lustre:index=0 lustre:fsname=sv9982 lustre:svname=sv9982:OST0000 SERVICENODE DEBUG: temporary OST2 device='lustre-ost2_2/ost2_2' Permanent disk data: Device: lustre-ost2_2/ost2_2 Target: sv9982:OST0001 Index: 1 Lustre FS: sv9982 Mount type: zfs Flags: 0x1062 (OST first_time update no_primnode ) Persistent mount opts: Parameters: mgsnode=192.168.203.152@tcp sys.timeout=20 failover.node=192.168.203.152@tcp1 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost2_2 /tmp/lustre-ost2_2 mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost2_2/ost2_2 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost2_2/ost2_2 properties lustre:mgsnode=192.168.203.152@tcp lustre:sys.timeout=20 lustre:failover.node=192.168.203.152@tcp1 lustre:autodegrade=on lustre:version=1 lustre:flags=4194 lustre:index=1 lustre:fsname=sv9982 lustre:svname=sv9982:OST0001 SERVICENODE DEBUG: starting MGS/MDT and temporary service-node targets start mds service on oleg352-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg352-server: oleg352-server.virtnet: executing set_default_debug -1 all pdsh@oleg352-client: oleg352-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 Start fs2ost1: mount -t lustre -o localrecov lustre-ost1_2/ost1_2 /mnt/lustre-fs2ost1 seq.cli-sv9982-OST0000-super.width=65536 oleg352-server: oleg352-server.virtnet: executing set_default_debug -1 all pdsh@oleg352-client: oleg352-server: ssh exited with exit code 1 Commit the device label on lustre-ost1_2/ost1_2 Started sv9982-OST0000 oleg352-server: missing pool name argument oleg352-server: usage: oleg352-server: trim [-dw] [-r ] [-c | -s] [ ...] pdsh@oleg352-client: oleg352-server: ssh exited with exit code 2 Start fs3ost2: mount -t lustre -o localrecov lustre-ost2_2/ost2_2 /mnt/lustre-fs3ost2 seq.cli-sv9982-OST0001-super.width=65536 oleg352-server: oleg352-server.virtnet: executing set_default_debug -1 all pdsh@oleg352-client: oleg352-server: ssh exited with exit code 1 Commit the device label on lustre-ost2_2/ost2_2 Started sv9982-OST0001 oleg352-server: missing pool name argument oleg352-server: usage: oleg352-server: trim [-dw] [-r ] [-c | -s] [ ...] pdsh@oleg352-client: oleg352-server: ssh exited with exit code 2 SERVICENODE DEBUG: after target startup SERVICENODE DEBUG: HOSTNAME=oleg352-client.virtnet OSS=oleg352-server SERVICENODE DEBUG: MODOPTS_PTLRPC='lbug_on_grant_miscount=1' SERVICENODE DEBUG: MODOPTS_LNET=' external_disc_visibility=nodemap_nidrange' SERVICENODE DEBUG: nets='tcp,tcp1' SERVICENODE DEBUG: nids='192.168.203.152@tcp,192.168.203.152@tcp1' SERVICENODE DEBUG: client LNet configuration net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: tcp local NI(s): - nid: 192.168.203.52@tcp status: up interfaces: 0: ens2 - net type: tcp1 local NI(s): - nid: 192.168.203.52@tcp1 status: up interfaces: 0: ens2 SERVICENODE DEBUG: OSS LNet configuration net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: tcp local NI(s): - nid: 192.168.203.152@tcp status: up interfaces: 0: ens2 - net type: tcp1 local NI(s): - nid: 192.168.203.152@tcp1 status: up interfaces: 0: ens2 peer: - primary nid: 192.168.203.152@tcp Multi-Rail: true peer ni: - nid: 192.168.203.152@tcp state: NA - nid: 192.168.203.152@tcp1 state: NA ping: - primary nid: 192.168.203.152@tcp Multi-Rail: false peer_ni: - nid: 192.168.203.152@tcp - nid: 192.168.203.152@tcp1 ping: - primary nid: 192.168.203.152@tcp Multi-Rail: false peer_ni: - nid: 192.168.203.152@tcp ping: - primary nid: 192.168.203.152@tcp1 Multi-Rail: false peer_ni: - nid: 192.168.203.152@tcp - nid: 192.168.203.152@tcp1 ping: - primary nid: 192.168.203.152@tcp Multi-Rail: false peer_ni: - nid: 192.168.203.152@tcp - nid: 192.168.203.152@tcp1 PASS 57l (264s) == conf-sanity test 57m: nodemap visibility class unions all configured client ranges ========================================================== 01:00:38 (1788152438) umount lustre on /mnt/lustre..... stop ost1 service on oleg352-server stop mds service on oleg352-server LNET unconfigure error 22: Invalid argument unloading modules via unload_modules_local on: 'oleg352-server' oleg352-server: oleg352-server.virtnet: executing unload_modules_local oleg352-server: oleg352-server: LNET unconfigure error 22: Invalid argument modules unloaded. setmodopts: MODOPTS_PTLRPC=lbug_on_grant_miscount=1 setmodopts: MODOPTS_LNET= external_disc_visibility=nodemap_nidrange SERVICENODE DEBUG: setup base_policy=none servicenode_policy=disabled SERVICENODE DEBUG: before load MODOPTS_PTLRPC='lbug_on_grant_miscount=1' SERVICENODE DEBUG: before load MODOPTS_LNET=' external_disc_visibility=nodemap_nidrange' /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ../lnet/lnet/lnet options: 'external_disc_visibility=nodemap_nidrange' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg352-server' oleg352-server: oleg352-server.virtnet: executing load_modules_local oleg352-server: Loading modules from /home/green/git/lustre-release/lustre oleg352-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg352-server: detected 4 online CPUs by sysfs oleg352-server: MODOPTS_LIBCFS= oleg352-server: Force libcfs to create 2 CPU partitions oleg352-server: ../lnet/lnet/lnet options: 'external_disc_visibility=nodemap_nidrange' oleg352-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg352-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg352-server: mdt/mdt options: 'mdt_enable_flr_ec=1' SERVICENODE DEBUG: modules loaded oleg352-server: oleg352-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre oleg352-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg352-client: oleg352-server: ssh exited with exit code 19 oleg352-server: oleg352-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre oleg352-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg352-client: oleg352-server: ssh exited with exit code 19 oleg352-server: oleg352-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre oleg352-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg352-client: oleg352-server: ssh exited with exit code 19 tunefs.lustre failed, reformatting instead Stopping clients: oleg352-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg352-client.virtnet /mnt/lustre2 (opts:-f) oleg352-server: oleg352-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg352-server' oleg352-server: oleg352-server.virtnet: executing load_modules_local oleg352-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg352-server: Loading modules from /home/green/git/lustre-release/lustre oleg352-server: detected 4 online CPUs by sysfs oleg352-server: MODOPTS_LIBCFS= oleg352-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg352-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg352-server: oleg352-server.virtnet: executing set_default_debug -1 all pdsh@oleg352-client: oleg352-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg352-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg352-server: oleg352-server.virtnet: executing set_default_debug -1 all pdsh@oleg352-client: oleg352-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg352-server: oleg352-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg352-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg352-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg352-server stop mds service on oleg352-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg352-server SERVICENODE DEBUG: writeconf_or_reformat complete oleg352-server: oleg352-server.virtnet: executing lnet_if_list SERVICENODE DEBUG: oss_if='ens2' client_if='ens2' oleg352-server: sh: /usr/sbin/ksocklnd-config: No such file or directory default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.52 default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.52 SERVICENODE DEBUG: net=tcp nid='192.168.203.152@tcp' SERVICENODE DEBUG: client net=tcp nid='192.168.203.52@tcp' SERVICENODE DEBUG: net=tcp1 nid='192.168.203.152@tcp1' SERVICENODE DEBUG: client net=tcp1 nid='192.168.203.52@tcp1' SERVICENODE DEBUG: before temporary target format SERVICENODE DEBUG: HOSTNAME=oleg352-client.virtnet OSS=oleg352-server SERVICENODE DEBUG: MODOPTS_PTLRPC='lbug_on_grant_miscount=1' SERVICENODE DEBUG: MODOPTS_LNET=' external_disc_visibility=nodemap_nidrange' SERVICENODE DEBUG: nets='tcp,tcp1' SERVICENODE DEBUG: nids='192.168.203.152@tcp,192.168.203.152@tcp1' SERVICENODE DEBUG: client LNet configuration net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: tcp local NI(s): - nid: 192.168.203.52@tcp status: up interfaces: 0: ens2 - net type: tcp1 local NI(s): - nid: 192.168.203.52@tcp1 status: up interfaces: 0: ens2 SERVICENODE DEBUG: OSS LNet configuration net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: tcp local NI(s): - nid: 192.168.203.152@tcp status: up interfaces: 0: ens2 - net type: tcp1 local NI(s): - nid: 192.168.203.152@tcp1 status: up interfaces: 0: ens2 peer: - primary nid: 192.168.203.152@tcp Multi-Rail: true peer ni: - nid: 192.168.203.152@tcp state: NA - nid: 192.168.203.152@tcp1 state: NA SERVICENODE DEBUG: temporary fsname='sv18821' SERVICENODE DEBUG: temporary OST1 device='lustre-ost1_2/ost1_2' Permanent disk data: Device: lustre-ost1_2/ost1_2 Target: sv18821:OST0000 Index: 0 Lustre FS: sv18821 Mount type: zfs Flags: 0x1062 (OST first_time update no_primnode ) Persistent mount opts: Parameters: mgsnode=192.168.203.152@tcp sys.timeout=20 failover.node=192.168.203.152@tcp autodegrade=on mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1_2/ost1_2 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1_2/ost1_2 properties lustre:mgsnode=192.168.203.152@tcp lustre:sys.timeout=20 lustre:failover.node=192.168.203.152@tcp lustre:autodegrade=on lustre:version=1 lustre:flags=4194 lustre:index=0 lustre:fsname=sv18821 lustre:svname=sv18821:OST0000 SERVICENODE DEBUG: temporary OST2 device='lustre-ost2_2/ost2_2' Permanent disk data: Device: lustre-ost2_2/ost2_2 Target: sv18821:OST0001 Index: 1 Lustre FS: sv18821 Mount type: zfs Flags: 0x1062 (OST first_time update no_primnode ) Persistent mount opts: Parameters: mgsnode=192.168.203.152@tcp sys.timeout=20 failover.node=192.168.203.152@tcp1 autodegrade=on mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost2_2/ost2_2 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost2_2/ost2_2 properties lustre:mgsnode=192.168.203.152@tcp lustre:sys.timeout=20 lustre:failover.node=192.168.203.152@tcp1 lustre:autodegrade=on lustre:version=1 lustre:flags=4194 lustre:index=1 lustre:fsname=sv18821 lustre:svname=sv18821:OST0001 SERVICENODE DEBUG: starting MGS/MDT and temporary service-node targets start mds service on oleg352-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg352-server: oleg352-server.virtnet: executing set_default_debug -1 all pdsh@oleg352-client: oleg352-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 Start fs2ost1: mount -t lustre -o localrecov lustre-ost1_2/ost1_2 /mnt/lustre-fs2ost1 seq.cli-sv18821-OST0000-super.width=65536 oleg352-server: oleg352-server.virtnet: executing set_default_debug -1 all pdsh@oleg352-client: oleg352-server: ssh exited with exit code 1 Commit the device label on lustre-ost1_2/ost1_2 Started sv18821-OST0000 oleg352-server: missing pool name argument oleg352-server: usage: oleg352-server: trim [-dw] [-r ] [-c | -s] [ ...] pdsh@oleg352-client: oleg352-server: ssh exited with exit code 2 Start fs3ost2: mount -t lustre -o localrecov lustre-ost2_2/ost2_2 /mnt/lustre-fs3ost2 seq.cli-sv18821-OST0001-super.width=65536 oleg352-server: oleg352-server.virtnet: executing set_default_debug -1 all pdsh@oleg352-client: oleg352-server: ssh exited with exit code 1 Commit the device label on lustre-ost2_2/ost2_2 Started sv18821-OST0001 oleg352-server: missing pool name argument oleg352-server: usage: oleg352-server: trim [-dw] [-r ] [-c | -s] [ ...] pdsh@oleg352-client: oleg352-server: ssh exited with exit code 2 SERVICENODE DEBUG: after target startup SERVICENODE DEBUG: HOSTNAME=oleg352-client.virtnet OSS=oleg352-server SERVICENODE DEBUG: MODOPTS_PTLRPC='lbug_on_grant_miscount=1' SERVICENODE DEBUG: MODOPTS_LNET=' external_disc_visibility=nodemap_nidrange' SERVICENODE DEBUG: nets='tcp,tcp1' SERVICENODE DEBUG: nids='192.168.203.152@tcp,192.168.203.152@tcp1' SERVICENODE DEBUG: client LNet configuration net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: tcp local NI(s): - nid: 192.168.203.52@tcp status: up interfaces: 0: ens2 - net type: tcp1 local NI(s): - nid: 192.168.203.52@tcp1 status: up interfaces: 0: ens2 SERVICENODE DEBUG: OSS LNet configuration net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: tcp local NI(s): - nid: 192.168.203.152@tcp status: up interfaces: 0: ens2 - net type: tcp1 local NI(s): - nid: 192.168.203.152@tcp1 status: up interfaces: 0: ens2 peer: - primary nid: 192.168.203.152@tcp Multi-Rail: true peer ni: - nid: 192.168.203.152@tcp state: NA - nid: 192.168.203.152@tcp1 state: NA ping: - primary nid: 192.168.203.152@tcp Multi-Rail: false peer_ni: - nid: 192.168.203.152@tcp - nid: 192.168.203.152@tcp1 ping: - primary nid: 192.168.203.152@tcp1 Multi-Rail: false peer_ni: - nid: 192.168.203.152@tcp - nid: 192.168.203.152@tcp1 PASS 57m (282s) == conf-sanity test 57n: independent nodemap visibility classes select independent views ========================================================== 01:05:20 (1788152720) umount lustre on /mnt/lustre..... stop ost1 service on oleg352-server stop mds service on oleg352-server LNET unconfigure error 22: Invalid argument unloading modules via unload_modules_local on: 'oleg352-server' oleg352-server: oleg352-server.virtnet: executing unload_modules_local oleg352-server: oleg352-server: LNET unconfigure error 22: Invalid argument modules unloaded. setmodopts: MODOPTS_PTLRPC=lbug_on_grant_miscount=1 setmodopts: MODOPTS_LNET= external_disc_visibility=nodemap_nidrange SERVICENODE DEBUG: setup base_policy=none servicenode_policy=disabled SERVICENODE DEBUG: before load MODOPTS_PTLRPC='lbug_on_grant_miscount=1' SERVICENODE DEBUG: before load MODOPTS_LNET=' external_disc_visibility=nodemap_nidrange' /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ../lnet/lnet/lnet options: 'external_disc_visibility=nodemap_nidrange' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg352-server' oleg352-server: oleg352-server.virtnet: executing load_modules_local oleg352-server: Loading modules from /home/green/git/lustre-release/lustre oleg352-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg352-server: detected 4 online CPUs by sysfs oleg352-server: MODOPTS_LIBCFS= oleg352-server: Force libcfs to create 2 CPU partitions oleg352-server: ../lnet/lnet/lnet options: 'external_disc_visibility=nodemap_nidrange' oleg352-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg352-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg352-server: mdt/mdt options: 'mdt_enable_flr_ec=1' SERVICENODE DEBUG: modules loaded oleg352-server: oleg352-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre oleg352-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg352-client: oleg352-server: ssh exited with exit code 19 oleg352-server: oleg352-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre oleg352-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg352-client: oleg352-server: ssh exited with exit code 19 oleg352-server: oleg352-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre oleg352-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg352-client: oleg352-server: ssh exited with exit code 19 tunefs.lustre failed, reformatting instead Stopping clients: oleg352-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg352-client.virtnet /mnt/lustre2 (opts:-f) oleg352-server: oleg352-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg352-server' oleg352-server: oleg352-server.virtnet: executing load_modules_local oleg352-server: Loading modules from /home/green/git/lustre-release/lustre oleg352-server: detected 4 online CPUs by sysfs oleg352-server: MODOPTS_LIBCFS= oleg352-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg352-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg352-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg352-server: oleg352-server.virtnet: executing set_default_debug -1 all pdsh@oleg352-client: oleg352-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg352-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg352-server: oleg352-server.virtnet: executing set_default_debug -1 all pdsh@oleg352-client: oleg352-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg352-server: oleg352-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg352-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg352-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg352-server stop mds service on oleg352-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg352-server SERVICENODE DEBUG: writeconf_or_reformat complete oleg352-server: oleg352-server.virtnet: executing lnet_if_list SERVICENODE DEBUG: oss_if='ens2' client_if='ens2' oleg352-server: sh: /usr/sbin/ksocklnd-config: No such file or directory default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.52 default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.52 SERVICENODE DEBUG: net=tcp nid='192.168.203.152@tcp' SERVICENODE DEBUG: client net=tcp nid='192.168.203.52@tcp' SERVICENODE DEBUG: net=tcp1 nid='192.168.203.152@tcp1' SERVICENODE DEBUG: client net=tcp1 nid='192.168.203.52@tcp1' SERVICENODE DEBUG: before temporary target format SERVICENODE DEBUG: HOSTNAME=oleg352-client.virtnet OSS=oleg352-server SERVICENODE DEBUG: MODOPTS_PTLRPC='lbug_on_grant_miscount=1' SERVICENODE DEBUG: MODOPTS_LNET=' external_disc_visibility=nodemap_nidrange' SERVICENODE DEBUG: nets='tcp,tcp1' SERVICENODE DEBUG: nids='192.168.203.152@tcp,192.168.203.152@tcp1' SERVICENODE DEBUG: client LNet configuration net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: tcp local NI(s): - nid: 192.168.203.52@tcp status: up interfaces: 0: ens2 - net type: tcp1 local NI(s): - nid: 192.168.203.52@tcp1 status: up interfaces: 0: ens2 SERVICENODE DEBUG: OSS LNet configuration net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: tcp local NI(s): - nid: 192.168.203.152@tcp status: up interfaces: 0: ens2 - net type: tcp1 local NI(s): - nid: 192.168.203.152@tcp1 status: up interfaces: 0: ens2 peer: - primary nid: 192.168.203.152@tcp Multi-Rail: true peer ni: - nid: 192.168.203.152@tcp state: NA - nid: 192.168.203.152@tcp1 state: NA SERVICENODE DEBUG: temporary fsname='sv14215' SERVICENODE DEBUG: temporary OST1 device='lustre-ost1_2/ost1_2' Permanent disk data: Device: lustre-ost1_2/ost1_2 Target: sv14215:OST0000 Index: 0 Lustre FS: sv14215 Mount type: zfs Flags: 0x1062 (OST first_time update no_primnode ) Persistent mount opts: Parameters: mgsnode=192.168.203.152@tcp sys.timeout=20 failover.node=192.168.203.152@tcp autodegrade=on mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1_2/ost1_2 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1_2/ost1_2 properties lustre:mgsnode=192.168.203.152@tcp lustre:sys.timeout=20 lustre:failover.node=192.168.203.152@tcp lustre:autodegrade=on lustre:version=1 lustre:flags=4194 lustre:index=0 lustre:fsname=sv14215 lustre:svname=sv14215:OST0000 SERVICENODE DEBUG: temporary OST2 device='lustre-ost2_2/ost2_2' Permanent disk data: Device: lustre-ost2_2/ost2_2 Target: sv14215:OST0001 Index: 1 Lustre FS: sv14215 Mount type: zfs Flags: 0x1062 (OST first_time update no_primnode ) Persistent mount opts: Parameters: mgsnode=192.168.203.152@tcp sys.timeout=20 failover.node=192.168.203.152@tcp1 autodegrade=on mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost2_2/ost2_2 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost2_2/ost2_2 properties lustre:mgsnode=192.168.203.152@tcp lustre:sys.timeout=20 lustre:failover.node=192.168.203.152@tcp1 lustre:autodegrade=on lustre:version=1 lustre:flags=4194 lustre:index=1 lustre:fsname=sv14215 lustre:svname=sv14215:OST0001 SERVICENODE DEBUG: starting MGS/MDT and temporary service-node targets start mds service on oleg352-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg352-server: oleg352-server.virtnet: executing set_default_debug -1 all pdsh@oleg352-client: oleg352-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 Start fs2ost1: mount -t lustre -o localrecov lustre-ost1_2/ost1_2 /mnt/lustre-fs2ost1 seq.cli-sv14215-OST0000-super.width=65536 oleg352-server: oleg352-server.virtnet: executing set_default_debug -1 all pdsh@oleg352-client: oleg352-server: ssh exited with exit code 1 Commit the device label on lustre-ost1_2/ost1_2 Started sv14215-OST0000 oleg352-server: missing pool name argument oleg352-server: usage: oleg352-server: trim [-dw] [-r ] [-c | -s] [ ...] pdsh@oleg352-client: oleg352-server: ssh exited with exit code 2 Start fs3ost2: mount -t lustre -o localrecov lustre-ost2_2/ost2_2 /mnt/lustre-fs3ost2 seq.cli-sv14215-OST0001-super.width=65536 oleg352-server: oleg352-server.virtnet: executing set_default_debug -1 all pdsh@oleg352-client: oleg352-server: ssh exited with exit code 1 Commit the device label on lustre-ost2_2/ost2_2 Started sv14215-OST0001 oleg352-server: missing pool name argument oleg352-server: usage: oleg352-server: trim [-dw] [-r ] [-c | -s] [ ...] pdsh@oleg352-client: oleg352-server: ssh exited with exit code 2 SERVICENODE DEBUG: after target startup SERVICENODE DEBUG: HOSTNAME=oleg352-client.virtnet OSS=oleg352-server SERVICENODE DEBUG: MODOPTS_PTLRPC='lbug_on_grant_miscount=1' SERVICENODE DEBUG: MODOPTS_LNET=' external_disc_visibility=nodemap_nidrange' SERVICENODE DEBUG: nets='tcp,tcp1' SERVICENODE DEBUG: nids='192.168.203.152@tcp,192.168.203.152@tcp1' SERVICENODE DEBUG: client LNet configuration net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: tcp local NI(s): - nid: 192.168.203.52@tcp status: up interfaces: 0: ens2 - net type: tcp1 local NI(s): - nid: 192.168.203.52@tcp1 status: up interfaces: 0: ens2 SERVICENODE DEBUG: OSS LNet configuration net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: tcp local NI(s): - nid: 192.168.203.152@tcp status: up interfaces: 0: ens2 - net type: tcp1 local NI(s): - nid: 192.168.203.152@tcp1 status: up interfaces: 0: ens2 peer: - primary nid: 192.168.203.152@tcp Multi-Rail: true peer ni: - nid: 192.168.203.152@tcp state: NA - nid: 192.168.203.152@tcp1 state: NA ping: - primary nid: 192.168.203.152@tcp Multi-Rail: false peer_ni: - nid: 192.168.203.152@tcp ping: - primary nid: 192.168.203.152@tcp1 Multi-Rail: false peer_ni: - nid: 192.168.203.152@tcp1 PASS 57n (295s) == conf-sanity test 57o: nodemap visibility composes with servicenode policy ========================================================== 01:10:15 (1788153015) umount lustre on /mnt/lustre..... stop ost1 service on oleg352-server stop mds service on oleg352-server LNET unconfigure error 22: Invalid argument unloading modules via unload_modules_local on: 'oleg352-server' oleg352-server: oleg352-server.virtnet: executing unload_modules_local oleg352-server: oleg352-server: LNET unconfigure error 22: Invalid argument modules unloaded. setmodopts: MODOPTS_PTLRPC=lbug_on_grant_miscount=1 setmodopts: MODOPTS_PTLRPC=lbug_on_grant_miscount=1 target_nid_policy=servicenode setmodopts: MODOPTS_LNET= external_disc_visibility=nodemap_nidrange SERVICENODE DEBUG: setup base_policy=none servicenode_policy=enabled SERVICENODE DEBUG: before load MODOPTS_PTLRPC='lbug_on_grant_miscount=1 target_nid_policy=servicenode' SERVICENODE DEBUG: before load MODOPTS_LNET=' external_disc_visibility=nodemap_nidrange' /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ../lnet/lnet/lnet options: 'external_disc_visibility=nodemap_nidrange' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1 target_nid_policy=servicenode' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg352-server' oleg352-server: oleg352-server.virtnet: executing load_modules_local oleg352-server: Loading modules from /home/green/git/lustre-release/lustre oleg352-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg352-server: detected 4 online CPUs by sysfs oleg352-server: MODOPTS_LIBCFS= oleg352-server: Force libcfs to create 2 CPU partitions oleg352-server: ../lnet/lnet/lnet options: 'external_disc_visibility=nodemap_nidrange' oleg352-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1 target_nid_policy=servicenode' oleg352-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg352-server: mdt/mdt options: 'mdt_enable_flr_ec=1' SERVICENODE DEBUG: modules loaded oleg352-server: oleg352-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre oleg352-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg352-client: oleg352-server: ssh exited with exit code 19 oleg352-server: oleg352-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre oleg352-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg352-client: oleg352-server: ssh exited with exit code 19 oleg352-server: oleg352-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre oleg352-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg352-client: oleg352-server: ssh exited with exit code 19 tunefs.lustre failed, reformatting instead Stopping clients: oleg352-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg352-client.virtnet /mnt/lustre2 (opts:-f) oleg352-server: oleg352-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg352-server' oleg352-server: oleg352-server.virtnet: executing load_modules_local oleg352-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg352-server: Loading modules from /home/green/git/lustre-release/lustre oleg352-server: detected 4 online CPUs by sysfs oleg352-server: MODOPTS_LIBCFS= oleg352-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg352-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg352-server: oleg352-server.virtnet: executing set_default_debug -1 all pdsh@oleg352-client: oleg352-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg352-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg352-server: oleg352-server.virtnet: executing set_default_debug -1 all pdsh@oleg352-client: oleg352-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg352-server: oleg352-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg352-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg352-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg352-server stop mds service on oleg352-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg352-server SERVICENODE DEBUG: writeconf_or_reformat complete oleg352-server: oleg352-server.virtnet: executing lnet_if_list SERVICENODE DEBUG: oss_if='ens2' client_if='ens2' oleg352-server: sh: /usr/sbin/ksocklnd-config: No such file or directory default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.52 default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.52 SERVICENODE DEBUG: net=tcp nid='192.168.203.152@tcp' SERVICENODE DEBUG: client net=tcp nid='192.168.203.52@tcp' SERVICENODE DEBUG: net=tcp1 nid='192.168.203.152@tcp1' SERVICENODE DEBUG: client net=tcp1 nid='192.168.203.52@tcp1' SERVICENODE DEBUG: before temporary target format SERVICENODE DEBUG: HOSTNAME=oleg352-client.virtnet OSS=oleg352-server SERVICENODE DEBUG: MODOPTS_PTLRPC='lbug_on_grant_miscount=1 target_nid_policy=servicenode' SERVICENODE DEBUG: MODOPTS_LNET=' external_disc_visibility=nodemap_nidrange' SERVICENODE DEBUG: nets='tcp,tcp1' SERVICENODE DEBUG: nids='192.168.203.152@tcp,192.168.203.152@tcp1' SERVICENODE DEBUG: client LNet configuration net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: tcp local NI(s): - nid: 192.168.203.52@tcp status: up interfaces: 0: ens2 - net type: tcp1 local NI(s): - nid: 192.168.203.52@tcp1 status: up interfaces: 0: ens2 SERVICENODE DEBUG: OSS LNet configuration net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: tcp local NI(s): - nid: 192.168.203.152@tcp status: up interfaces: 0: ens2 - net type: tcp1 local NI(s): - nid: 192.168.203.152@tcp1 status: up interfaces: 0: ens2 peer: - primary nid: 192.168.203.152@tcp Multi-Rail: true peer ni: - nid: 192.168.203.152@tcp state: NA SERVICENODE DEBUG: temporary fsname='sv6582' SERVICENODE DEBUG: temporary OST1 device='lustre-ost1_2/ost1_2' Permanent disk data: Device: lustre-ost1_2/ost1_2 Target: sv6582:OST0000 Index: 0 Lustre FS: sv6582 Mount type: zfs Flags: 0x1062 (OST first_time update no_primnode ) Persistent mount opts: Parameters: mgsnode=192.168.203.152@tcp sys.timeout=20 failover.node=192.168.203.152@tcp,192.168.203.152@tcp1 autodegrade=on mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1_2/ost1_2 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1_2/ost1_2 properties lustre:mgsnode=192.168.203.152@tcp lustre:sys.timeout=20 lustre:failover.node=192.168.203.152@tcp,192.168.203.152@tcp1 lustre:autodegrade=on lustre:version=1 lustre:flags=4194 lustre:index=0 lustre:fsname=sv6582 lustre:svname=sv6582:OST0000 SERVICENODE DEBUG: temporary OST2 device='lustre-ost2_2/ost2_2' Permanent disk data: Device: lustre-ost2_2/ost2_2 Target: sv6582:OST0001 Index: 1 Lustre FS: sv6582 Mount type: zfs Flags: 0x1062 (OST first_time update no_primnode ) Persistent mount opts: Parameters: mgsnode=192.168.203.152@tcp sys.timeout=20 failover.node=192.168.203.152@tcp1 autodegrade=on mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost2_2/ost2_2 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost2_2/ost2_2 properties lustre:mgsnode=192.168.203.152@tcp lustre:sys.timeout=20 lustre:failover.node=192.168.203.152@tcp1 lustre:autodegrade=on lustre:version=1 lustre:flags=4194 lustre:index=1 lustre:fsname=sv6582 lustre:svname=sv6582:OST0001 SERVICENODE DEBUG: starting MGS/MDT and temporary service-node targets start mds service on oleg352-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg352-server: oleg352-server.virtnet: executing set_default_debug -1 all pdsh@oleg352-client: oleg352-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 Start fs2ost1: mount -t lustre -o localrecov lustre-ost1_2/ost1_2 /mnt/lustre-fs2ost1 seq.cli-sv6582-OST0000-super.width=65536 oleg352-server: oleg352-server.virtnet: executing set_default_debug -1 all pdsh@oleg352-client: oleg352-server: ssh exited with exit code 1 Commit the device label on lustre-ost1_2/ost1_2 Started sv6582-OST0000 oleg352-server: missing pool name argument oleg352-server: usage: oleg352-server: trim [-dw] [-r ] [-c | -s] [ ...] pdsh@oleg352-client: oleg352-server: ssh exited with exit code 2 SERVICENODE DEBUG: after first target startup SERVICENODE DEBUG: HOSTNAME=oleg352-client.virtnet OSS=oleg352-server SERVICENODE DEBUG: MODOPTS_PTLRPC='lbug_on_grant_miscount=1 target_nid_policy=servicenode' SERVICENODE DEBUG: MODOPTS_LNET=' external_disc_visibility=nodemap_nidrange' SERVICENODE DEBUG: nets='tcp,tcp1' SERVICENODE DEBUG: nids='192.168.203.152@tcp,192.168.203.152@tcp1' SERVICENODE DEBUG: client LNet configuration net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: tcp local NI(s): - nid: 192.168.203.52@tcp status: up interfaces: 0: ens2 - net type: tcp1 local NI(s): - nid: 192.168.203.52@tcp1 status: up interfaces: 0: ens2 SERVICENODE DEBUG: OSS LNet configuration net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: tcp local NI(s): - nid: 192.168.203.152@tcp status: up interfaces: 0: ens2 - net type: tcp1 local NI(s): - nid: 192.168.203.152@tcp1 status: up interfaces: 0: ens2 peer: - primary nid: 192.168.203.152@tcp Multi-Rail: true peer ni: - nid: 192.168.203.152@tcp state: NA - nid: 192.168.203.152@tcp1 state: NA ping: - primary nid: 192.168.203.152@tcp Multi-Rail: false peer_ni: - nid: 192.168.203.152@tcp ping: - primary nid: 192.168.203.152@tcp1 Multi-Rail: false peer_ni: - nid: 192.168.203.152@tcp - nid: 192.168.203.152@tcp1 PASS 57o (297s) == conf-sanity test 57p: nodemap visibility includes server NIDs with routed reachability ========================================================== 01:15:12 (1788153312) umount lustre on /mnt/lustre..... stop ost1 service on oleg352-server stop mds service on oleg352-server LNET unconfigure error 22: Invalid argument unloading modules via unload_modules_local on: 'oleg352-server' oleg352-server: oleg352-server.virtnet: executing unload_modules_local oleg352-server: oleg352-server: LNET unconfigure error 22: Invalid argument modules unloaded. setmodopts: MODOPTS_PTLRPC=lbug_on_grant_miscount=1 setmodopts: MODOPTS_LNET= external_disc_visibility=nodemap_nidrange SERVICENODE DEBUG: setup base_policy=none servicenode_policy=disabled SERVICENODE DEBUG: before load MODOPTS_PTLRPC='lbug_on_grant_miscount=1' SERVICENODE DEBUG: before load MODOPTS_LNET=' external_disc_visibility=nodemap_nidrange' /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ../lnet/lnet/lnet options: 'external_disc_visibility=nodemap_nidrange' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg352-server' oleg352-server: oleg352-server.virtnet: executing load_modules_local oleg352-server: Loading modules from /home/green/git/lustre-release/lustre oleg352-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg352-server: detected 4 online CPUs by sysfs oleg352-server: MODOPTS_LIBCFS= oleg352-server: Force libcfs to create 2 CPU partitions oleg352-server: ../lnet/lnet/lnet options: 'external_disc_visibility=nodemap_nidrange' oleg352-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg352-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg352-server: mdt/mdt options: 'mdt_enable_flr_ec=1' SERVICENODE DEBUG: modules loaded oleg352-server: oleg352-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre oleg352-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg352-client: oleg352-server: ssh exited with exit code 19 oleg352-server: oleg352-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre oleg352-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg352-client: oleg352-server: ssh exited with exit code 19 oleg352-server: oleg352-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre oleg352-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg352-client: oleg352-server: ssh exited with exit code 19 tunefs.lustre failed, reformatting instead Stopping clients: oleg352-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg352-client.virtnet /mnt/lustre2 (opts:-f) oleg352-server: oleg352-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg352-server' oleg352-server: oleg352-server.virtnet: executing load_modules_local oleg352-server: Loading modules from /home/green/git/lustre-release/lustre oleg352-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg352-server: detected 4 online CPUs by sysfs oleg352-server: MODOPTS_LIBCFS= oleg352-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg352-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg352-server: oleg352-server.virtnet: executing set_default_debug -1 all pdsh@oleg352-client: oleg352-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg352-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg352-server: oleg352-server.virtnet: executing set_default_debug -1 all pdsh@oleg352-client: oleg352-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg352-server: oleg352-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg352-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg352-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg352-server stop mds service on oleg352-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg352-server SERVICENODE DEBUG: writeconf_or_reformat complete oleg352-server: oleg352-server.virtnet: executing lnet_if_list SERVICENODE DEBUG: oss_if='ens2' client_if='ens2' oleg352-server: sh: /usr/sbin/ksocklnd-config: No such file or directory default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.52 default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.52 oleg352-server: sh: /usr/sbin/ksocklnd-config: No such file or directory default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.52 default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.52 default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.52 SERVICENODE DEBUG: net=tcp nid='192.168.203.152@tcp' SERVICENODE DEBUG: client net=tcp nid='192.168.203.52@tcp' SERVICENODE DEBUG: net=tcp1 nid='192.168.203.152@tcp1' SERVICENODE DEBUG: client net=tcp1 nid='192.168.203.52@tcp1' SERVICENODE DEBUG: net=tcp2 nid='192.168.203.152@tcp2' SERVICENODE DEBUG: client net=tcp2 nid='192.168.203.52@tcp2' SERVICENODE DEBUG: before temporary target format SERVICENODE DEBUG: HOSTNAME=oleg352-client.virtnet OSS=oleg352-server SERVICENODE DEBUG: MODOPTS_PTLRPC='lbug_on_grant_miscount=1' SERVICENODE DEBUG: MODOPTS_LNET=' external_disc_visibility=nodemap_nidrange' SERVICENODE DEBUG: nets='tcp,tcp1,tcp2' SERVICENODE DEBUG: nids='192.168.203.152@tcp,192.168.203.152@tcp1,192.168.203.152@tcp2' SERVICENODE DEBUG: client LNet configuration net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: tcp local NI(s): - nid: 192.168.203.52@tcp status: up interfaces: 0: ens2 - net type: tcp1 local NI(s): - nid: 192.168.203.52@tcp1 status: up interfaces: 0: ens2 - net type: tcp2 local NI(s): - nid: 192.168.203.52@tcp2 status: up interfaces: 0: ens2 SERVICENODE DEBUG: OSS LNet configuration net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: tcp local NI(s): - nid: 192.168.203.152@tcp status: up interfaces: 0: ens2 - net type: tcp1 local NI(s): - nid: 192.168.203.152@tcp1 status: up interfaces: 0: ens2 - net type: tcp2 local NI(s): - nid: 192.168.203.152@tcp2 status: up interfaces: 0: ens2 peer: - primary nid: 192.168.203.152@tcp Multi-Rail: true peer ni: - nid: 192.168.203.152@tcp state: NA - nid: 192.168.203.152@tcp1 state: NA - nid: 192.168.203.152@tcp2 state: NA SERVICENODE DEBUG: temporary fsname='sv19286' SERVICENODE DEBUG: temporary OST1 device='lustre-ost1_2/ost1_2' Permanent disk data: Device: lustre-ost1_2/ost1_2 Target: sv19286:OST0000 Index: 0 Lustre FS: sv19286 Mount type: zfs Flags: 0x1062 (OST first_time update no_primnode ) Persistent mount opts: Parameters: mgsnode=192.168.203.152@tcp sys.timeout=20 failover.node=192.168.203.152@tcp autodegrade=on mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1_2/ost1_2 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1_2/ost1_2 properties lustre:mgsnode=192.168.203.152@tcp lustre:sys.timeout=20 lustre:failover.node=192.168.203.152@tcp lustre:autodegrade=on lustre:version=1 lustre:flags=4194 lustre:index=0 lustre:fsname=sv19286 lustre:svname=sv19286:OST0000 SERVICENODE DEBUG: temporary OST2 device='lustre-ost2_2/ost2_2' Permanent disk data: Device: lustre-ost2_2/ost2_2 Target: sv19286:OST0001 Index: 1 Lustre FS: sv19286 Mount type: zfs Flags: 0x1062 (OST first_time update no_primnode ) Persistent mount opts: Parameters: mgsnode=192.168.203.152@tcp sys.timeout=20 failover.node=192.168.203.152@tcp1 autodegrade=on mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost2_2/ost2_2 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost2_2/ost2_2 properties lustre:mgsnode=192.168.203.152@tcp lustre:sys.timeout=20 lustre:failover.node=192.168.203.152@tcp1 lustre:autodegrade=on lustre:version=1 lustre:flags=4194 lustre:index=1 lustre:fsname=sv19286 lustre:svname=sv19286:OST0001 SERVICENODE DEBUG: starting MGS/MDT and temporary service-node targets start mds service on oleg352-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg352-server: oleg352-server.virtnet: executing set_default_debug -1 all pdsh@oleg352-client: oleg352-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 Start fs2ost1: mount -t lustre -o localrecov lustre-ost1_2/ost1_2 /mnt/lustre-fs2ost1 seq.cli-sv19286-OST0000-super.width=65536 oleg352-server: oleg352-server.virtnet: executing set_default_debug -1 all pdsh@oleg352-client: oleg352-server: ssh exited with exit code 1 Commit the device label on lustre-ost1_2/ost1_2 Started sv19286-OST0000 oleg352-server: missing pool name argument oleg352-server: usage: oleg352-server: trim [-dw] [-r ] [-c | -s] [ ...] pdsh@oleg352-client: oleg352-server: ssh exited with exit code 2 Start fs3ost2: mount -t lustre -o localrecov lustre-ost2_2/ost2_2 /mnt/lustre-fs3ost2 seq.cli-sv19286-OST0001-super.width=65536 oleg352-server: oleg352-server.virtnet: executing set_default_debug -1 all pdsh@oleg352-client: oleg352-server: ssh exited with exit code 1 Commit the device label on lustre-ost2_2/ost2_2 Started sv19286-OST0001 oleg352-server: missing pool name argument oleg352-server: usage: oleg352-server: trim [-dw] [-r ] [-c | -s] [ ...] pdsh@oleg352-client: oleg352-server: ssh exited with exit code 2 SERVICENODE DEBUG: after target startup SERVICENODE DEBUG: HOSTNAME=oleg352-client.virtnet OSS=oleg352-server SERVICENODE DEBUG: MODOPTS_PTLRPC='lbug_on_grant_miscount=1' SERVICENODE DEBUG: MODOPTS_LNET=' external_disc_visibility=nodemap_nidrange' SERVICENODE DEBUG: nets='tcp,tcp1,tcp2' SERVICENODE DEBUG: nids='192.168.203.152@tcp,192.168.203.152@tcp1,192.168.203.152@tcp2' SERVICENODE DEBUG: client LNet configuration net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: tcp local NI(s): - nid: 192.168.203.52@tcp status: up interfaces: 0: ens2 - net type: tcp1 local NI(s): - nid: 192.168.203.52@tcp1 status: up interfaces: 0: ens2 - net type: tcp2 local NI(s): - nid: 192.168.203.52@tcp2 status: up interfaces: 0: ens2 SERVICENODE DEBUG: OSS LNet configuration net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: tcp local NI(s): - nid: 192.168.203.152@tcp status: up interfaces: 0: ens2 - net type: tcp1 local NI(s): - nid: 192.168.203.152@tcp1 status: up interfaces: 0: ens2 - net type: tcp2 local NI(s): - nid: 192.168.203.152@tcp2 status: up interfaces: 0: ens2 peer: - primary nid: 192.168.203.152@tcp Multi-Rail: true peer ni: - nid: 192.168.203.152@tcp state: NA - nid: 192.168.203.152@tcp1 state: NA - nid: 192.168.203.152@tcp2 state: NA SKIP: conf-sanity test_57p need a remote client with no Lustre server role for routing SKIP 57p (298s) == conf-sanity test 57q: active nodemap visibility fails closed while state is replaced ========================================================== 01:20:10 (1788153610) umount lustre on /mnt/lustre..... stop ost1 service on oleg352-server stop mds service on oleg352-server LNET unconfigure error 22: Invalid argument unloading modules via unload_modules_local on: 'oleg352-server' oleg352-server: oleg352-server.virtnet: executing unload_modules_local oleg352-server: oleg352-server: LNET unconfigure error 22: Invalid argument modules unloaded. setmodopts: MODOPTS_PTLRPC=lbug_on_grant_miscount=1 setmodopts: MODOPTS_LNET= external_disc_visibility=nodemap_nidrange SERVICENODE DEBUG: setup base_policy=none servicenode_policy=disabled SERVICENODE DEBUG: before load MODOPTS_PTLRPC='lbug_on_grant_miscount=1' SERVICENODE DEBUG: before load MODOPTS_LNET=' external_disc_visibility=nodemap_nidrange' /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ../lnet/lnet/lnet options: 'external_disc_visibility=nodemap_nidrange' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg352-server' oleg352-server: oleg352-server.virtnet: executing load_modules_local oleg352-server: Loading modules from /home/green/git/lustre-release/lustre oleg352-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg352-server: detected 4 online CPUs by sysfs oleg352-server: MODOPTS_LIBCFS= oleg352-server: Force libcfs to create 2 CPU partitions oleg352-server: ../lnet/lnet/lnet options: 'external_disc_visibility=nodemap_nidrange' oleg352-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg352-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg352-server: mdt/mdt options: 'mdt_enable_flr_ec=1' SERVICENODE DEBUG: modules loaded oleg352-server: oleg352-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre oleg352-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg352-client: oleg352-server: ssh exited with exit code 19 oleg352-server: oleg352-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre oleg352-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg352-client: oleg352-server: ssh exited with exit code 19 oleg352-server: oleg352-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre oleg352-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg352-client: oleg352-server: ssh exited with exit code 19 tunefs.lustre failed, reformatting instead Stopping clients: oleg352-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg352-client.virtnet /mnt/lustre2 (opts:-f) oleg352-server: oleg352-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg352-server' oleg352-server: oleg352-server.virtnet: executing load_modules_local oleg352-server: Loading modules from /home/green/git/lustre-release/lustre oleg352-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg352-server: detected 4 online CPUs by sysfs oleg352-server: MODOPTS_LIBCFS= oleg352-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg352-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg352-server: oleg352-server.virtnet: executing set_default_debug -1 all pdsh@oleg352-client: oleg352-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg352-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg352-server: oleg352-server.virtnet: executing set_default_debug -1 all pdsh@oleg352-client: oleg352-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg352-server: oleg352-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg352-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg352-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg352-server stop mds service on oleg352-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg352-server SERVICENODE DEBUG: writeconf_or_reformat complete oleg352-server: oleg352-server.virtnet: executing lnet_if_list SERVICENODE DEBUG: oss_if='ens2' client_if='ens2' oleg352-server: sh: /usr/sbin/ksocklnd-config: No such file or directory default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.52 default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.52 SERVICENODE DEBUG: net=tcp nid='192.168.203.152@tcp' SERVICENODE DEBUG: client net=tcp nid='192.168.203.52@tcp' SERVICENODE DEBUG: net=tcp1 nid='192.168.203.152@tcp1' SERVICENODE DEBUG: client net=tcp1 nid='192.168.203.52@tcp1' SERVICENODE DEBUG: before temporary target format SERVICENODE DEBUG: HOSTNAME=oleg352-client.virtnet OSS=oleg352-server SERVICENODE DEBUG: MODOPTS_PTLRPC='lbug_on_grant_miscount=1' SERVICENODE DEBUG: MODOPTS_LNET=' external_disc_visibility=nodemap_nidrange' SERVICENODE DEBUG: nets='tcp,tcp1' SERVICENODE DEBUG: nids='192.168.203.152@tcp,192.168.203.152@tcp1' SERVICENODE DEBUG: client LNet configuration net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: tcp local NI(s): - nid: 192.168.203.52@tcp status: up interfaces: 0: ens2 - net type: tcp1 local NI(s): - nid: 192.168.203.52@tcp1 status: up interfaces: 0: ens2 SERVICENODE DEBUG: OSS LNet configuration net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: tcp local NI(s): - nid: 192.168.203.152@tcp status: up interfaces: 0: ens2 - net type: tcp1 local NI(s): - nid: 192.168.203.152@tcp1 status: up interfaces: 0: ens2 peer: - primary nid: 192.168.203.152@tcp Multi-Rail: true peer ni: - nid: 192.168.203.152@tcp state: NA - nid: 192.168.203.152@tcp1 state: NA SERVICENODE DEBUG: temporary fsname='sv9948' SERVICENODE DEBUG: temporary OST1 device='lustre-ost1_2/ost1_2' Permanent disk data: Device: lustre-ost1_2/ost1_2 Target: sv9948:OST0000 Index: 0 Lustre FS: sv9948 Mount type: zfs Flags: 0x1062 (OST first_time update no_primnode ) Persistent mount opts: Parameters: mgsnode=192.168.203.152@tcp sys.timeout=20 failover.node=192.168.203.152@tcp autodegrade=on mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1_2/ost1_2 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1_2/ost1_2 properties lustre:mgsnode=192.168.203.152@tcp lustre:sys.timeout=20 lustre:failover.node=192.168.203.152@tcp lustre:autodegrade=on lustre:version=1 lustre:flags=4194 lustre:index=0 lustre:fsname=sv9948 lustre:svname=sv9948:OST0000 SERVICENODE DEBUG: temporary OST2 device='lustre-ost2_2/ost2_2' Permanent disk data: Device: lustre-ost2_2/ost2_2 Target: sv9948:OST0001 Index: 1 Lustre FS: sv9948 Mount type: zfs Flags: 0x1062 (OST first_time update no_primnode ) Persistent mount opts: Parameters: mgsnode=192.168.203.152@tcp sys.timeout=20 failover.node=192.168.203.152@tcp1 autodegrade=on mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost2_2/ost2_2 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost2_2/ost2_2 properties lustre:mgsnode=192.168.203.152@tcp lustre:sys.timeout=20 lustre:failover.node=192.168.203.152@tcp1 lustre:autodegrade=on lustre:version=1 lustre:flags=4194 lustre:index=1 lustre:fsname=sv9948 lustre:svname=sv9948:OST0001 SERVICENODE DEBUG: starting MGS/MDT and temporary service-node targets start mds service on oleg352-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg352-server: oleg352-server.virtnet: executing set_default_debug -1 all pdsh@oleg352-client: oleg352-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 Start fs2ost1: mount -t lustre -o localrecov lustre-ost1_2/ost1_2 /mnt/lustre-fs2ost1 seq.cli-sv9948-OST0000-super.width=65536 oleg352-server: oleg352-server.virtnet: executing set_default_debug -1 all pdsh@oleg352-client: oleg352-server: ssh exited with exit code 1 Commit the device label on lustre-ost1_2/ost1_2 Started sv9948-OST0000 oleg352-server: missing pool name argument oleg352-server: usage: oleg352-server: trim [-dw] [-r ] [-c | -s] [ ...] pdsh@oleg352-client: oleg352-server: ssh exited with exit code 2 Start fs3ost2: mount -t lustre -o localrecov lustre-ost2_2/ost2_2 /mnt/lustre-fs3ost2 seq.cli-sv9948-OST0001-super.width=65536 oleg352-server: oleg352-server.virtnet: executing set_default_debug -1 all pdsh@oleg352-client: oleg352-server: ssh exited with exit code 1 Commit the device label on lustre-ost2_2/ost2_2 Started sv9948-OST0001 oleg352-server: missing pool name argument oleg352-server: usage: oleg352-server: trim [-dw] [-r ] [-c | -s] [ ...] pdsh@oleg352-client: oleg352-server: ssh exited with exit code 2 SERVICENODE DEBUG: after target startup SERVICENODE DEBUG: HOSTNAME=oleg352-client.virtnet OSS=oleg352-server SERVICENODE DEBUG: MODOPTS_PTLRPC='lbug_on_grant_miscount=1' SERVICENODE DEBUG: MODOPTS_LNET=' external_disc_visibility=nodemap_nidrange' SERVICENODE DEBUG: nets='tcp,tcp1' SERVICENODE DEBUG: nids='192.168.203.152@tcp,192.168.203.152@tcp1' SERVICENODE DEBUG: client LNet configuration net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: tcp local NI(s): - nid: 192.168.203.52@tcp status: up interfaces: 0: ens2 - net type: tcp1 local NI(s): - nid: 192.168.203.52@tcp1 status: up interfaces: 0: ens2 SERVICENODE DEBUG: OSS LNet configuration net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: tcp local NI(s): - nid: 192.168.203.152@tcp status: up interfaces: 0: ens2 - net type: tcp1 local NI(s): - nid: 192.168.203.152@tcp1 status: up interfaces: 0: ens2 peer: - primary nid: 192.168.203.152@tcp Multi-Rail: true peer ni: - nid: 192.168.203.152@tcp state: NA - nid: 192.168.203.152@tcp1 state: NA ping: - primary nid: 192.168.203.152@tcp Multi-Rail: false peer_ni: - nid: 192.168.203.152@tcp fail_loc=0x80002404 fail_val=15 ping: - primary nid: 192.168.203.152@tcp Multi-Rail: false peer_ni: - nid: 192.168.203.152@tcp - nid: 192.168.203.152@tcp1 PASS 57q (341s) == conf-sanity test 57r: MGS restart publishes persisted nodemap visibility ========================================================== 01:25:51 (1788153951) umount lustre on /mnt/lustre..... stop ost1 service on oleg352-server stop mds service on oleg352-server LNET unconfigure error 22: Invalid argument unloading modules via unload_modules_local on: 'oleg352-server' oleg352-server: oleg352-server.virtnet: executing unload_modules_local oleg352-server: oleg352-server: LNET unconfigure error 22: Invalid argument modules unloaded. setmodopts: MODOPTS_PTLRPC=lbug_on_grant_miscount=1 setmodopts: MODOPTS_LNET= external_disc_visibility=nodemap_nidrange SERVICENODE DEBUG: setup base_policy=none servicenode_policy=disabled SERVICENODE DEBUG: before load MODOPTS_PTLRPC='lbug_on_grant_miscount=1' SERVICENODE DEBUG: before load MODOPTS_LNET=' external_disc_visibility=nodemap_nidrange' /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ../lnet/lnet/lnet options: 'external_disc_visibility=nodemap_nidrange' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg352-server' oleg352-server: oleg352-server.virtnet: executing load_modules_local oleg352-server: Loading modules from /home/green/git/lustre-release/lustre oleg352-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg352-server: detected 4 online CPUs by sysfs oleg352-server: MODOPTS_LIBCFS= oleg352-server: Force libcfs to create 2 CPU partitions oleg352-server: ../lnet/lnet/lnet options: 'external_disc_visibility=nodemap_nidrange' oleg352-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg352-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg352-server: mdt/mdt options: 'mdt_enable_flr_ec=1' SERVICENODE DEBUG: modules loaded oleg352-server: oleg352-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre oleg352-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg352-client: oleg352-server: ssh exited with exit code 19 oleg352-server: oleg352-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre oleg352-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg352-client: oleg352-server: ssh exited with exit code 19 oleg352-server: oleg352-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre oleg352-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg352-client: oleg352-server: ssh exited with exit code 19 tunefs.lustre failed, reformatting instead Stopping clients: oleg352-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg352-client.virtnet /mnt/lustre2 (opts:-f) oleg352-server: oleg352-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg352-server' oleg352-server: oleg352-server.virtnet: executing load_modules_local oleg352-server: Loading modules from /home/green/git/lustre-release/lustre oleg352-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg352-server: detected 4 online CPUs by sysfs oleg352-server: MODOPTS_LIBCFS= oleg352-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg352-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg352-server: oleg352-server.virtnet: executing set_default_debug -1 all pdsh@oleg352-client: oleg352-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg352-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg352-server: oleg352-server.virtnet: executing set_default_debug -1 all pdsh@oleg352-client: oleg352-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg352-server: oleg352-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg352-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg352-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg352-server stop mds service on oleg352-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg352-server SERVICENODE DEBUG: writeconf_or_reformat complete oleg352-server: oleg352-server.virtnet: executing lnet_if_list SERVICENODE DEBUG: oss_if='ens2' client_if='ens2' oleg352-server: sh: /usr/sbin/ksocklnd-config: No such file or directory default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.52 default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.52 SERVICENODE DEBUG: net=tcp nid='192.168.203.152@tcp' SERVICENODE DEBUG: client net=tcp nid='192.168.203.52@tcp' SERVICENODE DEBUG: net=tcp1 nid='192.168.203.152@tcp1' SERVICENODE DEBUG: client net=tcp1 nid='192.168.203.52@tcp1' SERVICENODE DEBUG: before temporary target format SERVICENODE DEBUG: HOSTNAME=oleg352-client.virtnet OSS=oleg352-server SERVICENODE DEBUG: MODOPTS_PTLRPC='lbug_on_grant_miscount=1' SERVICENODE DEBUG: MODOPTS_LNET=' external_disc_visibility=nodemap_nidrange' SERVICENODE DEBUG: nets='tcp,tcp1' SERVICENODE DEBUG: nids='192.168.203.152@tcp,192.168.203.152@tcp1' SERVICENODE DEBUG: client LNet configuration net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: tcp local NI(s): - nid: 192.168.203.52@tcp status: up interfaces: 0: ens2 - net type: tcp1 local NI(s): - nid: 192.168.203.52@tcp1 status: up interfaces: 0: ens2 SERVICENODE DEBUG: OSS LNet configuration net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: tcp local NI(s): - nid: 192.168.203.152@tcp status: up interfaces: 0: ens2 - net type: tcp1 local NI(s): - nid: 192.168.203.152@tcp1 status: up interfaces: 0: ens2 peer: - primary nid: 192.168.203.152@tcp Multi-Rail: true peer ni: - nid: 192.168.203.152@tcp state: NA - nid: 192.168.203.152@tcp1 state: NA SERVICENODE DEBUG: stopping setup after LNet config start mds service on oleg352-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg352-server: oleg352-server.virtnet: executing set_default_debug -1 all pdsh@oleg352-client: oleg352-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg352-server: oleg352-server.virtnet: executing lnet_if_list oleg352-server: rpc test_57r: @@@@@@ FAIL: Can't determine interface name for NID 192.168.203.152@tcp1 oleg352-server: Trace dump: oleg352-server: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7448:error() oleg352-server: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:8109:lnet_if_list() oleg352-server: = /home/green/git/lustre-release/lustre/tests/rpc.sh:20:main() oleg352-server: oleg352-server.virtnet: executing check_logdir /tmp/test_logs/1788154181 pdsh@oleg352-client: oleg352-server: ssh exited with exit code 1 ping: - primary nid: 192.168.203.152@tcp Multi-Rail: false peer_ni: - nid: 192.168.203.152@tcp Stopping clients: oleg352-client.virtnet /mnt/lustre (opts:) Stopping clients: oleg352-client.virtnet /mnt/lustre2 (opts:) Stopping /mnt/lustre-mds1 (opts:-f) on oleg352-server unloading modules via unload_modules_local on: 'oleg352-server' oleg352-server: oleg352-server.virtnet: executing unload_modules_local oleg352-server: modules unloaded. /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ../lnet/lnet/lnet options: 'external_disc_visibility=nodemap_nidrange' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg352-server' oleg352-server: oleg352-server.virtnet: executing load_modules_local oleg352-server: Loading modules from /home/green/git/lustre-release/lustre oleg352-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg352-server: detected 4 online CPUs by sysfs oleg352-server: MODOPTS_LIBCFS= oleg352-server: Force libcfs to create 2 CPU partitions oleg352-server: ../lnet/lnet/lnet options: 'external_disc_visibility=nodemap_nidrange' oleg352-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg352-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg352-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg352-server: oleg352-server.virtnet: executing lnet_if_list oleg352-server: sh: /usr/sbin/ksocklnd-config: No such file or directory start mds service on oleg352-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg352-server: oleg352-server.virtnet: executing set_default_debug -1 all pdsh@oleg352-client: oleg352-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 ping: - primary nid: 192.168.203.152@tcp Multi-Rail: false peer_ni: - nid: 192.168.203.152@tcp PASS 57r (358s) Stopping clients: oleg352-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg352-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg352-client: oleg352-server: ssh exited with exit code 2 oleg352-server: oleg352-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg352-server' oleg352-server: oleg352-server.virtnet: executing load_modules_local oleg352-server: Loading modules from /home/green/git/lustre-release/lustre oleg352-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg352-server: detected 4 online CPUs by sysfs oleg352-server: MODOPTS_LIBCFS= oleg352-server: Force libcfs to create 2 CPU partitions oleg352-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg352-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg352-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 2445 sec ========== 01:33:18 (1788154398) === conf-sanity: start cleanup 01:33:20 (1788154400) === === conf-sanity: finish cleanup 01:33:25 (1788154405) ===