-----============= acceptance-small: conf-sanity ============----- Tue Sep 8 17:34:01 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg242-server: /home/green/git/lustre-release/lustre/tests/except/conf-sanity.ex - see CLIENT_VERSION > v2_15_63-53-g1a6ef725c2 (34682650 > 34553653) for LU-16938, go 81 excepting tests: 32newtarball 108a 112a 112b 113 117 119 121 122a 123aa 123ab 123ac 123ad 123ae 123af 123ag 123ah 123ahi 123F 123G 123H 126 129 132 133 135 136 137 150 152 153a 153b 153c 155 802a skipping tests SLOW=no: 45 69 106 111 114 Stopping clients: oleg242-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg242-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg242-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg242-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg242-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg242-server oleg242-server: oleg242-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg242-server' oleg242-server: oleg242-server.virtnet: executing load_modules_local oleg242-server: Loading modules from /home/green/git/lustre-release/lustre oleg242-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg242-server: detected 4 online CPUs by sysfs oleg242-server: MODOPTS_LIBCFS= oleg242-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg242-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg242-server: oleg242-server.virtnet: executing set_default_debug -1 all pdsh@oleg242-client: oleg242-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg242-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg242-server: oleg242-server.virtnet: executing set_default_debug -1 all pdsh@oleg242-client: oleg242-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg242-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg242-server: oleg242-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg242-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg242-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg242-server stop mds service on oleg242-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg242-server == conf-sanity test 57b: initial registration from servicenode should not fail ========================================================== 17:37:10 (1788903430) oleg242-server: oleg242-server.virtnet: executing load_modules_local oleg242-server: Loading modules from /home/green/git/lustre-release/lustre oleg242-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg242-server: detected 4 online CPUs by sysfs oleg242-server: MODOPTS_LIBCFS= oleg242-server: Force libcfs to create 2 CPU partitions oleg242-server: oleg242-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre oleg242-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg242-client: oleg242-server: ssh exited with exit code 19 oleg242-server: oleg242-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre oleg242-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg242-client: oleg242-server: ssh exited with exit code 19 oleg242-server: oleg242-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre oleg242-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg242-client: oleg242-server: ssh exited with exit code 19 tunefs.lustre failed, reformatting instead Stopping clients: oleg242-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg242-client.virtnet /mnt/lustre2 (opts:-f) oleg242-server: oleg242-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg242-server' oleg242-server: oleg242-server.virtnet: executing load_modules_local oleg242-server: Loading modules from /home/green/git/lustre-release/lustre oleg242-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg242-server: detected 4 online CPUs by sysfs oleg242-server: MODOPTS_LIBCFS= oleg242-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg242-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg242-server: oleg242-server.virtnet: executing set_default_debug -1 all pdsh@oleg242-client: oleg242-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg242-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg242-server: oleg242-server.virtnet: executing set_default_debug -1 all pdsh@oleg242-client: oleg242-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg242-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg242-server: oleg242-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg242-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg242-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg242-server stop mds service on oleg242-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg242-server checking for existing Lustre data: found Read previous values: Device: lustre-ost1/ost1 Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x2 (OST ) Persistent mount opts: Parameters: sys.timeout=20 autodegrade=on mgsnode=192.168.202.142@tcp Permanent disk data: Device: lustre-ost1/ost1 Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x1042 (OST update no_primnode ) Persistent mount opts: Parameters: sys.timeout=20 autodegrade=on mgsnode=192.168.202.142@tcp failover.node=192.168.202.142@tcp autodegrade=on Writing lustre-ost1/ost1 properties lustre:sys.timeout=20 lustre:autodegrade=on lustre:mgsnode=192.168.202.142@tcp lustre:failover.node=192.168.202.142@tcp lustre:autodegrade=on lustre:version=1 lustre:flags=4162 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start mds service on oleg242-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg242-server: oleg242-server.virtnet: executing set_default_debug -1 all pdsh@oleg242-client: oleg242-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg242-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg242-server: oleg242-server.virtnet: executing set_default_debug -1 all pdsh@oleg242-client: oleg242-server: ssh exited with exit code 1 Started lustre-OST0000 oleg242-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 umount lustre on /mnt/lustre..... stop ost1 service on oleg242-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg242-server stop mds service on oleg242-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg242-server unloading modules via unload_modules_local on: 'oleg242-server' oleg242-server: oleg242-server.virtnet: executing unload_modules_local oleg242-server: modules unloaded. pdsh@oleg242-client: oleg242-client: ssh exited with exit code 2 pdsh@oleg242-client: oleg242-server: ssh exited with exit code 2 pdsh@oleg242-client: oleg242-server: ssh exited with exit code 2 pdsh@oleg242-client: oleg242-client: ssh exited with exit code 2 PASS 57b (151s) == conf-sanity test 57c: servicenode metadata preserves legacy discovery by default ========================================================== 17:39:41 (1788903581) SERVICENODE DEBUG: begin legacy-policy test umount lustre on /mnt/lustre..... stop ost1 service on oleg242-server stop mds service on oleg242-server LNET unconfigure error 22: Invalid argument unloading modules via unload_modules_local on: 'oleg242-server' oleg242-server: oleg242-server.virtnet: executing unload_modules_local oleg242-server: oleg242-server: LNET unconfigure error 22: Invalid argument modules unloaded. setmodopts: MODOPTS_PTLRPC=lbug_on_grant_miscount=1 SERVICENODE DEBUG: setup base_policy=none servicenode_policy=disabled SERVICENODE DEBUG: before load MODOPTS_PTLRPC='lbug_on_grant_miscount=1' SERVICENODE DEBUG: before load MODOPTS_LNET='' /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg242-server' oleg242-server: oleg242-server.virtnet: executing load_modules_local oleg242-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg242-server: Loading modules from /home/green/git/lustre-release/lustre oleg242-server: detected 4 online CPUs by sysfs oleg242-server: MODOPTS_LIBCFS= oleg242-server: Force libcfs to create 2 CPU partitions oleg242-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg242-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg242-server: mdt/mdt options: 'mdt_enable_flr_ec=1' SERVICENODE DEBUG: modules loaded oleg242-server: oleg242-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre oleg242-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg242-client: oleg242-server: ssh exited with exit code 19 oleg242-server: oleg242-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre oleg242-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg242-client: oleg242-server: ssh exited with exit code 19 oleg242-server: oleg242-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre oleg242-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg242-client: oleg242-server: ssh exited with exit code 19 tunefs.lustre failed, reformatting instead Stopping clients: oleg242-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg242-client.virtnet /mnt/lustre2 (opts:-f) oleg242-server: oleg242-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg242-server' oleg242-server: oleg242-server.virtnet: executing load_modules_local oleg242-server: Loading modules from /home/green/git/lustre-release/lustre oleg242-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg242-server: detected 4 online CPUs by sysfs oleg242-server: MODOPTS_LIBCFS= oleg242-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg242-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg242-server: oleg242-server.virtnet: executing set_default_debug -1 all pdsh@oleg242-client: oleg242-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg242-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg242-server: oleg242-server.virtnet: executing set_default_debug -1 all pdsh@oleg242-client: oleg242-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg242-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg242-server: oleg242-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg242-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg242-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg242-server stop mds service on oleg242-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg242-server SERVICENODE DEBUG: writeconf_or_reformat complete oleg242-server: oleg242-server.virtnet: executing lnet_if_list SERVICENODE DEBUG: oss_if='ens2' client_if='ens2' oleg242-server: sh: /usr/sbin/ksocklnd-config: No such file or directory SERVICENODE DEBUG: net=tcp nid='192.168.202.142@tcp' SERVICENODE DEBUG: net=tcp1 nid='192.168.202.142@tcp1' SERVICENODE DEBUG: before temporary target format SERVICENODE DEBUG: HOSTNAME=oleg242-client.virtnet OSS=oleg242-server SERVICENODE DEBUG: MODOPTS_PTLRPC='lbug_on_grant_miscount=1' SERVICENODE DEBUG: MODOPTS_LNET='' SERVICENODE DEBUG: nets='tcp,tcp1' SERVICENODE DEBUG: nids='192.168.202.142@tcp,192.168.202.142@tcp1' SERVICENODE DEBUG: client LNet configuration net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: tcp local NI(s): - nid: 192.168.202.42@tcp status: up interfaces: 0: ens2 - net type: tcp1 local NI(s): - nid: 192.168.202.42@tcp1 status: up interfaces: 0: ens2 SERVICENODE DEBUG: OSS LNet configuration net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: tcp local NI(s): - nid: 192.168.202.142@tcp status: up interfaces: 0: ens2 - net type: tcp1 local NI(s): - nid: 192.168.202.142@tcp1 status: up interfaces: 0: ens2 peer: - primary nid: 192.168.202.142@tcp Multi-Rail: true peer ni: - nid: 192.168.202.142@tcp state: NA - nid: 192.168.202.142@tcp1 state: NA SERVICENODE DEBUG: temporary fsname='sv31396' SERVICENODE DEBUG: temporary OST1 device='lustre-ost1_2/ost1_2' Permanent disk data: Device: lustre-ost1_2/ost1_2 Target: sv31396:OST0000 Index: 0 Lustre FS: sv31396 Mount type: zfs Flags: 0x1062 (OST first_time update no_primnode ) Persistent mount opts: Parameters: mgsnode=192.168.202.142@tcp sys.timeout=20 failover.node=192.168.202.142@tcp autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1_2 /tmp/lustre-ost1_2 mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1_2/ost1_2 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1_2/ost1_2 properties lustre:mgsnode=192.168.202.142@tcp lustre:sys.timeout=20 lustre:failover.node=192.168.202.142@tcp lustre:autodegrade=on lustre:version=1 lustre:flags=4194 lustre:index=0 lustre:fsname=sv31396 lustre:svname=sv31396:OST0000 SERVICENODE DEBUG: temporary OST2 device='lustre-ost2_2/ost2_2' Permanent disk data: Device: lustre-ost2_2/ost2_2 Target: sv31396:OST0001 Index: 1 Lustre FS: sv31396 Mount type: zfs Flags: 0x1062 (OST first_time update no_primnode ) Persistent mount opts: Parameters: mgsnode=192.168.202.142@tcp sys.timeout=20 failover.node=192.168.202.142@tcp1 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost2_2 /tmp/lustre-ost2_2 mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost2_2/ost2_2 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost2_2/ost2_2 properties lustre:mgsnode=192.168.202.142@tcp lustre:sys.timeout=20 lustre:failover.node=192.168.202.142@tcp1 lustre:autodegrade=on lustre:version=1 lustre:flags=4194 lustre:index=1 lustre:fsname=sv31396 lustre:svname=sv31396:OST0001 SERVICENODE DEBUG: starting MGS/MDT and temporary service-node targets start mds service on oleg242-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg242-server: oleg242-server.virtnet: executing set_default_debug -1 all pdsh@oleg242-client: oleg242-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 Start fs2ost1: mount -t lustre -o localrecov lustre-ost1_2/ost1_2 /mnt/lustre-fs2ost1 seq.cli-sv31396-OST0000-super.width=65536 oleg242-server: oleg242-server.virtnet: executing set_default_debug -1 all pdsh@oleg242-client: oleg242-server: ssh exited with exit code 1 Commit the device label on lustre-ost1_2/ost1_2 Started sv31396-OST0000 oleg242-server: missing pool name argument oleg242-server: usage: oleg242-server: trim [-dw] [-r ] [-c | -s] [ ...] pdsh@oleg242-client: oleg242-server: ssh exited with exit code 2 Start fs3ost2: mount -t lustre -o localrecov lustre-ost2_2/ost2_2 /mnt/lustre-fs3ost2 seq.cli-sv31396-OST0001-super.width=65536 oleg242-server: oleg242-server.virtnet: executing set_default_debug -1 all pdsh@oleg242-client: oleg242-server: ssh exited with exit code 1 Commit the device label on lustre-ost2_2/ost2_2 Started sv31396-OST0001 oleg242-server: missing pool name argument oleg242-server: usage: oleg242-server: trim [-dw] [-r ] [-c | -s] [ ...] pdsh@oleg242-client: oleg242-server: ssh exited with exit code 2 SERVICENODE DEBUG: after target startup SERVICENODE DEBUG: HOSTNAME=oleg242-client.virtnet OSS=oleg242-server SERVICENODE DEBUG: MODOPTS_PTLRPC='lbug_on_grant_miscount=1' SERVICENODE DEBUG: MODOPTS_LNET='' SERVICENODE DEBUG: nets='tcp,tcp1' SERVICENODE DEBUG: nids='192.168.202.142@tcp,192.168.202.142@tcp1' SERVICENODE DEBUG: client LNet configuration net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: tcp local NI(s): - nid: 192.168.202.42@tcp status: up interfaces: 0: ens2 - net type: tcp1 local NI(s): - nid: 192.168.202.42@tcp1 status: up interfaces: 0: ens2 SERVICENODE DEBUG: OSS LNet configuration net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: tcp local NI(s): - nid: 192.168.202.142@tcp status: up interfaces: 0: ens2 - net type: tcp1 local NI(s): - nid: 192.168.202.142@tcp1 status: up interfaces: 0: ens2 peer: - primary nid: 192.168.202.142@tcp Multi-Rail: true peer ni: - nid: 192.168.202.142@tcp state: NA - nid: 192.168.202.142@tcp1 state: NA SERVICENODE DEBUG: legacy expected full view='192.168.202.142@tcp,192.168.202.142@tcp1' SERVICENODE DEBUG: ping seed=192.168.202.142@tcp present='192.168.202.142@tcp,192.168.202.142@tcp1' absent='' SERVICENODE DEBUG: ping rc=0 ping: - primary nid: 192.168.202.142@tcp Multi-Rail: false peer_ni: - nid: 192.168.202.142@tcp - nid: 192.168.202.142@tcp1 SERVICENODE DEBUG: discover seed=192.168.202.142@tcp present='192.168.202.142@tcp,192.168.202.142@tcp1' absent='' SERVICENODE DEBUG: discover rc=0 discover: - primary nid: 192.168.202.142@tcp Multi-Rail: true peer_ni: - nid: 192.168.202.142@tcp - nid: 192.168.202.142@tcp1 SERVICENODE DEBUG: peer show rc=0 peer: - primary nid: 192.168.202.142@tcp Multi-Rail: true peer ni: - nid: 192.168.202.142@tcp state: NA - nid: 192.168.202.142@tcp1 state: NA SERVICENODE DEBUG: legacy-policy test complete PASS 57c (163s) == conf-sanity test 57d: servicenode policy splits OSS discovery views ========================================================== 17:42:24 (1788903744) umount lustre on /mnt/lustre..... stop ost1 service on oleg242-server stop mds service on oleg242-server LNET unconfigure error 22: Invalid argument unloading modules via unload_modules_local on: 'oleg242-server' oleg242-server: oleg242-server.virtnet: executing unload_modules_local oleg242-server: oleg242-server: LNET unconfigure error 22: Invalid argument modules unloaded. setmodopts: MODOPTS_PTLRPC=lbug_on_grant_miscount=1 setmodopts: MODOPTS_PTLRPC=lbug_on_grant_miscount=1 target_nid_policy=servicenode SERVICENODE DEBUG: setup base_policy=none servicenode_policy=enabled SERVICENODE DEBUG: before load MODOPTS_PTLRPC='lbug_on_grant_miscount=1 target_nid_policy=servicenode' SERVICENODE DEBUG: before load MODOPTS_LNET='' /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1 target_nid_policy=servicenode' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg242-server' oleg242-server: oleg242-server.virtnet: executing load_modules_local oleg242-server: Loading modules from /home/green/git/lustre-release/lustre oleg242-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg242-server: detected 4 online CPUs by sysfs oleg242-server: MODOPTS_LIBCFS= oleg242-server: Force libcfs to create 2 CPU partitions oleg242-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1 target_nid_policy=servicenode' oleg242-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg242-server: mdt/mdt options: 'mdt_enable_flr_ec=1' SERVICENODE DEBUG: modules loaded oleg242-server: oleg242-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre oleg242-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg242-client: oleg242-server: ssh exited with exit code 19 oleg242-server: oleg242-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre oleg242-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg242-client: oleg242-server: ssh exited with exit code 19 oleg242-server: oleg242-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre oleg242-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg242-client: oleg242-server: ssh exited with exit code 19 tunefs.lustre failed, reformatting instead Stopping clients: oleg242-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg242-client.virtnet /mnt/lustre2 (opts:-f) oleg242-server: oleg242-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg242-server' oleg242-server: oleg242-server.virtnet: executing load_modules_local oleg242-server: Loading modules from /home/green/git/lustre-release/lustre oleg242-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg242-server: detected 4 online CPUs by sysfs oleg242-server: MODOPTS_LIBCFS= oleg242-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg242-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg242-server: oleg242-server.virtnet: executing set_default_debug -1 all pdsh@oleg242-client: oleg242-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg242-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg242-server: oleg242-server.virtnet: executing set_default_debug -1 all pdsh@oleg242-client: oleg242-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg242-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg242-server: oleg242-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg242-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg242-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg242-server stop mds service on oleg242-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg242-server SERVICENODE DEBUG: writeconf_or_reformat complete oleg242-server: oleg242-server.virtnet: executing lnet_if_list SERVICENODE DEBUG: oss_if='ens2' client_if='ens2' oleg242-server: sh: /usr/sbin/ksocklnd-config: No such file or directory default via 192.168.202.254 dev ens2 192.168.202.0/24 dev ens2 proto kernel scope link src 192.168.202.42 default via 192.168.202.254 dev ens2 192.168.202.0/24 dev ens2 proto kernel scope link src 192.168.202.42 SERVICENODE DEBUG: net=tcp nid='192.168.202.142@tcp' SERVICENODE DEBUG: net=tcp1 nid='192.168.202.142@tcp1' SERVICENODE DEBUG: before temporary target format SERVICENODE DEBUG: HOSTNAME=oleg242-client.virtnet OSS=oleg242-server SERVICENODE DEBUG: MODOPTS_PTLRPC='lbug_on_grant_miscount=1 target_nid_policy=servicenode' SERVICENODE DEBUG: MODOPTS_LNET='' SERVICENODE DEBUG: nets='tcp,tcp1' SERVICENODE DEBUG: nids='192.168.202.142@tcp,192.168.202.142@tcp1' SERVICENODE DEBUG: client LNet configuration net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: tcp local NI(s): - nid: 192.168.202.42@tcp status: up interfaces: 0: ens2 - net type: tcp1 local NI(s): - nid: 192.168.202.42@tcp1 status: up interfaces: 0: ens2 SERVICENODE DEBUG: OSS LNet configuration net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: tcp local NI(s): - nid: 192.168.202.142@tcp status: up interfaces: 0: ens2 - net type: tcp1 local NI(s): - nid: 192.168.202.142@tcp1 status: up interfaces: 0: ens2 peer: - primary nid: 192.168.202.142@tcp Multi-Rail: true peer ni: - nid: 192.168.202.142@tcp state: NA SERVICENODE DEBUG: temporary fsname='sv5871' SERVICENODE DEBUG: temporary OST1 device='lustre-ost1_2/ost1_2' Permanent disk data: Device: lustre-ost1_2/ost1_2 Target: sv5871:OST0000 Index: 0 Lustre FS: sv5871 Mount type: zfs Flags: 0x1062 (OST first_time update no_primnode ) Persistent mount opts: Parameters: mgsnode=192.168.202.142@tcp sys.timeout=20 failover.node=192.168.202.142@tcp autodegrade=on mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1_2/ost1_2 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1_2/ost1_2 properties lustre:mgsnode=192.168.202.142@tcp lustre:sys.timeout=20 lustre:failover.node=192.168.202.142@tcp lustre:autodegrade=on lustre:version=1 lustre:flags=4194 lustre:index=0 lustre:fsname=sv5871 lustre:svname=sv5871:OST0000 SERVICENODE DEBUG: temporary OST2 device='lustre-ost2_2/ost2_2' Permanent disk data: Device: lustre-ost2_2/ost2_2 Target: sv5871:OST0001 Index: 1 Lustre FS: sv5871 Mount type: zfs Flags: 0x1062 (OST first_time update no_primnode ) Persistent mount opts: Parameters: mgsnode=192.168.202.142@tcp sys.timeout=20 failover.node=192.168.202.142@tcp1 autodegrade=on mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost2_2/ost2_2 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost2_2/ost2_2 properties lustre:mgsnode=192.168.202.142@tcp lustre:sys.timeout=20 lustre:failover.node=192.168.202.142@tcp1 lustre:autodegrade=on lustre:version=1 lustre:flags=4194 lustre:index=1 lustre:fsname=sv5871 lustre:svname=sv5871:OST0001 SERVICENODE DEBUG: starting MGS/MDT and temporary service-node targets start mds service on oleg242-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg242-server: oleg242-server.virtnet: executing set_default_debug -1 all pdsh@oleg242-client: oleg242-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 Start fs2ost1: mount -t lustre -o localrecov lustre-ost1_2/ost1_2 /mnt/lustre-fs2ost1 seq.cli-sv5871-OST0000-super.width=65536 oleg242-server: oleg242-server.virtnet: executing set_default_debug -1 all pdsh@oleg242-client: oleg242-server: ssh exited with exit code 1 Commit the device label on lustre-ost1_2/ost1_2 Started sv5871-OST0000 oleg242-server: missing pool name argument oleg242-server: usage: oleg242-server: trim [-dw] [-r ] [-c | -s] [ ...] pdsh@oleg242-client: oleg242-server: ssh exited with exit code 2 Start fs3ost2: mount -t lustre -o localrecov lustre-ost2_2/ost2_2 /mnt/lustre-fs3ost2 seq.cli-sv5871-OST0001-super.width=65536 oleg242-server: oleg242-server.virtnet: executing set_default_debug -1 all pdsh@oleg242-client: oleg242-server: ssh exited with exit code 1 Commit the device label on lustre-ost2_2/ost2_2 Started sv5871-OST0001 oleg242-server: missing pool name argument oleg242-server: usage: oleg242-server: trim [-dw] [-r ] [-c | -s] [ ...] pdsh@oleg242-client: oleg242-server: ssh exited with exit code 2 SERVICENODE DEBUG: after target startup SERVICENODE DEBUG: HOSTNAME=oleg242-client.virtnet OSS=oleg242-server SERVICENODE DEBUG: MODOPTS_PTLRPC='lbug_on_grant_miscount=1 target_nid_policy=servicenode' SERVICENODE DEBUG: MODOPTS_LNET='' SERVICENODE DEBUG: nets='tcp,tcp1' SERVICENODE DEBUG: nids='192.168.202.142@tcp,192.168.202.142@tcp1' SERVICENODE DEBUG: client LNet configuration net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: tcp local NI(s): - nid: 192.168.202.42@tcp status: up interfaces: 0: ens2 - net type: tcp1 local NI(s): - nid: 192.168.202.42@tcp1 status: up interfaces: 0: ens2 SERVICENODE DEBUG: OSS LNet configuration net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: tcp local NI(s): - nid: 192.168.202.142@tcp status: up interfaces: 0: ens2 - net type: tcp1 local NI(s): - nid: 192.168.202.142@tcp1 status: up interfaces: 0: ens2 peer: - primary nid: 192.168.202.142@tcp Multi-Rail: true peer ni: - nid: 192.168.202.142@tcp state: NA SERVICENODE DEBUG: ping seed=192.168.202.142@tcp present='192.168.202.142@tcp' absent='192.168.202.142@tcp1' SERVICENODE DEBUG: ping rc=0 ping: - primary nid: 192.168.202.142@tcp Multi-Rail: false peer_ni: - nid: 192.168.202.142@tcp SERVICENODE DEBUG: ping seed=192.168.202.142@tcp1 present='192.168.202.142@tcp1' absent='192.168.202.142@tcp' SERVICENODE DEBUG: ping rc=0 ping: - primary nid: 192.168.202.142@tcp1 Multi-Rail: false peer_ni: - nid: 192.168.202.142@tcp1 SERVICENODE DEBUG: discover seed=192.168.202.142@tcp present='192.168.202.142@tcp' absent='192.168.202.142@tcp1' SERVICENODE DEBUG: discover rc=0 discover: - primary nid: 192.168.202.142@tcp Multi-Rail: true peer_ni: - nid: 192.168.202.142@tcp SERVICENODE DEBUG: peer show rc=0 peer: - primary nid: 192.168.202.142@tcp Multi-Rail: true peer ni: - nid: 192.168.202.142@tcp state: NA SERVICENODE DEBUG: discover seed=192.168.202.142@tcp1 present='192.168.202.142@tcp1' absent='192.168.202.142@tcp' SERVICENODE DEBUG: discover rc=0 discover: - primary nid: 192.168.202.142@tcp1 Multi-Rail: true peer_ni: - nid: 192.168.202.142@tcp1 SERVICENODE DEBUG: peer show rc=0 peer: - primary nid: 192.168.202.142@tcp1 Multi-Rail: true peer ni: - nid: 192.168.202.142@tcp1 state: NA PASS 57d (140s) == conf-sanity test 57e: servicenode policy overrides self grouping ========================================================== 17:44:44 (1788903884) umount lustre on /mnt/lustre..... stop ost1 service on oleg242-server stop mds service on oleg242-server LNET unconfigure error 22: Invalid argument unloading modules via unload_modules_local on: 'oleg242-server' oleg242-server: oleg242-server.virtnet: executing unload_modules_local oleg242-server: oleg242-server: LNET unconfigure error 22: Invalid argument modules unloaded. setmodopts: MODOPTS_PTLRPC=lbug_on_grant_miscount=1 setmodopts: MODOPTS_PTLRPC=lbug_on_grant_miscount=1 target_nid_policy=servicenode setmodopts: MODOPTS_LNET= lnet_self_nets=tcp,tcp1 SERVICENODE DEBUG: setup base_policy=self servicenode_policy=enabled SERVICENODE DEBUG: before load MODOPTS_PTLRPC='lbug_on_grant_miscount=1 target_nid_policy=servicenode' SERVICENODE DEBUG: before load MODOPTS_LNET=' lnet_self_nets=tcp,tcp1' /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ../lnet/lnet/lnet options: 'lnet_self_nets=tcp,tcp1' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1 target_nid_policy=servicenode' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg242-server' oleg242-server: oleg242-server.virtnet: executing load_modules_local oleg242-server: Loading modules from /home/green/git/lustre-release/lustre oleg242-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg242-server: detected 4 online CPUs by sysfs oleg242-server: MODOPTS_LIBCFS= oleg242-server: Force libcfs to create 2 CPU partitions oleg242-server: ../lnet/lnet/lnet options: 'lnet_self_nets=tcp,tcp1' oleg242-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1 target_nid_policy=servicenode' oleg242-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg242-server: mdt/mdt options: 'mdt_enable_flr_ec=1' SERVICENODE DEBUG: modules loaded oleg242-server: oleg242-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre oleg242-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg242-client: oleg242-server: ssh exited with exit code 19 oleg242-server: oleg242-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre oleg242-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg242-client: oleg242-server: ssh exited with exit code 19 oleg242-server: oleg242-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre oleg242-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg242-client: oleg242-server: ssh exited with exit code 19 tunefs.lustre failed, reformatting instead Stopping clients: oleg242-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg242-client.virtnet /mnt/lustre2 (opts:-f) oleg242-server: oleg242-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg242-server' oleg242-server: oleg242-server.virtnet: executing load_modules_local oleg242-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg242-server: Loading modules from /home/green/git/lustre-release/lustre oleg242-server: detected 4 online CPUs by sysfs oleg242-server: MODOPTS_LIBCFS= oleg242-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg242-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg242-server: oleg242-server.virtnet: executing set_default_debug -1 all pdsh@oleg242-client: oleg242-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg242-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg242-server: oleg242-server.virtnet: executing set_default_debug -1 all pdsh@oleg242-client: oleg242-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg242-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg242-server: oleg242-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg242-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg242-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg242-server stop mds service on oleg242-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg242-server SERVICENODE DEBUG: writeconf_or_reformat complete oleg242-server: oleg242-server.virtnet: executing lnet_if_list SERVICENODE DEBUG: oss_if='ens2' client_if='ens2' oleg242-server: sh: /usr/sbin/ksocklnd-config: No such file or directory default via 192.168.202.254 dev ens2 192.168.202.0/24 dev ens2 proto kernel scope link src 192.168.202.42 default via 192.168.202.254 dev ens2 192.168.202.0/24 dev ens2 proto kernel scope link src 192.168.202.42 SERVICENODE DEBUG: net=tcp nid='192.168.202.142@tcp' SERVICENODE DEBUG: net=tcp1 nid='192.168.202.142@tcp1' SERVICENODE DEBUG: before temporary target format SERVICENODE DEBUG: HOSTNAME=oleg242-client.virtnet OSS=oleg242-server SERVICENODE DEBUG: MODOPTS_PTLRPC='lbug_on_grant_miscount=1 target_nid_policy=servicenode' SERVICENODE DEBUG: MODOPTS_LNET=' lnet_self_nets=tcp,tcp1' SERVICENODE DEBUG: nets='tcp,tcp1' SERVICENODE DEBUG: nids='192.168.202.142@tcp,192.168.202.142@tcp1' SERVICENODE DEBUG: client LNet configuration net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: tcp local NI(s): - nid: 192.168.202.42@tcp status: up interfaces: 0: ens2 - net type: tcp1 local NI(s): - nid: 192.168.202.42@tcp1 status: up interfaces: 0: ens2 SERVICENODE DEBUG: OSS LNet configuration net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: tcp local NI(s): - nid: 192.168.202.142@tcp status: up interfaces: 0: ens2 - net type: tcp1 local NI(s): - nid: 192.168.202.142@tcp1 status: up interfaces: 0: ens2 peer: - primary nid: 192.168.202.142@tcp Multi-Rail: true peer ni: - nid: 192.168.202.142@tcp state: NA SERVICENODE DEBUG: temporary fsname='sv3952' SERVICENODE DEBUG: temporary OST1 device='lustre-ost1_2/ost1_2' Permanent disk data: Device: lustre-ost1_2/ost1_2 Target: sv3952:OST0000 Index: 0 Lustre FS: sv3952 Mount type: zfs Flags: 0x1062 (OST first_time update no_primnode ) Persistent mount opts: Parameters: mgsnode=192.168.202.142@tcp sys.timeout=20 failover.node=192.168.202.142@tcp,192.168.202.142@tcp1 autodegrade=on mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1_2/ost1_2 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1_2/ost1_2 properties lustre:mgsnode=192.168.202.142@tcp lustre:sys.timeout=20 lustre:failover.node=192.168.202.142@tcp,192.168.202.142@tcp1 lustre:autodegrade=on lustre:version=1 lustre:flags=4194 lustre:index=0 lustre:fsname=sv3952 lustre:svname=sv3952:OST0000 SERVICENODE DEBUG: temporary OST2 device='lustre-ost2_2/ost2_2' Permanent disk data: Device: lustre-ost2_2/ost2_2 Target: sv3952:OST0001 Index: 1 Lustre FS: sv3952 Mount type: zfs Flags: 0x1062 (OST first_time update no_primnode ) Persistent mount opts: Parameters: mgsnode=192.168.202.142@tcp sys.timeout=20 failover.node=192.168.202.142@tcp1 autodegrade=on mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost2_2/ost2_2 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost2_2/ost2_2 properties lustre:mgsnode=192.168.202.142@tcp lustre:sys.timeout=20 lustre:failover.node=192.168.202.142@tcp1 lustre:autodegrade=on lustre:version=1 lustre:flags=4194 lustre:index=1 lustre:fsname=sv3952 lustre:svname=sv3952:OST0001 SERVICENODE DEBUG: starting MGS/MDT and temporary service-node targets start mds service on oleg242-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg242-server: oleg242-server.virtnet: executing set_default_debug -1 all pdsh@oleg242-client: oleg242-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 Start fs2ost1: mount -t lustre -o localrecov lustre-ost1_2/ost1_2 /mnt/lustre-fs2ost1 seq.cli-sv3952-OST0000-super.width=65536 oleg242-server: oleg242-server.virtnet: executing set_default_debug -1 all pdsh@oleg242-client: oleg242-server: ssh exited with exit code 1 Commit the device label on lustre-ost1_2/ost1_2 Started sv3952-OST0000 oleg242-server: missing pool name argument oleg242-server: usage: oleg242-server: trim [-dw] [-r ] [-c | -s] [ ...] pdsh@oleg242-client: oleg242-server: ssh exited with exit code 2 SERVICENODE DEBUG: after first target startup SERVICENODE DEBUG: HOSTNAME=oleg242-client.virtnet OSS=oleg242-server SERVICENODE DEBUG: MODOPTS_PTLRPC='lbug_on_grant_miscount=1 target_nid_policy=servicenode' SERVICENODE DEBUG: MODOPTS_LNET=' lnet_self_nets=tcp,tcp1' SERVICENODE DEBUG: nets='tcp,tcp1' SERVICENODE DEBUG: nids='192.168.202.142@tcp,192.168.202.142@tcp1' SERVICENODE DEBUG: client LNet configuration net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: tcp local NI(s): - nid: 192.168.202.42@tcp status: up interfaces: 0: ens2 - net type: tcp1 local NI(s): - nid: 192.168.202.42@tcp1 status: up interfaces: 0: ens2 SERVICENODE DEBUG: OSS LNet configuration net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: tcp local NI(s): - nid: 192.168.202.142@tcp status: up interfaces: 0: ens2 - net type: tcp1 local NI(s): - nid: 192.168.202.142@tcp1 status: up interfaces: 0: ens2 peer: - primary nid: 192.168.202.142@tcp Multi-Rail: true peer ni: - nid: 192.168.202.142@tcp state: NA - nid: 192.168.202.142@tcp1 state: NA SERVICENODE DEBUG: ping seed=192.168.202.142@tcp present='192.168.202.142@tcp,192.168.202.142@tcp1' absent='' SERVICENODE DEBUG: ping rc=0 ping: - primary nid: 192.168.202.142@tcp Multi-Rail: false peer_ni: - nid: 192.168.202.142@tcp - nid: 192.168.202.142@tcp1 SERVICENODE DEBUG: ping seed=192.168.202.142@tcp1 present='192.168.202.142@tcp,192.168.202.142@tcp1' absent='' SERVICENODE DEBUG: ping rc=0 ping: - primary nid: 192.168.202.142@tcp1 Multi-Rail: false peer_ni: - nid: 192.168.202.142@tcp - nid: 192.168.202.142@tcp1 SERVICENODE DEBUG: discover seed=192.168.202.142@tcp present='192.168.202.142@tcp,192.168.202.142@tcp1' absent='' SERVICENODE DEBUG: discover rc=0 discover: - primary nid: 192.168.202.142@tcp Multi-Rail: true peer_ni: - nid: 192.168.202.142@tcp - nid: 192.168.202.142@tcp1 SERVICENODE DEBUG: peer show rc=0 peer: - primary nid: 192.168.202.142@tcp Multi-Rail: true peer ni: - nid: 192.168.202.142@tcp state: NA - nid: 192.168.202.142@tcp1 state: NA SERVICENODE DEBUG: discover seed=192.168.202.142@tcp1 present='192.168.202.142@tcp,192.168.202.142@tcp1' absent='' SERVICENODE DEBUG: discover rc=0 discover: - primary nid: 192.168.202.142@tcp Multi-Rail: true peer_ni: - nid: 192.168.202.142@tcp - nid: 192.168.202.142@tcp1 SERVICENODE DEBUG: peer show rc=0 peer: - primary nid: 192.168.202.142@tcp Multi-Rail: true peer ni: - nid: 192.168.202.142@tcp state: NA - nid: 192.168.202.142@tcp1 state: NA PASS 57e (116s) == conf-sanity test 57f: servicenode policy overrides NUMA grouping ========================================================== 17:46:40 (1788904000) umount lustre on /mnt/lustre..... stop ost1 service on oleg242-server stop mds service on oleg242-server LNET unconfigure error 22: Invalid argument unloading modules via unload_modules_local on: 'oleg242-server' oleg242-server: oleg242-server.virtnet: executing unload_modules_local oleg242-server: oleg242-server: LNET unconfigure error 22: Invalid argument modules unloaded. setmodopts: MODOPTS_PTLRPC=lbug_on_grant_miscount=1 setmodopts: MODOPTS_PTLRPC=lbug_on_grant_miscount=1 target_nid_policy=servicenode setmodopts: MODOPTS_LNET= lnet_group_numa_nets=tcp,tcp1 SERVICENODE DEBUG: setup base_policy=numa servicenode_policy=enabled SERVICENODE DEBUG: before load MODOPTS_PTLRPC='lbug_on_grant_miscount=1 target_nid_policy=servicenode' SERVICENODE DEBUG: before load MODOPTS_LNET=' lnet_group_numa_nets=tcp,tcp1' /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ../lnet/lnet/lnet options: 'lnet_group_numa_nets=tcp,tcp1' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1 target_nid_policy=servicenode' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg242-server' oleg242-server: oleg242-server.virtnet: executing load_modules_local oleg242-server: Loading modules from /home/green/git/lustre-release/lustre oleg242-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg242-server: detected 4 online CPUs by sysfs oleg242-server: MODOPTS_LIBCFS= oleg242-server: Force libcfs to create 2 CPU partitions oleg242-server: ../lnet/lnet/lnet options: 'lnet_group_numa_nets=tcp,tcp1' oleg242-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1 target_nid_policy=servicenode' oleg242-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg242-server: mdt/mdt options: 'mdt_enable_flr_ec=1' SERVICENODE DEBUG: modules loaded fail_loc=0xe005 fail_val=2 oleg242-server: oleg242-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre oleg242-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg242-client: oleg242-server: ssh exited with exit code 19 oleg242-server: oleg242-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre oleg242-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg242-client: oleg242-server: ssh exited with exit code 19 oleg242-server: oleg242-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre oleg242-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg242-client: oleg242-server: ssh exited with exit code 19 tunefs.lustre failed, reformatting instead Stopping clients: oleg242-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg242-client.virtnet /mnt/lustre2 (opts:-f) oleg242-server: oleg242-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg242-server' oleg242-server: oleg242-server.virtnet: executing load_modules_local oleg242-server: Loading modules from /home/green/git/lustre-release/lustre oleg242-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg242-server: detected 4 online CPUs by sysfs oleg242-server: MODOPTS_LIBCFS= oleg242-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg242-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg242-server: oleg242-server.virtnet: executing set_default_debug -1 all pdsh@oleg242-client: oleg242-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg242-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg242-server: oleg242-server.virtnet: executing set_default_debug -1 all pdsh@oleg242-client: oleg242-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg242-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg242-server: oleg242-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg242-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg242-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg242-server stop mds service on oleg242-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg242-server SERVICENODE DEBUG: writeconf_or_reformat complete oleg242-server: oleg242-server.virtnet: executing lnet_if_list SERVICENODE DEBUG: oss_if='ens2' client_if='ens2' oleg242-server: sh: /usr/sbin/ksocklnd-config: No such file or directory default via 192.168.202.254 dev ens2 192.168.202.0/24 dev ens2 proto kernel scope link src 192.168.202.42 default via 192.168.202.254 dev ens2 192.168.202.0/24 dev ens2 proto kernel scope link src 192.168.202.42 SERVICENODE DEBUG: net=tcp nid='192.168.202.142@tcp' SERVICENODE DEBUG: net=tcp1 nid='192.168.202.142@tcp1' SERVICENODE DEBUG: before temporary target format SERVICENODE DEBUG: HOSTNAME=oleg242-client.virtnet OSS=oleg242-server SERVICENODE DEBUG: MODOPTS_PTLRPC='lbug_on_grant_miscount=1 target_nid_policy=servicenode' SERVICENODE DEBUG: MODOPTS_LNET=' lnet_group_numa_nets=tcp,tcp1' SERVICENODE DEBUG: nets='tcp,tcp1' SERVICENODE DEBUG: nids='192.168.202.142@tcp,192.168.202.142@tcp1' SERVICENODE DEBUG: client LNet configuration net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: tcp local NI(s): - nid: 192.168.202.42@tcp status: up interfaces: 0: ens2 - net type: tcp1 local NI(s): - nid: 192.168.202.42@tcp1 status: up interfaces: 0: ens2 SERVICENODE DEBUG: OSS LNet configuration net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: tcp local NI(s): - nid: 192.168.202.142@tcp status: up interfaces: 0: ens2 - net type: tcp1 local NI(s): - nid: 192.168.202.142@tcp1 status: up interfaces: 0: ens2 peer: - primary nid: 192.168.202.142@tcp Multi-Rail: true peer ni: - nid: 192.168.202.142@tcp state: NA SERVICENODE DEBUG: temporary fsname='sv18114' SERVICENODE DEBUG: temporary OST1 device='lustre-ost1_2/ost1_2' Permanent disk data: Device: lustre-ost1_2/ost1_2 Target: sv18114:OST0000 Index: 0 Lustre FS: sv18114 Mount type: zfs Flags: 0x1062 (OST first_time update no_primnode ) Persistent mount opts: Parameters: mgsnode=192.168.202.142@tcp sys.timeout=20 failover.node=192.168.202.142@tcp,192.168.202.142@tcp1 autodegrade=on mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1_2/ost1_2 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1_2/ost1_2 properties lustre:mgsnode=192.168.202.142@tcp lustre:sys.timeout=20 lustre:failover.node=192.168.202.142@tcp,192.168.202.142@tcp1 lustre:autodegrade=on lustre:version=1 lustre:flags=4194 lustre:index=0 lustre:fsname=sv18114 lustre:svname=sv18114:OST0000 SERVICENODE DEBUG: temporary OST2 device='lustre-ost2_2/ost2_2' Permanent disk data: Device: lustre-ost2_2/ost2_2 Target: sv18114:OST0001 Index: 1 Lustre FS: sv18114 Mount type: zfs Flags: 0x1062 (OST first_time update no_primnode ) Persistent mount opts: Parameters: mgsnode=192.168.202.142@tcp sys.timeout=20 failover.node=192.168.202.142@tcp1 autodegrade=on mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost2_2/ost2_2 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost2_2/ost2_2 properties lustre:mgsnode=192.168.202.142@tcp lustre:sys.timeout=20 lustre:failover.node=192.168.202.142@tcp1 lustre:autodegrade=on lustre:version=1 lustre:flags=4194 lustre:index=1 lustre:fsname=sv18114 lustre:svname=sv18114:OST0001 SERVICENODE DEBUG: starting MGS/MDT and temporary service-node targets start mds service on oleg242-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg242-server: oleg242-server.virtnet: executing set_default_debug -1 all pdsh@oleg242-client: oleg242-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 Start fs2ost1: mount -t lustre -o localrecov lustre-ost1_2/ost1_2 /mnt/lustre-fs2ost1 seq.cli-sv18114-OST0000-super.width=65536 oleg242-server: oleg242-server.virtnet: executing set_default_debug -1 all pdsh@oleg242-client: oleg242-server: ssh exited with exit code 1 Commit the device label on lustre-ost1_2/ost1_2 Started sv18114-OST0000 oleg242-server: missing pool name argument oleg242-server: usage: oleg242-server: trim [-dw] [-r ] [-c | -s] [ ...] pdsh@oleg242-client: oleg242-server: ssh exited with exit code 2 SERVICENODE DEBUG: after first target startup SERVICENODE DEBUG: HOSTNAME=oleg242-client.virtnet OSS=oleg242-server SERVICENODE DEBUG: MODOPTS_PTLRPC='lbug_on_grant_miscount=1 target_nid_policy=servicenode' SERVICENODE DEBUG: MODOPTS_LNET=' lnet_group_numa_nets=tcp,tcp1' SERVICENODE DEBUG: nets='tcp,tcp1' SERVICENODE DEBUG: nids='192.168.202.142@tcp,192.168.202.142@tcp1' SERVICENODE DEBUG: client LNet configuration net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: tcp local NI(s): - nid: 192.168.202.42@tcp status: up interfaces: 0: ens2 - net type: tcp1 local NI(s): - nid: 192.168.202.42@tcp1 status: up interfaces: 0: ens2 SERVICENODE DEBUG: OSS LNet configuration net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: tcp local NI(s): - nid: 192.168.202.142@tcp status: up interfaces: 0: ens2 - net type: tcp1 local NI(s): - nid: 192.168.202.142@tcp1 status: up interfaces: 0: ens2 peer: - primary nid: 192.168.202.142@tcp Multi-Rail: true peer ni: - nid: 192.168.202.142@tcp state: NA - nid: 192.168.202.142@tcp1 state: NA SERVICENODE DEBUG: ping seed=192.168.202.142@tcp present='192.168.202.142@tcp,192.168.202.142@tcp1' absent='' SERVICENODE DEBUG: ping rc=0 ping: - primary nid: 192.168.202.142@tcp Multi-Rail: false peer_ni: - nid: 192.168.202.142@tcp - nid: 192.168.202.142@tcp1 SERVICENODE DEBUG: ping seed=192.168.202.142@tcp1 present='192.168.202.142@tcp,192.168.202.142@tcp1' absent='' SERVICENODE DEBUG: ping rc=0 ping: - primary nid: 192.168.202.142@tcp1 Multi-Rail: false peer_ni: - nid: 192.168.202.142@tcp - nid: 192.168.202.142@tcp1 SERVICENODE DEBUG: discover seed=192.168.202.142@tcp present='192.168.202.142@tcp,192.168.202.142@tcp1' absent='' SERVICENODE DEBUG: discover rc=0 discover: - primary nid: 192.168.202.142@tcp Multi-Rail: true peer_ni: - nid: 192.168.202.142@tcp - nid: 192.168.202.142@tcp1 SERVICENODE DEBUG: peer show rc=0 peer: - primary nid: 192.168.202.142@tcp Multi-Rail: true peer ni: - nid: 192.168.202.142@tcp state: NA - nid: 192.168.202.142@tcp1 state: NA SERVICENODE DEBUG: discover seed=192.168.202.142@tcp1 present='192.168.202.142@tcp,192.168.202.142@tcp1' absent='' SERVICENODE DEBUG: discover rc=0 discover: - primary nid: 192.168.202.142@tcp Multi-Rail: true peer_ni: - nid: 192.168.202.142@tcp - nid: 192.168.202.142@tcp1 SERVICENODE DEBUG: peer show rc=0 peer: - primary nid: 192.168.202.142@tcp Multi-Rail: true peer ni: - nid: 192.168.202.142@tcp state: NA - nid: 192.168.202.142@tcp1 state: NA PASS 57f (116s) == conf-sanity test 57g: servicenode policy keeps ungrouped NIDs discoverable ========================================================== 17:48:36 (1788904116) umount lustre on /mnt/lustre..... stop ost1 service on oleg242-server stop mds service on oleg242-server LNET unconfigure error 22: Invalid argument unloading modules via unload_modules_local on: 'oleg242-server' oleg242-server: oleg242-server.virtnet: executing unload_modules_local oleg242-server: oleg242-server: LNET unconfigure error 22: Invalid argument modules unloaded. setmodopts: MODOPTS_PTLRPC=lbug_on_grant_miscount=1 setmodopts: MODOPTS_PTLRPC=lbug_on_grant_miscount=1 target_nid_policy=servicenode SERVICENODE DEBUG: setup base_policy=none servicenode_policy=enabled SERVICENODE DEBUG: before load MODOPTS_PTLRPC='lbug_on_grant_miscount=1 target_nid_policy=servicenode' SERVICENODE DEBUG: before load MODOPTS_LNET='' /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1 target_nid_policy=servicenode' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg242-server' oleg242-server: oleg242-server.virtnet: executing load_modules_local oleg242-server: Loading modules from /home/green/git/lustre-release/lustre oleg242-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg242-server: detected 4 online CPUs by sysfs oleg242-server: MODOPTS_LIBCFS= oleg242-server: Force libcfs to create 2 CPU partitions oleg242-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1 target_nid_policy=servicenode' oleg242-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg242-server: mdt/mdt options: 'mdt_enable_flr_ec=1' SERVICENODE DEBUG: modules loaded oleg242-server: oleg242-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre oleg242-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg242-client: oleg242-server: ssh exited with exit code 19 oleg242-server: oleg242-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre oleg242-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg242-client: oleg242-server: ssh exited with exit code 19 oleg242-server: oleg242-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre oleg242-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg242-client: oleg242-server: ssh exited with exit code 19 tunefs.lustre failed, reformatting instead Stopping clients: oleg242-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg242-client.virtnet /mnt/lustre2 (opts:-f) oleg242-server: oleg242-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg242-server' oleg242-server: oleg242-server.virtnet: executing load_modules_local oleg242-server: Loading modules from /home/green/git/lustre-release/lustre oleg242-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg242-server: detected 4 online CPUs by sysfs oleg242-server: MODOPTS_LIBCFS= oleg242-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg242-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg242-server: oleg242-server.virtnet: executing set_default_debug -1 all pdsh@oleg242-client: oleg242-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg242-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg242-server: oleg242-server.virtnet: executing set_default_debug -1 all pdsh@oleg242-client: oleg242-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg242-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg242-server: oleg242-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg242-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg242-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg242-server stop mds service on oleg242-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg242-server SERVICENODE DEBUG: writeconf_or_reformat complete oleg242-server: oleg242-server.virtnet: executing lnet_if_list SERVICENODE DEBUG: oss_if='ens2' client_if='ens2' oleg242-server: sh: /usr/sbin/ksocklnd-config: No such file or directory default via 192.168.202.254 dev ens2 192.168.202.0/24 dev ens2 proto kernel scope link src 192.168.202.42 default via 192.168.202.254 dev ens2 192.168.202.0/24 dev ens2 proto kernel scope link src 192.168.202.42 SERVICENODE DEBUG: net=tcp nid='192.168.202.142@tcp' SERVICENODE DEBUG: net=tcp1 nid='192.168.202.142@tcp1' SERVICENODE DEBUG: before temporary target format SERVICENODE DEBUG: HOSTNAME=oleg242-client.virtnet OSS=oleg242-server SERVICENODE DEBUG: MODOPTS_PTLRPC='lbug_on_grant_miscount=1 target_nid_policy=servicenode' SERVICENODE DEBUG: MODOPTS_LNET='' SERVICENODE DEBUG: nets='tcp,tcp1' SERVICENODE DEBUG: nids='192.168.202.142@tcp,192.168.202.142@tcp1' SERVICENODE DEBUG: client LNet configuration net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: tcp local NI(s): - nid: 192.168.202.42@tcp status: up interfaces: 0: ens2 - net type: tcp1 local NI(s): - nid: 192.168.202.42@tcp1 status: up interfaces: 0: ens2 SERVICENODE DEBUG: OSS LNet configuration net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: tcp local NI(s): - nid: 192.168.202.142@tcp status: up interfaces: 0: ens2 - net type: tcp1 local NI(s): - nid: 192.168.202.142@tcp1 status: up interfaces: 0: ens2 peer: - primary nid: 192.168.202.142@tcp Multi-Rail: true peer ni: - nid: 192.168.202.142@tcp state: NA SERVICENODE DEBUG: temporary fsname='sv23378' SERVICENODE DEBUG: temporary OST1 device='lustre-ost1_2/ost1_2' Permanent disk data: Device: lustre-ost1_2/ost1_2 Target: sv23378:OST0000 Index: 0 Lustre FS: sv23378 Mount type: zfs Flags: 0x1062 (OST first_time update no_primnode ) Persistent mount opts: Parameters: mgsnode=192.168.202.142@tcp sys.timeout=20 failover.node=192.168.202.142@tcp autodegrade=on mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1_2/ost1_2 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1_2/ost1_2 properties lustre:mgsnode=192.168.202.142@tcp lustre:sys.timeout=20 lustre:failover.node=192.168.202.142@tcp lustre:autodegrade=on lustre:version=1 lustre:flags=4194 lustre:index=0 lustre:fsname=sv23378 lustre:svname=sv23378:OST0000 SERVICENODE DEBUG: temporary OST2 device='lustre-ost2_2/ost2_2' Permanent disk data: Device: lustre-ost2_2/ost2_2 Target: sv23378:OST0001 Index: 1 Lustre FS: sv23378 Mount type: zfs Flags: 0x1062 (OST first_time update no_primnode ) Persistent mount opts: Parameters: mgsnode=192.168.202.142@tcp sys.timeout=20 failover.node=192.168.202.142@tcp1 autodegrade=on mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost2_2/ost2_2 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost2_2/ost2_2 properties lustre:mgsnode=192.168.202.142@tcp lustre:sys.timeout=20 lustre:failover.node=192.168.202.142@tcp1 lustre:autodegrade=on lustre:version=1 lustre:flags=4194 lustre:index=1 lustre:fsname=sv23378 lustre:svname=sv23378:OST0001 SERVICENODE DEBUG: starting MGS/MDT and temporary service-node targets start mds service on oleg242-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg242-server: oleg242-server.virtnet: executing set_default_debug -1 all pdsh@oleg242-client: oleg242-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 Start fs2ost1: mount -t lustre -o localrecov lustre-ost1_2/ost1_2 /mnt/lustre-fs2ost1 seq.cli-sv23378-OST0000-super.width=65536 oleg242-server: oleg242-server.virtnet: executing set_default_debug -1 all pdsh@oleg242-client: oleg242-server: ssh exited with exit code 1 Commit the device label on lustre-ost1_2/ost1_2 Started sv23378-OST0000 oleg242-server: missing pool name argument oleg242-server: usage: oleg242-server: trim [-dw] [-r ] [-c | -s] [ ...] pdsh@oleg242-client: oleg242-server: ssh exited with exit code 2 SERVICENODE DEBUG: after first target startup SERVICENODE DEBUG: HOSTNAME=oleg242-client.virtnet OSS=oleg242-server SERVICENODE DEBUG: MODOPTS_PTLRPC='lbug_on_grant_miscount=1 target_nid_policy=servicenode' SERVICENODE DEBUG: MODOPTS_LNET='' SERVICENODE DEBUG: nets='tcp,tcp1' SERVICENODE DEBUG: nids='192.168.202.142@tcp,192.168.202.142@tcp1' SERVICENODE DEBUG: client LNet configuration net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: tcp local NI(s): - nid: 192.168.202.42@tcp status: up interfaces: 0: ens2 - net type: tcp1 local NI(s): - nid: 192.168.202.42@tcp1 status: up interfaces: 0: ens2 SERVICENODE DEBUG: OSS LNet configuration net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: tcp local NI(s): - nid: 192.168.202.142@tcp status: up interfaces: 0: ens2 - net type: tcp1 local NI(s): - nid: 192.168.202.142@tcp1 status: up interfaces: 0: ens2 peer: - primary nid: 192.168.202.142@tcp Multi-Rail: true peer ni: - nid: 192.168.202.142@tcp state: NA SERVICENODE DEBUG: ping seed=192.168.202.142@tcp1 present='192.168.202.142@tcp1' absent='192.168.202.142@tcp' SERVICENODE DEBUG: ping rc=0 ping: - primary nid: 192.168.202.142@tcp1 Multi-Rail: false peer_ni: - nid: 192.168.202.142@tcp1 SERVICENODE DEBUG: discover seed=192.168.202.142@tcp1 present='192.168.202.142@tcp1' absent='192.168.202.142@tcp' SERVICENODE DEBUG: discover rc=0 discover: - primary nid: 192.168.202.142@tcp1 Multi-Rail: true peer_ni: - nid: 192.168.202.142@tcp1 SERVICENODE DEBUG: peer show rc=0 peer: - primary nid: 192.168.202.142@tcp1 Multi-Rail: true peer ni: - nid: 192.168.202.142@tcp1 state: NA SERVICENODE DEBUG: ping seed=192.168.202.142@tcp present='192.168.202.142@tcp' absent='192.168.202.142@tcp1' SERVICENODE DEBUG: ping rc=0 ping: - primary nid: 192.168.202.142@tcp Multi-Rail: false peer_ni: - nid: 192.168.202.142@tcp SERVICENODE DEBUG: starting temporary OST2 Start fs3ost2: mount -t lustre -o localrecov lustre-ost2_2/ost2_2 /mnt/lustre-fs3ost2 seq.cli-sv23378-OST0001-super.width=65536 oleg242-server: oleg242-server.virtnet: executing set_default_debug -1 all pdsh@oleg242-client: oleg242-server: ssh exited with exit code 1 Commit the device label on lustre-ost2_2/ost2_2 Started sv23378-OST0001 oleg242-server: missing pool name argument oleg242-server: usage: oleg242-server: trim [-dw] [-r ] [-c | -s] [ ...] pdsh@oleg242-client: oleg242-server: ssh exited with exit code 2 SERVICENODE DEBUG: after second target startup SERVICENODE DEBUG: HOSTNAME=oleg242-client.virtnet OSS=oleg242-server SERVICENODE DEBUG: MODOPTS_PTLRPC='lbug_on_grant_miscount=1 target_nid_policy=servicenode' SERVICENODE DEBUG: MODOPTS_LNET='' SERVICENODE DEBUG: nets='tcp,tcp1' SERVICENODE DEBUG: nids='192.168.202.142@tcp,192.168.202.142@tcp1' SERVICENODE DEBUG: client LNet configuration net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: tcp local NI(s): - nid: 192.168.202.42@tcp status: up interfaces: 0: ens2 - net type: tcp1 local NI(s): - nid: 192.168.202.42@tcp1 status: up interfaces: 0: ens2 peer: - primary nid: 192.168.202.142@tcp Multi-Rail: false peer ni: - nid: 192.168.202.142@tcp state: NA - primary nid: 192.168.202.142@tcp1 Multi-Rail: true peer ni: - nid: 192.168.202.142@tcp1 state: NA SERVICENODE DEBUG: OSS LNet configuration net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: tcp local NI(s): - nid: 192.168.202.142@tcp status: up interfaces: 0: ens2 - net type: tcp1 local NI(s): - nid: 192.168.202.142@tcp1 status: up interfaces: 0: ens2 peer: - primary nid: 192.168.202.142@tcp Multi-Rail: true peer ni: - nid: 192.168.202.142@tcp state: NA peer: - primary nid: 192.168.202.42@tcp1 Multi-Rail: true peer ni: - nid: 192.168.202.42@tcp1 state: NA peer: - primary nid: 192.168.202.42@tcp Multi-Rail: false peer ni: - nid: 192.168.202.42@tcp state: NA SERVICENODE DEBUG: ping seed=192.168.202.142@tcp present='192.168.202.142@tcp' absent='192.168.202.142@tcp1' SERVICENODE DEBUG: ping rc=0 ping: - primary nid: 192.168.202.142@tcp Multi-Rail: false peer_ni: - nid: 192.168.202.142@tcp SERVICENODE DEBUG: ping seed=192.168.202.142@tcp1 present='192.168.202.142@tcp1' absent='192.168.202.142@tcp' SERVICENODE DEBUG: ping rc=0 ping: - primary nid: 192.168.202.142@tcp1 Multi-Rail: true peer_ni: - nid: 192.168.202.142@tcp1 SERVICENODE DEBUG: discover seed=192.168.202.142@tcp present='192.168.202.142@tcp' absent='192.168.202.142@tcp1' SERVICENODE DEBUG: discover rc=0 discover: - primary nid: 192.168.202.142@tcp Multi-Rail: true peer_ni: - nid: 192.168.202.142@tcp SERVICENODE DEBUG: peer show rc=0 peer: - primary nid: 192.168.202.142@tcp Multi-Rail: true peer ni: - nid: 192.168.202.142@tcp state: NA SERVICENODE DEBUG: discover seed=192.168.202.142@tcp1 present='192.168.202.142@tcp1' absent='192.168.202.142@tcp' SERVICENODE DEBUG: discover rc=0 discover: - primary nid: 192.168.202.142@tcp1 Multi-Rail: true peer_ni: - nid: 192.168.202.142@tcp1 SERVICENODE DEBUG: peer show rc=0 peer: - primary nid: 192.168.202.142@tcp1 Multi-Rail: true peer ni: - nid: 192.168.202.142@tcp1 state: NA PASS 57g (240s) == conf-sanity test 57h: servicenode policy is active before the first target mounts ========================================================== 17:52:36 (1788904356) umount lustre on /mnt/lustre..... stop ost1 service on oleg242-server stop mds service on oleg242-server LNET unconfigure error 22: Invalid argument unloading modules via unload_modules_local on: 'oleg242-server' oleg242-server: oleg242-server.virtnet: executing unload_modules_local oleg242-server: oleg242-server: LNET unconfigure error 22: Invalid argument modules unloaded. setmodopts: MODOPTS_PTLRPC=lbug_on_grant_miscount=1 setmodopts: MODOPTS_PTLRPC=lbug_on_grant_miscount=1 target_nid_policy=servicenode SERVICENODE DEBUG: setup base_policy=none servicenode_policy=enabled SERVICENODE DEBUG: before load MODOPTS_PTLRPC='lbug_on_grant_miscount=1 target_nid_policy=servicenode' SERVICENODE DEBUG: before load MODOPTS_LNET='' /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1 target_nid_policy=servicenode' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg242-server' oleg242-server: oleg242-server.virtnet: executing load_modules_local oleg242-server: Loading modules from /home/green/git/lustre-release/lustre oleg242-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg242-server: detected 4 online CPUs by sysfs oleg242-server: MODOPTS_LIBCFS= oleg242-server: Force libcfs to create 2 CPU partitions oleg242-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1 target_nid_policy=servicenode' oleg242-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg242-server: mdt/mdt options: 'mdt_enable_flr_ec=1' SERVICENODE DEBUG: modules loaded oleg242-server: oleg242-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre oleg242-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg242-client: oleg242-server: ssh exited with exit code 19 oleg242-server: oleg242-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre oleg242-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg242-client: oleg242-server: ssh exited with exit code 19 oleg242-server: oleg242-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre oleg242-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg242-client: oleg242-server: ssh exited with exit code 19 tunefs.lustre failed, reformatting instead Stopping clients: oleg242-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg242-client.virtnet /mnt/lustre2 (opts:-f) oleg242-server: oleg242-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg242-server' oleg242-server: oleg242-server.virtnet: executing load_modules_local oleg242-server: Loading modules from /home/green/git/lustre-release/lustre oleg242-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg242-server: detected 4 online CPUs by sysfs oleg242-server: MODOPTS_LIBCFS= oleg242-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg242-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg242-server: oleg242-server.virtnet: executing set_default_debug -1 all pdsh@oleg242-client: oleg242-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg242-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg242-server: oleg242-server.virtnet: executing set_default_debug -1 all pdsh@oleg242-client: oleg242-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg242-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg242-server: oleg242-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg242-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg242-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg242-server stop mds service on oleg242-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg242-server SERVICENODE DEBUG: writeconf_or_reformat complete oleg242-server: oleg242-server.virtnet: executing lnet_if_list SERVICENODE DEBUG: oss_if='ens2' client_if='ens2' oleg242-server: sh: /usr/sbin/ksocklnd-config: No such file or directory default via 192.168.202.254 dev ens2 192.168.202.0/24 dev ens2 proto kernel scope link src 192.168.202.42 default via 192.168.202.254 dev ens2 192.168.202.0/24 dev ens2 proto kernel scope link src 192.168.202.42 SERVICENODE DEBUG: net=tcp nid='192.168.202.142@tcp' SERVICENODE DEBUG: net=tcp1 nid='192.168.202.142@tcp1' SERVICENODE DEBUG: before temporary target format SERVICENODE DEBUG: HOSTNAME=oleg242-client.virtnet OSS=oleg242-server SERVICENODE DEBUG: MODOPTS_PTLRPC='lbug_on_grant_miscount=1 target_nid_policy=servicenode' SERVICENODE DEBUG: MODOPTS_LNET='' SERVICENODE DEBUG: nets='tcp,tcp1' SERVICENODE DEBUG: nids='192.168.202.142@tcp,192.168.202.142@tcp1' SERVICENODE DEBUG: client LNet configuration net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: tcp local NI(s): - nid: 192.168.202.42@tcp status: up interfaces: 0: ens2 - net type: tcp1 local NI(s): - nid: 192.168.202.42@tcp1 status: up interfaces: 0: ens2 SERVICENODE DEBUG: OSS LNet configuration net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: tcp local NI(s): - nid: 192.168.202.142@tcp status: up interfaces: 0: ens2 - net type: tcp1 local NI(s): - nid: 192.168.202.142@tcp1 status: up interfaces: 0: ens2 peer: - primary nid: 192.168.202.142@tcp Multi-Rail: true peer ni: - nid: 192.168.202.142@tcp state: NA SERVICENODE DEBUG: stopping setup after LNet config SERVICENODE DEBUG: ping seed=192.168.202.142@tcp present='192.168.202.142@tcp' absent='192.168.202.142@tcp1' SERVICENODE DEBUG: ping rc=0 ping: - primary nid: 192.168.202.142@tcp Multi-Rail: false peer_ni: - nid: 192.168.202.142@tcp SERVICENODE DEBUG: ping seed=192.168.202.142@tcp1 present='192.168.202.142@tcp1' absent='192.168.202.142@tcp' SERVICENODE DEBUG: ping rc=0 ping: - primary nid: 192.168.202.142@tcp1 Multi-Rail: false peer_ni: - nid: 192.168.202.142@tcp1 SERVICENODE DEBUG: discover seed=192.168.202.142@tcp present='192.168.202.142@tcp' absent='192.168.202.142@tcp1' SERVICENODE DEBUG: discover rc=0 discover: - primary nid: 192.168.202.142@tcp Multi-Rail: true peer_ni: - nid: 192.168.202.142@tcp SERVICENODE DEBUG: peer show rc=0 peer: - primary nid: 192.168.202.142@tcp Multi-Rail: true peer ni: - nid: 192.168.202.142@tcp state: NA SERVICENODE DEBUG: discover seed=192.168.202.142@tcp1 present='192.168.202.142@tcp1' absent='192.168.202.142@tcp' SERVICENODE DEBUG: discover rc=0 discover: - primary nid: 192.168.202.142@tcp1 Multi-Rail: true peer_ni: - nid: 192.168.202.142@tcp1 SERVICENODE DEBUG: peer show rc=0 peer: - primary nid: 192.168.202.142@tcp1 Multi-Rail: true peer ni: - nid: 192.168.202.142@tcp1 state: NA PASS 57h (238s) == conf-sanity test 57i: servicenode Push never falls back to the merged view ========================================================== 17:56:35 (1788904595) umount lustre on /mnt/lustre..... stop ost1 service on oleg242-server stop mds service on oleg242-server LNET unconfigure error 22: Invalid argument unloading modules via unload_modules_local on: 'oleg242-server' oleg242-server: oleg242-server.virtnet: executing unload_modules_local oleg242-server: oleg242-server: LNET unconfigure error 22: Invalid argument modules unloaded. setmodopts: MODOPTS_PTLRPC=lbug_on_grant_miscount=1 setmodopts: MODOPTS_PTLRPC=lbug_on_grant_miscount=1 target_nid_policy=servicenode SERVICENODE DEBUG: setup base_policy=none servicenode_policy=enabled SERVICENODE DEBUG: before load MODOPTS_PTLRPC='lbug_on_grant_miscount=1 target_nid_policy=servicenode' SERVICENODE DEBUG: before load MODOPTS_LNET='' /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1 target_nid_policy=servicenode' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg242-server' oleg242-server: oleg242-server.virtnet: executing load_modules_local oleg242-server: Loading modules from /home/green/git/lustre-release/lustre oleg242-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg242-server: detected 4 online CPUs by sysfs oleg242-server: MODOPTS_LIBCFS= oleg242-server: Force libcfs to create 2 CPU partitions oleg242-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1 target_nid_policy=servicenode' oleg242-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg242-server: mdt/mdt options: 'mdt_enable_flr_ec=1' SERVICENODE DEBUG: modules loaded oleg242-server: oleg242-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre oleg242-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg242-client: oleg242-server: ssh exited with exit code 19 oleg242-server: oleg242-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre oleg242-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg242-client: oleg242-server: ssh exited with exit code 19 oleg242-server: oleg242-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre oleg242-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg242-client: oleg242-server: ssh exited with exit code 19 tunefs.lustre failed, reformatting instead Stopping clients: oleg242-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg242-client.virtnet /mnt/lustre2 (opts:-f) oleg242-server: oleg242-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg242-server' oleg242-server: oleg242-server.virtnet: executing load_modules_local oleg242-server: Loading modules from /home/green/git/lustre-release/lustre oleg242-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg242-server: detected 4 online CPUs by sysfs oleg242-server: MODOPTS_LIBCFS= oleg242-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg242-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg242-server: oleg242-server.virtnet: executing set_default_debug -1 all pdsh@oleg242-client: oleg242-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg242-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg242-server: oleg242-server.virtnet: executing set_default_debug -1 all pdsh@oleg242-client: oleg242-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg242-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg242-server: oleg242-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg242-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg242-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg242-server stop mds service on oleg242-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg242-server SERVICENODE DEBUG: writeconf_or_reformat complete oleg242-server: oleg242-server.virtnet: executing lnet_if_list SERVICENODE DEBUG: oss_if='ens2' client_if='ens2' oleg242-server: sh: /usr/sbin/ksocklnd-config: No such file or directory default via 192.168.202.254 dev ens2 192.168.202.0/24 dev ens2 proto kernel scope link src 192.168.202.42 default via 192.168.202.254 dev ens2 192.168.202.0/24 dev ens2 proto kernel scope link src 192.168.202.42 oleg242-server: sh: /usr/sbin/ksocklnd-config: No such file or directory default via 192.168.202.254 dev ens2 192.168.202.0/24 dev ens2 proto kernel scope link src 192.168.202.42 default via 192.168.202.254 dev ens2 192.168.202.0/24 dev ens2 proto kernel scope link src 192.168.202.42 default via 192.168.202.254 dev ens2 192.168.202.0/24 dev ens2 proto kernel scope link src 192.168.202.42 SERVICENODE DEBUG: net=tcp nid='192.168.202.142@tcp' SERVICENODE DEBUG: net=tcp1 nid='192.168.202.142@tcp1' SERVICENODE DEBUG: net=tcp2 nid='192.168.202.142@tcp2' SERVICENODE DEBUG: before temporary target format SERVICENODE DEBUG: HOSTNAME=oleg242-client.virtnet OSS=oleg242-server SERVICENODE DEBUG: MODOPTS_PTLRPC='lbug_on_grant_miscount=1 target_nid_policy=servicenode' SERVICENODE DEBUG: MODOPTS_LNET='' SERVICENODE DEBUG: nets='tcp,tcp1,tcp2' SERVICENODE DEBUG: nids='192.168.202.142@tcp,192.168.202.142@tcp1,192.168.202.142@tcp2' SERVICENODE DEBUG: client LNet configuration net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: tcp local NI(s): - nid: 192.168.202.42@tcp status: up interfaces: 0: ens2 - net type: tcp1 local NI(s): - nid: 192.168.202.42@tcp1 status: up interfaces: 0: ens2 - net type: tcp2 local NI(s): - nid: 192.168.202.42@tcp2 status: up interfaces: 0: ens2 SERVICENODE DEBUG: OSS LNet configuration net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: tcp local NI(s): - nid: 192.168.202.142@tcp status: up interfaces: 0: ens2 - net type: tcp1 local NI(s): - nid: 192.168.202.142@tcp1 status: up interfaces: 0: ens2 - net type: tcp2 local NI(s): - nid: 192.168.202.142@tcp2 status: up interfaces: 0: ens2 peer: - primary nid: 192.168.202.142@tcp Multi-Rail: true peer ni: - nid: 192.168.202.142@tcp state: NA SERVICENODE DEBUG: temporary fsname='sv25055' SERVICENODE DEBUG: temporary OST1 device='lustre-ost1_2/ost1_2' Permanent disk data: Device: lustre-ost1_2/ost1_2 Target: sv25055:OST0000 Index: 0 Lustre FS: sv25055 Mount type: zfs Flags: 0x1062 (OST first_time update no_primnode ) Persistent mount opts: Parameters: mgsnode=192.168.202.142@tcp sys.timeout=20 failover.node=192.168.202.142@tcp,192.168.202.142@tcp1 autodegrade=on mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1_2/ost1_2 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1_2/ost1_2 properties lustre:mgsnode=192.168.202.142@tcp lustre:sys.timeout=20 lustre:failover.node=192.168.202.142@tcp,192.168.202.142@tcp1 lustre:autodegrade=on lustre:version=1 lustre:flags=4194 lustre:index=0 lustre:fsname=sv25055 lustre:svname=sv25055:OST0000 SERVICENODE DEBUG: temporary OST2 device='lustre-ost2_2/ost2_2' Permanent disk data: Device: lustre-ost2_2/ost2_2 Target: sv25055:OST0001 Index: 1 Lustre FS: sv25055 Mount type: zfs Flags: 0x1062 (OST first_time update no_primnode ) Persistent mount opts: Parameters: mgsnode=192.168.202.142@tcp sys.timeout=20 failover.node=192.168.202.142@tcp autodegrade=on mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost2_2/ost2_2 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost2_2/ost2_2 properties lustre:mgsnode=192.168.202.142@tcp lustre:sys.timeout=20 lustre:failover.node=192.168.202.142@tcp lustre:autodegrade=on lustre:version=1 lustre:flags=4194 lustre:index=1 lustre:fsname=sv25055 lustre:svname=sv25055:OST0001 SERVICENODE DEBUG: starting MGS/MDT and temporary service-node targets start mds service on oleg242-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg242-server: oleg242-server.virtnet: executing set_default_debug -1 all pdsh@oleg242-client: oleg242-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 Start fs2ost1: mount -t lustre -o localrecov lustre-ost1_2/ost1_2 /mnt/lustre-fs2ost1 seq.cli-sv25055-OST0000-super.width=65536 oleg242-server: oleg242-server.virtnet: executing set_default_debug -1 all pdsh@oleg242-client: oleg242-server: ssh exited with exit code 1 Commit the device label on lustre-ost1_2/ost1_2 Started sv25055-OST0000 oleg242-server: missing pool name argument oleg242-server: usage: oleg242-server: trim [-dw] [-r ] [-c | -s] [ ...] pdsh@oleg242-client: oleg242-server: ssh exited with exit code 2 SERVICENODE DEBUG: after first target startup SERVICENODE DEBUG: HOSTNAME=oleg242-client.virtnet OSS=oleg242-server SERVICENODE DEBUG: MODOPTS_PTLRPC='lbug_on_grant_miscount=1 target_nid_policy=servicenode' SERVICENODE DEBUG: MODOPTS_LNET='' SERVICENODE DEBUG: nets='tcp,tcp1,tcp2' SERVICENODE DEBUG: nids='192.168.202.142@tcp,192.168.202.142@tcp1,192.168.202.142@tcp2' SERVICENODE DEBUG: client LNet configuration net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: tcp local NI(s): - nid: 192.168.202.42@tcp status: up interfaces: 0: ens2 - net type: tcp1 local NI(s): - nid: 192.168.202.42@tcp1 status: up interfaces: 0: ens2 - net type: tcp2 local NI(s): - nid: 192.168.202.42@tcp2 status: up interfaces: 0: ens2 SERVICENODE DEBUG: OSS LNet configuration net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: tcp local NI(s): - nid: 192.168.202.142@tcp status: up interfaces: 0: ens2 - net type: tcp1 local NI(s): - nid: 192.168.202.142@tcp1 status: up interfaces: 0: ens2 - net type: tcp2 local NI(s): - nid: 192.168.202.142@tcp2 status: up interfaces: 0: ens2 peer: - primary nid: 192.168.202.142@tcp Multi-Rail: true peer ni: - nid: 192.168.202.142@tcp state: NA - nid: 192.168.202.142@tcp1 state: NA SERVICENODE DEBUG: discover seed=192.168.202.142@tcp present='192.168.202.142@tcp' absent='192.168.202.142@tcp2' SERVICENODE DEBUG: discover rc=0 discover: - primary nid: 192.168.202.142@tcp Multi-Rail: true peer_ni: - nid: 192.168.202.142@tcp - nid: 192.168.202.142@tcp1 SERVICENODE DEBUG: peer show rc=0 peer: - primary nid: 192.168.202.142@tcp Multi-Rail: true peer ni: - nid: 192.168.202.142@tcp state: NA - nid: 192.168.202.142@tcp1 state: NA SERVICENODE DEBUG: starting temporary OST2 Start fs3ost2: mount -t lustre -o localrecov lustre-ost2_2/ost2_2 /mnt/lustre-fs3ost2 seq.cli-sv25055-OST0001-super.width=65536 oleg242-server: oleg242-server.virtnet: executing set_default_debug -1 all pdsh@oleg242-client: oleg242-server: ssh exited with exit code 1 Commit the device label on lustre-ost2_2/ost2_2 Started sv25055-OST0001 oleg242-server: missing pool name argument oleg242-server: usage: oleg242-server: trim [-dw] [-r ] [-c | -s] [ ...] pdsh@oleg242-client: oleg242-server: ssh exited with exit code 2 SERVICENODE DEBUG: after second target startup SERVICENODE DEBUG: HOSTNAME=oleg242-client.virtnet OSS=oleg242-server SERVICENODE DEBUG: MODOPTS_PTLRPC='lbug_on_grant_miscount=1 target_nid_policy=servicenode' SERVICENODE DEBUG: MODOPTS_LNET='' SERVICENODE DEBUG: nets='tcp,tcp1,tcp2' SERVICENODE DEBUG: nids='192.168.202.142@tcp,192.168.202.142@tcp1,192.168.202.142@tcp2' SERVICENODE DEBUG: client LNet configuration net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: tcp local NI(s): - nid: 192.168.202.42@tcp status: up interfaces: 0: ens2 - net type: tcp1 local NI(s): - nid: 192.168.202.42@tcp1 status: up interfaces: 0: ens2 - net type: tcp2 local NI(s): - nid: 192.168.202.42@tcp2 status: up interfaces: 0: ens2 peer: - primary nid: 192.168.202.142@tcp Multi-Rail: true peer ni: - nid: 192.168.202.142@tcp state: NA - nid: 192.168.202.142@tcp1 state: NA SERVICENODE DEBUG: OSS LNet configuration net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: tcp local NI(s): - nid: 192.168.202.142@tcp status: up interfaces: 0: ens2 - net type: tcp1 local NI(s): - nid: 192.168.202.142@tcp1 status: up interfaces: 0: ens2 - net type: tcp2 local NI(s): - nid: 192.168.202.142@tcp2 status: up interfaces: 0: ens2 peer: - primary nid: 192.168.202.142@tcp Multi-Rail: true peer ni: - nid: 192.168.202.142@tcp state: NA - nid: 192.168.202.142@tcp1 state: NA peer: - primary nid: 192.168.202.42@tcp Multi-Rail: true peer ni: - nid: 192.168.202.42@tcp state: NA SERVICENODE DEBUG: ping seed=192.168.202.142@tcp present='192.168.202.142@tcp' absent='192.168.202.142@tcp2' SERVICENODE DEBUG: ping rc=0 ping: - primary nid: 192.168.202.142@tcp Multi-Rail: true peer_ni: - nid: 192.168.202.142@tcp - nid: 192.168.202.142@tcp1 PASS 57i (277s) == conf-sanity test 57j: failnode targets contribute no servicenode group ========================================================== 18:01:12 (1788904872) umount lustre on /mnt/lustre..... stop ost1 service on oleg242-server stop mds service on oleg242-server LNET unconfigure error 22: Invalid argument unloading modules via unload_modules_local on: 'oleg242-server' oleg242-server: oleg242-server.virtnet: executing unload_modules_local oleg242-server: oleg242-server: LNET unconfigure error 22: Invalid argument modules unloaded. setmodopts: MODOPTS_PTLRPC=lbug_on_grant_miscount=1 setmodopts: MODOPTS_PTLRPC=lbug_on_grant_miscount=1 target_nid_policy=servicenode SERVICENODE DEBUG: setup base_policy=none servicenode_policy=enabled SERVICENODE DEBUG: before load MODOPTS_PTLRPC='lbug_on_grant_miscount=1 target_nid_policy=servicenode' SERVICENODE DEBUG: before load MODOPTS_LNET='' /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1 target_nid_policy=servicenode' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg242-server' oleg242-server: oleg242-server.virtnet: executing load_modules_local oleg242-server: Loading modules from /home/green/git/lustre-release/lustre oleg242-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg242-server: detected 4 online CPUs by sysfs oleg242-server: MODOPTS_LIBCFS= oleg242-server: Force libcfs to create 2 CPU partitions oleg242-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1 target_nid_policy=servicenode' oleg242-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg242-server: mdt/mdt options: 'mdt_enable_flr_ec=1' SERVICENODE DEBUG: modules loaded oleg242-server: oleg242-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre oleg242-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg242-client: oleg242-server: ssh exited with exit code 19 oleg242-server: oleg242-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre oleg242-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg242-client: oleg242-server: ssh exited with exit code 19 oleg242-server: oleg242-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre oleg242-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg242-client: oleg242-server: ssh exited with exit code 19 tunefs.lustre failed, reformatting instead Stopping clients: oleg242-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg242-client.virtnet /mnt/lustre2 (opts:-f) oleg242-server: oleg242-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg242-server' oleg242-server: oleg242-server.virtnet: executing load_modules_local oleg242-server: Loading modules from /home/green/git/lustre-release/lustre oleg242-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg242-server: detected 4 online CPUs by sysfs oleg242-server: MODOPTS_LIBCFS= oleg242-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg242-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg242-server: oleg242-server.virtnet: executing set_default_debug -1 all pdsh@oleg242-client: oleg242-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg242-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg242-server: oleg242-server.virtnet: executing set_default_debug -1 all pdsh@oleg242-client: oleg242-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg242-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg242-server: oleg242-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg242-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg242-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg242-server stop mds service on oleg242-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg242-server SERVICENODE DEBUG: writeconf_or_reformat complete oleg242-server: oleg242-server.virtnet: executing lnet_if_list SERVICENODE DEBUG: oss_if='ens2' client_if='ens2' oleg242-server: sh: /usr/sbin/ksocklnd-config: No such file or directory default via 192.168.202.254 dev ens2 192.168.202.0/24 dev ens2 proto kernel scope link src 192.168.202.42 default via 192.168.202.254 dev ens2 192.168.202.0/24 dev ens2 proto kernel scope link src 192.168.202.42 SERVICENODE DEBUG: net=tcp nid='192.168.202.142@tcp' SERVICENODE DEBUG: net=tcp1 nid='192.168.202.142@tcp1' SERVICENODE DEBUG: before temporary target format SERVICENODE DEBUG: HOSTNAME=oleg242-client.virtnet OSS=oleg242-server SERVICENODE DEBUG: MODOPTS_PTLRPC='lbug_on_grant_miscount=1 target_nid_policy=servicenode' SERVICENODE DEBUG: MODOPTS_LNET='' SERVICENODE DEBUG: nets='tcp,tcp1' SERVICENODE DEBUG: nids='192.168.202.142@tcp,192.168.202.142@tcp1' SERVICENODE DEBUG: client LNet configuration net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: tcp local NI(s): - nid: 192.168.202.42@tcp status: up interfaces: 0: ens2 - net type: tcp1 local NI(s): - nid: 192.168.202.42@tcp1 status: up interfaces: 0: ens2 SERVICENODE DEBUG: OSS LNet configuration net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: tcp local NI(s): - nid: 192.168.202.142@tcp status: up interfaces: 0: ens2 - net type: tcp1 local NI(s): - nid: 192.168.202.142@tcp1 status: up interfaces: 0: ens2 peer: - primary nid: 192.168.202.142@tcp Multi-Rail: true peer ni: - nid: 192.168.202.142@tcp state: NA SERVICENODE DEBUG: temporary fsname='sv15455' SERVICENODE DEBUG: temporary OST1 device='lustre-ost1_2/ost1_2' Permanent disk data: Device: lustre-ost1_2/ost1_2 Target: sv15455:OST0000 Index: 0 Lustre FS: sv15455 Mount type: zfs Flags: 0x62 (OST first_time update ) Persistent mount opts: Parameters: mgsnode=192.168.202.142@tcp sys.timeout=20 failover.node=192.168.202.42@tcp autodegrade=on mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1_2/ost1_2 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1_2/ost1_2 properties lustre:mgsnode=192.168.202.142@tcp lustre:sys.timeout=20 lustre:failover.node=192.168.202.42@tcp lustre:autodegrade=on lustre:version=1 lustre:flags=98 lustre:index=0 lustre:fsname=sv15455 lustre:svname=sv15455:OST0000 SERVICENODE DEBUG: temporary OST2 device='lustre-ost2_2/ost2_2' Permanent disk data: Device: lustre-ost2_2/ost2_2 Target: sv15455:OST0001 Index: 1 Lustre FS: sv15455 Mount type: zfs Flags: 0x62 (OST first_time update ) Persistent mount opts: Parameters: mgsnode=192.168.202.142@tcp sys.timeout=20 failover.node=192.168.202.42@tcp autodegrade=on mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost2_2/ost2_2 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost2_2/ost2_2 properties lustre:mgsnode=192.168.202.142@tcp lustre:sys.timeout=20 lustre:failover.node=192.168.202.42@tcp lustre:autodegrade=on lustre:version=1 lustre:flags=98 lustre:index=1 lustre:fsname=sv15455 lustre:svname=sv15455:OST0001 SERVICENODE DEBUG: starting MGS/MDT and temporary service-node targets start mds service on oleg242-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg242-server: oleg242-server.virtnet: executing set_default_debug -1 all pdsh@oleg242-client: oleg242-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 Start fs2ost1: mount -t lustre -o localrecov lustre-ost1_2/ost1_2 /mnt/lustre-fs2ost1 seq.cli-sv15455-OST0000-super.width=65536 oleg242-server: oleg242-server.virtnet: executing set_default_debug -1 all pdsh@oleg242-client: oleg242-server: ssh exited with exit code 1 Commit the device label on lustre-ost1_2/ost1_2 Started sv15455-OST0000 oleg242-server: missing pool name argument oleg242-server: usage: oleg242-server: trim [-dw] [-r ] [-c | -s] [ ...] pdsh@oleg242-client: oleg242-server: ssh exited with exit code 2 Start fs3ost2: mount -t lustre -o localrecov lustre-ost2_2/ost2_2 /mnt/lustre-fs3ost2 seq.cli-sv15455-OST0001-super.width=65536 oleg242-server: oleg242-server.virtnet: executing set_default_debug -1 all pdsh@oleg242-client: oleg242-server: ssh exited with exit code 1 Commit the device label on lustre-ost2_2/ost2_2 Started sv15455-OST0001 oleg242-server: missing pool name argument oleg242-server: usage: oleg242-server: trim [-dw] [-r ] [-c | -s] [ ...] pdsh@oleg242-client: oleg242-server: ssh exited with exit code 2 SERVICENODE DEBUG: after target startup SERVICENODE DEBUG: HOSTNAME=oleg242-client.virtnet OSS=oleg242-server SERVICENODE DEBUG: MODOPTS_PTLRPC='lbug_on_grant_miscount=1 target_nid_policy=servicenode' SERVICENODE DEBUG: MODOPTS_LNET='' SERVICENODE DEBUG: nets='tcp,tcp1' SERVICENODE DEBUG: nids='192.168.202.142@tcp,192.168.202.142@tcp1' SERVICENODE DEBUG: client LNet configuration net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: tcp local NI(s): - nid: 192.168.202.42@tcp status: up interfaces: 0: ens2 - net type: tcp1 local NI(s): - nid: 192.168.202.42@tcp1 status: up interfaces: 0: ens2 SERVICENODE DEBUG: OSS LNet configuration net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: tcp local NI(s): - nid: 192.168.202.142@tcp status: up interfaces: 0: ens2 - net type: tcp1 local NI(s): - nid: 192.168.202.142@tcp1 status: up interfaces: 0: ens2 peer: - primary nid: 192.168.202.142@tcp Multi-Rail: true peer ni: - nid: 192.168.202.142@tcp1 state: NA - nid: 192.168.202.142@tcp state: NA SERVICENODE DEBUG: ping seed=192.168.202.142@tcp present='192.168.202.142@tcp' absent='192.168.202.142@tcp1' SERVICENODE DEBUG: ping rc=0 ping: - primary nid: 192.168.202.142@tcp Multi-Rail: false peer_ni: - nid: 192.168.202.142@tcp SERVICENODE DEBUG: ping seed=192.168.202.142@tcp1 present='192.168.202.142@tcp1' absent='192.168.202.142@tcp' SERVICENODE DEBUG: ping rc=0 ping: - primary nid: 192.168.202.142@tcp1 Multi-Rail: false peer_ni: - nid: 192.168.202.142@tcp1 SERVICENODE DEBUG: discover seed=192.168.202.142@tcp present='192.168.202.142@tcp' absent='192.168.202.142@tcp1' SERVICENODE DEBUG: discover rc=0 discover: - primary nid: 192.168.202.142@tcp Multi-Rail: true peer_ni: - nid: 192.168.202.142@tcp SERVICENODE DEBUG: peer show rc=0 peer: - primary nid: 192.168.202.142@tcp Multi-Rail: true peer ni: - nid: 192.168.202.142@tcp state: NA PASS 57j (163s) == conf-sanity test 57k: unknown target_nid_policy stops the module load ========================================================== 18:03:55 (1788905035) umount lustre on /mnt/lustre..... stop ost1 service on oleg242-server stop mds service on oleg242-server LNET unconfigure error 22: Invalid argument unloading modules via unload_modules_local on: 'oleg242-server' oleg242-server: oleg242-server.virtnet: executing unload_modules_local oleg242-server: oleg242-server: LNET unconfigure error 22: Invalid argument modules unloaded. LNET unconfigure error 22: Invalid argument unloading modules via unload_modules_local on: 'oleg242-server' oleg242-server: oleg242-server.virtnet: executing unload_modules_local oleg242-server: oleg242-server: LNET unconfigure error 22: Invalid argument modules unloaded. oleg242-server: modprobe: FATAL: Module ptlrpc not found in directory /lib/modules/4.18.0rh8.10-debug pdsh@oleg242-client: oleg242-server: ssh exited with exit code 1 pdsh@oleg242-client: oleg242-server: ssh exited with exit code 1 pdsh@oleg242-client: oleg242-client: ssh exited with exit code 2 pdsh@oleg242-client: oleg242-server: ssh exited with exit code 2 pdsh@oleg242-client: oleg242-client: ssh exited with exit code 2 pdsh@oleg242-client: oleg242-server: ssh exited with exit code 2 LNET unconfigure error 22: Invalid argument unloading modules via unload_modules_local on: 'oleg242-server' oleg242-server: oleg242-server.virtnet: executing unload_modules_local oleg242-server: oleg242-server: LNET unconfigure error 22: Invalid argument modules unloaded. PASS 57k (16s) == conf-sanity test 57ka: IR READY respects servicenode target NID policy ========================================================== 18:04:11 (1788905051) umount lustre on /mnt/lustre..... stop ost1 service on oleg242-server stop mds service on oleg242-server LNET unconfigure error 22: Invalid argument unloading modules via unload_modules_local on: 'oleg242-server' oleg242-server: oleg242-server.virtnet: executing unload_modules_local oleg242-server: oleg242-server: LNET unconfigure error 22: Invalid argument modules unloaded. setmodopts: MODOPTS_PTLRPC=lbug_on_grant_miscount=1 setmodopts: MODOPTS_PTLRPC=lbug_on_grant_miscount=1 target_nid_policy=servicenode SERVICENODE DEBUG: setup base_policy=none servicenode_policy=enabled SERVICENODE DEBUG: before load MODOPTS_PTLRPC='lbug_on_grant_miscount=1 target_nid_policy=servicenode' SERVICENODE DEBUG: before load MODOPTS_LNET='' /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1 target_nid_policy=servicenode' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg242-server' oleg242-server: oleg242-server.virtnet: executing load_modules_local oleg242-server: Loading modules from /home/green/git/lustre-release/lustre oleg242-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg242-server: detected 4 online CPUs by sysfs oleg242-server: MODOPTS_LIBCFS= oleg242-server: Force libcfs to create 2 CPU partitions oleg242-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1 target_nid_policy=servicenode' oleg242-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg242-server: mdt/mdt options: 'mdt_enable_flr_ec=1' SERVICENODE DEBUG: modules loaded oleg242-server: oleg242-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre oleg242-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg242-client: oleg242-server: ssh exited with exit code 19 oleg242-server: oleg242-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre oleg242-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg242-client: oleg242-server: ssh exited with exit code 19 oleg242-server: oleg242-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre oleg242-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg242-client: oleg242-server: ssh exited with exit code 19 tunefs.lustre failed, reformatting instead Stopping clients: oleg242-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg242-client.virtnet /mnt/lustre2 (opts:-f) oleg242-server: oleg242-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg242-server' oleg242-server: oleg242-server.virtnet: executing load_modules_local oleg242-server: Loading modules from /home/green/git/lustre-release/lustre oleg242-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg242-server: detected 4 online CPUs by sysfs oleg242-server: MODOPTS_LIBCFS= oleg242-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg242-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg242-server: oleg242-server.virtnet: executing set_default_debug -1 all pdsh@oleg242-client: oleg242-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg242-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg242-server: oleg242-server.virtnet: executing set_default_debug -1 all pdsh@oleg242-client: oleg242-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg242-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg242-server: oleg242-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg242-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg242-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg242-server stop mds service on oleg242-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg242-server SERVICENODE DEBUG: writeconf_or_reformat complete oleg242-server: oleg242-server.virtnet: executing lnet_if_list SERVICENODE DEBUG: oss_if='ens2' client_if='ens2' oleg242-server: sh: /usr/sbin/ksocklnd-config: No such file or directory default via 192.168.202.254 dev ens2 192.168.202.0/24 dev ens2 proto kernel scope link src 192.168.202.42 default via 192.168.202.254 dev ens2 192.168.202.0/24 dev ens2 proto kernel scope link src 192.168.202.42 SERVICENODE DEBUG: net=tcp nid='192.168.202.142@tcp' SERVICENODE DEBUG: net=tcp1 nid='192.168.202.142@tcp1' SERVICENODE DEBUG: before temporary target format SERVICENODE DEBUG: HOSTNAME=oleg242-client.virtnet OSS=oleg242-server SERVICENODE DEBUG: MODOPTS_PTLRPC='lbug_on_grant_miscount=1 target_nid_policy=servicenode' SERVICENODE DEBUG: MODOPTS_LNET='' SERVICENODE DEBUG: nets='tcp,tcp1' SERVICENODE DEBUG: nids='192.168.202.142@tcp,192.168.202.142@tcp1' SERVICENODE DEBUG: client LNet configuration net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: tcp local NI(s): - nid: 192.168.202.42@tcp status: up interfaces: 0: ens2 - net type: tcp1 local NI(s): - nid: 192.168.202.42@tcp1 status: up interfaces: 0: ens2 SERVICENODE DEBUG: OSS LNet configuration net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: tcp local NI(s): - nid: 192.168.202.142@tcp status: up interfaces: 0: ens2 - net type: tcp1 local NI(s): - nid: 192.168.202.142@tcp1 status: up interfaces: 0: ens2 peer: - primary nid: 192.168.202.142@tcp Multi-Rail: true peer ni: - nid: 192.168.202.142@tcp state: NA SERVICENODE DEBUG: temporary fsname='sv30888' SERVICENODE DEBUG: temporary OST1 device='lustre-ost1_2/ost1_2' Permanent disk data: Device: lustre-ost1_2/ost1_2 Target: sv30888:OST0000 Index: 0 Lustre FS: sv30888 Mount type: zfs Flags: 0x1062 (OST first_time update no_primnode ) Persistent mount opts: Parameters: mgsnode=192.168.202.142@tcp sys.timeout=20 failover.node=192.168.202.142@tcp autodegrade=on mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1_2/ost1_2 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1_2/ost1_2 properties lustre:mgsnode=192.168.202.142@tcp lustre:sys.timeout=20 lustre:failover.node=192.168.202.142@tcp lustre:autodegrade=on lustre:version=1 lustre:flags=4194 lustre:index=0 lustre:fsname=sv30888 lustre:svname=sv30888:OST0000 SERVICENODE DEBUG: temporary OST2 device='lustre-ost2_2/ost2_2' Permanent disk data: Device: lustre-ost2_2/ost2_2 Target: sv30888:OST0001 Index: 1 Lustre FS: sv30888 Mount type: zfs Flags: 0x1062 (OST first_time update no_primnode ) Persistent mount opts: Parameters: mgsnode=192.168.202.142@tcp sys.timeout=20 failover.node=192.168.202.142@tcp1 autodegrade=on mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost2_2/ost2_2 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost2_2/ost2_2 properties lustre:mgsnode=192.168.202.142@tcp lustre:sys.timeout=20 lustre:failover.node=192.168.202.142@tcp1 lustre:autodegrade=on lustre:version=1 lustre:flags=4194 lustre:index=1 lustre:fsname=sv30888 lustre:svname=sv30888:OST0001 SERVICENODE DEBUG: starting MGS/MDT and temporary service-node targets start mds service on oleg242-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg242-server: oleg242-server.virtnet: executing set_default_debug -1 all pdsh@oleg242-client: oleg242-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 Start fs2ost1: mount -t lustre -o localrecov lustre-ost1_2/ost1_2 /mnt/lustre-fs2ost1 seq.cli-sv30888-OST0000-super.width=65536 oleg242-server: oleg242-server.virtnet: executing set_default_debug -1 all pdsh@oleg242-client: oleg242-server: ssh exited with exit code 1 Commit the device label on lustre-ost1_2/ost1_2 Started sv30888-OST0000 oleg242-server: missing pool name argument oleg242-server: usage: oleg242-server: trim [-dw] [-r ] [-c | -s] [ ...] pdsh@oleg242-client: oleg242-server: ssh exited with exit code 2 SERVICENODE DEBUG: after first target startup SERVICENODE DEBUG: HOSTNAME=oleg242-client.virtnet OSS=oleg242-server SERVICENODE DEBUG: MODOPTS_PTLRPC='lbug_on_grant_miscount=1 target_nid_policy=servicenode' SERVICENODE DEBUG: MODOPTS_LNET='' SERVICENODE DEBUG: nets='tcp,tcp1' SERVICENODE DEBUG: nids='192.168.202.142@tcp,192.168.202.142@tcp1' SERVICENODE DEBUG: client LNet configuration net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: tcp local NI(s): - nid: 192.168.202.42@tcp status: up interfaces: 0: ens2 - net type: tcp1 local NI(s): - nid: 192.168.202.42@tcp1 status: up interfaces: 0: ens2 SERVICENODE DEBUG: OSS LNet configuration net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: tcp local NI(s): - nid: 192.168.202.142@tcp status: up interfaces: 0: ens2 - net type: tcp1 local NI(s): - nid: 192.168.202.142@tcp1 status: up interfaces: 0: ens2 peer: - primary nid: 192.168.202.142@tcp Multi-Rail: true peer ni: - nid: 192.168.202.142@tcp state: NA Stopping /mnt/lustre-fs2ost1 (opts:-f) on oleg242-server fail_val=1 fail_loc=0x80000727 Start fs2ost1: mount -t lustre -o localrecov lustre-ost1_2/ost1_2 /mnt/lustre-fs2ost1 seq.cli-sv30888-OST0000-super.width=65536 oleg242-server: oleg242-server.virtnet: executing set_default_debug -1 all pdsh@oleg242-client: oleg242-server: ssh exited with exit code 1 Started sv30888-OST0000 fail_loc=0 fail_val=0 PASS 57ka (111s) Stopping clients: oleg242-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg242-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg242-client: oleg242-server: ssh exited with exit code 2 oleg242-server: oleg242-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg242-server' oleg242-server: oleg242-server.virtnet: executing load_modules_local oleg242-server: Loading modules from /home/green/git/lustre-release/lustre oleg242-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg242-server: detected 4 online CPUs by sysfs oleg242-server: MODOPTS_LIBCFS= oleg242-server: Force libcfs to create 2 CPU partitions oleg242-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg242-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg242-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 1946 sec ========== 18:06:29 (1788905189) === conf-sanity: start cleanup 18:06:30 (1788905190) === === conf-sanity: finish cleanup 18:06:31 (1788905191) ===