-----============= acceptance-small: conf-sanity ============----- Tue Sep 8 17:33:31 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg237-server: /home/green/git/lustre-release/lustre/tests/except/conf-sanity.ex - see CLIENT_VERSION > v2_15_63-53-g1a6ef725c2 (34682650 > 34553653) for LU-16938, go 81 excepting tests: 32newtarball 108a 112a 112b 113 117 119 121 122a 123aa 123ab 123ac 123ad 123ae 123af 123ag 123ah 123ahi 123F 123G 123H 126 129 132 133 135 136 137 150 152 153a 153b 153c 155 802a skipping tests SLOW=no: 45 69 106 111 114 Stopping clients: oleg237-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg237-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg237-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg237-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg237-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg237-server oleg237-server: oleg237-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg237-server' oleg237-server: oleg237-server.virtnet: executing load_modules_local oleg237-server: Loading modules from /home/green/git/lustre-release/lustre oleg237-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg237-server: detected 4 online CPUs by sysfs oleg237-server: MODOPTS_LIBCFS= oleg237-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg237-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg237-server: oleg237-server.virtnet: executing set_default_debug -1 all pdsh@oleg237-client: oleg237-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg237-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg237-server: oleg237-server.virtnet: executing set_default_debug -1 all pdsh@oleg237-client: oleg237-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg237-server: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg237-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg237-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg237-server stop mds service on oleg237-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg237-server == conf-sanity test 57b: initial registration from servicenode should not fail ========================================================== 17:36:28 (1788903388) oleg237-server: oleg237-server.virtnet: executing load_modules_local oleg237-server: Loading modules from /home/green/git/lustre-release/lustre oleg237-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg237-server: detected 4 online CPUs by sysfs oleg237-server: MODOPTS_LIBCFS= oleg237-server: Force libcfs to create 2 CPU partitions oleg237-server: oleg237-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre oleg237-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg237-client: oleg237-server: ssh exited with exit code 19 oleg237-server: oleg237-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre oleg237-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg237-client: oleg237-server: ssh exited with exit code 19 oleg237-server: oleg237-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre oleg237-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg237-client: oleg237-server: ssh exited with exit code 19 tunefs.lustre failed, reformatting instead Stopping clients: oleg237-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg237-client.virtnet /mnt/lustre2 (opts:-f) oleg237-server: oleg237-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg237-server' oleg237-server: oleg237-server.virtnet: executing load_modules_local oleg237-server: Loading modules from /home/green/git/lustre-release/lustre oleg237-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg237-server: detected 4 online CPUs by sysfs oleg237-server: MODOPTS_LIBCFS= oleg237-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg237-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg237-server: oleg237-server.virtnet: executing set_default_debug -1 all pdsh@oleg237-client: oleg237-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg237-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg237-server: oleg237-server.virtnet: executing set_default_debug -1 all pdsh@oleg237-client: oleg237-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg237-server: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg237-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg237-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg237-server stop mds service on oleg237-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg237-server checking for existing Lustre data: found Read previous values: Device: lustre-ost1/ost1 Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x2 (OST ) Persistent mount opts: Parameters: autodegrade=on sys.timeout=20 mgsnode=192.168.202.137@tcp Permanent disk data: Device: lustre-ost1/ost1 Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x1042 (OST update no_primnode ) Persistent mount opts: Parameters: autodegrade=on sys.timeout=20 mgsnode=192.168.202.137@tcp failover.node=192.168.202.137@tcp autodegrade=on Writing lustre-ost1/ost1 properties lustre:autodegrade=on lustre:sys.timeout=20 lustre:mgsnode=192.168.202.137@tcp lustre:failover.node=192.168.202.137@tcp lustre:autodegrade=on lustre:version=1 lustre:flags=4162 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start mds service on oleg237-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg237-server: oleg237-server.virtnet: executing set_default_debug -1 all pdsh@oleg237-client: oleg237-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg237-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg237-server: oleg237-server.virtnet: executing set_default_debug -1 all pdsh@oleg237-client: oleg237-server: ssh exited with exit code 1 Started lustre-OST0000 oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 umount lustre on /mnt/lustre..... stop ost1 service on oleg237-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg237-server stop mds service on oleg237-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg237-server unloading modules via unload_modules_local on: 'oleg237-server' oleg237-server: oleg237-server.virtnet: executing unload_modules_local oleg237-server: modules unloaded. pdsh@oleg237-client: oleg237-client: ssh exited with exit code 2 pdsh@oleg237-client: oleg237-server: ssh exited with exit code 2 pdsh@oleg237-client: oleg237-client: ssh exited with exit code 2 pdsh@oleg237-client: oleg237-server: ssh exited with exit code 2 PASS 57b (159s) == conf-sanity test 57c: servicenode metadata preserves legacy discovery by default ========================================================== 17:39:07 (1788903547) SERVICENODE DEBUG: begin legacy-policy test umount lustre on /mnt/lustre..... stop ost1 service on oleg237-server stop mds service on oleg237-server LNET unconfigure error 22: Invalid argument unloading modules via unload_modules_local on: 'oleg237-server' oleg237-server: oleg237-server.virtnet: executing unload_modules_local oleg237-server: oleg237-server: LNET unconfigure error 22: Invalid argument modules unloaded. setmodopts: MODOPTS_PTLRPC=lbug_on_grant_miscount=1 SERVICENODE DEBUG: setup base_policy=none servicenode_policy=disabled SERVICENODE DEBUG: before load MODOPTS_PTLRPC='lbug_on_grant_miscount=1' SERVICENODE DEBUG: before load MODOPTS_LNET='' /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg237-server' oleg237-server: oleg237-server.virtnet: executing load_modules_local oleg237-server: Loading modules from /home/green/git/lustre-release/lustre oleg237-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg237-server: detected 4 online CPUs by sysfs oleg237-server: MODOPTS_LIBCFS= oleg237-server: Force libcfs to create 2 CPU partitions oleg237-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg237-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg237-server: mdt/mdt options: 'mdt_enable_flr_ec=1' SERVICENODE DEBUG: modules loaded oleg237-server: oleg237-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre oleg237-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg237-client: oleg237-server: ssh exited with exit code 19 oleg237-server: oleg237-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre oleg237-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg237-client: oleg237-server: ssh exited with exit code 19 oleg237-server: oleg237-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre oleg237-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg237-client: oleg237-server: ssh exited with exit code 19 tunefs.lustre failed, reformatting instead Stopping clients: oleg237-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg237-client.virtnet /mnt/lustre2 (opts:-f) oleg237-server: oleg237-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg237-server' oleg237-server: oleg237-server.virtnet: executing load_modules_local oleg237-server: Loading modules from /home/green/git/lustre-release/lustre oleg237-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg237-server: detected 4 online CPUs by sysfs oleg237-server: MODOPTS_LIBCFS= oleg237-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg237-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg237-server: oleg237-server.virtnet: executing set_default_debug -1 all pdsh@oleg237-client: oleg237-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg237-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg237-server: oleg237-server.virtnet: executing set_default_debug -1 all pdsh@oleg237-client: oleg237-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg237-server: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg237-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg237-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg237-server stop mds service on oleg237-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg237-server SERVICENODE DEBUG: writeconf_or_reformat complete oleg237-server: oleg237-server.virtnet: executing lnet_if_list SERVICENODE DEBUG: oss_if='ens2' client_if='ens2' oleg237-server: sh: /usr/sbin/ksocklnd-config: No such file or directory SERVICENODE DEBUG: net=tcp nid='192.168.202.137@tcp' SERVICENODE DEBUG: net=tcp1 nid='192.168.202.137@tcp1' SERVICENODE DEBUG: before temporary target format SERVICENODE DEBUG: HOSTNAME=oleg237-client.virtnet OSS=oleg237-server SERVICENODE DEBUG: MODOPTS_PTLRPC='lbug_on_grant_miscount=1' SERVICENODE DEBUG: MODOPTS_LNET='' SERVICENODE DEBUG: nets='tcp,tcp1' SERVICENODE DEBUG: nids='192.168.202.137@tcp,192.168.202.137@tcp1' SERVICENODE DEBUG: client LNet configuration net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: tcp local NI(s): - nid: 192.168.202.37@tcp status: up interfaces: 0: ens2 - net type: tcp1 local NI(s): - nid: 192.168.202.37@tcp1 status: up interfaces: 0: ens2 SERVICENODE DEBUG: OSS LNet configuration net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: tcp local NI(s): - nid: 192.168.202.137@tcp status: up interfaces: 0: ens2 - net type: tcp1 local NI(s): - nid: 192.168.202.137@tcp1 status: up interfaces: 0: ens2 peer: - primary nid: 192.168.202.137@tcp Multi-Rail: true peer ni: - nid: 192.168.202.137@tcp state: NA - nid: 192.168.202.137@tcp1 state: NA SERVICENODE DEBUG: temporary fsname='sv21925' SERVICENODE DEBUG: temporary OST1 device='lustre-ost1_2/ost1_2' Permanent disk data: Device: lustre-ost1_2/ost1_2 Target: sv21925:OST0000 Index: 0 Lustre FS: sv21925 Mount type: zfs Flags: 0x1062 (OST first_time update no_primnode ) Persistent mount opts: Parameters: mgsnode=192.168.202.137@tcp sys.timeout=20 failover.node=192.168.202.137@tcp autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1_2 /tmp/lustre-ost1_2 mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1_2/ost1_2 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1_2/ost1_2 properties lustre:mgsnode=192.168.202.137@tcp lustre:sys.timeout=20 lustre:failover.node=192.168.202.137@tcp lustre:autodegrade=on lustre:version=1 lustre:flags=4194 lustre:index=0 lustre:fsname=sv21925 lustre:svname=sv21925:OST0000 SERVICENODE DEBUG: temporary OST2 device='lustre-ost2_2/ost2_2' Permanent disk data: Device: lustre-ost2_2/ost2_2 Target: sv21925:OST0001 Index: 1 Lustre FS: sv21925 Mount type: zfs Flags: 0x1062 (OST first_time update no_primnode ) Persistent mount opts: Parameters: mgsnode=192.168.202.137@tcp sys.timeout=20 failover.node=192.168.202.137@tcp1 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost2_2 /tmp/lustre-ost2_2 mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost2_2/ost2_2 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost2_2/ost2_2 properties lustre:mgsnode=192.168.202.137@tcp lustre:sys.timeout=20 lustre:failover.node=192.168.202.137@tcp1 lustre:autodegrade=on lustre:version=1 lustre:flags=4194 lustre:index=1 lustre:fsname=sv21925 lustre:svname=sv21925:OST0001 SERVICENODE DEBUG: starting MGS/MDT and temporary service-node targets start mds service on oleg237-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg237-server: oleg237-server.virtnet: executing set_default_debug -1 all pdsh@oleg237-client: oleg237-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 Start fs2ost1: mount -t lustre -o localrecov lustre-ost1_2/ost1_2 /mnt/lustre-fs2ost1 seq.cli-sv21925-OST0000-super.width=65536 oleg237-server: oleg237-server.virtnet: executing set_default_debug -1 all pdsh@oleg237-client: oleg237-server: ssh exited with exit code 1 Commit the device label on lustre-ost1_2/ost1_2 Started sv21925-OST0000 oleg237-server: missing pool name argument oleg237-server: usage: oleg237-server: trim [-dw] [-r ] [-c | -s] [ ...] pdsh@oleg237-client: oleg237-server: ssh exited with exit code 2 Start fs3ost2: mount -t lustre -o localrecov lustre-ost2_2/ost2_2 /mnt/lustre-fs3ost2 seq.cli-sv21925-OST0001-super.width=65536 oleg237-server: oleg237-server.virtnet: executing set_default_debug -1 all pdsh@oleg237-client: oleg237-server: ssh exited with exit code 1 Commit the device label on lustre-ost2_2/ost2_2 Started sv21925-OST0001 oleg237-server: missing pool name argument oleg237-server: usage: oleg237-server: trim [-dw] [-r ] [-c | -s] [ ...] pdsh@oleg237-client: oleg237-server: ssh exited with exit code 2 SERVICENODE DEBUG: after target startup SERVICENODE DEBUG: HOSTNAME=oleg237-client.virtnet OSS=oleg237-server SERVICENODE DEBUG: MODOPTS_PTLRPC='lbug_on_grant_miscount=1' SERVICENODE DEBUG: MODOPTS_LNET='' SERVICENODE DEBUG: nets='tcp,tcp1' SERVICENODE DEBUG: nids='192.168.202.137@tcp,192.168.202.137@tcp1' SERVICENODE DEBUG: client LNet configuration net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: tcp local NI(s): - nid: 192.168.202.37@tcp status: up interfaces: 0: ens2 - net type: tcp1 local NI(s): - nid: 192.168.202.37@tcp1 status: up interfaces: 0: ens2 SERVICENODE DEBUG: OSS LNet configuration net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: tcp local NI(s): - nid: 192.168.202.137@tcp status: up interfaces: 0: ens2 - net type: tcp1 local NI(s): - nid: 192.168.202.137@tcp1 status: up interfaces: 0: ens2 peer: - primary nid: 192.168.202.137@tcp Multi-Rail: true peer ni: - nid: 192.168.202.137@tcp state: NA - nid: 192.168.202.137@tcp1 state: NA SERVICENODE DEBUG: legacy expected full view='192.168.202.137@tcp,192.168.202.137@tcp1' SERVICENODE DEBUG: ping seed=192.168.202.137@tcp present='192.168.202.137@tcp,192.168.202.137@tcp1' absent='' SERVICENODE DEBUG: ping rc=0 ping: - primary nid: 192.168.202.137@tcp Multi-Rail: false peer_ni: - nid: 192.168.202.137@tcp - nid: 192.168.202.137@tcp1 SERVICENODE DEBUG: discover seed=192.168.202.137@tcp present='192.168.202.137@tcp,192.168.202.137@tcp1' absent='' SERVICENODE DEBUG: discover rc=0 discover: - primary nid: 192.168.202.137@tcp Multi-Rail: true peer_ni: - nid: 192.168.202.137@tcp - nid: 192.168.202.137@tcp1 SERVICENODE DEBUG: peer show rc=0 peer: - primary nid: 192.168.202.137@tcp Multi-Rail: true peer ni: - nid: 192.168.202.137@tcp state: NA - nid: 192.168.202.137@tcp1 state: NA SERVICENODE DEBUG: legacy-policy test complete PASS 57c (164s) == conf-sanity test 57d: servicenode policy splits OSS discovery views ========================================================== 17:41:51 (1788903711) umount lustre on /mnt/lustre..... stop ost1 service on oleg237-server stop mds service on oleg237-server LNET unconfigure error 22: Invalid argument unloading modules via unload_modules_local on: 'oleg237-server' oleg237-server: oleg237-server.virtnet: executing unload_modules_local oleg237-server: oleg237-server: LNET unconfigure error 22: Invalid argument modules unloaded. setmodopts: MODOPTS_PTLRPC=lbug_on_grant_miscount=1 setmodopts: MODOPTS_PTLRPC=lbug_on_grant_miscount=1 target_nid_policy=servicenode SERVICENODE DEBUG: setup base_policy=none servicenode_policy=enabled SERVICENODE DEBUG: before load MODOPTS_PTLRPC='lbug_on_grant_miscount=1 target_nid_policy=servicenode' SERVICENODE DEBUG: before load MODOPTS_LNET='' /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1 target_nid_policy=servicenode' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg237-server' oleg237-server: oleg237-server.virtnet: executing load_modules_local oleg237-server: Loading modules from /home/green/git/lustre-release/lustre oleg237-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg237-server: detected 4 online CPUs by sysfs oleg237-server: MODOPTS_LIBCFS= oleg237-server: Force libcfs to create 2 CPU partitions oleg237-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1 target_nid_policy=servicenode' oleg237-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg237-server: mdt/mdt options: 'mdt_enable_flr_ec=1' SERVICENODE DEBUG: modules loaded oleg237-server: oleg237-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre oleg237-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg237-client: oleg237-server: ssh exited with exit code 19 oleg237-server: oleg237-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre oleg237-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg237-client: oleg237-server: ssh exited with exit code 19 oleg237-server: oleg237-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre oleg237-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg237-client: oleg237-server: ssh exited with exit code 19 tunefs.lustre failed, reformatting instead Stopping clients: oleg237-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg237-client.virtnet /mnt/lustre2 (opts:-f) oleg237-server: oleg237-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg237-server' oleg237-server: oleg237-server.virtnet: executing load_modules_local oleg237-server: Loading modules from /home/green/git/lustre-release/lustre oleg237-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg237-server: detected 4 online CPUs by sysfs oleg237-server: MODOPTS_LIBCFS= oleg237-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg237-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg237-server: oleg237-server.virtnet: executing set_default_debug -1 all pdsh@oleg237-client: oleg237-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg237-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg237-server: oleg237-server.virtnet: executing set_default_debug -1 all pdsh@oleg237-client: oleg237-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg237-server: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg237-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg237-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg237-server stop mds service on oleg237-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg237-server SERVICENODE DEBUG: writeconf_or_reformat complete oleg237-server: oleg237-server.virtnet: executing lnet_if_list SERVICENODE DEBUG: oss_if='ens2' client_if='ens2' oleg237-server: sh: /usr/sbin/ksocklnd-config: No such file or directory default via 192.168.202.254 dev ens2 192.168.202.0/24 dev ens2 proto kernel scope link src 192.168.202.37 default via 192.168.202.254 dev ens2 192.168.202.0/24 dev ens2 proto kernel scope link src 192.168.202.37 SERVICENODE DEBUG: net=tcp nid='192.168.202.137@tcp' SERVICENODE DEBUG: net=tcp1 nid='192.168.202.137@tcp1' SERVICENODE DEBUG: before temporary target format SERVICENODE DEBUG: HOSTNAME=oleg237-client.virtnet OSS=oleg237-server SERVICENODE DEBUG: MODOPTS_PTLRPC='lbug_on_grant_miscount=1 target_nid_policy=servicenode' SERVICENODE DEBUG: MODOPTS_LNET='' SERVICENODE DEBUG: nets='tcp,tcp1' SERVICENODE DEBUG: nids='192.168.202.137@tcp,192.168.202.137@tcp1' SERVICENODE DEBUG: client LNet configuration net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: tcp local NI(s): - nid: 192.168.202.37@tcp status: up interfaces: 0: ens2 - net type: tcp1 local NI(s): - nid: 192.168.202.37@tcp1 status: up interfaces: 0: ens2 SERVICENODE DEBUG: OSS LNet configuration net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: tcp local NI(s): - nid: 192.168.202.137@tcp status: up interfaces: 0: ens2 - net type: tcp1 local NI(s): - nid: 192.168.202.137@tcp1 status: up interfaces: 0: ens2 peer: - primary nid: 192.168.202.137@tcp Multi-Rail: true peer ni: - nid: 192.168.202.137@tcp state: NA SERVICENODE DEBUG: temporary fsname='sv14345' SERVICENODE DEBUG: temporary OST1 device='lustre-ost1_2/ost1_2' Permanent disk data: Device: lustre-ost1_2/ost1_2 Target: sv14345:OST0000 Index: 0 Lustre FS: sv14345 Mount type: zfs Flags: 0x1062 (OST first_time update no_primnode ) Persistent mount opts: Parameters: mgsnode=192.168.202.137@tcp sys.timeout=20 failover.node=192.168.202.137@tcp autodegrade=on mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1_2/ost1_2 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1_2/ost1_2 properties lustre:mgsnode=192.168.202.137@tcp lustre:sys.timeout=20 lustre:failover.node=192.168.202.137@tcp lustre:autodegrade=on lustre:version=1 lustre:flags=4194 lustre:index=0 lustre:fsname=sv14345 lustre:svname=sv14345:OST0000 SERVICENODE DEBUG: temporary OST2 device='lustre-ost2_2/ost2_2' Permanent disk data: Device: lustre-ost2_2/ost2_2 Target: sv14345:OST0001 Index: 1 Lustre FS: sv14345 Mount type: zfs Flags: 0x1062 (OST first_time update no_primnode ) Persistent mount opts: Parameters: mgsnode=192.168.202.137@tcp sys.timeout=20 failover.node=192.168.202.137@tcp1 autodegrade=on mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost2_2/ost2_2 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost2_2/ost2_2 properties lustre:mgsnode=192.168.202.137@tcp lustre:sys.timeout=20 lustre:failover.node=192.168.202.137@tcp1 lustre:autodegrade=on lustre:version=1 lustre:flags=4194 lustre:index=1 lustre:fsname=sv14345 lustre:svname=sv14345:OST0001 SERVICENODE DEBUG: starting MGS/MDT and temporary service-node targets start mds service on oleg237-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg237-server: oleg237-server.virtnet: executing set_default_debug -1 all pdsh@oleg237-client: oleg237-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 Start fs2ost1: mount -t lustre -o localrecov lustre-ost1_2/ost1_2 /mnt/lustre-fs2ost1 seq.cli-sv14345-OST0000-super.width=65536 oleg237-server: oleg237-server.virtnet: executing set_default_debug -1 all pdsh@oleg237-client: oleg237-server: ssh exited with exit code 1 Commit the device label on lustre-ost1_2/ost1_2 Started sv14345-OST0000 oleg237-server: missing pool name argument oleg237-server: usage: oleg237-server: trim [-dw] [-r ] [-c | -s] [ ...] pdsh@oleg237-client: oleg237-server: ssh exited with exit code 2 Start fs3ost2: mount -t lustre -o localrecov lustre-ost2_2/ost2_2 /mnt/lustre-fs3ost2 seq.cli-sv14345-OST0001-super.width=65536 oleg237-server: oleg237-server.virtnet: executing set_default_debug -1 all pdsh@oleg237-client: oleg237-server: ssh exited with exit code 1 Commit the device label on lustre-ost2_2/ost2_2 Started sv14345-OST0001 oleg237-server: missing pool name argument oleg237-server: usage: oleg237-server: trim [-dw] [-r ] [-c | -s] [ ...] pdsh@oleg237-client: oleg237-server: ssh exited with exit code 2 SERVICENODE DEBUG: after target startup SERVICENODE DEBUG: HOSTNAME=oleg237-client.virtnet OSS=oleg237-server SERVICENODE DEBUG: MODOPTS_PTLRPC='lbug_on_grant_miscount=1 target_nid_policy=servicenode' SERVICENODE DEBUG: MODOPTS_LNET='' SERVICENODE DEBUG: nets='tcp,tcp1' SERVICENODE DEBUG: nids='192.168.202.137@tcp,192.168.202.137@tcp1' SERVICENODE DEBUG: client LNet configuration net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: tcp local NI(s): - nid: 192.168.202.37@tcp status: up interfaces: 0: ens2 - net type: tcp1 local NI(s): - nid: 192.168.202.37@tcp1 status: up interfaces: 0: ens2 SERVICENODE DEBUG: OSS LNet configuration net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: tcp local NI(s): - nid: 192.168.202.137@tcp status: up interfaces: 0: ens2 - net type: tcp1 local NI(s): - nid: 192.168.202.137@tcp1 status: up interfaces: 0: ens2 peer: - primary nid: 192.168.202.137@tcp Multi-Rail: true peer ni: - nid: 192.168.202.137@tcp state: NA SERVICENODE DEBUG: ping seed=192.168.202.137@tcp present='192.168.202.137@tcp' absent='192.168.202.137@tcp1' SERVICENODE DEBUG: ping rc=0 ping: - primary nid: 192.168.202.137@tcp Multi-Rail: false peer_ni: - nid: 192.168.202.137@tcp SERVICENODE DEBUG: ping seed=192.168.202.137@tcp1 present='192.168.202.137@tcp1' absent='192.168.202.137@tcp' SERVICENODE DEBUG: ping rc=0 ping: - primary nid: 192.168.202.137@tcp1 Multi-Rail: false peer_ni: - nid: 192.168.202.137@tcp1 SERVICENODE DEBUG: discover seed=192.168.202.137@tcp present='192.168.202.137@tcp' absent='192.168.202.137@tcp1' SERVICENODE DEBUG: discover rc=0 discover: - primary nid: 192.168.202.137@tcp Multi-Rail: true peer_ni: - nid: 192.168.202.137@tcp SERVICENODE DEBUG: peer show rc=0 peer: - primary nid: 192.168.202.137@tcp Multi-Rail: true peer ni: - nid: 192.168.202.137@tcp state: NA SERVICENODE DEBUG: discover seed=192.168.202.137@tcp1 present='192.168.202.137@tcp1' absent='192.168.202.137@tcp' SERVICENODE DEBUG: discover rc=0 discover: - primary nid: 192.168.202.137@tcp1 Multi-Rail: true peer_ni: - nid: 192.168.202.137@tcp1 SERVICENODE DEBUG: peer show rc=0 peer: - primary nid: 192.168.202.137@tcp1 Multi-Rail: true peer ni: - nid: 192.168.202.137@tcp1 state: NA PASS 57d (143s) == conf-sanity test 57e: servicenode policy overrides self grouping ========================================================== 17:44:14 (1788903854) umount lustre on /mnt/lustre..... stop ost1 service on oleg237-server stop mds service on oleg237-server LNET unconfigure error 22: Invalid argument unloading modules via unload_modules_local on: 'oleg237-server' oleg237-server: oleg237-server.virtnet: executing unload_modules_local oleg237-server: oleg237-server: LNET unconfigure error 22: Invalid argument modules unloaded. setmodopts: MODOPTS_PTLRPC=lbug_on_grant_miscount=1 setmodopts: MODOPTS_PTLRPC=lbug_on_grant_miscount=1 target_nid_policy=servicenode setmodopts: MODOPTS_LNET= lnet_self_nets=tcp,tcp1 SERVICENODE DEBUG: setup base_policy=self servicenode_policy=enabled SERVICENODE DEBUG: before load MODOPTS_PTLRPC='lbug_on_grant_miscount=1 target_nid_policy=servicenode' SERVICENODE DEBUG: before load MODOPTS_LNET=' lnet_self_nets=tcp,tcp1' /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ../lnet/lnet/lnet options: 'lnet_self_nets=tcp,tcp1' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1 target_nid_policy=servicenode' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg237-server' oleg237-server: oleg237-server.virtnet: executing load_modules_local oleg237-server: Loading modules from /home/green/git/lustre-release/lustre oleg237-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg237-server: detected 4 online CPUs by sysfs oleg237-server: MODOPTS_LIBCFS= oleg237-server: Force libcfs to create 2 CPU partitions oleg237-server: ../lnet/lnet/lnet options: 'lnet_self_nets=tcp,tcp1' oleg237-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1 target_nid_policy=servicenode' oleg237-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg237-server: mdt/mdt options: 'mdt_enable_flr_ec=1' SERVICENODE DEBUG: modules loaded oleg237-server: oleg237-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre oleg237-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg237-client: oleg237-server: ssh exited with exit code 19 oleg237-server: oleg237-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre oleg237-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg237-client: oleg237-server: ssh exited with exit code 19 oleg237-server: oleg237-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre oleg237-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg237-client: oleg237-server: ssh exited with exit code 19 tunefs.lustre failed, reformatting instead Stopping clients: oleg237-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg237-client.virtnet /mnt/lustre2 (opts:-f) oleg237-server: oleg237-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg237-server' oleg237-server: oleg237-server.virtnet: executing load_modules_local oleg237-server: Loading modules from /home/green/git/lustre-release/lustre oleg237-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg237-server: detected 4 online CPUs by sysfs oleg237-server: MODOPTS_LIBCFS= oleg237-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg237-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg237-server: oleg237-server.virtnet: executing set_default_debug -1 all pdsh@oleg237-client: oleg237-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg237-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg237-server: oleg237-server.virtnet: executing set_default_debug -1 all pdsh@oleg237-client: oleg237-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg237-server: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg237-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg237-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg237-server stop mds service on oleg237-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg237-server SERVICENODE DEBUG: writeconf_or_reformat complete oleg237-server: oleg237-server.virtnet: executing lnet_if_list SERVICENODE DEBUG: oss_if='ens2' client_if='ens2' oleg237-server: sh: /usr/sbin/ksocklnd-config: No such file or directory default via 192.168.202.254 dev ens2 192.168.202.0/24 dev ens2 proto kernel scope link src 192.168.202.37 default via 192.168.202.254 dev ens2 192.168.202.0/24 dev ens2 proto kernel scope link src 192.168.202.37 SERVICENODE DEBUG: net=tcp nid='192.168.202.137@tcp' SERVICENODE DEBUG: net=tcp1 nid='192.168.202.137@tcp1' SERVICENODE DEBUG: before temporary target format SERVICENODE DEBUG: HOSTNAME=oleg237-client.virtnet OSS=oleg237-server SERVICENODE DEBUG: MODOPTS_PTLRPC='lbug_on_grant_miscount=1 target_nid_policy=servicenode' SERVICENODE DEBUG: MODOPTS_LNET=' lnet_self_nets=tcp,tcp1' SERVICENODE DEBUG: nets='tcp,tcp1' SERVICENODE DEBUG: nids='192.168.202.137@tcp,192.168.202.137@tcp1' SERVICENODE DEBUG: client LNet configuration net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: tcp local NI(s): - nid: 192.168.202.37@tcp status: up interfaces: 0: ens2 - net type: tcp1 local NI(s): - nid: 192.168.202.37@tcp1 status: up interfaces: 0: ens2 SERVICENODE DEBUG: OSS LNet configuration net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: tcp local NI(s): - nid: 192.168.202.137@tcp status: up interfaces: 0: ens2 - net type: tcp1 local NI(s): - nid: 192.168.202.137@tcp1 status: up interfaces: 0: ens2 peer: - primary nid: 192.168.202.137@tcp Multi-Rail: true peer ni: - nid: 192.168.202.137@tcp state: NA SERVICENODE DEBUG: temporary fsname='sv13828' SERVICENODE DEBUG: temporary OST1 device='lustre-ost1_2/ost1_2' Permanent disk data: Device: lustre-ost1_2/ost1_2 Target: sv13828:OST0000 Index: 0 Lustre FS: sv13828 Mount type: zfs Flags: 0x1062 (OST first_time update no_primnode ) Persistent mount opts: Parameters: mgsnode=192.168.202.137@tcp sys.timeout=20 failover.node=192.168.202.137@tcp,192.168.202.137@tcp1 autodegrade=on mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1_2/ost1_2 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1_2/ost1_2 properties lustre:mgsnode=192.168.202.137@tcp lustre:sys.timeout=20 lustre:failover.node=192.168.202.137@tcp,192.168.202.137@tcp1 lustre:autodegrade=on lustre:version=1 lustre:flags=4194 lustre:index=0 lustre:fsname=sv13828 lustre:svname=sv13828:OST0000 SERVICENODE DEBUG: temporary OST2 device='lustre-ost2_2/ost2_2' Permanent disk data: Device: lustre-ost2_2/ost2_2 Target: sv13828:OST0001 Index: 1 Lustre FS: sv13828 Mount type: zfs Flags: 0x1062 (OST first_time update no_primnode ) Persistent mount opts: Parameters: mgsnode=192.168.202.137@tcp sys.timeout=20 failover.node=192.168.202.137@tcp1 autodegrade=on mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost2_2/ost2_2 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost2_2/ost2_2 properties lustre:mgsnode=192.168.202.137@tcp lustre:sys.timeout=20 lustre:failover.node=192.168.202.137@tcp1 lustre:autodegrade=on lustre:version=1 lustre:flags=4194 lustre:index=1 lustre:fsname=sv13828 lustre:svname=sv13828:OST0001 SERVICENODE DEBUG: starting MGS/MDT and temporary service-node targets start mds service on oleg237-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg237-server: oleg237-server.virtnet: executing set_default_debug -1 all pdsh@oleg237-client: oleg237-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 Start fs2ost1: mount -t lustre -o localrecov lustre-ost1_2/ost1_2 /mnt/lustre-fs2ost1 seq.cli-sv13828-OST0000-super.width=65536 oleg237-server: oleg237-server.virtnet: executing set_default_debug -1 all pdsh@oleg237-client: oleg237-server: ssh exited with exit code 1 Commit the device label on lustre-ost1_2/ost1_2 Started sv13828-OST0000 oleg237-server: missing pool name argument oleg237-server: usage: oleg237-server: trim [-dw] [-r ] [-c | -s] [ ...] pdsh@oleg237-client: oleg237-server: ssh exited with exit code 2 SERVICENODE DEBUG: after first target startup SERVICENODE DEBUG: HOSTNAME=oleg237-client.virtnet OSS=oleg237-server SERVICENODE DEBUG: MODOPTS_PTLRPC='lbug_on_grant_miscount=1 target_nid_policy=servicenode' SERVICENODE DEBUG: MODOPTS_LNET=' lnet_self_nets=tcp,tcp1' SERVICENODE DEBUG: nets='tcp,tcp1' SERVICENODE DEBUG: nids='192.168.202.137@tcp,192.168.202.137@tcp1' SERVICENODE DEBUG: client LNet configuration net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: tcp local NI(s): - nid: 192.168.202.37@tcp status: up interfaces: 0: ens2 - net type: tcp1 local NI(s): - nid: 192.168.202.37@tcp1 status: up interfaces: 0: ens2 SERVICENODE DEBUG: OSS LNet configuration net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: tcp local NI(s): - nid: 192.168.202.137@tcp status: up interfaces: 0: ens2 - net type: tcp1 local NI(s): - nid: 192.168.202.137@tcp1 status: up interfaces: 0: ens2 peer: - primary nid: 192.168.202.137@tcp Multi-Rail: true peer ni: - nid: 192.168.202.137@tcp state: NA - nid: 192.168.202.137@tcp1 state: NA SERVICENODE DEBUG: ping seed=192.168.202.137@tcp present='192.168.202.137@tcp,192.168.202.137@tcp1' absent='' SERVICENODE DEBUG: ping rc=0 ping: - primary nid: 192.168.202.137@tcp Multi-Rail: false peer_ni: - nid: 192.168.202.137@tcp - nid: 192.168.202.137@tcp1 SERVICENODE DEBUG: ping seed=192.168.202.137@tcp1 present='192.168.202.137@tcp,192.168.202.137@tcp1' absent='' SERVICENODE DEBUG: ping rc=0 ping: - primary nid: 192.168.202.137@tcp1 Multi-Rail: false peer_ni: - nid: 192.168.202.137@tcp - nid: 192.168.202.137@tcp1 SERVICENODE DEBUG: discover seed=192.168.202.137@tcp present='192.168.202.137@tcp,192.168.202.137@tcp1' absent='' SERVICENODE DEBUG: discover rc=0 discover: - primary nid: 192.168.202.137@tcp Multi-Rail: true peer_ni: - nid: 192.168.202.137@tcp - nid: 192.168.202.137@tcp1 SERVICENODE DEBUG: peer show rc=0 peer: - primary nid: 192.168.202.137@tcp Multi-Rail: true peer ni: - nid: 192.168.202.137@tcp state: NA - nid: 192.168.202.137@tcp1 state: NA SERVICENODE DEBUG: discover seed=192.168.202.137@tcp1 present='192.168.202.137@tcp,192.168.202.137@tcp1' absent='' SERVICENODE DEBUG: discover rc=0 discover: - primary nid: 192.168.202.137@tcp Multi-Rail: true peer_ni: - nid: 192.168.202.137@tcp - nid: 192.168.202.137@tcp1 SERVICENODE DEBUG: peer show rc=0 peer: - primary nid: 192.168.202.137@tcp Multi-Rail: true peer ni: - nid: 192.168.202.137@tcp state: NA - nid: 192.168.202.137@tcp1 state: NA PASS 57e (111s) == conf-sanity test 57f: servicenode policy overrides NUMA grouping ========================================================== 17:46:05 (1788903965) umount lustre on /mnt/lustre..... stop ost1 service on oleg237-server stop mds service on oleg237-server LNET unconfigure error 22: Invalid argument unloading modules via unload_modules_local on: 'oleg237-server' oleg237-server: oleg237-server.virtnet: executing unload_modules_local oleg237-server: oleg237-server: LNET unconfigure error 22: Invalid argument modules unloaded. setmodopts: MODOPTS_PTLRPC=lbug_on_grant_miscount=1 setmodopts: MODOPTS_PTLRPC=lbug_on_grant_miscount=1 target_nid_policy=servicenode setmodopts: MODOPTS_LNET= lnet_group_numa_nets=tcp,tcp1 SERVICENODE DEBUG: setup base_policy=numa servicenode_policy=enabled SERVICENODE DEBUG: before load MODOPTS_PTLRPC='lbug_on_grant_miscount=1 target_nid_policy=servicenode' SERVICENODE DEBUG: before load MODOPTS_LNET=' lnet_group_numa_nets=tcp,tcp1' /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ../lnet/lnet/lnet options: 'lnet_group_numa_nets=tcp,tcp1' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1 target_nid_policy=servicenode' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg237-server' oleg237-server: oleg237-server.virtnet: executing load_modules_local oleg237-server: Loading modules from /home/green/git/lustre-release/lustre oleg237-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg237-server: detected 4 online CPUs by sysfs oleg237-server: MODOPTS_LIBCFS= oleg237-server: Force libcfs to create 2 CPU partitions oleg237-server: ../lnet/lnet/lnet options: 'lnet_group_numa_nets=tcp,tcp1' oleg237-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1 target_nid_policy=servicenode' oleg237-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg237-server: mdt/mdt options: 'mdt_enable_flr_ec=1' SERVICENODE DEBUG: modules loaded fail_loc=0xe005 fail_val=2 oleg237-server: oleg237-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre oleg237-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg237-client: oleg237-server: ssh exited with exit code 19 oleg237-server: oleg237-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre oleg237-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg237-client: oleg237-server: ssh exited with exit code 19 oleg237-server: oleg237-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre oleg237-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg237-client: oleg237-server: ssh exited with exit code 19 tunefs.lustre failed, reformatting instead Stopping clients: oleg237-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg237-client.virtnet /mnt/lustre2 (opts:-f) oleg237-server: oleg237-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg237-server' oleg237-server: oleg237-server.virtnet: executing load_modules_local oleg237-server: Loading modules from /home/green/git/lustre-release/lustre oleg237-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg237-server: detected 4 online CPUs by sysfs oleg237-server: MODOPTS_LIBCFS= oleg237-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg237-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg237-server: oleg237-server.virtnet: executing set_default_debug -1 all pdsh@oleg237-client: oleg237-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg237-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg237-server: oleg237-server.virtnet: executing set_default_debug -1 all pdsh@oleg237-client: oleg237-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg237-server: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg237-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg237-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg237-server stop mds service on oleg237-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg237-server SERVICENODE DEBUG: writeconf_or_reformat complete oleg237-server: oleg237-server.virtnet: executing lnet_if_list SERVICENODE DEBUG: oss_if='ens2' client_if='ens2' oleg237-server: sh: /usr/sbin/ksocklnd-config: No such file or directory default via 192.168.202.254 dev ens2 192.168.202.0/24 dev ens2 proto kernel scope link src 192.168.202.37 default via 192.168.202.254 dev ens2 192.168.202.0/24 dev ens2 proto kernel scope link src 192.168.202.37 SERVICENODE DEBUG: net=tcp nid='192.168.202.137@tcp' SERVICENODE DEBUG: net=tcp1 nid='192.168.202.137@tcp1' SERVICENODE DEBUG: before temporary target format SERVICENODE DEBUG: HOSTNAME=oleg237-client.virtnet OSS=oleg237-server SERVICENODE DEBUG: MODOPTS_PTLRPC='lbug_on_grant_miscount=1 target_nid_policy=servicenode' SERVICENODE DEBUG: MODOPTS_LNET=' lnet_group_numa_nets=tcp,tcp1' SERVICENODE DEBUG: nets='tcp,tcp1' SERVICENODE DEBUG: nids='192.168.202.137@tcp,192.168.202.137@tcp1' SERVICENODE DEBUG: client LNet configuration net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: tcp local NI(s): - nid: 192.168.202.37@tcp status: up interfaces: 0: ens2 - net type: tcp1 local NI(s): - nid: 192.168.202.37@tcp1 status: up interfaces: 0: ens2 SERVICENODE DEBUG: OSS LNet configuration net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: tcp local NI(s): - nid: 192.168.202.137@tcp status: up interfaces: 0: ens2 - net type: tcp1 local NI(s): - nid: 192.168.202.137@tcp1 status: up interfaces: 0: ens2 peer: - primary nid: 192.168.202.137@tcp Multi-Rail: true peer ni: - nid: 192.168.202.137@tcp state: NA SERVICENODE DEBUG: temporary fsname='sv22044' SERVICENODE DEBUG: temporary OST1 device='lustre-ost1_2/ost1_2' Permanent disk data: Device: lustre-ost1_2/ost1_2 Target: sv22044:OST0000 Index: 0 Lustre FS: sv22044 Mount type: zfs Flags: 0x1062 (OST first_time update no_primnode ) Persistent mount opts: Parameters: mgsnode=192.168.202.137@tcp sys.timeout=20 failover.node=192.168.202.137@tcp,192.168.202.137@tcp1 autodegrade=on mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1_2/ost1_2 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1_2/ost1_2 properties lustre:mgsnode=192.168.202.137@tcp lustre:sys.timeout=20 lustre:failover.node=192.168.202.137@tcp,192.168.202.137@tcp1 lustre:autodegrade=on lustre:version=1 lustre:flags=4194 lustre:index=0 lustre:fsname=sv22044 lustre:svname=sv22044:OST0000 SERVICENODE DEBUG: temporary OST2 device='lustre-ost2_2/ost2_2' Permanent disk data: Device: lustre-ost2_2/ost2_2 Target: sv22044:OST0001 Index: 1 Lustre FS: sv22044 Mount type: zfs Flags: 0x1062 (OST first_time update no_primnode ) Persistent mount opts: Parameters: mgsnode=192.168.202.137@tcp sys.timeout=20 failover.node=192.168.202.137@tcp1 autodegrade=on mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost2_2/ost2_2 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost2_2/ost2_2 properties lustre:mgsnode=192.168.202.137@tcp lustre:sys.timeout=20 lustre:failover.node=192.168.202.137@tcp1 lustre:autodegrade=on lustre:version=1 lustre:flags=4194 lustre:index=1 lustre:fsname=sv22044 lustre:svname=sv22044:OST0001 SERVICENODE DEBUG: starting MGS/MDT and temporary service-node targets start mds service on oleg237-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg237-server: oleg237-server.virtnet: executing set_default_debug -1 all pdsh@oleg237-client: oleg237-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 Start fs2ost1: mount -t lustre -o localrecov lustre-ost1_2/ost1_2 /mnt/lustre-fs2ost1 seq.cli-sv22044-OST0000-super.width=65536 oleg237-server: oleg237-server.virtnet: executing set_default_debug -1 all pdsh@oleg237-client: oleg237-server: ssh exited with exit code 1 Commit the device label on lustre-ost1_2/ost1_2 Started sv22044-OST0000 oleg237-server: missing pool name argument oleg237-server: usage: oleg237-server: trim [-dw] [-r ] [-c | -s] [ ...] pdsh@oleg237-client: oleg237-server: ssh exited with exit code 2 SERVICENODE DEBUG: after first target startup SERVICENODE DEBUG: HOSTNAME=oleg237-client.virtnet OSS=oleg237-server SERVICENODE DEBUG: MODOPTS_PTLRPC='lbug_on_grant_miscount=1 target_nid_policy=servicenode' SERVICENODE DEBUG: MODOPTS_LNET=' lnet_group_numa_nets=tcp,tcp1' SERVICENODE DEBUG: nets='tcp,tcp1' SERVICENODE DEBUG: nids='192.168.202.137@tcp,192.168.202.137@tcp1' SERVICENODE DEBUG: client LNet configuration net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: tcp local NI(s): - nid: 192.168.202.37@tcp status: up interfaces: 0: ens2 - net type: tcp1 local NI(s): - nid: 192.168.202.37@tcp1 status: up interfaces: 0: ens2 SERVICENODE DEBUG: OSS LNet configuration net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: tcp local NI(s): - nid: 192.168.202.137@tcp status: up interfaces: 0: ens2 - net type: tcp1 local NI(s): - nid: 192.168.202.137@tcp1 status: up interfaces: 0: ens2 peer: - primary nid: 192.168.202.137@tcp Multi-Rail: true peer ni: - nid: 192.168.202.137@tcp state: NA - nid: 192.168.202.137@tcp1 state: NA SERVICENODE DEBUG: ping seed=192.168.202.137@tcp present='192.168.202.137@tcp,192.168.202.137@tcp1' absent='' SERVICENODE DEBUG: ping rc=0 ping: - primary nid: 192.168.202.137@tcp Multi-Rail: false peer_ni: - nid: 192.168.202.137@tcp - nid: 192.168.202.137@tcp1 SERVICENODE DEBUG: ping seed=192.168.202.137@tcp1 present='192.168.202.137@tcp,192.168.202.137@tcp1' absent='' SERVICENODE DEBUG: ping rc=0 ping: - primary nid: 192.168.202.137@tcp1 Multi-Rail: false peer_ni: - nid: 192.168.202.137@tcp - nid: 192.168.202.137@tcp1 SERVICENODE DEBUG: discover seed=192.168.202.137@tcp present='192.168.202.137@tcp,192.168.202.137@tcp1' absent='' SERVICENODE DEBUG: discover rc=0 discover: - primary nid: 192.168.202.137@tcp Multi-Rail: true peer_ni: - nid: 192.168.202.137@tcp - nid: 192.168.202.137@tcp1 SERVICENODE DEBUG: peer show rc=0 peer: - primary nid: 192.168.202.137@tcp Multi-Rail: true peer ni: - nid: 192.168.202.137@tcp state: NA - nid: 192.168.202.137@tcp1 state: NA SERVICENODE DEBUG: discover seed=192.168.202.137@tcp1 present='192.168.202.137@tcp,192.168.202.137@tcp1' absent='' SERVICENODE DEBUG: discover rc=0 discover: - primary nid: 192.168.202.137@tcp Multi-Rail: true peer_ni: - nid: 192.168.202.137@tcp - nid: 192.168.202.137@tcp1 SERVICENODE DEBUG: peer show rc=0 peer: - primary nid: 192.168.202.137@tcp Multi-Rail: true peer ni: - nid: 192.168.202.137@tcp state: NA - nid: 192.168.202.137@tcp1 state: NA PASS 57f (111s) == conf-sanity test 57g: servicenode policy keeps ungrouped NIDs discoverable ========================================================== 17:47:56 (1788904076) umount lustre on /mnt/lustre..... stop ost1 service on oleg237-server stop mds service on oleg237-server LNET unconfigure error 22: Invalid argument unloading modules via unload_modules_local on: 'oleg237-server' oleg237-server: oleg237-server.virtnet: executing unload_modules_local oleg237-server: oleg237-server: LNET unconfigure error 22: Invalid argument modules unloaded. setmodopts: MODOPTS_PTLRPC=lbug_on_grant_miscount=1 setmodopts: MODOPTS_PTLRPC=lbug_on_grant_miscount=1 target_nid_policy=servicenode SERVICENODE DEBUG: setup base_policy=none servicenode_policy=enabled SERVICENODE DEBUG: before load MODOPTS_PTLRPC='lbug_on_grant_miscount=1 target_nid_policy=servicenode' SERVICENODE DEBUG: before load MODOPTS_LNET='' /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1 target_nid_policy=servicenode' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg237-server' oleg237-server: oleg237-server.virtnet: executing load_modules_local oleg237-server: Loading modules from /home/green/git/lustre-release/lustre oleg237-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg237-server: detected 4 online CPUs by sysfs oleg237-server: MODOPTS_LIBCFS= oleg237-server: Force libcfs to create 2 CPU partitions oleg237-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1 target_nid_policy=servicenode' oleg237-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg237-server: mdt/mdt options: 'mdt_enable_flr_ec=1' SERVICENODE DEBUG: modules loaded oleg237-server: oleg237-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre oleg237-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg237-client: oleg237-server: ssh exited with exit code 19 oleg237-server: oleg237-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre oleg237-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg237-client: oleg237-server: ssh exited with exit code 19 oleg237-server: oleg237-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre oleg237-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg237-client: oleg237-server: ssh exited with exit code 19 tunefs.lustre failed, reformatting instead Stopping clients: oleg237-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg237-client.virtnet /mnt/lustre2 (opts:-f) oleg237-server: oleg237-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg237-server' oleg237-server: oleg237-server.virtnet: executing load_modules_local oleg237-server: Loading modules from /home/green/git/lustre-release/lustre oleg237-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg237-server: detected 4 online CPUs by sysfs oleg237-server: MODOPTS_LIBCFS= oleg237-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg237-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg237-server: oleg237-server.virtnet: executing set_default_debug -1 all pdsh@oleg237-client: oleg237-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg237-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg237-server: oleg237-server.virtnet: executing set_default_debug -1 all pdsh@oleg237-client: oleg237-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg237-server: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg237-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg237-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg237-server stop mds service on oleg237-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg237-server SERVICENODE DEBUG: writeconf_or_reformat complete oleg237-server: oleg237-server.virtnet: executing lnet_if_list SERVICENODE DEBUG: oss_if='ens2' client_if='ens2' oleg237-server: sh: /usr/sbin/ksocklnd-config: No such file or directory default via 192.168.202.254 dev ens2 192.168.202.0/24 dev ens2 proto kernel scope link src 192.168.202.37 default via 192.168.202.254 dev ens2 192.168.202.0/24 dev ens2 proto kernel scope link src 192.168.202.37 SERVICENODE DEBUG: net=tcp nid='192.168.202.137@tcp' SERVICENODE DEBUG: net=tcp1 nid='192.168.202.137@tcp1' SERVICENODE DEBUG: before temporary target format SERVICENODE DEBUG: HOSTNAME=oleg237-client.virtnet OSS=oleg237-server SERVICENODE DEBUG: MODOPTS_PTLRPC='lbug_on_grant_miscount=1 target_nid_policy=servicenode' SERVICENODE DEBUG: MODOPTS_LNET='' SERVICENODE DEBUG: nets='tcp,tcp1' SERVICENODE DEBUG: nids='192.168.202.137@tcp,192.168.202.137@tcp1' SERVICENODE DEBUG: client LNet configuration net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: tcp local NI(s): - nid: 192.168.202.37@tcp status: up interfaces: 0: ens2 - net type: tcp1 local NI(s): - nid: 192.168.202.37@tcp1 status: up interfaces: 0: ens2 SERVICENODE DEBUG: OSS LNet configuration net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: tcp local NI(s): - nid: 192.168.202.137@tcp status: up interfaces: 0: ens2 - net type: tcp1 local NI(s): - nid: 192.168.202.137@tcp1 status: up interfaces: 0: ens2 peer: - primary nid: 192.168.202.137@tcp Multi-Rail: true peer ni: - nid: 192.168.202.137@tcp state: NA SERVICENODE DEBUG: temporary fsname='sv14987' SERVICENODE DEBUG: temporary OST1 device='lustre-ost1_2/ost1_2' Permanent disk data: Device: lustre-ost1_2/ost1_2 Target: sv14987:OST0000 Index: 0 Lustre FS: sv14987 Mount type: zfs Flags: 0x1062 (OST first_time update no_primnode ) Persistent mount opts: Parameters: mgsnode=192.168.202.137@tcp sys.timeout=20 failover.node=192.168.202.137@tcp autodegrade=on mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1_2/ost1_2 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1_2/ost1_2 properties lustre:mgsnode=192.168.202.137@tcp lustre:sys.timeout=20 lustre:failover.node=192.168.202.137@tcp lustre:autodegrade=on lustre:version=1 lustre:flags=4194 lustre:index=0 lustre:fsname=sv14987 lustre:svname=sv14987:OST0000 SERVICENODE DEBUG: temporary OST2 device='lustre-ost2_2/ost2_2' Permanent disk data: Device: lustre-ost2_2/ost2_2 Target: sv14987:OST0001 Index: 1 Lustre FS: sv14987 Mount type: zfs Flags: 0x1062 (OST first_time update no_primnode ) Persistent mount opts: Parameters: mgsnode=192.168.202.137@tcp sys.timeout=20 failover.node=192.168.202.137@tcp1 autodegrade=on mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost2_2/ost2_2 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost2_2/ost2_2 properties lustre:mgsnode=192.168.202.137@tcp lustre:sys.timeout=20 lustre:failover.node=192.168.202.137@tcp1 lustre:autodegrade=on lustre:version=1 lustre:flags=4194 lustre:index=1 lustre:fsname=sv14987 lustre:svname=sv14987:OST0001 SERVICENODE DEBUG: starting MGS/MDT and temporary service-node targets start mds service on oleg237-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg237-server: oleg237-server.virtnet: executing set_default_debug -1 all pdsh@oleg237-client: oleg237-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 Start fs2ost1: mount -t lustre -o localrecov lustre-ost1_2/ost1_2 /mnt/lustre-fs2ost1 seq.cli-sv14987-OST0000-super.width=65536 oleg237-server: oleg237-server.virtnet: executing set_default_debug -1 all pdsh@oleg237-client: oleg237-server: ssh exited with exit code 1 Commit the device label on lustre-ost1_2/ost1_2 Started sv14987-OST0000 oleg237-server: missing pool name argument oleg237-server: usage: oleg237-server: trim [-dw] [-r ] [-c | -s] [ ...] pdsh@oleg237-client: oleg237-server: ssh exited with exit code 2 SERVICENODE DEBUG: after first target startup SERVICENODE DEBUG: HOSTNAME=oleg237-client.virtnet OSS=oleg237-server SERVICENODE DEBUG: MODOPTS_PTLRPC='lbug_on_grant_miscount=1 target_nid_policy=servicenode' SERVICENODE DEBUG: MODOPTS_LNET='' SERVICENODE DEBUG: nets='tcp,tcp1' SERVICENODE DEBUG: nids='192.168.202.137@tcp,192.168.202.137@tcp1' SERVICENODE DEBUG: client LNet configuration net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: tcp local NI(s): - nid: 192.168.202.37@tcp status: up interfaces: 0: ens2 - net type: tcp1 local NI(s): - nid: 192.168.202.37@tcp1 status: up interfaces: 0: ens2 SERVICENODE DEBUG: OSS LNet configuration net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: tcp local NI(s): - nid: 192.168.202.137@tcp status: up interfaces: 0: ens2 - net type: tcp1 local NI(s): - nid: 192.168.202.137@tcp1 status: up interfaces: 0: ens2 peer: - primary nid: 192.168.202.137@tcp Multi-Rail: true peer ni: - nid: 192.168.202.137@tcp state: NA SERVICENODE DEBUG: ping seed=192.168.202.137@tcp1 present='192.168.202.137@tcp1' absent='192.168.202.137@tcp' SERVICENODE DEBUG: ping rc=0 ping: - primary nid: 192.168.202.137@tcp1 Multi-Rail: false peer_ni: - nid: 192.168.202.137@tcp1 SERVICENODE DEBUG: discover seed=192.168.202.137@tcp1 present='192.168.202.137@tcp1' absent='192.168.202.137@tcp' SERVICENODE DEBUG: discover rc=0 discover: - primary nid: 192.168.202.137@tcp1 Multi-Rail: true peer_ni: - nid: 192.168.202.137@tcp1 SERVICENODE DEBUG: peer show rc=0 peer: - primary nid: 192.168.202.137@tcp1 Multi-Rail: true peer ni: - nid: 192.168.202.137@tcp1 state: NA SERVICENODE DEBUG: ping seed=192.168.202.137@tcp present='192.168.202.137@tcp' absent='192.168.202.137@tcp1' SERVICENODE DEBUG: ping rc=0 ping: - primary nid: 192.168.202.137@tcp Multi-Rail: false peer_ni: - nid: 192.168.202.137@tcp SERVICENODE DEBUG: starting temporary OST2 Start fs3ost2: mount -t lustre -o localrecov lustre-ost2_2/ost2_2 /mnt/lustre-fs3ost2 seq.cli-sv14987-OST0001-super.width=65536 oleg237-server: oleg237-server.virtnet: executing set_default_debug -1 all pdsh@oleg237-client: oleg237-server: ssh exited with exit code 1 Commit the device label on lustre-ost2_2/ost2_2 Started sv14987-OST0001 oleg237-server: missing pool name argument oleg237-server: usage: oleg237-server: trim [-dw] [-r ] [-c | -s] [ ...] pdsh@oleg237-client: oleg237-server: ssh exited with exit code 2 SERVICENODE DEBUG: after second target startup SERVICENODE DEBUG: HOSTNAME=oleg237-client.virtnet OSS=oleg237-server SERVICENODE DEBUG: MODOPTS_PTLRPC='lbug_on_grant_miscount=1 target_nid_policy=servicenode' SERVICENODE DEBUG: MODOPTS_LNET='' SERVICENODE DEBUG: nets='tcp,tcp1' SERVICENODE DEBUG: nids='192.168.202.137@tcp,192.168.202.137@tcp1' SERVICENODE DEBUG: client LNet configuration net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: tcp local NI(s): - nid: 192.168.202.37@tcp status: up interfaces: 0: ens2 - net type: tcp1 local NI(s): - nid: 192.168.202.37@tcp1 status: up interfaces: 0: ens2 peer: - primary nid: 192.168.202.137@tcp1 Multi-Rail: true peer ni: - nid: 192.168.202.137@tcp1 state: NA - primary nid: 192.168.202.137@tcp Multi-Rail: false peer ni: - nid: 192.168.202.137@tcp state: NA SERVICENODE DEBUG: OSS LNet configuration net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: tcp local NI(s): - nid: 192.168.202.137@tcp status: up interfaces: 0: ens2 - net type: tcp1 local NI(s): - nid: 192.168.202.137@tcp1 status: up interfaces: 0: ens2 peer: - primary nid: 192.168.202.137@tcp Multi-Rail: true peer ni: - nid: 192.168.202.137@tcp state: NA peer: - primary nid: 192.168.202.37@tcp1 Multi-Rail: true peer ni: - nid: 192.168.202.37@tcp1 state: NA peer: - primary nid: 192.168.202.37@tcp Multi-Rail: false peer ni: - nid: 192.168.202.37@tcp state: NA SERVICENODE DEBUG: ping seed=192.168.202.137@tcp present='192.168.202.137@tcp' absent='192.168.202.137@tcp1' SERVICENODE DEBUG: ping rc=0 ping: - primary nid: 192.168.202.137@tcp Multi-Rail: false peer_ni: - nid: 192.168.202.137@tcp SERVICENODE DEBUG: ping seed=192.168.202.137@tcp1 present='192.168.202.137@tcp1' absent='192.168.202.137@tcp' SERVICENODE DEBUG: ping rc=0 ping: - primary nid: 192.168.202.137@tcp1 Multi-Rail: true peer_ni: - nid: 192.168.202.137@tcp1 SERVICENODE DEBUG: discover seed=192.168.202.137@tcp present='192.168.202.137@tcp' absent='192.168.202.137@tcp1' SERVICENODE DEBUG: discover rc=0 discover: - primary nid: 192.168.202.137@tcp Multi-Rail: true peer_ni: - nid: 192.168.202.137@tcp SERVICENODE DEBUG: peer show rc=0 peer: - primary nid: 192.168.202.137@tcp Multi-Rail: true peer ni: - nid: 192.168.202.137@tcp state: NA SERVICENODE DEBUG: discover seed=192.168.202.137@tcp1 present='192.168.202.137@tcp1' absent='192.168.202.137@tcp' SERVICENODE DEBUG: discover rc=0 discover: - primary nid: 192.168.202.137@tcp1 Multi-Rail: true peer_ni: - nid: 192.168.202.137@tcp1 SERVICENODE DEBUG: peer show rc=0 peer: - primary nid: 192.168.202.137@tcp1 Multi-Rail: true peer ni: - nid: 192.168.202.137@tcp1 state: NA PASS 57g (133s) == conf-sanity test 57h: servicenode policy is active before the first target mounts ========================================================== 17:50:09 (1788904209) umount lustre on /mnt/lustre..... stop ost1 service on oleg237-server stop mds service on oleg237-server LNET unconfigure error 22: Invalid argument unloading modules via unload_modules_local on: 'oleg237-server' oleg237-server: oleg237-server.virtnet: executing unload_modules_local oleg237-server: oleg237-server: LNET unconfigure error 22: Invalid argument modules unloaded. setmodopts: MODOPTS_PTLRPC=lbug_on_grant_miscount=1 setmodopts: MODOPTS_PTLRPC=lbug_on_grant_miscount=1 target_nid_policy=servicenode SERVICENODE DEBUG: setup base_policy=none servicenode_policy=enabled SERVICENODE DEBUG: before load MODOPTS_PTLRPC='lbug_on_grant_miscount=1 target_nid_policy=servicenode' SERVICENODE DEBUG: before load MODOPTS_LNET='' /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1 target_nid_policy=servicenode' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg237-server' oleg237-server: oleg237-server.virtnet: executing load_modules_local oleg237-server: Loading modules from /home/green/git/lustre-release/lustre oleg237-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg237-server: detected 4 online CPUs by sysfs oleg237-server: MODOPTS_LIBCFS= oleg237-server: Force libcfs to create 2 CPU partitions oleg237-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1 target_nid_policy=servicenode' oleg237-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg237-server: mdt/mdt options: 'mdt_enable_flr_ec=1' SERVICENODE DEBUG: modules loaded oleg237-server: oleg237-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre oleg237-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg237-client: oleg237-server: ssh exited with exit code 19 oleg237-server: oleg237-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre oleg237-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg237-client: oleg237-server: ssh exited with exit code 19 oleg237-server: oleg237-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre oleg237-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg237-client: oleg237-server: ssh exited with exit code 19 tunefs.lustre failed, reformatting instead Stopping clients: oleg237-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg237-client.virtnet /mnt/lustre2 (opts:-f) oleg237-server: oleg237-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg237-server' oleg237-server: oleg237-server.virtnet: executing load_modules_local oleg237-server: Loading modules from /home/green/git/lustre-release/lustre oleg237-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg237-server: detected 4 online CPUs by sysfs oleg237-server: MODOPTS_LIBCFS= oleg237-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg237-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg237-server: oleg237-server.virtnet: executing set_default_debug -1 all pdsh@oleg237-client: oleg237-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg237-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg237-server: oleg237-server.virtnet: executing set_default_debug -1 all pdsh@oleg237-client: oleg237-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg237-server: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg237-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg237-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg237-server stop mds service on oleg237-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg237-server SERVICENODE DEBUG: writeconf_or_reformat complete oleg237-server: oleg237-server.virtnet: executing lnet_if_list SERVICENODE DEBUG: oss_if='ens2' client_if='ens2' oleg237-server: sh: /usr/sbin/ksocklnd-config: No such file or directory default via 192.168.202.254 dev ens2 192.168.202.0/24 dev ens2 proto kernel scope link src 192.168.202.37 default via 192.168.202.254 dev ens2 192.168.202.0/24 dev ens2 proto kernel scope link src 192.168.202.37 SERVICENODE DEBUG: net=tcp nid='192.168.202.137@tcp' SERVICENODE DEBUG: net=tcp1 nid='192.168.202.137@tcp1' SERVICENODE DEBUG: before temporary target format SERVICENODE DEBUG: HOSTNAME=oleg237-client.virtnet OSS=oleg237-server SERVICENODE DEBUG: MODOPTS_PTLRPC='lbug_on_grant_miscount=1 target_nid_policy=servicenode' SERVICENODE DEBUG: MODOPTS_LNET='' SERVICENODE DEBUG: nets='tcp,tcp1' SERVICENODE DEBUG: nids='192.168.202.137@tcp,192.168.202.137@tcp1' SERVICENODE DEBUG: client LNet configuration net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: tcp local NI(s): - nid: 192.168.202.37@tcp status: up interfaces: 0: ens2 - net type: tcp1 local NI(s): - nid: 192.168.202.37@tcp1 status: up interfaces: 0: ens2 SERVICENODE DEBUG: OSS LNet configuration net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: tcp local NI(s): - nid: 192.168.202.137@tcp status: up interfaces: 0: ens2 - net type: tcp1 local NI(s): - nid: 192.168.202.137@tcp1 status: up interfaces: 0: ens2 peer: - primary nid: 192.168.202.137@tcp Multi-Rail: true peer ni: - nid: 192.168.202.137@tcp state: NA SERVICENODE DEBUG: stopping setup after LNet config SERVICENODE DEBUG: ping seed=192.168.202.137@tcp present='192.168.202.137@tcp' absent='192.168.202.137@tcp1' SERVICENODE DEBUG: ping rc=0 ping: - primary nid: 192.168.202.137@tcp Multi-Rail: false peer_ni: - nid: 192.168.202.137@tcp SERVICENODE DEBUG: ping seed=192.168.202.137@tcp1 present='192.168.202.137@tcp1' absent='192.168.202.137@tcp' SERVICENODE DEBUG: ping rc=0 ping: - primary nid: 192.168.202.137@tcp1 Multi-Rail: false peer_ni: - nid: 192.168.202.137@tcp1 SERVICENODE DEBUG: discover seed=192.168.202.137@tcp present='192.168.202.137@tcp' absent='192.168.202.137@tcp1' SERVICENODE DEBUG: discover rc=0 discover: - primary nid: 192.168.202.137@tcp Multi-Rail: true peer_ni: - nid: 192.168.202.137@tcp SERVICENODE DEBUG: peer show rc=0 peer: - primary nid: 192.168.202.137@tcp Multi-Rail: true peer ni: - nid: 192.168.202.137@tcp state: NA SERVICENODE DEBUG: discover seed=192.168.202.137@tcp1 present='192.168.202.137@tcp1' absent='192.168.202.137@tcp' SERVICENODE DEBUG: discover rc=0 discover: - primary nid: 192.168.202.137@tcp1 Multi-Rail: true peer_ni: - nid: 192.168.202.137@tcp1 SERVICENODE DEBUG: peer show rc=0 peer: - primary nid: 192.168.202.137@tcp1 Multi-Rail: true peer ni: - nid: 192.168.202.137@tcp1 state: NA PASS 57h (238s) == conf-sanity test 57i: servicenode Push never falls back to the merged view ========================================================== 17:54:07 (1788904447) umount lustre on /mnt/lustre..... stop ost1 service on oleg237-server stop mds service on oleg237-server LNET unconfigure error 22: Invalid argument unloading modules via unload_modules_local on: 'oleg237-server' oleg237-server: oleg237-server.virtnet: executing unload_modules_local oleg237-server: oleg237-server: LNET unconfigure error 22: Invalid argument modules unloaded. setmodopts: MODOPTS_PTLRPC=lbug_on_grant_miscount=1 setmodopts: MODOPTS_PTLRPC=lbug_on_grant_miscount=1 target_nid_policy=servicenode SERVICENODE DEBUG: setup base_policy=none servicenode_policy=enabled SERVICENODE DEBUG: before load MODOPTS_PTLRPC='lbug_on_grant_miscount=1 target_nid_policy=servicenode' SERVICENODE DEBUG: before load MODOPTS_LNET='' /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1 target_nid_policy=servicenode' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg237-server' oleg237-server: oleg237-server.virtnet: executing load_modules_local oleg237-server: Loading modules from /home/green/git/lustre-release/lustre oleg237-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg237-server: detected 4 online CPUs by sysfs oleg237-server: MODOPTS_LIBCFS= oleg237-server: Force libcfs to create 2 CPU partitions oleg237-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1 target_nid_policy=servicenode' oleg237-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg237-server: mdt/mdt options: 'mdt_enable_flr_ec=1' SERVICENODE DEBUG: modules loaded oleg237-server: oleg237-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre oleg237-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg237-client: oleg237-server: ssh exited with exit code 19 oleg237-server: oleg237-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre oleg237-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg237-client: oleg237-server: ssh exited with exit code 19 oleg237-server: oleg237-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre oleg237-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg237-client: oleg237-server: ssh exited with exit code 19 tunefs.lustre failed, reformatting instead Stopping clients: oleg237-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg237-client.virtnet /mnt/lustre2 (opts:-f) oleg237-server: oleg237-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg237-server' oleg237-server: oleg237-server.virtnet: executing load_modules_local oleg237-server: Loading modules from /home/green/git/lustre-release/lustre oleg237-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg237-server: detected 4 online CPUs by sysfs oleg237-server: MODOPTS_LIBCFS= oleg237-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg237-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg237-server: oleg237-server.virtnet: executing set_default_debug -1 all pdsh@oleg237-client: oleg237-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg237-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg237-server: oleg237-server.virtnet: executing set_default_debug -1 all pdsh@oleg237-client: oleg237-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg237-server: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg237-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg237-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg237-server stop mds service on oleg237-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg237-server SERVICENODE DEBUG: writeconf_or_reformat complete oleg237-server: oleg237-server.virtnet: executing lnet_if_list SERVICENODE DEBUG: oss_if='ens2' client_if='ens2' oleg237-server: sh: /usr/sbin/ksocklnd-config: No such file or directory default via 192.168.202.254 dev ens2 192.168.202.0/24 dev ens2 proto kernel scope link src 192.168.202.37 default via 192.168.202.254 dev ens2 192.168.202.0/24 dev ens2 proto kernel scope link src 192.168.202.37 oleg237-server: sh: /usr/sbin/ksocklnd-config: No such file or directory default via 192.168.202.254 dev ens2 192.168.202.0/24 dev ens2 proto kernel scope link src 192.168.202.37 default via 192.168.202.254 dev ens2 192.168.202.0/24 dev ens2 proto kernel scope link src 192.168.202.37 default via 192.168.202.254 dev ens2 192.168.202.0/24 dev ens2 proto kernel scope link src 192.168.202.37 SERVICENODE DEBUG: net=tcp nid='192.168.202.137@tcp' SERVICENODE DEBUG: net=tcp1 nid='192.168.202.137@tcp1' SERVICENODE DEBUG: net=tcp2 nid='192.168.202.137@tcp2' SERVICENODE DEBUG: before temporary target format SERVICENODE DEBUG: HOSTNAME=oleg237-client.virtnet OSS=oleg237-server SERVICENODE DEBUG: MODOPTS_PTLRPC='lbug_on_grant_miscount=1 target_nid_policy=servicenode' SERVICENODE DEBUG: MODOPTS_LNET='' SERVICENODE DEBUG: nets='tcp,tcp1,tcp2' SERVICENODE DEBUG: nids='192.168.202.137@tcp,192.168.202.137@tcp1,192.168.202.137@tcp2' SERVICENODE DEBUG: client LNet configuration net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: tcp local NI(s): - nid: 192.168.202.37@tcp status: up interfaces: 0: ens2 - net type: tcp1 local NI(s): - nid: 192.168.202.37@tcp1 status: up interfaces: 0: ens2 - net type: tcp2 local NI(s): - nid: 192.168.202.37@tcp2 status: up interfaces: 0: ens2 SERVICENODE DEBUG: OSS LNet configuration net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: tcp local NI(s): - nid: 192.168.202.137@tcp status: up interfaces: 0: ens2 - net type: tcp1 local NI(s): - nid: 192.168.202.137@tcp1 status: up interfaces: 0: ens2 - net type: tcp2 local NI(s): - nid: 192.168.202.137@tcp2 status: up interfaces: 0: ens2 peer: - primary nid: 192.168.202.137@tcp Multi-Rail: true peer ni: - nid: 192.168.202.137@tcp state: NA SERVICENODE DEBUG: temporary fsname='sv29460' SERVICENODE DEBUG: temporary OST1 device='lustre-ost1_2/ost1_2' Permanent disk data: Device: lustre-ost1_2/ost1_2 Target: sv29460:OST0000 Index: 0 Lustre FS: sv29460 Mount type: zfs Flags: 0x1062 (OST first_time update no_primnode ) Persistent mount opts: Parameters: mgsnode=192.168.202.137@tcp sys.timeout=20 failover.node=192.168.202.137@tcp,192.168.202.137@tcp1 autodegrade=on mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1_2/ost1_2 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1_2/ost1_2 properties lustre:mgsnode=192.168.202.137@tcp lustre:sys.timeout=20 lustre:failover.node=192.168.202.137@tcp,192.168.202.137@tcp1 lustre:autodegrade=on lustre:version=1 lustre:flags=4194 lustre:index=0 lustre:fsname=sv29460 lustre:svname=sv29460:OST0000 SERVICENODE DEBUG: temporary OST2 device='lustre-ost2_2/ost2_2' Permanent disk data: Device: lustre-ost2_2/ost2_2 Target: sv29460:OST0001 Index: 1 Lustre FS: sv29460 Mount type: zfs Flags: 0x1062 (OST first_time update no_primnode ) Persistent mount opts: Parameters: mgsnode=192.168.202.137@tcp sys.timeout=20 failover.node=192.168.202.137@tcp autodegrade=on mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost2_2/ost2_2 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost2_2/ost2_2 properties lustre:mgsnode=192.168.202.137@tcp lustre:sys.timeout=20 lustre:failover.node=192.168.202.137@tcp lustre:autodegrade=on lustre:version=1 lustre:flags=4194 lustre:index=1 lustre:fsname=sv29460 lustre:svname=sv29460:OST0001 SERVICENODE DEBUG: starting MGS/MDT and temporary service-node targets start mds service on oleg237-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg237-server: oleg237-server.virtnet: executing set_default_debug -1 all pdsh@oleg237-client: oleg237-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 Start fs2ost1: mount -t lustre -o localrecov lustre-ost1_2/ost1_2 /mnt/lustre-fs2ost1 seq.cli-sv29460-OST0000-super.width=65536 oleg237-server: oleg237-server.virtnet: executing set_default_debug -1 all pdsh@oleg237-client: oleg237-server: ssh exited with exit code 1 Commit the device label on lustre-ost1_2/ost1_2 Started sv29460-OST0000 oleg237-server: missing pool name argument oleg237-server: usage: oleg237-server: trim [-dw] [-r ] [-c | -s] [ ...] pdsh@oleg237-client: oleg237-server: ssh exited with exit code 2 SERVICENODE DEBUG: after first target startup SERVICENODE DEBUG: HOSTNAME=oleg237-client.virtnet OSS=oleg237-server SERVICENODE DEBUG: MODOPTS_PTLRPC='lbug_on_grant_miscount=1 target_nid_policy=servicenode' SERVICENODE DEBUG: MODOPTS_LNET='' SERVICENODE DEBUG: nets='tcp,tcp1,tcp2' SERVICENODE DEBUG: nids='192.168.202.137@tcp,192.168.202.137@tcp1,192.168.202.137@tcp2' SERVICENODE DEBUG: client LNet configuration net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: tcp local NI(s): - nid: 192.168.202.37@tcp status: up interfaces: 0: ens2 - net type: tcp1 local NI(s): - nid: 192.168.202.37@tcp1 status: up interfaces: 0: ens2 - net type: tcp2 local NI(s): - nid: 192.168.202.37@tcp2 status: up interfaces: 0: ens2 SERVICENODE DEBUG: OSS LNet configuration net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: tcp local NI(s): - nid: 192.168.202.137@tcp status: up interfaces: 0: ens2 - net type: tcp1 local NI(s): - nid: 192.168.202.137@tcp1 status: up interfaces: 0: ens2 - net type: tcp2 local NI(s): - nid: 192.168.202.137@tcp2 status: up interfaces: 0: ens2 peer: - primary nid: 192.168.202.137@tcp Multi-Rail: true peer ni: - nid: 192.168.202.137@tcp state: NA - nid: 192.168.202.137@tcp1 state: NA peer: - primary nid: 192.168.202.137@tcp2 Multi-Rail: true peer ni: - nid: 192.168.202.137@tcp2 state: NA SERVICENODE DEBUG: discover seed=192.168.202.137@tcp present='192.168.202.137@tcp' absent='192.168.202.137@tcp2' SERVICENODE DEBUG: discover rc=0 discover: - primary nid: 192.168.202.137@tcp Multi-Rail: true peer_ni: - nid: 192.168.202.137@tcp - nid: 192.168.202.137@tcp1 SERVICENODE DEBUG: peer show rc=0 peer: - primary nid: 192.168.202.137@tcp Multi-Rail: true peer ni: - nid: 192.168.202.137@tcp state: NA - nid: 192.168.202.137@tcp1 state: NA SERVICENODE DEBUG: starting temporary OST2 Start fs3ost2: mount -t lustre -o localrecov lustre-ost2_2/ost2_2 /mnt/lustre-fs3ost2 seq.cli-sv29460-OST0001-super.width=65536 oleg237-server: oleg237-server.virtnet: executing set_default_debug -1 all pdsh@oleg237-client: oleg237-server: ssh exited with exit code 1 Commit the device label on lustre-ost2_2/ost2_2 Started sv29460-OST0001 oleg237-server: missing pool name argument oleg237-server: usage: oleg237-server: trim [-dw] [-r ] [-c | -s] [ ...] pdsh@oleg237-client: oleg237-server: ssh exited with exit code 2 SERVICENODE DEBUG: after second target startup SERVICENODE DEBUG: HOSTNAME=oleg237-client.virtnet OSS=oleg237-server SERVICENODE DEBUG: MODOPTS_PTLRPC='lbug_on_grant_miscount=1 target_nid_policy=servicenode' SERVICENODE DEBUG: MODOPTS_LNET='' SERVICENODE DEBUG: nets='tcp,tcp1,tcp2' SERVICENODE DEBUG: nids='192.168.202.137@tcp,192.168.202.137@tcp1,192.168.202.137@tcp2' SERVICENODE DEBUG: client LNet configuration net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: tcp local NI(s): - nid: 192.168.202.37@tcp status: up interfaces: 0: ens2 - net type: tcp1 local NI(s): - nid: 192.168.202.37@tcp1 status: up interfaces: 0: ens2 - net type: tcp2 local NI(s): - nid: 192.168.202.37@tcp2 status: up interfaces: 0: ens2 peer: - primary nid: 192.168.202.137@tcp Multi-Rail: true peer ni: - nid: 192.168.202.137@tcp state: NA - nid: 192.168.202.137@tcp1 state: NA SERVICENODE DEBUG: OSS LNet configuration net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: tcp local NI(s): - nid: 192.168.202.137@tcp status: up interfaces: 0: ens2 - net type: tcp1 local NI(s): - nid: 192.168.202.137@tcp1 status: up interfaces: 0: ens2 - net type: tcp2 local NI(s): - nid: 192.168.202.137@tcp2 status: up interfaces: 0: ens2 peer: - primary nid: 192.168.202.137@tcp Multi-Rail: true peer ni: - nid: 192.168.202.137@tcp state: NA - nid: 192.168.202.137@tcp1 state: NA peer: - primary nid: 192.168.202.137@tcp2 Multi-Rail: true peer ni: - nid: 192.168.202.137@tcp2 state: NA peer: - primary nid: 192.168.202.37@tcp Multi-Rail: true peer ni: - nid: 192.168.202.37@tcp state: NA SERVICENODE DEBUG: ping seed=192.168.202.137@tcp present='192.168.202.137@tcp' absent='192.168.202.137@tcp2' SERVICENODE DEBUG: ping rc=0 ping: - primary nid: 192.168.202.137@tcp Multi-Rail: true peer_ni: - nid: 192.168.202.137@tcp - nid: 192.168.202.137@tcp1 PASS 57i (305s) == conf-sanity test 57j: failnode targets contribute no servicenode group ========================================================== 17:59:12 (1788904752) umount lustre on /mnt/lustre..... stop ost1 service on oleg237-server stop mds service on oleg237-server LNET unconfigure error 22: Invalid argument unloading modules via unload_modules_local on: 'oleg237-server' oleg237-server: oleg237-server.virtnet: executing unload_modules_local oleg237-server: oleg237-server: LNET unconfigure error 22: Invalid argument modules unloaded. setmodopts: MODOPTS_PTLRPC=lbug_on_grant_miscount=1 setmodopts: MODOPTS_PTLRPC=lbug_on_grant_miscount=1 target_nid_policy=servicenode SERVICENODE DEBUG: setup base_policy=none servicenode_policy=enabled SERVICENODE DEBUG: before load MODOPTS_PTLRPC='lbug_on_grant_miscount=1 target_nid_policy=servicenode' SERVICENODE DEBUG: before load MODOPTS_LNET='' /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1 target_nid_policy=servicenode' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg237-server' oleg237-server: oleg237-server.virtnet: executing load_modules_local oleg237-server: Loading modules from /home/green/git/lustre-release/lustre oleg237-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg237-server: detected 4 online CPUs by sysfs oleg237-server: MODOPTS_LIBCFS= oleg237-server: Force libcfs to create 2 CPU partitions oleg237-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1 target_nid_policy=servicenode' oleg237-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg237-server: mdt/mdt options: 'mdt_enable_flr_ec=1' SERVICENODE DEBUG: modules loaded oleg237-server: oleg237-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre oleg237-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg237-client: oleg237-server: ssh exited with exit code 19 oleg237-server: oleg237-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre oleg237-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg237-client: oleg237-server: ssh exited with exit code 19 oleg237-server: oleg237-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre oleg237-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg237-client: oleg237-server: ssh exited with exit code 19 tunefs.lustre failed, reformatting instead Stopping clients: oleg237-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg237-client.virtnet /mnt/lustre2 (opts:-f) oleg237-server: oleg237-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg237-server' oleg237-server: oleg237-server.virtnet: executing load_modules_local oleg237-server: Loading modules from /home/green/git/lustre-release/lustre oleg237-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg237-server: detected 4 online CPUs by sysfs oleg237-server: MODOPTS_LIBCFS= oleg237-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg237-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg237-server: oleg237-server.virtnet: executing set_default_debug -1 all pdsh@oleg237-client: oleg237-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg237-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg237-server: oleg237-server.virtnet: executing set_default_debug -1 all pdsh@oleg237-client: oleg237-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg237-server: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg237-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg237-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg237-server stop mds service on oleg237-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg237-server SERVICENODE DEBUG: writeconf_or_reformat complete oleg237-server: oleg237-server.virtnet: executing lnet_if_list SERVICENODE DEBUG: oss_if='ens2' client_if='ens2' oleg237-server: sh: /usr/sbin/ksocklnd-config: No such file or directory default via 192.168.202.254 dev ens2 192.168.202.0/24 dev ens2 proto kernel scope link src 192.168.202.37 default via 192.168.202.254 dev ens2 192.168.202.0/24 dev ens2 proto kernel scope link src 192.168.202.37 SERVICENODE DEBUG: net=tcp nid='192.168.202.137@tcp' SERVICENODE DEBUG: net=tcp1 nid='192.168.202.137@tcp1' SERVICENODE DEBUG: before temporary target format SERVICENODE DEBUG: HOSTNAME=oleg237-client.virtnet OSS=oleg237-server SERVICENODE DEBUG: MODOPTS_PTLRPC='lbug_on_grant_miscount=1 target_nid_policy=servicenode' SERVICENODE DEBUG: MODOPTS_LNET='' SERVICENODE DEBUG: nets='tcp,tcp1' SERVICENODE DEBUG: nids='192.168.202.137@tcp,192.168.202.137@tcp1' SERVICENODE DEBUG: client LNet configuration net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: tcp local NI(s): - nid: 192.168.202.37@tcp status: up interfaces: 0: ens2 - net type: tcp1 local NI(s): - nid: 192.168.202.37@tcp1 status: up interfaces: 0: ens2 SERVICENODE DEBUG: OSS LNet configuration net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: tcp local NI(s): - nid: 192.168.202.137@tcp status: up interfaces: 0: ens2 - net type: tcp1 local NI(s): - nid: 192.168.202.137@tcp1 status: up interfaces: 0: ens2 peer: - primary nid: 192.168.202.137@tcp Multi-Rail: true peer ni: - nid: 192.168.202.137@tcp state: NA SERVICENODE DEBUG: temporary fsname='sv10701' SERVICENODE DEBUG: temporary OST1 device='lustre-ost1_2/ost1_2' Permanent disk data: Device: lustre-ost1_2/ost1_2 Target: sv10701:OST0000 Index: 0 Lustre FS: sv10701 Mount type: zfs Flags: 0x62 (OST first_time update ) Persistent mount opts: Parameters: mgsnode=192.168.202.137@tcp sys.timeout=20 failover.node=192.168.202.37@tcp autodegrade=on mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1_2/ost1_2 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1_2/ost1_2 properties lustre:mgsnode=192.168.202.137@tcp lustre:sys.timeout=20 lustre:failover.node=192.168.202.37@tcp lustre:autodegrade=on lustre:version=1 lustre:flags=98 lustre:index=0 lustre:fsname=sv10701 lustre:svname=sv10701:OST0000 SERVICENODE DEBUG: temporary OST2 device='lustre-ost2_2/ost2_2' Permanent disk data: Device: lustre-ost2_2/ost2_2 Target: sv10701:OST0001 Index: 1 Lustre FS: sv10701 Mount type: zfs Flags: 0x62 (OST first_time update ) Persistent mount opts: Parameters: mgsnode=192.168.202.137@tcp sys.timeout=20 failover.node=192.168.202.37@tcp autodegrade=on mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost2_2/ost2_2 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost2_2/ost2_2 properties lustre:mgsnode=192.168.202.137@tcp lustre:sys.timeout=20 lustre:failover.node=192.168.202.37@tcp lustre:autodegrade=on lustre:version=1 lustre:flags=98 lustre:index=1 lustre:fsname=sv10701 lustre:svname=sv10701:OST0001 SERVICENODE DEBUG: starting MGS/MDT and temporary service-node targets start mds service on oleg237-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg237-server: oleg237-server.virtnet: executing set_default_debug -1 all pdsh@oleg237-client: oleg237-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 Start fs2ost1: mount -t lustre -o localrecov lustre-ost1_2/ost1_2 /mnt/lustre-fs2ost1 seq.cli-sv10701-OST0000-super.width=65536 oleg237-server: oleg237-server.virtnet: executing set_default_debug -1 all pdsh@oleg237-client: oleg237-server: ssh exited with exit code 1 Commit the device label on lustre-ost1_2/ost1_2 Started sv10701-OST0000 oleg237-server: missing pool name argument oleg237-server: usage: oleg237-server: trim [-dw] [-r ] [-c | -s] [ ...] pdsh@oleg237-client: oleg237-server: ssh exited with exit code 2 Start fs3ost2: mount -t lustre -o localrecov lustre-ost2_2/ost2_2 /mnt/lustre-fs3ost2 seq.cli-sv10701-OST0001-super.width=65536 oleg237-server: oleg237-server.virtnet: executing set_default_debug -1 all pdsh@oleg237-client: oleg237-server: ssh exited with exit code 1 Commit the device label on lustre-ost2_2/ost2_2 Started sv10701-OST0001 oleg237-server: missing pool name argument oleg237-server: usage: oleg237-server: trim [-dw] [-r ] [-c | -s] [ ...] pdsh@oleg237-client: oleg237-server: ssh exited with exit code 2 SERVICENODE DEBUG: after target startup SERVICENODE DEBUG: HOSTNAME=oleg237-client.virtnet OSS=oleg237-server SERVICENODE DEBUG: MODOPTS_PTLRPC='lbug_on_grant_miscount=1 target_nid_policy=servicenode' SERVICENODE DEBUG: MODOPTS_LNET='' SERVICENODE DEBUG: nets='tcp,tcp1' SERVICENODE DEBUG: nids='192.168.202.137@tcp,192.168.202.137@tcp1' SERVICENODE DEBUG: client LNet configuration net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: tcp local NI(s): - nid: 192.168.202.37@tcp status: up interfaces: 0: ens2 - net type: tcp1 local NI(s): - nid: 192.168.202.37@tcp1 status: up interfaces: 0: ens2 SERVICENODE DEBUG: OSS LNet configuration net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: tcp local NI(s): - nid: 192.168.202.137@tcp status: up interfaces: 0: ens2 - net type: tcp1 local NI(s): - nid: 192.168.202.137@tcp1 status: up interfaces: 0: ens2 peer: - primary nid: 192.168.202.137@tcp Multi-Rail: true peer ni: - nid: 192.168.202.137@tcp1 state: NA - nid: 192.168.202.137@tcp state: NA SERVICENODE DEBUG: ping seed=192.168.202.137@tcp present='192.168.202.137@tcp' absent='192.168.202.137@tcp1' SERVICENODE DEBUG: ping rc=0 ping: - primary nid: 192.168.202.137@tcp Multi-Rail: false peer_ni: - nid: 192.168.202.137@tcp SERVICENODE DEBUG: ping seed=192.168.202.137@tcp1 present='192.168.202.137@tcp1' absent='192.168.202.137@tcp' SERVICENODE DEBUG: ping rc=0 ping: - primary nid: 192.168.202.137@tcp1 Multi-Rail: false peer_ni: - nid: 192.168.202.137@tcp1 SERVICENODE DEBUG: discover seed=192.168.202.137@tcp present='192.168.202.137@tcp' absent='192.168.202.137@tcp1' SERVICENODE DEBUG: discover rc=0 discover: - primary nid: 192.168.202.137@tcp Multi-Rail: true peer_ni: - nid: 192.168.202.137@tcp SERVICENODE DEBUG: peer show rc=0 peer: - primary nid: 192.168.202.137@tcp Multi-Rail: true peer ni: - nid: 192.168.202.137@tcp state: NA PASS 57j (192s) == conf-sanity test 57k: unknown target_nid_policy stops the module load ========================================================== 18:02:24 (1788904944) umount lustre on /mnt/lustre..... stop ost1 service on oleg237-server stop mds service on oleg237-server LNET unconfigure error 22: Invalid argument unloading modules via unload_modules_local on: 'oleg237-server' oleg237-server: oleg237-server.virtnet: executing unload_modules_local oleg237-server: oleg237-server: LNET unconfigure error 22: Invalid argument modules unloaded. LNET unconfigure error 22: Invalid argument unloading modules via unload_modules_local on: 'oleg237-server' oleg237-server: oleg237-server.virtnet: executing unload_modules_local oleg237-server: oleg237-server: LNET unconfigure error 22: Invalid argument modules unloaded. oleg237-server: modprobe: FATAL: Module ptlrpc not found in directory /lib/modules/4.18.0rh8.10-debug pdsh@oleg237-client: oleg237-server: ssh exited with exit code 1 pdsh@oleg237-client: oleg237-server: ssh exited with exit code 1 pdsh@oleg237-client: oleg237-server: ssh exited with exit code 2 pdsh@oleg237-client: oleg237-client: ssh exited with exit code 2 pdsh@oleg237-client: oleg237-client: ssh exited with exit code 2 pdsh@oleg237-client: oleg237-server: ssh exited with exit code 2 LNET unconfigure error 22: Invalid argument unloading modules via unload_modules_local on: 'oleg237-server' oleg237-server: oleg237-server.virtnet: executing unload_modules_local oleg237-server: oleg237-server: LNET unconfigure error 22: Invalid argument modules unloaded. PASS 57k (16s) == conf-sanity test 57ka: IR READY respects servicenode target NID policy ========================================================== 18:02:40 (1788904960) umount lustre on /mnt/lustre..... stop ost1 service on oleg237-server stop mds service on oleg237-server LNET unconfigure error 22: Invalid argument unloading modules via unload_modules_local on: 'oleg237-server' oleg237-server: oleg237-server.virtnet: executing unload_modules_local oleg237-server: oleg237-server: LNET unconfigure error 22: Invalid argument modules unloaded. setmodopts: MODOPTS_PTLRPC=lbug_on_grant_miscount=1 setmodopts: MODOPTS_PTLRPC=lbug_on_grant_miscount=1 target_nid_policy=servicenode SERVICENODE DEBUG: setup base_policy=none servicenode_policy=enabled SERVICENODE DEBUG: before load MODOPTS_PTLRPC='lbug_on_grant_miscount=1 target_nid_policy=servicenode' SERVICENODE DEBUG: before load MODOPTS_LNET='' /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1 target_nid_policy=servicenode' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg237-server' oleg237-server: oleg237-server.virtnet: executing load_modules_local oleg237-server: Loading modules from /home/green/git/lustre-release/lustre oleg237-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg237-server: detected 4 online CPUs by sysfs oleg237-server: MODOPTS_LIBCFS= oleg237-server: Force libcfs to create 2 CPU partitions oleg237-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1 target_nid_policy=servicenode' oleg237-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg237-server: mdt/mdt options: 'mdt_enable_flr_ec=1' SERVICENODE DEBUG: modules loaded oleg237-server: oleg237-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre oleg237-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg237-client: oleg237-server: ssh exited with exit code 19 oleg237-server: oleg237-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre oleg237-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg237-client: oleg237-server: ssh exited with exit code 19 oleg237-server: oleg237-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre oleg237-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg237-client: oleg237-server: ssh exited with exit code 19 tunefs.lustre failed, reformatting instead Stopping clients: oleg237-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg237-client.virtnet /mnt/lustre2 (opts:-f) oleg237-server: oleg237-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg237-server' oleg237-server: oleg237-server.virtnet: executing load_modules_local oleg237-server: Loading modules from /home/green/git/lustre-release/lustre oleg237-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg237-server: detected 4 online CPUs by sysfs oleg237-server: MODOPTS_LIBCFS= oleg237-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg237-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg237-server: oleg237-server.virtnet: executing set_default_debug -1 all pdsh@oleg237-client: oleg237-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg237-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg237-server: oleg237-server.virtnet: executing set_default_debug -1 all pdsh@oleg237-client: oleg237-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg237-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg237-server: oleg237-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg237-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg237-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg237-server stop mds service on oleg237-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg237-server SERVICENODE DEBUG: writeconf_or_reformat complete oleg237-server: oleg237-server.virtnet: executing lnet_if_list SERVICENODE DEBUG: oss_if='ens2' client_if='ens2' oleg237-server: sh: /usr/sbin/ksocklnd-config: No such file or directory default via 192.168.202.254 dev ens2 192.168.202.0/24 dev ens2 proto kernel scope link src 192.168.202.37 default via 192.168.202.254 dev ens2 192.168.202.0/24 dev ens2 proto kernel scope link src 192.168.202.37 SERVICENODE DEBUG: net=tcp nid='192.168.202.137@tcp' SERVICENODE DEBUG: net=tcp1 nid='192.168.202.137@tcp1' SERVICENODE DEBUG: before temporary target format SERVICENODE DEBUG: HOSTNAME=oleg237-client.virtnet OSS=oleg237-server SERVICENODE DEBUG: MODOPTS_PTLRPC='lbug_on_grant_miscount=1 target_nid_policy=servicenode' SERVICENODE DEBUG: MODOPTS_LNET='' SERVICENODE DEBUG: nets='tcp,tcp1' SERVICENODE DEBUG: nids='192.168.202.137@tcp,192.168.202.137@tcp1' SERVICENODE DEBUG: client LNet configuration net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: tcp local NI(s): - nid: 192.168.202.37@tcp status: up interfaces: 0: ens2 - net type: tcp1 local NI(s): - nid: 192.168.202.37@tcp1 status: up interfaces: 0: ens2 SERVICENODE DEBUG: OSS LNet configuration net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: tcp local NI(s): - nid: 192.168.202.137@tcp status: up interfaces: 0: ens2 - net type: tcp1 local NI(s): - nid: 192.168.202.137@tcp1 status: up interfaces: 0: ens2 peer: - primary nid: 192.168.202.137@tcp Multi-Rail: true peer ni: - nid: 192.168.202.137@tcp state: NA SERVICENODE DEBUG: temporary fsname='sv1019' SERVICENODE DEBUG: temporary OST1 device='lustre-ost1_2/ost1_2' Permanent disk data: Device: lustre-ost1_2/ost1_2 Target: sv1019:OST0000 Index: 0 Lustre FS: sv1019 Mount type: zfs Flags: 0x1062 (OST first_time update no_primnode ) Persistent mount opts: Parameters: mgsnode=192.168.202.137@tcp sys.timeout=20 failover.node=192.168.202.137@tcp autodegrade=on mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1_2/ost1_2 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1_2/ost1_2 properties lustre:mgsnode=192.168.202.137@tcp lustre:sys.timeout=20 lustre:failover.node=192.168.202.137@tcp lustre:autodegrade=on lustre:version=1 lustre:flags=4194 lustre:index=0 lustre:fsname=sv1019 lustre:svname=sv1019:OST0000 SERVICENODE DEBUG: temporary OST2 device='lustre-ost2_2/ost2_2' Permanent disk data: Device: lustre-ost2_2/ost2_2 Target: sv1019:OST0001 Index: 1 Lustre FS: sv1019 Mount type: zfs Flags: 0x1062 (OST first_time update no_primnode ) Persistent mount opts: Parameters: mgsnode=192.168.202.137@tcp sys.timeout=20 failover.node=192.168.202.137@tcp1 autodegrade=on mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost2_2/ost2_2 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost2_2/ost2_2 properties lustre:mgsnode=192.168.202.137@tcp lustre:sys.timeout=20 lustre:failover.node=192.168.202.137@tcp1 lustre:autodegrade=on lustre:version=1 lustre:flags=4194 lustre:index=1 lustre:fsname=sv1019 lustre:svname=sv1019:OST0001 SERVICENODE DEBUG: starting MGS/MDT and temporary service-node targets start mds service on oleg237-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg237-server: oleg237-server.virtnet: executing set_default_debug -1 all pdsh@oleg237-client: oleg237-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg237-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 Start fs2ost1: mount -t lustre -o localrecov lustre-ost1_2/ost1_2 /mnt/lustre-fs2ost1 seq.cli-sv1019-OST0000-super.width=65536 oleg237-server: oleg237-server.virtnet: executing set_default_debug -1 all pdsh@oleg237-client: oleg237-server: ssh exited with exit code 1 Commit the device label on lustre-ost1_2/ost1_2 Started sv1019-OST0000 oleg237-server: missing pool name argument oleg237-server: usage: oleg237-server: trim [-dw] [-r ] [-c | -s] [ ...] pdsh@oleg237-client: oleg237-server: ssh exited with exit code 2 SERVICENODE DEBUG: after first target startup SERVICENODE DEBUG: HOSTNAME=oleg237-client.virtnet OSS=oleg237-server SERVICENODE DEBUG: MODOPTS_PTLRPC='lbug_on_grant_miscount=1 target_nid_policy=servicenode' SERVICENODE DEBUG: MODOPTS_LNET='' SERVICENODE DEBUG: nets='tcp,tcp1' SERVICENODE DEBUG: nids='192.168.202.137@tcp,192.168.202.137@tcp1' SERVICENODE DEBUG: client LNet configuration net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: tcp local NI(s): - nid: 192.168.202.37@tcp status: up interfaces: 0: ens2 - net type: tcp1 local NI(s): - nid: 192.168.202.37@tcp1 status: up interfaces: 0: ens2 SERVICENODE DEBUG: OSS LNet configuration net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: tcp local NI(s): - nid: 192.168.202.137@tcp status: up interfaces: 0: ens2 - net type: tcp1 local NI(s): - nid: 192.168.202.137@tcp1 status: up interfaces: 0: ens2 peer: - primary nid: 192.168.202.137@tcp Multi-Rail: true peer ni: - nid: 192.168.202.137@tcp state: NA Stopping /mnt/lustre-fs2ost1 (opts:-f) on oleg237-server fail_val=1 fail_loc=0x80000727 Start fs2ost1: mount -t lustre -o localrecov lustre-ost1_2/ost1_2 /mnt/lustre-fs2ost1 seq.cli-sv1019-OST0000-super.width=65536 oleg237-server: oleg237-server.virtnet: executing set_default_debug -1 all pdsh@oleg237-client: oleg237-server: ssh exited with exit code 1 Started sv1019-OST0000 fail_loc=0 fail_val=0 PASS 57ka (134s) Stopping clients: oleg237-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg237-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg237-client: oleg237-server: ssh exited with exit code 2 oleg237-server: oleg237-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg237-server' oleg237-server: oleg237-server.virtnet: executing load_modules_local oleg237-server: Loading modules from /home/green/git/lustre-release/lustre oleg237-server: detected 4 online CPUs by sysfs oleg237-server: MODOPTS_LIBCFS= oleg237-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg237-server: Force libcfs to create 2 CPU partitions oleg237-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg237-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg237-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 1911 sec ========== 18:05:24 (1788905124) === conf-sanity: start cleanup 18:05:24 (1788905124) === === conf-sanity: finish cleanup 18:05:25 (1788905125) ===