== conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 12:00:08 (1773676808) checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.150@tcp sys.timeout=20 Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.150@tcp sys.timeout=20 Comment: just a comment for tail oleg350-server: tunefs.lustre: params are too long: oleg350-server: mgsnode=192.168.203.150@tcp:111.222.173.1@tcp,111.222.142.1@tcp:111.222.173.2@tcp,111.222.142.2@tcp:111.222.173.3@tcp,111.222.142.3@tcp:111.222.173.4@tcp,111.222.142.4@tcp:111.222.173.5@tcp,111.222.142.5@tcp:111.222.173.6@tcp,111.222.142.6@tcp:111.222.173.7@tcp,111.222.142.7@tcp:111.222.173.8@tcp,111.222.142.8@tcp:111.222.173.9@tcp,111.222.142.9@tcp:111.222.173.10@tcp,111.222.142.10@tcp:111.222.173.11@tcp,111.222.142.11@tcp:111.222.173.12@tcp,111.222.142.12@tcp:111.222.173.13@tcp,111.222.142.13@tcp:111.222.173.14@tcp,111.222.142.14@tcp:111.222.173.15@tcp,111.222.142.15@tcp:111.222.173.16@tcp,111.222.142.16@tcp:111.222.173.17@tcp,111.222.142.17@tcp:111.222.173.18@tcp,111.222.142.18@tcp:111.222.173.19@tcp,111.222.142.19@tcp:111.222.173.20@tcp,111.222.142.20@tcp:111.222.173.21@tcp,111.222.142.21@tcp:111.222.173.22@tcp,111.222.142.22@tcp:111.222.173.23@tcp,111.222.142.23@tcp:111.222.173.24@tcp,111.222.142.24@tcp:111.222.173.25@tcp,111.222.142.25@tcp:111.222.173.26@tcp,111.222.142.26@tcp:111.222.173.27@tcp,111.222.142.27@tcp:111.222.173.28@tcp,111.222.142.28@tcp:111.222.173.29@tcp,111.222.142.29@tcp:111.222.173.30@tcp,111.222.142.30@tcp:111.222.173.31@tcp,111.222.142.31@tcp:111.222.173.32@tcp,111.222.142.32@tcp:111.222.173.33@tcp,111.222.142.33@tcp:111.222.173.34@tcp,111.222.142.34@tcp:111.222.173.35@tcp,111.222.142.35@tcp:111.222.173.36@tcp,111.222.142.36@tcp:111.222.173.37@tcp,111.222.142.37@tcp:111.222.173.38@tcp,111.222.142.38@tcp:111.222.173.39@tcp,111.222.142.39@tcp:111.222.173.40@tcp,111.222.142.40@tcp:111.222.173.41@tcp,111.222.142.41@tcp:111.222.173.42@tcp,111.222.142.42@tcp:111.222.173.43@tcp,111.222.142.43@tcp:111.222.173.44@tcp,111.222.142.44@tcp:111.222.173.45@tcp,111.222.142.45@tcp:111.222.173.46@tcp,111.222.142.46@tcp:111.222.173.47@tcp,111.222.142.47@tcp:111.222.173.48@tcp,111.222.142.48@tcp:111.222.173.49@tcp,111.222.142.49@tcp:111.222.173.50@tcp,111.222.142.50@tcp sys.timeout=20 failover.node=111.222.173.1@tcp,111.222.142.1@tcp:111.222.173.2@tcp,111.222.142.2@tcp:111.222.173.3@tcp,111.222.142.3@tcp:111.222.173.4@tcp,111.222.142.4@tcp:111.222.173.5@tcp,111.222.142.5@tcp:111.222.173.6@tcp,111.222.142.6@tcp:111.222.173.7@tcp,111.222.142.7@tcp:111.222.173.8@tcp,111.222.142.8@tcp:111.222.173.9@tcp,111.222.142.9@tcp:111.222.173.10@tcp,111.222.142.10@tcp:111.222.173.11@tcp,111.222.142.11@tcp:111.222.173.12@tcp,111.222.142.12@tcp:111.222.173.13@tcp,111.222.142.13@tcp:111.222.173.14@tcp,111.222.142.14@tcp:111.222.173.15@tcp,111.222.142.15@tcp:111.222.173.16@tcp,111.222.142.16@tcp:111.222.173.17@tcp,111.222.142.17@tcp:111.222.173.18@tcp,111.222.142.18@tcp:111.222.173.19@tcp,111.222.142.19@tcp:111.222.173.20@tcp,111.222.142.20@tcp:111.222.173.21@tcp,111.222.142.21@tcp:111.222.173.22@tcp,111.222.142.22@tcp:111.222.173.23@tcp,111.222.142.23@tcp:111.222.173.24@tcp,111.222.142.24@tcp:111.222.173.25@tcp,111.222.142.25@tcp:111.222.173.26@tcp,111.222.142.26@tcp:111.222.173.27@tcp,111.222.142.27@tcp:111.222.173.28@tcp,111.222.142.28@tcp:111.222.173.29@tcp,111.222.142.29@tcp:111.222.173.30@tcp,111.222.142.30@tcp:111.222.173.31@tcp,111.222.142.31@tcp:111.222.173.32@tcp,111.222.142.32@tcp:111.222.173.33@tcp,111.222.142.33@tcp:111.222.173.34@tcp,111.222.142.34@tcp:111.222.173.35@tcp,111.222.142.35@tcp:111.222.173.36@tcp,111.222.142.36@tcp:111.222.173.37@tcp,111.222.142.37@tcp:111.222.173.38@tcp,111.222.142.38@tcp:111.222.173.39@tcp,111.222.142.39@tcp:111.222.173.40@tcp,111.222.142.40@tcp:111.222.173.41@tcp,111.222.142.41@tcp:111.222.173.42@tcp,111.222.142.42@tcp:111.222.173.43@tcp,111.222.142.43@tcp:111.222.173.44@tcp,111.222.142.44@tcp:111.222.173.45@tcp,111.222.142.45@tcp:111.222.173.46@tcp,111.222.142.46@tcp:111.222.173.47@tcp,111.222.142.47@tcp:111.222.173.48@tcp,111.222.142.48@tcp:111.222.173.49@tcp,111.222.142.49@tcp:111.222.173.50@tcp,111.222.142.50@tcp +failover.node==111.222.173.51@tcp,111.222.142.51@tcp oleg350-server: tunefs.lustre: exiting with 7 (Argument list too long) pdsh@oleg350-client: oleg350-server: ssh exited with exit code 7 start mds service on oleg350-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg350-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg350-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Started lustre-OST0000 oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid stop ost1 service on oleg350-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg350-server stop mds service on oleg350-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg350-server stop mds service on oleg350-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg350-server Found total 201 NIDs in parameters Stopping clients: oleg350-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg350-client.virtnet /mnt/lustre2 (opts:-f) oleg350-server: oleg350-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg350-server' oleg350-server: oleg350-server.virtnet: executing load_modules_local oleg350-server: Loading modules from /home/green/git/lustre-release/lustre oleg350-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg350-server: detected 4 online CPUs by sysfs oleg350-server: MODOPTS_LIBCFS= oleg350-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey