-----============= acceptance-small: conf-sanity ============----- Thu Apr 23 21:47:57 EDT 2026 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg222-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex': No such file or directory excepting tests: 102 106 115 32newtarball 108a 112a 112b 113 117 119 121 122a 123aa 123ab 123ac 123ad 123ae 123af 123ag 123ah 123ahi 123F 123G 123H 126 129 132 133 135 136 137 150 152 153a 153b 153c 155 802a skipping tests SLOW=no: 45 69 106 111 114 Stopping clients: oleg222-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg222-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg222-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg222-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg222-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg222-server oleg222-server: oleg222-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg222-server' oleg222-server: oleg222-server.virtnet: executing load_modules_local oleg222-server: Loading modules from /home/green/git/lustre-release/lustre oleg222-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg222-server: detected 4 online CPUs by sysfs oleg222-server: MODOPTS_LIBCFS= oleg222-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg222-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg222-server: oleg222-server.virtnet: executing set_default_debug -1 all pdsh@oleg222-client: oleg222-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg222-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg222-server: oleg222-server.virtnet: executing set_default_debug -1 all pdsh@oleg222-client: oleg222-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg222-server: oleg222-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg222-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg222-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg222-server stop mds service on oleg222-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg222-server == conf-sanity test 76a: set permanent params with lctl across mounts ========================================================== 21:50:37 (1776995437) start mds service on oleg222-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg222-server: oleg222-server.virtnet: executing set_default_debug -1 all pdsh@oleg222-client: oleg222-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg222-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg222-server: oleg222-server.virtnet: executing set_default_debug -1 all pdsh@oleg222-client: oleg222-server: ssh exited with exit code 1 Started lustre-OST0000 oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg222-client.virtnet: -o user_xattr,flock 192.168.202.122@tcp:/lustre /mnt/lustre Change MGS params max_dirty_mb: 462 new_max_dirty_mb: 452 Waiting 90s for '452' Updated after 2s: want '452' got '452' 452 Check the value is stored after remount Stopping clients: oleg222-client.virtnet /mnt/lustre (opts:) Stopping client oleg222-client.virtnet /mnt/lustre opts: Stopping clients: oleg222-client.virtnet /mnt/lustre2 (opts:) Stopping /mnt/lustre-mds1 (opts:-f) on oleg222-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg222-server Checking servers environments Checking clients oleg222-client.virtnet environments /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg222-server' oleg222-server: oleg222-server.virtnet: executing load_modules_local oleg222-server: Loading modules from /home/green/git/lustre-release/lustre oleg222-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg222-server: detected 4 online CPUs by sysfs oleg222-server: MODOPTS_LIBCFS= oleg222-server: Force libcfs to create 2 CPU partitions Setup mgs, mdt, osts Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg222-server: oleg222-server.virtnet: executing set_default_debug -1 all pdsh@oleg222-client: oleg222-server: ssh exited with exit code 1 Started lustre-MDT0000 Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg222-server: oleg222-server.virtnet: executing set_default_debug -1 all pdsh@oleg222-client: oleg222-server: ssh exited with exit code 1 Started lustre-OST0000 Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg222-server: oleg222-server.virtnet: executing set_default_debug -1 all pdsh@oleg222-client: oleg222-server: ssh exited with exit code 1 Commit the device label on lustre-ost2/ost2 Started lustre-OST0001 mount lustre on /mnt/lustre..... Starting client: oleg222-client.virtnet: -o user_xattr,flock 192.168.202.122@tcp:/lustre /mnt/lustre Starting client oleg222-client.virtnet: -o user_xattr,flock 192.168.202.122@tcp:/lustre /mnt/lustre Started clients oleg222-client.virtnet: 192.168.202.122@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff91fc07c0e800.idle_timeout=debug osc.lustre-OST0001-osc-ffff91fc07c0e800.idle_timeout=debug disable quota as required Change OST params client_cache_count: 128 new_client_cache_count: 256 Waiting 90s for '256' Updated after 2s: want '256' got '256' 256 Check the value is stored after remount Stopping clients: oleg222-client.virtnet /mnt/lustre (opts:) Stopping client oleg222-client.virtnet /mnt/lustre opts: Stopping clients: oleg222-client.virtnet /mnt/lustre2 (opts:) Stopping /mnt/lustre-mds1 (opts:-f) on oleg222-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg222-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg222-server Checking servers environments Checking clients oleg222-client.virtnet environments /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg222-server' oleg222-server: oleg222-server.virtnet: executing load_modules_local oleg222-server: Loading modules from /home/green/git/lustre-release/lustre oleg222-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg222-server: detected 4 online CPUs by sysfs oleg222-server: MODOPTS_LIBCFS= oleg222-server: Force libcfs to create 2 CPU partitions Setup mgs, mdt, osts Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg222-server: oleg222-server.virtnet: executing set_default_debug -1 all pdsh@oleg222-client: oleg222-server: ssh exited with exit code 1 Started lustre-MDT0000 Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg222-server: oleg222-server.virtnet: executing set_default_debug -1 all pdsh@oleg222-client: oleg222-server: ssh exited with exit code 1 Started lustre-OST0000 Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg222-server: oleg222-server.virtnet: executing set_default_debug -1 all pdsh@oleg222-client: oleg222-server: ssh exited with exit code 1 Started lustre-OST0001 mount lustre on /mnt/lustre..... Starting client: oleg222-client.virtnet: -o user_xattr,flock 192.168.202.122@tcp:/lustre /mnt/lustre Starting client oleg222-client.virtnet: -o user_xattr,flock 192.168.202.122@tcp:/lustre /mnt/lustre Started clients oleg222-client.virtnet: 192.168.202.122@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff91fc1885e800.idle_timeout=debug osc.lustre-OST0001-osc-ffff91fc1885e800.idle_timeout=debug disable quota as required 256 Stopping clients: oleg222-client.virtnet /mnt/lustre (opts:) Stopping client oleg222-client.virtnet /mnt/lustre opts: Stopping clients: oleg222-client.virtnet /mnt/lustre2 (opts:) Stopping /mnt/lustre-mds1 (opts:-f) on oleg222-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg222-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg222-server PASS 76a (263s) == conf-sanity test 76b: verify params log setup correctly ========================================================== 21:55:00 (1776995700) Stopping clients: oleg222-client.virtnet /mnt/lustre (opts:) Stopping clients: oleg222-client.virtnet /mnt/lustre2 (opts:) Checking servers environments Checking clients oleg222-client.virtnet environments /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg222-server' oleg222-server: oleg222-server.virtnet: executing load_modules_local oleg222-server: Loading modules from /home/green/git/lustre-release/lustre oleg222-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg222-server: detected 4 online CPUs by sysfs oleg222-server: MODOPTS_LIBCFS= oleg222-server: Force libcfs to create 2 CPU partitions Setup mgs, mdt, osts Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg222-server: oleg222-server.virtnet: executing set_default_debug -1 all pdsh@oleg222-client: oleg222-server: ssh exited with exit code 1 Started lustre-MDT0000 Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg222-server: oleg222-server.virtnet: executing set_default_debug -1 all pdsh@oleg222-client: oleg222-server: ssh exited with exit code 1 Started lustre-OST0000 Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg222-server: oleg222-server.virtnet: executing set_default_debug -1 all pdsh@oleg222-client: oleg222-server: ssh exited with exit code 1 Started lustre-OST0001 mount lustre on /mnt/lustre..... Starting client: oleg222-client.virtnet: -o user_xattr,flock 192.168.202.122@tcp:/lustre /mnt/lustre Starting client oleg222-client.virtnet: -o user_xattr,flock 192.168.202.122@tcp:/lustre /mnt/lustre Started clients oleg222-client.virtnet: 192.168.202.122@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff91fc012af000.idle_timeout=debug osc.lustre-OST0001-osc-ffff91fc012af000.idle_timeout=debug disable quota as required mgs.MGS.live.params= fsname: params flags: 0x20 gen: 1 Secure RPC Config Rules: imperative_recovery_state: state: startup nonir_clients: 0 nidtbl_version: 2 notify_duration_total: 0.000000000 notify_duation_max: 0.000000000 notify_count: 0 Stopping clients: oleg222-client.virtnet /mnt/lustre (opts:) Stopping client oleg222-client.virtnet /mnt/lustre opts: Stopping clients: oleg222-client.virtnet /mnt/lustre2 (opts:) Stopping /mnt/lustre-mds1 (opts:-f) on oleg222-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg222-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg222-server PASS 76b (123s) == conf-sanity test 76c: verify changelog_mask is applied with lctl set_param -P ========================================================== 21:57:03 (1776995823) Checking servers environments Checking clients oleg222-client.virtnet environments /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg222-server' oleg222-server: oleg222-server.virtnet: executing load_modules_local oleg222-server: Loading modules from /home/green/git/lustre-release/lustre oleg222-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg222-server: detected 4 online CPUs by sysfs oleg222-server: MODOPTS_LIBCFS= oleg222-server: Force libcfs to create 2 CPU partitions Setup mgs, mdt, osts Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg222-server: oleg222-server.virtnet: executing set_default_debug -1 all pdsh@oleg222-client: oleg222-server: ssh exited with exit code 1 Started lustre-MDT0000 Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg222-server: oleg222-server.virtnet: executing set_default_debug -1 all pdsh@oleg222-client: oleg222-server: ssh exited with exit code 1 Started lustre-OST0000 Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg222-server: oleg222-server.virtnet: executing set_default_debug -1 all pdsh@oleg222-client: oleg222-server: ssh exited with exit code 1 Started lustre-OST0001 mount lustre on /mnt/lustre..... Starting client: oleg222-client.virtnet: -o user_xattr,flock 192.168.202.122@tcp:/lustre /mnt/lustre Starting client oleg222-client.virtnet: -o user_xattr,flock 192.168.202.122@tcp:/lustre /mnt/lustre Started clients oleg222-client.virtnet: 192.168.202.122@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff91fc08e21800.idle_timeout=debug osc.lustre-OST0001-osc-ffff91fc08e21800.idle_timeout=debug disable quota as required Change changelog_mask pdsh@oleg222-client: oleg222-server: ssh exited with exit code 1 Check the value is stored after mds remount stop mds service on oleg222-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg222-server start mds service on oleg222-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg222-server: oleg222-server.virtnet: executing set_default_debug -1 all pdsh@oleg222-client: oleg222-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Stopping clients: oleg222-client.virtnet /mnt/lustre (opts:) Stopping client oleg222-client.virtnet /mnt/lustre opts: Stopping clients: oleg222-client.virtnet /mnt/lustre2 (opts:) Stopping /mnt/lustre-mds1 (opts:-f) on oleg222-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg222-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg222-server PASS 76c (133s) == conf-sanity test 76d: verify llite.*.xattr_cache can be set by 'lctl set_param -P' correctly ========================================================== 21:59:17 (1776995957) Checking servers environments Checking clients oleg222-client.virtnet environments /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg222-server' oleg222-server: oleg222-server.virtnet: executing load_modules_local oleg222-server: Loading modules from /home/green/git/lustre-release/lustre oleg222-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg222-server: detected 4 online CPUs by sysfs oleg222-server: MODOPTS_LIBCFS= oleg222-server: Force libcfs to create 2 CPU partitions Setup mgs, mdt, osts Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg222-server: oleg222-server.virtnet: executing set_default_debug -1 all pdsh@oleg222-client: oleg222-server: ssh exited with exit code 1 Started lustre-MDT0000 Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg222-server: oleg222-server.virtnet: executing set_default_debug -1 all pdsh@oleg222-client: oleg222-server: ssh exited with exit code 1 Started lustre-OST0000 Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg222-server: oleg222-server.virtnet: executing set_default_debug -1 all pdsh@oleg222-client: oleg222-server: ssh exited with exit code 1 Started lustre-OST0001 mount lustre on /mnt/lustre..... Starting client: oleg222-client.virtnet: -o user_xattr,flock 192.168.202.122@tcp:/lustre /mnt/lustre Starting client oleg222-client.virtnet: -o user_xattr,flock 192.168.202.122@tcp:/lustre /mnt/lustre Started clients oleg222-client.virtnet: 192.168.202.122@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff91fc08e24800.idle_timeout=debug osc.lustre-OST0001-osc-ffff91fc08e24800.idle_timeout=debug disable quota as required lctl set_param -P llite.*.xattr_cache=0 Waiting 90s for '0' Updated after 2s: want '0' got '0' Check llite.$inst.xattr_cache on client /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg222-client.virtnet /mnt/lustre (opts:) mount lustre on /mnt/lustre..... Starting client: oleg222-client.virtnet: -o user_xattr,flock 192.168.202.122@tcp:/lustre /mnt/lustre Check llite.$inst.xattr_cache on the new client /mnt/lustre2 mount lustre on /mnt/lustre2..... Starting client: oleg222-client.virtnet: -o user_xattr,flock 192.168.202.122@tcp:/lustre /mnt/lustre2 umount lustre on /mnt/lustre2..... Stopping client oleg222-client.virtnet /mnt/lustre2 (opts:) Stopping clients: oleg222-client.virtnet /mnt/lustre (opts:) Stopping client oleg222-client.virtnet /mnt/lustre opts: Stopping clients: oleg222-client.virtnet /mnt/lustre2 (opts:) Stopping /mnt/lustre-mds1 (opts:-f) on oleg222-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg222-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg222-server oleg222-server: No device found for name MGS: Invalid argument oleg222-server: This command must be run on the MGS. oleg222-server: error: executing set_param: No such device pdsh@oleg222-client: oleg222-server: ssh exited with exit code 19 Reader error: 'No devices found' at 0 PASS 76d (118s) == conf-sanity test 77: comma-separated MGS NIDs and failover node NIDs ========================================================== 22:01:15 (1776996075) pdsh@oleg222-client: oleg222-server: ssh exited with exit code 1 Permanent disk data: Target: test1234:MDT0000 Index: 0 Lustre FS: test1234 Mount type: zfs Flags: 0x65 (MDT MGS first_time update ) Persistent mount opts: Parameters: mgsnode=192.168.202.122@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-mdt1_2 /tmp/lustre-mdt1_2 mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-mdt1_2/mdt1_2 xattr=sa dnodesize=auto compression=off Writing lustre-mdt1_2/mdt1_2 properties lustre:mgsnode=192.168.202.122@tcp lustre:sys.timeout=20 lustre:mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity lustre:version=1 lustre:flags=101 lustre:index=0 lustre:fsname=test1234 lustre:svname=test1234:MDT0000 Start fs2mds: mount -t lustre -o localrecov lustre-mdt1_2/mdt1_2 /mnt/lustre-fs2mds oleg222-server: oleg222-server.virtnet: executing set_default_debug -1 all pdsh@oleg222-client: oleg222-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1_2/mdt1_2 Started test1234-MDT0000 Permanent disk data: Target: test1234:OST0000 Index: 0 Lustre FS: test1234 Mount type: zfs Flags: 0x62 (OST first_time update ) Persistent mount opts: Parameters: mgsnode=0.0.0.0@tcp,192.168.202.122@tcp,192.168.202.122@tcp:192.168.202.122@tcp:192.168.202.122@tcp sys.timeout=20 failover.node=1.2.3.4@tcp,4.3.2.1@tcp autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1_2 /tmp/lustre-ost1_2 mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1_2/ost1_2 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1_2/ost1_2 properties lustre:mgsnode=0.0.0.0@tcp,192.168.202.122@tcp,192.168.202.122@tcp:192.168.202.122@tcp:192.168.202.122@tcp lustre:sys.timeout=20 lustre:failover.node=1.2.3.4@tcp,4.3.2.1@tcp lustre:autodegrade=on lustre:version=1 lustre:flags=98 lustre:index=0 lustre:fsname=test1234 lustre:svname=test1234:OST0000 /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1839: OST_INDICES: bad array subscript Start fs2ost: mount -t lustre -o localrecov lustre-ost1_2/ost1_2 /mnt/lustre-fs2ost seq.cli-test1234-OST0000-super.width=65536 oleg222-server: oleg222-server.virtnet: executing set_default_debug -1 all pdsh@oleg222-client: oleg222-server: ssh exited with exit code 1 Commit the device label on lustre-ost1_2/ost1_2 Started test1234-OST0000 oleg222-server: fstrim: /mnt/lustre-fs2ost: the discard operation is not supported pdsh@oleg222-client: oleg222-server: ssh exited with exit code 1 setup single mount lustre success umount /mnt/lustre2 ... stopping fs2mds ... Stopping /mnt/lustre-fs2mds (opts:-f) on oleg222-server stopping fs2ost ... Stopping /mnt/lustre-fs2ost (opts:-f) on oleg222-server PASS 77 (74s) == conf-sanity test 78: run resize2fs on MDT and OST filesystems ========================================================== 22:02:30 (1776996150) SKIP: conf-sanity test_78 ldiskfs only test SKIP 78 (3s) == conf-sanity test 79: format MDT/OST without mgs option (should return errors) ========================================================== 22:02:33 (1776996153) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg222-server' oleg222-server: oleg222-server.virtnet: executing load_modules_local oleg222-server: Loading modules from /home/green/git/lustre-release/lustre oleg222-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg222-server: detected 4 online CPUs by sysfs oleg222-server: MODOPTS_LIBCFS= oleg222-server: Force libcfs to create 2 CPU partitions oleg222-server: oleg222-server: mkfs.lustre FATAL: Must specify --mgs or --mgsnode oleg222-server: mkfs.lustre: exiting with 22 (Invalid argument) pdsh@oleg222-client: oleg222-server: ssh exited with exit code 22 oleg222-server: oleg222-server: mkfs.lustre FATAL: Must specify --mgs or --mgsnode oleg222-server: mkfs.lustre: exiting with 22 (Invalid argument) pdsh@oleg222-client: oleg222-server: ssh exited with exit code 22 oleg222-server: oleg222-server: mkfs.lustre FATAL: Must specify --mgsnode oleg222-server: mkfs.lustre: exiting with 22 (Invalid argument) pdsh@oleg222-client: oleg222-server: ssh exited with exit code 22 Stopping clients: oleg222-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg222-client.virtnet /mnt/lustre2 (opts:-f) oleg222-server: oleg222-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg222-server' oleg222-server: oleg222-server.virtnet: executing load_modules_local oleg222-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg222-server: Loading modules from /home/green/git/lustre-release/lustre oleg222-server: detected 4 online CPUs by sysfs oleg222-server: MODOPTS_LIBCFS= oleg222-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg222-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg222-server: oleg222-server.virtnet: executing set_default_debug -1 all pdsh@oleg222-client: oleg222-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg222-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg222-server: oleg222-server.virtnet: executing set_default_debug -1 all pdsh@oleg222-client: oleg222-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg222-server: oleg222-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg222-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg222-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg222-server stop mds service on oleg222-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg222-server PASS 79 (156s) == conf-sanity test 80: mgc import reconnect race ======== 22:05:09 (1776996309) start mds service on oleg222-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg222-server: oleg222-server.virtnet: executing set_default_debug -1 all pdsh@oleg222-client: oleg222-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg222-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg222-server: oleg222-server.virtnet: executing set_default_debug -1 all pdsh@oleg222-client: oleg222-server: ssh exited with exit code 1 Started lustre-OST0000 oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid fail_val=10 fail_loc=0x906 fail_val=10 fail_loc=0x906 start ost2 service on oleg222-server Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg222-server: oleg222-server.virtnet: executing set_default_debug -1 all pdsh@oleg222-client: oleg222-server: ssh exited with exit code 1 Commit the device label on lustre-ost2/ost2 Started lustre-OST0001 oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid fail_loc=0 stop ost2 service on oleg222-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg222-server stop ost1 service on oleg222-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg222-server stop mds service on oleg222-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg222-server PASS 80 (128s) == conf-sanity test 81: sparse OST indexing ============== 22:07:17 (1776996437) SKIP: conf-sanity test_81 needs >= 3 OSTs SKIP 81 (3s) == conf-sanity test 82a: specify OSTs for file (succeed) or directory (succeed) ========================================================== 22:07:20 (1776996440) SKIP: conf-sanity test_82a needs >= 3 OSTs SKIP 82a (3s) == conf-sanity test 82b: specify OSTs for file with --pool and --ost-list options ========================================================== 22:07:23 (1776996443) SKIP: conf-sanity test_82b needs >= 4 OSTs SKIP 82b (4s) == conf-sanity test 82c: specify sparse OSTs for setstripe (should not crash) ========================================================== 22:07:27 (1776996447) SKIP: conf-sanity test_82c needs >= 4 OSTs SKIP 82c (3s) == conf-sanity test 83: ENOSPACE on OST doesn't cause message VFS: Busy inodes after unmount ... ========================================================== 22:07:30 (1776996450) SKIP: conf-sanity test_83 ldiskfs only test SKIP 83 (4s) == conf-sanity test 84: check recovery_hard_time ========= 22:07:34 (1776996454) start mds service on oleg222-server start mds service on oleg222-server Start mds1: mount -t lustre -o localrecov -o recovery_time_hard=60,recovery_time_soft=60 lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg222-server: oleg222-server.virtnet: executing set_default_debug -1 all pdsh@oleg222-client: oleg222-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg222-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg222-server: oleg222-server.virtnet: executing set_default_debug -1 all pdsh@oleg222-client: oleg222-server: ssh exited with exit code 1 Started lustre-OST0000 oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg222-server: oleg222-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg222-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec start ost2 service on oleg222-server Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg222-server: oleg222-server.virtnet: executing set_default_debug -1 all pdsh@oleg222-client: oleg222-server: ssh exited with exit code 1 Started lustre-OST0001 oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid oleg222-server: oleg222-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 oleg222-server: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec recovery_time=60, timeout=20, wrap_up=5 mount lustre on /mnt/lustre..... Starting client: oleg222-client.virtnet: -o user_xattr,flock 192.168.202.122@tcp:/lustre /mnt/lustre mount lustre on /mnt/lustre2..... Starting client: oleg222-client.virtnet: -o user_xattr,flock 192.168.202.122@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 383616 2048 379520 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 382976 2048 378880 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 382976 2048 378880 1% /mnt/lustre[OST:1] filesystem_summary: 765952 4096 757760 1% /mnt/lustre - open/close 467 (time 1776996542.59 total 10.02 last 46.61) - open/close 977 (time 1776996552.60 total 20.03 last 50.94) total: 1000 open/close in 20.43 seconds: 48.95 ops/second fail_loc=0x20000709 fail_val=5 Failing mds1 on oleg222-server Stopping /mnt/lustre-mds1 (opts:) on oleg222-server 22:09:20 (1776996560) shut down facet: mds1 facet_host: oleg222-server facet_failover_host: oleg222-server Failover mds1 to oleg222-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov -o recovery_time_hard=60,recovery_time_soft=60 lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg222-server: oleg222-server.virtnet: executing set_default_debug -1 all pdsh@oleg222-client: oleg222-server: ssh exited with exit code 1 Started lustre-MDT0000 22:09:43 (1776996583) targets are mounted 22:09:44 (1776996584) facet_failover done oleg222-client: error: invalid path '/mnt/lustre': Input/output error pdsh@oleg222-client: oleg222-client: ssh exited with exit code 5 recovery status status: COMPLETE recovery_start: 1776996577 recovery_duration: 60 completed_clients: 1/2 replayed_requests: 189 last_transno: 12884902081 VBR: DISABLED IR: DISABLED fail_loc=0 umount lustre on /mnt/lustre..... Stopping client oleg222-client.virtnet /mnt/lustre (opts:) umount lustre on /mnt/lustre2..... Stopping client oleg222-client.virtnet /mnt/lustre2 (opts:) stop ost1 service on oleg222-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg222-server stop ost2 service on oleg222-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg222-server stop mds service on oleg222-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg222-server PASS 84 (220s) == conf-sanity test 85: osd_ost init: fail ea_fid_set ==== 22:11:14 (1776996674) fail_loc=0x197 start ost1 service on oleg222-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg222-server: oleg222-server.virtnet: executing set_default_debug -1 all pdsh@oleg222-client: oleg222-server: ssh exited with exit code 1 Started lustre-OST0000 oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid stop ost1 service on oleg222-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg222-server PASS 85 (76s) == conf-sanity test 86: Replacing mkfs.lustre -G option == 22:12:31 (1776996751) SKIP: conf-sanity test_86 LU-6442: no such mkfs params for ZFS OSTs SKIP 86 (3s) == conf-sanity test 87: check if MDT inode can hold EAs with N stripes properly ========================================================== 22:12:33 (1776996753) SKIP: conf-sanity test_87 ldiskfs only test SKIP 87 (4s) == conf-sanity test 88: check the default mount options can be overridden ========================================================== 22:12:37 (1776996757) SKIP: conf-sanity test_88 LU-6662: no implementation for ZFS SKIP 88 (3s) == conf-sanity test 89: check tunefs --param and --erase-param{s} options ========================================================== 22:12:40 (1776996760) Failing mds1 on oleg222-server 22:12:45 (1776996765) shut down facet: mds1 facet_host: oleg222-server facet_failover_host: oleg222-server Failover mds1 to oleg222-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg222-server: oleg222-server.virtnet: executing set_default_debug -1 all pdsh@oleg222-client: oleg222-server: ssh exited with exit code 1 Started lustre-MDT0000 22:13:07 (1776996787) targets are mounted 22:13:07 (1776996787) facet_failover done pdsh@oleg222-client: oleg222-client: ssh exited with exit code 95 oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid pdsh@oleg222-client: oleg222-client: ssh exited with exit code 95 Stopping clients: oleg222-client.virtnet /mnt/lustre (opts:) Stopping clients: oleg222-client.virtnet /mnt/lustre2 (opts:) Stopping /mnt/lustre-mds1 (opts:-f) on oleg222-server tunefs --param failover.node=192.0.2.254@tcp0 tunefs --param failover.node=192.0.2.255@tcp0 tunefs --erase-param failover.node tunefs --erase-params tunefs --param failover.node=192.0.2.254@tcp0 --erase-params Stopping clients: oleg222-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg222-client.virtnet /mnt/lustre2 (opts:-f) oleg222-server: oleg222-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg222-server' oleg222-server: oleg222-server.virtnet: executing load_modules_local oleg222-server: Loading modules from /home/green/git/lustre-release/lustre oleg222-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg222-server: detected 4 online CPUs by sysfs oleg222-server: MODOPTS_LIBCFS= oleg222-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg222-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg222-server: oleg222-server.virtnet: executing set_default_debug -1 all pdsh@oleg222-client: oleg222-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg222-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg222-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg222-server: oleg222-server.virtnet: executing set_default_debug -1 all pdsh@oleg222-client: oleg222-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) failover.node=192.0.2.254@tcp0,osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg222-server: oleg222-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg222-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg222-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg222-server stop mds service on oleg222-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg222-server PASS 89 (197s) == conf-sanity test 90a: check max_mod_rpcs_in_flight is enforced ========================================================== 22:15:57 (1776996957) start mds service on oleg222-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg222-server: oleg222-server.virtnet: executing set_default_debug -1 all pdsh@oleg222-client: oleg222-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg222-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg222-server: oleg222-server.virtnet: executing set_default_debug -1 all pdsh@oleg222-client: oleg222-server: ssh exited with exit code 1 Started lustre-OST0000 oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg222-client.virtnet: -o user_xattr,flock 192.168.202.122@tcp:/lustre /mnt/lustre max_mod_rcps_in_flight is 7 creating 8 files ... fail_loc=0x159 launch 6 chmod in parallel ... fail_loc=0 launch 1 additional chmod in parallel ... /mnt/lustre/d90a.conf-sanity/file-7 has perms 0600 OK fail_loc=0x159 launch 7 chmod in parallel ... fail_loc=0 launch 1 additional chmod in parallel ... /mnt/lustre/d90a.conf-sanity/file-8 has perms 0644 OK umount lustre on /mnt/lustre..... Stopping client oleg222-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg222-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg222-server stop mds service on oleg222-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg222-server unloading modules via unload_modules_local on: 'oleg222-server' oleg222-server: oleg222-server.virtnet: executing unload_modules_local oleg222-server: modules unloaded. pdsh@oleg222-client: oleg222-client: ssh exited with exit code 2 pdsh@oleg222-client: oleg222-server: ssh exited with exit code 2 pdsh@oleg222-client: oleg222-client: ssh exited with exit code 2 pdsh@oleg222-client: oleg222-server: ssh exited with exit code 2 PASS 90a (123s) == conf-sanity test 90b: check max_mod_rpcs_in_flight is enforced after update ========================================================== 22:18:00 (1776997080) start mds service on oleg222-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg222-server' oleg222-server: oleg222-server.virtnet: executing load_modules_local oleg222-server: Loading modules from /home/green/git/lustre-release/lustre oleg222-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg222-server: detected 4 online CPUs by sysfs oleg222-server: MODOPTS_LIBCFS= oleg222-server: Force libcfs to create 2 CPU partitions oleg222-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg222-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg222-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg222-server: oleg222-server.virtnet: executing set_default_debug -1 all pdsh@oleg222-client: oleg222-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg222-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg222-server: oleg222-server.virtnet: executing set_default_debug -1 all pdsh@oleg222-client: oleg222-server: ssh exited with exit code 1 Started lustre-OST0000 oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg222-client.virtnet: -o user_xattr,flock 192.168.202.122@tcp:/lustre /mnt/lustre mdc.lustre-MDT0000-mdc-ffff91fc04a24800.max_mod_rpcs_in_flight=1 max_mod_rpcs_in_flight set to 1 creating 2 files ... fail_loc=0x159 launch 0 chmod in parallel ... fail_loc=0 launch 1 additional chmod in parallel ... /mnt/lustre/d90b.conf-sanity1/file-1 has perms 0600 OK fail_loc=0x159 launch 1 chmod in parallel ... fail_loc=0 launch 1 additional chmod in parallel ... /mnt/lustre/d90b.conf-sanity1/file-2 has perms 0644 OK mdc.lustre-MDT0000-mdc-ffff91fc04a24800.max_mod_rpcs_in_flight=5 max_mod_rpcs_in_flight set to 5 creating 6 files ... fail_loc=0x159 launch 4 chmod in parallel ... fail_loc=0 launch 1 additional chmod in parallel ... /mnt/lustre/d90b.conf-sanity2/file-5 has perms 0600 OK fail_loc=0x159 launch 5 chmod in parallel ... fail_loc=0 launch 1 additional chmod in parallel ... /mnt/lustre/d90b.conf-sanity2/file-6 has perms 0644 OK mdt_max_mod_rpcs_in_flight is 8 umount lustre on /mnt/lustre..... Stopping client oleg222-client.virtnet /mnt/lustre (opts:) error: get_param: param_path 'mdt/*/max_mod_rpcs_in_flight': No such file or directory the deprecated max_mod_rpcs_per_client parameter was involved mount lustre on /mnt/lustre..... Starting client: oleg222-client.virtnet: -o user_xattr,flock 192.168.202.122@tcp:/lustre /mnt/lustre mdc.lustre-MDT0000-mdc-ffff91fc041c0800.max_rpcs_in_flight=17 mdc.lustre-MDT0000-mdc-ffff91fc041c0800.max_mod_rpcs_in_flight=16 max_mod_rpcs_in_flight set to 16 creating 17 files ... fail_loc=0x159 launch 15 chmod in parallel ... fail_loc=0 launch 1 additional chmod in parallel ... /mnt/lustre/d90b.conf-sanity3/file-16 has perms 0600 OK fail_loc=0x159 launch 16 chmod in parallel ... fail_loc=0 launch 1 additional chmod in parallel ... /mnt/lustre/d90b.conf-sanity3/file-17 has perms 0644 OK error: get_param: param_path 'mdt/*/max_mod_rpcs_in_flight': No such file or directory the deprecated max_mod_rpcs_per_client parameter was involved umount lustre on /mnt/lustre..... Stopping client oleg222-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg222-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg222-server stop mds service on oleg222-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg222-server unloading modules via unload_modules_local on: 'oleg222-server' oleg222-server: oleg222-server.virtnet: executing unload_modules_local oleg222-server: modules unloaded. pdsh@oleg222-client: oleg222-client: ssh exited with exit code 2 pdsh@oleg222-client: oleg222-server: ssh exited with exit code 2 pdsh@oleg222-client: oleg222-client: ssh exited with exit code 2 pdsh@oleg222-client: oleg222-server: ssh exited with exit code 2 PASS 90b (212s) == conf-sanity test 90c: check max_mod_rpcs_in_flight update limits ========================================================== 22:21:32 (1776997292) start mds service on oleg222-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg222-server' oleg222-server: oleg222-server.virtnet: executing load_modules_local oleg222-server: Loading modules from /home/green/git/lustre-release/lustre oleg222-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg222-server: detected 4 online CPUs by sysfs oleg222-server: MODOPTS_LIBCFS= oleg222-server: Force libcfs to create 2 CPU partitions oleg222-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg222-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg222-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg222-server: oleg222-server.virtnet: executing set_default_debug -1 all pdsh@oleg222-client: oleg222-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg222-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg222-server: oleg222-server.virtnet: executing set_default_debug -1 all pdsh@oleg222-client: oleg222-server: ssh exited with exit code 1 Started lustre-OST0000 oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg222-client.virtnet: -o user_xattr,flock 192.168.202.122@tcp:/lustre /mnt/lustre max_rpcs_in_flight is 8 MDC max_mod_rpcs_in_flight is 7 mdt_max_mod_rpcs_in_flight is 8 error: get_param: param_path 'mdt/*/max_mod_rpcs_in_flight': No such file or directory the deprecated max_mod_rpcs_per_client parameter was involved mdc.lustre-MDT0000-mdc-ffff91fc08e27800.max_mod_rpcs_in_flight=8 umount lustre on /mnt/lustre..... Stopping client oleg222-client.virtnet /mnt/lustre (opts:) error: get_param: param_path 'mdt/*/max_mod_rpcs_in_flight': No such file or directory the deprecated max_mod_rpcs_per_client parameter was involved mount lustre on /mnt/lustre..... Starting client: oleg222-client.virtnet: -o user_xattr,flock 192.168.202.122@tcp:/lustre /mnt/lustre mdc.lustre-MDT0000-mdc-ffff91fc05631800.max_rpcs_in_flight=10 error: set_param: setting /sys/fs/lustre/mdc/lustre-MDT0000-mdc-ffff91fc05631800/max_mod_rpcs_in_flight=9: Numerical result out of range error: set_param: setting 'mdc/lustre-MDT0000-mdc-*/max_mod_rpcs_in_flight'='9': Numerical result out of range Stopping client oleg222-client.virtnet /mnt/lustre (opts:) Starting client: oleg222-client.virtnet: -o user_xattr,flock 192.168.202.122@tcp:/lustre /mnt/lustre error: get_param: param_path 'mdt/*/max_mod_rpcs_in_flight': No such file or directory the deprecated max_mod_rpcs_per_client parameter was involved umount lustre on /mnt/lustre..... Stopping client oleg222-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg222-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg222-server stop mds service on oleg222-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg222-server unloading modules via unload_modules_local on: 'oleg222-server' oleg222-server: oleg222-server.virtnet: executing unload_modules_local oleg222-server: modules unloaded. pdsh@oleg222-client: oleg222-server: ssh exited with exit code 2 pdsh@oleg222-client: oleg222-client: ssh exited with exit code 2 pdsh@oleg222-client: oleg222-server: ssh exited with exit code 2 pdsh@oleg222-client: oleg222-client: ssh exited with exit code 2 PASS 90c (121s) == conf-sanity test 90d: check one close RPC is allowed above max_mod_rpcs_in_flight ========================================================== 22:23:33 (1776997413) start mds service on oleg222-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg222-server' oleg222-server: oleg222-server.virtnet: executing load_modules_local oleg222-server: Loading modules from /home/green/git/lustre-release/lustre oleg222-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg222-server: detected 4 online CPUs by sysfs oleg222-server: MODOPTS_LIBCFS= oleg222-server: Force libcfs to create 2 CPU partitions oleg222-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg222-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg222-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg222-server: oleg222-server.virtnet: executing set_default_debug -1 all pdsh@oleg222-client: oleg222-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg222-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg222-server: oleg222-server.virtnet: executing set_default_debug -1 all pdsh@oleg222-client: oleg222-server: ssh exited with exit code 1 Started lustre-OST0000 oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg222-client.virtnet: -o user_xattr,flock 192.168.202.122@tcp:/lustre /mnt/lustre max_mod_rcps_in_flight is 7 creating 7 files ... multiop /mnt/lustre/d90d.conf-sanity/file-close vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7560 fail_loc=0x159 launch 7 chmod in parallel ... fail_loc=0 launch 1 additional close in parallel ... umount lustre on /mnt/lustre..... Stopping client oleg222-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg222-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg222-server stop mds service on oleg222-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg222-server unloading modules via unload_modules_local on: 'oleg222-server' oleg222-server: oleg222-server.virtnet: executing unload_modules_local oleg222-server: modules unloaded. pdsh@oleg222-client: oleg222-client: ssh exited with exit code 2 pdsh@oleg222-client: oleg222-server: ssh exited with exit code 2 pdsh@oleg222-client: oleg222-client: ssh exited with exit code 2 pdsh@oleg222-client: oleg222-server: ssh exited with exit code 2 PASS 90d (130s) == conf-sanity test 91: evict-by-nid support ============= 22:25:43 (1776997543) start mds service on oleg222-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg222-server' oleg222-server: oleg222-server.virtnet: executing load_modules_local oleg222-server: Loading modules from /home/green/git/lustre-release/lustre oleg222-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg222-server: detected 4 online CPUs by sysfs oleg222-server: MODOPTS_LIBCFS= oleg222-server: Force libcfs to create 2 CPU partitions oleg222-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg222-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg222-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg222-server: oleg222-server.virtnet: executing set_default_debug -1 all pdsh@oleg222-client: oleg222-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg222-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg222-server: oleg222-server.virtnet: executing set_default_debug -1 all pdsh@oleg222-client: oleg222-server: ssh exited with exit code 1 Started lustre-OST0000 oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg222-client.virtnet: -o user_xattr,flock 192.168.202.122@tcp:/lustre /mnt/lustre setup single mount lustre success list nids on mdt: mdt.lustre-MDT0000.exports.0@lo mdt.lustre-MDT0000.exports.192.168.202.22@tcp mdt.lustre-MDT0000.exports.clear uuid from 192\.168\.202\.22@tcp: mdt.lustre-MDT0000.exports.192.168.202.22@tcp.uuid=9d176cdb-c3ae-4a9b-a157-48c408aef677 manual umount lustre on /mnt/lustre.... evict 192\.168\.202\.22@tcp oleg222-server: error: read_param: '/sys/kernel/debug/lustre/mdt/lustre-MDT0000/exports/192.168.202.22@tcp/uuid': No such device pdsh@oleg222-client: oleg222-server: ssh exited with exit code 19 oleg222-server: error: read_param: '/sys/kernel/debug/lustre/obdfilter/lustre-OST0000/exports/192.168.202.22@tcp/uuid': No such device pdsh@oleg222-client: oleg222-server: ssh exited with exit code 19 oleg222-server: error: read_param: '/sys/kernel/debug/lustre/mdt/lustre-MDT0000/exports/192.168.202.22@tcp/uuid': No such device pdsh@oleg222-client: oleg222-server: ssh exited with exit code 19 oleg222-server: error: read_param: '/sys/kernel/debug/lustre/obdfilter/lustre-OST0000/exports/192.168.202.22@tcp/uuid': No such device pdsh@oleg222-client: oleg222-server: ssh exited with exit code 19 umount lustre on /mnt/lustre..... stop ost1 service on oleg222-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg222-server stop mds service on oleg222-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg222-server unloading modules via unload_modules_local on: 'oleg222-server' oleg222-server: oleg222-server.virtnet: executing unload_modules_local oleg222-server: modules unloaded. pdsh@oleg222-client: oleg222-client: ssh exited with exit code 2 pdsh@oleg222-client: oleg222-server: ssh exited with exit code 2 pdsh@oleg222-client: oleg222-server: ssh exited with exit code 2 pdsh@oleg222-client: oleg222-client: ssh exited with exit code 2 PASS 91 (147s) == conf-sanity test 92: ldev returns MGS NID correctly in command substitution ========================================================== 22:28:10 (1776997690) Host is oleg222-client.virtnet ----- /tmp/ldev.conf ----- oleg222-server - lustre-MGS0000 zfs:lustre-mdt1/mdt1 oleg222-server - lustre-OST0000 zfs:lustre-ost1/ost1 oleg222-server - lustre-OST0001 zfs:lustre-ost2/ost2 oleg222-server - lustre-MDT0000 zfs:lustre-mdt1/mdt1 --- END /tmp/ldev.conf --- ----- /tmp/nids ----- oleg222-server 192.168.202.122@tcp --- END /tmp/nids --- -- START OF LDEV OUTPUT -- lustre-OST0000: 192.168.202.122@tcp lustre-OST0001: 192.168.202.122@tcp lustre-MGS0000: 192.168.202.122@tcp lustre-MDT0000: 192.168.202.122@tcp --- END OF LDEV OUTPUT --- -- START OF LDEV OUTPUT -- lustre-MGS0000: 192.168.202.122@tcp lustre-OST0000: 192.168.202.122@tcp lustre-MDT0000: 192.168.202.122@tcp lustre-OST0001: 192.168.202.122@tcp --- END OF LDEV OUTPUT --- -- START OF LDEV OUTPUT -- lustre-MGS0000: 192.168.202.122@tcp lustre-OST0000: 192.168.202.122@tcp lustre-OST0001: 192.168.202.122@tcp lustre-MDT0000: 192.168.202.122@tcp --- END OF LDEV OUTPUT --- -- START OF LDEV OUTPUT -- lustre-OST0000: 192.168.202.122@tcp lustre-MGS0000: 192.168.202.122@tcp lustre-OST0001: 192.168.202.122@tcp lustre-MDT0000: 192.168.202.122@tcp --- END OF LDEV OUTPUT --- pdsh@oleg222-client: oleg222-server: ssh exited with exit code 2 pdsh@oleg222-client: oleg222-server: ssh exited with exit code 2 PASS 92 (9s) == conf-sanity test 93: register mulitple MDT at the same time ========================================================== 22:28:19 (1776997699) SKIP: conf-sanity test_93 needs >= 3 MDTs SKIP 93 (3s) == conf-sanity test 94: ldev outputs correct labels for file system name query ========================================================== 22:28:23 (1776997703) ----- /tmp/ldev.conf ----- oleg222-server - lustre-MGS0000 zfs:lustre-mdt1/mdt1 oleg222-server - lustre-OST0000 zfs:lustre-ost1/ost1 oleg222-server - lustre-OST0001 zfs:lustre-ost2/ost2 oleg222-server - lustre-MDT0000 zfs:lustre-mdt1/mdt1 --- END /tmp/ldev.conf --- ----- /tmp/nids ----- oleg222-server 192.168.202.122@tcp --- END /tmp/nids --- -- START OF LDEV OUTPUT -- lustre-MDT0000 lustre-MGS0000 lustre-OST0000 lustre-OST0001 --- END OF LDEV OUTPUT --- -- START OF EXPECTED OUTPUT -- lustre-MDT0000 lustre-MGS0000 lustre-OST0000 lustre-OST0001 --- END OF EXPECTED OUTPUT --- pdsh@oleg222-client: oleg222-server: ssh exited with exit code 2 pdsh@oleg222-client: oleg222-server: ssh exited with exit code 2 PASS 94 (8s) == conf-sanity test 95: ldev should only allow one label filter ========================================================== 22:28:31 (1776997711) ----- /tmp/ldev.conf ----- oleg222-server - lustre-MGS0000 zfs:lustre-mdt1/mdt1 oleg222-server - lustre-OST0000 zfs:lustre-ost1/ost1 oleg222-server - lustre-OST0001 zfs:lustre-ost2/ost2 oleg222-server - lustre-MDT0000 zfs:lustre-mdt1/mdt1 --- END /tmp/ldev.conf --- ----- /tmp/nids ----- oleg222-server 192.168.202.122@tcp --- END /tmp/nids --- pdsh@oleg222-client: oleg222-server: ssh exited with exit code 2 pdsh@oleg222-client: oleg222-server: ssh exited with exit code 2 PASS 95 (10s) == conf-sanity test 96: ldev returns hostname and backend fs correctly in command sub ========================================================== 22:28:41 (1776997721) ----- /tmp/ldev.conf ----- oleg222-server - lustre-MGS0000 zfs:lustre-mdt1/mdt1 oleg222-server - lustre-OST0000 zfs:lustre-ost1/ost1 oleg222-server - lustre-OST0001 zfs:lustre-ost2/ost2 oleg222-server - lustre-MDT0000 zfs:lustre-mdt1/mdt1 --- END /tmp/ldev.conf --- ----- /tmp/nids ----- oleg222-server 192.168.202.122@tcp --- END /tmp/nids --- -- START OF LDEV OUTPUT -- oleg222-server-zfs oleg222-server-zfs oleg222-server-zfs oleg222-server-zfs --- END OF LDEV OUTPUT --- -- START OF EXPECTED OUTPUT -- oleg222-server-zfs oleg222-server-zfs oleg222-server-zfs oleg222-server-zfs --- END OF EXPECTED OUTPUT --- pdsh@oleg222-client: oleg222-server: ssh exited with exit code 2 pdsh@oleg222-client: oleg222-server: ssh exited with exit code 2 PASS 96 (9s) == conf-sanity test 97: ldev returns correct ouput when querying based on role ========================================================== 22:28:50 (1776997730) ----- /tmp/ldev.conf ----- oleg222-server - lustre-MGS0000 zfs:lustre-mdt1/mdt1 oleg222-server - lustre-OST0000 zfs:lustre-ost1/ost1 oleg222-server - lustre-OST0001 zfs:lustre-ost2/ost2 oleg222-server - lustre-MDT0000 zfs:lustre-mdt1/mdt1 --- END /tmp/ldev.conf --- ----- /tmp/nids ----- oleg222-server 192.168.202.122@tcp --- END /tmp/nids --- MDT role -- START OF LDEV OUTPUT -- lustre-MDT0000 --- END OF LDEV OUTPUT --- -- START OF EXPECTED OUTPUT -- lustre-MDT0000 --- END OF EXPECTED OUTPUT --- OST role -- START OF LDEV OUTPUT -- lustre-OST0000 lustre-OST0001 --- END OF LDEV OUTPUT --- -- START OF EXPECTED OUTPUT -- lustre-OST0000 lustre-OST0001 --- END OF EXPECTED OUTPUT --- MGS role -- START OF LDEV OUTPUT -- lustre-MGS0000 --- END OF LDEV OUTPUT --- -- START OF EXPECTED OUTPUT -- lustre-MGS0000 --- END OF EXPECTED OUTPUT --- pdsh@oleg222-client: oleg222-server: ssh exited with exit code 2 pdsh@oleg222-client: oleg222-server: ssh exited with exit code 2 PASS 97 (10s) == conf-sanity test 98: Buffer-overflow check while parsing mount_opts ========================================================== 22:29:00 (1776997740) start mds service on oleg222-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg222-server' oleg222-server: oleg222-server.virtnet: executing load_modules_local oleg222-server: Loading modules from /home/green/git/lustre-release/lustre oleg222-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg222-server: detected 4 online CPUs by sysfs oleg222-server: MODOPTS_LIBCFS= oleg222-server: Force libcfs to create 2 CPU partitions oleg222-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg222-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg222-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg222-server: oleg222-server.virtnet: executing set_default_debug -1 all pdsh@oleg222-client: oleg222-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg222-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg222-server: oleg222-server.virtnet: executing set_default_debug -1 all pdsh@oleg222-client: oleg222-server: ssh exited with exit code 1 Started lustre-OST0000 oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg222-client.virtnet: -o user_xattr,flock 192.168.202.122@tcp:/lustre /mnt/lustre setup single mount lustre success arg[0] = mount.lustre arg[1] = -o arg[2] = remount,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr arg[3] = -v arg[4] = 192.168.202.122@tcp:/lustre arg[5] = /mnt/lustre source = 192.168.202.122@tcp:/lustre (192.168.202.122@tcp:/lustre), target = /mnt/lustre options(4517/4096) = remount,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,error: mount options too long, 4517 over limit 4096 user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr options(4517/4096) = remount,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,error: mount options too long, 4517 over limit 4096 umount lustre on /mnt/lustre..... Stopping client oleg222-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg222-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg222-server stop mds service on oleg222-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg222-server unloading modules via unload_modules_local on: 'oleg222-server' oleg222-server: oleg222-server.virtnet: executing unload_modules_local oleg222-server: modules unloaded. pdsh@oleg222-client: oleg222-client: ssh exited with exit code 2 pdsh@oleg222-client: oleg222-server: ssh exited with exit code 2 pdsh@oleg222-client: oleg222-client: ssh exited with exit code 2 pdsh@oleg222-client: oleg222-server: ssh exited with exit code 2 PASS 98 (115s) == conf-sanity test 99: Adding meta_bg option ============ 22:30:56 (1776997856) SKIP: conf-sanity test_99 ldiskfs only test SKIP 99 (3s) == conf-sanity test 100: check lshowmount lists MGS, MDT, OST and 0@lo ========================================================== 22:31:00 (1776997860) Stopping clients: oleg222-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg222-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg222-client: oleg222-server: ssh exited with exit code 2 oleg222-server: oleg222-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg222-server' oleg222-server: oleg222-server.virtnet: executing load_modules_local oleg222-server: Loading modules from /home/green/git/lustre-release/lustre oleg222-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg222-server: detected 4 online CPUs by sysfs oleg222-server: MODOPTS_LIBCFS= oleg222-server: Force libcfs to create 2 CPU partitions oleg222-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg222-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg222-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg222-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg222-server: oleg222-server.virtnet: executing set_default_debug -1 all pdsh@oleg222-client: oleg222-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg222-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg222-server: oleg222-server.virtnet: executing set_default_debug -1 all pdsh@oleg222-client: oleg222-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg222-client.virtnet: -o user_xattr,flock 192.168.202.122@tcp:/lustre /mnt/lustre setup single mount lustre success lustre-MDT0000: lustre-OST0000: umount lustre on /mnt/lustre..... Stopping client oleg222-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg222-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg222-server stop mds service on oleg222-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg222-server unloading modules via unload_modules_local on: 'oleg222-server' oleg222-server: oleg222-server.virtnet: executing unload_modules_local oleg222-server: modules unloaded. pdsh@oleg222-client: oleg222-client: ssh exited with exit code 2 pdsh@oleg222-client: oleg222-server: ssh exited with exit code 2 pdsh@oleg222-client: oleg222-server: ssh exited with exit code 2 pdsh@oleg222-client: oleg222-client: ssh exited with exit code 2 PASS 100 (174s) == conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 22:33:54 (1776998034) start mds service on oleg222-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg222-server' oleg222-server: oleg222-server.virtnet: executing load_modules_local oleg222-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg222-server: Loading modules from /home/green/git/lustre-release/lustre oleg222-server: detected 4 online CPUs by sysfs oleg222-server: MODOPTS_LIBCFS= oleg222-server: Force libcfs to create 2 CPU partitions oleg222-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg222-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg222-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg222-server: oleg222-server.virtnet: executing set_default_debug -1 all pdsh@oleg222-client: oleg222-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg222-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg222-server: oleg222-server.virtnet: executing set_default_debug -1 all pdsh@oleg222-client: oleg222-server: ssh exited with exit code 1 Started lustre-OST0000 oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg222-client.virtnet: -o user_xattr,flock 192.168.202.122@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 122 (time 1776998123.65 total 10.56 last 11.56) - open/close 306 (time 1776998135.03 total 21.93 last 16.18) - open/close 419 (time 1776998145.05 total 31.95 last 11.28) - open/close 603 (time 1776998155.90 total 42.80 last 16.96) - open/close 792 (time 1776998165.93 total 52.83 last 18.84) - open/close 889 (time 1776998178.81 total 65.71 last 7.53) - open/close 1118 (time 1776998188.83 total 75.73 last 22.86) - open/close 1309 (time 1776998200.88 total 87.79 last 15.84) - open/close 1515 (time 1776998212.33 total 99.23 last 17.99) - open/close 1701 (time 1776998223.35 total 110.26 last 16.87) - open/close 1868 (time 1776998234.09 total 121.00 last 15.55) - open/close 1944 (time 1776998244.42 total 131.33 last 7.36) - open/close 2143 (time 1776998254.51 total 141.41 last 19.74) - open/close 2351 (time 1776998265.46 total 152.36 last 18.99) - open/close 2521 (time 1776998276.03 total 162.94 last 16.08) - open/close 2718 (time 1776998286.96 total 173.87 last 18.02) - open/close 2902 (time 1776998296.97 total 183.87 last 18.39) - open/close 3116 (time 1776998309.27 total 196.17 last 17.40) - open/close 3259 (time 1776998320.05 total 206.95 last 13.27) - open/close 3377 (time 1776998330.89 total 217.79 last 10.88) - open/close 3523 (time 1776998341.45 total 228.36 last 13.82) - open/close 3757 (time 1776998352.14 total 239.04 last 21.90) - open/close 3927 (time 1776998366.46 total 253.36 last 11.87) - open/close 4021 (time 1776998376.65 total 263.55 last 9.23) - open/close 4229 (time 1776998386.81 total 273.72 last 20.46) - open/close 4423 (time 1776998396.83 total 283.73 last 19.37) - open/close 4585 (time 1776998406.85 total 293.75 last 16.16) - open/close 4776 (time 1776998417.41 total 304.32 last 18.08) - open/close 4936 (time 1776998427.53 total 314.43 last 15.82) - open/close 5208 (time 1776998438.39 total 325.29 last 25.04) - open/close 5405 (time 1776998448.58 total 335.48 last 19.33) - open/close 5656 (time 1776998459.33 total 346.23 last 23.36) - open/close 5855 (time 1776998469.61 total 356.52 last 19.34) - open/close 6103 (time 1776998480.63 total 367.54 last 22.51) - open/close 6339 (time 1776998490.90 total 377.81 last 22.98) - open/close 6505 (time 1776998501.23 total 388.14 last 16.07) - open/close 6752 (time 1776998514.13 total 401.04 last 19.14) - open/close 6991 (time 1776998524.42 total 411.33 last 23.23) - open/close 7215 (time 1776998535.40 total 422.31 last 20.40) - open/close 7445 (time 1776998546.29 total 433.19 last 21.13) - open/close 7588 (time 1776998556.29 total 443.19 last 14.30) - open/close 7683 (time 1776998566.90 total 453.80 last 8.96) - open/close 7806 (time 1776998576.91 total 463.81 last 12.29) - open/close 7979 (time 1776998586.94 total 473.84 last 17.25) - open/close 8115 (time 1776998597.48 total 484.39 last 12.90) - open/close 8262 (time 1776998608.76 total 495.67 last 13.03) - open/close 8471 (time 1776998618.99 total 505.90 last 20.43) - open/close 8571 (time 1776998629.04 total 515.95 last 9.95) - open/close 8752 (time 1776998639.23 total 526.13 last 17.78) - open/close 8969 (time 1776998649.24 total 536.15 last 21.66) - open/close 9113 (time 1776998659.98 total 546.88 last 13.41) - open/close 9184 (time 1776998670.34 total 557.25 last 6.85) - open/close 9385 (time 1776998680.35 total 567.26 last 20.08) - open/close 9604 (time 1776998690.35 total 577.26 last 21.90) - open/close 9833 (time 1776998700.39 total 587.30 last 22.82) - open/close 10000 (time 1776998710.16 total 597.07 last 17.09) - open/close 10179 (time 1776998720.17 total 607.07 last 17.89) - open/close 10304 (time 1776998730.19 total 617.10 last 12.47) - open/close 10464 (time 1776998742.00 total 628.91 last 13.55) - open/close 10593 (time 1776998752.00 total 638.91 last 12.90) - open/close 10690 (time 1776998762.39 total 649.29 last 9.34) - open/close 10888 (time 1776998772.59 total 659.49 last 19.41) - open/close 11027 (time 1776998782.71 total 669.61 last 13.73) - open/close 11290 (time 1776998793.47 total 680.37 last 24.44) - open/close 11488 (time 1776998803.49 total 690.39 last 19.76) - open/close 11710 (time 1776998813.50 total 700.41 last 22.17) - open/close 11820 (time 1776998823.64 total 710.54 last 10.85) - open/close 11979 (time 1776998834.43 total 721.33 last 14.73) - open/close 12136 (time 1776998844.79 total 731.69 last 15.15) - open/close 12313 (time 1776998855.43 total 742.33 last 16.64) - open/close 12605 (time 1776998868.29 total 755.20 last 22.70) - open/close 12858 (time 1776998878.52 total 765.42 last 24.74) - open/close 13112 (time 1776998888.83 total 775.74 last 24.63) - open/close 13210 (time 1776998899.20 total 786.11 last 9.45) - open/close 13326 (time 1776998909.20 total 796.11 last 11.60) - open/close 13451 (time 1776998919.47 total 806.38 last 12.17) - open/close 13577 (time 1776998929.47 total 816.38 last 12.60) - open/close 13699 (time 1776998939.48 total 826.39 last 12.19) - open/close 13868 (time 1776998949.49 total 836.40 last 16.88) - open/close 14037 (time 1776998959.61 total 846.51 last 16.71) - open/close 14151 (time 1776998969.63 total 856.53 last 11.38) - open/close 14354 (time 1776998979.64 total 866.55 last 20.27) - open/close 14528 (time 1776998989.92 total 876.82 last 16.93) - open/close 14775 (time 1776998999.93 total 886.84 last 24.66) - open/close 14971 (time 1776999009.98 total 896.88 last 19.52) - open/close 15172 (time 1776999020.03 total 906.94 last 19.99) - open/close 15344 (time 1776999030.04 total 916.94 last 17.19) - open/close 15522 (time 1776999040.05 total 926.96 last 17.78) - open/close 15748 (time 1776999050.07 total 936.97 last 22.57) - open/close 15967 (time 1776999060.47 total 947.38 last 21.04) - open/close 16164 (time 1776999071.21 total 958.12 last 18.34) - open/close 16343 (time 1776999084.15 total 971.05 last 13.84) - open/close 16497 (time 1776999094.19 total 981.09 last 15.34) - open/close 16655 (time 1776999105.20 total 992.10 last 14.35) - open/close 16859 (time 1776999115.51 total 1002.42 last 19.78) - open/close 17098 (time 1776999125.55 total 1012.46 last 23.81) - open/close 17300 (time 1776999135.58 total 1022.48 last 20.15) - open/close 17517 (time 1776999146.18 total 1033.08 last 20.47) - open/close 18261 (time 1776999156.18 total 1043.08 last 74.39) - open/close 19015 (time 1776999166.18 total 1053.09 last 75.37) - open/close 19618 (time 1776999176.20 total 1063.11 last 60.18) - open/close 20000 (time 1776999182.75 total 1069.66 last 58.32) - open/close 20765 (time 1776999192.77 total 1079.67 last 76.38) - open/close 21450 (time 1776999202.79 total 1089.70 last 68.34) - open/close 22108 (time 1776999212.81 total 1099.71 last 65.71) - open/close 22940 (time 1776999222.82 total 1109.73 last 83.08) - open/close 23634 (time 1776999232.82 total 1119.73 last 69.40) - open/close 24402 (time 1776999242.82 total 1129.73 last 76.78) - open/close 25201 (time 1776999252.83 total 1139.74 last 79.82) - open/close 26072 (time 1776999262.84 total 1149.74 last 87.06) - open/close 26949 (time 1776999272.87 total 1159.77 last 87.44) - open/close 27809 (time 1776999282.88 total 1169.78 last 85.93) - open/close 28591 (time 1776999292.88 total 1179.79 last 78.16) - open/close 29463 (time 1776999302.90 total 1189.81 last 87.03) - open/close 30000 (time 1776999310.58 total 1197.49 last 69.93) - open/close 30741 (time 1776999320.60 total 1207.50 last 73.97) - open/close 31505 (time 1776999330.60 total 1217.50 last 76.39) - open/close 32282 (time 1776999340.61 total 1227.51 last 77.66) - open/close 33074 (time 1776999350.61 total 1237.52 last 79.16) - open/close 33888 (time 1776999360.62 total 1247.52 last 81.36) - open/close 34619 (time 1776999370.62 total 1257.53 last 73.06) - open/close 35338 (time 1776999380.62 total 1267.53 last 71.89) - open/close 36178 (time 1776999390.64 total 1277.55 last 83.82) - open/close 37023 (time 1776999400.65 total 1287.56 last 84.40) - open/close 37881 (time 1776999410.66 total 1297.56 last 85.76) - open/close 38713 (time 1776999420.66 total 1307.57 last 83.18) - open/close 39573 (time 1776999430.67 total 1317.57 last 85.95) - open/close 40000 (time 1776999435.58 total 1322.49 last 86.92) - open/close 40808 (time 1776999445.59 total 1332.49 last 80.75) - open/close 41569 (time 1776999455.59 total 1342.49 last 76.09) - open/close 42329 (time 1776999465.61 total 1352.51 last 75.86) - open/close 43099 (time 1776999475.61 total 1362.51 last 76.98) - open/close 43864 (time 1776999485.61 total 1372.52 last 76.48) - open/close 44664 (time 1776999495.62 total 1382.52 last 79.96) - open/close 45523 (time 1776999505.63 total 1392.53 last 85.81) - open/close 46388 (time 1776999515.63 total 1402.54 last 86.45) - open/close 47198 (time 1776999525.65 total 1412.55 last 80.88) - open/close 48032 (time 1776999535.66 total 1422.56 last 83.32) - open/close 48876 (time 1776999545.66 total 1432.56 last 84.39) - open/close 49714 (time 1776999555.67 total 1442.57 last 83.73) total: 50000 open/close in 1445.82 seconds: 34.58 ops/second - unlinked 0 (time 1776999562 ; total 0 ; last 0) - unlinked 10000 (time 1776999692 ; total 130 ; last 130) - unlinked 20000 (time 1776999811 ; total 249 ; last 119) - unlinked 30000 (time 1776999929 ; total 367 ; last 118) - unlinked 40000 (time 1777000038 ; total 476 ; last 109) total: 50000 unlinks in 544 seconds: 91.911766 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg222-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg222-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg222-server stop mds service on oleg222-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg222-server unloading modules via unload_modules_local on: 'oleg222-server' oleg222-server: oleg222-server.virtnet: executing unload_modules_local oleg222-server: modules unloaded. pdsh@oleg222-client: oleg222-client: ssh exited with exit code 2 pdsh@oleg222-client: oleg222-server: ssh exited with exit code 2 pdsh@oleg222-client: oleg222-client: ssh exited with exit code 2 pdsh@oleg222-client: oleg222-server: ssh exited with exit code 2 PASS 101a (2099s) == conf-sanity test 101b: Race events DISCONNECT and ACTIVE in osp ========================================================== 23:08:53 (1777000133) start mds service on oleg222-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg222-server' oleg222-server: oleg222-server.virtnet: executing load_modules_local oleg222-server: Loading modules from /home/green/git/lustre-release/lustre oleg222-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg222-server: detected 4 online CPUs by sysfs oleg222-server: MODOPTS_LIBCFS= oleg222-server: Force libcfs to create 2 CPU partitions oleg222-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg222-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg222-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg222-server: oleg222-server.virtnet: executing set_default_debug -1 all pdsh@oleg222-client: oleg222-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg222-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg222-server: oleg222-server.virtnet: executing set_default_debug -1 all pdsh@oleg222-client: oleg222-server: ssh exited with exit code 1 Started lustre-OST0000 oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg222-client.virtnet: -o user_xattr,flock 192.168.202.122@tcp:/lustre /mnt/lustre fail_loc=0x80002107 fail_val=20 stop ost1 service on oleg222-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg222-server start ost1 service on oleg222-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg222-server: oleg222-server.virtnet: executing set_default_debug -1 all pdsh@oleg222-client: oleg222-server: ssh exited with exit code 1 Started lustre-OST0000 oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 11 sec oleg222-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff91fc05e9a000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff91fc05e9a000.ost_server_uuid in FULL state after 0 sec umount lustre on /mnt/lustre..... Stopping client oleg222-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg222-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg222-server stop mds service on oleg222-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg222-server unloading modules via unload_modules_local on: 'oleg222-server' oleg222-server: oleg222-server.virtnet: executing unload_modules_local oleg222-server: modules unloaded. pdsh@oleg222-client: oleg222-server: ssh exited with exit code 2 pdsh@oleg222-client: oleg222-client: ssh exited with exit code 2 pdsh@oleg222-client: oleg222-client: ssh exited with exit code 2 pdsh@oleg222-client: oleg222-server: ssh exited with exit code 2 PASS 101b (140s) SKIP: conf-sanity test_102 skipping excluded test 102 == conf-sanity test 103: rename filesystem name ========== 23:11:14 (1777000274) Checking servers environments Checking clients oleg222-client.virtnet environments /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg222-server' oleg222-server: oleg222-server.virtnet: executing load_modules_local oleg222-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg222-server: Loading modules from /home/green/git/lustre-release/lustre oleg222-server: detected 4 online CPUs by sysfs oleg222-server: MODOPTS_LIBCFS= oleg222-server: Force libcfs to create 2 CPU partitions oleg222-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg222-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg222-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Setup mgs, mdt, osts Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg222-server: oleg222-server.virtnet: executing set_default_debug -1 all pdsh@oleg222-client: oleg222-server: ssh exited with exit code 1 Started lustre-MDT0000 Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg222-server: oleg222-server.virtnet: executing set_default_debug -1 all pdsh@oleg222-client: oleg222-server: ssh exited with exit code 1 Started lustre-OST0000 Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg222-server: oleg222-server.virtnet: executing set_default_debug -1 all pdsh@oleg222-client: oleg222-server: ssh exited with exit code 1 Commit the device label on lustre-ost2/ost2 Started lustre-OST0001 mount lustre on /mnt/lustre..... Starting client: oleg222-client.virtnet: -o user_xattr,flock 192.168.202.122@tcp:/lustre /mnt/lustre Starting client oleg222-client.virtnet: -o user_xattr,flock 192.168.202.122@tcp:/lustre /mnt/lustre Started clients oleg222-client.virtnet: 192.168.202.122@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff91fc04931800.idle_timeout=debug osc.lustre-OST0001-osc-ffff91fc04931800.idle_timeout=debug setting jobstats to procname_uid Setting lustre.sys.jobid_var from disable to procname_uid Waiting 90s for 'procname_uid' Updated after 6s: want 'procname_uid' got 'procname_uid' disable quota as required oleg222-server: Pool lustre.pool1 created oleg222-server: Pool lustre.lustre created oleg222-server: OST lustre-OST0000_UUID added to pool lustre.lustre Stopping clients: oleg222-client.virtnet /mnt/lustre (opts:) Stopping client oleg222-client.virtnet /mnt/lustre opts: Stopping clients: oleg222-client.virtnet /mnt/lustre2 (opts:) Stopping /mnt/lustre-mds1 (opts:-f) on oleg222-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg222-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg222-server rename lustre to mylustre checking for existing Lustre data: found Read previous values: Target: lustre-MDT0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x5 (MDT MGS ) Persistent mount opts: Parameters: mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity sys.timeout=20 Permanent disk data: Target: mylustre-MDT0000 Index: 0 Lustre FS: mylustre Mount type: zfs Flags: 0x5 (MDT MGS ) Persistent mount opts: Parameters: mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity sys.timeout=20 lustre:svname=mylustre-MDT0000 cmd: zfs set mountpoint=/tmp/mntoTt51Z lustre-mdt1/mdt1 && zfs set canmount=on lustre-mdt1/mdt1 && zfs mount lustre-mdt1/mdt1 cmd: zfs umount lustre-mdt1/mdt1 && zfs set canmount=off lustre-mdt1/mdt1 && zfs set mountpoint=none lustre-mdt1/mdt1 Writing lustre-mdt1/mdt1 properties lustre:mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity lustre:sys.timeout=20 lustre:version=1 lustre:flags=5 lustre:index=0 lustre:fsname=mylustre lustre:svname=mylustre-MDT0000 checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x2 (OST ) Persistent mount opts: Parameters: sys.timeout=20 autodegrade=on mgsnode=192.168.202.122@tcp Permanent disk data: Target: mylustre-OST0000 Index: 0 Lustre FS: mylustre Mount type: zfs Flags: 0x2 (OST ) Persistent mount opts: Parameters: sys.timeout=20 autodegrade=on mgsnode=192.168.202.122@tcp autodegrade=on lustre:svname=mylustre-OST0000 cmd: zfs set mountpoint=/tmp/mntcfvpzZ lustre-ost1/ost1 && zfs set canmount=on lustre-ost1/ost1 && zfs mount lustre-ost1/ost1 cmd: zfs umount lustre-ost1/ost1 && zfs set canmount=off lustre-ost1/ost1 && zfs set mountpoint=none lustre-ost1/ost1 Writing lustre-ost1/ost1 properties lustre:sys.timeout=20 lustre:autodegrade=on lustre:mgsnode=192.168.202.122@tcp lustre:autodegrade=on lustre:version=1 lustre:flags=2 lustre:index=0 lustre:fsname=mylustre lustre:svname=mylustre-OST0000 checking for existing Lustre data: found Read previous values: Target: lustre-OST0001 Index: 1 Lustre FS: lustre Mount type: zfs Flags: 0x2 (OST ) Persistent mount opts: Parameters: sys.timeout=20 autodegrade=on mgsnode=192.168.202.122@tcp Permanent disk data: Target: mylustre-OST0001 Index: 1 Lustre FS: mylustre Mount type: zfs Flags: 0x2 (OST ) Persistent mount opts: Parameters: sys.timeout=20 autodegrade=on mgsnode=192.168.202.122@tcp autodegrade=on lustre:svname=mylustre-OST0001 cmd: zfs set mountpoint=/tmp/mntKOdAOo lustre-ost2/ost2 && zfs set canmount=on lustre-ost2/ost2 && zfs mount lustre-ost2/ost2 cmd: zfs umount lustre-ost2/ost2 && zfs set canmount=off lustre-ost2/ost2 && zfs set mountpoint=none lustre-ost2/ost2 Writing lustre-ost2/ost2 properties lustre:sys.timeout=20 lustre:autodegrade=on lustre:mgsnode=192.168.202.122@tcp lustre:autodegrade=on lustre:version=1 lustre:flags=2 lustre:index=1 lustre:fsname=mylustre lustre:svname=mylustre-OST0001 Checking servers environments Checking clients oleg222-client.virtnet environments /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg222-server' oleg222-server: oleg222-server.virtnet: executing load_modules_local oleg222-server: Loading modules from /home/green/git/lustre-release/lustre oleg222-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg222-server: detected 4 online CPUs by sysfs oleg222-server: MODOPTS_LIBCFS= oleg222-server: Force libcfs to create 2 CPU partitions Setup mgs, mdt, osts Start mds1: mount -t lustre -o localrecov mylustre-mdt1/mdt1 /mnt/lustre-mds1 oleg222-server: oleg222-server.virtnet: executing set_default_debug -1 all pdsh@oleg222-client: oleg222-server: ssh exited with exit code 1 Started mylustre-MDT0000 Start ost1: mount -t lustre -o localrecov mylustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-mylustre-OST0000-super.width=65536 oleg222-server: oleg222-server.virtnet: executing set_default_debug -1 all pdsh@oleg222-client: oleg222-server: ssh exited with exit code 1 Started mylustre-OST0000 Start ost2: mount -t lustre -o localrecov mylustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-mylustre-OST0001-super.width=65536 oleg222-server: oleg222-server.virtnet: executing set_default_debug -1 all pdsh@oleg222-client: oleg222-server: ssh exited with exit code 1 Started mylustre-OST0001 mount mylustre on /mnt/lustre..... Starting client: oleg222-client.virtnet: -o user_xattr,flock 192.168.202.122@tcp:/mylustre /mnt/lustre Starting client oleg222-client.virtnet: -o user_xattr,flock 192.168.202.122@tcp:/mylustre /mnt/lustre Started clients oleg222-client.virtnet: 192.168.202.122@tcp:/mylustre on /mnt/lustre type lustre (rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) Using TIMEOUT=20 osc.mylustre-OST0000-osc-ffff91fc10078800.idle_timeout=debug osc.mylustre-OST0001-osc-ffff91fc10078800.idle_timeout=debug disable quota as required File: /mnt/lustre/d103.conf-sanity/test-framework.sh Size: 313610 Blocks: 1027 IO Block: 4194304 regular file Device: c3aa56ceh/3282720462d Inode: 144115305935798275 Links: 1 Access: (0755/-rwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-04-23 23:12:23.000000000 -0400 Modify: 2026-04-23 23:12:23.000000000 -0400 Change: 2026-04-23 23:12:23.000000000 -0400 Birth: 2026-04-23 23:12:22.000000000 -0400 Pool: mylustre.pool1 Pool: mylustre.lustre mylustre-OST0000_UUID mylustre-OST0000_UUID oleg222-server: OST mylustre-OST0001_UUID added to pool mylustre.lustre Stopping clients: oleg222-client.virtnet /mnt/lustre (opts:) Stopping client oleg222-client.virtnet /mnt/lustre opts: Stopping clients: oleg222-client.virtnet /mnt/lustre2 (opts:) Stopping /mnt/lustre-mds1 (opts:-f) on oleg222-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg222-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg222-server rename mylustre to tfs checking for existing Lustre data: found Read previous values: Target: mylustre-MDT0000 Index: 0 Lustre FS: mylustre Mount type: zfs Flags: 0x5 (MDT MGS ) Persistent mount opts: Parameters: mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity sys.timeout=20 Permanent disk data: Target: tfs-MDT0000 Index: 0 Lustre FS: tfs Mount type: zfs Flags: 0x5 (MDT MGS ) Persistent mount opts: Parameters: mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity sys.timeout=20 lustre:svname=tfs-MDT0000 cmd: zfs set mountpoint=/tmp/mntRGjbrt mylustre-mdt1/mdt1 && zfs set canmount=on mylustre-mdt1/mdt1 && zfs mount mylustre-mdt1/mdt1 cmd: zfs umount mylustre-mdt1/mdt1 && zfs set canmount=off mylustre-mdt1/mdt1 && zfs set mountpoint=none mylustre-mdt1/mdt1 Writing mylustre-mdt1/mdt1 properties lustre:mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity lustre:sys.timeout=20 lustre:version=1 lustre:flags=5 lustre:index=0 lustre:fsname=tfs lustre:svname=tfs-MDT0000 checking for existing Lustre data: found Read previous values: Target: mylustre-OST0000 Index: 0 Lustre FS: mylustre Mount type: zfs Flags: 0x2 (OST ) Persistent mount opts: Parameters: sys.timeout=20 autodegrade=on mgsnode=192.168.202.122@tcp Permanent disk data: Target: tfs-OST0000 Index: 0 Lustre FS: tfs Mount type: zfs Flags: 0x2 (OST ) Persistent mount opts: Parameters: sys.timeout=20 autodegrade=on mgsnode=192.168.202.122@tcp autodegrade=on lustre:svname=tfs-OST0000 cmd: zfs set mountpoint=/tmp/mntDOpJl3 mylustre-ost1/ost1 && zfs set canmount=on mylustre-ost1/ost1 && zfs mount mylustre-ost1/ost1 cmd: zfs umount mylustre-ost1/ost1 && zfs set canmount=off mylustre-ost1/ost1 && zfs set mountpoint=none mylustre-ost1/ost1 Writing mylustre-ost1/ost1 properties lustre:sys.timeout=20 lustre:autodegrade=on lustre:mgsnode=192.168.202.122@tcp lustre:autodegrade=on lustre:version=1 lustre:flags=2 lustre:index=0 lustre:fsname=tfs lustre:svname=tfs-OST0000 checking for existing Lustre data: found Read previous values: Target: mylustre-OST0001 Index: 1 Lustre FS: mylustre Mount type: zfs Flags: 0x2 (OST ) Persistent mount opts: Parameters: sys.timeout=20 autodegrade=on mgsnode=192.168.202.122@tcp Permanent disk data: Target: tfs-OST0001 Index: 1 Lustre FS: tfs Mount type: zfs Flags: 0x2 (OST ) Persistent mount opts: Parameters: sys.timeout=20 autodegrade=on mgsnode=192.168.202.122@tcp autodegrade=on lustre:svname=tfs-OST0001 cmd: zfs set mountpoint=/tmp/mntnGKnOx mylustre-ost2/ost2 && zfs set canmount=on mylustre-ost2/ost2 && zfs mount mylustre-ost2/ost2 cmd: zfs umount mylustre-ost2/ost2 && zfs set canmount=off mylustre-ost2/ost2 && zfs set mountpoint=none mylustre-ost2/ost2 Writing mylustre-ost2/ost2 properties lustre:sys.timeout=20 lustre:autodegrade=on lustre:mgsnode=192.168.202.122@tcp lustre:autodegrade=on lustre:version=1 lustre:flags=2 lustre:index=1 lustre:fsname=tfs lustre:svname=tfs-OST0001 Checking servers environments Checking clients oleg222-client.virtnet environments /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg222-server' oleg222-server: oleg222-server.virtnet: executing load_modules_local oleg222-server: Loading modules from /home/green/git/lustre-release/lustre oleg222-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg222-server: detected 4 online CPUs by sysfs oleg222-server: MODOPTS_LIBCFS= oleg222-server: Force libcfs to create 2 CPU partitions Setup mgs, mdt, osts Start mds1: mount -t lustre -o localrecov tfs-mdt1/mdt1 /mnt/lustre-mds1 oleg222-server: oleg222-server.virtnet: executing set_default_debug -1 all pdsh@oleg222-client: oleg222-server: ssh exited with exit code 1 Started tfs-MDT0000 Start ost1: mount -t lustre -o localrecov tfs-ost1/ost1 /mnt/lustre-ost1 seq.cli-tfs-OST0000-super.width=65536 oleg222-server: oleg222-server.virtnet: executing set_default_debug -1 all pdsh@oleg222-client: oleg222-server: ssh exited with exit code 1 Started tfs-OST0000 Start ost2: mount -t lustre -o localrecov tfs-ost2/ost2 /mnt/lustre-ost2 seq.cli-tfs-OST0001-super.width=65536 oleg222-server: oleg222-server.virtnet: executing set_default_debug -1 all pdsh@oleg222-client: oleg222-server: ssh exited with exit code 1 Started tfs-OST0001 mount tfs on /mnt/lustre..... Starting client: oleg222-client.virtnet: -o user_xattr,flock 192.168.202.122@tcp:/tfs /mnt/lustre Starting client oleg222-client.virtnet: -o user_xattr,flock 192.168.202.122@tcp:/tfs /mnt/lustre Started clients oleg222-client.virtnet: 192.168.202.122@tcp:/tfs on /mnt/lustre type lustre (rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) Using TIMEOUT=20 osc.tfs-OST0000-osc-ffff91fc2d8ae000.idle_timeout=debug osc.tfs-OST0001-osc-ffff91fc2d8ae000.idle_timeout=debug disable quota as required File: /mnt/lustre/d103.conf-sanity/test-framework.sh Size: 313610 Blocks: 1027 IO Block: 4194304 regular file Device: 32e2fa5ah/853736026d Inode: 144115305935798275 Links: 1 Access: (0755/-rwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-04-23 23:12:23.000000000 -0400 Modify: 2026-04-23 23:12:23.000000000 -0400 Change: 2026-04-23 23:12:23.000000000 -0400 Birth: 2026-04-23 23:12:22.000000000 -0400 Pool: tfs.pool1 Pool: tfs.lustre tfs-OST0000_UUID tfs-OST0001_UUID tfs-OST0000_UUID Stopping clients: oleg222-client.virtnet /mnt/lustre (opts:) Stopping client oleg222-client.virtnet /mnt/lustre opts: Stopping clients: oleg222-client.virtnet /mnt/lustre2 (opts:) Stopping /mnt/lustre-mds1 (opts:-f) on oleg222-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg222-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg222-server rename tfs to lustre checking for existing Lustre data: found Read previous values: Target: tfs-MDT0000 Index: 0 Lustre FS: tfs Mount type: zfs Flags: 0x5 (MDT MGS ) Persistent mount opts: Parameters: mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity sys.timeout=20 Permanent disk data: Target: lustre-MDT0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x5 (MDT MGS ) Persistent mount opts: Parameters: mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity sys.timeout=20 lustre:svname=lustre-MDT0000 cmd: zfs set mountpoint=/tmp/mntg6Gw5p tfs-mdt1/mdt1 && zfs set canmount=on tfs-mdt1/mdt1 && zfs mount tfs-mdt1/mdt1 cmd: zfs umount tfs-mdt1/mdt1 && zfs set canmount=off tfs-mdt1/mdt1 && zfs set mountpoint=none tfs-mdt1/mdt1 Writing tfs-mdt1/mdt1 properties lustre:mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity lustre:sys.timeout=20 lustre:version=1 lustre:flags=5 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-MDT0000 checking for existing Lustre data: found Read previous values: Target: tfs-OST0000 Index: 0 Lustre FS: tfs Mount type: zfs Flags: 0x2 (OST ) Persistent mount opts: Parameters: sys.timeout=20 autodegrade=on mgsnode=192.168.202.122@tcp Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x2 (OST ) Persistent mount opts: Parameters: sys.timeout=20 autodegrade=on mgsnode=192.168.202.122@tcp autodegrade=on lustre:svname=lustre-OST0000 cmd: zfs set mountpoint=/tmp/mnt4tmnP1 tfs-ost1/ost1 && zfs set canmount=on tfs-ost1/ost1 && zfs mount tfs-ost1/ost1 cmd: zfs umount tfs-ost1/ost1 && zfs set canmount=off tfs-ost1/ost1 && zfs set mountpoint=none tfs-ost1/ost1 Writing tfs-ost1/ost1 properties lustre:sys.timeout=20 lustre:autodegrade=on lustre:mgsnode=192.168.202.122@tcp lustre:autodegrade=on lustre:version=1 lustre:flags=2 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 checking for existing Lustre data: found Read previous values: Target: tfs-OST0001 Index: 1 Lustre FS: tfs Mount type: zfs Flags: 0x2 (OST ) Persistent mount opts: Parameters: sys.timeout=20 autodegrade=on mgsnode=192.168.202.122@tcp Permanent disk data: Target: lustre-OST0001 Index: 1 Lustre FS: lustre Mount type: zfs Flags: 0x2 (OST ) Persistent mount opts: Parameters: sys.timeout=20 autodegrade=on mgsnode=192.168.202.122@tcp autodegrade=on lustre:svname=lustre-OST0001 cmd: zfs set mountpoint=/tmp/mntqBSClP tfs-ost2/ost2 && zfs set canmount=on tfs-ost2/ost2 && zfs mount tfs-ost2/ost2 cmd: zfs umount tfs-ost2/ost2 && zfs set canmount=off tfs-ost2/ost2 && zfs set mountpoint=none tfs-ost2/ost2 Writing tfs-ost2/ost2 properties lustre:sys.timeout=20 lustre:autodegrade=on lustre:mgsnode=192.168.202.122@tcp lustre:autodegrade=on lustre:version=1 lustre:flags=2 lustre:index=1 lustre:fsname=lustre lustre:svname=lustre-OST0001 Checking servers environments Checking clients oleg222-client.virtnet environments /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg222-server' oleg222-server: oleg222-server.virtnet: executing load_modules_local oleg222-server: Loading modules from /home/green/git/lustre-release/lustre oleg222-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg222-server: detected 4 online CPUs by sysfs oleg222-server: MODOPTS_LIBCFS= oleg222-server: Force libcfs to create 2 CPU partitions Setup mgs, mdt, osts Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg222-server: oleg222-server.virtnet: executing set_default_debug -1 all pdsh@oleg222-client: oleg222-server: ssh exited with exit code 1 Started lustre-MDT0000 Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg222-server: oleg222-server.virtnet: executing set_default_debug -1 all pdsh@oleg222-client: oleg222-server: ssh exited with exit code 1 Started lustre-OST0000 Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg222-server: oleg222-server.virtnet: executing set_default_debug -1 all pdsh@oleg222-client: oleg222-server: ssh exited with exit code 1 Started lustre-OST0001 mount lustre on /mnt/lustre..... Starting client: oleg222-client.virtnet: -o user_xattr,flock 192.168.202.122@tcp:/lustre /mnt/lustre Starting client oleg222-client.virtnet: -o user_xattr,flock 192.168.202.122@tcp:/lustre /mnt/lustre Started clients oleg222-client.virtnet: 192.168.202.122@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff91fc12014000.idle_timeout=debug osc.lustre-OST0001-osc-ffff91fc12014000.idle_timeout=debug disable quota as required PASS 103 (312s) == conf-sanity test 104a: Make sure user defined options are reflected in mount ========================================================== 23:16:26 (1777000586) mountfsopt: acl,user_xattr Stopping clients: oleg222-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg222-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg222-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg222-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg222-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg222-server oleg222-server: oleg222-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg222-server' oleg222-server: oleg222-server.virtnet: executing load_modules_local oleg222-server: Loading modules from /home/green/git/lustre-release/lustre oleg222-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg222-server: detected 4 online CPUs by sysfs oleg222-server: MODOPTS_LIBCFS= oleg222-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 Start mds1: mount -t lustre -o localrecov,noacl lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg222-server: oleg222-server.virtnet: executing set_default_debug -1 all pdsh@oleg222-client: oleg222-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg222-server: oleg222-server.virtnet: executing set_default_debug -1 all pdsh@oleg222-client: oleg222-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg222-server: oleg222-server.virtnet: executing set_default_debug -1 all pdsh@oleg222-client: oleg222-server: ssh exited with exit code 1 Commit the device label on lustre-ost2/ost2 Started lustre-OST0001 mount lustre on /mnt/lustre..... Starting client: oleg222-client.virtnet: -o user_xattr,flock 192.168.202.122@tcp:/lustre /mnt/lustre setfacl: /mnt/lustre: Operation not supported PASS 104a (111s) == conf-sanity test 104b: Mount uses last flock argument ========================================================== 23:18:17 (1777000697) mount lustre with opts flock,localflock on /mnt/lustre3..... Starting client: oleg222-client.virtnet: -o flock,localflock 192.168.202.122@tcp:/lustre /mnt/lustre3 192.168.202.122@tcp:/lustre on /mnt/lustre3 type lustre (rw,checksum,encrypt,noflock,localflock,lruresize,nolock,statfs_project,nouser_fid2path,nouser_xattr,verbose) umount lustre on /mnt/lustre3..... Stopping client oleg222-client.virtnet /mnt/lustre3 (opts:) mount lustre with opts localflock,flock on /mnt/lustre3..... Starting client: oleg222-client.virtnet: -o localflock,flock 192.168.202.122@tcp:/lustre /mnt/lustre3 192.168.202.122@tcp:/lustre on /mnt/lustre3 type lustre (rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,nouser_xattr,verbose) umount lustre on /mnt/lustre3..... Stopping client oleg222-client.virtnet /mnt/lustre3 (opts:) mount lustre with opts localflock,flock,noflock on /mnt/lustre3..... Starting client: oleg222-client.virtnet: -o localflock,flock,noflock 192.168.202.122@tcp:/lustre /mnt/lustre3 umount lustre on /mnt/lustre3..... Stopping client oleg222-client.virtnet /mnt/lustre3 (opts:) PASS 104b (8s) == conf-sanity test 105: check file creation for ro and rw bind mnt pt ========================================================== 23:18:26 (1777000706) umount lustre on /mnt/lustre..... Stopping client oleg222-client.virtnet /mnt/lustre (opts:-f) stop ost1 service on oleg222-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg222-server stop mds service on oleg222-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg222-server unloading modules via unload_modules_local on: 'oleg222-server' oleg222-server: oleg222-server.virtnet: executing unload_modules_local oleg222-server: oleg222-server: rmmod: ERROR: Module mdd is in use pdsh@oleg222-client: oleg222-server: ssh exited with exit code 2 modules unloaded. Stopping clients: oleg222-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg222-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-ost2 (opts:-f) on oleg222-server oleg222-server: oleg222-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg222-server' oleg222-server: oleg222-server.virtnet: executing load_modules_local oleg222-server: Loading modules from /home/green/git/lustre-release/lustre oleg222-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg222-server: detected 4 online CPUs by sysfs oleg222-server: MODOPTS_LIBCFS= oleg222-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg222-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg222-server: oleg222-server.virtnet: executing set_default_debug -1 all pdsh@oleg222-client: oleg222-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg222-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg222-server: oleg222-server.virtnet: executing set_default_debug -1 all pdsh@oleg222-client: oleg222-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg222-client.virtnet: -o user_xattr,flock 192.168.202.122@tcp:/lustre /mnt/lustre touch: cannot touch '/tmp/d105.conf-sanity/f105.conf-sanity': Read-only file system umount lustre on /mnt/lustre..... Stopping client oleg222-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg222-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg222-server stop mds service on oleg222-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg222-server unloading modules via unload_modules_local on: 'oleg222-server' oleg222-server: oleg222-server.virtnet: executing unload_modules_local oleg222-server: modules unloaded. pdsh@oleg222-client: oleg222-client: ssh exited with exit code 2 pdsh@oleg222-client: oleg222-server: ssh exited with exit code 2 pdsh@oleg222-client: oleg222-client: ssh exited with exit code 2 pdsh@oleg222-client: oleg222-server: ssh exited with exit code 2 PASS 105 (121s) SKIP: conf-sanity test_106 skipping SLOW test 106 == conf-sanity test 107: Unknown config param should not fail target mounting ========================================================== 23:20:28 (1777000828) start mds service on oleg222-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg222-server' oleg222-server: oleg222-server.virtnet: executing load_modules_local oleg222-server: Loading modules from /home/green/git/lustre-release/lustre oleg222-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg222-server: detected 4 online CPUs by sysfs oleg222-server: MODOPTS_LIBCFS= oleg222-server: Force libcfs to create 2 CPU partitions oleg222-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg222-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg222-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg222-server: oleg222-server.virtnet: executing set_default_debug -1 all pdsh@oleg222-client: oleg222-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg222-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg222-server: oleg222-server.virtnet: executing set_default_debug -1 all pdsh@oleg222-client: oleg222-server: ssh exited with exit code 1 Started lustre-OST0000 oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid stop ost1 service on oleg222-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg222-server stop mds service on oleg222-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg222-server unloading modules via unload_modules_local on: 'oleg222-server' oleg222-server: oleg222-server.virtnet: executing unload_modules_local oleg222-server: modules unloaded. /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg222-server' oleg222-server: oleg222-server.virtnet: executing load_modules_local oleg222-server: Loading modules from /home/green/git/lustre-release/lustre oleg222-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg222-server: detected 4 online CPUs by sysfs oleg222-server: MODOPTS_LIBCFS= oleg222-server: Force libcfs to create 2 CPU partitions oleg222-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg222-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg222-server: mdt/mdt options: 'mdt_enable_flr_ec=1' start ost1 service on oleg222-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg222-server: oleg222-server.virtnet: executing set_default_debug -1 all pdsh@oleg222-client: oleg222-server: ssh exited with exit code 1 Started lustre-OST0000 oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid umount lustre on /mnt/lustre..... stop ost1 service on oleg222-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg222-server stop mds service on oleg222-server unloading modules via unload_modules_local on: 'oleg222-server' oleg222-server: oleg222-server.virtnet: executing unload_modules_local oleg222-server: modules unloaded. pdsh@oleg222-client: oleg222-client: ssh exited with exit code 2 pdsh@oleg222-client: oleg222-server: ssh exited with exit code 2 pdsh@oleg222-client: oleg222-client: ssh exited with exit code 2 pdsh@oleg222-client: oleg222-server: ssh exited with exit code 2 PASS 107 (144s) SKIP: conf-sanity test_108a skipping ALWAYS excluded test 108a == conf-sanity test 108b: migrate from ZFS to ldiskfs ==== 23:22:53 (1777000973) SKIP: conf-sanity test_108b ldiskfs only test SKIP 108b (2s) == conf-sanity test 109a: test lctl clear_conf fsname ==== 23:22:55 (1777000975) SKIP: conf-sanity test_109a LU-8727: no implementation for ZFS SKIP 109a (2s) == conf-sanity test 109b: test lctl clear_conf one config ========================================================== 23:22:57 (1777000977) SKIP: conf-sanity test_109b LU-8727: no implementation for ZFS SKIP 109b (2s) == conf-sanity test 110: Adding large_dir with 3-level htree ========================================================== 23:22:59 (1777000979) SKIP: conf-sanity test_110 Only applicable to ldiskfs-based MDTs SKIP 110 (2s) SKIP: conf-sanity test_111 skipping SLOW test 111 SKIP: conf-sanity test_112a skipping ALWAYS excluded test 112a SKIP: conf-sanity test_112b skipping ALWAYS excluded test 112b SKIP: conf-sanity test_113 skipping ALWAYS excluded test 113 SKIP: conf-sanity test_114 skipping SLOW test 114 SKIP: conf-sanity test_115 skipping excluded test 115 == conf-sanity test 116: big size MDT support ============ 23:23:06 (1777000986) SKIP: conf-sanity test_116 ldiskfs only test SKIP 116 (1s) SKIP: conf-sanity test_117 skipping ALWAYS excluded test 117 SKIP: conf-sanity test_119 skipping ALWAYS excluded test 119 == conf-sanity test 120: cross-target rename should not create bad symlinks ========================================================== 23:23:09 (1777000989) SKIP: conf-sanity test_120 mdt count < 2 SKIP 120 (2s) Stopping clients: oleg222-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg222-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg222-client: oleg222-server: ssh exited with exit code 2 oleg222-server: oleg222-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg222-server' oleg222-server: oleg222-server.virtnet: executing load_modules_local oleg222-server: Loading modules from /home/green/git/lustre-release/lustre oleg222-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg222-server: detected 4 online CPUs by sysfs oleg222-server: MODOPTS_LIBCFS= oleg222-server: Force libcfs to create 2 CPU partitions oleg222-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg222-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg222-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 5792 sec ========== 23:24:31 (1777001071) === conf-sanity: start cleanup 23:24:34 (1777001074) === === conf-sanity: finish cleanup 23:24:38 (1777001078) ===