-----============= acceptance-small: conf-sanity ============----- Tue Sep 8 06:23:08 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg331-server: /home/green/git/lustre-release/lustre/tests/except/conf-sanity.ex - see CLIENT_VERSION >= v2_15_63-53-g1a6ef725c2 (34683431 >= 34553653) for LU-16938, go 81 - see CLIENT_VERSION >= v2_15_65-31-g74a5b9efaa (34683431 >= 34554143) for LU-18348, go 32b 32c - see CLIENT_VERSION >= v2_16_55-19-gf69f27cc35 (34683431 >= 34617107) for LU-17920, go 123ae 123ag - see CLIENT_VERSION >= v2_17_52-94-g86904ef943 (34683431 >= 34681950) for LU-19740, go 115 excepting tests: 102 106 115 32newtarball 112a 112b 113 117 119 121 122a 123aa 123ab 123ac 123ad 123ae 123af 123ag 123ah 123ahi 123F 123G 123H 126 129 132 133 135 136 137 150 152 153a 153b 153c 155 802a skipping tests SLOW=no: 45 69 106 111 114 Stopping clients: oleg331-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg331-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg331-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg331-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg331-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg331-server oleg331-server: oleg331-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg331-server' oleg331-server: oleg331-server.virtnet: executing load_modules_local oleg331-server: kmemleak disabled oleg331-server: Loading modules from /home/green/git/lustre-release/lustre oleg331-server: detected 4 online CPUs by sysfs oleg331-server: MODOPTS_LIBCFS= oleg331-server: Force libcfs to create 2 CPU partitions oleg331-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg331-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg331-server: oleg331-server.virtnet: executing set_default_debug -1 all pdsh@oleg331-client: oleg331-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg331-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg331-server: oleg331-server.virtnet: executing set_default_debug -1 all pdsh@oleg331-client: oleg331-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg331-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg331-server: oleg331-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg331-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg331-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg331-server stop mds service on oleg331-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg331-server == conf-sanity test 76a: set permanent params with lctl across mounts ========================================================== 06:25:55 (1788863155) start mds service on oleg331-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg331-server: oleg331-server.virtnet: executing set_default_debug -1 all pdsh@oleg331-client: oleg331-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg331-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg331-server: oleg331-server.virtnet: executing set_default_debug -1 all pdsh@oleg331-client: oleg331-server: ssh exited with exit code 1 Started lustre-OST0000 oleg331-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg331-client.virtnet: -o user_xattr,flock 192.168.203.131@tcp:/lustre /mnt/lustre Change MGS params max_dirty_mb: 462 new_max_dirty_mb: 452 Waiting 90s for '452' Updated after 2s: want '452' got '452' 452 Check the value is stored after remount Stopping clients: oleg331-client.virtnet /mnt/lustre (opts:) Stopping client oleg331-client.virtnet /mnt/lustre opts: Stopping clients: oleg331-client.virtnet /mnt/lustre2 (opts:) Stopping /mnt/lustre-mds1 (opts:-f) on oleg331-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg331-server Checking servers environments Checking clients oleg331-client.virtnet environments Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg331-server' oleg331-server: oleg331-server.virtnet: executing load_modules_local oleg331-server: kmemleak disabled oleg331-server: Loading modules from /home/green/git/lustre-release/lustre oleg331-server: detected 4 online CPUs by sysfs oleg331-server: MODOPTS_LIBCFS= oleg331-server: Force libcfs to create 2 CPU partitions oleg331-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Setup mgs, mdt, osts Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg331-server: oleg331-server.virtnet: executing set_default_debug -1 all pdsh@oleg331-client: oleg331-server: ssh exited with exit code 1 Started lustre-MDT0000 Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg331-server: oleg331-server.virtnet: executing set_default_debug -1 all pdsh@oleg331-client: oleg331-server: ssh exited with exit code 1 Started lustre-OST0000 Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg331-server: oleg331-server.virtnet: executing set_default_debug -1 all pdsh@oleg331-client: oleg331-server: ssh exited with exit code 1 Commit the device label on lustre-ost2/ost2 Started lustre-OST0001 mount lustre on /mnt/lustre..... Starting client: oleg331-client.virtnet: -o user_xattr,flock 192.168.203.131@tcp:/lustre /mnt/lustre Starting client oleg331-client.virtnet: -o user_xattr,flock 192.168.203.131@tcp:/lustre /mnt/lustre Started clients oleg331-client.virtnet: 192.168.203.131@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9b5920831000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9b5920831000.idle_timeout=debug disable quota as required Change OST params client_cache_count: 128 new_client_cache_count: 256 Waiting 90s for '256' Updated after 2s: want '256' got '256' 256 Check the value is stored after remount Stopping clients: oleg331-client.virtnet /mnt/lustre (opts:) Stopping client oleg331-client.virtnet /mnt/lustre opts: Stopping clients: oleg331-client.virtnet /mnt/lustre2 (opts:) Stopping /mnt/lustre-mds1 (opts:-f) on oleg331-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg331-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg331-server Checking servers environments Checking clients oleg331-client.virtnet environments Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg331-server' oleg331-server: oleg331-server.virtnet: executing load_modules_local oleg331-server: kmemleak disabled oleg331-server: Loading modules from /home/green/git/lustre-release/lustre oleg331-server: detected 4 online CPUs by sysfs oleg331-server: MODOPTS_LIBCFS= oleg331-server: Force libcfs to create 2 CPU partitions oleg331-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Setup mgs, mdt, osts Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg331-server: oleg331-server.virtnet: executing set_default_debug -1 all pdsh@oleg331-client: oleg331-server: ssh exited with exit code 1 Started lustre-MDT0000 Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg331-server: oleg331-server.virtnet: executing set_default_debug -1 all pdsh@oleg331-client: oleg331-server: ssh exited with exit code 1 Started lustre-OST0000 Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg331-server: oleg331-server.virtnet: executing set_default_debug -1 all pdsh@oleg331-client: oleg331-server: ssh exited with exit code 1 Started lustre-OST0001 mount lustre on /mnt/lustre..... Starting client: oleg331-client.virtnet: -o user_xattr,flock 192.168.203.131@tcp:/lustre /mnt/lustre Starting client oleg331-client.virtnet: -o user_xattr,flock 192.168.203.131@tcp:/lustre /mnt/lustre Started clients oleg331-client.virtnet: 192.168.203.131@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9b5918630800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9b5918630800.idle_timeout=debug disable quota as required 256 Stopping clients: oleg331-client.virtnet /mnt/lustre (opts:) Stopping client oleg331-client.virtnet /mnt/lustre opts: Stopping clients: oleg331-client.virtnet /mnt/lustre2 (opts:) Stopping /mnt/lustre-mds1 (opts:-f) on oleg331-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg331-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg331-server PASS 76a (278s) == conf-sanity test 76b: verify params log setup correctly ========================================================== 06:30:33 (1788863433) Stopping clients: oleg331-client.virtnet /mnt/lustre (opts:) Stopping clients: oleg331-client.virtnet /mnt/lustre2 (opts:) Checking servers environments Checking clients oleg331-client.virtnet environments Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg331-server' oleg331-server: oleg331-server.virtnet: executing load_modules_local oleg331-server: kmemleak disabled oleg331-server: Loading modules from /home/green/git/lustre-release/lustre oleg331-server: detected 4 online CPUs by sysfs oleg331-server: MODOPTS_LIBCFS= oleg331-server: Force libcfs to create 2 CPU partitions oleg331-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Setup mgs, mdt, osts Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg331-server: oleg331-server.virtnet: executing set_default_debug -1 all pdsh@oleg331-client: oleg331-server: ssh exited with exit code 1 Started lustre-MDT0000 Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg331-server: oleg331-server.virtnet: executing set_default_debug -1 all pdsh@oleg331-client: oleg331-server: ssh exited with exit code 1 Started lustre-OST0000 Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg331-server: oleg331-server.virtnet: executing set_default_debug -1 all pdsh@oleg331-client: oleg331-server: ssh exited with exit code 1 Started lustre-OST0001 mount lustre on /mnt/lustre..... Starting client: oleg331-client.virtnet: -o user_xattr,flock 192.168.203.131@tcp:/lustre /mnt/lustre Starting client oleg331-client.virtnet: -o user_xattr,flock 192.168.203.131@tcp:/lustre /mnt/lustre Started clients oleg331-client.virtnet: 192.168.203.131@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9b5918661000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9b5918661000.idle_timeout=debug disable quota as required mgs.MGS.live.params= fsname: params flags: 0x20 gen: 1 Secure RPC Config Rules: imperative_recovery_state: state: startup nonir_clients: 0 nidtbl_version: 2 notify_duration_total: 0.000000000 notify_duation_max: 0.000000000 notify_count: 0 Stopping clients: oleg331-client.virtnet /mnt/lustre (opts:) Stopping client oleg331-client.virtnet /mnt/lustre opts: Stopping clients: oleg331-client.virtnet /mnt/lustre2 (opts:) Stopping /mnt/lustre-mds1 (opts:-f) on oleg331-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg331-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg331-server PASS 76b (110s) == conf-sanity test 76c: verify changelog_mask is applied with lctl set_param -P ========================================================== 06:32:23 (1788863543) Checking servers environments Checking clients oleg331-client.virtnet environments Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg331-server' oleg331-server: oleg331-server.virtnet: executing load_modules_local oleg331-server: kmemleak disabled oleg331-server: Loading modules from /home/green/git/lustre-release/lustre oleg331-server: detected 4 online CPUs by sysfs oleg331-server: MODOPTS_LIBCFS= oleg331-server: Force libcfs to create 2 CPU partitions oleg331-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Setup mgs, mdt, osts Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg331-server: oleg331-server.virtnet: executing set_default_debug -1 all pdsh@oleg331-client: oleg331-server: ssh exited with exit code 1 Started lustre-MDT0000 Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg331-server: oleg331-server.virtnet: executing set_default_debug -1 all pdsh@oleg331-client: oleg331-server: ssh exited with exit code 1 Started lustre-OST0000 Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg331-server: oleg331-server.virtnet: executing set_default_debug -1 all pdsh@oleg331-client: oleg331-server: ssh exited with exit code 1 Started lustre-OST0001 mount lustre on /mnt/lustre..... Starting client: oleg331-client.virtnet: -o user_xattr,flock 192.168.203.131@tcp:/lustre /mnt/lustre Starting client oleg331-client.virtnet: -o user_xattr,flock 192.168.203.131@tcp:/lustre /mnt/lustre Started clients oleg331-client.virtnet: 192.168.203.131@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9b59192e2000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9b59192e2000.idle_timeout=debug disable quota as required Change changelog_mask pdsh@oleg331-client: oleg331-server: ssh exited with exit code 1 Check the value is stored after mds remount stop mds service on oleg331-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg331-server start mds service on oleg331-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg331-server: oleg331-server.virtnet: executing set_default_debug -1 all pdsh@oleg331-client: oleg331-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Stopping clients: oleg331-client.virtnet /mnt/lustre (opts:) Stopping client oleg331-client.virtnet /mnt/lustre opts: Stopping clients: oleg331-client.virtnet /mnt/lustre2 (opts:) Stopping /mnt/lustre-mds1 (opts:-f) on oleg331-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg331-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg331-server PASS 76c (119s) == conf-sanity test 76d: verify llite.*.xattr_cache can be set by 'lctl set_param -P' correctly ========================================================== 06:34:22 (1788863662) Checking servers environments Checking clients oleg331-client.virtnet environments Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg331-server' oleg331-server: oleg331-server.virtnet: executing load_modules_local oleg331-server: kmemleak disabled oleg331-server: Loading modules from /home/green/git/lustre-release/lustre oleg331-server: detected 4 online CPUs by sysfs oleg331-server: MODOPTS_LIBCFS= oleg331-server: Force libcfs to create 2 CPU partitions oleg331-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Setup mgs, mdt, osts Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg331-server: oleg331-server.virtnet: executing set_default_debug -1 all pdsh@oleg331-client: oleg331-server: ssh exited with exit code 1 Started lustre-MDT0000 Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg331-server: oleg331-server.virtnet: executing set_default_debug -1 all pdsh@oleg331-client: oleg331-server: ssh exited with exit code 1 Started lustre-OST0000 Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg331-server: oleg331-server.virtnet: executing set_default_debug -1 all pdsh@oleg331-client: oleg331-server: ssh exited with exit code 1 Started lustre-OST0001 mount lustre on /mnt/lustre..... Starting client: oleg331-client.virtnet: -o user_xattr,flock 192.168.203.131@tcp:/lustre /mnt/lustre Starting client oleg331-client.virtnet: -o user_xattr,flock 192.168.203.131@tcp:/lustre /mnt/lustre Started clients oleg331-client.virtnet: 192.168.203.131@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9b5908bac000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9b5908bac000.idle_timeout=debug disable quota as required lctl set_param -P llite.*.xattr_cache=0 Waiting 90s for '0' Updated after 2s: want '0' got '0' Check llite.$inst.xattr_cache on client /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg331-client.virtnet /mnt/lustre (opts:) mount lustre on /mnt/lustre..... Starting client: oleg331-client.virtnet: -o user_xattr,flock 192.168.203.131@tcp:/lustre /mnt/lustre Check llite.$inst.xattr_cache on the new client /mnt/lustre2 mount lustre on /mnt/lustre2..... Starting client: oleg331-client.virtnet: -o user_xattr,flock 192.168.203.131@tcp:/lustre /mnt/lustre2 umount lustre on /mnt/lustre2..... Stopping client oleg331-client.virtnet /mnt/lustre2 (opts:) Stopping clients: oleg331-client.virtnet /mnt/lustre (opts:) Stopping client oleg331-client.virtnet /mnt/lustre opts: Stopping clients: oleg331-client.virtnet /mnt/lustre2 (opts:) Stopping /mnt/lustre-mds1 (opts:-f) on oleg331-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg331-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg331-server oleg331-server: No device found for name MGS: Invalid argument oleg331-server: This command must be run on the MGS. oleg331-server: error: executing set_param: No such device pdsh@oleg331-client: oleg331-server: ssh exited with exit code 19 Reader error: 'No devices found' at 0 PASS 76d (104s) == conf-sanity test 77: comma-separated MGS NIDs and failover node NIDs ========================================================== 06:36:06 (1788863766) pdsh@oleg331-client: oleg331-server: ssh exited with exit code 1 Permanent disk data: Device: lustre-mdt1_2/mdt1_2 Target: test1234:MDT0000 Index: 0 Lustre FS: test1234 Mount type: zfs Flags: 0x65 (MDT MGS first_time update ) Persistent mount opts: Parameters: mgsnode=192.168.203.131@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-mdt1_2 /tmp/lustre-mdt1_2 mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-mdt1_2/mdt1_2 xattr=sa dnodesize=auto compression=off Writing lustre-mdt1_2/mdt1_2 properties lustre:mgsnode=192.168.203.131@tcp lustre:sys.timeout=20 lustre:mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity lustre:version=1 lustre:flags=101 lustre:index=0 lustre:fsname=test1234 lustre:svname=test1234:MDT0000 Start fs2mds: mount -t lustre -o localrecov lustre-mdt1_2/mdt1_2 /mnt/lustre-fs2mds oleg331-server: oleg331-server.virtnet: executing set_default_debug -1 all pdsh@oleg331-client: oleg331-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1_2/mdt1_2 Started test1234-MDT0000 Permanent disk data: Device: lustre-ost1_2/ost1_2 Target: test1234:OST0000 Index: 0 Lustre FS: test1234 Mount type: zfs Flags: 0x62 (OST first_time update ) Persistent mount opts: Parameters: mgsnode=0.0.0.0@tcp,192.168.203.131@tcp,192.168.203.131@tcp:192.168.203.131@tcp:192.168.203.131@tcp sys.timeout=20 failover.node=1.2.3.4@tcp,4.3.2.1@tcp autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1_2 /tmp/lustre-ost1_2 mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1_2/ost1_2 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1_2/ost1_2 properties lustre:mgsnode=0.0.0.0@tcp,192.168.203.131@tcp,192.168.203.131@tcp:192.168.203.131@tcp:192.168.203.131@tcp lustre:sys.timeout=20 lustre:failover.node=1.2.3.4@tcp,4.3.2.1@tcp lustre:autodegrade=on lustre:version=1 lustre:flags=98 lustre:index=0 lustre:fsname=test1234 lustre:svname=test1234:OST0000 /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1877: OST_INDICES: bad array subscript Start fs2ost: mount -t lustre -o localrecov lustre-ost1_2/ost1_2 /mnt/lustre-fs2ost seq.cli-test1234-OST0000-super.width=65536 oleg331-server: oleg331-server.virtnet: executing set_default_debug -1 all pdsh@oleg331-client: oleg331-server: ssh exited with exit code 1 Commit the device label on lustre-ost1_2/ost1_2 Started test1234-OST0000 setup single mount lustre success umount /mnt/lustre2 ... stopping fs2mds ... Stopping /mnt/lustre-fs2mds (opts:-f) on oleg331-server stopping fs2ost ... Stopping /mnt/lustre-fs2ost (opts:-f) on oleg331-server PASS 77 (66s) == conf-sanity test 78: run resize2fs on MDT and OST filesystems ========================================================== 06:37:12 (1788863832) SKIP: conf-sanity test_78 ldiskfs only test SKIP 78 (3s) == conf-sanity test 79: format MDT/OST without mgs option (should return errors) ========================================================== 06:37:15 (1788863835) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg331-server' oleg331-server: oleg331-server.virtnet: executing load_modules_local oleg331-server: kmemleak disabled oleg331-server: Loading modules from /home/green/git/lustre-release/lustre oleg331-server: detected 4 online CPUs by sysfs oleg331-server: MODOPTS_LIBCFS= oleg331-server: Force libcfs to create 2 CPU partitions oleg331-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt oleg331-server: oleg331-server: mkfs.lustre FATAL: Must specify --mgs or --mgsnode oleg331-server: mkfs.lustre: exiting with 22 (Invalid argument) pdsh@oleg331-client: oleg331-server: ssh exited with exit code 22 oleg331-server: oleg331-server: mkfs.lustre FATAL: Must specify --mgs or --mgsnode oleg331-server: mkfs.lustre: exiting with 22 (Invalid argument) pdsh@oleg331-client: oleg331-server: ssh exited with exit code 22 oleg331-server: oleg331-server: mkfs.lustre FATAL: Must specify --mgsnode oleg331-server: mkfs.lustre: exiting with 22 (Invalid argument) pdsh@oleg331-client: oleg331-server: ssh exited with exit code 22 Stopping clients: oleg331-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg331-client.virtnet /mnt/lustre2 (opts:-f) oleg331-server: oleg331-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg331-server' oleg331-server: oleg331-server.virtnet: executing load_modules_local oleg331-server: kmemleak disabled oleg331-server: Loading modules from /home/green/git/lustre-release/lustre oleg331-server: detected 4 online CPUs by sysfs oleg331-server: MODOPTS_LIBCFS= oleg331-server: Force libcfs to create 2 CPU partitions oleg331-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg331-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg331-server: oleg331-server.virtnet: executing set_default_debug -1 all pdsh@oleg331-client: oleg331-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg331-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg331-server: oleg331-server.virtnet: executing set_default_debug -1 all pdsh@oleg331-client: oleg331-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg331-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg331-server: oleg331-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg331-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg331-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg331-server stop mds service on oleg331-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg331-server PASS 79 (163s) == conf-sanity test 80: mgc import reconnect race ======== 06:39:58 (1788863998) start mds service on oleg331-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg331-server: oleg331-server.virtnet: executing set_default_debug -1 all pdsh@oleg331-client: oleg331-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg331-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg331-server: oleg331-server.virtnet: executing set_default_debug -1 all pdsh@oleg331-client: oleg331-server: ssh exited with exit code 1 Started lustre-OST0000 oleg331-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 fail_val=10 fail_loc=0x906 fail_val=10 fail_loc=0x906 start ost2 service on oleg331-server Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg331-server: oleg331-server.virtnet: executing set_default_debug -1 all pdsh@oleg331-client: oleg331-server: ssh exited with exit code 1 Commit the device label on lustre-ost2/ost2 Started lustre-OST0001 oleg331-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 fail_loc=0 stop ost2 service on oleg331-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg331-server stop ost1 service on oleg331-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg331-server stop mds service on oleg331-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg331-server PASS 80 (127s) == conf-sanity test 81: sparse OST indexing ============== 06:42:05 (1788864125) SKIP: conf-sanity test_81 needs >= 3 OSTs SKIP 81 (3s) == conf-sanity test 82a: specify OSTs for file (succeed) or directory (succeed) ========================================================== 06:42:08 (1788864128) SKIP: conf-sanity test_82a needs >= 3 OSTs SKIP 82a (4s) == conf-sanity test 82b: specify OSTs for file with --pool and --ost-list options ========================================================== 06:42:12 (1788864132) SKIP: conf-sanity test_82b needs >= 4 OSTs SKIP 82b (3s) == conf-sanity test 82c: specify sparse OSTs for setstripe (should not crash) ========================================================== 06:42:15 (1788864135) SKIP: conf-sanity test_82c needs >= 4 OSTs SKIP 82c (4s) == conf-sanity test 83: ENOSPACE on OST doesn't cause message VFS: Busy inodes after unmount ... ========================================================== 06:42:19 (1788864139) SKIP: conf-sanity test_83 ldiskfs only test SKIP 83 (3s) == conf-sanity test 84: check recovery_hard_time ========= 06:42:22 (1788864142) start mds service on oleg331-server start mds service on oleg331-server Start mds1: mount -t lustre -o localrecov -o recovery_time_hard=60,recovery_time_soft=60 lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg331-server: oleg331-server.virtnet: executing set_default_debug -1 all pdsh@oleg331-client: oleg331-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg331-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg331-server: oleg331-server.virtnet: executing set_default_debug -1 all pdsh@oleg331-client: oleg331-server: ssh exited with exit code 1 Started lustre-OST0000 oleg331-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg331-server: oleg331-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg331-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec start ost2 service on oleg331-server Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg331-server: oleg331-server.virtnet: executing set_default_debug -1 all pdsh@oleg331-client: oleg331-server: ssh exited with exit code 1 Started lustre-OST0001 oleg331-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg331-server: oleg331-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 oleg331-server: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec recovery_time=60, timeout=20, wrap_up=5 mount lustre on /mnt/lustre..... Starting client: oleg331-client.virtnet: -o user_xattr,flock 192.168.203.131@tcp:/lustre /mnt/lustre mount lustre on /mnt/lustre2..... Starting client: oleg331-client.virtnet: -o user_xattr,flock 192.168.203.131@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 383616 2048 379520 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 382976 2048 378880 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 382976 2048 378880 1% /mnt/lustre[OST:1] filesystem_summary: 765952 4096 757760 1% /mnt/lustre - open/close 489 (time 1788864244.54 total 10.00 last 48.89) - open/close 982 (time 1788864254.56 total 20.01 last 49.25) total: 1000 open/close in 20.36 seconds: 49.12 ops/second fail_loc=0x20000709 fail_val=5 Failing mds1 on oleg331-server Stopping /mnt/lustre-mds1 (opts:) on oleg331-server 06:44:21 (1788864261) shut down facet: mds1 facet_host: oleg331-server facet_failover_host: oleg331-server Failover mds1 to oleg331-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov -o recovery_time_hard=60,recovery_time_soft=60 lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg331-server: oleg331-server.virtnet: executing set_default_debug -1 all pdsh@oleg331-client: oleg331-server: ssh exited with exit code 1 Started lustre-MDT0000 06:44:53 (1788864293) targets are mounted 06:44:53 (1788864293) facet_failover done oleg331-client: error: invalid path '/mnt/lustre': Input/output error pdsh@oleg331-client: oleg331-client: ssh exited with exit code 5 recovery status status: COMPLETE recovery_start: 1788864290 recovery_duration: 60 completed_clients: 1/2 replayed_requests: 189 last_transno: 12884902081 VBR: DISABLED IR: DISABLED fail_loc=0 umount lustre on /mnt/lustre..... Stopping client oleg331-client.virtnet /mnt/lustre (opts:) umount lustre on /mnt/lustre2..... Stopping client oleg331-client.virtnet /mnt/lustre2 (opts:) stop ost1 service on oleg331-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg331-server stop ost2 service on oleg331-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg331-server stop mds service on oleg331-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg331-server PASS 84 (249s) == conf-sanity test 85: osd_ost init: fail ea_fid_set ==== 06:46:31 (1788864391) fail_loc=0x197 start ost1 service on oleg331-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg331-server: oleg331-server.virtnet: executing set_default_debug -1 all pdsh@oleg331-client: oleg331-server: ssh exited with exit code 1 Started lustre-OST0000 oleg331-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 stop ost1 service on oleg331-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg331-server PASS 85 (78s) == conf-sanity test 86: Replacing mkfs.lustre -G option == 06:47:49 (1788864469) SKIP: conf-sanity test_86 LU-6442: no such mkfs params for ZFS OSTs SKIP 86 (3s) == conf-sanity test 87: check if MDT inode can hold EAs with N stripes properly ========================================================== 06:47:52 (1788864472) SKIP: conf-sanity test_87 ldiskfs only test SKIP 87 (4s) == conf-sanity test 88a: tunefs.lustre persists a --mountfsoptions-only change ========================================================== 06:47:57 (1788864477) SKIP: conf-sanity test_88a LU-6662: no implementation for ZFS SKIP 88a (5s) == conf-sanity test 88b: check the default mount options can be overridden ========================================================== 06:48:01 (1788864481) SKIP: conf-sanity test_88b LU-6662: no implementation for ZFS SKIP 88b (3s) == conf-sanity test 89: check tunefs --param and --erase-param{s} options ========================================================== 06:48:04 (1788864484) Failing mds1 on oleg331-server 06:48:09 (1788864489) shut down facet: mds1 facet_host: oleg331-server facet_failover_host: oleg331-server Failover mds1 to oleg331-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg331-server: oleg331-server.virtnet: executing set_default_debug -1 all pdsh@oleg331-client: oleg331-server: ssh exited with exit code 1 Started lustre-MDT0000 06:48:33 (1788864513) targets are mounted 06:48:33 (1788864513) facet_failover done pdsh@oleg331-client: oleg331-client: ssh exited with exit code 95 oleg331-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 pdsh@oleg331-client: oleg331-client: ssh exited with exit code 95 Stopping clients: oleg331-client.virtnet /mnt/lustre (opts:) Stopping clients: oleg331-client.virtnet /mnt/lustre2 (opts:) Stopping /mnt/lustre-mds1 (opts:-f) on oleg331-server tunefs --param failover.node=192.0.2.254@tcp0 tunefs --param failover.node=192.0.2.255@tcp0 tunefs --erase-param failover.node tunefs --erase-params tunefs --param failover.node=192.0.2.254@tcp0 --erase-params Stopping clients: oleg331-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg331-client.virtnet /mnt/lustre2 (opts:-f) oleg331-server: oleg331-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg331-server' oleg331-server: oleg331-server.virtnet: executing load_modules_local oleg331-server: kmemleak disabled oleg331-server: Loading modules from /home/green/git/lustre-release/lustre oleg331-server: detected 4 online CPUs by sysfs oleg331-server: MODOPTS_LIBCFS= oleg331-server: Force libcfs to create 2 CPU partitions oleg331-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg331-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg331-server: oleg331-server.virtnet: executing set_default_debug -1 all pdsh@oleg331-client: oleg331-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg331-client.virtnet: executing wait_import_state_mount FULL failover.node=192.0.2.254@tcp0,mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg331-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg331-server: oleg331-server.virtnet: executing set_default_debug -1 all pdsh@oleg331-client: oleg331-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg331-client.virtnet: executing wait_import_state_mount (FULL|IDLE) failover.node=192.0.2.254@tcp0,osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg331-server: oleg331-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg331-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg331-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg331-server stop mds service on oleg331-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg331-server PASS 89 (181s) == conf-sanity test 90a: check max_mod_rpcs_in_flight is enforced ========================================================== 06:51:05 (1788864665) start mds service on oleg331-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg331-server: oleg331-server.virtnet: executing set_default_debug -1 all pdsh@oleg331-client: oleg331-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg331-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg331-server: oleg331-server.virtnet: executing set_default_debug -1 all pdsh@oleg331-client: oleg331-server: ssh exited with exit code 1 Started lustre-OST0000 oleg331-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg331-client.virtnet: -o user_xattr,flock 192.168.203.131@tcp:/lustre /mnt/lustre max_mod_rcps_in_flight is 7 creating 8 files ... fail_loc=0x159 launch 6 chmod in parallel ... fail_loc=0 launch 1 additional chmod in parallel ... /mnt/lustre/d90a.conf-sanity/file-7 has perms 0600 OK fail_loc=0x159 launch 7 chmod in parallel ... fail_loc=0 launch 1 additional chmod in parallel ... /mnt/lustre/d90a.conf-sanity/file-8 has perms 0644 OK umount lustre on /mnt/lustre..... Stopping client oleg331-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg331-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg331-server stop mds service on oleg331-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg331-server unloading modules via unload_modules_local on: 'oleg331-server' oleg331-server: oleg331-server.virtnet: executing unload_modules_local oleg331-server: kmemleak disabled modules unloaded. pdsh@oleg331-client: oleg331-client: ssh exited with exit code 2 pdsh@oleg331-client: oleg331-server: ssh exited with exit code 2 pdsh@oleg331-client: oleg331-client: ssh exited with exit code 2 pdsh@oleg331-client: oleg331-server: ssh exited with exit code 2 PASS 90a (120s) == conf-sanity test 90b: check max_mod_rpcs_in_flight is enforced after update ========================================================== 06:53:05 (1788864785) start mds service on oleg331-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x 1 green green 1931560 Sep 8 03:54 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg331-server' oleg331-server: oleg331-server.virtnet: executing load_modules_local oleg331-server: kmemleak disabled oleg331-server: Loading modules from /home/green/git/lustre-release/lustre oleg331-server: detected 4 online CPUs by sysfs oleg331-server: MODOPTS_LIBCFS= oleg331-server: Force libcfs to create 2 CPU partitions oleg331-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg331-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg331-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg331-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg331-server: oleg331-server.virtnet: executing set_default_debug -1 all pdsh@oleg331-client: oleg331-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg331-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg331-server: oleg331-server.virtnet: executing set_default_debug -1 all pdsh@oleg331-client: oleg331-server: ssh exited with exit code 1 Started lustre-OST0000 oleg331-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg331-client.virtnet: -o user_xattr,flock 192.168.203.131@tcp:/lustre /mnt/lustre mdc.lustre-MDT0000-mdc-ffff9b5913336000.max_mod_rpcs_in_flight=1 max_mod_rpcs_in_flight set to 1 creating 2 files ... fail_loc=0x159 launch 0 chmod in parallel ... fail_loc=0 launch 1 additional chmod in parallel ... /mnt/lustre/d90b.conf-sanity1/file-1 has perms 0600 OK fail_loc=0x159 launch 1 chmod in parallel ... fail_loc=0 launch 1 additional chmod in parallel ... /mnt/lustre/d90b.conf-sanity1/file-2 has perms 0644 OK mdc.lustre-MDT0000-mdc-ffff9b5913336000.max_mod_rpcs_in_flight=5 max_mod_rpcs_in_flight set to 5 creating 6 files ... fail_loc=0x159 launch 4 chmod in parallel ... fail_loc=0 launch 1 additional chmod in parallel ... /mnt/lustre/d90b.conf-sanity2/file-5 has perms 0600 OK fail_loc=0x159 launch 5 chmod in parallel ... fail_loc=0 launch 1 additional chmod in parallel ... /mnt/lustre/d90b.conf-sanity2/file-6 has perms 0644 OK mdt_max_mod_rpcs_in_flight is 8 umount lustre on /mnt/lustre..... Stopping client oleg331-client.virtnet /mnt/lustre (opts:) error: get_param: param_path 'mdt/*/max_mod_rpcs_in_flight': No such file or directory the deprecated max_mod_rpcs_per_client parameter was involved mount lustre on /mnt/lustre..... Starting client: oleg331-client.virtnet: -o user_xattr,flock 192.168.203.131@tcp:/lustre /mnt/lustre mdc.lustre-MDT0000-mdc-ffff9b5918666800.max_rpcs_in_flight=17 mdc.lustre-MDT0000-mdc-ffff9b5918666800.max_mod_rpcs_in_flight=16 max_mod_rpcs_in_flight set to 16 creating 17 files ... fail_loc=0x159 launch 15 chmod in parallel ... fail_loc=0 launch 1 additional chmod in parallel ... /mnt/lustre/d90b.conf-sanity3/file-16 has perms 0600 OK fail_loc=0x159 launch 16 chmod in parallel ... fail_loc=0 launch 1 additional chmod in parallel ... /mnt/lustre/d90b.conf-sanity3/file-17 has perms 0644 OK error: get_param: param_path 'mdt/*/max_mod_rpcs_in_flight': No such file or directory the deprecated max_mod_rpcs_per_client parameter was involved umount lustre on /mnt/lustre..... Stopping client oleg331-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg331-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg331-server stop mds service on oleg331-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg331-server unloading modules via unload_modules_local on: 'oleg331-server' oleg331-server: oleg331-server.virtnet: executing unload_modules_local oleg331-server: kmemleak disabled modules unloaded. pdsh@oleg331-client: oleg331-client: ssh exited with exit code 2 pdsh@oleg331-client: oleg331-server: ssh exited with exit code 2 pdsh@oleg331-client: oleg331-client: ssh exited with exit code 2 pdsh@oleg331-client: oleg331-server: ssh exited with exit code 2 PASS 90b (202s) == conf-sanity test 90c: check max_mod_rpcs_in_flight update limits ========================================================== 06:56:28 (1788864988) start mds service on oleg331-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x 1 green green 1931560 Sep 8 03:54 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg331-server' oleg331-server: oleg331-server.virtnet: executing load_modules_local oleg331-server: kmemleak disabled oleg331-server: Loading modules from /home/green/git/lustre-release/lustre oleg331-server: detected 4 online CPUs by sysfs oleg331-server: MODOPTS_LIBCFS= oleg331-server: Force libcfs to create 2 CPU partitions oleg331-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg331-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg331-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg331-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg331-server: oleg331-server.virtnet: executing set_default_debug -1 all pdsh@oleg331-client: oleg331-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg331-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg331-server: oleg331-server.virtnet: executing set_default_debug -1 all pdsh@oleg331-client: oleg331-server: ssh exited with exit code 1 Started lustre-OST0000 oleg331-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg331-client.virtnet: -o user_xattr,flock 192.168.203.131@tcp:/lustre /mnt/lustre max_rpcs_in_flight is 8 MDC max_mod_rpcs_in_flight is 7 mdt_max_mod_rpcs_in_flight is 8 error: get_param: param_path 'mdt/*/max_mod_rpcs_in_flight': No such file or directory the deprecated max_mod_rpcs_per_client parameter was involved mdc.lustre-MDT0000-mdc-ffff9b5919306800.max_mod_rpcs_in_flight=8 umount lustre on /mnt/lustre..... Stopping client oleg331-client.virtnet /mnt/lustre (opts:) error: get_param: param_path 'mdt/*/max_mod_rpcs_in_flight': No such file or directory the deprecated max_mod_rpcs_per_client parameter was involved mount lustre on /mnt/lustre..... Starting client: oleg331-client.virtnet: -o user_xattr,flock 192.168.203.131@tcp:/lustre /mnt/lustre mdc.lustre-MDT0000-mdc-ffff9b5910de7800.max_rpcs_in_flight=10 error: set_param: setting /sys/fs/lustre/mdc/lustre-MDT0000-mdc-ffff9b5910de7800/max_mod_rpcs_in_flight=9: Numerical result out of range error: set_param: setting 'mdc/lustre-MDT0000-mdc-*/max_mod_rpcs_in_flight'='9': Numerical result out of range Stopping client oleg331-client.virtnet /mnt/lustre (opts:) Starting client: oleg331-client.virtnet: -o user_xattr,flock 192.168.203.131@tcp:/lustre /mnt/lustre error: get_param: param_path 'mdt/*/max_mod_rpcs_in_flight': No such file or directory the deprecated max_mod_rpcs_per_client parameter was involved umount lustre on /mnt/lustre..... Stopping client oleg331-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg331-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg331-server stop mds service on oleg331-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg331-server unloading modules via unload_modules_local on: 'oleg331-server' oleg331-server: oleg331-server.virtnet: executing unload_modules_local oleg331-server: kmemleak disabled modules unloaded. pdsh@oleg331-client: oleg331-server: ssh exited with exit code 2 pdsh@oleg331-client: oleg331-client: ssh exited with exit code 2 pdsh@oleg331-client: oleg331-server: ssh exited with exit code 2 pdsh@oleg331-client: oleg331-client: ssh exited with exit code 2 PASS 90c (122s) == conf-sanity test 90d: check one close RPC is allowed above max_mod_rpcs_in_flight ========================================================== 06:58:31 (1788865111) start mds service on oleg331-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x 1 green green 1931560 Sep 8 03:54 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg331-server' oleg331-server: oleg331-server.virtnet: executing load_modules_local oleg331-server: kmemleak disabled oleg331-server: Loading modules from /home/green/git/lustre-release/lustre oleg331-server: detected 4 online CPUs by sysfs oleg331-server: MODOPTS_LIBCFS= oleg331-server: Force libcfs to create 2 CPU partitions oleg331-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg331-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg331-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg331-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg331-server: oleg331-server.virtnet: executing set_default_debug -1 all pdsh@oleg331-client: oleg331-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg331-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg331-server: oleg331-server.virtnet: executing set_default_debug -1 all pdsh@oleg331-client: oleg331-server: ssh exited with exit code 1 Started lustre-OST0000 oleg331-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg331-client.virtnet: -o user_xattr,flock 192.168.203.131@tcp:/lustre /mnt/lustre max_mod_rcps_in_flight is 7 creating 7 files ... multiop /mnt/lustre/d90d.conf-sanity/file-close vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7624 fail_loc=0x159 launch 7 chmod in parallel ... fail_loc=0 launch 1 additional close in parallel ... umount lustre on /mnt/lustre..... Stopping client oleg331-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg331-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg331-server stop mds service on oleg331-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg331-server unloading modules via unload_modules_local on: 'oleg331-server' oleg331-server: oleg331-server.virtnet: executing unload_modules_local oleg331-server: kmemleak disabled modules unloaded. pdsh@oleg331-client: oleg331-client: ssh exited with exit code 2 pdsh@oleg331-client: oleg331-server: ssh exited with exit code 2 pdsh@oleg331-client: oleg331-client: ssh exited with exit code 2 pdsh@oleg331-client: oleg331-server: ssh exited with exit code 2 PASS 90d (126s) == conf-sanity test 91: evict-by-nid support ============= 07:00:38 (1788865238) start mds service on oleg331-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x 1 green green 1931560 Sep 8 03:54 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg331-server' oleg331-server: oleg331-server.virtnet: executing load_modules_local oleg331-server: kmemleak disabled oleg331-server: Loading modules from /home/green/git/lustre-release/lustre oleg331-server: detected 4 online CPUs by sysfs oleg331-server: MODOPTS_LIBCFS= oleg331-server: Force libcfs to create 2 CPU partitions oleg331-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg331-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg331-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg331-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg331-server: oleg331-server.virtnet: executing set_default_debug -1 all pdsh@oleg331-client: oleg331-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg331-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg331-server: oleg331-server.virtnet: executing set_default_debug -1 all pdsh@oleg331-client: oleg331-server: ssh exited with exit code 1 Started lustre-OST0000 oleg331-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg331-client.virtnet: -o user_xattr,flock 192.168.203.131@tcp:/lustre /mnt/lustre setup single mount lustre success list nids on mdt: mdt.lustre-MDT0000.exports.0@lo mdt.lustre-MDT0000.exports.192.168.203.31@tcp mdt.lustre-MDT0000.exports.clear uuid from 192\.168\.203\.31@tcp: mdt.lustre-MDT0000.exports.192.168.203.31@tcp.uuid=2e71d0b6-b4c9-44d6-9a76-a19ad07c3380 manual umount lustre on /mnt/lustre.... evict 192\.168\.203\.31@tcp oleg331-server: error: read_param: '/sys/kernel/debug/lustre/mdt/lustre-MDT0000/exports/192.168.203.31@tcp/uuid': No such device pdsh@oleg331-client: oleg331-server: ssh exited with exit code 19 oleg331-server: error: read_param: '/sys/kernel/debug/lustre/obdfilter/lustre-OST0000/exports/192.168.203.31@tcp/uuid': No such device pdsh@oleg331-client: oleg331-server: ssh exited with exit code 19 oleg331-server: error: read_param: '/sys/kernel/debug/lustre/mdt/lustre-MDT0000/exports/192.168.203.31@tcp/uuid': No such device pdsh@oleg331-client: oleg331-server: ssh exited with exit code 19 oleg331-server: error: read_param: '/sys/kernel/debug/lustre/obdfilter/lustre-OST0000/exports/192.168.203.31@tcp/uuid': No such device pdsh@oleg331-client: oleg331-server: ssh exited with exit code 19 umount lustre on /mnt/lustre..... stop ost1 service on oleg331-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg331-server stop mds service on oleg331-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg331-server unloading modules via unload_modules_local on: 'oleg331-server' oleg331-server: oleg331-server.virtnet: executing unload_modules_local oleg331-server: kmemleak disabled modules unloaded. pdsh@oleg331-client: oleg331-client: ssh exited with exit code 2 pdsh@oleg331-client: oleg331-server: ssh exited with exit code 2 pdsh@oleg331-client: oleg331-server: ssh exited with exit code 2 pdsh@oleg331-client: oleg331-client: ssh exited with exit code 2 PASS 91 (142s) == conf-sanity test 92: ldev returns MGS NID correctly in command substitution ========================================================== 07:02:59 (1788865379) Host is oleg331-client.virtnet ----- /tmp/ldev.conf ----- oleg331-server - lustre-MGS0000 zfs:lustre-mdt1/mdt1 oleg331-server - lustre-OST0000 zfs:lustre-ost1/ost1 oleg331-server - lustre-OST0001 zfs:lustre-ost2/ost2 oleg331-server - lustre-MDT0000 zfs:lustre-mdt1/mdt1 --- END /tmp/ldev.conf --- ----- /tmp/nids ----- oleg331-server 192.168.203.131@tcp --- END /tmp/nids --- -- START OF LDEV OUTPUT -- lustre-OST0000: 192.168.203.131@tcp lustre-MGS0000: 192.168.203.131@tcp lustre-OST0001: 192.168.203.131@tcp lustre-MDT0000: 192.168.203.131@tcp --- END OF LDEV OUTPUT --- -- START OF LDEV OUTPUT -- lustre-OST0000: 192.168.203.131@tcp lustre-MDT0000: 192.168.203.131@tcp lustre-MGS0000: 192.168.203.131@tcp lustre-OST0001: 192.168.203.131@tcp --- END OF LDEV OUTPUT --- -- START OF LDEV OUTPUT -- lustre-OST0001: 192.168.203.131@tcp lustre-MDT0000: 192.168.203.131@tcp lustre-MGS0000: 192.168.203.131@tcp lustre-OST0000: 192.168.203.131@tcp --- END OF LDEV OUTPUT --- -- START OF LDEV OUTPUT -- lustre-OST0001: 192.168.203.131@tcp lustre-MDT0000: 192.168.203.131@tcp lustre-OST0000: 192.168.203.131@tcp lustre-MGS0000: 192.168.203.131@tcp --- END OF LDEV OUTPUT --- pdsh@oleg331-client: oleg331-server: ssh exited with exit code 2 pdsh@oleg331-client: oleg331-server: ssh exited with exit code 2 PASS 92 (8s) == conf-sanity test 93: register mulitple MDT at the same time ========================================================== 07:03:07 (1788865387) SKIP: conf-sanity test_93 needs >= 3 MDTs SKIP 93 (3s) == conf-sanity test 94: ldev outputs correct labels for file system name query ========================================================== 07:03:10 (1788865390) ----- /tmp/ldev.conf ----- oleg331-server - lustre-MGS0000 zfs:lustre-mdt1/mdt1 oleg331-server - lustre-OST0000 zfs:lustre-ost1/ost1 oleg331-server - lustre-OST0001 zfs:lustre-ost2/ost2 oleg331-server - lustre-MDT0000 zfs:lustre-mdt1/mdt1 --- END /tmp/ldev.conf --- ----- /tmp/nids ----- oleg331-server 192.168.203.131@tcp --- END /tmp/nids --- -- START OF LDEV OUTPUT -- lustre-MDT0000 lustre-MGS0000 lustre-OST0000 lustre-OST0001 --- END OF LDEV OUTPUT --- -- START OF EXPECTED OUTPUT -- lustre-MDT0000 lustre-MGS0000 lustre-OST0000 lustre-OST0001 --- END OF EXPECTED OUTPUT --- pdsh@oleg331-client: oleg331-server: ssh exited with exit code 2 pdsh@oleg331-client: oleg331-server: ssh exited with exit code 2 PASS 94 (8s) == conf-sanity test 95: ldev should only allow one label filter ========================================================== 07:03:18 (1788865398) ----- /tmp/ldev.conf ----- oleg331-server - lustre-MGS0000 zfs:lustre-mdt1/mdt1 oleg331-server - lustre-OST0000 zfs:lustre-ost1/ost1 oleg331-server - lustre-OST0001 zfs:lustre-ost2/ost2 oleg331-server - lustre-MDT0000 zfs:lustre-mdt1/mdt1 --- END /tmp/ldev.conf --- ----- /tmp/nids ----- oleg331-server 192.168.203.131@tcp --- END /tmp/nids --- pdsh@oleg331-client: oleg331-server: ssh exited with exit code 2 pdsh@oleg331-client: oleg331-server: ssh exited with exit code 2 PASS 95 (7s) == conf-sanity test 96: ldev returns hostname and backend fs correctly in command sub ========================================================== 07:03:25 (1788865405) ----- /tmp/ldev.conf ----- oleg331-server - lustre-MGS0000 zfs:lustre-mdt1/mdt1 oleg331-server - lustre-OST0000 zfs:lustre-ost1/ost1 oleg331-server - lustre-OST0001 zfs:lustre-ost2/ost2 oleg331-server - lustre-MDT0000 zfs:lustre-mdt1/mdt1 --- END /tmp/ldev.conf --- ----- /tmp/nids ----- oleg331-server 192.168.203.131@tcp --- END /tmp/nids --- -- START OF LDEV OUTPUT -- oleg331-server-zfs oleg331-server-zfs oleg331-server-zfs oleg331-server-zfs --- END OF LDEV OUTPUT --- -- START OF EXPECTED OUTPUT -- oleg331-server-zfs oleg331-server-zfs oleg331-server-zfs oleg331-server-zfs --- END OF EXPECTED OUTPUT --- pdsh@oleg331-client: oleg331-server: ssh exited with exit code 2 pdsh@oleg331-client: oleg331-server: ssh exited with exit code 2 PASS 96 (8s) == conf-sanity test 97: ldev returns correct ouput when querying based on role ========================================================== 07:03:33 (1788865413) ----- /tmp/ldev.conf ----- oleg331-server - lustre-MGS0000 zfs:lustre-mdt1/mdt1 oleg331-server - lustre-OST0000 zfs:lustre-ost1/ost1 oleg331-server - lustre-OST0001 zfs:lustre-ost2/ost2 oleg331-server - lustre-MDT0000 zfs:lustre-mdt1/mdt1 --- END /tmp/ldev.conf --- ----- /tmp/nids ----- oleg331-server 192.168.203.131@tcp --- END /tmp/nids --- MDT role -- START OF LDEV OUTPUT -- lustre-MDT0000 --- END OF LDEV OUTPUT --- -- START OF EXPECTED OUTPUT -- lustre-MDT0000 --- END OF EXPECTED OUTPUT --- OST role -- START OF LDEV OUTPUT -- lustre-OST0000 lustre-OST0001 --- END OF LDEV OUTPUT --- -- START OF EXPECTED OUTPUT -- lustre-OST0000 lustre-OST0001 --- END OF EXPECTED OUTPUT --- MGS role -- START OF LDEV OUTPUT -- lustre-MGS0000 --- END OF LDEV OUTPUT --- -- START OF EXPECTED OUTPUT -- lustre-MGS0000 --- END OF EXPECTED OUTPUT --- pdsh@oleg331-client: oleg331-server: ssh exited with exit code 2 pdsh@oleg331-client: oleg331-server: ssh exited with exit code 2 PASS 97 (7s) == conf-sanity test 98: Buffer-overflow check while parsing mount_opts ========================================================== 07:03:40 (1788865420) start mds service on oleg331-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x 1 green green 1931560 Sep 8 03:54 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg331-server' oleg331-server: oleg331-server.virtnet: executing load_modules_local oleg331-server: kmemleak disabled oleg331-server: Loading modules from /home/green/git/lustre-release/lustre oleg331-server: detected 4 online CPUs by sysfs oleg331-server: MODOPTS_LIBCFS= oleg331-server: Force libcfs to create 2 CPU partitions oleg331-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg331-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg331-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg331-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg331-server: oleg331-server.virtnet: executing set_default_debug -1 all pdsh@oleg331-client: oleg331-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg331-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg331-server: oleg331-server.virtnet: executing set_default_debug -1 all pdsh@oleg331-client: oleg331-server: ssh exited with exit code 1 Started lustre-OST0000 oleg331-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg331-client.virtnet: -o user_xattr,flock 192.168.203.131@tcp:/lustre /mnt/lustre setup single mount lustre success arg[0] = mount.lustre arg[1] = -o arg[2] = remount,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr arg[3] = -v arg[4] = 192.168.203.131@tcp:/lustre arg[5] = /mnt/lustre source = 192.168.203.131@tcp:/lustre (192.168.203.131@tcp:/lustre), target = /mnt/lustre options(4517/4096) = remount,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,error: mount options too long, 4517 over limit 4096 user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr options(4517/4096) = remount,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,user_xattr,error: mount options too long, 4517 over limit 4096 umount lustre on /mnt/lustre..... Stopping client oleg331-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg331-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg331-server stop mds service on oleg331-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg331-server unloading modules via unload_modules_local on: 'oleg331-server' oleg331-server: oleg331-server.virtnet: executing unload_modules_local oleg331-server: kmemleak disabled modules unloaded. pdsh@oleg331-client: oleg331-server: ssh exited with exit code 2 pdsh@oleg331-client: oleg331-client: ssh exited with exit code 2 pdsh@oleg331-client: oleg331-client: ssh exited with exit code 2 pdsh@oleg331-client: oleg331-server: ssh exited with exit code 2 PASS 98 (100s) == conf-sanity test 99: Adding meta_bg option ============ 07:05:21 (1788865521) SKIP: conf-sanity test_99 ldiskfs only test SKIP 99 (2s) == conf-sanity test 100: check lshowmount lists MGS, MDT, OST and 0@lo ========================================================== 07:05:24 (1788865524) Stopping clients: oleg331-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg331-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg331-client: oleg331-server: ssh exited with exit code 2 oleg331-server: oleg331-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x 1 green green 1931560 Sep 8 03:54 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg331-server' oleg331-server: oleg331-server.virtnet: executing load_modules_local oleg331-server: kmemleak disabled oleg331-server: Loading modules from /home/green/git/lustre-release/lustre oleg331-server: detected 4 online CPUs by sysfs oleg331-server: MODOPTS_LIBCFS= oleg331-server: Force libcfs to create 2 CPU partitions oleg331-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg331-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg331-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg331-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg331-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg331-server: oleg331-server.virtnet: executing set_default_debug -1 all pdsh@oleg331-client: oleg331-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg331-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg331-server: oleg331-server.virtnet: executing set_default_debug -1 all pdsh@oleg331-client: oleg331-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg331-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg331-client.virtnet: -o user_xattr,flock 192.168.203.131@tcp:/lustre /mnt/lustre setup single mount lustre success lustre-MDT0000: lustre-OST0000: umount lustre on /mnt/lustre..... Stopping client oleg331-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg331-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg331-server stop mds service on oleg331-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg331-server unloading modules via unload_modules_local on: 'oleg331-server' oleg331-server: oleg331-server.virtnet: executing unload_modules_local oleg331-server: kmemleak disabled modules unloaded. pdsh@oleg331-client: oleg331-client: ssh exited with exit code 2 pdsh@oleg331-client: oleg331-server: ssh exited with exit code 2 pdsh@oleg331-client: oleg331-client: ssh exited with exit code 2 pdsh@oleg331-client: oleg331-server: ssh exited with exit code 2 PASS 100 (153s) == conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 07:07:57 (1788865677) start mds service on oleg331-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x 1 green green 1931560 Sep 8 03:54 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg331-server' oleg331-server: oleg331-server.virtnet: executing load_modules_local oleg331-server: kmemleak disabled oleg331-server: Loading modules from /home/green/git/lustre-release/lustre oleg331-server: detected 4 online CPUs by sysfs oleg331-server: MODOPTS_LIBCFS= oleg331-server: Force libcfs to create 2 CPU partitions oleg331-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg331-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg331-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg331-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg331-server: oleg331-server.virtnet: executing set_default_debug -1 all pdsh@oleg331-client: oleg331-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg331-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg331-server: oleg331-server.virtnet: executing set_default_debug -1 all pdsh@oleg331-client: oleg331-server: ssh exited with exit code 1 Started lustre-OST0000 oleg331-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg331-client.virtnet: -o user_xattr,flock 192.168.203.131@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 139 (time 1788865762.89 total 10.57 last 13.16) - open/close 384 (time 1788865773.75 total 21.43 last 22.56) - open/close 595 (time 1788865784.13 total 31.81 last 20.33) - open/close 766 (time 1788865794.14 total 41.82 last 17.08) - open/close 943 (time 1788865804.16 total 51.84 last 17.67) - open/close 1085 (time 1788865814.85 total 62.53 last 13.28) - open/close 1219 (time 1788865824.91 total 72.59 last 13.32) - open/close 1432 (time 1788865835.13 total 82.81 last 20.84) - open/close 1633 (time 1788865845.56 total 93.24 last 19.28) - open/close 1853 (time 1788865855.69 total 103.37 last 21.72) - open/close 2051 (time 1788865865.71 total 113.39 last 19.75) - open/close 2222 (time 1788865875.72 total 123.40 last 17.09) - open/close 2471 (time 1788865885.73 total 133.41 last 24.88) - open/close 2708 (time 1788865895.73 total 143.41 last 23.70) - open/close 2846 (time 1788865905.73 total 153.41 last 13.79) - open/close 3055 (time 1788865915.74 total 163.42 last 20.88) - open/close 3289 (time 1788865925.75 total 173.43 last 23.40) - open/close 3471 (time 1788865935.78 total 183.46 last 18.14) - open/close 3684 (time 1788865946.26 total 193.94 last 20.32) - open/close 3899 (time 1788865956.34 total 204.02 last 21.32) - open/close 4112 (time 1788865966.77 total 214.45 last 20.42) - open/close 4271 (time 1788865976.79 total 224.47 last 15.87) - open/close 4463 (time 1788865987.12 total 234.80 last 18.58) - open/close 4692 (time 1788865997.39 total 245.07 last 22.32) - open/close 4941 (time 1788866008.25 total 255.93 last 22.92) - open/close 5202 (time 1788866018.26 total 265.94 last 26.08) - open/close 5415 (time 1788866028.53 total 276.21 last 20.73) - open/close 5580 (time 1788866038.53 total 286.21 last 16.50) - open/close 5709 (time 1788866049.31 total 296.99 last 11.97) - open/close 5827 (time 1788866061.06 total 308.74 last 10.04) - open/close 6025 (time 1788866072.13 total 319.81 last 17.89) - open/close 6287 (time 1788866082.45 total 330.13 last 25.39) - open/close 6490 (time 1788866092.49 total 340.17 last 20.23) - open/close 6684 (time 1788866102.60 total 350.28 last 19.18) - open/close 6870 (time 1788866112.62 total 360.30 last 18.58) - open/close 7095 (time 1788866123.26 total 370.93 last 21.15) - open/close 7238 (time 1788866133.72 total 381.40 last 13.66) - open/close 7395 (time 1788866143.73 total 391.41 last 15.69) - open/close 7610 (time 1788866154.82 total 402.50 last 19.40) - open/close 7849 (time 1788866165.63 total 413.31 last 22.10) - open/close 8093 (time 1788866175.64 total 423.32 last 24.38) - open/close 8300 (time 1788866185.64 total 433.32 last 20.70) - open/close 8553 (time 1788866196.54 total 444.22 last 23.22) - open/close 8793 (time 1788866206.54 total 454.22 last 23.99) - open/close 9051 (time 1788866216.54 total 464.22 last 25.79) - open/close 9251 (time 1788866226.55 total 474.23 last 19.99) - open/close 9486 (time 1788866237.07 total 484.74 last 22.34) - open/close 9708 (time 1788866247.13 total 494.81 last 22.05) - open/close 9973 (time 1788866258.50 total 506.18 last 23.32) - open/close 10188 (time 1788866268.52 total 516.20 last 21.46) - open/close 10409 (time 1788866278.52 total 526.20 last 22.10) - open/close 10618 (time 1788866288.76 total 536.44 last 20.40) - open/close 10842 (time 1788866299.01 total 546.69 last 21.86) - open/close 11067 (time 1788866309.44 total 557.12 last 21.57) - open/close 11299 (time 1788866320.40 total 568.08 last 21.18) - open/close 11457 (time 1788866330.77 total 578.45 last 15.24) - open/close 11633 (time 1788866341.75 total 589.43 last 16.02) - open/close 11827 (time 1788866351.76 total 599.44 last 19.38) - open/close 12041 (time 1788866362.23 total 609.91 last 20.43) - open/close 12289 (time 1788866373.06 total 620.74 last 22.91) - open/close 12545 (time 1788866383.07 total 630.75 last 25.57) - open/close 12798 (time 1788866393.35 total 641.03 last 24.61) - open/close 12993 (time 1788866403.96 total 651.64 last 18.38) - open/close 13239 (time 1788866414.01 total 661.69 last 24.48) - open/close 13434 (time 1788866425.11 total 672.79 last 17.56) - open/close 13683 (time 1788866435.83 total 683.51 last 23.23) - open/close 13918 (time 1788866446.40 total 694.08 last 22.23) - open/close 14151 (time 1788866456.40 total 704.08 last 23.30) - open/close 14372 (time 1788866466.42 total 714.10 last 22.06) - open/close 14594 (time 1788866476.87 total 724.55 last 21.24) - open/close 14827 (time 1788866487.35 total 735.03 last 22.23) - open/close 15039 (time 1788866497.78 total 745.46 last 20.32) - open/close 15253 (time 1788866508.30 total 755.98 last 20.35) - open/close 15343 (time 1788866518.32 total 766.00 last 8.98) - open/close 15551 (time 1788866528.33 total 776.01 last 20.78) - open/close 15737 (time 1788866539.14 total 786.82 last 17.21) - open/close 15994 (time 1788866549.14 total 796.82 last 25.69) - open/close 16225 (time 1788866560.09 total 807.77 last 21.11) - open/close 16421 (time 1788866570.65 total 818.32 last 18.56) - open/close 16631 (time 1788866580.93 total 828.61 last 20.43) - open/close 16786 (time 1788866591.34 total 839.02 last 14.88) - open/close 16960 (time 1788866601.35 total 849.03 last 17.39) - open/close 17140 (time 1788866611.96 total 859.64 last 16.97) - open/close 17372 (time 1788866623.03 total 870.71 last 20.94) - open/close 17584 (time 1788866633.73 total 881.41 last 19.82) - open/close 17762 (time 1788866643.75 total 891.43 last 17.77) - open/close 17839 (time 1788866653.95 total 901.63 last 7.55) - open/close 18036 (time 1788866663.96 total 911.64 last 19.68) - open/close 18241 (time 1788866674.37 total 922.05 last 19.68) - open/close 18495 (time 1788866685.24 total 932.92 last 23.38) - open/close 19041 (time 1788866695.24 total 942.92 last 54.56) - open/close 19839 (time 1788866705.26 total 952.94 last 79.64) - open/close 20000 (time 1788866707.66 total 955.34 last 67.17) - open/close 20676 (time 1788866717.68 total 965.36 last 67.49) - open/close 21356 (time 1788866727.70 total 975.38 last 67.86) - open/close 22011 (time 1788866737.71 total 985.39 last 65.40) - open/close 22700 (time 1788866747.72 total 995.40 last 68.83) - open/close 23359 (time 1788866757.73 total 1005.41 last 65.86) - open/close 24058 (time 1788866767.77 total 1015.45 last 69.59) - open/close 24753 (time 1788866777.78 total 1025.46 last 69.48) - open/close 25470 (time 1788866787.79 total 1035.47 last 71.62) - open/close 26174 (time 1788866797.79 total 1045.47 last 70.37) - open/close 26887 (time 1788866807.80 total 1055.48 last 71.24) - open/close 27597 (time 1788866817.80 total 1065.48 last 70.98) - open/close 28264 (time 1788866827.81 total 1075.49 last 66.67) - open/close 28939 (time 1788866837.82 total 1085.50 last 67.45) - open/close 29620 (time 1788866847.82 total 1095.50 last 68.08) - open/close 30000 (time 1788866853.28 total 1100.96 last 69.58) - open/close 30609 (time 1788866863.29 total 1110.97 last 60.82) - open/close 31230 (time 1788866873.30 total 1120.97 last 62.08) - open/close 31847 (time 1788866883.31 total 1130.99 last 61.58) - open/close 32526 (time 1788866893.32 total 1141.00 last 67.88) - open/close 33159 (time 1788866903.32 total 1151.00 last 63.27) - open/close 33834 (time 1788866913.35 total 1161.03 last 67.33) - open/close 34509 (time 1788866923.36 total 1171.04 last 67.42) - open/close 35153 (time 1788866933.36 total 1181.04 last 64.35) - open/close 35798 (time 1788866943.38 total 1191.06 last 64.42) - open/close 36472 (time 1788866953.38 total 1201.06 last 67.37) - open/close 37110 (time 1788866963.38 total 1211.06 last 63.79) - open/close 37802 (time 1788866973.39 total 1221.07 last 69.17) - open/close 38480 (time 1788866983.39 total 1231.07 last 67.78) - open/close 38965 (time 1788866993.39 total 1241.07 last 48.49) - open/close 39480 (time 1788867003.43 total 1251.11 last 51.31) - open/close 40000 (time 1788867012.85 total 1260.53 last 55.24) - open/close 40669 (time 1788867022.85 total 1270.53 last 66.89) - open/close 41256 (time 1788867032.86 total 1280.54 last 58.62) - open/close 41899 (time 1788867042.87 total 1290.54 last 64.27) - open/close 42558 (time 1788867052.87 total 1300.55 last 65.84) - open/close 43204 (time 1788867062.88 total 1310.56 last 64.55) - open/close 43887 (time 1788867072.91 total 1320.59 last 68.08) - open/close 44481 (time 1788867082.91 total 1330.59 last 59.40) - open/close 45039 (time 1788867092.92 total 1340.60 last 55.79) - open/close 45664 (time 1788867102.93 total 1350.61 last 62.39) - open/close 46332 (time 1788867112.94 total 1360.62 last 66.74) - open/close 46991 (time 1788867122.96 total 1370.63 last 65.82) - open/close 47670 (time 1788867132.96 total 1380.64 last 67.88) - open/close 48310 (time 1788867142.97 total 1390.65 last 63.92) - open/close 48958 (time 1788867152.97 total 1400.65 last 64.80) - open/close 49554 (time 1788867162.97 total 1410.65 last 59.59) total: 50000 open/close in 1417.65 seconds: 35.27 ops/second - unlinked 0 (time 1788867174 ; total 0 ; last 0) - unlinked 10000 (time 1788867333 ; total 159 ; last 159) - unlinked 20000 (time 1788867490 ; total 316 ; last 157) - unlinked 30000 (time 1788867647 ; total 473 ; last 157) - unlinked 40000 (time 1788867813 ; total 639 ; last 166) total: 50000 unlinks in 797 seconds: 62.735256 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg331-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg331-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg331-server stop mds service on oleg331-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg331-server unloading modules via unload_modules_local on: 'oleg331-server' oleg331-server: oleg331-server.virtnet: executing unload_modules_local oleg331-server: kmemleak disabled modules unloaded. pdsh@oleg331-client: oleg331-client: ssh exited with exit code 2 pdsh@oleg331-client: oleg331-server: ssh exited with exit code 2 pdsh@oleg331-client: oleg331-client: ssh exited with exit code 2 pdsh@oleg331-client: oleg331-server: ssh exited with exit code 2 PASS 101a (2342s) == conf-sanity test 101b: Race events DISCONNECT and ACTIVE in osp ========================================================== 07:46:59 (1788868019) start mds service on oleg331-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x 1 green green 1931560 Sep 8 03:54 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg331-server' oleg331-server: oleg331-server.virtnet: executing load_modules_local oleg331-server: kmemleak disabled oleg331-server: Loading modules from /home/green/git/lustre-release/lustre oleg331-server: detected 4 online CPUs by sysfs oleg331-server: MODOPTS_LIBCFS= oleg331-server: Force libcfs to create 2 CPU partitions oleg331-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg331-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg331-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg331-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg331-server: oleg331-server.virtnet: executing set_default_debug -1 all pdsh@oleg331-client: oleg331-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg331-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg331-server: oleg331-server.virtnet: executing set_default_debug -1 all pdsh@oleg331-client: oleg331-server: ssh exited with exit code 1 Started lustre-OST0000 oleg331-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg331-client.virtnet: -o user_xattr,flock 192.168.203.131@tcp:/lustre /mnt/lustre fail_loc=0x80002107 fail_val=20 stop ost1 service on oleg331-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg331-server start ost1 service on oleg331-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg331-server: oleg331-server.virtnet: executing set_default_debug -1 all pdsh@oleg331-client: oleg331-server: ssh exited with exit code 1 Started lustre-OST0000 oleg331-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec oleg331-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9b59095d2800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9b59095d2800.ost_server_uuid in FULL state after 0 sec umount lustre on /mnt/lustre..... Stopping client oleg331-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg331-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg331-server stop mds service on oleg331-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg331-server unloading modules via unload_modules_local on: 'oleg331-server' oleg331-server: oleg331-server.virtnet: executing unload_modules_local oleg331-server: kmemleak disabled modules unloaded. pdsh@oleg331-client: oleg331-server: ssh exited with exit code 2 pdsh@oleg331-client: oleg331-client: ssh exited with exit code 2 pdsh@oleg331-client: oleg331-client: ssh exited with exit code 2 pdsh@oleg331-client: oleg331-server: ssh exited with exit code 2 PASS 101b (175s) SKIP: conf-sanity test_102 skipping excluded test 102 == conf-sanity test 103: rename filesystem name ========== 07:49:57 (1788868197) Checking servers environments Checking clients oleg331-client.virtnet environments Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x 1 green green 1931560 Sep 8 03:54 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg331-server' oleg331-server: oleg331-server.virtnet: executing load_modules_local oleg331-server: kmemleak disabled oleg331-server: Loading modules from /home/green/git/lustre-release/lustre oleg331-server: detected 4 online CPUs by sysfs oleg331-server: MODOPTS_LIBCFS= oleg331-server: Force libcfs to create 2 CPU partitions oleg331-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg331-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg331-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg331-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Setup mgs, mdt, osts Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg331-server: oleg331-server.virtnet: executing set_default_debug -1 all pdsh@oleg331-client: oleg331-server: ssh exited with exit code 1 Started lustre-MDT0000 Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg331-server: oleg331-server.virtnet: executing set_default_debug -1 all pdsh@oleg331-client: oleg331-server: ssh exited with exit code 1 Started lustre-OST0000 Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg331-server: oleg331-server.virtnet: executing set_default_debug -1 all pdsh@oleg331-client: oleg331-server: ssh exited with exit code 1 Commit the device label on lustre-ost2/ost2 Started lustre-OST0001 mount lustre on /mnt/lustre..... Starting client: oleg331-client.virtnet: -o user_xattr,flock 192.168.203.131@tcp:/lustre /mnt/lustre Starting client oleg331-client.virtnet: -o user_xattr,flock 192.168.203.131@tcp:/lustre /mnt/lustre Started clients oleg331-client.virtnet: 192.168.203.131@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9b59138ed000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9b59138ed000.idle_timeout=debug setting jobstats to procname_uid Setting lustre.sys.jobid_var from disable to procname_uid Waiting 90s for 'procname_uid' Updated after 7s: want 'procname_uid' got 'procname_uid' disable quota as required oleg331-server: Pool lustre.pool1 created oleg331-server: Pool lustre.lustre created oleg331-server: OST lustre-OST0000_UUID added to pool lustre.lustre Waiting 90s for 'lustre-OST0000_UUID' Stopping clients: oleg331-client.virtnet /mnt/lustre (opts:) Stopping client oleg331-client.virtnet /mnt/lustre opts: Stopping clients: oleg331-client.virtnet /mnt/lustre2 (opts:) Stopping /mnt/lustre-mds1 (opts:-f) on oleg331-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg331-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg331-server rename lustre to mylustre checking for existing Lustre data: found Read previous values: Device: lustre-mdt1/mdt1 Target: lustre-MDT0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x5 (MDT MGS ) Persistent mount opts: Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity Permanent disk data: Device: lustre-mdt1/mdt1 Target: mylustre-MDT0000 Index: 0 Lustre FS: mylustre Mount type: zfs Flags: 0x5 (MDT MGS ) Persistent mount opts: Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity lustre:svname=mylustre-MDT0000 cmd: zfs set mountpoint=/tmp/mntDqprOH lustre-mdt1/mdt1 && zfs set canmount=on lustre-mdt1/mdt1 && zfs mount lustre-mdt1/mdt1 cmd: zfs umount lustre-mdt1/mdt1 && zfs set canmount=off lustre-mdt1/mdt1 && zfs set mountpoint=none lustre-mdt1/mdt1 Writing lustre-mdt1/mdt1 properties lustre:sys.timeout=20 lustre:mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity lustre:version=1 lustre:flags=5 lustre:index=0 lustre:fsname=mylustre lustre:svname=mylustre-MDT0000 checking for existing Lustre data: found Read previous values: Device: lustre-ost1/ost1 Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x2 (OST ) Persistent mount opts: Parameters: autodegrade=on sys.timeout=20 mgsnode=192.168.203.131@tcp Permanent disk data: Device: lustre-ost1/ost1 Target: mylustre-OST0000 Index: 0 Lustre FS: mylustre Mount type: zfs Flags: 0x2 (OST ) Persistent mount opts: Parameters: autodegrade=on sys.timeout=20 mgsnode=192.168.203.131@tcp autodegrade=on lustre:svname=mylustre-OST0000 cmd: zfs set mountpoint=/tmp/mntnWVJPk lustre-ost1/ost1 && zfs set canmount=on lustre-ost1/ost1 && zfs mount lustre-ost1/ost1 cmd: zfs umount lustre-ost1/ost1 && zfs set canmount=off lustre-ost1/ost1 && zfs set mountpoint=none lustre-ost1/ost1 Writing lustre-ost1/ost1 properties lustre:autodegrade=on lustre:sys.timeout=20 lustre:mgsnode=192.168.203.131@tcp lustre:autodegrade=on lustre:version=1 lustre:flags=2 lustre:index=0 lustre:fsname=mylustre lustre:svname=mylustre-OST0000 checking for existing Lustre data: found Read previous values: Device: lustre-ost2/ost2 Target: lustre-OST0001 Index: 1 Lustre FS: lustre Mount type: zfs Flags: 0x2 (OST ) Persistent mount opts: Parameters: autodegrade=on sys.timeout=20 mgsnode=192.168.203.131@tcp Permanent disk data: Device: lustre-ost2/ost2 Target: mylustre-OST0001 Index: 1 Lustre FS: mylustre Mount type: zfs Flags: 0x2 (OST ) Persistent mount opts: Parameters: autodegrade=on sys.timeout=20 mgsnode=192.168.203.131@tcp autodegrade=on lustre:svname=mylustre-OST0001 cmd: zfs set mountpoint=/tmp/mntuYeaka lustre-ost2/ost2 && zfs set canmount=on lustre-ost2/ost2 && zfs mount lustre-ost2/ost2 cmd: zfs umount lustre-ost2/ost2 && zfs set canmount=off lustre-ost2/ost2 && zfs set mountpoint=none lustre-ost2/ost2 Writing lustre-ost2/ost2 properties lustre:autodegrade=on lustre:sys.timeout=20 lustre:mgsnode=192.168.203.131@tcp lustre:autodegrade=on lustre:version=1 lustre:flags=2 lustre:index=1 lustre:fsname=mylustre lustre:svname=mylustre-OST0001 Checking servers environments Checking clients oleg331-client.virtnet environments Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg331-server' oleg331-server: oleg331-server.virtnet: executing load_modules_local oleg331-server: kmemleak disabled oleg331-server: Loading modules from /home/green/git/lustre-release/lustre oleg331-server: detected 4 online CPUs by sysfs oleg331-server: MODOPTS_LIBCFS= oleg331-server: Force libcfs to create 2 CPU partitions oleg331-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Setup mgs, mdt, osts Start mds1: mount -t lustre -o localrecov mylustre-mdt1/mdt1 /mnt/lustre-mds1 oleg331-server: oleg331-server.virtnet: executing set_default_debug -1 all pdsh@oleg331-client: oleg331-server: ssh exited with exit code 1 Started mylustre-MDT0000 Start ost1: mount -t lustre -o localrecov mylustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-mylustre-OST0000-super.width=65536 oleg331-server: oleg331-server.virtnet: executing set_default_debug -1 all pdsh@oleg331-client: oleg331-server: ssh exited with exit code 1 Started mylustre-OST0000 Start ost2: mount -t lustre -o localrecov mylustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-mylustre-OST0001-super.width=65536 oleg331-server: oleg331-server.virtnet: executing set_default_debug -1 all pdsh@oleg331-client: oleg331-server: ssh exited with exit code 1 Started mylustre-OST0001 mount mylustre on /mnt/lustre..... Starting client: oleg331-client.virtnet: -o user_xattr,flock 192.168.203.131@tcp:/mylustre /mnt/lustre Starting client oleg331-client.virtnet: -o user_xattr,flock 192.168.203.131@tcp:/mylustre /mnt/lustre Started clients oleg331-client.virtnet: 192.168.203.131@tcp:/mylustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Using TIMEOUT=20 osc.mylustre-OST0000-osc-ffff9b593838b800.idle_timeout=debug osc.mylustre-OST0001-osc-ffff9b593838b800.idle_timeout=debug disable quota as required File: /mnt/lustre/d103.conf-sanity/test-framework.sh Size: 319779 Blocks: 1027 IO Block: 4194304 regular file Device: c3aa56ceh/3282720462d Inode: 144115305935798275 Links: 1 Access: (0755/-rwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-08 07:51:41.000000000 -0400 Modify: 2026-09-08 07:51:41.000000000 -0400 Change: 2026-09-08 07:51:41.000000000 -0400 Birth: 2026-09-08 07:51:42.000000000 -0400 Pool: mylustre.pool1 Pool: mylustre.lustre mylustre-OST0000_UUID mylustre-OST0000_UUID oleg331-server: OST mylustre-OST0001_UUID added to pool mylustre.lustre Stopping clients: oleg331-client.virtnet /mnt/lustre (opts:) Stopping client oleg331-client.virtnet /mnt/lustre opts: Stopping clients: oleg331-client.virtnet /mnt/lustre2 (opts:) Stopping /mnt/lustre-mds1 (opts:-f) on oleg331-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg331-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg331-server rename mylustre to tfs checking for existing Lustre data: found Read previous values: Device: mylustre-mdt1/mdt1 Target: mylustre-MDT0000 Index: 0 Lustre FS: mylustre Mount type: zfs Flags: 0x5 (MDT MGS ) Persistent mount opts: Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity Permanent disk data: Device: mylustre-mdt1/mdt1 Target: tfs-MDT0000 Index: 0 Lustre FS: tfs Mount type: zfs Flags: 0x5 (MDT MGS ) Persistent mount opts: Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity lustre:svname=tfs-MDT0000 cmd: zfs set mountpoint=/tmp/mntB0Bjgg mylustre-mdt1/mdt1 && zfs set canmount=on mylustre-mdt1/mdt1 && zfs mount mylustre-mdt1/mdt1 cmd: zfs umount mylustre-mdt1/mdt1 && zfs set canmount=off mylustre-mdt1/mdt1 && zfs set mountpoint=none mylustre-mdt1/mdt1 Writing mylustre-mdt1/mdt1 properties lustre:sys.timeout=20 lustre:mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity lustre:version=1 lustre:flags=5 lustre:index=0 lustre:fsname=tfs lustre:svname=tfs-MDT0000 checking for existing Lustre data: found Read previous values: Device: mylustre-ost1/ost1 Target: mylustre-OST0000 Index: 0 Lustre FS: mylustre Mount type: zfs Flags: 0x2 (OST ) Persistent mount opts: Parameters: autodegrade=on sys.timeout=20 mgsnode=192.168.203.131@tcp Permanent disk data: Device: mylustre-ost1/ost1 Target: tfs-OST0000 Index: 0 Lustre FS: tfs Mount type: zfs Flags: 0x2 (OST ) Persistent mount opts: Parameters: autodegrade=on sys.timeout=20 mgsnode=192.168.203.131@tcp autodegrade=on lustre:svname=tfs-OST0000 cmd: zfs set mountpoint=/tmp/mntAjHuFW mylustre-ost1/ost1 && zfs set canmount=on mylustre-ost1/ost1 && zfs mount mylustre-ost1/ost1 cmd: zfs umount mylustre-ost1/ost1 && zfs set canmount=off mylustre-ost1/ost1 && zfs set mountpoint=none mylustre-ost1/ost1 Writing mylustre-ost1/ost1 properties lustre:autodegrade=on lustre:sys.timeout=20 lustre:mgsnode=192.168.203.131@tcp lustre:autodegrade=on lustre:version=1 lustre:flags=2 lustre:index=0 lustre:fsname=tfs lustre:svname=tfs-OST0000 checking for existing Lustre data: found Read previous values: Device: mylustre-ost2/ost2 Target: mylustre-OST0001 Index: 1 Lustre FS: mylustre Mount type: zfs Flags: 0x2 (OST ) Persistent mount opts: Parameters: autodegrade=on sys.timeout=20 mgsnode=192.168.203.131@tcp Permanent disk data: Device: mylustre-ost2/ost2 Target: tfs-OST0001 Index: 1 Lustre FS: tfs Mount type: zfs Flags: 0x2 (OST ) Persistent mount opts: Parameters: autodegrade=on sys.timeout=20 mgsnode=192.168.203.131@tcp autodegrade=on lustre:svname=tfs-OST0001 cmd: zfs set mountpoint=/tmp/mntYfMTkT mylustre-ost2/ost2 && zfs set canmount=on mylustre-ost2/ost2 && zfs mount mylustre-ost2/ost2 cmd: zfs umount mylustre-ost2/ost2 && zfs set canmount=off mylustre-ost2/ost2 && zfs set mountpoint=none mylustre-ost2/ost2 Writing mylustre-ost2/ost2 properties lustre:autodegrade=on lustre:sys.timeout=20 lustre:mgsnode=192.168.203.131@tcp lustre:autodegrade=on lustre:version=1 lustre:flags=2 lustre:index=1 lustre:fsname=tfs lustre:svname=tfs-OST0001 Checking servers environments Checking clients oleg331-client.virtnet environments Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg331-server' oleg331-server: oleg331-server.virtnet: executing load_modules_local oleg331-server: kmemleak disabled oleg331-server: Loading modules from /home/green/git/lustre-release/lustre oleg331-server: detected 4 online CPUs by sysfs oleg331-server: MODOPTS_LIBCFS= oleg331-server: Force libcfs to create 2 CPU partitions oleg331-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Setup mgs, mdt, osts Start mds1: mount -t lustre -o localrecov tfs-mdt1/mdt1 /mnt/lustre-mds1 oleg331-server: oleg331-server.virtnet: executing set_default_debug -1 all pdsh@oleg331-client: oleg331-server: ssh exited with exit code 1 Started tfs-MDT0000 Start ost1: mount -t lustre -o localrecov tfs-ost1/ost1 /mnt/lustre-ost1 seq.cli-tfs-OST0000-super.width=65536 oleg331-server: oleg331-server.virtnet: executing set_default_debug -1 all pdsh@oleg331-client: oleg331-server: ssh exited with exit code 1 Started tfs-OST0000 Start ost2: mount -t lustre -o localrecov tfs-ost2/ost2 /mnt/lustre-ost2 seq.cli-tfs-OST0001-super.width=65536 oleg331-server: oleg331-server.virtnet: executing set_default_debug -1 all pdsh@oleg331-client: oleg331-server: ssh exited with exit code 1 Started tfs-OST0001 mount tfs on /mnt/lustre..... Starting client: oleg331-client.virtnet: -o user_xattr,flock 192.168.203.131@tcp:/tfs /mnt/lustre Starting client oleg331-client.virtnet: -o user_xattr,flock 192.168.203.131@tcp:/tfs /mnt/lustre Started clients oleg331-client.virtnet: 192.168.203.131@tcp:/tfs on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Using TIMEOUT=20 osc.tfs-OST0000-osc-ffff9b5910d2f000.idle_timeout=debug osc.tfs-OST0001-osc-ffff9b5910d2f000.idle_timeout=debug disable quota as required File: /mnt/lustre/d103.conf-sanity/test-framework.sh Size: 319779 Blocks: 1027 IO Block: 4194304 regular file Device: 32e2fa5ah/853736026d Inode: 144115305935798275 Links: 1 Access: (0755/-rwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-08 07:51:41.000000000 -0400 Modify: 2026-09-08 07:51:41.000000000 -0400 Change: 2026-09-08 07:51:41.000000000 -0400 Birth: 2026-09-08 07:51:42.000000000 -0400 Pool: tfs.pool1 Pool: tfs.lustre tfs-OST0000_UUID tfs-OST0001_UUID tfs-OST0000_UUID Stopping clients: oleg331-client.virtnet /mnt/lustre (opts:) Stopping client oleg331-client.virtnet /mnt/lustre opts: Stopping clients: oleg331-client.virtnet /mnt/lustre2 (opts:) Stopping /mnt/lustre-mds1 (opts:-f) on oleg331-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg331-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg331-server rename tfs to lustre checking for existing Lustre data: found Read previous values: Device: tfs-mdt1/mdt1 Target: tfs-MDT0000 Index: 0 Lustre FS: tfs Mount type: zfs Flags: 0x5 (MDT MGS ) Persistent mount opts: Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity Permanent disk data: Device: tfs-mdt1/mdt1 Target: lustre-MDT0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x5 (MDT MGS ) Persistent mount opts: Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity lustre:svname=lustre-MDT0000 cmd: zfs set mountpoint=/tmp/mntPC4Hy9 tfs-mdt1/mdt1 && zfs set canmount=on tfs-mdt1/mdt1 && zfs mount tfs-mdt1/mdt1 cmd: zfs umount tfs-mdt1/mdt1 && zfs set canmount=off tfs-mdt1/mdt1 && zfs set mountpoint=none tfs-mdt1/mdt1 Writing tfs-mdt1/mdt1 properties lustre:sys.timeout=20 lustre:mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity lustre:version=1 lustre:flags=5 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-MDT0000 checking for existing Lustre data: found Read previous values: Device: tfs-ost1/ost1 Target: tfs-OST0000 Index: 0 Lustre FS: tfs Mount type: zfs Flags: 0x2 (OST ) Persistent mount opts: Parameters: autodegrade=on sys.timeout=20 mgsnode=192.168.203.131@tcp Permanent disk data: Device: tfs-ost1/ost1 Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x2 (OST ) Persistent mount opts: Parameters: autodegrade=on sys.timeout=20 mgsnode=192.168.203.131@tcp autodegrade=on lustre:svname=lustre-OST0000 cmd: zfs set mountpoint=/tmp/mnt4OcIzN tfs-ost1/ost1 && zfs set canmount=on tfs-ost1/ost1 && zfs mount tfs-ost1/ost1 cmd: zfs umount tfs-ost1/ost1 && zfs set canmount=off tfs-ost1/ost1 && zfs set mountpoint=none tfs-ost1/ost1 Writing tfs-ost1/ost1 properties lustre:autodegrade=on lustre:sys.timeout=20 lustre:mgsnode=192.168.203.131@tcp lustre:autodegrade=on lustre:version=1 lustre:flags=2 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 checking for existing Lustre data: found Read previous values: Device: tfs-ost2/ost2 Target: tfs-OST0001 Index: 1 Lustre FS: tfs Mount type: zfs Flags: 0x2 (OST ) Persistent mount opts: Parameters: autodegrade=on sys.timeout=20 mgsnode=192.168.203.131@tcp Permanent disk data: Device: tfs-ost2/ost2 Target: lustre-OST0001 Index: 1 Lustre FS: lustre Mount type: zfs Flags: 0x2 (OST ) Persistent mount opts: Parameters: autodegrade=on sys.timeout=20 mgsnode=192.168.203.131@tcp autodegrade=on lustre:svname=lustre-OST0001 cmd: zfs set mountpoint=/tmp/mntHISynM tfs-ost2/ost2 && zfs set canmount=on tfs-ost2/ost2 && zfs mount tfs-ost2/ost2 cmd: zfs umount tfs-ost2/ost2 && zfs set canmount=off tfs-ost2/ost2 && zfs set mountpoint=none tfs-ost2/ost2 Writing tfs-ost2/ost2 properties lustre:autodegrade=on lustre:sys.timeout=20 lustre:mgsnode=192.168.203.131@tcp lustre:autodegrade=on lustre:version=1 lustre:flags=2 lustre:index=1 lustre:fsname=lustre lustre:svname=lustre-OST0001 Checking servers environments Checking clients oleg331-client.virtnet environments Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg331-server' oleg331-server: oleg331-server.virtnet: executing load_modules_local oleg331-server: kmemleak disabled oleg331-server: Loading modules from /home/green/git/lustre-release/lustre oleg331-server: detected 4 online CPUs by sysfs oleg331-server: MODOPTS_LIBCFS= oleg331-server: Force libcfs to create 2 CPU partitions oleg331-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Setup mgs, mdt, osts Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg331-server: oleg331-server.virtnet: executing set_default_debug -1 all pdsh@oleg331-client: oleg331-server: ssh exited with exit code 1 Started lustre-MDT0000 Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg331-server: oleg331-server.virtnet: executing set_default_debug -1 all pdsh@oleg331-client: oleg331-server: ssh exited with exit code 1 Started lustre-OST0000 Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg331-server: oleg331-server.virtnet: executing set_default_debug -1 all pdsh@oleg331-client: oleg331-server: ssh exited with exit code 1 Started lustre-OST0001 mount lustre on /mnt/lustre..... Starting client: oleg331-client.virtnet: -o user_xattr,flock 192.168.203.131@tcp:/lustre /mnt/lustre Starting client oleg331-client.virtnet: -o user_xattr,flock 192.168.203.131@tcp:/lustre /mnt/lustre Started clients oleg331-client.virtnet: 192.168.203.131@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9b59312b6800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9b59312b6800.idle_timeout=debug disable quota as required PASS 103 (460s) == conf-sanity test 104a: Make sure user defined options are reflected in mount ========================================================== 07:57:37 (1788868657) mkfs.lustre --mountfsopts: user_xattr,recovery_time_soft=222 Stopping clients: oleg331-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg331-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg331-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg331-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg331-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg331-server oleg331-server: oleg331-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg331-server' oleg331-server: oleg331-server.virtnet: executing load_modules_local oleg331-server: kmemleak disabled oleg331-server: Loading modules from /home/green/git/lustre-release/lustre oleg331-server: detected 4 online CPUs by sysfs oleg331-server: MODOPTS_LIBCFS= oleg331-server: Force libcfs to create 2 CPU partitions oleg331-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 mount opts: -o localrecov,recovery_time_soft=188 Start mds1: mount -t lustre -o localrecov,recovery_time_soft=188 lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg331-server: oleg331-server.virtnet: executing set_default_debug -1 all pdsh@oleg331-client: oleg331-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000