== conf-sanity test 157c: test allow_register timeout functionality ========================================================== 06:51:41 (1789383101) start mds service on oleg135-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1931560 Sep 14 04:14 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg135-server' oleg135-server: oleg135-server.virtnet: executing load_modules_local oleg135-server: kmemleak disabled oleg135-server: Loading modules from /home/green/git/lustre-release/lustre oleg135-server: detected 4 online CPUs by sysfs oleg135-server: MODOPTS_LIBCFS= oleg135-server: Force libcfs to create 2 CPU partitions oleg135-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg135-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg135-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg135-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg135-server: oleg135-server.virtnet: executing set_default_debug -1 all pdsh@oleg135-client: oleg135-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg135-server: params: OBD_IOC_LLOG_PRINT failed: No such file or directory pdsh@oleg135-client: oleg135-server: ssh exited with exit code 2 pdsh@oleg135-client: oleg135-server: ssh exited with exit code 123 allow_register=unlimited oleg135-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg135-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg135-server: oleg135-server.virtnet: executing set_default_debug -1 all pdsh@oleg135-client: oleg135-server: ssh exited with exit code 1 Started lustre-OST0000 oleg135-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg135-client.virtnet: -o user_xattr,flock 192.168.201.135@tcp:/lustre /mnt/lustre allow_register=unlimited pdsh@oleg135-client: oleg135-server: ssh exited with exit code 1 Permanent disk data: Device: /tmp/lustre-ost1_2 Target: fs32324:OST0000 Index: 0 Lustre FS: fs32324 Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.135@tcp:192.168.201.135@tcp sys.timeout=20 formatting backing filesystem ldiskfs on /dev/loop0 target name fs32324:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L fs32324:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/loop0 200000k /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1890: OST_INDICES: bad array subscript pdsh@oleg135-client: oleg135-server: ssh exited with exit code 1 pdsh@oleg135-client: oleg135-server: ssh exited with exit code 1 Start fs2ost: mount -t lustre -o localrecov /dev/mapper/fs2ost_flakey /mnt/lustre-fs2ost seq.cli-fs32324-OST0000-super.width=65536 oleg135-server: oleg135-server.virtnet: executing set_default_debug -1 all pdsh@oleg135-client: oleg135-server: ssh exited with exit code 1 Commit the device label on /tmp/lustre-ost1_2 Started fs32324-OST0000 /mnt/lustre-fs2ost: 83.1 MiB (87142400 bytes) trimmed allow_register=60s obdfilter.fs32324-OST0000.recovery_status=status: INACTIVE Waiting for allow_register timeout to expire Waiting 66s for '1' Waiting 56s for '1' Waiting 46s for '1' Waiting 36s for '1' Waiting 26s for '1' Waiting 16s for '1' Updated after 60s: want '1' got '1' pdsh@oleg135-client: oleg135-server: ssh exited with exit code 1 Permanent disk data: Device: /tmp/lustre-ost2_2 Target: fs32324:OST0001 Index: 1 Lustre FS: fs32324 Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.135@tcp:192.168.201.135@tcp sys.timeout=20 formatting backing filesystem ldiskfs on /dev/loop1 target name fs32324:OST0001 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L fs32324:OST0001 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/loop1 200000k /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1890: OST_INDICES: bad array subscript pdsh@oleg135-client: oleg135-server: ssh exited with exit code 1 pdsh@oleg135-client: oleg135-server: ssh exited with exit code 1 Start fs3ost: mount -t lustre -o localrecov /dev/mapper/fs3ost_flakey /mnt/lustre-fs3ost oleg135-server: mount.lustre: mount /dev/mapper/fs3ost_flakey at /mnt/lustre-fs3ost failed: Permission denied pdsh@oleg135-client: oleg135-server: ssh exited with exit code 13 oleg135-server: error: set_param: param_path 'seq/cli-fs32324:OST0001-super/width': No such file or directory oleg135-server: error: set_param: setting 'seq/cli-fs32324:OST0001-super/width'='65536': No such file or directory pdsh@oleg135-client: oleg135-server: ssh exited with exit code 2 Start of /dev/mapper/fs3ost_flakey on fs3ost failed 13 allow_register=unlimited /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1890: OST_INDICES: bad array subscript Start fs3ost: mount -t lustre -o localrecov /dev/mapper/fs3ost_flakey /mnt/lustre-fs3ost seq.cli-fs32324-OST0001-super.width=65536 oleg135-server: oleg135-server.virtnet: executing set_default_debug -1 all pdsh@oleg135-client: oleg135-server: ssh exited with exit code 1 Commit the device label on /tmp/lustre-ost2_2 Started fs32324-OST0001 /mnt/lustre-fs3ost: 83.1 MiB (87146496 bytes) trimmed obdfilter.fs32324-OST0000.recovery_status=status: INACTIVE obdfilter.fs32324-OST0001.recovery_status=status: INACTIVE Cleaning up test 157c... Stopping /mnt/lustre-fs2ost (opts:-f) on oleg135-server Stopping /mnt/lustre-fs3ost (opts:-f) on oleg135-server umount lustre on /mnt/lustre..... Stopping client oleg135-client.virtnet /mnt/lustre (opts:) oleg135-server: params: OBD_IOC_LLOG_PRINT failed: No such file or directory pdsh@oleg135-client: oleg135-server: ssh exited with exit code 2 pdsh@oleg135-client: oleg135-server: ssh exited with exit code 123 allow_register=unlimited stop ost1 service on oleg135-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg135-server stop mds service on oleg135-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg135-server unloading modules via unload_modules_local on: 'oleg135-server' oleg135-server: oleg135-server.virtnet: executing unload_modules_local oleg135-server: kmemleak disabled modules unloaded.