== conf-sanity test 157c: test allow_register timeout functionality ========================================================== 15:13:33 (1788808413) start mds service on oleg656-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1921056 Sep 7 12:38 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg656-server' oleg656-server: oleg656-server.virtnet: executing load_modules_local oleg656-server: kmemleak disabled oleg656-server: Loading modules from /home/green/git/lustre-release/lustre oleg656-server: detected 4 online CPUs by sysfs oleg656-server: MODOPTS_LIBCFS= oleg656-server: Force libcfs to create 2 CPU partitions oleg656-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg656-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg656-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg656-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg656-server: oleg656-server.virtnet: executing set_default_debug -1 all pdsh@oleg656-client: oleg656-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg656-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg656-server: oleg656-server.virtnet: executing set_default_debug -1 all pdsh@oleg656-client: oleg656-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg656-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg656-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg656-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg656-server: oleg656-server.virtnet: executing set_default_debug -1 all pdsh@oleg656-client: oleg656-server: ssh exited with exit code 1 Started lustre-OST0000 oleg656-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg656-client.virtnet: -o user_xattr,flock 192.168.206.156@tcp:/lustre /mnt/lustre allow_register=unlimited pdsh@oleg656-client: oleg656-server: ssh exited with exit code 1 Permanent disk data: Device: /tmp/lustre-ost1_2 Target: fs27043:OST0000 Index: 0 Lustre FS: fs27043 Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.206.156@tcp:192.168.206.156@tcp sys.timeout=20 formatting backing filesystem ldiskfs on /dev/loop1 target name fs27043:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L fs27043:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/loop1 200000k /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1875: OST_INDICES: bad array subscript pdsh@oleg656-client: oleg656-server: ssh exited with exit code 1 pdsh@oleg656-client: oleg656-server: ssh exited with exit code 1 Start fs2ost: mount -t lustre -o localrecov /dev/mapper/fs2ost_flakey /mnt/lustre-fs2ost seq.cli-fs27043-OST0000-super.width=65536 oleg656-server: oleg656-server.virtnet: executing set_default_debug -1 all pdsh@oleg656-client: oleg656-server: ssh exited with exit code 1 Commit the device label on /tmp/lustre-ost1_2 Started fs27043-OST0000 /mnt/lustre-fs2ost: 83.1 MiB (87142400 bytes) trimmed allow_register=60s obdfilter.fs27043-OST0000.recovery_status=status: INACTIVE Waiting for allow_register timeout to expire Waiting 68s for '1' Waiting 58s for '1' Waiting 48s for '1' Waiting 28s for '1' Updated after 59s: want '1' got '1' pdsh@oleg656-client: oleg656-server: ssh exited with exit code 1 Permanent disk data: Device: /tmp/lustre-ost2_2 Target: fs27043:OST0001 Index: 1 Lustre FS: fs27043 Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.206.156@tcp:192.168.206.156@tcp sys.timeout=20 formatting backing filesystem ldiskfs on /dev/loop2 target name fs27043:OST0001 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L fs27043:OST0001 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/loop2 200000k /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1875: OST_INDICES: bad array subscript pdsh@oleg656-client: oleg656-server: ssh exited with exit code 1 pdsh@oleg656-client: oleg656-server: ssh exited with exit code 1 Start fs3ost: mount -t lustre -o localrecov /dev/mapper/fs3ost_flakey /mnt/lustre-fs3ost oleg656-server: mount.lustre: mount /dev/mapper/fs3ost_flakey at /mnt/lustre-fs3ost failed: Permission denied pdsh@oleg656-client: oleg656-server: ssh exited with exit code 13 oleg656-server: error: set_param: param_path 'seq/cli-fs27043:OST0001-super/width': No such file or directory oleg656-server: error: set_param: setting 'seq/cli-fs27043:OST0001-super/width'='65536': No such file or directory pdsh@oleg656-client: oleg656-server: ssh exited with exit code 2 Start of /dev/mapper/fs3ost_flakey on fs3ost failed 13 allow_register=unlimited /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1875: OST_INDICES: bad array subscript Start fs3ost: mount -t lustre -o localrecov /dev/mapper/fs3ost_flakey /mnt/lustre-fs3ost seq.cli-fs27043-OST0001-super.width=65536 oleg656-server: oleg656-server.virtnet: executing set_default_debug -1 all pdsh@oleg656-client: oleg656-server: ssh exited with exit code 1 Commit the device label on /tmp/lustre-ost2_2 Started fs27043-OST0001 /mnt/lustre-fs3ost: 83.1 MiB (87146496 bytes) trimmed obdfilter.fs27043-OST0000.recovery_status=status: INACTIVE obdfilter.fs27043-OST0001.recovery_status=status: INACTIVE Cleaning up test 157c... Stopping /mnt/lustre-fs2ost (opts:-f) on oleg656-server Stopping /mnt/lustre-fs3ost (opts:-f) on oleg656-server umount lustre on /mnt/lustre..... Stopping client oleg656-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg656-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg656-server stop mds service on oleg656-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg656-server stop mds service on oleg656-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg656-server unloading modules via unload_modules_local on: 'oleg656-server' oleg656-server: oleg656-server.virtnet: executing unload_modules_local oleg656-server: kmemleak disabled modules unloaded.