== conf-sanity test 157c: test allow_register timeout functionality ========================================================== 06:40:18 (1789382418) start mds service on oleg331-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1931560 Sep 14 04:14 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg331-server' oleg331-server: oleg331-server.virtnet: executing load_modules_local oleg331-server: kmemleak disabled oleg331-server: Loading modules from /home/green/git/lustre-release/lustre oleg331-server: detected 4 online CPUs by sysfs oleg331-server: MODOPTS_LIBCFS= oleg331-server: Force libcfs to create 2 CPU partitions oleg331-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg331-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg331-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg331-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg331-server: oleg331-server.virtnet: executing set_default_debug -1 all pdsh@oleg331-client: oleg331-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg331-server: params: OBD_IOC_LLOG_PRINT failed: No such file or directory pdsh@oleg331-client: oleg331-server: ssh exited with exit code 2 pdsh@oleg331-client: oleg331-server: ssh exited with exit code 123 allow_register=unlimited oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg331-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg331-server: oleg331-server.virtnet: executing set_default_debug -1 all pdsh@oleg331-client: oleg331-server: ssh exited with exit code 1 Started lustre-OST0000 oleg331-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg331-client.virtnet: -o user_xattr,flock 192.168.203.131@tcp:/lustre /mnt/lustre allow_register=unlimited pdsh@oleg331-client: oleg331-server: ssh exited with exit code 1 Permanent disk data: Device: /tmp/lustre-ost1_2 Target: fs17595:OST0000 Index: 0 Lustre FS: fs17595 Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.131@tcp:192.168.203.131@tcp sys.timeout=20 formatting backing filesystem ldiskfs on /dev/loop0 target name fs17595:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L fs17595:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/loop0 200000k /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1890: OST_INDICES: bad array subscript pdsh@oleg331-client: oleg331-server: ssh exited with exit code 1 pdsh@oleg331-client: oleg331-server: ssh exited with exit code 1 Start fs2ost: mount -t lustre -o localrecov /dev/mapper/fs2ost_flakey /mnt/lustre-fs2ost seq.cli-fs17595-OST0000-super.width=65536 oleg331-server: oleg331-server.virtnet: executing set_default_debug -1 all pdsh@oleg331-client: oleg331-server: ssh exited with exit code 1 Commit the device label on /tmp/lustre-ost1_2 Started fs17595-OST0000 /mnt/lustre-fs2ost: 83.1 MiB (87142400 bytes) trimmed allow_register=60s obdfilter.fs17595-OST0000.recovery_status=status: INACTIVE Waiting for allow_register timeout to expire Waiting 67s for '1' Waiting 57s for '1' Waiting 47s for '1' Waiting 37s for '1' Waiting 27s for '1' Waiting 17s for '1' Updated after 58s: want '1' got '1' pdsh@oleg331-client: oleg331-server: ssh exited with exit code 1 Permanent disk data: Device: /tmp/lustre-ost2_2 Target: fs17595:OST0001 Index: 1 Lustre FS: fs17595 Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.131@tcp:192.168.203.131@tcp sys.timeout=20 formatting backing filesystem ldiskfs on /dev/loop1 target name fs17595:OST0001 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L fs17595:OST0001 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/loop1 200000k /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1890: OST_INDICES: bad array subscript pdsh@oleg331-client: oleg331-server: ssh exited with exit code 1 pdsh@oleg331-client: oleg331-server: ssh exited with exit code 1 Start fs3ost: mount -t lustre -o localrecov /dev/mapper/fs3ost_flakey /mnt/lustre-fs3ost oleg331-server: mount.lustre: mount /dev/mapper/fs3ost_flakey at /mnt/lustre-fs3ost failed: Permission denied pdsh@oleg331-client: oleg331-server: ssh exited with exit code 13 oleg331-server: error: set_param: param_path 'seq/cli-fs17595:OST0001-super/width': No such file or directory oleg331-server: error: set_param: setting 'seq/cli-fs17595:OST0001-super/width'='65536': No such file or directory pdsh@oleg331-client: oleg331-server: ssh exited with exit code 2 Start of /dev/mapper/fs3ost_flakey on fs3ost failed 13 allow_register=unlimited /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1890: OST_INDICES: bad array subscript Start fs3ost: mount -t lustre -o localrecov /dev/mapper/fs3ost_flakey /mnt/lustre-fs3ost seq.cli-fs17595-OST0001-super.width=65536 oleg331-server: oleg331-server.virtnet: executing set_default_debug -1 all pdsh@oleg331-client: oleg331-server: ssh exited with exit code 1 Commit the device label on /tmp/lustre-ost2_2 Started fs17595-OST0001 /mnt/lustre-fs3ost: 83.1 MiB (87146496 bytes) trimmed obdfilter.fs17595-OST0000.recovery_status=status: INACTIVE obdfilter.fs17595-OST0001.recovery_status=status: INACTIVE Cleaning up test 157c... Stopping /mnt/lustre-fs2ost (opts:-f) on oleg331-server Stopping /mnt/lustre-fs3ost (opts:-f) on oleg331-server umount lustre on /mnt/lustre..... Stopping client oleg331-client.virtnet /mnt/lustre (opts:) oleg331-server: params: OBD_IOC_LLOG_PRINT failed: No such file or directory pdsh@oleg331-client: oleg331-server: ssh exited with exit code 2 pdsh@oleg331-client: oleg331-server: ssh exited with exit code 123 allow_register=unlimited stop ost1 service on oleg331-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg331-server stop mds service on oleg331-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg331-server unloading modules via unload_modules_local on: 'oleg331-server' oleg331-server: oleg331-server.virtnet: executing unload_modules_local oleg331-server: kmemleak disabled modules unloaded.