== conf-sanity test 24a: Multiple MDTs on a single node == 17:49:02 (1787608142) pdsh@oleg211-client: oleg211-server: ssh exited with exit code 1 Permanent disk data: Device: lustre-mdt1_2/mdt1_2 Target: 969362ae:MDT0000 Index: 0 Lustre FS: 969362ae Mount type: zfs Flags: 0x61 (MDT first_time update ) Persistent mount opts: Parameters: mgsnode=192.168.202.111@tcp:192.168.202.111@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-mdt1_2 /tmp/lustre-mdt1_2 mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-mdt1_2/mdt1_2 xattr=sa dnodesize=auto compression=off Writing lustre-mdt1_2/mdt1_2 properties lustre:mgsnode=192.168.202.111@tcp:192.168.202.111@tcp lustre:sys.timeout=20 lustre:mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity lustre:version=1 lustre:flags=97 lustre:index=0 lustre:fsname=969362ae lustre:svname=969362ae:MDT0000 Permanent disk data: Device: lustre-ost1_2/ost1_2 Target: 969362ae:OST0000 Index: 0 Lustre FS: 969362ae Mount type: zfs Flags: 0x62 (OST first_time update ) Persistent mount opts: Parameters: mgsnode=192.168.202.111@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1_2 /tmp/lustre-ost1_2 mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1_2/ost1_2 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1_2/ost1_2 properties lustre:mgsnode=192.168.202.111@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=98 lustre:index=0 lustre:fsname=969362ae lustre:svname=969362ae:OST0000 start mds service on oleg211-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg211-server' oleg211-server: oleg211-server.virtnet: executing load_modules_local oleg211-server: Loading modules from /home/green/git/lustre-release/lustre oleg211-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg211-server: detected 4 online CPUs by sysfs oleg211-server: MODOPTS_LIBCFS= oleg211-server: Force libcfs to create 2 CPU partitions oleg211-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg211-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg211-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg211-server: oleg211-server.virtnet: executing set_default_debug -1 all pdsh@oleg211-client: oleg211-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg211-server Start mds2: mount -t lustre -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg211-server: oleg211-server.virtnet: executing set_default_debug -1 all pdsh@oleg211-client: oleg211-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg211-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg211-server: oleg211-server.virtnet: executing set_default_debug -1 all pdsh@oleg211-client: oleg211-server: ssh exited with exit code 1 Started lustre-OST0000 oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg211-client.virtnet: -o user_xattr,flock 192.168.202.111@tcp:/lustre /mnt/lustre Start fs2mds: mount -t lustre -o localrecov lustre-mdt1_2/mdt1_2 /mnt/lustre-fs2mds oleg211-server: oleg211-server.virtnet: executing set_default_debug -1 all pdsh@oleg211-client: oleg211-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1_2/mdt1_2 Started 969362ae-MDT0000 /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1850: OST_INDICES: bad array subscript Start fs2ost: mount -t lustre -o localrecov lustre-ost1_2/ost1_2 /mnt/lustre-fs2ost seq.cli-969362ae-OST0000-super.width=65536 oleg211-server: oleg211-server.virtnet: executing set_default_debug -1 all pdsh@oleg211-client: oleg211-server: ssh exited with exit code 1 Commit the device label on lustre-ost1_2/ost1_2 Started 969362ae-OST0000 969362ae-MDT0000: Deregistered changelog user #1 setup single mount lustre success Failing fs2mds on oleg211-server Stopping /mnt/lustre-fs2mds (opts:) on oleg211-server 17:51:52 (1787608312) shut down facet: fs2mds facet_host: oleg211-server facet_failover_host: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4400: [: oleg211-server: unary operator expected mount facets: fs2mds Start fs2mds: mount -t lustre -o localrecov lustre-mdt1_2/mdt1_2 /mnt/lustre-fs2mds oleg211-server: oleg211-server.virtnet: executing set_default_debug -1 all pdsh@oleg211-client: oleg211-server: ssh exited with exit code 1 Started 969362ae-MDT0000 17:52:14 (1787608334) targets are mounted 17:52:14 (1787608334) facet_failover done Failing fs2ost on oleg211-server Stopping /mnt/lustre-fs2ost (opts:) on oleg211-server 17:52:17 (1787608337) shut down facet: fs2ost facet_host: oleg211-server facet_failover_host: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4400: [: oleg211-server: unary operator expected mount facets: fs2ost /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1850: OST_INDICES: bad array subscript Start fs2ost: mount -t lustre -o localrecov lustre-ost1_2/ost1_2 /mnt/lustre-fs2ost seq.cli-969362ae-OST0000-super.width=65536 oleg211-server: oleg211-server.virtnet: executing set_default_debug -1 all pdsh@oleg211-client: oleg211-server: ssh exited with exit code 1 Started 969362ae-OST0000 17:52:42 (1787608362) targets are mounted 17:52:42 (1787608362) facet_failover done Filesystem 1K-blocks Used Available Use% Mounted on devtmpfs 1890908 0 1890908 0% /dev tmpfs 1890908 0 1890908 0% /dev/shm tmpfs 1890908 16356 1874552 1% /run tmpfs 1890908 0 1890908 0% /sys/fs/cgroup /dev/nbd0 30787492 17460884 11737360 60% / none 1890908 24392 1866516 2% /var/lib/stateless/writable /dev/vda 73088 73088 0 100% /home/green/git/lustre-release 192.168.200.253:/exports/state/oleg211-client.virtnet 143911936 122829824 14439424 90% /var/lib/stateless/state 192.168.200.253:/exports/state/oleg211-client.virtnet/boot 143911936 122829824 14439424 90% /boot 192.168.200.253:/exports/state/oleg211-client.virtnet/etc/kdump.conf 143911936 122829824 14439424 90% /etc/kdump.conf tmpfs 378180 0 378180 0% /run/user/0 192.168.200.253:/exports/testreports/68607/testresults/conf-sanity1-zfs-DNE-rocky8.10_x86_64-rocky8.10_x86_64 54444715008 38448508928 15996206080 71% /tmp/testlogs 192.168.202.111@tcp:/lustre 382976 2048 378880 1% /mnt/lustre 192.168.202.111@tcp:/969362ae 244736 2048 240640 1% /mnt/lustre2 umount lustre on /mnt/lustre..... Stopping client oleg211-client.virtnet /mnt/lustre (opts:) stop mds service on oleg211-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg211-server stop mds service on oleg211-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg211-server umount /mnt/lustre2 ... stopping fs2mds ... Stopping /mnt/lustre-fs2mds (opts:-f) on oleg211-server stopping fs2ost ... Stopping /mnt/lustre-fs2ost (opts:-f) on oleg211-server stop ost1 service on oleg211-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg211-server stop mds service on oleg211-server stop mds service on oleg211-server unloading modules via unload_modules_local on: 'oleg211-server' oleg211-server: oleg211-server.virtnet: executing unload_modules_local oleg211-server: modules unloaded. pdsh@oleg211-client: oleg211-client: ssh exited with exit code 2 pdsh@oleg211-client: oleg211-server: ssh exited with exit code 2 pdsh@oleg211-client: oleg211-server: ssh exited with exit code 2 pdsh@oleg211-client: oleg211-client: ssh exited with exit code 2