-----============= acceptance-small: conf-sanity ============----- Mon May 25 07:58:56 EDT 2026 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg132-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex': No such file or directory excepting tests: 32newtarball skipping tests SLOW=no: 45 69 106 111 114 Stopping clients: oleg132-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg132-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg132-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg132-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg132-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg132-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg132-server oleg132-server: oleg132-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg132-server' oleg132-server: oleg132-server.virtnet: executing load_modules_local oleg132-server: Loading modules from /home/green/git/lustre-release/lustre oleg132-server: detected 4 online CPUs by sysfs oleg132-server: MODOPTS_LIBCFS= oleg132-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg132-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg132-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg132-server: oleg132-server.virtnet: executing set_default_debug -1 all pdsh@oleg132-client: oleg132-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg132-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg132-server: oleg132-server.virtnet: executing set_default_debug -1 all pdsh@oleg132-client: oleg132-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg132-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg132-server: oleg132-server.virtnet: executing set_default_debug -1 all pdsh@oleg132-client: oleg132-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg132-server: oleg132-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg132-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg132-server: oleg132-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg132-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg132-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg132-server stop mds service on oleg132-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg132-server stop mds service on oleg132-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg132-server == conf-sanity test 121: failover MGS ==================== 08:03:27 (1779710607) Stopping clients: oleg132-client.virtnet /mnt/lustre (opts:) Stopping clients: oleg132-client.virtnet /mnt/lustre2 (opts:) start mds service on oleg132-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg132-server: oleg132-server.virtnet: executing set_default_debug -1 all pdsh@oleg132-client: oleg132-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg132-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg132-server: oleg132-server.virtnet: executing set_default_debug -1 all pdsh@oleg132-client: oleg132-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid Failing mgs on oleg132-server Stopping /mnt/lustre-mds1 (opts:) on oleg132-server 08:04:44 (1779710684) shut down facet: mgs facet_host: oleg132-server facet_failover_host: oleg132-server Failover mgs to oleg132-server mount facets: mgs Start mgs: mount -t lustre /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg132-server: oleg132-server.virtnet: executing set_default_debug -1 all pdsh@oleg132-client: oleg132-server: ssh exited with exit code 1 Started lustre-MDT0000 08:05:19 (1779710719) targets are mounted 08:05:19 (1779710719) facet_failover done pdsh@oleg132-client: oleg132-client: ssh exited with exit code 95 oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid pdsh@oleg132-client: oleg132-client: ssh exited with exit code 95 stop mds service on oleg132-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg132-server stop mds service on oleg132-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg132-server start mds service on oleg132-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg132-server: oleg132-server.virtnet: executing set_default_debug -1 all pdsh@oleg132-client: oleg132-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg132-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg132-server: oleg132-server.virtnet: executing set_default_debug -1 all pdsh@oleg132-client: oleg132-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid Failing mgs on oleg132-server Stopping /mnt/lustre-mds1 (opts:) on oleg132-server 08:06:52 (1779710812) shut down facet: mgs facet_host: oleg132-server facet_failover_host: oleg132-server Failover mgs to oleg132-server mount facets: mgs Start mgs: mount -t lustre /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg132-server: oleg132-server.virtnet: executing set_default_debug -1 all pdsh@oleg132-client: oleg132-server: ssh exited with exit code 1 Started lustre-MDT0000 08:07:28 (1779710848) targets are mounted 08:07:28 (1779710848) facet_failover done pdsh@oleg132-client: oleg132-client: ssh exited with exit code 95 oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid pdsh@oleg132-client: oleg132-client: ssh exited with exit code 95 stop mds service on oleg132-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg132-server stop mds service on oleg132-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg132-server PASS 121 (280s) == conf-sanity test 122a: Check OST sequence update ====== 08:08:07 (1779710887) Stopping clients: oleg132-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg132-client.virtnet /mnt/lustre2 (opts:-f) oleg132-server: oleg132-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg132-server' oleg132-server: oleg132-server.virtnet: executing load_modules_local oleg132-server: Loading modules from /home/green/git/lustre-release/lustre oleg132-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg132-server: detected 4 online CPUs by sysfs oleg132-server: MODOPTS_LIBCFS= oleg132-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg132-server' oleg132-server: oleg132-server.virtnet: executing load_modules_local oleg132-server: Loading modules from /home/green/git/lustre-release/lustre oleg132-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg132-server: detected 4 online CPUs by sysfs oleg132-server: MODOPTS_LIBCFS= oleg132-server: Force libcfs to create 2 CPU partitions fail_loc=0x00001e0 start mds service on oleg132-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg132-server: oleg132-server.virtnet: executing set_default_debug -1 all pdsh@oleg132-client: oleg132-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg132-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg132-server: oleg132-server.virtnet: executing set_default_debug -1 all pdsh@oleg132-client: oleg132-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg132-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg132-server: oleg132-server.virtnet: executing set_default_debug -1 all pdsh@oleg132-client: oleg132-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg132-client.virtnet: -o user_xattr,flock 192.168.201.132@tcp:/lustre /mnt/lustre fail_loc=0 - open/close 443 (time 1779711121.82 total 10.03 last 44.15) total: 1000 open/close in 19.80 seconds: 50.50 ops/second umount lustre on /mnt/lustre..... Stopping client oleg132-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg132-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg132-server stop mds service on oleg132-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg132-server stop mds service on oleg132-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg132-server unloading modules via unload_modules_local on: 'oleg132-server' oleg132-server: oleg132-server.virtnet: executing unload_modules_local oleg132-server: modules unloaded. pdsh@oleg132-client: oleg132-client: ssh exited with exit code 2 pdsh@oleg132-client: oleg132-server: ssh exited with exit code 2 pdsh@oleg132-client: oleg132-client: ssh exited with exit code 2 pdsh@oleg132-client: oleg132-server: ssh exited with exit code 2 PASS 122a (314s) == conf-sanity test 123aa: llog_print works with FIDs and simple names ========================================================== 08:13:21 (1779711201) start mds service on oleg132-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg132-server' oleg132-server: oleg132-server.virtnet: executing load_modules_local oleg132-server: Loading modules from /home/green/git/lustre-release/lustre oleg132-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg132-server: detected 4 online CPUs by sysfs oleg132-server: MODOPTS_LIBCFS= oleg132-server: Force libcfs to create 2 CPU partitions oleg132-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg132-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg132-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg132-server: oleg132-server.virtnet: executing set_default_debug -1 all pdsh@oleg132-client: oleg132-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg132-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg132-server: oleg132-server.virtnet: executing set_default_debug -1 all pdsh@oleg132-client: oleg132-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg132-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg132-server: oleg132-server.virtnet: executing set_default_debug -1 all pdsh@oleg132-client: oleg132-server: ssh exited with exit code 1 Started lustre-OST0000 oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg132-client.virtnet: -o user_xattr,flock 192.168.201.132@tcp:/lustre /mnt/lustre 1 UP mgs MGS MGS 7 - { index: 2, event: attach, device: lustre-clilov, type: lov, UUID: lustre-clilov_UUID } - { index: 3, event: setup, device: lustre-clilov, UUID: } - { index: 6, event: attach, device: lustre-clilmv, type: lmv, UUID: lustre-clilmv_UUID } - { index: 7, event: setup, device: lustre-clilmv, UUID: } - { index: 10, event: new_profile, name: lustre-client, lov: lustre-clilov, lmv: lustre-clilmv } - { index: 2, event: attach, device: lustre-clilov, type: lov, UUID: lustre-clilov_UUID } - { index: 3, event: setup, device: lustre-clilov, UUID: } - { index: 6, event: attach, device: lustre-clilmv, type: lmv, UUID: lustre-clilmv_UUID } - { index: 7, event: setup, device: lustre-clilmv, UUID: } - { index: 10, event: new_profile, name: lustre-client, lov: lustre-clilov, lmv: lustre-clilmv } - { index: 13, event: add_uuid, nid: 192.168.201.132@tcp(0x20000c0a8c984), node: 192.168.201.132@tcp } - { index: 14, event: attach, device: lustre-MDT0000-mdc, type: mdc, UUID: lustre-clilmv_UUID } - { index: 15, event: setup, device: lustre-MDT0000-mdc, UUID: lustre-MDT0000_UUID, node: 192.168.201.132@tcp } - { index: 16, event: add_mdc, device: lustre-clilmv, mdt: lustre-MDT0000_UUID, index: 0, gen: 1, UUID: lustre-MDT0000-mdc_UUID } - { index: 22, event: add_uuid, nid: 192.168.201.132@tcp(0x20000c0a8c984), node: 192.168.201.132@tcp } - { index: 23, event: attach, device: lustre-MDT0001-mdc, type: mdc, UUID: lustre-clilmv_UUID } - { index: 24, event: setup, device: lustre-MDT0001-mdc, UUID: lustre-MDT0001_UUID, node: 192.168.201.132@tcp } - { index: 25, event: add_mdc, device: lustre-clilmv, mdt: lustre-MDT0001_UUID, index: 1, gen: 1, UUID: lustre-MDT0001-mdc_UUID } - { index: 31, event: add_uuid, nid: 192.168.201.132@tcp(0x20000c0a8c984), node: 192.168.201.132@tcp } - { index: 32, event: attach, device: lustre-OST0000-osc, type: osc, UUID: lustre-clilov_UUID } - { index: 33, event: setup, device: lustre-OST0000-osc, UUID: lustre-OST0000_UUID, node: 192.168.201.132@tcp } - { index: 34, event: add_osc, device: lustre-clilov, ost: lustre-OST0000_UUID, index: 0, gen: 1 } - { index: 37, event: set_timeout, num: 0x000014, parameter: sys.timeout=20 } PASS 123aa (139s) == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 08:15:41 (1779711341) PASS 123ab (17s) == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 08:15:58 (1779711358) PASS 123ac (14s) == conf-sanity test 123ad: llog_print shows all records == 08:16:12 (1779711372) PASS 123ad (23s) == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 08:16:35 (1779711395) - { index: 11, event: set_param, device: general, parameter: osc.*.max_dirty_mb, value: 462 } - { index: 46, event: conf_param, device: lustre-OST0000-osc, parameter: osc.max_dirty_mb=462 } - { index: 2, event: attach, device: lustre-clilov, type: lov, UUID: lustre-clilov_UUID } - { index: 3, event: setup, device: lustre-clilov, UUID: } - { index: 6, event: attach, device: lustre-clilmv, type: lmv, UUID: lustre-clilmv_UUID } - { index: 7, event: setup, device: lustre-clilmv, UUID: } - { index: 10, event: new_profile, name: lustre-client, lov: lustre-clilov, lmv: lustre-clilmv } - { index: 13, event: add_uuid, nid: 192.168.201.132@tcp(0x20000c0a8c984), node: 192.168.201.132@tcp } - { index: 14, event: attach, device: lustre-MDT0000-mdc, type: mdc, UUID: lustre-clilmv_UUID } - { index: 15, event: setup, device: lustre-MDT0000-mdc, UUID: lustre-MDT0000_UUID, node: 192.168.201.132@tcp } - { index: 16, event: add_mdc, device: lustre-clilmv, mdt: lustre-MDT0000_UUID, index: 0, gen: 1, UUID: lustre-MDT0000-mdc_UUID } - { index: 22, event: add_uuid, nid: 192.168.201.132@tcp(0x20000c0a8c984), node: 192.168.201.132@tcp } - { index: 23, event: attach, device: lustre-MDT0001-mdc, type: mdc, UUID: lustre-clilmv_UUID } - { index: 24, event: setup, device: lustre-MDT0001-mdc, UUID: lustre-MDT0001_UUID, node: 192.168.201.132@tcp } - { index: 25, event: add_mdc, device: lustre-clilmv, mdt: lustre-MDT0001_UUID, index: 1, gen: 1, UUID: lustre-MDT0001-mdc_UUID } - { index: 31, event: add_uuid, nid: 192.168.201.132@tcp(0x20000c0a8c984), node: 192.168.201.132@tcp } - { index: 32, event: attach, device: lustre-OST0000-osc, type: osc, UUID: lustre-clilov_UUID } - { index: 33, event: setup, device: lustre-OST0000-osc, UUID: lustre-OST0000_UUID, node: 192.168.201.132@tcp } - { index: 34, event: add_osc, device: lustre-clilov, ost: lustre-OST0000_UUID, index: 0, gen: 1 } - { index: 37, event: set_timeout, num: 0x000014, parameter: sys.timeout=20 } - { index: 43, event: conf_param, device: lustre-OST0000-osc, parameter: osc.max_pages_per_rpc=1024 } - { index: 46, event: conf_param, device: lustre-OST0000-osc, parameter: osc.max_dirty_mb=462 } - { index: 2, event: attach, device: lustre-clilov, type: lov, UUID: lustre-clilov_UUID } - { index: 3, event: setup, device: lustre-clilov, UUID: } - { index: 6, event: attach, device: lustre-clilmv, type: lmv, UUID: lustre-clilmv_UUID } - { index: 7, event: setup, device: lustre-clilmv, UUID: } - { index: 10, event: new_profile, name: lustre-client, lov: lustre-clilov, lmv: lustre-clilmv } - { index: 13, event: add_uuid, nid: 192.168.201.132@tcp(0x20000c0a8c984), node: 192.168.201.132@tcp } - { index: 14, event: attach, device: lustre-MDT0000-mdc, type: mdc, UUID: lustre-clilmv_UUID } - { index: 15, event: setup, device: lustre-MDT0000-mdc, UUID: lustre-MDT0000_UUID, node: 192.168.201.132@tcp } - { index: 16, event: add_mdc, device: lustre-clilmv, mdt: lustre-MDT0000_UUID, index: 0, gen: 1, UUID: lustre-MDT0000-mdc_UUID } - { index: 22, event: add_uuid, nid: 192.168.201.132@tcp(0x20000c0a8c984), node: 192.168.201.132@tcp } - { index: 23, event: attach, device: lustre-MDT0001-mdc, type: mdc, UUID: lustre-clilmv_UUID } - { index: 24, event: setup, device: lustre-MDT0001-mdc, UUID: lustre-MDT0001_UUID, node: 192.168.201.132@tcp } - { index: 25, event: add_mdc, device: lustre-clilmv, mdt: lustre-MDT0001_UUID, index: 1, gen: 1, UUID: lustre-MDT0001-mdc_UUID } - { index: 31, event: add_uuid, nid: 192.168.201.132@tcp(0x20000c0a8c984), node: 192.168.201.132@tcp } - { index: 32, event: attach, device: lustre-OST0000-osc, type: osc, UUID: lustre-clilov_UUID } - { index: 33, event: setup, device: lustre-OST0000-osc, UUID: lustre-OST0000_UUID, node: 192.168.201.132@tcp } - { index: 34, event: add_osc, device: lustre-clilov, ost: lustre-OST0000_UUID, index: 0, gen: 1 } - { index: 37, event: set_timeout, num: 0x000014, parameter: sys.timeout=20 } - { index: 43, event: conf_param, device: lustre-OST0000-osc, parameter: osc.max_pages_per_rpc=1024 } PASS 123ae (40s) == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 08:17:15 (1779711435) lctl --device MGS llog_catlist ... orig_clist: lustre-MDT0001 lustre-MDT0000 lustre-client lustre-OST0000 fail_loc=0x131b fail_val=2 new_clist: lustre-MDT0000 lustre-client lustre-OST0000 fail_loc=0 done lctl --device lustre-MDT0000 llog_catlist ... orig_clist: [0x1:0x2:0x0] fail_loc=0x131b fail_val=2 new_clist: fail_loc=0 done fail_loc=0 PASS 123af (22s) == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 08:17:38 (1779711458) PASS 123ag (22s) == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 08:18:00 (1779711480) del_ost: dry run for target lustre-OST0000 config_log: lustre-MDT0001 [DRY RUN] cancel catalog 'lustre-MDT0001:41':"- { index: 41, event: conf_param, device: lustre-OST0000-osc-MDT0001, parameter: osc.max_dirty_mb=462 }" [DRY RUN] cancel catalog 'lustre-MDT0001:29':"- { index: 29, event: add_osc, device: lustre-MDT0001-mdtlov, ost: lustre-OST0000_UUID, index: 0, gen: 1 }" [DRY RUN] cancel catalog 'lustre-MDT0001:28':"- { index: 28, event: setup, device: lustre-OST0000-osc-MDT0001, UUID: lustre-OST0000_UUID, node: 192.168.201.132@tcp }" [DRY RUN] cancel catalog 'lustre-MDT0001:27':"- { index: 27, event: attach, device: lustre-OST0000-osc-MDT0001, type: osc, UUID: lustre-MDT0001-mdtlov_UUID }" del_ost: no catalog entry deleted config_log: lustre-MDT0000 [DRY RUN] cancel catalog 'lustre-MDT0000:41':"- { index: 41, event: conf_param, device: lustre-OST0000-osc-MDT0000, parameter: osc.max_dirty_mb=462 }" [DRY RUN] cancel catalog 'lustre-MDT0000:29':"- { index: 29, event: add_osc, device: lustre-MDT0000-mdtlov, ost: lustre-OST0000_UUID, index: 0, gen: 1 }" [DRY RUN] cancel catalog 'lustre-MDT0000:28':"- { index: 28, event: setup, device: lustre-OST0000-osc-MDT0000, UUID: lustre-OST0000_UUID, node: 192.168.201.132@tcp }" [DRY RUN] cancel catalog 'lustre-MDT0000:27':"- { index: 27, event: attach, device: lustre-OST0000-osc-MDT0000, type: osc, UUID: lustre-MDT0000-mdtlov_UUID }" del_ost: no catalog entry deleted config_log: lustre-client [DRY RUN] cancel catalog 'lustre-client:34':"- { index: 34, event: add_osc, device: lustre-clilov, ost: lustre-OST0000_UUID, index: 0, gen: 1 }" [DRY RUN] cancel catalog 'lustre-client:33':"- { index: 33, event: setup, device: lustre-OST0000-osc, UUID: lustre-OST0000_UUID, node: 192.168.201.132@tcp }" [DRY RUN] cancel catalog 'lustre-client:32':"- { index: 32, event: attach, device: lustre-OST0000-osc, type: osc, UUID: lustre-clilov_UUID }" del_ost: no catalog entry deleted config_log: lustre-MDT0001 cancel catalog lustre-MDT0001 log_idx 41: done cancel catalog lustre-MDT0001 log_idx 29: done cancel catalog lustre-MDT0001 log_idx 28: done cancel catalog lustre-MDT0001 log_idx 27: done del_ost: cancelled 4 catalog entries config_log: lustre-MDT0000 cancel catalog lustre-MDT0000 log_idx 41: done cancel catalog lustre-MDT0000 log_idx 29: done cancel catalog lustre-MDT0000 log_idx 28: done cancel catalog lustre-MDT0000 log_idx 27: done del_ost: cancelled 4 catalog entries config_log: lustre-client cancel catalog lustre-client log_idx 34: done cancel catalog lustre-client log_idx 33: done cancel catalog lustre-client log_idx 32: done del_ost: cancelled 3 catalog entries umount lustre on /mnt/lustre..... Stopping client oleg132-client.virtnet /mnt/lustre (opts:) mount lustre on /mnt/lustre..... Starting client: oleg132-client.virtnet: -o user_xattr,flock 192.168.201.132@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg132-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg132-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg132-server stop mds service on oleg132-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg132-server stop mds service on oleg132-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg132-server unloading modules via unload_modules_local on: 'oleg132-server' oleg132-server: oleg132-server.virtnet: executing unload_modules_local oleg132-server: modules unloaded. Stopping clients: oleg132-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg132-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg132-client: oleg132-server: ssh exited with exit code 2 oleg132-server: oleg132-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg132-server' oleg132-server: oleg132-server.virtnet: executing load_modules_local oleg132-server: Loading modules from /home/green/git/lustre-release/lustre oleg132-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg132-server: detected 4 online CPUs by sysfs oleg132-server: MODOPTS_LIBCFS= oleg132-server: Force libcfs to create 2 CPU partitions oleg132-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg132-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg132-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg132-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg132-server: oleg132-server.virtnet: executing set_default_debug -1 all pdsh@oleg132-client: oleg132-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg132-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg132-server: oleg132-server.virtnet: executing set_default_debug -1 all pdsh@oleg132-client: oleg132-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg132-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg132-server: oleg132-server.virtnet: executing set_default_debug -1 all pdsh@oleg132-client: oleg132-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg132-server: oleg132-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg132-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg132-server: oleg132-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg132-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg132-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg132-server stop mds service on oleg132-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg132-server stop mds service on oleg132-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg132-server fail_loc=0 PASS 123ah (312s) == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 08:23:12 (1779711792) start mds service on oleg132-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg132-server: oleg132-server.virtnet: executing set_default_debug -1 all pdsh@oleg132-client: oleg132-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg132-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg132-server: oleg132-server.virtnet: executing set_default_debug -1 all pdsh@oleg132-client: oleg132-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg132-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg132-server: oleg132-server.virtnet: executing set_default_debug -1 all pdsh@oleg132-client: oleg132-server: ssh exited with exit code 1 Started lustre-OST0000 oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg132-client.virtnet: -o user_xattr,flock 192.168.201.132@tcp:/lustre /mnt/lustre oleg132-server: params: OBD_IOC_LLOG_PRINT failed: No such file or directory pdsh@oleg132-client: oleg132-server: ssh exited with exit code 2 - { index: 394, event: set_param, device: general, parameter: timeout, value: 129 } cleanup test 123ai timeout=20 timeout=20 PASS 123ai (288s) == conf-sanity test 123aj: check permanent TBF rules ===== 08:28:00 (1779712080) cleanup test 123aj oleg132-server: error: executing set_param: No such file or directory pdsh@oleg132-client: oleg132-server: ssh exited with exit code 2 ost.OSS.ost_io.nrs_policies=fifo PASS 123aj (61s) == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 08:29:02 (1779712142) oleg132-server: rm: cannot remove '/tmp/f123F.conf-sanity.yaml': No such file or directory pdsh@oleg132-client: oleg132-server: ssh exited with exit code 1 Unmounting FS Stopping clients: oleg132-client.virtnet /mnt/lustre (opts:) Stopping client oleg132-client.virtnet /mnt/lustre opts: Stopping clients: oleg132-client.virtnet /mnt/lustre2 (opts:) Stopping /mnt/lustre-mds1 (opts:-f) on oleg132-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg132-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg132-server Writeconf Remounting start mds service on oleg132-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg132-server: oleg132-server.virtnet: executing set_default_debug -1 all pdsh@oleg132-client: oleg132-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg132-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg132-server: oleg132-server.virtnet: executing set_default_debug -1 all pdsh@oleg132-client: oleg132-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg132-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg132-server: oleg132-server.virtnet: executing set_default_debug -1 all pdsh@oleg132-client: oleg132-server: ssh exited with exit code 1 Started lustre-OST0000 oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg132-client.virtnet: -o user_xattr,flock 192.168.201.132@tcp:/lustre /mnt/lustre Setting configuration parameters set_param: mdt.lustre-MDT0001.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity set_param: mdt.lustre-MDT0000.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity set_param: jobid_var=TESTNAME umount lustre on /mnt/lustre..... Stopping client oleg132-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg132-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg132-server stop mds service on oleg132-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg132-server stop mds service on oleg132-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg132-server unloading modules via unload_modules_local on: 'oleg132-server' oleg132-server: oleg132-server.virtnet: executing unload_modules_local oleg132-server: modules unloaded. pdsh@oleg132-client: oleg132-client: ssh exited with exit code 2 pdsh@oleg132-client: oleg132-server: ssh exited with exit code 2 pdsh@oleg132-client: oleg132-client: ssh exited with exit code 2 pdsh@oleg132-client: oleg132-server: ssh exited with exit code 2 PASS 123F (222s) == conf-sanity test 123G: clear and reset all parameters using apply_yaml ========================================================== 08:32:44 (1779712364) start mds service on oleg132-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg132-server' oleg132-server: oleg132-server.virtnet: executing load_modules_local oleg132-server: Loading modules from /home/green/git/lustre-release/lustre oleg132-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg132-server: detected 4 online CPUs by sysfs oleg132-server: MODOPTS_LIBCFS= oleg132-server: Force libcfs to create 2 CPU partitions oleg132-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg132-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg132-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg132-server: oleg132-server.virtnet: executing set_default_debug -1 all pdsh@oleg132-client: oleg132-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg132-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg132-server: oleg132-server.virtnet: executing set_default_debug -1 all pdsh@oleg132-client: oleg132-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg132-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg132-server: oleg132-server.virtnet: executing set_default_debug -1 all pdsh@oleg132-client: oleg132-server: ssh exited with exit code 1 Started lustre-OST0000 oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg132-client.virtnet: -o user_xattr,flock 192.168.201.132@tcp:/lustre /mnt/lustre oleg132-server: rm: cannot remove '/tmp/f123G.conf-sanity.yaml': No such file or directory pdsh@oleg132-client: oleg132-server: ssh exited with exit code 1 Unmounting FS Stopping clients: oleg132-client.virtnet /mnt/lustre (opts:) Stopping client oleg132-client.virtnet /mnt/lustre opts: Stopping clients: oleg132-client.virtnet /mnt/lustre2 (opts:) Stopping /mnt/lustre-mds1 (opts:-f) on oleg132-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg132-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg132-server Writeconf Remounting start mds service on oleg132-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg132-server: oleg132-server.virtnet: executing set_default_debug -1 all pdsh@oleg132-client: oleg132-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg132-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg132-server: oleg132-server.virtnet: executing set_default_debug -1 all pdsh@oleg132-client: oleg132-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg132-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg132-server: oleg132-server.virtnet: executing set_default_debug -1 all pdsh@oleg132-client: oleg132-server: ssh exited with exit code 1 Started lustre-OST0000 oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg132-client.virtnet: -o user_xattr,flock 192.168.201.132@tcp:/lustre /mnt/lustre Setting configuration parameters conf_param: lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity conf_param: lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity set_param: mdt.lustre-MDT0001.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity set_param: mdt.lustre-MDT0000.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity set_param: jobid_var=TESTNAME umount lustre on /mnt/lustre..... Stopping client oleg132-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg132-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg132-server stop mds service on oleg132-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg132-server stop mds service on oleg132-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg132-server unloading modules via unload_modules_local on: 'oleg132-server' oleg132-server: oleg132-server.virtnet: executing unload_modules_local oleg132-server: modules unloaded. pdsh@oleg132-client: oleg132-server: ssh exited with exit code 2 pdsh@oleg132-client: oleg132-client: ssh exited with exit code 2 pdsh@oleg132-client: oleg132-server: ssh exited with exit code 2 pdsh@oleg132-client: oleg132-client: ssh exited with exit code 2 PASS 123G (373s) == conf-sanity test 123H: check concurent accesses with 'lctl llog_print ========================================================== 08:38:57 (1779712737) start mds service on oleg132-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg132-server' oleg132-server: oleg132-server.virtnet: executing load_modules_local oleg132-server: Loading modules from /home/green/git/lustre-release/lustre oleg132-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg132-server: detected 4 online CPUs by sysfs oleg132-server: MODOPTS_LIBCFS= oleg132-server: Force libcfs to create 2 CPU partitions oleg132-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg132-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg132-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg132-server: oleg132-server.virtnet: executing set_default_debug -1 all pdsh@oleg132-client: oleg132-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg132-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg132-server: oleg132-server.virtnet: executing set_default_debug -1 all pdsh@oleg132-client: oleg132-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg132-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg132-server: oleg132-server.virtnet: executing set_default_debug -1 all pdsh@oleg132-client: oleg132-server: ssh exited with exit code 1 Started lustre-OST0000 oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg132-client.virtnet: -o user_xattr,flock 192.168.201.132@tcp:/lustre /mnt/lustre 20 744 oleg132-server: warning: setting '-P' option with '-d' PASS 123H (270s) == conf-sanity test 124: check failover after replace_nids ========================================================== 08:43:28 (1779713008) SKIP: conf-sanity test_124 needs MDT failover setup SKIP 124 (4s) == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 08:43:32 (1779713012) umount lustre on /mnt/lustre..... Stopping client oleg132-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg132-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg132-server stop mds service on oleg132-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg132-server stop mds service on oleg132-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg132-server unloading modules via unload_modules_local on: 'oleg132-server' oleg132-server: oleg132-server.virtnet: executing unload_modules_local oleg132-server: modules unloaded. oleg132-server: oleg132-server.virtnet: executing load_module ../libcfs/libcfs/libcfs fail_loc=0x60d oleg132-server: oleg132-server.virtnet: executing load_modules oleg132-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg132-server: Loading modules from /home/green/git/lustre-release/lustre oleg132-server: detected 4 online CPUs by sysfs oleg132-server: MODOPTS_LIBCFS= oleg132-server: Force libcfs to create 2 CPU partitions oleg132-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg132-server: quota/lquota options: 'hash_lqs_cur_bits=3' /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg132-server' clearing fail_loc on mds1 fail_loc=0 oleg132-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg132-server: oleg132-server.virtnet: executing load_modules_local oleg132-server: Loading modules from /home/green/git/lustre-release/lustre oleg132-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg132-server: detected 4 online CPUs by sysfs oleg132-server: MODOPTS_LIBCFS= oleg132-server: Force libcfs to create 2 CPU partitions Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg132-server: oleg132-server.virtnet: executing set_default_debug -1 all pdsh@oleg132-client: oleg132-server: ssh exited with exit code 1 Started lustre-MDT0000 PASS 126 (136s) == conf-sanity test 127: direct io overwrite on full ost ========================================================== 08:45:48 (1779713148) umount lustre on /mnt/lustre..... stop ost1 service on oleg132-server stop mds service on oleg132-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg132-server stop mds service on oleg132-server unloading modules via unload_modules_local on: 'oleg132-server' oleg132-server: oleg132-server.virtnet: executing unload_modules_local oleg132-server: modules unloaded. start mds service on oleg132-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg132-server' oleg132-server: oleg132-server.virtnet: executing load_modules_local oleg132-server: Loading modules from /home/green/git/lustre-release/lustre oleg132-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg132-server: detected 4 online CPUs by sysfs oleg132-server: MODOPTS_LIBCFS= oleg132-server: Force libcfs to create 2 CPU partitions oleg132-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg132-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg132-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg132-server: oleg132-server.virtnet: executing set_default_debug -1 all pdsh@oleg132-client: oleg132-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg132-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg132-server: oleg132-server.virtnet: executing set_default_debug -1 all pdsh@oleg132-client: oleg132-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg132-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg132-server: oleg132-server.virtnet: executing set_default_debug -1 all pdsh@oleg132-client: oleg132-server: ssh exited with exit code 1 Started lustre-OST0000 oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg132-client.virtnet: -o user_xattr,flock 192.168.201.132@tcp:/lustre /mnt/lustre Stopping clients: /mnt/lustre (opts:) pdsh@oleg132-client: no remote hosts specified osc.lustre-OST0000-osc-MDT0000.active updated after 0 sec (got 1) osc.lustre-OST0000-osc-MDT0001.active updated after 0 sec (got 1) Waiting for MDT destroys to complete oleg132-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg132-client: oleg132-server: ssh exited with exit code 1 dd: error writing '/mnt/lustre/f127.conf-sanity': No space left on device 124+0 records in 123+0 records out 128974848 bytes (129 MB, 123 MiB) copied, 14.5519 s, 8.9 MB/s 123+0 records in 123+0 records out 128974848 bytes (129 MB, 123 MiB) copied, 11.1878 s, 11.5 MB/s PASS 127 (215s) == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 08:49:23 (1779713363) SKIP: conf-sanity test_128 need separate mgs device SKIP 128 (5s) == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 08:49:28 (1779713368) Stopping clients: oleg132-client.virtnet /mnt/lustre (opts:) Stopping client oleg132-client.virtnet /mnt/lustre opts: Stopping clients: oleg132-client.virtnet /mnt/lustre2 (opts:) Stopping /mnt/lustre-mds1 (opts:-f) on oleg132-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg132-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg132-server start mds service on oleg132-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg132-server: oleg132-server.virtnet: executing set_default_debug -1 all pdsh@oleg132-client: oleg132-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg132-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg132-server: oleg132-server.virtnet: executing set_default_debug -1 all pdsh@oleg132-client: oleg132-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid Format ost1: /dev/mapper/ost1_flakey Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg132-server: mount.lustre: mount /dev/mapper/ost1_flakey at /mnt/lustre-ost1 failed: Address already in use oleg132-server: The target service's index is already in use. (/dev/mapper/ost1_flakey) pdsh@oleg132-client: oleg132-server: ssh exited with exit code 98 oleg132-server: error: set_param: param_path 'seq/cli-lustre:OST0000-super/width': No such file or directory oleg132-server: error: set_param: setting 'seq/cli-lustre:OST0000-super/width'='65536': No such file or directory pdsh@oleg132-client: oleg132-server: ssh exited with exit code 2 Start of /dev/mapper/ost1_flakey on ost1 failed 98 Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg132-server: mount.lustre: mount /dev/mapper/ost1_flakey at /mnt/lustre-ost1 failed: Address already in use oleg132-server: The target service's index is already in use. (/dev/mapper/ost1_flakey) pdsh@oleg132-client: oleg132-server: ssh exited with exit code 98 oleg132-server: error: set_param: param_path 'seq/cli-lustre:OST0000-super/width': No such file or directory oleg132-server: error: set_param: setting 'seq/cli-lustre:OST0000-super/width'='65536': No such file or directory pdsh@oleg132-client: oleg132-server: ssh exited with exit code 2 Start of /dev/mapper/ost1_flakey on ost1 failed 98 checking for existing Lustre data: found Read previous values: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x22 (OST first_time ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.132@tcp sys.timeout=20 Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre=OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x122 (OST first_time writeconf ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.132@tcp sys.timeout=20 Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg132-server: oleg132-server.virtnet: executing set_default_debug -1 all pdsh@oleg132-client: oleg132-server: ssh exited with exit code 1 Started lustre-OST0000 Stopping /mnt/lustre-ost1 (opts:) on oleg132-server stop mds service on oleg132-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg132-server stop mds service on oleg132-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg132-server PASS 129 (189s) == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 08:52:38 (1779713558) Checking servers environments Checking clients oleg132-client.virtnet environments /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg132-server' oleg132-server: oleg132-server.virtnet: executing load_modules_local oleg132-server: Loading modules from /home/green/git/lustre-release/lustre oleg132-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg132-server: detected 4 online CPUs by sysfs oleg132-server: MODOPTS_LIBCFS= oleg132-server: Force libcfs to create 2 CPU partitions Setup mgs, mdt, osts Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg132-server: oleg132-server.virtnet: executing set_default_debug -1 all pdsh@oleg132-client: oleg132-server: ssh exited with exit code 1 Started lustre-MDT0000 Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg132-server: oleg132-server.virtnet: executing set_default_debug -1 all pdsh@oleg132-client: oleg132-server: ssh exited with exit code 1 Started lustre-MDT0001 Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg132-server: oleg132-server.virtnet: executing set_default_debug -1 all pdsh@oleg132-client: oleg132-server: ssh exited with exit code 1 Started lustre-OST0000 Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg132-server: oleg132-server.virtnet: executing set_default_debug -1 all pdsh@oleg132-client: oleg132-server: ssh exited with exit code 1 Started lustre-OST0001 mount lustre on /mnt/lustre..... Starting client: oleg132-client.virtnet: -o user_xattr,flock 192.168.201.132@tcp:/lustre /mnt/lustre Starting client oleg132-client.virtnet: -o user_xattr,flock 192.168.201.132@tcp:/lustre /mnt/lustre Started clients oleg132-client.virtnet: 192.168.201.132@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8ab5e5428800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8ab5e5428800.idle_timeout=debug setting jobstats to procname_uid Setting lustre.sys.jobid_var from disable to procname_uid Waiting 90s for 'procname_uid' disable quota as required stop mds service on oleg132-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg132-server checking for existing Lustre data: found Read previous values: Device: /dev/mapper/mds2_flakey Target: lustre-MDT0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x1 (MDT ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: mgsnode=192.168.201.132@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity Permanent disk data: Device: /dev/mapper/mds2_flakey Target: lustre=MDT0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x101 (MDT writeconf ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: mgsnode=192.168.201.132@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity start mds service on oleg132-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg132-server: oleg132-server.virtnet: executing set_default_debug -1 all pdsh@oleg132-client: oleg132-server: ssh exited with exit code 1 Started lustre-MDT0001 PASS 130 (202s) == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 08:56:00 (1779713760) oleg132-server: debugfs 1.47.3-wc2 (11-Nov-2025) Checking servers environments Checking clients oleg132-client.virtnet environments /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg132-server' oleg132-server: oleg132-server.virtnet: executing load_modules_local oleg132-server: Loading modules from /home/green/git/lustre-release/lustre oleg132-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg132-server: detected 4 online CPUs by sysfs oleg132-server: MODOPTS_LIBCFS= oleg132-server: Force libcfs to create 2 CPU partitions Setup mgs, mdt, osts Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg132-server: mount.lustre: according to /etc/mtab /dev/mapper/mds1_flakey is already mounted on /mnt/lustre-mds1 pdsh@oleg132-client: oleg132-server: ssh exited with exit code 17 Start of /dev/mapper/mds1_flakey on mds1 failed 17 Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg132-server: mount.lustre: according to /etc/mtab /dev/mapper/mds2_flakey is already mounted on /mnt/lustre-mds2 pdsh@oleg132-client: oleg132-server: ssh exited with exit code 17 Start of /dev/mapper/mds2_flakey on mds2 failed 17 Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg132-server: mount.lustre: according to /etc/mtab /dev/mapper/ost1_flakey is already mounted on /mnt/lustre-ost1 pdsh@oleg132-client: oleg132-server: ssh exited with exit code 17 seq.cli-lustre-OST0000-super.width=65536 Start of /dev/mapper/ost1_flakey on ost1 failed 17 Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 oleg132-server: mount.lustre: according to /etc/mtab /dev/mapper/ost2_flakey is already mounted on /mnt/lustre-ost2 pdsh@oleg132-client: oleg132-server: ssh exited with exit code 17 seq.cli-lustre-OST0001-super.width=65536 Start of /dev/mapper/ost2_flakey on ost2 failed 17 mount lustre on /mnt/lustre..... Starting client: oleg132-client.virtnet: -o user_xattr,flock 192.168.201.132@tcp:/lustre /mnt/lustre mount.lustre: according to /etc/mtab 192.168.201.132@tcp:/lustre is already mounted on /mnt/lustre Starting client oleg132-client.virtnet: -o user_xattr,flock 192.168.201.132@tcp:/lustre /mnt/lustre Started clients oleg132-client.virtnet: 192.168.201.132@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8ab5e5428800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8ab5e5428800.idle_timeout=debug disable quota as required striped dir -i1 -c2 -H crush /mnt/lustre/d131.conf-sanity total: 512 open/close in 9.98 seconds: 51.31 ops/second striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d131.conf-sanity.inherit total: 128 open/close in 2.45 seconds: 52.34 ops/second Stopping clients: oleg132-client.virtnet /mnt/lustre (opts:) Stopping client oleg132-client.virtnet /mnt/lustre opts: Stopping clients: oleg132-client.virtnet /mnt/lustre2 (opts:) Stopping /mnt/lustre-mds1 (opts:-f) on oleg132-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg132-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg132-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg132-server file-level backup/restore on mds1:/dev/mapper/mds1_flakey backup data reformat new device Format mds1: /dev/mapper/mds1_flakey restore data remove recovery logs removed '/mnt/lustre-brpt/CATALOGS' file-level backup/restore on mds2:/dev/mapper/mds2_flakey backup data reformat new device Format mds2: /dev/mapper/mds2_flakey restore data remove recovery logs removed '/mnt/lustre-brpt/CATALOGS' Checking servers environments Checking clients oleg132-client.virtnet environments /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg132-server' oleg132-server: oleg132-server.virtnet: executing load_modules_local oleg132-server: Loading modules from /home/green/git/lustre-release/lustre oleg132-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg132-server: detected 4 online CPUs by sysfs oleg132-server: MODOPTS_LIBCFS= oleg132-server: Force libcfs to create 2 CPU partitions Setup mgs, mdt, osts Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg132-server: oleg132-server.virtnet: executing set_default_debug -1 all pdsh@oleg132-client: oleg132-server: ssh exited with exit code 1 Started lustre-MDT0000 Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg132-server: oleg132-server.virtnet: executing set_default_debug -1 all pdsh@oleg132-client: oleg132-server: ssh exited with exit code 1 Started lustre-MDT0001 Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg132-server: oleg132-server.virtnet: executing set_default_debug -1 all pdsh@oleg132-client: oleg132-server: ssh exited with exit code 1 Started lustre-OST0000 Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg132-server: oleg132-server.virtnet: executing set_default_debug -1 all pdsh@oleg132-client: oleg132-server: ssh exited with exit code 1 Started lustre-OST0001 mount lustre on /mnt/lustre..... Starting client: oleg132-client.virtnet: -o user_xattr,flock 192.168.201.132@tcp:/lustre /mnt/lustre Starting client oleg132-client.virtnet: -o user_xattr,flock 192.168.201.132@tcp:/lustre /mnt/lustre Started clients oleg132-client.virtnet: 192.168.201.132@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8ab5e542f000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8ab5e542f000.idle_timeout=debug disable quota as required total: 2 open/close in 0.13 seconds: 15.61 ops/second PASS 131 (531s) == conf-sanity test 132: hsm_actions processed after failover ========================================================== 09:04:51 (1779714291) Stopping clients: oleg132-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg132-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg132-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg132-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg132-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg132-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg132-server oleg132-server: oleg132-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg132-server' oleg132-server: oleg132-server.virtnet: executing load_modules_local oleg132-server: Loading modules from /home/green/git/lustre-release/lustre oleg132-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg132-server: detected 4 online CPUs by sysfs oleg132-server: MODOPTS_LIBCFS= oleg132-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg132-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg132-server: oleg132-server.virtnet: executing set_default_debug -1 all pdsh@oleg132-client: oleg132-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 stop mds service on oleg132-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg132-server checking for existing Lustre data: found Read previous values: Device: /dev/mapper/mds1_flakey Target: lustre-MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x5 (MDT MGS ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity Permanent disk data: Device: /dev/mapper/mds1_flakey Target: lustre-MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x45 (MDT MGS update ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity mdt.hsm_control=enabled start mds service on oleg132-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg132-server: oleg132-server.virtnet: executing set_default_debug -1 all pdsh@oleg132-client: oleg132-server: ssh exited with exit code 1 Started lustre-MDT0000 stop mds service on oleg132-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg132-server PASS 132 (212s) == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 09:08:24 (1779714504) SKIP: conf-sanity test_133 needs >= 4 OSTs SKIP 133 (6s) == conf-sanity test 134: check_iam works without faults == 09:08:29 (1779714509) Checking servers environments Checking clients oleg132-client.virtnet environments /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg132-server' oleg132-server: oleg132-server.virtnet: executing load_modules_local oleg132-server: Loading modules from /home/green/git/lustre-release/lustre oleg132-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg132-server: detected 4 online CPUs by sysfs oleg132-server: MODOPTS_LIBCFS= oleg132-server: Force libcfs to create 2 CPU partitions Setup mgs, mdt, osts Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg132-server: oleg132-server.virtnet: executing set_default_debug -1 all pdsh@oleg132-client: oleg132-server: ssh exited with exit code 1 Started lustre-MDT0000 Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg132-server: oleg132-server.virtnet: executing set_default_debug -1 all pdsh@oleg132-client: oleg132-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg132-server: oleg132-server.virtnet: executing set_default_debug -1 all pdsh@oleg132-client: oleg132-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg132-server: oleg132-server.virtnet: executing set_default_debug -1 all pdsh@oleg132-client: oleg132-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost2_flakey Started lustre-OST0001 mount lustre on /mnt/lustre..... Starting client: oleg132-client.virtnet: -o user_xattr,flock 192.168.201.132@tcp:/lustre /mnt/lustre Starting client oleg132-client.virtnet: -o user_xattr,flock 192.168.201.132@tcp:/lustre /mnt/lustre Started clients oleg132-client.virtnet: 192.168.201.132@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8ab5c75f0800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8ab5c75f0800.idle_timeout=debug disable quota as required striped dir -i0 -c2 -H crush /mnt/lustre/d134.conf-sanity Started LFSCK on the device lustre-MDT0000: scrub layout namespace recsize: 1 ptrsize: 4 indirect_levels: 0 idle_blocks: 0 Root entries: count 2, limit 203 000: [0:0x0:0x0] ptr: 1 Block 1,FIX leaf, Leaf block, count 1, limit 240 count 1 NO ERRORS 0 debugfs 1.47.3-wc2 (11-Nov-2025) Record type to print: lfsck_namespace Filesize 8192, blocks count 2 Root format: LFIX keysize: 16 recsize: 1 ptrsize: 4 indirect_levels: 0 idle_blocks: 0 Root entries: count 2, limit 203 000: [0:0x0:0x0] ptr: 1 Block 1,FIX leaf, Leaf block, count 1, limit 240 000: [0:0x0:0x0] 0x0 () count 1 NO ERRORS dd if=/dev/urandom of=/tmp/d134.conf-sanity/lfsck_namespace_14 bs=2 conv=notrunc count=1 seek=30 Filesize 8192, blocks count 2 Root format: LFIX keysize: 16 recsize: 1 ptrsize: 4 indirect_levels: 0 idle_blocks: 0 Root entries: count 2, limit 203 000: [0:0x0:0x0] ptr: 1 Block 1,FIX leaf, Leaf block, count 1, limit 240 count 1 NO ERRORS 0 pdsh@oleg132-client: oleg132-server: ssh exited with exit code 255 debugfs 1.47.3-wc2 (11-Nov-2025) Record type to print: lfsck_namespace Filesize 8192, blocks count 2 Root format: LFIX keysize: 16 recsize: 1 ptrsize: 4 indirect_levels: 0 idle_blocks: 0 Root entries: count 2, limit 203 000: [0:0x0:0x0] ptr: 1 Block 1,FIX leaf, Leaf block, count 1, limit 240 000: [0:0x0:0x0] 0x0 () count 1 NO ERRORS dd if=/dev/urandom of=/tmp/d134.conf-sanity/lfsck_namespace_15 bs=2 conv=notrunc count=1 seek=5 Filesize 8192, blocks count 2 Root format: LFIX keysize: 16 recsize: 34803 ptrsize: 4 indirect_levels: 0 idle_blocks: 0 Too large record + key or too small block, 34823, 4096 Root node is insane FINISHED WITH ERRORS 255 debugfs 1.47.3-wc2 (11-Nov-2025) Record type to print: nodemap Filesize 8192, blocks count 2 Root format: LFIX keysize: 8 recsize: 32 ptrsize: 4 indirect_levels: 0 idle_blocks: 0 Root entries: count 2, limit 340 000: { id: 0x0, type: empty(0) } ptr: 1 Block 1,FIX leaf, Leaf block, count 5, limit 102 000: { id: 0x0, type: empty(0) } {} 001: { id: 0x0, type: cluster(1), subtype: cluster } { name: default, flag: 0xb8, flag2: 0x4, squash_uid: 65534, squash_gid: 65534, squash_projid: 65534} 002: { id: 0x0, type: global(15) } { is_active: 0 } 003: { id: 0x1, type: cluster(1), subtype: cluster } { name: test_134, flag: 0xb8, flag2: 0x4, squash_uid: 65534, squash_gid: 65534, squash_projid: 65534} 004: { id: 0x1, type: range(2) } { start_nid: 121.23.2.100@tcp, end_nid: 121.23.2.120@tcp, type 0x0 } count 5 NO ERRORS dd if=/dev/urandom of=/tmp/d134.conf-sanity/nodemap bs=2 conv=notrunc count=1 seek=26 Filesize 8192, blocks count 2 Root format: LFIX keysize: 8 recsize: 32 ptrsize: 4 indirect_levels: 0 idle_blocks: 0 Root entries: count 2, limit 340 000: { id: 0x0, type: empty(0) } ptr: 1 Block 1,FIX leaf, Leaf block, count 5, limit 102 count 5 NO ERRORS 0 PASS 134 (1098s) == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 09:26:47 (1779715607) Stopping clients: oleg132-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg132-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg132-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg132-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg132-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg132-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg132-server oleg132-server: oleg132-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg132-server' oleg132-server: oleg132-server.virtnet: executing load_modules_local oleg132-server: Loading modules from /home/green/git/lustre-release/lustre oleg132-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg132-server: detected 4 online CPUs by sysfs oleg132-server: MODOPTS_LIBCFS= oleg132-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg132-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg132-server: oleg132-server.virtnet: executing set_default_debug -1 all pdsh@oleg132-client: oleg132-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg132-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg132-server: oleg132-server.virtnet: executing set_default_debug -1 all pdsh@oleg132-client: oleg132-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg132-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg132-server: oleg132-server.virtnet: executing set_default_debug -1 all pdsh@oleg132-client: oleg132-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg132-client.virtnet: -o user_xattr,flock 192.168.201.132@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H fnv_1a_64 /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 385 (time 1779715900.92 total 10.01 last 38.46) - open/close 770 (time 1779715910.93 total 20.02 last 38.45) - open/close 1163 (time 1779715920.93 total 30.03 last 39.28) - open/close 1580 (time 1779715930.94 total 40.04 last 41.65) - open/close 2090 (time 1779715940.95 total 50.04 last 50.99) - open/close 2615 (time 1779715950.95 total 60.05 last 52.48) - open/close 3112 (time 1779715960.96 total 70.05 last 49.66) - open/close 3667 (time 1779715970.97 total 80.06 last 55.45) - open/close 4103 (time 1779715980.98 total 90.07 last 43.55) total: 4500 open/close in 98.04 seconds: 45.90 ops/second - /unlink 939 (time 1779716005.80 total 10.01 last 93.84) - /unlink 1786 (time 1779716015.80 total 20.01 last 84.69) - /unlink 2670 (time 1779716025.81 total 30.02 last 88.31) - /unlink 3613 (time 1779716035.81 total 40.02 last 94.29) total: 4500 /unlink in 49.32 seconds: 91.23 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 521 (time 1779716062.92 total 10.02 last 52.00) - open/close 1016 (time 1779716072.93 total 20.03 last 49.46) - open/close 1491 (time 1779716082.93 total 30.03 last 47.49) - open/close 1926 (time 1779716092.95 total 40.04 last 43.45) - open/close 2377 (time 1779716102.97 total 50.06 last 45.01) - open/close 2869 (time 1779716112.97 total 60.06 last 49.18) - open/close 3363 (time 1779716122.98 total 70.07 last 49.36) - open/close 3875 (time 1779716132.99 total 80.09 last 51.11) - open/close 4459 (time 1779716143.00 total 90.10 last 58.33) total: 4500 open/close in 90.62 seconds: 49.66 ops/second - /unlink 928 (time 1779716160.61 total 10.01 last 92.74) - /unlink 1868 (time 1779716170.63 total 20.03 last 93.80) - /unlink 2847 (time 1779716180.64 total 30.04 last 97.82) - /unlink 3809 (time 1779716190.65 total 40.05 last 96.11) total: 4500 /unlink in 47.84 seconds: 94.06 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 392 (time 1779716216.21 total 10.01 last 39.17) - open/close 809 (time 1779716226.24 total 20.04 last 41.57) - open/close 1282 (time 1779716236.25 total 30.05 last 47.25) - open/close 1775 (time 1779716246.28 total 40.07 last 49.18) - open/close 2287 (time 1779716256.29 total 50.08 last 51.14) - open/close 2822 (time 1779716266.29 total 60.08 last 53.50) - open/close 3345 (time 1779716276.30 total 70.09 last 52.25) - open/close 3798 (time 1779716286.32 total 80.12 last 45.19) - open/close 4227 (time 1779716296.33 total 90.12 last 42.89) total: 4500 open/close in 95.22 seconds: 47.26 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 976 (time 1779716318.98 total 10.01 last 97.53) - /unlink 1998 (time 1779716329.00 total 20.02 last 102.09) - /unlink 2835 (time 1779716339.00 total 30.02 last 83.67) - /unlink 3678 (time 1779716349.00 total 40.02 last 84.29) total: 4500 /unlink in 49.75 seconds: 90.46 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 608 (time 1779716376.23 total 10.03 last 60.65) - open/close 1195 (time 1779716386.24 total 20.03 last 58.68) - open/close 1640 (time 1779716396.24 total 30.03 last 44.50) - open/close 2059 (time 1779716406.26 total 40.05 last 41.79) - open/close 2577 (time 1779716416.29 total 50.08 last 51.68) - open/close 3054 (time 1779716426.31 total 60.11 last 47.56) - open/close 3516 (time 1779716436.35 total 70.14 last 46.06) - open/close 3933 (time 1779716446.38 total 80.17 last 41.57) - open/close 4449 (time 1779716456.39 total 90.18 last 51.53) total: 4500 open/close in 91.40 seconds: 49.23 ops/second - /unlink 978 (time 1779716474.97 total 10.00 last 97.76) - /unlink 1793 (time 1779716484.97 total 20.00 last 81.50) - /unlink 2773 (time 1779716494.98 total 30.01 last 97.95) - /unlink 3618 (time 1779716504.98 total 40.01 last 84.46) total: 4500 /unlink in 49.91 seconds: 90.15 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 443 (time 1779716532.49 total 10.00 last 44.29) - open/close 999 (time 1779716542.50 total 20.01 last 55.54) - open/close 1547 (time 1779716552.53 total 30.04 last 54.67) - open/close 2065 (time 1779716562.54 total 40.05 last 51.75) - open/close 2588 (time 1779716572.54 total 50.05 last 52.28) - open/close 3041 (time 1779716582.55 total 60.06 last 45.25) - open/close 3519 (time 1779716592.58 total 70.10 last 47.63) - open/close 3956 (time 1779716602.59 total 80.10 last 43.66) - open/close 4465 (time 1779716612.61 total 90.12 last 50.81) total: 4500 open/close in 90.93 seconds: 49.49 ops/second - /unlink 885 (time 1779716631.08 total 10.00 last 88.48) - /unlink 1695 (time 1779716641.09 total 20.01 last 80.92) - /unlink 2511 (time 1779716651.09 total 30.01 last 81.59) - /unlink 3300 (time 1779716661.11 total 40.03 last 78.77) total: 4500 /unlink in 47.40 seconds: 94.94 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 1641 (time 1779716680.88 total 10.00 last 164.10) - open/close 3780 (time 1779716690.88 total 20.00 last 213.86) total: 4500 open/close in 22.23 seconds: 202.44 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 total: 4500 /unlink in 6.74 seconds: 667.86 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 3072 (time 1779716712.96 total 10.01 last 307.03) total: 4500 open/close in 15.07 seconds: 298.67 ops/second total: 4500 /unlink in 7.73 seconds: 581.86 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 3030 (time 1779716738.92 total 10.00 last 302.90) total: 4500 open/close in 14.74 seconds: 305.37 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 total: 4500 /unlink in 8.53 seconds: 527.49 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 2847 (time 1779716765.54 total 10.00 last 284.65) total: 4500 open/close in 16.82 seconds: 267.60 ops/second - /unlink 4261 (time 1779716784.41 total 10.00 last 426.01) total: 4500 /unlink in 10.57 seconds: 425.93 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 3009 (time 1779716796.87 total 10.00 last 300.89) total: 4500 open/close in 15.06 seconds: 298.80 ops/second total: 4500 /unlink in 7.75 seconds: 580.68 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 2976 (time 1779716822.90 total 10.00 last 297.50) total: 4500 open/close in 15.13 seconds: 297.43 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 total: 4500 /unlink in 6.27 seconds: 718.00 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 4024 (time 1779716847.28 total 10.00 last 402.40) total: 4500 open/close in 11.30 seconds: 398.37 ops/second total: 4500 /unlink in 6.69 seconds: 672.37 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 4290 (time 1779716868.25 total 10.00 last 428.93) total: 4500 open/close in 10.52 seconds: 427.84 ops/second total: 4500 /unlink in 5.34 seconds: 842.34 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 - open/close 3981 (time 1779716886.95 total 10.00 last 398.05) total: 4500 open/close in 11.38 seconds: 395.28 ops/second lustre-MDT0000: clear the changelog for cl1 to record #246998 total: 4500 /unlink in 6.05 seconds: 743.19 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 total: 4500 open/close in 9.89 seconds: 455.18 ops/second total: 4500 /unlink in 5.10 seconds: 881.50 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 total: 4500 open/close in 9.66 seconds: 466.02 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 total: 4500 /unlink in 4.95 seconds: 909.95 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 total: 4500 open/close in 9.72 seconds: 463.07 ops/second total: 4500 /unlink in 5.02 seconds: 895.74 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 total: 4500 open/close in 9.14 seconds: 492.35 ops/second total: 4500 /unlink in 5.36 seconds: 839.96 ops/second lustre-MDT0000: clear the changelog for cl1 to record #324998 total: 4500 open/close in 9.47 seconds: 475.35 ops/second lustre-MDT0000: clear the changelog for cl1 to record #337998 total: 4500 /unlink in 5.41 seconds: 832.34 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11196: 94413 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg132-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg132-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg132-server stop mds service on oleg132-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg132-server stop mds service on oleg132-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg132-server unloading modules via unload_modules_local on: 'oleg132-server' oleg132-server: oleg132-server.virtnet: executing unload_modules_local oleg132-server: modules unloaded. PASS 135 (1411s) == conf-sanity test 136: don't panic with bad obdecho setup ========================================================== 09:50:18 (1779717018) Stopping clients: oleg132-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg132-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg132-client: oleg132-server: ssh exited with exit code 2 oleg132-server: oleg132-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg132-server' oleg132-server: oleg132-server.virtnet: executing load_modules_local oleg132-server: Loading modules from /home/green/git/lustre-release/lustre oleg132-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg132-server: detected 4 online CPUs by sysfs oleg132-server: MODOPTS_LIBCFS= oleg132-server: Force libcfs to create 2 CPU partitions oleg132-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg132-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg132-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg132-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg132-server: oleg132-server.virtnet: executing set_default_debug -1 all pdsh@oleg132-client: oleg132-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg132-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg132-server: oleg132-server.virtnet: executing set_default_debug -1 all pdsh@oleg132-client: oleg132-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg132-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg132-server: oleg132-server.virtnet: executing set_default_debug -1 all pdsh@oleg132-client: oleg132-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg132-client.virtnet: -o user_xattr,flock 192.168.201.132@tcp:/lustre /mnt/lustre oleg132-server: oleg132-server.virtnet: executing load_module obdecho/obdecho oleg132-server: error: setup: Invalid argument pdsh@oleg132-client: oleg132-server: ssh exited with exit code 22 oleg132-server: error: test_mkdir: No such file or directory pdsh@oleg132-client: oleg132-server: ssh exited with exit code 2 Stopping clients: oleg132-client.virtnet /mnt/lustre (opts:) Stopping client oleg132-client.virtnet /mnt/lustre opts: Stopping clients: oleg132-client.virtnet /mnt/lustre2 (opts:) Stopping /mnt/lustre-mds1 (opts:-f) on oleg132-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg132-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg132-server Stopping clients: oleg132-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg132-client.virtnet /mnt/lustre2 (opts:-f) oleg132-server: oleg132-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg132-server' oleg132-server: oleg132-server.virtnet: executing load_modules_local oleg132-server: Loading modules from /home/green/git/lustre-release/lustre oleg132-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg132-server: detected 4 online CPUs by sysfs oleg132-server: MODOPTS_LIBCFS= oleg132-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg132-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg132-server: oleg132-server.virtnet: executing set_default_debug -1 all pdsh@oleg132-client: oleg132-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg132-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg132-server: oleg132-server.virtnet: executing set_default_debug -1 all pdsh@oleg132-client: oleg132-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg132-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg132-server: oleg132-server.virtnet: executing set_default_debug -1 all pdsh@oleg132-client: oleg132-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg132-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg132-server: oleg132-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg132-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg132-server: oleg132-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg132-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg132-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg132-server stop mds service on oleg132-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg132-server stop mds service on oleg132-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg132-server PASS 136 (152s) == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 09:52:50 (1779717170) SKIP: conf-sanity test_137 needs >= 3 MDTs SKIP 137 (1s) == conf-sanity test 140: remove_updatelog script actions ========================================================== 09:52:52 (1779717172) pdsh@oleg132-client: oleg132-server: ssh exited with exit code 1 start mds service on oleg132-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg132-server: oleg132-server.virtnet: executing set_default_debug -1 all pdsh@oleg132-client: oleg132-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg132-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg132-server: oleg132-server.virtnet: executing set_default_debug -1 all pdsh@oleg132-client: oleg132-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg132-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg132-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1