-----============= acceptance-small: conf-sanity ============----- Mon Sep 14 12:36:04 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg229-server: /home/green/git/lustre-release/lustre/tests/except/conf-sanity.ex - see CLIENT_VERSION >= v2_15_63-53-g1a6ef725c2 (34683431 >= 34553653) for LU-16938, go 81 - see CLIENT_VERSION >= v2_15_65-31-g74a5b9efaa (34683431 >= 34554143) for LU-18348, go 32b 32c - see CLIENT_VERSION >= v2_16_55-19-gf69f27cc35 (34683431 >= 34617107) for LU-17920, go 123ae 123ag - see CLIENT_VERSION >= v2_17_52-94-g86904ef943 (34683431 >= 34681950) for LU-19740, go 115 excepting tests: 32newtarball skipping tests SLOW=no: 45 69 106 111 114 Stopping clients: oleg229-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg229-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg229-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg229-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg229-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg229-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg229-server oleg229-server: oleg229-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg229-server' oleg229-server: oleg229-server.virtnet: executing load_modules_local oleg229-server: kmemleak disabled oleg229-server: Loading modules from /home/green/git/lustre-release/lustre oleg229-server: detected 4 online CPUs by sysfs oleg229-server: MODOPTS_LIBCFS= oleg229-server: Force libcfs to create 2 CPU partitions oleg229-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg229-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg229-server: oleg229-server.virtnet: executing set_default_debug -1 all pdsh@oleg229-client: oleg229-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg229-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg229-server: oleg229-server.virtnet: executing set_default_debug -1 all pdsh@oleg229-client: oleg229-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg229-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg229-server: oleg229-server.virtnet: executing set_default_debug -1 all pdsh@oleg229-client: oleg229-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg229-server: oleg229-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg229-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg229-server: oleg229-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg229-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg229-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg229-server stop mds service on oleg229-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg229-server stop mds service on oleg229-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg229-server == conf-sanity test 121: failover MGS ==================== 12:40:08 (1789404008) Stopping clients: oleg229-client.virtnet /mnt/lustre (opts:) Stopping clients: oleg229-client.virtnet /mnt/lustre2 (opts:) start mds service on oleg229-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg229-server: oleg229-server.virtnet: executing set_default_debug -1 all pdsh@oleg229-client: oleg229-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg229-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg229-server: oleg229-server.virtnet: executing set_default_debug -1 all pdsh@oleg229-client: oleg229-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 Failing mgs on oleg229-server Stopping /mnt/lustre-mds1 (opts:) on oleg229-server 12:41:09 (1789404069) shut down facet: mgs facet_host: oleg229-server facet_failover_host: oleg229-server Failover mgs to oleg229-server mount facets: mgs Start mgs: mount -t lustre /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg229-server: oleg229-server.virtnet: executing set_default_debug -1 all pdsh@oleg229-client: oleg229-server: ssh exited with exit code 1 Started lustre-MDT0000 12:41:35 (1789404095) targets are mounted 12:41:35 (1789404095) facet_failover done pdsh@oleg229-client: oleg229-client: ssh exited with exit code 95 oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 pdsh@oleg229-client: oleg229-client: ssh exited with exit code 95 stop mds service on oleg229-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg229-server stop mds service on oleg229-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg229-server start mds service on oleg229-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg229-server: oleg229-server.virtnet: executing set_default_debug -1 all pdsh@oleg229-client: oleg229-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg229-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg229-server: oleg229-server.virtnet: executing set_default_debug -1 all pdsh@oleg229-client: oleg229-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 Failing mgs on oleg229-server Stopping /mnt/lustre-mds1 (opts:) on oleg229-server 12:42:44 (1789404164) shut down facet: mgs facet_host: oleg229-server facet_failover_host: oleg229-server Failover mgs to oleg229-server mount facets: mgs Start mgs: mount -t lustre /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg229-server: oleg229-server.virtnet: executing set_default_debug -1 all pdsh@oleg229-client: oleg229-server: ssh exited with exit code 1 Started lustre-MDT0000 12:43:17 (1789404197) targets are mounted 12:43:17 (1789404197) facet_failover done pdsh@oleg229-client: oleg229-client: ssh exited with exit code 95 oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 pdsh@oleg229-client: oleg229-client: ssh exited with exit code 95 stop mds service on oleg229-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg229-server stop mds service on oleg229-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg229-server PASS 121 (231s) == conf-sanity test 122a: Check OST sequence update ====== 12:43:59 (1789404239) Stopping clients: oleg229-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg229-client.virtnet /mnt/lustre2 (opts:-f) oleg229-server: oleg229-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg229-server' oleg229-server: oleg229-server.virtnet: executing load_modules_local oleg229-server: kmemleak disabled oleg229-server: Loading modules from /home/green/git/lustre-release/lustre oleg229-server: detected 4 online CPUs by sysfs oleg229-server: MODOPTS_LIBCFS= oleg229-server: Force libcfs to create 2 CPU partitions oleg229-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg229-server' oleg229-server: oleg229-server.virtnet: executing load_modules_local oleg229-server: kmemleak disabled oleg229-server: Loading modules from /home/green/git/lustre-release/lustre oleg229-server: detected 4 online CPUs by sysfs oleg229-server: MODOPTS_LIBCFS= oleg229-server: Force libcfs to create 2 CPU partitions oleg229-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt fail_loc=0x00001e0 start mds service on oleg229-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg229-server: oleg229-server.virtnet: executing set_default_debug -1 all pdsh@oleg229-client: oleg229-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg229-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg229-server: oleg229-server.virtnet: executing set_default_debug -1 all pdsh@oleg229-client: oleg229-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg229-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg229-server: oleg229-server.virtnet: executing set_default_debug -1 all pdsh@oleg229-client: oleg229-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg229-client.virtnet: -o user_xattr,flock 192.168.202.129@tcp:/lustre /mnt/lustre fail_loc=0 - open/close 821 (time 1789404410.96 total 10.01 last 82.05) total: 1000 open/close in 12.06 seconds: 82.92 ops/second umount lustre on /mnt/lustre..... Stopping client oleg229-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg229-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg229-server stop mds service on oleg229-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg229-server stop mds service on oleg229-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg229-server unloading modules via unload_modules_local on: 'oleg229-server' oleg229-server: oleg229-server.virtnet: executing unload_modules_local oleg229-server: kmemleak disabled modules unloaded. pdsh@oleg229-client: oleg229-server: ssh exited with exit code 2 pdsh@oleg229-client: oleg229-client: ssh exited with exit code 2 pdsh@oleg229-client: oleg229-client: ssh exited with exit code 2 pdsh@oleg229-client: oleg229-server: ssh exited with exit code 2 PASS 122a (229s) == conf-sanity test 123aa: llog_print works with FIDs and simple names ========================================================== 12:47:48 (1789404468) start mds service on oleg229-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1931560 Sep 14 05:02 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg229-server' oleg229-server: oleg229-server.virtnet: executing load_modules_local oleg229-server: kmemleak disabled oleg229-server: Loading modules from /home/green/git/lustre-release/lustre oleg229-server: detected 4 online CPUs by sysfs oleg229-server: MODOPTS_LIBCFS= oleg229-server: Force libcfs to create 2 CPU partitions oleg229-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg229-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg229-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg229-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg229-server: oleg229-server.virtnet: executing set_default_debug -1 all pdsh@oleg229-client: oleg229-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg229-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg229-server: oleg229-server.virtnet: executing set_default_debug -1 all pdsh@oleg229-client: oleg229-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg229-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg229-server: oleg229-server.virtnet: executing set_default_debug -1 all pdsh@oleg229-client: oleg229-server: ssh exited with exit code 1 Started lustre-OST0000 oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg229-client.virtnet: -o user_xattr,flock 192.168.202.129@tcp:/lustre /mnt/lustre 1 UP mgs MGS MGS 7 - { index: 2, event: attach, device: lustre-clilov, type: lov, UUID: lustre-clilov_UUID } - { index: 3, event: setup, device: lustre-clilov, UUID: } - { index: 6, event: attach, device: lustre-clilmv, type: lmv, UUID: lustre-clilmv_UUID } - { index: 7, event: setup, device: lustre-clilmv, UUID: } - { index: 10, event: new_profile, name: lustre-client, lov: lustre-clilov, lmv: lustre-clilmv } - { index: 2, event: attach, device: lustre-clilov, type: lov, UUID: lustre-clilov_UUID } - { index: 3, event: setup, device: lustre-clilov, UUID: } - { index: 6, event: attach, device: lustre-clilmv, type: lmv, UUID: lustre-clilmv_UUID } - { index: 7, event: setup, device: lustre-clilmv, UUID: } - { index: 10, event: new_profile, name: lustre-client, lov: lustre-clilov, lmv: lustre-clilmv } - { index: 13, event: add_uuid, nid: 192.168.202.129@tcp(0x20000c0a8ca81), node: 192.168.202.129@tcp } - { index: 14, event: attach, device: lustre-MDT0000-mdc, type: mdc, UUID: lustre-clilmv_UUID } - { index: 15, event: setup, device: lustre-MDT0000-mdc, UUID: lustre-MDT0000_UUID, node: 192.168.202.129@tcp } - { index: 16, event: add_mdc, device: lustre-clilmv, mdt: lustre-MDT0000_UUID, index: 0, gen: 1, UUID: lustre-MDT0000-mdc_UUID } - { index: 22, event: add_uuid, nid: 192.168.202.129@tcp(0x20000c0a8ca81), node: 192.168.202.129@tcp } - { index: 23, event: attach, device: lustre-MDT0001-mdc, type: mdc, UUID: lustre-clilmv_UUID } - { index: 24, event: setup, device: lustre-MDT0001-mdc, UUID: lustre-MDT0001_UUID, node: 192.168.202.129@tcp } - { index: 25, event: add_mdc, device: lustre-clilmv, mdt: lustre-MDT0001_UUID, index: 1, gen: 1, UUID: lustre-MDT0001-mdc_UUID } - { index: 31, event: add_uuid, nid: 192.168.202.129@tcp(0x20000c0a8ca81), node: 192.168.202.129@tcp } - { index: 32, event: attach, device: lustre-OST0000-osc, type: osc, UUID: lustre-clilov_UUID } - { index: 33, event: setup, device: lustre-OST0000-osc, UUID: lustre-OST0000_UUID, node: 192.168.202.129@tcp } - { index: 34, event: add_osc, device: lustre-clilov, ost: lustre-OST0000_UUID, index: 0, gen: 1 } - { index: 37, event: set_timeout, num: 0x000014, parameter: sys.timeout=20 } PASS 123aa (114s) == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 12:49:42 (1789404582) PASS 123ab (12s) == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 12:49:54 (1789404594) PASS 123ac (9s) == conf-sanity test 123ad: llog_print shows all records == 12:50:03 (1789404603) PASS 123ad (14s) == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 12:50:17 (1789404617) - { index: 11, event: set_param, device: general, parameter: osc.*.max_dirty_mb, value: 462 } - { index: 46, event: conf_param, device: lustre-OST0000-osc, parameter: osc.max_dirty_mb=462 } - { index: 2, event: attach, device: lustre-clilov, type: lov, UUID: lustre-clilov_UUID } - { index: 3, event: setup, device: lustre-clilov, UUID: } - { index: 6, event: attach, device: lustre-clilmv, type: lmv, UUID: lustre-clilmv_UUID } - { index: 7, event: setup, device: lustre-clilmv, UUID: } - { index: 10, event: new_profile, name: lustre-client, lov: lustre-clilov, lmv: lustre-clilmv } - { index: 13, event: add_uuid, nid: 192.168.202.129@tcp(0x20000c0a8ca81), node: 192.168.202.129@tcp } - { index: 14, event: attach, device: lustre-MDT0000-mdc, type: mdc, UUID: lustre-clilmv_UUID } - { index: 15, event: setup, device: lustre-MDT0000-mdc, UUID: lustre-MDT0000_UUID, node: 192.168.202.129@tcp } - { index: 16, event: add_mdc, device: lustre-clilmv, mdt: lustre-MDT0000_UUID, index: 0, gen: 1, UUID: lustre-MDT0000-mdc_UUID } - { index: 22, event: add_uuid, nid: 192.168.202.129@tcp(0x20000c0a8ca81), node: 192.168.202.129@tcp } - { index: 23, event: attach, device: lustre-MDT0001-mdc, type: mdc, UUID: lustre-clilmv_UUID } - { index: 24, event: setup, device: lustre-MDT0001-mdc, UUID: lustre-MDT0001_UUID, node: 192.168.202.129@tcp } - { index: 25, event: add_mdc, device: lustre-clilmv, mdt: lustre-MDT0001_UUID, index: 1, gen: 1, UUID: lustre-MDT0001-mdc_UUID } - { index: 31, event: add_uuid, nid: 192.168.202.129@tcp(0x20000c0a8ca81), node: 192.168.202.129@tcp } - { index: 32, event: attach, device: lustre-OST0000-osc, type: osc, UUID: lustre-clilov_UUID } - { index: 33, event: setup, device: lustre-OST0000-osc, UUID: lustre-OST0000_UUID, node: 192.168.202.129@tcp } - { index: 34, event: add_osc, device: lustre-clilov, ost: lustre-OST0000_UUID, index: 0, gen: 1 } - { index: 37, event: set_timeout, num: 0x000014, parameter: sys.timeout=20 } - { index: 43, event: conf_param, device: lustre-OST0000-osc, parameter: osc.max_pages_per_rpc=1024 } - { index: 46, event: conf_param, device: lustre-OST0000-osc, parameter: osc.max_dirty_mb=462 } - { index: 2, event: attach, device: lustre-clilov, type: lov, UUID: lustre-clilov_UUID } - { index: 3, event: setup, device: lustre-clilov, UUID: } - { index: 6, event: attach, device: lustre-clilmv, type: lmv, UUID: lustre-clilmv_UUID } - { index: 7, event: setup, device: lustre-clilmv, UUID: } - { index: 10, event: new_profile, name: lustre-client, lov: lustre-clilov, lmv: lustre-clilmv } - { index: 13, event: add_uuid, nid: 192.168.202.129@tcp(0x20000c0a8ca81), node: 192.168.202.129@tcp } - { index: 14, event: attach, device: lustre-MDT0000-mdc, type: mdc, UUID: lustre-clilmv_UUID } - { index: 15, event: setup, device: lustre-MDT0000-mdc, UUID: lustre-MDT0000_UUID, node: 192.168.202.129@tcp } - { index: 16, event: add_mdc, device: lustre-clilmv, mdt: lustre-MDT0000_UUID, index: 0, gen: 1, UUID: lustre-MDT0000-mdc_UUID } - { index: 22, event: add_uuid, nid: 192.168.202.129@tcp(0x20000c0a8ca81), node: 192.168.202.129@tcp } - { index: 23, event: attach, device: lustre-MDT0001-mdc, type: mdc, UUID: lustre-clilmv_UUID } - { index: 24, event: setup, device: lustre-MDT0001-mdc, UUID: lustre-MDT0001_UUID, node: 192.168.202.129@tcp } - { index: 25, event: add_mdc, device: lustre-clilmv, mdt: lustre-MDT0001_UUID, index: 1, gen: 1, UUID: lustre-MDT0001-mdc_UUID } - { index: 31, event: add_uuid, nid: 192.168.202.129@tcp(0x20000c0a8ca81), node: 192.168.202.129@tcp } - { index: 32, event: attach, device: lustre-OST0000-osc, type: osc, UUID: lustre-clilov_UUID } - { index: 33, event: setup, device: lustre-OST0000-osc, UUID: lustre-OST0000_UUID, node: 192.168.202.129@tcp } - { index: 34, event: add_osc, device: lustre-clilov, ost: lustre-OST0000_UUID, index: 0, gen: 1 } - { index: 37, event: set_timeout, num: 0x000014, parameter: sys.timeout=20 } - { index: 43, event: conf_param, device: lustre-OST0000-osc, parameter: osc.max_pages_per_rpc=1024 } PASS 123ae (26s) == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 12:50:43 (1789404643) lctl --device MGS llog_catlist ... orig_clist: lustre-client lustre-OST0000 lustre-MDT0000 lustre-MDT0001 fail_loc=0x131b fail_val=2 new_clist: lustre-OST0000 lustre-MDT0000 lustre-MDT0001 fail_loc=0 done lctl --device lustre-MDT0000 llog_catlist ... orig_clist: [0x1:0x2:0x0] fail_loc=0x131b fail_val=2 new_clist: fail_loc=0 done fail_loc=0 PASS 123af (16s) == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 12:50:59 (1789404659) PASS 123ag (16s) == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 12:51:15 (1789404675) del_ost: dry run for target lustre-OST0000 config_log: lustre-client [DRY RUN] cancel catalog 'lustre-client:34':"- { index: 34, event: add_osc, device: lustre-clilov, ost: lustre-OST0000_UUID, index: 0, gen: 1 }" [DRY RUN] cancel catalog 'lustre-client:33':"- { index: 33, event: setup, device: lustre-OST0000-osc, UUID: lustre-OST0000_UUID, node: 192.168.202.129@tcp }" [DRY RUN] cancel catalog 'lustre-client:32':"- { index: 32, event: attach, device: lustre-OST0000-osc, type: osc, UUID: lustre-clilov_UUID }" del_ost: no catalog entry deleted config_log: lustre-MDT0000 [DRY RUN] cancel catalog 'lustre-MDT0000:41':"- { index: 41, event: conf_param, device: lustre-OST0000-osc-MDT0000, parameter: osc.max_dirty_mb=462 }" [DRY RUN] cancel catalog 'lustre-MDT0000:29':"- { index: 29, event: add_osc, device: lustre-MDT0000-mdtlov, ost: lustre-OST0000_UUID, index: 0, gen: 1 }" [DRY RUN] cancel catalog 'lustre-MDT0000:28':"- { index: 28, event: setup, device: lustre-OST0000-osc-MDT0000, UUID: lustre-OST0000_UUID, node: 192.168.202.129@tcp }" [DRY RUN] cancel catalog 'lustre-MDT0000:27':"- { index: 27, event: attach, device: lustre-OST0000-osc-MDT0000, type: osc, UUID: lustre-MDT0000-mdtlov_UUID }" del_ost: no catalog entry deleted config_log: lustre-MDT0001 [DRY RUN] cancel catalog 'lustre-MDT0001:41':"- { index: 41, event: conf_param, device: lustre-OST0000-osc-MDT0001, parameter: osc.max_dirty_mb=462 }" [DRY RUN] cancel catalog 'lustre-MDT0001:29':"- { index: 29, event: add_osc, device: lustre-MDT0001-mdtlov, ost: lustre-OST0000_UUID, index: 0, gen: 1 }" [DRY RUN] cancel catalog 'lustre-MDT0001:28':"- { index: 28, event: setup, device: lustre-OST0000-osc-MDT0001, UUID: lustre-OST0000_UUID, node: 192.168.202.129@tcp }" [DRY RUN] cancel catalog 'lustre-MDT0001:27':"- { index: 27, event: attach, device: lustre-OST0000-osc-MDT0001, type: osc, UUID: lustre-MDT0001-mdtlov_UUID }" del_ost: no catalog entry deleted config_log: lustre-client cancel catalog lustre-client log_idx 34: done cancel catalog lustre-client log_idx 33: done cancel catalog lustre-client log_idx 32: done del_ost: cancelled 3 catalog entries config_log: lustre-MDT0000 cancel catalog lustre-MDT0000 log_idx 41: done cancel catalog lustre-MDT0000 log_idx 29: done cancel catalog lustre-MDT0000 log_idx 28: done cancel catalog lustre-MDT0000 log_idx 27: done del_ost: cancelled 4 catalog entries config_log: lustre-MDT0001 cancel catalog lustre-MDT0001 log_idx 41: done cancel catalog lustre-MDT0001 log_idx 29: done cancel catalog lustre-MDT0001 log_idx 28: done cancel catalog lustre-MDT0001 log_idx 27: done del_ost: cancelled 4 catalog entries umount lustre on /mnt/lustre..... Stopping client oleg229-client.virtnet /mnt/lustre (opts:) mount lustre on /mnt/lustre..... Starting client: oleg229-client.virtnet: -o user_xattr,flock 192.168.202.129@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg229-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg229-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg229-server stop mds service on oleg229-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg229-server stop mds service on oleg229-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg229-server unloading modules via unload_modules_local on: 'oleg229-server' oleg229-server: oleg229-server.virtnet: executing unload_modules_local oleg229-server: kmemleak disabled modules unloaded. Stopping clients: oleg229-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg229-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg229-client: oleg229-server: ssh exited with exit code 2 oleg229-server: oleg229-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1931560 Sep 14 05:02 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg229-server' oleg229-server: oleg229-server.virtnet: executing load_modules_local oleg229-server: kmemleak disabled oleg229-server: Loading modules from /home/green/git/lustre-release/lustre oleg229-server: detected 4 online CPUs by sysfs oleg229-server: MODOPTS_LIBCFS= oleg229-server: Force libcfs to create 2 CPU partitions oleg229-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg229-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg229-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg229-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg229-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg229-server: oleg229-server.virtnet: executing set_default_debug -1 all pdsh@oleg229-client: oleg229-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg229-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg229-server: oleg229-server.virtnet: executing set_default_debug -1 all pdsh@oleg229-client: oleg229-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg229-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg229-server: oleg229-server.virtnet: executing set_default_debug -1 all pdsh@oleg229-client: oleg229-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg229-server: oleg229-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg229-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg229-server: oleg229-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg229-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg229-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg229-server stop mds service on oleg229-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg229-server stop mds service on oleg229-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg229-server fail_loc=0 PASS 123ah (243s) == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 12:55:19 (1789404919) start mds service on oleg229-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg229-server: oleg229-server.virtnet: executing set_default_debug -1 all pdsh@oleg229-client: oleg229-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg229-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg229-server: oleg229-server.virtnet: executing set_default_debug -1 all pdsh@oleg229-client: oleg229-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg229-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg229-server: oleg229-server.virtnet: executing set_default_debug -1 all pdsh@oleg229-client: oleg229-server: ssh exited with exit code 1 Started lustre-OST0000 oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg229-client.virtnet: -o user_xattr,flock 192.168.202.129@tcp:/lustre /mnt/lustre oleg229-server: params: OBD_IOC_LLOG_PRINT failed: No such file or directory pdsh@oleg229-client: oleg229-server: ssh exited with exit code 2 - { index: 394, event: set_param, device: general, parameter: timeout, value: 129 } cleanup test 123ai timeout=20 timeout=20 PASS 123ai (216s) == conf-sanity test 123aj: check permanent TBF rules ===== 12:58:55 (1789405135) cleanup test 123aj oleg229-server: error: executing set_param: No such file or directory pdsh@oleg229-client: oleg229-server: ssh exited with exit code 2 ost.OSS.ost_io.nrs_policies=fifo PASS 123aj (46s) == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 12:59:40 (1789405180) oleg229-server: rm: cannot remove '/tmp/f123F.conf-sanity.yaml': No such file or directory pdsh@oleg229-client: oleg229-server: ssh exited with exit code 1 Unmounting FS Stopping clients: oleg229-client.virtnet /mnt/lustre (opts:) Stopping client oleg229-client.virtnet /mnt/lustre opts: Stopping clients: oleg229-client.virtnet /mnt/lustre2 (opts:) Stopping /mnt/lustre-mds1 (opts:-f) on oleg229-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg229-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg229-server Writeconf Remounting start mds service on oleg229-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg229-server: oleg229-server.virtnet: executing set_default_debug -1 all pdsh@oleg229-client: oleg229-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg229-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg229-server: oleg229-server.virtnet: executing set_default_debug -1 all pdsh@oleg229-client: oleg229-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg229-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg229-server: oleg229-server.virtnet: executing set_default_debug -1 all pdsh@oleg229-client: oleg229-server: ssh exited with exit code 1 Started lustre-OST0000 oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg229-client.virtnet: -o user_xattr,flock 192.168.202.129@tcp:/lustre /mnt/lustre Setting configuration parameters set_param: mdt.lustre-MDT0001.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity set_param: mdt.lustre-MDT0000.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity set_param: jobid_var=TESTNAME umount lustre on /mnt/lustre..... Stopping client oleg229-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg229-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg229-server stop mds service on oleg229-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg229-server stop mds service on oleg229-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg229-server unloading modules via unload_modules_local on: 'oleg229-server' oleg229-server: oleg229-server.virtnet: executing unload_modules_local oleg229-server: kmemleak disabled modules unloaded. pdsh@oleg229-client: oleg229-client: ssh exited with exit code 2 pdsh@oleg229-client: oleg229-server: ssh exited with exit code 2 pdsh@oleg229-client: oleg229-server: ssh exited with exit code 2 pdsh@oleg229-client: oleg229-client: ssh exited with exit code 2 PASS 123F (186s) == conf-sanity test 123G: clear and reset all parameters using apply_yaml ========================================================== 13:02:46 (1789405366) start mds service on oleg229-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1931560 Sep 14 05:02 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg229-server' oleg229-server: oleg229-server.virtnet: executing load_modules_local oleg229-server: kmemleak disabled oleg229-server: Loading modules from /home/green/git/lustre-release/lustre oleg229-server: detected 4 online CPUs by sysfs oleg229-server: MODOPTS_LIBCFS= oleg229-server: Force libcfs to create 2 CPU partitions oleg229-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg229-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg229-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg229-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg229-server: oleg229-server.virtnet: executing set_default_debug -1 all pdsh@oleg229-client: oleg229-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg229-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg229-server: oleg229-server.virtnet: executing set_default_debug -1 all pdsh@oleg229-client: oleg229-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg229-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg229-server: oleg229-server.virtnet: executing set_default_debug -1 all pdsh@oleg229-client: oleg229-server: ssh exited with exit code 1 Started lustre-OST0000 oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg229-client.virtnet: -o user_xattr,flock 192.168.202.129@tcp:/lustre /mnt/lustre oleg229-server: rm: cannot remove '/tmp/f123G.conf-sanity.yaml': No such file or directory pdsh@oleg229-client: oleg229-server: ssh exited with exit code 1 Unmounting FS Stopping clients: oleg229-client.virtnet /mnt/lustre (opts:) Stopping client oleg229-client.virtnet /mnt/lustre opts: Stopping clients: oleg229-client.virtnet /mnt/lustre2 (opts:) Stopping /mnt/lustre-mds1 (opts:-f) on oleg229-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg229-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg229-server Writeconf Remounting start mds service on oleg229-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg229-server: oleg229-server.virtnet: executing set_default_debug -1 all pdsh@oleg229-client: oleg229-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg229-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg229-server: oleg229-server.virtnet: executing set_default_debug -1 all pdsh@oleg229-client: oleg229-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg229-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg229-server: oleg229-server.virtnet: executing set_default_debug -1 all pdsh@oleg229-client: oleg229-server: ssh exited with exit code 1 Started lustre-OST0000 oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg229-client.virtnet: -o user_xattr,flock 192.168.202.129@tcp:/lustre /mnt/lustre Setting configuration parameters conf_param: lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity conf_param: lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity set_param: mdt.lustre-MDT0001.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity set_param: mdt.lustre-MDT0000.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity set_param: jobid_var=TESTNAME umount lustre on /mnt/lustre..... Stopping client oleg229-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg229-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg229-server stop mds service on oleg229-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg229-server stop mds service on oleg229-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg229-server unloading modules via unload_modules_local on: 'oleg229-server' oleg229-server: oleg229-server.virtnet: executing unload_modules_local oleg229-server: kmemleak disabled modules unloaded. pdsh@oleg229-client: oleg229-client: ssh exited with exit code 2 pdsh@oleg229-client: oleg229-server: ssh exited with exit code 2 pdsh@oleg229-client: oleg229-server: ssh exited with exit code 2 pdsh@oleg229-client: oleg229-client: ssh exited with exit code 2 PASS 123G (288s) == conf-sanity test 123H: check concurent accesses with 'lctl llog_print ========================================================== 13:07:34 (1789405654) start mds service on oleg229-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1931560 Sep 14 05:02 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg229-server' oleg229-server: oleg229-server.virtnet: executing load_modules_local oleg229-server: kmemleak disabled oleg229-server: Loading modules from /home/green/git/lustre-release/lustre oleg229-server: detected 4 online CPUs by sysfs oleg229-server: MODOPTS_LIBCFS= oleg229-server: Force libcfs to create 2 CPU partitions oleg229-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg229-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg229-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg229-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg229-server: oleg229-server.virtnet: executing set_default_debug -1 all pdsh@oleg229-client: oleg229-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg229-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg229-server: oleg229-server.virtnet: executing set_default_debug -1 all pdsh@oleg229-client: oleg229-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg229-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg229-server: oleg229-server.virtnet: executing set_default_debug -1 all pdsh@oleg229-client: oleg229-server: ssh exited with exit code 1 Started lustre-OST0000 oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg229-client.virtnet: -o user_xattr,flock 192.168.202.129@tcp:/lustre /mnt/lustre 20 744 oleg229-server: warning: setting '-P' option with '-d' PASS 123H (211s) == conf-sanity test 124: check failover after replace_nids ========================================================== 13:11:06 (1789405866) SKIP: conf-sanity test_124 needs MDT failover setup SKIP 124 (4s) == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 13:11:09 (1789405869) umount lustre on /mnt/lustre..... Stopping client oleg229-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg229-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg229-server stop mds service on oleg229-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg229-server stop mds service on oleg229-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg229-server unloading modules via unload_modules_local on: 'oleg229-server' oleg229-server: oleg229-server.virtnet: executing unload_modules_local oleg229-server: kmemleak disabled modules unloaded. oleg229-server: oleg229-server.virtnet: executing load_module ../libcfs/libcfs/libcfs fail_loc=0x60d oleg229-server: oleg229-server.virtnet: executing load_modules oleg229-server: kmemleak disabled oleg229-server: Loading modules from /home/green/git/lustre-release/lustre oleg229-server: detected 4 online CPUs by sysfs oleg229-server: MODOPTS_LIBCFS= oleg229-server: Force libcfs to create 2 CPU partitions oleg229-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg229-server: quota/lquota options: 'hash_lqs_cur_bits=3' Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1931560 Sep 14 05:02 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg229-server' oleg229-server: oleg229-server.virtnet: executing load_modules_local oleg229-server: kmemleak disabled oleg229-server: Loading modules from /home/green/git/lustre-release/lustre oleg229-server: detected 4 online CPUs by sysfs oleg229-server: MODOPTS_LIBCFS= oleg229-server: Force libcfs to create 2 CPU partitions clearing fail_loc on mds1 fail_loc=0 oleg229-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg229-server: insmod: ERROR: could not insert module /home/green/git/lustre-release/lustre/lod/lod.ko: File exists pdsh@oleg229-client: oleg229-server: ssh exited with exit code 1 oleg229-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg229-server: oleg229-server.virtnet: executing set_default_debug -1 all pdsh@oleg229-client: oleg229-server: ssh exited with exit code 1 Started lustre-MDT0000 PASS 126 (106s) == conf-sanity test 127: direct io overwrite on full ost ========================================================== 13:12:55 (1789405975) umount lustre on /mnt/lustre..... stop ost1 service on oleg229-server stop mds service on oleg229-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg229-server stop mds service on oleg229-server unloading modules via unload_modules_local on: 'oleg229-server' oleg229-server: oleg229-server.virtnet: executing unload_modules_local oleg229-server: kmemleak disabled modules unloaded. start mds service on oleg229-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1931560 Sep 14 05:02 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg229-server' oleg229-server: oleg229-server.virtnet: executing load_modules_local oleg229-server: kmemleak disabled oleg229-server: Loading modules from /home/green/git/lustre-release/lustre oleg229-server: detected 4 online CPUs by sysfs oleg229-server: MODOPTS_LIBCFS= oleg229-server: Force libcfs to create 2 CPU partitions oleg229-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg229-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg229-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg229-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg229-server: oleg229-server.virtnet: executing set_default_debug -1 all pdsh@oleg229-client: oleg229-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg229-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg229-server: oleg229-server.virtnet: executing set_default_debug -1 all pdsh@oleg229-client: oleg229-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg229-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg229-server: oleg229-server.virtnet: executing set_default_debug -1 all pdsh@oleg229-client: oleg229-server: ssh exited with exit code 1 Started lustre-OST0000 oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg229-client.virtnet: -o user_xattr,flock 192.168.202.129@tcp:/lustre /mnt/lustre Stopping clients: /mnt/lustre (opts:) pdsh@oleg229-client: no remote hosts specified osc.lustre-OST0000-osc-MDT0000.active updated after 0 sec (got 1) osc.lustre-OST0000-osc-MDT0001.active updated after 0 sec (got 1) Waiting for MDT destroys to complete oleg229-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg229-client: oleg229-server: ssh exited with exit code 1 dd: error writing '/mnt/lustre/f127.conf-sanity': No space left on device 124+0 records in 123+0 records out 128974848 bytes (129 MB, 123 MiB) copied, 8.22135 s, 15.7 MB/s 123+0 records in 123+0 records out 128974848 bytes (129 MB, 123 MiB) copied, 8.06978 s, 16.0 MB/s PASS 127 (181s) == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 13:15:56 (1789406156) SKIP: conf-sanity test_128 need separate mgs device SKIP 128 (4s) == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 13:16:00 (1789406160) Stopping clients: oleg229-client.virtnet /mnt/lustre (opts:) Stopping client oleg229-client.virtnet /mnt/lustre opts: Stopping clients: oleg229-client.virtnet /mnt/lustre2 (opts:) Stopping /mnt/lustre-mds1 (opts:-f) on oleg229-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg229-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg229-server start mds service on oleg229-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg229-server: oleg229-server.virtnet: executing set_default_debug -1 all pdsh@oleg229-client: oleg229-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg229-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg229-server: oleg229-server.virtnet: executing set_default_debug -1 all pdsh@oleg229-client: oleg229-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 Format ost1: /dev/mapper/ost1_flakey Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg229-server: mount.lustre: mount /dev/mapper/ost1_flakey at /mnt/lustre-ost1 failed: Address already in use oleg229-server: The target service's index is already in use. (/dev/mapper/ost1_flakey) pdsh@oleg229-client: oleg229-server: ssh exited with exit code 98 oleg229-server: error: set_param: param_path 'seq/cli-lustre:OST0000-super/width': No such file or directory oleg229-server: error: set_param: setting 'seq/cli-lustre:OST0000-super/width'='65536': No such file or directory pdsh@oleg229-client: oleg229-server: ssh exited with exit code 2 Start of /dev/mapper/ost1_flakey on ost1 failed 98 Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg229-server: mount.lustre: mount /dev/mapper/ost1_flakey at /mnt/lustre-ost1 failed: Address already in use oleg229-server: The target service's index is already in use. (/dev/mapper/ost1_flakey) pdsh@oleg229-client: oleg229-server: ssh exited with exit code 98 oleg229-server: error: set_param: param_path 'seq/cli-lustre:OST0000-super/width': No such file or directory oleg229-server: error: set_param: setting 'seq/cli-lustre:OST0000-super/width'='65536': No such file or directory pdsh@oleg229-client: oleg229-server: ssh exited with exit code 2 Start of /dev/mapper/ost1_flakey on ost1 failed 98 checking for existing Lustre data: found Read previous values: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x22 (OST first_time ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.202.129@tcp sys.timeout=20 Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre=OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x122 (OST first_time writeconf ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.202.129@tcp sys.timeout=20 Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg229-server: oleg229-server.virtnet: executing set_default_debug -1 all pdsh@oleg229-client: oleg229-server: ssh exited with exit code 1 Started lustre-OST0000 Stopping /mnt/lustre-ost1 (opts:) on oleg229-server stop mds service on oleg229-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg229-server stop mds service on oleg229-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg229-server PASS 129 (144s) == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 13:18:24 (1789406304) Checking servers environments Checking clients oleg229-client.virtnet environments Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg229-server' oleg229-server: oleg229-server.virtnet: executing load_modules_local oleg229-server: kmemleak disabled oleg229-server: Loading modules from /home/green/git/lustre-release/lustre oleg229-server: detected 4 online CPUs by sysfs oleg229-server: MODOPTS_LIBCFS= oleg229-server: Force libcfs to create 2 CPU partitions oleg229-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Setup mgs, mdt, osts Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg229-server: oleg229-server.virtnet: executing set_default_debug -1 all pdsh@oleg229-client: oleg229-server: ssh exited with exit code 1 Started lustre-MDT0000 Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg229-server: oleg229-server.virtnet: executing set_default_debug -1 all pdsh@oleg229-client: oleg229-server: ssh exited with exit code 1 Started lustre-MDT0001 Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg229-server: oleg229-server.virtnet: executing set_default_debug -1 all pdsh@oleg229-client: oleg229-server: ssh exited with exit code 1 Started lustre-OST0000 Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg229-server: oleg229-server.virtnet: executing set_default_debug -1 all pdsh@oleg229-client: oleg229-server: ssh exited with exit code 1 Started lustre-OST0001 mount lustre on /mnt/lustre..... Starting client: oleg229-client.virtnet: -o user_xattr,flock 192.168.202.129@tcp:/lustre /mnt/lustre Starting client oleg229-client.virtnet: -o user_xattr,flock 192.168.202.129@tcp:/lustre /mnt/lustre Started clients oleg229-client.virtnet: 192.168.202.129@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Using TIMEOUT=20 osc.lustre-OST0000-osc-ffffa09a8a096800.idle_timeout=debug osc.lustre-OST0001-osc-ffffa09a8a096800.idle_timeout=debug setting jobstats to procname_uid Setting lustre.sys.jobid_var from disable to procname_uid disable quota as required stop mds service on oleg229-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg229-server checking for existing Lustre data: found Read previous values: Device: /dev/mapper/mds2_flakey Target: lustre-MDT0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x1 (MDT ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: mgsnode=192.168.202.129@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity Permanent disk data: Device: /dev/mapper/mds2_flakey Target: lustre=MDT0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x101 (MDT writeconf ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: mgsnode=192.168.202.129@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity start mds service on oleg229-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg229-server: oleg229-server.virtnet: executing set_default_debug -1 all pdsh@oleg229-client: oleg229-server: ssh exited with exit code 1 Started lustre-MDT0001 PASS 130 (136s) == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 13:20:40 (1789406440) oleg229-server: debugfs 1.47.3-wc2 (11-Nov-2025) Checking servers environments Checking clients oleg229-client.virtnet environments Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg229-server' oleg229-server: oleg229-server.virtnet: executing load_modules_local oleg229-server: kmemleak disabled oleg229-server: Loading modules from /home/green/git/lustre-release/lustre oleg229-server: detected 4 online CPUs by sysfs oleg229-server: MODOPTS_LIBCFS= oleg229-server: Force libcfs to create 2 CPU partitions oleg229-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Setup mgs, mdt, osts Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg229-server: mount.lustre: according to /etc/mtab /dev/mapper/mds1_flakey is already mounted on /mnt/lustre-mds1 pdsh@oleg229-client: oleg229-server: ssh exited with exit code 17 Start of /dev/mapper/mds1_flakey on mds1 failed 17 Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg229-server: mount.lustre: according to /etc/mtab /dev/mapper/mds2_flakey is already mounted on /mnt/lustre-mds2 pdsh@oleg229-client: oleg229-server: ssh exited with exit code 17 Start of /dev/mapper/mds2_flakey on mds2 failed 17 Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg229-server: mount.lustre: according to /etc/mtab /dev/mapper/ost1_flakey is already mounted on /mnt/lustre-ost1 pdsh@oleg229-client: oleg229-server: ssh exited with exit code 17 seq.cli-lustre-OST0000-super.width=65536 Start of /dev/mapper/ost1_flakey on ost1 failed 17 Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 oleg229-server: mount.lustre: according to /etc/mtab /dev/mapper/ost2_flakey is already mounted on /mnt/lustre-ost2 pdsh@oleg229-client: oleg229-server: ssh exited with exit code 17 seq.cli-lustre-OST0001-super.width=65536 Start of /dev/mapper/ost2_flakey on ost2 failed 17 mount lustre on /mnt/lustre..... Starting client: oleg229-client.virtnet: -o user_xattr,flock 192.168.202.129@tcp:/lustre /mnt/lustre mount.lustre: according to /etc/mtab 192.168.202.129@tcp:/lustre is already mounted on /mnt/lustre Starting client oleg229-client.virtnet: -o user_xattr,flock 192.168.202.129@tcp:/lustre /mnt/lustre Started clients oleg229-client.virtnet: 192.168.202.129@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Using TIMEOUT=20 osc.lustre-OST0000-osc-ffffa09a8a096800.idle_timeout=debug osc.lustre-OST0001-osc-ffffa09a8a096800.idle_timeout=debug disable quota as required striped dir -i1 -c2 -H crush2 /mnt/lustre/d131.conf-sanity total: 512 open/close in 6.49 seconds: 78.90 ops/second striped dir -i1 -c2 -H all_char /mnt/lustre/d131.conf-sanity.inherit total: 128 open/close in 1.55 seconds: 82.58 ops/second Stopping clients: oleg229-client.virtnet /mnt/lustre (opts:) Stopping client oleg229-client.virtnet /mnt/lustre opts: Stopping clients: oleg229-client.virtnet /mnt/lustre2 (opts:) Stopping /mnt/lustre-mds1 (opts:-f) on oleg229-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg229-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg229-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg229-server file-level backup/restore on mds1:/dev/mapper/mds1_flakey backup data reformat new device Format mds1: /dev/mapper/mds1_flakey restore data remove recovery logs removed '/mnt/lustre-brpt/CATALOGS' file-level backup/restore on mds2:/dev/mapper/mds2_flakey backup data reformat new device Format mds2: /dev/mapper/mds2_flakey restore data remove recovery logs removed '/mnt/lustre-brpt/CATALOGS' Checking servers environments Checking clients oleg229-client.virtnet environments Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg229-server' oleg229-server: oleg229-server.virtnet: executing load_modules_local oleg229-server: kmemleak disabled oleg229-server: Loading modules from /home/green/git/lustre-release/lustre oleg229-server: detected 4 online CPUs by sysfs oleg229-server: MODOPTS_LIBCFS= oleg229-server: Force libcfs to create 2 CPU partitions oleg229-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Setup mgs, mdt, osts Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg229-server: oleg229-server.virtnet: executing set_default_debug -1 all pdsh@oleg229-client: oleg229-server: ssh exited with exit code 1 Started lustre-MDT0000 Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg229-server: oleg229-server.virtnet: executing set_default_debug -1 all pdsh@oleg229-client: oleg229-server: ssh exited with exit code 1 Started lustre-MDT0001 Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg229-server: oleg229-server.virtnet: executing set_default_debug -1 all pdsh@oleg229-client: oleg229-server: ssh exited with exit code 1 Started lustre-OST0000 Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg229-server: oleg229-server.virtnet: executing set_default_debug -1 all pdsh@oleg229-client: oleg229-server: ssh exited with exit code 1 Started lustre-OST0001 mount lustre on /mnt/lustre..... Starting client: oleg229-client.virtnet: -o user_xattr,flock 192.168.202.129@tcp:/lustre /mnt/lustre Starting client oleg229-client.virtnet: -o user_xattr,flock 192.168.202.129@tcp:/lustre /mnt/lustre Started clients oleg229-client.virtnet: 192.168.202.129@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Using TIMEOUT=20 osc.lustre-OST0000-osc-ffffa09a8b2c7000.idle_timeout=debug osc.lustre-OST0001-osc-ffffa09a8b2c7000.idle_timeout=debug disable quota as required total: 2 open/close in 0.10 seconds: 19.37 ops/second PASS 131 (383s) == conf-sanity test 132: hsm_actions processed after failover ========================================================== 13:27:03 (1789406823) Stopping clients: oleg229-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg229-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg229-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg229-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg229-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg229-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg229-server oleg229-server: oleg229-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg229-server' oleg229-server: oleg229-server.virtnet: executing load_modules_local oleg229-server: kmemleak disabled oleg229-server: Loading modules from /home/green/git/lustre-release/lustre oleg229-server: detected 4 online CPUs by sysfs oleg229-server: MODOPTS_LIBCFS= oleg229-server: Force libcfs to create 2 CPU partitions oleg229-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg229-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg229-server: oleg229-server.virtnet: executing set_default_debug -1 all pdsh@oleg229-client: oleg229-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 stop mds service on oleg229-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg229-server checking for existing Lustre data: found Read previous values: Device: /dev/mapper/mds1_flakey Target: lustre-MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x5 (MDT MGS ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity Permanent disk data: Device: /dev/mapper/mds1_flakey Target: lustre-MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x45 (MDT MGS update ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity mdt.hsm_control=enabled start mds service on oleg229-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg229-server: oleg229-server.virtnet: executing set_default_debug -1 all pdsh@oleg229-client: oleg229-server: ssh exited with exit code 1 Started lustre-MDT0000 stop mds service on oleg229-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg229-server PASS 132 (169s) == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 13:29:52 (1789406992) SKIP: conf-sanity test_133 needs >= 4 OSTs SKIP 133 (4s) == conf-sanity test 134: check_iam works without faults == 13:29:56 (1789406996) Checking servers environments Checking clients oleg229-client.virtnet environments Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg229-server' oleg229-server: oleg229-server.virtnet: executing load_modules_local oleg229-server: kmemleak disabled oleg229-server: Loading modules from /home/green/git/lustre-release/lustre oleg229-server: detected 4 online CPUs by sysfs oleg229-server: MODOPTS_LIBCFS= oleg229-server: Force libcfs to create 2 CPU partitions oleg229-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Setup mgs, mdt, osts Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg229-server: oleg229-server.virtnet: executing set_default_debug -1 all pdsh@oleg229-client: oleg229-server: ssh exited with exit code 1 Started lustre-MDT0000 Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg229-server: oleg229-server.virtnet: executing set_default_debug -1 all pdsh@oleg229-client: oleg229-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg229-server: oleg229-server.virtnet: executing set_default_debug -1 all pdsh@oleg229-client: oleg229-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg229-server: oleg229-server.virtnet: executing set_default_debug -1 all pdsh@oleg229-client: oleg229-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost2_flakey Started lustre-OST0001 mount lustre on /mnt/lustre..... Starting client: oleg229-client.virtnet: -o user_xattr,flock 192.168.202.129@tcp:/lustre /mnt/lustre Starting client oleg229-client.virtnet: -o user_xattr,flock 192.168.202.129@tcp:/lustre /mnt/lustre Started clients oleg229-client.virtnet: 192.168.202.129@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Using TIMEOUT=20 osc.lustre-OST0000-osc-ffffa09a8a097000.idle_timeout=debug osc.lustre-OST0001-osc-ffffa09a8a097000.idle_timeout=debug disable quota as required striped dir -i0 -c2 -H crush2 /mnt/lustre/d134.conf-sanity Started LFSCK on the device lustre-MDT0000: scrub layout namespace ptrsize: 4 indirect_levels: 0 idle_blocks: 0 Root entries: count 2, limit 203 000: [0:0x0:0x0] ptr: 1 Block 1,FIX leaf, Leaf block, count 1, limit 240 count 1 NO ERRORS 0 debugfs 1.47.3-wc2 (11-Nov-2025) Record type to print: lfsck_namespace Filesize 8192, blocks count 2 Root format: LFIX keysize: 16 recsize: 1 ptrsize: 4 indirect_levels: 0 idle_blocks: 0 Root entries: count 2, limit 203 000: [0:0x0:0x0] ptr: 1 Block 1,FIX leaf, Leaf block, count 1, limit 240 000: [0:0x0:0x0] 0x0 () count 1 NO ERRORS dd if=/dev/urandom of=/tmp/d134.conf-sanity/lfsck_namespace_14 bs=2 conv=notrunc count=1 seek=31 Filesize 8192, blocks count 2 Root format: LFIX keysize: 16 recsize: 1 ptrsize: 4 indirect_levels: 0 idle_blocks: 0 Root entries: count 2, limit 203 000: [0:0x0:0x0] ptr: 1 Block 1,FIX leaf, Leaf block, count 1, limit 240 count 1 NO ERRORS 0 debugfs 1.47.3-wc2 (11-Nov-2025) Record type to print: lfsck_namespace Filesize 8192, blocks count 2 Root format: LFIX keysize: 16 recsize: 1 ptrsize: 4 indirect_levels: 0 idle_blocks: 0 Root entries: count 2, limit 203 000: [0:0x0:0x0] ptr: 1 Block 1,FIX leaf, Leaf block, count 1, limit 240 000: [0:0x0:0x0] 0x0 () count 1 NO ERRORS dd if=/dev/urandom of=/tmp/d134.conf-sanity/lfsck_namespace_15 bs=2 conv=notrunc count=1 seek=12 Filesize 8192, blocks count 2 Root format: LFIX keysize: 16 recsize: 1 ptrsize: 4 indirect_levels: 0 idle_blocks: 0 Root entries: count 2, limit 203 000: [0:0x0:0x0] ptr: 1 Block 1,FIX leaf, Leaf block, count 1, limit 240 count 1 NO ERRORS 0 debugfs 1.47.3-wc2 (11-Nov-2025) Record type to print: nodemap Filesize 8192, blocks count 2 Root format: LFIX keysize: 8 recsize: 32 ptrsize: 4 indirect_levels: 0 idle_blocks: 0 Root entries: count 2, limit 340 000: { id: 0x0, type: empty(0) } ptr: 1 Block 1,FIX leaf, Leaf block, count 5, limit 102 000: { id: 0x0, type: empty(0) } {} 001: { id: 0x0, type: cluster(1), subtype: cluster } { name: default, flag: 0xb8, flag2: 0x4, squash_uid: 65534, squash_gid: 65534, squash_projid: 65534} 002: { id: 0x0, type: global(15) } { is_active: 0 } 003: { id: 0x1, type: cluster(1), subtype: cluster } { name: test_134, flag: 0xb8, flag2: 0x4, squash_uid: 65534, squash_gid: 65534, squash_projid: 65534} 004: { id: 0x1, type: range(2) } { start_nid: 121.23.2.100@tcp, end_nid: 121.23.2.120@tcp, type 0x0 } count 5 NO ERRORS dd if=/dev/urandom of=/tmp/d134.conf-sanity/nodemap bs=2 conv=notrunc count=1 seek=16 Filesize 8192, blocks count 2 Root format: LFIX keysize: 8 recsize: 32 ptrsize: 4 indirect_levels: 0 idle_blocks: 0 Root entries: count 2, limit 340 000: { id: 0x0, type: empty(0) } ptr: 1 Block 1,FIX leaf, Leaf block, count 5, limit 102 count 5 NO ERRORS 0 PASS 134 (757s) == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 13:42:33 (1789407753) Stopping clients: oleg229-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg229-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg229-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg229-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg229-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg229-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg229-server oleg229-server: oleg229-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg229-server' oleg229-server: oleg229-server.virtnet: executing load_modules_local oleg229-server: kmemleak disabled oleg229-server: Loading modules from /home/green/git/lustre-release/lustre oleg229-server: detected 4 online CPUs by sysfs oleg229-server: MODOPTS_LIBCFS= oleg229-server: Force libcfs to create 2 CPU partitions oleg229-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg229-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg229-server: oleg229-server.virtnet: executing set_default_debug -1 all pdsh@oleg229-client: oleg229-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg229-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg229-server: oleg229-server.virtnet: executing set_default_debug -1 all pdsh@oleg229-client: oleg229-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg229-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg229-server: oleg229-server.virtnet: executing set_default_debug -1 all pdsh@oleg229-client: oleg229-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg229-client.virtnet: -o user_xattr,flock 192.168.202.129@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H fnv_1a_64 /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 92 (time 1789407958.79 total 10.02 last 9.18) - open/close 200 (time 1789407968.84 total 20.07 last 10.74) - open/close 304 (time 1789407978.87 total 30.10 last 10.37) - open/close 394 (time 1789407988.89 total 40.13 last 8.98) - open/close 492 (time 1789407998.90 total 50.14 last 9.79) - open/close 597 (time 1789408008.94 total 60.17 last 10.46) - open/close 712 (time 1789408018.95 total 70.18 last 11.49) - open/close 818 (time 1789408028.99 total 80.23 last 10.55) - open/close 928 (time 1789408039.10 total 90.34 last 10.88) - open/close 1043 (time 1789408049.18 total 100.41 last 11.41) - open/close 1151 (time 1789408059.27 total 110.50 last 10.71) - open/close 1262 (time 1789408069.31 total 120.55 last 11.05) - open/close 1373 (time 1789408079.38 total 130.61 last 11.03) - open/close 1482 (time 1789408089.44 total 140.67 last 10.83) - open/close 1553 (time 1789408099.47 total 150.70 last 7.08) - open/close 1642 (time 1789408109.48 total 160.71 last 8.89) - open/close 1737 (time 1789408119.56 total 170.80 last 9.42) - open/close 1857 (time 1789408129.59 total 180.82 last 11.97) - open/close 1952 (time 1789408139.64 total 190.87 last 9.45) - open/close 2053 (time 1789408149.70 total 200.93 last 10.04) - open/close 2147 (time 1789408159.73 total 210.97 last 9.37) - open/close 2236 (time 1789408169.81 total 221.05 last 8.83) - open/close 2349 (time 1789408179.84 total 231.07 last 11.27) - open/close 2477 (time 1789408189.86 total 241.09 last 12.78) - open/close 2586 (time 1789408199.92 total 251.16 last 10.83) - open/close 2663 (time 1789408210.11 total 261.34 last 7.56) - open/close 2745 (time 1789408220.14 total 271.37 last 8.18) - open/close 2843 (time 1789408230.16 total 281.40 last 9.77) - open/close 2962 (time 1789408240.17 total 291.40 last 11.90) - open/close 3043 (time 1789408250.19 total 301.42 last 8.08) - open/close 3135 (time 1789408260.33 total 311.57 last 9.07) - open/close 3248 (time 1789408270.39 total 321.62 last 11.24) - open/close 3361 (time 1789408280.45 total 331.68 last 11.23) - open/close 3460 (time 1789408290.50 total 341.74 last 9.85) - open/close 3583 (time 1789408300.53 total 351.76 last 12.27) - open/close 3712 (time 1789408310.55 total 361.79 last 12.87) - open/close 3842 (time 1789408320.56 total 371.79 last 12.99) - open/close 3980 (time 1789408330.61 total 381.84 last 13.73) - open/close 4112 (time 1789408340.61 total 391.84 last 13.20) - open/close 4250 (time 1789408350.64 total 401.88 last 13.75) - open/close 4366 (time 1789408360.65 total 411.88 last 11.59) - open/close 4475 (time 1789408370.70 total 421.93 last 10.84) total: 4500 open/close in 423.73 seconds: 10.62 ops/second - /unlink 294 (time 1789408388.96 total 10.01 last 29.36) - /unlink 578 (time 1789408398.97 total 20.01 last 28.40) - /unlink 875 (time 1789408408.99 total 30.04 last 29.62) - /unlink 1180 (time 1789408419.02 total 40.07 last 30.41) - /unlink 1476 (time 1789408429.05 total 50.10 last 29.51) - /unlink 1778 (time 1789408439.07 total 60.12 last 30.14) - /unlink 2076 (time 1789408449.07 total 70.12 last 29.80) - /unlink 2299 (time 1789408459.12 total 80.17 last 22.19) - /unlink 2582 (time 1789408469.15 total 90.20 last 28.22) - /unlink 2865 (time 1789408479.18 total 100.22 last 28.22) - /unlink 3166 (time 1789408489.20 total 110.24 last 30.04) - /unlink 3538 (time 1789408499.21 total 120.26 last 37.14) - /unlink 3796 (time 1789408509.26 total 130.30 last 25.69) - /unlink 4091 (time 1789408519.27 total 140.32 last 29.45) - /unlink 4382 (time 1789408529.28 total 150.33 last 29.07) total: 4500 /unlink in 153.83 seconds: 29.25 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 75 (time 1789408546.80 total 10.01 last 7.49) - open/close 165 (time 1789408556.82 total 20.03 last 8.98) - open/close 277 (time 1789408566.86 total 30.06 last 11.16) - open/close 405 (time 1789408576.90 total 40.10 last 12.75) - open/close 525 (time 1789408586.93 total 50.14 last 11.96) - open/close 642 (time 1789408596.98 total 60.19 last 11.64) - open/close 737 (time 1789408607.02 total 70.23 last 9.46) - open/close 844 (time 1789408617.05 total 80.26 last 10.67) - open/close 965 (time 1789408627.12 total 90.33 last 12.01) - open/close 1051 (time 1789408637.14 total 100.35 last 8.58) - open/close 1138 (time 1789408647.16 total 110.37 last 8.68) - open/close 1244 (time 1789408657.26 total 120.47 last 10.50) - open/close 1356 (time 1789408667.27 total 130.48 last 11.19) - open/close 1471 (time 1789408677.27 total 140.48 last 11.50) - open/close 1560 (time 1789408687.27 total 150.48 last 8.90) - open/close 1676 (time 1789408697.37 total 160.58 last 11.49) - open/close 1786 (time 1789408707.43 total 170.64 last 10.93) - open/close 1901 (time 1789408717.47 total 180.68 last 11.46) - open/close 1998 (time 1789408727.51 total 190.72 last 9.67) - open/close 2077 (time 1789408737.60 total 200.81 last 7.83) - open/close 2147 (time 1789408747.64 total 210.85 last 6.97) - open/close 2250 (time 1789408757.64 total 220.85 last 10.30) - open/close 2344 (time 1789408767.65 total 230.86 last 9.39) - open/close 2448 (time 1789408777.70 total 240.91 last 10.35) - open/close 2563 (time 1789408787.73 total 250.94 last 11.47) - open/close 2688 (time 1789408797.73 total 260.94 last 12.50) - open/close 2788 (time 1789408807.77 total 270.98 last 9.96) - open/close 2873 (time 1789408817.83 total 281.03 last 8.45) - open/close 2947 (time 1789408827.88 total 291.09 last 7.36) - open/close 3033 (time 1789408837.96 total 301.16 last 8.54) - open/close 3133 (time 1789408848.06 total 311.27 last 9.90) - open/close 3239 (time 1789408858.14 total 321.34 last 10.52) - open/close 3340 (time 1789408868.14 total 331.35 last 10.09) - open/close 3434 (time 1789408878.15 total 341.36 last 9.39) - open/close 3529 (time 1789408888.24 total 351.45 last 9.42) - open/close 3627 (time 1789408898.36 total 361.57 last 9.68) - open/close 3718 (time 1789408908.42 total 371.63 last 9.05) - open/close 3816 (time 1789408918.54 total 381.75 last 9.68) - open/close 3917 (time 1789408928.56 total 391.77 last 10.09) - open/close 4025 (time 1789408938.61 total 401.82 last 10.74) - open/close 4105 (time 1789408948.63 total 411.84 last 7.99) - open/close 4207 (time 1789408958.70 total 421.91 last 10.13) - open/close 4296 (time 1789408968.82 total 432.03 last 8.79) - open/close 4399 (time 1789408978.91 total 442.12 last 10.21) - open/close 4499 (time 1789408988.96 total 452.16 last 9.96) total: 4500 open/close in 452.16 seconds: 9.95 ops/second - /unlink 229 (time 1789409003.87 total 10.06 last 22.75) - /unlink 498 (time 1789409013.90 total 20.09 last 26.84) - /unlink 768 (time 1789409023.92 total 30.11 last 26.94) - /unlink 1061 (time 1789409033.94 total 40.13 last 29.23) - /unlink 1345 (time 1789409043.95 total 50.14 last 28.36) - /unlink 1656 (time 1789409053.98 total 60.17 last 31.00) - /unlink 1890 (time 1789409064.01 total 70.20 last 23.33) - /unlink 2166 (time 1789409074.02 total 80.21 last 27.59) - /unlink 2434 (time 1789409084.06 total 90.25 last 26.68) - /unlink 2680 (time 1789409094.09 total 100.28 last 24.54) - /unlink 2977 (time 1789409104.09 total 110.28 last 29.70) - /unlink 3271 (time 1789409114.12 total 120.31 last 29.30) - /unlink 3572 (time 1789409124.13 total 130.32 last 30.07) - /unlink 3859 (time 1789409134.14 total 140.33 last 28.68) - /unlink 4150 (time 1789409144.15 total 150.34 last 29.07) - /unlink 4434 (time 1789409154.16 total 160.35 last 28.38) total: 4500 /unlink in 162.92 seconds: 27.62 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 85 (time 1789409172.98 total 10.08 last 8.43) - open/close 197 (time 1789409183.08 total 20.18 last 11.09) - open/close 304 (time 1789409193.13 total 30.23 last 10.65) - open/close 423 (time 1789409203.14 total 40.24 last 11.89) - open/close 517 (time 1789409213.17 total 50.27 last 9.37) - open/close 622 (time 1789409223.17 total 60.27 last 10.50) - open/close 722 (time 1789409233.29 total 70.40 last 9.87) - open/close 794 (time 1789409243.46 total 80.57 last 7.08) - open/close 886 (time 1789409253.51 total 90.61 last 9.16) - open/close 996 (time 1789409263.62 total 100.73 last 10.88) - open/close 1114 (time 1789409273.64 total 110.75 last 11.78) - open/close 1204 (time 1789409283.72 total 120.82 last 8.93) - open/close 1298 (time 1789409293.89 total 130.99 last 9.25) - open/close 1372 (time 1789409304.04 total 141.15 last 7.29) - open/close 1450 (time 1789409314.11 total 151.22 last 7.74) - open/close 1557 (time 1789409324.19 total 161.29 last 10.62) - open/close 1676 (time 1789409334.25 total 171.35 last 11.83) - open/close 1787 (time 1789409344.31 total 181.42 last 11.03) - open/close 1872 (time 1789409354.33 total 191.43 last 8.49) - open/close 1950 (time 1789409364.47 total 201.57 last 7.69) - open/close 2042 (time 1789409374.48 total 211.59 last 9.18) - open/close 2135 (time 1789409384.52 total 221.62 last 9.27) - open/close 2244 (time 1789409394.61 total 231.71 last 10.80) - open/close 2341 (time 1789409404.64 total 241.75 last 9.66) - open/close 2456 (time 1789409414.73 total 251.83 last 11.41) - open/close 2576 (time 1789409424.77 total 261.87 last 11.95) - open/close 2674 (time 1789409434.82 total 271.93 last 9.74) - open/close 2788 (time 1789409444.88 total 281.99 last 11.33) - open/close 2912 (time 1789409454.91 total 292.02 last 12.36) - open/close 3021 (time 1789409464.97 total 302.07 last 10.84) - open/close 3139 (time 1789409474.98 total 312.09 last 11.78) - open/close 3252 (time 1789409485.04 total 322.14 last 11.24) - open/close 3340 (time 1789409495.06 total 332.16 last 8.78) - open/close 3432 (time 1789409505.18 total 342.29 last 9.09) - open/close 3531 (time 1789409515.19 total 352.30 last 9.89) - open/close 3631 (time 1789409525.24 total 362.35 last 9.95) - open/close 3751 (time 1789409535.28 total 372.38 last 11.96) - open/close 3864 (time 1789409545.29 total 382.39 last 11.29) - open/close 3965 (time 1789409555.31 total 392.42 last 10.07) - open/close 4072 (time 1789409565.32 total 402.43 last 10.69) - open/close 4188 (time 1789409575.36 total 412.46 last 11.56) - open/close 4331 (time 1789409585.43 total 422.53 last 14.20) - open/close 4444 (time 1789409595.53 total 432.63 last 11.18) total: 4500 open/close in 437.98 seconds: 10.27 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 351 (time 1789409614.62 total 10.01 last 35.07) - /unlink 725 (time 1789409624.63 total 20.02 last 37.37) - /unlink 1027 (time 1789409634.64 total 30.03 last 30.16) - /unlink 1320 (time 1789409644.65 total 40.04 last 29.27) - /unlink 1495 (time 1789409654.77 total 50.16 last 17.30) - /unlink 1695 (time 1789409664.80 total 60.19 last 19.94) - /unlink 1950 (time 1789409674.81 total 70.20 last 25.46) - /unlink 2250 (time 1789409684.83 total 80.22 last 29.94) - /unlink 2535 (time 1789409694.86 total 90.25 last 28.41) - /unlink 2690 (time 1789409704.89 total 100.27 last 15.46) - /unlink 2890 (time 1789409714.91 total 110.30 last 19.96) - /unlink 3173 (time 1789409724.93 total 120.32 last 28.23) - /unlink 3384 (time 1789409734.95 total 130.34 last 21.06) - /unlink 3630 (time 1789409744.96 total 140.35 last 24.58) - /unlink 3880 (time 1789409755.01 total 150.40 last 24.88) - /unlink 4122 (time 1789409765.02 total 160.41 last 24.16) - /unlink 4367 (time 1789409775.03 total 170.42 last 24.48) total: 4500 /unlink in 175.47 seconds: 25.64 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 108 (time 1789409795.51 total 10.04 last 10.75) - open/close 218 (time 1789409805.52 total 20.06 last 10.98) - open/close 325 (time 1789409815.55 total 30.09 last 10.67) - open/close 418 (time 1789409825.65 total 40.19 last 9.21) - open/close 529 (time 1789409835.68 total 50.22 last 11.07) - open/close 637 (time 1789409845.75 total 60.28 last 10.73) - open/close 716 (time 1789409855.76 total 70.30 last 7.89) - open/close 823 (time 1789409865.78 total 80.31 last 10.68) - open/close 939 (time 1789409875.87 total 90.41 last 11.49) - open/close 1030 (time 1789409885.93 total 100.47 last 9.04) - open/close 1117 (time 1789409895.94 total 110.48 last 8.69) - open/close 1204 (time 1789409906.07 total 120.60 last 8.59) - open/close 1309 (time 1789409916.11 total 130.65 last 10.45) - open/close 1410 (time 1789409926.16 total 140.69 last 10.06) - open/close 1502 (time 1789409936.18 total 150.71 last 9.18) - open/close 1603 (time 1789409946.27 total 160.81 last 10.01) - open/close 1692 (time 1789409956.38 total 170.92 last 8.80) - open/close 1791 (time 1789409966.43 total 180.97 last 9.85) - open/close 1912 (time 1789409976.54 total 191.07 last 11.98) - open/close 1995 (time 1789409986.55 total 201.08 last 8.29) - open/close 2091 (time 1789409996.58 total 211.12 last 9.57) - open/close 2197 (time 1789410006.64 total 221.18 last 10.53) - open/close 2302 (time 1789410016.69 total 231.22 last 10.45) - open/close 2393 (time 1789410026.72 total 241.26 last 9.07) - open/close 2493 (time 1789410036.75 total 251.29 last 9.97) - open/close 2593 (time 1789410046.86 total 261.39 last 9.89) - open/close 2702 (time 1789410056.91 total 271.45 last 10.84) - open/close 2807 (time 1789410066.99 total 281.53 last 10.42) - open/close 2906 (time 1789410077.08 total 291.61 last 9.81) - open/close 2993 (time 1789410087.21 total 301.75 last 8.59) - open/close 3057 (time 1789410097.38 total 311.92 last 6.29) - open/close 3125 (time 1789410107.39 total 321.93 last 6.79) - open/close 3215 (time 1789410117.45 total 331.99 last 8.95) - open/close 3333 (time 1789410127.49 total 342.03 last 11.75) - open/close 3428 (time 1789410137.51 total 352.05 last 9.48) - open/close 3524 (time 1789410147.57 total 362.11 last 9.55) - open/close 3592 (time 1789410157.60 total 372.13 last 6.78) - open/close 3696 (time 1789410167.78 total 382.32 last 10.21) - open/close 3798 (time 1789410177.87 total 392.41 last 10.11) - open/close 3889 (time 1789410187.89 total 402.43 last 9.08) - open/close 3980 (time 1789410197.99 total 412.52 last 9.01) - open/close 4049 (time 1789410207.99 total 422.53 last 6.90) - open/close 4154 (time 1789410218.01 total 432.54 last 10.48) - open/close 4259 (time 1789410228.02 total 442.56 last 10.48) - open/close 4346 (time 1789410238.03 total 452.56 last 8.70) - open/close 4454 (time 1789410248.09 total 462.63 last 10.73) total: 4500 open/close in 467.78 seconds: 9.62 ops/second - /unlink 283 (time 1789410268.02 total 10.01 last 28.27) - /unlink 535 (time 1789410278.04 total 20.03 last 25.15) - /unlink 781 (time 1789410288.06 total 30.05 last 24.56) - /unlink 1040 (time 1789410298.09 total 40.08 last 25.81) - /unlink 1319 (time 1789410308.10 total 50.08 last 27.89) - /unlink 1566 (time 1789410318.10 total 60.09 last 24.69) - /unlink 1824 (time 1789410328.12 total 70.11 last 25.74) - /unlink 2097 (time 1789410338.13 total 80.12 last 27.27) - /unlink 2371 (time 1789410348.15 total 90.13 last 27.37) - /unlink 2640 (time 1789410358.19 total 100.18 last 26.77) - /unlink 2877 (time 1789410368.20 total 110.19 last 23.68) - /unlink 3103 (time 1789410378.24 total 120.23 last 22.51) - /unlink 3345 (time 1789410388.25 total 130.24 last 24.18) - /unlink 3590 (time 1789410398.26 total 140.25 last 24.47) - /unlink 3834 (time 1789410408.28 total 150.27 last 24.35) - /unlink 4098 (time 1789410418.30 total 160.29 last 26.36) - /unlink 4370 (time 1789410428.30 total 170.29 last 27.20) total: 4500 /unlink in 175.36 seconds: 25.66 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 84 (time 1789410448.61 total 10.03 last 8.37) - open/close 151 (time 1789410458.67 total 20.09 last 6.66) - open/close 246 (time 1789410468.72 total 30.14 last 9.45) - open/close 344 (time 1789410478.76 total 40.17 last 9.77) - open/close 435 (time 1789410488.78 total 50.20 last 9.08) - open/close 541 (time 1789410498.80 total 60.22 last 10.57) - open/close 663 (time 1789410508.86 total 70.28 last 12.13) - open/close 779 (time 1789410518.95 total 80.37 last 11.50) - open/close 889 (time 1789410528.98 total 90.40 last 10.97) - open/close 1011 (time 1789410539.04 total 100.46 last 12.12) - open/close 1128 (time 1789410549.07 total 110.49 last 11.67) - open/close 1247 (time 1789410559.08 total 120.50 last 11.88) - open/close 1349 (time 1789410569.26 total 130.68 last 10.02) - open/close 1419 (time 1789410579.30 total 140.72 last 6.97) - open/close 1506 (time 1789410589.32 total 150.74 last 8.68) - open/close 1615 (time 1789410599.36 total 160.78 last 10.85) - open/close 1708 (time 1789410609.46 total 170.88 last 9.21) - open/close 1812 (time 1789410619.56 total 180.97 last 10.30) - open/close 1935 (time 1789410629.56 total 190.98 last 12.30) - open/close 2039 (time 1789410639.67 total 201.09 last 10.29) - open/close 2129 (time 1789410649.68 total 211.10 last 8.99) - open/close 2235 (time 1789410659.75 total 221.17 last 10.53) - open/close 2352 (time 1789410669.76 total 231.17 last 11.70) - open/close 2468 (time 1789410679.88 total 241.30 last 11.46) - open/close 2573 (time 1789410689.92 total 251.33 last 10.46) - open/close 2680 (time 1789410700.03 total 261.45 last 10.58) - open/close 2788 (time 1789410710.05 total 271.46 last 10.79) - open/close 2905 (time 1789410720.07 total 281.49 last 11.67) - open/close 3009 (time 1789410730.10 total 291.52 last 10.37) - open/close 3106 (time 1789410740.12 total 301.54 last 9.68) - open/close 3213 (time 1789410750.19 total 311.61 last 10.62) - open/close 3329 (time 1789410760.23 total 321.65 last 11.56) - open/close 3454 (time 1789410770.32 total 331.73 last 12.39) - open/close 3559 (time 1789410780.37 total 341.79 last 10.45) - open/close 3675 (time 1789410790.41 total 351.83 last 11.55) - open/close 3779 (time 1789410800.47 total 361.89 last 10.34) - open/close 3864 (time 1789410810.49 total 371.91 last 8.49) - open/close 3975 (time 1789410820.54 total 381.95 last 11.05) - open/close 4084 (time 1789410830.62 total 392.04 last 10.81) - open/close 4196 (time 1789410840.72 total 402.14 last 11.09) - open/close 4281 (time 1789410850.84 total 412.26 last 8.40) - open/close 4387 (time 1789410860.84 total 422.26 last 10.60) total: 4500 open/close in 432.14 seconds: 10.41 ops/second - /unlink 269 (time 1789410885.14 total 10.01 last 26.88) - /unlink 565 (time 1789410895.17 total 20.03 last 29.53) - /unlink 887 (time 1789410905.18 total 30.05 last 32.15) - /unlink 1218 (time 1789410915.19 total 40.06 last 33.07) - /unlink 1437 (time 1789410925.20 total 50.07 last 21.87) - /unlink 1667 (time 1789410935.22 total 60.09 last 22.95) - /unlink 1905 (time 1789410945.28 total 70.14 last 23.67) - /unlink 2167 (time 1789410955.31 total 80.18 last 26.11) - /unlink 2421 (time 1789410965.35 total 90.21 last 25.32) - /unlink 2710 (time 1789410975.36 total 100.23 last 28.85) - /unlink 2957 (time 1789410985.39 total 110.26 last 24.63) - /unlink 3230 (time 1789410995.43 total 120.30 last 27.19) - /unlink 3510 (time 1789411005.46 total 130.33 last 27.91) - /unlink 3840 (time 1789411015.48 total 140.34 last 32.96) - /unlink 4130 (time 1789411025.50 total 150.36 last 28.93) - /unlink 4442 (time 1789411035.52 total 160.39 last 31.13) total: 4500 /unlink in 161.75 seconds: 27.82 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998