-----============= acceptance-small: conf-sanity ============----- Wed Mar 6 23:11:26 EST 2024 excepting tests: 32newtarball 110 skipping tests SLOW=no: 45 69 106 111 114 Stopping clients: oleg340-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg340-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg340-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg340-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg340-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg340-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg340-server oleg340-server: oleg340-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory loading modules on: 'oleg340-server' oleg340-server: oleg340-server.virtnet: executing load_modules_local oleg340-server: Loading modules from /home/green/git/lustre-release/lustre oleg340-server: detected 4 online CPUs by sysfs oleg340-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format mds2: lustre-mdt2/mdt2 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg340-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg340-server: oleg340-server.virtnet: executing set_default_debug -1 all pdsh@oleg340-client: oleg340-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 start mds service on oleg340-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg340-server: oleg340-server.virtnet: executing set_default_debug -1 all pdsh@oleg340-client: oleg340-server: ssh exited with exit code 1 Commit the device label on lustre-mdt2/mdt2 Started lustre-MDT0001 oleg340-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg340-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg340-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg340-server: oleg340-server.virtnet: executing set_default_debug -1 all pdsh@oleg340-client: oleg340-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg340-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg340-server: oleg340-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg340-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg340-server: oleg340-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg340-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg340-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg340-server stop mds service on oleg340-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg340-server stop mds service on oleg340-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg340-server debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 35b: Continue reconnection retries, if the active server is busy ========================================================== 23:12:45 (1709784765) start mds service on oleg340-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg340-server: oleg340-server.virtnet: executing set_default_debug -1 all pdsh@oleg340-client: oleg340-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg340-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg340-server: oleg340-server.virtnet: executing set_default_debug -1 all pdsh@oleg340-client: oleg340-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg340-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg340-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg340-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg340-server: oleg340-server.virtnet: executing set_default_debug -1 all pdsh@oleg340-client: oleg340-server: ssh exited with exit code 1 Started lustre-OST0000 oleg340-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg340-client.virtnet: -o user_xattr,flock oleg340-server@tcp:/lustre /mnt/lustre debug=ha conf-sanity.sh test_35b 2024-03-0623h13m02s Set up a fake failnode for the MDS at_max=0 at_max=0 Injecting EBUSY on MDS fail_loc=0x80000136 mdc.lustre-MDT0000-mdc-ffff88012a4f5800.stats=clear mdc.lustre-MDT0001-mdc-ffff88012a4f5800.stats=clear Creating a test file and stat it File: '/mnt/lustre/d35b.conf-sanity/f35b.conf-sanity' Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115238826934274 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-03-06 23:13:45.000000000 -0500 Modify: 2024-03-06 23:13:45.000000000 -0500 Change: 2024-03-06 23:13:45.000000000 -0500 Birth: - Stop injecting EBUSY on MDS fail_loc=0 done at_max=600 at_max=600 Debug log: 66 lines, 66 kept, 0 dropped, 0 bad. umount lustre on /mnt/lustre..... Stopping client oleg340-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg340-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg340-server stop mds service on oleg340-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg340-server stop mds service on oleg340-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg340-server unloading modules on: 'oleg340-server' oleg340-server: oleg340-server.virtnet: executing unload_modules_local modules unloaded. oleg340-server: oleg340-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre oleg340-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg340-client: oleg340-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg340-server: oleg340-server: tunefs.lustre FATAL: Device lustre-mdt2/mdt2 has not been formatted with mkfs.lustre oleg340-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg340-client: oleg340-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg340-server: oleg340-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre oleg340-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg340-client: oleg340-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg340-server: oleg340-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre oleg340-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg340-client: oleg340-server: ssh exited with exit code 19 checking for existing Lustre data: not found tunefs failed, reformatting instead Stopping clients: oleg340-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg340-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg340-client: oleg340-server: ssh exited with exit code 2 oleg340-server: oleg340-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg340-server' oleg340-server: oleg340-server.virtnet: executing load_modules_local oleg340-server: Loading modules from /home/green/git/lustre-release/lustre oleg340-server: detected 4 online CPUs by sysfs oleg340-server: Force libcfs to create 2 CPU partitions oleg340-server: libkmod: kmod_module_get_refcnt: could not read integer from '/sys/module/pcc_cpufreq/refcnt': 'No such device' oleg340-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory oleg340-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg340-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg340-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format mds2: lustre-mdt2/mdt2 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg340-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg340-server: oleg340-server.virtnet: executing set_default_debug -1 all pdsh@oleg340-client: oleg340-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 start mds service on oleg340-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg340-server: oleg340-server.virtnet: executing set_default_debug -1 all pdsh@oleg340-client: oleg340-server: ssh exited with exit code 1 Commit the device label on lustre-mdt2/mdt2 Started lustre-MDT0001 oleg340-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg340-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg340-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg340-server: oleg340-server.virtnet: executing set_default_debug -1 all pdsh@oleg340-client: oleg340-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg340-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg340-server: oleg340-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg340-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec oleg340-server: oleg340-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg340-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg340-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg340-server stop mds service on oleg340-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg340-server stop mds service on oleg340-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg340-server PASS 35b (136s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 23:15:02 (1709784902) start mds service on oleg340-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg340-server: oleg340-server.virtnet: executing set_default_debug -1 all pdsh@oleg340-client: oleg340-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg340-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg340-server: oleg340-server.virtnet: executing set_default_debug -1 all pdsh@oleg340-client: oleg340-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg340-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg340-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg340-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg340-server: oleg340-server.virtnet: executing set_default_debug -1 all pdsh@oleg340-client: oleg340-server: ssh exited with exit code 1 Started lustre-OST0000 oleg340-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg340-client.virtnet: -o user_xattr,flock oleg340-server@tcp:/lustre /mnt/lustre copying 10 files to /mnt/lustre/d38.conf-sanity tar: Removing leading `/' from member names umount lustre on /mnt/lustre..... Stopping client oleg340-client.virtnet /mnt/lustre (opts:) osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=44 osp.lustre-OST0000-osc-MDT0001.prealloc_next_id=34 stop mds service on oleg340-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg340-server stop mds service on oleg340-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg340-server delete lov_objid file on MDS 000000 43 000008 start mds service on oleg340-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg340-server: oleg340-server.virtnet: executing set_default_debug -1 all pdsh@oleg340-client: oleg340-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg340-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg340-server: oleg340-server.virtnet: executing set_default_debug -1 all pdsh@oleg340-client: oleg340-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg340-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg340-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg340-client.virtnet: -o user_xattr,flock oleg340-server@tcp:/lustre /mnt/lustre osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=2 osp.lustre-OST0000-osc-MDT0001.prealloc_next_id=66 umount lustre on /mnt/lustre..... Stopping client oleg340-client.virtnet /mnt/lustre (opts:) stop mds service on oleg340-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg340-server stop mds service on oleg340-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg340-server 000000 2 000008 oleg340-server: 8+0 records in oleg340-server: 8+0 records out oleg340-server: 4096 bytes (4.1 kB) copied, 0.000677667 s, 6.0 MB/s start mds service on oleg340-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg340-server: oleg340-server.virtnet: executing set_default_debug -1 all pdsh@oleg340-client: oleg340-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg340-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg340-server: oleg340-server.virtnet: executing set_default_debug -1 all pdsh@oleg340-client: oleg340-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg340-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg340-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg340-client.virtnet: -o user_xattr,flock oleg340-server@tcp:/lustre /mnt/lustre osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=35 osp.lustre-OST0000-osc-MDT0001.prealloc_next_id=98 umount lustre on /mnt/lustre..... Stopping client oleg340-client.virtnet /mnt/lustre (opts:) stop mds service on oleg340-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg340-server stop mds service on oleg340-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg340-server 000000 34 000008 files compared the same umount lustre on /mnt/lustre..... stop ost1 service on oleg340-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg340-server stop mds service on oleg340-server stop mds service on oleg340-server unloading modules on: 'oleg340-server' oleg340-server: oleg340-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg340-client: oleg340-client: ssh exited with exit code 2 pdsh@oleg340-client: oleg340-server: ssh exited with exit code 2 pdsh@oleg340-client: oleg340-client: ssh exited with exit code 2 pdsh@oleg340-client: oleg340-server: ssh exited with exit code 2 PASS 38 (96s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg340-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg340-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg340-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg340-client: oleg340-server: ssh exited with exit code 2 oleg340-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg340-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 23:16:40 (1709785000) Stopping clients: oleg340-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg340-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg340-client: oleg340-server: ssh exited with exit code 2 oleg340-server: oleg340-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg340-server' oleg340-server: oleg340-server.virtnet: executing load_modules_local oleg340-server: Loading modules from /home/green/git/lustre-release/lustre oleg340-server: detected 4 online CPUs by sysfs oleg340-server: Force libcfs to create 2 CPU partitions oleg340-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg340-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format mds2: lustre-mdt2/mdt2 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg340-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg340-server: oleg340-server.virtnet: executing set_default_debug -1 all pdsh@oleg340-client: oleg340-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 start mds service on oleg340-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg340-server: oleg340-server.virtnet: executing set_default_debug -1 all pdsh@oleg340-client: oleg340-server: ssh exited with exit code 1 Commit the device label on lustre-mdt2/mdt2 Started lustre-MDT0001 oleg340-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg340-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg340-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg340-server: oleg340-server.virtnet: executing set_default_debug -1 all pdsh@oleg340-client: oleg340-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg340-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg340-client.virtnet: -o user_xattr,flock oleg340-server@tcp:/lustre /mnt/lustre setup single mount lustre success fail_loc=0x142 start ost2 service on oleg340-server Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg340-server: oleg340-server.virtnet: executing set_default_debug -1 all pdsh@oleg340-client: oleg340-server: ssh exited with exit code 1 Commit the device label on lustre-ost2/ost2 Started lustre-OST0001 oleg340-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec stop ost2 service on oleg340-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg340-server umount lustre on /mnt/lustre..... Stopping client oleg340-client.virtnet /mnt/lustre (opts:-f) stop ost1 service on oleg340-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg340-server stop mds service on oleg340-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg340-server stop mds service on oleg340-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg340-server unloading modules on: 'oleg340-server' oleg340-server: oleg340-server.virtnet: executing unload_modules_local modules unloaded. oleg340-server: oleg340-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre oleg340-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg340-client: oleg340-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg340-server: oleg340-server: tunefs.lustre FATAL: Device lustre-mdt2/mdt2 has not been formatted with mkfs.lustre oleg340-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg340-client: oleg340-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg340-server: oleg340-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre oleg340-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg340-client: oleg340-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg340-server: oleg340-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre oleg340-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg340-client: oleg340-server: ssh exited with exit code 19 checking for existing Lustre data: not found tunefs failed, reformatting instead Stopping clients: oleg340-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg340-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg340-client: oleg340-server: ssh exited with exit code 2 oleg340-server: oleg340-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg340-server' oleg340-server: oleg340-server.virtnet: executing load_modules_local oleg340-server: Loading modules from /home/green/git/lustre-release/lustre oleg340-server: detected 4 online CPUs by sysfs oleg340-server: Force libcfs to create 2 CPU partitions oleg340-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg340-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format mds2: lustre-mdt2/mdt2 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg340-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg340-server: oleg340-server.virtnet: executing set_default_debug -1 all pdsh@oleg340-client: oleg340-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 start mds service on oleg340-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg340-server: oleg340-server.virtnet: executing set_default_debug -1 all pdsh@oleg340-client: oleg340-server: ssh exited with exit code 1 Commit the device label on lustre-mdt2/mdt2 Started lustre-MDT0001 oleg340-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg340-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg340-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg340-server: oleg340-server.virtnet: executing set_default_debug -1 all pdsh@oleg340-client: oleg340-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg340-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg340-server: oleg340-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg340-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg340-server: oleg340-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg340-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg340-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg340-server stop mds service on oleg340-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg340-server stop mds service on oleg340-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg340-server PASS 51 (143s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:19:05 (1709785145) start mds service on oleg340-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg340-server: oleg340-server.virtnet: executing set_default_debug -1 all pdsh@oleg340-client: oleg340-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg340-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg340-server: oleg340-server.virtnet: executing set_default_debug -1 all pdsh@oleg340-client: oleg340-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg340-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg340-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg340-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg340-server: oleg340-server.virtnet: executing set_default_debug -1 all pdsh@oleg340-client: oleg340-server: ssh exited with exit code 1 Started lustre-OST0000 oleg340-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg340-client.virtnet: -o user_xattr,flock oleg340-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 731 (time 1709785174.34 total 10.63 last 68.75) - open/close 1424 (time 1709785185.04 total 21.33 last 64.79) - open/close 2042 (time 1709785195.46 total 31.76 last 59.27) - open/close 2665 (time 1709785206.04 total 42.33 last 58.91) - open/close 3241 (time 1709785216.61 total 52.90 last 54.48) - open/close 3908 (time 1709785227.53 total 63.82 last 61.08) - open/close 4430 (time 1709785238.16 total 74.45 last 49.12) - open/close 4970 (time 1709785248.99 total 85.28 last 49.86) - open/close 5476 (time 1709785259.75 total 96.04 last 47.01) - open/close 5946 (time 1709785270.49 total 106.79 last 43.75) - open/close 6195 (time 1709785280.49 total 116.79 last 24.90) - open/close 6428 (time 1709785291.44 total 127.73 last 21.29) - open/close 6876 (time 1709785302.34 total 138.63 last 41.09) - open/close 7339 (time 1709785313.24 total 149.53 last 42.49) - open/close 7820 (time 1709785324.21 total 160.50 last 43.83) - open/close 8361 (time 1709785334.67 total 170.97 last 51.72) - open/close 8815 (time 1709785344.81 total 181.10 last 44.79) - open/close 9316 (time 1709785355.22 total 191.51 last 48.13) - open/close 9892 (time 1709785366.10 total 202.39 last 52.93) - open/close 10437 (time 1709785376.10 total 212.39 last 54.49) - open/close 10783 (time 1709785386.10 total 222.40 last 34.59) - open/close 10964 (time 1709785396.24 total 232.54 last 17.85) - open/close 11570 (time 1709785406.25 total 242.54 last 60.59) - open/close 13038 (time 1709785416.25 total 252.54 last 146.73) - open/close 14438 (time 1709785426.26 total 262.55 last 139.93) - open/close 15764 (time 1709785436.26 total 272.55 last 132.59) - open/close 17016 (time 1709785446.26 total 282.55 last 125.19) - open/close 18159 (time 1709785456.26 total 292.55 last 114.29) - open/close 19311 (time 1709785466.26 total 302.56 last 115.13) - open/close 20000 (time 1709785472.45 total 308.74 last 111.33) - open/close 21002 (time 1709785482.46 total 318.75 last 100.13) - open/close 21986 (time 1709785492.46 total 328.75 last 98.38) - open/close 22914 (time 1709785502.47 total 338.76 last 92.70) - open/close 23802 (time 1709785512.47 total 348.77 last 88.78) - open/close 24682 (time 1709785522.48 total 358.77 last 87.99) - open/close 25583 (time 1709785532.48 total 368.77 last 90.08) - open/close 26381 (time 1709785542.49 total 378.78 last 79.73) - open/close 27215 (time 1709785552.49 total 388.78 last 83.34) - open/close 28000 (time 1709785562.50 total 398.79 last 78.48) - open/close 28815 (time 1709785572.50 total 408.79 last 81.48) - open/close 29621 (time 1709785582.51 total 418.80 last 80.52) - open/close 30000 (time 1709785587.18 total 423.48 last 81.05) - open/close 30750 (time 1709785597.19 total 433.48 last 74.94) - open/close 31448 (time 1709785607.19 total 443.49 last 69.78) - open/close 32128 (time 1709785617.20 total 453.50 last 67.93) - open/close 32776 (time 1709785627.21 total 463.50 last 64.79) - open/close 33462 (time 1709785637.21 total 473.51 last 68.55) - open/close 34085 (time 1709785647.22 total 483.52 last 62.23) - open/close 34703 (time 1709785657.24 total 493.53 last 61.71) - open/close 35329 (time 1709785667.25 total 503.54 last 62.53) - open/close 35965 (time 1709785677.26 total 513.55 last 63.54) - open/close 36514 (time 1709785687.27 total 523.56 last 54.86) - open/close 37082 (time 1709785697.27 total 533.56 last 56.79) - open/close 37651 (time 1709785707.28 total 543.57 last 56.85) - open/close 38247 (time 1709785717.28 total 553.58 last 59.56) - open/close 38805 (time 1709785727.30 total 563.59 last 55.72) - open/close 39346 (time 1709785737.30 total 573.59 last 54.10) - open/close 39895 (time 1709785747.30 total 583.60 last 54.86) - open/close 40432 (time 1709785757.32 total 593.61 last 53.64) - open/close 40964 (time 1709785767.32 total 603.61 last 53.18) - open/close 41504 (time 1709785777.34 total 613.63 last 53.90) - open/close 42049 (time 1709785787.35 total 623.65 last 54.41) - open/close 42557 (time 1709785797.36 total 633.66 last 50.75) - open/close 43061 (time 1709785807.37 total 643.66 last 50.35) - open/close 43567 (time 1709785817.37 total 653.67 last 50.60) - open/close 44064 (time 1709785827.38 total 663.68 last 49.65) - open/close 44564 (time 1709785837.39 total 673.68 last 49.96) - open/close 45051 (time 1709785847.39 total 683.68 last 48.69) - open/close 45540 (time 1709785857.40 total 693.69 last 48.88) - open/close 46040 (time 1709785867.41 total 703.70 last 49.94) - open/close 46536 (time 1709785877.42 total 713.71 last 49.56) - open/close 47047 (time 1709785887.44 total 723.73 last 51.01) - open/close 47521 (time 1709785897.44 total 733.74 last 47.36) - open/close 47991 (time 1709785907.46 total 743.75 last 46.94) - open/close 48454 (time 1709785917.47 total 753.76 last 46.25) - open/close 48896 (time 1709785927.49 total 763.78 last 44.12) - open/close 49352 (time 1709785937.49 total 773.79 last 45.56) - open/close 49803 (time 1709785947.51 total 783.80 last 45.03) total: 50000 open/close in 788.31 seconds: 63.43 ops/second - unlinked 0 (time 1709785952 ; total 0 ; last 0) - unlinked 10000 (time 1709786227 ; total 275 ; last 275) - unlinked 20000 (time 1709786476 ; total 524 ; last 249) - unlinked 30000 (time 1709786722 ; total 770 ; last 246) - unlinked 40000 (time 1709786826 ; total 874 ; last 104) total: 50000 unlinks in 926 seconds: 53.995682 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg340-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg340-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg340-server stop mds service on oleg340-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg340-server stop mds service on oleg340-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg340-server unloading modules on: 'oleg340-server' oleg340-server: oleg340-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg340-client: oleg340-client: ssh exited with exit code 2 pdsh@oleg340-client: oleg340-server: ssh exited with exit code 2 pdsh@oleg340-client: oleg340-client: ssh exited with exit code 2 pdsh@oleg340-client: oleg340-server: ssh exited with exit code 2 PASS 101a (1755s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg340-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg340-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg340-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg340-client: oleg340-server: ssh exited with exit code 2 oleg340-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg340-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == conf-sanity test 101b: Race events DISCONNECT and ACTIVE in osp ========================================================== 23:48:23 (1709786903) start mds service on oleg340-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg340-server' oleg340-server: oleg340-server.virtnet: executing load_modules_local oleg340-server: Loading modules from /home/green/git/lustre-release/lustre oleg340-server: detected 4 online CPUs by sysfs oleg340-server: Force libcfs to create 2 CPU partitions oleg340-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg340-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg340-server: oleg340-server.virtnet: executing set_default_debug -1 all pdsh@oleg340-client: oleg340-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg340-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg340-server: oleg340-server.virtnet: executing set_default_debug -1 all pdsh@oleg340-client: oleg340-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg340-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg340-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg340-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg340-server: oleg340-server.virtnet: executing set_default_debug -1 all pdsh@oleg340-client: oleg340-server: ssh exited with exit code 1 Started lustre-OST0000 oleg340-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg340-client.virtnet: -o user_xattr,flock oleg340-server@tcp:/lustre /mnt/lustre fail_loc=0x80002107 fail_val=20 stop ost1 service on oleg340-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg340-server start ost1 service on oleg340-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg340-server: oleg340-server.virtnet: executing set_default_debug -1 all pdsh@oleg340-client: oleg340-server: ssh exited with exit code 1 Started lustre-OST0000 oleg340-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 14 sec oleg340-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8800a839a000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800a839a000.ost_server_uuid in FULL state after 0 sec umount lustre on /mnt/lustre..... Stopping client oleg340-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg340-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg340-server stop mds service on oleg340-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg340-server stop mds service on oleg340-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg340-server unloading modules on: 'oleg340-server' oleg340-server: oleg340-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg340-client: oleg340-client: ssh exited with exit code 2 pdsh@oleg340-client: oleg340-server: ssh exited with exit code 2 pdsh@oleg340-client: oleg340-client: ssh exited with exit code 2 pdsh@oleg340-client: oleg340-server: ssh exited with exit code 2 PASS 101b (133s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg340-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg340-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory Stopping clients: oleg340-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg340-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg340-client: oleg340-server: ssh exited with exit code 2 oleg340-server: oleg340-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg340-server' oleg340-server: oleg340-server.virtnet: executing load_modules_local oleg340-server: Loading modules from /home/green/git/lustre-release/lustre oleg340-server: detected 4 online CPUs by sysfs oleg340-server: Force libcfs to create 2 CPU partitions oleg340-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg340-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format mds2: lustre-mdt2/mdt2 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 2395 sec ========== 23:51:21 (1709787081) === conf-sanity: start cleanup 23:51:22 (1709787082) === === conf-sanity: finish cleanup 23:51:22 (1709787082) ===