-----============= acceptance-small: conf-sanity ============----- Wed Mar 6 23:10:49 EST 2024 excepting tests: 32newtarball 110 skipping tests SLOW=no: 45 69 106 111 114 Stopping clients: oleg140-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg140-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg140-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg140-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg140-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg140-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg140-server oleg140-server: oleg140-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg140-server' oleg140-server: oleg140-server.virtnet: executing load_modules_local oleg140-server: Loading modules from /home/green/git/lustre-release/lustre oleg140-server: detected 4 online CPUs by sysfs oleg140-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format mds2: lustre-mdt2/mdt2 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg140-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg140-server: oleg140-server.virtnet: executing set_default_debug -1 all pdsh@oleg140-client: oleg140-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 start mds service on oleg140-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg140-server: oleg140-server.virtnet: executing set_default_debug -1 all pdsh@oleg140-client: oleg140-server: ssh exited with exit code 1 Commit the device label on lustre-mdt2/mdt2 Started lustre-MDT0001 oleg140-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg140-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg140-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg140-server: oleg140-server.virtnet: executing set_default_debug -1 all pdsh@oleg140-client: oleg140-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg140-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg140-server: oleg140-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg140-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg140-server: oleg140-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg140-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg140-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg140-server stop mds service on oleg140-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg140-server stop mds service on oleg140-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg140-server debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 35b: Continue reconnection retries, if the active server is busy ========================================================== 23:12:00 (1709784720) start mds service on oleg140-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg140-server: oleg140-server.virtnet: executing set_default_debug -1 all pdsh@oleg140-client: oleg140-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg140-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg140-server: oleg140-server.virtnet: executing set_default_debug -1 all pdsh@oleg140-client: oleg140-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg140-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg140-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg140-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg140-server: oleg140-server.virtnet: executing set_default_debug -1 all pdsh@oleg140-client: oleg140-server: ssh exited with exit code 1 Started lustre-OST0000 oleg140-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg140-client.virtnet: -o user_xattr,flock oleg140-server@tcp:/lustre /mnt/lustre debug=ha conf-sanity.sh test_35b 2024-03-0623h12m17s Set up a fake failnode for the MDS at_max=0 at_max=0 Injecting EBUSY on MDS fail_loc=0x80000136 mdc.lustre-MDT0000-mdc-ffff8800b584d800.stats=clear mdc.lustre-MDT0001-mdc-ffff8800b584d800.stats=clear Creating a test file and stat it File: '/mnt/lustre/d35b.conf-sanity/f35b.conf-sanity' Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115238826934274 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-03-06 23:13:00.000000000 -0500 Modify: 2024-03-06 23:13:00.000000000 -0500 Change: 2024-03-06 23:13:00.000000000 -0500 Birth: - Stop injecting EBUSY on MDS fail_loc=0 done at_max=600 at_max=600 Debug log: 63 lines, 63 kept, 0 dropped, 0 bad. umount lustre on /mnt/lustre..... Stopping client oleg140-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg140-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg140-server stop mds service on oleg140-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg140-server stop mds service on oleg140-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg140-server unloading modules on: 'oleg140-server' oleg140-server: oleg140-server.virtnet: executing unload_modules_local modules unloaded. oleg140-server: oleg140-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre oleg140-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg140-client: oleg140-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg140-server: oleg140-server: tunefs.lustre FATAL: Device lustre-mdt2/mdt2 has not been formatted with mkfs.lustre oleg140-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg140-client: oleg140-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg140-server: oleg140-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre oleg140-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg140-client: oleg140-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg140-server: oleg140-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre oleg140-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg140-client: oleg140-server: ssh exited with exit code 19 checking for existing Lustre data: not found tunefs failed, reformatting instead Stopping clients: oleg140-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg140-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg140-client: oleg140-server: ssh exited with exit code 2 oleg140-server: oleg140-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg140-server' oleg140-server: oleg140-server.virtnet: executing load_modules_local oleg140-server: Loading modules from /home/green/git/lustre-release/lustre oleg140-server: detected 4 online CPUs by sysfs oleg140-server: Force libcfs to create 2 CPU partitions oleg140-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg140-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg140-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format mds2: lustre-mdt2/mdt2 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg140-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg140-server: oleg140-server.virtnet: executing set_default_debug -1 all pdsh@oleg140-client: oleg140-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 start mds service on oleg140-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg140-server: oleg140-server.virtnet: executing set_default_debug -1 all pdsh@oleg140-client: oleg140-server: ssh exited with exit code 1 Commit the device label on lustre-mdt2/mdt2 Started lustre-MDT0001 oleg140-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg140-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg140-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg140-server: oleg140-server.virtnet: executing set_default_debug -1 all pdsh@oleg140-client: oleg140-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg140-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg140-server: oleg140-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg140-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg140-server: oleg140-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg140-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg140-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg140-server stop mds service on oleg140-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg140-server stop mds service on oleg140-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg140-server PASS 35b (141s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 23:14:23 (1709784863) start mds service on oleg140-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg140-server: oleg140-server.virtnet: executing set_default_debug -1 all pdsh@oleg140-client: oleg140-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg140-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg140-server: oleg140-server.virtnet: executing set_default_debug -1 all pdsh@oleg140-client: oleg140-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg140-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg140-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg140-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg140-server: oleg140-server.virtnet: executing set_default_debug -1 all pdsh@oleg140-client: oleg140-server: ssh exited with exit code 1 Started lustre-OST0000 oleg140-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg140-client.virtnet: -o user_xattr,flock oleg140-server@tcp:/lustre /mnt/lustre copying 10 files to /mnt/lustre/d38.conf-sanity tar: Removing leading `/' from member names umount lustre on /mnt/lustre..... Stopping client oleg140-client.virtnet /mnt/lustre (opts:) osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=44 osp.lustre-OST0000-osc-MDT0001.prealloc_next_id=34 stop mds service on oleg140-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg140-server stop mds service on oleg140-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg140-server delete lov_objid file on MDS 000000 43 000008 start mds service on oleg140-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg140-server: oleg140-server.virtnet: executing set_default_debug -1 all pdsh@oleg140-client: oleg140-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg140-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg140-server: oleg140-server.virtnet: executing set_default_debug -1 all pdsh@oleg140-client: oleg140-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg140-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg140-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg140-client.virtnet: -o user_xattr,flock oleg140-server@tcp:/lustre /mnt/lustre osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=2 osp.lustre-OST0000-osc-MDT0001.prealloc_next_id=66 umount lustre on /mnt/lustre..... Stopping client oleg140-client.virtnet /mnt/lustre (opts:) stop mds service on oleg140-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg140-server stop mds service on oleg140-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg140-server 000000 2 000008 oleg140-server: 8+0 records in oleg140-server: 8+0 records out oleg140-server: 4096 bytes (4.1 kB) copied, 0.000554658 s, 7.4 MB/s start mds service on oleg140-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg140-server: oleg140-server.virtnet: executing set_default_debug -1 all pdsh@oleg140-client: oleg140-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg140-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg140-server: oleg140-server.virtnet: executing set_default_debug -1 all pdsh@oleg140-client: oleg140-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg140-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg140-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg140-client.virtnet: -o user_xattr,flock oleg140-server@tcp:/lustre /mnt/lustre osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=35 osp.lustre-OST0000-osc-MDT0001.prealloc_next_id=98 umount lustre on /mnt/lustre..... Stopping client oleg140-client.virtnet /mnt/lustre (opts:) stop mds service on oleg140-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg140-server stop mds service on oleg140-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg140-server 000000 34 000008 files compared the same umount lustre on /mnt/lustre..... stop ost1 service on oleg140-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg140-server stop mds service on oleg140-server stop mds service on oleg140-server unloading modules on: 'oleg140-server' oleg140-server: oleg140-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg140-client: oleg140-client: ssh exited with exit code 2 pdsh@oleg140-client: oleg140-server: ssh exited with exit code 2 pdsh@oleg140-client: oleg140-client: ssh exited with exit code 2 pdsh@oleg140-client: oleg140-server: ssh exited with exit code 2 PASS 38 (84s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg140-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg140-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg140-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg140-client: oleg140-server: ssh exited with exit code 2 oleg140-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg140-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 23:15:49 (1709784949) Stopping clients: oleg140-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg140-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg140-client: oleg140-server: ssh exited with exit code 2 oleg140-server: oleg140-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg140-server' oleg140-server: oleg140-server.virtnet: executing load_modules_local oleg140-server: Loading modules from /home/green/git/lustre-release/lustre oleg140-server: detected 4 online CPUs by sysfs oleg140-server: Force libcfs to create 2 CPU partitions oleg140-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg140-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format mds2: lustre-mdt2/mdt2 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg140-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg140-server: oleg140-server.virtnet: executing set_default_debug -1 all pdsh@oleg140-client: oleg140-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 start mds service on oleg140-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg140-server: oleg140-server.virtnet: executing set_default_debug -1 all pdsh@oleg140-client: oleg140-server: ssh exited with exit code 1 Commit the device label on lustre-mdt2/mdt2 Started lustre-MDT0001 oleg140-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg140-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg140-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg140-server: oleg140-server.virtnet: executing set_default_debug -1 all pdsh@oleg140-client: oleg140-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg140-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg140-client.virtnet: -o user_xattr,flock oleg140-server@tcp:/lustre /mnt/lustre setup single mount lustre success fail_loc=0x142 start ost2 service on oleg140-server Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg140-server: oleg140-server.virtnet: executing set_default_debug -1 all pdsh@oleg140-client: oleg140-server: ssh exited with exit code 1 Commit the device label on lustre-ost2/ost2 Started lustre-OST0001 oleg140-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 2 sec stop ost2 service on oleg140-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg140-server umount lustre on /mnt/lustre..... Stopping client oleg140-client.virtnet /mnt/lustre (opts:-f) stop ost1 service on oleg140-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg140-server stop mds service on oleg140-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg140-server stop mds service on oleg140-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg140-server unloading modules on: 'oleg140-server' oleg140-server: oleg140-server.virtnet: executing unload_modules_local modules unloaded. oleg140-server: oleg140-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre oleg140-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg140-client: oleg140-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg140-server: oleg140-server: tunefs.lustre FATAL: Device lustre-mdt2/mdt2 has not been formatted with mkfs.lustre oleg140-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg140-client: oleg140-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg140-server: oleg140-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre oleg140-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg140-client: oleg140-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg140-server: oleg140-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre oleg140-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg140-client: oleg140-server: ssh exited with exit code 19 checking for existing Lustre data: not found tunefs failed, reformatting instead Stopping clients: oleg140-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg140-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg140-client: oleg140-server: ssh exited with exit code 2 oleg140-server: oleg140-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg140-server' oleg140-server: oleg140-server.virtnet: executing load_modules_local oleg140-server: Loading modules from /home/green/git/lustre-release/lustre oleg140-server: detected 4 online CPUs by sysfs oleg140-server: Force libcfs to create 2 CPU partitions oleg140-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg140-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format mds2: lustre-mdt2/mdt2 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg140-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg140-server: oleg140-server.virtnet: executing set_default_debug -1 all pdsh@oleg140-client: oleg140-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 start mds service on oleg140-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg140-server: oleg140-server.virtnet: executing set_default_debug -1 all pdsh@oleg140-client: oleg140-server: ssh exited with exit code 1 Commit the device label on lustre-mdt2/mdt2 Started lustre-MDT0001 oleg140-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg140-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg140-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg140-server: oleg140-server.virtnet: executing set_default_debug -1 all pdsh@oleg140-client: oleg140-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg140-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg140-server: oleg140-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg140-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec oleg140-server: oleg140-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg140-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg140-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg140-server stop mds service on oleg140-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg140-server stop mds service on oleg140-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg140-server PASS 51 (154s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:18:25 (1709785105) start mds service on oleg140-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg140-server: oleg140-server.virtnet: executing set_default_debug -1 all pdsh@oleg140-client: oleg140-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg140-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg140-server: oleg140-server.virtnet: executing set_default_debug -1 all pdsh@oleg140-client: oleg140-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg140-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg140-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg140-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg140-server: oleg140-server.virtnet: executing set_default_debug -1 all pdsh@oleg140-client: oleg140-server: ssh exited with exit code 1 Started lustre-OST0000 oleg140-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg140-client.virtnet: -o user_xattr,flock oleg140-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 789 (time 1709785133.92 total 10.75 last 73.38) - open/close 1514 (time 1709785144.65 total 21.48 last 67.56) - open/close 2144 (time 1709785155.32 total 32.15 last 59.07) - open/close 2767 (time 1709785165.88 total 42.72 last 58.97) - open/close 3353 (time 1709785176.56 total 53.40 last 54.87) - open/close 3922 (time 1709785189.20 total 66.04 last 45.01) - open/close 4503 (time 1709785199.90 total 76.73 last 54.32) - open/close 5061 (time 1709785210.61 total 87.44 last 52.11) - open/close 5650 (time 1709785221.34 total 98.17 last 54.89) - open/close 6027 (time 1709785231.71 total 108.54 last 36.36) - open/close 6490 (time 1709785242.29 total 119.12 last 43.75) - open/close 6934 (time 1709785252.84 total 129.68 last 42.08) - open/close 7364 (time 1709785263.35 total 140.19 last 40.90) - open/close 7805 (time 1709785273.91 total 150.74 last 41.78) - open/close 8193 (time 1709785284.50 total 161.33 last 36.65) - open/close 8537 (time 1709785295.02 total 171.85 last 32.70) - open/close 8997 (time 1709785305.94 total 182.77 last 42.11) - open/close 9408 (time 1709785316.59 total 193.43 last 38.57) - open/close 9867 (time 1709785327.51 total 204.34 last 42.05) - open/close 10000 (time 1709785330.09 total 206.92 last 51.60) - open/close 10367 (time 1709785340.71 total 217.54 last 34.56) - open/close 10831 (time 1709785350.74 total 227.58 last 46.23) - open/close 11171 (time 1709785361.17 total 238.00 last 32.61) - open/close 13572 (time 1709785371.17 total 248.01 last 240.05) - open/close 15727 (time 1709785381.17 total 258.01 last 215.47) - open/close 17548 (time 1709785391.18 total 268.01 last 182.03) - open/close 18807 (time 1709785401.18 total 278.01 last 125.86) - open/close 20000 (time 1709785408.88 total 285.71 last 154.92) - open/close 21392 (time 1709785418.89 total 295.72 last 139.14) - open/close 22704 (time 1709785428.89 total 305.73 last 131.10) - open/close 23950 (time 1709785438.90 total 315.74 last 124.49) - open/close 25231 (time 1709785448.91 total 325.74 last 128.02) - open/close 26199 (time 1709785458.91 total 335.75 last 96.74) - open/close 27370 (time 1709785468.92 total 345.75 last 117.03) - open/close 28462 (time 1709785478.92 total 355.76 last 109.17) - open/close 29469 (time 1709785488.93 total 365.76 last 100.63) - open/close 30000 (time 1709785495.04 total 371.87 last 86.95) - open/close 30781 (time 1709785505.04 total 381.88 last 78.05) - open/close 31660 (time 1709785515.05 total 391.88 last 87.86) - open/close 32496 (time 1709785525.05 total 401.89 last 83.54) - open/close 33277 (time 1709785535.05 total 411.89 last 78.10) - open/close 34135 (time 1709785545.06 total 421.90 last 85.72) - open/close 34976 (time 1709785555.07 total 431.91 last 84.03) - open/close 35742 (time 1709785565.09 total 441.92 last 76.50) - open/close 36531 (time 1709785575.09 total 451.93 last 78.85) - open/close 37313 (time 1709785585.10 total 461.93 last 78.13) - open/close 38087 (time 1709785595.11 total 471.94 last 77.35) - open/close 38784 (time 1709785605.11 total 481.95 last 69.66) - open/close 39474 (time 1709785615.12 total 491.95 last 68.98) - open/close 40000 (time 1709785622.47 total 499.30 last 71.53) - open/close 40678 (time 1709785632.48 total 509.31 last 67.73) - open/close 41372 (time 1709785642.49 total 519.32 last 69.33) - open/close 42014 (time 1709785652.49 total 529.32 last 64.20) - open/close 42653 (time 1709785662.49 total 539.33 last 63.89) - open/close 43299 (time 1709785672.50 total 549.33 last 64.54) - open/close 43888 (time 1709785682.52 total 559.35 last 58.79) - open/close 44468 (time 1709785692.52 total 569.36 last 57.98) - open/close 45019 (time 1709785702.53 total 579.37 last 55.04) - open/close 45614 (time 1709785712.55 total 589.38 last 59.42) - open/close 46183 (time 1709785722.56 total 599.40 last 56.81) - open/close 46758 (time 1709785732.57 total 609.40 last 57.49) - open/close 47368 (time 1709785742.57 total 619.41 last 60.95) - open/close 47930 (time 1709785752.58 total 629.41 last 56.19) - open/close 48498 (time 1709785762.58 total 639.41 last 56.79) - open/close 49044 (time 1709785772.59 total 649.42 last 54.55) - open/close 49610 (time 1709785782.60 total 659.44 last 56.51) total: 50000 open/close in 666.60 seconds: 75.01 ops/second - unlinked 0 (time 1709785790 ; total 0 ; last 0) - unlinked 10000 (time 1709785990 ; total 200 ; last 200) - unlinked 20000 (time 1709786137 ; total 347 ; last 147) - unlinked 30000 (time 1709786233 ; total 443 ; last 96) - unlinked 40000 (time 1709786295 ; total 505 ; last 62) total: 50000 unlinks in 539 seconds: 92.764381 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg140-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg140-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg140-server stop mds service on oleg140-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg140-server stop mds service on oleg140-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg140-server unloading modules on: 'oleg140-server' oleg140-server: oleg140-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg140-client: oleg140-client: ssh exited with exit code 2 pdsh@oleg140-client: oleg140-server: ssh exited with exit code 2 pdsh@oleg140-client: oleg140-client: ssh exited with exit code 2 pdsh@oleg140-client: oleg140-server: ssh exited with exit code 2 PASS 101a (1242s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg140-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg140-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg140-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg140-client: oleg140-server: ssh exited with exit code 2 oleg140-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg140-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == conf-sanity test 101b: Race events DISCONNECT and ACTIVE in osp ========================================================== 23:39:09 (1709786349) start mds service on oleg140-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg140-server' oleg140-server: oleg140-server.virtnet: executing load_modules_local oleg140-server: Loading modules from /home/green/git/lustre-release/lustre oleg140-server: detected 4 online CPUs by sysfs oleg140-server: Force libcfs to create 2 CPU partitions oleg140-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg140-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg140-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg140-server: oleg140-server.virtnet: executing set_default_debug -1 all pdsh@oleg140-client: oleg140-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg140-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg140-server: oleg140-server.virtnet: executing set_default_debug -1 all pdsh@oleg140-client: oleg140-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg140-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg140-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg140-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg140-server: oleg140-server.virtnet: executing set_default_debug -1 all pdsh@oleg140-client: oleg140-server: ssh exited with exit code 1 Started lustre-OST0000 oleg140-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg140-client.virtnet: -o user_xattr,flock oleg140-server@tcp:/lustre /mnt/lustre fail_loc=0x80002107 fail_val=20 stop ost1 service on oleg140-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg140-server start ost1 service on oleg140-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg140-server: oleg140-server.virtnet: executing set_default_debug -1 all pdsh@oleg140-client: oleg140-server: ssh exited with exit code 1 Started lustre-OST0000 oleg140-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 9 sec oleg140-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff880136a92800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff880136a92800.ost_server_uuid in FULL state after 0 sec umount lustre on /mnt/lustre..... Stopping client oleg140-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg140-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg140-server stop mds service on oleg140-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg140-server stop mds service on oleg140-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg140-server unloading modules on: 'oleg140-server' oleg140-server: oleg140-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg140-client: oleg140-client: ssh exited with exit code 2 pdsh@oleg140-client: oleg140-server: ssh exited with exit code 2 pdsh@oleg140-client: oleg140-client: ssh exited with exit code 2 pdsh@oleg140-client: oleg140-server: ssh exited with exit code 2 PASS 101b (94s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg140-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg140-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory Stopping clients: oleg140-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg140-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg140-client: oleg140-server: ssh exited with exit code 2 oleg140-server: oleg140-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg140-server' oleg140-server: oleg140-server.virtnet: executing load_modules_local oleg140-server: Loading modules from /home/green/git/lustre-release/lustre oleg140-server: detected 4 online CPUs by sysfs oleg140-server: Force libcfs to create 2 CPU partitions oleg140-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg140-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg140-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format mds2: lustre-mdt2/mdt2 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 1816 sec ========== 23:41:05 (1709786465) === conf-sanity: start cleanup 23:41:06 (1709786466) === === conf-sanity: finish cleanup 23:41:06 (1709786466) ===