-----============= acceptance-small: conf-sanity ============----- Wed Mar 6 23:45:00 EST 2024 excepting tests: 32newtarball 110 skipping tests SLOW=no: 45 69 106 111 114 Stopping clients: oleg308-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg308-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg308-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg308-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg308-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg308-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg308-server oleg308-server: oleg308-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg308-server' oleg308-server: oleg308-server.virtnet: executing load_modules_local oleg308-server: Loading modules from /home/green/git/lustre-release/lustre oleg308-server: detected 4 online CPUs by sysfs oleg308-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format mds2: lustre-mdt2/mdt2 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg308-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg308-server: oleg308-server.virtnet: executing set_default_debug -1 all pdsh@oleg308-client: oleg308-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 start mds service on oleg308-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg308-server: oleg308-server.virtnet: executing set_default_debug -1 all pdsh@oleg308-client: oleg308-server: ssh exited with exit code 1 Commit the device label on lustre-mdt2/mdt2 Started lustre-MDT0001 oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg308-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg308-server: oleg308-server.virtnet: executing set_default_debug -1 all pdsh@oleg308-client: oleg308-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg308-server: oleg308-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg308-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg308-server: oleg308-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg308-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg308-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg308-server stop mds service on oleg308-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg308-server stop mds service on oleg308-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg308-server debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 35b: Continue reconnection retries, if the active server is busy ========================================================== 23:46:27 (1709786787) start mds service on oleg308-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg308-server: oleg308-server.virtnet: executing set_default_debug -1 all pdsh@oleg308-client: oleg308-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg308-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg308-server: oleg308-server.virtnet: executing set_default_debug -1 all pdsh@oleg308-client: oleg308-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg308-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg308-server: oleg308-server.virtnet: executing set_default_debug -1 all pdsh@oleg308-client: oleg308-server: ssh exited with exit code 1 Started lustre-OST0000 oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg308-client.virtnet: -o user_xattr,flock oleg308-server@tcp:/lustre /mnt/lustre debug=ha conf-sanity.sh test_35b 2024-03-0623h46m57s Set up a fake failnode for the MDS at_max=0 at_max=0 Injecting EBUSY on MDS fail_loc=0x80000136 mdc.lustre-MDT0000-mdc-ffff8800b3f71000.stats=clear mdc.lustre-MDT0001-mdc-ffff8800b3f71000.stats=clear Creating a test file and stat it File: '/mnt/lustre/d35b.conf-sanity/f35b.conf-sanity' Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115238826934274 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-03-06 23:47:43.000000000 -0500 Modify: 2024-03-06 23:47:43.000000000 -0500 Change: 2024-03-06 23:47:43.000000000 -0500 Birth: - Stop injecting EBUSY on MDS fail_loc=0 done at_max=600 at_max=600 Debug log: 88 lines, 88 kept, 0 dropped, 0 bad. umount lustre on /mnt/lustre..... Stopping client oleg308-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg308-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg308-server stop mds service on oleg308-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg308-server stop mds service on oleg308-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg308-server unloading modules on: 'oleg308-server' oleg308-server: oleg308-server.virtnet: executing unload_modules_local modules unloaded. oleg308-server: oleg308-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre oleg308-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg308-client: oleg308-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg308-server: oleg308-server: tunefs.lustre FATAL: Device lustre-mdt2/mdt2 has not been formatted with mkfs.lustre oleg308-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg308-client: oleg308-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg308-server: oleg308-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre oleg308-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg308-client: oleg308-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg308-server: oleg308-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre oleg308-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg308-client: oleg308-server: ssh exited with exit code 19 checking for existing Lustre data: not found tunefs failed, reformatting instead Stopping clients: oleg308-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg308-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg308-client: oleg308-server: ssh exited with exit code 2 oleg308-server: oleg308-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg308-server' oleg308-server: oleg308-server.virtnet: executing load_modules_local oleg308-server: Loading modules from /home/green/git/lustre-release/lustre oleg308-server: detected 4 online CPUs by sysfs oleg308-server: Force libcfs to create 2 CPU partitions oleg308-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg308-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format mds2: lustre-mdt2/mdt2 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg308-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg308-server: oleg308-server.virtnet: executing set_default_debug -1 all pdsh@oleg308-client: oleg308-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 start mds service on oleg308-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg308-server: oleg308-server.virtnet: executing set_default_debug -1 all pdsh@oleg308-client: oleg308-server: ssh exited with exit code 1 Commit the device label on lustre-mdt2/mdt2 Started lustre-MDT0001 oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg308-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg308-server: oleg308-server.virtnet: executing set_default_debug -1 all pdsh@oleg308-client: oleg308-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg308-server: oleg308-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg308-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg308-server: oleg308-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg308-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg308-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg308-server stop mds service on oleg308-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg308-server stop mds service on oleg308-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg308-server PASS 35b (224s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 23:50:13 (1709787013) start mds service on oleg308-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg308-server: oleg308-server.virtnet: executing set_default_debug -1 all pdsh@oleg308-client: oleg308-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg308-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg308-server: oleg308-server.virtnet: executing set_default_debug -1 all pdsh@oleg308-client: oleg308-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg308-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg308-server: oleg308-server.virtnet: executing set_default_debug -1 all pdsh@oleg308-client: oleg308-server: ssh exited with exit code 1 Started lustre-OST0000 oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg308-client.virtnet: -o user_xattr,flock oleg308-server@tcp:/lustre /mnt/lustre copying 10 files to /mnt/lustre/d38.conf-sanity tar: Removing leading `/' from member names umount lustre on /mnt/lustre..... Stopping client oleg308-client.virtnet /mnt/lustre (opts:) osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=44 osp.lustre-OST0000-osc-MDT0001.prealloc_next_id=34 stop mds service on oleg308-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg308-server stop mds service on oleg308-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg308-server delete lov_objid file on MDS 000000 43 000008 start mds service on oleg308-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg308-server: oleg308-server.virtnet: executing set_default_debug -1 all pdsh@oleg308-client: oleg308-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg308-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg308-server: oleg308-server.virtnet: executing set_default_debug -1 all pdsh@oleg308-client: oleg308-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg308-client.virtnet: -o user_xattr,flock oleg308-server@tcp:/lustre /mnt/lustre osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=2 osp.lustre-OST0000-osc-MDT0001.prealloc_next_id=66 umount lustre on /mnt/lustre..... Stopping client oleg308-client.virtnet /mnt/lustre (opts:) stop mds service on oleg308-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg308-server stop mds service on oleg308-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg308-server 000000 2 000008 oleg308-server: 8+0 records in oleg308-server: 8+0 records out oleg308-server: 4096 bytes (4.1 kB) copied, 0.00312484 s, 1.3 MB/s start mds service on oleg308-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg308-server: oleg308-server.virtnet: executing set_default_debug -1 all pdsh@oleg308-client: oleg308-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg308-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg308-server: oleg308-server.virtnet: executing set_default_debug -1 all pdsh@oleg308-client: oleg308-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg308-client.virtnet: -o user_xattr,flock oleg308-server@tcp:/lustre /mnt/lustre osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=35 osp.lustre-OST0000-osc-MDT0001.prealloc_next_id=98 umount lustre on /mnt/lustre..... Stopping client oleg308-client.virtnet /mnt/lustre (opts:) stop mds service on oleg308-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg308-server stop mds service on oleg308-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg308-server 000000 34 000008 files compared the same umount lustre on /mnt/lustre..... stop ost1 service on oleg308-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg308-server stop mds service on oleg308-server stop mds service on oleg308-server unloading modules on: 'oleg308-server' oleg308-server: oleg308-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg308-client: oleg308-server: ssh exited with exit code 2 pdsh@oleg308-client: oleg308-client: ssh exited with exit code 2 pdsh@oleg308-client: oleg308-server: ssh exited with exit code 2 pdsh@oleg308-client: oleg308-client: ssh exited with exit code 2 PASS 38 (171s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg308-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg308-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg308-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg308-client: oleg308-server: ssh exited with exit code 2 oleg308-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg308-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 23:53:09 (1709787189) Stopping clients: oleg308-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg308-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg308-client: oleg308-server: ssh exited with exit code 2 oleg308-server: oleg308-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg308-server' oleg308-server: oleg308-server.virtnet: executing load_modules_local oleg308-server: Loading modules from /home/green/git/lustre-release/lustre oleg308-server: detected 4 online CPUs by sysfs oleg308-server: Force libcfs to create 2 CPU partitions oleg308-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg308-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format mds2: lustre-mdt2/mdt2 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg308-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg308-server: oleg308-server.virtnet: executing set_default_debug -1 all pdsh@oleg308-client: oleg308-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 start mds service on oleg308-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg308-server: oleg308-server.virtnet: executing set_default_debug -1 all pdsh@oleg308-client: oleg308-server: ssh exited with exit code 1 Commit the device label on lustre-mdt2/mdt2 Started lustre-MDT0001 oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg308-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg308-server: oleg308-server.virtnet: executing set_default_debug -1 all pdsh@oleg308-client: oleg308-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg308-client.virtnet: -o user_xattr,flock oleg308-server@tcp:/lustre /mnt/lustre setup single mount lustre success fail_loc=0x142 start ost2 service on oleg308-server Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg308-server: oleg308-server.virtnet: executing set_default_debug -1 all pdsh@oleg308-client: oleg308-server: ssh exited with exit code 1 Commit the device label on lustre-ost2/ost2 Started lustre-OST0001 oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec stop ost2 service on oleg308-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg308-server umount lustre on /mnt/lustre..... Stopping client oleg308-client.virtnet /mnt/lustre (opts:-f) stop ost1 service on oleg308-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg308-server stop mds service on oleg308-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg308-server stop mds service on oleg308-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg308-server unloading modules on: 'oleg308-server' oleg308-server: oleg308-server.virtnet: executing unload_modules_local modules unloaded. oleg308-server: oleg308-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre oleg308-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg308-client: oleg308-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg308-server: oleg308-server: tunefs.lustre FATAL: Device lustre-mdt2/mdt2 has not been formatted with mkfs.lustre oleg308-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg308-client: oleg308-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg308-server: oleg308-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre oleg308-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg308-client: oleg308-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg308-server: oleg308-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre oleg308-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg308-client: oleg308-server: ssh exited with exit code 19 checking for existing Lustre data: not found tunefs failed, reformatting instead Stopping clients: oleg308-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg308-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg308-client: oleg308-server: ssh exited with exit code 2 oleg308-server: oleg308-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg308-server' oleg308-server: oleg308-server.virtnet: executing load_modules_local oleg308-server: Loading modules from /home/green/git/lustre-release/lustre oleg308-server: detected 4 online CPUs by sysfs oleg308-server: Force libcfs to create 2 CPU partitions oleg308-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg308-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format mds2: lustre-mdt2/mdt2 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg308-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg308-server: oleg308-server.virtnet: executing set_default_debug -1 all pdsh@oleg308-client: oleg308-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 start mds service on oleg308-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg308-server: oleg308-server.virtnet: executing set_default_debug -1 all pdsh@oleg308-client: oleg308-server: ssh exited with exit code 1 Commit the device label on lustre-mdt2/mdt2 Started lustre-MDT0001 oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg308-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg308-server: oleg308-server.virtnet: executing set_default_debug -1 all pdsh@oleg308-client: oleg308-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg308-server: oleg308-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg308-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg308-server: oleg308-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg308-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg308-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg308-server stop mds service on oleg308-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg308-server stop mds service on oleg308-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg308-server PASS 51 (286s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:58:00 (1709787480) start mds service on oleg308-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg308-server: oleg308-server.virtnet: executing set_default_debug -1 all pdsh@oleg308-client: oleg308-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg308-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg308-server: oleg308-server.virtnet: executing set_default_debug -1 all pdsh@oleg308-client: oleg308-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg308-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg308-server: oleg308-server.virtnet: executing set_default_debug -1 all pdsh@oleg308-client: oleg308-server: ssh exited with exit code 1 Started lustre-OST0000 oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg308-client.virtnet: -o user_xattr,flock oleg308-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 220 (time 1709787532.43 total 10.12 last 21.74) - open/close 363 (time 1709787542.72 total 20.41 last 13.90) - open/close 536 (time 1709787553.01 total 30.70 last 16.81) - open/close 767 (time 1709787563.93 total 41.62 last 21.15) - open/close 938 (time 1709787574.75 total 52.44 last 15.80) - open/close 1184 (time 1709787585.22 total 62.91 last 23.50) - open/close 1346 (time 1709787596.01 total 73.70 last 15.02) - open/close 1565 (time 1709787606.02 total 83.71 last 21.88) - open/close 1709 (time 1709787616.64 total 94.33 last 13.55) - open/close 1936 (time 1709787627.21 total 104.90 last 21.48) - open/close 2111 (time 1709787637.27 total 114.96 last 17.40) - open/close 2249 (time 1709787648.09 total 125.78 last 12.75) - open/close 2485 (time 1709787658.39 total 136.08 last 22.93) - open/close 2617 (time 1709787669.08 total 146.77 last 12.35) - open/close 2772 (time 1709787679.49 total 157.18 last 14.88) - open/close 2922 (time 1709787689.73 total 167.42 last 14.66) - open/close 3136 (time 1709787700.42 total 178.10 last 20.02) - open/close 3343 (time 1709787710.58 total 188.27 last 20.36) - open/close 3570 (time 1709787721.14 total 198.83 last 21.50) - open/close 3678 (time 1709787731.14 total 208.83 last 10.80) - open/close 3866 (time 1709787741.62 total 219.31 last 17.94) - open/close 4005 (time 1709787751.95 total 229.64 last 13.45) - open/close 4179 (time 1709787762.69 total 240.38 last 16.21) - open/close 4293 (time 1709787773.71 total 251.40 last 10.34) - open/close 4460 (time 1709787783.83 total 261.52 last 16.51) - open/close 4610 (time 1709787794.16 total 271.85 last 14.52) - open/close 4766 (time 1709787804.20 total 281.88 last 15.54) - open/close 4891 (time 1709787814.20 total 291.89 last 12.50) - open/close 5120 (time 1709787824.66 total 302.35 last 21.89) - open/close 5240 (time 1709787834.74 total 312.43 last 11.90) - open/close 5397 (time 1709787845.55 total 323.24 last 14.53) - open/close 5567 (time 1709787855.72 total 333.40 last 16.72) - open/close 5682 (time 1709787866.06 total 343.75 last 11.12) - open/close 5842 (time 1709787876.23 total 353.92 last 15.74) - open/close 6049 (time 1709787886.26 total 363.95 last 20.63) - open/close 6221 (time 1709787896.89 total 374.58 last 16.18) - open/close 6449 (time 1709787906.91 total 384.59 last 22.77) - open/close 6629 (time 1709787917.10 total 394.79 last 17.65) - open/close 6784 (time 1709787927.12 total 404.80 last 15.48) - open/close 6855 (time 1709787937.81 total 415.49 last 6.64) - open/close 7156 (time 1709787947.81 total 425.50 last 30.09) - open/close 7251 (time 1709787957.82 total 435.50 last 9.50) - open/close 7450 (time 1709787968.30 total 445.99 last 18.97) - open/close 7642 (time 1709787978.98 total 456.67 last 17.98) - open/close 7890 (time 1709787989.06 total 466.75 last 24.62) - open/close 8055 (time 1709788004.70 total 482.38 last 10.55) - open/close 8210 (time 1709788014.72 total 492.41 last 15.46) - open/close 8414 (time 1709788025.34 total 503.03 last 19.20) - open/close 8653 (time 1709788036.38 total 514.06 last 21.66) - open/close 8933 (time 1709788046.39 total 524.07 last 27.97) - open/close 9095 (time 1709788056.52 total 534.20 last 15.99) - open/close 9273 (time 1709788066.53 total 544.21 last 17.78) - open/close 9487 (time 1709788076.58 total 554.27 last 21.28) - open/close 9646 (time 1709788086.93 total 564.62 last 15.37) - open/close 10000 (time 1709788093.07 total 570.75 last 57.66) - open/close 10749 (time 1709788103.08 total 580.76 last 74.83) - open/close 11418 (time 1709788113.09 total 590.78 last 66.82) - open/close 12066 (time 1709788123.09 total 600.78 last 64.79) - open/close 12733 (time 1709788133.10 total 610.79 last 66.62) - open/close 13384 (time 1709788143.12 total 620.80 last 65.00) - open/close 13982 (time 1709788153.13 total 630.81 last 59.75) - open/close 14562 (time 1709788163.15 total 640.84 last 57.87) - open/close 15120 (time 1709788173.15 total 650.84 last 55.76) - open/close 15675 (time 1709788183.16 total 660.84 last 55.49) - open/close 16219 (time 1709788193.17 total 670.86 last 54.32) - open/close 16735 (time 1709788203.21 total 680.90 last 51.39) - open/close 17223 (time 1709788213.22 total 690.91 last 48.76) - open/close 17769 (time 1709788223.22 total 700.91 last 54.57) - open/close 18307 (time 1709788233.23 total 710.92 last 53.78) - open/close 18809 (time 1709788243.24 total 720.93 last 50.15) - open/close 19258 (time 1709788253.24 total 730.93 last 44.88) - open/close 19740 (time 1709788263.26 total 740.94 last 48.13) - open/close 20000 (time 1709788269.25 total 746.93 last 43.42) - open/close 20438 (time 1709788279.26 total 756.95 last 43.74) - open/close 20865 (time 1709788289.27 total 766.96 last 42.65) - open/close 21284 (time 1709788299.29 total 776.98 last 41.82) - open/close 21708 (time 1709788309.31 total 787.00 last 42.32) - open/close 22109 (time 1709788319.32 total 797.01 last 40.05) - open/close 22539 (time 1709788329.33 total 807.02 last 42.97) - open/close 22993 (time 1709788339.35 total 817.04 last 45.31) - open/close 23418 (time 1709788349.36 total 827.04 last 42.47) - open/close 23864 (time 1709788359.36 total 837.04 last 44.60) - open/close 24235 (time 1709788369.37 total 847.05 last 37.06) - open/close 24617 (time 1709788379.38 total 857.06 last 38.16) - open/close 24988 (time 1709788389.38 total 867.07 last 37.08) - open/close 25367 (time 1709788399.40 total 877.09 last 37.81) - open/close 25759 (time 1709788409.42 total 887.10 last 39.15) - open/close 26159 (time 1709788419.43 total 897.11 last 39.97) - open/close 26550 (time 1709788429.44 total 907.13 last 39.03) - open/close 26925 (time 1709788439.47 total 917.16 last 37.41) - open/close 27292 (time 1709788449.49 total 927.18 last 36.62) - open/close 27680 (time 1709788459.50 total 937.19 last 38.74) - open/close 28022 (time 1709788469.52 total 947.20 last 34.16) - open/close 28398 (time 1709788479.53 total 957.21 last 37.56) - open/close 28752 (time 1709788489.55 total 967.24 last 35.30) - open/close 29084 (time 1709788499.57 total 977.26 last 33.13) - open/close 29435 (time 1709788509.58 total 987.27 last 35.08) - open/close 29794 (time 1709788519.60 total 997.29 last 35.82) - open/close 30000 (time 1709788525.44 total 1003.13 last 35.30) - open/close 30350 (time 1709788535.47 total 1013.15 last 34.91) - open/close 30708 (time 1709788545.48 total 1023.17 last 35.75) - open/close 31057 (time 1709788555.50 total 1033.19 last 34.83) - open/close 31398 (time 1709788565.51 total 1043.20 last 34.05) - open/close 31757 (time 1709788575.52 total 1053.21 last 35.86) - open/close 32112 (time 1709788585.53 total 1063.22 last 35.48) - open/close 32463 (time 1709788595.56 total 1073.25 last 34.99) - open/close 32819 (time 1709788605.58 total 1083.27 last 35.51) - open/close 33162 (time 1709788615.61 total 1093.30 last 34.20) - open/close 33516 (time 1709788625.62 total 1103.31 last 35.36) - open/close 33857 (time 1709788635.64 total 1113.33 last 34.05) - open/close 34200 (time 1709788645.64 total 1123.33 last 34.30) - open/close 34540 (time 1709788655.67 total 1133.35 last 33.91) - open/close 34878 (time 1709788665.68 total 1143.36 last 33.77) - open/close 35194 (time 1709788675.69 total 1153.37 last 31.57) - open/close 35508 (time 1709788685.69 total 1163.38 last 31.38) - open/close 35810 (time 1709788695.72 total 1173.41 last 30.11) - open/close 36122 (time 1709788705.75 total 1183.44 last 31.11) - open/close 36428 (time 1709788715.76 total 1193.45 last 30.57) - open/close 36743 (time 1709788725.78 total 1203.47 last 31.45) - open/close 37064 (time 1709788735.79 total 1213.48 last 32.06) - open/close 37383 (time 1709788745.82 total 1223.51 last 31.82) - open/close 37691 (time 1709788755.85 total 1233.54 last 30.69) - open/close 37987 (time 1709788765.86 total 1243.55 last 29.57) - open/close 38249 (time 1709788775.86 total 1253.55 last 26.20) - open/close 38530 (time 1709788785.87 total 1263.56 last 28.09) - open/close 38801 (time 1709788795.92 total 1273.61 last 26.96) - open/close 39059 (time 1709788805.95 total 1283.63 last 25.73) - open/close 39124 (time 1709788819.75 total 1297.44 last 4.71) - open/close 39386 (time 1709788829.78 total 1307.47 last 26.12) - open/close 39668 (time 1709788839.78 total 1317.47 last 28.18) - open/close 39938 (time 1709788849.79 total 1327.48 last 26.99) - open/close 40000 (time 1709788852.12 total 1329.81 last 26.53) - open/close 40256 (time 1709788862.14 total 1339.82 last 25.57) - open/close 40504 (time 1709788872.14 total 1349.83 last 24.79) - open/close 40746 (time 1709788882.18 total 1359.87 last 24.11) - open/close 41034 (time 1709788892.20 total 1369.89 last 28.74) - open/close 41318 (time 1709788902.21 total 1379.90 last 28.37) - open/close 41592 (time 1709788912.23 total 1389.91 last 27.36) - open/close 41873 (time 1709788922.25 total 1399.94 last 28.03) - open/close 42151 (time 1709788932.28 total 1409.97 last 27.71) - open/close 42410 (time 1709788942.29 total 1419.98 last 25.88) - open/close 42667 (time 1709788952.34 total 1430.03 last 25.57) - open/close 42918 (time 1709788962.36 total 1440.05 last 25.04) - open/close 43179 (time 1709788972.37 total 1450.06 last 26.09) - open/close 43447 (time 1709788982.40 total 1460.09 last 26.71) - open/close 43706 (time 1709788992.45 total 1470.14 last 25.77) - open/close 43941 (time 1709789002.49 total 1480.17 last 23.42) - open/close 44194 (time 1709789012.50 total 1490.18 last 25.28) - open/close 44443 (time 1709789022.52 total 1500.20 last 24.85) - open/close 44681 (time 1709789032.54 total 1510.23 last 23.74) - open/close 44916 (time 1709789042.57 total 1520.26 last 23.44) - open/close 45161 (time 1709789052.57 total 1530.26 last 24.49) - open/close 45408 (time 1709789062.60 total 1540.29 last 24.64) - open/close 45647 (time 1709789072.60 total 1550.29 last 23.90) - open/close 45862 (time 1709789082.62 total 1560.31 last 21.46) - open/close 46098 (time 1709789092.64 total 1570.33 last 23.56) - open/close 46333 (time 1709789102.65 total 1580.33 last 23.49) - open/close 46572 (time 1709789112.68 total 1590.37 last 23.83) - open/close 46805 (time 1709789122.70 total 1600.39 last 23.24) - open/close 47028 (time 1709789132.73 total 1610.42 last 22.23) - open/close 47251 (time 1709789142.76 total 1620.45 last 22.24) - open/close 47480 (time 1709789152.80 total 1630.49 last 22.81) - open/close 47709 (time 1709789162.82 total 1640.50 last 22.87) - open/close 47947 (time 1709789172.84 total 1650.53 last 23.73) - open/close 48183 (time 1709789182.86 total 1660.55 last 23.56) - open/close 48407 (time 1709789192.88 total 1670.57 last 22.36) - open/close 48631 (time 1709789202.91 total 1680.60 last 22.33) - open/close 48846 (time 1709789212.95 total 1690.63 last 21.43) - open/close 49072 (time 1709789222.97 total 1700.66 last 22.55) - open/close 49297 (time 1709789232.98 total 1710.67 last 22.48) - open/close 49523 (time 1709789242.98 total 1720.67 last 22.59) - open/close 49739 (time 1709789253.02 total 1730.71 last 21.53) - open/close 49959 (time 1709789263.03 total 1740.71 last 21.99) total: 50000 open/close in 1742.55 seconds: 28.69 ops/second - unlinked 0 (time 1709789266 ; total 0 ; last 0) - unlinked 10000 (time 1709789805 ; total 539 ; last 539) - unlinked 20000 (time 1709790231 ; total 965 ; last 426) - unlinked 30000 (time 1709790423 ; total 1157 ; last 192) - unlinked 40000 (time 1709790492 ; total 1226 ; last 69) total: 50000 unlinks in 1260 seconds: 39.682541 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg308-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg308-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg308-server stop mds service on oleg308-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg308-server stop mds service on oleg308-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg308-server unloading modules on: 'oleg308-server' oleg308-server: oleg308-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg308-client: oleg308-client: ssh exited with exit code 2 pdsh@oleg308-client: oleg308-server: ssh exited with exit code 2 pdsh@oleg308-client: oleg308-client: ssh exited with exit code 2 pdsh@oleg308-client: oleg308-server: ssh exited with exit code 2 PASS 101a (3064s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg308-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg308-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg308-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg308-client: oleg308-server: ssh exited with exit code 2 oleg308-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg308-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == conf-sanity test 101b: Race events DISCONNECT and ACTIVE in osp ========================================================== 00:49:06 (1709790546) start mds service on oleg308-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg308-server' oleg308-server: oleg308-server.virtnet: executing load_modules_local oleg308-server: Loading modules from /home/green/git/lustre-release/lustre oleg308-server: detected 4 online CPUs by sysfs oleg308-server: Force libcfs to create 2 CPU partitions oleg308-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg308-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg308-server: oleg308-server.virtnet: executing set_default_debug -1 all pdsh@oleg308-client: oleg308-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg308-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg308-server: oleg308-server.virtnet: executing set_default_debug -1 all pdsh@oleg308-client: oleg308-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg308-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg308-server: oleg308-server.virtnet: executing set_default_debug -1 all pdsh@oleg308-client: oleg308-server: ssh exited with exit code 1 Started lustre-OST0000 oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg308-client.virtnet: -o user_xattr,flock oleg308-server@tcp:/lustre /mnt/lustre fail_loc=0x80002107 fail_val=20 stop ost1 service on oleg308-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg308-server start ost1 service on oleg308-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg308-server: oleg308-server.virtnet: executing set_default_debug -1 all pdsh@oleg308-client: oleg308-server: ssh exited with exit code 1 Started lustre-OST0000 oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 11 sec oleg308-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8800b5854800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b5854800.ost_server_uuid in FULL state after 0 sec umount lustre on /mnt/lustre..... Stopping client oleg308-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg308-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg308-server stop mds service on oleg308-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg308-server stop mds service on oleg308-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg308-server unloading modules on: 'oleg308-server' oleg308-server: oleg308-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg308-client: oleg308-client: ssh exited with exit code 2 pdsh@oleg308-client: oleg308-server: ssh exited with exit code 2 pdsh@oleg308-client: oleg308-client: ssh exited with exit code 2 pdsh@oleg308-client: oleg308-server: ssh exited with exit code 2 PASS 101b (94s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg308-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg308-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory Stopping clients: oleg308-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg308-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg308-client: oleg308-server: ssh exited with exit code 2 oleg308-server: oleg308-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg308-server' oleg308-server: oleg308-server.virtnet: executing load_modules_local oleg308-server: Loading modules from /home/green/git/lustre-release/lustre oleg308-server: detected 4 online CPUs by sysfs oleg308-server: Force libcfs to create 2 CPU partitions oleg308-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg308-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format mds2: lustre-mdt2/mdt2 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 3972 sec ========== 00:51:12 (1709790672) === conf-sanity: start cleanup 00:51:12 (1709790672) === === conf-sanity: finish cleanup 00:51:12 (1709790672) ===