-----============= acceptance-small: conf-sanity ============----- Wed Mar 6 23:10:20 EST 2024 excepting tests: 32newtarball 110 skipping tests SLOW=no: 45 69 106 111 114 Stopping clients: oleg346-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg346-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg346-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg346-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg346-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg346-server oleg346-server: oleg346-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg346-server' oleg346-server: oleg346-server.virtnet: executing load_modules_local oleg346-server: Loading modules from /home/green/git/lustre-release/lustre oleg346-server: detected 4 online CPUs by sysfs oleg346-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg346-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg346-server: oleg346-server.virtnet: executing set_default_debug -1 all pdsh@oleg346-client: oleg346-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg346-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg346-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg346-server: oleg346-server.virtnet: executing set_default_debug -1 all pdsh@oleg346-client: oleg346-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg346-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg346-server: oleg346-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg346-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg346-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg346-server stop mds service on oleg346-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg346-server debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 35b: Continue reconnection retries, if the active server is busy ========================================================== 23:11:22 (1709784682) start mds service on oleg346-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg346-server: oleg346-server.virtnet: executing set_default_debug -1 all pdsh@oleg346-client: oleg346-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg346-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg346-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg346-server: oleg346-server.virtnet: executing set_default_debug -1 all pdsh@oleg346-client: oleg346-server: ssh exited with exit code 1 Started lustre-OST0000 oleg346-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg346-client.virtnet: -o user_xattr,flock oleg346-server@tcp:/lustre /mnt/lustre debug=ha conf-sanity.sh test_35b 2024-03-0623h11m36s Set up a fake failnode for the MDS at_max=0 at_max=0 Injecting EBUSY on MDS fail_loc=0x80000136 mdc.lustre-MDT0000-mdc-ffff8800aad39000.stats=clear Creating a test file and stat it File: '/mnt/lustre/d35b.conf-sanity/f35b.conf-sanity' Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115238826934274 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-03-06 23:12:19.000000000 -0500 Modify: 2024-03-06 23:12:19.000000000 -0500 Change: 2024-03-06 23:12:19.000000000 -0500 Birth: - Stop injecting EBUSY on MDS fail_loc=0 done at_max=600 at_max=600 Debug log: 54 lines, 54 kept, 0 dropped, 0 bad. umount lustre on /mnt/lustre..... Stopping client oleg346-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg346-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg346-server stop mds service on oleg346-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg346-server unloading modules on: 'oleg346-server' oleg346-server: oleg346-server.virtnet: executing unload_modules_local modules unloaded. oleg346-server: oleg346-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre oleg346-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg346-client: oleg346-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg346-server: oleg346-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre oleg346-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg346-client: oleg346-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg346-server: oleg346-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre oleg346-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg346-client: oleg346-server: ssh exited with exit code 19 checking for existing Lustre data: not found tunefs failed, reformatting instead Stopping clients: oleg346-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg346-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg346-client: oleg346-server: ssh exited with exit code 2 oleg346-server: oleg346-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg346-server' oleg346-server: oleg346-server.virtnet: executing load_modules_local oleg346-server: Loading modules from /home/green/git/lustre-release/lustre oleg346-server: detected 4 online CPUs by sysfs oleg346-server: Force libcfs to create 2 CPU partitions oleg346-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg346-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg346-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg346-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg346-server: oleg346-server.virtnet: executing set_default_debug -1 all pdsh@oleg346-client: oleg346-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg346-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg346-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg346-server: oleg346-server.virtnet: executing set_default_debug -1 all pdsh@oleg346-client: oleg346-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg346-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg346-server: oleg346-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg346-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg346-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg346-server stop mds service on oleg346-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg346-server PASS 35b (112s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 23:13:15 (1709784795) start mds service on oleg346-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg346-server: oleg346-server.virtnet: executing set_default_debug -1 all pdsh@oleg346-client: oleg346-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg346-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg346-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg346-server: oleg346-server.virtnet: executing set_default_debug -1 all pdsh@oleg346-client: oleg346-server: ssh exited with exit code 1 Started lustre-OST0000 oleg346-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg346-client.virtnet: -o user_xattr,flock oleg346-server@tcp:/lustre /mnt/lustre copying 10 files to /mnt/lustre/d38.conf-sanity tar: Removing leading `/' from member names umount lustre on /mnt/lustre..... Stopping client oleg346-client.virtnet /mnt/lustre (opts:) osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=44 stop mds service on oleg346-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg346-server delete lov_objid file on MDS 000000 43 000008 start mds service on oleg346-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg346-server: oleg346-server.virtnet: executing set_default_debug -1 all pdsh@oleg346-client: oleg346-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg346-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg346-client.virtnet: -o user_xattr,flock oleg346-server@tcp:/lustre /mnt/lustre osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=2 umount lustre on /mnt/lustre..... Stopping client oleg346-client.virtnet /mnt/lustre (opts:) stop mds service on oleg346-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg346-server 000000 2 000008 oleg346-server: 8+0 records in oleg346-server: 8+0 records out oleg346-server: 4096 bytes (4.1 kB) copied, 0.000660095 s, 6.2 MB/s start mds service on oleg346-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg346-server: oleg346-server.virtnet: executing set_default_debug -1 all pdsh@oleg346-client: oleg346-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg346-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg346-client.virtnet: -o user_xattr,flock oleg346-server@tcp:/lustre /mnt/lustre osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=35 umount lustre on /mnt/lustre..... Stopping client oleg346-client.virtnet /mnt/lustre (opts:) stop mds service on oleg346-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg346-server 000000 34 000008 files compared the same umount lustre on /mnt/lustre..... stop ost1 service on oleg346-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg346-server stop mds service on oleg346-server unloading modules on: 'oleg346-server' oleg346-server: oleg346-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg346-client: oleg346-client: ssh exited with exit code 2 pdsh@oleg346-client: oleg346-server: ssh exited with exit code 2 pdsh@oleg346-client: oleg346-client: ssh exited with exit code 2 pdsh@oleg346-client: oleg346-server: ssh exited with exit code 2 PASS 38 (56s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg346-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg346-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg346-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg346-client: oleg346-server: ssh exited with exit code 2 oleg346-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg346-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 23:14:12 (1709784852) Stopping clients: oleg346-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg346-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg346-client: oleg346-server: ssh exited with exit code 2 oleg346-server: oleg346-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_refcnt: could not read integer from '/sys/module/acpi_cpufreq/refcnt': 'No such device' libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg346-server' oleg346-server: oleg346-server.virtnet: executing load_modules_local oleg346-server: Loading modules from /home/green/git/lustre-release/lustre oleg346-server: detected 4 online CPUs by sysfs oleg346-server: Force libcfs to create 2 CPU partitions oleg346-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg346-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg346-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg346-server: oleg346-server.virtnet: executing set_default_debug -1 all pdsh@oleg346-client: oleg346-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg346-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg346-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg346-server: oleg346-server.virtnet: executing set_default_debug -1 all pdsh@oleg346-client: oleg346-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg346-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg346-client.virtnet: -o user_xattr,flock oleg346-server@tcp:/lustre /mnt/lustre setup single mount lustre success fail_loc=0x142 start ost2 service on oleg346-server Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg346-server: oleg346-server.virtnet: executing set_default_debug -1 all pdsh@oleg346-client: oleg346-server: ssh exited with exit code 1 Commit the device label on lustre-ost2/ost2 Started lustre-OST0001 oleg346-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec stop ost2 service on oleg346-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg346-server umount lustre on /mnt/lustre..... Stopping client oleg346-client.virtnet /mnt/lustre (opts:-f) stop ost1 service on oleg346-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg346-server stop mds service on oleg346-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg346-server unloading modules on: 'oleg346-server' oleg346-server: oleg346-server.virtnet: executing unload_modules_local modules unloaded. oleg346-server: oleg346-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre oleg346-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg346-client: oleg346-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg346-server: oleg346-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre oleg346-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg346-client: oleg346-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg346-server: oleg346-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre oleg346-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg346-client: oleg346-server: ssh exited with exit code 19 checking for existing Lustre data: not found tunefs failed, reformatting instead Stopping clients: oleg346-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg346-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg346-client: oleg346-server: ssh exited with exit code 2 oleg346-server: oleg346-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg346-server' oleg346-server: oleg346-server.virtnet: executing load_modules_local oleg346-server: Loading modules from /home/green/git/lustre-release/lustre oleg346-server: detected 4 online CPUs by sysfs oleg346-server: Force libcfs to create 2 CPU partitions oleg346-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg346-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg346-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg346-server: oleg346-server.virtnet: executing set_default_debug -1 all pdsh@oleg346-client: oleg346-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg346-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg346-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg346-server: oleg346-server.virtnet: executing set_default_debug -1 all pdsh@oleg346-client: oleg346-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg346-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg346-server: oleg346-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg346-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg346-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg346-server stop mds service on oleg346-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg346-server PASS 51 (120s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:16:14 (1709784974) start mds service on oleg346-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg346-server: oleg346-server.virtnet: executing set_default_debug -1 all pdsh@oleg346-client: oleg346-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg346-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg346-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg346-server: oleg346-server.virtnet: executing set_default_debug -1 all pdsh@oleg346-client: oleg346-server: ssh exited with exit code 1 Started lustre-OST0000 oleg346-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg346-client.virtnet: -o user_xattr,flock oleg346-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 883 (time 1709784999.48 total 10.78 last 81.90) - open/close 1406 (time 1709785009.60 total 20.90 last 51.68) - open/close 2010 (time 1709785019.98 total 31.28 last 58.19) - open/close 2654 (time 1709785030.50 total 41.80 last 61.20) - open/close 3206 (time 1709785040.84 total 52.14 last 53.43) - open/close 3766 (time 1709785051.27 total 62.57 last 53.69) - open/close 4274 (time 1709785061.61 total 72.90 last 49.14) - open/close 4849 (time 1709785072.25 total 83.55 last 54.02) - open/close 5371 (time 1709785082.70 total 94.00 last 49.95) - open/close 5818 (time 1709785093.01 total 104.31 last 43.35) - open/close 6244 (time 1709785103.33 total 114.63 last 41.29) - open/close 6721 (time 1709785113.82 total 125.12 last 45.46) - open/close 7202 (time 1709785124.55 total 135.85 last 44.84) - open/close 7622 (time 1709785134.93 total 146.23 last 40.46) - open/close 8028 (time 1709785145.43 total 156.73 last 38.67) - open/close 8453 (time 1709785156.14 total 167.44 last 39.67) - open/close 8861 (time 1709785166.70 total 178.00 last 38.64) - open/close 9301 (time 1709785176.70 total 188.00 last 44.00) - open/close 9755 (time 1709785186.71 total 198.01 last 45.38) - open/close 10000 (time 1709785192.73 total 204.03 last 40.69) - open/close 10464 (time 1709785202.73 total 214.03 last 46.39) - open/close 10821 (time 1709785213.31 total 224.61 last 33.76) - open/close 11239 (time 1709785223.37 total 234.67 last 41.52) - open/close 13400 (time 1709785233.38 total 244.68 last 216.03) - open/close 15235 (time 1709785243.38 total 254.68 last 183.45) - open/close 16784 (time 1709785253.38 total 264.68 last 154.87) - open/close 18280 (time 1709785263.39 total 274.69 last 149.50) - open/close 19668 (time 1709785273.39 total 284.69 last 138.80) - open/close 20000 (time 1709785276.24 total 287.54 last 116.25) - open/close 21140 (time 1709785286.25 total 297.55 last 113.95) - open/close 22334 (time 1709785296.26 total 307.55 last 119.32) - open/close 23520 (time 1709785306.26 total 317.56 last 118.55) - open/close 24684 (time 1709785316.27 total 327.57 last 116.30) - open/close 25775 (time 1709785326.27 total 337.57 last 109.05) - open/close 26808 (time 1709785336.28 total 347.58 last 103.22) - open/close 27834 (time 1709785346.29 total 357.59 last 102.54) - open/close 28782 (time 1709785356.29 total 367.59 last 94.74) - open/close 29722 (time 1709785366.29 total 377.59 last 93.99) - open/close 30000 (time 1709785369.49 total 380.79 last 87.07) - open/close 30873 (time 1709785379.49 total 390.79 last 87.28) - open/close 31676 (time 1709785389.49 total 400.79 last 80.28) - open/close 32456 (time 1709785399.50 total 410.80 last 77.96) - open/close 33139 (time 1709785409.51 total 420.81 last 68.22) - open/close 33832 (time 1709785419.52 total 430.81 last 69.25) - open/close 34513 (time 1709785429.53 total 440.82 last 68.03) - open/close 35197 (time 1709785439.54 total 450.84 last 68.31) - open/close 35865 (time 1709785449.54 total 460.84 last 66.77) - open/close 36519 (time 1709785459.56 total 470.86 last 65.27) - open/close 37166 (time 1709785469.58 total 480.88 last 64.60) - open/close 37801 (time 1709785479.58 total 490.88 last 63.48) - open/close 38390 (time 1709785489.58 total 500.88 last 58.88) - open/close 38969 (time 1709785499.60 total 510.90 last 57.82) - open/close 39539 (time 1709785509.61 total 520.91 last 56.95) - open/close 40000 (time 1709785518.01 total 529.31 last 54.88) - open/close 40552 (time 1709785528.02 total 539.32 last 55.13) - open/close 41113 (time 1709785538.03 total 549.32 last 56.08) - open/close 41650 (time 1709785548.03 total 559.33 last 53.67) - open/close 42206 (time 1709785558.04 total 569.34 last 55.54) - open/close 42717 (time 1709785568.05 total 579.35 last 51.06) - open/close 43275 (time 1709785578.06 total 589.35 last 55.77) - open/close 43812 (time 1709785588.08 total 599.38 last 53.56) - open/close 44334 (time 1709785598.10 total 609.39 last 52.13) - open/close 44854 (time 1709785608.10 total 619.40 last 51.98) - open/close 45365 (time 1709785618.11 total 629.41 last 51.06) - open/close 45843 (time 1709785628.12 total 639.42 last 47.73) - open/close 46294 (time 1709785638.13 total 649.43 last 45.05) - open/close 46766 (time 1709785648.15 total 659.45 last 47.14) - open/close 47242 (time 1709785658.15 total 669.45 last 47.57) - open/close 47732 (time 1709785668.17 total 679.47 last 48.92) - open/close 48214 (time 1709785678.19 total 689.48 last 48.13) - open/close 48664 (time 1709785688.20 total 699.50 last 44.94) - open/close 49119 (time 1709785698.21 total 709.51 last 45.43) - open/close 49497 (time 1709785708.23 total 719.53 last 37.74) - open/close 49906 (time 1709785718.23 total 729.53 last 40.90) total: 50000 open/close in 731.76 seconds: 68.33 ops/second - unlinked 0 (time 1709785721 ; total 0 ; last 0) - unlinked 10000 (time 1709785965 ; total 244 ; last 244) - unlinked 20000 (time 1709786150 ; total 429 ; last 185) - unlinked 30000 (time 1709786272 ; total 551 ; last 122) - unlinked 40000 (time 1709786368 ; total 647 ; last 96) total: 50000 unlinks in 688 seconds: 72.674416 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg346-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg346-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg346-server stop mds service on oleg346-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg346-server unloading modules on: 'oleg346-server' oleg346-server: oleg346-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg346-client: oleg346-client: ssh exited with exit code 2 pdsh@oleg346-client: oleg346-server: ssh exited with exit code 2 pdsh@oleg346-client: oleg346-client: ssh exited with exit code 2 pdsh@oleg346-client: oleg346-server: ssh exited with exit code 2 PASS 101a (1450s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg346-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg346-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg346-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg346-client: oleg346-server: ssh exited with exit code 2 oleg346-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg346-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == conf-sanity test 101b: Race events DISCONNECT and ACTIVE in osp ========================================================== 23:40:26 (1709786426) start mds service on oleg346-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg346-server' oleg346-server: oleg346-server.virtnet: executing load_modules_local oleg346-server: Loading modules from /home/green/git/lustre-release/lustre oleg346-server: detected 4 online CPUs by sysfs oleg346-server: Force libcfs to create 2 CPU partitions oleg346-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg346-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg346-server: oleg346-server.virtnet: executing set_default_debug -1 all pdsh@oleg346-client: oleg346-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg346-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg346-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg346-server: oleg346-server.virtnet: executing set_default_debug -1 all pdsh@oleg346-client: oleg346-server: ssh exited with exit code 1 Started lustre-OST0000 oleg346-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg346-client.virtnet: -o user_xattr,flock oleg346-server@tcp:/lustre /mnt/lustre fail_loc=0x80002107 fail_val=20 stop ost1 service on oleg346-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg346-server start ost1 service on oleg346-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg346-server: oleg346-server.virtnet: executing set_default_debug -1 all pdsh@oleg346-client: oleg346-server: ssh exited with exit code 1 Started lustre-OST0000 oleg346-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 6 sec oleg346-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8800b624d800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b624d800.ost_server_uuid in FULL state after 0 sec umount lustre on /mnt/lustre..... Stopping client oleg346-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg346-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg346-server stop mds service on oleg346-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg346-server unloading modules on: 'oleg346-server' oleg346-server: oleg346-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg346-client: oleg346-client: ssh exited with exit code 2 pdsh@oleg346-client: oleg346-server: ssh exited with exit code 2 pdsh@oleg346-client: oleg346-client: ssh exited with exit code 2 pdsh@oleg346-client: oleg346-server: ssh exited with exit code 2 PASS 101b (82s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg346-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg346-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory Stopping clients: oleg346-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg346-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg346-client: oleg346-server: ssh exited with exit code 2 oleg346-server: oleg346-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg346-server' oleg346-server: oleg346-server.virtnet: executing load_modules_local oleg346-server: Loading modules from /home/green/git/lustre-release/lustre oleg346-server: detected 4 online CPUs by sysfs oleg346-server: Force libcfs to create 2 CPU partitions oleg346-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg346-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 1911 sec ========== 23:42:11 (1709786531) === conf-sanity: start cleanup 23:42:11 (1709786531) === === conf-sanity: finish cleanup 23:42:11 (1709786531) ===