-----============= acceptance-small: conf-sanity ============----- Wed Mar 6 23:43:31 EST 2024 excepting tests: 32newtarball 110 skipping tests SLOW=no: 45 69 106 111 114 Stopping clients: oleg141-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg141-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg141-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg141-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg141-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg141-server oleg141-server: oleg141-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg141-server' oleg141-server: oleg141-server.virtnet: executing load_modules_local oleg141-server: Loading modules from /home/green/git/lustre-release/lustre oleg141-server: detected 4 online CPUs by sysfs oleg141-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg141-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg141-server: oleg141-server.virtnet: executing set_default_debug -1 all pdsh@oleg141-client: oleg141-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg141-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg141-server: oleg141-server.virtnet: executing set_default_debug -1 all pdsh@oleg141-client: oleg141-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg141-server: oleg141-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg141-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg141-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg141-server stop mds service on oleg141-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg141-server debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 35b: Continue reconnection retries, if the active server is busy ========================================================== 23:44:41 (1709786681) start mds service on oleg141-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg141-server: oleg141-server.virtnet: executing set_default_debug -1 all pdsh@oleg141-client: oleg141-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg141-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg141-server: oleg141-server.virtnet: executing set_default_debug -1 all pdsh@oleg141-client: oleg141-server: ssh exited with exit code 1 Started lustre-OST0000 oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg141-client.virtnet: -o user_xattr,flock oleg141-server@tcp:/lustre /mnt/lustre debug=ha conf-sanity.sh test_35b 2024-03-0623h45m02s Set up a fake failnode for the MDS at_max=0 at_max=0 Injecting EBUSY on MDS fail_loc=0x80000136 mdc.lustre-MDT0000-mdc-ffff8800aadba800.stats=clear Creating a test file and stat it File: '/mnt/lustre/d35b.conf-sanity/f35b.conf-sanity' Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115238826934274 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-03-06 23:45:47.000000000 -0500 Modify: 2024-03-06 23:45:47.000000000 -0500 Change: 2024-03-06 23:45:47.000000000 -0500 Birth: - Stop injecting EBUSY on MDS fail_loc=0 done at_max=600 at_max=600 Debug log: 58 lines, 58 kept, 0 dropped, 0 bad. umount lustre on /mnt/lustre..... Stopping client oleg141-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg141-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg141-server stop mds service on oleg141-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg141-server unloading modules on: 'oleg141-server' oleg141-server: oleg141-server.virtnet: executing unload_modules_local modules unloaded. oleg141-server: oleg141-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre oleg141-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg141-client: oleg141-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg141-server: oleg141-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre oleg141-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg141-client: oleg141-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg141-server: oleg141-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre oleg141-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg141-client: oleg141-server: ssh exited with exit code 19 checking for existing Lustre data: not found tunefs failed, reformatting instead Stopping clients: oleg141-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg141-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg141-client: oleg141-server: ssh exited with exit code 2 oleg141-server: oleg141-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg141-server' oleg141-server: oleg141-server.virtnet: executing load_modules_local oleg141-server: Loading modules from /home/green/git/lustre-release/lustre oleg141-server: detected 4 online CPUs by sysfs oleg141-server: Force libcfs to create 2 CPU partitions oleg141-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg141-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg141-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg141-server: oleg141-server.virtnet: executing set_default_debug -1 all pdsh@oleg141-client: oleg141-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg141-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg141-server: oleg141-server.virtnet: executing set_default_debug -1 all pdsh@oleg141-client: oleg141-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg141-server: oleg141-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg141-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg141-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg141-server stop mds service on oleg141-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg141-server PASS 35b (169s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 23:47:34 (1709786854) start mds service on oleg141-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg141-server: oleg141-server.virtnet: executing set_default_debug -1 all pdsh@oleg141-client: oleg141-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg141-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg141-server: oleg141-server.virtnet: executing set_default_debug -1 all pdsh@oleg141-client: oleg141-server: ssh exited with exit code 1 Started lustre-OST0000 oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg141-client.virtnet: -o user_xattr,flock oleg141-server@tcp:/lustre /mnt/lustre copying 10 files to /mnt/lustre/d38.conf-sanity tar: Removing leading `/' from member names umount lustre on /mnt/lustre..... Stopping client oleg141-client.virtnet /mnt/lustre (opts:) osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=44 stop mds service on oleg141-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg141-server delete lov_objid file on MDS 000000 43 000008 start mds service on oleg141-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg141-server: oleg141-server.virtnet: executing set_default_debug -1 all pdsh@oleg141-client: oleg141-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg141-client.virtnet: -o user_xattr,flock oleg141-server@tcp:/lustre /mnt/lustre osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=2 umount lustre on /mnt/lustre..... Stopping client oleg141-client.virtnet /mnt/lustre (opts:) stop mds service on oleg141-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg141-server 000000 2 000008 oleg141-server: 8+0 records in oleg141-server: 8+0 records out oleg141-server: 4096 bytes (4.1 kB) copied, 0.000519666 s, 7.9 MB/s start mds service on oleg141-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg141-server: oleg141-server.virtnet: executing set_default_debug -1 all pdsh@oleg141-client: oleg141-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg141-client.virtnet: -o user_xattr,flock oleg141-server@tcp:/lustre /mnt/lustre osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=35 umount lustre on /mnt/lustre..... Stopping client oleg141-client.virtnet /mnt/lustre (opts:) stop mds service on oleg141-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg141-server 000000 34 000008 files compared the same umount lustre on /mnt/lustre..... stop ost1 service on oleg141-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg141-server stop mds service on oleg141-server unloading modules on: 'oleg141-server' oleg141-server: oleg141-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg141-client: oleg141-client: ssh exited with exit code 2 pdsh@oleg141-client: oleg141-server: ssh exited with exit code 2 pdsh@oleg141-client: oleg141-client: ssh exited with exit code 2 pdsh@oleg141-client: oleg141-server: ssh exited with exit code 2 PASS 38 (96s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg141-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg141-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg141-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg141-client: oleg141-server: ssh exited with exit code 2 oleg141-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg141-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 23:49:14 (1709786954) Stopping clients: oleg141-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg141-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg141-client: oleg141-server: ssh exited with exit code 2 oleg141-server: oleg141-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg141-server' oleg141-server: oleg141-server.virtnet: executing load_modules_local oleg141-server: Loading modules from /home/green/git/lustre-release/lustre oleg141-server: detected 4 online CPUs by sysfs oleg141-server: Force libcfs to create 2 CPU partitions oleg141-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg141-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg141-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg141-server: oleg141-server.virtnet: executing set_default_debug -1 all pdsh@oleg141-client: oleg141-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg141-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg141-server: oleg141-server.virtnet: executing set_default_debug -1 all pdsh@oleg141-client: oleg141-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg141-client.virtnet: -o user_xattr,flock oleg141-server@tcp:/lustre /mnt/lustre setup single mount lustre success fail_loc=0x142 start ost2 service on oleg141-server Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg141-server: oleg141-server.virtnet: executing set_default_debug -1 all pdsh@oleg141-client: oleg141-server: ssh exited with exit code 1 Commit the device label on lustre-ost2/ost2 Started lustre-OST0001 oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec stop ost2 service on oleg141-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg141-server umount lustre on /mnt/lustre..... Stopping client oleg141-client.virtnet /mnt/lustre (opts:-f) stop ost1 service on oleg141-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg141-server stop mds service on oleg141-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg141-server unloading modules on: 'oleg141-server' oleg141-server: oleg141-server.virtnet: executing unload_modules_local modules unloaded. oleg141-server: oleg141-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre oleg141-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg141-client: oleg141-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg141-server: oleg141-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre oleg141-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg141-client: oleg141-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg141-server: oleg141-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre oleg141-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg141-client: oleg141-server: ssh exited with exit code 19 checking for existing Lustre data: not found tunefs failed, reformatting instead Stopping clients: oleg141-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg141-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg141-client: oleg141-server: ssh exited with exit code 2 oleg141-server: oleg141-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg141-server' oleg141-server: oleg141-server.virtnet: executing load_modules_local oleg141-server: Loading modules from /home/green/git/lustre-release/lustre oleg141-server: detected 4 online CPUs by sysfs oleg141-server: Force libcfs to create 2 CPU partitions oleg141-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg141-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg141-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg141-server: oleg141-server.virtnet: executing set_default_debug -1 all pdsh@oleg141-client: oleg141-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg141-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg141-server: oleg141-server.virtnet: executing set_default_debug -1 all pdsh@oleg141-client: oleg141-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg141-server: oleg141-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg141-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg141-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg141-server stop mds service on oleg141-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg141-server PASS 51 (192s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:52:31 (1709787151) start mds service on oleg141-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg141-server: oleg141-server.virtnet: executing set_default_debug -1 all pdsh@oleg141-client: oleg141-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg141-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg141-server: oleg141-server.virtnet: executing set_default_debug -1 all pdsh@oleg141-client: oleg141-server: ssh exited with exit code 1 Started lustre-OST0000 oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg141-client.virtnet: -o user_xattr,flock oleg141-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 188 (time 1709787185.05 total 10.50 last 17.90) - open/close 391 (time 1709787195.06 total 20.51 last 20.28) - open/close 565 (time 1709787205.69 total 31.14 last 16.38) - open/close 765 (time 1709787215.72 total 41.18 last 19.93) - open/close 986 (time 1709787226.44 total 51.89 last 20.62) - open/close 1237 (time 1709787237.39 total 62.84 last 22.94) - open/close 1414 (time 1709787247.39 total 72.84 last 17.69) - open/close 1590 (time 1709787257.80 total 83.25 last 16.91) - open/close 1817 (time 1709787268.79 total 94.24 last 20.65) - open/close 1956 (time 1709787279.07 total 104.52 last 13.52) - open/close 2170 (time 1709787289.11 total 114.56 last 21.32) - open/close 2356 (time 1709787299.11 total 124.57 last 18.59) - open/close 2593 (time 1709787309.45 total 134.90 last 22.93) - open/close 2792 (time 1709787319.45 total 144.91 last 19.89) - open/close 3003 (time 1709787329.84 total 155.29 last 20.31) - open/close 3183 (time 1709787340.79 total 166.25 last 16.43) - open/close 3391 (time 1709787351.77 total 177.22 last 18.95) - open/close 3518 (time 1709787361.81 total 187.26 last 12.65) - open/close 3695 (time 1709787371.81 total 197.26 last 17.69) - open/close 3883 (time 1709787381.98 total 207.43 last 18.50) - open/close 4089 (time 1709787392.14 total 217.59 last 20.27) - open/close 4276 (time 1709787402.84 total 228.29 last 17.47) - open/close 4440 (time 1709787412.85 total 238.30 last 16.39) - open/close 4546 (time 1709787423.08 total 248.53 last 10.36) - open/close 4728 (time 1709787433.26 total 258.72 last 17.86) - open/close 4840 (time 1709787443.27 total 268.72 last 11.20) - open/close 5040 (time 1709787453.28 total 278.73 last 19.98) - open/close 5292 (time 1709787463.34 total 288.79 last 25.05) - open/close 5413 (time 1709787473.36 total 298.81 last 12.07) - open/close 5545 (time 1709787483.37 total 308.82 last 13.19) - open/close 5757 (time 1709787494.36 total 319.81 last 19.28) - open/close 5893 (time 1709787504.38 total 329.83 last 13.57) - open/close 6051 (time 1709787514.62 total 340.07 last 15.44) - open/close 6238 (time 1709787525.10 total 350.55 last 17.85) - open/close 6372 (time 1709787535.50 total 360.95 last 12.88) - open/close 6570 (time 1709787546.22 total 371.67 last 18.47) - open/close 6887 (time 1709787556.22 total 381.67 last 31.69) - open/close 7073 (time 1709787567.18 total 392.63 last 16.97) - open/close 7211 (time 1709787578.09 total 403.54 last 12.65) - open/close 7412 (time 1709787588.26 total 413.71 last 19.76) - open/close 7629 (time 1709787598.27 total 423.72 last 21.68) - open/close 7802 (time 1709787608.28 total 433.73 last 17.29) - open/close 8038 (time 1709787618.74 total 444.19 last 22.57) - open/close 8268 (time 1709787629.50 total 454.95 last 21.38) - open/close 8515 (time 1709787639.74 total 465.19 last 24.11) - open/close 8648 (time 1709787650.66 total 476.11 last 12.18) - open/close 8817 (time 1709787670.29 total 495.75 last 8.61) - open/close 8999 (time 1709787680.98 total 506.43 last 17.03) - open/close 9142 (time 1709787691.71 total 517.17 last 13.32) - open/close 9281 (time 1709787701.86 total 527.32 last 13.69) - open/close 9515 (time 1709787712.82 total 538.27 last 21.36) - open/close 9770 (time 1709787723.76 total 549.21 last 23.30) - open/close 9984 (time 1709787733.77 total 559.22 last 21.39) - open/close 10102 (time 1709787744.34 total 569.79 last 11.17) - open/close 10617 (time 1709787754.35 total 579.80 last 51.45) - open/close 11179 (time 1709787764.35 total 589.80 last 56.17) - open/close 11748 (time 1709787774.36 total 599.81 last 56.86) - open/close 12298 (time 1709787784.36 total 609.81 last 55.00) - open/close 12755 (time 1709787794.37 total 619.82 last 45.68) - open/close 13236 (time 1709787804.38 total 629.84 last 48.01) - open/close 13764 (time 1709787814.39 total 639.84 last 52.79) - open/close 14240 (time 1709787824.41 total 649.86 last 47.50) - open/close 14687 (time 1709787834.41 total 659.87 last 44.67) - open/close 15098 (time 1709787844.43 total 669.88 last 41.05) - open/close 15469 (time 1709787854.44 total 679.89 last 37.03) - open/close 15885 (time 1709787864.47 total 689.93 last 41.47) - open/close 16288 (time 1709787874.48 total 699.93 last 40.26) - open/close 16686 (time 1709787884.51 total 709.96 last 39.70) - open/close 17058 (time 1709787894.52 total 719.97 last 37.16) - open/close 17440 (time 1709787904.52 total 729.98 last 38.19) - open/close 17826 (time 1709787914.55 total 740.00 last 38.49) - open/close 18181 (time 1709787924.56 total 750.01 last 35.48) - open/close 18530 (time 1709787934.58 total 760.03 last 34.81) - open/close 18899 (time 1709787944.62 total 770.07 last 36.76) - open/close 19307 (time 1709787954.64 total 780.09 last 40.72) - open/close 19741 (time 1709787964.68 total 790.13 last 43.22) - open/close 20000 (time 1709787971.73 total 797.18 last 36.78) - open/close 20427 (time 1709787981.75 total 807.20 last 42.60) - open/close 20861 (time 1709787991.77 total 817.22 last 43.32) - open/close 21272 (time 1709788001.78 total 827.23 last 41.05) - open/close 21678 (time 1709788011.78 total 837.23 last 40.60) - open/close 22048 (time 1709788021.78 total 847.23 last 36.99) - open/close 22399 (time 1709788031.79 total 857.25 last 35.05) - open/close 22774 (time 1709788041.82 total 867.27 last 37.40) - open/close 23159 (time 1709788051.83 total 877.29 last 38.45) - open/close 23544 (time 1709788061.84 total 887.29 last 38.47) - open/close 23917 (time 1709788071.85 total 897.30 last 37.26) - open/close 24320 (time 1709788081.86 total 907.31 last 40.28) - open/close 24693 (time 1709788091.87 total 917.32 last 37.26) - open/close 25066 (time 1709788101.89 total 927.34 last 37.23) - open/close 25396 (time 1709788111.91 total 937.36 last 32.93) - open/close 25757 (time 1709788121.92 total 947.37 last 36.05) - open/close 26081 (time 1709788131.94 total 957.39 last 32.33) - open/close 26428 (time 1709788141.97 total 967.42 last 34.62) - open/close 26762 (time 1709788151.98 total 977.43 last 33.37) - open/close 27081 (time 1709788161.98 total 987.44 last 31.87) - open/close 27401 (time 1709788172.01 total 997.46 last 31.93) - open/close 27729 (time 1709788182.02 total 1007.47 last 32.74) - open/close 28052 (time 1709788192.04 total 1017.49 last 32.25) - open/close 28395 (time 1709788202.05 total 1027.50 last 34.26) - open/close 28757 (time 1709788212.06 total 1037.51 last 36.19) - open/close 29098 (time 1709788222.06 total 1047.52 last 34.07) - open/close 29440 (time 1709788232.08 total 1057.53 last 34.15) - open/close 29768 (time 1709788242.10 total 1067.55 last 32.73) - open/close 30000 (time 1709788249.76 total 1075.21 last 30.29) - open/close 30271 (time 1709788259.78 total 1085.23 last 27.06) - open/close 30561 (time 1709788269.79 total 1095.24 last 28.97) - open/close 30868 (time 1709788279.82 total 1105.27 last 30.60) - open/close 31197 (time 1709788289.83 total 1115.28 last 32.85) - open/close 31514 (time 1709788299.86 total 1125.31 last 31.61) - open/close 31828 (time 1709788309.88 total 1135.34 last 31.33) - open/close 32126 (time 1709788319.92 total 1145.37 last 29.70) - open/close 32432 (time 1709788329.95 total 1155.40 last 30.49) - open/close 32724 (time 1709788339.97 total 1165.42 last 29.16) - open/close 33015 (time 1709788349.98 total 1175.43 last 29.07) - open/close 33322 (time 1709788360.00 total 1185.45 last 30.64) - open/close 33619 (time 1709788370.01 total 1195.46 last 29.67) - open/close 33918 (time 1709788380.01 total 1205.46 last 29.89) - open/close 34207 (time 1709788390.03 total 1215.48 last 28.84) - open/close 34494 (time 1709788400.06 total 1225.51 last 28.62) - open/close 34789 (time 1709788410.08 total 1235.53 last 29.46) - open/close 35074 (time 1709788420.08 total 1245.54 last 28.47) - open/close 35344 (time 1709788430.11 total 1255.56 last 26.92) - open/close 35627 (time 1709788440.16 total 1265.61 last 28.17) - open/close 35882 (time 1709788450.20 total 1275.65 last 25.39) - open/close 36152 (time 1709788460.24 total 1285.69 last 26.89) - open/close 36409 (time 1709788470.24 total 1295.69 last 25.70) - open/close 36699 (time 1709788480.24 total 1305.69 last 29.00) - open/close 36970 (time 1709788490.26 total 1315.71 last 27.06) - open/close 37250 (time 1709788500.29 total 1325.74 last 27.91) - open/close 37530 (time 1709788510.31 total 1335.76 last 27.94) - open/close 37807 (time 1709788520.32 total 1345.77 last 27.67) - open/close 38075 (time 1709788530.33 total 1355.78 last 26.78) - open/close 38356 (time 1709788540.34 total 1365.79 last 28.07) - open/close 38638 (time 1709788550.37 total 1375.82 last 28.11) - open/close 38915 (time 1709788560.39 total 1385.84 last 27.66) - open/close 39194 (time 1709788570.42 total 1395.87 last 27.80) - open/close 39444 (time 1709788580.44 total 1405.90 last 24.95) - open/close 39711 (time 1709788590.45 total 1415.90 last 26.68) - open/close 39977 (time 1709788600.47 total 1425.92 last 26.55) - open/close 40243 (time 1709788610.49 total 1435.94 last 26.55) - open/close 40512 (time 1709788620.52 total 1445.97 last 26.81) - open/close 40789 (time 1709788630.54 total 1456.00 last 27.64) - open/close 41034 (time 1709788640.56 total 1466.01 last 24.46) - open/close 41286 (time 1709788650.59 total 1476.04 last 25.14) - open/close 41532 (time 1709788660.60 total 1486.05 last 24.57) - open/close 41785 (time 1709788670.64 total 1496.09 last 25.20) - open/close 42018 (time 1709788680.68 total 1506.13 last 23.20) - open/close 42249 (time 1709788690.71 total 1516.17 last 23.03) - open/close 42491 (time 1709788700.72 total 1526.17 last 24.19) - open/close 42741 (time 1709788710.74 total 1536.19 last 24.95) - open/close 42984 (time 1709788720.76 total 1546.21 last 24.26) - open/close 43228 (time 1709788730.78 total 1556.23 last 24.35) - open/close 43464 (time 1709788740.82 total 1566.27 last 23.50) - open/close 43695 (time 1709788750.82 total 1576.27 last 23.10) - open/close 43915 (time 1709788760.83 total 1586.28 last 21.98) - open/close 44136 (time 1709788770.84 total 1596.29 last 22.07) - open/close 44362 (time 1709788780.86 total 1606.31 last 22.57) - open/close 44596 (time 1709788790.88 total 1616.33 last 23.35) - open/close 44819 (time 1709788800.88 total 1626.34 last 22.28) - open/close 45050 (time 1709788810.89 total 1636.34 last 23.08) - open/close 45279 (time 1709788820.93 total 1646.38 last 22.82) - open/close 45512 (time 1709788830.96 total 1656.41 last 23.23) - open/close 45740 (time 1709788841.00 total 1666.45 last 22.71) - open/close 45968 (time 1709788851.02 total 1676.47 last 22.76) - open/close 46182 (time 1709788861.04 total 1686.49 last 21.36) - open/close 46395 (time 1709788871.06 total 1696.51 last 21.25) - open/close 46607 (time 1709788881.06 total 1706.51 last 21.20) - open/close 46810 (time 1709788891.09 total 1716.54 last 20.23) - open/close 47026 (time 1709788901.11 total 1726.56 last 21.57) - open/close 47235 (time 1709788911.12 total 1736.57 last 20.88) - open/close 47445 (time 1709788921.19 total 1746.64 last 20.86) - open/close 47654 (time 1709788931.19 total 1756.64 last 20.89) - open/close 47869 (time 1709788941.21 total 1766.66 last 21.46) - open/close 48077 (time 1709788951.23 total 1776.68 last 20.75) - open/close 48282 (time 1709788961.28 total 1786.73 last 20.41) - open/close 48488 (time 1709788971.32 total 1796.77 last 20.51) - open/close 48698 (time 1709788981.34 total 1806.79 last 20.96) - open/close 48899 (time 1709788991.39 total 1816.84 last 20.00) - open/close 49098 (time 1709789001.40 total 1826.85 last 19.87) - open/close 49301 (time 1709789011.41 total 1836.86 last 20.28) - open/close 49512 (time 1709789021.44 total 1846.89 last 21.04) - open/close 49727 (time 1709789031.45 total 1856.91 last 21.47) - open/close 49938 (time 1709789041.47 total 1866.92 last 21.06) total: 50000 open/close in 1869.90 seconds: 26.74 ops/second - unlinked 0 (time 1709789045 ; total 0 ; last 0) - unlinked 10000 (time 1709789623 ; total 578 ; last 578) - unlinked 20000 (time 1709790088 ; total 1043 ; last 465) - unlinked 30000 (time 1709790398 ; total 1353 ; last 310) - unlinked 40000 (time 1709790544 ; total 1499 ; last 146) total: 50000 unlinks in 1541 seconds: 32.446465 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg141-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg141-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg141-server stop mds service on oleg141-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg141-server unloading modules on: 'oleg141-server' oleg141-server: oleg141-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg141-client: oleg141-client: ssh exited with exit code 2 pdsh@oleg141-client: oleg141-server: ssh exited with exit code 2 pdsh@oleg141-client: oleg141-client: ssh exited with exit code 2 pdsh@oleg141-client: oleg141-server: ssh exited with exit code 2 PASS 101a (3446s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg141-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg141-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg141-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg141-client: oleg141-server: ssh exited with exit code 2 oleg141-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg141-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == conf-sanity test 101b: Race events DISCONNECT and ACTIVE in osp ========================================================== 00:49:59 (1709790599) start mds service on oleg141-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg141-server' oleg141-server: oleg141-server.virtnet: executing load_modules_local oleg141-server: Loading modules from /home/green/git/lustre-release/lustre oleg141-server: detected 4 online CPUs by sysfs oleg141-server: Force libcfs to create 2 CPU partitions oleg141-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg141-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg141-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg141-server: oleg141-server.virtnet: executing set_default_debug -1 all pdsh@oleg141-client: oleg141-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg141-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg141-server: oleg141-server.virtnet: executing set_default_debug -1 all pdsh@oleg141-client: oleg141-server: ssh exited with exit code 1 Started lustre-OST0000 oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg141-client.virtnet: -o user_xattr,flock oleg141-server@tcp:/lustre /mnt/lustre fail_loc=0x80002107 fail_val=20 stop ost1 service on oleg141-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg141-server start ost1 service on oleg141-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg141-server: oleg141-server.virtnet: executing set_default_debug -1 all pdsh@oleg141-client: oleg141-server: ssh exited with exit code 1 Started lustre-OST0000 oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 12 sec oleg141-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8800af0ac000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800af0ac000.ost_server_uuid in FULL state after 0 sec umount lustre on /mnt/lustre..... Stopping client oleg141-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg141-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg141-server stop mds service on oleg141-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg141-server unloading modules on: 'oleg141-server' oleg141-server: oleg141-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg141-client: oleg141-client: ssh exited with exit code 2 pdsh@oleg141-client: oleg141-server: ssh exited with exit code 2 pdsh@oleg141-client: oleg141-client: ssh exited with exit code 2 pdsh@oleg141-client: oleg141-server: ssh exited with exit code 2 PASS 101b (85s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg141-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg141-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory Stopping clients: oleg141-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg141-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg141-client: oleg141-server: ssh exited with exit code 2 oleg141-server: oleg141-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg141-server' oleg141-server: oleg141-server.virtnet: executing load_modules_local oleg141-server: Loading modules from /home/green/git/lustre-release/lustre oleg141-server: detected 4 online CPUs by sysfs oleg141-server: Force libcfs to create 2 CPU partitions oleg141-server: libkmod: kmod_module_get_refcnt: could not read integer from '/sys/module/acpi_cpufreq/refcnt': 'No such device' oleg141-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg141-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 4094 sec ========== 00:51:46 (1709790706) === conf-sanity: start cleanup 00:51:47 (1709790707) === === conf-sanity: finish cleanup 00:51:47 (1709790707) ===