-----============= acceptance-small: conf-sanity ============----- Wed Mar 6 23:42:56 EST 2024 excepting tests: 32newtarball 110 skipping tests SLOW=no: 45 69 106 111 114 Stopping clients: oleg129-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg129-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg129-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg129-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg129-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg129-server oleg129-server: oleg129-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg129-server' oleg129-server: oleg129-server.virtnet: executing load_modules_local oleg129-server: Loading modules from /home/green/git/lustre-release/lustre oleg129-server: detected 4 online CPUs by sysfs oleg129-server: Force libcfs to create 2 CPU partitions oleg129-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg129-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg129-server: oleg129-server.virtnet: executing set_default_debug -1 all pdsh@oleg129-client: oleg129-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg129-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg129-server: oleg129-server.virtnet: executing set_default_debug -1 all pdsh@oleg129-client: oleg129-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg129-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg129-server: oleg129-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg129-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg129-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg129-server stop mds service on oleg129-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg129-server debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 35b: Continue reconnection retries, if the active server is busy ========================================================== 23:44:30 (1709786670) start mds service on oleg129-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg129-server: oleg129-server.virtnet: executing set_default_debug -1 all pdsh@oleg129-client: oleg129-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg129-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg129-server: oleg129-server.virtnet: executing set_default_debug -1 all pdsh@oleg129-client: oleg129-server: ssh exited with exit code 1 Started lustre-OST0000 oleg129-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg129-client.virtnet: -o user_xattr,flock oleg129-server@tcp:/lustre /mnt/lustre debug=ha conf-sanity.sh test_35b 2024-03-0623h44m44s Set up a fake failnode for the MDS at_max=0 at_max=0 Injecting EBUSY on MDS fail_loc=0x80000136 mdc.lustre-MDT0000-mdc-ffff88012a52c800.stats=clear Creating a test file and stat it File: '/mnt/lustre/d35b.conf-sanity/f35b.conf-sanity' Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115238826934274 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-03-06 23:45:30.000000000 -0500 Modify: 2024-03-06 23:45:30.000000000 -0500 Change: 2024-03-06 23:45:30.000000000 -0500 Birth: - Stop injecting EBUSY on MDS fail_loc=0 done at_max=600 at_max=600 Debug log: 58 lines, 58 kept, 0 dropped, 0 bad. umount lustre on /mnt/lustre..... Stopping client oleg129-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg129-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg129-server stop mds service on oleg129-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg129-server unloading modules on: 'oleg129-server' oleg129-server: oleg129-server.virtnet: executing unload_modules_local modules unloaded. oleg129-server: oleg129-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre oleg129-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg129-client: oleg129-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg129-server: oleg129-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre oleg129-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg129-client: oleg129-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg129-server: oleg129-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre oleg129-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg129-client: oleg129-server: ssh exited with exit code 19 checking for existing Lustre data: not found tunefs failed, reformatting instead Stopping clients: oleg129-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg129-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg129-client: oleg129-server: ssh exited with exit code 2 oleg129-server: oleg129-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg129-server' oleg129-server: oleg129-server.virtnet: executing load_modules_local oleg129-server: Loading modules from /home/green/git/lustre-release/lustre oleg129-server: detected 4 online CPUs by sysfs oleg129-server: Force libcfs to create 2 CPU partitions oleg129-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg129-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg129-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg129-server: oleg129-server.virtnet: executing set_default_debug -1 all pdsh@oleg129-client: oleg129-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg129-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg129-server: oleg129-server.virtnet: executing set_default_debug -1 all pdsh@oleg129-client: oleg129-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg129-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg129-server: oleg129-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg129-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg129-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg129-server stop mds service on oleg129-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg129-server PASS 35b (156s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 23:47:10 (1709786830) start mds service on oleg129-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg129-server: oleg129-server.virtnet: executing set_default_debug -1 all pdsh@oleg129-client: oleg129-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg129-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg129-server: oleg129-server.virtnet: executing set_default_debug -1 all pdsh@oleg129-client: oleg129-server: ssh exited with exit code 1 Started lustre-OST0000 oleg129-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg129-client.virtnet: -o user_xattr,flock oleg129-server@tcp:/lustre /mnt/lustre copying 10 files to /mnt/lustre/d38.conf-sanity tar: Removing leading `/' from member names umount lustre on /mnt/lustre..... Stopping client oleg129-client.virtnet /mnt/lustre (opts:) osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=44 stop mds service on oleg129-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg129-server delete lov_objid file on MDS 000000 43 000008 start mds service on oleg129-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg129-server: oleg129-server.virtnet: executing set_default_debug -1 all pdsh@oleg129-client: oleg129-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg129-client.virtnet: -o user_xattr,flock oleg129-server@tcp:/lustre /mnt/lustre osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=2 umount lustre on /mnt/lustre..... Stopping client oleg129-client.virtnet /mnt/lustre (opts:) stop mds service on oleg129-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg129-server 000000 2 000008 oleg129-server: 8+0 records in oleg129-server: 8+0 records out oleg129-server: 4096 bytes (4.1 kB) copied, 0.00287162 s, 1.4 MB/s start mds service on oleg129-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg129-server: oleg129-server.virtnet: executing set_default_debug -1 all pdsh@oleg129-client: oleg129-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg129-client.virtnet: -o user_xattr,flock oleg129-server@tcp:/lustre /mnt/lustre osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=35 umount lustre on /mnt/lustre..... Stopping client oleg129-client.virtnet /mnt/lustre (opts:) stop mds service on oleg129-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg129-server 000000 34 000008 files compared the same umount lustre on /mnt/lustre..... stop ost1 service on oleg129-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg129-server stop mds service on oleg129-server unloading modules on: 'oleg129-server' oleg129-server: oleg129-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg129-client: oleg129-client: ssh exited with exit code 2 pdsh@oleg129-client: oleg129-server: ssh exited with exit code 2 pdsh@oleg129-client: oleg129-client: ssh exited with exit code 2 pdsh@oleg129-client: oleg129-server: ssh exited with exit code 2 PASS 38 (100s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg129-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg129-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg129-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg129-client: oleg129-server: ssh exited with exit code 2 oleg129-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg129-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 23:48:53 (1709786933) Stopping clients: oleg129-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg129-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg129-client: oleg129-server: ssh exited with exit code 2 oleg129-server: oleg129-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg129-server' oleg129-server: oleg129-server.virtnet: executing load_modules_local oleg129-server: Loading modules from /home/green/git/lustre-release/lustre oleg129-server: detected 4 online CPUs by sysfs oleg129-server: Force libcfs to create 2 CPU partitions oleg129-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg129-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg129-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg129-server: oleg129-server.virtnet: executing set_default_debug -1 all pdsh@oleg129-client: oleg129-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg129-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg129-server: oleg129-server.virtnet: executing set_default_debug -1 all pdsh@oleg129-client: oleg129-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg129-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg129-client.virtnet: -o user_xattr,flock oleg129-server@tcp:/lustre /mnt/lustre setup single mount lustre success fail_loc=0x142 start ost2 service on oleg129-server Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg129-server: oleg129-server.virtnet: executing set_default_debug -1 all pdsh@oleg129-client: oleg129-server: ssh exited with exit code 1 Commit the device label on lustre-ost2/ost2 Started lustre-OST0001 oleg129-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec stop ost2 service on oleg129-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg129-server umount lustre on /mnt/lustre..... Stopping client oleg129-client.virtnet /mnt/lustre (opts:-f) stop ost1 service on oleg129-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg129-server stop mds service on oleg129-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg129-server unloading modules on: 'oleg129-server' oleg129-server: oleg129-server.virtnet: executing unload_modules_local modules unloaded. oleg129-server: oleg129-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre oleg129-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg129-client: oleg129-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg129-server: oleg129-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre oleg129-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg129-client: oleg129-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg129-server: oleg129-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre oleg129-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg129-client: oleg129-server: ssh exited with exit code 19 checking for existing Lustre data: not found tunefs failed, reformatting instead Stopping clients: oleg129-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg129-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg129-client: oleg129-server: ssh exited with exit code 2 oleg129-server: oleg129-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg129-server' oleg129-server: oleg129-server.virtnet: executing load_modules_local oleg129-server: Loading modules from /home/green/git/lustre-release/lustre oleg129-server: detected 4 online CPUs by sysfs oleg129-server: Force libcfs to create 2 CPU partitions oleg129-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg129-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg129-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg129-server: oleg129-server.virtnet: executing set_default_debug -1 all pdsh@oleg129-client: oleg129-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg129-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg129-server: oleg129-server.virtnet: executing set_default_debug -1 all pdsh@oleg129-client: oleg129-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg129-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg129-server: oleg129-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg129-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg129-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg129-server stop mds service on oleg129-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg129-server PASS 51 (194s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:52:11 (1709787131) start mds service on oleg129-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg129-server: oleg129-server.virtnet: executing set_default_debug -1 all pdsh@oleg129-client: oleg129-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg129-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg129-server: oleg129-server.virtnet: executing set_default_debug -1 all pdsh@oleg129-client: oleg129-server: ssh exited with exit code 1 Started lustre-OST0000 oleg129-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg129-client.virtnet: -o user_xattr,flock oleg129-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 152 (time 1709787166.24 total 10.48 last 14.50) - open/close 369 (time 1709787176.98 total 21.22 last 20.20) - open/close 527 (time 1709787187.45 total 31.69 last 15.09) - open/close 659 (time 1709787197.45 total 41.69 last 13.20) - open/close 856 (time 1709787207.46 total 51.70 last 19.69) - open/close 1070 (time 1709787217.46 total 61.70 last 21.40) - open/close 1310 (time 1709787227.74 total 71.98 last 23.34) - open/close 1481 (time 1709787237.74 total 81.98 last 17.09) - open/close 1743 (time 1709787248.51 total 92.75 last 24.33) - open/close 1863 (time 1709787259.33 total 103.57 last 11.09) - open/close 1966 (time 1709787269.97 total 114.21 last 9.68) - open/close 2107 (time 1709787279.98 total 124.22 last 14.08) - open/close 2250 (time 1709787292.21 total 136.45 last 11.69) - open/close 2483 (time 1709787302.82 total 147.06 last 21.96) - open/close 2641 (time 1709787313.43 total 157.68 last 14.88) - open/close 2748 (time 1709787323.51 total 167.76 last 10.62) - open/close 2881 (time 1709787334.08 total 178.32 last 12.59) - open/close 3089 (time 1709787344.14 total 188.38 last 20.67) - open/close 3281 (time 1709787354.28 total 198.53 last 18.93) - open/close 3439 (time 1709787364.93 total 209.17 last 14.85) - open/close 3668 (time 1709787375.62 total 219.86 last 21.41) - open/close 3918 (time 1709787385.63 total 229.87 last 24.98) - open/close 4075 (time 1709787395.65 total 239.89 last 15.67) - open/close 4312 (time 1709787406.57 total 250.81 last 21.71) - open/close 4492 (time 1709787416.96 total 261.20 last 17.32) - open/close 4747 (time 1709787427.79 total 272.03 last 23.54) - open/close 4893 (time 1709787437.79 total 282.03 last 14.60) - open/close 5091 (time 1709787447.81 total 292.05 last 19.76) - open/close 5299 (time 1709787457.83 total 302.07 last 20.75) - open/close 5451 (time 1709787467.85 total 312.09 last 15.17) - open/close 5593 (time 1709787478.01 total 322.25 last 13.98) - open/close 5750 (time 1709787488.82 total 333.06 last 14.52) - open/close 5971 (time 1709787498.83 total 343.07 last 22.07) - open/close 6061 (time 1709787508.96 total 353.20 last 8.88) - open/close 6295 (time 1709787519.73 total 363.97 last 21.74) - open/close 6438 (time 1709787529.73 total 373.97 last 14.29) - open/close 6662 (time 1709787540.38 total 384.63 last 21.03) - open/close 6869 (time 1709787550.39 total 394.63 last 20.68) - open/close 7088 (time 1709787560.82 total 405.06 last 21.00) - open/close 7347 (time 1709787571.20 total 415.44 last 24.95) - open/close 7448 (time 1709787581.20 total 425.44 last 10.10) - open/close 7601 (time 1709787591.21 total 435.45 last 15.29) - open/close 7740 (time 1709787601.22 total 445.46 last 13.89) - open/close 7907 (time 1709787612.21 total 456.45 last 15.19) - open/close 7970 (time 1709787622.84 total 467.09 last 5.93) - open/close 8123 (time 1709787633.07 total 477.32 last 14.96) - open/close 8231 (time 1709787643.08 total 487.32 last 10.80) - open/close 8467 (time 1709787653.47 total 497.71 last 22.72) - open/close 8647 (time 1709787663.48 total 507.72 last 17.97) - open/close 8741 (time 1709787673.49 total 517.73 last 9.39) - open/close 8874 (time 1709787684.04 total 528.28 last 12.61) - open/close 9003 (time 1709787694.87 total 539.11 last 11.91) - open/close 9252 (time 1709787705.80 total 550.04 last 22.78) - open/close 9365 (time 1709787715.80 total 560.05 last 11.30) - open/close 9558 (time 1709787726.04 total 570.28 last 18.86) - open/close 10000 (time 1709787734.99 total 579.24 last 49.34) - open/close 10577 (time 1709787745.01 total 589.26 last 57.58) - open/close 11102 (time 1709787755.02 total 599.27 last 52.45) - open/close 11616 (time 1709787765.03 total 609.27 last 51.36) - open/close 12152 (time 1709787775.07 total 619.31 last 53.42) - open/close 12651 (time 1709787785.07 total 629.32 last 49.86) - open/close 13101 (time 1709787795.10 total 639.34 last 44.88) - open/close 13611 (time 1709787805.11 total 649.36 last 50.94) - open/close 14093 (time 1709787815.13 total 659.37 last 48.14) - open/close 14589 (time 1709787825.15 total 669.39 last 49.47) - open/close 15026 (time 1709787835.16 total 679.41 last 43.65) - open/close 15413 (time 1709787845.17 total 689.41 last 38.70) - open/close 15836 (time 1709787855.17 total 699.41 last 42.27) - open/close 16276 (time 1709787865.21 total 709.45 last 43.84) - open/close 16674 (time 1709787875.22 total 719.46 last 39.75) - open/close 17072 (time 1709787885.24 total 729.48 last 39.72) - open/close 17477 (time 1709787895.25 total 739.49 last 40.48) - open/close 17862 (time 1709787905.27 total 749.51 last 38.42) - open/close 18222 (time 1709787915.29 total 759.53 last 35.91) - open/close 18574 (time 1709787925.29 total 769.54 last 35.19) - open/close 18913 (time 1709787935.34 total 779.58 last 33.74) - open/close 19264 (time 1709787945.35 total 789.60 last 35.06) - open/close 19672 (time 1709787955.36 total 799.60 last 40.78) - open/close 20000 (time 1709787963.49 total 807.73 last 40.36) - open/close 20372 (time 1709787973.50 total 817.74 last 37.14) - open/close 20820 (time 1709787983.52 total 827.76 last 44.72) - open/close 21223 (time 1709787993.52 total 837.77 last 40.28) - open/close 21596 (time 1709788003.53 total 847.77 last 37.29) - open/close 22000 (time 1709788013.55 total 857.79 last 40.31) - open/close 22352 (time 1709788023.57 total 867.81 last 35.12) - open/close 22711 (time 1709788033.61 total 877.85 last 35.77) - open/close 23107 (time 1709788043.61 total 887.86 last 39.57) - open/close 23476 (time 1709788053.63 total 897.87 last 36.84) - open/close 23854 (time 1709788063.64 total 907.88 last 37.77) - open/close 24213 (time 1709788073.66 total 917.90 last 35.83) - open/close 24610 (time 1709788083.66 total 927.91 last 39.67) - open/close 24983 (time 1709788093.68 total 937.92 last 37.23) - open/close 25337 (time 1709788103.71 total 947.95 last 35.31) - open/close 25685 (time 1709788113.75 total 958.00 last 34.64) - open/close 26025 (time 1709788123.77 total 968.01 last 33.96) - open/close 26359 (time 1709788133.77 total 978.01 last 33.39) - open/close 26716 (time 1709788143.78 total 988.02 last 35.67) - open/close 27022 (time 1709788153.78 total 998.02 last 30.59) - open/close 27333 (time 1709788163.79 total 1008.03 last 31.08) - open/close 27650 (time 1709788173.79 total 1018.03 last 31.69) - open/close 27975 (time 1709788183.80 total 1028.04 last 32.45) - open/close 28296 (time 1709788193.82 total 1038.07 last 32.03) - open/close 28649 (time 1709788203.83 total 1048.07 last 35.29) - open/close 29004 (time 1709788213.84 total 1058.08 last 35.44) - open/close 29331 (time 1709788223.84 total 1068.09 last 32.70) - open/close 29677 (time 1709788233.85 total 1078.10 last 34.57) - open/close 29997 (time 1709788243.87 total 1088.11 last 31.95) - open/close 30290 (time 1709788253.88 total 1098.12 last 29.26) - open/close 30577 (time 1709788263.89 total 1108.13 last 28.68) - open/close 30882 (time 1709788273.90 total 1118.14 last 30.47) - open/close 31183 (time 1709788283.90 total 1128.15 last 30.08) - open/close 31509 (time 1709788293.93 total 1138.17 last 32.52) - open/close 31827 (time 1709788303.93 total 1148.17 last 31.80) - open/close 32134 (time 1709788313.94 total 1158.18 last 30.66) - open/close 32441 (time 1709788323.97 total 1168.21 last 30.61) - open/close 32746 (time 1709788333.98 total 1178.23 last 30.46) - open/close 33034 (time 1709788343.99 total 1188.23 last 28.77) - open/close 33331 (time 1709788354.02 total 1198.26 last 29.62) - open/close 33641 (time 1709788364.02 total 1208.27 last 30.98) - open/close 33925 (time 1709788374.05 total 1218.29 last 28.34) - open/close 34223 (time 1709788384.08 total 1228.32 last 29.70) - open/close 34512 (time 1709788394.08 total 1238.33 last 28.89) - open/close 34803 (time 1709788404.09 total 1248.34 last 29.07) - open/close 35093 (time 1709788414.11 total 1258.35 last 28.96) - open/close 35371 (time 1709788424.12 total 1268.36 last 27.76) - open/close 35651 (time 1709788434.15 total 1278.39 last 27.92) - open/close 35916 (time 1709788444.17 total 1288.41 last 26.44) - open/close 36182 (time 1709788454.19 total 1298.43 last 26.54) - open/close 36439 (time 1709788464.22 total 1308.47 last 25.62) - open/close 36721 (time 1709788474.25 total 1318.49 last 28.13) - open/close 36994 (time 1709788484.28 total 1328.53 last 27.20) - open/close 37274 (time 1709788494.33 total 1338.58 last 27.86) - open/close 37550 (time 1709788504.36 total 1348.61 last 27.51) - open/close 37828 (time 1709788514.38 total 1358.63 last 27.75) - open/close 38102 (time 1709788524.39 total 1368.63 last 27.38) - open/close 38373 (time 1709788534.40 total 1378.64 last 27.08) - open/close 38660 (time 1709788544.40 total 1388.65 last 28.67) - open/close 38930 (time 1709788554.41 total 1398.65 last 26.99) - open/close 39207 (time 1709788564.41 total 1408.66 last 27.68) - open/close 39478 (time 1709788574.45 total 1418.70 last 26.99) - open/close 39732 (time 1709788584.50 total 1428.74 last 25.29) - open/close 40000 (time 1709788594.51 total 1438.75 last 26.77) - open/close 40259 (time 1709788604.52 total 1448.76 last 25.87) - open/close 40524 (time 1709788614.52 total 1458.76 last 26.49) - open/close 40797 (time 1709788624.53 total 1468.77 last 27.28) - open/close 41063 (time 1709788634.55 total 1478.79 last 26.55) - open/close 41307 (time 1709788644.56 total 1488.80 last 24.38) - open/close 41559 (time 1709788654.56 total 1498.80 last 25.18) - open/close 41812 (time 1709788664.59 total 1508.84 last 25.22) - open/close 42058 (time 1709788674.61 total 1518.85 last 24.56) - open/close 42290 (time 1709788684.61 total 1528.85 last 23.20) - open/close 42529 (time 1709788694.65 total 1538.89 last 23.82) - open/close 42773 (time 1709788704.68 total 1548.92 last 24.32) - open/close 43017 (time 1709788714.72 total 1558.96 last 24.31) - open/close 43258 (time 1709788724.74 total 1568.98 last 24.04) - open/close 43496 (time 1709788734.76 total 1579.00 last 23.76) - open/close 43730 (time 1709788744.79 total 1589.03 last 23.33) - open/close 43963 (time 1709788754.79 total 1599.04 last 23.28) - open/close 44188 (time 1709788764.81 total 1609.06 last 22.46) - open/close 44400 (time 1709788774.83 total 1619.07 last 21.16) - open/close 44629 (time 1709788784.85 total 1629.09 last 22.86) - open/close 44862 (time 1709788794.89 total 1639.13 last 23.21) - open/close 45085 (time 1709788804.92 total 1649.16 last 22.23) - open/close 45316 (time 1709788814.94 total 1659.18 last 23.05) - open/close 45546 (time 1709788824.97 total 1669.21 last 22.94) - open/close 45776 (time 1709788834.97 total 1679.22 last 22.98) - open/close 45999 (time 1709788845.00 total 1689.24 last 22.24) - open/close 46220 (time 1709788855.04 total 1699.28 last 22.01) - open/close 46425 (time 1709788865.04 total 1709.28 last 20.50) - open/close 46639 (time 1709788875.06 total 1719.30 last 21.37) - open/close 46847 (time 1709788885.10 total 1729.34 last 20.71) - open/close 47054 (time 1709788895.11 total 1739.35 last 20.67) - open/close 47270 (time 1709788905.12 total 1749.36 last 21.58) - open/close 47482 (time 1709788915.16 total 1759.40 last 21.11) - open/close 47690 (time 1709788925.21 total 1769.45 last 20.71) - open/close 47903 (time 1709788935.21 total 1779.46 last 21.28) - open/close 48119 (time 1709788945.26 total 1789.50 last 21.50) - open/close 48328 (time 1709788955.28 total 1799.53 last 20.85) - open/close 48532 (time 1709788965.33 total 1809.57 last 20.30) - open/close 48740 (time 1709788975.37 total 1819.61 last 20.73) - open/close 48944 (time 1709788985.39 total 1829.63 last 20.36) - open/close 49145 (time 1709788995.42 total 1839.66 last 20.04) - open/close 49347 (time 1709789005.45 total 1849.69 last 20.14) - open/close 49556 (time 1709789015.47 total 1859.71 last 20.86) - open/close 49774 (time 1709789025.51 total 1869.76 last 21.70) - open/close 49991 (time 1709789035.52 total 1879.76 last 21.69) total: 50000 open/close in 1880.15 seconds: 26.59 ops/second - unlinked 0 (time 1709789037 ; total 0 ; last 0) - unlinked 10000 (time 1709789610 ; total 573 ; last 573) - unlinked 20000 (time 1709790077 ; total 1040 ; last 467) - unlinked 30000 (time 1709790387 ; total 1350 ; last 310) - unlinked 40000 (time 1709790524 ; total 1487 ; last 137) total: 50000 unlinks in 1535 seconds: 32.573292 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg129-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg129-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg129-server stop mds service on oleg129-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg129-server unloading modules on: 'oleg129-server' oleg129-server: oleg129-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg129-client: oleg129-client: ssh exited with exit code 2 pdsh@oleg129-client: oleg129-server: ssh exited with exit code 2 pdsh@oleg129-client: oleg129-client: ssh exited with exit code 2 pdsh@oleg129-client: oleg129-server: ssh exited with exit code 2 PASS 101a (3455s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg129-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg129-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg129-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg129-client: oleg129-server: ssh exited with exit code 2 oleg129-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg129-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == conf-sanity test 101b: Race events DISCONNECT and ACTIVE in osp ========================================================== 00:49:48 (1709790588) start mds service on oleg129-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg129-server' oleg129-server: oleg129-server.virtnet: executing load_modules_local oleg129-server: Loading modules from /home/green/git/lustre-release/lustre oleg129-server: detected 4 online CPUs by sysfs oleg129-server: Force libcfs to create 2 CPU partitions oleg129-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg129-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg129-server: oleg129-server.virtnet: executing set_default_debug -1 all pdsh@oleg129-client: oleg129-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg129-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg129-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg129-server: oleg129-server.virtnet: executing set_default_debug -1 all pdsh@oleg129-client: oleg129-server: ssh exited with exit code 1 Started lustre-OST0000 oleg129-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg129-client.virtnet: -o user_xattr,flock oleg129-server@tcp:/lustre /mnt/lustre fail_loc=0x80002107 fail_val=20 stop ost1 service on oleg129-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg129-server start ost1 service on oleg129-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg129-server: oleg129-server.virtnet: executing set_default_debug -1 all pdsh@oleg129-client: oleg129-server: ssh exited with exit code 1 Started lustre-OST0000 oleg129-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 12 sec oleg129-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8800b62d6800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b62d6800.ost_server_uuid in FULL state after 0 sec umount lustre on /mnt/lustre..... Stopping client oleg129-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg129-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg129-server stop mds service on oleg129-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg129-server unloading modules on: 'oleg129-server' oleg129-server: oleg129-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg129-client: oleg129-client: ssh exited with exit code 2 pdsh@oleg129-client: oleg129-server: ssh exited with exit code 2 pdsh@oleg129-client: oleg129-client: ssh exited with exit code 2 pdsh@oleg129-client: oleg129-server: ssh exited with exit code 2 PASS 101b (85s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg129-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg129-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory Stopping clients: oleg129-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg129-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg129-client: oleg129-server: ssh exited with exit code 2 oleg129-server: oleg129-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg129-server' oleg129-server: oleg129-server.virtnet: executing load_modules_local oleg129-server: Loading modules from /home/green/git/lustre-release/lustre oleg129-server: detected 4 online CPUs by sysfs oleg129-server: Force libcfs to create 2 CPU partitions oleg129-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg129-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 4119 sec ========== 00:51:36 (1709790696) === conf-sanity: start cleanup 00:51:36 (1709790696) === === conf-sanity: finish cleanup 00:51:36 (1709790696) ===