-----============= acceptance-small: conf-sanity ============----- Wed Mar 6 23:38:45 EST 2024 excepting tests: 32newtarball 110 skipping tests SLOW=no: 45 69 106 111 114 Stopping clients: oleg201-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg201-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg201-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg201-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg201-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg201-server oleg201-server: oleg201-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg201-server' oleg201-server: oleg201-server.virtnet: executing load_modules_local oleg201-server: Loading modules from /home/green/git/lustre-release/lustre oleg201-server: detected 4 online CPUs by sysfs oleg201-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg201-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg201-server: oleg201-server.virtnet: executing set_default_debug -1 all pdsh@oleg201-client: oleg201-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg201-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg201-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg201-server: oleg201-server.virtnet: executing set_default_debug -1 all pdsh@oleg201-client: oleg201-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg201-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg201-server: oleg201-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg201-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg201-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg201-server stop mds service on oleg201-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg201-server debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 35b: Continue reconnection retries, if the active server is busy ========================================================== 23:39:46 (1709786386) start mds service on oleg201-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg201-server: oleg201-server.virtnet: executing set_default_debug -1 all pdsh@oleg201-client: oleg201-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg201-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg201-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg201-server: oleg201-server.virtnet: executing set_default_debug -1 all pdsh@oleg201-client: oleg201-server: ssh exited with exit code 1 Started lustre-OST0000 oleg201-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg201-client.virtnet: -o user_xattr,flock oleg201-server@tcp:/lustre /mnt/lustre debug=ha conf-sanity.sh test_35b 2024-03-0623h40m05s Set up a fake failnode for the MDS at_max=0 at_max=0 Injecting EBUSY on MDS fail_loc=0x80000136 mdc.lustre-MDT0000-mdc-ffff8800aa7ed000.stats=clear Creating a test file and stat it File: '/mnt/lustre/d35b.conf-sanity/f35b.conf-sanity' Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115238826934274 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-03-06 23:40:48.000000000 -0500 Modify: 2024-03-06 23:40:48.000000000 -0500 Change: 2024-03-06 23:40:48.000000000 -0500 Birth: - Stop injecting EBUSY on MDS fail_loc=0 done at_max=600 at_max=600 Debug log: 72 lines, 72 kept, 0 dropped, 0 bad. umount lustre on /mnt/lustre..... Stopping client oleg201-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg201-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg201-server stop mds service on oleg201-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg201-server unloading modules on: 'oleg201-server' oleg201-server: oleg201-server.virtnet: executing unload_modules_local modules unloaded. oleg201-server: oleg201-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre oleg201-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg201-client: oleg201-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg201-server: oleg201-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre oleg201-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg201-client: oleg201-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg201-server: oleg201-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre oleg201-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg201-client: oleg201-server: ssh exited with exit code 19 checking for existing Lustre data: not found tunefs failed, reformatting instead Stopping clients: oleg201-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg201-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg201-client: oleg201-server: ssh exited with exit code 2 oleg201-server: oleg201-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg201-server' oleg201-server: oleg201-server.virtnet: executing load_modules_local oleg201-server: Loading modules from /home/green/git/lustre-release/lustre oleg201-server: detected 4 online CPUs by sysfs oleg201-server: Force libcfs to create 2 CPU partitions oleg201-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory oleg201-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg201-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg201-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg201-server: oleg201-server.virtnet: executing set_default_debug -1 all pdsh@oleg201-client: oleg201-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg201-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg201-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg201-server: oleg201-server.virtnet: executing set_default_debug -1 all pdsh@oleg201-client: oleg201-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg201-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg201-server: oleg201-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg201-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg201-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg201-server stop mds service on oleg201-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg201-server PASS 35b (122s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 23:41:49 (1709786509) start mds service on oleg201-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg201-server: oleg201-server.virtnet: executing set_default_debug -1 all pdsh@oleg201-client: oleg201-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg201-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg201-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg201-server: oleg201-server.virtnet: executing set_default_debug -1 all pdsh@oleg201-client: oleg201-server: ssh exited with exit code 1 Started lustre-OST0000 oleg201-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg201-client.virtnet: -o user_xattr,flock oleg201-server@tcp:/lustre /mnt/lustre copying 10 files to /mnt/lustre/d38.conf-sanity tar: Removing leading `/' from member names umount lustre on /mnt/lustre..... Stopping client oleg201-client.virtnet /mnt/lustre (opts:) osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=44 stop mds service on oleg201-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg201-server delete lov_objid file on MDS 000000 43 000008 start mds service on oleg201-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg201-server: oleg201-server.virtnet: executing set_default_debug -1 all pdsh@oleg201-client: oleg201-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg201-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg201-client.virtnet: -o user_xattr,flock oleg201-server@tcp:/lustre /mnt/lustre osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=2 umount lustre on /mnt/lustre..... Stopping client oleg201-client.virtnet /mnt/lustre (opts:) stop mds service on oleg201-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg201-server 000000 2 000008 oleg201-server: 8+0 records in oleg201-server: 8+0 records out oleg201-server: 4096 bytes (4.1 kB) copied, 0.000854783 s, 4.8 MB/s start mds service on oleg201-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg201-server: oleg201-server.virtnet: executing set_default_debug -1 all pdsh@oleg201-client: oleg201-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg201-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg201-client.virtnet: -o user_xattr,flock oleg201-server@tcp:/lustre /mnt/lustre osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=35 umount lustre on /mnt/lustre..... Stopping client oleg201-client.virtnet /mnt/lustre (opts:) stop mds service on oleg201-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg201-server 000000 34 000008 files compared the same umount lustre on /mnt/lustre..... stop ost1 service on oleg201-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg201-server stop mds service on oleg201-server unloading modules on: 'oleg201-server' oleg201-server: oleg201-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg201-client: oleg201-client: ssh exited with exit code 2 pdsh@oleg201-client: oleg201-server: ssh exited with exit code 2 pdsh@oleg201-client: oleg201-client: ssh exited with exit code 2 pdsh@oleg201-client: oleg201-server: ssh exited with exit code 2 PASS 38 (71s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg201-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg201-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg201-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg201-client: oleg201-server: ssh exited with exit code 2 oleg201-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg201-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 23:43:03 (1709786583) Stopping clients: oleg201-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg201-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg201-client: oleg201-server: ssh exited with exit code 2 oleg201-server: oleg201-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_size: failed to read coresize from /sys/module/pcc_cpufreq ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg201-server' oleg201-server: oleg201-server.virtnet: executing load_modules_local oleg201-server: Loading modules from /home/green/git/lustre-release/lustre oleg201-server: detected 4 online CPUs by sysfs oleg201-server: Force libcfs to create 2 CPU partitions oleg201-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg201-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg201-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg201-server: oleg201-server.virtnet: executing set_default_debug -1 all pdsh@oleg201-client: oleg201-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg201-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg201-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg201-server: oleg201-server.virtnet: executing set_default_debug -1 all pdsh@oleg201-client: oleg201-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg201-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg201-client.virtnet: -o user_xattr,flock oleg201-server@tcp:/lustre /mnt/lustre setup single mount lustre success fail_loc=0x142 start ost2 service on oleg201-server Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg201-server: oleg201-server.virtnet: executing set_default_debug -1 all pdsh@oleg201-client: oleg201-server: ssh exited with exit code 1 Commit the device label on lustre-ost2/ost2 Started lustre-OST0001 oleg201-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec stop ost2 service on oleg201-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg201-server umount lustre on /mnt/lustre..... Stopping client oleg201-client.virtnet /mnt/lustre (opts:-f) stop ost1 service on oleg201-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg201-server stop mds service on oleg201-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg201-server unloading modules on: 'oleg201-server' oleg201-server: oleg201-server.virtnet: executing unload_modules_local modules unloaded. oleg201-server: oleg201-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre oleg201-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg201-client: oleg201-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg201-server: oleg201-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre oleg201-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg201-client: oleg201-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg201-server: oleg201-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre oleg201-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg201-client: oleg201-server: ssh exited with exit code 19 checking for existing Lustre data: not found tunefs failed, reformatting instead Stopping clients: oleg201-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg201-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg201-client: oleg201-server: ssh exited with exit code 2 oleg201-server: oleg201-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg201-server' oleg201-server: oleg201-server.virtnet: executing load_modules_local oleg201-server: Loading modules from /home/green/git/lustre-release/lustre oleg201-server: detected 4 online CPUs by sysfs oleg201-server: Force libcfs to create 2 CPU partitions oleg201-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg201-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg201-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg201-server: oleg201-server.virtnet: executing set_default_debug -1 all pdsh@oleg201-client: oleg201-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg201-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg201-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg201-server: oleg201-server.virtnet: executing set_default_debug -1 all pdsh@oleg201-client: oleg201-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg201-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg201-server: oleg201-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg201-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg201-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg201-server stop mds service on oleg201-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg201-server PASS 51 (160s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:45:46 (1709786746) start mds service on oleg201-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg201-server: oleg201-server.virtnet: executing set_default_debug -1 all pdsh@oleg201-client: oleg201-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg201-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg201-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg201-server: oleg201-server.virtnet: executing set_default_debug -1 all pdsh@oleg201-client: oleg201-server: ssh exited with exit code 1 Started lustre-OST0000 oleg201-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg201-client.virtnet: -o user_xattr,flock oleg201-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 212 (time 1709786776.26 total 10.00 last 21.19) - open/close 395 (time 1709786786.96 total 20.70 last 17.10) - open/close 634 (time 1709786797.25 total 31.00 last 23.22) - open/close 864 (time 1709786807.26 total 41.01 last 22.98) - open/close 1053 (time 1709786817.33 total 51.08 last 18.75) - open/close 1208 (time 1709786827.34 total 61.08 last 15.50) - open/close 1412 (time 1709786838.20 total 71.95 last 18.78) - open/close 1697 (time 1709786849.07 total 82.82 last 26.22) - open/close 1956 (time 1709786860.01 total 93.76 last 23.68) - open/close 2258 (time 1709786870.68 total 104.42 last 28.31) - open/close 2474 (time 1709786881.11 total 114.86 last 20.69) - open/close 2723 (time 1709786891.86 total 125.61 last 23.17) - open/close 2915 (time 1709786902.30 total 136.05 last 18.39) - open/close 3122 (time 1709786912.80 total 146.54 last 19.73) - open/close 3333 (time 1709786923.30 total 157.04 last 20.10) - open/close 3565 (time 1709786933.30 total 167.05 last 23.20) - open/close 3800 (time 1709786943.77 total 177.52 last 22.44) - open/close 4062 (time 1709786954.36 total 188.10 last 24.75) - open/close 4355 (time 1709786964.36 total 198.11 last 29.29) - open/close 4564 (time 1709786975.30 total 209.05 last 19.10) - open/close 4754 (time 1709786985.71 total 219.45 last 18.26) - open/close 5031 (time 1709786996.35 total 230.09 last 26.03) - open/close 5315 (time 1709787006.35 total 240.10 last 28.39) - open/close 5452 (time 1709787016.65 total 250.39 last 13.31) - open/close 5646 (time 1709787026.83 total 260.58 last 19.05) - open/close 5789 (time 1709787037.40 total 271.15 last 13.53) - open/close 5950 (time 1709787048.20 total 281.95 last 14.90) - open/close 6124 (time 1709787058.40 total 292.15 last 17.06) - open/close 6208 (time 1709787069.12 total 302.87 last 7.84) - open/close 6309 (time 1709787079.16 total 312.91 last 10.06) - open/close 6440 (time 1709787089.56 total 323.31 last 12.60) - open/close 6704 (time 1709787100.40 total 334.15 last 24.34) - open/close 6943 (time 1709787110.84 total 344.59 last 22.90) - open/close 7217 (time 1709787121.15 total 354.89 last 26.59) - open/close 7522 (time 1709787132.19 total 365.94 last 27.62) - open/close 7644 (time 1709787142.83 total 376.58 last 11.47) - open/close 7925 (time 1709787153.24 total 386.99 last 26.98) - open/close 8086 (time 1709787163.79 total 397.54 last 15.27) - open/close 8237 (time 1709787173.92 total 407.67 last 14.90) - open/close 8377 (time 1709787184.06 total 417.81 last 13.82) - open/close 8557 (time 1709787194.47 total 428.22 last 17.29) - open/close 8776 (time 1709787204.48 total 438.23 last 21.88) - open/close 8918 (time 1709787222.21 total 455.96 last 8.01) - open/close 9006 (time 1709787232.22 total 465.97 last 8.79) - open/close 9149 (time 1709787242.35 total 476.10 last 14.12) - open/close 9300 (time 1709787252.36 total 486.11 last 15.08) - open/close 9498 (time 1709787262.38 total 496.13 last 19.76) - open/close 9783 (time 1709787272.40 total 506.15 last 28.44) - open/close 10000 (time 1709787276.20 total 509.95 last 57.17) - open/close 10566 (time 1709787286.21 total 519.95 last 56.56) - open/close 11078 (time 1709787296.21 total 529.95 last 51.20) - open/close 11640 (time 1709787306.22 total 539.96 last 56.15) - open/close 12169 (time 1709787316.22 total 549.97 last 52.88) - open/close 12713 (time 1709787326.22 total 559.97 last 54.38) - open/close 13162 (time 1709787336.24 total 569.99 last 44.84) - open/close 13624 (time 1709787346.25 total 580.00 last 46.13) - open/close 14088 (time 1709787356.26 total 590.00 last 46.38) - open/close 14598 (time 1709787366.26 total 600.01 last 50.98) - open/close 15033 (time 1709787376.28 total 610.03 last 43.40) - open/close 15453 (time 1709787386.30 total 620.05 last 41.93) - open/close 15867 (time 1709787396.36 total 630.11 last 41.15) - open/close 16234 (time 1709787406.37 total 640.12 last 36.66) - open/close 16621 (time 1709787416.37 total 650.12 last 38.70) - open/close 17046 (time 1709787426.37 total 660.12 last 42.49) - open/close 17420 (time 1709787436.39 total 670.14 last 37.32) - open/close 17878 (time 1709787446.40 total 680.15 last 45.79) - open/close 18237 (time 1709787456.42 total 690.16 last 35.83) - open/close 18615 (time 1709787466.42 total 700.17 last 37.77) - open/close 18932 (time 1709787476.43 total 710.17 last 31.69) - open/close 19216 (time 1709787495.18 total 728.93 last 15.14) - open/close 19612 (time 1709787505.22 total 738.97 last 39.45) - open/close 19932 (time 1709787515.27 total 749.02 last 31.85) - open/close 20263 (time 1709787525.28 total 759.03 last 33.06) - open/close 20583 (time 1709787535.28 total 769.03 last 32.00) - open/close 20905 (time 1709787545.29 total 779.04 last 32.17) - open/close 21224 (time 1709787555.32 total 789.06 last 31.82) - open/close 21265 (time 1709787567.85 total 801.60 last 3.27) - open/close 21604 (time 1709787577.86 total 811.61 last 33.87) - open/close 21860 (time 1709787596.56 total 830.31 last 13.69) - open/close 22186 (time 1709787606.57 total 840.31 last 32.58) - open/close 22517 (time 1709787616.58 total 850.33 last 33.05) - open/close 22837 (time 1709787626.58 total 860.33 last 31.99) - open/close 23079 (time 1709787644.58 total 878.33 last 13.45) - open/close 23343 (time 1709787654.58 total 888.33 last 26.39) - open/close 23669 (time 1709787664.60 total 898.35 last 32.55) - open/close 23978 (time 1709787674.61 total 908.35 last 30.88) - open/close 24315 (time 1709787684.67 total 918.42 last 33.49) - open/close 24640 (time 1709787694.71 total 928.46 last 32.38) - open/close 24930 (time 1709787704.74 total 938.48 last 28.92) - open/close 25206 (time 1709787714.76 total 948.51 last 27.54) - open/close 25477 (time 1709787724.78 total 958.53 last 27.04) - open/close 25748 (time 1709787734.81 total 968.55 last 27.04) - open/close 26022 (time 1709787744.91 total 978.65 last 27.13) - open/close 26307 (time 1709787754.92 total 988.67 last 28.45) - open/close 26576 (time 1709787764.95 total 998.70 last 26.82) - open/close 26892 (time 1709787775.04 total 1008.78 last 31.34) - open/close 27197 (time 1709787785.05 total 1018.80 last 30.45) - open/close 27474 (time 1709787795.06 total 1028.80 last 27.69) - open/close 27791 (time 1709787805.13 total 1038.88 last 31.47) - open/close 28059 (time 1709787815.17 total 1048.92 last 26.69) - open/close 28339 (time 1709787825.18 total 1058.93 last 27.97) - open/close 28652 (time 1709787835.21 total 1068.95 last 31.22) - open/close 28937 (time 1709787845.23 total 1078.98 last 28.43) - open/close 29197 (time 1709787855.24 total 1088.99 last 25.98) - open/close 29440 (time 1709787865.25 total 1099.00 last 24.28) - open/close 29740 (time 1709787875.27 total 1109.02 last 29.94) - open/close 30000 (time 1709787885.19 total 1118.94 last 26.21) - open/close 30227 (time 1709787895.22 total 1128.97 last 22.64) - open/close 30477 (time 1709787905.22 total 1138.97 last 24.99) - open/close 30740 (time 1709787915.23 total 1148.98 last 26.27) - open/close 30978 (time 1709787925.26 total 1159.01 last 23.74) - open/close 31240 (time 1709787935.29 total 1169.04 last 26.13) - open/close 31515 (time 1709787945.31 total 1179.06 last 27.43) - open/close 31836 (time 1709787955.33 total 1189.08 last 32.05) - open/close 32142 (time 1709787965.34 total 1199.09 last 30.57) - open/close 32444 (time 1709787975.37 total 1209.12 last 30.11) - open/close 32736 (time 1709787985.38 total 1219.13 last 29.16) - open/close 33050 (time 1709787995.40 total 1229.15 last 31.35) - open/close 33390 (time 1709788005.40 total 1239.15 last 33.99) - open/close 33740 (time 1709788015.42 total 1249.17 last 34.93) - open/close 34097 (time 1709788025.46 total 1259.21 last 35.56) - open/close 34467 (time 1709788035.48 total 1269.22 last 36.96) - open/close 34848 (time 1709788045.51 total 1279.25 last 37.99) - open/close 35221 (time 1709788055.53 total 1289.28 last 37.19) - open/close 35575 (time 1709788065.54 total 1299.29 last 35.39) - open/close 35921 (time 1709788075.54 total 1309.29 last 34.60) - open/close 36306 (time 1709788085.54 total 1319.29 last 38.49) - open/close 36656 (time 1709788095.56 total 1329.31 last 34.93) - open/close 37002 (time 1709788105.56 total 1339.31 last 34.59) - open/close 37356 (time 1709788115.58 total 1349.33 last 35.33) - open/close 37692 (time 1709788125.60 total 1359.35 last 33.54) - open/close 38041 (time 1709788135.61 total 1369.36 last 34.86) - open/close 38390 (time 1709788145.61 total 1379.36 last 34.89) - open/close 38734 (time 1709788155.63 total 1389.38 last 34.34) - open/close 39089 (time 1709788165.64 total 1399.39 last 35.45) - open/close 39435 (time 1709788175.67 total 1409.42 last 34.52) - open/close 39772 (time 1709788185.68 total 1419.43 last 33.66) - open/close 40000 (time 1709788192.23 total 1425.98 last 34.80) - open/close 40340 (time 1709788202.24 total 1435.99 last 33.96) - open/close 40682 (time 1709788212.25 total 1445.99 last 34.19) - open/close 41028 (time 1709788222.26 total 1456.00 last 34.56) - open/close 41365 (time 1709788232.26 total 1466.01 last 33.69) - open/close 41691 (time 1709788242.27 total 1476.01 last 32.58) - open/close 42024 (time 1709788252.27 total 1486.02 last 33.29) - open/close 42350 (time 1709788262.28 total 1496.03 last 32.57) - open/close 42669 (time 1709788272.30 total 1506.05 last 31.81) - open/close 42987 (time 1709788282.34 total 1516.08 last 31.70) - open/close 43283 (time 1709788292.34 total 1526.08 last 29.60) - open/close 43601 (time 1709788302.35 total 1536.10 last 31.74) - open/close 43912 (time 1709788312.36 total 1546.10 last 31.10) - open/close 44218 (time 1709788322.36 total 1556.11 last 30.58) - open/close 44519 (time 1709788332.39 total 1566.13 last 30.03) - open/close 44822 (time 1709788342.40 total 1576.15 last 30.27) - open/close 45118 (time 1709788352.42 total 1586.17 last 29.54) - open/close 45406 (time 1709788362.44 total 1596.19 last 28.74) - open/close 45701 (time 1709788372.46 total 1606.21 last 29.43) - open/close 45994 (time 1709788382.49 total 1616.24 last 29.21) - open/close 46263 (time 1709788392.51 total 1626.25 last 26.87) - open/close 46563 (time 1709788402.53 total 1636.28 last 29.94) - open/close 46858 (time 1709788412.56 total 1646.30 last 29.42) - open/close 47144 (time 1709788422.56 total 1656.31 last 28.59) - open/close 47435 (time 1709788432.57 total 1666.32 last 29.07) - open/close 47712 (time 1709788442.59 total 1676.34 last 27.65) - open/close 47987 (time 1709788452.59 total 1686.34 last 27.49) - open/close 48267 (time 1709788462.60 total 1696.34 last 27.99) - open/close 48545 (time 1709788472.62 total 1706.37 last 27.74) - open/close 48842 (time 1709788482.65 total 1716.40 last 29.60) - open/close 49116 (time 1709788492.66 total 1726.41 last 27.38) - open/close 49397 (time 1709788502.68 total 1736.43 last 28.03) - open/close 49686 (time 1709788512.70 total 1746.45 last 28.85) - open/close 49972 (time 1709788522.71 total 1756.46 last 28.56) total: 50000 open/close in 1757.43 seconds: 28.45 ops/second - unlinked 0 (time 1709788524 ; total 0 ; last 0) - unlinked 10000 (time 1709788942 ; total 418 ; last 418) - unlinked 20000 (time 1709789322 ; total 798 ; last 380) - unlinked 30000 (time 1709789607 ; total 1083 ; last 285) - unlinked 40000 (time 1709789762 ; total 1238 ; last 155) total: 50000 unlinks in 1290 seconds: 38.759689 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg201-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg201-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg201-server stop mds service on oleg201-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg201-server unloading modules on: 'oleg201-server' oleg201-server: oleg201-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg201-client: oleg201-client: ssh exited with exit code 2 pdsh@oleg201-client: oleg201-server: ssh exited with exit code 2 pdsh@oleg201-client: oleg201-client: ssh exited with exit code 2 pdsh@oleg201-client: oleg201-server: ssh exited with exit code 2 PASS 101a (3082s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg201-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg201-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg201-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg201-client: oleg201-server: ssh exited with exit code 2 oleg201-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg201-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == conf-sanity test 101b: Race events DISCONNECT and ACTIVE in osp ========================================================== 00:37:10 (1709789830) start mds service on oleg201-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg201-server' oleg201-server: oleg201-server.virtnet: executing load_modules_local oleg201-server: Loading modules from /home/green/git/lustre-release/lustre oleg201-server: detected 4 online CPUs by sysfs oleg201-server: Force libcfs to create 2 CPU partitions oleg201-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory oleg201-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg201-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg201-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg201-server: oleg201-server.virtnet: executing set_default_debug -1 all pdsh@oleg201-client: oleg201-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg201-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg201-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg201-server: oleg201-server.virtnet: executing set_default_debug -1 all pdsh@oleg201-client: oleg201-server: ssh exited with exit code 1 Started lustre-OST0000 oleg201-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg201-client.virtnet: -o user_xattr,flock oleg201-server@tcp:/lustre /mnt/lustre fail_loc=0x80002107 fail_val=20 stop ost1 service on oleg201-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg201-server start ost1 service on oleg201-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg201-server: oleg201-server.virtnet: executing set_default_debug -1 all pdsh@oleg201-client: oleg201-server: ssh exited with exit code 1 Started lustre-OST0000 oleg201-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 12 sec oleg201-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8800ab2a2800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800ab2a2800.ost_server_uuid in FULL state after 0 sec umount lustre on /mnt/lustre..... Stopping client oleg201-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg201-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg201-server stop mds service on oleg201-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg201-server unloading modules on: 'oleg201-server' oleg201-server: oleg201-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg201-client: oleg201-client: ssh exited with exit code 2 pdsh@oleg201-client: oleg201-server: ssh exited with exit code 2 pdsh@oleg201-client: oleg201-client: ssh exited with exit code 2 pdsh@oleg201-client: oleg201-server: ssh exited with exit code 2 PASS 101b (86s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg201-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg201-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory Stopping clients: oleg201-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg201-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg201-client: oleg201-server: ssh exited with exit code 2 oleg201-server: oleg201-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg201-server' oleg201-server: oleg201-server.virtnet: executing load_modules_local oleg201-server: Loading modules from /home/green/git/lustre-release/lustre oleg201-server: detected 4 online CPUs by sysfs oleg201-server: Force libcfs to create 2 CPU partitions oleg201-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg201-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 3613 sec ========== 00:38:58 (1709789938) === conf-sanity: start cleanup 00:38:59 (1709789939) === === conf-sanity: finish cleanup 00:38:59 (1709789939) ===