-----============= acceptance-small: conf-sanity ============----- Wed Mar 6 23:43:44 EST 2024 excepting tests: 32newtarball 110 skipping tests SLOW=no: 45 69 106 111 114 Stopping clients: oleg328-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg328-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg328-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg328-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg328-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg328-server oleg328-server: oleg328-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg328-server' oleg328-server: oleg328-server.virtnet: executing load_modules_local oleg328-server: Loading modules from /home/green/git/lustre-release/lustre oleg328-server: detected 4 online CPUs by sysfs oleg328-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg328-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg328-server: oleg328-server.virtnet: executing set_default_debug -1 all pdsh@oleg328-client: oleg328-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg328-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg328-server: oleg328-server.virtnet: executing set_default_debug -1 all pdsh@oleg328-client: oleg328-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg328-server: oleg328-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg328-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg328-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg328-server stop mds service on oleg328-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg328-server debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 35b: Continue reconnection retries, if the active server is busy ========================================================== 23:44:56 (1709786696) start mds service on oleg328-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg328-server: oleg328-server.virtnet: executing set_default_debug -1 all pdsh@oleg328-client: oleg328-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg328-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg328-server: oleg328-server.virtnet: executing set_default_debug -1 all pdsh@oleg328-client: oleg328-server: ssh exited with exit code 1 Started lustre-OST0000 oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg328-client.virtnet: -o user_xattr,flock oleg328-server@tcp:/lustre /mnt/lustre debug=ha conf-sanity.sh test_35b 2024-03-0623h45m14s Set up a fake failnode for the MDS at_max=0 at_max=0 Injecting EBUSY on MDS fail_loc=0x80000136 mdc.lustre-MDT0000-mdc-ffff8800afac9000.stats=clear Creating a test file and stat it File: '/mnt/lustre/d35b.conf-sanity/f35b.conf-sanity' Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115238826934274 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-03-06 23:45:59.000000000 -0500 Modify: 2024-03-06 23:45:59.000000000 -0500 Change: 2024-03-06 23:45:59.000000000 -0500 Birth: - Stop injecting EBUSY on MDS fail_loc=0 done at_max=600 at_max=600 Debug log: 76 lines, 76 kept, 0 dropped, 0 bad. umount lustre on /mnt/lustre..... Stopping client oleg328-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg328-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg328-server stop mds service on oleg328-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg328-server unloading modules on: 'oleg328-server' oleg328-server: oleg328-server.virtnet: executing unload_modules_local modules unloaded. oleg328-server: oleg328-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre oleg328-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg328-client: oleg328-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg328-server: oleg328-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre oleg328-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg328-client: oleg328-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg328-server: oleg328-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre oleg328-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg328-client: oleg328-server: ssh exited with exit code 19 checking for existing Lustre data: not found tunefs failed, reformatting instead Stopping clients: oleg328-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg328-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg328-client: oleg328-server: ssh exited with exit code 2 oleg328-server: oleg328-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg328-server' oleg328-server: oleg328-server.virtnet: executing load_modules_local oleg328-server: Loading modules from /home/green/git/lustre-release/lustre oleg328-server: detected 4 online CPUs by sysfs oleg328-server: Force libcfs to create 2 CPU partitions oleg328-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg328-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg328-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg328-server: oleg328-server.virtnet: executing set_default_debug -1 all pdsh@oleg328-client: oleg328-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg328-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg328-server: oleg328-server.virtnet: executing set_default_debug -1 all pdsh@oleg328-client: oleg328-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg328-server: oleg328-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg328-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg328-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg328-server stop mds service on oleg328-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg328-server PASS 35b (131s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 23:47:09 (1709786829) start mds service on oleg328-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg328-server: oleg328-server.virtnet: executing set_default_debug -1 all pdsh@oleg328-client: oleg328-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg328-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg328-server: oleg328-server.virtnet: executing set_default_debug -1 all pdsh@oleg328-client: oleg328-server: ssh exited with exit code 1 Started lustre-OST0000 oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg328-client.virtnet: -o user_xattr,flock oleg328-server@tcp:/lustre /mnt/lustre copying 10 files to /mnt/lustre/d38.conf-sanity tar: Removing leading `/' from member names umount lustre on /mnt/lustre..... Stopping client oleg328-client.virtnet /mnt/lustre (opts:) osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=44 stop mds service on oleg328-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg328-server delete lov_objid file on MDS 000000 43 000008 start mds service on oleg328-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg328-server: oleg328-server.virtnet: executing set_default_debug -1 all pdsh@oleg328-client: oleg328-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg328-client.virtnet: -o user_xattr,flock oleg328-server@tcp:/lustre /mnt/lustre osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=2 umount lustre on /mnt/lustre..... Stopping client oleg328-client.virtnet /mnt/lustre (opts:) stop mds service on oleg328-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg328-server 000000 2 000008 oleg328-server: 8+0 records in oleg328-server: 8+0 records out oleg328-server: 4096 bytes (4.1 kB) copied, 0.00149315 s, 2.7 MB/s start mds service on oleg328-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg328-server: oleg328-server.virtnet: executing set_default_debug -1 all pdsh@oleg328-client: oleg328-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg328-client.virtnet: -o user_xattr,flock oleg328-server@tcp:/lustre /mnt/lustre osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=35 umount lustre on /mnt/lustre..... Stopping client oleg328-client.virtnet /mnt/lustre (opts:) stop mds service on oleg328-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg328-server 000000 34 000008 files compared the same umount lustre on /mnt/lustre..... stop ost1 service on oleg328-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg328-server stop mds service on oleg328-server unloading modules on: 'oleg328-server' oleg328-server: oleg328-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg328-client: oleg328-client: ssh exited with exit code 2 pdsh@oleg328-client: oleg328-server: ssh exited with exit code 2 pdsh@oleg328-client: oleg328-server: ssh exited with exit code 2 pdsh@oleg328-client: oleg328-client: ssh exited with exit code 2 PASS 38 (78s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg328-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg328-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg328-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg328-client: oleg328-server: ssh exited with exit code 2 oleg328-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg328-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 23:48:30 (1709786910) Stopping clients: oleg328-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg328-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg328-client: oleg328-server: ssh exited with exit code 2 oleg328-server: oleg328-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg328-server' oleg328-server: oleg328-server.virtnet: executing load_modules_local oleg328-server: Loading modules from /home/green/git/lustre-release/lustre oleg328-server: detected 4 online CPUs by sysfs oleg328-server: Force libcfs to create 2 CPU partitions oleg328-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg328-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg328-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg328-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg328-server: oleg328-server.virtnet: executing set_default_debug -1 all pdsh@oleg328-client: oleg328-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg328-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg328-server: oleg328-server.virtnet: executing set_default_debug -1 all pdsh@oleg328-client: oleg328-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg328-client.virtnet: -o user_xattr,flock oleg328-server@tcp:/lustre /mnt/lustre setup single mount lustre success fail_loc=0x142 start ost2 service on oleg328-server Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg328-server: oleg328-server.virtnet: executing set_default_debug -1 all pdsh@oleg328-client: oleg328-server: ssh exited with exit code 1 Commit the device label on lustre-ost2/ost2 Started lustre-OST0001 oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec stop ost2 service on oleg328-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg328-server umount lustre on /mnt/lustre..... Stopping client oleg328-client.virtnet /mnt/lustre (opts:-f) stop ost1 service on oleg328-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg328-server stop mds service on oleg328-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg328-server unloading modules on: 'oleg328-server' oleg328-server: oleg328-server.virtnet: executing unload_modules_local modules unloaded. oleg328-server: oleg328-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre oleg328-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg328-client: oleg328-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg328-server: oleg328-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre oleg328-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg328-client: oleg328-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg328-server: oleg328-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre oleg328-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg328-client: oleg328-server: ssh exited with exit code 19 checking for existing Lustre data: not found tunefs failed, reformatting instead Stopping clients: oleg328-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg328-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg328-client: oleg328-server: ssh exited with exit code 2 oleg328-server: oleg328-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg328-server' oleg328-server: oleg328-server.virtnet: executing load_modules_local oleg328-server: Loading modules from /home/green/git/lustre-release/lustre oleg328-server: detected 4 online CPUs by sysfs oleg328-server: Force libcfs to create 2 CPU partitions oleg328-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg328-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg328-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg328-server: oleg328-server.virtnet: executing set_default_debug -1 all pdsh@oleg328-client: oleg328-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg328-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg328-server: oleg328-server.virtnet: executing set_default_debug -1 all pdsh@oleg328-client: oleg328-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg328-server: oleg328-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg328-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg328-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg328-server stop mds service on oleg328-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg328-server PASS 51 (188s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:51:42 (1709787102) start mds service on oleg328-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg328-server: oleg328-server.virtnet: executing set_default_debug -1 all pdsh@oleg328-client: oleg328-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg328-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg328-server: oleg328-server.virtnet: executing set_default_debug -1 all pdsh@oleg328-client: oleg328-server: ssh exited with exit code 1 Started lustre-OST0000 oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg328-client.virtnet: -o user_xattr,flock oleg328-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 250 (time 1709787139.24 total 12.69 last 19.69) - open/close 444 (time 1709787149.58 total 23.03 last 18.78) - open/close 580 (time 1709787162.22 total 35.67 last 10.76) - open/close 729 (time 1709787172.22 total 45.67 last 14.89) - open/close 908 (time 1709787182.65 total 56.10 last 17.17) - open/close 1100 (time 1709787192.68 total 66.13 last 19.13) - open/close 1279 (time 1709787202.70 total 76.15 last 17.88) - open/close 1489 (time 1709787213.21 total 86.66 last 19.97) - open/close 1750 (time 1709787223.79 total 97.25 last 24.66) - open/close 1947 (time 1709787234.81 total 108.26 last 17.89) - open/close 2264 (time 1709787244.81 total 118.26 last 31.69) - open/close 2426 (time 1709787255.29 total 128.74 last 15.46) - open/close 2584 (time 1709787265.30 total 138.75 last 15.79) - open/close 2769 (time 1709787275.63 total 149.08 last 17.91) - open/close 2965 (time 1709787285.94 total 159.39 last 19.01) - open/close 3186 (time 1709787296.13 total 169.58 last 21.68) - open/close 3400 (time 1709787306.66 total 180.11 last 20.33) - open/close 3562 (time 1709787316.68 total 190.13 last 16.17) - open/close 3721 (time 1709787326.69 total 200.14 last 15.89) - open/close 3836 (time 1709787336.69 total 210.14 last 11.50) - open/close 3999 (time 1709787346.96 total 220.41 last 15.87) - open/close 4098 (time 1709787356.97 total 230.42 last 9.89) - open/close 4380 (time 1709787367.95 total 241.40 last 25.67) - open/close 4536 (time 1709787378.67 total 252.12 last 14.56) - open/close 4750 (time 1709787389.14 total 262.59 last 20.44) - open/close 4964 (time 1709787399.87 total 273.32 last 19.94) - open/close 5145 (time 1709787409.89 total 283.34 last 18.08) - open/close 5340 (time 1709787419.90 total 293.35 last 19.48) - open/close 5500 (time 1709787429.92 total 303.37 last 15.97) - open/close 5651 (time 1709787439.92 total 313.37 last 15.10) - open/close 5872 (time 1709787449.93 total 323.38 last 22.08) - open/close 6027 (time 1709787460.69 total 334.14 last 14.41) - open/close 6259 (time 1709787470.70 total 344.15 last 23.18) - open/close 6447 (time 1709787480.93 total 354.38 last 18.37) - open/close 6655 (time 1709787491.47 total 364.92 last 19.74) - open/close 6811 (time 1709787502.11 total 375.56 last 14.66) - open/close 7031 (time 1709787512.25 total 385.70 last 21.70) - open/close 7184 (time 1709787522.67 total 396.12 last 14.68) - open/close 7352 (time 1709787532.68 total 406.13 last 16.79) - open/close 7503 (time 1709787542.68 total 416.13 last 15.09) - open/close 7664 (time 1709787553.23 total 426.68 last 15.26) - open/close 7833 (time 1709787563.23 total 436.69 last 16.90) - open/close 7948 (time 1709787573.42 total 446.87 last 11.29) - open/close 8247 (time 1709787584.16 total 457.61 last 27.83) - open/close 8451 (time 1709787595.13 total 468.58 last 18.60) - open/close 8612 (time 1709787605.46 total 478.91 last 15.59) - open/close 8733 (time 1709787616.39 total 489.85 last 11.07) - open/close 8906 (time 1709787627.00 total 500.45 last 16.32) - open/close 9067 (time 1709787637.00 total 510.45 last 16.09) - open/close 9219 (time 1709787647.83 total 521.28 last 14.03) - open/close 9360 (time 1709787658.75 total 532.20 last 12.92) - open/close 9472 (time 1709787668.75 total 542.20 last 11.20) - open/close 9593 (time 1709787679.18 total 552.63 last 11.59) - open/close 9770 (time 1709787689.69 total 563.14 last 16.85) - open/close 9968 (time 1709787699.69 total 573.15 last 19.79) - open/close 10127 (time 1709787709.74 total 583.19 last 15.83) - open/close 10309 (time 1709787719.76 total 593.21 last 18.16) - open/close 10433 (time 1709787729.77 total 603.22 last 12.39) - open/close 10650 (time 1709787739.78 total 613.23 last 21.68) - open/close 11115 (time 1709787749.79 total 623.24 last 46.46) - open/close 11616 (time 1709787759.82 total 633.27 last 49.95) - open/close 12101 (time 1709787769.82 total 643.27 last 48.47) - open/close 12556 (time 1709787779.83 total 653.28 last 45.45) - open/close 13025 (time 1709787789.84 total 663.29 last 46.87) - open/close 13509 (time 1709787799.85 total 673.30 last 48.35) - open/close 13987 (time 1709787809.85 total 683.31 last 47.77) - open/close 14458 (time 1709787819.86 total 693.31 last 47.08) - open/close 14945 (time 1709787829.87 total 703.33 last 48.63) - open/close 15435 (time 1709787839.88 total 713.33 last 48.99) - open/close 15911 (time 1709787849.88 total 723.33 last 47.59) - open/close 16399 (time 1709787859.88 total 733.33 last 48.79) - open/close 16841 (time 1709787869.90 total 743.35 last 44.11) - open/close 17110 (time 1709787887.70 total 761.15 last 15.11) - open/close 17545 (time 1709787897.71 total 771.16 last 43.45) - open/close 17997 (time 1709787907.73 total 781.18 last 45.12) - open/close 18400 (time 1709787917.77 total 791.22 last 40.14) - open/close 18803 (time 1709787927.78 total 801.23 last 40.27) - open/close 19163 (time 1709787937.80 total 811.25 last 35.93) - open/close 19545 (time 1709787947.83 total 821.29 last 38.06) - open/close 19894 (time 1709787957.84 total 831.29 last 34.89) - open/close 20000 (time 1709787960.76 total 834.21 last 36.25) - open/close 20412 (time 1709787970.77 total 844.22 last 41.17) - open/close 20784 (time 1709787980.78 total 854.23 last 37.15) - open/close 21159 (time 1709787990.80 total 864.25 last 37.42) - open/close 21545 (time 1709788000.81 total 874.26 last 38.59) - open/close 21924 (time 1709788010.81 total 884.26 last 37.87) - open/close 22296 (time 1709788020.84 total 894.29 last 37.11) - open/close 22644 (time 1709788030.84 total 904.29 last 34.80) - open/close 22983 (time 1709788040.85 total 914.30 last 33.85) - open/close 23326 (time 1709788050.86 total 924.31 last 34.27) - open/close 23665 (time 1709788060.90 total 934.35 last 33.78) - open/close 24012 (time 1709788070.92 total 944.37 last 34.61) - open/close 24403 (time 1709788080.94 total 954.39 last 39.03) - open/close 24804 (time 1709788090.96 total 964.41 last 40.01) - open/close 25199 (time 1709788100.96 total 974.41 last 39.49) - open/close 25578 (time 1709788110.99 total 984.44 last 37.82) - open/close 25971 (time 1709788120.99 total 994.44 last 39.27) - open/close 26366 (time 1709788131.01 total 1004.46 last 39.45) - open/close 26769 (time 1709788141.01 total 1014.46 last 40.27) - open/close 27144 (time 1709788151.02 total 1024.47 last 37.46) - open/close 27511 (time 1709788161.04 total 1034.49 last 36.63) - open/close 27896 (time 1709788171.06 total 1044.52 last 38.41) - open/close 28247 (time 1709788181.08 total 1054.54 last 35.03) - open/close 28610 (time 1709788191.11 total 1064.56 last 36.20) - open/close 28958 (time 1709788201.12 total 1074.57 last 34.77) - open/close 29320 (time 1709788211.14 total 1084.59 last 36.13) - open/close 29661 (time 1709788221.15 total 1094.61 last 34.05) - open/close 30000 (time 1709788230.47 total 1103.93 last 36.38) - open/close 30342 (time 1709788240.49 total 1113.94 last 34.15) - open/close 30672 (time 1709788250.49 total 1123.95 last 32.98) - open/close 30981 (time 1709788260.52 total 1133.97 last 30.82) - open/close 31113 (time 1709788275.87 total 1149.32 last 8.60) - open/close 31424 (time 1709788285.89 total 1159.34 last 31.03) - open/close 31723 (time 1709788295.91 total 1169.36 last 29.83) - open/close 32029 (time 1709788305.93 total 1179.38 last 30.53) - open/close 32329 (time 1709788315.96 total 1189.41 last 29.93) - open/close 32634 (time 1709788325.97 total 1199.42 last 30.47) - open/close 32967 (time 1709788336.00 total 1209.45 last 33.19) - open/close 33284 (time 1709788346.03 total 1219.48 last 31.61) - open/close 33610 (time 1709788356.04 total 1229.49 last 32.56) - open/close 33926 (time 1709788366.08 total 1239.53 last 31.48) - open/close 34218 (time 1709788376.11 total 1249.56 last 29.12) - open/close 34534 (time 1709788386.14 total 1259.59 last 31.50) - open/close 34837 (time 1709788396.16 total 1269.61 last 30.22) - open/close 35147 (time 1709788406.18 total 1279.63 last 30.95) - open/close 35442 (time 1709788416.21 total 1289.66 last 29.41) - open/close 35762 (time 1709788426.25 total 1299.70 last 31.88) - open/close 35882 (time 1709788441.59 total 1315.04 last 7.82) - open/close 36182 (time 1709788451.62 total 1325.07 last 29.92) - open/close 36491 (time 1709788461.63 total 1335.08 last 30.86) - open/close 36781 (time 1709788471.65 total 1345.10 last 28.95) - open/close 37092 (time 1709788481.67 total 1355.12 last 31.04) - open/close 37349 (time 1709788491.68 total 1365.13 last 25.68) - open/close 37612 (time 1709788501.71 total 1375.16 last 26.22) - open/close 37900 (time 1709788511.73 total 1385.18 last 28.73) - open/close 38202 (time 1709788521.74 total 1395.19 last 30.17) - open/close 38478 (time 1709788531.75 total 1405.21 last 27.56) - open/close 38754 (time 1709788541.78 total 1415.23 last 27.52) - open/close 39041 (time 1709788551.80 total 1425.25 last 28.64) - open/close 39314 (time 1709788561.81 total 1435.26 last 27.28) - open/close 39608 (time 1709788571.82 total 1445.28 last 29.36) - open/close 39899 (time 1709788581.85 total 1455.30 last 29.03) - open/close 40000 (time 1709788585.44 total 1458.89 last 28.16) - open/close 40283 (time 1709788595.45 total 1468.90 last 28.26) - open/close 40573 (time 1709788605.46 total 1478.92 last 28.96) - open/close 40861 (time 1709788615.49 total 1488.94 last 28.74) - open/close 41142 (time 1709788625.51 total 1498.96 last 28.03) - open/close 41413 (time 1709788635.53 total 1508.98 last 27.04) - open/close 41647 (time 1709788645.54 total 1518.99 last 23.38) - open/close 41851 (time 1709788655.59 total 1529.04 last 20.31) - open/close 42054 (time 1709788665.62 total 1539.07 last 20.24) - open/close 42282 (time 1709788675.64 total 1549.09 last 22.76) - open/close 42528 (time 1709788685.66 total 1559.11 last 24.54) - open/close 42775 (time 1709788695.68 total 1569.13 last 24.66) - open/close 43030 (time 1709788705.70 total 1579.15 last 25.45) - open/close 43268 (time 1709788715.72 total 1589.17 last 23.75) - open/close 43513 (time 1709788725.74 total 1599.19 last 24.45) - open/close 43784 (time 1709788735.75 total 1609.20 last 27.08) - open/close 44058 (time 1709788745.77 total 1619.22 last 27.35) - open/close 44321 (time 1709788755.80 total 1629.25 last 26.21) - open/close 44575 (time 1709788765.83 total 1639.28 last 25.34) - open/close 44812 (time 1709788775.83 total 1649.28 last 23.69) - open/close 45048 (time 1709788785.84 total 1659.29 last 23.58) - open/close 45277 (time 1709788795.89 total 1669.35 last 22.78) - open/close 45503 (time 1709788805.90 total 1679.35 last 22.60) - open/close 45722 (time 1709788815.90 total 1689.35 last 21.88) - open/close 45935 (time 1709788825.93 total 1699.38 last 21.25) - open/close 46171 (time 1709788835.96 total 1709.41 last 23.52) - open/close 46420 (time 1709788845.97 total 1719.43 last 24.87) - open/close 46644 (time 1709788856.00 total 1729.45 last 22.35) - open/close 46874 (time 1709788866.03 total 1739.48 last 22.92) - open/close 47098 (time 1709788876.07 total 1749.52 last 22.31) - open/close 47324 (time 1709788886.10 total 1759.55 last 22.53) - open/close 47573 (time 1709788896.13 total 1769.58 last 24.82) - open/close 47797 (time 1709788906.17 total 1779.62 last 22.32) - open/close 48034 (time 1709788916.19 total 1789.65 last 23.64) - open/close 48273 (time 1709788926.22 total 1799.68 last 23.83) - open/close 48504 (time 1709788936.23 total 1809.68 last 23.09) - open/close 48726 (time 1709788946.24 total 1819.69 last 22.17) - open/close 48957 (time 1709788956.26 total 1829.71 last 23.07) - open/close 49187 (time 1709788966.27 total 1839.72 last 22.97) - open/close 49430 (time 1709788976.27 total 1849.72 last 24.30) - open/close 49671 (time 1709788986.29 total 1859.74 last 24.05) - open/close 49904 (time 1709788996.29 total 1869.74 last 23.29) total: 50000 open/close in 1873.91 seconds: 26.68 ops/second - unlinked 0 (time 1709789001 ; total 0 ; last 0) - unlinked 10000 (time 1709789485 ; total 484 ; last 484) - unlinked 20000 (time 1709789861 ; total 860 ; last 376) - unlinked 30000 (time 1709790092 ; total 1091 ; last 231) - unlinked 40000 (time 1709790218 ; total 1217 ; last 126) total: 50000 unlinks in 1269 seconds: 39.401104 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg328-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg328-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg328-server stop mds service on oleg328-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg328-server unloading modules on: 'oleg328-server' oleg328-server: oleg328-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg328-client: oleg328-client: ssh exited with exit code 2 pdsh@oleg328-client: oleg328-server: ssh exited with exit code 2 pdsh@oleg328-client: oleg328-client: ssh exited with exit code 2 pdsh@oleg328-client: oleg328-server: ssh exited with exit code 2 PASS 101a (3178s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg328-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg328-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg328-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg328-client: oleg328-server: ssh exited with exit code 2 oleg328-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg328-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == conf-sanity test 101b: Race events DISCONNECT and ACTIVE in osp ========================================================== 00:44:42 (1709790282) start mds service on oleg328-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg328-server' oleg328-server: oleg328-server.virtnet: executing load_modules_local oleg328-server: Loading modules from /home/green/git/lustre-release/lustre oleg328-server: detected 4 online CPUs by sysfs oleg328-server: Force libcfs to create 2 CPU partitions oleg328-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg328-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg328-server: oleg328-server.virtnet: executing set_default_debug -1 all pdsh@oleg328-client: oleg328-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg328-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg328-server: oleg328-server.virtnet: executing set_default_debug -1 all pdsh@oleg328-client: oleg328-server: ssh exited with exit code 1 Started lustre-OST0000 oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg328-client.virtnet: -o user_xattr,flock oleg328-server@tcp:/lustre /mnt/lustre fail_loc=0x80002107 fail_val=20 stop ost1 service on oleg328-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg328-server start ost1 service on oleg328-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg328-server: oleg328-server.virtnet: executing set_default_debug -1 all pdsh@oleg328-client: oleg328-server: ssh exited with exit code 1 Started lustre-OST0000 oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 5 sec oleg328-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff88012a60b800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff88012a60b800.ost_server_uuid in FULL state after 0 sec umount lustre on /mnt/lustre..... Stopping client oleg328-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg328-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg328-server stop mds service on oleg328-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg328-server unloading modules on: 'oleg328-server' oleg328-server: oleg328-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg328-client: oleg328-client: ssh exited with exit code 2 pdsh@oleg328-client: oleg328-server: ssh exited with exit code 2 pdsh@oleg328-client: oleg328-client: ssh exited with exit code 2 pdsh@oleg328-client: oleg328-server: ssh exited with exit code 2 PASS 101b (85s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg328-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg328-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory Stopping clients: oleg328-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg328-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg328-client: oleg328-server: ssh exited with exit code 2 oleg328-server: oleg328-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg328-server' oleg328-server: oleg328-server.virtnet: executing load_modules_local oleg328-server: Loading modules from /home/green/git/lustre-release/lustre oleg328-server: detected 4 online CPUs by sysfs oleg328-server: Force libcfs to create 2 CPU partitions oleg328-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg328-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 3763 sec ========== 00:46:28 (1709790388) === conf-sanity: start cleanup 00:46:29 (1709790389) === === conf-sanity: finish cleanup 00:46:29 (1709790389) ===