-----============= acceptance-small: conf-sanity ============----- Wed Mar 6 23:41:53 EST 2024 excepting tests: 32newtarball 110 skipping tests SLOW=no: 45 69 106 111 114 Stopping clients: oleg319-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg319-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg319-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg319-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg319-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg319-server oleg319-server: oleg319-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory loading modules on: 'oleg319-server' oleg319-server: oleg319-server.virtnet: executing load_modules_local oleg319-server: Loading modules from /home/green/git/lustre-release/lustre oleg319-server: detected 4 online CPUs by sysfs oleg319-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg319-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg319-server: oleg319-server.virtnet: executing set_default_debug -1 all pdsh@oleg319-client: oleg319-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg319-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg319-server: oleg319-server.virtnet: executing set_default_debug -1 all pdsh@oleg319-client: oleg319-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg319-server: oleg319-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg319-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg319-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg319-server stop mds service on oleg319-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg319-server debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 35b: Continue reconnection retries, if the active server is busy ========================================================== 23:43:24 (1709786604) start mds service on oleg319-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg319-server: oleg319-server.virtnet: executing set_default_debug -1 all pdsh@oleg319-client: oleg319-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg319-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg319-server: oleg319-server.virtnet: executing set_default_debug -1 all pdsh@oleg319-client: oleg319-server: ssh exited with exit code 1 Started lustre-OST0000 oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg319-client.virtnet: -o user_xattr,flock oleg319-server@tcp:/lustre /mnt/lustre debug=ha conf-sanity.sh test_35b 2024-03-0623h43m46s Set up a fake failnode for the MDS at_max=0 at_max=0 Injecting EBUSY on MDS fail_loc=0x80000136 mdc.lustre-MDT0000-mdc-ffff8800b6d04800.stats=clear Creating a test file and stat it File: '/mnt/lustre/d35b.conf-sanity/f35b.conf-sanity' Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115238826934274 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-03-06 23:44:33.000000000 -0500 Modify: 2024-03-06 23:44:33.000000000 -0500 Change: 2024-03-06 23:44:33.000000000 -0500 Birth: - Stop injecting EBUSY on MDS fail_loc=0 done at_max=600 at_max=600 Debug log: 74 lines, 74 kept, 0 dropped, 0 bad. umount lustre on /mnt/lustre..... Stopping client oleg319-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg319-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg319-server stop mds service on oleg319-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg319-server unloading modules on: 'oleg319-server' oleg319-server: oleg319-server.virtnet: executing unload_modules_local modules unloaded. oleg319-server: oleg319-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre oleg319-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg319-client: oleg319-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg319-server: oleg319-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre oleg319-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg319-client: oleg319-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg319-server: oleg319-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre oleg319-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg319-client: oleg319-server: ssh exited with exit code 19 checking for existing Lustre data: not found tunefs failed, reformatting instead Stopping clients: oleg319-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg319-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg319-client: oleg319-server: ssh exited with exit code 2 oleg319-server: oleg319-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg319-server' oleg319-server: oleg319-server.virtnet: executing load_modules_local oleg319-server: Loading modules from /home/green/git/lustre-release/lustre oleg319-server: detected 4 online CPUs by sysfs oleg319-server: Force libcfs to create 2 CPU partitions oleg319-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg319-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg319-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg319-server: oleg319-server.virtnet: executing set_default_debug -1 all pdsh@oleg319-client: oleg319-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg319-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg319-server: oleg319-server.virtnet: executing set_default_debug -1 all pdsh@oleg319-client: oleg319-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg319-server: oleg319-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg319-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg319-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg319-server stop mds service on oleg319-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg319-server PASS 35b (141s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 23:45:47 (1709786747) start mds service on oleg319-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg319-server: oleg319-server.virtnet: executing set_default_debug -1 all pdsh@oleg319-client: oleg319-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg319-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg319-server: oleg319-server.virtnet: executing set_default_debug -1 all pdsh@oleg319-client: oleg319-server: ssh exited with exit code 1 Started lustre-OST0000 oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg319-client.virtnet: -o user_xattr,flock oleg319-server@tcp:/lustre /mnt/lustre copying 10 files to /mnt/lustre/d38.conf-sanity tar: Removing leading `/' from member names umount lustre on /mnt/lustre..... Stopping client oleg319-client.virtnet /mnt/lustre (opts:) osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=44 stop mds service on oleg319-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg319-server delete lov_objid file on MDS 000000 43 000008 start mds service on oleg319-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg319-server: oleg319-server.virtnet: executing set_default_debug -1 all pdsh@oleg319-client: oleg319-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg319-client.virtnet: -o user_xattr,flock oleg319-server@tcp:/lustre /mnt/lustre osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=2 umount lustre on /mnt/lustre..... Stopping client oleg319-client.virtnet /mnt/lustre (opts:) stop mds service on oleg319-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg319-server 000000 2 000008 oleg319-server: 8+0 records in oleg319-server: 8+0 records out oleg319-server: 4096 bytes (4.1 kB) copied, 0.000664598 s, 6.2 MB/s start mds service on oleg319-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg319-server: oleg319-server.virtnet: executing set_default_debug -1 all pdsh@oleg319-client: oleg319-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg319-client.virtnet: -o user_xattr,flock oleg319-server@tcp:/lustre /mnt/lustre osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=35 umount lustre on /mnt/lustre..... Stopping client oleg319-client.virtnet /mnt/lustre (opts:) stop mds service on oleg319-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg319-server 000000 34 000008 files compared the same umount lustre on /mnt/lustre..... stop ost1 service on oleg319-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg319-server stop mds service on oleg319-server unloading modules on: 'oleg319-server' oleg319-server: oleg319-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg319-client: oleg319-client: ssh exited with exit code 2 pdsh@oleg319-client: oleg319-server: ssh exited with exit code 2 pdsh@oleg319-client: oleg319-client: ssh exited with exit code 2 pdsh@oleg319-client: oleg319-server: ssh exited with exit code 2 PASS 38 (76s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg319-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg319-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg319-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg319-client: oleg319-server: ssh exited with exit code 2 oleg319-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg319-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 23:47:06 (1709786826) Stopping clients: oleg319-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg319-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg319-client: oleg319-server: ssh exited with exit code 2 oleg319-server: oleg319-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg319-server' oleg319-server: oleg319-server.virtnet: executing load_modules_local oleg319-server: Loading modules from /home/green/git/lustre-release/lustre oleg319-server: detected 4 online CPUs by sysfs oleg319-server: Force libcfs to create 2 CPU partitions oleg319-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg319-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg319-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg319-server: oleg319-server.virtnet: executing set_default_debug -1 all pdsh@oleg319-client: oleg319-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg319-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg319-server: oleg319-server.virtnet: executing set_default_debug -1 all pdsh@oleg319-client: oleg319-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg319-client.virtnet: -o user_xattr,flock oleg319-server@tcp:/lustre /mnt/lustre setup single mount lustre success fail_loc=0x142 start ost2 service on oleg319-server Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg319-server: oleg319-server.virtnet: executing set_default_debug -1 all pdsh@oleg319-client: oleg319-server: ssh exited with exit code 1 Commit the device label on lustre-ost2/ost2 Started lustre-OST0001 oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 2 sec stop ost2 service on oleg319-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg319-server umount lustre on /mnt/lustre..... Stopping client oleg319-client.virtnet /mnt/lustre (opts:-f) stop ost1 service on oleg319-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg319-server stop mds service on oleg319-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg319-server unloading modules on: 'oleg319-server' oleg319-server: oleg319-server.virtnet: executing unload_modules_local modules unloaded. oleg319-server: oleg319-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre oleg319-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg319-client: oleg319-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg319-server: oleg319-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre oleg319-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg319-client: oleg319-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg319-server: oleg319-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre oleg319-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg319-client: oleg319-server: ssh exited with exit code 19 checking for existing Lustre data: not found tunefs failed, reformatting instead Stopping clients: oleg319-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg319-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg319-client: oleg319-server: ssh exited with exit code 2 oleg319-server: oleg319-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg319-server' oleg319-server: oleg319-server.virtnet: executing load_modules_local oleg319-server: Loading modules from /home/green/git/lustre-release/lustre oleg319-server: detected 4 online CPUs by sysfs oleg319-server: Force libcfs to create 2 CPU partitions oleg319-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg319-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg319-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg319-server: oleg319-server.virtnet: executing set_default_debug -1 all pdsh@oleg319-client: oleg319-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg319-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg319-server: oleg319-server.virtnet: executing set_default_debug -1 all pdsh@oleg319-client: oleg319-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg319-server: oleg319-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg319-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg319-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg319-server stop mds service on oleg319-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg319-server PASS 51 (180s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:50:10 (1709787010) start mds service on oleg319-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg319-server: oleg319-server.virtnet: executing set_default_debug -1 all pdsh@oleg319-client: oleg319-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg319-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg319-server: oleg319-server.virtnet: executing set_default_debug -1 all pdsh@oleg319-client: oleg319-server: ssh exited with exit code 1 Started lustre-OST0000 oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg319-client.virtnet: -o user_xattr,flock oleg319-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 131 (time 1709787041.25 total 10.56 last 12.41) - open/close 421 (time 1709787051.96 total 21.27 last 27.06) - open/close 721 (time 1709787062.70 total 32.00 last 27.95) - open/close 944 (time 1709787073.44 total 42.75 last 20.75) - open/close 1165 (time 1709787083.44 total 52.75 last 22.10) - open/close 1349 (time 1709787093.63 total 62.94 last 18.06) - open/close 1493 (time 1709787103.66 total 72.96 last 14.36) - open/close 1677 (time 1709787113.67 total 82.97 last 18.38) - open/close 1848 (time 1709787124.43 total 93.73 last 15.89) - open/close 2135 (time 1709787135.20 total 104.51 last 26.63) - open/close 2384 (time 1709787145.50 total 114.81 last 24.17) - open/close 2623 (time 1709787155.69 total 124.99 last 23.47) - open/close 2757 (time 1709787165.72 total 135.02 last 13.36) - open/close 3023 (time 1709787175.92 total 145.22 last 26.08) - open/close 3148 (time 1709787185.97 total 155.27 last 12.44) - open/close 3308 (time 1709787196.92 total 166.23 last 14.61) - open/close 3434 (time 1709787206.94 total 176.25 last 12.57) - open/close 3646 (time 1709787217.64 total 186.94 last 19.83) - open/close 3813 (time 1709787228.08 total 197.38 last 15.99) - open/close 3967 (time 1709787238.24 total 207.55 last 15.15) - open/close 4123 (time 1709787249.03 total 218.34 last 14.46) - open/close 4220 (time 1709787259.66 total 228.97 last 9.13) - open/close 4454 (time 1709787269.66 total 238.97 last 23.40) - open/close 4599 (time 1709787280.24 total 249.54 last 13.71) - open/close 4772 (time 1709787290.33 total 259.63 last 17.15) - open/close 4884 (time 1709787300.33 total 269.64 last 11.19) - open/close 5054 (time 1709787311.33 total 280.64 last 15.46) - open/close 5290 (time 1709787321.34 total 290.65 last 23.57) - open/close 5441 (time 1709787331.36 total 300.66 last 15.08) - open/close 5620 (time 1709787342.33 total 311.64 last 16.31) - open/close 5744 (time 1709787353.78 total 323.08 last 10.83) - open/close 5895 (time 1709787364.71 total 334.01 last 13.81) - open/close 6072 (time 1709787374.99 total 344.30 last 17.22) - open/close 6247 (time 1709787385.68 total 354.99 last 16.36) - open/close 6348 (time 1709787395.78 total 365.09 last 10.00) - open/close 6400 (time 1709787413.42 total 382.73 last 2.95) - open/close 6533 (time 1709787423.76 total 393.07 last 12.86) - open/close 6749 (time 1709787433.77 total 403.07 last 21.60) - open/close 6893 (time 1709787443.78 total 413.09 last 14.37) - open/close 6989 (time 1709787455.83 total 425.13 last 7.97) - open/close 7127 (time 1709787466.28 total 435.59 last 13.21) - open/close 7290 (time 1709787477.30 total 446.61 last 14.79) - open/close 7471 (time 1709787487.90 total 457.21 last 17.07) - open/close 7649 (time 1709787497.93 total 467.24 last 17.75) - open/close 7800 (time 1709787510.40 total 479.71 last 12.11) - open/close 7960 (time 1709787520.72 total 490.02 last 15.51) - open/close 8085 (time 1709787530.90 total 500.20 last 12.28) - open/close 8202 (time 1709787541.38 total 510.69 last 11.16) - open/close 8350 (time 1709787551.55 total 520.85 last 14.56) - open/close 8490 (time 1709787562.11 total 531.42 last 13.25) - open/close 8671 (time 1709787572.86 total 542.17 last 16.83) - open/close 8913 (time 1709787583.90 total 553.21 last 21.93) - open/close 9063 (time 1709787594.42 total 563.73 last 14.26) - open/close 9254 (time 1709787604.47 total 573.78 last 19.01) - open/close 9446 (time 1709787615.03 total 584.33 last 18.19) - open/close 9627 (time 1709787625.03 total 594.34 last 18.09) - open/close 10000 (time 1709787632.18 total 601.49 last 52.17) - open/close 10212 (time 1709787647.59 total 616.90 last 13.75) - open/close 10713 (time 1709787657.60 total 626.91 last 50.07) - open/close 11193 (time 1709787667.64 total 636.95 last 47.81) - open/close 11652 (time 1709787677.65 total 646.95 last 45.87) - open/close 12106 (time 1709787687.67 total 656.98 last 45.28) - open/close 12514 (time 1709787697.69 total 667.00 last 40.72) - open/close 12895 (time 1709787707.70 total 677.00 last 38.08) - open/close 13309 (time 1709787717.70 total 687.00 last 41.40) - open/close 13780 (time 1709787727.71 total 697.02 last 47.04) - open/close 14233 (time 1709787737.72 total 707.02 last 45.26) - open/close 14595 (time 1709787747.72 total 717.03 last 36.19) - open/close 14958 (time 1709787757.72 total 727.03 last 36.30) - open/close 15390 (time 1709787767.74 total 737.04 last 43.15) - open/close 15797 (time 1709787777.74 total 747.05 last 40.67) - open/close 16200 (time 1709787787.75 total 757.06 last 40.25) - open/close 16631 (time 1709787797.76 total 767.07 last 43.05) - open/close 17054 (time 1709787807.77 total 777.07 last 42.29) - open/close 17472 (time 1709787817.79 total 787.10 last 41.69) - open/close 17593 (time 1709787831.80 total 801.11 last 8.64) - open/close 18000 (time 1709787841.82 total 811.13 last 40.62) - open/close 18409 (time 1709787851.85 total 821.15 last 40.81) - open/close 18827 (time 1709787861.85 total 831.16 last 41.77) - open/close 19191 (time 1709787871.90 total 841.21 last 36.23) - open/close 19545 (time 1709787881.92 total 851.23 last 35.32) - open/close 19924 (time 1709787891.93 total 861.24 last 37.87) - open/close 20000 (time 1709787893.98 total 863.29 last 37.05) - open/close 20382 (time 1709787903.98 total 873.29 last 38.20) - open/close 20768 (time 1709787913.99 total 883.30 last 38.57) - open/close 21115 (time 1709787924.03 total 893.34 last 34.56) - open/close 21431 (time 1709787934.06 total 903.37 last 31.50) - open/close 21773 (time 1709787944.07 total 913.37 last 34.19) - open/close 22115 (time 1709787954.09 total 923.39 last 34.14) - open/close 22462 (time 1709787964.09 total 933.39 last 34.69) - open/close 22807 (time 1709787974.11 total 943.41 last 34.43) - open/close 23145 (time 1709787984.12 total 953.42 last 33.77) - open/close 23515 (time 1709787994.12 total 963.42 last 37.00) - open/close 23888 (time 1709788004.14 total 973.44 last 37.22) - open/close 24241 (time 1709788014.14 total 983.45 last 35.28) - open/close 24596 (time 1709788024.16 total 993.47 last 35.42) - open/close 24913 (time 1709788034.19 total 1003.50 last 31.62) - open/close 25244 (time 1709788044.19 total 1013.50 last 33.09) - open/close 25569 (time 1709788054.21 total 1023.52 last 32.45) - open/close 25884 (time 1709788064.22 total 1033.53 last 31.47) - open/close 26224 (time 1709788074.22 total 1043.53 last 33.99) - open/close 26640 (time 1709788084.24 total 1053.55 last 41.53) - open/close 27027 (time 1709788094.25 total 1063.55 last 38.67) - open/close 27414 (time 1709788104.26 total 1073.57 last 38.65) - open/close 27800 (time 1709788114.28 total 1083.59 last 38.52) - open/close 28193 (time 1709788124.30 total 1093.61 last 39.22) - open/close 28574 (time 1709788134.30 total 1103.61 last 38.10) - open/close 28965 (time 1709788144.33 total 1113.63 last 39.00) - open/close 29327 (time 1709788154.35 total 1123.66 last 36.10) - open/close 29701 (time 1709788164.35 total 1133.66 last 37.40) - open/close 30000 (time 1709788172.28 total 1141.58 last 37.72) - open/close 30359 (time 1709788182.30 total 1151.61 last 35.82) - open/close 30717 (time 1709788192.34 total 1161.65 last 35.65) - open/close 31059 (time 1709788202.35 total 1171.65 last 34.18) - open/close 31409 (time 1709788212.35 total 1181.66 last 35.00) - open/close 31767 (time 1709788222.37 total 1191.67 last 35.74) - open/close 32108 (time 1709788232.38 total 1201.69 last 34.05) - open/close 32443 (time 1709788242.38 total 1211.69 last 33.49) - open/close 32775 (time 1709788252.39 total 1221.70 last 33.17) - open/close 33105 (time 1709788262.40 total 1231.71 last 32.96) - open/close 33431 (time 1709788272.41 total 1241.72 last 32.57) - open/close 33754 (time 1709788282.42 total 1251.73 last 32.27) - open/close 34054 (time 1709788292.45 total 1261.75 last 29.92) - open/close 34364 (time 1709788302.47 total 1271.77 last 30.93) - open/close 34670 (time 1709788312.48 total 1281.79 last 30.55) - open/close 34973 (time 1709788322.50 total 1291.81 last 30.24) - open/close 35293 (time 1709788332.52 total 1301.83 last 31.93) - open/close 35613 (time 1709788342.54 total 1311.85 last 31.93) - open/close 35932 (time 1709788352.58 total 1321.88 last 31.80) - open/close 36259 (time 1709788362.60 total 1331.91 last 32.61) - open/close 36554 (time 1709788372.63 total 1341.94 last 29.42) - open/close 36866 (time 1709788382.63 total 1351.94 last 31.20) - open/close 37169 (time 1709788392.66 total 1361.96 last 30.22) - open/close 37474 (time 1709788402.68 total 1371.99 last 30.43) - open/close 37773 (time 1709788412.70 total 1382.00 last 29.85) - open/close 38090 (time 1709788422.70 total 1392.01 last 31.68) - open/close 38384 (time 1709788432.71 total 1402.01 last 29.40) - open/close 38705 (time 1709788442.72 total 1412.03 last 32.05) - open/close 38992 (time 1709788452.73 total 1422.04 last 28.67) - open/close 39282 (time 1709788462.75 total 1432.05 last 28.96) - open/close 39579 (time 1709788472.76 total 1442.07 last 29.67) - open/close 39879 (time 1709788482.77 total 1452.08 last 29.97) - open/close 40000 (time 1709788487.29 total 1456.60 last 26.75) - open/close 40261 (time 1709788497.33 total 1466.63 last 26.01) - open/close 40548 (time 1709788507.36 total 1476.66 last 28.61) - open/close 40842 (time 1709788517.37 total 1486.67 last 29.38) - open/close 41126 (time 1709788527.40 total 1496.70 last 28.31) - open/close 41409 (time 1709788537.43 total 1506.73 last 28.21) - open/close 41699 (time 1709788547.44 total 1516.74 last 28.98) - open/close 41976 (time 1709788557.45 total 1526.76 last 27.66) - open/close 42257 (time 1709788567.46 total 1536.77 last 28.08) - open/close 42545 (time 1709788577.48 total 1546.78 last 28.74) - open/close 42830 (time 1709788587.49 total 1556.80 last 28.46) - open/close 43106 (time 1709788597.53 total 1566.83 last 27.51) - open/close 43391 (time 1709788607.54 total 1576.85 last 28.46) - open/close 43669 (time 1709788617.57 total 1586.87 last 27.73) - open/close 43941 (time 1709788627.57 total 1596.87 last 27.19) - open/close 44203 (time 1709788637.57 total 1606.88 last 26.19) - open/close 44418 (time 1709788647.59 total 1616.90 last 21.47) - open/close 44617 (time 1709788657.62 total 1626.93 last 19.84) - open/close 44820 (time 1709788667.64 total 1636.95 last 20.26) - open/close 45063 (time 1709788677.67 total 1646.97 last 24.24) - open/close 45314 (time 1709788687.68 total 1656.99 last 25.07) - open/close 45569 (time 1709788697.68 total 1666.99 last 25.50) - open/close 45816 (time 1709788707.69 total 1676.99 last 24.68) - open/close 46055 (time 1709788717.71 total 1687.02 last 23.84) - open/close 46312 (time 1709788727.72 total 1697.03 last 25.67) - open/close 46582 (time 1709788737.75 total 1707.06 last 26.93) - open/close 46848 (time 1709788747.77 total 1717.07 last 26.56) - open/close 47106 (time 1709788757.78 total 1727.09 last 25.75) - open/close 47354 (time 1709788767.82 total 1737.13 last 24.70) - open/close 47593 (time 1709788777.83 total 1747.14 last 23.88) - open/close 47835 (time 1709788787.85 total 1757.16 last 24.16) - open/close 48060 (time 1709788797.90 total 1767.21 last 22.38) - open/close 48297 (time 1709788807.95 total 1777.25 last 23.60) - open/close 48528 (time 1709788817.99 total 1787.29 last 23.01) - open/close 48746 (time 1709788827.99 total 1797.30 last 21.79) - open/close 48991 (time 1709788838.01 total 1807.31 last 24.46) - open/close 49230 (time 1709788848.03 total 1817.33 last 23.85) - open/close 49464 (time 1709788858.05 total 1827.35 last 23.36) - open/close 49685 (time 1709788868.05 total 1837.36 last 22.08) - open/close 49909 (time 1709788878.06 total 1847.37 last 22.38) total: 50000 open/close in 1851.37 seconds: 27.01 ops/second - unlinked 0 (time 1709788883 ; total 0 ; last 0) - unlinked 10000 (time 1709789342 ; total 459 ; last 459) - unlinked 20000 (time 1709789715 ; total 832 ; last 373) - unlinked 30000 (time 1709789966 ; total 1083 ; last 251) - unlinked 40000 (time 1709790082 ; total 1199 ; last 116) total: 50000 unlinks in 1251 seconds: 39.968025 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg319-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg319-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg319-server stop mds service on oleg319-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg319-server unloading modules on: 'oleg319-server' oleg319-server: oleg319-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg319-client: oleg319-client: ssh exited with exit code 2 pdsh@oleg319-client: oleg319-server: ssh exited with exit code 2 pdsh@oleg319-client: oleg319-client: ssh exited with exit code 2 pdsh@oleg319-client: oleg319-server: ssh exited with exit code 2 PASS 101a (3135s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg319-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg319-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg319-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg319-client: oleg319-server: ssh exited with exit code 2 oleg319-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg319-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == conf-sanity test 101b: Race events DISCONNECT and ACTIVE in osp ========================================================== 00:42:27 (1709790147) start mds service on oleg319-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg319-server' oleg319-server: oleg319-server.virtnet: executing load_modules_local oleg319-server: Loading modules from /home/green/git/lustre-release/lustre oleg319-server: detected 4 online CPUs by sysfs oleg319-server: Force libcfs to create 2 CPU partitions oleg319-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg319-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg319-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg319-server: oleg319-server.virtnet: executing set_default_debug -1 all pdsh@oleg319-client: oleg319-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg319-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg319-server: oleg319-server.virtnet: executing set_default_debug -1 all pdsh@oleg319-client: oleg319-server: ssh exited with exit code 1 Started lustre-OST0000 oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg319-client.virtnet: -o user_xattr,flock oleg319-server@tcp:/lustre /mnt/lustre fail_loc=0x80002107 fail_val=20 stop ost1 service on oleg319-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg319-server start ost1 service on oleg319-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg319-server: oleg319-server.virtnet: executing set_default_debug -1 all pdsh@oleg319-client: oleg319-server: ssh exited with exit code 1 Started lustre-OST0000 oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 5 sec oleg319-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8800aa9cf000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800aa9cf000.ost_server_uuid in FULL state after 0 sec umount lustre on /mnt/lustre..... Stopping client oleg319-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg319-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg319-server stop mds service on oleg319-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg319-server unloading modules on: 'oleg319-server' oleg319-server: oleg319-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg319-client: oleg319-client: ssh exited with exit code 2 pdsh@oleg319-client: oleg319-server: ssh exited with exit code 2 pdsh@oleg319-client: oleg319-client: ssh exited with exit code 2 pdsh@oleg319-client: oleg319-server: ssh exited with exit code 2 PASS 101b (89s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg319-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg319-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory Stopping clients: oleg319-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg319-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg319-client: oleg319-server: ssh exited with exit code 2 oleg319-server: oleg319-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg319-server' oleg319-server: oleg319-server.virtnet: executing load_modules_local oleg319-server: Loading modules from /home/green/git/lustre-release/lustre oleg319-server: detected 4 online CPUs by sysfs oleg319-server: Force libcfs to create 2 CPU partitions oleg319-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg319-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 3746 sec ========== 00:44:20 (1709790260) === conf-sanity: start cleanup 00:44:21 (1709790261) === === conf-sanity: finish cleanup 00:44:21 (1709790261) ===