-----============= acceptance-small: conf-sanity ============----- Wed Mar 6 23:44:04 EST 2024 excepting tests: 32newtarball 110 skipping tests SLOW=no: 45 69 106 111 114 Stopping clients: oleg445-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg445-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg445-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg445-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg445-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg445-server oleg445-server: oleg445-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg445-server' oleg445-server: oleg445-server.virtnet: executing load_modules_local oleg445-server: Loading modules from /home/green/git/lustre-release/lustre oleg445-server: detected 4 online CPUs by sysfs oleg445-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg445-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg445-server: oleg445-server.virtnet: executing set_default_debug -1 all pdsh@oleg445-client: oleg445-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg445-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg445-server: oleg445-server.virtnet: executing set_default_debug -1 all pdsh@oleg445-client: oleg445-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg445-server: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg445-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg445-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg445-server stop mds service on oleg445-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg445-server debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 35b: Continue reconnection retries, if the active server is busy ========================================================== 23:45:35 (1709786735) start mds service on oleg445-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg445-server: oleg445-server.virtnet: executing set_default_debug -1 all pdsh@oleg445-client: oleg445-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg445-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg445-server: oleg445-server.virtnet: executing set_default_debug -1 all pdsh@oleg445-client: oleg445-server: ssh exited with exit code 1 Started lustre-OST0000 oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg445-client.virtnet: -o user_xattr,flock oleg445-server@tcp:/lustre /mnt/lustre debug=ha conf-sanity.sh test_35b 2024-03-0623h45m52s Set up a fake failnode for the MDS at_max=0 at_max=0 Injecting EBUSY on MDS fail_loc=0x80000136 mdc.lustre-MDT0000-mdc-ffff8800aac9e800.stats=clear Creating a test file and stat it File: '/mnt/lustre/d35b.conf-sanity/f35b.conf-sanity' Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115238826934274 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-03-06 23:46:37.000000000 -0500 Modify: 2024-03-06 23:46:37.000000000 -0500 Change: 2024-03-06 23:46:37.000000000 -0500 Birth: - Stop injecting EBUSY on MDS fail_loc=0 done at_max=600 at_max=600 Debug log: 76 lines, 76 kept, 0 dropped, 0 bad. umount lustre on /mnt/lustre..... Stopping client oleg445-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg445-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg445-server stop mds service on oleg445-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg445-server unloading modules on: 'oleg445-server' oleg445-server: oleg445-server.virtnet: executing unload_modules_local modules unloaded. oleg445-server: oleg445-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre oleg445-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg445-client: oleg445-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg445-server: oleg445-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre oleg445-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg445-client: oleg445-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg445-server: oleg445-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre oleg445-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg445-client: oleg445-server: ssh exited with exit code 19 checking for existing Lustre data: not found tunefs failed, reformatting instead Stopping clients: oleg445-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg445-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg445-client: oleg445-server: ssh exited with exit code 2 oleg445-server: oleg445-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg445-server' oleg445-server: oleg445-server.virtnet: executing load_modules_local oleg445-server: Loading modules from /home/green/git/lustre-release/lustre oleg445-server: detected 4 online CPUs by sysfs oleg445-server: Force libcfs to create 2 CPU partitions oleg445-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg445-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg445-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg445-server: oleg445-server.virtnet: executing set_default_debug -1 all pdsh@oleg445-client: oleg445-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg445-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg445-server: oleg445-server.virtnet: executing set_default_debug -1 all pdsh@oleg445-client: oleg445-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg445-server: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg445-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg445-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg445-server stop mds service on oleg445-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg445-server PASS 35b (143s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 23:48:01 (1709786881) start mds service on oleg445-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg445-server: oleg445-server.virtnet: executing set_default_debug -1 all pdsh@oleg445-client: oleg445-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg445-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg445-server: oleg445-server.virtnet: executing set_default_debug -1 all pdsh@oleg445-client: oleg445-server: ssh exited with exit code 1 Started lustre-OST0000 oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg445-client.virtnet: -o user_xattr,flock oleg445-server@tcp:/lustre /mnt/lustre copying 10 files to /mnt/lustre/d38.conf-sanity tar: Removing leading `/' from member names umount lustre on /mnt/lustre..... Stopping client oleg445-client.virtnet /mnt/lustre (opts:) osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=44 stop mds service on oleg445-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg445-server delete lov_objid file on MDS 000000 43 000008 start mds service on oleg445-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg445-server: oleg445-server.virtnet: executing set_default_debug -1 all pdsh@oleg445-client: oleg445-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg445-client.virtnet: -o user_xattr,flock oleg445-server@tcp:/lustre /mnt/lustre mount.lustre: mount oleg445-server@tcp:/lustre at /mnt/lustre failed: Input/output error Is the MGS running? conf-sanity test_38: @@@@@@ FAIL: mount_client /mnt/lustre failed Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:6975:error() = /home/green/git/lustre-release/lustre/tests/conf-sanity.sh:3521:test_38() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7315:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7375:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7186:run_test() = /home/green/git/lustre-release/lustre/tests/conf-sanity.sh:3569:main() Dumping lctl log to /tmp/testlogs//conf-sanity.test_38.*.1709786926.log rsync: chown "/tmp/testlogs/.conf-sanity.test_38.debug_log.oleg445-server.1709786926.log.3HYvkb" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.conf-sanity.test_38.dmesg.oleg445-server.1709786926.log.m4RkQ9" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] FAIL 38 (49s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 23:48:58 (1709786938) Stopping clients: oleg445-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg445-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg445-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg445-server oleg445-server: oleg445-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg445-server' oleg445-server: oleg445-server.virtnet: executing load_modules_local oleg445-server: Loading modules from /home/green/git/lustre-release/lustre oleg445-server: detected 4 online CPUs by sysfs oleg445-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg445-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg445-server: oleg445-server.virtnet: executing set_default_debug -1 all pdsh@oleg445-client: oleg445-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg445-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg445-server: oleg445-server.virtnet: executing set_default_debug -1 all pdsh@oleg445-client: oleg445-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg445-client.virtnet: -o user_xattr,flock oleg445-server@tcp:/lustre /mnt/lustre setup single mount lustre success fail_loc=0x142 start ost2 service on oleg445-server Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg445-server: oleg445-server.virtnet: executing set_default_debug -1 all pdsh@oleg445-client: oleg445-server: ssh exited with exit code 1 Commit the device label on lustre-ost2/ost2 Started lustre-OST0001 oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 1 sec stop ost2 service on oleg445-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg445-server umount lustre on /mnt/lustre..... Stopping client oleg445-client.virtnet /mnt/lustre (opts:-f) stop ost1 service on oleg445-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg445-server stop mds service on oleg445-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg445-server unloading modules on: 'oleg445-server' oleg445-server: oleg445-server.virtnet: executing unload_modules_local modules unloaded. oleg445-server: oleg445-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre oleg445-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg445-client: oleg445-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg445-server: oleg445-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre oleg445-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg445-client: oleg445-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg445-server: oleg445-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre oleg445-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg445-client: oleg445-server: ssh exited with exit code 19 checking for existing Lustre data: not found tunefs failed, reformatting instead Stopping clients: oleg445-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg445-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg445-client: oleg445-server: ssh exited with exit code 2 oleg445-server: oleg445-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg445-server' oleg445-server: oleg445-server.virtnet: executing load_modules_local oleg445-server: Loading modules from /home/green/git/lustre-release/lustre oleg445-server: detected 4 online CPUs by sysfs oleg445-server: Force libcfs to create 2 CPU partitions oleg445-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg445-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg445-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg445-server: oleg445-server.virtnet: executing set_default_debug -1 all pdsh@oleg445-client: oleg445-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg445-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg445-server: oleg445-server.virtnet: executing set_default_debug -1 all pdsh@oleg445-client: oleg445-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg445-server: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg445-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg445-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg445-server stop mds service on oleg445-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg445-server PASS 51 (192s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:52:14 (1709787134) start mds service on oleg445-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg445-server: oleg445-server.virtnet: executing set_default_debug -1 all pdsh@oleg445-client: oleg445-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg445-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg445-server: oleg445-server.virtnet: executing set_default_debug -1 all pdsh@oleg445-client: oleg445-server: ssh exited with exit code 1 Started lustre-OST0000 oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg445-client.virtnet: -o user_xattr,flock oleg445-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 181 (time 1709787165.30 total 10.51 last 17.22) - open/close 433 (time 1709787175.65 total 20.85 last 24.36) - open/close 692 (time 1709787185.87 total 31.08 last 25.33) - open/close 873 (time 1709787196.30 total 41.51 last 17.36) - open/close 1119 (time 1709787207.08 total 52.29 last 22.82) - open/close 1411 (time 1709787217.08 total 62.29 last 29.19) - open/close 1559 (time 1709787228.11 total 73.31 last 13.43) - open/close 1786 (time 1709787239.13 total 84.34 last 20.59) - open/close 1898 (time 1709787250.03 total 95.24 last 10.27) - open/close 2111 (time 1709787260.11 total 105.31 last 21.14) - open/close 2410 (time 1709787270.11 total 115.32 last 29.89) - open/close 2573 (time 1709787280.90 total 126.10 last 15.11) - open/close 2729 (time 1709787291.73 total 136.93 last 14.41) - open/close 2868 (time 1709787302.10 total 147.31 last 13.39) - open/close 3053 (time 1709787312.47 total 157.68 last 17.84) - open/close 3267 (time 1709787322.49 total 167.70 last 21.35) - open/close 3501 (time 1709787333.30 total 178.51 last 21.65) - open/close 3734 (time 1709787343.31 total 188.51 last 23.29) - open/close 3896 (time 1709787353.81 total 199.02 last 15.42) - open/close 4112 (time 1709787364.02 total 209.22 last 21.17) - open/close 4280 (time 1709787374.03 total 219.24 last 16.78) - open/close 4451 (time 1709787384.15 total 229.35 last 16.90) - open/close 4680 (time 1709787394.54 total 239.75 last 22.04) - open/close 4924 (time 1709787404.55 total 249.76 last 24.37) - open/close 5132 (time 1709787414.56 total 259.76 last 20.79) - open/close 5298 (time 1709787427.09 total 272.29 last 13.25) - open/close 5521 (time 1709787437.69 total 282.89 last 21.03) - open/close 5622 (time 1709787447.71 total 292.92 last 10.08) - open/close 5756 (time 1709787457.72 total 302.93 last 13.38) - open/close 5985 (time 1709787467.84 total 313.05 last 22.63) - open/close 6176 (time 1709787477.94 total 323.15 last 18.91) - open/close 6339 (time 1709787488.13 total 333.33 last 16.00) - open/close 6500 (time 1709787498.15 total 343.36 last 16.06) - open/close 6710 (time 1709787508.16 total 353.37 last 20.98) - open/close 6917 (time 1709787518.16 total 363.37 last 20.69) - open/close 7108 (time 1709787528.18 total 373.38 last 19.08) - open/close 7231 (time 1709787538.91 total 384.11 last 11.46) - open/close 7339 (time 1709787548.92 total 394.12 last 10.79) - open/close 7513 (time 1709787559.81 total 405.01 last 15.98) - open/close 7812 (time 1709787569.82 total 415.02 last 29.87) - open/close 7978 (time 1709787580.84 total 426.05 last 15.06) - open/close 8207 (time 1709787591.33 total 436.53 last 21.84) - open/close 8442 (time 1709787601.34 total 446.55 last 23.46) - open/close 8628 (time 1709787611.49 total 456.70 last 18.33) - open/close 8851 (time 1709787622.96 total 468.17 last 19.44) - open/close 9180 (time 1709787633.79 total 478.99 last 30.39) - open/close 9289 (time 1709787643.80 total 489.00 last 10.89) - open/close 9525 (time 1709787654.75 total 499.96 last 21.54) - open/close 9694 (time 1709787665.75 total 510.95 last 15.37) - open/close 9876 (time 1709787676.37 total 521.58 last 17.13) - open/close 10000 (time 1709787682.24 total 527.45 last 21.14) - open/close 10581 (time 1709787692.24 total 537.45 last 58.08) - open/close 11169 (time 1709787702.25 total 547.46 last 58.75) - open/close 11715 (time 1709787712.26 total 557.47 last 54.53) - open/close 12245 (time 1709787722.27 total 567.48 last 52.96) - open/close 12822 (time 1709787732.29 total 577.49 last 57.61) - open/close 13424 (time 1709787742.29 total 587.50 last 60.17) - open/close 13992 (time 1709787752.30 total 597.50 last 56.76) - open/close 14468 (time 1709787762.31 total 607.51 last 47.57) - open/close 14947 (time 1709787772.34 total 617.55 last 47.72) - open/close 15398 (time 1709787782.38 total 627.58 last 44.95) - open/close 15846 (time 1709787792.40 total 637.60 last 44.71) - open/close 16309 (time 1709787802.41 total 647.61 last 46.25) - open/close 16777 (time 1709787812.43 total 657.63 last 46.72) - open/close 17200 (time 1709787822.44 total 667.65 last 42.23) - open/close 17580 (time 1709787832.45 total 677.66 last 37.96) - open/close 18023 (time 1709787842.46 total 687.66 last 44.29) - open/close 18459 (time 1709787852.47 total 697.68 last 43.52) - open/close 18930 (time 1709787862.48 total 707.69 last 47.07) - open/close 19385 (time 1709787872.49 total 717.69 last 45.47) - open/close 19858 (time 1709787882.50 total 727.70 last 47.25) - open/close 20000 (time 1709787885.48 total 730.69 last 47.61) - open/close 20435 (time 1709787895.50 total 740.70 last 43.43) - open/close 20868 (time 1709787905.52 total 750.72 last 43.22) - open/close 21262 (time 1709787915.54 total 760.75 last 39.29) - open/close 21681 (time 1709787925.55 total 770.76 last 41.85) - open/close 22091 (time 1709787935.57 total 780.78 last 40.94) - open/close 22478 (time 1709787945.58 total 790.79 last 38.65) - open/close 22878 (time 1709787955.59 total 800.80 last 39.96) - open/close 23268 (time 1709787965.60 total 810.80 last 38.98) - open/close 23649 (time 1709787975.63 total 820.83 last 37.98) - open/close 24019 (time 1709787985.65 total 830.85 last 36.94) - open/close 24411 (time 1709787995.67 total 840.87 last 39.11) - open/close 24818 (time 1709788005.69 total 850.89 last 40.63) - open/close 25212 (time 1709788015.71 total 860.91 last 39.32) - open/close 25578 (time 1709788025.71 total 870.92 last 36.58) - open/close 25908 (time 1709788035.74 total 880.94 last 32.91) - open/close 26248 (time 1709788045.74 total 890.95 last 33.98) - open/close 26592 (time 1709788055.76 total 900.96 last 34.36) - open/close 26935 (time 1709788065.76 total 910.96 last 34.29) - open/close 27292 (time 1709788075.77 total 920.98 last 35.65) - open/close 27641 (time 1709788085.78 total 930.98 last 34.89) - open/close 27999 (time 1709788095.80 total 941.01 last 35.71) - open/close 28349 (time 1709788105.86 total 951.06 last 34.80) - open/close 28687 (time 1709788115.88 total 961.08 last 33.74) - open/close 29034 (time 1709788125.88 total 971.09 last 34.67) - open/close 29346 (time 1709788135.91 total 981.12 last 31.12) - open/close 29668 (time 1709788145.92 total 991.12 last 32.18) - open/close 29978 (time 1709788155.93 total 1001.13 last 30.97) - open/close 30262 (time 1709788165.93 total 1011.14 last 28.39) - open/close 30565 (time 1709788175.95 total 1021.15 last 30.25) - open/close 30895 (time 1709788185.98 total 1031.18 last 32.90) - open/close 31223 (time 1709788196.01 total 1041.22 last 32.70) - open/close 31517 (time 1709788206.02 total 1051.23 last 29.36) - open/close 31820 (time 1709788216.03 total 1061.23 last 30.29) - open/close 32105 (time 1709788226.05 total 1071.25 last 28.45) - open/close 32407 (time 1709788236.07 total 1081.28 last 30.11) - open/close 32679 (time 1709788246.08 total 1091.28 last 27.19) - open/close 32975 (time 1709788256.12 total 1101.32 last 29.48) - open/close 33293 (time 1709788266.12 total 1111.33 last 31.78) - open/close 33578 (time 1709788276.13 total 1121.34 last 28.48) - open/close 33882 (time 1709788286.16 total 1131.37 last 30.30) - open/close 34189 (time 1709788296.18 total 1141.38 last 30.65) - open/close 34487 (time 1709788306.18 total 1151.38 last 29.80) - open/close 34776 (time 1709788316.19 total 1161.39 last 28.88) - open/close 35060 (time 1709788326.20 total 1171.41 last 28.36) - open/close 35349 (time 1709788336.23 total 1181.43 last 28.82) - open/close 35620 (time 1709788346.26 total 1191.46 last 27.02) - open/close 35885 (time 1709788356.26 total 1201.47 last 26.49) - open/close 36133 (time 1709788366.29 total 1211.49 last 24.74) - open/close 36399 (time 1709788376.29 total 1221.50 last 26.59) - open/close 36666 (time 1709788386.32 total 1231.53 last 26.62) - open/close 36930 (time 1709788396.34 total 1241.54 last 26.35) - open/close 37202 (time 1709788406.35 total 1251.56 last 27.16) - open/close 37469 (time 1709788416.39 total 1261.59 last 26.61) - open/close 37723 (time 1709788426.42 total 1271.62 last 25.33) - open/close 37997 (time 1709788436.44 total 1281.65 last 27.33) - open/close 38258 (time 1709788446.45 total 1291.65 last 26.08) - open/close 38528 (time 1709788456.46 total 1301.67 last 26.96) - open/close 38797 (time 1709788466.49 total 1311.69 last 26.84) - open/close 39065 (time 1709788476.49 total 1321.69 last 26.79) - open/close 39324 (time 1709788486.51 total 1331.72 last 25.84) - open/close 39592 (time 1709788496.53 total 1341.73 last 26.76) - open/close 39668 (time 1709788510.53 total 1355.73 last 5.43) - open/close 39929 (time 1709788520.55 total 1365.76 last 26.03) - open/close 40000 (time 1709788523.69 total 1368.89 last 22.66) - open/close 40260 (time 1709788533.73 total 1378.93 last 25.90) - open/close 40517 (time 1709788543.74 total 1388.95 last 25.66) - open/close 40780 (time 1709788553.77 total 1398.98 last 26.22) - open/close 41037 (time 1709788563.78 total 1408.99 last 25.68) - open/close 41294 (time 1709788573.82 total 1419.02 last 25.61) - open/close 41553 (time 1709788583.84 total 1429.04 last 25.84) - open/close 41821 (time 1709788593.85 total 1439.05 last 26.77) - open/close 42087 (time 1709788603.85 total 1449.06 last 26.58) - open/close 42342 (time 1709788613.88 total 1459.08 last 25.45) - open/close 42607 (time 1709788623.88 total 1469.09 last 26.48) - open/close 42881 (time 1709788633.90 total 1479.11 last 27.34) - open/close 43153 (time 1709788643.93 total 1489.14 last 27.12) - open/close 43419 (time 1709788653.95 total 1499.16 last 26.55) - open/close 43682 (time 1709788663.97 total 1509.18 last 26.25) - open/close 43940 (time 1709788674.00 total 1519.21 last 25.72) - open/close 44205 (time 1709788684.03 total 1529.24 last 26.43) - open/close 44480 (time 1709788694.05 total 1539.25 last 27.46) - open/close 44740 (time 1709788704.05 total 1549.25 last 26.00) - open/close 44989 (time 1709788714.09 total 1559.29 last 24.80) - open/close 45221 (time 1709788724.10 total 1569.30 last 23.17) - open/close 45467 (time 1709788734.10 total 1579.30 last 24.60) - open/close 45711 (time 1709788744.13 total 1589.33 last 24.33) - open/close 45944 (time 1709788754.14 total 1599.34 last 23.28) - open/close 46188 (time 1709788764.16 total 1609.36 last 24.35) - open/close 46410 (time 1709788774.17 total 1619.37 last 22.19) - open/close 46634 (time 1709788784.17 total 1629.37 last 22.39) - open/close 46869 (time 1709788794.19 total 1639.40 last 23.44) - open/close 47095 (time 1709788804.20 total 1649.40 last 22.59) - open/close 47322 (time 1709788814.22 total 1659.42 last 22.66) - open/close 47548 (time 1709788824.24 total 1669.45 last 22.54) - open/close 47780 (time 1709788834.26 total 1679.46 last 23.16) - open/close 48006 (time 1709788844.29 total 1689.50 last 22.52) - open/close 48217 (time 1709788854.31 total 1699.51 last 21.07) - open/close 48445 (time 1709788864.33 total 1709.53 last 22.75) - open/close 48676 (time 1709788874.36 total 1719.56 last 23.03) - open/close 48900 (time 1709788884.40 total 1729.60 last 22.32) - open/close 49105 (time 1709788894.45 total 1739.66 last 20.39) - open/close 49311 (time 1709788904.48 total 1749.69 last 20.53) - open/close 49524 (time 1709788914.49 total 1759.70 last 21.28) - open/close 49725 (time 1709788924.49 total 1769.70 last 20.09) - open/close 49931 (time 1709788934.53 total 1779.73 last 20.53) total: 50000 open/close in 1782.99 seconds: 28.04 ops/second - unlinked 0 (time 1709788939 ; total 0 ; last 0) - unlinked 10000 (time 1709789458 ; total 519 ; last 519) - unlinked 20000 (time 1709789880 ; total 941 ; last 422) - unlinked 30000 (time 1709790158 ; total 1219 ; last 278) - unlinked 40000 (time 1709790261 ; total 1322 ; last 103) total: 50000 unlinks in 1357 seconds: 36.845985 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg445-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg445-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg445-server stop mds service on oleg445-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg445-server unloading modules on: 'oleg445-server' oleg445-server: oleg445-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg445-client: oleg445-client: ssh exited with exit code 2 pdsh@oleg445-client: oleg445-server: ssh exited with exit code 2 pdsh@oleg445-client: oleg445-client: ssh exited with exit code 2 pdsh@oleg445-client: oleg445-server: ssh exited with exit code 2 PASS 101a (3176s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg445-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg445-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg445-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg445-client: oleg445-server: ssh exited with exit code 2 oleg445-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg445-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == conf-sanity test 101b: Race events DISCONNECT and ACTIVE in osp ========================================================== 00:45:11 (1709790311) start mds service on oleg445-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg445-server' oleg445-server: oleg445-server.virtnet: executing load_modules_local oleg445-server: Loading modules from /home/green/git/lustre-release/lustre oleg445-server: detected 4 online CPUs by sysfs oleg445-server: Force libcfs to create 2 CPU partitions oleg445-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg445-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg445-server: oleg445-server.virtnet: executing set_default_debug -1 all pdsh@oleg445-client: oleg445-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg445-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg445-server: oleg445-server.virtnet: executing set_default_debug -1 all pdsh@oleg445-client: oleg445-server: ssh exited with exit code 1 Started lustre-OST0000 oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg445-client.virtnet: -o user_xattr,flock oleg445-server@tcp:/lustre /mnt/lustre fail_loc=0x80002107 fail_val=20 stop ost1 service on oleg445-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg445-server start ost1 service on oleg445-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg445-server: oleg445-server.virtnet: executing set_default_debug -1 all pdsh@oleg445-client: oleg445-server: ssh exited with exit code 1 Started lustre-OST0000 oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 7 sec oleg445-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8800aac98800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800aac98800.ost_server_uuid in FULL state after 0 sec umount lustre on /mnt/lustre..... Stopping client oleg445-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg445-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg445-server stop mds service on oleg445-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg445-server unloading modules on: 'oleg445-server' oleg445-server: oleg445-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg445-client: oleg445-client: ssh exited with exit code 2 pdsh@oleg445-client: oleg445-server: ssh exited with exit code 2 pdsh@oleg445-client: oleg445-client: ssh exited with exit code 2 pdsh@oleg445-client: oleg445-server: ssh exited with exit code 2 PASS 101b (81s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg445-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg445-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory Stopping clients: oleg445-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg445-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg445-client: oleg445-server: ssh exited with exit code 2 oleg445-server: oleg445-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg445-server' oleg445-server: oleg445-server.virtnet: executing load_modules_local oleg445-server: Loading modules from /home/green/git/lustre-release/lustre oleg445-server: detected 4 online CPUs by sysfs oleg445-server: Force libcfs to create 2 CPU partitions oleg445-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg445-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 3765 sec ========== 00:46:50 (1709790410) conf-sanity: FAIL: test_38 mount_client /mnt/lustre failed === conf-sanity: start cleanup 00:46:50 (1709790410) === === conf-sanity: finish cleanup 00:46:50 (1709790410) ===