-----============= acceptance-small: conf-sanity ============----- Wed Mar 6 23:39:07 EST 2024 excepting tests: 32newtarball 110 skipping tests SLOW=no: 45 69 106 111 114 Stopping clients: oleg344-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg344-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg344-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg344-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg344-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg344-server oleg344-server: oleg344-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg344-server' oleg344-server: oleg344-server.virtnet: executing load_modules_local oleg344-server: Loading modules from /home/green/git/lustre-release/lustre oleg344-server: detected 4 online CPUs by sysfs oleg344-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg344-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg344-server: oleg344-server.virtnet: executing set_default_debug -1 all pdsh@oleg344-client: oleg344-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg344-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg344-server: oleg344-server.virtnet: executing set_default_debug -1 all pdsh@oleg344-client: oleg344-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg344-server: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg344-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg344-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg344-server stop mds service on oleg344-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg344-server debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 35b: Continue reconnection retries, if the active server is busy ========================================================== 23:40:00 (1709786400) start mds service on oleg344-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg344-server: oleg344-server.virtnet: executing set_default_debug -1 all pdsh@oleg344-client: oleg344-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg344-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg344-server: oleg344-server.virtnet: executing set_default_debug -1 all pdsh@oleg344-client: oleg344-server: ssh exited with exit code 1 Started lustre-OST0000 oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg344-client.virtnet: -o user_xattr,flock oleg344-server@tcp:/lustre /mnt/lustre debug=ha conf-sanity.sh test_35b 2024-03-0623h40m15s Set up a fake failnode for the MDS at_max=0 at_max=0 Injecting EBUSY on MDS fail_loc=0x80000136 mdc.lustre-MDT0000-mdc-ffff88012aacd000.stats=clear Creating a test file and stat it File: '/mnt/lustre/d35b.conf-sanity/f35b.conf-sanity' Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115238826934274 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-03-06 23:40:59.000000000 -0500 Modify: 2024-03-06 23:40:59.000000000 -0500 Change: 2024-03-06 23:40:59.000000000 -0500 Birth: - Stop injecting EBUSY on MDS fail_loc=0 done at_max=600 at_max=600 Debug log: 54 lines, 54 kept, 0 dropped, 0 bad. umount lustre on /mnt/lustre..... Stopping client oleg344-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg344-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg344-server stop mds service on oleg344-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg344-server unloading modules on: 'oleg344-server' oleg344-server: oleg344-server.virtnet: executing unload_modules_local modules unloaded. oleg344-server: oleg344-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre oleg344-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg344-client: oleg344-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg344-server: oleg344-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre oleg344-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg344-client: oleg344-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg344-server: oleg344-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre oleg344-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg344-client: oleg344-server: ssh exited with exit code 19 checking for existing Lustre data: not found tunefs failed, reformatting instead Stopping clients: oleg344-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg344-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg344-client: oleg344-server: ssh exited with exit code 2 oleg344-server: oleg344-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg344-server' oleg344-server: oleg344-server.virtnet: executing load_modules_local oleg344-server: Loading modules from /home/green/git/lustre-release/lustre oleg344-server: detected 4 online CPUs by sysfs oleg344-server: Force libcfs to create 2 CPU partitions oleg344-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg344-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg344-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg344-server: oleg344-server.virtnet: executing set_default_debug -1 all pdsh@oleg344-client: oleg344-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg344-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg344-server: oleg344-server.virtnet: executing set_default_debug -1 all pdsh@oleg344-client: oleg344-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg344-server: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg344-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg344-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg344-server stop mds service on oleg344-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg344-server PASS 35b (115s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 23:41:57 (1709786517) start mds service on oleg344-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg344-server: oleg344-server.virtnet: executing set_default_debug -1 all pdsh@oleg344-client: oleg344-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg344-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg344-server: oleg344-server.virtnet: executing set_default_debug -1 all pdsh@oleg344-client: oleg344-server: ssh exited with exit code 1 Started lustre-OST0000 oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg344-client.virtnet: -o user_xattr,flock oleg344-server@tcp:/lustre /mnt/lustre copying 10 files to /mnt/lustre/d38.conf-sanity tar: Removing leading `/' from member names umount lustre on /mnt/lustre..... Stopping client oleg344-client.virtnet /mnt/lustre (opts:) osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=44 stop mds service on oleg344-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg344-server delete lov_objid file on MDS 000000 43 000008 start mds service on oleg344-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg344-server: oleg344-server.virtnet: executing set_default_debug -1 all pdsh@oleg344-client: oleg344-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg344-client.virtnet: -o user_xattr,flock oleg344-server@tcp:/lustre /mnt/lustre mount.lustre: mount oleg344-server@tcp:/lustre at /mnt/lustre failed: Input/output error Is the MGS running? conf-sanity test_38: @@@@@@ FAIL: mount_client /mnt/lustre failed Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:6975:error() = /home/green/git/lustre-release/lustre/tests/conf-sanity.sh:3521:test_38() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7315:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7375:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7186:run_test() = /home/green/git/lustre-release/lustre/tests/conf-sanity.sh:3569:main() Dumping lctl log to /tmp/testlogs//conf-sanity.test_38.*.1709786555.log rsync: chown "/tmp/testlogs/.conf-sanity.test_38.debug_log.oleg344-server.1709786555.log.zTyP8Z" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.conf-sanity.test_38.dmesg.oleg344-server.1709786555.log.3bF3OS" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] FAIL 38 (42s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 23:42:45 (1709786565) Stopping clients: oleg344-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg344-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg344-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg344-server oleg344-server: oleg344-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg344-server' oleg344-server: oleg344-server.virtnet: executing load_modules_local oleg344-server: Loading modules from /home/green/git/lustre-release/lustre oleg344-server: detected 4 online CPUs by sysfs oleg344-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg344-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg344-server: oleg344-server.virtnet: executing set_default_debug -1 all pdsh@oleg344-client: oleg344-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg344-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg344-server: oleg344-server.virtnet: executing set_default_debug -1 all pdsh@oleg344-client: oleg344-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg344-client.virtnet: -o user_xattr,flock oleg344-server@tcp:/lustre /mnt/lustre setup single mount lustre success fail_loc=0x142 start ost2 service on oleg344-server Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg344-server: oleg344-server.virtnet: executing set_default_debug -1 all pdsh@oleg344-client: oleg344-server: ssh exited with exit code 1 Commit the device label on lustre-ost2/ost2 Started lustre-OST0001 oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec stop ost2 service on oleg344-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg344-server umount lustre on /mnt/lustre..... Stopping client oleg344-client.virtnet /mnt/lustre (opts:-f) stop ost1 service on oleg344-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg344-server stop mds service on oleg344-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg344-server unloading modules on: 'oleg344-server' oleg344-server: oleg344-server.virtnet: executing unload_modules_local modules unloaded. oleg344-server: oleg344-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre oleg344-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg344-client: oleg344-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg344-server: oleg344-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre oleg344-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg344-client: oleg344-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg344-server: oleg344-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre oleg344-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg344-client: oleg344-server: ssh exited with exit code 19 checking for existing Lustre data: not found tunefs failed, reformatting instead Stopping clients: oleg344-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg344-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg344-client: oleg344-server: ssh exited with exit code 2 oleg344-server: oleg344-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg344-server' oleg344-server: oleg344-server.virtnet: executing load_modules_local oleg344-server: Loading modules from /home/green/git/lustre-release/lustre oleg344-server: detected 4 online CPUs by sysfs oleg344-server: Force libcfs to create 2 CPU partitions oleg344-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg344-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg344-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg344-server: oleg344-server.virtnet: executing set_default_debug -1 all pdsh@oleg344-client: oleg344-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg344-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg344-server: oleg344-server.virtnet: executing set_default_debug -1 all pdsh@oleg344-client: oleg344-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg344-server: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg344-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg344-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg344-server stop mds service on oleg344-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg344-server PASS 51 (177s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:45:45 (1709786745) start mds service on oleg344-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg344-server: oleg344-server.virtnet: executing set_default_debug -1 all pdsh@oleg344-client: oleg344-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg344-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg344-server: oleg344-server.virtnet: executing set_default_debug -1 all pdsh@oleg344-client: oleg344-server: ssh exited with exit code 1 Started lustre-OST0000 oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg344-client.virtnet: -o user_xattr,flock oleg344-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 321 (time 1709786775.61 total 10.57 last 30.36) - open/close 567 (time 1709786785.86 total 20.83 last 23.99) - open/close 835 (time 1709786796.63 total 31.59 last 24.90) - open/close 1173 (time 1709786807.36 total 42.32 last 31.50) - open/close 1425 (time 1709786817.55 total 52.52 last 24.73) - open/close 1693 (time 1709786837.33 total 72.30 last 13.54) - open/close 1865 (time 1709786847.57 total 82.54 last 16.80) - open/close 2090 (time 1709786858.16 total 93.13 last 21.25) - open/close 2344 (time 1709786868.17 total 103.13 last 25.39) - open/close 2560 (time 1709786878.85 total 113.81 last 20.23) - open/close 2751 (time 1709786889.28 total 124.25 last 18.30) - open/close 2941 (time 1709786900.28 total 135.25 last 17.28) - open/close 3190 (time 1709786910.28 total 145.25 last 24.90) - open/close 3440 (time 1709786921.32 total 156.28 last 22.65) - open/close 3641 (time 1709786931.34 total 166.31 last 20.05) - open/close 3827 (time 1709786943.58 total 178.55 last 15.19) - open/close 3982 (time 1709786953.64 total 188.60 last 15.42) - open/close 4106 (time 1709786963.81 total 198.78 last 12.19) - open/close 4310 (time 1709786976.42 total 211.39 last 16.18) - open/close 4508 (time 1709786987.00 total 221.97 last 18.71) - open/close 4751 (time 1709786997.01 total 231.98 last 24.28) - open/close 4915 (time 1709787008.73 total 243.70 last 13.99) - open/close 5070 (time 1709787018.98 total 253.95 last 15.13) - open/close 5201 (time 1709787029.10 total 264.07 last 12.94) - open/close 5390 (time 1709787039.10 total 274.07 last 18.90) - open/close 5591 (time 1709787049.16 total 284.13 last 19.99) - open/close 5855 (time 1709787059.45 total 294.42 last 25.66) - open/close 6035 (time 1709787070.28 total 305.24 last 16.62) - open/close 6178 (time 1709787080.28 total 315.25 last 14.29) - open/close 6401 (time 1709787090.29 total 325.26 last 22.28) - open/close 6576 (time 1709787100.31 total 335.27 last 17.47) - open/close 6751 (time 1709787110.80 total 345.77 last 16.67) - open/close 7006 (time 1709787120.82 total 355.78 last 25.47) - open/close 7129 (time 1709787131.31 total 366.28 last 11.72) - open/close 7270 (time 1709787142.02 total 376.98 last 13.17) - open/close 7326 (time 1709787155.79 total 390.76 last 4.07) - open/close 7470 (time 1709787166.48 total 401.45 last 13.47) - open/close 7645 (time 1709787176.88 total 411.84 last 16.83) - open/close 7796 (time 1709787196.71 total 431.68 last 7.61) - open/close 7943 (time 1709787206.72 total 441.69 last 14.68) - open/close 8087 (time 1709787217.33 total 452.30 last 13.57) - open/close 8243 (time 1709787227.93 total 462.90 last 14.72) - open/close 8482 (time 1709787238.55 total 473.51 last 22.52) - open/close 8768 (time 1709787248.55 total 483.52 last 28.59) - open/close 9367 (time 1709787258.56 total 493.53 last 59.80) - open/close 9936 (time 1709787268.57 total 503.54 last 56.85) - open/close 10452 (time 1709787278.59 total 513.56 last 51.49) - open/close 11036 (time 1709787288.61 total 523.57 last 58.33) - open/close 11612 (time 1709787298.62 total 533.58 last 57.55) - open/close 12154 (time 1709787308.62 total 543.59 last 54.17) - open/close 12687 (time 1709787318.63 total 553.59 last 53.27) - open/close 12901 (time 1709787373.32 total 608.29 last 3.91) - open/close 13380 (time 1709787383.32 total 618.29 last 47.88) - open/close 13843 (time 1709787393.37 total 628.34 last 46.06) - open/close 14260 (time 1709787403.40 total 638.37 last 41.58) - open/close 14702 (time 1709787413.43 total 648.40 last 44.08) - open/close 15158 (time 1709787423.44 total 658.40 last 45.57) - open/close 15603 (time 1709787433.44 total 668.40 last 44.50) - open/close 16083 (time 1709787443.45 total 678.42 last 47.92) - open/close 16528 (time 1709787453.48 total 688.45 last 44.39) - open/close 16923 (time 1709787463.49 total 698.45 last 39.47) - open/close 17300 (time 1709787473.49 total 708.46 last 37.68) - open/close 17731 (time 1709787483.51 total 718.48 last 43.02) - open/close 18113 (time 1709787493.51 total 728.48 last 38.20) - open/close 18444 (time 1709787503.52 total 738.49 last 33.07) - open/close 18854 (time 1709787513.54 total 748.51 last 40.92) - open/close 19226 (time 1709787523.54 total 758.51 last 37.18) - open/close 19544 (time 1709787533.55 total 768.52 last 31.79) - open/close 19633 (time 1709787547.27 total 782.23 last 6.49) - open/close 19978 (time 1709787557.29 total 792.26 last 34.41) - open/close 20264 (time 1709787567.30 total 802.27 last 28.58) - open/close 20584 (time 1709787577.32 total 812.29 last 31.93) - open/close 20917 (time 1709787587.33 total 822.30 last 33.26) - open/close 21235 (time 1709787597.34 total 832.31 last 31.78) - open/close 21504 (time 1709787656.23 total 891.20 last 4.57) - open/close 21848 (time 1709787666.24 total 901.21 last 34.38) - open/close 22152 (time 1709787676.27 total 911.24 last 30.30) - open/close 22456 (time 1709787686.28 total 921.25 last 30.36) - open/close 22709 (time 1709787696.29 total 931.26 last 25.28) - open/close 22974 (time 1709787706.31 total 941.27 last 26.46) - open/close 23238 (time 1709787716.32 total 951.29 last 26.36) - open/close 23550 (time 1709787726.36 total 961.33 last 31.07) - open/close 23867 (time 1709787736.39 total 971.36 last 31.60) - open/close 24147 (time 1709787746.40 total 981.36 last 27.99) - open/close 24410 (time 1709787756.41 total 991.37 last 26.27) - open/close 24722 (time 1709787766.43 total 1001.40 last 31.12) - open/close 25004 (time 1709787776.44 total 1011.41 last 28.17) - open/close 25292 (time 1709787786.44 total 1021.41 last 28.80) - open/close 25597 (time 1709787796.46 total 1031.42 last 30.46) - open/close 25908 (time 1709787806.46 total 1041.43 last 31.09) - open/close 26229 (time 1709787816.47 total 1051.44 last 32.06) - open/close 26522 (time 1709787826.50 total 1061.47 last 29.21) - open/close 26865 (time 1709787836.51 total 1071.47 last 34.29) - open/close 27153 (time 1709787857.09 total 1092.06 last 13.99) - open/close 27468 (time 1709787867.13 total 1102.10 last 31.39) - open/close 27744 (time 1709787877.16 total 1112.13 last 27.52) - open/close 28036 (time 1709787887.16 total 1122.13 last 29.19) - open/close 28330 (time 1709787897.21 total 1132.17 last 29.27) - open/close 28644 (time 1709787907.21 total 1142.18 last 31.39) - open/close 28937 (time 1709787917.21 total 1152.18 last 29.30) - open/close 29224 (time 1709787927.23 total 1162.20 last 28.65) - open/close 29504 (time 1709787937.36 total 1172.32 last 27.64) - open/close 29787 (time 1709787947.40 total 1182.37 last 28.19) - open/close 30000 (time 1709787954.77 total 1189.73 last 28.91) - open/close 30303 (time 1709787964.78 total 1199.75 last 30.25) - open/close 30587 (time 1709787974.84 total 1209.80 last 28.25) - open/close 30873 (time 1709787984.88 total 1219.85 last 28.47) - open/close 31178 (time 1709787994.93 total 1229.89 last 30.38) - open/close 31485 (time 1709788004.95 total 1239.92 last 30.63) - open/close 31778 (time 1709788014.96 total 1249.93 last 29.25) - open/close 32055 (time 1709788024.98 total 1259.95 last 27.65) - open/close 32316 (time 1709788035.00 total 1269.97 last 26.06) - open/close 32589 (time 1709788045.01 total 1279.98 last 27.26) - open/close 32858 (time 1709788055.02 total 1289.99 last 26.88) - open/close 33125 (time 1709788065.03 total 1300.00 last 26.67) - open/close 33418 (time 1709788075.05 total 1310.02 last 29.25) - open/close 33759 (time 1709788085.07 total 1320.04 last 34.03) - open/close 34064 (time 1709788095.07 total 1330.04 last 30.50) - open/close 34365 (time 1709788105.07 total 1340.04 last 30.10) - open/close 34688 (time 1709788115.09 total 1350.06 last 32.24) - open/close 35017 (time 1709788125.10 total 1360.07 last 32.87) - open/close 35336 (time 1709788135.11 total 1370.08 last 31.88) - open/close 35666 (time 1709788145.14 total 1380.11 last 32.89) - open/close 35963 (time 1709788155.16 total 1390.13 last 29.64) - open/close 36262 (time 1709788165.19 total 1400.15 last 29.84) - open/close 36562 (time 1709788175.20 total 1410.17 last 29.96) - open/close 36866 (time 1709788185.22 total 1420.19 last 30.33) - open/close 37166 (time 1709788195.25 total 1430.22 last 29.91) - open/close 37465 (time 1709788205.26 total 1440.23 last 29.88) - open/close 37764 (time 1709788215.26 total 1450.23 last 29.89) - open/close 38068 (time 1709788225.28 total 1460.25 last 30.34) - open/close 38356 (time 1709788235.29 total 1470.26 last 28.76) - open/close 38648 (time 1709788245.32 total 1480.28 last 29.13) - open/close 38922 (time 1709788255.35 total 1490.32 last 27.30) - open/close 39197 (time 1709788265.37 total 1500.34 last 27.46) - open/close 39476 (time 1709788275.38 total 1510.35 last 27.87) - open/close 39735 (time 1709788285.41 total 1520.38 last 25.82) - open/close 39990 (time 1709788295.43 total 1530.39 last 25.45) - open/close 40251 (time 1709788305.46 total 1540.43 last 26.02) - open/close 40525 (time 1709788315.48 total 1550.45 last 27.33) - open/close 40789 (time 1709788325.51 total 1560.48 last 26.33) - open/close 41067 (time 1709788335.51 total 1570.48 last 27.80) - open/close 41342 (time 1709788345.54 total 1580.50 last 27.43) - open/close 41613 (time 1709788355.56 total 1590.53 last 27.03) - open/close 41889 (time 1709788365.56 total 1600.53 last 27.60) - open/close 42146 (time 1709788375.56 total 1610.53 last 25.70) - open/close 42420 (time 1709788385.58 total 1620.55 last 27.35) - open/close 42682 (time 1709788395.59 total 1630.56 last 26.17) - open/close 42947 (time 1709788405.62 total 1640.59 last 26.42) - open/close 43200 (time 1709788415.64 total 1650.61 last 25.24) - open/close 43471 (time 1709788425.66 total 1660.63 last 27.04) - open/close 43737 (time 1709788435.68 total 1670.65 last 26.55) - open/close 44013 (time 1709788445.71 total 1680.68 last 27.52) - open/close 44259 (time 1709788455.72 total 1690.69 last 24.58) - open/close 44517 (time 1709788465.74 total 1700.71 last 25.75) - open/close 44786 (time 1709788475.77 total 1710.73 last 26.83) - open/close 45046 (time 1709788485.80 total 1720.77 last 25.91) - open/close 45272 (time 1709788495.81 total 1730.78 last 22.58) - open/close 45516 (time 1709788505.83 total 1740.80 last 24.35) - open/close 45764 (time 1709788515.84 total 1750.81 last 24.76) - open/close 46020 (time 1709788525.87 total 1760.83 last 25.55) - open/close 46255 (time 1709788535.88 total 1770.84 last 23.47) - open/close 46508 (time 1709788545.88 total 1780.84 last 25.30) - open/close 46761 (time 1709788555.90 total 1790.87 last 25.24) - open/close 47012 (time 1709788565.92 total 1800.89 last 25.04) - open/close 47271 (time 1709788575.93 total 1810.90 last 25.89) - open/close 47526 (time 1709788585.95 total 1820.92 last 25.44) - open/close 47774 (time 1709788595.96 total 1830.93 last 24.79) - open/close 48030 (time 1709788605.97 total 1840.94 last 25.57) - open/close 48283 (time 1709788616.00 total 1850.97 last 25.23) - open/close 48532 (time 1709788626.03 total 1861.00 last 24.82) - open/close 48771 (time 1709788636.06 total 1871.03 last 23.82) - open/close 48975 (time 1709788646.09 total 1881.05 last 20.36) - open/close 49154 (time 1709788656.12 total 1891.09 last 17.84) - open/close 49335 (time 1709788666.16 total 1901.13 last 18.02) - open/close 49555 (time 1709788676.18 total 1911.14 last 21.97) - open/close 49775 (time 1709788686.20 total 1921.17 last 21.94) total: 50000 open/close in 1930.89 seconds: 25.89 ops/second - unlinked 0 (time 1709788697 ; total 0 ; last 0) - unlinked 10000 (time 1709789172 ; total 475 ; last 475) - unlinked 20000 (time 1709789562 ; total 865 ; last 390) - unlinked 30000 (time 1709789856 ; total 1159 ; last 294) - unlinked 40000 (time 1709789999 ; total 1302 ; last 143) total: 50000 unlinks in 1355 seconds: 36.900368 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg344-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg344-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg344-server stop mds service on oleg344-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg344-server unloading modules on: 'oleg344-server' oleg344-server: oleg344-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg344-client: oleg344-client: ssh exited with exit code 2 pdsh@oleg344-client: oleg344-server: ssh exited with exit code 2 pdsh@oleg344-client: oleg344-client: ssh exited with exit code 2 pdsh@oleg344-client: oleg344-server: ssh exited with exit code 2 PASS 101a (3321s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg344-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg344-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg344-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg344-client: oleg344-server: ssh exited with exit code 2 oleg344-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg344-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == conf-sanity test 101b: Race events DISCONNECT and ACTIVE in osp ========================================================== 00:41:08 (1709790068) start mds service on oleg344-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg344-server' oleg344-server: oleg344-server.virtnet: executing load_modules_local oleg344-server: Loading modules from /home/green/git/lustre-release/lustre oleg344-server: detected 4 online CPUs by sysfs oleg344-server: Force libcfs to create 2 CPU partitions oleg344-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg344-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg344-server: oleg344-server.virtnet: executing set_default_debug -1 all pdsh@oleg344-client: oleg344-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg344-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg344-server: oleg344-server.virtnet: executing set_default_debug -1 all pdsh@oleg344-client: oleg344-server: ssh exited with exit code 1 Started lustre-OST0000 oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg344-client.virtnet: -o user_xattr,flock oleg344-server@tcp:/lustre /mnt/lustre fail_loc=0x80002107 fail_val=20 stop ost1 service on oleg344-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg344-server start ost1 service on oleg344-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg344-server: oleg344-server.virtnet: executing set_default_debug -1 all pdsh@oleg344-client: oleg344-server: ssh exited with exit code 1 Started lustre-OST0000 oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 5 sec oleg344-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8800aa20c000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800aa20c000.ost_server_uuid in FULL state after 0 sec umount lustre on /mnt/lustre..... Stopping client oleg344-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg344-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg344-server stop mds service on oleg344-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg344-server unloading modules on: 'oleg344-server' oleg344-server: oleg344-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg344-client: oleg344-client: ssh exited with exit code 2 pdsh@oleg344-client: oleg344-server: ssh exited with exit code 2 pdsh@oleg344-client: oleg344-client: ssh exited with exit code 2 pdsh@oleg344-client: oleg344-server: ssh exited with exit code 2 PASS 101b (88s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg344-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg344-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory Stopping clients: oleg344-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg344-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg344-client: oleg344-server: ssh exited with exit code 2 oleg344-server: oleg344-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg344-server' oleg344-server: oleg344-server.virtnet: executing load_modules_local oleg344-server: Loading modules from /home/green/git/lustre-release/lustre oleg344-server: detected 4 online CPUs by sysfs oleg344-server: Force libcfs to create 2 CPU partitions oleg344-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg344-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 3833 sec ========== 00:43:01 (1709790181) conf-sanity: FAIL: test_38 mount_client /mnt/lustre failed === conf-sanity: start cleanup 00:43:02 (1709790182) === === conf-sanity: finish cleanup 00:43:02 (1709790182) ===