-----============= acceptance-small: conf-sanity ============----- Wed Mar 6 23:45:36 EST 2024 excepting tests: 32newtarball 110 skipping tests SLOW=no: 45 69 106 111 114 Stopping clients: oleg419-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg419-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg419-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg419-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg419-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg419-server oleg419-server: oleg419-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory loading modules on: 'oleg419-server' oleg419-server: oleg419-server.virtnet: executing load_modules_local oleg419-server: Loading modules from /home/green/git/lustre-release/lustre oleg419-server: detected 4 online CPUs by sysfs oleg419-server: Force libcfs to create 2 CPU partitions oleg419-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg419-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg419-server: oleg419-server.virtnet: executing set_default_debug -1 all pdsh@oleg419-client: oleg419-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg419-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg419-server: oleg419-server.virtnet: executing set_default_debug -1 all pdsh@oleg419-client: oleg419-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg419-server: oleg419-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg419-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg419-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg419-server stop mds service on oleg419-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg419-server debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 35b: Continue reconnection retries, if the active server is busy ========================================================== 23:46:43 (1709786803) start mds service on oleg419-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg419-server: oleg419-server.virtnet: executing set_default_debug -1 all pdsh@oleg419-client: oleg419-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg419-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg419-server: oleg419-server.virtnet: executing set_default_debug -1 all pdsh@oleg419-client: oleg419-server: ssh exited with exit code 1 Started lustre-OST0000 oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg419-client.virtnet: -o user_xattr,flock oleg419-server@tcp:/lustre /mnt/lustre debug=ha conf-sanity.sh test_35b 2024-03-0623h46m58s Set up a fake failnode for the MDS at_max=0 at_max=0 Injecting EBUSY on MDS fail_loc=0x80000136 mdc.lustre-MDT0000-mdc-ffff8800b1536800.stats=clear Creating a test file and stat it File: '/mnt/lustre/d35b.conf-sanity/f35b.conf-sanity' Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115238826934274 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-03-06 23:47:45.000000000 -0500 Modify: 2024-03-06 23:47:45.000000000 -0500 Change: 2024-03-06 23:47:45.000000000 -0500 Birth: - Stop injecting EBUSY on MDS fail_loc=0 done at_max=600 at_max=600 Debug log: 76 lines, 76 kept, 0 dropped, 0 bad. umount lustre on /mnt/lustre..... Stopping client oleg419-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg419-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg419-server stop mds service on oleg419-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg419-server unloading modules on: 'oleg419-server' oleg419-server: oleg419-server.virtnet: executing unload_modules_local modules unloaded. oleg419-server: oleg419-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre oleg419-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg419-client: oleg419-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg419-server: oleg419-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre oleg419-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg419-client: oleg419-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg419-server: oleg419-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre oleg419-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg419-client: oleg419-server: ssh exited with exit code 19 checking for existing Lustre data: not found tunefs failed, reformatting instead Stopping clients: oleg419-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg419-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg419-client: oleg419-server: ssh exited with exit code 2 oleg419-server: oleg419-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg419-server' oleg419-server: oleg419-server.virtnet: executing load_modules_local oleg419-server: Loading modules from /home/green/git/lustre-release/lustre oleg419-server: detected 4 online CPUs by sysfs oleg419-server: Force libcfs to create 2 CPU partitions oleg419-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg419-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg419-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg419-server: oleg419-server.virtnet: executing set_default_debug -1 all pdsh@oleg419-client: oleg419-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg419-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg419-server: oleg419-server.virtnet: executing set_default_debug -1 all pdsh@oleg419-client: oleg419-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg419-server: oleg419-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg419-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg419-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg419-server stop mds service on oleg419-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg419-server PASS 35b (157s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 23:49:23 (1709786963) start mds service on oleg419-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg419-server: oleg419-server.virtnet: executing set_default_debug -1 all pdsh@oleg419-client: oleg419-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg419-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg419-server: oleg419-server.virtnet: executing set_default_debug -1 all pdsh@oleg419-client: oleg419-server: ssh exited with exit code 1 Started lustre-OST0000 oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg419-client.virtnet: -o user_xattr,flock oleg419-server@tcp:/lustre /mnt/lustre copying 10 files to /mnt/lustre/d38.conf-sanity tar: Removing leading `/' from member names umount lustre on /mnt/lustre..... Stopping client oleg419-client.virtnet /mnt/lustre (opts:) osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=44 stop mds service on oleg419-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg419-server delete lov_objid file on MDS 000000 43 000008 start mds service on oleg419-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg419-server: oleg419-server.virtnet: executing set_default_debug -1 all pdsh@oleg419-client: oleg419-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg419-client.virtnet: -o user_xattr,flock oleg419-server@tcp:/lustre /mnt/lustre osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=2 umount lustre on /mnt/lustre..... Stopping client oleg419-client.virtnet /mnt/lustre (opts:) stop mds service on oleg419-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg419-server 000000 2 000008 oleg419-server: 8+0 records in oleg419-server: 8+0 records out oleg419-server: 4096 bytes (4.1 kB) copied, 0.00220582 s, 1.9 MB/s start mds service on oleg419-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg419-server: oleg419-server.virtnet: executing set_default_debug -1 all pdsh@oleg419-client: oleg419-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg419-client.virtnet: -o user_xattr,flock oleg419-server@tcp:/lustre /mnt/lustre mount.lustre: mount oleg419-server@tcp:/lustre at /mnt/lustre failed: Input/output error Is the MGS running? conf-sanity test_38: @@@@@@ FAIL: mount_client /mnt/lustre failed Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:6975:error() = /home/green/git/lustre-release/lustre/tests/conf-sanity.sh:3550:test_38() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7315:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7375:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7186:run_test() = /home/green/git/lustre-release/lustre/tests/conf-sanity.sh:3569:main() Dumping lctl log to /tmp/testlogs//conf-sanity.test_38.*.1709787024.log rsync: chown "/tmp/testlogs/.conf-sanity.test_38.debug_log.oleg419-server.1709787024.log.UpmgwD" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.conf-sanity.test_38.dmesg.oleg419-server.1709787024.log.j2DEUI" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] FAIL 38 (65s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 23:50:35 (1709787035) Stopping clients: oleg419-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg419-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg419-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg419-server oleg419-server: oleg419-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg419-server' oleg419-server: oleg419-server.virtnet: executing load_modules_local oleg419-server: Loading modules from /home/green/git/lustre-release/lustre oleg419-server: detected 4 online CPUs by sysfs oleg419-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg419-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg419-server: oleg419-server.virtnet: executing set_default_debug -1 all pdsh@oleg419-client: oleg419-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg419-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg419-server: oleg419-server.virtnet: executing set_default_debug -1 all pdsh@oleg419-client: oleg419-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg419-client.virtnet: -o user_xattr,flock oleg419-server@tcp:/lustre /mnt/lustre setup single mount lustre success fail_loc=0x142 start ost2 service on oleg419-server Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg419-server: oleg419-server.virtnet: executing set_default_debug -1 all pdsh@oleg419-client: oleg419-server: ssh exited with exit code 1 Commit the device label on lustre-ost2/ost2 Started lustre-OST0001 oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec stop ost2 service on oleg419-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg419-server umount lustre on /mnt/lustre..... Stopping client oleg419-client.virtnet /mnt/lustre (opts:-f) stop ost1 service on oleg419-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg419-server stop mds service on oleg419-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg419-server unloading modules on: 'oleg419-server' oleg419-server: oleg419-server.virtnet: executing unload_modules_local modules unloaded. oleg419-server: oleg419-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre oleg419-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg419-client: oleg419-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg419-server: oleg419-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre oleg419-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg419-client: oleg419-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg419-server: oleg419-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre oleg419-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg419-client: oleg419-server: ssh exited with exit code 19 checking for existing Lustre data: not found tunefs failed, reformatting instead Stopping clients: oleg419-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg419-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg419-client: oleg419-server: ssh exited with exit code 2 oleg419-server: oleg419-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg419-server' oleg419-server: oleg419-server.virtnet: executing load_modules_local oleg419-server: Loading modules from /home/green/git/lustre-release/lustre oleg419-server: detected 4 online CPUs by sysfs oleg419-server: Force libcfs to create 2 CPU partitions oleg419-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg419-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg419-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg419-server: oleg419-server.virtnet: executing set_default_debug -1 all pdsh@oleg419-client: oleg419-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg419-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg419-server: oleg419-server.virtnet: executing set_default_debug -1 all pdsh@oleg419-client: oleg419-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg419-server: oleg419-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg419-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg419-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg419-server stop mds service on oleg419-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg419-server PASS 51 (191s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:53:49 (1709787229) start mds service on oleg419-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg419-server: oleg419-server.virtnet: executing set_default_debug -1 all pdsh@oleg419-client: oleg419-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg419-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg419-server: oleg419-server.virtnet: executing set_default_debug -1 all pdsh@oleg419-client: oleg419-server: ssh exited with exit code 1 Started lustre-OST0000 oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg419-client.virtnet: -o user_xattr,flock oleg419-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 257 (time 1709787262.33 total 10.04 last 25.59) - open/close 497 (time 1709787272.36 total 20.07 last 23.93) - open/close 714 (time 1709787282.52 total 30.23 last 21.36) - open/close 888 (time 1709787297.12 total 44.83 last 11.92) - open/close 1069 (time 1709787307.14 total 54.85 last 18.06) - open/close 1288 (time 1709787317.50 total 65.21 last 21.14) - open/close 1474 (time 1709787328.09 total 75.80 last 17.56) - open/close 1619 (time 1709787338.31 total 86.03 last 14.18) - open/close 1801 (time 1709787350.90 total 98.61 last 14.46) - open/close 2036 (time 1709787361.35 total 109.07 last 22.47) - open/close 2216 (time 1709787372.36 total 120.07 last 16.36) - open/close 2363 (time 1709787382.68 total 130.40 last 14.24) - open/close 2575 (time 1709787393.11 total 140.82 last 20.33) - open/close 2780 (time 1709787403.12 total 150.83 last 20.48) - open/close 2956 (time 1709787413.86 total 161.57 last 16.39) - open/close 3158 (time 1709787424.26 total 171.98 last 19.42) - open/close 3386 (time 1709787434.43 total 182.14 last 22.43) - open/close 3524 (time 1709787444.92 total 192.63 last 13.16) - open/close 3634 (time 1709787455.89 total 203.60 last 10.02) - open/close 3853 (time 1709787466.91 total 214.62 last 19.88) - open/close 4083 (time 1709787476.92 total 224.63 last 22.98) - open/close 4172 (time 1709787488.42 total 236.14 last 7.73) - open/close 4337 (time 1709787498.53 total 246.25 last 16.32) - open/close 4531 (time 1709787508.55 total 256.26 last 19.37) - open/close 4723 (time 1709787519.15 total 266.86 last 18.11) - open/close 4892 (time 1709787529.30 total 277.02 last 16.64) - open/close 5040 (time 1709787539.43 total 287.15 last 14.61) - open/close 5247 (time 1709787550.93 total 298.64 last 18.01) - open/close 5501 (time 1709787560.93 total 308.65 last 25.39) - open/close 5812 (time 1709787571.85 total 319.56 last 28.50) - open/close 6124 (time 1709787581.85 total 329.57 last 31.17) - open/close 6319 (time 1709787592.81 total 340.52 last 17.80) - open/close 6539 (time 1709787602.81 total 350.53 last 21.99) - open/close 6769 (time 1709787613.38 total 361.09 last 21.77) - open/close 6992 (time 1709787625.74 total 373.45 last 18.05) - open/close 7270 (time 1709787636.12 total 383.83 last 26.77) - open/close 7468 (time 1709787648.15 total 395.86 last 16.46) - open/close 7717 (time 1709787667.91 total 415.62 last 12.60) - open/close 7877 (time 1709787677.91 total 425.63 last 15.99) - open/close 8001 (time 1709787688.26 total 435.97 last 11.99) - open/close 8262 (time 1709787698.28 total 446.00 last 26.03) - open/close 8367 (time 1709787708.29 total 456.00 last 10.50) - open/close 8623 (time 1709787718.31 total 466.03 last 25.53) - open/close 8809 (time 1709787728.34 total 476.06 last 18.54) - open/close 8981 (time 1709787738.88 total 486.60 last 16.32) - open/close 9177 (time 1709787748.91 total 496.62 last 19.56) - open/close 9403 (time 1709787758.91 total 506.62 last 22.60) - open/close 9609 (time 1709787768.92 total 516.64 last 20.57) - open/close 9961 (time 1709787778.94 total 526.65 last 35.15) - open/close 10503 (time 1709787788.94 total 536.66 last 54.16) - open/close 11023 (time 1709787798.95 total 546.66 last 51.98) - open/close 11645 (time 1709787809.05 total 556.77 last 61.56) - open/close 12195 (time 1709787819.06 total 566.78 last 54.95) - open/close 12665 (time 1709787829.06 total 576.78 last 47.00) - open/close 13215 (time 1709787839.07 total 586.78 last 54.96) - open/close 13773 (time 1709787849.09 total 596.80 last 55.68) - open/close 13981 (time 1709787863.97 total 611.68 last 13.98) - open/close 14528 (time 1709787873.97 total 621.69 last 54.67) - open/close 15098 (time 1709787883.99 total 631.71 last 56.88) - open/close 15631 (time 1709787894.00 total 641.72 last 53.25) - open/close 16149 (time 1709787904.01 total 651.73 last 51.75) - open/close 16633 (time 1709787914.04 total 661.75 last 48.27) - open/close 17100 (time 1709787924.06 total 671.77 last 46.60) - open/close 17569 (time 1709787934.07 total 681.79 last 46.84) - open/close 18019 (time 1709787944.09 total 691.81 last 44.91) - open/close 18484 (time 1709787954.10 total 701.82 last 46.45) - open/close 18935 (time 1709787964.12 total 711.84 last 45.01) - open/close 19372 (time 1709787974.15 total 721.86 last 43.60) - open/close 19801 (time 1709787984.15 total 731.87 last 42.87) - open/close 20000 (time 1709787988.65 total 736.37 last 44.22) - open/close 20453 (time 1709787998.67 total 746.39 last 45.22) - open/close 20900 (time 1709788008.68 total 756.39 last 44.67) - open/close 21317 (time 1709788018.70 total 766.41 last 41.61) - open/close 21748 (time 1709788028.70 total 776.42 last 43.09) - open/close 22165 (time 1709788038.72 total 786.44 last 41.62) - open/close 22542 (time 1709788048.74 total 796.46 last 37.63) - open/close 22928 (time 1709788058.75 total 806.46 last 38.58) - open/close 23301 (time 1709788068.77 total 816.48 last 37.22) - open/close 23710 (time 1709788078.79 total 826.50 last 40.82) - open/close 24103 (time 1709788088.79 total 836.51 last 39.29) - open/close 24484 (time 1709788098.81 total 846.52 last 38.03) - open/close 24851 (time 1709788108.82 total 856.53 last 36.67) - open/close 25231 (time 1709788118.83 total 866.55 last 37.94) - open/close 25607 (time 1709788128.85 total 876.57 last 37.54) - open/close 25953 (time 1709788138.89 total 886.60 last 34.48) - open/close 26301 (time 1709788148.92 total 896.63 last 34.70) - open/close 26636 (time 1709788158.92 total 906.63 last 33.49) - open/close 26957 (time 1709788168.94 total 916.66 last 32.03) - open/close 27282 (time 1709788178.94 total 926.66 last 32.49) - open/close 27642 (time 1709788188.95 total 936.66 last 35.99) - open/close 27967 (time 1709788198.97 total 946.69 last 32.41) - open/close 28288 (time 1709788208.98 total 956.69 last 32.08) - open/close 28610 (time 1709788219.00 total 966.71 last 32.14) - open/close 28929 (time 1709788229.02 total 976.73 last 31.84) - open/close 29236 (time 1709788239.03 total 986.75 last 30.64) - open/close 29548 (time 1709788249.06 total 996.77 last 31.13) - open/close 29895 (time 1709788259.07 total 1006.79 last 34.65) - open/close 30000 (time 1709788262.12 total 1009.83 last 34.50) - open/close 30313 (time 1709788272.13 total 1019.85 last 31.24) - open/close 30619 (time 1709788282.15 total 1029.86 last 30.56) - open/close 30943 (time 1709788292.16 total 1039.88 last 32.34) - open/close 31262 (time 1709788302.19 total 1049.91 last 31.81) - open/close 31572 (time 1709788312.19 total 1059.91 last 31.00) - open/close 31879 (time 1709788322.21 total 1069.92 last 30.64) - open/close 32182 (time 1709788332.22 total 1079.93 last 30.27) - open/close 32478 (time 1709788342.22 total 1089.94 last 29.58) - open/close 32763 (time 1709788352.23 total 1099.94 last 28.50) - open/close 33041 (time 1709788362.24 total 1109.95 last 27.77) - open/close 33324 (time 1709788372.26 total 1119.97 last 28.24) - open/close 33608 (time 1709788382.27 total 1129.98 last 28.37) - open/close 33891 (time 1709788392.28 total 1140.00 last 28.26) - open/close 34181 (time 1709788402.32 total 1150.03 last 28.90) - open/close 34462 (time 1709788412.35 total 1160.07 last 28.00) - open/close 34743 (time 1709788422.38 total 1170.10 last 28.01) - open/close 35023 (time 1709788432.41 total 1180.12 last 27.93) - open/close 35307 (time 1709788442.43 total 1190.15 last 28.33) - open/close 35590 (time 1709788452.44 total 1200.15 last 28.29) - open/close 35854 (time 1709788462.46 total 1210.18 last 26.33) - open/close 36124 (time 1709788472.48 total 1220.20 last 26.95) - open/close 36398 (time 1709788482.49 total 1230.21 last 27.37) - open/close 36685 (time 1709788492.49 total 1240.21 last 28.70) - open/close 36974 (time 1709788502.49 total 1250.21 last 28.89) - open/close 37274 (time 1709788512.52 total 1260.23 last 29.93) - open/close 37538 (time 1709788522.55 total 1270.26 last 26.32) - open/close 37809 (time 1709788532.56 total 1280.27 last 27.07) - open/close 38075 (time 1709788542.58 total 1290.30 last 26.54) - open/close 38345 (time 1709788552.61 total 1300.32 last 26.94) - open/close 38613 (time 1709788562.61 total 1310.33 last 26.79) - open/close 38882 (time 1709788572.63 total 1320.35 last 26.84) - open/close 39154 (time 1709788582.64 total 1330.35 last 27.19) - open/close 39434 (time 1709788592.65 total 1340.36 last 27.96) - open/close 39716 (time 1709788602.66 total 1350.38 last 28.16) - open/close 39980 (time 1709788612.68 total 1360.40 last 26.35) - open/close 40251 (time 1709788622.71 total 1370.43 last 27.02) - open/close 40536 (time 1709788632.73 total 1380.44 last 28.45) - open/close 40820 (time 1709788642.73 total 1390.44 last 28.39) - open/close 41099 (time 1709788652.75 total 1400.46 last 27.85) - open/close 41366 (time 1709788662.75 total 1410.47 last 26.69) - open/close 41630 (time 1709788672.76 total 1420.47 last 26.39) - open/close 41891 (time 1709788682.78 total 1430.50 last 26.03) - open/close 42162 (time 1709788692.81 total 1440.52 last 27.02) - open/close 42421 (time 1709788702.81 total 1450.53 last 25.89) - open/close 42677 (time 1709788712.84 total 1460.56 last 25.53) - open/close 42919 (time 1709788722.87 total 1470.58 last 24.13) - open/close 43170 (time 1709788732.90 total 1480.61 last 25.04) - open/close 43414 (time 1709788742.90 total 1490.62 last 24.38) - open/close 43656 (time 1709788752.92 total 1500.64 last 24.15) - open/close 43906 (time 1709788762.92 total 1510.64 last 25.00) - open/close 44163 (time 1709788772.94 total 1520.66 last 25.65) - open/close 44399 (time 1709788782.96 total 1530.67 last 23.56) - open/close 44640 (time 1709788792.96 total 1540.68 last 24.09) - open/close 44877 (time 1709788802.98 total 1550.70 last 23.65) - open/close 45114 (time 1709788812.99 total 1560.71 last 23.68) - open/close 45349 (time 1709788823.03 total 1570.75 last 23.41) - open/close 45587 (time 1709788833.04 total 1580.76 last 23.77) - open/close 45824 (time 1709788843.07 total 1590.78 last 23.64) - open/close 46046 (time 1709788853.11 total 1600.82 last 22.12) - open/close 46281 (time 1709788863.11 total 1610.83 last 23.49) - open/close 46521 (time 1709788873.12 total 1620.83 last 23.99) - open/close 46758 (time 1709788883.15 total 1630.86 last 23.63) - open/close 46981 (time 1709788893.19 total 1640.91 last 22.20) - open/close 47209 (time 1709788903.21 total 1650.92 last 22.76) - open/close 47442 (time 1709788913.23 total 1660.95 last 23.24) - open/close 47666 (time 1709788923.26 total 1670.97 last 22.35) - open/close 47890 (time 1709788933.27 total 1680.99 last 22.36) - open/close 48118 (time 1709788943.31 total 1691.02 last 22.72) - open/close 48342 (time 1709788953.34 total 1701.06 last 22.32) - open/close 48569 (time 1709788963.39 total 1711.10 last 22.60) - open/close 48789 (time 1709788973.43 total 1721.14 last 21.92) - open/close 49005 (time 1709788983.44 total 1731.16 last 21.56) - open/close 49222 (time 1709788993.49 total 1741.20 last 21.61) - open/close 49446 (time 1709789003.51 total 1751.22 last 22.36) - open/close 49669 (time 1709789013.55 total 1761.26 last 22.22) - open/close 49885 (time 1709789023.55 total 1771.26 last 21.60) total: 50000 open/close in 1776.31 seconds: 28.15 ops/second - unlinked 0 (time 1709789029 ; total 0 ; last 0) - unlinked 10000 (time 1709789554 ; total 525 ; last 525) - unlinked 20000 (time 1709789964 ; total 935 ; last 410) - unlinked 30000 (time 1709790212 ; total 1183 ; last 248) - unlinked 40000 (time 1709790297 ; total 1268 ; last 85) total: 50000 unlinks in 1295 seconds: 38.610039 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg419-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg419-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg419-server stop mds service on oleg419-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg419-server unloading modules on: 'oleg419-server' oleg419-server: oleg419-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg419-client: oleg419-client: ssh exited with exit code 2 pdsh@oleg419-client: oleg419-server: ssh exited with exit code 2 pdsh@oleg419-client: oleg419-client: ssh exited with exit code 2 pdsh@oleg419-client: oleg419-server: ssh exited with exit code 2 PASS 101a (3108s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg419-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg419-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg419-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg419-client: oleg419-server: ssh exited with exit code 2 oleg419-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg419-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == conf-sanity test 101b: Race events DISCONNECT and ACTIVE in osp ========================================================== 00:45:38 (1709790338) start mds service on oleg419-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg419-server' oleg419-server: oleg419-server.virtnet: executing load_modules_local oleg419-server: Loading modules from /home/green/git/lustre-release/lustre oleg419-server: detected 4 online CPUs by sysfs oleg419-server: Force libcfs to create 2 CPU partitions oleg419-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg419-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg419-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg419-server: oleg419-server.virtnet: executing set_default_debug -1 all pdsh@oleg419-client: oleg419-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg419-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg419-server: oleg419-server.virtnet: executing set_default_debug -1 all pdsh@oleg419-client: oleg419-server: ssh exited with exit code 1 Started lustre-OST0000 oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg419-client.virtnet: -o user_xattr,flock oleg419-server@tcp:/lustre /mnt/lustre fail_loc=0x80002107 fail_val=20 stop ost1 service on oleg419-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg419-server start ost1 service on oleg419-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg419-server: oleg419-server.virtnet: executing set_default_debug -1 all pdsh@oleg419-client: oleg419-server: ssh exited with exit code 1 Started lustre-OST0000 oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 8 sec oleg419-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff88012aaa6000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff88012aaa6000.ost_server_uuid in FULL state after 0 sec umount lustre on /mnt/lustre..... Stopping client oleg419-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg419-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg419-server stop mds service on oleg419-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg419-server unloading modules on: 'oleg419-server' oleg419-server: oleg419-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg419-client: oleg419-client: ssh exited with exit code 2 pdsh@oleg419-client: oleg419-server: ssh exited with exit code 2 pdsh@oleg419-client: oleg419-client: ssh exited with exit code 2 pdsh@oleg419-client: oleg419-server: ssh exited with exit code 2 PASS 101b (80s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg419-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg419-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory Stopping clients: oleg419-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg419-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg419-client: oleg419-server: ssh exited with exit code 2 oleg419-server: oleg419-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg419-server' oleg419-server: oleg419-server.virtnet: executing load_modules_local oleg419-server: Loading modules from /home/green/git/lustre-release/lustre oleg419-server: detected 4 online CPUs by sysfs oleg419-server: Force libcfs to create 2 CPU partitions oleg419-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg419-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 3699 sec ========== 00:47:15 (1709790435) conf-sanity: FAIL: test_38 mount_client /mnt/lustre failed === conf-sanity: start cleanup 00:47:15 (1709790435) === === conf-sanity: finish cleanup 00:47:15 (1709790435) ===