-----============= acceptance-small: conf-sanity ============----- Wed Mar 6 23:40:41 EST 2024 excepting tests: 32newtarball 110 skipping tests SLOW=no: 45 69 106 111 114 Stopping clients: oleg435-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg435-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg435-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg435-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg435-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg435-server oleg435-server: oleg435-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg435-server' oleg435-server: oleg435-server.virtnet: executing load_modules_local oleg435-server: Loading modules from /home/green/git/lustre-release/lustre oleg435-server: detected 4 online CPUs by sysfs oleg435-server: Force libcfs to create 2 CPU partitions oleg435-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg435-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg435-server: oleg435-server.virtnet: executing set_default_debug -1 all pdsh@oleg435-client: oleg435-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg435-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg435-server: oleg435-server.virtnet: executing set_default_debug -1 all pdsh@oleg435-client: oleg435-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg435-server: oleg435-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg435-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg435-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg435-server stop mds service on oleg435-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg435-server debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 35b: Continue reconnection retries, if the active server is busy ========================================================== 23:41:41 (1709786501) start mds service on oleg435-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg435-server: oleg435-server.virtnet: executing set_default_debug -1 all pdsh@oleg435-client: oleg435-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg435-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg435-server: oleg435-server.virtnet: executing set_default_debug -1 all pdsh@oleg435-client: oleg435-server: ssh exited with exit code 1 Started lustre-OST0000 oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg435-client.virtnet: -o user_xattr,flock oleg435-server@tcp:/lustre /mnt/lustre debug=ha conf-sanity.sh test_35b 2024-03-0623h41m57s Set up a fake failnode for the MDS at_max=0 at_max=0 Injecting EBUSY on MDS fail_loc=0x80000136 mdc.lustre-MDT0000-mdc-ffff8800aa267800.stats=clear Creating a test file and stat it File: '/mnt/lustre/d35b.conf-sanity/f35b.conf-sanity' Size: 0 Blocks: 0 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115238826934274 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-03-06 23:42:39.000000000 -0500 Modify: 2024-03-06 23:42:39.000000000 -0500 Change: 2024-03-06 23:42:39.000000000 -0500 Birth: - Stop injecting EBUSY on MDS fail_loc=0 done at_max=600 at_max=600 Debug log: 74 lines, 74 kept, 0 dropped, 0 bad. umount lustre on /mnt/lustre..... Stopping client oleg435-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg435-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg435-server stop mds service on oleg435-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg435-server unloading modules on: 'oleg435-server' oleg435-server: oleg435-server.virtnet: executing unload_modules_local modules unloaded. oleg435-server: tunefs.lustre: Unable to mount /dev/mapper/mds1_flakey: No such device oleg435-server: Is the ldiskfs module available? oleg435-server: oleg435-server: tunefs.lustre FATAL: failed to write local files oleg435-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg435-client: oleg435-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x5 (MDT MGS ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity Permanent disk data: Target: lustre=MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x105 (MDT MGS writeconf ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity oleg435-server: tunefs.lustre: Unable to mount /dev/mapper/ost1_flakey: No such device oleg435-server: Is the ldiskfs module available? oleg435-server: oleg435-server: tunefs.lustre FATAL: failed to write local files oleg435-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg435-client: oleg435-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x2 (OST ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.204.135@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x102 (OST writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.204.135@tcp sys.timeout=20 oleg435-server: tunefs.lustre: Unable to mount /dev/mapper/ost2_flakey: No such device oleg435-server: Is the ldiskfs module available? oleg435-server: oleg435-server: tunefs.lustre FATAL: failed to write local files oleg435-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg435-client: oleg435-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.204.135@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x162 (OST first_time update writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.204.135@tcp sys.timeout=20 tunefs failed, reformatting instead Stopping clients: oleg435-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg435-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg435-client: oleg435-server: ssh exited with exit code 2 oleg435-server: oleg435-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg435-server' oleg435-server: oleg435-server.virtnet: executing load_modules_local oleg435-server: Loading modules from /home/green/git/lustre-release/lustre oleg435-server: detected 4 online CPUs by sysfs oleg435-server: Force libcfs to create 2 CPU partitions oleg435-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg435-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg435-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg435-server: oleg435-server.virtnet: executing set_default_debug -1 all pdsh@oleg435-client: oleg435-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg435-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg435-server: oleg435-server.virtnet: executing set_default_debug -1 all pdsh@oleg435-client: oleg435-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg435-server: oleg435-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg435-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg435-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg435-server stop mds service on oleg435-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg435-server PASS 35b (121s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 23:43:45 (1709786625) start mds service on oleg435-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg435-server: oleg435-server.virtnet: executing set_default_debug -1 all pdsh@oleg435-client: oleg435-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg435-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg435-server: oleg435-server.virtnet: executing set_default_debug -1 all pdsh@oleg435-client: oleg435-server: ssh exited with exit code 1 Started lustre-OST0000 oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg435-client.virtnet: -o user_xattr,flock oleg435-server@tcp:/lustre /mnt/lustre copying 10 files to /mnt/lustre/d38.conf-sanity tar: Removing leading `/' from member names umount lustre on /mnt/lustre..... Stopping client oleg435-client.virtnet /mnt/lustre (opts:) osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=44 stop mds service on oleg435-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg435-server delete lov_objid file on MDS 000000 43 000008 start mds service on oleg435-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg435-server: oleg435-server.virtnet: executing set_default_debug -1 all pdsh@oleg435-client: oleg435-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg435-client.virtnet: -o user_xattr,flock oleg435-server@tcp:/lustre /mnt/lustre osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=2 umount lustre on /mnt/lustre..... Stopping client oleg435-client.virtnet /mnt/lustre (opts:) stop mds service on oleg435-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg435-server 000000 2 000008 oleg435-server: 8+0 records in oleg435-server: 8+0 records out oleg435-server: 4096 bytes (4.1 kB) copied, 0.000891844 s, 4.6 MB/s start mds service on oleg435-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg435-server: oleg435-server.virtnet: executing set_default_debug -1 all pdsh@oleg435-client: oleg435-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg435-client.virtnet: -o user_xattr,flock oleg435-server@tcp:/lustre /mnt/lustre mount.lustre: mount oleg435-server@tcp:/lustre at /mnt/lustre failed: Input/output error Is the MGS running? conf-sanity test_38: @@@@@@ FAIL: mount_client /mnt/lustre failed Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:6975:error() = /home/green/git/lustre-release/lustre/tests/conf-sanity.sh:3550:test_38() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7315:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7375:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7186:run_test() = /home/green/git/lustre-release/lustre/tests/conf-sanity.sh:3569:main() Dumping lctl log to /tmp/testlogs//conf-sanity.test_38.*.1709786683.log rsync: chown "/tmp/testlogs/.conf-sanity.test_38.debug_log.oleg435-server.1709786683.log.nwXFnt" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.conf-sanity.test_38.dmesg.oleg435-server.1709786683.log.7aR6Ya" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] FAIL 38 (61s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 23:44:50 (1709786690) Stopping clients: oleg435-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg435-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg435-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg435-server oleg435-server: oleg435-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg435-server' oleg435-server: oleg435-server.virtnet: executing load_modules_local oleg435-server: Loading modules from /home/green/git/lustre-release/lustre oleg435-server: detected 4 online CPUs by sysfs oleg435-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg435-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg435-server: oleg435-server.virtnet: executing set_default_debug -1 all pdsh@oleg435-client: oleg435-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg435-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg435-server: oleg435-server.virtnet: executing set_default_debug -1 all pdsh@oleg435-client: oleg435-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg435-client.virtnet: -o user_xattr,flock oleg435-server@tcp:/lustre /mnt/lustre setup single mount lustre success fail_loc=0x142 start ost2 service on oleg435-server Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg435-server: oleg435-server.virtnet: executing set_default_debug -1 all pdsh@oleg435-client: oleg435-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost2_flakey Started lustre-OST0001 oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec stop ost2 service on oleg435-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg435-server umount lustre on /mnt/lustre..... Stopping client oleg435-client.virtnet /mnt/lustre (opts:-f) stop ost1 service on oleg435-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg435-server stop mds service on oleg435-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg435-server unloading modules on: 'oleg435-server' oleg435-server: oleg435-server.virtnet: executing unload_modules_local modules unloaded. oleg435-server: tunefs.lustre: Unable to mount /dev/mapper/mds1_flakey: No such device oleg435-server: Is the ldiskfs module available? oleg435-server: oleg435-server: tunefs.lustre FATAL: failed to write local files oleg435-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg435-client: oleg435-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x5 (MDT MGS ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity Permanent disk data: Target: lustre=MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x105 (MDT MGS writeconf ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity oleg435-server: tunefs.lustre: Unable to mount /dev/mapper/ost1_flakey: No such device oleg435-server: Is the ldiskfs module available? oleg435-server: oleg435-server: tunefs.lustre FATAL: failed to write local files oleg435-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg435-client: oleg435-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x2 (OST ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.204.135@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x102 (OST writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.204.135@tcp sys.timeout=20 oleg435-server: tunefs.lustre: Unable to mount /dev/mapper/ost2_flakey: No such device oleg435-server: Is the ldiskfs module available? oleg435-server: oleg435-server: tunefs.lustre FATAL: failed to write local files oleg435-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg435-client: oleg435-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x2 (OST ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.204.135@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x102 (OST writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.204.135@tcp sys.timeout=20 tunefs failed, reformatting instead Stopping clients: oleg435-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg435-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg435-client: oleg435-server: ssh exited with exit code 2 oleg435-server: oleg435-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg435-server' oleg435-server: oleg435-server.virtnet: executing load_modules_local oleg435-server: Loading modules from /home/green/git/lustre-release/lustre oleg435-server: detected 4 online CPUs by sysfs oleg435-server: Force libcfs to create 2 CPU partitions oleg435-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg435-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg435-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg435-server: oleg435-server.virtnet: executing set_default_debug -1 all pdsh@oleg435-client: oleg435-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg435-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg435-server: oleg435-server.virtnet: executing set_default_debug -1 all pdsh@oleg435-client: oleg435-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg435-server: oleg435-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg435-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg435-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg435-server stop mds service on oleg435-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg435-server PASS 51 (167s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:47:40 (1709786860) start mds service on oleg435-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg435-server: oleg435-server.virtnet: executing set_default_debug -1 all pdsh@oleg435-client: oleg435-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg435-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg435-server: oleg435-server.virtnet: executing set_default_debug -1 all pdsh@oleg435-client: oleg435-server: ssh exited with exit code 1 Started lustre-OST0000 oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg435-client.virtnet: -o user_xattr,flock oleg435-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 281 (time 1709786891.24 total 10.00 last 28.09) - open/close 540 (time 1709786901.25 total 20.02 last 25.87) - open/close 719 (time 1709786911.55 total 30.31 last 17.39) - open/close 939 (time 1709786921.95 total 40.72 last 21.15) - open/close 1139 (time 1709786932.91 total 51.68 last 18.25) - open/close 1349 (time 1709786943.66 total 62.42 last 19.55) - open/close 1518 (time 1709786954.46 total 73.23 last 15.64) - open/close 1643 (time 1709786965.13 total 83.89 last 11.73) - open/close 1804 (time 1709786975.41 total 94.18 last 15.65) - open/close 2031 (time 1709786985.42 total 104.19 last 22.68) - open/close 2262 (time 1709786996.40 total 115.17 last 21.03) - open/close 2443 (time 1709787006.74 total 125.51 last 17.50) - open/close 2674 (time 1709787016.75 total 135.51 last 23.09) - open/close 3062 (time 1709787027.31 total 146.08 last 36.73) - open/close 3171 (time 1709787037.70 total 156.46 last 10.49) - open/close 3335 (time 1709787048.18 total 166.95 last 15.65) - open/close 3555 (time 1709787058.18 total 176.95 last 22.00) - open/close 3692 (time 1709787068.19 total 186.96 last 13.69) - open/close 3940 (time 1709787079.72 total 198.48 last 21.51) - open/close 4333 (time 1709787090.59 total 209.35 last 36.16) - open/close 4460 (time 1709787101.11 total 219.88 last 12.07) - open/close 4687 (time 1709787112.11 total 230.87 last 20.64) - open/close 4908 (time 1709787122.67 total 241.44 last 20.93) - open/close 5122 (time 1709787132.69 total 251.45 last 21.37) - open/close 5351 (time 1709787143.22 total 261.99 last 21.73) - open/close 5546 (time 1709787153.23 total 272.00 last 19.48) - open/close 5752 (time 1709787163.59 total 282.35 last 19.89) - open/close 5974 (time 1709787174.34 total 293.11 last 20.64) - open/close 6249 (time 1709787184.44 total 303.20 last 27.24) - open/close 6476 (time 1709787194.45 total 313.21 last 22.68) - open/close 6658 (time 1709787204.49 total 323.26 last 18.12) - open/close 6850 (time 1709787214.50 total 333.27 last 19.18) - open/close 7086 (time 1709787224.50 total 343.27 last 23.60) - open/close 7249 (time 1709787234.51 total 353.27 last 16.30) - open/close 7534 (time 1709787244.51 total 363.28 last 28.48) - open/close 7666 (time 1709787254.79 total 373.55 last 12.84) - open/close 7923 (time 1709787265.25 total 384.02 last 24.57) - open/close 8002 (time 1709787276.78 total 395.55 last 6.85) - open/close 8222 (time 1709787286.80 total 405.56 last 21.96) - open/close 8311 (time 1709787296.80 total 415.57 last 8.90) - open/close 8440 (time 1709787307.71 total 426.47 last 11.83) - open/close 8586 (time 1709787319.02 total 437.78 last 12.91) - open/close 8749 (time 1709787331.38 total 450.14 last 13.19) - open/close 8923 (time 1709787341.39 total 460.15 last 17.38) - open/close 9076 (time 1709787352.11 total 470.87 last 14.27) - open/close 9220 (time 1709787362.81 total 481.58 last 13.45) - open/close 9426 (time 1709787372.83 total 491.60 last 20.56) - open/close 9563 (time 1709787383.86 total 502.63 last 12.42) - open/close 9720 (time 1709787394.02 total 512.79 last 15.45) - open/close 10000 (time 1709787400.71 total 519.48 last 41.87) - open/close 10593 (time 1709787410.72 total 529.48 last 59.27) - open/close 11141 (time 1709787420.72 total 539.48 last 54.79) - open/close 11661 (time 1709787430.73 total 549.50 last 51.93) - open/close 12190 (time 1709787440.75 total 559.51 last 52.83) - open/close 12703 (time 1709787450.75 total 569.51 last 51.29) - open/close 13188 (time 1709787460.75 total 579.51 last 48.49) - open/close 13663 (time 1709787470.77 total 589.53 last 47.40) - open/close 14191 (time 1709787480.77 total 599.54 last 52.78) - open/close 14633 (time 1709787490.78 total 609.54 last 44.18) - open/close 15093 (time 1709787500.79 total 619.55 last 45.98) - open/close 15547 (time 1709787510.79 total 629.55 last 45.39) - open/close 15982 (time 1709787520.79 total 639.55 last 43.48) - open/close 16383 (time 1709787530.80 total 649.56 last 40.06) - open/close 16804 (time 1709787540.81 total 659.57 last 42.07) - open/close 17223 (time 1709787550.83 total 669.59 last 41.81) - open/close 17650 (time 1709787560.85 total 679.61 last 42.63) - open/close 17801 (time 1709787575.08 total 693.84 last 10.61) - open/close 17942 (time 1709787589.39 total 708.16 last 9.85) - open/close 18420 (time 1709787599.41 total 718.18 last 47.71) - open/close 18874 (time 1709787609.45 total 728.21 last 45.23) - open/close 19319 (time 1709787619.47 total 738.23 last 44.42) - open/close 19734 (time 1709787629.48 total 748.24 last 41.46) - open/close 20000 (time 1709787636.17 total 754.94 last 39.73) - open/close 20412 (time 1709787646.19 total 764.96 last 41.12) - open/close 20852 (time 1709787656.21 total 774.98 last 43.91) - open/close 21255 (time 1709787666.21 total 784.98 last 40.29) - open/close 21631 (time 1709787676.24 total 795.01 last 37.48) - open/close 21986 (time 1709787686.26 total 805.02 last 35.45) - open/close 22322 (time 1709787696.26 total 815.03 last 33.59) - open/close 22684 (time 1709787706.27 total 825.04 last 36.16) - open/close 23022 (time 1709787716.30 total 835.07 last 33.69) - open/close 23383 (time 1709787726.32 total 845.08 last 36.04) - open/close 23775 (time 1709787736.33 total 855.09 last 39.17) - open/close 24191 (time 1709787746.33 total 865.10 last 41.58) - open/close 24548 (time 1709787756.33 total 875.10 last 35.70) - open/close 24875 (time 1709787766.35 total 885.11 last 32.65) - open/close 25188 (time 1709787776.36 total 895.13 last 31.25) - open/close 25422 (time 1709787795.22 total 913.99 last 12.41) - open/close 25757 (time 1709787805.24 total 924.00 last 33.45) - open/close 26082 (time 1709787815.24 total 934.01 last 32.49) - open/close 26382 (time 1709787825.26 total 944.03 last 29.94) - open/close 26715 (time 1709787835.27 total 954.04 last 33.27) - open/close 27062 (time 1709787845.30 total 964.06 last 34.61) - open/close 27405 (time 1709787855.30 total 974.07 last 34.29) - open/close 27795 (time 1709787865.32 total 984.08 last 38.94) - open/close 28145 (time 1709787875.33 total 994.10 last 34.95) - open/close 28521 (time 1709787885.36 total 1004.12 last 37.51) - open/close 28865 (time 1709787895.37 total 1014.13 last 34.37) - open/close 29206 (time 1709787905.37 total 1024.14 last 34.07) - open/close 29524 (time 1709787915.41 total 1034.17 last 31.70) - open/close 29843 (time 1709787925.43 total 1044.20 last 31.81) - open/close 30000 (time 1709787930.49 total 1049.25 last 31.07) - open/close 30334 (time 1709787940.49 total 1059.25 last 33.40) - open/close 30646 (time 1709787950.52 total 1069.28 last 31.11) - open/close 30957 (time 1709787960.53 total 1079.30 last 31.04) - open/close 31268 (time 1709787970.56 total 1089.32 last 31.03) - open/close 31570 (time 1709787980.59 total 1099.36 last 30.10) - open/close 31889 (time 1709787990.61 total 1109.37 last 31.84) - open/close 32218 (time 1709788000.62 total 1119.38 last 32.88) - open/close 32558 (time 1709788010.64 total 1129.40 last 33.94) - open/close 32875 (time 1709788020.65 total 1139.42 last 31.65) - open/close 33170 (time 1709788030.66 total 1149.43 last 29.47) - open/close 33462 (time 1709788040.67 total 1159.43 last 29.18) - open/close 33743 (time 1709788050.69 total 1169.45 last 28.05) - open/close 34034 (time 1709788060.70 total 1179.46 last 29.07) - open/close 34325 (time 1709788070.70 total 1189.47 last 29.08) - open/close 34628 (time 1709788080.72 total 1199.48 last 30.26) - open/close 34934 (time 1709788090.75 total 1209.51 last 30.51) - open/close 35224 (time 1709788100.78 total 1219.54 last 28.90) - open/close 35517 (time 1709788110.81 total 1229.58 last 29.20) - open/close 35814 (time 1709788120.82 total 1239.58 last 29.69) - open/close 36103 (time 1709788130.83 total 1249.59 last 28.88) - open/close 36375 (time 1709788140.85 total 1259.61 last 27.14) - open/close 36663 (time 1709788150.89 total 1269.65 last 28.69) - open/close 36925 (time 1709788160.91 total 1279.67 last 26.14) - open/close 37184 (time 1709788170.94 total 1289.71 last 25.81) - open/close 37454 (time 1709788180.98 total 1299.74 last 26.90) - open/close 37732 (time 1709788190.99 total 1309.75 last 27.78) - open/close 37994 (time 1709788201.01 total 1319.77 last 26.14) - open/close 38258 (time 1709788211.03 total 1329.80 last 26.34) - open/close 38520 (time 1709788221.07 total 1339.84 last 26.10) - open/close 38788 (time 1709788231.11 total 1349.87 last 26.70) - open/close 39029 (time 1709788241.14 total 1359.90 last 24.03) - open/close 39277 (time 1709788251.16 total 1369.93 last 24.73) - open/close 39559 (time 1709788261.19 total 1379.95 last 28.13) - open/close 39820 (time 1709788271.21 total 1389.98 last 26.03) - open/close 40000 (time 1709788278.23 total 1397.00 last 25.64) - open/close 40064 (time 1709788291.83 total 1410.60 last 4.71) - open/close 40329 (time 1709788301.88 total 1420.64 last 26.38) - open/close 40585 (time 1709788311.92 total 1430.68 last 25.50) - open/close 40837 (time 1709788321.95 total 1440.72 last 25.11) - open/close 41098 (time 1709788331.97 total 1450.74 last 26.05) - open/close 41343 (time 1709788341.98 total 1460.74 last 24.49) - open/close 41579 (time 1709788351.99 total 1470.75 last 23.58) - open/close 41819 (time 1709788362.01 total 1480.77 last 23.95) - open/close 42055 (time 1709788372.05 total 1490.82 last 23.50) - open/close 42298 (time 1709788382.07 total 1500.84 last 24.26) - open/close 42537 (time 1709788392.09 total 1510.86 last 23.85) - open/close 42781 (time 1709788402.15 total 1520.91 last 24.27) - open/close 43020 (time 1709788412.17 total 1530.93 last 23.84) - open/close 43261 (time 1709788422.20 total 1540.96 last 24.04) - open/close 43505 (time 1709788432.22 total 1550.98 last 24.35) - open/close 43748 (time 1709788442.24 total 1561.00 last 24.24) - open/close 43986 (time 1709788452.27 total 1571.03 last 23.74) - open/close 44224 (time 1709788462.28 total 1581.05 last 23.76) - open/close 44469 (time 1709788472.30 total 1591.06 last 24.46) - open/close 44701 (time 1709788482.32 total 1601.09 last 23.15) - open/close 44943 (time 1709788492.35 total 1611.12 last 24.12) - open/close 45188 (time 1709788502.37 total 1621.14 last 24.45) - open/close 45440 (time 1709788512.38 total 1631.14 last 25.18) - open/close 45664 (time 1709788522.39 total 1641.15 last 22.39) - open/close 45897 (time 1709788532.42 total 1651.18 last 23.23) - open/close 46131 (time 1709788542.46 total 1661.22 last 23.31) - open/close 46368 (time 1709788552.50 total 1671.26 last 23.59) - open/close 46601 (time 1709788562.54 total 1681.30 last 23.22) - open/close 46833 (time 1709788572.58 total 1691.34 last 23.10) - open/close 47068 (time 1709788582.59 total 1701.35 last 23.47) - open/close 47307 (time 1709788592.62 total 1711.38 last 23.82) - open/close 47549 (time 1709788602.66 total 1721.42 last 24.11) - open/close 47780 (time 1709788612.67 total 1731.44 last 23.07) - open/close 48019 (time 1709788622.71 total 1741.48 last 23.80) - open/close 48266 (time 1709788632.72 total 1751.48 last 24.69) - open/close 48513 (time 1709788642.72 total 1761.49 last 24.70) - open/close 48755 (time 1709788652.74 total 1771.51 last 24.15) - open/close 48992 (time 1709788662.77 total 1781.53 last 23.65) - open/close 49220 (time 1709788672.81 total 1791.57 last 22.71) - open/close 49467 (time 1709788682.85 total 1801.61 last 24.60) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49698) error: No space left on device total: 49698 open/close in 1810.73 seconds: 27.45 ops/second - unlinked 0 (time 1709788693 ; total 0 ; last 0) - unlinked 10000 (time 1709789187 ; total 494 ; last 494) - unlinked 20000 (time 1709789607 ; total 914 ; last 420) - unlinked 30000 (time 1709789912 ; total 1219 ; last 305) - unlinked 40000 (time 1709790059 ; total 1366 ; last 147) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49698) error: No such file or directory total: 49698 unlinks in 1423 seconds: 34.924809 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg435-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg435-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg435-server stop mds service on oleg435-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg435-server unloading modules on: 'oleg435-server' oleg435-server: oleg435-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg435-client: oleg435-client: ssh exited with exit code 2 pdsh@oleg435-client: oleg435-server: ssh exited with exit code 2 pdsh@oleg435-client: oleg435-client: ssh exited with exit code 2 pdsh@oleg435-client: oleg435-server: ssh exited with exit code 2 PASS 101a (3272s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg435-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg435-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg435-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg435-client: oleg435-server: ssh exited with exit code 2 oleg435-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg435-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == conf-sanity test 101b: Race events DISCONNECT and ACTIVE in osp ========================================================== 00:42:14 (1709790134) start mds service on oleg435-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg435-server' oleg435-server: oleg435-server.virtnet: executing load_modules_local oleg435-server: Loading modules from /home/green/git/lustre-release/lustre oleg435-server: detected 4 online CPUs by sysfs oleg435-server: Force libcfs to create 2 CPU partitions oleg435-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg435-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg435-server: oleg435-server.virtnet: executing set_default_debug -1 all pdsh@oleg435-client: oleg435-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg435-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg435-server: oleg435-server.virtnet: executing set_default_debug -1 all pdsh@oleg435-client: oleg435-server: ssh exited with exit code 1 Started lustre-OST0000 oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg435-client.virtnet: -o user_xattr,flock oleg435-server@tcp:/lustre /mnt/lustre fail_loc=0x80002107 fail_val=20 stop ost1 service on oleg435-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg435-server start ost1 service on oleg435-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg435-server: oleg435-server.virtnet: executing set_default_debug -1 all pdsh@oleg435-client: oleg435-server: ssh exited with exit code 1 Started lustre-OST0000 oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 5 sec oleg435-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff88012a538000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff88012a538000.ost_server_uuid in FULL state after 0 sec umount lustre on /mnt/lustre..... Stopping client oleg435-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg435-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg435-server stop mds service on oleg435-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg435-server unloading modules on: 'oleg435-server' oleg435-server: oleg435-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg435-client: oleg435-client: ssh exited with exit code 2 pdsh@oleg435-client: oleg435-server: ssh exited with exit code 2 pdsh@oleg435-client: oleg435-client: ssh exited with exit code 2 pdsh@oleg435-client: oleg435-server: ssh exited with exit code 2 PASS 101b (88s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg435-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg435-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory Stopping clients: oleg435-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg435-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg435-client: oleg435-server: ssh exited with exit code 2 oleg435-server: oleg435-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg435-server' oleg435-server: oleg435-server.virtnet: executing load_modules_local oleg435-server: Loading modules from /home/green/git/lustre-release/lustre oleg435-server: detected 4 online CPUs by sysfs oleg435-server: Force libcfs to create 2 CPU partitions oleg435-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg435-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey == conf-sanity test complete, duration 3800 sec ========== 00:44:01 (1709790241) conf-sanity: FAIL: test_38 mount_client /mnt/lustre failed === conf-sanity: start cleanup 00:44:01 (1709790241) === === conf-sanity: finish cleanup 00:44:01 (1709790241) ===