-----============= acceptance-small: conf-sanity ============----- Wed Mar 6 23:39:02 EST 2024 excepting tests: 32newtarball 110 skipping tests SLOW=no: 45 69 106 111 114 Stopping clients: oleg414-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg414-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg414-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg414-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg414-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg414-server oleg414-server: oleg414-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg414-server' oleg414-server: oleg414-server.virtnet: executing load_modules_local oleg414-server: Loading modules from /home/green/git/lustre-release/lustre oleg414-server: detected 4 online CPUs by sysfs oleg414-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg414-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg414-server: oleg414-server.virtnet: executing set_default_debug -1 all pdsh@oleg414-client: oleg414-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg414-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg414-server: oleg414-server.virtnet: executing set_default_debug -1 all pdsh@oleg414-client: oleg414-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg414-server: oleg414-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg414-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg414-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg414-server stop mds service on oleg414-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg414-server debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 35b: Continue reconnection retries, if the active server is busy ========================================================== 23:40:04 (1709786404) start mds service on oleg414-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg414-server: oleg414-server.virtnet: executing set_default_debug -1 all pdsh@oleg414-client: oleg414-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg414-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg414-server: oleg414-server.virtnet: executing set_default_debug -1 all pdsh@oleg414-client: oleg414-server: ssh exited with exit code 1 Started lustre-OST0000 oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg414-client.virtnet: -o user_xattr,flock oleg414-server@tcp:/lustre /mnt/lustre debug=ha conf-sanity.sh test_35b 2024-03-0623h40m19s Set up a fake failnode for the MDS at_max=0 at_max=0 Injecting EBUSY on MDS fail_loc=0x80000136 mdc.lustre-MDT0000-mdc-ffff8800abacf800.stats=clear Creating a test file and stat it File: '/mnt/lustre/d35b.conf-sanity/f35b.conf-sanity' Size: 0 Blocks: 0 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115238826934274 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-03-06 23:41:02.000000000 -0500 Modify: 2024-03-06 23:41:02.000000000 -0500 Change: 2024-03-06 23:41:02.000000000 -0500 Birth: - Stop injecting EBUSY on MDS fail_loc=0 done at_max=600 at_max=600 Debug log: 74 lines, 74 kept, 0 dropped, 0 bad. umount lustre on /mnt/lustre..... Stopping client oleg414-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg414-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg414-server stop mds service on oleg414-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg414-server unloading modules on: 'oleg414-server' oleg414-server: oleg414-server.virtnet: executing unload_modules_local modules unloaded. oleg414-server: tunefs.lustre: Unable to mount /dev/mapper/mds1_flakey: No such device oleg414-server: Is the ldiskfs module available? oleg414-server: oleg414-server: tunefs.lustre FATAL: failed to write local files oleg414-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg414-client: oleg414-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x5 (MDT MGS ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity Permanent disk data: Target: lustre=MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x105 (MDT MGS writeconf ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity oleg414-server: tunefs.lustre: Unable to mount /dev/mapper/ost1_flakey: No such device oleg414-server: Is the ldiskfs module available? oleg414-server: oleg414-server: tunefs.lustre FATAL: failed to write local files oleg414-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg414-client: oleg414-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x2 (OST ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.204.114@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x102 (OST writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.204.114@tcp sys.timeout=20 oleg414-server: tunefs.lustre: Unable to mount /dev/mapper/ost2_flakey: No such device oleg414-server: Is the ldiskfs module available? oleg414-server: oleg414-server: tunefs.lustre FATAL: failed to write local files oleg414-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg414-client: oleg414-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.204.114@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x162 (OST first_time update writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.204.114@tcp sys.timeout=20 tunefs failed, reformatting instead Stopping clients: oleg414-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg414-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg414-client: oleg414-server: ssh exited with exit code 2 oleg414-server: oleg414-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg414-server' oleg414-server: oleg414-server.virtnet: executing load_modules_local oleg414-server: Loading modules from /home/green/git/lustre-release/lustre oleg414-server: detected 4 online CPUs by sysfs oleg414-server: Force libcfs to create 2 CPU partitions oleg414-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg414-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg414-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg414-server: oleg414-server.virtnet: executing set_default_debug -1 all pdsh@oleg414-client: oleg414-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg414-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg414-server: oleg414-server.virtnet: executing set_default_debug -1 all pdsh@oleg414-client: oleg414-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg414-server: oleg414-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg414-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg414-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg414-server stop mds service on oleg414-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg414-server PASS 35b (109s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 23:41:55 (1709786515) start mds service on oleg414-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg414-server: oleg414-server.virtnet: executing set_default_debug -1 all pdsh@oleg414-client: oleg414-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg414-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg414-server: oleg414-server.virtnet: executing set_default_debug -1 all pdsh@oleg414-client: oleg414-server: ssh exited with exit code 1 Started lustre-OST0000 oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg414-client.virtnet: -o user_xattr,flock oleg414-server@tcp:/lustre /mnt/lustre copying 10 files to /mnt/lustre/d38.conf-sanity tar: Removing leading `/' from member names umount lustre on /mnt/lustre..... Stopping client oleg414-client.virtnet /mnt/lustre (opts:) osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=44 stop mds service on oleg414-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg414-server delete lov_objid file on MDS 000000 43 000008 start mds service on oleg414-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg414-server: oleg414-server.virtnet: executing set_default_debug -1 all pdsh@oleg414-client: oleg414-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg414-client.virtnet: -o user_xattr,flock oleg414-server@tcp:/lustre /mnt/lustre osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=2 umount lustre on /mnt/lustre..... Stopping client oleg414-client.virtnet /mnt/lustre (opts:) stop mds service on oleg414-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg414-server 000000 2 000008 oleg414-server: 8+0 records in oleg414-server: 8+0 records out oleg414-server: 4096 bytes (4.1 kB) copied, 0.000635983 s, 6.4 MB/s start mds service on oleg414-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg414-server: oleg414-server.virtnet: executing set_default_debug -1 all pdsh@oleg414-client: oleg414-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg414-client.virtnet: -o user_xattr,flock oleg414-server@tcp:/lustre /mnt/lustre osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=35 umount lustre on /mnt/lustre..... Stopping client oleg414-client.virtnet /mnt/lustre (opts:) stop mds service on oleg414-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg414-server 000000 34 000008 files compared the same umount lustre on /mnt/lustre..... stop ost1 service on oleg414-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg414-server stop mds service on oleg414-server unloading modules on: 'oleg414-server' oleg414-server: oleg414-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg414-client: oleg414-client: ssh exited with exit code 2 pdsh@oleg414-client: oleg414-server: ssh exited with exit code 2 pdsh@oleg414-client: oleg414-client: ssh exited with exit code 2 pdsh@oleg414-client: oleg414-server: ssh exited with exit code 2 PASS 38 (62s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg414-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg414-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg414-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg414-client: oleg414-server: ssh exited with exit code 2 oleg414-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg414-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 23:42:59 (1709786579) Stopping clients: oleg414-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg414-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg414-client: oleg414-server: ssh exited with exit code 2 oleg414-server: oleg414-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg414-server' oleg414-server: oleg414-server.virtnet: executing load_modules_local oleg414-server: Loading modules from /home/green/git/lustre-release/lustre oleg414-server: detected 4 online CPUs by sysfs oleg414-server: Force libcfs to create 2 CPU partitions oleg414-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg414-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg414-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg414-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg414-server: oleg414-server.virtnet: executing set_default_debug -1 all pdsh@oleg414-client: oleg414-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg414-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg414-server: oleg414-server.virtnet: executing set_default_debug -1 all pdsh@oleg414-client: oleg414-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg414-client.virtnet: -o user_xattr,flock oleg414-server@tcp:/lustre /mnt/lustre setup single mount lustre success fail_loc=0x142 start ost2 service on oleg414-server Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg414-server: oleg414-server.virtnet: executing set_default_debug -1 all pdsh@oleg414-client: oleg414-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost2_flakey Started lustre-OST0001 oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec stop ost2 service on oleg414-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg414-server umount lustre on /mnt/lustre..... Stopping client oleg414-client.virtnet /mnt/lustre (opts:-f) stop ost1 service on oleg414-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg414-server stop mds service on oleg414-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg414-server unloading modules on: 'oleg414-server' oleg414-server: oleg414-server.virtnet: executing unload_modules_local modules unloaded. oleg414-server: tunefs.lustre: Unable to mount /dev/mapper/mds1_flakey: No such device oleg414-server: Is the ldiskfs module available? oleg414-server: oleg414-server: tunefs.lustre FATAL: failed to write local files oleg414-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg414-client: oleg414-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x5 (MDT MGS ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity Permanent disk data: Target: lustre=MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x105 (MDT MGS writeconf ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity oleg414-server: tunefs.lustre: Unable to mount /dev/mapper/ost1_flakey: No such device oleg414-server: Is the ldiskfs module available? oleg414-server: oleg414-server: tunefs.lustre FATAL: failed to write local files oleg414-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg414-client: oleg414-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x2 (OST ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.204.114@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x102 (OST writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.204.114@tcp sys.timeout=20 oleg414-server: tunefs.lustre: Unable to mount /dev/mapper/ost2_flakey: No such device oleg414-server: Is the ldiskfs module available? oleg414-server: oleg414-server: tunefs.lustre FATAL: failed to write local files oleg414-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg414-client: oleg414-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x2 (OST ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.204.114@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x102 (OST writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.204.114@tcp sys.timeout=20 tunefs failed, reformatting instead Stopping clients: oleg414-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg414-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg414-client: oleg414-server: ssh exited with exit code 2 oleg414-server: oleg414-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg414-server' oleg414-server: oleg414-server.virtnet: executing load_modules_local oleg414-server: Loading modules from /home/green/git/lustre-release/lustre oleg414-server: detected 4 online CPUs by sysfs oleg414-server: Force libcfs to create 2 CPU partitions oleg414-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg414-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg414-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg414-server: oleg414-server.virtnet: executing set_default_debug -1 all pdsh@oleg414-client: oleg414-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg414-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg414-server: oleg414-server.virtnet: executing set_default_debug -1 all pdsh@oleg414-client: oleg414-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg414-server: oleg414-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg414-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg414-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg414-server stop mds service on oleg414-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg414-server PASS 51 (163s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:45:45 (1709786745) start mds service on oleg414-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg414-server: oleg414-server.virtnet: executing set_default_debug -1 all pdsh@oleg414-client: oleg414-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg414-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg414-server: oleg414-server.virtnet: executing set_default_debug -1 all pdsh@oleg414-client: oleg414-server: ssh exited with exit code 1 Started lustre-OST0000 oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg414-client.virtnet: -o user_xattr,flock oleg414-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 247 (time 1709786774.54 total 10.50 last 23.52) - open/close 601 (time 1709786784.75 total 20.71 last 34.68) - open/close 904 (time 1709786795.40 total 31.37 last 28.44) - open/close 1044 (time 1709786805.40 total 41.37 last 14.00) - open/close 1317 (time 1709786815.85 total 51.82 last 26.13) - open/close 1548 (time 1709786826.49 total 62.46 last 21.71) - open/close 1863 (time 1709786837.39 total 73.36 last 28.89) - open/close 2113 (time 1709786847.65 total 83.62 last 24.37) - open/close 2538 (time 1709786858.39 total 94.36 last 39.57) - open/close 2799 (time 1709786868.40 total 104.37 last 26.08) - open/close 2904 (time 1709786878.53 total 114.50 last 10.36) - open/close 3004 (time 1709786889.63 total 125.60 last 9.01) - open/close 3194 (time 1709786900.39 total 136.36 last 17.65) - open/close 3491 (time 1709786910.43 total 146.39 last 29.60) - open/close 3661 (time 1709786920.43 total 156.40 last 16.99) - open/close 3801 (time 1709786930.47 total 166.44 last 13.95) - open/close 3964 (time 1709786941.16 total 177.13 last 15.25) - open/close 4309 (time 1709786951.18 total 187.15 last 34.43) - open/close 4446 (time 1709786963.16 total 199.12 last 11.44) - open/close 4657 (time 1709786973.73 total 209.70 last 19.96) - open/close 4836 (time 1709786983.73 total 219.70 last 17.89) - open/close 5179 (time 1709786994.16 total 230.13 last 32.88) - open/close 5445 (time 1709787005.16 total 241.13 last 24.19) - open/close 5572 (time 1709787016.05 total 252.01 last 11.67) - open/close 5794 (time 1709787026.60 total 262.57 last 21.04) - open/close 6052 (time 1709787037.33 total 273.29 last 24.05) - open/close 6266 (time 1709787047.77 total 283.74 last 20.49) - open/close 6451 (time 1709787057.81 total 293.78 last 18.43) - open/close 6615 (time 1709787068.25 total 304.21 last 15.72) - open/close 6716 (time 1709787078.25 total 314.22 last 10.09) - open/close 6890 (time 1709787088.28 total 324.24 last 17.36) - open/close 7092 (time 1709787098.28 total 334.25 last 20.19) - open/close 7407 (time 1709787108.29 total 344.26 last 31.47) - open/close 7576 (time 1709787118.63 total 354.60 last 16.34) - open/close 7862 (time 1709787128.81 total 364.78 last 28.09) - open/close 8012 (time 1709787139.34 total 375.30 last 14.25) - open/close 8208 (time 1709787149.39 total 385.36 last 19.50) - open/close 8476 (time 1709787159.39 total 395.36 last 26.79) - open/close 8692 (time 1709787169.92 total 405.89 last 20.52) - open/close 8970 (time 1709787179.92 total 415.89 last 27.79) - open/close 9283 (time 1709787190.85 total 426.81 last 28.66) - open/close 9454 (time 1709787200.87 total 436.84 last 17.06) - open/close 9640 (time 1709787211.48 total 447.45 last 17.53) - open/close 9813 (time 1709787222.15 total 458.12 last 16.21) - open/close 10000 (time 1709787231.29 total 467.26 last 20.46) - open/close 10502 (time 1709787250.38 total 486.35 last 26.30) - open/close 11108 (time 1709787260.39 total 496.35 last 60.55) - open/close 11703 (time 1709787270.39 total 506.36 last 59.49) - open/close 12208 (time 1709787280.50 total 516.47 last 49.93) - open/close 12747 (time 1709787290.53 total 526.49 last 53.77) - open/close 13275 (time 1709787300.54 total 536.50 last 52.76) - open/close 13720 (time 1709787310.54 total 546.50 last 44.49) - open/close 14268 (time 1709787320.55 total 556.52 last 54.70) - open/close 14756 (time 1709787330.58 total 566.54 last 48.70) - open/close 15184 (time 1709787340.59 total 576.55 last 42.75) - open/close 15666 (time 1709787350.59 total 586.56 last 48.18) - open/close 16173 (time 1709787360.61 total 596.58 last 50.60) - open/close 16606 (time 1709787370.63 total 606.60 last 43.23) - open/close 17043 (time 1709787380.64 total 616.61 last 43.65) - open/close 17507 (time 1709787390.65 total 626.61 last 46.37) - open/close 17946 (time 1709787400.66 total 636.63 last 43.84) - open/close 18346 (time 1709787410.66 total 646.63 last 39.98) - open/close 18730 (time 1709787420.70 total 656.66 last 38.27) - open/close 18825 (time 1709787435.34 total 671.31 last 6.49) - open/close 19197 (time 1709787445.37 total 681.33 last 37.11) - open/close 19259 (time 1709787458.13 total 694.10 last 4.86) - open/close 19600 (time 1709787468.14 total 704.10 last 34.09) - open/close 19699 (time 1709787481.85 total 717.82 last 7.22) - open/close 20000 (time 1709787489.78 total 725.75 last 37.95) - open/close 20341 (time 1709787499.82 total 735.79 last 33.97) - open/close 20663 (time 1709787509.85 total 745.81 last 32.12) - open/close 20959 (time 1709787519.86 total 755.83 last 29.56) - open/close 21276 (time 1709787529.88 total 765.85 last 31.64) - open/close 21625 (time 1709787539.91 total 775.87 last 34.81) - open/close 21985 (time 1709787549.93 total 785.90 last 35.90) - open/close 22326 (time 1709787559.96 total 795.93 last 34.00) - open/close 22694 (time 1709787569.98 total 805.94 last 36.75) - open/close 23071 (time 1709787579.98 total 815.95 last 37.69) - open/close 23321 (time 1709787597.42 total 833.38 last 14.34) - open/close 23698 (time 1709787607.45 total 843.42 last 37.56) - open/close 24059 (time 1709787617.47 total 853.44 last 36.04) - open/close 24395 (time 1709787627.49 total 863.46 last 33.52) - open/close 24737 (time 1709787637.52 total 873.48 last 34.12) - open/close 25100 (time 1709787647.53 total 883.49 last 36.27) - open/close 25472 (time 1709787657.53 total 893.50 last 37.18) - open/close 25835 (time 1709787667.54 total 903.50 last 36.29) - open/close 26154 (time 1709787677.54 total 913.50 last 31.90) - open/close 26450 (time 1709787687.54 total 923.51 last 29.59) - open/close 26750 (time 1709787697.55 total 933.52 last 29.97) - open/close 27037 (time 1709787707.56 total 943.53 last 28.68) - open/close 27327 (time 1709787717.57 total 953.54 last 28.96) - open/close 27635 (time 1709787727.60 total 963.56 last 30.72) - open/close 27983 (time 1709787737.61 total 973.58 last 34.74) - open/close 28366 (time 1709787747.64 total 983.60 last 38.22) - open/close 28690 (time 1709787757.65 total 993.62 last 32.36) - open/close 28983 (time 1709787767.68 total 1003.65 last 29.21) - open/close 29268 (time 1709787777.68 total 1013.65 last 28.48) - open/close 29528 (time 1709787787.69 total 1023.66 last 25.97) - open/close 29782 (time 1709787797.70 total 1033.67 last 25.38) - open/close 30000 (time 1709787804.55 total 1040.51 last 31.86) - open/close 30298 (time 1709787814.57 total 1050.54 last 29.72) - open/close 30579 (time 1709787824.61 total 1060.58 last 28.00) - open/close 30861 (time 1709787834.64 total 1070.60 last 28.12) - open/close 31179 (time 1709787844.65 total 1080.62 last 31.76) - open/close 31500 (time 1709787854.66 total 1090.63 last 32.05) - open/close 31852 (time 1709787864.68 total 1100.65 last 35.15) - open/close 32190 (time 1709787874.69 total 1110.65 last 33.77) - open/close 32536 (time 1709787884.71 total 1120.67 last 34.53) - open/close 32876 (time 1709787894.74 total 1130.70 last 33.89) - open/close 33203 (time 1709787904.75 total 1140.72 last 32.66) - open/close 33511 (time 1709787914.76 total 1150.73 last 30.76) - open/close 33831 (time 1709787924.78 total 1160.74 last 31.96) - open/close 34139 (time 1709787934.79 total 1170.76 last 30.76) - open/close 34454 (time 1709787944.80 total 1180.76 last 31.47) - open/close 34765 (time 1709787954.82 total 1190.79 last 31.03) - open/close 35070 (time 1709787964.82 total 1200.79 last 30.48) - open/close 35378 (time 1709787974.85 total 1210.82 last 30.71) - open/close 35681 (time 1709787984.86 total 1220.82 last 30.29) - open/close 35989 (time 1709787994.87 total 1230.84 last 30.76) - open/close 36303 (time 1709788004.90 total 1240.86 last 31.32) - open/close 36610 (time 1709788014.90 total 1250.86 last 30.69) - open/close 36915 (time 1709788024.91 total 1260.88 last 30.46) - open/close 37212 (time 1709788034.95 total 1270.91 last 29.58) - open/close 37503 (time 1709788044.97 total 1280.94 last 29.03) - open/close 37784 (time 1709788054.99 total 1290.96 last 28.04) - open/close 38076 (time 1709788065.02 total 1300.99 last 29.12) - open/close 38360 (time 1709788075.05 total 1311.01 last 28.33) - open/close 38651 (time 1709788085.06 total 1321.03 last 29.05) - open/close 38943 (time 1709788095.09 total 1331.06 last 29.11) - open/close 39232 (time 1709788105.11 total 1341.08 last 28.85) - open/close 39526 (time 1709788115.13 total 1351.10 last 29.35) - open/close 39812 (time 1709788125.14 total 1361.11 last 28.56) - open/close 40000 (time 1709788131.92 total 1367.89 last 27.74) - open/close 40268 (time 1709788141.94 total 1377.91 last 26.74) - open/close 40543 (time 1709788151.99 total 1387.96 last 27.37) - open/close 40783 (time 1709788162.04 total 1398.00 last 23.89) - open/close 41051 (time 1709788172.05 total 1408.02 last 26.76) - open/close 41317 (time 1709788182.08 total 1418.05 last 26.54) - open/close 41592 (time 1709788192.11 total 1428.08 last 27.41) - open/close 41859 (time 1709788202.15 total 1438.12 last 26.60) - open/close 42115 (time 1709788212.18 total 1448.15 last 25.51) - open/close 42360 (time 1709788222.20 total 1458.17 last 24.46) - open/close 42619 (time 1709788232.22 total 1468.18 last 25.85) - open/close 42862 (time 1709788242.25 total 1478.21 last 24.23) - open/close 43103 (time 1709788252.25 total 1488.21 last 24.10) - open/close 43368 (time 1709788262.26 total 1498.23 last 26.46) - open/close 43619 (time 1709788272.28 total 1508.25 last 25.06) - open/close 43874 (time 1709788282.29 total 1518.25 last 25.48) - open/close 44135 (time 1709788292.31 total 1528.28 last 26.03) - open/close 44385 (time 1709788302.33 total 1538.30 last 24.96) - open/close 44627 (time 1709788312.34 total 1548.30 last 24.18) - open/close 44867 (time 1709788322.38 total 1558.34 last 23.91) - open/close 45115 (time 1709788332.39 total 1568.36 last 24.76) - open/close 45352 (time 1709788342.45 total 1578.41 last 23.57) - open/close 45582 (time 1709788352.48 total 1588.45 last 22.92) - open/close 45812 (time 1709788362.50 total 1598.47 last 22.96) - open/close 46041 (time 1709788372.54 total 1608.51 last 22.80) - open/close 46270 (time 1709788382.56 total 1618.52 last 22.86) - open/close 46498 (time 1709788392.59 total 1628.56 last 22.72) - open/close 46731 (time 1709788402.62 total 1638.58 last 23.25) - open/close 46962 (time 1709788412.64 total 1648.60 last 23.05) - open/close 47191 (time 1709788422.67 total 1658.64 last 22.82) - open/close 47420 (time 1709788432.69 total 1668.66 last 22.86) - open/close 47652 (time 1709788442.70 total 1678.67 last 23.18) - open/close 47884 (time 1709788452.70 total 1688.67 last 23.20) - open/close 48120 (time 1709788462.74 total 1698.71 last 23.51) - open/close 48351 (time 1709788472.74 total 1708.71 last 23.09) - open/close 48579 (time 1709788482.76 total 1718.73 last 22.76) - open/close 48818 (time 1709788492.80 total 1728.77 last 23.81) - open/close 49052 (time 1709788502.80 total 1738.77 last 23.40) - open/close 49295 (time 1709788512.83 total 1748.79 last 24.25) - open/close 49512 (time 1709788522.86 total 1758.83 last 21.62) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49698) error: No space left on device total: 49698 open/close in 1767.11 seconds: 28.12 ops/second - unlinked 0 (time 1709788532 ; total 0 ; last 0) - unlinked 10000 (time 1709789019 ; total 487 ; last 487) - unlinked 20000 (time 1709789423 ; total 891 ; last 404) - unlinked 30000 (time 1709789730 ; total 1198 ; last 307) - unlinked 40000 (time 1709789900 ; total 1368 ; last 170) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49698) error: No such file or directory total: 49698 unlinks in 1430 seconds: 34.753845 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg414-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg414-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg414-server stop mds service on oleg414-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg414-server unloading modules on: 'oleg414-server' oleg414-server: oleg414-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg414-client: oleg414-client: ssh exited with exit code 2 pdsh@oleg414-client: oleg414-server: ssh exited with exit code 2 pdsh@oleg414-client: oleg414-client: ssh exited with exit code 2 pdsh@oleg414-client: oleg414-server: ssh exited with exit code 2 PASS 101a (3232s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg414-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg414-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg414-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg414-client: oleg414-server: ssh exited with exit code 2 oleg414-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg414-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == conf-sanity test 101b: Race events DISCONNECT and ACTIVE in osp ========================================================== 00:39:39 (1709789979) start mds service on oleg414-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg414-server' oleg414-server: oleg414-server.virtnet: executing load_modules_local oleg414-server: Loading modules from /home/green/git/lustre-release/lustre oleg414-server: detected 4 online CPUs by sysfs oleg414-server: Force libcfs to create 2 CPU partitions oleg414-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg414-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg414-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg414-server: oleg414-server.virtnet: executing set_default_debug -1 all pdsh@oleg414-client: oleg414-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg414-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg414-server: oleg414-server.virtnet: executing set_default_debug -1 all pdsh@oleg414-client: oleg414-server: ssh exited with exit code 1 Started lustre-OST0000 oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg414-client.virtnet: -o user_xattr,flock oleg414-server@tcp:/lustre /mnt/lustre fail_loc=0x80002107 fail_val=20 stop ost1 service on oleg414-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg414-server start ost1 service on oleg414-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg414-server: oleg414-server.virtnet: executing set_default_debug -1 all pdsh@oleg414-client: oleg414-server: ssh exited with exit code 1 Started lustre-OST0000 oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 5 sec oleg414-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff88012c44d800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff88012c44d800.ost_server_uuid in FULL state after 0 sec umount lustre on /mnt/lustre..... Stopping client oleg414-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg414-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg414-server stop mds service on oleg414-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg414-server unloading modules on: 'oleg414-server' oleg414-server: oleg414-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg414-client: oleg414-client: ssh exited with exit code 2 pdsh@oleg414-client: oleg414-server: ssh exited with exit code 2 pdsh@oleg414-client: oleg414-client: ssh exited with exit code 2 pdsh@oleg414-client: oleg414-server: ssh exited with exit code 2 PASS 101b (87s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg414-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg414-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory Stopping clients: oleg414-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg414-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg414-client: oleg414-server: ssh exited with exit code 2 oleg414-server: oleg414-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg414-server' oleg414-server: oleg414-server.virtnet: executing load_modules_local oleg414-server: Loading modules from /home/green/git/lustre-release/lustre oleg414-server: detected 4 online CPUs by sysfs oleg414-server: Force libcfs to create 2 CPU partitions oleg414-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg414-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey == conf-sanity test complete, duration 3745 sec ========== 00:41:28 (1709790088) === conf-sanity: start cleanup 00:41:28 (1709790088) === === conf-sanity: finish cleanup 00:41:29 (1709790089) ===