-----============= acceptance-small: conf-sanity ============----- Wed Mar 6 23:39:33 EST 2024 excepting tests: 32newtarball 110 skipping tests SLOW=no: 45 69 106 111 114 Stopping clients: oleg232-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg232-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg232-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg232-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg232-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg232-server oleg232-server: oleg232-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg232-server' oleg232-server: oleg232-server.virtnet: executing load_modules_local oleg232-server: Loading modules from /home/green/git/lustre-release/lustre oleg232-server: detected 4 online CPUs by sysfs oleg232-server: Force libcfs to create 2 CPU partitions oleg232-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg232-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg232-server: oleg232-server.virtnet: executing set_default_debug -1 all pdsh@oleg232-client: oleg232-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg232-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg232-server: oleg232-server.virtnet: executing set_default_debug -1 all pdsh@oleg232-client: oleg232-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg232-server: oleg232-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg232-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg232-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg232-server stop mds service on oleg232-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg232-server debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 35b: Continue reconnection retries, if the active server is busy ========================================================== 23:40:40 (1709786440) start mds service on oleg232-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg232-server: oleg232-server.virtnet: executing set_default_debug -1 all pdsh@oleg232-client: oleg232-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg232-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg232-server: oleg232-server.virtnet: executing set_default_debug -1 all pdsh@oleg232-client: oleg232-server: ssh exited with exit code 1 Started lustre-OST0000 oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg232-client.virtnet: -o user_xattr,flock oleg232-server@tcp:/lustre /mnt/lustre debug=ha conf-sanity.sh test_35b 2024-03-0623h40m58s Set up a fake failnode for the MDS at_max=0 at_max=0 Injecting EBUSY on MDS fail_loc=0x80000136 mdc.lustre-MDT0000-mdc-ffff88012aab1800.stats=clear Creating a test file and stat it File: '/mnt/lustre/d35b.conf-sanity/f35b.conf-sanity' Size: 0 Blocks: 0 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115238826934274 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-03-06 23:41:41.000000000 -0500 Modify: 2024-03-06 23:41:41.000000000 -0500 Change: 2024-03-06 23:41:41.000000000 -0500 Birth: - Stop injecting EBUSY on MDS fail_loc=0 done at_max=600 at_max=600 Debug log: 56 lines, 56 kept, 0 dropped, 0 bad. umount lustre on /mnt/lustre..... Stopping client oleg232-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg232-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg232-server stop mds service on oleg232-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg232-server unloading modules on: 'oleg232-server' oleg232-server: oleg232-server.virtnet: executing unload_modules_local modules unloaded. oleg232-server: tunefs.lustre: Unable to mount /dev/mapper/mds1_flakey: No such device oleg232-server: Is the ldiskfs module available? oleg232-server: oleg232-server: tunefs.lustre FATAL: failed to write local files oleg232-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg232-client: oleg232-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x5 (MDT MGS ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity Permanent disk data: Target: lustre=MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x105 (MDT MGS writeconf ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity oleg232-server: tunefs.lustre: Unable to mount /dev/mapper/ost1_flakey: No such device oleg232-server: Is the ldiskfs module available? oleg232-server: oleg232-server: tunefs.lustre FATAL: failed to write local files oleg232-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg232-client: oleg232-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x2 (OST ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.202.132@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x102 (OST writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.202.132@tcp sys.timeout=20 oleg232-server: tunefs.lustre: Unable to mount /dev/mapper/ost2_flakey: No such device oleg232-server: Is the ldiskfs module available? oleg232-server: oleg232-server: tunefs.lustre FATAL: failed to write local files oleg232-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg232-client: oleg232-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.202.132@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x162 (OST first_time update writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.202.132@tcp sys.timeout=20 tunefs failed, reformatting instead Stopping clients: oleg232-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg232-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg232-client: oleg232-server: ssh exited with exit code 2 oleg232-server: oleg232-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg232-server' oleg232-server: oleg232-server.virtnet: executing load_modules_local oleg232-server: Loading modules from /home/green/git/lustre-release/lustre oleg232-server: detected 4 online CPUs by sysfs oleg232-server: Force libcfs to create 2 CPU partitions oleg232-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg232-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg232-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg232-server: oleg232-server.virtnet: executing set_default_debug -1 all pdsh@oleg232-client: oleg232-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg232-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg232-server: oleg232-server.virtnet: executing set_default_debug -1 all pdsh@oleg232-client: oleg232-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg232-server: oleg232-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg232-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg232-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg232-server stop mds service on oleg232-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg232-server PASS 35b (121s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 23:42:43 (1709786563) start mds service on oleg232-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg232-server: oleg232-server.virtnet: executing set_default_debug -1 all pdsh@oleg232-client: oleg232-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg232-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg232-server: oleg232-server.virtnet: executing set_default_debug -1 all pdsh@oleg232-client: oleg232-server: ssh exited with exit code 1 Started lustre-OST0000 oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg232-client.virtnet: -o user_xattr,flock oleg232-server@tcp:/lustre /mnt/lustre copying 10 files to /mnt/lustre/d38.conf-sanity tar: Removing leading `/' from member names umount lustre on /mnt/lustre..... Stopping client oleg232-client.virtnet /mnt/lustre (opts:) osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=44 stop mds service on oleg232-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg232-server delete lov_objid file on MDS 000000 43 000008 start mds service on oleg232-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg232-server: oleg232-server.virtnet: executing set_default_debug -1 all pdsh@oleg232-client: oleg232-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg232-client.virtnet: -o user_xattr,flock oleg232-server@tcp:/lustre /mnt/lustre osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=2 umount lustre on /mnt/lustre..... Stopping client oleg232-client.virtnet /mnt/lustre (opts:) stop mds service on oleg232-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg232-server 000000 2 000008 oleg232-server: 8+0 records in oleg232-server: 8+0 records out oleg232-server: 4096 bytes (4.1 kB) copied, 0.0013779 s, 3.0 MB/s start mds service on oleg232-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg232-server: oleg232-server.virtnet: executing set_default_debug -1 all pdsh@oleg232-client: oleg232-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg232-client.virtnet: -o user_xattr,flock oleg232-server@tcp:/lustre /mnt/lustre osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=35 umount lustre on /mnt/lustre..... Stopping client oleg232-client.virtnet /mnt/lustre (opts:) stop mds service on oleg232-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg232-server 000000 34 000008 files compared the same umount lustre on /mnt/lustre..... stop ost1 service on oleg232-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg232-server stop mds service on oleg232-server unloading modules on: 'oleg232-server' oleg232-server: oleg232-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg232-client: oleg232-client: ssh exited with exit code 2 pdsh@oleg232-client: oleg232-server: ssh exited with exit code 2 pdsh@oleg232-client: oleg232-client: ssh exited with exit code 2 pdsh@oleg232-client: oleg232-server: ssh exited with exit code 2 PASS 38 (79s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg232-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg232-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg232-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg232-client: oleg232-server: ssh exited with exit code 2 oleg232-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg232-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 23:44:05 (1709786645) Stopping clients: oleg232-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg232-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg232-client: oleg232-server: ssh exited with exit code 2 oleg232-server: oleg232-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg232-server' oleg232-server: oleg232-server.virtnet: executing load_modules_local oleg232-server: Loading modules from /home/green/git/lustre-release/lustre oleg232-server: detected 4 online CPUs by sysfs oleg232-server: Force libcfs to create 2 CPU partitions oleg232-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg232-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg232-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg232-server: oleg232-server.virtnet: executing set_default_debug -1 all pdsh@oleg232-client: oleg232-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg232-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg232-server: oleg232-server.virtnet: executing set_default_debug -1 all pdsh@oleg232-client: oleg232-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg232-client.virtnet: -o user_xattr,flock oleg232-server@tcp:/lustre /mnt/lustre setup single mount lustre success fail_loc=0x142 start ost2 service on oleg232-server Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg232-server: oleg232-server.virtnet: executing set_default_debug -1 all pdsh@oleg232-client: oleg232-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost2_flakey Started lustre-OST0001 oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 4 sec stop ost2 service on oleg232-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg232-server umount lustre on /mnt/lustre..... Stopping client oleg232-client.virtnet /mnt/lustre (opts:-f) stop ost1 service on oleg232-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg232-server stop mds service on oleg232-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg232-server unloading modules on: 'oleg232-server' oleg232-server: oleg232-server.virtnet: executing unload_modules_local modules unloaded. oleg232-server: tunefs.lustre: Unable to mount /dev/mapper/mds1_flakey: No such device oleg232-server: Is the ldiskfs module available? oleg232-server: oleg232-server: tunefs.lustre FATAL: failed to write local files oleg232-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg232-client: oleg232-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x5 (MDT MGS ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity Permanent disk data: Target: lustre=MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x105 (MDT MGS writeconf ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity oleg232-server: tunefs.lustre: Unable to mount /dev/mapper/ost1_flakey: No such device oleg232-server: Is the ldiskfs module available? oleg232-server: oleg232-server: tunefs.lustre FATAL: failed to write local files oleg232-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg232-client: oleg232-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x2 (OST ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.202.132@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x102 (OST writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.202.132@tcp sys.timeout=20 oleg232-server: tunefs.lustre: Unable to mount /dev/mapper/ost2_flakey: No such device oleg232-server: Is the ldiskfs module available? oleg232-server: oleg232-server: tunefs.lustre FATAL: failed to write local files oleg232-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg232-client: oleg232-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x2 (OST ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.202.132@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x102 (OST writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.202.132@tcp sys.timeout=20 tunefs failed, reformatting instead Stopping clients: oleg232-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg232-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg232-client: oleg232-server: ssh exited with exit code 2 oleg232-server: oleg232-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg232-server' oleg232-server: oleg232-server.virtnet: executing load_modules_local oleg232-server: Loading modules from /home/green/git/lustre-release/lustre oleg232-server: detected 4 online CPUs by sysfs oleg232-server: Force libcfs to create 2 CPU partitions oleg232-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg232-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg232-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg232-server: oleg232-server.virtnet: executing set_default_debug -1 all pdsh@oleg232-client: oleg232-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg232-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg232-server: oleg232-server.virtnet: executing set_default_debug -1 all pdsh@oleg232-client: oleg232-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg232-server: oleg232-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg232-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec stop ost1 service on oleg232-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg232-server stop mds service on oleg232-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg232-server PASS 51 (161s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:46:50 (1709786810) start mds service on oleg232-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg232-server: oleg232-server.virtnet: executing set_default_debug -1 all pdsh@oleg232-client: oleg232-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg232-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg232-server: oleg232-server.virtnet: executing set_default_debug -1 all pdsh@oleg232-client: oleg232-server: ssh exited with exit code 1 Started lustre-OST0000 oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg232-client.virtnet: -o user_xattr,flock oleg232-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 193 (time 1709786842.53 total 10.29 last 18.75) - open/close 443 (time 1709786853.25 total 21.01 last 23.33) - open/close 698 (time 1709786863.26 total 31.02 last 25.47) - open/close 1033 (time 1709786873.75 total 41.51 last 31.94) - open/close 1236 (time 1709786884.32 total 52.08 last 19.20) - open/close 1448 (time 1709786894.92 total 62.68 last 20.01) - open/close 1683 (time 1709786905.71 total 73.47 last 21.77) - open/close 1981 (time 1709786916.09 total 83.85 last 28.73) - open/close 2169 (time 1709786926.84 total 94.60 last 17.48) - open/close 2373 (time 1709786939.76 total 107.52 last 15.80) - open/close 2597 (time 1709786949.77 total 117.53 last 22.38) - open/close 2746 (time 1709786960.19 total 127.95 last 14.30) - open/close 3041 (time 1709786970.21 total 137.97 last 29.45) - open/close 3260 (time 1709786983.17 total 150.93 last 16.89) - open/close 3473 (time 1709786994.09 total 161.85 last 19.51) - open/close 3730 (time 1709787005.03 total 172.79 last 23.50) - open/close 3933 (time 1709787015.50 total 183.26 last 19.38) - open/close 4159 (time 1709787026.16 total 193.92 last 21.21) - open/close 4338 (time 1709787037.16 total 204.92 last 16.26) - open/close 4580 (time 1709787047.94 total 215.70 last 22.45) - open/close 4756 (time 1709787058.58 total 226.34 last 16.54) - open/close 4939 (time 1709787068.59 total 236.35 last 18.29) - open/close 5109 (time 1709787078.59 total 246.35 last 16.99) - open/close 5270 (time 1709787088.62 total 256.38 last 16.06) - open/close 5400 (time 1709787100.89 total 268.65 last 10.60) - open/close 5576 (time 1709787111.89 total 279.65 last 16.01) - open/close 5776 (time 1709787122.54 total 290.30 last 18.77) - open/close 5943 (time 1709787132.69 total 300.45 last 16.46) - open/close 6093 (time 1709787142.69 total 310.45 last 15.00) - open/close 6284 (time 1709787153.40 total 321.16 last 17.82) - open/close 6426 (time 1709787163.41 total 331.17 last 14.20) - open/close 6474 (time 1709787175.35 total 343.11 last 4.02) - open/close 6611 (time 1709787195.04 total 362.80 last 6.96) - open/close 6770 (time 1709787205.21 total 372.97 last 15.62) - open/close 6974 (time 1709787217.80 total 385.56 last 16.21) - open/close 7222 (time 1709787228.40 total 396.17 last 23.38) - open/close 7426 (time 1709787238.73 total 406.49 last 19.77) - open/close 7585 (time 1709787248.74 total 416.50 last 15.87) - open/close 7709 (time 1709787258.76 total 426.52 last 12.38) - open/close 7877 (time 1709787268.98 total 436.74 last 16.43) - open/close 8025 (time 1709787279.53 total 447.29 last 14.03) - open/close 8174 (time 1709787289.55 total 457.31 last 14.87) - open/close 8371 (time 1709787299.97 total 467.73 last 18.89) - open/close 8523 (time 1709787310.64 total 478.40 last 14.25) - open/close 8742 (time 1709787321.49 total 489.25 last 20.18) - open/close 8900 (time 1709787331.72 total 499.48 last 15.45) - open/close 9060 (time 1709787341.88 total 509.64 last 15.75) - open/close 9203 (time 1709787351.91 total 519.67 last 14.25) - open/close 9388 (time 1709787361.92 total 529.68 last 18.50) - open/close 9529 (time 1709787372.05 total 539.81 last 13.91) - open/close 9709 (time 1709787382.08 total 549.84 last 17.95) - open/close 10000 (time 1709787391.38 total 559.14 last 31.30) - open/close 10397 (time 1709787401.43 total 569.19 last 39.50) - open/close 10803 (time 1709787411.43 total 579.19 last 40.59) - open/close 11238 (time 1709787421.43 total 589.19 last 43.48) - open/close 11692 (time 1709787431.45 total 599.21 last 45.32) - open/close 11861 (time 1709787446.53 total 614.29 last 11.21) - open/close 12258 (time 1709787456.56 total 624.32 last 39.58) - open/close 12662 (time 1709787466.56 total 634.32 last 40.38) - open/close 12987 (time 1709787486.57 total 654.33 last 16.25) - open/close 13420 (time 1709787496.61 total 664.37 last 43.11) - open/close 13841 (time 1709787506.61 total 674.37 last 42.09) - open/close 14204 (time 1709787516.64 total 684.40 last 36.20) - open/close 14564 (time 1709787526.65 total 694.41 last 35.98) - open/close 14918 (time 1709787536.65 total 704.41 last 35.39) - open/close 15226 (time 1709787546.68 total 714.44 last 30.71) - open/close 15535 (time 1709787556.71 total 724.47 last 30.80) - open/close 15882 (time 1709787566.72 total 734.48 last 34.69) - open/close 16248 (time 1709787576.72 total 744.48 last 36.59) - open/close 16574 (time 1709787586.76 total 754.52 last 32.48) - open/close 16914 (time 1709787596.76 total 764.52 last 33.98) - open/close 17245 (time 1709787606.77 total 774.53 last 33.07) - open/close 17560 (time 1709787616.79 total 784.55 last 31.44) - open/close 17894 (time 1709787626.80 total 794.56 last 33.37) - open/close 18254 (time 1709787636.81 total 804.57 last 35.98) - open/close 18604 (time 1709787646.82 total 814.58 last 34.95) - open/close 18900 (time 1709787656.84 total 824.60 last 29.55) - open/close 19232 (time 1709787666.87 total 834.63 last 33.11) - open/close 19528 (time 1709787676.88 total 844.64 last 29.56) - open/close 19846 (time 1709787686.89 total 854.65 last 31.77) - open/close 20000 (time 1709787692.55 total 860.31 last 27.19) - open/close 20314 (time 1709787702.58 total 870.34 last 31.32) - open/close 20625 (time 1709787712.59 total 880.35 last 31.08) - open/close 20889 (time 1709787722.65 total 890.41 last 26.23) - open/close 21177 (time 1709787732.66 total 900.42 last 28.79) - open/close 21466 (time 1709787742.66 total 910.42 last 28.88) - open/close 21747 (time 1709787752.67 total 920.43 last 28.07) - open/close 22019 (time 1709787762.72 total 930.48 last 27.08) - open/close 22290 (time 1709787772.75 total 940.51 last 27.03) - open/close 22588 (time 1709787782.76 total 950.52 last 29.75) - open/close 22880 (time 1709787792.78 total 960.54 last 29.15) - open/close 23185 (time 1709787802.79 total 970.55 last 30.49) - open/close 23194 (time 1709787814.13 total 981.89 last 0.79) - open/close 23455 (time 1709787824.17 total 991.93 last 25.99) - open/close 23715 (time 1709787843.72 total 1011.48 last 13.30) - open/close 23958 (time 1709787853.74 total 1021.50 last 24.24) - open/close 24204 (time 1709787863.80 total 1031.56 last 24.47) - open/close 24461 (time 1709787873.81 total 1041.57 last 25.67) - open/close 24723 (time 1709787883.82 total 1051.58 last 26.16) - open/close 24969 (time 1709787893.86 total 1061.62 last 24.51) - open/close 25196 (time 1709787903.94 total 1071.70 last 22.52) - open/close 25463 (time 1709787913.98 total 1081.74 last 26.60) - open/close 25708 (time 1709787924.02 total 1091.78 last 24.39) - open/close 25957 (time 1709787934.03 total 1101.79 last 24.87) - open/close 26228 (time 1709787944.05 total 1111.81 last 27.06) - open/close 26538 (time 1709787954.07 total 1121.83 last 30.93) - open/close 26872 (time 1709787964.08 total 1131.84 last 33.38) - open/close 27175 (time 1709787974.08 total 1141.84 last 30.29) - open/close 27483 (time 1709787984.10 total 1151.86 last 30.74) - open/close 27804 (time 1709787994.12 total 1161.88 last 32.02) - open/close 28163 (time 1709788004.13 total 1171.89 last 35.85) - open/close 28537 (time 1709788014.14 total 1181.90 last 37.40) - open/close 28934 (time 1709788024.15 total 1191.91 last 39.65) - open/close 29327 (time 1709788034.17 total 1201.93 last 39.22) - open/close 29739 (time 1709788044.17 total 1211.93 last 41.18) - open/close 30000 (time 1709788050.75 total 1218.51 last 39.68) - open/close 30394 (time 1709788060.75 total 1228.51 last 39.39) - open/close 30762 (time 1709788070.75 total 1238.51 last 36.80) - open/close 31141 (time 1709788080.76 total 1248.52 last 37.89) - open/close 31531 (time 1709788090.77 total 1258.53 last 38.96) - open/close 31893 (time 1709788100.79 total 1268.55 last 36.12) - open/close 32260 (time 1709788110.80 total 1278.56 last 36.64) - open/close 32627 (time 1709788120.81 total 1288.57 last 36.70) - open/close 32988 (time 1709788130.81 total 1298.57 last 36.07) - open/close 33348 (time 1709788140.82 total 1308.58 last 35.99) - open/close 33703 (time 1709788150.83 total 1318.59 last 35.45) - open/close 34060 (time 1709788160.84 total 1328.60 last 35.66) - open/close 34429 (time 1709788170.86 total 1338.62 last 36.83) - open/close 34790 (time 1709788180.88 total 1348.64 last 36.04) - open/close 35149 (time 1709788190.90 total 1358.66 last 35.83) - open/close 35502 (time 1709788200.92 total 1368.68 last 35.23) - open/close 35853 (time 1709788210.92 total 1378.68 last 35.08) - open/close 36209 (time 1709788220.93 total 1388.69 last 35.59) - open/close 36558 (time 1709788230.94 total 1398.70 last 34.84) - open/close 36895 (time 1709788240.95 total 1408.71 last 33.68) - open/close 37238 (time 1709788250.96 total 1418.72 last 34.25) - open/close 37583 (time 1709788260.99 total 1428.75 last 34.42) - open/close 37907 (time 1709788271.01 total 1438.77 last 32.32) - open/close 38230 (time 1709788281.02 total 1448.78 last 32.28) - open/close 38533 (time 1709788291.04 total 1458.80 last 30.22) - open/close 38848 (time 1709788301.06 total 1468.82 last 31.46) - open/close 39157 (time 1709788311.06 total 1478.82 last 30.90) - open/close 39468 (time 1709788321.09 total 1488.85 last 31.01) - open/close 39773 (time 1709788331.12 total 1498.88 last 30.41) - open/close 40000 (time 1709788338.96 total 1506.72 last 28.96) - open/close 40296 (time 1709788348.97 total 1516.73 last 29.57) - open/close 40590 (time 1709788358.98 total 1526.74 last 29.38) - open/close 40882 (time 1709788368.98 total 1536.74 last 29.19) - open/close 41168 (time 1709788378.99 total 1546.75 last 28.58) - open/close 41446 (time 1709788389.00 total 1556.76 last 27.77) - open/close 41736 (time 1709788399.02 total 1566.78 last 28.92) - open/close 42039 (time 1709788409.05 total 1576.81 last 30.21) - open/close 42327 (time 1709788419.05 total 1586.81 last 28.80) - open/close 42623 (time 1709788429.05 total 1596.82 last 29.59) - open/close 42926 (time 1709788439.07 total 1606.83 last 30.24) - open/close 43212 (time 1709788449.11 total 1616.87 last 28.50) - open/close 43504 (time 1709788459.12 total 1626.88 last 29.18) - open/close 43786 (time 1709788469.13 total 1636.89 last 28.16) - open/close 44083 (time 1709788479.14 total 1646.90 last 29.66) - open/close 44375 (time 1709788489.16 total 1656.92 last 29.15) - open/close 44670 (time 1709788499.18 total 1666.94 last 29.44) - open/close 44959 (time 1709788509.19 total 1676.95 last 28.87) - open/close 45254 (time 1709788519.22 total 1686.98 last 29.41) - open/close 45542 (time 1709788529.25 total 1697.01 last 28.72) - open/close 45828 (time 1709788539.26 total 1707.02 last 28.56) - open/close 46108 (time 1709788549.28 total 1717.04 last 27.94) - open/close 46398 (time 1709788559.29 total 1727.05 last 28.97) - open/close 46682 (time 1709788569.31 total 1737.07 last 28.35) - open/close 46971 (time 1709788579.32 total 1747.08 last 28.86) - open/close 47263 (time 1709788589.33 total 1757.09 last 29.17) - open/close 47557 (time 1709788599.36 total 1767.12 last 29.32) - open/close 47848 (time 1709788609.38 total 1777.14 last 29.05) - open/close 48141 (time 1709788619.38 total 1787.14 last 29.30) - open/close 48421 (time 1709788629.41 total 1797.17 last 27.91) - open/close 48690 (time 1709788639.42 total 1807.18 last 26.86) - open/close 48955 (time 1709788649.46 total 1817.22 last 26.41) - open/close 49227 (time 1709788659.48 total 1827.24 last 27.14) - open/close 49496 (time 1709788669.50 total 1837.26 last 26.85) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49698) error: No space left on device total: 49698 open/close in 1844.66 seconds: 26.94 ops/second - unlinked 0 (time 1709788678 ; total 1 ; last 1) - unlinked 10000 (time 1709789122 ; total 445 ; last 444) - unlinked 20000 (time 1709789510 ; total 833 ; last 388) - unlinked 30000 (time 1709789777 ; total 1100 ; last 267) - unlinked 40000 (time 1709789912 ; total 1235 ; last 135) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49698) error: No such file or directory total: 49698 unlinks in 1279 seconds: 38.856918 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg232-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg232-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg232-server stop mds service on oleg232-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg232-server unloading modules on: 'oleg232-server' oleg232-server: oleg232-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg232-client: oleg232-client: ssh exited with exit code 2 pdsh@oleg232-client: oleg232-server: ssh exited with exit code 2 pdsh@oleg232-client: oleg232-client: ssh exited with exit code 2 pdsh@oleg232-client: oleg232-server: ssh exited with exit code 2 PASS 101a (3158s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg232-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg232-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg232-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg232-client: oleg232-server: ssh exited with exit code 2 oleg232-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg232-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == conf-sanity test 101b: Race events DISCONNECT and ACTIVE in osp ========================================================== 00:39:30 (1709789970) start mds service on oleg232-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg232-server' oleg232-server: oleg232-server.virtnet: executing load_modules_local oleg232-server: Loading modules from /home/green/git/lustre-release/lustre oleg232-server: detected 4 online CPUs by sysfs oleg232-server: Force libcfs to create 2 CPU partitions oleg232-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg232-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg232-server: oleg232-server.virtnet: executing set_default_debug -1 all pdsh@oleg232-client: oleg232-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg232-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg232-server: oleg232-server.virtnet: executing set_default_debug -1 all pdsh@oleg232-client: oleg232-server: ssh exited with exit code 1 Started lustre-OST0000 oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg232-client.virtnet: -o user_xattr,flock oleg232-server@tcp:/lustre /mnt/lustre fail_loc=0x80002107 fail_val=20 stop ost1 service on oleg232-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg232-server start ost1 service on oleg232-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg232-server: oleg232-server.virtnet: executing set_default_debug -1 all pdsh@oleg232-client: oleg232-server: ssh exited with exit code 1 Started lustre-OST0000 oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 7 sec oleg232-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff88012fef8000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff88012fef8000.ost_server_uuid in FULL state after 0 sec umount lustre on /mnt/lustre..... Stopping client oleg232-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg232-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg232-server stop mds service on oleg232-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg232-server unloading modules on: 'oleg232-server' oleg232-server: oleg232-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg232-client: oleg232-client: ssh exited with exit code 2 pdsh@oleg232-client: oleg232-server: ssh exited with exit code 2 pdsh@oleg232-client: oleg232-client: ssh exited with exit code 2 pdsh@oleg232-client: oleg232-server: ssh exited with exit code 2 PASS 101b (85s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg232-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg232-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory Stopping clients: oleg232-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg232-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg232-client: oleg232-server: ssh exited with exit code 2 oleg232-server: oleg232-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg232-server' oleg232-server: oleg232-server.virtnet: executing load_modules_local oleg232-server: Loading modules from /home/green/git/lustre-release/lustre oleg232-server: detected 4 online CPUs by sysfs oleg232-server: Force libcfs to create 2 CPU partitions oleg232-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg232-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey == conf-sanity test complete, duration 3699 sec ========== 00:41:13 (1709790073) === conf-sanity: start cleanup 00:41:13 (1709790073) === === conf-sanity: finish cleanup 00:41:13 (1709790073) ===