-----============= acceptance-small: conf-sanity ============----- Wed Mar 6 23:40:12 EST 2024 excepting tests: 32newtarball 110 skipping tests SLOW=no: 45 69 106 111 114 Stopping clients: oleg457-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg457-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg457-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg457-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg457-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg457-server oleg457-server: oleg457-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg457-server' oleg457-server: oleg457-server.virtnet: executing load_modules_local oleg457-server: Loading modules from /home/green/git/lustre-release/lustre oleg457-server: detected 4 online CPUs by sysfs oleg457-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg457-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg457-server: oleg457-server.virtnet: executing set_default_debug -1 all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg457-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg457-server: oleg457-server.virtnet: executing set_default_debug -1 all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg457-server: oleg457-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg457-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg457-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg457-server stop mds service on oleg457-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg457-server debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 35b: Continue reconnection retries, if the active server is busy ========================================================== 23:41:14 (1709786474) start mds service on oleg457-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg457-server: oleg457-server.virtnet: executing set_default_debug -1 all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg457-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg457-server: oleg457-server.virtnet: executing set_default_debug -1 all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-OST0000 oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg457-client.virtnet: -o user_xattr,flock oleg457-server@tcp:/lustre /mnt/lustre debug=ha conf-sanity.sh test_35b 2024-03-0623h41m28s Set up a fake failnode for the MDS at_max=0 at_max=0 Injecting EBUSY on MDS fail_loc=0x80000136 mdc.lustre-MDT0000-mdc-ffff880130333800.stats=clear Creating a test file and stat it File: '/mnt/lustre/d35b.conf-sanity/f35b.conf-sanity' Size: 0 Blocks: 0 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115238826934274 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-03-06 23:42:11.000000000 -0500 Modify: 2024-03-06 23:42:11.000000000 -0500 Change: 2024-03-06 23:42:11.000000000 -0500 Birth: - Stop injecting EBUSY on MDS fail_loc=0 done at_max=600 at_max=600 Debug log: 54 lines, 54 kept, 0 dropped, 0 bad. umount lustre on /mnt/lustre..... Stopping client oleg457-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg457-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg457-server stop mds service on oleg457-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg457-server unloading modules on: 'oleg457-server' oleg457-server: oleg457-server.virtnet: executing unload_modules_local modules unloaded. oleg457-server: tunefs.lustre: Unable to mount /dev/mapper/mds1_flakey: No such device oleg457-server: Is the ldiskfs module available? oleg457-server: oleg457-server: tunefs.lustre FATAL: failed to write local files oleg457-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg457-client: oleg457-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x5 (MDT MGS ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity Permanent disk data: Target: lustre=MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x105 (MDT MGS writeconf ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity oleg457-server: tunefs.lustre: Unable to mount /dev/mapper/ost1_flakey: No such device oleg457-server: Is the ldiskfs module available? oleg457-server: oleg457-server: tunefs.lustre FATAL: failed to write local files oleg457-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg457-client: oleg457-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x2 (OST ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.204.157@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x102 (OST writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.204.157@tcp sys.timeout=20 oleg457-server: tunefs.lustre: Unable to mount /dev/mapper/ost2_flakey: No such device oleg457-server: Is the ldiskfs module available? oleg457-server: oleg457-server: tunefs.lustre FATAL: failed to write local files oleg457-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg457-client: oleg457-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.204.157@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x162 (OST first_time update writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.204.157@tcp sys.timeout=20 tunefs failed, reformatting instead Stopping clients: oleg457-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg457-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg457-client: oleg457-server: ssh exited with exit code 2 oleg457-server: oleg457-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg457-server' oleg457-server: oleg457-server.virtnet: executing load_modules_local oleg457-server: Loading modules from /home/green/git/lustre-release/lustre oleg457-server: detected 4 online CPUs by sysfs oleg457-server: Force libcfs to create 2 CPU partitions oleg457-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg457-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg457-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg457-server: oleg457-server.virtnet: executing set_default_debug -1 all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg457-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg457-server: oleg457-server.virtnet: executing set_default_debug -1 all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg457-server: oleg457-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg457-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg457-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg457-server stop mds service on oleg457-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg457-server PASS 35b (117s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 23:43:13 (1709786593) start mds service on oleg457-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg457-server: oleg457-server.virtnet: executing set_default_debug -1 all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg457-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg457-server: oleg457-server.virtnet: executing set_default_debug -1 all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-OST0000 oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg457-client.virtnet: -o user_xattr,flock oleg457-server@tcp:/lustre /mnt/lustre copying 10 files to /mnt/lustre/d38.conf-sanity tar: Removing leading `/' from member names umount lustre on /mnt/lustre..... Stopping client oleg457-client.virtnet /mnt/lustre (opts:) osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=44 stop mds service on oleg457-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg457-server delete lov_objid file on MDS 000000 43 000008 start mds service on oleg457-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg457-server: oleg457-server.virtnet: executing set_default_debug -1 all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg457-client.virtnet: -o user_xattr,flock oleg457-server@tcp:/lustre /mnt/lustre osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=2 umount lustre on /mnt/lustre..... Stopping client oleg457-client.virtnet /mnt/lustre (opts:) stop mds service on oleg457-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg457-server 000000 2 000008 oleg457-server: 8+0 records in oleg457-server: 8+0 records out oleg457-server: 4096 bytes (4.1 kB) copied, 0.00043465 s, 9.4 MB/s start mds service on oleg457-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg457-server: oleg457-server.virtnet: executing set_default_debug -1 all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg457-client.virtnet: -o user_xattr,flock oleg457-server@tcp:/lustre /mnt/lustre osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=35 umount lustre on /mnt/lustre..... Stopping client oleg457-client.virtnet /mnt/lustre (opts:) stop mds service on oleg457-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg457-server 000000 34 000008 files compared the same umount lustre on /mnt/lustre..... stop ost1 service on oleg457-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg457-server stop mds service on oleg457-server unloading modules on: 'oleg457-server' oleg457-server: oleg457-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg457-client: oleg457-server: ssh exited with exit code 2 pdsh@oleg457-client: oleg457-client: ssh exited with exit code 2 pdsh@oleg457-client: oleg457-client: ssh exited with exit code 2 pdsh@oleg457-client: oleg457-server: ssh exited with exit code 2 PASS 38 (76s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg457-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg457-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg457-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg457-client: oleg457-server: ssh exited with exit code 2 oleg457-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg457-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 23:44:32 (1709786672) Stopping clients: oleg457-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg457-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg457-client: oleg457-server: ssh exited with exit code 2 oleg457-server: oleg457-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg457-server' oleg457-server: oleg457-server.virtnet: executing load_modules_local oleg457-server: Loading modules from /home/green/git/lustre-release/lustre oleg457-server: detected 4 online CPUs by sysfs oleg457-server: Force libcfs to create 2 CPU partitions oleg457-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg457-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg457-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg457-server: oleg457-server.virtnet: executing set_default_debug -1 all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg457-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg457-server: oleg457-server.virtnet: executing set_default_debug -1 all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg457-client.virtnet: -o user_xattr,flock oleg457-server@tcp:/lustre /mnt/lustre setup single mount lustre success fail_loc=0x142 start ost2 service on oleg457-server Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg457-server: oleg457-server.virtnet: executing set_default_debug -1 all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost2_flakey Started lustre-OST0001 oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 2 sec stop ost2 service on oleg457-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg457-server umount lustre on /mnt/lustre..... Stopping client oleg457-client.virtnet /mnt/lustre (opts:-f) stop ost1 service on oleg457-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg457-server stop mds service on oleg457-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg457-server unloading modules on: 'oleg457-server' oleg457-server: oleg457-server.virtnet: executing unload_modules_local modules unloaded. oleg457-server: tunefs.lustre: Unable to mount /dev/mapper/mds1_flakey: No such device oleg457-server: Is the ldiskfs module available? oleg457-server: oleg457-server: tunefs.lustre FATAL: failed to write local files oleg457-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg457-client: oleg457-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x5 (MDT MGS ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity Permanent disk data: Target: lustre=MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x105 (MDT MGS writeconf ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity oleg457-server: tunefs.lustre: Unable to mount /dev/mapper/ost1_flakey: No such device oleg457-server: Is the ldiskfs module available? oleg457-server: oleg457-server: tunefs.lustre FATAL: failed to write local files oleg457-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg457-client: oleg457-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x2 (OST ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.204.157@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x102 (OST writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.204.157@tcp sys.timeout=20 oleg457-server: tunefs.lustre: Unable to mount /dev/mapper/ost2_flakey: No such device oleg457-server: Is the ldiskfs module available? oleg457-server: oleg457-server: tunefs.lustre FATAL: failed to write local files oleg457-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg457-client: oleg457-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x2 (OST ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.204.157@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x102 (OST writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.204.157@tcp sys.timeout=20 tunefs failed, reformatting instead Stopping clients: oleg457-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg457-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg457-client: oleg457-server: ssh exited with exit code 2 oleg457-server: oleg457-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg457-server' oleg457-server: oleg457-server.virtnet: executing load_modules_local oleg457-server: Loading modules from /home/green/git/lustre-release/lustre oleg457-server: detected 4 online CPUs by sysfs oleg457-server: Force libcfs to create 2 CPU partitions oleg457-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg457-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg457-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg457-server: oleg457-server.virtnet: executing set_default_debug -1 all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg457-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg457-server: oleg457-server.virtnet: executing set_default_debug -1 all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg457-server: oleg457-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg457-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg457-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg457-server stop mds service on oleg457-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg457-server PASS 51 (162s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:47:16 (1709786836) start mds service on oleg457-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg457-server: oleg457-server.virtnet: executing set_default_debug -1 all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg457-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg457-server: oleg457-server.virtnet: executing set_default_debug -1 all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-OST0000 oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg457-client.virtnet: -o user_xattr,flock oleg457-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 329 (time 1709786865.22 total 10.65 last 30.90) - open/close 586 (time 1709786875.94 total 21.37 last 23.97) - open/close 836 (time 1709786886.45 total 31.88 last 23.79) - open/close 1057 (time 1709786896.45 total 41.88 last 22.08) - open/close 1345 (time 1709786906.87 total 52.30 last 27.65) - open/close 1557 (time 1709786917.26 total 62.69 last 20.41) - open/close 1735 (time 1709786927.27 total 72.70 last 17.79) - open/close 1912 (time 1709786937.57 total 83.00 last 17.19) - open/close 2125 (time 1709786947.78 total 93.21 last 20.85) - open/close 2380 (time 1709786957.79 total 103.22 last 25.49) - open/close 2575 (time 1709786975.78 total 121.21 last 10.84) - open/close 2795 (time 1709786985.79 total 131.22 last 21.99) - open/close 3012 (time 1709786996.72 total 142.15 last 19.85) - open/close 3321 (time 1709787006.78 total 152.21 last 30.72) - open/close 3524 (time 1709787017.66 total 163.09 last 18.66) - open/close 3821 (time 1709787027.77 total 173.20 last 29.37) - open/close 4192 (time 1709787037.77 total 183.20 last 37.10) - open/close 4397 (time 1709787047.80 total 193.23 last 20.45) - open/close 4566 (time 1709787058.24 total 203.67 last 16.18) - open/close 4871 (time 1709787068.27 total 213.70 last 30.43) - open/close 5015 (time 1709787078.45 total 223.88 last 14.15) - open/close 5243 (time 1709787088.55 total 233.98 last 22.57) - open/close 5584 (time 1709787098.55 total 243.98 last 34.10) - open/close 5793 (time 1709787108.56 total 253.99 last 20.88) - open/close 5909 (time 1709787119.24 total 264.67 last 10.86) - open/close 6133 (time 1709787129.34 total 274.77 last 22.19) - open/close 6305 (time 1709787139.67 total 285.10 last 16.64) - open/close 6461 (time 1709787149.93 total 295.36 last 15.20) - open/close 6646 (time 1709787160.19 total 305.62 last 18.04) - open/close 6833 (time 1709787171.10 total 316.53 last 17.15) - open/close 7197 (time 1709787181.11 total 326.54 last 36.35) - open/close 7414 (time 1709787193.48 total 338.91 last 17.54) - open/close 7562 (time 1709787204.24 total 349.67 last 13.75) - open/close 7699 (time 1709787215.11 total 360.54 last 12.60) - open/close 7808 (time 1709787226.05 total 371.48 last 9.96) - open/close 7957 (time 1709787236.65 total 382.08 last 14.06) - open/close 8108 (time 1709787246.66 total 392.09 last 15.10) - open/close 8235 (time 1709787257.38 total 402.81 last 11.85) - open/close 8452 (time 1709787267.39 total 412.82 last 21.68) - open/close 8632 (time 1709787277.60 total 423.03 last 17.62) - open/close 8768 (time 1709787287.85 total 433.28 last 13.27) - open/close 8900 (time 1709787297.87 total 443.30 last 13.16) - open/close 9099 (time 1709787307.88 total 453.31 last 19.89) - open/close 9376 (time 1709787318.50 total 463.93 last 26.09) - open/close 9577 (time 1709787329.15 total 474.58 last 18.87) - open/close 9815 (time 1709787339.65 total 485.08 last 22.66) - open/close 10000 (time 1709787348.16 total 493.59 last 21.76) - open/close 10596 (time 1709787358.16 total 503.59 last 59.59) - open/close 11120 (time 1709787378.32 total 523.75 last 25.98) - open/close 11670 (time 1709787388.32 total 533.75 last 55.00) - open/close 12199 (time 1709787398.33 total 543.76 last 52.87) - open/close 12727 (time 1709787408.34 total 553.77 last 52.74) - open/close 13213 (time 1709787418.34 total 563.77 last 48.59) - open/close 13662 (time 1709787428.34 total 573.77 last 44.89) - open/close 14128 (time 1709787438.37 total 583.80 last 46.47) - open/close 14614 (time 1709787448.39 total 593.82 last 48.49) - open/close 15067 (time 1709787458.41 total 603.84 last 45.23) - open/close 15536 (time 1709787468.44 total 613.87 last 46.78) - open/close 15966 (time 1709787478.44 total 623.87 last 42.97) - open/close 16375 (time 1709787488.45 total 633.88 last 40.85) - open/close 16811 (time 1709787498.47 total 643.90 last 43.55) - open/close 17211 (time 1709787508.47 total 653.90 last 39.99) - open/close 17599 (time 1709787518.49 total 663.92 last 38.70) - open/close 18019 (time 1709787528.50 total 673.93 last 41.97) - open/close 18422 (time 1709787538.51 total 683.94 last 40.28) - open/close 18851 (time 1709787548.51 total 693.94 last 42.88) - open/close 19289 (time 1709787558.52 total 703.95 last 43.75) - open/close 19751 (time 1709787568.57 total 714.00 last 45.97) - open/close 20000 (time 1709787574.89 total 720.32 last 39.40) - open/close 20449 (time 1709787584.90 total 730.33 last 44.85) - open/close 20914 (time 1709787594.92 total 740.35 last 46.43) - open/close 20919 (time 1709787606.11 total 751.54 last 0.45) - open/close 21351 (time 1709787616.12 total 761.55 last 43.14) - open/close 21793 (time 1709787626.14 total 771.57 last 44.10) - open/close 22208 (time 1709787636.16 total 781.59 last 41.44) - open/close 22632 (time 1709787646.16 total 791.59 last 42.37) - open/close 23085 (time 1709787656.17 total 801.60 last 45.26) - open/close 23523 (time 1709787666.18 total 811.61 last 43.76) - open/close 23908 (time 1709787676.19 total 821.62 last 38.49) - open/close 24284 (time 1709787686.21 total 831.63 last 37.53) - open/close 24701 (time 1709787696.30 total 841.73 last 41.32) - open/close 25075 (time 1709787706.30 total 851.73 last 37.38) - open/close 25404 (time 1709787716.34 total 861.77 last 32.77) - open/close 25754 (time 1709787726.35 total 871.78 last 34.96) - open/close 26132 (time 1709787736.36 total 881.79 last 37.77) - open/close 26538 (time 1709787746.36 total 891.79 last 40.58) - open/close 26906 (time 1709787756.39 total 901.82 last 36.70) - open/close 27088 (time 1709787772.94 total 918.37 last 11.00) - open/close 27355 (time 1709787782.94 total 928.37 last 26.70) - open/close 27649 (time 1709787793.02 total 938.45 last 29.17) - open/close 27952 (time 1709787813.37 total 958.80 last 14.89) - open/close 28278 (time 1709787823.38 total 968.81 last 32.58) - open/close 28613 (time 1709787833.40 total 978.83 last 33.44) - open/close 28977 (time 1709787843.41 total 988.84 last 36.37) - open/close 29336 (time 1709787853.43 total 998.86 last 35.80) - open/close 29732 (time 1709787863.45 total 1008.88 last 39.52) - open/close 30000 (time 1709787870.61 total 1016.04 last 37.46) - open/close 30387 (time 1709787880.62 total 1026.05 last 38.63) - open/close 30780 (time 1709787890.64 total 1036.07 last 39.23) - open/close 31179 (time 1709787900.67 total 1046.10 last 39.80) - open/close 31521 (time 1709787910.67 total 1056.10 last 34.19) - open/close 31879 (time 1709787920.68 total 1066.11 last 35.74) - open/close 32215 (time 1709787930.70 total 1076.13 last 33.56) - open/close 32584 (time 1709787940.71 total 1086.14 last 36.83) - open/close 32955 (time 1709787950.72 total 1096.15 last 37.10) - open/close 33367 (time 1709787960.74 total 1106.17 last 41.12) - open/close 33706 (time 1709787970.74 total 1116.17 last 33.89) - open/close 34045 (time 1709787980.80 total 1126.23 last 33.70) - open/close 34421 (time 1709787990.81 total 1136.24 last 37.55) - open/close 34781 (time 1709788000.81 total 1146.24 last 36.00) - open/close 35048 (time 1709788010.84 total 1156.27 last 26.63) - open/close 35308 (time 1709788020.85 total 1166.28 last 25.98) - open/close 35676 (time 1709788030.86 total 1176.29 last 36.77) - open/close 36042 (time 1709788040.86 total 1186.29 last 36.58) - open/close 36400 (time 1709788050.89 total 1196.32 last 35.70) - open/close 36723 (time 1709788060.90 total 1206.33 last 32.27) - open/close 37060 (time 1709788070.93 total 1216.36 last 33.61) - open/close 37399 (time 1709788080.93 total 1226.36 last 33.89) - open/close 37751 (time 1709788090.94 total 1236.37 last 35.15) - open/close 38064 (time 1709788100.95 total 1246.38 last 31.29) - open/close 38374 (time 1709788110.97 total 1256.40 last 30.92) - open/close 38706 (time 1709788120.99 total 1266.42 last 33.14) - open/close 39021 (time 1709788131.01 total 1276.44 last 31.42) - open/close 39298 (time 1709788141.05 total 1286.48 last 27.61) - open/close 39590 (time 1709788151.06 total 1296.49 last 29.16) - open/close 39827 (time 1709788161.08 total 1306.51 last 23.64) - open/close 40000 (time 1709788168.52 total 1313.95 last 23.25) - open/close 40283 (time 1709788178.55 total 1323.98 last 28.22) - open/close 40581 (time 1709788188.58 total 1334.01 last 29.72) - open/close 40832 (time 1709788198.61 total 1344.04 last 25.03) - open/close 41124 (time 1709788208.62 total 1354.05 last 29.18) - open/close 41419 (time 1709788218.64 total 1364.07 last 29.42) - open/close 41717 (time 1709788228.66 total 1374.09 last 29.76) - open/close 42004 (time 1709788238.68 total 1384.11 last 28.64) - open/close 42285 (time 1709788248.71 total 1394.14 last 28.02) - open/close 42607 (time 1709788258.71 total 1404.14 last 32.17) - open/close 42919 (time 1709788268.72 total 1414.15 last 31.17) - open/close 43198 (time 1709788278.75 total 1424.18 last 27.82) - open/close 43481 (time 1709788288.75 total 1434.18 last 28.30) - open/close 43789 (time 1709788298.76 total 1444.19 last 30.77) - open/close 44049 (time 1709788308.78 total 1454.21 last 25.97) - open/close 44308 (time 1709788318.79 total 1464.22 last 25.88) - open/close 44600 (time 1709788328.79 total 1474.22 last 29.19) - open/close 44898 (time 1709788338.81 total 1484.24 last 29.74) - open/close 45178 (time 1709788348.84 total 1494.27 last 27.92) - open/close 45452 (time 1709788358.85 total 1504.28 last 27.35) - open/close 45749 (time 1709788368.86 total 1514.29 last 29.68) - open/close 46036 (time 1709788378.87 total 1524.30 last 28.67) - open/close 46327 (time 1709788388.89 total 1534.32 last 29.05) - open/close 46625 (time 1709788398.91 total 1544.34 last 29.73) - open/close 46916 (time 1709788408.93 total 1554.36 last 29.06) - open/close 47171 (time 1709788418.95 total 1564.38 last 25.44) - open/close 47426 (time 1709788428.95 total 1574.38 last 25.50) - open/close 47678 (time 1709788438.97 total 1584.40 last 25.16) - open/close 47932 (time 1709788449.00 total 1594.43 last 25.31) - open/close 48186 (time 1709788459.02 total 1604.45 last 25.36) - open/close 48439 (time 1709788469.03 total 1614.46 last 25.27) - open/close 48700 (time 1709788479.06 total 1624.49 last 26.03) - open/close 48936 (time 1709788489.06 total 1634.49 last 23.59) - open/close 49173 (time 1709788499.07 total 1644.50 last 23.68) - open/close 49448 (time 1709788509.09 total 1654.52 last 27.44) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49698) error: No space left on device total: 49698 open/close in 1663.58 seconds: 29.87 ops/second - unlinked 0 (time 1709788519 ; total 0 ; last 0) - unlinked 10000 (time 1709788958 ; total 439 ; last 439) - unlinked 20000 (time 1709789315 ; total 796 ; last 357) - unlinked 30000 (time 1709789575 ; total 1056 ; last 260) - unlinked 40000 (time 1709789712 ; total 1193 ; last 137) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49698) error: No such file or directory total: 49698 unlinks in 1240 seconds: 40.079033 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg457-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg457-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg457-server stop mds service on oleg457-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg457-server unloading modules on: 'oleg457-server' oleg457-server: oleg457-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg457-client: oleg457-client: ssh exited with exit code 2 pdsh@oleg457-client: oleg457-server: ssh exited with exit code 2 pdsh@oleg457-client: oleg457-client: ssh exited with exit code 2 pdsh@oleg457-client: oleg457-server: ssh exited with exit code 2 PASS 101a (2939s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg457-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg457-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg457-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg457-client: oleg457-server: ssh exited with exit code 2 oleg457-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg457-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == conf-sanity test 101b: Race events DISCONNECT and ACTIVE in osp ========================================================== 00:36:17 (1709789777) start mds service on oleg457-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg457-server' oleg457-server: oleg457-server.virtnet: executing load_modules_local oleg457-server: Loading modules from /home/green/git/lustre-release/lustre oleg457-server: detected 4 online CPUs by sysfs oleg457-server: Force libcfs to create 2 CPU partitions oleg457-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg457-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg457-server: oleg457-server.virtnet: executing set_default_debug -1 all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg457-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg457-server: oleg457-server.virtnet: executing set_default_debug -1 all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-OST0000 oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg457-client.virtnet: -o user_xattr,flock oleg457-server@tcp:/lustre /mnt/lustre fail_loc=0x80002107 fail_val=20 stop ost1 service on oleg457-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg457-server start ost1 service on oleg457-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg457-server: oleg457-server.virtnet: executing set_default_debug -1 all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-OST0000 oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 5 sec oleg457-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8800accde800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800accde800.ost_server_uuid in FULL state after 0 sec umount lustre on /mnt/lustre..... Stopping client oleg457-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg457-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg457-server stop mds service on oleg457-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg457-server unloading modules on: 'oleg457-server' oleg457-server: oleg457-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg457-client: oleg457-client: ssh exited with exit code 2 pdsh@oleg457-client: oleg457-server: ssh exited with exit code 2 pdsh@oleg457-client: oleg457-client: ssh exited with exit code 2 pdsh@oleg457-client: oleg457-server: ssh exited with exit code 2 PASS 101b (89s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg457-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg457-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory Stopping clients: oleg457-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg457-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg457-client: oleg457-server: ssh exited with exit code 2 oleg457-server: oleg457-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg457-server' oleg457-server: oleg457-server.virtnet: executing load_modules_local oleg457-server: Loading modules from /home/green/git/lustre-release/lustre oleg457-server: detected 4 online CPUs by sysfs oleg457-server: Force libcfs to create 2 CPU partitions oleg457-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg457-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey == conf-sanity test complete, duration 3474 sec ========== 00:38:07 (1709789887) === conf-sanity: start cleanup 00:38:07 (1709789887) === === conf-sanity: finish cleanup 00:38:07 (1709789887) ===