-----============= acceptance-small: conf-sanity ============----- Wed Mar 6 23:10:29 EST 2024 excepting tests: 32newtarball 110 skipping tests SLOW=no: 45 69 106 111 114 Stopping clients: oleg447-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg447-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg447-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg447-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg447-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg447-server oleg447-server: oleg447-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg447-server' oleg447-server: oleg447-server.virtnet: executing load_modules_local oleg447-server: Loading modules from /home/green/git/lustre-release/lustre oleg447-server: detected 4 online CPUs by sysfs oleg447-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg447-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg447-server: oleg447-server.virtnet: executing set_default_debug -1 all pdsh@oleg447-client: oleg447-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg447-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg447-server: oleg447-server.virtnet: executing set_default_debug -1 all pdsh@oleg447-client: oleg447-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg447-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg447-server: oleg447-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg447-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg447-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg447-server stop mds service on oleg447-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg447-server debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 35b: Continue reconnection retries, if the active server is busy ========================================================== 23:11:33 (1709784693) start mds service on oleg447-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg447-server: oleg447-server.virtnet: executing set_default_debug -1 all pdsh@oleg447-client: oleg447-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg447-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg447-server: oleg447-server.virtnet: executing set_default_debug -1 all pdsh@oleg447-client: oleg447-server: ssh exited with exit code 1 Started lustre-OST0000 oleg447-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg447-client.virtnet: -o user_xattr,flock oleg447-server@tcp:/lustre /mnt/lustre debug=ha conf-sanity.sh test_35b 2024-03-0623h11m48s Set up a fake failnode for the MDS at_max=0 at_max=0 Injecting EBUSY on MDS fail_loc=0x80000136 mdc.lustre-MDT0000-mdc-ffff88012a53e000.stats=clear Creating a test file and stat it File: '/mnt/lustre/d35b.conf-sanity/f35b.conf-sanity' Size: 0 Blocks: 0 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115238826934274 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-03-06 23:12:32.000000000 -0500 Modify: 2024-03-06 23:12:32.000000000 -0500 Change: 2024-03-06 23:12:32.000000000 -0500 Birth: - Stop injecting EBUSY on MDS fail_loc=0 done at_max=600 at_max=600 Debug log: 74 lines, 74 kept, 0 dropped, 0 bad. umount lustre on /mnt/lustre..... Stopping client oleg447-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg447-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg447-server stop mds service on oleg447-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg447-server unloading modules on: 'oleg447-server' oleg447-server: oleg447-server.virtnet: executing unload_modules_local modules unloaded. oleg447-server: tunefs.lustre: Unable to mount /dev/mapper/mds1_flakey: No such device oleg447-server: Is the ldiskfs module available? oleg447-server: oleg447-server: tunefs.lustre FATAL: failed to write local files oleg447-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg447-client: oleg447-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x5 (MDT MGS ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity Permanent disk data: Target: lustre=MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x105 (MDT MGS writeconf ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity oleg447-server: tunefs.lustre: Unable to mount /dev/mapper/ost1_flakey: No such device oleg447-server: Is the ldiskfs module available? oleg447-server: oleg447-server: tunefs.lustre FATAL: failed to write local files oleg447-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg447-client: oleg447-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x2 (OST ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.204.147@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x102 (OST writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.204.147@tcp sys.timeout=20 oleg447-server: tunefs.lustre: Unable to mount /dev/mapper/ost2_flakey: No such device oleg447-server: Is the ldiskfs module available? oleg447-server: oleg447-server: tunefs.lustre FATAL: failed to write local files oleg447-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg447-client: oleg447-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.204.147@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x162 (OST first_time update writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.204.147@tcp sys.timeout=20 tunefs failed, reformatting instead Stopping clients: oleg447-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg447-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg447-client: oleg447-server: ssh exited with exit code 2 oleg447-server: oleg447-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg447-server' oleg447-server: oleg447-server.virtnet: executing load_modules_local oleg447-server: Loading modules from /home/green/git/lustre-release/lustre oleg447-server: detected 4 online CPUs by sysfs oleg447-server: Force libcfs to create 2 CPU partitions oleg447-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg447-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg447-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg447-server: oleg447-server.virtnet: executing set_default_debug -1 all pdsh@oleg447-client: oleg447-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg447-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg447-server: oleg447-server.virtnet: executing set_default_debug -1 all pdsh@oleg447-client: oleg447-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg447-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg447-server: oleg447-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg447-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec stop ost1 service on oleg447-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg447-server stop mds service on oleg447-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg447-server PASS 35b (123s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 23:13:37 (1709784817) start mds service on oleg447-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg447-server: oleg447-server.virtnet: executing set_default_debug -1 all pdsh@oleg447-client: oleg447-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg447-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg447-server: oleg447-server.virtnet: executing set_default_debug -1 all pdsh@oleg447-client: oleg447-server: ssh exited with exit code 1 Started lustre-OST0000 oleg447-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg447-client.virtnet: -o user_xattr,flock oleg447-server@tcp:/lustre /mnt/lustre copying 10 files to /mnt/lustre/d38.conf-sanity tar: Removing leading `/' from member names umount lustre on /mnt/lustre..... Stopping client oleg447-client.virtnet /mnt/lustre (opts:) osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=44 stop mds service on oleg447-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg447-server delete lov_objid file on MDS 000000 43 000008 start mds service on oleg447-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg447-server: oleg447-server.virtnet: executing set_default_debug -1 all pdsh@oleg447-client: oleg447-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg447-client.virtnet: -o user_xattr,flock oleg447-server@tcp:/lustre /mnt/lustre osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=2 umount lustre on /mnt/lustre..... Stopping client oleg447-client.virtnet /mnt/lustre (opts:) stop mds service on oleg447-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg447-server 000000 2 000008 oleg447-server: 8+0 records in oleg447-server: 8+0 records out oleg447-server: 4096 bytes (4.1 kB) copied, 0.000753865 s, 5.4 MB/s start mds service on oleg447-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg447-server: oleg447-server.virtnet: executing set_default_debug -1 all pdsh@oleg447-client: oleg447-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg447-client.virtnet: -o user_xattr,flock oleg447-server@tcp:/lustre /mnt/lustre osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=35 umount lustre on /mnt/lustre..... Stopping client oleg447-client.virtnet /mnt/lustre (opts:) stop mds service on oleg447-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg447-server 000000 34 000008 files compared the same umount lustre on /mnt/lustre..... stop ost1 service on oleg447-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg447-server stop mds service on oleg447-server unloading modules on: 'oleg447-server' oleg447-server: oleg447-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg447-client: oleg447-client: ssh exited with exit code 2 pdsh@oleg447-client: oleg447-server: ssh exited with exit code 2 pdsh@oleg447-client: oleg447-client: ssh exited with exit code 2 pdsh@oleg447-client: oleg447-server: ssh exited with exit code 2 PASS 38 (70s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg447-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg447-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg447-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg447-client: oleg447-server: ssh exited with exit code 2 oleg447-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg447-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 23:14:49 (1709784889) Stopping clients: oleg447-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg447-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg447-client: oleg447-server: ssh exited with exit code 2 oleg447-server: oleg447-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg447-server' oleg447-server: oleg447-server.virtnet: executing load_modules_local oleg447-server: Loading modules from /home/green/git/lustre-release/lustre oleg447-server: detected 4 online CPUs by sysfs oleg447-server: Force libcfs to create 2 CPU partitions oleg447-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg447-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg447-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg447-server: oleg447-server.virtnet: executing set_default_debug -1 all pdsh@oleg447-client: oleg447-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg447-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg447-server: oleg447-server.virtnet: executing set_default_debug -1 all pdsh@oleg447-client: oleg447-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg447-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg447-client.virtnet: -o user_xattr,flock oleg447-server@tcp:/lustre /mnt/lustre setup single mount lustre success fail_loc=0x142 start ost2 service on oleg447-server Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg447-server: oleg447-server.virtnet: executing set_default_debug -1 all pdsh@oleg447-client: oleg447-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost2_flakey Started lustre-OST0001 oleg447-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 1 sec stop ost2 service on oleg447-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg447-server umount lustre on /mnt/lustre..... Stopping client oleg447-client.virtnet /mnt/lustre (opts:-f) stop ost1 service on oleg447-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg447-server stop mds service on oleg447-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg447-server unloading modules on: 'oleg447-server' oleg447-server: oleg447-server.virtnet: executing unload_modules_local modules unloaded. oleg447-server: tunefs.lustre: Unable to mount /dev/mapper/mds1_flakey: No such device oleg447-server: Is the ldiskfs module available? oleg447-server: oleg447-server: tunefs.lustre FATAL: failed to write local files oleg447-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg447-client: oleg447-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x5 (MDT MGS ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity Permanent disk data: Target: lustre=MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x105 (MDT MGS writeconf ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity oleg447-server: tunefs.lustre: Unable to mount /dev/mapper/ost1_flakey: No such device oleg447-server: Is the ldiskfs module available? oleg447-server: oleg447-server: tunefs.lustre FATAL: failed to write local files oleg447-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg447-client: oleg447-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x2 (OST ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.204.147@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x102 (OST writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.204.147@tcp sys.timeout=20 oleg447-server: tunefs.lustre: Unable to mount /dev/mapper/ost2_flakey: No such device oleg447-server: Is the ldiskfs module available? oleg447-server: oleg447-server: tunefs.lustre FATAL: failed to write local files oleg447-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg447-client: oleg447-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x2 (OST ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.204.147@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x102 (OST writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.204.147@tcp sys.timeout=20 tunefs failed, reformatting instead Stopping clients: oleg447-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg447-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg447-client: oleg447-server: ssh exited with exit code 2 oleg447-server: oleg447-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg447-server' oleg447-server: oleg447-server.virtnet: executing load_modules_local oleg447-server: Loading modules from /home/green/git/lustre-release/lustre oleg447-server: detected 4 online CPUs by sysfs oleg447-server: Force libcfs to create 2 CPU partitions oleg447-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg447-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg447-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg447-server: oleg447-server.virtnet: executing set_default_debug -1 all pdsh@oleg447-client: oleg447-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg447-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg447-server: oleg447-server.virtnet: executing set_default_debug -1 all pdsh@oleg447-client: oleg447-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg447-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg447-server: oleg447-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg447-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg447-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg447-server stop mds service on oleg447-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg447-server PASS 51 (130s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:17:01 (1709785021) start mds service on oleg447-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg447-server: oleg447-server.virtnet: executing set_default_debug -1 all pdsh@oleg447-client: oleg447-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg447-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg447-server: oleg447-server.virtnet: executing set_default_debug -1 all pdsh@oleg447-client: oleg447-server: ssh exited with exit code 1 Started lustre-OST0000 oleg447-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg447-client.virtnet: -o user_xattr,flock oleg447-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 683 (time 1709785047.50 total 10.37 last 65.83) - open/close 1528 (time 1709785058.27 total 21.14 last 78.49) - open/close 2078 (time 1709785068.65 total 31.53 last 52.96) - open/close 2430 (time 1709785078.65 total 41.53 last 35.20) - open/close 2926 (time 1709785089.07 total 51.94 last 47.62) - open/close 3310 (time 1709785099.07 total 61.94 last 38.39) - open/close 3851 (time 1709785109.40 total 72.28 last 52.36) - open/close 4424 (time 1709785119.90 total 82.77 last 54.61) - open/close 5039 (time 1709785130.56 total 93.43 last 57.68) - open/close 5640 (time 1709785141.39 total 104.26 last 55.50) - open/close 6180 (time 1709785152.23 total 115.11 last 49.78) - open/close 6603 (time 1709785162.76 total 125.63 last 40.19) - open/close 6929 (time 1709785173.05 total 135.92 last 31.67) - open/close 7409 (time 1709785183.84 total 146.71 last 44.50) - open/close 7679 (time 1709785194.17 total 157.04 last 26.14) - open/close 8023 (time 1709785204.38 total 167.25 last 33.68) - open/close 8539 (time 1709785214.39 total 177.26 last 51.57) - open/close 8966 (time 1709785225.19 total 188.06 last 39.52) - open/close 9396 (time 1709785235.19 total 198.06 last 42.99) - open/close 9795 (time 1709785245.94 total 208.81 last 37.11) - open/close 10000 (time 1709785250.84 total 213.71 last 41.84) - open/close 10223 (time 1709785260.91 total 223.78 last 22.15) - open/close 10612 (time 1709785270.91 total 233.79 last 38.89) - open/close 11423 (time 1709785280.92 total 243.79 last 81.07) - open/close 12915 (time 1709785290.92 total 253.79 last 149.19) - open/close 14262 (time 1709785300.92 total 263.80 last 134.60) - open/close 15706 (time 1709785310.92 total 273.80 last 144.39) - open/close 16917 (time 1709785320.93 total 283.80 last 121.02) - open/close 18308 (time 1709785330.94 total 293.81 last 139.03) - open/close 19472 (time 1709785340.94 total 303.81 last 116.39) - open/close 20000 (time 1709785345.45 total 308.32 last 117.11) - open/close 21117 (time 1709785355.45 total 318.32 last 111.64) - open/close 22160 (time 1709785365.46 total 328.33 last 104.24) - open/close 23095 (time 1709785375.46 total 338.33 last 93.50) - open/close 23959 (time 1709785385.46 total 348.34 last 86.34) - open/close 24751 (time 1709785395.47 total 358.35 last 79.11) - open/close 25524 (time 1709785405.48 total 368.36 last 77.23) - open/close 26204 (time 1709785415.48 total 378.36 last 67.99) - open/close 27048 (time 1709785425.49 total 388.37 last 84.31) - open/close 27770 (time 1709785435.50 total 398.38 last 72.15) - open/close 28501 (time 1709785445.52 total 408.39 last 72.99) - open/close 29235 (time 1709785455.52 total 418.39 last 73.38) - open/close 29971 (time 1709785465.53 total 428.40 last 73.56) - open/close 30642 (time 1709785475.53 total 438.41 last 67.04) - open/close 31211 (time 1709785485.55 total 448.42 last 56.82) - open/close 31865 (time 1709785495.55 total 458.43 last 65.37) - open/close 32473 (time 1709785505.57 total 468.45 last 60.68) - open/close 33127 (time 1709785515.58 total 478.46 last 65.33) - open/close 33654 (time 1709785525.59 total 488.46 last 52.69) - open/close 34225 (time 1709785535.60 total 498.47 last 57.02) - open/close 34795 (time 1709785545.60 total 508.48 last 56.98) - open/close 35383 (time 1709785555.60 total 518.48 last 58.80) - open/close 35954 (time 1709785565.61 total 528.49 last 57.05) - open/close 36548 (time 1709785575.63 total 538.50 last 59.31) - open/close 37120 (time 1709785585.65 total 548.52 last 57.09) - open/close 37625 (time 1709785595.65 total 558.53 last 50.46) - open/close 38174 (time 1709785605.66 total 568.54 last 54.86) - open/close 38702 (time 1709785615.67 total 578.55 last 52.75) - open/close 39157 (time 1709785625.68 total 588.55 last 45.47) - open/close 39645 (time 1709785635.68 total 598.55 last 48.80) - open/close 40000 (time 1709785642.51 total 605.38 last 51.98) - open/close 40457 (time 1709785652.52 total 615.40 last 45.63) - open/close 40909 (time 1709785662.54 total 625.41 last 45.13) - open/close 41412 (time 1709785672.55 total 635.42 last 50.25) - open/close 41880 (time 1709785682.55 total 645.43 last 46.79) - open/close 42330 (time 1709785692.56 total 655.43 last 44.97) - open/close 42779 (time 1709785702.59 total 665.46 last 44.77) - open/close 43187 (time 1709785712.59 total 675.47 last 40.77) - open/close 43601 (time 1709785722.60 total 685.47 last 41.39) - open/close 44086 (time 1709785732.60 total 695.47 last 48.48) - open/close 44534 (time 1709785742.61 total 705.49 last 44.74) - open/close 44995 (time 1709785752.63 total 715.51 last 46.02) - open/close 45429 (time 1709785762.63 total 725.51 last 43.40) - open/close 45875 (time 1709785772.65 total 735.52 last 44.54) - open/close 46294 (time 1709785782.65 total 745.52 last 41.90) - open/close 46729 (time 1709785792.65 total 755.52 last 43.48) - open/close 47166 (time 1709785802.67 total 765.55 last 43.60) - open/close 47550 (time 1709785812.68 total 775.55 last 38.39) - open/close 48013 (time 1709785822.69 total 785.56 last 46.26) - open/close 48423 (time 1709785832.69 total 795.56 last 40.98) - open/close 48851 (time 1709785842.69 total 805.56 last 42.80) - open/close 49221 (time 1709785852.72 total 815.59 last 36.90) - open/close 49592 (time 1709785862.74 total 825.61 last 37.01) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49698) error: No space left on device total: 49698 open/close in 828.37 seconds: 59.99 ops/second - unlinked 0 (time 1709785866 ; total 0 ; last 0) - unlinked 10000 (time 1709786147 ; total 281 ; last 281) - unlinked 20000 (time 1709786354 ; total 488 ; last 207) - unlinked 30000 (time 1709786478 ; total 612 ; last 124) - unlinked 40000 (time 1709786540 ; total 674 ; last 62) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49698) error: No such file or directory total: 49698 unlinks in 713 seconds: 69.702667 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg447-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg447-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg447-server stop mds service on oleg447-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg447-server unloading modules on: 'oleg447-server' oleg447-server: oleg447-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg447-client: oleg447-client: ssh exited with exit code 2 pdsh@oleg447-client: oleg447-server: ssh exited with exit code 2 pdsh@oleg447-client: oleg447-client: ssh exited with exit code 2 pdsh@oleg447-client: oleg447-server: ssh exited with exit code 2 PASS 101a (1581s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg447-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg447-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg447-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg447-client: oleg447-server: ssh exited with exit code 2 oleg447-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg447-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == conf-sanity test 101b: Race events DISCONNECT and ACTIVE in osp ========================================================== 23:43:24 (1709786604) start mds service on oleg447-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg447-server' oleg447-server: oleg447-server.virtnet: executing load_modules_local oleg447-server: Loading modules from /home/green/git/lustre-release/lustre oleg447-server: detected 4 online CPUs by sysfs oleg447-server: Force libcfs to create 2 CPU partitions oleg447-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg447-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg447-server: oleg447-server.virtnet: executing set_default_debug -1 all pdsh@oleg447-client: oleg447-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg447-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg447-server: oleg447-server.virtnet: executing set_default_debug -1 all pdsh@oleg447-client: oleg447-server: ssh exited with exit code 1 Started lustre-OST0000 oleg447-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg447-client.virtnet: -o user_xattr,flock oleg447-server@tcp:/lustre /mnt/lustre fail_loc=0x80002107 fail_val=20 stop ost1 service on oleg447-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg447-server start ost1 service on oleg447-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg447-server: oleg447-server.virtnet: executing set_default_debug -1 all pdsh@oleg447-client: oleg447-server: ssh exited with exit code 1 Started lustre-OST0000 oleg447-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 1 sec oleg447-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8800abf61800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800abf61800.ost_server_uuid in FULL state after 0 sec umount lustre on /mnt/lustre..... Stopping client oleg447-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg447-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg447-server stop mds service on oleg447-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg447-server unloading modules on: 'oleg447-server' oleg447-server: oleg447-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg447-client: oleg447-client: ssh exited with exit code 2 pdsh@oleg447-client: oleg447-server: ssh exited with exit code 2 pdsh@oleg447-client: oleg447-client: ssh exited with exit code 2 pdsh@oleg447-client: oleg447-server: ssh exited with exit code 2 PASS 101b (94s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg447-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg447-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory Stopping clients: oleg447-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg447-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg447-client: oleg447-server: ssh exited with exit code 2 oleg447-server: oleg447-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg447-server' oleg447-server: oleg447-server.virtnet: executing load_modules_local oleg447-server: Loading modules from /home/green/git/lustre-release/lustre oleg447-server: detected 4 online CPUs by sysfs oleg447-server: Force libcfs to create 2 CPU partitions oleg447-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg447-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey == conf-sanity test complete, duration 2098 sec ========== 23:45:28 (1709786728) === conf-sanity: start cleanup 23:45:29 (1709786729) === === conf-sanity: finish cleanup 23:45:29 (1709786729) ===