-----============= acceptance-small: conf-sanity ============----- Wed Mar 6 23:39:46 EST 2024 excepting tests: 32newtarball 110 skipping tests SLOW=no: 45 69 106 111 114 Stopping clients: oleg332-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg332-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg332-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg332-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg332-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg332-server oleg332-server: oleg332-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory loading modules on: 'oleg332-server' oleg332-server: oleg332-server.virtnet: executing load_modules_local oleg332-server: Loading modules from /home/green/git/lustre-release/lustre oleg332-server: detected 4 online CPUs by sysfs oleg332-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg332-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg332-server: oleg332-server.virtnet: executing set_default_debug -1 all pdsh@oleg332-client: oleg332-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg332-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg332-server: oleg332-server.virtnet: executing set_default_debug -1 all pdsh@oleg332-client: oleg332-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg332-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg332-server: oleg332-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg332-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg332-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg332-server stop mds service on oleg332-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg332-server debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 35b: Continue reconnection retries, if the active server is busy ========================================================== 23:40:53 (1709786453) start mds service on oleg332-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg332-server: oleg332-server.virtnet: executing set_default_debug -1 all pdsh@oleg332-client: oleg332-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg332-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg332-server: oleg332-server.virtnet: executing set_default_debug -1 all pdsh@oleg332-client: oleg332-server: ssh exited with exit code 1 Started lustre-OST0000 oleg332-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg332-client.virtnet: -o user_xattr,flock oleg332-server@tcp:/lustre /mnt/lustre debug=ha conf-sanity.sh test_35b 2024-03-0623h41m09s Set up a fake failnode for the MDS at_max=0 at_max=0 Injecting EBUSY on MDS fail_loc=0x80000136 mdc.lustre-MDT0000-mdc-ffff880136952800.stats=clear Creating a test file and stat it File: '/mnt/lustre/d35b.conf-sanity/f35b.conf-sanity' Size: 0 Blocks: 0 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115238826934274 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-03-06 23:41:52.000000000 -0500 Modify: 2024-03-06 23:41:52.000000000 -0500 Change: 2024-03-06 23:41:52.000000000 -0500 Birth: - Stop injecting EBUSY on MDS fail_loc=0 done at_max=600 at_max=600 Debug log: 54 lines, 54 kept, 0 dropped, 0 bad. umount lustre on /mnt/lustre..... Stopping client oleg332-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg332-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg332-server stop mds service on oleg332-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg332-server unloading modules on: 'oleg332-server' oleg332-server: oleg332-server.virtnet: executing unload_modules_local modules unloaded. oleg332-server: tunefs.lustre: Unable to mount /dev/mapper/mds1_flakey: No such device oleg332-server: Is the ldiskfs module available? oleg332-server: oleg332-server: tunefs.lustre FATAL: failed to write local files oleg332-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg332-client: oleg332-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x5 (MDT MGS ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity Permanent disk data: Target: lustre=MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x105 (MDT MGS writeconf ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity oleg332-server: tunefs.lustre: Unable to mount /dev/mapper/ost1_flakey: No such device oleg332-server: Is the ldiskfs module available? oleg332-server: oleg332-server: tunefs.lustre FATAL: failed to write local files oleg332-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg332-client: oleg332-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x2 (OST ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.203.132@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x102 (OST writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.203.132@tcp sys.timeout=20 oleg332-server: tunefs.lustre: Unable to mount /dev/mapper/ost2_flakey: No such device oleg332-server: Is the ldiskfs module available? oleg332-server: oleg332-server: tunefs.lustre FATAL: failed to write local files oleg332-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg332-client: oleg332-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.203.132@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x162 (OST first_time update writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.203.132@tcp sys.timeout=20 tunefs failed, reformatting instead Stopping clients: oleg332-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg332-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg332-client: oleg332-server: ssh exited with exit code 2 oleg332-server: oleg332-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg332-server' oleg332-server: oleg332-server.virtnet: executing load_modules_local oleg332-server: Loading modules from /home/green/git/lustre-release/lustre oleg332-server: detected 4 online CPUs by sysfs oleg332-server: Force libcfs to create 2 CPU partitions oleg332-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg332-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg332-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg332-server: oleg332-server.virtnet: executing set_default_debug -1 all pdsh@oleg332-client: oleg332-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg332-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg332-server: oleg332-server.virtnet: executing set_default_debug -1 all pdsh@oleg332-client: oleg332-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg332-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg332-server: oleg332-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg332-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg332-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg332-server stop mds service on oleg332-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg332-server PASS 35b (138s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 23:43:14 (1709786594) start mds service on oleg332-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg332-server: oleg332-server.virtnet: executing set_default_debug -1 all pdsh@oleg332-client: oleg332-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg332-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg332-server: oleg332-server.virtnet: executing set_default_debug -1 all pdsh@oleg332-client: oleg332-server: ssh exited with exit code 1 Started lustre-OST0000 oleg332-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg332-client.virtnet: -o user_xattr,flock oleg332-server@tcp:/lustre /mnt/lustre copying 10 files to /mnt/lustre/d38.conf-sanity tar: Removing leading `/' from member names umount lustre on /mnt/lustre..... Stopping client oleg332-client.virtnet /mnt/lustre (opts:) osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=44 stop mds service on oleg332-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg332-server delete lov_objid file on MDS 000000 43 000008 start mds service on oleg332-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg332-server: oleg332-server.virtnet: executing set_default_debug -1 all pdsh@oleg332-client: oleg332-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg332-client.virtnet: -o user_xattr,flock oleg332-server@tcp:/lustre /mnt/lustre osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=2 umount lustre on /mnt/lustre..... Stopping client oleg332-client.virtnet /mnt/lustre (opts:) stop mds service on oleg332-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg332-server 000000 2 000008 oleg332-server: 8+0 records in oleg332-server: 8+0 records out oleg332-server: 4096 bytes (4.1 kB) copied, 0.000682716 s, 6.0 MB/s start mds service on oleg332-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg332-server: oleg332-server.virtnet: executing set_default_debug -1 all pdsh@oleg332-client: oleg332-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg332-client.virtnet: -o user_xattr,flock oleg332-server@tcp:/lustre /mnt/lustre osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=35 umount lustre on /mnt/lustre..... Stopping client oleg332-client.virtnet /mnt/lustre (opts:) stop mds service on oleg332-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg332-server 000000 34 000008 files compared the same umount lustre on /mnt/lustre..... stop ost1 service on oleg332-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg332-server stop mds service on oleg332-server unloading modules on: 'oleg332-server' oleg332-server: oleg332-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg332-client: oleg332-client: ssh exited with exit code 2 pdsh@oleg332-client: oleg332-server: ssh exited with exit code 2 pdsh@oleg332-client: oleg332-server: ssh exited with exit code 2 pdsh@oleg332-client: oleg332-client: ssh exited with exit code 2 PASS 38 (86s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg332-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg332-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg332-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg332-client: oleg332-server: ssh exited with exit code 2 oleg332-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg332-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 23:44:43 (1709786683) Stopping clients: oleg332-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg332-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg332-client: oleg332-server: ssh exited with exit code 2 oleg332-server: oleg332-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg332-server' oleg332-server: oleg332-server.virtnet: executing load_modules_local oleg332-server: Loading modules from /home/green/git/lustre-release/lustre oleg332-server: detected 4 online CPUs by sysfs oleg332-server: Force libcfs to create 2 CPU partitions oleg332-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg332-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg332-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg332-server: oleg332-server.virtnet: executing set_default_debug -1 all pdsh@oleg332-client: oleg332-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg332-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg332-server: oleg332-server.virtnet: executing set_default_debug -1 all pdsh@oleg332-client: oleg332-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg332-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg332-client.virtnet: -o user_xattr,flock oleg332-server@tcp:/lustre /mnt/lustre setup single mount lustre success fail_loc=0x142 start ost2 service on oleg332-server Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg332-server: oleg332-server.virtnet: executing set_default_debug -1 all pdsh@oleg332-client: oleg332-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost2_flakey Started lustre-OST0001 oleg332-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec stop ost2 service on oleg332-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg332-server umount lustre on /mnt/lustre..... Stopping client oleg332-client.virtnet /mnt/lustre (opts:-f) stop ost1 service on oleg332-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg332-server stop mds service on oleg332-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg332-server unloading modules on: 'oleg332-server' oleg332-server: oleg332-server.virtnet: executing unload_modules_local modules unloaded. oleg332-server: tunefs.lustre: Unable to mount /dev/mapper/mds1_flakey: No such device oleg332-server: Is the ldiskfs module available? oleg332-server: oleg332-server: tunefs.lustre FATAL: failed to write local files oleg332-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg332-client: oleg332-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x5 (MDT MGS ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity Permanent disk data: Target: lustre=MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x105 (MDT MGS writeconf ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity oleg332-server: tunefs.lustre: Unable to mount /dev/mapper/ost1_flakey: No such device oleg332-server: Is the ldiskfs module available? oleg332-server: oleg332-server: tunefs.lustre FATAL: failed to write local files oleg332-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg332-client: oleg332-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x2 (OST ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.203.132@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x102 (OST writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.203.132@tcp sys.timeout=20 oleg332-server: tunefs.lustre: Unable to mount /dev/mapper/ost2_flakey: No such device oleg332-server: Is the ldiskfs module available? oleg332-server: oleg332-server: tunefs.lustre FATAL: failed to write local files oleg332-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg332-client: oleg332-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x2 (OST ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.203.132@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x102 (OST writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.203.132@tcp sys.timeout=20 tunefs failed, reformatting instead Stopping clients: oleg332-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg332-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg332-client: oleg332-server: ssh exited with exit code 2 oleg332-server: oleg332-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg332-server' oleg332-server: oleg332-server.virtnet: executing load_modules_local oleg332-server: Loading modules from /home/green/git/lustre-release/lustre oleg332-server: detected 4 online CPUs by sysfs oleg332-server: Force libcfs to create 2 CPU partitions oleg332-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg332-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg332-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg332-server: oleg332-server.virtnet: executing set_default_debug -1 all pdsh@oleg332-client: oleg332-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg332-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg332-server: oleg332-server.virtnet: executing set_default_debug -1 all pdsh@oleg332-client: oleg332-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg332-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg332-server: oleg332-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg332-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg332-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg332-server stop mds service on oleg332-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg332-server PASS 51 (152s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:47:17 (1709786837) start mds service on oleg332-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg332-server: oleg332-server.virtnet: executing set_default_debug -1 all pdsh@oleg332-client: oleg332-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg332-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg332-server: oleg332-server.virtnet: executing set_default_debug -1 all pdsh@oleg332-client: oleg332-server: ssh exited with exit code 1 Started lustre-OST0000 oleg332-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg332-client.virtnet: -o user_xattr,flock oleg332-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 263 (time 1709786868.90 total 10.82 last 24.31) - open/close 409 (time 1709786879.67 total 21.58 last 13.56) - open/close 681 (time 1709786890.60 total 32.51 last 24.89) - open/close 882 (time 1709786901.42 total 43.33 last 18.57) - open/close 1047 (time 1709786911.52 total 53.44 last 16.33) - open/close 1297 (time 1709786922.28 total 64.20 last 23.22) - open/close 1559 (time 1709786932.29 total 74.20 last 26.19) - open/close 1921 (time 1709786943.21 total 85.13 last 33.14) - open/close 2008 (time 1709786953.53 total 95.44 last 8.43) - open/close 2161 (time 1709786963.63 total 105.55 last 15.14) - open/close 2369 (time 1709786981.00 total 122.91 last 11.98) - open/close 2628 (time 1709786991.62 total 133.53 last 24.39) - open/close 2827 (time 1709787002.24 total 144.15 last 18.73) - open/close 3036 (time 1709787012.24 total 154.16 last 20.90) - open/close 3291 (time 1709787022.25 total 164.17 last 25.48) - open/close 3467 (time 1709787032.66 total 174.58 last 16.91) - open/close 3615 (time 1709787042.66 total 184.58 last 14.79) - open/close 3762 (time 1709787053.47 total 195.38 last 13.61) - open/close 4048 (time 1709787063.83 total 205.75 last 27.59) - open/close 4263 (time 1709787073.84 total 215.76 last 21.47) - open/close 4475 (time 1709787084.54 total 226.46 last 19.82) - open/close 4775 (time 1709787094.55 total 236.47 last 29.96) - open/close 5012 (time 1709787105.09 total 247.01 last 22.49) - open/close 5130 (time 1709787125.03 total 266.95 last 5.92) - open/close 5318 (time 1709787135.04 total 276.95 last 18.80) - open/close 5450 (time 1709787145.25 total 287.17 last 12.92) - open/close 5702 (time 1709787155.26 total 297.17 last 25.18) - open/close 5924 (time 1709787166.90 total 308.82 last 19.06) - open/close 6144 (time 1709787177.73 total 319.64 last 20.33) - open/close 6442 (time 1709787187.79 total 329.70 last 29.62) - open/close 6654 (time 1709787198.66 total 340.58 last 19.49) - open/close 6751 (time 1709787209.14 total 351.06 last 9.26) - open/close 6956 (time 1709787219.81 total 361.73 last 19.22) - open/close 7131 (time 1709787230.35 total 372.27 last 16.60) - open/close 7280 (time 1709787240.42 total 382.34 last 14.79) - open/close 7529 (time 1709787250.45 total 392.37 last 24.83) - open/close 7622 (time 1709787260.46 total 402.38 last 9.29) - open/close 7783 (time 1709787270.81 total 412.73 last 15.55) - open/close 7945 (time 1709787281.01 total 422.92 last 15.89) - open/close 8095 (time 1709787291.02 total 432.94 last 14.98) - open/close 8303 (time 1709787301.68 total 443.59 last 19.52) - open/close 8542 (time 1709787311.88 total 453.80 last 23.42) - open/close 8804 (time 1709787321.89 total 463.81 last 26.18) - open/close 8992 (time 1709787332.81 total 474.73 last 17.21) - open/close 9275 (time 1709787342.82 total 484.74 last 28.27) - open/close 9484 (time 1709787353.29 total 495.21 last 19.96) - open/close 9774 (time 1709787363.30 total 505.21 last 28.98) - open/close 9892 (time 1709787374.14 total 516.06 last 10.88) - open/close 10000 (time 1709787380.47 total 522.38 last 17.08) - open/close 10468 (time 1709787390.48 total 532.39 last 46.75) - open/close 11043 (time 1709787400.49 total 542.40 last 57.45) - open/close 11547 (time 1709787410.50 total 552.41 last 50.36) - open/close 12079 (time 1709787420.53 total 562.44 last 53.04) - open/close 12663 (time 1709787430.54 total 572.45 last 58.33) - open/close 12963 (time 1709787446.92 total 588.83 last 18.32) - open/close 13470 (time 1709787456.92 total 598.84 last 50.66) - open/close 13892 (time 1709787466.94 total 608.85 last 42.16) - open/close 14268 (time 1709787476.96 total 618.87 last 37.52) - open/close 14728 (time 1709787486.97 total 628.89 last 45.93) - open/close 15216 (time 1709787496.99 total 638.91 last 48.69) - open/close 15697 (time 1709787506.99 total 648.91 last 48.09) - open/close 16162 (time 1709787517.00 total 658.92 last 46.46) - open/close 16564 (time 1709787527.03 total 668.94 last 40.10) - open/close 16965 (time 1709787537.03 total 678.94 last 40.09) - open/close 17389 (time 1709787547.03 total 688.95 last 42.38) - open/close 17763 (time 1709787557.04 total 698.96 last 37.36) - open/close 18137 (time 1709787567.07 total 708.99 last 37.30) - open/close 18495 (time 1709787577.09 total 719.00 last 35.75) - open/close 18857 (time 1709787587.09 total 729.00 last 36.19) - open/close 19179 (time 1709787597.10 total 739.01 last 32.17) - open/close 19519 (time 1709787607.13 total 749.05 last 33.89) - open/close 19746 (time 1709787624.31 total 766.23 last 13.21) - open/close 19886 (time 1709787639.11 total 781.03 last 9.46) - open/close 20000 (time 1709787642.71 total 784.63 last 31.68) - open/close 20347 (time 1709787652.73 total 794.65 last 34.63) - open/close 20467 (time 1709787667.85 total 809.76 last 7.94) - open/close 20809 (time 1709787677.86 total 819.78 last 34.15) - open/close 21139 (time 1709787687.90 total 829.82 last 32.87) - open/close 21398 (time 1709787697.93 total 839.85 last 25.82) - open/close 21686 (time 1709787707.96 total 849.88 last 28.71) - open/close 21953 (time 1709787717.98 total 859.89 last 26.67) - open/close 22286 (time 1709787727.99 total 869.90 last 33.27) - open/close 22610 (time 1709787737.99 total 879.90 last 32.39) - open/close 22912 (time 1709787748.01 total 889.93 last 30.13) - open/close 23174 (time 1709787758.03 total 899.95 last 26.14) - open/close 23481 (time 1709787768.05 total 909.96 last 30.66) - open/close 23786 (time 1709787778.06 total 919.97 last 30.47) - open/close 24111 (time 1709787788.06 total 929.97 last 32.50) - open/close 24200 (time 1709787802.27 total 944.19 last 6.26) - open/close 24542 (time 1709787812.28 total 954.19 last 34.20) - open/close 24864 (time 1709787822.29 total 964.21 last 32.15) - open/close 25205 (time 1709787832.31 total 974.23 last 34.02) - open/close 25553 (time 1709787842.36 total 984.27 last 34.65) - open/close 25877 (time 1709787852.37 total 994.28 last 32.36) - open/close 26206 (time 1709787862.39 total 1004.31 last 32.83) - open/close 26527 (time 1709787872.41 total 1014.32 last 32.06) - open/close 26833 (time 1709787882.41 total 1024.33 last 30.58) - open/close 27157 (time 1709787892.44 total 1034.36 last 32.30) - open/close 27465 (time 1709787902.57 total 1044.49 last 30.40) - open/close 27791 (time 1709787912.59 total 1054.51 last 32.53) - open/close 28081 (time 1709787922.62 total 1064.54 last 28.92) - open/close 28348 (time 1709787942.78 total 1084.69 last 13.25) - open/close 28631 (time 1709787952.79 total 1094.71 last 28.26) - open/close 28942 (time 1709787962.83 total 1104.74 last 31.00) - open/close 29260 (time 1709787972.84 total 1114.75 last 31.76) - open/close 29568 (time 1709787982.85 total 1124.77 last 30.76) - open/close 29890 (time 1709787992.89 total 1134.80 last 32.07) - open/close 30000 (time 1709787996.33 total 1138.24 last 31.99) - open/close 30323 (time 1709788006.35 total 1148.26 last 32.24) - open/close 30625 (time 1709788016.38 total 1158.30 last 30.09) - open/close 30904 (time 1709788026.39 total 1168.30 last 27.88) - open/close 31192 (time 1709788036.39 total 1178.31 last 28.79) - open/close 31467 (time 1709788046.41 total 1188.32 last 27.45) - open/close 31738 (time 1709788056.43 total 1198.35 last 27.03) - open/close 32015 (time 1709788066.46 total 1208.38 last 27.62) - open/close 32305 (time 1709788076.46 total 1218.38 last 28.99) - open/close 32661 (time 1709788086.50 total 1228.41 last 35.49) - open/close 33003 (time 1709788096.51 total 1238.43 last 34.13) - open/close 33338 (time 1709788106.52 total 1248.43 last 33.50) - open/close 33681 (time 1709788116.52 total 1258.44 last 34.27) - open/close 34019 (time 1709788126.54 total 1268.45 last 33.75) - open/close 34379 (time 1709788136.57 total 1278.48 last 35.90) - open/close 34740 (time 1709788146.58 total 1288.50 last 36.04) - open/close 35068 (time 1709788156.60 total 1298.51 last 32.75) - open/close 35401 (time 1709788166.61 total 1308.52 last 33.28) - open/close 35741 (time 1709788176.62 total 1318.53 last 33.96) - open/close 36060 (time 1709788186.64 total 1328.56 last 31.82) - open/close 36376 (time 1709788196.67 total 1338.59 last 31.51) - open/close 36688 (time 1709788206.68 total 1348.60 last 31.17) - open/close 37003 (time 1709788216.68 total 1358.60 last 31.50) - open/close 37124 (time 1709788231.48 total 1373.40 last 8.18) - open/close 37430 (time 1709788241.50 total 1383.42 last 30.54) - open/close 37724 (time 1709788251.52 total 1393.43 last 29.36) - open/close 38004 (time 1709788261.54 total 1403.46 last 27.93) - open/close 38291 (time 1709788271.58 total 1413.49 last 28.59) - open/close 38578 (time 1709788281.61 total 1423.52 last 28.62) - open/close 38853 (time 1709788291.61 total 1433.52 last 27.49) - open/close 39133 (time 1709788301.63 total 1443.55 last 27.93) - open/close 39410 (time 1709788311.63 total 1453.55 last 27.70) - open/close 39692 (time 1709788321.66 total 1463.58 last 28.12) - open/close 39975 (time 1709788331.69 total 1473.60 last 28.24) - open/close 40264 (time 1709788341.70 total 1483.61 last 28.87) - open/close 40552 (time 1709788351.72 total 1493.64 last 28.73) - open/close 40830 (time 1709788361.75 total 1503.67 last 27.71) - open/close 41080 (time 1709788371.75 total 1513.67 last 24.99) - open/close 41337 (time 1709788381.78 total 1523.70 last 25.63) - open/close 41602 (time 1709788391.79 total 1533.70 last 26.49) - open/close 41875 (time 1709788401.80 total 1543.72 last 27.26) - open/close 42141 (time 1709788411.82 total 1553.73 last 26.56) - open/close 42441 (time 1709788421.82 total 1563.74 last 29.98) - open/close 42702 (time 1709788431.85 total 1573.76 last 26.04) - open/close 42974 (time 1709788441.87 total 1583.78 last 27.14) - open/close 43233 (time 1709788451.89 total 1593.81 last 25.83) - open/close 43495 (time 1709788461.90 total 1603.81 last 26.19) - open/close 43740 (time 1709788471.93 total 1613.84 last 24.43) - open/close 44011 (time 1709788481.96 total 1623.88 last 27.01) - open/close 44252 (time 1709788491.99 total 1633.90 last 24.03) - open/close 44492 (time 1709788502.01 total 1643.93 last 23.95) - open/close 44740 (time 1709788512.04 total 1653.96 last 24.72) - open/close 45007 (time 1709788522.05 total 1663.97 last 26.68) - open/close 45265 (time 1709788532.05 total 1673.97 last 25.79) - open/close 45527 (time 1709788542.09 total 1684.00 last 26.11) - open/close 45792 (time 1709788552.10 total 1694.01 last 26.48) - open/close 46044 (time 1709788562.11 total 1704.03 last 25.15) - open/close 46311 (time 1709788572.14 total 1714.06 last 26.62) - open/close 46580 (time 1709788582.16 total 1724.07 last 26.87) - open/close 46850 (time 1709788592.18 total 1734.09 last 26.95) - open/close 47119 (time 1709788602.21 total 1744.13 last 26.80) - open/close 47384 (time 1709788612.23 total 1754.15 last 26.44) - open/close 47656 (time 1709788622.25 total 1764.16 last 27.16) - open/close 47928 (time 1709788632.28 total 1774.20 last 27.11) - open/close 48191 (time 1709788642.29 total 1784.21 last 26.28) - open/close 48462 (time 1709788652.29 total 1794.21 last 27.10) - open/close 48724 (time 1709788662.30 total 1804.21 last 26.19) - open/close 48981 (time 1709788672.30 total 1814.21 last 25.69) - open/close 49226 (time 1709788682.32 total 1824.24 last 24.44) - open/close 49461 (time 1709788692.35 total 1834.27 last 23.43) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49698) error: No space left on device total: 49698 open/close in 1844.25 seconds: 26.95 ops/second - unlinked 0 (time 1709788703 ; total 0 ; last 0) - unlinked 10000 (time 1709789179 ; total 476 ; last 476) - unlinked 20000 (time 1709789608 ; total 905 ; last 429) - unlinked 30000 (time 1709789951 ; total 1248 ; last 343) - unlinked 40000 (time 1709790143 ; total 1440 ; last 192) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49698) error: No such file or directory total: 49698 unlinks in 1500 seconds: 33.132000 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg332-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg332-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg332-server stop mds service on oleg332-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg332-server unloading modules on: 'oleg332-server' oleg332-server: oleg332-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg332-client: oleg332-client: ssh exited with exit code 2 pdsh@oleg332-client: oleg332-server: ssh exited with exit code 2 pdsh@oleg332-client: oleg332-client: ssh exited with exit code 2 pdsh@oleg332-client: oleg332-server: ssh exited with exit code 2 PASS 101a (3383s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg332-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg332-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg332-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg332-client: oleg332-server: ssh exited with exit code 2 oleg332-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg332-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == conf-sanity test 101b: Race events DISCONNECT and ACTIVE in osp ========================================================== 00:43:42 (1709790222) start mds service on oleg332-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg332-server' oleg332-server: oleg332-server.virtnet: executing load_modules_local oleg332-server: Loading modules from /home/green/git/lustre-release/lustre oleg332-server: detected 4 online CPUs by sysfs oleg332-server: Force libcfs to create 2 CPU partitions oleg332-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg332-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg332-server: oleg332-server.virtnet: executing set_default_debug -1 all pdsh@oleg332-client: oleg332-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg332-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg332-server: oleg332-server.virtnet: executing set_default_debug -1 all pdsh@oleg332-client: oleg332-server: ssh exited with exit code 1 Started lustre-OST0000 oleg332-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg332-client.virtnet: -o user_xattr,flock oleg332-server@tcp:/lustre /mnt/lustre fail_loc=0x80002107 fail_val=20 stop ost1 service on oleg332-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg332-server start ost1 service on oleg332-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg332-server: oleg332-server.virtnet: executing set_default_debug -1 all pdsh@oleg332-client: oleg332-server: ssh exited with exit code 1 Started lustre-OST0000 oleg332-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 5 sec oleg332-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff88012a495000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff88012a495000.ost_server_uuid in FULL state after 0 sec umount lustre on /mnt/lustre..... Stopping client oleg332-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg332-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg332-server stop mds service on oleg332-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg332-server unloading modules on: 'oleg332-server' oleg332-server: oleg332-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg332-client: oleg332-client: ssh exited with exit code 2 pdsh@oleg332-client: oleg332-server: ssh exited with exit code 2 pdsh@oleg332-client: oleg332-client: ssh exited with exit code 2 pdsh@oleg332-client: oleg332-server: ssh exited with exit code 2 PASS 101b (88s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg332-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg332-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory Stopping clients: oleg332-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg332-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg332-client: oleg332-server: ssh exited with exit code 2 oleg332-server: oleg332-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg332-server' oleg332-server: oleg332-server.virtnet: executing load_modules_local oleg332-server: Loading modules from /home/green/git/lustre-release/lustre oleg332-server: detected 4 online CPUs by sysfs oleg332-server: Force libcfs to create 2 CPU partitions oleg332-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg332-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey == conf-sanity test complete, duration 3945 sec ========== 00:45:31 (1709790331) === conf-sanity: start cleanup 00:45:32 (1709790332) === === conf-sanity: finish cleanup 00:45:32 (1709790332) ===