-----============= acceptance-small: conf-sanity ============----- Wed Mar 6 23:39:38 EST 2024 excepting tests: 32newtarball 110 skipping tests SLOW=no: 45 69 106 111 114 Stopping clients: oleg257-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg257-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg257-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg257-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg257-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg257-server oleg257-server: oleg257-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg257-server' oleg257-server: oleg257-server.virtnet: executing load_modules_local oleg257-server: Loading modules from /home/green/git/lustre-release/lustre oleg257-server: detected 4 online CPUs by sysfs oleg257-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg257-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg257-server: oleg257-server.virtnet: executing set_default_debug -1 all pdsh@oleg257-client: oleg257-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg257-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg257-server: oleg257-server.virtnet: executing set_default_debug -1 all pdsh@oleg257-client: oleg257-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg257-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg257-server: oleg257-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg257-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg257-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg257-server stop mds service on oleg257-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg257-server debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 35b: Continue reconnection retries, if the active server is busy ========================================================== 23:40:44 (1709786444) start mds service on oleg257-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg257-server: oleg257-server.virtnet: executing set_default_debug -1 all pdsh@oleg257-client: oleg257-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg257-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg257-server: oleg257-server.virtnet: executing set_default_debug -1 all pdsh@oleg257-client: oleg257-server: ssh exited with exit code 1 Started lustre-OST0000 oleg257-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg257-client.virtnet: -o user_xattr,flock oleg257-server@tcp:/lustre /mnt/lustre debug=ha conf-sanity.sh test_35b 2024-03-0623h41m01s Set up a fake failnode for the MDS at_max=0 at_max=0 Injecting EBUSY on MDS fail_loc=0x80000136 mdc.lustre-MDT0000-mdc-ffff8800b6d16000.stats=clear Creating a test file and stat it File: '/mnt/lustre/d35b.conf-sanity/f35b.conf-sanity' Size: 0 Blocks: 0 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115238826934274 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-03-06 23:41:44.000000000 -0500 Modify: 2024-03-06 23:41:44.000000000 -0500 Change: 2024-03-06 23:41:44.000000000 -0500 Birth: - Stop injecting EBUSY on MDS fail_loc=0 done at_max=600 at_max=600 Debug log: 74 lines, 74 kept, 0 dropped, 0 bad. umount lustre on /mnt/lustre..... Stopping client oleg257-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg257-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg257-server stop mds service on oleg257-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg257-server unloading modules on: 'oleg257-server' oleg257-server: oleg257-server.virtnet: executing unload_modules_local modules unloaded. oleg257-server: tunefs.lustre: Unable to mount /dev/mapper/mds1_flakey: No such device oleg257-server: Is the ldiskfs module available? oleg257-server: oleg257-server: tunefs.lustre FATAL: failed to write local files oleg257-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg257-client: oleg257-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x5 (MDT MGS ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity Permanent disk data: Target: lustre=MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x105 (MDT MGS writeconf ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity oleg257-server: tunefs.lustre: Unable to mount /dev/mapper/ost1_flakey: No such device oleg257-server: Is the ldiskfs module available? oleg257-server: oleg257-server: tunefs.lustre FATAL: failed to write local files oleg257-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg257-client: oleg257-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x2 (OST ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.202.157@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x102 (OST writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.202.157@tcp sys.timeout=20 oleg257-server: tunefs.lustre: Unable to mount /dev/mapper/ost2_flakey: No such device oleg257-server: Is the ldiskfs module available? oleg257-server: oleg257-server: tunefs.lustre FATAL: failed to write local files oleg257-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg257-client: oleg257-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.202.157@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x162 (OST first_time update writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.202.157@tcp sys.timeout=20 tunefs failed, reformatting instead Stopping clients: oleg257-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg257-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg257-client: oleg257-server: ssh exited with exit code 2 oleg257-server: oleg257-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg257-server' oleg257-server: oleg257-server.virtnet: executing load_modules_local oleg257-server: Loading modules from /home/green/git/lustre-release/lustre oleg257-server: detected 4 online CPUs by sysfs oleg257-server: Force libcfs to create 2 CPU partitions oleg257-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg257-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg257-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg257-server: oleg257-server.virtnet: executing set_default_debug -1 all pdsh@oleg257-client: oleg257-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg257-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg257-server: oleg257-server.virtnet: executing set_default_debug -1 all pdsh@oleg257-client: oleg257-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg257-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg257-server: oleg257-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg257-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg257-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg257-server stop mds service on oleg257-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg257-server PASS 35b (120s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 23:42:46 (1709786566) start mds service on oleg257-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg257-server: oleg257-server.virtnet: executing set_default_debug -1 all pdsh@oleg257-client: oleg257-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg257-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg257-server: oleg257-server.virtnet: executing set_default_debug -1 all pdsh@oleg257-client: oleg257-server: ssh exited with exit code 1 Started lustre-OST0000 oleg257-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg257-client.virtnet: -o user_xattr,flock oleg257-server@tcp:/lustre /mnt/lustre copying 10 files to /mnt/lustre/d38.conf-sanity tar: Removing leading `/' from member names umount lustre on /mnt/lustre..... Stopping client oleg257-client.virtnet /mnt/lustre (opts:) osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=44 stop mds service on oleg257-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg257-server delete lov_objid file on MDS 000000 43 000008 start mds service on oleg257-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg257-server: oleg257-server.virtnet: executing set_default_debug -1 all pdsh@oleg257-client: oleg257-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg257-client.virtnet: -o user_xattr,flock oleg257-server@tcp:/lustre /mnt/lustre osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=2 umount lustre on /mnt/lustre..... Stopping client oleg257-client.virtnet /mnt/lustre (opts:) stop mds service on oleg257-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg257-server 000000 2 000008 oleg257-server: 8+0 records in oleg257-server: 8+0 records out oleg257-server: 4096 bytes (4.1 kB) copied, 0.000932396 s, 4.4 MB/s start mds service on oleg257-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg257-server: oleg257-server.virtnet: executing set_default_debug -1 all pdsh@oleg257-client: oleg257-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg257-client.virtnet: -o user_xattr,flock oleg257-server@tcp:/lustre /mnt/lustre osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=35 umount lustre on /mnt/lustre..... Stopping client oleg257-client.virtnet /mnt/lustre (opts:) stop mds service on oleg257-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg257-server 000000 34 000008 files compared the same umount lustre on /mnt/lustre..... stop ost1 service on oleg257-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg257-server stop mds service on oleg257-server unloading modules on: 'oleg257-server' oleg257-server: oleg257-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg257-client: oleg257-client: ssh exited with exit code 2 pdsh@oleg257-client: oleg257-server: ssh exited with exit code 2 pdsh@oleg257-client: oleg257-client: ssh exited with exit code 2 pdsh@oleg257-client: oleg257-server: ssh exited with exit code 2 PASS 38 (80s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg257-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg257-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg257-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg257-client: oleg257-server: ssh exited with exit code 2 oleg257-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg257-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 23:44:08 (1709786648) Stopping clients: oleg257-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg257-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg257-client: oleg257-server: ssh exited with exit code 2 oleg257-server: oleg257-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg257-server' oleg257-server: oleg257-server.virtnet: executing load_modules_local oleg257-server: Loading modules from /home/green/git/lustre-release/lustre oleg257-server: detected 4 online CPUs by sysfs oleg257-server: Force libcfs to create 2 CPU partitions oleg257-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg257-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg257-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg257-server: oleg257-server.virtnet: executing set_default_debug -1 all pdsh@oleg257-client: oleg257-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg257-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg257-server: oleg257-server.virtnet: executing set_default_debug -1 all pdsh@oleg257-client: oleg257-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg257-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg257-client.virtnet: -o user_xattr,flock oleg257-server@tcp:/lustre /mnt/lustre setup single mount lustre success fail_loc=0x142 start ost2 service on oleg257-server Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg257-server: oleg257-server.virtnet: executing set_default_debug -1 all pdsh@oleg257-client: oleg257-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost2_flakey Started lustre-OST0001 oleg257-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec stop ost2 service on oleg257-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg257-server umount lustre on /mnt/lustre..... Stopping client oleg257-client.virtnet /mnt/lustre (opts:-f) stop ost1 service on oleg257-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg257-server stop mds service on oleg257-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg257-server unloading modules on: 'oleg257-server' oleg257-server: oleg257-server.virtnet: executing unload_modules_local modules unloaded. oleg257-server: tunefs.lustre: Unable to mount /dev/mapper/mds1_flakey: No such device oleg257-server: Is the ldiskfs module available? oleg257-server: oleg257-server: tunefs.lustre FATAL: failed to write local files oleg257-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg257-client: oleg257-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x5 (MDT MGS ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity Permanent disk data: Target: lustre=MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x105 (MDT MGS writeconf ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity oleg257-server: tunefs.lustre: Unable to mount /dev/mapper/ost1_flakey: No such device oleg257-server: Is the ldiskfs module available? oleg257-server: oleg257-server: tunefs.lustre FATAL: failed to write local files oleg257-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg257-client: oleg257-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x2 (OST ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.202.157@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x102 (OST writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.202.157@tcp sys.timeout=20 oleg257-server: tunefs.lustre: Unable to mount /dev/mapper/ost2_flakey: No such device oleg257-server: Is the ldiskfs module available? oleg257-server: oleg257-server: tunefs.lustre FATAL: failed to write local files oleg257-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg257-client: oleg257-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x2 (OST ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.202.157@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x102 (OST writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.202.157@tcp sys.timeout=20 tunefs failed, reformatting instead Stopping clients: oleg257-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg257-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg257-client: oleg257-server: ssh exited with exit code 2 oleg257-server: oleg257-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg257-server' oleg257-server: oleg257-server.virtnet: executing load_modules_local oleg257-server: Loading modules from /home/green/git/lustre-release/lustre oleg257-server: detected 4 online CPUs by sysfs oleg257-server: Force libcfs to create 2 CPU partitions oleg257-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg257-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg257-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg257-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg257-server: oleg257-server.virtnet: executing set_default_debug -1 all pdsh@oleg257-client: oleg257-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg257-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg257-server: oleg257-server.virtnet: executing set_default_debug -1 all pdsh@oleg257-client: oleg257-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg257-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg257-server: oleg257-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg257-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg257-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg257-server stop mds service on oleg257-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg257-server PASS 51 (164s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:46:56 (1709786816) start mds service on oleg257-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg257-server: oleg257-server.virtnet: executing set_default_debug -1 all pdsh@oleg257-client: oleg257-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg257-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg257-server: oleg257-server.virtnet: executing set_default_debug -1 all pdsh@oleg257-client: oleg257-server: ssh exited with exit code 1 Started lustre-OST0000 oleg257-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg257-client.virtnet: -o user_xattr,flock oleg257-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 234 (time 1709786850.71 total 10.76 last 21.75) - open/close 432 (time 1709786860.73 total 20.78 last 19.76) - open/close 588 (time 1709786871.58 total 31.63 last 14.38) - open/close 772 (time 1709786881.78 total 41.83 last 18.03) - open/close 1011 (time 1709786891.80 total 51.84 last 23.87) - open/close 1313 (time 1709786902.67 total 62.72 last 27.77) - open/close 1496 (time 1709786912.90 total 72.94 last 17.90) - open/close 1725 (time 1709786922.90 total 82.95 last 22.90) - open/close 1955 (time 1709786933.72 total 93.77 last 21.25) - open/close 2167 (time 1709786943.73 total 103.77 last 21.19) - open/close 2425 (time 1709786954.34 total 114.39 last 24.30) - open/close 2579 (time 1709786965.90 total 125.94 last 13.33) - open/close 2817 (time 1709786976.33 total 136.38 last 22.81) - open/close 3022 (time 1709786987.05 total 147.10 last 19.12) - open/close 3189 (time 1709786997.46 total 157.51 last 16.04) - open/close 3382 (time 1709787008.47 total 168.52 last 17.53) - open/close 3572 (time 1709787018.47 total 178.52 last 19.00) - open/close 3704 (time 1709787028.67 total 188.72 last 12.94) - open/close 3888 (time 1709787039.59 total 199.64 last 16.84) - open/close 4048 (time 1709787050.40 total 210.45 last 14.80) - open/close 4227 (time 1709787060.69 total 220.74 last 17.41) - open/close 4335 (time 1709787070.70 total 230.75 last 10.79) - open/close 4472 (time 1709787080.96 total 241.01 last 13.35) - open/close 4652 (time 1709787091.94 total 251.98 last 16.41) - open/close 4757 (time 1709787102.63 total 262.67 last 9.82) - open/close 4914 (time 1709787113.35 total 273.40 last 14.64) - open/close 5070 (time 1709787123.36 total 283.41 last 15.59) - open/close 5204 (time 1709787133.40 total 293.44 last 13.35) - open/close 5303 (time 1709787143.41 total 303.46 last 9.89) - open/close 5434 (time 1709787153.93 total 313.98 last 12.45) - open/close 5541 (time 1709787163.94 total 323.99 last 10.69) - open/close 5696 (time 1709787173.97 total 334.02 last 15.46) - open/close 5740 (time 1709787190.81 total 350.86 last 2.61) - open/close 5894 (time 1709787201.06 total 361.11 last 15.02) - open/close 6211 (time 1709787211.70 total 371.74 last 29.80) - open/close 6352 (time 1709787222.26 total 382.31 last 13.35) - open/close 6426 (time 1709787232.86 total 392.91 last 6.98) - open/close 6632 (time 1709787243.15 total 403.20 last 20.02) - open/close 6815 (time 1709787253.16 total 413.21 last 18.29) - open/close 7054 (time 1709787263.17 total 423.22 last 23.87) - open/close 7230 (time 1709787274.16 total 434.21 last 16.01) - open/close 7370 (time 1709787284.56 total 444.61 last 13.46) - open/close 7544 (time 1709787295.53 total 455.58 last 15.87) - open/close 7804 (time 1709787305.54 total 465.59 last 25.96) - open/close 7930 (time 1709787316.50 total 476.55 last 11.50) - open/close 8141 (time 1709787326.51 total 486.56 last 21.08) - open/close 8432 (time 1709787336.86 total 496.91 last 28.12) - open/close 8687 (time 1709787346.87 total 506.91 last 25.48) - open/close 8821 (time 1709787356.88 total 516.92 last 13.39) - open/close 9024 (time 1709787367.00 total 527.05 last 20.04) - open/close 9222 (time 1709787377.04 total 537.08 last 19.74) - open/close 9410 (time 1709787387.05 total 547.10 last 18.78) - open/close 9722 (time 1709787397.06 total 557.11 last 31.14) - open/close 10000 (time 1709787403.77 total 563.82 last 41.48) - open/close 10456 (time 1709787413.78 total 573.83 last 45.55) - open/close 10899 (time 1709787423.78 total 583.83 last 44.29) - open/close 11338 (time 1709787433.79 total 593.84 last 43.86) - open/close 11365 (time 1709787445.69 total 605.74 last 2.27) - open/close 11806 (time 1709787455.69 total 615.74 last 44.10) - open/close 12249 (time 1709787465.72 total 625.77 last 44.17) - open/close 12595 (time 1709787475.75 total 635.80 last 34.51) - open/close 13052 (time 1709787485.76 total 645.80 last 45.68) - open/close 13499 (time 1709787495.77 total 655.82 last 44.62) - open/close 13909 (time 1709787505.80 total 665.85 last 40.90) - open/close 14293 (time 1709787515.81 total 675.86 last 38.35) - open/close 14669 (time 1709787525.81 total 685.86 last 37.60) - open/close 15026 (time 1709787535.85 total 695.89 last 35.57) - open/close 15391 (time 1709787545.87 total 705.92 last 36.41) - open/close 15763 (time 1709787555.87 total 715.92 last 37.20) - open/close 16115 (time 1709787565.90 total 725.95 last 35.11) - open/close 16486 (time 1709787575.93 total 735.98 last 36.98) - open/close 16863 (time 1709787585.94 total 745.99 last 37.66) - open/close 17216 (time 1709787595.98 total 756.02 last 35.18) - open/close 17578 (time 1709787605.98 total 766.03 last 36.18) - open/close 17929 (time 1709787616.00 total 776.05 last 35.05) - open/close 18275 (time 1709787626.02 total 786.07 last 34.53) - open/close 18650 (time 1709787636.05 total 796.10 last 37.37) - open/close 19022 (time 1709787646.05 total 806.10 last 37.19) - open/close 19298 (time 1709787656.06 total 816.10 last 27.60) - open/close 19635 (time 1709787666.08 total 826.13 last 33.63) - open/close 19928 (time 1709787676.09 total 836.13 last 29.27) - open/close 20000 (time 1709787678.54 total 838.59 last 29.37) - open/close 20316 (time 1709787688.54 total 848.59 last 31.59) - open/close 20646 (time 1709787698.56 total 858.61 last 32.94) - open/close 20959 (time 1709787708.56 total 868.61 last 31.30) - open/close 21238 (time 1709787718.57 total 878.62 last 27.88) - open/close 21501 (time 1709787728.58 total 888.63 last 26.27) - open/close 21795 (time 1709787738.60 total 898.64 last 29.36) - open/close 22076 (time 1709787748.60 total 908.65 last 28.09) - open/close 22360 (time 1709787758.61 total 918.66 last 28.36) - open/close 22650 (time 1709787768.65 total 928.70 last 28.89) - open/close 22941 (time 1709787778.66 total 938.71 last 29.06) - open/close 23217 (time 1709787799.24 total 959.29 last 13.41) - open/close 23493 (time 1709787809.25 total 969.30 last 27.58) - open/close 23783 (time 1709787819.27 total 979.31 last 28.95) - open/close 24075 (time 1709787829.28 total 989.33 last 29.15) - open/close 24372 (time 1709787839.31 total 999.35 last 29.63) - open/close 24628 (time 1709787849.35 total 1009.40 last 25.48) - open/close 24878 (time 1709787859.35 total 1019.40 last 25.00) - open/close 25109 (time 1709787869.36 total 1029.41 last 23.07) - open/close 25386 (time 1709787879.37 total 1039.41 last 27.70) - open/close 25643 (time 1709787889.37 total 1049.42 last 25.68) - open/close 25881 (time 1709787899.41 total 1059.46 last 23.72) - open/close 26123 (time 1709787909.43 total 1069.48 last 24.15) - open/close 26388 (time 1709787919.44 total 1079.49 last 26.47) - open/close 26637 (time 1709787929.44 total 1089.49 last 24.89) - open/close 26906 (time 1709787939.44 total 1099.49 last 26.90) - open/close 27186 (time 1709787949.46 total 1109.50 last 27.96) - open/close 27504 (time 1709787959.46 total 1119.51 last 31.80) - open/close 27790 (time 1709787969.47 total 1129.52 last 28.56) - open/close 28085 (time 1709787979.48 total 1139.53 last 29.47) - open/close 28379 (time 1709787989.50 total 1149.55 last 29.36) - open/close 28706 (time 1709787999.50 total 1159.55 last 32.69) - open/close 29051 (time 1709788009.55 total 1169.60 last 34.35) - open/close 29424 (time 1709788019.55 total 1179.60 last 37.27) - open/close 29814 (time 1709788029.56 total 1189.61 last 38.96) - open/close 30000 (time 1709788034.26 total 1194.31 last 39.63) - open/close 30382 (time 1709788044.26 total 1204.31 last 38.18) - open/close 30762 (time 1709788054.28 total 1214.33 last 37.95) - open/close 31129 (time 1709788064.30 total 1224.35 last 36.60) - open/close 31474 (time 1709788074.32 total 1234.37 last 34.45) - open/close 31852 (time 1709788084.33 total 1244.38 last 37.74) - open/close 32212 (time 1709788094.34 total 1254.39 last 35.97) - open/close 32557 (time 1709788104.36 total 1264.41 last 34.44) - open/close 32905 (time 1709788114.36 total 1274.41 last 34.79) - open/close 33250 (time 1709788124.37 total 1284.42 last 34.47) - open/close 33588 (time 1709788134.40 total 1294.44 last 33.72) - open/close 33912 (time 1709788144.41 total 1304.46 last 32.35) - open/close 34237 (time 1709788154.43 total 1314.48 last 32.44) - open/close 34571 (time 1709788164.45 total 1324.50 last 33.33) - open/close 34908 (time 1709788174.46 total 1334.51 last 33.68) - open/close 35233 (time 1709788184.47 total 1344.52 last 32.45) - open/close 35559 (time 1709788194.47 total 1354.52 last 32.60) - open/close 35880 (time 1709788204.49 total 1364.54 last 32.03) - open/close 36206 (time 1709788214.50 total 1374.55 last 32.57) - open/close 36536 (time 1709788224.52 total 1384.57 last 32.95) - open/close 36858 (time 1709788234.54 total 1394.59 last 32.13) - open/close 37172 (time 1709788244.56 total 1404.61 last 31.33) - open/close 37488 (time 1709788254.58 total 1414.62 last 31.57) - open/close 37795 (time 1709788264.61 total 1424.65 last 30.61) - open/close 38103 (time 1709788274.61 total 1434.66 last 30.80) - open/close 38392 (time 1709788284.63 total 1444.68 last 28.84) - open/close 38667 (time 1709788294.64 total 1454.69 last 27.48) - open/close 38945 (time 1709788304.65 total 1464.70 last 27.76) - open/close 39221 (time 1709788314.68 total 1474.72 last 27.54) - open/close 39493 (time 1709788324.70 total 1484.75 last 27.14) - open/close 39757 (time 1709788334.73 total 1494.78 last 26.32) - open/close 40000 (time 1709788343.81 total 1503.86 last 26.75) - open/close 40279 (time 1709788353.84 total 1513.89 last 27.82) - open/close 40561 (time 1709788363.84 total 1523.89 last 28.19) - open/close 40838 (time 1709788373.85 total 1533.90 last 27.68) - open/close 41116 (time 1709788383.88 total 1543.93 last 27.71) - open/close 41375 (time 1709788393.91 total 1553.96 last 25.83) - open/close 41663 (time 1709788403.94 total 1563.99 last 28.72) - open/close 41947 (time 1709788413.96 total 1574.01 last 28.34) - open/close 42225 (time 1709788423.98 total 1584.03 last 27.74) - open/close 42508 (time 1709788433.99 total 1594.04 last 28.28) - open/close 42780 (time 1709788444.03 total 1604.07 last 27.10) - open/close 43033 (time 1709788454.03 total 1614.07 last 25.30) - open/close 43290 (time 1709788464.04 total 1624.09 last 25.65) - open/close 43465 (time 1709788482.05 total 1642.10 last 9.72) - open/close 43716 (time 1709788492.06 total 1652.10 last 25.09) - open/close 43978 (time 1709788502.06 total 1662.11 last 26.18) - open/close 44249 (time 1709788512.07 total 1672.12 last 27.08) - open/close 44543 (time 1709788522.10 total 1682.15 last 29.31) - open/close 44823 (time 1709788532.13 total 1692.18 last 27.93) - open/close 45106 (time 1709788542.16 total 1702.21 last 28.21) - open/close 45385 (time 1709788552.19 total 1712.24 last 27.82) - open/close 45663 (time 1709788562.22 total 1722.27 last 27.71) - open/close 45947 (time 1709788572.22 total 1732.27 last 28.39) - open/close 46225 (time 1709788582.23 total 1742.28 last 27.78) - open/close 46510 (time 1709788592.23 total 1752.28 last 28.49) - open/close 46771 (time 1709788602.23 total 1762.28 last 26.10) - open/close 47028 (time 1709788612.26 total 1772.31 last 25.64) - open/close 47288 (time 1709788622.28 total 1782.33 last 25.95) - open/close 47543 (time 1709788632.29 total 1792.34 last 25.46) - open/close 47791 (time 1709788642.33 total 1802.38 last 24.70) - open/close 48044 (time 1709788652.37 total 1812.42 last 25.20) - open/close 48295 (time 1709788662.41 total 1822.46 last 25.01) - open/close 48560 (time 1709788672.43 total 1832.48 last 26.44) - open/close 48819 (time 1709788682.46 total 1842.50 last 25.84) - open/close 49068 (time 1709788692.47 total 1852.52 last 24.86) - open/close 49325 (time 1709788702.47 total 1862.52 last 25.69) - open/close 49578 (time 1709788712.48 total 1872.53 last 25.28) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49698) error: No space left on device total: 49698 open/close in 1877.28 seconds: 26.47 ops/second - unlinked 0 (time 1709788718 ; total 0 ; last 0) - unlinked 10000 (time 1709789184 ; total 466 ; last 466) - unlinked 20000 (time 1709789581 ; total 863 ; last 397) - unlinked 30000 (time 1709789866 ; total 1148 ; last 285) - unlinked 40000 (time 1709790013 ; total 1295 ; last 147) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49698) error: No such file or directory total: 49698 unlinks in 1336 seconds: 37.199100 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg257-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg257-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg257-server stop mds service on oleg257-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg257-server unloading modules on: 'oleg257-server' oleg257-server: oleg257-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg257-client: oleg257-client: ssh exited with exit code 2 pdsh@oleg257-client: oleg257-server: ssh exited with exit code 2 pdsh@oleg257-client: oleg257-client: ssh exited with exit code 2 pdsh@oleg257-client: oleg257-server: ssh exited with exit code 2 PASS 101a (3252s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg257-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg257-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg257-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg257-client: oleg257-server: ssh exited with exit code 2 oleg257-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg257-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == conf-sanity test 101b: Race events DISCONNECT and ACTIVE in osp ========================================================== 00:41:10 (1709790070) start mds service on oleg257-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg257-server' oleg257-server: oleg257-server.virtnet: executing load_modules_local oleg257-server: Loading modules from /home/green/git/lustre-release/lustre oleg257-server: detected 4 online CPUs by sysfs oleg257-server: Force libcfs to create 2 CPU partitions oleg257-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg257-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg257-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg257-server: oleg257-server.virtnet: executing set_default_debug -1 all pdsh@oleg257-client: oleg257-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg257-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg257-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg257-server: oleg257-server.virtnet: executing set_default_debug -1 all pdsh@oleg257-client: oleg257-server: ssh exited with exit code 1 Started lustre-OST0000 oleg257-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg257-client.virtnet: -o user_xattr,flock oleg257-server@tcp:/lustre /mnt/lustre fail_loc=0x80002107 fail_val=20 stop ost1 service on oleg257-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg257-server start ost1 service on oleg257-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg257-server: oleg257-server.virtnet: executing set_default_debug -1 all pdsh@oleg257-client: oleg257-server: ssh exited with exit code 1 Started lustre-OST0000 oleg257-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 12 sec oleg257-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8800ad3c7000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800ad3c7000.ost_server_uuid in FULL state after 0 sec umount lustre on /mnt/lustre..... Stopping client oleg257-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg257-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg257-server stop mds service on oleg257-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg257-server unloading modules on: 'oleg257-server' oleg257-server: oleg257-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg257-client: oleg257-client: ssh exited with exit code 2 pdsh@oleg257-client: oleg257-server: ssh exited with exit code 2 pdsh@oleg257-client: oleg257-client: ssh exited with exit code 2 pdsh@oleg257-client: oleg257-server: ssh exited with exit code 2 PASS 101b (84s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg257-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg257-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory Stopping clients: oleg257-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg257-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg257-client: oleg257-server: ssh exited with exit code 2 oleg257-server: oleg257-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg257-server' oleg257-server: oleg257-server.virtnet: executing load_modules_local oleg257-server: Loading modules from /home/green/git/lustre-release/lustre oleg257-server: detected 4 online CPUs by sysfs oleg257-server: Force libcfs to create 2 CPU partitions oleg257-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg257-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey == conf-sanity test complete, duration 3793 sec ========== 00:42:51 (1709790171) === conf-sanity: start cleanup 00:42:52 (1709790172) === === conf-sanity: finish cleanup 00:42:52 (1709790172) ===