-----============= acceptance-small: conf-sanity ============----- Wed Mar 6 23:40:23 EST 2024 excepting tests: 32newtarball 110 skipping tests SLOW=no: 45 69 106 111 114 Stopping clients: oleg313-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg313-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg313-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg313-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg313-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg313-server oleg313-server: oleg313-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_refcnt: could not read integer from '/sys/module/acpi_cpufreq/refcnt': 'No such device' libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory loading modules on: 'oleg313-server' oleg313-server: oleg313-server.virtnet: executing load_modules_local oleg313-server: Loading modules from /home/green/git/lustre-release/lustre oleg313-server: detected 4 online CPUs by sysfs oleg313-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg313-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg313-server: oleg313-server.virtnet: executing set_default_debug -1 all pdsh@oleg313-client: oleg313-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 oleg313-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg313-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg313-server: oleg313-server.virtnet: executing set_default_debug -1 all pdsh@oleg313-client: oleg313-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg313-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg313-server: oleg313-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg313-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg313-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg313-server stop mds service on oleg313-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg313-server debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 35b: Continue reconnection retries, if the active server is busy ========================================================== 23:41:27 (1709786487) start mds service on oleg313-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg313-server: oleg313-server.virtnet: executing set_default_debug -1 all pdsh@oleg313-client: oleg313-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg313-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg313-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg313-server: oleg313-server.virtnet: executing set_default_debug -1 all pdsh@oleg313-client: oleg313-server: ssh exited with exit code 1 Started lustre-OST0000 oleg313-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg313-client.virtnet: -o user_xattr,flock oleg313-server@tcp:/lustre /mnt/lustre debug=ha conf-sanity.sh test_35b 2024-03-0623h41m43s Set up a fake failnode for the MDS at_max=0 at_max=0 Injecting EBUSY on MDS fail_loc=0x80000136 mdc.lustre-MDT0000-mdc-ffff8800aa4e2800.stats=clear Creating a test file and stat it File: '/mnt/lustre/d35b.conf-sanity/f35b.conf-sanity' Size: 0 Blocks: 0 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115238826934274 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-03-06 23:42:26.000000000 -0500 Modify: 2024-03-06 23:42:26.000000000 -0500 Change: 2024-03-06 23:42:26.000000000 -0500 Birth: - Stop injecting EBUSY on MDS fail_loc=0 done at_max=600 at_max=600 Debug log: 74 lines, 74 kept, 0 dropped, 0 bad. umount lustre on /mnt/lustre..... Stopping client oleg313-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg313-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg313-server stop mds service on oleg313-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg313-server unloading modules on: 'oleg313-server' oleg313-server: oleg313-server.virtnet: executing unload_modules_local modules unloaded. oleg313-server: tunefs.lustre: Unable to mount /dev/mapper/mds1_flakey: No such device oleg313-server: Is the ldiskfs module available? oleg313-server: oleg313-server: tunefs.lustre FATAL: failed to write local files oleg313-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg313-client: oleg313-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x5 (MDT MGS ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity Permanent disk data: Target: lustre=MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x105 (MDT MGS writeconf ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity oleg313-server: tunefs.lustre: Unable to mount /dev/mapper/ost1_flakey: No such device oleg313-server: Is the ldiskfs module available? oleg313-server: oleg313-server: tunefs.lustre FATAL: failed to write local files oleg313-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg313-client: oleg313-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x2 (OST ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.203.113@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x102 (OST writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.203.113@tcp sys.timeout=20 oleg313-server: tunefs.lustre: Unable to mount /dev/mapper/ost2_flakey: No such device oleg313-server: Is the ldiskfs module available? oleg313-server: oleg313-server: tunefs.lustre FATAL: failed to write local files oleg313-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg313-client: oleg313-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.203.113@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x162 (OST first_time update writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.203.113@tcp sys.timeout=20 tunefs failed, reformatting instead Stopping clients: oleg313-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg313-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg313-client: oleg313-server: ssh exited with exit code 2 oleg313-server: oleg313-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg313-server' oleg313-server: oleg313-server.virtnet: executing load_modules_local oleg313-server: Loading modules from /home/green/git/lustre-release/lustre oleg313-server: detected 4 online CPUs by sysfs oleg313-server: Force libcfs to create 2 CPU partitions oleg313-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg313-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg313-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg313-server: oleg313-server.virtnet: executing set_default_debug -1 all pdsh@oleg313-client: oleg313-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 oleg313-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg313-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg313-server: oleg313-server.virtnet: executing set_default_debug -1 all pdsh@oleg313-client: oleg313-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg313-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg313-server: oleg313-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg313-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg313-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg313-server stop mds service on oleg313-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg313-server PASS 35b (148s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 23:43:59 (1709786639) start mds service on oleg313-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg313-server: oleg313-server.virtnet: executing set_default_debug -1 all pdsh@oleg313-client: oleg313-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg313-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg313-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg313-server: oleg313-server.virtnet: executing set_default_debug -1 all pdsh@oleg313-client: oleg313-server: ssh exited with exit code 1 Started lustre-OST0000 oleg313-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg313-client.virtnet: -o user_xattr,flock oleg313-server@tcp:/lustre /mnt/lustre copying 10 files to /mnt/lustre/d38.conf-sanity tar: Removing leading `/' from member names umount lustre on /mnt/lustre..... Stopping client oleg313-client.virtnet /mnt/lustre (opts:) osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=44 stop mds service on oleg313-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg313-server delete lov_objid file on MDS 000000 43 000008 start mds service on oleg313-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg313-server: oleg313-server.virtnet: executing set_default_debug -1 all pdsh@oleg313-client: oleg313-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg313-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg313-client.virtnet: -o user_xattr,flock oleg313-server@tcp:/lustre /mnt/lustre osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=2 umount lustre on /mnt/lustre..... Stopping client oleg313-client.virtnet /mnt/lustre (opts:) stop mds service on oleg313-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg313-server 000000 2 000008 oleg313-server: 8+0 records in oleg313-server: 8+0 records out oleg313-server: 4096 bytes (4.1 kB) copied, 0.000409233 s, 10.0 MB/s start mds service on oleg313-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg313-server: oleg313-server.virtnet: executing set_default_debug -1 all pdsh@oleg313-client: oleg313-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg313-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg313-client.virtnet: -o user_xattr,flock oleg313-server@tcp:/lustre /mnt/lustre osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=35 umount lustre on /mnt/lustre..... Stopping client oleg313-client.virtnet /mnt/lustre (opts:) stop mds service on oleg313-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg313-server 000000 34 000008 files compared the same umount lustre on /mnt/lustre..... stop ost1 service on oleg313-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg313-server stop mds service on oleg313-server unloading modules on: 'oleg313-server' oleg313-server: oleg313-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg313-client: oleg313-client: ssh exited with exit code 2 pdsh@oleg313-client: oleg313-server: ssh exited with exit code 2 pdsh@oleg313-client: oleg313-client: ssh exited with exit code 2 pdsh@oleg313-client: oleg313-server: ssh exited with exit code 2 PASS 38 (77s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg313-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg313-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg313-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg313-client: oleg313-server: ssh exited with exit code 2 oleg313-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg313-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 23:45:19 (1709786719) Stopping clients: oleg313-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg313-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg313-client: oleg313-server: ssh exited with exit code 2 oleg313-server: oleg313-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg313-server' oleg313-server: oleg313-server.virtnet: executing load_modules_local oleg313-server: Loading modules from /home/green/git/lustre-release/lustre oleg313-server: detected 4 online CPUs by sysfs oleg313-server: Force libcfs to create 2 CPU partitions oleg313-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg313-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg313-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg313-server: oleg313-server.virtnet: executing set_default_debug -1 all pdsh@oleg313-client: oleg313-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 oleg313-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg313-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg313-server: oleg313-server.virtnet: executing set_default_debug -1 all pdsh@oleg313-client: oleg313-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg313-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg313-client.virtnet: -o user_xattr,flock oleg313-server@tcp:/lustre /mnt/lustre setup single mount lustre success fail_loc=0x142 start ost2 service on oleg313-server Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg313-server: oleg313-server.virtnet: executing set_default_debug -1 all pdsh@oleg313-client: oleg313-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost2_flakey Started lustre-OST0001 oleg313-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec stop ost2 service on oleg313-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg313-server umount lustre on /mnt/lustre..... Stopping client oleg313-client.virtnet /mnt/lustre (opts:-f) stop ost1 service on oleg313-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg313-server stop mds service on oleg313-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg313-server unloading modules on: 'oleg313-server' oleg313-server: oleg313-server.virtnet: executing unload_modules_local modules unloaded. oleg313-server: tunefs.lustre: Unable to mount /dev/mapper/mds1_flakey: No such device oleg313-server: Is the ldiskfs module available? oleg313-server: oleg313-server: tunefs.lustre FATAL: failed to write local files oleg313-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg313-client: oleg313-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x5 (MDT MGS ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity Permanent disk data: Target: lustre=MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x105 (MDT MGS writeconf ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity oleg313-server: tunefs.lustre: Unable to mount /dev/mapper/ost1_flakey: No such device oleg313-server: Is the ldiskfs module available? oleg313-server: oleg313-server: tunefs.lustre FATAL: failed to write local files oleg313-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg313-client: oleg313-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x2 (OST ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.203.113@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x102 (OST writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.203.113@tcp sys.timeout=20 oleg313-server: tunefs.lustre: Unable to mount /dev/mapper/ost2_flakey: No such device oleg313-server: Is the ldiskfs module available? oleg313-server: oleg313-server: tunefs.lustre FATAL: failed to write local files oleg313-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg313-client: oleg313-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x2 (OST ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.203.113@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x102 (OST writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.203.113@tcp sys.timeout=20 tunefs failed, reformatting instead Stopping clients: oleg313-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg313-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg313-client: oleg313-server: ssh exited with exit code 2 oleg313-server: oleg313-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg313-server' oleg313-server: oleg313-server.virtnet: executing load_modules_local oleg313-server: Loading modules from /home/green/git/lustre-release/lustre oleg313-server: detected 4 online CPUs by sysfs oleg313-server: Force libcfs to create 2 CPU partitions oleg313-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg313-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg313-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg313-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg313-server: oleg313-server.virtnet: executing set_default_debug -1 all pdsh@oleg313-client: oleg313-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 oleg313-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg313-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg313-server: oleg313-server.virtnet: executing set_default_debug -1 all pdsh@oleg313-client: oleg313-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg313-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg313-server: oleg313-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg313-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg313-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg313-server stop mds service on oleg313-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg313-server PASS 51 (153s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:47:55 (1709786875) start mds service on oleg313-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg313-server: oleg313-server.virtnet: executing set_default_debug -1 all pdsh@oleg313-client: oleg313-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg313-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg313-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg313-server: oleg313-server.virtnet: executing set_default_debug -1 all pdsh@oleg313-client: oleg313-server: ssh exited with exit code 1 Started lustre-OST0000 oleg313-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg313-client.virtnet: -o user_xattr,flock oleg313-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 209 (time 1709786905.37 total 10.01 last 20.88) - open/close 474 (time 1709786915.99 total 20.64 last 24.93) - open/close 724 (time 1709786926.66 total 31.30 last 23.44) - open/close 915 (time 1709786937.57 total 42.21 last 17.50) - open/close 1154 (time 1709786948.34 total 52.98 last 22.19) - open/close 1292 (time 1709786958.36 total 63.00 last 13.77) - open/close 1509 (time 1709786968.86 total 73.51 last 20.66) - open/close 1656 (time 1709786979.48 total 84.12 last 13.85) - open/close 1879 (time 1709786989.48 total 94.12 last 22.29) - open/close 2030 (time 1709787000.69 total 105.33 last 13.47) - open/close 2304 (time 1709787011.36 total 116.00 last 25.68) - open/close 2480 (time 1709787021.36 total 126.01 last 17.60) - open/close 2627 (time 1709787031.61 total 136.25 last 14.35) - open/close 2800 (time 1709787042.16 total 146.80 last 16.40) - open/close 3037 (time 1709787052.16 total 156.80 last 23.68) - open/close 3203 (time 1709787062.17 total 166.81 last 16.59) - open/close 3365 (time 1709787072.49 total 177.13 last 15.69) - open/close 3481 (time 1709787082.49 total 187.14 last 11.60) - open/close 3730 (time 1709787093.29 total 197.93 last 23.07) - open/close 3919 (time 1709787103.32 total 207.96 last 18.84) - open/close 4071 (time 1709787113.46 total 218.10 last 14.99) - open/close 4253 (time 1709787123.57 total 228.21 last 18.00) - open/close 4386 (time 1709787133.84 total 238.49 last 12.94) - open/close 4570 (time 1709787143.99 total 248.63 last 18.13) - open/close 4574 (time 1709787155.13 total 259.78 last 0.36) - open/close 4680 (time 1709787165.14 total 269.78 last 10.59) - open/close 4834 (time 1709787175.69 total 280.34 last 14.59) - open/close 4973 (time 1709787186.66 total 291.30 last 12.68) - open/close 5178 (time 1709787196.66 total 301.30 last 20.49) - open/close 5353 (time 1709787207.06 total 311.70 last 16.83) - open/close 5443 (time 1709787219.83 total 324.47 last 7.05) - open/close 5706 (time 1709787230.83 total 335.47 last 23.90) - open/close 5845 (time 1709787241.37 total 346.01 last 13.19) - open/close 5998 (time 1709787251.57 total 356.21 last 15.00) - open/close 6228 (time 1709787262.59 total 367.23 last 20.86) - open/close 6463 (time 1709787272.68 total 377.32 last 23.30) - open/close 6596 (time 1709787282.68 total 387.33 last 13.29) - open/close 6796 (time 1709787292.69 total 397.33 last 20.00) - open/close 6930 (time 1709787303.65 total 408.29 last 12.22) - open/close 7072 (time 1709787314.10 total 418.74 last 13.59) - open/close 7284 (time 1709787334.48 total 439.12 last 10.40) - open/close 7465 (time 1709787344.54 total 449.18 last 17.99) - open/close 7590 (time 1709787355.19 total 459.83 last 11.74) - open/close 7742 (time 1709787365.42 total 470.06 last 14.85) - open/close 7911 (time 1709787376.03 total 480.67 last 15.94) - open/close 8065 (time 1709787386.06 total 490.70 last 15.35) - open/close 8235 (time 1709787396.11 total 500.76 last 16.91) - open/close 8458 (time 1709787406.13 total 510.77 last 22.26) - open/close 8635 (time 1709787416.88 total 521.52 last 16.47) - open/close 8849 (time 1709787427.29 total 531.93 last 20.55) - open/close 8987 (time 1709787437.30 total 541.94 last 13.79) - open/close 9182 (time 1709787447.92 total 552.57 last 18.35) - open/close 9330 (time 1709787458.03 total 562.67 last 14.64) - open/close 9556 (time 1709787468.08 total 572.72 last 22.49) - open/close 9725 (time 1709787478.91 total 583.55 last 15.61) - open/close 9847 (time 1709787493.47 total 598.11 last 8.38) - open/close 10000 (time 1709787496.66 total 601.31 last 47.93) - open/close 10467 (time 1709787506.68 total 611.32 last 46.65) - open/close 10929 (time 1709787516.70 total 621.34 last 46.10) - open/close 11367 (time 1709787526.70 total 631.35 last 43.76) - open/close 11811 (time 1709787536.71 total 641.35 last 44.39) - open/close 11823 (time 1709787548.05 total 652.70 last 1.06) - open/close 12233 (time 1709787558.09 total 662.73 last 40.85) - open/close 12596 (time 1709787568.11 total 672.75 last 36.24) - open/close 12967 (time 1709787578.13 total 682.77 last 37.01) - open/close 13325 (time 1709787588.15 total 692.79 last 35.74) - open/close 13653 (time 1709787598.19 total 702.83 last 32.65) - open/close 13999 (time 1709787608.19 total 712.84 last 34.60) - open/close 14362 (time 1709787618.21 total 722.85 last 36.23) - open/close 14717 (time 1709787628.23 total 732.87 last 35.44) - open/close 15085 (time 1709787638.24 total 742.88 last 36.77) - open/close 15402 (time 1709787648.24 total 752.88 last 31.69) - open/close 15772 (time 1709787658.25 total 762.89 last 36.96) - open/close 16108 (time 1709787668.27 total 772.91 last 33.54) - open/close 16438 (time 1709787678.29 total 782.93 last 32.92) - open/close 16772 (time 1709787688.30 total 792.95 last 33.35) - open/close 17088 (time 1709787698.37 total 803.01 last 31.39) - open/close 17353 (time 1709787708.41 total 813.05 last 26.39) - open/close 17667 (time 1709787718.46 total 823.10 last 31.24) - open/close 18002 (time 1709787728.48 total 833.12 last 33.44) - open/close 18295 (time 1709787738.51 total 843.16 last 29.21) - open/close 18567 (time 1709787748.54 total 853.19 last 27.12) - open/close 18860 (time 1709787758.55 total 863.19 last 29.29) - open/close 19176 (time 1709787768.55 total 873.19 last 31.59) - open/close 19486 (time 1709787778.57 total 883.21 last 30.93) - open/close 19796 (time 1709787788.58 total 893.23 last 30.97) - open/close 20000 (time 1709787795.30 total 899.94 last 30.36) - open/close 20310 (time 1709787805.33 total 909.97 last 30.91) - open/close 20651 (time 1709787815.35 total 919.99 last 34.03) - open/close 20954 (time 1709787825.38 total 930.02 last 30.22) - open/close 21308 (time 1709787835.40 total 940.04 last 35.33) - open/close 21610 (time 1709787845.41 total 950.05 last 30.17) - open/close 21929 (time 1709787855.45 total 960.09 last 31.78) - open/close 22188 (time 1709787865.47 total 970.11 last 25.84) - open/close 22489 (time 1709787875.51 total 980.15 last 30.00) - open/close 22768 (time 1709787885.51 total 990.15 last 27.89) - open/close 23047 (time 1709787895.53 total 1000.17 last 27.85) - open/close 23348 (time 1709787905.54 total 1010.18 last 30.05) - open/close 23659 (time 1709787915.56 total 1020.20 last 31.05) - open/close 23959 (time 1709787925.56 total 1030.20 last 29.99) - open/close 24236 (time 1709787935.60 total 1040.24 last 27.60) - open/close 24520 (time 1709787945.61 total 1050.25 last 28.36) - open/close 24786 (time 1709787955.63 total 1060.27 last 26.55) - open/close 25054 (time 1709787965.65 total 1070.30 last 26.73) - open/close 25317 (time 1709787975.67 total 1080.31 last 26.25) - open/close 25617 (time 1709787985.68 total 1090.32 last 29.99) - open/close 25917 (time 1709787995.68 total 1100.32 last 29.98) - open/close 26212 (time 1709788005.70 total 1110.34 last 29.44) - open/close 26502 (time 1709788015.71 total 1120.35 last 28.99) - open/close 26778 (time 1709788025.73 total 1130.37 last 27.55) - open/close 27041 (time 1709788035.75 total 1140.39 last 26.24) - open/close 27308 (time 1709788045.77 total 1150.41 last 26.64) - open/close 27571 (time 1709788055.80 total 1160.44 last 26.23) - open/close 27829 (time 1709788065.84 total 1170.48 last 25.68) - open/close 28104 (time 1709788075.86 total 1180.50 last 27.46) - open/close 28457 (time 1709788085.87 total 1190.51 last 35.27) - open/close 28809 (time 1709788095.87 total 1200.51 last 35.18) - open/close 29150 (time 1709788105.89 total 1210.53 last 34.04) - open/close 29489 (time 1709788115.89 total 1220.54 last 33.88) - open/close 29834 (time 1709788125.90 total 1230.54 last 34.47) - open/close 30000 (time 1709788130.60 total 1235.24 last 35.36) - open/close 30358 (time 1709788140.62 total 1245.26 last 35.72) - open/close 30701 (time 1709788150.62 total 1255.26 last 34.29) - open/close 31032 (time 1709788160.64 total 1265.28 last 33.03) - open/close 31374 (time 1709788170.66 total 1275.30 last 34.13) - open/close 31700 (time 1709788180.69 total 1285.33 last 32.50) - open/close 32024 (time 1709788190.69 total 1295.34 last 32.38) - open/close 32332 (time 1709788200.70 total 1305.34 last 30.80) - open/close 32646 (time 1709788210.70 total 1315.34 last 31.39) - open/close 32964 (time 1709788220.70 total 1325.34 last 31.79) - open/close 33299 (time 1709788230.72 total 1335.37 last 33.43) - open/close 33608 (time 1709788240.73 total 1345.37 last 30.90) - open/close 33918 (time 1709788250.74 total 1355.38 last 30.97) - open/close 34204 (time 1709788260.74 total 1365.38 last 28.58) - open/close 34504 (time 1709788270.75 total 1375.39 last 29.99) - open/close 34797 (time 1709788280.77 total 1385.41 last 29.23) - open/close 35075 (time 1709788290.79 total 1395.43 last 27.73) - open/close 35352 (time 1709788300.83 total 1405.47 last 27.61) - open/close 35632 (time 1709788310.84 total 1415.48 last 27.96) - open/close 35915 (time 1709788320.85 total 1425.49 last 28.29) - open/close 35994 (time 1709788334.77 total 1439.41 last 5.67) - open/close 36294 (time 1709788344.80 total 1449.44 last 29.91) - open/close 36596 (time 1709788354.83 total 1459.47 last 30.11) - open/close 36882 (time 1709788364.86 total 1469.50 last 28.52) - open/close 37170 (time 1709788374.87 total 1479.51 last 28.76) - open/close 37457 (time 1709788384.90 total 1489.54 last 28.63) - open/close 37741 (time 1709788394.90 total 1499.54 last 28.40) - open/close 38031 (time 1709788404.93 total 1509.57 last 28.91) - open/close 38304 (time 1709788414.96 total 1519.60 last 27.22) - open/close 38602 (time 1709788424.98 total 1529.62 last 29.73) - open/close 38877 (time 1709788435.01 total 1539.65 last 27.42) - open/close 39147 (time 1709788445.01 total 1549.66 last 27.00) - open/close 39411 (time 1709788455.03 total 1559.67 last 26.36) - open/close 39679 (time 1709788465.05 total 1569.69 last 26.75) - open/close 39956 (time 1709788475.07 total 1579.71 last 27.63) - open/close 40223 (time 1709788485.11 total 1589.75 last 26.60) - open/close 40477 (time 1709788495.11 total 1599.75 last 25.40) - open/close 40749 (time 1709788505.13 total 1609.77 last 27.14) - open/close 41014 (time 1709788515.14 total 1619.78 last 26.49) - open/close 41282 (time 1709788525.17 total 1629.81 last 26.71) - open/close 41549 (time 1709788535.19 total 1639.84 last 26.63) - open/close 41818 (time 1709788545.21 total 1649.85 last 26.86) - open/close 42077 (time 1709788555.23 total 1659.87 last 25.84) - open/close 42338 (time 1709788565.24 total 1669.88 last 26.08) - open/close 42614 (time 1709788575.26 total 1679.90 last 27.53) - open/close 42887 (time 1709788585.27 total 1689.91 last 27.27) - open/close 43151 (time 1709788595.29 total 1699.93 last 26.35) - open/close 43425 (time 1709788605.31 total 1709.95 last 27.35) - open/close 43695 (time 1709788615.34 total 1719.98 last 26.92) - open/close 43966 (time 1709788625.36 total 1730.00 last 27.04) - open/close 44222 (time 1709788635.38 total 1740.02 last 25.55) - open/close 44457 (time 1709788645.41 total 1750.05 last 23.43) - open/close 44682 (time 1709788655.45 total 1760.09 last 22.41) - open/close 44919 (time 1709788665.48 total 1770.12 last 23.62) - open/close 45163 (time 1709788675.50 total 1780.15 last 24.35) - open/close 45397 (time 1709788685.51 total 1790.15 last 23.39) - open/close 45639 (time 1709788695.52 total 1800.16 last 24.16) - open/close 45878 (time 1709788705.56 total 1810.20 last 23.82) - open/close 46115 (time 1709788715.61 total 1820.26 last 23.57) - open/close 46368 (time 1709788725.63 total 1830.28 last 25.25) - open/close 46632 (time 1709788735.65 total 1840.30 last 26.35) - open/close 46901 (time 1709788745.66 total 1850.30 last 26.89) - open/close 47149 (time 1709788755.68 total 1860.33 last 24.74) - open/close 47379 (time 1709788765.69 total 1870.33 last 23.00) - open/close 47593 (time 1709788775.71 total 1880.35 last 21.35) - open/close 47819 (time 1709788785.72 total 1890.36 last 22.58) - open/close 48048 (time 1709788795.75 total 1900.39 last 22.82) - open/close 48264 (time 1709788805.79 total 1910.43 last 21.51) - open/close 48489 (time 1709788815.82 total 1920.46 last 22.44) - open/close 48706 (time 1709788825.83 total 1930.47 last 21.68) - open/close 48935 (time 1709788835.86 total 1940.50 last 22.82) - open/close 49161 (time 1709788845.91 total 1950.55 last 22.49) - open/close 49374 (time 1709788855.91 total 1960.55 last 21.30) - open/close 49579 (time 1709788865.96 total 1970.60 last 20.40) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49698) error: No space left on device total: 49698 open/close in 1976.07 seconds: 25.15 ops/second - unlinked 0 (time 1709788872 ; total 0 ; last 0) - unlinked 10000 (time 1709789367 ; total 495 ; last 495) - unlinked 20000 (time 1709789808 ; total 936 ; last 441) - unlinked 30000 (time 1709790121 ; total 1249 ; last 313) - unlinked 40000 (time 1709790285 ; total 1413 ; last 164) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49698) error: No such file or directory total: 49698 unlinks in 1462 seconds: 33.993160 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg313-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg313-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg313-server stop mds service on oleg313-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg313-server unloading modules on: 'oleg313-server' oleg313-server: oleg313-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg313-client: oleg313-server: ssh exited with exit code 2 pdsh@oleg313-client: oleg313-client: ssh exited with exit code 2 pdsh@oleg313-client: oleg313-client: ssh exited with exit code 2 pdsh@oleg313-client: oleg313-server: ssh exited with exit code 2 PASS 101a (3474s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg313-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg313-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg313-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg313-client: oleg313-server: ssh exited with exit code 2 oleg313-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg313-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == conf-sanity test 101b: Race events DISCONNECT and ACTIVE in osp ========================================================== 00:45:51 (1709790351) start mds service on oleg313-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg313-server' oleg313-server: oleg313-server.virtnet: executing load_modules_local oleg313-server: Loading modules from /home/green/git/lustre-release/lustre oleg313-server: detected 4 online CPUs by sysfs oleg313-server: Force libcfs to create 2 CPU partitions oleg313-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg313-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg313-server: oleg313-server.virtnet: executing set_default_debug -1 all pdsh@oleg313-client: oleg313-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg313-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg313-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg313-server: oleg313-server.virtnet: executing set_default_debug -1 all pdsh@oleg313-client: oleg313-server: ssh exited with exit code 1 Started lustre-OST0000 oleg313-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg313-client.virtnet: -o user_xattr,flock oleg313-server@tcp:/lustre /mnt/lustre fail_loc=0x80002107 fail_val=20 stop ost1 service on oleg313-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg313-server start ost1 service on oleg313-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg313-server: oleg313-server.virtnet: executing set_default_debug -1 all pdsh@oleg313-client: oleg313-server: ssh exited with exit code 1 Started lustre-OST0000 oleg313-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 7 sec oleg313-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8800acb7a800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800acb7a800.ost_server_uuid in FULL state after 0 sec umount lustre on /mnt/lustre..... Stopping client oleg313-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg313-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg313-server stop mds service on oleg313-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg313-server unloading modules on: 'oleg313-server' oleg313-server: oleg313-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg313-client: oleg313-client: ssh exited with exit code 2 pdsh@oleg313-client: oleg313-server: ssh exited with exit code 2 pdsh@oleg313-client: oleg313-client: ssh exited with exit code 2 pdsh@oleg313-client: oleg313-server: ssh exited with exit code 2 PASS 101b (86s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg313-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg313-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory Stopping clients: oleg313-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg313-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg313-client: oleg313-server: ssh exited with exit code 2 oleg313-server: oleg313-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg313-server' oleg313-server: oleg313-server.virtnet: executing load_modules_local oleg313-server: Loading modules from /home/green/git/lustre-release/lustre oleg313-server: detected 4 online CPUs by sysfs oleg313-server: Force libcfs to create 2 CPU partitions oleg313-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg313-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey == conf-sanity test complete, duration 4030 sec ========== 00:47:34 (1709790454) === conf-sanity: start cleanup 00:47:35 (1709790455) === === conf-sanity: finish cleanup 00:47:35 (1709790455) ===