-----============= acceptance-small: conf-sanity ============----- Wed Mar 6 23:16:11 EST 2024 excepting tests: 32newtarball 110 skipping tests SLOW=no: 45 69 106 111 114 Stopping clients: oleg125-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg125-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg125-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg125-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg125-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg125-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg125-server oleg125-server: oleg125-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg125-server' oleg125-server: oleg125-server.virtnet: executing load_modules_local oleg125-server: Loading modules from /home/green/git/lustre-release/lustre oleg125-server: detected 4 online CPUs by sysfs oleg125-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg125-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg125-server: oleg125-server.virtnet: executing set_default_debug -1 all pdsh@oleg125-client: oleg125-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg125-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg125-server: oleg125-server.virtnet: executing set_default_debug -1 all pdsh@oleg125-client: oleg125-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg125-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg125-server: oleg125-server.virtnet: executing set_default_debug -1 all pdsh@oleg125-client: oleg125-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg125-server: oleg125-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg125-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg125-server: oleg125-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg125-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg125-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg125-server stop mds service on oleg125-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg125-server stop mds service on oleg125-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg125-server debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 35b: Continue reconnection retries, if the active server is busy ========================================================== 23:17:34 (1709785054) start mds service on oleg125-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg125-server: oleg125-server.virtnet: executing set_default_debug -1 all pdsh@oleg125-client: oleg125-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg125-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg125-server: oleg125-server.virtnet: executing set_default_debug -1 all pdsh@oleg125-client: oleg125-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg125-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg125-server: oleg125-server.virtnet: executing set_default_debug -1 all pdsh@oleg125-client: oleg125-server: ssh exited with exit code 1 Started lustre-OST0000 oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg125-client.virtnet: -o user_xattr,flock oleg125-server@tcp:/lustre /mnt/lustre debug=ha conf-sanity.sh test_35b 2024-03-0623h17m53s Set up a fake failnode for the MDS at_max=0 at_max=0 Injecting EBUSY on MDS fail_loc=0x80000136 mdc.lustre-MDT0000-mdc-ffff8800b0a05800.stats=clear mdc.lustre-MDT0001-mdc-ffff8800b0a05800.stats=clear Creating a test file and stat it File: '/mnt/lustre/d35b.conf-sanity/f35b.conf-sanity' Size: 0 Blocks: 0 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115238826934274 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-03-06 23:18:36.000000000 -0500 Modify: 2024-03-06 23:18:36.000000000 -0500 Change: 2024-03-06 23:18:36.000000000 -0500 Birth: - Stop injecting EBUSY on MDS fail_loc=0 done at_max=600 at_max=600 Debug log: 63 lines, 63 kept, 0 dropped, 0 bad. umount lustre on /mnt/lustre..... Stopping client oleg125-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg125-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg125-server stop mds service on oleg125-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg125-server stop mds service on oleg125-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg125-server unloading modules on: 'oleg125-server' oleg125-server: oleg125-server.virtnet: executing unload_modules_local modules unloaded. oleg125-server: tunefs.lustre: Unable to mount /dev/mapper/mds1_flakey: No such device oleg125-server: Is the ldiskfs module available? oleg125-server: oleg125-server: tunefs.lustre FATAL: failed to write local files oleg125-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg125-client: oleg125-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x5 (MDT MGS ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity Permanent disk data: Target: lustre=MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x105 (MDT MGS writeconf ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity oleg125-server: tunefs.lustre: Unable to mount /dev/mapper/mds2_flakey: No such device oleg125-server: Is the ldiskfs module available? oleg125-server: oleg125-server: tunefs.lustre FATAL: failed to write local files oleg125-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg125-client: oleg125-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-MDT0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x1 (MDT ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: mgsnode=192.168.201.125@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity Permanent disk data: Target: lustre=MDT0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x101 (MDT writeconf ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: mgsnode=192.168.201.125@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity oleg125-server: tunefs.lustre: Unable to mount /dev/mapper/ost1_flakey: No such device oleg125-server: Is the ldiskfs module available? oleg125-server: oleg125-server: tunefs.lustre FATAL: failed to write local files oleg125-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg125-client: oleg125-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x2 (OST ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.201.125@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x102 (OST writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.201.125@tcp sys.timeout=20 oleg125-server: tunefs.lustre: Unable to mount /dev/mapper/ost2_flakey: No such device oleg125-server: Is the ldiskfs module available? oleg125-server: oleg125-server: tunefs.lustre FATAL: failed to write local files oleg125-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg125-client: oleg125-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.201.125@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x162 (OST first_time update writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.201.125@tcp sys.timeout=20 tunefs failed, reformatting instead Stopping clients: oleg125-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg125-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg125-client: oleg125-server: ssh exited with exit code 2 oleg125-server: oleg125-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg125-server' oleg125-server: oleg125-server.virtnet: executing load_modules_local oleg125-server: Loading modules from /home/green/git/lustre-release/lustre oleg125-server: detected 4 online CPUs by sysfs oleg125-server: Force libcfs to create 2 CPU partitions oleg125-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg125-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg125-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg125-server: oleg125-server.virtnet: executing set_default_debug -1 all pdsh@oleg125-client: oleg125-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg125-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg125-server: oleg125-server.virtnet: executing set_default_debug -1 all pdsh@oleg125-client: oleg125-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg125-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg125-server: oleg125-server.virtnet: executing set_default_debug -1 all pdsh@oleg125-client: oleg125-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg125-server: oleg125-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg125-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec oleg125-server: oleg125-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg125-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg125-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg125-server stop mds service on oleg125-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg125-server stop mds service on oleg125-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg125-server PASS 35b (141s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 23:19:57 (1709785197) start mds service on oleg125-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg125-server: oleg125-server.virtnet: executing set_default_debug -1 all pdsh@oleg125-client: oleg125-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg125-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg125-server: oleg125-server.virtnet: executing set_default_debug -1 all pdsh@oleg125-client: oleg125-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg125-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg125-server: oleg125-server.virtnet: executing set_default_debug -1 all pdsh@oleg125-client: oleg125-server: ssh exited with exit code 1 Started lustre-OST0000 oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg125-client.virtnet: -o user_xattr,flock oleg125-server@tcp:/lustre /mnt/lustre copying 10 files to /mnt/lustre/d38.conf-sanity tar: Removing leading `/' from member names umount lustre on /mnt/lustre..... Stopping client oleg125-client.virtnet /mnt/lustre (opts:) osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=44 osp.lustre-OST0000-osc-MDT0001.prealloc_next_id=34 stop mds service on oleg125-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg125-server stop mds service on oleg125-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg125-server delete lov_objid file on MDS 000000 43 000008 start mds service on oleg125-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg125-server: oleg125-server.virtnet: executing set_default_debug -1 all pdsh@oleg125-client: oleg125-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg125-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg125-server: oleg125-server.virtnet: executing set_default_debug -1 all pdsh@oleg125-client: oleg125-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg125-client.virtnet: -o user_xattr,flock oleg125-server@tcp:/lustre /mnt/lustre osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=2 osp.lustre-OST0000-osc-MDT0001.prealloc_next_id=66 umount lustre on /mnt/lustre..... Stopping client oleg125-client.virtnet /mnt/lustre (opts:) stop mds service on oleg125-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg125-server stop mds service on oleg125-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg125-server 000000 2 000008 oleg125-server: 8+0 records in oleg125-server: 8+0 records out oleg125-server: 4096 bytes (4.1 kB) copied, 0.000357104 s, 11.5 MB/s start mds service on oleg125-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg125-server: oleg125-server.virtnet: executing set_default_debug -1 all pdsh@oleg125-client: oleg125-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg125-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg125-server: oleg125-server.virtnet: executing set_default_debug -1 all pdsh@oleg125-client: oleg125-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg125-client.virtnet: -o user_xattr,flock oleg125-server@tcp:/lustre /mnt/lustre osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=35 osp.lustre-OST0000-osc-MDT0001.prealloc_next_id=98 umount lustre on /mnt/lustre..... Stopping client oleg125-client.virtnet /mnt/lustre (opts:) stop mds service on oleg125-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg125-server stop mds service on oleg125-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg125-server 000000 34 000008 files compared the same umount lustre on /mnt/lustre..... stop ost1 service on oleg125-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg125-server stop mds service on oleg125-server stop mds service on oleg125-server unloading modules on: 'oleg125-server' oleg125-server: oleg125-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg125-client: oleg125-client: ssh exited with exit code 2 pdsh@oleg125-client: oleg125-server: ssh exited with exit code 2 pdsh@oleg125-client: oleg125-client: ssh exited with exit code 2 pdsh@oleg125-client: oleg125-server: ssh exited with exit code 2 PASS 38 (105s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg125-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg125-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg125-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg125-client: oleg125-server: ssh exited with exit code 2 oleg125-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg125-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 23:21:44 (1709785304) Stopping clients: oleg125-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg125-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg125-client: oleg125-server: ssh exited with exit code 2 oleg125-server: oleg125-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg125-server' oleg125-server: oleg125-server.virtnet: executing load_modules_local oleg125-server: Loading modules from /home/green/git/lustre-release/lustre oleg125-server: detected 4 online CPUs by sysfs oleg125-server: Force libcfs to create 2 CPU partitions oleg125-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg125-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg125-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg125-server: oleg125-server.virtnet: executing set_default_debug -1 all pdsh@oleg125-client: oleg125-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg125-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg125-server: oleg125-server.virtnet: executing set_default_debug -1 all pdsh@oleg125-client: oleg125-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg125-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg125-server: oleg125-server.virtnet: executing set_default_debug -1 all pdsh@oleg125-client: oleg125-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg125-client.virtnet: -o user_xattr,flock oleg125-server@tcp:/lustre /mnt/lustre setup single mount lustre success fail_loc=0x142 start ost2 service on oleg125-server Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg125-server: oleg125-server.virtnet: executing set_default_debug -1 all pdsh@oleg125-client: oleg125-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost2_flakey Started lustre-OST0001 oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 1 sec stop ost2 service on oleg125-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg125-server umount lustre on /mnt/lustre..... Stopping client oleg125-client.virtnet /mnt/lustre (opts:-f) stop ost1 service on oleg125-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg125-server stop mds service on oleg125-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg125-server stop mds service on oleg125-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg125-server unloading modules on: 'oleg125-server' oleg125-server: oleg125-server.virtnet: executing unload_modules_local modules unloaded. oleg125-server: tunefs.lustre: Unable to mount /dev/mapper/mds1_flakey: No such device oleg125-server: Is the ldiskfs module available? oleg125-server: oleg125-server: tunefs.lustre FATAL: failed to write local files oleg125-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg125-client: oleg125-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x5 (MDT MGS ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity Permanent disk data: Target: lustre=MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x105 (MDT MGS writeconf ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity oleg125-server: tunefs.lustre: Unable to mount /dev/mapper/mds2_flakey: No such device oleg125-server: Is the ldiskfs module available? oleg125-server: oleg125-server: tunefs.lustre FATAL: failed to write local files oleg125-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg125-client: oleg125-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-MDT0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x1 (MDT ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: mgsnode=192.168.201.125@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity Permanent disk data: Target: lustre=MDT0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x101 (MDT writeconf ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: mgsnode=192.168.201.125@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity oleg125-server: tunefs.lustre: Unable to mount /dev/mapper/ost1_flakey: No such device oleg125-server: Is the ldiskfs module available? oleg125-server: oleg125-server: tunefs.lustre FATAL: failed to write local files oleg125-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg125-client: oleg125-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x2 (OST ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.201.125@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x102 (OST writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.201.125@tcp sys.timeout=20 oleg125-server: tunefs.lustre: Unable to mount /dev/mapper/ost2_flakey: No such device oleg125-server: Is the ldiskfs module available? oleg125-server: oleg125-server: tunefs.lustre FATAL: failed to write local files oleg125-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg125-client: oleg125-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x2 (OST ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.201.125@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x102 (OST writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.201.125@tcp sys.timeout=20 tunefs failed, reformatting instead Stopping clients: oleg125-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg125-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg125-client: oleg125-server: ssh exited with exit code 2 oleg125-server: oleg125-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg125-server' oleg125-server: oleg125-server.virtnet: executing load_modules_local oleg125-server: Loading modules from /home/green/git/lustre-release/lustre oleg125-server: detected 4 online CPUs by sysfs oleg125-server: Force libcfs to create 2 CPU partitions oleg125-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg125-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg125-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg125-server: oleg125-server.virtnet: executing set_default_debug -1 all pdsh@oleg125-client: oleg125-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg125-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg125-server: oleg125-server.virtnet: executing set_default_debug -1 all pdsh@oleg125-client: oleg125-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg125-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg125-server: oleg125-server.virtnet: executing set_default_debug -1 all pdsh@oleg125-client: oleg125-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg125-server: oleg125-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg125-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg125-server: oleg125-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg125-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg125-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg125-server stop mds service on oleg125-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg125-server stop mds service on oleg125-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg125-server PASS 51 (152s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:24:18 (1709785458) start mds service on oleg125-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg125-server: oleg125-server.virtnet: executing set_default_debug -1 all pdsh@oleg125-client: oleg125-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg125-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg125-server: oleg125-server.virtnet: executing set_default_debug -1 all pdsh@oleg125-client: oleg125-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg125-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg125-server: oleg125-server.virtnet: executing set_default_debug -1 all pdsh@oleg125-client: oleg125-server: ssh exited with exit code 1 Started lustre-OST0000 oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg125-client.virtnet: -o user_xattr,flock oleg125-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 770 (time 1709785487.72 total 10.53 last 73.13) - open/close 1413 (time 1709785498.23 total 21.04 last 61.14) - open/close 2023 (time 1709785508.94 total 31.75 last 56.96) - open/close 2640 (time 1709785519.40 total 42.21 last 59.01) - open/close 3260 (time 1709785530.03 total 52.84 last 58.32) - open/close 3771 (time 1709785540.42 total 63.24 last 49.16) - open/close 4295 (time 1709785550.88 total 73.69 last 50.13) - open/close 4850 (time 1709785561.39 total 84.20 last 52.80) - open/close 5447 (time 1709785572.17 total 94.99 last 55.36) - open/close 5915 (time 1709785582.53 total 105.35 last 45.16) - open/close 6423 (time 1709785593.06 total 115.87 last 48.27) - open/close 6904 (time 1709785603.89 total 126.70 last 44.41) - open/close 7412 (time 1709785614.72 total 137.53 last 46.93) - open/close 7753 (time 1709785624.94 total 147.76 last 33.34) - open/close 8165 (time 1709785635.58 total 158.39 last 38.75) - open/close 8661 (time 1709785646.55 total 169.36 last 45.20) - open/close 9252 (time 1709785657.28 total 180.10 last 55.05) - open/close 9752 (time 1709785667.46 total 190.27 last 49.15) - open/close 10000 (time 1709785670.66 total 193.47 last 77.53) - open/close 10358 (time 1709785681.47 total 204.28 last 33.10) - open/close 11010 (time 1709785691.47 total 214.29 last 65.18) - open/close 11381 (time 1709785701.97 total 224.79 last 35.33) - open/close 11769 (time 1709785711.98 total 234.79 last 38.79) - open/close 12775 (time 1709785721.98 total 244.79 last 100.56) - open/close 14282 (time 1709785731.98 total 254.80 last 150.64) - open/close 15901 (time 1709785741.98 total 264.80 last 161.89) - open/close 17238 (time 1709785751.99 total 274.80 last 133.61) - open/close 18494 (time 1709785761.99 total 284.81 last 125.59) - open/close 19625 (time 1709785772.00 total 294.81 last 113.05) - open/close 20000 (time 1709785775.24 total 298.06 last 115.46) - open/close 21139 (time 1709785785.25 total 308.06 last 113.87) - open/close 22177 (time 1709785795.25 total 318.06 last 103.80) - open/close 23167 (time 1709785805.25 total 328.07 last 98.92) - open/close 24098 (time 1709785815.26 total 338.07 last 93.08) - open/close 24961 (time 1709785825.26 total 348.07 last 86.29) - open/close 25772 (time 1709785835.27 total 358.08 last 81.03) - open/close 26516 (time 1709785845.28 total 368.09 last 74.32) - open/close 27270 (time 1709785855.29 total 378.10 last 75.34) - open/close 27970 (time 1709785865.30 total 388.11 last 69.92) - open/close 28699 (time 1709785875.30 total 398.12 last 72.85) - open/close 29391 (time 1709785885.31 total 408.12 last 69.18) - open/close 30000 (time 1709785894.38 total 417.19 last 67.13) - open/close 30604 (time 1709785904.39 total 427.21 last 60.31) - open/close 31193 (time 1709785914.40 total 437.21 last 58.85) - open/close 31834 (time 1709785924.40 total 447.22 last 64.08) - open/close 32408 (time 1709785934.42 total 457.23 last 57.33) - open/close 33004 (time 1709785944.43 total 467.24 last 59.53) - open/close 33595 (time 1709785954.44 total 477.25 last 59.05) - open/close 34150 (time 1709785964.44 total 487.26 last 55.47) - open/close 34703 (time 1709785974.45 total 497.27 last 55.24) - open/close 35232 (time 1709785984.46 total 507.27 last 52.87) - open/close 35731 (time 1709785994.47 total 517.28 last 49.85) - open/close 36232 (time 1709786004.49 total 527.30 last 50.01) - open/close 36724 (time 1709786014.49 total 537.31 last 49.17) - open/close 37205 (time 1709786024.51 total 547.33 last 48.02) - open/close 37681 (time 1709786034.52 total 557.34 last 47.54) - open/close 38123 (time 1709786044.54 total 567.35 last 44.15) - open/close 38592 (time 1709786054.54 total 577.35 last 46.88) - open/close 39056 (time 1709786064.55 total 587.36 last 46.36) - open/close 39550 (time 1709786074.56 total 597.37 last 49.35) - open/close 39975 (time 1709786084.57 total 607.38 last 42.46) - open/close 40449 (time 1709786094.58 total 617.40 last 47.33) - open/close 40919 (time 1709786104.60 total 627.42 last 46.90) - open/close 41351 (time 1709786114.62 total 637.43 last 43.15) - open/close 41803 (time 1709786124.64 total 647.45 last 45.11) - open/close 42256 (time 1709786134.65 total 657.46 last 45.25) - open/close 42711 (time 1709786144.67 total 667.48 last 45.40) - open/close 43158 (time 1709786154.69 total 677.50 last 44.62) - open/close 43590 (time 1709786164.69 total 687.50 last 43.19) - open/close 44043 (time 1709786174.70 total 697.52 last 45.24) - open/close 44452 (time 1709786184.72 total 707.53 last 40.85) - open/close 44862 (time 1709786194.74 total 717.55 last 40.91) - open/close 45253 (time 1709786204.75 total 727.57 last 39.03) - open/close 45671 (time 1709786214.76 total 737.58 last 41.77) - open/close 46090 (time 1709786224.78 total 747.59 last 41.82) - open/close 46528 (time 1709786234.79 total 757.61 last 43.75) - open/close 46964 (time 1709786244.80 total 767.62 last 43.56) - open/close 47421 (time 1709786254.82 total 777.64 last 45.61) - open/close 47866 (time 1709786264.83 total 787.64 last 44.46) - open/close 48278 (time 1709786274.84 total 797.65 last 41.18) - open/close 48733 (time 1709786284.86 total 807.67 last 45.41) - open/close 49153 (time 1709786294.88 total 817.69 last 41.90) - open/close 49537 (time 1709786304.89 total 827.71 last 38.34) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49600) error: No space left on device total: 49600 open/close in 829.98 seconds: 59.76 ops/second - unlinked 0 (time 1709786308 ; total 0 ; last 0) - unlinked 10000 (time 1709786592 ; total 284 ; last 284) - unlinked 20000 (time 1709786887 ; total 579 ; last 295) - unlinked 30000 (time 1709787151 ; total 843 ; last 264) - unlinked 40000 (time 1709787324 ; total 1016 ; last 173) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49600) error: No such file or directory total: 49600 unlinks in 1110 seconds: 44.684685 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg125-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg125-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg125-server stop mds service on oleg125-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg125-server stop mds service on oleg125-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg125-server unloading modules on: 'oleg125-server' oleg125-server: oleg125-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg125-client: oleg125-client: ssh exited with exit code 2 pdsh@oleg125-client: oleg125-server: ssh exited with exit code 2 pdsh@oleg125-client: oleg125-client: ssh exited with exit code 2 pdsh@oleg125-client: oleg125-server: ssh exited with exit code 2 PASS 101a (1994s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg125-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg125-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg125-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg125-client: oleg125-server: ssh exited with exit code 2 oleg125-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg125-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == conf-sanity test 101b: Race events DISCONNECT and ACTIVE in osp ========================================================== 23:57:36 (1709787456) start mds service on oleg125-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg125-server' oleg125-server: oleg125-server.virtnet: executing load_modules_local oleg125-server: Loading modules from /home/green/git/lustre-release/lustre oleg125-server: detected 4 online CPUs by sysfs oleg125-server: Force libcfs to create 2 CPU partitions oleg125-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg125-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg125-server: oleg125-server.virtnet: executing set_default_debug -1 all pdsh@oleg125-client: oleg125-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg125-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg125-server: oleg125-server.virtnet: executing set_default_debug -1 all pdsh@oleg125-client: oleg125-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg125-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg125-server: oleg125-server.virtnet: executing set_default_debug -1 all pdsh@oleg125-client: oleg125-server: ssh exited with exit code 1 Started lustre-OST0000 oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg125-client.virtnet: -o user_xattr,flock oleg125-server@tcp:/lustre /mnt/lustre fail_loc=0x80002107 fail_val=20 stop ost1 service on oleg125-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg125-server start ost1 service on oleg125-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg125-server: oleg125-server.virtnet: executing set_default_debug -1 all pdsh@oleg125-client: oleg125-server: ssh exited with exit code 1 Started lustre-OST0000 oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec oleg125-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8800ac79f800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800ac79f800.ost_server_uuid in FULL state after 0 sec umount lustre on /mnt/lustre..... Stopping client oleg125-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg125-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg125-server stop mds service on oleg125-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg125-server stop mds service on oleg125-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg125-server unloading modules on: 'oleg125-server' oleg125-server: oleg125-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg125-client: oleg125-client: ssh exited with exit code 2 pdsh@oleg125-client: oleg125-server: ssh exited with exit code 2 pdsh@oleg125-client: oleg125-server: ssh exited with exit code 2 pdsh@oleg125-client: oleg125-client: ssh exited with exit code 2 PASS 101b (142s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg125-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg125-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory Stopping clients: oleg125-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg125-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg125-client: oleg125-server: ssh exited with exit code 2 oleg125-server: oleg125-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg125-server' oleg125-server: oleg125-server.virtnet: executing load_modules_local oleg125-server: Loading modules from /home/green/git/lustre-release/lustre oleg125-server: detected 4 online CPUs by sysfs oleg125-server: Force libcfs to create 2 CPU partitions oleg125-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg125-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey == conf-sanity test complete, duration 2669 sec ========== 00:00:40 (1709787640) === conf-sanity: start cleanup 00:00:41 (1709787641) === === conf-sanity: finish cleanup 00:00:41 (1709787641) ===