-----============= acceptance-small: conf-sanity ============----- Wed Mar 6 23:16:21 EST 2024 excepting tests: 32newtarball 110 skipping tests SLOW=no: 45 69 106 111 114 Stopping clients: oleg227-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg227-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg227-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg227-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg227-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg227-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg227-server oleg227-server: oleg227-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg227-server' oleg227-server: oleg227-server.virtnet: executing load_modules_local oleg227-server: Loading modules from /home/green/git/lustre-release/lustre oleg227-server: detected 4 online CPUs by sysfs oleg227-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg227-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg227-server: oleg227-server.virtnet: executing set_default_debug -1 all pdsh@oleg227-client: oleg227-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg227-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg227-server: oleg227-server.virtnet: executing set_default_debug -1 all pdsh@oleg227-client: oleg227-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg227-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg227-server: oleg227-server.virtnet: executing set_default_debug -1 all pdsh@oleg227-client: oleg227-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg227-server: oleg227-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg227-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg227-server: oleg227-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg227-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg227-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg227-server stop mds service on oleg227-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg227-server stop mds service on oleg227-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg227-server debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 35b: Continue reconnection retries, if the active server is busy ========================================================== 23:17:43 (1709785063) start mds service on oleg227-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg227-server: oleg227-server.virtnet: executing set_default_debug -1 all pdsh@oleg227-client: oleg227-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg227-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg227-server: oleg227-server.virtnet: executing set_default_debug -1 all pdsh@oleg227-client: oleg227-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg227-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg227-server: oleg227-server.virtnet: executing set_default_debug -1 all pdsh@oleg227-client: oleg227-server: ssh exited with exit code 1 Started lustre-OST0000 oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg227-client.virtnet: -o user_xattr,flock oleg227-server@tcp:/lustre /mnt/lustre debug=ha conf-sanity.sh test_35b 2024-03-0623h18m01s Set up a fake failnode for the MDS at_max=0 at_max=0 Injecting EBUSY on MDS fail_loc=0x80000136 mdc.lustre-MDT0000-mdc-ffff880130fd4800.stats=clear mdc.lustre-MDT0001-mdc-ffff880130fd4800.stats=clear Creating a test file and stat it File: '/mnt/lustre/d35b.conf-sanity/f35b.conf-sanity' Size: 0 Blocks: 0 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115238826934274 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-03-06 23:18:44.000000000 -0500 Modify: 2024-03-06 23:18:44.000000000 -0500 Change: 2024-03-06 23:18:44.000000000 -0500 Birth: - Stop injecting EBUSY on MDS fail_loc=0 done at_max=600 at_max=600 Debug log: 66 lines, 66 kept, 0 dropped, 0 bad. umount lustre on /mnt/lustre..... Stopping client oleg227-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg227-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg227-server stop mds service on oleg227-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg227-server stop mds service on oleg227-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg227-server unloading modules on: 'oleg227-server' oleg227-server: oleg227-server.virtnet: executing unload_modules_local modules unloaded. oleg227-server: tunefs.lustre: Unable to mount /dev/mapper/mds1_flakey: No such device oleg227-server: Is the ldiskfs module available? oleg227-server: oleg227-server: tunefs.lustre FATAL: failed to write local files oleg227-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg227-client: oleg227-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x5 (MDT MGS ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity Permanent disk data: Target: lustre=MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x105 (MDT MGS writeconf ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity oleg227-server: tunefs.lustre: Unable to mount /dev/mapper/mds2_flakey: No such device oleg227-server: Is the ldiskfs module available? oleg227-server: oleg227-server: tunefs.lustre FATAL: failed to write local files oleg227-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg227-client: oleg227-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-MDT0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x1 (MDT ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: mgsnode=192.168.202.127@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity Permanent disk data: Target: lustre=MDT0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x101 (MDT writeconf ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: mgsnode=192.168.202.127@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity oleg227-server: tunefs.lustre: Unable to mount /dev/mapper/ost1_flakey: No such device oleg227-server: Is the ldiskfs module available? oleg227-server: oleg227-server: tunefs.lustre FATAL: failed to write local files oleg227-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg227-client: oleg227-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x2 (OST ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.202.127@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x102 (OST writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.202.127@tcp sys.timeout=20 oleg227-server: tunefs.lustre: Unable to mount /dev/mapper/ost2_flakey: No such device oleg227-server: Is the ldiskfs module available? oleg227-server: oleg227-server: tunefs.lustre FATAL: failed to write local files oleg227-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg227-client: oleg227-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.202.127@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x162 (OST first_time update writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.202.127@tcp sys.timeout=20 tunefs failed, reformatting instead Stopping clients: oleg227-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg227-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg227-client: oleg227-server: ssh exited with exit code 2 oleg227-server: oleg227-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg227-server' oleg227-server: oleg227-server.virtnet: executing load_modules_local oleg227-server: Loading modules from /home/green/git/lustre-release/lustre oleg227-server: detected 4 online CPUs by sysfs oleg227-server: Force libcfs to create 2 CPU partitions oleg227-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg227-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg227-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg227-server: oleg227-server.virtnet: executing set_default_debug -1 all pdsh@oleg227-client: oleg227-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg227-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg227-server: oleg227-server.virtnet: executing set_default_debug -1 all pdsh@oleg227-client: oleg227-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg227-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg227-server: oleg227-server.virtnet: executing set_default_debug -1 all pdsh@oleg227-client: oleg227-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg227-server: oleg227-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg227-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg227-server: oleg227-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg227-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg227-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg227-server stop mds service on oleg227-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg227-server stop mds service on oleg227-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg227-server PASS 35b (137s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 23:20:01 (1709785201) start mds service on oleg227-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg227-server: oleg227-server.virtnet: executing set_default_debug -1 all pdsh@oleg227-client: oleg227-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg227-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg227-server: oleg227-server.virtnet: executing set_default_debug -1 all pdsh@oleg227-client: oleg227-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg227-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg227-server: oleg227-server.virtnet: executing set_default_debug -1 all pdsh@oleg227-client: oleg227-server: ssh exited with exit code 1 Started lustre-OST0000 oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg227-client.virtnet: -o user_xattr,flock oleg227-server@tcp:/lustre /mnt/lustre copying 10 files to /mnt/lustre/d38.conf-sanity tar: Removing leading `/' from member names umount lustre on /mnt/lustre..... Stopping client oleg227-client.virtnet /mnt/lustre (opts:) osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=44 osp.lustre-OST0000-osc-MDT0001.prealloc_next_id=34 stop mds service on oleg227-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg227-server stop mds service on oleg227-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg227-server delete lov_objid file on MDS 000000 43 000008 start mds service on oleg227-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg227-server: oleg227-server.virtnet: executing set_default_debug -1 all pdsh@oleg227-client: oleg227-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg227-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg227-server: oleg227-server.virtnet: executing set_default_debug -1 all pdsh@oleg227-client: oleg227-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg227-client.virtnet: -o user_xattr,flock oleg227-server@tcp:/lustre /mnt/lustre osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=2 osp.lustre-OST0000-osc-MDT0001.prealloc_next_id=66 umount lustre on /mnt/lustre..... Stopping client oleg227-client.virtnet /mnt/lustre (opts:) stop mds service on oleg227-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg227-server stop mds service on oleg227-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg227-server 000000 2 000008 oleg227-server: 8+0 records in oleg227-server: 8+0 records out oleg227-server: 4096 bytes (4.1 kB) copied, 0.000375004 s, 10.9 MB/s start mds service on oleg227-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg227-server: oleg227-server.virtnet: executing set_default_debug -1 all pdsh@oleg227-client: oleg227-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg227-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg227-server: oleg227-server.virtnet: executing set_default_debug -1 all pdsh@oleg227-client: oleg227-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg227-client.virtnet: -o user_xattr,flock oleg227-server@tcp:/lustre /mnt/lustre osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=35 osp.lustre-OST0000-osc-MDT0001.prealloc_next_id=98 umount lustre on /mnt/lustre..... Stopping client oleg227-client.virtnet /mnt/lustre (opts:) stop mds service on oleg227-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg227-server stop mds service on oleg227-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg227-server 000000 34 000008 files compared the same umount lustre on /mnt/lustre..... stop ost1 service on oleg227-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg227-server stop mds service on oleg227-server stop mds service on oleg227-server unloading modules on: 'oleg227-server' oleg227-server: oleg227-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg227-client: oleg227-client: ssh exited with exit code 2 pdsh@oleg227-client: oleg227-server: ssh exited with exit code 2 pdsh@oleg227-client: oleg227-client: ssh exited with exit code 2 pdsh@oleg227-client: oleg227-server: ssh exited with exit code 2 PASS 38 (102s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg227-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg227-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg227-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg227-client: oleg227-server: ssh exited with exit code 2 oleg227-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg227-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 23:21:45 (1709785305) Stopping clients: oleg227-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg227-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg227-client: oleg227-server: ssh exited with exit code 2 oleg227-server: oleg227-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg227-server' oleg227-server: oleg227-server.virtnet: executing load_modules_local oleg227-server: Loading modules from /home/green/git/lustre-release/lustre oleg227-server: detected 4 online CPUs by sysfs oleg227-server: Force libcfs to create 2 CPU partitions oleg227-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg227-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg227-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg227-server: oleg227-server.virtnet: executing set_default_debug -1 all pdsh@oleg227-client: oleg227-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg227-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg227-server: oleg227-server.virtnet: executing set_default_debug -1 all pdsh@oleg227-client: oleg227-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg227-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg227-server: oleg227-server.virtnet: executing set_default_debug -1 all pdsh@oleg227-client: oleg227-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg227-client.virtnet: -o user_xattr,flock oleg227-server@tcp:/lustre /mnt/lustre setup single mount lustre success fail_loc=0x142 start ost2 service on oleg227-server Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg227-server: oleg227-server.virtnet: executing set_default_debug -1 all pdsh@oleg227-client: oleg227-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost2_flakey Started lustre-OST0001 oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 1 sec stop ost2 service on oleg227-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg227-server umount lustre on /mnt/lustre..... Stopping client oleg227-client.virtnet /mnt/lustre (opts:-f) stop ost1 service on oleg227-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg227-server stop mds service on oleg227-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg227-server stop mds service on oleg227-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg227-server unloading modules on: 'oleg227-server' oleg227-server: oleg227-server.virtnet: executing unload_modules_local modules unloaded. oleg227-server: tunefs.lustre: Unable to mount /dev/mapper/mds1_flakey: No such device oleg227-server: Is the ldiskfs module available? oleg227-server: oleg227-server: tunefs.lustre FATAL: failed to write local files oleg227-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg227-client: oleg227-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x5 (MDT MGS ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity Permanent disk data: Target: lustre=MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x105 (MDT MGS writeconf ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity oleg227-server: tunefs.lustre: Unable to mount /dev/mapper/mds2_flakey: No such device oleg227-server: Is the ldiskfs module available? oleg227-server: oleg227-server: tunefs.lustre FATAL: failed to write local files oleg227-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg227-client: oleg227-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-MDT0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x1 (MDT ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: mgsnode=192.168.202.127@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity Permanent disk data: Target: lustre=MDT0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x101 (MDT writeconf ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: mgsnode=192.168.202.127@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity oleg227-server: tunefs.lustre: Unable to mount /dev/mapper/ost1_flakey: No such device oleg227-server: Is the ldiskfs module available? oleg227-server: oleg227-server: tunefs.lustre FATAL: failed to write local files oleg227-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg227-client: oleg227-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x2 (OST ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.202.127@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x102 (OST writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.202.127@tcp sys.timeout=20 oleg227-server: tunefs.lustre: Unable to mount /dev/mapper/ost2_flakey: No such device oleg227-server: Is the ldiskfs module available? oleg227-server: oleg227-server: tunefs.lustre FATAL: failed to write local files oleg227-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg227-client: oleg227-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x2 (OST ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.202.127@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x102 (OST writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.202.127@tcp sys.timeout=20 tunefs failed, reformatting instead Stopping clients: oleg227-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg227-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg227-client: oleg227-server: ssh exited with exit code 2 oleg227-server: oleg227-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg227-server' oleg227-server: oleg227-server.virtnet: executing load_modules_local oleg227-server: Loading modules from /home/green/git/lustre-release/lustre oleg227-server: detected 4 online CPUs by sysfs oleg227-server: Force libcfs to create 2 CPU partitions oleg227-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg227-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg227-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg227-server: oleg227-server.virtnet: executing set_default_debug -1 all pdsh@oleg227-client: oleg227-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg227-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg227-server: oleg227-server.virtnet: executing set_default_debug -1 all pdsh@oleg227-client: oleg227-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg227-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg227-server: oleg227-server.virtnet: executing set_default_debug -1 all pdsh@oleg227-client: oleg227-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg227-server: oleg227-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg227-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec oleg227-server: oleg227-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg227-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg227-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg227-server stop mds service on oleg227-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg227-server stop mds service on oleg227-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg227-server PASS 51 (154s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:24:21 (1709785461) start mds service on oleg227-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg227-server: oleg227-server.virtnet: executing set_default_debug -1 all pdsh@oleg227-client: oleg227-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg227-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg227-server: oleg227-server.virtnet: executing set_default_debug -1 all pdsh@oleg227-client: oleg227-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg227-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg227-server: oleg227-server.virtnet: executing set_default_debug -1 all pdsh@oleg227-client: oleg227-server: ssh exited with exit code 1 Started lustre-OST0000 oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg227-client.virtnet: -o user_xattr,flock oleg227-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 584 (time 1709785490.45 total 10.39 last 56.20) - open/close 1049 (time 1709785500.60 total 20.54 last 45.81) - open/close 1429 (time 1709785510.61 total 30.54 last 37.99) - open/close 1836 (time 1709785520.67 total 40.61 last 40.44) - open/close 2220 (time 1709785531.62 total 51.56 last 35.06) - open/close 2748 (time 1709785542.16 total 62.10 last 50.13) - open/close 3417 (time 1709785553.16 total 73.10 last 60.80) - open/close 3859 (time 1709785563.57 total 83.51 last 42.46) - open/close 4415 (time 1709785574.34 total 94.28 last 51.63) - open/close 4808 (time 1709785585.66 total 105.60 last 34.71) - open/close 5329 (time 1709785595.66 total 115.60 last 52.10) - open/close 5589 (time 1709785606.63 total 126.57 last 23.69) - open/close 5836 (time 1709785616.63 total 136.57 last 24.70) - open/close 6328 (time 1709785626.64 total 146.58 last 49.19) - open/close 6727 (time 1709785637.41 total 157.35 last 37.05) - open/close 7214 (time 1709785647.41 total 167.35 last 48.70) - open/close 7693 (time 1709785658.36 total 178.30 last 43.74) - open/close 8079 (time 1709785669.07 total 189.01 last 36.03) - open/close 8513 (time 1709785679.08 total 199.02 last 43.36) - open/close 8948 (time 1709785689.50 total 209.44 last 41.77) - open/close 9199 (time 1709785700.91 total 220.85 last 21.99) - open/close 9456 (time 1709785711.42 total 231.36 last 24.46) - open/close 9795 (time 1709785721.42 total 241.36 last 33.89) - open/close 10000 (time 1709785726.63 total 246.57 last 39.33) - open/close 11581 (time 1709785736.63 total 256.57 last 158.09) - open/close 12988 (time 1709785746.64 total 266.57 last 140.68) - open/close 14349 (time 1709785756.64 total 276.58 last 136.07) - open/close 15599 (time 1709785766.64 total 286.58 last 124.99) - open/close 16770 (time 1709785776.64 total 296.58 last 117.03) - open/close 17851 (time 1709785786.65 total 306.59 last 108.06) - open/close 18901 (time 1709785796.65 total 316.59 last 104.94) - open/close 19875 (time 1709785806.66 total 326.60 last 97.31) - open/close 20737 (time 1709785816.66 total 336.60 last 86.20) - open/close 21534 (time 1709785826.67 total 346.61 last 79.66) - open/close 22260 (time 1709785836.68 total 356.62 last 72.55) - open/close 23097 (time 1709785846.69 total 366.63 last 83.61) - open/close 23807 (time 1709785856.70 total 376.64 last 70.90) - open/close 24533 (time 1709785866.71 total 386.65 last 72.56) - open/close 25360 (time 1709785876.71 total 396.65 last 82.65) - open/close 26166 (time 1709785886.72 total 406.66 last 80.57) - open/close 26861 (time 1709785896.73 total 416.67 last 69.41) - open/close 27508 (time 1709785906.74 total 426.68 last 64.61) - open/close 28121 (time 1709785916.75 total 436.69 last 61.29) - open/close 28738 (time 1709785926.76 total 446.70 last 61.64) - open/close 29377 (time 1709785936.76 total 456.70 last 63.87) - open/close 29991 (time 1709785946.77 total 466.71 last 61.34) - open/close 30603 (time 1709785956.78 total 476.72 last 61.13) - open/close 31210 (time 1709785966.78 total 486.72 last 60.69) - open/close 31808 (time 1709785976.80 total 496.74 last 59.71) - open/close 32393 (time 1709785986.82 total 506.75 last 58.40) - open/close 32977 (time 1709785996.83 total 516.77 last 58.31) - open/close 33548 (time 1709786006.85 total 526.78 last 57.02) - open/close 34113 (time 1709786016.85 total 536.79 last 56.46) - open/close 34650 (time 1709786026.86 total 546.80 last 53.68) - open/close 35170 (time 1709786036.86 total 556.80 last 51.97) - open/close 35691 (time 1709786046.86 total 566.80 last 52.09) - open/close 36226 (time 1709786056.87 total 576.81 last 53.47) - open/close 36744 (time 1709786066.89 total 586.83 last 51.72) - open/close 37238 (time 1709786076.89 total 596.83 last 49.36) - open/close 37741 (time 1709786086.90 total 606.84 last 50.26) - open/close 38252 (time 1709786096.91 total 616.85 last 51.06) - open/close 38753 (time 1709786106.91 total 626.85 last 50.08) - open/close 39254 (time 1709786116.92 total 636.86 last 50.08) - open/close 39749 (time 1709786126.93 total 646.87 last 49.44) - open/close 40000 (time 1709786132.12 total 652.06 last 48.34) - open/close 40468 (time 1709786142.13 total 662.07 last 46.76) - open/close 40958 (time 1709786152.14 total 672.08 last 48.94) - open/close 41435 (time 1709786162.15 total 682.09 last 47.65) - open/close 41924 (time 1709786172.16 total 692.10 last 48.87) - open/close 42411 (time 1709786182.16 total 702.10 last 48.68) - open/close 42899 (time 1709786192.17 total 712.11 last 48.77) - open/close 43361 (time 1709786202.19 total 722.13 last 46.11) - open/close 43827 (time 1709786212.21 total 732.15 last 46.49) - open/close 44273 (time 1709786222.22 total 742.16 last 44.55) - open/close 44749 (time 1709786232.23 total 752.17 last 47.57) - open/close 45188 (time 1709786242.24 total 762.18 last 43.86) - open/close 45642 (time 1709786252.25 total 772.19 last 45.36) - open/close 46096 (time 1709786262.27 total 782.21 last 45.32) - open/close 46583 (time 1709786272.27 total 792.21 last 48.70) - open/close 47048 (time 1709786282.27 total 802.21 last 46.49) - open/close 47393 (time 1709786292.27 total 812.21 last 34.49) - open/close 47738 (time 1709786302.29 total 822.23 last 34.43) - open/close 48076 (time 1709786312.30 total 832.24 last 33.77) - open/close 48461 (time 1709786322.32 total 842.26 last 38.44) - open/close 48777 (time 1709786332.32 total 852.26 last 31.59) - open/close 49109 (time 1709786342.34 total 862.28 last 33.15) - open/close 49466 (time 1709786352.34 total 872.28 last 35.69) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49600) error: No space left on device total: 49600 open/close in 876.84 seconds: 56.57 ops/second - unlinked 0 (time 1709786358 ; total 0 ; last 0) - unlinked 10000 (time 1709786679 ; total 321 ; last 321) - unlinked 20000 (time 1709786991 ; total 633 ; last 312) - unlinked 30000 (time 1709787240 ; total 882 ; last 249) - unlinked 40000 (time 1709787422 ; total 1064 ; last 182) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49600) error: No such file or directory total: 49600 unlinks in 1154 seconds: 42.980934 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg227-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg227-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg227-server stop mds service on oleg227-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg227-server stop mds service on oleg227-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg227-server unloading modules on: 'oleg227-server' oleg227-server: oleg227-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg227-client: oleg227-server: ssh exited with exit code 2 pdsh@oleg227-client: oleg227-client: ssh exited with exit code 2 pdsh@oleg227-client: oleg227-server: ssh exited with exit code 2 pdsh@oleg227-client: oleg227-client: ssh exited with exit code 2 PASS 101a (2084s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg227-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg227-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg227-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg227-client: oleg227-server: ssh exited with exit code 2 oleg227-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg227-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == conf-sanity test 101b: Race events DISCONNECT and ACTIVE in osp ========================================================== 23:59:08 (1709787548) start mds service on oleg227-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg227-server' oleg227-server: oleg227-server.virtnet: executing load_modules_local oleg227-server: Loading modules from /home/green/git/lustre-release/lustre oleg227-server: detected 4 online CPUs by sysfs oleg227-server: Force libcfs to create 2 CPU partitions oleg227-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg227-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg227-server: oleg227-server.virtnet: executing set_default_debug -1 all pdsh@oleg227-client: oleg227-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg227-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg227-server: oleg227-server.virtnet: executing set_default_debug -1 all pdsh@oleg227-client: oleg227-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg227-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg227-server: oleg227-server.virtnet: executing set_default_debug -1 all pdsh@oleg227-client: oleg227-server: ssh exited with exit code 1 Started lustre-OST0000 oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg227-client.virtnet: -o user_xattr,flock oleg227-server@tcp:/lustre /mnt/lustre fail_loc=0x80002107 fail_val=20 stop ost1 service on oleg227-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg227-server start ost1 service on oleg227-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg227-server: oleg227-server.virtnet: executing set_default_debug -1 all pdsh@oleg227-client: oleg227-server: ssh exited with exit code 1 Started lustre-OST0000 oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec oleg227-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8800ab130800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800ab130800.ost_server_uuid in FULL state after 0 sec umount lustre on /mnt/lustre..... Stopping client oleg227-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg227-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg227-server stop mds service on oleg227-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg227-server stop mds service on oleg227-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg227-server unloading modules on: 'oleg227-server' oleg227-server: oleg227-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg227-client: oleg227-client: ssh exited with exit code 2 pdsh@oleg227-client: oleg227-server: ssh exited with exit code 2 pdsh@oleg227-client: oleg227-client: ssh exited with exit code 2 pdsh@oleg227-client: oleg227-server: ssh exited with exit code 2 PASS 101b (138s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg227-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg227-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory Stopping clients: oleg227-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg227-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg227-client: oleg227-server: ssh exited with exit code 2 oleg227-server: oleg227-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg227-server' oleg227-server: oleg227-server.virtnet: executing load_modules_local oleg227-server: Loading modules from /home/green/git/lustre-release/lustre oleg227-server: detected 4 online CPUs by sysfs oleg227-server: Force libcfs to create 2 CPU partitions oleg227-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg227-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey == conf-sanity test complete, duration 2750 sec ========== 00:02:11 (1709787731) === conf-sanity: start cleanup 00:02:13 (1709787733) === === conf-sanity: finish cleanup 00:02:13 (1709787733) ===