-----============= acceptance-small: conf-sanity ============----- Wed Mar 6 23:41:39 EST 2024 excepting tests: 32newtarball 110 skipping tests SLOW=no: 45 69 106 111 114 Stopping clients: oleg256-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg256-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg256-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg256-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg256-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg256-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg256-server oleg256-server: oleg256-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg256-server' oleg256-server: oleg256-server.virtnet: executing load_modules_local oleg256-server: Loading modules from /home/green/git/lustre-release/lustre oleg256-server: detected 4 online CPUs by sysfs oleg256-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg256-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg256-server: oleg256-server.virtnet: executing set_default_debug -1 all pdsh@oleg256-client: oleg256-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg256-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg256-server: oleg256-server.virtnet: executing set_default_debug -1 all pdsh@oleg256-client: oleg256-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg256-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg256-server: oleg256-server.virtnet: executing set_default_debug -1 all pdsh@oleg256-client: oleg256-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg256-server: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg256-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg256-server: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg256-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg256-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg256-server stop mds service on oleg256-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg256-server stop mds service on oleg256-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg256-server debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 35b: Continue reconnection retries, if the active server is busy ========================================================== 23:43:18 (1709786598) start mds service on oleg256-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg256-server: oleg256-server.virtnet: executing set_default_debug -1 all pdsh@oleg256-client: oleg256-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg256-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg256-server: oleg256-server.virtnet: executing set_default_debug -1 all pdsh@oleg256-client: oleg256-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg256-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg256-server: oleg256-server.virtnet: executing set_default_debug -1 all pdsh@oleg256-client: oleg256-server: ssh exited with exit code 1 Started lustre-OST0000 oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg256-client.virtnet: -o user_xattr,flock oleg256-server@tcp:/lustre /mnt/lustre debug=ha conf-sanity.sh test_35b 2024-03-0623h43m51s Set up a fake failnode for the MDS at_max=0 at_max=0 Injecting EBUSY on MDS fail_loc=0x80000136 mdc.lustre-MDT0000-mdc-ffff88012abfe000.stats=clear mdc.lustre-MDT0001-mdc-ffff88012abfe000.stats=clear Creating a test file and stat it File: '/mnt/lustre/d35b.conf-sanity/f35b.conf-sanity' Size: 0 Blocks: 0 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115238826934274 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-03-06 23:44:37.000000000 -0500 Modify: 2024-03-06 23:44:37.000000000 -0500 Change: 2024-03-06 23:44:37.000000000 -0500 Birth: - Stop injecting EBUSY on MDS fail_loc=0 done at_max=600 at_max=600 Debug log: 88 lines, 88 kept, 0 dropped, 0 bad. umount lustre on /mnt/lustre..... Stopping client oleg256-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg256-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg256-server stop mds service on oleg256-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg256-server stop mds service on oleg256-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg256-server unloading modules on: 'oleg256-server' oleg256-server: oleg256-server.virtnet: executing unload_modules_local modules unloaded. oleg256-server: tunefs.lustre: Unable to mount /dev/mapper/mds1_flakey: No such device oleg256-server: Is the ldiskfs module available? oleg256-server: oleg256-server: tunefs.lustre FATAL: failed to write local files oleg256-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg256-client: oleg256-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x5 (MDT MGS ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity Permanent disk data: Target: lustre=MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x105 (MDT MGS writeconf ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity oleg256-server: tunefs.lustre: Unable to mount /dev/mapper/mds2_flakey: No such device oleg256-server: Is the ldiskfs module available? oleg256-server: oleg256-server: tunefs.lustre FATAL: failed to write local files oleg256-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg256-client: oleg256-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-MDT0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x1 (MDT ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: mgsnode=192.168.202.156@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity Permanent disk data: Target: lustre=MDT0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x101 (MDT writeconf ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: mgsnode=192.168.202.156@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity oleg256-server: tunefs.lustre: Unable to mount /dev/mapper/ost1_flakey: No such device oleg256-server: Is the ldiskfs module available? oleg256-server: oleg256-server: tunefs.lustre FATAL: failed to write local files oleg256-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg256-client: oleg256-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x2 (OST ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.202.156@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x102 (OST writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.202.156@tcp sys.timeout=20 oleg256-server: tunefs.lustre: Unable to mount /dev/mapper/ost2_flakey: No such device oleg256-server: Is the ldiskfs module available? oleg256-server: oleg256-server: tunefs.lustre FATAL: failed to write local files oleg256-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg256-client: oleg256-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.202.156@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x162 (OST first_time update writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.202.156@tcp sys.timeout=20 tunefs failed, reformatting instead Stopping clients: oleg256-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg256-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg256-client: oleg256-server: ssh exited with exit code 2 oleg256-server: oleg256-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg256-server' oleg256-server: oleg256-server.virtnet: executing load_modules_local oleg256-server: Loading modules from /home/green/git/lustre-release/lustre oleg256-server: detected 4 online CPUs by sysfs oleg256-server: Force libcfs to create 2 CPU partitions oleg256-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg256-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg256-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg256-server: oleg256-server.virtnet: executing set_default_debug -1 all pdsh@oleg256-client: oleg256-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg256-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg256-server: oleg256-server.virtnet: executing set_default_debug -1 all pdsh@oleg256-client: oleg256-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg256-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg256-server: oleg256-server.virtnet: executing set_default_debug -1 all pdsh@oleg256-client: oleg256-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg256-server: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg256-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec oleg256-server: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg256-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg256-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg256-server stop mds service on oleg256-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg256-server stop mds service on oleg256-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg256-server PASS 35b (187s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 23:46:27 (1709786787) start mds service on oleg256-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg256-server: oleg256-server.virtnet: executing set_default_debug -1 all pdsh@oleg256-client: oleg256-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg256-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg256-server: oleg256-server.virtnet: executing set_default_debug -1 all pdsh@oleg256-client: oleg256-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg256-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg256-server: oleg256-server.virtnet: executing set_default_debug -1 all pdsh@oleg256-client: oleg256-server: ssh exited with exit code 1 Started lustre-OST0000 oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg256-client.virtnet: -o user_xattr,flock oleg256-server@tcp:/lustre /mnt/lustre copying 10 files to /mnt/lustre/d38.conf-sanity tar: Removing leading `/' from member names umount lustre on /mnt/lustre..... Stopping client oleg256-client.virtnet /mnt/lustre (opts:) osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=44 osp.lustre-OST0000-osc-MDT0001.prealloc_next_id=34 stop mds service on oleg256-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg256-server stop mds service on oleg256-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg256-server delete lov_objid file on MDS 000000 43 000008 start mds service on oleg256-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg256-server: oleg256-server.virtnet: executing set_default_debug -1 all pdsh@oleg256-client: oleg256-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg256-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg256-server: oleg256-server.virtnet: executing set_default_debug -1 all pdsh@oleg256-client: oleg256-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg256-client.virtnet: -o user_xattr,flock oleg256-server@tcp:/lustre /mnt/lustre osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=2 osp.lustre-OST0000-osc-MDT0001.prealloc_next_id=66 umount lustre on /mnt/lustre..... Stopping client oleg256-client.virtnet /mnt/lustre (opts:) stop mds service on oleg256-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg256-server stop mds service on oleg256-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg256-server 000000 2 000008 oleg256-server: 8+0 records in oleg256-server: 8+0 records out oleg256-server: 4096 bytes (4.1 kB) copied, 0.00124426 s, 3.3 MB/s start mds service on oleg256-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg256-server: oleg256-server.virtnet: executing set_default_debug -1 all pdsh@oleg256-client: oleg256-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg256-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg256-server: oleg256-server.virtnet: executing set_default_debug -1 all pdsh@oleg256-client: oleg256-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg256-client.virtnet: -o user_xattr,flock oleg256-server@tcp:/lustre /mnt/lustre osp.lustre-OST0000-osc-MDT0000.prealloc_next_id=35 osp.lustre-OST0000-osc-MDT0001.prealloc_next_id=98 umount lustre on /mnt/lustre..... Stopping client oleg256-client.virtnet /mnt/lustre (opts:) stop mds service on oleg256-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg256-server stop mds service on oleg256-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg256-server 000000 34 000008 files compared the same umount lustre on /mnt/lustre..... stop ost1 service on oleg256-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg256-server stop mds service on oleg256-server stop mds service on oleg256-server unloading modules on: 'oleg256-server' oleg256-server: oleg256-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg256-client: oleg256-server: ssh exited with exit code 2 pdsh@oleg256-client: oleg256-client: ssh exited with exit code 2 pdsh@oleg256-client: oleg256-client: ssh exited with exit code 2 pdsh@oleg256-client: oleg256-server: ssh exited with exit code 2 PASS 38 (134s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg256-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg256-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg256-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg256-client: oleg256-server: ssh exited with exit code 2 oleg256-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg256-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 23:48:44 (1709786924) Stopping clients: oleg256-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg256-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg256-client: oleg256-server: ssh exited with exit code 2 oleg256-server: oleg256-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg256-server' oleg256-server: oleg256-server.virtnet: executing load_modules_local oleg256-server: Loading modules from /home/green/git/lustre-release/lustre oleg256-server: detected 4 online CPUs by sysfs oleg256-server: Force libcfs to create 2 CPU partitions oleg256-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg256-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg256-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg256-server: oleg256-server.virtnet: executing set_default_debug -1 all pdsh@oleg256-client: oleg256-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg256-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg256-server: oleg256-server.virtnet: executing set_default_debug -1 all pdsh@oleg256-client: oleg256-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg256-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg256-server: oleg256-server.virtnet: executing set_default_debug -1 all pdsh@oleg256-client: oleg256-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg256-client.virtnet: -o user_xattr,flock oleg256-server@tcp:/lustre /mnt/lustre setup single mount lustre success fail_loc=0x142 start ost2 service on oleg256-server Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg256-server: oleg256-server.virtnet: executing set_default_debug -1 all pdsh@oleg256-client: oleg256-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost2_flakey Started lustre-OST0001 oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec stop ost2 service on oleg256-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg256-server umount lustre on /mnt/lustre..... Stopping client oleg256-client.virtnet /mnt/lustre (opts:-f) stop ost1 service on oleg256-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg256-server stop mds service on oleg256-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg256-server stop mds service on oleg256-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg256-server unloading modules on: 'oleg256-server' oleg256-server: oleg256-server.virtnet: executing unload_modules_local modules unloaded. oleg256-server: tunefs.lustre: Unable to mount /dev/mapper/mds1_flakey: No such device oleg256-server: Is the ldiskfs module available? oleg256-server: oleg256-server: tunefs.lustre FATAL: failed to write local files oleg256-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg256-client: oleg256-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x5 (MDT MGS ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity Permanent disk data: Target: lustre=MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x105 (MDT MGS writeconf ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity oleg256-server: tunefs.lustre: Unable to mount /dev/mapper/mds2_flakey: No such device oleg256-server: Is the ldiskfs module available? oleg256-server: oleg256-server: tunefs.lustre FATAL: failed to write local files oleg256-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg256-client: oleg256-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-MDT0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x1 (MDT ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: mgsnode=192.168.202.156@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity Permanent disk data: Target: lustre=MDT0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x101 (MDT writeconf ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: mgsnode=192.168.202.156@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity oleg256-server: tunefs.lustre: Unable to mount /dev/mapper/ost1_flakey: No such device oleg256-server: Is the ldiskfs module available? oleg256-server: oleg256-server: tunefs.lustre FATAL: failed to write local files oleg256-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg256-client: oleg256-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x2 (OST ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.202.156@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x102 (OST writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.202.156@tcp sys.timeout=20 oleg256-server: tunefs.lustre: Unable to mount /dev/mapper/ost2_flakey: No such device oleg256-server: Is the ldiskfs module available? oleg256-server: oleg256-server: tunefs.lustre FATAL: failed to write local files oleg256-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg256-client: oleg256-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x2 (OST ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.202.156@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x102 (OST writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.202.156@tcp sys.timeout=20 tunefs failed, reformatting instead Stopping clients: oleg256-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg256-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg256-client: oleg256-server: ssh exited with exit code 2 oleg256-server: oleg256-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg256-server' oleg256-server: oleg256-server.virtnet: executing load_modules_local oleg256-server: Loading modules from /home/green/git/lustre-release/lustre oleg256-server: detected 4 online CPUs by sysfs oleg256-server: Force libcfs to create 2 CPU partitions oleg256-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg256-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg256-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg256-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg256-server: oleg256-server.virtnet: executing set_default_debug -1 all pdsh@oleg256-client: oleg256-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg256-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg256-server: oleg256-server.virtnet: executing set_default_debug -1 all pdsh@oleg256-client: oleg256-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg256-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg256-server: oleg256-server.virtnet: executing set_default_debug -1 all pdsh@oleg256-client: oleg256-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg256-server: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg256-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg256-server: oleg256-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg256-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg256-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg256-server stop mds service on oleg256-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg256-server stop mds service on oleg256-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg256-server PASS 51 (249s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:52:57 (1709787177) start mds service on oleg256-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg256-server: oleg256-server.virtnet: executing set_default_debug -1 all pdsh@oleg256-client: oleg256-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg256-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg256-server: oleg256-server.virtnet: executing set_default_debug -1 all pdsh@oleg256-client: oleg256-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg256-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg256-server: oleg256-server.virtnet: executing set_default_debug -1 all pdsh@oleg256-client: oleg256-server: ssh exited with exit code 1 Started lustre-OST0000 oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg256-client.virtnet: -o user_xattr,flock oleg256-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 275 (time 1709787229.68 total 10.87 last 25.31) - open/close 493 (time 1709787239.82 total 21.01 last 21.49) - open/close 646 (time 1709787250.06 total 31.25 last 14.95) - open/close 815 (time 1709787260.07 total 41.26 last 16.88) - open/close 1063 (time 1709787270.48 total 51.67 last 23.82) - open/close 1215 (time 1709787280.69 total 61.89 last 14.88) - open/close 1411 (time 1709787291.48 total 72.67 last 18.17) - open/close 1555 (time 1709787302.37 total 83.56 last 13.22) - open/close 1767 (time 1709787312.79 total 93.98 last 20.34) - open/close 2039 (time 1709787323.59 total 104.78 last 25.19) - open/close 2354 (time 1709787334.88 total 116.07 last 27.90) - open/close 2553 (time 1709787345.10 total 126.29 last 19.47) - open/close 2706 (time 1709787355.67 total 136.86 last 14.48) - open/close 3096 (time 1709787365.69 total 146.88 last 38.92) - open/close 3306 (time 1709787375.69 total 156.88 last 20.99) - open/close 3530 (time 1709787386.68 total 167.87 last 20.38) - open/close 3750 (time 1709787396.77 total 177.97 last 21.80) - open/close 3940 (time 1709787406.83 total 188.02 last 18.90) - open/close 4159 (time 1709787417.45 total 198.64 last 20.61) - open/close 4255 (time 1709787428.34 total 209.53 last 8.82) - open/close 4442 (time 1709787438.64 total 219.83 last 18.14) - open/close 4574 (time 1709787449.20 total 230.39 last 12.50) - open/close 4773 (time 1709787459.96 total 241.16 last 18.49) - open/close 4914 (time 1709787470.87 total 252.06 last 12.93) - open/close 5101 (time 1709787481.67 total 262.86 last 17.32) - open/close 5278 (time 1709787491.72 total 272.91 last 17.61) - open/close 5395 (time 1709787501.73 total 282.92 last 11.69) - open/close 5593 (time 1709787511.80 total 292.99 last 19.66) - open/close 5744 (time 1709787522.44 total 303.63 last 14.19) - open/close 5870 (time 1709787533.33 total 314.52 last 11.57) - open/close 5992 (time 1709787543.35 total 324.54 last 12.18) - open/close 6187 (time 1709787553.35 total 334.54 last 19.50) - open/close 6320 (time 1709787563.36 total 344.55 last 13.28) - open/close 6417 (time 1709787574.41 total 355.60 last 8.78) - open/close 6576 (time 1709787584.70 total 365.89 last 15.46) - open/close 6767 (time 1709787594.71 total 375.90 last 19.07) - open/close 6894 (time 1709787605.51 total 386.70 last 11.76) - open/close 7170 (time 1709787615.52 total 396.71 last 27.57) - open/close 7332 (time 1709787625.53 total 406.72 last 16.19) - open/close 7520 (time 1709787635.77 total 416.96 last 18.35) - open/close 7657 (time 1709787646.18 total 427.38 last 13.15) - open/close 7858 (time 1709787656.19 total 437.38 last 20.09) - open/close 8017 (time 1709787666.55 total 447.74 last 15.35) - open/close 8104 (time 1709787677.55 total 458.74 last 7.91) - open/close 8289 (time 1709787688.02 total 469.21 last 17.67) - open/close 8401 (time 1709787698.03 total 479.22 last 11.18) - open/close 8567 (time 1709787708.51 total 489.70 last 15.84) - open/close 8761 (time 1709787718.51 total 499.70 last 19.40) - open/close 8933 (time 1709787728.76 total 509.95 last 16.79) - open/close 9104 (time 1709787739.24 total 520.43 last 16.31) - open/close 9240 (time 1709787749.78 total 530.98 last 12.90) - open/close 9373 (time 1709787759.93 total 541.12 last 13.11) - open/close 9531 (time 1709787770.04 total 551.23 last 15.63) - open/close 9583 (time 1709787780.35 total 561.54 last 5.05) - open/close 9908 (time 1709787790.36 total 571.55 last 32.45) - open/close 10000 (time 1709787799.31 total 580.50 last 10.28) - open/close 10191 (time 1709787810.00 total 591.19 last 17.87) - open/close 10619 (time 1709787820.01 total 601.20 last 42.74) - open/close 11120 (time 1709787839.90 total 621.09 last 25.19) - open/close 11332 (time 1709787855.70 total 636.89 last 13.42) - open/close 11808 (time 1709787865.72 total 646.91 last 47.50) - open/close 12357 (time 1709787875.73 total 656.92 last 54.85) - open/close 12861 (time 1709787885.74 total 666.93 last 50.36) - open/close 13318 (time 1709787895.74 total 676.93 last 45.69) - open/close 13780 (time 1709787905.76 total 686.95 last 46.11) - open/close 14261 (time 1709787915.77 total 696.96 last 48.02) - open/close 14703 (time 1709787925.80 total 706.99 last 44.08) - open/close 15162 (time 1709787935.81 total 717.00 last 45.85) - open/close 15648 (time 1709787945.82 total 727.01 last 48.53) - open/close 16185 (time 1709787955.84 total 737.03 last 53.63) - open/close 16711 (time 1709787965.85 total 747.04 last 52.55) - open/close 17200 (time 1709787975.86 total 757.06 last 48.82) - open/close 17689 (time 1709787985.88 total 767.07 last 48.82) - open/close 18180 (time 1709787995.89 total 777.08 last 49.05) - open/close 18759 (time 1709788005.90 total 787.09 last 57.87) - open/close 19327 (time 1709788015.90 total 797.09 last 56.76) - open/close 19893 (time 1709788025.92 total 807.11 last 56.51) - open/close 20000 (time 1709788027.95 total 809.14 last 52.70) - open/close 20588 (time 1709788037.95 total 819.14 last 58.78) - open/close 21161 (time 1709788047.95 total 829.14 last 57.30) - open/close 21729 (time 1709788057.96 total 839.15 last 56.75) - open/close 22250 (time 1709788067.97 total 849.16 last 52.07) - open/close 22777 (time 1709788077.97 total 859.17 last 52.67) - open/close 23344 (time 1709788087.98 total 869.18 last 56.64) - open/close 23843 (time 1709788097.99 total 879.18 last 49.87) - open/close 24347 (time 1709788108.00 total 889.19 last 50.36) - open/close 24851 (time 1709788118.01 total 899.21 last 50.32) - open/close 25322 (time 1709788128.03 total 909.22 last 47.02) - open/close 25828 (time 1709788138.04 total 919.23 last 50.55) - open/close 26313 (time 1709788148.06 total 929.25 last 48.41) - open/close 26796 (time 1709788158.08 total 939.27 last 48.22) - open/close 27285 (time 1709788168.09 total 949.28 last 48.84) - open/close 27760 (time 1709788178.11 total 959.30 last 47.41) - open/close 28222 (time 1709788188.13 total 969.32 last 46.10) - open/close 28703 (time 1709788198.14 total 979.33 last 48.04) - open/close 29156 (time 1709788208.15 total 989.34 last 45.29) - open/close 29625 (time 1709788218.16 total 999.36 last 46.81) - open/close 30000 (time 1709788226.41 total 1007.60 last 45.48) - open/close 30445 (time 1709788236.43 total 1017.62 last 44.42) - open/close 30891 (time 1709788246.43 total 1027.62 last 44.57) - open/close 31327 (time 1709788256.46 total 1037.65 last 43.49) - open/close 31745 (time 1709788266.47 total 1047.67 last 41.74) - open/close 32174 (time 1709788276.48 total 1057.67 last 42.86) - open/close 32568 (time 1709788286.49 total 1067.68 last 39.37) - open/close 32964 (time 1709788296.49 total 1077.69 last 39.59) - open/close 33367 (time 1709788306.50 total 1087.69 last 40.29) - open/close 33773 (time 1709788316.52 total 1097.71 last 40.52) - open/close 34157 (time 1709788326.52 total 1107.71 last 38.40) - open/close 34541 (time 1709788336.53 total 1117.72 last 38.37) - open/close 34927 (time 1709788346.55 total 1127.74 last 38.50) - open/close 35295 (time 1709788356.56 total 1137.75 last 36.76) - open/close 35672 (time 1709788366.56 total 1147.76 last 37.69) - open/close 36033 (time 1709788376.57 total 1157.76 last 36.08) - open/close 36392 (time 1709788386.59 total 1167.78 last 35.82) - open/close 36750 (time 1709788396.59 total 1177.78 last 35.79) - open/close 37128 (time 1709788406.61 total 1187.80 last 37.73) - open/close 37496 (time 1709788416.64 total 1197.83 last 36.71) - open/close 37854 (time 1709788426.64 total 1207.83 last 35.79) - open/close 38205 (time 1709788436.66 total 1217.85 last 35.05) - open/close 38546 (time 1709788446.67 total 1227.86 last 34.06) - open/close 38896 (time 1709788456.68 total 1237.87 last 34.95) - open/close 39234 (time 1709788466.70 total 1247.90 last 33.72) - open/close 39585 (time 1709788476.73 total 1257.93 last 35.00) - open/close 39943 (time 1709788486.75 total 1267.94 last 35.75) - open/close 40281 (time 1709788496.76 total 1277.95 last 33.75) - open/close 40620 (time 1709788506.77 total 1287.96 last 33.88) - open/close 40969 (time 1709788516.77 total 1297.96 last 34.89) - open/close 41307 (time 1709788526.79 total 1307.98 last 33.73) - open/close 41635 (time 1709788536.79 total 1317.98 last 32.80) - open/close 41965 (time 1709788546.80 total 1327.99 last 32.97) - open/close 42298 (time 1709788556.81 total 1338.00 last 33.26) - open/close 42642 (time 1709788566.84 total 1348.03 last 34.32) - open/close 42974 (time 1709788576.84 total 1358.03 last 33.18) - open/close 43311 (time 1709788586.87 total 1368.06 last 33.62) - open/close 43658 (time 1709788596.88 total 1378.07 last 34.67) - open/close 43993 (time 1709788606.88 total 1388.07 last 33.48) - open/close 44325 (time 1709788616.90 total 1398.09 last 33.14) - open/close 44659 (time 1709788626.91 total 1408.10 last 33.37) - open/close 44970 (time 1709788636.91 total 1418.10 last 31.10) - open/close 45274 (time 1709788646.93 total 1428.12 last 30.34) - open/close 45590 (time 1709788656.96 total 1438.15 last 31.50) - open/close 45893 (time 1709788666.97 total 1448.16 last 30.26) - open/close 46202 (time 1709788676.98 total 1458.17 last 30.87) - open/close 46505 (time 1709788687.01 total 1468.20 last 30.22) - open/close 46810 (time 1709788697.01 total 1478.20 last 30.49) - open/close 47110 (time 1709788707.04 total 1488.23 last 29.91) - open/close 47410 (time 1709788717.06 total 1498.26 last 29.93) - open/close 47719 (time 1709788727.08 total 1508.27 last 30.86) - open/close 48015 (time 1709788737.08 total 1518.27 last 29.58) - open/close 48302 (time 1709788747.09 total 1528.28 last 28.67) - open/close 48584 (time 1709788757.10 total 1538.29 last 28.19) - open/close 48872 (time 1709788767.12 total 1548.31 last 28.72) - open/close 49155 (time 1709788777.13 total 1558.32 last 28.29) - open/close 49433 (time 1709788787.13 total 1568.32 last 27.79) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49600) error: No space left on device total: 49600 open/close in 1574.26 seconds: 31.51 ops/second - unlinked 0 (time 1709788794 ; total 0 ; last 0) - unlinked 10000 (time 1709789243 ; total 449 ; last 449) - unlinked 20000 (time 1709789614 ; total 820 ; last 371) - unlinked 30000 (time 1709789870 ; total 1076 ; last 256) - unlinked 40000 (time 1709789993 ; total 1199 ; last 123) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49600) error: No such file or directory total: 49600 unlinks in 1235 seconds: 40.161942 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg256-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg256-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg256-server stop mds service on oleg256-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg256-server stop mds service on oleg256-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg256-server unloading modules on: 'oleg256-server' oleg256-server: oleg256-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg256-client: oleg256-client: ssh exited with exit code 2 pdsh@oleg256-client: oleg256-server: ssh exited with exit code 2 pdsh@oleg256-client: oleg256-client: ssh exited with exit code 2 pdsh@oleg256-client: oleg256-server: ssh exited with exit code 2 PASS 101a (2873s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg256-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg256-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg256-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg256-client: oleg256-server: ssh exited with exit code 2 oleg256-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg256-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == conf-sanity test 101b: Race events DISCONNECT and ACTIVE in osp ========================================================== 00:40:51 (1709790051) start mds service on oleg256-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg256-server' oleg256-server: oleg256-server.virtnet: executing load_modules_local oleg256-server: Loading modules from /home/green/git/lustre-release/lustre oleg256-server: detected 4 online CPUs by sysfs oleg256-server: Force libcfs to create 2 CPU partitions oleg256-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg256-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg256-server: oleg256-server.virtnet: executing set_default_debug -1 all pdsh@oleg256-client: oleg256-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg256-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg256-server: oleg256-server.virtnet: executing set_default_debug -1 all pdsh@oleg256-client: oleg256-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg256-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg256-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg256-server: oleg256-server.virtnet: executing set_default_debug -1 all pdsh@oleg256-client: oleg256-server: ssh exited with exit code 1 Started lustre-OST0000 oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg256-client.virtnet: -o user_xattr,flock oleg256-server@tcp:/lustre /mnt/lustre fail_loc=0x80002107 fail_val=20 stop ost1 service on oleg256-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg256-server start ost1 service on oleg256-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg256-server: oleg256-server.virtnet: executing set_default_debug -1 all pdsh@oleg256-client: oleg256-server: ssh exited with exit code 1 Started lustre-OST0000 oleg256-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 8 sec oleg256-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8800a8ccf800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800a8ccf800.ost_server_uuid in FULL state after 0 sec umount lustre on /mnt/lustre..... Stopping client oleg256-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg256-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg256-server stop mds service on oleg256-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg256-server stop mds service on oleg256-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg256-server unloading modules on: 'oleg256-server' oleg256-server: oleg256-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg256-client: oleg256-client: ssh exited with exit code 2 pdsh@oleg256-client: oleg256-server: ssh exited with exit code 2 pdsh@oleg256-client: oleg256-client: ssh exited with exit code 2 pdsh@oleg256-client: oleg256-server: ssh exited with exit code 2 PASS 101b (95s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg256-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg256-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory Stopping clients: oleg256-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg256-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg256-client: oleg256-server: ssh exited with exit code 2 oleg256-server: oleg256-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg256-server' oleg256-server: oleg256-server.virtnet: executing load_modules_local oleg256-server: Loading modules from /home/green/git/lustre-release/lustre oleg256-server: detected 4 online CPUs by sysfs oleg256-server: Force libcfs to create 2 CPU partitions oleg256-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg256-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey == conf-sanity test complete, duration 3667 sec ========== 00:42:46 (1709790166) === conf-sanity: start cleanup 00:42:46 (1709790166) === === conf-sanity: finish cleanup 00:42:46 (1709790166) ===