-----============= acceptance-small: conf-sanity ============----- Sun Mar 10 15:47:21 EDT 2024 excepting tests: 32newtarball 110 skipping tests SLOW=no: 45 69 106 111 114 Stopping clients: oleg254-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg254-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg254-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg254-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg254-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg254-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg254-server oleg254-server: oleg254-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg254-server' oleg254-server: oleg254-server.virtnet: executing load_modules_local oleg254-server: Loading modules from /home/green/git/lustre-release/lustre oleg254-server: detected 4 online CPUs by sysfs oleg254-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg254-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug -1 all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg254-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg254-server: oleg254-server.virtnet: executing set_default_debug -1 all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg254-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg254-server: oleg254-server.virtnet: executing set_default_debug -1 all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg254-server: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg254-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg254-server: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg254-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg254-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg254-server stop mds service on oleg254-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg254-server stop mds service on oleg254-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg254-server debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 15:48:44 (1710100124) start mds service on oleg254-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug -1 all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg254-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg254-server: oleg254-server.virtnet: executing set_default_debug -1 all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg254-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg254-server: oleg254-server.virtnet: executing set_default_debug -1 all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-OST0000 oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg254-server: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg254-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec oleg254-server: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg254-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 2 sec stop ost1 service on oleg254-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg254-server stop mds service on oleg254-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg254-server stop mds service on oleg254-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg254-server PASS 21a (33s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 15:49:18 (1710100158) start ost1 service on oleg254-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg254-server: oleg254-server.virtnet: executing set_default_debug -1 all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-OST0000 oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid start mds service on oleg254-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug -1 all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg254-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg254-server: oleg254-server.virtnet: executing set_default_debug -1 all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid oleg254-server: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg254-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg254-server: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg254-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop mds service on oleg254-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg254-server stop mds service on oleg254-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg254-server stop ost1 service on oleg254-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg254-server PASS 21b (223s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 15:53:04 (1710100384) start mds service on oleg254-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug -1 all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg254-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg254-server: oleg254-server.virtnet: executing set_default_debug -1 all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid Client mount with ost in logs, but none running start ost1 service on oleg254-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg254-server: oleg254-server.virtnet: executing set_default_debug -1 all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-OST0000 oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg254-server: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg254-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 10 sec oleg254-server: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg254-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg254-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg254-server mount lustre on /mnt/lustre..... Starting client: oleg254-client.virtnet: -o user_xattr,flock oleg254-server@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg254-client.virtnet /mnt/lustre (opts:-f) PASS Client mount with a running ost start ost1 service on oleg254-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg254-server: oleg254-server.virtnet: executing set_default_debug -1 all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-OST0000 oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg254-client.virtnet: -o user_xattr,flock oleg254-server@tcp:/lustre /mnt/lustre oleg254-server: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg254-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg254-server: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg254-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec oleg254-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8800ae138800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800ae138800.ost_server_uuid in FULL state after 0 sec setup single mount lustre success PASS umount lustre on /mnt/lustre..... Stopping client oleg254-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg254-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg254-server stop mds service on oleg254-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg254-server stop mds service on oleg254-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg254-server unloading modules on: 'oleg254-server' oleg254-server: oleg254-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg254-client: oleg254-client: ssh exited with exit code 2 pdsh@oleg254-client: oleg254-server: ssh exited with exit code 2 pdsh@oleg254-client: oleg254-server: ssh exited with exit code 2 pdsh@oleg254-client: oleg254-client: ssh exited with exit code 2 PASS 22 (83s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg254-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg254-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg254-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg254-client: oleg254-server: ssh exited with exit code 2 oleg254-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg254-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == conf-sanity test 46a: handle ost additional - wide striped file ========================================================== 15:54:30 (1710100470) Testing with 2 OSTs Stopping clients: oleg254-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg254-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg254-client: oleg254-server: ssh exited with exit code 2 oleg254-server: oleg254-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg254-server' oleg254-server: oleg254-server.virtnet: executing load_modules_local oleg254-server: Loading modules from /home/green/git/lustre-release/lustre oleg254-server: detected 4 online CPUs by sysfs oleg254-server: Force libcfs to create 2 CPU partitions oleg254-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg254-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg254-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug -1 all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg254-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg254-server: oleg254-server.virtnet: executing set_default_debug -1 all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg254-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg254-server: oleg254-server.virtnet: executing set_default_debug -1 all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg254-server: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg254-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg254-server: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg254-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg254-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg254-server stop mds service on oleg254-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg254-server stop mds service on oleg254-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg254-server start mds service on oleg254-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug -1 all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg254-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg254-server: oleg254-server.virtnet: executing set_default_debug -1 all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg254-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg254-server: oleg254-server.virtnet: executing set_default_debug -1 all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-OST0000 oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg254-server: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg254-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg254-server: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg254-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 2 sec mount lustre on /mnt/lustre..... Starting client: oleg254-client.virtnet: -o user_xattr,flock oleg254-server@tcp:/lustre /mnt/lustre Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg254-server: oleg254-server.virtnet: executing set_default_debug -1 all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost2_flakey Started lustre-OST0001 oleg254-server: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 oleg254-server: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 1 sec oleg254-server: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 oleg254-server: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec oleg254-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8800b6055800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b6055800.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre2..... Starting client: oleg254-client.virtnet: -o user_xattr,flock oleg254-server@tcp:/lustre /mnt/lustre2 /mnt/lustre2 stripe_count: -1 stripe_size: 4194304 pattern: raid0 stripe_offset: -1 /mnt/lustre2/widestripe lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 34 0x22 0x280000401 1 2 0x2 0x2c0000401 File: '/mnt/lustre/widestripe' Size: 3 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115238826934273 Links: 1 Access: (0674/-rw-rwxr--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-03-10 15:56:01.000000000 -0400 Modify: 2024-03-10 15:56:01.000000000 -0400 Change: 2024-03-10 15:56:01.000000000 -0400 Birth: - umount lustre on /mnt/lustre2..... Stopping client oleg254-client.virtnet /mnt/lustre2 (opts:) umount lustre on /mnt/lustre..... Stopping client oleg254-client.virtnet /mnt/lustre (opts:) Stopping /mnt/lustre-ost2 (opts:-f) on oleg254-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg254-server stop mds service on oleg254-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg254-server stop mds service on oleg254-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg254-server stop ost1 service on oleg254-server stop mds service on oleg254-server stop mds service on oleg254-server unloading modules on: 'oleg254-server' oleg254-server: oleg254-server.virtnet: executing unload_modules_local modules unloaded. oleg254-server: tunefs.lustre: Unable to mount /dev/mapper/mds1_flakey: No such device oleg254-server: Is the ldiskfs module available? oleg254-server: oleg254-server: tunefs.lustre FATAL: failed to write local files oleg254-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg254-client: oleg254-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x5 (MDT MGS ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity Permanent disk data: Target: lustre=MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x105 (MDT MGS writeconf ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity oleg254-server: tunefs.lustre: Unable to mount /dev/mapper/mds2_flakey: No such device oleg254-server: Is the ldiskfs module available? oleg254-server: oleg254-server: tunefs.lustre FATAL: failed to write local files oleg254-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg254-client: oleg254-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-MDT0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x1 (MDT ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: mgsnode=192.168.202.154@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity Permanent disk data: Target: lustre=MDT0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x101 (MDT writeconf ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: mgsnode=192.168.202.154@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity oleg254-server: tunefs.lustre: Unable to mount /dev/mapper/ost1_flakey: No such device oleg254-server: Is the ldiskfs module available? oleg254-server: oleg254-server: tunefs.lustre FATAL: failed to write local files oleg254-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg254-client: oleg254-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x2 (OST ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.202.154@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x102 (OST writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.202.154@tcp sys.timeout=20 oleg254-server: tunefs.lustre: Unable to mount /dev/mapper/ost2_flakey: No such device oleg254-server: Is the ldiskfs module available? oleg254-server: oleg254-server: tunefs.lustre FATAL: failed to write local files oleg254-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg254-client: oleg254-server: ssh exited with exit code 19 checking for existing Lustre data: found Read previous values: Target: lustre-OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x2 (OST ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.202.154@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x102 (OST writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.202.154@tcp sys.timeout=20 tunefs failed, reformatting instead Stopping clients: oleg254-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg254-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg254-client: oleg254-server: ssh exited with exit code 2 oleg254-server: oleg254-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg254-server' oleg254-server: oleg254-server.virtnet: executing load_modules_local oleg254-server: Loading modules from /home/green/git/lustre-release/lustre oleg254-server: detected 4 online CPUs by sysfs oleg254-server: Force libcfs to create 2 CPU partitions oleg254-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg254-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg254-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug -1 all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg254-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg254-server: oleg254-server.virtnet: executing set_default_debug -1 all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg254-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg254-server: oleg254-server.virtnet: executing set_default_debug -1 all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg254-server: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg254-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec oleg254-server: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg254-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg254-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg254-server stop mds service on oleg254-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg254-server stop mds service on oleg254-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg254-server PASS 46a (197s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 50b: lazystatfs all servers down ===== 15:57:49 (1710100669) start mds service on oleg254-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug -1 all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg254-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg254-server: oleg254-server.virtnet: executing set_default_debug -1 all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg254-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg254-server: oleg254-server.virtnet: executing set_default_debug -1 all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-OST0000 oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg254-client.virtnet: -o user_xattr,flock oleg254-server@tcp:/lustre /mnt/lustre llite.lustre-ffff88012b2af000.lazystatfs=1 stop ost1 service on oleg254-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg254-server oleg254-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff88012b2af000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff88012b2af000.ost_server_uuid in DISCONN state after 0 sec 0 UP mgc MGC192.168.202.154@tcp 0f2aa32f-822f-49f4-bb45-40727b9de188 5 1 UP lov lustre-clilov-ffff88012b2af000 12765da6-0f73-45a4-aa9f-1c4d3420894d 4 2 UP lmv lustre-clilmv-ffff88012b2af000 12765da6-0f73-45a4-aa9f-1c4d3420894d 5 3 UP mdc lustre-MDT0000-mdc-ffff88012b2af000 12765da6-0f73-45a4-aa9f-1c4d3420894d 5 4 UP mdc lustre-MDT0001-mdc-ffff88012b2af000 12765da6-0f73-45a4-aa9f-1c4d3420894d 5 5 UP osc lustre-OST0000-osc-ffff88012b2af000 12765da6-0f73-45a4-aa9f-1c4d3420894d 5 OSCs should all be DISCONN multiop /mnt/lustre vf_ TMPPIPE=/tmp/multiop_open_wait_pipe.7346 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 95248 1632 84960 2% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 95248 1508 85084 2% /mnt/lustre[MDT:1] filesystem_summary: 190496 3140 170044 2% /mnt/lustre /home/green/git/lustre-release/lustre/tests/conf-sanity.sh: line 4282: kill: (517) - No such process umount lustre on /mnt/lustre..... Stopping client oleg254-client.virtnet /mnt/lustre (opts:) stop mds service on oleg254-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg254-server stop mds service on oleg254-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg254-server PASS 50b (47s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 50c: lazystatfs one server down ====== 15:58:38 (1710100718) start mds service on oleg254-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug -1 all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg254-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg254-server: oleg254-server.virtnet: executing set_default_debug -1 all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg254-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg254-server: oleg254-server.virtnet: executing set_default_debug -1 all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-OST0000 oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid start ost2 service on oleg254-server Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg254-server: oleg254-server.virtnet: executing set_default_debug -1 all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost2_flakey Started lustre-OST0001 oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg254-client.virtnet: -o user_xattr,flock oleg254-server@tcp:/lustre /mnt/lustre llite.lustre-ffff8800aaeaf000.lazystatfs=1 stop ost1 service on oleg254-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg254-server oleg254-server: oleg254-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg254-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec oleg254-server: oleg254-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg254-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec multiop /mnt/lustre vf_ TMPPIPE=/tmp/multiop_open_wait_pipe.7346 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 95248 1668 84924 2% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 95248 1532 85060 2% /mnt/lustre[MDT:1] lustre-OST0001_UUID 142216 1524 126692 2% /mnt/lustre[OST:1] filesystem_summary: 142216 1524 126692 2% /mnt/lustre /home/green/git/lustre-release/lustre/tests/conf-sanity.sh: line 4282: kill: (3342) - No such process umount lustre on /mnt/lustre..... Stopping client oleg254-client.virtnet /mnt/lustre (opts:) stop ost2 service on oleg254-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg254-server stop mds service on oleg254-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg254-server stop mds service on oleg254-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg254-server checking for existing Lustre data: found Read previous values: Target: lustre-MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x5 (MDT MGS ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity Permanent disk data: Target: lustre=MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x105 (MDT MGS writeconf ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity checking for existing Lustre data: found Read previous values: Target: lustre-MDT0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x1 (MDT ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: mgsnode=192.168.202.154@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity Permanent disk data: Target: lustre=MDT0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x101 (MDT writeconf ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: mgsnode=192.168.202.154@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x2 (OST ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.202.154@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x102 (OST writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.202.154@tcp sys.timeout=20 checking for existing Lustre data: found Read previous values: Target: lustre-OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x2 (OST ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.202.154@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x102 (OST writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.202.154@tcp sys.timeout=20 PASS 50c (64s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 50e: normal statfs all servers down == 15:59:45 (1710100785) Stopping clients: oleg254-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg254-client.virtnet /mnt/lustre2 (opts:-f) oleg254-server: oleg254-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg254-server' oleg254-server: oleg254-server.virtnet: executing load_modules_local oleg254-server: Loading modules from /home/green/git/lustre-release/lustre oleg254-server: detected 4 online CPUs by sysfs oleg254-server: Force libcfs to create 2 CPU partitions oleg254-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg254-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug -1 all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg254-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg254-server: oleg254-server.virtnet: executing set_default_debug -1 all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg254-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg254-server: oleg254-server.virtnet: executing set_default_debug -1 all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg254-server: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg254-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg254-server: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg254-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg254-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg254-server stop mds service on oleg254-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg254-server stop mds service on oleg254-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg254-server start mds service on oleg254-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug -1 all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg254-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg254-server: oleg254-server.virtnet: executing set_default_debug -1 all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg254-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg254-server: oleg254-server.virtnet: executing set_default_debug -1 all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-OST0000 oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg254-server: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg254-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 9 sec oleg254-server: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg254-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg254-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg254-server oleg254-server: oleg254-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg254-server: oleg254-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 mount lustre on /mnt/lustre..... Starting client: oleg254-client.virtnet: -o user_xattr,flock oleg254-server@tcp:/lustre /mnt/lustre llite.lustre-ffff8800aaeaa000.lazystatfs=0 multiop /mnt/lustre v_f TMPPIPE=/tmp/multiop_open_wait_pipe.7346 start ost1 service on oleg254-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg254-server: oleg254-server.virtnet: executing set_default_debug -1 all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-OST0000 oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec umount lustre on /mnt/lustre..... Stopping client oleg254-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg254-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg254-server stop mds service on oleg254-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg254-server stop mds service on oleg254-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg254-server PASS 50e (152s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 50f: normal statfs one server in down ========================================================== 16:02:21 (1710100941) start mds service on oleg254-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug -1 all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg254-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg254-server: oleg254-server.virtnet: executing set_default_debug -1 all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg254-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg254-server: oleg254-server.virtnet: executing set_default_debug -1 all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-OST0000 oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg254-server: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg254-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 5 sec oleg254-server: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg254-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec start ost2 service on oleg254-server Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg254-server: oleg254-server.virtnet: executing set_default_debug -1 all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost2_flakey Started lustre-OST0001 oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid oleg254-server: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 oleg254-server: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg254-server: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 oleg254-server: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost2 service on oleg254-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg254-server oleg254-server: oleg254-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 oleg254-server: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec oleg254-server: oleg254-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 oleg254-server: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg254-client.virtnet: -o user_xattr,flock oleg254-server@tcp:/lustre /mnt/lustre llite.lustre-ffff88012afa6000.lazystatfs=0 multiop /mnt/lustre v_f TMPPIPE=/tmp/multiop_open_wait_pipe.7346 start ost2 service on oleg254-server Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg254-server: oleg254-server.virtnet: executing set_default_debug -1 all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-OST0001 oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec stop ost2 service on oleg254-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg254-server umount lustre on /mnt/lustre..... Stopping client oleg254-client.virtnet /mnt/lustre (opts:-f) stop ost1 service on oleg254-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg254-server stop mds service on oleg254-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg254-server stop mds service on oleg254-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg254-server checking for existing Lustre data: found Read previous values: Target: lustre-MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x5 (MDT MGS ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity Permanent disk data: Target: lustre=MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x105 (MDT MGS writeconf ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity checking for existing Lustre data: found Read previous values: Target: lustre-MDT0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x1 (MDT ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: mgsnode=192.168.202.154@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity Permanent disk data: Target: lustre=MDT0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x101 (MDT writeconf ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: mgsnode=192.168.202.154@tcp sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x2 (OST ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.202.154@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x102 (OST writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.202.154@tcp sys.timeout=20 checking for existing Lustre data: found Read previous values: Target: lustre-OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x2 (OST ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.202.154@tcp sys.timeout=20 Permanent disk data: Target: lustre=OST0001 Index: 1 Lustre FS: lustre Mount type: ldiskfs Flags: 0x102 (OST writeconf ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.202.154@tcp sys.timeout=20 PASS 50f (138s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 69: replace an OST with the same index ========================================================== 16:04:43 (1710101083) start mds service on oleg254-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug -1 all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg254-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg254-server: oleg254-server.virtnet: executing set_default_debug -1 all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg254-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg254-server: oleg254-server.virtnet: executing set_default_debug -1 all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-OST0000 oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg254-client.virtnet: -o user_xattr,flock oleg254-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49440 inodes available. Want 99872. rc=0 - open/close 1140 (time 1710101128.64 total 10.01 last 113.90) - open/close 1190 (time 1710101140.18 total 21.55 last 4.33) - open/close 2242 (time 1710101150.18 total 31.55 last 105.19) - open/close 3111 (time 1710101160.19 total 41.56 last 86.82) - open/close 4026 (time 1710101170.19 total 51.56 last 91.49) - open/close 4175 (time 1710101182.96 total 64.33 last 11.67) - open/close 4939 (time 1710101192.97 total 74.34 last 76.32) - open/close 5701 (time 1710101202.97 total 84.34 last 76.20) - open/close 6483 (time 1710101212.98 total 94.35 last 78.15) - open/close 7184 (time 1710101222.99 total 104.36 last 70.04) - open/close 7314 (time 1710101236.04 total 117.41 last 9.96) - open/close 7935 (time 1710101246.05 total 127.42 last 62.04) - open/close 8582 (time 1710101266.44 total 147.81 last 31.73) - open/close 9205 (time 1710101276.45 total 157.82 last 62.22) - open/close 9842 (time 1710101286.47 total 167.84 last 63.63) total: 10000 open/close in 170.37 seconds: 58.70 ops/second - unlinked 0 (time 1710101290 ; total 0 ; last 0) total: 10000 unlinks in 87 seconds: 114.942528 unlinks/second - open/close 1076 (time 1710101389.29 total 10.01 last 107.52) - open/close 2042 (time 1710101399.30 total 20.01 last 96.54) - open/close 2957 (time 1710101409.31 total 30.02 last 91.42) - open/close 3805 (time 1710101419.31 total 40.03 last 84.75) - open/close 4614 (time 1710101429.31 total 50.03 last 80.88) - open/close 5416 (time 1710101439.31 total 60.03 last 80.19) - open/close 6155 (time 1710101449.32 total 70.04 last 73.86) - open/close 6845 (time 1710101459.33 total 80.05 last 68.94) - open/close 7488 (time 1710101469.42 total 90.13 last 63.74) - open/close 8067 (time 1710101479.49 total 100.20 last 57.50) - open/close 8667 (time 1710101489.49 total 110.21 last 59.96) - open/close 9249 (time 1710101499.50 total 120.22 last 58.15) - open/close 9815 (time 1710101509.51 total 130.23 last 56.54) total: 10000 open/close in 133.79 seconds: 74.74 ops/second - unlinked 0 (time 1710101515 ; total 0 ; last 0) total: 10000 unlinks in 85 seconds: 117.647057 unlinks/second - open/close 6 (time 1710101613.79 total 11.14 last 0.54) - open/close 1040 (time 1710101634.52 total 31.88 last 49.86) - open/close 1970 (time 1710101644.52 total 41.88 last 93.00) - open/close 2885 (time 1710101654.53 total 51.88 last 91.49) - open/close 3374 (time 1710101670.99 total 68.35 last 29.69) - open/close 4248 (time 1710101681.00 total 78.35 last 87.35) - open/close 5060 (time 1710101691.01 total 88.36 last 81.14) - open/close 5854 (time 1710101701.01 total 98.37 last 79.36) - open/close 6595 (time 1710101711.01 total 108.37 last 74.08) - open/close 7343 (time 1710101721.03 total 118.38 last 74.70) - open/close 7354 (time 1710101732.26 total 129.61 last 0.98) - open/close 8026 (time 1710101742.27 total 139.63 last 67.12) - open/close 8604 (time 1710101752.28 total 149.63 last 57.75) - open/close 9217 (time 1710101762.29 total 159.65 last 61.23) - open/close 9847 (time 1710101772.30 total 169.65 last 62.95) total: 10000 open/close in 171.82 seconds: 58.20 ops/second - unlinked 0 (time 1710101776 ; total 0 ; last 0) total: 10000 unlinks in 81 seconds: 123.456787 unlinks/second - open/close 1093 (time 1710101869.92 total 10.00 last 109.27) - open/close 2033 (time 1710101879.93 total 20.01 last 93.94) - open/close 2927 (time 1710101889.94 total 30.02 last 89.33) - open/close 3715 (time 1710101909.69 total 49.77 last 39.89) - open/close 4528 (time 1710101919.70 total 59.78 last 81.23) - open/close 5317 (time 1710101929.70 total 69.78 last 78.90) - open/close 5997 (time 1710101939.71 total 79.79 last 67.91) - open/close 6617 (time 1710101960.00 total 100.08 last 30.56) - open/close 7263 (time 1710101970.01 total 110.09 last 64.55) - open/close 7968 (time 1710101980.01 total 120.09 last 70.50) - open/close 8641 (time 1710101990.02 total 130.10 last 67.26) - open/close 9287 (time 1710102000.04 total 140.11 last 64.48) - open/close 9907 (time 1710102010.04 total 150.12 last 61.98) total: 10000 open/close in 151.57 seconds: 65.98 ops/second - unlinked 0 (time 1710102013 ; total 0 ; last 0) total: 10000 unlinks in 91 seconds: 109.890106 unlinks/second - open/close 1047 (time 1710102116.38 total 10.00 last 104.69) - open/close 2012 (time 1710102126.38 total 20.00 last 96.46) - open/close 2950 (time 1710102136.39 total 30.01 last 93.72) - open/close 3788 (time 1710102146.39 total 40.01 last 83.80) - open/close 4621 (time 1710102156.39 total 50.01 last 83.30) - open/close 5423 (time 1710102166.40 total 60.02 last 80.18) - open/close 6203 (time 1710102176.40 total 70.02 last 77.98) - open/close 6935 (time 1710102186.41 total 80.03 last 73.14) - open/close 7600 (time 1710102196.41 total 90.03 last 66.46) - open/close 8296 (time 1710102206.42 total 100.04 last 69.57) - open/close 8914 (time 1710102216.42 total 110.04 last 61.80) - open/close 9533 (time 1710102226.42 total 120.04 last 61.89) total: 10000 open/close in 127.09 seconds: 78.68 ops/second - unlinked 0 (time 1710102235 ; total 0 ; last 0) total: 10000 unlinks in 93 seconds: 107.526878 unlinks/second - open/close 818 (time 1710102349.36 total 18.65 last 43.87) - open/close 1893 (time 1710102359.37 total 28.65 last 107.45) - open/close 2821 (time 1710102369.37 total 38.65 last 92.78) - open/close 3340 (time 1710102386.17 total 55.46 last 30.88) - open/close 4186 (time 1710102396.18 total 65.47 last 84.55) - open/close 5012 (time 1710102406.19 total 75.48 last 82.53) - open/close 5802 (time 1710102416.20 total 85.48 last 78.94) - open/close 6550 (time 1710102426.20 total 95.49 last 74.78) - open/close 7290 (time 1710102436.20 total 105.49 last 73.99) - open/close 7534 (time 1710102450.57 total 119.86 last 16.98) - open/close 8210 (time 1710102460.57 total 129.86 last 67.59) - open/close 8938 (time 1710102470.57 total 139.86 last 72.80) - open/close 9553 (time 1710102480.58 total 149.86 last 61.48) total: 10000 open/close in 156.60 seconds: 63.86 ops/second - unlinked 0 (time 1710102489 ; total 0 ; last 0) total: 10000 unlinks in 77 seconds: 129.870132 unlinks/second - open/close 1093 (time 1710102578.44 total 10.00 last 109.28) - open/close 2123 (time 1710102588.45 total 20.01 last 102.93) - open/close 3062 (time 1710102598.45 total 30.02 last 93.84) - open/close 4038 (time 1710102608.46 total 40.02 last 97.53) - open/close 4873 (time 1710102618.48 total 50.05 last 83.30) - open/close 5735 (time 1710102628.48 total 60.05 last 86.20) - open/close 6528 (time 1710102638.50 total 70.06 last 79.17) - open/close 7316 (time 1710102648.50 total 80.07 last 78.77) - open/close 8059 (time 1710102658.50 total 90.07 last 74.30) - open/close 8746 (time 1710102668.51 total 100.08 last 68.62) - open/close 9421 (time 1710102678.52 total 110.09 last 67.43) total: 10000 open/close in 118.36 seconds: 84.49 ops/second - unlinked 0 (time 1710102688 ; total 0 ; last 0) total: 10000 unlinks in 74 seconds: 135.135132 unlinks/second - open/close 626 (time 1710102779.64 total 15.66 last 39.98) - open/close 1820 (time 1710102789.65 total 25.67 last 119.30) - open/close 2903 (time 1710102799.66 total 35.68 last 108.22) - open/close 3943 (time 1710102809.67 total 45.68 last 103.93) - open/close 4981 (time 1710102819.67 total 55.68 last 103.78) - open/close 5976 (time 1710102829.67 total 65.68 last 99.49) - open/close 6923 (time 1710102839.67 total 75.69 last 94.65) - open/close 7828 (time 1710102849.68 total 85.70 last 90.45) - open/close 8700 (time 1710102859.68 total 95.70 last 87.19) - open/close 9220 (time 1710102876.91 total 112.92 last 30.18) total: 10000 open/close in 122.15 seconds: 81.86 ops/second - unlinked 0 (time 1710102887 ; total 0 ; last 0) total: 10000 unlinks in 57 seconds: 175.438599 unlinks/second - open/close 1729 (time 1710102956.15 total 10.00 last 172.87) - open/close 3347 (time 1710102966.16 total 20.01 last 161.74) - open/close 4854 (time 1710102976.16 total 30.01 last 150.69) - open/close 6475 (time 1710102986.16 total 40.01 last 162.02) - open/close 8093 (time 1710102996.17 total 50.02 last 161.74) - open/close 9472 (time 1710103006.17 total 60.02 last 137.85) total: 10000 open/close in 64.10 seconds: 156.00 ops/second - unlinked 0 (time 1710103011 ; total 0 ; last 0) total: 10000 unlinks in 40 seconds: 250.000000 unlinks/second - open/close 2963 (time 1710103061.97 total 10.00 last 296.21) - open/close 5200 (time 1710103071.97 total 20.00 last 223.68) - open/close 7129 (time 1710103081.97 total 30.01 last 192.81) - open/close 8951 (time 1710103091.97 total 40.01 last 182.19) total: 10000 open/close in 46.11 seconds: 216.87 ops/second - unlinked 0 (time 1710103098 ; total 0 ; last 0) total: 10000 unlinks in 27 seconds: 370.370361 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg254-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg254-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg254-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.202.154@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k Writing CONFIGS/mountdata start ost1 service on oleg254-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg254-server: oleg254-server.virtnet: executing set_default_debug -1 all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-OST0000 oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg254-server: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg254-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 10 sec oleg254-server: oleg254-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg254-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg254-client.virtnet: -o user_xattr,flock oleg254-server@tcp:/lustre /mnt/lustre On OST0, 17649 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg254-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg254-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg254-server stop mds service on oleg254-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg254-server stop mds service on oleg254-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg254-server unloading modules on: 'oleg254-server' oleg254-server: oleg254-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg254-client: oleg254-client: ssh exited with exit code 2 pdsh@oleg254-client: oleg254-server: ssh exited with exit code 2 pdsh@oleg254-client: oleg254-client: ssh exited with exit code 2 pdsh@oleg254-client: oleg254-server: ssh exited with exit code 2 PASS 69 (2088s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg254-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg254-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg254-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg254-client: oleg254-server: ssh exited with exit code 2 oleg254-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg254-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == conf-sanity test 93: register mulitple MDT at the same time ========================================================== 16:39:33 (1710103173) SKIP: conf-sanity test_93 needs >= 3 MDTs SKIP 93 (1s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg254-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg254-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory Stopping clients: oleg254-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg254-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg254-client: oleg254-server: ssh exited with exit code 2 oleg254-server: oleg254-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg254-server' oleg254-server: oleg254-server.virtnet: executing load_modules_local oleg254-server: Loading modules from /home/green/git/lustre-release/lustre oleg254-server: detected 4 online CPUs by sysfs oleg254-server: Force libcfs to create 2 CPU partitions oleg254-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg254-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg254-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey == conf-sanity test complete, duration 3153 sec ========== 16:39:54 (1710103194) === conf-sanity: start cleanup 16:39:54 (1710103194) === === conf-sanity: finish cleanup 16:39:54 (1710103194) ===