-----============= acceptance-small: conf-sanity ============----- Sun Mar 10 15:45:31 EDT 2024 excepting tests: 32newtarball 110 skipping tests SLOW=no: 45 69 106 111 114 Stopping clients: oleg116-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg116-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg116-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg116-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg116-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg116-server oleg116-server: oleg116-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory loading modules on: 'oleg116-server' oleg116-server: oleg116-server.virtnet: executing load_modules_local oleg116-server: Loading modules from /home/green/git/lustre-release/lustre oleg116-server: detected 4 online CPUs by sysfs oleg116-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg116-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg116-server: oleg116-server.virtnet: executing set_default_debug -1 all pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg116-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg116-server: oleg116-server.virtnet: executing set_default_debug -1 all pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg116-server: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg116-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg116-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg116-server stop mds service on oleg116-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg116-server debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 15:47:01 (1710100021) start mds service on oleg116-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg116-server: oleg116-server.virtnet: executing set_default_debug -1 all pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg116-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg116-server: oleg116-server.virtnet: executing set_default_debug -1 all pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 Started lustre-OST0000 oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg116-server: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg116-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 4 sec stop ost1 service on oleg116-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg116-server stop mds service on oleg116-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg116-server PASS 21a (21s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 15:47:24 (1710100044) start ost1 service on oleg116-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg116-server: oleg116-server.virtnet: executing set_default_debug -1 all pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 Started lustre-OST0000 oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid start mds service on oleg116-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg116-server: oleg116-server.virtnet: executing set_default_debug -1 all pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg116-server: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg116-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop mds service on oleg116-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg116-server stop ost1 service on oleg116-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg116-server PASS 21b (196s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 15:50:42 (1710100242) start mds service on oleg116-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg116-server: oleg116-server.virtnet: executing set_default_debug -1 all pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid Client mount with ost in logs, but none running start ost1 service on oleg116-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg116-server: oleg116-server.virtnet: executing set_default_debug -1 all pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 Started lustre-OST0000 oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg116-server: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg116-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec stop ost1 service on oleg116-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg116-server mount lustre on /mnt/lustre..... Starting client: oleg116-client.virtnet: -o user_xattr,flock oleg116-server@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg116-client.virtnet /mnt/lustre (opts:-f) PASS Client mount with a running ost start ost1 service on oleg116-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg116-server: oleg116-server.virtnet: executing set_default_debug -1 all pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 Started lustre-OST0000 oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg116-client.virtnet: -o user_xattr,flock oleg116-server@tcp:/lustre /mnt/lustre oleg116-server: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg116-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec oleg116-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8800aae62000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800aae62000.ost_server_uuid in FULL state after 0 sec setup single mount lustre success PASS umount lustre on /mnt/lustre..... Stopping client oleg116-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg116-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg116-server stop mds service on oleg116-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg116-server unloading modules on: 'oleg116-server' oleg116-server: oleg116-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg116-client: oleg116-client: ssh exited with exit code 2 pdsh@oleg116-client: oleg116-server: ssh exited with exit code 2 pdsh@oleg116-client: oleg116-client: ssh exited with exit code 2 pdsh@oleg116-client: oleg116-server: ssh exited with exit code 2 PASS 22 (52s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg116-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg116-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg116-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg116-client: oleg116-server: ssh exited with exit code 2 oleg116-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg116-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == conf-sanity test 46a: handle ost additional - wide striped file ========================================================== 15:51:35 (1710100295) Testing with 2 OSTs Stopping clients: oleg116-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg116-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg116-client: oleg116-server: ssh exited with exit code 2 oleg116-server: oleg116-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg116-server' oleg116-server: oleg116-server.virtnet: executing load_modules_local oleg116-server: Loading modules from /home/green/git/lustre-release/lustre oleg116-server: detected 4 online CPUs by sysfs oleg116-server: Force libcfs to create 2 CPU partitions oleg116-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg116-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg116-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg116-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg116-server: oleg116-server.virtnet: executing set_default_debug -1 all pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg116-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg116-server: oleg116-server.virtnet: executing set_default_debug -1 all pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg116-server: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg116-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg116-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg116-server stop mds service on oleg116-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg116-server start mds service on oleg116-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg116-server: oleg116-server.virtnet: executing set_default_debug -1 all pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg116-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg116-server: oleg116-server.virtnet: executing set_default_debug -1 all pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 Started lustre-OST0000 oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg116-server: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg116-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 4 sec mount lustre on /mnt/lustre..... Starting client: oleg116-client.virtnet: -o user_xattr,flock oleg116-server@tcp:/lustre /mnt/lustre Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg116-server: oleg116-server.virtnet: executing set_default_debug -1 all pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 Commit the device label on lustre-ost2/ost2 Started lustre-OST0001 oleg116-server: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 oleg116-server: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg116-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff88012db0c000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff88012db0c000.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre2..... Starting client: oleg116-client.virtnet: -o user_xattr,flock oleg116-server@tcp:/lustre /mnt/lustre2 /mnt/lustre2 stripe_count: -1 stripe_size: 4194304 pattern: raid0 stripe_offset: -1 /mnt/lustre2/widestripe lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 2 0x2 0x280000400 0 34 0x22 0x240000400 File: '/mnt/lustre/widestripe' Size: 3 Blocks: 2 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115238826934273 Links: 1 Access: (0674/-rw-rwxr--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-03-10 15:52:31.000000000 -0400 Modify: 2024-03-10 15:52:31.000000000 -0400 Change: 2024-03-10 15:52:32.000000000 -0400 Birth: - umount lustre on /mnt/lustre2..... Stopping client oleg116-client.virtnet /mnt/lustre2 (opts:) umount lustre on /mnt/lustre..... Stopping client oleg116-client.virtnet /mnt/lustre (opts:) Stopping /mnt/lustre-ost2 (opts:-f) on oleg116-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg116-server stop mds service on oleg116-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg116-server stop ost1 service on oleg116-server stop mds service on oleg116-server unloading modules on: 'oleg116-server' oleg116-server: oleg116-server.virtnet: executing unload_modules_local modules unloaded. oleg116-server: oleg116-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre oleg116-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg116-client: oleg116-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg116-server: oleg116-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre oleg116-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg116-client: oleg116-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg116-server: oleg116-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre oleg116-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg116-client: oleg116-server: ssh exited with exit code 19 checking for existing Lustre data: not found tunefs failed, reformatting instead Stopping clients: oleg116-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg116-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg116-client: oleg116-server: ssh exited with exit code 2 oleg116-server: oleg116-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg116-server' oleg116-server: oleg116-server.virtnet: executing load_modules_local oleg116-server: Loading modules from /home/green/git/lustre-release/lustre oleg116-server: detected 4 online CPUs by sysfs oleg116-server: Force libcfs to create 2 CPU partitions oleg116-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg116-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg116-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg116-server: oleg116-server.virtnet: executing set_default_debug -1 all pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg116-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg116-server: oleg116-server.virtnet: executing set_default_debug -1 all pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg116-server: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg116-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg116-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg116-server stop mds service on oleg116-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg116-server PASS 46a (150s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 50b: lazystatfs all servers down ===== 15:54:07 (1710100447) start mds service on oleg116-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg116-server: oleg116-server.virtnet: executing set_default_debug -1 all pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg116-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg116-server: oleg116-server.virtnet: executing set_default_debug -1 all pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 Started lustre-OST0000 oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg116-client.virtnet: -o user_xattr,flock oleg116-server@tcp:/lustre /mnt/lustre llite.lustre-ffff8800a9674800.lazystatfs=1 stop ost1 service on oleg116-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg116-server oleg116-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff8800a9674800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800a9674800.ost_server_uuid in DISCONN state after 0 sec 0 UP mgc MGC192.168.201.116@tcp e76c9293-34f5-4115-bfec-04c6871edee1 5 1 UP lov lustre-clilov-ffff8800a9674800 f8fc0e86-b21d-4220-ad06-dfdbda5d6802 4 2 UP lmv lustre-clilmv-ffff8800a9674800 f8fc0e86-b21d-4220-ad06-dfdbda5d6802 5 3 UP mdc lustre-MDT0000-mdc-ffff8800a9674800 f8fc0e86-b21d-4220-ad06-dfdbda5d6802 5 4 UP osc lustre-OST0000-osc-ffff8800a9674800 f8fc0e86-b21d-4220-ad06-dfdbda5d6802 5 OSCs should all be DISCONN multiop /mnt/lustre vf_ TMPPIPE=/tmp/multiop_open_wait_pipe.6721 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 383616 3200 378368 1% /mnt/lustre[MDT:0] filesystem_summary: 383616 3200 378368 1% /mnt/lustre /home/green/git/lustre-release/lustre/tests/conf-sanity.sh: line 4282: kill: (27473) - No such process umount lustre on /mnt/lustre..... Stopping client oleg116-client.virtnet /mnt/lustre (opts:) stop mds service on oleg116-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg116-server PASS 50b (43s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 50c: lazystatfs one server down ====== 15:54:54 (1710100494) start mds service on oleg116-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg116-server: oleg116-server.virtnet: executing set_default_debug -1 all pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg116-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg116-server: oleg116-server.virtnet: executing set_default_debug -1 all pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 Started lustre-OST0000 oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid start ost2 service on oleg116-server Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg116-server: oleg116-server.virtnet: executing set_default_debug -1 all pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 Commit the device label on lustre-ost2/ost2 Started lustre-OST0001 oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg116-client.virtnet: -o user_xattr,flock oleg116-server@tcp:/lustre /mnt/lustre llite.lustre-ffff8800b5c7f000.lazystatfs=1 stop ost1 service on oleg116-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg116-server oleg116-server: oleg116-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg116-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec multiop /mnt/lustre vf_ TMPPIPE=/tmp/multiop_open_wait_pipe.6721 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 383616 3328 378240 1% /mnt/lustre[MDT:0] lustre-OST0001_UUID 382976 3072 377856 1% /mnt/lustre[OST:1] filesystem_summary: 382976 3072 377856 1% /mnt/lustre /home/green/git/lustre-release/lustre/tests/conf-sanity.sh: line 4282: kill: (29716) - No such process umount lustre on /mnt/lustre..... Stopping client oleg116-client.virtnet /mnt/lustre (opts:) stop ost2 service on oleg116-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg116-server stop mds service on oleg116-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg116-server oleg116-server: oleg116-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre oleg116-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg116-client: oleg116-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg116-server: oleg116-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre oleg116-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg116-client: oleg116-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg116-server: oleg116-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre oleg116-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg116-client: oleg116-server: ssh exited with exit code 19 checking for existing Lustre data: not found tunefs failed, reformatting instead Stopping clients: oleg116-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg116-client.virtnet /mnt/lustre2 (opts:-f) oleg116-server: oleg116-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg116-server' oleg116-server: oleg116-server.virtnet: executing load_modules_local oleg116-server: Loading modules from /home/green/git/lustre-release/lustre oleg116-server: detected 4 online CPUs by sysfs oleg116-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg116-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg116-server: oleg116-server.virtnet: executing set_default_debug -1 all pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg116-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg116-server: oleg116-server.virtnet: executing set_default_debug -1 all pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg116-server: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg116-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg116-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg116-server stop mds service on oleg116-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg116-server PASS 50c (87s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 50e: normal statfs all servers down == 15:56:22 (1710100582) Stopping clients: oleg116-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg116-client.virtnet /mnt/lustre2 (opts:-f) oleg116-server: oleg116-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg116-server' oleg116-server: oleg116-server.virtnet: executing load_modules_local oleg116-server: Loading modules from /home/green/git/lustre-release/lustre oleg116-server: detected 4 online CPUs by sysfs oleg116-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg116-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg116-server: oleg116-server.virtnet: executing set_default_debug -1 all pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg116-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg116-server: oleg116-server.virtnet: executing set_default_debug -1 all pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg116-server: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg116-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg116-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg116-server stop mds service on oleg116-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg116-server start mds service on oleg116-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg116-server: oleg116-server.virtnet: executing set_default_debug -1 all pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg116-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg116-server: oleg116-server.virtnet: executing set_default_debug -1 all pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 Started lustre-OST0000 oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg116-server: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg116-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg116-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg116-server oleg116-server: oleg116-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 mount lustre on /mnt/lustre..... Starting client: oleg116-client.virtnet: -o user_xattr,flock oleg116-server@tcp:/lustre /mnt/lustre llite.lustre-ffff88012abad000.lazystatfs=0 multiop /mnt/lustre v_f TMPPIPE=/tmp/multiop_open_wait_pipe.6721 start ost1 service on oleg116-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg116-server: oleg116-server.virtnet: executing set_default_debug -1 all pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 Started lustre-OST0000 oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec umount lustre on /mnt/lustre..... Stopping client oleg116-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg116-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg116-server stop mds service on oleg116-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg116-server PASS 50e (117s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 50f: normal statfs one server in down ========================================================== 15:58:21 (1710100701) start mds service on oleg116-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg116-server: oleg116-server.virtnet: executing set_default_debug -1 all pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg116-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg116-server: oleg116-server.virtnet: executing set_default_debug -1 all pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 Started lustre-OST0000 oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg116-server: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg116-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec start ost2 service on oleg116-server Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg116-server: oleg116-server.virtnet: executing set_default_debug -1 all pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 Commit the device label on lustre-ost2/ost2 Started lustre-OST0001 oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid oleg116-server: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 oleg116-server: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost2 service on oleg116-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg116-server oleg116-server: oleg116-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 oleg116-server: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg116-client.virtnet: -o user_xattr,flock oleg116-server@tcp:/lustre /mnt/lustre llite.lustre-ffff8800b6ccb000.lazystatfs=0 multiop /mnt/lustre v_f TMPPIPE=/tmp/multiop_open_wait_pipe.6721 start ost2 service on oleg116-server Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg116-server: oleg116-server.virtnet: executing set_default_debug -1 all pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 Started lustre-OST0001 oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec stop ost2 service on oleg116-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg116-server umount lustre on /mnt/lustre..... Stopping client oleg116-client.virtnet /mnt/lustre (opts:-f) stop ost1 service on oleg116-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg116-server stop mds service on oleg116-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg116-server oleg116-server: oleg116-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre oleg116-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg116-client: oleg116-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg116-server: oleg116-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre oleg116-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg116-client: oleg116-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg116-server: oleg116-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre oleg116-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg116-client: oleg116-server: ssh exited with exit code 19 checking for existing Lustre data: not found tunefs failed, reformatting instead Stopping clients: oleg116-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg116-client.virtnet /mnt/lustre2 (opts:-f) oleg116-server: oleg116-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg116-server' oleg116-server: oleg116-server.virtnet: executing load_modules_local oleg116-server: Loading modules from /home/green/git/lustre-release/lustre oleg116-server: detected 4 online CPUs by sysfs oleg116-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg116-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg116-server: oleg116-server.virtnet: executing set_default_debug -1 all pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg116-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg116-server: oleg116-server.virtnet: executing set_default_debug -1 all pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg116-server: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg116-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg116-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg116-server stop mds service on oleg116-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg116-server PASS 50f (126s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 69: replace an OST with the same index ========================================================== 16:00:30 (1710100830) start mds service on oleg116-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg116-server: oleg116-server.virtnet: executing set_default_debug -1 all pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg116-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg116-server: oleg116-server.virtnet: executing set_default_debug -1 all pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 Started lustre-OST0000 oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg116-client.virtnet: -o user_xattr,flock oleg116-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11872 inodes available. Want 99936. rc=0 - open/close 227 (time 1710100861.18 total 13.07 last 17.37) - open/close 1277 (time 1710100871.19 total 23.08 last 104.90) - open/close 2195 (time 1710100881.19 total 33.08 last 91.79) - open/close 3139 (time 1710100891.20 total 43.09 last 94.26) - open/close 3991 (time 1710100901.20 total 53.09 last 85.20) - open/close 4698 (time 1710100911.21 total 63.10 last 70.67) - open/close 5421 (time 1710100921.21 total 73.10 last 72.30) - open/close 6158 (time 1710100931.21 total 83.11 last 73.65) - open/close 6878 (time 1710100941.23 total 93.12 last 71.91) - open/close 7600 (time 1710100951.24 total 103.13 last 72.10) - open/close 8293 (time 1710100961.25 total 113.14 last 69.26) - open/close 8893 (time 1710100971.26 total 123.15 last 59.91) - open/close 9487 (time 1710100981.31 total 133.20 last 59.10) total: 10000 open/close in 143.04 seconds: 69.91 ops/second - unlinked 0 (time 1710100993 ; total 0 ; last 0) total: 10000 unlinks in 89 seconds: 112.359550 unlinks/second - open/close 1054 (time 1710101094.25 total 10.00 last 105.36) - open/close 2028 (time 1710101104.25 total 20.00 last 97.39) - open/close 2878 (time 1710101114.26 total 30.01 last 84.92) - open/close 3598 (time 1710101124.26 total 40.02 last 71.97) - open/close 4205 (time 1710101134.27 total 50.03 last 60.63) - open/close 4821 (time 1710101144.28 total 60.03 last 61.58) - open/close 5487 (time 1710101154.28 total 70.03 last 66.58) - open/close 6113 (time 1710101164.29 total 80.04 last 62.55) - open/close 6675 (time 1710101174.30 total 90.05 last 56.14) - open/close 7220 (time 1710101184.31 total 100.06 last 54.44) - open/close 7817 (time 1710101194.31 total 110.07 last 59.70) - open/close 8312 (time 1710101204.33 total 120.09 last 49.39) - open/close 8817 (time 1710101214.34 total 130.09 last 50.47) - open/close 9288 (time 1710101224.34 total 140.10 last 47.08) - open/close 9706 (time 1710101234.35 total 150.11 last 41.76) total: 10000 open/close in 157.07 seconds: 63.66 ops/second - unlinked 0 (time 1710101243 ; total 0 ; last 0) total: 10000 unlinks in 116 seconds: 86.206894 unlinks/second - open/close 809 (time 1710101372.08 total 10.00 last 80.89) - open/close 1588 (time 1710101382.10 total 20.01 last 77.80) - open/close 2336 (time 1710101392.11 total 30.02 last 74.72) - open/close 3040 (time 1710101402.11 total 40.03 last 70.38) - open/close 3679 (time 1710101412.12 total 50.04 last 63.84) - open/close 4266 (time 1710101422.14 total 60.06 last 58.59) - open/close 4885 (time 1710101432.15 total 70.07 last 61.83) - open/close 5450 (time 1710101442.15 total 80.07 last 56.48) - open/close 6033 (time 1710101452.16 total 90.08 last 58.23) - open/close 6546 (time 1710101462.21 total 100.12 last 51.09) - open/close 7036 (time 1710101472.21 total 110.13 last 48.96) - open/close 7546 (time 1710101482.24 total 120.16 last 50.87) - open/close 8066 (time 1710101492.26 total 130.18 last 51.87) - open/close 8533 (time 1710101502.27 total 140.19 last 46.66) - open/close 8986 (time 1710101512.29 total 150.21 last 45.20) - open/close 9340 (time 1710101530.56 total 168.48 last 19.38) - open/close 9728 (time 1710101540.57 total 178.49 last 38.76) total: 10000 open/close in 184.80 seconds: 54.11 ops/second - unlinked 0 (time 1710101549 ; total 0 ; last 0) total: 10000 unlinks in 115 seconds: 86.956520 unlinks/second - open/close 830 (time 1710101677.10 total 10.00 last 83.00) - open/close 1701 (time 1710101687.10 total 20.00 last 87.06) - open/close 2455 (time 1710101697.12 total 30.02 last 75.30) - open/close 3175 (time 1710101707.17 total 40.07 last 71.60) - open/close 3842 (time 1710101717.21 total 50.11 last 66.47) - open/close 4423 (time 1710101727.21 total 60.11 last 58.09) - open/close 5029 (time 1710101737.21 total 70.12 last 60.55) - open/close 5595 (time 1710101747.22 total 80.12 last 56.59) - open/close 6148 (time 1710101757.23 total 90.14 last 55.20) - open/close 6685 (time 1710101767.27 total 100.17 last 53.53) - open/close 7219 (time 1710101777.28 total 110.18 last 53.35) - open/close 7765 (time 1710101787.29 total 120.19 last 54.54) - open/close 8265 (time 1710101797.30 total 130.21 last 49.92) - open/close 8748 (time 1710101807.41 total 140.31 last 47.80) - open/close 9205 (time 1710101817.43 total 150.33 last 45.61) - open/close 9632 (time 1710101827.43 total 160.34 last 42.68) total: 10000 open/close in 169.17 seconds: 59.11 ops/second - unlinked 0 (time 1710101838 ; total 0 ; last 0) total: 10000 unlinks in 111 seconds: 90.090088 unlinks/second - open/close 876 (time 1710101961.99 total 10.00 last 87.59) - open/close 1659 (time 1710101971.99 total 20.00 last 78.29) - open/close 2372 (time 1710101981.99 total 30.00 last 71.29) - open/close 3108 (time 1710101992.01 total 40.02 last 73.48) - open/close 3781 (time 1710102002.01 total 50.03 last 67.26) - open/close 4403 (time 1710102012.03 total 60.04 last 62.12) - open/close 5006 (time 1710102022.04 total 70.05 last 60.22) - open/close 5482 (time 1710102041.88 total 89.90 last 23.99) - open/close 6076 (time 1710102051.89 total 99.90 last 59.37) - open/close 6576 (time 1710102061.89 total 109.90 last 49.99) - open/close 7152 (time 1710102071.90 total 119.91 last 57.54) - open/close 7648 (time 1710102081.92 total 129.93 last 49.50) - open/close 8157 (time 1710102091.92 total 139.94 last 50.89) - open/close 8625 (time 1710102101.96 total 149.97 last 46.65) - open/close 9031 (time 1710102111.97 total 159.98 last 40.56) - open/close 9479 (time 1710102121.97 total 169.99 last 44.77) - open/close 9954 (time 1710102131.98 total 180.00 last 47.45) total: 10000 open/close in 180.79 seconds: 55.31 ops/second - unlinked 0 (time 1710102135 ; total 0 ; last 0) total: 10000 unlinks in 139 seconds: 71.942444 unlinks/second - open/close 869 (time 1710102287.42 total 10.01 last 86.85) - open/close 1621 (time 1710102297.43 total 20.02 last 75.10) - open/close 2367 (time 1710102307.44 total 30.03 last 74.54) - open/close 3102 (time 1710102317.45 total 40.03 last 73.45) - open/close 3699 (time 1710102327.46 total 50.04 last 59.65) - open/close 4320 (time 1710102337.47 total 60.05 last 62.02) - open/close 4936 (time 1710102347.47 total 70.06 last 61.59) - open/close 5534 (time 1710102357.49 total 80.07 last 59.72) - open/close 6133 (time 1710102367.50 total 90.08 last 59.83) - open/close 6726 (time 1710102377.50 total 100.09 last 59.26) - open/close 7189 (time 1710102387.51 total 110.09 last 46.28) - open/close 7625 (time 1710102397.52 total 120.10 last 43.56) - open/close 8113 (time 1710102407.53 total 130.11 last 48.76) - open/close 8595 (time 1710102417.53 total 140.11 last 48.18) - open/close 9084 (time 1710102427.54 total 150.13 last 48.83) - open/close 9611 (time 1710102437.56 total 160.14 last 52.63) total: 10000 open/close in 168.98 seconds: 59.18 ops/second - unlinked 0 (time 1710102449 ; total 0 ; last 0) total: 10000 unlinks in 124 seconds: 80.645164 unlinks/second - open/close 873 (time 1710102585.87 total 10.00 last 87.29) - open/close 1649 (time 1710102595.87 total 20.01 last 77.57) - open/close 2389 (time 1710102605.87 total 30.01 last 73.97) - open/close 3042 (time 1710102615.88 total 40.02 last 65.26) - open/close 3650 (time 1710102625.89 total 50.03 last 60.74) - open/close 4283 (time 1710102635.90 total 60.04 last 63.23) - open/close 4886 (time 1710102645.90 total 70.04 last 60.28) - open/close 5456 (time 1710102655.93 total 80.07 last 56.86) - open/close 6017 (time 1710102665.94 total 90.07 last 56.05) - open/close 6378 (time 1710102684.46 total 108.60 last 19.49) - open/close 6896 (time 1710102694.50 total 118.64 last 51.59) - open/close 7363 (time 1710102704.52 total 128.65 last 46.62) - open/close 7820 (time 1710102714.53 total 138.66 last 45.66) - open/close 8263 (time 1710102724.54 total 148.67 last 44.25) - open/close 8697 (time 1710102734.56 total 158.69 last 43.31) - open/close 9187 (time 1710102744.57 total 168.71 last 48.92) - open/close 9623 (time 1710102754.58 total 178.71 last 43.59) total: 10000 open/close in 186.31 seconds: 53.67 ops/second - unlinked 0 (time 1710102765 ; total 0 ; last 0) total: 10000 unlinks in 124 seconds: 80.645164 unlinks/second - open/close 791 (time 1710102901.52 total 10.01 last 79.03) - open/close 1581 (time 1710102911.52 total 20.01 last 78.98) - open/close 2324 (time 1710102921.55 total 30.04 last 74.07) - open/close 3043 (time 1710102931.56 total 40.06 last 71.80) - open/close 3668 (time 1710102941.57 total 50.06 last 62.48) - open/close 4262 (time 1710102951.57 total 60.06 last 59.38) - open/close 4850 (time 1710102961.58 total 70.07 last 58.74) - open/close 5437 (time 1710102971.59 total 80.08 last 58.64) - open/close 5980 (time 1710102981.59 total 90.09 last 54.29) - open/close 6534 (time 1710102991.60 total 100.10 last 55.35) - open/close 6689 (time 1710103005.53 total 114.02 last 11.13) - open/close 7175 (time 1710103015.53 total 124.02 last 48.60) - open/close 7678 (time 1710103025.53 total 134.03 last 50.27) - open/close 7828 (time 1710103039.72 total 148.21 last 10.57) - open/close 8298 (time 1710103049.74 total 158.23 last 46.90) - open/close 8791 (time 1710103059.74 total 168.23 last 49.30) - open/close 9277 (time 1710103069.78 total 178.27 last 48.41) - open/close 9693 (time 1710103079.82 total 188.31 last 41.44) total: 10000 open/close in 195.19 seconds: 51.23 ops/second - unlinked 0 (time 1710103089 ; total 0 ; last 0) total: 10000 unlinks in 105 seconds: 95.238098 unlinks/second - open/close 915 (time 1710103206.41 total 10.01 last 91.43) - open/close 1742 (time 1710103216.42 total 20.01 last 82.66) - open/close 2561 (time 1710103226.42 total 30.02 last 81.88) - open/close 3256 (time 1710103236.42 total 40.02 last 69.49) - open/close 3981 (time 1710103246.43 total 50.02 last 72.44) - open/close 4649 (time 1710103256.44 total 60.03 last 66.74) - open/close 5344 (time 1710103266.44 total 70.03 last 69.49) - open/close 6051 (time 1710103276.45 total 80.04 last 70.66) - open/close 6690 (time 1710103286.46 total 90.05 last 63.81) - open/close 7296 (time 1710103296.47 total 100.06 last 60.56) - open/close 7921 (time 1710103306.48 total 110.07 last 62.45) - open/close 8541 (time 1710103316.48 total 120.08 last 61.95) - open/close 9141 (time 1710103326.49 total 130.08 last 59.98) - open/close 9737 (time 1710103336.49 total 140.09 last 59.57) total: 10000 open/close in 144.97 seconds: 68.98 ops/second - unlinked 0 (time 1710103343 ; total 0 ; last 0) total: 10000 unlinks in 86 seconds: 116.279068 unlinks/second - open/close 1050 (time 1710103441.49 total 10.01 last 104.91) - open/close 2041 (time 1710103451.50 total 20.02 last 99.02) - open/close 2987 (time 1710103461.50 total 30.02 last 94.57) - open/close 3915 (time 1710103471.50 total 40.03 last 92.75) - open/close 4759 (time 1710103481.51 total 50.03 last 84.34) - open/close 5629 (time 1710103491.52 total 60.04 last 86.94) - open/close 6554 (time 1710103501.52 total 70.04 last 92.45) - open/close 7428 (time 1710103511.53 total 80.05 last 87.38) - open/close 8333 (time 1710103521.53 total 90.05 last 90.45) - open/close 9215 (time 1710103531.53 total 100.05 last 88.20) total: 10000 open/close in 109.46 seconds: 91.35 ops/second - unlinked 0 (time 1710103542 ; total 0 ; last 0) total: 10000 unlinks in 53 seconds: 188.679245 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg116-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg116-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg116-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.201.116@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.201.116@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg116-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg116-server: oleg116-server.virtnet: executing set_default_debug -1 all pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 Started lustre-OST0000 oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg116-server: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg116-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 7 sec mount lustre on /mnt/lustre..... Starting client: oleg116-client.virtnet: -o user_xattr,flock oleg116-server@tcp:/lustre /mnt/lustre On OST0, 17669 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg116-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg116-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg116-server stop mds service on oleg116-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg116-server unloading modules on: 'oleg116-server' oleg116-server: oleg116-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg116-client: oleg116-client: ssh exited with exit code 2 pdsh@oleg116-client: oleg116-server: ssh exited with exit code 2 pdsh@oleg116-client: oleg116-client: ssh exited with exit code 2 pdsh@oleg116-client: oleg116-server: ssh exited with exit code 2 PASS 69 (2805s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg116-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg116-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg116-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg116-client: oleg116-server: ssh exited with exit code 2 oleg116-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg116-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == conf-sanity test 93: register mulitple MDT at the same time ========================================================== 16:47:17 (1710103637) SKIP: conf-sanity test_93 needs >= 3 MDTs SKIP 93 (0s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg116-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg116-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory Stopping clients: oleg116-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg116-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg116-client: oleg116-server: ssh exited with exit code 2 oleg116-server: oleg116-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg116-server' oleg116-server: oleg116-server.virtnet: executing load_modules_local oleg116-server: Loading modules from /home/green/git/lustre-release/lustre oleg116-server: detected 4 online CPUs by sysfs oleg116-server: Force libcfs to create 2 CPU partitions oleg116-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg116-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 3726 sec ========== 16:47:38 (1710103658) === conf-sanity: start cleanup 16:47:38 (1710103658) === === conf-sanity: finish cleanup 16:47:38 (1710103658) ===