-----============= acceptance-small: conf-sanity ============----- Sun Mar 10 15:45:23 EDT 2024 excepting tests: 32newtarball 110 skipping tests SLOW=no: 45 69 106 111 114 Stopping clients: oleg214-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg214-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg214-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg214-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg214-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg214-server oleg214-server: oleg214-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg214-server' oleg214-server: oleg214-server.virtnet: executing load_modules_local oleg214-server: Loading modules from /home/green/git/lustre-release/lustre oleg214-server: detected 4 online CPUs by sysfs oleg214-server: Force libcfs to create 2 CPU partitions oleg214-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg214-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg214-server: oleg214-server.virtnet: executing set_default_debug -1 all pdsh@oleg214-client: oleg214-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg214-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg214-server: oleg214-server.virtnet: executing set_default_debug -1 all pdsh@oleg214-client: oleg214-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg214-server: oleg214-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg214-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg214-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg214-server stop mds service on oleg214-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg214-server debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 15:46:39 (1710099999) start mds service on oleg214-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg214-server: oleg214-server.virtnet: executing set_default_debug -1 all pdsh@oleg214-client: oleg214-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg214-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg214-server: oleg214-server.virtnet: executing set_default_debug -1 all pdsh@oleg214-client: oleg214-server: ssh exited with exit code 1 Started lustre-OST0000 oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg214-server: oleg214-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg214-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 4 sec stop ost1 service on oleg214-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg214-server stop mds service on oleg214-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg214-server PASS 21a (29s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 15:47:09 (1710100029) start ost1 service on oleg214-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg214-server: oleg214-server.virtnet: executing set_default_debug -1 all pdsh@oleg214-client: oleg214-server: ssh exited with exit code 1 Started lustre-OST0000 oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid start mds service on oleg214-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg214-server: oleg214-server.virtnet: executing set_default_debug -1 all pdsh@oleg214-client: oleg214-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg214-server: oleg214-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg214-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop mds service on oleg214-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg214-server stop ost1 service on oleg214-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg214-server PASS 21b (197s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 15:50:27 (1710100227) start mds service on oleg214-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg214-server: oleg214-server.virtnet: executing set_default_debug -1 all pdsh@oleg214-client: oleg214-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid Client mount with ost in logs, but none running start ost1 service on oleg214-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg214-server: oleg214-server.virtnet: executing set_default_debug -1 all pdsh@oleg214-client: oleg214-server: ssh exited with exit code 1 Started lustre-OST0000 oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg214-server: oleg214-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg214-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 4 sec stop ost1 service on oleg214-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg214-server mount lustre on /mnt/lustre..... Starting client: oleg214-client.virtnet: -o user_xattr,flock oleg214-server@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg214-client.virtnet /mnt/lustre (opts:-f) PASS Client mount with a running ost start ost1 service on oleg214-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg214-server: oleg214-server.virtnet: executing set_default_debug -1 all pdsh@oleg214-client: oleg214-server: ssh exited with exit code 1 Started lustre-OST0000 oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg214-client.virtnet: -o user_xattr,flock oleg214-server@tcp:/lustre /mnt/lustre oleg214-server: oleg214-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg214-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec oleg214-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8800aac64800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800aac64800.ost_server_uuid in FULL state after 0 sec setup single mount lustre success PASS umount lustre on /mnt/lustre..... Stopping client oleg214-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg214-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg214-server stop mds service on oleg214-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg214-server unloading modules on: 'oleg214-server' oleg214-server: oleg214-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg214-client: oleg214-client: ssh exited with exit code 2 pdsh@oleg214-client: oleg214-server: ssh exited with exit code 2 pdsh@oleg214-client: oleg214-client: ssh exited with exit code 2 pdsh@oleg214-client: oleg214-server: ssh exited with exit code 2 PASS 22 (51s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg214-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg214-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg214-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg214-client: oleg214-server: ssh exited with exit code 2 oleg214-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg214-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == conf-sanity test 46a: handle ost additional - wide striped file ========================================================== 15:51:20 (1710100280) Testing with 2 OSTs Stopping clients: oleg214-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg214-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg214-client: oleg214-server: ssh exited with exit code 2 oleg214-server: oleg214-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg214-server' oleg214-server: oleg214-server.virtnet: executing load_modules_local oleg214-server: Loading modules from /home/green/git/lustre-release/lustre oleg214-server: detected 4 online CPUs by sysfs oleg214-server: Force libcfs to create 2 CPU partitions oleg214-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg214-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg214-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg214-server: oleg214-server.virtnet: executing set_default_debug -1 all pdsh@oleg214-client: oleg214-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg214-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg214-server: oleg214-server.virtnet: executing set_default_debug -1 all pdsh@oleg214-client: oleg214-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg214-server: oleg214-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg214-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg214-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg214-server stop mds service on oleg214-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg214-server start mds service on oleg214-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg214-server: oleg214-server.virtnet: executing set_default_debug -1 all pdsh@oleg214-client: oleg214-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg214-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg214-server: oleg214-server.virtnet: executing set_default_debug -1 all pdsh@oleg214-client: oleg214-server: ssh exited with exit code 1 Started lustre-OST0000 oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg214-server: oleg214-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg214-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 4 sec mount lustre on /mnt/lustre..... Starting client: oleg214-client.virtnet: -o user_xattr,flock oleg214-server@tcp:/lustre /mnt/lustre Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg214-server: oleg214-server.virtnet: executing set_default_debug -1 all pdsh@oleg214-client: oleg214-server: ssh exited with exit code 1 Commit the device label on lustre-ost2/ost2 Started lustre-OST0001 oleg214-server: oleg214-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 oleg214-server: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg214-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8800b43ea000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b43ea000.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre2..... Starting client: oleg214-client.virtnet: -o user_xattr,flock oleg214-server@tcp:/lustre /mnt/lustre2 /mnt/lustre2 stripe_count: -1 stripe_size: 4194304 pattern: raid0 stripe_offset: -1 /mnt/lustre2/widestripe lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 2 0x2 0x280000400 0 34 0x22 0x240000400 File: '/mnt/lustre/widestripe' Size: 3 Blocks: 2 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115238826934273 Links: 1 Access: (0674/-rw-rwxr--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-03-10 15:52:19.000000000 -0400 Modify: 2024-03-10 15:52:19.000000000 -0400 Change: 2024-03-10 15:52:20.000000000 -0400 Birth: - umount lustre on /mnt/lustre2..... Stopping client oleg214-client.virtnet /mnt/lustre2 (opts:) umount lustre on /mnt/lustre..... Stopping client oleg214-client.virtnet /mnt/lustre (opts:) Stopping /mnt/lustre-ost2 (opts:-f) on oleg214-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg214-server stop mds service on oleg214-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg214-server stop ost1 service on oleg214-server stop mds service on oleg214-server unloading modules on: 'oleg214-server' oleg214-server: oleg214-server.virtnet: executing unload_modules_local modules unloaded. oleg214-server: oleg214-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre oleg214-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg214-client: oleg214-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg214-server: oleg214-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre oleg214-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg214-client: oleg214-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg214-server: oleg214-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre oleg214-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg214-client: oleg214-server: ssh exited with exit code 19 checking for existing Lustre data: not found tunefs failed, reformatting instead Stopping clients: oleg214-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg214-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg214-client: oleg214-server: ssh exited with exit code 2 oleg214-server: oleg214-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg214-server' oleg214-server: oleg214-server.virtnet: executing load_modules_local oleg214-server: Loading modules from /home/green/git/lustre-release/lustre oleg214-server: detected 4 online CPUs by sysfs oleg214-server: Force libcfs to create 2 CPU partitions oleg214-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg214-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg214-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg214-server: oleg214-server.virtnet: executing set_default_debug -1 all pdsh@oleg214-client: oleg214-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg214-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg214-server: oleg214-server.virtnet: executing set_default_debug -1 all pdsh@oleg214-client: oleg214-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg214-server: oleg214-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg214-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg214-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg214-server stop mds service on oleg214-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg214-server PASS 46a (134s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 50b: lazystatfs all servers down ===== 15:53:36 (1710100416) start mds service on oleg214-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg214-server: oleg214-server.virtnet: executing set_default_debug -1 all pdsh@oleg214-client: oleg214-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg214-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg214-server: oleg214-server.virtnet: executing set_default_debug -1 all pdsh@oleg214-client: oleg214-server: ssh exited with exit code 1 Started lustre-OST0000 oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg214-client.virtnet: -o user_xattr,flock oleg214-server@tcp:/lustre /mnt/lustre llite.lustre-ffff88012a44a800.lazystatfs=1 stop ost1 service on oleg214-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg214-server oleg214-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff88012a44a800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff88012a44a800.ost_server_uuid in DISCONN state after 0 sec 0 UP mgc MGC192.168.202.114@tcp 10f5514a-4358-4860-9da3-e517c9a15fb9 5 1 UP lov lustre-clilov-ffff88012a44a800 c07cd8e4-a630-42ee-ba85-049cf46666c1 4 2 UP lmv lustre-clilmv-ffff88012a44a800 c07cd8e4-a630-42ee-ba85-049cf46666c1 5 3 UP mdc lustre-MDT0000-mdc-ffff88012a44a800 c07cd8e4-a630-42ee-ba85-049cf46666c1 5 4 UP osc lustre-OST0000-osc-ffff88012a44a800 c07cd8e4-a630-42ee-ba85-049cf46666c1 5 OSCs should all be DISCONN multiop /mnt/lustre vf_ TMPPIPE=/tmp/multiop_open_wait_pipe.6728 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 383616 3200 378368 1% /mnt/lustre[MDT:0] filesystem_summary: 383616 3200 378368 1% /mnt/lustre /home/green/git/lustre-release/lustre/tests/conf-sanity.sh: line 4282: kill: (27457) - No such process umount lustre on /mnt/lustre..... Stopping client oleg214-client.virtnet /mnt/lustre (opts:) stop mds service on oleg214-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg214-server PASS 50b (44s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 50c: lazystatfs one server down ====== 15:54:22 (1710100462) start mds service on oleg214-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg214-server: oleg214-server.virtnet: executing set_default_debug -1 all pdsh@oleg214-client: oleg214-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg214-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg214-server: oleg214-server.virtnet: executing set_default_debug -1 all pdsh@oleg214-client: oleg214-server: ssh exited with exit code 1 Started lustre-OST0000 oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid start ost2 service on oleg214-server Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg214-server: oleg214-server.virtnet: executing set_default_debug -1 all pdsh@oleg214-client: oleg214-server: ssh exited with exit code 1 Commit the device label on lustre-ost2/ost2 Started lustre-OST0001 oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg214-client.virtnet: -o user_xattr,flock oleg214-server@tcp:/lustre /mnt/lustre llite.lustre-ffff8800aa0b9000.lazystatfs=1 stop ost1 service on oleg214-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg214-server oleg214-server: oleg214-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg214-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec multiop /mnt/lustre vf_ TMPPIPE=/tmp/multiop_open_wait_pipe.6728 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 383616 3328 378240 1% /mnt/lustre[MDT:0] lustre-OST0001_UUID 382976 3072 377856 1% /mnt/lustre[OST:1] filesystem_summary: 382976 3072 377856 1% /mnt/lustre /home/green/git/lustre-release/lustre/tests/conf-sanity.sh: line 4282: kill: (29693) - No such process umount lustre on /mnt/lustre..... Stopping client oleg214-client.virtnet /mnt/lustre (opts:) stop ost2 service on oleg214-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg214-server stop mds service on oleg214-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg214-server oleg214-server: oleg214-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre oleg214-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg214-client: oleg214-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg214-server: oleg214-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre oleg214-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg214-client: oleg214-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg214-server: oleg214-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre oleg214-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg214-client: oleg214-server: ssh exited with exit code 19 checking for existing Lustre data: not found tunefs failed, reformatting instead Stopping clients: oleg214-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg214-client.virtnet /mnt/lustre2 (opts:-f) oleg214-server: oleg214-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg214-server' oleg214-server: oleg214-server.virtnet: executing load_modules_local oleg214-server: Loading modules from /home/green/git/lustre-release/lustre oleg214-server: detected 4 online CPUs by sysfs oleg214-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg214-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg214-server: oleg214-server.virtnet: executing set_default_debug -1 all pdsh@oleg214-client: oleg214-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg214-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg214-server: oleg214-server.virtnet: executing set_default_debug -1 all pdsh@oleg214-client: oleg214-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg214-server: oleg214-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg214-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg214-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg214-server stop mds service on oleg214-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg214-server PASS 50c (87s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 50e: normal statfs all servers down == 15:55:50 (1710100550) Stopping clients: oleg214-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg214-client.virtnet /mnt/lustre2 (opts:-f) oleg214-server: oleg214-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg214-server' oleg214-server: oleg214-server.virtnet: executing load_modules_local oleg214-server: Loading modules from /home/green/git/lustre-release/lustre oleg214-server: detected 4 online CPUs by sysfs oleg214-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg214-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg214-server: oleg214-server.virtnet: executing set_default_debug -1 all pdsh@oleg214-client: oleg214-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg214-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg214-server: oleg214-server.virtnet: executing set_default_debug -1 all pdsh@oleg214-client: oleg214-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg214-server: oleg214-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg214-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg214-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg214-server stop mds service on oleg214-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg214-server start mds service on oleg214-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg214-server: oleg214-server.virtnet: executing set_default_debug -1 all pdsh@oleg214-client: oleg214-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg214-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg214-server: oleg214-server.virtnet: executing set_default_debug -1 all pdsh@oleg214-client: oleg214-server: ssh exited with exit code 1 Started lustre-OST0000 oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg214-server: oleg214-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg214-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec stop ost1 service on oleg214-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg214-server oleg214-server: oleg214-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg214-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg214-client.virtnet: -o user_xattr,flock oleg214-server@tcp:/lustre /mnt/lustre llite.lustre-ffff8800b639b000.lazystatfs=0 multiop /mnt/lustre v_f TMPPIPE=/tmp/multiop_open_wait_pipe.6728 start ost1 service on oleg214-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg214-server: oleg214-server.virtnet: executing set_default_debug -1 all pdsh@oleg214-client: oleg214-server: ssh exited with exit code 1 Started lustre-OST0000 oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec umount lustre on /mnt/lustre..... Stopping client oleg214-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg214-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg214-server stop mds service on oleg214-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg214-server PASS 50e (93s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 50f: normal statfs one server in down ========================================================== 15:57:25 (1710100645) start mds service on oleg214-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg214-server: oleg214-server.virtnet: executing set_default_debug -1 all pdsh@oleg214-client: oleg214-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg214-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg214-server: oleg214-server.virtnet: executing set_default_debug -1 all pdsh@oleg214-client: oleg214-server: ssh exited with exit code 1 Started lustre-OST0000 oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg214-server: oleg214-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg214-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec start ost2 service on oleg214-server Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg214-server: oleg214-server.virtnet: executing set_default_debug -1 all pdsh@oleg214-client: oleg214-server: ssh exited with exit code 1 Commit the device label on lustre-ost2/ost2 Started lustre-OST0001 oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid oleg214-server: oleg214-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 oleg214-server: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost2 service on oleg214-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg214-server oleg214-server: oleg214-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 oleg214-server: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg214-client.virtnet: -o user_xattr,flock oleg214-server@tcp:/lustre /mnt/lustre llite.lustre-ffff8800aa0aa800.lazystatfs=0 multiop /mnt/lustre v_f TMPPIPE=/tmp/multiop_open_wait_pipe.6728 start ost2 service on oleg214-server Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg214-server: oleg214-server.virtnet: executing set_default_debug -1 all pdsh@oleg214-client: oleg214-server: ssh exited with exit code 1 Started lustre-OST0001 oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 1 sec stop ost2 service on oleg214-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg214-server umount lustre on /mnt/lustre..... Stopping client oleg214-client.virtnet /mnt/lustre (opts:-f) stop ost1 service on oleg214-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg214-server stop mds service on oleg214-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg214-server oleg214-server: oleg214-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre oleg214-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg214-client: oleg214-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg214-server: oleg214-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre oleg214-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg214-client: oleg214-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg214-server: oleg214-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre oleg214-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg214-client: oleg214-server: ssh exited with exit code 19 checking for existing Lustre data: not found tunefs failed, reformatting instead Stopping clients: oleg214-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg214-client.virtnet /mnt/lustre2 (opts:-f) oleg214-server: oleg214-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg214-server' oleg214-server: oleg214-server.virtnet: executing load_modules_local oleg214-server: Loading modules from /home/green/git/lustre-release/lustre oleg214-server: detected 4 online CPUs by sysfs oleg214-server: Force libcfs to create 2 CPU partitions oleg214-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg214-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg214-server: oleg214-server.virtnet: executing set_default_debug -1 all pdsh@oleg214-client: oleg214-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg214-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg214-server: oleg214-server.virtnet: executing set_default_debug -1 all pdsh@oleg214-client: oleg214-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg214-server: oleg214-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg214-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg214-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg214-server stop mds service on oleg214-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg214-server PASS 50f (117s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 69: replace an OST with the same index ========================================================== 15:59:22 (1710100762) start mds service on oleg214-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg214-server: oleg214-server.virtnet: executing set_default_debug -1 all pdsh@oleg214-client: oleg214-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg214-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg214-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg214-server: oleg214-server.virtnet: executing set_default_debug -1 all pdsh@oleg214-client: oleg214-server: ssh exited with exit code 1 Started lustre-OST0000 oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg214-client.virtnet: -o user_xattr,flock oleg214-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11872 inodes available. Want 99936. rc=0 - open/close 1662 (time 1710100790.16 total 10.00 last 166.14) - open/close 2999 (time 1710100800.17 total 20.01 last 133.59) - open/close 4165 (time 1710100810.18 total 30.02 last 116.52) - open/close 5168 (time 1710100820.19 total 40.03 last 100.22) - open/close 6168 (time 1710100830.19 total 50.03 last 99.95) - open/close 7170 (time 1710100840.19 total 60.03 last 100.20) - open/close 8233 (time 1710100850.19 total 70.03 last 106.30) - open/close 9120 (time 1710100860.20 total 80.04 last 88.60) - open/close 9984 (time 1710100870.22 total 90.06 last 86.29) total: 10000 open/close in 90.23 seconds: 110.82 ops/second - unlinked 0 (time 1710100871 ; total 0 ; last 0) total: 10000 unlinks in 65 seconds: 153.846161 unlinks/second - open/close 1099 (time 1710100948.19 total 10.00 last 109.89) - open/close 2111 (time 1710100958.20 total 20.01 last 101.14) - open/close 3168 (time 1710100968.21 total 30.02 last 105.54) - open/close 4232 (time 1710100978.22 total 40.03 last 106.33) - open/close 5160 (time 1710100988.23 total 50.04 last 92.71) - open/close 6107 (time 1710100998.24 total 60.05 last 94.62) - open/close 7043 (time 1710101008.24 total 70.06 last 93.52) - open/close 7801 (time 1710101018.25 total 80.07 last 75.73) - open/close 8585 (time 1710101028.26 total 90.07 last 78.36) - open/close 9419 (time 1710101038.26 total 100.07 last 83.38) total: 10000 open/close in 108.50 seconds: 92.17 ops/second - unlinked 0 (time 1710101048 ; total 0 ; last 0) total: 10000 unlinks in 87 seconds: 114.942528 unlinks/second - open/close 1104 (time 1710101147.29 total 10.00 last 110.35) - open/close 2023 (time 1710101157.29 total 20.01 last 91.86) - open/close 2880 (time 1710101167.30 total 30.02 last 85.62) - open/close 3772 (time 1710101177.31 total 40.02 last 89.14) - open/close 4590 (time 1710101187.31 total 50.03 last 81.77) - open/close 5321 (time 1710101197.32 total 60.04 last 73.04) - open/close 6048 (time 1710101207.33 total 70.05 last 72.58) - open/close 6803 (time 1710101217.34 total 80.05 last 75.49) - open/close 7428 (time 1710101227.34 total 90.06 last 62.47) - open/close 8036 (time 1710101237.35 total 100.07 last 60.73) - open/close 8688 (time 1710101247.36 total 110.08 last 65.15) - open/close 9357 (time 1710101257.36 total 120.08 last 66.88) - open/close 9988 (time 1710101267.37 total 130.09 last 63.03) total: 10000 open/close in 130.27 seconds: 76.76 ops/second - unlinked 0 (time 1710101269 ; total 0 ; last 0) total: 10000 unlinks in 89 seconds: 112.359550 unlinks/second - open/close 1017 (time 1710101370.30 total 10.01 last 101.62) - open/close 1995 (time 1710101380.30 total 20.01 last 97.80) - open/close 2958 (time 1710101390.30 total 30.01 last 96.26) - open/close 3767 (time 1710101400.30 total 40.01 last 80.89) - open/close 4556 (time 1710101410.31 total 50.03 last 78.80) - open/close 5280 (time 1710101420.32 total 60.03 last 72.34) - open/close 6043 (time 1710101430.33 total 70.04 last 76.24) - open/close 6802 (time 1710101440.33 total 80.05 last 75.88) - open/close 7523 (time 1710101450.34 total 90.05 last 72.09) - open/close 8170 (time 1710101460.36 total 100.07 last 64.57) - open/close 8803 (time 1710101470.36 total 110.07 last 63.30) - open/close 9418 (time 1710101480.37 total 120.08 last 61.40) total: 10000 open/close in 129.97 seconds: 76.94 ops/second - unlinked 0 (time 1710101492 ; total 0 ; last 0) total: 10000 unlinks in 81 seconds: 123.456787 unlinks/second - open/close 1126 (time 1710101585.50 total 10.00 last 112.60) - open/close 2156 (time 1710101595.50 total 20.01 last 102.92) - open/close 3068 (time 1710101605.51 total 30.02 last 91.13) - open/close 4083 (time 1710101615.51 total 40.02 last 101.49) - open/close 4914 (time 1710101625.52 total 50.03 last 83.03) - open/close 5771 (time 1710101635.52 total 60.03 last 85.66) - open/close 6547 (time 1710101645.54 total 70.04 last 77.49) - open/close 7279 (time 1710101655.57 total 80.07 last 72.99) - open/close 8095 (time 1710101665.64 total 90.14 last 81.04) - open/close 8798 (time 1710101675.64 total 100.15 last 70.28) - open/close 9481 (time 1710101685.64 total 110.15 last 68.30) total: 10000 open/close in 117.64 seconds: 85.00 ops/second - unlinked 0 (time 1710101694 ; total 0 ; last 0) total: 10000 unlinks in 78 seconds: 128.205124 unlinks/second - open/close 1211 (time 1710101784.54 total 10.00 last 121.10) - open/close 2283 (time 1710101794.55 total 20.01 last 107.08) - open/close 3218 (time 1710101804.56 total 30.02 last 93.45) - open/close 4108 (time 1710101814.56 total 40.02 last 88.94) - open/close 4978 (time 1710101824.57 total 50.03 last 86.97) - open/close 5805 (time 1710101834.57 total 60.03 last 82.64) - open/close 6610 (time 1710101844.58 total 70.04 last 80.45) - open/close 7341 (time 1710101854.59 total 80.05 last 73.02) - open/close 8007 (time 1710101864.59 total 90.05 last 66.57) - open/close 8536 (time 1710101884.36 total 109.82 last 26.76) - open/close 9145 (time 1710101904.74 total 130.19 last 29.89) - open/close 9804 (time 1710101914.75 total 140.21 last 65.81) total: 10000 open/close in 143.27 seconds: 69.80 ops/second - unlinked 0 (time 1710101919 ; total 0 ; last 0) total: 10000 unlinks in 82 seconds: 121.951218 unlinks/second - open/close 1128 (time 1710102013.37 total 10.01 last 112.73) - open/close 2108 (time 1710102023.37 total 20.01 last 97.99) - open/close 3089 (time 1710102033.38 total 30.02 last 98.03) - open/close 3620 (time 1710102049.93 total 46.56 last 32.09) - open/close 4542 (time 1710102059.94 total 56.57 last 92.10) - open/close 5358 (time 1710102069.94 total 66.58 last 81.54) - open/close 6159 (time 1710102079.95 total 76.59 last 80.02) - open/close 6919 (time 1710102089.96 total 86.60 last 75.97) - open/close 7350 (time 1710102106.80 total 103.44 last 25.58) - open/close 8075 (time 1710102116.81 total 113.45 last 72.47) - open/close 8734 (time 1710102126.82 total 123.46 last 65.82) - open/close 9403 (time 1710102136.83 total 133.46 last 66.87) total: 10000 open/close in 143.16 seconds: 69.85 ops/second - unlinked 0 (time 1710102148 ; total 0 ; last 0) total: 10000 unlinks in 79 seconds: 126.582275 unlinks/second - open/close 1160 (time 1710102239.48 total 10.00 last 115.99) - open/close 2258 (time 1710102249.48 total 20.00 last 109.77) - open/close 3329 (time 1710102259.48 total 30.01 last 107.05) - open/close 4214 (time 1710102269.48 total 40.01 last 88.48) - open/close 5024 (time 1710102279.49 total 50.02 last 80.94) - open/close 5766 (time 1710102289.49 total 60.02 last 74.18) - open/close 6559 (time 1710102299.58 total 70.10 last 78.66) - open/close 7348 (time 1710102309.58 total 80.10 last 78.89) - open/close 8093 (time 1710102319.59 total 90.11 last 74.41) - open/close 8815 (time 1710102329.59 total 100.12 last 72.17) - open/close 9424 (time 1710102339.61 total 110.13 last 60.81) total: 10000 open/close in 118.10 seconds: 84.67 ops/second - unlinked 0 (time 1710102349 ; total 0 ; last 0) total: 10000 unlinks in 76 seconds: 131.578949 unlinks/second - open/close 1199 (time 1710102438.09 total 10.00 last 119.86) - open/close 2209 (time 1710102448.10 total 20.01 last 100.94) - open/close 3156 (time 1710102458.10 total 30.02 last 94.64) - open/close 4072 (time 1710102468.11 total 40.02 last 91.58) - open/close 4930 (time 1710102478.11 total 50.02 last 85.78) - open/close 5772 (time 1710102488.11 total 60.02 last 84.17) - open/close 6559 (time 1710102498.11 total 70.02 last 78.69) - open/close 7298 (time 1710102508.12 total 80.03 last 73.86) - open/close 7981 (time 1710102518.13 total 90.04 last 68.24) - open/close 8620 (time 1710102528.14 total 100.05 last 63.82) - open/close 9341 (time 1710102538.15 total 110.07 last 72.01) total: 10000 open/close in 119.72 seconds: 83.53 ops/second - unlinked 0 (time 1710102549 ; total 0 ; last 0) total: 10000 unlinks in 74 seconds: 135.135132 unlinks/second - open/close 1107 (time 1710102635.82 total 10.00 last 110.66) - open/close 2231 (time 1710102645.82 total 20.01 last 112.37) - open/close 3191 (time 1710102655.82 total 30.01 last 95.98) - open/close 4073 (time 1710102665.85 total 40.03 last 87.97) - open/close 4976 (time 1710102675.85 total 50.04 last 90.28) - open/close 5831 (time 1710102685.86 total 60.05 last 85.40) - open/close 6689 (time 1710102695.87 total 70.05 last 85.75) - open/close 7524 (time 1710102705.87 total 80.06 last 83.43) - open/close 8240 (time 1710102715.88 total 90.06 last 71.59) - open/close 8964 (time 1710102725.89 total 100.08 last 72.31) - open/close 9661 (time 1710102735.89 total 110.08 last 69.68) total: 10000 open/close in 114.69 seconds: 87.19 ops/second - unlinked 0 (time 1710102742 ; total 0 ; last 0) total: 10000 unlinks in 72 seconds: 138.888885 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg214-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg214-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg214-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.202.114@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.202.114@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg214-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg214-server: oleg214-server.virtnet: executing set_default_debug -1 all pdsh@oleg214-client: oleg214-server: ssh exited with exit code 1 Started lustre-OST0000 oleg214-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg214-server: oleg214-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg214-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 9 sec mount lustre on /mnt/lustre..... Starting client: oleg214-client.virtnet: -o user_xattr,flock oleg214-server@tcp:/lustre /mnt/lustre On OST0, 7189 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg214-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg214-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg214-server stop mds service on oleg214-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg214-server unloading modules on: 'oleg214-server' oleg214-server: oleg214-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg214-client: oleg214-client: ssh exited with exit code 2 pdsh@oleg214-client: oleg214-server: ssh exited with exit code 2 pdsh@oleg214-client: oleg214-client: ssh exited with exit code 2 pdsh@oleg214-client: oleg214-server: ssh exited with exit code 2 PASS 69 (2099s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg214-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg214-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg214-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg214-client: oleg214-server: ssh exited with exit code 2 oleg214-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg214-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == conf-sanity test 93: register mulitple MDT at the same time ========================================================== 16:34:23 (1710102863) SKIP: conf-sanity test_93 needs >= 3 MDTs SKIP 93 (2s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg214-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg214-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory Stopping clients: oleg214-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg214-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg214-client: oleg214-server: ssh exited with exit code 2 oleg214-server: oleg214-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg214-server' oleg214-server: oleg214-server.virtnet: executing load_modules_local oleg214-server: Loading modules from /home/green/git/lustre-release/lustre oleg214-server: detected 4 online CPUs by sysfs oleg214-server: Force libcfs to create 2 CPU partitions oleg214-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg214-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 2968 sec ========== 16:34:52 (1710102892) === conf-sanity: start cleanup 16:34:52 (1710102892) === === conf-sanity: finish cleanup 16:34:52 (1710102892) ===