-----============= acceptance-small: conf-sanity ============----- Sun Mar 10 15:46:49 EDT 2024 excepting tests: 32newtarball 110 skipping tests SLOW=no: 45 69 106 111 114 Stopping clients: oleg232-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg232-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg232-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg232-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg232-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg232-server oleg232-server: oleg232-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory loading modules on: 'oleg232-server' oleg232-server: oleg232-server.virtnet: executing load_modules_local oleg232-server: Loading modules from /home/green/git/lustre-release/lustre oleg232-server: detected 4 online CPUs by sysfs oleg232-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg232-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg232-server: oleg232-server.virtnet: executing set_default_debug -1 all pdsh@oleg232-client: oleg232-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg232-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg232-server: oleg232-server.virtnet: executing set_default_debug -1 all pdsh@oleg232-client: oleg232-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg232-server: oleg232-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg232-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg232-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg232-server stop mds service on oleg232-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg232-server debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 15:47:40 (1710100060) start mds service on oleg232-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg232-server: oleg232-server.virtnet: executing set_default_debug -1 all pdsh@oleg232-client: oleg232-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg232-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg232-server: oleg232-server.virtnet: executing set_default_debug -1 all pdsh@oleg232-client: oleg232-server: ssh exited with exit code 1 Started lustre-OST0000 oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg232-server: oleg232-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg232-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 4 sec stop ost1 service on oleg232-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg232-server stop mds service on oleg232-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg232-server PASS 21a (21s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 15:48:03 (1710100083) start ost1 service on oleg232-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg232-server: oleg232-server.virtnet: executing set_default_debug -1 all pdsh@oleg232-client: oleg232-server: ssh exited with exit code 1 Started lustre-OST0000 oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid start mds service on oleg232-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg232-server: oleg232-server.virtnet: executing set_default_debug -1 all pdsh@oleg232-client: oleg232-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg232-server: oleg232-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg232-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop mds service on oleg232-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg232-server stop ost1 service on oleg232-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg232-server PASS 21b (197s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 15:51:21 (1710100281) start mds service on oleg232-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg232-server: oleg232-server.virtnet: executing set_default_debug -1 all pdsh@oleg232-client: oleg232-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid Client mount with ost in logs, but none running start ost1 service on oleg232-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg232-server: oleg232-server.virtnet: executing set_default_debug -1 all pdsh@oleg232-client: oleg232-server: ssh exited with exit code 1 Started lustre-OST0000 oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg232-server: oleg232-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg232-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 4 sec stop ost1 service on oleg232-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg232-server mount lustre on /mnt/lustre..... Starting client: oleg232-client.virtnet: -o user_xattr,flock oleg232-server@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg232-client.virtnet /mnt/lustre (opts:-f) PASS Client mount with a running ost start ost1 service on oleg232-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg232-server: oleg232-server.virtnet: executing set_default_debug -1 all pdsh@oleg232-client: oleg232-server: ssh exited with exit code 1 Started lustre-OST0000 oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg232-client.virtnet: -o user_xattr,flock oleg232-server@tcp:/lustre /mnt/lustre oleg232-server: oleg232-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg232-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec oleg232-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff88013102b000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff88013102b000.ost_server_uuid in FULL state after 0 sec setup single mount lustre success PASS umount lustre on /mnt/lustre..... Stopping client oleg232-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg232-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg232-server stop mds service on oleg232-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg232-server unloading modules on: 'oleg232-server' oleg232-server: oleg232-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg232-client: oleg232-client: ssh exited with exit code 2 pdsh@oleg232-client: oleg232-server: ssh exited with exit code 2 pdsh@oleg232-client: oleg232-client: ssh exited with exit code 2 pdsh@oleg232-client: oleg232-server: ssh exited with exit code 2 PASS 22 (48s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg232-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg232-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg232-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg232-client: oleg232-server: ssh exited with exit code 2 oleg232-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg232-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == conf-sanity test 46a: handle ost additional - wide striped file ========================================================== 15:52:11 (1710100331) Testing with 2 OSTs Stopping clients: oleg232-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg232-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg232-client: oleg232-server: ssh exited with exit code 2 oleg232-server: oleg232-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg232-server' oleg232-server: oleg232-server.virtnet: executing load_modules_local oleg232-server: Loading modules from /home/green/git/lustre-release/lustre oleg232-server: detected 4 online CPUs by sysfs oleg232-server: Force libcfs to create 2 CPU partitions oleg232-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg232-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg232-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg232-server: oleg232-server.virtnet: executing set_default_debug -1 all pdsh@oleg232-client: oleg232-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg232-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg232-server: oleg232-server.virtnet: executing set_default_debug -1 all pdsh@oleg232-client: oleg232-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg232-server: oleg232-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg232-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg232-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg232-server stop mds service on oleg232-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg232-server start mds service on oleg232-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg232-server: oleg232-server.virtnet: executing set_default_debug -1 all pdsh@oleg232-client: oleg232-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg232-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg232-server: oleg232-server.virtnet: executing set_default_debug -1 all pdsh@oleg232-client: oleg232-server: ssh exited with exit code 1 Started lustre-OST0000 oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg232-server: oleg232-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg232-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg232-client.virtnet: -o user_xattr,flock oleg232-server@tcp:/lustre /mnt/lustre Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg232-server: oleg232-server.virtnet: executing set_default_debug -1 all pdsh@oleg232-client: oleg232-server: ssh exited with exit code 1 Commit the device label on lustre-ost2/ost2 Started lustre-OST0001 oleg232-server: oleg232-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 oleg232-server: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 2 sec oleg232-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8800b6e7e800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800b6e7e800.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre2..... Starting client: oleg232-client.virtnet: -o user_xattr,flock oleg232-server@tcp:/lustre /mnt/lustre2 /mnt/lustre2 stripe_count: -1 stripe_size: 4194304 pattern: raid0 stripe_offset: -1 /mnt/lustre2/widestripe lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 2 0x2 0x280000400 0 34 0x22 0x240000400 File: '/mnt/lustre/widestripe' Size: 3 Blocks: 2 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115238826934273 Links: 1 Access: (0674/-rw-rwxr--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-03-10 15:53:18.000000000 -0400 Modify: 2024-03-10 15:53:18.000000000 -0400 Change: 2024-03-10 15:53:19.000000000 -0400 Birth: - umount lustre on /mnt/lustre2..... Stopping client oleg232-client.virtnet /mnt/lustre2 (opts:) umount lustre on /mnt/lustre..... Stopping client oleg232-client.virtnet /mnt/lustre (opts:) Stopping /mnt/lustre-ost2 (opts:-f) on oleg232-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg232-server stop mds service on oleg232-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg232-server stop ost1 service on oleg232-server stop mds service on oleg232-server unloading modules on: 'oleg232-server' oleg232-server: oleg232-server.virtnet: executing unload_modules_local modules unloaded. oleg232-server: oleg232-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre oleg232-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg232-client: oleg232-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg232-server: oleg232-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre oleg232-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg232-client: oleg232-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg232-server: oleg232-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre oleg232-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg232-client: oleg232-server: ssh exited with exit code 19 checking for existing Lustre data: not found tunefs failed, reformatting instead Stopping clients: oleg232-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg232-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg232-client: oleg232-server: ssh exited with exit code 2 oleg232-server: oleg232-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg232-server' oleg232-server: oleg232-server.virtnet: executing load_modules_local oleg232-server: Loading modules from /home/green/git/lustre-release/lustre oleg232-server: detected 4 online CPUs by sysfs oleg232-server: Force libcfs to create 2 CPU partitions oleg232-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg232-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg232-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg232-server: oleg232-server.virtnet: executing set_default_debug -1 all pdsh@oleg232-client: oleg232-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg232-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg232-server: oleg232-server.virtnet: executing set_default_debug -1 all pdsh@oleg232-client: oleg232-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg232-server: oleg232-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg232-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg232-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg232-server stop mds service on oleg232-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg232-server PASS 46a (149s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 50b: lazystatfs all servers down ===== 15:54:42 (1710100482) start mds service on oleg232-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg232-server: oleg232-server.virtnet: executing set_default_debug -1 all pdsh@oleg232-client: oleg232-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg232-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg232-server: oleg232-server.virtnet: executing set_default_debug -1 all pdsh@oleg232-client: oleg232-server: ssh exited with exit code 1 Started lustre-OST0000 oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg232-client.virtnet: -o user_xattr,flock oleg232-server@tcp:/lustre /mnt/lustre llite.lustre-ffff8800ab34d800.lazystatfs=1 stop ost1 service on oleg232-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg232-server oleg232-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff8800ab34d800.ost_server_uuid 50 0 UP mgc MGC192.168.202.132@tcp 437b9105-f801-405d-8069-8ae59d99e420 5 1 UP lov lustre-clilov-ffff8800ab34d800 df1664ec-d83b-4ea3-a55a-5836f6bf7521 4 2 UP lmv lustre-clilmv-ffff8800ab34d800 df1664ec-d83b-4ea3-a55a-5836f6bf7521 5 3 UP mdc lustre-MDT0000-mdc-ffff8800ab34d800 df1664ec-d83b-4ea3-a55a-5836f6bf7521 5 4 UP osc lustre-OST0000-osc-ffff8800ab34d800 df1664ec-d83b-4ea3-a55a-5836f6bf7521 5 OSCs should all be DISCONN multiop /mnt/lustre vf_ TMPPIPE=/tmp/multiop_open_wait_pipe.6739 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 383616 3200 378368 1% /mnt/lustre[MDT:0] filesystem_summary: 383616 3200 378368 1% /mnt/lustre /home/green/git/lustre-release/lustre/tests/conf-sanity.sh: line 4282: kill: (27300) - No such process umount lustre on /mnt/lustre..... Stopping client oleg232-client.virtnet /mnt/lustre (opts:) stop mds service on oleg232-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg232-server PASS 50b (44s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 50c: lazystatfs one server down ====== 15:55:28 (1710100528) start mds service on oleg232-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg232-server: oleg232-server.virtnet: executing set_default_debug -1 all pdsh@oleg232-client: oleg232-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg232-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg232-server: oleg232-server.virtnet: executing set_default_debug -1 all pdsh@oleg232-client: oleg232-server: ssh exited with exit code 1 Started lustre-OST0000 oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid start ost2 service on oleg232-server Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg232-server: oleg232-server.virtnet: executing set_default_debug -1 all pdsh@oleg232-client: oleg232-server: ssh exited with exit code 1 Commit the device label on lustre-ost2/ost2 Started lustre-OST0001 oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg232-client.virtnet: -o user_xattr,flock oleg232-server@tcp:/lustre /mnt/lustre llite.lustre-ffff880136b03800.lazystatfs=1 stop ost1 service on oleg232-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg232-server oleg232-server: oleg232-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg232-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec multiop /mnt/lustre vf_ TMPPIPE=/tmp/multiop_open_wait_pipe.6739 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 383616 3328 378240 1% /mnt/lustre[MDT:0] lustre-OST0001_UUID 382976 3072 377856 1% /mnt/lustre[OST:1] filesystem_summary: 382976 3072 377856 1% /mnt/lustre /home/green/git/lustre-release/lustre/tests/conf-sanity.sh: line 4282: kill: (29532) - No such process umount lustre on /mnt/lustre..... Stopping client oleg232-client.virtnet /mnt/lustre (opts:) stop ost2 service on oleg232-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg232-server stop mds service on oleg232-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg232-server oleg232-server: oleg232-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre oleg232-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg232-client: oleg232-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg232-server: oleg232-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre oleg232-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg232-client: oleg232-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg232-server: oleg232-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre oleg232-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg232-client: oleg232-server: ssh exited with exit code 19 checking for existing Lustre data: not found tunefs failed, reformatting instead Stopping clients: oleg232-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg232-client.virtnet /mnt/lustre2 (opts:-f) oleg232-server: oleg232-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory loading modules on: 'oleg232-server' oleg232-server: oleg232-server.virtnet: executing load_modules_local oleg232-server: Loading modules from /home/green/git/lustre-release/lustre oleg232-server: detected 4 online CPUs by sysfs oleg232-server: Force libcfs to create 2 CPU partitions oleg232-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg232-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg232-server: oleg232-server.virtnet: executing set_default_debug -1 all pdsh@oleg232-client: oleg232-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg232-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg232-server: oleg232-server.virtnet: executing set_default_debug -1 all pdsh@oleg232-client: oleg232-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg232-server: oleg232-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg232-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg232-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg232-server stop mds service on oleg232-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg232-server PASS 50c (83s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 50e: normal statfs all servers down == 15:56:53 (1710100613) Stopping clients: oleg232-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg232-client.virtnet /mnt/lustre2 (opts:-f) oleg232-server: oleg232-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg232-server' oleg232-server: oleg232-server.virtnet: executing load_modules_local oleg232-server: Loading modules from /home/green/git/lustre-release/lustre oleg232-server: detected 4 online CPUs by sysfs oleg232-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg232-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg232-server: oleg232-server.virtnet: executing set_default_debug -1 all pdsh@oleg232-client: oleg232-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg232-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg232-server: oleg232-server.virtnet: executing set_default_debug -1 all pdsh@oleg232-client: oleg232-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg232-server: oleg232-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg232-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg232-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg232-server stop mds service on oleg232-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg232-server start mds service on oleg232-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg232-server: oleg232-server.virtnet: executing set_default_debug -1 all pdsh@oleg232-client: oleg232-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg232-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg232-server: oleg232-server.virtnet: executing set_default_debug -1 all pdsh@oleg232-client: oleg232-server: ssh exited with exit code 1 Started lustre-OST0000 oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg232-server: oleg232-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg232-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec stop ost1 service on oleg232-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg232-server oleg232-server: oleg232-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg232-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg232-client.virtnet: -o user_xattr,flock oleg232-server@tcp:/lustre /mnt/lustre llite.lustre-ffff88013102d800.lazystatfs=0 multiop /mnt/lustre v_f TMPPIPE=/tmp/multiop_open_wait_pipe.6739 start ost1 service on oleg232-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg232-server: oleg232-server.virtnet: executing set_default_debug -1 all pdsh@oleg232-client: oleg232-server: ssh exited with exit code 1 Started lustre-OST0000 oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec umount lustre on /mnt/lustre..... Stopping client oleg232-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg232-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg232-server stop mds service on oleg232-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg232-server PASS 50e (98s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 50f: normal statfs one server in down ========================================================== 15:58:33 (1710100713) start mds service on oleg232-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg232-server: oleg232-server.virtnet: executing set_default_debug -1 all pdsh@oleg232-client: oleg232-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg232-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg232-server: oleg232-server.virtnet: executing set_default_debug -1 all pdsh@oleg232-client: oleg232-server: ssh exited with exit code 1 Started lustre-OST0000 oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg232-server: oleg232-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg232-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec start ost2 service on oleg232-server Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg232-server: oleg232-server.virtnet: executing set_default_debug -1 all pdsh@oleg232-client: oleg232-server: ssh exited with exit code 1 Commit the device label on lustre-ost2/ost2 Started lustre-OST0001 oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid oleg232-server: oleg232-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 oleg232-server: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost2 service on oleg232-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg232-server oleg232-server: oleg232-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 oleg232-server: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg232-client.virtnet: -o user_xattr,flock oleg232-server@tcp:/lustre /mnt/lustre llite.lustre-ffff880131029800.lazystatfs=0 multiop /mnt/lustre v_f TMPPIPE=/tmp/multiop_open_wait_pipe.6739 start ost2 service on oleg232-server Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg232-server: oleg232-server.virtnet: executing set_default_debug -1 all pdsh@oleg232-client: oleg232-server: ssh exited with exit code 1 Started lustre-OST0001 oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec stop ost2 service on oleg232-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg232-server umount lustre on /mnt/lustre..... Stopping client oleg232-client.virtnet /mnt/lustre (opts:-f) stop ost1 service on oleg232-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg232-server stop mds service on oleg232-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg232-server oleg232-server: oleg232-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre oleg232-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg232-client: oleg232-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg232-server: oleg232-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre oleg232-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg232-client: oleg232-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg232-server: oleg232-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre oleg232-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg232-client: oleg232-server: ssh exited with exit code 19 checking for existing Lustre data: not found tunefs failed, reformatting instead Stopping clients: oleg232-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg232-client.virtnet /mnt/lustre2 (opts:-f) oleg232-server: oleg232-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg232-server' oleg232-server: oleg232-server.virtnet: executing load_modules_local oleg232-server: Loading modules from /home/green/git/lustre-release/lustre oleg232-server: detected 4 online CPUs by sysfs oleg232-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg232-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg232-server: oleg232-server.virtnet: executing set_default_debug -1 all pdsh@oleg232-client: oleg232-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg232-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg232-server: oleg232-server.virtnet: executing set_default_debug -1 all pdsh@oleg232-client: oleg232-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg232-server: oleg232-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg232-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg232-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg232-server stop mds service on oleg232-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg232-server PASS 50f (134s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 69: replace an OST with the same index ========================================================== 16:00:49 (1710100849) start mds service on oleg232-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg232-server: oleg232-server.virtnet: executing set_default_debug -1 all pdsh@oleg232-client: oleg232-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg232-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg232-server: oleg232-server.virtnet: executing set_default_debug -1 all pdsh@oleg232-client: oleg232-server: ssh exited with exit code 1 Started lustre-OST0000 oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg232-client.virtnet: -o user_xattr,flock oleg232-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11872 inodes available. Want 99936. rc=0 - open/close 1458 (time 1710100880.01 total 10.00 last 145.78) - open/close 2896 (time 1710100890.01 total 20.00 last 143.76) - open/close 4117 (time 1710100900.01 total 30.01 last 122.09) - open/close 5292 (time 1710100910.02 total 40.01 last 117.40) - open/close 6118 (time 1710100920.03 total 50.02 last 82.54) - open/close 7022 (time 1710100930.03 total 60.02 last 90.39) - open/close 7790 (time 1710100940.05 total 70.04 last 76.69) - open/close 8442 (time 1710100950.06 total 80.05 last 65.08) - open/close 9062 (time 1710100960.07 total 90.06 last 61.95) - open/close 9769 (time 1710100970.08 total 100.07 last 70.62) total: 10000 open/close in 103.21 seconds: 96.89 ops/second - unlinked 0 (time 1710100974 ; total 0 ; last 0) total: 10000 unlinks in 71 seconds: 140.845078 unlinks/second - open/close 1170 (time 1710101057.65 total 10.01 last 116.92) - open/close 2144 (time 1710101067.66 total 20.02 last 97.29) - open/close 2805 (time 1710101086.41 total 38.76 last 35.27) - open/close 3649 (time 1710101096.41 total 48.76 last 84.39) - open/close 4474 (time 1710101106.41 total 58.77 last 82.47) - open/close 5202 (time 1710101116.41 total 68.77 last 72.79) - open/close 5870 (time 1710101126.41 total 78.77 last 66.78) - open/close 6562 (time 1710101136.42 total 88.78 last 69.13) - open/close 7273 (time 1710101146.43 total 98.79 last 71.04) - open/close 7914 (time 1710101156.45 total 108.80 last 64.02) - open/close 8514 (time 1710101166.47 total 118.83 last 59.83) - open/close 9126 (time 1710101176.50 total 128.85 last 61.06) - open/close 9683 (time 1710101186.52 total 138.88 last 55.57) total: 10000 open/close in 145.04 seconds: 68.95 ops/second - unlinked 0 (time 1710101194 ; total 0 ; last 0) total: 10000 unlinks in 104 seconds: 96.153847 unlinks/second - open/close 914 (time 1710101310.40 total 10.00 last 91.39) - open/close 1774 (time 1710101320.41 total 20.01 last 85.95) - open/close 2638 (time 1710101330.42 total 30.02 last 86.32) - open/close 3377 (time 1710101340.43 total 40.03 last 73.79) - open/close 4036 (time 1710101350.43 total 50.03 last 65.88) - open/close 4725 (time 1710101360.45 total 60.05 last 68.80) - open/close 5406 (time 1710101370.46 total 70.06 last 68.02) - open/close 6112 (time 1710101380.47 total 80.07 last 70.55) - open/close 6784 (time 1710101390.47 total 90.07 last 67.20) - open/close 7406 (time 1710101400.47 total 100.07 last 62.17) - open/close 8018 (time 1710101410.48 total 110.08 last 61.14) - open/close 8591 (time 1710101420.49 total 120.09 last 57.26) - open/close 9169 (time 1710101430.51 total 130.11 last 57.69) - open/close 9787 (time 1710101440.51 total 140.11 last 61.80) total: 10000 open/close in 144.12 seconds: 69.39 ops/second - unlinked 0 (time 1710101446 ; total 0 ; last 0) total: 10000 unlinks in 96 seconds: 104.166664 unlinks/second - open/close 1020 (time 1710101554.56 total 10.00 last 102.00) - open/close 1924 (time 1710101564.56 total 20.00 last 90.39) - open/close 2798 (time 1710101574.57 total 30.01 last 87.37) - open/close 3572 (time 1710101584.57 total 40.01 last 77.34) - open/close 4343 (time 1710101594.58 total 50.02 last 77.06) - open/close 5029 (time 1710101604.60 total 60.04 last 68.47) - open/close 5764 (time 1710101614.60 total 70.04 last 73.47) - open/close 6452 (time 1710101624.61 total 80.05 last 68.77) - open/close 7152 (time 1710101634.62 total 90.06 last 69.89) - open/close 7769 (time 1710101644.62 total 100.06 last 61.69) - open/close 8370 (time 1710101654.63 total 110.07 last 60.06) - open/close 9031 (time 1710101664.64 total 120.07 last 66.06) - open/close 9615 (time 1710101674.65 total 130.08 last 58.34) total: 10000 open/close in 137.02 seconds: 72.98 ops/second - unlinked 0 (time 1710101683 ; total 0 ; last 0) total: 10000 unlinks in 102 seconds: 98.039215 unlinks/second - open/close 1012 (time 1710101797.47 total 10.01 last 101.10) - open/close 1886 (time 1710101807.47 total 20.02 last 87.35) - open/close 2663 (time 1710101817.48 total 30.02 last 77.64) - open/close 3483 (time 1710101827.48 total 40.03 last 81.98) - open/close 4267 (time 1710101837.49 total 50.03 last 78.37) - open/close 4991 (time 1710101847.50 total 60.04 last 72.35) - open/close 5618 (time 1710101857.50 total 70.05 last 62.65) - open/close 6225 (time 1710101867.51 total 80.05 last 60.64) - open/close 6823 (time 1710101877.53 total 90.07 last 59.69) - open/close 7444 (time 1710101887.54 total 100.08 last 62.04) - open/close 8014 (time 1710101897.55 total 110.09 last 56.94) - open/close 8627 (time 1710101907.57 total 120.11 last 61.19) - open/close 9170 (time 1710101917.58 total 130.12 last 54.24) - open/close 9746 (time 1710101927.59 total 140.13 last 57.58) total: 10000 open/close in 145.38 seconds: 68.78 ops/second - unlinked 0 (time 1710101934 ; total 0 ; last 0) total: 10000 unlinks in 97 seconds: 103.092781 unlinks/second - open/close 1075 (time 1710102043.47 total 10.00 last 107.46) - open/close 2015 (time 1710102053.48 total 20.01 last 93.93) - open/close 2831 (time 1710102063.49 total 30.02 last 81.52) - open/close 3647 (time 1710102073.49 total 40.03 last 81.53) - open/close 4454 (time 1710102083.51 total 50.04 last 80.62) - open/close 5156 (time 1710102093.51 total 60.05 last 70.15) - open/close 5873 (time 1710102103.52 total 70.05 last 71.67) - open/close 6487 (time 1710102113.52 total 80.05 last 61.40) - open/close 7094 (time 1710102123.52 total 90.05 last 60.69) - open/close 7722 (time 1710102133.53 total 100.06 last 62.73) - open/close 8287 (time 1710102143.55 total 110.09 last 56.38) - open/close 8860 (time 1710102153.55 total 120.09 last 57.28) - open/close 9384 (time 1710102163.58 total 130.11 last 52.29) - open/close 9899 (time 1710102173.58 total 140.11 last 51.48) total: 10000 open/close in 142.01 seconds: 70.42 ops/second - unlinked 0 (time 1710102177 ; total 0 ; last 0) total: 10000 unlinks in 105 seconds: 95.238098 unlinks/second - open/close 1015 (time 1710102294.36 total 10.00 last 101.45) - open/close 1932 (time 1710102304.37 total 20.01 last 91.61) - open/close 2793 (time 1710102314.38 total 30.02 last 86.07) - open/close 3624 (time 1710102324.38 total 40.02 last 83.06) - open/close 4317 (time 1710102334.39 total 50.03 last 69.25) - open/close 5073 (time 1710102344.39 total 60.03 last 75.58) - open/close 5815 (time 1710102354.40 total 70.04 last 74.14) - open/close 6455 (time 1710102364.40 total 80.04 last 63.98) - open/close 7093 (time 1710102374.41 total 90.05 last 63.78) - open/close 7747 (time 1710102384.41 total 100.05 last 65.37) - open/close 8300 (time 1710102394.42 total 110.06 last 55.24) - open/close 8879 (time 1710102404.43 total 120.07 last 57.85) - open/close 9473 (time 1710102414.44 total 130.08 last 59.36) total: 10000 open/close in 139.99 seconds: 71.44 ops/second - unlinked 0 (time 1710102426 ; total 0 ; last 0) total: 10000 unlinks in 102 seconds: 98.039215 unlinks/second - open/close 1072 (time 1710102540.23 total 10.01 last 107.12) - open/close 2079 (time 1710102550.24 total 20.01 last 100.65) - open/close 2968 (time 1710102560.25 total 30.02 last 88.84) - open/close 3806 (time 1710102570.26 total 40.03 last 83.70) - open/close 4621 (time 1710102580.27 total 50.04 last 81.40) - open/close 5384 (time 1710102590.29 total 60.06 last 76.19) - open/close 6093 (time 1710102600.29 total 70.06 last 70.86) - open/close 6821 (time 1710102610.31 total 80.08 last 72.66) - open/close 7488 (time 1710102620.33 total 90.10 last 66.59) - open/close 8127 (time 1710102630.33 total 100.10 last 63.90) - open/close 8747 (time 1710102640.33 total 110.11 last 61.97) - open/close 9325 (time 1710102650.34 total 120.11 last 57.77) - open/close 9855 (time 1710102660.35 total 130.12 last 52.94) total: 10000 open/close in 132.86 seconds: 75.27 ops/second - unlinked 0 (time 1710102665 ; total 0 ; last 0) total: 10000 unlinks in 87 seconds: 114.942528 unlinks/second - open/close 1079 (time 1710102764.69 total 10.00 last 107.89) - open/close 2182 (time 1710102774.70 total 20.01 last 110.19) - open/close 3165 (time 1710102784.70 total 30.01 last 98.30) - open/close 4066 (time 1710102794.70 total 40.02 last 90.07) - open/close 4930 (time 1710102804.71 total 50.02 last 86.33) - open/close 5738 (time 1710102814.71 total 60.02 last 80.79) - open/close 6560 (time 1710102824.73 total 70.04 last 82.08) - open/close 7298 (time 1710102834.74 total 80.05 last 73.70) - open/close 8052 (time 1710102844.76 total 90.07 last 75.25) - open/close 8734 (time 1710102854.78 total 100.09 last 68.08) - open/close 9434 (time 1710102864.78 total 110.09 last 69.97) total: 10000 open/close in 117.45 seconds: 85.14 ops/second - unlinked 0 (time 1710102873 ; total 0 ; last 0) total: 10000 unlinks in 67 seconds: 149.253738 unlinks/second - open/close 1574 (time 1710102951.98 total 10.00 last 157.35) - open/close 3060 (time 1710102961.99 total 20.01 last 148.54) - open/close 4506 (time 1710102971.99 total 30.01 last 144.52) - open/close 5947 (time 1710102981.99 total 40.01 last 144.09) - open/close 7464 (time 1710102992.00 total 50.01 last 151.68) - open/close 8842 (time 1710103002.00 total 60.02 last 137.73) total: 10000 open/close in 69.15 seconds: 144.61 ops/second - unlinked 0 (time 1710103012 ; total 0 ; last 0) total: 10000 unlinks in 42 seconds: 238.095245 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg232-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg232-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg232-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.202.132@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.202.132@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg232-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg232-server: oleg232-server.virtnet: executing set_default_debug -1 all pdsh@oleg232-client: oleg232-server: ssh exited with exit code 1 Started lustre-OST0000 oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg232-server: oleg232-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg232-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec mount lustre on /mnt/lustre..... Starting client: oleg232-client.virtnet: -o user_xattr,flock oleg232-server@tcp:/lustre /mnt/lustre On OST0, 17717 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg232-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg232-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg232-server stop mds service on oleg232-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg232-server unloading modules on: 'oleg232-server' oleg232-server: oleg232-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg232-client: oleg232-client: ssh exited with exit code 2 pdsh@oleg232-client: oleg232-server: ssh exited with exit code 2 pdsh@oleg232-client: oleg232-client: ssh exited with exit code 2 pdsh@oleg232-client: oleg232-server: ssh exited with exit code 2 PASS 69 (2238s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg232-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg232-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg232-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg232-client: oleg232-server: ssh exited with exit code 2 oleg232-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg232-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == conf-sanity test 93: register mulitple MDT at the same time ========================================================== 16:38:09 (1710103089) SKIP: conf-sanity test_93 needs >= 3 MDTs SKIP 93 (1s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg232-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg232-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory Stopping clients: oleg232-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg232-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg232-client: oleg232-server: ssh exited with exit code 2 oleg232-server: oleg232-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg232-server' oleg232-server: oleg232-server.virtnet: executing load_modules_local oleg232-server: Loading modules from /home/green/git/lustre-release/lustre oleg232-server: detected 4 online CPUs by sysfs oleg232-server: Force libcfs to create 2 CPU partitions oleg232-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg232-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 3099 sec ========== 16:38:28 (1710103108) === conf-sanity: start cleanup 16:38:29 (1710103109) === === conf-sanity: finish cleanup 16:38:29 (1710103109) ===