-----============= acceptance-small: conf-sanity ============----- Sun Mar 10 15:45:50 EDT 2024 excepting tests: 32newtarball 110 skipping tests SLOW=no: 45 69 106 111 114 Stopping clients: oleg247-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg247-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg247-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg247-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg247-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg247-server oleg247-server: oleg247-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg247-server' oleg247-server: oleg247-server.virtnet: executing load_modules_local oleg247-server: Loading modules from /home/green/git/lustre-release/lustre oleg247-server: detected 4 online CPUs by sysfs oleg247-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg247-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg247-server: oleg247-server.virtnet: executing set_default_debug -1 all pdsh@oleg247-client: oleg247-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg247-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg247-server: oleg247-server.virtnet: executing set_default_debug -1 all pdsh@oleg247-client: oleg247-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg247-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg247-server: oleg247-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg247-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg247-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg247-server stop mds service on oleg247-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg247-server debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 15:46:48 (1710100008) start mds service on oleg247-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg247-server: oleg247-server.virtnet: executing set_default_debug -1 all pdsh@oleg247-client: oleg247-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg247-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg247-server: oleg247-server.virtnet: executing set_default_debug -1 all pdsh@oleg247-client: oleg247-server: ssh exited with exit code 1 Started lustre-OST0000 oleg247-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg247-server: oleg247-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg247-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec stop ost1 service on oleg247-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg247-server stop mds service on oleg247-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg247-server PASS 21a (20s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 15:47:10 (1710100030) start ost1 service on oleg247-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg247-server: oleg247-server.virtnet: executing set_default_debug -1 all pdsh@oleg247-client: oleg247-server: ssh exited with exit code 1 Started lustre-OST0000 oleg247-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid start mds service on oleg247-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg247-server: oleg247-server.virtnet: executing set_default_debug -1 all pdsh@oleg247-client: oleg247-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg247-server: oleg247-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg247-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop mds service on oleg247-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg247-server stop ost1 service on oleg247-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg247-server PASS 21b (197s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 15:50:28 (1710100228) start mds service on oleg247-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg247-server: oleg247-server.virtnet: executing set_default_debug -1 all pdsh@oleg247-client: oleg247-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid Client mount with ost in logs, but none running start ost1 service on oleg247-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg247-server: oleg247-server.virtnet: executing set_default_debug -1 all pdsh@oleg247-client: oleg247-server: ssh exited with exit code 1 Started lustre-OST0000 oleg247-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg247-server: oleg247-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg247-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 5 sec stop ost1 service on oleg247-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg247-server mount lustre on /mnt/lustre..... Starting client: oleg247-client.virtnet: -o user_xattr,flock oleg247-server@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg247-client.virtnet /mnt/lustre (opts:-f) PASS Client mount with a running ost start ost1 service on oleg247-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg247-server: oleg247-server.virtnet: executing set_default_debug -1 all pdsh@oleg247-client: oleg247-server: ssh exited with exit code 1 Started lustre-OST0000 oleg247-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg247-client.virtnet: -o user_xattr,flock oleg247-server@tcp:/lustre /mnt/lustre oleg247-server: oleg247-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg247-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg247-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8800b64c0800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b64c0800.ost_server_uuid in FULL state after 0 sec setup single mount lustre success PASS umount lustre on /mnt/lustre..... Stopping client oleg247-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg247-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg247-server stop mds service on oleg247-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg247-server unloading modules on: 'oleg247-server' oleg247-server: oleg247-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg247-client: oleg247-client: ssh exited with exit code 2 pdsh@oleg247-client: oleg247-server: ssh exited with exit code 2 pdsh@oleg247-client: oleg247-client: ssh exited with exit code 2 pdsh@oleg247-client: oleg247-server: ssh exited with exit code 2 PASS 22 (40s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg247-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg247-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg247-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg247-client: oleg247-server: ssh exited with exit code 2 oleg247-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg247-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == conf-sanity test 46a: handle ost additional - wide striped file ========================================================== 15:51:10 (1710100270) Testing with 2 OSTs Stopping clients: oleg247-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg247-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg247-client: oleg247-server: ssh exited with exit code 2 oleg247-server: oleg247-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg247-server' oleg247-server: oleg247-server.virtnet: executing load_modules_local oleg247-server: Loading modules from /home/green/git/lustre-release/lustre oleg247-server: detected 4 online CPUs by sysfs oleg247-server: Force libcfs to create 2 CPU partitions oleg247-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg247-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg247-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg247-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg247-server: oleg247-server.virtnet: executing set_default_debug -1 all pdsh@oleg247-client: oleg247-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg247-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg247-server: oleg247-server.virtnet: executing set_default_debug -1 all pdsh@oleg247-client: oleg247-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg247-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg247-server: oleg247-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg247-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg247-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg247-server stop mds service on oleg247-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg247-server start mds service on oleg247-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg247-server: oleg247-server.virtnet: executing set_default_debug -1 all pdsh@oleg247-client: oleg247-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg247-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg247-server: oleg247-server.virtnet: executing set_default_debug -1 all pdsh@oleg247-client: oleg247-server: ssh exited with exit code 1 Started lustre-OST0000 oleg247-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg247-server: oleg247-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg247-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 4 sec mount lustre on /mnt/lustre..... Starting client: oleg247-client.virtnet: -o user_xattr,flock oleg247-server@tcp:/lustre /mnt/lustre Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg247-server: oleg247-server.virtnet: executing set_default_debug -1 all pdsh@oleg247-client: oleg247-server: ssh exited with exit code 1 Commit the device label on lustre-ost2/ost2 Started lustre-OST0001 oleg247-server: oleg247-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 oleg247-server: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg247-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8800af084000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800af084000.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre2..... Starting client: oleg247-client.virtnet: -o user_xattr,flock oleg247-server@tcp:/lustre /mnt/lustre2 /mnt/lustre2 stripe_count: -1 stripe_size: 4194304 pattern: raid0 stripe_offset: -1 /mnt/lustre2/widestripe lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 2 0x2 0x280000400 0 34 0x22 0x240000400 File: '/mnt/lustre/widestripe' Size: 3 Blocks: 2 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115238826934273 Links: 1 Access: (0674/-rw-rwxr--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-03-10 15:52:08.000000000 -0400 Modify: 2024-03-10 15:52:08.000000000 -0400 Change: 2024-03-10 15:52:08.000000000 -0400 Birth: - umount lustre on /mnt/lustre2..... Stopping client oleg247-client.virtnet /mnt/lustre2 (opts:) umount lustre on /mnt/lustre..... Stopping client oleg247-client.virtnet /mnt/lustre (opts:) Stopping /mnt/lustre-ost2 (opts:-f) on oleg247-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg247-server stop mds service on oleg247-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg247-server stop ost1 service on oleg247-server stop mds service on oleg247-server unloading modules on: 'oleg247-server' oleg247-server: oleg247-server.virtnet: executing unload_modules_local modules unloaded. oleg247-server: oleg247-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre oleg247-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg247-client: oleg247-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg247-server: oleg247-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre oleg247-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg247-client: oleg247-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg247-server: oleg247-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre oleg247-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg247-client: oleg247-server: ssh exited with exit code 19 checking for existing Lustre data: not found tunefs failed, reformatting instead Stopping clients: oleg247-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg247-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg247-client: oleg247-server: ssh exited with exit code 2 oleg247-server: oleg247-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg247-server' oleg247-server: oleg247-server.virtnet: executing load_modules_local oleg247-server: Loading modules from /home/green/git/lustre-release/lustre oleg247-server: detected 4 online CPUs by sysfs oleg247-server: Force libcfs to create 2 CPU partitions oleg247-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory oleg247-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg247-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg247-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg247-server: oleg247-server.virtnet: executing set_default_debug -1 all pdsh@oleg247-client: oleg247-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg247-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg247-server: oleg247-server.virtnet: executing set_default_debug -1 all pdsh@oleg247-client: oleg247-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg247-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg247-server: oleg247-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg247-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg247-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg247-server stop mds service on oleg247-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg247-server PASS 46a (136s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 50b: lazystatfs all servers down ===== 15:53:28 (1710100408) start mds service on oleg247-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg247-server: oleg247-server.virtnet: executing set_default_debug -1 all pdsh@oleg247-client: oleg247-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg247-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg247-server: oleg247-server.virtnet: executing set_default_debug -1 all pdsh@oleg247-client: oleg247-server: ssh exited with exit code 1 Started lustre-OST0000 oleg247-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg247-client.virtnet: -o user_xattr,flock oleg247-server@tcp:/lustre /mnt/lustre llite.lustre-ffff8800af16b000.lazystatfs=1 stop ost1 service on oleg247-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg247-server oleg247-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff8800af16b000.ost_server_uuid 50 0 UP mgc MGC192.168.202.147@tcp 2bff66d1-201e-4b8e-af13-3cc38ed0041d 5 1 UP lov lustre-clilov-ffff8800af16b000 3c1f25ff-f18c-4963-bb1e-7bc60f391013 4 2 UP lmv lustre-clilmv-ffff8800af16b000 3c1f25ff-f18c-4963-bb1e-7bc60f391013 5 3 UP mdc lustre-MDT0000-mdc-ffff8800af16b000 3c1f25ff-f18c-4963-bb1e-7bc60f391013 5 4 UP osc lustre-OST0000-osc-ffff8800af16b000 3c1f25ff-f18c-4963-bb1e-7bc60f391013 5 OSCs should all be DISCONN multiop /mnt/lustre vf_ TMPPIPE=/tmp/multiop_open_wait_pipe.6758 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 383616 3200 378368 1% /mnt/lustre[MDT:0] filesystem_summary: 383616 3200 378368 1% /mnt/lustre /home/green/git/lustre-release/lustre/tests/conf-sanity.sh: line 4282: kill: (27311) - No such process umount lustre on /mnt/lustre..... Stopping client oleg247-client.virtnet /mnt/lustre (opts:) stop mds service on oleg247-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg247-server PASS 50b (44s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 50c: lazystatfs one server down ====== 15:54:14 (1710100454) start mds service on oleg247-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg247-server: oleg247-server.virtnet: executing set_default_debug -1 all pdsh@oleg247-client: oleg247-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg247-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg247-server: oleg247-server.virtnet: executing set_default_debug -1 all pdsh@oleg247-client: oleg247-server: ssh exited with exit code 1 Started lustre-OST0000 oleg247-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid start ost2 service on oleg247-server Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg247-server: oleg247-server.virtnet: executing set_default_debug -1 all pdsh@oleg247-client: oleg247-server: ssh exited with exit code 1 Commit the device label on lustre-ost2/ost2 Started lustre-OST0001 oleg247-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg247-client.virtnet: -o user_xattr,flock oleg247-server@tcp:/lustre /mnt/lustre llite.lustre-ffff88012b2b4000.lazystatfs=1 stop ost1 service on oleg247-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg247-server oleg247-server: oleg247-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg247-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec multiop /mnt/lustre vf_ TMPPIPE=/tmp/multiop_open_wait_pipe.6758 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 383616 3328 378240 1% /mnt/lustre[MDT:0] lustre-OST0001_UUID 382976 3072 377856 1% /mnt/lustre[OST:1] filesystem_summary: 382976 3072 377856 1% /mnt/lustre /home/green/git/lustre-release/lustre/tests/conf-sanity.sh: line 4282: kill: (29546) - No such process umount lustre on /mnt/lustre..... Stopping client oleg247-client.virtnet /mnt/lustre (opts:) stop ost2 service on oleg247-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg247-server stop mds service on oleg247-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg247-server oleg247-server: oleg247-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre oleg247-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg247-client: oleg247-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg247-server: oleg247-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre oleg247-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg247-client: oleg247-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg247-server: oleg247-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre oleg247-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg247-client: oleg247-server: ssh exited with exit code 19 checking for existing Lustre data: not found tunefs failed, reformatting instead Stopping clients: oleg247-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg247-client.virtnet /mnt/lustre2 (opts:-f) oleg247-server: oleg247-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory loading modules on: 'oleg247-server' oleg247-server: oleg247-server.virtnet: executing load_modules_local oleg247-server: Loading modules from /home/green/git/lustre-release/lustre oleg247-server: detected 4 online CPUs by sysfs oleg247-server: Force libcfs to create 2 CPU partitions oleg247-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg247-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg247-server: oleg247-server.virtnet: executing set_default_debug -1 all pdsh@oleg247-client: oleg247-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg247-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg247-server: oleg247-server.virtnet: executing set_default_debug -1 all pdsh@oleg247-client: oleg247-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg247-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg247-server: oleg247-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg247-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg247-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg247-server stop mds service on oleg247-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg247-server PASS 50c (87s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 50e: normal statfs all servers down == 15:55:43 (1710100543) Stopping clients: oleg247-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg247-client.virtnet /mnt/lustre2 (opts:-f) oleg247-server: oleg247-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg247-server' oleg247-server: oleg247-server.virtnet: executing load_modules_local oleg247-server: Loading modules from /home/green/git/lustre-release/lustre oleg247-server: detected 4 online CPUs by sysfs oleg247-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg247-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg247-server: oleg247-server.virtnet: executing set_default_debug -1 all pdsh@oleg247-client: oleg247-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg247-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg247-server: oleg247-server.virtnet: executing set_default_debug -1 all pdsh@oleg247-client: oleg247-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg247-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg247-server: oleg247-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg247-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg247-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg247-server stop mds service on oleg247-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg247-server start mds service on oleg247-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg247-server: oleg247-server.virtnet: executing set_default_debug -1 all pdsh@oleg247-client: oleg247-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg247-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg247-server: oleg247-server.virtnet: executing set_default_debug -1 all pdsh@oleg247-client: oleg247-server: ssh exited with exit code 1 Started lustre-OST0000 oleg247-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg247-server: oleg247-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg247-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec stop ost1 service on oleg247-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg247-server oleg247-server: oleg247-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg247-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg247-client.virtnet: -o user_xattr,flock oleg247-server@tcp:/lustre /mnt/lustre llite.lustre-ffff88013766e800.lazystatfs=0 multiop /mnt/lustre v_f TMPPIPE=/tmp/multiop_open_wait_pipe.6758 start ost1 service on oleg247-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg247-server: oleg247-server.virtnet: executing set_default_debug -1 all pdsh@oleg247-client: oleg247-server: ssh exited with exit code 1 Started lustre-OST0000 oleg247-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec umount lustre on /mnt/lustre..... Stopping client oleg247-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg247-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg247-server stop mds service on oleg247-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg247-server PASS 50e (90s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 50f: normal statfs one server in down ========================================================== 15:57:15 (1710100635) start mds service on oleg247-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg247-server: oleg247-server.virtnet: executing set_default_debug -1 all pdsh@oleg247-client: oleg247-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg247-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg247-server: oleg247-server.virtnet: executing set_default_debug -1 all pdsh@oleg247-client: oleg247-server: ssh exited with exit code 1 Started lustre-OST0000 oleg247-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg247-server: oleg247-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg247-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec start ost2 service on oleg247-server Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg247-server: oleg247-server.virtnet: executing set_default_debug -1 all pdsh@oleg247-client: oleg247-server: ssh exited with exit code 1 Commit the device label on lustre-ost2/ost2 Started lustre-OST0001 oleg247-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid oleg247-server: oleg247-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 oleg247-server: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost2 service on oleg247-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg247-server oleg247-server: oleg247-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 oleg247-server: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg247-client.virtnet: -o user_xattr,flock oleg247-server@tcp:/lustre /mnt/lustre llite.lustre-ffff8800b5d1a000.lazystatfs=0 multiop /mnt/lustre v_f TMPPIPE=/tmp/multiop_open_wait_pipe.6758 start ost2 service on oleg247-server Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg247-server: oleg247-server.virtnet: executing set_default_debug -1 all pdsh@oleg247-client: oleg247-server: ssh exited with exit code 1 Started lustre-OST0001 oleg247-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 1 sec stop ost2 service on oleg247-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg247-server umount lustre on /mnt/lustre..... Stopping client oleg247-client.virtnet /mnt/lustre (opts:-f) stop ost1 service on oleg247-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg247-server stop mds service on oleg247-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg247-server oleg247-server: oleg247-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre oleg247-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg247-client: oleg247-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg247-server: oleg247-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre oleg247-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg247-client: oleg247-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg247-server: oleg247-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre oleg247-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg247-client: oleg247-server: ssh exited with exit code 19 checking for existing Lustre data: not found tunefs failed, reformatting instead Stopping clients: oleg247-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg247-client.virtnet /mnt/lustre2 (opts:-f) oleg247-server: oleg247-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg247-server' oleg247-server: oleg247-server.virtnet: executing load_modules_local oleg247-server: Loading modules from /home/green/git/lustre-release/lustre oleg247-server: detected 4 online CPUs by sysfs oleg247-server: Force libcfs to create 2 CPU partitions oleg247-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg247-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg247-server: oleg247-server.virtnet: executing set_default_debug -1 all pdsh@oleg247-client: oleg247-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg247-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg247-server: oleg247-server.virtnet: executing set_default_debug -1 all pdsh@oleg247-client: oleg247-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg247-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg247-server: oleg247-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg247-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg247-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg247-server stop mds service on oleg247-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg247-server PASS 50f (128s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 69: replace an OST with the same index ========================================================== 15:59:25 (1710100765) start mds service on oleg247-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg247-server: oleg247-server.virtnet: executing set_default_debug -1 all pdsh@oleg247-client: oleg247-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg247-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg247-server: oleg247-server.virtnet: executing set_default_debug -1 all pdsh@oleg247-client: oleg247-server: ssh exited with exit code 1 Started lustre-OST0000 oleg247-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg247-client.virtnet: -o user_xattr,flock oleg247-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11872 inodes available. Want 99936. rc=0 - open/close 1638 (time 1710100793.06 total 10.00 last 163.75) - open/close 2966 (time 1710100803.06 total 20.01 last 132.76) - open/close 4112 (time 1710100813.06 total 30.01 last 114.59) - open/close 5168 (time 1710100823.06 total 40.01 last 105.58) - open/close 6181 (time 1710100833.06 total 50.01 last 101.30) - open/close 7279 (time 1710100843.06 total 60.01 last 109.77) - open/close 8291 (time 1710100853.07 total 70.02 last 101.16) - open/close 9128 (time 1710100863.07 total 80.02 last 83.70) total: 10000 open/close in 89.88 seconds: 111.26 ops/second - unlinked 0 (time 1710100874 ; total 0 ; last 0) total: 10000 unlinks in 65 seconds: 153.846161 unlinks/second - open/close 1149 (time 1710100951.80 total 10.00 last 114.84) - open/close 2106 (time 1710100961.81 total 20.01 last 95.67) - open/close 3219 (time 1710100971.81 total 30.01 last 111.27) - open/close 4182 (time 1710100981.81 total 40.01 last 96.26) - open/close 5145 (time 1710100991.82 total 50.02 last 96.22) - open/close 6111 (time 1710101001.83 total 60.03 last 96.51) - open/close 6928 (time 1710101011.84 total 70.04 last 81.61) - open/close 7658 (time 1710101021.84 total 80.04 last 72.99) - open/close 8485 (time 1710101031.85 total 90.05 last 82.67) - open/close 9233 (time 1710101041.85 total 100.05 last 74.78) - open/close 9855 (time 1710101051.86 total 110.06 last 62.15) total: 10000 open/close in 112.33 seconds: 89.02 ops/second - unlinked 0 (time 1710101055 ; total 0 ; last 0) total: 10000 unlinks in 91 seconds: 109.890106 unlinks/second - open/close 1039 (time 1710101158.32 total 10.00 last 103.89) - open/close 1999 (time 1710101168.32 total 20.00 last 95.99) - open/close 2937 (time 1710101178.33 total 30.01 last 93.71) - open/close 3760 (time 1710101188.34 total 40.02 last 82.24) - open/close 4558 (time 1710101198.35 total 50.03 last 79.69) - open/close 5271 (time 1710101208.35 total 60.03 last 71.28) - open/close 6014 (time 1710101218.36 total 70.04 last 74.25) - open/close 6632 (time 1710101228.37 total 80.05 last 61.73) - open/close 7257 (time 1710101238.37 total 90.06 last 62.47) - open/close 7894 (time 1710101248.38 total 100.07 last 63.63) - open/close 8505 (time 1710101258.39 total 110.07 last 61.07) - open/close 9070 (time 1710101268.40 total 120.09 last 56.42) - open/close 9625 (time 1710101278.42 total 130.10 last 55.42) total: 10000 open/close in 137.18 seconds: 72.89 ops/second - unlinked 0 (time 1710101287 ; total 0 ; last 0) total: 10000 unlinks in 99 seconds: 101.010101 unlinks/second - open/close 1028 (time 1710101398.46 total 10.00 last 102.76) - open/close 1922 (time 1710101408.47 total 20.01 last 89.30) - open/close 2784 (time 1710101418.49 total 30.03 last 86.07) - open/close 3619 (time 1710101428.51 total 40.05 last 83.35) - open/close 4410 (time 1710101438.51 total 50.05 last 79.08) - open/close 5174 (time 1710101448.52 total 60.06 last 76.34) - open/close 5866 (time 1710101458.53 total 70.07 last 69.11) - open/close 6511 (time 1710101468.54 total 80.08 last 64.45) - open/close 7112 (time 1710101478.54 total 90.08 last 60.09) - open/close 7712 (time 1710101488.64 total 100.18 last 59.42) - open/close 8305 (time 1710101498.64 total 110.18 last 59.27) - open/close 8885 (time 1710101508.66 total 120.20 last 57.92) - open/close 9465 (time 1710101518.66 total 130.20 last 57.98) total: 10000 open/close in 139.57 seconds: 71.65 ops/second - unlinked 0 (time 1710101530 ; total 0 ; last 0) total: 10000 unlinks in 86 seconds: 116.279068 unlinks/second - open/close 1097 (time 1710101628.35 total 10.01 last 109.61) - open/close 2111 (time 1710101638.35 total 20.01 last 101.39) - open/close 2985 (time 1710101648.36 total 30.02 last 87.33) - open/close 3866 (time 1710101658.36 total 40.02 last 88.03) - open/close 4753 (time 1710101668.37 total 50.04 last 88.60) - open/close 5548 (time 1710101678.37 total 60.04 last 79.50) - open/close 6330 (time 1710101688.39 total 70.05 last 78.11) - open/close 7102 (time 1710101698.39 total 80.05 last 77.15) - open/close 7809 (time 1710101708.39 total 90.06 last 70.69) - open/close 8518 (time 1710101718.41 total 100.07 last 70.77) - open/close 9154 (time 1710101728.42 total 110.08 last 63.53) - open/close 9829 (time 1710101738.49 total 120.15 last 67.08) total: 10000 open/close in 123.05 seconds: 81.27 ops/second - unlinked 0 (time 1710101743 ; total 0 ; last 0) total: 10000 unlinks in 84 seconds: 119.047623 unlinks/second - open/close 1086 (time 1710101838.95 total 10.00 last 108.58) - open/close 2100 (time 1710101848.96 total 20.01 last 101.34) - open/close 3004 (time 1710101858.96 total 30.01 last 90.37) - open/close 3831 (time 1710101868.98 total 40.03 last 82.53) - open/close 4658 (time 1710101878.99 total 50.04 last 82.67) - open/close 5445 (time 1710101889.00 total 60.04 last 78.63) - open/close 6217 (time 1710101899.00 total 70.05 last 77.15) - open/close 7028 (time 1710101909.02 total 80.06 last 80.99) - open/close 7125 (time 1710101921.64 total 92.69 last 7.68) - open/close 7792 (time 1710101931.67 total 102.72 last 66.50) - open/close 8417 (time 1710101941.68 total 112.73 last 62.47) - open/close 9002 (time 1710101951.68 total 122.73 last 58.49) - open/close 9587 (time 1710101961.68 total 132.73 last 58.47) total: 10000 open/close in 139.44 seconds: 71.71 ops/second - unlinked 0 (time 1710101970 ; total 0 ; last 0) total: 10000 unlinks in 80 seconds: 125.000000 unlinks/second - open/close 1097 (time 1710102062.26 total 10.01 last 109.64) - open/close 2145 (time 1710102072.26 total 20.01 last 104.78) - open/close 3155 (time 1710102082.26 total 30.01 last 100.98) - open/close 4073 (time 1710102092.27 total 40.01 last 91.78) - open/close 4977 (time 1710102102.28 total 50.02 last 90.29) - open/close 5746 (time 1710102112.28 total 60.03 last 76.86) - open/close 6509 (time 1710102122.29 total 70.04 last 76.23) - open/close 7240 (time 1710102132.30 total 80.05 last 73.03) - open/close 7909 (time 1710102142.31 total 90.05 last 66.87) - open/close 8621 (time 1710102152.31 total 100.06 last 71.17) - open/close 9271 (time 1710102162.32 total 110.07 last 64.93) - open/close 9911 (time 1710102172.34 total 120.08 last 63.91) total: 10000 open/close in 121.45 seconds: 82.34 ops/second - unlinked 0 (time 1710102175 ; total 0 ; last 0) total: 10000 unlinks in 80 seconds: 125.000000 unlinks/second - open/close 1150 (time 1710102267.18 total 10.01 last 114.92) - open/close 2162 (time 1710102277.19 total 20.01 last 101.19) - open/close 3066 (time 1710102287.19 total 30.01 last 90.36) - open/close 3689 (time 1710102304.66 total 47.49 last 35.65) - open/close 4588 (time 1710102314.67 total 57.50 last 89.81) - open/close 5491 (time 1710102324.68 total 67.50 last 90.24) - open/close 6255 (time 1710102334.70 total 77.53 last 76.22) - open/close 7033 (time 1710102344.72 total 87.55 last 77.64) - open/close 7796 (time 1710102354.76 total 97.58 last 76.06) - open/close 8505 (time 1710102364.76 total 107.58 last 70.88) - open/close 9208 (time 1710102374.77 total 117.59 last 70.25) - open/close 9890 (time 1710102384.78 total 127.60 last 68.11) total: 10000 open/close in 129.19 seconds: 77.41 ops/second - unlinked 0 (time 1710102388 ; total 0 ; last 0) total: 10000 unlinks in 79 seconds: 126.582275 unlinks/second - open/close 1186 (time 1710102478.93 total 10.00 last 118.59) - open/close 2276 (time 1710102488.95 total 20.02 last 108.84) - open/close 3256 (time 1710102498.95 total 30.02 last 97.97) - open/close 4226 (time 1710102508.95 total 40.02 last 96.98) - open/close 5117 (time 1710102518.96 total 50.03 last 89.05) - open/close 5950 (time 1710102528.96 total 60.03 last 83.29) - open/close 6784 (time 1710102538.97 total 70.04 last 83.34) - open/close 7588 (time 1710102548.97 total 80.04 last 80.36) - open/close 8268 (time 1710102559.03 total 90.10 last 67.59) - open/close 8999 (time 1710102569.04 total 100.11 last 73.05) - open/close 9689 (time 1710102579.04 total 110.11 last 68.99) total: 10000 open/close in 114.89 seconds: 87.04 ops/second - unlinked 0 (time 1710102585 ; total 0 ; last 0) total: 10000 unlinks in 78 seconds: 128.205124 unlinks/second - open/close 1185 (time 1710102676.01 total 10.00 last 118.44) - open/close 2256 (time 1710102686.01 total 20.01 last 107.09) - open/close 3324 (time 1710102696.02 total 30.02 last 106.69) - open/close 4287 (time 1710102706.03 total 40.02 last 96.22) - open/close 5123 (time 1710102716.03 total 50.03 last 83.54) - open/close 5962 (time 1710102726.04 total 60.04 last 83.81) - open/close 6324 (time 1710102741.48 total 75.48 last 23.45) - open/close 7111 (time 1710102751.49 total 85.49 last 78.59) - open/close 7877 (time 1710102761.50 total 95.50 last 76.50) - open/close 8696 (time 1710102771.51 total 105.51 last 81.86) - open/close 9453 (time 1710102781.52 total 115.52 last 75.60) total: 10000 open/close in 122.74 seconds: 81.47 ops/second - unlinked 0 (time 1710102790 ; total 0 ; last 0) total: 10000 unlinks in 70 seconds: 142.857147 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg247-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg247-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg247-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.202.147@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.202.147@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg247-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg247-server: oleg247-server.virtnet: executing set_default_debug -1 all pdsh@oleg247-client: oleg247-server: ssh exited with exit code 1 Started lustre-OST0000 oleg247-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg247-server: oleg247-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg247-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 4 sec mount lustre on /mnt/lustre..... Starting client: oleg247-client.virtnet: -o user_xattr,flock oleg247-server@tcp:/lustre /mnt/lustre On OST0, 7277 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg247-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg247-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg247-server stop mds service on oleg247-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg247-server unloading modules on: 'oleg247-server' oleg247-server: oleg247-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg247-client: oleg247-client: ssh exited with exit code 2 pdsh@oleg247-client: oleg247-server: ssh exited with exit code 2 pdsh@oleg247-client: oleg247-client: ssh exited with exit code 2 pdsh@oleg247-client: oleg247-server: ssh exited with exit code 2 PASS 69 (2134s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg247-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg247-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg247-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg247-client: oleg247-server: ssh exited with exit code 2 oleg247-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg247-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == conf-sanity test 93: register mulitple MDT at the same time ========================================================== 16:35:02 (1710102902) SKIP: conf-sanity test_93 needs >= 3 MDTs SKIP 93 (1s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg247-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg247-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory Stopping clients: oleg247-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg247-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg247-client: oleg247-server: ssh exited with exit code 2 oleg247-server: oleg247-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg247-server' oleg247-server: oleg247-server.virtnet: executing load_modules_local oleg247-server: Loading modules from /home/green/git/lustre-release/lustre oleg247-server: detected 4 online CPUs by sysfs oleg247-server: Force libcfs to create 2 CPU partitions oleg247-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory oleg247-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg247-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 2977 sec ========== 16:35:28 (1710102928) === conf-sanity: start cleanup 16:35:29 (1710102929) === === conf-sanity: finish cleanup 16:35:29 (1710102929) ===