-----============= acceptance-small: conf-sanity ============----- Sun Mar 10 15:46:09 EDT 2024 excepting tests: 32newtarball 110 skipping tests SLOW=no: 45 69 106 111 114 Stopping clients: oleg210-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg210-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg210-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg210-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg210-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg210-server oleg210-server: oleg210-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg210-server' oleg210-server: oleg210-server.virtnet: executing load_modules_local oleg210-server: Loading modules from /home/green/git/lustre-release/lustre oleg210-server: detected 4 online CPUs by sysfs oleg210-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg210-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg210-server: oleg210-server.virtnet: executing set_default_debug -1 all pdsh@oleg210-client: oleg210-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg210-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg210-server: oleg210-server.virtnet: executing set_default_debug -1 all pdsh@oleg210-client: oleg210-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg210-server: oleg210-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg210-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg210-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg210-server stop mds service on oleg210-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg210-server debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 15:47:11 (1710100031) start mds service on oleg210-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg210-server: oleg210-server.virtnet: executing set_default_debug -1 all pdsh@oleg210-client: oleg210-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg210-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg210-server: oleg210-server.virtnet: executing set_default_debug -1 all pdsh@oleg210-client: oleg210-server: ssh exited with exit code 1 Started lustre-OST0000 oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg210-server: oleg210-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg210-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 4 sec stop ost1 service on oleg210-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg210-server stop mds service on oleg210-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg210-server PASS 21a (21s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 15:47:34 (1710100054) start ost1 service on oleg210-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg210-server: oleg210-server.virtnet: executing set_default_debug -1 all pdsh@oleg210-client: oleg210-server: ssh exited with exit code 1 Started lustre-OST0000 oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid start mds service on oleg210-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg210-server: oleg210-server.virtnet: executing set_default_debug -1 all pdsh@oleg210-client: oleg210-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg210-server: oleg210-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg210-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop mds service on oleg210-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg210-server stop ost1 service on oleg210-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg210-server PASS 21b (197s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 15:50:52 (1710100252) start mds service on oleg210-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg210-server: oleg210-server.virtnet: executing set_default_debug -1 all pdsh@oleg210-client: oleg210-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid Client mount with ost in logs, but none running start ost1 service on oleg210-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg210-server: oleg210-server.virtnet: executing set_default_debug -1 all pdsh@oleg210-client: oleg210-server: ssh exited with exit code 1 Started lustre-OST0000 oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg210-server: oleg210-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg210-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 4 sec stop ost1 service on oleg210-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg210-server mount lustre on /mnt/lustre..... Starting client: oleg210-client.virtnet: -o user_xattr,flock oleg210-server@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg210-client.virtnet /mnt/lustre (opts:-f) PASS Client mount with a running ost start ost1 service on oleg210-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg210-server: oleg210-server.virtnet: executing set_default_debug -1 all pdsh@oleg210-client: oleg210-server: ssh exited with exit code 1 Started lustre-OST0000 oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg210-client.virtnet: -o user_xattr,flock oleg210-server@tcp:/lustre /mnt/lustre oleg210-server: oleg210-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg210-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec oleg210-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8800aabc3000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800aabc3000.ost_server_uuid in FULL state after 0 sec setup single mount lustre success PASS umount lustre on /mnt/lustre..... Stopping client oleg210-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg210-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg210-server stop mds service on oleg210-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg210-server unloading modules on: 'oleg210-server' oleg210-server: oleg210-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg210-client: oleg210-client: ssh exited with exit code 2 pdsh@oleg210-client: oleg210-server: ssh exited with exit code 2 pdsh@oleg210-client: oleg210-client: ssh exited with exit code 2 pdsh@oleg210-client: oleg210-server: ssh exited with exit code 2 PASS 22 (51s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg210-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg210-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg210-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg210-client: oleg210-server: ssh exited with exit code 2 oleg210-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg210-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == conf-sanity test 46a: handle ost additional - wide striped file ========================================================== 15:51:44 (1710100304) Testing with 2 OSTs Stopping clients: oleg210-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg210-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg210-client: oleg210-server: ssh exited with exit code 2 oleg210-server: oleg210-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg210-server' oleg210-server: oleg210-server.virtnet: executing load_modules_local oleg210-server: Loading modules from /home/green/git/lustre-release/lustre oleg210-server: detected 4 online CPUs by sysfs oleg210-server: Force libcfs to create 2 CPU partitions oleg210-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory oleg210-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg210-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg210-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg210-server: oleg210-server.virtnet: executing set_default_debug -1 all pdsh@oleg210-client: oleg210-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg210-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg210-server: oleg210-server.virtnet: executing set_default_debug -1 all pdsh@oleg210-client: oleg210-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg210-server: oleg210-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg210-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg210-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg210-server stop mds service on oleg210-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg210-server start mds service on oleg210-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg210-server: oleg210-server.virtnet: executing set_default_debug -1 all pdsh@oleg210-client: oleg210-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg210-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg210-server: oleg210-server.virtnet: executing set_default_debug -1 all pdsh@oleg210-client: oleg210-server: ssh exited with exit code 1 Started lustre-OST0000 oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg210-server: oleg210-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg210-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec mount lustre on /mnt/lustre..... Starting client: oleg210-client.virtnet: -o user_xattr,flock oleg210-server@tcp:/lustre /mnt/lustre Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg210-server: oleg210-server.virtnet: executing set_default_debug -1 all pdsh@oleg210-client: oleg210-server: ssh exited with exit code 1 Commit the device label on lustre-ost2/ost2 Started lustre-OST0001 oleg210-server: oleg210-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 oleg210-server: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg210-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff88012e549800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff88012e549800.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre2..... Starting client: oleg210-client.virtnet: -o user_xattr,flock oleg210-server@tcp:/lustre /mnt/lustre2 /mnt/lustre2 stripe_count: -1 stripe_size: 4194304 pattern: raid0 stripe_offset: -1 /mnt/lustre2/widestripe lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 34 0x22 0x240000400 1 2 0x2 0x280000400 File: '/mnt/lustre/widestripe' Size: 3 Blocks: 2 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115238826934273 Links: 1 Access: (0674/-rw-rwxr--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-03-10 15:52:44.000000000 -0400 Modify: 2024-03-10 15:52:44.000000000 -0400 Change: 2024-03-10 15:52:45.000000000 -0400 Birth: - umount lustre on /mnt/lustre2..... Stopping client oleg210-client.virtnet /mnt/lustre2 (opts:) umount lustre on /mnt/lustre..... Stopping client oleg210-client.virtnet /mnt/lustre (opts:) Stopping /mnt/lustre-ost2 (opts:-f) on oleg210-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg210-server stop mds service on oleg210-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg210-server stop ost1 service on oleg210-server stop mds service on oleg210-server unloading modules on: 'oleg210-server' oleg210-server: oleg210-server.virtnet: executing unload_modules_local modules unloaded. oleg210-server: oleg210-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre oleg210-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg210-client: oleg210-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg210-server: oleg210-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre oleg210-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg210-client: oleg210-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg210-server: oleg210-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre oleg210-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg210-client: oleg210-server: ssh exited with exit code 19 checking for existing Lustre data: not found tunefs failed, reformatting instead Stopping clients: oleg210-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg210-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg210-client: oleg210-server: ssh exited with exit code 2 oleg210-server: oleg210-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg210-server' oleg210-server: oleg210-server.virtnet: executing load_modules_local oleg210-server: Loading modules from /home/green/git/lustre-release/lustre oleg210-server: detected 4 online CPUs by sysfs oleg210-server: Force libcfs to create 2 CPU partitions oleg210-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg210-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg210-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg210-server: oleg210-server.virtnet: executing set_default_debug -1 all pdsh@oleg210-client: oleg210-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg210-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg210-server: oleg210-server.virtnet: executing set_default_debug -1 all pdsh@oleg210-client: oleg210-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg210-server: oleg210-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg210-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg210-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg210-server stop mds service on oleg210-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg210-server PASS 46a (137s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 50b: lazystatfs all servers down ===== 15:54:03 (1710100443) start mds service on oleg210-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg210-server: oleg210-server.virtnet: executing set_default_debug -1 all pdsh@oleg210-client: oleg210-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg210-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg210-server: oleg210-server.virtnet: executing set_default_debug -1 all pdsh@oleg210-client: oleg210-server: ssh exited with exit code 1 Started lustre-OST0000 oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg210-client.virtnet: -o user_xattr,flock oleg210-server@tcp:/lustre /mnt/lustre llite.lustre-ffff8800a9d37800.lazystatfs=1 stop ost1 service on oleg210-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg210-server oleg210-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff8800a9d37800.ost_server_uuid 50 0 UP mgc MGC192.168.202.110@tcp 684347ed-9623-4391-8e5f-b2d81bf847cf 5 1 UP lov lustre-clilov-ffff8800a9d37800 f77b7584-2276-4c3a-bb7a-e42c7135b823 4 2 UP lmv lustre-clilmv-ffff8800a9d37800 f77b7584-2276-4c3a-bb7a-e42c7135b823 5 3 UP mdc lustre-MDT0000-mdc-ffff8800a9d37800 f77b7584-2276-4c3a-bb7a-e42c7135b823 5 4 UP osc lustre-OST0000-osc-ffff8800a9d37800 f77b7584-2276-4c3a-bb7a-e42c7135b823 5 OSCs should all be DISCONN multiop /mnt/lustre vf_ TMPPIPE=/tmp/multiop_open_wait_pipe.6733 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 383616 3200 378368 1% /mnt/lustre[MDT:0] filesystem_summary: 383616 3200 378368 1% /mnt/lustre /home/green/git/lustre-release/lustre/tests/conf-sanity.sh: line 4282: kill: (27302) - No such process umount lustre on /mnt/lustre..... Stopping client oleg210-client.virtnet /mnt/lustre (opts:) stop mds service on oleg210-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg210-server PASS 50b (45s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 50c: lazystatfs one server down ====== 15:54:50 (1710100490) start mds service on oleg210-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg210-server: oleg210-server.virtnet: executing set_default_debug -1 all pdsh@oleg210-client: oleg210-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg210-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg210-server: oleg210-server.virtnet: executing set_default_debug -1 all pdsh@oleg210-client: oleg210-server: ssh exited with exit code 1 Started lustre-OST0000 oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid start ost2 service on oleg210-server Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg210-server: oleg210-server.virtnet: executing set_default_debug -1 all pdsh@oleg210-client: oleg210-server: ssh exited with exit code 1 Commit the device label on lustre-ost2/ost2 Started lustre-OST0001 oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg210-client.virtnet: -o user_xattr,flock oleg210-server@tcp:/lustre /mnt/lustre llite.lustre-ffff8800aabc5000.lazystatfs=1 stop ost1 service on oleg210-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg210-server oleg210-server: oleg210-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg210-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec multiop /mnt/lustre vf_ TMPPIPE=/tmp/multiop_open_wait_pipe.6733 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 383616 3328 378240 1% /mnt/lustre[MDT:0] lustre-OST0001_UUID 382976 3072 377856 1% /mnt/lustre[OST:1] filesystem_summary: 382976 3072 377856 1% /mnt/lustre /home/green/git/lustre-release/lustre/tests/conf-sanity.sh: line 4282: kill: (29537) - No such process umount lustre on /mnt/lustre..... Stopping client oleg210-client.virtnet /mnt/lustre (opts:) stop ost2 service on oleg210-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg210-server stop mds service on oleg210-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg210-server oleg210-server: oleg210-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre oleg210-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg210-client: oleg210-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg210-server: oleg210-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre oleg210-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg210-client: oleg210-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg210-server: oleg210-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre oleg210-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg210-client: oleg210-server: ssh exited with exit code 19 checking for existing Lustre data: not found tunefs failed, reformatting instead Stopping clients: oleg210-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg210-client.virtnet /mnt/lustre2 (opts:-f) oleg210-server: oleg210-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg210-server' oleg210-server: oleg210-server.virtnet: executing load_modules_local oleg210-server: Loading modules from /home/green/git/lustre-release/lustre oleg210-server: detected 4 online CPUs by sysfs oleg210-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg210-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg210-server: oleg210-server.virtnet: executing set_default_debug -1 all pdsh@oleg210-client: oleg210-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg210-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg210-server: oleg210-server.virtnet: executing set_default_debug -1 all pdsh@oleg210-client: oleg210-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg210-server: oleg210-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg210-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg210-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg210-server stop mds service on oleg210-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg210-server PASS 50c (82s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 50e: normal statfs all servers down == 15:56:14 (1710100574) Stopping clients: oleg210-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg210-client.virtnet /mnt/lustre2 (opts:-f) oleg210-server: oleg210-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg210-server' oleg210-server: oleg210-server.virtnet: executing load_modules_local oleg210-server: Loading modules from /home/green/git/lustre-release/lustre oleg210-server: detected 4 online CPUs by sysfs oleg210-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg210-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg210-server: oleg210-server.virtnet: executing set_default_debug -1 all pdsh@oleg210-client: oleg210-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg210-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg210-server: oleg210-server.virtnet: executing set_default_debug -1 all pdsh@oleg210-client: oleg210-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg210-server: oleg210-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg210-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec stop ost1 service on oleg210-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg210-server stop mds service on oleg210-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg210-server start mds service on oleg210-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg210-server: oleg210-server.virtnet: executing set_default_debug -1 all pdsh@oleg210-client: oleg210-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg210-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg210-server: oleg210-server.virtnet: executing set_default_debug -1 all pdsh@oleg210-client: oleg210-server: ssh exited with exit code 1 Started lustre-OST0000 oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg210-server: oleg210-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg210-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg210-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg210-server oleg210-server: oleg210-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg210-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg210-client.virtnet: -o user_xattr,flock oleg210-server@tcp:/lustre /mnt/lustre llite.lustre-ffff8800aabc5800.lazystatfs=0 multiop /mnt/lustre v_f TMPPIPE=/tmp/multiop_open_wait_pipe.6733 start ost1 service on oleg210-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg210-server: oleg210-server.virtnet: executing set_default_debug -1 all pdsh@oleg210-client: oleg210-server: ssh exited with exit code 1 Started lustre-OST0000 oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec umount lustre on /mnt/lustre..... Stopping client oleg210-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg210-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg210-server stop mds service on oleg210-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg210-server PASS 50e (98s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 50f: normal statfs one server in down ========================================================== 15:57:54 (1710100674) start mds service on oleg210-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg210-server: oleg210-server.virtnet: executing set_default_debug -1 all pdsh@oleg210-client: oleg210-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg210-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg210-server: oleg210-server.virtnet: executing set_default_debug -1 all pdsh@oleg210-client: oleg210-server: ssh exited with exit code 1 Started lustre-OST0000 oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg210-server: oleg210-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg210-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec start ost2 service on oleg210-server Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg210-server: oleg210-server.virtnet: executing set_default_debug -1 all pdsh@oleg210-client: oleg210-server: ssh exited with exit code 1 Commit the device label on lustre-ost2/ost2 Started lustre-OST0001 oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid oleg210-server: oleg210-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 oleg210-server: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 3 sec stop ost2 service on oleg210-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg210-server oleg210-server: oleg210-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 oleg210-server: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg210-client.virtnet: -o user_xattr,flock oleg210-server@tcp:/lustre /mnt/lustre llite.lustre-ffff8800ac593000.lazystatfs=0 multiop /mnt/lustre v_f TMPPIPE=/tmp/multiop_open_wait_pipe.6733 start ost2 service on oleg210-server Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg210-server: oleg210-server.virtnet: executing set_default_debug -1 all pdsh@oleg210-client: oleg210-server: ssh exited with exit code 1 Started lustre-OST0001 oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec stop ost2 service on oleg210-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg210-server umount lustre on /mnt/lustre..... Stopping client oleg210-client.virtnet /mnt/lustre (opts:-f) stop ost1 service on oleg210-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg210-server stop mds service on oleg210-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg210-server oleg210-server: oleg210-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre oleg210-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg210-client: oleg210-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg210-server: oleg210-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre oleg210-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg210-client: oleg210-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg210-server: oleg210-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre oleg210-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg210-client: oleg210-server: ssh exited with exit code 19 checking for existing Lustre data: not found tunefs failed, reformatting instead Stopping clients: oleg210-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg210-client.virtnet /mnt/lustre2 (opts:-f) oleg210-server: oleg210-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory loading modules on: 'oleg210-server' oleg210-server: oleg210-server.virtnet: executing load_modules_local oleg210-server: Loading modules from /home/green/git/lustre-release/lustre oleg210-server: detected 4 online CPUs by sysfs oleg210-server: Force libcfs to create 2 CPU partitions oleg210-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg210-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg210-server: oleg210-server.virtnet: executing set_default_debug -1 all pdsh@oleg210-client: oleg210-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg210-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg210-server: oleg210-server.virtnet: executing set_default_debug -1 all pdsh@oleg210-client: oleg210-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg210-server: oleg210-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg210-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg210-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg210-server stop mds service on oleg210-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg210-server PASS 50f (129s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 69: replace an OST with the same index ========================================================== 16:00:06 (1710100806) start mds service on oleg210-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg210-server: oleg210-server.virtnet: executing set_default_debug -1 all pdsh@oleg210-client: oleg210-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg210-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg210-server: oleg210-server.virtnet: executing set_default_debug -1 all pdsh@oleg210-client: oleg210-server: ssh exited with exit code 1 Started lustre-OST0000 oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg210-client.virtnet: -o user_xattr,flock oleg210-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11872 inodes available. Want 99936. rc=0 - open/close 1355 (time 1710100833.74 total 10.01 last 135.35) - open/close 2783 (time 1710100843.74 total 20.01 last 142.78) - open/close 4033 (time 1710100853.75 total 30.02 last 124.96) - open/close 5094 (time 1710100863.75 total 40.02 last 106.05) - open/close 6149 (time 1710100873.76 total 50.03 last 105.44) - open/close 7148 (time 1710100883.77 total 60.04 last 99.79) - open/close 8103 (time 1710100893.77 total 70.04 last 95.50) - open/close 9006 (time 1710100903.78 total 80.04 last 90.23) - open/close 9831 (time 1710100913.79 total 90.06 last 82.40) total: 10000 open/close in 93.03 seconds: 107.49 ops/second - unlinked 0 (time 1710100918 ; total 0 ; last 0) total: 10000 unlinks in 72 seconds: 138.888885 unlinks/second - open/close 1437 (time 1710101001.79 total 10.00 last 143.67) - open/close 2602 (time 1710101011.80 total 20.01 last 116.40) - open/close 3573 (time 1710101021.81 total 30.02 last 97.01) - open/close 4664 (time 1710101031.81 total 40.02 last 109.09) - open/close 5702 (time 1710101041.82 total 50.03 last 103.71) - open/close 6570 (time 1710101051.83 total 60.04 last 86.73) - open/close 7393 (time 1710101061.84 total 70.05 last 82.22) - open/close 8106 (time 1710101071.84 total 80.05 last 71.26) - open/close 8746 (time 1710101081.85 total 90.05 last 63.99) - open/close 9400 (time 1710101091.88 total 100.08 last 65.20) total: 10000 open/close in 109.31 seconds: 91.48 ops/second - unlinked 0 (time 1710101103 ; total 0 ; last 0) total: 10000 unlinks in 85 seconds: 117.647057 unlinks/second - open/close 959 (time 1710101200.89 total 10.00 last 95.89) - open/close 1899 (time 1710101210.90 total 20.01 last 93.93) - open/close 2769 (time 1710101220.91 total 30.02 last 86.90) - open/close 3523 (time 1710101230.91 total 40.02 last 75.40) - open/close 4270 (time 1710101240.91 total 50.02 last 74.69) - open/close 5025 (time 1710101250.92 total 60.03 last 75.44) - open/close 5732 (time 1710101260.93 total 70.04 last 70.63) - open/close 6374 (time 1710101270.96 total 80.08 last 63.97) - open/close 7025 (time 1710101280.98 total 90.09 last 65.01) - open/close 7634 (time 1710101291.06 total 100.18 last 60.38) - open/close 8240 (time 1710101301.07 total 110.19 last 60.54) - open/close 8828 (time 1710101311.08 total 120.20 last 58.74) - open/close 9393 (time 1710101321.11 total 130.22 last 56.34) total: 10000 open/close in 140.09 seconds: 71.38 ops/second - unlinked 0 (time 1710101333 ; total 0 ; last 0) total: 10000 unlinks in 87 seconds: 114.942528 unlinks/second - open/close 1043 (time 1710101432.37 total 10.00 last 104.27) - open/close 2023 (time 1710101442.38 total 20.01 last 97.91) - open/close 2903 (time 1710101452.38 total 30.02 last 87.96) - open/close 3683 (time 1710101462.39 total 40.02 last 77.94) - open/close 4502 (time 1710101472.40 total 50.03 last 81.83) - open/close 5221 (time 1710101482.42 total 60.05 last 71.76) - open/close 5910 (time 1710101492.42 total 70.06 last 68.86) - open/close 6554 (time 1710101502.44 total 80.07 last 64.32) - open/close 7195 (time 1710101512.44 total 90.08 last 64.06) - open/close 7800 (time 1710101522.48 total 100.12 last 60.27) - open/close 8415 (time 1710101532.48 total 110.12 last 61.48) - open/close 8989 (time 1710101542.49 total 120.12 last 57.38) - open/close 9586 (time 1710101552.50 total 130.13 last 59.63) total: 10000 open/close in 137.51 seconds: 72.72 ops/second - unlinked 0 (time 1710101562 ; total 0 ; last 0) total: 10000 unlinks in 86 seconds: 116.279068 unlinks/second - open/close 1117 (time 1710101660.50 total 10.04 last 111.21) - open/close 2096 (time 1710101670.50 total 20.05 last 97.86) - open/close 2996 (time 1710101680.51 total 30.05 last 89.95) - open/close 3845 (time 1710101690.53 total 40.08 last 84.71) - open/close 4725 (time 1710101700.53 total 50.08 last 87.97) - open/close 5512 (time 1710101710.53 total 60.08 last 78.70) - open/close 6292 (time 1710101720.54 total 70.09 last 77.95) - open/close 7016 (time 1710101730.55 total 80.10 last 72.32) - open/close 7705 (time 1710101740.57 total 90.11 last 68.79) - open/close 8300 (time 1710101750.58 total 100.12 last 59.45) - open/close 8940 (time 1710101760.59 total 110.13 last 63.94) - open/close 9543 (time 1710101770.59 total 120.14 last 60.26) total: 10000 open/close in 127.34 seconds: 78.53 ops/second - unlinked 0 (time 1710101779 ; total 0 ; last 0) total: 10000 unlinks in 83 seconds: 120.481926 unlinks/second - open/close 1017 (time 1710101874.48 total 10.01 last 101.57) - open/close 2005 (time 1710101884.50 total 20.03 last 98.63) - open/close 2914 (time 1710101894.51 total 30.04 last 90.81) - open/close 3835 (time 1710101904.52 total 40.05 last 92.03) - open/close 4630 (time 1710101914.52 total 50.05 last 79.49) - open/close 5403 (time 1710101924.52 total 60.05 last 77.28) - open/close 6100 (time 1710101934.54 total 70.07 last 69.58) - open/close 6722 (time 1710101944.54 total 80.07 last 62.18) - open/close 7339 (time 1710101954.54 total 90.07 last 61.68) - open/close 7949 (time 1710101964.55 total 100.08 last 60.95) - open/close 8598 (time 1710101974.56 total 110.09 last 64.87) - open/close 9212 (time 1710101984.57 total 120.10 last 61.33) - open/close 9861 (time 1710101994.58 total 130.11 last 64.80) total: 10000 open/close in 132.48 seconds: 75.48 ops/second - unlinked 0 (time 1710101998 ; total 0 ; last 0) total: 10000 unlinks in 96 seconds: 104.166664 unlinks/second - open/close 1083 (time 1710102106.12 total 10.01 last 108.19) - open/close 2040 (time 1710102116.13 total 20.02 last 95.60) - open/close 2928 (time 1710102126.13 total 30.02 last 88.79) - open/close 3834 (time 1710102136.14 total 40.03 last 90.56) - open/close 4627 (time 1710102146.15 total 50.03 last 79.23) - open/close 5449 (time 1710102156.15 total 60.04 last 82.14) - open/close 6137 (time 1710102166.15 total 70.04 last 68.79) - open/close 6874 (time 1710102176.17 total 80.06 last 73.60) - open/close 7560 (time 1710102186.17 total 90.06 last 68.60) - open/close 8190 (time 1710102196.18 total 100.06 last 62.96) - open/close 8834 (time 1710102206.19 total 110.08 last 64.31) - open/close 9422 (time 1710102216.20 total 120.09 last 58.75) total: 10000 open/close in 129.89 seconds: 76.99 ops/second - unlinked 0 (time 1710102227 ; total 0 ; last 0) total: 10000 unlinks in 83 seconds: 120.481926 unlinks/second - open/close 1155 (time 1710102322.55 total 10.00 last 115.48) - open/close 2114 (time 1710102332.55 total 20.01 last 95.86) - open/close 3065 (time 1710102342.56 total 30.01 last 95.07) - open/close 4016 (time 1710102352.56 total 40.01 last 95.06) - open/close 4842 (time 1710102362.57 total 50.02 last 82.51) - open/close 5663 (time 1710102372.58 total 60.03 last 82.03) - open/close 6478 (time 1710102382.59 total 70.04 last 81.43) - open/close 7165 (time 1710102392.59 total 80.04 last 68.69) - open/close 7811 (time 1710102402.60 total 90.05 last 64.55) - open/close 8496 (time 1710102412.62 total 100.08 last 68.32) - open/close 9118 (time 1710102422.63 total 110.08 last 62.20) - open/close 9784 (time 1710102432.63 total 120.08 last 66.57) total: 10000 open/close in 123.62 seconds: 80.90 ops/second - unlinked 0 (time 1710102437 ; total 0 ; last 0) total: 10000 unlinks in 79 seconds: 126.582275 unlinks/second - open/close 1058 (time 1710102528.22 total 10.00 last 105.76) - open/close 2052 (time 1710102538.23 total 20.01 last 99.34) - open/close 3061 (time 1710102548.23 total 30.01 last 100.88) - open/close 3965 (time 1710102558.24 total 40.02 last 90.32) - open/close 4773 (time 1710102568.24 total 50.03 last 80.75) - open/close 5560 (time 1710102578.25 total 60.04 last 78.61) - open/close 6264 (time 1710102588.27 total 70.06 last 70.27) - open/close 6934 (time 1710102598.28 total 80.06 last 66.94) - open/close 7657 (time 1710102608.29 total 90.07 last 72.26) - open/close 8323 (time 1710102618.34 total 100.12 last 66.26) - open/close 8973 (time 1710102628.34 total 110.12 last 65.00) - open/close 9570 (time 1710102638.35 total 120.13 last 59.63) total: 10000 open/close in 126.61 seconds: 78.98 ops/second - unlinked 0 (time 1710102646 ; total 0 ; last 0) total: 10000 unlinks in 102 seconds: 98.039215 unlinks/second - open/close 1216 (time 1710102760.61 total 10.01 last 121.53) - open/close 2367 (time 1710102770.61 total 20.01 last 115.08) - open/close 3466 (time 1710102780.61 total 30.01 last 109.90) - open/close 4468 (time 1710102790.62 total 40.02 last 100.12) - open/close 5355 (time 1710102800.63 total 50.02 last 88.63) - open/close 6280 (time 1710102810.63 total 60.03 last 92.47) - open/close 7201 (time 1710102820.64 total 70.03 last 92.03) - open/close 8074 (time 1710102830.64 total 80.04 last 87.25) - open/close 8914 (time 1710102840.65 total 90.05 last 83.94) - open/close 9521 (time 1710102858.73 total 108.12 last 33.58) total: 10000 open/close in 113.97 seconds: 87.74 ops/second - unlinked 0 (time 1710102866 ; total 0 ; last 0) total: 10000 unlinks in 59 seconds: 169.491531 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg210-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg210-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg210-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.202.110@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.202.110@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg210-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg210-server: oleg210-server.virtnet: executing set_default_debug -1 all pdsh@oleg210-client: oleg210-server: ssh exited with exit code 1 Started lustre-OST0000 oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg210-server: oleg210-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg210-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 14 sec mount lustre on /mnt/lustre..... Starting client: oleg210-client.virtnet: -o user_xattr,flock oleg210-server@tcp:/lustre /mnt/lustre On OST0, 14877 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg210-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg210-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg210-server stop mds service on oleg210-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg210-server unloading modules on: 'oleg210-server' oleg210-server: oleg210-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg210-client: oleg210-client: ssh exited with exit code 2 pdsh@oleg210-client: oleg210-server: ssh exited with exit code 2 pdsh@oleg210-client: oleg210-client: ssh exited with exit code 2 pdsh@oleg210-client: oleg210-server: ssh exited with exit code 2 PASS 69 (2169s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg210-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg210-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg210-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg210-client: oleg210-server: ssh exited with exit code 2 oleg210-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg210-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == conf-sanity test 93: register mulitple MDT at the same time ========================================================== 16:36:17 (1710102977) SKIP: conf-sanity test_93 needs >= 3 MDTs SKIP 93 (1s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg210-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg210-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory Stopping clients: oleg210-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg210-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg210-client: oleg210-server: ssh exited with exit code 2 oleg210-server: oleg210-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg210-server' oleg210-server: oleg210-server.virtnet: executing load_modules_local oleg210-server: Loading modules from /home/green/git/lustre-release/lustre oleg210-server: detected 4 online CPUs by sysfs oleg210-server: Force libcfs to create 2 CPU partitions oleg210-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg210-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 3029 sec ========== 16:36:39 (1710102999) === conf-sanity: start cleanup 16:36:39 (1710102999) === === conf-sanity: finish cleanup 16:36:39 (1710102999) ===