-----============= acceptance-small: conf-sanity ============----- Sun Mar 10 15:46:51 EDT 2024 excepting tests: 32newtarball 110 skipping tests SLOW=no: 45 69 106 111 114 Stopping clients: oleg108-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg108-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg108-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg108-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg108-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg108-server oleg108-server: oleg108-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg108-server' oleg108-server: oleg108-server.virtnet: executing load_modules_local oleg108-server: Loading modules from /home/green/git/lustre-release/lustre oleg108-server: detected 4 online CPUs by sysfs oleg108-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg108-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg108-server: oleg108-server.virtnet: executing set_default_debug -1 all pdsh@oleg108-client: oleg108-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg108-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg108-server: oleg108-server.virtnet: executing set_default_debug -1 all pdsh@oleg108-client: oleg108-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg108-server: oleg108-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg108-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg108-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg108-server stop mds service on oleg108-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg108-server debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 15:47:43 (1710100063) start mds service on oleg108-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg108-server: oleg108-server.virtnet: executing set_default_debug -1 all pdsh@oleg108-client: oleg108-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg108-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg108-server: oleg108-server.virtnet: executing set_default_debug -1 all pdsh@oleg108-client: oleg108-server: ssh exited with exit code 1 Started lustre-OST0000 oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg108-server: oleg108-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg108-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 4 sec stop ost1 service on oleg108-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg108-server stop mds service on oleg108-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg108-server PASS 21a (21s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 15:48:06 (1710100086) start ost1 service on oleg108-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg108-server: oleg108-server.virtnet: executing set_default_debug -1 all pdsh@oleg108-client: oleg108-server: ssh exited with exit code 1 Started lustre-OST0000 oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid start mds service on oleg108-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg108-server: oleg108-server.virtnet: executing set_default_debug -1 all pdsh@oleg108-client: oleg108-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg108-server: oleg108-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg108-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop mds service on oleg108-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg108-server stop ost1 service on oleg108-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg108-server PASS 21b (196s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 15:51:24 (1710100284) start mds service on oleg108-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg108-server: oleg108-server.virtnet: executing set_default_debug -1 all pdsh@oleg108-client: oleg108-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid Client mount with ost in logs, but none running start ost1 service on oleg108-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg108-server: oleg108-server.virtnet: executing set_default_debug -1 all pdsh@oleg108-client: oleg108-server: ssh exited with exit code 1 Started lustre-OST0000 oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg108-server: oleg108-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg108-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 4 sec stop ost1 service on oleg108-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg108-server mount lustre on /mnt/lustre..... Starting client: oleg108-client.virtnet: -o user_xattr,flock oleg108-server@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg108-client.virtnet /mnt/lustre (opts:-f) PASS Client mount with a running ost start ost1 service on oleg108-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg108-server: oleg108-server.virtnet: executing set_default_debug -1 all pdsh@oleg108-client: oleg108-server: ssh exited with exit code 1 Started lustre-OST0000 oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg108-client.virtnet: -o user_xattr,flock oleg108-server@tcp:/lustre /mnt/lustre oleg108-server: oleg108-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg108-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec oleg108-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8800aecd2800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800aecd2800.ost_server_uuid in FULL state after 0 sec setup single mount lustre success PASS umount lustre on /mnt/lustre..... Stopping client oleg108-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg108-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg108-server stop mds service on oleg108-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg108-server unloading modules on: 'oleg108-server' oleg108-server: oleg108-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg108-client: oleg108-client: ssh exited with exit code 2 pdsh@oleg108-client: oleg108-server: ssh exited with exit code 2 pdsh@oleg108-client: oleg108-client: ssh exited with exit code 2 pdsh@oleg108-client: oleg108-server: ssh exited with exit code 2 PASS 22 (50s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg108-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg108-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg108-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg108-client: oleg108-server: ssh exited with exit code 2 oleg108-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg108-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == conf-sanity test 46a: handle ost additional - wide striped file ========================================================== 15:52:16 (1710100336) Testing with 2 OSTs Stopping clients: oleg108-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg108-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg108-client: oleg108-server: ssh exited with exit code 2 oleg108-server: oleg108-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg108-server' oleg108-server: oleg108-server.virtnet: executing load_modules_local oleg108-server: Loading modules from /home/green/git/lustre-release/lustre oleg108-server: detected 4 online CPUs by sysfs oleg108-server: Force libcfs to create 2 CPU partitions oleg108-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg108-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg108-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg108-server: oleg108-server.virtnet: executing set_default_debug -1 all pdsh@oleg108-client: oleg108-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg108-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg108-server: oleg108-server.virtnet: executing set_default_debug -1 all pdsh@oleg108-client: oleg108-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg108-server: oleg108-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg108-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg108-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg108-server stop mds service on oleg108-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg108-server start mds service on oleg108-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg108-server: oleg108-server.virtnet: executing set_default_debug -1 all pdsh@oleg108-client: oleg108-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg108-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg108-server: oleg108-server.virtnet: executing set_default_debug -1 all pdsh@oleg108-client: oleg108-server: ssh exited with exit code 1 Started lustre-OST0000 oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg108-server: oleg108-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg108-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg108-client.virtnet: -o user_xattr,flock oleg108-server@tcp:/lustre /mnt/lustre Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg108-server: oleg108-server.virtnet: executing set_default_debug -1 all pdsh@oleg108-client: oleg108-server: ssh exited with exit code 1 Commit the device label on lustre-ost2/ost2 Started lustre-OST0001 oleg108-server: oleg108-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 oleg108-server: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 2 sec oleg108-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8800a9c15000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800a9c15000.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre2..... Starting client: oleg108-client.virtnet: -o user_xattr,flock oleg108-server@tcp:/lustre /mnt/lustre2 /mnt/lustre2 stripe_count: -1 stripe_size: 4194304 pattern: raid0 stripe_offset: -1 /mnt/lustre2/widestripe lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 34 0x22 0x240000400 1 2 0x2 0x280000400 File: '/mnt/lustre/widestripe' Size: 3 Blocks: 2 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115238826934273 Links: 1 Access: (0674/-rw-rwxr--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-03-10 15:53:30.000000000 -0400 Modify: 2024-03-10 15:53:30.000000000 -0400 Change: 2024-03-10 15:53:31.000000000 -0400 Birth: - umount lustre on /mnt/lustre2..... Stopping client oleg108-client.virtnet /mnt/lustre2 (opts:) umount lustre on /mnt/lustre..... Stopping client oleg108-client.virtnet /mnt/lustre (opts:) Stopping /mnt/lustre-ost2 (opts:-f) on oleg108-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg108-server stop mds service on oleg108-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg108-server stop ost1 service on oleg108-server stop mds service on oleg108-server unloading modules on: 'oleg108-server' oleg108-server: oleg108-server.virtnet: executing unload_modules_local modules unloaded. oleg108-server: oleg108-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre oleg108-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg108-client: oleg108-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg108-server: oleg108-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre oleg108-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg108-client: oleg108-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg108-server: oleg108-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre oleg108-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg108-client: oleg108-server: ssh exited with exit code 19 checking for existing Lustre data: not found tunefs failed, reformatting instead Stopping clients: oleg108-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg108-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg108-client: oleg108-server: ssh exited with exit code 2 oleg108-server: oleg108-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg108-server' oleg108-server: oleg108-server.virtnet: executing load_modules_local oleg108-server: Loading modules from /home/green/git/lustre-release/lustre oleg108-server: detected 4 online CPUs by sysfs oleg108-server: Force libcfs to create 2 CPU partitions oleg108-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg108-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg108-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg108-server: oleg108-server.virtnet: executing set_default_debug -1 all pdsh@oleg108-client: oleg108-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg108-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg108-server: oleg108-server.virtnet: executing set_default_debug -1 all pdsh@oleg108-client: oleg108-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg108-server: oleg108-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg108-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg108-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg108-server stop mds service on oleg108-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg108-server PASS 46a (148s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 50b: lazystatfs all servers down ===== 15:54:46 (1710100486) start mds service on oleg108-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg108-server: oleg108-server.virtnet: executing set_default_debug -1 all pdsh@oleg108-client: oleg108-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg108-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg108-server: oleg108-server.virtnet: executing set_default_debug -1 all pdsh@oleg108-client: oleg108-server: ssh exited with exit code 1 Started lustre-OST0000 oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg108-client.virtnet: -o user_xattr,flock oleg108-server@tcp:/lustre /mnt/lustre llite.lustre-ffff8800b5b47000.lazystatfs=1 stop ost1 service on oleg108-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg108-server oleg108-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff8800b5b47000.ost_server_uuid 50 0 UP mgc MGC192.168.201.108@tcp 43e2cc4f-5eb3-4ac0-bfa7-a951359f0437 5 1 UP lov lustre-clilov-ffff8800b5b47000 9b11ba6a-df29-4352-9b26-c5ec3c8b875a 4 2 UP lmv lustre-clilmv-ffff8800b5b47000 9b11ba6a-df29-4352-9b26-c5ec3c8b875a 5 3 UP mdc lustre-MDT0000-mdc-ffff8800b5b47000 9b11ba6a-df29-4352-9b26-c5ec3c8b875a 5 4 UP osc lustre-OST0000-osc-ffff8800b5b47000 9b11ba6a-df29-4352-9b26-c5ec3c8b875a 5 OSCs should all be DISCONN multiop /mnt/lustre vf_ TMPPIPE=/tmp/multiop_open_wait_pipe.6736 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 383616 3200 378368 1% /mnt/lustre[MDT:0] filesystem_summary: 383616 3200 378368 1% /mnt/lustre /home/green/git/lustre-release/lustre/tests/conf-sanity.sh: line 4282: kill: (27303) - No such process umount lustre on /mnt/lustre..... Stopping client oleg108-client.virtnet /mnt/lustre (opts:) stop mds service on oleg108-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg108-server PASS 50b (47s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 50c: lazystatfs one server down ====== 15:55:35 (1710100535) start mds service on oleg108-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg108-server: oleg108-server.virtnet: executing set_default_debug -1 all pdsh@oleg108-client: oleg108-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg108-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg108-server: oleg108-server.virtnet: executing set_default_debug -1 all pdsh@oleg108-client: oleg108-server: ssh exited with exit code 1 Started lustre-OST0000 oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid start ost2 service on oleg108-server Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg108-server: oleg108-server.virtnet: executing set_default_debug -1 all pdsh@oleg108-client: oleg108-server: ssh exited with exit code 1 Commit the device label on lustre-ost2/ost2 Started lustre-OST0001 oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg108-client.virtnet: -o user_xattr,flock oleg108-server@tcp:/lustre /mnt/lustre llite.lustre-ffff8800b586c000.lazystatfs=1 stop ost1 service on oleg108-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg108-server oleg108-server: oleg108-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg108-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec multiop /mnt/lustre vf_ TMPPIPE=/tmp/multiop_open_wait_pipe.6736 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 383616 3328 378240 1% /mnt/lustre[MDT:0] lustre-OST0001_UUID 382976 3072 377856 1% /mnt/lustre[OST:1] filesystem_summary: 382976 3072 377856 1% /mnt/lustre /home/green/git/lustre-release/lustre/tests/conf-sanity.sh: line 4282: kill: (29531) - No such process umount lustre on /mnt/lustre..... Stopping client oleg108-client.virtnet /mnt/lustre (opts:) stop ost2 service on oleg108-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg108-server stop mds service on oleg108-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg108-server oleg108-server: oleg108-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre oleg108-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg108-client: oleg108-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg108-server: oleg108-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre oleg108-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg108-client: oleg108-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg108-server: oleg108-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre oleg108-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg108-client: oleg108-server: ssh exited with exit code 19 checking for existing Lustre data: not found tunefs failed, reformatting instead Stopping clients: oleg108-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg108-client.virtnet /mnt/lustre2 (opts:-f) oleg108-server: oleg108-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg108-server' oleg108-server: oleg108-server.virtnet: executing load_modules_local oleg108-server: Loading modules from /home/green/git/lustre-release/lustre oleg108-server: detected 4 online CPUs by sysfs oleg108-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg108-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg108-server: oleg108-server.virtnet: executing set_default_debug -1 all pdsh@oleg108-client: oleg108-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg108-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg108-server: oleg108-server.virtnet: executing set_default_debug -1 all pdsh@oleg108-client: oleg108-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg108-server: oleg108-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg108-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg108-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg108-server stop mds service on oleg108-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg108-server PASS 50c (76s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 50e: normal statfs all servers down == 15:56:53 (1710100613) Stopping clients: oleg108-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg108-client.virtnet /mnt/lustre2 (opts:-f) oleg108-server: oleg108-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory loading modules on: 'oleg108-server' oleg108-server: oleg108-server.virtnet: executing load_modules_local oleg108-server: Loading modules from /home/green/git/lustre-release/lustre oleg108-server: detected 4 online CPUs by sysfs oleg108-server: Force libcfs to create 2 CPU partitions oleg108-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg108-server: libkmod: kmod_module_get_refcnt: could not read integer from '/sys/module/pcc_cpufreq/refcnt': 'No such device' oleg108-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory oleg108-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory oleg108-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg108-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg108-server: oleg108-server.virtnet: executing set_default_debug -1 all pdsh@oleg108-client: oleg108-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg108-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg108-server: oleg108-server.virtnet: executing set_default_debug -1 all pdsh@oleg108-client: oleg108-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg108-server: oleg108-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg108-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg108-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg108-server stop mds service on oleg108-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg108-server start mds service on oleg108-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg108-server: oleg108-server.virtnet: executing set_default_debug -1 all pdsh@oleg108-client: oleg108-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg108-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg108-server: oleg108-server.virtnet: executing set_default_debug -1 all pdsh@oleg108-client: oleg108-server: ssh exited with exit code 1 Started lustre-OST0000 oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg108-server: oleg108-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg108-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 4 sec stop ost1 service on oleg108-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg108-server oleg108-server: oleg108-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 mount lustre on /mnt/lustre..... Starting client: oleg108-client.virtnet: -o user_xattr,flock oleg108-server@tcp:/lustre /mnt/lustre llite.lustre-ffff8800b6d14800.lazystatfs=0 multiop /mnt/lustre v_f TMPPIPE=/tmp/multiop_open_wait_pipe.6736 start ost1 service on oleg108-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg108-server: oleg108-server.virtnet: executing set_default_debug -1 all pdsh@oleg108-client: oleg108-server: ssh exited with exit code 1 Started lustre-OST0000 oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 1 sec umount lustre on /mnt/lustre..... Stopping client oleg108-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg108-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg108-server stop mds service on oleg108-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg108-server PASS 50e (131s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 50f: normal statfs one server in down ========================================================== 15:59:07 (1710100747) start mds service on oleg108-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg108-server: oleg108-server.virtnet: executing set_default_debug -1 all pdsh@oleg108-client: oleg108-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg108-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg108-server: oleg108-server.virtnet: executing set_default_debug -1 all pdsh@oleg108-client: oleg108-server: ssh exited with exit code 1 Started lustre-OST0000 oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg108-server: oleg108-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg108-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec start ost2 service on oleg108-server Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg108-server: oleg108-server.virtnet: executing set_default_debug -1 all pdsh@oleg108-client: oleg108-server: ssh exited with exit code 1 Commit the device label on lustre-ost2/ost2 Started lustre-OST0001 oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid oleg108-server: oleg108-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 oleg108-server: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost2 service on oleg108-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg108-server oleg108-server: oleg108-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 oleg108-server: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg108-client.virtnet: -o user_xattr,flock oleg108-server@tcp:/lustre /mnt/lustre llite.lustre-ffff88012a55d000.lazystatfs=0 multiop /mnt/lustre v_f TMPPIPE=/tmp/multiop_open_wait_pipe.6736 start ost2 service on oleg108-server Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg108-server: oleg108-server.virtnet: executing set_default_debug -1 all pdsh@oleg108-client: oleg108-server: ssh exited with exit code 1 Started lustre-OST0001 oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 1 sec stop ost2 service on oleg108-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg108-server umount lustre on /mnt/lustre..... Stopping client oleg108-client.virtnet /mnt/lustre (opts:-f) stop ost1 service on oleg108-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg108-server stop mds service on oleg108-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg108-server oleg108-server: oleg108-server: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre oleg108-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg108-client: oleg108-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg108-server: oleg108-server: tunefs.lustre FATAL: Device lustre-ost1/ost1 has not been formatted with mkfs.lustre oleg108-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg108-client: oleg108-server: ssh exited with exit code 19 checking for existing Lustre data: not found oleg108-server: oleg108-server: tunefs.lustre FATAL: Device lustre-ost2/ost2 has not been formatted with mkfs.lustre oleg108-server: tunefs.lustre: exiting with 19 (No such device) pdsh@oleg108-client: oleg108-server: ssh exited with exit code 19 checking for existing Lustre data: not found tunefs failed, reformatting instead Stopping clients: oleg108-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg108-client.virtnet /mnt/lustre2 (opts:-f) oleg108-server: oleg108-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory loading modules on: 'oleg108-server' oleg108-server: oleg108-server.virtnet: executing load_modules_local oleg108-server: Loading modules from /home/green/git/lustre-release/lustre oleg108-server: detected 4 online CPUs by sysfs oleg108-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg108-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg108-server: oleg108-server.virtnet: executing set_default_debug -1 all pdsh@oleg108-client: oleg108-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg108-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg108-server: oleg108-server.virtnet: executing set_default_debug -1 all pdsh@oleg108-client: oleg108-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg108-server: oleg108-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg108-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg108-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg108-server stop mds service on oleg108-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg108-server PASS 50f (142s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 69: replace an OST with the same index ========================================================== 16:01:32 (1710100892) start mds service on oleg108-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg108-server: oleg108-server.virtnet: executing set_default_debug -1 all pdsh@oleg108-client: oleg108-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg108-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg108-server: oleg108-server.virtnet: executing set_default_debug -1 all pdsh@oleg108-client: oleg108-server: ssh exited with exit code 1 Started lustre-OST0000 oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg108-client.virtnet: -o user_xattr,flock oleg108-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11872 inodes available. Want 99936. rc=0 - open/close 1073 (time 1710100925.48 total 10.01 last 107.20) - open/close 2013 (time 1710100935.49 total 20.01 last 93.99) - open/close 2966 (time 1710100945.50 total 30.02 last 95.20) - open/close 3949 (time 1710100955.50 total 40.03 last 98.25) - open/close 4744 (time 1710100965.51 total 50.04 last 79.42) - open/close 5529 (time 1710100975.52 total 60.04 last 78.43) - open/close 6204 (time 1710100985.53 total 70.05 last 67.44) - open/close 6858 (time 1710100995.54 total 80.06 last 65.32) - open/close 7473 (time 1710101005.54 total 90.07 last 61.48) - open/close 8168 (time 1710101015.54 total 100.07 last 69.50) - open/close 8780 (time 1710101025.55 total 110.07 last 61.16) - open/close 9369 (time 1710101035.55 total 120.08 last 58.89) - open/close 9994 (time 1710101045.56 total 130.09 last 62.42) total: 10000 open/close in 130.18 seconds: 76.82 ops/second - unlinked 0 (time 1710101048 ; total 0 ; last 0) total: 10000 unlinks in 93 seconds: 107.526878 unlinks/second - open/close 903 (time 1710101153.61 total 10.02 last 90.13) - open/close 1767 (time 1710101163.62 total 20.02 last 86.36) - open/close 2516 (time 1710101173.62 total 30.03 last 74.87) - open/close 3235 (time 1710101194.32 total 50.73 last 34.73) - open/close 3833 (time 1710101204.33 total 60.73 last 59.79) - open/close 4431 (time 1710101214.33 total 70.74 last 59.77) - open/close 5076 (time 1710101224.33 total 80.74 last 64.48) - open/close 5609 (time 1710101234.37 total 90.78 last 53.12) - open/close 6159 (time 1710101244.38 total 100.79 last 54.94) - open/close 6674 (time 1710101254.40 total 110.81 last 51.41) - open/close 7213 (time 1710101264.40 total 120.81 last 53.90) - open/close 7655 (time 1710101274.40 total 130.81 last 44.18) - open/close 8147 (time 1710101284.40 total 140.81 last 49.19) - open/close 8696 (time 1710101294.41 total 150.81 last 54.88) - open/close 9151 (time 1710101304.41 total 160.82 last 45.49) - open/close 9558 (time 1710101314.41 total 170.82 last 40.69) - open/close 9943 (time 1710101324.42 total 180.83 last 38.47) total: 10000 open/close in 182.31 seconds: 54.85 ops/second - unlinked 0 (time 1710101328 ; total 0 ; last 0) total: 10000 unlinks in 118 seconds: 84.745766 unlinks/second - open/close 883 (time 1710101458.98 total 10.06 last 87.77) - open/close 1617 (time 1710101468.98 total 20.07 last 73.36) - open/close 2256 (time 1710101478.99 total 30.07 last 63.88) - open/close 2922 (time 1710101489.00 total 40.08 last 66.53) - open/close 3567 (time 1710101499.01 total 50.09 last 64.44) - open/close 4149 (time 1710101509.01 total 60.09 last 58.19) - open/close 4801 (time 1710101519.01 total 70.09 last 65.17) - open/close 5369 (time 1710101529.03 total 80.11 last 56.69) - open/close 5917 (time 1710101539.03 total 90.12 last 54.77) - open/close 6439 (time 1710101549.04 total 100.12 last 52.17) - open/close 6976 (time 1710101559.05 total 110.13 last 53.65) - open/close 7528 (time 1710101569.05 total 120.14 last 55.18) - open/close 7960 (time 1710101579.12 total 130.20 last 42.90) - open/close 8413 (time 1710101589.13 total 140.21 last 45.26) - open/close 8866 (time 1710101599.15 total 150.23 last 45.23) - open/close 8987 (time 1710101612.54 total 163.62 last 9.04) - open/close 9423 (time 1710101622.56 total 173.64 last 43.51) - open/close 9838 (time 1710101632.58 total 183.66 last 41.43) total: 10000 open/close in 187.52 seconds: 53.33 ops/second - unlinked 0 (time 1710101639 ; total 0 ; last 0) total: 10000 unlinks in 116 seconds: 86.206894 unlinks/second - open/close 846 (time 1710101767.85 total 10.00 last 84.58) - open/close 1698 (time 1710101777.90 total 20.05 last 84.83) - open/close 2414 (time 1710101787.91 total 30.06 last 71.51) - open/close 3138 (time 1710101797.92 total 40.07 last 72.31) - open/close 3767 (time 1710101807.98 total 50.13 last 62.51) - open/close 4307 (time 1710101817.99 total 60.14 last 53.97) - open/close 4845 (time 1710101828.01 total 70.16 last 53.68) - open/close 5455 (time 1710101838.01 total 80.16 last 60.98) - open/close 6038 (time 1710101848.02 total 90.17 last 58.27) - open/close 6601 (time 1710101858.07 total 100.22 last 55.99) - open/close 7137 (time 1710101868.07 total 110.22 last 53.59) - open/close 7618 (time 1710101878.08 total 120.22 last 48.09) - open/close 8151 (time 1710101888.08 total 130.23 last 53.29) - open/close 8652 (time 1710101898.08 total 140.23 last 50.09) - open/close 9145 (time 1710101908.14 total 150.29 last 49.00) - open/close 9585 (time 1710101918.15 total 160.30 last 43.96) total: 10000 open/close in 169.84 seconds: 58.88 ops/second - unlinked 0 (time 1710101930 ; total 0 ; last 0) total: 10000 unlinks in 112 seconds: 89.285713 unlinks/second - open/close 861 (time 1710102054.45 total 10.01 last 86.05) - open/close 1618 (time 1710102064.45 total 20.01 last 75.67) - open/close 2307 (time 1710102084.88 total 40.44 last 33.72) - open/close 2930 (time 1710102094.91 total 50.47 last 62.10) - open/close 3583 (time 1710102104.96 total 60.51 last 65.02) - open/close 4195 (time 1710102114.96 total 70.51 last 61.19) - open/close 4803 (time 1710102124.98 total 80.54 last 60.66) - open/close 5426 (time 1710102134.98 total 90.54 last 62.29) - open/close 5971 (time 1710102145.00 total 100.56 last 54.42) - open/close 6517 (time 1710102155.01 total 110.57 last 54.54) - open/close 7018 (time 1710102165.02 total 120.58 last 50.04) - open/close 7483 (time 1710102175.04 total 130.60 last 46.42) - open/close 7965 (time 1710102185.04 total 140.60 last 48.20) - open/close 8504 (time 1710102195.04 total 150.60 last 53.90) - open/close 8962 (time 1710102205.07 total 160.62 last 45.68) - open/close 9411 (time 1710102215.11 total 170.67 last 44.71) - open/close 9860 (time 1710102225.12 total 180.68 last 44.86) total: 10000 open/close in 183.88 seconds: 54.38 ops/second - unlinked 0 (time 1710102231 ; total 0 ; last 0) total: 10000 unlinks in 112 seconds: 89.285713 unlinks/second - open/close 861 (time 1710102355.41 total 10.00 last 86.07) - open/close 1691 (time 1710102365.41 total 20.01 last 82.97) - open/close 2500 (time 1710102375.43 total 30.03 last 80.70) - open/close 3144 (time 1710102385.44 total 40.04 last 64.34) - open/close 3802 (time 1710102395.48 total 50.08 last 65.54) - open/close 4381 (time 1710102405.49 total 60.09 last 57.87) - open/close 4921 (time 1710102415.50 total 70.10 last 53.95) - open/close 5554 (time 1710102425.51 total 80.11 last 63.23) - open/close 6153 (time 1710102435.51 total 90.11 last 59.90) - open/close 6716 (time 1710102445.51 total 100.11 last 56.30) - open/close 7222 (time 1710102455.53 total 110.12 last 50.52) - open/close 7719 (time 1710102465.53 total 120.13 last 49.68) - open/close 8230 (time 1710102475.53 total 130.13 last 51.08) - open/close 8735 (time 1710102485.54 total 140.14 last 50.48) - open/close 9209 (time 1710102495.58 total 150.18 last 47.20) - open/close 9676 (time 1710102505.59 total 160.19 last 46.67) total: 10000 open/close in 167.44 seconds: 59.72 ops/second - unlinked 0 (time 1710102515 ; total 0 ; last 0) total: 10000 unlinks in 113 seconds: 88.495575 unlinks/second - open/close 886 (time 1710102641.18 total 10.00 last 88.60) - open/close 1705 (time 1710102651.18 total 20.00 last 81.89) - open/close 2479 (time 1710102661.19 total 30.01 last 77.35) - open/close 3147 (time 1710102671.24 total 40.06 last 66.47) - open/close 3777 (time 1710102681.25 total 50.07 last 62.91) - open/close 4376 (time 1710102691.26 total 60.08 last 59.87) - open/close 4948 (time 1710102701.26 total 70.08 last 57.16) - open/close 5501 (time 1710102711.29 total 80.11 last 55.18) - open/close 6041 (time 1710102721.31 total 90.13 last 53.86) - open/close 6548 (time 1710102731.33 total 100.15 last 50.58) - open/close 7063 (time 1710102741.35 total 110.17 last 51.43) - open/close 7549 (time 1710102751.35 total 120.17 last 48.59) - open/close 8131 (time 1710102761.36 total 130.18 last 58.15) - open/close 8627 (time 1710102771.36 total 140.18 last 49.57) - open/close 9121 (time 1710102781.36 total 150.18 last 49.39) - open/close 9615 (time 1710102791.37 total 160.19 last 49.37) total: 10000 open/close in 167.85 seconds: 59.58 ops/second - unlinked 0 (time 1710102801 ; total 0 ; last 0) total: 10000 unlinks in 116 seconds: 86.206894 unlinks/second - open/close 855 (time 1710102930.91 total 10.01 last 85.43) - open/close 1630 (time 1710102940.92 total 20.01 last 77.49) - open/close 2318 (time 1710102950.93 total 30.02 last 68.73) - open/close 2995 (time 1710102960.94 total 40.03 last 67.62) - open/close 3646 (time 1710102970.96 total 50.05 last 64.95) - open/close 4262 (time 1710102980.96 total 60.06 last 61.59) - open/close 4873 (time 1710102990.97 total 70.06 last 61.07) - open/close 5112 (time 1710103006.25 total 85.34 last 15.64) - open/close 5615 (time 1710103016.26 total 95.35 last 50.27) - open/close 6159 (time 1710103026.26 total 105.36 last 54.36) - open/close 6689 (time 1710103036.27 total 115.36 last 52.97) - open/close 7209 (time 1710103046.27 total 125.36 last 51.99) - open/close 7767 (time 1710103056.28 total 135.38 last 55.71) - open/close 8012 (time 1710103072.61 total 151.70 last 15.01) - open/close 8515 (time 1710103082.62 total 161.71 last 50.25) - open/close 9036 (time 1710103092.63 total 171.72 last 52.07) - open/close 9484 (time 1710103102.63 total 181.72 last 44.79) - open/close 9941 (time 1710103112.64 total 191.74 last 45.63) total: 10000 open/close in 193.16 seconds: 51.77 ops/second - unlinked 0 (time 1710103116 ; total 0 ; last 0) total: 10000 unlinks in 104 seconds: 96.153847 unlinks/second - open/close 845 (time 1710103232.79 total 10.00 last 84.47) - open/close 1654 (time 1710103242.80 total 20.01 last 80.82) - open/close 2483 (time 1710103252.81 total 30.02 last 82.82) - open/close 3216 (time 1710103262.82 total 40.03 last 73.28) - open/close 4021 (time 1710103272.82 total 50.03 last 80.47) - open/close 4779 (time 1710103282.83 total 60.04 last 75.70) - open/close 5426 (time 1710103292.84 total 70.05 last 64.67) - open/close 6133 (time 1710103302.84 total 80.05 last 70.69) - open/close 6796 (time 1710103312.85 total 90.06 last 66.22) - open/close 7462 (time 1710103322.87 total 100.07 last 66.51) - open/close 7509 (time 1710103334.65 total 111.86 last 3.99) - open/close 8139 (time 1710103344.66 total 121.87 last 62.94) - open/close 8765 (time 1710103354.67 total 131.88 last 62.58) - open/close 9267 (time 1710103374.00 total 151.21 last 25.96) - open/close 9884 (time 1710103384.01 total 161.22 last 61.67) total: 10000 open/close in 163.30 seconds: 61.24 ops/second - unlinked 0 (time 1710103388 ; total 0 ; last 0) total: 10000 unlinks in 83 seconds: 120.481926 unlinks/second - open/close 1114 (time 1710103482.66 total 10.00 last 111.36) - open/close 2276 (time 1710103492.66 total 20.01 last 116.15) - open/close 3454 (time 1710103502.67 total 30.01 last 117.73) - open/close 4496 (time 1710103512.67 total 40.01 last 104.19) - open/close 5561 (time 1710103522.67 total 50.02 last 106.43) - open/close 6604 (time 1710103532.67 total 60.02 last 104.29) - open/close 7554 (time 1710103542.68 total 70.02 last 94.98) - open/close 8491 (time 1710103552.68 total 80.03 last 93.63) - open/close 9416 (time 1710103562.69 total 90.04 last 92.43) total: 10000 open/close in 96.00 seconds: 104.17 ops/second - unlinked 0 (time 1710103569 ; total 0 ; last 0) total: 10000 unlinks in 46 seconds: 217.391312 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg108-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg108-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg108-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.201.108@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.201.108@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg108-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg108-server: oleg108-server.virtnet: executing set_default_debug -1 all pdsh@oleg108-client: oleg108-server: ssh exited with exit code 1 Started lustre-OST0000 oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg108-server: oleg108-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg108-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 11 sec mount lustre on /mnt/lustre..... Starting client: oleg108-client.virtnet: -o user_xattr,flock oleg108-server@tcp:/lustre /mnt/lustre On OST0, 8037 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg108-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg108-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg108-server stop mds service on oleg108-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg108-server unloading modules on: 'oleg108-server' oleg108-server: oleg108-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg108-client: oleg108-client: ssh exited with exit code 2 pdsh@oleg108-client: oleg108-server: ssh exited with exit code 2 pdsh@oleg108-client: oleg108-client: ssh exited with exit code 2 pdsh@oleg108-client: oleg108-server: ssh exited with exit code 2 PASS 69 (2764s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg108-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg108-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg108-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg108-client: oleg108-server: ssh exited with exit code 2 oleg108-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg108-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == conf-sanity test 93: register mulitple MDT at the same time ========================================================== 16:47:38 (1710103658) SKIP: conf-sanity test_93 needs >= 3 MDTs SKIP 93 (1s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg108-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg108-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory Stopping clients: oleg108-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg108-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg108-client: oleg108-server: ssh exited with exit code 2 oleg108-server: oleg108-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg108-server' oleg108-server: oleg108-server.virtnet: executing load_modules_local oleg108-server: Loading modules from /home/green/git/lustre-release/lustre oleg108-server: detected 4 online CPUs by sysfs oleg108-server: Force libcfs to create 2 CPU partitions oleg108-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory oleg108-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg108-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 3666 sec ========== 16:47:58 (1710103678) === conf-sanity: start cleanup 16:47:58 (1710103678) === === conf-sanity: finish cleanup 16:47:58 (1710103678) ===