-----============= acceptance-small: conf-sanity ============----- Thu May 1 16:34:05 EDT 2025 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg306-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex: No such file or directory excepting tests: 32b 32c 32newtarball 108a 112a 112b 113 117 119 121 122a 123aa 123ab 123ac 123ad 123ae 123af 123ag 123ah 123ahi 123F 123G 123H 126 129 132 133 135 136 137 150 152 153a 153b 153c 155 802a Stopping clients: oleg306-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg306-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg306-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg306-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg306-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg306-server oleg306-server: oleg306-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg306-server' oleg306-server: oleg306-server.virtnet: executing load_modules_local oleg306-server: Loading modules from /home/green/git/lustre-release/lustre oleg306-server: detected 4 online CPUs by sysfs oleg306-server: MODOPTS_LIBCFS= oleg306-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg306-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg306-server: oleg306-server.virtnet: executing set_default_debug -1 all pdsh@oleg306-client: oleg306-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg306-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg306-server: oleg306-server.virtnet: executing set_default_debug -1 all pdsh@oleg306-client: oleg306-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg306-server: oleg306-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg306-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg306-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg306-server stop mds service on oleg306-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg306-server == conf-sanity test 45: long unlink handling in ptlrpcd == 16:35:37 (1746131737) start mds service on oleg306-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg306-server: oleg306-server.virtnet: executing set_default_debug -1 all pdsh@oleg306-client: oleg306-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg306-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg306-server: oleg306-server.virtnet: executing set_default_debug -1 all pdsh@oleg306-client: oleg306-server: ssh exited with exit code 1 Started lustre-OST0000 oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg306-client.virtnet: -o user_xattr,flock 192.168.203.106@tcp:/lustre /mnt/lustre setup single mount lustre success stop mds service on oleg306-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg306-server sleep 60 sec fail_loc=0x8000050f sleep 10 sec manual umount lustre on /mnt/lustre.... df: '/mnt/lustre': Cannot send after transport endpoint shutdown fail_loc=0x0 start mds service on oleg306-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg306-server: oleg306-server.virtnet: executing set_default_debug -1 all pdsh@oleg306-client: oleg306-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg306-client.virtnet: -o user_xattr,flock 192.168.203.106@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg306-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg306-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg306-server stop mds service on oleg306-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg306-server unloading modules via unload_modules_local on: 'oleg306-server' oleg306-server: oleg306-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg306-client: oleg306-server: ssh exited with exit code 2 pdsh@oleg306-client: oleg306-client: ssh exited with exit code 2 pdsh@oleg306-client: oleg306-server: ssh exited with exit code 2 pdsh@oleg306-client: oleg306-client: ssh exited with exit code 2 PASS 45 (141s) == conf-sanity test 69: replace an OST with the same index ========================================================== 16:37:58 (1746131878) start mds service on oleg306-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg306-server' oleg306-server: oleg306-server.virtnet: executing load_modules_local oleg306-server: Loading modules from /home/green/git/lustre-release/lustre oleg306-server: detected 4 online CPUs by sysfs oleg306-server: MODOPTS_LIBCFS= oleg306-server: Force libcfs to create 2 CPU partitions oleg306-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg306-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg306-server: oleg306-server.virtnet: executing set_default_debug -1 all pdsh@oleg306-client: oleg306-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg306-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg306-server: oleg306-server.virtnet: executing set_default_debug -1 all pdsh@oleg306-client: oleg306-server: ssh exited with exit code 1 Started lustre-OST0000 oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg306-client.virtnet: -o user_xattr,flock 192.168.203.106@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11872 inodes available. Want 99872. rc=0 - open/close 1219 (time 1746131925.51 total 10.00 last 121.86) - open/close 2142 (time 1746131935.52 total 20.01 last 92.22) - open/close 3085 (time 1746131945.53 total 30.02 last 94.26) - open/close 4073 (time 1746131955.53 total 40.02 last 98.77) - open/close 5190 (time 1746131965.53 total 50.02 last 111.65) - open/close 6143 (time 1746131975.54 total 60.03 last 95.22) - open/close 7032 (time 1746131985.54 total 70.03 last 88.89) - open/close 7914 (time 1746131995.54 total 80.03 last 88.19) - open/close 8927 (time 1746132005.55 total 90.04 last 101.24) - open/close 9950 (time 1746132015.56 total 100.05 last 102.18) total: 10000 open/close in 100.60 seconds: 99.41 ops/second - unlinked 0 (time 1746132018 ; total 0 ; last 0) total: 10000 unlinks in 64 seconds: 156.250000 unlinks/second - open/close 1084 (time 1746132093.95 total 10.01 last 108.33) - open/close 2145 (time 1746132103.95 total 20.01 last 106.02) - open/close 3076 (time 1746132113.97 total 30.02 last 93.00) - open/close 4027 (time 1746132123.97 total 40.03 last 95.04) - open/close 4989 (time 1746132133.98 total 50.03 last 96.17) - open/close 6008 (time 1746132143.98 total 60.04 last 101.85) - open/close 6947 (time 1746132154.00 total 70.05 last 93.76) - open/close 7833 (time 1746132164.00 total 80.06 last 88.55) - open/close 8736 (time 1746132174.01 total 90.07 last 90.24) - open/close 9710 (time 1746132184.01 total 100.07 last 97.36) total: 10000 open/close in 102.88 seconds: 97.20 ops/second - unlinked 0 (time 1746132188 ; total 0 ; last 0) total: 10000 unlinks in 60 seconds: 166.666672 unlinks/second - open/close 912 (time 1746132261.05 total 10.01 last 91.12) - open/close 1846 (time 1746132271.08 total 20.03 last 93.17) - open/close 3113 (time 1746132281.08 total 30.04 last 126.63) - open/close 4118 (time 1746132291.09 total 40.05 last 100.43) - open/close 5271 (time 1746132301.10 total 50.05 last 115.23) - open/close 6347 (time 1746132311.10 total 60.05 last 107.58) - open/close 7505 (time 1746132321.10 total 70.05 last 115.80) - open/close 8394 (time 1746132331.12 total 80.07 last 88.75) - open/close 9350 (time 1746132341.12 total 90.08 last 95.54) total: 10000 open/close in 96.53 seconds: 103.60 ops/second - unlinked 0 (time 1746132349 ; total 0 ; last 0) total: 10000 unlinks in 57 seconds: 175.438599 unlinks/second - open/close 897 (time 1746132418.10 total 10.00 last 89.67) - open/close 2119 (time 1746132428.10 total 20.01 last 122.17) - open/close 3164 (time 1746132438.11 total 30.02 last 104.39) - open/close 4025 (time 1746132448.12 total 40.02 last 86.03) - open/close 4992 (time 1746132458.12 total 50.03 last 96.68) - open/close 6199 (time 1746132468.14 total 60.05 last 120.46) - open/close 7098 (time 1746132478.14 total 70.05 last 89.89) - open/close 7994 (time 1746132488.16 total 80.07 last 89.42) - open/close 8917 (time 1746132498.17 total 90.07 last 92.26) total: 10000 open/close in 98.45 seconds: 101.58 ops/second - unlinked 0 (time 1746132508 ; total 0 ; last 0) total: 10000 unlinks in 62 seconds: 161.290329 unlinks/second - open/close 983 (time 1746132582.55 total 10.00 last 98.28) - open/close 2114 (time 1746132592.55 total 20.01 last 113.04) - open/close 3048 (time 1746132602.56 total 30.01 last 93.39) - open/close 4049 (time 1746132612.57 total 40.02 last 99.98) - open/close 5136 (time 1746132622.57 total 50.03 last 108.63) - open/close 6218 (time 1746132632.58 total 60.03 last 108.19) - open/close 7299 (time 1746132642.58 total 70.03 last 108.08) - open/close 8476 (time 1746132652.58 total 80.04 last 117.63) - open/close 9645 (time 1746132662.58 total 90.04 last 116.89) total: 10000 open/close in 93.82 seconds: 106.59 ops/second - unlinked 0 (time 1746132668 ; total 0 ; last 0) total: 10000 unlinks in 58 seconds: 172.413788 unlinks/second - open/close 1062 (time 1746132737.99 total 10.02 last 106.03) - open/close 2065 (time 1746132748.00 total 20.03 last 100.12) - open/close 3124 (time 1746132758.01 total 30.03 last 105.89) - open/close 4362 (time 1746132768.01 total 40.04 last 123.76) - open/close 5334 (time 1746132778.01 total 50.04 last 97.16) - open/close 6287 (time 1746132788.02 total 60.05 last 95.25) - open/close 7460 (time 1746132798.02 total 70.05 last 117.28) - open/close 8506 (time 1746132808.02 total 80.05 last 104.57) - open/close 9487 (time 1746132818.04 total 90.07 last 97.88) total: 10000 open/close in 95.23 seconds: 105.01 ops/second - unlinked 0 (time 1746132825 ; total 0 ; last 0) total: 10000 unlinks in 53 seconds: 188.679245 unlinks/second - open/close 955 (time 1746132890.44 total 10.00 last 95.49) - open/close 2031 (time 1746132900.45 total 20.01 last 107.56) - open/close 3334 (time 1746132910.45 total 30.01 last 130.29) - open/close 4432 (time 1746132920.45 total 40.01 last 109.73) - open/close 5619 (time 1746132930.45 total 50.01 last 118.68) - open/close 7012 (time 1746132940.47 total 60.03 last 139.09) - open/close 7984 (time 1746132950.48 total 70.04 last 97.13) - open/close 9050 (time 1746132960.48 total 80.04 last 106.59) total: 10000 open/close in 88.20 seconds: 113.38 ops/second - unlinked 0 (time 1746132969 ; total 0 ; last 0) total: 10000 unlinks in 49 seconds: 204.081635 unlinks/second - open/close 1371 (time 1746133030.55 total 10.01 last 137.00) - open/close 2514 (time 1746133040.56 total 20.01 last 114.24) - open/close 3565 (time 1746133050.56 total 30.02 last 105.07) - open/close 4695 (time 1746133060.57 total 40.02 last 112.93) - open/close 5955 (time 1746133070.57 total 50.03 last 125.95) - open/close 7128 (time 1746133080.57 total 60.03 last 117.29) - open/close 8402 (time 1746133090.58 total 70.03 last 127.37) - open/close 9877 (time 1746133100.60 total 80.05 last 147.19) total: 10000 open/close in 81.22 seconds: 123.12 ops/second - unlinked 0 (time 1746133103 ; total 0 ; last 0) total: 10000 unlinks in 53 seconds: 188.679245 unlinks/second - open/close 1202 (time 1746133168.71 total 10.00 last 120.19) - open/close 2211 (time 1746133178.71 total 20.01 last 100.85) - open/close 3244 (time 1746133188.72 total 30.01 last 103.20) - open/close 4344 (time 1746133198.72 total 40.02 last 109.98) - open/close 5750 (time 1746133208.72 total 50.02 last 140.56) - open/close 7055 (time 1746133218.74 total 60.04 last 130.25) - open/close 8398 (time 1746133228.75 total 70.04 last 134.26) - open/close 9493 (time 1746133238.75 total 80.05 last 109.44) total: 10000 open/close in 84.16 seconds: 118.82 ops/second - unlinked 0 (time 1746133244 ; total 0 ; last 0) total: 10000 unlinks in 45 seconds: 222.222229 unlinks/second - open/close 1089 (time 1746133301.70 total 10.00 last 108.90) - open/close 2308 (time 1746133311.70 total 20.00 last 121.88) - open/close 3605 (time 1746133321.71 total 30.01 last 129.60) - open/close 4776 (time 1746133331.72 total 40.02 last 117.00) - open/close 6102 (time 1746133341.72 total 50.02 last 132.57) - open/close 7521 (time 1746133351.72 total 60.02 last 141.90) - open/close 8808 (time 1746133361.72 total 70.02 last 128.68) total: 10000 open/close in 79.25 seconds: 126.18 ops/second - unlinked 0 (time 1746133372 ; total 0 ; last 0) total: 10000 unlinks in 38 seconds: 263.157898 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg306-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg306-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg306-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.203.106@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.203.106@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg306-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg306-server: oleg306-server.virtnet: executing set_default_debug -1 all pdsh@oleg306-client: oleg306-server: ssh exited with exit code 1 Started lustre-OST0000 oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg306-server: oleg306-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg306-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec mount lustre on /mnt/lustre..... Starting client: oleg306-client.virtnet: -o user_xattr,flock 192.168.203.106@tcp:/lustre /mnt/lustre On OST0, 6773 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg306-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg306-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg306-server stop mds service on oleg306-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg306-server unloading modules via unload_modules_local on: 'oleg306-server' oleg306-server: oleg306-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg306-client: oleg306-client: ssh exited with exit code 2 pdsh@oleg306-client: oleg306-server: ssh exited with exit code 2 pdsh@oleg306-client: oleg306-server: ssh exited with exit code 2 pdsh@oleg306-client: oleg306-client: ssh exited with exit code 2 PASS 69 (1582s) == conf-sanity test 111: Adding large_dir with over 2GB directory ========================================================== 17:04:20 (1746133460) SKIP: conf-sanity test_111 Only applicable to ldiskfs-based MDTs SKIP 111 (2s) Stopping clients: oleg306-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg306-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg306-client: oleg306-server: ssh exited with exit code 2 oleg306-server: oleg306-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg306-server' oleg306-server: oleg306-server.virtnet: executing load_modules_local oleg306-server: Loading modules from /home/green/git/lustre-release/lustre oleg306-server: detected 4 online CPUs by sysfs oleg306-server: MODOPTS_LIBCFS= oleg306-server: Force libcfs to create 2 CPU partitions oleg306-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg306-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 1852 sec ========== 17:04:58 (1746133498) === conf-sanity: start cleanup 17:04:59 (1746133499) === === conf-sanity: finish cleanup 17:05:01 (1746133501) ===