-----============= acceptance-small: conf-sanity ============----- Wed Apr 30 23:43:08 EDT 2025 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg453-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex: No such file or directory excepting tests: 32b 32c 32newtarball 108a 112a 112b 113 117 119 121 122a 123aa 123ab 123ac 123ad 123ae 123af 123ag 123ah 123ahi 123F 123G 123H 126 129 132 133 135 136 137 150 152 153a 153b 153c 155 802a Stopping clients: oleg453-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg453-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg453-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg453-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg453-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg453-server oleg453-server: oleg453-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg453-server' oleg453-server: oleg453-server.virtnet: executing load_modules_local oleg453-server: Loading modules from /home/green/git/lustre-release/lustre oleg453-server: detected 4 online CPUs by sysfs oleg453-server: MODOPTS_LIBCFS= oleg453-server: Force libcfs to create 2 CPU partitions oleg453-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg453-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg453-server: oleg453-server.virtnet: executing set_default_debug -1 all pdsh@oleg453-client: oleg453-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg453-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg453-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg453-server: oleg453-server.virtnet: executing set_default_debug -1 all pdsh@oleg453-client: oleg453-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg453-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg453-server: oleg453-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg453-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg453-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg453-server stop mds service on oleg453-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg453-server == conf-sanity test 45: long unlink handling in ptlrpcd == 23:44:28 (1746071068) start mds service on oleg453-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg453-server: oleg453-server.virtnet: executing set_default_debug -1 all pdsh@oleg453-client: oleg453-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg453-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg453-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg453-server: oleg453-server.virtnet: executing set_default_debug -1 all pdsh@oleg453-client: oleg453-server: ssh exited with exit code 1 Started lustre-OST0000 oleg453-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg453-client.virtnet: -o user_xattr,flock 192.168.204.153@tcp:/lustre /mnt/lustre setup single mount lustre success stop mds service on oleg453-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg453-server sleep 60 sec fail_loc=0x8000050f sleep 10 sec manual umount lustre on /mnt/lustre.... df: '/mnt/lustre': Cannot send after transport endpoint shutdown fail_loc=0x0 start mds service on oleg453-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg453-server: oleg453-server.virtnet: executing set_default_debug -1 all pdsh@oleg453-client: oleg453-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg453-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg453-client.virtnet: -o user_xattr,flock 192.168.204.153@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg453-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg453-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg453-server stop mds service on oleg453-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg453-server unloading modules via unload_modules_local on: 'oleg453-server' oleg453-server: oleg453-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg453-client: oleg453-client: ssh exited with exit code 2 pdsh@oleg453-client: oleg453-server: ssh exited with exit code 2 pdsh@oleg453-client: oleg453-server: ssh exited with exit code 2 pdsh@oleg453-client: oleg453-client: ssh exited with exit code 2 PASS 45 (126s) == conf-sanity test 69: replace an OST with the same index ========================================================== 23:46:34 (1746071194) start mds service on oleg453-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg453-server' oleg453-server: oleg453-server.virtnet: executing load_modules_local oleg453-server: Loading modules from /home/green/git/lustre-release/lustre oleg453-server: detected 4 online CPUs by sysfs oleg453-server: MODOPTS_LIBCFS= oleg453-server: Force libcfs to create 2 CPU partitions oleg453-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg453-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg453-server: oleg453-server.virtnet: executing set_default_debug -1 all pdsh@oleg453-client: oleg453-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg453-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg453-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg453-server: oleg453-server.virtnet: executing set_default_debug -1 all pdsh@oleg453-client: oleg453-server: ssh exited with exit code 1 Started lustre-OST0000 oleg453-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg453-client.virtnet: -o user_xattr,flock 192.168.204.153@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11872 inodes available. Want 99872. rc=0 - open/close 1620 (time 1746071236.03 total 10.00 last 161.95) - open/close 3161 (time 1746071246.03 total 20.01 last 154.05) - open/close 4391 (time 1746071256.04 total 30.02 last 122.88) - open/close 5664 (time 1746071266.05 total 40.02 last 127.24) - open/close 6828 (time 1746071276.05 total 50.03 last 116.31) - open/close 8159 (time 1746071286.06 total 60.03 last 133.02) - open/close 9151 (time 1746071296.07 total 70.05 last 99.06) total: 10000 open/close in 78.47 seconds: 127.44 ops/second - unlinked 0 (time 1746071306 ; total 0 ; last 0) total: 10000 unlinks in 42 seconds: 238.095245 unlinks/second - open/close 1722 (time 1746071359.54 total 10.01 last 172.09) - open/close 3207 (time 1746071369.54 total 20.01 last 148.40) - open/close 4758 (time 1746071379.55 total 30.02 last 155.02) - open/close 6121 (time 1746071389.55 total 40.02 last 136.28) - open/close 7672 (time 1746071399.56 total 50.03 last 154.97) - open/close 9077 (time 1746071409.56 total 60.03 last 140.45) total: 10000 open/close in 64.92 seconds: 154.04 ops/second - unlinked 0 (time 1746071415 ; total 0 ; last 0) total: 10000 unlinks in 32 seconds: 312.500000 unlinks/second - open/close 1745 (time 1746071459.00 total 10.00 last 174.46) - open/close 3794 (time 1746071469.00 total 20.00 last 204.84) - open/close 5523 (time 1746071479.01 total 30.01 last 172.81) - open/close 7536 (time 1746071489.01 total 40.01 last 201.27) - open/close 9489 (time 1746071499.02 total 50.02 last 195.09) total: 10000 open/close in 53.80 seconds: 185.86 ops/second - unlinked 0 (time 1746071504 ; total 0 ; last 0) total: 10000 unlinks in 40 seconds: 250.000000 unlinks/second - open/close 1183 (time 1746071555.98 total 10.00 last 118.26) - open/close 2535 (time 1746071565.98 total 20.01 last 135.13) - open/close 3724 (time 1746071575.98 total 30.01 last 118.90) - open/close 4925 (time 1746071585.99 total 40.01 last 120.06) - open/close 6128 (time 1746071595.99 total 50.01 last 120.27) - open/close 7406 (time 1746071605.99 total 60.02 last 127.76) - open/close 8739 (time 1746071615.99 total 70.02 last 133.29) - open/close 9924 (time 1746071626.00 total 80.02 last 118.44) total: 10000 open/close in 80.79 seconds: 123.78 ops/second - unlinked 0 (time 1746071628 ; total 0 ; last 0) total: 10000 unlinks in 47 seconds: 212.765961 unlinks/second - open/close 1381 (time 1746071686.79 total 10.00 last 138.08) - open/close 2788 (time 1746071696.79 total 20.00 last 140.66) - open/close 4650 (time 1746071706.80 total 30.01 last 186.08) - open/close 6639 (time 1746071716.80 total 40.02 last 198.81) - open/close 8398 (time 1746071726.80 total 50.02 last 175.84) - open/close 9824 (time 1746071736.81 total 60.02 last 142.58) total: 10000 open/close in 61.41 seconds: 162.83 ops/second - unlinked 0 (time 1746071740 ; total 0 ; last 0) total: 10000 unlinks in 43 seconds: 232.558136 unlinks/second - open/close 1312 (time 1746071795.13 total 10.01 last 131.03) - open/close 2875 (time 1746071805.13 total 20.01 last 156.30) - open/close 4770 (time 1746071815.14 total 30.02 last 189.43) - open/close 6602 (time 1746071825.14 total 40.02 last 183.18) - open/close 9003 (time 1746071835.14 total 50.02 last 240.09) total: 10000 open/close in 55.26 seconds: 180.96 ops/second - unlinked 0 (time 1746071841 ; total 0 ; last 0) total: 10000 unlinks in 22 seconds: 454.545441 unlinks/second - open/close 1822 (time 1746071874.61 total 10.00 last 182.17) - open/close 3344 (time 1746071884.62 total 20.01 last 152.03) - open/close 4686 (time 1746071894.62 total 30.02 last 134.13) - open/close 6648 (time 1746071904.63 total 40.02 last 196.10) - open/close 8988 (time 1746071914.63 total 50.03 last 233.94) total: 10000 open/close in 55.15 seconds: 181.31 ops/second - unlinked 0 (time 1746071920 ; total 0 ; last 0) total: 10000 unlinks in 22 seconds: 454.545441 unlinks/second - open/close 2115 (time 1746071953.95 total 10.00 last 211.44) - open/close 4777 (time 1746071963.95 total 20.00 last 266.18) - open/close 7033 (time 1746071973.95 total 30.01 last 225.55) - open/close 9365 (time 1746071983.95 total 40.01 last 233.20) total: 10000 open/close in 42.14 seconds: 237.31 ops/second - unlinked 0 (time 1746071987 ; total 0 ; last 0) total: 10000 unlinks in 21 seconds: 476.190491 unlinks/second - open/close 2577 (time 1746072019.41 total 10.00 last 257.61) - open/close 4918 (time 1746072029.41 total 20.01 last 234.07) - open/close 6776 (time 1746072039.41 total 30.01 last 185.77) - open/close 8791 (time 1746072049.41 total 40.01 last 201.48) total: 10000 open/close in 46.34 seconds: 215.80 ops/second - unlinked 0 (time 1746072056 ; total 0 ; last 0) total: 10000 unlinks in 20 seconds: 500.000000 unlinks/second - open/close 2887 (time 1746072087.81 total 10.00 last 288.68) - open/close 5468 (time 1746072097.81 total 20.00 last 258.09) - open/close 7799 (time 1746072107.82 total 30.00 last 233.06) total: 10000 open/close in 37.57 seconds: 266.16 ops/second - unlinked 0 (time 1746072116 ; total 0 ; last 0) total: 10000 unlinks in 24 seconds: 416.666656 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg453-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg453-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg453-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.204.153@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.204.153@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg453-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg453-server: oleg453-server.virtnet: executing set_default_debug -1 all pdsh@oleg453-client: oleg453-server: ssh exited with exit code 1 Started lustre-OST0000 oleg453-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg453-server: oleg453-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg453-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 4 sec mount lustre on /mnt/lustre..... Starting client: oleg453-client.virtnet: -o user_xattr,flock 192.168.204.153@tcp:/lustre /mnt/lustre On OST0, 7829 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg453-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg453-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg453-server stop mds service on oleg453-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg453-server unloading modules via unload_modules_local on: 'oleg453-server' oleg453-server: oleg453-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg453-client: oleg453-client: ssh exited with exit code 2 pdsh@oleg453-client: oleg453-server: ssh exited with exit code 2 pdsh@oleg453-client: oleg453-client: ssh exited with exit code 2 pdsh@oleg453-client: oleg453-server: ssh exited with exit code 2 PASS 69 (986s) == conf-sanity test 111: Adding large_dir with over 2GB directory ========================================================== 00:03:01 (1746072181) SKIP: conf-sanity test_111 Only applicable to ldiskfs-based MDTs SKIP 111 (1s) Stopping clients: oleg453-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg453-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg453-client: oleg453-server: ssh exited with exit code 2 oleg453-server: oleg453-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg453-server' oleg453-server: oleg453-server.virtnet: executing load_modules_local oleg453-server: Loading modules from /home/green/git/lustre-release/lustre oleg453-server: detected 4 online CPUs by sysfs oleg453-server: MODOPTS_LIBCFS= oleg453-server: Force libcfs to create 2 CPU partitions oleg453-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg453-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 1218 sec ========== 00:03:27 (1746072207) === conf-sanity: start cleanup 00:03:28 (1746072208) === === conf-sanity: finish cleanup 00:03:29 (1746072209) ===