-----============= acceptance-small: conf-sanity ============----- Thu May 1 17:06:24 EDT 2025 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg406-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex: No such file or directory excepting tests: 32b 32c 32newtarball 108a 112a 112b 113 117 119 121 122a 123aa 123ab 123ac 123ad 123ae 123af 123ag 123ah 123ahi 123F 123G 123H 126 129 132 133 135 136 137 150 152 153a 153b 153c 155 802a Stopping clients: oleg406-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg406-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg406-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg406-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg406-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg406-server oleg406-server: oleg406-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory loading modules on: 'oleg406-server' oleg406-server: oleg406-server.virtnet: executing load_modules_local oleg406-server: Loading modules from /home/green/git/lustre-release/lustre oleg406-server: detected 4 online CPUs by sysfs oleg406-server: MODOPTS_LIBCFS= oleg406-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg406-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg406-server: oleg406-server.virtnet: executing set_default_debug -1 all pdsh@oleg406-client: oleg406-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg406-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg406-server: oleg406-server.virtnet: executing set_default_debug -1 all pdsh@oleg406-client: oleg406-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg406-server: oleg406-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg406-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg406-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg406-server stop mds service on oleg406-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg406-server == conf-sanity test 45: long unlink handling in ptlrpcd == 17:07:36 (1746133656) start mds service on oleg406-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg406-server: oleg406-server.virtnet: executing set_default_debug -1 all pdsh@oleg406-client: oleg406-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg406-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg406-server: oleg406-server.virtnet: executing set_default_debug -1 all pdsh@oleg406-client: oleg406-server: ssh exited with exit code 1 Started lustre-OST0000 oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg406-client.virtnet: -o user_xattr,flock 192.168.204.106@tcp:/lustre /mnt/lustre setup single mount lustre success stop mds service on oleg406-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg406-server sleep 60 sec fail_loc=0x8000050f sleep 10 sec manual umount lustre on /mnt/lustre.... df: '/mnt/lustre': Cannot send after transport endpoint shutdown fail_loc=0x0 start mds service on oleg406-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg406-server: oleg406-server.virtnet: executing set_default_debug -1 all pdsh@oleg406-client: oleg406-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg406-client.virtnet: -o user_xattr,flock 192.168.204.106@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg406-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg406-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg406-server stop mds service on oleg406-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg406-server unloading modules via unload_modules_local on: 'oleg406-server' oleg406-server: oleg406-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg406-client: oleg406-client: ssh exited with exit code 2 pdsh@oleg406-client: oleg406-server: ssh exited with exit code 2 pdsh@oleg406-client: oleg406-client: ssh exited with exit code 2 pdsh@oleg406-client: oleg406-server: ssh exited with exit code 2 PASS 45 (130s) == conf-sanity test 69: replace an OST with the same index ========================================================== 17:09:46 (1746133786) start mds service on oleg406-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg406-server' oleg406-server: oleg406-server.virtnet: executing load_modules_local oleg406-server: Loading modules from /home/green/git/lustre-release/lustre oleg406-server: detected 4 online CPUs by sysfs oleg406-server: MODOPTS_LIBCFS= oleg406-server: Force libcfs to create 2 CPU partitions oleg406-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg406-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg406-server: oleg406-server.virtnet: executing set_default_debug -1 all pdsh@oleg406-client: oleg406-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg406-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg406-server: oleg406-server.virtnet: executing set_default_debug -1 all pdsh@oleg406-client: oleg406-server: ssh exited with exit code 1 Started lustre-OST0000 oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg406-client.virtnet: -o user_xattr,flock 192.168.204.106@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11872 inodes available. Want 99872. rc=0 - open/close 2219 (time 1746133825.47 total 10.01 last 221.77) - open/close 4060 (time 1746133835.48 total 20.01 last 184.07) - open/close 6337 (time 1746133845.48 total 30.01 last 227.64) - open/close 8722 (time 1746133855.48 total 40.01 last 238.45) total: 10000 open/close in 45.97 seconds: 217.53 ops/second - unlinked 0 (time 1746133862 ; total 0 ; last 0) total: 10000 unlinks in 34 seconds: 294.117645 unlinks/second - open/close 1716 (time 1746133908.07 total 10.01 last 171.49) - open/close 3197 (time 1746133918.07 total 20.01 last 148.06) - open/close 4481 (time 1746133928.09 total 30.02 last 128.25) - open/close 5865 (time 1746133938.09 total 40.03 last 138.34) - open/close 7641 (time 1746133948.09 total 50.03 last 177.55) total: 10000 open/close in 58.77 seconds: 170.15 ops/second - unlinked 0 (time 1746133957 ; total 0 ; last 0) total: 10000 unlinks in 19 seconds: 526.315796 unlinks/second - open/close 2119 (time 1746133988.04 total 10.01 last 211.77) - open/close 3658 (time 1746133998.04 total 20.01 last 153.90) - open/close 5150 (time 1746134008.04 total 30.01 last 149.18) - open/close 6617 (time 1746134018.04 total 40.01 last 146.68) - open/close 8125 (time 1746134028.05 total 50.01 last 150.74) - open/close 9673 (time 1746134038.05 total 60.02 last 154.73) total: 10000 open/close in 63.23 seconds: 158.16 ops/second - unlinked 0 (time 1746134042 ; total 0 ; last 0) total: 10000 unlinks in 31 seconds: 322.580658 unlinks/second - open/close 2147 (time 1746134084.22 total 10.00 last 214.66) - open/close 4022 (time 1746134094.22 total 20.00 last 187.46) - open/close 6488 (time 1746134104.22 total 30.00 last 246.59) - open/close 8301 (time 1746134114.22 total 40.00 last 181.29) - open/close 9771 (time 1746134124.22 total 50.01 last 146.93) total: 10000 open/close in 51.55 seconds: 193.99 ops/second - unlinked 0 (time 1746134127 ; total 0 ; last 0) total: 10000 unlinks in 40 seconds: 250.000000 unlinks/second - open/close 1649 (time 1746134178.87 total 10.00 last 164.83) - open/close 3470 (time 1746134188.87 total 20.01 last 182.02) - open/close 5523 (time 1746134198.88 total 30.01 last 205.29) - open/close 7197 (time 1746134208.88 total 40.01 last 167.39) - open/close 8457 (time 1746134218.88 total 50.02 last 125.92) - open/close 9694 (time 1746134228.88 total 60.02 last 123.69) total: 10000 open/close in 62.37 seconds: 160.34 ops/second - unlinked 0 (time 1746134232 ; total 0 ; last 0) total: 10000 unlinks in 30 seconds: 333.333344 unlinks/second - open/close 1675 (time 1746134273.64 total 10.00 last 167.50) - open/close 3430 (time 1746134283.64 total 20.00 last 175.46) - open/close 5720 (time 1746134293.64 total 30.01 last 228.93) - open/close 8232 (time 1746134303.64 total 40.01 last 251.17) total: 10000 open/close in 46.30 seconds: 215.97 ops/second - unlinked 0 (time 1746134310 ; total 0 ; last 0) total: 10000 unlinks in 23 seconds: 434.782623 unlinks/second - open/close 2348 (time 1746134344.78 total 10.00 last 234.70) - open/close 4148 (time 1746134354.78 total 20.01 last 179.90) - open/close 5786 (time 1746134364.78 total 30.01 last 163.78) - open/close 7125 (time 1746134374.78 total 40.01 last 133.88) - open/close 8681 (time 1746134384.79 total 50.01 last 155.58) total: 10000 open/close in 56.20 seconds: 177.95 ops/second - unlinked 0 (time 1746134392 ; total 0 ; last 0) total: 10000 unlinks in 36 seconds: 277.777771 unlinks/second - open/close 1753 (time 1746134439.73 total 10.00 last 175.29) - open/close 3674 (time 1746134449.74 total 20.01 last 191.98) - open/close 5623 (time 1746134459.74 total 30.01 last 194.85) - open/close 7079 (time 1746134469.74 total 40.01 last 145.60) - open/close 8454 (time 1746134479.75 total 50.02 last 137.42) - open/close 9574 (time 1746134489.75 total 60.02 last 111.90) total: 10000 open/close in 63.03 seconds: 158.65 ops/second - unlinked 0 (time 1746134494 ; total 0 ; last 0) total: 10000 unlinks in 44 seconds: 227.272720 unlinks/second - open/close 1513 (time 1746134549.75 total 10.00 last 151.24) - open/close 3150 (time 1746134559.75 total 20.01 last 163.68) - open/close 4673 (time 1746134569.75 total 30.01 last 152.29) - open/close 6295 (time 1746134579.76 total 40.01 last 162.17) - open/close 8005 (time 1746134589.76 total 50.01 last 170.97) - open/close 9798 (time 1746134599.76 total 60.01 last 179.27) total: 10000 open/close in 61.45 seconds: 162.73 ops/second - unlinked 0 (time 1746134602 ; total 0 ; last 0) total: 10000 unlinks in 43 seconds: 232.558136 unlinks/second - open/close 1258 (time 1746134657.56 total 10.00 last 125.80) - open/close 2429 (time 1746134667.58 total 20.01 last 116.96) - open/close 3893 (time 1746134677.58 total 30.02 last 146.30) - open/close 5641 (time 1746134687.59 total 40.02 last 174.74) - open/close 7250 (time 1746134697.59 total 50.03 last 160.83) - open/close 8412 (time 1746134707.59 total 60.03 last 116.17) - open/close 9753 (time 1746134717.60 total 70.03 last 134.04) total: 10000 open/close in 71.60 seconds: 139.67 ops/second - unlinked 0 (time 1746134720 ; total 0 ; last 0) total: 10000 unlinks in 44 seconds: 227.272720 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg406-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg406-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg406-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.204.106@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.204.106@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg406-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg406-server: oleg406-server.virtnet: executing set_default_debug -1 all pdsh@oleg406-client: oleg406-server: ssh exited with exit code 1 Started lustre-OST0000 oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg406-server: oleg406-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg406-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 6 sec mount lustre on /mnt/lustre..... Starting client: oleg406-client.virtnet: -o user_xattr,flock 192.168.204.106@tcp:/lustre /mnt/lustre On OST0, 18357 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg406-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg406-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg406-server stop mds service on oleg406-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg406-server unloading modules via unload_modules_local on: 'oleg406-server' oleg406-server: oleg406-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg406-client: oleg406-client: ssh exited with exit code 2 pdsh@oleg406-client: oleg406-server: ssh exited with exit code 2 pdsh@oleg406-client: oleg406-client: ssh exited with exit code 2 pdsh@oleg406-client: oleg406-server: ssh exited with exit code 2 PASS 69 (1017s) == conf-sanity test 111: Adding large_dir with over 2GB directory ========================================================== 17:26:43 (1746134803) SKIP: conf-sanity test_111 Only applicable to ldiskfs-based MDTs SKIP 111 (1s) Stopping clients: oleg406-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg406-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg406-client: oleg406-server: ssh exited with exit code 2 oleg406-server: oleg406-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg406-server' oleg406-server: oleg406-server.virtnet: executing load_modules_local oleg406-server: Loading modules from /home/green/git/lustre-release/lustre oleg406-server: detected 4 online CPUs by sysfs oleg406-server: MODOPTS_LIBCFS= oleg406-server: Force libcfs to create 2 CPU partitions oleg406-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg406-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 1243 sec ========== 17:27:08 (1746134828) === conf-sanity: start cleanup 17:27:08 (1746134828) === === conf-sanity: finish cleanup 17:27:09 (1746134829) ===