-----============= acceptance-small: conf-sanity ============----- Thu May 1 15:26:53 EDT 2025 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg210-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex: No such file or directory excepting tests: 32b 32c 32newtarball 108a 112a 112b 113 117 119 121 122a 123aa 123ab 123ac 123ad 123ae 123af 123ag 123ah 123ahi 123F 123G 123H 126 129 132 133 135 136 137 150 152 153a 153b 153c 155 802a Stopping clients: oleg210-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg210-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg210-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg210-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg210-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg210-server oleg210-server: oleg210-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg210-server' oleg210-server: oleg210-server.virtnet: executing load_modules_local oleg210-server: Loading modules from /home/green/git/lustre-release/lustre oleg210-server: detected 4 online CPUs by sysfs oleg210-server: MODOPTS_LIBCFS= oleg210-server: Force libcfs to create 2 CPU partitions oleg210-server: libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg210-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg210-server: oleg210-server.virtnet: executing set_default_debug -1 all pdsh@oleg210-client: oleg210-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg210-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg210-server: oleg210-server.virtnet: executing set_default_debug -1 all pdsh@oleg210-client: oleg210-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg210-server: oleg210-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg210-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg210-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg210-server stop mds service on oleg210-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg210-server == conf-sanity test 45: long unlink handling in ptlrpcd == 15:28:14 (1746127694) start mds service on oleg210-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg210-server: oleg210-server.virtnet: executing set_default_debug -1 all pdsh@oleg210-client: oleg210-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg210-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg210-server: oleg210-server.virtnet: executing set_default_debug -1 all pdsh@oleg210-client: oleg210-server: ssh exited with exit code 1 Started lustre-OST0000 oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg210-client.virtnet: -o user_xattr,flock 192.168.202.110@tcp:/lustre /mnt/lustre setup single mount lustre success stop mds service on oleg210-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg210-server sleep 60 sec fail_loc=0x8000050f sleep 10 sec manual umount lustre on /mnt/lustre.... df: '/mnt/lustre': Cannot send after transport endpoint shutdown fail_loc=0x0 start mds service on oleg210-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg210-server: oleg210-server.virtnet: executing set_default_debug -1 all pdsh@oleg210-client: oleg210-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg210-client.virtnet: -o user_xattr,flock 192.168.202.110@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg210-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg210-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg210-server stop mds service on oleg210-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg210-server unloading modules via unload_modules_local on: 'oleg210-server' oleg210-server: oleg210-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg210-client: oleg210-client: ssh exited with exit code 2 pdsh@oleg210-client: oleg210-server: ssh exited with exit code 2 pdsh@oleg210-client: oleg210-client: ssh exited with exit code 2 pdsh@oleg210-client: oleg210-server: ssh exited with exit code 2 PASS 45 (135s) == conf-sanity test 69: replace an OST with the same index ========================================================== 15:30:29 (1746127829) start mds service on oleg210-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg210-server' oleg210-server: oleg210-server.virtnet: executing load_modules_local oleg210-server: Loading modules from /home/green/git/lustre-release/lustre oleg210-server: detected 4 online CPUs by sysfs oleg210-server: MODOPTS_LIBCFS= oleg210-server: Force libcfs to create 2 CPU partitions oleg210-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg210-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg210-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg210-server: oleg210-server.virtnet: executing set_default_debug -1 all pdsh@oleg210-client: oleg210-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg210-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg210-server: oleg210-server.virtnet: executing set_default_debug -1 all pdsh@oleg210-client: oleg210-server: ssh exited with exit code 1 Started lustre-OST0000 oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg210-client.virtnet: -o user_xattr,flock 192.168.202.110@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11872 inodes available. Want 99872. rc=0 - open/close 1474 (time 1746127874.51 total 10.00 last 147.38) - open/close 2871 (time 1746127884.51 total 20.00 last 139.65) - open/close 4140 (time 1746127894.52 total 30.01 last 126.81) - open/close 5392 (time 1746127904.54 total 40.03 last 125.01) - open/close 6661 (time 1746127914.54 total 50.03 last 126.83) - open/close 7933 (time 1746127924.55 total 60.04 last 127.14) - open/close 9138 (time 1746127934.55 total 70.05 last 120.39) total: 10000 open/close in 77.96 seconds: 128.27 ops/second - unlinked 0 (time 1746127944 ; total 0 ; last 0) total: 10000 unlinks in 42 seconds: 238.095245 unlinks/second - open/close 1248 (time 1746127998.19 total 10.00 last 124.78) - open/close 2492 (time 1746128008.20 total 20.01 last 124.30) - open/close 3782 (time 1746128018.20 total 30.01 last 128.97) - open/close 4918 (time 1746128028.20 total 40.02 last 113.54) - open/close 6133 (time 1746128038.22 total 50.03 last 121.33) - open/close 7229 (time 1746128048.22 total 60.04 last 109.55) - open/close 8405 (time 1746128058.23 total 70.04 last 117.55) - open/close 9592 (time 1746128068.23 total 80.04 last 118.69) total: 10000 open/close in 82.84 seconds: 120.72 ops/second - unlinked 0 (time 1746128072 ; total 0 ; last 0) total: 10000 unlinks in 56 seconds: 178.571426 unlinks/second - open/close 1047 (time 1746128140.66 total 10.00 last 104.66) - open/close 2182 (time 1746128150.66 total 20.01 last 113.43) - open/close 3487 (time 1746128160.66 total 30.01 last 130.48) - open/close 4717 (time 1746128170.67 total 40.02 last 122.93) - open/close 6038 (time 1746128180.67 total 50.02 last 132.05) - open/close 7484 (time 1746128190.68 total 60.03 last 144.48) - open/close 8652 (time 1746128200.69 total 70.04 last 116.72) - open/close 9813 (time 1746128210.70 total 80.04 last 116.00) total: 10000 open/close in 81.79 seconds: 122.26 ops/second - unlinked 0 (time 1746128214 ; total 0 ; last 0) total: 10000 unlinks in 33 seconds: 303.030304 unlinks/second - open/close 2129 (time 1746128258.92 total 10.00 last 212.85) - open/close 3895 (time 1746128268.92 total 20.00 last 176.58) - open/close 5726 (time 1746128278.93 total 30.01 last 183.00) - open/close 7227 (time 1746128288.93 total 40.01 last 150.04) - open/close 9177 (time 1746128298.94 total 50.02 last 194.96) total: 10000 open/close in 53.52 seconds: 186.85 ops/second - unlinked 0 (time 1746128303 ; total 0 ; last 0) total: 10000 unlinks in 36 seconds: 277.777771 unlinks/second - open/close 1502 (time 1746128350.47 total 10.00 last 150.17) - open/close 2951 (time 1746128360.47 total 20.00 last 144.89) - open/close 4225 (time 1746128370.48 total 30.01 last 127.36) - open/close 5446 (time 1746128380.50 total 40.03 last 121.85) - open/close 6550 (time 1746128390.50 total 50.03 last 110.39) - open/close 7619 (time 1746128400.52 total 60.04 last 106.71) - open/close 8731 (time 1746128410.52 total 70.05 last 111.13) - open/close 9936 (time 1746128420.52 total 80.05 last 120.50) total: 10000 open/close in 80.54 seconds: 124.15 ops/second - unlinked 0 (time 1746128423 ; total 0 ; last 0) total: 10000 unlinks in 58 seconds: 172.413788 unlinks/second - open/close 981 (time 1746128493.75 total 10.02 last 97.94) - open/close 1981 (time 1746128503.77 total 20.03 last 99.82) - open/close 2944 (time 1746128513.77 total 30.04 last 96.26) - open/close 3853 (time 1746128523.81 total 40.08 last 90.54) - open/close 4815 (time 1746128533.83 total 50.09 last 96.06) - open/close 5841 (time 1746128543.83 total 60.10 last 102.53) - open/close 6882 (time 1746128553.84 total 70.11 last 104.00) - open/close 7890 (time 1746128563.85 total 80.12 last 100.73) - open/close 8843 (time 1746128573.87 total 90.14 last 95.08) - open/close 9869 (time 1746128583.87 total 100.14 last 102.60) total: 10000 open/close in 101.30 seconds: 98.72 ops/second - unlinked 0 (time 1746128587 ; total 0 ; last 0) total: 10000 unlinks in 71 seconds: 140.845078 unlinks/second - open/close 954 (time 1746128670.76 total 10.00 last 95.39) - open/close 1873 (time 1746128680.77 total 20.00 last 91.88) - open/close 2741 (time 1746128690.77 total 30.00 last 86.78) - open/close 3679 (time 1746128700.77 total 40.01 last 93.78) - open/close 4619 (time 1746128710.77 total 50.01 last 93.98) - open/close 5512 (time 1746128720.78 total 60.02 last 89.22) - open/close 6391 (time 1746128730.78 total 70.02 last 87.89) - open/close 7159 (time 1746128740.79 total 80.03 last 76.75) - open/close 8031 (time 1746128750.80 total 90.03 last 87.15) - open/close 8978 (time 1746128760.80 total 100.04 last 94.66) total: 10000 open/close in 109.43 seconds: 91.38 ops/second - unlinked 0 (time 1746128772 ; total 0 ; last 0) total: 10000 unlinks in 55 seconds: 181.818176 unlinks/second - open/close 1201 (time 1746128839.64 total 10.00 last 120.09) - open/close 2498 (time 1746128849.65 total 20.01 last 129.60) - open/close 3745 (time 1746128859.65 total 30.01 last 124.69) - open/close 5039 (time 1746128869.65 total 40.01 last 129.37) - open/close 6169 (time 1746128879.66 total 50.02 last 112.91) - open/close 7417 (time 1746128889.69 total 60.05 last 124.43) - open/close 8456 (time 1746128899.69 total 70.05 last 103.89) - open/close 9506 (time 1746128909.69 total 80.05 last 104.96) total: 10000 open/close in 84.03 seconds: 119.00 ops/second - unlinked 0 (time 1746128915 ; total 0 ; last 0) total: 10000 unlinks in 55 seconds: 181.818176 unlinks/second - open/close 1241 (time 1746128982.48 total 10.00 last 124.08) - open/close 2547 (time 1746128992.49 total 20.01 last 130.50) - open/close 3863 (time 1746129002.49 total 30.01 last 131.57) - open/close 5253 (time 1746129012.49 total 40.01 last 138.97) - open/close 6510 (time 1746129022.50 total 50.02 last 125.63) - open/close 7662 (time 1746129032.50 total 60.02 last 115.19) - open/close 8880 (time 1746129042.50 total 70.03 last 121.73) total: 10000 open/close in 79.46 seconds: 125.84 ops/second - unlinked 0 (time 1746129053 ; total 0 ; last 0) total: 10000 unlinks in 47 seconds: 212.765961 unlinks/second - open/close 1183 (time 1746129112.67 total 10.01 last 118.17) - open/close 2343 (time 1746129122.67 total 20.01 last 115.96) - open/close 3547 (time 1746129132.68 total 30.03 last 120.25) - open/close 4740 (time 1746129142.69 total 40.03 last 119.21) - open/close 6078 (time 1746129152.70 total 50.04 last 133.74) - open/close 7564 (time 1746129162.70 total 60.05 last 148.50) - open/close 8998 (time 1746129172.71 total 70.05 last 143.34) total: 10000 open/close in 77.05 seconds: 129.79 ops/second - unlinked 0 (time 1746129181 ; total 0 ; last 0) total: 10000 unlinks in 42 seconds: 238.095245 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg210-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg210-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg210-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.202.110@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.202.110@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg210-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg210-server: oleg210-server.virtnet: executing set_default_debug -1 all pdsh@oleg210-client: oleg210-server: ssh exited with exit code 1 Started lustre-OST0000 oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg210-server: oleg210-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg210-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 6 sec mount lustre on /mnt/lustre..... Starting client: oleg210-client.virtnet: -o user_xattr,flock 192.168.202.110@tcp:/lustre /mnt/lustre On OST0, 17693 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg210-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg210-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg210-server stop mds service on oleg210-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg210-server unloading modules via unload_modules_local on: 'oleg210-server' oleg210-server: oleg210-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg210-client: oleg210-client: ssh exited with exit code 2 pdsh@oleg210-client: oleg210-server: ssh exited with exit code 2 pdsh@oleg210-client: oleg210-client: ssh exited with exit code 2 pdsh@oleg210-client: oleg210-server: ssh exited with exit code 2 PASS 69 (1442s) == conf-sanity test 111: Adding large_dir with over 2GB directory ========================================================== 15:54:31 (1746129271) SKIP: conf-sanity test_111 Only applicable to ldiskfs-based MDTs SKIP 111 (2s) Stopping clients: oleg210-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg210-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg210-client: oleg210-server: ssh exited with exit code 2 oleg210-server: oleg210-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg210-server' oleg210-server: oleg210-server.virtnet: executing load_modules_local oleg210-server: Loading modules from /home/green/git/lustre-release/lustre oleg210-server: detected 4 online CPUs by sysfs oleg210-server: MODOPTS_LIBCFS= oleg210-server: Force libcfs to create 2 CPU partitions oleg210-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg210-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 1690 sec ========== 15:55:03 (1746129303) === conf-sanity: start cleanup 15:55:04 (1746129304) === === conf-sanity: finish cleanup 15:55:05 (1746129305) ===