-----============= acceptance-small: conf-sanity ============----- Thu May 1 19:56:01 EDT 2025 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg119-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex: No such file or directory excepting tests: 32b 32c 32newtarball 108a 112a 112b 113 117 119 121 122a 123aa 123ab 123ac 123ad 123ae 123af 123ag 123ah 123ahi 123F 123G 123H 126 129 132 133 135 136 137 150 152 153a 153b 153c 155 802a Stopping clients: oleg119-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg119-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg119-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg119-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg119-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg119-server oleg119-server: oleg119-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg119-server' oleg119-server: oleg119-server.virtnet: executing load_modules_local oleg119-server: Loading modules from /home/green/git/lustre-release/lustre oleg119-server: detected 4 online CPUs by sysfs oleg119-server: MODOPTS_LIBCFS= oleg119-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg119-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg119-server: oleg119-server.virtnet: executing set_default_debug -1 all pdsh@oleg119-client: oleg119-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg119-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg119-server: oleg119-server.virtnet: executing set_default_debug -1 all pdsh@oleg119-client: oleg119-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg119-server: oleg119-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg119-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg119-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg119-server stop mds service on oleg119-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg119-server == conf-sanity test 45: long unlink handling in ptlrpcd == 19:57:32 (1746143852) start mds service on oleg119-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg119-server: oleg119-server.virtnet: executing set_default_debug -1 all pdsh@oleg119-client: oleg119-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg119-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg119-server: oleg119-server.virtnet: executing set_default_debug -1 all pdsh@oleg119-client: oleg119-server: ssh exited with exit code 1 Started lustre-OST0000 oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg119-client.virtnet: -o user_xattr,flock 192.168.201.119@tcp:/lustre /mnt/lustre setup single mount lustre success stop mds service on oleg119-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg119-server sleep 60 sec fail_loc=0x8000050f sleep 10 sec manual umount lustre on /mnt/lustre.... df: '/mnt/lustre': Cannot send after transport endpoint shutdown fail_loc=0x0 start mds service on oleg119-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg119-server: oleg119-server.virtnet: executing set_default_debug -1 all pdsh@oleg119-client: oleg119-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg119-client.virtnet: -o user_xattr,flock 192.168.201.119@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg119-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg119-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg119-server stop mds service on oleg119-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg119-server unloading modules via unload_modules_local on: 'oleg119-server' oleg119-server: oleg119-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg119-client: oleg119-client: ssh exited with exit code 2 pdsh@oleg119-client: oleg119-server: ssh exited with exit code 2 pdsh@oleg119-client: oleg119-client: ssh exited with exit code 2 pdsh@oleg119-client: oleg119-server: ssh exited with exit code 2 PASS 45 (142s) == conf-sanity test 69: replace an OST with the same index ========================================================== 19:59:55 (1746143995) start mds service on oleg119-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg119-server' oleg119-server: oleg119-server.virtnet: executing load_modules_local oleg119-server: Loading modules from /home/green/git/lustre-release/lustre oleg119-server: detected 4 online CPUs by sysfs oleg119-server: MODOPTS_LIBCFS= oleg119-server: Force libcfs to create 2 CPU partitions oleg119-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg119-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg119-server: oleg119-server.virtnet: executing set_default_debug -1 all pdsh@oleg119-client: oleg119-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg119-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg119-server: oleg119-server.virtnet: executing set_default_debug -1 all pdsh@oleg119-client: oleg119-server: ssh exited with exit code 1 Started lustre-OST0000 oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg119-client.virtnet: -o user_xattr,flock 192.168.201.119@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11872 inodes available. Want 99872. rc=0 - open/close 991 (time 1746144049.61 total 10.01 last 99.05) - open/close 2159 (time 1746144059.62 total 20.01 last 116.71) - open/close 3226 (time 1746144069.62 total 30.02 last 106.64) - open/close 4361 (time 1746144079.63 total 40.03 last 113.39) - open/close 5553 (time 1746144089.63 total 50.03 last 119.18) - open/close 6658 (time 1746144099.64 total 60.03 last 110.46) - open/close 7879 (time 1746144109.64 total 70.04 last 122.05) - open/close 9009 (time 1746144119.64 total 80.04 last 112.98) total: 10000 open/close in 89.10 seconds: 112.23 ops/second - unlinked 0 (time 1746144130 ; total 0 ; last 0) total: 10000 unlinks in 57 seconds: 175.438599 unlinks/second - open/close 1028 (time 1746144199.79 total 10.00 last 102.78) - open/close 2326 (time 1746144209.79 total 20.01 last 129.75) - open/close 3562 (time 1746144219.80 total 30.01 last 123.49) - open/close 4737 (time 1746144229.81 total 40.02 last 117.38) - open/close 6042 (time 1746144239.81 total 50.03 last 130.49) - open/close 7153 (time 1746144249.81 total 60.03 last 111.05) - open/close 8321 (time 1746144259.82 total 70.03 last 116.76) - open/close 9600 (time 1746144269.82 total 80.04 last 127.84) total: 10000 open/close in 83.76 seconds: 119.39 ops/second - unlinked 0 (time 1746144275 ; total 0 ; last 0) total: 10000 unlinks in 53 seconds: 188.679245 unlinks/second - open/close 1087 (time 1746144340.72 total 10.00 last 108.65) - open/close 2138 (time 1746144350.73 total 20.01 last 105.07) - open/close 3198 (time 1746144360.73 total 30.01 last 105.99) - open/close 4167 (time 1746144370.73 total 40.01 last 96.86) - open/close 5036 (time 1746144380.73 total 50.01 last 86.89) - open/close 5932 (time 1746144390.74 total 60.02 last 89.51) - open/close 6841 (time 1746144400.75 total 70.03 last 90.84) - open/close 7813 (time 1746144410.76 total 80.04 last 97.08) - open/close 8643 (time 1746144420.77 total 90.05 last 82.94) - open/close 9699 (time 1746144430.77 total 100.05 last 105.53) total: 10000 open/close in 102.51 seconds: 97.55 ops/second - unlinked 0 (time 1746144435 ; total 0 ; last 0) total: 10000 unlinks in 56 seconds: 178.571426 unlinks/second - open/close 859 (time 1746144504.43 total 10.01 last 85.81) - open/close 1768 (time 1746144514.43 total 20.01 last 90.89) - open/close 2778 (time 1746144524.43 total 30.01 last 101.00) - open/close 3951 (time 1746144534.44 total 40.02 last 117.19) - open/close 4938 (time 1746144544.45 total 50.03 last 98.63) - open/close 5865 (time 1746144554.46 total 60.04 last 92.61) - open/close 6869 (time 1746144564.46 total 70.04 last 100.39) - open/close 7773 (time 1746144574.46 total 80.04 last 90.36) - open/close 8803 (time 1746144584.47 total 90.05 last 102.97) - open/close 9960 (time 1746144594.47 total 100.05 last 115.67) total: 10000 open/close in 100.41 seconds: 99.59 ops/second - unlinked 0 (time 1746144596 ; total 0 ; last 0) total: 10000 unlinks in 48 seconds: 208.333328 unlinks/second - open/close 1267 (time 1746144656.37 total 10.01 last 126.62) - open/close 2473 (time 1746144666.37 total 20.01 last 120.52) - open/close 3567 (time 1746144676.37 total 30.01 last 109.40) - open/close 4652 (time 1746144686.38 total 40.02 last 108.45) - open/close 5811 (time 1746144696.38 total 50.02 last 115.90) - open/close 6843 (time 1746144706.39 total 60.03 last 103.09) - open/close 8040 (time 1746144716.39 total 70.03 last 119.65) - open/close 9083 (time 1746144726.40 total 80.04 last 104.22) total: 10000 open/close in 88.65 seconds: 112.80 ops/second - unlinked 0 (time 1746144736 ; total 0 ; last 0) total: 10000 unlinks in 62 seconds: 161.290329 unlinks/second - open/close 1180 (time 1746144810.02 total 10.00 last 117.94) - open/close 2285 (time 1746144820.02 total 20.01 last 110.45) - open/close 3258 (time 1746144830.03 total 30.02 last 97.17) - open/close 4353 (time 1746144840.04 total 40.03 last 109.42) - open/close 5533 (time 1746144850.04 total 50.03 last 117.97) - open/close 6609 (time 1746144860.05 total 60.04 last 107.50) - open/close 7657 (time 1746144870.06 total 70.05 last 104.75) - open/close 8802 (time 1746144880.08 total 80.06 last 114.30) - open/close 9967 (time 1746144890.08 total 90.07 last 116.42) total: 10000 open/close in 90.40 seconds: 110.62 ops/second - unlinked 0 (time 1746144892 ; total 0 ; last 0) total: 10000 unlinks in 51 seconds: 196.078430 unlinks/second - open/close 1181 (time 1746144955.10 total 10.00 last 118.04) - open/close 2457 (time 1746144965.10 total 20.00 last 127.60) - open/close 3713 (time 1746144975.11 total 30.01 last 125.56) - open/close 4848 (time 1746144985.12 total 40.02 last 113.38) - open/close 6147 (time 1746144995.12 total 50.02 last 129.83) - open/close 7388 (time 1746145005.12 total 60.03 last 124.06) - open/close 8401 (time 1746145015.13 total 70.03 last 101.29) - open/close 9352 (time 1746145025.13 total 80.04 last 95.04) total: 10000 open/close in 86.61 seconds: 115.46 ops/second - unlinked 0 (time 1746145033 ; total 0 ; last 0) total: 10000 unlinks in 64 seconds: 156.250000 unlinks/second - open/close 1120 (time 1746145109.62 total 10.00 last 111.97) - open/close 2119 (time 1746145119.64 total 20.02 last 99.74) - open/close 3050 (time 1746145129.64 total 30.02 last 93.09) - open/close 4022 (time 1746145139.64 total 40.02 last 97.17) - open/close 5078 (time 1746145149.66 total 50.04 last 105.41) - open/close 6026 (time 1746145159.67 total 60.05 last 94.70) - open/close 7029 (time 1746145169.67 total 70.05 last 100.28) - open/close 7983 (time 1746145179.68 total 80.06 last 95.32) - open/close 8887 (time 1746145189.69 total 90.07 last 90.34) - open/close 9862 (time 1746145199.69 total 100.07 last 97.47) total: 10000 open/close in 101.35 seconds: 98.67 ops/second - unlinked 0 (time 1746145203 ; total 0 ; last 0) total: 10000 unlinks in 56 seconds: 178.571426 unlinks/second - open/close 1180 (time 1746145271.44 total 10.00 last 117.98) - open/close 2496 (time 1746145281.45 total 20.01 last 131.49) - open/close 3752 (time 1746145291.45 total 30.01 last 125.59) - open/close 4769 (time 1746145301.47 total 40.03 last 101.51) - open/close 6001 (time 1746145311.48 total 50.04 last 123.06) - open/close 6994 (time 1746145321.48 total 60.04 last 99.27) - open/close 7969 (time 1746145331.48 total 70.05 last 97.48) - open/close 8920 (time 1746145341.49 total 80.05 last 95.06) - open/close 9983 (time 1746145351.49 total 90.05 last 106.26) total: 10000 open/close in 90.20 seconds: 110.87 ops/second - unlinked 0 (time 1746145353 ; total 0 ; last 0) total: 10000 unlinks in 53 seconds: 188.679245 unlinks/second - open/close 1195 (time 1746145418.93 total 10.01 last 119.39) - open/close 2243 (time 1746145428.93 total 20.01 last 104.77) - open/close 3384 (time 1746145438.93 total 30.01 last 114.08) - open/close 4438 (time 1746145448.93 total 40.02 last 105.38) - open/close 5414 (time 1746145458.94 total 50.02 last 97.55) - open/close 6497 (time 1746145468.95 total 60.03 last 108.24) - open/close 7755 (time 1746145478.95 total 70.03 last 125.76) - open/close 8727 (time 1746145488.95 total 80.03 last 97.19) - open/close 9731 (time 1746145498.96 total 90.04 last 100.27) total: 10000 open/close in 92.50 seconds: 108.11 ops/second - unlinked 0 (time 1746145503 ; total 0 ; last 0) total: 10000 unlinks in 54 seconds: 185.185181 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg119-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg119-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg119-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.201.119@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.201.119@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg119-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg119-server: oleg119-server.virtnet: executing set_default_debug -1 all pdsh@oleg119-client: oleg119-server: ssh exited with exit code 1 Started lustre-OST0000 oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg119-server: oleg119-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg119-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg119-client.virtnet: -o user_xattr,flock 192.168.201.119@tcp:/lustre /mnt/lustre On OST0, 13005 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg119-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg119-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg119-server stop mds service on oleg119-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg119-server unloading modules via unload_modules_local on: 'oleg119-server' oleg119-server: oleg119-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg119-client: oleg119-client: ssh exited with exit code 2 pdsh@oleg119-client: oleg119-server: ssh exited with exit code 2 pdsh@oleg119-client: oleg119-client: ssh exited with exit code 2 pdsh@oleg119-client: oleg119-server: ssh exited with exit code 2 PASS 69 (1617s) == conf-sanity test 111: Adding large_dir with over 2GB directory ========================================================== 20:26:52 (1746145612) SKIP: conf-sanity test_111 Only applicable to ldiskfs-based MDTs SKIP 111 (2s) Stopping clients: oleg119-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg119-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg119-client: oleg119-server: ssh exited with exit code 2 oleg119-server: oleg119-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg119-server' oleg119-server: oleg119-server.virtnet: executing load_modules_local oleg119-server: Loading modules from /home/green/git/lustre-release/lustre oleg119-server: detected 4 online CPUs by sysfs oleg119-server: MODOPTS_LIBCFS= oleg119-server: Force libcfs to create 2 CPU partitions oleg119-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg119-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 1887 sec ========== 20:27:29 (1746145649) === conf-sanity: start cleanup 20:27:30 (1746145650) === === conf-sanity: finish cleanup 20:27:32 (1746145652) ===