-----============= acceptance-small: conf-sanity ============----- Thu May 1 18:59:16 EDT 2025 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg137-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex: No such file or directory excepting tests: 32b 32c 32newtarball 108a 112a 112b 113 117 119 121 122a 123aa 123ab 123ac 123ad 123ae 123af 123ag 123ah 123ahi 123F 123G 123H 126 129 132 133 135 136 137 150 152 153a 153b 153c 155 802a Stopping clients: oleg137-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg137-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg137-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg137-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg137-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg137-server oleg137-server: oleg137-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg137-server' oleg137-server: oleg137-server.virtnet: executing load_modules_local oleg137-server: Loading modules from /home/green/git/lustre-release/lustre oleg137-server: detected 4 online CPUs by sysfs oleg137-server: MODOPTS_LIBCFS= oleg137-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg137-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg137-server: oleg137-server.virtnet: executing set_default_debug -1 all pdsh@oleg137-client: oleg137-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg137-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg137-server: oleg137-server.virtnet: executing set_default_debug -1 all pdsh@oleg137-client: oleg137-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg137-server: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg137-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg137-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg137-server stop mds service on oleg137-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg137-server == conf-sanity test 45: long unlink handling in ptlrpcd == 19:00:58 (1746140458) start mds service on oleg137-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg137-server: oleg137-server.virtnet: executing set_default_debug -1 all pdsh@oleg137-client: oleg137-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg137-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg137-server: oleg137-server.virtnet: executing set_default_debug -1 all pdsh@oleg137-client: oleg137-server: ssh exited with exit code 1 Started lustre-OST0000 oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg137-client.virtnet: -o user_xattr,flock 192.168.201.137@tcp:/lustre /mnt/lustre setup single mount lustre success stop mds service on oleg137-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg137-server sleep 60 sec fail_loc=0x8000050f sleep 10 sec manual umount lustre on /mnt/lustre.... df: '/mnt/lustre': Cannot send after transport endpoint shutdown fail_loc=0x0 start mds service on oleg137-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg137-server: oleg137-server.virtnet: executing set_default_debug -1 all pdsh@oleg137-client: oleg137-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg137-client.virtnet: -o user_xattr,flock 192.168.201.137@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg137-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg137-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg137-server stop mds service on oleg137-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg137-server unloading modules via unload_modules_local on: 'oleg137-server' oleg137-server: oleg137-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg137-client: oleg137-server: ssh exited with exit code 2 pdsh@oleg137-client: oleg137-client: ssh exited with exit code 2 pdsh@oleg137-client: oleg137-client: ssh exited with exit code 2 pdsh@oleg137-client: oleg137-server: ssh exited with exit code 2 PASS 45 (188s) == conf-sanity test 69: replace an OST with the same index ========================================================== 19:04:06 (1746140646) start mds service on oleg137-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg137-server' oleg137-server: oleg137-server.virtnet: executing load_modules_local oleg137-server: Loading modules from /home/green/git/lustre-release/lustre oleg137-server: detected 4 online CPUs by sysfs oleg137-server: MODOPTS_LIBCFS= oleg137-server: Force libcfs to create 2 CPU partitions oleg137-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg137-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg137-server: oleg137-server.virtnet: executing set_default_debug -1 all pdsh@oleg137-client: oleg137-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg137-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg137-server: oleg137-server.virtnet: executing set_default_debug -1 all pdsh@oleg137-client: oleg137-server: ssh exited with exit code 1 Started lustre-OST0000 oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg137-client.virtnet: -o user_xattr,flock 192.168.201.137@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11872 inodes available. Want 99872. rc=0 - open/close 885 (time 1746140711.51 total 10.00 last 88.49) - open/close 1756 (time 1746140721.52 total 20.01 last 87.00) - open/close 2703 (time 1746140731.53 total 30.02 last 94.63) - open/close 3615 (time 1746140741.53 total 40.02 last 91.20) - open/close 4457 (time 1746140751.54 total 50.02 last 84.16) - open/close 5355 (time 1746140761.55 total 60.04 last 89.69) - open/close 6233 (time 1746140771.55 total 70.04 last 87.78) - open/close 7163 (time 1746140781.55 total 80.04 last 92.96) - open/close 8069 (time 1746140791.56 total 90.04 last 90.59) - open/close 8980 (time 1746140801.56 total 100.05 last 91.07) - open/close 9928 (time 1746140811.57 total 110.06 last 94.71) total: 10000 open/close in 110.96 seconds: 90.12 ops/second - unlinked 0 (time 1746140814 ; total 0 ; last 0) total: 10000 unlinks in 73 seconds: 136.986298 unlinks/second - open/close 851 (time 1746140900.51 total 10.01 last 85.04) - open/close 1800 (time 1746140910.51 total 20.01 last 94.89) - open/close 2738 (time 1746140920.52 total 30.02 last 93.70) - open/close 3581 (time 1746140930.52 total 40.02 last 84.30) - open/close 4419 (time 1746140940.54 total 50.03 last 83.66) - open/close 5258 (time 1746140950.55 total 60.04 last 83.82) - open/close 6150 (time 1746140960.55 total 70.05 last 89.15) - open/close 7073 (time 1746140970.56 total 80.06 last 92.22) - open/close 7988 (time 1746140980.57 total 90.07 last 91.39) - open/close 8824 (time 1746140990.58 total 100.08 last 83.54) - open/close 9851 (time 1746141000.58 total 110.08 last 102.65) total: 10000 open/close in 111.27 seconds: 89.87 ops/second - unlinked 0 (time 1746141003 ; total 0 ; last 0) total: 10000 unlinks in 54 seconds: 185.185181 unlinks/second - open/close 806 (time 1746141069.92 total 10.01 last 80.51) - open/close 1599 (time 1746141079.92 total 20.02 last 79.26) - open/close 2696 (time 1746141089.93 total 30.03 last 109.57) - open/close 3799 (time 1746141099.94 total 40.03 last 110.23) - open/close 4679 (time 1746141109.95 total 50.04 last 87.92) - open/close 5596 (time 1746141119.96 total 60.05 last 91.60) - open/close 6646 (time 1746141129.97 total 70.07 last 104.88) - open/close 7782 (time 1746141139.98 total 80.07 last 113.50) - open/close 8813 (time 1746141149.99 total 90.08 last 103.05) - open/close 9814 (time 1746141159.99 total 100.09 last 100.03) total: 10000 open/close in 101.87 seconds: 98.17 ops/second - unlinked 0 (time 1746141164 ; total 0 ; last 0) total: 10000 unlinks in 61 seconds: 163.934433 unlinks/second - open/close 1193 (time 1746141236.92 total 10.00 last 119.28) - open/close 2303 (time 1746141246.92 total 20.01 last 110.96) - open/close 3266 (time 1746141256.93 total 30.01 last 96.26) - open/close 4136 (time 1746141266.93 total 40.02 last 86.94) - open/close 5148 (time 1746141276.94 total 50.03 last 101.11) - open/close 6291 (time 1746141286.95 total 60.03 last 114.25) - open/close 7464 (time 1746141296.95 total 70.04 last 117.22) - open/close 8600 (time 1746141306.96 total 80.04 last 113.55) - open/close 9779 (time 1746141316.96 total 90.04 last 117.90) total: 10000 open/close in 91.62 seconds: 109.14 ops/second - unlinked 0 (time 1746141320 ; total 0 ; last 0) total: 10000 unlinks in 51 seconds: 196.078430 unlinks/second - open/close 1088 (time 1746141384.58 total 10.01 last 108.73) - open/close 2021 (time 1746141394.58 total 20.01 last 93.26) - open/close 3024 (time 1746141404.58 total 30.01 last 100.29) - open/close 4131 (time 1746141414.58 total 40.01 last 110.66) - open/close 5297 (time 1746141424.60 total 50.03 last 116.47) - open/close 6324 (time 1746141434.60 total 60.03 last 102.66) - open/close 7433 (time 1746141444.60 total 70.03 last 110.88) - open/close 8591 (time 1746141454.61 total 80.04 last 115.66) - open/close 9649 (time 1746141464.61 total 90.05 last 105.78) total: 10000 open/close in 93.32 seconds: 107.16 ops/second - unlinked 0 (time 1746141469 ; total 0 ; last 0) total: 10000 unlinks in 66 seconds: 151.515152 unlinks/second - open/close 1139 (time 1746141547.88 total 10.00 last 113.88) - open/close 2386 (time 1746141557.89 total 20.01 last 124.66) - open/close 3585 (time 1746141567.89 total 30.01 last 119.90) - open/close 5251 (time 1746141577.89 total 40.01 last 166.55) - open/close 6824 (time 1746141587.89 total 50.01 last 157.28) - open/close 7791 (time 1746141597.90 total 60.02 last 96.65) - open/close 8677 (time 1746141607.90 total 70.02 last 88.56) - open/close 9572 (time 1746141617.90 total 80.02 last 89.49) total: 10000 open/close in 84.05 seconds: 118.98 ops/second - unlinked 0 (time 1746141623 ; total 0 ; last 0) total: 10000 unlinks in 54 seconds: 185.185181 unlinks/second - open/close 1034 (time 1746141689.59 total 10.01 last 103.34) - open/close 2019 (time 1746141699.60 total 20.01 last 98.44) - open/close 3000 (time 1746141709.60 total 30.01 last 98.09) - open/close 4123 (time 1746141719.60 total 40.01 last 112.28) - open/close 5008 (time 1746141729.61 total 50.02 last 88.45) - open/close 5871 (time 1746141739.61 total 60.02 last 86.30) - open/close 6792 (time 1746141749.63 total 70.05 last 91.87) - open/close 7699 (time 1746141759.64 total 80.05 last 90.67) - open/close 8642 (time 1746141769.64 total 90.06 last 94.21) - open/close 9556 (time 1746141779.67 total 100.09 last 91.13) total: 10000 open/close in 105.18 seconds: 95.08 ops/second - unlinked 0 (time 1746141786 ; total 0 ; last 0) total: 10000 unlinks in 59 seconds: 169.491531 unlinks/second - open/close 1144 (time 1746141857.29 total 10.00 last 114.38) - open/close 2224 (time 1746141867.29 total 20.00 last 107.98) - open/close 3239 (time 1746141877.30 total 30.01 last 101.42) - open/close 4239 (time 1746141887.30 total 40.02 last 99.94) - open/close 5126 (time 1746141897.31 total 50.02 last 88.65) - open/close 6084 (time 1746141907.32 total 60.03 last 95.73) - open/close 7058 (time 1746141917.32 total 70.04 last 97.32) - open/close 8116 (time 1746141927.33 total 80.05 last 105.71) - open/close 9113 (time 1746141937.34 total 90.05 last 99.66) total: 10000 open/close in 97.76 seconds: 102.29 ops/second - unlinked 0 (time 1746141946 ; total 0 ; last 0) total: 10000 unlinks in 51 seconds: 196.078430 unlinks/second - open/close 1092 (time 1746142009.58 total 10.00 last 109.15) - open/close 2428 (time 1746142019.58 total 20.01 last 133.56) - open/close 3473 (time 1746142029.60 total 30.02 last 104.32) - open/close 4397 (time 1746142039.60 total 40.03 last 92.39) - open/close 5391 (time 1746142049.60 total 50.03 last 99.39) - open/close 6312 (time 1746142059.62 total 60.05 last 91.93) - open/close 7250 (time 1746142069.62 total 70.05 last 93.79) - open/close 8189 (time 1746142079.62 total 80.05 last 93.90) - open/close 8926 (time 1746142089.63 total 90.06 last 73.64) - open/close 9889 (time 1746142099.64 total 100.07 last 96.19) total: 10000 open/close in 101.14 seconds: 98.87 ops/second - unlinked 0 (time 1746142103 ; total 1 ; last 1) total: 10000 unlinks in 65 seconds: 153.846161 unlinks/second - open/close 766 (time 1746142179.88 total 10.02 last 76.47) - open/close 1518 (time 1746142189.88 total 20.02 last 75.18) - open/close 2588 (time 1746142199.89 total 30.03 last 106.94) - open/close 3590 (time 1746142209.90 total 40.03 last 100.14) - open/close 4578 (time 1746142219.90 total 50.04 last 98.76) - open/close 5552 (time 1746142229.91 total 60.04 last 97.33) - open/close 6509 (time 1746142239.91 total 70.05 last 95.64) - open/close 7475 (time 1746142249.91 total 80.05 last 96.59) - open/close 8403 (time 1746142259.92 total 90.06 last 92.73) - open/close 9246 (time 1746142269.93 total 100.06 last 84.26) - open/close 9990 (time 1746142279.94 total 110.07 last 74.32) total: 10000 open/close in 110.18 seconds: 90.76 ops/second - unlinked 0 (time 1746142282 ; total 0 ; last 0) total: 10000 unlinks in 61 seconds: 163.934433 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg137-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg137-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg137-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.201.137@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.201.137@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg137-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg137-server: oleg137-server.virtnet: executing set_default_debug -1 all pdsh@oleg137-client: oleg137-server: ssh exited with exit code 1 Started lustre-OST0000 oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg137-server: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg137-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg137-client.virtnet: -o user_xattr,flock 192.168.201.137@tcp:/lustre /mnt/lustre On OST0, 13773 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg137-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg137-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg137-server stop mds service on oleg137-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg137-server unloading modules via unload_modules_local on: 'oleg137-server' oleg137-server: oleg137-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg137-client: oleg137-server: ssh exited with exit code 2 pdsh@oleg137-client: oleg137-client: ssh exited with exit code 2 pdsh@oleg137-client: oleg137-client: ssh exited with exit code 2 pdsh@oleg137-client: oleg137-server: ssh exited with exit code 2 PASS 69 (1755s) == conf-sanity test 111: Adding large_dir with over 2GB directory ========================================================== 19:33:21 (1746142401) SKIP: conf-sanity test_111 Only applicable to ldiskfs-based MDTs SKIP 111 (2s) Stopping clients: oleg137-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg137-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg137-client: oleg137-server: ssh exited with exit code 2 oleg137-server: oleg137-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg137-server' oleg137-server: oleg137-server.virtnet: executing load_modules_local oleg137-server: Loading modules from /home/green/git/lustre-release/lustre oleg137-server: detected 4 online CPUs by sysfs oleg137-server: MODOPTS_LIBCFS= oleg137-server: Force libcfs to create 2 CPU partitions oleg137-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg137-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 2090 sec ========== 19:34:07 (1746142447) === conf-sanity: start cleanup 19:34:08 (1746142448) === === conf-sanity: finish cleanup 19:34:10 (1746142450) ===