-----============= acceptance-small: conf-sanity ============----- Wed Apr 30 23:52:54 EDT 2025 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg144-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex: No such file or directory excepting tests: 32b 32c 32newtarball 108a 112a 112b 113 117 119 121 122a 123aa 123ab 123ac 123ad 123ae 123af 123ag 123ah 123ahi 123F 123G 123H 126 129 132 133 135 136 137 150 152 153a 153b 153c 155 802a Stopping clients: oleg144-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg144-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg144-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg144-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg144-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg144-server oleg144-server: oleg144-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_refcnt: could not read integer from '/sys/module/acpi_cpufreq/refcnt': 'No such device' loading modules on: 'oleg144-server' oleg144-server: oleg144-server.virtnet: executing load_modules_local oleg144-server: Loading modules from /home/green/git/lustre-release/lustre oleg144-server: detected 4 online CPUs by sysfs oleg144-server: MODOPTS_LIBCFS= oleg144-server: Force libcfs to create 2 CPU partitions oleg144-server: libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg144-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg144-server: oleg144-server.virtnet: executing set_default_debug -1 all pdsh@oleg144-client: oleg144-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg144-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg144-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg144-server: oleg144-server.virtnet: executing set_default_debug -1 all pdsh@oleg144-client: oleg144-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg144-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg144-server: oleg144-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg144-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg144-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg144-server stop mds service on oleg144-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg144-server == conf-sanity test 45: long unlink handling in ptlrpcd == 23:54:49 (1746071689) start mds service on oleg144-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg144-server: oleg144-server.virtnet: executing set_default_debug -1 all pdsh@oleg144-client: oleg144-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg144-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg144-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg144-server: oleg144-server.virtnet: executing set_default_debug -1 all pdsh@oleg144-client: oleg144-server: ssh exited with exit code 1 Started lustre-OST0000 oleg144-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg144-client.virtnet: -o user_xattr,flock 192.168.201.144@tcp:/lustre /mnt/lustre setup single mount lustre success stop mds service on oleg144-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg144-server sleep 60 sec fail_loc=0x8000050f sleep 10 sec manual umount lustre on /mnt/lustre.... df: '/mnt/lustre': Cannot send after transport endpoint shutdown fail_loc=0x0 start mds service on oleg144-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg144-server: oleg144-server.virtnet: executing set_default_debug -1 all pdsh@oleg144-client: oleg144-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg144-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg144-client.virtnet: -o user_xattr,flock 192.168.201.144@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg144-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg144-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg144-server stop mds service on oleg144-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg144-server unloading modules via unload_modules_local on: 'oleg144-server' oleg144-server: oleg144-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg144-client: oleg144-client: ssh exited with exit code 2 pdsh@oleg144-client: oleg144-server: ssh exited with exit code 2 pdsh@oleg144-client: oleg144-client: ssh exited with exit code 2 pdsh@oleg144-client: oleg144-server: ssh exited with exit code 2 PASS 45 (183s) == conf-sanity test 69: replace an OST with the same index ========================================================== 23:57:53 (1746071873) start mds service on oleg144-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg144-server' oleg144-server: oleg144-server.virtnet: executing load_modules_local oleg144-server: Loading modules from /home/green/git/lustre-release/lustre oleg144-server: detected 4 online CPUs by sysfs oleg144-server: MODOPTS_LIBCFS= oleg144-server: Force libcfs to create 2 CPU partitions oleg144-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg144-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg144-server: oleg144-server.virtnet: executing set_default_debug -1 all pdsh@oleg144-client: oleg144-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg144-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg144-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg144-server: oleg144-server.virtnet: executing set_default_debug -1 all pdsh@oleg144-client: oleg144-server: ssh exited with exit code 1 Started lustre-OST0000 oleg144-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg144-client.virtnet: -o user_xattr,flock 192.168.201.144@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11872 inodes available. Want 99872. rc=0 - open/close 750 (time 1746071937.64 total 10.00 last 74.99) - open/close 1484 (time 1746071947.64 total 20.01 last 73.34) - open/close 2190 (time 1746071957.66 total 30.02 last 70.51) - open/close 2979 (time 1746071967.66 total 40.03 last 78.86) - open/close 3791 (time 1746071977.67 total 50.04 last 81.13) - open/close 4532 (time 1746071987.68 total 60.05 last 74.03) - open/close 5323 (time 1746071997.69 total 70.05 last 79.02) - open/close 6135 (time 1746072007.70 total 80.06 last 81.14) - open/close 6909 (time 1746072017.72 total 90.09 last 77.20) - open/close 7783 (time 1746072027.73 total 100.10 last 87.33) - open/close 8541 (time 1746072037.73 total 110.10 last 75.76) - open/close 9284 (time 1746072047.75 total 120.12 last 74.19) total: 10000 open/close in 130.06 seconds: 76.89 ops/second - unlinked 0 (time 1746072060 ; total 0 ; last 0) total: 10000 unlinks in 72 seconds: 138.888885 unlinks/second - open/close 883 (time 1746072144.46 total 10.00 last 88.28) - open/close 1696 (time 1746072154.47 total 20.01 last 81.22) - open/close 2439 (time 1746072164.47 total 30.02 last 74.25) - open/close 3458 (time 1746072174.48 total 40.03 last 101.81) - open/close 4548 (time 1746072184.49 total 50.03 last 108.95) - open/close 5544 (time 1746072194.49 total 60.04 last 99.55) - open/close 6270 (time 1746072204.51 total 70.05 last 72.49) - open/close 7026 (time 1746072214.51 total 80.06 last 75.54) - open/close 7861 (time 1746072224.52 total 90.07 last 83.45) - open/close 8685 (time 1746072234.52 total 100.07 last 82.37) - open/close 9508 (time 1746072244.53 total 110.08 last 82.24) total: 10000 open/close in 116.42 seconds: 85.89 ops/second - unlinked 0 (time 1746072253 ; total 0 ; last 0) total: 10000 unlinks in 85 seconds: 117.647057 unlinks/second - open/close 826 (time 1746072351.23 total 10.00 last 82.56) - open/close 1765 (time 1746072361.25 total 20.02 last 93.72) - open/close 2496 (time 1746072371.26 total 30.03 last 73.07) - open/close 3340 (time 1746072381.26 total 40.03 last 84.38) - open/close 4095 (time 1746072391.26 total 50.03 last 75.47) - open/close 4868 (time 1746072401.27 total 60.04 last 77.22) - open/close 5733 (time 1746072411.29 total 70.06 last 86.37) - open/close 6512 (time 1746072421.30 total 80.07 last 77.80) - open/close 7282 (time 1746072431.31 total 90.09 last 76.91) - open/close 8157 (time 1746072441.33 total 100.10 last 87.36) - open/close 8950 (time 1746072451.33 total 110.11 last 79.27) - open/close 9725 (time 1746072461.33 total 120.11 last 77.49) total: 10000 open/close in 123.70 seconds: 80.84 ops/second - unlinked 0 (time 1746072467 ; total 0 ; last 0) total: 10000 unlinks in 82 seconds: 121.951218 unlinks/second - open/close 764 (time 1746072562.40 total 10.00 last 76.37) - open/close 1690 (time 1746072572.41 total 20.01 last 92.57) - open/close 2492 (time 1746072582.41 total 30.01 last 80.15) - open/close 3236 (time 1746072592.42 total 40.02 last 74.33) - open/close 3928 (time 1746072602.42 total 50.02 last 69.19) - open/close 4629 (time 1746072612.43 total 60.03 last 70.05) - open/close 5424 (time 1746072622.44 total 70.04 last 79.44) - open/close 6264 (time 1746072632.45 total 80.05 last 83.90) - open/close 7017 (time 1746072642.45 total 90.05 last 75.29) - open/close 7809 (time 1746072652.46 total 100.06 last 79.13) - open/close 8601 (time 1746072662.46 total 110.06 last 79.17) - open/close 9366 (time 1746072672.47 total 120.07 last 76.50) total: 10000 open/close in 128.39 seconds: 77.89 ops/second - unlinked 0 (time 1746072683 ; total 0 ; last 0) total: 10000 unlinks in 79 seconds: 126.582275 unlinks/second - open/close 760 (time 1746072775.50 total 10.00 last 75.97) - open/close 1541 (time 1746072785.50 total 20.01 last 78.09) - open/close 2493 (time 1746072795.51 total 30.01 last 95.14) - open/close 3381 (time 1746072805.51 total 40.01 last 88.80) - open/close 4247 (time 1746072815.51 total 50.02 last 86.57) - open/close 5024 (time 1746072825.53 total 60.03 last 77.59) - open/close 5843 (time 1746072835.53 total 70.03 last 81.87) - open/close 6583 (time 1746072845.54 total 80.04 last 73.97) - open/close 7396 (time 1746072855.54 total 90.04 last 81.25) - open/close 8111 (time 1746072865.55 total 100.05 last 71.47) - open/close 8824 (time 1746072875.57 total 110.07 last 71.16) - open/close 9497 (time 1746072885.57 total 120.08 last 67.25) total: 10000 open/close in 126.99 seconds: 78.75 ops/second - unlinked 0 (time 1746072895 ; total 0 ; last 0) total: 10000 unlinks in 72 seconds: 138.888885 unlinks/second - open/close 701 (time 1746072980.51 total 10.01 last 70.02) - open/close 1423 (time 1746072990.52 total 20.01 last 72.19) - open/close 2233 (time 1746073000.53 total 30.03 last 80.90) - open/close 3055 (time 1746073010.53 total 40.03 last 82.20) - open/close 3912 (time 1746073020.53 total 50.03 last 85.66) - open/close 4689 (time 1746073030.53 total 60.03 last 77.69) - open/close 5473 (time 1746073040.55 total 70.04 last 78.30) - open/close 6234 (time 1746073050.56 total 80.05 last 76.04) - open/close 7061 (time 1746073060.56 total 90.06 last 82.63) - open/close 7821 (time 1746073070.57 total 100.07 last 75.95) - open/close 8715 (time 1746073080.57 total 110.07 last 89.40) - open/close 9741 (time 1746073090.58 total 120.08 last 102.48) total: 10000 open/close in 123.32 seconds: 81.09 ops/second - unlinked 0 (time 1746073096 ; total 0 ; last 0) total: 10000 unlinks in 75 seconds: 133.333328 unlinks/second - open/close 976 (time 1746073183.44 total 10.00 last 97.57) - open/close 1923 (time 1746073193.45 total 20.01 last 94.60) - open/close 2922 (time 1746073203.46 total 30.02 last 99.85) - open/close 4057 (time 1746073213.47 total 40.03 last 113.34) - open/close 4956 (time 1746073223.47 total 50.04 last 89.88) - open/close 5786 (time 1746073233.48 total 60.04 last 82.96) - open/close 6669 (time 1746073243.49 total 70.05 last 88.22) - open/close 7701 (time 1746073253.49 total 80.06 last 103.12) - open/close 8700 (time 1746073263.50 total 90.06 last 99.81) - open/close 9792 (time 1746073273.51 total 100.07 last 109.14) total: 10000 open/close in 102.42 seconds: 97.64 ops/second - unlinked 0 (time 1746073278 ; total 0 ; last 0) total: 10000 unlinks in 63 seconds: 158.730164 unlinks/second - open/close 939 (time 1746073353.52 total 10.00 last 93.87) - open/close 1838 (time 1746073363.53 total 20.02 last 89.77) - open/close 2761 (time 1746073373.54 total 30.02 last 92.25) - open/close 3579 (time 1746073383.55 total 40.03 last 81.72) - open/close 4433 (time 1746073393.55 total 50.04 last 85.37) - open/close 5248 (time 1746073403.56 total 60.05 last 81.43) - open/close 6072 (time 1746073413.57 total 70.05 last 82.36) - open/close 6751 (time 1746073423.57 total 80.06 last 67.85) - open/close 7480 (time 1746073433.58 total 90.07 last 72.84) - open/close 8242 (time 1746073443.58 total 100.07 last 76.20) - open/close 9024 (time 1746073453.59 total 110.07 last 78.17) - open/close 9916 (time 1746073463.60 total 120.08 last 89.10) total: 10000 open/close in 121.00 seconds: 82.65 ops/second - unlinked 0 (time 1746073466 ; total 0 ; last 0) total: 10000 unlinks in 68 seconds: 147.058823 unlinks/second - open/close 854 (time 1746073547.21 total 10.00 last 85.38) - open/close 1668 (time 1746073557.21 total 20.01 last 81.35) - open/close 2438 (time 1746073567.21 total 30.01 last 76.99) - open/close 3205 (time 1746073577.22 total 40.01 last 76.67) - open/close 4067 (time 1746073587.22 total 50.02 last 86.17) - open/close 4844 (time 1746073597.23 total 60.03 last 77.64) - open/close 5692 (time 1746073607.25 total 70.04 last 84.64) - open/close 6572 (time 1746073617.26 total 80.06 last 87.87) - open/close 7392 (time 1746073627.27 total 90.07 last 81.93) - open/close 8319 (time 1746073637.27 total 100.07 last 92.68) - open/close 9104 (time 1746073647.28 total 110.08 last 78.42) - open/close 9913 (time 1746073657.29 total 120.09 last 80.86) total: 10000 open/close in 121.33 seconds: 82.42 ops/second - unlinked 0 (time 1746073661 ; total 0 ; last 0) total: 10000 unlinks in 74 seconds: 135.135132 unlinks/second - open/close 971 (time 1746073748.32 total 10.01 last 96.98) - open/close 1911 (time 1746073758.33 total 20.02 last 93.90) - open/close 2708 (time 1746073768.34 total 30.03 last 79.67) - open/close 3431 (time 1746073778.34 total 40.03 last 72.28) - open/close 4168 (time 1746073788.35 total 50.04 last 73.66) - open/close 5040 (time 1746073798.35 total 60.04 last 87.18) - open/close 5890 (time 1746073808.36 total 70.05 last 84.92) - open/close 6636 (time 1746073818.37 total 80.06 last 74.53) - open/close 7586 (time 1746073828.38 total 90.07 last 94.86) - open/close 8336 (time 1746073838.38 total 100.07 last 74.99) - open/close 9077 (time 1746073848.41 total 110.10 last 73.94) - open/close 9857 (time 1746073858.42 total 120.11 last 77.92) total: 10000 open/close in 121.54 seconds: 82.28 ops/second - unlinked 0 (time 1746073862 ; total 0 ; last 0) total: 10000 unlinks in 66 seconds: 151.515152 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg144-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg144-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg144-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.201.144@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.201.144@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg144-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg144-server: oleg144-server.virtnet: executing set_default_debug -1 all pdsh@oleg144-client: oleg144-server: ssh exited with exit code 1 Started lustre-OST0000 oleg144-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg144-server: oleg144-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg144-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 5 sec mount lustre on /mnt/lustre..... Starting client: oleg144-client.virtnet: -o user_xattr,flock 192.168.201.144@tcp:/lustre /mnt/lustre On OST0, 3461 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg144-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg144-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg144-server stop mds service on oleg144-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg144-server unloading modules via unload_modules_local on: 'oleg144-server' oleg144-server: oleg144-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg144-client: oleg144-server: ssh exited with exit code 2 pdsh@oleg144-client: oleg144-client: ssh exited with exit code 2 pdsh@oleg144-client: oleg144-client: ssh exited with exit code 2 pdsh@oleg144-client: oleg144-server: ssh exited with exit code 2 PASS 69 (2119s) == conf-sanity test 111: Adding large_dir with over 2GB directory ========================================================== 00:33:13 (1746073993) SKIP: conf-sanity test_111 Only applicable to ldiskfs-based MDTs SKIP 111 (2s) Stopping clients: oleg144-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg144-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg144-client: oleg144-server: ssh exited with exit code 2 oleg144-server: oleg144-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg144-server' oleg144-server: oleg144-server.virtnet: executing load_modules_local oleg144-server: Loading modules from /home/green/git/lustre-release/lustre oleg144-server: detected 4 online CPUs by sysfs oleg144-server: MODOPTS_LIBCFS= oleg144-server: Force libcfs to create 2 CPU partitions oleg144-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg144-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 2464 sec ========== 00:33:59 (1746074039) === conf-sanity: start cleanup 00:34:00 (1746074040) === === conf-sanity: finish cleanup 00:34:02 (1746074042) ===