-----============= acceptance-small: conf-sanity ============----- Thu May 1 18:25:41 EDT 2025 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg460-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex: No such file or directory excepting tests: 32b 32c 32newtarball 108a 112a 112b 113 117 119 121 122a 123aa 123ab 123ac 123ad 123ae 123af 123ag 123ah 123ahi 123F 123G 123H 126 129 132 133 135 136 137 150 152 153a 153b 153c 155 802a Stopping clients: oleg460-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg460-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg460-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg460-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg460-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg460-server oleg460-server: oleg460-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg460-server' oleg460-server: oleg460-server.virtnet: executing load_modules_local oleg460-server: Loading modules from /home/green/git/lustre-release/lustre oleg460-server: detected 4 online CPUs by sysfs oleg460-server: MODOPTS_LIBCFS= oleg460-server: Force libcfs to create 2 CPU partitions oleg460-server: libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg460-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg460-server: oleg460-server.virtnet: executing set_default_debug -1 all pdsh@oleg460-client: oleg460-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg460-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg460-server: oleg460-server.virtnet: executing set_default_debug -1 all pdsh@oleg460-client: oleg460-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg460-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg460-server: oleg460-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg460-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg460-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg460-server stop mds service on oleg460-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg460-server == conf-sanity test 45: long unlink handling in ptlrpcd == 18:26:54 (1746138414) start mds service on oleg460-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg460-server: oleg460-server.virtnet: executing set_default_debug -1 all pdsh@oleg460-client: oleg460-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg460-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg460-server: oleg460-server.virtnet: executing set_default_debug -1 all pdsh@oleg460-client: oleg460-server: ssh exited with exit code 1 Started lustre-OST0000 oleg460-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg460-client.virtnet: -o user_xattr,flock 192.168.204.160@tcp:/lustre /mnt/lustre setup single mount lustre success stop mds service on oleg460-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg460-server sleep 60 sec fail_loc=0x8000050f sleep 10 sec manual umount lustre on /mnt/lustre.... df: '/mnt/lustre': Cannot send after transport endpoint shutdown fail_loc=0x0 start mds service on oleg460-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg460-server: oleg460-server.virtnet: executing set_default_debug -1 all pdsh@oleg460-client: oleg460-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg460-client.virtnet: -o user_xattr,flock 192.168.204.160@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg460-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg460-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg460-server stop mds service on oleg460-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg460-server unloading modules via unload_modules_local on: 'oleg460-server' oleg460-server: oleg460-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg460-client: oleg460-server: ssh exited with exit code 2 pdsh@oleg460-client: oleg460-client: ssh exited with exit code 2 pdsh@oleg460-client: oleg460-client: ssh exited with exit code 2 pdsh@oleg460-client: oleg460-server: ssh exited with exit code 2 PASS 45 (142s) == conf-sanity test 69: replace an OST with the same index ========================================================== 18:29:16 (1746138556) start mds service on oleg460-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg460-server' oleg460-server: oleg460-server.virtnet: executing load_modules_local oleg460-server: Loading modules from /home/green/git/lustre-release/lustre oleg460-server: detected 4 online CPUs by sysfs oleg460-server: MODOPTS_LIBCFS= oleg460-server: Force libcfs to create 2 CPU partitions oleg460-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg460-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg460-server: oleg460-server.virtnet: executing set_default_debug -1 all pdsh@oleg460-client: oleg460-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg460-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg460-server: oleg460-server.virtnet: executing set_default_debug -1 all pdsh@oleg460-client: oleg460-server: ssh exited with exit code 1 Started lustre-OST0000 oleg460-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg460-client.virtnet: -o user_xattr,flock 192.168.204.160@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11872 inodes available. Want 99872. rc=0 - open/close 1347 (time 1746138599.08 total 10.00 last 134.66) - open/close 2785 (time 1746138609.08 total 20.01 last 143.76) - open/close 3982 (time 1746138619.10 total 30.02 last 119.48) - open/close 5345 (time 1746138629.10 total 40.02 last 136.29) - open/close 6777 (time 1746138639.10 total 50.03 last 143.14) - open/close 8422 (time 1746138649.11 total 60.03 last 164.44) total: 10000 open/close in 69.65 seconds: 143.58 ops/second - unlinked 0 (time 1746138660 ; total 0 ; last 0) total: 10000 unlinks in 42 seconds: 238.095245 unlinks/second - open/close 1175 (time 1746138713.76 total 10.00 last 117.45) - open/close 2232 (time 1746138723.77 total 20.01 last 105.68) - open/close 3243 (time 1746138733.77 total 30.01 last 101.06) - open/close 4455 (time 1746138743.77 total 40.01 last 121.19) - open/close 5690 (time 1746138753.77 total 50.01 last 123.48) - open/close 7069 (time 1746138763.78 total 60.02 last 137.78) - open/close 8344 (time 1746138773.78 total 70.02 last 127.48) - open/close 9748 (time 1746138783.79 total 80.03 last 140.33) total: 10000 open/close in 81.99 seconds: 121.96 ops/second - unlinked 0 (time 1746138787 ; total 0 ; last 0) total: 10000 unlinks in 48 seconds: 208.333328 unlinks/second - open/close 1296 (time 1746138847.10 total 10.00 last 129.57) - open/close 2608 (time 1746138857.10 total 20.01 last 131.14) - open/close 3830 (time 1746138867.10 total 30.01 last 122.19) - open/close 4967 (time 1746138877.11 total 40.01 last 113.65) - open/close 6138 (time 1746138887.11 total 50.01 last 117.08) - open/close 7268 (time 1746138897.11 total 60.02 last 112.94) - open/close 8335 (time 1746138907.13 total 70.03 last 106.57) - open/close 9365 (time 1746138917.13 total 80.03 last 102.97) total: 10000 open/close in 85.90 seconds: 116.42 ops/second - unlinked 0 (time 1746138925 ; total 0 ; last 0) total: 10000 unlinks in 49 seconds: 204.081635 unlinks/second - open/close 950 (time 1746138986.97 total 10.00 last 94.99) - open/close 1975 (time 1746138996.97 total 20.00 last 102.50) - open/close 3062 (time 1746139006.97 total 30.00 last 108.70) - open/close 4116 (time 1746139016.97 total 40.00 last 105.39) - open/close 5179 (time 1746139026.97 total 50.00 last 106.29) - open/close 6299 (time 1746139036.98 total 60.01 last 111.91) - open/close 7411 (time 1746139046.98 total 70.02 last 111.15) - open/close 8545 (time 1746139056.98 total 80.02 last 113.37) - open/close 9543 (time 1746139067.00 total 90.03 last 99.68) total: 10000 open/close in 94.83 seconds: 105.45 ops/second - unlinked 0 (time 1746139073 ; total 0 ; last 0) total: 10000 unlinks in 50 seconds: 200.000000 unlinks/second - open/close 1675 (time 1746139134.54 total 10.00 last 167.44) - open/close 2822 (time 1746139144.55 total 20.01 last 114.64) - open/close 3925 (time 1746139154.55 total 30.01 last 110.28) - open/close 5173 (time 1746139164.56 total 40.02 last 124.74) - open/close 6343 (time 1746139174.57 total 50.02 last 116.90) - open/close 7283 (time 1746139184.57 total 60.03 last 93.93) - open/close 8310 (time 1746139194.58 total 70.04 last 102.65) - open/close 9584 (time 1746139204.59 total 80.04 last 127.29) total: 10000 open/close in 83.83 seconds: 119.29 ops/second - unlinked 0 (time 1746139210 ; total 0 ; last 0) total: 10000 unlinks in 52 seconds: 192.307693 unlinks/second - open/close 1185 (time 1746139274.60 total 10.00 last 118.45) - open/close 2561 (time 1746139284.61 total 20.01 last 137.53) - open/close 3769 (time 1746139294.62 total 30.02 last 120.68) - open/close 4682 (time 1746139304.62 total 40.02 last 91.27) - open/close 5633 (time 1746139314.62 total 50.02 last 95.09) - open/close 6752 (time 1746139324.63 total 60.03 last 111.80) - open/close 8099 (time 1746139334.63 total 70.03 last 134.68) - open/close 9215 (time 1746139344.64 total 80.04 last 111.55) total: 10000 open/close in 86.86 seconds: 115.13 ops/second - unlinked 0 (time 1746139353 ; total 0 ; last 0) total: 10000 unlinks in 55 seconds: 181.818176 unlinks/second - open/close 1237 (time 1746139420.78 total 10.00 last 123.66) - open/close 2352 (time 1746139430.78 total 20.01 last 111.48) - open/close 3341 (time 1746139440.79 total 30.02 last 98.80) - open/close 4290 (time 1746139450.80 total 40.02 last 94.86) - open/close 5222 (time 1746139460.80 total 50.03 last 93.14) - open/close 6158 (time 1746139470.81 total 60.03 last 93.56) - open/close 7200 (time 1746139480.82 total 70.04 last 104.12) - open/close 8091 (time 1746139490.82 total 80.04 last 89.05) - open/close 8950 (time 1746139500.82 total 90.04 last 85.89) - open/close 9983 (time 1746139510.83 total 100.05 last 103.26) total: 10000 open/close in 100.14 seconds: 99.87 ops/second - unlinked 0 (time 1746139513 ; total 0 ; last 0) total: 10000 unlinks in 48 seconds: 208.333328 unlinks/second - open/close 1126 (time 1746139573.16 total 10.00 last 112.57) - open/close 2392 (time 1746139583.17 total 20.01 last 126.47) - open/close 3721 (time 1746139593.18 total 30.02 last 132.76) - open/close 4730 (time 1746139603.18 total 40.02 last 100.90) - open/close 5755 (time 1746139613.19 total 50.03 last 102.47) - open/close 7065 (time 1746139623.19 total 60.03 last 130.97) - open/close 8300 (time 1746139633.19 total 70.03 last 123.45) - open/close 9307 (time 1746139643.19 total 80.03 last 100.70) total: 10000 open/close in 85.96 seconds: 116.34 ops/second - unlinked 0 (time 1746139650 ; total 0 ; last 0) total: 10000 unlinks in 52 seconds: 192.307693 unlinks/second - open/close 1064 (time 1746139714.06 total 10.00 last 106.38) - open/close 2083 (time 1746139724.06 total 20.01 last 101.86) - open/close 3104 (time 1746139734.07 total 30.01 last 102.06) - open/close 4039 (time 1746139744.07 total 40.02 last 93.45) - open/close 4957 (time 1746139754.08 total 50.02 last 91.76) - open/close 5895 (time 1746139764.08 total 60.03 last 93.76) - open/close 6832 (time 1746139774.08 total 70.03 last 93.68) - open/close 7877 (time 1746139784.08 total 80.03 last 104.50) - open/close 9084 (time 1746139794.08 total 90.03 last 120.68) total: 10000 open/close in 99.04 seconds: 100.97 ops/second - unlinked 0 (time 1746139805 ; total 0 ; last 0) total: 10000 unlinks in 46 seconds: 217.391312 unlinks/second - open/close 1421 (time 1746139863.02 total 10.00 last 142.07) - open/close 2690 (time 1746139873.02 total 20.00 last 126.87) - open/close 3873 (time 1746139883.02 total 30.01 last 118.27) - open/close 5191 (time 1746139893.04 total 40.02 last 131.57) - open/close 6466 (time 1746139903.04 total 50.03 last 127.47) - open/close 7715 (time 1746139913.04 total 60.03 last 124.88) - open/close 8807 (time 1746139923.04 total 70.03 last 109.18) total: 10000 open/close in 79.90 seconds: 125.16 ops/second - unlinked 0 (time 1746139934 ; total 0 ; last 0) total: 10000 unlinks in 56 seconds: 178.571426 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg460-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg460-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg460-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.204.160@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.204.160@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg460-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg460-server: oleg460-server.virtnet: executing set_default_debug -1 all pdsh@oleg460-client: oleg460-server: ssh exited with exit code 1 Started lustre-OST0000 oleg460-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg460-server: oleg460-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg460-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 7 sec mount lustre on /mnt/lustre..... Starting client: oleg460-client.virtnet: -o user_xattr,flock 192.168.204.160@tcp:/lustre /mnt/lustre On OST0, 5261 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg460-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg460-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg460-server stop mds service on oleg460-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg460-server unloading modules via unload_modules_local on: 'oleg460-server' oleg460-server: oleg460-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg460-client: oleg460-server: ssh exited with exit code 2 pdsh@oleg460-client: oleg460-client: ssh exited with exit code 2 pdsh@oleg460-client: oleg460-client: ssh exited with exit code 2 pdsh@oleg460-client: oleg460-server: ssh exited with exit code 2 PASS 69 (1493s) == conf-sanity test 111: Adding large_dir with over 2GB directory ========================================================== 18:54:09 (1746140049) SKIP: conf-sanity test_111 Only applicable to ldiskfs-based MDTs SKIP 111 (2s) Stopping clients: oleg460-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg460-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg460-client: oleg460-server: ssh exited with exit code 2 oleg460-server: oleg460-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg460-server' oleg460-server: oleg460-server.virtnet: executing load_modules_local oleg460-server: Loading modules from /home/green/git/lustre-release/lustre oleg460-server: detected 4 online CPUs by sysfs oleg460-server: MODOPTS_LIBCFS= oleg460-server: Force libcfs to create 2 CPU partitions oleg460-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg460-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 1748 sec ========== 18:54:49 (1746140089) === conf-sanity: start cleanup 18:54:49 (1746140089) === === conf-sanity: finish cleanup 18:54:51 (1746140091) ===