-----============= acceptance-small: conf-sanity ============----- Thu May 1 22:58:26 EDT 2025 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg419-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex: No such file or directory excepting tests: 32b 32c 32newtarball 108a 112a 112b 113 117 119 121 122a 123aa 123ab 123ac 123ad 123ae 123af 123ag 123ah 123ahi 123F 123G 123H 126 129 132 133 135 136 137 150 152 153a 153b 153c 155 802a Stopping clients: oleg419-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg419-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg419-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg419-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg419-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg419-server oleg419-server: oleg419-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg419-server' oleg419-server: oleg419-server.virtnet: executing load_modules_local oleg419-server: Loading modules from /home/green/git/lustre-release/lustre oleg419-server: detected 4 online CPUs by sysfs oleg419-server: MODOPTS_LIBCFS= oleg419-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg419-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg419-server: oleg419-server.virtnet: executing set_default_debug -1 all pdsh@oleg419-client: oleg419-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg419-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg419-server: oleg419-server.virtnet: executing set_default_debug -1 all pdsh@oleg419-client: oleg419-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg419-server: oleg419-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg419-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg419-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg419-server stop mds service on oleg419-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg419-server == conf-sanity test 45: long unlink handling in ptlrpcd == 22:59:46 (1746154786) start mds service on oleg419-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg419-server: oleg419-server.virtnet: executing set_default_debug -1 all pdsh@oleg419-client: oleg419-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg419-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg419-server: oleg419-server.virtnet: executing set_default_debug -1 all pdsh@oleg419-client: oleg419-server: ssh exited with exit code 1 Started lustre-OST0000 oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg419-client.virtnet: -o user_xattr,flock 192.168.204.119@tcp:/lustre /mnt/lustre setup single mount lustre success stop mds service on oleg419-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg419-server sleep 60 sec fail_loc=0x8000050f sleep 10 sec manual umount lustre on /mnt/lustre.... df: '/mnt/lustre': Cannot send after transport endpoint shutdown fail_loc=0x0 start mds service on oleg419-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg419-server: oleg419-server.virtnet: executing set_default_debug -1 all pdsh@oleg419-client: oleg419-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg419-client.virtnet: -o user_xattr,flock 192.168.204.119@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg419-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg419-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg419-server stop mds service on oleg419-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg419-server unloading modules via unload_modules_local on: 'oleg419-server' oleg419-server: oleg419-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg419-client: oleg419-client: ssh exited with exit code 2 pdsh@oleg419-client: oleg419-server: ssh exited with exit code 2 pdsh@oleg419-client: oleg419-client: ssh exited with exit code 2 pdsh@oleg419-client: oleg419-server: ssh exited with exit code 2 PASS 45 (145s) == conf-sanity test 69: replace an OST with the same index ========================================================== 23:02:11 (1746154931) start mds service on oleg419-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg419-server' oleg419-server: oleg419-server.virtnet: executing load_modules_local oleg419-server: Loading modules from /home/green/git/lustre-release/lustre oleg419-server: detected 4 online CPUs by sysfs oleg419-server: MODOPTS_LIBCFS= oleg419-server: Force libcfs to create 2 CPU partitions oleg419-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg419-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg419-server: oleg419-server.virtnet: executing set_default_debug -1 all pdsh@oleg419-client: oleg419-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg419-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg419-server: oleg419-server.virtnet: executing set_default_debug -1 all pdsh@oleg419-client: oleg419-server: ssh exited with exit code 1 Started lustre-OST0000 oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg419-client.virtnet: -o user_xattr,flock 192.168.204.119@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11872 inodes available. Want 99872. rc=0 - open/close 1010 (time 1746154974.23 total 10.01 last 100.94) - open/close 2121 (time 1746154984.23 total 20.01 last 111.04) - open/close 3599 (time 1746154994.24 total 30.01 last 147.75) - open/close 5091 (time 1746155004.24 total 40.02 last 149.09) - open/close 6545 (time 1746155014.25 total 50.03 last 145.29) - open/close 7759 (time 1746155024.28 total 60.06 last 121.02) - open/close 8809 (time 1746155034.29 total 70.07 last 104.92) total: 10000 open/close in 78.75 seconds: 126.99 ops/second - unlinked 0 (time 1746155044 ; total 0 ; last 0) total: 10000 unlinks in 41 seconds: 243.902435 unlinks/second - open/close 1611 (time 1746155097.45 total 10.00 last 161.06) - open/close 3101 (time 1746155107.46 total 20.01 last 148.94) - open/close 4466 (time 1746155117.46 total 30.01 last 136.50) - open/close 5748 (time 1746155127.46 total 40.01 last 128.20) - open/close 7366 (time 1746155137.46 total 50.01 last 161.72) - open/close 8811 (time 1746155147.47 total 60.02 last 144.45) total: 10000 open/close in 69.32 seconds: 144.26 ops/second - unlinked 0 (time 1746155158 ; total 0 ; last 0) total: 10000 unlinks in 39 seconds: 256.410248 unlinks/second - open/close 1403 (time 1746155209.84 total 10.00 last 140.30) - open/close 2542 (time 1746155219.85 total 20.00 last 113.86) - open/close 3940 (time 1746155229.85 total 30.01 last 139.70) - open/close 5160 (time 1746155239.86 total 40.01 last 121.98) - open/close 6433 (time 1746155249.86 total 50.01 last 127.28) - open/close 7886 (time 1746155259.86 total 60.02 last 145.26) - open/close 9537 (time 1746155269.86 total 70.02 last 165.10) total: 10000 open/close in 73.26 seconds: 136.50 ops/second - unlinked 0 (time 1746155274 ; total 0 ; last 0) total: 10000 unlinks in 44 seconds: 227.272720 unlinks/second - open/close 1187 (time 1746155330.67 total 10.00 last 118.67) - open/close 2442 (time 1746155340.68 total 20.02 last 125.33) - open/close 3607 (time 1746155350.68 total 30.02 last 116.48) - open/close 5025 (time 1746155360.69 total 40.02 last 141.78) - open/close 6180 (time 1746155370.70 total 50.03 last 115.39) - open/close 7415 (time 1746155380.70 total 60.03 last 123.42) - open/close 8583 (time 1746155390.71 total 70.04 last 116.75) - open/close 9869 (time 1746155400.71 total 80.05 last 128.52) total: 10000 open/close in 81.00 seconds: 123.45 ops/second - unlinked 0 (time 1746155403 ; total 0 ; last 0) total: 10000 unlinks in 44 seconds: 227.272720 unlinks/second - open/close 1234 (time 1746155459.39 total 10.00 last 123.37) - open/close 2420 (time 1746155469.40 total 20.00 last 118.57) - open/close 3705 (time 1746155479.41 total 30.01 last 128.38) - open/close 4835 (time 1746155489.41 total 40.02 last 112.97) - open/close 5890 (time 1746155499.41 total 50.02 last 105.45) - open/close 7257 (time 1746155509.42 total 60.02 last 136.66) - open/close 8788 (time 1746155519.42 total 70.03 last 152.98) total: 10000 open/close in 79.46 seconds: 125.84 ops/second - unlinked 0 (time 1746155530 ; total 0 ; last 0) total: 10000 unlinks in 45 seconds: 222.222229 unlinks/second - open/close 1377 (time 1746155586.96 total 10.00 last 137.66) - open/close 2882 (time 1746155596.96 total 20.00 last 150.48) - open/close 4720 (time 1746155606.97 total 30.01 last 183.64) - open/close 6050 (time 1746155616.97 total 40.02 last 132.97) - open/close 7405 (time 1746155626.97 total 50.02 last 135.46) - open/close 8738 (time 1746155636.97 total 60.02 last 133.27) total: 10000 open/close in 68.97 seconds: 144.99 ops/second - unlinked 0 (time 1746155647 ; total 0 ; last 0) total: 10000 unlinks in 38 seconds: 263.157898 unlinks/second - open/close 2001 (time 1746155696.41 total 10.01 last 199.92) - open/close 3951 (time 1746155706.41 total 20.01 last 194.91) - open/close 5414 (time 1746155716.41 total 30.02 last 146.28) - open/close 6849 (time 1746155726.43 total 40.03 last 143.34) - open/close 7994 (time 1746155736.43 total 50.03 last 114.41) - open/close 9287 (time 1746155746.44 total 60.04 last 129.24) total: 10000 open/close in 66.14 seconds: 151.19 ops/second - unlinked 0 (time 1746155754 ; total 0 ; last 0) total: 10000 unlinks in 46 seconds: 217.391312 unlinks/second - open/close 1028 (time 1746155812.42 total 10.01 last 102.72) - open/close 2181 (time 1746155822.42 total 20.01 last 115.28) - open/close 3321 (time 1746155832.42 total 30.01 last 113.96) - open/close 4506 (time 1746155842.42 total 40.02 last 118.46) - open/close 5778 (time 1746155852.43 total 50.02 last 127.13) - open/close 7029 (time 1746155862.44 total 60.03 last 124.98) - open/close 7925 (time 1746155872.45 total 70.05 last 89.48) - open/close 8777 (time 1746155882.46 total 80.05 last 85.18) - open/close 9911 (time 1746155892.47 total 90.06 last 113.28) total: 10000 open/close in 91.04 seconds: 109.84 ops/second - unlinked 0 (time 1746155895 ; total 0 ; last 0) total: 10000 unlinks in 47 seconds: 212.765961 unlinks/second - open/close 1080 (time 1746155955.03 total 10.01 last 107.92) - open/close 2199 (time 1746155965.04 total 20.01 last 111.84) - open/close 3376 (time 1746155975.04 total 30.01 last 117.67) - open/close 4374 (time 1746155985.05 total 40.02 last 99.72) - open/close 5391 (time 1746155995.05 total 50.03 last 101.65) - open/close 6469 (time 1746156005.06 total 60.03 last 107.74) - open/close 7557 (time 1746156015.06 total 70.03 last 108.80) - open/close 8767 (time 1746156025.07 total 80.05 last 120.81) - open/close 9918 (time 1746156035.07 total 90.05 last 115.07) total: 10000 open/close in 90.86 seconds: 110.06 ops/second - unlinked 0 (time 1746156037 ; total 0 ; last 0) total: 10000 unlinks in 53 seconds: 188.679245 unlinks/second - open/close 1282 (time 1746156102.45 total 10.00 last 128.19) - open/close 2612 (time 1746156112.46 total 20.01 last 132.87) - open/close 3698 (time 1746156122.47 total 30.02 last 108.49) - open/close 4691 (time 1746156132.48 total 40.03 last 99.26) - open/close 6087 (time 1746156142.49 total 50.03 last 139.50) - open/close 7247 (time 1746156152.49 total 60.04 last 115.95) - open/close 8293 (time 1746156162.49 total 70.04 last 104.59) - open/close 9483 (time 1746156172.50 total 80.04 last 118.93) total: 10000 open/close in 84.24 seconds: 118.70 ops/second - unlinked 0 (time 1746156178 ; total 0 ; last 0) total: 10000 unlinks in 57 seconds: 175.438599 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg419-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg419-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg419-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.204.119@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.204.119@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg419-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg419-server: oleg419-server.virtnet: executing set_default_debug -1 all pdsh@oleg419-client: oleg419-server: ssh exited with exit code 1 Started lustre-OST0000 oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg419-server: oleg419-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg419-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg419-client.virtnet: -o user_xattr,flock 192.168.204.119@tcp:/lustre /mnt/lustre On OST0, 6845 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg419-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg419-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg419-server stop mds service on oleg419-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg419-server unloading modules via unload_modules_local on: 'oleg419-server' oleg419-server: oleg419-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg419-client: oleg419-client: ssh exited with exit code 2 pdsh@oleg419-client: oleg419-server: ssh exited with exit code 2 pdsh@oleg419-client: oleg419-server: ssh exited with exit code 2 pdsh@oleg419-client: oleg419-client: ssh exited with exit code 2 PASS 69 (1353s) == conf-sanity test 111: Adding large_dir with over 2GB directory ========================================================== 23:24:45 (1746156285) SKIP: conf-sanity test_111 Only applicable to ldiskfs-based MDTs SKIP 111 (1s) Stopping clients: oleg419-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg419-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg419-client: oleg419-server: ssh exited with exit code 2 oleg419-server: oleg419-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg419-server' oleg419-server: oleg419-server.virtnet: executing load_modules_local oleg419-server: Loading modules from /home/green/git/lustre-release/lustre oleg419-server: detected 4 online CPUs by sysfs oleg419-server: MODOPTS_LIBCFS= oleg419-server: Force libcfs to create 2 CPU partitions oleg419-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg419-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg419-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 1614 sec ========== 23:25:20 (1746156320) === conf-sanity: start cleanup 23:25:21 (1746156321) === === conf-sanity: finish cleanup 23:25:23 (1746156323) ===