-----============= acceptance-small: conf-sanity ============----- Thu May 1 23:27:08 EDT 2025 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg329-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex: No such file or directory excepting tests: 32b 32c 32newtarball 108a 112a 112b 113 117 119 121 122a 123aa 123ab 123ac 123ad 123ae 123af 123ag 123ah 123ahi 123F 123G 123H 126 129 132 133 135 136 137 150 152 153a 153b 153c 155 802a Stopping clients: oleg329-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg329-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg329-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg329-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg329-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg329-server oleg329-server: oleg329-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg329-server' oleg329-server: oleg329-server.virtnet: executing load_modules_local oleg329-server: Loading modules from /home/green/git/lustre-release/lustre oleg329-server: detected 4 online CPUs by sysfs oleg329-server: MODOPTS_LIBCFS= oleg329-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg329-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg329-server: oleg329-server.virtnet: executing set_default_debug -1 all pdsh@oleg329-client: oleg329-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg329-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg329-server: oleg329-server.virtnet: executing set_default_debug -1 all pdsh@oleg329-client: oleg329-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg329-server: oleg329-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg329-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg329-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg329-server stop mds service on oleg329-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg329-server == conf-sanity test 45: long unlink handling in ptlrpcd == 23:28:27 (1746156507) start mds service on oleg329-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg329-server: oleg329-server.virtnet: executing set_default_debug -1 all pdsh@oleg329-client: oleg329-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg329-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg329-server: oleg329-server.virtnet: executing set_default_debug -1 all pdsh@oleg329-client: oleg329-server: ssh exited with exit code 1 Started lustre-OST0000 oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg329-client.virtnet: -o user_xattr,flock 192.168.203.129@tcp:/lustre /mnt/lustre setup single mount lustre success stop mds service on oleg329-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg329-server sleep 60 sec fail_loc=0x8000050f sleep 10 sec manual umount lustre on /mnt/lustre.... df: '/mnt/lustre': Cannot send after transport endpoint shutdown fail_loc=0x0 start mds service on oleg329-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg329-server: oleg329-server.virtnet: executing set_default_debug -1 all pdsh@oleg329-client: oleg329-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg329-client.virtnet: -o user_xattr,flock 192.168.203.129@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg329-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg329-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg329-server stop mds service on oleg329-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg329-server unloading modules via unload_modules_local on: 'oleg329-server' oleg329-server: oleg329-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg329-client: oleg329-client: ssh exited with exit code 2 pdsh@oleg329-client: oleg329-server: ssh exited with exit code 2 pdsh@oleg329-client: oleg329-client: ssh exited with exit code 2 pdsh@oleg329-client: oleg329-server: ssh exited with exit code 2 PASS 45 (133s) == conf-sanity test 69: replace an OST with the same index ========================================================== 23:30:41 (1746156641) start mds service on oleg329-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg329-server' oleg329-server: oleg329-server.virtnet: executing load_modules_local oleg329-server: Loading modules from /home/green/git/lustre-release/lustre oleg329-server: detected 4 online CPUs by sysfs oleg329-server: MODOPTS_LIBCFS= oleg329-server: Force libcfs to create 2 CPU partitions oleg329-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg329-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg329-server: oleg329-server.virtnet: executing set_default_debug -1 all pdsh@oleg329-client: oleg329-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg329-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg329-server: oleg329-server.virtnet: executing set_default_debug -1 all pdsh@oleg329-client: oleg329-server: ssh exited with exit code 1 Started lustre-OST0000 oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg329-client.virtnet: -o user_xattr,flock 192.168.203.129@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11872 inodes available. Want 99872. rc=0 - open/close 1157 (time 1746156682.36 total 10.01 last 115.58) - open/close 2261 (time 1746156692.37 total 20.01 last 110.36) - open/close 3346 (time 1746156702.37 total 30.02 last 108.47) - open/close 4625 (time 1746156712.38 total 40.02 last 127.84) - open/close 5732 (time 1746156722.38 total 50.03 last 110.65) - open/close 6873 (time 1746156732.39 total 60.04 last 113.98) - open/close 8134 (time 1746156742.40 total 70.04 last 126.02) - open/close 9328 (time 1746156752.40 total 80.05 last 119.32) total: 10000 open/close in 85.72 seconds: 116.66 ops/second - unlinked 0 (time 1746156759 ; total 0 ; last 0) total: 10000 unlinks in 43 seconds: 232.558136 unlinks/second - open/close 1415 (time 1746156814.06 total 10.00 last 141.48) - open/close 2725 (time 1746156824.06 total 20.00 last 130.99) - open/close 3865 (time 1746156834.06 total 30.00 last 113.99) - open/close 5014 (time 1746156844.07 total 40.01 last 114.83) - open/close 6176 (time 1746156854.07 total 50.02 last 116.11) - open/close 7226 (time 1746156864.08 total 60.02 last 104.99) - open/close 8239 (time 1746156874.08 total 70.02 last 101.27) - open/close 9287 (time 1746156884.08 total 80.03 last 104.73) total: 10000 open/close in 86.13 seconds: 116.11 ops/second - unlinked 0 (time 1746156891 ; total 0 ; last 0) total: 10000 unlinks in 57 seconds: 175.438599 unlinks/second - open/close 1340 (time 1746156959.67 total 10.00 last 133.99) - open/close 2293 (time 1746156969.68 total 20.01 last 95.25) - open/close 3280 (time 1746156979.68 total 30.01 last 98.67) - open/close 4615 (time 1746156989.69 total 40.01 last 133.43) - open/close 5803 (time 1746156999.69 total 50.02 last 118.79) - open/close 6795 (time 1746157009.69 total 60.02 last 99.20) - open/close 7730 (time 1746157019.69 total 70.02 last 93.46) - open/close 8763 (time 1746157029.71 total 80.03 last 103.16) - open/close 9717 (time 1746157039.71 total 90.04 last 95.32) total: 10000 open/close in 92.53 seconds: 108.07 ops/second - unlinked 0 (time 1746157044 ; total 0 ; last 0) total: 10000 unlinks in 51 seconds: 196.078430 unlinks/second - open/close 1017 (time 1746157107.37 total 10.01 last 101.63) - open/close 1972 (time 1746157117.38 total 20.01 last 95.46) - open/close 2975 (time 1746157127.38 total 30.02 last 100.24) - open/close 4020 (time 1746157137.39 total 40.02 last 104.43) - open/close 5225 (time 1746157147.39 total 50.03 last 120.46) - open/close 6345 (time 1746157157.40 total 60.04 last 111.90) - open/close 7427 (time 1746157167.40 total 70.04 last 108.19) - open/close 8397 (time 1746157177.41 total 80.04 last 96.98) - open/close 9519 (time 1746157187.41 total 90.04 last 112.17) total: 10000 open/close in 93.61 seconds: 106.82 ops/second - unlinked 0 (time 1746157192 ; total 0 ; last 0) total: 10000 unlinks in 40 seconds: 250.000000 unlinks/second - open/close 1489 (time 1746157243.88 total 10.00 last 148.87) - open/close 2812 (time 1746157253.88 total 20.00 last 132.29) - open/close 4051 (time 1746157263.89 total 30.01 last 123.86) - open/close 5210 (time 1746157273.89 total 40.01 last 115.84) - open/close 6642 (time 1746157283.90 total 50.01 last 143.15) - open/close 8174 (time 1746157293.90 total 60.02 last 153.13) - open/close 9586 (time 1746157303.90 total 70.02 last 141.12) total: 10000 open/close in 72.94 seconds: 137.10 ops/second - unlinked 0 (time 1746157308 ; total 0 ; last 0) total: 10000 unlinks in 45 seconds: 222.222229 unlinks/second - open/close 1369 (time 1746157364.96 total 10.00 last 136.85) - open/close 2616 (time 1746157374.97 total 20.02 last 124.52) - open/close 3935 (time 1746157384.98 total 30.02 last 131.85) - open/close 5375 (time 1746157394.98 total 40.02 last 143.96) - open/close 7033 (time 1746157404.99 total 50.03 last 165.68) - open/close 8396 (time 1746157414.99 total 60.04 last 136.22) - open/close 9920 (time 1746157424.99 total 70.04 last 152.37) total: 10000 open/close in 70.49 seconds: 141.87 ops/second - unlinked 0 (time 1746157426 ; total 0 ; last 0) total: 10000 unlinks in 32 seconds: 312.500000 unlinks/second - open/close 1597 (time 1746157469.49 total 10.00 last 159.68) - open/close 3406 (time 1746157479.50 total 20.00 last 180.87) - open/close 4965 (time 1746157489.50 total 30.00 last 155.89) - open/close 6847 (time 1746157499.50 total 40.00 last 188.18) - open/close 8795 (time 1746157509.50 total 50.01 last 194.76) total: 10000 open/close in 56.98 seconds: 175.50 ops/second - unlinked 0 (time 1746157517 ; total 0 ; last 0) total: 10000 unlinks in 36 seconds: 277.777771 unlinks/second - open/close 1434 (time 1746157564.55 total 10.01 last 143.32) - open/close 3159 (time 1746157574.55 total 20.01 last 172.49) - open/close 4801 (time 1746157584.55 total 30.01 last 164.15) - open/close 6545 (time 1746157594.55 total 40.01 last 174.34) - open/close 7860 (time 1746157604.56 total 50.02 last 131.42) - open/close 9370 (time 1746157614.56 total 60.02 last 150.96) total: 10000 open/close in 65.32 seconds: 153.10 ops/second - unlinked 0 (time 1746157621 ; total 0 ; last 0) total: 10000 unlinks in 40 seconds: 250.000000 unlinks/second - open/close 1277 (time 1746157673.04 total 10.00 last 127.67) - open/close 2826 (time 1746157683.04 total 20.01 last 154.86) - open/close 4192 (time 1746157693.05 total 30.01 last 136.52) - open/close 5738 (time 1746157703.05 total 40.01 last 154.58) - open/close 7252 (time 1746157713.05 total 50.01 last 151.40) - open/close 8695 (time 1746157723.05 total 60.02 last 144.21) total: 10000 open/close in 69.48 seconds: 143.92 ops/second - unlinked 0 (time 1746157734 ; total 0 ; last 0) total: 10000 unlinks in 45 seconds: 222.222229 unlinks/second - open/close 1201 (time 1746157790.97 total 10.00 last 120.10) - open/close 2571 (time 1746157800.97 total 20.00 last 136.97) - open/close 3690 (time 1746157810.97 total 30.00 last 111.89) - open/close 4774 (time 1746157820.97 total 40.01 last 108.37) - open/close 5960 (time 1746157830.98 total 50.01 last 118.57) - open/close 7083 (time 1746157840.99 total 60.02 last 112.16) - open/close 8103 (time 1746157850.99 total 70.02 last 101.99) - open/close 9142 (time 1746157861.01 total 80.04 last 103.69) total: 10000 open/close in 87.65 seconds: 114.09 ops/second - unlinked 0 (time 1746157870 ; total 0 ; last 0) total: 10000 unlinks in 49 seconds: 204.081635 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg329-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg329-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg329-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.203.129@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.203.129@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg329-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg329-server: oleg329-server.virtnet: executing set_default_debug -1 all pdsh@oleg329-client: oleg329-server: ssh exited with exit code 1 Started lustre-OST0000 oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg329-server: oleg329-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg329-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 4 sec mount lustre on /mnt/lustre..... Starting client: oleg329-client.virtnet: -o user_xattr,flock 192.168.203.129@tcp:/lustre /mnt/lustre On OST0, 17669 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg329-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg329-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg329-server stop mds service on oleg329-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg329-server unloading modules via unload_modules_local on: 'oleg329-server' oleg329-server: oleg329-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg329-client: oleg329-client: ssh exited with exit code 2 pdsh@oleg329-client: oleg329-server: ssh exited with exit code 2 pdsh@oleg329-client: oleg329-client: ssh exited with exit code 2 pdsh@oleg329-client: oleg329-server: ssh exited with exit code 2 PASS 69 (1326s) == conf-sanity test 111: Adding large_dir with over 2GB directory ========================================================== 23:52:47 (1746157967) SKIP: conf-sanity test_111 Only applicable to ldiskfs-based MDTs SKIP 111 (2s) Stopping clients: oleg329-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg329-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg329-client: oleg329-server: ssh exited with exit code 2 oleg329-server: oleg329-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg329-server' oleg329-server: oleg329-server.virtnet: executing load_modules_local oleg329-server: Loading modules from /home/green/git/lustre-release/lustre oleg329-server: detected 4 online CPUs by sysfs oleg329-server: MODOPTS_LIBCFS= oleg329-server: Force libcfs to create 2 CPU partitions oleg329-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg329-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 1571 sec ========== 23:53:20 (1746158000) === conf-sanity: start cleanup 23:53:20 (1746158000) === === conf-sanity: finish cleanup 23:53:22 (1746158002) ===