-----============= acceptance-small: conf-sanity ============----- Thu May 1 21:01:16 EDT 2025 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg425-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex: No such file or directory excepting tests: 32b 32c 32newtarball 108a 112a 112b 113 117 119 121 122a 123aa 123ab 123ac 123ad 123ae 123af 123ag 123ah 123ahi 123F 123G 123H 126 129 132 133 135 136 137 150 152 153a 153b 153c 155 802a Stopping clients: oleg425-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg425-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg425-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg425-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg425-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg425-server oleg425-server: oleg425-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory loading modules on: 'oleg425-server' oleg425-server: oleg425-server.virtnet: executing load_modules_local oleg425-server: Loading modules from /home/green/git/lustre-release/lustre oleg425-server: detected 4 online CPUs by sysfs oleg425-server: MODOPTS_LIBCFS= oleg425-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg425-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg425-server: oleg425-server.virtnet: executing set_default_debug -1 all pdsh@oleg425-client: oleg425-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg425-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg425-server: oleg425-server.virtnet: executing set_default_debug -1 all pdsh@oleg425-client: oleg425-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg425-server: oleg425-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg425-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg425-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg425-server stop mds service on oleg425-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg425-server == conf-sanity test 45: long unlink handling in ptlrpcd == 21:02:34 (1746147754) start mds service on oleg425-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg425-server: oleg425-server.virtnet: executing set_default_debug -1 all pdsh@oleg425-client: oleg425-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg425-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg425-server: oleg425-server.virtnet: executing set_default_debug -1 all pdsh@oleg425-client: oleg425-server: ssh exited with exit code 1 Started lustre-OST0000 oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg425-client.virtnet: -o user_xattr,flock 192.168.204.125@tcp:/lustre /mnt/lustre setup single mount lustre success stop mds service on oleg425-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg425-server sleep 60 sec fail_loc=0x8000050f sleep 10 sec manual umount lustre on /mnt/lustre.... df: '/mnt/lustre': Cannot send after transport endpoint shutdown fail_loc=0x0 start mds service on oleg425-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg425-server: oleg425-server.virtnet: executing set_default_debug -1 all pdsh@oleg425-client: oleg425-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg425-client.virtnet: -o user_xattr,flock 192.168.204.125@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg425-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg425-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg425-server stop mds service on oleg425-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg425-server unloading modules via unload_modules_local on: 'oleg425-server' oleg425-server: oleg425-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg425-client: oleg425-client: ssh exited with exit code 2 pdsh@oleg425-client: oleg425-server: ssh exited with exit code 2 pdsh@oleg425-client: oleg425-client: ssh exited with exit code 2 pdsh@oleg425-client: oleg425-server: ssh exited with exit code 2 PASS 45 (139s) == conf-sanity test 69: replace an OST with the same index ========================================================== 21:04:53 (1746147893) start mds service on oleg425-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg425-server' oleg425-server: oleg425-server.virtnet: executing load_modules_local oleg425-server: Loading modules from /home/green/git/lustre-release/lustre oleg425-server: detected 4 online CPUs by sysfs oleg425-server: MODOPTS_LIBCFS= oleg425-server: Force libcfs to create 2 CPU partitions oleg425-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg425-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg425-server: oleg425-server.virtnet: executing set_default_debug -1 all pdsh@oleg425-client: oleg425-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg425-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg425-server: oleg425-server.virtnet: executing set_default_debug -1 all pdsh@oleg425-client: oleg425-server: ssh exited with exit code 1 Started lustre-OST0000 oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg425-client.virtnet: -o user_xattr,flock 192.168.204.125@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11872 inodes available. Want 99872. rc=0 - open/close 1125 (time 1746147932.48 total 10.01 last 112.38) - open/close 2267 (time 1746147942.49 total 20.02 last 114.04) - open/close 3445 (time 1746147952.51 total 30.04 last 117.63) - open/close 4736 (time 1746147962.51 total 40.04 last 129.02) - open/close 5985 (time 1746147972.52 total 50.05 last 124.83) - open/close 7482 (time 1746147982.52 total 60.05 last 149.65) - open/close 8780 (time 1746147992.53 total 70.06 last 129.68) - open/close 9949 (time 1746148002.53 total 80.06 last 116.88) total: 10000 open/close in 80.50 seconds: 124.22 ops/second - unlinked 0 (time 1746148004 ; total 0 ; last 0) total: 10000 unlinks in 51 seconds: 196.078430 unlinks/second - open/close 1229 (time 1746148067.62 total 10.00 last 122.84) - open/close 2356 (time 1746148077.62 total 20.01 last 112.66) - open/close 3458 (time 1746148087.63 total 30.02 last 110.09) - open/close 4613 (time 1746148097.64 total 40.02 last 115.46) - open/close 5732 (time 1746148107.64 total 50.03 last 111.85) - open/close 6918 (time 1746148117.64 total 60.03 last 118.58) - open/close 8289 (time 1746148127.65 total 70.03 last 137.04) - open/close 9781 (time 1746148137.65 total 80.03 last 149.20) total: 10000 open/close in 81.45 seconds: 122.78 ops/second - unlinked 0 (time 1746148140 ; total 0 ; last 0) total: 10000 unlinks in 32 seconds: 312.500000 unlinks/second - open/close 1643 (time 1746148183.98 total 10.00 last 164.28) - open/close 3468 (time 1746148193.98 total 20.00 last 182.50) - open/close 5080 (time 1746148203.99 total 30.01 last 161.10) - open/close 6833 (time 1746148213.99 total 40.01 last 175.26) - open/close 8343 (time 1746148223.99 total 50.01 last 150.98) - open/close 9561 (time 1746148233.99 total 60.01 last 121.77) total: 10000 open/close in 64.17 seconds: 155.85 ops/second - unlinked 0 (time 1746148240 ; total 0 ; last 0) total: 10000 unlinks in 44 seconds: 227.272720 unlinks/second - open/close 1461 (time 1746148295.51 total 10.01 last 145.99) - open/close 2568 (time 1746148305.52 total 20.01 last 110.68) - open/close 3801 (time 1746148315.52 total 30.01 last 123.28) - open/close 4864 (time 1746148325.52 total 40.02 last 106.24) - open/close 5971 (time 1746148335.53 total 50.02 last 110.68) - open/close 7083 (time 1746148345.54 total 60.03 last 111.08) - open/close 8335 (time 1746148355.54 total 70.03 last 125.16) - open/close 9542 (time 1746148365.54 total 80.03 last 120.69) total: 10000 open/close in 83.24 seconds: 120.13 ops/second - unlinked 0 (time 1746148370 ; total 0 ; last 0) total: 10000 unlinks in 48 seconds: 208.333328 unlinks/second - open/close 1156 (time 1746148430.38 total 10.01 last 115.53) - open/close 2190 (time 1746148440.41 total 20.03 last 103.14) - open/close 3389 (time 1746148450.41 total 30.04 last 119.81) - open/close 4545 (time 1746148460.41 total 40.04 last 115.59) - open/close 5651 (time 1746148470.42 total 50.05 last 110.51) - open/close 6960 (time 1746148480.42 total 60.05 last 130.90) - open/close 8623 (time 1746148490.43 total 70.05 last 166.27) total: 10000 open/close in 77.87 seconds: 128.42 ops/second - unlinked 0 (time 1746148499 ; total 0 ; last 0) total: 10000 unlinks in 30 seconds: 333.333344 unlinks/second - open/close 1273 (time 1746148541.14 total 10.01 last 127.14) - open/close 2609 (time 1746148551.15 total 20.02 last 133.47) - open/close 3999 (time 1746148561.15 total 30.03 last 138.95) - open/close 5294 (time 1746148571.16 total 40.03 last 129.44) - open/close 6352 (time 1746148581.16 total 50.03 last 105.76) - open/close 7456 (time 1746148591.17 total 60.04 last 110.30) - open/close 8680 (time 1746148601.17 total 70.04 last 122.39) - open/close 9739 (time 1746148611.18 total 80.05 last 105.79) total: 10000 open/close in 82.44 seconds: 121.30 ops/second - unlinked 0 (time 1746148615 ; total 0 ; last 0) total: 10000 unlinks in 42 seconds: 238.095245 unlinks/second - open/close 1326 (time 1746148668.89 total 10.00 last 132.54) - open/close 2840 (time 1746148678.89 total 20.01 last 151.38) - open/close 4292 (time 1746148688.89 total 30.01 last 145.12) - open/close 5720 (time 1746148698.90 total 40.01 last 142.79) - open/close 7243 (time 1746148708.90 total 50.02 last 152.23) - open/close 8587 (time 1746148718.91 total 60.03 last 134.23) - open/close 9642 (time 1746148728.92 total 70.03 last 105.47) total: 10000 open/close in 72.48 seconds: 137.97 ops/second - unlinked 0 (time 1746148732 ; total 0 ; last 0) total: 10000 unlinks in 46 seconds: 217.391312 unlinks/second - open/close 1280 (time 1746148790.52 total 10.00 last 127.94) - open/close 2494 (time 1746148800.52 total 20.01 last 121.35) - open/close 3526 (time 1746148810.53 total 30.01 last 103.15) - open/close 4617 (time 1746148820.53 total 40.01 last 109.10) - open/close 5800 (time 1746148830.53 total 50.02 last 118.27) - open/close 7026 (time 1746148840.53 total 60.02 last 122.55) - open/close 8180 (time 1746148850.54 total 70.02 last 115.35) - open/close 9495 (time 1746148860.54 total 80.03 last 131.47) total: 10000 open/close in 83.31 seconds: 120.04 ops/second - unlinked 0 (time 1746148865 ; total 0 ; last 0) total: 10000 unlinks in 51 seconds: 196.078430 unlinks/second - open/close 1285 (time 1746148928.55 total 10.00 last 128.48) - open/close 2583 (time 1746148938.55 total 20.00 last 129.79) - open/close 3866 (time 1746148948.55 total 30.01 last 128.25) - open/close 5031 (time 1746148958.56 total 40.01 last 116.46) - open/close 6475 (time 1746148968.56 total 50.01 last 144.32) - open/close 7735 (time 1746148978.57 total 60.02 last 125.91) - open/close 8860 (time 1746148988.58 total 70.03 last 112.44) total: 10000 open/close in 78.08 seconds: 128.07 ops/second - unlinked 0 (time 1746148998 ; total 0 ; last 0) total: 10000 unlinks in 36 seconds: 277.777771 unlinks/second - open/close 1412 (time 1746149046.16 total 10.01 last 141.10) - open/close 3184 (time 1746149056.17 total 20.01 last 177.12) - open/close 4794 (time 1746149066.17 total 30.01 last 160.99) - open/close 6120 (time 1746149076.17 total 40.02 last 132.50) - open/close 7337 (time 1746149086.18 total 50.02 last 121.66) - open/close 8838 (time 1746149096.19 total 60.03 last 149.97) total: 10000 open/close in 68.75 seconds: 145.46 ops/second - unlinked 0 (time 1746149106 ; total 0 ; last 0) total: 10000 unlinks in 47 seconds: 212.765961 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg425-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg425-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg425-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.204.125@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.204.125@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg425-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg425-server: oleg425-server.virtnet: executing set_default_debug -1 all pdsh@oleg425-client: oleg425-server: ssh exited with exit code 1 Started lustre-OST0000 oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg425-server: oleg425-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg425-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 6 sec mount lustre on /mnt/lustre..... Starting client: oleg425-client.virtnet: -o user_xattr,flock 192.168.204.125@tcp:/lustre /mnt/lustre On OST0, 17717 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg425-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg425-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg425-server stop mds service on oleg425-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg425-server unloading modules via unload_modules_local on: 'oleg425-server' oleg425-server: oleg425-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg425-client: oleg425-client: ssh exited with exit code 2 pdsh@oleg425-client: oleg425-server: ssh exited with exit code 2 pdsh@oleg425-client: oleg425-client: ssh exited with exit code 2 pdsh@oleg425-client: oleg425-server: ssh exited with exit code 2 PASS 69 (1307s) == conf-sanity test 111: Adding large_dir with over 2GB directory ========================================================== 21:26:40 (1746149200) SKIP: conf-sanity test_111 Only applicable to ldiskfs-based MDTs SKIP 111 (1s) Stopping clients: oleg425-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg425-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg425-client: oleg425-server: ssh exited with exit code 2 oleg425-server: oleg425-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg425-server' oleg425-server: oleg425-server.virtnet: executing load_modules_local oleg425-server: Loading modules from /home/green/git/lustre-release/lustre oleg425-server: detected 4 online CPUs by sysfs oleg425-server: MODOPTS_LIBCFS= oleg425-server: Force libcfs to create 2 CPU partitions oleg425-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg425-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 1551 sec ========== 21:27:08 (1746149228) === conf-sanity: start cleanup 21:27:08 (1746149228) === === conf-sanity: finish cleanup 21:27:10 (1746149230) ===