-----============= acceptance-small: conf-sanity ============----- Fri Apr 25 18:23:38 EDT 2025 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg328-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex: No such file or directory excepting tests: 32b 32c 32newtarball 108a 112a 112b 113 117 119 121 122a 123aa 123ab 123ac 123ad 123ae 123af 123ag 123ah 123ahi 123F 123G 123H 126 129 132 133 135 136 137 150 152 153a 153b 153c 155 802a Stopping clients: oleg328-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg328-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg328-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg328-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg328-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg328-server oleg328-server: oleg328-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg328-server' oleg328-server: oleg328-server.virtnet: executing load_modules_local oleg328-server: Loading modules from /home/green/git/lustre-release/lustre oleg328-server: detected 4 online CPUs by sysfs oleg328-server: MODOPTS_LIBCFS= oleg328-server: Force libcfs to create 2 CPU partitions oleg328-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg328-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg328-server: oleg328-server.virtnet: executing set_default_debug -1 all pdsh@oleg328-client: oleg328-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg328-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg328-server: oleg328-server.virtnet: executing set_default_debug -1 all pdsh@oleg328-client: oleg328-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg328-server: oleg328-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg328-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg328-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg328-server stop mds service on oleg328-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg328-server == conf-sanity test 45: long unlink handling in ptlrpcd == 18:25:08 (1745619908) start mds service on oleg328-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg328-server: oleg328-server.virtnet: executing set_default_debug -1 all pdsh@oleg328-client: oleg328-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg328-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg328-server: oleg328-server.virtnet: executing set_default_debug -1 all pdsh@oleg328-client: oleg328-server: ssh exited with exit code 1 Started lustre-OST0000 oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg328-client.virtnet: -o user_xattr,flock 192.168.203.128@tcp:/lustre /mnt/lustre setup single mount lustre success stop mds service on oleg328-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg328-server sleep 60 sec fail_loc=0x8000050f sleep 10 sec manual umount lustre on /mnt/lustre.... df: '/mnt/lustre': Cannot send after transport endpoint shutdown fail_loc=0x0 start mds service on oleg328-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg328-server: oleg328-server.virtnet: executing set_default_debug -1 all pdsh@oleg328-client: oleg328-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg328-client.virtnet: -o user_xattr,flock 192.168.203.128@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg328-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg328-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg328-server stop mds service on oleg328-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg328-server unloading modules via unload_modules_local on: 'oleg328-server' oleg328-server: oleg328-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg328-client: oleg328-server: ssh exited with exit code 2 pdsh@oleg328-client: oleg328-client: ssh exited with exit code 2 pdsh@oleg328-client: oleg328-server: ssh exited with exit code 2 pdsh@oleg328-client: oleg328-client: ssh exited with exit code 2 PASS 45 (138s) == conf-sanity test 69: replace an OST with the same index ========================================================== 18:27:26 (1745620046) start mds service on oleg328-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg328-server' oleg328-server: oleg328-server.virtnet: executing load_modules_local oleg328-server: Loading modules from /home/green/git/lustre-release/lustre oleg328-server: detected 4 online CPUs by sysfs oleg328-server: MODOPTS_LIBCFS= oleg328-server: Force libcfs to create 2 CPU partitions oleg328-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg328-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg328-server: oleg328-server.virtnet: executing set_default_debug -1 all pdsh@oleg328-client: oleg328-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg328-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg328-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg328-server: oleg328-server.virtnet: executing set_default_debug -1 all pdsh@oleg328-client: oleg328-server: ssh exited with exit code 1 Started lustre-OST0000 oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg328-client.virtnet: -o user_xattr,flock 192.168.203.128@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11872 inodes available. Want 99872. rc=0 - open/close 847 (time 1745620090.18 total 10.01 last 84.63) - open/close 1797 (time 1745620100.18 total 20.01 last 94.94) - open/close 2664 (time 1745620110.19 total 30.02 last 86.65) - open/close 3569 (time 1745620120.20 total 40.02 last 90.46) - open/close 4525 (time 1745620130.20 total 50.03 last 95.58) - open/close 5424 (time 1745620140.20 total 60.03 last 89.90) - open/close 6493 (time 1745620150.20 total 70.03 last 106.83) - open/close 7674 (time 1745620160.21 total 80.04 last 118.07) - open/close 8731 (time 1745620170.21 total 90.04 last 105.62) total: 10000 open/close in 98.62 seconds: 101.40 ops/second - unlinked 0 (time 1745620180 ; total 0 ; last 0) total: 10000 unlinks in 40 seconds: 250.000000 unlinks/second - open/close 1745 (time 1745620232.13 total 10.00 last 174.47) - open/close 3428 (time 1745620242.14 total 20.01 last 168.24) - open/close 4847 (time 1745620252.14 total 30.01 last 141.84) - open/close 6445 (time 1745620262.15 total 40.02 last 159.62) - open/close 7577 (time 1745620272.16 total 50.03 last 113.16) - open/close 9197 (time 1745620282.16 total 60.03 last 161.93) total: 10000 open/close in 65.18 seconds: 153.42 ops/second - unlinked 0 (time 1745620288 ; total 0 ; last 0) total: 10000 unlinks in 38 seconds: 263.157898 unlinks/second - open/close 1724 (time 1745620337.68 total 10.00 last 172.40) - open/close 3188 (time 1745620347.68 total 20.00 last 146.34) - open/close 4489 (time 1745620357.68 total 30.01 last 130.08) - open/close 5986 (time 1745620367.69 total 40.01 last 149.64) - open/close 7856 (time 1745620377.69 total 50.01 last 186.94) - open/close 9685 (time 1745620387.70 total 60.02 last 182.82) total: 10000 open/close in 62.21 seconds: 160.74 ops/second - unlinked 0 (time 1745620391 ; total 0 ; last 0) total: 10000 unlinks in 32 seconds: 312.500000 unlinks/second - open/close 2046 (time 1745620435.06 total 10.00 last 204.50) - open/close 4127 (time 1745620445.07 total 20.01 last 208.02) - open/close 6660 (time 1745620455.07 total 30.01 last 253.22) - open/close 9346 (time 1745620465.07 total 40.01 last 268.52) total: 10000 open/close in 42.94 seconds: 232.89 ops/second - unlinked 0 (time 1745620469 ; total 0 ; last 0) total: 10000 unlinks in 28 seconds: 357.142853 unlinks/second - open/close 1745 (time 1745620508.97 total 10.00 last 174.46) - open/close 3476 (time 1745620518.98 total 20.01 last 172.97) - open/close 5593 (time 1745620528.98 total 30.01 last 211.63) - open/close 7603 (time 1745620538.99 total 40.02 last 200.90) - open/close 9184 (time 1745620549.00 total 50.03 last 157.88) - open/close 9884 (time 1745620559.02 total 60.05 last 69.87) total: 10000 open/close in 61.49 seconds: 162.62 ops/second - unlinked 0 (time 1745620562 ; total 0 ; last 0) total: 10000 unlinks in 58 seconds: 172.413788 unlinks/second - open/close 965 (time 1745620632.15 total 10.00 last 96.46) - open/close 1869 (time 1745620642.15 total 20.01 last 90.39) - open/close 2873 (time 1745620652.15 total 30.01 last 100.35) - open/close 3951 (time 1745620662.15 total 40.01 last 107.78) - open/close 4954 (time 1745620672.16 total 50.01 last 100.29) - open/close 6140 (time 1745620682.16 total 60.02 last 118.51) - open/close 7222 (time 1745620692.16 total 70.02 last 108.19) - open/close 8599 (time 1745620702.17 total 80.02 last 137.68) - open/close 9904 (time 1745620712.17 total 90.02 last 130.48) total: 10000 open/close in 90.87 seconds: 110.05 ops/second - unlinked 0 (time 1745620714 ; total 0 ; last 0) total: 10000 unlinks in 47 seconds: 212.765961 unlinks/second - open/close 1376 (time 1745620772.88 total 10.01 last 137.51) - open/close 2742 (time 1745620782.89 total 20.01 last 136.54) - open/close 4307 (time 1745620792.89 total 30.01 last 156.48) - open/close 5653 (time 1745620802.89 total 40.01 last 134.59) - open/close 6949 (time 1745620812.89 total 50.01 last 129.59) - open/close 8250 (time 1745620822.89 total 60.02 last 130.04) - open/close 9536 (time 1745620832.90 total 70.03 last 128.51) total: 10000 open/close in 72.91 seconds: 137.16 ops/second - unlinked 0 (time 1745620837 ; total 0 ; last 0) total: 10000 unlinks in 40 seconds: 250.000000 unlinks/second - open/close 1406 (time 1745620889.30 total 10.00 last 140.59) - open/close 2826 (time 1745620899.30 total 20.00 last 141.99) - open/close 4623 (time 1745620909.31 total 30.00 last 179.64) - open/close 6268 (time 1745620919.31 total 40.01 last 164.49) - open/close 8233 (time 1745620929.31 total 50.01 last 196.42) total: 10000 open/close in 59.01 seconds: 169.46 ops/second - unlinked 0 (time 1745620939 ; total 0 ; last 0) total: 10000 unlinks in 34 seconds: 294.117645 unlinks/second - open/close 1731 (time 1745620984.73 total 10.00 last 173.04) - open/close 3684 (time 1745620994.74 total 20.02 last 195.04) - open/close 5537 (time 1745621004.76 total 30.03 last 185.05) - open/close 7626 (time 1745621014.76 total 40.04 last 208.80) - open/close 9751 (time 1745621024.77 total 50.04 last 212.44) total: 10000 open/close in 51.02 seconds: 195.99 ops/second - unlinked 0 (time 1745621026 ; total 0 ; last 0) total: 10000 unlinks in 21 seconds: 476.190491 unlinks/second - open/close 2542 (time 1745621058.87 total 10.01 last 254.01) - open/close 5437 (time 1745621068.88 total 20.01 last 289.40) - open/close 7835 (time 1745621078.88 total 30.01 last 239.71) - open/close 9819 (time 1745621088.88 total 40.02 last 198.37) total: 10000 open/close in 41.14 seconds: 243.09 ops/second - unlinked 0 (time 1745621091 ; total 0 ; last 0) total: 10000 unlinks in 30 seconds: 333.333344 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg328-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg328-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg328-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.203.128@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.203.128@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg328-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg328-server: oleg328-server.virtnet: executing set_default_debug -1 all pdsh@oleg328-client: oleg328-server: ssh exited with exit code 1 Started lustre-OST0000 oleg328-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg328-server: oleg328-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg328-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 8 sec mount lustre on /mnt/lustre..... Starting client: oleg328-client.virtnet: -o user_xattr,flock 192.168.203.128@tcp:/lustre /mnt/lustre On OST0, 14357 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg328-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg328-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg328-server stop mds service on oleg328-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg328-server unloading modules via unload_modules_local on: 'oleg328-server' oleg328-server: oleg328-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg328-client: oleg328-client: ssh exited with exit code 2 pdsh@oleg328-client: oleg328-server: ssh exited with exit code 2 pdsh@oleg328-client: oleg328-client: ssh exited with exit code 2 pdsh@oleg328-client: oleg328-server: ssh exited with exit code 2 PASS 69 (1118s) == conf-sanity test 111: Adding large_dir with over 2GB directory ========================================================== 18:46:04 (1745621164) SKIP: conf-sanity test_111 Only applicable to ldiskfs-based MDTs SKIP 111 (1s) Stopping clients: oleg328-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg328-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg328-client: oleg328-server: ssh exited with exit code 2 oleg328-server: oleg328-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg328-server' oleg328-server: oleg328-server.virtnet: executing load_modules_local oleg328-server: Loading modules from /home/green/git/lustre-release/lustre oleg328-server: detected 4 online CPUs by sysfs oleg328-server: MODOPTS_LIBCFS= oleg328-server: Force libcfs to create 2 CPU partitions oleg328-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg328-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 1373 sec ========== 18:46:32 (1745621192) === conf-sanity: start cleanup 18:46:33 (1745621193) === === conf-sanity: finish cleanup 18:46:34 (1745621194) ===